Why copying backups is not a substitute for replication functionality

We will make this discussion less abstract by providing an example. A common backup strategy is to run your backups to the master backup respository every day (maybe automatically via cron) and to move (or rotate) this backup offsite (or at least to another physically separate media) periodically. For illustration purposes, let's assume you have a separate backup disk stored offsite that you will update weekly.³² Once each week you will add the newly made backups from your master backup to the backup copy on the external(offsite) disk (which you temporarily bring onsite and connect to your computer).

Copying the new backup only with ``cp -a'' is a bad idea, because the newly copied directories (backups) will not be hard linked to the existing ones on the external disk.³³ You can use linkToDirs.pl to link (and copy) the new backups in the master backup to the existing ones in the backup copy on the external disk. Using linkToDirs.pl is nice for ad hoc replications, but not the best for planned and automated ones.

Another common way to copy the new backups to the external disk is to use synchronization tools like rsync. There are two issues with this approach: 1^st, it takes very long if you have lots of backups and 2^nd, you will replicate every fault on you master backup disk to your backup, and that's really not what you want. Imagine, your disk for the master backup gets a block error in a file from the backup one month ago. So the affected file is broken in your backup. If you now synchronize the disk with e.g. rsync, you will copy the broken file. In the worst of all cases, you can destroy your whole Backup by this method (without getting more security). If you use the replication from storeBackup, old data is not affected in the replication.

BUT STOP: What if the newly copied data is broken because some sectors of the disk are seriously broken or you have to deal with broken RAM or any other reason which leads to incorrect data in your master backup? Will you ever determine that parts of your data in the backup are broken? The backup program storeBackup will tell you the same as rsync about that - nothing, because it is not in their control. For this reason you should run storeBackupCheckBackup.pl which recalculates check sums for every file periodically on your backup(s). By running this program, you are able to see faults in old backups which you are able to correct manually if you have a replica. And you are able to see in an early stage if your new backups are broken. Therefore, we suggest to run storeBackupCheckBackup.pl on new backups every week or so on the master backup and on the copy plus to run it on old backups (which may take a long time) every few months.

If you recognize errors on your hard disk, you should investigate deeper into the problem and not hesitate to replace the disk.

The basic idea of storeBackup's replication feature is to solve the issues described above. A replication means we have the same state in two different locations (e.g., in the master backup and in the backup copy). That's what we have done in the description above with the cp -a command. Let's say, this was the backup from Monday. After a day we have a change (a new backup on Tuesday) in the master backup. For the replication, we need just the differences between the backup from Monday and the backup from Tuesday. If we have some clever algorithm to get all the changes (deltas) from the backup of Monday to the backup of Tuesday, we could transport these changes to the backup copy on the external disk and rebuild the full backup (with all links, permissions and so on) on the external disk. As a result, the backup on the external disk contains exactly the same information as the master backup.

If we want to connect the external disk only once a week³⁴ we need a place to store the differences. We will have these deltas from Monday to Tuesday, from Tuesday to Wednesday etc. What we are doing, is to rebuild the complete and full backups on the backup copy disk, e.g.:

• Backup from Tuesday $\longrightarrow$ rebuild from full backup from Monday plus Deltas from Monday to Tuesday
• Backup from Wednesday $\longrightarrow$ rebuild from full backup from Tuesday plus Deltas from Tuesday to Wednesday
• Backup from Thursday $\longrightarrow$ rebuild from full backup from Wednesday plus Deltas from Wednesday to Thursday
• ...

This means we need the deltas between two consecutive backups in the master backup. In principle, there are two ways to get these:

1. Calculation of the differences, which means something like a ``reverse de-duplication'' (storeBackup searches for files (or part of files) with identical contents and hard links them).
2. Identification of the differences directly when creating the backups. At that point, the differences are calculated and known.

The second way is the typical way used for replications, e.g. used in database or LDAP replication. StoreBackup also uses this way to replicate.

StoreBackup generates deltas to (one or more) existing backups with the option lateLinks and temporarily stores them in a "delta cache".³⁵ (See section 7.6 for more information about how to configure it.)

The storeBackup replication functionality provides the following features:

• Replication of backups takes advantage of storeBackup's existing capability to store just the differences to the former backup(s) plus the information needed to reconstruct this to a full backup.
• You can configure which series have to be replicated. At the master backup you do not have to know the number of copies you want to make. This provides decoupling of source (master backup) and target (backup copy)).
• Replication of backups can be fully automated. Along with several configuration files that are needed for replication, you simply use cron (or similar) to run storeBackupUpdateBackup.pl. You also can run storeBackupDel.pl to automatically delete old backups in the replication.
• replica disk(s) do not need to be permanently attached. You can decide if you want to connect one or more of the replica disk(s) permanently or not (so you need manual intervention).) This may depend on the location you replicate to (external via WAN or via LAN) and your backup strategy.
• For security reasons, you can set up the different storage places in a way that the master backup does not have to be accessible from the copies (and vice versa).
• Replication is asynchronous. This means you may connect your disk1 for backup copy number 1 on even weekends and backup copy number 2 on disk2 on uneven weekends. But as a result you get the same data on both backup disks, regardless of your individual deletion scheme of each backup including the master backup.
• Deletion of old backups on the replicated backups can be done for replica(s) and master backup with the same tool (storeBackupDel.pl). Especially, you can use different deletion schemes, e.g., to additionally store very long term backups on your (slower) replica medium(s).
• Replicas behave like normal storeBackup backups. Naturally, you can run storeBackupCheckBackup.pl on the replicas, too.
• It is robust. If something goes wrong with the replication (so the deltas are lost on the way to the replica backup) for whatever reason, you can use linkToDirs.pl to generate a new identical status (backup version) to be able to continue with replication of backups. You can do this without copying everything (from the master backup or from another replica) by just copying / hard linking missing delta. (But you need direct access between the affected backups / replicas during that time or you have to perform an intermediate copy.) Another advantage of this proceeding is that you do not copy possible faults in older backups from one backup copy to the other one, like you possibly would when using standard synchronization tools.

In short, if you make a ``normal'' backup (without replication) with storeBackup.pl, you typically have one place (see option backupDir) where you store your backups. This will be called the master backup. (It is the same as what we have called the master backup repository in other sections of this document.) If the disk (or e.g., the file) system for this ``master backup'' fails, you will lose the backup and therefore the history of your data. It is a form of data loss that can be prevented with storeBackup's replication feature.