How storeBackup's replication works

If you wish to create advanced replication configurations, this section will be of interest to you. The following picture shows the principle of the data flow when replicating backups:

\includegraphics[scale=0.95]{replication}

Important: Before you can start replicating backups, you have to copy a common version of the backup from the master backup to the locations where the copies will reside. The reason for that is that replication is based on incremental backups (option lateLinks of storeBackup.pl), and they always have to be linked against the backup version to which these incremental backups were generated. You can use cp -a or (better) linkToDirs.pl to copy these (old) backups.

If your master backup is located at /masterBackup and you want to copy the series series1, series2 and series3 to /extDisk1/stbu (which will be the top level of your backup copy), you can run:51

# linkToDirs.pl /masterBackup/series1 /masterBackup/series2 /masterBackup/series3 \
  --targetDir /extDisk1/stbu

In the example shown in the picture above, there is a master backup where all the backups are stored. In this master backup, there are three different series called series1, series2 and series3. In each of these series, there are three incremental backups (shown as circles) created with option lateLinks and which have to be replicated.

The configuration file located in the top level directory of the master backup (storeBackupBaseTree.conf) has been configured that two (series1, series2) of the three series have to be copied onto the external disks. The configuration file would look e.g., as follows (without comments):

backupTreeName=myMasterBackup
backupType=master
seriesToDistribute=series1 series2
deltaCache=/deltaCache

You can generate this type of configuration file by typing

storeBackupUpdateBackup.pl --genBackupBaseTreeConf directory

where directory is the top level directory of your master backup. After its generation, edit the configuration file depending to your needs.

The example configuration above tells storeBackup (to be precise storeBackupUpdateBackup.pl) that you are calling your master backup myMasterBackup (choose a name whatever you want, it has nothing to do with a directory name - it is a unique identifier you choose and which will point to an actual directory) and that this is your master backup. The configuration file also defines, that series1 and series2 have to be copied and that the central hub, your ``deltaCache'' is in directory /deltacache. After copying the incremental backup via storeBackupUpdateBackup.pl just as described above, the same run of that program starts generating the missing hard links (etc.) in the master backup so the incremental backups become full backups.

In this example, I assume that the server (or generally the box) to save has two internal disks. On /, the file system and the user data are located, while the second disk, mounted at /backup stores the master backup (and maybe data you do not want to backup). To locate the ``deltaCache'' on the first disk which has to be saved makes sense: If the first (operating system) disk is broken, you can use the backup disk to restore it later. If the backup disk is broken, you still have the delta to the external disks on the first one. Naturally, if both are broken at the same time, you only have the external ones and may lose some days (if you do not do a daily sync).

In the deltaCache, you have to generate (see below) and configure a configuration file named deltaCache.conf in the top level directory with the following contents:

backupCopy0= 'extDisk1' series1 series2
backupCopy1= 'extDisk2' series1
;backupCopy2=
;backupCopy3=
;backupCopy4=
;backupCopy5=
;backupCopy6=
;backupCopy7=
;backupCopy8=
;backupCopy9=

You have to define two copy targets because in this example you want to make two copies. The first line (backupCopy0) in the configuration file above defines that series1 and series2 have to be copied to the backup copy with the unique identifier extDisk1. The second line (backupCopy1) tells storeBackup to copy series1 to the backup with the unique identifier extDisk2.

You can generate this type of configuration file by typing

storeBackupUpdateBackup.pl --genCopyStationConf directory

where directory is the top level directory of your deltaCache. After its generation, edit the configuration file depending to your needs.

Finally, you have to edit a configuration file for the place where you want to replicate your data. As (in this example) you want to replicate to two different storage systems (called extDisk1 and extDisk2) you have to generate two configurations files in the top level of these replicas:

storeBackupUpdateBackup.pl --genBackupBaseTreeConf directory

Where directory is the top level directory of your backup copy. The generated configuration file with the name storeBackupBaseTree.conf will be stored in that top level directory. After its generation, edit the configuration file depending to your needs; in this example it is:

backupTreeName=extDisk1
backupType=copy
seriesToDistribute= series1 series2
deltaCache=/deltaCache

The Options backupTreeName and seriesToDistribute must fit to the corresponding entry of deltaCache.conf (see above, backupCopy0).

When you call:

storeBackupUpdateBackup.pl --backupDir directory

where directory is the top level backup directory (with the series below), it will copy the backups in deltaCache from the configured series to the specified location.

You might want to add some other options to storeBackupUpdateBackup.pl, e.g.:

storeBackupUpdateBackup.pl --progressReport 200 --archiveDurationCopyStation 32d -b directory

This means, it will print a progress report and delete backups in deltaCache after a month, but only if they were delivered and hard linked successfully to the replica backups.

Finally, you have to repeat the last step of configuring a configuration file for extDisk2, too:

backupTreeName=extDisk2
backupType=copy
seriesToDistribute= series1
deltaCache=/deltaCache

That's it. Now each run of storeBackupUpdateBackup.pl on the master backup will copy the required backup deltas (= backups generated with lateLinks) and finalize the backups to be complete ones (not incremental ones). Running storeBackupUpdateBackup.pl on the replica backups (extDisk1 and extDisk2) will copy those deltas from the deltaCache to the selected replica. Finally, one of those calls of storeBackupUpdateBackup.pl will delete those deltas from deltaCache (depending on option --archiveDurationCopyStation).

Heinz-Josef Claes 2014-04-20