
= Using Multiple Backups for Relax-and-Recover

== Basics

Currently multiple backups are only supported for:

* the internal BACKUP=NETFS method with BACKUP_TYPE=""
* the internal BACKUP=BLOCKCLONE method
* the external BACKUP=BORG method

In general multiple backups are not supported for
BACKUP_TYPE=incremental or BACKUP_TYPE=differential
because those require special backup archive file names.

=== The basic idea behind

A "rear mkbackup" run can be split into
a "rear mkrescue" run plus a "rear mkbackuponly" run
and the result is still the same.

Accordingly "rear mkbackup" can be split into
a single "rear mkrescue" plus multiple "rear mkbackuponly"
where each particular "rear mkbackuponly" backups only a
particular part of the files of the system, for example:

* a backup of the files of the basic system
* a backup of the files in the /home directories
* a backup of the files in the /opt directory

Multiple "rear mkbackuponly" require that each particular
"rear mkbackuponly" uses a specific ReaR configuration file
that specifies how that particular "rear mkbackuponly" must be done.

Therefore the '-C' command line parameter is needed where
an additional ReaR configuration file can be specified.

=== The basic way how to create multiple backups

Have common settings in /etc/rear/local.conf

For each particular backup specify its parameters in
a separated additional configuration files like

----
/etc/rear/basic_system.conf
/etc/rear/home_backup.conf
/etc/rear/opt_backup.conf
----

First create the ReaR recovery/rescue system ISO image
together with a backup of the files of the basic system:

----
rear -C basic_system mkbackup
----

Then backup the files in the /home directories:

----
rear -C home_backup mkbackuponly
----

Afterwards backup the files in the /opt directory:

----
rear -C opt_backup mkbackuponly
----

=== The basic way how to recover with multiple backups

The basic idea how to recover with multiple backups is
to split the "rear recover" into an initial recovery
of the basic system followed by several backup restore
operations as follows:

Boot the ReaR recovery/rescue system.

In the ReaR recovery/rescue system do the following:

First recover the basic system:

----
rear -C basic_system recover
----

Then restore the files in the /home directories:

----
rear -C home_backup restoreonly
----

Afterwards restore the files in the /opt directory:

----
rear -C opt_backup restoreonly
----

Finally reboot the recreated system.

For more internal details and some background information see
https://github.com/rear/rear/issues/1088

== Relax-and-Recover Setup for Multiple Backups

Assume for example multiple backups should be done
using the NETFS backup method with 'tar' as backup program
to get separated backups for:

* the files of the basic system
* the files in the /home directories
* the files in the /opt directory

Those four configuration files could be used:

./etc/rear/local.conf
[source,bash]
----
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://your.NFS.server.IP/path/to/your/rear/backup
----

./etc/rear/basic_system.conf
[source,bash]
----
this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/home/*' '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"
----

./etc/rear/home_backup.conf
[source,bash]
----
this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/home/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"
----

./etc/rear/opt_backup.conf
[source,bash]
----
this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"
----

The BACKUP_ONLY_INCLUDE setting is described in conf/default.conf.

With those config files creating the ReaR recovery/rescue system ISO image
and subsequently backup the files of the system could be done like:

----
rear mkrescue
rear -C basic_system mkbackuponly
rear -C home_backup mkbackuponly
rear -C opt_backup mkbackuponly
----

Recovery of that system could be done by calling in the
ReaR recovery/rescue system:

----
rear -C basic_system recover
rear -C home_backup restoreonly
rear -C opt_backup restoreonly
----

Note that system recovery with multiple backups requires that
first and foremost the basic system is recovered where all files
must be restored that are needed to install the bootloader and
to boot the basic system into a normal usable state.

Nowadays systemd usually needs files in the /usr directory
so that in practice in particular all files in the /usr directory
must be restored during the initial basic system recovery
plus whatever else is needed to boot and run the basic system.

Multiple backups cannot be used to spilt the files of the basic system
into several backups. The files of the basic system must be in one
single backup and that backup must be restored during the initial
recovery of the basic system.

== Relax-and-Recover Setup for Different Backup Methods

Because multiple backups are used via separated additional
configuration files, different backup methods can be used.

Assume for example multiple backups should be used to get
separated backups for the files of the basic system
using the NETFS backup method with 'tar' as backup program
and to backup the files in the /home directory
using the BORG backup method.

The configuration files could be like the following:

./etc/rear/local.conf
[source,bash]
----
OUTPUT=ISO
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" borg locale )
COPY_AS_IS=( "${COPY_AS_IS[@]}" "/borg/keys" )
----

./etc/rear/basic_system.conf
[source,bash]
----
this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/home/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://your.NFS.server.IP/path/to/your/rear/backup
----

./etc/rear/home_backup.conf
[source,bash]
----
this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP=BORG
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/home/*' )
BORGBACKUP_ARCHIVE_PREFIX="rear"
BORGBACKUP_HOST="borg.server.name"
BORGBACKUP_USERNAME="borg_server_username"
BORGBACKUP_REPO="/path/to/borg/repository/on/borg/server"
BORGBACKUP_PRUNE_HOURLY=5
BORGBACKUP_PRUNE_WEEKLY=2
BORGBACKUP_COMPRESSION="zlib,9"
BORGBACKUP_ENC_TYPE="keyfile"
export BORG_KEYS_DIR="/borg/keys"
export BORG_CACHE_DIR="/borg/cache"
export BORG_PASSPHRASE="a1b2c3_d4e5f6"
export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="yes"
export BORG_REMOTE_PATH="/usr/local/bin/borg"
----

Using different backup methods requires to get all the binaries
and all other needed files of all used backup methods into the
ReaR recovery/rescue system during "rear mkbackup/mkrescue".

Those binaries and other needed files must be manually specified
via REQUIRED_PROGS and COPY_AS_IS in /etc/rear/local.conf
(regarding REQUIRED_PROGS and COPY_AS_IS see conf/default.conf).

With those config files creating the ReaR recovery/rescue system ISO image
together with a 'tar' backup of the files of the basic system and
a separated Borg backup of the files in /home could be done like:

----
rear -C home_backup mkbackuponly
rear -C basic_system mkbackup
----

In contrast to the other examples above the Borg backup is run first
because Borg creates encryption keys during repository initialization.
This ensures the right /borg/keys is created before it will be copied into
the ReaR recovery/rescue system by the subsequent "rear mkbackup/mkrescue".
Alternatively the ReaR recovery/rescue system could be created again
after the Borg backup is done like:

----
rear -C basic_system mkbackup
rear -C home_backup mkbackuponly
rear -C basic_system mkrescue
----

Recovery of that system could be done by calling in the
ReaR recovery/rescue system:

----
rear -C basic_system recover
rear -C home_backup restoreonly
----

== Running Multiple Backups and Restores in Parallel

When the files in multiple backups are separated from each other
it should work to run multiple backups or multiple restores in parallel.

Whether or not that actually works in your particular case
depends on how you made the backups in your particular case.

For sufficiently well separated backups it should work
to run multiple different

----
rear -C backup_config mkbackuponly
----

or multiple different

----
rear -C backup_config restoreonly
----

in parallel.

Running in parallel is only supported for mkbackuponly and restoreonly.

For example like

----
rear -C backup1 mkbackuponly & rear -C backup2 mkbackuponly & wait
----

or

----
rear -C backup1 restoreonly & rear -C backup2 restoreonly & wait
----

ReaR's default logging is not prepared for multiple simultaneous runs
and also ReaR's current progress subsystem is not prepared for that.
On the terminal the messages from different simultaneous runs are
indistinguishable and the current progress subsystem additionally
outputs subsequent messages on one same line which results
illegible and meaningless output on the terminal.

Therefore additional parameters must be set to make ReaR's messages
and the progress subsystem output appropriate for parallel runs.

Simultaneously running ReaR workflows require unique messages and
unique logfile names.

Therefore the PID ('$$') is specified to be used as message prefix
for all ReaR messages and it is also added to the LOGFILE value.

The parameters MESSAGE_PREFIX PROGRESS_MODE and PROGRESS_WAIT_SECONDS
are described in conf/default.conf.

For example a setup for parallel runs of mkbackuponly and restoreonly
could look like the following:

./etc/rear/local.conf
[source,bash]
----
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://your.NFS.server.IP/path/to/your/rear/backup
MESSAGE_PREFIX="$$: "
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="3"
----

./etc/rear/basic_system.conf
[source,bash]
----
this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}-$$.log"
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/home/*' '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"
----

./etc/rear/home_backup.conf
----
this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}-$$.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/home/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"
----

./etc/rear/opt_backup.conf
[source,bash]
----
this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}-$$.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"
----

With those config files creating the ReaR recovery/rescue system ISO image
together with a backup of the files of the basic system and then
backup the files in /home and /opt in parallel could be done like:

----
rear -C basic_system mkbackup
rear -C home_backup mkbackuponly & rear -C opt_backup mkbackuponly & wait
----

Recovery of that system could be done by calling in the
ReaR recovery/rescue system:

----
rear -C basic_system recover
rear -C home_backup restoreonly & rear -C opt_backup restoreonly & wait
----

Even on a relatively small system with a single CPU
running multiple backups and restores in parallel
can be somewhat faster compared to sequential processing.

On powerful systems with multiple CPUs, much main memory, fast storage access,
and fast access to the backups it is in practice mandatory to split
a single huge backup of the whole system into separated parts and
run at least the restores in parallel to utilize powerful hardware
and be as fast as possible in case of emergency and time pressure
during a real disaster recovery.

Remember that system recovery with multiple backups requires that
first and foremost the basic system is recovered where all files
must be restored that are needed to install the bootloader and
to boot the basic system into a normal usable state so that
'rear recover' cannot run in parallel with 'rear restoreonly'.

