Overview
While Vultr does have a backup system for entire system images, this works at the block level, and requires that the backup is restored to a VM before the data can be restored. Additionally, backups are only stored for a short amount of time, and do not provide a true incremental backup solution.
Enter Duplicity – Duplicity builds upon the legendary rsync and provides the ability to store incremental backups along with encrypting the data at rest via GPG. It is Posix compliant, and only transfers deltas between backup runs, thus reducing the overall bandwidth requirement.
Setting up environment
Terminology
- Source host – Server which will have its data backed up. For this tutorial, it has the IP address: 10.1.10.1
- Backup host – Destination server for backups. For this tutorial, it has the IP address: 10.1.10.2
- /backupdir – Source directory on source host used for backups in this tutorial. You can change this to match your environment.
- /destdir – Destination directory on backup host used for backups in this tutorial. You can change this to match your environment.
- Full backup – Entire copy of the source dataset.
- Incremental backup – Copy of all the changes made since the last backup.
Installing Duplicity
Ubuntu 14.04:
sudo apt-get update
sudo apt-get install duplicity python-paramiko
CentOS (requires EPEL):
sudo yum install duplicity python-paramiko
Ubuntu 12.04/Debian 7:
sudo apt-get update
sudo apt-get install ncftp python-paramiko python-pycryptopp lftp python-boto python-dev librsync-dev
wget https://launchpad.net/duplicity/0.7-series/0.7.02/+download/duplicity-0.7.02.tar.gz
tar xzvf duplicity*
cd duplicity*
sudo python setup.py install
We have to install from source as the Duplicity package within Debian 7 and Ubuntu 12.04 are broken due to a change in the backend SSH library.
Double check that Duplicity is installed by running:
duplicity -v
It should return the following output (version may be different):
duplicity 0.6.18
Setting up key-less authentication for SSH
The next step is to setup certificate-based authentication for SSH between the backup host and the source host. This will enable the source server to SSH into the backup host without typing in a passphrase. Vultr has a great article which explains how to do this: How Do I Generate SSH Keys.
Backups
Running the first (full) backup
Let’s run a full backup! This will send a full copy of data from the source server to the destination.
duplicity full -v --no-encryption --include="/sourcedir/" --exclude="**" / ssh://user@backupserver:22/destdir/
You can add additional folders by including multiple --include="[dir]" statements.
The --no-encryption specifies that the data is not to be encrypted at the destination. Data will be encrypted during transport as it’s passing through the SSH tunnel.
The --exclude="**" / option is a trick to backup everything that is only in the include list (and nothing else).
Running incremental backup
Running an incremental backup is very simple – just change the full flag to the incremental flag.
duplicity incremental -v --no-encryption --include="/sourcedir/" --exclude="**" / ssh://user@backupserver:22/destdir/
Automation
Write automated scripts
Having to run these commands every time you need a backup is a drag – what if we had scripts to handle it for us?
Full backup script
Run the command.
nano /usr/local/bin/backup-full
Add the following content.
#!/bin/bash
duplicity full -v --no-encryption --include="/sourcedir/" --exclude="**" / ssh://user@backupserver:22/destdir/
Incremental backup script
nano /usr/local/bin/backup-incremental
Add the following content.
#!/bin/bash
duplicity incremental -v --no-encryption --include="/sourcedir/" --exclude="**" / ssh://user@backupserver:22/destdir/
Make scripts executable
To make the scripts executable, run the following command.
chmod +x /usr/local/bin/backup-*
Now you can perform a backup by running backup-full and backup-incremental from within the shell. Pretty cool!
Setup cron
Let’s make the backups automatic! By setting up cron to run the above scripts at specified times, we can make sure that backups are performed at regular intervals.
Run the following command.
crontab -e
Add the following to the bottom of the file.
10 01 * * 1,2,3,4,5,6 backup-incremental
10 01 * * 7 backup-full
This will run a full backup every Sunday at 1:10 AM, and will run incremental backups every other day at 1:10 AM as well.
Restoration
Godzilla has destroyed Seattle and we need to be able to get the data back from the backup VPS in New York!
duplicity --no-encryption --file-to-restore / ssh://user@backupserver:22/destdir/
If we need to restore data from 3 days ago:
duplicity --no-encryption -t 3D --file-to-restore / ssh://user@backupserver:22/destdir/
The -t 3D option means restore a backup from three days ago. Similar options like -t 1M (for one month ago) or -t 5H (for 5 hours ago) also work.