Economical append-only offsite backups with restic and Wasabi on Debian 10
This guide shows how to install and configure the free backup software restic on Debian Buster to store off-site backups at the cloud storage company Wasabi. The main advantage of this guide is that it provides obligatory append-only backups. This means that once a backup has been made, it is secure in the sense that it cannot be overwritten or erased from the system the backup was made from. This means that the backup archives are protected from both natural disasters at the server of origin (for example fire) as well as man-made accidents (such a mistake at the root shell prompt). The backups are also protected in the sense that they are stored encrypted, which means that only the person or system that has the password for the backup can read it.
Please note that this guide is provided as-is without any warranty whatsoever. The backup software restic has not reached version 1.0 yet, and you should therefore never use it as your only backup solution. To be protected for eventual mishaps, please make sure to take a backup of the system you intend to use for the purpose of this guide before you begin. Also make sure to test that the backup you have taken really works through a test restore.
The guide is based and tested on Debian Buster (Debian 10). Older Debian versions are out of scope of this guide, but if you want to experiment on your own it should be possible to add the Debian Buster apt package repository to install recent versions of restic, tinyproxy and rclone also on Debian Jessie or Stretch.
This guide is structured in five parts:
In the first part, we will prepare an account at the Wasabi cloud storage where we later will store the backups. Wasabi provides a free 30 day trial for testing purposes.
Secondly, we will install and configure rclone, which on one hand provides access to the storage at Wasabi, and on the other hand offers an access interface for restic towards the storage.
Thirdly, tinyproxy - a simple proxy server - will be prepared, as it is needed as an intermediate for making the storage to work in an append-only manner together with restic.
Fourthly, we will set up the backup program restic.
As the very last step we will test both making a backup and a partial restore.
1 Prepare an account at Wasabi cloud storage
1.1 Start a Wasabi account
Wasabi has a 30-day free trial which can be found here:
https://wasabi.com/sign-up/
After filling out the form at the web page above you will receive your access data for the web interface of Wasabi. Use your access data to log in at this page:
https://console.wasabisys.com/
1.2 Create one bucket for data and another for locking
We will now create two so-called "Buckets". A bucket is a storage container in which you store data.
Create the first bucket through clicking on the red icon named Create Bucket which you find in the top right of the web page. The first bucket will be where all the backup data will be stored, which we will refer to later as the DATABUCKET. You know need to choose a bucket name. For example, you can use the following format as a template for the bucket name:
yourdomain.com-restic-data
Please replace yourdomain.com in the template name above with the domain name of the computer you want to backup. After setting the DATABUCKET name, please write it down. You will need it later to replace the placeholder text DATABUCKET in the templates that will follow later in this guide.
Then choose the region you want the bucket to be in. For optimum performance, it is good to choose the region/continent of the server you want to backup. So in case your server is in Europe you could choose:
eu-central-1
Then press Next.
On the next page two choices are displayed: for Bucket Versioning and Bucket Logging respectively. You can let these choices stay at their default values, i.e. Suspend Versioning and Suspend Logging. Then click Next, and click Create Bucket.
Now, repeat the same procedure and create another bucket. This second bucket will only be used to store the locking information of restic. We will later refer to it as the LOCKBUCKET. Click on the Create Bucket button again, and now choose a name for this second bucket, for example using this format:
yourdomain.com-restic-locks
Please replace yourdomain.com in the template above with the domain name of the computer to be backed up. After setting the LOCKBUCKET name, please write it down. You will need it later to replace LOCKBUCKET in the templates that will follow later in this guide.
(Restic uses locks to make sure that only one client at a time does backups or restores. The reason we need a separate bucket for the locking information is that restic needs to be able to both add and delete locks, and therefore we need a bucket that permits deletions.)
1.3 Create a user
Click on the IAM icon in the topmost icon row. You have now entered Identity & Access management. Click on Users in the left column. Then click on the red icon named + Create user and choose a logical user name, such as the domain name of the server you want to backup. For example, the user name could be system1.example.com. After setting the user name, please write it down so that you can replace USERNAME in the templates that will follow later in this guide with your value.
Under the Access section in the dialogue, check the checkbox named Programmatic (create API key). Then click Next. Create a group for the user. If all your servers were situated under one main domain, this main domain could be a suitable group name. Now click Next. On the following page (policy creation) just click Next. Finally, click on the Create user button.
To the right of the text Secret Key towards the bottom of the dialogue window you find a small link named Show, that you should click on. Save both the ACCESS KEY and the SECRET KEY in a text file. You will need this information later when we configure the access to Wasabi from the client computer you will backup.
1.4 Create a policy
In order for the user we have created to be allowed access to the two buckets, we need to create an access policy.
Click on Policies in the left column of the Wasabi web interface. Then click on the red button Create Policy. You can choose the domain name of the computer you want to backup for the policy name, for example system1.example.com . Write the policy name in the top left field.
Below is a template you will use as a base for the actual policy. Copy the template to a text editor, and then replace the two occurrences of DATABUCKET in the text file with the name you gave to the first bucket, and the two occurrences of LOCKBUCKET with the name you chose for the second bucket.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "arn:aws:s3:::*"
},
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
"arn:aws:s3:::DATABUCKET",
"arn:aws:s3:::DATABUCKET/*",
"arn:aws:s3:::LOCKBUCKET",
"arn:aws:s3:::LOCKBUCKET/*"
]
}
]
}
Copy the policy you have created from your text editor and paste it in the big, main field of the Wasabi web page dialogue window. Then click Save.
1.5 Apply the policy
The policy now needs to be applied to the user. Click on Users to the left and then click on the user you have created. Now click on the Permissions tab (to the right in the grey horizontal menu). Then start typing the first letters of the policy name you chose when you created the policy in the text field under Start typing to find policies for the user. Click on the name of your policy. Now you have applied the policy to the user.
1.6 Set the data bucket to be immutable (append-only)
For security reasons we want it to be impossible to overwrite already made backups from the client (the computer the backups are made from). In order to accomplish this we use the Wasabi immutable feature. In the web interface of Wasabi click on Storage (in the top bar). Then click on the DATABUCKET (probably the one that ends with restic-data). Click on the white cogwheel for Settings (towards the upper right). Then click on Compliance in the white upper bar. Click on the slider to the right of Compliance Mode to activate this mode. If you want to you can activate Delete After Retention but you have to make sure to put a large value in Retention Time, this is the minimum time every file stays after the client tries to delete it. For example, put 100 years here. (It seems like you cannot leave this field empty for an infinite retention time.) This means that data is safe for 100 years during which period it cannot be deleted nor modified. Press Save to save your settings.
1.7 Create the lock folder
In the web interface of Wasabi click on Storage (in the top bar). Then click on the LOCKBUCKET (the one that ends with restic- locks).
Now press the green button named Create folder. Name the folder locks so that it can be used for the restic lock files later on.
We are now finished with the configuration in the Wasabi web interface. In the next section, we will start to work at the terminal of the computer we want to backup.
2 Install and configure rclone
2.1 First, make a backup via your existing system
Now it is time to access the computer the backups shall be taken from.
Please remember to take a backup of this computer with an existing system! Also please test that the backup works through a test restore before you continue. In case something would not work as expected, it is critical to be able to revert to a functioning stage.
2.2 Install rclone
All lines shown below should be typed into the terminal followed by Enter.
Connect to the terminal of the computer you want to make backups from, for example via SSH. Then issue this command to become root:
su -
followed by Enter and the root password. Root access is needed, as we will install new programs from the Debian repository.
Now install rclone:
apt install rclone
2.3 Configure rclone to connect to Wasabi
Then to configure rclone, type this command:
rclone config
Then type:
n
Now it is time to choose a name for the config. Please note that. (dot) is not allowed in the name, but on the other hand - (dash) is allowed. In order to name your rclone configuration use this template, but first replace yourdomain-com with your own domain name, first replacing. (dots) in the domain name with - (hyphens):
wasabi-yourdomain-com-restic
Save the name you give the rclone config, as you later will replace the placeholder RCLONECONFIG with the name.
Then print 4 for Amazon S3 Compliant Storage Providers and Enter
Then print 7 for Wasabi Object Storage and Enter
Then print 1 for Enter AWS credentials in the next step and Enter
Copy the ACCESS KEY that you saved earlier in a file and paste it, then press Enter
Then copy and paste the SECRET KEY you also saved in the same file earlier, followed by Enter
Print the region you choose for your buckets. You can find the region if you click on Storage in the top row of the Wasabi web interface. For example type this region for Europe and then Enter:
eu-central-1
For Endpoint for S3 API, type this if the bucket is in the EU region followed by Enter:
s3.eu-central-1.wasabisys.com
If your bucket is in the US, please find the name of your endpoint in this document:
https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-regions-
Leave Location constraint empty and press Enter
Press 1 for Owner gets FULL_CONTROL and then Enter
Press y for Edit advanced config and then Enter
Press Enter for default value for chunk size
Press Enter for default value for disable checksum
Press Enter for default value for session token
Type 16 for Upload concurrency and then Enter
Press Enter for default value for force_path_style
Press Enter for default value for v2_auth followed by Enter
Press y for Yes this is OK and Enter
Press q to quit config.
Now the configuration for rclone is finished.
2.4 Test that rclone works with Wasabi
Now create a test file through this command :
echo test >> /tmp/test.txt
Transfer the test file through the command below. However, you first need to change RCLONECONFIG to your value, and also DATABUCKET to your value for this parameter. Everything is on one line:
rclone -v sync /tmp/test.txt RCLONECONFIG:DATABUCKET/
If everything works you should now see the test.txt file in the Wasabi web interface for the DATABUCKET. Please note that sometimes it can take up to approx. 1 minute until the file is shown. Also test a transfer with the other bucket, the one ending with restic-locks:
rclone -v sync /tmp/test.txt RCLONECONFIG:LOCKBUCKET/
Now check the Wasabi web interface again, for the text file that should be in the LOCKBUCKET.
2.5 Setup rclone to serve restic via systemd
Rclone has a built-in feature of using a cloud storage space (Wasabi in our case) and serve it to restic in a compatible way. This way of running rclone is done via the command rclone serve restic which works as a server process. We will need two of these server processes: one main for almost everything, and a second one which only deals with the restic lock files.
First, create a restic user:
adduser restic
Copy the rclone configuration from the root user to the restic user:
su restic
mkdir -p /home/restic/.config/rclone
exit
cp -a /root/.config/rclone/rclone.conf /home/restic/.config/rclone/
chown restic.restic /home/restic/.config/rclone/rclone.conf
Create a systemd control file, for example through the nano editor:
nano /etc/systemd/system/restic-data.service
The file should contain the data below. Please note that as before you need to change the RCLONECONFIG value as well as the DATABUCKET value to your settings.
[Unit]
Description=Rclone serve restic data on wasabi
After=network.target
[Service]
Type=simple
User=restic
Group=restic
ExecStart=/usr/bin/rclone serve restic RCLONECONFIG:DATABUCKET --addr=127.0.0.1:8001 --append-only --retries 10 --transfers 20 --s3-upload-concurrency 8 --s3-chunk-size 16M
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
WantedBy=multi-user.target
Save the file. The purpose of it is for rclone to provide a restic interface towards the Wasabi cloud storage bucket. The systemd service we have just created will start automatically on the next boot, but first we need to activate it through these commands:
systemctl daemon-reload
systemctl enable restic-data
systemctl start restic-data
Then we check that the service have started:
systemctl status restic-data
You should now see Active: active (running) if everything works correctly. Press q to quit the status display.
Now we have installed the rclone serve restic service on port 8001 which connects to a bucket that is append-only and will store the backups.
Next, we need another service on port 8002 for the restic locking system, connected to the second bucket which is read+write. Therefore we will repeat the steps we just have done once more with minor modifications.
Create a systemd control file :
nano /etc/systemd/system/restic-locks.service
The file should contain the data below. The line that begins with ExecStart= is a single, long line split into three lines below. Please change the RCLONECONFIG value as well as the LOCKBUCKET value to your settings.
[Unit]
Description=Rclone serve restic locks on wasabi
After=network.target
[Service]
Type=simple
User=restic
Group=restic
ExecStart=/usr/bin/rclone serve restic RCLONECONFIG:LOCKBUCKET --addr=127.0.0.1:8002
Restart=always
RestartSec=5
StartLimitInterval=0
[Install]
