Using Rclone with Azure for a Low-cost Backup

July 27, 2018
by Blaize Stewart

While backups are often one of the most overlooked planks in a comprehensive data security plan, they are are probably among the most important things one can do for data security. It’s works as an insurance policy against data loss which can be caused by a myriad of things ranging from accidental deletion, to drive failure, to ransomeware attacks.

A good backup strategy usually will not co-locate the backup data with the original for a number of reasons. A few might be things like fires and theft. In the past, data was backed up to media removable and stored offsite in a place such as a safe deposit or the like. Nowadays with the high-speed internet and readily available cloud-based storage, backing up over the internet to the cloud is a possibility.

One such cloud storage is Azure Blob Storage. Originally, Azure only had 1 tier for blob storage that was general purpose. Recently though, Azure introduced storage tiers for Azure Storage accounts, it opened up blob storage to a whole new set of use cases. The 3 storage tiers are hot, cool, and archive. Hot storage is intended for applications that need data to be readily available and that will be read and written to fairly often. Archive storage is intended for long-term archival of data. Data is not stored in a readily available state, so to recover the data requires that it go through a “hydration” process that can take a lot of time. Cool storage sits between hot and archive offering a lower-cost option that is available for use, but not intended for access. Cool storage in most regions is .01 per GB per month. This means that one terrabyte is roughly $10 a month. Azure does not charge for writing to cool storage, but it does charge for reading from cool storage. Given that the intent of this is a backup, you need only read from it in the event of data loss.

Azure Storage is only half the equation. To get data onto Azure Storage, you need a utility/agent that will move data from your local computer to the storage account and this is where RClone comes in. RClone is a command line utility that performs one-way syncs between your local data and the cloud. When it runs, it looks for changes on the local file system, then uploads those changes to the storage account. Anything unchanged is left alone. The initial upload will obviously take some time, but once it’s finished only changes are sent up.

To be clear, Azure does have a backup as a service offering, which can be used for more robust backups and schemes. However if you’re looking for a simple solution, this little “hack” might just be for you.

Setting up a Storage Account

Setting up a storage account in the Azure Portal is easy.

Select Create a resource ► Storage ► Storage account – blob, file, table, and queue. This will open the blade to configure the storage account.

Use the following settings in the blade.

Name: Give the account a unique DNS name.
Deployment Model: Resource manager
Account kind: Blob storage
Location: Select a region near you. Check to see which region is most cost effective here.
Replication: Locally-redundant storage (LRS)
Performance: Standard
Access tier (default): Cool
Secure transfer required: Enabled
Subscription: Select from the list if you have more than one.
Resource Group: Create new, then type in a name for the new group.
Virtual networks: Unselectable as it’s greyed out.
Data Lake Storage Gen2: Unselectable as it’s greyed out.

Once you’ve filled out the form, click Create. It usually takes less than a minute to provision.

Configuring RClone

First, download Rclone from Rclone’s site and Unzip Rclone and store Rclone somewhere on your drive that’s easy to find.
Launch a Terminal, CMD, or Powershell session and CD to the RClone folder.
Launch the interactive configuration.
```
 rclone config
```

At the prompt, select n for New remote and press Enter.

 No remotes found - make a new one
 n) New remote
 s) Set configuration password
 q) Quit config
 n/s/q> n

For the name, enter azure and press Enter.
```
 name> azure
```

For Storage enter 15 for Microsoft Azure Blob Storage and press Enter.

  1 / Alias for a existing remote
     "alias"
  2 / Amazon Drive
     "amazon cloud drive"
 ...
 15 / Microsoft Azure Blob Storage
     "azureblob"
 ...
 23 / Yandex Disk
     "yandex"
 24 / http Connection
     "http"
 Storage> 15

For Storage Account Name, type in the same name you gave in the first setting (the unique DNS Name) when you configured the storage account above.
```
 Storage Account Name
 account> blaizebackup
```
Now, you need to retrieve a key. To do this, go back to the Azure Portal and select Resource groups ► The name of the resource group you created above ► then the storage account you created.
Select Resource Group
Select Access keys, then click the Copy button to key 1.
Copy Key
Paste the key into the config CLI for Storage Account Key and press Enter.
```
 Storage Account Key
 key> tj/+mJVQ...==
```
For Endpoint for the service - leave blank normally, just press Enter unless you are using Azure Gov Cloud or something other than the standard Azure offering.

Review the settings, type in y, then press Enter.

 Remote config
 --------------------
 [azure]
 type = azureblob
 account = blaizebackup
 key = tj/+...==
 endpoint =
 --------------------
 y) Yes this is OK
 e) Edit this remote
 d) Delete this remote
 y/e/d> y

You’ll go back to the config menu where you’ll see your remote is now configured. Type in q to quit.

 Current remotes:

 Name                 Type
 ====                 ====
 azure                azureblob

 e) Edit existing remote
 n) New remote
 d) Delete remote
 r) Rename remote
 c) Copy remote
 s) Set configuration password
 q) Quit config
 e/n/d/r/c/s/q> q

Use Rclone to create a container on Azure. The syntax is rclone mkdir remote:container, where remote is the name of the remote you created with rclone config and containername is the name of the blob container you’ll create on Azure.
```
 rclone mkdir azure:backup
```

Now, Rclone is configured to talk to Azure and use it for backups.

Syncing a Directory

Rclone will sync a local directory with the remote container, storing all the files in the local directory in the container. Rclone uses the syntax, rclone sync source destination, where source is the local folder and destination is the container on Azure you just created.

rclone sync /path/to/my/backup/directory azure:backup

Scheduling a Job

Scheduling a backup is important to automating backups. Depending on your platform will depend on how you do this. Windows can use Task Scheduler while Mac OS and Linux can use crontabs.

Before scheduling a job, make sure you have done your initial upload and it has completed.

Windows

Create a text file called backup.bat somewhere on your computer and paste in the command you used in the section on Syncing a Directory. It will look something like the following. Specify the full path to the rclone.exe and don’t forget to save the file.
```
 C:fullpathtorclone.exe sync "C:pathtomybackupdirectory" azure:backup
```
Use schtasks to schedule a job. This utility takes a number of parameters.
- /RU – the user to run the job as. This is needed if the the user you want to use is logged out.
- /RP – the password for the user.
- /SC – set to DAILY
- /TN – the name of the job. Call it Backup
- /TR – the path to the backup.bat file you just created.
- /ST – the time to start the task. This is in the 24 hour time format. 01:05:00 is 1:05 AM. 13:05:00 would be 1:05 PM.
```
 schtasks /Create /RU username /RP "password" /SC DAILY /TN Backup /TR C:pathtobackup.bat /ST 01:05:00
```

If you want to back up multiple directories, simply add multiple containers using rclone mkdir and add a new line for each directory in the batch file for the source and corresponding destination container.

Mac and Linux

Create a text file called backup.sh somewhere on your computer, and paste the command you used in the section on Syncing a Directory. It will look something like the following. Specify the full path to the rclone executable and don’t forget to save the file.
```
 #!/bin/sh
 /full/path/to/rclone sync /path/to/my/backup/directory azure:backup
```
Make the script executable with chmod.
```
 chmod +x backup.sh
```
Edit crontabs.
```
 sudo crontab -e
```
Add an entry to the bottom of the crontabs file. Crontabs are straight forward: the first 5 fields represent in order minutes, hours, days, months, and weekdays. Using * will denote all. To make the backup.sh run at Daily at 1:05 AM, use something that looks like this:
```
 5 1 * * * /full/path/to/backup.sh
```
Save the crontabs and you’re ready to go.

If you want to back up multiple directories, simply add multiple containers using rclone mkdir and add a new line for each directory in the script for the source and corresponding destination container.

Conclusion

This simple utility offers a nice way to backup local data to Azure and will work for a lot of simple and even some more complex use cases. Here are a few Do’s and Dont’s

Dos

Backup documents, pictures, videos, content, and other sorts of files you can’t stand to lose.
Schedule a daily backup to make sure stuff does get backed up regularly.
Do check to make sure things are backing up occasionally.

Don’ts

Don’t backup programs and program directories.
Don’t use this for source control.
Don’t assume that you’ll never need a backup.

Happy Backing Up!

Need Help With An Azure Project?

Azure Consulting Azure Training

Stay Informed

We deliver solutions that accelerate the value of Azure.

Ready to experience the full power of Microsoft Azure?