Since I switched from VMware to Debian, I have struggled to find a proper solution for my KVM backups.

I tried hard!

A top-rated solution doesn’t like recent Linux kernels. Another one doesn’t support Debian 11. Another one put me into permission hell, and another didn’t even work as the first install requires a server with a GUI.
Why is it so difficult?

So, I left my comfort zone of dealing with a web dashboard and looked for command line options.

Good news: I found one, and besides the first steps at the beginning, it works very well!

Enter: virtnbdbackup

What a name, can you pronounce it?

You can find the documentation here.
I’m on Debian, so be careful with my steps below as they might differ slightly from other distros.

Allow incremental backups

The tool requires a few changes on all VMs.

Yes, that includes the templates, so you don’t have to repeat them in the future.

Issue this command; in my example, “mssql” is the name of the VM:

virsh edit mssql

Delete the very first line and replace it with this one:

<domain type='kvm' id='1' xmlns:qemu=‘http://libvirt.org/schemas/domain/qemu/1.0'>

Now scroll to the bottom and right before </domain> paste these lines:

 <qemu:capabilities>
   <qemu:add capability='incremental-backup'/>
 </qemu:capabilities>

It should look like this:

The change requires a restart of the VM, but as we need exclusive access to the disk image for our next step, I will shut it down.

Converting the image files

In one of my first attempts, I received an error message about “dirty bitmaps”.

The documentation let me down, as it only said, “consider migrating your qcow files to version 3 format”. What’s that about, now?

It took some google-fu to understand that there are different versions of the qcow image file format.

The latest one, and even that has existed for many years, is version 3.
 To make it worse, the files are still called qcow2.

There’s no qcow3.

Instead, they use an internal version number, and 0.10 is v2 while 1.1 is v3.

Got it? Well, it doesn’t matter, as here are the steps.

qemu-img convert -f qcow2 -O qcow2 -o compat=1.1 /path/to/your-image /path/to/your-new-image

Using my mssql example, it looks like this:

qemu-img convert -f qcow2 -O qcow2 -o compat=1.1 /mnt/vm/mssql /mnt/vm/mssqlx

Depending on the image size, this might take a few minutes, but eventually, you end up with two files on the datastore.

I suggest renaming mssql to mssqlold and the mssqlx to mssql.

Now start the VM and see if everything works.

Fine? Great, delete mssqlold.

Rinse, and repeat with all your VMs.

The first KVM backup

Now with all the prep work out of the way, everything else is easy.
For the first time, use this command to initiate the backup:

virtnbdbackup -d VMNAME -l full -o /backup/location/VMNAME

Again, here’s my example:

virtnbdbackup -d mssql -l full -o /mnt/vm-backup/mssql

The path /mnt/vm-backup leads to one of my Qnap devices.
It will take a few minutes and look like this:

Once the first full backup is finished, we continue to use incremental backups for the future.
It’s a similar command:

virtnbdbackup -d mssql -l inc -o /mnt/vm-backup/mssql

To automate it, I suggest visiting the crontab-generator.
Here are my results:

Crontab -e
0 2 * * * virtnbdbackup -d mssql -l inc -o /mnt/vm-backup/mssql >/dev/null 2>&1
5 2 * * * virtnbdbackup -d hco -l inc -o /mnt/vm-backup/hco >/dev/null 2>&1
10 2 * * * virtnbdbackup -d docker01 -l inc -o /mnt/vm-backup/docker01 >/dev/null 2>&1
15 2 * * * virtnbdbackup -d plex -l inc -o /mnt/vm-backup/plex >/dev/null 2>&1
20 2 * * * virtnbdbackup -d parrot -l inc -o /mnt/vm-backup/parrot >/dev/null 2>&1 

Basically, I start at 02:00 in the night and give each backup 5 minutes; that’s plenty of time for an incremental update. The documentation explains that multiple jobs simultaneously are supported, but I’m in no rush.

Restoring a VM

A backup isn’t a backup until it’s tested.
I deleted one of my VMs via the Cockpit dashboard for my test. You can confirm the deletion with this command:

virsh list

I’m honest with you; I didn’t use my mssql box for the test; I used an unimportant one. But I made sure I created a few incrementals after applying changes before killing it.

This command will provide information about the required disk space etc.:

virtnbdrestore -i /tmp/backupset/ -o dump

The command for a full restore is this:

virtnbdrestore -i /tmp/backupset/ -o /tmp/restore

In my case:

virtnbdrestore -i /mnt/vm-backup/parrot/ -o dump
virtnbdrestore -i /mnt/vm-backup/parrot/ -o /mnt/vm03/parrot

I received an error message:

Restore path [/mnt/vm03/parrot] seems not to be an libvirt managed pool, skipping refresh.

What’s that? I’m not sure!
But according to the almighty internet, it helps to send a “virsh pool-refresh vm03,” but maybe I could have ignored it.

I will check it next time!

Inside the folder, I find both the XML and the image. I’m jumping into Cockpit, importing a VM, pointing to the image, and starting it.

It’s working fine, including the last changes I applied!

Okay, the restore process could be a little more convenient than placing the XML file in its location, but it’ll do for me.

That’s it. The preparations took a while, but it didn’t require an additional VM for a management system, and it took me less than two hours from discover virtnbdbackup until I could backup and restore a machine.

I’m wondering why all those vendors struggle.

Anyway, not my problem. I’m sorted.

More homelab posts: