Consistently backup your virtual machines using libvirt and zfs – part 1

How to backup virtual machines is a pretty interesting topic and a lot could be said. COW file systems like zfs or btrfs actually do most of the job for you, thanks to their snapshotting capabilities. Unfortunately that’s not enough to get consistent backups, because taking a snapshot of a running VM is very similar to unplugging the power cord. In most cases this isn’t as bad as it sounds, but it is extremely bad if you’re running databases or so. That means you will get corrupt data, which is something we want to avoid at all costs. But how to avoid that? Shutting down the machines before taking the snapshots could be a solution, but that’s only viable if you do daily backups at most. What if we want hourly snapshots? That’s simply unfeasible. The next thing we could do is to pause the VM, take a snapshot of the disk, dump the ram and the EFI vars and then resume the guest. That would be way better, but it still involves some kind of downtime. Is it possible to get it any better? If you use qcow2 you could use its internal snapshotting features to do live snapshots of the state of the machine, but that unfortunately doesn’t work anymore if you use UEFI and it’s also not so well maintained. Also you probably want to use ZVOLs, so no way.
The best alternative out there are libvirt external snapshots. They allow you to freeze the VM image (be it a raw file, qcow2 or zvol), take a dump of the ram and then keep writing all subsequent writes to an external qcow2 file. Actually we don’t really need the external qcow2 file at all, because we can use zfs to track the diff instead. It means that as soon as we created the libvirt snapshot we can immediately take a zfs snapshot and then merge back the external file into the original image.
I use sanoid to take the zfs snapshots and I wanted to keep using it. Unfortunately it didn’t support pre/post scripts, but luckily there were some patches floating around. They didn’t expose all the things I needed in order to get it working, so I made my own fork where I forward ported the patch to latest git master, plus adding additional features to get all the data I needed: https://github.com/darkbasic/sanoid
If you’re using Arch Linux here is a PKGBUILD which tracks my branch, with the addition of systemd timers which the AUR package didn’t have: sanoid-git.tar

Let’s see how it’s implemented:

zfs list
rpool/VM 36.0G 357G 24K none
rpool/VM/fedora28 9.04G 41.0G 8.21G /var/lib/libvirt/images/fedora28
rpool/VM/win2k16 26.9G 73.1G 17.3G /var/lib/libvirt/images/win2k16
rpool/VM_snapshots 34K 357G 34K /var/lib/libvirt/snapshots

As you can see I have a dataset called VM which contains an additional dataset for each VM. There I also store the nvram with the EFI VARS, because it’s important to backup them as well. Additionally I have another dataset called VM_snapshots which I use to store the external qcow2 diff. Its only purpose is to avoid that it gets snapshotted along with the rest of the machine: we don’t need it and it will cease to exist a few seconds later.

Here is my sanoid config:

[rpool/VM]
use_template = production,scripts
recursive = yes
# if you want sanoid to manage the child datasets but leave this one alone, set process_children_only.
process_children_only = yes

[template_production]
hourly = 36
daily = 30
monthly = 3
yearly = 0
autosnap = yes
autoprune = yes

[template_scripts]
### run script before snapshot
### dataset name will be supplied as an environment variable $SANOID_TARGET
pre_snapshot_script = /opt/scripts/prescript.sh
### run script after snapshot
### dataset name will be supplied as an environment variable $SANOID_TARGET
post_snapshot_script = /opt/scripts/postscript.sh
### don't take an inconsistent snapshot
#no_inconsistent_snapshot = yes
### run post_snapshot_script when pre_snapshot_script is failing
#force_post_snapshot_script = yes

This is the content of my prescript:

#!/bin/bash
DOMAIN=${SANOID_TARGET##*/}
SNAPSHOT_NAME=${SANOID_SNAPNAME}
RAM_BACKUP=/mem

# Backup xml
cp /etc/libvirt/qemu/${DOMAIN}.xml /var/lib/libvirt/images/${DOMAIN}/

# Find out if running or not
STATE=`virsh dominfo $DOMAIN | grep "State" | cut -d " " -f 11`

if [ "$STATE" = "running" ]; then
# Take a libvirt snapshot
virsh snapshot-create-as ${DOMAIN} ${SNAPSHOT_NAME} \
--diskspec vda,snapshot=external,file=/var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.disk.qcow2 \
--memspec file=/var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.mem.qcow2,snapshot=external \
--atomic
fi

exit 0

Again, you will need my fork of sanoid in order to get pre-post scripts supports and in particular the additional environment variables. Hopefully soon it won’t be necessary anymore.

What’s going on? First we check if the machine is running, because if it isn’t a regular zfs snapshot will be enough. If it’s running, on the other hand, we do an external libvirt snapshot and we dump the memory.
Now all subsequent writes will go through the external qcow2 and sanoid will take the zfs snapshot.

This is the content of my postscript:

#!/bin/bash
DOMAIN=${SANOID_TARGET##*/}
SNAPSHOT_NAME=${SANOID_SNAPNAME}
RAM_BACKUP=/mem

# Find out if running or not
STATE=`virsh dominfo $DOMAIN | grep "State" | cut -d " " -f 11`

if [ "$STATE" = "running" ]; then
# Commits content from top images into base and adjust the base image as the current active image (--pivot)
virsh blockcommit ${DOMAIN} vda --active --wait --pivot

# Delete snapshot
rm /var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.disk.qcow2

# Once the 'blockpull' operation above is complete, we can clean-up the tracking of snapshots by libvirt to reflect the new reality
virsh snapshot-delete ${DOMAIN} ${SNAPSHOT_NAME} --metadata

# Move the ram to a bigger and cheaper drive.
mkdir ${RAM_BACKUP}/${DOMAIN} 2> /dev/null
mv /var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.mem.qcow2 ${RAM_BACKUP}/${DOMAIN}/
fi

exit 0

As soon as the snapshot is taken we want to merge the external qcow2 file back to the original image using blockcommit. We don’t need it because zfs will take care of the diff. Now it’s time to backup our precious ram dump. We don’t want to waste our Optane 3D XPoint memory with it, so it will get stored on a slower and cheaper drive.

What’s next? We still need more sanoid hooks, in particular pre/post pruning scripts because we want to delete our ram dumps when the old snapshots get deleted. I will probably implement it sooner or later, but since I don’t know Perl patches are welcome.

We also want to send/receive our snapshots to an off site machine (zfs snapshots are not backups), but that’s for part 2!

If you want to further look into the topic I suggest you to read the following:
https://www.spinics.net/lists/virt-tools/msg11470.html
https://wiki.libvirt.org/page/I_created_an_external_snapshot,_but_libvirt_will_not_let_me_delete_or_revert_to_it
https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit
https://blog.programster.org/kvm-external-snapshots
https://www.redhat.com/archives/libvirt-users/2013-October/msg00018.html
https://kashyapc.fedorapeople.org/virt/lc-2012/lceu-2012-virt-snapshots-kashyap-chamarthy.pdf
https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-illustration.txt
https://kashyapc.fedorapeople.org/virt/lc-2012/live-backup-with-external-disk-snapshots-and-blockpull.txt
https://iclykofte.com/kvm-live-online-backups-external-and-internal/
https://wiki.libvirt.org/page/Live-merge-an-entire-disk-image-chain-including-current-active-disk

10 comments to Consistently backup your virtual machines using libvirt and zfs – part 1

  • Hurrah! At last I got a web site from where I know how to in fact get valuable information regarding
    my study and knowledge.

  • I appreciate, lead to I found just what I was taking a look for.

    You’ve ended my 4 day long hunt! God Bless you man. Have a nice day.
    Bye

  • It’s really a great and helpful piece of information. I’m glad that
    you simply shared this helpful info with us. Please stay us informed like this.
    Thank you for sharing.

  • Thanks , I have recently been searching for info about this
    topic for a while and yours is the best I’ve discovered so far.
    But, what concerning the bottom line? Are you positive concerning the
    supply?

  • Have you ever thought about publishing an e-book or guest authoring on other
    sites? I have a blog based on the same topics you discuss
    and would really like to have you share some stories/information. I know my viewers would enjoy your work.
    If you are even remotely interested, feel free to send
    me an e mail.

  • What’s up, I wish for to subscribe for this blog to take most up-to-date updates, thus where can i do it please assist.

  • Ute

    I’d like to thank you for the efforts you’ve put in writing this website.

    I am hoping to see the same high-grade blog posts from you in the future as well.
    In truth, your creative writing abilities has encouraged me to get my very own website now
    😉

  • Coco Gauff marches into Battle of South Palm Beach County against Naomi Osaka
    in Open third round

  • The state and Best Soccer Gambling Agent

    One of the best soccer gambling sites operating in the Hard anodized cookware region, including Indonesia, is Sbobet.
    Known as typically the site that won two awards since the
    best user, Sbobet has been serving its customers since 2004 and carries on to survive as the oldest and greatest internet gambling internet site in Asia.

    Using the license he obtained from typically the Isle of Men Wagering Supervision Commission, as well as the Government regarding the Republic of the Philippines for the Hard anodized cookware region, Sbobet
    proved that will their ambition to control
    the world gambling market was not merely a stone cold bluff.
    But sadly Sbobet struggles in touching the gambler in Indonesia for the reason that supplier Sbobet is an global site that uses British and is also forbidden by legal entities inside Indonesia.
    Therefore, Depobola is present as an official soccer gambling agent who assists you register with
    Sgd777 accounts.

    Many articles that will provide commentary on Bola tangkas
    emphasize that this brand name provides a very complete wagering gambling service, covering greater than 1500 sports events weekly.
    With successful sponsorship with some of the top sports teams
    in the English Premier League, such as West Ham,
    Swansea Metropolis and Southampton, it offers confirmed that football
    is their particular true identity. And with very lucrative promotions, it’s not with out
    reason the rating supplier continues to skyrocket in the eyes of observers and loyal customers.

    Soccer Gambling Video games Available on Sbobet
    Even though this sort of sports betting sport is the primary attraction presented, it
    does not imply that gambling sites that usually are identical in blue perform
    not offer other types of games. There are still a great many other
    games that members can choose to enjoy with Depobola official gambling agents such as casino games, racing, lottery, Online Slots and
    so about.

    As the key product, regarding course, this provider offers provided the best soccer gambling games
    that are very complete and varied. Presently there are greater than 20 sorts of sports which can be chosen as betting objects and there are many varieties of games that members can choose from the most popular kinds such as Asian Handicap, Correct Score, 1X2, U / U, O or E, to Mix Parlay.

    Asian Handicap
    You could say this is the most well-liked soccer gambling game
    performed. Even though the game is quite tricky, in reality the
    Asian handicap has never been abandoned by its fans. Some recognition from bettor related
    to be able to this game is since the game is pretty challenging and difficult to guess due to
    the fact it involves voor or perhaps handicap features.

    Correct Report
    As the name indicates, in this game gamers are asked to guess the
    score that happens in the match specifically. The game is split up into two types: the very
    first is half time or perhaps HT where the player will guess the
    score of the match until the halftime break; in addition to the second is FEET or full time exactly where players will guess the
    particular score at the ending in the match.
    1X2 (Double Chance)
    The name associated with the game is used from the betting alternatives offered
    as part of the game: just one, if the player wager for a home team;
    2, if a participant bets for an away team; and X
    in case a player bets regarding a draw. In a new double chance the gamer
    can double the opportunity of victory by choosing 2 of the 3
    available choices such as: 1X, 2X, or 12.

    Mix Parlay
    Mix Parlay is actually a gambling game exactly where players will play various kinds market games into
    1 bet. The winnings of this game will be very
    large since the funds that players place on bets will be multiplied by the likelihood of several of the games that will players play.

    Features That Become Advantages of Sgd777
    If you think the huge benefits are only the total and wide variants regarding the
    game, it indicates an individual don’t really know your pet because there are several other advantages
    that you need to know and also make this specific site never abandoned by simply
    its members.

    Very Aggressive Odds
    Odds offered are extremely competitive when compared to other similar sites.
    Not really only that, this Supplier also indiscriminately serves their customers.
    If other websites will reject players that have won a lot,
    nevertheless not with Sbobet because they will welcome every
    consumer who chose them as a partner to play nicely
    and friendly.
    High Gamble Reduce
    Although limited, yet online betting limits are incredibly high.

    Because of the particular height, you will feel like Sbobet’s guess seems
    limitless. Therefore, you can play just as much as you
    want with us as much since you would like to place a bet.

Leave a Reply to Becks-berry Cancel reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>