Consistently backup your virtual machines using libvirt and zfs – part 1

How to backup virtual machines is a pretty interesting topic and a lot could be said. COW file systems like zfs or btrfs actually do most of the job for you, thanks to their snapshotting capabilities. Unfortunately that’s not enough to get consistent backups, because taking a snapshot of a running VM is very similar to unplugging the power cord. In most cases this isn’t as bad as it sounds, but it is extremely bad if you’re running databases or so. That means you will get corrupt data, which is something we want to avoid at all costs. But how to avoid that? Shutting down the machines before taking the snapshots could be a solution, but that’s only viable if you do daily backups at most. What if we want hourly snapshots? That’s simply unfeasible. The next thing we could do is to pause the VM, take a snapshot of the disk, dump the ram and the EFI vars and then resume the guest. That would be way better, but it still involves some kind of downtime. Is it possible to get it any better? If you use qcow2 you could use its internal snapshotting features to do live snapshots of the state of the machine, but that unfortunately doesn’t work anymore if you use UEFI and it’s also not so well maintained. Also you probably want to use ZVOLs, so no way.
The best alternative out there are libvirt external snapshots. They allow you to freeze the VM image (be it a raw file, qcow2 or zvol), take a dump of the ram and then keep writing all subsequent writes to an external qcow2 file. Actually we don’t really need the external qcow2 file at all, because we can use zfs to track the diff instead. It means that as soon as we created the libvirt snapshot we can immediately take a zfs snapshot and then merge back the external file into the original image.
I use sanoid to take the zfs snapshots and I wanted to keep using it. Unfortunately it didn’t support pre/post scripts, but luckily there were some patches floating around. They didn’t expose all the things I needed in order to get it working, so I made my own fork where I forward ported the patch to latest git master, plus adding additional features to get all the data I needed: https://github.com/darkbasic/sanoid
If you’re using Arch Linux here is a PKGBUILD which tracks my branch, with the addition of systemd timers which the AUR package didn’t have: sanoid-git.tar

Let’s see how it’s implemented:

zfs list
rpool/VM 36.0G 357G 24K none
rpool/VM/fedora28 9.04G 41.0G 8.21G /var/lib/libvirt/images/fedora28
rpool/VM/win2k16 26.9G 73.1G 17.3G /var/lib/libvirt/images/win2k16
rpool/VM_snapshots 34K 357G 34K /var/lib/libvirt/snapshots

As you can see I have a dataset called VM which contains an additional dataset for each VM. There I also store the nvram with the EFI VARS, because it’s important to backup them as well. Additionally I have another dataset called VM_snapshots which I use to store the external qcow2 diff. Its only purpose is to avoid that it gets snapshotted along with the rest of the machine: we don’t need it and it will cease to exist a few seconds later.

Here is my sanoid config:

[rpool/VM]
use_template = production,scripts
recursive = yes
# if you want sanoid to manage the child datasets but leave this one alone, set process_children_only.
process_children_only = yes

[template_production]
hourly = 36
daily = 30
monthly = 3
yearly = 0
autosnap = yes
autoprune = yes

[template_scripts]
### run script before snapshot
### dataset name will be supplied as an environment variable $SANOID_TARGET
pre_snapshot_script = /opt/scripts/prescript.sh
### run script after snapshot
### dataset name will be supplied as an environment variable $SANOID_TARGET
post_snapshot_script = /opt/scripts/postscript.sh
### don't take an inconsistent snapshot
#no_inconsistent_snapshot = yes
### run post_snapshot_script when pre_snapshot_script is failing
#force_post_snapshot_script = yes

This is the content of my prescript:

#!/bin/bash
DOMAIN=${SANOID_TARGET##*/}
SNAPSHOT_NAME=${SANOID_SNAPNAME}
RAM_BACKUP=/mem

# Backup xml
cp /etc/libvirt/qemu/${DOMAIN}.xml /var/lib/libvirt/images/${DOMAIN}/

# Find out if running or not
STATE=`virsh dominfo $DOMAIN | grep "State" | cut -d " " -f 11`

if [ "$STATE" = "running" ]; then
# Take a libvirt snapshot
virsh snapshot-create-as ${DOMAIN} ${SNAPSHOT_NAME} \
--diskspec vda,snapshot=external,file=/var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.disk.qcow2 \
--memspec file=/var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.mem.qcow2,snapshot=external \
--atomic
fi

exit 0

Again, you will need my fork of sanoid in order to get pre-post scripts supports and in particular the additional environment variables. Hopefully soon it won’t be necessary anymore.

What’s going on? First we check if the machine is running, because if it isn’t a regular zfs snapshot will be enough. If it’s running, on the other hand, we do an external libvirt snapshot and we dump the memory.
Now all subsequent writes will go through the external qcow2 and sanoid will take the zfs snapshot.

This is the content of my postscript:

#!/bin/bash
DOMAIN=${SANOID_TARGET##*/}
SNAPSHOT_NAME=${SANOID_SNAPNAME}
RAM_BACKUP=/mem

# Find out if running or not
STATE=`virsh dominfo $DOMAIN | grep "State" | cut -d " " -f 11`

if [ "$STATE" = "running" ]; then
# Commits content from top images into base and adjust the base image as the current active image (--pivot)
virsh blockcommit ${DOMAIN} vda --active --wait --pivot

# Delete snapshot
rm /var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.disk.qcow2

# Once the 'blockpull' operation above is complete, we can clean-up the tracking of snapshots by libvirt to reflect the new reality
virsh snapshot-delete ${DOMAIN} ${SNAPSHOT_NAME} --metadata

# Move the ram to a bigger and cheaper drive.
mkdir ${RAM_BACKUP}/${DOMAIN} 2> /dev/null
mv /var/lib/libvirt/snapshots/${DOMAIN}.${SNAPSHOT_NAME}.mem.qcow2 ${RAM_BACKUP}/${DOMAIN}/
fi

exit 0

As soon as the snapshot is taken we want to merge the external qcow2 file back to the original image using blockcommit. We don’t need it because zfs will take care of the diff. Now it’s time to backup our precious ram dump. We don’t want to waste our Optane 3D XPoint memory with it, so it will get stored on a slower and cheaper drive.

What’s next? We still need more sanoid hooks, in particular pre/post pruning scripts because we want to delete our ram dumps when the old snapshots get deleted. I will probably implement it sooner or later, but since I don’t know Perl patches are welcome.

We also want to send/receive our snapshots to an off site machine (zfs snapshots are not backups), but that’s for part 2!

If you want to further look into the topic I suggest you to read the following:
https://www.spinics.net/lists/virt-tools/msg11470.html
https://wiki.libvirt.org/page/I_created_an_external_snapshot,_but_libvirt_will_not_let_me_delete_or_revert_to_it
https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit
https://blog.programster.org/kvm-external-snapshots
https://www.redhat.com/archives/libvirt-users/2013-October/msg00018.html
https://kashyapc.fedorapeople.org/virt/lc-2012/lceu-2012-virt-snapshots-kashyap-chamarthy.pdf
https://kashyapc.fedorapeople.org/virt/lc-2012/snapshots-illustration.txt
https://kashyapc.fedorapeople.org/virt/lc-2012/live-backup-with-external-disk-snapshots-and-blockpull.txt
https://iclykofte.com/kvm-live-online-backups-external-and-internal/
https://wiki.libvirt.org/page/Live-merge-an-entire-disk-image-chain-including-current-active-disk

16 comments to Consistently backup your virtual machines using libvirt and zfs – part 1

  • Hurrah! At last I got a web site from where I know how to in fact get valuable information regarding
    my study and knowledge.

  • I appreciate, lead to I found just what I was taking a look for.

    You’ve ended my 4 day long hunt! God Bless you man. Have a nice day.
    Bye

  • It’s really a great and helpful piece of information. I’m glad that
    you simply shared this helpful info with us. Please stay us informed like this.
    Thank you for sharing.

  • Thanks , I have recently been searching for info about this
    topic for a while and yours is the best I’ve discovered so far.
    But, what concerning the bottom line? Are you positive concerning the
    supply?

  • Have you ever thought about publishing an e-book or guest authoring on other
    sites? I have a blog based on the same topics you discuss
    and would really like to have you share some stories/information. I know my viewers would enjoy your work.
    If you are even remotely interested, feel free to send
    me an e mail.

  • What’s up, I wish for to subscribe for this blog to take most up-to-date updates, thus where can i do it please assist.

  • Ute

    I’d like to thank you for the efforts you’ve put in writing this website.

    I am hoping to see the same high-grade blog posts from you in the future as well.
    In truth, your creative writing abilities has encouraged me to get my very own website now
    😉

  • Coco Gauff marches into Battle of South Palm Beach County against Naomi Osaka
    in Open third round

  • The state and Best Soccer Gambling Agent

    One of the best soccer gambling sites operating in the Hard anodized cookware region, including Indonesia, is Sbobet.
    Known as typically the site that won two awards since the
    best user, Sbobet has been serving its customers since 2004 and carries on to survive as the oldest and greatest internet gambling internet site in Asia.

    Using the license he obtained from typically the Isle of Men Wagering Supervision Commission, as well as the Government regarding the Republic of the Philippines for the Hard anodized cookware region, Sbobet
    proved that will their ambition to control
    the world gambling market was not merely a stone cold bluff.
    But sadly Sbobet struggles in touching the gambler in Indonesia for the reason that supplier Sbobet is an global site that uses British and is also forbidden by legal entities inside Indonesia.
    Therefore, Depobola is present as an official soccer gambling agent who assists you register with
    Sgd777 accounts.

    Many articles that will provide commentary on Bola tangkas
    emphasize that this brand name provides a very complete wagering gambling service, covering greater than 1500 sports events weekly.
    With successful sponsorship with some of the top sports teams
    in the English Premier League, such as West Ham,
    Swansea Metropolis and Southampton, it offers confirmed that football
    is their particular true identity. And with very lucrative promotions, it’s not with out
    reason the rating supplier continues to skyrocket in the eyes of observers and loyal customers.

    Soccer Gambling Video games Available on Sbobet
    Even though this sort of sports betting sport is the primary attraction presented, it
    does not imply that gambling sites that usually are identical in blue perform
    not offer other types of games. There are still a great many other
    games that members can choose to enjoy with Depobola official gambling agents such as casino games, racing, lottery, Online Slots and
    so about.

    As the key product, regarding course, this provider offers provided the best soccer gambling games
    that are very complete and varied. Presently there are greater than 20 sorts of sports which can be chosen as betting objects and there are many varieties of games that members can choose from the most popular kinds such as Asian Handicap, Correct Score, 1X2, U / U, O or E, to Mix Parlay.

    Asian Handicap
    You could say this is the most well-liked soccer gambling game
    performed. Even though the game is quite tricky, in reality the
    Asian handicap has never been abandoned by its fans. Some recognition from bettor related
    to be able to this game is since the game is pretty challenging and difficult to guess due to
    the fact it involves voor or perhaps handicap features.

    Correct Report
    As the name indicates, in this game gamers are asked to guess the
    score that happens in the match specifically. The game is split up into two types: the very
    first is half time or perhaps HT where the player will guess the
    score of the match until the halftime break; in addition to the second is FEET or full time exactly where players will guess the
    particular score at the ending in the match.
    1X2 (Double Chance)
    The name associated with the game is used from the betting alternatives offered
    as part of the game: just one, if the player wager for a home team;
    2, if a participant bets for an away team; and X
    in case a player bets regarding a draw. In a new double chance the gamer
    can double the opportunity of victory by choosing 2 of the 3
    available choices such as: 1X, 2X, or 12.

    Mix Parlay
    Mix Parlay is actually a gambling game exactly where players will play various kinds market games into
    1 bet. The winnings of this game will be very
    large since the funds that players place on bets will be multiplied by the likelihood of several of the games that will players play.

    Features That Become Advantages of Sgd777
    If you think the huge benefits are only the total and wide variants regarding the
    game, it indicates an individual don’t really know your pet because there are several other advantages
    that you need to know and also make this specific site never abandoned by simply
    its members.

    Very Aggressive Odds
    Odds offered are extremely competitive when compared to other similar sites.
    Not really only that, this Supplier also indiscriminately serves their customers.
    If other websites will reject players that have won a lot,
    nevertheless not with Sbobet because they will welcome every
    consumer who chose them as a partner to play nicely
    and friendly.
    High Gamble Reduce
    Although limited, yet online betting limits are incredibly high.

    Because of the particular height, you will feel like Sbobet’s guess seems
    limitless. Therefore, you can play just as much as you
    want with us as much since you would like to place a bet.

  • Some individuals on the other hand pay a lot money at one time to boost
    the chance of winning. The system where betting money is thru your bank is secure to work with, and the transaction is always between your account and the
    Ladbrokes. Simple system described in steps.
    Many are simple individuals just struggling to get by. This guide
    gives all the info it’s essential to get into the game.
    But the thing is that to extend your chances of
    winning, you need to position a guess every day. As a substitute, you are able
    to get all the information you need from that same site.
    As lengthy because the proprietor of the Bitcoin looks after
    it correctly utilizing a high-safety wallet and two-issue authentication, it is going
    to be extremely difficult for anybody else to get entry to their funds.

    At any level, nonetheless, the bookie can soar in and make adjustments to the traces that can be found to his players if knows he might be getting heavy one sided action.

    If you are one of those individuals who doesnt like cats, Im sorry to hear that.
    For the above instance, it will look like -7.5 (-110).

    Other gamers can legally buy Mega Tens of millions Lottery tickets online by means of the lottery agent you see
    talked about above. For instance, MEGA Tens of millions is a multi-state lottery and it operates
    in eleven states. As an illustration, you wouldn’t need to choose simply any race horse
    in hopes of profitable a guess. 5. Wager on sports by putting
    your wager. Once you click on the sport of your selecting you’ll want to
    bookmark the web page so you’ll always have essentially the
    most current and updated sports betting odds at your disposal.
    In this technique, chances are you’ll select a low quantity from the
    given set equivalent to 1-50 like 1. Some successful numbers
    have two consecutive numerals, although not on a regular basis.
    After which with the time you saved do something more
    productive than you’d have finished appearing like a
    poor particular person.

  • It’s truly very difficult in this active life to listen news
    on TV, so I only use web for that reason, and get the hottest information.

  • Betberry is one of the particular Official Joker123 gambling internet site agents in Indonesia,
    which also provides various kinds top quality gambling games for example those provided by
    Joker123. Right here you can enjoy a new variety
    of exciting wagering games such as casino, slots, and gambling that is currently popular among followers on the planet gambling that is usually shooting fish or likewise known as
    fish hunter.

    Choosing a Joker123 gambling site to be a new gambling partner is not an easy matter.
    Not a few gambling players in typically the world who feel the particular consequences of choosing partners to play
    gambling carelessly. Obviously you do not want to get trapped
    in the problem, right? Therefore, choose only official wagering agents like Betberry, because only with Betberry you can feel the excitement in playing online betting to the fullest.
    There is no fraud, embezzlement or even theft because Betberry
    functions under an official certificate and is guaranteed to be able to
    be protected from all acts of cyber crime.

    Typically the Advantages of Playing Joker123 Gambling on Betberry
    For anybody who are still cautious to come along, here are the benefits associated with playing
    gambling on the Joker123 Internet site.

    Have an Official Permit
    Whether it’s Betberry, which runs its business in Philippines or Joker123, which provides served countless gambling gamers for years,
    both operate under official licenses. This particular ensures that both Betberry
    in addition to Joker Gaming are certain to have the finest and professional service quality.
    All of the games available are also definitely standardized in addition to have
    the highest high quality in its class.
    Complete Game Variant
    At Joker gaming you may enjoy various sorts of gambling games that
    you want. Starting through the joker slot, joker casino, before the
    fish capturing game that is very fascinating. No need to get worried you
    will feel dullness because in Joker Gambling you will not run out associated with
    games, there will always be challenging online games that
    you can take pleasure in here.
    You can likewise enjoy the latest characteristic, live casino!
    In this particular game you can feel the sensation like playing gambling directly in luxurious gambling houses similar to Todas las Vegas or Macau.
    You may also invite beautiful dealers to interact with you throughout the game to get rid
    of boredom.
    Quick Registration Method
    No need for charge cards, foreign bank accounts, or even deposits with very high amounts.

    Register as the member at Betberry, all you require is
    an ID credit card, contact number, and an lively email address.
    The enrollment process can be designed in a few minutes plus all for free.
    Obtain a user ID and pass word that will be used to log in to
    be able to Joker123, only once you sign up you can enjoy almost all available Joker123 gambling online games.

    Easy Play Access
    Right now there is no need to go out of your home or
    apart from home to appreciate the thrilling gambling game because with Betberry everything is achievable.
    You can enjoy hundreds associated with Joker123 gambling games straight throughout your computer or laptop computer screen.
    But if a person are a person who else travels a lot probably
    you want to try out the Joker123 application of which can be directly mounted to the smartphone
    you might have.
    Betberry provides an alternate Joker123 link if the main site address is unavailable.
    This link is used in order to avoid government blockages
    of which block customers when these people want to access the overall game on Joker123.
    So conserve alternative links to make it easier for you to accessibility the game.

    24 Hr Customer Service
    If you encounter difficulties throughout the game or perhaps need
    help with all the online game provided by Betberry in addition to Joker123, don’t hesitate to contact Betberry customer services whenever you need.
    Betberry customer service is always active 24 hours
    a day and is ready to help you whenever needed.
    Secure Monetary Dealings
    As an recognized agent in Indonesia, of course we want to give the best
    service with regard to its customers, including in phrases of financial transactions.
    On this particular site you can bring out all transactions securely
    through leading local banking institutions in Indonesia such because BNI, BCA, BRI,
    Mandiri and CIMB Niaga.
    Thus don’t hesitate anymore, sign up an account on the
    particular Joker123 gambling site along with Betberry and enjoy all the ease of being able to access all available games.
    All the best!

  • The Best and Most Trusted On the internet Sbobet Agent
    Playing on gambling online sites has become a familiar thing for fans of this online game to use their luck.
    Especially in a rustic that forbids all varieties of gambling routines such as Indonesia, Sgd777 agents are increasingly mushrooming and are becoming
    the popular alternative for bettors to channel their hobbies and interests safely.

    In modern periods, bettors no longer need to go far to Las Vegas
    or Macau to appear for bookies. Only together with a home computer or even laptop,
    even more basic, smartphone and internet network, whoever and wherever a person are,
    can immediately perform online gambling comfortably and freely.

    Indeed, finding a new trusted Sbobet agent in order to be used being a perform partner is a challenging problem.

    With so many sites offering a selection of attractive features and promos, it’s difficult to
    be able to distinguish which types really provide professional
    services and which types are fake.

    Most persons will pick the biggest gambling site directly that may be chosen by typically the bettor, but the drawback
    is often required a huge deposit in order to register
    / join the particular game. Even some people, you will find
    those who locate it difficult if an individual have to use websites that
    use English.

    Ultimately, the players, especially within Indonesia, inevitably have
    to find an Indonesian online gambling agent who certainly has international quality games and is able to provide the
    particular best service but alternatively can provide peace of mind in playing in addition to will not create a damaged wallet.

    Tips for Selecting a Trusted Soccer Gambling Broker
    There are several huge names online that you can select from as
    play partners. Between Depobola. This agent will be arguably
    the biggest gambling promoter in Indonesia that has served its users for years.
    Listed here are the reasons why Depobola will be very popular
    like a reliable Sbobet Indonesia agent.

    daily Customer Support
    The 1st feature you must have got is having CS or even Customer Support that is always online at just
    about all times. This is meant to ensure that the comfort and overall flexibility of members when playing always awake.

    Any difficulties and confusion faced by players can end
    up being immediately resolved by making use of Customer Support who can constantly be contacted
    for 24 hours straight. And avoid agents whose customer services is hard to contact, that could be that the site is fraudulent.

    Safe & Fast Financial Dealings
    Not all trusted sports gambling agents have a very great financial transaction system.
    A few of them require pretty a long time, even up to days plus cannot guarantee the protection of their customers whenever transacting.

    Unlike the truth along with Depobola. Financial transactions
    such as deposits, withdrawals, and transfers can be done in just minutes in addition to of course use a new high-level
    security alarm. In addition, the process can end
    up being carried out easily since it has worked carefully with assorted major banks in the
    country.
    Several types of Games
    If in the past gambling only struggled about lottery and soccer gambling, now Depobola
    provides exciting innovations by presenting a variety
    of interesting and fun games. Not simply as a possible agent of Sbobet Online,
    variations of credit card games such as poker or perhaps dominoes, casino games this kind of as
    roulette or baccarat, to the slots offered.
    All game products are usually guaranteed to have grade-A top quality since they are provided
    by leading gaming providers such since Sbobet, Maxbet while others.
    Just at Depobola, online gambling gives the most complete sport making sure
    the project you can devote time without feeling fed up.

    Attractive Promos and Bonuses
    The last tip for choosing the best site is usually to be
    sure that the site offers a huge selection of promos and bonuses that will are a
    shame in case you miss them. This feature will make the game easier and much more
    fun.
    Depobola has prepared various advertisements and bonuses that can be obtained by members in an easy
    way. For example , like a welcome bonus promo, where you
    only require to register as a new member, then you
    certainly automatically have the right to be capable of geting the
    bonus.
    There are usually still a great many other promos proposed by the best Depobola
    agents. Therefore, don’t wait virtually any longer and immediately sign up yourself now to sign up for with countless other users to feel the enjoyment of playing with
    Trusted Online Sbobet agents within Indonesia.

  • Why do you spam other people’s web sites to promote your own, cunt?

Leave a Reply to trachycarpus fortunei Cancel reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>