Editing Introduction to a Self Managed Life: a 13 hour & 28 minute presentation by FUTO software (section)

= Step 7: Backing up virtual machines =

Now that we have a giant storage array that will continue working even in the event of multiple drive deaths, we can set up our virtual machines to back up regularly. This way, if we destroy one with idiocy, or if it becomes corrupt, we can restore it instantly to what it was like before the mess happened.

<span id="backup-script-creation"></span>
==== 7.1 Backup script creation: ====

This script below will allow you to have your virtual machines backed up automatically. It does the following:

* Shuts down the virtual machine
* Copies its disk image qcow2 file to the <code>/mediapool/vmbackups</code> zfs dataset
* Copies its configuration so it can be set up again
* Keeps five backups but deletes the oldest ones after you have five.

This means the following:

* You can mess things up by deleting files you weren’t supposed to, mess up configurations and programs, and restore everything to where it was last week with one or two kindergarten level GNU/Linux commands.
* You can migrate this to another computer entirely &amp; start the virtual machine up there.

shuts each virtual machine down one by one, backs up the virtual

Save this as <code>/root/vm_backup.sh</code>:

<pre># Open the text editor
sudo nano -w /root/vm_backup.sh</pre>
<pre>#!/bin/bash

# thank you to stack overflow for giving me the courage to wade through 100s of posts and hack together something that looks like it works. 

# config for backups
BACKUP_DIR=&quot;/mediapool/vmbackups&quot;
LOG_FILE=&quot;/var/log/vm_backups.log&quot;
RETENTION_DAYS=56  # how long to keep backups

# Function to write messages to our log file`
log_message() {
    # Get the current timestamp and message
    echo &quot;[$(date '+%Y-%m-%d %H:%M:%S')] $1&quot; | tee -a &quot;$LOG_FILE&quot;
}

# Function to find the actual disk path for a VM when the default path doesn't exist
# Uses virsh dumpxml to get the disk source path from the VM's XML configuration
find_vm_disk_path() {
    local vm_name=$1
    # Get the VM's XML configuration and extract the first disk source path
    # Using grep with -o to only output the matched portion
    # Using sed to extract just the path part from the source attribute
    local disk_path=$(virsh dumpxml &quot;$vm_name&quot; | grep -o &quot;source file='[^']*'&quot; | head -n1 | sed &quot;s/source file='\(.*\)'/\1/&quot;)
    
    # Check if we found a path and if it exists
    if [ -n &quot;$disk_path&quot; ] &amp;&amp; [ -f &quot;$disk_path&quot; ]; then
        echo &quot;$disk_path&quot;
        return 0
    else
        return 1
    fi
}

# main backup function 
backup_vm() {
    local virtual_machine_name=$1  # The name of the virtual machine we're backing up
    local date_stamp=$(date +%Y%m%d)  # Today's date for the backup file name
    local source_file=&quot;/var/lib/libvirt/images/${virtual_machine_name}.qcow2&quot;  # Where the virtual machine is
    
    # If the default path doesn't exist, try to find the actual disk path
    if [ ! -f &quot;$source_file&quot; ]; then
        log_message &quot;Default disk path not found for ${virtual_machine_name}, searching XML configuration...&quot;
        local found_path=$(find_vm_disk_path &quot;$virtual_machine_name&quot;)
        
        # If we found a valid path, use it instead
        if [ -n &quot;$found_path&quot; ]; then
            log_message &quot;Found alternate disk path: ${found_path}&quot;
            source_file=&quot;$found_path&quot;
        fi
    fi
    
    local backup_file=&quot;${BACKUP_DIR}/${virtual_machine_name}-${date_stamp}.qcow2&quot;  # Where we're putting the backup of it
    local config_file=&quot;${BACKUP_DIR}/${virtual_machine_name}-${date_stamp}.xml&quot;  # Where it saves the virtual machine config
    
    # Check if source file exists before attempting backup
    if [ ! -f &quot;$source_file&quot; ]; then
        log_message &quot;ERROR: Source file $source_file does not exist for ${virtual_machine_name}&quot;
        return 1
    fi
    
    # Announce backup is starting
    log_message &quot;Starting backup process for ${virtual_machine_name}&quot;
    
    # Save virtual machine's config
    virsh dumpxml &quot;$virtual_machine_name&quot; &gt; &quot;$config_file&quot;
    
    # Set ownership and permissions for config file
    chown libvirt-qemu:kvm &quot;$config_file&quot;
    chmod 644 &quot;$config_file&quot;

    # Try to shut down the virtual machine nicely 
    log_message &quot;Shutting down ${virtual_machine_name}&quot;
    virsh shutdown &quot;$virtual_machine_name&quot;
    
    # Wait patiently for the virtual machine to shut down 
    local count=0
    while [ &quot;$(virsh domstate $virtual_machine_name)&quot; != &quot;shut off&quot; ] &amp;&amp; [ $count -lt 30 ]; do
        sleep 10
        count=$((count + 1))
    done
    
    # If it doesn't turn off, make it turn off(like holding the power button)
    if [ &quot;$(virsh domstate $virtual_machine_name)&quot; != &quot;shut off&quot; ]; then
        log_message &quot;WARNING: Force shutting down ${virtual_machine_name}&quot;
        virsh destroy &quot;$virtual_machine_name&quot;
        sleep 10
    fi
    
    # Make sure it's actually off - trust but verify
    if [ &quot;$(virsh domstate $virtual_machine_name)&quot; != &quot;shut off&quot; ]; then
        log_message &quot;ERROR: Failed to shut down ${virtual_machine_name}&quot;
        return 1
    fi
    
    # Create the backup - doesn't use compression since qemu-img convert compression is single threaded and insanely slow
    log_message &quot;Creating backup of ${virtual_machine_name}&quot;
    if ! qemu-img convert -p -f qcow2 -O qcow2 &quot;$source_file&quot; &quot;$backup_file&quot;; then
        log_message &quot;ERROR: Backup failed for ${virtual_machine_name}&quot;
        virsh start &quot;$virtual_machine_name&quot;
        return 1
    fi
    
    # Set ownership and permissions for backup file
    chown libvirt-qemu:kvm &quot;$backup_file&quot;
    chmod 644 &quot;$backup_file&quot;

    # Make sure the backup isn't insanely small since that means this didn't work
    # Fixed stat command for Linux systems
    local source_size=$(stat -c%s &quot;$source_file&quot;)
    local backup_size=$(stat -c%s &quot;$backup_file&quot;)
    if [ &quot;$backup_size&quot; -lt 1048576 ]; then  # Less than 1MB is suspicious - like a $5 &quot;genuine&quot; Rolex
        log_message &quot;ERROR: Backup file suspiciously small for ${virtual_machine_name}&quot;
        rm -f &quot;$backup_file&quot; &quot;$config_file&quot;
        virsh start &quot;$virtual_machine_name&quot;
        return 1
    fi
    
    # Turn virtual machine back on when backup is done. 
    log_message &quot;Starting ${virtual_machine_name}&quot;
    virsh start &quot;$virtual_machine_name&quot;
    
    # Wait for it to come back online 
    count=0
    while [ &quot;$(virsh domstate $virtual_machine_name)&quot; != &quot;running&quot; ] &amp;&amp; [ $count -lt 12 ]; do
        sleep 5
        count=$((count + 1))
    done
    
    # Make sure it actually started(inspect what you expect)
    if [ &quot;$(virsh domstate $virtual_machine_name)&quot; != &quot;running&quot; ]; then
        log_message &quot;ERROR: Failed to start ${virtual_machine_name}&quot;
        return 1
    fi
    
    # announce that it worked
    log_message &quot;Backup of ${virtual_machine_name} completed!&quot;
    
    # Clean up old backups - because nobody likes a full hard drive
    log_message &quot;Cleaning up old backups for ${virtual_machine_name}&quot;
    find &quot;$BACKUP_DIR&quot; -name &quot;${virtual_machine_name}-*.qcow2&quot; -mtime +${RETENTION_DAYS} -exec rm -f {} \;  # Delete old qcow2 files
    find &quot;$BACKUP_DIR&quot; -name &quot;${virtual_machine_name}-*.xml&quot; -mtime +${RETENTION_DAYS} -exec rm -f {} \;   # Delete old xml files
}

# Start of the main backup process
log_message &quot;Starting backup process&quot;

# Make sure we're running as root 
if [ &quot;$EUID&quot; -ne 0 ]; then
    log_message &quot;ERROR: Must run as root&quot;
    exit 1
fi

# Check if the backup directory exists 
if [ ! -d &quot;$BACKUP_DIR&quot; ]; then
    log_message &quot;ERROR: Backup directory $BACKUP_DIR does not exist&quot;
    exit 1
fi

# Get list of ALL virtual machines, not just running ones
# Changed to list all VMs instead of just running ones
VMS=($(virsh list --all --name))

# Check if we have enough disk space to back up
available_space=$(df -B1 &quot;$BACKUP_DIR&quot; | awk 'NR==2 {print $4}')
required_space=0

# Calculate how much space we need
for virtual_machine in &quot;${VMS[@]}&quot;; do
    if [ -n &quot;$virtual_machine&quot; ]; then
        # Try the default path first
        local_path=&quot;/var/lib/libvirt/images/${virtual_machine}.qcow2&quot;
        
        # If default path doesn't exist, try to find actual path
        if [ ! -f &quot;$local_path&quot; ]; then
            local_path=$(find_vm_disk_path &quot;$virtual_machine&quot;) || continue
        fi
        
        if [ -f &quot;$local_path&quot; ]; then
            virtual_machine_size=$(du -b &quot;$local_path&quot; 2&gt;/dev/null | cut -f1)
            required_space=$((required_space + virtual_machine_size))
        fi
    fi
done

# Make sure we have enough space 
if [ &quot;$available_space&quot; -lt &quot;$required_space&quot; ]; then
    log_message &quot;ERROR: Insufficient space in backup directory&quot;
    exit 1
fi

# loop for backing up every virtual machine
for virtual_machine in &quot;${VMS[@]}&quot;; do
    if [ -n &quot;$virtual_machine&quot; ]; then
        backup_vm &quot;$virtual_machine&quot;
    fi
done

# announce it's all done
log_message &quot;Backup process completed!&quot;
</pre>
<blockquote>'''Nerd note: This script would be laughed out of the room for use in production environments for major web companies &amp; datacenters.''' This script turns off the virtual machine to back it up.

This means that at 1 AM, the service goes down. This would be unacceptable in a production environment where people expect the service to be available 24/7.

There are ways to do live backups where you flush mysql tables and lock them, make redis background save, pause call processing in asterisk, pause io, create atomic snapshots, coordinate with databases of all the different programs…. the audience of this guide is a person running a home server in his closet. Do you really want to subject a beginner to docker volumes that may not be in a consistent state, email delivery/receipt being interrupted, database transactionst hat are messed with in the middle fo a write, corrupt call recordings, partially written large files, all so someone can get live backups of a server in their closet, you monster?

If you need that level of uptime, you’re not a newbie reading this guide. or you are, and you need to hire a consultant to set you up with something like veeam.

To subject a newbie to the risk of error/corruption/screwups that comes with doing live backups for these things when they’re at the level of this guide being helpful to them is cruel.
</blockquote>
<span id="set-permissions-so-script-works"></span>
==== 7.2 Set permissions so script works ====

This won’t work if we don’t give it permissions to be executable.

<pre># Make script executable
sudo chmod +x /root/vm_backup.sh

# Test script permissions
sudo -u root /root/vm_backup.sh</pre>
<span id="tell-computer-to-run-script-every-week-at-1-am-on-sunday"></span>
==== 7.3 Tell computer to run script every week, at 1 AM on Sunday ====

Cron is a scheduler. You can tell cron to run a command, a script, etc. once a week, once a month, twice a day, every 10 minutes. We’re going to set this to back up at 1 AM every Sunday.

<ol style="list-style-type: decimal;">
<li><p>Open root’s crontab:</p>
<pre>sudo crontab -e</pre></li>
<li><p>Add this line:</p>
<pre>0 1 * * 0 /root/vm_backup.sh &gt;&gt; /var/log/vm_backup.log 2&gt;&amp;1</pre>
<p>This will:</p>
<ul>
<li><p>Run at 1:00 AM every Sunday</p></li>
<li><p>Log output to <code>/var/log/vm_backup.log</code></p></li>
<li><p>Include both standard output and errors in the log</p></li>
<li><p>The virtual machine will be down while the transfer occurs</p></li></ul>

<p>If anyone is calling Rossmann Repair Group at 1 AM on a Sunday morning, they deserve to get a busy signal. Actually they deserve allison smith telling them to get the fk out of here. but a busy signal will suffice.</p></li></ol>

==== 7.4 Make sure cron is running ====

<pre>sudo systemctl status cron</pre>
View scheduled cron jobs:

<pre>sudo crontab -l</pre>
<span id="step-8-restoring-a-virtual-machine-from-a-backup"></span>

== Step 8: Restoring a virtual machine from a backup ==

So you messed up and deleted everything inside your virtual machine. You want to go back to where you were before.

Remember:

'''A BACKUP PLAN IS ONLY AS GOOD AS HOW EASY IT IS TO RESTORE FROM A BACKUP!'''

<span id="basic-restore"></span>
=== Basic Restore ===

By “basic restore” I mean what to do when you messed up a program configuration or deleted files inside a virtual machine or corrupted something accidentally. You want to go back to the image of the virtual machine you had before, on the same happycloud host computer.

<span id="before-you-start"></span>
==== 8.1 Before You Start ====

I’m assuming the following is true: - Your virtual machine is already defined in Virtual Machine Manager(you see it when you run virtual machine manager GUI) - Your backups are in <code>/mediapool/vmbackups</code> - The backups were created using the qemu-img backup script I provided above - You just need to restore the virtual machine’s disk because you messed up some files or programs

<span id="find-your-backup"></span>
==== 8.2 Find Your Backup ====

# List available backups for your virtual machine:

<pre>ls -l /mediapool/vmbackups/name-of-your-virtual-machine-*.qcow2</pre>
You’ll see files named like this: - <code>name-of-your-virtual-machine-20240101.qcow2</code> - <code>name-of-your-virtual-machine-20240108.qcow2</code>

These are the disk image files that have all of the data/programs/databases/operating system.

Each backup will have an XML file to go with it: - <code>name-of-your-virtual-machine-20240101.xml</code> - <code>name-of-your-virtual-machine-20240108.xml</code>

These are the files that tell virtual machine manager all of the details about your virtual machine(RAM/CPU, hardware setup, etc.)

Pick the most recent backup before you screwed something up.

<span id="fast-restore"></span>
==== Fast Restore: ====

# Turn off the virtual machine

<pre># Shut down the virtual machine gracefully
virsh shutdown name-of-your-virtual-machine

# Wait until it's actually off. Check status with:
virsh list --all</pre>
<ol start="2" style="list-style-type: decimal;">
<li>Backup Current Disk (just in case)</li></ol>

<pre># Move the current (messed up/broken) disk with date
mv /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2 /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2.broken-$(date +%Y%m%d)</pre>
<ol start="3" style="list-style-type: decimal;">
<li>Restore Backup</li></ol>

<pre># a cool command to put the virtual machine back where it was
qemu-img convert -p -f qcow2 -O qcow2 /mediapool/vmbackups/name-of-your-virtual-machine-20240101.qcow2 /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2

# set permissions so that our virtual machine management stuff can use it.
chown libvirt-qemu:kvm /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2
chmod 644 /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2</pre>
<ol start="4" style="list-style-type: decimal;">
<li>Start the Virtual Machine</li></ol>

<pre>virsh start name-of-your-virtual-machine</pre>
<span id="check-the-restore"></span>
==== Check the Restore ====

# Watch the virtual machine console in Virtual Machine Manager to make sure it boots
# Try logging in when it’s up
# Check that services(mailcow, immich, syncthing) actually work

<span id="complicated-restore"></span>
=== Complicated Restore ===

Let’s say you destroyed more. You also messed up the virtual machine’s configuration in virsh. You edited the xml file for the virtual machine or messed with its settings in the '''Virtual Machine Manager''' GUI, and now nothing works.

For a complete restore of both disk &amp; configuration:

# Remove the current virtual machine:

<pre>virsh destroy name-of-your-virtual-machine
virsh undefine name-of-your-virtual-machine</pre>
<ol start="2" style="list-style-type: decimal;">
<li>Restore the Disk:</li></ol>

<pre># Convert the compressed backup to the images directory
qemu-img convert -p -f qcow2 -O qcow2 /mediapool/vmbackups/name-of-your-virtual-machine-20240101.qcow2 /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2

# Fix permissions
chown libvirt-qemu:kvm /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2
chmod 644 /var/lib/libvirt/images/name-of-your-virtual-machine.qcow2</pre>
<ol start="3" style="list-style-type: decimal;">
<li>Restore the virtual machine config:</li></ol>

<pre># The backup includes the XML configuration file
virsh define /mediapool/vmbackups/name-of-your-virtual-machine-20240101.xml</pre>
<ol start="4" style="list-style-type: decimal;">
<li>Start the VM:</li></ol>

<pre>virsh start name-of-your-virtual-machine</pre>
<span id="common-screwups"></span>
=== Common screwups ===

# '''“Failed to convert image”''':
#* Make sure you have enough disk space
#* Check that the backup file isn’t corrupted
# '''“Failed to start VM”''':
#* Usually permissions. Everyone is excited to realize they had a backup file whilst copying it back; in the excitement of realizing you actually HAVE a backup, nobody remembers to set permissions on the backup file.
#* Check that the XML file matches the system config. use virtual machine manager for this to see if anything sticks out in the GUI as a stupid mistake.
#* Verify all paths exist
# '''“Could not access storage file”''': Check paths in both:
#* <code>/var/lib/libvirt/images/</code>
#* The virtual machine XML config
#* Make sure permissions are right (644 for files)

<span id="verifying-success"></span>
=== Verifying Success ===

After restoration, verify: 1. VM boots properly 2. Network connectivity works 3. All services start correctly 4. Data and configurations are as expected 5. Check logs for any errors

If something isn’t right, you can always try an older backup - they’re kept for 56 days.

<span id="accessing-your-samba-share-from-any-device"></span>