KVM

DNS

dnsmasq and named won't run on the same machine without some tweaks since they both want to bind to port 53. The solution is to alter each config to listen on specific IPs only.

/etc/named.conf - named should listen on the external IP and localhost.

listen-on port 53 { 127.0.0.1; 213.229.103.79; };
listen-on-v6 port 53 { ::1; 2a02:af8:3:2000::7982; };

/etc/dnsmasq.conf - dnsmasq should listen on the virbr0 interface only

listen-address=192.168.122.1
bind-interfaces

If you prefer, you can use interface=virbr0 instead of listen-address=192.168.122.1

Networking

Configuring networking on KVM to work with individually routed IPs (or a small subnet of routed IPs) where the routed IPs aren't related to the primary IP of the host involves creating a virtual bridge, enabling some firewall rules and manually creating some routes on both the host and guest.

Virtual bridge configuration

Virtual bridge definition is as follows. The IP address used can be anything private since it's only used internally for routing.

<network>
  <name>routed</name>
  <forward mode='route'/>
  <bridge name='virbr1' dev='eth0' delay='0' />
  <ip address='192.168.123.1' netmask='255.255.255.255'>
  </ip>
</network>

Save the above as net-routed.xml and then create/start the network.

# virsh net-define net-routed.xml
# virsh net-start routed
# virsh net-autostart routed

Startup hooks

Define the IP address(es) to be routed in /etc/libvirt/hooks/routed-ips

ROUTED_GW="192.168.123.1"
ROUTED_DEV="virbr1"
ROUTED_IPS="92.48.112.177 92.48.112.178 92.48.112.179"

This qemu/libvirt script uses the above file and should be created as /etc/libvirt/hooks/qemu (don't forget to set the permissions as +x). The additions to manage the iptables rules were added by me, the original script only added the routes.

#!/bin/sh
# Found at http://blog.gadi.cc/single-ip-routing-in-libvirt/
# Add individual IPs for our routed network to the routing table
#
# Since no hook exists for net-start, the best we can do is check if
# all the IPs are added everytime a VM is launched, without re-adding.
# When a net-destroy occurs, the routes will be automatically removed.
. `dirname $0`/routed-ips
if [ "$2" == "start" ]; then
   for IP in $ROUTED_IPS ; do
       if [ "`ip route list | grep $IP`" == "" ] ; then
           ip route add $IP via $ROUTED_GW dev $ROUTED_DEV
       fi
       # Remove the old firewall rules if present
       iptables -D FORWARD -d $IP -o virbr1 -j ACCEPT 
       iptables -D FORWARD -s $IP -i virbr1 -j ACCEPT 
       # Add them back in before
       iptables -I FORWARD -d $IP -o virbr1 -j ACCEPT 
       iptables -I FORWARD -s $IP -i virbr1 -j ACCEPT 
   done
fi
exit 0

The script is run like this during the startup phase of virtual machines.

# /etc/libvirt/hooks/qemu guest_name start begin -

Guest kickstart

Guest kickstart config should contain the following sections. The post-install script creates default routing via the virtual bridge internal IP.

network --device eth0 --bootproto static --ip=92.48.112.178 --netmask=255.255.255.255 --nameserver=213.229.103.79

%post

cat > /etc/sysconfig/network-script/route-eth0 <<EOF
192.168.123.1 dev eth0
default via 192.168.123.1 dev eth0
EOF

Adding serial console after install

CentOS 6.6

You'll need to make sure the grub config has console=ttyS0 as a kernel parameter.

initctl start serial DEV=ttyS0 SPEED=9600

All being well, you should then be able to use virsh console to connect and get a login prompt.

Hot-add disk to running VM

# qemu-img create newdisk.img 10G

# cat > newdisk.xml <<EOF
disk type='file' device='disk'>
  <driver name='qemu' type='qcow2'/>
  <source file='/path/to/newdisk.img'/>
  <target dev='vdb' bus='virtio'/>
</disk>
EOF

# virsh attach-device <domain name> /path/to/disk.xml

Check the guest to see if the disk was hotplug-inserted. The kernel should be triggered, as can be checked with dmesg:

virtio-pci 0000:00:06.0: irq 30 for MSI/MSI-X
 vdb: unknown partition table

Snapshot disk images

Disk images need to be qcow2 format to be able to have snapshots made so first of all, convert the raw disk to qcow2. With the VM powered off:

# qemu-img convert -p -O qcow2 vmname.dsk vmname.dsk.qcow2
# virsh edit vmname

Change

<driver name='qemu' type='raw' cache='none'/>
<source file='/kvm/vmname.dsk'/>

to

<driver name='qemu' type='qcow2' cache='none'/>
<source file='/kvm/vmname.dsk.qcow2'/>

To create a snapshot

# virsh snapshot-list vmname
Name                 Creation Time             State
------------------------------------------------------------
Before OS upgrade    2014-08-04 13:27:55 +0100 shutoff

# virsh snapshot-create-as vmname "After OS upgrade"
Domain snapshot After OS upgrade created

# virsh snapshot-list vmname
Name                 Creation Time             State
------------------------------------------------------------
After OS upgrade     2014-08-14 13:32:49 +0100 running
Before OS upgrade    2014-08-04 13:27:55 +0100 shutoff