Internal LAN on LXC

When I wrote my previous post about setting up LXC, one of the things I found was that when installing the lxc package, it went ahead and created an "lxcbr0" interface.

It turns out that this interface is actually an "internal network" which you can connect your VM's to if you want them talking to each other directly, as opposed to any network which the host is also on.

To setup my VM's I just added another interface and connected it to the "lxcbr0" bridge, by adding the following lines to the configuration:

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lxcbr0
lxc.network.hwaddr = 4a:49:43:49:79:ef

and then configuring the interface in the "interfaces" file:

auto eth1
iface eth1 inet static
    address 10.0.3.2
    netmask 255.255.255.0

Then I did the same to another VM and they were able to talk to each other.

Btrfs snapshots and LXC

In my previous post I talked about LXC, which is a light-weight virtualization technology for Linux. One thing which LXC lacks is the ability to make snapshots. As the VM is running as a regular process in RAM, at the moment it's not possible to just make a copy of the ram file, as it is in VMware etc...

So, in order to work around this, we're only going to be taking snapshots of VM's when they're shutdown and we're going to be making use of the Btrfs snapshot functionality.

First of all, we need to create a Btrfs filesystem. I assume that you have a spare drive or partition which you can use for this:

$ sudo mkfs.btrfs -L btrfs-test /dev/sda8 

WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL 
WARNING! - see http://btrfs.wiki.kernel.org before using 

fs created label btrfs-test on /dev/sda8 nodesize 4096 leafsize 4096 sectorsize 4096 size 10.00GB 
Btrfs Btrfs v0.19

Now we can mount the filesystem:

$ sudo mount /dev/sda8 /lxc

Before we set about creating our VM's, we're going to create some "subvolumes", which we'll be able to snapshot. We're going to use this feature of Btrfs to handle the snapshotting of our VM's.

$ sudo btrfs subvolume create /lxc/vm0
Create subvolume '/lxc/vm0'

Now that we've done this, we can go ahead and create a VM using the template scripts, and configure it, as described in my previous post.

$ sudo /usr/lib/lxc/templates/lxc-ubuntu -p /lxc/vm0 ...

Now that we've created a brand new VM, we're going to create a snapshot of it's "clean" state, so that we can roll back to it, should something go wrong. We do this by creating a snapshot called "clean" of the /lxc/vm0 subvolume.

$ cd /lxc/
$ sudo btrfs subvolume snapshot vm0 vm0-clean
Create a snapshot of 'vm0' in './vm0-clean'
$ sudo btrfs subvolume list /lxc
ID 256 top level 5 path vm0
ID 258 top level 5 path vm0-clean

Now we can start the VM:

$ sudo lxc-start -n vm0 -f /lxc/vm0/config

And install/configure our software:

$ sudo apt-get install apache2 postgresql

Now, suppose that we're humming along nicely, but then realise that we've made a mistake and installed apache2 instead of tomcat6 and postgresql instead of mysql-server and want to start over. All we would need to do is to delete the "vm0" subvolume and rename the "vm0-clean" directory to "vm0":

$ sudo btrfs subvolume delete /lxc/vm0
Delete subvolume '/lxc/vm0'
$ sudo btrfs subvolume list /lxc
ID 258 top level 5 path vm0-clean
$ sudo mv vm0-clean vm0
$ sudo btrfs subvolume list /lxc
ID 258 top level 5 path vm0

Notice how the ID of the snapshot doesn't change, even though we've renamed it.

Now, when we start up the VM, we can see that the "apache2" and "postgresql" packages haven't been installed yet, because we've rolled our VM back to the original snapshot that we've taken.

Now, perhaps the scenario with installing the wrong packages isn't that realistic (it would probably be easier to just remove the packages instead of rolling back the snapshots), however, this was just chosen to demonstrate the capabilities of the technology and you can probably imagine a scenario where a snapshot would be more useful. e.g. testing out a software upgrade, which you're concerned might break some functionality.

NOTE: Another way of using snapshots is to mount them directly, by passing the "-o subvol=..." option at mount time, as described at: http://btrfs.ipv5.de/index.php?title=SysadminGuide#Managing_snapshots

LXC on Ubuntu

This post talks about how to setup LXC (Linux Containers) on Ubuntu 12.04. LXC is an operating system-level virtualization technology, which allow you to run multiple virtual machines on one host.

There are quite a few limitations to this type of virtualization, when compared with the type of full, emulator style virtualization that VMware, VirtualBox etc... use. One of the main ones is that you'll be unable to run different operating systems on the virtualization host. i.e. we can only run Linux VM's on our host.

The big advantage is performance. Because the host doesn't have to bother with all of the code which does virtual hardware emulation, the virtual machines run a lot faster in general.

So, to setup LXC, we first need to install it:

sudo apt-get install lxc lxctl uuid btrfs-tools

The 'lxctl', 'uuid' and 'btrfs-tools' packages aren't really needed, but come recommended, and it doesn't hurt to install them.

Now, at this point I did a reboot, which may not be necessary, and afterwards checked the LXC configuration using:

lxc-checkconfig

You should find that all of the different settings are set to "enabled".

Now that we've got LXC installed, we can go ahead and start creating our first VM. Luckily, the 'lxc' package comes with a set of template scripts, which make setting up a VM easy. These scripts are located under '/usr/lib/lxc/templates' and to create our first VM we run:

sudo /usr/lib/lxc/templates/lxc-ubuntu -p /lxc/vm0/
 
Where '/lxc/vm0' is the path to the VM. Note that you will have to create this directory as it doesn't exist by default.

Once you run that script, you'll see a lot of output from the VM getting created and initialized. From the output and from looking at the template script, it looks like the template takes all of the currently installed packages, copies them over to the VM filesystem and configures them.

Once, this is done, we need to configure the networking, which I found to be the trickiest part of the setup. Because the guest VM is using the same hardware as the host, it has the ability to use the network interface attached to the host. Now, obviously you don't want both the host and the guest using the same interface, as it will lead to IP address and MAC address conflicts. So, the guest should have a distinct MAC and/or IP address.

There are several ways to configure the networking for the guest VM's and there are example configuration files of the many ways that they can be configured under file:///usr/share/doc/lxc/examples/ (note that you can enter this location into your web browser and it should load). For our purposes, we're going to go with the lxc-veth.conf file, which sets up a virtual network interface connected to a network bridge which we have to create.

So, firstly, we need to create a network bridge, which is done by adding the following lines to the /etc/network/interfaces file:

auto br0
iface br0 inet dhcp
bridge_ports eth0

This will create the bridge that we're going to connect the virtual network interface of the VM to. In order to enable it, restart the 'networking' service:

sudo service networking restart

Note that we have to create a bridge, even if we've only got one physical interface to connect to it.

Then add the relevant lines from the example config to the VM configuration file, under /lxc/vm0/config:

...
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = 4a:49:43:49:79:bf
lxc.network.ipv4 = 10.0.0.0/24
...

Note that the IP network should match that of your host interface, otherwise you might have some problems getting an IP through DHCP. Also, note that the MAC address is just taken from the example file an was probably randomly generated.

That should complete the configuration. We are now free to start up the VM using the 'lxc-start' command:

sudo lxc-start -n vm0 -f /lxc/vm0/config

This should start up the VM and bring up the console on the terminal screen. For the default Ubuntu template, you can log in using ubuntu/ubuntu username/password pair.

Once you've logged in, you can confirm that the VM has a different IP address to that of the host and start configuring it. 

Funnily enough, just as I've finished writing up this post, I stumbled upon a Launchpad blog post talking about how they're using LXC to speed up their testing: http://blog.launchpad.net/general/parallelising-the-unparallelisable