Btrfs snapshots and LXC

In my previous post I talked about LXC, which is a light-weight virtualization technology for Linux. One thing which LXC lacks is the ability to make snapshots. As the VM is running as a regular process in RAM, at the moment it's not possible to just make a copy of the ram file, as it is in VMware etc...

So, in order to work around this, we're only going to be taking snapshots of VM's when they're shutdown and we're going to be making use of the Btrfs snapshot functionality.

First of all, we need to create a Btrfs filesystem. I assume that you have a spare drive or partition which you can use for this:

$ sudo mkfs.btrfs -L btrfs-test /dev/sda8 

WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL 
WARNING! - see http://btrfs.wiki.kernel.org before using 

fs created label btrfs-test on /dev/sda8 nodesize 4096 leafsize 4096 sectorsize 4096 size 10.00GB 
Btrfs Btrfs v0.19

Now we can mount the filesystem:

$ sudo mount /dev/sda8 /lxc

Before we set about creating our VM's, we're going to create some "subvolumes", which we'll be able to snapshot. We're going to use this feature of Btrfs to handle the snapshotting of our VM's.

$ sudo btrfs subvolume create /lxc/vm0
Create subvolume '/lxc/vm0'

Now that we've done this, we can go ahead and create a VM using the template scripts, and configure it, as described in my previous post.

$ sudo /usr/lib/lxc/templates/lxc-ubuntu -p /lxc/vm0 ...

Now that we've created a brand new VM, we're going to create a snapshot of it's "clean" state, so that we can roll back to it, should something go wrong. We do this by creating a snapshot called "clean" of the /lxc/vm0 subvolume.

$ cd /lxc/
$ sudo btrfs subvolume snapshot vm0 vm0-clean
Create a snapshot of 'vm0' in './vm0-clean'
$ sudo btrfs subvolume list /lxc
ID 256 top level 5 path vm0
ID 258 top level 5 path vm0-clean

Now we can start the VM:

$ sudo lxc-start -n vm0 -f /lxc/vm0/config

And install/configure our software:

$ sudo apt-get install apache2 postgresql

Now, suppose that we're humming along nicely, but then realise that we've made a mistake and installed apache2 instead of tomcat6 and postgresql instead of mysql-server and want to start over. All we would need to do is to delete the "vm0" subvolume and rename the "vm0-clean" directory to "vm0":

$ sudo btrfs subvolume delete /lxc/vm0
Delete subvolume '/lxc/vm0'
$ sudo btrfs subvolume list /lxc
ID 258 top level 5 path vm0-clean
$ sudo mv vm0-clean vm0
$ sudo btrfs subvolume list /lxc
ID 258 top level 5 path vm0

Notice how the ID of the snapshot doesn't change, even though we've renamed it.

Now, when we start up the VM, we can see that the "apache2" and "postgresql" packages haven't been installed yet, because we've rolled our VM back to the original snapshot that we've taken.

Now, perhaps the scenario with installing the wrong packages isn't that realistic (it would probably be easier to just remove the packages instead of rolling back the snapshots), however, this was just chosen to demonstrate the capabilities of the technology and you can probably imagine a scenario where a snapshot would be more useful. e.g. testing out a software upgrade, which you're concerned might break some functionality.

NOTE: Another way of using snapshots is to mount them directly, by passing the "-o subvol=..." option at mount time, as described at: http://btrfs.ipv5.de/index.php?title=SysadminGuide#Managing_snapshots

LXC on Ubuntu

This post talks about how to setup LXC (Linux Containers) on Ubuntu 12.04. LXC is an operating system-level virtualization technology, which allow you to run multiple virtual machines on one host.

There are quite a few limitations to this type of virtualization, when compared with the type of full, emulator style virtualization that VMware, VirtualBox etc... use. One of the main ones is that you'll be unable to run different operating systems on the virtualization host. i.e. we can only run Linux VM's on our host.

The big advantage is performance. Because the host doesn't have to bother with all of the code which does virtual hardware emulation, the virtual machines run a lot faster in general.

So, to setup LXC, we first need to install it:

sudo apt-get install lxc lxctl uuid btrfs-tools

The 'lxctl', 'uuid' and 'btrfs-tools' packages aren't really needed, but come recommended, and it doesn't hurt to install them.

Now, at this point I did a reboot, which may not be necessary, and afterwards checked the LXC configuration using:

lxc-checkconfig

You should find that all of the different settings are set to "enabled".

Now that we've got LXC installed, we can go ahead and start creating our first VM. Luckily, the 'lxc' package comes with a set of template scripts, which make setting up a VM easy. These scripts are located under '/usr/lib/lxc/templates' and to create our first VM we run:

sudo /usr/lib/lxc/templates/lxc-ubuntu -p /lxc/vm0/
 
Where '/lxc/vm0' is the path to the VM. Note that you will have to create this directory as it doesn't exist by default.

Once you run that script, you'll see a lot of output from the VM getting created and initialized. From the output and from looking at the template script, it looks like the template takes all of the currently installed packages, copies them over to the VM filesystem and configures them.

Once, this is done, we need to configure the networking, which I found to be the trickiest part of the setup. Because the guest VM is using the same hardware as the host, it has the ability to use the network interface attached to the host. Now, obviously you don't want both the host and the guest using the same interface, as it will lead to IP address and MAC address conflicts. So, the guest should have a distinct MAC and/or IP address.

There are several ways to configure the networking for the guest VM's and there are example configuration files of the many ways that they can be configured under file:///usr/share/doc/lxc/examples/ (note that you can enter this location into your web browser and it should load). For our purposes, we're going to go with the lxc-veth.conf file, which sets up a virtual network interface connected to a network bridge which we have to create.

So, firstly, we need to create a network bridge, which is done by adding the following lines to the /etc/network/interfaces file:

auto br0
iface br0 inet dhcp
bridge_ports eth0

This will create the bridge that we're going to connect the virtual network interface of the VM to. In order to enable it, restart the 'networking' service:

sudo service networking restart

Note that we have to create a bridge, even if we've only got one physical interface to connect to it.

Then add the relevant lines from the example config to the VM configuration file, under /lxc/vm0/config:

...
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = 4a:49:43:49:79:bf
lxc.network.ipv4 = 10.0.0.0/24
...

Note that the IP network should match that of your host interface, otherwise you might have some problems getting an IP through DHCP. Also, note that the MAC address is just taken from the example file an was probably randomly generated.

That should complete the configuration. We are now free to start up the VM using the 'lxc-start' command:

sudo lxc-start -n vm0 -f /lxc/vm0/config

This should start up the VM and bring up the console on the terminal screen. For the default Ubuntu template, you can log in using ubuntu/ubuntu username/password pair.

Once you've logged in, you can confirm that the VM has a different IP address to that of the host and start configuring it. 

Funnily enough, just as I've finished writing up this post, I stumbled upon a Launchpad blog post talking about how they're using LXC to speed up their testing: http://blog.launchpad.net/general/parallelising-the-unparallelisable

Programming and premature optimization

I recently watched a lecture by Jonathan Blow, which can be found here. In it he talks about the kinds of challenges that you have to go through being an independent game developer. The main point to take away is that you really have to take a holistic view of the whole "business" and not just the "beauty of computer science" bits which programmers tend to focus on. Once of the main points he covers is to avoid at all costs the sin of premature optimization. The argument goes that if you spend too much time looking for the "perfect" way of doing something, you're likely going to over optimize it and have both wasted your own time on a problem, where you won't get the return on effort that you needed and ended up with code which can't be easily reused.

One of the ways that he gets this message across is to take a look at one of his games, Braid, which it turns out is approximately 90,000 lines of code. He makes the point that the industry average for "lines of code" per programmer, per year is 3,250. At this rate, it would take some ridiculous amount of time, like 28 years to produce a game. So, in order to launch a game by yourself you have to be "super productive" and spending time on anything which won't lead to the game launching (like implementing complex algorithms, which only optimize the game by 0.5%) will without a doubt result in failure.

It's a compelling point and one that can be seen at work in the real world. It's often not the "best" code that ends up being successful, but rather code that ships. The trade-off I guess is that this "ship first" mentality means that a lot of the time, the user ends up with poor quality code, buggy and resistant to upgrades.

Converting param values in Grails

There are some cases in Grails where you want to compare values as Integers instead of the default String objects. There are two ways to do this. The first is to use either the "parseInt" or "valueOf" methods which are original Java methods. The other way to skin this cat is to call the "int()" method on the params object:

if(Integer.parseInt(params.user.id) != user.id){
...
}
if(params.int('user.id') != user.id){
...
}

There are other methods which the params object has to easily convert HTTP post values to well known data types, such as:

param.short(...)
param.byte(...)
param.long(...)
param.double(...)
param.boolean(...)

Another nice feature is that these methods accept a second optional parameter, which is the default that the value is set to, in the case that there is an error in the conversion:

def price = prams.float("seventy", 0.0)

This behaviour is documented in the Grails documentation here. Although, there doesn't seem to be a comprehensive list of the methods available at the time of writing.

STS/Eclipse version control plugins

STS/Eclipse has a number of plugins which make it easy to work with your version control system right in your IDE. Which one you install largely depends on which version control system you're working with.

If you're working with Subversion, you'll want to use the Subversive plugin. There is also a Subclipse plugin which apparently does the same thing, but I haven't had time to try it out as I've been quite happy with Subversive and if it's not broken...

If you're working with Git, you'll want to use the EGit plugin. I've used this plugin to hook up my projects with GitHub and haven't had any major dramas with it so far.

Grails - Setting failOnError globally

One of the small annoyances with Grails that I've found is that the application doesn't fail when a call to the "save" method fails. One of the ways to fix this is to pass the "failOnError" parameter to the save method, set to true:

def book = new Book(title: "The Shining").save(failOnError: true)

However, this gets annoying, having to pass the parameter every time that you call the "save" method. A solution is to declare it as the default setting and forget about it.

This can be done in Config.groovy, by adding the following line:

grails.gorm.failOnError=true

You can also add this configuration to specific packages, in case that you didn't want the configuration to apply to all of the packages used in your application:

grails.gorm.failOnError = ['com.companyname.somepackage','com.companyname.someotherpackage']

From: http://grails.org/doc/latest/guide/conf.html#configGORM

Override the toString method in Grails

The Grails framework is pretty smart in that if you have objects that are related to another object, it will allow you to associate an object at creation time. For example, if you had an User class, which had a "hasMany" relationship with the Post class, when creating a new Post object you would see a drop down allowing you to select a User:

Note that this is assuming that you've used the "generate-all" command to create the default scaffolding. The screen listing the Posts also shows the User:

However, as we can see from the above screenshot the values in the drop down and the User field don't seem to make much sense. The reason for the values are that by default the scaffolding generator calls the "toString" method on the object. Which is set to return the class name of the object, plus it's unique id. While this makes sense to have as the default, we need to change it in order to make it easier for people to read.

Fortunately, changing it is quite straightforward, with us only needing to define (override) the toString method. Simply add the following to the User domain class:

String toString(){
  return username
}

After putting this into the User controller, restart the grails application and you should see the usernames of the users in the drop down when creating a new Post:

This works for any class that you can think of. Simply override the toString method to make it more presentable to the people using the site. I do this almost automatically for all of my domain classes.