Faster volume mounts with Docker development

Docker is a great technology, but in a previous post I mentioned the issues boot2docker has on Mac OS X, specifically with volume mounts.

Since a significant proportional of our development team (including myself) develop on Mac OS X, these issues affect us on a daily basis, so I did some digging to find out the causes of this slowdown and what (if anything) can be done to fix it.

To summarise the problem:

• Docker operates using Linux container (LXC) technology and a library called libcontainer that requires specific extensions provided by the Linux kernel. Since Mac OS is based on a non-linux Unix kernel, it does not have the required features, and will not run natively.
• The officially supported way to do this in Mac OS is via Boot2Docker, a very neat command line utility that bundles a lightweight linux VM with some terminal control commands. It works by starting an linux virtual machine in virtualbox, which runs the docker server and your containers.
• The docker command line client still runs on the mac OS terminal, but talks to the virtual Linux server over the network.
• When doing web development in docker, a key feature is volume mounting directories from the Mac OS host machine into a container, as you can start a server and see changes as you modify files in your editor. The only other way to get data into docker is to add it to the container file system at build time, which means running a slow container rebuild command each time you change a file.
• Boot2Docker accomplishes this volume mounting through virtualbox and it's 'shared folders' functionality. Virtualbox shared folders are notoriously slow, especially for writes and don't sync file change notifications from the host system.
• This is awful if you are trying to dockerise a disk-heavy activity such as a grunt-based frontend build. On typical projects we can be waiting for hundreds of seconds to complete a build, compared to single digits on the host machine. Trying to do an npm install or bower install inside a running container can be super slow. All these things work perfectly and fast on Linux host docker environments.
• Anything in your development workflow which requires 'watching' files in a mounted directory (e.g. sass, python runserver, grunt watch) is also slow to respond. A typical response time after saving a sass file on my host machine is 5-10 seconds inside the container, compared to an apparently instantaneous response on the host.

What we can do to fix it

The key option is mounting the host machine disk into the VM via some different system to virtualbox shared folders. Several of these options propose using the same VM and linking to virtualbox using the faster NFS, whereas the last involves reversing the process, and sharing the folder back to the host from inside the VM using samba.

Build a custom boot2docker virtual machine

This idea involves taking a pre-built disk image from the boot2docker project, and using a VM management tool like vagrant to start a VM with this, and configure the required NFS config. Vagrant makes the NFS setup very easy. Following some instructions in a github issue (which I've lost the link for), I built a vagrantfile around this concept. I also wrapped the vagrantfile in a brew package repository to make it easily installable.

To install:

• Uses a homebrew formula from a custom brew tap
• We need to first install the dependencies
  • brew install caskroom/cask/brew-cask
  • brew cask install vagrant
• brew tap grahamgilchrist/custom
• brew install vagrant-boot2docker
• brew install docker (currently needs to be 1.4.1 to match the virtual image)
• You now have the ability to run the vagrant-boot2docker command in any shell to manage the linux image

Usage:

• vagrant-boot2docker up starts the virtual linux image
  • The machine IP will be output as the last line from the up command. This is the IP to use for accessing the container output
  • If the vagrant box won't start with an error about resources already being used then you probably have not shut down an old virtual image. Open virtualbox and close/remove any VMs there with vagrant in the name
  • In each terminal you want to use docker, you need to run export DOCKER_HOST=tcp://192.168.33.10:2375
• vagrant-boot2docker suspend/destroy stops the virtual linux image

This installs a custom vagrantfile and shell script to manage it. The script passes through any arguments to the custom vagrant instance. e.g. vagrant-boot2docker up will start the vagrant image

This proved good for a while, but the system did seem to occasionally suffer from annoying VM crashes during periods of intense disk activity, requiring a restart of vagrant/virtualbox. The biggest pain with this method however is the maintenance. Since I am maintaining the vagrantfile, but not the boot2docker image, it relies on either the third party to keep the image maintained, or for me to build it myself on each docker release. Sadly this proved to be too much maintenance cost, so I gave up with this approach. The vagrantfile is still in the repo though, for anyone who wishes to base this idea on an updated image.

Tweak your virtualbox settings

You don't have to live with the stock boot2docker VM either. It's possible to modify the official VM image to add NFS mounts. Blackfire.io have written an extremely good guide to overcoming boot2docker issues on Mac OS by modifying the VM config, including how to add NFS mounts. I highly suggest you read this guide:

• http://blog.blackfire.io/how-we-use-docker.html

My only issue with adopting this is that updates to boot2docker will wipe out these changes, so it makes updating docker a pain. I'm sure the NFS mount setup could be scripted however to ease this pain, perhaps even as a custom brew package, but again this does require maintenance time with each docker release.

Hodor

• https://github.com/gansbrest/hodor

Hodor, Hodor, Hodor.

I've not actually tried Hodor yet, but it promises to offer a great ecosystem around managing docker development projects. As well as providing faster NFS volume mounts,it also eases some other Mac OS docker headaches such as port mapping and ssh key sharing. These are all very promising features and I'd like to give it a go. The reason I haven't tried it yet, is that I am already very happy with the container orchestration provided by docker-compose, so I don't want to convert that process to another system which is less popular and supported.

Do everything in your VM

The final option, and the one I've settled on for now, is to stop fighting and just run everything inside Linux. So instead of using the boot2docker virtual machine, I create my own lightweight linux VM and run docker (server and client) directly inside that. I ssh to this machine from my mac OS terminal and basically do all my development work inside of this machine instead of using the mac terminal. Projects are cloned directly into the VM filesystem, and shared back out over the network with samba. This means docker container disk operations run at near native speed (for the VM), with the tradeoff that only occasionally saving files is slower over samba. However since saving a file is a manual process which happens infrequently compared to automated disk operations, this is largely not noticeable.

Of course I have to ssh into the VM in each window I want to use it, but in practice I find this no more annoying than having to do boot2docker up or set the boot2docker environment variables in each terminal.

I have a strictly 'docker-only' policy for software on this machine, so the only software and configuration it needs is:

• ssh
• git
• samba
• docker
• docker-compose

Of course, this requires some knowledge of linux, and installing command line tools, which can be a barrier to team members not familiar with this OS. However, if all you do is install git, docker and docker-compose, this environment can be quite easy to setup. At some point I hope to write a vagrantfile similar to the custom boot2docker above to simplify the provision of this VM for myself and other users.

The future

I hope this helps someone out there facing similar issues, and gives you some ideas to solve it. My hope is that eventually the official boot2docker setup will adopt some sort of NFS mount which can be seamlessly installed and updated with documentation. Until that point, we are stuck with either fully working inside a linux VM, or getting you hands dirty with tweaking virtualbox settings.