Setting up docker with xhyve

An alternative You can skip all this and check out dlite, which integrates xhyve with docker without going through the hassle of dealing with docker-machine and fiddling with environment variable shenanigans. I've since switched to that over docker-machine-driver-xhyve.


One of the pain points I have had with using docker on OS X is that performance can take a hit due to needing a Type 2 hypervisor (which runs more like a regular program that doesn't have direct access to hardware) to work (e.g. virtualbox, VMWare).

Here's where xhyve comes in. xhyve takes advantage of the Hypervisor.framework that was introduced in OS X 10.10, which brings Type 1 hypervisor technology (direct, host-level access to hardware) to OS X. @mist64, the main author of xhyve has an excellent introduction written up on it.

As this is all very new technology, I did not find many guides on how to use xhyve with my docker workflow, so I've written up my notes here.

docker with xhyve

A solution I've been using to get that takes advantage of @zchee's docker-machine-driver-xhyve.

I've used to solution to replace my previous, virtualbox-driven docker workflow.

Before you begin

If you have a previous installation of the docker toolset, whether through Docker Toolbox or homebrew, you'll need to uninstall them all.

You can use docker's own uninstall script:

sh -c "$(curl -fsSl https://raw.githubusercontent.com/docker/toolbox/master/osx/uninstall.sh)"

If you installed docker with homebrew, uninstall docker, docker-compose, docker-machine:

brew uninstall docker docker-compose docker-machine

The tools

All of these can be installed without homebrew, but I'm too lazy to go over that.

The tools:

  • docker (homebrew) v1.9.x
  • docker-compose (homebrew) v1.5.x
  • docker-machine (from source or binary)
  • docker-machine-driver-xhyve (from source or binary)
  • xhyve (homebrew)

Installation (Automated)

BEWARE Dragons here.

I've attempted to automate this process with a script, but YMMV.

Installation (binaries)

UPDATED 06/01/2016 - @nathanleclaire pointed out that there are binaries of docker-machine-driver-xhyve here. Not sure how I missed that before, but I've updated this to account for the pre-compiled binaries. :)

Install the docker toolset:

brew update
brew install docker docker-compose docker-machine

You'll then need to change permissions for the docker-machine-driver-xhyve binary:

sudo chown root:wheel /usr/local/bin/docker-machine-driver-xhyve
sudo chmod u+s /usr/local/bin/docker-machine-driver-xhyve

See the releases page for installation without home-brew.

Manual Installation (from source)

Installing dependencies

docker-machine-driver-xhyve and docker-machine both require go, so you will need to install go alongside docker, docker-compose, and xhyve. We will be building docker-machine from source.

brew update
brew install xhyve docker go
brew install docker-compose --without-docker ---without-docker-machine

Then, set up the $GOPATH which will be necessary to proceed:

mkdir ~/.go
export GOPATH=~/.go

Installing docker-machine

Retrieve the docker-machine source and then build and install:

go get github.com/docker/machine
cd $GOPATH/src/github.com/docker/machine
make build
make install # this will copy the bin to /usr/local/bin

Installing docker-machine-driver-xhyve

Retrieve the docker-machine-driver-xhyve source and then build and install that:

export GO15VENDOREXPERIMENT=1
go get -u github.com/zchee/docker-machine-driver-xhyve
cd $GOPATH/src/github.com/zchee/docker-machine-driver-xhyve
make build
make install
sudo chown root:wheel $GOPATH/bin/docker-machine-driver-xhyve
sudo chmod u+s $GOPATH/bin/docker-machine-driver-xhyve

Creating the xhyve docker-machine

NOTE I had to temporarily turn off the OS X Firewall in order for this to work and to disconnect from any active VPNs.


Assuming things went smoothly (let's hope), we can now proceed to creating the xhyve docker-machine. Hooray!

docker-machine create dev --driver xhyve --xhyve-experimental-nfs-share

This will create a docker-machine running on xhyve, and then set up a tinycorelinux-based boot2docker VM.

docker-machine

I'd recommend having the --xhyve-experimental-nfs-share argument passed if you want to mount shared volumes. This will mount /Users as an NFS folder in the tinycorelinux environment that docker on OS X needs.

If you have an existing /etc/exports (used to configure NFS on OS X), temporarily move it to /etc/exports as the NFS setup by this docker-machine driver will not work otherwise.

sudo mv /etc/exports{,.backup}

NOTE: if you are using a VPN, turn it off before using this step.

Testing

Try running the hello world docker example:

hello world

Hacking away

I've successfully used this on projects that use multiple containers without having to change any configuration in docker-compose.yml.