Minimal OpenBSD Virtual Machine

OpenBSD has its own hypervisor (vmm) for few years. Its primary goal is to run OpenBSD guests. Recently I played around vmm to create a minimal virtual machine without a virtual disk! It is useful when you need to create lots of similar virtual machines without worrying about accidental data corruption by them.

Virtual machine managers need a start point to start. The most obvious way is a firmware which detects and loads boot loader and other option is the boot loader itself. However some virtual machine managers (like OpenBSD vmm) are capable of loading a kernel directly. Loading a kernel directly is much faster than the other two options, while virtual machines are unable to modify kernels; so it’s the best option to run many virtual machines.

By default OpenBSD kernel need a disk to mount on /, but you can change this behavior to few other options:

1. CD: using CD as root is a bad idea because it is read-only while kernel needs to write something (more accurately because I couldn’t do that)
2. network: OpenBSD supports NFS on root, but you need a network accessible server with distinct disk spaces for virtual machines
3. ramdisk: ramdisk is the best alternative to real disk for /; however it is limited (in my experience up to about 100 MB)

Therefore it is best to keep installation size under 100 MB; however you might use CD, or NFS for /usr (because /usr is the only read-only-able option). Usual basic installation of OpenBSD needs at least 700 MB disk space; so you need to remove unneeded files manually. The most aggressive way is that you install the base and required packages, set every file access time to an old date (say Jan 1, 1970), reboot, execute all binaries you might need, and finally remove every file that aren’t accessed recently.

# set access time to an old date
find / -exec touch -a -t 197001010000 '{}' '+'
reboot
...
# do whatever you might need to do
# (login, ping, traceroute, ...)
...
# remove files that aren't accessed recently
find / -atime 1 -prune -exec rm -f '{}' '+'
# remove empty directories
find / -type d -exec rmdir '{}' '+'

To reduce installation size down to 100 MB, you need to disable unneeded rc daemons and disable all re-orderings in /etc/rc (because they need /usr/bin/clang which is too big). It is almost safe, because all libraries has been re-ordered at least one time at the first boot after installation. Also keep in mind that you don’t need /boot and /bsd* files, because vmm will load kernel directly.

After removing all unneeded files, it’s time to build your own kernel:

# extract sys.tar.gz into /usr/src
...
# change kernel config
cd /usr/src/sys/arch/$(machine)/compile/GENERIC
sed -i -Ee'/(^conf|vmm|drm|gpu|agp)/d' ../../conf/GENERIC
cat <<-END >>../../conf/GENERIC
option          MINIROOTSIZE=204800
config          bsd root on rd0a
pseudo-device   rd 1
END
# now build kernel
make obj
make config
make -j4

Now you need to insert your installation files in the kernel:

# extract ramdisk
mv /usr/obj/sys/arch/$(machine)/compile/GENERIC/bsd mybsd
rdsetroot -x mybsd rd.img
# disklabel
vnconfig vnd0 rd.img
dd if=/dev/zero of=/dev/rvnd0c count=8
print 'X\na a\n0\n*\n\n512\n\nn\n1\nw\nq\n' | disklabel -E vnd0
newfs -i 16384 -m 0 vnd0a
mount /dev/vnd0a /mnt
# now copy files
...
# now change fstab
cat <<-END >/mnt/etc/fstab
/dev/rd0a /     ffs     rw
swap    /tmp    mfs     rw,nodev,nosuid,-s=64M
END
# insert ramdisk
umount /mnt
vnconfig -u vnd0
rdsetroot mybsd rd.img

Now you can run many virtual machines without any more configuration:

vmctl start -L -b mybsd vm1
vmctl start -L -b mybsd vm2
vmctl start -L -b mybsd vm3
vmctl start -L -b mybsd vm4