After messing with Docker on my laptop, I thought it would be interesting to setup a VM on my FreeBSD server to run RancherOS. I've been using vm-bhyve to manage the VMs, and have been running Ubuntu without much problem, so I figured another Linux distro would be fine ... but ended up opening a whole can of worms and while I did get it running eventually, I learned more about grub and booting on bhyve than I really wanted. I thought I'd jot down some notes here for future reference.
To start with, bhyve is not a general hypervisor that can boot any PC-compatible disk or CD image you throw at it, the way something like KVM, VMWare, or Parallels can. It doesn't start a VM in 16-bit mode and go through a old-school BIOS boot sequence where it reads a Master Boot Record and executes whatever's there. It knows how to load a FreeBSD kernel, and with grub2-bhyve it can boot disks and CDs that use Grub2 - such as Ubuntu.
Unfortunately, RancherOS doesn't use grub, instead it uses Syslinux/ISOLinux on their ISO images and harddisk installations. When bhyve boots using the grub loader, it doesn't find any grub menu on the disk, and just drops you into a grub command prompt.
GNU GRUB version 2.00 Minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions. Anywhere else TAB lists possible device or file completions. grub>
Fortunately, the grub commandline is like a mini-os, with lots of abilities to look around the disks, and it turns out manually boot things like RancherOS.
The first command to run is:
so that future commands don't just scroll off the screen.
displays a list of commands,
help <command> gives a short explanation.
ls lets you start poking around, in this case giving:
(cd0) (cd0,msdos1) (host)
Now we're getting somewhere. Trying
ls (cd0) gives
Device cd0: Filesystem type iso9660 - Label `RancherOS' - Last modification time 2018-09-19 03:09:12 Wednesday, UUID 2018-09-19-03-09-12-00 - Total size 176128 sectors
ls -l (cd0)/ gives
DIR 20180919030909 boot/ DIR 20180919030912 rancheros/
OK, a boot directory, getting closer.
ls -l (cd0)/boot gives
170 20180919030909 global.cfg 66978212 20180919030909 initrd-v1.4.1 DIR 20180919030909 isolinux/ 1373 20180919030909 linux-current.cfg 12734 20180919030909 rancher.png 5523216 20180919030909 vmlinuz-4.14.67-rancher2
There we go, isolinux, but no grub files, no wonder it doesn't boot. After lots and lots of messing around learning grub, I was able to get an initial boot of the CD image from the grub> prompt with:
linux (cd0)/boot/vmlinuz-4.14.67-rancher2 initrd (cd0)/boot/initrd-v1.4.1 boot
And it started! After lots of Linux boot output I was rewarded with:
, , ______ _ _____ _____TM ,------------|'------'| | ___ \ | | / _ / ___| / . '-' |- | |_/ /__ _ _ __ ___| |__ ___ _ __ | | | \ '--. \/| | | | // _' | '_ \ / __| '_ \ / _ \ '__' | | | |'--. \ | .________.'----' | |\ \ (_| | | | | (__| | | | __/ | | \_/ /\__/ / | | | | \_| \_\__,_|_| |_|\___|_| |_|\___|_| \___/\____/ \___/ \___/ Linux 4.14.67-rancher2 RancherOS #1 SMP Thu Sep 13 15:37:04 UTC 2018 rancher ttyS0 docker-sys: 172.18.42.1 eth0: 10.66.0.48 lo: 127.0.0.1 rancher login:
Very cool, but what's the login? Userid is
rancher, but there is no default password. According to the rancher docs, the ISO image is supposed to auto-login. Now what?
After rebooting and getting back to the grub> prompt, and digging around more, I found that
cat (cd0)/boot/global.cfg showed:
APPEND rancher.autologin=tty1 rancher.autologin=ttyS0 rancher.autologin=ttyS1 rancher.autologin=ttyS1 console=tty1 console=ttyS0 console=ttyS1 printk.devkmsg=on panic=10
Ah, LInux command parameters including autologin stuff. To apply them it ended up being (again at the grub> prompt):
linux (cd0)/boot/vmlinuz-4.14.67-rancher2 rancher.autologin=tty1 rancher.autologin=ttyS0 rancher.autologin=ttyS1 rancher.autologin=ttyS1 console=tty1 console=ttyS0 console=ttyS1 printk.devkmsg=on panic=10 initrd (cd0)/boot/initrd-v1.4.1 boot
(that commandline could probably be simplified since we can see from the banner that our VM console is ttyS0, so we probably don't need the params relating to tty1 or ttyS1) This time I got the cattle banner from above, and a beautiful:
Autologin default [rancher@rancher ~]$
sudo -s (not requiring a password) gives root access. At that point you can do whatever, including installing onto a harddisk.
To get a RancherOS harddisk installation to boot, you'd have to go through similar steps with grub in exploring around the (hd0,1) disk to find the kernel, initrd, and kernel params. The grub commands for booting can be saved permanently in the vm-bhyve config for this machine with
grub_runX lines like:
grub_run0="linux (hd0,1)/boot/vmlinuz-4.14.67-rancher2 rancher.autologin=ttyS0 printk.devkmsg=on rancher.state.dev=LABEL=RANCHER_STATE rancher.state.wait panic=10 console=tty0" grub_run1="initrd (hd0,1)/boot/initrd-v1.4.1" grub_run2="boot"
So the full vm-bhyve config file looks like (in case you're wondering - I hate it when people give snippets of code but don't show where it should go exactly):
loader="grub" grub_run0="linux (hd0,1)/boot/vmlinuz-4.14.67-rancher2 rancher.autologin=ttyS0 printk.devkmsg=on rancher.state.dev=LABEL=RANCHER_STATE rancher.state.wait panic=10 console=tty0" grub_run1="initrd (hd0,1)/boot/initrd-v1.4.1" grub_run2="boot" cpu=2 memory=2048M network0_type="virtio-net" network0_switch="public" disk0_type="virtio-blk" disk0_name="disk0" disk0_dev="sparse-zvol" uuid="7f9fc9e5-c835-11e8-a327-e03f49af0c7d" network0_mac="58:9c:fc:05:2a:04"
With that, my VM now boots without manual intervention, even though the virtual disk doesn't use grub.