Bhyve

[x] BHYVE_FIRMWARE  Required to run UEFI guests
[x] EXAMPLES        Install example guest templates
[x] GRUB2_BHYVE     Required to run any guests that need a Grub bootloader
[x] TMUX            Tmux console access instead of cu/nmdm

Configuration

vm-bhyve

Fist we need to create a directory where vm-bhyve will store it’s configuration as well as the various VMs created:

zfs create -o mountpoint=/var/bhyvevm-bhyve directory main/bhyvevm-bhyve dataset  # Create vm-bhyve directory
zfs set compress=zstd main/bhyvevm-bhyve dataset                # Enable compression if desired

Enabling vm-bhyve and specifying it’s directory is done in the system configuration file rc.conf:

vm_enable="YES"
vm_dir="zfs:main/bhyvevm-bhyve dataset"

Allows vm-bhyve to initialize it’s infrastructure:

vm init

Various templates are available in /usr/local/share/examples/vm-bhyve/, they should be copied (and adapted) in the .templates directory inside the defined in vm_dir (in case of a zfs filesystem, the mountpoint is used).

cp /usr/local/share/examples/vm-bhyve/* /var/bhyvevm-bhyve directory/.templates/

Networking

The bridge interface on which bhyve will attach is manually configured, instead of letting vm-bhyve manage it, so it’s easier to integrate on other part of the system.

Create the vmnet0 interface and assign it an IP address, the interface is also given a group name jail to ease management in complex firewall rules:

cloned_interfaces="bridge1"
ifconfig_bridge1_name="vmnet0Dedicated network interface"
ifconfig_vmnet0Dedicated network interface="inet 192.168.100.1/24 group jail"

If we want to give the VM access to the outside world we need to have the host act as a gateway, and also as here the VMs have been defined on a private network NAT must be enabled:

nat pass on wan0External interface inet from vmnet0Dedicated network interface:network to any -> wan0External interface

gateway_enable="yes"
pf_enable="yes"

It is possible to use, for a more complex configuration, a full featured DHCP, TFTP and DNS server to provide the necessary network services for booting and network access, but we have chosen here to use dnsmasq which conveniently provide all these services with a simple configuration:

# Select listening interface
interface=vmnet0Dedicated network interface        # Select the interface on which to listen
except-interface=lo0    # Ensure we don't listen on loopback 
bind-interfaces         # Bind to interfaces instead of listening on wildcard

# DNS (with DNSSEC enabled)
domain-needed
conf-file=/usr/local/share/dnsmasq/trust-anchors.conf
dnssec
dnssec-check-unsigned

# DHCP
dhcp-authoritative
dhcp-range=192.168.100.10,192.168.100.150Range of dynamically allocated addresses,24hLease time
# Using statically allocated addresses for some guests
#  infinite = maximum lease time
dhcp-host=debianHostname,192.168.100.55Assigned address                      # from provided hostname
dhcp-host=58:9c:fc:0d:84:bdMAC address,192.168.100.52Assigned address,infiniteLease time  # from MAC address

We now let vm-bhyve know that we have created an interface where it will be able to connect the VMs:

# List of available switch
switch_list="bhyve-bridge"

# Switch definition
type_bhyve-bridge="manual"
bridge_bhyve-bridge="vmnet0Dedicated network interface"

options {
        listen-on       { 192.168.1.1; 192.168.100.1List of IP addresses on which to bind };
        listen-on-v6    { none; };
}

VM

Templates

As we are using zfs, the various templates will be adapted to use a sparse zvol as disk, and it will be named root

disk0_name="root"
disk0_dev="sparse-zvol"

Console

There is a virtio console, which provide a socket where to connect (with socat for example), but:

• there’s no way to use console port feature (➞ no /dev/hvc? in linux)
• emergency write is advertised, but is a no-op

For a true console, we need to use vm console which connect (defined when configuring the VM) to an emulated serial line:

• from null-modem using cu and nmdm device
• from stdio/stdout using tmnux

Shared directory

• Only available in FreeBSD 13.
• This option is not directly handled by vm command, the configuration file vm.conf need to be edited.
• A free PCI slot (option -s) need to be selected, check vm-bhyve.log to see what was assigned.

bhyve_options="-s 15PCI slot,virtio-9p,fooShare name=/path/to/fooShared directory,barShare name=/path/to/bar"Shared directory

Sound

• Only available in FreeBSD 13.
• This option is not directly handled by vm command, the configuration file vm.conf need to be edited.
• A free PCI slot (option -s) need to be selected, check vm-bhyve.log to see what was assigned.

bhyve_options="-s 16PCI slot,hda,play=/dev/dsp,rec=/dev/dsp"

Entropy driver

To supply high-quality randomness from the hypervisor to the guest:

virt_random="yes"

The guest will also need a specific driver:

• FreeBSD: virtio_random
• Windows: see virtio-win

Guests

Windows

Executing command at startup:

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run]
"F Drive"="subst F: D:\\mount\\db"
"G Drive"="subst G: D:\\mount\\log"

Linux

Console

The virtio console is available through named devices in /dev/virtio-ports/ directory (which hold seem links to numbered device /dev/vport?p?).

usermod -a -G dialout johndoeUser login
usermod -a -G tty johndoeUser login

Shared directory

# See: https://www.kernel.org/doc/html/latest/filesystems/9p.html
mount -t 9p -o trans=virtio,cache=mmap,msize=512000 sharename /path/to/mountpoint

Commands

vm create -t debian debian  # Create VM from template
vm configure debian         # Perform extra configuration

debian_cd_repos_url=https://cdimage.debian.org/debian-cd/current/amd64/iso-cd
vm iso ${debian_cd_repos_url}/debian-10.9.0-amd64-netinst.iso

vm install debian debian-10.9.0-amd64-netinst.iso

vm start debian

vm console debian

vm stop debian        # Ask for gracefull shutdown
vm poweroff debian    # Terminate immeditately

Troubleshouting

All ports busy (vm console)

# vm console debian
all ports busy

There is a cu process already connected to the console, find it and kill it