Installing FreeBSD on OVH

Tim Chase — Sun, 06 Nov 2016 19:48:15 GMT

While OVH has a number of Linux-based options for their low-end VPS offerings, I wanted to try installing FreeBSD. As far as I can tell, OVH doesn't offer the ability to provide a bootable .iso or .img installer image for their VPS offerings (unlike their dedicated server instances). Fortunately, their VPS offers a recovery console with SSH access along with the use of dd, gzip/gunzip, and xz. This grants SSH access to the recovery console, allowing me to write the disk image directly to the virtual drive which I learned from murf.

The following was all done within a fairly stock FreeBSD 11 install (ZFS-on-root).

My first problem was that the official downloadable .raw hard-drive image was something like 20.1GB which was just slightly larger than my available 20.0GB of space so I couldn't directly write the image provided on the FreeBSD site.

Time to do it the hard way and build my own image.

Preliminary reconnaissance

First, we need to gather some network information from the OVH install. I'm assuming your initial machine image was either an Ubuntu or Debian install. If not, modify accordingly, but the goal is to obtain the following from your management console and your /etc/networks.d/* or /etc/network/* file:

your static IPv4 address (should be available in your management console or as the address ${EXTERNAL_IPv4} line in your interface file, or by issuing ipconfig eth0 and looking for the IPv4 address)
your static IPv6 address (should be available in your management console or from ipconfig eth0 referenced later as ${EXTERNAL_IPv6} )
your default gateway (should be available in your interface file from the line that reads post-up /sbin/ip route add ${GATEWAY_IPv4} dev eth0 or from the output of route | grep default )
your DNS server (should be available in your interface file from the line that reads dns-nameserver ${DNS_NAMESERVER} or nameserver ${DNS_NAMESERVER} )
your DNS search suffix (should be available in your interface file from the line that reads dns-search ${DNS_SEARCH} however this may be optional)

With these items noted, we can use them later on when creating the corresponding configurations in FreeBSD.

Create a local image file

First, create a drive-image file that is the right size:

user@localhost$ truncate -s 20G $HOME/freebsd.img

This will create a 20GB image file that should fit exactly in the available space on the OVH VPS. If you're using the lowest-end VPS, change that to 10G to make a 10GB drive instead.

Create a device for the file

In order to install to this file as a disk, FreeBSD needs to recognize it. This can be done with the mdconfig command as root. This needs to be done as root so first su - to root:

user@localhost$ su -
Password: ********

On most systems, there won't already be a md0 device, but it's good to check first:

mdconfig -l

On most systems, that will return no pre-existing devices so you can use 0 as the device-number, but if other md devices exist, add 1 to the highest device number returned.

Create the (in this case md0 ) disk-backed memory device:

mdconfig -f ~user/freebsd.img -u 0

This will create a md0 device to which FreeBSD can be installed. Note that if you already have a md0 device, change the -u 0 to a higher number that doesn't already exist

Install FreeBSD

Before installing, it's good to specify the root-pool as something other than the default pools you likely have:

export ZFSBOOT_POOL_NAME=ovhzpool
bsdinstall

With the md0 device created we can run bsdinstall and choose md0 as our target drive.

For the Keymap Selection, continue with the default keymap unless you have reason not to.

For the hostname, this post uses "ovh".

For your mirror selection, choose something geographically close.

If you're using UFS on your local machine or installing UFS on your OVH server, feel free to investigate whether the other automated installers work for you. However, if your current/host setup uses ZFS like this guide, choosing "Guided Root-on-ZFS" or the "Manual Disk Setup (experts)" in bsdinstall will find your existing zpools & GELI devices and try to forcibly detach them , killing your local system in the process, requiring a reboot. So for ZFS-on-existing-ZFS, use the "Shell" option. Also, make note of the warning/instructions:

Use this shell to set up partitions for the new system. When finished, mount the system at /mnt and place an fstab file for the new system at /tmp/bsdinstall_etc/fstab . Then type 'exit'. You can also enter the partition editor at any time by entering bsdinstall partedit.

confirm that our ZFSBOOT_POOL_NAME is set correctly

export ZFSBOOT_POOL_NAME=ovhzpool

Clear any existing traces of a partition table. If it doesn't have an existing partition table, it may complain, but since the goal is to nuke it if it does exist, any such error can be safely ignored.

gpart destroy -F md0

Create a GPT partition table

gpart create -s gpt md0

Create a 512k boot partition

gpart add -t freebsd-boot -s 512k -l boot md0

Install the boot-loader code on the first (only so far) partition

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 md0

Create a ZFS data partition, filling the rest of the drive

gpart add -t freebsd-zfs -l ${ZFSBOOT_POOL_NAME} -a 1M md0

Create an encrypted disk (optionally add -T to skip passing TRIM commands to the SSD as this can leak information about how much space is actually in use)

geli init -e AES-XTS -l 128 -s 4096 -b -g gpt/${ZFSBOOT_POOL_NAME}

At this point, it will prompt you to enter and confirm your GELI password. Next, attach this GELI device:

geli attach gpt/${ZFSBOOT_POOL_NAME}

Now, create a pool on that encrypted device. With a single-but-larger disk, I'd set copies=2 on my root pool to get a little redundancy by default on everything. But since 20GB is a little tight, I set it selectively on my datasets for user data and assume I can regenerate anything in system datasets. Your own needs will determine whether you make everything redundant, just your user data, or nothing.

zpool create -R /mnt -O canmount=off -O mountpoint=/ -O atime=off -O compression=on ${ZFSBOOT_POOL_NAME} gpt/${ZFSBOOT_POOL_NAME}.eli
zfs create -o mountpoint=/ ${ZFSBOOT_POOL_NAME}/ROOT
zpool set bootfs=${ZFSBOOT_POOL_NAME}/ROOT ${ZFSBOOT_POOL_NAME}
zfs create -o copies=2 ${ZFSBOOT_POOL_NAME}/home
zfs create -o canmount=off ${ZFSBOOT_POOL_NAME}/usr
zfs create ${ZFSBOOT_POOL_NAME}/usr/local
zfs create ${ZFSBOOT_POOL_NAME}/usr/local/jails
zfs create ${ZFSBOOT_POOL_NAME}/usr/obj
zfs create ${ZFSBOOT_POOL_NAME}/usr/src
zfs create ${ZFSBOOT_POOL_NAME}/usr/ports
zfs create ${ZFSBOOT_POOL_NAME}/usr/ports/distfiles
zfs create -o canmount=off ${ZFSBOOT_POOL_NAME}/var
zfs create ${ZFSBOOT_POOL_NAME}/var/log
zfs create ${ZFSBOOT_POOL_NAME}/var/tmp
zfs create ${ZFSBOOT_POOL_NAME}/tmp

Because of a quirk in the installer, it requires /usr/freebsd-dist/MANIFEST to be present in the chroot environment. It also seems to clean this up if the installer hiccups for any reason, so I recommend putting a copy into your local directory. It may already be on your install media, or you can download a copy from the corresponding directory on the FTP site. Check your mirror, architecture, and release, but it should be something like

mkdir freebsd-dist
cd freebsd-dist
fetch ftp://ftp8.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.0-RELEASE/MANIFEST
cd ..

and then copying it into /usr/

cp -va freebsd-dist /usr/

That way, you'll have a local copy in the event something goes awry and you have to recreate the /usr/freebsd-dist/MANIFEST again.

Finally, exit the shell to return to the installer.

exit

The rest of the installer should be fairly straight-forward. At the end of the install, it will prompt to drop to a shell. Do that.

First, create/edit /etc/rc.conf to include the following lines, making use of the information gleaned in the reconnaissance step above.

ifconfig_vtnet0="inet $EXTERNAL_IPv4 netmask 255.255.255.255 broadcast $EXTERNAL_IPv4"
static_routes="net1 net2"
route_net1="$GATEWAY_IPv4 -interface vtnet0"
route_net2="default $GATEWAY_IPv4 "
ifconfig_vtnet0_ipv6="inet6 $EXTERNAL_IPv6 prefixlen 64"
ipv6_defaultrouter="$GATEWAY_IPv6"

If you plan to use jails, don't let syslogd listen on all addresses and it's worthwhile to set up a loopback interface to use while you're in the /etc/rc.conf

syslogd_flags="-ss"
cloned_interfaces="lo1"

Next up, edit /etc/resolv.conf using the DNS information gathered during reconnaissance to contain the nameserver and optionally the search

nameserver $DNS_NAMESERVER
search $DNS_SEARCH

Assuming you set up a user during the install and that they're a member of the wheel group to let you su - and perform administrative commands, it's worth editing the /etc/ssh/sshd_config to allow SSH login and prohibit root login by adding/modifying these lines:

ListenAddress $EXTERNAL_IP
PermitRootLogin no

You may also want to add your SSH public key to $USER/.ssh/authorized_keys for your non-root $USER so you can SSH in without a password.

With changes done in the chroot we can now exit the chroot and return to the parent shell.

exit

Freeing up the image

Before shipping the image over to the server, it first needs to be shut down cleanly. Start by unmounting all the new ZFS items under. The following finds them and creates a series of commands that can be used to unmount them:

mount | awk '/\/mnt/{print "umount " $3}' | sort -r

If they all look good, you can execute them.

mount | awk '/\/mnt/{print "umount " $3}' | sort -r | sh

ZFS still has these pools active so disconnect them

zpool export $ZFSBOOT_POOL_NAME

Detach the GELI devices if needed.

geli detach /dev/gpt/${ZFSBOOT_POOL_NAME}.eli

With all filesystems disassociated from the md0 it can now be disconnected too.

mdconfig -d -u 0

Can now leave the root shell and return to your unprivileged user

exit

Reconnecting

In the event you want to make further modifications, it's helpful to know how to reattach the md0, attach the GELI volume, and import the ZFS pool again. So as root

mdconfig -f ~user/freebsd.img -u 0
geli attach gpt/$ZFSBOOT_POOL_NAME
zpool import -R /mnt $ZFSBOOT_POOL_NAME

From here, you can make changes and then repeat the steps for disconnecting the disk image.

Sending the image over

Start by compressing the disk image to save bandwidth. My 20GB image compressed to ~400MB, or roughly 2%. I keep the original image around in case I need to reconnect to it to make changes I forgot about. The gzip process takes a little while.

gzip --keep freebsd.img

Log into your VPS console and restart your VPS in rescue mode.

Once it has restarted, you should receive information via email on how to log into that console. Use this information to SSH into the rescue image.

ssh root@$EXTERNAL_IPv4



Your normal root drive will be mounted under the rescue 
/mnt/
First you have to find out its name.  It might be
vdb1,
sdb1,
or something similar so it will need to be unmounted before overwriting.
Once done, disconnect.



mount | grep /mnt
umount /mnt/vdb1
exit

Next, upload the image to the drive in question (make sure you specify the correct root device based on the mount point, such as vdb or sdb ).

ssh root@$EXTERNAL_IPv4 "gunzip | dd of=/dev/sdb bs=1M" < freebsd.img.gz

or, if you have pv installed, you can get a rough estimate of the time needed

pv freebsd.img.gz | ssh root@$EXTERNAL_IPv4 "gunzip | dd of=/dev/sdb bs=1M"

Depending on your internet speed, this may take a while. However, once the image has finished transferring, you should be able to go into your management console and reboot the server. If all has gone well, your server should start booting.

Note that, since we have an encrypted drive, we need to log into the KVM console to enter the boot password. It should then boot in that KVM console to the login prompt. If you can get in there, then you can attempt to SSH in directly.

ssh $USER@$EXTERNAL_IPv4



If you installed your public key in
$USER/.ssh/authorized_keys
it should let you right in.
If you didn't, you can enter your credentials and should hopefully be up
and running.

Tim's blog (Posts about OVH)