Upon learning that
OpenBSD
was
discontinuing
sqlite
in the base system,
I joked that the
BCHS
web stack would need to drop
sqlite
and investigate new options for a web-stack available purely in the
OpenBSD base system.
So continuing that joke, I proposed the
HOFFA
web stack:
httpd,
OpenBSD, flat-files, and
awk.
Inspired by a recent post on Reddit where someone created an
amber monochrome theme on OpenBSD running CWM
I decided to undertake a similar venture to do a green-on-black theme
for CWM on OpenBSD.
While
OVH
has a number of Linux-based options for their low-end VPS offerings, I
wanted to try installing
FreeBSD.
As far as I can tell, OVH doesn't offer the ability to provide a bootable
.iso
or
.img
installer image for their VPS offerings (unlike their dedicated server
instances).
Fortunately, their VPS offers a recovery console with SSH access along with
the use of
dd,
gzip/gunzip,
and
xz.
This grants SSH access to the recovery console, allowing me to write the disk
image directly to the virtual drive which I learned from
murf.
The following was all done within a fairly stock FreeBSD 11 install (ZFS-on-root).
My first problem was that the official downloadable
.rawhard-drive image
was something like 20.1GB which was just
slightly
larger than my available 20.0GB of space so I couldn't directly write the
image provided on the FreeBSD site.
Time to do it the hard way and build my own image.
Preliminary reconnaissance
First, we need to gather some network information from the OVH install.
I'm assuming your initial machine image was either an Ubuntu or Debian
install.
If not, modify accordingly, but the goal is to obtain the following
from your management console and your
/etc/networks.d/*
or
/etc/network/*
file:
your static IPv4 address (should be available in your management
console or as the
address ${EXTERNAL_IPv4}
line in your interface file, or by issuing
ipconfig eth0
and looking for the IPv4 address)
your static IPv6 address (should be available in your management
console or from
ipconfig eth0
referenced later as
${EXTERNAL_IPv6}
)
your default gateway (should be available in your interface file
from the line that reads
post-up /sbin/ip route add ${GATEWAY_IPv4} dev eth0
or from the output of
route | grep default
)
your DNS server (should be available in your interface file from
the line that reads
dns-nameserver ${DNS_NAMESERVER}
or
nameserver ${DNS_NAMESERVER}
)
your DNS search suffix (should be available in your interface file from
the line that reads
dns-search ${DNS_SEARCH}
however this may be optional)
With these items noted, we can use them later on when creating the
corresponding configurations in FreeBSD.
Create a local image file
First, create a drive-image file that is the right size:
user@localhost$ truncate -s 20G$HOME/freebsd.img
This will create a 20GB image file that should fit exactly in the available
space on the OVH VPS.
If you're using the lowest-end VPS, change that to 10G to make a 10GB
drive instead.
Create a device for the file
In order to install to this file as a disk, FreeBSD needs to recognize it.
This can be done with the
mdconfig
command as root.
This needs to be done as
root
so first
su -
to root:
user@localhost$ su -
Password: ********
On most systems, there won't already be a
md0
device, but it's good to check first:
mdconfig -l
On most systems, that will return no pre-existing devices so you can use
0
as the device-number, but if other
md
devices exist, add 1 to the highest device number returned.
Create the (in this case
md0
) disk-backed memory device:
mdconfig -f ~user/freebsd.img -u 0
This will create a
md0
device to which FreeBSD can be installed.
Note that if you already have a
md0
device, change the
-u 0
to a higher number that doesn't already exist
Install FreeBSD
Before installing, it's good to specify the root-pool as something other
than the default pools you likely have:
export ZFSBOOT_POOL_NAME=ovhzpool
bsdinstall
With the
md0
device created we can run
bsdinstall
and choose
md0
as our target drive.
For the Keymap Selection, continue with the default keymap unless you have
reason not to.
For the hostname, this post uses "ovh".
For your mirror selection, choose something geographically close.
If you're using UFS on your local machine or installing UFS on your OVH
server, feel free to investigate whether the other automated installers
work for you.
However, if your current/host setup uses ZFS like this guide,
choosing "Guided Root-on-ZFS" or the "Manual Disk Setup (experts)" in
bsdinstall
will find your existing zpools & GELI devices and try to forcibly
detach them
,
killing your local system in the process, requiring a reboot.
So for ZFS-on-existing-ZFS, use the "Shell" option.
Also, make note of the warning/instructions:
Use this shell to set up partitions for the new system.
When finished, mount the system at
/mnt
and place an fstab file for the new system at
/tmp/bsdinstall_etc/fstab
.
Then type 'exit'.
You can also enter the partition editor at any time by entering
bsdinstall partedit.
confirm that our ZFSBOOT_POOL_NAME is set correctly
export ZFSBOOT_POOL_NAME=ovhzpool
Clear any existing traces of a partition table.
If it doesn't have an existing partition table, it may complain, but
since the goal is to nuke it if it does exist, any such error can be
safely ignored.
gpart destroy -F md0
Create a GPT partition table
gpart create -s gpt md0
Create a 512k boot partition
gpart add -t freebsd-boot -s 512k -l boot md0
Install the boot-loader code on the first (only so far) partition
Create a ZFS data partition, filling the rest of the drive
gpart add -t freebsd-zfs -l ${ZFSBOOT_POOL_NAME} -a 1M md0
Create an encrypted disk
(optionally add -T to skip passing TRIM commands to the SSD
as this can leak information about how much space is actually in use)
At this point, it will prompt you to enter and confirm your GELI
password.
Next, attach this GELI device:
geli attach gpt/${ZFSBOOT_POOL_NAME}
Now, create a pool on that encrypted device.
With a single-but-larger disk, I'd set
copies=2
on my root pool to get a little redundancy by default on everything.
But since 20GB is a little tight, I set it selectively on my datasets
for user data and assume I can regenerate anything in system datasets.
Your own needs will determine whether you make everything redundant,
just your user data, or nothing.
Because of a quirk in the installer, it requires
/usr/freebsd-dist/MANIFEST
to be present in the
chroot
environment.
It also seems to clean this up if the installer hiccups for any reason,
so I recommend putting a copy into your local directory.
It may already be on your install media, or you can download a copy
from the corresponding directory on the FTP site.
Check your mirror, architecture, and release, but it should be something
like
mkdir freebsd-dist
cd freebsd-dist
fetch ftp://ftp8.freebsd.org/pub/FreeBSD/releases/amd64/amd64/11.0-RELEASE/MANIFEST
cd ..
and then copying it into
/usr/
cp -va freebsd-dist /usr/
That way, you'll have a local copy in the event something goes awry and
you have to recreate the
/usr/freebsd-dist/MANIFEST
again.
Finally,
exit
the shell to return to the installer.
exit
The rest of the installer should be fairly straight-forward.
At the end of the install, it will prompt to drop to a shell.
Do that.
First, create/edit
/etc/rc.conf
to include the following lines, making use of the information gleaned
in the
reconnaissance
step above.
If you plan to use jails, don't let syslogd listen on all addresses
and it's worthwhile to set up a loopback interface to use while you're
in the
/etc/rc.conf
syslogd_flags="-ss"
cloned_interfaces="lo1"
Next up, edit
/etc/resolv.conf
using the DNS information gathered during
reconnaissance
to contain the
nameserver
and optionally the
search
nameserver $DNS_NAMESERVER
search $DNS_SEARCH
Assuming you set up a user during the install and that they're a member
of the
wheel
group to let you
su -
and perform administrative commands, it's worth editing the
/etc/ssh/sshd_config
to allow SSH login and prohibit root login by adding/modifying these
lines:
ListenAddress $EXTERNAL_IP
PermitRootLogin no
You may also want to add your SSH public key to
$USER/.ssh/authorized_keys
for your non-root
$USER
so you can SSH in without a password.
With changes done in the
chroot
we can now exit the
chroot
and return to the parent shell.
exit
Freeing up the image
Before shipping the image over to the server, it first needs to be shut
down cleanly.
Start by unmounting all the new ZFS items under.
The following finds them and creates a series of commands that can be
used to unmount them:
mount | awk '/\/mnt/{print "umount " $3}' | sort -r
If they all look good, you can execute them.
mount | awk '/\/mnt/{print "umount " $3}' | sort -r | sh
ZFS still has these pools active so disconnect them
zpool export $ZFSBOOT_POOL_NAME
Detach the GELI devices if needed.
geli detach /dev/gpt/${ZFSBOOT_POOL_NAME}.eli
With all filesystems disassociated from the
md0
it can now be disconnected too.
mdconfig -d -u 0
Can now leave the root shell and return to your unprivileged user
exit
Reconnecting
In the event you want to make further modifications, it's helpful to
know how to reattach the
md0,
attach the GELI volume, and import the ZFS pool again.
So as root
Start by compressing the disk image to save bandwidth.
My 20GB image compressed to ~400MB, or roughly 2%.
I keep the original image around in case I need to reconnect to it to
make changes I forgot about.
The
gzip
process takes a little while.
gzip --keep freebsd.img
Log into your VPS console and restart your VPS in rescue mode.
Once it has restarted, you should receive information via email on how
to log into that console.
Use this information to SSH into the rescue image.
ssh root@$EXTERNAL_IPv4
Your normal root drive will be mounted under the rescue
/mnt/
First you have to find out its name. It might be
vdb1,
sdb1,
or something similar so it will need to be unmounted before overwriting.
Once done, disconnect.
mount | grep /mnt
umount /mnt/vdb1
exit
Next, upload the image to the drive in question (make sure you specify
the correct root device based on the mount point, such as
vdb
or
sdb
).
Depending on your internet speed, this may take a while.
However, once the image has finished transferring, you should be able to
go into your management console and reboot the server.
If all has gone well, your server should start booting.
Note that, since we have an encrypted drive, we need to log into the KVM
console to enter the boot password.
It should then boot in that KVM console to the login prompt.
If you can get in there, then you can attempt to SSH in directly.
ssh $USER@$EXTERNAL_IPv4
If you installed your public key in
$USER/.ssh/authorized_keys
it should let you right in.
If you didn't, you can enter your credentials and should hopefully be up
and running.
As the weirdo behind the somewhat tongue-in-cheek
@ed1conf
account on
Twitter
and
Mastodon,
account I'm occasionally asked "Why
ed(1)?"
Hopefully some of my reasons for learning & using
ed(1)
can pique your interest in taking the time to get to know
this little piece of history.
This assumes that you have
virtualenvwrapper.sh
installed which isolates this installation from other projects
you may have installed locally.
Create a virtual environment
bash$mkvirtualenv myenv ↵
This command creates a new virtual environment named
"myenv". You can choose any name you like as long
as it makes sense to you. Because I use this virtual
environment for Nikola, I unimaginatively call mine
"nikola". Once the command has completed,
virtualenvwrapper automatically puts you in that
virtual environment as shown by the prefix "(myenv)".
Optionally integrating shell completion
Adding
shell completion
is optional, but a nice addition that Nikola provides. The
following two commands enable Nikola-specific tab-completion
when you activate your virtual environment, and disable it when
you deactivate the virtual environment.
If you use zsh instead of bash, you
can specify "--shell zsh" instead of
"--shell bash".
Deactivating the virtual environment
(myenv)bash$ deactivate ↵
This exits the virtual environment you created.
Alternatively, you can just quit your command shell using
"exit ↵". During initial setup,
you can skip this step, as you'll just need to reactivate the
virtual environment to install Nikola.
Reactivating the virtual environment
bash$ workon myenv ↵
This will activate the "myenv" environment again,
as well as put the environment-name indicator in your prompt.
If you didn't deactivate the virtual environment using
deactivate, then you won't need to reactivate
your virtual environment.
Install Nikola
(myenv)bash$pip install Nikola ↵
Inside your virtual environment (as shown by the
environment-name inside parentheses before your usual prompt),
this uses pip to fetch the latest version of Nikola
and install it. This brings in a number of dependencies, some
of which may need to be compiled. As such, if you don't have a
compiler installed on your machine, you'll want to do that
before this step.
Create a new blog
(myenv)bash$ cd /path/to/your/blog ↵
(myenv)bash$ nikola init myblog ↵
You can choose a name that you like for the folder instead of
"myblog" (or similarly boring reasons, I named mine
"blog"). This creates a folder called
/path/to/your/blog/myblog in which your
content, templates, cache, and output will go. It also prompts
you for the name of your site, the main author's name, the main
author's email address, a description of the site, the root URL
for the site (which should end with a trailing slash such as
"http://example.com/"), the languages you intend to
use, your time zone, and which comment system you want to use
(if any).
Configuration
Nikola's configuration takes place in
"/path/to/your/blog/myblog/conf.py".
Opening this file in your
favorite text editor,
you'll notice that the answers you gave during the
"nikola init" process appear as the values to
various settings within this file.
By default, Nikola expects markup to be
reST
but I prefer the straight-forward nature of HTML. So my first
stop was to add HTML as a processed language. By putting it
first in the list, it becomes the default for
new_post which saves me from forgetting to specify
the template as "-f html" every time I generate a
new post. To do this, I modified the "POSTS =
(…)" and "PAGES = (…)" to
include the following lines:
I want to have my templates contain additional metadata fields
for author (allowing me to easily override this for
guest posts) and category. To do this, I found the
comment containing ADDITIONAL_METADATA and added
the following lines:
ADDITIONAL_METADATA = {
"author": BLOG_AUTHOR, # default to me
"category": "uncategorized",
}
I also customized the CONTENT_FOOTER to show a
minimal copyright notice:
To round out some personal preferences, I explored the
Nikola handbook
and added the following settings in their respective places:
PRETTY_URLS = True
SOCIAL_BUTTONS_CODE = "" # I don't want social buttons
COPY_SOURCES = SHOW_SOURCELINK = False # not very useful with HTML fragments
USE_BUNDLES = False # may change this later
Finally, I wanted to enable automated deployment of my blog to
my web-hosting service. To deploy without needing to enter the
password for my hosting service, I set up SSH
keys and used ssh-copy-id to push the public
key up to my server. With that in place, I adjusted the
(formerly empty) list of commands DEPLOY_COMMANDS .
Why a static site generator over a dynamic blog engine?
I prefer to work in raw HTML because I've been doing it for
years, and the markup is readable. For some reason,
Markdown,
Textile,
and
reST
drive me nuts—mostly because it's easy for me to enter
things I don't intend as markup, and when I want to do complex
markup, I have to look up the obscure syntax in the
documentation. Every. Single. Time.
Most of the blogging engines (such as
Wordpress
and
Drupal
) do allow for directly entering
raw HTML, but it always feels like I'm fighting against the
grain. Every time I go to compose a post, I have to toggle
into a special "HTML entry" mode. One wrong turn and some of
the engines will "sanitize" my HTML, reformatting everything.
A blogging engine should treat my content as sacred and
never try to second-guess (or reformat) what
I want.
To deploy these dynamic blog engines, it requires a server
with the ability to run code (usually
PHP
but some used mod_perl, mod_python,
or mod_cgibin) and a backing database such as
MySQL.
Granted, you can find free and low-cost hosting that provides
simple PHP/MySQL access, so this isn't a huge hurdle, but it
does place additional restrictions on which services you can
use.
Some of the cheap/free shared-hosting plans put so many hosts
co-located on the same hardware that performance becomes an
issue—both database processing and the engine's
processing. With a static site generator, there is very
little overhead in serving static pages.
The more dynamic components in play, the more likely it is
that vulnerabilities lurk due to the extra exposure. With a
static site, far fewer parts are in motion to be attacked so I
don't worry about vulnerabilities in PHP, Wordpress, or MySQL.
Many of the blog engines don't support versioning of the
content. I can keep my static site content under revision
control with
git,
Mercurial,
Subversion,
or a multitude of others.
Why Nikola rather than the other static site generators?
I compared several static site generators, both using a
static site generator comparison site
and my own testing. Top contenders included the following:
As I tested them, I started to develop a list of criteria that
mattered to me:
As a Python developer, I had a bias towards those built in
Python. It allows me to poke around under the hood, create
patches when I find things I want to change, and gain a deeper
understanding of how things worked in the event I hit edge
cases.
Documentation mattered. Several projects had documentation
that was incomplete, hard to navigate, or required JavaScript
(which prevented the docs from working nicely on one of my
e-readers).
Additionally, some projects seem less than active in their
development. According to the
static site generator comparison site,
some hadn't been updated in months or years while others have
had sustained, ongoing development.
Most importantly, I wanted the ability to compose in HTML
fragments rather than being forced to use Markdown, Textile,
or reST.
So I ended up settling on
Nikola
as it's developed in Python, allows me to use HTML fragments,
has thorough documentation, and is actively developed.
I've long had an interest in starting up a simple blog for the
purpose of documenting various projects on which I'm working.
So it seems that the first thing I should blog about is the
process of choosing and configuring blogging software.
Hand-coded HTML
I started posting things on the web when I was in college where
every page was hand-crafted HTML and dumped in
~/public_html/ of my account on the Computer
Science web server (I believe it started as a
DEC ULTRIX,
but then became a Red Hat box) and everything went live thanks to
Apache
running mod_userdir. Even before using HTML, I
frequently used WordPerfect 5.1's "Reveal Codes" ability to
inspect the control tags/codes used by the document which saved
me much grief. I still prefer HTML over modern markup
competitors like
Markdown,
Textile,
or
reST
because HTML's tags are consistent. When trying the other
markup languages, I frequently get stung by entering text that
turns out to mean something to the markup engine, and when I
reach for some of the more powerful features (tables, acronyms,
code blocks, etc), I have to go to the reference materials every
single time to make sure I get it correct. In HTML, all markup
consists of an opening < followed by a closing
>, and entities are always escaped with
&. Consistent—the way I like it.
However, any time I wanted to change the look and feel of the
site, I had to touch every single file (at least those that I
cared about) and I also had to remember to update any index
pages every time I added or renamed a page.
Server Side Includes
At some point, I explored using
Server side includes
to make the site look more uniform. While it was a bit of a
hassle, it had the advantage that I could update the look of the
site by changing a couple included files. I still had to
manually update indexes, but it was a step in the right
direction. Again, for deployment, since my college server was
running
Apache
with mod_include installed,
Dynamic blogging platforms
Then blogging platforms such as
Wordpress
and
Drupal
came along with dynamic sites backed by a database. These held
a lot more promise, allowing me to maintain my content and site
theme independently.
Static Site Generators
Finally, static site generators have grown in popularity. They
allow for lower server demands, a smaller attack-surface, and a
separation of the content from the structure it populates and
its . This site has been created with
Nikola,
a static site generator developed in
Python.