Tim's blog (Posts about ed)

Old Computer Challenge

Tim Chase — Mon, 10 Jul 2023 13:52:33 GMT

History

This is the third year in which Solene has run the Old Computer Challenge and I've tried to participate in each of them.

Participating in these requires a little flexibility since I work remotely for $DAYJOB. The low-end computer challenges posed less issue since neither the VPN nor rdesktop required much in the way of resources. Using the smaller screen-dimensions on the older laptops made it a bit more challenging, but I made it work.

None of these challenges shifted any usage to my phone. I prefer not to use my phone for anything beyond the barest of essentials: phone-calls, texting, podcasts, timers, lists, and weather.

First year (2021)

The first year and third year both focused on limited hardware,

I chose a Gateway Solo 1200 as my primary machine for the first year. Boasting an 800MHz Celeron processor, a 120GB spinning-rust HDD, a 10mbit wired LAN connection (it also had an internal wi0 wireless card and a PCMCIA Intel wifi option, but both only supported WEP rather than WPA2 so I stuck with the wired option), and upgraded to its maximum of 320MB of RAM. This machine infamously arrived on 9/11 with the UPS driver delivering it as I watched the twin towers fall on TV.

The machine ran the latest release of OpenBSD without issue. The limited CPU & RAM limited my choice of software notably. Fortunately, other than web-browsing, much of what I do happens at the command-line.

Email: I had used Claws Mail (a GUI mail program) for many years but it started using more and more system resources. So I had aspired to switch to mutt or neomutt. The Old Computer Challenge gave me the kick I needed. Dealing with multiple accounts and catch-all mailboxes posed the worst pain-points. Otherwise, it ran fine within the limited system resources. And they provide a lot of power to mow through piles of email.
Music: I've long used cmus for playing my music collection, and pianobar for streaming. Both ran fine even on this ancient hardware.
Calendar: I've moved my calendaring to remind and it runs fine at the CLI. It did have a noticeable lag on startup but I suspect that my 3000+ reminders/events cause that. When I winnow it down to a much more sensible volume of reminders, it runs in a blink.
Coding: All my coding happens at the CLI using a mix of vi, vim, and ed for editing, and doing version-control with git or rcs so not much changed here. I did find notable startup lag both in starting vim and executing Python code. It made me appreciate the fast startup times for utilities that compiled down to native code. I also found myself using awk in a lot of places since it had a faster startup time than Python.
RSS: I've long used rss2email to gather my RSS feeds and deliver them to my inbox, reducing the RSS-reader issue to a mail issue. I experienced no disruption here, since mutt let me keep reading my feeds just as I had done in Claws.
Social media: I accessed Twitter with Rainbowstream, Mastodon with Tootstream, and rtv (now obsolete) for Reddit. For text posts and commenting, I loved them all. But for image/video posts, they fell short. I wish I had a quality CLI interface for Facebook to keep in touch with friends & family who only share things there.
Office stuff: Thankfully, I don't have to deal with Office documents often. And almost never outside of $DAYJOB so I could use Word or Excel remotely. I did install Abiword for the occasional MS-Word document and Gnumeric for the occasional spreadsheet. Both provided reasonable fidelity and speed while running within the confines of the limited hardware.
Gaming
: I don't game much, so this didn't impact me much. I think I played a couple rounds of cribbage(6) and atc(6) as a proof-of-concept, but certainly no high-end FPS games here.

Web browsing hurt the most. Firefox & Chromium? Completely unusable without gobs of RAM. For some basic browsing, lynx and dillo provided lightweight options, while Epiphany clocked in at barely-usable (but still better than Firefox & Chromium) for sites requiring JavaScript.

Second year (2022)

The second year focused more on limiting network usage (both total-time and bandwidth).

I had to segregate life here since $DAYJOB requires remoting into my work machine so I didn't count that time against my allotted 1hr.

I didn't know how to count my cron job that downloads my podcasts nightly since I don't have much control over how long they run or how much data they download. I decided that, since the challenge only ran for a week, and I batch podcasts roughly every three weeks, I could load a fresh batch to my player before the challenge, disable the cron job for the week, and then re-enable the cron job after completing the challenge. Not quite the spirit of the challenge, but also a lot like how I would download things in high-school, where I would walk to the local campus library to download large files and bring them home.

Email didn't pose a great concern, since OfflineIMAP let me batch download my emails from the server, and my local MTA would batch up outbound emails until I reconnected, sending them all to my smart-host mail-server in one go.

However, the second year really cut into social-media usage. Its model simply doesn't accommodate offline use well.

Third year (2023)

Similar to the first year the tools remained largely the same. However this year I did the challenge while on vacation. Cheating? Maybe. But also enforcing since I didn't take any other laptop. This time I took a Dell Mini10 netbook with me. This hand-me-down came to me with 2GB of RAM, but I'd made a few upgrades:

replaced the 120MB HDD with a 60GB SSD giving a bit of extra pep
replaced the rubbish Broadcom wireless half-height PCI card with an Atheros chipset
installed OpenBSD 7.3 in place of Windows Vista

The netbook has no fan, relying on passive cooling instead. This meant that using apm -L kept the system running cool. I could manually apm -H to get the full 1.x GHz but it came with a warm price, discouraging me from doing so.

The tiny 1024×600 screen resolution gave even greater constraints when remoting into $DAYJOB but, that helped me stay in vacation-mode rather than try to sneak in hours. Additionally, X seemed to think the display offered 1024×768 resolution, so everything rendered with a squishing/scaling that ruined friends' pictures. And equally bad, the Poulsbo chipset lacked support in X, so it rendered very slowly using VESA. But I had times where I could watch text render character-by-character, and could type full paragraphs of text before the first couple words appeared on the screen. With better graphics-support, I suspect it would have felt notably snappier.

Future challenges

After returning from that vacation, I purchased a new laptop for travel, and got rid of four of my old junker laptops (my beloved rejoices at fewer laptops on my desk). I still have the Mini10 and a PPC iBook G4 running OpenBSD, so I can participate in future challenges.

Using mail(1)

Tim Chase — Wed, 14 Dec 2022 17:05:34 GMT

Intro

CONGRATULATIONS! Your OpenBSD install has been successfully completed!

When you login to your new system the first time, please read your mail
using the 'mail' command.

OpenBSD's post-install message

Oh, no! You have a fresh install and the only way to read your welcome message from Theo requires using mail. Or perhaps you've exited some long-running program and your shell informs you

You have mail in /var/mail/$USER

However, upon launching mail, most users find that the interface has all the friendliness of ed(1).

For decades, Unix-like systems have offered mail(1) as a way to read the system mailbox and send mail. While not as flashy or featureful as more modern MUAs like s-nail, Mutt, NeoMutt, Alpine, or Aerc (or even graphical MUAs like Claws Mail, Thunderbird, KMail, or Gnome Evolution), it provides a surprising bit of functionality hidden under the hood.

History

Over the years, a variety of CLI programs have called themselves some form of "mail", including mail, Mail, mailx, heirloom-mail, and s-nail, with subtle differences in each of them. For purposes of this post, I'll try to stick mostly to mail/mailx, as defined by POSIX which means

no IMAP/POP/SMTP support
no Maildir or MH support, only classic mbox format
no built-in spam filtering
no threading
no native support for MIME types other than text/plain (so no text/html)

that you might find in some iterations such as heirloom mailx. Specifically, all examples here should work in mail as provided by FreeBSD & OpenBSD base systems.

This post assumes some basic system configuration and will discuss the three essential modes of operation:

Additionally, this post will cover some common configuration settings as well as some tips & tricks.

System configuration expectations

By default, most Linux & BSD systems provide local mail receipt & delivery out of the box on a fresh install, whether with sendmail, exim, Postfix, or my favorite, OpenSMTPD.

To exchange mail with external systems, you'll have to configure your MTA as well as possibly modify your DNS to point your MX record at your server, set up SPF & DMARC and you might want some sort of spam-filtering solution in place. However, since MTA choice & configuration fall outside the scope of this post, use whatever you prefer as long as your MDA delivers to the user's system mbox (usually /var/mail/$USER or /var/spool/mail/$USER) and your MTA has compatibility with sendmail (usually done through a binary or link at /usr/sbin/sendmail).

Finally, mail assumes that it can lock mailboxes at the file-system level. On local file-systems, this doesn't generally pose a problem. However, if your system mbox resides on an NFS share with unreliable locking, you may encounter issues/conflicts if more than one process attempts to modify the mailbox.

Essential modes of operation

Send mode

The most simple of the modes, this expects the email message body on stdin, an optional subject passed with the -s option, CC & BCC fields with the -c & -b options, followed by the list of recipient(s).

$ echo "ask about surgery" |
> mail -s "Call Mom" $USER

$ calendar |
> mail -s "Our calendar for $(date +"%Y-%m-%d")" \
> me@example.net spouse@example.net

$ git log -10 |
> mail -s "What I've done this week" \
> -b owner@example.com \
> -c boss@example.com \
> coworker@example.com

Examples of using mail in command pipelines

Send mode works well in scripts to email output of commands, and often cron & at use mail to send the results of each task to the owner.

If mail reads from an interactive terminal, it affords tilde-escape operations for editing the message. When finished with the message, use an EOF with ctrl+d to send the message.

Mail reading mode

Starting `mail`

If you currently have mail in your system mailbox, POSIX shells should inform you (roughly) every $MAILCHECK seconds (default=600, or 10 minutes) if the last-modified time of the system mailbox (as specified by $MAIL) has changed. Launching mail with no arguments opens mail, displays a screenful of message headers, and leaves you at an "&" prompt.

⋮
You have mail in /var/mail/demo
$ mail
Mail version 8.1.2 01/15/2001.  Type ? for help.
"/var/mail/demo": 2 messages 2 new
>N  1 root@example.com  Thu Dec 15 01:30   29/931   example.com daily
 N  2 root@example.com  Fri Dec 23 01:30   32/963   example.com daily
&

Example of how mail greets you

Exiting `mail`

To quit back to your shell without modifying the mailbox, use the x/exit command.

& x
$

quitting

If you use the q/quit command or type ctrl+d to send an EOF, it will change the message status of unread messages from N (new) to U (unread). By default, if you have read any of the messages in your system mailbox, they will get moved to ~/mbox upon quitting with q. The hold setting prevents mail from moving messages out of the system mailbox unless you explicitly (re)move them.

Listing messages

By default, when mail starts up, it shows you a screenful of message-summaries, one per line. Each message has an identifying number/index that should remain valid for the entire session. The header-listing consists of the following fields:

flags (New, Unread, Preserved, M send to mbox, and * saved)
message index
sender
date
size (in lines and bytes)
subject

If you have more than one screenful of messages, you can use the z command to list the next screenful, z- to list the previous screenful, and h to re-display the current screenful. When you first start mail, it displays the first page of message-summaries as if you had typed h.

To list message summaries matching particular criteria, you can use the f/from command with a message-list filter. Message-list filters can consist of a single message number,

& f 5
 N  5 root@example.com  Thu Dec 15 01:30   29/931   example.com daily

show summary for message 5

multiple message numbers,

& f 5 7 9
 N  5 root@example.com  Thu Dec 15 01:30   29/931   example.com daily
 N  7 root@example.com  Sat Jan  7 00:08   22/1191  Cron <upload@example.com>
 N  9 root@example.com  Sat Jan  7 00:17 13789/971549 Cron <upload@example.com>

show summary for messages 5, 7, and 9

or a range of numbers:

& f 5-8
 N  5 root@example.com  Thu Dec 15 01:30   29/931   example.com daily
 N  7 root@example.com  Sat Jan  7 00:08   22/1191  Cron <upload@example.com>
 N  8 bob@example.edu   Sat Jan  7 00:08   32/1625  Comments on your thesis

show summary for messages 5–8

You can use a $ to indicate the last message in the mailbox:

& f 7-$
 N  7 root@example.com  Sat Jan  7 00:08   22/1191  Cron <@example.com>
 N  8 bob@example.edu   Sat Jan  7 00:08   32/1625  Comments on your thesis
 N  9 root@example.com  Sat Jan  7 00:17 13789/971549 Cron <upload@example.com>

show summary for messages 7 through the last one

These can combine to list particular messages by number & range:

& f 2 4-6 8-$
    2 root@example.com  Wed Dec 14 01:30   29/931   example.com daily
 U  4 tim               Wed Dec 14 21:11   16/597   Call Mom
 N  5 root@example.com  Thu Dec 15 01:30   29/931   example.com daily
 N  8 bob@example.edu   Sat Jan  7 00:08   32/1625  Comments on your thesis
 N  9 root@example.com  Sat Jan  7 00:17 13789/971549 Cron <upload@example.com>

show summary for messages 2, 4–6, and 8 through the last one

If you want to refer to all messages in the mailbox, you can use *

& f *
⋮

show summary for every message in the mailbox

though this may scroll past if you have more than one screenful of messages.

Additionally, you can list messages by category of message

:d: deleted messages
:n: new messages
:o: old messages
:r: read messages
:u: unread messages

There's a subtle difference between "new" and "old" messages that might matter to some users, but that distinction has never mattered much to me.

& f :r
    2 root@example.com  Wed Dec 14 01:30   29/931   example.com daily

show summary for messages that you have read

Finally, you can list messages by searching. If you use a regular string, mail will search the "From" header for mail from that recipient:

& f bob@example.edu
 N  8 bob@example.edu   Sat Jan  7 00:08   32/1625  Comments on your thesis

show summary for messages from bob@example.edu

You can use partial matching as well:

& f bob
 N  8 bob@example.edu   Sat Jan  7 00:08   32/1625  Comments on your thesis

show summary for messages from senders containing "bob"

If you prefix the search string with a /, it will search messages' Subject field:

& f /thesis
 N  8 bob@example.edu   Sat Jan  7 00:08   32/1625  Comments on your thesis

show summary for messages where the subject contains "thesis"

Displaying messages

To display a message, use the p/print command. Without a message-list following the p/print command, mail will print the message headers and the body of the current message. If the length of the message exceeds the height of the screen (or the value of the crt setting, if set), the headers and body will get sent to your $PAGER.

& p 8
Message 8:
From bob@example.edu Sat Jan  7 00:08:53 2023
Return-Path: <bob@example.edu>
X-Original-To: demo@example.net
Delivered-To: demo@example.net
Received: by example.net (Postfix, from userid 1234)
        id 533AC0FFEE; Sat,  7 Jan 2023 00:08:53 +0000 (UTC)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Message-Id: <20230125170817.533AC0FFEE@example.edu>
Date: Sat Jan  7 00:08:53 2023 -0600 (CST)
From: Professor Bob <bob@example.edu>
To: demo@example.net
Subject: Comments on your thesis

Wow, I've never seen such rubbish.
⋮

printing message #8

A lot of headers clutter that output, so I often use retain (or ignore) to winnow these to what I care about. I retain only those headers I want to see:

& retain cc date subject
& p 8
Message 8:
From bob@example.edu Sat Jan  7 00:08:53 2023
Date: Sat Jan  7 00:08:53 2023 -0600 (CST)
Subject: Comments on your thesis

Wow, I've never seen such rubbish.
⋮

setting retained headers

Much better. Well, except for the quality of the thesis.

After using retain, if you subsequently want to see all the headers, you can use the P/Print (with a capital "P") command.

Deleting (and undeleting) messages

The d/delete command deletes the current message, while u/undelete undeletes a deleted message. Both accept a list of message-IDs like the print command. You can undelete messages as long as you haven't closed the current mailbox file. If you accidentally delete a bunch of messages remember you can always use x to quit safely without writing any changes,

& d 3 8 5
& u /thesis

deleting & undeleting messages

Replying to messages

After reading a message, you can use the r/reply command to reply to all the recipients, or the R/Reply command to reply only to the author of the message. By default, mail doesn't include the original message, so you may want to use the ~m command to read the current message into the reply, prefixed with indentprefix.

The section on mail composition has more details on how to edit your reply.

Composing new messages

To compose a new message from within mail, use the m/mail command, specifying the recipient.

& mail customerservice@example.com
Subject: broken frobniculator
I bought a new frobniculator last week
but it broke within the first 10 minutes
of use.

How can I obtain a replacement?

Thanks!
.

composing a new message

The section on mail composition has more details on how to compose your message.

Filing messages

Once you have read a message, rather than delete it, you may prefer to file it in a separate mbox archive so it doesn't clutter your main inbox. Use the s/save command to specify an mbox file. You can specify a message-list between the save command and the mbox filename to save multiple messages in one command.

Alternatively, if you would like to save a copy of the message, while keeping the original in the current mailbox, use the c/copy command which has the same syntax as save.

& p /thesis
⋮
& s thesis
"thesis" [New file]
& copy 10 12 bills
"bills" [Appended]

filing messages in another folder

Changing mailboxes

Great. Now you can read mail from your system mailbox. And you can file mail in other mbox stores. How do you access those folders? You can either specify it on the shell command-line with -f or within mail using the file command. The folder setting allows you to prefix a folder-name with "+" to refer to mbox stores in that folder regardless of your current working-directory. Additionally, you can use "%" to refer to your system mailbox, "#" to refer to the previous folder you had open, and "&" to refer to your MBOX folder.

& set folder="mail"
& copy 8 +thesis
& folders
bills   thesis

$ mail -f thesis
⋮
& folders
bills   thesis
& file +bills
⋮ # messages from the "bills" folder
& file %
⋮ # messages in the $MAIL spool

opening different mailboxes

Checking for new mail

While in a mail session, new mail might arrive in your system mailbox (or an external process might have modified your current folder). Use the inc command to "incorporate" new mail.

& h
 N  1 bob@example.edu   Sat Jan  7 00:08   32/1625  Comments on your thesis
& inc
"/var/mail/$USER": 2 messages 2 new
& h
 N  1 bob@example.edu   Sat Jan  7 00:08   32/1625  Comments on your thesis
 N  2 bob@example.edu   Sat Jan  9 11:33   48/2158  More comments

Incorporating new mail

Mail composition mode

When creating a new message, either via the shell

$ mail -s "My subj" user@example.com

or mail command-line

& m user@example.com
Subject: My subject

by default mail lets you enter your message one line at time and terminate it with a period or EOF:

& m bob@example.edu
Subject: thesis feedback
Thanks for your helpful feedback.
.

Tilde escapes

While convenient to type your message and send it, sometimes you want to edit the message in your $EDITOR, revise the list of recipients, change the Subject line, or include a message's contents for a point-by-point reply. By entering a message-body line beginning with a tilde followed by a command-letter, mail lets you make such modifications. And if you want to abandon the message, use the ~q command to save the draft in ~/dead.letter, or use the ~x to abandon the message completely. If you saved a draft, you can read it back in with ~d.

Using ~? will give you the full list of allowed tilde commands.

Use ~t recipient, ~c recipient, and ~b recipient to add recipients to the "To:", "CC:", and "BCC:" recipient-lists respectively. The ~s lets you edit your "Subject:" header. Or use ~h to edit each header in turn.

& reply /thesis
~c dean@example.edu
Hey, Bob,
Copying the dean to get her input.
⋮

adding to the CC list

Most usefully, the ~e and ~v commands let you edit the current message in your $EDITOR and your $VISUAL editors respectively. When you return from your editor, you can continue to append to the message where you left off. To display the message as it currently stands, use the ~p tilde command.

As with other mailers, sometimes you want to include the content of other messages. The ~f message-list & ~m message-list commands let you read one or more messages from your current mailbox into the current message. If you omit the message-list, mail uses the current message. If you have set the indentprefix option (I recommend the internet standard "> "), the ~m will prefix each line with the corresponding value:

& set indentprefix="> "
& m bob@example.edu
Subject: thesis feedback
~m /thesis

Thanks for your helpful feedback.
.

as commonly used for replies. By default, only the retained headers get included. If you want all the headers, use the uppercase versions ~F & ~M instead.

Alternatively, you might want to read output from an external file (with ~r filename) or command (with ~! command)

& m jsmith@example.net
Subject: having trouble with my Acme frobniculator interface
I had trouble loading the firmware.
Here's my dmesg output:
~! dmesg

and my config file:
~r /etc/acme-frobniculator

Could you point me in the right direction?

Thanks!

including files and command-output in a message

Finally, sometimes you want to pass your message through an external utility like rot13(6), b1ff(6), or a spell-check utility. The ~| command command lets you filter the message.

& m unclejoe@example.mil
Subject: dirty joke

mud, mud, mud
~| rot13
(continue)
~p
-------
Message contains:
To: unclejoe@example.mil
Subject: joke

zhq, zhq, zhq
(continue)

including files and command-output in a message

Beware: if editing over ssh Both mail & ssh use tilde-escapes at the beginning of the line. To send a tilde over ssh you may have to hit ~ twice to send the tilde to mail.

Address book

Not having the best memory, I prefer to let my computer keep track of contacts. I want to type "john" as the recipient and let my mail program figure out who I mean. Do I want to forget any family members when sending out important family-wide updates? No way! The "alias" file contains tuples beginning with the keyword alias followed by the nickname you want to use, and one or more addresses to expand into.

$ cat ~/aliases
alias mom mom@example.net
alias dad dad@example.net
alias john john@example.com
alias joe unclejoe@example.mil
alias family john, joe, mom, dad
alias bob bob@example.edu
alias dean dean@example.edu
alias thesis bob, dean

a sample alias file

This lets me compose a message to "john" or "family" and have mail expand to the correct addresses.

& mail family
Subject: Family crisis
Joe ate all the ice cream again!
.

using an alias

Settings

Your ~/.mailrc file can contain startup commands & settings to tweak the behavior of mail so this section covers the ones I find useful. I've included my ~/.mailrc below as an example.

As listed above, I set my retain to a very small subset of headers that I actually want to see, usually "Subject", "Date", and "CC".

crt: By default, mail determines the number of lines based on the display device and uses this to determine when to pass the output through $PAGER for display. Setting crt lets you override this. For instance, on one machine, I set this 120 lines and commonly read my 3–4 system messages by issuing p * to display all the messages. Any one message falls below that threshold, letting me see an individual message without paging. But I find it handy to read all of them together in $PAGER in one quick skim.
folder: By default, mail assumes all your mail-folders reside in your $HOME directory. If you use the folders command to list available mbox files, it may return lots of non-mbox files. Instead, if you store your mail in a subdirectory, you can set that here. Additionally, regardless of your current working-directory, you can prefix folder-names with a "+" to refer to folders in your folder directory
header: Each time you open a folder (whether at startup or using the file command), mail displays a screen-full of headers. If you don't want this behavior, set noheader.
hold: When quitting and leaving your system mailbox, by default mail will move every read-message into your MBOX store (~/mbox by default). By setting hold, mail will leave messages in your system mail-store until you manually move or delete them.
indentprefix: When replying to a message, common convention suggests that you prefix each quoted line with the string "> " The indentprefix setting controls this.
MBOX: This specifies your default "mbox" local mail store, defaulting to ~/mbox but I prefer to set this to a file within my ~/mail hierarchy
record: When sending a message, often times you want to keep that message around. The record setting lets you specify the folder in which to put a copy. Unfortunately, there's no folder-name for the current folder, but you can use something like +sent to keep them all in the same folder.
searchheaders: Normally when specifying a message-list by searching with /, mail will only search the Subject header. However, if you enable searchheaders, you can prefix your search term with a header-name to search. This lets you do things like f /To:dean@example.edu (to search for all messages sent to the dean) or f /Date:Jul (to search for messages sent in July). If you omit the header, it defaults to searching the Subject header as usual.

As promised, an excerpt of my ~/.mailrc

$ cat ~/.mailrc
# I prefer to keep my mbox files 
# in a mail/ subdirectory
# to keep from cluttering my
# home directory.
set folder=mail
set MBOX=+inbox
set record=+sent
set hold
set searchheaders
set indentprefix="> "
set crt=120
retain cc
retain date
retain subject
source ~/aliases

My ~/.mailrc file

Tips & tricks

Deleting all-but XYZ

I receive a number of messages from system-administration processes. Some come daily, some weekly, some monthly. Some have security advisories, while others give general system information. So I usually want to delete the non-security-related ones. However, the subject lines make this more challenging. The trick? Delete all of them, then undelete the security-related ones:

& d root@myhost.example.com
& u /insecurity

deleting all system messages, then undeleting security-related ones

I can then review these more-relevant messages without the noise of the other messages.

Quickly processing messages

On some machines, I want to read all the mail in my $PAGER pressing space to show a screen-full at a time and then delete all the messages unless something surprising shows up.

& p *
⋮
& d *

displaying all messages, then deleting them

Other times, I want to read each message and process as I go. After displaying a message, I can hit enter (with no command) to display the next message automatically. Alternatively, the dp command will delete the current message and print the next message in one go. It may only save one keypress, but I find it useful

& 1
Message 1:
⋮
& 
Message 2:
⋮
& dp

displaying sequential messages; deleting and displaying in one command

Aggressive pf(4) configuration for SSH protection

Tim Chase — Sun, 29 Mar 2020 18:14:54 GMT

If some unknown person came up to your house and started trying to open each window, jiggling each door handle, and punching in random codes on your garage-door opener, you could safely assume that they sought trouble and that you should stop them. Similarly, if some unknown computer on the internet started probing at ports on your server where you offer no services, it should set off alarm-bells and you should prevent this miscreant from interacting with your server. Especially SSH access. You know what services your machine provides. If anybody attempts to connect to some other service on your machine, in all likelihood they don't have your best interests in mind.

In this post, we configure pf on OpenBSD to catch inappropriate connection-attempts and block the offender.

Assumptions

This post assumes a few things about you:

OS installed that can run pf

While I wrote this article using pf on OpenBSD, many of the aspects should translate to FreeBSD even though FreeBSD's version of pf represents an older snapshot from when their codebases diverged. Particularly,

I think the abstract egress interface-group may not exist so you'd have to specify the interface directly (either inline or as a macro)
Where I use doas, you either have to install sudo or doas or else you must su - to become root for many of these commands.
in the _geoip script, you would need to replace the use of ftp with either fetch, wget, or curl

Administrative rights

This should go without saying, but if you don't administer the machine, you can't modify the firewall configuration.

Basic CLI

In this case, we have minimal work to do at the command-line but it helps to know your way around. OpenBSD has a template /etc/examples/pf.conf you might want to copy as a starting-point; you might find it easier to cd into /etc directory; or you might use tmux and ssh to access the server. Note that editing /etc/pf.conf can leave your server inaccessible (like cutting off the tree-branch while sitting on it), so you'll want to make sure that you have console access — locally, via a serial connection/KVM, or VNC connection.

Basic networking knowledge

The basic networking ideas of UDP, TCP, ports, services, etc. If you need to get up to speed here you might want a resource like Michael W. Lucas's Networking for System Administrators, O'Reilly's TCP/IP Network Administration, or some other such book.

Edit text-files

This focuses primarily on modifying your /etc/pf.conf which you can edit with your favorite $EDITOR, whether nano, emacs/mg, vi/vim/neovim, ed, cat, a magnetized needle and a steady hand, or butterflies. But this guide assumes that you can edit text files and save them.

Setup

For purposes here, I'll use OpenBSD's doas command to elevate privileges but you can use sudo if you have it installed on FreeBSD or execute these commands under a su - to root if you like to live dangerously. This assumes that doas (or sudo) has a configuration that lets you run the commands in question.

This also assumes starting with an empty (or mostly-empty if you did doas cp /etc/examples/pf.conf /etc/ ) pf.conf file. If you already have an existing configuration, you'll need to work out how these changes integrate with your existing setup.

Goals

This post covers several tricks to protect the actual SSH port.

change port on which SSH listens
flag any IP addresses that touch our server's minefield and prevent them from accessing SSH
if anybody actually finds the SSH port and hammers on it, ban them too
limit the country/countries from which we can SSH

Change the `sshd` port number

For purposes of this post, I configure sshd to listen on port 2345 but you should choose a different one that suits you. This doesn't give much in the way of security, but it reduces some of the noise in your log-files and makes the minefield work by making it unknown and surrounded by trip-ports.

Change the listening port

Edit your /etc/ssh/sshd_config file and specify the Port by adding/editing the line to read

Port 2345

/etc/ssh/sshd_config

Save the file and restart sshd

user$ doas rcctl restart sshd

Note: If you make this configuration change while SSHed into the server, you might cut off your access. As mentioned, make sure that you have console, serial, or VNC access in case you need to recover from making an error here.

Change the default connecting port

On my local machine, I find it painful to remember how to specify the port to various ssh commands. Some take -p $PORTNUM or -P $PORTNUM, while others take -o Port=$PORTNUM and yet others expect it as an environment variable. To simplify this, edit (or create) your ~/.ssh/config to specify the port-number there

host example
    HostName example.com
    Port 2345

~/.ssh/config

Now ssh, scp, sftp, rsync, etc, should all default to the right port-number without you needing to specify it explicitly.

Creating a minefield

Here we configure a minefield of ports such that if anybody tries to connect on any of these fake ports, they get added to a table of troublemakers. For the minefield, I default to 1024:9999 but if you want a broader range, check the output of

user$ sysctl net.inet.ip.port{first,last}

which gives a larger swath. If you extend it below 1024, make sure you don't interfere with other services that you might actually want to provide.

##########
# macros #
##########
minefield="1024:9999"
ssh_alternate_port=2345
⋮
##########
# tables #
##########
table <troublemakers> persist
⋮
#########
# rules #
#########
block quick proto tcp from <troublemakers> \
  to (egress) port $ssh_alternate_port

## these are just randomly probing

# port 22 is a dead-giveaway
# because we know we moved our SSH server elsewhere
# Also, we have no telnet or SMB
# so anyone poking there
# is up to no good
pass in on egress proto tcp \
  from any \
  to (egress) port { \
    telnet, \
    ssh, \
    netbios-ns, \
    netbios-ssn, \
    microsoft-ds \
  } \
  synproxy state \
  tag trouble

# tag stuff in the range $minefield as trouble
pass in on egress proto tcp from any to (egress) port $minefield \
  synproxy state \
  tag trouble

# unless it's to our hole-in-one
pass in proto tcp from any to (egress) port $ssh_alternate_port tag good

# if we're still trouble, add to the troublemakers
pass proto tcp from any to (egress) port $ssh_alternate_port \
  tagged trouble \
  synproxy state \
  (max-src-conn 1, max-src-conn-rate 1/10, \
  overload <troublemakers> flush global \
  )
⋮

/etc/pf.conf

We only specify TCP because malicious actors could spoof UDP source-addresses, blocking innocent folks.

Feel free to add other services to the list of services you don't provide (I have telnet, SSH, and SMB in my list) if you notice strangers knocking elsewhere.

We tag packets as "trouble", and then tag the one port we know as "good". If a packet still has the "trouble" tag, we use some trickery (involving max-src-conn & max-src-conn-rate to limit them to one connection and to one attempt every 10 seconds) to add them to the troublemakers table, banning all future SSH access for them. The flush global keyword kills any other existing connections this offending visitor currently has.

We also need the synproxy state which allows pf to handle the connection to ports that don't actually have services backing them. (Aside: I think that all ports that don't have a real service behind them need the synproxy state but it might turn out that only the overload item needs it.)

With all that in place, any attempt to connect to a port in our minefield (or select services we know we don't run) results in the remote IP address getting added to the troublemakers table and subsequently blocked from our SSH port.

Rate-limiting connections

If an attacker does manage to guess our actual port 2345 on the first try, they will likely try to hammer on it. So we want to rate-limit them:

##########
# Tables #
##########
table <bruteforce> persist
⋮
#########
# Rules #
#########
block quick proto tcp \
  from <bruteforce> \
  to (egress) port $ssh_alternate_port

# these made it through to the SSH port but abusing it
pass proto tcp from any to (egress) \
  port {$ssh_alternate_port} \
  flags S/SA keep state \
  (max-src-conn 5, max-src-conn-rate 5/5, \
  overload <bruteforce> flush global \
  )

/etc/pf.conf

This limits to 5 concurrent connections (max-src-conn 5) and 5 connection-attempts during a 10-second window (max-src-conn-rate 5/10).

Limiting by country

To begin with, I know that I will almost certainly never connect to SSH from outside the United States. IPDeny.com provides regularly-updated lists of IPv4 & IPv6 ranges for each country that I can use as a rough determination of country-of-origin. Yes, this can get thrown off by VPNs but it cuts off a surprising number of attackers.

Download GeoIP data

First, create the folder where pf will find the GeoIP tables. Inside that, create a tmp/ directory so we can download there, compare the results with what we already have, and move the new data atop the old data atomically:

user$ DEST="/usr/local/share/geoip"
user$ doas mkdir -p "$DEST/tmp"
user$ doas chmod 755 "$DEST"
user$ doas chmod 750 "$DEST/tmp"

For now, we will manually download the IPv4 and IPv6 GeoIP blocks:

user$ DEST="/usr/local/share/geoip"
user$ V4URL="https://www.ipdeny.com/ipblocks/data/aggregated/us-aggregated.zone"
user$ V6URL="https://www.ipdeny.com/ipv6/ipaddresses/aggregated/us-aggregated.zone"
user$ V4DEST="$DEST/ipv4_us-aggregated.zone"
user$ V6DEST="$DEST/ipv6_us-aggregated.zone"
user$ ftp -o "$V4DEST" "$V4URL"
user$ ftp -o "$V6DEST" "$V6URL"

Now with these files, we can point pf at them and prohibit access from non-US visitors:

##########
# Tables #
##########
table <domesticv4> \
  persist file "/usr/local/share/geoip/ipv4_us-aggregated.zone"
table <domesticv6> \
  persist file "/usr/local/share/geoip/ipv6_us-aggregated.zone"
⋮
#########
# Rules #
#########
# non-domestic IPv4/IPv6 connections simply not allowed to touch SSH
block quick inet proto tcp \
  from ! <domesticv4> \
  to (egress) port $ssh_alternate_port
block quick inet6 proto tcp \
  from ! <domesticv6> \
  to (egress) port $ssh_alternate_port

/etc/pf.conf

With this in place, any visitors from a non-US IP address simply cannot access the alternate SSH port.

Automating GeoIP downloading

Occasionally the GeoIP ranges change so you may want to download them automatically and have pf pick them up automatically. For this we'll create a stand-alone _geoip user, modify the permissions of the GeoIP data directory, set up a script to automatically download GeoIP data, compare it to existing data, and (if it changed) replace the old data and restart pf.

Create a `_geoip` user

Here we use the adduser command to create a new user. I put this user in the "daemon" login-class, specify an empty password, and then disable password logins:

user$ doas adduser _geoip
Use option ``-silent'' if you don't want to see all warnings and questions.

Reading /etc/shells
Check /etc/master.passwd
Check /etc/group

Ok, let's go.
Don't worry about mistakes. There will be a chance later to correct any input.
Enter username []: _geoip
Enter full name []: GeoIP downloader
Enter shell csh git-shell ksh nologin sh [ksh]:
Uid [1002]:
Login group _geoip [_geoip]:
Login group is ``_geoip''. Invite _geoip into other groups: guest no 
[no]:
Login class authpf bgpd daemon default pbuild staff unbound
[default]: daemon
Enter password []:
Disable password logins for the user? (y/n) [n]: y

Name:  _geoip
Password:    ****
Fullname:    GeoIP downloader
Uid:   1002
Gid:   1002 (_geoip)
Groups:      _geoip
Login Class: daemon
HOME:  /home/_geoip
Shell:       /bin/ksh

You could specify the shell as nologin but then have to do some work-arounds to get a shell to create the script and set up the cron job. After configuring the setup below, you can go back and use chsh to change the shell to nologin for added lockdown.

Modify permissions on GeoIP data directory

Now we need to make sure that the _geoip user can write to the directories

user$ doas chown -R _geoip:_geoip "$DEST"

Allow the `_geoip` user access to restart `pf`

In order for the _geoip user to instruct pf to reload its configuration and pick up the new GeoIP files, we need to add a line in our /etc/doas.conf file:

permit nopass _geoip cmd /sbin/pfctl args -f /etc/pf.conf

/etc/doas.conf

This allows the _geoip user to run only pfctl as specified by full path, and only with the arguments -f /etc/pf.conf to minimize things. If you want to lock it down even further you can limit the _geoip user to only reload those two tables with something like this (untested):

permit nopass _geoip \
  cmd /sbin/pfctl \
  args -t domesticv4 -T replace -f "/usr/local/share/geoip/ipv4_us-aggregated.zone"
permit nopass _geoip \
  cmd /sbin/pfctl \
  args -t domesticv6 -T replace -f "/usr/local/share/geoip/ipv6_us-aggregated.zone"

/etc/doas.conf

If you choose to go this route, make sure that you update the script below to use

/sbin/pfctl -t domesticv4 -T replace -f "/usr/local/share/geoip/ipv4_us-aggregated.zone"
/sbin/pfctl -t domesticv6 -T replace -f "/usr/local/share/geoip/ipv6_us-aggregated.zone"

~_geoip/bin/update_zone.sh

instead.

Create a script to download & install GeoIP info

I switch to the _geoip user and create the script to use

user$ doas su - _geoip
_geoip$ mkdir -p ~/bin
_geoip$ touch ~/bin/update_zone.sh
_geoip$ chmod +x ~/bin/update_zone.sh

I then used my $EDITOR to populate that ~/bin/update_zone.sh file with the following script:

#!/bin/sh
FTPOPTS="-MV"
DEST="/usr/local/share/geoip"
V4URL="https://www.ipdeny.com/ipblocks/data/aggregated/us-aggregated.zone"
V6URL="https://www.ipdeny.com/ipv6/ipaddresses/aggregated/us-aggregated.zone"
V4TMP="$DEST/tmp/ipv4_us-aggregated.zone"
V4DEST="$DEST/ipv4_us-aggregated.zone"
V6TMP="$DEST/tmp/ipv6_us-aggregated.zone"
V6DEST="$DEST/ipv6_us-aggregated.zone"

CHANGED=false
# fetch them to the temp location
# and if successful and they differ,
# replace the original
ftp $FTPOPTS -o $V4TMP "$V4URL" && ! cmp -s "$V4TMP" "$V4DEST" && mv "$V4TMP" "$V4DEST" && CHANGED=true
ftp $FTPOPTS -o $V6TMP "$V6URL" && ! cmp -s "$V6TMP" "$V6DEST" && mv "$V6TMP" "$V6DEST" && CHANGED=true

# if it changed, reload pf.conf
$CHANGED && (echo "Loading new domestic zones for SSH blocking" ; doas /sbin/pfctl -f /etc/pf.conf )

~_geoip/bin/update_zone.sh" id="script

Scheduling the download

On OpenBSD 6.7 and later cron allows the use of the ~ operator to indicate a randomly chosen time to help distribute the running of tasks without everything happening at one time. So after issuing su - _geoip to become the _geoip user, I used

_geoip$ crontab -e

to edit the crontab and add an entry to run the script:

# min, hr, day-of-mon, mon, day-of-wk
~ 1~3 * * 2 $HOME/bin/install_us_zone.sh

crontab -e

This picks some random time between 1:00am and 3:59am on Tuesday ("2") to run our script. If you use another cron or an older version on OpenBSD, choose an appropriate time. The IPDeny terms-and-conditions offer very liberal usage limits so you could download them more frequently. However, I find that the data doesn't change too frequently, so weekly works for me.

Final file

Putting it all together my final file looks something like this (including the initial entries from /etc/examples/pf.conf)

##########
# Macros #
##########
# see also `sysctl net.inet.ip.port{first,last}`
minefield="1024:9999"
ssh_alternate_port=2345
⋮

##########
# Tables #
##########
table <bruteforce> persist
table <troublemakers> persist
table <domesticv4> persist \
  file "/usr/local/share/geoip/ipv4_us-aggregated.zone"
table <domesticv6> persist \
  file "/usr/local/share/geoip/ipv6_us-aggregated.zone"
⋮

###########
# Options #
###########
set skip on lo
⋮

#########
# Rules #
#########
block return    # block stateless traffic
pass      # establish keep-state

# block brute-forcers
block quick proto tcp from <bruteforce> \
  to (egress) port $ssh_alternate_port
block quick proto tcp from <troublemakers> \
  to (egress) port $ssh_alternate_port

# non-domestic v4/v6 connections simply not allowed to touch SSH
block quick inet proto tcp \
  from ! <domesticv4> \
  to (egress) port $ssh_alternate_port
block quick inet6 proto tcp \
  from ! <domesticv6> \
  to (egress) port $ssh_alternate_port

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

# these made it through to the SSH port but abusing it
pass proto tcp from any to (egress) \
  port {$ssh_alternate_port} \
  flags S/SA keep state \
  (max-src-conn 5, max-src-conn-rate 5/5, \
  overload <bruteforce> flush global \
  )

## these are just randomly probing

# port 22 is a dead-giveaway
# because we know we moved our SSH server elsewhere
# Also, we have no telnet or SMB
# so anyone poking there
# is up to no good
pass in on egress proto tcp \
  from any \
  to (egress) port { \
    telnet, \
    ssh, \
    netbios-ns, \
    netbios-ssn, \
    microsoft-ds \
  } \
  synproxy state \
  tag trouble

# tag stuff in the range $minefield as trouble
pass in on egress proto tcp from any to (egress) port $minefield \
  synproxy state \
  tag trouble

# unless it's to our hole-in-one
pass in proto tcp from any to (egress) port $ssh_alternate_port tag good

# if we're still trouble, add to the troublemakers
pass proto tcp from any to (egress) port $ssh_alternate_port \
  tagged trouble \
  synproxy state \
  (max-src-conn 1, max-src-conn-rate 1/10, \
  overload <troublemakers> flush global \
  )

pass proto tcp from any to any port {http https} \
  keep state (max-src-conn 10, max-src-conn-rate 10/3)

⋮

/etc/pf.conf

Conclusion

With all this in place, the SSH port now has protection from non-domestic IP addresses, folks probing around randomly to ports they have no business connecting to, and folks who actually find the port but bang against it too hard. From here, one might also implement fail2ban, blacklistd, or sshguard to monitor your sshd logs, watching for failed logins and then banning based on repeated login failures.

How I use remind(1)

Tim Chase — Sun, 16 Feb 2020 19:35:08 GMT

Overview

The remind calendar utility offers an amazing degree of power for doing calendar-related scheduling unmatched by any other calendar program I've used. If you haven't encountered it before, this presentation (PDF) gives a good overview of some of the features, as do Linux Journal and 43 Folders. In this post, I'll walk through how I use both the basic and advanced features.

This post makes a couple assumptions about you as the reader:

General working knowledge of the CLI: While most of the shell commands should work in the majority of Unix-like shells (/bin/sh, bash, ksh, zsh, or csh/tcsh) some have slight syntax differences. Such differences might include the for loop syntax (csh/tcsh has a different syntax than the sh style loops I use), and what variables the shell provides (such as $COLUMNS that bash & ksh provide, which you might have to define or hard-code before using in other shells). If you build from source into a local ~/bin/ directory, you will want to know about modifying your $PATH as well. You don't need a Ph.D. in using the command-line to do any of these, but I do assume basic directory creation & navigation skills using mkdir & cd, as well as moving/renaming and copying with mv/cp and output redirection. I also have a section on using git or rcs to version your reminders but that section aims to help you integrate whatever version-control software you already use rather than try to teach either as a requirement.
General competency with a text editor: Reminder files are pure plain-text. So you can edit them with your favorite $EDITOR, whether nano, emacs/mg, vi/vim/neovim, ed, cat, a magnetized needle and a steady hand, or butterflies. But this guide assumes that you can edit text files and save them.
You have remind already installed or can install it: If you administer your personal system, you should have root/sudo/doas access to install remind from a binary package (available in most repositories) or possibly install a C compiler like gcc/clang to install from source. If you do not administer your system, you can either talk to your system administrator and ask them to install remind (try offering baked goods); or you can install from source, specifying a user-local directory such as $HOME/bin/ (as long as you include that in your $PATH, you have a C compiler on the system, and your administrator didn't mark your $HOME/ directory's mount-point as noexec).

Installing

As a package

Initially released in 1990 via USENET on comp.sources.misc, nearly every package repository should provide remind, making it as easy to install as any other package:

root@freebsd# pkg install remind

FreeBSD

user@openbsd$ doas pkg_add remind

OpenBSD

user@debian$ sudo apt-get install remind

Debian/Ubuntu based

On some systems, this has no dependencies; on other systems, this brings in some GUI components (Tcl/tk) because the package might also include the optional graphical tkRemind component.

From source

Alternatively, download the source directly from the latest tarball (and hopefully verify it using gpg with the PGP key which should have the fingerprint 738E:4D95:4052:902C:147D:07B2:685A:5A5E:511D:30E2)

user$ VER=0.3.03.00
user$ wget https://dianne.skoll.ca/projects/remind/download/remind-${VER}.tar.gz{,.sig}
user$ gpg --verify remind-${VER}.tar.gz.sig  xvfz remind-${VER}.tar.gz
user$ tar xvfz remind-${VER}.tar.gz
user$ mv remind-${VER} remind

or if you want to download from git:

user$ git clone https://dianne.skoll.ca/projects/remind/git/Remind.git remind

and then compile it yourself:

user$ cd remind
user$ ./configure
user$ make
user$ sudo make install

If you prefer not to make install you can move/copy the resulting binaries from remind/src/{remind,rem2ps} to a ~/bin/ directory (in your $PATH), create a link from remind to rem, and put the man pages from remind/man/*.1 in your $MANPATH.

Getting started

Creating your first `remind` file

By default, invoking rem expects data in ~/.reminders and you can just dump all your reminders in there (though remind also lets you split reminders up into individual files). When invoked as remind rather than rem you must specify the reminder file directly:

user$ remind path/to/reminders.rem

The most basic reminder consists of REM followed by a date of some sort, followed by MSG and the text of the reminder. A few things to keep in mind

remind ignores blank lines. and treats lines beginning with a # as comments
to continue a reminder on the next line, end the previous line with a backslash ("\")
If you only include part of the date remind assumes the missing parts repeat.
You can specify either the numeric day of the month or the day of the week (in which case it will repeat on that day of the week).
For the month and week-day, remind does not distinguish between upper/lower-case and thus treats "Jan", "jan", "JaN", "jAn", and "JAN" all as the same.
Additionally, while remind requires the first three characters of months ("Jan") and weekday names ("Wed"), you may also fully spell out the month ("January") or weekday name ("Wednesday") if that makes it easier for you to read.
If you provide more than one weekday, remind will match any of them.
Alternatively, you can write the date in YYYY-MM-DD format
If you omit all of the values, omit the REM too

$EDITOR

~/.reminders

# this is a comment
# these and the next (blank) line are ignored

REM Jan 1 2020 MSG The first day of the 2020
REM 2020-1-1 MSG Also the first day of the 2020
REM Jan 1 MSG The first day of the year
REM 1 MSG The first day of every month
REM Jan MSG Every day in each January
REM January MSG Also every day in each January
REM Jan 1 MSG Every January 1st
REM Jan 2020 MSG Every day in January 2020
REM Wed MSG Every Wednesday
REM Wednesday MSG Also every Wednesday
REM Wed Jan MSG Every Wednesday in every January
REM Wed Jan 2020 MSG Every Wednesday in January 2020
REM Mon Wed Fri MSG Every Monday, Wednesday, and Friday
REM Sat Sun Jan 2020 MSG Every weekend in January of 2020
MSG With no REM, this happens every single day
REM Jan 1 2020 \
  MSG This line has a continuation

~/.reminders

user$ remind ~/.reminders 2020-01-01
user$ rem 2020-01-01

^st

remind

user$ rem

By specifying a day of the week and a numeric date, remind will find the next date-match and will then scan forward until it finds the matching day of the week.

REM Mon 1 MSG First Monday of the month
REM Mon 1 Jan MSG First Monday in January
REM Mon 1 Jan 2020 MSG First Monday in January, 2020
REM Mon 8 MSG Second Monday of the month
REM Mon 15 MSG Third Monday of the month
REM Mon 22 MSG Fourth Monday of the month
REM Mon 29 MSG Infrequent Fifth Monday of the month
REM Sat Sun 15 MSG Third Saturday or Sunday of the month

~/.reminders

Beware

^th

SATISFY

same

Also note that this can have interesting side-effects such as trying to find Labor Day.

Similarly, our city offers a program for recycling items that we shouldn't put in our recycle bin (hazardous household waste, electronics, etc). These events occur quarterly on the Saturday after the first Friday of the month. We can express the date as

REM Sat 2 MSG Recycling center drop-off day

~/.reminders

SATISFY

REM Sat 2 SATISFY [$Tm % 3 == 2] \
  MSG Recycling center drop-off day

~/.reminders

Adding time information

To indicate the time and length of an event use AT (in military/24-hr time format; I wish the parser allowed for 12-hr AM/PM notation, but alas) and optional DURATION (in hours and minutes). Here I have a doctor's appointment at 2:30pm but don't know how long it will run. And I go to the gym on Tuesday/Thursday at 5:00am for an hour and fifteen minutes:

REM Jan 30 2020 AT 14:30 MSG Doctor's appointment
REM Tue Thu AT 5:00 DURATION 1:15 MSG Gym

~/.reminders

week/month view

substitution filters

MSG

Getting advance notice

Sometimes you want a bit of advanced warning like knowing about your doctor's appointment tomorrow. Perhaps you want a week of notice before your mother's birthday so you can get something in the mail. Or you want a 60 days of notice before you have to pay taxes so you can get all your paperwork in order. remind uses +n and ++n to give you n days of advanced warning. The single + skips over OMIT dates, while ++ ignores OMIT dates. For now, use the ++n form for simplicity.

REM Jan 1 2020 ++1 AT 14:00 MSG Doctor's appointment
REM May 12 ++7 MSG Mom's birthday
REM Apr 15 ++60 MSG Taxes

~/.reminders

remind

week or month calendar views

-pa

-sa

remind

adjust the reminder text based on the number of days out

Adjusting dates

To adjust a date backwards, use -n or --n to specify the number of days. At first glance, this provides little benefit over just doing the math and hard-coding it for normal reminders:

REM 15 --3 MSG Really the 12th
REM 12 MSG Just use the 12th instead

~/.reminders

REM 1 --1 MSG Last day of the month
REM 1 --7 MSG One week left in the month
REM 1 Wed --7 MSG Last Wednesday of the month
REM Feb 1 Wed --7 MSG Last Wednesday in January

~/.reminders

As with +n vs. ++n, the single "-" jumps over OMIT dates, (so they can end up going back more than n days if they encounter an intervening holiday); while the double "--" ignores OMIT dates. These adjustments can combine

REM 1 --1 ++3 MSG Last day of the month

~/.reminders

Repeating reminders

As shown above, if you omit portions of a REM statement, remind treats the missing date part as a repeat. This lets you create reminders such as birthdays and anniversaries, holidays, weekly events like church, or invoicing customers on a given day of each month:

REM Jan 13 MSG Steve's birthday
REM Feb 18 ++7 MSG Mom & Dad's anniversary
REM Apr 1 ++1 MSG April Fool's Day
REM Sun AT 9:00 MSG Church
REM 15 MSG Invoice customers

~/.reminders

On occasion I need a reminder every N days like picking up a 90-day prescription that started on January 27^th of 2019. To do this, use an asterisk followed by the number of days to repeat.

REM Jan 27 2019 *90 MSG Pick up medication

~/.reminders

example of a medication schedule

For repeating events, remind lets you specify date ranges using FROM, UNTIL, and THROUGH keywords. For example, if your vacation starts on December 21^st and repeats daily (*1) until January 1st you can use:

REM Dec 21 2019 *1 UNTIL Jan 1 2020 MSG Vacation

~/.reminders

THROUGH

*1

REM Dec 21 2019 THROUGH Jan 1 2020 MSG Vacation

~/.reminders

*2

^st

REM Jan 1 2019 *2 UNTIL Jan 31 2020 MSG Take medication

~/.reminders

^th

REM Wed FROM Aug 21 2019 UNTIL Jun 1 2020 MSG Game Night
REM 12 FROM Aug 21 2019 UNTIL Jun 1 2020 MSG School year meeting

~/.reminders

Modifying the output

Other views

If you have no reminders for a given day, remind prints "No reminders." Otherwise, remind prints a banner of the form "Reminders for Wednesday, 1st January, 2020 (today)" followed by an agenda-view of the reminders for that day. If you want more than one day worth, specify the number of days with "*n". Note that you should quote the "*" either by escaping it with a backslash or by wrapping it in quotes. Otherwise the shell might expand it to a filename in your current directory. To show four days of reminders:

user$ rem \*4
user$ rem '*4'

remind also offers an ASCII calendar view for either the month or week:

user$ rem -c # a month view
user$ rem -c2 # two month view
user$ rem -c 2019-8-1 # one-month-view of 2019-08-01
user$ rem -c+ # a week view
user$ rem -c+2 # two week view

-w$COLUMNS

user$ rem -c+2 -w$COLUMNS

remind

TTY

Other message-types

Up to this point, we've only discussed the MSG message-type. remind also provides CAL for events that should only show up in the week/month view but not the agenda view, and MSF which behaves the same as the MSG except it wraps long text in the agenda view.

REM Jan 1 2020 CAL This only appears on the calendar view
REM Jan 1 2020 MSF This is a very long reminder \
  and includes several lines \
  of text all with the escaping backslash \
  but when you view it \
  in the agenda view \
  it will be wrapped \
  so that it formats nicely.

~/.reminders

remind

RUN

SPECIAL

PS

PSFILE

Getting rid of blank lines

I find it annoying to get a blank line after each reminder. You can prevent this by ending reminder text with a %

REM Jan 30 2020 +3 MSG Doctor's appointment%

~/.reminders

%

every single reminder

Selectively showing text

You might want certain text to appear on the daily agenda calendar but you don't need that level of detail on the week/month calendar view. You can wrap the text you want on the week/month calendar view with %"…%"

REM Jan 30 2020 +3 MSF %"Doctor%"'s appointment \
  123 Main St., Anytown, NY\
  800-555-1212

~/.reminders

%"

Substitution filter

Your reminder-message can include escape sequences (beginning with a %) that adjust based on the context. So instead of a reminder that just reads "Mom's birthday" you can use

REM May 12 ++7 MSG Mom's birthday %b

~/.reminders

remind

in 7 days' time

in 2 days' time

tomorrow

today

man

remind

SUBSTITUTION FILTER

%b: described above, producing "in N days' time", "tomorrow", and "today"
%l: replaced with "on YYYY-MM-DD", "tomorrow", and "today"
%c: replaced with the day-of-the-week of the event such as "on Wednesday", then "tomorrow", and "today"
%2: replaced with the AT time in 12-hour am/pm time, e.g. "at 2:30pm"

REM Jan 1 2020 +1 AT 14:30 MSG Doctor appointment %b %2

~/.reminders

remind prints a "banner" before each day's agenda. The default BANNER contains "Reminders for %w, %d%s %m, %y%o:" but you can use any of the substitution filter formatting sequences that fit your wants. The extra blank line after the BANNER bothers me so let's get rid of that by putting a % at the end:

BANNER Reminders for %w, %d%s %m, %y%o:%

~/.reminders

BANNER Reminders for %w, %m %d%s, %y%o:%

~/.reminders

When I use rem '*3' to get 3 days worth of reminders I dislike how it formats the dividers because they all flow together visually, making it hard for me to read. Fortunately, remind lets you use an IF/ELSE/ENDIF block to conditionally tweak this as well:

IF defined("subsequent_iteration")
  BANNER ---------------------%_%w, %m %d%s, %y%o:%
ELSE
  BANNER %w, %m %d%s, %y%o:%
  SET subsequent_iteration 1
  PRESERVE subsequent_iteration
ENDIF

~/.reminders

subsequent_iteration

BANNER

subsequent_iteration

PRESERVE

subsequent_iteration

remind

ELSE

Sorting

By default, remind prints agenda items in the order in which they appear in the reminder files. Using the -g flag tells remind to sort the output. Following the -g comes up to four "a" or "d" characters to indicate ascending or descending order. Positionally, these indicate

the trigger date of the event
the trigger time of the event
the priority of the event
whether to sort timed events before untimed events

rem -gaad

When sorting reminders, I like to have a visual indication between where today's reminders stop and subsequent days' advanced notices so I override the special sortbanner() function. Much like the BANNER, this lets you define a separator that comes between these events. I use the $SortByDate, $SortByPrio, and $SortByTime variables to determine if any sort-order creates these logical breaks, and then use a substitution filter in my function-result:

SET banner_str "…"
IF $SortByDate + $SortByPrio + $SortByTime > 0
  BANNER %
  FSET sortbanner(x) iif(x == $U, \
    banner_str, \
    "----%B----%" \
    )
ELSE
  BANNER [banner_str]
ENDIF

~/.reminders

BANNER %

sortbanner()

%B

%b

Next instance of reminders

If you want to know the next time a reminder will occur, remind offers the "next" (-n) flag. Beware as this produces ~900 reminders for me (one for each of my 1600+ reminders yet to happen in the future), but you can grep this output or pipe it to $PAGER

user$ rem -n |
>  grep Alice |
>  sort |
>  less

File management

Organizing reminders

While remind will happily let you dump all your reminders in ~/.reminders, it also provides functionality to split and combine files using the INCLUDE directive. I like to organize reminders into files based on reminder-type, so I created a directory to house them all

user$ cd
user$ # don't tromp over existing reminders if we have them
user$ mv .reminders original_reminders.rem
user$ mkdir -p ~/.config/remind
user$ echo 'SET remind_dir getenv("HOME") + "/.config/remind/"' > \
>  ~/.config/remind/reminders.rem

~/.reminders

user$ ln -s .config/remind/reminders.rem ~/.reminders
user$ cd ~/.config/remind

remind

expressions

INCLUDE

rsync

$HOME

Alternatively, you can create a link to the containing folder instead of a reminder file full of INCLUDE lines:

user$ ln -s .config/remind/ ~/.reminders

remind

*.rem

010-birthdays.rem

015-anniversaries.rem

020-work.rem

Initial file types

I start by creating a helpers.rem file to store a variety of helper functions and constants.

user$ touch ~/.config/remind/helpers.rem

INCLUDE

reminders.rem

user$ for f in ushol birthdays anniversaries work bills …
>  do
>  echo "INCLUDE [filedir()]/helpers.rem" > ~/.config/remind/${f}.rem
>  echo "INCLUDE [remind_dir]/${f}.rem" >> ~/.config/remind/reminders.rem
>  done

an individual file for myself
one for my wife
one for each kid
one for each kid's school calendar
birthdays
anniversaries
family events
household chores
finances
church events
one for each place I volunteer
US holidays
events at our local library
work

man-page-per-day

following

INCLUDE

one for the kids (combining both kids' personal calendars and their school calendars)
one for the whole family (combining my calendar and my wife's calendar with the combined kids' calendar)
and of course the master reminders.rem that contains everything

To save a bit of processing time, I wrap my helpers.rem in a guard so that it only gets processed once:

IF !defined("HaveHelpers")
SET HaveHelpers 1
# make sure this IF guard gets closed at the bottom
⋮
# all my helpers go here
⋮
ENDIF # HaveHelpers

helpers.rem

At first glance, splitting out each file doesn't seem to offer any advantages. However, this lets you do things like show a calendar of birthdays only without all the other reminders, or show just a school calendar.

user$ remind -c ~/.config/remind/birthdays.rem

The majority of my ushol.rem file comes from the examples on the remind website. It creates a bunch of OMIT entries useful for any reminders that get bumped because of US holidays.

Advanced features

While all that is useful, other calendar programs like Google Calendar, Microsoft Outlook, or even the venerable calendar(1) CLI utility can do many of those things. However remind really shines when you want to do things that these others simply can't.

Expression evaluation

For even more power & flexibility, remind lets you define variables & functions as well as evaluate expressions (documented in the man-pages as "EXPRESSION PASTING").

To set a variable, use SET and remind will evaluate the right-hand side (the "4 - 3" in the example), assigning the result to the variable on the left-hand side (the myvar in the example):

SET myvar 4 - 3

~/.reminders

FSET

add_three

x

FSET add_three(x) 3 + x

~/.reminders

REM Jan 1 2020 +1 MSG 4 - 3 = [4 - 3]
REM Jan 1 2020 +1 MSG 4 - 3 = [myvar]
REM Jan 1 2020 +1 MSG 4 - 3 = [add_three(-2)]
REM Jan [4 - 3] 2020 +1 MSG 4 - 3 = 1
REM Jan 1 [add_three(2017)] +1 MSG 4 - 3 = 1
REM Jan 1 2020 +[myvar] MSG 4 - 3 = 1

~/.reminders

SET VacationStart date(2020, 7, 15)
SET VacationEnd VacationStart + 10
REM [VacationStart - 60] OMIT Sat Sun BEFORE \
  MSG Submit time-off paperwork
REM [VacationStart - 50] MSG Buy tickets
REM [VacationStart - 3] OMIT Sun BEFORE MSG Stop mail delivery
REM [VacationStart - 1] MSG Pack bags
REM [VacationStart] THROUGH [VacationEnd] MSG Vacation
REM [VacationEnd + 1] OMIT Sat Sun AFTER MSG Back to work

~/.reminders

OMIT

Using functions in reminder text

If I know the birthday or anniversary of a friend, I like to include that in the reminder text. So in my helpers.rem I have

⋮
FSET born(y) "(" + ($Uy-y) + "yo)"
FSET married(y) (ord($Uy-y)) + " anniversary"
⋮

helpers.rem

The $Uy holds the year of the currently-rendering event whether today, the date passed on the command line, one of the subsequent days in a multi-day agenda view, or on a week/month calendar view. So by subtracting the birthday/anniversary year from the current-event's date, you get which birthday/anniversary. The ord() function turns a number into its ordinal representation ("1" → "1st", "2" → "2nd", "3" → "3rd", "4" → "4th", …)

⋮
REM Apr 1 +7 MSG %"Bob & Alice [married(1999)]%" %b
⋮
REM Jan 14 +7 MSG %"Bob [born(1980)]%" %b
⋮

~/.reminders

Bob & Alice 20th anniversary tomorrow
⋮
Bob (40yo) in 3 days' time

Days until an annual event

As an example of function use, say you want to determine the number of days until an annual event. If it has already passed this year, you want to refer to the event next year; if it hasn't already passed this year, you want to refer to the event this year. A function wraps up this functionality nicely:

FSET DaysUntil(dt, m, d) iif(\
    date(year(dt), m, d) < dt,\
    date(year(dt)+1, m, d),\
    date(year(dt), m, d)\
    ) - dt
FSET DaysTillStr(dt, m, d) \
    DaysUntil(dt, m, d) + \
    plural(DaysUntil(dt, m, d), " day")
FSET ChristmasDays(dt) DaysTillStr(dt, 12, 25)
MSG %"[ChristmasDays($T)] until Christmas%"%

~/.reminders

When run, this will produce output

user$ rem 2020-12-1
Reminders for Tuesday, 1st December, 2020:

24 days until Christmas
user$ rem 2020-12-28
Reminders for Monday, 28th December, 2020:

362 days until Christmas

Specifying the priority of a reminder

While I don't currently use task priority much, remind allows you to provide a PRIORITY modifier (an integer between 0 and 9999) for an event. If unspecified, remind sets the priority to $DefaultPrio (5000 by default). As far as I can tell, there's no inherent meaning to the priority, so if it makes more sense to you that 9999 means "high priority" and 0 means "low priority", then use them as such; if it makes more sense to you that 0 means "high priority" and 9999 means "low priority", use that. I tend to declare a couple constants using SET and then use those, making it easier to reverse the meaning should I need to.

SET PRIORITY_HIGH 9999
SET PRIORITY_LOW 0
REM Jan 1 2020 \
    AT 10:00 DURATION 1:00 \
    PRIORITY [PRIORITY_HIGH] \
    MSG Very important meeting
REM Jan 1 2020 \
    AT 14:00 DURATION 1:00 \
    PRIORITY [PRIORITY_LOW] \
    MSG Unimportant thing

~/.reminders

Adding a prefix/suffix to agenda items

remind provides a pair of special functions, msgprefix(p) and msgsuffix(p), that emit a prefix & suffix around each agenda item. These functions take an explicit priority parameter (although you can reference other variables and functions) allowing you to do things like

# Annotate high-vs-low priority
FSET is_low_prio(p) p < $DefaultPrio
FSET is_high_prio(p) p > $DefaultPrio

FSET msgprefix(p) \
    iif(is_high_prio(p), "+ ", \
    iif(is_low_prio(p), "- ", \
    "  "))

# add some asterisks after high-priority tasks
FSET msgsuffix(p) \
    iif(is_high_prio(p), " *****", "")

~/.reminders

is_low_prio

is_high_prio

tricks from the section on color

FSET msgprefix(p) \
    iif(is_high_prio(p), Red, \
    iif(is_low_prio(p), Gry, \
    Nrm))
FSET msgsuffix(p) Nrm

~/.reminders

Omitting and shifting dates

Some events shift around or get cancelled based on whether they fall on particular days like holidays, weekends, or even just for arbitrary reasons ("somebody else booked the conference room so we need to move it to next week"). The OMIT keyword tells remind how to identify these cases.

Omitting/shifting for a single event

You might have a project at work that runs for a couple weeks but you don't want your agenda to display it on weekends because you have work/life boundaries. Use the OMIT keyword to tell which days, and SKIP to tell remind to ignore events that fall on these days. Put them in your REM command:

REM Jan 2 2020 THROUGH Jan 15 2020 \
    OMIT Sat Sun SKIP \
    MSG Work project

~/.reminders

^th

AFTER

REM 15 OMIT Sat Sun AFTER MSG Payday

~/.reminders

^th

BEFORE

REM 15 OMIT Sat Sun BEFORE MSG Pay water bill

~/.reminders

Omitting/shifting for multiple events

Certain events such as federal holidays can displace multiple reminders. Rather than try and OMIT every single holiday in every single reminder, remind provides bare OMIT entries similar to REM but they get added to a contextual omit-list. If you have important weekly meetings on Monday and Friday and an unimportant meeting on Wednesday, the following will bump the important Monday meeting back to Tuesday if Christmas fell on Monday, will bump the important Friday meeting back to Thursday if Christmas fell on a Friday, and cancel the unimportant Wednesday meeting if Christmas fell on a Wednesday,

OMIT Jul 4 MSG 4th of July
OMIT Dec 25 MSG Christmas
REM Mon AFTER MSG Important Monday Meeting
REM Wed SKIP MSG Unimportant Wednesday meeting
REM Fri BEFORE MSG Important Friday Meeting

~/.reminders

user$ rem 2015-12-24 '*2' # got moved to 24th
user$ rem 2017-12-25 '*2' # got moved to 26th
user$ rem 2019-12-24 '*3' # got cancelled

^th

OMIT

Omit notice and adjusting backwards

As discussed in the advanced-notice and adjusting-dates sections, remind can give you advanced notice of reminders and adjust dates backwards a certain number of days. In that section, I recommended using the double ++/-- because they ignore OMIT dates, making them easier to understand. However sometimes you want extra days' notice if intervening OMIT holidays happens (such as closing the bank or post-office) in which case you should use the single + form. Likewise, use -n to find the last day of the month, but you want it to skip over OMIT dates.

OMIT Dec 25 MSG Christmas
REM Dec 26 +1 MSG shows on the 24th, 25th, and 26th
REM Dec 26 ++1 MSG shows on the 25th and 26th but not 24th
REM Jan 1 -7 MSG shows on the 24th
REM Jan 1 --7 MSG shows on the 25th

~/.reminders

Omit contexts

Because not all places or calendars use the same list of holidays to determine OMIT days, remind lets you create custom OMIT contexts with PUSH (ending those temporary contexts with POP) and optionally use CLEAR to temporarily remove previous OMIT dates. As an example, our kids' schools close for (most) federal holidays but each has its own list of other days off such as in-service days and bad-weather make-up days.

# our son's school has certain days off
PUSH
OMIT Apr 27 2020 MSG %"In service day%" \
  No school for son
OMIT May 22 2020 MSG %"Bad-weather make-up day%" \
  No school for son
REM Mon Tue Wed Thu Fri \
  FROM Aug 14 2019 \
  UNTIL Jun 1 2020 \
  SKIP \
  MSG %"Son's school%"
POP
# our daughter's school has other days off
PUSH
OMIT Apr 13 2020 MSG %"In service day%" \
  No school for daughter
OMIT May 8 2020 MSG %"Bad-weather make-up day%" \
  No school for daughter
REM Mon Tue Wed Thu Fri \
  FROM Aug 14 2019 \
  UNTIL Jun 1 2020 \
  SKIP \
  MSG %"Daughter's school%"
POP

~/.reminders

OMIT

PUSH
CLEAR
OMIT 5 Oct 2019 MSG %"No breakfast get-together%" \
  (venue unavailable)
REM Sat 1 +1 AT 07:45 DURATION 1:30 \
  OMIT Sun Mon Tue Wed Thur Fri AFTER \
  FROM Aug 31 2019 \
  MSG %"Breakfast get-together%" %b %2
POP

~/.reminders

the CLEAR inside the PUSH/POP block means that my global OMIT list of holidays doesn't impact the breakfast dates
if future events bump the event I only need to add them to the OMIT list at the top
by having a local OMIT list as well (consisting of the non-Saturdays) the AFTER postpones to the following Saturday rather than pushing it to the next day (Sunday)

In reality, I also use CLEAR with the kids' school calendars too because not every federally-recognized holiday closes the schools so I find it easiest to just hard-code all the days-off for each school year.

Frankly, as a guideline, I lean towards putting every OMIT inside its own PUSH/CLEAR/POP block.

Omit function

On occasion you might find it easier to describe the days you want to OMIT with a function. remind lets you create a such a function that takes a date as the only argument, and returns 0 if it should consider the date, or returns non-0 if it should treat the date as omitted. Use the OMITFUNC modifier to tell remind to ignore the local and global OMITs and use the OMITFUNC instead.

FSET myomit(d) …
REM OMITFUNC myomit MSG Some event

~/.reminders

OMIT

isomitted()

OMITFUNC

FSET near_a_new_moon(d) \
  moonphase(d) < 20 || \
  moonphase(d) > 340
FSET allmyomits(d) \
  wkdaynum(d) == 0 || \
  wkdaynum(d) == 6 || \
  isomitted(d) || \
  near_a_new_moon(d)

REM OMITFUNC allmyomits AFTER \
  MSF This doesn't happen \
  on weekends, \
  OMIT dates, \
  or around a new moon.

~/.reminders

moonphase()

near_a_new_moon

Using `OMIT` for more complex dates

In certain cases you might want to create an OMIT that falls on certain days that otherwise work with regular reminders. However if you try to do something like

OMIT Mon Feb 15 MSG President's Day

~/.reminders

remind

OMIT

trigdate()

OMIT

REM Mon Feb 15 MSG President's Day
OMIT [trigdate()]

~/.reminders

this might have surprising side-effects if you don't take care where you start your search

Using `SATISFY` to conditionally test events

Sometimes you want remind to perform additional tests on a date to determine whether an event should occur, and if not, continue seeking the next matching date and test that. The SATISFY clause lets you evaluate an expression to do just that. You might only want something to happen every N^th month or test that something only occurs on certain days. As previously suggested when discussing the N^th weekday of a month you might need to check that the date falls in the same month.

REM 1 SATISFY [$Tm % 3 == 1] \
  MSG The first of every 3rd month starting January
REM 1 SATISFY [$Tm % 3 == 2] \
  MSG The first of every 3rd month starting February
REM 1 SATISFY [$Tm % 3 == 0] \
  MSG The first of every 3rd month starting March

REM MAYBE-UNCOMPUTABLE \
  SATISFY [$U == realtoday()] \
  MSG Only appears today
REM [realtoday()] \
  MSG Also only appears today

REM 13 SATISFY [$Uw == 5] MSG Friday the 13th

REM Mon 29 SATISFY [monnum($T - 7) = $Tm] \
  MSG Infrequent Fifth Monday of the month

PUSH
CLEAR
⋮
OMIT Dec 23 2019 THROUGH Jan 6 2020 \
  SATISFY [$Uw == 1] \
  MSG %"No club%"
⋮
REM Mon \
  FROM Aug 1 2019 \
  UNTIL May 31 2020 \
  OMIT SKIP \
  MSG Club
POP

~/.reminders

The first three reminders trigger on the first of each month. However remind will test the month-number ($Um), dividing it by 3 and getting the remainder ("%" does division and results in the modulus/remainder). If the remainder is 1, we have January/April/July/October; if the remainder is 2, we have February/May/August/November; if the remainder is 0, we have March/June/September/December.
The second two reminders test if the currently-considered date (today(), the same as $U) matches the real-world date (realtoday()) and only displays the MSG portion today.
To find Friday the 13^th, we can't use REM Fri 13 because this finds the first Friday on or after the 13^th of each month. That might or might not coincide with the 13^th. Instead, we find all the 13^ths and then use the SATISFY clause to reject those that don't fall on a Friday (0=Sunday, …, 5=Friday, 6=Saturday)
To find the 5^th Monday we need to use the same procedure for finding the other N^th weekdays of the month (Mon 29) but then check that the month of the resulting date ($Um) falls in the same month (monnum()) as the date seven days before (-7) today's date ($U, the date currently under consideration).
In the final pair of entries, I have a club meeting every Monday night from August through May. However, club takes off during Christmas break (and some other days I've not included here), so the OMIT takes care of this. However, I don't want notifications on every day during the break telling me not to attend club. I only want notifications on the Mondays during the break. Using the SATISFY clause lets me limit these so they only fall on Mondays ($Uw == 1).

Fine-grained scheduling

Sometimes you want advanced warning of an event at certain intervals like ++n provides, but you don't want those warnings on every day during that interval. The WARN modifier lets you provide a function that tells remind which days you want reminders. The function gets called repeatedly first with 1, then with 2, etc. The search stops if one of the following conditions occur:

remind encounters the current number of days out for the next instance of this reminder
it encounters a zero (today)
the function results cease decreasing monotonically (they must continue decreasing, so "10, 5, 7, 0" will bail at 7 because it went up from 5)

choose()

FSET mywarnfunc(i) choose(i, 30, 10, 5, 2, 0)
REM Jan 1 2020 WARN mywarnfunc \
  MSG Get car inspected

~/.reminders

remind

mywarnfunc

choose()

^st

remind

mywarnfunc

choose()

^st

remind

^st

OMIT

FSET mywarnfunc(i) choose(i, 30, -10, 5, -2, 0)
REM Jan 1 2020 \
  WARN mywarnfunc \
  MSG Get car inspected

~/.reminders

OMIT

remind

SCHED

AT

FSET mynotice(i) choose(i, 4*60, 2*60, 30, 0)
REM January 1 2020 AT 10:00 \
  SCHED mynotice \
  MSG Doctor

~/.reminders

PRECISE SCHEDULING

man

Miscellaneous tricks

As I've built up my remind files, I've developed a collection of tricks and tips that I use for various scenarios.

Labels

Because I have a file for each type of reminder, I find it handy to prefix each item with its filename using the special msgprefix(x) function (which I also use for colorizing agenda reminders):

# fileprefix() reduces the current
# path+filename+extension
# to just the filename
# /path/to/file.rem -> "FILE"
FSET fileprefix() upper( \
  substr( \
    filename(), \
    strlen(remind_dir)+1, \
      strlen(filename())-4 \
      ) \
  )
FSET msgprefix(x) fileprefix() + ": "

reminders.rem

Friday, 27th December, 2019 (today):
USHOL: Chanukah 5
BIRTHDAYS: David (10yo) in 3 days' time
BIRTHDAYS: Annabelle (12yo)
BIRTHDAYS: Nate today
MANPAGES: man 1 ed
SCHOOL: Christmas: No School
FINANCES: Pay gas bill (today)
HOUSEHOLD: Garbage day (Delayed) today

msgprefix(x)

SET topic ""
FSET msgprefix(x) iif(strlen(topic) > 1, topic + ": ", "")
FSET born(y) "(" + ($Uy-y) + "yo)"
FSET married(y) (ord($Uy-y)) + " anniversary"
SET topic "Birthdays"
REM Jan 1 MSG %"Mom%" [born(1950)]
REM Jan 6 MSG %"Dianne%" [born(1970)]
SET topic "Anniv"
REM Apr 1 +7 MSG %"Bob & Alice [married(1999)]%" %b
SET topic "Work"
REM Fri AT 7:30 MSG %"Meeting%" %2

~/.reminders

Color

As a rule, I generally want plain-text reminders, whether emailing them to myself or displaying a calendar with rem -c. But when at the command-line, I like a bit of color.

General per-reminder foreground color

As one of the SPECIAL codes, remind offers COLOR (or it will happily accept "COLOUR") for reminders that alters the color used when generating a week/month calendar with -cc+ or -cc. The entry takes three numbers as arguments, the Red, Green, and Blue values, each ranging from 0 to 255, followed directly by the MSG-style text (without the MSG keyword):

REM Jan 1 SPECIAL COLOR 255 0 255 \
  Happy New Year in bright magenta

~/.reminders

This makes it easy to colorize a single reminder, but it only works in week/month view (or external apps that support it via the -p option), not the agenda view (unless you have version 3.3.0 or later where the -@ option enables color in the agenda view). However sometimes I want to colorize whole swaths of reminders and do it on the agenda view. I want all my birthday and anniversary reminders in green, my personal reminders in blue, my wife's reminders in bright magenta, the kids' reminders in cyan, church/volunteering reminders in magenta, work in yellow, and some low-priority entries in gray. I don't want to have to specify the color for each and every one of my 1,600+ reminders.

Prior to the addition of `$DefaultColor` in version 3.3.0

To do this, I first add a number of ANSI escape codes as constants to my helpers.rem that I can then use elsewhere.

⋮
# ANSI colors
SET Esc CHAR(27)
SET Nrm Esc + "[0m"
SET Blk Esc + "[0;30m"
SET Red Esc + "[0;31m"
SET Grn Esc + "[0;32m"
SET Ylw Esc + "[0;33m"
SET Blu Esc + "[0;34m"
SET Mag Esc + "[0;35m"
SET Cyn Esc + "[0;36m"
SET Wht Esc + "[0;37m"
SET Gry Esc + "[30;1m"
SET BrRed Esc + "[31;1m"
SET BrGrn Esc + "[32;1m"
SET BrYlw Esc + "[33;1m"
SET BrBlu Esc + "[34;1m"
SET BrMag Esc + "[35;1m"
SET BrCyn Esc + "[36;1m"
SET BrWht Esc + "[37;1m"
⋮

helpers.rem

Then, in my base reminders.rem file, I have the following:

IF defined("COLOR")
  SET bannercolor BrWht
  SET calcolor Nrm
  SET labelcolor Nrm
  # if desired, use the priority "x"
  # to change the prefix color
  FSET prefixcolor(x) Nrm
  FSET color() calcolor
  FSET msgsuffix(x) Nrm
ELSE
  SET bannercolor ""
  SET calcolor ""
  SET labelcolor ""
  FSET prefixcolor(x) ""
  FSET color() ""
ENDIF

FSET msgprefix(x) \
  prefixcolor(x) + \
  fileprefix(x) + \
  ": " + \
  color()

IF defined("subsequent_iteration")
  BANNER [labelcolor]=====================[bannercolor]%_%w, %d%s %m, %y%o:%
ELSE
  BANNER [bannercolor]%w, %d%s %m, %y%o:%
  SET subsequent_iteration 1
  PRESERVE subsequent_iteration
ENDIF

reminders.rem

(the section on labels provides the fileprefix function)

COLOR

I then set a color at the top of each of my files after including the helpers.rem and it changes the color of everything following, allowing me to have multiple color blocks in the same file

INCLUDE [filedir()]/helpers.rem
SET calcolor Ylw
REM 1 MSG %"Send invoice%"
⋮

work.rem

INCLUDE [filedir()]/helpers.rem
SET calcolor BrBlu
REM Jan 10 MSG %"Alice%"
REM Jan 14 +7 MSG %"Bob [born(1980)]%" %b
SET calcolor Blu
REM Jan 6 MSG %"Dianne's birthday%"
⋮

birthdays.rem

msgprefix()

msgsuffix()

remind

set the prefix color for the labels
determine and display the file-name-based prefix
switch to whatever color calcolor currently contains
display the reminder text
reset back to a normal color

I can then use rem -iCOLOR=1 to get my results in color. (ignore the fact that the events actually fall on different dates)

BIRTHDAYS: Alice
BIRTHDAYS: Bob (39yo)
BIRTHDAYS: Dianne
WORK: Send invoice

calprefix()

calsuffix()

After the addition of `$DefaultColor` in version 3.3.0

I submitted a patch that has (with modifications) made it into the mainline codebase in version 3.3.0, released January 31^st, 2020. This patch allows you to set a $DefaultColor variable combining the benefits of the COLOR keyword (appearing on the week/month calendars and getting passed to external utilities) with the benefits of my calcolor/msgprefix()/msgsuffix() trick above (colorizing the agenda view, without needing to specify the color for every single event).

⋮
SET $DefaultColor "0 255 0"
# birthdays for which I take some sort of action
REM Jan 10 MSG %"Alice%"
REM Jan 14 +7 MSG %"Bob [born(1980)]%" %b
SET $DefaultColor "0 128 0"
# birthdays I want to know about
REM Jan 1 MSG %"Eric's Mom's birthday%"
SET $DefaultColor "-1 -1 -1"
# Other misc. birthdays in the original default color
REM Jan 6 MSG %"Dianne's birthday%"
⋮

birthdays.rem

remind

-@

Raw output

While pretty output works well for humans, sometimes I desire to pipe output to another program. I have no interest in trying to parse the week or month calendar views. The agenda view can serve as a primitive & fragile output format for post-processing. However, remind provides the -p and -s options to produce machine-readable output. This output format works much better with tools such as grep, sed, awk, and python/perl/ruby, as well as the tkremind program. remind also allows your reminders to convey additional metadata to these back-ends. Given this reminder:

REM 2020-01-29 \
  AT 8:30 DURATION 3:15 \
  TAG work \
  TAG home \
  SPECIAL COLOR 255 128 0 \
  my message

~/.reminders

user$ rem -s 2020-01-29
2020/01/29 COLOR work,home 195 510 255 128 0 8:30-11:45am  my message
user$ rem -p 2020-01-29
# rem2ps begin
January 2020 31 3 0
Sunday Monday Tuesday Wednesday Thursday Friday Saturday
December 31
February 29
2020/01/29 COLOR work,home 195 510 255 128 0 8:30-11:45am  my message
# rem2ps end

-s

The date in YYYY/MM/DD format
The SPECIAL type (in this case COLOR)
a comma-separated list of TAGs associated with the reminder
the duration in minutes
the start-time in minutes-from-midnight (510 = 60 minutes * 8 hours)
the $RED, $GREEN, and $BLUE values from the COLOR (255, 128, and 0 here)
the body of the message, prepended with the time (if present)

-p

a literal # rem2ps begin
a line showing the month currently rendering, the current year, the current month, the number of days in the month, the week-day (0=Sunday, 1=Monday, … 6=Saturday) of the first day of the month, and a boolean 0/1 based on whether the calendar should display Monday first (based on the "-m" option passed to remind)
the days of the week in the local language
the next month and its number of days
the previous month and its number of days
the list of reminders like -s produces
a literal # rem2ps end

# rem2ps end

*

`SPECIAL` reminders

While I don't give much space to them, remind's SPECIAL token lets you pass out-of-band information to external back-ends. This can include things like changing the color of an entry (with SPECIAL COLOR $RED $GREEN $BLUE), shading the background of a calendar square (with SPECIAL SHADE $GRAYLEVEL or SPECIAL SHADE $RED $GREEN $BLUE), drawing the phase-of-the-moon (with SPECIAL MOON $PHASE), or adding per-week annotations (with SPECIAL WEEK $TEXT). I use rem2ps in the example regarding relative dates but don't employ any of the SPECIAL commands therein.

`RUN` reminders

The RUN message-type acts much like a MSG message-type except that after passing the body through the substitution filter it executes the resulting command instead of displaying the result. Beware of quoting and escaping issues. While you can do things like

REM Jan 31 +3 RUN echo Hello %b,

~/.reminders

echo Hello in 2 days' time

REM 1 ONCE RUN backup.sh [$U]

~/.reminders

backup.sh 2020-01-01

ONCE

remind

atime

/etc/fstab

noatime

ONCE

every

I keep a list of "to-do" items so I tried including those in my reminders:

REM SATISFY [$U == realtoday()] RUN head -5 ~/todo.txt

~/.reminders

I also experimented with keeping larger stretches of prose in an external file like

BEGIN: gifts
⋮
END: gifts
BEGIN: FakeCON
Flight:
 Southwest Flight 3141 DAL → BWI
 Departs: 1:41pm
 Arrives: 3:14pm
 Confirmation number: 1414
Car:
 Thrifty reservation# 15278-3955
Hotel:
 Marriott
 753 Old Oak
 Reservation: 20200115732
 Member#: 1783613807
END: FakeCON
BEGIN: other
⋮
END: other

annotations.txt

RUN

FSET external_annotation(marker) \
    "sed -n '/^BEGIN: *" + marker + "$/,/^END: *" + marker + "$/p' " + \
    filedir() + "/annotations.txt"
⋮
REM Feb 18 2020 RUN [external_annotation("FakeCON")]

~/.reminders

RUN

Tagging

Similar to the SPECIAL keyword, the TAG keyword lets you assign one or more tags to your reminder.

REM 15 TAG work MSG Run commission report
REM 1 TAG home TAG chores MSG Test smoke alarms
REM 1 TAG car TAG chores MSG Check oil levels

~/.reminders

remind

raw-output

write monthly letters to our grandparents

REM Jan 18 2020 TAG letter \
  MSG Date to see the new Jumanji movie

~/.reminders

-s

user$ rem -s 2020-01-01 |
>  awk '$3 ~ /letter/'

using a RUN directive

REM 1 MSF [iif($Tm % 2, "Honey", "Hubby")] \
  writes %"Grandparent letter%"
REM 1 RUN rem -sr [$U - 1] | \
  awk '$3 ~ /letter/{for (i=2; i<6; i++) $i=""; print}'

~/.reminders

-r

remind

RUN

[$U - 1]

remind

I wish remind had more native functionality to support tags such as a trigtags() function (to return the tags for the current reminder), or a hastag() function (to ask if the current reminder has a given tag). This would help me filter output or expand tags in message-bodies. Alas, not yet. I look forward to some day (hopefully) when these are even more useful. Perhaps for my next patch.

Shell tips & tricks

Aliases

I've set up a couple bash aliases in my shell to minimize typing:

⋮
alias 1='rem -gaad -iCOLOR=1'
for d in {2..6} ; do
    alias ${i}='rem -gaad -iCOLOR=1 "*"'${i}
done
⋮

~/.bash_aliases

3

Crontab

I have crontab entries that email me my today/tomorrow reminders each day and a full week of reminders on Sunday morning:

@daily rem -g '*2'
45 6 * * 0 rem '*7'

crontab

Version control

I find it useful to keep my reminder directory, ~/.config/remind/, in version control. I use git for multiple files, but you could just as easily use mercurial, Subversion, or even CVS or rcs. This gives me a lot of freedom to experiment with complex reminder syntax. If I mess something up, I can always revert to the known-good state. Also, I can delete reminders that have passed so they don't clutter my calendar or increase processing time, but still have historical record of their existence. I can also git push & git pull changes between different machines to keep my reminders in sync.

user$ cd ~/.config/remind
user$ git init .
user$ git add *.rem
user$ git commit -m "Initial checkin"
⋮
modify reminder files
⋮
user$ git commit -am "Added John's birthday"

git

user$ cd
user$ mkdir RCS
user$ echo "Initial checkin" | ci -l .reminders
user$ ed -p"* " .reminders
3141
* a
REM Apr 1 MSG Tom's birthday%
.
* wq
3170
user$ echo "Added Tom's birthday" | ci -l .reminders

rcs

Speech

The readable nature of the output makes it easy to pipe output to a program like espeak (or another text-to-speech engine which you would also have to install) to read your reminders aloud. You might create a custom spoken.rem file that you don't INCLUDE in your regular reminders.rem in which you remove the BANNER, emit section dividers (if you don't choose to use display labels), and INCLUDE a select subset of calendars:

BANNER %
MSG Birthdays%
INCLUDE [filedir()]/birthdays.rem
MSG Personal items%
INCLUDE [filedir()]/tim.rem
MSG Work items%
INCLUDE [filedir()]/work.rem

spoken.rem

[filedir()]

spoken.rem

And to read them:

user$ remind ~/.config/remind/spoken.rem | espeak

Easier or more boring recipes

Car registration

We need to get our car inspected and get our (re)registration submitted by the end of February so I want notifications starting at the beginning of February:

REM Mar 1 --1 ++[26 + isleap($Uy)] \
  SATISFY [$Uy > 2019] \
  MSG %"Car reg./inspection due%" %b

household.rem

Mar 1 --1

++[26 + isleap($Uy)]

SATISFY

Maturing bonds

I purchased a couple simple-interest bonds and want to know when to expect the interest payment checks. They arrive around the 25^th of the month (or AFTER if it falls on a Sunday when the USPS doesn't deliver mail), every 6 months until the end of the bond-term. But remind makes this pretty easy:

REM 25 UNTIL Feb 26, 2025 \
  OMIT Sun AFTER \
  SATISFY [$Tm % 6 == 2] \
  MSG %"Expect bond check for $31.41%"

finances.rem

SATISFY

^th

$Tm % 6 == 2

Notice until action

I send birthday cards to certain friends & family members so I want about a week of notice to get a card, write a note, and get it in the mail. However, once I've sent the card, I don't need further advanced warning reminders. By updating that year in the expression once I've completed the task I no longer get advanced warning for the given year. Similar to the car registration example, in this case I still want to receive notification on the day of the birthday too, not drop the reminder entirely.

FSET last_sent(y) iif($Uy > y, 7, 0)
⋮
# once I've sent a card to Grandma in 2020
# I'll change the "2019" to "2020":
REM Jul 6 +[last_sent(2019)] MSG %"Grandma%" %b

birthdays.rem

+

Messages tweaked based on date

As a Christmas gift to our grandparents, we take turns hand-writing a letter to them each month. So on the first of each month, I want a reminder, but also want to know whose turn:

REM 1 MSF [iif($Tm % 2, "Honey", "Hubby")] \
  writes %"Grandparent letter%"

household.rem

REM Sat MSF Scrub %"[ \
  choose((weekno() % 3)+1, \
  "Kitchen", \
  "Master", \
  "Kids'") \
  ] grout%"

household.rem

Selectively getting advanced notice

For friends I see at church on Sunday I would like advance notice of their birthday on Sunday but don't want additional reminders during the intervening days of the week. For this, I use an expression evaluation to dynamically determine the number of days worth of notice:

FSET church_notice() iif($Uw == 0, 6, 0)
⋮
REM Jan 16 ++[church_notice()] MSG %"Tony %c%"

~/.reminders

^th

Annoying credit-reporting company rules

I learned the hard way that the credit-reporting companies will not let you order your credit report on the same day each year because a full year hasn't actually passed. Yes, I'd love to pull my free credit report on the same day of each year. However, I need to pull them more than 365 days (366 on a leap-year) apart. So I use the repeat modifier to get reminded every 367 days. I also intersperse them throughout the year rather than all at once so that I get a more frequent sampling in case something has gone wrong:

REM Jan 1 2019 *367 MSF Order credit report: %"Equifax%" \
  (1-877-322-8228)%_\
  https://www.annualcreditreport.com/index.action
REM May 1 2019 *367 MSF Order credit report: %"Experian%" \
  (1-877-322-8228)%_\
  https://www.annualcreditreport.com/index.action
REM Sep 13 2019 *367 MSF Order credit report: %"TransUnion%" \
  (1-877-322-8228)%_\
  https://www.annualcreditreport.com/index.action

~/.reminders

Post-holiday candy sales

I tend to stock up on chocolate when it goes on sale after some of the big candy holidays. (I keep stashes of chocolate hidden around the house so that when my beloved has a hankering for a certain type, I can often play the miracle-worker and bring it out of hiding). So I use remind to make sure I don't forget:

REM Feb 15 MSG %"Post-Valentine candy sales%"%
REM [easterdate($Uy) + 1] MSG %"Post-Easter candy sales%"%
REM Nov 1 MSG %"Post-Halloween candy sales%"%
REM Dec 26 MSG %"Post-Christmas candy sales%"%

~/.reminders

remind

easterdate()

Did any of these reminders trigger?

Sometimes you have several events and you want to provide some SPECIAL modifier for all of them. You can use remind's special $NumTrig variable to check this:

SET n $NumTrig
REM Sat Sun SATISFY 1
REM Feb 14 MSG Valentine's Day
REM [easterdate()] MSG Easter
IF $NumTrig > n
  # At least one triggered
  REM SPECIAL SHADE 75
ENDIF

~/.reminders

$NumTrig

IF

SPECIAL SHADE

FizzBuzz

Often in interviews the interviewer will instruct the candidate to whiteboard the classic fizzbuzz solution where the program emits n numbers but if the number is divisible by 3 print "fizz", if the number is divisible by 5 print "buzz", and if the number is divisible by 3 & 5 print "fizzbuzz" (otherwise just printing the number). So in case you ever need to do this using remind:

FSET doy() $U - date($Uy-1, 12, 31)
MSG [iif(doy() % 15 == 0, \
  "fizzbuzz", \
  iif(doy() % 5 == 0, \
    "buzz", \
    iif(doy() % 3 == 0, \
      "fizz", \
      doy() \
      ) \
    ) \
  )]

~/.reminders

doy()

$U

^st

More complex calendar recipes

Shifting garbage day

The go-to for demonstrating the power of remind involves shifting your garbage collection day back if a holiday occurs earlier in the week:

PUSH
CLEAR

INCLUDE [filedir()]/holidays.rem

SET garbday 4
SET garbdayname wkday(garbday)
FSET _garbhol(x) wkdaynum(x) == garbday \
  && slide(x - garbday, garbday) != x
FSET _garbdel() iif($Tw != garbday, " (Delayed)", "")
REM [garbdayname] +1 AFTER OMITFUNC _garbhol \
  MSG %"Garbage day[_garbdel()]%" %b%
POP

~/.reminders

garbday

OMIT

holiday.rem

PUSH

CLEAR

POP

remind

Pretty date-diff

While remind lets you subtract two dates to get the number of days between them, sometimes you want that difference expresses as a number of years, months, and days. To do that, I have this ymd_diff function which returns values like "3y 2m 11d" or "3y 1m 14d ago" which you might find more useful:

FSET ymd_diff_gt(d1, d2) \
  iif(year(d2) > year(d1), \
    (year(d2) - year(d1)) - ( \
      (monnum(d2) < monnum(d1)) \
      || \
      (monnum(d2) == monnum(d1) && day(d2) < day(d1)) \
      ), \
    0 \
    ) + "y " + \
  iif(monnum(d2) > monnum(d1), \
    (monnum(d2) - monnum(d1)) - (day(d2) < day(d1)), \
    iif(monnum(d2) < monnum(d1), \
      (12 + monnum(d2) - monnum(d1)) - (day(d2) < day(d1)), \
      iif(day(d2) < day(d1), 11, 0) \
      )) + "m " + \
  iif(day(d2) >= day(d1), \
    day(d2) - day(d1), \
    (daysinmon(monnum(d1), year(d2)) + day(d2)) - day(d1) \
    ) + "d"
FSET ymd_diff(d1, d2) \
  iif(d1 > d2, ymd_diff_gt(d2, d1) + " ago",\
  ymd_diff_gt(d1, d2) \
  )

~/.reminders

Generating reminder files programmatically

One of my calendars gives me a daily man-page to read based on my list of system man-pages looking something like

SET ManStart date(2020,1,1)
REM [ManStart + 0] MSG man 5 %"fstab%"%
REM [ManStart + 1] MSG man 1 %"rs%"%
REM [ManStart + 2] MSG man 1 %"awk%"%
REM [ManStart + 3] MSG man 6 %"cribbage%"%
⋮

manpages.rem

user$ cat remify_manpages.awk
BEGIN {
 print "SET ManStart date(2020,1,5)"
}
$7 ~ /^[1678]$/ {
 printf("REM [ManStart + %i] MSG man %s %%\"%s%%\"%%\n", NR - 1, $7, $6)
}

user$ find /usr/share/man/man[1678]/ -maxdepth 2 -type f \
>  | grep -vi perl \
>  | sort -R \
>  | awk -F'[./]' -f remify_manpages.awk > manpages.rem

man

/usr/share/man/man[1678]/

perl

awk

remind

Relative date calculations for event-planning

I serve on the leadership team for a local non-profit that works to provide clothing-vouchers to kids on the school free/reduced-lunch program. However the calendar changes each year based on certain fixed dates including when the school counselors have their training before school starts, when school starts, and which weekends the shopping event(s) take(s) place. I use SET to define the fixed-dates, and calculate the other dates based off them.

SET CounselorTraining date(2019,7,31)
SET SchoolStart date(2019,8,15)
SET Event date(2019,10,19)

SET ApplicationsGoHome (Event - 47)

REM [CounselorTraining] MSG %"Attend counselor training%"
REM [SchoolStart] MSG %"First day of school%"
REM [ApplicationsGoHome - 3] \
  MSG %"Remind counselors applications will be going home%"
REM [ApplicationsGoHome] \
  MSG %"Applications go home with kids%"
REM [ApplicationsGoHome] \
  MSG %"School calls families to let them know to watch for applications%"
REM [Event - 37] \
  MSG %"Remind counselors that applications must be back [Event - 33]%"
REM [Event - 33] MSG %"Check funding levels%"
REM [Event - 32] MSG %"Pick up applications from school%"
REM [Event - 32] THROUGH [Event - 26] MSG %"Data entry%"
REM [Event - 10] MSG %"Print acceptance letters%"
REM [Event - 8] MSG %"Email counselors to expect letters%"
REM [Event - 8] MSG %"Acceptance letters go home%"
REM [Event - 5] MSG %"Email volunteers a reminder%"
REM [Event] MSG %"Event%"
REM [Event + 2] MSG %"Email volunteers to thank%"

~/.reminders

OMIT

BEFORE

rem -p3 2019-08-01 | rem2ps | ps2pdf - calendar.pdf

Medicine schedule

One of our kids takes a medication that we receive in a 30-day supply. This means we need to pick it up at the pharmacy at least the day before to ensure uninterrupted dosing. The pharmacy occasionally needs a business day or two to get the medication in, so I like to drop off the prescription a couple days early. Which means I need to pick up the prescription at the doctor's office before that. Which means I need to call the doctor's office 72hr before that. And all of those dates can get shifted by weekends and holidays. On a less-powerful calendar, I'd schedule a reminder every 30 days and then hope I remember to take weekends and holidays into consideration.

On the other hand, remind makes this fairly straightforward to express. While doing this, I'd also like to get nagging reminders and advanced notice if I haven't yet picked up the pills. But once I have, I don't want continued reminders. First we'll begin by creating an OMIT context to hold the holidays that the pharmacy & doctor's office close.

PUSH
CLEAR
# the Dr. and pharmacy close on these dates
⋮
OMIT Dec 24
OMIT Dec 25
⋮
# reminders will go here
⋮
POP

~/.reminders

⋮
SET MostRecentlyGotPills date(2020, 1, 24)
# Started taking the medication on
SET MedStartDate date(2019, 3, 18)
# we get 30 pills
SET MedFreq 30
# Dr. office requests 72hr of notice
# but remind needs it in days
SET RXLeadTime (72/24)
⋮

~/.reminders

⋮
REM [MedStartDate] *[MedFreq] SATISFY 1
SET PillsRunOut trigdate()
⋮

~/.reminders

SATISFY 1

trigdate()

From here, we know that we need to pick up the pills the day before then (possibly more than one day before if there's an intervening holiday or weekend):

⋮
# need to pick up meds at least one day before we run out
REM [PillsRunOut] -1 +1 \
    OMIT Sun BEFORE \
    SATISFY [$U > MostRecentlyGotPills] \
    MSG %"Pick up pills%" at pharmacy %b
⋮

~/.reminders

-1

OMIT

SATISFY

+1

Once we know when we need to pick up the pills, we can capture that date (again, with trigdate()) and use it to determine when we need to pick up the prescription and take it to the pharmacy (the day before, but the doctor's office doesn't open for prescriptions on weekends or over holidays so it might get bumped back further).

⋮
SET RealPillPickUpDate trigdate()
REM [RealPillPickUpDate - 1] +1 \
    OMIT Sat Sun BEFORE \
    SATISFY [$U > MostRecentlyGotPills] \
    MSF %"Pick up Rx%" at Dr. %b \
    and take to pharmacy
⋮

~/.reminders

⋮
SET RealRxPickUpDate trigdate()
REM [RealRxPickUpDate] -[RXLeadTime] \
    OMIT Sat Sun BEFORE \
    SATISFY [$U > MostRecentlyGotPills] \
    MSF %"Call Dr. office to request Rx refill%" %b
⋮

~/.reminders

SATISFY

This produces a final series of variables and reminders that notify me at appropriate times and do all the holiday/weekend math so we don't accidentally end up without medication:

################
# Update here: #
################
SET MostRecentlyGotPills date(2020, 1, 24)

###################
# Configure here: #
###################
SET MedStartDate date(2019, 11, 27)
# we get 30 pills
SET MedFreq 30
# Dr. office requests 72hr of notice
# but remind needs it in days
SET RXLeadTime (72/24)

PUSH
CLEAR
# the Dr. and pharmacy close on these dates
OMIT Jan 1
⋮
OMIT Dec 24
OMIT Dec 25

#########################

# when's the next occurrence?
REM [MedStartDate] *[MedFreq] SATISFY 1
SET PillsRunOut trigdate()

# need to pick up meds at least one day before we run out
REM [PillsRunOut] -1 +1 \
    OMIT Sat Sun BEFORE \
    SATISFY [$U > MostRecentlyGotPills] \
    MSG %"Pick up pills%" at pharmacy %b
SET RealPillPickUpDate trigdate()
REM [RealPillPickUpDate - 1] +1 \
    OMIT Sat Sun BEFORE \
    SATISFY [$U > MostRecentlyGotPills] \
    MSF %"Pick up Rx%" at Dr. %b \
    and take to pharmacy
SET RealRxPickUpDate trigdate()
REM [RealRxPickUpDate] -[RXLeadTime] \
    OMIT Sat Sun BEFORE \
    SATISFY [$U > MostRecentlyGotPills] \
    MSF %"Call Dr. office to request Rx refill%" %b
POP

~/.reminders

MostRecentlyGotPills

MedStartDate

MedFreq

RXLeadTime

Scanning forward from an earlier date

Sometimes you have an OMIT that has happened recently and would thus impact how advanced notice with + and adjusting dates with BEFORE/AFTER/SKIP calculations occur, but because remind first determines the OMIT dates based on the "today" in question, it might not adjust properly. For example, consider a floating holiday like Labor Day (the first Monday in September):

REM Mon 1 Sep SATISFY 1
OMIT [$T] MSG Labor Day

~/.reminders

September 7^th, 2020

next

September 6^th, 2021

SCANFROM

REM Mon 1 Sep SCANFROM [$U-7] SATISFY 1
OMIT [$T] MSG Labor Day

~/.reminders

^th

# Start looking for July 4th
# beginning 7 days ago:
REM 4 July SCANFROM [$U - 7] SATISFY 1

# if it fell on a Saturday
# observed on the preceding Friday
IF wkdaynum(trigdate()) == 6
  REM [trigdate()] MSG Independence day (actual)
  OMIT [trigdate()-1] MSG Independence day (observed)
ELSE
  # if it fell on a Sunday
  # observed on the following Monday
  IF wkdaynum(trigdate()) == 0
    REM [trigdate()] MSG Independence day (actual)
    OMIT [trigdate()+1] MSG Independence day (observed)
  ELSE
    # otherwise, observed=actual
    OMIT [trigdate()] MSG Independence day
  ENDIF
ENDIF

~/.reminders

defs.rem

School N-day cycles ignoring holidays & weekends

Sometimes you want to know the number of days between two dates, without counting holidays or other OMIT dates. This often shows up in school schedules where a 4- or 6-day cycle often rotates across the standard 5-day week. The nonomitted() function lets you do this with minimal fuss. However, the start-date (the first parameter) must be less-than or equal to the end-date or remind will produce an error.

SET FirstDayOfSchool date(2019, 8, 14)
# a 4-day cycle
SET SchoolPhase 4
PUSH
CLEAR
# list of school holidays
# and in-service days:
OMIT Jan 1
⋮
OMIT Dec 23 2019 THROUGH Jan 6 2020 \
  SATISFY [$Uw > 0 && $Uw < 6] \
  MSG %"Christmas: No school%"%
⋮
OMIT Mar 13 2020 MSG %"In-service: No school%"%

IF $U >= FirstDayOfSchool
  SET DayNumber nonomitted(  \
    FirstDayOfSchool, \
    $U, \
    "SAT", "SUN" \
    )
  SET schedule DayNumber % SchoolPhase + 1
  REM Aug 13 2021 AT 7:15 THROUGH May 21 2020 \
    SKIP \
    OMIT Sat Sun \
    MSG %"School%" %2 ([choose( \
      schedule, \
      "music", \
      "art", \
      "language", \
      "health" \
      )])%
ENDIF

~/.reminders

Daemon mode

In addition to invoking remind manually or from scripts, remind also offers a "daemon mode" that will run in the background and fire off commands as AT events come around.

user$ remind -z5 -k'espeak %s &' ~/.reminders

~/.reminders

remind

~/.reminders

AT

remind

espeak

%s

xmessage

zenity

user$ remind -z5 -k'xmessage %s &' ~/.reminders

~/.reminders

xmessage

notify-send

user$ remind -z5 -k'notify-send %s &' ~/.reminders

~/.reminders

MSG

Your reminder file can use the $Daemon variable to test to see if remind is running in daemon mode:

REM Jan 5 2020 AT 10:30 MSG [ \
  iif($Daemon, "Hey, you! ", "") \
  ] Doctor's appointment

~/.reminders

use espeak to speak your reminders

Other utilities

Generating iCalendar/`.ics` files

Great. Now you have all your reminders generating just the way you want. How do you share them with other people? Most other cough*inferior*cough calendar programs still support .ics format files to describe events. You can use the rem2ics utility to convert the output of remind into a .ics file that you can import into Google Calendar, Outlook, or iCalendar. Note: While it will export all of your reminders including repeating reminders, it will not include enough meta-information to preserve things like repeats, so the events show up in the external calendar as individual events.

user$ rem -s12 |
>  TZ=CST8CDT rem2ics -do >reminders.ics

reminders.ics

Other external utilities

Outside the scope of this article you'll find things like tkremind (a GUI front-end to remind) and wyrd (a TUI front-end to remind). I don't currently use either of these so I'll leave those for the reader to explore.

Troubleshooting & debugging

While the vast majority of my reminders pose no issues, occasionally one ends up complex enough that I want to make sure it really works like I expect it to. Because these really are code, I appreciate having some tools to debug them.

Echoing into `remind -`

As a first line of debugging, I usually try to isolate the single reminder and then run remind with the date I want to test. If you specify a filename of "-", remind will read reminders from stdin which lets you echo a single reminder to test it:

user$ echo 'REM 1 --1 MSG Last' |
>  remind - 2020-1-31

echo

user$ (echo 'OMIT Dec 31'
>  echo 'REM 1 --1 MSG Last'
>  ) | remind - 2020-12-31

user$ echo REM 1 --1 MSG Last \
>  remind - 2020-12-31 '*33'

user$ for m in {1..12}
>  do
>  echo "REM 1 --1 MSG Last" |
>  remind - 2020-${m}-30
>  done

Using a custom `.reminders` file

Sometimes you really do need a lengthier test case. For these, I back-up my ~/.reminders file,

user$ cp ~/.reminders{,_orig}

user$ $EDITOR .reminders # create test case

user$ mv ~/{.,working_}reminders

user$ mv ~/.reminders{_orig,} # restore the original

$EDITOR

~/working_reminders

Alternatively, I also have my machines configured with a "guest" user, so sometimes it ends up faster to log in as the "guest" (who has no ~/.reminders file of consequence), blow away any previous test ~/.reminders file, and test there on an isolated case without risk to my real reminders.

Showing LOTS of dates

Sometimes I find it easiest to look at a spew of calendars and eyeball the results. I like to do this in month view, producing a whole year and paging through the results with my $PAGER (less in my case).

user$ rem -c12 -w$COLUMNS 2020-1-1 | less

-p/-s

user$ rem -p12 2020-1-1 | less

Printing values in `MSG` output

In the same classic tradition of printf() debugging, I also find it useful to see what remind thinks a variable holds:

SET SomeVar some complex expression
REM … MSG SomeVar=[SomeVar] on [$T]/[$U]

~/.reminders

Debugging flags

For really getting under the hood remind has a -d option that turns on additional debugging output. It gets verbose quickly, but invoking

user$ rem -dextvlf 2>&1 | less

man

x

t

v

Wrap up

When I first heard about remind it sounded interesting, but I tried multiple times before it finally clicked for me (much like git which took me about three serious attempts before I felt like I understood it). But once I figured out that I could express things that no other calendar program let me do, I have trouble keeping my primary calendar in anything else. I use rsync and git to keep my calendar in sync across multiple machines without exposing my calendar info to third-parties like Google or Microsoft. Hopefully this post gives you the motivation to evaluate it for your hardest calendar test-cases.

And if you come up with strange questions, use-cases, or other recipes for using remind feel free to contact me via email, on the remind mailing list, or over on Twitter @gumnos and I'll do my best to help out.

Setting up a terminal screen-reader on OpenBSD

Tim Chase — Sun, 19 May 2019 23:52:20 GMT

This post walks through installing the screen-reader yasr and configuring it to use speech-dispatcher with a TTS engine.

This assumes you have successfully installed a stock OpenBSD system and that your user has doas access to install software.

Install required packages

First, install the requisite packages from the repos:

$ doas pkg_add speech-dispatcher espeak flite

The flite package is optional, but can provide you with an alternate set of voices.

Currently yasr isn't available as an OpenBSD package, but it's fairly easy to install. You should be able to obtain the source code from the yasr source repository.

$  git clone https://github.com/mgorse/yasr

and to build it

$ cd yasr/yasr
$ make

At some point, hopefully yasr will be in the package repositories, making it as easy as pkg_add yasr.

Ensure that sound works

If you haven't already, make sure that your OpenBSD box can play audio. You can check this by using

$ aucat -i somefile.wav

If you don't have any .wav files on your machine, speech-dispatcher provides /usr/local/share/sounds/speech-dispatcher/test.wav so

aucat -i /usr/local/share/sounds/speech-dispatcher/test.wav

should play a short flourish. If it is too quiet, use mixerctl to set the volume or try additional OpenBSD audio troubleshooting techniques.

Set up `speech-dispatcher`

To create a user configuration file in ~/.config/speech-dispatcher/ use

$ spd-conf

with the default output of "espeak" (you can later try changing this to "flite" but I leave that as an exercise for the reader) and output via libao. You can also set the default output speed. I prefer it a little faster than the default but it might be good to start with the default and then tune it later. As you configure, it should offer you the option to run some tests. If you choose "yes" for the first test, you should hear "Speech dispatcher works." I recommend skipping the other tests. With this in place, you should now have a ~/.config/speech-dispatcher/ directory with various configuration settings within.

The yasr screen-reader requires speech-dispatcher to listen on an inet socket rather than a Unix socket, so edit the config file with your favorite text editor and uncomment/change the CommunicationMethod from "unix_socket" to "inet_socket"

$ ed ~/.config/speech-dispatcher/speechd.conf
/CommunicationMethod.*unix_socket
a
CommunicationMethod "inet_socket"
.

wq

This does open a local port (6560) allowing other processes on the same machine to connect to speech-dispatcher just so you're aware. If you choose to change the port, just make sure to also change it in the yasr configuration later.

Configure `yasr`

Copy the default yasr configuration file for modification:

$ cp ~/tmp/yasr-0.6.9/yasr.conf ~/.yasr.conf

Make the following changes:

Comment out all the synthesizer=… lines except the synthesizer=speech dispatcher line, uncommenting that one
Uncomment the synthesizer port=127.0.0.1:6560 line
Set the shell=… to your preferred shell. OpenBSD doesn't install bash at /bin/bash so you'll either want to use /bin/ksh or, (if you have bash installed) /usr/local/bin/bash, or your preferred default shell.

You can change other aspects of yasr in here, but for now, leave the defaults.

Testing

It took me a bit to figure out that speech-dispatcher listens for a connection but times-out (by default, 5 seconds) and then quits. So you need to invoke it and then immediately launch yasr

$ speech-dispatcher && ~/tmp/yasr-0.6.9/yasr/yasr

If all is working successfully it should start yasr and you should hear it read your prompt.

Getting started with `yasr`

The ~/.yasr.conf file lists all of the keyboard commands for both normal/interactive mode (the default) and for review mode. Keys are given by ASCII value written in hex.

Normal/interactive mode commands
Hex value(s)	Keyboard command	Action
0x01	Ctrl+A	say cursor
0x0c	Ctrl+L	say line
0x0e	Ctrl+N	bypass
0x18	Ctrl+X	flush
0x1b0b	Ctrl+Alt+K	keyboard wizard
0x1b0f	Ctrl+Alt+O	options menu
0x1b13	Ctrl+Alt+S	write configuration
0x1b30	Alt+0	say line:256
0x1b62	Alt+b	say character:-1
0x1b63	Alt+c	say character
0x1b64	Alt+d	say word
0x1b65	Alt+e	read cursor to bottom
0x1b69	Alt+i	reinitialize
0x1b6b	Alt+k	say line:-1
0x1b6c	Alt+l	say line
0x1b6d	Alt+m	say line:1
0x1b72	Alt+r	toggle review mode
0x1b74	Alt+t	read top to cursor
0x1b77	Alt+w	read screen
0x1b78	Alt+x	flush:1

When you enter review-mode, commands go to yasr directly instead of being passed to the application.

Review-mode commands
Hex value(s)	Keyboard command	Action
0x20	space	say review cursor
0x24	$	end of line
0x28	(	previous paragraph
0x29	)	next paragraph
0x3c	<	search back
0x3e	>	search forward
0x5e	^	beginning of line
0x60	`	ascii
0x62	b	say character:-1
0x63	c	say character
0x65	e	read cursor to bottom
0x66	f	find
0x6a	j	say line:1:0
0x6b	k	say line:-1:0
0x6c	l	say line
0x6d	m	say line:1:0
0x6e	n	bypass
0x74	t	read top to cursor
0x76	v	set option:6
0x77	w	read screen
0x78	x	say word:1
0x7a	z	say word:-1
0x1b73	Alt+s	set option:5
0x1b5b41	Up arrow	say line:-1:1
0x1b5b42	Down arrow	say line:1:1
0x1b5b44	Left arrow	say character:-1
0x1b5b43	Right arrow	say character:1

I'm not quite sure what several of those do, but feel free to experiment with them.

Next steps

From here, you can tweak your yasr keyboard commands in your ~/.yasr.conf or change your speech-dispatcher settings in your ~/.config/speech-dispatcher/speechd.conf and ~/.config/speech-dispatcher/modules/espeak.conf files.

You might copy/link the yasr binary into some place on your $PATH such as

$ mkdir ~/bin
$ ln -s ~/tmp/yasr-0.6.9/yasr/yasr ~/bin

If you're feeling adventuous and prefer flite you might try configuring speech-dispatcher to use that instead.

Finally, it helps to have a suite of applications that play nicely with text-to-speech output. A few you might want to try:

Local mail: mail/mailx
Remote mail: s-nail (available as a package)
Calendar: remind (available as a package)
Web-browsing: lynx or edbrowse
RSS: rss2email or newsboat (both available as as packages)
IRC: ii (available as a package) or tinyirc (not currently available as a package)
Task management: taskwarrior or devtodo (both available as as packages)
Music: cmus or mpd/mpc (both available as as packages)
Terminal multiplexing: tmux or GNU screen (available as packages) It helps to either hide the status-bar or remove the clock so that it doesn't constantly spam you with time updates
Games:
- bsdgames collection
- frotz interpreter for Infocom-style text adventures

CLI best practices

Tim Chase — Sat, 20 Apr 2019 01:08:31 GMT

This article pulls together a number of tips and best practices for building CLI applications.

Input

Use standard library input functions to read from stdin that can play nicely with rlwrap(1) and other input sources.
If using readline functionality, provide an option to disable it.
Only use readline functionality if you have used isatty(3) to test if stdin is a terminal.
Don't pass sensitive information like passwords and API tokens on the command-line since they will appear in the output of ps(1). Instead store them in a file with appropriate ownership & permissions or pass them to the application in an environment variable.
If you need to interactively read sensitive information such as passwords, read directly from /dev/tty or use curses/ncurses calls to prevent sensitive input from displaying on the screen.

Output

Though obvious, put errors on stderr and output on stdout.
Only show progress meters if requested via a command-line option.
Allow tunable output verbosity, preferrably including a completely silent mode (often -q/--quiet or -s/--silent) and increasing levels of info/debugging output (often one or more -v/--verbose options).
Make color optional/smart by respecting $NO_COLOR if set, or $COLOR={always,never,auto}. If $COLOR (or $PROGNAME_COLOR) is not set, use isatty(3) to establish a default.
Default output format should have one record per line with easy-to-discern/use column delimiters, not one record spanning multiple lines (unless requested as an output option like sqlite's ".mode line"). Use command-line options to specify other output formats such as JSON, XML, YAML, S-expressions, CSV or TSV.
Don't produce binary output on stdout unless explicitly requested.
Print numbers unformatted unless requested otherwise. This might include internationalizing/localizing things like currency markers, thousands separators, and decimal separators. Or this might involve humanizing units, often with a -k (Kilobytes), -m (Megabytes), -g (Gigibytes), or -h/--humanize flag.
Offer formatting control for dates/time including internationalizing/localizing or explicitly specifying a format string. If using a format string, stick to one standard: preferrably strftime(3) strings (%Y-%m-%d %H:%M), but CLDR format date strings, or PHP style date format strings are also common in certain communities.

Command-line options & switches

Provide both short (-x) and long (--long-option) versions of options. On non-GNU systems, long options are less expected.
Use common command-line option conventions:
- -h/--help should give an example of usage, a description of what the program does, along with a list of options and their descriptions. Note that -h may conflict with "human output".
- -v/--version should give version information. Note that -v may conflict with verbosity flags.
- If a command has side-effets such as deleting/renaming/modifying files or transferring large volumes of data, providea -n/--dry-run to let the user see what will happen without actually performing the requested action.
- -- to separate options and arguments.
- Use - as a file-name to read from stdin.
- Use @filename to read data from filename instead of data provided directly on the command-line.
For long options that take arguments, allow either an equals sign or a space to separate the option from its arguments. E.g. --name=Sam or --name Sam.
It can be beneficial to provide an option to expose expected completions for shell command-line completion functions.

Defaults

If an option can have a default value, provide one.
If output is truncated to the width of the tty, use curses calls to determine the width and fall back to 80 columns otherwise.
If spawning an external editor, check the following environment variables and binaries in order
1. $PROGNAME_EDITOR
2. $VISUAL
3. $EDITOR
4. vi(1)
5. ed(1)
using the first one that matches.

Other considerations

Where possible, allow the user to specify options in any order. Requiring options in a particular order makes things more fragile.
If one tool provides multiple actions, use the form "programname {global options} action {action-specific options}". Examples include
- git (git commit)
- Subversion (svn commit)
- Mercurial (hg commit)
- CVS (cvs commit)
- taskwarrior (task list)
- managing services (service httpd start)
Use $LC_CTYPE to determine the encoding of stdin/stdout but allow the user to specify alternate encodings and error/replacement strategies.
Return a 0 exit status on success. Return a non-0 exit status if any errors occurred, preferrably using standard values from /usr/include/sysexits.h.
Store local user configuration in ~/.config/$PROGNAME/
Settings should be determined in the following order (from lowest priority to highest precedence)
1. built in configuration
2. system-wide configuration from either /etc/${PROGNAME}rc, /etc/${PROGNAME}.conf, /etc/${PROGNAME}/*, /usr/local/etc/${PROGNAME}rc or /usr/local/etc/${PROGNAME}.conf
3. user configuration file in ~/.config/$PROGNAME/*
4. environment variables
5. command-line options
On Microsoft Windows, these conventions may differ. Notably, the use of /option:value instead of --option=value

Other resources

GNU standards for CLIs
Stack Overflow discussion on good CLIs
Exploring CLI Best Practices by Kevin Deisz
How to CLI
Crafting a command line experience that developers will love by Nick Parsons

HOFFA: httpd, OpenBSD, flat-files, and awk

Tim Chase — Sat, 09 Jun 2018 00:17:32 GMT

Upon learning that OpenBSD was discontinuing sqlite in the base system, I joked that the BCHS web stack would need to drop sqlite and investigate new options for a web-stack available purely in the OpenBSD base system. So continuing that joke, I proposed the HOFFA web stack: httpd, OpenBSD, flat-files, and awk.

Server configuration

This walk-through assumes you've managed to follow one of the many fine OpenBSD installation guides and now have a fresh OpenBSD system with no packages installed.

Begin by copying the template httpd.conf into the /etc directory:

$ doas cp /etc/examples/httpd.conf /etc/

Next, edit /etc/httpd.conf and in your server block, add a location directive to point to your cgi-bin/ directory (this is relative to your $CHROOT so the absolute path ends up being /var/www/cgi-bin/ by default)

server "example.com" {
    ⋮
    location "/cgi-bin/*" {
        fastcgi
        root "/"
    }
    ⋮
}

Now with the web server configured, it, and the slowcgi proxy need to be enabled.

$ doas rcctl enable slowcgi
$ doas rcctl enable httpd
$ doas rcctl start slowcgi
$ doas rcctl check slowcgi
$ doas rcctl start httpd

(the rcctl check slowcgi may be optional but it doesn't hurt to confirm that it's running)

Populating binaries & libraries in the `$CHROOT`

With the server and CGI proxy running, binaries need to be put where they'll be found. As this is HOFFA this uses awk

$ doas cp `which awk` /var/www/bin/

then create the library directories and copy in the needed libraries. First, find out which libraries awk needs:

$ ldd `which awk`
/usr/bin/awk:
 Start    End      Type  Open Ref GrpRef Name
 17b0e000 37b20000 exe   1    0   0      /usr/bin/awk
 05321000 2532c000 rlib  0    1   0      /usr/lib/libm.so.10.1
 07f46000 27f76000 rlib  0    1   0      /usr/lib/libc.so.92.3
 02875000 02875000 ld.so 0    1   0      /usr/libexec/ld.so

so we need to create those library directories in the $CHROOT

$ doas mkdir -p /var/www/usr/lib /var/www/usr/libexec

and copy the libraries in

$ doas cp /usr/lib/lib[mc].so* /var/www/usr/lib/
$ doas cp /usr/libexec/ld.so /var/www/usr/libexec/

Hello world

Now with the web server & slowcgi in place, along with the required binaries & libraries, it's time to write some code.

$ cd /var/www/cgi-bin
$ doas ed example.awk
a
#!/bin/awk -f
BEGIN {
 printf("Status: 200 OK\r\n");
 printf("Content-type: text/plain\r\n\r\n");
 print "Hello", ENVIRON["REMOTE_ADDR"], "from awk!";
}
.
wq
$ doas chown www:www example.awk
$ doas chmod +x example.awk

With this in place, it should now be possible to point your browser at your newly configured script. If your browser is on the same machine you can visit http://localhost/cgi-bin/example.awk or if it's on a remote machine, use its domain-name or IP address http://example.com/cgi-bin/example.awk

Unsorted ed(1) Tips and Tricks

Tim Chase — Sat, 06 Feb 2016 23:35:09 GMT

Edit text like a pro with ed(1)

I came across Ten unsorted vi/vim tricks, volumes one & Ten unsorted vi/vim tricks, volume two and thought I'd compile a similar list for ed(1).

In this post I will present counterpoints with easy & but useful tricks using ed(1) commands and, in the Unix philosophy, external tools.

You might ask, why ed(1)?

ed(1) is always^* there. Since ed(1) is required by POSIX, you should find it in any *nix/Linux distribution. Well, it should be there. Sadly, many distros are now dropping ed(1) from their core installations, making it an optional add-on package.
ed(1) is small but mighty. On various systems at my disposal, ed(1) clocks in at between 51k and 183k, while nano is 2-4x as large, vi/vim is 3-20x as large, and emacs clocks in at over 8 megs. Yet ed(1) provides a lot of functionality in those meager few bytes.
ed(1) needs no configuration. You don't have to worry about sitting down at a foreign machine and now knowing how ed(1) is configured. Yes, there are some subtle differences between GNU ed(1) and BSD ed(1) but the vast majority is the same.

Getting some help

ed(1) is known for its terse "?" reply to any issue. You can request in-line help with the "h" command for a bit of a hint. Additionally, the man page should cover everything you need to know.

s/oof/bar/
?
h
No match

Search and replace

To search, use the "/" or "?" command followed by your search pattern.

/pattern/n
18 this line contains "pattern"

To search and replace on the current line, use the "s" command:

s/foo/bar/g

or, apply the changes over a range:

4,18s/foo/bar/g

Show line numbers

To show line numbers in ed(1) add the suffix "n" to your command, for example

,s/foo/bar/n
g/pattern/n
10zn

Execute an external command from within `ed(1)`

Use the "!" command to execute

!ls
file.txt
!

Insert an existing file into the current one

In case you need to insert the contents of a file into the one you're currently writing, use the "r" command.

r file.txt
1625

Note that ed(1) lets you know how many bytes were added to your file.

Display changes performed since last save

This trick uses diff to display the difference between the file on disk and the current editing buffer.

w !diff -u file.txt -

Indendent and un-indenting lines

To indent a range of lines, substitute the beginning of the line with either spaces or tabs:

12,18s/^/    /
.,+4s/^/{tab}/

To unindent, strip off that number of leading indentation:

12,18s/^    //
12,18s/^ \{4\}//
.,+4s/^{tab}//

Undo & redo

ed(1) provides one level of undo/redo, using the "u" command to toggle between them:

$ ed
1a
hello
world
.
2d
,n
1 hello
u
,n
1 hello
2 world
u
,n
1 hello
Q

Insert the output of an external command

Similar to the insertion of a file above, ed(1) lets you use the "r !" command to read the output of an external program:

r !ls -l

Record commands

Command history

Since ed(1) appends to the terminal window rather than overwriting and redrawing the screen, the entire editing history for the given ed(1) session is available for review.

Using ed(1) as a password manager

Tim Chase — Fri, 06 Nov 2015 21:36:29 GMT

I recently came across a post on using vim as a password manager so I thought I'd post a companion article on using ed(1) as a password manager.

The main goal is to be able to keep the secrets encrypted at all times on disk, only decrypting within the active ed(1) session.

BSD ed(1) offers an x command (well, OpenBSD removed it in revision 1.37 of /src/bin/ed/main.c "because DES") that allows for weak DES encryption of the file. However, we want stronger encryption and portability between various versions of ed(1) so will ignore this option.

Writing our encrypted password file to disk

To begin, we'll open up ed(1) and add some password content

user@hostname$ ed
a
https://example.com
Username: demo
Password: Pa$$w0rd1

imaps://example.edu
Username: jstudent@example.edu
Password: ed(1)uc8

.

Now, instead of writing the unencrypted password file, we'll use gnupg to encrypt the file and write it out to the disk by piping the file through gpg instructing it to write to our password file. Note the trailing "-" which tells gpg to read the input from stdin. gpg will prompt for a passphrase and confirmation of that password.

w !gpg --symmetric --output passwords.gpg -
Enter passphrase: Password1
Repeat passphrase: Password1
127

ed(1) informs us that it successfully piped our 127 bytes of data through gpg but we can confirm that the passwords.gpg file was written and then we can quit to go about our day:

!ls passwords.gpg
passwords.gpg
!
q

Reading our encrypted password file from disk

Now, we want to be able to look up a password to enter at some future point. So we fire up ed(1) and decrypt our passwords.

user@hostname$ ed
r !gpg --decrypt passwords.gpg
Enter passphrase: Password1

We want to look up our log-in credentials for our email server so we issue

?imap.*edu
imaps://example.edu
+
Username: jstudent@example.edu
+
Password: ed(1)uc8
Q

(Yes, using "+" can be replaced with just hitting "<Enter>") Alternatively, we could use grep(1) or sed(1) to filter the results and show some context:

user@hostname$ gpg --decrypt passwords.gpg | grep -A2 example.com
Enter passphrase: Password1
https://example.com
Username: demo
Password: Pa$$w0rd1
user@hostname$ gpg --decrypt passwords.gpg | sed -n '/example.com/,/^$/p'
Enter passphrase: Password1
https://example.com
Username: demo
Password: Pa$$w0rd1

Modifying our password lists

Now we want to modify our document and/or change our master-password:

user@hostname$ ed
r !gpg --decrypt passwords.gpg
Enter passphrase: Password1
127
3s/Pa..w0rd1/Pbuttwrd1
Password: Pbuttwrd1
$a
https://twitter.com/
Username: ed1conf
Password: EyeDonutThinkSew

.
w !gpg --symmetric --output passwords.gpg -
Enter passphrase: NewPassword2
Repeat passphrase: NewPassword2
File `passwords.gpg` exists. Overwrite? (y/N) y
194

And there you have it: using ed(1) as a password manager.

Why ed(1)?

Tim Chase — Thu, 10 Sep 2015 01:55:08 GMT

As the weirdo behind the somewhat tongue-in-cheek @ed1conf account on Twitter and Mastodon, account I'm occasionally asked "Why ed(1)?" Hopefully some of my reasons for learning & using ed(1) can pique your interest in taking the time to get to know this little piece of history.

Ubiquity

Sometimes your favorite $EDITOR is installed, sometimes it's not. Some, like vi/vim are just about everywhere. Other times, you would have to have sufficient privileges/space to install or compile your editor of choice. But if you know ed, nearly every Linux/BSD/Mac has it installed because it's part of the POSIX standard. It's even small enough to fit on most recovery media without breaking the bank. But between ed and vi/vim, I know that I can get things done even when I'm on a new machine.

Sometimes it's the only editor you have

Several times in my life ed has been the only editor available in certain environments.

At $DAYJOB[-1] the Linux-based router needed some configuration changes that the web interface didn't accommodate. So a quick terminal connection later (telnet, sigh), I discovered that ed was the only editor available. No problem. Edited the config file and we were back up and running with the proper changes.
At the same $DAYJOB[-1], I developed software for a ruggedized hand-held device and its attached printer. This was back when PDAs were just becoming popular, so this thing was a brick. The DOS-based operating system had no built-in editor, meaning editing files usually meant laboriously copying the file over a serial link to the PC, editing it there, then sending it back down. I longed for the ability to cut that time down but very few of the full-screen editors I tried were even able to successfully paint on the small LCD screen-buffer properly, and of those that did, the on-screen keyboard took up so much screen real-estate as to make them useless. So I installed a DOS build of ed and it worked like a charm (better than edlin.exe that I also tried). Editing turn-around and testing went from 15-20 minutes down to 3-5 minutes per iteration.
Some platforms such as Heroku provide only ed as their editor. Not usually an issue since most of the time you're not trying to edit live on the server. But if you need to do it, it's nice to know how.
On some MUD games and old BBSes, the text-editor is often an ed-like editor.

Visible editing history

Unless you have a key-echoing utility like Screenkey or Screenflick, it's hard for an audience to see exactly what you did when you're editing during a presentation. It's nice to for the audience to be able to see exactly what you typed if they're trying to follow along.

All commands are ASCII text

Sometimes your terminal emulator or keyboard isn't configured correctly. Function keys, arrows, alt- and meta-modifiers may not transmit properly. Since all of ed's commands are basic ASCII, it works even if your keyboard/terminal is unable to send extended characters properly.

It works when `$TERM` is messed up

Likewise, your $TERM setting can get messed up. Sometimes full-screen terminal applications leave the screen in a state where everything is somewhat garbled. Yes, there's reset which will let you reset the terminal back to some sensible defaults, but sometimes your termcap database has trouble too. An editor that only uses stdin and stdout can save your bacon.

Accessibility

Because ed reads all of its commands from stdin and writes all output to stdout in a serial fashion, it's very usable in a screen-reader like yasr or speakup allowing you to edit text without a screen. If you've never edited text sightless, give it a try some time.

Scriptability

Because ed reads all of its commands from stdin it's easy to write a script that will edit a file in an automated fashion.

Previous output

On occasion, I want to see the output of one or more previous shell commands while I continue to edit. A full-screen editor takes over the entire screen, preventing me from seeing that output. With ed the previous output is right there and remains in my scroll-back buffer for reference while I edit. I find this particularly useful when using \e in psql or mysql if my $EDITOR is set to ed. This allows me to edit the SQL while keeping the results of my previous query visible.

Small, fast, light

On resource-constrained systems, sometimes you need something light like ed where the executable and memory-footprint are measured in kilobytes rather than megabytes. This is less of a problem on most systems these days, but with small resource-constrained SOC and embedded boards running Linux or BSD, a light-weight yet powerful editor can help.

Usable on low-bandwidth/high-latency connections

Sometimes you are connected by a slow or very laggy connection. Whether this is a satellite uplink, a 300-baud serial connection, or a congested WAN link, sometimes you simply want to edit productively without the overhead of repainting the screen.

Showing off

Finally, there's a small measure of grey-beard prestige that comes with using an editor that baffles so many people. It's a fast way to demonstrate that I'm not some newbie with cert-only knowledge, but that I enjoy Unix history and working at the command-line. Or maybe it shows that I'm just a little crazy.

Tim's blog (Posts about ed)

Old Computer Challenge

History

First year (2021)

Second year (2022)

Third year (2023)

Future challenges

Using mail(1)

Intro

History

System configuration expectations

Essential modes of operation

Send mode

Mail reading mode

Starting mail

Exiting mail

Listing messages

Displaying messages

Deleting (and undeleting) messages

Replying to messages

Composing new messages

Filing messages

Changing mailboxes

Checking for new mail

Mail composition mode

Tilde escapes

Address book

Settings

Tips & tricks

Deleting all-but XYZ

Quickly processing messages

Aggressive pf(4) configuration for SSH protection

Assumptions

Setup

Goals

Change the sshd port number

Change the listening port

Change the default connecting port

Creating a minefield

Rate-limiting connections

Limiting by country

Download GeoIP data

Automating GeoIP downloading

Create a _geoip user

Modify permissions on GeoIP data directory

Allow the _geoip user access to restart pf

Create a script to download & install GeoIP info

Scheduling the download

Final file

Conclusion

How I use remind(1)

Overview

Installing

As a package

From source

Getting started

Creating your first remind file

Adding time information

Getting advance notice

Adjusting dates

Repeating reminders

Modifying the output

Other views

Other message-types

Getting rid of blank lines

Selectively showing text

Substitution filter

Tweaking the banner

Sorting

Sub-BANNERs for sorting

Next instance of reminders

File management

Organizing reminders

Initial file types

Advanced features

Expression evaluation

Using functions in reminder text

Days until an annual event

Specifying the priority of a reminder

Adding a prefix/suffix to agenda items

Starting `mail`

Exiting `mail`

Change the `sshd` port number

Create a `_geoip` user

Allow the `_geoip` user access to restart `pf`

Creating your first `remind` file

Using `OMIT` for more complex dates

Using `SATISFY` to conditionally test events

Prior to the addition of `$DefaultColor` in version 3.3.0

After the addition of `$DefaultColor` in version 3.3.0

`SPECIAL` reminders

`RUN` reminders

Generating iCalendar/`.ics` files

Echoing into `remind -`

Using a custom `.reminders` file

Printing values in `MSG` output

Set up `speech-dispatcher`

Configure `yasr`

Getting started with `yasr`