There are a couple of terminal emulators which allow for display of arbitrary images in the terminal (I mean, jpegs and pngs and the like, not (just) sixel), notably Kitty and WezTerm.

WezTerm with iTerm2 compatible image protocol support, and built-in imgcat command, alongside of (experimental but seemingly working) support for Kitty’s image display protocol (enable this in WezTerm by adding to your wezterm.lua config the line enable_kitty_graphics=true).

I mainly use Fish shell, and here I record a couple of notes on convenience customisations in Fish for working with in-terminal image display in Kitty and WezTerm. But, I have a POSIX-compatible (including Bash) version (less well tested) included below as well.

WezTerm’s command to display an image in the terminal is wezterm imgcat </path/to/image/file>, and Kitty’s is kitten icat </path/to/image/file>.

I’m playing with both terminals, and have a hard time remembering things, so one thing I’ve done is to alias these commands in fish to just icat.

More significantly, while WezTerm seems to have a smart auto-sizing of the image to downsize the rendering to make sure the image fits fully within the current terminal window, Kitty only seems to guarantee downsized display to make sure the image fits the current width of the terminal window, but does not downsize to fit the current height of the terminal window.

Thus, in Kitty, we get things like this:

(….hmm, boots aren’t a kitty…)

While in WezTerm, we get:

And with appropriate configuration from below added, the new (easy-to-remember) icat command works to produce in-terminal image display as in WezTerm:

(…and there’s the kitty in Kitty!)

Here are the relevant lines that can be added to your shell configuration file to both make the image display command in both Kitty and WezTerm both simply icat, and also resize the display of the image in Kitty to make sure it fits lengthwise in the current terminal window:

Fish shell

#################### BEGIN image display things ######################

# image display things for kitty [https://sw.kovidgoyal.net/kitty/]
#                        & wezterm [https://wezfurlong.org/wezterm/]
#
# - use 'icat $path/to/image' (in either) to display image in terminal
# - on inner workings of the image display protocols, see:
#   - for kitty:    https://sw.kovidgoyal.net/kitty/kittens/icat/
#   - for wezterm:  https://wezterm.org/cli/imgcat.html

# if we're in kitty:
if [ "$TERM" = "xterm-kitty" ]
  alias icat-full "kitten icat" # alias the default kitty icat behaviour to `icat-full'

  # function icat-auto
  function icat  # define auto-resizing version of kitty's `icat'
    # calculate kitty window size parameters:
    set -l kittysize (kitten icat --print-window-size) # get kitty window size
    set -l kittywidth (echo $kittysize | cut -dx -f1) # extract width of kitty window
    set -l kittyheight (echo $kittysize | cut -dx -f2) # extract height of kitty window
    set -l argcount (count $argv)    # count how argumnets to icat there are

    # calculate target image size parameters: [requires ImageMagick!]
    set -l imagesize (identify -format '%wx%h' "$argv[$argcount]")  # `identify' is part of ImageMagick
    set -l imagewidth (echo $imagesize | cut -dx -f1) # extract width of image
    set -l imageheight (echo $imagesize | cut -dx -f2) # extract height of image
    set -l reducebyy 1.0  # initialise and set to '100%' [no change] y-axis reduction percentage
    set -l reducebyx 1.0  # initialise and set to '100%' [no change] x-axis reduction percentage

    # test if target image is bigger than kitty window
    if [ $imageheight -gt $kittyheight ] # if it's taller than the terminal window
                                         # divide terminal window height by image height
      set reducebyy (math $kittyheight / $imageheight)
    end
    if [ $imagewidth -gt $kittywidth ]  # if it's widre than the terminal window
                                         # divide terminal window width by image width
      set reducebyx (math $kittywidth / $imagewidth)
    end
    if [ $reducebyy -lt $reducebyx ]  # set both reducebyx, reducebyy to smallest of 2 values
      set reducebyx $reducebyy
    else
      set reducebyy $reducebyx
    end
    # calculate target image height and width to fit fully in terminal window
    set imageheight (math floor (math $reducebyy x $imageheight))
    set imagewidth (math floor (math $reducebyx x $imagewidth))

    # call the `kitten icat' kitty command using the calculated image size
    if [ "$argcount" -gt 1 ] # if the user has passed more args than just the target image filepath
                             # then pass all of those separately along with `--use-window-size' parameter
      command kitten icat --use-window-size $COLUMNS,$LINES,"$imagewidth","$imageheight" $argv[1..(math (count $argv) - 1)] $argv[(count $argv)]
    else
      command kitten icat --use-window-size $COLUMNS,$LINES,"$imagewidth","$imageheight" $argv[1]
    end
  end

  # if we're in WezTerm, alias `icat' to wezterm's imgcat
  else if [ "$TERM_PROGRAM" = "WezTerm" ]
    alias icat "wezterm imgcat"
    # NOTE: currently not sure best way of emulating kitty's default image display behaviour
    #      - mainly because I don't know how to get the WezTerm window size in a
    #        fashion that's window-manager neutral and neutral between
    #        graphical display protocol (i.e., X11 vs Wayland)

    # draft version of default kitty icat-like behaviour for `imgcat'
    # function icat-full
    #   set -l imagesize (identify -format '%w' "$argv[$argcount]")
    # end
  # alias icat-full "wezterm imgcat --width 100%"
end

#################### END image display things ######################

POSIX shell (including Bash)

#################### BEGIN image display things ######################

# image display things for kitty [https://sw.kovidgoyal.net/kitty/]
#                        & wezterm [https://wezfurlong.org/wezterm/]
#
# - use 'icat $path/to/image' (in either) to display image in terminal
# - on inner workings of the image display protocols, see:
#   - for kitty:    https://sw.kovidgoyal.net/kitty/kittens/icat/
#   - for wezterm:  https://wezterm.org/cli/imgcat.html

# if we're in kitty:
if [ "$TERM" = "xterm-kitty" ]
then
    alias icat_full="kitten icat" # alias the default kitty icat behaviour to `icat-full'

    icat () { # define auto-resizing version of kitty's `icat'
        # calculate kitty window size parameters:
        kittysize="$(kitten icat --print-window-size)" # get kitty window size
        kittywidth="$(echo $kittysize | cut -dx -f1)" # extract width of kitty window
        kittyheight="$(echo $kittysize | cut -dx -f2)" # extract height of kitty window

        # calculate target image size parameters: [requires ImageMagick!]
        icat_last_arg_pos="\${$#}"
        icat_last_arg_content="$(eval echo \"$icat_last_arg_pos\")"
        identify_command="identify -format '%wx%h' \"$icat_last_arg_content\"" # `identify' is part of ImageMagick
        imagesize="$(eval $identify_command)"
        imagewidth="$(echo $imagesize | cut -dx -f1)" # extract width of image
        imageheight="$(echo $imagesize | cut -dx -f2)" # extract height of image
        reducebyy=100  # initialise and set to '100%' [no change] y-axis reduction percentage
        reducebyx=100  # initialise and set to '100%' [no change] x-axis reduction percentage

        # test if target image is bigger than kitty window
        if [ "$imageheight" -gt "$kittyheight" ] # if it's taller than the terminal window
        then
            # divide terminal window height by image height
            reducebyy=$(( ( kittyheight * 100 ) / imageheight ))
        fi
        if [ "$imagewidth" -gt "$kittywidth" ]  # if it's wider than the terminal window
        then
            # divide terminal window width by image width
            reducebyx=$(( ( kittywidth * 100 ) / imagewidth ))
        fi
        if [ "$reducebyy" -lt "$reducebyx" ]  # set both reducebyx, reducebyy to smallest of 2 values
        then
            reducebyx="$reducebyy"
        else
            reducebyy="$reducebyx"
        fi
        # calculate target image height and width to fit fully in terminal window
        imageheight=$(( ( reducebyy * imageheight ) / 100 ))
        imagewidth=$(( ( reducebyx * imagewidth )  / 100 ))
        kitten icat --use-window-size $COLUMNS,$LINES,$imagewidth,$imageheight "$@"
    }
    # if we're in WezTerm, alias `icat' to wezterm's imgcat
elif [ "$TERM_PROGRAM" = "WezTerm" ]
then
    alias icat="wezterm imgcat"
fi

#################### END image display things ######################

Note about built-in capacity

I confirmed with the developer, Kovid Goyal, that currently Kitty does not have an “autofit” option (other than it’s default autofit to fit width), see GitHub issue #9201. But Kovid seems to have gone ahead and added a built-it --fit option for Kitty, so a built-in way of managing image display with respect to window size should be available natively in an upcoming Kitty release.

• some sort of persistence (a “bouncer” or the always-connected server)
• use with some sort of IRC client in Emacs
• some sort of reasonable mobile phone client

It ended up longer than I intended.

tldr

• Using some of the more IRCv3 tools, particularly the cluster of things by Simon Ser and co (soju, goguma, gamja - see below) makes for a much better IRC experience, and easily supports synchronised access across multiple devices (e.g., desktop, laptops, mobile).
  • soju provides a very capable bouncer
  • goguma provides an excellent mobile client
  • gamja is a very user-friendly web-client front-end
  • and these three things can complement each other in a complete IRC setup system
• we can make such a soju-based set-up work well with Emacs IRC clients
• soju and gamja can be self-hosted; but there are also two paid/hosted instances currently available:
  • SourceHut’s chat.sr.ht (on a fork of soju with a different feature-set)
  • IRC Today’s service (I think closer to mainline soju, but not certain)
• we can leverage some of the IRCv3 functions nicely in Emacs

The most useful bits here may be notes on how to set up clients to work with SourceHut’s chat.sr.ht, particularly Goguma and Emacs/ERC, and some of the cool features of IRCv3 things [modernising IRC], like the soju bouncer (which is used by chat.sr.ht), for which see the section below “Smooth & Modern IRC Implementation & Practice (mobile, Emacs)", which you might jump to if you don’t want to read a bunch of preamble.

History, theory, background

Pre-history: wee chats in musl pies

Long ago now I spent a while setting up a RaspberryPi 3b (running musl flavoured Void Linux) as a Weechat Server/Bouncer in order to make using IRC less painful. This involved a lot of steps, including setting up a NoIP script and SSL certs with Let’s Encrypt, and setting up scripts to auto-fetch new certs. But once up, it worked well (unless my home internet went out), and Weechat had a nice mobile app, so I could connect both on desktop/laptop with an Emacs IRC client and also had a pocket connection via my mobile phone ¹. This gave me a persistent connection and log history and all that.

Then I started using Matrix (=a modern, but heavy, “IRC replacement”, itself theoretically a Slack, Discord, &c. competitor) more, (in part) since some projects have chosen it as a more modern/capable alternative to IRC, and, soon after, it was the case that most of the IRC rooms I was in were bridged to Matrix anyway. And then it seemed perhaps more convenient just to access everything in one place, since some things were only Matrix and not on IRC, it made sense for that to be Matrix.

And then the Matrix bridges mostly shut down.

Path of the prodIRCal son

I toyed off and on with getting re-set up on IRC, but there has been a lot of other things going on in life, and I didn’t feel like trying to set up my rpi mini-server IRC bouncer again.

I tried some other hosted IRC bouncer services. But some of these wouldn’t let me connect with on a VPN.

I found one good free (as in freedom and free as in no paisa) ZNC bouncer service, which is FreeIRC.org ²

The ZNC bouncer worked fine on Emacs (with ERC; I’ve gone back and forth between ERC and Circe, but this time I was trying ERC again).³ And I eventually figured out how to connect to multiple networks through the ZNC bouncer.

But the best thing I could find on mobile was the Revolution IRC client, and, while it’s a nice enough front-end, it struggled with maintaining a connection to the ZNC bouncer I was using. (I asked about this in an IRC room, and the general consensus was that mobile’s just not a good place to try to do IRC, but I remembered that via the Weechat android app or through Glowing Bear I’d had a good mobile IRC experience years ago.)

IRCv3, and other Korean tubers

I also came across another IRC mobile app, Goguma, but couldn’t get it work. But, frustrated with the situation, I wanted to see if, assuming I could somehow get it work, Goguma might be a better mobile solution.⁴

I ultimately ended up stumbling across an interesting lobste.rs discussion of Goguma which pointed to it being part of a set of IRCv3-aware software, including an IRC bouncer soju, which a number of the commenters on the lobste.rs thread comparing the experience favourably against using ZNC.

A cluster of IRCv3 things:⁵

• Goguma (Korean 고구마 goguma “sweet potato”): IRC mobile client written in Flutter by Simon Ser
• Gamja (Korean 감자 gamja “potato”): simple IRC web client written in Javascript by Simon Ser
• Soju (Korean 소주 soju “a distilled alcoholic beverage”): IRC bouncer written in Go by Simon Ser
• Senpai (Japanese 先輩 sèńpáí “senior in social standing or level of education/skill; an elder”): a modern terminal IRC written in Go, started by ‘taiite', who handed development over to ‘delthas'.
• Ergo (a play on ergonomic [and “ergonomadic"] and Go(lang), so I suppose ultimately from Ancient Greek ἔργον érgon “work”, but not sure that’s relevant): a modern IRC server written in Go [Jeremy Latt, Daniel Oaks, Shivaram Lingamneni] implementing bleeding-edge IRCv3 features/support

The first three things are the most relevant for us (well, me). Ergo sounds great, but I’m not running an IRC server myself at the moment; and I’ve got a whole wealth of choices of Emacs IRC clients if I’m on desktop/laptop, so Senpai doesn’t seem relevant either for my use case.

Trains, Wayland, and Things that Go

The first three of the IRCv3-looking things are all by Simon Ser, who was lately at Drew DeVault‘s SourceHut, but seems to have left sometime in 2024⁶ and now works at Open Source Railway Designer (something to do with making tools for railway infrastructure simulation, which sounds quite cool: open source trains). Ser obviously works a good bit on IRC-related software, and also a lot of things in Go, and has been involved with Wayland-related projected (taking over maintainership of Sway (a Wayland window manager) and wlroots (Wayland general purpose compositor underlying Sway and other things) from DeVault some years back).

Here’s Thomas Flament & Simon Ser talking about IRCv3 and soju, senpai, gamja, goguma at FOSDEM ‘25 in Brussels:

Chatting on IRC in 2025: grandpa, what’s up? [video link with captions]

In any case, one of the paid-only features that SourceHut offers is via chat.sr.ht, an IRC bouncer, which is running on a fork of soju, with a web-client based on gamja.

As far as I can tell, Sourcehut’s servers are now (all? mainly?) in Amsterdam.⁷

chat.sr.ht isn’t the only hosted IRC bouncer service which runs some version of the soju bouncer, there is also IRC Today [discussed also on lobste.rs], which is hosted in Paris on Scaleway servers.⁸

So, there are at least two apparently EU-based hosted “IRCv3-forward” bouncers.

Smooth & Modern IRC Implementation & Practice (mobile, Emacs)

How to manage to drink rice liquor in a hut: preliminaries

In any case, I signed up for the SourceHut offering at chat.sr.ht, which uses their fork of soju.

Here’s what I did to get it set up for mobile and in Emacs (it seemed worth documenting, in what was meant to be a short post….):

The main things one needs to know are in the chat.sr.hut ‘manpages’, especially the ‘quickstart guide’. But here are a few more notes for specific set-up on Goguma and Emacs.

The first thing one needs to do for any “third-party” client (this includes Goguma or any Emacs IRC client) is generate an OAuth 2.0 personal access token in your SourceHut account. This will be effectively your “password” (with your SourceHut login name as your username).

Nb: on mobile, you have to be careful to actually manage to copy the entire token with “Select all” or the like - I was stuck for a long time not being able to make Goguma work because I’d not managed to copy the entire token, and this wasn’t at all obvious in a mobile browser. On desktop, it’s not an issue.

Goguma (mobile): One potato, two potato, sweet potato

For Goguma, you’re just going to put in:

• Server: chat.sr.ht
• Nickname: <your SourceHut username>
• Password: <an OAuth2.0 token you generated as above>

And you’re in. And here’s what it looks like:

And there’s a setting in Goguma (“Compact message list”) to make it look less like a mobile messaging app and just have plain text if you prefer that:

And you can have Goguma show you notifications if someone mentions your nick. Here is an example showing on a Pebble watch:

(IRC on one’s wrist; it truly is 2025.)

Emacs machines and authority tokens: what to put in your authinfo.gpg and init.el

In your ~/.authinfo.gpg file, add a line like this (with <>‘ed items to be replaced fully; don’t actually include any <>'s here or above or elsewhere; where "my-sourcehut-id" is your actual sourcehut login id in quotes):

machine chat.sr.ht login "my-sourcehut-id" password "<one of your OAuth 2.0 tokens generated at SourceHut meta>"

And then in your ~.emacs.d/init.el, assuming you’re using ERC and have that configured otherwise to your liking, make a user-function for each actual IRC server you want to connect with via chat.sr.ht/soju, like this (assuming your user-id is "my-sourcehut-id": replace appropriately— and you might have different :nick's (“nicknames” ≈ user-handles/user-names/identities) on different servers, of course, so adjust those as well; but your :username is going to be consistently your SourceHut username followed by “/<the-particular-irc-server-address>"):

;; just so we don't have to type it each time, a wrapping `let',
;; and so after this our `server', `port', & `passwd' will be thus defined
;; (you could do `:nick' like this too if it's the same everywhere)
(let ((server "chat.sr.ht")
      (port 6697)
      (passwd
       (cadr
        (auth-source-user-and-password "chat.sr.ht" "my-sourcehut-id"))))
  ;; `auth-source-user-and-password' returns a list of `login' and `password'
  ;; from your .authinfo.gpg; but we just need the second, so `cadr' (= return
  ;; the `car' of the `cdr'), because since the list is going to be something
  ;; like `(mylogin mypassword)', will give us just the atomic `mypassword',
  ;; which is what we want [because the `cdr' of `(mylogin mypassword)' will be
  ;; `(mypassword)' and the `car' of `(mypassword)' will be just `mypassword'].)

  (defun erc-libera ()
    "Connect to Libera.Chat IRC server."
    (interactive)
    (erc-tls
     :nick "emacsuser007"
     :server server
     :port port
     :user "my-sourcehut-id/irc.libera.chat"
     :password passwd))

  (defun erc-oftc ()
    "Connect to OFTC IRC server."
    (interactive)
    (erc-tls
     :nick "emacsuser007"
     :server server
     :port port
     :user "my-sourcehut-id/irc.oftc.net"
     :password passwd))

  (defun erc-ircnow ()
    "Connect to IRCNow IRC server."
    (interactive)
    (erc-tls
     :nick "emacsuser007"
     :server server
     :port port
     :user "my-sourcehut-id/irc6.ircnow.org"
     :password passwd))

  (defun erc-ergo ()
    "Connect to IRCv3-forward Ergo IRC server."
    (interactive)
    (erc-tls
     :nick "emacsuser007"
     :server server
     :port port
     :user "my-sourcehut-id/irc.ergo.chat"
     :password passwd)))

Here we’re making connections for Libera, OFTC, IRCNow, and Ergo, the IRCv3-forward server implemented in Go we mentioned above.⁹

But you can add lots of different servers to your bouncer, and both in Goguma and Emacs/ERC (just make a new erc-<server> function as above), you’ll just be able to get to all of the different rooms in one place and don’t necessary need to worry about which server a particular thing is on after getting it set up.

And then you can make a “connect to all the things function” like this:

(defun erc-connect-all ()
  "Connect to all IRC servers above."
  (interactive)
  ;; we're just calling all of the functions we defined above here:
  (erc-libera)
  (erc-oftc)
  (erc-ircnow)
  (erc-ergo))

And what it looks like (well, it looks like however you’ve configured ERC or whatever Emacs IRC client you’re using, but, in case you want to see a possible way it could look):

If you get knocked offline momentarily, ERC will try to reconnect to the bouncer. If you are knocked off or close Emacs for a while, you might want to catch up on what you missed recently. And there’s a nice IRCv3 feature implemented in soju (and in chat.sr.ht) for this, as discussed in the next section.

Leveraging IRCv3 features in Emacs

We can also do another convenient thing for our Emacs IRC configuration now. IRCv3 defines a “chathistory” extension, as an easy way of retrieving earlier IRC room content. This doesn’t work most places, but it works on any bouncers based on soju (see notes below), the syntax being /quote CHATHISTORY LATEST <channelname> * <number-of-lines-to-fetch>.

We can make this more user-friendly in Emacs with something like this, an interactive command that fetches history for whatever IRC room it’s called in (a 100 lines by default; prefix C-u to enter a different number):

(defun erc-chatsrht-give-me-more-irc-history ()
  "Get more history for current IRC buffer (IRCv3 only).

Defaults to 100 lines of history; when C-u prefixed, asks user for
number of lines to fetch.

If using an IRCv3 capable server/bouncer (like chat.sr.ht), fetch the
chat history via the IRCv3 chathistory extension. (Currently, only
soju-based servers implement this feature; see:
https://ircv3.net/software/clients)

For more on chathistory, see:
 - https://man.sr.ht/chat.sr.ht/bouncer-usage.md#chat-history-logs
 - https://ircv3.net/specs/extensions/chathistory
 - https://soju.im/doc/soju.1.html"
  (interactive)
  (if (not (member
            (with-current-buffer (current-buffer)
              major-mode)
            '(erc-mode
              circe-mode
              rcirc-mode)))
      (message "not an IRC buffer; ignoring")
    (let ((lines 100)
          (channel (buffer-name)))
      (when current-prefix-arg
        (progn
          (setq lines
                (read-number (format "How many lines to fetch: ") lines))))
      (erc-send-input
       (concat "/quote CHATHISTORY LATEST " channel " * " (number-to-string lines))
       t))))

Then you can just call erc-chatsrht-give-me-more-irc-history in an IRC buffer (you might find a convenient keybinding for it) to get prior IRC chat in that buffer.

This sort of thing makes Emacs a very pleasant environment to manage IRC in.

Now, for actually adding servers to your chat.sr.ht bouncer in the first place, you might try out the gamja web client (see below).

Inline images and files in Goguma and Emacs

Oh. Gamja (as least as implemented by chat.sr.ht) doesn’t seem to do it, but Goguma fetches image links and displays a preview:

[Note: chat.sr.ht’s soju fork doesn’t currently offer direct file-upload and is perhaps unlikely to, but this seems to be implemented in upstream soju and maybe on IRC Today.]

Additional note: Goguma doesn’t do this by default; you need to go into the main “Settings” menu and enable “Display link previews”. (I also have “Send & display typing indicators” turned on here.)

And you can make ERC do similarly, with respect to fetching posted image links, with the erc-image package from MELPA. I have a use-package definition like this:

(use-package erc-image
  :ensure t
  :after erc
  :config
  (setq erc-image-inline-rescale 300) ; maybe set bigger
  (add-to-list 'erc-modules 'image))

And then you’ll see something like this if you post an image link in ERC:

I’d like to figure out how one might better manage it, but — in terms of handling posting local files — there’s a (not-on-MELPA) package imgbb.el which will upload images in Emacs (when passed a file-path or interactively from a file-chooser) to ImgBB, and then put the link in the clipboard.¹⁰

[Note: For better or worse, erc-image will pull images from http as well as https addresses; while Goguma seems only to do the latter.]

Web-client: The sweetness of non-sweet potatoes on the web

The web interface for chat.sr.ht (at https://chat.sr.ht) is, again, based on gamja (“potato”) (discussed above) and gamja is a nice interface indeed.

And, for making your initial connections to different IRC servers, it may be easier just to use this web interface, especially if you’ve already got accounts set up on any of the servers, as for most of them you’ll be able to enter your username and password and so on in the “Add server” interface and not have to do all of the /msg NickServ .... stuff one usually does. (If you don’t have an account already, you’ll have to register in someway, maybe by messaging Nick).

Looks like this:

In fact, even for signing in/up on a brand new server, at least on some servers, gamja lets you click on “login” or “register” and pulls up a pop-up box to fill in password, email, &c. So it’s quite convenient (and you may not have to message Nick manually).

Wrap-up & evaluation

There are interesting IRC things still going on, both in terms of active IRC communities (though this varies; #emacs is quite active, #dwarffortress isn’t; Discord has eaten into some spaces), and innovations in IRC technologies and tools.

I’m not sure what the best thing is terms of choice of a soju/gajma ecosystem. chat.sr.ht seems to be the cheapest hosted option; IRC Today is more expensive but may currently be more featureful. SourceHut has had trouble with being DDOS’ed/going off-line (see, e.g., SourceHut network outage post-mortem).

Self-hosted set-ups are another option, but are likely to be more fiddly/effort to run/maintain than a “professional” service. But here are a few links to descriptions of people’s self-hosted soju setups:

• self-hosting on Alpine Linux: Self-Hosting Soju
• self-hosting on OpenBSD: Setting up an IRC bouncer (soju) on OpenBSD · Hugo’s weblog
• more self-hosting on Alpine Linux: Self-Hosted IRC with Soju & Senpai | Isaac Ganoung’s Website
• self-hosting on IRCNow: IRCNow | Soju / Guide

Currently, chat.sr.ht is working well for me, even if it may be behind on some mainline soju features.

Appendix: etymologies

You probably don’t need to know any of this. It’s not going to help you log into Goguma or make your Emacs config work.

But the naming of a lot of the IRC things above is strange and I’m a linguist by trade and it’s hard for me to avoid paying a lot of attention to words and meanings of words and connections of words.

(While on the topic of mostly irrelevant things: I have some other-language interference (Hindi here) with the Korean names of some of the software discussed here. Specifically, gamja makes me think either गंजा ganjā “bald” or गाँजा gā̃jā “ganja, cannabis”; and soju keeps making me think of an elided version of सो जाऊं so jāū̃ “shall I go to sleep?” [“नहीं! आप आई॰आर॰सी॰ बाउंसर हैं, आपको सो नहीं जाना चाहिए! No! You’re an IRC bouncer: you should not go to sleep!"])

At any rate, here are, unasked for, etymological notes on the four IRCv3 things with Korean or Japanese names, largely sourced from Wiktionary, as indicated, with some connecting text:

Goguma

[mainly sourced from: https://en.wiktionary.org/wiki/%EA%B3%A0%EA%B5%AC%EB%A7%88]

Named apparently after Korean 고구마 goguma “sweet potato” (cp. the web client by the same team, gamja “potato”). First attested in the Mulmyeonggo (물명고 / 物名考), 1824, as Early Modern Korean 고금아 (Yale: kokuma), borrowed from Japanese 孝行芋 (kōkō imo), a term used in the Tsushima dialect. Some earlier attestations are known, but they are in the context of quoting the dialectal Japanese word, not in a Korean context.

There’s also apparently a recent “Internet slang” sense: “a plot development which frustrates the reader (e.g. the protagonist fails to achieve their goal)” [from c. 2012].

The Japanese word 孝行芋 (kōkō imo) from which Korean 고구마 goguma is borrowed is apparently 孝行 (kōkō) [? “dutiful”??] + 芋 (imo) from Old Japanese, attested in the Nihon Shoki of 720, referring to “edible tubers”. May be a shift from older form うも (umo), ultimately from Proto-Japonic *umo. Cognate with Okinawan 芋 (‘nmu). [see: https://en.wiktionary.org/wiki/%E5%AD%9D%E8%A1%8C%E8%8A%8B#Japanese https://en.wiktionary.org/wiki/%E8%8A%8B#Japanese ]

The Japanese word 芋 (imo) is connected to Chinese 芋 [Old *ɢʷas, Modern Mandarin yù ] “taro”.

Seemingly a phono-semantic compound (形聲 / 形声, OC *ɢʷa, *ɢʷas): semantic 艸 (“grass; plant”) + phonetic 于 (OC *ɢʷa) – “taro”.

Compare Proto-Hmong-Mien *wouH (“taro”), Burmese ဝ (wa., “elephant foot yam”), Tibetan གྲོ་མ (gro ma, “ Argentina anserina (syn. Potentilla anserina), a plant with small edible tubers”).

There are various theories on how all these words are related:

• Schuessler (2007) considers it to be an areal word, comparing it to the Hmong-Mien and Burmese words. Schuessler (2015) does not consider the Tibetan word to be cognate.

• Blench (2012) suggests that the Chinese word is borrowed from Proto-Hmong-Mien and that the Burmese word may be a late loan from Old Chinese.

• STEDT reconstructs Proto-Sino-Tibetan *g/s-rwa (“taro; yam; tuber”), whence the Tibetan word. This etymon is regarded as allofamically related this word and 薯 (OC *djas).

• Gong Hwang-cherng (2002) and Baxter and Sagart (2017) also suggest that this word is related to the Tibetan word.

  [on the Chinese, see: https://en.wiktionary.org/wiki/%E8%8A%8B]

Gamja

[sourced from: https://en.wiktionary.org/wiki/%EA%B0%90%EC%9E%90]

Seemingly named from Korean 감자 gamja “potato”, which is a nativisation of the Sino-Korean term 감저 (甘藷, gamjeo, “lesser yam (Dioscorea esculenta)”). First attested 1766 in Korea, then referring to the sweet potato (Ipomoea batatas). The word came to refer to both “potato” and “sweet potato” in the nineteenth century, and later lost its original meaning. (So gamja meant “sweet potato” first; but partially supplanted by goguma.)

With 甘藷, gamjeo apparently borrowed from Early Modern Korean 감져 (kamcye), itself from 甘藷 [https://en.wiktionary.org/wiki/%E7%94%98%E8%97%B7].

[There’s also homophonous verbal form: 감자 (gamja) (plain hortative of 감다)

1. let’s close (our eyes)

2. let’s wash

   which is presumably unrelated.]

Soju

[source: https://en.wikipedia.org/wiki/Soju]

Seemingly named for Korean Soju (소주; 燒酒), which means “burned liquor”: with the first syllable, so (소; 燒; “burn”), referring to the heat of distillation and the second syllable, ju (주; 酒), meaning “alcoholic drink”. Etymological dictionaries record that China’s shaozhou (shāojiǔ, 烧酒), Japan’s shochu (shōchū, 焼酎), and Korea’s soju (soju, 燒酒) have the same etymology. [A Wikipedian has here added a note about unreliable sources(?).]

Another name for soju is noju (노주; 露酒; “dew liquor”), with its first syllable, no/ro (노/로; 露; “dew”), likening the droplets of the collected alcohol during the distilling process to dewdrops.

The origin of soju dates back to 13th-century Goryeo. The Yuan Mongols [the imperial dynasty of China founded by Kublai Khan, grandson of Genghis Khan] acquired the technique of distilling arak from the Persians during their invasions of the Levant, Anatolia, and Persia, and in turn introduced it to the Korean Peninsula during the Mongol invasions of Korea (1231–1259). Distilleries were set up around the city of Gaegyeong, the then-capital (current Kaesong). In the areas surrounding Kaesong, soju is still called arak-ju (아락주). Andong soju, the direct root of modern South Korean soju varieties, started as the home-brewed liquor developed in the city of Andong, where the Yuan Mongols’ logistics base was located during this era.

Senpai

[source: https://en.wiktionary.org/wiki/%E5%85%88%E8%BC%A9#Japanese]

Obviously named for Japanese 先輩 senpai “senior/superior in social standing or education or skill; an elder” (the homepage has a tagline “Your everyday IRC student” and also “Welcome home, desune~” and also a smiling anime-ish cat-ish logo).

The Japanese word was borrowed from the Middle Chinese 先輩 (sen pwojH). A doublet of ソンベ (sonbe) [With Japanese sonbe being itself borrowed from Korean 선배(先輩) (seonbae), which means “(chiefly in Korean media) sunbae (upperclassman or senior, in the context of Korea)"].

Both ultimately from Chinese 先輩 [ xiānbèi ] “older generation; senior; elder; ancestor; predecessor”.

In Chapter 9 of The Little Schemer (“and Again, and Again, and Again,…"), it starts off by querying the reader if they want caviar and how to find it in a list, and then essentially gets (from caviar and grits) into issues around the halting problem.

But a number of pages into this chapter the book queries of a particular function: “Is this function total?” and presents the following:

(define C
  (lambda (n)
    (cond
     ((one? n) 1)
     (else
      (cond
       ((even? n) (C (÷ n 2)))
       (else (C (add1 (× 3 n)))))))))

I didn’t know what C was (a number of readers probably have recognised it already, but please don’t spoil it for the others just yet).

But whether we get the point of the function or not,we can implement it in Emacs Lisp, translating from the Scheme into Emacs Lisp (defining an equivalent of even? as an Emacs Lisp evenp first) as:

;; -*- lexical-binding: t; -*-
(defun evenp (n)
  "Returns `t' if `n' is even."
  (if (= (% n 2) 0)
      t
    nil))

(defun C (n)
  "Function C from Little Schemer p.155."
  (cond
   ((= n 1)
    1)
   ((evenp n)
    (C (/ n 2)))
   (t
    (C (1+ (* 3 n))))))

And we can try running this on various numbers:

(C 1) ; 1
(C 9) ; 1
(C 108) ; 1
(C 837799) ; 1
(C 8400511) ; 1
(C 63728126) ; 1

That is, for all of these numbers it just outputs 1. Well, we can try a couple of other values:

(C 0) ; cl-prin1: (error "Lisp nesting exceeds ‘max-lisp-eval-depth’")
(C 63728127) ; cl-prin1: (error "Lisp nesting exceeds ‘max-lisp-eval-depth’")

So C fails for 0 and numbers above 63728126, with a “excessive lisp nesting”-type issue.

But since the only non-error result we’re getting is 1, we could write a much simpler function.¹

;; -*- lexical-binding: t; -*-
(defun ersatz-C (n)
  "Returns 1 if `n' is 1;
enter an infinite nesting loop
if `n' is 0 or above 63728126;
for any other integer value,
pause briefly for sake of plausibly,
and then return 1."
  (cond
   ((= n 1)
    1)
   ((= n 0)
    (ersatz-C n))
   ((> n 63728126)
    (ersatz-C n))
   (t (progn
        (sleep-for 0.005)
        1))))

But obviously function C is doing something more interesting. So, let’s make it actually show us what it’s doing:

;; -*- lexical-binding: t; -*-
(defun C-verbose (n &optional count)
  "For a number `n', return `1' if it's `1'; otherwise,
if it's even, run the same function on half of `n';
else, run the same function on one more than three times
`n'."
  (let ((count (or count 0)))
    (print (format "Step %s: %s" count n))
    (setq count (1+ count))
    (cond
     ((= n 1)
      1)
     ((evenp n)
      (C-verbose (/ n 2) count))
     (t
      (C-verbose (1+ (* 3 n)) count)))))

(C-verbose 7) ; =
;; "Step 0: 7"
;; "Step 1: 22"
;; "Step 2: 11"
;; "Step 3: 34"
;; "Step 4: 17"
;; "Step 5: 52"
;; "Step 6: 26"
;; "Step 7: 13"
;; "Step 8: 40"
;; "Step 9: 20"
;; "Step 10: 10"
;; "Step 11: 5"
;; "Step 12: 16"
;; "Step 13: 8"
;; "Step 14: 4"
;; "Step 15: 2"
;; "Step 16: 1"

(C-verbose 9) ; =
;; "Step 0: 9"
;; "Step 1: 28"
;; "Step 2: 14"
;; "Step 3: 7"
;; "Step 4: 22"
;; "Step 5: 11"
;; "Step 6: 34"
;; "Step 7: 17"
;; "Step 8: 52"
;; "Step 9: 26"
;; "Step 10: 13"
;; "Step 11: 40"
;; "Step 12: 20"
;; "Step 13: 10"
;; "Step 14: 5"
;; "Step 15: 16"
;; "Step 16: 8"
;; "Step 17: 4"
;; "Step 18: 2"
;; "Step 19: 1"

Perhaps not surprisingly, we see that setting n=9 makes the function take more steps to reach 1 than setting n=7. But it’s not actually a direct linear relation, because if we set n=10, it’s a much shorter path:

(C-verbose 10) ; =
;; "Step 0: 10"
;; "Step 1: 5"
;; "Step 2: 16"
;; "Step 3: 8"
;; "Step 4: 4"
;; "Step 5: 2"
;; "Step 6: 1"

So we might still wonder about exactly what’s going on with function C, how it works, and what’s the point of it anyway.

Conjectures about simple arithmetic operations and integer pathways

The Little Schemer does give a clue about what C is, referring to the question of whether it’s a total function or not:

It doesn’t yield a value for 0, but otherwise nobody knows. Thank you, Lothar Collatz (1910–1990).

A bit of research and we can find quite a bit about C: it turns out to be the Collatz conjecture: one of the most famous unsolved problems in mathematics.

The Wikipedia page on the Collatz conjecture is useful here in laying out basics. As well, Eric Roosendaal‘s “On the 3x + 1 problem”. And Quantamagazine’s “The Simple Math Problem We Still Can’t Solve” presents a fairly accessible overview, including recent developments.

There’s some very interesting visualisations of the Collatz Functions paths as a Collatz tree, which is a rather aesthetically-appealling visual object. Here’s Algoritmarte’s 3D live render, based on a discussion from the Numberphile channel (for Numberphile’s original video, done by hand, see the fn):²

But Collatz conjecture is an unsolved problem that looks like it should certainly be solvable. It seems quite trivial in some sense: inductive logic would seem to suggest that the same basic patterns would repeat, even if there are more of them, or they are longer, given that every number that’s been tested has converged to 1 (or, more precisely, fallen into the “4-2-1” loop) and so this tempts people in trying to solve it. But Paul Erdős said of it “Mathematics may not be ready for such problems”.³

Well, we’ll not try to solve it, but rather use it for thinking about issues of recursion and long calculations in Emacs Lisp, and doing some practical engineering-type testing of different methods of dealing with such functions.

One thing we might note, as a sort of aside, is that The Little Schemer‘s C isn’t properly the Collatz function. The conjecture is really about whether the process will ever reach the repeating “4-2-1” loop, because the process of dividing even numbers by 2 and multiple odd numbers and adding 1 to them would actually mean that once we’ve reached 1, we need to multiple it by 3 and add 1, which means we’re back at 4, which is even, so we’ll divide by 2 and get 2, which is even, so we divide by 2 and get 1, which is odd, so we…. And so on.

We can write this “real Collatz” function:

;; -*- lexical-binding: t; -*-
(defun real-collatz (n)
  "For an integer `n', if it's evan, divide it by 2;
if it's odd, multiple it by 3 and add 1."
  (cond
   ((evenp n)
    (real-collatz (/ n 2)))
   (t (real-collatz (1+ (* n 3))))))

If we now try to feed it 9, we’ll hit the lisp-nesting error, but if we turn on toggle-debug-on-error, we can inspect what’s going on, and it’s exactly the “4-2-1” loop we predicted:

(real-collatz 9) ; cl-prin1: (error "Lisp nesting exceeds ‘max-lisp-eval-depth’")

;; DEBUG:
....
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(8)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(16)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(5)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(10)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(20)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(40)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(13)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(26)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(52)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(17)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(34)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(11)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(22)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(7)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(14)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(28)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(9)
  eval((real-collatz 9) nil)
  elisp--eval-last-sexp(nil)
.....

We’ll make use of it later on though, somewhat (essentially with the n=1 condition displaced.)

Dealing with long winding paths, through hailstorms

Ok, so two things. One, let’s not just print things (longer paths are going to make his quite messy), but return a list of the steps our function goes through and a count of them. We do still want to be able to inspect the path patterns. (Wikipedia notes on this “[t]he sequence of numbers involved [in the paths] is sometimes referred to as the hailstone sequence, hailstone numbers or hailstone numerals (because the values are usually subject to multiple descents and ascents like hailstones in a cloud)".)

And, two, let’s use the cl-labels trick from last time to get tail-call optimisation in Emacs.

(And a minor thing: not bother with a separate evenp function; just use the calculation directly.)

;; -*- lexical-binding: t; -*-
(defun collatz-cllabels-tco (n)
  "Calculates the 'Collatz path' for `n',
return a list of all of the steps cons'ed
with count of the steps in the path.

[Uses `cl-labels' to get tail-call optimisation.]"
  (cl-labels ((collatz* (n r)
                  (setq r (cons n r))
                  (cond
                     ((= n 1)
                      r)
                     ((= (% n 2) 0)
                        (collatz* (/ n 2) r))
                     (t
                        (collatz* (1+ (* 3 n)) r)))))
    (let* ((clst (collatz* n nil))
          (clngth (1- (length clst))))
      (cons (nreverse clst) clngth))))

Here you will note that we’ve got our collatz* function (defined in cl-labels) being self-recursive only properly in tail positions, so it can be tail-call optimised. We start by calling collatz* with arguments n (whatever value we passed for n) and nil, with collatz* taking two arguments, n and r and setting r to the cons of n with r (we’ll pass this value on, collecting the steps), and then implementing the rest of the Collatz function in the same way as previously, except passing along the r value as our collector. At the end, we’ll count the steps and cons the list of the steps themselves, using nreverse to flip the list over (because it’ll have the last results on the “outer” left and the first results on the “inner” right), with the step count.

Some examples:

(collatz-cllabels-tco 9) ; ((9 28 14 7 22 11 34 17 52 26 13 40 20 10 5 16 8 4 2 1) . 20)
                ; path from 9 to 1; 20 steps

(collatz-cllabels-tco 21) ; ((21 64 32 16 8 4 2 1) . 8)
                 ; path from 21 to 1; 8 steps

Excellent, but what about “63728127”?:— the number that tripped us up before:

;; -*- lexical-binding: t; -*-

(collatz-cllabels-tco 63728127) ; it works!

;; ...but the result is quite long, you can try it yourself;
;; use setq to be able to see all of it, e.g.,
;; (setq c-result (collatz-cllabels-tco 63728127)).

;; for, let's just see how many steps it is:
(defun collatz-return-steps (n)
  "Output just the number of steps that
`n' takes to reach 1."
  (format "%s takes %s steps to reach 1"
          n
          (cdr
           (collatz-cllabels-tco n))))


(collatz-return-steps 63728127) ; =
"63728127 takes 949 steps to reach 1"

“Let’s try it with a bigger number!", you say. Well, we can try it on most-positive-fixnum (which is I think probably set to 2305843009213693951 by default) — but remember the size of the number and the number of steps is not a direct linear relation, so you might be disappointed:

(collatz-return-steps most-positive-fixnum) ; =
"2305843009213693951 takes 860 steps to reach 1"

This is less than for 63728127.

Ok, so let’s make something really big then. 2^1000 + 1.

(collatz-return-steps (1+ (expt 2 1000))) ; =
"10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069377 takes 7248 steps to reach 1"

It takes 7,248 steps. What about 2^10000 + 1?

(collatz-return-steps (1+ (expt 2 10000))) ; =
"19950631168807583848837421626835850838234968318861924548520089498529438830221946631919961684036194597899331129423209124271556491349413781117593785932096323957855730046793794526765246551266059895520550086918193311542508608460618104685509074866089624888090489894838009253941633257850621568309473902556912388065225096643874441046759871626985453222868538161694315775629640762836880760732228535091641476183956381458969463899410840960536267821064621427333394036525565649530603142680234969400335934316651459297773279665775606172582031407994198179607378245683762280037302885487251900834464581454650557929601414833921615734588139257095379769119277800826957735674444123062018757836325502728323789270710373802866393031428133241401624195671690574061419654342324638801248856147305207431992259611796250130992860241708340807605932320161268492288496255841312844061536738951487114256315111089745514203313820202931640957596464756010405845841566072044962867016515061920631004186422275908670900574606417856951911456055068251250406007519842261898059237118054444788072906395242548339221982707404473162376760846613033778706039803413197133493654622700563169937455508241780972810983291314403571877524768509857276937926433221599399876886660808368837838027643282775172273657572744784112294389733810861607423253291974813120197604178281965697475898164531258434135959862784130128185406283476649088690521047580882615823961985770122407044330583075869039319604603404973156583208672105913300903752823415539745394397715257455290510212310947321610753474825740775273986348298498340756937955646638621874569499279016572103701364433135817214311791398222983845847334440270964182851005072927748364550578634501100852987812389473928699540834346158807043959118985815145779177143619698728131459483783202081474982171858011389071228250905826817436220577475921417653715687725614904582904992461028630081535583308130101987675856234343538955409175623400844887526162643568648833519463720377293240094456246923254350400678027273837755376406726898636241037491410966718557050759098100246789880178271925953381282421954028302759408448955014676668389697996886241636313376393903373455801407636741877711055384225739499110186468219696581651485130494222369947714763069155468217682876200362777257723781365331611196811280792669481887201298643660768551639860534602297871557517947385246369446923087894265948217008051120322365496288169035739121368338393591756418733850510970271613915439590991598154654417336311656936031122249937969999226781732358023111862644575299135758175008199839236284615249881088960232244362173771618086357015468484058622329792853875623486556440536962622018963571028812361567512543338303270029097668650568557157505516727518899194129711337690149916181315171544007728650573189557450920330185304847113818315407324053319038462084036421763703911550639789000742853672196280903477974533320468368795868580237952218629120080742819551317948157624448298518461509704888027274721574688131594750409732115080498190455803416826949787141316063210686391511681774304792596709377 takes 72378 steps to reach 1"

72,378 steps.

Can we go one bigger?

(collatz-return-steps (1+ (expt 2 100000)))
;; Debugger entered--Lisp error: (overflow-error)

No. But it’s because Emacs has maximum-integer-width set to 65536 (so a number can only be 65,536 digits long). We can set it to something bigger, and it’ll work.

(let ((integer-width 99999999))
  (collatz-return-steps (1+ (expt 2 100000)))) ; =

;; I'll spare you the number `n' itself as it's realy quite long,
;; but that number takes 717,858 steps to reach 1.

You’ll note, if you’ve been evaluating along in your own Emacs that these results have been nearly instantaneous so far. This one takes a bit longer.

(I subsequently tried doing collatz-cllabels-tco on 2^1000000 + 1, but with only 16Gb in my machine, Linux’s OOM-killer killed Emacs before it reached the end. I should probably try setting up some sort of swap better.)

Streaming though the hail

We can also try the stream package we looked at before, which creates “lazy” conses, of potentially infinite sequences, which can be evaluated one piece at a time, as needed.

(defun collatz-stream (n)
  "For a number `n', return `1' if it's `1'; otherwise,
if it's even, run the same function on half of `n';
else, run the same function on one more than three times
`n'.
(Also, collect each step into a list and, when `n' is `1',
print the list.)
[Uses the `stream' package for lazy-evaluated conses.]"
  (cl-labels ((collatz* (n)
                (stream-cons n
                             ;; note that this is the `real' collatz func
                             ;; no checking for 1 here; we do it below
                              (cond
                               ((= (% n 2) 0)
                                (collatz* (/ n 2)))
                               (t (collatz* (1+ (* 3 n))))))))
    (let ((clst (collatz* n))
          (last 0)
          (out nil))
      (while (not (= last 1)) ;; stop when we reach 1
        (setq last (stream-pop clst))
        (setq out (cons last out)))
      (let* ((steps (1- (length out)))
            (rlist (nreverse out)))
        (cons rlist steps)))))

This, however, turns out to in fact be much slower than our cl-labels tco recursion.

In fact, we can quantify this a bit further using Adam Porter‘s bench-multi macro:

(bench-multi
  :times 5
  :forms (("collatz-stream" (let ((integer-width 99999999))
            (collatz-stream (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco" (let ((integer-width 99999999))
            (collatz-cllabels-tco (1+ (expt 2 10000)))))))

Here the forms are ranked in order by fastest first; the second column says how much slower the form is compared against the fastest; then there are total runtime (for all iterations; we did here 5 for each); and total garbage collections performed by Emacs during those tests; and the total amount of time the garbage collection tasks themselves took.

I’m not quite sure why collatz-stream is this much slower than collatz-cllabels-tco is the case. My guess is that because of the delayed/lazy evaluation, we aren’t able to get real reduction of stack frames because we’re only evaluating one piece/instance of the Collatz function on each stream-pop, and there’s no advantage to using cl-labels here really.

The stream method certainly triggers more garbage collection, which is part of it, though I’m not entirely sure why it should involve triggering more garbage collection. (I suppose we could play with garbage collection settings in emacs and see if we can deal garbage collection and make the stream method better.)

Caught in loops still?

But what about a plain looping method for the Collatz function. We generally saw with the Fibonacci series that loops seemed more efficient.

We can try with both plain while loops and also cl-loops:

(defun collatz-while-loop (n)
  "Collatz function using emacs `while' loop."
  (let ((m n)
        (clst (list n)))
    (while (not (= m 1))
      (setq m
            (cond
             ((= (% m 2) 0)
              (/ m 2))
             (t (1+ (* 3 m)))))
      (setq clst (cons m clst)))
          (let* ((steps (1- (length clst)))
            (rlist (nreverse clst)))
            (cons rlist steps))))

(defun collatz-cl-loop (n)
  "Collatz function using `cl-loop' with
`while' clause condition."
  (let ((m n)
        (clst (list n)))
    (cl-loop while (not (= m 1))
             do (progn
                  (setq m
                        (cond
                         ((= (% m 2) 0)
                          (/ m 2))
                         (t (1+ (* 3 m)))))
                  (setq clst (cons m clst)))
             (let* ((steps (1- (length clst)))
                    (rlist (nreverse clst)))
               (cons rlist steps))))

And now comparing all of the methods:

(bench-multi
  :times 5
  :forms (("collatz-stream"
           (let ((integer-width 99999999))
             (collatz-stream (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco (1+ (expt 2 10000)))))
          ("collatz-while-loop"
           (let ((integer-width 99999999))
             (collatz-while-loop (1+ (expt 2 10000)))))
          ("collatz-cl-loop"
           (let ((integer-width 99999999))
             (collatz-cl-loop (1+ (expt 2 10000)))))))

We find the following results:

Both types of loops turn out faster than even collatz-cllabels-tco, with the plain emacs while loop method seeming slightly faster than the cl-loop method, but probably within margin of error (or maybe just a bit of extra machine in cl-loop which loses a small amount of efficiency.)

For our TCO tail-recurive cll maybe it’s the additional garbage collection that slows down collatz-cllabels-tco.

What if we just look at the two loop methods, and run more trials?

(bench-multi
  :times 100
  :forms (("collatz-while-loop"
           (let ((integer-width 99999999))
             (collatz-while-loop (1+ (expt 2 10000)))))
          ("collatz-cl-loop"
           (let ((integer-width 99999999))
             (collatz-cl-loop (1+ (expt 2 10000)))))))

Our results are:

Still the plain while is slightly faster than cl-loop.

An attempt at promoting our recursive tco function

I’d really like to somehow improve our recursive TCO collatz function.

What if we pull the collector r out in a higher-scoping let and so don’t have to pass it into collatz* each time; thus:

(defun collatz-cllabels-tco-improved (n)
  "Calculates the 'Collatz path' for `n',
return a list of all of the steps cons'ed
with count of the steps in the path.

[Uses `cl-labels' to get tail-call optimisation.]"
  (let ((r nil))
    (cl-labels ((collatz* (n)
                    (setq r (cons n r))
                    (cond
                       ((= n 1)
                        r)
                       ((= (% n 2) 0)
                          (collatz* (/ n 2)))
                       (t
                          (collatz* (1+ (* 3 n)))))))
      (let* ((clst (collatz* n))
            (clngth (length clst)))
        (cons (nreverse clst) clngth)))))

(bench-multi
  :times 10
  :forms (("collatz-cllabels-tco"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco-improved"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco-improved (1+ (expt 2 10000)))))))

Here are the results:

We do seem to eke out a bit more performance.

But compared to the loops, it’s still not as performant:

(bench-multi
  :times 10
  :forms (("collatz-cllabels-tco"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco-improved"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco-improved (1+ (expt 2 10000)))))
          ("collatz-while-loop"
           (let ((integer-width 99999999))
             (collatz-while-loop (1+ (expt 2 10000)))))
          ("collatz-cl-loop"
           (let ((integer-width 99999999))
             (collatz-cl-loop (1+ (expt 2 10000)))))))

We seem to have done a bit better. Now collatz-cllabels-tco-improved is only 1.26x slower than the fastest, collatz-while-loop. We save an extra gc over the older collatz-cllabels-tco, but still have to run an additional one compared to the two loop functions.

Putting all of the methods we’ve come up with together and running more trials:

(bench-multi
  :times 100
  :forms (("collatz-stream"
           (let ((integer-width 99999999))
             (collatz-stream (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco-improved"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco-improved (1+ (expt 2 10000)))))
          ("collatz-while-loop"
           (let ((integer-width 99999999))
             (collatz-while-loop (1+ (expt 2 10000)))))
          ("collatz-cl-loop"
           (let ((integer-width 99999999))
             (collatz-cl-loop (1+ (expt 2 10000)))))))

Here, interestingly, we end up with collatz-cl-loop slightly edging out collatz-while-loop. And our favourite (well, my favourite) collatz-cllabels-tco-improved function doesn’t do too poorly: it’s only x1.28 slower than the fastest (collatz-cl-loop); and even the old collatz-cllabels-tco with additional argument passing is only x1.53 slower

The collatz-stream function remains significantly slower, x19 so.

There is an obvious correlation between runtime and garbage collection, (19 vs 20 vs 33 vs 47 vs 961), so again possibly tweaking garbage collection could change things. Although, if we get rid of the garbage collection runtime differences we end up with:

Now collatz-while-loop is again indeed actually the fastest, but still just barely more so than collatz-cl-loop; both recursive TCO’ing cl-labels aren’t actually too much slower than the loops, though the improved version is indeed better; and collatz-stream is still quite slow in comparison. So garbage collection is part of the difference, but not the only one, especially for the streams method.

Escaping (at least some) loops: out of recursion and into… different recursion (next time)

This was really an excursus (on an excursus). And, next time — really — we’ll deal with the Y Combinator. I know this one really wasn’t a lambda calculus post at all, but an excursion on our previous excursion into recursion, here mostly practical considerations for Emacs. But recursion is crucial for the upcoming discussion of the Y Combinator, and paradoxes (and their uses). And that’ll be interesting for lambda calculus, and its connections with Lisp.

And, after we’re through with Y Combinators and Fibonacci sequences and barbers who shave everyone who does’t shave themselves, and how these things all tie together, we can get to something I’ve been working on for a while:— given that, connections aside, Lisp isn’t lambda calculus: can we come up with a good implementation of lambda calculus in Lisp?

(And specifically in Emacs Lisp: which is maybe one of the Lisps least naturally suited for this: a Scheme or Clojure would be a better choice really. But it’s more fun to try to get it to work in Elisp, with all of the additional challenges it presents.)

A couple of excerpts from previous quotes from McCarthy on the subject to set the stage:

…And so, the way in which to [be able to handle function passing/higher order functions] was to borrow from Church’s Lambda Calculus, to borrow the lambda definition. Now, having borrowed this notation, one the myths concerning LISP that people think up or invent for themselves becomes apparent, and that is that LISP is somehow a realization of the lambda calculus, or that was the intention. The truth is that I didn’t understand the lambda calculus, really. In particular, I didn’t understand that you really could do conditional expressions in recursion in some sense in the pure lambda calculus.…

[McCarthy 1978a:190¹]

…Writing eval required inventing a notation representing LISP functions as LISP data, and such a notation was devised for the purposes of the paper with no thought that it would be used to express LISP programs in practice. Logical completeness required that the notation used to express functions used as functional arguments be extended to provide for recursive functions, and the LABEL notation was invented by Nathaniel Rochester for that purpose. D.M.R. Park pointed out that LABEL was logically unnecessary since the result could be achieved using only LAMBDA — by a construction analogous to Church’s Y-operator, albeit in a more complicated way.…

[McCarthy 1978a:179¹]

Examining Church’s Y Combinator will be something we return to (probably in a number of posts), but I’ll defer discussion of it for the moment.

For now, let’s consider recursion in lisps. We’ll be talking a lot of recursion in Emacs Lisp today in fact.

Self-reference, self-embedding

Recursion is a concept or process depends on a simpler or previous version of itself. It’s ubiquitous, including in the natural world: Wikipedia notes that “Shapes that seem to have been created by recursive processes sometimes appear in plants and animals, such as in branching structures in which one large part branches out into two or more similar smaller parts. One example is Romanesco broccoli.”

Recursion is a thing I deal with a lot in my day job, as it is a feature of natural language, especially syntax and semantics. To provide a quick illustration — though this is not at all how modern generative syntax is done anymore — consider phrase structure grammar and phrase structure rules used by Noam Chomsky and his colleagues in the 1950s.

Sentences (and linguistics objects generally) have formal structure, and it is part of the productive/creative nature of language that we might envision this structure as involving abstract structure rules that can be expanded in different ways (and then have vocabulary filled in).

A phrase structure rule will generally have the form A → [B C], indicating that A may be rewritten or expanded into [B C]. (In the following, I’ll use round brackets ()'s to denote optional constituents.)

So, a subset of these rules for English might include something like (note that some things have multiple rules that can apply to them):

1. S → NP VP
2. NP → (Det) N
3. N → (AdjP)* N
4. VP → V
5. VP → V NP (NP)
6. VP → V CP
7. CP → (Comp) S
...

(Where S is “sentence”; Det is a determiner (like “the”, “a”); NP is a noun phrase; Nₙ is a noun head; AdjP is an adjective phrase; VP is a verb phrase; V is a verb head; CP is a complementiser phrase; Comp is a complementiser (like “that”).)

So we can rewrite S as NP VP (1) and then rewrite NP VP as Det N VP (2, choosing an optional Det) and then Det N VP as Det N V (4) and then insert lexical items of the appropriate category into the ‘heads’ (the non-P elements). So we might choose “the” for Det and “cat” for N and “purrs” for V, and get the sentence “the cat purrs”.

But note that some of the rules allow for expansion into elements that contain expansions back into themselves. So rule (1) allows an S to exapnd into NP VP and rule (6) allows for a VP to expand into V CP and rule (7) allows for a CP to expand into an S. At which point we can apply rule (1) again to expand the new S into NP VP, and then repeat this process as many times as we like:

S
NP VP
N VP
N V CP
N V Comp S
N V Comp NP VP
N V Comp N VP
N V Comp N V CP
N V Comp N V Comp S
N V Comp N V Comp NP VP
N V Comp N V Comp N VP
N V Comp N V Comp N V CP
N V Comp N V Comp N V Comp S
N V Comp N V Comp N V Comp NP VP
N V Comp N V Comp N V Comp N VP
N V Comp N V Comp N V Comp N V CP
N V Comp N V Comp N V Comp N V Comp S
N V Comp N V Comp N V Comp N V Comp NP VP
...

And it’s easy to imagine examples of what such a sentence could be like, e.g., “John said that Sita said that Bill said that Mary said that Ram said that Kim said that…". It won’t be infinitely long, but there’s no particular theoretical bound on how long it could be (memory processing and finite human lifespans will impose practical limits, of course).

On the formal semantics side, things are similar: consider that logical languages too (which are often used to formalise natural language semantics) allow for recursion. Propositional logic construction with Boolean operators too have no theoretical upper limits: we can write (t ↔ (p ∧ (q ∨ (r → (s ∧ ¬¬¬¬¬¬t))))),² and there’s nothing which prevents composing this bit of formalism with yet another bit, and so on.

And in mathematics and computer science, recursion is often a thing which suggests itself.

For instance, though there are other ways of calculating it, the Fibonacci sequence³, i.e., a sequence of numbers in which each element is the sum of the two elements that precede it (e.g. 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144…).

A natural way of writing an equation to calculate these is something like:

1. Fib(0) = 0 as base case 1.
2. Fib(1) = 1 as base case 2.
3. For all integers n > 1, Fib(n) = Fib(n − 1) + Fib(n − 2).

Rule 3 makes reference to itself. I.e., in order (by this method) to calculate Fib(6), you have to calculate Fib(5), for which you have to calculate Fib(4)), for which you have to calculate Fib(3), for which you have to calculate Fib(2), which you can then base on rules (1) & (2): you can add 0 and 1 (= Fib(0) and Fib(1)) together to get Fib(2), and then you can calculate Fib(3) by adding Fib(1) and Fib(2) and so on.

Cursed recursion

In early Lisp(s), despite the concern around recursion, writing recursive functions was/is not always pragmatically viable, because it can lead to stack overflows (potential infinities are hard in practice).

Scheme, a Lisp dialect originally created at MIT by Guy L. Steele, Jr. and Gerald Jay Sussman, was the first Lisp to implement tail call optimisation, which is a way of making significantly recursive functions viable, making tail calls similar in memory requirement to their equivalent loops.

Before talking (a little bit more) about tail recursion, let’s look at concrete examples of a tail recursive function and its loop equivalent. We’ve mentioned Fibonacci numbers already, so let’s try to write functions calculate these (in Emacs Lisp).

Following the mathematical abstraction for the Fibonacci sequence above, we could write a function like this:

;; -*- lexical-binding: t; -*-
(defun fib1 (n a b)
  "Calculate the first `n' Fibonacci numbers, recursively."
  (if (< n 1)
      a
    (cons a
          (fib1 (1- n) b (+ a b)))))

I’ve tried to make this as simple as possible, we could make it nicer (say, with a wrapper function or some some of flet) so that we didn’t have to pass in the initial values. But, keeping it simple (for now): fib1 is a function taking three arguments: n, the quantity of Fibonacci numbers to return; a, the first number to start with; and b, the second number to start with.

Following the schema above, we’re going to pass 0 for a and 1 for b. Let’s get the first ten numbers of the sequence, and pass 10 for n:

(fib1 10 0 1)  ; (0 1 1 2 3 5 8 13 21 34 . 55)

I know, the output is a bit ugly because we’re just cons’ing the results and so it’s not a proper list, but we’re keeping things simple.

Let’s walk through how it works. The function first looks at n, if n is less than 1, it returns a (whatever it is). If n isn’t less than 1, we return the cons of a with the result of calling fib1 itself on “n minus 1” b and “a plus b”.

So if we start with n=3, a=0, b=1, that is, evaluate (fib1 3 0 1), the function would say, well, 3 isn’t less than 1, so I’m going to create a cons with 0 and the result of calling (fib1 2 1 1) (2 = “n minus 1”, because n is currently 3; 1 because b=1; and 1 because a + b = 0 + 1 = 1).

So at this point we have a cons that looks like this:

(0 . (fib1 2 1 1))  ;; [= (cons 0 (fib1 2 1 1))]

When we evaluate (fib1 2 1 1), n is still not less than 1, so we’re going to cons the current value of a (which is 1) with another call to fib1: (fib1 1 1 2) (1 = “n minus 1”, because n is currently 2; 1 because b=1; and 2 because a + b = 1 + 1 = 2).

So now we have:

(0 1 . (fib1 1 1 2))  ;; [= (cons 0 (cons 1 (fib1 1 1 2)))]

Now we have to evaluate (fib1 1 1 2). n is still not less than 1; so we create another cons of 1 (as a is currently 1) with yet another call of fib1: (fib1 0 2 3) (0 = “n minus 1”, because n is currently 1; 2 because b=2; and 3 because a + b = 1 + 2 = 3). And so now:

(0 1 1 . (fib1 0 2 3))  ;; [= (cons 0 (cons 1 (cons 1 (fib1 0 2 3))))]

And, finally, evaluating (fib1 0 2 3), now n is less than one, so we take the first branch of the conditional and just return a, which is 2. So the result of starting with (fib1 3 0 1) is:

(0 1 1 . 2)  ;; [= (cons 0 (cons 1 (cons 1 2)))]

And you can try this with other values of n, e.g., try evaluating (fib1 100 0 1) to get the first 100 members of the sequence.⁴

But, at least for me on Emacs 30.0.93, 529 is the limit. If we try (fib1 520 0 1), the debugger pops up with a 1622 line long error, which begins:

Debugger entered--Lisp error: (excessive-lisp-nesting 1622)
  cl-print--cons-tail(excessive-lisp-nesting (1601) #<buffer  *temp*>)
  #f(compiled-function (object stream) #<bytecode 0x491d55561699a09>)(#0 #<buffer  *temp*>)
  apply(#f(compiled-function (object stream) #<bytecode 0x491d55561699a09>) (#0 #<buffer  *temp*>))
  #f(compiled-function (&rest args) #<bytecode 0x1c31d892b8046a8b>)()
  #f(compiled-function (cl--cnm object stream) #<bytecode 0x7c361f66f109692>)(#f(compiled-function (&rest args) #<bytecode 0x1c31d892b8046a8b>) #0 #<buffer  *temp*>)
  apply(#f(compiled-function (cl--cnm object stream) #<bytecode 0x7c361f66f109692>) #f(compiled-function (&rest args) #<bytecode 0x1c31d892b8046a8b>) (#0 #<buffer  *temp*>))
  #f(compiled-function (object stream) #<bytecode 0x1f277a9a6dc403fa>)(#0 #<buffer  *temp*>)
  apply(#f(compiled-function (object stream) #<bytecode 0x1f277a9a6dc403fa>) #0 #<buffer  *temp*>)
  cl-print-object(#0 #<buffer  *temp*>)
  cl-prin1(#0 #<buffer  *temp*>)
  backtrace--print(#0 #<buffer  *temp*>)
  cl-print-to-string-with-limit(backtrace--print #0 5000)
  backtrace--print-to-string(#0 nil)
  backtrace-print-to-string(#0)
  debugger--insert-header((error #0 :backtrace-base eval-expression--debug))
  #f(compiled-function () #<bytecode 0x299e53d67d1ac62>)()
  backtrace-print()
  debugger-setup-buffer((error #0 :backtrace-base eval-expression--debug))
  debug(error #0 :backtrace-base eval-expression--debug)
  eval-expression--debug(#0)
  (if (< n 1) a (cons a (fib1 (1- n) b (+ a b))))
  fib1(0 259396630450514843945535792456880074043523940756078363514486570322782139633750401577338505233670220572153381665 419712564636128966418863068957011388899128076671547993021605479585858227224221424221791102364954108601240491394)
  (cons a (fib1 (1- n) b (+ a b)))
  (if (< n 1) a (cons a (fib1 (1- n) b (+ a b))))
  fib1(1 160315934185614122473327276500131314855604135915469629507118909263076087590471022644452597131283888029087109729 259396630450514843945535792456880074043523940756078363514486570322782139633750401577338505233670220572153381665)
  (cons a (fib1 (1- n) b (+ a b)))
  (if (< n 1) a (cons a (fib1 (1- n) b (+ a b))))
  fib1(2 99080696264900721472208515956748759187919804840608734007367661059706052043279378932885908102386332543066271936 160315934185614122473327276500131314855604135915469629507118909263076087590471022644452597131283888029087109729)
....

Because we’ve built up a long, deeply-embedded list of conses and Emacs has a limit of how deep it’s willing/able to go (you can see above that we’ve almost made it to the end, just needing to calculate fib1(0), when Emacs decides it’s had enough).

In Scheme and elsewhere, self-recursive calls at the ends (“tails”) of functions can be optimised to avoid these sorts of stack overflows (excessive-lisp-nesting).⁵ Tail-call optimisation lets procedure calls in tail positions be treated a specialised GOTO statements, which can be efficiently processed:

…only in cases where structures are explicitly declared to be dynamically referenced should the compiler be forced to leave them on the stack in an otherwise tail-recursive situation. In general, procedure calls may be usefully thought of as GOTO statements which also pass parameters, and can be uniformly encoded as JUMP instructions. This is a simple, universal technique, to be contrasted with […] more powerful recursion-removal techniques…

[Steele 1977:155⁶]

But our fib1 function doesn’t do this, and we end up flooded the stack with too many conses.

Back to loops

Recursive functions are perhaps most idiomatic in Scheme (among lisps, I mean). Some implementations of Common Lisp can do tail-call optimisation, but loops are perhaps more common, and certainly in Emacs Lisp (for reasons you can see above), loops are usually what are used. And so we can write a new Fibonacci function with a loop. It won’t be nearly as pretty, but it’ll work better.

Here’s one possible implementation:

;; -*- lexical-binding: t; -*-
(defun fib2 (n a b)
  "Calculate the first `n' Fibonacci numbers,
in a loop."
  (let ((result nil))
    (dotimes (c n)
      (setq result (cons a result))
      (setq tempa b)
      (setq b (+ a b))
      (setq a tempa))
    (nreverse result)))

(fib2 10 0 1) ; (0 1 1 2 3 5 8 13 21 34)

(setq bigfib-50000 (fib2 50000 0 1)) ; this will work - we can get 50,000 numbers

The result is prettier at least: a proper rather than an improper list (because we started by cons'ing onto an empty list). Our fib2 function itself isn’t as mathematically pleasing as our fib1 function, we end up with a lot of setq's (the nreverse at the end reverses our list, because the way we build up our list is by cons'ing the first results first, so they end up at the end until we flip them with nreverse). But it works well. If you try to (fib2 100000 0 1), it’ll fail, but not because of stack overflow, just because we end up with numbers that are too big for Emacs. But you can certain get the over 50,000 members of the Fibonacci sequence, which is much better than fib1's limit of 529.

And dotimes is just one loop procedure available. (See cl-loop for a more powerful one.)

Optimal Tail with Emacs

Ok, so, practically, we should probably generally prefer loops over tail-recursive functions in Emacs. But, what if we just like the latter more?⁷ Are there any other possibilities?

Wilfred Hughes has an emacs package tco.el which implements a special macro for writing tail-recursive functions.⁸ It works by replacing each self-call with a thunk, and wrapping the function body in a loop that repeatedly evaluates the thunk. Thus a function foo defined with the defun-tco macro:

(defun-tco foo (...)
  (...)
  (foo (...)))

would be re-written as:

(defun foo (...)
   (flet (foo-thunk (...)
               (...)
               (lambda () (foo-thunk (...))))
     (let ((result (apply foo-thunk (...))))
       (while (functionp result)
         (setq result (funcall result)))
       result)))

And this delays evaluation in such a way as to avoid stack overflows. Unfortunately, at least currently for me (Emacs 30.0.93 again), tco.el seems to have some issues.

In Emacs 28.1, cl-labels (one of the ways of sort of doing let's for functions) gained some limited tail-call optimisation (as did named-let, which uses cl-labels).

;; -*- lexical-binding: t; -*-
(defun fib3 (n)
  "Calculate the first `n' Fibonacci numbers,
recursively, with limited tail-call optimisation
through `cl-labels'?!"
  (cl-labels ((fib* (n a b)
                (if (< n 1)
                    a
                  (cons a
                        (fib* (1- n)
                              b
                              (+ a b))))))
    (fib* n 0 1)))

(setq bigfib3 (fib3 397)) ; 396 highest that works

At least the way I’ve written it, it seems to suffer an overflow even sooner (at 397 rather than 529 as for our fib1), because it has to come back and do the cons after the tail-call — so fib* isn’t actually in the tail. (We can fix this in at least one way, which we’ll do in fib5.)

We could try to write an accumulator as a hack, where we try to do ours conses one at a time and pass along the results, but this fares no better than our fib1:

;; -*- lexical-binding: t; -*-
(defun fib4 (n a b accum)
  "Calculate the first `n' Fibonacci numbers, recursively,
but collect conses as we go and keep track of the length of
the `accum' cp. against `n'."
  (let* ((accum (cons a accum))
         (accum-lng (length accum)))
    (if (< n accum-lng)
        (nreverse accum)
      (fib4 n b (+ b a) accum))))

(setq bigfib4-529 (fib4 529 0 1 nil)) ; last good
(setq bigfib4-530 (fib4 530 0 1 nil)) ; overflows

If we combine cl-labels and the accumulator trick, however, we do seem to be able to escape stack overflows, because now we’ve got fib* properly in the tail:

;; -*- lexical-binding: t; -*-
(defun fib5 (n)
  "Calculate the first `n' Fibonacci numbers, recursively,
using both cl-labels and the accumulator trick."
  (cl-labels ((fib* (a b accum)
                   (let* ((accum (cons a accum))
                         (accum-lng (length accum)))
                     (if (< n accum-lng)
                         (nreverse accum)
                         (fib* b (+ b a) accum)))))
            (fib* 0 1 nil)))

(setq bigfib5-10000 (fib5 10000)) ; ok
(setq bigfib5-50000 (fib5 50000)) ; very slow, but ok

Now we’re back in the realms of what our fib2 non-recursive loop-style function could do. Although (setq bigfib5-50000 (fib5 50000)) calculates very slowly (worse than our looping fib2), so that’s not ideal.

Stream of Conses-ness

But here’s another possibility: Nicholas Pettton‘s stream package for emacs, where “streams” are delayed evaluations of cons cells.

;; -*- lexical-binding: t; -*-
(defun fib6 (n)
  "Return a list of the first `n' Fibonacci numbers,
implemented as stream of (delayed evaluation) conses."
  (cl-labels ((fibonacci-populate (a b)
                (stream-cons a (fibonacci-populate b (+ a b)))))
    (let ((fibonacci-stream
           (fibonacci-populate 0 1))
          (fibs nil))
      (dotimes (c n)
        (setq fibs (cons (stream-pop fibonacci-stream) fibs)))
      (nreverse fibs))))

(setq fib6-10k (fib6 10000)) ; ok
(setq fib6-50k (fib6 50000)) ; little slow, but works
(setq fib6-100k (fib6 100000)) ; little slow & overflow error

This works well. (fib6 50000) still turns out to run a bit slower than our (fib2 50000), so loops are still probably more efficient, but streams are pretty interesting. They can can used to represent infinite sequences. So here, above, fibonacci-stream (set by (fibonacci-populate 0 1)) is actually an infinite stream of Fibonaccis numbers, but lazily evaluated, so we just get the next one each time we call stream-pop on our fibonacci-stream local variable. (What happens is that stream-pop takes the car of fibonacci-stream, evaluates and returns it, and then sets fibonacci-stream to be its cdr (i.e., popping off and “discarding” the first element; which which captured in our fibs collector.))

Cascades of Fibonacci numbers

Oh, incidentally and irrelevantly, if you inspect the contents of your fib6-50k, it’s very aesthetically pleasing, a cascade of numbers:

excessive-lisp-nesting: (overflow-error)

I had hoped to get to the Y Combinator today (and think I might have suggested a promise of that), for that’s where things really get interesting. And we need to get back to lambda calculus, of course. But we may be near the limits of excessive lisp nesting ourselves here.

However, the recursion discussion here has set the stage for the Y Combinator, which we’ve already talked a couple of times, especially in connection to John McCarthy’s claims about “not really understanding” lambda calculus and the fact that these really centre on his not seeing how one could get recursion without direct Self-reference (and thus the need for LABEL) because of not knowing about the Y Combinator.

And, the Y Combinator ties in with all sorts of other curious things. Paradoxes, types, calligraphy.

[Thus, I ended up with an excursus on this excursus as the next post. I’ll put a link here to the proper third part when it’s up.]

A particular application of lambda calculus is a very salient part of my “day job” as a formal semanticist of natural language. And my interests in Emacs and lisp(s) feel like they tie in here as well—though that’s a question in itself which is probably as good of a starting point into this (planned) series of posts as any.

There is much to explore: origins of John McCarthy‘s Lisp and Alonzo Church‘s lambda calculus; encodings of the simple made complex by restriction to a limited set of tools; recursion, fixed points, and paradoxes; infinities, philosophy, and engineering. But much of this requires stage setting.

And finding an exact entry point is yet tricky. But perhaps we start with λ: the divining rod, the wizard’s crooked staff, as it is the key component of much magic of a sort.

Lisp: LAMBDA the Ultimate?

We’ll start on the programming side, before turning to more philosophical or mathematical abstractions, with lambda.

It is now not only Lisps which contain lambda as a keyword, many/most programming languages have lambda as a keyword, usually for the introduction of an anonymous function. That is, an unnamed function, sometimes for one-off use.

But in McCarthy’s original formulation of LISP in 1958, LAMBDA was used as the basis for the implementation of functions generally:

Let f be an expression that stands for a function of two integer variables. It should make sense to write f (3, 4) and the value of this expression should be determined. The expression y^2 + x does not meet this requirement; y^2 + x(3, 4) is not a conventional notation, and if we attempted to define it we would be uncertain whether its value would turn out to be 13 or 19. Church calls an expression like y2 + x, a form. A form can be converted into a function if we can determine the correspondence between the variables occurring in the form and the ordered list of arguments of the desired function. This is accomplished by Church’s λ-notation. [p.6]

….

{λ[[x_1;…; x_n]; 𝓔]}∗ is (LAMBDA, (x∗_1,…, x∗_n), 𝓔∗). [p.16]

[McCarthy 1960:6, 16¹]

As in lambda calculus, LAMBDA binds variables, and replaces any occurrences of them in the scope of the operator with whatever it receives as arguments.

So the expression:

(LAMBDA (x y z) (+ (* y x) (* z x)))

if given the arguments 5, 2, 3, would replace x's with 5; y's with 2, and z's with 3:

(+ (* 2 5) (* 3 5))  ;; = (+ 10 15) = 25

The illusion of a blue-suffused platonic universe of car's

The origin of lambda keywords in LISP, and the origin of LISP’s LAMBDA in lambda calculus has suggested the idea that LISP was something like an implementation of lambda calculus as a programming language, and the certain mysticism² attaching to both suggests perhaps a tighter surface association than there is direct evidence for.

This is the topic of a 2019 blog post based on his talk for the Heart of Clojure conference in Daniel Szmulewicz expands on the theme “Lisp is not a realization of the Lambda Calculus”.³ One of the points Szmulewicz draws attention to is McCarthy’s own words:

…one of the myths concerning LISP that people think up or invent for themselves becomes apparent, and that is that LISP is somehow a realization of the lambda calculus, or that was the intention. The truth is that I didn’t understand the lambda calculus, really.

[McCarthy 1978b:190⁴]

In the paper version of the talk, McCarthy makes a similar point, but it’s worth looking at it in the larger context:

…how do you talk about the sum of the derivatives, and in programming it, there were clearly two kinds of programs that could be written.

One is where you have a sum of a fixed number of terms, like just two, where you regard a sum as a binary operation. And then you could write down the formula easy enough. But the other was where you have a sum of an indefinite number of terms, and you’d really like to make that work too. To make that work, what you want to be able to talk about is doing the same operation on all the elements of a lit. You want to be able to get a new list whose elements are obtained from the old list by just taking each element and doing a certain operation to it.

In order to describe that, one has to have a notation for functions. So one could write this function called mapcar. This says, “Apply the function f to all the elements of the list.” If the list is null then you’re going to get a NIL here. Otherwise you are going to apply the function to the first element of the list and put that onto a front of a list which is obtained by doing the same operation again to the rest of the list. So that’s mapcar. It wasn’t called mapcar then. It was called maplist, but maplist is something different, which I will describe in just a moment.

That was fine for that recursive definition of applying a function to everything on the list. No new ideas were required. But then, how do you write these functions?

And so, the way in which to do that was to borrow from Church’s Lambda Calculus, to borrow the lambda definition. Now, having borrowed this notation, one the myths concerning LISP that people think up or invent for themselves becomes apparent, and that is that LISP is somehow a realization of the lambda calculus, or that was the intention. The truth is that I didn’t understand the lambda calculus, really. In particular, I didn’t understand that you really could do conditional expressions in recursion in some sense in the pure lambda calculus. So, it wasn’t an attempt to make the lambda calculus practical, although if someone had started out with that intention, he might have ended up with something like LISP.

[McCarthy 1978a:189–190⁵]

The Discovery of LISP

Two bits from the end of this I want to highlight. The first, well, it’ll come up in future posts, and probably later in this post itself, and it has to do with recursion:

“I didn’t understand that you really could do conditional expressions in recursion in some sense in the pure lambda calculus”

And the second is that McCarthy hedges his “LISP as a realisation of the lambda calculus is myth” stance slightly:

“So, it wasn’t an attempt to make the lambda calculus practical, although if someone had started out with that intention, he might have ended up with something like LISP."

This does fit rather well with (and perhaps suggested) the framing that Paul Graham does of McCarthy as the “discoverer” of Lisp — like Euclid of geometry — rather than its inventor in his “The Roots of Lisp” paper:

In 1960, John McCarthy published a remarkable paper in which he did for programming something like what Euclid did for geometry. He showed how, given a handful of simple operators and a notation for functions, you can build a whole programming language. He called this language Lisp, for “List Processing,” because one of his key ideas was to use a simple data structure called a list for both code and data.

It’s worth understanding what McCarthy discovered, not just as a landmark in the history of computers, but as a model for what programming is tending to become in our own time. …. In this article I’m going to try to explain in the simplest possible terms what McCarthy discovered. The point is not just to learn about an interesting theoretical result someone figured out forty years ago, but to show where languages are heading. The unusual thing about Lisp—in fact, the defining quality of Lisp—is that it can be written in itself.

[Graham 2002:1⁶ (emphasis mine)]

Graham’s paper itself, as well as some of Graham’s other publications/postings (e.g., talking about Lisp as a sort of “secret weapon” in comparison to “blub languages”)⁷ is perhaps another contributor to the mystique of Lisp. (With the flip side of “Lisp as the Cleveriest Hacker’s Secret Weapon” enchanted coin being “the Curse of Lisp”⁸.)

There are other components of the history of LISP, which are suggestive of discovery rather than invention. McCarthy’s initial aim for LISP was more akin that of the Turing Machine: as a formal abstraction describing a mathematical model whose components were simple and few but yet was capable of performing any (and all) arbitrary computational operation:

One mathematical consideration that influenced LISP was to express programs as applicative expressions built up from variables and constants using functions. I considered it important to make these expressions obey the usual mathematical laws allowing replacement of expressions by expressions giving the same value. The motive was to allow proofs of properties of programs using ordinary mathematical methods. This is only possible to the extent that side effects can be avoided. Unfortunately, side effects are often a great convenience when computational efficiency is important, and “functions” with side effects are present in LISP. However, the so-called pure LISP is free of side effects, and Cartwright (1976) and Cartwright and McCarthy (1978) show how to represent pure LISP programs by sentences and schemata in first-order logic and prove their properties. This is an additional vindication of the striving for mathematical neatness, because it is now easier to prove that pure LISP programs meet their specifications than it is for any other programming language in extensive use. (Fans of other programming languages are challenged to write a program to concatenate lists and prove that the operation is associative.)

Another way to show that LISP was neater than Turing machines was to write a universal LISP function and show that is briefer and more comprehensible than the description of a universal Turing machine. This was the LISP function eval[e,a], which computers the value of a LISP expression e, the second argument a being a list of assignments of values to variables. (a is needed to make the recursion work.) Writing eval required inventing a notation representing LISP functions as LISP data, and such a notation was devised for the purposes of the paper with no thought that it would be used to express LISP programs in practice. Logical completeness required that the notation used to express functions used as functional arguments be extended to provide for recursive functions, and the LABEL notation was invented by Nathaniel Rochester for that purpose. D.M.R. Park pointed out that LABEL was logically unnecessary since the result could be achieved using only LAMBDA — by a construction analogous to Church’s Y-operator, albeit in a more complicated way.

S.R. Russell noticed that eval could serve as an interpreter for LISP, promptly hand coded it, and we now had a programming language with an interpreter.

[McCarthy 1978:179⁵]

(There’s a lot going on in this passage, and its context.

Notions of reasons to avoid side-effects (an ideal of “functional programming”, emphasised in Haskell and Clojure (another lisp)): thus the “functional” mode of lambda calculus rather than the “everything is state” mode of Turing machines and their infinitely long memory tapes.

Recursion again (which we’ll get to, repeatedly).

And Alonzo Church, the originator (discoverer?)^{see “X” below} of lambda calculus, who we’ll talk more about soon, and who was also Alan Turing’s PhD supervisor at Princeton.

But, first: let’s turn back to the topic of the concrete instantiation of Lisp on physical hardware by Stephen Russell.)

Elsewhere, McCarthy makes clear that he hadn’t thought at that point of LISP being instantiatable but as a theoretical exploration of computing at an abstract level. But, instead, the theoretical description translated fairly easily and directly to a runnable program, a LISP interpreter:

This EVAL was written and published in the paper and Steve Russell said, “look, why don’t I program this EVAL” … and I said to him, “ho, ho, you’re confusing theory with practice, this EVAL is intended for reading, not for computing”. But he went ahead and did it. That is, he compiled the EVAL in my paper into IBM 704 machine code, fixing bugs, and then advertised this as a Lisp interpreter, which it certainly was. So at that point Lisp had essentially the form that it has today…

[McCarthy 1974a:307⁹]

“No compute. Only read.” “Hell, EVAL that.”

This is the EVAL code “in the paper” referred to above:

eval[e; a] = [
   atom [e] → assoc [e; a];

   atom [car [e]] → [
        eq [car [e]; QUOTE] → cadr [e];
        eq [car [e]; ATOM] → atom [eval [cadr [e]; a]];
        eq [car [e]; EQ] → [eval [cadr [e]; a] = eval [caddr [e]; a]];
        eq [car [e]; COND] → evcon [cdr [e]; a];
        eq [car [e]; CAR] → car [eval [cadr [e]; a]];
        eq [car [e]; CDR] → cdr [eval [cadr [e]; a]];
        eq [car [e]; CONS] → cons [eval [cadr [e]; a]; eval [caddr [e]; a]];
        T → eval [cons [assoc [car [e]; a]; evlis [cdr [e]; a]]; a]];

   eq [caar [e]; LABEL] → eval [cons [caddar [e]; cdr [e]];
                                cons [list [cadar [e]; car [e]; a]]];

   eq [caar [e]; LAMBDA] → eval [caddar [e];
                                 append [pair [cadar [e];
                                               evlis [cdr [e]; a]; a]]]

evcon[c; a] = [eval[caar[c]; a] → eval[cadar[c]; a]; T → evcon[cdr[c]; a]]

evlis[m; a] = [null[m] → NIL; T → cons[eval[car[m]; a]; evlis[cdr[m]; a]]]

Translated into slightly more familiar Lisp style (with added named-function-making label's), it is:¹⁰

(label eval
       (lambda (e a)
         (cond
           ((atom e) (assoc e a))
           ((atom (car e))
            (cond
              ((eq (car e) 'quote) (cadr e))
              ((eq (car e) 'atom)  (atom  (eval (cadr e) a)))
              ((eq (car e) 'eq)    (eq    (eval (cadr e) a)
                                          (eval (caddr e) a)))
              ((eq (car e) 'car)   (car   (eval (cadr e) a)))
              ((eq (car e) 'cons)  (cons  (eval (cadr e) a)
                                          (eval (caddr e) a)))
              ((eq (car e) 'cond)  (evcon (cdr e) a))
              ('t                  (eval (cons (assoc (car e) a)
                                               (cdr e))
                                         a))))
           ((eq (caar e) 'label)   (eval (cons (caddar e) (cdr e))
                                         (cons (list (cadar e) (car e)) a)))
           ((eq (caar e) 'lambda)  (eval (caddar e)
                                         (append (pair (cadar e)
                                                       (evlis (cdr e) a))
                                                 a))))))

(label evcon
       (lambda (c a)
         (cond ((eval (caar c) a)
                (eval (cadar c) a))
               ('t
                (evcon (cdr c) a)))))

(label evlis
       (lambda (m a)
         (cond ((null m) '())
               ('t (cons (eval  (car m) a)
                         (evlis (cdr m) a))))))

The above, given the few additional definitions for convenience immediately following, is a full LISP interpreter.¹¹

(label null
       (lambda (x)
         (eq x '())))

(label and
       (lambda (x y)
         (cond (x (cond (y 't) ('t '())))
               ('t '()))))

(label not
       (lambda (x)
         (cond (x '())
               ('t 't))))

(label append
       (lambda (x y)
         (cond ((null x) y)
               ('t (cons (car x)
                         (append (cdr x) y))))))

(label pair
       (lambda (x y)
         (cond ((and (null x) (null y)) '())
               ((and (not (atom x)) (not (atom y)))
                (cons (list (car x) (car y))
                      (pair (cdr x) (cdr y)))))))

(label assoc
       (lambda (x y)
         (cond ((eq (caar y) x) (cadar y))
               ('t (assoc x (cdr y))))))

The brevity of the code combined with the details of the story of the implementation: seemingly the theoretical code works without translation, suggesting a sort of natural discovery. But, the nature of even the theoretical, pre-implementation code did not exist in some Platonic heaven of mathematics, as can be seen by the nature of some of the operations, particularly car and cdr (often in modern Lisps, especially Scheme and Scheme-influenced Lisps, rendered instead more transparently as first and rest.)

The Non-Platonic mechanics of 1950s IBM mainframes

LISP was designed with the IBM 704-style architecture in mind, and car and cdr make some reference to particular details of the hardware, where the IBM 704 had “address” and “decrement” fields in memory index registers (≈locations in physical RAM), and car referenced the “address” field of the register (so CAR = “C-ontents of the A-ddress part of the R-egister”) and cdr the “decrement” field of the register (CDR = “C-ontents of the D-ecrement part of the R-egister”).¹²

Lists in Lisp are singly-linked lists, with each node in the list having a “value” field and a “next” field, which points to the next node; these “value” and “next” fields correspond to the car and cdr operations. So, if we have a list like (a b c), the first node in the list has a “value” field of a and a “next” field pointing at another node, with this second node actually itself being — not b — but rather the list (b c). A proper list in Lisp is nil-terminated: that is, the last item in the list is actually nil (which is the empty list '()).

The operation cons above (for CONstructor) is a pair-forming operation (where the pairs correspond to the “value” and “next” fields), returning what are variously called (in different lisps) “conses” or “pairs”. Not all conses/pairs are lists (at least in most Lisps), since a proper list in Lisp is nil-terminated.

The operation (cons 'a 'b) will return a cons'ed object with essentially a single node, where the “value” field will be a and the “next” field will be b. Lisps will usually print such non-list conses (sometimes called “improper lists”) as “dotted lists”, i.e., (cons 'a 'b) will print out as (a . b).

The list (a b c) is actually the result of doing (cons a (cons b (cons c nil))) and would be printed in dotted-pair notation as (a . (b . (c . nil))) [which is equivalent to (a . (b . (c . ()))), since nil is the empty list].

Unsurprisingly, a lot of early/traditional Lisp programming involves manipulations of lists. But all modern Lisps implement other types of data structures as well, including vectors/arrays, hash tables, objects, and so on.

But, on the main topic of (eq 'lisp 'lambdacalculus), others have also pointed out the concrete hardware-connections of LISP from early days, telling against the “Lisp-as-pure-formal-invention-discovery” or “Lisp as (semi-)direct implementation of lambda calculus” notions:

Lisp was intended to be implemented on a computer from day 0. For their IBM 704. Actually Lisp is based on earlier programming experience. From 56 onwards John McCarthy implemented Lisp ideas in Fortran & FLPL. Then 58 the implementation of Lisp was started. 59 a first runnable version was there. 1960 there was the Lisp 1 implementation. The widely known paper on the Lisp implementation and recursive function theory was published in 1960. But his original prime motivation was not to have a notation for recursive function theory, it was to have a list processing programming language for their IBM 704 for AI research.

Lisp as designed by McCarthy was very different from lambda calculus.

[going on to point to the Stoyan (1984) Early LISP history (1956 - 1959) paper.]

— lispm‘s comment on r/lisp thread on this topic

There’s obviously much more to explore for the early history of Lisp and the nature of its connections to lambda calculus, but this much at least should give a general sense of the distinctions/divergences between Lisp and lambda calculus, while not ignoring important interconnections between them.

Reaching the '() of the line

And so while it’s tempting to delve off into other interesting features (homoiconicity!) of LISP/Lisps and their history and dialects (the Common Lisp of Endor; Schemes, Rackets, Chickens and Guile (oh my!), Clojure, Fennel and others), I’d wanted to get to lambda calculus proper much earlier in this piece already, and so we’ll set our (lispy) parens down for a moment, and trade in our LAMBDA for a λ.

Cattle-prodding functions: the lambda calculus

The aforementioned Alonzo Church¹³, a Princeton mathematician who supervised 31 doctoral students during his career, influencing others important researchers (including Haskell Curry [for whom the programming language Haskell is named; as well as the operating of currying]), developed (the) lambda calculus as part of his research into the foundations of mathematics.

Lambda calculus is Turing complete, thus equivalent in computation power to a Turing machine and a universal model of computation.

There are very few bits of machinery in basic untyped lambda calculus. (A reason for which it seemed to be attractive to McCarthy as a touchstone.)

Lambda calculus has variables and lambdas; function application; a reduction operation (which may follow function application); and a convenience variable-renaming operation.

More specifically, we have:

1. variables, like x, which are characters or strings representing “a parameter”.
2. lambda abstraction: essentially just the definition of a function, specifying its input (by a bound variable, say, λx) and returning an output (say, M). E.g., the expression λx.M will take an input, and replace any and all instances of x in the body M¹⁴ with whatever the input was. (The . separates the lambda and specification of bound variable (here x) (“input taker”) from the body (the “output”).)
3. function application: a representation like (M N), the function M applies to N, where both M and N are some sort of lambda terms.
4. β-reduction (beta reduction): bound variables inside body of the expression are replaced by inputs “taken” by the lambda expressions. The basic form: ((λx.M) N) → (M[x := N]). (That is, an expression (λx.M) combining with an expression N returns (M) where where all instances of x inside of M are replaced with N.)

(Setting aside rule 4 for the moment.)

For example, we might have an expression:

λf.λx.(f x)

This would be an expression which combines, one at a time, with two inputs, the lambdas operating from left to right, and then applying the f input to the x input.

To see this in full working order, we need to specific one of the two remaining operations (both reduction operations):

So, turning to rule 4 for β-reduction, taking our expression from above and providing it with inputs, and walking through the (two) application+β-reduction steps one at a time:

((λf.λx.(f x)) b a) =

((λx.(b x)) a) =

(b a)

That is, first, the leftmost lambda, λf, “takes” the leftmost argument b (in “taking” an argument, it “discharges” and disappears) and the (single) bound instance of the variable f in the body is replaced by b. Then, the same thing happens with the remaining lambda, λx, and the remaining argument, a. The result is a function where b applies to a. (Though since in this case there are no more lambdas, nothing more happens.)

Since (b a) looks somewhat unexciting/opaque, we can imagine a sort of hybrid proper untyped lambda calculus/Lisp hybrid language — let’s call this toy language of ours ΛΙΣΠ — and illustrate what things might look like there (assuming here that numbers are numbers and #'* is a lispy prefix multiplication function):

((λf.λx.λy.(f x y)) #'* 6 7) =

((λx.λy.(* x y)) 6 7) =

((λy.(* 6 y)) 7) =

(* 6 7) =

42

This isn’t how mathematics works in classical untyped lambda calculus — because we only have the 4 rules/entities enumerated above [plus a variable-clash reduction operation called α-reduction]¹⁵ and nothing else¹⁶: no integers, no stipulated mathematical operations, no car or cdr or cons or eq or anything — we can do all of these things in lambda calculus with the tools we have, and we’ll explore that in another post, but for now I just wanted to show you the toy ΛΙΣΠ language snippet as I find something that feels a bit more familiar and concrete can be helpful for understanding the notional unpinnings of what’s going on in lambda calculus.

Ok, so there’s no integers or cars, but what’s all this about cattle prods?

Well, why is λ / LAMBDA the “function”-making operator? Maybe just eeny-meeny-miney-moe amongst Greek letters, but at least at one point Church explained that [A. Church, 7 July 1964. Unpublished letter to Harald Dickson, §2] that it came from the notation “x̂” used for class-abstraction by Whitehead and Russell in their Principia Mathematica (which we’re refer to later on), by first modifying “x̂” to “^⁣x” to (and then for better visibility to “∧x”) to distinguish function-abstraction from class-abstraction, and then changing “∧” (which is similar to uppercase Greek “Λ”) to (lowercase Greek) “λ” for ease of printing (and presumably to avoid confusion with other mathematical uses of “∧”, e.g., logical AND).¹⁷ [So maybe “x̂” ⇒ “^⁣x” → “∧x” → “λx”.]

The Greek lambda (uppercase Λ, lowercase λ) as an ortheme itself derives ultimately from a Semitic abjad, specifically from the Phoenician lāmd 𐤋, which (like most letters began as a pictogram of sorts) is considered to originate from something like an ox-goad, i.e., a cattle prod, or else a shepherd’s crook, i.e., a pastoral staff. (The reconstructed Proto-Semitic word *lamed- means a “goad”.)¹⁸

Are We in Platonic Heaven Yet?

Lambda calculus, not being tied to any particular hardware and being a true formula abstraction, feels like something that might have a better claim to being something like a property of the universe that one might discover (I admit I find it hard not to feel something of the sort — but then I’ve used lambda calculus for work on natural language within a framework that wants to assign some sort of reality to our formalisations, so it’s hard not to be pulled in this direction), however, Church says (of his own formalism):

We do not attach any character of uniqueness or absolute truth to any particular system of logic. The entities of formal logic are abstractions, invented because of their use in describing and systematizing facts of experience or observation, and their properties, determined in rough outline by this intended use, depend for their exact character on the arbitrary choice of the inventor.

[Alonzo Church 1932:348¹⁹]

(This seems part of Church’s constructivist philosophy, in common with Frege.)²⁰

What’s next? : λy.(equal? (cdr L) y)

We’ve pulled at a lot of disparate threads here, trying to explore the nature of the connections between Lisp and (the) lambda calculus. Neither the idea that Lisp is a direct instantiation of lambda calculus nor the idea that McCarthy was largely ignorant of properties of lambda calculus are quite right. But that there are an interesting interplay of connections.

But. This is really to set the stage for me to talk about things I’m interested in which draw on different aspects of Lisp(s) and lambda calculus and formal or applied applications to do with one or the other.

I was going to talk about Montague Grammar, because it’s fascinating (and it’s my day job), for which lambda calculus is a crucial component.²¹ But we’re at length now, so it should be another day.

What I do want to look at next is a combination of Lisp and lambda calculus, in various ways, starting with attempts to implement aspects of lambda calculus in Emacs Lisp, and the challenges therein.

[fingers crossed that (next 'blog) does not eval to undefined.]

(Update: (eval (next 'blog)): Part 2: Recursion Excursion.)

Sometimes you need to kill something in Linux. Sometimes it makes sense to use some sort interactive process monitor, like the process table in Plasma’s System Monitor, or top or htop or bottom or some other sort of top.¹ (Or, if you’re in an X11 environment rather than a Wayland one, you could use xkill.²)

killing with killall

You can often get by with killall, e.g., if you want to kill all running Firefox applications:

killall firefox

killing with pkill

Or you could use pkill (which has a number of options) in much the same way.

These sorts of approaches don’t always work. Sometimes (this is true in Guix³ a lot) processes are named in ways that killall or pkill don’t match.

listing processes and process ids with ps

You can list running processes in Linux with ps -ef, and this will show you all running processes. It’s too much, obviously, there are a lot of things going on in your system. So, you can filter it by “piping” the output of ps -ef through grep, e.g., ps -ef | grep -i emacs, which might show you something like:

emacsomancer    15188 26384  0 21:05 ?        00:00:02 /gnu/store/5vkx1cf1d2k9dj974vgd77yx0fdis284-emacs-pdf-tools-1.1.0/bin/epdfinfo
emacsomancer    23294 26384  0 21:33 ?        00:00:00 /home/emacsomancer/.guix-home/profile/bin/emacs --quick --batch --load /home/emacsomancer/.emacs.d/eln-cache/30.0.93-e51375a4/el-job-child-8a892b0e-c7b5e3df.eln --eval (el-job-child--work #'org-node-parser--collect-dangerously nil)
emacsomancer    23550  2250  0 21:34 pts/4    00:00:00 grep --color=auto -i emacs
emacsomancer    26384     1  5 19:47 ?        00:05:42 /home/emacsomancer/.guix-home/profile/bin/emacs --daemon --debug-init
emacsomancer    26513 26384  0 19:47 ?        00:00:03 /home/emacsomancer/.guix-home/profile/bin/emacs --quick --batch --load /home/emacsomancer/.emacs.d/eln-cache/30.0.93-e51375a4/el-job-child-8a892b0e-c7b5e3df.eln --eval (el-job-child--work #'org-node-parser--collect-dangerously t)
emacsomancer    26514 26384  0 19:47 ?        00:00:02 /home/emacsomancer/.guix-home/profile/bin/emacs --quick --batch --load /home/emacsomancer/.emacs.d/eln-cache/30.0.93-e51375a4/el-job-child-8a892b0e-c7b5e3df.eln --eval (el-job-child--work #'org-node-parser--collect-dangerously t)
emacsomancer    26515 26384  0 19:47 ?        00:00:02 /home/emacsomancer/.guix-home/profile/bin/emacs --quick --batch --load /home/emacsomancer/.emacs.d/eln-cache/30.0.93-e51375a4/el-job-child-8a892b0e-c7b5e3df.eln --eval (el-job-child--work #'org-node-parser--collect-dangerously t)
emacsomancer    26516 26384  0 19:47 ?        00:00:01 /home/emacsomancer/.guix-home/profile/bin/emacs --quick --batch --load /home/emacsomancer/.emacs.d/eln-cache/30.0.93-e51375a4/el-job-child-8a892b0e-c7b5e3df.eln --eval (el-job-child--work #'org-node-parser--collect-dangerously t)
emacsomancer    26517 26384  0 19:47 ?        00:00:02 /home/emacsomancer/.guix-home/profile/bin/emacs --quick --batch --load /home/emacsomancer/.emacs.d/eln-cache/30.0.93-e51375a4/el-job-child-8a892b0e-c7b5e3df.eln --eval (el-job-child--work #'org-node-parser--collect-dangerously t)
emacsomancer    26518 26384  0 19:47 ?        00:00:02 /home/emacsomancer/.guix-home/profile/bin/emacs --quick --batch --load /home/emacsomancer/.emacs.d/eln-cache/30.0.93-e51375a4/el-job-child-8a892b0e-c7b5e3df.eln --eval (el-job-child--work #'org-node-parser--collect-dangerously t)
emacsomancer    26519 26384  0 19:47 ?        00:00:01 /home/emacsomancer/.guix-home/profile/bin/emacs --quick --batch --load /home/emacsomancer/.emacs.d/eln-cache/30.0.93-e51375a4/el-job-child-8a892b0e-c7b5e3df.eln --eval (el-job-child--work #'org-node-parser--collect-dangerously t)
emacsomancer    26750 23962  0 19:48 tty8     00:00:00 /home/emacsomancer/.guix-home/profile/bin/emacsclient -c
emacsomancer    26761 23962  0 19:48 tty8     00:00:00 /home/emacsomancer/.guix-home/profile/bin/emacsclient -c -e (mu4e)

Since you’ve piped ps through grep, you’re missing the explanatory column headers row (because that row doesn’t contain the string “emacs” and so has been filtered out by grep), which are:

UID        PID  PPID  C STIME TTY          TIME CMD

So, then, picking one of the entries above, let’s line them up:

UID             PID    PPID  C STIME TTY      TIME     CMD
emacsomancer    26384     1  5 19:47 ?        00:05:42 /home/emacsomancer/.guix-home/profile/bin/emacs --daemon --debug-init

The UID is just the name of “user” who executed the process (probably you). The CMD is what the process actually is, which is important. The other important piece here is the PID, which is the Process ID or Processor identifier, for with this you can have your system kill (or do other things) to a very specific process (usually an application, or a process it’s spun off). The other fields aren’t relevant for us here.⁴

kill (or, how to kill things nicely)

With the PID, you can do various things with a process, such as kill. This could be a simple kill 26384 to kill the emacs --daemon process.

Sometimes, processes don’t want to die and a simple kill won’t work. kill is equivalent to kill -TERM or kill -15, which essentially nicely ask the process to stop. This is often good, because if the process has some sort of “exit things” it does, like saving backups of open files, it will do those things first before stopping.

But sometimes the reason you’re in the command line in the first place is because a process is hung, and so the simple kill won’t do any good.

kill -9 (or, what if politely killing doesn’t work)

In that case you can do kill -9, which is “the unsafe way of brutally murdering a process. It’s equivalent to pulling the power cord, and may cause data corruption."⁵ This often not what you want to do, but is useful for when you need to force something to stop immediately or when it’s not responding to polite kill's. (Though kill -9 / SIGKILL won’t kill zombies, because they’re dead already and are just waiting for their parent processes to reap them.)

Advanced pkill (or, what if there are too many things to kill one by one?)

If you recall the ps -ef | grep -i emacs output from above, there are often a bunch of associated processes. I’ve often ended up having to try to run kill (or kill -9) on a bunch of PIDs one by one until finally the application shut down (cases where killall didn’t work for one reason or other).

We can instead use our process lister (ps -ef) with piping to pkill. E.g., like this in the case of killing all processes named “emacs”:

ps -ef | pkill -f emacs

Careful, because, especially if the “name” is short, it might be a substring of other running processes. (I.e., you probably don’t want to try something like ps -ef | pkill -f e. It would kill emacs --daemon, true, but it would also kill any other process with an e in it.)

You can check first by piping ps -ef through pgrep:

ps -ef | pgrep -l emacs # show the list of all the processes (and their names) to be killed first

to see a list of the names of the processes that would be killed (for our emacs example).

Or, if you need a bit more information, do instead:

ps -ef | pgrep -a emacs # show the list of all the processes (and their full command line, not just name) to be killed first

to see the full command line (including but not limited to the process’s name).

And, just like kill above, pkill is by default the “polite kill”. If you’re dealing with stubborn processes, you can add a -9 to order SIGKILL, e.g.:

ps -ef | pkill -9 -f emacs

(If you had a completely hung Emacs, say.)

What if you need to do something other than killing a bunch of things?

This is all well and good if all you need to do is kill, but sometimes there are other things to do.

For instance, I finally got Mullvad VPN‘s own graphical client to work on Guix,⁶ but for some reason the graphical interface split tunnelling doesn’t work for me on Guix (usually the Mullvad interface would allow you open a specific application outside of the VPN tunnel (i.e. on your regular connection)). Fortunately, Mullvad also has a command-line interface and one can add currently running applications to be excluded from the VPN tunnel (this is also useful even if your Mullvad GUI split-tunnelling is working for not having to shut down and re-open applications if you want them excluded from your VPN tunnel). But it does it by PID, e.g.:

mullvad split-tunnel add 26384

(if we were excluding the emacs --daemon PID from the above example from the VPN tunnel, say.)

One of the obvious things one might want to exclude from a VPN tunnel is a particular browser (e.g., there’s a site that doesn’t like the VPN, so you open up a second browser outside of the VPN to use to access that site).

But browsers often involve a number of processes, and I found myself having to manually run the mullvad split-tunnel add command on ten different PIDs, and check mullvad.net each time to see if that was the one I needed or not.

This is both time-consuming and frustrating and, just like kill'ing, we can instead do it at scale.

Now, the mullvad split-tunnel command itself doesn’t have a built-in facility to either add processes by name or more than one in a single shot, but we can write a loop over a list of PIDs and have the loop execute mullvad split tunnel add ... for each one.

In Bash (“the Bourne Again SHell”), if you wanted to exclude Firefox from the Mullvad VPN tunnel, you could do:

for pid in $(pgrep -u $(id -u) -f "firefox"); do mullvad split-tunnel add "$pid"; done

Just like for pkill, you can double-check before what would be added to the tunnel exclusion by doing one of:

ps -ef | pgrep -l firefox # show the list of all the processes (and their names) to be killed first
ps -ef | pgrep -a firefox # show the list of all the processes (and their full command line, not just name) to be killed first

If you’re using Fish shell (“FInally, a command line SHell for the 90s”), instead of the Bash line above, you would use:

for pid in $(pgrep -u $(id -u) -f "firefox"); mullvad split-tunnel add "$pid"; end

But what if you don’t want to remember arcane incantations?

I don’t, or, well, won’t. I can probably find it in my shell history once I’ve run it on a machine, but we can do better than that.

We can instead add a function to the shell.

For Bash, you would add to your ~/.bash_profile:

function mullvad-split-tunnel-by-name() {
  for pid in $(pgrep -u "$(id -u)" -f "$1")
  do
      mullvad split-tunnel add "$pid"
      printf "added to split tunnel: %s\n" "$(ps -p "$pid" -o command= | awk '{print $1}')"
  done
}

(You don’t need the printf bit if you don’t want, but this way you’ll see a proper list of the names of all of the excluded processes, which the mullvad split-tunnel add command doesn’t do by itself.)

Then (once your .bash_profile is source'ed)⁷, you could just enter the following in the terminal to have all firefox processes excluded from the Mullvad VPN tunnel:

mullvad-split-tunnel-by-name firefox

(You can choose a shorter name for your function than mullvad-split-tunnel-by-name of course, but for me such is fine with TAB completion.)

And similarly for other things you wanted to exclude from the VPN tunnel (e.g., mullvad-split-tunnel-by-name chromium).

For Fish, you would add a new file to your ~/.config/fish/functions/ directory (say ~/.config/fish/functions/mullvad-functions.fish) with the following content:

function mullvad-split-tunnel-by-name --description 'adds all matching instances to mullvads split tunnel'
  for pid in $(pgrep -u $(id -u) -f $argv)
      mullvad split-tunnel add "$pid"
      printf "added to split tunnel: %s\n" "$(ps -p $pid -o command= | awk '{print $1}')"
  end
end

(Again, the printf line is optional. And again you should source it or restart your shell.)

A couple of other useful mullvad split-tunnel functions

These can be defined in similar fashion to the above mullvad-split-tunnel-by-name: one for removing things from the VPN excluded list, and for one for showing what’s currently on the list by name. (since mullvad split-tunnel list just spits back a list of raw PIDs, which isn’t very helpful for identifying what’s actually being split tunnelled….)

• removing processes from exclude list

  Like mullvad-split-tunnel-by-name, but for removing excluded processes (i.e., re-including them in VPN tunnel):

  • In Bash:

    function mullvad-remove-process-by-name() {
       for pid in $(pgrep -u "$(id -u)" -f "$1")
       do
           mullvad split-tunnel delete "$pid"
           printf "removed from split tunnel: %s\n" "$(ps -p "$pid" -o command= | awk '{print $1}')"
       done
    }
    

  • In Fish:

  function mullvad-remove-process-by-name --description 'remove all matching instances to mullvads split tunnel'
    for pid in $(pgrep -u $(id -u) -f $argv)
        mullvad split-tunnel delete "$pid"
        printf "removed from split tunnel: %s\n" "$(ps -p $pid -o command= | awk '{print $1}')"
    end
  end
  

• listing processes in exclude list by name rather than PID

  Checking on what’s currently split out from the VPN tunnel: this provide a listing of processes currently excluded from the VPN tunnel by name (in a couple of options):

  • In Bash:

  # list full command line info associated with each PID in list
  function mullvad-list-excluded-processes-by-long-name() {
      mullvad_vpn_exclude_array=( $(mullvad split-tunnel list) )
      mvre='^[0-9]+$'
      printf "Processes excluded from VPN tunnel:\n\n"
      for key in "${!mullvad_vpn_exclude_array[@]}"
      do
          if  [[ ${mullvad_vpn_exclude_array[$key]} =~ $mvre ]]
          then
              printf "PID %s = %s\n" "${mullvad_vpn_exclude_array[$key]}" "$(ps -p "${mullvad_vpn_exclude_array[$key]}" -o command=)"
          fi
      done
  }
  
  # list just process name info associated with each PID in list
  function mullvad-list-excluded-processes-by-short-name() {
      mullvad_vpn_exclude_array=( $(mullvad split-tunnel list) )
      mvre='^[0-9]+$'
      printf "Processes excluded from VPN tunnel:\n\n"
      for key in "${!mullvad_vpn_exclude_array[@]}"
      do
          if  [[ ${mullvad_vpn_exclude_array[$key]} =~ $mvre ]]
          then
              printf "PID %s = %s\n" "${mullvad_vpn_exclude_array[$key]}" "$(ps -p "${mullvad_vpn_exclude_array[$key]}" -o comm=)"
          fi
      done
  }
  

  • Similarly for Fish:

    function mullvad-list-excluded-processes-by-long-name --description 'lists excluded processes in mullvads split tunnel by long command name'
        printf "Processes excluded from VPN tunnel:\n"
        for key in $(mullvad split-tunnel list)
            if string match -qr '^[0-9]+$' -- "$key"
                 printf "PID %s: %s\n" "$key" "$(ps -p $key -o command=)"
            end
        end
    end
    
    function mullvad-list-excluded-processes-by-short-name --description 'lists excluded processes in mullvads split tunnel by short process name'
        echo "Processes excluded from VPN tunnel:"
        for key in $(mullvad split-tunnel list)
            if string match -qr '^[0-9]+$' -- "$key"
                 printf "PID %s: %s\n" "$key" "$(ps -p $key -o comm=)"
            end
        end
    end
    

Beyond tunnelling and killing

These techniques — including defining various named speciality functions (in whatever shell) — could obviously be adapted for other similar cases of wanting to run a command on multiple processes sharing (all or part of) a name, whenever you find you’re tired of wasting time manually mucking about with heaps of PIDs.

Trying to Cope with Emacs on mobile

I’ve tried a number of different solutions for managing sync’ed Org files on mobile¹, and some of these are useful for some purposes, but to being able to access and add to my Org-Roam notes, I’ve found I really need a full-blooded Emacs instance.²

And dealing with interacting with Emacs on a touchscreen interface/touchscreen keyboard is a bit of a nightmare.

Here, we’ll try to improve it a little. Into the rabbit-hole….

Note about using Termux for Emacs on Android

You’re probably better off installing Termux from F-Droid rather than from the Google Play Store [see the note from the official Termux maintainers at: https://github.com/termux/termux-app/discussions/4000]. This is especially true is you want to use Emacs in Termux for managing Org-roam files which are shared/kept in sync with your Emacs on desktop machines, which you can do with Syncthing (use Catfriend1’s Syncthing-fork [Github]/[F-Droid] on Android), in order to be able to grant Termux (and thus Emacs) access to your shared Org files.

Termux Extra Keys

Termux has a very useful feature adding extra keys above the system keyboard. These are quite useful, including ‘sticky’ CTRL and ALT keys. (In addition, one can hold volume-down and press a key for CTRL + [that-key] or volume-up and press a key for ALT + [that-key]; with volume-up + t inputting TAB. At least on my phone, using the volume keys as modifiers is fairly awkward/uncomfortable, so I prefer these extra keys.)

[Nb.: These keys are toggled on and off by pressing volume-up + k or volume-up + q. This is important to note, for I once toggled them off by accident and couldn’t understand what had happened to them for several months.]

Also worth mentioning is the fact that:

• swipe left on extra keys brings up swipe-compatible/regular keyboard typing
• swipe right on regular keyboard typing area (where extra keys was) will get you back to extra keys

The first of these lets you use any of your keyboard’s regular features and so can be quite useful.

See the Termux manual on Extra Keys for a description and basic configuration and usage instructions.

Extra Keys for better Emacs navigation

Here, I explore the possibilities offered by the advanced configuration options for Extra Keys for navigating Emacs in Termux, especially Org-Roam or Org-node ³, to create faster/easier ways of calling commonly used commands since the user-interface to Emacs with an on-screen keyboard is often somewhat frustrating compared to the using Emacs with a physical keyboard. In addition to extra tappable keys, Extra Keys also allows for different outcomes when the user does swipe-up on a key, including the outputting of key sequences, which can include modifier keys like CTRL. This allows the possibility of loading them with Emacs key sequences which would otherwise be much slower to access with an onscreen keyboard.

What I wanted was:

• An easy, single-key (no modifier) way of doing forward and backward searches. It is painful to either hold volume-down or have to keep on toggling the Extra Keys’ CTRL for each next/previous result. So I recruited ♡ and ♤ for forward and back searches, respectively (because ♡ is sort of pointed down and ♤ is sort of pointing up), and bound these appropriately in my Emacs configuration (see below).

• other convenient single-key buffer navigation interface features, e.g., arrow keys and page-up and page-down

• some convenience things like closing all but the current window (delete-other-windows = C-x 1), opening up iBuffer, switch-buffer

• slashes and - and ~ available immediately to tap insert (not under my keyboard’s symbol toggle or long-hold)

• ability to call some Org-mode/Org-node commands quickly: including finding nodes, inserting links to nodes, capturing new dailies, committing captures, going to today’s daily entry, moving forward and backwards through the dailies, toggling the org-roam backlinks buffer

  

Example configuration

The configuration outlined below is partially specific for my Emacs configuration, especially for the C-c (CTRL c) keys, but some things involve default Emacs keybindings and it can at least serve as a model for what one can do. To configure Extra Keys, you need to modify the extra-keys part of ~/.termux/termux.properties in this fashion:

extra-keys = [[ \
  {key: 'ESC', popup: {macro: "CTRL x 1", display: "C-x 1"}}, \
  {key: 'CTRL', popup: {macro: "CTRL c", display: "C-c"}}, \
  {key: 'ALT', popup: {macro: "ALT x", display: "M-x"}}, \
  {key: '♡', popup: {macro: "CTRL x r b", display: "bookmarks"}}, \
  {key: '~', popup: {macro: "CTRL c n d", display: "today"}}, \
  {key: 'UP', popup: {macro: "CTRL c n n", display: "new note"}}, \
  {key: '-', popup: {macro: "CTRL c CTRL k", display: "cancel note"}}, \
  {key: 'PGUP', popup: {macro: "CTRL c n y", display: "prev note"}}], \
  [ \
  {key: 'TAB', popup: {macro: "CTRL x b", display: "alttab"}}, \
  {key: '/', popup: {macro: "CTRL c n f", display: "find node"}}, \
  {key: '\\\\', popup: {macro: "CTRL c n i", display: "insert link"}}, \
  {key: '♤', popup: {macro: "CTRL x CTRL b", display: "ibuffer"}}, \
  {key: 'LEFT', popup: {macro: "CTRL c n l", display: "backlinks"}}, \
  {key: 'DOWN', popup: {macro: "CTRL c CTRL c", display: "commit"}}, \
  {key: 'RIGHT', popup: {macro: "CTRL c n v", display: "goto-date"}}, \
  {key: 'PGDN', popup: {macro: "CTRL c n t", display: "next note"}}]]