emacs on The Neo-Babbage Files

Trials and Visions of Internet Relay Chat

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Fri, 28 Mar 2025 13:15:00 -0500

This was meant to be a shortish bit on a couple of points on a decent IRC (Internet Relay Chat) set-up, including:

some sort of persistence (a “bouncer” or the always-connected server)
use with some sort of IRC client in Emacs
some sort of reasonable mobile phone client

It ended up longer than I intended.

tldr

Using some of the more IRCv3 tools, particularly the cluster of things by Simon Ser and co (soju, goguma, gamja - see below) makes for a much better IRC experience, and easily supports synchronised access across multiple devices (e.g., desktop, laptops, mobile).
- soju provides a very capable bouncer
- goguma provides an excellent mobile client
- gamja is a very user-friendly web-client front-end
- and these three things can complement each other in a complete IRC setup system
we can make such a soju-based set-up work well with Emacs IRC clients
soju and gamja can be self-hosted; but there are also two paid/hosted instances currently available:
- SourceHut’s chat.sr.ht (on a fork of soju with a different feature-set)
- IRC Today’s service (I think closer to mainline soju, but not certain)
we can leverage some of the IRCv3 functions nicely in Emacs

The most useful bits here may be notes on how to set up clients to work with SourceHut’s chat.sr.ht, particularly Goguma and Emacs/ERC, and some of the cool features of IRCv3 things [modernising IRC], like the soju bouncer (which is used by chat.sr.ht), for which see the section below “Smooth & Modern IRC Implementation & Practice (mobile, Emacs)", which you might jump to if you don’t want to read a bunch of preamble.

History, theory, background

Pre-history: wee chats in musl pies

Long ago now I spent a while setting up a RaspberryPi 3b (running musl flavoured Void Linux) as a Weechat Server/Bouncer in order to make using IRC less painful. This involved a lot of steps, including setting up a NoIP script and SSL certs with Let’s Encrypt, and setting up scripts to auto-fetch new certs. But once up, it worked well (unless my home internet went out), and Weechat had a nice mobile app, so I could connect both on desktop/laptop with an Emacs IRC client and also had a pocket connection via my mobile phone ¹. This gave me a persistent connection and log history and all that.

Then I started using Matrix (=a modern, but heavy, “IRC replacement”, itself theoretically a Slack, Discord, &c. competitor) more, (in part) since some projects have chosen it as a more modern/capable alternative to IRC, and, soon after, it was the case that most of the IRC rooms I was in were bridged to Matrix anyway. And then it seemed perhaps more convenient just to access everything in one place, since some things were only Matrix and not on IRC, it made sense for that to be Matrix.

And then the Matrix bridges mostly shut down.

Path of the prodIRCal son

I toyed off and on with getting re-set up on IRC, but there has been a lot of other things going on in life, and I didn’t feel like trying to set up my rpi mini-server IRC bouncer again.

I tried some other hosted IRC bouncer services. But some of these wouldn’t let me connect with on a VPN.

I found one good free (as in freedom and free as in no paisa) ZNC bouncer service, which is FreeIRC.org ²

The ZNC bouncer worked fine on Emacs (with ERC; I’ve gone back and forth between ERC and Circe, but this time I was trying ERC again).³ And I eventually figured out how to connect to multiple networks through the ZNC bouncer.

But the best thing I could find on mobile was the Revolution IRC client, and, while it’s a nice enough front-end, it struggled with maintaining a connection to the ZNC bouncer I was using. (I asked about this in an IRC room, and the general consensus was that mobile’s just not a good place to try to do IRC, but I remembered that via the Weechat android app or through Glowing Bear I’d had a good mobile IRC experience years ago.)

IRCv3, and other Korean tubers

I also came across another IRC mobile app, Goguma, but couldn’t get it work. But, frustrated with the situation, I wanted to see if, assuming I could somehow get it work, Goguma might be a better mobile solution.⁴

I ultimately ended up stumbling across an interesting lobste.rs discussion of Goguma which pointed to it being part of a set of IRCv3-aware software, including an IRC bouncer soju, which a number of the commenters on the lobste.rs thread comparing the experience favourably against using ZNC.

A cluster of IRCv3 things:⁵

Goguma (Korean 고구마 goguma “sweet potato”): IRC mobile client written in Flutter by Simon Ser
Gamja (Korean 감자 gamja “potato”): simple IRC web client written in Javascript by Simon Ser
Soju (Korean 소주 soju “a distilled alcoholic beverage”): IRC bouncer written in Go by Simon Ser
Senpai (Japanese 先輩 sèńpáí “senior in social standing or level of education/skill; an elder”): a modern terminal IRC written in Go, started by ‘taiite', who handed development over to ‘delthas'.
Ergo (a play on ergonomic [and “ergonomadic"] and Go(lang), so I suppose ultimately from Ancient Greek ἔργον érgon “work”, but not sure that’s relevant): a modern IRC server written in Go [Jeremy Latt, Daniel Oaks, Shivaram Lingamneni] implementing bleeding-edge IRCv3 features/support

The first three things are the most relevant for us (well, me). Ergo sounds great, but I’m not running an IRC server myself at the moment; and I’ve got a whole wealth of choices of Emacs IRC clients if I’m on desktop/laptop, so Senpai doesn’t seem relevant either for my use case.

Trains, Wayland, and Things that Go

The first three of the IRCv3-looking things are all by Simon Ser, who was lately at Drew DeVault‘s SourceHut, but seems to have left sometime in 2024⁶ and now works at Open Source Railway Designer (something to do with making tools for railway infrastructure simulation, which sounds quite cool: open source trains). Ser obviously works a good bit on IRC-related software, and also a lot of things in Go, and has been involved with Wayland-related projected (taking over maintainership of Sway (a Wayland window manager) and wlroots (Wayland general purpose compositor underlying Sway and other things) from DeVault some years back).

Here’s Thomas Flament & Simon Ser talking about IRCv3 and soju, senpai, gamja, goguma at FOSDEM ‘25 in Brussels:

Chatting on IRC in 2025: grandpa, what’s up? [video link with captions]

In any case, one of the paid-only features that SourceHut offers is via chat.sr.ht, an IRC bouncer, which is running on a fork of soju, with a web-client based on gamja.

As far as I can tell, Sourcehut’s servers are now (all? mainly?) in Amsterdam.⁷

chat.sr.ht isn’t the only hosted IRC bouncer service which runs some version of the soju bouncer, there is also IRC Today [discussed also on lobste.rs], which is hosted in Paris on Scaleway servers.⁸

So, there are at least two apparently EU-based hosted “IRCv3-forward” bouncers.

Smooth & Modern IRC Implementation & Practice (mobile, Emacs)

How to manage to drink rice liquor in a hut: preliminaries

In any case, I signed up for the SourceHut offering at chat.sr.ht, which uses their fork of soju.

Here’s what I did to get it set up for mobile and in Emacs (it seemed worth documenting, in what was meant to be a short post….):

The main things one needs to know are in the chat.sr.hut ‘manpages’, especially the ‘quickstart guide’. But here are a few more notes for specific set-up on Goguma and Emacs.

The first thing one needs to do for any “third-party” client (this includes Goguma or any Emacs IRC client) is generate an OAuth 2.0 personal access token in your SourceHut account. This will be effectively your “password” (with your SourceHut login name as your username).

Nb: on mobile, you have to be careful to actually manage to copy the entire token with “Select all” or the like - I was stuck for a long time not being able to make Goguma work because I’d not managed to copy the entire token, and this wasn’t at all obvious in a mobile browser. On desktop, it’s not an issue.

Goguma (mobile): One potato, two potato, sweet potato

For Goguma, you’re just going to put in:

Server: chat.sr.ht
Nickname: <your SourceHut username>
Password: <an OAuth2.0 token you generated as above>

And you’re in. And here’s what it looks like:

Figure 1: Goguma with normal “bubble” display mode in #sr.ht@libera.chat

And there’s a setting in Goguma (“Compact message list”) to make it look less like a mobile messaging app and just have plain text if you prefer that:

Figure 2: Goguma in “compact message list” display mode in #sr.ht@libera.chat

And you can have Goguma show you notifications if someone mentions your nick. Here is an example showing on a Pebble watch:

Figure 3: Goguma nick mention notification displaying on Pebble watch

(IRC on one’s wrist; it truly is 2025.)

Emacs machines and authority tokens: what to put in your authinfo.gpg and init.el

In your ~/.authinfo.gpg file, add a line like this (with <>‘ed items to be replaced fully; don’t actually include any <>'s here or above or elsewhere; where "my-sourcehut-id" is your actual sourcehut login id in quotes):

machine chat.sr.ht login "my-sourcehut-id" password "<one of your OAuth 2.0 tokens generated at SourceHut meta>"

And then in your ~.emacs.d/init.el, assuming you’re using ERC and have that configured otherwise to your liking, make a user-function for each actual IRC server you want to connect with via chat.sr.ht/soju, like this (assuming your user-id is "my-sourcehut-id": replace appropriately— and you might have different :nick's (“nicknames” ≈ user-handles/user-names/identities) on different servers, of course, so adjust those as well; but your :username is going to be consistently your SourceHut username followed by “/<the-particular-irc-server-address>"):


;; just so we don't have to type it each time, a wrapping `let',
;; and so after this our `server', `port', & `passwd' will be thus defined
;; (you could do `:nick' like this too if it's the same everywhere)
(let ((server "chat.sr.ht")
      (port 6697)
      (passwd
       (cadr
        (auth-source-user-and-password "chat.sr.ht" "my-sourcehut-id"))))
  ;; `auth-source-user-and-password' returns a list of `login' and `password'
  ;; from your .authinfo.gpg; but we just need the second, so `cadr' (= return
  ;; the `car' of the `cdr'), because since the list is going to be something
  ;; like `(mylogin mypassword)', will give us just the atomic `mypassword',
  ;; which is what we want [because the `cdr' of `(mylogin mypassword)' will be
  ;; `(mypassword)' and the `car' of `(mypassword)' will be just `mypassword'].)

  (defun erc-libera ()
    "Connect to Libera.Chat IRC server."
    (interactive)
    (erc-tls
     :nick "emacsuser007"
     :server server
     :port port
     :user "my-sourcehut-id/irc.libera.chat"
     :password passwd))

  (defun erc-oftc ()
    "Connect to OFTC IRC server."
    (interactive)
    (erc-tls
     :nick "emacsuser007"
     :server server
     :port port
     :user "my-sourcehut-id/irc.oftc.net"
     :password passwd))

  (defun erc-ircnow ()
    "Connect to IRCNow IRC server."
    (interactive)
    (erc-tls
     :nick "emacsuser007"
     :server server
     :port port
     :user "my-sourcehut-id/irc6.ircnow.org"
     :password passwd))

  (defun erc-ergo ()
    "Connect to IRCv3-forward Ergo IRC server."
    (interactive)
    (erc-tls
     :nick "emacsuser007"
     :server server
     :port port
     :user "my-sourcehut-id/irc.ergo.chat"
     :password passwd)))

Here we’re making connections for Libera, OFTC, IRCNow, and Ergo, the IRCv3-forward server implemented in Go we mentioned above.⁹

But you can add lots of different servers to your bouncer, and both in Goguma and Emacs/ERC (just make a new erc-<server> function as above), you’ll just be able to get to all of the different rooms in one place and don’t necessary need to worry about which server a particular thing is on after getting it set up.

And then you can make a “connect to all the things function” like this:

(defun erc-connect-all ()
  "Connect to all IRC servers above."
  (interactive)
  ;; we're just calling all of the functions we defined above here:
  (erc-libera)
  (erc-oftc)
  (erc-ircnow)
  (erc-ergo))

And what it looks like (well, it looks like however you’ve configured ERC or whatever Emacs IRC client you’re using, but, in case you want to see a possible way it could look):

Figure 4: in ERC on chat.sr.ht visiting #sr.ht@libera.chat

If you get knocked offline momentarily, ERC will try to reconnect to the bouncer. If you are knocked off or close Emacs for a while, you might want to catch up on what you missed recently. And there’s a nice IRCv3 feature implemented in soju (and in chat.sr.ht) for this, as discussed in the next section.

Leveraging IRCv3 features in Emacs

We can also do another convenient thing for our Emacs IRC configuration now. IRCv3 defines a “chathistory” extension, as an easy way of retrieving earlier IRC room content. This doesn’t work most places, but it works on any bouncers based on soju (see notes below), the syntax being /quote CHATHISTORY LATEST <channelname> * <number-of-lines-to-fetch>.

We can make this more user-friendly in Emacs with something like this, an interactive command that fetches history for whatever IRC room it’s called in (a 100 lines by default; prefix C-u to enter a different number):

(defun erc-chatsrht-give-me-more-irc-history ()
  "Get more history for current IRC buffer (IRCv3 only).

Defaults to 100 lines of history; when C-u prefixed, asks user for
number of lines to fetch.

If using an IRCv3 capable server/bouncer (like chat.sr.ht), fetch the
chat history via the IRCv3 chathistory extension. (Currently, only
soju-based servers implement this feature; see:
https://ircv3.net/software/clients)

For more on chathistory, see:
 - https://man.sr.ht/chat.sr.ht/bouncer-usage.md#chat-history-logs
 - https://ircv3.net/specs/extensions/chathistory
 - https://soju.im/doc/soju.1.html"
  (interactive)
  (if (not (member
            (with-current-buffer (current-buffer)
              major-mode)
            '(erc-mode
              circe-mode
              rcirc-mode)))
      (message "not an IRC buffer; ignoring")
    (let ((lines 100)
          (channel (buffer-name)))
      (when current-prefix-arg
        (progn
          (setq lines
                (read-number (format "How many lines to fetch: ") lines))))
      (erc-send-input
       (concat "/quote CHATHISTORY LATEST " channel " * " (number-to-string lines))
       t))))

Then you can just call erc-chatsrht-give-me-more-irc-history in an IRC buffer (you might find a convenient keybinding for it) to get prior IRC chat in that buffer.

This sort of thing makes Emacs a very pleasant environment to manage IRC in.

Now, for actually adding servers to your chat.sr.ht bouncer in the first place, you might try out the gamja web client (see below).

Inline images and files in Goguma and Emacs

Oh. Gamja (as least as implemented by chat.sr.ht) doesn’t seem to do it, but Goguma fetches image links and displays a preview:

Figure 5: in the goguma mobile client, an image preview in #dwarffortress@libera.chat

[Note: chat.sr.ht’s soju fork doesn’t currently offer direct file-upload and is perhaps unlikely to, but this seems to be implemented in upstream soju and maybe on IRC Today.]

Additional note: Goguma doesn’t do this by default; you need to go into the main “Settings” menu and enable “Display link previews”. (I also have “Send & display typing indicators” turned on here.)

And you can make ERC do similarly, with respect to fetching posted image links, with the erc-image package from MELPA. I have a use-package definition like this:

(use-package erc-image
  :ensure t
  :after erc
  :config
  (setq erc-image-inline-rescale 300) ; maybe set bigger
  (add-to-list 'erc-modules 'image))

And then you’ll see something like this if you post an image link in ERC:

Figure 6: in ERC, an image preview in #dwarffortress@libera.chat

I’d like to figure out how one might better manage it, but — in terms of handling posting local files — there’s a (not-on-MELPA) package imgbb.el which will upload images in Emacs (when passed a file-path or interactively from a file-chooser) to ImgBB, and then put the link in the clipboard.¹⁰

[Note: For better or worse, erc-image will pull images from http as well as https addresses; while Goguma seems only to do the latter.]

Web-client: The sweetness of non-sweet potatoes on the web

The web interface for chat.sr.ht (at https://chat.sr.ht) is, again, based on gamja (“potato”) (discussed above) and gamja is a nice interface indeed.

Figure 7: in the gamja web-client at chat.sr.ht visiting #sr.ht@libera.chat

And, for making your initial connections to different IRC servers, it may be easier just to use this web interface, especially if you’ve already got accounts set up on any of the servers, as for most of them you’ll be able to enter your username and password and so on in the “Add server” interface and not have to do all of the /msg NickServ .... stuff one usually does. (If you don’t have an account already, you’ll have to register in someway, maybe by messaging Nick).

Looks like this:

Figure 8: in the gamja web-client at chat.sr.ht in the server-add menu

In fact, even for signing in/up on a brand new server, at least on some servers, gamja lets you click on “login” or “register” and pulls up a pop-up box to fill in password, email, &c. So it’s quite convenient (and you may not have to message Nick manually).

Wrap-up & evaluation

There are interesting IRC things still going on, both in terms of active IRC communities (though this varies; #emacs is quite active, #dwarffortress isn’t; Discord has eaten into some spaces), and innovations in IRC technologies and tools.

I’m not sure what the best thing is terms of choice of a soju/gajma ecosystem. chat.sr.ht seems to be the cheapest hosted option; IRC Today is more expensive but may currently be more featureful. SourceHut has had trouble with being DDOS’ed/going off-line (see, e.g., SourceHut network outage post-mortem).

Self-hosted set-ups are another option, but are likely to be more fiddly/effort to run/maintain than a “professional” service. But here are a few links to descriptions of people’s self-hosted soju setups:

self-hosting on Alpine Linux: Self-Hosting Soju
self-hosting on OpenBSD: Setting up an IRC bouncer (soju) on OpenBSD · Hugo’s weblog
more self-hosting on Alpine Linux: Self-Hosted IRC with Soju & Senpai | Isaac Ganoung’s Website
self-hosting on IRCNow: IRCNow | Soju / Guide

Currently, chat.sr.ht is working well for me, even if it may be behind on some mainline soju features.

Appendix: etymologies

You probably don’t need to know any of this. It’s not going to help you log into Goguma or make your Emacs config work.

But the naming of a lot of the IRC things above is strange and I’m a linguist by trade and it’s hard for me to avoid paying a lot of attention to words and meanings of words and connections of words.

(While on the topic of mostly irrelevant things: I have some other-language interference (Hindi here) with the Korean names of some of the software discussed here. Specifically, gamja makes me think either गंजा ganjā “bald” or गाँजा gā̃jā “ganja, cannabis”; and soju keeps making me think of an elided version of सो जाऊं so jāū̃ “shall I go to sleep?” [“नहीं! आप आई॰आर॰सी॰ बाउंसर हैं, आपको सो नहीं जाना चाहिए! No! You’re an IRC bouncer: you should not go to sleep!"])

At any rate, here are, unasked for, etymological notes on the four IRCv3 things with Korean or Japanese names, largely sourced from Wiktionary, as indicated, with some connecting text:

Goguma

[mainly sourced from: https://en.wiktionary.org/wiki/%EA%B3%A0%EA%B5%AC%EB%A7%88]

Named apparently after Korean 고구마 goguma “sweet potato” (cp. the web client by the same team, gamja “potato”). First attested in the Mulmyeonggo (물명고 / 物名考), 1824, as Early Modern Korean 고금아 (Yale: kokuma), borrowed from Japanese 孝行芋 (kōkō imo), a term used in the Tsushima dialect. Some earlier attestations are known, but they are in the context of quoting the dialectal Japanese word, not in a Korean context.

There’s also apparently a recent “Internet slang” sense: “a plot development which frustrates the reader (e.g. the protagonist fails to achieve their goal)” [from c. 2012].

The Japanese word 孝行芋 (kōkō imo) from which Korean 고구마 goguma is borrowed is apparently 孝行 (kōkō) [? “dutiful”??] + 芋 (imo) from Old Japanese, attested in the Nihon Shoki of 720, referring to “edible tubers”. May be a shift from older form うも (umo), ultimately from Proto-Japonic *umo. Cognate with Okinawan 芋 (‘nmu). [see: https://en.wiktionary.org/wiki/%E5%AD%9D%E8%A1%8C%E8%8A%8B#Japanese https://en.wiktionary.org/wiki/%E8%8A%8B#Japanese ]

The Japanese word 芋 (imo) is connected to Chinese 芋 [Old *ɢʷas, Modern Mandarin yù ] “taro”.

Seemingly a phono-semantic compound (形聲 / 形声, OC *ɢʷa, *ɢʷas): semantic 艸 (“grass; plant”) + phonetic 于 (OC *ɢʷa) – “taro”.

Compare Proto-Hmong-Mien *wouH (“taro”), Burmese ဝ (wa., “elephant foot yam”), Tibetan གྲོ་མ (gro ma, “ Argentina anserina (syn. Potentilla anserina), a plant with small edible tubers”).

There are various theories on how all these words are related:

Schuessler (2007) considers it to be an areal word, comparing it to the Hmong-Mien and Burmese words. Schuessler (2015) does not consider the Tibetan word to be cognate.
Blench (2012) suggests that the Chinese word is borrowed from Proto-Hmong-Mien and that the Burmese word may be a late loan from Old Chinese.
STEDT reconstructs Proto-Sino-Tibetan *g/s-rwa (“taro; yam; tuber”), whence the Tibetan word. This etymon is regarded as allofamically related this word and 薯 (OC *djas).
Gong Hwang-cherng (2002) and Baxter and Sagart (2017) also suggest that this word is related to the Tibetan word.

[on the Chinese, see: https://en.wiktionary.org/wiki/%E8%8A%8B]

Gamja

[sourced from: https://en.wiktionary.org/wiki/%EA%B0%90%EC%9E%90]

Seemingly named from Korean 감자 gamja “potato”, which is a nativisation of the Sino-Korean term 감저 (甘藷, gamjeo, “lesser yam (Dioscorea esculenta)”). First attested 1766 in Korea, then referring to the sweet potato (Ipomoea batatas). The word came to refer to both “potato” and “sweet potato” in the nineteenth century, and later lost its original meaning. (So gamja meant “sweet potato” first; but partially supplanted by goguma.)

With 甘藷, gamjeo apparently borrowed from Early Modern Korean 감져 (kamcye), itself from 甘藷 [https://en.wiktionary.org/wiki/%E7%94%98%E8%97%B7].

[There’s also homophonous verbal form: 감자 (gamja) (plain hortative of 감다)

let’s close (our eyes)
let’s wash

which is presumably unrelated.]

Soju

[source: https://en.wikipedia.org/wiki/Soju]

Seemingly named for Korean Soju (소주; 燒酒), which means “burned liquor”: with the first syllable, so (소; 燒; “burn”), referring to the heat of distillation and the second syllable, ju (주; 酒), meaning “alcoholic drink”. Etymological dictionaries record that China’s shaozhou (shāojiǔ, 烧酒), Japan’s shochu (shōchū, 焼酎), and Korea’s soju (soju, 燒酒) have the same etymology. [A Wikipedian has here added a note about unreliable sources(?).]

Another name for soju is noju (노주; 露酒; “dew liquor”), with its first syllable, no/ro (노/로; 露; “dew”), likening the droplets of the collected alcohol during the distilling process to dewdrops.

The origin of soju dates back to 13th-century Goryeo. The Yuan Mongols [the imperial dynasty of China founded by Kublai Khan, grandson of Genghis Khan] acquired the technique of distilling arak from the Persians during their invasions of the Levant, Anatolia, and Persia, and in turn introduced it to the Korean Peninsula during the Mongol invasions of Korea (1231–1259). Distilleries were set up around the city of Gaegyeong, the then-capital (current Kaesong). In the areas surrounding Kaesong, soju is still called arak-ju (아락주). Andong soju, the direct root of modern South Korean soju varieties, started as the home-brewed liquor developed in the city of Andong, where the Yuan Mongols’ logistics base was located during this era.

Senpai

[source: https://en.wiktionary.org/wiki/%E5%85%88%E8%BC%A9#Japanese]

Obviously named for Japanese 先輩 senpai “senior/superior in social standing or education or skill; an elder” (the homepage has a tagline “Your everyday IRC student” and also “Welcome home, desune~” and also a smiling anime-ish cat-ish logo).

The Japanese word was borrowed from the Middle Chinese 先輩 (sen pwojH). A doublet of ソンベ (sonbe) [With Japanese sonbe being itself borrowed from Korean 선배(先輩) (seonbae), which means “(chiefly in Korean media) sunbae (upperclassman or senior, in the context of Korea)"].

Both ultimately from Chinese 先輩 [ xiānbèi ] “older generation; senior; elder; ancestor; predecessor”.

See: https://weechat.org/about/interfaces/ ↩︎
See https://wiki.ircnow.org/index.php?n=FreeIRC.About; freeirc.org is part of the IRCNow network:— a group/community with a lot of useful free and paid services of various sorts, including shell accounts and VPSes and such, and a lot of freely accessible Unix/OpenBSD server admin tutorials.

I also thought about the SDF Public Access Unix System[sdf.org], which has shell accounts and an IRC bouncer, but didn’t end up trying their bouncer. ↩︎
Alongside of ERC and Circe, another popular Emacs IRC client is rcirc. But the Emacswiki lists at least nine different Emacs IRC clients. ↩︎
Goguma exists in an official version for Android on F-Droid [on the official F-Droid respository; and a ‘nightly’ version on Goguma’s own F-Droid repository]; with unofficial/community versions on Google Play for Android and on the App Store for iOS. At the moment I’d recommend the official F-Droid version; I had some issues with missing messages on the ‘nightly’ version. ↩︎
I’m not entirely sure why Goguma is named after the Korean word for “sweet potato”. Though Wiktionary does report there’s also apparently a recent “Internet slang” sense of the word: “a plot development which frustrates the reader (e.g. the protagonist fails to achieve their goal)” [from c. 2012]; that may be a red herring.

It’s obviously connected to one of Ser’s other IRC applications, gamja, and maybe it’s just wanting to name things after Korean words for root vegetables for some reason. I would have said it was a play on Go, but Goguma isn’t written in Go.

Though soju (which is written in Go), while also a Korean word, doesn’t fit quite the root vegetable pattern. (Nor does senpai, but that’s by different authors [though Korean 先輩 (in hangeul 선배) seonbae “upperclassman, senior” exists too, and is cognate with senpai, both ultimately borrowed from Middle Chinese 先輩(sen pwojH) “elder, senior, ancestor”]. Well, see more of this in the appendix, if it’s your sort of thing. ↩︎
I think this is what is referred to in the June 2024 SourceHut blog post, “…You may have heard that we also had to part ways with one of our staff members recently. This reduces our headcount to two. For the time being we will not be hiring a replacement, but our near-future plans are achievable with our current headcount. Though we usually aim for transparency to the maximum extent possible, we will not be sharing further details about this departure, as a matter of reasonable privacy.” ↩︎
see 2024-06-04 blog report “The state of SourceHut and our plans for the future”, where DeVault says “Also, as a happy side-effect of our surprise move to Amsterdam, SourceHut’s datacenter installation is now entirely powered by renewable energy sources. We have also finally rolled out IPv6 support for most SourceHut services as part of our migration!” ↩︎
In the “Who is behind IRC Today” FAQ, they say “We have co-developed the open source piece of software used in our service, soju.” I suspect this “we” includes at least, delthas (the current developer of senpai mentioned above, who also has a number of commits to the soju repo, and who is in France), and Thomas Flament.

(For whatever reason, there seem to be at least two open source developers who are both (a) French, and with interests in (b) IRCv3 and (c) Golang and (d) Korean or Japanese foodstuffs/culture.) ↩︎
If you’re on IRC at all, you probably want to be on Libera (which is where the main body of Freenode went after Freenode was bought by the founder of Private Internet Access VPN, Andrew Lee, pretender to the defunct throne of Joseon and the Korean Empire; Korea seems tied up in strange ways to IRC), and lots of free software projects which aren’t on Libera are on OFTC [=Open and Free Technology Community] (e.g., Alpine Linux). ↩︎

With Elpaca I use the following:

(use-package imgbb
  :ensure (:host github :repo "ecraven/imgbb.el"))

With Emacs 30+'s vc-package you should be able to do something like:

(use-package imgbb
  :vc (imgbb :url "https://github.com/ecraven/imgbb.el"
                    :branch "master")
  :ensure t)

We might try to make something a bit better/more integrated, but it works well enough for now. ↩︎

C-c-c-conjecturing, and dealing with recursion in Emacs (more excursus)

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Mon, 24 Feb 2025 04:10:00 -0600

I’m not putting this in the lambda-calculus series, though it touches on issues from the last post in the series, but specifically issues of recursion. I was curious to go back and recall how The Little Schemer dealt with problems of recursion (and the Y Combinator (which we still haven’t got properly to yet, but we will, I promise)).

In Chapter 9 of The Little Schemer (“and Again, and Again, and Again,…"), it starts off by querying the reader if they want caviar and how to find it in a list, and then essentially gets (from caviar and grits) into issues around the halting problem.

Figure 1: Duane Bibby illustration heading Chapter 9 of “The Little Schemer”

But a number of pages into this chapter the book queries of a particular function: “Is this function total?” and presents the following:

(define C
  (lambda (n)
    (cond
     ((one? n) 1)
     (else
      (cond
       ((even? n) (C (÷ n 2)))
       (else (C (add1 (× 3 n)))))))))

Code Snippet 1: a function puzzle from "The Little Schemer" (p.155)

I didn’t know what C was (a number of readers probably have recognised it already, but please don’t spoil it for the others just yet).

But whether we get the point of the function or not,we can implement it in Emacs Lisp, translating from the Scheme into Emacs Lisp (defining an equivalent of even? as an Emacs Lisp evenp first) as:

;; -*- lexical-binding: t; -*-
(defun evenp (n)
  "Returns `t' if `n' is even."
  (if (= (% n 2) 0)
      t
    nil))

(defun C (n)
  "Function C from Little Schemer p.155."
  (cond
   ((= n 1)
    1)
   ((evenp n)
    (C (/ n 2)))
   (t
    (C (1+ (* 3 n))))))

Code Snippet 2: translation of the The Little Schemer 'C' function into Elisp

And we can try running this on various numbers:

(C 1) ; 1
(C 9) ; 1
(C 108) ; 1
(C 837799) ; 1
(C 8400511) ; 1
(C 63728126) ; 1

That is, for all of these numbers it just outputs 1. Well, we can try a couple of other values:

(C 0) ; cl-prin1: (error "Lisp nesting exceeds ‘max-lisp-eval-depth’")
(C 63728127) ; cl-prin1: (error "Lisp nesting exceeds ‘max-lisp-eval-depth’")

So C fails for 0 and numbers above 63728126, with a “excessive lisp nesting”-type issue.

But since the only non-error result we’re getting is 1, we could write a much simpler function.¹

;; -*- lexical-binding: t; -*-
(defun ersatz-C (n)
  "Returns 1 if `n' is 1;
enter an infinite nesting loop
if `n' is 0 or above 63728126;
for any other integer value,
pause briefly for sake of plausibly,
and then return 1."
  (cond
   ((= n 1)
    1)
   ((= n 0)
    (ersatz-C n))
   ((> n 63728126)
    (ersatz-C n))
   (t (progn
        (sleep-for 0.005)
        1))))

Code Snippet 3: A simpler function to return "1"

But obviously function C is doing something more interesting. So, let’s make it actually show us what it’s doing:

;; -*- lexical-binding: t; -*-
(defun C-verbose (n &optional count)
  "For a number `n', return `1' if it's `1'; otherwise,
if it's even, run the same function on half of `n';
else, run the same function on one more than three times
`n'."
  (let ((count (or count 0)))
    (print (format "Step %s: %s" count n))
    (setq count (1+ count))
    (cond
     ((= n 1)
      1)
     ((evenp n)
      (C-verbose (/ n 2) count))
     (t
      (C-verbose (1+ (* 3 n)) count)))))

(C-verbose 7) ; =
;; "Step 0: 7"
;; "Step 1: 22"
;; "Step 2: 11"
;; "Step 3: 34"
;; "Step 4: 17"
;; "Step 5: 52"
;; "Step 6: 26"
;; "Step 7: 13"
;; "Step 8: 40"
;; "Step 9: 20"
;; "Step 10: 10"
;; "Step 11: 5"
;; "Step 12: 16"
;; "Step 13: 8"
;; "Step 14: 4"
;; "Step 15: 2"
;; "Step 16: 1"

(C-verbose 9) ; =
;; "Step 0: 9"
;; "Step 1: 28"
;; "Step 2: 14"
;; "Step 3: 7"
;; "Step 4: 22"
;; "Step 5: 11"
;; "Step 6: 34"
;; "Step 7: 17"
;; "Step 8: 52"
;; "Step 9: 26"
;; "Step 10: 13"
;; "Step 11: 40"
;; "Step 12: 20"
;; "Step 13: 10"
;; "Step 14: 5"
;; "Step 15: 16"
;; "Step 16: 8"
;; "Step 17: 4"
;; "Step 18: 2"
;; "Step 19: 1"

Code Snippet 4: Write `C' to make it show us what it's doing; with some examples of use

Perhaps not surprisingly, we see that setting n=9 makes the function take more steps to reach 1 than setting n=7. But it’s not actually a direct linear relation, because if we set n=10, it’s a much shorter path:

(C-verbose 10) ; =
;; "Step 0: 10"
;; "Step 1: 5"
;; "Step 2: 16"
;; "Step 3: 8"
;; "Step 4: 4"
;; "Step 5: 2"
;; "Step 6: 1"

So we might still wonder about exactly what’s going on with function C, how it works, and what’s the point of it anyway.

Conjectures about simple arithmetic operations and integer pathways

The Little Schemer does give a clue about what C is, referring to the question of whether it’s a total function or not:

It doesn’t yield a value for 0, but otherwise nobody knows. Thank you, Lothar Collatz (1910–1990).

A bit of research and we can find quite a bit about C: it turns out to be the Collatz conjecture: one of the most famous unsolved problems in mathematics.

The Wikipedia page on the Collatz conjecture is useful here in laying out basics. As well, Eric Roosendaal‘s “On the 3x + 1 problem”. And Quantamagazine’s “The Simple Math Problem We Still Can’t Solve” presents a fairly accessible overview, including recent developments.

There’s some very interesting visualisations of the Collatz Functions paths as a Collatz tree, which is a rather aesthetically-appealling visual object. Here’s Algoritmarte’s 3D live render, based on a discussion from the Numberphile channel (for Numberphile’s original video, done by hand, see the fn):²

Algoritmarte's Collatz Tree visualisation

But Collatz conjecture is an unsolved problem that looks like it should certainly be solvable. It seems quite trivial in some sense: inductive logic would seem to suggest that the same basic patterns would repeat, even if there are more of them, or they are longer, given that every number that’s been tested has converged to 1 (or, more precisely, fallen into the “4-2-1” loop) and so this tempts people in trying to solve it. But Paul Erdős said of it “Mathematics may not be ready for such problems”.³

Figure 2: https://xkcd.com/710/ [see: https://www.explainxkcd.com/wiki/index.php/710:_Collatz_Conjecture ]

Well, we’ll not try to solve it, but rather use it for thinking about issues of recursion and long calculations in Emacs Lisp, and doing some practical engineering-type testing of different methods of dealing with such functions.

One thing we might note, as a sort of aside, is that The Little Schemer‘s C isn’t properly the Collatz function. The conjecture is really about whether the process will ever reach the repeating “4-2-1” loop, because the process of dividing even numbers by 2 and multiple odd numbers and adding 1 to them would actually mean that once we’ve reached 1, we need to multiple it by 3 and add 1, which means we’re back at 4, which is even, so we’ll divide by 2 and get 2, which is even, so we divide by 2 and get 1, which is odd, so we…. And so on.

We can write this “real Collatz” function:

;; -*- lexical-binding: t; -*-
(defun real-collatz (n)
  "For an integer `n', if it's evan, divide it by 2;
if it's odd, multiple it by 3 and add 1."
  (cond
   ((evenp n)
    (real-collatz (/ n 2)))
   (t (real-collatz (1+ (* n 3))))))

Code Snippet 5: the real basic Collatz operations

If we now try to feed it 9, we’ll hit the lisp-nesting error, but if we turn on toggle-debug-on-error, we can inspect what’s going on, and it’s exactly the “4-2-1” loop we predicted:

(real-collatz 9) ; cl-prin1: (error "Lisp nesting exceeds ‘max-lisp-eval-depth’")

;; DEBUG:
....
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(1)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(2)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(4)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(8)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(16)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(5)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(10)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(20)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(40)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(13)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(26)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(52)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(17)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(34)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(11)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(22)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(7)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(14)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(28)
  (cond ((evenp n) (real-collatz (/ n 2))) (t (real-collatz (1+ (* n 3)))))
  real-collatz(9)
  eval((real-collatz 9) nil)
  elisp--eval-last-sexp(nil)
.....

We’ll make use of it later on though, somewhat (essentially with the n=1 condition displaced.)

Dealing with long winding paths, through hailstorms

Ok, so two things. One, let’s not just print things (longer paths are going to make his quite messy), but return a list of the steps our function goes through and a count of them. We do still want to be able to inspect the path patterns. (Wikipedia notes on this “[t]he sequence of numbers involved [in the paths] is sometimes referred to as the hailstone sequence, hailstone numbers or hailstone numerals (because the values are usually subject to multiple descents and ascents like hailstones in a cloud)".)

And, two, let’s use the cl-labels trick from last time to get tail-call optimisation in Emacs.

(And a minor thing: not bother with a separate evenp function; just use the calculation directly.)

;; -*- lexical-binding: t; -*-
(defun collatz-cllabels-tco (n)
  "Calculates the 'Collatz path' for `n',
return a list of all of the steps cons'ed
with count of the steps in the path.

[Uses `cl-labels' to get tail-call optimisation.]"
  (cl-labels ((collatz* (n r)
                  (setq r (cons n r))
                  (cond
                     ((= n 1)
                      r)
                     ((= (% n 2) 0)
                        (collatz* (/ n 2) r))
                     (t
                        (collatz* (1+ (* 3 n)) r)))))
    (let* ((clst (collatz* n nil))
          (clngth (1- (length clst))))
      (cons (nreverse clst) clngth))))

Code Snippet 6: A tail-call optimising implementation of a recursive Collatz function

Here you will note that we’ve got our collatz* function (defined in cl-labels) being self-recursive only properly in tail positions, so it can be tail-call optimised. We start by calling collatz* with arguments n (whatever value we passed for n) and nil, with collatz* taking two arguments, n and r and setting r to the cons of n with r (we’ll pass this value on, collecting the steps), and then implementing the rest of the Collatz function in the same way as previously, except passing along the r value as our collector. At the end, we’ll count the steps and cons the list of the steps themselves, using nreverse to flip the list over (because it’ll have the last results on the “outer” left and the first results on the “inner” right), with the step count.

Some examples:

(collatz-cllabels-tco 9) ; ((9 28 14 7 22 11 34 17 52 26 13 40 20 10 5 16 8 4 2 1) . 20)
                ; path from 9 to 1; 20 steps

(collatz-cllabels-tco 21) ; ((21 64 32 16 8 4 2 1) . 8)
                 ; path from 21 to 1; 8 steps

Excellent, but what about “63728127”?:— the number that tripped us up before:

;; -*- lexical-binding: t; -*-

(collatz-cllabels-tco 63728127) ; it works!

;; ...but the result is quite long, you can try it yourself;
;; use setq to be able to see all of it, e.g.,
;; (setq c-result (collatz-cllabels-tco 63728127)).

;; for, let's just see how many steps it is:
(defun collatz-return-steps (n)
  "Output just the number of steps that
`n' takes to reach 1."
  (format "%s takes %s steps to reach 1"
          n
          (cdr
           (collatz-cllabels-tco n))))


(collatz-return-steps 63728127) ; =
"63728127 takes 949 steps to reach 1"

Code Snippet 7: simple counting of Collatz steps on the path

“Let’s try it with a bigger number!", you say. Well, we can try it on most-positive-fixnum (which is I think probably set to 2305843009213693951 by default) — but remember the size of the number and the number of steps is not a direct linear relation, so you might be disappointed:

(collatz-return-steps most-positive-fixnum) ; =
"2305843009213693951 takes 860 steps to reach 1"

This is less than for 63728127.

Ok, so let’s make something really big then. 2^1000 + 1.

(collatz-return-steps (1+ (expt 2 1000))) ; =
"10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069377 takes 7248 steps to reach 1"

It takes 7,248 steps. What about 2^10000 + 1?

(collatz-return-steps (1+ (expt 2 10000))) ; =
"19950631168807583848837421626835850838234968318861924548520089498529438830221946631919961684036194597899331129423209124271556491349413781117593785932096323957855730046793794526765246551266059895520550086918193311542508608460618104685509074866089624888090489894838009253941633257850621568309473902556912388065225096643874441046759871626985453222868538161694315775629640762836880760732228535091641476183956381458969463899410840960536267821064621427333394036525565649530603142680234969400335934316651459297773279665775606172582031407994198179607378245683762280037302885487251900834464581454650557929601414833921615734588139257095379769119277800826957735674444123062018757836325502728323789270710373802866393031428133241401624195671690574061419654342324638801248856147305207431992259611796250130992860241708340807605932320161268492288496255841312844061536738951487114256315111089745514203313820202931640957596464756010405845841566072044962867016515061920631004186422275908670900574606417856951911456055068251250406007519842261898059237118054444788072906395242548339221982707404473162376760846613033778706039803413197133493654622700563169937455508241780972810983291314403571877524768509857276937926433221599399876886660808368837838027643282775172273657572744784112294389733810861607423253291974813120197604178281965697475898164531258434135959862784130128185406283476649088690521047580882615823961985770122407044330583075869039319604603404973156583208672105913300903752823415539745394397715257455290510212310947321610753474825740775273986348298498340756937955646638621874569499279016572103701364433135817214311791398222983845847334440270964182851005072927748364550578634501100852987812389473928699540834346158807043959118985815145779177143619698728131459483783202081474982171858011389071228250905826817436220577475921417653715687725614904582904992461028630081535583308130101987675856234343538955409175623400844887526162643568648833519463720377293240094456246923254350400678027273837755376406726898636241037491410966718557050759098100246789880178271925953381282421954028302759408448955014676668389697996886241636313376393903373455801407636741877711055384225739499110186468219696581651485130494222369947714763069155468217682876200362777257723781365331611196811280792669481887201298643660768551639860534602297871557517947385246369446923087894265948217008051120322365496288169035739121368338393591756418733850510970271613915439590991598154654417336311656936031122249937969999226781732358023111862644575299135758175008199839236284615249881088960232244362173771618086357015468484058622329792853875623486556440536962622018963571028812361567512543338303270029097668650568557157505516727518899194129711337690149916181315171544007728650573189557450920330185304847113818315407324053319038462084036421763703911550639789000742853672196280903477974533320468368795868580237952218629120080742819551317948157624448298518461509704888027274721574688131594750409732115080498190455803416826949787141316063210686391511681774304792596709377 takes 72378 steps to reach 1"

72,378 steps.

Can we go one bigger?

(collatz-return-steps (1+ (expt 2 100000)))
;; Debugger entered--Lisp error: (overflow-error)

No. But it’s because Emacs has maximum-integer-width set to 65536 (so a number can only be 65,536 digits long). We can set it to something bigger, and it’ll work.

(let ((integer-width 99999999))
  (collatz-return-steps (1+ (expt 2 100000)))) ; =

;; I'll spare you the number `n' itself as it's realy quite long,
;; but that number takes 717,858 steps to reach 1.

You’ll note, if you’ve been evaluating along in your own Emacs that these results have been nearly instantaneous so far. This one takes a bit longer.

(I subsequently tried doing collatz-cllabels-tco on 2^1000000 + 1, but with only 16Gb in my machine, Linux’s OOM-killer killed Emacs before it reached the end. I should probably try setting up some sort of swap better.)

Streaming though the hail

We can also try the stream package we looked at before, which creates “lazy” conses, of potentially infinite sequences, which can be evaluated one piece at a time, as needed.

(defun collatz-stream (n)
  "For a number `n', return `1' if it's `1'; otherwise,
if it's even, run the same function on half of `n';
else, run the same function on one more than three times
`n'.
(Also, collect each step into a list and, when `n' is `1',
print the list.)
[Uses the `stream' package for lazy-evaluated conses.]"
  (cl-labels ((collatz* (n)
                (stream-cons n
                             ;; note that this is the `real' collatz func
                             ;; no checking for 1 here; we do it below
                              (cond
                               ((= (% n 2) 0)
                                (collatz* (/ n 2)))
                               (t (collatz* (1+ (* 3 n))))))))
    (let ((clst (collatz* n))
          (last 0)
          (out nil))
      (while (not (= last 1)) ;; stop when we reach 1
        (setq last (stream-pop clst))
        (setq out (cons last out)))
      (let* ((steps (1- (length out)))
            (rlist (nreverse out)))
        (cons rlist steps)))))

Code Snippet 8: Using lazy cons sequences in "stream" for Collatz calculations

This, however, turns out to in fact be much slower than our cl-labels tco recursion.

In fact, we can quantify this a bit further using Adam Porter‘s bench-multi macro:

(bench-multi
  :times 5
  :forms (("collatz-stream" (let ((integer-width 99999999))
            (collatz-stream (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco" (let ((integer-width 99999999))
            (collatz-cllabels-tco (1+ (expt 2 10000)))))))

Code Snippet 9: bench-marking cl-labels vs stream collatz functions

Here the forms are ranked in order by fastest first; the second column says how much slower the form is compared against the fastest; then there are total runtime (for all iterations; we did here 5 for each); and total garbage collections performed by Emacs during those tests; and the total amount of time the garbage collection tasks themselves took.

Form	x fastest	Total runtime	# of GCs	Total GC runtime
collatz-cllabels-tco	fastest	1.310546	2	0.484381
collatz-stream	12.72	16.665472	45	11.808777

I’m not quite sure why collatz-stream is this much slower than collatz-cllabels-tco is the case. My guess is that because of the delayed/lazy evaluation, we aren’t able to get real reduction of stack frames because we’re only evaluating one piece/instance of the Collatz function on each stream-pop, and there’s no advantage to using cl-labels here really.

The stream method certainly triggers more garbage collection, which is part of it, though I’m not entirely sure why it should involve triggering more garbage collection. (I suppose we could play with garbage collection settings in emacs and see if we can deal garbage collection and make the stream method better.)

Caught in loops still?

But what about a plain looping method for the Collatz function. We generally saw with the Fibonacci series that loops seemed more efficient.

We can try with both plain while loops and also cl-loops:

(defun collatz-while-loop (n)
  "Collatz function using emacs `while' loop."
  (let ((m n)
        (clst (list n)))
    (while (not (= m 1))
      (setq m
            (cond
             ((= (% m 2) 0)
              (/ m 2))
             (t (1+ (* 3 m)))))
      (setq clst (cons m clst)))
          (let* ((steps (1- (length clst)))
            (rlist (nreverse clst)))
            (cons rlist steps))))

(defun collatz-cl-loop (n)
  "Collatz function using `cl-loop' with
`while' clause condition."
  (let ((m n)
        (clst (list n)))
    (cl-loop while (not (= m 1))
             do (progn
                  (setq m
                        (cond
                         ((= (% m 2) 0)
                          (/ m 2))
                         (t (1+ (* 3 m)))))
                  (setq clst (cons m clst)))
             (let* ((steps (1- (length clst)))
                    (rlist (nreverse clst)))
               (cons rlist steps))))

Code Snippet 10: looping methods for the Collatz function

And now comparing all of the methods:

(bench-multi
  :times 5
  :forms (("collatz-stream"
           (let ((integer-width 99999999))
             (collatz-stream (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco (1+ (expt 2 10000)))))
          ("collatz-while-loop"
           (let ((integer-width 99999999))
             (collatz-while-loop (1+ (expt 2 10000)))))
          ("collatz-cl-loop"
           (let ((integer-width 99999999))
             (collatz-cl-loop (1+ (expt 2 10000)))))))

Code Snippet 11: bench-marking our Collatz function implementations

We find the following results:

Form	x fastest	Total runtime	# of GCs	Total GC runtime
collatz-while-loop	fastest	0.876315	1	0.240661
collatz-cl-loop	1.02	0.896261	1	0.235273
collatz-cllabels-tco	1.43	1.255294	2	0.449736
collatz-stream	18.77	16.446478	48	11.677753

Both types of loops turn out faster than even collatz-cllabels-tco, with the plain emacs while loop method seeming slightly faster than the cl-loop method, but probably within margin of error (or maybe just a bit of extra machine in cl-loop which loses a small amount of efficiency.)

For our TCO tail-recurive cll maybe it’s the additional garbage collection that slows down collatz-cllabels-tco.

What if we just look at the two loop methods, and run more trials?

(bench-multi
  :times 100
  :forms (("collatz-while-loop"
           (let ((integer-width 99999999))
             (collatz-while-loop (1+ (expt 2 10000)))))
          ("collatz-cl-loop"
           (let ((integer-width 99999999))
             (collatz-cl-loop (1+ (expt 2 10000)))))))

Code Snippet 12: bench-marking loop methods of implementing the Collatz function

Our results are:

Form	x fastest	Total runtime	# of GCs	Total GC runtime
collatz-while-loop	fastest	17.584334	20	4.851656
collatz-cl-loop	1.03	18.027418	20	5.070158

Still the plain while is slightly faster than cl-loop.

An attempt at promoting our recursive tco function

I’d really like to somehow improve our recursive TCO collatz function.

What if we pull the collector r out in a higher-scoping let and so don’t have to pass it into collatz* each time; thus:

(defun collatz-cllabels-tco-improved (n)
  "Calculates the 'Collatz path' for `n',
return a list of all of the steps cons'ed
with count of the steps in the path.

[Uses `cl-labels' to get tail-call optimisation.]"
  (let ((r nil))
    (cl-labels ((collatz* (n)
                    (setq r (cons n r))
                    (cond
                       ((= n 1)
                        r)
                       ((= (% n 2) 0)
                          (collatz* (/ n 2)))
                       (t
                          (collatz* (1+ (* 3 n)))))))
      (let* ((clst (collatz* n))
            (clngth (length clst)))
        (cons (nreverse clst) clngth)))))

Code Snippet 13: trying to eke out more performance for TCO cl-labels recursive Collatz function

(bench-multi
  :times 10
  :forms (("collatz-cllabels-tco"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco-improved"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco-improved (1+ (expt 2 10000)))))))

Code Snippet 14: bench-marking the two cl-labels recursive Collatz functions

Here are the results:

Form	x fastest	Total runtime	# of GCs	Total GC runtime
collatz-cllabels-tco-improved	fastest	2.198572	3	0.719563
collatz-cllabels-tco	1.17	2.579470	4	0.923288

We do seem to eke out a bit more performance.

But compared to the loops, it’s still not as performant:

(bench-multi
  :times 10
  :forms (("collatz-cllabels-tco"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco-improved"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco-improved (1+ (expt 2 10000)))))
          ("collatz-while-loop"
           (let ((integer-width 99999999))
             (collatz-while-loop (1+ (expt 2 10000)))))
          ("collatz-cl-loop"
           (let ((integer-width 99999999))
             (collatz-cl-loop (1+ (expt 2 10000)))))))

Code Snippet 15: benchmarking two cl-labels and two loop Collatz implementations

We seem to have done a bit better. Now collatz-cllabels-tco-improved is only 1.26x slower than the fastest, collatz-while-loop. We save an extra gc over the older collatz-cllabels-tco, but still have to run an additional one compared to the two loop functions.

Form	x fastest	Total runtime	# of GCs	Total GC runtime
collatz-while-loop	fastest	1.754987	2	0.486154
collatz-cl-loop	1.01	1.769489	2	0.486728
collatz-cllabels-tco-improved	1.26	2.208662	3	0.732637
collatz-cllabels-tco	1.44	2.528099	4	0.899793

Putting all of the methods we’ve come up with together and running more trials:

(bench-multi
  :times 100
  :forms (("collatz-stream"
           (let ((integer-width 99999999))
             (collatz-stream (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco (1+ (expt 2 10000)))))
          ("collatz-cllabels-tco-improved"
           (let ((integer-width 99999999))
             (collatz-cllabels-tco-improved (1+ (expt 2 10000)))))
          ("collatz-while-loop"
           (let ((integer-width 99999999))
             (collatz-while-loop (1+ (expt 2 10000)))))
          ("collatz-cl-loop"
           (let ((integer-width 99999999))
             (collatz-cl-loop (1+ (expt 2 10000)))))))

Code Snippet 16: full benchmark of all of our Collatz implementations

Here, interestingly, we end up with collatz-cl-loop slightly edging out collatz-while-loop. And our favourite (well, my favourite) collatz-cllabels-tco-improved function doesn’t do too poorly: it’s only x1.28 slower than the fastest (collatz-cl-loop); and even the old collatz-cllabels-tco with additional argument passing is only x1.53 slower

Form	x fastest	Total runtime	# of GCs	Total GC runtime
collatz-cl-loop	fastest	17.733980	19	4.817641
collatz-while-loop	1.01	17.850171	20	4.977408
collatz-cllabels-tco-improved	1.28	22.736898	33	7.961263
collatz-cllabels-tco	1.53	27.146827	47	10.735285
collatz-stream	18.97	336.443089	961	240.167718

The collatz-stream function remains significantly slower, x19 so.

There is an obvious correlation between runtime and garbage collection, (19 vs 20 vs 33 vs 47 vs 961), so again possibly tweaking garbage collection could change things. Although, if we get rid of the garbage collection runtime differences we end up with:

Form	Total runtime w/o GC	x fastest
collatz-while-loop	12.872762999999999	fastest
collatz-cl-loop	12.916338999999999	1.003
collatz-cllabels-tco-improved	14.775635000000001	1.148
collatz-cllabels-tco	16.411541999999997	1.275
collatz-stream	96.27537099999998	7.479

Now collatz-while-loop is again indeed actually the fastest, but still just barely more so than collatz-cl-loop; both recursive TCO’ing cl-labels aren’t actually too much slower than the loops, though the improved version is indeed better; and collatz-stream is still quite slow in comparison. So garbage collection is part of the difference, but not the only one, especially for the streams method.

Escaping (at least some) loops: out of recursion and into… different recursion (next time)

This was really an excursus (on an excursus). And, next time — really — we’ll deal with the Y Combinator. I know this one really wasn’t a lambda calculus post at all, but an excursion on our previous excursion into recursion, here mostly practical considerations for Emacs. But recursion is crucial for the upcoming discussion of the Y Combinator, and paradoxes (and their uses). And that’ll be interesting for lambda calculus, and its connections with Lisp.

And, after we’re through with Y Combinators and Fibonacci sequences and barbers who shave everyone who does’t shave themselves, and how these things all tie together, we can get to something I’ve been working on for a while:— given that, connections aside, Lisp isn’t lambda calculus: can we come up with a good implementation of lambda calculus in Lisp?

(And specifically in Emacs Lisp: which is maybe one of the Lisps least naturally suited for this: a Scheme or Clojure would be a better choice really. But it’s more fun to try to get it to work in Elisp, with all of the additional challenges it presents.)

(I’m perhaps very slightly giving the game away; or, at least providing a clue in the naming of it if you can think of what C-words “ersatz” rhymes with.) ↩︎
See The Collatz Tree | Algoritmarte. And the video it’s based on https://www.youtube.com/watch?v=LqKpkdRRLZw
↩︎
The Quantamagazine article notes:

Do not try to solve this math problem.

You will be tempted. This problem is simply stated, easily understood, and all too inviting. Just pick a number, any number: If the number is even, cut it in half; if it’s odd, triple it and add 1. Take that new number and repeat the process, again and again. If you keep this up, you’ll eventually get stuck in a loop. At least, that’s what we think will happen.

Take 10 for example: 10 is even, so we cut it in half to get 5. Since 5 is odd, we triple it and add 1. Now we have 16, which is even, so we halve it to get 8, then halve that to get 4, then halve it again to get 2, and once more to get 1. Since 1 is odd, we triple it and add 1. Now we’re back at 4, and we know where this goes: 4 goes to 2 which goes to 1 which goes to 4, and so on. We’re stuck in a loop.

Or try 11: It’s odd, so we triple it and add 1. Now we have 34, which is even, so we halve it to get 17, triple that and add 1 to get 52, halve that to get 26 and again to get 13, triple that and add 1 to get 40, halve that to get 20, then 10, then 5, triple that and add 1 to get 16, and halve that to get 8, then 4, 2 and 1. And we’re stuck in the loop again.

The infamous Collatz conjecture says that if you start with any positive integer, you’ll always end up in this loop. And you’ll probably ignore my warning about trying to solve it: It just seems too simple and too orderly to resist understanding. In fact, it would be hard to find a mathematician who hasn’t played around with this problem.

I couldn’t ignore it when I first learned of it in school. My friends and I spent days trading thrilling insights that never seemed to get us any closer to an answer. But the Collatz conjecture is infamous for a reason: Even though every number that’s ever been tried ends up in that loop, we’re still not sure it’s always true. Despite all the attention, it’s still just a conjecture.

….

It’s easy to verify that the Collatz conjecture is true for any particular number: Just compute the orbit until you arrive at 1. But to see why it’s hard to prove for every number, let’s explore a slightly simpler function, ℊ.

g(n)= {n/2n+1, if n is even

{n+1, if n is odd

We might conjecture that orbits under ℊ always get to 1. I’ll call this the “Nollatz” conjecture, but we could also call it the n + 1 conjecture. We could explore this by testing more orbits, but knowing something is true for a bunch of numbers — even 268 of them — isn’t a proof that it’s true for every number. Fortunately, the Nollatz conjecture can actually be proved. Here’s how.

First, we know that half of a positive integer is always less than the integer itself. So if n is even and positive, then ℊ(n) = n/2 < n. In other words, when an orbit reaches an even number, the next number will always be smaller. …. This tells us that when an orbit under ℊ reaches an odd number greater than 1, we’ll always be at a smaller number two steps later. And now we can outline a proof of the Nollatz conjecture: Anywhere in our orbit, whether at an even or an odd number, we’ll trend downward. The only exception is when we hit 1 at the bottom of our descent. But once we hit 1 we’re trapped the loop, just as we conjectured. … [For f, the Collatz conjencture,] as with ℊ, applying f to an even number makes it smaller. And as with ℊ, applying f to an odd number produces an even number, which means we know what happens next: f will cut the new number in half.

But here’s where our argument falls apart. Unlike our example above, this number is bigger than n: 3n+12 = 3n2 + 12, and 3n2 = 1.5n, which is always bigger than n. The key to our proof of the Nollatz conjecture was that an odd number must end up smaller two steps later, but this isn’t true in the Collatz case. Our argument won’t work.

↩︎

Lambda Calculus and Lisp, part 2 (recursion excursion)

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Thu, 20 Feb 2025 01:26:00 -0600

From the previous entry in this series, one of the things of note in discussing the nature of the connections between LISP and (the) lambda calculus was John McCarthy’s concern about recursion and higher-order functions.

A couple of excerpts from previous quotes from McCarthy on the subject to set the stage:

…And so, the way in which to [be able to handle function passing/higher order functions] was to borrow from Church’s Lambda Calculus, to borrow the lambda definition. Now, having borrowed this notation, one the myths concerning LISP that people think up or invent for themselves becomes apparent, and that is that LISP is somehow a realization of the lambda calculus, or that was the intention. The truth is that I didn’t understand the lambda calculus, really. In particular, I didn’t understand that you really could do conditional expressions in recursion in some sense in the pure lambda calculus.…

[McCarthy 1978a:190¹]

…Writing eval required inventing a notation representing LISP functions as LISP data, and such a notation was devised for the purposes of the paper with no thought that it would be used to express LISP programs in practice. Logical completeness required that the notation used to express functions used as functional arguments be extended to provide for recursive functions, and the LABEL notation was invented by Nathaniel Rochester for that purpose. D.M.R. Park pointed out that LABEL was logically unnecessary since the result could be achieved using only LAMBDA — by a construction analogous to Church’s Y-operator, albeit in a more complicated way.…

[McCarthy 1978a:179¹]

Examining Church’s Y Combinator will be something we return to (probably in a number of posts), but I’ll defer discussion of it for the moment.

For now, let’s consider recursion in lisps. We’ll be talking a lot of recursion in Emacs Lisp today in fact.

Self-reference, self-embedding

Recursion is a concept or process depends on a simpler or previous version of itself. It’s ubiquitous, including in the natural world: Wikipedia notes that “Shapes that seem to have been created by recursive processes sometimes appear in plants and animals, such as in branching structures in which one large part branches out into two or more similar smaller parts. One example is Romanesco broccoli.”

Figure 1: Romanesco broccoli (Brassica oleracea), from Wikipedia

Recursion is a thing I deal with a lot in my day job, as it is a feature of natural language, especially syntax and semantics. To provide a quick illustration — though this is not at all how modern generative syntax is done anymore — consider phrase structure grammar and phrase structure rules used by Noam Chomsky and his colleagues in the 1950s.

Sentences (and linguistics objects generally) have formal structure, and it is part of the productive/creative nature of language that we might envision this structure as involving abstract structure rules that can be expanded in different ways (and then have vocabulary filled in).

A phrase structure rule will generally have the form A → [B C], indicating that A may be rewritten or expanded into [B C]. (In the following, I’ll use round brackets ()'s to denote optional constituents.)

So, a subset of these rules for English might include something like (note that some things have multiple rules that can apply to them):

1. S → NP VP
2. NP → (Det) N
3. N → (AdjP)* N
4. VP → V
5. VP → V NP (NP)
6. VP → V CP
7. CP → (Comp) S
...

Code Snippet 1: a snippet of phrase structure grammar rules for English [Nb.: again, not prolog, but maybe the best fontlocking choice here]

(Where S is “sentence”; Det is a determiner (like “the”, “a”); NP is a noun phrase; Nₙ is a noun head; AdjP is an adjective phrase; VP is a verb phrase; V is a verb head; CP is a complementiser phrase; Comp is a complementiser (like “that”).)

So we can rewrite S as NP VP (1) and then rewrite NP VP as Det N VP (2, choosing an optional Det) and then Det N VP as Det N V (4) and then insert lexical items of the appropriate category into the ‘heads’ (the non-P elements). So we might choose “the” for Det and “cat” for N and “purrs” for V, and get the sentence “the cat purrs”.

But note that some of the rules allow for expansion into elements that contain expansions back into themselves. So rule (1) allows an S to exapnd into NP VP and rule (6) allows for a VP to expand into V CP and rule (7) allows for a CP to expand into an S. At which point we can apply rule (1) again to expand the new S into NP VP, and then repeat this process as many times as we like:

S
NP VP
N VP
N V CP
N V Comp S
N V Comp NP VP
N V Comp N VP
N V Comp N V CP
N V Comp N V Comp S
N V Comp N V Comp NP VP
N V Comp N V Comp N VP
N V Comp N V Comp N V CP
N V Comp N V Comp N V Comp S
N V Comp N V Comp N V Comp NP VP
N V Comp N V Comp N V Comp N VP
N V Comp N V Comp N V Comp N V CP
N V Comp N V Comp N V Comp N V Comp S
N V Comp N V Comp N V Comp N V Comp NP VP
...

Code Snippet 2: a recursive English sentence expansion [Nb.: again, not prolog, but maybe the best fontlocking choice here]

And it’s easy to imagine examples of what such a sentence could be like, e.g., “John said that Sita said that Bill said that Mary said that Ram said that Kim said that…". It won’t be infinitely long, but there’s no particular theoretical bound on how long it could be (memory processing and finite human lifespans will impose practical limits, of course).

On the formal semantics side, things are similar: consider that logical languages too (which are often used to formalise natural language semantics) allow for recursion. Propositional logic construction with Boolean operators too have no theoretical upper limits: we can write (t ↔ (p ∧ (q ∨ (r → (s ∧ ¬¬¬¬¬¬t))))),² and there’s nothing which prevents composing this bit of formalism with yet another bit, and so on.

And in mathematics and computer science, recursion is often a thing which suggests itself.

For instance, though there are other ways of calculating it, the Fibonacci sequence³, i.e., a sequence of numbers in which each element is the sum of the two elements that precede it (e.g. 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144…).

A natural way of writing an equation to calculate these is something like:

Fib(0) = 0 as base case 1.

Fib(1) = 1 as base case 2.

For all integers n > 1, Fib(n) = Fib(n − 1) + Fib(n − 2).

Rule 3 makes reference to itself. I.e., in order (by this method) to calculate Fib(6), you have to calculate Fib(5), for which you have to calculate Fib(4)), for which you have to calculate Fib(3), for which you have to calculate Fib(2), which you can then base on rules (1) & (2): you can add 0 and 1 (= Fib(0) and Fib(1)) together to get Fib(2), and then you can calculate Fib(3) by adding Fib(1) and Fib(2) and so on.

Cursed recursion

In early Lisp(s), despite the concern around recursion, writing recursive functions was/is not always pragmatically viable, because it can lead to stack overflows (potential infinities are hard in practice).

Scheme, a Lisp dialect originally created at MIT by Guy L. Steele, Jr. and Gerald Jay Sussman, was the first Lisp to implement tail call optimisation, which is a way of making significantly recursive functions viable, making tail calls similar in memory requirement to their equivalent loops.

Before talking (a little bit more) about tail recursion, let’s look at concrete examples of a tail recursive function and its loop equivalent. We’ve mentioned Fibonacci numbers already, so let’s try to write functions calculate these (in Emacs Lisp).

Following the mathematical abstraction for the Fibonacci sequence above, we could write a function like this:

;; -*- lexical-binding: t; -*-
(defun fib1 (n a b)
  "Calculate the first `n' Fibonacci numbers, recursively."
  (if (< n 1)
      a
    (cons a
          (fib1 (1- n) b (+ a b)))))

Code Snippet 3: a first go at a recursive elisp function for fibonacci numbers

I’ve tried to make this as simple as possible, we could make it nicer (say, with a wrapper function or some some of flet) so that we didn’t have to pass in the initial values. But, keeping it simple (for now): fib1 is a function taking three arguments: n, the quantity of Fibonacci numbers to return; a, the first number to start with; and b, the second number to start with.

Following the schema above, we’re going to pass 0 for a and 1 for b. Let’s get the first ten numbers of the sequence, and pass 10 for n:

(fib1 10 0 1)  ; (0 1 1 2 3 5 8 13 21 34 . 55)

I know, the output is a bit ugly because we’re just cons’ing the results and so it’s not a proper list, but we’re keeping things simple.

Let’s walk through how it works. The function first looks at n, if n is less than 1, it returns a (whatever it is). If n isn’t less than 1, we return the cons of a with the result of calling fib1 itself on “n minus 1” b and “a plus b”.

So if we start with n=3, a=0, b=1, that is, evaluate (fib1 3 0 1), the function would say, well, 3 isn’t less than 1, so I’m going to create a cons with 0 and the result of calling (fib1 2 1 1) (2 = “n minus 1”, because n is currently 3; 1 because b=1; and 1 because a + b = 0 + 1 = 1).

So at this point we have a cons that looks like this:

(0 . (fib1 2 1 1))  ;; [= (cons 0 (fib1 2 1 1))]

When we evaluate (fib1 2 1 1), n is still not less than 1, so we’re going to cons the current value of a (which is 1) with another call to fib1: (fib1 1 1 2) (1 = “n minus 1”, because n is currently 2; 1 because b=1; and 2 because a + b = 1 + 1 = 2).

So now we have:

(0 1 . (fib1 1 1 2))  ;; [= (cons 0 (cons 1 (fib1 1 1 2)))]

Now we have to evaluate (fib1 1 1 2). n is still not less than 1; so we create another cons of 1 (as a is currently 1) with yet another call of fib1: (fib1 0 2 3) (0 = “n minus 1”, because n is currently 1; 2 because b=2; and 3 because a + b = 1 + 2 = 3). And so now:

(0 1 1 . (fib1 0 2 3))  ;; [= (cons 0 (cons 1 (cons 1 (fib1 0 2 3))))]

And, finally, evaluating (fib1 0 2 3), now n is less than one, so we take the first branch of the conditional and just return a, which is 2. So the result of starting with (fib1 3 0 1) is:

(0 1 1 . 2)  ;; [= (cons 0 (cons 1 (cons 1 2)))]

And you can try this with other values of n, e.g., try evaluating (fib1 100 0 1) to get the first 100 members of the sequence.⁴

But, at least for me on Emacs 30.0.93, 529 is the limit. If we try (fib1 520 0 1), the debugger pops up with a 1622 line long error, which begins:

Debugger entered--Lisp error: (excessive-lisp-nesting 1622)
  cl-print--cons-tail(excessive-lisp-nesting (1601) #<buffer  *temp*>)
  #f(compiled-function (object stream) #<bytecode 0x491d55561699a09>)(#0 #<buffer  *temp*>)
  apply(#f(compiled-function (object stream) #<bytecode 0x491d55561699a09>) (#0 #<buffer  *temp*>))
  #f(compiled-function (&rest args) #<bytecode 0x1c31d892b8046a8b>)()
  #f(compiled-function (cl--cnm object stream) #<bytecode 0x7c361f66f109692>)(#f(compiled-function (&rest args) #<bytecode 0x1c31d892b8046a8b>) #0 #<buffer  *temp*>)
  apply(#f(compiled-function (cl--cnm object stream) #<bytecode 0x7c361f66f109692>) #f(compiled-function (&rest args) #<bytecode 0x1c31d892b8046a8b>) (#0 #<buffer  *temp*>))
  #f(compiled-function (object stream) #<bytecode 0x1f277a9a6dc403fa>)(#0 #<buffer  *temp*>)
  apply(#f(compiled-function (object stream) #<bytecode 0x1f277a9a6dc403fa>) #0 #<buffer  *temp*>)
  cl-print-object(#0 #<buffer  *temp*>)
  cl-prin1(#0 #<buffer  *temp*>)
  backtrace--print(#0 #<buffer  *temp*>)
  cl-print-to-string-with-limit(backtrace--print #0 5000)
  backtrace--print-to-string(#0 nil)
  backtrace-print-to-string(#0)
  debugger--insert-header((error #0 :backtrace-base eval-expression--debug))
  #f(compiled-function () #<bytecode 0x299e53d67d1ac62>)()
  backtrace-print()
  debugger-setup-buffer((error #0 :backtrace-base eval-expression--debug))
  debug(error #0 :backtrace-base eval-expression--debug)
  eval-expression--debug(#0)
  (if (< n 1) a (cons a (fib1 (1- n) b (+ a b))))
  fib1(0 259396630450514843945535792456880074043523940756078363514486570322782139633750401577338505233670220572153381665 419712564636128966418863068957011388899128076671547993021605479585858227224221424221791102364954108601240491394)
  (cons a (fib1 (1- n) b (+ a b)))
  (if (< n 1) a (cons a (fib1 (1- n) b (+ a b))))
  fib1(1 160315934185614122473327276500131314855604135915469629507118909263076087590471022644452597131283888029087109729 259396630450514843945535792456880074043523940756078363514486570322782139633750401577338505233670220572153381665)
  (cons a (fib1 (1- n) b (+ a b)))
  (if (< n 1) a (cons a (fib1 (1- n) b (+ a b))))
  fib1(2 99080696264900721472208515956748759187919804840608734007367661059706052043279378932885908102386332543066271936 160315934185614122473327276500131314855604135915469629507118909263076087590471022644452597131283888029087109729)
....

Code Snippet 4: beginning of excessive-lisp-nesting error for our fib1 function

Because we’ve built up a long, deeply-embedded list of conses and Emacs has a limit of how deep it’s willing/able to go (you can see above that we’ve almost made it to the end, just needing to calculate fib1(0), when Emacs decides it’s had enough).

In Scheme and elsewhere, self-recursive calls at the ends (“tails”) of functions can be optimised to avoid these sorts of stack overflows (excessive-lisp-nesting).⁵ Tail-call optimisation lets procedure calls in tail positions be treated a specialised GOTO statements, which can be efficiently processed:

…only in cases where structures are explicitly declared to be dynamically referenced should the compiler be forced to leave them on the stack in an otherwise tail-recursive situation. In general, procedure calls may be usefully thought of as GOTO statements which also pass parameters, and can be uniformly encoded as JUMP instructions. This is a simple, universal technique, to be contrasted with […] more powerful recursion-removal techniques…

[Steele 1977:155⁶]

But our fib1 function doesn’t do this, and we end up flooded the stack with too many conses.

Back to loops

Recursive functions are perhaps most idiomatic in Scheme (among lisps, I mean). Some implementations of Common Lisp can do tail-call optimisation, but loops are perhaps more common, and certainly in Emacs Lisp (for reasons you can see above), loops are usually what are used. And so we can write a new Fibonacci function with a loop. It won’t be nearly as pretty, but it’ll work better.

Here’s one possible implementation:

;; -*- lexical-binding: t; -*-
(defun fib2 (n a b)
  "Calculate the first `n' Fibonacci numbers,
in a loop."
  (let ((result nil))
    (dotimes (c n)
      (setq result (cons a result))
      (setq tempa b)
      (setq b (+ a b))
      (setq a tempa))
    (nreverse result)))

(fib2 10 0 1) ; (0 1 1 2 3 5 8 13 21 34)

(setq bigfib-50000 (fib2 50000 0 1)) ; this will work - we can get 50,000 numbers

Code Snippet 5: a second go at a fibonacci function, with looping

The result is prettier at least: a proper rather than an improper list (because we started by cons'ing onto an empty list). Our fib2 function itself isn’t as mathematically pleasing as our fib1 function, we end up with a lot of setq's (the nreverse at the end reverses our list, because the way we build up our list is by cons'ing the first results first, so they end up at the end until we flip them with nreverse). But it works well. If you try to (fib2 100000 0 1), it’ll fail, but not because of stack overflow, just because we end up with numbers that are too big for Emacs. But you can certain get the over 50,000 members of the Fibonacci sequence, which is much better than fib1's limit of 529.

And dotimes is just one loop procedure available. (See cl-loop for a more powerful one.)

Optimal Tail with Emacs

Ok, so, practically, we should probably generally prefer loops over tail-recursive functions in Emacs. But, what if we just like the latter more?⁷ Are there any other possibilities?

Wilfred Hughes has an emacs package tco.el which implements a special macro for writing tail-recursive functions.⁸ It works by replacing each self-call with a thunk, and wrapping the function body in a loop that repeatedly evaluates the thunk. Thus a function foo defined with the defun-tco macro:

(defun-tco foo (...)
  (...)
  (foo (...)))

would be re-written as:

(defun foo (...)
   (flet (foo-thunk (...)
               (...)
               (lambda () (foo-thunk (...))))
     (let ((result (apply foo-thunk (...))))
       (while (functionp result)
         (setq result (funcall result)))
       result)))

And this delays evaluation in such a way as to avoid stack overflows. Unfortunately, at least currently for me (Emacs 30.0.93 again), tco.el seems to have some issues.

In Emacs 28.1, cl-labels (one of the ways of sort of doing let's for functions) gained some limited tail-call optimisation (as did named-let, which uses cl-labels).

;; -*- lexical-binding: t; -*-
(defun fib3 (n)
  "Calculate the first `n' Fibonacci numbers,
recursively, with limited tail-call optimisation
through `cl-labels'?!"
  (cl-labels ((fib* (n a b)
                (if (< n 1)
                    a
                  (cons a
                        (fib* (1- n)
                              b
                              (+ a b))))))
    (fib* n 0 1)))

(setq bigfib3 (fib3 397)) ; 396 highest that works

Code Snippet 6: a third go at a fibonacci function, with cl-labels

At least the way I’ve written it, it seems to suffer an overflow even sooner (at 397 rather than 529 as for our fib1), because it has to come back and do the cons after the tail-call — so fib* isn’t actually in the tail. (We can fix this in at least one way, which we’ll do in fib5.)

We could try to write an accumulator as a hack, where we try to do ours conses one at a time and pass along the results, but this fares no better than our fib1:

;; -*- lexical-binding: t; -*-
(defun fib4 (n a b accum)
  "Calculate the first `n' Fibonacci numbers, recursively,
but collect conses as we go and keep track of the length of
the `accum' cp. against `n'."
  (let* ((accum (cons a accum))
         (accum-lng (length accum)))
    (if (< n accum-lng)
        (nreverse accum)
      (fib4 n b (+ b a) accum))))

(setq bigfib4-529 (fib4 529 0 1 nil)) ; last good
(setq bigfib4-530 (fib4 530 0 1 nil)) ; overflows

Code Snippet 7: a fourth go at a fibonacci function, with an accumulator

If we combine cl-labels and the accumulator trick, however, we do seem to be able to escape stack overflows, because now we’ve got fib* properly in the tail:

;; -*- lexical-binding: t; -*-
(defun fib5 (n)
  "Calculate the first `n' Fibonacci numbers, recursively,
using both cl-labels and the accumulator trick."
  (cl-labels ((fib* (a b accum)
                   (let* ((accum (cons a accum))
                         (accum-lng (length accum)))
                     (if (< n accum-lng)
                         (nreverse accum)
                         (fib* b (+ b a) accum)))))
            (fib* 0 1 nil)))

(setq bigfib5-10000 (fib5 10000)) ; ok
(setq bigfib5-50000 (fib5 50000)) ; very slow, but ok

Code Snippet 8: a fifth go at a fibonacci function, with cl-labels and an accumulator

Now we’re back in the realms of what our fib2 non-recursive loop-style function could do. Although (setq bigfib5-50000 (fib5 50000)) calculates very slowly (worse than our looping fib2), so that’s not ideal.

Stream of Conses-ness

But here’s another possibility: Nicholas Pettton‘s stream package for emacs, where “streams” are delayed evaluations of cons cells.

;; -*- lexical-binding: t; -*-
(defun fib6 (n)
  "Return a list of the first `n' Fibonacci numbers,
implemented as stream of (delayed evaluation) conses."
  (cl-labels ((fibonacci-populate (a b)
                (stream-cons a (fibonacci-populate b (+ a b)))))
    (let ((fibonacci-stream
           (fibonacci-populate 0 1))
          (fibs nil))
      (dotimes (c n)
        (setq fibs (cons (stream-pop fibonacci-stream) fibs)))
      (nreverse fibs))))

(setq fib6-10k (fib6 10000)) ; ok
(setq fib6-50k (fib6 50000)) ; little slow, but works
(setq fib6-100k (fib6 100000)) ; little slow & overflow error

Code Snippet 9: a sixth go at a fibonacci function, with delayed evaluation conses

This works well. (fib6 50000) still turns out to run a bit slower than our (fib2 50000), so loops are still probably more efficient, but streams are pretty interesting. They can can used to represent infinite sequences. So here, above, fibonacci-stream (set by (fibonacci-populate 0 1)) is actually an infinite stream of Fibonaccis numbers, but lazily evaluated, so we just get the next one each time we call stream-pop on our fibonacci-stream local variable. (What happens is that stream-pop takes the car of fibonacci-stream, evaluates and returns it, and then sets fibonacci-stream to be its cdr (i.e., popping off and “discarding” the first element; which which captured in our fibs collector.))

Cascades of Fibonacci numbers

Oh, incidentally and irrelevantly, if you inspect the contents of your fib6-50k, it’s very aesthetically pleasing, a cascade of numbers:

Figure 2: fibonacci numbers burst forth from their seeds and spill out into the buffer

`excessive-lisp-nesting`: (overflow-error)

I had hoped to get to the Y Combinator today (and think I might have suggested a promise of that), for that’s where things really get interesting. And we need to get back to lambda calculus, of course. But we may be near the limits of excessive lisp nesting ourselves here.

However, the recursion discussion here has set the stage for the Y Combinator, which we’ve already talked a couple of times, especially in connection to John McCarthy’s claims about “not really understanding” lambda calculus and the fact that these really centre on his not seeing how one could get recursion without direct Self-reference (and thus the need for LABEL) because of not knowing about the Y Combinator.

Figure 3: a close-up of a section of Michał “phoe” Herda’s hand-illuminated Y Combinator Codex showing part of a Fibonacci defun

And, the Y Combinator ties in with all sorts of other curious things. Paradoxes, types, calligraphy.

[Thus, I ended up with an excursus on this excursus as the next post. I’ll put a link here to the proper third part when it’s up.]

McCarthy, John. 1978a. History of Lisp. In History of programming languages, ed. Richard L. Wexelblat, 173–185. New York: Association for Computing Machinery. https://dl.acm.org/doi/10.1145/800025.1198360 ↩︎
“t if and only if p and q or if r then s and not not not not not not t” ↩︎
A number of Indian philosophers, at least as far back as Virahāṅka (ca. AD 600–800), gave formulations for what is usually called the Fibonacci sequence. See Singh, P. (1985). The so-called fibonacci numbers in ancient and medieval India. Historia Mathematica, 12(3), 229–244. https://doi.org/10.1016/0315-0860(85)90021-7 [pdf] ↩︎
You might actually want to do something like (setq my-fib1-100 (fib1 100 0 1)) to put the result into a variable, because the echo area at the bottom of Emacs isn’t big enough for all of the numbers. (Oh, to eval things in an Emacs buffer, put your cursor/point at the end of the expression and press C-x C-e. But don’t do that for this one. If you want Emacs to just stick the results in directly into the buffer rather than echoing them, press C-u C-x C-e. But don’t do that here either, because Emacs will still end up printing an ellipsis because it thinks it’s too long.) And then press C-h v and type my-fib1-100 and enter to see the result. ↩︎
See further here, for instance, for more about tail calls and tail call optimisation. ↩︎
Steele, Guy L., Jr. 1977. Debunking the “expensive procedure call” myth or, procedure call implementations considered harmful or, LAMBDA: The Ultimate GOTO. ACM ‘77: Proceedings of the 1977 annual conference, 153–162. [https://dl.acm.org/doi/10.1145/800179.810196] ↩︎
If you’ve read any of Paul Graham’s Common Lisp books (e.g., On Lisp) or any of the Little Schemer books (the latter with Duane Bibby‘s lovely artwork in them), you may be disposed towards using recursion rather than loops.

Figure 4: close-up of Duane Bibby’s cover illustration for “The Little Schemer”

↩︎
One of my emacs packages, Equake, actually used to use tco.el (until commit #0ab0801 [2020-08-24 Mon]). ↩︎

Using Emacs and Org-Roam/Org-Node on Android (with Termux Extra Keys)

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Thu, 30 Jan 2025 14:15:00 -0600

My main use of Emacs on Android (via Termux) is to be able to access and add to my Org-roam library of notes. This post is primarily about some Termux features which improve the user experience for this use case. [Update [2025-02-03 Mon]: added some screenshots throughout to give a sense of what it looks like.]

Trying to Cope with Emacs on mobile

I’ve tried a number of different solutions for managing sync’ed Org files on mobile¹, and some of these are useful for some purposes, but to being able to access and add to my Org-Roam notes, I’ve found I really need a full-blooded Emacs instance.²

And dealing with interacting with Emacs on a touchscreen interface/touchscreen keyboard is a bit of a nightmare.

Here, we’ll try to improve it a little. Into the rabbit-hole….

Note about using Termux for Emacs on Android

You’re probably better off installing Termux from F-Droid rather than from the Google Play Store [see the note from the official Termux maintainers at: https://github.com/termux/termux-app/discussions/4000]. This is especially true is you want to use Emacs in Termux for managing Org-roam files which are shared/kept in sync with your Emacs on desktop machines, which you can do with Syncthing (use Catfriend1’s Syncthing-fork [Github]/[F-Droid] on Android), in order to be able to grant Termux (and thus Emacs) access to your shared Org files.

Termux Extra Keys

Termux has a very useful feature adding extra keys above the system keyboard. These are quite useful, including ‘sticky’ CTRL and ALT keys. (In addition, one can hold volume-down and press a key for CTRL + [that-key] or volume-up and press a key for ALT + [that-key]; with volume-up + t inputting TAB. At least on my phone, using the volume keys as modifiers is fairly awkward/uncomfortable, so I prefer these extra keys.)

[Nb.: These keys are toggled on and off by pressing volume-up + k or volume-up + q. This is important to note, for I once toggled them off by accident and couldn’t understand what had happened to them for several months.]

Also worth mentioning is the fact that:

swipe left on extra keys brings up swipe-compatible/regular keyboard typing
swipe right on regular keyboard typing area (where extra keys was) will get you back to extra keys

The first of these lets you use any of your keyboard’s regular features and so can be quite useful.

See the Termux manual on Extra Keys for a description and basic configuration and usage instructions.

Here, I explore the possibilities offered by the advanced configuration options for Extra Keys for navigating Emacs in Termux, especially Org-Roam or Org-node ³, to create faster/easier ways of calling commonly used commands since the user-interface to Emacs with an on-screen keyboard is often somewhat frustrating compared to the using Emacs with a physical keyboard. In addition to extra tappable keys, Extra Keys also allows for different outcomes when the user does swipe-up on a key, including the outputting of key sequences, which can include modifier keys like CTRL. This allows the possibility of loading them with Emacs key sequences which would otherwise be much slower to access with an onscreen keyboard.

What I wanted was:

An easy, single-key (no modifier) way of doing forward and backward searches. It is painful to either hold volume-down or have to keep on toggling the Extra Keys’ CTRL for each next/previous result. So I recruited ♡ and ♤ for forward and back searches, respectively (because ♡ is sort of pointed down and ♤ is sort of pointing up), and bound these appropriately in my Emacs configuration (see below).
other convenient single-key buffer navigation interface features, e.g., arrow keys and page-up and page-down
some convenience things like closing all but the current window (delete-other-windows = C-x 1), opening up iBuffer, switch-buffer
slashes and - and ~ available immediately to tap insert (not under my keyboard’s symbol toggle or long-hold)
ability to call some Org-mode/Org-node commands quickly: including finding nodes, inserting links to nodes, capturing new dailies, committing captures, going to today’s daily entry, moving forward and backwards through the dailies, toggling the org-roam backlinks buffer

Example configuration

The configuration outlined below is partially specific for my Emacs configuration, especially for the C-c (CTRL c) keys, but some things involve default Emacs keybindings and it can at least serve as a model for what one can do. To configure Extra Keys, you need to modify the extra-keys part of ~/.termux/termux.properties in this fashion:

extra-keys = [[ \
  {key: 'ESC', popup: {macro: "CTRL x 1", display: "C-x 1"}}, \
  {key: 'CTRL', popup: {macro: "CTRL c", display: "C-c"}}, \
  {key: 'ALT', popup: {macro: "ALT x", display: "M-x"}}, \
  {key: '♡', popup: {macro: "CTRL x r b", display: "bookmarks"}}, \
  {key: '~', popup: {macro: "CTRL c n d", display: "today"}}, \
  {key: 'UP', popup: {macro: "CTRL c n n", display: "new note"}}, \
  {key: '-', popup: {macro: "CTRL c CTRL k", display: "cancel note"}}, \
  {key: 'PGUP', popup: {macro: "CTRL c n y", display: "prev note"}}], \
  [ \
  {key: 'TAB', popup: {macro: "CTRL x b", display: "alttab"}}, \
  {key: '/', popup: {macro: "CTRL c n f", display: "find node"}}, \
  {key: '\\\\', popup: {macro: "CTRL c n i", display: "insert link"}}, \
  {key: '♤', popup: {macro: "CTRL x CTRL b", display: "ibuffer"}}, \
  {key: 'LEFT', popup: {macro: "CTRL c n l", display: "backlinks"}}, \
  {key: 'DOWN', popup: {macro: "CTRL c CTRL c", display: "commit"}}, \
  {key: 'RIGHT', popup: {macro: "CTRL c n v", display: "goto-date"}}, \
  {key: 'PGDN', popup: {macro: "CTRL c n t", display: "next note"}}]]

Here is a rough summary of what this does:

key	do on key tap	do on key swipe up	display on swipe up	what key swipe up does
`ESC`	‘esc’	`C-x 1`	“C-x 1”	`delete-other-windows`
`CTRL`	‘ctrl’ (`C-`) toggle	`C-c`	“C-c”	start `C-c` prefix command
`ALT`	‘alt’ (`M-`) toggle	`M-x`	“M-x”	`M-x`
`♡`	`isearch-forward`	`C-x r b`	“bookmarks”	`bookmark-jump`
`~`	‘~’	`C-c n d`	“today”	`org-node-goto-today`
`↑`	‘up’	`C-c n n`	“new note”	`org-roam-dailies-capture-today`
`-`	‘-’	`C-c C-k`	“cancel note”	`C-c C-k` to cancel capture
`PGUP`	‘pageup’	`C-c n y`	“prev note”	`org-node-goto-next-day`
`TAB`	‘tab key’	`C-x b`	“alttab”	`consult-buffer` (`switch-to-buffer`)
`/`	‘/’	`C-c n f`	“find node”	`org-node-find`
`\`	‘\’	`C-c n i`	“insert link”	`org-node-insert-link*`
`♤`	`isearch-backward`	`C-x C-b`	“ibuffer”	`ibuffer`
`←`	‘left’	`C-c n l`	“backlinks”	`org-roam-buffer-toggle`
`↓`	‘down’	`C-c C-c`	“commit”	`C-c C-c` to ‘commit’
`→`	‘right’	`C-c n v`	“goto-date”	`org-node-goto-daily`
`PGDN`	‘pagedown’	`C-c n t`	“next note”	`org-node-goto-prev-day`

In case you do use Org-node (and, if you’re trying to do Org-roam things on Android, I would again recommend considering Org-node, if only for making interactions fast enough to not be completely frustrating), you could implement the following interactive functions:

(defun org-node-goto-daily ()
  (interactive)
  (org-node-seq--jump "d"))

(defun org-node-goto-today ()
  (interactive)
  (org-node-seq-goto "d" (format-time-string "%F")))

(defun org-node-goto-next-day ()
  (interactive nil org-mode)
  (org-node-seq--goto-next "d"))

(defun org-node-goto-prev-day ()
  (interactive nil org-mode)
  (org-node-seq--goto-previous "d"))

(defun org-roam-dailies-capture-today ()
  (interactive)
  (org-roam-dailies-capture-today nil "d"))

And then bind:

org-node-insert-link* to “C-c n i”
org-node-find to “C-c n f”
org-node-goto-daily to “C-c n v”
org-node-goto-today to “C-c n d”
org-node-goto-next-day to “C-c n t” ('[t]omorrow’)
org-node-goto-prev-day to “C-c n y” ('[y]esterday’)
org-roam-dailies-capture-today to “C-c n n”
org-roam-buffer-toggle to “C-c n l”

Most of the other keybindings implemented above are default Emacs bindings.

In terms of how this improves the Termux Emacs experience, consider that you could open up Emacs in Termux and then proceed by:

swiping up on ↑ to open up a new Org-[roam/node] note in today’s daily journal
then swiping left on the extra keys pane to get into normal keyboard entry mode
swipe/glide-typing the text of your note quickly, and pressing enter
swiping right on the now clear virtual keyboard pane to bring back the extra keys
swiping up on \ to insert (‘splice in’) a org-id link to another Org-roam node or two
swiping up on ↓ to commit/complete your daily note
swiping up on PGUP to see yesterday’s Org-roam daily journal
swiping up on / to search for your “Termux” Org-roam node to add notes about Extra Keys to your Termux Org-roam node/file
tapping ♡ to search through the entry to see all the places you made reference to “extra keys”
swiping up on ← to see what backlinks you have for “Termux”
swiping up on ESC to get back to single window view
swiping up on ♤ to get to IBuffer to be able to manage your buffers
swiping up on ↑ to open another new daily note in today’s journal
changing your mind and swiping up on - to cancel the note capture
…..

This is all a lot faster than having to tap CTRL and then type c n n to start a new entry in today’s Org-roam daily journal, typing it out (tapping CTRL and then c n i each time you want to link text to another node), and then tapping CTRL and then c and then CTRL and then c to commit it, and so on.

For a bonus reduction in modifier and key tapping, consider using a modal mode in Emacs: I particularly like meow.

Appendix: Key Mnemonics

A potentially useful/emblematic list of mnemonics for the keys. [I use ☝️ to indicate swiping up on a key below.] Some of these mnemonics are stretches, but there are only so many keys and a particular set of things one wants, so stretches are sometimes required.

♡ is a forwards/downwards search because the heart points down
♤ is a backwards/upwards search because the spade points up [and we use these pointy characters indicating card suits in preference to others because I’m not going to be typing them on Android ever]
(all other single tap keys just do what it says on the tin, inserting characters, toggling modifiers, &c.)
☝️ ♡ is bookmarks because these are ‘favourites’ because you ‘♡’ them
☝️ ♤ is IBuffer because if you have icons enabled in your IBuffer⁴ then you’ll see icons and ♤ is like an icon (I know this one is a stretch, but….⁵)
☝️ ESC is delete-other-windows because you’re ‘escaping’ from them
☝️ CTRL is C-c because C-c starts with CTRL
☝️ ALT is M-x because M-x starts with ALT
☝️ ~ is org-node-goto-today because in Unix ~ is “user’s home directory” and today’s daily Org-roam entry is a sort of a ‘home’ (it makes sense to me anyway)
☝️ - is C-c C-k is which ‘cancel’ in lots of places (like capture templates) and - is a negative sign and so can be understood like “don’t do it!”
☝️ ↑ is org-roam-dailies-capture-today because this is the function to capture a new Org-roam daily note and ↑ is like “opening up something”
☝️ ↓ is C-c C-c which is in many places in Emacs something like ‘commit’, and for capture templates it is the ‘finish capture’ binding and if ↑ is opening a new note, then ↓ is finishing it, putting it ‘down’
☝️ ← is open up Org-roam backlinks buffer because ← is pointing ‘backwards’
☝️ → is org-node-goto-daily because we’re looking ‘forward’ towards a specific date and going to it
☝️ PGUP is org-node-goto-prev-day because we’re going ‘up’/‘backwards’ in the calendar (of daily Org-roam journal entries)
☝️ PGDN is org-node-goto-prev-day because we’re going ‘down’/‘forwards’ in the calendar (of daily Org-roam journal entries)
☝️ TAB is consult-buffer / switch-to-buffer because this function is sort of like Alt-TAB in many window environments
☝️ / is org-node-find because / is the search key in Vim, and also associated with searches elsewhere in Unixland (the marker of beginning/end of regexs, often used for searches) [and because / is the opposite slash from \, which we’re going to use for the complementary operation to insert links]
☝️ \ is org-node-insert-link* because in many environments \ is some sort of ‘splice’ indicator⁶, and inserting a link to a node is also sort of a splice [and because \ is the opposite slash from / which performs a complementary operation]

E.g., orgzly, organice. ↩︎
There is a full GUI Emacs now on Android, but I don’t know how to get it to be able to access files shared via Syncthing, and, anyway, you wouldn’t be able to do the sort of convenient keyboard hacks there I’m going to discuss in this post. ↩︎
I highly recommend looking at Martin Edström’s Org-node package if you’re trying to manage Org-roam notes on Android; the performance differences between the two packages are especially noticeable on Android (orders of magnitude⁷). And the two packages can live alongside each other, so you can keep your existing Org-roam set-up going while you set-up/experiment with Org-node. ↩︎
With a package like https://github.com/seagle0128/all-the-icons-ibuffer. On IBuffer itself, see https://www.emacswiki.org/emacs/IbufferMode. ↩︎
Well, okay, so here’s another one: think of IBuffer as trying to right your buffer organisation like Two, Five, and Seven (of Spades) in Alice in Wonderland trying sort out the Queen of Hearts’ rose-bushes (by painting the roses red): ↩︎
From https://port70.net/~nsz/c/c11/n1570.html#5.1.1.2:

Each instance of a backslash character (\) immediately followed by a new-line character is deleted, splicing physical source lines to form logical source lines. Only the last backslash on any physical source line shall be eligible for being part of such a splice. A source file that is not empty shall end in a new-line character, which shall not be immediately preceded by a backslash character before any such splicing takes place.

↩︎
Android 10/11 seems to have introduced which restricts how regular applications/users can interact with the filesystem, for ‘security’ reasons, implemented via Filesystem in Userspace (FUSE), which makes operations which interact with the file system sometimes very sluggish. Accessing directories with lots of files, as one might have in an Org-roam directory, can be amazingly slow. The increases exponentially with the number of Org-roam files.

Org-node is implemented in such a way [cf. https://github.com/meedstrom/org-node/issues/26] that it caches/stores and accesses many things in ways that avoid directly accessing the file-system (especially in org-node-fakeroam), where Org-roam does not. ↩︎

Guix, XIM, Emacs, Multi_key, Shft+SPC

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Thu, 30 Jan 2025 12:22:00 -0600

— Describe in single words only the good things that come into your mind about… <Multi_key>.

— <Multi_key>?

— Yeah.

— Let me tell you about <Multi_key>….

"<Multi_key> is undefined"

A bit out of order, but things tangle, a problem I’m having on my Guix machine with Emacs.

It’s connected with the Lucid toolkit¹, but only in an indirect causal sort of way:— I’m using the standard GTK3 toolkit on this particular build of Emacs on Guix. (A custom build, admittedly, which might be part of the issue, but where exactly, I’m not sure.) I’ll probably say some other things connected to this later, but for the moment forget about Lucid. It’s just a GTK3 toolkit build.²

So, skipping over large swathes of story, I’ve got a seemingly mostly-functioning build of Emacs 30.0.93 running on my Guix laptop (which finally has a functioning StumpWM environment again), and I go to type æ or þ or á or λ or something using Right_Alt as a compose key (defined by xmodmap) and Emacs, without blinking an eye, flashes at me, saying "<Multi_key> is undefined".

I try various things to do with XIM and environment variables³ and none of it works. I finally come across a post on a FreeBSD bugs tracker⁴ which suggests a possible fix of setting (setq x-gtk-use-native-input 't).

And, though it feels like a hack for something that’s gone wrong in a deeper way, it seems to work, and I can type ɔ and Þ and γ and so on.

But then I’m reading a PDF (using PDF Tools) and I go to page back up to read an earlier bit by pressing Shift and SPACE and I find myself paging down to the end of the document instead.

The “native input” method apparently won’t recognise Shft+SPACE, but just sees SPACE and so pages down.

So if x-gtk-use-native-input is nil, I can’t use the compose key in Emacs, and if it’s t then I can’t scroll properly in PDFs.

I thought vaguely about writing a macro to process the ‘kragen’ XCompose definitions into a bunch of things of the form:

(define-key global-map [(Multi_key) (s) (s)] (lambda ()
                                                   (interactive)
                                                 (insert 223)))

and then have my init.el load the whole thing in. But I don’t think that would quite work anyway, since sometimes I want to say insert a special character as part of a search and not insert it into a text buffer.

So I came up with a different stupid solution. It’s not perfect, but it’s at least somewhat working while I try to figure out what went wrong in the first place.

Here it is:

(when (equal (system-name) "guix-laptop")
  ;; on this Guix/StumpWM machine's Emacs, xcompose doesn't work:
  ;; Emacs says "I don't know what <Multi_key> means"
  ;; (well, it actually says "<Multi_key> is undefined").
  ;; As per https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=278167
  ;; we can 'fix' this if we set (setq x-gtk-use-native-input 't).
  ;; Then xcompose will will work again ... but now Shift-SPACE won't

  ;; So...

  (setq x-gtk-use-native-input 'nil) ;; make sure Shift-SPACE works...

  ;; So, Emacs says "<Multi_key> is undefined".
  ;; Well, then, we can define it.
  ;; Namely as a function which sets `x-gtk-use-native-input' to `t',
  ;; and then calls an external shell-command to input <Multi_key>
  ;; (so that it acts like Right_Alt was pressed), and then sets a timer running
  ;; that turns `x-gtk-use-native-input' to `nil' after 10 seconds
  ;; (so that Shift-SPACE will work again).

  (defun bms/xinput-hack ()
    (interactive)
    ;; stop the timer if there is one running already:
    (when bms/xinput-timer-running
      (cancel-timer bms/xinput-timer-running))
    ;; set the input method to allow xcompose to work:
    (setq x-gtk-use-native-input 't)
    ;; must have `xdotool' installed:
    (shell-command "xdotool key Multi_key")
    ;; record the name of the timer to stop it,
    ;; so we don't end up with tonnes of timers somehow
    (setq bms/xinput-timer-running
          (run-at-time 10 nil
          ;; this sets it for 10 secs; maybe could be shorter,
          ;; but you don't want it turning off in the middle of sequences
                       (lambda ()
                         (setq x-gtk-use-native-input 'nil)))))

  ;; And then we bind <Multi_key> to that function.
  ;; (There, now it's not undefined anymore.)

  (define-key global-map (kbd "<Multi_key>") #'bms/xinput-hack))
  ;; A stupid solution to a stupid problem.

See Emacs Using the Lucid toolkit is blazingly fast | r/emacs for why this might be interesting. This is the whole “out of order” thing, but it’s not directly relevant for this particular post. ↩︎
Though, on the plain, boring, ordinary GTK3 toolkit and its relation to Emacs see “Buttery Smooth Emacs”. ↩︎
And I’ve already got (setf (getenv "GTK_IM_MODULE") "xim") and (setf (getenv "QT_IM_MODULE") "xim" setting environment variables in my init.lisp config for StumpWM here anyway. ↩︎
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=278167 ↩︎

Equake: A Geas on Gnomish Smiths

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Wed, 08 Jun 2022 10:54:00 -0600

A new version of Equake, the drop-down “terminal emulator” for Emacs, should be hitting Melpa shortly. This version includes a number of bug fixes, and some new features.

Jeff Kowalski added code for a “close Equake frame on loss of focus feature” (similar to the Tilda feature) and a number of bug fixes and code-cleanup.

Further: I’m (half-)jokingly calling this the Geas on Gnomish Smiths release as I’ve finally figured out how to make it behave properly under GNOME Shell Wayland.

To my knowledge, the only other currently working drop-down terminal for GNOME Shell Wayland is the JavaScript GNOME extension ddterm. Wayland (at least GNOME Shell’s Wayland) seems very restrictive in how non-user-initiated events can affect window properties. E.g., it is difficult to programmatically affect window focus in GNOME Shell under Wayland. The default “hide window” behaviour of Equake doesn’t work.

However, I did have a legacy feature where closing/hiding Equake involved destroying the frame rather than hiding it. This is somewhat complicated, as re-invoking Equake then involves creating a new frame and restoring the tabs and buffer history. This is how Equake originally worked, but it is simpler (and slightly faster) just to hide the frame on “close” and then unhide it on “re-open”, and this became the default behaviour for Equake some time ago. But I left the old code in place as another option.

However, various changes apparently ended up with the restoration behaviour not working quite properly. I have fixed this (more or less, see https://gitlab.com/emacsomancer/equake/-/issues/26). With that fix in place, we can then trick GNOME Shell under Wayland into behaving properly. Rather than hiding the Equake frame on “close”, we destroy it and then recreate it on “open”. This involves quite a bit of trickery behind the scenes. Essentially, opening a new window under GNOME Shell Wayland can result in that window being placed on top of any existing windows (though to do it properly, employing a it of Elisp (select-frame-set-input-focus (selected-frame)) seems to result in the new Emacs client frame being on top (and focussed) [this is a potentially useful trick for opening regular new Emacsclient frames as well].

But, the usual way that Equake frames are created internally is via Emacs make-frame, which doesn’t automatically focus the new frame. So rather than doing this, we need to have the user call emacsclient -c and then transform that frame into an Equake frame. A new Equake function equake--transform-existing-frame-into-equake-frame takes care of this. The transformation and the old code restoring tabs and history doesn’t quite take care of everything though as the point/cursor is left by default at the top of the frame, which is not the user-expected behaviour. So an addition bit of elisp (goto-char (1- (point-max))) generally moves the point/cursor to the end of the buffer where the user might expect it.

So I have a bit of shell script as a helper function which is what one should create a keybinding for in GNOME Shell Wayland:

#!/bin/sh

equakestatus=$(emacsclient -n -e '(frame-live-p (alist-get (equake--get-monitor) equake--frame))')

if [ "$equakestatus" = "nil" ]; then
    emacsclient -c -e "(progn (select-frame-set-input-focus (selected-frame))
                              (equake--transform-existing-frame-into-equake-frame)
                              (goto-char (1- (point-max))))"
else
    emacsclient -n -e '(progn (setq equake-use-frame-hide nil)
                              (equake-invoke))'
fi

This checks if there is a live Equake frame; if there is, it is hidden; otherwise a new frame is created and transformed.

Finally, in order to be able to get the “Always on Top” behaviour, the new Equake frame transmutation function calls (shell-command "wmctrl -r :ACTIVE: -b toggle,above") (so one needs wmctrl installed), which seems to trigger the “Always on Top” feature at least for Xwayland windows.

And, presto, change-o, voilà, we have a version of Equake which exhibits its normal X11 behaviour in GNOME Shell Wayland.

(As long as Emacs is run as an Xwayland application; otherwise the “always on top” behaviour doesn’t work properly.)

Social network platforms (IRC, Matrix, Discord, Slack, and others)

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Wed, 27 Oct 2021 14:22:23 -0600

Following on the last post

Lisp

Common Lisp:
- …

Clojure: slack
- Racket: slack

Automatically adding information to Org-roam file properties

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Tue, 10 Aug 2021 21:12:00 -0600

This expands on a feature I included in the setup for using Org-roam on Android/LineageOS in the last post, specifically automatically adding properties to newly created Org-roam files.

Since Org-roam v2 creates a top properties drawer (with an :ID: tag) anyway, it is nice to stick other information there as well. Specifically, information that could be useful in some situation, but which usually we don’t want to see, like :AUTHOR: (it’s probably you, and you know who you are), :CREATION_TIME: (and why not use Unix epoch time?), and so on. I have org drawers fold themselves automatically, so the normally-useless information doesn’t distract me.

We can do this by leveraging Org-roam’s org-roam-capture-new-node-hook, and some org-roam-add-property function calls, as below.

But, while we’re at it, we might also record where a note was made from. There are a number of ways we might do this, but an easy one (only requiring curl and an active Internet connection) is using ipinfo.io. curl ipinfo.io will give you a bunch of information in JSON format about your internet provider, including latitude and longitude, which will likely be at least somewhere near your present location. And curl ipinfo.io/loc will return just latitude,longitude.

  (defun bms/add-other-auto-props-to-org-roam-properties ()
    ;; if the file already exists, don't do anything, otherwise...
    (unless (file-exists-p (buffer-file-name))
      ;; if there's also a CREATION_TIME property, don't modify it
      (unless (org-find-property "CREATION_TIME")
        ;; otherwise, add a Unix epoch timestamp for CREATION_TIME prop
        ;; (this is what "%s" does - see http://doc.endlessparentheses.com/Fun/format-time-string )
        (org-roam-add-property
         (format-time-string "%s"
                             (nth 5
                                  (file-attributes (buffer-file-name))))
         "CREATION_TIME"))
      ;; similarly for AUTHOR and MAIL properties
      (unless (org-find-property "AUTHOR")
        (org-roam-add-property roam-user "AUTHOR"))
      (unless (org-find-property "MAIL")
        (org-roam-add-property roam-email "MAIL"))
      ;; also add the latitude and longitude
      (unless (org-find-property "LAT_LONG")
        ;; recheck location:
        (bms/get-lat-long-from-ipinfo)
        (org-roam-add-property (concat (number-to-string calendar-latitude) "," (number-to-string calendar-longitude)) "LAT-LONG"))))

  ;; hook to be run whenever an org-roam capture completes
  (add-hook 'org-roam-capture-new-node-hook #'bms/add-other-auto-props-to-org-roam-properties)

;; function to find latitude & longitude
;;                      (requires curl to be installed on system)
(setq calendar-latitude 0)
(setq calendar-longitude 0)
(defun bms/get-lat-long-from-ipinfo ()
  (let*
      ((latlong (substring
                 (shell-command-to-string "curl -s 'ipinfo.io/loc'")
                   0 -1))
       (latlong-list (split-string latlong ",")))
    (setq calendar-latitude (string-to-number (car latlong-list)))
    (setq calendar-longitude (string-to-number (cadr latlong-list)))))

You might also calculate/set calendar-latitude and calendar-longitude in other ways. Including just hard-coding them for stationary machines. On Android, we could in theory make use of the Termux command termux-location, which queries the device’s GPS. But unfortunately it doesn’t always work (if it can’t find a good connection to a GPS satellite) and even when it does work it’s slow, so it’s not something you’d want to call every time you made a note. GeoClue would be another possible source.

(If you’re using a VPN, you’ll want to escape from it somehow to get something closer to your real location. How you do this will vary based on your VPN provider and other factors. (If you’re calling from Emacs, and you use something like Mullvad, you may want to revise the shell-command-to-string to call up a bash session/script, then exclude that specific bash session/script from the VPN, and then call curl, so that the call references your “real” IP. E.g. if you’re using Mullvad, then:

#!/bin/bash
PID=`echo $$`
mullvad split-tunnel pid add "${PID}"
curl ipinfo.io/loc # for lat/long ; `curl ipinfo.io` for full info

might give you a start on something.))

Let me know if you think of other properties that could be useful to automatically add to Org-roam file properties.

Org-roam on Android

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Mon, 09 Aug 2021 20:06:00 -0500

I’ve been using the note-taking Zettelkasten-ish Org-roam system for a few months and it’s been very useful to me, just as a low-friction way of making more notes and easily finding and/or (re)discovering notes that I’ve made.

It’s pretty useful to be able to have access to these notes, and be able to quickly add notes, on mobile as well. I thought it might be useful to include here some notes on how to do, since (especially since v2 of Org-roam) there are some hurdles.

On Android/LineageOS install the F-Droid app store, and then from there install Termux. Open Termux and install four things we’ll need (strictly speaking you don’t need curl and ripgrep, but they’ll be useful): Emacs, sqlite, curl, and ripgrep via pkg install emacs sqlite curl ripgrep.

You can then open up Emacs via emacs and get started.

I include here a commented partial version of my ~/.emacs.d/init.el configuration file for my Termux/Emacs Org-roam setup:

;; BASIC SETUP:

;; package setup - bootstrap the package system
(require 'package)
(setq package-enable-at-startup nil)
(setq gnutls-algorithm-priority "NORMAL:-VERS-TLS1.3")
(setq package-archives
      '(("GNU ELPA"     . "https://elpa.gnu.org/packages/")
        ("ORG"		. "https://orgmode.org/elpa/")
        ("MELPA Stable" . "https://stable.melpa.org/packages/")
        ("MELPA"        . "https://melpa.org/packages/"))
      package-archive-priorities
      '(("ORG"		. 20)
        ("MELPA"        . 15)
        ("MELPA Stable" . 10)
        ("GNU ELPA"     . 5)))

(package-initialize)

;; Bootstrap `use-package'
(unless (package-installed-p 'use-package)
  (package-refresh-contents)
  (package-install 'use-package))

(eval-when-compile
  (require 'use-package))

;; for Termux-specific things; useful if you want to share
;; configs across platforms
(defvar termux-p
  (not (null (getenv "ANDROID_ROOT")))
  "If non-nil, GNU Emacs is running on Termux.")

(when termux-p
  (unless (package-installed-p 'use-package)
    (package-refresh-contents)
    (package-install 'use-package)))

;; This makes Emacs in Termux use your Android browser for opening urls
(setq browse-url-browser-function 'browse-url-xdg-open)

;; mouse
;; enable mouse reporting for terminal emulators
;; this lets you scroll around by swiping
(unless window-system
  (xterm-mouse-mode 1)
  (global-set-key [mouse-4] (lambda ()
                              (interactive)
                              (scroll-down 1)))
  (global-set-key [mouse-5] (lambda ()
                              (interactive)
                              (scroll-up 1))))

;; ORG
(use-package org
  :ensure t
  :ensure org-plus-contrib
  :init
  (setq org-src-fontify-natively t)
  :config
  ;; (add-to-list 'auto-mode-alist '("\\.org\\'" . org-mode))
  (define-key org-mode-map (kbd "M-p") 'org-metaup)
  (define-key org-mode-map (kbd "M-n") 'org-metadown)
  (setq org-catch-invisible-edits 'show-and-error)
  (setq org-cycle-separator-lines -1)
  (setq org-return-follows-link t)
  (setq org-export-with-toc 'nil)
  (setq org-startup-folded 'content)
  (setq org-ellipsis "⇣")
  ;; **** use regular android apps to view pdfs & images *****
  (when termux-p
    (add-to-list 'org-file-apps '("\\.pdf\\'" . "termux-open %s"))
    (add-to-list 'org-file-apps '("\\.png\\'" . "termux-open %s"))
    (add-to-list 'org-file-apps '("\\.jpg\\'" . "termux-open %s"))
    (add-to-list 'org-file-apps '("\\.jpeg\\'" . "termux-open %s")))
  ;; needed for <s etc. expansion of code-blocks
  (require 'org-tempo))

;; define our Org-roam user and their email (set to your desired name/email)
(defvar roam-user "Some User"
  "The name of the Org-roam note author.")
(defvar roam-email "roman@mode.org"
  "The public email of that author.")

(setq org-roam-v2-ack t)

;; we need this package for v2 of Org-oram
(use-package emacsql-sqlite3
  :ensure t)

;; If you've replicated my setup; otherwise change to the Termux
;; local path.
(setq org-roam-directory (file-truename "~/Documents/Org/org-roam/"))

;; org-roam
(use-package org-roam
  :ensure t
  :custom
  (setq org-roam-db-location (file-truename "~"))
  (org-roam-directory (file-truename "~/Documents/Org/org-roam/"))
  :bind (("C-c n l" . org-roam-buffer-toggle)
         ("C-c n f" . org-roam-node-find)
         ("C-c n r" . org-roam-graph)
         ("C-c n i" . org-roam-node-insert)
         ("C-c n c" . org-roam-capture)
         ("C-c n g" . org-id-get-create)
         ;; Dailies
         ("C-c n n" . org-roam-dailies-capture-today)
         ("C-c n d" . org-roam-dailies-goto-today) ; find toDay
         ("C-c n v" . org-roam-dailies-goto-date)
         ("C-c n f" . org-roam-dailies-goto-next-note)
         ("C-c n b" . org-roam-dailies-goto-previous-note))
  :config

  ;; this is a chunglak's hack to get sqlite to work on Android with org-roam v2:
  ;; from: https://github.com/org-roam/org-roam/issues/1605#issuecomment-885997237
  (defun org-roam-db ()
    "Entrypoint to the Org-roam sqlite database.
Initializes and stores the database, and the database connection.
Performs a database upgrade when required."
    (unless (and (org-roam-db--get-connection)
                 (emacsql-live-p (org-roam-db--get-connection)))
      (let ((init-db (not (file-exists-p org-roam-db-location))))
        (make-directory (file-name-directory org-roam-db-location) t)
        (let ((conn (emacsql-sqlite3 org-roam-db-location)))
          (emacsql conn [:pragma (= foreign_keys ON)])
          (set-process-query-on-exit-flag (emacsql-process conn) nil)
          (puthash (expand-file-name org-roam-directory)
                   conn
                   org-roam-db--connection)
          (when init-db
            (org-roam-db--init conn))
          (let* ((version (caar (emacsql conn "PRAGMA user_version")))
                 (version (org-roam-db--upgrade-maybe conn version)))
            (cond
             ((> version org-roam-db-version)
              (emacsql-close conn)
              (user-error
               "The Org-roam database was created with a newer Org-roam version.  "
               "You need to update the Org-roam package"))
             ((< version org-roam-db-version)
              (emacsql-close conn)
              (error "BUG: The Org-roam database scheme changed %s"
                     "and there is no upgrade path")))))))
    (org-roam-db--get-connection))
  (defun org-roam-db--init (db)
    "Initialize database DB with the correct schema and user version."
    (emacsql-with-transaction db
      (emacsql db "PRAGMA foreign_keys = ON") ;; added
      (emacsql db [:pragma (= foreign_keys ON)])
      (pcase-dolist (`(,table ,schema) org-roam-db--table-schemata)
        (emacsql db [:create-table $i1 $S2] table schema))
      (pcase-dolist (`(,index-name ,table ,columns) org-roam-db--table-indices)
        (emacsql db [:create-index $i1 :on $i2 $S3] index-name table columns))
      (emacsql db (format "PRAGMA user_version = %s" org-roam-db-version))))
  ;; end chunglak hack

  (org-roam-setup)
  ;; If using org-roam-protocol
  (require 'org-roam-protocol))

;; These are my capture templates:
(setq org-roam-capture-templates
      `(("d" "default" plain "%?" :if-new
         (file+head "%<%Y%m%d%H%M%S>-${slug}.org"
                    ,(concat "#+title: ${title}\n"
                             "#+date: %U\n\n"))
         :unnarrowed t)))

(setq org-roam-dailies-directory "~/Documents/Org/org-roam/daily")

(setq org-roam-dailies-capture-templates
      `(("d" "default" entry "* %?" :if-new
         (file+head "%(concat org-roam-dailies-directory \"/%<%Y-%m-%d>.org\")"
                    ,(concat "#+title: %<%Y-%m-%d>" "\n"
                             "#+filetags: :daily_journal:\n\n")))))

;; deft - one way to search Org-roam notes, but not the fastest (see below)
(use-package deft
  :ensure t
  :config
  :after org
  :bind
  ("C-c r d" . deft)
  :custom
  (deft-recursive t)
  (deft-use-filter-string-for-filename t)
  (deft-default-extension "org")
  (deft-directory "~/Documents/Org/org-roam/"))

;; Here end the basic setup, but....

;; SOME OTHER THINGS YOU MIGHT ADD

;; bars seems pointless here, but if you like, don't do this
(menu-bar-mode -1)
(tool-bar-mode -1)

;; You could use a different theme
(use-package cyberpunk-theme
  :ensure t
  :config
  (load-theme 'cyberpunk))

;;;;;;;;;;;;;;;;;;;;;
;; Spell-checking ;;;
;;;;;;;;;;;;;;;;;;;;;
(require 'flymake)
(setq ispell-program-name "hunspell") ; could be ispell as well, depending on your preferences
(setq ispell-dictionary "en_GB") ; this can obviously be set to any language your spell-checking program supports
;; I installed the en_GB ones, but these don't come in Termux by default. To add arbitrary hunspell languages, see:
;; https://www.reddit.com/r/termux/comments/k5o6mp/new_hunspell_dictionaries/?
;; in summary:
;; - how to add new: copy .aff and .dic files in /data/data/com.termux/files/usr/share/hunspell/
;; - where to get new: https://www.freeoffice.com/en/download/dictionaries

(dolist (hook '(org-mode-hook))
  (add-hook hook (lambda () (flyspell-mode 1))))
(add-hook 'org-mode-hook (lambda () (setq ispell-parser 'tex))) ; make orgmode recognise LaTeX syntax [from http://stackoverflow.com/questions/11646880/flyspell-in-org-mode-recognize-latex-syntax-like-auctex ]
(add-hook 'text-mode-hook #'flyspell-mode)

;;;;;;;;;;;;;;;
;; Undo-Tree ;; - a new undo package
;;;;;;;;;;;;;;;
(use-package undo-tree
  :ensure t
  :config
  ;; (setq undo-tree-auto-save-history 1)
  ;; Each node in the undo tree should have a timestamp.
  (setq undo-tree-visualizer-timestamps t)
  ;; Show a diff window displaying changes between undo nodes.
  (setq undo-tree-visualizer-diff t)
  (global-undo-tree-mode))

;; display time and date in modeline, if you like
(setq display-time-day-and-date t)
(display-time-mode 1)

;; prettier bullets
(use-package org-bullets
  :ensure t
  :config
  (add-hook 'org-mode-hook (lambda () (org-bullets-mode 1)))
  (setq org-bullets-bullet-list '("⋇" "∴" "∵" "∷" "∺")))

;; A nice way of quickly adding links.
;; (Though in Termux, you first must paste from your
;; Android clipboard and then copy/kill via Emacs before
;; it'll work.)
(use-package org-cliplink
  :ensure t
  :config
  (define-key org-mode-map (kbd "C-c o c") #'org-cliplink))

;; This is also not needed, but adds some (dubiously) useful properties
;; to the Org-roam file's property drawer.

;; First, set up a system for getting location
;; (we could also try to leverage termux's built-in
;; GPS location abilities via `termux-location`, but
;; it seems a bit slow and doesn't even always work if
;; your device can't get a good satellite connection.)
(setq calendar-latitude 0)
(setq calendar-longitude 0)
(defun bms/get-lat-long-from-ipinfo ()
  (let*
      ((latlong (substring
                 (shell-command-to-string "curl -s 'ipinfo.io/loc'") 0 -1))
       (latlong-list (split-string latlong ",")))
    (setq calendar-latitude (string-to-number (car latlong-list)))
    (setq calendar-longitude (string-to-number (cadr latlong-list)))))

(defun bms/add-other-auto-props-to-org-roam-properties ()
  (unless (file-exists-p (buffer-file-name))
    (unless (org-find-property "CREATION_TIME")
      (org-roam-add-property (format-time-string "%s"
                                                 (nth 5
                                                      (file-attributes (buffer-file-name))))
                             "CREATION_TIME"))
    (unless (org-find-property "AUTHOR")
      (org-roam-add-property roam-user "AUTHOR"))
    (unless (org-find-property "MAIL")
      (org-roam-add-property roam-email "MAIL"))
    (unless (org-find-property "LAT_LONG")
      (bms/get-lat-long-from-ipinfo)
      (org-roam-add-property (concat (number-to-string calendar-latitude) "," (number-to-string calendar-longitude)) "LAT-LONG"))))

(add-hook 'org-roam-capture-new-node-hook #'bms/add-other-auto-props-to-org-roam-properties)

;; You could use Ivy or Helm or the default, but I
;; like Selectrum, Consult & friends. Plus we can leverage
;; Consult for a nice alternative to deft for note-searching.
;; You'll need this to use my ripgrep note searching feature below.

;; selectrum
(use-package selectrum
  :ensure t
  :config
  (selectrum-mode +1))

;; ;; prescient  - T9
(use-package prescient
  :ensure t
  :config
  (setq prescient-persist-mode t)
  (setq prescient-filter-method '(literal regexp initialism fuzzy))) ;; added fuzzy

(use-package orderless
  :ensure t
  :init (icomplete-mode)                ; optional but recommended!
  :custom (completion-styles '(orderless))
  :config
  (setq orderless-matching-styles '(orderless-flex))
  ;; This means that the company-capf backend will automatically use orderless, but following issue exists:
  ;; Pressing SPC takes you out of completion, so with the default separator you are limited to one component,
  ;; which is no fun. To fix this add a separator that is allowed to occur in identifiers, for example, for
  ;; Emacs Lisp code you could use an ampersand:
  (setq orderless-component-separator "[ &]")

  ;; The matching portions of candidates aren’t highlighted. But while you can’t get different faces for
  ;; different components, you can at least get the matches highlighted in the sole available face with this configuration
  (defun just-one-face (fn &rest args)
    (let ((orderless-match-faces [completions-common-part]))
      (apply fn args)))

  (advice-add 'company-capf--candidates :around #'just-one-face))

(use-package selectrum-prescient
  :ensure t
  :config
  ;; to make sorting and filtering more intelligent
  (selectrum-prescient-mode +1)

  ;; Filtering with orderless
  (setq selectrum-refine-candidates-function #'orderless-filter)
  (setq selectrum-highlight-candidates-function #'orderless-highlight-matches)

  ;; If you also configure `completion-styles` for orderless you might want to use the
  ;; following advice because orderless isn't well suited for initial gathering of
  ;; candidates by completion in region.
  (advice-add #'completion--category-override :filter-return
              (defun completion-in-region-style-setup+ (res)
                "Fallback to default styles for region completions with orderless."
                (or res
                    ;; Don't use orderless for initial candidate gathering.
                    (and completion-in-region-mode-predicate
                         (not (minibufferp))
                         (equal '(orderless) completion-styles)
                         '(basic partial-completion emacs22)))))

  ;; Minibuffer-actions with embark

  ;; You should bind embark commands like embark-act, embark-act-noexit
  ;; and embark-export in minibuffer-local-map (as embark commands are not selectrum specific).
  ;; For available commands and other embark configurations see the embark documentation and its wiki.

  (defun current-candidate+category ()
    (when selectrum-is-active
      (cons (selectrum--get-meta 'category)
            (selectrum-get-current-candidate))))

  (add-hook 'embark-target-finders #'current-candidate+category)

  (defun current-candidates+category ()
    (when selectrum-is-active
      (cons (selectrum--get-meta 'category)
            (selectrum-get-current-candidates
             ;; Pass relative file names for dired.
             minibuffer-completing-file-name))))

  (add-hook 'embark-candidate-collectors #'current-candidates+category)
  ;; No unnecessary computation delay after injection.
  (add-hook 'embark-setup-hook 'selectrum-set-selected-candidate)

  ;; The following is not selectrum specific but included here for convenience.
  ;; If you don't want to use which-key as a key prompter skip the following code.

  (setq embark-action-indicator
        (lambda (map) (which-key--show-keymap "Embark" map nil nil 'no-paging)
          #'which-key--hide-popup-ignore-command)
        embark-become-indicator embark-action-indicator)

  ;; to save your command history on disk, so the sorting gets more
  ;; intelligent over time
  (prescient-persist-mode +1))

;; Example configuration for Consult
(use-package consult
  ;; Replace bindings. Lazily loaded due by `use-package'.
  :bind (("C-x M-:" . consult-complex-command)
         ("C-c h" . consult-history)
         ("C-c m" . consult-mode-command)
         ("C-x b" . consult-buffer)
         ("C-x 4 b" . consult-buffer-other-window)
         ("C-x 5 b" . consult-buffer-other-frame)
         ("C-x r x" . consult-register)
         ("C-x r b" . consult-bookmark)
         ("M-g g" . consult-goto-line)
         ("M-g M-g" . consult-goto-line)
         ("M-g o" . consult-outline)       ;; "M-s o" is a good alternative.
         ("M-g l" . consult-line)          ;; "M-s l" is a good alternative.
         ("M-g m" . consult-mark)          ;; I recommend to bind Consult navigation
         ("M-g k" . consult-global-mark)   ;; commands under the "M-g" prefix.
         ("M-g r" . consult-ripgrep)      ;; or consult-grep, consult-ripgrep
         ("M-g f" . consult-find)          ;; or consult-locate, my-fdfind
         ("M-g i" . consult-project-imenu) ;; or consult-imenu
         ("M-g e" . consult-error)
         ("M-s m" . consult-multi-occur)
         ("M-y" . consult-yank-pop)
         ("<help> a" . consult-apropos))

  ;; The :init configuration is always executed (Not lazy!)
  :init

  ;; Custom command wrappers. It is generally encouraged to write your own
  ;; commands based on the Consult commands. Some commands have arguments which
  ;; allow tweaking. Furthermore global configuration variables can be set
  ;; locally in a let-binding.
  (defun my-fdfind (&optional dir)
    (interactive "P")
    (let ((consult-find-command '("fdfind" "--color=never" "--full-path")))
      (consult-find dir)))

  ;; Replace `multi-occur' with `consult-multi-occur', which is a drop-in replacement.
  (fset 'multi-occur #'consult-multi-occur)

  ;; Configure other variables and modes in the :config section, after lazily loading the package
  :config

  ;; Configure preview. Note that the preview-key can also be configured on a
  ;; per-command basis via `consult-config'.
  ;; (setq consult-preview-key 'any) ;; any key triggers preview, the default

  ;; Optionally configure narrowing key.
  ;; Both < and C-+ work reasonably well.
  (setq consult-narrow-key "<") ;; (kbd "C-+")
  ;; Optionally make narrowing help available in the minibuffer.
  ;; Probably not needed if you are using which-key.
  ;; (define-key consult-narrow-map (vconcat consult-narrow-key "?") #'consult-narrow-help)

  ;; Optional configure a view library to be used by `consult-buffer'.
  ;; The view library must provide two functions, one to open the view by name,
  ;; and one function which must return a list of views as strings.
  ;; Example: https://github.com/minad/bookmark-view/
  ;; (setq consult-view-open-function #'bookmark-jump
  ;;       consult-view-list-function #'bookmark-view-names)

  ;; Optionally configure a function which returns the project root directory
  ;; (autoload 'projectile-project-root "projectile")
  ;; (setq consult-project-root-function #'projectile-project-root)
  )

;; Optionally add the `consult-flycheck' command.
(use-package consult-flycheck
  :bind (:map flycheck-command-map
              ("!" . consult-flycheck)))

;; Optionally enable richer annotations using the Marginalia package
(use-package marginalia
  :ensure t
  ;; The :init configuration is always executed (Not lazy!)
  :init
  ;; Must be in the :init section of use-package such that the mode gets
  ;; enabled right away. Note that this forces loading the package.
  (marginalia-mode))

(use-package embark
  :ensure t
  :bind
  ("C-S-a" . embark-act)               ; pick some comfortable binding
  :config
  ;; For Selectrum users:
  (defun current-candidate+category ()
    (when selectrum-is-active
      (cons (selectrum--get-meta 'category)
            (selectrum-get-current-candidate))))

  (add-hook 'embark-target-finders #'current-candidate+category)

  (defun current-candidates+category ()
    (when selectrum-is-active
      (cons (selectrum--get-meta 'category)
            (selectrum-get-current-candidates
             ;; Pass relative file names for dired.
             minibuffer-completing-file-name))))

  (add-hook 'embark-candidate-collectors #'current-candidates+category)

  ;; No unnecessary computation delay after injection.
  (add-hook 'embark-setup-hook 'selectrum-set-selected-candidate))

;; org-roam-rg-search - this is a much faster way to search Org-roam notes:
;; requires the Selectrum+Consult setup immediately preceding.
;; Use C-c r r to search notes via consult's ripgrep interface
(defun bms/org-roam-rg-search ()
  "Search org-roam directory using consult-ripgrep. With live-preview."
  (interactive)
  (let ((consult-ripgrep "rg --null --ignore-case --type org --line-buffered --color=always --max-columns=500 --no-heading --line-number . -e ARG OPTS"))
    (consult-ripgrep org-roam-directory)))

(global-set-key (kbd "C-c rr") 'bms/org-roam-rg-search)

;; speed-keys - see https://github.com/alhassy/emacs.d#manipulating-sections
(setq org-use-speed-commands t)

;; On an org-heading, C-a goes to after the star, heading markers. To use speed keys, run C-a C-a to get to the star markers.
;; C-e goes to the end of the heading, not including the tags.
(setq org-special-ctrl-a/e t)

;;drag images into orgmode
(use-package org-download
  :ensure t
  :config
  (add-hook 'dired-mode-hook 'org-download-enable)
  (global-set-key (kbd "C-c o i") #'org-download-yank)
  (setq org-download-method 'attach))

(defun bms/org-attach-insert-link (&optional in-emacs)
  "Insert attachment from list."
  (interactive "P")
  (let ((attach-dir (org-attach-dir)))
    (if attach-dir
        (let* ((file (pcase (org-attach-file-list attach-dir)
                       (`(,file) file)
                       (files (completing-read "Insert attachment: "
                                               (mapcar #'list files) nil t))))
               (path (expand-file-name file attach-dir))
               (desc (file-name-nondirectory path)))
          (let ((initial-input
                 (cond
                  ((not org-link-make-description-function) desc)
                  (t (condition-case nil
                         (funcall org-link-make-description-function link desc)
                       (error
                        (message "Can't get link description from %S"
                                 (symbol-name org-link-make-description-function))
                        (sit-for 2)
                        nil))))))
            (setq desc (if (called-interactively-p 'any)
                           (read-string "Description: " initial-input)
                         initial-input))
            (org-insert-link nil path (concat "attachment:" desc))))
      (error "No attachment directory exist"))))
(define-key org-mode-map (kbd "C-c o l") #'bms/org-attach-insert-link)

;; in case you want some things not in melpa
;; you'll need it for the remaining things below
(use-package quelpa
  :ensure t)

(use-package quelpa-use-package
  :ensure t)

;; A bit of sugar for the visual appearance of Org syntax
;; Use if you like.
(use-package org-appear
  :ensure t
  :quelpa (org-appear :fetcher github :repo "awth13/org-appear")
  :config
  (setq org-hide-emphasis-markers t)
  (add-hook 'org-mode-hook 'org-appear-mode))

;; You don't need this, but it's cool and it does work on Android:
;; see https://github.com/org-roam/org-roam-ui for features
(use-package org-roam-ui
  :ensure t
  :quelpa (org-roam-ui :fetcher github :repo "org-roam/org-roam-ui" :branch "main" :files ("*.el" "out"))
  :after org-roam
  :hook (org-roam . org-roam-ui-mode))

An excellent way of keeping Org notes (and files more generally) in sync between desktop, laptop, and mobile devices is Syncthing. On Android I recommend using the Syncthing-Fork app (via F-Droid), which has various improvements over the default Syncthing app on Android, including better file-access features. (On iOS there is now a third-party solution for syncing via Syncthing: Möbius-Sync. I have no idea how to use Emacs/Org-mode on iOS though, but I recall hearing about some ways of running a Linux shell on iOS like iSH, so possibly there’s some way.)

I have Syncthing sync my Org files to a directory in my main “home” directory on Android Documents/Org and then in Termux created a Documents directory and inside of that directory created a symlink to my actual Org directory via ln -s storage/shared/Documents/Org Org. I’ve found that is easier for allowing Syncthing to have access to the files in order to keep them in sync. (And having my Org files live at ~/Documents/Org in Termux mimics the directory structure on my Linux boxes, which makes lots of things easier in terms of sharing configurations.)

Dynamically set pdf-tools annotation colours

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Thu, 27 May 2021 12:25:03 -0600

In Emacs, pdf-tools can be used to add annotations to a PDF document. It can be useful to have multiple annotation colours though, and be able to set these on the fly.

Here’s an example of how to do it with four colours:

;; annotation colours
(defun bms/pdf-annot-colour-blue ()
  (interactive)
  (setq pdf-annot-default-markup-annotation-properties
        '((label . "") (color . "blue") (popup-is-open)))
  (message "%s" (propertize "Annotation colour set to blue." 'face '(:foreground "blue"))))

(defun bms/pdf-annot-colour-yellow ()
  (interactive)
  (setq pdf-annot-default-markup-annotation-properties
        '((label . "") (color . "yellow") (popup-is-open)))
    (message "%s" (propertize "Annotation colour set to yellow." 'face '(:foreground "yellow"))))

(defun bms/pdf-annot-colour-red ()
  (interactive)
  (setq pdf-annot-default-markup-annotation-properties
        '((label . "") (color . "red") (popup-is-open)))
  (message "%s" (propertize "Annotation colour set to red." 'face '(:foreground "red"))))

(defun bms/pdf-annot-colour-orange ()
  (interactive)
  (setq pdf-annot-default-markup-annotation-properties
        '((label . "") (color . "orange") (popup-is-open)))
  (message "%s" (propertize "Annotation colour set to orange." 'face '(:foreground "orange"))))

;; rebind keys for pdf-tools
(defun bms/pdf-tools-mode-config ()
  "Set pdf-tools keybindings."
  (local-set-key (kbd "R") #'bms/pdf-annot-colour-red)
  (local-set-key (kbd "L") #'bms/pdf-annot-colour-blue)
  (local-set-key (kbd "O") #'bms/pdf-annot-colour-orange)
  (local-set-key (kbd "Y") #'bms/pdf-annot-colour-yellow))

;; add to pdf-view-mode-hook
(add-hook 'pdf-view-mode-hook #'bms/pdf-tools-mode-config)

Example in use (though some of the colours might be better chosen, e.g. a lighter shade of blue):

Twiddler config for Emacs

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Sat, 30 May 2020 22:00:00 -0600

The Twiddler [here’s archive.org’s link, as the site seems to be down as I write this], a one-handed chording keyboard, has a longish history of being associated with Emacs. Here’s 1990s Alan Alda interviewing Thad Starner, who’s using a wearable-computing device foreshadowing Google Glass, using a Twiddler mk 1 to interact with Emacs (using the Remembrance Agent):

I’ve long been intrigued by this one-hand, non-tethered input method and finally got a Twiddler 3. I was unfortunately stymied on progress early on due to a loose battery (requiring cracking the entire device open), but have got it back working and am slowly trying to integrate it into my workflow.

One place where it would be particularly useful is with a mobile device (e.g. smart phone), especially as one can run full Emacs in Termux on Android.

The Twiddler is a configurable device, and obviously using it with Emacs calls for some Emacs-specific configuration. Here is my current working configuration:

M-x tab-space: Emacs-centric layout for the Twiddler 3

With important Emacs combinations like C-x, M-x, C-g mapped to chords (along with a chord prefix, s-F for my personal StumpWM config), and some chords set up for comfortable navigating with God Mode and avy. And lots of space for expansion.

Beautiful and Free Unicode Typefaces, for editor and printer (including a comparison of Latin Modern and Computer Modern Unicode)

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Sun, 04 Aug 2019 02:25:00 -0600

For my academic papers, I often need a typeface with a wide range of characters and diacritic combinations. Basic diacritics are supported by a wide range of fonts, but more specialised diacritics and particularly combinations of diacritics only work well in a handful of typefaces. I write my papers in TeX, which has two components: the typeface used to set the paper in (La)TeX and the typeface/font used inside Emacs, where I write the papers.

Typeset typefaces

The choice of typeface is particularly relevant for using XeLaTeX or LuaLaTeX, where the standard default unicode typeface family is Latin Modern, based on Donald Knuth‘s original Computer Modern family of typefaces designed in METAFONT, and the general recommendation is to use Latin Modern rather than the earlier Computer Modern Unicode.

Generally reasons given for preferring Latin Modern include the fact that Latin Modern involves handmade vectorisation, with revised metrics, and additional glyph coverage, including particularly diacritic characters; whereas the Computer Modern Unicode fonts were converted from METAFONT sources using mftrace with autotrace backend and fontforge (former pfaedit) automatically.

Some of the revised metrics of Latin Modern include arguably better positioning of acute and grave accents, more appropriately-sized umlauts, and (here tastes may vary) a better looking Eszett (ß).

[Note that throughout this post, you can right-click on the images and select “View Image” to see a larger/zoomable version of the image.]

However, in my tests, Computer Modern Unicode actually performs better at least with respect to handling diacritics. In fact, Computer Modern Unicode is one of a handful of typefaces which actually properly handle a large range of diacritics and combinations of diacritics, as can be seen in the image below, showing a variety of serif, sans serif, and monospace typefaces, respectively, typeset in XeLaTeX:

[The source file from which all of these samples were generated is included below in this footnote: ¹.]

Note that in many typefaces, including all of the Latin Modern faces, combinations of acute accents and macrons (e.g. the first character in the tests: ā́, also not handled well by IM Fell Great Primer, the font this blog used to use for its body text; now switched to Computer Modern Unicode Sans) are very poorly handled, often with the accent and macron being overlaid, rather than the accent properly stacking on top of the macron.

Even fonts that properly handle this often fail for other diacritics: note that EB Garamond and Liberation Serif don’t seem to properly handle underrings (e.g. R̥r̥, also not handled that well by IM Fell Great Primer).

Even the Linux Libertine faces (here represented by their Libertinus continuations), which overall fare well, don’t handle the acute+macron combination well in the italic version ā́.

Thus the only serif faces which handle the full range of characters/diacritics are the didone (a style arising in the late 18th-century, popular in the 19th-c.) Computer Modern Unicode; Junicode (based on a 17th-century font used for George Hickes‘s Linguarum Vett. Septentrionalium Thesaurus); and the early 20th-century styled Times New Romanesque Noto Serif.

For the sans serif faces, in addition to the Computer Modern Unicode Sans Serif face (in some ways a ‘skeletal’, un-didone-ised version of Computer Modern Roman), Mozilla’s Fira Sans (here represented by FiraGO, an updated/extended version), and Noto Sans – the latter two both being vaguely ‘humanist’ sans serifs, though Noto Sans perhaps has some ‘american gothic/grotesque’ features – also manage the full range of characters/diacritics, but doesn’t have an italic face.

For the monospace faces, again the Computer Modern Typewriter face (an ‘Egyptian’ typewriter face, not unlike Courier) manages the full range of characters/diacritics, as do Noto Sans Mono and the Iosevka faces (here represented by my own customised version, Iosevka Oak Vise).

Not all of these typefaces are of equal beauty. Here are some examples of running text, single sentences first:

And full paragraphs (for a subset of the typefaces; only including those that manage at least a majority of the characters and diacritic combinations):²

Only a few typefaces handle the full range of characters and diacritic combinations. For the serif faces:

CMU Serif
Noto Serif
Junicode

For the sans faces:

CMU Sans Serif
Noto Sans
Fira Sans (FiraGO)

For the monospaced faces:

CMU Typewriter
Noto Sans Mono
Iosevka

Here is a comparison only of these “winners” (and Noto Sans Mono perhaps is not a winner, lacking an italic face):

So, if you want to use a font family with good performance across the full range of fonts, Computer Modern Unicode (CMU) or Noto seem like the best bets, and the former is much more aesthetically-pleasing than the latter (in my opinion), which also lacks an italic font for its mono version. Junicode is also a beautiful and very functional serif face, which I often use.

The Latin Modern faces turn out, despite the received wisdom, to be inferior in terms of diacritic coverage, and switching from Latin Modern to Computer Modern Unicode (CMU) has significantly reduced the amount of frustration I have in trying to typeset my papers.

Editor typefaces

The editor font is the one I spend most of my time staring at, so I also want something good here too.

Despite DejaVu Sans Mono apparently not properly rendering all character/diacritic combinations in LaTeX, it handles all of these perfectly well when used as the default font in Emacs:

DejaVu Sans Mono was for many years my preferred font for Emacs (and I assume DejaVu Sans Mono-derived fonts like Menlo or Hack will also behave well).

Noto Sans Mono also works fine, but it’s not my favourite:

CMU Typewriter, despite rendering well in LaTeX, has problems with certain combination when used as the Emacs font, and anyway doesn’t look as good as the other choices as an editor font for whatever reason:³

These days, I prefer to use a customised version of Iosevka, which also handles all of the characters/diacritic combinations perfectly:

I plan to write more extensively about Iosevka in another post.

Summary: Best Bets

For typesetting papers requiring a full-range of unicode roman characters and diacritics (for non-roman, the Noto fonts are great), the Computer Modern Unicode (CMU) faces⁴ are the best bets, along with Junicode.

For editor work requiring a full-range of unicode roman characters and diacritics, use a DejaVu Sans Mono font or derivative, or one of the Iosevka variants.

Source file. ↩︎
I’m not sure why Latin Modern Typewriter ends up with a ragged right margin. ↩︎
Ubuntu Mono, when I tried setting it as the default font in Emacs and opening the .tex file from which the pdf screenshots shown here were taken, caused Emacs to crash! ↩︎
E.g., CMU Serif, CMU Sans Serif, CMU Typewriter; there are a number of others, including also CMU Concrete Roman, used below for the non-title text (the title text is set in Latin Modern Dunhill): ↩︎

Top 50 IF list 2019

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Tue, 30 Jul 2019 11:32:00 -0600

Emily Short wrote a blog post a fortnight ago or so discussing her nominees for Victor Gijsbers’ Top 50 Interactive Fiction Games of All Time list, 2019 edition. The contest closes on the 31st of July 2019 (i.e. in 2 days, as of the day I write this), and I was thinking about what games would be on my list. This has also resulted, perhaps more importantly, with me having a list of games I still need to play.

Both lists below have the games hyperlinked to their Interactive Fiction Database page, which generally includes reviews and game files to download (or links/suggestions of where to get the game).

First, my personal list of favourites/best text adventures/interactive fiction that I’ve completed or at least played most (er, at least half?) of, ordered roughly in the order I encountered them:

Game	Year	Author	Publisher
Zork I	1980	Marc Blank & Dave Lebling	Infocom
Zork II	1981	Dave Lebling & Marc Blank	Infocom
Enchanter	1983	Marc Blank & Dave Lebling	Infocom
Planetfall	1983	Steve Meretzky	Infocom
The Hitchhiker’s Guide to the Galaxy	1984	Douglas Adams & Steve Meretzky	Infocom
Wishbringer	1985	Brian Moriarty	Infocom
The Pawn	1985	Rob Steggles et al.	Magnetic Scrolls
Spellbreaker	1985	Dave Lebling	Infocom
The Guild of Thieves	1987	Rob Steggles	Magnetic Scrolls
Knight Orc	1987	Pete Austin	Level 9
Dunnet	1982	Ron Schnell
Curses!	1993	Graham Nelson
Spider & Web	1998	Andrew Plotkin
Counterfeit Monkey	2012	Emily Short
Coloratura	2013	Lynnea Glasser
Mentula Mancanus: Apocolocyntosis	2011	Adam Thornton

Some of these may be being boosted by nostalgia, but I think they all pretty much hold up and are interesting/innovative in some fashion.

Looking over the author names, I remember when playing Infocom games back in the 80s suspecting that “Marc Blank” was a pseudonym….i.e. Last Name: [blank].

Dunnet is an interesting entry also in that it was originally written in Maclisp for the DECSYSTEM-20, then ported to Emacs Lisp in 1992, and it is the Emacs version (M-x dunnet) that I’m familiar with.

A list of games I still need to play (never played before) or finish (haven’t made significant progress, though I spent a pretty long time with the Silicon Dreams games, Anchorhead, and Varicella, but don’t think I’ve made much progress (or I’ve forgotten what progress I made)), roughly in order of precedence (i.e. how soon I plan to play):

Game	Year	Author	Publisher
Anchorhead	1998	Michael Gentry
Cragne Manor	2018	[numerous]
Galatea	2000	Emily Short
Savoir-Faire	2002	Emily Short
Hadean Lands	2014	Andrew Plotkin
Varicella	1999	Adam Cadre
Endless, Nameless	2012	Adam Cadre
Bronze	2006	Emily Short
Balances	1994	Graham Nelson
Jigsaw	1995	Graham Nelson
Blue Lacuna	2008	Aaron Reed
A Beauty Cold and Austere	2017	Mike Spivey
Lime Ergot	2014	Caleb Wilson (as Rust Blight)
Suveh Nux	2007	David Fisher
Delusions	1996	C. E. Forman
Slouching Towards Bedlam	2003	Star Foster & Daniel Ravipinto
A Mind Forever Voyaging	1985	Steve Meretzky	Infocom
Trinity	1986	Brian Moriarty	Infocom
Snowball [Silicon Dreams]	1983	Mike Austin et al.	Level 9
Return to Eden [Silicon Dreams]	1984	Nick Austin et al.	Level 9
The Worm in Paradise [Silicon Dreams]	1985	Mike Austin et al.	Level 9
The Shadow in the Cathedral	2009	Ian Finley & John Ingold
Make It Good	2009	Jon Ingold
1893: A World’s Fair Mystery	2002	Peter Nepstad
Red Moon	1985	David Williamson et al.	Level 9
Shade	2000	Andrew Plotkin
Stationfall	1987	Steve Meretzky	Infocom
Rameses	2000	Stephen Bond
The Moonlit Tower	2002	Yoon Ha Lee
Bureaucracy	1987	Douglas Adams	Infocom

All of the games listed here, in either list, are playable in either Malyon or Gargoyle (with the exception, I think, of Hadean Lands), an admittedly rather arbitrary criterion.

Auto-generate “creator” PDF metadata in AUCTeX using yasnippet

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Wed, 29 May 2019 22:59:00 -0600

After struggling with some poorly-handled, apparently “reset” proofs introducing heaps of errors (despite my providing a .tex source) for the past few days,¹ I thought about providing automated pdf-tags indicating creation tools used for my TeX-produced documents. Real, professionally-typeset documents deserve to have the tools used to produce them properly recognised in their metadata. So here’s a yasnippet which generates auto-populated hyperref options to generate a pdf-creator tag indicating the version of Emacs, AUCTeX, and distro used:

# -*- mode: snippet -*-
# name: version-hyperref
# key: hyperv
# --
\usepackage[pdfusetitle,
 pdfcreator={`(replace-regexp-in-string "_" "-"
  (concat (replace-regexp-in-string "\n" "" (substring (emacs-version) 0 (1- (cl-search "(" (emacs-version))) ))
     " with AUCTeX " (pkg-info-version-info 'auctex)
   (if (shell-command-to-string "type -p lsb_release > /dev/null")
      (concat
        " on "
      (substring (shell-command-to-string "lsb_release -sd") 1 (- (length (shell-command-to-string "lsb_release -sd")) 2))
        " ("
      (substring (shell-command-to-string "lsb_release -sr") 0 (- (length (shell-command-to-string "lsb_release -sr")) 1))
        " '"
      (substring (shell-command-to-string "lsb_release -sc") 0 (- (length (shell-command-to-string "lsb_release -sc")) 1))
         "'" " release, using the "
      (replace-regexp-in-string "\n$" "" (shell-command-to-string "uname -r")) " kernel)")
     (if (shell-command-to-string "type -p guix > /dev/null")
       (concat
         "on Guix System "
       (shell-command-to-string "guix system -V | awk 'NR==1{printf $5}'")
         " (using the "
       (replace-regexp-in-string "\n$" "" (shell-command-to-string "uname -r")) " kernel)")))))`}]{hyperref}

Snippets can execute elisp code placed between `...`. The version of Emacs (with some of the additional less-relevant information removed via substring combined with cl-search (for “(", the beginning of the additional information)), and AUCTeX can easily be done using Emacs-internal functions, (emacs-version) and (pkg-info-version-info 'auctex), respectively.

For the operating-system information, we rely on external calls to the shell via Emacs’s shell-command-to-string function. Many distros will have lsb_release application (from the Linux Standard Base project) available (and presumably usually part of the base-install). Running lsb_release --help reveals a helpful set of options:

SYNOPSIS
      lsb_release [OPTION]...
OPTIONS
      −v, −−version
      Display the version of the LSB specification against which the distribution is compliant.

      −i, −−id
      Display the string id of the distributor.

      −d, −−description
      Display the single line text description of the distribution.

      −r, −−release
      Display the release number of the distribution.

      −c, −−codename
      Display the codename according to the distribution release.

      −a, −−all
      Display all of the above information.

      −s, −−short
      Display all of the above information in short output format.

      −h, −−help
      Display this message.
LSB Version:	1.0

And we can get various bits of information about the distribution out via passing different flags to lsb_release (combined with the -s “short description” flag).

Some distros don’t use the lsb_release, and looking at various methods neofetch employs to manage to suss out the distro was instructive. I added a method to allow for detecting the version of Guix System distribution, and it could be easily extended (in neofetch-style) to employ other methods for additional distros. (But the lsb_release method should handle the majority of distros.)

Finally, we get can the kernel version out via the ubiquitous uname application, via uname -r.

Thus calling this snippet on Void Linux results in a TeX block like this:

\usepackage[pdfusetitle,
pdfcreator={GNU Emacs 26.2 with AUCTeX 12.1.2 on Void Linux
(rolling 'void' release, using the 5.1.5-1 kernel)}]{hyperref}

(The hyperref option pdfusetitle will make use of the values of the TeX-defined \author{...} and \title={...} commands to populate the author and title pdf metadata. If you do anything funky with these fields (like adding a \thanks{...} command to your title), you may need to take out the pdfusetitle option and manually specify these tags via pdfauthor={...} etc.)

On Guix System (née GuixSD) the result will be something like:

\usepackage[pdfusetitle,
pdfcreator={GNU Emacs 26.2 with AUCTeX 12.1.2
on Guix System 1.0.1-1.8204295 (using the 5.1.2-gnu kernel)}]{hyperref}

An amusing side-discovery of putting this snippet together is that Arch Linux multiply – and insistently – reminds the uname caller of the distro the kernel belongs to. I.e., using this snippet on my Arch machine resulted in:

 \usepackage[pdfusetitle,
pdfcreator={GNU Emacs 26.2 with AUCTeX 12.1.2 on
Arch Linux (rolling 'n/a' release, using the 5.1.5-arch1-2-ARCH kernel)}]{hyperref}

The horrifying reality of “professional” desktop publishing software stands in contrast to real professional documents produced in (La)TeX; and any resulting professionalism in the output of the former results from squeezing additional unpaid labour from researchers who have to correct the errors of “professional” typesetters. ↩︎

Equake(!) Quake-style overlay console in StumpWM

Benjamin.Slade@fakeEmailToMakeValidatorHappy.com (Benjamin Slade) — Wed, 22 May 2019 22:26:00 -0500

I’ve been alternatively using both KDE Plasma 5 and StumpWM on various machines and have got a working model for using the Equake drop-down in StumpWM.

The StumpWM #'invoke-equake command hides (using StumpWM native hide-window, rather than Emacs’s make-frame-invisible as the latter creates various issues in finding and fetching the Equake window) the Equake frame if it’s the currently active window; it searches through all windows in all groups on the current screen/monitor, and calls emacsclient -n -e '(equake-invoke)' to create an Equake frame if no extant Equake window is found; and if an Equake window does already exist for the current screen, it is yanked into the current group, pulled into the current frame, and unhidden (if necessary).

This seems to work pretty well, though I haven’t figured out how to have StumpWM ‘skip’ the Equake frame when cycling through Emacs windows. (And, of course, having a proper partial height floating Equake window would be ideal.)

EDIT: Here’s a better, more Quake-like set-up, which actually makes use of recently added partial-height floating windows in non-floating groups:

(defun calc-equake-width ()
  (let ((screen-width (caddr (with-input-from-stringp (s (run-shell-command "emacsclient -n -e '(equake-find-workarea-of-current-screen (equake-calculate-mouse-location (display-monitor-attributes-list)) (display-monitor-attributes-list))'" t)) (read s))))
        (desired-width-perc (read-from-string (run-shell-command "emacsclient -n -e 'equake-size-width'" t))))
    (truncate (* screen-width desired-width-perc))))

(defun calc-equake-height ()
  (let ((screen-height (cadddr (with-input-from-string (s (run-shell-command "emacsclient -n -e '(equake-find-workarea-of-current-screen (equake-calculate-mouse-location (display-monitor-attributes-list)) (display-monitor-attributes-list))'" t)) (read s))))
        (desired-height-perc (read-from-string (run-shell-command "emacsclient -n -e 'equake-size-height'" t))))
    (truncate (* screen-height desired-height-perc))))

(setq *equake-width* 1368) ; TODO: programmatically get screen dimensions before Emacs starts
(setq *equake-height* 768)

(defcommand invoke-equake () ()
  (let* ((on-top-windows (group-on-top-windows (current-group)))
         (equake-on-top (find-equake-in-group on-top-windows)))
    (if equake-on-top
        (progn (setf (group-on-top-windows (current-group)) (remove equake-on-top on-top-windows))
               (unfloat-window equake-on-top (current-group))
               (hide-window equake-on-top)) ;; then hide Equake window via native Stumpwm method.)
      (let ((found-equake (find-equake-globally (screen-groups (current-screen))))) ; Otherwise, search all groups of current screen for Equake window:
        (if (not found-equake)          ; If Equake cannot be found,
            (progn
              (run-shell-command "emacsclient -n -e '(equake-invoke)'") ; then invoke Equake via emacs function.
              (setq *equake-height* (calc-equake-height)) ; delay calculation of height & width setting until 1st time equake invoked
              (setq *equake-width* (calc-equake-width))) ; (otherwise Emacs may not be fully loaded)
          (progn (raise-window found-equake)
                   (move-window-to-group found-equake (current-group)) ; But if Equake window is found, move it to the current group,
                   (unhide-window found-equake) ; unhide window, in case hidden
                   (float-window found-equake (current-group)) ; float window
                   (float-window-move-resize (find-equake-globally (screen-groups (current-screen))) :width *equake-width* :height *equake-height*) ; set size
                   (focus-window found-equake)
                   (toggle-always-on-top))))))) ; make on top

(defun find-equake-in-group (windows-list)
  "Search through WINDOWS-LIST, i.e. all windows of a group, for an Equake window. Sub-component of '#find-equake-globally."
  (let ((current-searched-window (car windows-list)))
    (if (equal current-searched-window 'nil)
        'nil
        (if (search "*EQUAKE*[" (window-name current-searched-window))
            current-searched-window
            (find-equake-in-group (cdr windows-list))))))

(defun find-equake-globally (group-list)
  "Recursively search through GROUP-LIST, a list of all groups on current screen, for an Equake window."
  (if (equal (car group-list) 'nil)
      'nil
      (let ((equake-window (find-equake-in-group (list-windows (car group-list)))))
        (if equake-window
            equake-window               ; stop if found and return window
            (find-equake-globally (cdr group-list))))))

;; Set the mouse focus policy;  - :click is best for proper Equake functioning
(setf *mouse-focus-policy* :click) ;; also StumpWM default (I think)

;; Keybindings
;; bind to an appropriate key
(define-key *top-map* (kbd "F12") "invoke-equake")

This is what it looks like:

video link here