Sysadmin on George Honeywood's blog

It's all optics

Sun, 01 Nov 2026 20:41:29 +0000

In computer networking, if you need to send data at a rate greater than 1 gigabit/second, you’ll have to look a bit further than the ubiquitous 1000BASE-T.

10GBASE‑T — a 10 gigabit/second iteration of the 1000BASE-T standard — does exist, but it isn’t too popular. 10GBASE‑T uses the same twisted pair cables as 1000BASE-T (CAT 6/CAT 5e), but it is more power hungry (computationally intensive), higher latency, and is more expensive than other options available.

The easy option if you’re only going small distances — less than 5 meters, i.e. within a single datacenter rack — is to use DAC cables (direct attach copper, colloquially known as twinax). DAC cables can handle high speeds, and are usually cheaper than using optical networking.

However, if you need to send data further than within a single rack, at 10 gigabit/second or faster, you’ll probably want to use optical networking. In a simple form you’ll have some setup like this:

So in total, to connect two servers together via a switch, you’ll need four optics, and two fibre optic cables. This is obviously a bit more complicated than using DAC cables, where you’d just need the two cables! Such is the price of extra range and flexibility. This is all Ethernet, despite the very different physical forms.

In an enterprise setup, where you want redundancy, it gets rather more complicated. One method is to configure two switches in an MLAG (multi-chassis link aggregation group), then patch each server up to both of the MLAG switches. Once you’re patched up, configure link aggregation using LACP on the servers and switches:

The idea behind this arrangement is that you can tolerate either a switch failure, or the loss of one of the paths up to the switches — in both scenarios the server will stay connected to the network. An adjacent benefit here is that you can use the bandwidth of both links in aggregate, so long as you’re using layer3+4 packet hashing and there are multiple TCP connections.

NICs

Servers generally do not come with networking integrated onto the motherboard like desktop PCs. Instead, external NICs are used, either in traditional PCIe slots, or via hotswappable OCP cards (these also use PCIe transport).

These allow you to configure your server based on the uplink bandwidth and amount of ports required.

Optics

Optics are what take the electrical signals from the NIC and serialize them into the light that will be sent over the fibre optic cable. On the other end of the connection, the remote optic will deserialize the light back into electrical signals to be handled by the NIC.

Duplex LC connector on an SFP optic

Optics vary wildly in price depending on the vendor selling them, the rated bandwidth, and the type of fibre they work with. Wikipedia has nice lists of the different standards for optics, for 40G/100G, and 200G+.

SFPs and form factors

NIC ports and optics come in various physical form factors. These are standardized as different kinds of SFPs (small factor pluggables):

SFP: SFP is the smallest of the SFP form-factors. There are different specifications under this size, including SFP, SFP+, and SFP28. Usually these ports handle between 1 and 25 gigabit/s speeds.
QSFP (quad-SFP): Within this size, common specifications are QSFP, QSFP+, QSFP28, QSFP56. These ports are generally used for 40+ gigabit/s speeds. Internally, a single QSFP port is made up of four SFP ports known as lanes. This means you can do funky things, like use a “breakout” cable to connect a single 100G QSFP28 port on a switch to 4x servers, each with a SFP28 port (each server uplinked at 25G).
OSFP (octal-SFP): the largest physical size. This is 8x SFP lanes, used for very high bandwidth connections.

Connectors

There are three kinds of connectors I’ve encountered:

LC: These are probably the most common connector seen in datacenters, generally in duplex (two fibre strands), where one strand is used for sending traffic (TX) and the other for receiving (RX).
MPO: These contain many fibre strands (typically 8-24) in a single connector.
One usecase of these high strand counts is trunk cabling between racks. Say you need to connect a rack with 12 servers to a different rack which has your switch in it. In this scenario, you’ve got some options:
- Run 12x duplex LC terminated cables between the racks. This can get fiddly with so many cables.
- Run a single MPO-24 cable (24 strands) between the racks. This MPO-24 cable can then terminate into a cassette on both ends, which will change the presentation into 12x duplex LC ports. From the cassettes you can then run short patch cables to the optics in the switchports and server NICs.
  High strand counts are also handy for breakout applications. For example, if you have a MPO-8 connector on the 1x QSFP side of the breakout, you can split this out into 4x duplex LC connectors on the SFP side.
- You can also buy cables which bundle multiple fibres into a single physical cable, but are split out at the ends. You’d probably want to use some kind of patch panel to make the cable management easier.
SC: I’ve not seen these in the datacenter, instead commonly used for fibre-to-the-premises installs (GPON), where a single fibre strand is used (simplex).

These connectors come with different, incompatible end faces on the fibre strands. The common ones are UPC (blue connectors), and APC (green connectors).
APC connectors have favourable light loss and light return characteristics compared to UPC, which can become more important with longer fibre runs. You need to make sure if you’ve got UPC optics, you have UPC cables!

Another concern is polarity. With duplex LC connectors, you need to make sure the RX side on one optic is connected to the TX side on the other optic (and vice versa). If RX is connected to RX, you won’t get a link up! Generally you can swap the fibres round in the duplex LC connector, so it’s possible to fix at least.

This polarity problem gets worse with MPO cables and their high strand counts. Cables come in different types, A, B, and C, which are used in different cabling polarity “methods”. With an MPO-8 cable, you need to make sure the 4x TX strands hit the 4x RX strands on the other optic, in the right order.

Another thing to be a bit careful with MPO cables is that the different fibre strands don’t always line up when you mix MPO standards. You can mix and match MPO-8 and MPO-12 (the inner 4 strands of the MPO-12 will be unused). However, if you try to mix MPO-12 and MPO-24, none of the strands will line up!

Fibre optic cables

Fibre optic cables come in two main types: multimode and singlemode. If you’ve got multimode fibre, you’ll need to use multimode optics, and vice versa for singlemode. Light in multimode fibre experiences modal dispersion, so reach is limited compared to singlemode fibre.

OS2 (yellow) duplex singlemode fibre with UPC (blue) LC connector

OS2 is a singlemode fibre standard, colour-coded yellow. OS2 fibre is capable of reaching 40 kilometres when used with -ER optics – much further than multimode!

Probably the most common type of multimode fibre is OM4, which is usually colour-coded aqua. OM4 supports distances of around 100 to 150 meters. OM5 is a revision of this standard, which can extend the reach of OM4 by around 50 meters.

The economics here can be tricky. Multimode cables tend to be more expensive than singlemode cables, but singlemode optics are often more expensive than multimode optics.
Historically multimode fibre would be used for shorter runs, e.g. within a DC cage, and singlemode used when longer distances are required. However, costs for singlemode optics have gone down, so multimode has become a bit rarer.

Multiplexing

In their simplest form, optics send only a single wavelength of light down a fibre strand, but it is possible to send multiple wavelengths down a single strand — this is known as multiplexing. This is more common with singlemode fibre than multimode. There are a couple of different usecases here:

Bidirectional optics. BiDi optics both send and receive light on a single strand of fibre (simplex). Simplex fibre is handy for the residential usecase of hiding a fibre along a wall, or under doors. G.657B3 bend insensitive fibre can be less than a millimeter wide, with a minimum bend radius of circa 5mm.
High bandwidth. Multiplexing is useful if you want to increase your throughput over a single strand. Higher speed links are usually made up of multiple combined lanes of traffic, and the simplest way to transport this is to use multiple strands of fibre. For example, 100GBASE-SR4 optics use 8 strands of multimode fibre, 4 for each direction, each strand running at 25G.
However, say you’re reusing an existing run of OS2 duplex fibre, you can use 100GBASE-LR4 optics instead, which multiplex 4x 25G wavelengths onto each directional strand.

Optics which use multiplexing tend to be more expensive than non-multiplexing ones — this is why multimode optics tend to be cheaper than singlemode optics!

Footnote

There are lots of things I’ve missed out or glossed over here. FEC (forward error correction) in particular is very interesting. I’d also like to cover link auto-negotiation, but I’d need to learn how that works first! Bear in mind I’m writing from the blinkered perspective of a Linux sysadmin, rather than a network engineer or fibre tech — I’ve never spliced a fibre!

Bodging a brouter

Fri, 17 Oct 2025 19:20:35 +0100

I wanted to do something a bit weird the other day.
I’ve got two networks. 192.168.0.1/24 and 192.168.1.1/24, which are at different sites — .1. is at my parent’s house, .0. is at my flat.
A Jellyfin media server lives on the .1. network, and I want to be able to reach that from my the .0. network, where I have a Roku with the Jellyfin app installed.

The really easy option here is to add a port forward on the .1. side router, such that you can reach the Jellyfin server from the public internet — this is exactly what I used to do for Plex. However, exposing things to the internet isn’t usually a great idea from a security perspective, especially when you aren’t great at keeping things up to date!

I’ve got Tailscale (which is a fancy Wireguard VPN solution) set up on the .1. side, so I can reach that network from specific devices that have the Tailscale app installed, but unfortunately there isn’t an app for the Roku. What makes things a bit more complicated is that the router for the .0. network is a rubbish ISP provided one. I can’t easily avoid using it as the flat has 4G/LTE internet, don’t ask! As such I can’t configure any extra routes or install the Tailscale VPN on the router itself.

My first pass at this was very lazy. On my desktop on the .0. side I’ve got Tailscale running, so it can reach the Jellyfin server over on .1. fine. All that’s required is a little TCP proxy ¹:

desktop$ socat -dd TCP4-LISTEN:8096,fork,reuseaddr TCP4:192.168.1.30:8096

Then on the Roku I can configure the Jellyfin client to hit the desktop LAN .0. IP and the connection is forwarded on to the remote Jellyfin server at 192.168.1.30. This worked perfectly fine, but having to keep my desktop on whenever I wanted to use Jellyfin was a bit annoying.

My next idea was to just put this on a Raspberry Pi which I can leave on permenantly without using too much expensive electricity. The downside with this socat trick is that it’s a bit static, if you wanted to talk to multiple endpoints you’d need to listen on multiple ports and remember all the mappings.

What I really wanted was a proper routed setup, where any .0. device can route to anything on the .1. network. This is where I had a devious idea — to use the Raspberry Pi as a bridge for all traffic on the way to my router. Bridging is where a device with two NICs, links them, such that traffic coming in on one, is sent straight back out the other.
For my purpose the Pi can’t act purely as a bridge though, it also needs to do some routing of the .1. network bound packets, such that they pass through the tailscale tunnel.

As far as I can tell, this is a bit of an unusual thing to want to do — it’s called bridge routing (or brouting). It doesn’t seem to easy to do this with a usual Linux bridge device ², but thankfully it’s pretty simple to configure something that behaves the same.

These are the interfaces on my brouter:

root@raspberry:~# networkctl
IDX LINK TYPE OPERATIONAL SETUP
 1 lo loopback carrier unmanaged
 2 eth0 ether routable configured
 3 eth1 ether routable configured
 4 tailscale0 none routable unmanaged

I’m using eth0 as the upstream side (plugged into the ISP router) and eth1 as downstream (plugged into a switch and WAP).
You’ll need some kernel config:

root@raspberry:~# cat /etc/sysctl.d/99-proxyarp.conf
net.ipv4.ip_forward=1
net.ipv4.conf.eth0.proxy_arp=1
net.ipv4.conf.eth1.proxy_arp=1

ip_forward makes the kernel to retransmit packets that weren’t destined for a local IP address.
proxy_arp causes ARP traffic coming in on one interface to be sent back out of the other, replacing the MAC addresses such that layer 2 works as expected.

Once you’ve got your kernel parameters set, you need a bit of iptables magic to handle the routing part of the equation. We want to mark the .1. bound packets, so that they are forced to route via our tailscale0 interface:

iptables -t mangle -A PREROUTING -d 192.168.1.0/24 -j MARK --set-mark 1
ip rule add fwmark 1 table 100
ip route add default dev tailscale0 table 100

This isn’t quite enough to make things work unfortunately. Checking the traffic, I can see packets headed out over the Tailscale interface, but no replies coming back:

root@raspberry:~# tcpdump -i tailscale0 'icmp' -n
18:53:28.804692 IP 192.168.0.3 > 192.168.1.30: ICMP echo request, id 253, seq 1, length 64
18:53:29.805010 IP 192.168.0.3 > 192.168.1.30: ICMP echo request, id 253, seq 2, length 64
18:53:30.829186 IP 192.168.0.3 > 192.168.1.30: ICMP echo request, id 253, seq 3, length 64

This is because devices on the .1. net can’t route directly back to .0.. To make this work we have to SNAT (source NAT) the traffic such that it appears to come from the local side of the Tailscale tunnel:

iptables -t nat -A POSTROUTING -o tailscale0 -j MASQUERADE

Now we see replies coming back too! 100.120.7.128 is the local IP of the tailscale0 interface.

root@raspberry:~# tcpdump -i tailscale0 'icmp' -n
18:56:50.428553 IP 100.120.7.128 > 192.168.1.30: ICMP echo request, id 254, seq 1, length 64
18:56:50.499516 IP 192.168.1.30 > 100.120.7.128: ICMP echo reply, id 254, seq 1, length 64
18:56:51.429393 IP 100.120.7.128 > 192.168.1.30: ICMP echo request, id 254, seq 2, length 64
18:56:51.517161 IP 192.168.1.30 > 100.120.7.128: ICMP echo reply, id 254, seq 2, length 64
18:56:52.431187 IP 100.120.7.128 > 192.168.1.30: ICMP echo request, id 254, seq 3, length 64
18:56:52.509740 IP 192.168.1.30 > 100.120.7.128: ICMP echo reply, id 254, seq 3, length 64

I figure I better include a diagram here:

The final piece of the puzzle is that this setup breaks DHCP (dynamic IP configuration) — as it uses broadcast traffic, and the kernel won’t forward broadcast traffic. This is pretty easy to work around, you can install a little daemon to proxy traffic recieved downstream on eth1 up to the ISP router on eth0. I used isc-dhcp-relay and the config is trivial:

root@raspberry:~# cat /etc/default/isc-dhcp-relay
# What servers should the DHCP relay forward requests to?
SERVERS="192.168.0.1"
# On what interfaces should the DHCP relay (dhrelay) serve DHCP requests?
INTERFACES="eth1 eth0"

The network flow for DHCP now looks like this, where 192.168.0.125 is eth0’s local IP and 192.168.0.126 is eth1’s:

root@raspberry:~# tcpdump -i any 'port bootps' -n
19:12:11.5112 eth1 B IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 312
19:12:11.5114 eth0 Out IP 192.168.0.125.67 > 192.168.0.1.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 312
19:12:11.5345 eth0 In IP 192.168.0.1.67 > 192.168.0.126.67: BOOTP/DHCP, Reply, length 300
19:12:11.5346 eth1 Out IP 192.168.0.126.67 > 0.0.0.0.68: BOOTP/DHCP, Reply, length 300
19:12:11.5418 eth1 B IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 308
19:12:11.5419 eth0 Out IP 192.168.0.125.67 > 192.168.0.1.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 308
19:12:11.5684 eth0 In IP 192.168.0.1.67 > 192.168.0.126.67: BOOTP/DHCP, Reply, length 300
19:12:11.5685 eth1 Out IP 192.168.0.126.67 > 192.168.0.53.68: BOOTP/DHCP, Reply, length 300
19:12:11.5760 eth1 B IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 318
19:12:11.5761 eth0 Out IP 192.168.0.125.67 > 192.168.0.1.67: BOOTP/DHCP, Request from aa:bb:cc:dd:ee:ff, length 318
19:12:11.5814 eth0 In IP 192.168.0.1.67 > 192.168.0.126.67: BOOTP/DHCP, Reply, length 300
19:12:11.5816 eth1 Out IP 192.168.0.126.67 > 192.168.0.53.68: BOOTP/DHCP, Reply, length 300

The advantage of this setup is that it’s transparent to downstream devices, they think they are just talking to the ISP router normally. That it just happens to sit on the path is useful if something goes wrong with the Pi ³, I can just physically unplug it and I won’t lose anything other than my extra route.

It probably would have been easier if I had set up my Pi with OpenWRT or similar, and had it act as a normal router, downstream of the ISP provided router. Well, at least this way I learnt some stuff!

You could also use a HTTP reverse proxy for this, but these tools usually need a bit more config than socat does. ↩︎
It might be possible via the br_netfilter kernel module and net.bridge.bridge-nf-call-iptables=1 parameter, but I couldn’t figure out how to make it work! ↩︎
SD card failure being what I am afraid of here! ↩︎

ZFS snapshots and how to sync them

Sat, 07 Dec 2024 20:50:25 +0000

I am quite a big fan of ZFS. One of its useful features is snapshotting, allowing you to record the state of your filesystem at a specific point in time. This means if you later change or delete files, you will be able to roll-back to this previous snapshot, or restore individual files. So far, so normal — even NTFS has a kind of snapshot/restore capability.

What makes ZFS’s snapshots even more useful is what they enable in terms of filesystem synchronization. My use case is that I have 10 TB of data at a remote site that I want to back up to my local desktop. I am constrained by a slow internet connection, so I need a solution that doesn’t use much bandwidth.

What I can do is to have a matching snapshot pair on both sides, then take another snapshot on the sending end. ZFS can then find which blocks have changed on the sending side, and send and replay the differences to the receiving side. Unfortunately you still have to do a single initial replication on the whole dataset — you can’t get around that!

In my case the initial replication was going to take more than a week over the slow link; I ended up sneakernetting the data between sites to speed up the process. While you can achieve incremental results with a tool like rsync, the process isn’t as efficient, as it has to check to see if each file has been updated.

syncoid is the tool I use to automate the zfs snapshot, zfs send | receive process. I run a command like this to pull all the changes since the previous run:

root@desktop:~# syncoid --recursive --skip-parent root@192.168.1.10:tank tank-offsite

Here tank is a pool on the remote machine at 192.168.1.10, and tank-offsite is mounted on the local desktop. I don’t use the desktop machine that runs this backup consistently, it suspended most of the time, so I needed a way to schedule this job regularly.

systemd to the rescue! I created a service to run the syncoid command, and a timer to run the service on a schedule. The magic here is the WakeSystem= timer option — which resumes my desktop from suspend when the timer triggers. Here is my config — it is pleasantly simple:

root@desktop:~# systemctl cat tank-offsite.{service,timer}
# /etc/systemd/system/tank-offsite.service
[Unit]
Description=Backup tank to tank-offsite
Requires=network-online.target
After=network-online.target

[Service]
Type=oneshot

# wait for the network to be usable
ExecStartPre=sh -c 'until ping -c 1 192.168.1.10; do sleep 1; done'
# do the sync
ExecStart=/usr/local/sbin/syncoid --recursive --skip-parent root@192.168.1.10:tank tank-offsite
# suspend the machine again
ExecStopPost=/usr/bin/systemctl suspend

# /etc/systemd/system/tank-offsite.timer
[Unit]
Description=Backup tank to tank-offsite on a schedule

[Timer]
OnCalendar=Mon-Sun 03:00
WakeSystem=true

[Install]
WantedBy=timers.target

The only little hack here is the ExecStartPre= line; my first pass at this didn’t include it. I was hoping that After=network-online.target would be enough to make sure the network stack was up before the syncoid command would run. Evidently not:

$ journalctl -u tank-offsite
Dec 22 03:00:28 desktop systemd[1]: Starting tank-offsite.service - Backup tank to tank-offsite...
Dec 22 03:00:29 desktop syncoid[162504]: ssh: connect to host 192.168.1.10 port 22: Network is unreachable
Dec 22 03:00:29 desktop syncoid[162505]: ssh: connect to host 192.168.1.10 port 22: Network is unreachable
Dec 22 03:00:29 desktop syncoid[162423]: CRITICAL ERROR: ssh connection echo test failed for root@192.168.1.10 with exit code 255 at /usr/local/sbin/syncoid line 1714.
Dec 22 03:00:29 desktop systemd[1]: tank-offsite.service: Control process exited, code=exited, status=1/FAILURE
Dec 22 03:00:29 desktop systemd[1]: tank-offsite.service: Failed with result 'exit-code'.
Dec 22 03:00:29 desktop systemd[1]: Failed to start tank-offsite.service - Backup tank to tank-offsite.

systemd’s docs have some discussion of this scenario. They suggest adding a separate service that runs the same ping check with Before=network-online.target specified. This means that any services depending on network-online.target will only start once the ping has been successful. As I only have a single service that cares about this, keeping it all together in the same unit with ExecStartPre= made sense to me.

You may have noticed a slight issue with this timer setup: my machine will always suspend after the backup job finishes. This is the desired outcome when the timer has caused the desktop to wake up, but it will probably be quite annoying if I were using the machine at the time. Luckily I’m usually fast asleep before 03:00! I couldn’t find a neat way of avoiding this.

One thing that I’d still like to get working is ZFS’s nop-write support. My use for this is that I have a different daily job that tars up data from my VPS, overwriting a file on disk. This means that every day I have to sync the full tar file, as from ZFS’s perspective it is completely new data, even though there will have been very few changes. The data in question is only about 1.3 GB — so it isn’t too slow, but if I can avoid having to copy it at all then it would be preferable. Alternatively, I could just swap to using rsync for this job, and avoid re-writing the data at all. ZFS’s compression should replicate the benefits I was getting from compressing the tar file.

a few days later

Getting nop-write working was indeed really just as simple as swapping the dataset checksum setting to sha256 (or you can pick one of the other options)

$ zfs set checksum=sha256 pool_name/dataset_name

Now even though the job rewrites this large tar file daily, there is nothing for syncoid to copy over! (well, only 84 KB compared to 1.3 GB)

$ journalctl -qu tank-offsite --grep 'tank/backup@' | tail -n 2
# without `nop-write`
Dec 28 03:00:43 desktop syncoid[337636]: Sending incremental tank/backup@syncoid_desktop_2024-12-27:23:38:04-GMT00:00 ... syncoid_desktop_2024-12-28:03:00:43-GMT00:00 (~ 1.3 GB):
# with `nop-write`
Dec 28 13:28:38 desktop syncoid[26300]: Sending incremental tank/backup@syncoid_desktop_2024-12-28:13:12:15-GMT00:00 ... syncoid_desktop_2024-12-28:13:28:38-GMT00:00 (~ 84 KB):

I am now questioning why I didn’t use rsync in the first place, however. nop-write saves me from replicating the unchanged overwritten file, by preventing the writes from actually hitting the disk. It doesn’t avoid the whole daily download from the VPS though! ZFS couldn’t possibly know what writes would be no-ops without having the new data to checksum against what is already on disk.

The only slight gotcha with nop-write is that you need to make sure that you don’t truncate the file you rewrite. For example something like this won’t work:

$ ssh $VPS_HOSTNAME 'tar -cf - /var/www/' > var-www.tar

If you do this, then the shell first truncates the file before writing the new data. There is probably a clever bash way of avoiding the truncation, but using dd with the conv=notrunc option is a bit more self documenting:

$ ssh $VPS_HOSTNAME 'tar -cf - /var/www/' | dd of=var-www.tar conv=notrunc bs=1M

Although nop-write sounds very compelling at first – it is actually fairly niche. Most of the time it is possible (and more efficient) to avoid rewriting unchanged data!

a few weeks later

At some point I decided to enable some power management settings on my desktop. Usually, my desktop is suspended. However, sometimes one of my meddling cats will step on the keyboard, waking it up, so it is handy if the machine automatically suspends after being idle for some amount of time. Auto-suspend however created a bit of headache for this backup job:

Jan 10 03:00:44 desktop systemd[1]: Starting tank-offsite.service - Backup tank to tank-offsite...
Jan 10 03:00:55 desktop sh[296224]: PING 192.168.1.10 (192.168.1.10) 56(84) bytes of data.
Jan 10 03:00:55 desktop sh[296224]: 64 bytes from 192.168.1.10: icmp_seq=1 ttl=62 time=12.8 ms
Jan 10 03:00:55 desktop sh[296224]: --- 192.168.1.10 ping statistics ---
Jan 10 03:00:55 desktop sh[296224]: 1 packets transmitted, 1 received, 0% packet loss, time 0ms
Jan 10 03:00:55 desktop sh[296224]: rtt min/avg/max/mdev = 12.842/12.842/12.842/0.000 ms
Jan 10 03:00:56 desktop syncoid[296226]: Sending incremental tank/backup@syncoid_desktop_2025-01-09:03:00:51-GMT00:00 ... syncoid_desktop_2025-01-10:03:00:56-GMT00:00 (~ 3.2 MB):
Jan 10 03:01:11 desktop syncoid[296226]: Sending incremental tank/backup/email@syncoid_desktop_2025-01-09:03:01:11-GMT00:00 ... syncoid_desktop_2025-01-10:03:01:11-GMT00:00 (~ 1.5 MB):
Jan 10 03:01:13 desktop syncoid[296226]: Sending incremental tank/backup/lxc@syncoid_desktop_2025-01-09:03:01:13-GMT00:00 ... syncoid_desktop_2025-01-10:03:01:13-GMT00:00 (~ 4 KB):
Jan 10 03:01:14 desktop syncoid[296226]: Sending incremental tank/backup/pbs@syncoid_desktop_2025-01-09:03:01:13-GMT00:00 ... syncoid_desktop_2025-01-10:03:01:14-GMT00:00 (~ 1.0 GB):
Jan 10 08:52:35 desktop syncoid[296484]: cannot receive incremental stream: dataset is busy
Jan 10 19:29:53 desktop syncoid[296488]: lzop: Broken pipe: <stdout>
Jan 10 19:29:53 desktop syncoid[296226]: CRITICAL ERROR: ssh -S /tmp/syncoid-root@192.168.1.10-1736478055-2255 root@192.168.1.10 ' zfs send -I '"'"'tank/backup/pbs'"'"'@'"'"'syncoid_desktop_2025-01-09:03:01:13-G'"'"' '>
Jan 10 19:29:55 desktop syncoid[296226]: Sending incremental tank/backup/veeam@syncoid_desktop_2025-01-09:03:15:13-GMT00:00 ... syncoid_desktop_2025-01-10:19:29:55-GMT00:00 (~ 4 KB):
Jan 10 19:29:59 desktop syncoid[296226]: Sending incremental tank/data@syncoid_desktop_2025-01-09:05:52:20-GMT00:00 ... syncoid_desktop_2025-01-10:19:29:58-GMT00:00 (~ 35.8 MB):
Jan 10 19:30:39 desktop systemd[1]: tank-offsite.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 10 19:30:39 desktop systemd[1]: tank-offsite.service: Failed with result 'exit-code'.
Jan 10 19:30:39 desktop systemd[1]: Failed to start tank-offsite.service - Backup tank to tank-offsite.
Jan 10 19:30:39 desktop systemd[1]: tank-offsite.service: Consumed 31.336s CPU time, 43.7M memory peak.

It wasn’t immediately clear what was going wrong to me, as I’d completely forgotten that auto-suspend was enabled. What made it more obvious was this other journal entry:

Jan 10 03:15:40 desktop systemd-logind[1367]: The system will suspend now!
Jan 10 03:15:40 desktop ModemManager[1645]: <msg> [sleep-monitor-systemd] system is about to suspend
Jan 10 03:15:40 desktop NetworkManager[2071]: <info> [1736478940.0158] manager: sleep: sleep requested (sleeping: no enabled: yes)
Jan 10 03:15:40 desktop NetworkManager[2071]: <info> [1736478940.0163] manager: NetworkManager state is now ASLEEP
Jan 10 03:15:40 desktop systemd[1]: Reached target sleep.target - Sleep.
Jan 10 03:15:40 desktop systemd[1]: Starting nvidia-suspend.service - NVIDIA system suspend actions...

Conveniently, it is quite easy to add “inhibitions” that prevent the system from sleeping. I added a systemd-inhibit call on the ExecStart= line of the unit:

ExecStart=/usr/bin/systemd-inhibit /usr/local/sbin/syncoid [-snip-]

This wasn’t quite enough, however. There is a race condition if you call systemd-inhibit while the machine is still waking from sleep:

Jan 12 03:00:51 desktop systemd-inhibit[398074]: Failed to inhibit: The operation inhibition has been requested for is already running
Jan 12 03:00:51 desktop systemd[1]: tank-offsite.service: Main process exited, code=exited, status=1/FAILUR

I’ve hacked around this by adding a 30s sleep in another ExecStartPre= line. I’m somewhat surprised there isn’t a built-in option to inhibit sleep while a unit is running, but I suppose this is a niche thing to do.