Coordinated Reboots for bootc Hosts

TL;DR – My bootc hosts stage new images every hour, but nothing rebooted them into those images. Now a small timer on each host checks a time window, its own health and its neighbours, takes a lock from a ~200-line Python FleetLock server, and only then reboots. No Kubernetes, no etcd, and protocol-compatible with the Fedora CoreOS tooling in case I outgrow it.

In how I tweak my base image I described how I replaced bootc-fetch-apply-updates.timer with my own bootc-fetch.timer: every hour bootc update pulls the newest image and stages it, but doesn’t reboot. I ended that post with “the actual reboot is a separate step I control manually or via another timer”. This post is about that missing step.

If you are new to bootc, the motivation post explains why I use it, and the posts on the matrix build and conditional builds cover the image chain c-os:base → c-os:cloud → c-os:docker → c-os-deployments:<site>, built nightly by Forgejo Actions on Fedora 44.

Staged Updates Are Useless Until You Reboot

A staged image is a promise: “next boot, you’ll run the new version”. Until that boot happens, the host keeps running the old kernel, the old packages and the old config. So someone has to reboot it.

Doing that by hand doesn’t scale, and I forget. The obvious alternative – a timer that runs systemctl reboot every night at 03:00 – is worse. My Docker Swarm nodes would all go down at the same minute, and hosts that depend on shared storage would come back while the storage host is still booting. A blind timer turns “update the fleet” into “take the fleet offline”.

What I actually want is:

  • reboot only inside a quiet time window,
  • only if there is something new staged,
  • only if this host and the things it depends on are healthy,
  • and never too many hosts of the same kind at once.

First Idea: Just Tell the Others

My first idea was a kind of messaging: before rebooting, a host announces “I’m rebooting now”, and every other host checks for such announcements before making its own.

That’s a classic check-then-act race. Two hosts check at the same moment, both see nothing, both announce, both reboot. Shortening the gap doesn’t fix it, it only makes the bug rarer and harder to reproduce. What you need is an atomic operation: “give me the lock if it’s free” as a single step, decided by exactly one party.

A lock also has a nice failure mode. If a host takes the lock and never comes back, the lock simply stays held. Nobody else reboots, and when I look at the status I see which host is stuck. Stuck-but-safe beats a fleet rebooting itself into a broken image one by one.

Prior Art: Zincati and FleetLock

Fedora CoreOS solved this a while ago. Its update agent, Zincati, talks to a lock server using the FleetLock protocol, which is pleasantly small:

  • POST /v1/pre-reboot – ask for a reboot slot
  • POST /v1/steady-state – give the slot back after the reboot

The usual server is airlock, which keeps its state in etcd. That’s a solid design, but for a handful of hosts it means running an etcd cluster plus another container just to hand out one lock.

So I wrote my own FleetLock server: about 200 lines of Python, no etcd, no extra container. Because it speaks the same protocol, I can swap it for airlock later without touching the clients.

The Server

fleetlock-server.py ships in the c-os:cloud image together with fleetlock-server.service, which is disabled by default – I only enable it on the one host that hands out the locks. The full source is in the gorgeOS repo.

A few design decisions do most of the work:

Single-threaded on purpose. It uses the plain http.server.HTTPServer, not the threading variant. Requests are handled one at a time, so “check if a slot is free and take it” is atomic without any locking code.

State on disk before the answer. Holders are stored in /var/lib/fleetlock/state.json, and the file is written before the client gets its 200:

def save_state(state: dict) -> None:
    tmp = STATE_FILE + ".tmp"
    with open(tmp, "w") as f:
        json.dump(state, f, indent=2)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, STATE_FILE)

fsync plus rename means the server survives its own reboot – which matters, because the lock server is itself a bootc host that wants updates. For each holder it stores the machine-id, hostname, IP and when the lock was taken.

Groups with slots. Not every host competes with every other host. The config defines groups, each with a number of slots. Members are matched by hostname, IP or machine-id, or a client can name its group in the request. A host in several groups needs a free slot in every one of them:

default_slots: 1   # hosts in no group share the group 'default'
groups:
  - name: db
    slots: 1
    members:
      - db1
      - db2
      - 10.0.0.23

The config is re-read on every request, so changing groups needs no restart. There’s also a GET /v1/status endpoint (not part of the protocol) that shows who holds which lock.

What if the lock server reboots? Clients treat “unreachable” as “no”. While the server is down, nobody reboots – the safe default. If that ever becomes a problem, a three-node etcd with airlock is the HA upgrade path.

The Client

The client lives in c-os:base: bootc-reboot.py, run by bootc-reboot.timer every 15 minutes. The service has ConditionPathExists=/etc/bootc-reboot.yaml, so on hosts without a config it does nothing at all. That’s how I opt hosts in one by one.

Each run follows the same chain, and every step can say “not now”:

  1. Am I inside the reboot window?
  2. Is a new image staged? (bootc status --json → status.staged)
  3. Am I healthy? systemctl is-system-running must say running (or degraded, if allowed), and /run/nologin must not exist.
  4. Are my dependencies up? Optional remote checks: a host:port must accept TCP, a URL must return 2xx, and at least min_up hosts in a CIDR must answer on a port.
  5. Take the lock with POST /v1/pre-reboot.
  6. Write a marker containing the current boot_id, then reboot.

After the reboot, the next run finds the marker. If its boot_id differs from the current one, the reboot really happened. The client then waits until the host is healthy and only then sends steady-state to release the lock. If the new image doesn’t come up properly, the lock stays held and the next host waits.

An example config:

server: http://10.0.0.10:7450
group: default

window:
  timezone: UTC
  days: [mon, tue, wed, thu, fri, sat, sun]
  start: "02:00"
  end: "05:00"

timeout: 2

require_hosts:
  - 10.0.0.5:443
  - https://status.example.lan/health

range:
  cidr: 10.0.0.0/24
  port: 22
  min_up: 7

allow_degraded: false

The window may cross midnight, and leaving it out allows reboots at any time.

Things That Bit Me

This is the part I’d have liked to read before I started.

Timer ordering. My first timer used OnBootSec. On a slow boot, “5 minutes after boot” can be reached before the system is even up, so the timer fired right away and judged a half-booted host. Now the timer is ordered after multi-user.target and uses OnActiveSec, which counts from timer activation:

[Unit]
After=multi-user.target

[Timer]
OnActiveSec=5min
OnUnitActiveSec=15min
RandomizedDelaySec=2min

A host stuck at “System is booting up”. One host never finished booting and refused every login with “System is booting up. Unprivileged users are not permitted to log in yet”. That message comes from /run/nologin, which is only removed by systemd-user-sessions.service – and that unit waits for remote-fs.target. A remote mount that hangs keeps the whole host in “booting” forever.

The watchdog from the base-image post already rebooted in that case, but it destroyed all evidence. Debugging from a VM console is painful, so I extended it: after 5 minutes it now also checks systemd-user-sessions.service, and before rebooting it writes a diagnostics log to /var/log/multiuser-reboot/ with the job list, failed units, ordering cycles, mounts and bootc status.

sshd and /var/home. I had RequiresMountsFor=/var/home on sshd. That sounds reasonable until the home mount fails: then sshd doesn’t start, and the one tool I need to fix the mount is gone. Now it’s ordering only:

[Unit]
# Only ordering: a failed /var/home mount must not block sshd.
After=var-home.mount

Timezones. bootc hosts default to UTC, Forgejo’s cron runs in UTC, and I think in Vienna time. That’s one conversion too many to get right in my head. The fix was window.timezone in the client config, and a rule for myself: everything in UTC.

Scheduling the Pipeline

With the reboot window fixed, the build schedule lines up behind it:

  • 23:00 UTC – the c-os chain builds (base, cloud, docker)
  • 01:00 UTC – the deployment images build on top of it
  • 02:00–05:00 UTC – reboot window

The hourly bootc-fetch.timer stages the new image in between, so by 02:00 it’s waiting on every host.

There are two caveats. Manual runs and builds triggered by a push ignore this schedule, so their images get picked up in the next window. And if the c-os chain runs long, the deployment builds at 01:00 use yesterday’s base image. That’s not dangerous, just one day late.

Limitations and Next Steps

  • No authentication. Anyone on the network can take or release a lock. That’s acceptable on my trusted LAN, and only there.
  • A single server. If it’s down, nothing reboots. Safe, but it can quietly delay updates.
  • Possible watchdog reboot loop. If a host never gets healthy, the watchdog will keep rebooting it. The diagnostics logs make that visible, but nothing stops it yet.
  • etcd/airlock later, if the fleet or the need for HA grows. Thanks to the protocol, the clients won’t need to change.

Takeaways

  • Fetching and staging updates is the easy half; deciding when to reboot is the real problem.
  • “Announce and check” is a race. Use an atomic lock, decided by a single party.
  • A lock that stays held when something breaks is a feature: the fleet stops and tells you where.
  • Use OnActiveSec after multi-user.target, not OnBootSec, for anything that judges system health.
  • Write diagnostics to disk before an automatic reboot, or you’ll never know why it happened.
← Prev → Next