gorgeOS-Cloud

In the last post I walked through the base image that every one of my servers shares. This time I want to look at the first layer on top of it: the cloud image, which adds everything a virtual machine running in a hypervisor environment needs.

The full source is in my gorge.os repository.

Layered images

One of the things I really like about BootC is that you can layer images just like you layer application containers. The cloud image simply starts from the base:

FROM git.m1ch.eu/oci/c-os:base-43

Everything from the base (packages, services, shell defaults) is already there. The cloud image only adds what is specific to VMs. This keeps each layer small and focused, and means I only need to rebuild the cloud image when something in the cloud-specific config changes.

Packages

Four packages are added on top of the base.

qemu-guest-agent – the standard guest agent for KVM/QEMU hosts. It lets the hypervisor query the VM for its IP addresses, freeze the filesystem for consistent snapshots, and execute guest commands during provisioning. Hard to live without when you manage VMs from Proxmox or libvirt.

cloud-init – handles first-boot configuration: setting the hostname, injecting SSH keys, and mounting cloud-provided filesystems. Even though BootC manages the OS image, cloud-init is still the right tool for the instance-specific setup that varies between VMs.

virtiofsd – the VirtioFS daemon. It allows the hypervisor host to share a directory into the VM over a high-performance virtio transport. I use it to share config and data directories into VMs without going through the network stack.

keepalived – implements VRRP for a floating IP between two or more hosts. Useful for high-availability setups where one VM should take over if another goes down.

Monitoring with Beszel

I use Beszel as a lightweight monitoring hub. Every host runs the Beszel agent, which streams metrics (CPU, memory, disk, network) to a central Beszel hub.

Installing the agent

Instead of packaging the agent as an RPM, I download the latest release directly from GitHub during the image build:

RUN curl -sL "https://github.com/henrygd/beszel/releases/latest/download/\
beszel-agent_$(uname -s)_$(uname -m | sed -e 's/x86_64/amd64/' ...).tar.gz" \
    | tar -xz --directory /usr/local/bin beszel-agent

The uname -m translation handles amd64, arm and arm64 in one line, so the same Containerfile works for all my target architectures.

Dedicated system user

Rather than running the agent as root, I create a dedicated beszel user via sysusers.d:

u  beszel  -  "beszel agent"  /nonexistent  /bin/false
m  beszel  disk

The disk group membership is needed so the agent can read disk I/O stats directly. This is the minimal set of permissions required; No shell, or home directory required.

Service and credentials

The systemd unit ships in the image, but the credentials do not:

[Service]
ExecStart=/usr/local/bin/beszel-agent
User=beszel
Environment="LISTEN=45876"
# Environment="KEY=<public key>"
# Environment="TOKEN=<token>"

The commented-out variables are intentional. Adding them to the image would bake secrets into every layer that inherits from this one. Instead, I set them per host using systemctl edit beszel-agent.service, which writes a drop-in under /etc/systemd/system/ – a path that lives in the mutable /etc overlay and survives image updates.

The unit also has a solid set of security hardening options: ProtectSystem=strict, NoNewPrivileges, ProtectHome=read-only, LockPersonality, and so on. For a long-running network-facing daemon, this is worth the few extra lines.

WireGuard tunnel monitor

Most of my VMs are connected via WireGuard. WireGuard is excellent, but it does not automatically reconnect a tunnel after a network interruption. The interface stays up even if no handshake has happened in hours. On a VM that might restart or lose connectivity briefly, this is a real operational problem.

wg-monitor.sh solves it. The script checks the age of the most recent handshake on a given interface and restarts the WireGuard interface if it is stale.

How it works

for l in $(wg show "${INTERFACE}" latest-handshakes | cut -f2); do
    age=$(( now_ - l ))
    if [[ "$age" -gt "$MAX_AGE" ]]; then
        STALE=true
    fi
done

It iterates over all peers. If any peer’s last handshake is older than MAX_AGE_SECONDS (default: 180 s), the interface is considered stale and restarted with systemctl restart wg-quick@<interface>.

Rate limiting

Blindly restarting every minute when a peer is permanently unreachable would cause a restart storm. To avoid this, the script keeps a log of restart timestamps in /var/run/wg-monitor-<interface>.restarts and enforces a limit of 3 restarts per hour:

if [[ "$recent" -ge "$RATE_LIMIT" ]]; then
    log "WARNING: Rate limit reached -- skipping restart."
fi

Template service

The service and timer use systemd’s @ instance template syntax, so a single pair of unit files covers any number of WireGuard interfaces:

# Enable monitoring for wg0 and wg1
systemctl enable --now wg-monitor@wg0.timer
systemctl enable --now wg-monitor@wg1.timer

The check interval (60 s) and the staleness threshold (MAX_AGE_SECONDS) can be overridden per interface via systemctl edit without touching the unit files in the image.

That covers the cloud layer. In the next post I will look how I run a docker swarm on a several bootc-systems.

← Prev → Next