Glusterfs down 🫩

This morning I woke up and the radio was not playing. I soon figured out, that my DNS, AdGuard, was for some reason down.

I started with the debugging, even before I had breakfast. Not evan a coffee!

The docker-container for AdGuard was running and reported no error. So I dug deeper and deeper, just to find out, that several containers showed strange behavior. For example did Traefik had som error that it could not find the middlewares, that where configured via environment variables.

After I while i fond, that there was a problem with the access of some configuration files. I use Glusterfs to store the container configurations and so far I had no problems. But today something was amiss. The Glusterfs drives where mounted and I could access the folders, but when I tried to open a file I got “Glusterfs Transport endpoint is not connected”. What does that mean I wonder?

Turns out, that one brick of the Glusterfs was not connected. In /var/log/glusterfs/glustershd.log I found the error, that I had a peer rejected state-error. With that I could find the answer rather fast.

# Stop glusterd
systemctl stop glusterd.service 
# In /var/lib/glusterd, delete everything except glusterd.info (the UUID file)
mv /var/lib/glusterd{,.bak}
mkdir /var/lib/glusterd
cp /var/lib/glusterd{.bak,}/glusterd.info
# Start glusterd
systemctl start glusterd.service 
# Probe one of the good peers
gluster peer probe <other gluster host name> 
# Restart glusterd, check 'gluster peer status'
systemctl restart glusterd.service 

You may need to restart glusterd another time or two, keep checking peer status.

So after 1h I could finally get my breakfast.