Sovereignty: Is there a SysAdmin in the room?

For months my private cloud ran like a charm, suddenly my Gitea runners stopped working. In the runner output I noticed the message that my disk was full. Pocket‑ID, my identity provider, had some occasional hick‑ups. This slow decay of my cloud started the day I launched the individual services, no magic fairies that kept my cloud in a pristine condition. I omitted the obvious, I omitted some old skool system administration. Something that you get for free if you use one of the big hyperscalers.

The moment I realised I was the SysAdmin

When you build your own sovereign cloud, you become the platform team.

You build it… You run it… You own it!

This is something that you need to realize when you build your own cloud, regardless of you build it out of curiosity or if you build it to run your business. You need to be able to fix your own shit. You need to have access to somebody to do the maintenance, even if that someone is you!

No shit, Sherlock!

My Gitea runner didn’t stop because of some deep technical mystery. It stopped because I didn’t think oh the most basic rule: clean up your mess. Runner images piled up, /dev/sda1 filled up, and suddenly half my services behaved like they were running through wet cement. After running docker system prune -af to resolve the disk space issue, the Gitea Actions runner stopped working with the following error:

Error response from daemon: failed to resolve reference "registry.vd-sande.nl/sources/gitea-build-image:latest": pull access denied,
repository does not exist or may require authorization: authorization failed: no basic auth credentials

The runner had previously cached the image locally. After the prune, the cache was gone and the runner needed to pull it again, but the disk was already full again before it could. Another problem to solve!

So the prune didn’t fix anything. It just exposed the real problem: the disk layout itself.

Time to dive into the rabbit hole.

Diagnosing the Disk

Check the available space:

df -h /

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       111G  111G     0 100% /

The root partition was at 100%. Checking what was using the space:

sudo du -sh /var/lib/*

101G    /var/lib/docker
11G     /var/lib/containerd
2.7G    /var/lib/registry

Docker was using 101G, and docker system prune -af reclaimed 0 bytes because all images were actively used by running containers.

Checking the physical disk:

lsblk -d -o NAME,SIZE,TYPE,MODEL

NAME   SIZE TYPE MODEL
sda  232.9G disk Samsung SSD 870 EVO 250GB

The physical disk is 232.9GB but the partition is only 113.2GB. There was room to grow, but the partition layout was in the way.

The Partition Layout Problem

Get the partition information:

sudo parted /dev/sda print free

The layout showed:

/dev/sda1  *    2048       237299711   113.2G  Linux
/dev/sda2       237301758  250068991     6.1G  Extended
/dev/sda5       237301760  250068991     6.1G  Linux swap
(free)          250068992  488397167   ~113.5G

The ~113GB of free space was after the swap partition, not directly after sda1. Simply extending sda1 was blocked by sda2.

Freeing Enough Space to Work

Before making any partition changes, we needed at least a few MB free:

sudo journalctl --vacuum-size=50M

This freed ~50MB from archived journal files. This gave me just enough breathing room to proceed.

Living on the Edge

Desperate time, need desperate measures. In my case I had to expand a full Disk on a running Docker host without losing data. This is where the power of Linux shines, you can do things beyond imagination on a running system. Things like repartioning a disk without losing data. This was exact the thing I was going to do.

The Fix: Remove Swap, Extend sda1, Recreate Swap as a File

Step 1 — Disable swap and open fdisk

sudo swapoff -a
sudo fdisk /dev/sda

Step 2 — Repartition interactively

Inside fdisk, in order:

d        # delete partition
5        # remove sda5 (swap logical volume)
d        # delete partition
2        # remove sda2 (extended container)
d        # delete partition
1        # remove sda1
n        # new partition
p        # primary
1        # partition number 1
2048     # start sector — MUST match the original!
         # (press Enter to use default end = full disk)
N        # Do NOT remove the ext4 signature!
a        # set bootable flag
1        # on partition 1
w        # write and exit

Critical: The start sector must stay at 2048. Changing it would destroy the filesystem. Answering N to the signature removal question preserves all existing data.

Step 3 — Reboot

The kernel cannot reread the partition table of a mounted root filesystem live. A reboot is required.

sudo reboot

Step 4 — Expand the filesystem after reboot

sudo resize2fs /dev/sda1
df -h /

Step 5 — Recreate swap as a swapfile

sudo fallocate -l 6G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

The root partition now uses the full available disk space (~240GB), all Docker containers continue running, and the swap is replaced with a file — functionally identical to the old partition. BOOM!

Mitigation is good, prevention is better!

In the future we don’t want to run in the same mess again. So we have to schedule some small background jobs that keeps our Docker host sane.

1. Weekly Docker cleanup cron job

Removes stopped containers, unused networks, build cache older than 7 days, and unused images older than 30 days. Running containers and their images are not affected.

sudo tee /etc/cron.weekly/docker-cleanup > /dev/null << 'EOF'
#!/bin/bash
docker system prune -f --filter "until=168h"
docker image prune -af --filter "until=720h"
EOF
sudo chmod +x /etc/cron.weekly/docker-cleanup

2. Permanent journal size limit

Prevents systemd journal from growing unboundedly.

sudo tee -a /etc/systemd/journald.conf > /dev/null << 'EOF'

[Journal]
SystemMaxUse=200M
RuntimeMaxUse=50M
EOF
sudo systemctl restart systemd-journald

3. Docker container log limits

Caps log file size per container. Without this, long-running containers can write gigabytes of logs over time.

sudo tee /etc/docker/daemon.json > /dev/null << 'EOF'
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
EOF
sudo systemctl restart docker

Note: The log limit only applies to containers started after this change. Existing containers need to be recreated to pick it up.

Time to get serious

The disk is fixed, the system is running again, and everything looks fine. This is exactly how you end up in trouble the next time. Without signals, without alerts, without anything poking when things start to fall apart. I’m basically relying on my future self to magically remember to check everything.

Spoiler: he won’t.

I need tooling that does this for me. Something that actually watches my systems instead of assuming I will. Something that allows me to schedule maintenance taks easily. Should I go full blown and use Prometheus with Grafana, or maybe something lighter. Hard to say if that’s complete overkill for my private Cloud, or exactly what I need.

Either way, there’s plenty to explore. If somebody has some advice for me? Please let me know :)

To be continued.