The corpse on life support

I had a CentOS 7 server running for years, long before Capaldi regenerated. It managed a password vault my friends and family used, a private container registry, several websites, and the LDAP directory that all my other machines relied on for user info. The server reached EOL in June 2024 and had gone two years without any security updates. It made me nervous, but I kept it behind a firewall and regularly changed my SSH keys on a non-standard port.

You can't fix this with a simple yum update. The repositories are frozen, the mirrors 404, and the box just sits there collecting CVEs like the TARDIS collecting strays. Once a system is end-of-life, patching isn't really an option. The only real solution is to rebuild on a supported OS and carefully migrate everything over. This is the first of three posts about how I did it: planning, making decisions, and setting up the new server. I named it Radar, to go along with Sherman (my k3s cluster) and Hawkeye (my development and monitoring box).

Regenerate, don't resurrect

The first temptation is always the in-place upgrade. Leap CentOS 7 straight to something current, keep the box, save yourself the migration. Don't. An in-place major-version upgrade of a production host is the Kessel Run without running the numbers. Sure, Han made it... once, in the fastest ship in the galaxy, with the plot on his side. You're not flying the Falcon, and the cargo is everyone's passwords.

Starting fresh with a clean build gives you a system you can actually understand: a known starting point, no years of leftover clutter, and a chance to improve the setup instead of repeating old mistakes. So, set up a new server with a supported OS, migrate your services, and then put the old one out of its misery for good.

Picking the OS

I chose Rocky Linux 10. I've got RHEL/CentOS muscle memory going back to 2003, dnf/yum, SELinux, firewalld all live where my hands expect them, and it's supported into the mid-2030s. AlmaLinux would've been an equally correct answer. I picked Rocky on familiarity, not religion. CloudLinux was off the table: it's a paid, shared-hosting-density OS, the wrong tool for a single-tenant Docker host.

Everything in a container

The old box was a hybrid. Some services ran in Docker, others were bolted to the host with hand-maintained nginx vhosts. The host-native half is exactly the part that's miserable to migrate, so the call was easy: everything goes in a container, and ingress runs through Traefik instead of a dozen .conf files I'd been hand-editing for years. That's a dozen piles of tech debt I was done carrying. The payoff is reproducibility. The next time I move this box it's a git pull and a compose up, not an archaeology dig. No sprinting out ahead of the boulder with the idol in one hand. And we're not all as lucky as Harrison Ford.

Tailscale first, on purpose

This is the most important decision in the whole series, so I'll say it plainly: the very first thing I installed on the new box, before the firewall, before the SSH hardening, before anything that could lock me out, was the out-of-band way back in.

Build the escape pod before you light the engines. The Death Star had a thermal exhaust port too. The difference: I wanted mine, and I knew exactly where it was. Concretely, that meant Tailscale with SSH enabled, which gives me a path to the box that doesn't depend on the host's sshd, firewall, or user database. Every dangerous step that came later, I did through that channel, precisely so a fat-fingered config couldn't strand me. Hold that thought, it pays off before the end of this post.

The kernel landmine

First real fight: Docker wouldn't start. dockerd died complaining about iptables and a missing addrtype kernel module. The new box's OVH minimal image had booted a kernel with no matching kernel-modules package installed, so the netfilter bits Docker needs to build its NAT rules weren't on disk. The fix was almost annoyingly simple: a newer kernel was already installed, just not booted. Reboot into it, the modules show up, Docker starts clean.

The transferable lesson, because you'll hit this on any fresh cloud image: fully update and reboot into the updated kernel before you trust Docker's networking. Don't debug iptables on a kernel you're about to replace anyway.

Killing the directory

The box also ran my OpenLDAP directory. That had to go too, and not just because its host was dying. A self-hosted directory at this scale is a single point of failure wearing a tie. When it hiccups, logins hiccup everywhere. The client stack is a different animal on every OS (SSSD here, nslcd there), and the certificate upkeep had already bitten me more than once. The plan: retire it for plain local users with matching UID/GIDs and SSH keys, machine by machine.

The part where I lock myself out of two boxes at once

Here's the war story, because the clean writeups are lies of omission. To convert a directory user to a local one, you want the name-service resolver to stop handing back the LDAP entry. So on the other boxes I went to stop the resolver, while I was logged in as the very LDAP user it was resolving.

The session died under me. The daemon stayed down. And because that user could no longer be resolved at login, I was simultaneously locked out of public SSH on two machines. A genuine "these aren't the droids you're looking for" moment, except the droids were my own SSH sessions and they were already slipping away.

Recovery? The escape pod from three sections ago. tailscale ssh root@<box>. Root resolves from local files (it was never an LDAP user), completely independent of the daemon I'd just shot, and that got me back in to restart it. This is why Tailscale goes on first. Not because the docs say so, but because the night you need it, it's the only door left standing.

Day one, still standing

By the end of day one, the new box was up, reachable, running Docker, built from the ground up for local users and key-based access, with no directory client in sight. It was also wide open in all the ways a fresh box is. That's the subject of part two: barring the doors, the YubiKey, the break-glass account I hid in the walls, and the afternoon I lowered my own shields with a single dnf command.

Part two is where I make it defensible. And nearly undo all of it with one command.

Full Series

Three posts, one migration:

vps01 is dead. Long live Radar. The dead two-years-EOL CentOS 7 box, and why you regenerate instead of resurrect. (You're here.)
Barring the Doors: Hardware Keys and Self-Inflicted Wounds. Hardening the new host with sysctl, fail2ban, SELinux and a YubiKey, plus the firewall I removed by accident.
Pull, Don't Push: No Spare Key Under the Doormat. Keyless deploys that never reach into the box. (You're here.)