The last door

Radar is built, hardened, and locked down. Two posts of paranoia bought a box with barred doors, a hardware key on the front gate, and a break-glass account hidden in the walls. Now the unglamorous part that quietly undoes all of it: I need to get code onto the box automatically, every time I push, without handing back everything the first two posts earned.

Part One got the engine running. Part Two made it defensible. This one has to keep it that way while still letting the robots in, through a side gate I actually control.

The default way to deploy from CI is to hand your pipeline an SSH private key and let it shell into the box. Don't. A root-capable SSH key sitting in a CI system is the Falcon left running in a Mos Eisley parking lot. It's fine right up until it very much isn't, and "it" is a leaked secret, a poisoned dependency, or one of the dozen quiet ways third-party CI gets owned while you sleep.

Five doors, one I walked through

There are more ways to get code from CI onto a box than are worth listing. I weighed five of them in earnest, walked past four, and here's the reasoning on each.

A self-hosted runner is a persistent agent living on your production host, running whatever the pipeline hands it. That is keeping a rancor in your own pit and trusting it to only eat what you point at. The day a workflow gets poisoned, it runs on the exact box you spent two posts hardening. Hard pass.

An unauthenticated webhook is an unpublicized URL that kicks off a deploy when something pokes it. That is security by obscurity, and an unmarked door is still a door. Now you're authenticating, rate-limiting, and patching an internet-facing thing you invented specifically to avoid an internet-facing thing. No.

An authenticated webhook is that same door with a real lock instead of a hidden one: it checks a signature before it acts, so obscurity stops being the plan. This is a genuinely good tool. I run one elsewhere in this stack to sync content. But the deploy path is the one thing that can pull and run a root-capable image, and for that I didn't want any inbound trigger at all. A locked door is still a door on the street, and the morning the lock has a bug, someone's already leaning on it. Better. Not chosen.

A scheduled pull finally turns the traffic around: the box wakes on a timer, checks the registry, and grabs a new image if one showed up. No inbound hole at all, which is most of the fight won. I actually shipped this version first, and it worked fine. The catch is the timer. Poll every minute and you spend all day asking "anything?" and hearing "no." Poll every hour and your deploy waits in line behind a clock. I wanted the release to land the moment the image finished building, not whenever the next tick came around. Close. Still not it.

A pull over the mesh is the one I walked through. Same outbound direction as the timer, but on demand instead of on a schedule. The pipeline never reaches into the box at all. It builds an image, pushes it to a registry, and pokes the box to go fetch it. The box reaches out for its own orders like a good little astromech. No inbound door, no standing poller, and nothing that outlives the run.

How the good door works

The pipeline authenticates to my mesh with short-lived federated tokens: minted per run, scoped to that run, gone the second it ends. There's no long-lived key sitting in CI for anyone to steal, because nothing outlives a single job. The run joins the mesh as a throwaway node, connects to one deliberately crippled deploy account, and says its piece.

And the deploy is a pull. The pipeline's only real power is "push an image to the registry," which it obviously has, because it just built the thing. The trigger it sends is barely a sentence: go get the new one. The box already trusts that registry. It pulls the image and brings the stack up. CI never touches the host filesystem, never runs a host command, never gets a shell.

Four layers, each dumb on its own

The security here isn't one clever trick. It's four boring ones, stacked so that beating any single layer buys you almost nothing.

  1. The mesh ACL. The CI identity can reach exactly one account on exactly one box, and nothing else on the network. Not the other machines, not other users, not other ports. It gets a hall pass to a single room.
  2. The deploy account's "shell" is not a shell. It's a tiny dispatcher that refuses interactive sessions outright. It reads only the first word of whatever it's handed, so a cute deploy-app; rm -rf / collapses to a harmless deploy-app. It rejects anything that isn't a plain alphanumeric stack name, and it flatly refuses to touch a stack that hasn't been explicitly opted in with a marker file. There is no eval anywhere in it, ever.
  3. Sudo scoped to one binary. Even if you somehow talked your way past the dispatcher, that account can run exactly one program as root: the deploy script. Not a shell. Not a wildcard. One thing. And the grant is execute-only: run that one script as root, nothing else. The account can't edit it or repoint it; the script is root-owned, so the deploy user can't change a byte. It can read it, but that's a plain script with no secret in it, and reading the lock isn't the same as holding the key.
  4. The deploy script stays in its lane. It pulls the named image, brings the stack up, warms the cache, and does nothing else.

Defense in depth, like a Russian doll, if a Russian doll could rm -rf you. Any one layer looks too simple to matter. Stacked, they mean a fully compromised pipeline gets you a broom closet with a very heavy door.

The door that said knock, but not enter

And here's the part you probably came for: the self-inflicted wound. I take my own competence for granted, the way most of us do, and it gets me every time. Flawless work is rare, and a writeup that leaves the screwup out is a lie of omission, the kind that makes everyone look perfect. Heads up: we aren't.

The first time I ran the finished pipeline, it hung. Not an error, not a denial. Just a deploy sitting there knocking politely until it gave up two minutes later. Everything looked right. The token minted, the identity was correct, the deploy account existed, and the permissions read exactly how I meant them.

The catch: on a mesh, "who is allowed to log in" and "which packets are allowed to reach the port" are two separate rules. I'd written the first and forgotten the second. The access rule cheerfully said this identity may knock on that door. The packet filter never said the door may open. So my own pipeline stood on the porch with its credentials in hand, talking to a wall.

One line in the mesh policy fixed it: let the CI identity reach the deploy box on the one port that matters. The lesson is the boring kind that only lands after it's cost you a couple of minutes and a chunk of ego: a rule that grants an identity is not the same as a rule that passes a packet. Check both, or watch your robot knock on a door it's technically allowed to open and physically cannot reach.

So what can a compromised pipeline actually do?

This is the only question that matters, so I'll answer it plainly. If an attacker fully owns one of my CI runs, they can tell the box to pull and run the latest image. That image is the one the pipeline built anyway, so it's a power they already had. That is the entire list. They cannot get a shell, run arbitrary commands, deploy a stack that was never allowed, reach the host filesystem, or pivot to anything else on the network. The blast radius is a closet, not the bridge, and nowhere near the self-destruct codes.

The thread through all three

The same pattern secures anything else I deploy: push to a registry, let the target pull through a command-scoped identity, keep every inbound hole closed, and never mint a credential that outlives a run. You pay for the expensive part once, the dispatcher, the ACL, the scoped sudo, and after that a new service is a single marker file.

That's the whole migration. Off a dead box, onto a hardened one, with a deploy path that doesn't bleed. If there's one thread running through all three parts, it's this: build it like you expect to be breached, because one day someone will try. The goal was never a perfect wall. Perfect walls don't exist, and the people certain theirs is perfect are just the ones who haven't been breached yet. The goal is that when someone gets a foothold, a leaked key, a bad dependency, my own -y at two in the morning, what they find is a broom closet with a very heavy door. And that there's still one hidden way back in, for the nights the someone is me.

Full Series

Three posts, one migration:

  1. vps01 is dead. Long live Radar. The dead two-years-EOL CentOS 7 box, and why you regenerate instead of resurrect.
  2. Barring the Doors: Hardware Keys and Self-Inflicted Wounds. Hardening the new host with sysctl, fail2ban, SELinux and a YubiKey, plus the firewall I removed by accident.
  3. Pull, Don't Push: No Spare Key Under the Doormat. Keyless deploys that never reach into the box. (You're here.)