When the CI runner is also the production box

Every workflow here runs on a self-hosted runner. There are three: one for lint and test, one on the dev box, one that is literally the production VM. GitHub Actions is only the trigger and orchestration layer — no hosted-runner minutes are billed anywhere.

The payoff on the production runner is direct: "run this command on the runner" and "run this command on the server that needs to end up hosting the new containers" are the same sentence. No separate SSH-deploy step, no secret key to manage for it, no drift between "what CI built" and "what the server pulls."

The trap that took a while to see

A CI job that stands up part of the stack — docker compose up -d --build backend to smoke-test it — was intermittently failing, and I first assumed it was a stale container left behind by a cancelled prior run. Added a docker compose down cleanup step. Still failed.

The real cause: Docker container names are unique per daemon, not per Compose project. A container_name: override is not namespaced by the project label the way volumes and networks are. Since the runner executes directly on the same box and same Docker daemon as the live production stack, and the dev compose file used the same container_name values as production, the CI job's up was colliding with currently-running production containers of the identical name. The cleanup step could never have helped — it's scoped to the CI job's own project label, and production runs under a different one.

The fix was to prefix every dev/CI container name distinctly from production's. Obvious once seen. Invisible until then, because the failure looked exactly like an ordinary flaky-cleanup problem.

What to keep from this

• Self-hosted runners are worth it for cost and for erasing the deploy hop — but you've traded isolation for it. The runner shares a kernel, a Docker daemon, disk, and RAM with whatever else is on that box.
• Give every environment its own Compose project name and its own explicit container names. The two are namespaced differently and you need both to be distinct.
• Never let a CI job run a broad docker system prune or anything with --volumes on a shared box. A stopped production container and its volume can be swept in the same pass — this is exactly how a monitoring tool's entire history got permanently deleted once.
• Watch RAM headroom, not just CPU. A runner mid-build plus the production stack plus a monitoring stack can sum past a small VM's ceiling, and the thing that gets OOM-killed is not always the one you'd want.