The CI/CD chain, and "green" meaning nothing

Not one monolithic pipeline — a chain of purpose-specific workflows triggering off each other:

• ci.yml on every push: backend lint, typecheck, migrations, schema drift, unit tests; a separate pgBouncer smoke job; frontend typecheck, lint, build; a full Playwright E2E job — parallel where possible, with a notify job depending on all of them, reporting pass/fail regardless of outcome.
• build-push.yml after CI succeeds on main, pushing images to the registry.
• deploy-production.yml after a successful build-push — pulls the images, runs prisma migrate deploy, starts the stack, health-checks it, rolls back automatically on failure.
• rollback-production.yml — a manual workflow for rolling back after the automated check already passed but a problem surfaces later.

Chaining with workflow_run rather than one giant job means each stage can be re-run or debugged on its own.

Self-hosted runners

Both CI and production deploy run on self-hosted runners. Two reasons: cost, and the fact that the production runner is the production VM — so "run commands on the runner" doubles as "run these Docker commands on the server that needs to end up running the new containers," with no separate SSH deploy step. The trade is being responsible for keeping that machine alive and patched; if the runner goes down, CI silently stops until someone fixes the runner itself.

concurrency:
  group: deploy-production
  cancel-in-progress: false

Only one production deploy at a time — a second trigger queues rather than racing the first, because two simultaneous docker compose up against the same stack is a genuinely bad time. cancel-in-progress: false means a new deploy waits rather than killing the current one mid-flight.

The round I actually remember

Standing this up took seven rounds in a single day. After everything went green, nothing had been built. The workflow_run chaining was silently broken by design under the conditions those jobs ran in — it just didn't run them, and didn't complain. I'd "verified" the pipeline by reading the workflow files, which it turns out is worth approximately nothing. Merged the build jobs into the main pipeline instead of chaining.

Second time in this project that "the config looks right" and "the thing actually happened" turned out to be different claims.

One more choice: notifications fire on success too, not just failure. For a solo developer, a confirmed "deploy succeeded" notification is what lets you actually walk away instead of anxiously refreshing the Actions tab.