devops
Two deploy paths, and knowing which one you're on
Two deploy paths, and knowing which one you're on
A rolling container swap has a window — a few seconds — where the new schema is live and some old backend processes are still serving requests. Whether that window is harmless or an incident depends entirely on the migration, and the runbook exists to make me decide which before I type the deploy command.
Path A — routine
Additive-only migrations — a new nullable column, a new table, a new enum value — are safe with old code running against the new schema. Old code just doesn't use the new thing. No maintenance mode, no ceremony. The deploy script takes a backup, builds each image, runs the migration, brings the stack up, polls the health endpoint, scans every service's logs for real errors in a short window, and prunes build cache on the way out.
Path B — breaking
A column drop or rename, a new NOT NULL without a default, a type change. Old backend code will misbehave against this schema, so:
- Raise the app-level maintenance flag first — a real feature flag enforced at the edge, not just a static page. On this platform it needs an emailed one-time code to toggle, specifically so it can't be flipped casually or by a single compromised session.
- Confirm the maintenance page is actually showing to a normal user before touching anything. That's the real "users see it's down" moment, not the toggle.
- Run the deploy.
- Only after the health check passes and the log scan is clean, toggle maintenance off.
The maintenance guard deliberately still lets the operator's own admin surface, /auth, and /health through — otherwise turning maintenance on locks you out of the screen that turns it off. That exclusion isn't an edge case; it's the first thing to get right.
Things that are neither A nor B and still bite
- A new index on a table that will hold real data goes in its own migration, built concurrently — a plain index build takes a write lock on the table for its entire duration.
SET NOT NULLon an existing column errors if any row is currently null. Check first; it's not automatic.- A migration reporting "27 rows written" is evidence about the migration, not about the feature. Re-drive the user-visible flow before calling it done — I've watched a backfill report success while the feature stayed broken because a second filter, elsewhere, was also excluding the rows.