A deploy that rolls itself back at 2am

With nobody watching a deploy in real time, this is the difference between a bad deploy self-healing in two minutes and a bad deploy staying down until I wake up and notice.

Four steps: snapshot the currently running image tag, deploy the new one, health-check it, and on failure automatically redeploy the snapshot and health-check that too.

Snapshot before touching anything. Without recording what's currently running, "roll back" has no defined target.

current="$(docker inspect --format='{{index .Config.Image}}' app-backend)"
echo "$current" | sed -E 's#.*-backend:##' > "$LAST_GOOD_TAG_FILE"

Migrations run as a one-off container, separate from starting the stack, so a migration failure stops the deploy before schema-incompatible code is running against the database:

docker compose pull
docker compose run --rm --no-deps backend sh -c 'DATABASE_URL="$DIRECT_URL" npx prisma migrate deploy'
docker compose up -d

Health check with a bounded retry loop, not a single check. A single check right after docker compose up -d almost always fails — the container needs real time to start and connect. The loop gives it up to two minutes before concluding it's actually broken.

for i in $(seq 1 30); do
  body="$(docker compose exec -T backend curl -sS http://localhost:3002/health)"
  if echo "$body" | grep -q '"ok":true'; then exit 0; fi
  sleep 4
done

Rollback triggers only on failure, and isn't blindly trusted either — it goes through the same health-check loop. If even the rollback doesn't come back healthy, the workflow exits non-zero and pages for manual intervention. No infinite auto-retry that could mask a genuinely broken state.

What it doesn't catch

A health check passing only means the backend responds to /health — not that the deploy is correct. A subtle logic bug, a broken-but-non-crashing feature, a slow memory leak all sail straight through. Automatic rollback catches "broken and down," not "technically running but wrong." That second category still needs real monitoring, error tracking, and someone actually looking.