Healing stuck state on every boot

Any long-running process managing stateful workflows can die at exactly the wrong moment and leave something half-finished. Nobody moves it out of that state, because the process that would have done so simply stopped existing.

Concretely: the backend dies while an operator is mid-call with a customer at a counter. The token is stuck in called forever, and the counter believes it's occupied. Nothing in the system will ever notice on its own.

The pattern

Run a recovery pass on every boot, hooked into the framework's startup lifecycle.

@Injectable()
export class RecoveryService implements OnModuleInit {
  async onModuleInit() {
    if (!recoveryEnabled()) return;
    await this.recoverStuckState();
  }

  async recoverStuckState() {
    const stuck = await this.findRecordsStuckInIntermediateState({
      olderThan: STALE_THRESHOLD,
    });
    for (const record of stuck) {
      await this.forceTransitionToSafeState(record);
    }
  }
}

A few details that matter:

• A time threshold, not "any record in this state." Something that entered the state ten seconds ago isn't stuck, it's mid-flight. I use 30 minutes for a called token.
• Idempotent by construction. It runs on every startup, not only after a crash. The logic doesn't need to know whether the last shutdown was clean — it just correctly identifies genuinely stuck records either way. No flag to get wrong.
• An env-var escape hatch. When restoring a known-good backup, you explicitly don't want automatic healing second-guessing the restored data on the next boot.
• Check what points at the stuck record, not just the record. The counter holds a pointer to its "current token." If recovery pushes that token back to queued but doesn't clear the counter's pointer, the counter looks permanently busy with a token nobody's serving. Easy to miss if you only think about the primary entity.

Build it before you need it

This class of bug is invisible until the exact bad-timing crash happens, then very visible and annoying to fix by hand — someone has to notice the stuck record, work out why, and intervene without making it worse. Discovering the need mid-incident, with a real customer standing at a counter that won't clear, is a worse way to learn it.