EduQate

Five layers, one dependency direction: clients, shared packages, domain modules, core concerns, infrastructure. Every arrow points down.

Four client apps share a small set of packages — types, an API client, query hooks, and two UI kits that import nothing else internal. Below them are nineteen domain modules (~21k lines) over sixteen core concerns (~11k lines): auth, policy resolution, settings, a transactional outbox, storage. Then Postgres, Redis, object storage and a Rust image service.

The point isn't tidiness. It's that I can read the blast radius of a change straight off the diagram — touch a shared package and every flow below it counts as unverified until I've driven it again.

Two exceptions, found by counting imports instead of trusting the diagram. The outbox dispatcher reaches up into notification modules to route events, and auth reaches up for the user record. Both are fixable with an interface owned by the lower layer, and neither has cost me anything yet.

A transactional outbox instead of calling side effects inline. Submitting an application owes the user a confirmation email and a notification. Calling those inside the same request that writes to the database leaves a real gap: if the email API has a bad moment right after the commit, the confirmation is gone and nothing says so. Instead I write the event in the same transaction as the change, and a separate process delivers it afterwards, retrying on failure.

A settings hierarchy that's really a safety override. Every configurable behaviour has a platform default, an organisation can override it, and individual branches can override it again. On the surface that reads as flexibility. The real reason is that a branch admin can make a genuinely harmful call, and someone with more authority above them needs to be able to correct it.

Session invalidation for tokens that are stateless by design. A version counter on each user, baked into every token when it's issued and checked on every request. Forcing a full logout is one increment — every token that user holds stops working, and there's no blacklist to maintain.

A form builder, and an argument about who holds the pen. Every organisation needs its own application form, and the open question was who builds it. The instinct was to let admins configure their own. I pushed back: staff aren't necessarily technical, and a live form is exactly where a small mistake becomes a real problem mid-cycle with no support team on standby. Form authoring stays platform-side with a preview-and-confirm step, and forms are versioned so every submission is locked to the version it was filled against.

Schema drift caught in CI. A check that diffs the schema definition against the applied migrations on every run and fails the moment they disagree, instead of letting the gap turn up later as a confusing bug.

The half of this job that isn't in the codebase. I own the environment it runs in as much as the code.

One command brings the whole stack up locally — database, connection pooler, Redis, the Rust image service, the API, three web apps, a reverse proxy — with every dependency gated on a real health check, so the API can't boot against a database that isn't ready yet. Dev containers are name-prefixed away from production, after a CI failure I traced back to the runner sharing a Docker daemon with the live stack: a same-named compose up was colliding with running production containers, not leftovers from a cancelled run.

Production runs on three isolated Docker networks — public web, edge, data — so the database is only reachable from the one service that bridges to it. Nothing is published to the host. The only way in is a tunnel that dials outward, so there are no inbound ports to attack, and it runs as two replicas for failover.

The edge proxy default-denies any host it doesn't recognise, caps request body size and slow-client timeouts before a request reaches the app, and serves a branded maintenance page for the few seconds the backend is gone during a container swap.

The backend runs as a process cluster, not a single process — the runtime pins one process per CPU core. Everything clustering would otherwise break — rate-limit counters, socket rooms, OAuth exchange codes, background-job claims — moved to Redis first, and the boot refuses to start in production without it, so it can't quietly degrade to per-process behaviour.

Every workflow runs on self-hosted runners and declares a least-privilege permission block. Deploys aren't a CI job: a script on the server builds the images there, applies migrations, waits for the backend health check, scans every service's logs, and emails me the result either way.

The pipeline that gates a build: lint, typecheck, a migration-drift check that fails the moment the schema file and the applied migrations disagree, a smoke job that runs the whole stack *through* the connection pooler to catch the bug class that once corrupted data, dependency audits, CodeQL, and a Trivy scan for vulnerable packages and leaked secrets.

I stopped trusting “green” here twice. A cross-file workflow chain silently never fired, because that kind of trigger only evaluates from the default branch. No error anywhere; the run was simply never attempted. I collapsed the chain into one file, where job-to-job dependencies carry no such caveat. Separately, a whole CI run once passed every check while nothing had actually been built.

The database is dumped nightly, shipped offsite and pruned on a retention window — and a separate job restores the latest backup into a throwaway container every week and checks the tables come back non-empty. A backup script exiting zero is not the same claim as a restorable backup.

Two deploy paths, both written down. A routine one, where additive-only migrations are safe with old code running through the few seconds of a swap. And a breaking-schema one that raises a maintenance flag first — gated so it can't be toggled casually — runs the migration behind it, and lifts it only once a log scan across every service comes back clean.

The scars are in the runbook too. Never run a volume prune on a box where the database volume could be momentarily unattached. Give production and monitoring their own compose project names, so a teardown on the wrong file can't reach the wrong volume. New indexes on tables that will hold real data go in their own migration, built concurrently, because a plain index build locks the table for its whole duration.

One global guard chain — throttle, authenticate, check role, check scope — in a fixed order, deny-by-default at every layer. A route that declares no role is closed. A route that declares a scope still has to re-derive and re-check ownership inside the service, because a request header can outrank the URL parameter and the guard on its own isn't enough.

Passwords: a slow hash, a breach-list check on every path that can set one, per-account lockout rather than per-IP, and a dummy hash on the no-such-user path so response timing can't enumerate accounts.

Tokens: stateless JWTs with a per-user version counter for instant revocation, refresh rotation with reuse detection that kills the whole session on a replayed token, refresh held in an httpOnly cookie and the access token in memory only — never local storage.

Rate limiting keys on the authenticated user, not the IP. Keying on IP means everyone behind one office network shares a single budget, which looks exactly like an attack. That needed the auth step to run before the limiter, which is deliberate ordering rather than an accident.