EduQate, part two — everything the ledger caught
The verification discipline from the last chapter ran for months. This is what it found. Some of these are in the Problems section in more clinical detail; here I get to tell them as stories.
The number that lied in every response
I was fixing something else — a fee-management endpoint that was rejecting every field I sent it with a generic "unrecognized property" error. That turned out to be an import type on the DTO, which strips the runtime metadata that the framework's validation needs to recognise a field at all. Fixed it.
Then, instead of just checking that the request now returned 200, I read the response body. And the fee amount wasn't a number. It was {"s":1,"e":4,"d":[15000]} — the raw internal guts of a precision-decimal library, leaking straight through the API.
Here's the mechanism, because it's a good cautionary tale. There's a global response interceptor whose job is to stringify BigInt values, which JSON can't serialize natively. To do that generically, it walks any object-shaped value in the response and rebuilds it property by property. That generic rebuild strips the prototype off anything object-shaped that passes through it — including the precision-decimal type, which has a perfectly good toJSON() that only works while its prototype is intact.
So every Decimal in every response was wrong. Fees, payments, installments, everywhere, silently, for who knows how long. Nothing threw. Every endpoint returned 200. A typecheck can't see it. A "does it succeed" test can't see it. Reading the calling code won't show it to you, because the corruption happens after the handler, in shared middleware nobody re-reads once it works.
The fix is one explicit instanceof check before the generic walk. The lesson is bigger than the fix: generic code that tears down and rebuilds arbitrary objects to solve one narrow problem will eventually meet a type it wasn't written for. When that code sits in a global position, the blast radius is the entire API at once.
The same bug, four times, three different reasons
The DTO thing above wasn't isolated. The identical symptom — "the endpoint strips my entire request body" — showed up four separate times.
First on fees, from the import type. Then on document requirements, because that DTO had no validation decorators at all, just plain property declarations, and an undecorated property gets stripped under whitelist validation. Third on form submission, where the field is a genuine union type — a form answer can be a string, an array, a number, a boolean, or null depending on the field — and nobody had decorated it because no single type-check decorator is correct for a union. (The answer, which I didn't know existed until then, is a decorator that accepts any value and just requires the key be present.)
After the second one I stopped fixing these one at a time and just grepped the whole codebase for the pattern: DTO classes with declarations and no validation decorators. That caught a fourth instance before I burned a cycle on it. When a root cause turns out to be an undocumented convention, sweep for it immediately instead of tripping over each instance like it's a surprise every time.
Every customer could click Apply. Every customer hit a wall.
The single most important action in the entire product, for its primary user, is apply. I had tested it exhaustively — as a staff member, logged in as an admin, verifying the review-and-approval side.
Then, one day, I logged in as an ordinary customer and clicked the actual Apply button for the first time in a while.
Four error toasts, stacked. A 403 and three "no active form found" 404s, all at once. On every organisation. For every customer. Always.
The 403 came from an endpoint the apply flow calls to check whether an organisation currently has an open intake window. That endpoint is admin-only by design — it was built for staff, it returns internal fields, and it's guarded by an ownership check that only makes sense for a staff account tied to a specific organisation. A customer has no business calling it. Nothing was wrong with the endpoint. It was just the wrong door, and the apply flow had been knocking on it since it was written.
Nothing logged. Nothing 500'd. Nothing paged anyone. It just quietly turned the core conversion action of the product into a button that always fails, for everyone, until somebody clicked it with the browser console open. It could have shipped exactly like that.
The tempting fix was to add the customer role to that endpoint's allowed list. I traced the authorization chain first and found a second, independent guard checking organizational ownership — which a customer account doesn't and shouldn't have data for. Loosening the role list would have just swapped one denial for a different one, and touching the ownership guard would have changed behaviour for every other route that depends on it. So instead: a new, deliberately narrow, genuinely public endpoint that answers only the one question a prospective customer needs — "is this specific organisation accepting applications right now" — filtered server-side, no client-supplied override, no organizational data attached. The staff endpoint and its guards were left completely alone.
An endpoint's authorization can be entirely correct for the caller it was designed around, and still be the wrong endpoint for a different caller who later starts depending on it. "Does this user have permission" and "should this user be knocking here at all" are different questions. Testing only as staff answers one of them.
Two gyroscopes and an out-of-memory crash
I got a live OutOfMemoryError from the running mobile app — the renderer failing to allocate sixteen bytes on a nearly-full heap. A tiny allocation failing like that usually means something has been leaking for a while, not that one big allocation blew the budget.
Two decorative components — the tilt-reactive geometry things from the design system — each independently subscribed to the device's rotation sensor. A real native subscription. And they're used together, constantly, on nearly every screen. So up to two live native sensor listeners per screen, at two polling intervals, for as long as the screen stayed mounted. Plus the sensor hook has a history of not tearing its listener down cleanly on hot-reload, and I'd been hot-reloading one of those exact screens repeatedly right before the crash.
The fast fix was to rip the sensor out of one component. Nobody would notice a decorative diamond that stopped tilting. I didn't do that. I built a shared provider that subscribes to the sensor once, at the app root, and hands the value down. The defect wasn't that the effect existed — it was that two components independently reached for the same native resource instead of one owner holding it. Deleting one would've fixed that night's crash and left the pattern free to recur the next time something wanted device orientation. Same instinct as connection pooling, honestly: two things quietly reaching for the same shared resource is worth noticing even when each one looks fine alone.
Payments: deferred, then actually unwired, then unwired again
When the core lifecycle was verified, payments was next. I deferred it — explicitly, in the ledger — rather than half-verifying a flow I wasn't ready to stand behind.
But "deferred" isn't a state a user can be in. A "not yet verified" row in a ledger reads as pending, when the honest status was not turned on. And a half-built payment flow that a user can physically reach is a worse failure mode than no feature at all. So I removed the entry points. Then, a few days later, under an instruction to go further, I added one guard at the controller level that 404s every payment route before it touches any payment logic — so the whole "payments are off" fact lives in one line at the top of one file, and any future route on that controller is covered automatically.
That second pass also caught something the first had missed. The payment provider's webhook endpoint is public — it has to be, so the real provider can call it. Which made it the one payment route genuinely reachable by an outside caller, the entire time I thought payments were "unwired." The new guard runs independently of the auth layer, so it's covered too. I verified this with a real customer token in hand — all four routes 404 — rather than an anonymous request that might just be getting blocked by auth coincidentally.
"No button links to it" and "it is not reachable" are different claims, especially in a file-based router where any file in the routes folder is reachable by URL whether the UI links to it or not.
The day the load test redirected itself
I set out to do a production-depth code review. It found three real bugs and a dependency-override config that had never done anything. Then it turned into building a k6 load-testing setup. Then the load test surfaced that the backend had never once run as more than a single process. Then that led to clustering. Then the clustering question exposed a rate-limiter design gap. One session, four different projects, each one falling out of the last.
The load test itself had bugs I had to fix before it produced a trustworthy number — wrong ports, endpoints that didn't exist, WebSocket auth sent as a query parameter the server never reads. And a self-inflicted one: the test's own login requests, each doing a real (deliberately slow) password hash, were landing in the same latency metric as the requests I actually cared about, dragging the average up. A load test's own setup phase needs its own tag or it measures itself.
Once the numbers were real: ramping virtual users against a single process, the 95th-percentile latency climbed sharply past 100 concurrent users. And removing the artificial pacing between requests made throughput worse, not better — which is the signature of a saturated single-threaded process, not one with headroom. docker stats confirmed it in one line: one CPU core pegged at 100%, nineteen sitting idle.
The runtime only ever executes on one thread per process. More cores on the host do nothing for a single un-clustered process. So: multiple worker processes. Not Kubernetes — Kubernetes solves scaling across machines, which is a problem I don't have on one server, and it costs a control plane and manifests for no payoff here. A process manager in cluster mode: near-zero code, plus auto-restart and zero-downtime reload for free.
Before turning that on, I audited every piece of shared state that breaks under clustering. Two things came back clean — the rate limiter and the realtime layer were already backed by a shared store. But I found three background jobs guarded only by an in-process flag, two of which would happily double-process the same work across multiple processes. Fixed those to an atomic per-row claim before any side effect.
The rate limiter counted the wrong thing
And then, checking whether clustering was safe, I read how the rate limiter identifies "one caller." It was keyed on network address. For every route. Including authenticated ones.
Which means: in production, every customer on the same office or building WiFi, and every staff member behind the same office NAT, shared a single rate-limit budget. One heavy user could silently throttle everyone else on the same network — indistinguishable, from the outside, from a deliberate attack. Nothing in the code looked wrong. The default (key on IP) is a completely reasonable choice for the one case it was built for — an anonymous login attempt, which has no other identity to key on — and a genuinely bad one for every authenticated request behind a shared connection.
The fix keys on the authenticated user's identity when there is one, falling back to network address only for requests that haven't logged in yet. This needed a small reordering of the request pipeline, so that the component which knows who you are runs before the rate limiter needs that information.
I verified it two ways that actually matter. Logged in as two different users from the same network address, hammered a protected endpoint with both, and checked the actual counter keys in Redis — two independent budgets, not one shared one. Then, the more convincing test: ran the app as two clustered workers with a deliberately tiny limit and hammered it with one user's traffic. Got exactly the shared limit, not double it. Which proves the fix holds across processes and isn't just correct-looking code that happened to work in a single-process test.
CI: green, and yet nothing had been built
Standing up the CI pipeline for real, on a self-hosted runner, took seven rounds in a single day. Round three is the one I tell people about.
Everything went green. Every job, every check, all passing. And nothing had been built. Not a typo. The mechanism that was supposed to chain the build jobs off the test jobs was silently broken by design under the exact conditions those jobs ran in — it just... didn't run them, and didn't complain. I'd previously "verified" this pipeline by reading the workflow files, which it turns out is worth approximately nothing.
Second time in this project that "the config looks right" and "the thing actually happened" turned out to be completely different claims. (The first time was the CSP — but that's the next chapter.)
Fifteen businesses, open and priced and unable to take a single order
A redesign re-keyed the application forms from one grouping to a finer one — from organisation-and-category down to organisation-and-category-and-grade — and deleted the old fallback resolution paths as part of the change. That consequence was identified in advance, written down, and formally accepted by the project owner. "Some old records will stop resolving." Agreed. Moving on.
What nobody did was enumerate it.
Then the demo dataset got replaced with a real, production-shaped one — dozens of real businesses with real names, real fee tables, real coordinates. And production-like data immediately surfaced what the toy dataset had hidden: 15 of 25 active organisations could not be applied to, and nothing anywhere said so.
A real customer could open one of them — full rating, active status, open badges on every option, a full fee table, a prominent red Apply button — click through, pick anything, and land on "No application form is currently available" with the Continue button permanently greyed out. The old-style forms these organisations had could never resolve under the new model, while still being flagged published everywhere the UI looks.
It's invisible from both directions. The staff user's own dashboard says their form is published, so they believe they're live. The customer sees a fully-populated listing with real prices and a big call to action. Nothing errors. No 4xx, no 5xx, clean console on both sides. The only way to find it is to walk the entire flow as a real customer on an organisation nobody thought to test.
And then the fix wasn't the fix. I ran the data backfill. It reported the rows it had written. The flow was still dead — because the resolver was independently also filtering on a column the old-style records don't populate. I only found that second bug by re-driving the flow in a browser, rather than trusting the row count the migration printed. Applyable organisations went from 8, to 23, of 25.
Two things I keep from this. A migration reporting "27 rows written" is evidence about the migration, not about the feature — verify the user-visible outcome, not the step you happen to be able to count. And: accepting a consequence and measuring it are different actions, and only one of them changes anything. "Some records will stop resolving" and "these fifteen named organisations cannot accept a single order as of today" are the same fact at two resolutions, and only the second one makes anybody act. Any change that removes a resolution path should ship with the query that lists exactly what stops resolving.
Signing in on your laptop killed your phone
There was a single column on the user record holding a single session token id. Which means an account could structurally only represent one live session. Log in on your phone, and your laptop's session silently starts failing. Log in again on the laptop, and your phone dies.
It reads as a flaky app, not a bug, because nothing errors at the moment it happens — the surviving session keeps working fine until its access token expires up to two hours later, and then you're just "logged out" for no visible reason. This was live for months and nobody reported it clearly, because "I got logged out" is what users say about every session problem and it never sounds structural.
Proven with two curl "devices" before I touched anything, fixed with a proper sessions table, re-proven with the same script.
The day I deleted live data with my own hands
This is the part of the project I'd most want to remember, so it's going in the story with no varnish.
I was verifying maintenance mode through the API. I sent a PATCH to the platform settings endpoint with a partial config object — just the two fields I was testing.
The merge logic built the new settings blob as a shallow spread over whatever the request happened to carry. So a partial payload replaced the entire blob. Every setting outside my two test fields — gone.
I recovered it and fixed the merge. But the lasting change is a rule I now follow without exception: don't test a write with a hand-rolled partial API body. Drive it through the UI, which sends the shape the server was actually designed for. And capture the current state before you touch anything, either way. I was testing the exact endpoint whose merge semantics were broken, with the exact payload shape that triggers the bug. Both of those rules, followed, would have independently prevented it.
The audit trail that audited nothing
I toggled a single notification preference through the UI and then read the audit table, expecting to see one row describing my change.
Three bugs. The list of "keys to exclude from the audit" contained the two blob fields where every settings value on every tab actually lives — so the audit faithfully recorded the handful of top-level fields nobody ever changes and ignored everything else. A type-comparison bug meant every save also logged two phantom rows. And the endpoint that lets you read the audit log ignored tenancy entirely, which I proved by reading another organisation's log with my own credentials.
An audit feature that runs, writes rows, and audits the wrong things is worse than no audit feature, because it produces a document people will trust.
(Related, found the same week: the "last modified" timestamp never advanced on 34 of 40 tables. Every relative time in the product — "updated 3 hours ago" — was quietly a creation date wearing a modification date's label. Nothing crashes. The number shown is a real number. That's exactly why it lasted.)
The app was unusable on a phone, and one app had no navigation at all
The complaint was one sentence: fine on desktop, very messy on a phone, text falling out of containers, buttons at the edge.
All true, and worse than I expected. Measured at 390 pixels wide:
The admin and super-admin apps had a fixed-width sidebar with no breakpoint, squeezing the main content area down to 130 pixels. Page titles rendered one letter per line, vertically. List item text measured zero pixels wide.
The customer-facing web app, logged in, had exactly one link on the entire page. Not one nav item — one <a href>. No way to reach the dashboard, documents, messages, profile, or settings at all, because the whole navigation lived in a sidebar that was set to only appear on large screens, with no mobile equivalent. Logged out, the header ran 32 pixels off the right edge of the screen.
The uncomfortable part: every single one of these is visible by resizing a browser window, and the app had been through multiple design sweeps without anyone doing that.
A whole microservice, running perfectly, that nothing used
Owner reports a slow page. Lighthouse score of 70, a 3.3-second largest-contentful-paint, a 3.3-megabyte payload. First question is whether server-side rendering would help. It wouldn't — these are auth-gated dashboards, no search crawlers, and SSR just moves the data wait from one place to another.
The actual cause: there is a dedicated image-optimisation microservice. Written in Rust. It generates resized thumbnail variants of every image on upload and stores references to them. It had been deployed, healthy, passing its health check, and burning CPU on every single upload for months — and nothing served its output. Every image resolver in the entire product signed a URL to the full-resolution original instead. Exactly one code path, in one recently-built feature, ever asked for a variant.
So its entire output was dead weight in object storage. And it surfaced as a page-speed complaint, weeks later, on a metric nobody was watching — not as an alert, not as a failing check, because a component that's completely bypassed stays green forever if its health check only measures its own liveness.
The health check answered "can I respond?" — which was true and useless. Nothing anywhere answered "is my output reaching a user?"
The fix wired each image resolver to a size, with a fallback to the original when a variant is missing. But the part that changed how I work is the logging that shipped with the fix: the service now logs every variant it produced, with dimensions and bytes, and the consumer logs which storage key it actually served and whether that was a variant or a fallback. Because the failure mode here wasn't a crash — it was silence. And I added a CI test that spawns the real binary, sends it a real image over raw HTTP, and asserts that both variants come back as valid JPEGs within their intended dimensions — so that "the thumbnailer is running" and "the thumbnailer produces correct output" stop being the same claim.
Maintenance mode that in-app browsers strolled straight past
With maintenance mode switched on, a normal browser correctly showed the maintenance page. An in-app browser — the one inside a messaging or social app — rendered the normal application, with no data.
Tracing it found something structurally wrong rather than a compatibility quirk. The backend enforced the flag correctly. But the screen users see was a client-side component that only renders after a successful network poll to that backend. So enforcement of a platform-wide block lived entirely in code running on the client — code that had to complete a successful cross-origin network call in order to learn that it should block itself. Any client where that call quietly fails — which restricted in-app browsers do more aggressively — renders the ordinary app shell forever, because the very channel that would deliver the "stop" instruction is the one failing.
The fix moved enforcement to the reverse proxy, which every request passes through and no client controls. And it reused what was already there rather than adding a parallel mechanism: a public gate route left off the existing guard's allow-list, so the existing guard produces the block with no new logic; a non-2xx response promoted into an error status that flows into the error handler already serving the branded page.
Two scoping decisions came with it. Check only real page navigations, so static assets don't pay for a round trip. And exclude the operator's own hostname — because otherwise switching maintenance mode on locks you out of the only screen that switches it off. That exclusion isn't an edge case. It's the first thing to get right.
A control that depends on the controlled party's cooperation is not a control.
Browser auto-translate was crashing the app
Chrome's built-in page translation rewrites text nodes in place, underneath React's reconciler, which then tries to remove or replace children that aren't the nodes it remembers, and the app dies with a DOM exception that looks exactly like a React bug and is not one. One attribute — translate="no" on the root HTML of all three web apps — fixes it. It will never reproduce in a test environment. It needs a real user whose browser language differs from the page's.
Which is sort of the theme of everything in this chapter.
There's more — I haven't even mentioned the coverage program that took backend line coverage from 30% to 68% and built out security test suites for cross-tenant isolation across nine resource types, or the production Content-Security-Policy that silently blocked both the PDF viewer and error reporting in the one environment where it mattered, or the several times Google OAuth failed because of LAN IPs in a way that looked like an integration bug every single time and never was.
But you get the shape of it. The expensive bugs live outside the application — in the edge config, the deploy config, the client-side enforcement, the shared middleware nobody re-reads. None of them fail a test. None of them reproduce locally. None of them appear on any architecture diagram. And the only reliable way I've found to catch them is to stop trusting that the code does what it says, and go watch it.