devops
What a load test actually tells you
What a load test actually tells you
Without one, "can this handle real traffic" is a guess based on how the app feels while I click around — which tells me nothing about concurrent behaviour or where latency actually starts climbing. k6 scripts a virtual-user ramp against real endpoints and reports percentile latencies — p95/p99, not an average that hides the tail people actually experience — plus throughput and failure rate.
export const options = {
scenarios: { ramp: { executor: "ramping-vus", stages: [
{ duration: "30s", target: 20 }, { duration: "1m", target: 100 },
]}},
thresholds: { "http_req_duration{endpoint:api_get}": ["p(95)<300"] },
};
The mistake that isn't obvious until it happens
Any script with a non-trivial setup() — logging in a pool of test accounts — has those login requests land in the same latency metric as the requests actually under test, by default.
A login does a real bcrypt compare, deliberately slow. If setup logs in 20 accounts and the real test fires a few hundred requests early on, those 20 slow logins drag the aggregate p95 to a number that looks alarming and means nothing about the endpoint I'm measuring. My first real run showed p95 of 229ms for simple GETs on a stack under no meaningful load, and it took me a minute to work out why.
The fix: tag setup requests and test requests separately, scope the thresholds to the tag that matters. Cheap to do, easy to forget, and the first symptom is a number confusingly worse than reality.
When more realistic load made the numbers worse
Removing the artificial think-time between iterations should naively just mean more requests and proportionally worse numbers. What actually happened: throughput dropped while latency rose, simultaneously. Not a proportional trade — close to a fingerprint for a saturated single-threaded bottleneck, where requests queue behind one thread that can't get to them faster no matter how many are waiting.
I confirmed it directly with docker stats during the run: one core pegged near 100%, everything else idle. That's what turned "latency got worse" into "this specific process is the bottleneck" — the only version of the finding that tells you what to fix. (It's also what led to PM2 clustering.)
What it isn't
Running the load generator and the system under test on the same machine biases the result — they compete for the same cores. The absolute millisecond figures are pessimistic; the shape of the finding — where latency starts climbing, whether removing pacing helps or hurts — is still informative. It's not a substitute for a real capacity benchmark on production-equivalent hardware, or for first defining what "concurrent" means for your actual traffic. A thousand idle open tabs and a thousand people submitting the same form at once share a headline number and nothing else.