Infrastructure
The half of the job that isn't in a repository
I don't hand deployment to a platform. The servers, the edge, the backups and the recovery are mine, including the one serving this page.
- Weekly storage use
- −80%
- Frontend first load
- 2× faster
- Backup tiers
- 3 · one offline
- Reverse proxies
- Caddy · nginx · Traefik
What this page is served from
The host, from the metal up.
- 01Dedicated physical serverBare metal I rent and administer. Not a managed platform.
- 02IncusSystem containers for hard separation between tenants on one box.
- 03DockerThe application containers inside each Incus guest.
- 04nginxTLS termination and host routing at the edge.
- 05This site, and the project sitesfiras.flamouchi.com and the five project subdomains.
This page is served from a box I run
WhyA portfolio that talks about infrastructure and runs on a managed platform is making a claim it can't back.
The site you're reading, and the five project sites beside it, run on a dedicated physical server: Incus for system containers, Docker inside them, nginx terminating TLS and routing by host.
The same practices run the production platform I build at work — the backup tiers, the load-aware scheduling and the erasure scheme below are all in use there, not just here. This site is the instance I can show you the inside of.
Three reverse proxies, picked per job
WhyThe edge is where a bad default becomes a production incident, and the three tools fail differently.
Caddy where automatic HTTPS is the whole point and the config should stay short. nginx where I need precise control over routing, limits and buffering. Traefik for load balancing across Docker containers, where the proxy should discover backends as they come and go rather than be told about them.
All three terminate TLS, cap request body size and slow-client timeouts before a request reaches an application, and default-deny any host they don't recognise.
Backups on three tiers, one of them physical
WhyA backup that has never been restored is a belief, not a backup.
Nightly dumps go offsite to object storage on a retention window. A separate job restores the latest backup into a throwaway container and asserts the tables come back non-empty, because a backup script exiting zero is not the same claim as a restorable backup.
The third tier is physical and deliberately offline. A systemd path unit watches for a specific flash disk being mounted and starts the backup when it appears — plug the disk in and it runs, with no command to remember. It checks for an existing backup at that date first, so re-mounting the same disk doesn't start the job again. An offline copy can't be reached by anything that reaches the server.
Backups that run before the server falls over
WhyThe box had crashed under load before, and the backup window was whenever cron said so.
Backups are scheduled against server load rather than the clock, so a heavy period doesn't get a heavy job on top of it.
Load average is also watched for the climb that has preceded a crash on this hardware. When it shows, the databases and the infrastructure state are pushed off the box ahead of the failure rather than recovered after it. That has fired on a real crash, which is the only reason I trust it.
Deleting a user from backups that already exist
WhyAn erasure request covers the backups too, and a backup you can rewrite isn't a backup.
You cannot go back into an immutable archive and surgically remove one person. So the data is encrypted per user before it is ever written, and erasure destroys that user's key instead of the archive.
The rows stay where they are and become permanently unreadable, including in every historical copy at once. The backup chain keeps its integrity and the erasure is real rather than promised.
A local environment on Proxmox and Incus
WhyCI and staging need real machines, and a single runner doing everything is a queue waiting to happen.
The on-premise environment runs on Proxmox for virtual machines and Incus for system containers, so each workload gets its own isolated machine instead of sharing one.
CI work is spread across several self-hosted runners rather than piling onto one, so a heavy job on one runner doesn't hold up everyone else's.
Scaling up, because scaling out wasn't affordable
WhyA second machine was not in the budget, so the machine that exists had to do more.
Horizontal scaling assumes you can buy another node. With one box, headroom has to come out of the work itself.
The biggest single win was images. A Rust service downscales uploads to 640px and compresses them before they reach object storage. Weekly storage use fell by 80%, and the frontend got about twice as fast on first load, because the bytes it waits for are a fraction of what they were. Storage and egress is the bill that grows quietly and is hardest to walk back.