OpenDerja_TN — a schema is the one thing you can't take back

Twelve million people write Tunisian Derja every day — in texts, comments, captions — and there is almost no proper, continuous, openly-licensed record of it that you can actually build a tool on. There have been standardisation efforts. There are a few corpora. There are other Arabic-dialect crowdsourcing platforms. But not all of it together, maintained continuously, as one open service with a working translator on top.

So OpenDerja_TN is that: contributors write, correct, and confirm Derja from their own region; out of it comes a citable dataset, a lexicon, and a live translator built directly on the corpus. It's a non-profit from the first line of code — not a vibe, a structural decision.

This is the El Kef mountains game again, if you read chapter two. Something that's ours, that nobody's taking care of properly, so. The scale is just very different now.

The one decision you genuinely cannot undo

Most of the architecture here is my reused blueprint. What's genuinely new is the data model, and nearly every hard call in it comes from a single observation:

A dialect is a continuum, and a schema that forces a single answer destroys information that can never be recovered.

Code can be rewritten in a weekend. Months of community annotation cannot. Every place the schema forces one value where reality has several is information destroyed at write time, and no later modelling pass gets it back. So:

Region is a set, not a value. A word can be northwestern and northern at the same time. It can be pan-Tunisian. It can be northwestern in one era and general in another. Store one region per record — which is the default shape of a database column and makes every query simpler — and you've made an unrecoverable false choice on every single row. So region is a set, and a separate field records whether the item is regional at all, because "which regions" and "is this even regional" are two different questions and collapsing them makes both unanswerable.

Annotations attach to character spans. Always. Even the ones that apply to a whole sentence — its register, its era — get written as a span covering the full text, rather than as a null span meaning "the whole thing." One shape for every annotation means one consensus mechanism, one query path, one set of invariants. Allow the null and every consumer of the data handles two cases forever.

Tokens are annotations, not a hierarchy. Because standardisation rewrites text, and an offset-anchored annotation can migrate along with a rewrite, while a parallel token hierarchy silently drifts out of sync.

Consensus is keyed by the full span — segment, type, start offset, end offset — not just segment and type. Key it on segment-and-type alone and two annotations on different parts of the same sentence overwrite each other, with no error, with no way to notice afterward.

The task idempotency key carries a mandatory slot. Idempotency keys exist to stop duplicate work. But a key that's too coarse doesn't create duplicates — it collapses work that was supposed to happen in parallel. A segment that needs three independent confirmations is three separate tasks by design; key them on just segment-and-type and those three become one, and the corpus quietly has a third of the confirmation it reports having. The slot is mandatory rather than optional because an optional field with a sensible default is a field somebody forgets, and the failure here is silent.

The train/dev/test split is set at insert and never, ever changed. It's the only thing that makes an accuracy claim honest. One row moved between splits after the fact invalidates every number computed since.

Machine output is never fact. Anything a model produces enters as a suggestion — attributed to a system user, flagged, requiring human confirmation before it can touch a canonical field. In a corpus people will cite, the difference between "a Tunisian said this" and "a model guessed this" is the entire value of the dataset, so that boundary is structural, not a convention.

Restraint, written down so nobody helpfully undoes it

There are three language planes in the architecture: TypeScript for the control plane, Rust for a future data plane, Python for a future intelligence plane. Only the TypeScript one has code.

The temptation with a split like that is to start writing the Rust immediately, because it's the fun part. The rule that stops me: the control plane stays TypeScript permanently, and a stage moves to Rust only when it's measured to be a bottleneck at real corpus scale — not when it seems like it would be faster. The Rust modules exist as empty, pre-scaffolded seams so the eventual move is cheap. That they're still empty is the point.

And I wrote that into the project's own ledger — explicitly, "these empty modules are not a gap to fill, do not touch them until the scrapers exist end to end and have been benchmarked" — because a deliberate absence looks exactly like an unfinished job, and a future me (or a future contributor) will find those empty files and helpfully start filling them in. A deliberate presence explains itself. A deliberate absence needs documentation.

The test suite that got smaller instead of failing

A monorepo restructure moved most of the paths in the project. Afterward, the test run was green.

It was green because 13 of 57 test suites could no longer resolve their imports and had simply stopped running. Among the dead ones was a 400-line spec covering the task engine — the single most important piece of domain logic in the whole codebase — silently not executing. Nothing failed. The summary line looked healthy, just shorter. And nobody reads a suite count.

Fixing the module resolution took the run from misleadingly-green to 59 suites and 302 tests genuinely passing.

A suite that can't resolve its imports and a suite that doesn't exist are indistinguishable in a summary. After any structural move, the number worth checking is the count, not the colour.

Why non-profit is a structural decision

Phases one and two of this project are scoped so that if it stops after phase two, it has still produced something that didn't exist before: a published, citable, openly-licensed, regionally-tagged Derja dataset, plus a public tool a non-technical Tunisian can actually use. Everything after that is upside, not a requirement for the work to have been worth doing.

For a solo project with no funding and no deadline, defining a stopping point that still counts as success is a design decision as much as a planning one. It's the decision that makes the thing survivable. I learned that the hard way on other projects where "done" was always just over the next hill.