OpenDerja_TN

Region is a set, not a value. A segment can belong to two regions at once, and storing one region would force a false choice on every row. A separate field records whether the item is regional at all, because "which regions" and "is this regional" are different questions.

Annotations always attach to character spans — including the ones that apply to a whole segment, which are written as a span covering the full text rather than a null span. One shape for every annotation means one consensus mechanism and one query path, instead of every consumer handling two cases forever.

Tokens are annotations, not a hierarchy. Standardisation rewrites text. Offset-anchored annotations migrate with a rewrite; a parallel token hierarchy silently desyncs.

Consensus is keyed by the full span, not by segment and type. Otherwise two annotations on different parts of one sentence overwrite each other with no error.

Task idempotency keys carry a mandatory slot. Empty for record-scoped tasks, the character range for span-scoped ones, an ordinal for the N-of-a-kind case. Making it mandatory rather than optional is the whole decision: an optional field with a sensible default is a field somebody omits, and the failure here is silent.

The train/dev/test split is set at insert and never changed. It's the only thing that makes an accuracy claim honest.

Machine output is never fact. Model output enters as a suggestion, attributed to a system user and flagged, and needs human confirmation before it reaches a canonical field.

Three planes exist — TypeScript, Rust, Python — but only the TypeScript one has code in it. A stage moves to Rust when it's measured as a bottleneck at real corpus scale, not when it seems like it would be faster. The Rust modules are scaffolded as empty seams so the move stays cheap, and the fact that they're still empty is written into the project ledger so nobody later reads them as a gap.

No ML model, no Python inference and no GPU dependency before the corpus is real. Nothing in a request path is allowed to await inference. The nlp and media services are health-check-only stubs until their phase arrives.

Every contributor screen is one question, three seconds, no explanation needed. Trust levels, task types and routing are never visible to a contributor. The corpus only gets built if people who aren't linguists can contribute without learning the system first.

Phases one and two are scoped so that if the project stops after phase two, it has still produced something that didn't exist before: a published, citable, regionally-tagged Derja dataset, and a public tool a non-technical Tunisian can use. Defining a stopping point that still counts as success is what makes a project with no funding and no deadline survivable.

The backend is written and tested. The next work — the contributor front-end, the scraper sources, the translator on top of the corpus — is where contributors come in. There's a bilingual architecture ledger (EN + FR) recording the non-negotiable rules and why each one exists, so a new contributor doesn't have to re-derive a settled decision. The repository is private for now and goes public with that opening.