A Mixture-of-Experts language model built end to end in PyTorch — own tokenizer, own data pipeline, own training loop. Nothing pretrained, nothing inherited. This is the whole system, stage by stage, with the numbers each stage actually produced.
Each stage consumes the output of the last, and can only run once the one before it has finished. Two are irreversible: the tokenizer fixes the vocabulary every later stage depends on, and pretraining fixes the weights everything after it merely adjusts.
Byte-level BPE over a 20 GB sample drawn from the same mixture the model will train on. Starting from raw bytes, the most frequent adjacent pair is merged into a new token, 43,861 times over.
Hindi is sampled at twice its share of the training mix. Devanagari needs more merges than Latin to encode efficiently, and an under-merged Hindi vocabulary silently doubles the token cost of every Hindi document for the entire run — a mistake you would only discover four days in.
Same meaning, 2.8× the tokens. That ratio is the price of a 57% English mixture — and it is fixed for the life of the model.
The reserved block — chat, reasoning, tool-use, 32 spares and 4,096 audio slots — costs 0.7% of the model and had to exist before pretraining. Added later, those embeddings start from noise while everything else has seen 18B tokens, and never catch up.
Raw text is never written to disk. It streams from HuggingFace, is tokenized in flight, and only uint16 ids are stored — turning a ~500 GB text problem into a 49 GB one.
Each becomes its own .bin of uint16 ids, with an EOS after every document. Batches are capped by characters, not document count — document sizes span four orders of magnitude here, and a fixed 2,000-document batch of Gutenberg is ~225M tokens, enough to overshoot a 2M quota a hundredfold.
Every shard carries the whole mixture. Without interleaving the model sees three days of English, then a day of Hindi — and forgets as it goes. 98 shards, 250M tokens each, 24.5B total.
Every token enters as one of 48,000 ids, becomes a 1024-dimensional vector, and passes through 16 blocks. Each block does two things: attention, which looks at other tokens, and a feed-forward network, which does the actual work.
Filled is what runs for a given token; the outline is what is stored. Almost all of the model is MoE weight, and almost none of it fires — 1,318.8M total against 280.0M active, 4.71× sparse.
This is the whole idea. Each MoE layer holds 48 routed experts plus one shared expert. A router scores the 48 and picks the best four; the shared expert always runs. The other 44 are not touched for that token.
Capacity comes from memory, cost comes from compute. Adding experts raises the parameter count without raising the work per token — which is why a 1.32B model answers on a CPU at ten tokens a second.
It absorbs what every token needs — basic grammar, common patterns — so the routed experts are free to specialise instead of each having to be a generalist.
Routing on raw embeddings is near-random, because the model has not learned anything yet. A router that collapses in the first layer never recovers: a few experts take every token and the rest never train.
Each expert carries a bias added to its routing score; overloaded experts get nudged down, idle ones up. It happens outside the loss, so balance costs the objective nothing. An auxiliary loss fights the thing you are training for.
Load stayed between 1.9% and 2.4% against a 2.1% uniform, with zero dead experts across all 17,166 steps. Collapse is the failure mode this architecture is most exposed to, and it did not happen.
Log-likelihood scoring, 500 examples per task, length-normalised. A results table that lists only wins is not evidence, so the two that went the other way are here too.
| Task | Random | Base | SFT | DPO |
|---|---|---|---|---|
| HellaSwag | 25.0 | 38.4 | 39.8 | 40.4 |
| ARC-easy | 25.0 | 45.0 | 44.8 | 45.0 |
| PIQA | 50.0 | 62.6 | 65.4 | 65.6 |
| WinoGrande | 50.0 | 50.6 | 49.0 | 49.0 |
ARC-easy and PIQA sit well clear of chance, so the model learned real commonsense rather than fluent grammar alone. WinoGrande sits at chance — the pronoun-resolution reasoning it measures never arrived, which is the sharpest statement available of what 0.28B active parameters do not buy.
DPO did not generalise. Its training loop reported 66.25% preference accuracy; on held-out pairs it came out at 47.5% against a 50% baseline, and the benchmark columns agree — SFT and DPO are within noise of each other. The stage cost six hours and bought nothing measurable. It went unnoticed until afterwards because neither sft.py nor dpo.py had any validation at all.
The model runs on the GPU box; this site runs on Vercel. The hard part is not connecting them — it is that the tunnel between them gets a new hostname every time it restarts.
The address is published, not configured. The supervisor writes each new hostname to backend.json in the repo, and the browser reads it at runtime from raw.githubusercontent.com. Baking it into an environment variable meant a redeploy after every restart — and the redeploy is triggered by the very commit carrying the new address, so the gap was guaranteed. When a request fails the cached address is dropped and the next poll re-reads the file, so the site recovers on its own.
Two layers of supervision, each covering what the other cannot. serve.sh health-checks the model server and the public URL, restarting either within ten seconds. systemd watches serve.sh — the one thing it cannot watch itself — and lingering keeps it alive across logout and reboot.
Every one of these was silent — the code ran, produced output, and was wrong. They are the part of the project most likely to be useful to somebody else.
Top-4 routing over 48 experts leaves some experts with no tokens in a micro-batch, so their gradients never arrive and their DDP buckets never become ready. Different ranks skip different experts, so the collectives desync. find_unused_parameters=True is not optional here — and it was present in train.py but missing from sft.py and dpo.py, which is where it surfaced.
DPO scores a chosen and a rejected answer before a single backward pass. DDP marks each parameter ready when its gradient arrives, so two forwards into one backward marks everything twice and aborts. The fix is what the reference implementations do: concatenate both candidates along the batch axis and forward once.
The model emitted <|end_turn|> three tokens in with probability 0.83, so every reply was a fragment. Blocking the stop token for the first 32 steps forces it to elaborate. Separately, top_p was accepted as an argument and never applied — every reply was really plain top-k 50.
Two real bugs, and neither was the cause. The dump streams alphabetically, so the first 500k articles were A–C: Albert Einstein was there, Tokyo was not. Shuffling fixed that and the number did not move, because the median article is ~1,300 characters and a random sample is almost all stubs. What remained is arithmetic — any few hundred thousand articles is 2–6% of Wikipedia. Retrieval over uploaded documents shipped instead, where coverage is total by construction.
The chat app styled header, main and aside as bare element selectors, and the stylesheet is shared. header { display: flex } turned the landing page's header into a row and laid the nav and the hero side by side, vertically centred, halfway down the page.
The supervisor restarted the server forever while health checks passed: an orphan held port 8000, so every new server exited with EADDRINUSE while the orphan answered the probe. Health checks also ran ten seconds after start, killing each server mid-load of its 5.3 GB of weights. And a quick tunnel can hang after its preflight without ever registering — a live process behind a hostname that resolves to nothing.
Upload a document and it answers from that, with the passage it used shown underneath. No account needed.
Open the chat