48 + 1 experts
Top-4 routing with sigmoid scoring and bias-based load balancing. One shared expert always fires, absorbing what every token needs so the routed ones can specialise. Layer 0 stays dense — routing on raw embeddings collapses early.
A 1.32-billion-parameter Mixture-of-Experts model, written from first principles in PyTorch. Own tokenizer, own data pipeline, own training loop — nothing pretrained, nothing inherited.
$ pip install torch tokenizers huggingface_hub
$ wget huggingface.co/Abhisingh-18/Sutra-1.3B-Chat/…/inference.py
$ python inference.py "What is machine learning?"
Sutra-1.3B on cpu: 1.32B total, 0.28B active
Machine learning is a branch of artificial intelligence that deals with the study and prediction of complex data. It involves using algorithms to analyse large amounts of data to make predictions about future outcomes.Four of forty-eight experts run for any given token, so the model carries 1.32B parameters of capacity at 0.28B parameters of compute. That ratio is why it answers on a CPU at ten tokens a second.
Top-4 routing with sigmoid scoring and bias-based load balancing. One shared expert always fires, absorbing what every token needs so the routed ones can specialise. Layer 0 stays dense — routing on raw embeddings collapses early.
Multi-head Latent Attention compresses keys and values into a 256-wide latent before projection, with rotary position carried on a decoupled 32-dimension head split.
Atomic checkpoints, batches that are a deterministic function of step and rank, and a loss-spike guard that rolls back. A four-day run does not finish uninterrupted.
Log-likelihood scoring over 500 examples per task, length-normalised. Reproduce with python -m src.eval --compare.
| Task | Random | Base | SFT | DPO |
|---|---|---|---|---|
| HellaSwag | 25.0 | 38.4 | 39.8 | 40.4 |
| ARC-easy | 25.0 | 45.0 | 44.8 | 45.0 |
| PIQA | 50.0 | 62.6 | 65.4 | 65.6 |
| WinoGrande | 50.0 | 50.6 | 49.0 | 49.0 |
ARC-easy and PIQA sit well clear of chance, so the model learned real commonsense rather than fluent grammar alone. WinoGrande sits at chance — the pronoun-resolution reasoning it measures never arrived, which is the sharpest statement available of what 0.28B active parameters do not buy.
A 48,000-token BPE vocabulary over English and Devanagari, with chat, reasoning and 4,096 audio tokens reserved up front. Added later, their embeddings would start from noise while everything else had seen 18B tokens.
18B tokens across 4× RTX 6000 Ada at 33% MFU. Held-out perplexity 15.00, zero dead experts, and one loss spike across 17,166 steps that recovered on its own.
200,000 conversations over three epochs. Held-out perplexity 5.49, with the third epoch measuring best — so the extra epochs bought quality rather than overfitting.
DPO across 100,000 preference pairs. Held-out accuracy landed at 47.5% against a 50% baseline, so this stage did not generalise. Reported here rather than quietly dropped.
Trained on 18B tokens — roughly 500× less than comparable 1B models. That gap shows up in specific, predictable ways, and pretending otherwise would only waste your time.
Upload a document and it answers from that, with the passage it used shown underneath. No account needed.
Open the chat