सूत्र — a thread, a formula
SUTRA

Built from nothing. No pretrained weights. No fine-tune of someone else's model. A 1.32B-parameter language model, trained solo.

01 — Scale
0.00B Total parameters
0.00B Active per token
Sparsity  4.7×
02 — Architecture
Multi-head Latent Attention
2,048
288
Bytes cached / token
48 routed + 1 shared
KV cache  7.1× smaller Routing  top-4 of 48

Auxiliary-loss-free expert balancing, implemented from the papers — nothing borrowed off the shelf.

03 — Forged
Five stages, one pipeline
Tokenizer48K BPE
Data18B tokens
Pretrain4d 9h
SFT18h
DPO6h
4× RTX 6000 Ada GPUs 18,000,000,000 tokens
04 — Proof
Measured, not claimed
PIQA65.6%
ARC-easy45.0%

WinoGrande still sits at chance. We're not hiding that — 0.28B active parameters don't buy pronoun-resolution reasoning yet.

Talk to it.
Open the chat Read the code Get the weights

Free API. No signup to start.

SUTRA-1.3B · MIXTURE-OF-EXPERTS · TRAINED FROM SCRATCH