Verse 3
A bigger, harder, completely from-scratch language-model build. The target is roughly 7 billion parameters, with FSDP making multi-GPU training practical.
The next jump
is not small.
From scratch
No API wrapper and no simple fine-tune. Verse 3 keeps the from-scratch philosophy: understand the architecture, training system and failure modes by building them.
~7B parameters
The architecture will be scaled by validating layers, model width, heads and FFN dimensions rather than blindly copying a reference configuration.
FSDP
Parameters, gradients and optimizer state are planned to be sharded across GPUs so the full model does not need to be replicated on every device.
Same DNA.
More scale.
Verse 3 stays in the same architectural family as the existing model: decoder-only Transformer, RoPE, RMSNorm and SwiGLU. The major change is scale plus the training infrastructure needed to support it.
RoPE
Rotary positional embeddings.
RMSNorm
Normalization stays familiar.
SwiGLU
Feed-forward capacity scales with the model.
140 billion
tokens.
≈ 140,000,000,000 training tokens
The plan is to scale the existing FineWeb-Edu pipeline. At this size, storage and throughput become real engineering constraints; disk capacity and hardware will be checked before a large run starts.
Plan first.
Then earn the numbers.
01 — Validate the 7B configuration
Calculate the parameter count and memory requirements before committing to a long run.
02 — Add opt-in FSDP
Keep the existing 2B path working while adding sharded training, mixed precision and tested checkpoint handling.
03 — Prove training works
Run real training and verify that loss decreases instead of treating a successful launch as proof.
04 — Benchmark the checkpoint
Run the WikiText-103 evaluation and publish the measured perplexity. The ~9 figure remains a target until that happens.
No pretending.
FSDP should make the 7B tier practical — if the node fits.
The actual GPU count, VRAM and communication overhead will be calculated against the rented multi-GPU node before training.
~9 perplexity is not a promise.
The target is only a target. The public result will be whatever the trained checkpoint actually measures.
Verse 3 is being built.
The page is a living plan for the next model. As the work moves from architecture → training → benchmark, the plan will become a record of what actually happened.
Follow the build on GitHub ↗