Verse 1
Uncensored
A 2,023,822,080-parameter decoder-only Transformer built from the ground up in PyTorch: model architecture, tokenizer, data pipeline, training loop, checkpoints, evaluation and chat interface.
The model itself,
not an API wrapper.
Transformer architecture
A hand-written decoder-only Transformer in PyTorch using self-attention, rotary position embeddings, RMSNorm, SwiGLU and residual connections.
Fresh BPE tokenizer
The project trains its own tokenizer on the supplied corpus rather than borrowing the tokenizer of a pretrained model.
Pretraining pipeline
The training system handles next-token prediction, optimization, scheduling, checkpoints and multi-GPU gradient synchronization.
Instruction stage
An optional second training stage can teach a completed base checkpoint to respond to instruction/response examples.
Evaluation
The repository includes evaluation for loss/perplexity and generated samples rather than treating compilation as proof of model quality.
Local interface
Once a checkpoint is trained, the chat script loads it for local text generation.
Every stage is visible.
29 blocks.
2.02 billion parameters.
Each Transformer block follows the same core path. Stack 29 of them at a model width of 2,304 and the configured network reaches 2,023,822,080 trainable parameters.
Built does not mean
fully trained.
The thing stopping the full run is compute.
The repository and architecture are built, and the pipeline has been smoke-tested at a tiny scale. But training the full 2.02B-parameter model toward the intended 20-billion-token run requires serious GPU time. I don't currently have the budget to pay for that full training run, so I am not going to pretend a random-weight or tiny-test checkpoint is a finished 2B model.
Status: architecture ready · full-scale pretraining waiting on computeTest the pipeline
before spending money.
Model + tokenizer pipeline
The repository contains the complete path from corpus preparation to a freshly trained tokenizer and model construction.
Checkpoint bugs fixed
The training code now creates checkpoint directories correctly, accepts the tokenizer's real vocabulary size and respects configured gradient clipping.
Tiny end-to-end smoke test
A deliberately shrunk CPU configuration was used to verify training, checkpoint saving and loading, and generation. That validates the software path — not the quality of the untrained full 2B model.
Full 20B-token run
Still pending. This is the expensive part, and any future loss/perplexity claims will come from a real trained checkpoint.
The code is public.
The repository contains the model, configuration, tokenizer, data utilities, training loop, instruction-training stage, evaluation and chat interface. You can inspect the actual build instead of taking the portfolio page at face value.
View Verse 1 on GitHub ↗See what comes next →