Model build
From spec to a working model
The architecture, tokenizer, and training loop came together. Four bugs stood between the spec and a passing run on the H100.
- Steps complete
- 3,051 / 3,051
- Throughput
- 220–250k tok/s
- Resume test
- Passed
BubblSpace MoE-0.9B-A0.2B
From the first working debug run to 10 billion tokens of English pretraining and the first Phase 2 results. Follow the decisions, failures, and measurements that shaped the model.
Model build
The architecture, tokenizer, and training loop came together. Four bugs stood between the spec and a passing run on the H100.
Phase 1
We turned FineWeb-Edu into 100 shards, trained a base model on one H100, and released the checkpoint. Then we tested what its fluent completions actually meant.
“Washington, DC.” … “This city is in California.”
Released checkpoint · step_19073
Phase 2 design
Three revisions to the data mix turned the weaknesses of the English base into a plan for learning code, tool calls, and entity relationships.
Main run · 10B
Anneal · 2B
Phase 2 preflight
The first smoke test was slower than expected. A profile changed the diagnosis, and one background thread recovered most of the lost throughput.
Projected full run · ~48h before, ~39h after
Phase 2 launch
Code and tool-call scores began to move. English benchmarks slipped, and the binding probes gave us a harder question to take into the anneal.
Binding probes showed no clear gain.