BubblSpace MoE-0.9B-A0.2B

The training story

From the first working debug run to 10 billion tokens of English pretraining and the first Phase 2 results. Follow the decisions, failures, and measurements that shaped the model.

  1. Model build

    From spec to a working model

    The architecture, tokenizer, and training loop came together. Four bugs stood between the spec and a passing run on the H100.

    H100 debug run · recorded results
    Steps complete
    3,051 / 3,051
    Throughput
    220–250k tok/s
    Resume test
    Passed
  2. Phase 1

    The 10B-token English base

    We turned FineWeb-Edu into 100 shards, trained a base model on one H100, and released the checkpoint. Then we tested what its fluent completions actually meant.

    Phase 1 completion excerpt
    “Washington, DC.” … “This city is in California.”

    Released checkpoint · step_19073

  3. Phase 2 design

    Designing the next training run

    Three revisions to the data mix turned the weaknesses of the English base into a plan for learning code, tool calls, and entity relationships.

    Planned share by token group

    Main run · 10B

    Anneal · 2B

    • English and STEM41.5% to 34%
    • Code49% to 38%
    • Tool use8% to 22%
    • Entity binding1.5% to 6%
  4. Phase 2 preflight

    The GPU was waiting

    The first smoke test was slower than expected. A profile changed the diagnosis, and one background thread recovered most of the lost throughput.

    Smoke-test throughput
    Serial loader69–71k tok/s
    Background prefetch85.5k tok/s

    Projected full run · ~48h before, ~39h after

  5. Phase 2 launch

    What the first 7,000 steps showed

    Code and tool-call scores began to move. English benchmarks slipped, and the binding probes gave us a harder question to take into the anneal.

    Phase 1 final compared with early Phase 2
    Code output prediction · step 5k
    0.0% to 8.5%
    Valid call with expected tool · step 7k
    0% to 48%
    English benchmark average · step 5k
    47.18 to 45.40

    Binding probes showed no clear gain.

Follow the work