18 November 2026·Evals

Mid-training eval trajectory for BubblSpace MoE-0.9B-A0.2B (steps 7,000–9,000)

Across 2,000 steps of FineWeb-Edu pretraining (~37% to ~47% of the 10B-token run), LAMBADA perplexity dropped monotonically from 58.1 to 49.3 while PIQA rose to 65.9%.

BubblSpace MoE-0.9B-A0.2B is a base-pretrained (non-instruction-tuned) sparse model (~915M total / ~211M active params) evaluated with lm-evaluation-harness on loglikelihood-scored multiple-choice and continuation tasks.

We checkpointed and evaluated the 10B-token FineWeb-Edu pretraining run at steps 7,000, 8,000, and 9,000 out of 19,073 total planned steps (global batch size 524,288 tokens, sequence length 2,048). Each checkpoint is scored with lm-evaluation-harness without instruction tuning.

Checkpoint progression

TaskMetricstep_7000step_8000step_9000
PIQAacc_norm64.7%64.5%65.9%
ARC-Easyacc_norm51.1%50.7%50.8%
HellaSwagacc_norm37.0%38.0%38.5%
LAMBADA (openai)acc29.0%31.0%30.6%
LAMBADA (openai)perplexity58.150.349.3
WinoGrandeacc52.4%51.7%51.5%
ARC-Challengeacc_norm28.0%27.6%27.2%

Observations

  • LAMBADA perplexity improved monotonically across every 1,000-step window (58.1 -> 50.3 -> 49.3), a 15% reduction relative to step_7000.
  • PIQA moved +1.4 points between step_8000 and step_9000 (clear of its ~1.1% standard error), and HellaSwag climbed steadily from 37.0% to 38.5%.
  • Reasoning-heavy benchmarks (ARC-Challenge and WinoGrande) remain flat near random baseline, which is expected at 211M active parameters and ~4.7B tokens seen.