18 November 2026·Evals
Mid-training eval trajectory for BubblSpace MoE-0.9B-A0.2B (steps 7,000–9,000)
Across 2,000 steps of FineWeb-Edu pretraining (~37% to ~47% of the 10B-token run), LAMBADA perplexity dropped monotonically from 58.1 to 49.3 while PIQA rose to 65.9%.
BubblSpace MoE-0.9B-A0.2B is a base-pretrained (non-instruction-tuned) sparse model (~915M total / ~211M active params) evaluated with lm-evaluation-harness on loglikelihood-scored multiple-choice and continuation tasks.
We checkpointed and evaluated the 10B-token FineWeb-Edu pretraining run at steps 7,000, 8,000, and 9,000 out of 19,073 total planned steps (global batch size 524,288 tokens, sequence length 2,048). Each checkpoint is scored with lm-evaluation-harness without instruction tuning.
Checkpoint progression
| Task | Metric | step_7000 | step_8000 | step_9000 |
|---|---|---|---|---|
| PIQA | acc_norm | 64.7% | 64.5% | 65.9% |
| ARC-Easy | acc_norm | 51.1% | 50.7% | 50.8% |
| HellaSwag | acc_norm | 37.0% | 38.0% | 38.5% |
| LAMBADA (openai) | acc | 29.0% | 31.0% | 30.6% |
| LAMBADA (openai) | perplexity | 58.1 | 50.3 | 49.3 |
| WinoGrande | acc | 52.4% | 51.7% | 51.5% |
| ARC-Challenge | acc_norm | 28.0% | 27.6% | 27.2% |
Observations
- LAMBADA perplexity improved monotonically across every 1,000-step window (58.1 -> 50.3 -> 49.3), a 15% reduction relative to step_7000.
- PIQA moved +1.4 points between step_8000 and step_9000 (clear of its ~1.1% standard error), and HellaSwag climbed steadily from 37.0% to 38.5%.
- Reasoning-heavy benchmarks (ARC-Challenge and WinoGrande) remain flat near random baseline, which is expected at 211M active parameters and ~4.7B tokens seen.