← FireHacker

Writing

Essays and deep-dives on BubblSpace training runs, profiling, and open AI research.

21 November 2026 · 8 min read

Phase 2 pre-flight: profiling showed data loading, not the document mask

Before launching Phase 2 of BubblSpace MoE-0.9B-A0.2B on a single H100, I profiled one training step: the new document mask costs ~3.7%, while a serial NFS-bound data loader idled the GPU for ~1.37 s every step. A background prefetch thread took throughput from ~69k to ~85.5k tok/s, cutting the projected run from ~48h to ~39h without changing what the model sees.