21 November 2026·8 min read·Amardeep Singh Sidhu (FireHacker)

Phase 2 pre-flight: profiling showed data loading, not the document mask

Before launching Phase 2 of BubblSpace MoE-0.9B-A0.2B on a single H100, I profiled one training step: the new document mask costs ~3.7%, while a serial NFS-bound data loader idled the GPU for ~1.37 s every step. A background prefetch thread took throughput from ~69k to ~85.5k tok/s, cutting the projected run from ~48h to ~39h without changing what the model sees.

Status at time of writing: Phase 2 has not launched yet. Everything below happened in the pre-flight window: a 100-step smoke test, a profiling pass, one fix, and a second smoke test. Numbers are medians from those runs on 1× H100 PCIe.

Why this post

Phase 1 of BubblSpace MoE-0.9B-A0.2B (~915M total / ~211M active parameters, 64 experts, top-8) finished its 10B-token FineWeb-Edu run at about 87k tokens/sec, roughly 32 hours on one H100. Phase 2 continues from that checkpoint on a 12B-token mixture of English and STEM text, code, agentic traces and a synthetic contrastive-binding corpus. It also adds three things Phase 1 never had: a causal + same-document attention mask via flex_attention, a sampler that draws every row from one of 16 token pools on a step-dependent schedule, and on-the-fly fill-in-the-middle for code.

The first 100-step smoke test passed every correctness check: weights and optimizer state loaded cleanly from the Phase 1 checkpoint, flex_attention compiled under torch.compile, grad_norm settled through the LR re-warm, and checkpoints saved. But it ran at ~69–71k tok/s. At that speed, 12B tokens is about 48 hours, not the 38–39 I had planned. Before committing two days of H100 time I wanted to know where the missing ~9 hours were going, without changing a single sampled token.

My guess was wrong

My working assumption was that the new document mask was the cost. Phase 1 used PyTorch's fused causal attention (is_causal=True), the fastest path there is. flex_attention is the opposite end of the trade-off: its flexibility is high, since it supports arbitrary combinations of masks and biases via score_mod and mask_mod functions. Ours is a two-line mask_mod (allow kv_idx ≤ q_idx, and only when both tokens share a document id derived from <|endoftext|> positions), compiled into a block-sparse kernel. Flexible usually means slower, so I expected it to be most of the gap. I measured instead of assuming.

The profile

A small script timed each piece of one training step in isolation on the H100 box, with CUDA synchronization around every measurement. One optimizer step is 16 gradient-accumulation micro-batches of 16 × 2048 tokens.

Work (per micro-batch unless noted)Median time
Data loading (get_batch)85.8 ms
Document mask build (create_block_mask)8.0 ms
Forward + backward WITH doc mask (incl. mask build)379.3 ms
Forward + backward WITHOUT mask (Phase 1 path)365.6 ms
Optimizer step, Muon + AdamW (once per step)130 ms

What it said

  • The document mask is nearly free: 379 vs 366 ms is ~3.7%, mask construction included. flex_attention is almost as fast as the fused causal kernel here, so it was not the gap.
  • Data loading is the gap. 86 ms × 16 micro-batches ≈ 1.37 s per step during which the GPU sits idle, because the loader ran serially on the main training thread. That alone accounts for the drop from ~87k to ~69k tok/s.
  • The likely cause is random reads from network-attached storage: each micro-batch now pulls 16 windows from different pools. Phase 1 had the same access pattern but one pool, so it paid less.
  • torch.compile recompiles were not a problem: the ones logged were a one-time specialization per layer index at warm-up (the same code Phase 1 used), not a recompile on every batch.

The profile also gave a ceiling. With data loading fully overlapped with compute, the projected step is ~6.2 s, about 84.6k tok/s, within ~3% of Phase 1, or roughly 39 hours for 12B tokens instead of 48.

The fix: overlap, don't optimize

I didn't try to make the loader faster. I moved it off the critical path. A background thread assembles micro-batches into a bounded queue (depth 8) while the GPU computes; the training loop just pops the next ready batch. One thread is enough because 86 ms of loading per micro-batch is far below 379 ms of compute, and Python's GIL isn't a concern because the thread spends most of its time waiting on network reads, which release it.

python
class BatchPrefetcher:
    def __init__(self, dataset, micro_batch, grad_accum, start_step, total_steps, depth=8):
        self._q = queue.Queue(maxsize=depth)

        def produce():
            # same order the training loop consumes: grad_accum batches per step
            for step in range(start_step, total_steps):
                for _ in range(grad_accum):
                    self._q.put(dataset.get_batch_cpu(micro_batch, step))  # pinned CPU tensors
            self._q.put(DONE)

        threading.Thread(target=produce, daemon=True).start()

    def next(self, device):
        x, y = self._q.get()
        return x.to(device, non_blocking=True), y.to(device, non_blocking=True)

The constraint I cared about most was that the fix must not change what the model sees. The sampler's mixture depends on the step (Stage 2A plateau vs Stage 2B anneal), so the producer walks the steps in exactly the order the training loop consumes them. A unit test checks this directly: with the same seeds, the prefetched batches are bit-for-bit identical, in order, to calling get_batch inline, including across a stage boundary. The full suite (16 tests, including the original Phase 1 tests) passes.

Effect

Smoke test #1 (serial loader)Smoke test #2 (prefetch)
Steady-state throughput~69–71k tok/s~85.5k tok/s
Projected Phase 2 wall-clock (12B tokens)~48h~39h
Loss / val losses / grad_normbaselinetracks smoke #1 (same data, same order)

The measured ~85.5k tok/s lands right on the profile's ~84.6k ceiling, within ~2% of Phase 1's 87k. About 9 hours of H100 time per run went to the GPU waiting on file reads, and none of it showed up as an error, only as a lower number in the logs.

What I'd do again

  • Smoke-test before profiling. The first 100 steps answered the correctness questions (does the checkpoint load, does flex_attention compile, is the re-warm stable) so the profile could focus purely on time.
  • Profile the pieces, not the whole. One end-to-end tok/s number told me something was slow; only the per-component timings told me what, and they contradicted my guess.
  • Make the speed fix provably behavior-neutral. An order-equivalence test is cheap; discovering two days later that a 'speed-up' silently changed the data mixture would not be.

Next

The prefetcher is implemented, unit-tested and smoke-confirmed, and all 16 Phase 2 pools are on the training box with verified token counts. The remaining gate is the launch itself: Phase 2 continues from the Phase 1 checkpoint for 22,888 steps (12B tokens) on one H100, logged to W&B as a new run whose x-axis continues Phase 1's step count. I'll write up what it learns, especially whether the contrastive-binding data moves the entity-binding probe, once there's something real to report.