Phase 2 launch
What the first 7,000 steps showed
Code and tool-call scores began to move. English benchmarks slipped, and the binding probes gave us a harder question to take into the anneal.
- Code output prediction · step 5k
- 0.0% to 8.5%
- Valid call with expected tool · step 7k
- 0% to 48%
- English benchmark average · step 5k
- 47.18 to 45.40
Binding probes showed no clear gain.
Phase 2 preflight established that the loader could keep up with the GPU. This chapter records the launch and the snapshot through roughly step 7,000.
Launch
Launched with your go-ahead at 18:01 UTC, 21 Nov:
python3 scripts/train.py configs/phase2.yaml \
--init_from /path/to/checkpoints/step_19073.pt
It appears in W&B as run moe0-9b-a0-2b-phase02 (lec9ww5h), in the same project and group as Phase 1, with global_step continuing from 19,073 so the two runs line up on one axis.
Mistake: I launched without python -u, so the log file is block-buffered and shows nothing for long stretches. Two monitors expired with no events before I switched to reading metrics from the W&B API. Future launches use -u.
Training health
| Steps | Loss | Load-balance loss | z-loss | grad_norm mean / max | tok/s |
|---|---|---|---|---|---|
| 0–500 (re-warm) | 1.93 | 1.16 | 0.116 | 0.35 / 3.1 | 84.2k |
| 1,000–1,500 | 1.64 | 1.05 | 0.096 | 0.19 / 0.21 | 84.4k |
| 2,500–3,000 | 1.56 | 1.03 | 0.090 | 0.19 / 0.77 | 84.0k |
| 4,000–4,500 | 1.52 | 1.02 | 0.085 | 0.17 / 0.18 | 84.3k |
| 6,500–7,000 | 1.50 | 1.01 | 0.080 | 0.17 / 0.21 | 84.0k |
- No crashes and no restarts.
- A liveness check (every 10, then 40 minutes) never fired.
- The one grad-norm spike (3.1) is the re-warm, and it settled within ~80 steps, as in the smoke test.
- Load-balance loss started at ~1.16 as the router adapted to code and agentic tokens, and is now back near the ideal 1.0.
- At this checkpoint, the projected finish was ~08:00–09:30 UTC, 23 Nov.
Per-pool validation loss (held-out, every 250 steps):
| Group | Step 0 | Step 1,000 | Step 6,750 |
|---|---|---|---|
| Code | 3.51 | 1.35 | 1.16 |
| Agentic | 2.93 | 1.22 | 1.09 |
| English | 2.67 | 2.26 | 2.21 |
| Contrastive | 3.87 | 0.59 | 0.59 |
| FineWeb-Edu alone | 2.50 | 2.65 | 2.76 ↑ |
Reading the router properly
You asked how to read W&B's tokens_per_expert histogram. Explaining it turned up a real flaw in the chart: it sums counts by expert index across all 16 layers. Expert 37 in layer 0 and expert 37 in layer 9 are unrelated, so the chart can look balanced while individual layers are badly skewed, or the reverse.
scripts/analyze_routing.py measures routing per (layer, expert), separately for each data group, on held-out windows. It reports load CV, entropy, starved (<0.1× fair share) and overloaded (>3×) experts, and domain specialization (an expert's share of a domain vs. the rest). It runs on the A10 against relayed checkpoints, so training is never touched.
| Phase 1 final | P2 1k | 3k | 5k | 7k | |
|---|---|---|---|---|---|
| Mean load CV | 0.623 | 0.385 | 0.318 | 0.293 | 0.285 |
| Overloaded experts | 21 | 5 | 0 | 0 | 0 |
| Starved experts | 0 | 2 | 0 | 1 | 0 |
At step 1,000, layer-0 experts 39 and 58 were starved (~0.09× load), and I flagged them as a death risk. They recovered. Since then one layer-0 expert is briefly starved at some checkpoints (17 at step 2,000, 31 at step 5,000), a different one each time. That's rotation, not death.
The analysis also shows real specialization emerging. For example, a layer-1 expert receives code tokens at ~16× the rate of other domains, and a layer-7 expert takes contrastive text at ~12×.
Building the eval suite
Phase 1's six English benchmarks can't see code, tool calls or binding. You asked for more evals for agentic coding. The plan is in eval-plan-phase2.md, and every eval runs on the A10, never on the training machine.
What was built:
scripts/eval_phase2_quick.py, every 1,000 steps:- fill-in-the-middle exact match on held-out code
- agentic tool-call format: emits, closes, valid JSON, name in tools, name = gold, argument keys = gold, ends turn
- reasoning
<think>open/close - fixed greedy prompts for side-by-side inspection
- Probe v3: 70 binding cases whose sentence structures are disjoint from both v2 and the contrastive corpus. If v2 rises and v3 doesn't, the model memorized templates instead of learning to bind.
scripts/eval_phase2_milestone.py: CRUXEval-O (predict a program's output, 2-shot official prompt) and BFCL v3 simple / multiple / parallel, converted to our<tools>/<tool_call>format and scored by AST match. Neither executes model-written code. HumanEval/MBPP do, so they wait for a sandbox.scripts/contamination_check.py: 13-gram overlap of BFCL against the raw ToolMind, Hermes and Glaive files. It found 11 of 603 questions, which are excluded from scoring.
The determinism discovery. While adding the KV cache for faster generation, cached and uncached greedy outputs diverged after a few tokens. That looked like a cache bug. It wasn't. Two identical uncached forward passes also differed by up to ~0.1 in the logits: non-deterministic CUDA kernels introduce tiny float differences, and those flip near-tied top-8 router choices (0–12 tokens per layer got a different expert set between runs). For evals that means the same checkpoint scores differently run to run. src/eval/determinism.py (deterministic algorithms plus a fixed cuBLAS workspace) makes runs bit-identical, and every eval script calls it first. The Phase 1 baseline was re-scored under it: v2 25 → 24/100, the honest number.
The eval pipeline:
H100 (training) A10 (evals)
step_N.pt saved ──► relay_ckpts.sh ──rsync, 40 MB/s cap──► phase2_step_N.pt
│
eval_watch.sh: probes v2/v3, quick, routing
(+ CRUXEval/BFCL, English at milestones)
│
results ► benchmark/phase2/*.json
checkpoint deleted after evals
The bandwidth cap keeps the relay from competing with training's data reads on network storage. After the step-5,000 English result, the pipeline was extended: the English suite now runs at steps 7,000, 9,000, …, 21,000 and the final 22,888, with CRUXEval/BFCL again at 19,000 and the final step.
Results so far
The full tables are in eval-notes.md. In short:
| Phase 1 final | Phase 2 best so far | |
|---|---|---|
| CRUXEval-O pass@1 | 0.0% | 8.5% (step 5k, 94% parsable) |
| BFCL simple / multiple / parallel | 0 / 0 / 0% | 9 / 6 / 0% (step 5k) |
| Agentic: valid call, name = gold | 0% | 48% (step 7k) |
| FIM exact match | 0% | 5% (steps 6k–7k) |
Reasoning: closes <think> |
0% | 0% |
| Probe v2 / v3 | 24 / 4 | 25–31 / 3–6 (noise) |
| English 6-task average | 47.18 | 45.40 (step 5k, −1.78) |
Honest read:
- Code and agentic formats are being learned. These moved off zero on external benchmarks. On held-out episodes, step 1,000 generations went from echoing the user to writing well-formed calls like
<tool_call>{"name": "generate_random_color", "arguments": {}}</tool_call>. - Reasoning opens
<think>and rambles. Many BFCL misses are the model starting a ToolMind-style think block and never reaching the call. - Binding hasn't moved. Contrastive validation loss flattened at ~0.59 by step ~500, so the model fitted the corpus's surface statistics almost immediately, but neither probe shows transfer. The anneal (contrastive ×4) is the last chance within this run. If it fails, binding needs a different approach after Phase 2, not more of the same data.
- English forgetting is real.
- HellaSwag −2.5 and ARC-Easy −3.3 are several standard errors.
- FineWeb-Edu validation loss rose 0.26, while Wikipedia and Cosmopedia improved.
- Part of the gap is the comparison: Phase 1 was measured after its learning rate decayed, and step 5,000 is at peak learning rate. The anneal should recover some of it.
- I first framed this as a decision for you, then corrected it to "watch". When the numbers crossed my own 1-point threshold, I moved it to "measure the trend". The decision point is step 19,000: if the English trend is still falling, Stage 2B restarts from the step-19,000 checkpoint with a small FineWeb-Edu share instead of zero, losing essentially no progress.
Mistakes and lessons
pkill -f/pgrep -fmatched the remote shell itself (three times across Phase 2), so the command killed its own shell. Use bracketed patterns ([t]rain.py) or kill by PID.- Unbuffered logs: use
python -ufor long runs, and monitor through W&B, not the log file. - The KV-cache "bug" was MoE nondeterminism. Measure the baseline's own variance before blaming new code.
- A watcher's exit code isn't a verdict. The eval-queue watcher exited 1 because its final error-grep found no tracebacks. Read the output, not the status.
- Escalation discipline: I first presented the FineWeb question as a decision needed today. When you asked what it rested on, the honest answer was one validation-loss signal and a decision ~20 hours away. Present watch items as watch items.
Where things stood at this point
- Training: step ~7,000 / 22,888, healthy, Stage 2A.
- Next: English trend at 9k, 11k, 13k, 15k, 17k and 19k. The FineWeb decision before step 19,000. The anneal.
- Open questions at this checkpoint:
- HumanEval/MBPP in a Docker sandbox on the A10 (no network, CPU/memory/time limits).
- Fixing the eval's "name in tools" check, which scores below the stricter "name = gold" and so must be mis-parsing one tool-list format.
- The mid-run call on distillation.
- The HF model card shows Phase 2 progress, but the weights on HF are still Phase 1's final checkpoint. Phase 2 weights had yet to be released at this snapshot.