plan(policy-quality): Phase 2 plan refinements (tolerance bands + decision tables)
Three polish items from the plan review: 1. Task 2.4 Step 5 — replaced vague "must not regress" with a numeric tolerance-band table: multi_fold best_val_metric ±15% per fold, Best Sharpe ≥ 20 floor, and hard F_Half/F_Full ≥ 0.05 + cf_flip ≥ 0.1 BLOCKERS. Anchors to the baseline metrics doc values captured at policy-quality-baseline. 2. Task 2.6 — converted the ad-hoc text decision at Step 1 into the same five-row decision table format Task 2.1 uses (DELETE / KEEP / KEEP-as- safety-net / INCONCLUSIVE / DELETE-because-inseparable). Step 2 ablation got its own 4-row outcome table keyed to ent_mag delta, multi-fold Sharpe regression, and NaN appearance. 3. Task 2.7 Step 1 — added "Repeat 3× with different seeds" clause and explicit total wall-clock note: ~75 min (3 × 25 min) for the ablation pass before the Step 2 decision. Aligns with the Cross-cutting concern #6 about sample-noise rejection. No task count change (still 11: 2.0–2.10). Net code-delta estimate unchanged. Standing-rule compliance unchanged (no stubs / no atomics / no quickfixes / no hiding / no feature flags / no push-per-task).
This commit is contained in:
jgrusewski
@@ -652,7 +652,21 @@ for t in magnitude_distribution reward_component_audit controller_activity \
|
||||
"trainers::dqn::smoke_tests::$t" --ignored --nocapture 2>&1 | tail -5
|
||||
done
|
||||
```
|
||||
All five must pass. multi_fold_convergence in particular must still show ≥ 2 / 3 folds with Best Sharpe > 0 — if it regresses after Task 2.4, the relocation to C51 target smoothing is mis-calibrated; revisit Step 1.
|
||||
All five must pass. Explicit numeric tolerance band against `policy-quality-baseline`:
|
||||
|
||||
| Metric | Baseline | Tolerance | Action on breach |
|
||||
|---|---|---|---|
|
||||
| `multi_fold_convergence` Best val_metric, fold 0 | 0.028 | ±15% | revisit Step 1 |
|
||||
| `multi_fold_convergence` Best val_metric, fold 1 | 0.018 | ±15% | revisit Step 1 |
|
||||
| `multi_fold_convergence` Best val_metric, fold 2 | 0.019 | ±15% | revisit Step 1 |
|
||||
| `multi_fold_convergence` Best Sharpe, any fold | 51 / 39 / 87 | ≥ 20 (conservative floor) | revisit Step 1 |
|
||||
| `magnitude_distribution` F_Half | 0.150 | ≥ 0.05 (smoke threshold) | BLOCK — a reward-kernel change that breaks magnitude diversity is unshippable |
|
||||
| `magnitude_distribution` F_Full | 0.255 | ≥ 0.05 | BLOCK |
|
||||
| `reward_component_audit` cf_flip | 0.619 | ≥ 0.1 | BLOCK (cf-flip is the dominant remaining reward contributor) |
|
||||
|
||||
Any metric outside its tolerance band → the C51 target-smoothing relocation is mis-calibrated. Root-cause the mis-match (is the `-10 * (1 - exp(target/10))` shape too aggressive on rare-large-negative targets? Is it being applied twice? Is target_v_range clamp interacting?) before re-committing.
|
||||
|
||||
Baseline values are from `docs/superpowers/specs/2026-04-21-policy-quality-baseline-metrics.md`; re-consult for exact figures if baseline has advanced.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
@@ -981,17 +995,39 @@ From `/tmp/foxhunt_smoke/magnitude_distribution_grad_split.log`:
|
||||
```bash
|
||||
grep "grad_split" /tmp/foxhunt_smoke/magnitude_distribution_grad_split.log \
|
||||
| awk -F'c51=' '{print $2}' | awk '{print $1}' | sort -n | head -10
|
||||
# Compute mean across epochs:
|
||||
grep "grad_split" /tmp/foxhunt_smoke/magnitude_distribution_grad_split.log \
|
||||
| awk -F'c51=' '{print $2}' | awk '{print $1}' \
|
||||
| awk '{s+=$1; n++} END {if (n) print "mean_c51="s/n}'
|
||||
# Also capture iqn/cql/ens for the ablation path decision:
|
||||
grep "grad_split" /tmp/foxhunt_smoke/magnitude_distribution_grad_split.log \
|
||||
| awk -F'iqn=' '{print $2}' | awk '{s+=$1; n++} END {if (n) print "mean_iqn="s/n}'
|
||||
```
|
||||
|
||||
Decision:
|
||||
- If `grad_mag_c51` mean > 0.1 across epochs → C51 (including its entropy term) IS sending gradient to magnitude; entropy regularization is plausibly contributing. **Re-measure by disabling only the entropy term**, not the whole C51 path — see Step 2.
|
||||
- If `grad_mag_c51` mean < 0.01 → C51 path is nearly dead to magnitude; entropy term within it cannot be doing useful work. **DELETE.**
|
||||
**Decision table** (mean `grad_mag_c51` ratio across 20 epochs is the keystone input):
|
||||
|
||||
- [ ] **Step 2: (Conditional) ablate entropy with a single-variable comparison**
|
||||
| Task 2.0 observation | E4 verdict | Action |
|
||||
|---|---|---|
|
||||
| `grad_mag_c51 < 0.01` across 20/20 epochs | **DELETE** | Skip Step 2, go to Step 3 |
|
||||
| `grad_mag_c51 > 0.3` across most epochs AND H4 fix already spread magnitude Q-values | **KEEP** | Skip Step 2, go to Step 4 (document evidence) |
|
||||
| `grad_mag_c51 ∈ [0.01, 0.3]` OR mixed-signal across epochs | **INCONCLUSIVE** | Run Step 2 ablation |
|
||||
| `grad_mag_c51 > 0.1` AND Task 2.1 Branch A/B landed but magnitude still collapsed | **KEEP** as safety net, reclassify as `entropy_is_load-bearing` | Step 4 with caveat documentation |
|
||||
| `grad_mag_c51` ≈ `grad_mag_iqn` (entropy term drowned in IQN bulk) | **DELETE** — contribution is not separable from IQN | Step 3 |
|
||||
|
||||
Only if Step 1 is inconclusive: run the 5-smoke suite with `entropy_coefficient = 0.0` in `dqn-smoketest.toml`, compare `ent_mag` trajectory across 20 epochs against baseline. If `ent_mag` drops > 0.05 on any fold, KEEP (entropy is load-bearing). Otherwise DELETE.
|
||||
Populate `/tmp/policy-quality-task26-verdict.txt` with the chosen row for reference in the commit message.
|
||||
|
||||
This ablation is ~12 min local (3 folds × 20 epochs on RTX 3050). Do NOT commit the `entropy_coefficient = 0.0` TOML change — revert after the comparison.
|
||||
- [ ] **Step 2: (Conditional — only if Step 1 verdict is INCONCLUSIVE) ablate entropy with a single-variable comparison**
|
||||
|
||||
Run the 5-smoke suite with `entropy_coefficient = 0.0` in `dqn-smoketest.toml`. Compare `ent_mag` trajectory across 20 epochs against the `policy-quality-baseline` tag. Ablation decision table:
|
||||
|
||||
| Ablation outcome | Verdict | Rationale |
|
||||
|---|---|---|
|
||||
| `ent_mag` drops > 0.05 on any fold | **KEEP** | Entropy is doing measurable work; don't silently remove it |
|
||||
| `ent_mag` unchanged (±0.02) | **DELETE** | Redundant with NoisyNets + count bonus |
|
||||
| Multi-fold `Best Sharpe` regresses > 15% | **KEEP** | Even if entropy doesn't visibly move `ent_mag`, it's stabilising learning — keep with commit-message note |
|
||||
| Any fold NaNs | **KEEP + investigate** | Ablation revealed a hidden stability dependency — file as separate bug, do not land DELETE |
|
||||
|
||||
This ablation is ~12 min local (3 folds × 20 epochs on RTX 3050). Do NOT commit the `entropy_coefficient = 0.0` TOML change — revert after the comparison with `git checkout -- config/training/dqn-smoketest.toml`.
|
||||
|
||||
- [ ] **Step 3: (If DELETE) remove the term**
|
||||
|
||||
@@ -1091,12 +1127,22 @@ for t in controller_activity multi_fold_convergence magnitude_distribution; do
|
||||
done
|
||||
```
|
||||
|
||||
Expected runtime: ~25 min total (3 smokes × ~8 min each). Record:
|
||||
Expected runtime per ablation pass: ~25 min (3 smokes × ~8 min each). Record:
|
||||
- `controller_activity` final `[CTRL_FIRE]` line — the ablated grad_clip column should be `0.000`.
|
||||
- `multi_fold_convergence` Best Sharpe per fold — must pass the ≥ 2 / 3 folds > 0 gate.
|
||||
- `magnitude_distribution` — must pass `F_Half ≥ 0.05` AND `F_Full ≥ 0.05`.
|
||||
|
||||
**`git checkout -- crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs`** to revert the override.
|
||||
**Repeat 3×** with different seeds (set `foxhunt_seed = 1`, `2`, `3` in the smoke helper OR vary via CLI if exposed). A single ablation run can produce a Sharpe fluctuation of ±5-10 purely from CUDA-stream + stochastic-action-sampling noise at smoke scale — three repeats let us distinguish real regression from sample noise. **Total Task 2.7 Step 1 wall-clock: ~75 min (3 × 25 min).**
|
||||
|
||||
Record all 3 runs' Best Sharpe + F_Half + F_Full per fold in a small table before moving to Step 2:
|
||||
|
||||
```
|
||||
run 1: fold 0 Sharpe=<>, fold 1 Sharpe=<>, fold 2 Sharpe=<>; F_Half=<>, F_Full=<>
|
||||
run 2: fold 0 Sharpe=<>, fold 1 Sharpe=<>, fold 2 Sharpe=<>; F_Half=<>, F_Full=<>
|
||||
run 3: fold 0 Sharpe=<>, fold 1 Sharpe=<>, fold 2 Sharpe=<>; F_Half=<>, F_Full=<>
|
||||
```
|
||||
|
||||
**`git checkout -- crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs`** to revert the override after all 3 runs complete.
|
||||
|
||||
- [ ] **Step 2: Interpret the data**
|
||||
|
||||
|
||||
Reference in New Issue
Block a user