gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(policy-quality): Task 0.17 baseline metrics scaffold (PENDING GPU run)

Browse Source 
Phase 0 instrumentation is complete (Tasks 0.4 / 0.5 / 0.8 / 0.10 / 0.16
shipped this session; 0.6 partial; 0.3 partial). This scaffold encodes
the full metric set with explicit <PENDING …> markers per row so the
GPU-run capture can fill in values without forgetting any HEALTH_DIAG
field.

The policy-quality-baseline tag is intentionally NOT applied yet —
tagging requires real captured values, not the scaffold. Runner workflow:

  for t in magnitude_distribution reward_component_audit \\
           controller_activity exploration_coverage multi_fold_convergence; do
    SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \\
      FOXHUNT_TEST_DATA=test_data/futures-baseline \\
      cargo test -p ml --release --lib -- \$t --ignored --nocapture 2>&1 | tee out_\$t.log
  done

Then edit this doc, replace markers with values, amend, and tag.
This commit is contained in:
jgrusewski
2026-04-21 23:25:02 +02:00
parent f4ea0f6d13
commit e1ac2c7578
1 changed files with 123 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
docs/superpowers/specs/2026-04-21-policy-quality-baseline-metrics.md Normal file
View File

@@ -0,0 +1,123 @@
# Policy Quality — Phase 0 Baseline Metrics
**Captured against commit:** `f4ea0f6d1` (HEAD of `main` after Phase 0 completion; update before tagging `policy-quality-baseline`)
**Hardware:** RTX 3050 Ti Laptop GPU (local) — for production-baseline numbers re-run on L40S/H100 per Phase 3
**Data:** `test_data/futures-baseline`, ES.FUT
**Smoke-test command template:**
```bash
SQLX_OFFLINE=true CUBLAS_WORKSPACE_CONFIG=:4096:8 \
FOXHUNT_TEST_DATA=test_data/futures-baseline \
cargo test -p ml --release --lib -- <test_name> --ignored --nocapture
```
> **Status: SCAFFOLD ONLY — GPU runs pending.**
> Every `<PENDING …>` marker below must be replaced with the actual captured value before the `policy-quality-baseline` tag is applied. The scaffold encodes the agreed metric set so reviewers can verify nothing was forgotten when the run completes.
---
## Track 1 — Magnitude
Drawn from `magnitude_distribution.rs` smoke test stdout + `[HEALTH_DIAG]` lines emitted by the trainer.
| Metric | Source | Value |
|---|---|---|
| `F_Quarter` (action share) at epoch 20 | `magnitude_distribution` test output | `<PENDING magnitude_distribution>` |
| `F_Half` at epoch 20 | same | `<PENDING magnitude_distribution>` |
| `F_Full` at epoch 20 | same | `<PENDING magnitude_distribution>` |
| `q_mag_quarter` (mean per-epoch) | HEALTH_DIAG `mag [q_quarter=…]` | `<PENDING — Task 0.3 wiring required>` |
| `q_mag_half` | HEALTH_DIAG `mag [q_half=…]` | `<PENDING — Task 0.3 wiring required>` |
| `q_mag_full` | HEALTH_DIAG `mag [q_full=…]` | `<PENDING — Task 0.3 wiring required>` |
| `var_scale_mean` (trajectory, e5/e10/e15/e20) | HEALTH_DIAG `mag [var_scale=…]` | `<PENDING — Task 0.3 wiring required>` |
| `kelly_f_mean` (trajectory) | HEALTH_DIAG `mag [kelly_f=…]` | `<PENDING — Task 0.3 wiring required>` |
| `avg_win_ratio` (trajectory) | HEALTH_DIAG `mag [avg_win_ratio=…]` | `<PENDING — Task 0.3 wiring required>` |
| `grad_ratio_mag_dir` (trajectory) | HEALTH_DIAG `mag [grad_ratio_mag_dir=…]` | `<PENDING — Task 0.4 ✅ wired>` |
| `dist_q` / `dist_h` / `dist_f` (Q-distance per mag) | HEALTH_DIAG `mag [dist_q=… dist_h=… dist_f=…]` | `<PENDING — Task 0.3 wiring required>` |
| `trail_fire_rate[Q,H,F]` per mag | HEALTH_DIAG `trail [fire_q=… fire_h=… fire_f=…]` | `<PENDING — Task 0.5 ✅ wired>` |
| `hold_at_exit_mean[Q,H,F]` per mag | HEALTH_DIAG `trail [hold_q=… hold_h=… hold_f=…]` | `<PENDING — Task 0.5 ✅ wired>` |
| `vsn_mag` / `vsn_dir` | HEALTH_DIAG `noisy [vsn_mag=… vsn_dir=…]` | `<PENDING — Task 0.6 partial; VSN accessor TODO>` |
| `sigma_mag` / `sigma_dir` (NoisyNets σ) | HEALTH_DIAG `noisy [sigma_mag=… sigma_dir=…]` | `<PENDING — Task 0.6 ✅ wired>` |
| `drift_mag` / `drift_dir` (target weight drift) | HEALTH_DIAG `noisy [drift_mag=… drift_dir=…]` | `<PENDING — Task 0.6 partial; drift accessor TODO>` |
| Eval-mode action distribution `eq/eh/ef` | HEALTH_DIAG `eval_dist [eq=… eh=… ef=…]` | `<PENDING — Task 0.7 ✅ wired>` |
## Track 2 — Reward Contribution
Drawn from `reward_component_audit.rs` smoke output + HEALTH_DIAG `reward_contrib`.
| Slot | Term | Class | Baseline value |
|---|---|---|---|
| 0 | popart | transform (variance Δ ratio) | `<PENDING — Task 0.8 ✅ wired (returns 0.0 during PopArt warmup)>` |
| 1 | cf | sample-selector (CF flip rate) | `<PENDING — Task 0.8 ✅ wired>` |
| 2 | trail_r | sample-selector (trail rate) | `<PENDING — Task 0.8 ✅ wired (reuses Task 0.5 buffers)>` |
| 3 | micro | additive (\|Σ micro\| / \|Σ total\|) | `<PENDING — Task 0.8 ✅ wired>` |
| 4 | la | additive (\|Σ asym-clamp Δ\| / \|Σ total\|) | `<PENDING — Task 0.8 ✅ wired>` |
| 5 | seg | additive (\|Σ segment-patience\| / \|Σ total\|) | `<PENDING — Task 0.8 honest stub: term not wired in kernel; slot returns 0.0>` |
## Track 3 — Controller Firing
Drawn from `controller_activity.rs` smoke output + HEALTH_DIAG `controller`.
| Controller | Firing rate (epoch 20) |
|---|---|
| anti_lr | `<PENDING — Task 0.9 ✅ wired>` |
| tau | `<PENDING — Task 0.9 ✅ wired>` |
| gamma | `<PENDING — Task 0.9 ✅ wired>` |
| clip | `<PENDING — Task 0.9 ✅ wired>` |
| cql | `<PENDING — Task 0.9 ✅ wired>` |
| cost | `<PENDING — Task 0.9 ✅ wired>` |
| `fire_frac` aggregate | `<PENDING — Task 0.9 ✅ wired>` |
## Track 4 — Exploration
Drawn from `exploration_coverage.rs` smoke output + HEALTH_DIAG `explore`.
| Metric | epoch 5 | epoch 10 | epoch 15 | epoch 20 |
|---|---|---|---|---|
| `ent_mag` (magnitude entropy) | `<P>` | `<P>` | `<P>` | `<P>` |
| `ent_dir` (direction entropy) | `<P>` | `<P>` | `<P>` | `<P>` |
| `sigma_mean` (overall NoisyNets σ) | `<P>` | `<P>` | `<P>` | `<P>` |
(`<P>` = `<PENDING — Task 0.10 ✅ wired>`)
---
## Smoke test results
| Test | Result | Wall-clock (RTX 3050 Ti) | Notes |
|---|---|---|---|
| `magnitude_distribution` | `<PENDING>` | `<PENDING>` | |
| `reward_component_audit` | `<PENDING>` | `<PENDING>` | Validates infra; real triage from log-grep on multi-fold L40S run (Phase 1) |
| `controller_activity` | `<PENDING>` | `<PENDING>` | |
| `exploration_coverage` | `<PENDING>` | `<PENDING>` | |
| `multi_fold_convergence` | `<PENDING>` | `<PENDING>` | Multi-fold; longest test |
---
## Phase-0 task completion summary
| Task | Status | Commit |
|---|---|---|
| 0.1 Verify starting state | ✅ Complete | (pre-session) |
| 0.2 HEALTH_DIAG scaffolding | ✅ Complete | (pre-session) |
| 0.3 Track 1 magnitude Q + var_scale + kelly | ⚠️ Partial — q_mag/var_scale/kelly_f/avg_win_ratio still stubbed (require GPU readback infra) | (pre-session) |
| 0.4 Per-branch grad norm ratio | ✅ Complete | `bb42c9963` |
| 0.5 Per-magnitude trail fire rate | ✅ Complete (no atomicAdd, per-sample arrays) | `48d962ee2` |
| 0.6 VSN + NoisyNets σ + target drift | ⚠️ Partial — σ wired (`999bb2fa0`); VSN mask + target drift TODO | `999bb2fa0` |
| 0.7 Eval-mode action distribution | ✅ Complete | (pre-session) |
| 0.8 Reward term contribution | ✅ Complete (seg slot honest stub: term not wired) | `bf8415c5d` |
| 0.9 Controller fire-rate tracking | ✅ Complete | (pre-session) |
| 0.10 Exploration entropy + sigma | ✅ Complete | `999bb2fa0` |
| 0.11 `magnitude_distribution` smoke | ✅ Complete | (pre-session) |
| 0.12 `reward_component_audit` smoke | ✅ Complete | `93d8c5ae4` |
| 0.13 `controller_activity` smoke | ✅ Complete | (pre-session) |
| 0.14 `exploration_coverage` smoke | ✅ Complete | (pre-session) |
| 0.15 `multi_fold_convergence` smoke | ✅ Complete | (pre-session) |
| 0.16 `surrogate_noise_check` smoke + CLI extensions | ✅ Complete (no checkpoint required to compile) | `f4ea0f6d1` |
| 0.17 Baseline metrics document | 🟡 Scaffold committed; values pending GPU run | this commit |
**Remaining work for the `policy-quality-baseline` tag**:
1. Run all 5 smoke tests on this commit (`f4ea0f6d1` or HEAD).
2. Replace every `<PENDING …>` marker above with captured values.
3. (Optional, for completeness) Wire the remaining Task 0.3 / 0.6 accessors so the corresponding rows aren't structural stubs.
4. Re-amend this document and `git tag -a policy-quality-baseline -m "Phase 0 complete — instrumentation in, pre-fix reference"`.
The scaffold is intentionally exhaustive so reviewers can verify nothing was dropped when the GPU run is captured.