design(policy-quality): revision 2 — full 4-track V7 + outside-the-box
Addresses critique from second self-review: 1. SCOPE GAP fixed: spec now covers all 4 V7 categories the user asked for. Added Track 3 (controllers) and Track 4 (exploration) with their own smoke tests (controller_activity, exploration_coverage) and HEALTH_DIAG fields. Previous version only had action-space + reward. 2. H1-H5 expanded to H1-H10 with 5 new codebase-specific hypotheses: - H6: Regime-adaptive trailing stop closing Full-positions early - H7: VSN / NoisyNets masking magnitude features - H8: Target tau too slow for magnitude propagation - H9: Data genuinely favors Quarter (no bug, delete magnitude branch) - H10: Entropy regularization + argmax tie ordering bias 3. H3 REFRAMED from "Kelly bug" to design critique — Kelly as hard multiplier vs soft gate. Fix sketch updated to offer replacement. 4. Feature branch + tag strategy replaces "single branch" containment. feat/policy-quality isolates work from main until Phase 4 merge via PR. git tag policy-quality-baseline gives named rollback anchor. 5. SURROGATE-NOISE CHECK added as MANDATORY validation gate — catches look-ahead/leakage bugs that would otherwise produce false-positive edges. Permanent smoke artifact. 6. Validation gate restructured: B + surrogate-noise MANDATORY; A criteria soft; controller-activity soft. Partial success path explicit per-gate. 7. Risk register expanded — added scenarios for B-mandatory failure (feat branch stays unmerged), all-10-hypotheses-REJECTED (fall back to H9 delete-branch), surrogate-noise failure (halt spec, open diagnostic spec), multi-load-bearing-controllers (tech debt doc). 8. Budget corrected to 5.5-6.5 hrs L40S (was underestimated 3.5-4), with per-track breakdown. Spec now: 464 lines, 10 sections, covers 4 V7 tracks × ~10 hypotheses + 8 reward terms + 7 controllers + 5 exploration mechanisms = ~30 audit items total.
This commit is contained in:
jgrusewski
@@ -53,42 +53,53 @@ Failing a gate does not trigger a revert. A follow-up spec (`docs/superpowers/sp
|
||||
|
||||
## 3. Architecture
|
||||
|
||||
Single branch. Phase-gated. Two parallel investigation tracks that merge into one convergence commit.
|
||||
Feature-branch–isolated. Phase-gated. **Four** parallel V7 investigation tracks (honoring the systematic-across-all-categories scope) that merge into one convergence commit on the feature branch, validated, then merged to main via PR.
|
||||
|
||||
```
|
||||
main timeline:
|
||||
feat/policy-quality branch:
|
||||
|
||||
[Phase 0] commit: measurement substrate
|
||||
[Phase 0] commit: measurement substrate. Tagged policy-quality-baseline.
|
||||
↓ (baseline run against this commit — reference metrics frozen)
|
||||
|
||||
[Phase 1] NO commits. Two parallel investigation tracks in local worktree
|
||||
or throwaway branches. Findings documented in spec follow-up.
|
||||
[Phase 1] NO commits on feat/*. Four parallel investigation tracks live
|
||||
on throwaway branches wip/pq-track{1,2,3,4}, pushed to origin
|
||||
for crash safety.
|
||||
↓
|
||||
|
||||
[Phase 2] commit: single convergence fix. Reward changes + magnitude sizer
|
||||
fix + deleted sub-noise terms. All confirmed bugs fixed in one
|
||||
commit.
|
||||
[Phase 2] commit: single convergence fix on feat/policy-quality.
|
||||
All confirmed bugs across all 4 tracks fix together.
|
||||
↓
|
||||
|
||||
[Phase 3] commit(s): validation tweaks from 6-fold run. Success-criteria
|
||||
gate checked. Follow-up notes if partial.
|
||||
[Phase 3] commit(s) on feat/*: validation tweaks from 6-fold run.
|
||||
Success-criteria gate checked.
|
||||
↓
|
||||
|
||||
[Phase 4] Merge feat/policy-quality → main via PR. Only on all gates green.
|
||||
If validation fails, feature branch stays, main untouched.
|
||||
```
|
||||
|
||||
**Key architectural decisions:**
|
||||
|
||||
- **Feature branch for isolation.** Every change lives on `feat/policy-quality` until Phase 4 merge. `main` stays clean until validation passes. Revert is "don't merge."
|
||||
- **Measurement first.** No fix lands until its detection is codified in a smoke test.
|
||||
- **No mid-investigation commits.** Phase 1 is exploratory; anything that lands pollutes the Phase 0 baseline.
|
||||
- **No mid-investigation commits on feature branch.** Phase 1 is exploratory; only the convergence commit lands on `feat/policy-quality`.
|
||||
- **All confirmed findings fix together.** No ranking, no "most likely cause" — V7 discipline gates inclusion (measurement confirms a real bug), every confirmed bug ships in Phase 2.
|
||||
- **Greenfield checkpoint policy.** Any action-space or weight-shape change is allowed. Existing `.safetensors` artifacts are training-only, no one is running them in production.
|
||||
- **6-fold × 50-epoch L40S gate.** No "production policy quality" claim without that evidence.
|
||||
- **Tagged baseline for rollback.** `git tag policy-quality-baseline` at Phase 0. If merged-to-main code ever regresses, `git reset --hard policy-quality-baseline` restores known state.
|
||||
|
||||
**Budget estimate:** ~3–5 days elapsed.
|
||||
**Budget estimate:** ~4–6 days elapsed.
|
||||
|
||||
- L40S validation runs: ~1 hour each, budget 3 runs = **3 hrs** (one after Phase 2, up to 2 more to iterate if partial)
|
||||
- Local smoke runs (RTX 3050): ~3 min each, budget unlimited
|
||||
- Investigation HEALTH_DIAG runs during Phase 1 use the same Argo workflows, 20-epoch × 2-fold = ~12 min on L40S, budget 3 = ~30 min
|
||||
- Investigation runs during Phase 1 — 4 tracks × 2-3 dedicated measurement runs each:
|
||||
- Track 1 (magnitude): forced-exploration epoch experiment (~1 hr run) + baseline (~12 min) + per-hypothesis follow-up = ~1.5 hr
|
||||
- Track 2 (reward audit): 1 instrumented 20-epoch run (~12 min) is enough to measure all 8 terms
|
||||
- Track 3 (controllers): 1 instrumented run + 1 comparison run with selected controllers disabled = ~25 min
|
||||
- Track 4 (exploration): 1 instrumented run (~12 min)
|
||||
- Total investigation GPU: **~2.5 hrs**
|
||||
|
||||
Total GPU: **~3.5–4 hrs L40S** (not 6–10 — earlier estimate was conservative).
|
||||
Total GPU: **~5.5–6.5 hrs L40S**.
|
||||
|
||||
**Target GPU pool:** `ci-training-l40s` throughout. H100 only if L40S queue saturates (no expected architecture-specific differences).
|
||||
|
||||
@@ -116,10 +127,25 @@ All live under `crates/ml/src/trainers/dqn/smoke_tests/`. Each is `#[ignore]`-ga
|
||||
- Terms below their class's threshold are triaged for DELETE in Phase 2.
|
||||
- Implementation detail: the test uses one training run with instrumentation, not 8 separate runs.
|
||||
|
||||
**`controller_activity.rs`** — Track 3 gate
|
||||
- Train 20 epochs, record per-controller firing rate.
|
||||
- Assert: no single controller (anti-LR, adaptive-tau, adaptive-gamma, grad-clip, CQL-alpha, cost-anneal) fires in > 50% of epochs. High firing rate = load-bearing → production risk.
|
||||
|
||||
**`exploration_coverage.rs`** — Track 4 gate
|
||||
- Train 20 epochs, record per-epoch magnitude-branch action entropy.
|
||||
- Assert: entropy ≥ 0.5 at epoch 5, ≥ 0.3 at epoch 20. Collapse-speed check — if magnitude entropy hits 0 before epoch 20, something is quenching exploration.
|
||||
|
||||
**`multi_fold_convergence.rs`** — fast local variant of the full gate
|
||||
- 3 folds × 20 epochs (vs 6 × 50 on L40S). Runs ~5 min on RTX 3050.
|
||||
- Assert: all 3 folds complete without NaN, Best Sharpe > 0 on at least 2/3.
|
||||
|
||||
**`surrogate_noise_check.rs`** — false-positive detector
|
||||
- Use the trained model from Phase 2 or Phase 3.
|
||||
- On validation data, REPLACE model actions with uniformly random actions (same distribution shape as trained model).
|
||||
- Run backtest with random actions.
|
||||
- Assert: random-action Sharpe ≤ 0.5× trained-model Sharpe. If random actions produce comparable Sharpe, we have measurement/leakage bugs (look-ahead, data re-use, stale features) — not a real edge.
|
||||
- Critical production-readiness gate. Permanent artifact — guards against future measurement regressions.
|
||||
|
||||
### 4.2 HEALTH_DIAG extensions
|
||||
|
||||
Add to the existing HEALTH_DIAG log line in `training_loop.rs`. All forward-only, no backward gradient, cost is one scalar readback per epoch per field.
|
||||
@@ -132,7 +158,16 @@ Track 1 fields (magnitude diagnosis):
|
||||
- `action_dist_quarter`, `action_dist_half`, `action_dist_full` — epoch-level usage fractions
|
||||
|
||||
Track 2 fields (reward V7 audit):
|
||||
- `reward_contrib_popart`, `reward_contrib_cf_flip`, `reward_contrib_trail`, `reward_contrib_micro`, `reward_contrib_loss_aversion`, `reward_contrib_segment_patience` — per-term magnitude as fraction of step_return
|
||||
- `reward_contrib_popart`, `reward_contrib_cf_flip`, `reward_contrib_trail`, `reward_contrib_micro`, `reward_contrib_loss_aversion`, `reward_contrib_segment_patience` — per-term magnitude as fraction of step_return (see class-specific measurement in §4.1)
|
||||
|
||||
Track 3 fields (controllers V7 audit):
|
||||
- `controller_fire_{anti_lr, adaptive_tau, adaptive_gamma, grad_clip, cql_alpha, cost_anneal}` — boolean per epoch
|
||||
- `controllers_firing_fraction` — aggregate: fraction of epochs where ANY controller fired non-trivially
|
||||
|
||||
Track 4 fields (exploration V7 audit):
|
||||
- `explore_entropy_magnitude` — action-entropy restricted to magnitude branch
|
||||
- `explore_entropy_direction` — same for direction branch (control variable)
|
||||
- `noisy_net_sigma_mean` — average NoisyNets σ across weights
|
||||
|
||||
### 4.3 Phase 0 exit criteria
|
||||
|
||||
@@ -146,31 +181,41 @@ Track 2 fields (reward V7 audit):
|
||||
|
||||
**Branch discipline during Phase 1:**
|
||||
|
||||
- No commits land on `main`.
|
||||
- Each track uses a throwaway local branch (`wip/policy-quality-track1`, `wip/policy-quality-track2`) that IS pushed to origin as backup — losing a multi-day investigation to a disk crash is unacceptable.
|
||||
- Track branches are never merged. At Phase 2, findings are re-applied by hand on a fresh `main` branch, and the throwaway branches are deleted.
|
||||
- No commits land on `main` OR on `feat/policy-quality`.
|
||||
- Each track uses a throwaway local branch (`wip/pq-track1-magnitude`, `wip/pq-track2-reward`, `wip/pq-track3-controllers`, `wip/pq-track4-exploration`) pushed to origin as backup — losing a multi-day investigation to a disk crash is unacceptable.
|
||||
- Track branches are never merged. At Phase 2, findings are re-applied on `feat/policy-quality` as a single commit, and the wip branches are deleted.
|
||||
|
||||
This gives us crash safety (branches pushed) + baseline cleanliness (nothing lands on main).
|
||||
This gives us crash safety (branches pushed) + feature-branch cleanliness (nothing lands on `feat/policy-quality` until Phase 2).
|
||||
|
||||
### 5.1 Track 1 — Magnitude collapse diagnosis
|
||||
|
||||
Five candidate root-cause hypotheses. Each gets a **CONFIRMED** or **REJECTED** verdict based on the measurement signal.
|
||||
Ten candidate root-cause hypotheses. Each gets a **CONFIRMED** or **REJECTED** verdict based on the measurement signal. H1–H5 are common RL-trading causes; H6–H10 are codebase-specific possibilities surfaced in spec review.
|
||||
|
||||
| # | Hypothesis | Detection signal | Detection threshold |
|
||||
|---|---|---|---|
|
||||
| H1 | Reward gradient asymmetry — Full-position reward variance is higher than Quarter's; policy correctly avoids high-variance bins. | During a forced-exploration epoch (epsilon=1.0 for 1 epoch at mid-training), sample 100 states × 3 magnitudes each, log realized step-return per (state, magnitude). Compare mean/variance across magnitudes. | Ratio `mean(R_full) / std(R_full)` is ≤ 0.5× `mean(R_quarter) / std(R_quarter)` — policy correctly prefers Quarter under Sharpe-per-trade logic. |
|
||||
| H1 | Reward gradient asymmetry — Full-position reward variance is higher than Quarter's; policy correctly avoids high-variance bins. | Forced-exploration epoch (epsilon=1.0 for 1 epoch at mid-training), sample 100 states × 3 magnitudes each, log realized step-return per (state, magnitude). Compare mean/variance across magnitudes. | `mean(R_full)/std(R_full)` is ≤ 0.5× `mean(R_quarter)/std(R_quarter)` — policy correctly prefers Quarter under Sharpe-per-trade logic. |
|
||||
| H2 | Var[Q] conviction-sizer over-shrinking — `effective_max_pos *= 1/(1 + sqrt(Var[Q]))` permanently collapses the target toward Quarter. | `var_scale_mean` field | `var_scale_mean < 0.3` across ≥10/20 epochs |
|
||||
| H3 | Kelly cap early-firing — Kelly stats (computed from Quarter-only history) cap at Quarter after 20+ trades. | `kelly_f_mean` field | `kelly_f_mean ≤ 0.25` across ≥10/20 epochs |
|
||||
| H3 | **Kelly as hard multiplier is too aggressive for marginal edges** — Kelly formula `f = (bp − q)/b` gives small `f` when win rate × avg_win ≈ avg_loss (marginal edges, typical in trading). The code multiplies `effective_max_pos *= kelly_f`, shrinking the envelope before magnitude selection even sees it. Not a Kelly bug; a design question: should Kelly be a hard multiplier or a soft gate? | `kelly_f_mean` AND `avg_win / avg_loss` stats from training | `kelly_f_mean ≤ 0.25` AND `avg_win / avg_loss < 1.3` — confirms marginal-edge scenario where Kelly's low value is correct math but wrong policy tool. |
|
||||
| H4 | Magnitude-head gradient starvation — the per-action-branch gradient (*not* the per-loss-component IQN/CQL/C51/Ens split) to the magnitude head is much smaller than to the direction head. Magnitude weights stay near init. | `grad_ratio_mag_dir` field, measured as end-of-epoch average of per-step ‖∇ W[mag_head]‖ ÷ ‖∇ W[direction_head]‖ (pre-clip). | `grad_ratio_mag_dir < 0.1` across ≥15/20 epochs |
|
||||
| H5 | Bootstrap bias — initial random exploration anchored Quarter, epsilon decayed before the policy escaped. | Action entropy vs epsilon schedule | entropy < 0.3 before epoch 5 AND epsilon below 0.1 |
|
||||
| H6 | Regime-adaptive trailing stop (wired 2026-04-21) closes Full-position trades prematurely. Full moves hit profit-retreat thresholds faster than Quarter moves because P&L magnitude is larger. | Per-magnitude trailing-stop firing rate, plus mean hold-time-at-exit per magnitude. | Trail firing rate on Full positions > 3× rate on Quarter positions, AND mean hold-time-at-exit on Full < 0.5× Quarter. |
|
||||
| H7 | VSN (variable selection network) or NoisyNets-injected noise disproportionately masks/perturbs the magnitude-head's input features, so effective signal-to-noise is worse than direction head. | VSN mask distribution per branch (forward hook), NoisyNets σ per weight tensor. | Magnitude head mean-mask weight < 0.5× direction head, OR NoisyNets σ on w_b1fc > 2× other branch weights. |
|
||||
| H8 | Target-network tau too slow for magnitude propagation — because magnitude distribution is concentrated at Quarter, target updates rarely propagate signals about Half/Full (never sampled enough), effective learning rate is lower for those bins. | Target-net weight drift per branch (tau × online_grad accumulated). | Magnitude-head target drift < 0.1× direction-head drift cumulative. |
|
||||
| H9 | **Not a bug: data favors Quarter.** Most bars are low-volatility range; optimal sizing per bar IS Quarter. Removing magnitude branch entirely would be the correct design, not a fix. | Per-magnitude Sharpe during forced-exploration epoch (H1 measurement reused). | If all three magnitudes show similar Sharpe (no statistically significant difference), Quarter-collapse is correct behaviour and magnitude branch is dead code. |
|
||||
| H10 | Entropy regularization + argmax tie-breaking: entropy bonus encourages uniform distribution across magnitudes; argmax at evaluation picks the first bin (Quarter by index) on any near-tie. Training-time exploration looks uniform but evaluation is quarter-biased. | Compare training-mode action distribution vs eval-mode action distribution. | Training-mode magnitude entropy > 0.8 (near-uniform) AND eval-mode `F_Quarter` > 90% (collapse at eval only). |
|
||||
|
||||
**Fix sketch (if confirmed):**
|
||||
|
||||
- H1 → rebalance reward gradient per magnitude (advantage weighting by bin, or per-bin loss scale)
|
||||
- H2 → clamp `var_scale ≥ 0.5` OR remove entirely if redundant with Kelly
|
||||
- H3 → decouple Kelly from magnitude conditioning (use Kelly as position scaler, not magnitude quantizer)
|
||||
- H4 → investigate why magnitude-head gradient is small. Candidates: magnitude-branch loss weight too low in the 4-branch advantage sum, magnitude head's `w_b1fc` dimensionality causing numerical suppression, VSN (variable selection network) masking magnitude features. Fix by rebalancing the 4-branch advantage weights OR adjusting initial weight scale for the magnitude head. (NOT the IQN/CQL/C51/Ens loss-component budget — that's orthogonal.)
|
||||
- H3 → replace Kelly hard multiplier with a soft gate (e.g., `if kelly_f < 0.3 then magnitude ≤ Half`, keeping magnitude head free to choose); OR remove Kelly entirely and accept greater variance in exchange for proper action space exploration
|
||||
- H4 → rebalance 4-branch advantage weights OR adjust initial weight scale for the magnitude head
|
||||
- H5 → extend exploration warmup for magnitude branch only
|
||||
- H6 → feed per-magnitude scale into the trailing-stop distance (larger positions get proportionally wider stops), OR disable trailing stop entirely if H6 + H9 both confirmed
|
||||
- H7 → mask VSN's effect on magnitude features; reduce NoisyNets σ on magnitude weights
|
||||
- H8 → increase tau OR use per-branch target update rates (direction faster, magnitude slower with sample-weighted updates)
|
||||
- H9 → delete magnitude branch, reduce action space from 108 to 36 (direction × order × urgency only). Massive simplification if this hypothesis confirms.
|
||||
- H10 → break argmax ties by Q-value margin (no bin ordering bias); OR match training/eval distributions via stochastic eval rollouts
|
||||
|
||||
### 5.2 Track 2 — Reward V7 audit
|
||||
|
||||
@@ -205,9 +250,84 @@ Every reward signal gets the V7 3-step treatment.
|
||||
| Contribution is meaningful AND invariant is not otherwise covered | KEEP |
|
||||
| Contribution appears meaningful but code is half-wired (e.g., forward only, no backward) | WIRE PROPERLY or DELETE based on backward-completion effort |
|
||||
|
||||
### 5.3 Phase 1 exit
|
||||
### 5.3 Track 3 — Controllers V7 audit
|
||||
|
||||
Each track produces a triage list. Verdicts + evidence + proposed Phase-2 fix per item.
|
||||
Every adaptive controller currently shaping training gets a V7 3-step treatment.
|
||||
|
||||
**Controllers currently active:**
|
||||
|
||||
| # | Controller | Current role |
|
||||
|---|---|---|
|
||||
| C1 | Anti-intuitive LR (asymmetric, wired 2026-04-21) | Fires every epoch after warmup; boosts LR on good signal, dampens on sustained bad |
|
||||
| C2 | Adaptive tau (target-net update rate) | Adjusts based on Q-drift |
|
||||
| C3 | Adaptive gamma (discount) | Curriculum: low gamma early, high later |
|
||||
| C4 | Adaptive grad clip | Threshold adjusts to running grad-norm distribution |
|
||||
| C5 | CQL alpha schedule | Pessimism weight vs online data confidence |
|
||||
| C6 | Cost-anneal schedule (`cost_anneal_ptr`) | Tx-cost curriculum: 0 early, full later |
|
||||
| C7 | Learning-rate scheduler (base) | Step decay or cosine |
|
||||
|
||||
**Audit question per controller:** is this load-bearing (firing often, without it training breaks) or diagnostic (firing rarely, just catching edge cases)? A production-deployable policy should have controllers be diagnostic.
|
||||
|
||||
**Decision matrix per controller:**
|
||||
|
||||
| Evidence | Decision |
|
||||
|---|---|
|
||||
| Firing rate > 50% of epochs AND disabling it breaks training | LOAD-BEARING — keep, document the necessity, flag as production-readiness concern |
|
||||
| Firing rate < 10% of epochs | DIAGNOSTIC — keep, no action |
|
||||
| Firing rate 10–50% AND disabling it doesn't break training | CANDIDATE FOR DELETE — measure impact of removal, delete if within noise |
|
||||
| Controller's trigger condition is symptom of upstream bug | ROOT-CAUSE FIX — fix upstream, controller becomes diagnostic |
|
||||
|
||||
**Track 3 smoke test deliverable:**
|
||||
```rust
|
||||
// crates/ml/src/trainers/dqn/smoke_tests/controller_activity.rs
|
||||
fn test_controllers_not_load_bearing() {
|
||||
// Train 20 epochs. Record per-controller firing rate.
|
||||
// Assert: no single controller fires in > 50% of epochs.
|
||||
// High firing = policy can't hold on its own = production risk.
|
||||
}
|
||||
```
|
||||
|
||||
**HEALTH_DIAG extensions (Track 3):**
|
||||
- `controller_fire_{anti_lr, adaptive_tau, adaptive_gamma, grad_clip, cql_alpha, cost_anneal}` — boolean per epoch, aggregated over window
|
||||
|
||||
### 5.4 Track 4 — Exploration V7 audit
|
||||
|
||||
**Exploration mechanisms currently active:**
|
||||
|
||||
| # | Mechanism | Current role |
|
||||
|---|---|---|
|
||||
| E1 | Epsilon-greedy | Random action with probability ε, decays over training |
|
||||
| E2 | NoisyNets | Injects parametric noise in fc layers for exploration |
|
||||
| E3 | Count-based bonus (`count_bonus=true`) | Reward bonus for rarely-visited (state, action) pairs |
|
||||
| E4 | Entropy regularization | Pushes policy toward uniform action dist |
|
||||
| E5 | Counterfactual flip (R3) | Data-level exploration via mirroring (already in Track 2) |
|
||||
|
||||
**Audit question per mechanism:** are these redundant with each other, and is the net exploration pressure at the action/magnitude level appropriate?
|
||||
|
||||
**V7 audit template per mechanism:**
|
||||
1. Identify signal — what exploration problem does this solve?
|
||||
2. Check better-form — does another mechanism already provide this?
|
||||
3. Measure — log magnitude-level action entropy, "novel state" count, per-mechanism contribution estimate
|
||||
4. Decide — DELETE redundant, TUNE under-powered, KEEP covering unique problem
|
||||
|
||||
**Track 4 smoke test deliverable:**
|
||||
```rust
|
||||
// crates/ml/src/trainers/dqn/smoke_tests/exploration_coverage.rs
|
||||
fn test_exploration_covers_magnitude_space() {
|
||||
// Train 20 epochs. Record per-epoch magnitude-level action entropy.
|
||||
// Assert: entropy over magnitudes ≥ 0.5 at epoch 5, ≥ 0.3 at epoch 20.
|
||||
// If entropy collapses too fast, magnitude space is under-explored.
|
||||
}
|
||||
```
|
||||
|
||||
**HEALTH_DIAG extensions (Track 4):**
|
||||
- `explore_entropy_magnitude` — action-entropy restricted to magnitude branch
|
||||
- `explore_entropy_direction` — same for direction branch (control)
|
||||
- `noisy_net_sigma_mean` — average NoisyNets σ across weights
|
||||
|
||||
### 5.5 Phase 1 exit
|
||||
|
||||
Each track produces a triage list. Verdicts + evidence + proposed Phase-2 fix per item. Four track outputs consolidate into one Phase 2 commit.
|
||||
|
||||
---
|
||||
|
||||
@@ -226,18 +346,27 @@ Fixes confirmed by Track 1 (magnitude collapse):
|
||||
|
||||
Deleted by Track 2 (reward V7 audit):
|
||||
- R<id> (<term>): sub-noise / better-form covered, deleted
|
||||
- ...
|
||||
|
||||
Kept by Track 2 (measurement confirmed real signal):
|
||||
- R<id>: retained, <evidence summary>
|
||||
|
||||
Fixes confirmed by Track 3 (controllers V7 audit):
|
||||
- C<id> (<controller>): <root-cause fix OR removed>
|
||||
- Note if any controller remains LOAD-BEARING — flagged for sub-project B follow-up.
|
||||
|
||||
Fixes confirmed by Track 4 (exploration V7 audit):
|
||||
- E<id> (<mechanism>): <tune OR delete>
|
||||
|
||||
Smoke tests added (Phase 0):
|
||||
- magnitude_distribution — asserts ≥5% per bin
|
||||
- reward_component_audit — asserts each active term ≥1% contrib
|
||||
- multi_fold_convergence (fast local variant)
|
||||
- magnitude_distribution (Track 1)
|
||||
- reward_component_audit (Track 2)
|
||||
- controller_activity (Track 3)
|
||||
- exploration_coverage (Track 4)
|
||||
- multi_fold_convergence (fast local variant of Phase 3 gate)
|
||||
- surrogate_noise_check (false-positive detector)
|
||||
|
||||
Validation gate (Phase 3):
|
||||
- 6-fold × 50-epoch L40S run, success criteria A+B pending
|
||||
- 6-fold × 50-epoch L40S run, success criteria A+B+surrogate-noise pending
|
||||
```
|
||||
|
||||
**Rules:**
|
||||
@@ -267,8 +396,10 @@ Existing walk-forward config produces 6 folds on `test_data/futures-baseline`.
|
||||
| A — Trade count (soft) | ≥100 trades per val window on ≥5/6 folds (per-fold, not averaged) | Metrics kernel |
|
||||
| **B — Multi-fold Sharpe (MANDATORY)** | Best val Sharpe annualized > 10 on ≥5/6 folds | `training_loop` Best Sharpe log |
|
||||
| **B — Multi-fold WinRate (MANDATORY)** | Val WinRate > 55% on ≥5/6 folds | Metrics kernel |
|
||||
| **Surrogate-noise (MANDATORY)** | Random-action Sharpe ≤ 0.5× trained-model Sharpe | `surrogate_noise_check.rs` smoke |
|
||||
| Controller-activity (soft) | No controller fires in > 50% of epochs | HEALTH_DIAG `controller_fire_*` fields |
|
||||
|
||||
Criterion B gates MUST pass for the spec to close successfully. Criterion A gates are soft — if B passes but some A gate misses, the spec closes with a documented A follow-up.
|
||||
Criterion B gates AND surrogate-noise check MUST pass for the spec to close successfully and merge to main (Phase 4). Criterion A gates are soft — if B and surrogate-noise pass but some A gate misses, the spec closes with a documented A follow-up. Controller-activity is soft — high firing rates go into a tech-debt follow-up note (sub-project B concern).
|
||||
|
||||
### 7.3 Partial-success handling
|
||||
|
||||
@@ -284,8 +415,12 @@ Left in the tree after the spec closes:
|
||||
|
||||
- `crates/ml/src/trainers/dqn/smoke_tests/magnitude_distribution.rs`
|
||||
- `crates/ml/src/trainers/dqn/smoke_tests/reward_component_audit.rs`
|
||||
- `crates/ml/src/trainers/dqn/smoke_tests/controller_activity.rs`
|
||||
- `crates/ml/src/trainers/dqn/smoke_tests/exploration_coverage.rs`
|
||||
- `crates/ml/src/trainers/dqn/smoke_tests/multi_fold_convergence.rs`
|
||||
- HEALTH_DIAG extensions (Track 1 + Track 2 fields, all behavioral-code-path reusable)
|
||||
- `crates/ml/src/trainers/dqn/smoke_tests/surrogate_noise_check.rs`
|
||||
- HEALTH_DIAG extensions across all 4 tracks (all behavioral-code-path reusable)
|
||||
- `git tag policy-quality-baseline` (Phase 0 anchor for rollback)
|
||||
|
||||
These are the regression coverage for the invariants this spec establishes.
|
||||
|
||||
@@ -306,10 +441,14 @@ These are the regression coverage for the invariants this spec establishes.
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|---|---|---|---|
|
||||
| Phase 1 investigation finds no confirmed bugs (all H1–H5 REJECTED) | Low | Medium | Track 2 reward audit still produces findings; if both tracks come up empty, upgrade spec to include controller audit (sub-project A scope creep). |
|
||||
| Phase 2 fix passes smokes but validation (Phase 3) still misses action-dist gate | Medium | Medium | Partial-success path. Follow-up spec addresses remaining deltas. |
|
||||
| Phase 1 finds all 10 Track-1 hypotheses REJECTED | Low | High | Track 2/3/4 almost certainly surface findings; if all 4 tracks come up empty, magnitude collapse might be **H9 (not a bug, data favors Quarter)**. Delete magnitude branch per H9 fix sketch — action-space simplification is still a production win. |
|
||||
| H9 confirmed: magnitude branch deletion breaks backwards compat more than expected | Medium | Medium | Greenfield policy covers this — we accepted retrain cost. Smoke tests validate pre/post equivalence in non-magnitude dimensions. |
|
||||
| Phase 2 fix passes smokes but validation (Phase 3) misses action-dist (soft) gate while passing B (mandatory) | Medium | Low | Soft-gate partial-success path. Follow-up spec addresses A deltas without blocking merge. |
|
||||
| Phase 2 fix passes smokes but validation misses B (mandatory) | Medium | High | Feature branch stays unmerged. `feat/policy-quality` kept for rerun attempts. No pollution of main. If 3 iterations fail, escalate: reward or architecture change is the real need (sub-project A scope exceeded). |
|
||||
| Surrogate-noise check fails (random actions match trained model) | Low | Critical | Blocks merge. Indicates look-ahead or data-leak bug that invalidates all training to date. Halt spec, open dedicated diagnostic spec. |
|
||||
| Retraining cost runs beyond 10 hrs L40S | Low | Low | Iteration budget capped at 3 full validation runs. If still failing, triage with faster subset runs. |
|
||||
| Greenfield changes invalidate smoke-test baselines (circular dependency) | Low | High | Phase 0 baseline captured in text notes before any fix. Smokes measure relative deltas, not absolute thresholds where possible. |
|
||||
| Controller-audit (Track 3) finds multiple load-bearing controllers that can't be removed in this spec | Medium | Medium | Document as tech-debt, flag for sub-project B. Spec can still close if policy is profitable. But "load-bearing controllers = not production-deployable" remains a known concern. |
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user