# Track 4 — Exploration V7 Audit Triage

**Phase:** Phase 1 — Track 4 (Exploration V7 audit)
**Captured against commit:** `5c70c68a1` (tag: post Track 1 triage landing)
**Hardware:** RTX 3050 Ti Laptop GPU (4 GB) — **preliminary smoke-scale verdicts**; L40S validation pending per plan §Task 1.1
**Data source:** baseline `magnitude_distribution.rs` + `exploration_coverage.rs` smoke runs (20 epochs × 3 folds, 5000-bar slice, `dqn-smoketest` profile).
**Log refs:** `/tmp/foxhunt_smoke/magnitude_distribution.log`, `/tmp/foxhunt_smoke/exploration_coverage.log` (local; not committed).

---

## Executive summary

Preliminary triage against spec §5.4 exploration mechanisms (E1–E5). At smoke scale:

- **1 KEEP** — E1 (epsilon-greedy); the only mechanism preserving exploration pressure once σ-anneal and ε-decay end and argmax dominates.
- **1 TUNE** — E2 (NoisyNets); scheduler is wired but the HEALTH_DIAG `sigma_mean` accessor reports the *raw* σ tensor (constant 0.0320) instead of the anneal-scaled effective σ. Instrumentation gap → can't confirm scheduler contribution.
- **1 DELETE-CANDIDATE** — E4 (entropy regularization in C51); `entropy_coefficient = 0.001` is ≤ reward magnitude by design (per the in-code comment at `config.rs:1412`) and entropy is already ~1.0 from NoisyNets + wide Q; the term is not producing observable exploration pressure. Provisional DELETE; final verdict blocked on the gradient decomposition that Track 1 also needs for H4.
- **1 KEEP** — E3 (count-based UCB bonus); uniquely addresses the magnitude-coverage failure mode seen in fold 3 (ent_mag drops from 0.94 → 0.68 mid-training before recovering).
- **1 CROSS-REFERENCED** — E5 (counterfactual flip); Track 2 handles this as R3 at the reward level. Confirmed wired at `experience_kernels.cu:1252-1261` and gated by `exploration_scale`.

**Most load-bearing cross-track interaction**: Track 1 CONFIRMED H10 (eval-time argmax collapses to Quarter despite training `ent_mag ≈ 0.99`). Track 4 confirms the corollary: **training-time exploration is working correctly** across all three mechanisms that measurably fire (E1 ε≈0.01 floor, E2 σ-noise, E3 UCB). The exploration pressure is *not* the bug — it's being discarded at the eval argmax boundary. The Phase 2 fix for the observed Quarter collapse is Track 1's H10 patch (match training/eval distributions or Q-margin tie-break), **not** more exploration in E1–E4.

---

## Per-mechanism verdicts

### E1 — Epsilon-greedy

**Current role:** random action with probability ε, decays over training (spec §5.4).

**Wiring check:**
- Config (`crates/ml/src/trainers/dqn/config.rs:432-436, 1282-1284`): `epsilon_start=0.3`, `epsilon_end=0.02`, `epsilon_decay=0.995` (production default); smoke config (`config/training/dqn-smoketest.toml:39-41`) overrides to `epsilon_start=0.3`, `epsilon_end=0.02`, `epsilon_decay=0.85` (fast decay).
- Hot-path apply: `crates/ml/src/trainers/dqn/trainer/action.rs:82-85` reads `agent.get_effective_epsilon()`, volatility-adjusts, passes to the GPU fused action selector. Per-branch multiplier `epsilon_magnitude_mult` at line 122.
- Cold-path apply: `action.rs:161-167` uses `rng.gen_range(0..5)` (5-level exposure space — this is the single-action path used only by tests; the batch hot-path uses GPU-factored selection, so the 0..5 range here is a legacy artifact matching Track 1's H10 finding on single-action eval paths, not a smoke-run pressure).
- Decay step: `agent.update_epsilon()` is gated by `if !true` in `training_loop.rs:2861` — effectively disabled because NoisyNets are always on. So training-time ε decays only through the `epsilon_decay` hyperparameter applied elsewhere (epsilon schedule is driven by the explicit start/end/decay curve, not by `update_epsilon`).

**Measured contribution:**
- Per-epoch training logs: `Exploration: entropy=0.811–0.816, epsilon=0.0100` by epoch 20 (below the configured `epsilon_end=0.02` floor because volatility adjustment nudges it lower).
- ε is at its floor for ~all 20 epochs in the smoke run; per-step ε-random action selection is producing ~1% random-branch picks per sample.

**Verdict:** **KEEP**

**Rationale:** At the ε=0.01 floor, E1 contributes a small but non-zero fraction of random-branch actions every batch. It's the only mechanism that still injects exploration pressure **after** the σ-scheduler completes its anneal (at step ~10k, which at smoke scale means epoch ≈ 3 of fold 1 onward — σ-anneal is essentially done by epoch 5). Without E1, post-anneal training would fall back entirely to count-bonus + argmax; ε-greedy is the safety floor. Its measured cost is minimal and it's what keeps branch action entropy above ~0.8 when Q-values sharpen later in training.

**Caveat:** the cold-path `epsilon_greedy_action` at `action.rs:161-167` uses a 0..5 range that predates the 4-branch refactor (the MEMORY.md entry confirms 4-branch was added 2026-04-08; this function signature is from the old 9-level code). It's only used by tests, not smoke or production, so the pressure it'd add is zero in measured runs. Flagging it as tech debt; not material to this triage.

---

### E2 — NoisyNets

**Current role:** injects parametric noise in FC layers for exploration (spec §5.4).

**Wiring check:**
- Layer impl: `crates/ml-dqn/src/noisy_layers.rs` — σ tensor initialized to `sigma_init / sqrt(in_features)`. At smoke, `sigma_init=0.3` and `in_features=88` (the SH2+3 direction-conditioned input layout confirmed in MEMORY.md) → σ ≈ 0.0320, matching HEALTH_DIAG `sigma_mean=0.0320` exactly.
- Scheduler: `crates/ml-dqn/src/noisy_sigma_scheduler.rs` — linear anneal from `initial_sigma=0.8` to `final_sigma=0.4` over `noisy_sigma_anneal_steps=10000` (config default at `config.rs:1495-1497`), with a `SIGMA_FLOOR=0.3` clamp.
- Scheduler wiring: `trainer/constructor.rs:481-490` — always instantiated; `training_loop.rs:2867-2874` — steps `train_step_count` times per epoch, then `agent.set_noise_sigma_scale(sigma_scale)` and `reset_noise_with_sigma(scale)`.
- **Instrumentation gap:** `reset_noise_with_sigma` at `noisy_layers.rs:161-211` scales only the *noise epsilon samples* (`weight_epsilon`, `bias_epsilon`), **not** the underlying σ tensor. The `sigma_mean` accessor at line 345 reads the unchanged σ tensor and returns the same value every epoch (0.0320). So HEALTH_DIAG `sigma_mean` cannot confirm whether annealing is happening on the effective noise magnitude; the *reported* 0.0320 is consistent with "sigma_init fixed at 0.3" whether or not the scheduler scale is active.

**Measured contribution:**
- `sigma_mean=0.0320` every epoch (20/20) in all 3 folds of the baseline run — by construction, this is what the accessor reads, not a measurement of whether σ-annealing fires.
- Effective noise scale: at 58 steps/epoch × 20 epochs × 3 folds = ~3480 training steps, scheduler progress ≈ 35% of its 10k-step schedule → effective `sigma_scale` should be 0.8 + 0.35·(0.4 − 0.8) = 0.66 by end of smoke. This would lower the noise magnitude by ~34% over the run, but the σ tensor itself is untouched, so we see a flat reading.

**Verdict:** **TUNE** — specifically, fix the `sigma_mean` accessor to report `σ_tensor · sigma_scale` (or report the scheduler's current `sigma_scale` as a separate HEALTH_DIAG field) before drawing any firmer conclusion. **Do not delete**: NoisyNets are widely established as effective for 4-branch factored action heads, and with high action-space cardinality (3×3×3×3 = 81) the per-weight noise is load-bearing for state-conditioned exploration. But we can't measure contribution without fixing instrumentation.

**Rationale:** Track 1 H7 already REJECTED the symmetric "NoisyNets mask magnitude more than direction" hypothesis at the σ-tensor level, which rules out asymmetric harm. But Track 1 H7 is REJECTED on the same broken accessor; a correct instrumentation may still flip the verdict. Track 4's independent concern is a different question: "is E2 contributing *at all* as exploration pressure?" — and we cannot answer that without reading the scheduler-adjusted effective σ.

---

### E3 — Count-based bonus (`count_bonus_coefficient`)

**Current role:** reward bonus for rarely-visited (state, action) pairs (spec §5.4). In this codebase it's implemented as an additive UCB bonus at the *action-selection* stage, not a reward-term — see wiring below.

**Wiring check:**
- Config: `config/training/dqn-smoketest.toml:44` — `count_bonus_coefficient = 0.1`. Production/localdev match. Hyperopt bounds `[0.0, 0.3]` at `config/training/dqn-hyperopt.toml:53`.
- Config object: `config.rs:743, 1414` — `count_bonus_coefficient: Some(0.1)` in the default.
- Apply site: `trainer/action.rs:105-118` — every batch, `agent.get_count_bonuses_branched()` returns per-branch UCB bonuses; scaled by the coefficient; uploaded to GPU action selector via `set_count_bonuses(&be_f32, &bo_f32, &bu_f32)`. This adds the bonus to the Q-values consumed by `select_actions_branching`.
- Reset: `training_loop.rs:2878-2879` — counts reset per epoch boundary.
- Training-time confirmation: the `Exploration:` log line at `training_loop.rs:3160-3164` reports `count_bonus=true` when `count_bonus_coefficient > 0.0` — the smoke log shows this is true for all 20 epochs of all 3 folds.

**Measured contribution:**
- "novel state" count is not wired as a HEALTH_DIAG field yet. The proxy is the magnitude entropy trajectory.
- Fold 3 (entries 41–60 of the exploration-coverage entropy history) is the one epoch sequence where `ent_mag` demonstrates a visible decay/recovery arc: 0.94 → 0.87 → 0.80 → **0.68 (epoch 13)** → 0.87 (epoch 19). The recovery from 0.68 to 0.87 is consistent with per-epoch count-bonus reset nudging under-explored magnitude bins back into selection. Folds 1 and 2 never drop below 0.95, leaving their contribution unmeasurable at smoke scale (saturation).

**Verdict:** **KEEP**

**Rationale:** E3 is the only mechanism that provides *state-conditioned* exploration pressure tied to per-branch action coverage. E1 is uniform-random; E2 is parametric noise on weights (uncoupled from coverage); E4 is a distributional regularizer. When magnitude entropy starts decaying (fold 3 is the smoke-scale example), E3 is structurally the right thing to lean on — and the fold 3 recovery is suggestive that it does. Until the fold-3 anomaly is understood at L40S scale, DELETE would be premature. Flag for production-scale re-measurement with an explicit "novel-branch-action rate" HEALTH_DIAG field so contribution can be quantified rather than inferred.

---

### E4 — Entropy regularization

**Current role:** pushes policy toward uniform action distribution (spec §5.4). Implemented as a C51-logit penalty term, not a policy-gradient entropy bonus.

**Wiring check:**
- Config: `config/training/dqn-smoketest.toml:43` — `entropy_coefficient = 0.001`. Same in production/localdev.
- Config object: `config.rs:739, 1412` — `entropy_coefficient: 0.001` with in-code comment: `"Must be ≤ reward magnitude (~0.001). Old 0.01 was 10x reward → entropy dominated learning."` (explicit design intent to keep this term subdominant.)
- Apply site: `fused_training.rs:388` — passed to `c51_grad_kernel` as `entropy_coefficient: dqn.config.entropy_coefficient as f32`.
- Kernel: `crates/ml/src/cuda_pipeline/c51_grad_kernel.cu:25, 64-68` — adds `inv_batch * entropy_coeff * (1.0 + lp_clamped)` to the combined gradient if `entropy_coeff > 0.0f`.

**Measured contribution:**
- `ent_mag` baseline across all 3 folds × 20 epochs is 0.68–0.99 (fold 3 low: 0.68; all other folds ≥0.95). Near-uniform.
- But: training-time entropy is already high for other reasons:
  1. H4 (Track 1 CONFIRMED): magnitude-head gradient ≈ 0 → Q-values near init → naturally near-uniform. E4 can't push what's already uniform.
  2. NoisyNets injects weight-space noise → per-sample Q-values differ across samples independently of E4.
  3. Count-bonus keeps rare actions artificially attractive.
- With `entropy_coefficient = 0.001` sized to stay ≤ reward magnitude, the per-gradient contribution is ~1e-4 · (1 + log_prob) per (b, d, j) — at typical `log_prob` values near `log(1/num_atoms)` the term is ~1e-4 in magnitude, ≈ 1/10000th of the Bellman CE gradient at typical scales. By construction, this is sub-noise.

**Verdict:** **DELETE-CANDIDATE** (provisional — needs Track 1's per-component gradient decomposition before final).

**Rationale:** The coefficient is explicitly sized to be smaller than the primary loss term (per the code comment), and the branch entropy it's supposed to push is already maxed out by E2/E3 plus the H4 "no gradient to magnitude" pathology. It's a fifth mechanism covering an invariant (action-uniformity) that three other mechanisms already enforce. Either:
- **(a)** it's inactive in measurable contribution and can be removed without affecting policy (DELETE), or
- **(b)** it's doing useful Q-logit smoothing we haven't measured (KEEP).

Resolving the ambiguity requires the per-loss-component gradient decomposition that Track 1's H4 fix also depends on (IQN / CQL / C51 / Ens split). Once we can read the C51-entropy contribution separately, we can decide. Until then, provisional DELETE — the Occam-razor case is that a 0.001-weighted term sized explicitly *not* to dominate learning is not the thing breaking policy quality.

**Note on overlap with H10:** even if E4 were working perfectly at training time, Track 1 H10 shows eval-time argmax erases that pressure. Adding or tuning E4 doesn't help eval behavior unless training-eval alignment is also fixed — which is Track 1's job.

---

### E5 — Counterfactual flip (R3)

**Current role:** data-level exploration via mirroring (spec §5.4 note: "already in Track 2").

**Wiring check (confirming cross-track handoff):**
- Kernel: `crates/ml/src/cuda_pipeline/experience_kernels.cu:1252-1261` — `do_flip = (flip_u < exploration_scale * cf_ratio) ? 1 : 0`. Effective flip probability is `exploration_scale * cf_ratio` where `cf_ratio` is the adaptive ratio computed from learning health.
- Collector: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs:742-743, 1391, 1458, 1475, 2754-2755, 2837` — `current_cf_ratio()` reads from cached learning health; passed as kernel arg; `last_cf_ratio_eff` tracked for HEALTH_DIAG.
- Reward plumbing: kernel flips reward sign at lines 2128, 2153-2158, 2187-2192 conditionally on `do_flip`.
- `exploration_scale` is gated to 0.0 in validation mode (see `experience_kernels.cu:1517-1523`) so CF flips are training-only — consistent with E5 being a training-time exploration mechanism.

**Measured contribution (at R3 level):** handled by Track 2.

**Verdict:** **CROSS-REFERENCED (Track 2)**

**Rationale:** The spec explicitly scopes E5 as R3 in Track 2's audit; wiring here just confirms the mechanism is live (`cf_ratio_eff = 0.66` per the baseline HEALTH_DIAG, `reward_contrib [cf=0.592]` at epoch 20 of fold 3). Track 4 defers to Track 2's measurement.

---

## Cross-mechanism synthesis

**1. Training-time exploration is not the bug.** Across all 3 folds and 20 epochs, `ent_mag` stays ≥ 0.68 and is usually ≥ 0.95. Branch-joint entropy is 0.81–0.82 at the end. The system is exploring in action space during training. Track 1 H10 is the downstream failure: eval-time argmax discards all four mechanisms' pressure.

**2. E1 + E2 are not obviously redundant.** E1 is discrete and uniform-random over branches; E2 is continuous-weight parametric noise with state-conditioned outputs. They cover different exploration shapes. At smoke, E2 gets effectively zeroed from the post-anneal sigma_scale → E1's ε-floor becomes the only explicit randomness outside of count bonuses. Deleting E1 would leave the post-anneal regime relying entirely on E2 (which may itself be reducing via scheduler) + E3 + E4 — fragile. Keep both; measure E2 contribution properly first.

**3. E3 is the load-bearing coverage-aware mechanism.** The fold 3 entropy trajectory (0.94 → 0.68 → 0.87) is the one smoke-scale signal where a mechanism could plausibly recover a genuine coverage collapse. If E3 is removed, fold 3 at 0.68 may not recover. This needs to be replicated at L40S before committing.

**4. E4 is cosmetic at its current weight.** A 0.001-weighted entropy term on C51 logits, intentionally sized to be 1/10x the reward magnitude, is not going to fight Track 1 H4's zero-gradient problem in the magnitude head. If H4 is fixed in Phase 2, the need for E4 goes further down, not up (the gradient path itself starts carrying the learning signal). Provisional DELETE.

**5. E5 is Track 2's domain.** Confirmed wired; contribution measured elsewhere.

**6. Fold-3 anomaly is the one signal worth following.** Folds 1 and 2 stay saturated at near-uniform entropy, so they cannot distinguish "E3 doing useful work" from "policy naturally uniform." Fold 3 is the only fold with a real coverage dynamic. A single-seed smoke anomaly is weak evidence; L40S replication across multiple seeds is required before committing to any E3 tuning.

**7. Cross-track H10 interaction restated.** Phase 2 should **not** add exploration pressure at the training-time mechanisms (E1–E4 all work). Phase 2 should match training/eval distributions per Track 1's H10 fix (Q-margin tie-break or stochastic eval rollouts). If that lands, E4's marginal contribution becomes even more irrelevant (training entropy naturally persists; nothing is silently discarded at eval), strengthening the DELETE case for E4.

---

## Proposed Phase 2 changes (Track 4)

Ordered by simplification-first (per spec §5.5 conflict-resolution rule):

1. **DELETE (provisional, blocked on H4 gradient decomposition)** — remove `entropy_coefficient` from `DQNHyperparameters`, `c51_grad_kernel` arg, and `config/training/dqn-*.toml [exploration]` sections. Re-run baseline smoke and confirm `ent_mag` does not drop materially. Retain if drop > 0.05 across any fold. Gate this DELETE on Track 1's per-component gradient split landing first — if C51-entropy turns out to be *the* gradient to the magnitude head, the DELETE order flips to KEEP and the fix moves to rescaling that path.

2. **TUNE (instrumentation, not policy)** — fix `sigma_mean` HEALTH_DIAG accessor at `crates/ml-dqn/src/noisy_layers.rs:345` to return `σ_tensor_mean · current_sigma_scale` (or add a separate `sigma_scale_effective` field to HEALTH_DIAG `noisy [...]`). Required before any firm verdict on E2 contribution. Pure accessor fix; zero behavior change.

3. **KEEP** — E1 (epsilon-greedy at ε-floor=0.01), E3 (count-based UCB bonus at coeff=0.1).

4. **Tech-debt flag (not Phase 2)** — `action.rs:161-167 epsilon_greedy_action` still uses the pre-4-branch 0..5 exposure range; it's only called by tests so zero measured pressure, but it will mislead anyone who reads it. Rename or delete in a follow-up cleanup pass, not here.

5. **No change** to E5 — defer to Track 2 verdict.

---

## Limitations of this preliminary triage

- **Smoke scale ≠ production scale.** `dqn-smoketest` runs 5000 bars, batch=64, buffer=256 across 3 folds × 20 epochs. L40S production is 176k bars × 50+ epochs × batch=16384. ε-schedule, σ-anneal, and count-bonus dynamics all run for orders of magnitude more steps at production — some verdicts may flip. Specifically, E2's effective σ-scale at smoke (~0.66 by end) is much closer to its initial 0.8 than the production-run endpoint (~0.4 or floor). Re-measure all five verdicts at L40S.
- **`sigma_mean` accessor is broken.** Until instrumentation fix lands, E2's contribution cannot be firmly assessed. TUNE verdict is gated on this fix; DELETE cannot be argued without it.
- **"Novel state" count is not instrumented.** E3's contribution is inferred from the fold-3 entropy recovery only. A direct per-branch "novel action coverage" HEALTH_DIAG field would give a cleaner measurement.
- **Fold-3 single-seed anomaly.** The 0.94 → 0.68 → 0.87 magnitude-entropy trajectory is from one seed on one fold. Could be noise; could be a real coverage failure mode. Must replicate across multiple seeds before deriving any E3 tuning decision.
- **Per-component gradient decomposition missing.** E4 verdict is blocked on the same IQN/CQL/C51/Ens gradient split that Track 1 H4 needs. Land once, reuse.
- **Cold-path `epsilon_greedy_action` 0..5 range** is tech debt unrelated to the audit but surfaced during wiring check; flagged for cleanup pass.

## Next Track 4 steps

- Land the `sigma_mean` accessor fix as a prerequisite (small, independent, low-risk).
- Add `novel_branch_action_rate` to HEALTH_DIAG for E3 direct measurement.
- Re-run `magnitude_distribution` + `exploration_coverage` smokes on L40S with multiple seeds to determine whether the fold-3 entropy dip replicates.
- After Track 1's H4 per-component gradient split lands, revisit E4 DELETE with direct C51-entropy-term contribution data.
- Decide whether the fold-3 anomaly warrants a separate micro-instrumentation task (per-fold magnitude-bin visit counts over time).
