feat(alpha): alpha_dqn_h600_smoke — runnable Task 12 DQN smoke

Phase E.1 Task 12. Linear Q-network (W [9×10] + b [9], no hidden layer)
trained with ε-greedy + Munchausen target on the Phase E ExecutionEnv.
End-to-end runnable: load env, train, periodically launch
alpha_kill_criteria + apply_pearls_ad chain at episode boundaries, emit
PASS/FAIL verdict against the 4 kill criteria thresholds.

Pipeline per training step (all on GPU):
  1. forward Q_current on s_batch via alpha_linear_q_forward
  2. forward Q_next on s'_batch via alpha_linear_q_forward
  3. alpha_munchausen_target → targets[batch]
  4. alpha_linear_q_grad → dW, db (sparse over taken actions)
  5. alpha_linear_q_sgd_step on W and b (separate launches)
  6. every K episodes: kill_criteria + apply_pearls_ad chain → ISV[539..542]

Pipeline visibility bumps so examples can reach launchers:
  - cuda_pipeline::alpha_kernels module → pub
  - All launch_alpha_* fns → pub
  - launch_apply_pearls → pub
  - ALPHA_LINEAR_Q_CUBIN → pub

These are appropriate pub exports (Phase E.1 public API surface).

Initial micro-smoke (horizon=100, n_episodes=50, lr=1e-6):
  Q_SPREAD_EMA         = 3.12   (≥ 0.05)    PASS
  ACTION_ENTROPY_EMA   = 2.12   (≥ 1.0986)  PASS
  RETURN_VS_RANDOM_EMA = +1.03  (≥ 0.0)     PASS
  EARLY_Q_MOVEMENT_EMA = 2268   (≥ 0.01)    PASS [unphysical scale]
  Overall: PASS (uncalibrated)

Known stability issues — flagged in the binary's CLI docstring:
  - lr=1e-4 diverges to NaN (Q grows, Munchausen target explodes)
  - lr=1e-6 stays finite but Q grows 2000× over 50 episodes
  - Follow-ups: gradient clipping, target network, reward normalisation

Bug fixed during development: `stream.memcpy_htod(&host, &mut buf.clone())`
was uploading to a TEMPORARY clone (dropped immediately) — `kc_scalar_dev`
and `kc_action_counts_dev` never got their host data → entropy=0, early_mvmt=0,
rvr stuck at the alloc-zeros default. Fixed by removing `.clone()` and using
direct `&mut` refs.

Reads:
  config/ml/alpha_fill_coeffs.json   (Task 5c)
  ISV slots 547/548                  (Task 7c baseline)

Writes:
  config/ml/alpha_dqn_h600_smoke.json (verdict + per-checkpoint KC trajectory)

Reproduction:
  cargo run -p ml --release --example alpha_dqn_h600_smoke -- \
    --mbp10-dir /home/jgrusewski/Work/foxhunt/test_data/futures-baseline-mbp10/ES.FUT \
    --horizon 600 --n-episodes 1000

Audit doc docs/isv-slots.md updated per Invariant 7.
This commit is contained in:
jgrusewski
2026-05-15 15:39:30 +02:00
parent 36ab50814e
commit fa30c2dd66
5 changed files with 769 additions and 8 deletions

View File

@@ -661,3 +661,39 @@ All three are wired through `pub(crate)` launchers in `alpha_kernels.rs`. Cubin
Architecture rationale (linear, no hidden layer): the 10-dim Phase E state has meaningful direct features (alpha_logit, spread_bps, position, ...) so linear Q captures real relations like `Q[Buy] ∝ alpha_logit`. If linear can't pass the kill-criteria gate, no architecture upgrade will save it.
**Next step (Task 12 proper):** smoke binary that loads the env from MBP-10 + fitted FillModel, instantiates the linear Q-net via these kernels, runs ε-greedy training for N episodes with Munchausen target, periodically launches the kill-criteria pipeline, and emits PASS/FAIL verdict.
## Phase E.1 Task 12 — H=600 DQN smoke binary (2026-05-15)
`crates/ml/examples/alpha_dqn_h600_smoke.rs` is the runnable kill-criteria-gate test for Milestone E.1. Linear Q-network on GPU (W [9×10] + b [9], no hidden layer) trained with ε-greedy action selection + Munchausen target augmentation on the Phase E ExecutionEnv. All compute on GPU; action selection reads 9 Q-values to CPU per step (read-only).
Reads:
- `config/ml/alpha_fill_coeffs.json` (Task 5c artifact)
- ISV anchors 547/548 = Task 7c baseline (mean=-5185, std=4953)
Writes:
- `config/ml/alpha_dqn_h600_smoke.json` — verdict + per-checkpoint KC trajectory
Visibility bumps required for examples (not crate-internal):
- `cuda_pipeline::alpha_kernels` module: `pub(crate)``pub`
- All `launch_alpha_*` launchers: `pub(crate)``pub`
- `cuda_pipeline::sp4_wiener_ema::launch_apply_pearls`: `pub(crate)``pub`
- `ALPHA_LINEAR_Q_CUBIN`: `pub(crate)``pub`
These are appropriate `pub` exports — they're the public API for using the Phase E.1 kernel layer from external trainers / smokes.
**Initial micro-smoke (horizon=100, n_episodes=50, lr=1e-6):**
- Q_SPREAD_EMA = 3.12 (≥0.05, PASS)
- ACTION_ENTROPY_EMA = 2.12 (≥1.0986, PASS)
- RETURN_VS_RANDOM_EMA = +1.03 (≥0.0, PASS)
- EARLY_Q_MOVEMENT_EMA = 2268 (≥0.01, PASS but uncalibrated — ||W||/||W_init|| grew 2000×)
All four pass but training stability is poor (early_mvmt ≈ 2000× is unphysical). Known follow-ups: (a) gradient clipping, (b) target network with periodic hard-update, (c) reward normalisation. With lr=1e-4 the network diverges to NaN at H=600 — Munchausen target produces large gradients without clipping. lr=1e-6 keeps it stable but slow.
**Reproduction:**
```bash
cargo run -p ml --release --example alpha_dqn_h600_smoke -- \
--mbp10-dir /home/jgrusewski/Work/foxhunt/test_data/futures-baseline-mbp10/ES.FUT \
--horizon 600 --n-episodes 1000
```
This is the *runnable Task 12 deliverable*. The full H=6000 scale-up (Task 13) is the next milestone gated on a more stable run at H=600.