feat(alpha): alpha_dqn_h600_smoke — runnable Task 12 DQN smoke
Phase E.1 Task 12. Linear Q-network (W [9×10] + b [9], no hidden layer)
trained with ε-greedy + Munchausen target on the Phase E ExecutionEnv.
End-to-end runnable: load env, train, periodically launch
alpha_kill_criteria + apply_pearls_ad chain at episode boundaries, emit
PASS/FAIL verdict against the 4 kill criteria thresholds.
Pipeline per training step (all on GPU):
1. forward Q_current on s_batch via alpha_linear_q_forward
2. forward Q_next on s'_batch via alpha_linear_q_forward
3. alpha_munchausen_target → targets[batch]
4. alpha_linear_q_grad → dW, db (sparse over taken actions)
5. alpha_linear_q_sgd_step on W and b (separate launches)
6. every K episodes: kill_criteria + apply_pearls_ad chain → ISV[539..542]
Pipeline visibility bumps so examples can reach launchers:
- cuda_pipeline::alpha_kernels module → pub
- All launch_alpha_* fns → pub
- launch_apply_pearls → pub
- ALPHA_LINEAR_Q_CUBIN → pub
These are appropriate pub exports (Phase E.1 public API surface).
Initial micro-smoke (horizon=100, n_episodes=50, lr=1e-6):
Q_SPREAD_EMA = 3.12 (≥ 0.05) PASS
ACTION_ENTROPY_EMA = 2.12 (≥ 1.0986) PASS
RETURN_VS_RANDOM_EMA = +1.03 (≥ 0.0) PASS
EARLY_Q_MOVEMENT_EMA = 2268 (≥ 0.01) PASS [unphysical scale]
Overall: PASS (uncalibrated)
Known stability issues — flagged in the binary's CLI docstring:
- lr=1e-4 diverges to NaN (Q grows, Munchausen target explodes)
- lr=1e-6 stays finite but Q grows 2000× over 50 episodes
- Follow-ups: gradient clipping, target network, reward normalisation
Bug fixed during development: `stream.memcpy_htod(&host, &mut buf.clone())`
was uploading to a TEMPORARY clone (dropped immediately) — `kc_scalar_dev`
and `kc_action_counts_dev` never got their host data → entropy=0, early_mvmt=0,
rvr stuck at the alloc-zeros default. Fixed by removing `.clone()` and using
direct `&mut` refs.
Reads:
config/ml/alpha_fill_coeffs.json (Task 5c)
ISV slots 547/548 (Task 7c baseline)
Writes:
config/ml/alpha_dqn_h600_smoke.json (verdict + per-checkpoint KC trajectory)
Reproduction:
cargo run -p ml --release --example alpha_dqn_h600_smoke -- \
--mbp10-dir /home/jgrusewski/Work/foxhunt/test_data/futures-baseline-mbp10/ES.FUT \
--horizon 600 --n-episodes 1000
Audit doc docs/isv-slots.md updated per Invariant 7.
This commit is contained in:
@@ -661,3 +661,39 @@ All three are wired through `pub(crate)` launchers in `alpha_kernels.rs`. Cubin
|
||||
Architecture rationale (linear, no hidden layer): the 10-dim Phase E state has meaningful direct features (alpha_logit, spread_bps, position, ...) so linear Q captures real relations like `Q[Buy] ∝ alpha_logit`. If linear can't pass the kill-criteria gate, no architecture upgrade will save it.
|
||||
|
||||
**Next step (Task 12 proper):** smoke binary that loads the env from MBP-10 + fitted FillModel, instantiates the linear Q-net via these kernels, runs ε-greedy training for N episodes with Munchausen target, periodically launches the kill-criteria pipeline, and emits PASS/FAIL verdict.
|
||||
|
||||
## Phase E.1 Task 12 — H=600 DQN smoke binary (2026-05-15)
|
||||
|
||||
`crates/ml/examples/alpha_dqn_h600_smoke.rs` is the runnable kill-criteria-gate test for Milestone E.1. Linear Q-network on GPU (W [9×10] + b [9], no hidden layer) trained with ε-greedy action selection + Munchausen target augmentation on the Phase E ExecutionEnv. All compute on GPU; action selection reads 9 Q-values to CPU per step (read-only).
|
||||
|
||||
Reads:
|
||||
- `config/ml/alpha_fill_coeffs.json` (Task 5c artifact)
|
||||
- ISV anchors 547/548 = Task 7c baseline (mean=-5185, std=4953)
|
||||
|
||||
Writes:
|
||||
- `config/ml/alpha_dqn_h600_smoke.json` — verdict + per-checkpoint KC trajectory
|
||||
|
||||
Visibility bumps required for examples (not crate-internal):
|
||||
- `cuda_pipeline::alpha_kernels` module: `pub(crate)` → `pub`
|
||||
- All `launch_alpha_*` launchers: `pub(crate)` → `pub`
|
||||
- `cuda_pipeline::sp4_wiener_ema::launch_apply_pearls`: `pub(crate)` → `pub`
|
||||
- `ALPHA_LINEAR_Q_CUBIN`: `pub(crate)` → `pub`
|
||||
|
||||
These are appropriate `pub` exports — they're the public API for using the Phase E.1 kernel layer from external trainers / smokes.
|
||||
|
||||
**Initial micro-smoke (horizon=100, n_episodes=50, lr=1e-6):**
|
||||
- Q_SPREAD_EMA = 3.12 (≥0.05, PASS)
|
||||
- ACTION_ENTROPY_EMA = 2.12 (≥1.0986, PASS)
|
||||
- RETURN_VS_RANDOM_EMA = +1.03 (≥0.0, PASS)
|
||||
- EARLY_Q_MOVEMENT_EMA = 2268 (≥0.01, PASS but uncalibrated — ||W||/||W_init|| grew 2000×)
|
||||
|
||||
All four pass but training stability is poor (early_mvmt ≈ 2000× is unphysical). Known follow-ups: (a) gradient clipping, (b) target network with periodic hard-update, (c) reward normalisation. With lr=1e-4 the network diverges to NaN at H=600 — Munchausen target produces large gradients without clipping. lr=1e-6 keeps it stable but slow.
|
||||
|
||||
**Reproduction:**
|
||||
```bash
|
||||
cargo run -p ml --release --example alpha_dqn_h600_smoke -- \
|
||||
--mbp10-dir /home/jgrusewski/Work/foxhunt/test_data/futures-baseline-mbp10/ES.FUT \
|
||||
--horizon 600 --n-episodes 1000
|
||||
```
|
||||
|
||||
This is the *runnable Task 12 deliverable*. The full H=6000 scale-up (Task 13) is the next milestone gated on a more stable run at H=600.
|
||||
|
||||
Reference in New Issue
Block a user