feat(dqn): add train_active_frac HEALTH_DIAG metric
Counterpart to existing val_active_frac. Reports the fraction of TRAINING rollout actions that were Long or Short. Required for L3 verification gate of Plan D (Phase 3) which checks Thompson is generating directional exploration during training. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -581,6 +581,10 @@ impl DQNTrainer {
|
||||
training_sharpe_ema_initialized: false,
|
||||
last_action_entropy: None,
|
||||
last_magnitude_dist: [0.0_f32; 3],
|
||||
// Plan C Task 5 — populated each epoch from monitor.action_counts in
|
||||
// training_loop.rs; consumed by HEALTH_DIAG `train_active_frac` line
|
||||
// emitted alongside val_active_frac in `consume_validation_loss`.
|
||||
last_train_active_frac: 0.0_f32,
|
||||
explore_entropy_mag_history: Vec::new(),
|
||||
prev_controller_values: super::ControllerPrevValues::default(),
|
||||
controller_fire_counts: super::ControllerFireCounts::default(),
|
||||
|
||||
@@ -857,6 +857,23 @@ impl DQNTrainer {
|
||||
window_bars,
|
||||
);
|
||||
|
||||
// Plan C Task 5 — training-rollout counterpart to val_active_frac.
|
||||
// Source: `monitor.action_counts` aggregated over the previous
|
||||
// epoch's training rollout (12-bin layout: dir*3 + mag); cached on
|
||||
// the trainer at the end of `training_loop.rs`'s per-epoch metrics
|
||||
// block (`self.last_train_active_frac`). `val_active_frac` above
|
||||
// measures direction commitment during the EVAL backtest; this
|
||||
// measures it during TRAINING rollout — divergence localises
|
||||
// whether Thompson is generating directional exploration during
|
||||
// experience collection (Plan D L3 verification gate, target
|
||||
// > 0.40 = at least 40% of training-rollout actions are Long or
|
||||
// Short, not Hold/Flat). Emitted next to val_active_frac so the
|
||||
// two lines are adjacent in HEALTH_DIAG output and easy to compare.
|
||||
tracing::info!(
|
||||
"HEALTH_DIAG[{}]: train_active_frac={:.4} (Long+Short during training rollout)",
|
||||
epoch, self.last_train_active_frac,
|
||||
);
|
||||
|
||||
m.sharpe as f64
|
||||
} else {
|
||||
0.0
|
||||
|
||||
@@ -331,6 +331,20 @@ pub struct DQNTrainer {
|
||||
/// summed across all direction bins, normalized to [0, 1].
|
||||
pub(crate) last_magnitude_dist: [f32; 3],
|
||||
|
||||
/// Plan C Task 5 — fraction of training-rollout actions where direction ∈
|
||||
/// {Short, Long} (i.e. the policy committed to a directional bet rather
|
||||
/// than Hold/Flat). Counterpart to the kernel-derived val_active_frac
|
||||
/// emitted alongside the val [...] block in `consume_validation_loss`.
|
||||
/// Required for Plan D's L3 verification gate (target > 0.40 = at least
|
||||
/// 40% of training-rollout actions are Long or Short, evidence Thompson
|
||||
/// is generating directional exploration during training).
|
||||
///
|
||||
/// Computed each epoch in the training_loop from `monitor.action_counts`
|
||||
/// (12-bin layout: dir*3 + mag, dir ∈ {Short=0, Hold=1, Long=2, Flat=3});
|
||||
/// active = bins[0..3] (Short) + bins[6..9] (Long), denominator = all 12.
|
||||
/// `0.0` until the first epoch finishes its training rollout.
|
||||
pub(crate) last_train_active_frac: f32,
|
||||
|
||||
/// Task 0.10 — per-epoch magnitude-branch action entropy history. Each
|
||||
/// entry is (epoch_0_indexed, normalized_entropy). Used by the
|
||||
/// exploration_coverage smoke test to assert the magnitude branch
|
||||
|
||||
@@ -2870,6 +2870,20 @@ impl DQNTrainer {
|
||||
// Task 0.14: append to exploration entropy history for smoke-test readback.
|
||||
self.explore_entropy_mag_history.push((epoch as u32, ent_mag));
|
||||
|
||||
// Plan C Task 5 — training-rollout active fraction (counterpart to
|
||||
// the kernel-derived val_active_frac emitted from
|
||||
// `consume_validation_loss`). "Active" = direction ∈ {Short, Long}.
|
||||
// 12-bin layout (dir*3 + mag): Short=[0..3], Hold=[3..6], Long=[6..9],
|
||||
// Flat=[9..12]; denominator = all 12 bins (every training step
|
||||
// contributes one tally). Cached on the trainer so the next epoch's
|
||||
// `consume_validation_loss` emits it next to val_active_frac.
|
||||
// Required for Plan D's L3 verification gate (target > 0.40).
|
||||
self.last_train_active_frac = if action_total_epoch > 0 {
|
||||
(dir0 + dir2).clamp(0.0, 1.0)
|
||||
} else {
|
||||
0.0_f32
|
||||
};
|
||||
|
||||
// Task 0.9: adaptive-controller fire detection.
|
||||
//
|
||||
// A controller "fires" iff it made an ADAPTIVE INTERVENTION this
|
||||
|
||||
@@ -1981,3 +1981,51 @@ clean (warnings unchanged from baseline). `cargo test -p ml --lib
|
||||
--no-run` clean. No fingerprint change — buffer layouts and ISV
|
||||
slots unchanged from the kernel's perspective.
|
||||
|
||||
## Plan C Task 5 (2026-04-29) — `train_active_frac` HEALTH_DIAG metric
|
||||
|
||||
Added the training-rollout counterpart to the existing
|
||||
`val_active_frac` metric so Plan D's L3 verification gate has a
|
||||
direct signal that Thompson is generating directional exploration
|
||||
during training (target: `train_active_frac > 0.40`, i.e. at least
|
||||
40% of training-rollout actions land on Short or Long rather than
|
||||
Hold/Flat). `val_active_frac` is sourced from the eval-backtest
|
||||
kernel's per-direction `buy_count`/`sell_count`/`hold_count`
|
||||
reduction; `train_active_frac` is sourced from the in-memory
|
||||
`monitor.action_counts` aggregator (12-bin layout, `dir*3 + mag`,
|
||||
direction ∈ {Short=0, Hold=1, Long=2, Flat=3}). Active fraction =
|
||||
`(Σ bins[0..3] + Σ bins[6..9]) / Σ bins[0..12]`.
|
||||
|
||||
Wire-up:
|
||||
|
||||
- `crates/ml/src/trainers/dqn/trainer/mod.rs` — added
|
||||
`last_train_active_frac: f32` field on `DQNTrainer` (cached each
|
||||
epoch from the in-loop monitor; consumed by the next epoch's
|
||||
`consume_validation_loss` so the two `*_active_frac` lines are
|
||||
emitted adjacent in HEALTH_DIAG output).
|
||||
- `crates/ml/src/trainers/dqn/trainer/constructor.rs` —
|
||||
`last_train_active_frac: 0.0_f32` initialisation (zero until the
|
||||
first epoch finishes its training rollout).
|
||||
- `crates/ml/src/trainers/dqn/trainer/training_loop.rs` —
|
||||
populates `self.last_train_active_frac` in the per-epoch metrics
|
||||
block immediately after the existing `dist_q/dist_h/dist_f` +
|
||||
`ent_dir`/`ent_mag` calculations, reusing the already-computed
|
||||
`dir0` (Short) and `dir2` (Long) per-direction shares (full-12-bin
|
||||
denominator, identical to `monitor.action_counts.iter().sum()`),
|
||||
with the same `action_total_epoch > 0` guard. Clamped to
|
||||
`[0.0, 1.0]` defensively.
|
||||
- `crates/ml/src/trainers/dqn/trainer/metrics.rs` — emits
|
||||
`HEALTH_DIAG[{}]: train_active_frac=X.XXXX (Long+Short during
|
||||
training rollout)` from `consume_validation_loss` immediately
|
||||
after the existing `val [...]` block (which contains
|
||||
`val_active_frac`). The two metrics now appear next to each
|
||||
other in every epoch's HEALTH_DIAG output: `val_active_frac`
|
||||
measures direction commitment during the eval backtest while
|
||||
`train_active_frac` measures it during training rollout —
|
||||
divergence localises whether Thompson is stochastic enough at
|
||||
experience-collection time even when eval (deterministic argmax)
|
||||
collapses to Hold/Flat.
|
||||
|
||||
No fingerprint change — buffer layouts, ISV slots, and kernel
|
||||
signatures are untouched. Verification:
|
||||
`SQLX_OFFLINE=true cargo check -p ml --lib` clean.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user