foxhunt/docs/superpowers/specs/2026-04-20-adaptive-learning-dynamics-design.md

Adaptive Learning Dynamics — LearningHealth System

Problem

The DQN trading model collapses to uniform Q-values across all actions after ~8 epochs of training. The 4 direction branches converge to identical Q-values, making argmax essentially random. WinRate stuck at 15-20% (random baseline), val_Sharpe_raw at -0.48.

The collapse is architectural, not a single sign flip. It results from a fixed-point attractor formed by interaction of multiple training mechanisms:

1. Stochastic Expected SARSA target (c51_loss_kernel.cu:605): when Q-gap is small, softmax weights become near-uniform → random sampling targets each action equally → online Q learns toward mean → reinforces collapse
2. CQL regularization (alpha=0.1): explicitly penalizes Q-differences via logsumexp(Q) - Q(taken) → pushes Q toward uniform, especially during collapse
3. Tau annealing to 0.0005: target network freezes once collapsed, locking in bad state
4. IQN gradient budget (40%): starves C51 directional learning of gradient
5. Advantage centering: dueling formula Q = V + A - mean_a(A) zeros out differences when advantage variance collapses

These mechanisms interact via positive feedback: collapsing Q reduces gradient signal → reduces ability to differentiate → further collapse.

The training metrics confirm the attractor:

• Q-gap: peaks 0.66 at epoch 8, drops to 0.05 by epoch 27 (13x decline)
• Q-var: drops 47x (0.118 → 0.0025) over same period
• Atom utilization: 37% → 14% (C51 distribution narrowing)

Solution

A unified LearningHealth signal that senses training health and continuously adapts learning hyperparameters. The collapse attractor becomes a temporary state with self-correcting recovery rather than a fixed point.

Core principle: Training hyperparameters are OUTPUTS of the temporal pipeline, not static schedules. The system meta-learns its own training settings based on observed health.

Architecture

Three layers:

┌─────────────────────────────────────────────────────────┐
│ LAYER 3: Enhancements & Novel Mechanisms (P1-P4, N1-N8) │
│  Each wired to LearningHealth signal where applicable   │
└─────────────────────────────────────────────────────────┘
                          ↑
┌─────────────────────────────────────────────────────────┐
│ LAYER 2: Core Adaptive Mechanisms (4 Gems)              │
│  CQL, Gradient Budget, Tau, Expected SARSA temperature  │
└─────────────────────────────────────────────────────────┘
                          ↑
┌─────────────────────────────────────────────────────────┐
│ LAYER 1: Sensing — LearningHealth scalar [0, 1]         │
│  6 components combined, EMA smoothed, broadcast to GPU  │
└─────────────────────────────────────────────────────────┘

Layer 1: LearningHealth Signal

A scalar learning_health ∈ [0, 1] computed at each epoch boundary, broadcast via pinned device-mapped memory.

Composition

learning_health = 0.25 × q_gap_norm
                + 0.15 × q_var_norm
                + 0.15 × atom_util_norm
                + 0.15 × grad_stable
                + 0.10 × ens_agree
                + 0.10 × grad_consistency
                + 0.10 × spectral_gap_norm    // P2 — included as 7th component

Where each component is a smoothstep-normalized signal in [0, 1]:

Component	Source	Healthy threshold	Formula
q_gap_norm	Q-gap (max - 2nd max)	0.5	smoothstep(0.01, 0.5, q_gap_ema)
q_var_norm	Q-value variance	0.1	smoothstep(0.001, 0.1, q_var_ema)
atom_util_norm	C51 atom utilization	70%	smoothstep(0.2, 0.7, atom_util)
grad_stable	Gradient norm stability	10	1 - smoothstep(10, 100, grad_norm_ema)
ens_agree	Ensemble agreement	1.0	1 - ensemble_disagreement
grad_consistency	Cosine of successive grads	0.5	smoothstep(-0.2, 0.5, cos(g_t, g_{t-1})) (P3)
spectral_gap_norm	SVD on Q matrix	2.0	1 - smoothstep(2.0, 10.0, sigma_1/sigma_2) (P2)

Meta-Q prediction (N8) does NOT enter the composition — it is logged separately as a leading indicator (see logging section). If meta-Q reliably predicts collapse, future revisions can incorporate it into composition.

Smoothing & Safeguards

• Warmup: learning_health = 0.5 for first 3 epochs (neutral, allow initialization)
• EMA: health_t = 0.9 × health_{t-1} + 0.1 × health_new (prevents flapping)
• Bounds: clamp to [0.2, 0.95] (never fully trust or distrust)
• Hysteresis: drop-threshold (0.4) and rise-threshold (0.6) differ to prevent oscillation

Storage

• Extend ISV signals buffer from 12 to 13 entries (pinned device-mapped)
• Add learning_health at index [12]
• Update path: end of process_epoch_boundary → compute on host → pinned HtoD copy
• Read path: kernels access via existing isv_signals_ptr (zero-copy, same stream)

Layer 2: Core Adaptive Mechanisms (Gems G2-G5)

G1 is the LearningHealth signal itself (Layer 1). Layer 2 contains the four gems that consume it.

G2: Uncertainty-Gated CQL

cql_alpha_eff = cql_base × (1 - regime_stability) × health

• During collapse (health=0): CQL OFF (prevents reinforcing collapse)
• During healthy training in volatile regime (regime_stability=0, health=1): CQL fully ON
• During healthy training in stable regime (regime_stability=1, health=1): CQL OFF (trust Q)

cql_base = 0.1 (current static value).

G3: Health-Coupled Tau

tau_eff = max(tau_scheduled, 0.01 × (1 - health))

• During healthy training: tau follows existing cosine annealing schedule (0.005 → 0.0005)
• During collapse (health=0): tau bumps up to 0.01 minimum, unfreezing target network
• Target can track recovery rather than locking in collapsed state

G4: Temperature-Continuous Expected SARSA

Replace stochastic sampling at c51_loss_kernel.cu:605 with continuous-temperature softmax:

// Compute softmax weights with health-adaptive temperature
float tau_health_factor = 1.0f + 5.0f * (1.0f - learning_health);
float tau_eff = fmaxf(q_gap_local, tau_floor) * tau_health_factor;
// ... compute action_weights[a] = exp((eq[a] - max_eq) / tau_eff) ...

// Stochastic sampling stays — but softmax sharpness varies with health
// Healthy: tau_eff small → sharp softmax → near-argmax behavior → strong gradient
// Collapsed: tau_eff large → uniform softmax → random sample → breaks symmetry

No hardcoded threshold. Continuous adaptation.

G5: Gradient Budget Allocation

iqn_budget_eff = 0.10 + 0.30 × health
cql_budget_eff = 0.10 × (1 - regime_stability) × health
ens_budget_eff = 0.05
c51_budget_eff = 1.0 - iqn_budget_eff - cql_budget_eff - ens_budget_eff

• Collapsed: C51 gets ~85% of gradient (focus on directional learning)
• Healthy: balanced 40/10/5/45 split (current static allocation)

Layer 3: Pearls & Novels

P1: Health-Weighted PER Priorities

Boost priorities of experiences where the action DIFFERED from the batch mean during collapse. These are the "directional learning signal" — they teach the network to differentiate.

priority = (|TD| + epsilon)^alpha × diversity_multiplier
where diversity_multiplier = 1.0 + 2.0 × (1 - health) × |action - mean_action_in_batch|

When healthy: diversity_multiplier = 1.0 (standard PER). When collapsed: diverse-action experiences get up to 3x priority boost.

P2: Spectral Collapse Detection

Already incorporated as 7th component in Layer 1 composition. Implementation: compute SVD on the batch Q-value matrix Q[B, n_actions] per epoch.

spectral_gap = sigma_1 / sigma_2  // ratio of 1st to 2nd singular value
spectral_gap_norm = 1 - smoothstep(2.0, 10.0, spectral_gap)

• Healthy Q: rank ≈ n_actions, spectral_gap small (e.g., 1.5) → spectral_gap_norm near 1.0
• Collapsed Q: rank ≈ 1, spectral_gap huge (e.g., 100+) → spectral_gap_norm near 0.0

SVD computed cheaply via cuSOLVER on Q[B, n_actions] once per epoch. For our 4 direction branches, n_actions=4, so SVD is on a [B, 4] matrix — fast.

P3: Gradient Direction Consistency

Already incorporated as 6th component in Layer 1. Implementation: track Adam's first moment vector m_t between successive updates, compute cosine similarity.

g_consistency = cos(m_t, m_{t-1}) = (m_t · m_{t-1}) / (||m_t|| × ||m_{t-1}||)

• High cosine (1.0): gradients pointing same direction → consistent learning
• Low/negative cosine (-1.0): gradients contradicting → noisy/random learning

P4: Temporal-Coupled Gamma

Replace fixed gamma=0.99 with regime-coupled discount:

gamma_eff = gamma_base + 0.005 × (regime_stability - 0.5) - 0.05 × (1 - health)
gamma_eff = clamp(gamma_eff, 0.9, 0.995)

• Stable regime + healthy: gamma_eff ≈ 0.995 (long-horizon value)
• Transitioning regime + healthy: gamma_eff ≈ 0.985 (shorter horizon, reactive)
• During collapse: gamma_eff ≈ 0.94 (focus on immediate clear reward signal)

N1: Temporal Self-Distillation

Keep rolling snapshots of Q-network weights at high-health epochs. Add a KL distillation loss when collapse detected:

if learning_health < 0.4:
    distill_loss = KL(current_Q || best_snapshot_Q)
    total_loss += 0.1 × (1 - health) × distill_loss

Snapshots stored as [5] ring buffer of weight checkpoints, updated when learning_health > 0.7 AND new max Q-gap observed. Self-healing via own past healthy state.

N2: Q-Gap Barrier Constraint

Explicit loss term forcing Q-gap above a minimum:

min_required_q_gap = 0.05 × health  // higher threshold when healthy
barrier_loss = max(0, min_required_q_gap - current_q_gap)^2
total_loss += 0.5 × barrier_loss

Makes Q-collapse literally high-loss. The barrier scales with health: aggressive enforcement when healthy, relaxed during recovery.

N3: Plasticity Injection

Replace fixed shrink-perturb schedule (shrink_perturb_interval) with health-triggered:

if learning_health < 0.3 for >= 3 consecutive epochs:
    trigger shrink_perturb (existing kernel)
    log "Plasticity injection triggered: health=X for N epochs"

Existing shrink_perturb_alpha and shrink_perturb_sigma reused. Health-triggered = adaptive, not periodic.

N4: Counterfactual Curriculum via Health

Current: 50% of experiences are counterfactual (fixed in env_step kernel).

Adaptive ratio:

cf_ratio_eff = 0.5 + 0.3 × (1 - health)

• Healthy: 50% CF (current default)
• Collapsed: 80% CF (more "what if I did opposite" training to escape collapse)

Implementation: pass cf_ratio_eff to env_step kernel as parameter, replace hardcoded flip_u < 0.5f with flip_u < cf_ratio_eff.

N5: Information Bottleneck Replacing CQL's Role

Add an Information Bottleneck loss that penalizes lack of state-dependence in Q:

For each action a:
    q_var_state = variance(Q(s_i, a) for s_i in batch)
    
ib_penalty = sum_a max(0, min_q_var - q_var_state[a])
total_loss += ib_weight × (1 - health) × ib_penalty

min_q_var = 0.01 (require Q-values to vary across states by at least 0.01).

Conceptually opposite to CQL:

• CQL: penalize Q for actions not seen → pushes Q UNIFORM
• IB: penalize Q for not depending on state → pushes Q DIFFERENTIATED

Both run simultaneously per user requirement (CQL for conservatism via G2, IB for state-dependence via N5). The forces are orthogonal: CQL operates per-state, IB operates across-states.

N6: Ensemble as Collapse Oracle

We have 5 ensemble Q-network heads. Currently used for diversity bonus.

Additional use as collapse early warning:

For each pair of ensemble heads (i, j):
    pairwise_q_gap[i][j] = mean(|Q_i(s, a) - Q_j(s, a)|)
    
ensemble_collapse_score = 1 - smoothstep(0.01, 0.1, mean(pairwise_q_gap))

If ensemble_collapse_score > 0.8 for ≥ 1 epoch, trigger N3 (plasticity injection) immediately, bypassing N3's normal "3 consecutive epochs" requirement. Ensemble agreement on uniform Q is catastrophic — all heads have fallen into the same local minimum, recovery requires immediate weight perturbation.

N7: Contrarian Override (severe collapse failsafe)

If WinRate < 40% for ≥ 5 consecutive epochs AND learning_health < 0.3:

For next K epochs (K=2):
    action = argmin(Q)  instead of argmax(Q)

Edgy mechanism: uses systematic anti-correlation as signal. Only activates during severe predictable collapse. Logs prominently. Disables automatically when WinRate recovers to ≥45%.

N8: Meta-Q Network

Auxiliary 3-layer MLP that predicts P(collapse in next K epochs):

• Architecture: 7 inputs → 32 hidden (ReLU) → 16 hidden (ReLU) → 1 sigmoid output
• Inputs (7-dim feature vector, computed each epoch):
  1. q_gap_ema
  2. grad_norm_ema (log-scaled)
  3. atom_util
  4. ens_disagreement
  5. regime_stability (from ISV[11])
  6. spectral_gap_norm (from P2)
  7. grad_consistency (from P3)
• Output: scalar in [0, 1] — predicted collapse probability for K=5 epochs ahead
• Training: BCE loss against observed collapse status (1 if learning_health < 0.3 occurred in next K epochs, else 0)
• Storage: rolling buffer of last 200 (input, target) pairs
• Optimizer: separate Adam (lr=1e-3), small batch (32) once per epoch — negligible compute
• Wiring: prediction meta_q_pred is logged in HEALTH_DIAG as a leading indicator. NOT added to LearningHealth composition until validated to predict reliably (≥ 80% accuracy on held-out window). Future revision can incorporate.

Logging (Debugging-First)

Per-epoch log line with all components for debugging:

HEALTH_DIAG: health=0.73 (ema=0.71)
  components [q_gap=0.82 q_var=0.65 atoms=0.91 grad_stable=0.74 ens_agree=0.55 grad_cos=0.62 spectral=0.78]
  effective [cql_alpha=0.027 iqn_budget=0.32 cql_budget=0.024 c51_budget=0.61 tau=0.005 sarsa_tau=1.2 gamma=0.985 cf_ratio=0.58]
  novels [distill=off barrier=0.0 plasticity=ready ib=0.18 ensemble_collapse=0.05 contrarian=off meta_q_pred=0.12]

Every adaptive value is logged. If anything goes wrong, the log line tells you which mechanism is misfiring.

Files Changed

File	Change
File	Change
---	---
gpu_dqn_trainer.rs	LearningHealth field, snapshot ring buffer, N1/N2 wiring
c51_loss_kernel.cu	G4 (Expected SARSA temperature), N2 barrier loss, N5 IB penalty kernel, P2 SVD spectral gap computation
fused_training.rs	G5 gradient budget computation, G3 tau computation, P3 grad consistency tracking, P4 adaptive gamma
experience_kernels.cu	N4 cf_ratio parameter (replace hardcoded flip_u < 0.5f)
gpu_replay_buffer.rs	P1 priority computation update (action-diversity multiplier)
training_loop.rs	LearningHealth computation, N3 plasticity trigger, N6 ensemble oracle, N7 contrarian override, HEALTH_DIAG logging
metrics.rs	Q-gap EMA, Q-var EMA, grad_norm EMA tracking
ml-core/src/isv_signals.rs (or wherever ISV buffer is defined)	Extend buffer size from 12 to 13, add LEARNING_HEALTH_INDEX = 12 constant
New: crates/ml/src/cuda_pipeline/meta_q_network.rs	N8 meta-Q network (small MLP, separate from main Q)
New: crates/ml/src/cuda_pipeline/q_snapshot.rs	N1 ring buffer of historical Q-network snapshots

Testing

1. Unit tests for LearningHealth composition: feed known component values, verify scalar output
2. Smoke test verifying log line presence: HEALTH_DIAG appears every epoch with all fields populated
3. Collapse-recovery test: artificially induce Q-collapse (zero out a layer), verify mechanisms activate
4. Comparison run: train baseline (current) vs adaptive (this spec) for 50 epochs on local fxcache, compare:
   • Q-gap trajectory (should not collapse)
   • val_Sharpe trajectory (should improve, not stuck at -0.48)
   • WinRate trajectory (should rise above 50% baseline)
5. Hyperparameter sweep: vary cql_base, health_weights, smoothing rate to find robust defaults

Success Criteria

• Q-gap stays above 0.1 throughout 50 epochs (no collapse)
• Q-var stays above 0.05 throughout (sustained variance)
• Atom utilization stays above 30% (C51 distribution healthy)
• WinRate exceeds 55% on validation by epoch 20 (genuinely-learning policy beats baseline ~25%)
• val_Sharpe_raw improves by epoch 30 (not stuck at -0.48)
• HEALTH_DIAG log shows continuous adaptation (no constant values)
• Plasticity injection triggers <= 3 times in 50 epochs (rare automatic recovery)
• Contrarian override never triggers (severe failsafe should not be needed)

Out of Scope

• Tuning the WEIGHTS of the 6 health components (assumed reasonable defaults, can tune later)
• Replacing the entire dueling architecture (advantage centering stays)
• Implementing curriculum learning beyond N4 (counterfactual curriculum)
• Multi-instrument training extensions