spec: adaptive learning dynamics — LearningHealth + 4 gems + 4 pearls + 8 novels

Fixes Q-value collapse via unified LearningHealth signal that senses training health (6 components) and continuously adapts: - CQL regularization (regime + health gated) - Gradient budget (IQN/CQL/Ens/C51 dynamic allocation) - Tau target EMA (health-coupled) - Expected SARSA temperature (continuous, no hardcoded threshold) Plus 4 pearls (PER priorities, spectral detection, gradient consistency, adaptive gamma) and 8 novels (self-distillation, barrier loss, plasticity injection, CF curriculum, information bottleneck, ensemble oracle, contrarian override, meta-Q network). Core principle: training hyperparameters are OUTPUTS of the temporal pipeline, not static schedules. The system meta-learns its own settings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 18:55:26 +02:00
parent 6e0cb3d62c
commit ff5e578bc3
1 changed files with 361 additions and 0 deletions
--- a/docs/superpowers/specs/2026-04-20-adaptive-learning-dynamics-design.md
+++ b/docs/superpowers/specs/2026-04-20-adaptive-learning-dynamics-design.md
@@ -0,0 +1,361 @@
+# Adaptive Learning Dynamics — LearningHealth System
+
+## Problem
+
+The DQN trading model collapses to uniform Q-values across all actions after ~8 epochs of training. The 4 direction branches converge to identical Q-values, making argmax essentially random. WinRate stuck at 15-20% (random baseline), val_Sharpe_raw at -0.48.
+
+The collapse is **architectural**, not a single sign flip. It results from a fixed-point attractor formed by interaction of multiple training mechanisms:
+
+1. **Stochastic Expected SARSA target** (`c51_loss_kernel.cu:605`): when Q-gap is small, softmax weights become near-uniform → random sampling targets each action equally → online Q learns toward mean → reinforces collapse
+2. **CQL regularization (alpha=0.1)**: explicitly penalizes Q-differences via `logsumexp(Q) - Q(taken)` → pushes Q toward uniform, especially during collapse
+3. **Tau annealing to 0.0005**: target network freezes once collapsed, locking in bad state
+4. **IQN gradient budget (40%)**: starves C51 directional learning of gradient
+5. **Advantage centering**: dueling formula `Q = V + A - mean_a(A)` zeros out differences when advantage variance collapses
+
+These mechanisms interact via positive feedback: collapsing Q reduces gradient signal → reduces ability to differentiate → further collapse.
+
+The training metrics confirm the attractor:
+- Q-gap: peaks 0.66 at epoch 8, drops to 0.05 by epoch 27 (13x decline)
+- Q-var: drops 47x (0.118 → 0.0025) over same period
+- Atom utilization: 37% → 14% (C51 distribution narrowing)
+
+## Solution
+
+A unified `LearningHealth` signal that senses training health and continuously adapts learning hyperparameters. The collapse attractor becomes a temporary state with self-correcting recovery rather than a fixed point.
+
+**Core principle**: Training hyperparameters are OUTPUTS of the temporal pipeline, not static schedules. The system meta-learns its own training settings based on observed health.
+
+## Architecture
+
+Three layers:
+
+```
+┌─────────────────────────────────────────────────────────┐
+│ LAYER 3: Enhancements & Novel Mechanisms (P1-P4, N1-N8) │
+│  Each wired to LearningHealth signal where applicable   │
+└─────────────────────────────────────────────────────────┘
+                          ↑
+┌─────────────────────────────────────────────────────────┐
+│ LAYER 2: Core Adaptive Mechanisms (4 Gems)              │
+│  CQL, Gradient Budget, Tau, Expected SARSA temperature  │
+└─────────────────────────────────────────────────────────┘
+                          ↑
+┌─────────────────────────────────────────────────────────┐
+│ LAYER 1: Sensing — LearningHealth scalar [0, 1]         │
+│  6 components combined, EMA smoothed, broadcast to GPU  │
+└─────────────────────────────────────────────────────────┘
+```
+
+## Layer 1: LearningHealth Signal
+
+A scalar `learning_health ∈ [0, 1]` computed at each epoch boundary, broadcast via pinned device-mapped memory.
+
+### Composition
+
+```
+learning_health = 0.30 × q_gap_norm
+                + 0.20 × q_var_norm
+                + 0.15 × atom_util_norm
+                + 0.15 × grad_stable
+                + 0.10 × ens_agree
+                + 0.10 × grad_consistency
+```
+
+Where each component is a smoothstep-normalized signal in [0, 1]:
+
+| Component | Source | Healthy threshold | Formula |
+|---|---|---|---|
+| `q_gap_norm` | Q-gap (max - 2nd max) | 0.5 | `smoothstep(0.01, 0.5, q_gap_ema)` |
+| `q_var_norm` | Q-value variance | 0.1 | `smoothstep(0.001, 0.1, q_var_ema)` |
+| `atom_util_norm` | C51 atom utilization | 70% | `smoothstep(0.2, 0.7, atom_util)` |
+| `grad_stable` | Gradient norm stability | 10 | `1 - smoothstep(10, 100, grad_norm_ema)` |
+| `ens_agree` | Ensemble agreement | 1.0 | `1 - ensemble_disagreement` |
+| `grad_consistency` | Cosine of successive grads | 0.5 | `smoothstep(-0.2, 0.5, cos(g_t, g_{t-1}))` (P3) |
+
+### Smoothing & Safeguards
+
+- **Warmup**: `learning_health = 0.5` for first 3 epochs (neutral, allow initialization)
+- **EMA**: `health_t = 0.9 × health_{t-1} + 0.1 × health_new` (prevents flapping)
+- **Bounds**: clamp to `[0.2, 0.95]` (never fully trust or distrust)
+- **Hysteresis**: drop-threshold (0.4) and rise-threshold (0.6) differ to prevent oscillation
+
+### Storage
+
+- Extend ISV signals buffer from 12 to 13 entries (pinned device-mapped)
+- Add `learning_health` at index `[12]`
+- Update path: end of `process_epoch_boundary` → compute on host → pinned HtoD copy
+- Read path: kernels access via existing `isv_signals_ptr` (zero-copy, same stream)
+
+## Layer 2: Core Adaptive Mechanisms (Gems)
+
+### G2: Uncertainty-Gated CQL
+
+```
+cql_alpha_eff = cql_base × (1 - regime_stability) × health
+```
+
+- During collapse (`health=0`): CQL OFF (prevents reinforcing collapse)
+- During healthy training in volatile regime (`regime_stability=0`, `health=1`): CQL fully ON
+- During healthy training in stable regime (`regime_stability=1`, `health=1`): CQL OFF (trust Q)
+
+`cql_base = 0.1` (current static value).
+
+### G3: Health-Coupled Tau
+
+```
+tau_eff = max(tau_scheduled, 0.01 × (1 - health))
+```
+
+- During healthy training: tau follows existing cosine annealing schedule (0.005 → 0.0005)
+- During collapse (`health=0`): tau bumps up to 0.01 minimum, unfreezing target network
+- Target can track recovery rather than locking in collapsed state
+
+### G4: Temperature-Continuous Expected SARSA
+
+Replace stochastic sampling at `c51_loss_kernel.cu:605` with continuous-temperature softmax:
+
+```c
+// Compute softmax weights with health-adaptive temperature
+float tau_health_factor = 1.0f + 5.0f * (1.0f - learning_health);
+float tau_eff = fmaxf(q_gap_local, tau_floor) * tau_health_factor;
+// ... compute action_weights[a] = exp((eq[a] - max_eq) / tau_eff) ...
+
+// Stochastic sampling stays — but softmax sharpness varies with health
+// Healthy: tau_eff small → sharp softmax → near-argmax behavior → strong gradient
+// Collapsed: tau_eff large → uniform softmax → random sample → breaks symmetry
+```
+
+No hardcoded threshold. Continuous adaptation.
+
+### Gradient Budget Allocation
+
+```
+iqn_budget_eff = 0.10 + 0.30 × health
+cql_budget_eff = 0.10 × (1 - regime_stability) × health
+ens_budget_eff = 0.05
+c51_budget_eff = 1.0 - iqn_budget_eff - cql_budget_eff - ens_budget_eff
+```
+
+- Collapsed: C51 gets ~85% of gradient (focus on directional learning)
+- Healthy: balanced 40/10/5/45 split (current static allocation)
+
+## Layer 3: Pearls & Novels
+
+### P1: Health-Weighted PER Priorities
+
+Boost priorities of experiences where the action DIFFERED from the batch mean during collapse. These are the "directional learning signal" — they teach the network to differentiate.
+
+```
+priority = (|TD| + epsilon)^alpha × diversity_multiplier
+where diversity_multiplier = 1.0 + 2.0 × (1 - health) × |action - mean_action_in_batch|
+```
+
+When healthy: diversity_multiplier = 1.0 (standard PER).
+When collapsed: diverse-action experiences get up to 3x priority boost.
+
+### P2: Spectral Collapse Detection (additional health input)
+
+Compute SVD on the batch Q-value matrix `Q[B, n_actions]`. Use spectral gap as collapse detector:
+
+```
+spectral_gap = sigma_1 / sigma_2  // ratio of 1st to 2nd singular value
+spectral_gap_norm = 1 - smoothstep(2.0, 10.0, spectral_gap)
+```
+
+- Healthy Q: rank ≈ n_actions, spectral_gap small (e.g., 1.5)
+- Collapsed Q: rank ≈ 1, spectral_gap huge (e.g., 100+)
+
+This becomes an additional input to LearningHealth (could replace `q_var_norm` or be added as 7th component with rebalanced weights).
+
+### P3: Gradient Direction Consistency
+
+Already incorporated as 6th component in Layer 1. Implementation: track Adam's first moment vector `m_t` between successive updates, compute cosine similarity.
+
+```
+g_consistency = cos(m_t, m_{t-1}) = (m_t · m_{t-1}) / (||m_t|| × ||m_{t-1}||)
+```
+
+- High cosine (1.0): gradients pointing same direction → consistent learning
+- Low/negative cosine (-1.0): gradients contradicting → noisy/random learning
+
+### P4: Temporal-Coupled Gamma
+
+Replace fixed `gamma=0.99` with regime-coupled discount:
+
+```
+gamma_eff = gamma_base + 0.005 × (regime_stability - 0.5) - 0.05 × (1 - health)
+gamma_eff = clamp(gamma_eff, 0.9, 0.995)
+```
+
+- Stable regime + healthy: gamma_eff ≈ 0.995 (long-horizon value)
+- Transitioning regime + healthy: gamma_eff ≈ 0.985 (shorter horizon, reactive)
+- During collapse: gamma_eff ≈ 0.94 (focus on immediate clear reward signal)
+
+### N1: Temporal Self-Distillation
+
+Keep rolling snapshots of Q-network weights at high-health epochs. Add a KL distillation loss when collapse detected:
+
+```
+if learning_health < 0.4:
+    distill_loss = KL(current_Q || best_snapshot_Q)
+    total_loss += 0.1 × (1 - health) × distill_loss
+```
+
+Snapshots stored as `[5]` ring buffer of weight checkpoints, updated when `learning_health > 0.7` AND new max Q-gap observed. Self-healing via own past healthy state.
+
+### N2: Q-Gap Barrier Constraint
+
+Explicit loss term forcing Q-gap above a minimum:
+
+```
+min_required_q_gap = 0.05 × health  // higher threshold when healthy
+barrier_loss = max(0, min_required_q_gap - current_q_gap)^2
+total_loss += 0.5 × barrier_loss
+```
+
+Makes Q-collapse literally high-loss. The barrier scales with health: aggressive enforcement when healthy, relaxed during recovery.
+
+### N3: Plasticity Injection
+
+Replace fixed shrink-perturb schedule (`shrink_perturb_interval`) with health-triggered:
+
+```
+if learning_health < 0.3 for >= 3 consecutive epochs:
+    trigger shrink_perturb (existing kernel)
+    log "Plasticity injection triggered: health=X for N epochs"
+```
+
+Existing `shrink_perturb_alpha` and `shrink_perturb_sigma` reused. Health-triggered = adaptive, not periodic.
+
+### N4: Counterfactual Curriculum via Health
+
+Current: 50% of experiences are counterfactual (fixed in `env_step` kernel).
+
+Adaptive ratio:
+
+```
+cf_ratio_eff = 0.5 + 0.3 × (1 - health)
+```
+
+- Healthy: 50% CF (current default)
+- Collapsed: 80% CF (more "what if I did opposite" training to escape collapse)
+
+Implementation: pass `cf_ratio_eff` to env_step kernel as parameter, replace hardcoded `flip_u < 0.5f` with `flip_u < cf_ratio_eff`.
+
+### N5: Information Bottleneck Replacing CQL's Role
+
+Add an Information Bottleneck loss that penalizes lack of state-dependence in Q:
+
+```
+For each action a:
+    q_var_state = variance(Q(s_i, a) for s_i in batch)
+    
+ib_penalty = sum_a max(0, min_q_var - q_var_state[a])
+total_loss += ib_weight × (1 - health) × ib_penalty
+```
+
+`min_q_var = 0.01` (require Q-values to vary across states by at least 0.01).
+
+Conceptually opposite to CQL:
+- CQL: penalize Q for actions not seen → pushes Q UNIFORM
+- IB: penalize Q for not depending on state → pushes Q DIFFERENTIATED
+
+Both run simultaneously per user requirement (CQL for conservatism via G2, IB for state-dependence via N5). The forces are orthogonal: CQL operates per-state, IB operates across-states.
+
+### N6: Ensemble as Collapse Oracle
+
+We have 5 ensemble Q-network heads. Currently used for diversity bonus.
+
+Additional use as collapse early warning:
+
+```
+For each pair of ensemble heads (i, j):
+    pairwise_q_gap[i][j] = mean(|Q_i(s, a) - Q_j(s, a)|)
+    
+ensemble_collapse_score = 1 - smoothstep(0.01, 0.1, mean(pairwise_q_gap))
+```
+
+If ALL ensemble heads agree on uniform Q (low pairwise gap), catastrophic collapse imminent. Trigger N3 (plasticity injection) regardless of N3's normal threshold.
+
+### N7: Contrarian Override (severe collapse failsafe)
+
+If WinRate < 40% for ≥ 5 consecutive epochs AND learning_health < 0.3:
+
+```
+For next K epochs (K=2):
+    action = argmin(Q)  instead of argmax(Q)
+```
+
+Edgy mechanism: uses systematic anti-correlation as signal. Only activates during severe predictable collapse. Logs prominently. Disables automatically when WinRate recovers to ≥45%.
+
+### N8: Meta-Q Network
+
+Auxiliary 3-layer MLP (state_dim → 32 → 16 → 1) that predicts `P(collapse in next K epochs)`:
+
+- Input: aggregated state statistics over rolling window (Q-gap_ema, grad_norm_ema, atom_util, ens_disagreement, regime_stability)
+- Output: scalar in [0, 1]
+- Training: supervised — target is observed collapse status (1 if learning_health < 0.3 in next K epochs, else 0)
+- Use: prediction added as additional EARLY indicator into LearningHealth (catches collapse BEFORE it happens)
+
+Training the meta-network: small batch each epoch from rolling buffer of past health observations. Cheap to train (small network), powerful as leading indicator.
+
+## Logging (Debugging-First)
+
+Per-epoch log line with all components for debugging:
+
+```
+HEALTH_DIAG: health=0.73 (ema=0.71)
+  components [q_gap=0.82 q_var=0.65 atoms=0.91 grad_stable=0.74 ens_agree=0.55 grad_cos=0.62]
+  spectral [gap_norm=0.78]
+  effective [cql=0.027 iqn_budget=0.32 cql_budget=0.024 c51_budget=0.61 tau=0.005 sarsa_temp=1.2 gamma=0.985 cf_ratio=0.58]
+  novels [distill=off barrier=0.0 plasticity=ready ib=0.18 ensemble_collapse=0.05 contrarian=off meta_q_pred=0.12]
+```
+
+Every adaptive value is logged. If anything goes wrong, the log line tells you which mechanism is misfiring.
+
+## Files Changed
+
+| File | Change |
+|---|---|
+| `gpu_dqn_trainer.rs` | LearningHealth field, snapshot ring buffer, IB loss kernel, N1/N2 wiring |
+| `c51_loss_kernel.cu` | G4 (Expected SARSA temperature), N2 barrier loss, N5 IB loss kernel |
+| `fused_training.rs` | Gradient budget computation, tau computation, P3 grad consistency tracking |
+| `gpu_attention.rs` | (no changes — already aligned via state layout) |
+| `experience_kernels.cu` | N4 cf_ratio parameter, P1 diversity-weighted priority |
+| `gpu_replay_buffer.rs` | P1 priority computation update |
+| `training_loop.rs` | LearningHealth computation, N3 plasticity trigger, N6 ensemble oracle, N7 contrarian override, HEALTH_DIAG logging |
+| `metrics.rs` | Q-gap EMA, Q-var EMA, grad_norm EMA tracking |
+| `ml_core/state_layout.rs` (or new file) | LEARNING_HEALTH_INDEX = 12 constant |
+| `ml_core/isv_signals.rs` (if exists) | Extend buffer size from 12 to 13 |
+| `c51_loss_kernel.cu` | N5 IB penalty kernel |
+| New: `meta_q_network.rs` | N8 meta-Q network (small MLP, separate from main Q) |
+| New: `q_snapshot.rs` | N1 ring buffer of historical Q-network snapshots |
+
+## Testing
+
+1. **Unit tests for LearningHealth composition**: feed known component values, verify scalar output
+2. **Smoke test verifying log line presence**: HEALTH_DIAG appears every epoch with all fields populated
+3. **Collapse-recovery test**: artificially induce Q-collapse (zero out a layer), verify mechanisms activate
+4. **Comparison run**: train baseline (current) vs adaptive (this spec) for 50 epochs on local fxcache, compare:
+   - Q-gap trajectory (should not collapse)
+   - val_Sharpe trajectory (should improve, not stuck at -0.48)
+   - WinRate trajectory (should rise above 50% baseline)
+5. **Hyperparameter sweep**: vary `cql_base`, `health_weights`, smoothing rate to find robust defaults
+
+## Success Criteria
+
+- Q-gap stays above 0.1 throughout 50 epochs (no collapse)
+- Q-var stays above 0.05 throughout (sustained variance)
+- Atom utilization stays above 30% (C51 distribution healthy)
+- WinRate exceeds 35% on validation by epoch 20
+- val_Sharpe_raw improves by epoch 30 (not stuck at -0.48)
+- HEALTH_DIAG log shows continuous adaptation (no constant values)
+- Plasticity injection triggers <= 3 times in 50 epochs (rare automatic recovery)
+- Contrarian override never triggers (severe failsafe should not be needed)
+
+## Out of Scope
+
+- Tuning the WEIGHTS of the 6 health components (assumed reasonable defaults, can tune later)
+- Replacing the entire dueling architecture (advantage centering stays)
+- Implementing curriculum learning beyond N4 (counterfactual curriculum)
+- Multi-instrument training extensions