# Adaptive Learning Dynamics — LearningHealth System

## Problem

The DQN trading model collapses to uniform Q-values across all actions after ~8 epochs of training. The 4 direction branches converge to identical Q-values, making argmax essentially random. WinRate stuck at 15-20% (random baseline), val_Sharpe_raw at -0.48.

The collapse is **architectural**, not a single sign flip. It results from a fixed-point attractor formed by interaction of multiple training mechanisms:

1. **Stochastic Expected SARSA target** (`c51_loss_kernel.cu:605`): when Q-gap is small, softmax weights become near-uniform → random sampling targets each action equally → online Q learns toward mean → reinforces collapse
2. **CQL regularization (alpha=0.1)**: explicitly penalizes Q-differences via `logsumexp(Q) - Q(taken)` → pushes Q toward uniform, especially during collapse
3. **Tau annealing to 0.0005**: target network freezes once collapsed, locking in bad state
4. **IQN gradient budget (40%)**: starves C51 directional learning of gradient
5. **Advantage centering**: dueling formula `Q = V + A - mean_a(A)` zeros out differences when advantage variance collapses

These mechanisms interact via positive feedback: collapsing Q reduces gradient signal → reduces ability to differentiate → further collapse.

The training metrics confirm the attractor:
- Q-gap: peaks 0.66 at epoch 8, drops to 0.05 by epoch 27 (13x decline)
- Q-var: drops 47x (0.118 → 0.0025) over same period
- Atom utilization: 37% → 14% (C51 distribution narrowing)

## Solution

A unified `LearningHealth` signal that senses training health and continuously adapts learning hyperparameters. The collapse attractor becomes a temporary state with self-correcting recovery rather than a fixed point.

**Core principle**: Training hyperparameters are OUTPUTS of the temporal pipeline, not static schedules. The system meta-learns its own training settings based on observed health.

## Architecture

Three layers:

```
┌─────────────────────────────────────────────────────────┐
│ LAYER 3: Enhancements & Novel Mechanisms (P1-P4, N1-N8) │
│  Each wired to LearningHealth signal where applicable   │
└─────────────────────────────────────────────────────────┘
                          ↑
┌─────────────────────────────────────────────────────────┐
│ LAYER 2: Core Adaptive Mechanisms (4 Gems)              │
│  CQL, Gradient Budget, Tau, Expected SARSA temperature  │
└─────────────────────────────────────────────────────────┘
                          ↑
┌─────────────────────────────────────────────────────────┐
│ LAYER 1: Sensing — LearningHealth scalar [0, 1]         │
│  6 components combined, EMA smoothed, broadcast to GPU  │
└─────────────────────────────────────────────────────────┘
```

## Layer 1: LearningHealth Signal

A scalar `learning_health ∈ [0, 1]` computed at each epoch boundary, broadcast via pinned device-mapped memory.

### Composition

```
learning_health = 0.25 × q_gap_norm
                + 0.15 × q_var_norm
                + 0.15 × atom_util_norm
                + 0.15 × grad_stable
                + 0.10 × ens_agree
                + 0.10 × grad_consistency
                + 0.10 × spectral_gap_norm    // P2 — included as 7th component
```

Where each component is a smoothstep-normalized signal in [0, 1]:

| Component | Source | Healthy threshold | Formula |
|---|---|---|---|
| `q_gap_norm` | Q-gap (max - 2nd max) | 0.5 | `smoothstep(0.01, 0.5, q_gap_ema)` |
| `q_var_norm` | Q-value variance | 0.1 | `smoothstep(0.001, 0.1, q_var_ema)` |
| `atom_util_norm` | C51 atom utilization | 70% | `smoothstep(0.2, 0.7, atom_util)` |
| `grad_stable` | Gradient norm stability | 10 | `1 - smoothstep(10, 100, grad_norm_ema)` |
| `ens_agree` | Ensemble agreement | 1.0 | `1 - ensemble_disagreement` |
| `grad_consistency` | Cosine of successive grads | 0.5 | `smoothstep(-0.2, 0.5, cos(g_t, g_{t-1}))` (P3) |
| `spectral_gap_norm` | SVD on Q matrix | 2.0 | `1 - smoothstep(2.0, 10.0, sigma_1/sigma_2)` (P2) |

**Meta-Q prediction (N8) does NOT enter the composition** — it is logged separately as a leading indicator (see logging section). If meta-Q reliably predicts collapse, future revisions can incorporate it into composition.

### Smoothing & Safeguards

- **Warmup**: `learning_health = 0.5` for first 3 epochs (neutral, allow initialization)
- **EMA**: `health_t = 0.9 × health_{t-1} + 0.1 × health_new` (prevents flapping)
- **Bounds**: clamp to `[0.2, 0.95]` (never fully trust or distrust)
- **Hysteresis**: drop-threshold (0.4) and rise-threshold (0.6) differ to prevent oscillation

### Storage

- Extend ISV signals buffer from 12 to 13 entries (pinned device-mapped)
- Add `learning_health` at index `[12]`
- Update path: end of `process_epoch_boundary` → compute on host → pinned HtoD copy
- Read path: kernels access via existing `isv_signals_ptr` (zero-copy, same stream)

## Layer 2: Core Adaptive Mechanisms (Gems G2-G5)

G1 is the LearningHealth signal itself (Layer 1). Layer 2 contains the four gems that consume it.

### G2: Uncertainty-Gated CQL

```
cql_alpha_eff = cql_base × (1 - regime_stability) × health
```

- During collapse (`health=0`): CQL OFF (prevents reinforcing collapse)
- During healthy training in volatile regime (`regime_stability=0`, `health=1`): CQL fully ON
- During healthy training in stable regime (`regime_stability=1`, `health=1`): CQL OFF (trust Q)

`cql_base = 0.1` (current static value).

### G3: Health-Coupled Tau

```
tau_eff = max(tau_scheduled, 0.01 × (1 - health))
```

- During healthy training: tau follows existing cosine annealing schedule (0.005 → 0.0005)
- During collapse (`health=0`): tau bumps up to 0.01 minimum, unfreezing target network
- Target can track recovery rather than locking in collapsed state

### G4: Temperature-Continuous Expected SARSA

Replace stochastic sampling at `c51_loss_kernel.cu:605` with continuous-temperature softmax:

```c
// Compute softmax weights with health-adaptive temperature
float tau_health_factor = 1.0f + 5.0f * (1.0f - learning_health);
float tau_eff = fmaxf(q_gap_local, tau_floor) * tau_health_factor;
// ... compute action_weights[a] = exp((eq[a] - max_eq) / tau_eff) ...

// Stochastic sampling stays — but softmax sharpness varies with health
// Healthy: tau_eff small → sharp softmax → near-argmax behavior → strong gradient
// Collapsed: tau_eff large → uniform softmax → random sample → breaks symmetry
```

No hardcoded threshold. Continuous adaptation.

### G5: Gradient Budget Allocation

```
iqn_budget_eff = 0.10 + 0.30 × health
cql_budget_eff = 0.10 × (1 - regime_stability) × health
ens_budget_eff = 0.05
c51_budget_eff = 1.0 - iqn_budget_eff - cql_budget_eff - ens_budget_eff
```

- Collapsed: C51 gets ~85% of gradient (focus on directional learning)
- Healthy: balanced 40/10/5/45 split (current static allocation)

## Layer 3: Pearls & Novels

### P1: Health-Weighted PER Priorities

Boost priorities of experiences where the action DIFFERED from the batch mean during collapse. These are the "directional learning signal" — they teach the network to differentiate.

```
priority = (|TD| + epsilon)^alpha × diversity_multiplier
where diversity_multiplier = 1.0 + 2.0 × (1 - health) × |action - mean_action_in_batch|
```

When healthy: diversity_multiplier = 1.0 (standard PER).
When collapsed: diverse-action experiences get up to 3x priority boost.

### P2: Spectral Collapse Detection

Already incorporated as 7th component in Layer 1 composition. Implementation: compute SVD on the batch Q-value matrix `Q[B, n_actions]` per epoch.

```
spectral_gap = sigma_1 / sigma_2  // ratio of 1st to 2nd singular value
spectral_gap_norm = 1 - smoothstep(2.0, 10.0, spectral_gap)
```

- Healthy Q: rank ≈ n_actions, spectral_gap small (e.g., 1.5) → spectral_gap_norm near 1.0
- Collapsed Q: rank ≈ 1, spectral_gap huge (e.g., 100+) → spectral_gap_norm near 0.0

SVD computed cheaply via cuSOLVER on `Q[B, n_actions]` once per epoch. For our 4 direction branches, n_actions=4, so SVD is on a `[B, 4]` matrix — fast.

### P3: Gradient Direction Consistency

Already incorporated as 6th component in Layer 1. Implementation: track Adam's first moment vector `m_t` between successive updates, compute cosine similarity.

```
g_consistency = cos(m_t, m_{t-1}) = (m_t · m_{t-1}) / (||m_t|| × ||m_{t-1}||)
```

- High cosine (1.0): gradients pointing same direction → consistent learning
- Low/negative cosine (-1.0): gradients contradicting → noisy/random learning

### P4: Temporal-Coupled Gamma

Replace fixed `gamma=0.99` with regime-coupled discount:

```
gamma_eff = gamma_base + 0.005 × (regime_stability - 0.5) - 0.05 × (1 - health)
gamma_eff = clamp(gamma_eff, 0.9, 0.995)
```

- Stable regime + healthy: gamma_eff ≈ 0.995 (long-horizon value)
- Transitioning regime + healthy: gamma_eff ≈ 0.985 (shorter horizon, reactive)
- During collapse: gamma_eff ≈ 0.94 (focus on immediate clear reward signal)

### N1: Temporal Self-Distillation

Keep rolling snapshots of Q-network weights at high-health epochs. Add a KL distillation loss when collapse detected:

```
if learning_health < 0.4:
    distill_loss = KL(current_Q || best_snapshot_Q)
    total_loss += 0.1 × (1 - health) × distill_loss
```

Snapshots stored as `[5]` ring buffer of weight checkpoints, updated when `learning_health > 0.7` AND new max Q-gap observed. Self-healing via own past healthy state.

### N2: Q-Gap Barrier Constraint

Explicit loss term forcing Q-gap above a minimum:

```
min_required_q_gap = 0.05 × health  // higher threshold when healthy
barrier_loss = max(0, min_required_q_gap - current_q_gap)^2
total_loss += 0.5 × barrier_loss
```

Makes Q-collapse literally high-loss. The barrier scales with health: aggressive enforcement when healthy, relaxed during recovery.

### N3: Plasticity Injection

Replace fixed shrink-perturb schedule (`shrink_perturb_interval`) with health-triggered:

```
if learning_health < 0.3 for >= 3 consecutive epochs:
    trigger shrink_perturb (existing kernel)
    log "Plasticity injection triggered: health=X for N epochs"
```

Existing `shrink_perturb_alpha` and `shrink_perturb_sigma` reused. Health-triggered = adaptive, not periodic.

### N4: Counterfactual Curriculum via Health

Current: 50% of experiences are counterfactual (fixed in `env_step` kernel).

Adaptive ratio:

```
cf_ratio_eff = 0.5 + 0.3 × (1 - health)
```

- Healthy: 50% CF (current default)
- Collapsed: 80% CF (more "what if I did opposite" training to escape collapse)

Implementation: pass `cf_ratio_eff` to env_step kernel as parameter, replace hardcoded `flip_u < 0.5f` with `flip_u < cf_ratio_eff`.

### N5: Information Bottleneck Replacing CQL's Role

Add an Information Bottleneck loss that penalizes lack of state-dependence in Q:

```
For each action a:
    q_var_state = variance(Q(s_i, a) for s_i in batch)
    
ib_penalty = sum_a max(0, min_q_var - q_var_state[a])
total_loss += ib_weight × (1 - health) × ib_penalty
```

`min_q_var = 0.01` (require Q-values to vary across states by at least 0.01).

Conceptually opposite to CQL:
- CQL: penalize Q for actions not seen → pushes Q UNIFORM
- IB: penalize Q for not depending on state → pushes Q DIFFERENTIATED

Both run simultaneously per user requirement (CQL for conservatism via G2, IB for state-dependence via N5). The forces are orthogonal: CQL operates per-state, IB operates across-states.

### N6: Ensemble as Collapse Oracle

We have 5 ensemble Q-network heads. Currently used for diversity bonus.

Additional use as collapse early warning:

```
For each pair of ensemble heads (i, j):
    pairwise_q_gap[i][j] = mean(|Q_i(s, a) - Q_j(s, a)|)
    
ensemble_collapse_score = 1 - smoothstep(0.01, 0.1, mean(pairwise_q_gap))
```

If ensemble_collapse_score > 0.8 for ≥ 1 epoch, trigger N3 (plasticity injection) immediately, bypassing N3's normal "3 consecutive epochs" requirement. Ensemble agreement on uniform Q is catastrophic — all heads have fallen into the same local minimum, recovery requires immediate weight perturbation.

### N7: Contrarian Override (severe collapse failsafe)

If WinRate < 40% for ≥ 5 consecutive epochs AND learning_health < 0.3:

```
For next K epochs (K=2):
    action = argmin(Q)  instead of argmax(Q)
```

Edgy mechanism: uses systematic anti-correlation as signal. Only activates during severe predictable collapse. Logs prominently. Disables automatically when WinRate recovers to ≥45%.

### N8: Meta-Q Network

Auxiliary 3-layer MLP that predicts `P(collapse in next K epochs)`:

- **Architecture**: 7 inputs → 32 hidden (ReLU) → 16 hidden (ReLU) → 1 sigmoid output
- **Inputs** (7-dim feature vector, computed each epoch):
  1. q_gap_ema
  2. grad_norm_ema (log-scaled)
  3. atom_util
  4. ens_disagreement
  5. regime_stability (from ISV[11])
  6. spectral_gap_norm (from P2)
  7. grad_consistency (from P3)
- **Output**: scalar in [0, 1] — predicted collapse probability for K=5 epochs ahead
- **Training**: BCE loss against observed collapse status (1 if learning_health < 0.3 occurred in next K epochs, else 0)
- **Storage**: rolling buffer of last 200 (input, target) pairs
- **Optimizer**: separate Adam (lr=1e-3), small batch (32) once per epoch — negligible compute
- **Wiring**: prediction `meta_q_pred` is logged in HEALTH_DIAG as a leading indicator. NOT added to LearningHealth composition until validated to predict reliably (≥ 80% accuracy on held-out window). Future revision can incorporate.

## Logging (Debugging-First)

Per-epoch log line with all components for debugging:

```
HEALTH_DIAG: health=0.73 (ema=0.71)
  components [q_gap=0.82 q_var=0.65 atoms=0.91 grad_stable=0.74 ens_agree=0.55 grad_cos=0.62 spectral=0.78]
  effective [cql_alpha=0.027 iqn_budget=0.32 cql_budget=0.024 c51_budget=0.61 tau=0.005 sarsa_tau=1.2 gamma=0.985 cf_ratio=0.58]
  novels [distill=off barrier=0.0 plasticity=ready ib=0.18 ensemble_collapse=0.05 contrarian=off meta_q_pred=0.12]
```

Every adaptive value is logged. If anything goes wrong, the log line tells you which mechanism is misfiring.

## Files Changed

| File | Change |
|---|---|
| File | Change |
|---|---|
| `gpu_dqn_trainer.rs` | LearningHealth field, snapshot ring buffer, N1/N2 wiring |
| `c51_loss_kernel.cu` | G4 (Expected SARSA temperature), N2 barrier loss, N5 IB penalty kernel, P2 SVD spectral gap computation |
| `fused_training.rs` | G5 gradient budget computation, G3 tau computation, P3 grad consistency tracking, P4 adaptive gamma |
| `experience_kernels.cu` | N4 cf_ratio parameter (replace hardcoded `flip_u < 0.5f`) |
| `gpu_replay_buffer.rs` | P1 priority computation update (action-diversity multiplier) |
| `training_loop.rs` | LearningHealth computation, N3 plasticity trigger, N6 ensemble oracle, N7 contrarian override, HEALTH_DIAG logging |
| `metrics.rs` | Q-gap EMA, Q-var EMA, grad_norm EMA tracking |
| `ml-core/src/isv_signals.rs` (or wherever ISV buffer is defined) | Extend buffer size from 12 to 13, add `LEARNING_HEALTH_INDEX = 12` constant |
| New: `crates/ml/src/cuda_pipeline/meta_q_network.rs` | N8 meta-Q network (small MLP, separate from main Q) |
| New: `crates/ml/src/cuda_pipeline/q_snapshot.rs` | N1 ring buffer of historical Q-network snapshots |

## Testing

1. **Unit tests for LearningHealth composition**: feed known component values, verify scalar output
2. **Smoke test verifying log line presence**: HEALTH_DIAG appears every epoch with all fields populated
3. **Collapse-recovery test**: artificially induce Q-collapse (zero out a layer), verify mechanisms activate
4. **Comparison run**: train baseline (current) vs adaptive (this spec) for 50 epochs on local fxcache, compare:
   - Q-gap trajectory (should not collapse)
   - val_Sharpe trajectory (should improve, not stuck at -0.48)
   - WinRate trajectory (should rise above 50% baseline)
5. **Hyperparameter sweep**: vary `cql_base`, `health_weights`, smoothing rate to find robust defaults

## Success Criteria

- Q-gap stays above 0.1 throughout 50 epochs (no collapse)
- Q-var stays above 0.05 throughout (sustained variance)
- Atom utilization stays above 30% (C51 distribution healthy)
- WinRate exceeds 55% on validation by epoch 20 (genuinely-learning policy beats baseline ~25%)
- val_Sharpe_raw improves by epoch 30 (not stuck at -0.48)
- HEALTH_DIAG log shows continuous adaptation (no constant values)
- Plasticity injection triggers <= 3 times in 50 epochs (rare automatic recovery)
- Contrarian override never triggers (severe failsafe should not be needed)

## Out of Scope

- Tuning the WEIGHTS of the 6 health components (assumed reasonable defaults, can tune later)
- Replacing the entire dueling architecture (advantage centering stays)
- Implementing curriculum learning beyond N4 (counterfactual curriculum)
- Multi-instrument training extensions
