spec: Introspective State Vector (ISV) — model self-awareness module
Replaces hardcoded Q-drift penalty with learnable self-awareness: 8 ISV signals (q_drift, grad_norm, td_error, ensemble_var, velocity, reward, atom_util, loss) computed by pure GPU kernel, fed through encoder MLP (8→16→8) for branch gating + drift-conditioned gamma + risk branch input. Includes recursive confidence head (predict own TD-error), branch confidence routing (replaces regime_branch_gate), and temporal ISV memory (K=4 rolling buffer with learned decay). 10 new weight tensors (68→78), 582 new params, ~39KB VRAM. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,711 @@
|
||||
# Introspective State Vector (ISV) — Design Spec
|
||||
|
||||
**Goal:** Replace the hardcoded Q-drift penalty with a self-awareness module that feeds training dynamics back into the model as learnable features. Instead of external penalties fighting the model, the model OBSERVES its own state (drift, uncertainty, gradient pressure, performance) and learns context-dependent responses. The model becomes meta-cognitive — it knows when it's drifting, uncertain, or degrading, and adapts its actions accordingly.
|
||||
|
||||
**Core Principle:** Penalty = blind force. Signal = learnable information. The model should understand it's drifting, not be punished for it.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
Market Features [B, 42]
|
||||
│
|
||||
┌────┴────┐
|
||||
│ Trunk │ (shared layers s1→s2, unchanged)
|
||||
└────┬────┘
|
||||
│
|
||||
h_s2 [B, SH2]
|
||||
│
|
||||
┌────┴──────────────────────────────────────────┐
|
||||
│ │
|
||||
│ ISV Signals [8 pinned scalars, GPU-readable]: │
|
||||
│ ┌─ [0] q_drift (normalized) │
|
||||
│ ├─ [1] grad_norm_ema │
|
||||
│ ├─ [2] td_error_ema │
|
||||
│ ├─ [3] ensemble_var_ema │
|
||||
│ ├─ [4] ensemble_var_delta (velocity) │
|
||||
│ ├─ [5] return_ema (performance) │
|
||||
│ ├─ [6] atom_utilization │
|
||||
│ └─ [7] loss_ema │
|
||||
│ │ │
|
||||
│ ISV Encoder MLP │
|
||||
│ 8 → 16 (ReLU) → 8 │
|
||||
│ │ │
|
||||
│ ISV Embedding [8] │
|
||||
│ │ │
|
||||
│ ┌────┴──── Temporal ISV ────┐ │
|
||||
│ │ Rolling buffer [K=4, 8] │ │
|
||||
│ │ Decay-weighted average │ │
|
||||
│ └──────────┬────────────────┘ │
|
||||
│ │ │
|
||||
└───────────────┼─────────────────────────────────┘
|
||||
│
|
||||
┌─────────────┼──────────────┬──────────┬──────────┐
|
||||
│ │ │ │ │
|
||||
Direction Magnitude Order Urgency Risk
|
||||
(ISV gate) (ISV gate) (ISV gate) (ISV gate) (h_s2 || ISV_raw)
|
||||
│ │ │ │ │
|
||||
confidence confidence confidence confidence │
|
||||
score score score score │
|
||||
│ │ │ │ │
|
||||
└──────── Branch Confidence Routing ──────────────┘
|
||||
│
|
||||
Drift-Conditioned γ
|
||||
(ISV → effective_gamma)
|
||||
│
|
||||
Final Action
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Component Catalog
|
||||
|
||||
### PEARLS (established meta-RL)
|
||||
|
||||
**P1: TD-Error as Feature**
|
||||
The temporal-difference error (predicted Q minus actual return) is the most fundamental learning signal in RL. RL² (Duan et al. 2016) showed that feeding TD-error back as an observation lets the model learn its own learning rate. High TD-error means "my predictions were wrong" — the model can learn to be cautious. We already compute TD-error in the C51 loss kernel. Route its EMA to ISV signal slot [2].
|
||||
|
||||
**P2: Action Entropy as Feature**
|
||||
Policy entropy measures decisiveness. Low entropy = confident (one action dominates). High entropy = uncertain (uniform distribution). Computed from C51 atom probabilities: `H = -sum(p * log(p))` over 51 atoms. We track this as part of the entropy regularization already. Route the batch-mean entropy to ISV via `atom_utilization` (slot [6]) which is a proxy — high utilization correlates with low entropy (atoms are well-spread, distribution is peaked).
|
||||
|
||||
**P3: EMA of Own Returns**
|
||||
Running average of the model's realized training returns. When return_ema is falling, the model sees "my strategy is degrading." This is what a human trader does — checks the equity curve. Route training Sharpe EMA (already computed in training_loop.rs) to ISV slot [5] via pinned memory.
|
||||
|
||||
### GEMS (less known, solid foundation)
|
||||
|
||||
**G1: Ensemble Disagreement Velocity**
|
||||
Not just current ensemble variance, but its rate of change. Growing disagreement = entering unknown territory (distribution shift). Shrinking = converging on a learned pattern. The derivative of uncertainty is more informative than the level. Computed as `delta = (ensemble_var - ensemble_var_prev) / max(ensemble_var_prev, 1e-6)`. Positive delta = "I'm becoming more confused." ISV slot [4].
|
||||
|
||||
**G2: Gradient Norm as Feature**
|
||||
The magnitude of gradients tells the model how aggressively the optimizer is correcting it. High gradient norm = "the optimizer thinks I'm very wrong." The model can learn: when gradient pressure is high, trust current weights less, be more conservative. We already compute gradient norms for clipping (`grad_norm_pinned`). EMA it and route to ISV slot [1].
|
||||
|
||||
**G3: Per-Branch Calibration (via Recursive Confidence — see N4)**
|
||||
Track `predicted_Q - actual_return` per branch over a rolling window. A branch that's confidently wrong gets a negative calibration score. Folded into the Recursive Confidence novel (N4) below.
|
||||
|
||||
### NOVELS (genuinely new)
|
||||
|
||||
**N1: Introspective State Vector (ISV) — Unified Self-Awareness Module**
|
||||
Instead of ad-hoc per-signal routing, ALL self-monitoring signals are packed into a single 8-dimensional vector that flows through a dedicated encoder MLP. The model develops a unified "how am I doing?" representation. No existing system provides comprehensive introspection — most meta-RL feeds back 1-2 signals, not a full state vector. The ISV is global (same for all samples in batch), broadcast via pinned device-mapped memory (zero memcpy).
|
||||
|
||||
**N2: Drift-Conditioned Discount (γ as Learned Function)**
|
||||
Instead of fixed or externally-adapted gamma, the model's OWN drift assessment modulates the discount factor:
|
||||
|
||||
```
|
||||
effective_gamma = base_gamma * sigmoid(w_gamma @ ISV_embedding + b_gamma)
|
||||
```
|
||||
|
||||
High drift → model shortens its horizon ("my far-future Q-values are unreliable, focus near-term"). Low drift → model extends horizon ("Q-values are stable, I can trust longer predictions"). Gamma becomes endogenous — the model controls its own planning horizon based on self-assessed reliability. The sigmoid output ranges [0.5, 1.0] (clamped), so effective_gamma stays in [base_gamma*0.5, base_gamma].
|
||||
|
||||
**N3: Temporal ISV — Pattern Memory for Training Dynamics**
|
||||
A rolling buffer of K=4 recent ISV snapshots with learned decay weights. The model sees not just "I'm drifting now" but "I've been drifting for 4 steps" or "drift just spiked after being stable." Computed as weighted average: `ISV_temporal[k] = sum(decay^t * ISV_history[t])` for each feature k. The decay weight is a single learned scalar (per ISV feature = 8 decay weights). Lightweight: 4×8=32 float buffer + 8 learned weights. No Mamba2 needed for 8 scalars.
|
||||
|
||||
**N4: Recursive Confidence — the Model Predicts Its Own Error**
|
||||
An auxiliary head that predicts `|Q_predicted - Q_actual|` — the model's own future prediction error:
|
||||
|
||||
```
|
||||
predicted_error = sigmoid(w_conf @ h_s2 + b_conf) → [B, 1]
|
||||
```
|
||||
|
||||
Supervised by actual absolute TD-error from the PREVIOUS step (lagged target, stored in pinned memory). The model learns to predict WHEN it will be wrong. When predicted_error is high, the risk branch automatically reduces exposure (predicted_error feeds into risk_budget_forward as an extra input). This creates a recursive self-improvement loop — the model's error prediction gets better each epoch, making it increasingly self-aware of its own limitations.
|
||||
|
||||
Loss: `L_conf = MSE(predicted_error, lagged_td_error_abs)`, weighted at 0.01× main loss (auxiliary, never dominates). Gradient flows through the shared trunk, teaching the trunk to encode error-predictive features.
|
||||
|
||||
**N5: Branch Confidence Routing — Selective Self-Doubt**
|
||||
Each branch computes a confidence score from its Q-distribution:
|
||||
|
||||
```
|
||||
confidence_d = max(softmax(Q_d)) / (1/A_d) → ratio to uniform
|
||||
```
|
||||
|
||||
When confidence is close to 1.0 (near uniform = no preference), the branch is uncertain. When the ISV shows instability (high drift, high grad_norm), uncertain branches get dampened:
|
||||
|
||||
```
|
||||
effective_weight_d = confidence_d * sigmoid(ISV_gate_d)
|
||||
Q_gated[b, d, a] = effective_weight_d * Q_advantage[b, d, a]
|
||||
```
|
||||
|
||||
The model develops selective self-doubt — it can be confident about direction but uncertain about magnitude, and act accordingly. This prevents one uncertain branch from poisoning the action selection. Replaces the current `regime_branch_gate` with a strictly more powerful mechanism that incorporates both market regime AND model confidence.
|
||||
|
||||
---
|
||||
|
||||
## ISV Signal Computation (Pure GPU — Zero CPU Involvement)
|
||||
|
||||
All 8 ISV signals are stored in a **pinned device-mapped buffer** `isv_signals [8]`. A single-thread GPU kernel `isv_signal_update` reads from existing GPU-accessible pointers, computes all EMA updates, and writes the ISV vector in-place. **No CPU in the hot path.** The kernel runs after `graph_adam` replay, before the next `graph_forward`.
|
||||
|
||||
| Slot | Signal | GPU Source Pointer | Formula (computed in kernel) |
|
||||
|------|--------|-------------------|------------------------------|
|
||||
| 0 | `q_drift` | `q_mean_scratch_dev_ptr`, `q_mean_ema_dev_ptr` | `(q_mean - ema) / max(\|ema\|, 0.1)` |
|
||||
| 1 | `grad_norm_ema` | `grad_norm_dev_ptr` | `0.95 * old + 0.05 * sqrt(grad_norm_sq)` |
|
||||
| 2 | `td_error_ema` | `td_error_scratch_dev_ptr` (new pinned scalar) | `0.95 * old + 0.05 * mean_abs_td` |
|
||||
| 3 | `ensemble_var_ema` | `ensemble_var_scratch_dev_ptr` (new pinned scalar) | `0.95 * old + 0.05 * mean_var` |
|
||||
| 4 | `ensemble_var_delta` | Slot [3] internal | `(new_ema - old_ema) / max(old_ema, 1e-6)` |
|
||||
| 5 | `reward_ema` | `reward_scratch_dev_ptr` (new pinned scalar) | `0.95 * old + 0.05 * batch_mean_reward` |
|
||||
| 6 | `atom_util` | `atom_util_scratch_dev_ptr` (new pinned scalar) | EMA of atom utilization |
|
||||
| 7 | `loss_ema` | `total_loss_dev_ptr` | `0.95 * old + 0.05 * total_loss` |
|
||||
|
||||
**New pinned scalars needed:** `td_error_scratch`, `ensemble_var_scratch`, `reward_scratch`, `atom_util_scratch` — each a single f32, pinned device-mapped. Written by existing GPU kernels (C51 loss writes mean |TD|, ensemble aggregate writes mean variance, etc.) via `atomicExch` or simple write at the end of those kernels. The `isv_signal_update` kernel then reads all of them.
|
||||
|
||||
### CUDA Kernel: `isv_signal_update`
|
||||
|
||||
Single-thread kernel that runs after each training step. Reads GPU-accessible scalars, computes EMA updates, writes ISV vector. Also shifts the temporal history buffer.
|
||||
|
||||
```cuda
|
||||
extern "C" __global__ void isv_signal_update(
|
||||
float* __restrict__ isv_signals, /* [8] pinned device-mapped — ISV vector */
|
||||
float* __restrict__ isv_history, /* [K*8] pinned device-mapped — temporal buffer */
|
||||
const float* __restrict__ q_mean_ptr, /* [1] pinned — current batch Q-mean */
|
||||
const float* __restrict__ q_mean_ema_ptr, /* [1] pinned — Q-mean EMA */
|
||||
const float* __restrict__ grad_norm_ptr, /* [1] pinned — gradient norm² from Adam */
|
||||
const float* __restrict__ td_error_ptr, /* [1] pinned — batch mean |TD-error| */
|
||||
const float* __restrict__ ens_var_ptr, /* [1] pinned — batch mean ensemble var */
|
||||
const float* __restrict__ reward_ptr, /* [1] pinned — batch mean reward */
|
||||
const float* __restrict__ atom_util_ptr, /* [1] pinned — atom utilization */
|
||||
const float* __restrict__ loss_ptr, /* [1] pinned — total loss */
|
||||
int K /* temporal history length (4) */
|
||||
) {
|
||||
if (threadIdx.x != 0) return;
|
||||
|
||||
float alpha = 0.05f; /* EMA smoothing */
|
||||
|
||||
/* ── Shift temporal history: prepend current ISV, drop oldest ── */
|
||||
for (int t = K - 1; t > 0; t--)
|
||||
for (int k = 0; k < 8; k++)
|
||||
isv_history[t * 8 + k] = isv_history[(t - 1) * 8 + k];
|
||||
for (int k = 0; k < 8; k++)
|
||||
isv_history[k] = isv_signals[k]; /* save current before overwrite */
|
||||
|
||||
/* ── Compute new ISV signals ── */
|
||||
|
||||
/* [0] Q-drift: normalized deviation from EMA */
|
||||
float q_mean = q_mean_ptr[0];
|
||||
float q_ema = q_mean_ema_ptr[0];
|
||||
isv_signals[0] = (q_mean - q_ema) / fmaxf(fabsf(q_ema), 0.1f);
|
||||
|
||||
/* [1] Gradient norm EMA (grad_norm_ptr stores norm², take sqrt) */
|
||||
float gn = sqrtf(fmaxf(grad_norm_ptr[0], 0.0f));
|
||||
isv_signals[1] = (1.0f - alpha) * isv_signals[1] + alpha * gn;
|
||||
|
||||
/* [2] TD-error EMA */
|
||||
isv_signals[2] = (1.0f - alpha) * isv_signals[2] + alpha * td_error_ptr[0];
|
||||
|
||||
/* [3] Ensemble variance EMA (save old for delta) */
|
||||
float old_ens_ema = isv_signals[3];
|
||||
isv_signals[3] = (1.0f - alpha) * old_ens_ema + alpha * ens_var_ptr[0];
|
||||
|
||||
/* [4] Ensemble variance velocity */
|
||||
isv_signals[4] = (isv_signals[3] - old_ens_ema) / fmaxf(old_ens_ema, 1e-6f);
|
||||
|
||||
/* [5] Reward EMA (batch-level, not epoch-level) */
|
||||
isv_signals[5] = (1.0f - alpha) * isv_signals[5] + alpha * reward_ptr[0];
|
||||
|
||||
/* [6] Atom utilization */
|
||||
isv_signals[6] = (1.0f - alpha) * isv_signals[6] + alpha * atom_util_ptr[0];
|
||||
|
||||
/* [7] Loss EMA */
|
||||
isv_signals[7] = (1.0f - alpha) * isv_signals[7] + alpha * loss_ptr[0];
|
||||
}
|
||||
```
|
||||
|
||||
**Launch:** grid=(1,1,1), block=(1,1,1). Single thread, ~40 FLOPs. Negligible.
|
||||
|
||||
**Integration:** The existing kernels need small additions to write their batch-mean scalars to pinned scratch pointers:
|
||||
- C51 loss kernel: at the end, one thread writes `mean(|td_errors|)` to `td_error_scratch_dev_ptr`
|
||||
- Ensemble aggregate kernel: at the end, one thread writes `mean(ensemble_var)` to `ensemble_var_scratch_dev_ptr`
|
||||
- Experience collector reward path: write batch mean reward to `reward_scratch_dev_ptr`
|
||||
- Atom utilization kernel: already produces a scalar — route to `atom_util_scratch_dev_ptr`
|
||||
|
||||
These are all single-write additions to existing kernels, not new kernel launches.
|
||||
|
||||
---
|
||||
|
||||
## New Weight Tensors
|
||||
|
||||
NUM_WEIGHT_TENSORS: 68 → 78 (10 new tensors).
|
||||
|
||||
| Index | Name | Shape | Purpose |
|
||||
|-------|------|-------|---------|
|
||||
| 68 | `w_isv_fc1` | [16, 8] | ISV encoder layer 1 weights |
|
||||
| 69 | `b_isv_fc1` | [16] | ISV encoder layer 1 bias |
|
||||
| 70 | `w_isv_fc2` | [8, 16] | ISV encoder layer 2 weights |
|
||||
| 71 | `b_isv_fc2` | [8] | ISV encoder layer 2 bias |
|
||||
| 72 | `w_isv_gate` | [4, 8] | Branch confidence gating weights |
|
||||
| 73 | `b_isv_gate` | [4] | Branch confidence gating bias |
|
||||
| 74 | `w_isv_gamma` | [1, 8] | Drift-conditioned gamma weights |
|
||||
| 75 | `b_isv_gamma` | [1] | Drift-conditioned gamma bias |
|
||||
| 76 | `w_conf_fc` | [1, SH2] | Recursive confidence head weights |
|
||||
| 77 | `b_conf_fc` | [1] | Recursive confidence head bias |
|
||||
|
||||
Plus 8 learned decay weights for temporal ISV (stored in `isv_decay_pinned [8]`, not in param buffer — updated by separate SGD like risk branch).
|
||||
|
||||
**Total new parameters in param buffer (indices 68-77):** 128 + 16 + 128 + 8 + 32 + 4 + 8 + 1 + SH2 + 1 = 326 + SH2. With SH2=256 (production default): **582 parameters**. Plus 8 decay weights in pinned memory (separate SGD). Negligible vs ~580K existing.
|
||||
|
||||
**Existing tensor size change:** `w_risk_fc` at index [64] grows from `AH * SH2 = 128 * 256 = 32768` to `AH * (SH2 + 9) = 128 * 265 = 33920` (+1152 elements). This is the ONLY existing tensor that changes — but it has blast radius: `risk_aligned_count`, `padded_byte_offset(param_sizes, 64)`, `risk_grad_buf`, and all risk SGD pointer arithmetic must be updated atomically with `compute_param_sizes`. The `risk_budget_backward` kernel stride changes from `j * SH2` to `j * (SH2 + 9)`.
|
||||
|
||||
---
|
||||
|
||||
## CUDA Kernel: `isv_forward`
|
||||
|
||||
Single kernel computes ISV encoder + branch gating + drift-conditioned gamma. One thread per sample.
|
||||
|
||||
```cuda
|
||||
extern "C" __global__ void isv_forward(
|
||||
const float* __restrict__ isv_signals_dev_ptr, /* [8] pinned device-mapped */
|
||||
const float* __restrict__ isv_history, /* [K=4, 8] temporal buffer */
|
||||
const float* __restrict__ w_fc1, /* [16, 8] */
|
||||
const float* __restrict__ b_fc1, /* [16] */
|
||||
const float* __restrict__ w_fc2, /* [8, 16] */
|
||||
const float* __restrict__ b_fc2, /* [8] */
|
||||
const float* __restrict__ isv_decay, /* [8] learned decay weights */
|
||||
const float* __restrict__ w_gate, /* [4, 8] */
|
||||
const float* __restrict__ b_gate, /* [4] */
|
||||
const float* __restrict__ w_gamma, /* [1, 8] */
|
||||
const float* __restrict__ b_gamma, /* [1] */
|
||||
float* __restrict__ isv_embedding_out, /* [8] shared (single-instance) */
|
||||
float* __restrict__ branch_gate_out, /* [4] shared (single-instance) */
|
||||
float* __restrict__ gamma_mod_out, /* [1] shared (single-instance) */
|
||||
int K
|
||||
) {
|
||||
/* Single thread — ISV is global, not per-sample */
|
||||
if (threadIdx.x != 0) return;
|
||||
|
||||
/* Step 1: Temporal ISV — decay-weighted average of history + current */
|
||||
float isv_temporal[8];
|
||||
for (int k = 0; k < 8; k++) {
|
||||
float decay = 1.0f / (1.0f + expf(-isv_decay[k])); /* sigmoid → (0,1) */
|
||||
float weighted_sum = isv_signals_dev_ptr[k]; /* current (weight=1) */
|
||||
float weight_sum = 1.0f;
|
||||
for (int t = 0; t < K; t++) {
|
||||
float w = powf(decay, (float)(t + 1));
|
||||
weighted_sum += w * isv_history[t * 8 + k];
|
||||
weight_sum += w;
|
||||
}
|
||||
isv_temporal[k] = weighted_sum / weight_sum;
|
||||
}
|
||||
|
||||
/* Step 2: ISV encoder — FC1 (8→16, ReLU) → FC2 (16→8) */
|
||||
float hidden[16];
|
||||
for (int j = 0; j < 16; j++) {
|
||||
float val = b_fc1[j];
|
||||
for (int k = 0; k < 8; k++)
|
||||
val += w_fc1[j * 8 + k] * isv_temporal[k];
|
||||
hidden[j] = fmaxf(val, 0.0f); /* ReLU */
|
||||
}
|
||||
|
||||
float embedding[8];
|
||||
for (int k = 0; k < 8; k++) {
|
||||
float val = b_fc2[k];
|
||||
for (int j = 0; j < 16; j++)
|
||||
val += w_fc2[k * 16 + j] * hidden[j];
|
||||
embedding[k] = val;
|
||||
}
|
||||
|
||||
/* Write embedding (single instance — downstream kernels read via pointer) */
|
||||
for (int k = 0; k < 8; k++)
|
||||
isv_embedding_out[k] = embedding[k];
|
||||
|
||||
/* Step 3: Branch gating — softmax(W_gate @ embedding + b_gate) */
|
||||
float logits[4];
|
||||
float max_logit = -1e9f;
|
||||
for (int d = 0; d < 4; d++) {
|
||||
float val = b_gate[d];
|
||||
for (int k = 0; k < 8; k++)
|
||||
val += w_gate[d * 8 + k] * embedding[k];
|
||||
logits[d] = val;
|
||||
max_logit = fmaxf(max_logit, logits[d]);
|
||||
}
|
||||
float sum_exp = 0.0f;
|
||||
for (int d = 0; d < 4; d++) {
|
||||
logits[d] = expf(logits[d] - max_logit);
|
||||
sum_exp += logits[d];
|
||||
}
|
||||
for (int d = 0; d < 4; d++)
|
||||
branch_gate_out[d] = logits[d] / sum_exp;
|
||||
|
||||
/* Step 4: Drift-conditioned gamma — sigmoid(W_gamma @ embedding + b_gamma) */
|
||||
float gamma_raw = b_gamma[0];
|
||||
for (int k = 0; k < 8; k++)
|
||||
gamma_raw += w_gamma[k] * embedding[k];
|
||||
/* Map to [0.5, 1.0] — 0.5 + 0.5*sigmoid(x). No clamp needed — formula
|
||||
* naturally produces this range. x→-∞: 0.5, x→+∞: 1.0. */
|
||||
gamma_mod_out[0] = 0.5f + 0.5f / (1.0f + expf(-gamma_raw));
|
||||
}
|
||||
```
|
||||
|
||||
**Launch:** grid=(1, 1, 1), block=(1, 1, 1). Single thread — ISV signals are global (same for all samples), so computing once and broadcasting is correct. The output buffers (`isv_embedding_out [8]`, `branch_gate_out [4]`, `gamma_mod_out [1]`) are single-instance, not per-sample. Downstream kernels read these shared buffers via pointer.
|
||||
|
||||
**Temporal ISV ring buffer convention:** `isv_history[0..8]` = most recent snapshot (1 step ago), `isv_history[8..16]` = 2 steps ago, etc. CPU prepends new snapshots by shifting existing entries down: `memmove(&history[8], &history[0], (K-1)*8*sizeof(float))`, then writes current ISV to `history[0..8]`.
|
||||
|
||||
---
|
||||
|
||||
## CUDA Kernel: `recursive_confidence_forward`
|
||||
|
||||
Auxiliary head that predicts the model's own TD-error.
|
||||
|
||||
```cuda
|
||||
extern "C" __global__ void recursive_confidence_forward(
|
||||
const float* __restrict__ h_s2, /* [B, SH2] */
|
||||
const float* __restrict__ w_conf, /* [1, SH2] */
|
||||
const float* __restrict__ b_conf, /* [1] */
|
||||
float* __restrict__ predicted_error, /* [B] */
|
||||
int B, int SH2
|
||||
) {
|
||||
int i = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
if (i >= B) return;
|
||||
|
||||
const float* h = h_s2 + (long long)i * SH2;
|
||||
float val = b_conf[0];
|
||||
for (int k = 0; k < SH2; k++)
|
||||
val += w_conf[k] * h[k];
|
||||
/* Sigmoid → [0, 1] range for error magnitude */
|
||||
predicted_error[i] = 1.0f / (1.0f + expf(-val));
|
||||
}
|
||||
```
|
||||
|
||||
**Supervision:** `L_conf = mean((predicted_error - lagged_td_error_abs)^2)` at 0.01× weight. `lagged_td_error_abs` is the batch-mean absolute TD-error from the PREVIOUS training step, stored in a pinned scalar.
|
||||
|
||||
---
|
||||
|
||||
## CUDA Kernel: `branch_confidence_routing`
|
||||
|
||||
Replaces `regime_branch_gate`. Combines ISV gating with per-branch confidence.
|
||||
|
||||
**Note on market regime:** The current `regime_branch_gate` uses ADX (state[40]) and CUSUM (state[41]) as market regime features. These are NOT lost — the trunk already processes them as input features, so branches see market regime through h_s2. ISV adds a NEW dimension: training dynamics awareness. `branch_confidence_routing` is strictly more powerful because it combines ISV (how the model is doing) with per-branch confidence (how certain each branch is).
|
||||
|
||||
```cuda
|
||||
extern "C" __global__ void branch_confidence_routing(
|
||||
float* __restrict__ q_values, /* [B, total_actions] in-place */
|
||||
const float* __restrict__ branch_gate, /* [4] shared from isv_forward */
|
||||
int B,
|
||||
int b0_size, int b1_size, int b2_size, int b3_size
|
||||
) {
|
||||
int i = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
if (i >= B) return;
|
||||
|
||||
int branch_sizes[4] = { b0_size, b1_size, b2_size, b3_size };
|
||||
int total_actions = b0_size + b1_size + b2_size + b3_size;
|
||||
int branch_offset = 0;
|
||||
|
||||
for (int d = 0; d < 4; d++) {
|
||||
int A_d = branch_sizes[d];
|
||||
|
||||
/* Branch confidence from Q-value separation:
|
||||
* Q_max - Q_mean within the branch, normalized by Q_range.
|
||||
* High separation = confident (one action clearly preferred).
|
||||
* Low separation = uncertain (all actions similar).
|
||||
* Uses q_values directly (expected Q per action, already computed). */
|
||||
float q_max = -1e9f;
|
||||
float q_sum = 0.0f;
|
||||
for (int a = 0; a < A_d; a++) {
|
||||
float q = q_values[(long long)i * total_actions + branch_offset + a];
|
||||
q_max = fmaxf(q_max, q);
|
||||
q_sum += q;
|
||||
}
|
||||
float q_mean = q_sum / (float)A_d;
|
||||
float separation = q_max - q_mean;
|
||||
/* Normalize: sigmoid maps separation to (0, 1). Scale=5 gives
|
||||
* ~0.5 at separation=0 (uniform), ~0.99 at separation=1. */
|
||||
float confidence = 1.0f / (1.0f + expf(-5.0f * separation));
|
||||
|
||||
/* Combined gate: ISV importance × branch confidence.
|
||||
* ISV gate from isv_forward (how training is going).
|
||||
* Confidence from Q-value separation (how decisive this branch is).
|
||||
* Floor at 0.3 — never fully mute a branch. */
|
||||
float isv_gate = branch_gate[d]; /* shared — same for all samples */
|
||||
float effective_weight = isv_gate * fmaxf(confidence, 0.3f);
|
||||
|
||||
/* Apply to Q-values for this branch (in-place) */
|
||||
for (int a = 0; a < A_d; a++) {
|
||||
q_values[(long long)i * total_actions + branch_offset + a] *= effective_weight;
|
||||
}
|
||||
branch_offset += A_d;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Key change from regime_branch_gate:** No `q_distributions` parameter needed — confidence is computed directly from the Q-values that are already in the buffer. This avoids the incorrect indexing of packed advantage logits and is computationally cheaper (A_d additions per branch instead of A_d×NA exponentials).
|
||||
|
||||
---
|
||||
|
||||
## Risk Branch Integration
|
||||
|
||||
The risk branch currently takes `h_s2 [B, SH2]` as input. With ISV, it receives `h_s2 || ISV_raw || predicted_error`:
|
||||
|
||||
```
|
||||
risk_input [B, SH2 + 8 + 1] = concat(h_s2, isv_signals, predicted_error)
|
||||
```
|
||||
|
||||
This gives the risk branch the most detailed self-awareness. It sees:
|
||||
- **h_s2**: market state representation
|
||||
- **ISV raw signals**: 8 training dynamics indicators
|
||||
- **predicted_error**: the model's own error prediction
|
||||
|
||||
The risk branch learns: "when I see high drift AND high predicted error AND high ensemble disagreement, reduce risk_budget → smaller positions, tighter CVaR."
|
||||
|
||||
**Change to `risk_budget_forward`:**
|
||||
|
||||
```cuda
|
||||
extern "C" __global__ void risk_budget_forward(
|
||||
const float* __restrict__ h_s2,
|
||||
const float* __restrict__ isv_signals_dev_ptr, /* [8] NEW: raw ISV */
|
||||
const float* __restrict__ predicted_error, /* [B] NEW: recursive confidence */
|
||||
const float* __restrict__ w_risk_fc, /* [AH, SH2+9] NEW: wider input */
|
||||
const float* __restrict__ b_risk_fc,
|
||||
const float* __restrict__ w_risk_out,
|
||||
const float* __restrict__ b_risk_out,
|
||||
float* __restrict__ risk_hidden,
|
||||
float* __restrict__ risk_budget_out,
|
||||
int B, int SH2, int AH
|
||||
) {
|
||||
int i = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
if (i >= B) return;
|
||||
|
||||
int input_dim = SH2 + 9; /* SH2 + 8 ISV + 1 predicted_error */
|
||||
const float* h = h_s2 + (long long)i * SH2;
|
||||
float* h_risk = risk_hidden + (long long)i * AH;
|
||||
|
||||
for (int j = 0; j < AH; j++) {
|
||||
float val = b_risk_fc[j];
|
||||
/* h_s2 portion */
|
||||
for (int k = 0; k < SH2; k++)
|
||||
val += w_risk_fc[(long long)j * input_dim + k] * h[k];
|
||||
/* ISV signals (broadcast — same for all samples) */
|
||||
for (int k = 0; k < 8; k++)
|
||||
val += w_risk_fc[(long long)j * input_dim + SH2 + k] * isv_signals_dev_ptr[k];
|
||||
/* Predicted error (per-sample) */
|
||||
val += w_risk_fc[(long long)j * input_dim + SH2 + 8] * predicted_error[i];
|
||||
h_risk[j] = fmaxf(val, 0.0f);
|
||||
}
|
||||
|
||||
float raw = b_risk_out[0];
|
||||
for (int j = 0; j < AH; j++)
|
||||
raw += w_risk_out[j] * h_risk[j];
|
||||
risk_budget_out[i] = 1.0f / (1.0f + expf(-raw));
|
||||
}
|
||||
```
|
||||
|
||||
**Weight tensor change:** `w_risk_fc` at index [64] grows from `AH * SH2` to `AH * (SH2 + 9)`. This is the ONLY existing tensor that changes size.
|
||||
|
||||
---
|
||||
|
||||
## Drift-Conditioned Gamma — Integration
|
||||
|
||||
The `gamma_mod_out [B]` from `isv_forward` modulates the Bellman target discount:
|
||||
|
||||
```
|
||||
effective_gamma[b] = base_gamma * gamma_mod[b]
|
||||
```
|
||||
|
||||
Where `gamma_mod` ∈ [0.5, 1.0], so effective_gamma ∈ [base_gamma*0.5, base_gamma].
|
||||
|
||||
**Integration point:** In `c51_loss_kernel.cu`, the `gamma` parameter becomes `gamma_buf [B]` — a per-sample buffer filled by `isv_forward`. The kernel reads `gamma_buf[b]` instead of a scalar `gamma`.
|
||||
|
||||
Currently `c51_loss_kernel` takes `float gamma` as a scalar. Change to:
|
||||
|
||||
```cuda
|
||||
const float* __restrict__ gamma_buf /* [B] per-sample effective gamma */
|
||||
```
|
||||
|
||||
And in the Bellman projection:
|
||||
```cuda
|
||||
float g = gamma_buf[b]; /* was: float g = gamma; */
|
||||
```
|
||||
|
||||
The per-sample gamma buffer is allocated once (`[max_batch_size]`) and filled by a `fill_gamma_buf` kernel that broadcasts `base_gamma * gamma_mod_out[0]` to all B entries. When ISV is not yet trained (early epochs), `gamma_mod_out[0]` starts at 0.75 (zero-init → sigmoid(0)=0.5 → 0.5+0.5*0.5=0.75), so gamma starts slightly conservative and adapts.
|
||||
|
||||
**CUDA Graph compatibility:** `c51_loss_kernel.cu` must be recompiled with the new signature. Set `graph_forward = None` after the kernel change to force recapture on next `train_step()`.
|
||||
|
||||
---
|
||||
|
||||
## Forward Pass Integration Order
|
||||
|
||||
```
|
||||
1. trunk forward (cuBLAS) → h_s2 [B, SH2]
|
||||
2. mamba2_step() → h_enriched [B, SH2]
|
||||
3. isv_forward() → isv_embedding [B, 8], branch_gate [B, 4], gamma_mod [B]
|
||||
4. recursive_confidence_forward() → predicted_error [B]
|
||||
5. fill gamma_buf = base_gamma * gamma_mod
|
||||
6. risk_budget_forward(h_s2, isv_signals, predicted_error) → risk_budget [B]
|
||||
7. branch heads forward (cuBLAS) → Q values [B, total_actions]
|
||||
8. compute_expected_q() → expected Q per action
|
||||
9. branch_confidence_routing(Q, branch_gate) → gated Q (uses Q-value separation for confidence)
|
||||
10. apply_risk_budget() → scaled Q
|
||||
11. action selection
|
||||
```
|
||||
|
||||
Steps 3-5 are new. Step 6 changes (wider input). Step 9 replaces `regime_branch_gate` — needs Q-values already computed (step 8) to measure per-branch confidence from Q-value separation.
|
||||
|
||||
---
|
||||
|
||||
## Backward Pass
|
||||
|
||||
**ISV encoder backward:** The ISV encoder weights receive gradients from two paths:
|
||||
1. Branch gating loss (through branch_confidence_routing)
|
||||
2. Gamma modulation loss (through C51 Bellman target)
|
||||
|
||||
Both are small MLP backward passes (8→16→8). Computed analytically in a `isv_backward` kernel, gradients accumulated into the main `grad_buf` at offsets 68-75.
|
||||
|
||||
**Recursive confidence backward:** MSE loss between predicted_error and lagged_td_error_abs. Gradient flows through w_conf_fc into the trunk gradient (accumulated into existing `bw_d_h_s2`).
|
||||
|
||||
**Temporal ISV decay backward:** The 8 decay weights are updated by separate SGD (like risk branch), not part of main Adam. Learning rate: 0.001. Gradient: chain rule through the decay-weighted average formula.
|
||||
|
||||
**Risk branch backward:** Already exists but must be updated for wider input. The existing kernel at `experience_kernels.cu:4750` uses stride `j * SH2` for `d_w_risk_fc`. Must change to `j * (SH2 + 9)` AND add gradient accumulation for the 9 new ISV+error columns:
|
||||
|
||||
```cuda
|
||||
/* Existing: gradient for h_s2 portion (columns 0..SH2) */
|
||||
for (int k = 0; k < SH2; k++)
|
||||
atomicAdd(&d_w_risk_fc[(long long)j * input_dim + k], d_hj * h[k]);
|
||||
/* NEW: gradient for ISV portion (columns SH2..SH2+8) */
|
||||
for (int k = 0; k < 8; k++)
|
||||
atomicAdd(&d_w_risk_fc[(long long)j * input_dim + SH2 + k], d_hj * isv_signals_dev_ptr[k]);
|
||||
/* NEW: gradient for predicted_error (column SH2+8) */
|
||||
atomicAdd(&d_w_risk_fc[(long long)j * input_dim + SH2 + 8], d_hj * predicted_error[i]);
|
||||
```
|
||||
|
||||
Where `input_dim = SH2 + 9`. The `risk_grad_buf` must be reallocated to match the new `risk_aligned_count`.
|
||||
|
||||
---
|
||||
|
||||
## Pinned Memory Layout
|
||||
|
||||
New pinned allocations (all `cuMemAllocHost` + `cuMemHostGetDevicePointer`):
|
||||
|
||||
| Name | Size | Purpose | Written by |
|
||||
|------|------|---------|-----------|
|
||||
| `isv_signals_pinned` | 8 × f32 = 32 B | ISV signal vector | `isv_signal_update` GPU kernel |
|
||||
| `isv_history_pinned` | 4 × 8 × f32 = 128 B | Temporal ISV rolling buffer | `isv_signal_update` GPU kernel |
|
||||
| `isv_decay_pinned` | 8 × f32 = 32 B | Temporal decay weights (learned) | ISV decay SGD GPU kernel |
|
||||
| `td_error_scratch_pinned` | 1 × f32 = 4 B | Batch mean \|TD-error\| | C51 loss kernel (1 atomicExch) |
|
||||
| `ensemble_var_scratch_pinned` | 1 × f32 = 4 B | Batch mean ensemble variance | Ensemble aggregate kernel |
|
||||
| `reward_scratch_pinned` | 1 × f32 = 4 B | Batch mean reward | Reward aggregation |
|
||||
| `atom_util_scratch_pinned` | 1 × f32 = 4 B | Atom utilization | Atom util kernel |
|
||||
| `lagged_td_error_pinned` | 1 × f32 = 4 B | Previous step's mean \|TD-error\| (recursive confidence target) | `isv_signal_update` copies from slot [2] before overwrite |
|
||||
|
||||
Total: 212 bytes. All written by GPU kernels, read by GPU kernels. **Zero CPU involvement.**
|
||||
|
||||
---
|
||||
|
||||
## What ISV Replaces
|
||||
|
||||
| Old Component | Replacement | Why |
|
||||
|---------------|-------------|-----|
|
||||
| Hardcoded Q-drift penalty (removed in TEA) | ISV slot [0] q_drift → branch gating + risk input | Model learns context-dependent drift response |
|
||||
| `regime_branch_gate` (4 inputs: ADX, CUSUM, Q_gap, atom_util) | `branch_confidence_routing` (ISV embedding + per-branch confidence) | Strictly more powerful — incorporates both market regime AND model self-awareness |
|
||||
| Fixed gamma / externally-adapted gamma | Drift-conditioned gamma | Model controls its own planning horizon |
|
||||
| External E1 Q-correction (additive bias) | ISV → risk_budget reduction when drift is high | Model learns WHEN correction is needed, not a blanket offset |
|
||||
|
||||
**E1 enrichment (Q-correction) is NOT removed** — it provides the external "ground truth" signal from validation. ISV provides the internal signal from training dynamics. They are complementary: E1 corrects from eval data (slow, epoch-level), ISV adapts from training dynamics (fast, per-step).
|
||||
|
||||
---
|
||||
|
||||
## Initialization
|
||||
|
||||
- ISV encoder weights: Xavier uniform (small MLP, standard init)
|
||||
- Branch gate weights: zero-initialized biases, small Xavier weights → initial softmax ≈ uniform (no branch preferred)
|
||||
- Gamma weights: zero-initialized → initial sigmoid = 0.5 → initial gamma_mod = 0.75 → slightly conservative (shorter horizon while ISV learns)
|
||||
- Recursive confidence weights: zero-initialized → initial predicted_error ≈ 0.5 (moderate uncertainty)
|
||||
- Decay weights: initialized to 0.0 → sigmoid(0) = 0.5 → moderate decay (geometric with ratio 0.5)
|
||||
- Risk branch wider input: existing SH2 portion from current checkpoint, new ISV+error columns zero-initialized (no disruption to existing risk behavior until ISV trains)
|
||||
|
||||
---
|
||||
|
||||
## VRAM Budget
|
||||
|
||||
| Buffer | Size (B=4096, SH2=256, AH=128) |
|
||||
|--------|------|
|
||||
| `isv_embedding_out` [8] | 32 B |
|
||||
| `branch_gate_out` [4] | 16 B |
|
||||
| `gamma_mod_out` [1] | 4 B |
|
||||
| `gamma_buf` [B] | 16 KB |
|
||||
| `predicted_error` [B] | 16 KB |
|
||||
| ISV weight tensors (582 params) | 2.3 KB |
|
||||
| `w_risk_fc` growth (+1152 elements) | 4.5 KB |
|
||||
| Pinned memory (signals + history + decay + lagged_td) | 196 B |
|
||||
| **Total** | **~39 KB** |
|
||||
|
||||
Negligible. Less than 0.0001% of H100 80GB.
|
||||
|
||||
---
|
||||
|
||||
## Target Network
|
||||
|
||||
ISV weights (indices 68-77) are **online-only** — they are NOT copied to `target_params_buf`. Rationale: ISV conditioning reflects the model's current training dynamics (drift, gradient pressure, loss trajectory). The target network should use stable behavior independent of training dynamics:
|
||||
|
||||
- **Target network forward:** Use `base_gamma` directly (no drift-conditioned gamma). Fill `gamma_buf` with `base_gamma` for target Q computation.
|
||||
- **Target network gating:** Use uniform branch gates (all 0.25) — no ISV-based branch suppression for Bellman targets.
|
||||
- **Target network risk:** `risk_budget_forward` for target uses zeros for ISV slots and 0.5 for predicted_error — neutral conditioning.
|
||||
|
||||
This means `target_params_buf` stays at its current size (indices 0-67). The ISV encoder, gamma, and gate weights only exist in `params_buf` and `grad_buf`. No EMA sync needed for ISV weights.
|
||||
|
||||
---
|
||||
|
||||
## What the Model Learns
|
||||
|
||||
With ISV, the model develops context-dependent responses to its own state:
|
||||
|
||||
| Scenario | ISV Signals | Learned Response |
|
||||
|----------|-------------|------------------|
|
||||
| Stable training, low drift | q_drift≈0, grad_norm low, loss steady | Long gamma (trust far-future Q), full risk_budget |
|
||||
| Q-drift high, trending market | q_drift>0, return_ema positive | Moderate gamma (drift is from real gains), maintain risk |
|
||||
| Q-drift high, mean-reverting | q_drift>0, return_ema falling | Short gamma (overestimation danger), reduce risk |
|
||||
| High uncertainty, exploring | ensemble_var high, ensemble_var_delta>0 | Reduce magnitude, prefer flat, low risk_budget |
|
||||
| Loss spike, gradient pressure | loss_ema spiking, grad_norm high | Conservative actions, short gamma, low confidence routing |
|
||||
| Model converging | td_error falling, loss falling | Extend gamma, increase risk_budget, trust branch confidence |
|
||||
|
||||
The key difference from hardcoded penalties: the model can learn that the SAME drift level means different things in different contexts. Q-drift=+3 during a strong trend is fine. Q-drift=+3 during consolidation is dangerous. ISV + market features let the model make this distinction.
|
||||
|
||||
---
|
||||
|
||||
## Training Step Flow (Pure GPU)
|
||||
|
||||
```
|
||||
Per training step:
|
||||
1. replay graph_forward → C51/MSE loss, gradients, td_errors
|
||||
2. inject auxiliary gradients → IQN, ensemble, CQL into grad_buf
|
||||
3. replay graph_adam → Adam update, grad_norm computed
|
||||
4. isv_signal_update() → reads all GPU scalars, updates ISV [8] + history [K,8]
|
||||
5. (next step's forward):
|
||||
5a. isv_forward() → encoder MLP, branch gate, gamma_mod (reads ISV from step 4)
|
||||
5b. recursive_confidence_fwd() → predicted_error [B]
|
||||
5c. fill gamma_buf → base_gamma * gamma_mod[0], broadcast to [B]
|
||||
5d. risk_budget_forward() → reads h_s2 + ISV signals + predicted_error
|
||||
5e. branch heads (cuBLAS) → Q values
|
||||
5f. branch_confidence_routing()→ gated Q
|
||||
5g. apply_risk_budget() → scaled Q
|
||||
5h. action selection
|
||||
```
|
||||
|
||||
Step 4 (`isv_signal_update`) is a standalone kernel launch — NOT captured in graph_forward or graph_adam. It runs between graph replays on the training stream. Steps 5a-5c are new kernel launches inside graph_forward capture.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **Pinned scratch scalars** — allocate td_error_scratch, ensemble_var_scratch, reward_scratch, atom_util_scratch (4 pinned scalars). Add single-write lines to existing C51 loss, ensemble aggregate, and atom util kernels.
|
||||
2. **ISV signal buffer + history** — allocate isv_signals [8], isv_history [K=4, 8], isv_decay [8] pinned. Write `isv_signal_update` kernel. Wire after graph_adam replay.
|
||||
3. **ISV encoder kernel** (`isv_forward`) — single-thread MLP forward, branch gating output, gamma modulation output. Write kernel + Rust launch.
|
||||
4. **Temporal ISV** — decay-weighted average in isv_forward (reads isv_history + isv_decay).
|
||||
5. **Drift-conditioned gamma** — change c51_loss_kernel gamma from scalar `float` to `const float* gamma_buf [B]`. Recompile cubin. Add `fill_gamma_buf` kernel (broadcasts `base_gamma * gamma_mod[0]`). Set `graph_forward = None` for recapture.
|
||||
6. **Branch confidence routing** — write `branch_confidence_routing` kernel. Replace `regime_branch_gate` at all call sites: `apply_regime_gate()`, `reduce_current_q_stats()`, `submit_forward_ops_ddqn()`. Drop `regime_gate_kernel` field.
|
||||
7. **Risk branch wider input** — change `w_risk_fc` in `compute_param_sizes` from `AH * SH2` to `AH * (SH2 + 9)`. Update `risk_budget_forward` + `risk_budget_backward` kernels for wider stride. Reallocate `risk_grad_buf`. Update `risk_aligned_count` and all `padded_byte_offset(param_sizes, 64)` callers.
|
||||
8. **Recursive confidence head** — write `recursive_confidence_forward` kernel + MSE backward. Route `lagged_td_error_pinned` from `isv_signal_update` (copies slot [2] before overwrite).
|
||||
9. **Weight tensors** — NUM_WEIGHT_TENSORS 68→78, extend `compute_param_sizes`, `xavier_init_params_buf`, `fan_dims`. Target network: ISV weights online-only (indices 68-77 NOT in target_params_buf).
|
||||
10. **Graph recapture + smoke tests** — invalidate all graphs (`graph_forward = None`, `graph_adam = None`, `graph_forward_ddqn = None`). Run smoke test: `FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- test_generalization_components_smoke --include-ignored --nocapture`. Run compute-sanitizer: `compute-sanitizer --tool memcheck --print-limit 5 target/debug/deps/ml-* "test_generalization_components_smoke" --test-threads=1 --include-ignored`. **Target: 0 errors.** Run full test suite: `SQLX_OFFLINE=true cargo test -p ml --lib`. **Target: 899+ passed, 0 failed.**
|
||||
|
||||
---
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| ISV signals are global (same for all batch samples) — no per-sample variation | Per-sample predicted_error from recursive confidence head adds sample-level variation. Risk branch combines global ISV with per-sample h_s2. Branch confidence uses per-sample Q-value separation. |
|
||||
| ISV encoder is tiny (582 params) — may underfit | 8→16→8 is sufficient for 8 input features. If needed, widen to 8→32→16. |
|
||||
| One-step lag in ISV signals | Correct behavior — model reacts to recent dynamics, not current-batch. Same as how humans read yesterday's P&L to adjust today's risk. |
|
||||
| Drift-conditioned gamma changes Bellman target — same gamma for all samples in batch | Single gamma_mod broadcast to all B samples is appropriate: ISV is a global training-dynamics signal, not market-dependent. Per-sample gamma variation comes from the base system (external adaptive_gamma). |
|
||||
| Risk branch input change (SH2→SH2+9) breaks existing checkpoints | Zero-initialize new columns. Existing SH2 weights load normally. Risk behavior unchanged until ISV trains. |
|
||||
| `w_risk_fc` resize ripples through risk_aligned_count, risk_grad_buf, backward stride | All must be updated atomically. Implementation plan step 6 handles this as a single unit. |
|
||||
| `risk_budget_backward` stride change (SH2 → SH2+9) | Explicit kernel update with new stride and ISV+error gradient accumulation (see Backward Pass section). |
|
||||
| Branch confidence routing × ISV gate product can approach 0 for one branch | Floor at 0.3 for confidence. ISV gate softmax ensures minimum ~0.05 per branch. Product floor: 0.05 × 0.3 = 0.015 — branch never fully muted. |
|
||||
| Recursive confidence may learn to always predict 0.5 (uninformative) | MSE loss with actual TD-error provides real supervision signal. If TD-errors vary, the head WILL learn to predict them. Early epochs: TD-error is high and variable → strong learning signal. |
|
||||
| `regime_branch_gate` call sites must all be replaced | enumerate: `apply_regime_gate()`, `reduce_current_q_stats()`, `submit_forward_ops_ddqn()`. Replace with `branch_confidence_routing`. Drop `regime_gate_kernel` field. |
|
||||
| Target network uses ISV weights | ISV weights are online-only (see Target Network section). Target forward uses neutral defaults (base_gamma, uniform gates). |
|
||||
Reference in New Issue
Block a user