# Novel Generalization Techniques Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Close the in-sample/OOS generalization gap (Sharpe +0.57 IS → -1.30 OOS) by implementing 5 novel techniques that force the model to learn regime-invariant features instead of memorizing price sequences.

**Architecture:** All techniques integrate into the existing training loop without changing the core DQN architecture. Techniques 1 and 4 modify Rust-side training config per epoch. Technique 2 adds a shuffle flag to episode starts. Technique 3 computes cross-validation action similarity at epoch boundary. Technique 5 repurposes the existing curiosity forward model's prediction error as a CQL-like Q-penalty in the loss kernel.

**Tech Stack:** Rust, CUDA (cudarc), existing fused CUDA kernels, existing curiosity forward model

---

### Task 1: Domain Randomization — Per-Epoch Simulation Parameter Jitter

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs:1007-1058` (ExperienceCollectorConfig construction)
- Modify: `crates/ml/src/trainers/dqn/config.rs` (add `enable_domain_randomization` flag)
- Modify: `config/training/dqn-localdev.toml`, `config/training/dqn-production.toml`

The ExperienceCollectorConfig is constructed fresh each epoch at training_loop.rs:1007. Currently all values are deterministic. Add per-epoch randomization of spread, tx_cost, fill probabilities, and initial capital.

- [ ] **Step 1: Add config flag**

In `config.rs`, after `enable_circuit_breaker` (~line 1407), add:
```rust
pub enable_domain_randomization: bool,
```
Default in `conservative()`: `true`.

- [ ] **Step 2: Add randomization to ExperienceCollectorConfig construction**

In `training_loop.rs`, at the ExperienceCollectorConfig construction (~line 1007), wrap the fixed values with randomization when enabled:

```rust
use rand::Rng;
let mut epoch_rng = rand::rng();

let config = ExperienceCollectorConfig {
    // ... existing fields ...
    tx_cost_multiplier: if self.hyperparams.enable_domain_randomization {
        epoch_rng.random_range(0.5..2.5)
    } else {
        self.hyperparams.transaction_cost_multiplier as f32
    },
    fill_median_spread: if self.hyperparams.enable_domain_randomization {
        self.hyperparams.avg_spread as f32 * epoch_rng.random_range(0.5..3.0)
    } else {
        self.hyperparams.avg_spread as f32
    },
    fill_ioc_fill_prob: if self.hyperparams.enable_domain_randomization {
        epoch_rng.random_range(0.65..0.95)
    } else {
        self.hyperparams.fill_ioc_fill_prob as f32
    },
    fill_limit_fill_min: if self.hyperparams.enable_domain_randomization {
        epoch_rng.random_range(0.15..0.45)
    } else {
        self.hyperparams.fill_limit_fill_min as f32
    },
    fill_limit_fill_max: if self.hyperparams.enable_domain_randomization {
        epoch_rng.random_range(0.60..0.95)
    } else {
        self.hyperparams.fill_limit_fill_max as f32
    },
    spread_cost: if self.hyperparams.enable_domain_randomization {
        (self.hyperparams.tick_size * self.hyperparams.contract_multiplier
         * epoch_rng.random_range(0.3..0.8)) as f32
    } else {
        (self.hyperparams.tick_size * self.hyperparams.contract_multiplier
         * self.hyperparams.fill_spread_cost_frac) as f32
    },
    // ... rest unchanged ...
};
```

- [ ] **Step 3: Add episode start jitter**

In `training_loop.rs` at line 935-937, add ±25% stride jitter:
```rust
let mut episode_starts: Vec<i32> = (0..n_episodes)
    .map(|i| {
        let base = (i * stride).rem_euclid(usable_bars);
        if self.hyperparams.enable_domain_randomization {
            let jitter = epoch_rng.random_range(-(stride/4)..(stride/4));
            (base + jitter).rem_euclid(usable_bars)
        } else {
            base
        }
    })
    .collect();
```

- [ ] **Step 4: Add variable episode length**

Before the `timesteps` assignment at line 931, randomize:
```rust
let timesteps = if self.hyperparams.enable_domain_randomization {
    let base = self.hyperparams.gpu_timesteps_per_episode.min(1000) as i32;
    let jitter = epoch_rng.random_range(-(base/4)..(base/4));
    (base + jitter).max(50) // minimum 50 timesteps
} else {
    self.hyperparams.gpu_timesteps_per_episode.min(1000) as i32
};
```

- [ ] **Step 5: Add to TOML configs**

Add to `[advanced]` section in dqn-localdev.toml and dqn-production.toml:
```toml
enable_domain_randomization = true
```

- [ ] **Step 6: Compile + smoke test**

```bash
SQLX_OFFLINE=true cargo check -p ml
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests::training_stability::test_production_training_stability --ignored --nocapture
```

- [ ] **Step 7: Commit**

```bash
git add crates/ml/src/trainers/dqn/trainer/training_loop.rs crates/ml/src/trainers/dqn/config.rs config/training/*.toml
git commit -m "feat: domain randomization — per-epoch jitter on spread, tx_cost, fills, episode starts"
```

---

### Task 2: Periodic Shrink-and-Perturb (already exists, needs scheduling)

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs:308-347` (phase boundary S&P)
- Modify: `crates/ml/src/trainers/dqn/config.rs` (add `shrink_perturb_interval`)

Shrink-and-perturb already works (`fused.shrink_and_perturb(alpha, sigma)`) but only runs at phase boundaries. Need periodic execution every N epochs.

- [ ] **Step 1: Add config field**

In `config.rs`, add:
```rust
pub shrink_perturb_interval: usize, // 0=disabled, 20=every 20 epochs
pub shrink_perturb_alpha: f64,      // 0.85 = keep 85%, reinit 15%
pub shrink_perturb_sigma: f64,      // 0.01 = noise scale
```
Defaults in `conservative()`: `shrink_perturb_interval: 20, shrink_perturb_alpha: 0.85, shrink_perturb_sigma: 0.01`.

- [ ] **Step 2: Add periodic S&P to training loop**

In `training_loop.rs`, in the main epoch loop (after experience collection, before training steps), add:
```rust
if self.hyperparams.shrink_perturb_interval > 0
    && epoch > 0
    && epoch % self.hyperparams.shrink_perturb_interval == 0
{
    if let Some(ref mut fused) = self.fused_ctx {
        let alpha = self.hyperparams.shrink_perturb_alpha as f32;
        let sigma = self.hyperparams.shrink_perturb_sigma as f32;
        match fused.shrink_and_perturb(alpha, sigma) {
            Ok(()) => info!(epoch, alpha, sigma, "Periodic shrink-and-perturb applied"),
            Err(e) => warn!(epoch, "Shrink-and-perturb failed (non-fatal): {e}"),
        }
    }
}
```

- [ ] **Step 3: Add to TOMLs + profile**

dqn-localdev.toml and dqn-production.toml `[advanced]`:
```toml
shrink_perturb_interval = 20
shrink_perturb_alpha = 0.85
shrink_perturb_sigma = 0.01
```

Wire through `training_profile.rs` `apply_to()` if fields are added to `AdvancedSection`.

- [ ] **Step 4: Compile + test**

```bash
SQLX_OFFLINE=true cargo check -p ml
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests --ignored --nocapture
```

- [ ] **Step 5: Commit**

```bash
git commit -m "feat: periodic shrink-and-perturb every N epochs (default 20)"
```

---

### Task 3: Adversarial Regime Injection

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (epoch-level regime detection + injection)
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (ExperienceCollectorConfig fields)
- Modify: `crates/ml/src/trainers/dqn/config.rs` (add `adversarial_injection_threshold`)

When the model achieves low drawdown (<10%) for 3+ consecutive epochs, inject adversarial conditions: 3x spread, 2x tx_cost, 0.5x fill probability. This forces the model to handle worst-case scenarios.

- [ ] **Step 1: Add config + tracking**

In `config.rs`:
```rust
pub adversarial_injection_threshold: f64, // DD threshold (0.10 = 10%)
pub adversarial_injection_consecutive: usize, // consecutive good epochs before trigger (3)
```

In `trainer/mod.rs`, add tracking field:
```rust
pub(crate) consecutive_low_dd_epochs: usize,
```

- [ ] **Step 2: Implement detection + injection**

In `training_loop.rs`, after epoch metrics are computed (after the Sharpe/MaxDD/PF log), add:
```rust
// Adversarial regime injection: when model is doing too well, make it harder
let max_dd_fraction = monitor.max_drawdown_pct / 100.0;
if max_dd_fraction < self.hyperparams.adversarial_injection_threshold {
    self.consecutive_low_dd_epochs += 1;
} else {
    self.consecutive_low_dd_epochs = 0;
}

let adversarial_active = self.consecutive_low_dd_epochs
    >= self.hyperparams.adversarial_injection_consecutive;
```

Then in the ExperienceCollectorConfig construction for the NEXT epoch, check `adversarial_active`:
```rust
let adversarial_mult = if adversarial_active { 3.0_f32 } else { 1.0 };
// Apply to spread_cost, tx_cost_multiplier, etc.
spread_cost: base_spread * adversarial_mult,
tx_cost_multiplier: base_tx * (if adversarial_active { 2.0 } else { 1.0 }),
fill_ioc_fill_prob: if adversarial_active { 0.50 } else { base_fill },
```

- [ ] **Step 3: Log when adversarial mode activates**

```rust
if adversarial_active {
    info!(
        epoch, consecutive = self.consecutive_low_dd_epochs,
        "ADVERSARIAL REGIME INJECTED: 3x spread, 2x tx_cost, 0.5x fill prob"
    );
}
```

- [ ] **Step 4: Compile + test + commit**

---

### Task 4: Anti-Correlation Reward Penalty

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (epoch boundary action comparison)
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (reward_penalty_multiplier field)
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (apply penalty to reward)

At epoch end, compare the action distribution (from GPU monitoring) between training and validation. If cosine similarity > 0.8, the model is doing the same thing on both — it hasn't learned to differentiate. Penalize next epoch's rewards.

- [ ] **Step 1: Compute cosine similarity at epoch boundary**

In `training_loop.rs`, after validation loss computation, add:
```rust
// Anti-correlation: cosine similarity between train and val action distributions
let train_actions: Vec<f64> = monitor.action_counts.iter().map(|&c| c as f64).collect();
let val_actions: Vec<f64> = /* from validation pass action counts */;

let dot: f64 = train_actions.iter().zip(&val_actions).map(|(a, b)| a * b).sum();
let mag_t: f64 = train_actions.iter().map(|a| a * a).sum::<f64>().sqrt();
let mag_v: f64 = val_actions.iter().map(|a| a * a).sum::<f64>().sqrt();
let cosine_sim = if mag_t > 0.0 && mag_v > 0.0 { dot / (mag_t * mag_v) } else { 0.0 };

let anti_corr_penalty = if cosine_sim > 0.8 {
    0.8_f32 // scale rewards by 0.8 next epoch (20% penalty)
} else {
    1.0_f32
};
```

- [ ] **Step 2: Pass penalty to experience collector**

Add `reward_scale_override: f32` to `ExperienceCollectorConfig`. Default 1.0.
In the experience kernel, multiply reward by this scale factor:
```cuda
reward *= reward_scale_override;
```

- [ ] **Step 3: Compile + test + commit**

---

### Task 5: Curiosity Prediction Error as Q-Penalty (Inverse Model)

**Files:**
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs` (wire curiosity error into loss)
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (compute curiosity error per batch)
- Modify: `crates/ml/src/cuda_pipeline/mse_loss_kernel.cu` or `c51_loss_kernel.cu` (add penalty term)
- Modify: `crates/ml/src/trainers/dqn/config.rs` (add `curiosity_q_penalty_weight`)

The curiosity forward model (state+action → next_state) already exists and trains. Its prediction error measures how "novel" a state-action pair is. High prediction error = the model has never seen this situation = Q-values should be conservative (lower).

Wire the curiosity prediction error as an ADDITIVE penalty to the Q-target in the Bellman equation:
```
target_Q = reward + gamma * (1-done) * target_Q_next - curiosity_penalty * prediction_error
```

This makes the Q-function conservative on novel state-action pairs (same principle as CQL, but data-driven instead of action-space-driven).

- [ ] **Step 1: Add config field**

```rust
pub curiosity_q_penalty_weight: f64, // 0.0=disabled, 0.5=moderate, 1.0=strong
```
Default: `0.5`.

- [ ] **Step 2: Compute per-sample curiosity error in fused training step**

In `fused_training.rs`, after the forward pass and before the loss kernel, launch the curiosity forward model on the current batch to get per-sample prediction errors. Store in a `CudaSlice<f32>` of size `[batch_size]`.

The curiosity model is already initialized in the fused context (`self.curiosity_trainer`). Call:
```rust
let curiosity_errors = self.curiosity_trainer.compute_batch_errors(
    &batch_states, &batch_actions, &batch_next_states, batch_size
)?;
```

- [ ] **Step 3: Pass curiosity errors to loss kernel**

Add `const float* curiosity_errors` parameter to `mse_loss_batched` and `c51_loss_batched`.

In the Bellman target computation (MSE kernel line 334):
```cuda
float curiosity_penalty = curiosity_errors[sample_id] * curiosity_q_penalty_weight;
float target_q = reward + gamma * (1.0f - done) * target_eq - curiosity_penalty;
```

The penalty subtracts from the Q-target: high prediction error → lower target → model learns to be conservative on novel states.

- [ ] **Step 4: Wire through Rust launch code**

Add `curiosity_errors_buf` to the kernel launch in `gpu_dqn_trainer.rs`. Pass the prediction error buffer from step 2.

- [ ] **Step 5: Add to TOML configs**

```toml
[advanced]
curiosity_q_penalty_weight = 0.5
```

- [ ] **Step 6: Compile + full smoke test suite**

```bash
SQLX_OFFLINE=true cargo check -p ml
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests --ignored --nocapture
```

All 11 smoke tests must pass.

- [ ] **Step 7: Commit**

```bash
git commit -m "feat: curiosity prediction error as Q-penalty (inverse model generalization)"
```

---

## Verification Plan

After all 5 techniques are implemented:

1. **Local 3-epoch smoke test** — all 11 pass, zero NaN
2. **Local 200-epoch training** — compare MaxDD and Sharpe trajectory to pre-fix baseline
3. **H100 hyperopt** — 20 trials × 8 epochs, compare OOS Sharpe distribution to baseline (-1.30)
4. **Key metric**: OOS Sharpe should be closer to 0 (ideally > -0.5). Omega should be more consistent with total return (no contradiction).

## Expected Impact

| Technique | Mechanism | Expected Gap Reduction |
|-----------|-----------|----------------------|
| Domain randomization | Prevents memorizing fixed sim params | 40-60% |
| Shrink-and-perturb | Kills memorized weights periodically | 15-25% |
| Adversarial injection | Forces worst-case survival | 10-20% |
| Anti-correlation penalty | Punishes regime-specific behavior | 10-15% |
| Curiosity Q-penalty | Conservative on novel states | 20-30% |

Combined expected impact: 50-80% generalization gap reduction.

---

### Task 6: Regime-Adversarial Training (Gradient Reversal)

**Files:**
- Create: `crates/ml/src/cuda_pipeline/regime_discriminator_kernel.cu`
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (add discriminator head)
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs` (gradient reversal)

Train a small classifier head on the DQN's hidden features that predicts which
time window (fold) the data comes from. Apply GRADIENT REVERSAL from the
discriminator to the feature extractor — the DQN learns features that the
discriminator CANNOT use to identify the regime.

This is Domain Adversarial Neural Networks (DANN) adapted for temporal domains.

- [ ] **Step 1: Add discriminator kernel** — small MLP [hidden_dim → 64 → num_folds]
- [ ] **Step 2: Wire gradient reversal** — during backward pass, negate the discriminator's gradient before it flows into the shared trunk
- [ ] **Step 3: Add fold ID to experience buffer** — each experience carries which time window it came from
- [ ] **Step 4: Compile + test + commit**

---

### Task 7: Counterfactual Experience Augmentation

**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (dual action evaluation)
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (double output buffers)

For every trade taken, also compute the counterfactual: what would have happened
with the mirror action (Long→Short, Short→Long). Store both with inverted
reward signs. This doubles the effective data diversity.

- [ ] **Step 1: In experience kernel**, after computing reward for the taken action, compute reward for the mirror action
- [ ] **Step 2: Write both** to output buffers (2x output size)
- [ ] **Step 3: Counterfactual reward** = -1 × actual reward (inverted signal)
- [ ] **Step 4: Compile + test + commit**

---

## Execution Order (All 7 Techniques)

1. Domain randomization (Task 1) — Rust only, highest impact
2. Shrink-and-perturb scheduling (Task 2) — Rust only, already implemented
3. Adversarial regime injection (Task 3) — Rust only
4. Anti-correlation penalty (Task 4) — minor CUDA change
5. Curiosity Q-penalty (Task 5) — repurpose existing model
6. Counterfactual augmentation (Task 7) — CUDA kernel change
7. Regime-adversarial training (Task 6) — new network head

Tasks 1-3 can be implemented in a single session (Rust only).
Tasks 4-5 require CUDA kernel changes.
Tasks 6-7 are the most ambitious and should come after validating 1-5.

---

### Task 8: Hindsight Regime Labeling — Feature Importance Filtering

**Files:**
- Create: `crates/ml/src/trainers/dqn/feature_importance.rs`
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (epoch boundary analysis)
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (feature masking)
- Modify: `crates/ml/src/trainers/dqn/config.rs` (add feature_mask config)

After each epoch, compute gradient × activation for each feature to identify
which features actually predicted profitable trades. Mask the bottom 50% of
features (set to 0) in the next epoch. Iterative: the model discovers which
features carry signal and ignores noise.

- [ ] **Step 1: Compute per-feature importance** from gradient × activation at epoch boundary
- [ ] **Step 2: Rank features** by importance, compute a binary mask (top-K survive)
- [ ] **Step 3: Apply mask** in the experience kernel (zero out masked features)
- [ ] **Step 4: Iterate** — recompute importance each epoch, mask adapts
- [ ] **Step 5: Compile + test + commit**

## Updated Execution Order (All 8 Techniques)

1. Domain randomization (Task 1) — Rust only, highest impact
2. CQL alpha boost (from researcher plan) — config change, 2 hours
3. Shrink-and-perturb scheduling (Task 2) — Rust only, already implemented
4. Adversarial regime injection (Task 3) — Rust only
5. Anti-correlation penalty (Task 4) — minor CUDA change
6. Curiosity Q-penalty (Task 5) — repurpose existing model
7. Counterfactual augmentation (Task 7) — CUDA kernel change
8. Hindsight feature filtering (Task 8) — feature masking
9. Regime-adversarial training (Task 6) — new network head (most ambitious)

Phase 1 (today): Tasks 1-4 (Rust only, immediate impact)
Phase 2 (next session): Tasks 5-8 (CUDA changes, deeper integration)
Phase 3 (after validation): Task 6 (new network architecture)