foxhunt/docs/superpowers/plans/2026-03-29-novel-generalization.md

Novel Generalization Techniques Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Close the in-sample/OOS generalization gap (Sharpe +0.57 IS → -1.30 OOS) by implementing 5 novel techniques that force the model to learn regime-invariant features instead of memorizing price sequences.

Architecture: All techniques integrate into the existing training loop without changing the core DQN architecture. Techniques 1 and 4 modify Rust-side training config per epoch. Technique 2 adds a shuffle flag to episode starts. Technique 3 computes cross-validation action similarity at epoch boundary. Technique 5 repurposes the existing curiosity forward model's prediction error as a CQL-like Q-penalty in the loss kernel.

Tech Stack: Rust, CUDA (cudarc), existing fused CUDA kernels, existing curiosity forward model

---

Task 1: Domain Randomization — Per-Epoch Simulation Parameter Jitter

Files:

• Modify: crates/ml/src/trainers/dqn/trainer/training_loop.rs:1007-1058 (ExperienceCollectorConfig construction)
• Modify: crates/ml/src/trainers/dqn/config.rs (add enable_domain_randomization flag)
• Modify: config/training/dqn-localdev.toml, config/training/dqn-production.toml

The ExperienceCollectorConfig is constructed fresh each epoch at training_loop.rs:1007. Currently all values are deterministic. Add per-epoch randomization of spread, tx_cost, fill probabilities, and initial capital.

• Step 1: Add config flag

In config.rs, after enable_circuit_breaker (~line 1407), add:

pub enable_domain_randomization: bool,

Default in conservative(): true.

• Step 2: Add randomization to ExperienceCollectorConfig construction

In training_loop.rs, at the ExperienceCollectorConfig construction (~line 1007), wrap the fixed values with randomization when enabled:

use rand::Rng;
let mut epoch_rng = rand::rng();

let config = ExperienceCollectorConfig {
    // ... existing fields ...
    tx_cost_multiplier: if self.hyperparams.enable_domain_randomization {
        epoch_rng.random_range(0.5..2.5)
    } else {
        self.hyperparams.transaction_cost_multiplier as f32
    },
    fill_median_spread: if self.hyperparams.enable_domain_randomization {
        self.hyperparams.avg_spread as f32 * epoch_rng.random_range(0.5..3.0)
    } else {
        self.hyperparams.avg_spread as f32
    },
    fill_ioc_fill_prob: if self.hyperparams.enable_domain_randomization {
        epoch_rng.random_range(0.65..0.95)
    } else {
        self.hyperparams.fill_ioc_fill_prob as f32
    },
    fill_limit_fill_min: if self.hyperparams.enable_domain_randomization {
        epoch_rng.random_range(0.15..0.45)
    } else {
        self.hyperparams.fill_limit_fill_min as f32
    },
    fill_limit_fill_max: if self.hyperparams.enable_domain_randomization {
        epoch_rng.random_range(0.60..0.95)
    } else {
        self.hyperparams.fill_limit_fill_max as f32
    },
    spread_cost: if self.hyperparams.enable_domain_randomization {
        (self.hyperparams.tick_size * self.hyperparams.contract_multiplier
         * epoch_rng.random_range(0.3..0.8)) as f32
    } else {
        (self.hyperparams.tick_size * self.hyperparams.contract_multiplier
         * self.hyperparams.fill_spread_cost_frac) as f32
    },
    // ... rest unchanged ...
};

• Step 3: Add episode start jitter

In training_loop.rs at line 935-937, add ±25% stride jitter:

let mut episode_starts: Vec<i32> = (0..n_episodes)
    .map(|i| {
        let base = (i * stride).rem_euclid(usable_bars);
        if self.hyperparams.enable_domain_randomization {
            let jitter = epoch_rng.random_range(-(stride/4)..(stride/4));
            (base + jitter).rem_euclid(usable_bars)
        } else {
            base
        }
    })
    .collect();

• Step 4: Add variable episode length

Before the timesteps assignment at line 931, randomize:

let timesteps = if self.hyperparams.enable_domain_randomization {
    let base = self.hyperparams.gpu_timesteps_per_episode.min(1000) as i32;
    let jitter = epoch_rng.random_range(-(base/4)..(base/4));
    (base + jitter).max(50) // minimum 50 timesteps
} else {
    self.hyperparams.gpu_timesteps_per_episode.min(1000) as i32
};

• Step 5: Add to TOML configs

Add to [advanced] section in dqn-localdev.toml and dqn-production.toml:

enable_domain_randomization = true

• Step 6: Compile + smoke test

SQLX_OFFLINE=true cargo check -p ml
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests::training_stability::test_production_training_stability --ignored --nocapture

• Step 7: Commit

git add crates/ml/src/trainers/dqn/trainer/training_loop.rs crates/ml/src/trainers/dqn/config.rs config/training/*.toml
git commit -m "feat: domain randomization — per-epoch jitter on spread, tx_cost, fills, episode starts"

---

Task 2: Periodic Shrink-and-Perturb (already exists, needs scheduling)

Files:

• Modify: crates/ml/src/trainers/dqn/trainer/training_loop.rs:308-347 (phase boundary S&P)
• Modify: crates/ml/src/trainers/dqn/config.rs (add shrink_perturb_interval)

Shrink-and-perturb already works (fused.shrink_and_perturb(alpha, sigma)) but only runs at phase boundaries. Need periodic execution every N epochs.

• Step 1: Add config field

In config.rs, add:

pub shrink_perturb_interval: usize, // 0=disabled, 20=every 20 epochs
pub shrink_perturb_alpha: f64,      // 0.85 = keep 85%, reinit 15%
pub shrink_perturb_sigma: f64,      // 0.01 = noise scale

Defaults in conservative(): shrink_perturb_interval: 20, shrink_perturb_alpha: 0.85, shrink_perturb_sigma: 0.01.

• Step 2: Add periodic S&P to training loop

In training_loop.rs, in the main epoch loop (after experience collection, before training steps), add:

if self.hyperparams.shrink_perturb_interval > 0
    && epoch > 0
    && epoch % self.hyperparams.shrink_perturb_interval == 0
{
    if let Some(ref mut fused) = self.fused_ctx {
        let alpha = self.hyperparams.shrink_perturb_alpha as f32;
        let sigma = self.hyperparams.shrink_perturb_sigma as f32;
        match fused.shrink_and_perturb(alpha, sigma) {
            Ok(()) => info!(epoch, alpha, sigma, "Periodic shrink-and-perturb applied"),
            Err(e) => warn!(epoch, "Shrink-and-perturb failed (non-fatal): {e}"),
        }
    }
}

• Step 3: Add to TOMLs + profile

dqn-localdev.toml and dqn-production.toml [advanced]:

shrink_perturb_interval = 20
shrink_perturb_alpha = 0.85
shrink_perturb_sigma = 0.01

Wire through training_profile.rs apply_to() if fields are added to AdvancedSection.

• Step 4: Compile + test

SQLX_OFFLINE=true cargo check -p ml
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests --ignored --nocapture

• Step 5: Commit

git commit -m "feat: periodic shrink-and-perturb every N epochs (default 20)"

---

Task 3: Adversarial Regime Injection

Files:

• Modify: crates/ml/src/trainers/dqn/trainer/training_loop.rs (epoch-level regime detection + injection)
• Modify: crates/ml/src/cuda_pipeline/gpu_experience_collector.rs (ExperienceCollectorConfig fields)
• Modify: crates/ml/src/trainers/dqn/config.rs (add adversarial_injection_threshold)

When the model achieves low drawdown (<10%) for 3+ consecutive epochs, inject adversarial conditions: 3x spread, 2x tx_cost, 0.5x fill probability. This forces the model to handle worst-case scenarios.

• Step 1: Add config + tracking

In config.rs:

pub adversarial_injection_threshold: f64, // DD threshold (0.10 = 10%)
pub adversarial_injection_consecutive: usize, // consecutive good epochs before trigger (3)

In trainer/mod.rs, add tracking field:

pub(crate) consecutive_low_dd_epochs: usize,

• Step 2: Implement detection + injection

In training_loop.rs, after epoch metrics are computed (after the Sharpe/MaxDD/PF log), add:

// Adversarial regime injection: when model is doing too well, make it harder
let max_dd_fraction = monitor.max_drawdown_pct / 100.0;
if max_dd_fraction < self.hyperparams.adversarial_injection_threshold {
    self.consecutive_low_dd_epochs += 1;
} else {
    self.consecutive_low_dd_epochs = 0;
}

let adversarial_active = self.consecutive_low_dd_epochs
    >= self.hyperparams.adversarial_injection_consecutive;

Then in the ExperienceCollectorConfig construction for the NEXT epoch, check adversarial_active:

let adversarial_mult = if adversarial_active { 3.0_f32 } else { 1.0 };
// Apply to spread_cost, tx_cost_multiplier, etc.
spread_cost: base_spread * adversarial_mult,
tx_cost_multiplier: base_tx * (if adversarial_active { 2.0 } else { 1.0 }),
fill_ioc_fill_prob: if adversarial_active { 0.50 } else { base_fill },

• Step 3: Log when adversarial mode activates

if adversarial_active {
    info!(
        epoch, consecutive = self.consecutive_low_dd_epochs,
        "ADVERSARIAL REGIME INJECTED: 3x spread, 2x tx_cost, 0.5x fill prob"
    );
}

• Step 4: Compile + test + commit

---

Task 4: Anti-Correlation Reward Penalty

Files:

• Modify: crates/ml/src/trainers/dqn/trainer/training_loop.rs (epoch boundary action comparison)
• Modify: crates/ml/src/cuda_pipeline/gpu_experience_collector.rs (reward_penalty_multiplier field)
• Modify: crates/ml/src/cuda_pipeline/experience_kernels.cu (apply penalty to reward)

At epoch end, compare the action distribution (from GPU monitoring) between training and validation. If cosine similarity > 0.8, the model is doing the same thing on both — it hasn't learned to differentiate. Penalize next epoch's rewards.

• Step 1: Compute cosine similarity at epoch boundary

In training_loop.rs, after validation loss computation, add:

// Anti-correlation: cosine similarity between train and val action distributions
let train_actions: Vec<f64> = monitor.action_counts.iter().map(|&c| c as f64).collect();
let val_actions: Vec<f64> = /* from validation pass action counts */;

let dot: f64 = train_actions.iter().zip(&val_actions).map(|(a, b)| a * b).sum();
let mag_t: f64 = train_actions.iter().map(|a| a * a).sum::<f64>().sqrt();
let mag_v: f64 = val_actions.iter().map(|a| a * a).sum::<f64>().sqrt();
let cosine_sim = if mag_t > 0.0 && mag_v > 0.0 { dot / (mag_t * mag_v) } else { 0.0 };

let anti_corr_penalty = if cosine_sim > 0.8 {
    0.8_f32 // scale rewards by 0.8 next epoch (20% penalty)
} else {
    1.0_f32
};

• Step 2: Pass penalty to experience collector

Add reward_scale_override: f32 to ExperienceCollectorConfig. Default 1.0. In the experience kernel, multiply reward by this scale factor:

reward *= reward_scale_override;

• Step 3: Compile + test + commit

---

Task 5: Curiosity Prediction Error as Q-Penalty (Inverse Model)

Files:

• Modify: crates/ml/src/trainers/dqn/fused_training.rs (wire curiosity error into loss)
• Modify: crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs (compute curiosity error per batch)
• Modify: crates/ml/src/cuda_pipeline/mse_loss_kernel.cu or c51_loss_kernel.cu (add penalty term)
• Modify: crates/ml/src/trainers/dqn/config.rs (add curiosity_q_penalty_weight)

The curiosity forward model (state+action → next_state) already exists and trains. Its prediction error measures how "novel" a state-action pair is. High prediction error = the model has never seen this situation = Q-values should be conservative (lower).

Wire the curiosity prediction error as an ADDITIVE penalty to the Q-target in the Bellman equation:

target_Q = reward + gamma * (1-done) * target_Q_next - curiosity_penalty * prediction_error

This makes the Q-function conservative on novel state-action pairs (same principle as CQL, but data-driven instead of action-space-driven).

• Step 1: Add config field

pub curiosity_q_penalty_weight: f64, // 0.0=disabled, 0.5=moderate, 1.0=strong

Default: 0.5.

• Step 2: Compute per-sample curiosity error in fused training step

In fused_training.rs, after the forward pass and before the loss kernel, launch the curiosity forward model on the current batch to get per-sample prediction errors. Store in a CudaSlice<f32> of size [batch_size].

The curiosity model is already initialized in the fused context (self.curiosity_trainer). Call:

let curiosity_errors = self.curiosity_trainer.compute_batch_errors(
    &batch_states, &batch_actions, &batch_next_states, batch_size
)?;

• Step 3: Pass curiosity errors to loss kernel

Add const float* curiosity_errors parameter to mse_loss_batched and c51_loss_batched.

In the Bellman target computation (MSE kernel line 334):

float curiosity_penalty = curiosity_errors[sample_id] * curiosity_q_penalty_weight;
float target_q = reward + gamma * (1.0f - done) * target_eq - curiosity_penalty;

The penalty subtracts from the Q-target: high prediction error → lower target → model learns to be conservative on novel states.

• Step 4: Wire through Rust launch code

Add curiosity_errors_buf to the kernel launch in gpu_dqn_trainer.rs. Pass the prediction error buffer from step 2.

• Step 5: Add to TOML configs

[advanced]
curiosity_q_penalty_weight = 0.5

• Step 6: Compile + full smoke test suite

SQLX_OFFLINE=true cargo check -p ml
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests --ignored --nocapture

All 11 smoke tests must pass.

• Step 7: Commit

git commit -m "feat: curiosity prediction error as Q-penalty (inverse model generalization)"

---

Verification Plan

After all 5 techniques are implemented:

1. Local 3-epoch smoke test — all 11 pass, zero NaN
2. Local 200-epoch training — compare MaxDD and Sharpe trajectory to pre-fix baseline
3. H100 hyperopt — 20 trials × 8 epochs, compare OOS Sharpe distribution to baseline (-1.30)
4. Key metric: OOS Sharpe should be closer to 0 (ideally > -0.5). Omega should be more consistent with total return (no contradiction).

Expected Impact

Technique	Mechanism	Expected Gap Reduction
Domain randomization	Prevents memorizing fixed sim params	40-60%
Shrink-and-perturb	Kills memorized weights periodically	15-25%
Adversarial injection	Forces worst-case survival	10-20%
Anti-correlation penalty	Punishes regime-specific behavior	10-15%
Curiosity Q-penalty	Conservative on novel states	20-30%

Combined expected impact: 50-80% generalization gap reduction.

---

Task 6: Regime-Adversarial Training (Gradient Reversal)

Files:

• Create: crates/ml/src/cuda_pipeline/regime_discriminator_kernel.cu
• Modify: crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs (add discriminator head)
• Modify: crates/ml/src/trainers/dqn/fused_training.rs (gradient reversal)

Train a small classifier head on the DQN's hidden features that predicts which time window (fold) the data comes from. Apply GRADIENT REVERSAL from the discriminator to the feature extractor — the DQN learns features that the discriminator CANNOT use to identify the regime.

This is Domain Adversarial Neural Networks (DANN) adapted for temporal domains.

• Step 1: Add discriminator kernel — small MLP [hidden_dim → 64 → num_folds]
• Step 2: Wire gradient reversal — during backward pass, negate the discriminator's gradient before it flows into the shared trunk
• Step 3: Add fold ID to experience buffer — each experience carries which time window it came from
• Step 4: Compile + test + commit

---

Task 7: Counterfactual Experience Augmentation

Files:

• Modify: crates/ml/src/cuda_pipeline/experience_kernels.cu (dual action evaluation)
• Modify: crates/ml/src/cuda_pipeline/gpu_experience_collector.rs (double output buffers)

For every trade taken, also compute the counterfactual: what would have happened with the mirror action (Long→Short, Short→Long). Store both with inverted reward signs. This doubles the effective data diversity.

• Step 1: In experience kernel, after computing reward for the taken action, compute reward for the mirror action
• Step 2: Write both to output buffers (2x output size)
• Step 3: Counterfactual reward = -1 × actual reward (inverted signal)
• Step 4: Compile + test + commit

---

Execution Order (All 7 Techniques)

1. Domain randomization (Task 1) — Rust only, highest impact
2. Shrink-and-perturb scheduling (Task 2) — Rust only, already implemented
3. Adversarial regime injection (Task 3) — Rust only
4. Anti-correlation penalty (Task 4) — minor CUDA change
5. Curiosity Q-penalty (Task 5) — repurpose existing model
6. Counterfactual augmentation (Task 7) — CUDA kernel change
7. Regime-adversarial training (Task 6) — new network head

Tasks 1-3 can be implemented in a single session (Rust only). Tasks 4-5 require CUDA kernel changes. Tasks 6-7 are the most ambitious and should come after validating 1-5.

---

Task 8: Hindsight Regime Labeling — Feature Importance Filtering

Files:

• Create: crates/ml/src/trainers/dqn/feature_importance.rs
• Modify: crates/ml/src/trainers/dqn/trainer/training_loop.rs (epoch boundary analysis)
• Modify: crates/ml/src/cuda_pipeline/experience_kernels.cu (feature masking)
• Modify: crates/ml/src/trainers/dqn/config.rs (add feature_mask config)

After each epoch, compute gradient × activation for each feature to identify which features actually predicted profitable trades. Mask the bottom 50% of features (set to 0) in the next epoch. Iterative: the model discovers which features carry signal and ignores noise.

• Step 1: Compute per-feature importance from gradient × activation at epoch boundary
• Step 2: Rank features by importance, compute a binary mask (top-K survive)
• Step 3: Apply mask in the experience kernel (zero out masked features)
• Step 4: Iterate — recompute importance each epoch, mask adapts
• Step 5: Compile + test + commit

Updated Execution Order (All 8 Techniques)

1. Domain randomization (Task 1) — Rust only, highest impact
2. CQL alpha boost (from researcher plan) — config change, 2 hours
3. Shrink-and-perturb scheduling (Task 2) — Rust only, already implemented
4. Adversarial regime injection (Task 3) — Rust only
5. Anti-correlation penalty (Task 4) — minor CUDA change
6. Curiosity Q-penalty (Task 5) — repurpose existing model
7. Counterfactual augmentation (Task 7) — CUDA kernel change
8. Hindsight feature filtering (Task 8) — feature masking
9. Regime-adversarial training (Task 6) — new network head (most ambitious)

Phase 1 (today): Tasks 1-4 (Rust only, immediate impact) Phase 2 (next session): Tasks 5-8 (CUDA changes, deeper integration) Phase 3 (after validation): Task 6 (new network architecture)