Files
foxhunt/docs/superpowers/plans/2026-03-29-novel-generalization.md

481 lines
19 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Novel Generalization Techniques Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Close the in-sample/OOS generalization gap (Sharpe +0.57 IS → -1.30 OOS) by implementing 5 novel techniques that force the model to learn regime-invariant features instead of memorizing price sequences.
**Architecture:** All techniques integrate into the existing training loop without changing the core DQN architecture. Techniques 1 and 4 modify Rust-side training config per epoch. Technique 2 adds a shuffle flag to episode starts. Technique 3 computes cross-validation action similarity at epoch boundary. Technique 5 repurposes the existing curiosity forward model's prediction error as a CQL-like Q-penalty in the loss kernel.
**Tech Stack:** Rust, CUDA (cudarc), existing fused CUDA kernels, existing curiosity forward model
---
### Task 1: Domain Randomization — Per-Epoch Simulation Parameter Jitter
**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs:1007-1058` (ExperienceCollectorConfig construction)
- Modify: `crates/ml/src/trainers/dqn/config.rs` (add `enable_domain_randomization` flag)
- Modify: `config/training/dqn-localdev.toml`, `config/training/dqn-production.toml`
The ExperienceCollectorConfig is constructed fresh each epoch at training_loop.rs:1007. Currently all values are deterministic. Add per-epoch randomization of spread, tx_cost, fill probabilities, and initial capital.
- [ ] **Step 1: Add config flag**
In `config.rs`, after `enable_circuit_breaker` (~line 1407), add:
```rust
pub enable_domain_randomization: bool,
```
Default in `conservative()`: `true`.
- [ ] **Step 2: Add randomization to ExperienceCollectorConfig construction**
In `training_loop.rs`, at the ExperienceCollectorConfig construction (~line 1007), wrap the fixed values with randomization when enabled:
```rust
use rand::Rng;
let mut epoch_rng = rand::rng();
let config = ExperienceCollectorConfig {
// ... existing fields ...
tx_cost_multiplier: if self.hyperparams.enable_domain_randomization {
epoch_rng.random_range(0.5..2.5)
} else {
self.hyperparams.transaction_cost_multiplier as f32
},
fill_median_spread: if self.hyperparams.enable_domain_randomization {
self.hyperparams.avg_spread as f32 * epoch_rng.random_range(0.5..3.0)
} else {
self.hyperparams.avg_spread as f32
},
fill_ioc_fill_prob: if self.hyperparams.enable_domain_randomization {
epoch_rng.random_range(0.65..0.95)
} else {
self.hyperparams.fill_ioc_fill_prob as f32
},
fill_limit_fill_min: if self.hyperparams.enable_domain_randomization {
epoch_rng.random_range(0.15..0.45)
} else {
self.hyperparams.fill_limit_fill_min as f32
},
fill_limit_fill_max: if self.hyperparams.enable_domain_randomization {
epoch_rng.random_range(0.60..0.95)
} else {
self.hyperparams.fill_limit_fill_max as f32
},
spread_cost: if self.hyperparams.enable_domain_randomization {
(self.hyperparams.tick_size * self.hyperparams.contract_multiplier
* epoch_rng.random_range(0.3..0.8)) as f32
} else {
(self.hyperparams.tick_size * self.hyperparams.contract_multiplier
* self.hyperparams.fill_spread_cost_frac) as f32
},
// ... rest unchanged ...
};
```
- [ ] **Step 3: Add episode start jitter**
In `training_loop.rs` at line 935-937, add ±25% stride jitter:
```rust
let mut episode_starts: Vec<i32> = (0..n_episodes)
.map(|i| {
let base = (i * stride).rem_euclid(usable_bars);
if self.hyperparams.enable_domain_randomization {
let jitter = epoch_rng.random_range(-(stride/4)..(stride/4));
(base + jitter).rem_euclid(usable_bars)
} else {
base
}
})
.collect();
```
- [ ] **Step 4: Add variable episode length**
Before the `timesteps` assignment at line 931, randomize:
```rust
let timesteps = if self.hyperparams.enable_domain_randomization {
let base = self.hyperparams.gpu_timesteps_per_episode.min(1000) as i32;
let jitter = epoch_rng.random_range(-(base/4)..(base/4));
(base + jitter).max(50) // minimum 50 timesteps
} else {
self.hyperparams.gpu_timesteps_per_episode.min(1000) as i32
};
```
- [ ] **Step 5: Add to TOML configs**
Add to `[advanced]` section in dqn-localdev.toml and dqn-production.toml:
```toml
enable_domain_randomization = true
```
- [ ] **Step 6: Compile + smoke test**
```bash
SQLX_OFFLINE=true cargo check -p ml
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests::training_stability::test_production_training_stability --ignored --nocapture
```
- [ ] **Step 7: Commit**
```bash
git add crates/ml/src/trainers/dqn/trainer/training_loop.rs crates/ml/src/trainers/dqn/config.rs config/training/*.toml
git commit -m "feat: domain randomization — per-epoch jitter on spread, tx_cost, fills, episode starts"
```
---
### Task 2: Periodic Shrink-and-Perturb (already exists, needs scheduling)
**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs:308-347` (phase boundary S&P)
- Modify: `crates/ml/src/trainers/dqn/config.rs` (add `shrink_perturb_interval`)
Shrink-and-perturb already works (`fused.shrink_and_perturb(alpha, sigma)`) but only runs at phase boundaries. Need periodic execution every N epochs.
- [ ] **Step 1: Add config field**
In `config.rs`, add:
```rust
pub shrink_perturb_interval: usize, // 0=disabled, 20=every 20 epochs
pub shrink_perturb_alpha: f64, // 0.85 = keep 85%, reinit 15%
pub shrink_perturb_sigma: f64, // 0.01 = noise scale
```
Defaults in `conservative()`: `shrink_perturb_interval: 20, shrink_perturb_alpha: 0.85, shrink_perturb_sigma: 0.01`.
- [ ] **Step 2: Add periodic S&P to training loop**
In `training_loop.rs`, in the main epoch loop (after experience collection, before training steps), add:
```rust
if self.hyperparams.shrink_perturb_interval > 0
&& epoch > 0
&& epoch % self.hyperparams.shrink_perturb_interval == 0
{
if let Some(ref mut fused) = self.fused_ctx {
let alpha = self.hyperparams.shrink_perturb_alpha as f32;
let sigma = self.hyperparams.shrink_perturb_sigma as f32;
match fused.shrink_and_perturb(alpha, sigma) {
Ok(()) => info!(epoch, alpha, sigma, "Periodic shrink-and-perturb applied"),
Err(e) => warn!(epoch, "Shrink-and-perturb failed (non-fatal): {e}"),
}
}
}
```
- [ ] **Step 3: Add to TOMLs + profile**
dqn-localdev.toml and dqn-production.toml `[advanced]`:
```toml
shrink_perturb_interval = 20
shrink_perturb_alpha = 0.85
shrink_perturb_sigma = 0.01
```
Wire through `training_profile.rs` `apply_to()` if fields are added to `AdvancedSection`.
- [ ] **Step 4: Compile + test**
```bash
SQLX_OFFLINE=true cargo check -p ml
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests --ignored --nocapture
```
- [ ] **Step 5: Commit**
```bash
git commit -m "feat: periodic shrink-and-perturb every N epochs (default 20)"
```
---
### Task 3: Adversarial Regime Injection
**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (epoch-level regime detection + injection)
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (ExperienceCollectorConfig fields)
- Modify: `crates/ml/src/trainers/dqn/config.rs` (add `adversarial_injection_threshold`)
When the model achieves low drawdown (<10%) for 3+ consecutive epochs, inject adversarial conditions: 3x spread, 2x tx_cost, 0.5x fill probability. This forces the model to handle worst-case scenarios.
- [ ] **Step 1: Add config + tracking**
In `config.rs`:
```rust
pub adversarial_injection_threshold: f64, // DD threshold (0.10 = 10%)
pub adversarial_injection_consecutive: usize, // consecutive good epochs before trigger (3)
```
In `trainer/mod.rs`, add tracking field:
```rust
pub(crate) consecutive_low_dd_epochs: usize,
```
- [ ] **Step 2: Implement detection + injection**
In `training_loop.rs`, after epoch metrics are computed (after the Sharpe/MaxDD/PF log), add:
```rust
// Adversarial regime injection: when model is doing too well, make it harder
let max_dd_fraction = monitor.max_drawdown_pct / 100.0;
if max_dd_fraction < self.hyperparams.adversarial_injection_threshold {
self.consecutive_low_dd_epochs += 1;
} else {
self.consecutive_low_dd_epochs = 0;
}
let adversarial_active = self.consecutive_low_dd_epochs
>= self.hyperparams.adversarial_injection_consecutive;
```
Then in the ExperienceCollectorConfig construction for the NEXT epoch, check `adversarial_active`:
```rust
let adversarial_mult = if adversarial_active { 3.0_f32 } else { 1.0 };
// Apply to spread_cost, tx_cost_multiplier, etc.
spread_cost: base_spread * adversarial_mult,
tx_cost_multiplier: base_tx * (if adversarial_active { 2.0 } else { 1.0 }),
fill_ioc_fill_prob: if adversarial_active { 0.50 } else { base_fill },
```
- [ ] **Step 3: Log when adversarial mode activates**
```rust
if adversarial_active {
info!(
epoch, consecutive = self.consecutive_low_dd_epochs,
"ADVERSARIAL REGIME INJECTED: 3x spread, 2x tx_cost, 0.5x fill prob"
);
}
```
- [ ] **Step 4: Compile + test + commit**
---
### Task 4: Anti-Correlation Reward Penalty
**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (epoch boundary action comparison)
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (reward_penalty_multiplier field)
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (apply penalty to reward)
At epoch end, compare the action distribution (from GPU monitoring) between training and validation. If cosine similarity > 0.8, the model is doing the same thing on both — it hasn't learned to differentiate. Penalize next epoch's rewards.
- [ ] **Step 1: Compute cosine similarity at epoch boundary**
In `training_loop.rs`, after validation loss computation, add:
```rust
// Anti-correlation: cosine similarity between train and val action distributions
let train_actions: Vec<f64> = monitor.action_counts.iter().map(|&c| c as f64).collect();
let val_actions: Vec<f64> = /* from validation pass action counts */;
let dot: f64 = train_actions.iter().zip(&val_actions).map(|(a, b)| a * b).sum();
let mag_t: f64 = train_actions.iter().map(|a| a * a).sum::<f64>().sqrt();
let mag_v: f64 = val_actions.iter().map(|a| a * a).sum::<f64>().sqrt();
let cosine_sim = if mag_t > 0.0 && mag_v > 0.0 { dot / (mag_t * mag_v) } else { 0.0 };
let anti_corr_penalty = if cosine_sim > 0.8 {
0.8_f32 // scale rewards by 0.8 next epoch (20% penalty)
} else {
1.0_f32
};
```
- [ ] **Step 2: Pass penalty to experience collector**
Add `reward_scale_override: f32` to `ExperienceCollectorConfig`. Default 1.0.
In the experience kernel, multiply reward by this scale factor:
```cuda
reward *= reward_scale_override;
```
- [ ] **Step 3: Compile + test + commit**
---
### Task 5: Curiosity Prediction Error as Q-Penalty (Inverse Model)
**Files:**
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs` (wire curiosity error into loss)
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (compute curiosity error per batch)
- Modify: `crates/ml/src/cuda_pipeline/mse_loss_kernel.cu` or `c51_loss_kernel.cu` (add penalty term)
- Modify: `crates/ml/src/trainers/dqn/config.rs` (add `curiosity_q_penalty_weight`)
The curiosity forward model (state+action → next_state) already exists and trains. Its prediction error measures how "novel" a state-action pair is. High prediction error = the model has never seen this situation = Q-values should be conservative (lower).
Wire the curiosity prediction error as an ADDITIVE penalty to the Q-target in the Bellman equation:
```
target_Q = reward + gamma * (1-done) * target_Q_next - curiosity_penalty * prediction_error
```
This makes the Q-function conservative on novel state-action pairs (same principle as CQL, but data-driven instead of action-space-driven).
- [ ] **Step 1: Add config field**
```rust
pub curiosity_q_penalty_weight: f64, // 0.0=disabled, 0.5=moderate, 1.0=strong
```
Default: `0.5`.
- [ ] **Step 2: Compute per-sample curiosity error in fused training step**
In `fused_training.rs`, after the forward pass and before the loss kernel, launch the curiosity forward model on the current batch to get per-sample prediction errors. Store in a `CudaSlice<f32>` of size `[batch_size]`.
The curiosity model is already initialized in the fused context (`self.curiosity_trainer`). Call:
```rust
let curiosity_errors = self.curiosity_trainer.compute_batch_errors(
&batch_states, &batch_actions, &batch_next_states, batch_size
)?;
```
- [ ] **Step 3: Pass curiosity errors to loss kernel**
Add `const float* curiosity_errors` parameter to `mse_loss_batched` and `c51_loss_batched`.
In the Bellman target computation (MSE kernel line 334):
```cuda
float curiosity_penalty = curiosity_errors[sample_id] * curiosity_q_penalty_weight;
float target_q = reward + gamma * (1.0f - done) * target_eq - curiosity_penalty;
```
The penalty subtracts from the Q-target: high prediction error → lower target → model learns to be conservative on novel states.
- [ ] **Step 4: Wire through Rust launch code**
Add `curiosity_errors_buf` to the kernel launch in `gpu_dqn_trainer.rs`. Pass the prediction error buffer from step 2.
- [ ] **Step 5: Add to TOML configs**
```toml
[advanced]
curiosity_q_penalty_weight = 0.5
```
- [ ] **Step 6: Compile + full smoke test suite**
```bash
SQLX_OFFLINE=true cargo check -p ml
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests --ignored --nocapture
```
All 11 smoke tests must pass.
- [ ] **Step 7: Commit**
```bash
git commit -m "feat: curiosity prediction error as Q-penalty (inverse model generalization)"
```
---
## Verification Plan
After all 5 techniques are implemented:
1. **Local 3-epoch smoke test** — all 11 pass, zero NaN
2. **Local 200-epoch training** — compare MaxDD and Sharpe trajectory to pre-fix baseline
3. **H100 hyperopt** — 20 trials × 8 epochs, compare OOS Sharpe distribution to baseline (-1.30)
4. **Key metric**: OOS Sharpe should be closer to 0 (ideally > -0.5). Omega should be more consistent with total return (no contradiction).
## Expected Impact
| Technique | Mechanism | Expected Gap Reduction |
|-----------|-----------|----------------------|
| Domain randomization | Prevents memorizing fixed sim params | 40-60% |
| Shrink-and-perturb | Kills memorized weights periodically | 15-25% |
| Adversarial injection | Forces worst-case survival | 10-20% |
| Anti-correlation penalty | Punishes regime-specific behavior | 10-15% |
| Curiosity Q-penalty | Conservative on novel states | 20-30% |
Combined expected impact: 50-80% generalization gap reduction.
---
### Task 6: Regime-Adversarial Training (Gradient Reversal)
**Files:**
- Create: `crates/ml/src/cuda_pipeline/regime_discriminator_kernel.cu`
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (add discriminator head)
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs` (gradient reversal)
Train a small classifier head on the DQN's hidden features that predicts which
time window (fold) the data comes from. Apply GRADIENT REVERSAL from the
discriminator to the feature extractor — the DQN learns features that the
discriminator CANNOT use to identify the regime.
This is Domain Adversarial Neural Networks (DANN) adapted for temporal domains.
- [ ] **Step 1: Add discriminator kernel** — small MLP [hidden_dim → 64 → num_folds]
- [ ] **Step 2: Wire gradient reversal** — during backward pass, negate the discriminator's gradient before it flows into the shared trunk
- [ ] **Step 3: Add fold ID to experience buffer** — each experience carries which time window it came from
- [ ] **Step 4: Compile + test + commit**
---
### Task 7: Counterfactual Experience Augmentation
**Files:**
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (dual action evaluation)
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (double output buffers)
For every trade taken, also compute the counterfactual: what would have happened
with the mirror action (Long→Short, Short→Long). Store both with inverted
reward signs. This doubles the effective data diversity.
- [ ] **Step 1: In experience kernel**, after computing reward for the taken action, compute reward for the mirror action
- [ ] **Step 2: Write both** to output buffers (2x output size)
- [ ] **Step 3: Counterfactual reward** = -1 × actual reward (inverted signal)
- [ ] **Step 4: Compile + test + commit**
---
## Execution Order (All 7 Techniques)
1. Domain randomization (Task 1) — Rust only, highest impact
2. Shrink-and-perturb scheduling (Task 2) — Rust only, already implemented
3. Adversarial regime injection (Task 3) — Rust only
4. Anti-correlation penalty (Task 4) — minor CUDA change
5. Curiosity Q-penalty (Task 5) — repurpose existing model
6. Counterfactual augmentation (Task 7) — CUDA kernel change
7. Regime-adversarial training (Task 6) — new network head
Tasks 1-3 can be implemented in a single session (Rust only).
Tasks 4-5 require CUDA kernel changes.
Tasks 6-7 are the most ambitious and should come after validating 1-5.
---
### Task 8: Hindsight Regime Labeling — Feature Importance Filtering
**Files:**
- Create: `crates/ml/src/trainers/dqn/feature_importance.rs`
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (epoch boundary analysis)
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (feature masking)
- Modify: `crates/ml/src/trainers/dqn/config.rs` (add feature_mask config)
After each epoch, compute gradient × activation for each feature to identify
which features actually predicted profitable trades. Mask the bottom 50% of
features (set to 0) in the next epoch. Iterative: the model discovers which
features carry signal and ignores noise.
- [ ] **Step 1: Compute per-feature importance** from gradient × activation at epoch boundary
- [ ] **Step 2: Rank features** by importance, compute a binary mask (top-K survive)
- [ ] **Step 3: Apply mask** in the experience kernel (zero out masked features)
- [ ] **Step 4: Iterate** — recompute importance each epoch, mask adapts
- [ ] **Step 5: Compile + test + commit**
## Updated Execution Order (All 8 Techniques)
1. Domain randomization (Task 1) — Rust only, highest impact
2. CQL alpha boost (from researcher plan) — config change, 2 hours
3. Shrink-and-perturb scheduling (Task 2) — Rust only, already implemented
4. Adversarial regime injection (Task 3) — Rust only
5. Anti-correlation penalty (Task 4) — minor CUDA change
6. Curiosity Q-penalty (Task 5) — repurpose existing model
7. Counterfactual augmentation (Task 7) — CUDA kernel change
8. Hindsight feature filtering (Task 8) — feature masking
9. Regime-adversarial training (Task 6) — new network head (most ambitious)
Phase 1 (today): Tasks 1-4 (Rust only, immediate impact)
Phase 2 (next session): Tasks 5-8 (CUDA changes, deeper integration)
Phase 3 (after validation): Task 6 (new network architecture)