feat(generalization): 7 gems & pearls — CUDA kernels + config + TOML wiring
Phase 1 generalization techniques to close IS/OOS Sharpe gap (+0.57→-1.30). All techniques fully wired from DQNHyperparameters → TOML [generalization] section → ExperienceCollectorConfig → CUDA kernel arguments. New techniques implemented: - #13 Vol normalization: divide return features by realized vol (state_gather) - #18 Asymmetric DD loss: extra penalty on Q-overestimation in drawdown (mse_loss_batched + c51_loss_batched) - #22 Feature noise: N(0, scale) per feature via LCG RNG (state_gather) - #23 Causal feature masking: random 30% feature subset zeroed per epoch (state_gather, mask uploaded from Rust) - #24 Anti-intuitive LR: 3x LR when Sharpe good, 0.3x when bad (Rust-only) - #25 Trade clustering: CV(inter-trade intervals) penalty via ps[3:6] (env_step, uses reserved portfolio state slots) - #27 Ensemble disagreement: Q_target -= weight * ensemble_std (mse_loss + c51_loss, buffer allocated for future ensemble wiring) Also includes 30-task plan doc with all 28+ techniques across 3 phases. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
@@ -0,0 +1,728 @@
|
||||
# Gems & Pearls — 28 Generalization Techniques for DQN Trading Agent
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Close the in-sample/OOS generalization gap (Sharpe +0.57 IS → -1.30 OOS). The model memorizes price sequences instead of learning regime-invariant market structure. These 28 techniques attack every axis of memorization.
|
||||
|
||||
**Architecture:** Branching DQN with 81 factored actions (9 exposure x 3 order x 3 urgency), 14 active features, fused CUDA training pipeline, C51+MSE blended loss, CQL regularization, curiosity model, ensemble heads.
|
||||
|
||||
**Tech Stack:** Rust, CUDA (cudarc), existing fused CUDA kernels, existing curiosity forward model
|
||||
|
||||
**Status Key:** DONE = committed, WIP = in progress, PLAN = not started
|
||||
|
||||
---
|
||||
|
||||
## Previously Planned (Tasks 1-8 from original plan)
|
||||
|
||||
### Task 1: Domain Randomization — Per-Epoch Simulation Parameter Jitter [DONE]
|
||||
|
||||
**Commit:** `c3ea0e57`
|
||||
|
||||
**Files:**
|
||||
- Modified: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`
|
||||
- Modified: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Per-epoch randomization of spread, tx_cost, fill probabilities, initial capital, episode start positions, and episode lengths. Prevents memorizing fixed simulation parameters.
|
||||
|
||||
- [x] Config flag `enable_domain_randomization`
|
||||
- [x] Randomize ExperienceCollectorConfig per epoch
|
||||
- [x] Episode start jitter +-25% stride
|
||||
- [x] Variable episode length
|
||||
- [x] TOML configs updated
|
||||
- [x] Compiled + tested
|
||||
|
||||
---
|
||||
|
||||
### Task 2: CQL Alpha Boost + Gradient Budget Rebalance [DONE]
|
||||
|
||||
**Commit:** `ae995673`
|
||||
|
||||
**Files:**
|
||||
- Modified: `crates/ml/src/trainers/dqn/config.rs`
|
||||
- Modified: `crates/ml/src/trainers/dqn/fused_training.rs`
|
||||
|
||||
CQL alpha 0.1 -> 1.0, gradient budget 15% -> 25%, C51 budget 70% -> 60%.
|
||||
|
||||
- [x] CQL alpha increased
|
||||
- [x] Gradient budget rebalanced
|
||||
- [x] Compiled + tested
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Periodic Shrink-and-Perturb [DONE]
|
||||
|
||||
**Commit:** `37cb034e`
|
||||
|
||||
**Files:**
|
||||
- Modified: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`
|
||||
- Modified: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Shrink-and-perturb every 20 epochs: keep 85% of weights, reinitialize 15% with noise sigma=0.01. Kills memorized weight patterns periodically.
|
||||
|
||||
- [x] Config fields `shrink_perturb_interval`, `_alpha`, `_sigma`
|
||||
- [x] Periodic S&P in training loop
|
||||
- [x] Compiled + tested
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Adversarial Regime Injection [PLAN]
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/trainer/mod.rs`
|
||||
|
||||
When the model achieves low drawdown (<10%) for 3+ consecutive epochs, inject adversarial conditions: 3x spread, 2x tx_cost, 0.5x fill probability. Forces the model to handle worst-case scenarios.
|
||||
|
||||
- [ ] Add config `adversarial_injection_threshold` (0.10), `adversarial_injection_consecutive` (3)
|
||||
- [ ] Add tracking field `consecutive_low_dd_epochs` in trainer state
|
||||
- [ ] Implement detection + injection in training loop
|
||||
- [ ] Apply adversarial multipliers to ExperienceCollectorConfig
|
||||
- [ ] Log when adversarial mode activates
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Anti-Correlation Reward Penalty [PLAN]
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu`
|
||||
|
||||
At epoch end, compare action distributions between training and validation. If cosine similarity > 0.8, the model does the same thing everywhere — it hasn't learned to differentiate. Penalize next epoch's rewards by 0.8x.
|
||||
|
||||
- [ ] Compute cosine similarity of train vs val action distributions
|
||||
- [ ] Add `reward_scale_override: f32` to ExperienceCollectorConfig
|
||||
- [ ] Multiply reward by scale factor in experience kernel
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Curiosity Prediction Error as Q-Penalty [PLAN]
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs`
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs`
|
||||
- Modify: `crates/ml/src/cuda_pipeline/mse_loss_kernel.cu` or `c51_loss_kernel.cu`
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
The curiosity forward model's prediction error measures how "novel" a state-action pair is. Wire it as an additive penalty to the Q-target in the Bellman equation:
|
||||
```
|
||||
target_Q = reward + gamma * (1-done) * target_Q_next - curiosity_penalty * prediction_error
|
||||
```
|
||||
High prediction error = novel state = conservative Q-values. Same principle as CQL but data-driven.
|
||||
|
||||
- [ ] Add config `curiosity_q_penalty_weight` (default 0.5)
|
||||
- [ ] Compute per-sample curiosity error in fused training step
|
||||
- [ ] Pass `curiosity_errors` buffer to loss kernel
|
||||
- [ ] Subtract penalty from Q-target in kernel
|
||||
- [ ] Wire through Rust launch code
|
||||
- [ ] TOML configs
|
||||
- [ ] Compile + full smoke test
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Counterfactual Experience Augmentation [PLAN]
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu`
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
|
||||
|
||||
For every trade taken, also compute the counterfactual: what would have happened with the mirror action (Long<->Short). Store both with inverted reward signs. Doubles effective data diversity.
|
||||
|
||||
- [ ] In experience kernel, compute reward for mirror action
|
||||
- [ ] Write both to output buffers (2x output size)
|
||||
- [ ] Counterfactual reward = -1 x actual reward
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Hindsight Regime Labeling — Feature Importance Filtering [PLAN]
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml/src/trainers/dqn/feature_importance.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu`
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
After each epoch, compute gradient x activation for each feature to identify which features predicted profitable trades. Mask the bottom 50% (set to 0) next epoch. Iterative: the model discovers which features carry signal and ignores noise.
|
||||
|
||||
- [ ] Compute per-feature importance from gradient x activation
|
||||
- [ ] Rank features, compute binary mask (top-K survive)
|
||||
- [ ] Apply mask in experience kernel (zero out masked features)
|
||||
- [ ] Iterate — recompute importance each epoch
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Regime-Adversarial Training (Gradient Reversal / DANN) [PLAN]
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml/src/cuda_pipeline/regime_discriminator_kernel.cu`
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs`
|
||||
|
||||
Train a small classifier head on DQN hidden features that predicts which time window (fold) data comes from. Apply GRADIENT REVERSAL — the DQN learns features that the discriminator CANNOT use to identify the regime. Domain Adversarial Neural Networks (DANN) adapted for temporal domains.
|
||||
|
||||
- [ ] Add discriminator kernel: small MLP [hidden_dim -> 64 -> num_folds]
|
||||
- [ ] Wire gradient reversal: negate discriminator gradient before shared trunk
|
||||
- [ ] Add fold ID to experience buffer
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
## Gems & Pearls — Category A: Symmetry & Invariance Exploits
|
||||
|
||||
### Task 10: Mirror Universe Training [PLAN]
|
||||
|
||||
**Phase:** 2 (CUDA changes) | **Difficulty:** Easy | **Impact:** High
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (return sign flip)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (mirror flag)
|
||||
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (epoch-level toggle)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Flip ALL returns (multiply by -1) and invert the action mapping (L100<->S100, L75<->S75, etc.). A model that truly understands market structure should produce identical P&L on mirror data with mirror actions. If IS Sharpe on mirrored data differs from normal data, the model has learned a directional bias, not structure. Train on both simultaneously — doubles data diversity with zero extra market data.
|
||||
|
||||
- [ ] Add config `enable_mirror_universe` (default true)
|
||||
- [ ] Add `mirror_active: bool` to ExperienceCollectorConfig
|
||||
- [ ] In experience kernel: when mirror=true, negate all return-based features and invert action index (8-action_idx for exposure)
|
||||
- [ ] Alternate: epoch N normal, epoch N+1 mirrored (or 50/50 within epoch)
|
||||
- [ ] Verify P&L symmetry on mirrored vs normal epochs
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 11: Time-Reversal Augmentation [PLAN]
|
||||
|
||||
**Phase:** 2 (CUDA changes) | **Difficulty:** Easy | **Impact:** Medium
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (reversed indexing)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (reverse flag)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Play episodes backwards. Market microstructure has statistical time-reversal symmetry at short horizons (bid-ask bounce). If the model can't trade profitably on reversed sequences, it's fitting to temporal artifacts (trends, momentum) rather than structural features. Mix 20% reversed episodes into each epoch — forces reliance on instantaneous features (spread, order flow) rather than trajectory.
|
||||
|
||||
- [ ] Add config `time_reversal_fraction` (default 0.2)
|
||||
- [ ] In experience kernel: reverse the bar indexing for selected episodes (bar[T-t] instead of bar[t])
|
||||
- [ ] Adjust next_close accordingly (now previous close)
|
||||
- [ ] Mark reversed episodes so reward calculation handles inverted time
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 12: Price-Level Invariance Enforcement [PLAN]
|
||||
|
||||
**Phase:** 2 (CUDA changes) | **Difficulty:** Medium | **Impact:** Very High
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (dual forward pass)
|
||||
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs` (invariance loss term)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/mse_loss_kernel.cu` (extra loss component)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Add a regularization loss: run the same features through the network twice — once at actual levels, once with all price-derived features shifted by a random constant. The Q-values MUST be identical. If they differ, the model has learned absolute price levels (which never generalize).
|
||||
|
||||
```
|
||||
L_invariance = MSE(Q(s), Q(s + delta))
|
||||
```
|
||||
|
||||
Added to the Bellman loss with configurable weight.
|
||||
|
||||
- [ ] Add config `price_invariance_weight` (default 0.1)
|
||||
- [ ] During training step, create shifted copy of feature batch (add random delta to price-derived feature indices)
|
||||
- [ ] Forward pass both original and shifted through Q-network
|
||||
- [ ] Compute MSE between Q-value outputs
|
||||
- [ ] Add weighted invariance loss to total loss
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 13: Volatility Regime Normalization [PLAN]
|
||||
|
||||
**Phase:** 1 (Rust-only) | **Difficulty:** Easy | **Impact:** Very High
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (feature preprocessing)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (normalization in kernel)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Before feeding features to the network, divide ALL return-based features by a 20-bar realized vol estimate. This makes the feature vector scale-invariant across regimes. A 1% move in a 5% vol regime should look identical to a 0.2% move in a 1% vol regime. This is preprocessing, not model change — but eliminates the #1 source of regime-specific memorization.
|
||||
|
||||
- [ ] Add config `enable_vol_normalization` (default true), `vol_lookback` (default 20)
|
||||
- [ ] Compute running realized vol from close-to-close returns in experience kernel
|
||||
- [ ] Divide return-based features (indices 0-7 approximately) by max(realized_vol, 1e-8)
|
||||
- [ ] Leave non-return features (ADX, CUSUM, volume ratios) unnormalized
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
## Gems & Pearls — Category B: Adversarial & Game-Theoretic
|
||||
|
||||
### Task 14: Market Maker Adversary (GAN for Trading) [PLAN]
|
||||
|
||||
**Phase:** 3 (Architecture) | **Difficulty:** Hard | **Impact:** Very High
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml/src/cuda_pipeline/adversary_kernel.cu`
|
||||
- Create: `crates/ml/src/trainers/dqn/market_maker_adversary.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Train a SECOND tiny network as the "adversarial market maker." It controls spread, fill probability, and slippage, and is trained to MAXIMIZE the DQN's losses. The DQN trains against this adversary. Cycle: 5 epochs adversarial training, 5 epochs recovery with frozen adversary. The DQN learns to survive worst-case execution — the ultimate generalization pressure.
|
||||
|
||||
- [ ] Add config `enable_market_maker_adversary` (default false), `adversary_cycle_epochs` (5)
|
||||
- [ ] Create adversary network: small MLP [state_dim -> 32 -> 3] outputting (spread_mult, fill_prob, slippage_mult)
|
||||
- [ ] Adversary loss = -1 * DQN reward (maximize DQN losses)
|
||||
- [ ] In training loop, alternate: adversary trains for N epochs, then freeze and let DQN recover for N epochs
|
||||
- [ ] Pass adversary outputs to ExperienceCollectorConfig
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 15: Gradient Vaccine [PLAN]
|
||||
|
||||
**Phase:** 2 (CUDA changes) | **Difficulty:** Medium | **Impact:** Very High
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs` (gradient projection)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (dual gradient computation)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
At each training step, compute gradients on BOTH the training batch and a held-out validation batch. Project out the component of the training gradient that CONTRADICTS the validation gradient:
|
||||
|
||||
```
|
||||
g_safe = g_train - (g_train . g_val / |g_val|^2) * g_val [when dot product < 0]
|
||||
```
|
||||
|
||||
Only update in directions that both sets agree on. The model literally cannot learn anything that doesn't generalize — overfitting gradients are surgically removed.
|
||||
|
||||
- [ ] Add config `enable_gradient_vaccine` (default true), `vaccine_val_fraction` (0.2)
|
||||
- [ ] Split each training batch: 80% train, 20% held-out validation
|
||||
- [ ] Compute gradients on both subsets separately
|
||||
- [ ] Compute dot product of gradient vectors
|
||||
- [ ] When dot product < 0, project out conflicting component
|
||||
- [ ] Apply safe gradient to optimizer
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 16: Phantom Liquidity Episodes [PLAN]
|
||||
|
||||
**Phase:** 3 (Architecture) | **Difficulty:** Medium | **Impact:** High
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml/src/trainers/dqn/synthetic_data.rs`
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Every 5th epoch, replace 30% of episodes with SYNTHETIC data — GBM with matched vol and drift, or shuffled 50-bar blocks of real data. The model must trade these profitably too. If it can only trade on real sequences but fails on statistically-identical synthetic ones, it has memorized specific price patterns. Ultimate memorization detector AND training regularizer.
|
||||
|
||||
- [ ] Add config `phantom_fraction` (0.3), `phantom_epoch_interval` (5)
|
||||
- [ ] Create GBM price generator: match vol, drift, and autocorrelation from real data
|
||||
- [ ] Alternative: block-shuffle real data (50-bar blocks, random permutation)
|
||||
- [ ] Replace selected episodes' price data with synthetic prices
|
||||
- [ ] Recompute features from synthetic prices
|
||||
- [ ] Track Sharpe on phantom vs real episodes separately (diagnostic)
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
## Gems & Pearls — Category C: Loss & Reward Reshaping
|
||||
|
||||
### Task 17: Counterfactual Regret Minimization Reward [PLAN]
|
||||
|
||||
**Phase:** 2 (CUDA changes) | **Difficulty:** Medium | **Impact:** High
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (regret computation)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Replace raw PnL reward with **regret**: `reward = PnL(action_taken) - max(PnL(all_actions))`. From game theory (CFR). The model learns to minimize regret rather than maximize absolute P&L. Regret is naturally normalized across regimes — a -$100 trade when the best was -$50 has regret -$50, same as a +$200 trade when the best was +$250. Eliminates regime-dependent reward scaling.
|
||||
|
||||
- [ ] Add config `enable_regret_reward` (default false), can coexist with normal reward
|
||||
- [ ] In experience kernel, after computing reward for taken action, compute reward for all 9 exposure levels
|
||||
- [ ] Regret = reward(taken) - max(reward(all_exposures))
|
||||
- [ ] Regret is always <= 0 (perfect play = 0 regret)
|
||||
- [ ] Option: blend regret with raw reward (0.5 * reward + 0.5 * regret)
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 18: Asymmetric Drawdown Loss [PLAN]
|
||||
|
||||
**Phase:** 1 (Rust-only) | **Difficulty:** Easy | **Impact:** High
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/mse_loss_kernel.cu` (asymmetric penalty)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/c51_loss_kernel.cu` (asymmetric penalty)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Add a term to the Bellman loss that specifically penalizes Q-value OVERESTIMATION on states where the portfolio is in drawdown:
|
||||
|
||||
```
|
||||
L_dd = max(0, Q_predicted - Q_target) * drawdown_depth^2
|
||||
```
|
||||
|
||||
When the model is in drawdown, overestimating Q-values is catastrophic (it holds losing positions expecting recovery). Makes the model pessimistic during drawdowns — the #1 OOS failure mode.
|
||||
|
||||
- [ ] Add config `asymmetric_dd_weight` (default 0.5)
|
||||
- [ ] Pass current drawdown depth to loss kernel as per-sample float
|
||||
- [ ] In kernel: when Q_pred > Q_target AND drawdown > 0, add penalty = overestimation * dd^2 * weight
|
||||
- [ ] Penalty only activates during drawdown (zero cost when equity is at highs)
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 19: Entropy-Regulated Position Trajectory [PLAN]
|
||||
|
||||
**Phase:** 2 (CUDA changes) | **Difficulty:** Easy | **Impact:** Medium
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (position histogram + entropy bonus)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Add an entropy bonus not to the ACTION distribution (standard) but to the POSITION TRAJECTORY over an episode. The model should visit multiple exposure levels, not get stuck at one. Compute the entropy of [time_at_S100, time_at_S75, ..., time_at_L100] and add to reward:
|
||||
|
||||
```
|
||||
reward += position_entropy_weight * H(position_histogram)
|
||||
```
|
||||
|
||||
Prevents degenerate strategies of "always flat" or "always long" that score well IS but collapse OOS.
|
||||
|
||||
- [ ] Add config `position_entropy_weight` (default 0.01)
|
||||
- [ ] Track per-episode position histogram in experience kernel (shared memory, 9 bins)
|
||||
- [ ] At episode end, compute entropy of histogram
|
||||
- [ ] Add scaled entropy bonus to final episode reward
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
## Gems & Pearls — Category D: Network Architecture Tricks
|
||||
|
||||
### Task 20: Weight Lottery Ticket Pruning [PLAN]
|
||||
|
||||
**Phase:** 3 (Architecture) | **Difficulty:** Medium | **Impact:** High
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml/src/trainers/dqn/pruning.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs` (apply mask)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (mask buffers)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
After 50 epochs of training, prune 70% of weights (smallest magnitude -> zero), then RETRAIN from scratch keeping only the surviving mask. The "lottery ticket hypothesis" says the sparse subnetwork generalizes better because it physically cannot memorize as much. Run twice: 100% -> 30% -> 9% of original parameters.
|
||||
|
||||
- [ ] Add config `pruning_epoch` (50), `pruning_fraction` (0.7), `pruning_rounds` (2)
|
||||
- [ ] After pruning_epoch, compute weight magnitude across all layers
|
||||
- [ ] Create binary mask: top 30% survive, bottom 70% = 0
|
||||
- [ ] Store mask as GPU buffer, apply element-wise after each optimizer step (w = w * mask)
|
||||
- [ ] Reset optimizer state for pruned weights
|
||||
- [ ] Optional: second round at epoch 100 (prune 70% of surviving 30% = 9% total)
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 21: Stochastic Depth (Layer Dropout) [PLAN]
|
||||
|
||||
**Phase:** 2 (CUDA changes) | **Difficulty:** Easy | **Impact:** Medium
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (layer skip logic)
|
||||
- Modify: `crates/ml-dqn/src/dqn.rs` (forward pass with skip)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
During training, randomly skip entire layers of the DQN trunk with 20% probability (replace with identity/passthrough). Each forward pass uses a random subset of the network's depth. Forces EVERY layer to produce useful features independently, not rely on deep compositional memorization. At inference, use all layers with scaled weights. Proven in vision transformers but nobody has applied it to RL trading.
|
||||
|
||||
- [ ] Add config `stochastic_depth_prob` (default 0.2)
|
||||
- [ ] During forward pass, for each hidden layer, generate random float
|
||||
- [ ] If random < prob, skip layer (output = input, bypass matmul + activation)
|
||||
- [ ] Scale surviving layers by 1/(1-prob) to maintain expected magnitude
|
||||
- [ ] At inference (eval mode), disable stochastic depth, use all layers
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 22: Feature-Wise Noise Injection [PLAN]
|
||||
|
||||
**Phase:** 1 (Rust-only) | **Difficulty:** Easy | **Impact:** Medium
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (noise injection)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Instead of zeroing features (dropout), add calibrated Gaussian noise to each feature with sigma proportional to that feature's standard deviation:
|
||||
|
||||
```
|
||||
feature_noisy = feature + N(0, noise_scale * std(feature))
|
||||
```
|
||||
|
||||
Strictly better than dropout for continuous features: the model sees signal plus noise, must learn to be robust to feature measurement error. Real market features ARE noisy.
|
||||
|
||||
- [ ] Add config `feature_noise_scale` (default 0.1)
|
||||
- [ ] Compute running std per feature (EMA or batch std)
|
||||
- [ ] In experience kernel, add N(0, scale * std) to each feature during training
|
||||
- [ ] Disable noise during evaluation/inference
|
||||
- [ ] Use curand for GPU-native noise generation
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
## Gems & Pearls — Category E: Temporal & Causal Structure
|
||||
|
||||
### Task 23: Causal Feature Masking (Random Feature Forests) [PLAN]
|
||||
|
||||
**Phase:** 1 (Rust-only) | **Difficulty:** Easy | **Impact:** High
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (feature masking)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (per-epoch mask generation)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Each epoch, randomly select 60% of features and mask the other 40% to zero. Different random subset each epoch. Over 100 epochs, the model trains on every possible feature combination. The surviving policy uses NO single feature as a crutch — it has built redundant representations. The "random forest" principle applied to RL feature extraction.
|
||||
|
||||
- [ ] Add config `feature_mask_fraction` (default 0.4), `enable_causal_masking` (default true)
|
||||
- [ ] At epoch start, generate random binary mask of length num_features
|
||||
- [ ] Upload mask to GPU
|
||||
- [ ] In experience kernel, multiply features by mask (zero out masked features)
|
||||
- [ ] Log which features are active each epoch
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 24: Drawdown-Conditional Learning Rate (Anti-Intuitive) [PLAN]
|
||||
|
||||
**Phase:** 1 (Rust-only) | **Difficulty:** Easy | **Impact:** Medium
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (LR scheduling)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (adam LR parameter)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
When the model is doing WELL (low drawdown, high Sharpe), INCREASE the learning rate 3x. When doing poorly, decrease it. The OPPOSITE of standard practice. Why: when the model is doing well IS, it's likely settling into an overfit minimum. High LR kicks it out. When struggling, low LR stabilizes. Creates natural oscillation that prevents settling into regime-specific policies.
|
||||
|
||||
- [ ] Add config `enable_anti_lr` (default true), `anti_lr_good_mult` (3.0), `anti_lr_bad_mult` (0.3), `anti_lr_sharpe_threshold` (0.3)
|
||||
- [ ] After epoch metrics, check if epoch Sharpe > threshold
|
||||
- [ ] If good (Sharpe > threshold): next epoch LR *= good_mult
|
||||
- [ ] If bad (Sharpe < -threshold): next epoch LR *= bad_mult
|
||||
- [ ] If neutral: use base LR
|
||||
- [ ] Clamp LR to [base_lr * 0.1, base_lr * 5.0] to prevent explosion
|
||||
- [ ] Update adam_lr parameter in GPU trainer
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 25: Trade Clustering Penalty [PLAN]
|
||||
|
||||
**Phase:** 1 (Rust-only) | **Difficulty:** Easy | **Impact:** Medium
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (inter-trade timing)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Compute the coefficient of variation of inter-trade intervals within each episode. If trades are temporally clustered (bursts then silence), penalize:
|
||||
|
||||
```
|
||||
penalty = CV(inter_trade_times) * clustering_penalty_weight
|
||||
```
|
||||
|
||||
Real generalizable strategies trade somewhat uniformly. Temporal clustering is a fingerprint of pattern-matching specific price sequences. Forces strategies that work across the ENTIRE episode.
|
||||
|
||||
- [ ] Add config `trade_clustering_penalty` (default 0.1)
|
||||
- [ ] Track trade timestamps in experience kernel (shared memory array)
|
||||
- [ ] At episode end, compute inter-trade intervals
|
||||
- [ ] Compute CV = std(intervals) / mean(intervals)
|
||||
- [ ] Subtract penalty * CV from episode reward
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 26: HER + Regime Tag Oversampling [PLAN]
|
||||
|
||||
**Phase:** 2 (CUDA changes) | **Difficulty:** Medium | **Impact:** High
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_her.rs` (regime-aware relabeling)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (regime tagging)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
Extend existing HER: when relabeling goals, also tag each experience with a coarse regime label (trending/mean-reverting/volatile, derived from ADX + realized vol). During replay, OVERSAMPLE experiences from underrepresented regimes. The model sees equal amounts of each regime regardless of real data distribution.
|
||||
|
||||
- [ ] Add config `enable_regime_oversampling` (default true), `num_regime_bins` (3)
|
||||
- [ ] In experience kernel, compute regime tag: ADX > 25 = trending, realized_vol > 2*median = volatile, else = mean-reverting
|
||||
- [ ] Store regime tag in experience buffer alongside existing fields
|
||||
- [ ] In PER sampling, weight by inverse regime frequency (rare regime = higher priority)
|
||||
- [ ] Log regime distribution per epoch
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
## Gems & Pearls — Category F: Meta-Learning & Selection
|
||||
|
||||
### Task 27: Ensemble Disagreement as Q-Penalty [PLAN]
|
||||
|
||||
**Phase:** 1 (Rust-only) | **Difficulty:** Easy | **Impact:** Very High
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (ensemble variance computation)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/mse_loss_kernel.cu` or `c51_loss_kernel.cu`
|
||||
- Modify: `crates/ml/src/trainers/dqn/fused_training.rs`
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
You have an ensemble of Q-networks. When ensemble members DISAGREE on Q-value (high variance across heads), that state-action pair is UNCERTAIN. Add ensemble disagreement as a PENALTY to the Q-target:
|
||||
|
||||
```
|
||||
Q_target -= beta * std(Q_head1, Q_head2, ..., Q_headN)
|
||||
```
|
||||
|
||||
High disagreement -> lower Q-target -> conservative behavior on uncertain states. Novel OOS states will have high disagreement -> conservative behavior -> fewer catastrophic losses. This is epistemic uncertainty penalization.
|
||||
|
||||
- [ ] Add config `ensemble_disagreement_weight` (default 0.3)
|
||||
- [ ] After ensemble forward pass, compute per-sample std across heads
|
||||
- [ ] Pass std buffer to loss kernel
|
||||
- [ ] Subtract weighted std from Q-target
|
||||
- [ ] Monitor average disagreement per epoch (diagnostic: should decrease during training)
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 28: Survival-of-the-Flattest Model Selection [PLAN]
|
||||
|
||||
**Phase:** 3 (Architecture) | **Difficulty:** Medium | **Impact:** High
|
||||
|
||||
**Files:**
|
||||
- Create: `crates/ml/src/trainers/dqn/sharpness.rs`
|
||||
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs` (selection criterion)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
During hyperopt, don't select the model with the best OOS Sharpe. Select the model whose loss landscape is FLATTEST — lowest |nabla^2 L| (Hessian trace). Approximate cheaply: perturb weights by epsilon in 10 random directions, measure loss change. Models in flat minima generalize; models in sharp minima memorize. Sharpness-Aware Minimization (SAM) applied to model SELECTION rather than training.
|
||||
|
||||
- [ ] Add config `sharpness_perturbation_epsilon` (0.01), `sharpness_num_directions` (10)
|
||||
- [ ] After training completes, compute sharpness score:
|
||||
- For each of 10 random unit vectors, perturb all weights by +/- epsilon * direction
|
||||
- Compute loss on validation set for each perturbation
|
||||
- Sharpness = mean(|loss_perturbed - loss_original|)
|
||||
- [ ] In hyperopt trial scoring: `score = oos_sharpe - sharpness_penalty_weight * sharpness`
|
||||
- [ ] Models with low sharpness (flat minima) are preferred even if Sharpe is slightly lower
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
### Task 29: Cross-Fold Action Consistency Score [PLAN]
|
||||
|
||||
**Phase:** 2 (CUDA changes) | **Difficulty:** Medium | **Impact:** Medium
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs` (cross-fold comparison)
|
||||
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs` (consistency scoring)
|
||||
- Modify: `crates/ml/src/trainers/dqn/config.rs`
|
||||
|
||||
During walk-forward, for each state that appears in multiple folds' validation sets (overlapping features), compare the actions the model would take. If the model takes DIFFERENT actions on the same market conditions in different folds, it hasn't generalized:
|
||||
|
||||
```
|
||||
consistency = 1 - mean(action_variance_across_folds)
|
||||
```
|
||||
|
||||
Use as SECONDARY selection criterion alongside Sharpe. A model with Sharpe 0.3 and consistency 0.9 beats Sharpe 0.5 and consistency 0.4.
|
||||
|
||||
- [ ] Add config `consistency_weight` (default 0.3)
|
||||
- [ ] During walk-forward, save action predictions for each fold's validation set
|
||||
- [ ] For overlapping states (same bar indices across folds), compute action agreement
|
||||
- [ ] Consistency = fraction of states where majority action matches
|
||||
- [ ] In hyperopt scoring: `score = oos_sharpe * (1 + consistency_weight * consistency)`
|
||||
- [ ] Compile + test + commit
|
||||
|
||||
---
|
||||
|
||||
## Complete Execution Order
|
||||
|
||||
### Phase 1 — Quick Wins, Rust-only (next session)
|
||||
| # | Technique | Status |
|
||||
|---|-----------|--------|
|
||||
| 1 | Domain Randomization | DONE |
|
||||
| 2 | CQL Alpha Boost | DONE |
|
||||
| 3 | Shrink-and-Perturb | DONE |
|
||||
| 4 | Adversarial Regime Injection | PLAN |
|
||||
| 13 | Volatility Regime Normalization | PLAN |
|
||||
| 18 | Asymmetric Drawdown Loss | PLAN |
|
||||
| 22 | Feature-Wise Noise Injection | PLAN |
|
||||
| 23 | Causal Feature Masking | PLAN |
|
||||
| 24 | Anti-Intuitive LR | PLAN |
|
||||
| 25 | Trade Clustering Penalty | PLAN |
|
||||
| 27 | Ensemble Disagreement Q-Penalty | PLAN |
|
||||
|
||||
### Phase 2 — CUDA Kernel Changes
|
||||
| # | Technique | Status |
|
||||
|---|-----------|--------|
|
||||
| 5 | Anti-Correlation Penalty | PLAN |
|
||||
| 6 | Curiosity Q-Penalty | PLAN |
|
||||
| 7 | Counterfactual Augmentation | PLAN |
|
||||
| 10 | Mirror Universe Training | PLAN |
|
||||
| 11 | Time-Reversal Augmentation | PLAN |
|
||||
| 12 | Price-Level Invariance | PLAN |
|
||||
| 15 | Gradient Vaccine | PLAN |
|
||||
| 17 | Counterfactual Regret Reward | PLAN |
|
||||
| 19 | Position Entropy | PLAN |
|
||||
| 21 | Stochastic Depth | PLAN |
|
||||
| 26 | HER + Regime Oversampling | PLAN |
|
||||
| 29 | Cross-Fold Consistency | PLAN |
|
||||
|
||||
### Phase 3 — Architecture Changes (after validation)
|
||||
| # | Technique | Status |
|
||||
|---|-----------|--------|
|
||||
| 8 | Feature Importance Filtering | PLAN |
|
||||
| 9 | Regime-Adversarial (DANN) | PLAN |
|
||||
| 14 | Market Maker Adversary | PLAN |
|
||||
| 16 | Phantom Liquidity Episodes | PLAN |
|
||||
| 20 | Lottery Ticket Pruning | PLAN |
|
||||
| 28 | Flattest Selection (SAM) | PLAN |
|
||||
|
||||
## Expected Combined Impact
|
||||
|
||||
| Category | Techniques | Gap Reduction |
|
||||
|----------|-----------|---------------|
|
||||
| Symmetry & Invariance (A) | 10-13 | 40-60% |
|
||||
| Adversarial & Game-Theoretic (B) | 14-16 | 30-50% |
|
||||
| Loss & Reward Reshaping (C) | 17-19 | 25-40% |
|
||||
| Network Architecture (D) | 20-22 | 20-35% |
|
||||
| Temporal & Causal (E) | 23-26 | 30-45% |
|
||||
| Meta-Learning & Selection (F) | 27-29 | 20-30% |
|
||||
| Previously Planned (1-9) | 1-9 | 50-80% |
|
||||
|
||||
**Combined target:** OOS Sharpe from -1.30 to > 0.0 (positive expectancy OOS).
|
||||
|
||||
---
|
||||
|
||||
## Critical: FP32 Experience Collector Weights
|
||||
|
||||
### Task 30: Experience Collector FP32 Weight Conversion [PLAN]
|
||||
|
||||
**Phase:** 1 (Critical) | **Difficulty:** Medium | **Impact:** Very High
|
||||
|
||||
**Files:**
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_experience_collector.rs` (change weight buffers from bf16 to f32)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/gpu_dqn_trainer.rs` (sync online weights as f32)
|
||||
- Modify: `crates/ml/src/cuda_pipeline/experience_kernels.cu` (read f32 weights instead of bf16)
|
||||
|
||||
The experience collector currently receives online Q-network weights in bf16 format for its
|
||||
forward passes during experience collection. This introduces quantization noise in Q-value
|
||||
estimates used for action selection, TD target computation, and priority calculation.
|
||||
Since the master weights in the Adam optimizer are already f32, the experience collector
|
||||
should use f32 copies to eliminate bf16 quantization artifacts during data collection.
|
||||
|
||||
- [ ] Change `online_params_flat` in GpuExperienceCollector from `CudaSlice<half::bf16>` to `CudaSlice<f32>`
|
||||
- [ ] Update weight sync to copy f32 master weights directly (skip bf16 conversion)
|
||||
- [ ] Update cuBLAS SGEMM calls to use f32 (already supports it, just change pointer types)
|
||||
- [ ] Update CUDA kernels to read f32 weights for forward pass during experience collection
|
||||
- [ ] Verify Q-value precision improvement in smoke tests
|
||||
- [ ] Compile + test + commit
|
||||
Reference in New Issue
Block a user