ba8fb8eedd
Integrate all PPO improvement modules into the core training paths: - Symlog value predictions in compute_value_loss (MLP + LSTM) - Adaptive entropy auto-tuning replaces fixed entropy_coeff - Percentile P5/P95 advantage scaling for heavy-tailed returns - DAPO clip_epsilon_high wired in all 7 PPOConfig construction sites - Shape mismatch fix in adaptive_entropy (unsqueeze scalar) 2726 tests pass, 0 clippy errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
76 lines
3.8 KiB
Markdown
76 lines
3.8 KiB
Markdown
# PPO Improvements Design — All Tiers
|
|
|
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
|
|
|
**Goal:** Comprehensive PPO improvements — 13D search space, symlog critic, DAPO clipping, adaptive entropy, percentile scaling, curiosity port, reward shaping, ExO-PPO trajectory replay, composite risk-adjusted reward.
|
|
|
|
**Architecture:** Modular additions to existing PPO pipeline. Each improvement is independent and tested separately. New modules are created for symlog, adaptive entropy, percentile scaler, trajectory replay buffer, and composite reward. Existing PPOConfig, PPOParams, and training loop are extended.
|
|
|
|
**Tech Stack:** Candle v0.9.1 (Rust), existing PPO modules in `crates/ml/src/ppo/`, hyperopt adapter in `crates/ml/src/hyperopt/adapters/ppo.rs`
|
|
|
|
---
|
|
|
|
## Phase 1: Search Space Expansion (7D → 13D)
|
|
|
|
Expand PPOParams from 7 to 13 dimensions. New params: gae_gamma, gae_lambda, mini_batch_size, max_grad_norm, max_position_absolute, clip_epsilon_high.
|
|
|
|
**Files:** `crates/ml/src/hyperopt/adapters/ppo.rs`
|
|
|
|
## Phase 2: Symlog Value Predictions
|
|
|
|
Add `symlog(x) = sign(x) * ln(|x| + 1)` and `symexp(x) = sign(x) * (exp(|x|) - 1)` transforms. Apply symlog to return targets in compute_value_loss(). Apply symexp when recovering values for advantage computation.
|
|
|
|
**Files:** New `crates/ml/src/ppo/symlog.rs`, modify `crates/ml/src/ppo/ppo.rs` (compute_value_loss), modify `crates/ml/src/ppo/gae.rs` (compute_gae to output symlog-compatible returns)
|
|
|
|
## Phase 3: DAPO Asymmetric Clipping
|
|
|
|
Default `clip_epsilon_high = Some(0.28)` in PPOConfig. Already implemented in compute_policy_loss — just set the default.
|
|
|
|
**Files:** `crates/ml/src/ppo/ppo.rs` (PPOConfig::default)
|
|
|
|
## Phase 4: Adaptive Entropy Coefficient
|
|
|
|
Learnable `log(alpha)` parameter auto-tuned via dual gradient descent. Target entropy = -0.5 * ln(num_actions). Extra optimizer step per batch: `alpha_loss = -alpha * (log_pi + target_entropy).mean()`.
|
|
|
|
**Files:** New `crates/ml/src/ppo/adaptive_entropy.rs`, modify `crates/ml/src/ppo/ppo.rs` (PPO struct + training loop)
|
|
|
|
## Phase 5: Percentile Advantage Scaling
|
|
|
|
Track running P5/P95 of returns with EMA decay (0.99). Scale advantages by 1/(P95-P5) instead of std. Robust to heavy-tailed financial returns.
|
|
|
|
**Files:** New `crates/ml/src/ppo/percentile_scaler.rs`, modify `crates/ml/src/ppo/ppo.rs` or `gae.rs` (advantage normalization)
|
|
|
|
## Phase 6: Curiosity Module Port
|
|
|
|
Port `dqn/curiosity.rs` CuriosityModule to PPO. It's already agent-agnostic (uses FactoredAction). Wire intrinsic reward into PPO trajectory collection. Add `curiosity_weight` to PPOConfig and hyperopt.
|
|
|
|
**Files:** Modify `crates/ml/src/hyperopt/adapters/ppo.rs` (wire curiosity), modify `crates/ml/src/ppo/ppo.rs` (PPOConfig + optional CuriosityModule)
|
|
|
|
## Phase 7: Reward Shaping Port
|
|
|
|
Port DQN reward components to PPO: hold penalty (discourages inactivity), rolling Sharpe (risk-adjusted), diversity penalty (smooth quadratic from DQN fix).
|
|
|
|
**Files:** New `crates/ml/src/ppo/reward_shaping.rs`, wire into hyperopt adapter
|
|
|
|
## Phase 8: ExO-PPO Trajectory Replay
|
|
|
|
FIFO buffer holding M=4 past rollouts. Importance-weighted updates with exponential attenuation outside clip bounds. 4x sample efficiency.
|
|
|
|
**Files:** New `crates/ml/src/ppo/trajectory_replay.rs`, modify PPO training loop
|
|
|
|
## Phase 9: Composite Risk-Adjusted Reward
|
|
|
|
Multi-component reward: `R = w1*return - w2*downside_dev + w3*differential_return`. Components computed from rolling windows in PortfolioTracker.
|
|
|
|
**Files:** New `crates/ml/src/ppo/composite_reward.rs`, wire into hyperopt adapter
|
|
|
|
## Phase 10: CUDA Fix + Validation
|
|
|
|
Commit PPO CUDA cleanup (already coded), run full test suite, verify 0 clippy.
|
|
|
|
---
|
|
|
|
## Implementation Order
|
|
|
|
Phases 1-5 are independent (parallel). Phases 6-7 depend on Phase 1 (new hyperopt params). Phase 8 is independent. Phase 9 depends on Phase 7. Phase 10 is final.
|