foxhunt/docs/plans/2026-03-03-ppo-improvements-design.md

PPO Improvements Design — All Tiers

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Comprehensive PPO improvements — 13D search space, symlog critic, DAPO clipping, adaptive entropy, percentile scaling, curiosity port, reward shaping, ExO-PPO trajectory replay, composite risk-adjusted reward.

Architecture: Modular additions to existing PPO pipeline. Each improvement is independent and tested separately. New modules are created for symlog, adaptive entropy, percentile scaler, trajectory replay buffer, and composite reward. Existing PPOConfig, PPOParams, and training loop are extended.

Tech Stack: Candle v0.9.1 (Rust), existing PPO modules in crates/ml/src/ppo/, hyperopt adapter in crates/ml/src/hyperopt/adapters/ppo.rs

---

Phase 1: Search Space Expansion (7D → 13D)

Expand PPOParams from 7 to 13 dimensions. New params: gae_gamma, gae_lambda, mini_batch_size, max_grad_norm, max_position_absolute, clip_epsilon_high.

Files: crates/ml/src/hyperopt/adapters/ppo.rs

Phase 2: Symlog Value Predictions

Add symlog(x) = sign(x) * ln(|x| + 1) and symexp(x) = sign(x) * (exp(|x|) - 1) transforms. Apply symlog to return targets in compute_value_loss(). Apply symexp when recovering values for advantage computation.

Files: New crates/ml/src/ppo/symlog.rs, modify crates/ml/src/ppo/ppo.rs (compute_value_loss), modify crates/ml/src/ppo/gae.rs (compute_gae to output symlog-compatible returns)

Phase 3: DAPO Asymmetric Clipping

Default clip_epsilon_high = Some(0.28) in PPOConfig. Already implemented in compute_policy_loss — just set the default.

Files: crates/ml/src/ppo/ppo.rs (PPOConfig::default)

Phase 4: Adaptive Entropy Coefficient

Learnable log(alpha) parameter auto-tuned via dual gradient descent. Target entropy = -0.5 * ln(num_actions). Extra optimizer step per batch: alpha_loss = -alpha * (log_pi + target_entropy).mean().

Files: New crates/ml/src/ppo/adaptive_entropy.rs, modify crates/ml/src/ppo/ppo.rs (PPO struct + training loop)

Phase 5: Percentile Advantage Scaling

Track running P5/P95 of returns with EMA decay (0.99). Scale advantages by 1/(P95-P5) instead of std. Robust to heavy-tailed financial returns.

Files: New crates/ml/src/ppo/percentile_scaler.rs, modify crates/ml/src/ppo/ppo.rs or gae.rs (advantage normalization)

Phase 6: Curiosity Module Port

Port dqn/curiosity.rs CuriosityModule to PPO. It's already agent-agnostic (uses FactoredAction). Wire intrinsic reward into PPO trajectory collection. Add curiosity_weight to PPOConfig and hyperopt.

Files: Modify crates/ml/src/hyperopt/adapters/ppo.rs (wire curiosity), modify crates/ml/src/ppo/ppo.rs (PPOConfig + optional CuriosityModule)

Phase 7: Reward Shaping Port

Port DQN reward components to PPO: hold penalty (discourages inactivity), rolling Sharpe (risk-adjusted), diversity penalty (smooth quadratic from DQN fix).

Files: New crates/ml/src/ppo/reward_shaping.rs, wire into hyperopt adapter

Phase 8: ExO-PPO Trajectory Replay

FIFO buffer holding M=4 past rollouts. Importance-weighted updates with exponential attenuation outside clip bounds. 4x sample efficiency.

Files: New crates/ml/src/ppo/trajectory_replay.rs, modify PPO training loop

Phase 9: Composite Risk-Adjusted Reward

Multi-component reward: R = w1*return - w2*downside_dev + w3*differential_return. Components computed from rolling windows in PortfolioTracker.

Files: New crates/ml/src/ppo/composite_reward.rs, wire into hyperopt adapter

Phase 10: CUDA Fix + Validation

Commit PPO CUDA cleanup (already coded), run full test suite, verify 0 clippy.

---

Implementation Order

Phases 1-5 are independent (parallel). Phases 6-7 depend on Phase 1 (new hyperopt params). Phase 8 is independent. Phase 9 depends on Phase 7. Phase 10 is final.