# PPO Improvements Design — All Tiers

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Comprehensive PPO improvements — 13D search space, symlog critic, DAPO clipping, adaptive entropy, percentile scaling, curiosity port, reward shaping, ExO-PPO trajectory replay, composite risk-adjusted reward.

**Architecture:** Modular additions to existing PPO pipeline. Each improvement is independent and tested separately. New modules are created for symlog, adaptive entropy, percentile scaler, trajectory replay buffer, and composite reward. Existing PPOConfig, PPOParams, and training loop are extended.

**Tech Stack:** Candle v0.9.1 (Rust), existing PPO modules in `crates/ml/src/ppo/`, hyperopt adapter in `crates/ml/src/hyperopt/adapters/ppo.rs`

---

## Phase 1: Search Space Expansion (7D → 13D)

Expand PPOParams from 7 to 13 dimensions. New params: gae_gamma, gae_lambda, mini_batch_size, max_grad_norm, max_position_absolute, clip_epsilon_high.

**Files:** `crates/ml/src/hyperopt/adapters/ppo.rs`

## Phase 2: Symlog Value Predictions

Add `symlog(x) = sign(x) * ln(|x| + 1)` and `symexp(x) = sign(x) * (exp(|x|) - 1)` transforms. Apply symlog to return targets in compute_value_loss(). Apply symexp when recovering values for advantage computation.

**Files:** New `crates/ml/src/ppo/symlog.rs`, modify `crates/ml/src/ppo/ppo.rs` (compute_value_loss), modify `crates/ml/src/ppo/gae.rs` (compute_gae to output symlog-compatible returns)

## Phase 3: DAPO Asymmetric Clipping

Default `clip_epsilon_high = Some(0.28)` in PPOConfig. Already implemented in compute_policy_loss — just set the default.

**Files:** `crates/ml/src/ppo/ppo.rs` (PPOConfig::default)

## Phase 4: Adaptive Entropy Coefficient

Learnable `log(alpha)` parameter auto-tuned via dual gradient descent. Target entropy = -0.5 * ln(num_actions). Extra optimizer step per batch: `alpha_loss = -alpha * (log_pi + target_entropy).mean()`.

**Files:** New `crates/ml/src/ppo/adaptive_entropy.rs`, modify `crates/ml/src/ppo/ppo.rs` (PPO struct + training loop)

## Phase 5: Percentile Advantage Scaling

Track running P5/P95 of returns with EMA decay (0.99). Scale advantages by 1/(P95-P5) instead of std. Robust to heavy-tailed financial returns.

**Files:** New `crates/ml/src/ppo/percentile_scaler.rs`, modify `crates/ml/src/ppo/ppo.rs` or `gae.rs` (advantage normalization)

## Phase 6: Curiosity Module Port

Port `dqn/curiosity.rs` CuriosityModule to PPO. It's already agent-agnostic (uses FactoredAction). Wire intrinsic reward into PPO trajectory collection. Add `curiosity_weight` to PPOConfig and hyperopt.

**Files:** Modify `crates/ml/src/hyperopt/adapters/ppo.rs` (wire curiosity), modify `crates/ml/src/ppo/ppo.rs` (PPOConfig + optional CuriosityModule)

## Phase 7: Reward Shaping Port

Port DQN reward components to PPO: hold penalty (discourages inactivity), rolling Sharpe (risk-adjusted), diversity penalty (smooth quadratic from DQN fix).

**Files:** New `crates/ml/src/ppo/reward_shaping.rs`, wire into hyperopt adapter

## Phase 8: ExO-PPO Trajectory Replay

FIFO buffer holding M=4 past rollouts. Importance-weighted updates with exponential attenuation outside clip bounds. 4x sample efficiency.

**Files:** New `crates/ml/src/ppo/trajectory_replay.rs`, modify PPO training loop

## Phase 9: Composite Risk-Adjusted Reward

Multi-component reward: `R = w1*return - w2*downside_dev + w3*differential_return`. Components computed from rolling windows in PortfolioTracker.

**Files:** New `crates/ml/src/ppo/composite_reward.rs`, wire into hyperopt adapter

## Phase 10: CUDA Fix + Validation

Commit PPO CUDA cleanup (already coded), run full test suite, verify 0 clippy.

---

## Implementation Order

Phases 1-5 are independent (parallel). Phases 6-7 depend on Phase 1 (new hyperopt params). Phase 8 is independent. Phase 9 depends on Phase 7. Phase 10 is final.