Per spec §7 (v3 architecture reset, promoted from v2 Phase B into CRT.1).
Single-cache-line layout. Every reader and writer of open_trade_state
migrates in this commit per feedback_no_partial_refactor.
New 64-byte layout (existing 24-byte fields preserved at original offsets):
Offset Size Field
0 8 entry_ts_ns (u64)
8 4 entry_px_x100 (i32)
12 4 entry_size (i32)
16 4 realized_at_open (f32)
20 4 conviction_at_entry (f32) ← NEW
24 4 conviction_ema (f32) ← NEW
28 16 conviction_per_horizon_ema [4 × f32] ← NEW
44 4 pnl_adjusted_conviction_ema (f32) ← NEW
48 4 peak_unrealized_pnl (f32) ← NEW
52 4 degradation_consecutive_events (u32) ← NEW
56 4 disagreement_consecutive_events (u32) ← NEW
60 1 horizon_idx_dominant_at_entry (u8) ← NEW
61 3 pad
The 24-byte prefix is unchanged so downstream readers (stop_check_isv in
decision_policy.cu, max_hold event-rate check in resting_orders.cu) need
only update the stride constant. The new fields at offsets 20..63 are
populated by subsequent CRT.1 tasks (C1.2 multi-horizon conviction, C1.4
composite exit signal).
Open branch in pnl_track.cu zeros the new fields implicitly because
alloc_zeros on the device slot zero-initialises everything; subsequent
writes only touch the 0-23 byte range (existing fields). Close branch
reset-loop iterates OPEN_TRADE_STATE_BYTES which now zeros all 64.
Files changed:
crates/ml-backtesting/src/lob/mod.rs — pub const OPEN_TRADE_STATE_BYTES: usize = 64
crates/ml-backtesting/cuda/pnl_track.cu — #define OPEN_TRADE_STATE_BYTES 64
crates/ml-backtesting/cuda/resting_orders.cu — comment + stride literal 24→64
crates/ml-backtesting/tests/stop_controller.rs — open_trade_state_64_byte_layout test
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build a new alpha-native RL trainer in `ml-alpha` that replaces the per-bar discrete-action DQN smoke approach (`alpha_baseline`) with a PPO actor-critic over multi-horizon alpha heads on a frozen Mamba2 perception trunk. Designed from the ground up for **multi-bar, multi-horizon trading** with continuous position sizing, decision-stride as a first-class concept, and IBKR-realistic passive-execution economics.
**Architecture (3-layer):**
1.**Perception** — Mamba2 trunk + multi-horizon alpha heads predicting `P(up | horizon=h)` for `h ∈ {30, 100, 1000, 6000}` bars. Trained supervised on labelled fxcache data. Frozen after pretrain.
2.**Decision policy** — small PPO actor-critic. Inputs: multi-horizon alpha probs + market state + position state. Outputs: direction categorical, hold-horizon categorical, position-size Gaussian, value scalar.
3.**Risk / execution** — Kelly-cap position sizing, max-drawdown gate, passive-limit posting via existing `ExecutionEnv`.
**Tech stack:** Rust 1.85+, cudarc CUDA, cuBLAS, Mamba2 SSM (reuse from `ml-alpha::mamba2_block`), PPO actor-critic on top, GAE for advantage estimation, multi-horizon supervised loss for perception.
**Rationale for PPO over DQN:** The existing fused-CUDA DQN trainer (`gpu_dqn_trainer.rs`, 1.9 MB) is sophisticated but was engineered for a different problem regime — per-bar discrete actions with distributional Q and synthetic reward shaping. We have no validation that this fits minute-horizon alpha with continuous position sizing. PPO addresses several DQN failure modes empirically observed today: bootstrapping noise at per-bar level (today's coin-flip problem), no native continuous action space (we want continuous position size), exploration via ε-greedy/Thompson clunky in trading. PPO's GAE + policy-gradient avoids these by design.
---
## File Structure
Following Path B (new binary in `ml-alpha`, library dep on `ml` only for kernels we genuinely need to share):
```
crates/ml-alpha/
├── src/
│ ├── lib.rs (modify: expose new modules)
│ ├── mamba2_block.rs (existing, reuse)
│ ├── fxcache_reader.rs (existing, reuse)
│ ├── multi_horizon_labels.rs (existing, extend to multi-h labels)
## Phase 0: Pre-work (current 9-fold result + plan baseline)
- [ ]**Step 1: Wait for the running `alpha-cv-6jlm5` workflow to complete.** Need the 9-fold stride=200 + scaled-training result on full 9Q before committing to a baseline. Captures the **target Sharpe to beat** with the new PPO trainer.
Run: `argo get alpha-cv-6jlm5 -n foxhunt` until status = Succeeded.
Then: aggregate the 9 fold JSONs into a single summary table.
Expected: mean Sharpe at quarter-tick cost, std across 9 folds, break-even cost.
- [ ]**Step 2: Record baseline numbers in a memory file**
(Boilerplate: copy from `alpha_train_stacker.rs`, swap out single-horizon loss for multi-horizon BCE, persist `perception_weights.bin` + `multi_horizon_logits_cache.bin`)
- [ ]**Step 1: Scaffold the CLI (steal from `alpha_train_stacker.rs`)**
The env wraps `ml::env::execution_env::ExecutionEnv` and exposes a stride-aware decision interface: `decide_at(bar_idx)` returns `Some(state)` only at decision points; calling `step(action)` advances `decision_stride` bars at once and returns the segment reward.
- [ ]**Step 1: Test that stride correctly gates action emission**
Fused MLP forward: trunk → 3 logit heads + value. Single kernel launch per minibatch.
- [ ]**Step 1: Write CUDA C source**
- [ ]**Step 2: Add NVRTC build hook in `build.rs`**
- [ ]**Step 3: Wire into `Policy::forward_gpu` (replaces the CPU forward)**
- [ ]**Step 4: Verify bit-equivalence against CPU forward on small batch**
### Task 4.2: PPO loss + backward kernel
- [ ] Fused kernel: given (new_log_probs, old_log_probs, advantages, returns, values, entropies) compute total loss + per-parameter gradients in one pass.
- [ ] Verify bit-equivalence on small batch.
### Task 4.3: GAE on GPU
- [ ] Sequential dependency makes parallelism hard; standard approach is reverse-prefix-scan. Single block thread-cooperative scan since rollout sizes are small (<10K).
---
## Phase 5: Walk-forward CV + cluster integration
### Task 5.1: Native walk-forward driver
**Files:**
- Create: `crates/ml-alpha/src/walk_forward.rs`
Iterate over fold offsets, train PPO from scratch on each fold, backtest on held-out portion.
git commit -m "infra(argo): alpha-ppo workflow for PPO 9-fold training"
git push
```
- [ ]**Step 3: Submit + monitor**
```bash
./scripts/argo-alpha-ppo.sh --branch main
```
Expected wall: ~90-120 min on L40S (perception 15 min + 9 folds × 10-12 min).
---
## Phase 6: Validation gate
Before declaring the new trainer ready, must clear the **DQN 9-fold baseline** captured in Phase 0 by a meaningful margin.
Acceptance criteria:
- [ ] Mean Sharpe at quarter-tick cost ≥ baseline + 0.5 (one std).
- [ ] Standard deviation across 9 folds ≤ baseline std (no regime overfitting).
- [ ] Break-even cost ≥ baseline (no economic regression).
- [ ] No fold worse than baseline worst fold by more than 1 Sharpe (robustness check).
If criteria are not met:
- [ ] Investigate failure modes. Common: PPO entropy collapse (policy gets too confident → no exploration), value-function lag (advantage misestimation), perception over/under-fit. Each has a standard remediation.
If criteria are met:
- [ ] Update memory `project_alpha_ppo_baseline.md` with the new bar.
- [ ] Document the architectural choice in a `pearl_*.md` entry (memory writes happen via Write tool with the auto-memory format).
---
## Self-Review
**Spec coverage:** Plan covers perception (multi-horizon labels + heads + pretraining binary), policy network (3-head categorical + value), env (decision-stride + segment reward), PPO loss + GAE, training loop, GPU acceleration, walk-forward, cluster workflow, validation gate. The user's three answered design decisions (Path B / branched-as-categorical / multi-horizon blend / hybrid 55-dim state) are all baked in.
**Placeholder scan:** Three `todo!()` markers in code snippets are illustrative; they get replaced when the task is executed (the body of `alpha_train_perception::main` follows the alpha_train_stacker template). No "TBD" / "later" markers.
**Type consistency:**`AlphaState`, `PolicyAction`, `Direction`, `Horizon`, `SizeBin`, `PolicyOutput`, `PpoLossResult`, `MultiHorizonLabels`, `MultiHorizonHead`, `AlphaExecutionEnv` defined once, referenced consistently. State dimension 55 = 42 + 4 + 6 + 3 verified in `alpha_state.rs` test.
**Ambiguity callout:** The size space (8 discrete bins vs Gaussian continuous) is a soft decision in the plan. Phase 3 implements discrete bins for PPO simplicity; if size-resolution matters empirically, swap to Gaussian (10–20 LOC change in `policy.rs`). Noted in Task 2.3.
**Total scope:** ~3 weeks of focused work for Phases 1–5; Phase 6 is the validation gate. Phase 0 baseline capture is blocking on the current `alpha-cv-6jlm5` cluster run.
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.