# Hyperopt FxCache Migration + Dead Code Removal — Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Migrate hyperopt to the zero-copy fxcache training path, then remove the old `Vec<(FeatureVector, Vec<f64>)>` training API entirely — eliminating ~500 lines of dead code.

**Architecture:** Hyperopt loads fxcache once, stores as `Arc<FxCacheData>`. Per trial: create DQNTrainer once (reuse across trials with reset_for_fold), call init_from_fxcache once, train_fold_from_slices per trial. After migration, delete: train_with_preloaded_data, train_with_shared_data, train_with_data_full_loop, features_to_trainer_format, DqnGpuData::upload (tuple variant), init_gpu_data, init_gpu_raw_buffers.

**Tech Stack:** Rust 1.85, cudarc 0.17.3

---

## Current Hyperopt Data Flow

```
1. preload_data() → load_training_data() → extract_ml_features() on CPU
2. Store as Arc<Vec<(FeatureVector, Vec<f64>)>> (4M heap allocs)
3. Per trial: DQNTrainer::new() → train_with_shared_data(Arc ref)
4. train_with_shared_data → train_with_data_full_loop (OLD path)
```

## Target Flow

```
1. preload_data() → load_fxcache() (470MB binary, no CPU feature extraction)
2. Store as Arc<FxCacheData> (contiguous arrays, zero per-bar alloc)
3. First trial: DQNTrainer::new() + init_from_fxcache() (one-time GPU upload)
4. Per trial: reset_for_fold() + train_fold_from_slices() (NEW path)
5. DELETE: train_with_preloaded_data, train_with_shared_data, train_with_data_full_loop
```

## File Structure

### Modified Files

| File | Changes |
|------|---------|
| `crates/ml/src/hyperopt/adapters/dqn.rs` | Load fxcache, store as Arc<FxCacheData>, use train_fold_from_slices per trial |
| `crates/ml/src/trainers/dqn/trainer/mod.rs` | Delete train_with_preloaded_data, train_with_shared_data |
| `crates/ml/src/trainers/dqn/trainer/training_loop.rs` | Delete train_with_data_full_loop, init_gpu_data, init_gpu_raw_buffers, collect_gpu_experiences, run_training_steps (OLD versions) |
| `crates/ml/src/cuda_pipeline/mod.rs` | Delete DqnGpuData::upload (tuple variant) |

### Dead Code to Remove After Migration

The following functions/methods become dead code once hyperopt uses the new path:

1. `DQNTrainer::train_with_preloaded_data` — accepts `Vec<(FeatureVector, Vec<f64>)>`
2. `DQNTrainer::train_with_shared_data` — accepts `Arc<Vec<(FeatureVector, Vec<f64>)>>`
3. `DQNTrainer::train_with_data_full_loop` — the old epoch loop
4. `DQNTrainer::init_gpu_data` — uploads via DqnGpuData::upload (tuple)
5. `DQNTrainer::init_gpu_raw_buffers` — flattens tuples then uploads
6. `DQNTrainer::collect_gpu_experiences` — old experience collection (takes training_data slice)
7. `DQNTrainer::run_training_steps` — old training loop (takes training_data slice)
8. `DqnGpuData::upload` — tuple-based upload
9. `features_to_trainer_format` / `features_to_trainer_format_fast` — convert features to tuples
10. `GpuBufferPool::upload_dqn` — tuple-based upload pool

---

### Task 1: Migrate hyperopt preload_data to fxcache

**Files:**
- Modify: `crates/ml/src/hyperopt/adapters/dqn.rs`

Replace `preloaded_training_data: Option<Arc<Vec<(FeatureVector, Vec<f64>)>>>` with fxcache data.

- [ ] **Step 1: Change preloaded data fields**

Replace:
```rust
preloaded_training_data: Option<Arc<Vec<(FeatureVector, Vec<f64>)>>>,
preloaded_val_data: Option<Arc<Vec<(FeatureVector, Vec<f64>)>>>,
```
With:
```rust
preloaded_fxcache: Option<Arc<fxcache::FxCacheData>>,
preloaded_train_end: usize,  // index where training ends, validation starts
```

- [ ] **Step 2: Rewrite preload_data to load fxcache**

- [ ] **Step 3: Rewrite evaluate_candidate to use train_fold_from_slices**

- [ ] **Step 4: Compile and test**

- [ ] **Step 5: Commit**

---

### Task 2: Delete old training API (dead code removal)

**Files:**
- Modify: `crates/ml/src/trainers/dqn/trainer/mod.rs`
- Modify: `crates/ml/src/trainers/dqn/trainer/training_loop.rs`
- Modify: `crates/ml/src/cuda_pipeline/mod.rs`

- [ ] **Step 1: Delete train_with_preloaded_data**
- [ ] **Step 2: Delete train_with_shared_data**  
- [ ] **Step 3: Delete train_with_data_full_loop**
- [ ] **Step 4: Delete init_gpu_data**
- [ ] **Step 5: Delete init_gpu_raw_buffers**
- [ ] **Step 6: Delete collect_gpu_experiences (old)**
- [ ] **Step 7: Delete run_training_steps (old)**
- [ ] **Step 8: Delete DqnGpuData::upload (tuple variant)**
- [ ] **Step 9: Delete GpuBufferPool::upload_dqn**
- [ ] **Step 10: Compile and fix any remaining references**
- [ ] **Step 11: Commit**

---

### Task 3: Update smoketests to use only the new path

**Files:**
- Modify: `crates/ml/src/trainers/dqn/smoke_tests/training_stability.rs`

The old smoketest (`test_gpu_collector_auto_initializes`) uses `trainer.train(&data_dir, ...)` which calls the old path. Replace with the fxcache path.

- [ ] **Step 1: Replace old smoketest to use fxcache path**
- [ ] **Step 2: Delete test_gpu_collector_auto_initializes (replaced by test_fxcache_zero_copy_training)**
- [ ] **Step 3: Compile and test**
- [ ] **Step 4: Commit**

---

### Task 4: Validate

- [ ] **Step 1: Run both smoketests locally**
- [ ] **Step 2: Run hyperopt smoke trial**
- [ ] **Step 3: Full workspace compile**
- [ ] **Step 4: Commit and push**