foxhunt/docs/superpowers/plans/2026-04-03-hyperopt-fxcache-migration.md

Hyperopt FxCache Migration + Dead Code Removal — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Migrate hyperopt to the zero-copy fxcache training path, then remove the old Vec<(FeatureVector, Vec<f64>)> training API entirely — eliminating ~500 lines of dead code.

Architecture: Hyperopt loads fxcache once, stores as Arc<FxCacheData>. Per trial: create DQNTrainer once (reuse across trials with reset_for_fold), call init_from_fxcache once, train_fold_from_slices per trial. After migration, delete: train_with_preloaded_data, train_with_shared_data, train_with_data_full_loop, features_to_trainer_format, DqnGpuData::upload (tuple variant), init_gpu_data, init_gpu_raw_buffers.

Tech Stack: Rust 1.85, cudarc 0.17.3

---

Current Hyperopt Data Flow

1. preload_data() → load_training_data() → extract_ml_features() on CPU
2. Store as Arc<Vec<(FeatureVector, Vec<f64>)>> (4M heap allocs)
3. Per trial: DQNTrainer::new() → train_with_shared_data(Arc ref)
4. train_with_shared_data → train_with_data_full_loop (OLD path)

Target Flow

1. preload_data() → load_fxcache() (470MB binary, no CPU feature extraction)
2. Store as Arc<FxCacheData> (contiguous arrays, zero per-bar alloc)
3. First trial: DQNTrainer::new() + init_from_fxcache() (one-time GPU upload)
4. Per trial: reset_for_fold() + train_fold_from_slices() (NEW path)
5. DELETE: train_with_preloaded_data, train_with_shared_data, train_with_data_full_loop

File Structure

Modified Files

File	Changes
crates/ml/src/hyperopt/adapters/dqn.rs	Load fxcache, store as Arc, use train_fold_from_slices per trial
crates/ml/src/trainers/dqn/trainer/mod.rs	Delete train_with_preloaded_data, train_with_shared_data
crates/ml/src/trainers/dqn/trainer/training_loop.rs	Delete train_with_data_full_loop, init_gpu_data, init_gpu_raw_buffers, collect_gpu_experiences, run_training_steps (OLD versions)
crates/ml/src/cuda_pipeline/mod.rs	Delete DqnGpuData::upload (tuple variant)

Dead Code to Remove After Migration

The following functions/methods become dead code once hyperopt uses the new path:

1. DQNTrainer::train_with_preloaded_data — accepts Vec<(FeatureVector, Vec<f64>)>
2. DQNTrainer::train_with_shared_data — accepts Arc<Vec<(FeatureVector, Vec<f64>)>>
3. DQNTrainer::train_with_data_full_loop — the old epoch loop
4. DQNTrainer::init_gpu_data — uploads via DqnGpuData::upload (tuple)
5. DQNTrainer::init_gpu_raw_buffers — flattens tuples then uploads
6. DQNTrainer::collect_gpu_experiences — old experience collection (takes training_data slice)
7. DQNTrainer::run_training_steps — old training loop (takes training_data slice)
8. DqnGpuData::upload — tuple-based upload
9. features_to_trainer_format / features_to_trainer_format_fast — convert features to tuples
10. GpuBufferPool::upload_dqn — tuple-based upload pool

---

Task 1: Migrate hyperopt preload_data to fxcache

Files:

• Modify: crates/ml/src/hyperopt/adapters/dqn.rs

Replace preloaded_training_data: Option<Arc<Vec<(FeatureVector, Vec<f64>)>>> with fxcache data.

• Step 1: Change preloaded data fields

Replace:

preloaded_training_data: Option<Arc<Vec<(FeatureVector, Vec<f64>)>>>,
preloaded_val_data: Option<Arc<Vec<(FeatureVector, Vec<f64>)>>>,

With:

preloaded_fxcache: Option<Arc<fxcache::FxCacheData>>,
preloaded_train_end: usize,  // index where training ends, validation starts

• Step 2: Rewrite preload_data to load fxcache

• Step 3: Rewrite evaluate_candidate to use train_fold_from_slices

• Step 4: Compile and test

• Step 5: Commit

---

Task 2: Delete old training API (dead code removal)

Files:

• Modify: crates/ml/src/trainers/dqn/trainer/mod.rs

• Modify: crates/ml/src/trainers/dqn/trainer/training_loop.rs

• Modify: crates/ml/src/cuda_pipeline/mod.rs

• Step 1: Delete train_with_preloaded_data

• Step 2: Delete train_with_shared_data

• Step 3: Delete train_with_data_full_loop

• Step 4: Delete init_gpu_data

• Step 5: Delete init_gpu_raw_buffers

• Step 6: Delete collect_gpu_experiences (old)

• Step 7: Delete run_training_steps (old)

• Step 8: Delete DqnGpuData::upload (tuple variant)

• Step 9: Delete GpuBufferPool::upload_dqn

• Step 10: Compile and fix any remaining references

• Step 11: Commit

---

Task 3: Update smoketests to use only the new path

Files:

• Modify: crates/ml/src/trainers/dqn/smoke_tests/training_stability.rs

The old smoketest (test_gpu_collector_auto_initializes) uses trainer.train(&data_dir, ...) which calls the old path. Replace with the fxcache path.

• Step 1: Replace old smoketest to use fxcache path
• Step 2: Delete test_gpu_collector_auto_initializes (replaced by test_fxcache_zero_copy_training)
• Step 3: Compile and test
• Step 4: Commit

---

Task 4: Validate

• Step 1: Run both smoketests locally
• Step 2: Run hyperopt smoke trial
• Step 3: Full workspace compile
• Step 4: Commit and push