foxhunt/docs/superpowers/plans/2026-03-17-cuda-candle-elimination.md

CUDA Pipeline Candle Elimination — Integration Plan

For agentic workers: REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Eliminate all Candle Tensor usage from the 11 GPU pipeline files, merge 7 worktree branches, and validate on H100.

Architecture: Each GPU pipeline file is migrated from Candle Tensor ops to pure cudarc CudaSlice + CUDA kernels. A single forked CudaStream (owned by DQNTrainer) is passed to all components. No Candle default stream (stream 0) in the hot path. After integration, upgrade cudarc 0.17.3 → 0.19.3 for lifetime fixes and new APIs.

Tech Stack: Rust, cudarc (CudaSlice, CudaStream, NVRTC), CUDA kernels (.cu)

---

Active Worktrees

Worktree	Branch	Target Files	Status
agent-a1cbdbd9	worktree-agent-a1cbdbd9	use_* flag elimination (config.rs, dqn.rs, 15+ files)	Complete
agent-a4737e1f	worktree-agent-a4737e1f	gpu_replay_buffer.rs + replay_buffer_type.rs	Running
agent-a3abf665	worktree-agent-a3abf665	gpu_action_selector.rs	Running
agent-a94b8948	worktree-agent-a94b8948	signal_adapter.rs	Running
agent-aec3cba5	worktree-agent-aec3cba5	gpu_weights.rs + gpu_iql_trainer.rs + gpu_training_guard.rs	Running
agent-af70b6d2	worktree-agent-af70b6d2	gpu_experience_collector.rs	Running
main worktree	worktree-cuda-walltime-reduction	Stream unification + sync fix + shmem fix	Current

Phase 1: Collect Agent Results (when agents complete)

Task 1: Verify each agent worktree compiles

Files: Each agent's worktree

• Step 1: Check agent-a1cbdbd9 (use_ flags)*

cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-a1cbdbd9
SQLX_OFFLINE=true cargo check -p ml -p ml-dqn --lib 2>&1 | grep error | head -5

Expected: zero errors

• Step 2: Check agent-a4737e1f (gpu_replay_buffer)

cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-a4737e1f
SQLX_OFFLINE=true cargo check -p ml-dqn --lib 2>&1 | grep error | head -5

• Step 3: Check agent-a3abf665 (gpu_action_selector)

cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-a3abf665
SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | grep error | head -5

• Step 4: Check agent-a94b8948 (signal_adapter)

cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-a94b8948
SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | grep error | head -5

• Step 5: Check agent-aec3cba5 (gpu_weights + iql + guard)

cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-aec3cba5
SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | grep error | head -5

• Step 6: Check agent-af70b6d2 (gpu_experience_collector)

cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-af70b6d2
SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | grep error | head -5

---

Phase 2: Cherry-pick into main worktree

Each agent made commits on its own branch. Cherry-pick them onto the main worktree-cuda-walltime-reduction branch in dependency order.

Task 2: Merge use_* flag elimination first (no conflicts expected)

Files: crates/ml/src/trainers/dqn/config.rs, crates/ml-dqn/src/dqn.rs, 15+ files

• Step 1: Cherry-pick the use_ commit*

cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/cuda-walltime-reduction
git cherry-pick worktree-agent-a1cbdbd9 --no-commit

• Step 2: Compile check

SQLX_OFFLINE=true cargo check -p ml -p ml-dqn --lib

• Step 3: Fix conflicts if any, then commit

git commit -m "refactor(dqn): eliminate use_* boolean flags — all features always on"

Task 3: Merge gpu_replay_buffer migration (highest conflict risk)

Files: crates/ml-dqn/src/gpu_replay_buffer.rs, crates/ml-dqn/src/replay_buffer_type.rs

• Step 1: Cherry-pick

git cherry-pick worktree-agent-a4737e1f --no-commit

• Step 2: Resolve conflicts with stream unification changes

Key conflict areas:

• GpuBatch struct: agent changed fields from Tensor to CudaSlice, main worktree added indices field

• insert_batch: agent rewrote to CudaSlice, main worktree added stream sync

• priorities_tensor(): main worktree added this for GPU PER update kernel

• Step 3: Update callers in fused_training.rs

The fused training loop uses gpu_batch.states, gpu_batch.indices, etc. If GpuBatch changed from Tensor to CudaSlice, update run_full_step() accordingly.

• Step 4: Compile check + commit

Task 4: Merge gpu_experience_collector migration

Files: crates/ml/src/cuda_pipeline/gpu_experience_collector.rs

• Step 1: Cherry-pick

git cherry-pick worktree-agent-af70b6d2 --no-commit

• Step 2: Resolve conflicts

Key: cuda_slice_to_tensor_f32() was modified in both main (added sync) and agent (eliminated entirely). Take the agent's version (no Tensor at all).

• Step 3: Compile check + commit

Task 5: Merge gpu_action_selector migration

Files: crates/ml/src/cuda_pipeline/gpu_action_selector.rs

• Step 1: Cherry-pick
• Step 2: Update callers in trainer/action.rs (Tensor → CudaSlice args)
• Step 3: Compile check + commit

Task 6: Merge signal_adapter migration

Files: crates/ml/src/cuda_pipeline/signal_adapter.rs

• Step 1: Cherry-pick
• Step 2: Update callers in inference.rs if any
• Step 3: Compile check + commit

Task 7: Merge gpu_weights + gpu_iql + gpu_training_guard

Files: crates/ml/src/cuda_pipeline/gpu_weights.rs, gpu_iql_trainer.rs, gpu_training_guard.rs

• Step 1: Cherry-pick
• Step 2: Resolve conflicts with fused_training.rs (IQL changes)
• Step 3: Compile check + commit

---

Phase 3: Full integration compile + fix

Task 8: Workspace-wide compilation

Files: All modified files

• Step 1: Full workspace check

SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep 'error\[' | wc -l

• Step 2: Fix caller mismatches

After all merges, callers that pass Tensor where CudaSlice is now expected need updating:

• crates/ml/src/trainers/dqn/trainer/training_loop.rs — experience batch handling

• crates/ml/src/trainers/dqn/trainer/action.rs — action selector calls

• crates/ml/src/trainers/dqn/trainer/train_step.rs — fused training batch

• crates/ml/src/trainers/dqn/fused_training.rs — GpuBatch field access

• crates/ml/src/trainers/dqn/trainer/metrics.rs — monitoring

• crates/ml/examples/evaluate_baseline.rs — evaluation

• crates/ml/examples/train_baseline_rl.rs — training binary

• crates/ml/tests/*.rs — integration tests

• Step 3: Fix all errors iteratively

SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep 'error\[' -A 3 | head -50
# Fix → recheck → repeat until zero errors

• Step 4: Zero warnings check

SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep 'warning\[' | wc -l

• Step 5: Commit integration fixes

git commit -m "fix: resolve integration conflicts after Candle elimination merge"

---

Phase 4: Candle audit — verify elimination is complete

Task 9: Audit remaining Candle refs in GPU pipeline

• Step 1: Count remaining Candle refs

grep -rn 'Tensor::\|candle_core::Tensor\|\.to_dtype(\|\.narrow(\|\.slice_scatter(' \
  crates/ml/src/cuda_pipeline/ crates/ml-dqn/src/gpu_replay_buffer.rs \
  crates/ml-dqn/src/replay_buffer_type.rs | wc -l

Expected: 0 (or only in test code)

• Step 2: Fix any remaining refs

• Step 3: Verify no Candle Device in GPU hot path

grep -rn 'candle_core::Device\|Device::Cuda' crates/ml/src/cuda_pipeline/ | grep -v test | grep -v '//' | wc -l

---

Phase 5: Test on RTX 3050 + H100

Task 10: Local RTX 3050 tests

• Step 1: ml-dqn lib tests

SQLX_OFFLINE=true cargo test -p ml-dqn --lib -- --test-threads=1

Expected: 392+ passed, 0 failed

• Step 2: ml lib tests (non-training)

SQLX_OFFLINE=true cargo test -p ml --lib -- --test-threads=1 \
  --skip smoke_tests::feature_coverage --skip smoke_tests::training_stability

Expected: 890+ passed, 0 failed

• Step 3: Production smoke test

FOXHUNT_TEST_DATA=/path/to/test_data SQLX_OFFLINE=true \
  timeout 300 cargo test -p ml --lib smoke_tests::feature_coverage -- --test-threads=1

Expected: 1 passed (may need release mode for speed)

Task 11: H100 CI pipeline

• Step 1: Push and submit

git push origin worktree-cuda-walltime-reduction
argo submit --from workflowtemplate/gpu-test-pipeline \
  -p commit-ref=worktree-cuda-walltime-reduction \
  -p models=dqn -p test-scope=all -n foxhunt

• Step 2: Monitor pipeline

argo get <workflow-name> -n foxhunt
kubectl logs -n foxhunt <pod-name> -c main --tail=30

Expected: All DQN tests pass, epoch completes in <60s

• Step 3: Verify no hangs

Check that "GPU collected N experiences" is followed by "Epoch complete" within 60s.

---

Phase 6: cudarc 0.19.3 upgrade (after Candle elimination is clean)

Task 12: Upgrade cudarc dependency

Files: Cargo.toml (workspace), Cargo.lock

• Step 1: Update candle-core's cudarc dep

Check if candle v0.9.1 supports cudarc 0.19.3. If not, use [patch.crates-io]:

[patch.crates-io]
cudarc = { version = "0.19.3" }

• Step 2: Fix lifetime changes

0.19.3 changes CudaView/CudaViewMut lifetimes:

• 'b: 'a → 'b on slice/transmute methods
• CudaViewMut methods return CudaViewMut<'b, T> not Self

Grep for affected patterns:

grep -rn 'as_view\|slice_mut\|split_at_mut\|transmute' crates/ml/src/cuda_pipeline/

• Step 3: Replace deprecated APIs

• memcpy_stod → clone_htod

• memcpy_dtov → clone_dtoh

• Manual compute cap parsing → context.compute_capability()

• Step 4: Full compile + test

---

Phase 7: Squash and PR

Task 13: Clean up git history

• Step 1: Interactive rebase to squash

git rebase -i origin/main

Squash into logical commits:

1. fix(cuda): shmem tile overflow + event tracking (original bug fixes)
2. refactor(cuda): unify onto single forked CudaStream
3. refactor(cuda): eliminate Candle from GPU pipeline
4. refactor(dqn): remove use_* boolean flags
5. chore: upgrade cudarc 0.17.3 → 0.19.3 (if included)

• Step 2: Push and create MR

git push origin worktree-cuda-walltime-reduction --force-with-lease

Create MR on GitLab targeting main.