# CUDA Pipeline Candle Elimination — Integration Plan

> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Eliminate all Candle Tensor usage from the 11 GPU pipeline files, merge 7 worktree branches, and validate on H100.

**Architecture:** Each GPU pipeline file is migrated from Candle Tensor ops to pure cudarc CudaSlice + CUDA kernels. A single forked CudaStream (owned by DQNTrainer) is passed to all components. No Candle default stream (stream 0) in the hot path. After integration, upgrade cudarc 0.17.3 → 0.19.3 for lifetime fixes and new APIs.

**Tech Stack:** Rust, cudarc (CudaSlice, CudaStream, NVRTC), CUDA kernels (.cu)

---

## Active Worktrees

| Worktree | Branch | Target Files | Status |
|----------|--------|-------------|--------|
| `agent-a1cbdbd9` | `worktree-agent-a1cbdbd9` | `use_*` flag elimination (config.rs, dqn.rs, 15+ files) | Complete |
| `agent-a4737e1f` | `worktree-agent-a4737e1f` | gpu_replay_buffer.rs + replay_buffer_type.rs | Running |
| `agent-a3abf665` | `worktree-agent-a3abf665` | gpu_action_selector.rs | Running |
| `agent-a94b8948` | `worktree-agent-a94b8948` | signal_adapter.rs | Running |
| `agent-aec3cba5` | `worktree-agent-aec3cba5` | gpu_weights.rs + gpu_iql_trainer.rs + gpu_training_guard.rs | Running |
| `agent-af70b6d2` | `worktree-agent-af70b6d2` | gpu_experience_collector.rs | Running |
| main worktree | `worktree-cuda-walltime-reduction` | Stream unification + sync fix + shmem fix | Current |

## Phase 1: Collect Agent Results (when agents complete)

### Task 1: Verify each agent worktree compiles

**Files:** Each agent's worktree

- [ ] **Step 1: Check agent-a1cbdbd9 (use_* flags)**
```bash
cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-a1cbdbd9
SQLX_OFFLINE=true cargo check -p ml -p ml-dqn --lib 2>&1 | grep error | head -5
```
Expected: zero errors

- [ ] **Step 2: Check agent-a4737e1f (gpu_replay_buffer)**
```bash
cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-a4737e1f
SQLX_OFFLINE=true cargo check -p ml-dqn --lib 2>&1 | grep error | head -5
```

- [ ] **Step 3: Check agent-a3abf665 (gpu_action_selector)**
```bash
cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-a3abf665
SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | grep error | head -5
```

- [ ] **Step 4: Check agent-a94b8948 (signal_adapter)**
```bash
cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-a94b8948
SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | grep error | head -5
```

- [ ] **Step 5: Check agent-aec3cba5 (gpu_weights + iql + guard)**
```bash
cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-aec3cba5
SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | grep error | head -5
```

- [ ] **Step 6: Check agent-af70b6d2 (gpu_experience_collector)**
```bash
cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/agent-af70b6d2
SQLX_OFFLINE=true cargo check -p ml --lib 2>&1 | grep error | head -5
```

---

## Phase 2: Cherry-pick into main worktree

Each agent made commits on its own branch. Cherry-pick them onto the main `worktree-cuda-walltime-reduction` branch in dependency order.

### Task 2: Merge use_* flag elimination first (no conflicts expected)

**Files:** crates/ml/src/trainers/dqn/config.rs, crates/ml-dqn/src/dqn.rs, 15+ files

- [ ] **Step 1: Cherry-pick the use_* commit**
```bash
cd /home/jgrusewski/Work/foxhunt/.claude/worktrees/cuda-walltime-reduction
git cherry-pick worktree-agent-a1cbdbd9 --no-commit
```

- [ ] **Step 2: Compile check**
```bash
SQLX_OFFLINE=true cargo check -p ml -p ml-dqn --lib
```

- [ ] **Step 3: Fix conflicts if any, then commit**
```bash
git commit -m "refactor(dqn): eliminate use_* boolean flags — all features always on"
```

### Task 3: Merge gpu_replay_buffer migration (highest conflict risk)

**Files:** crates/ml-dqn/src/gpu_replay_buffer.rs, crates/ml-dqn/src/replay_buffer_type.rs

- [ ] **Step 1: Cherry-pick**
```bash
git cherry-pick worktree-agent-a4737e1f --no-commit
```

- [ ] **Step 2: Resolve conflicts with stream unification changes**

Key conflict areas:
- `GpuBatch` struct: agent changed fields from Tensor to CudaSlice, main worktree added `indices` field
- `insert_batch`: agent rewrote to CudaSlice, main worktree added stream sync
- `priorities_tensor()`: main worktree added this for GPU PER update kernel

- [ ] **Step 3: Update callers in fused_training.rs**

The fused training loop uses `gpu_batch.states`, `gpu_batch.indices`, etc. If GpuBatch changed from Tensor to CudaSlice, update `run_full_step()` accordingly.

- [ ] **Step 4: Compile check + commit**

### Task 4: Merge gpu_experience_collector migration

**Files:** crates/ml/src/cuda_pipeline/gpu_experience_collector.rs

- [ ] **Step 1: Cherry-pick**
```bash
git cherry-pick worktree-agent-af70b6d2 --no-commit
```

- [ ] **Step 2: Resolve conflicts**

Key: `cuda_slice_to_tensor_f32()` was modified in both main (added sync) and agent (eliminated entirely). Take the agent's version (no Tensor at all).

- [ ] **Step 3: Compile check + commit**

### Task 5: Merge gpu_action_selector migration

**Files:** crates/ml/src/cuda_pipeline/gpu_action_selector.rs

- [ ] **Step 1: Cherry-pick**
- [ ] **Step 2: Update callers in trainer/action.rs** (Tensor → CudaSlice args)
- [ ] **Step 3: Compile check + commit**

### Task 6: Merge signal_adapter migration

**Files:** crates/ml/src/cuda_pipeline/signal_adapter.rs

- [ ] **Step 1: Cherry-pick**
- [ ] **Step 2: Update callers in inference.rs if any**
- [ ] **Step 3: Compile check + commit**

### Task 7: Merge gpu_weights + gpu_iql + gpu_training_guard

**Files:** crates/ml/src/cuda_pipeline/gpu_weights.rs, gpu_iql_trainer.rs, gpu_training_guard.rs

- [ ] **Step 1: Cherry-pick**
- [ ] **Step 2: Resolve conflicts with fused_training.rs** (IQL changes)
- [ ] **Step 3: Compile check + commit**

---

## Phase 3: Full integration compile + fix

### Task 8: Workspace-wide compilation

**Files:** All modified files

- [ ] **Step 1: Full workspace check**
```bash
SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep 'error\[' | wc -l
```

- [ ] **Step 2: Fix caller mismatches**

After all merges, callers that pass Tensor where CudaSlice is now expected need updating:
- `crates/ml/src/trainers/dqn/trainer/training_loop.rs` — experience batch handling
- `crates/ml/src/trainers/dqn/trainer/action.rs` — action selector calls
- `crates/ml/src/trainers/dqn/trainer/train_step.rs` — fused training batch
- `crates/ml/src/trainers/dqn/fused_training.rs` — GpuBatch field access
- `crates/ml/src/trainers/dqn/trainer/metrics.rs` — monitoring
- `crates/ml/examples/evaluate_baseline.rs` — evaluation
- `crates/ml/examples/train_baseline_rl.rs` — training binary
- `crates/ml/tests/*.rs` — integration tests

- [ ] **Step 3: Fix all errors iteratively**
```bash
SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep 'error\[' -A 3 | head -50
# Fix → recheck → repeat until zero errors
```

- [ ] **Step 4: Zero warnings check**
```bash
SQLX_OFFLINE=true cargo check --workspace 2>&1 | grep 'warning\[' | wc -l
```

- [ ] **Step 5: Commit integration fixes**
```bash
git commit -m "fix: resolve integration conflicts after Candle elimination merge"
```

---

## Phase 4: Candle audit — verify elimination is complete

### Task 9: Audit remaining Candle refs in GPU pipeline

- [ ] **Step 1: Count remaining Candle refs**
```bash
grep -rn 'Tensor::\|candle_core::Tensor\|\.to_dtype(\|\.narrow(\|\.slice_scatter(' \
  crates/ml/src/cuda_pipeline/ crates/ml-dqn/src/gpu_replay_buffer.rs \
  crates/ml-dqn/src/replay_buffer_type.rs | wc -l
```
Expected: 0 (or only in test code)

- [ ] **Step 2: Fix any remaining refs**

- [ ] **Step 3: Verify no Candle Device in GPU hot path**
```bash
grep -rn 'candle_core::Device\|Device::Cuda' crates/ml/src/cuda_pipeline/ | grep -v test | grep -v '//' | wc -l
```

---

## Phase 5: Test on RTX 3050 + H100

### Task 10: Local RTX 3050 tests

- [ ] **Step 1: ml-dqn lib tests**
```bash
SQLX_OFFLINE=true cargo test -p ml-dqn --lib -- --test-threads=1
```
Expected: 392+ passed, 0 failed

- [ ] **Step 2: ml lib tests (non-training)**
```bash
SQLX_OFFLINE=true cargo test -p ml --lib -- --test-threads=1 \
  --skip smoke_tests::feature_coverage --skip smoke_tests::training_stability
```
Expected: 890+ passed, 0 failed

- [ ] **Step 3: Production smoke test**
```bash
FOXHUNT_TEST_DATA=/path/to/test_data SQLX_OFFLINE=true \
  timeout 300 cargo test -p ml --lib smoke_tests::feature_coverage -- --test-threads=1
```
Expected: 1 passed (may need release mode for speed)

### Task 11: H100 CI pipeline

- [ ] **Step 1: Push and submit**
```bash
git push origin worktree-cuda-walltime-reduction
argo submit --from workflowtemplate/gpu-test-pipeline \
  -p commit-ref=worktree-cuda-walltime-reduction \
  -p models=dqn -p test-scope=all -n foxhunt
```

- [ ] **Step 2: Monitor pipeline**
```bash
argo get <workflow-name> -n foxhunt
kubectl logs -n foxhunt <pod-name> -c main --tail=30
```
Expected: All DQN tests pass, epoch completes in <60s

- [ ] **Step 3: Verify no hangs**

Check that "GPU collected N experiences" is followed by "Epoch complete" within 60s.

---

## Phase 6: cudarc 0.19.3 upgrade (after Candle elimination is clean)

### Task 12: Upgrade cudarc dependency

**Files:** Cargo.toml (workspace), Cargo.lock

- [ ] **Step 1: Update candle-core's cudarc dep**

Check if candle v0.9.1 supports cudarc 0.19.3. If not, use `[patch.crates-io]`:
```toml
[patch.crates-io]
cudarc = { version = "0.19.3" }
```

- [ ] **Step 2: Fix lifetime changes**

0.19.3 changes CudaView/CudaViewMut lifetimes:
- `'b: 'a` → `'b` on slice/transmute methods
- `CudaViewMut` methods return `CudaViewMut<'b, T>` not `Self`

Grep for affected patterns:
```bash
grep -rn 'as_view\|slice_mut\|split_at_mut\|transmute' crates/ml/src/cuda_pipeline/
```

- [ ] **Step 3: Replace deprecated APIs**
- `memcpy_stod` → `clone_htod`
- `memcpy_dtov` → `clone_dtoh`
- Manual compute cap parsing → `context.compute_capability()`

- [ ] **Step 4: Full compile + test**

---

## Phase 7: Squash and PR

### Task 13: Clean up git history

- [ ] **Step 1: Interactive rebase to squash**
```bash
git rebase -i origin/main
```
Squash into logical commits:
1. `fix(cuda): shmem tile overflow + event tracking` (original bug fixes)
2. `refactor(cuda): unify onto single forked CudaStream`
3. `refactor(cuda): eliminate Candle from GPU pipeline`
4. `refactor(dqn): remove use_* boolean flags`
5. `chore: upgrade cudarc 0.17.3 → 0.19.3` (if included)

- [ ] **Step 2: Push and create MR**
```bash
git push origin worktree-cuda-walltime-reduction --force-with-lease
```
Create MR on GitLab targeting `main`.
