# Single Shared cuBLAS Handle for Deterministic Training

## Problem

3 CublasForward instances create 3 separate cuBLAS handles on 1 CUDA stream.
NVIDIA guarantees bit-wise reproducibility only with 1 handle per stream.
Multiple handles contaminate cuBLAS internal state, causing training to
diverge at step 37 (epoch 2 start, after experience collection).

## Solution

Split CublasForward into SharedCublasHandle (single, Arc-shared) and
CublasGemmSet (per-component cached descriptors). Same for CublasBackward.

## Architecture

### SharedCublasHandle (created once, Arc-shared)

- cuBLAS handle (cublasHandle_t) with CUBLAS_PEDANTIC_MATH
- cublasLt handle (cublasLtHandle_t)
- Workspace buffer (32MB CudaSlice<u8>) with cublasSetWorkspace
- cublasLt workspace buffer (32MB CudaSlice<u8>)
- Bias/ReLU kernels (CudaFunction) compiled from precompiled cubin
- AlgoGetIds cache: deterministic algo selection per (shape, compute_type)
- Stream reference (Arc<CudaStream>)

### CublasGemmSet (per-component, owns cached descriptors)

- batch_size, network dimensions (shared_h1, shared_h2, value_h, adv_h, etc.)
- gemm_cache: HashMap<FwdGemmKey, CachedGemmDesc>
- gemm_relu_cache: HashMap<FwdGemmKey, CachedGemmDesc> (fused RELU_BIAS)
- branch_workspace_ptrs: [u64; 4] (per-branch Lt workspace for parallel dispatch)
- branch_workspace_bufs: [CudaSlice<u8>; 4]
- state_dim_padded, s1_input_dim, s1_ldb
- Arc<SharedCublasHandle> reference

### CublasBackwardSet (per-component, owns backward descriptors)

- gemm_cache: HashMap<BwdGemmKey, CachedBwdGemmDesc>
- bias_grad_kernel: CudaFunction
- relu_mask_kernel: CudaFunction
- Lt workspace (separate from forward to avoid contention during graph capture)
- Arc<SharedCublasHandle> reference

## Construction Flow

```
FusedTrainingCtx::new(stream)
  handle = SharedCublasHandle::new(stream)       // ONE handle for everything
  training_fwd = CublasGemmSet::new(handle, b=64) 
  ddqn_fwd = CublasGemmSet::new(handle, b=64)
  backward = CublasBackwardSet::new(handle)
  experience_collector = GpuExperienceCollector::new(handle, b=32)
```

## GEMM Dispatch

CublasGemmSet::sgemm_f32() calls:
1. Look up cached descriptor by shape key
2. If miss: create descriptor using handle.lt_handle, cache it
3. Call cublasLtMatmul with handle.lt_handle + handle.lt_workspace
4. Algo from AlgoGetIds (deterministic, cached on SharedCublasHandle)

## Dead Code Removal

- Delete CublasForward::new (replaced by SharedCublasHandle::new + CublasGemmSet::new)
- Delete CublasBackward::new (replaced by CublasBackwardSet::new)
- Delete SendSyncCublasHandle from batched_forward.rs (moved to SharedCublasHandle)
- Delete SendSyncCublasLtHandle from batched_forward.rs (moved to SharedCublasHandle)
- Delete duplicate handle creation in gpu_experience_collector.rs
- Delete duplicate handle creation in gpu_dqn_trainer.rs (cublas_forward_ddqn handle)
- Delete double_dqn_stream fork (DDQN now on main stream, no separate stream needed)
- Delete pass1_event / pass3_event (no cross-stream sync needed)
- Delete eval_td_snapshot / eval_loss_snapshot if graph_forward replay is replaced
- Delete the separate eval graph capture code (exp_fwd_graph in experience collector)
- Clean all underscore-prefixed dead fields

## Files Changed

| File | Change |
|------|--------|
| batched_forward.rs | Split CublasForward -> SharedCublasHandle + CublasGemmSet |
| batched_backward.rs | CublasBackward -> CublasBackwardSet using shared handle |
| gpu_dqn_trainer.rs | Create SharedCublasHandle, pass Arc to all components |
| gpu_experience_collector.rs | Accept Arc<SharedCublasHandle>, delete own handle |
| fused_training.rs | Wire shared handle, delete double_dqn_stream |
| mod.rs | Export SharedCublasHandle |

## Success Criteria

4 consecutive cargo test runs produce identical val_Sharpe across all 30 epochs.
