====================================================================
DETAILED CUDA 12.9 VERIFICATION REPORT
====================================================================

BINARY LOCATIONS (Absolute Paths)
--------------------------------------------------------------------
/home/jgrusewski/Work/foxhunt/target/release/examples/train_mamba2_parquet
/home/jgrusewski/Work/foxhunt/target/release/examples/train_tft_parquet
/home/jgrusewski/Work/foxhunt/target/release/examples/train_dqn
/home/jgrusewski/Work/foxhunt/target/release/examples/train_ppo

LIBRARY DEPENDENCY ANALYSIS (ldd output)
--------------------------------------------------------------------

train_mamba2_parquet CUDA libraries:
	libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1 (0x00007a6b418fd000)
	libcurand.so.10 => /usr/local/cuda-12.9/lib64/libcurand.so.10 (0x00007a6b37000000)
	libcublas.so.12 => /usr/local/cuda-12.9/lib64/libcublas.so.12 (0x00007a6b30800000)
	libcublasLt.so.12 => /usr/local/cuda-12.9/lib64/libcublasLt.so.12 (0x00007a6afde00000)

train_tft_parquet CUDA libraries:
	libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1 (0x0000725c8f530000)
	libcurand.so.10 => /usr/local/cuda-12.9/lib64/libcurand.so.10 (0x0000725c84c00000)
	libcublas.so.12 => /usr/local/cuda-12.9/lib64/libcublas.so.12 (0x0000725c7e400000)
	libcublasLt.so.12 => /usr/local/cuda-12.9/lib64/libcublasLt.so.12 (0x0000725c4ba00000)

train_dqn CUDA libraries:
	libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1 (0x00007b013af68000)
	libcurand.so.10 => /usr/local/cuda-12.9/lib64/libcurand.so.10 (0x00007b0130600000)
	libcublas.so.12 => /usr/local/cuda-12.9/lib64/libcublas.so.12 (0x00007b0129e00000)
	libcublasLt.so.12 => /usr/local/cuda-12.9/lib64/libcublasLt.so.12 (0x00007b00f7400000)

train_ppo CUDA libraries:
	libcuda.so.1 => /lib/x86_64-linux-gnu/libcuda.so.1 (0x00007f1ffbceb000)
	libcurand.so.10 => /usr/local/cuda-12.9/lib64/libcurand.so.10 (0x00007f1ff1400000)
	libcublas.so.12 => /usr/local/cuda-12.9/lib64/libcublas.so.12 (0x00007f1feac00000)
	libcublasLt.so.12 => /usr/local/cuda-12.9/lib64/libcublasLt.so.12 (0x00007f1fb8200000)

VERIFICATION SUMMARY
--------------------------------------------------------------------
✅ All binaries link to /usr/local/cuda-12.9/lib64/ libraries
✅ No references to cuda-13.0 libraries found
✅ Critical libraries verified:
   - libcublas.so.12 (NOT libcublas.so.13)
   - libcublasLt.so.12
   - libcurand.so.10

RUNPOD COMPATIBILITY
--------------------------------------------------------------------
Runpod Docker Container CUDA Version: 12.9.1
Local Compilation CUDA Version:       12.9
Binary CUDA Linkage:                  12.9 (libcublas.so.12)

Expected Runtime Behavior:
  ✅ Binaries will find libcublas.so.12 in Runpod container
  ✅ No "cannot open shared object file" errors
  ✅ Full CUDA GPU acceleration enabled

Previous Issue (FIXED):
  ❌ Binaries linked to libcublas.so.13 (not available in Runpod)
  ✅ Now linked to libcublas.so.12 (available in Runpod CUDA 12.9.1)

====================================================================