gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(ml): Complete hyperopt infrastructure + documentation

Browse Source 
Changes:
- CLAUDE.md: Update OOM fix validation status
- Add comprehensive documentation (30+ markdown reports)
- LSTM encoder varmap bug fix (tft/lstm_encoder.rs:290)
- Quantized LSTM layer matching fix (tft/quantized_lstm.rs)
- Hyperopt paths module (ml/src/hyperopt/paths.rs)
- Training path tests for all adapters (DQN, MAMBA-2, PPO, TFT)
- Checkpoint integrity tests
- Script cleanup: Remove 29 obsolete deployment scripts
- Archive old scripts to scripts/archive/
- New deployment utilities: check_gpu_availability.py, monitor_hyperopt.sh

Validation:
- OOM fixes validated: 5/5 trials successful (pod b6kc3mc5lbjiro)
- Batch-size-max 256 tested successfully
- All hyperopt adapters working correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2025-10-29 19:52:21 +01:00
parent 59cce96d9d
commit e61e8f54da
111 changed files with 15450 additions and 838 deletions
Download Patch File Download Diff File Expand all files Collapse all files

170
ml/src/hyperopt/adapters/async_data_loader.rs
View File

@@ -39,7 +39,7 @@ use anyhow::Result;
use candle_core::{Device, Tensor};
use std::sync::mpsc::{sync_channel, Receiver, SyncSender, TryRecvError};
use std::thread::{self, JoinHandle};
use tracing::{debug, info, warn};
use tracing::{debug, warn};
use crate::MLError;
@@ -121,14 +121,20 @@ impl AsyncDataLoader {
}
let total_batches = (data.len() + batch_size - 1) / batch_size;
let device_clone = device.clone();
info!(
"Creating AsyncDataLoader: {} samples, batch_size={}, prefetch={}, batches={}",
// CRITICAL FIX: Explicitly clone device for storage
// Issue: device.clone() on &Device might not properly clone CUDA devices
// Solution: Use Device::clone() explicitly to ensure proper cloning
let device_owned = (*device).clone();
let device_clone = device_owned.clone();
debug!(
"Creating AsyncDataLoader: {} samples, batch_size={}, prefetch={}, batches={}, device={:?}",
data.len(),
batch_size,
prefetch_count,
total_batches
total_batches,
device_owned
);
// Create bounded channel - blocks if full (backpressure)
@@ -144,7 +150,7 @@ impl AsyncDataLoader {
prefetch_thread: Some(prefetch_thread),
total_batches,
current_batch: 0,
device: device.clone(),
device: device_owned,
})
}
@@ -152,9 +158,11 @@ impl AsyncDataLoader {
///
/// This runs in a separate thread and:
/// 1. Chunks data into batches
/// 2. Concatenates tensors for each batch
/// 3. Transfers to GPU
/// 4. Sends via channel to training loop
/// 2. Concatenates tensors for each batch ON CPU
/// 3. Sends via channel to training loop
///
/// NOTE: GPU transfer happens in main thread to avoid CUDA context issues.
/// CUDA contexts are thread-local, so transferring in background thread can fail.
///
/// Stops when:
/// - All batches processed
@@ -164,15 +172,15 @@ impl AsyncDataLoader {
data: Vec<(Tensor, Tensor)>,
batch_size: usize,
sender: SyncSender<Result<(Tensor, Tensor), MLError>>,
device: Device,
_device: Device, // Unused - kept for API compatibility
) {
debug!("Prefetch worker started: {} samples", data.len());
for (batch_idx, batch_data) in data.chunks(batch_size).enumerate() {
// Note: SyncSender doesn't have is_disconnected(), we'll rely on send() error instead
// Prepare batch on CPU
let batch_result = Self::prepare_batch(batch_data, &device);
// Prepare batch on CPU ONLY (no GPU transfer in background thread)
let batch_result = Self::prepare_batch_cpu(batch_data);
// Send to training loop (blocks if channel full)
if let Err(e) = sender.send(batch_result) {
@@ -188,21 +196,20 @@ impl AsyncDataLoader {
debug!("Prefetch worker finished");
}
/// Prepare a single batch: concatenate tensors and move to GPU
/// Prepare a single batch on CPU: concatenate tensors
///
/// This is the CPU-intensive operation we want to overlap with GPU training.
/// GPU transfer happens in the main thread to avoid CUDA context issues.
///
/// # Arguments
///
/// * `batch_data` - Slice of (feature, target) tensor pairs
/// * `device` - GPU device to transfer to
///
/// # Returns
///
/// Batched tensors on GPU, or error if preparation fails
fn prepare_batch(
/// Batched tensors on CPU, or error if preparation fails
fn prepare_batch_cpu(
batch_data: &[(Tensor, Tensor)],
device: &Device,
) -> Result<(Tensor, Tensor), MLError> {
if batch_data.is_empty() {
return Err(MLError::InvalidInput("Empty batch".to_string()));
@@ -238,21 +245,7 @@ impl AsyncDataLoader {
})?
};
// Transfer to GPU (most expensive operation - now async!)
let batched_features = batched_features
.to_device(device)
.map_err(|e| MLError::TensorCreationError {
operation: "transfer features to device".to_string(),
reason: e.to_string(),
})?;
let batched_targets = batched_targets
.to_device(device)
.map_err(|e| MLError::TensorCreationError {
operation: "transfer targets to device".to_string(),
reason: e.to_string(),
})?;
// Return CPU tensors - GPU transfer happens in main thread
Ok((batched_features, batched_targets))
}
@@ -260,19 +253,85 @@ impl AsyncDataLoader {
///
/// Returns `None` when all batches consumed or on error.
///
/// NOTE: This method transfers tensors from CPU to GPU in the main thread
/// to avoid CUDA context issues. The background thread only prepares batches on CPU.
///
/// # Returns
///
/// - `Some((features, targets))` - Next batch ready on GPU
/// - `None` - No more batches or error occurred
pub fn next_batch(&mut self) -> Option<(Tensor, Tensor)> {
debug!("next_batch() called: current_batch={}, total_batches={}",
self.current_batch, self.total_batches);
if self.current_batch >= self.total_batches {
debug!("next_batch() returning None: reached total_batches");
return None;
}
debug!("next_batch() waiting for receiver.recv()...");
match self.receiver.recv() {
Ok(Ok((features, targets))) => {
self.current_batch += 1;
Some((features, targets))
debug!("Received batch {} from prefetch worker (features: {:?}, targets: {:?})",
self.current_batch, features.device(), targets.device());
// Transfer to GPU in main thread (CUDA context is valid here)
debug!(
"Batch {} - Before transfer: features device={:?}, targets device={:?}, target device={:?}",
self.current_batch,
features.device(),
targets.device(),
self.device
);
let features_gpu = match features.to_device(&self.device) {
Ok(t) => {
// CRITICAL FIX: Verify transfer actually happened
if t.device().is_cuda() != self.device.is_cuda() {
warn!(
"Device transfer failed: expected {:?}, got {:?} (to_device returned wrong device)",
self.device, t.device()
);
return None;
}
debug!(
"Batch {} - After transfer: features device={:?}",
self.current_batch, t.device()
);
t
}
Err(e) => {
warn!("Failed to transfer features to GPU at batch {}: {}",
self.current_batch - 1, e);
return None;
}
};
let targets_gpu = match targets.to_device(&self.device) {
Ok(t) => {
// CRITICAL FIX: Verify transfer actually happened
if t.device().is_cuda() != self.device.is_cuda() {
warn!(
"Device transfer failed: expected {:?}, got {:?} (to_device returned wrong device)",
self.device, t.device()
);
return None;
}
debug!(
"Batch {} - After transfer: targets device={:?}",
self.current_batch, t.device()
);
t
}
Err(e) => {
warn!("Failed to transfer targets to GPU at batch {}: {}",
self.current_batch - 1, e);
return None;
}
};
Some((features_gpu, targets_gpu))
}
Ok(Err(e)) => {
warn!("Batch preparation error at batch {}: {}", self.current_batch, e);
@@ -290,9 +349,12 @@ impl AsyncDataLoader {
///
/// Useful for checking if data is ready without waiting.
///
/// NOTE: This method transfers tensors from CPU to GPU in the main thread
/// to avoid CUDA context issues.
///
/// # Returns
///
/// - `Some((features, targets))` - Batch ready immediately
/// - `Some((features, targets))` - Batch ready immediately on GPU
/// - `None` - No batch ready yet (try again later) or stream ended
pub fn try_next_batch(&mut self) -> Option<(Tensor, Tensor)> {
if self.current_batch >= self.total_batches {
@@ -302,7 +364,45 @@ impl AsyncDataLoader {
match self.receiver.try_recv() {
Ok(Ok((features, targets))) => {
self.current_batch += 1;
Some((features, targets))
// Transfer to GPU in main thread (CUDA context is valid here)
debug!(
"try_next_batch - Batch {} - Before transfer: features device={:?}, targets device={:?}, target device={:?}",
self.current_batch,
features.device(),
targets.device(),
self.device
);
let features_gpu = match features.to_device(&self.device) {
Ok(t) => {
debug!(
"try_next_batch - Batch {} - After transfer: features device={:?}",
self.current_batch, t.device()
);
t
}
Err(e) => {
warn!("Failed to transfer features to GPU: {}", e);
return None;
}
};
let targets_gpu = match targets.to_device(&self.device) {
Ok(t) => {
debug!(
"try_next_batch - Batch {} - After transfer: targets device={:?}",
self.current_batch, t.device()
);
t
}
Err(e) => {
warn!("Failed to transfer targets to GPU: {}", e);
return None;
}
};
Some((features_gpu, targets_gpu))
}
Ok(Err(e)) => {
warn!("Batch preparation error: {}", e);