7.8 KiB
GPU Test Fixes + Feature Extraction Pipeline Fix
Goal: Fix all 14 failing GPU tests so cargo test -p ml --lib and integration tests pass locally on RTX 3050, and fix OFI=false in the training pipeline so fxcache OFI data is used.
Scope: Bug fixes only — no new features, no refactoring beyond what's needed.
Root Causes (6 bugs → 14 test failures + 1 pipeline issue)
Bug 1: GPU profile TOML values diverged from test assertions (2 tests)
Files: crates/ml-core/src/gpu/profile.rs lines 292-308
The A100 and default GPU profile TOMLs were updated (batch_size, buffer_size) but the unit tests still assert old values.
| Profile | Field | TOML value | Test expects |
|---|---|---|---|
| A100 | batch_size |
2048 | 512 |
| A100 | buffer_size |
0 | 200_000 |
| Default | buffer_size |
0 | 50_000 |
Fix: Update test assertions to match current TOML values.
Tests fixed: test_embedded_a100_parses, test_embedded_default_parses
Bug 2: GPU smoke tests allocate host vec with wrong size for memcpy (2 tests)
File: crates/ml/tests/smoke_test_real_data.rs lines 573-574, 734-735
Tests pre-allocate vec![0.0f32; expected_total] with expected_total = n_episodes * timesteps, then call stream.memcpy_dtoh(&batch.rewards, &mut rewards_host). The GPU buffer may be larger than expected_total due to padding or episode boundary handling, causing assert!(dst.len() >= src.len()) panic in cudarc.
Fix: Size host vecs from actual GPU buffer length:
let mut rewards_host = vec![0.0f32; batch.rewards.len()];
stream.memcpy_dtoh(&batch.rewards, &mut rewards_host).unwrap();
Tests fixed: smoke_gpu_real_ohlcv_forward, smoke_gpu_real_data_noisy_distributional
Bug 3: GPU residency tests panic on CUDA init (4 tests)
File: crates/ml/src/trainers/dqn/smoke_tests/gpu_residency.rs lines 28-32
smoke_stream() calls cuda_device() which may fail if the CUDA device is busy or not initialized. The tests should handle init failure gracefully rather than panicking with an unhelpful error from cudarc internals.
Fix: Investigate cuda_device() in helpers.rs — the panic is likely a downstream effect of Bug 2's buffer size issue (these tests also do memcpy). Read the actual test bodies to identify the specific memcpy or kernel launch that fails. If CUDA init is the issue, add proper error propagation. If it's a buffer size issue (same as Bug 2), apply the same fix.
Tests fixed: test_gpu_replay_buffer_priority_update_valid, test_gpu_replay_buffer_proportional_sample_valid, test_gpu_replay_buffer_rank_based_sample_valid, test_searchsorted_gpu_correctness
Bug 4: dqn-smoketest profile sets data_source = "mbp10" with wrong path (3 tests)
File: config/training/dqn-smoketest.toml line 23-24
The smoketest profile sets data_source = "mbp10" and mbp10_data_dir = "test_data/futures-baseline-mbp10". The DQNTrainer then passes the symbol as a subdirectory, creating test_data/futures-baseline-mbp10/ES.FUT/. But somewhere in the path resolution (likely crates/ml/src/data_loader.rs lines 124-130 or crates/ml-features/src/mbp10_loader.rs), an /ohlcv suffix gets appended, creating the non-existent path test_data/futures-baseline-mbp10/ohlcv.
Fix: Two options:
- Option A (recommended): Change
dqn-smoketest.tomltodata_source = "ohlcv"— smoketests don't need MBP-10 OFI features, they just need to verify the training loop works. - Option B: Fix the path resolution that appends
/ohlcvto MBP-10 paths.
Option A is safer because smoketests should be fast and minimal. OFI correctness is tested separately.
Tests fixed: test_early_stopping_terminates_with_error, test_gradient_collapse_propagates_error, test_healthy_training_completes_successfully
Bug 5: CQL alpha default changed (1 test)
File: crates/ml/tests/dqn_action_collapse_fix_test.rs line 145
Test asserts DQNHyperparameters::default().cql_alpha == 0.1. Either the default was changed or the field was renamed/restructured.
Fix: Check actual default value in DQNHyperparameters::default() and update the test assertion, or restore the correct default if it was accidentally changed.
Tests fixed: test_cql_configurable_via_hyperparameters
Bug 6: training_profile assertion expects old values (2 tests)
File: crates/ml/src/training_profile.rs lines 1080-1102
test_apply_to_dqn_production asserts hp.mbp10_data_dir == "test_data/futures-baseline-mbp10" and test_apply_to_dqn_smoketest asserts !hp.mbp10_data_dir.is_empty(). These may fail if the profile loading changed.
Fix: Verify the profile TOML values and update assertions to match. These are unit tests for config parsing — they should assert what the TOML actually contains.
Tests fixed: test_apply_to_dqn_production, test_apply_to_dqn_smoketest, test_load_embedded_dqn_smoketest
Bug 7: DQN training pipeline tests (5 tests) — verification needed
File: crates/ml/tests/dqn_training_pipeline_test.rs
These tests (test_dqn_trains_on_es_fut, test_dqn_loss_decreases, etc.) have graceful skip logic — they return Ok(()) if test data isn't found. If they're failing rather than skipping, the test data exists but something downstream fails (possibly Bug 4's MBP-10 path issue propagating into the trainer).
Fix: Run locally to determine if they skip or fail. If they fail, trace the error — likely the same data_source = "mbp10" profile issue from Bug 4.
Tests fixed: test_dqn_checkpoint_save_load, test_dqn_epsilon_greedy, test_dqn_loss_decreases, test_dqn_q_value_predictions, test_dqn_trains_on_es_fut
Pipeline Fix: OFI=false
Symptom: init_from_fxcache: 697732 bars uploaded to GPU (143.7 MB, OFI=false) — the fxcache on PVC has OFI data (precompute wrote 8-dim OFI) but training doesn't use it.
Root cause: The train_baseline_rl binary's fxcache discovery didn't find the cache via FOXHUNT_FEATURE_CACHE_DIR=/feature-cache. It fell through to the DBN fallback path which creates ofi = vec![[0.0; 8]; n] — all zeros. The init_from_fxcache then sees empty OFI and sets OFI=false.
Why discovery failed: The old binary (compiled before our --feature-cache-dir commit) uses env var lookup → walk-up heuristic. The env var points to /feature-cache which exists (PVC mounted), but the fxcache key calculation uses the data paths. If the key doesn't match (different paths for mbp10/trades on the old binary vs what precompute used), the lookup returns None.
Fix: Verify the fxcache key calculation uses the same paths in both precompute_features and train_baseline_rl. The key is computed from calculate_dbn_cache_key_full(symbol_dir, mbp10, trades). If the paths differ between precompute (which uses Argo template parameters) and training (which uses CLI args), the keys won't match.
Concrete steps:
- Run the new binary (with
--feature-cache-dir) locally with the same data paths precompute uses - Verify the cache key matches
- If paths differ, normalize them before key calculation
Verification
All fixes verified locally on RTX 3050:
# Unit tests
SQLX_OFFLINE=true cargo test -p ml-core --lib -- gpu::profile
SQLX_OFFLINE=true cargo test -p ml --lib -- smoke_tests
# Integration tests
SQLX_OFFLINE=true cargo test -p ml --test smoke_test_real_data
SQLX_OFFLINE=true cargo test -p ml --test dqn_early_stopping_termination_test
SQLX_OFFLINE=true cargo test -p ml --test dqn_action_collapse_fix_test
SQLX_OFFLINE=true cargo test -p ml --test dqn_training_pipeline_test
# Smoke test (ignored, needs test data)
FOXHUNT_TEST_DATA=test_data/futures-baseline cargo test -p ml --lib -- smoke_tests --ignored --nocapture
All tests must pass (not skip) on a machine with CUDA (RTX 3050) and local test data.