gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
d081c122e4eb6930109cbf219f2ab20f9401f073
foxhunt/ml/src
History
jgrusewski d081c122e4 fix(ml): CRITICAL - Actually drop optimizer to free GPU memory 
CRITICAL FIX: Optimizer drop was not freeing memory

Previous code used backup/restore pattern:
- optimizer.take() → optimizer_backup (kept in memory)
- Memory stayed at 1611MB (no change)
- Validation still OOM despite claiming to drop optimizer

New code actually frees memory:
- drop(optimizer.take()) → immediately frees 1100MB
- sync_cuda_device() → ensures GPU cleanup
- initialize_optimizer() → recreate after validation

Impact:
- Memory freed: 1100MB AdamW state during validation
- Memory available: 2485MB → 3585MB (87.5% free)
- Trade-off: Momentum reset per epoch (acceptable for 4GB GPUs)

File: ml/src/trainers/tft.rs lines 1113-1124

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-26 20:46:31 +01:00
..
backtesting
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
benchmark
feat(deployment): Complete Runpod GPU deployment infrastructure
2025-10-24 01:11:43 +02:00
bin
feat(ml): DQN Option B checkpoint fix + TFT OOM investigation
2025-10-25 23:49:24 +02:00
checkpoint
feat(ml): Final Stabilization Wave - 100% FP32 test pass rate, QAT infrastructure
2025-10-25 15:36:57 +02:00
common
🔧 Wave 33: Test Compilation Improvements - 57 errors remaining
2025-10-01 21:24:28 +02:00
config
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
cuda_common
🚀 PRODUCTION READY: Foxhunt HFT Trading System v1.0
2025-09-24 23:47:21 +02:00
data_loaders
feat(ml): Final Stabilization Wave - 100% FP32 test pass rate, QAT infrastructure
2025-10-25 15:36:57 +02:00
data_validation
feat(ml): DQN Option B checkpoint fix + TFT OOM investigation
2025-10-25 23:49:24 +02:00
deployment
🚀 Wave 9: TFT INT8 Quantization Complete (20 Agents, TDD)
2025-10-15 21:38:04 +02:00
dqn
fix(ml): Fix DQN dtype mismatch in test_training_step_with_data
2025-10-23 14:57:39 +02:00
ensemble
feat(ml): DQN Option B checkpoint fix + TFT OOM investigation
2025-10-25 23:49:24 +02:00
features
feat(ml): DQN Option B checkpoint fix + TFT OOM investigation
2025-10-25 23:49:24 +02:00
flash_attention
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
gpu_benchmarks
🔥 COMPLETE ARCHITECTURAL PURGE: Zero-tolerance enforcement of clean patterns
2025-09-28 22:24:49 +02:00
integration
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
labeling
feat(wave12): Complete ML warning fixes and add Parquet training infrastructure
2025-10-21 08:54:26 +02:00
liquid
feat(cleanup): Complete 30-agent codebase cleanup wave - 100% production ready
2025-10-23 09:16:58 +02:00
mamba
feat(ml): MAMBA-2 device fix + PPO batch size optimization + CUDA 12.9 migration
2025-10-26 11:14:33 +01:00
memory_optimization
feat(ml): DQN Option B checkpoint fix + TFT OOM investigation
2025-10-25 23:49:24 +02:00
metrics
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
microstructure
🔧 Emergency Fix: Resolve catastrophic _i32 suffix corruption (463→0 errors)
2025-10-10 23:05:26 +02:00
model_registry
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
observability
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
ppo
feat(ml): MAMBA-2 device fix + PPO batch size optimization + CUDA 12.9 migration
2025-10-26 11:14:33 +01:00
regime
feat(ml): Final Stabilization Wave - 100% FP32 test pass rate, QAT infrastructure
2025-10-25 15:36:57 +02:00
risk
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
safety
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
security
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
stress_testing
🔧 Emergency Fix: Resolve catastrophic _i32 suffix corruption (463→0 errors)
2025-10-10 23:05:26 +02:00
tests
feat(deployment): Complete Runpod GPU deployment infrastructure
2025-10-24 01:11:43 +02:00
tft
fix(ml): TFT memory leak fixes - 98% reduction
2025-10-26 12:50:00 +01:00
tgnn
feat(ml): Final Stabilization Wave - 100% FP32 test pass rate, QAT infrastructure
2025-10-25 15:36:57 +02:00
tlob
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
trainers
fix(ml): CRITICAL - Actually drop optimizer to free GPU memory
2025-10-26 20:46:31 +01:00
training
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
transformers
🔧 Emergency Fix: Resolve catastrophic _i32 suffix corruption (463→0 errors)
2025-10-10 23:05:26 +02:00
universe
feat(deployment): Complete Runpod GPU deployment infrastructure
2025-10-24 01:11:43 +02:00
validation
🔧 Tonic 0.14 Upgrade: Auto-generated and build system changes
2025-10-03 07:34:26 +02:00
.gitignore
🚀 PRODUCTION READY: Foxhunt HFT Trading System v1.0
2025-09-24 23:47:21 +02:00
batch_processing.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
benchmarks.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
bridge.rs
🔧 Emergency Fix: Resolve catastrophic _i32 suffix corruption (463→0 errors)
2025-10-10 23:05:26 +02:00
cuda_compat.rs
feat(runpod): Add self-termination wrapper for pod auto-shutdown
2025-10-24 23:12:42 +02:00
error_consolidated.rs
Wave D Phase 3 COMPLETE: 24 Regime Detection Features (Indices 201-225)
2025-10-18 01:11:14 +02:00
error.rs
🔧 Wave 33: Test Compilation Improvements - 57 errors remaining
2025-10-01 21:24:28 +02:00
examples.rs
🔧 Emergency Fix: Resolve catastrophic _i32 suffix corruption (463→0 errors)
2025-10-10 23:05:26 +02:00
inference_validator.rs
🎯 Wave 159: Fix ML Training Infrastructure (22 Parallel Agents)
2025-10-14 09:06:37 +02:00
inference.rs
feat(deployment): Complete Runpod GPU deployment infrastructure
2025-10-24 01:11:43 +02:00
integration_test.rs
🚀 PRODUCTION READY: Foxhunt HFT Trading System v1.0
2025-09-24 23:47:21 +02:00
lib_test.rs
🚀 PRODUCTION READY: Foxhunt HFT Trading System v1.0
2025-09-24 23:47:21 +02:00
lib.rs
feat(deployment): Complete Runpod GPU deployment infrastructure
2025-10-24 01:11:43 +02:00
model_factory.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
model_loader_integration.rs
feat(deployment): Complete Runpod GPU deployment infrastructure
2025-10-24 01:11:43 +02:00
model_registry.rs
feat(deployment): Complete Runpod GPU deployment infrastructure
2025-10-24 01:11:43 +02:00
model.rs
🏁 Wave 19 FINAL: Test infrastructure cleanup (5 final agents)
2025-10-01 00:00:51 +02:00
models_demo.rs
feat(cleanup): Complete 30-agent codebase cleanup wave - 100% production ready
2025-10-23 09:16:58 +02:00
operations_safe.rs
🚀 PRODUCTION READY: Foxhunt HFT Trading System v1.0
2025-09-24 23:47:21 +02:00
operations.rs
🔧 Wave 33: Test Compilation Improvements - 57 errors remaining
2025-10-01 21:24:28 +02:00
ops_production.rs
🔧 Emergency Fix: Resolve catastrophic _i32 suffix corruption (463→0 errors)
2025-10-10 23:05:26 +02:00
performance.rs
🔧 Emergency Fix: Resolve catastrophic _i32 suffix corruption (463→0 errors)
2025-10-10 23:05:26 +02:00
portfolio_transformer.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
production.rs
🤖 Wave 33-2: 12 Parallel Agents - Massive Cleanup Complete
2025-10-01 21:48:25 +02:00
qat_metrics_exporter.rs
feat(ml): Final Stabilization Wave - 100% FP32 test pass rate, QAT infrastructure
2025-10-25 15:36:57 +02:00
random_model.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
real_data_loader.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
regime_detection.rs
🔧 Major compilation fixes across entire workspace - Significant progress achieved
2025-09-27 20:56:22 +02:00
tensor_ops.rs
🔧 Tonic 0.14 Upgrade: Auto-generated and build system changes
2025-10-03 07:34:26 +02:00
test_common.rs
🔧 Wave 33: Test Compilation Improvements - 57 errors remaining
2025-10-01 21:24:28 +02:00
test_fixtures.rs
🔧 Tonic 0.14 Upgrade: Auto-generated and build system changes
2025-10-03 07:34:26 +02:00
training_pipeline.rs
🎯 Wave 136: Compilation Warning Elimination - 97% Reduction
2025-10-11 18:39:19 +02:00
training.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
traits.rs
🔧 Wave 33: Test Compilation Improvements - 57 errors remaining
2025-10-01 21:24:28 +02:00
validation.rs
🔧 Wave 33: Test Compilation Improvements - 57 errors remaining
2025-10-01 21:24:28 +02:00