gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
bbd59e386f9804630de6db23e8435c93eeb01743
foxhunt/ml/examples
History
jgrusewski bbd59e386f fix(ml): Aggressive validation cache clearing + CLI defaults 
Final Memory Leak Fixes:

1. Aggressive Cache Clearing (every batch)
   - Changed from every 10 batches to EVERY batch in validation loop
   - Prevents any cache accumulation during validation
   - Impact: <100MB validation memory usage (was 2500MB+ OOM)
   - Trade-off: ~2-5% slower validation (acceptable for stability)

2. CLI Parser Dynamic Defaults
   - validation_batch_size: hardcoded '32' → Option<usize>
   - Automatically defaults to match training batch_size
   - Users can run --batch-size 1 without --validation-batch-size 1

Files Modified:
- ml/src/trainers/tft.rs (cache clearing every batch)
- ml/examples/train_tft_parquet.rs (CLI Option<usize> default)

Expected Result:
- Small dataset (batch_size=1): 5/5 epochs without OOM
- Validation: Stable memory <200MB throughout
- Development ready for small dataset testing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-26 20:40:11 +01:00
..
ab_test_demonstration.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
adaptive_ml_backtest.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
analyze_dqn_checkpoints.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
backtest_ensemble.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
benchmark_cuda_speedup.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
benchmark_future_decoder.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
benchmark_ppo_optimization.rs
feat(ml): Final Stabilization Wave - 100% FP32 test pass rate, QAT infrastructure
2025-10-25 15:36:57 +02:00
benchmark_streaming_vs_batch.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
benchmark_weight_caching.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
check_feature_count.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
check_performance_regression.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
check_tft_weight_init.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
comprehensive_model_backtest.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
convert_6e_dbn_to_parquet.rs
feat(wave12): Complete ML warning fixes and add Parquet training infrastructure
2025-10-21 08:54:26 +02:00
convert_6e_parquet_simple.rs
feat(wave12): Complete ML warning fixes and add Parquet training infrastructure
2025-10-21 08:54:26 +02:00
create_small_parquet_files.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
cross_validation_backtest.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
cuda_test.rs
🔧 Tonic 0.14 Upgrade: Auto-generated and build system changes
2025-10-03 07:34:26 +02:00
download_l2_data.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
download_l2_test.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
download_training_data.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
ensemble_visualization.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
feature_importance_analysis.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
generate_calibration_dataset.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
gpu_memory_benchmark.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
gpu_memory_monitor.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
gpu_training_benchmark.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
inference_benchmark.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
MAMBA2_DBN_INTEGRATION.md
🚀 Wave 160 Phase 2: ML Training Infrastructure + TLOB Investigation
2025-10-14 10:42:56 +02:00
measure_dqn_memory.rs
feat(cleanup): Complete 30-agent codebase cleanup wave - 100% production ready
2025-10-23 09:16:58 +02:00
model_diversity_analysis.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
model_registry_api.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
optimize_barriers.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
optimize_batch_sizes.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
optimize_ensemble_weights.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
profile_model_memory.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
profile_tft_int8_memory.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
profile_training_memory_180d.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
quantize_tft_varmap.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
quantized_checkpoint_demo.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
quick_checkpoint_analysis.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
quick_performance_benchmark.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
README_MAMBA2_PARQUET.md
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
real_time_inference_benchmark.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
register_trained_models.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
retrain_all_models.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
six_model_ensemble.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
test_adaptive_regime_detection.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
test_cuda_basic.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
test_dbn_loading.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
test_dbn_prices.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
test_ensemble.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
test_future_decoder.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
test_gpu_hardware.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
test_gradient_checkpointing.rs
feat(ml): 25-agent cleanup wave - QAT fixes + clippy + tests (Agents 1-25)
2025-10-23 10:43:52 +02:00
test_memory_optimization.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
test_parquet_schema.rs
feat(wave12): Complete ML warning fixes and add Parquet training infrastructure
2025-10-21 08:54:26 +02:00
test_symmetric_quantization.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
test_tft_fp32_varmap.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
tft_int8_calibration_simple.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
tft_int8_calibration.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
train_dqn_es_fut.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
train_dqn.rs
fix(ml): DQN early stopping checkpoint naming (Option B)
2025-10-25 23:19:40 +02:00
train_liquid_dbn.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
train_mamba2_dbn.rs
feat(wave12): Complete ML warning fixes and add Parquet training infrastructure
2025-10-21 08:54:26 +02:00
train_mamba2_parquet.rs
feat(ml): Final Stabilization Wave - 100% FP32 test pass rate, QAT infrastructure
2025-10-25 15:36:57 +02:00
train_mamba2.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
train_ppo_es_fut.rs
feat(ml): Final Stabilization Wave - 100% FP32 test pass rate, QAT infrastructure
2025-10-25 15:36:57 +02:00
train_ppo_extended.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
train_ppo_parquet.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
train_ppo.rs
feat(ml): MAMBA-2 device fix + PPO batch size optimization + CUDA 12.9 migration
2025-10-26 11:14:33 +01:00
train_tft_dbn.rs
feat(ml): Implement Quantization-Aware Training (QAT) for TFT model
2025-10-21 21:13:11 +02:00
train_tft_parquet.rs
fix(ml): Aggressive validation cache clearing + CLI defaults
2025-10-26 20:40:11 +01:00
train_tft_qat.rs
feat(ml): Implement Quantization-Aware Training (QAT) for TFT model
2025-10-21 21:13:11 +02:00
train_tft.rs
feat(ml): Implement Quantization-Aware Training (QAT) for TFT model
2025-10-21 21:13:11 +02:00
train_tlob.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
tune_hyperparameters.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
validate_225_features_databento.rs
feat(wave12): Complete ML warning fixes and add Parquet training infrastructure
2025-10-21 08:54:26 +02:00
validate_225_features_runtime.rs
feat(wave9-11): Complete 225-feature integration and service migration
2025-10-20 21:54:39 +02:00
validate_checkpoints.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
validate_databento_files.rs
feat(wave12): Complete ML warning fixes and add Parquet training infrastructure
2025-10-21 08:54:26 +02:00
validate_dqn_225_features.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
validate_dqn_225_simple.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
validate_dqn_real_training.rs
feat(wave12): Complete ML warning fixes and add Parquet training infrastructure
2025-10-21 08:54:26 +02:00
validate_dqn_simple.rs
feat(wave12): Complete ML warning fixes and add Parquet training infrastructure
2025-10-21 08:54:26 +02:00
validate_features_1_50.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
validate_features_151_200.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
validate_per_channel_quantization.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
validate_ppo_checkpoints.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
validate_quantile_loss.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
validate_regime_features.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
validate_tft_int8_accuracy.rs
feat(wave1-2): Complete multi-model training architecture + TLI commands
2025-10-22 20:50:43 +02:00
validate_wave_c_features_51_150.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
verify_225_feature_values.rs
feat(wave9-11): Complete 225-feature integration and service migration
2025-10-20 21:54:39 +02:00
verify_225_features_wave9.rs
feat(wave9-11): Complete 225-feature integration and service migration
2025-10-20 21:54:39 +02:00
verify_dbn_loader_zero_free.rs
feat(wave9-11): Complete 225-feature integration and service migration
2025-10-20 21:54:39 +02:00
verify_feature_dims.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
verify_grn_weight_init.rs
feat(wave-d): Complete Wave D Phase 6 with 240+ parallel agents
2025-10-19 09:10:55 +02:00
verify_mamba2_dimensions.rs
feat(wave9-11): Complete 225-feature integration and service migration
2025-10-20 21:54:39 +02:00
wave_c_backtest.rs
feat(wave9-11): Complete 225-feature integration and service migration
2025-10-20 21:54:39 +02:00
wave_comparison_full_features.rs
feat(wave9-11): Complete 225-feature integration and service migration
2025-10-20 21:54:39 +02:00
wave_comparison_simple.rs
feat(wave9-11): Complete 225-feature integration and service migration
2025-10-20 21:54:39 +02:00
wave_d_backtest.rs
feat(wave9-11): Complete 225-feature integration and service migration
2025-10-20 21:54:39 +02:00