jgrusewski 92e9181dc4 feat(ml): Fix TFT QAT device mismatch + MAMBA2 memory leak (33 agents) ...

Critical Fixes Applied:
- TFT QAT device mismatch (3 bugs): Fixed CPU/CUDA tensor operations in qat.rs and qat_tft.rs
- QAT integration wiring: Created TFTModel trait, QAT wrapper now functional
- MAMBA2 750MB memory leak: Eliminated Vec accumulation (80% reduction)
- Tensor clone optimization: 28.6% reduction (28→20 clones)
- OOM handling: Auto-retry with batch size halving
- SSM state management: Epoch-level clearing added
- GPU memory profiling: Leak detection every 100 batches
- Device consistency tests: Validate QAT device handling
- DQN/PPO regression fixes: Tensor rank bugs resolved

Performance Improvements:
- TFT training: 2.1× faster expected (75s→35s/epoch)
- MAMBA2 memory: 80% reduction (1,757MB→350MB @ epoch 50)
- GPU memory budget: 46% reduction (815MB→440MB)
- Test pass rate: 99.22% (1,278/1,288)

Documentation:
- FINAL_DEPLOYMENT_SUMMARY.md: Comprehensive deployment summary
- RUNPOD_DEPLOYMENT_READY.md: Complete setup guide (8,400+ lines)
- FIX_SUMMARY_WAVE_TFT_MAMBA2.md: Technical fix details (642 lines)
- RUST_TENSOR_MEMORY_PATTERNS.md: Memory best practices (400+ lines)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-10-23 01:02:00 +02:00

..

production

🚀 Wave 160 Phase 6: CUDA Mandatory + TDD Testing + TFT Complete (21 Agents)

2025-10-14 23:13:34 +02:00

staging

feat(wave-d): Complete Phase 6 agents G15-G19 - memory optimization + performance validation

2025-10-18 18:14:34 +02:00

test

🎯 Wave 159: Fix ML Training Infrastructure (22 Parallel Agents)

2025-10-14 09:06:37 +02:00

test_tft

🚀 Wave 160 Phase 5: Complete ML Ensemble + Production Deployment (27 Agents)

2025-10-14 18:41:48 +02:00

dqn_epoch_10.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

dqn_epoch_20.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

dqn_epoch_30.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

dqn_epoch_40.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

dqn_epoch_50.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

dqn_final_epoch1.safetensors

🚀 Wave 160 Phase 5: Complete ML Ensemble + Production Deployment (27 Agents)

2025-10-14 18:41:48 +02:00

dqn_final_epoch2.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

dqn_final_epoch3.safetensors

feat(wave1-2): Complete multi-model training architecture + TLI commands

2025-10-22 20:50:43 +02:00

dqn_final_epoch30.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

dqn_final_epoch100.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

ppo_actor_epoch_1.safetensors

feat(wave12): E2E validation complete - 225-feature pipeline ready

2025-10-22 22:48:04 +02:00

ppo_actor_epoch_3.safetensors

feat(wave1-2): Complete multi-model training architecture + TLI commands

2025-10-22 20:50:43 +02:00

ppo_actor_epoch_10.safetensors

feat(ml): Fix TFT QAT device mismatch + MAMBA2 memory leak (33 agents)

2025-10-23 01:02:00 +02:00

ppo_actor_epoch_20.safetensors

feat(ml): Fix TFT QAT device mismatch + MAMBA2 memory leak (33 agents)

2025-10-23 01:02:00 +02:00

ppo_actor_epoch_30.safetensors

feat(ml): Fix TFT QAT device mismatch + MAMBA2 memory leak (33 agents)

2025-10-23 01:02:00 +02:00

ppo_actor_wave12_epoch30.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

ppo_checkpoint_epoch_1.safetensors

feat(wave12): E2E validation complete - 225-feature pipeline ready

2025-10-22 22:48:04 +02:00

ppo_checkpoint_epoch_3.safetensors

feat(wave1-2): Complete multi-model training architecture + TLI commands

2025-10-22 20:50:43 +02:00

ppo_checkpoint_epoch_10.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

ppo_checkpoint_epoch_20.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

ppo_checkpoint_epoch_30.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

ppo_critic_epoch_1.safetensors

feat(wave12): E2E validation complete - 225-feature pipeline ready

2025-10-22 22:48:04 +02:00

ppo_critic_epoch_3.safetensors

feat(wave1-2): Complete multi-model training architecture + TLI commands

2025-10-22 20:50:43 +02:00

ppo_critic_epoch_10.safetensors

feat(ml): Fix TFT QAT device mismatch + MAMBA2 memory leak (33 agents)

2025-10-23 01:02:00 +02:00

ppo_critic_epoch_20.safetensors

feat(ml): Fix TFT QAT device mismatch + MAMBA2 memory leak (33 agents)

2025-10-23 01:02:00 +02:00

ppo_critic_epoch_30.safetensors

feat(ml): Fix TFT QAT device mismatch + MAMBA2 memory leak (33 agents)

2025-10-23 01:02:00 +02:00

ppo_critic_wave12_epoch30.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

tft_225_epoch_0.json

feat(ml): Fix TFT QAT device mismatch + MAMBA2 memory leak (33 agents)

2025-10-23 01:02:00 +02:00

tft_225_epoch_0.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

training_results_20251013_161141.json

🎯 Wave 159: Fix ML Training Infrastructure (22 Parallel Agents)

2025-10-14 09:06:37 +02:00