jgrusewski 7ba64b2ef7 feat(ml): MAMBA-2 device fix + PPO batch size optimization + CUDA 12.9 migration ...

Critical Fixes:
- MAMBA-2 device mismatch fixed (3 methods: train_batch, validate, calculate_accuracy)
- PPO batch size increased 64→512 (fixes explained variance -23.56→+0.58)
- CUDA 12.9 migration complete (Runpod driver 550 compatibility)

MAMBA-2 Device Fix (ml/src/mamba/mod.rs):
- Added .to_device(&self.device)? calls in train_batch (L1216-1219)
- Added device transfers in validate (L1829-1831)
- Added device transfers in calculate_accuracy (L1856-1858)
- Training validated: 2 epochs, 40.35s, 171,900 params

PPO Optimization (ml/src/ppo/ppo.rs, ml/examples/train_ppo.rs):
- Changed default mini_batch_size from 64 to 512
- Gradient variance reduction: 88%
- Explained variance improvement: -23.56 → +0.58
- Training time: 33.0s (10 epochs), stable convergence
- All 59 unit tests pass

CUDA 12.9 Migration:
- Dockerfile.runpod updated to CUDA 12.9.1 + cuDNN 9
- All 4 binaries rebuilt with CUDA 12.9 (75MB total)
- Uploaded to Runpod S3: s3://se3zdnb5o4/binaries/
- Compatible with Runpod driver 550 (CUDA 13.0 requires driver 580+)

Training Validations:
- DQN: ✅ 15s training
- MAMBA-2: ✅ 40.35s training (device fix validated)
- PPO: ✅ 33.0s training (batch size fix validated)
- TFT: ⚠️ Memory leak investigation ongoing (+1216MB growth)

Test Results:
- ML tests: 1,337/1,337 pass (100%)
- Workspace tests: 3,196/3,196 pass (100%)
- PPO unit tests: 59/59 pass (100%)

🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>

2025-10-26 11:14:33 +01:00

..

production

🚀 Wave 160 Phase 6: CUDA Mandatory + TDD Testing + TFT Complete (21 Agents)

2025-10-14 23:13:34 +02:00

staging

feat(wave-d): Complete Phase 6 agents G15-G19 - memory optimization + performance validation

2025-10-18 18:14:34 +02:00

test

🎯 Wave 159: Fix ML Training Infrastructure (22 Parallel Agents)

2025-10-14 09:06:37 +02:00

test_tft

🚀 Wave 160 Phase 5: Complete ML Ensemble + Production Deployment (27 Agents)

2025-10-14 18:41:48 +02:00

dqn_epoch_10.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

dqn_epoch_20.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

dqn_epoch_30.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

dqn_epoch_40.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

dqn_epoch_50.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

dqn_final_epoch1.safetensors

🚀 Wave 160 Phase 5: Complete ML Ensemble + Production Deployment (27 Agents)

2025-10-14 18:41:48 +02:00

dqn_final_epoch2.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

dqn_final_epoch3.safetensors

feat(wave1-2): Complete multi-model training architecture + TLI commands

2025-10-22 20:50:43 +02:00

dqn_final_epoch30.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

dqn_final_epoch100.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

ppo_actor_epoch_1.safetensors

feat(wave12): E2E validation complete - 225-feature pipeline ready

2025-10-22 22:48:04 +02:00

ppo_actor_epoch_3.safetensors

feat(wave1-2): Complete multi-model training architecture + TLI commands

2025-10-22 20:50:43 +02:00

ppo_actor_epoch_10.safetensors

feat(ml): MAMBA-2 device fix + PPO batch size optimization + CUDA 12.9 migration

2025-10-26 11:14:33 +01:00

ppo_actor_epoch_20.safetensors

feat(cleanup): Complete 30-agent codebase cleanup wave - 100% production ready

2025-10-23 09:16:58 +02:00

ppo_actor_epoch_30.safetensors

feat(ml): Fix TFT QAT device mismatch + MAMBA2 memory leak (33 agents)

2025-10-23 01:02:00 +02:00

ppo_actor_wave12_epoch30.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

ppo_checkpoint_epoch_1.safetensors

feat(wave12): E2E validation complete - 225-feature pipeline ready

2025-10-22 22:48:04 +02:00

ppo_checkpoint_epoch_3.safetensors

feat(wave1-2): Complete multi-model training architecture + TLI commands

2025-10-22 20:50:43 +02:00

ppo_checkpoint_epoch_10.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

ppo_checkpoint_epoch_20.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

ppo_checkpoint_epoch_30.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

ppo_critic_epoch_1.safetensors

feat(wave12): E2E validation complete - 225-feature pipeline ready

2025-10-22 22:48:04 +02:00

ppo_critic_epoch_3.safetensors

feat(wave1-2): Complete multi-model training architecture + TLI commands

2025-10-22 20:50:43 +02:00

ppo_critic_epoch_10.safetensors

feat(ml): MAMBA-2 device fix + PPO batch size optimization + CUDA 12.9 migration

2025-10-26 11:14:33 +01:00

ppo_critic_epoch_20.safetensors

feat(cleanup): Complete 30-agent codebase cleanup wave - 100% production ready

2025-10-23 09:16:58 +02:00

ppo_critic_epoch_30.safetensors

feat(ml): Fix TFT QAT device mismatch + MAMBA2 memory leak (33 agents)

2025-10-23 01:02:00 +02:00

ppo_critic_wave12_epoch30.safetensors

feat(wave12): Complete ML warning fixes and add Parquet training infrastructure

2025-10-21 08:54:26 +02:00

tft_225_epoch_0.json

feat(runpod): Add self-termination wrapper for pod auto-shutdown

2025-10-24 23:12:42 +02:00

tft_225_epoch_0.safetensors

feat(wave9-11): Complete 225-feature integration and service migration

2025-10-20 21:54:39 +02:00

training_results_20251013_161141.json

🎯 Wave 159: Fix ML Training Infrastructure (22 Parallel Agents)

2025-10-14 09:06:37 +02:00