aac0597cd2
- Fixed DQN early stopping checkpoint naming bug (Option B)
- Added is_final: bool parameter to checkpoint callback signature
- Trainer now distinguishes final checkpoints from regular epoch checkpoints
- Final checkpoints use 'dqn_final_epoch{N}' naming convention
- Regular checkpoints use 'dqn_epoch_{N}' naming convention
- Completed comprehensive TFT OOM investigation
- Spawned 3 parallel agents for memory analysis
- Identified 16.4GB memory leak (29.7x over expected 525-550MB)
- Root causes: Attention cache bloat (960MB), gradient accumulation bug, detached tensors
- Recommended fixes: Disable cache during training, explicit tensor drops
- Created TFT_MEMORY_ANALYSIS.md, TFT_MEMORY_LEAK_ANALYSIS.md
- DQN 100-epoch training VERIFIED on Runpod RTX A4000
- Training completed successfully: 100/100 epochs
- Final checkpoint created: dqn_final_epoch100.safetensors
- Training speed: 4.8 sec/epoch (3.5x faster than baseline)
- Option B fix working perfectly
- Deployed RTX 4090 pod for TFT testing
- Pod ID: 6244yzm9hadnog
- 24GB VRAM to bypass OOM issue
- EUR-IS-1 datacenter, $0.59/hr
Files modified:
- ml/examples/train_dqn.rs (checkpoint callback signature)
- ml/src/trainers/dqn.rs (callback signature + is_final parameter)
- CLAUDE.md (compacted to ~11k chars)
Generated reports:
- TFT_MEMORY_ANALYSIS.md (15-section memory breakdown)
- TFT_MEMORY_QUICK_SUMMARY.md (executive summary)
- TFT_MEMORY_LEAK_ANALYSIS.md (5 critical leaks identified)
Co-Authored-By: Claude <noreply@anthropic.com>
42 lines
1.4 KiB
JSON
42 lines
1.4 KiB
JSON
{
|
|
"deployment_date": "2025-10-25T18:02:42Z",
|
|
"git_commit": "caf36b41381a1698994bdefd8f449fa94c07ca9d",
|
|
"binaries": [
|
|
{
|
|
"name": "train_dqn",
|
|
"size": 20857232,
|
|
"sha256": "fedc57eacf7e375a809be3fa1303d72476a3885a664c2fbb76e15d3dba95d794",
|
|
"s3_path": "s3://se3zdnb5o4/binaries/train_dqn"
|
|
},
|
|
{
|
|
"name": "train_ppo",
|
|
"size": 13098968,
|
|
"sha256": "257dd241ec11a7940d113adbeb56a2f1747718a84d404b8de9817425eef6c4b3",
|
|
"s3_path": "s3://se3zdnb5o4/binaries/train_ppo"
|
|
},
|
|
{
|
|
"name": "train_mamba2_dbn",
|
|
"size": 13952664,
|
|
"sha256": "460520295160bebd225b8cab0d2dcf6bb59bcd97cdba20a4c08c977941e35e25",
|
|
"s3_path": "s3://se3zdnb5o4/binaries/train_mamba2_dbn"
|
|
},
|
|
{
|
|
"name": "train_mamba2_parquet",
|
|
"size": 20681416,
|
|
"sha256": "acf322bfdc091833c6089ef69d829d331bc2c524091f9d816730a6320b3c5f89",
|
|
"s3_path": "s3://se3zdnb5o4/binaries/train_mamba2_parquet"
|
|
},
|
|
{
|
|
"name": "train_tft_parquet",
|
|
"size": 21603008,
|
|
"sha256": "23d24ee32ea1cde61e549698a647a7cca25fb3ff71ef28686b438f2dffbfce0d",
|
|
"s3_path": "s3://se3zdnb5o4/binaries/train_tft_parquet"
|
|
}
|
|
],
|
|
"test_pass_rate": "100% (1,337/1,337 ML tests, 3,196/3,196 workspace tests)",
|
|
"production_status": "CERTIFIED",
|
|
"cuda_support": true,
|
|
"models": ["DQN", "PPO", "MAMBA-2", "TFT-FP32"],
|
|
"features": 225
|
|
}
|