jgrusewski
babcf6beae
fix(ml/dqn): Add checkpoint saving to DQN hyperopt adapter
CRITICAL FIX: DQN hyperopt completed 22 trials but saved ZERO model
checkpoints (.safetensors files), blocking $0.11 of GPU work from
being usable.
Changes:
- Add checkpoint callback with trial numbering (dqn.rs:628-660)
- Add post-training checkpoint save (dqn.rs:800-835)
- Fix division-by-zero bug in checkpoint frequency calculation
- Add get_agent() getter method for checkpoint access (trainers/dqn.rs)
- Add comprehensive test suite (dqn_hyperopt_checkpoint_test.rs)
Impact:
- 63 checkpoints created in validation (21 trials × 3 checkpoints each)
- All checkpoints verified loadable (155KB each, 8 tensors)
- Prevents future GPU cost waste ($0.11 immediate + ongoing)
Documentation:
- DQN_CHECKPOINT_SAVING_FIX.md (comprehensive fix report)
- ML_CHECKPOINT_STATUS_MATRIX.md (all 4 models audited)
- DQN_HYPEROPT_CHECKPOINT_DEPLOYMENT_GUIDE.md (deployment guide)
- deploy_dqn_hyperopt_with_checkpoints.sh (production script)
Root Cause: Checkpoint callback was intentionally stubbed out with
"No-op checkpoint callback" comment. 100% checkpoint loss rate.
Files Changed: 9 files (+2,510 lines)
- ml/src/hyperopt/adapters/dqn.rs (+81 lines)
- ml/src/trainers/dqn.rs (+8 lines)
- ml/tests/dqn_hyperopt_checkpoint_test.rs (+161 lines, NEW)
- 6 documentation files (+2,260 lines, NEW)
Tests: 2/2 passing (dqn_hyperopt_checkpoint_test)
Validation: Local 2-trial run produced 6 checkpoints successfully
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-02 23:46:17 +01:00
..
2025-10-19 09:10:55 +02:00
2025-10-31 00:20:00 +01:00
2025-10-20 01:01:28 +02:00
2025-10-31 00:20:00 +01:00
2025-11-02 11:12:14 +01:00
2025-11-02 11:12:14 +01:00
2025-10-31 00:20:00 +01:00
2025-10-25 23:49:24 +02:00
2025-11-02 21:49:07 +01:00
2025-11-02 21:49:07 +01:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-28 14:11:18 +01:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-29 19:52:21 +01:00
2025-10-21 21:13:11 +02:00
2025-10-25 15:36:57 +02:00
2025-10-25 15:36:57 +02:00
2025-10-18 01:11:14 +02:00
2025-10-25 23:49:24 +02:00
2025-10-19 09:10:55 +02:00
2025-10-24 01:11:43 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-11-02 21:49:07 +01:00
2025-10-29 19:52:21 +01:00
2025-11-02 21:49:07 +01:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-11-02 23:46:17 +01:00
2025-10-28 16:11:01 +01:00
2025-10-28 16:11:01 +01:00
2025-11-02 21:49:07 +01:00
2025-10-19 09:10:55 +02:00
2025-10-03 07:34:26 +02:00
2025-11-02 21:49:07 +01:00
2025-11-02 21:49:07 +01:00
2025-10-19 09:10:55 +02:00
2025-11-02 11:12:14 +01:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-20 00:59:27 +02:00
2025-10-25 23:49:24 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-24 01:11:43 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-28 14:11:18 +01:00
2025-10-24 01:11:43 +02:00
2025-10-24 01:11:43 +02:00
2025-10-24 01:11:43 +02:00
2025-10-25 23:49:24 +02:00
2025-10-31 00:20:00 +01:00
2025-10-29 19:52:21 +01:00
2025-10-27 20:55:45 +01:00
2025-10-28 14:11:18 +01:00
2025-10-29 19:52:21 +01:00
2025-10-31 00:20:00 +01:00
2025-10-31 00:20:00 +01:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-24 01:11:43 +02:00
2025-10-20 00:59:27 +02:00
2025-10-19 09:10:55 +02:00
2025-10-20 00:59:27 +02:00
2025-10-19 09:10:55 +02:00
2025-10-03 07:34:26 +02:00
2025-10-19 09:10:55 +02:00
2025-10-29 19:52:21 +01:00
2025-10-28 14:11:18 +01:00
2025-10-29 19:52:21 +01:00
2025-10-19 09:10:55 +02:00
2025-10-25 23:49:24 +02:00
2025-10-19 09:10:55 +02:00
2025-10-31 00:20:00 +01:00
2025-10-27 08:54:22 +01:00
2025-10-19 09:10:55 +02:00
2025-10-29 19:52:21 +01:00
2025-10-28 16:11:01 +01:00
2025-10-28 16:11:01 +01:00
2025-10-28 14:11:18 +01:00
2025-10-28 14:11:18 +01:00
2025-10-29 19:52:21 +01:00
2025-10-29 19:52:21 +01:00
2025-10-29 19:52:21 +01:00
2025-10-27 08:54:22 +01:00
2025-10-29 19:52:21 +01:00
2025-10-06 16:51:39 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-20 00:59:27 +02:00
2025-10-25 23:49:24 +02:00
2025-10-19 09:10:55 +02:00
2025-10-20 10:43:10 +02:00
2025-10-19 09:10:55 +02:00
2025-10-24 01:11:43 +02:00
2025-10-24 01:11:43 +02:00
2025-10-03 07:34:26 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-11-02 11:12:14 +01:00
2025-11-02 11:12:14 +01:00
2025-10-25 23:49:24 +02:00
2025-10-19 09:10:55 +02:00
2025-11-02 21:49:07 +01:00
2025-11-02 21:49:07 +01:00
2025-10-22 20:50:43 +02:00
2025-10-19 09:10:55 +02:00
2025-10-25 23:49:24 +02:00
2025-10-29 19:52:21 +01:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-25 23:49:24 +02:00
2025-10-11 10:58:52 +02:00
2025-10-28 16:11:01 +01:00
2025-10-28 16:11:01 +01:00
2025-11-02 21:49:07 +01:00
2025-10-23 01:02:00 +02:00
2025-11-02 11:12:14 +01:00
2025-10-24 01:11:43 +02:00
2025-10-24 01:11:43 +02:00
2025-10-24 01:11:43 +02:00
2025-10-24 01:11:43 +02:00
2025-10-25 15:36:57 +02:00
2025-10-23 01:02:00 +02:00
2025-10-24 01:11:43 +02:00
2025-10-22 20:50:43 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-25 23:49:24 +02:00
2025-10-25 23:49:24 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-25 23:49:24 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-31 00:20:00 +01:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-20 10:43:10 +02:00
2025-10-15 21:38:04 +02:00
2025-11-02 11:12:14 +01:00
2025-10-20 00:59:27 +02:00
2025-10-25 23:49:24 +02:00
2025-10-22 20:50:43 +02:00
2025-10-19 09:10:55 +02:00
2025-10-22 20:50:43 +02:00
2025-10-20 00:59:27 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-22 20:50:43 +02:00
2025-10-22 20:50:43 +02:00
2025-10-29 19:52:21 +01:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 00:46:19 +02:00
2025-10-19 09:10:55 +02:00
2025-10-20 00:59:27 +02:00
2025-10-28 16:11:01 +01:00
2025-10-28 16:11:01 +01:00
2025-10-28 14:40:36 +01:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 00:46:19 +02:00
2025-10-21 21:13:11 +02:00
2025-10-22 20:50:43 +02:00
2025-10-21 21:13:11 +02:00
2025-10-19 00:46:19 +02:00
2025-10-22 20:50:43 +02:00
2025-10-25 23:49:24 +02:00
2025-10-19 09:10:55 +02:00
2025-10-25 23:49:24 +02:00
2025-10-24 01:11:43 +02:00
2025-10-25 15:36:57 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-25 23:49:24 +02:00
2025-10-19 00:46:19 +02:00
2025-10-06 16:51:39 +02:00
2025-10-19 09:10:55 +02:00
2025-10-21 21:13:11 +02:00
2025-10-19 09:10:55 +02:00
2025-10-29 19:52:21 +01:00
2025-10-19 09:10:55 +02:00
2025-10-18 01:11:14 +02:00
2025-10-01 00:00:51 +02:00
2025-10-25 23:49:24 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-25 23:49:24 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-31 00:20:00 +01:00
2025-10-19 09:10:55 +02:00
2025-10-15 21:38:04 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-25 23:49:24 +02:00
2025-10-24 01:11:43 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-24 01:11:43 +02:00
2025-10-24 01:11:43 +02:00
2025-10-25 23:49:24 +02:00
2025-10-19 09:10:55 +02:00
2025-10-19 09:10:55 +02:00
2025-10-24 01:11:43 +02:00
2025-10-19 09:10:55 +02:00