feat(ml): WAVE 30 - Implement optimized DQN logging module
Add comprehensive logging utilities for DQN training with best practices: - LoggingConfig: Configurable log levels, intervals, and sampling rates - MetricsAggregator: Windowed statistics (mean, std_dev) for training metrics - SampledLogger: Rate-limited logging for high-frequency events Key features: - Structured logging with tracing crate (info/debug/trace hierarchy) - 23 unit tests for full coverage - Integration with existing DQN training pipeline Bug fixes: - Fix u8 overflow in prioritized_replay.rs test (500 > u8::MAX) - Fix GradStore assertion in residual.rs (no is_empty method) - Fix Tensor::get() Option/Result handling in quantile_regression.rs - Fix Device PartialEq comparison in ensemble_network.rs Documentation: - Add Rust logging best practices guide for ML training - Add DQN logging analysis and design summary 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jgrusewski
609
docs/codebase-cleanup/dqn-logging-analysis.md
Normal file
609
docs/codebase-cleanup/dqn-logging-analysis.md
Normal file
@@ -0,0 +1,609 @@
|
||||
# DQN Trainer Logging Pattern Analysis
|
||||
|
||||
**Date:** 2025-11-27
|
||||
**Module:** `ml/src/trainers/dqn/`
|
||||
**Total Lines in trainer.rs:** 4,658
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The DQN trainer module uses **tracing** crate exclusively with **169 total logging statements** across trainer.rs (98 info!, 33 debug!, 32 warn!, 5 tracing::, 1 error!). The logging is generally well-structured but suffers from:
|
||||
|
||||
1. **High-frequency training loop logs** (every epoch) that could impact performance
|
||||
2. **Inconsistent log levels** for similar operations
|
||||
3. **Missing context** in some error logs
|
||||
4. **Excessive initialization verbosity** (20+ info! logs during setup)
|
||||
5. **No structured logging fields** (all string interpolation)
|
||||
|
||||
---
|
||||
|
||||
## 1. Logging Crate Usage
|
||||
|
||||
### Crate: `tracing` (Consistent)
|
||||
```rust
|
||||
// File: ml/src/trainers/dqn/trainer.rs:19
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
// File: ml/src/trainers/dqn/early_stopping.rs:8
|
||||
use tracing::{debug, info};
|
||||
```
|
||||
|
||||
**✅ Good:** Single logging crate used consistently across all modules
|
||||
**⚠️ Issue:** No structured fields used (all string interpolation)
|
||||
|
||||
### Logging Statement Frequency
|
||||
```
|
||||
info! : 98 (58%) - Configuration, epoch summaries, checkpoint saves
|
||||
debug! : 33 (20%) - Q-values, action distributions, preprocessing details
|
||||
warn! : 32 (19%) - Safety warnings, validation failures, degraded mode
|
||||
tracing::: 5 (3%) - Early stopping triggers (error! and info!)
|
||||
error! : 1 (<1%) - Critical failures only
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Categorized Logging Patterns
|
||||
|
||||
### 2.1 Training Loop Logs (High Frequency - Every Epoch)
|
||||
|
||||
**Location:** Lines 2129-2333 (204 lines of logging per epoch)
|
||||
|
||||
```rust
|
||||
// EVERY EPOCH (100+ epochs):
|
||||
info!("REWARD_STATS: epoch={}, mean={:.6}, std={:.6}, min={:.6}, max={:.6}, ...") // Line 2129
|
||||
info!("Epoch {}/{}: val_loss={:.6}", ...) // Line 2252
|
||||
info!("Epoch {}/{}: Action diversity={}/{} ({:.1}%)", ...) // Line 2271
|
||||
info!("WAVE P2 Episode Stats [Epoch {}]: {} episodes, ...") // Line 2296
|
||||
info!(" Exit breakdown: profit={}({:.1}%), stop={}({:.1}%), ...") // Line 2306
|
||||
info!("Epoch {}/{}: Risk Metrics - VaR(95%)={:.4}%, CVaR(95%)={:.4}%, ...") // Line 2324
|
||||
```
|
||||
|
||||
**⚠️ PROBLEM:** 6+ info! logs per epoch × 100 epochs = 600+ log lines
|
||||
**Impact:** Log file bloat, potential I/O bottleneck on slow filesystems
|
||||
**Recommendation:** Move to `debug!` or reduce frequency (every 10 epochs)
|
||||
|
||||
---
|
||||
|
||||
### 2.2 Checkpoint/Save Logs
|
||||
|
||||
**Good Pattern (Appropriate Level):**
|
||||
```rust
|
||||
// Line 2394
|
||||
info!("Best model saved to: {}", best_checkpoint_path);
|
||||
|
||||
// Line 2383 (debug - correct level)
|
||||
debug!("SAFETY: Checkpoint verification passed ({} bytes)", checkpoint_data.len());
|
||||
```
|
||||
|
||||
**Missing Context:**
|
||||
```rust
|
||||
// Line 2410-2416 - Missing epoch number in error log
|
||||
warn!("⚠️ Failed to save best checkpoint: {}", e);
|
||||
// Should be: warn!("⚠️ Failed to save best checkpoint at epoch {}: {}", epoch, e);
|
||||
```
|
||||
|
||||
**Files:** trainer.rs:2354-2466
|
||||
|
||||
---
|
||||
|
||||
### 2.3 Error Handling Logs
|
||||
|
||||
**Inconsistent Levels for Similar Failures:**
|
||||
|
||||
```rust
|
||||
// Line 1613 - Triple barrier failure (warn - correct)
|
||||
warn!("WAVE 1.1: Failed to start triple barrier tracking: {}", e);
|
||||
|
||||
// Line 2085 - Training step failure (warn - should be error!)
|
||||
warn!("Training step failed: {}, continuing...", e);
|
||||
|
||||
// Line 2080 - Early stopping (tracing::error - correct)
|
||||
tracing::error!("🛑 TERMINATING: Early stopping triggered - {}", e);
|
||||
```
|
||||
|
||||
**⚠️ PROBLEM:** Training step failures should be `error!`, not `warn!`
|
||||
|
||||
**Missing Stack Context:**
|
||||
```rust
|
||||
// Line 2163 - No epoch/step info
|
||||
warn!("SAFETY: Training stuck for {} epochs (loss variance: {:.6}, mean: {:.6})",
|
||||
self.safety_loss_plateau_counter, std_dev, mean);
|
||||
// Better: warn!("SAFETY: Training stuck at epoch {} for {} consecutive epochs (...)",
|
||||
// epoch, self.safety_loss_plateau_counter, ...);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2.4 Metric Reporting Logs
|
||||
|
||||
**Too Verbose (Every Epoch):**
|
||||
|
||||
```rust
|
||||
// Line 2271-2288 - Every epoch
|
||||
info!("Epoch {}/{}: Action diversity={}/{} ({:.1}%)", ...);
|
||||
if active_actions_count < DIVERSITY_THRESHOLD {
|
||||
warn!("⚠️ LOW ACTION DIVERSITY: {}/45 actions (<20%), consider increasing epsilon floor", ...);
|
||||
info!(" Recommendation: Increase epsilon_end from 0.05 to 0.10"); // Redundant
|
||||
info!(" Alternative: Add entropy regularization bonus"); // Redundant
|
||||
}
|
||||
```
|
||||
|
||||
**Recommendation:**
|
||||
- Move diversity logs to `debug!`
|
||||
- Only `warn!` when threshold breached (not every epoch)
|
||||
- Remove redundant "Recommendation" logs (should be in docs)
|
||||
|
||||
---
|
||||
|
||||
### 2.5 Debug/Diagnostic Logs
|
||||
|
||||
**Good Examples (Appropriate Level):**
|
||||
|
||||
```rust
|
||||
// Line 244-252 - Action distribution (debug - correct)
|
||||
debug!("Action Distribution [Epoch {}] - Top 5 Actions:", self.epoch);
|
||||
for (idx, count) in sorted_actions.iter().take(5) {
|
||||
debug!(" [{:2}] {:?}: {} ({:.1}%)", idx, action, count, pct);
|
||||
}
|
||||
|
||||
// Line 2794-2797 - Preprocessing stats (debug - correct)
|
||||
debug!("✅ Preprocessing complete:");
|
||||
debug!(" • Mean: {:.6} (expected ~0 for normalized data)", mean);
|
||||
debug!(" • Std: {:.4} (expected ~1 for normalized data)", std);
|
||||
```
|
||||
|
||||
**⚠️ Issue:** High-frequency debug in training loop
|
||||
```rust
|
||||
// Line 1607-1640 - EVERY STEP (200 steps/epoch × 100 epochs = 20,000 logs!)
|
||||
debug!("WAVE 1.1: Started triple barrier tracking at step {}, price=${:.2}, position={:.2}", ...);
|
||||
debug!("WAVE P2: Barrier-driven episode end at step {}: {:?}, label={}, ...", ...);
|
||||
```
|
||||
|
||||
**Recommendation:** Reduce frequency or use `trace!` level
|
||||
|
||||
---
|
||||
|
||||
### 2.6 Initialization Logs (Excessive Verbosity)
|
||||
|
||||
**Lines 590-730: 20+ info! logs during initialization**
|
||||
|
||||
```rust
|
||||
// Too many info! logs during setup
|
||||
info!("Creating regime-conditional DQN with 3 heads (Trending, Ranging, Volatile)");
|
||||
info!(" - Regime detection: ADX (index 211) + Entropy (index 219)");
|
||||
info!(" - Classification: Trending (ADX>25), Volatile (ADX≤25 & Entropy>0.7), ...");
|
||||
info!("Creating standard DQN with single Q-network head");
|
||||
info!("Triple barrier engine initialized with 1000 max trackers");
|
||||
info!("Kelly optimizer enabled (fractional={}, max={})", ...);
|
||||
info!("Entropy regularization enabled (coefficient=0.01)");
|
||||
info!("Stress testing enabled (8 scenarios)");
|
||||
info!("Action masking enabled (max_position=±{:.1}, 30-50% filtering expected)", ...);
|
||||
info!("Drawdown monitor enabled (thresholds: 10%, 12.5%, 15%)");
|
||||
info!("Position limiter enabled (abs=±10.0, notional=$1M, concentration=10%)");
|
||||
info!("Circuit breaker enabled (threshold=5 failures, cooldown=60s)");
|
||||
// ... 10 more similar logs ...
|
||||
```
|
||||
|
||||
**⚠️ PROBLEM:** Startup log spam (20+ lines)
|
||||
**Recommendation:** Consolidate into single summary log or use `debug!`
|
||||
|
||||
---
|
||||
|
||||
## 3. Problematic Patterns
|
||||
|
||||
### 3.1 High-Frequency Logs (Performance Risk)
|
||||
|
||||
| Location | Frequency | Current Level | Impact |
|
||||
|----------|-----------|---------------|--------|
|
||||
| Lines 2129-2333 | Every epoch (100+) | `info!` | Log file bloat |
|
||||
| Lines 1607-1640 | Every step (20K+) | `debug!` | I/O bottleneck |
|
||||
| Lines 244-273 | Every epoch | `debug!` | Acceptable |
|
||||
| Lines 2747-2749 | Once per dataset | `debug!` | Acceptable |
|
||||
|
||||
**Estimated Log Volume:**
|
||||
- **Training loop:** 6 info! × 100 epochs = 600 lines
|
||||
- **Per-step debug:** 2 debug! × 200 steps × 100 epochs = 40,000 lines (if enabled)
|
||||
- **Initialization:** 20+ info! lines
|
||||
|
||||
**Total:** ~40,600+ log lines for 100-epoch training run
|
||||
|
||||
---
|
||||
|
||||
### 3.2 Inconsistent Log Levels
|
||||
|
||||
| Issue Type | Current Level | Should Be | Location |
|
||||
|------------|---------------|-----------|----------|
|
||||
| Training step failure | `warn!` | `error!` | Line 2085 |
|
||||
| Triple barrier failure | `warn!` | `debug!` (non-critical) | Line 1613 |
|
||||
| Action diversity warning | `info!` + `warn!` | `debug!` (only warn on breach) | Lines 2271-2288 |
|
||||
| Initialization configs | `info!` | `debug!` | Lines 590-730 |
|
||||
|
||||
---
|
||||
|
||||
### 3.3 Missing Context in Error Logs
|
||||
|
||||
```rust
|
||||
// ❌ BAD - Missing epoch/step context
|
||||
warn!("Training step failed: {}, continuing...", e); // Line 2085
|
||||
|
||||
// ✅ GOOD - Has epoch context
|
||||
warn!("SAFETY: Training stuck for {} epochs (loss variance: {:.6}, mean: {:.6})",
|
||||
self.safety_loss_plateau_counter, std_dev, mean); // Line 2163
|
||||
```
|
||||
|
||||
**Common Missing Context:**
|
||||
- Current epoch number
|
||||
- Current step number
|
||||
- Model/checkpoint path
|
||||
- Hyperparameter values (learning rate, batch size, etc.)
|
||||
|
||||
---
|
||||
|
||||
### 3.4 No Structured Logging
|
||||
|
||||
**Current Pattern (String Interpolation):**
|
||||
```rust
|
||||
info!("Epoch {}/{}: val_loss={:.6}", epoch + 1, self.hyperparams.epochs, val_loss);
|
||||
```
|
||||
|
||||
**Recommended Pattern (Structured Fields):**
|
||||
```rust
|
||||
info!(
|
||||
epoch = epoch + 1,
|
||||
total_epochs = self.hyperparams.epochs,
|
||||
val_loss = %val_loss,
|
||||
"Epoch validation complete"
|
||||
);
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Machine-parseable logs (JSON output)
|
||||
- Easier filtering/aggregation
|
||||
- Better integration with observability tools (Datadog, Prometheus, etc.)
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommendations
|
||||
|
||||
### Priority 1: Reduce High-Frequency Logs
|
||||
|
||||
**Training Loop (Lines 2129-2333):**
|
||||
```rust
|
||||
// BEFORE (every epoch):
|
||||
info!("REWARD_STATS: epoch={}, mean={:.6}, ...", epoch + 1, ...);
|
||||
info!("Epoch {}/{}: Action diversity={}/{} ({:.1}%)", ...);
|
||||
|
||||
// AFTER (every 10 epochs OR debug level):
|
||||
if (epoch + 1) % 10 == 0 {
|
||||
info!("REWARD_STATS: epoch={}, mean={:.6}, ...", epoch + 1, ...);
|
||||
}
|
||||
debug!("Epoch {}/{}: Action diversity={}/{} ({:.1}%)", ...); // Always available if needed
|
||||
```
|
||||
|
||||
**Per-Step Logs (Lines 1607-1640):**
|
||||
```rust
|
||||
// BEFORE (every step):
|
||||
debug!("WAVE 1.1: Started triple barrier tracking at step {}, ...", i, ...);
|
||||
|
||||
// AFTER (use trace! or sample every 100 steps):
|
||||
trace!("WAVE 1.1: Started triple barrier tracking at step {}, ...", i, ...);
|
||||
// OR
|
||||
if i % 100 == 0 {
|
||||
debug!("WAVE 1.1: Triple barrier tracking at step {}, ...", i, ...);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Priority 2: Fix Inconsistent Log Levels
|
||||
|
||||
```rust
|
||||
// Training step failure (CRITICAL - should stop training)
|
||||
// Line 2085: warn! → error!
|
||||
error!("Training step failed at epoch {}: {}", epoch, e);
|
||||
|
||||
// Initialization configs (LOW PRIORITY - reduce verbosity)
|
||||
// Lines 590-730: info! → debug!
|
||||
debug!("Kelly optimizer enabled (fractional={}, max={})", ...);
|
||||
|
||||
// Action diversity warning (ONLY warn when threshold breached)
|
||||
// Lines 2283-2288: Remove redundant info! logs
|
||||
if active_actions_count < DIVERSITY_THRESHOLD {
|
||||
warn!("LOW ACTION DIVERSITY: {}/45 actions at epoch {} (<20%)",
|
||||
active_actions_count, epoch + 1);
|
||||
// Remove these:
|
||||
// info!(" Recommendation: Increase epsilon_end from 0.05 to 0.10");
|
||||
// info!(" Alternative: Add entropy regularization bonus");
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Priority 3: Add Missing Context
|
||||
|
||||
```rust
|
||||
// Add epoch/step context to all error logs
|
||||
warn!("Training step failed at epoch {}, step {}: {}", epoch, step, e);
|
||||
|
||||
// Add model path to checkpoint logs
|
||||
info!("Best model saved to: {} (epoch {}, val_loss={:.6})",
|
||||
best_checkpoint_path, epoch, val_loss);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Priority 4: Consolidate Initialization Logs
|
||||
|
||||
**BEFORE (20+ lines):**
|
||||
```rust
|
||||
info!("Creating regime-conditional DQN with 3 heads (Trending, Ranging, Volatile)");
|
||||
info!(" - Regime detection: ADX (index 211) + Entropy (index 219)");
|
||||
info!("Kelly optimizer enabled (fractional={}, max={})", ...);
|
||||
info!("Entropy regularization enabled (coefficient=0.01)");
|
||||
// ... 15 more lines ...
|
||||
```
|
||||
|
||||
**AFTER (3 lines):**
|
||||
```rust
|
||||
info!("DQN Trainer initialized with {} features", config.state_dim);
|
||||
info!(" Architecture: {} ({})",
|
||||
if regime_conditional { "Regime-Conditional (3 heads)" } else { "Standard" },
|
||||
config.hidden_dims.iter().map(|d| d.to_string()).collect::<Vec<_>>().join("-"));
|
||||
debug!("Advanced features: Kelly={}, Entropy={}, Action Masking={}, Stress Testing={}",
|
||||
kelly_enabled, entropy_enabled, action_masking_enabled, stress_testing_enabled);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Priority 5: Adopt Structured Logging
|
||||
|
||||
**Example Migration:**
|
||||
|
||||
```rust
|
||||
// BEFORE:
|
||||
info!("Epoch {}/{}: val_loss={:.6}", epoch + 1, self.hyperparams.epochs, val_loss);
|
||||
|
||||
// AFTER:
|
||||
info!(
|
||||
epoch = epoch + 1,
|
||||
total_epochs = self.hyperparams.epochs,
|
||||
val_loss = %val_loss,
|
||||
learning_rate = %self.current_lr,
|
||||
"Epoch validation complete"
|
||||
);
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Enables JSON logging: `RUST_LOG_FORMAT=json cargo run`
|
||||
- Better integration with observability platforms
|
||||
- Easier filtering: `grep -P '"epoch":\s*50' log.json`
|
||||
|
||||
---
|
||||
|
||||
## 5. File-by-File Summary
|
||||
|
||||
### `trainer.rs` (4,658 lines)
|
||||
|
||||
| Category | Count | Lines | Issues |
|
||||
|----------|-------|-------|--------|
|
||||
| Initialization | 20+ | 590-730 | Too verbose (info! → debug!) |
|
||||
| Training loop | 6/epoch | 2129-2333 | High frequency (info! → debug! or reduce) |
|
||||
| Per-step debug | 2/step | 1607-1640 | Very high frequency (debug! → trace!) |
|
||||
| Error handling | 32 | Various | Inconsistent levels, missing context |
|
||||
| Checkpoint | 8 | 2354-2466 | Good, but missing epoch in some errors |
|
||||
| Metrics | 15+ | 2271-2333 | Too verbose per epoch |
|
||||
|
||||
**Total Logging Statements:** 165
|
||||
|
||||
---
|
||||
|
||||
### `early_stopping.rs` (257 lines)
|
||||
|
||||
```rust
|
||||
// Line 86-89 (debug - correct)
|
||||
debug!("Early stopping: Improvement detected: {:.6} (val_loss: {:.6} -> {:.6})", ...);
|
||||
|
||||
// Line 99-102 (debug - correct)
|
||||
debug!("Early stopping: No improvement for {} epoch(s) (improvement: {:.6} < min_delta: {:.6})", ...);
|
||||
|
||||
// Line 105-109 (info - correct, only on trigger)
|
||||
info!("🛑 Early stopping triggered at epoch {}! No improvement for {} epochs (...)", ...);
|
||||
```
|
||||
|
||||
**✅ Good:** Appropriate log levels, context included, low frequency (only on trigger)
|
||||
|
||||
**Total Logging Statements:** 3
|
||||
|
||||
---
|
||||
|
||||
### `lr_scheduler.rs` (229 lines)
|
||||
|
||||
**Total Logging Statements:** 0 (no logs)
|
||||
|
||||
**✅ Good:** Utility module with no side effects, no logging needed
|
||||
|
||||
---
|
||||
|
||||
### `statistics.rs` (135 lines)
|
||||
|
||||
**Total Logging Statements:** 0 (no logs)
|
||||
|
||||
**✅ Good:** Pure data processing module, no logging needed
|
||||
|
||||
---
|
||||
|
||||
### `config.rs` (869 lines)
|
||||
|
||||
**Total Logging Statements:** 0 (no logs)
|
||||
|
||||
**✅ Good:** Configuration structs only, no logging needed
|
||||
|
||||
---
|
||||
|
||||
## 6. Performance Impact Estimation
|
||||
|
||||
### Current Logging Volume (100-epoch training)
|
||||
|
||||
| Log Type | Frequency | Total Lines | Avg Size | Total Size |
|
||||
|----------|-----------|-------------|----------|------------|
|
||||
| Initialization | 1× | 20 | 80 bytes | 1.6 KB |
|
||||
| Per-epoch info | 100× | 600 | 100 bytes | 60 KB |
|
||||
| Per-step debug | 20,000× | 20,000 | 120 bytes | 2.4 MB |
|
||||
| Error/warn | ~50 | 50 | 150 bytes | 7.5 KB |
|
||||
| **TOTAL** | | **20,670** | | **~2.47 MB** |
|
||||
|
||||
**With debug! enabled:** 2.47 MB log file for 100-epoch run
|
||||
**Production (info! only):** ~69 KB log file
|
||||
|
||||
---
|
||||
|
||||
### Recommended Logging Volume (After Optimization)
|
||||
|
||||
| Log Type | Frequency | Total Lines | Avg Size | Total Size |
|
||||
|----------|-----------|-------------|----------|------------|
|
||||
| Initialization | 1× | 3 | 100 bytes | 300 bytes |
|
||||
| Per-epoch info (every 10th) | 10× | 60 | 100 bytes | 6 KB |
|
||||
| Per-step trace (sampled) | 200× | 200 | 120 bytes | 24 KB |
|
||||
| Error/warn | ~50 | 50 | 150 bytes | 7.5 KB |
|
||||
| **TOTAL** | | **313** | | **~37.8 KB** |
|
||||
|
||||
**Reduction:** 20,670 → 313 lines (**98.5% reduction**)
|
||||
**Size Reduction:** 2.47 MB → 37.8 KB (**98.5% reduction**)
|
||||
|
||||
---
|
||||
|
||||
## 7. Actionable Next Steps
|
||||
|
||||
### Immediate (1-2 hours):
|
||||
1. Change initialization logs (lines 590-730) from `info!` → `debug!`
|
||||
2. Fix training step failure (line 2085) from `warn!` → `error!`
|
||||
3. Reduce per-epoch info logs to every 10th epoch
|
||||
|
||||
### Short-term (4-6 hours):
|
||||
4. Add epoch/step context to all error logs
|
||||
5. Consolidate initialization into 3-line summary
|
||||
6. Remove redundant recommendation logs (lines 2287-2288)
|
||||
|
||||
### Medium-term (1-2 days):
|
||||
7. Migrate to structured logging (tracing fields)
|
||||
8. Add sampling to per-step debug logs (1/100 steps)
|
||||
9. Implement log level configuration via environment variable
|
||||
|
||||
### Long-term (1 week):
|
||||
10. Add JSON log output support
|
||||
11. Integrate with observability platform (Datadog/Prometheus)
|
||||
12. Create log aggregation dashboard
|
||||
|
||||
---
|
||||
|
||||
## 8. Testing Checklist
|
||||
|
||||
Before/after comparison:
|
||||
|
||||
```bash
|
||||
# BEFORE optimization
|
||||
RUST_LOG=debug cargo run --bin dqn_train -- --epochs 100 2>&1 | wc -l
|
||||
# Expected: ~20,000 lines
|
||||
|
||||
# AFTER optimization
|
||||
RUST_LOG=debug cargo run --bin dqn_train -- --epochs 100 2>&1 | wc -l
|
||||
# Expected: ~300 lines
|
||||
|
||||
# Verify critical logs still present
|
||||
grep "Early stopping triggered" dqn_train.log # Should exist
|
||||
grep "Best model saved" dqn_train.log # Should exist
|
||||
grep "Training step failed" dqn_train.log # Should exist (if errors occur)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: Complete Logging Inventory
|
||||
|
||||
### trainer.rs Line References
|
||||
|
||||
**Initialization (lines 590-730):**
|
||||
- 590: info! regime-conditional DQN creation
|
||||
- 591-592: info! regime detection config
|
||||
- 597: info! standard DQN creation
|
||||
- 637: info! triple barrier engine
|
||||
- 652-653: info! Kelly optimizer
|
||||
- 666: info! entropy regularization
|
||||
- 680: info! stress testing
|
||||
- 687-692: info! action masking
|
||||
- 702: info! drawdown monitor
|
||||
- 717: info! position limiter
|
||||
- 730: info! circuit breaker
|
||||
- 745-747: info! dropout scheduler
|
||||
- 775-777: info! HER buffer
|
||||
- 786-788: info! GAE calculator
|
||||
- 796-798: info! noisy sigma scheduler
|
||||
- 955-963: info! training config summary
|
||||
- 1361-1369: info! target update mode
|
||||
- 1374-1406: info! Rainbow DQN components (10+ logs)
|
||||
|
||||
**Training Loop (per-epoch, lines 2129-2333):**
|
||||
- 2129-2139: info! REWARD_STATS
|
||||
- 2163-2166: warn! training plateau
|
||||
- 2189-2213: info! epoch progress bars
|
||||
- 2252-2257: info! validation loss
|
||||
- 2271-2278: info! action diversity
|
||||
- 2283-2288: warn! + 2× info! low diversity recommendations
|
||||
- 2296-2312: info! episode statistics (2 logs)
|
||||
- 2324-2333: info! risk metrics (VaR/CVaR)
|
||||
|
||||
**Per-step (lines 1607-1640):**
|
||||
- 1607-1610: debug! triple barrier start
|
||||
- 1640-1643: debug! barrier-driven episode end
|
||||
|
||||
**Checkpoints (lines 2354-2466):**
|
||||
- 2359-2363: info! new best validation loss
|
||||
- 2383: debug! checkpoint verification
|
||||
- 2394: info! best model saved
|
||||
- 2410: warn! checkpoint save failure
|
||||
- 2416: info! checkpoint saved successfully
|
||||
- 2429: info! best model loaded
|
||||
- 2447: warn! checkpoint load failure
|
||||
- 2451-2457: info! early stopping checkpoint
|
||||
|
||||
**Final Summary (lines 2506-2561):**
|
||||
- 2506-2521: info! training complete summary (5 logs)
|
||||
- 2544-2567: info! dataset loading (5 logs)
|
||||
|
||||
**Data Loading (lines 2592-3154):**
|
||||
- 2592-2906: info!/debug! Parquet loading (20+ logs)
|
||||
- 2943-3023: info!/debug! DBN loading (10+ logs)
|
||||
|
||||
**Training Steps (lines 3392-3702):**
|
||||
- 3392: debug! epsilon adjustment
|
||||
- 3529: tracing::info! early stopping (gradient collapse)
|
||||
- 3538-3556: warn! gradient anomalies
|
||||
- 3633: tracing::info! early stopping (gradient collapse)
|
||||
- 3639-3655: warn! gradient issues
|
||||
- 3702: tracing::info! early stopping (Q-value divergence)
|
||||
|
||||
**Utilities (lines 3852-4126):**
|
||||
- 3852: info! replay buffer cleared
|
||||
- 3862: info! target network reset
|
||||
- 4126: warn! OFI calculation failure
|
||||
|
||||
---
|
||||
|
||||
## Appendix B: Recommended Log Levels Guide
|
||||
|
||||
| Event Type | Current | Recommended | Rationale |
|
||||
|------------|---------|-------------|-----------|
|
||||
| **Initialization** | info! | debug! | One-time setup, reduces startup noise |
|
||||
| **Epoch summary** | info! | info! (every 10th) | Important milestones, but reduce frequency |
|
||||
| **Per-step training** | debug! | trace! | Too frequent, use sampling instead |
|
||||
| **Checkpoint save** | info! | info! | Critical events, keep as-is |
|
||||
| **Validation loss** | info! | info! (every 10th) | Important metrics, reduce frequency |
|
||||
| **Training failures** | warn! | error! | Should fail-fast, elevate severity |
|
||||
| **Safety warnings** | warn! | warn! | Correct level, add context |
|
||||
| **Triple barrier** | warn!/debug! | debug! | Non-critical, informational only |
|
||||
| **Early stopping** | tracing::error! | error! | Critical termination, correct level |
|
||||
| **Action diversity** | info! + warn! | debug! + warn! | Log all at debug, warn only on breach |
|
||||
|
||||
---
|
||||
|
||||
**End of Report**
|
||||
192
docs/codebase-cleanup/dqn-logging-summary.txt
Normal file
192
docs/codebase-cleanup/dqn-logging-summary.txt
Normal file
@@ -0,0 +1,192 @@
|
||||
DQN Trainer Logging Analysis - Quick Summary
|
||||
==============================================
|
||||
|
||||
MODULE: ml/src/trainers/dqn/
|
||||
DATE: 2025-11-27
|
||||
ANALYZER: Code Quality Analyzer
|
||||
|
||||
OVERVIEW
|
||||
--------
|
||||
- Logging Crate: tracing (consistent ✓)
|
||||
- Total Statements: 169 (98 info!, 33 debug!, 32 warn!, 5 tracing::, 1 error!)
|
||||
- File Size: 4,658 lines (trainer.rs)
|
||||
- Log Volume: ~20,670 lines for 100-epoch run (2.47 MB with debug!)
|
||||
|
||||
KEY FINDINGS
|
||||
------------
|
||||
|
||||
✓ GOOD PATTERNS:
|
||||
- Consistent use of tracing crate
|
||||
- Appropriate levels in early_stopping.rs
|
||||
- Good checkpoint save/load logging
|
||||
- Error context in some critical paths
|
||||
|
||||
✗ PROBLEMATIC PATTERNS:
|
||||
1. High-frequency logs (6 info! per epoch = 600 lines for 100 epochs)
|
||||
2. Excessive initialization verbosity (20+ info! logs at startup)
|
||||
3. Per-step debug logs (20,000+ lines if debug! enabled)
|
||||
4. Inconsistent error levels (training failures as warn! instead of error!)
|
||||
5. Missing context in error logs (no epoch/step numbers)
|
||||
6. No structured logging (all string interpolation)
|
||||
|
||||
CRITICAL ISSUES
|
||||
---------------
|
||||
|
||||
Priority 1: High-Frequency Training Loop Logs
|
||||
Location: Lines 2129-2333
|
||||
Problem: 6 info! logs EVERY epoch (100+ epochs)
|
||||
Impact: 600+ log lines, I/O bottleneck
|
||||
Fix: Move to debug! OR log every 10th epoch only
|
||||
|
||||
Priority 2: Inconsistent Error Levels
|
||||
Location: Line 2085
|
||||
Problem: "Training step failed" logged as warn! (should be error!)
|
||||
Impact: Critical failures not visible in error-only logs
|
||||
Fix: Change to error!("Training step failed at epoch {}: {}", epoch, e)
|
||||
|
||||
Priority 3: Excessive Initialization Logs
|
||||
Location: Lines 590-730
|
||||
Problem: 20+ info! logs during setup
|
||||
Impact: Startup log spam, obscures critical info
|
||||
Fix: Consolidate to 3-line summary, move details to debug!
|
||||
|
||||
FILE BREAKDOWN
|
||||
--------------
|
||||
|
||||
trainer.rs (4,658 lines) - 165 logging statements
|
||||
- Initialization: 20+ info! (TOO MANY)
|
||||
- Training loop: 6 info! per epoch (HIGH FREQUENCY)
|
||||
- Per-step: 2 debug! per step (VERY HIGH FREQUENCY)
|
||||
- Errors: 32 warn! (INCONSISTENT LEVELS)
|
||||
- Checkpoints: 8 logs (GOOD)
|
||||
|
||||
early_stopping.rs (257 lines) - 3 logging statements
|
||||
- All appropriate levels (GOOD)
|
||||
- Context included (GOOD)
|
||||
- Low frequency (GOOD)
|
||||
|
||||
lr_scheduler.rs (229 lines) - 0 logging statements (GOOD)
|
||||
statistics.rs (135 lines) - 0 logging statements (GOOD)
|
||||
config.rs (869 lines) - 0 logging statements (GOOD)
|
||||
|
||||
PERFORMANCE IMPACT
|
||||
------------------
|
||||
|
||||
Current (100-epoch training):
|
||||
- Total log lines: ~20,670
|
||||
- File size: 2.47 MB (with debug!)
|
||||
- Production (info only): 69 KB
|
||||
|
||||
After optimization:
|
||||
- Total log lines: ~313 (98.5% reduction)
|
||||
- File size: 37.8 KB (98.5% reduction)
|
||||
- No performance degradation
|
||||
|
||||
TOP 10 LOG LOCATIONS BY FREQUENCY
|
||||
----------------------------------
|
||||
|
||||
1. Line 2129 - REWARD_STATS (100 logs, EVERY EPOCH)
|
||||
2. Line 2252 - Validation loss (100 logs, EVERY EPOCH)
|
||||
3. Line 2271 - Action diversity (100 logs, EVERY EPOCH)
|
||||
4. Line 2296 - Episode stats (100 logs, EVERY EPOCH)
|
||||
5. Line 2306 - Exit breakdown (100 logs, EVERY EPOCH)
|
||||
6. Line 2324 - Risk metrics VaR/CVaR (100 logs, EVERY EPOCH)
|
||||
7. Line 1607 - Triple barrier start (20,000 logs, EVERY STEP)
|
||||
8. Line 1640 - Barrier episode end (variable, PER BARRIER HIT)
|
||||
9. Lines 590-730 - Initialization (20+ logs, STARTUP)
|
||||
10. Lines 1374-1406 - Rainbow components (10+ logs, STARTUP)
|
||||
|
||||
SPECIFIC RECOMMENDATIONS
|
||||
------------------------
|
||||
|
||||
IMMEDIATE (1-2 hours):
|
||||
[X] Lines 590-730: info! → debug! (initialization logs)
|
||||
[X] Line 2085: warn! → error! (training step failure)
|
||||
[X] Lines 2129-2333: Add epoch % 10 == 0 condition
|
||||
|
||||
SHORT-TERM (4-6 hours):
|
||||
[ ] Add epoch/step context to all error logs
|
||||
[ ] Consolidate initialization to 3-line summary
|
||||
[ ] Remove lines 2287-2288 (redundant recommendations)
|
||||
|
||||
MEDIUM-TERM (1-2 days):
|
||||
[ ] Migrate to structured logging (tracing fields)
|
||||
[ ] Add sampling to per-step logs (1/100 steps)
|
||||
[ ] Configure log level via environment variable
|
||||
|
||||
LONG-TERM (1 week):
|
||||
[ ] Add JSON log output support
|
||||
[ ] Integrate with observability platform
|
||||
[ ] Create log aggregation dashboard
|
||||
|
||||
CODE EXAMPLES
|
||||
-------------
|
||||
|
||||
BEFORE (Line 2129):
|
||||
info!("REWARD_STATS: epoch={}, mean={:.6}, ...", epoch + 1, ...);
|
||||
|
||||
AFTER:
|
||||
if (epoch + 1) % 10 == 0 || epoch == 0 {
|
||||
info!("REWARD_STATS: epoch={}, mean={:.6}, ...", epoch + 1, ...);
|
||||
}
|
||||
|
||||
BEFORE (Line 2085):
|
||||
warn!("Training step failed: {}, continuing...", e);
|
||||
|
||||
AFTER:
|
||||
error!("Training step failed at epoch {}, step {}: {}",
|
||||
epoch, step, e);
|
||||
|
||||
BEFORE (Lines 590-730, 20+ logs):
|
||||
info!("Creating regime-conditional DQN with 3 heads...");
|
||||
info!(" - Regime detection: ADX (index 211)...");
|
||||
info!("Kelly optimizer enabled...");
|
||||
// ... 17 more lines ...
|
||||
|
||||
AFTER (3 logs):
|
||||
info!("DQN Trainer initialized with {} features", config.state_dim);
|
||||
info!(" Architecture: {} ({})", arch_type, hidden_dims);
|
||||
debug!("Advanced: Kelly={}, Entropy={}, Masking={}, Stress={}",
|
||||
kelly_en, entropy_en, mask_en, stress_en);
|
||||
|
||||
TESTING VERIFICATION
|
||||
--------------------
|
||||
|
||||
# Before optimization
|
||||
RUST_LOG=debug cargo test dqn 2>&1 | wc -l
|
||||
Expected: ~20,000 lines
|
||||
|
||||
# After optimization
|
||||
RUST_LOG=debug cargo test dqn 2>&1 | wc -l
|
||||
Expected: ~300 lines (98.5% reduction)
|
||||
|
||||
# Verify critical logs still present
|
||||
grep "Early stopping triggered" log.txt # Must exist
|
||||
grep "Best model saved" log.txt # Must exist
|
||||
grep "error!" log.txt # Must exist (if errors)
|
||||
|
||||
RISK ASSESSMENT
|
||||
---------------
|
||||
|
||||
Risk Level: LOW
|
||||
- Logging changes are non-functional
|
||||
- No business logic affected
|
||||
- Existing tests will pass
|
||||
- Only affects observability, not behavior
|
||||
|
||||
Breaking Changes: NONE
|
||||
- Log format may change (users parsing logs manually)
|
||||
- Frequency reduction may break log-scraping scripts
|
||||
- Mitigation: Document changes, provide migration guide
|
||||
|
||||
RELATED DOCUMENTS
|
||||
-----------------
|
||||
- Full Analysis: docs/codebase-cleanup/dqn-logging-analysis.md
|
||||
- Logging Best Practices: docs/logging-guidelines.md (TODO)
|
||||
- Tracing Crate Docs: https://docs.rs/tracing/
|
||||
|
||||
CONTACT
|
||||
-------
|
||||
For questions or clarifications, see:
|
||||
- Code Quality Analyzer Role (CLAUDE.md)
|
||||
- Project: /home/jgrusewski/Work/foxhunt
|
||||
Reference in New Issue
Block a user