foxhunt/docs/codebase-cleanup/dqn-logging-analysis.md

DQN Trainer Logging Pattern Analysis

Date: 2025-11-27 Module: ml/src/trainers/dqn/ Total Lines in trainer.rs: 4,658

---

Executive Summary

The DQN trainer module uses tracing crate exclusively with 169 total logging statements across trainer.rs (98 info!, 33 debug!, 32 warn!, 5 tracing::, 1 error!). The logging is generally well-structured but suffers from:

1. High-frequency training loop logs (every epoch) that could impact performance
2. Inconsistent log levels for similar operations
3. Missing context in some error logs
4. Excessive initialization verbosity (20+ info! logs during setup)
5. No structured logging fields (all string interpolation)

---

1. Logging Crate Usage

Crate: tracing (Consistent)

// File: ml/src/trainers/dqn/trainer.rs:19
use tracing::{debug, info, warn};

// File: ml/src/trainers/dqn/early_stopping.rs:8
use tracing::{debug, info};

✅ Good: Single logging crate used consistently across all modules ⚠️ Issue: No structured fields used (all string interpolation)

Logging Statement Frequency

info!    : 98 (58%)  - Configuration, epoch summaries, checkpoint saves
debug!   : 33 (20%)  - Q-values, action distributions, preprocessing details
warn!    : 32 (19%)  - Safety warnings, validation failures, degraded mode
tracing::: 5  (3%)   - Early stopping triggers (error! and info!)
error!   : 1  (<1%)  - Critical failures only

---

2. Categorized Logging Patterns

2.1 Training Loop Logs (High Frequency - Every Epoch)

Location: Lines 2129-2333 (204 lines of logging per epoch)

// EVERY EPOCH (100+ epochs):
info!("REWARD_STATS: epoch={}, mean={:.6}, std={:.6}, min={:.6}, max={:.6}, ...")  // Line 2129
info!("Epoch {}/{}: val_loss={:.6}", ...)                                           // Line 2252
info!("Epoch {}/{}: Action diversity={}/{} ({:.1}%)", ...)                         // Line 2271
info!("WAVE P2 Episode Stats [Epoch {}]: {} episodes, ...")                        // Line 2296
info!("  Exit breakdown: profit={}({:.1}%), stop={}({:.1}%), ...")                 // Line 2306
info!("Epoch {}/{}: Risk Metrics - VaR(95%)={:.4}%, CVaR(95%)={:.4}%, ...")        // Line 2324

⚠️ PROBLEM: 6+ info! logs per epoch × 100 epochs = 600+ log lines Impact: Log file bloat, potential I/O bottleneck on slow filesystems Recommendation: Move to debug! or reduce frequency (every 10 epochs)

---

2.2 Checkpoint/Save Logs

Good Pattern (Appropriate Level):

// Line 2394
info!("Best model saved to: {}", best_checkpoint_path);

// Line 2383 (debug - correct level)
debug!("SAFETY: Checkpoint verification passed ({} bytes)", checkpoint_data.len());

Missing Context:

// Line 2410-2416 - Missing epoch number in error log
warn!("⚠️ Failed to save best checkpoint: {}", e);
// Should be: warn!("⚠️ Failed to save best checkpoint at epoch {}: {}", epoch, e);

Files: trainer.rs:2354-2466

---

2.3 Error Handling Logs

Inconsistent Levels for Similar Failures:

// Line 1613 - Triple barrier failure (warn - correct)
warn!("WAVE 1.1: Failed to start triple barrier tracking: {}", e);

// Line 2085 - Training step failure (warn - should be error!)
warn!("Training step failed: {}, continuing...", e);

// Line 2080 - Early stopping (tracing::error - correct)
tracing::error!("🛑 TERMINATING: Early stopping triggered - {}", e);

⚠️ PROBLEM: Training step failures should be error!, not warn!

Missing Stack Context:

// Line 2163 - No epoch/step info
warn!("SAFETY: Training stuck for {} epochs (loss variance: {:.6}, mean: {:.6})",
      self.safety_loss_plateau_counter, std_dev, mean);
// Better: warn!("SAFETY: Training stuck at epoch {} for {} consecutive epochs (...)",
//              epoch, self.safety_loss_plateau_counter, ...);

---

2.4 Metric Reporting Logs

Too Verbose (Every Epoch):

// Line 2271-2288 - Every epoch
info!("Epoch {}/{}: Action diversity={}/{} ({:.1}%)", ...);
if active_actions_count < DIVERSITY_THRESHOLD {
    warn!("⚠️  LOW ACTION DIVERSITY: {}/45 actions (<20%), consider increasing epsilon floor", ...);
    info!("   Recommendation: Increase epsilon_end from 0.05 to 0.10");  // Redundant
    info!("   Alternative: Add entropy regularization bonus");            // Redundant
}

Recommendation:

• Move diversity logs to debug!
• Only warn! when threshold breached (not every epoch)
• Remove redundant "Recommendation" logs (should be in docs)

---

2.5 Debug/Diagnostic Logs

Good Examples (Appropriate Level):

// Line 244-252 - Action distribution (debug - correct)
debug!("Action Distribution [Epoch {}] - Top 5 Actions:", self.epoch);
for (idx, count) in sorted_actions.iter().take(5) {
    debug!("  [{:2}] {:?}: {} ({:.1}%)", idx, action, count, pct);
}

// Line 2794-2797 - Preprocessing stats (debug - correct)
debug!("✅ Preprocessing complete:");
debug!("  • Mean: {:.6} (expected ~0 for normalized data)", mean);
debug!("  • Std: {:.4} (expected ~1 for normalized data)", std);

⚠️ Issue: High-frequency debug in training loop

// Line 1607-1640 - EVERY STEP (200 steps/epoch × 100 epochs = 20,000 logs!)
debug!("WAVE 1.1: Started triple barrier tracking at step {}, price=${:.2}, position={:.2}", ...);
debug!("WAVE P2: Barrier-driven episode end at step {}: {:?}, label={}, ...", ...);

Recommendation: Reduce frequency or use trace! level

---

2.6 Initialization Logs (Excessive Verbosity)

Lines 590-730: 20+ info! logs during initialization

// Too many info! logs during setup
info!("Creating regime-conditional DQN with 3 heads (Trending, Ranging, Volatile)");
info!("  - Regime detection: ADX (index 211) + Entropy (index 219)");
info!("  - Classification: Trending (ADX>25), Volatile (ADX≤25 & Entropy>0.7), ...");
info!("Creating standard DQN with single Q-network head");
info!("Triple barrier engine initialized with 1000 max trackers");
info!("Kelly optimizer enabled (fractional={}, max={})", ...);
info!("Entropy regularization enabled (coefficient=0.01)");
info!("Stress testing enabled (8 scenarios)");
info!("Action masking enabled (max_position=±{:.1}, 30-50% filtering expected)", ...);
info!("Drawdown monitor enabled (thresholds: 10%, 12.5%, 15%)");
info!("Position limiter enabled (abs=±10.0, notional=$1M, concentration=10%)");
info!("Circuit breaker enabled (threshold=5 failures, cooldown=60s)");
// ... 10 more similar logs ...

⚠️ PROBLEM: Startup log spam (20+ lines) Recommendation: Consolidate into single summary log or use debug!

---

3. Problematic Patterns

3.1 High-Frequency Logs (Performance Risk)

Location	Frequency	Current Level	Impact
Lines 2129-2333	Every epoch (100+)	info!	Log file bloat
Lines 1607-1640	Every step (20K+)	debug!	I/O bottleneck
Lines 244-273	Every epoch	debug!	Acceptable
Lines 2747-2749	Once per dataset	debug!	Acceptable

Estimated Log Volume:

• Training loop: 6 info! × 100 epochs = 600 lines
• Per-step debug: 2 debug! × 200 steps × 100 epochs = 40,000 lines (if enabled)
• Initialization: 20+ info! lines

Total: ~40,600+ log lines for 100-epoch training run

---

3.2 Inconsistent Log Levels

Issue Type	Current Level	Should Be	Location
Training step failure	warn!	error!	Line 2085
Triple barrier failure	warn!	debug! (non-critical)	Line 1613
Action diversity warning	info! + warn!	debug! (only warn on breach)	Lines 2271-2288
Initialization configs	info!	debug!	Lines 590-730

---

3.3 Missing Context in Error Logs

// ❌ BAD - Missing epoch/step context
warn!("Training step failed: {}, continuing...", e);  // Line 2085

// ✅ GOOD - Has epoch context
warn!("SAFETY: Training stuck for {} epochs (loss variance: {:.6}, mean: {:.6})",
      self.safety_loss_plateau_counter, std_dev, mean);  // Line 2163

Common Missing Context:

• Current epoch number
• Current step number
• Model/checkpoint path
• Hyperparameter values (learning rate, batch size, etc.)

---

3.4 No Structured Logging

Current Pattern (String Interpolation):

info!("Epoch {}/{}: val_loss={:.6}", epoch + 1, self.hyperparams.epochs, val_loss);

Recommended Pattern (Structured Fields):

info!(
    epoch = epoch + 1,
    total_epochs = self.hyperparams.epochs,
    val_loss = %val_loss,
    "Epoch validation complete"
);

Benefits:

• Machine-parseable logs (JSON output)
• Easier filtering/aggregation
• Better integration with observability tools (Datadog, Prometheus, etc.)

---

4. Recommendations

Priority 1: Reduce High-Frequency Logs

Training Loop (Lines 2129-2333):

// BEFORE (every epoch):
info!("REWARD_STATS: epoch={}, mean={:.6}, ...", epoch + 1, ...);
info!("Epoch {}/{}: Action diversity={}/{} ({:.1}%)", ...);

// AFTER (every 10 epochs OR debug level):
if (epoch + 1) % 10 == 0 {
    info!("REWARD_STATS: epoch={}, mean={:.6}, ...", epoch + 1, ...);
}
debug!("Epoch {}/{}: Action diversity={}/{} ({:.1}%)", ...);  // Always available if needed

Per-Step Logs (Lines 1607-1640):

// BEFORE (every step):
debug!("WAVE 1.1: Started triple barrier tracking at step {}, ...", i, ...);

// AFTER (use trace! or sample every 100 steps):
trace!("WAVE 1.1: Started triple barrier tracking at step {}, ...", i, ...);
// OR
if i % 100 == 0 {
    debug!("WAVE 1.1: Triple barrier tracking at step {}, ...", i, ...);
}

---

Priority 2: Fix Inconsistent Log Levels

// Training step failure (CRITICAL - should stop training)
// Line 2085: warn! → error!
error!("Training step failed at epoch {}: {}", epoch, e);

// Initialization configs (LOW PRIORITY - reduce verbosity)
// Lines 590-730: info! → debug!
debug!("Kelly optimizer enabled (fractional={}, max={})", ...);

// Action diversity warning (ONLY warn when threshold breached)
// Lines 2283-2288: Remove redundant info! logs
if active_actions_count < DIVERSITY_THRESHOLD {
    warn!("LOW ACTION DIVERSITY: {}/45 actions at epoch {} (<20%)",
          active_actions_count, epoch + 1);
    // Remove these:
    // info!("   Recommendation: Increase epsilon_end from 0.05 to 0.10");
    // info!("   Alternative: Add entropy regularization bonus");
}

---

Priority 3: Add Missing Context

// Add epoch/step context to all error logs
warn!("Training step failed at epoch {}, step {}: {}", epoch, step, e);

// Add model path to checkpoint logs
info!("Best model saved to: {} (epoch {}, val_loss={:.6})",
      best_checkpoint_path, epoch, val_loss);

---

Priority 4: Consolidate Initialization Logs

BEFORE (20+ lines):

info!("Creating regime-conditional DQN with 3 heads (Trending, Ranging, Volatile)");
info!("  - Regime detection: ADX (index 211) + Entropy (index 219)");
info!("Kelly optimizer enabled (fractional={}, max={})", ...);
info!("Entropy regularization enabled (coefficient=0.01)");
// ... 15 more lines ...

AFTER (3 lines):

info!("DQN Trainer initialized with {} features", config.state_dim);
info!("  Architecture: {} ({})",
      if regime_conditional { "Regime-Conditional (3 heads)" } else { "Standard" },
      config.hidden_dims.iter().map(|d| d.to_string()).collect::<Vec<_>>().join("-"));
debug!("Advanced features: Kelly={}, Entropy={}, Action Masking={}, Stress Testing={}",
       kelly_enabled, entropy_enabled, action_masking_enabled, stress_testing_enabled);

---

Priority 5: Adopt Structured Logging

Example Migration:

// BEFORE:
info!("Epoch {}/{}: val_loss={:.6}", epoch + 1, self.hyperparams.epochs, val_loss);

// AFTER:
info!(
    epoch = epoch + 1,
    total_epochs = self.hyperparams.epochs,
    val_loss = %val_loss,
    learning_rate = %self.current_lr,
    "Epoch validation complete"
);

Benefits:

• Enables JSON logging: RUST_LOG_FORMAT=json cargo run
• Better integration with observability platforms
• Easier filtering: grep -P '"epoch":\s*50' log.json

---

5. File-by-File Summary

trainer.rs (4,658 lines)

Category	Count	Lines	Issues
Initialization	20+	590-730	Too verbose (info! → debug!)
Training loop	6/epoch	2129-2333	High frequency (info! → debug! or reduce)
Per-step debug	2/step	1607-1640	Very high frequency (debug! → trace!)
Error handling	32	Various	Inconsistent levels, missing context
Checkpoint	8	2354-2466	Good, but missing epoch in some errors
Metrics	15+	2271-2333	Too verbose per epoch

Total Logging Statements: 165

---

early_stopping.rs (257 lines)

// Line 86-89 (debug - correct)
debug!("Early stopping: Improvement detected: {:.6} (val_loss: {:.6} -> {:.6})", ...);

// Line 99-102 (debug - correct)
debug!("Early stopping: No improvement for {} epoch(s) (improvement: {:.6} < min_delta: {:.6})", ...);

// Line 105-109 (info - correct, only on trigger)
info!("🛑 Early stopping triggered at epoch {}! No improvement for {} epochs (...)", ...);

✅ Good: Appropriate log levels, context included, low frequency (only on trigger)

Total Logging Statements: 3

---

lr_scheduler.rs (229 lines)

Total Logging Statements: 0 (no logs)

✅ Good: Utility module with no side effects, no logging needed

---

statistics.rs (135 lines)

Total Logging Statements: 0 (no logs)

✅ Good: Pure data processing module, no logging needed

---

config.rs (869 lines)

Total Logging Statements: 0 (no logs)

✅ Good: Configuration structs only, no logging needed

---

6. Performance Impact Estimation

Current Logging Volume (100-epoch training)

Log Type	Frequency	Total Lines	Avg Size	Total Size
Initialization	1×	20	80 bytes	1.6 KB
Per-epoch info	100×	600	100 bytes	60 KB
Per-step debug	20,000×	20,000	120 bytes	2.4 MB
Error/warn	~50	50	150 bytes	7.5 KB
TOTAL		20,670		~2.47 MB

With debug! enabled: 2.47 MB log file for 100-epoch run Production (info! only): ~69 KB log file

---

Recommended Logging Volume (After Optimization)

Log Type	Frequency	Total Lines	Avg Size	Total Size
Initialization	1×	3	100 bytes	300 bytes
Per-epoch info (every 10th)	10×	60	100 bytes	6 KB
Per-step trace (sampled)	200×	200	120 bytes	24 KB
Error/warn	~50	50	150 bytes	7.5 KB
TOTAL		313		~37.8 KB

Reduction: 20,670 → 313 lines (98.5% reduction) Size Reduction: 2.47 MB → 37.8 KB (98.5% reduction)

---

7. Actionable Next Steps

Immediate (1-2 hours):

1. Change initialization logs (lines 590-730) from info! → debug!
2. Fix training step failure (line 2085) from warn! → error!
3. Reduce per-epoch info logs to every 10th epoch

Short-term (4-6 hours):

4. Add epoch/step context to all error logs
5. Consolidate initialization into 3-line summary
6. Remove redundant recommendation logs (lines 2287-2288)

Medium-term (1-2 days):

7. Migrate to structured logging (tracing fields)
8. Add sampling to per-step debug logs (1/100 steps)
9. Implement log level configuration via environment variable

Long-term (1 week):

10. Add JSON log output support
11. Integrate with observability platform (Datadog/Prometheus)
12. Create log aggregation dashboard

---

8. Testing Checklist

Before/after comparison:

# BEFORE optimization
RUST_LOG=debug cargo run --bin dqn_train -- --epochs 100 2>&1 | wc -l
# Expected: ~20,000 lines

# AFTER optimization
RUST_LOG=debug cargo run --bin dqn_train -- --epochs 100 2>&1 | wc -l
# Expected: ~300 lines

# Verify critical logs still present
grep "Early stopping triggered" dqn_train.log  # Should exist
grep "Best model saved" dqn_train.log          # Should exist
grep "Training step failed" dqn_train.log      # Should exist (if errors occur)

---

Appendix A: Complete Logging Inventory

trainer.rs Line References

Initialization (lines 590-730):

• 590: info! regime-conditional DQN creation
• 591-592: info! regime detection config
• 597: info! standard DQN creation
• 637: info! triple barrier engine
• 652-653: info! Kelly optimizer
• 666: info! entropy regularization
• 680: info! stress testing
• 687-692: info! action masking
• 702: info! drawdown monitor
• 717: info! position limiter
• 730: info! circuit breaker
• 745-747: info! dropout scheduler
• 775-777: info! HER buffer
• 786-788: info! GAE calculator
• 796-798: info! noisy sigma scheduler
• 955-963: info! training config summary
• 1361-1369: info! target update mode
• 1374-1406: info! Rainbow DQN components (10+ logs)

Training Loop (per-epoch, lines 2129-2333):

• 2129-2139: info! REWARD_STATS
• 2163-2166: warn! training plateau
• 2189-2213: info! epoch progress bars
• 2252-2257: info! validation loss
• 2271-2278: info! action diversity
• 2283-2288: warn! + 2× info! low diversity recommendations
• 2296-2312: info! episode statistics (2 logs)
• 2324-2333: info! risk metrics (VaR/CVaR)

Per-step (lines 1607-1640):

• 1607-1610: debug! triple barrier start
• 1640-1643: debug! barrier-driven episode end

Checkpoints (lines 2354-2466):

• 2359-2363: info! new best validation loss
• 2383: debug! checkpoint verification
• 2394: info! best model saved
• 2410: warn! checkpoint save failure
• 2416: info! checkpoint saved successfully
• 2429: info! best model loaded
• 2447: warn! checkpoint load failure
• 2451-2457: info! early stopping checkpoint

Final Summary (lines 2506-2561):

• 2506-2521: info! training complete summary (5 logs)
• 2544-2567: info! dataset loading (5 logs)

Data Loading (lines 2592-3154):

• 2592-2906: info!/debug! Parquet loading (20+ logs)
• 2943-3023: info!/debug! DBN loading (10+ logs)

Training Steps (lines 3392-3702):

• 3392: debug! epsilon adjustment
• 3529: tracing::info! early stopping (gradient collapse)
• 3538-3556: warn! gradient anomalies
• 3633: tracing::info! early stopping (gradient collapse)
• 3639-3655: warn! gradient issues
• 3702: tracing::info! early stopping (Q-value divergence)

Utilities (lines 3852-4126):

• 3852: info! replay buffer cleared
• 3862: info! target network reset
• 4126: warn! OFI calculation failure

---

Appendix B: Recommended Log Levels Guide

Event Type	Current	Recommended	Rationale
Initialization	info!	debug!	One-time setup, reduces startup noise
Epoch summary	info!	info! (every 10th)	Important milestones, but reduce frequency
Per-step training	debug!	trace!	Too frequent, use sampling instead
Checkpoint save	info!	info!	Critical events, keep as-is
Validation loss	info!	info! (every 10th)	Important metrics, reduce frequency
Training failures	warn!	error!	Should fail-fast, elevate severity
Safety warnings	warn!	warn!	Correct level, add context
Triple barrier	warn!/debug!	debug!	Non-critical, informational only
Early stopping	tracing::error!	error!	Critical termination, correct level
Action diversity	info! + warn!	debug! + warn!	Log all at debug, warn only on breach

---

End of Report