foxhunt/DQN_TEMPORAL_CHRONOLOGY_INVESTIGATION.md

DQN Temporal Chronology Investigation - Critical Findings

Date: 2025-11-04 Investigator: Claude (Sonnet 4.5) Status: ✅ INVESTIGATION COMPLETE Priority: 🔴 CRITICAL - Temporal leakage detected

---

Executive Summary

Investigation into suspected temporal data leakage revealed CRITICAL FINDINGS:

1. ✅ Temporal leakage CONFIRMED: ES_FUT_unseen_90d.parquet contains data from 2024 (Aug-Nov), while training data is from 2025 (Apr-Oct) - 261 days backwards in time
2. ✅ Valid evaluation exists: ES_FUT_unseen.parquet contains chronologically correct data from 2025-10-20 to 2025-11-03 (14 days after training ends)
3. ❌ User's claim was backwards: Claimed "98% SELL → 94.5% HOLD" but actual behavior is "0.3% SELL → 5.3% SELL" (17.6x increase on invalid data)
4. ⚠️ Model issue detected: DQN shows extreme conservatism (99.4% HOLD) on valid data with negative Sharpe ratio (-7.00)

---

Data Chronology Analysis

Complete Dataset Inventory

Dataset	Date Range	Duration	Rows	Temporal Validity
ES_FUT_180d.parquet	2025-04-23 → 2025-10-19	179 days	174,053	Training data
ES_FUT_unseen.parquet	2025-10-20 → 2025-11-03	14 days	14,520	✅ VALID (after training)
ES_FUT_unseen_90d.parquet	2024-08-04 → 2024-11-01	88 days	89,393	❌ INVALID (before training)

Temporal Relationship Diagram

Timeline (chronological order):

2024-08-04                  2024-11-01       2025-04-23                  2025-10-19  2025-10-20    2025-11-03
    |                           |                |                           |           |             |
    |<--- ES_FUT_unseen_90d --->|                |<--  TRAINING DATA (180d) -->|           |<-- EVAL  -->|
    |     (88 days, INVALID)    |                |                           |           |  (14 days)  |
    |                           |                |                           |           |             |
    |                           |                |                           |           |             |
    |<--------- 261 days BEFORE training ------->|                           |           |             |
    |                                            |                           |           |             |
    ❌ TEMPORAL LEAKAGE ZONE                     ✅ TRAINING ZONE            ✅ VALID EVAL

Gap Analysis

• ES_FUT_unseen.parquet: Starts 0 days after training ends (perfect temporal split)
• ES_FUT_unseen_90d.parquet: Starts 261 days BEFORE training begins (complete temporal inversion)

---

Model Behavior Comparison

Evaluation Results Side-by-Side

Evaluation #1: ES_FUT_unseen.parquet (VALID)

Dataset: 2025-10-20 to 2025-11-03 (14 days, 14,420 bars) Temporal Status: ✅ Chronologically AFTER training (correct out-of-sample test)

Metric	Value	Interpretation
BUY actions	43 (0.3%)	Extremely rare
SELL actions	45 (0.3%)	Extremely rare
HOLD actions	14,332 (99.4%)	Dominant behavior
Total trades	36	Very few
Win rate	19.4%	Poor
Total P&L	-$373.25	Losing
Sharpe ratio	-7.00	Severely negative
Max drawdown	-$377.25	Deep

Interpretation: Model exhibits extreme risk aversion, refusing to trade in almost all scenarios. This suggests overfitting to a specific 2025 market regime where HOLD was the optimal strategy during training.

---

Evaluation #2: ES_FUT_unseen_90d.parquet (INVALID)

Dataset: 2024-08-04 to 2024-11-01 (88 days, 89,293 bars) Temporal Status: ❌ Chronologically BEFORE training (temporal leakage)

Metric	Value	Interpretation
BUY actions	250 (0.3%)	Same as valid eval
SELL actions	4,697 (5.3%)	17.6x MORE than valid
HOLD actions	84,346 (94.5%)	Still dominant
Total trades	308	8.6x more
Win rate	18.2%	Poor
Total P&L	-$1,724.25	Worse losses
Sharpe ratio	-4.13	Negative (but less severe)

Interpretation: When confronted with 2024 market patterns (never seen during training), model increases SELL actions 17.6x. This is temporal overfitting - model learned 2025-specific patterns and panics when presented with unfamiliar 2024 volatility.

---

Behavior Flip Analysis

User's Claim vs. Reality

❌ User claimed: "98% SELL → 94.5% HOLD" ✅ Actual behavior: "0.3% SELL → 5.3% SELL" (17.6x increase on invalid data)

The user's perception was inverted. The model does NOT become more conservative on old data - it becomes MORE AGGRESSIVE (more SELL actions) because it's confused by unfamiliar patterns.

Why More SELL Actions on 2024 Data?

Hypothesis: Temporal overfitting and regime mismatch.

1. 2025 Training Regime (Apr-Oct 2025):

   • Model learned specific 2025 market patterns
   • Optimal strategy during training: HOLD (low volatility, range-bound)
   • Reward structure favored inaction

2. 2024 Evaluation Regime (Aug-Nov 2024):

   • Different volatility profile (unseen patterns)
   • Q-network interprets unfamiliar patterns as "high risk"
   • Model defaults to SELL as risk-reduction strategy

3. Result: When model sees 2024 data it never trained on:

   • Q-values become less confident
   • Unfamiliar patterns → interpreted as threat signals
   • SELL actions increase 17.6x (45 → 4,697)
   • Model is "panicking" and trying to exit positions

---

Root Cause: Temporal Leakage

How This Happened

1. Training data: ES_FUT_180d.parquet (2025-04-23 to 2025-10-19)
2. Intended evaluation: ES_FUT_unseen.parquet (2025-10-20 to 2025-11-03)
3. Accidental evaluation: ES_FUT_unseen_90d.parquet (2024-08-04 to 2024-11-01)

The ES_FUT_unseen_90d.parquet file was likely downloaded as "90-day historical data" in 2024, then accidentally used for evaluation against a model trained on 2025 data.

Why This Invalidates Results

Temporal leakage violates fundamental ML principle: test data must come from a time period AFTER training data.

When evaluation data is from the PAST:

• Model hasn't learned those patterns (impossible, time travel)
• Results measure "confusion" not "generalization"
• Cannot predict production performance
• Misleading metrics (Sharpe, win rate, P&L all meaningless)

---

Validation of Correct Dataset

ES_FUT_unseen.parquet - Chronological Verification

# Executed verification (2025-11-04):
Training data (ES_FUT_180d.parquet):
  - Start: 2025-04-23 00:00:00+00:00
  - End:   2025-10-19 23:59:00+00:00
  - Duration: 179 days
  - Rows: 174,053

Evaluation data (ES_FUT_unseen.parquet):
  - Start: 2025-10-20 00:00:00+00:00  ← NEXT DAY after training
  - End:   2025-11-03 12:59:00+00:00
  - Duration: 14 days
  - Rows: 14,520

✅ Temporal gap: 0 days (perfect split)
✅ Chronologically valid out-of-sample test

---

Critical Model Issue: Extreme Conservatism

Valid Evaluation Results (ES_FUT_unseen.parquet)

The chronologically correct evaluation reveals a CRITICAL FLAW:

99.4% HOLD behavior with:

• Win rate: 19.4%
• Sharpe ratio: -7.00 (severely negative)
• Total P&L: -$373.25 (losing money)
• Only 36 trades in 14,420 bars (0.25% trade frequency)

Why This is NOT Production-Ready

1. Too Conservative: Model refuses to trade even when opportunities exist
2. Negative Returns: -$373.25 P&L over 14 days is unacceptable
3. Poor Risk-Adjusted Returns: Sharpe ratio of -7.00 indicates risk is not compensated
4. Low Win Rate: 19.4% means 80.6% of trades lose money

Root Cause: Reward Function Imbalance

The DQN's reward structure likely over-penalizes trading and over-rewards HOLD:

• HOLD has zero commission cost → "safe" choice
• BUY/SELL incur $2.50 commission per side ($5 round-trip)
• Model learns: "Don't trade, avoid commissions, minimize losses"

This is passive strategy overfitting - model learned to do nothing instead of trade profitably.

---

Recommendations

Immediate Actions (Today)

1. ✅ DISCARD all ES_FUT_unseen_90d.parquet results (temporal leakage)
2. ✅ ACCEPT ES_FUT_unseen.parquet as ground truth (chronologically valid)
3. ❌ DO NOT deploy current DQN model to production (99.4% HOLD is not viable)
4. 📝 Document this finding in CLAUDE.md (prevent future temporal leakage)

Short-Term Fixes (This Week)

1. Download fresh evaluation data:

   # Download 30-90 days of data AFTER training period
   python3 scripts/python/data/download_es_90d_multi_contract.py \
     --start-date 2025-10-20 \
     --end-date 2025-12-31 \
     --output test_data/ES_FUT_evaluation_2025Q4.parquet
   

2. Add temporal validation tests:

   // ml/tests/temporal_chronology_test.rs
   #[test]
   fn test_evaluation_data_after_training() {
       let train_end = load_training_data_end_date();
       let eval_start = load_evaluation_data_start_date();
       assert!(eval_start > train_end,
               "Evaluation must start AFTER training ends");
   }
   

3. Update evaluation binary with validation:

   // ml/examples/evaluate_dqn.rs
   fn validate_temporal_chronology(
       training_end: DateTime<Utc>,
       eval_start: DateTime<Utc>
   ) -> Result<()> {
       if eval_start <= training_end {
           return Err(anyhow!(
               "TEMPORAL LEAKAGE: Eval starts before/during training"
           ));
       }
       Ok(())
   }
   

Medium-Term Retraining (Next 2 Weeks)

DQN model requires complete retraining with:

1. Diverse Market Regimes:

   • Bull market data (trending up)
   • Bear market data (trending down)
   • Sideways market data (range-bound)
   • High volatility periods
   • Low volatility periods

2. Adjusted Reward Function:

   // Current (too conservative):
   reward = match action {
       BUY | SELL => pnl - commission,
       HOLD => 0.0,  // ← Too favorable
   };
   
   // Proposed (balanced):
   reward = match action {
       BUY | SELL => pnl - commission,
       HOLD => -opportunity_cost,  // Penalize excessive inaction
   };
   

3. Longer Training Period:

   • Current: 179 days (Apr-Oct 2025)
   • Proposed: 365+ days (multiple seasons, regimes)

4. Longer Evaluation Period:

   • Current: 14 days (too short for statistical significance)
   • Proposed: 60-90 days minimum

Long-Term Improvements (Next Month)

1. Implement regime detection:

   • Classify market as bull/bear/sideways
   • Train separate DQN models per regime
   • Deploy appropriate model based on current regime

2. Add ensemble methods:

   • Combine DQN with PPO, TFT, MAMBA-2
   • Vote or weight predictions
   • Reduce single-model risk

3. Production monitoring:

   • Track action distribution in real-time
   • Alert if HOLD > 95% (model broken)
   • Alert if temporal drift detected

---

Lessons Learned

Data Management

1. Always validate temporal chronology before evaluation
2. Name datasets with date ranges (e.g., ES_FUT_2025-10-20_to_2025-11-03.parquet)
3. Store training/eval metadata (start date, end date, regime)
4. Automate temporal validation in CI/CD pipeline

Model Training

1. HOLD bias is real - reward functions matter critically
2. Single-regime training = overfitting - need diverse data
3. Short evaluation periods mislead - need 60-90 days minimum
4. 99% HOLD is a red flag - model learned to do nothing

Evaluation Best Practices

1. Always check date ranges before running eval
2. Plot action distributions to spot anomalies early
3. Compare against random/naive baselines
4. Validate Sharpe ratio is positive before production

---

Appendix: Reproduction Steps

Verify Temporal Chronology

# 1. Check all dataset date ranges
python3 << 'EOF'
import pandas as pd
import glob

for f in glob.glob('test_data/ES_FUT*.parquet'):
    df = pd.read_parquet(f)
    if 'ts_event' in df.columns:
        timestamps = df['ts_event']
    else:
        timestamps = df.index
    print(f"{f}:")
    print(f"  {timestamps.min()} → {timestamps.max()}")
    print(f"  Duration: {(timestamps.max() - timestamps.min()).days} days")
    print()
EOF

Re-run Valid Evaluation

# 2. Run evaluation with chronologically correct data
cargo run -p ml --example evaluate_dqn --release --features cuda -- \
  --model-path ml/trained_models/dqn_best_model.safetensors \
  --parquet-file test_data/ES_FUT_unseen.parquet \
  --output-json /tmp/dqn_eval_VALID.json

# Expected output:
# BUY:  43 (0.3%)
# SELL: 45 (0.3%)
# HOLD: 14332 (99.4%)

Re-run Invalid Evaluation (for comparison)

# 3. Run evaluation with temporally-invalid data
cargo run -p ml --example evaluate_dqn --release --features cuda -- \
  --model-path ml/trained_models/dqn_best_model.safetensors \
  --parquet-file test_data/ES_FUT_unseen_90d.parquet \
  --output-json /tmp/dqn_eval_INVALID.json

# Expected output:
# BUY:  250 (0.3%)
# SELL: 4697 (5.3%)  ← 17.6x more than valid
# HOLD: 84346 (94.5%)

---

Conclusion

Temporal leakage confirmed and analyzed. The ES_FUT_unseen_90d.parquet file contains data from 261 days before training began, making all results from it invalid.

The chronologically correct evaluation (ES_FUT_unseen.parquet) reveals the DQN model is extremely conservative (99.4% HOLD) and not production-ready (Sharpe -7.00, losing money).

Action required: Retrain DQN with diverse market regimes, balanced reward function, and longer time horizon.

---

Report Generated: 2025-11-04 20:30 UTC Investigation Duration: 45 minutes Status: ✅ COMPLETE Next Steps: Document in CLAUDE.md, retrain DQN model