foxhunt/scripts/EPSILON_FIX3_VALIDATION_README.md

Fix #3 Epsilon Decay Validation Script

Overview

Comprehensive validation test script for Fix #3: Epsilon Decay Range Change ([0.990, 0.999] → [0.95, 0.99]).

Expected Outcome: Break 100% HOLD bias, enable action diversity.

Script Location

/home/jgrusewski/Work/foxhunt/scripts/validate_epsilon_fix3.sh

Usage

Basic Run

cd /home/jgrusewski/Work/foxhunt
./scripts/validate_epsilon_fix3.sh

Expected Runtime

• Duration: ~5-10 minutes (5 trials × 10 epochs)
• GPU: RTX 3050 Ti (CUDA-accelerated)
• Output: /tmp/ml_training/fix3_validation/

Success Criteria

The script validates Fix #3 using three criteria:

✅ Criterion 1: At least 1 trial with <90% HOLD

• Target: Break the 100% HOLD bias
• Threshold: At least 1 trial showing <90% HOLD actions
• Pass: trials_with_diversity >= 1

✅ Criterion 2: Action diversity >0%

• Target: BUY or SELL actions observed
• Threshold: min_hold_pct < 100%
• Pass: At least one trial shows non-zero BUY/SELL percentage

✅ Criterion 3: Epsilon values varying

• Target: Hyperopt explores epsilon_decay space
• Threshold: At least 2 unique epsilon values across trials
• Pass: unique_epsilons > 1

Output Files

1. Raw Log File

/tmp/ml_training/fix3_validation/test_YYYYMMDD_HHMMSS.log

Complete hyperopt output including:

• Trial parameters
• Training progress
• Action distributions
• Objective values

2. Summary Report

/tmp/ml_training/fix3_validation/summary_YYYYMMDD_HHMMSS.txt

Structured analysis including:

• Epsilon decay values observed
• Action distribution statistics
• Success criteria validation
• Overall assessment (PASS/FAIL)

Example Output

Successful Validation

==================================================
Fix #3 Epsilon Decay Validation Summary
==================================================

Expected Epsilon Decay Range: [0.95, 0.99]
Actual Epsilon Values Observed:
  Trial 1: 0.976
    ✅ Within expected range [0.95, 0.99]
  Trial 2: 0.982
    ✅ Within expected range [0.95, 0.99]
  Trial 3: 0.968
    ✅ Within expected range [0.95, 0.99]

Action Distribution Analysis:
  Trials Analyzed: 5
  Trials with <90% HOLD: 3
  Min HOLD %: 72%
  Max HOLD %: 95%

Success Criteria Validation:
✅ Criterion 1 PASS: At least 1 trial with <90% HOLD (3 trials)
✅ Criterion 2 PASS: Action diversity detected (min HOLD=72%)
✅ Criterion 3 PASS: Epsilon values varying across trials (3 unique values)

Overall Assessment:
✅ VALIDATION PASSED: Fix #3 successfully breaks 100% HOLD bias

Key Improvements:
  - Action diversity enabled (BUY/SELL actions observed)
  - HOLD percentage reduced to 72% (min)
  - 3/5 trials show <90% HOLD

Failed Validation

Success Criteria Validation:
❌ Criterion 1 FAIL: No trials with <90% HOLD (all trials ≥90% HOLD)
❌ Criterion 2 FAIL: 100% HOLD bias persists

Overall Assessment:
❌ VALIDATION FAILED: 100% HOLD bias persists

Possible Issues:
  - Epsilon decay range change not effective
  - Other hyperparameters overriding epsilon effect
  - Training epochs insufficient for exploration

Exit Codes

• 0: VALIDATION PASSED (all criteria met)
• 1: VALIDATION FAILED (one or more criteria failed)

Configuration

Default settings (edit script to customize):

TRIALS=5                               # Number of hyperopt trials
EPOCHS=10                              # Training epochs per trial
PARQUET_FILE="test_data/ES_FUT_180d.parquet"  # Input data
OUTPUT_DIR="/tmp/ml_training/fix3_validation"  # Results directory

Dependencies

• Rust toolchain: cargo (release mode)
• CUDA: RTX 3050 Ti GPU
• Parquet file: test_data/ES_FUT_180d.parquet
• Binary: ml/examples/hyperopt_dqn_demo.rs

Troubleshooting

Error: Parquet file not found

ERROR: Parquet file not found: test_data/ES_FUT_180d.parquet

Fix: Ensure parquet file exists:

ls -lh test_data/ES_FUT_180d.parquet

Error: CUDA not available

Fix: Verify GPU access:

nvidia-smi
cargo build --release -p ml --features cuda

No epsilon values extracted

Possible causes:

1. Log format changed (check hyperopt_dqn_demo.rs output)
2. Trials failed early (check raw log file)
3. Regex parsing issue (verify log manually)

Related Files

• Fix Implementation: /home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs (lines 44-45)
• Hyperopt Example: /home/jgrusewski/Work/foxhunt/ml/examples/hyperopt_dqn_demo.rs
• Training Binary: /home/jgrusewski/Work/foxhunt/ml/examples/train_dqn.rs

Quick Test (1 trial)

For rapid validation (1-2 minutes):

cd /home/jgrusewski/Work/foxhunt
cargo run --release -p ml --example hyperopt_dqn_demo --features cuda -- \
  --parquet-file test_data/ES_FUT_180d.parquet --trials 1 --epochs 5

Look for:

• epsilon_decay: 0.9XX (in range [0.95, 0.99])
• Action distribution: BUY=X%, SELL=Y%, HOLD=Z% (Z < 100%)

Next Steps

1. Run validation: ./scripts/validate_epsilon_fix3.sh
2. Review summary: cat /tmp/ml_training/fix3_validation/summary_*.txt
3. If PASS: Proceed to full hyperopt (50-100 trials)
4. If FAIL: Investigate logs, check for conflicting hyperparameters

Support

For issues or questions:

1. Check raw log file for errors
2. Verify CUDA/GPU availability
3. Review hyperopt_dqn_demo.rs output format
4. Consult CLAUDE.md for DQN bug fix context