foxhunt/scripts/EPSILON_FIX3_VALIDATION_README.md

# Fix #3 Epsilon Decay Validation Script

## Overview
Comprehensive validation test script for **Fix #3: Epsilon Decay Range Change** (`[0.990, 0.999]` → `[0.95, 0.99]`).

**Expected Outcome**: Break 100% HOLD bias, enable action diversity.

## Script Location
```bash
/home/jgrusewski/Work/foxhunt/scripts/validate_epsilon_fix3.sh
```

## Usage

### Basic Run
```bash
cd /home/jgrusewski/Work/foxhunt
./scripts/validate_epsilon_fix3.sh
```

### Expected Runtime
- **Duration**: ~5-10 minutes (5 trials × 10 epochs)
- **GPU**: RTX 3050 Ti (CUDA-accelerated)
- **Output**: `/tmp/ml_training/fix3_validation/`

## Success Criteria

The script validates Fix #3 using three criteria:

### ✅ Criterion 1: At least 1 trial with <90% HOLD
- **Target**: Break the 100% HOLD bias
- **Threshold**: At least 1 trial showing <90% HOLD actions
- **Pass**: `trials_with_diversity >= 1`

### ✅ Criterion 2: Action diversity >0%
- **Target**: BUY or SELL actions observed
- **Threshold**: min_hold_pct < 100%
- **Pass**: At least one trial shows non-zero BUY/SELL percentage

### ✅ Criterion 3: Epsilon values varying
- **Target**: Hyperopt explores epsilon_decay space
- **Threshold**: At least 2 unique epsilon values across trials
- **Pass**: `unique_epsilons > 1`

## Output Files

### 1. Raw Log File
```
/tmp/ml_training/fix3_validation/test_YYYYMMDD_HHMMSS.log
```
Complete hyperopt output including:
- Trial parameters
- Training progress
- Action distributions
- Objective values

### 2. Summary Report
```
/tmp/ml_training/fix3_validation/summary_YYYYMMDD_HHMMSS.txt
```
Structured analysis including:
- Epsilon decay values observed
- Action distribution statistics
- Success criteria validation
- Overall assessment (PASS/FAIL)

## Example Output

### Successful Validation
```
==================================================
Fix #3 Epsilon Decay Validation Summary
==================================================

Expected Epsilon Decay Range: [0.95, 0.99]
Actual Epsilon Values Observed:
  Trial 1: 0.976
    ✅ Within expected range [0.95, 0.99]
  Trial 2: 0.982
    ✅ Within expected range [0.95, 0.99]
  Trial 3: 0.968
    ✅ Within expected range [0.95, 0.99]

Action Distribution Analysis:
  Trials Analyzed: 5
  Trials with <90% HOLD: 3
  Min HOLD %: 72%
  Max HOLD %: 95%

Success Criteria Validation:
✅ Criterion 1 PASS: At least 1 trial with <90% HOLD (3 trials)
✅ Criterion 2 PASS: Action diversity detected (min HOLD=72%)
✅ Criterion 3 PASS: Epsilon values varying across trials (3 unique values)

Overall Assessment:
✅ VALIDATION PASSED: Fix #3 successfully breaks 100% HOLD bias

Key Improvements:
  - Action diversity enabled (BUY/SELL actions observed)
  - HOLD percentage reduced to 72% (min)
  - 3/5 trials show <90% HOLD
```

### Failed Validation
```
Success Criteria Validation:
❌ Criterion 1 FAIL: No trials with <90% HOLD (all trials ≥90% HOLD)
❌ Criterion 2 FAIL: 100% HOLD bias persists

Overall Assessment:
❌ VALIDATION FAILED: 100% HOLD bias persists

Possible Issues:
  - Epsilon decay range change not effective
  - Other hyperparameters overriding epsilon effect
  - Training epochs insufficient for exploration
```

## Exit Codes

- `0`: VALIDATION PASSED (all criteria met)
- `1`: VALIDATION FAILED (one or more criteria failed)

## Configuration

Default settings (edit script to customize):

```bash
TRIALS=5                               # Number of hyperopt trials
EPOCHS=10                              # Training epochs per trial
PARQUET_FILE="test_data/ES_FUT_180d.parquet"  # Input data
OUTPUT_DIR="/tmp/ml_training/fix3_validation"  # Results directory
```

## Dependencies

- **Rust toolchain**: `cargo` (release mode)
- **CUDA**: RTX 3050 Ti GPU
- **Parquet file**: `test_data/ES_FUT_180d.parquet`
- **Binary**: `ml/examples/hyperopt_dqn_demo.rs`

## Troubleshooting

### Error: Parquet file not found
```bash
ERROR: Parquet file not found: test_data/ES_FUT_180d.parquet
```
**Fix**: Ensure parquet file exists:
```bash
ls -lh test_data/ES_FUT_180d.parquet
```

### Error: CUDA not available
**Fix**: Verify GPU access:
```bash
nvidia-smi
cargo build --release -p ml --features cuda
```

### No epsilon values extracted
**Possible causes**:
1. Log format changed (check `hyperopt_dqn_demo.rs` output)
2. Trials failed early (check raw log file)
3. Regex parsing issue (verify log manually)

## Related Files

- **Fix Implementation**: `/home/jgrusewski/Work/foxhunt/ml/src/hyperopt/adapters/dqn.rs` (lines 44-45)
- **Hyperopt Example**: `/home/jgrusewski/Work/foxhunt/ml/examples/hyperopt_dqn_demo.rs`
- **Training Binary**: `/home/jgrusewski/Work/foxhunt/ml/examples/train_dqn.rs`

## Quick Test (1 trial)

For rapid validation (1-2 minutes):

```bash
cd /home/jgrusewski/Work/foxhunt
cargo run --release -p ml --example hyperopt_dqn_demo --features cuda -- \
  --parquet-file test_data/ES_FUT_180d.parquet --trials 1 --epochs 5
```

Look for:
- `epsilon_decay: 0.9XX` (in range [0.95, 0.99])
- `Action distribution: BUY=X%, SELL=Y%, HOLD=Z%` (Z < 100%)

## Next Steps

1. **Run validation**: `./scripts/validate_epsilon_fix3.sh`
2. **Review summary**: `cat /tmp/ml_training/fix3_validation/summary_*.txt`
3. **If PASS**: Proceed to full hyperopt (50-100 trials)
4. **If FAIL**: Investigate logs, check for conflicting hyperparameters

## Support

For issues or questions:
1. Check raw log file for errors
2. Verify CUDA/GPU availability
3. Review hyperopt_dqn_demo.rs output format
4. Consult CLAUDE.md for DQN bug fix context