foxhunt/AGENT_152_VALIDATION_CHECKLIST.md

Agent 152: E2E 225-Feature Training Validation Checklist

Date: 2025-10-22 Status: ⏳ Pending Execution Test Plan: See AGENT_152_PHASE_5_E2E_TEST_PLAN.md Quick Guide: See AGENT_152_PHASE_5_QUICK_SUMMARY.md

---

🎯 Quick Test (Scenario 1) - MANDATORY

Estimated Time: 15 minutes Model: PPO Dataset: ES_FUT_small.parquet (~500 bars) Command:

cargo run --release -p ml --example train_ppo_parquet -- \
  --parquet-file test_data/ES_FUT_small.parquet \
  --epochs 1 --batch-size 8 --no-early-stopping

Pre-Flight Checks

• Working directory: /home/jgrusewski/Work/foxhunt
• File exists: test_data/ES_FUT_small.parquet (~25KB)
• Rust toolchain: cargo --version works
• (Optional) GPU available: nvidia-smi works

Execution

• STARTED: Training command executed at: ____:____ (HH:MM)
• Compilation completed without errors
• Training started without errors

Critical Validation Points (IN ORDER)

1. Data Loading

• ✅ Console shows: "📊 Loading market data from Parquet file..."
• ✅ Console shows: "✅ Loaded XXX OHLCV bars" (XXX should be ~100-500)
• ❌ No errors: "Failed to open Parquet file"
• ❌ No errors: "No bars loaded from Parquet file"

Data loading result: [ ] PASS [ ] FAIL

---

2. Feature Extraction (MOST CRITICAL)

• ✅ Console shows: "🏗️ Extracting 225-dimensional feature vectors..."
• ✅ Console shows: "✅ Extracted XXX feature vectors (dim=225, warmup bars skipped=50)"
  • Required text: Must see "dim=225" explicitly
  • Feature count should be ~50 bars less than loaded bars (due to warmup)
• ✅ Console shows: "✅ Feature extraction complete: XXX samples"
• ❌ No errors: "State dimension mismatch: expected 225, got X"
• ❌ No errors: "No features extracted"

Feature extraction result: [ ] PASS [ ] FAIL

If FAIL: See Troubleshooting → Issue 2 in test plan

---

3. Model Initialization

• ✅ Console shows: "✅ PPO trainer initialized (state_dim=225)"
  • Required text: Must see "state_dim=225" explicitly
• ❌ No errors: "Failed to create PPO trainer"
• ❌ No errors: "CUDA out of memory" (if GPU is used)

Model initialization result: [ ] PASS [ ] FAIL

If GPU OOM: Retry with --batch-size 4

---

4. Training Execution

• ✅ Console shows: "🏋️ Starting training..."
• ✅ Console shows epoch progress: "📊 Epoch 1/1: policy_loss=X.XXXX, value_loss=X.XXXX, ..."
• ✅ Training completes: "✅ Training completed successfully!"
• ❌ No errors: "Training failed"
• ❌ No NaN/Inf errors: "loss=nan" or "loss=inf"

Training execution result: [ ] PASS [ ] FAIL

---

5. Model Checkpointing

• ✅ Console shows: "💾 Final checkpoint saved to: ml/trained_models/ppo_checkpoint_epoch_1.safetensors"
• ✅ Console shows: "📁 Model files saved to: ml/trained_models"
• ❌ No errors: "Failed to save checkpoint"

Model checkpointing result: [ ] PASS [ ] FAIL

---

6. Policy Convergence (Optional, but indicates quality)

• Console shows: "🔍 Policy Convergence Analysis:"
• Console shows: "• Policy updates (KL > 0): X" where X > 0
• Console shows: "✅ PASS: Policy updates detected (KL divergence > 0)"
• Console shows: "✅ PASS: Value network learning (explained variance > 0.5)"

Policy convergence result: [ ] PASS [ ] PARTIAL [ ] FAIL

Note: Even if FAIL, this is non-critical for quick validation

---

Post-Execution Verification

Filesystem Checks

# Run these commands:
ls -lh ml/trained_models/ppo_*.safetensors | tail -5

• ✅ File exists: ml/trained_models/ppo_actor_epoch_1.safetensors
• ✅ File exists: ml/trained_models/ppo_critic_epoch_1.safetensors
• ✅ File exists: ml/trained_models/ppo_checkpoint_epoch_1.safetensors
• ✅ All 3 files have size >1MB (non-zero)
• ✅ Timestamps match training run time

Filesystem result: [ ] PASS [ ] FAIL

---

Training Metrics

Actual Results (fill in from console output):

• Loaded bars: ______
• Feature vectors: ______ (should be ~50 less than loaded bars)
• Feature dimension: ______ (MUST be 225)
• State dimension: ______ (MUST be 225)
• Epochs completed: 1 / 1
• Training time: ______ seconds
• Policy updates (KL > 0): ______ / 1
• Final policy loss: ______
• Final value loss: ______
• Explained variance: ______
• GPU used: [ ] Yes [ ] No (CPU fallback)

---

🎯 Overall Scenario 1 Result

PASSED if ALL of the following are true:

1. ✅ Feature extraction shows "dim=225"
2. ✅ Model initialization shows "state_dim=225"
3. ✅ Training completes 1 epoch without errors
4. ✅ All 3 model files saved successfully
5. ✅ File sizes are non-zero (>1MB)

Final Result: [ ] ✅ PASS [ ] ⚠️ PARTIAL [ ] ❌ FAIL

Completion Time: : (HH:MM) Total Duration: ______ minutes

---

📝 Notes / Issues Encountered

Console Output (paste key lines):

[Paste critical console output here, especially lines showing "225-dimensional" and "state_dim=225"]

Errors Encountered:

[Paste any error messages here]

Troubleshooting Steps Taken:

• N/A - No issues
• Reduced batch size from 8 to 4 (GPU OOM)
• Rebuilt ML crate (cargo clean -p ml && cargo build --release -p ml)
• Other: ___________________________________

---

🔬 Medium Test (Scenario 2) - OPTIONAL

Status: [ ] Not Started [ ] In Progress [ ] Complete [ ] Skipped

Model: MAMBA-2 Dataset: ZN_FUT_90d_clean.parquet (~3.5K bars) Epochs: 10 Estimated Time: 30 minutes

Key Validations

• Wave D configuration detected: feature_count: 225
• Input shape validated: [1, 60, 225]
• All 10 epochs complete without errors
• Checkpoints saved at epoch 10
• training_losses.csv and training_metrics.json created

Result: [ ] PASS [ ] FAIL [ ] SKIPPED

Notes:

[Optional: Paste any notes from Scenario 2]

---

🏆 Full Test (Scenario 3) - OPTIONAL

Status: [ ] Not Started [ ] In Progress [ ] Complete [ ] Skipped

Models: TFT + PPO Dataset: ES_FUT_180d.parquet (~12.5K bars) Epochs: 20 (TFT) + 30 (PPO) Estimated Time: 1 hour

Key Validations

• TFT trained for 20 epochs with 225 features
• PPO trained for 30 epochs with 225 features
• Policy convergence achieved (KL divergence > 0)
• Value network learning (explained variance > 0.5)
• Cross-model validation passes (TFT + PPO + MAMBA-2)

Result: [ ] PASS [ ] FAIL [ ] SKIPPED

Notes:

[Optional: Paste any notes from Scenario 3]

---

✅ Final Certification

Agent 152 Phase 5 Validation: [ ] ✅ CERTIFIED [ ] ⚠️ PARTIAL [ ] ❌ FAILED

Certification Requirements (all must be checked):

• Scenario 1 (Quick Test) completed successfully
• 225-dimensional feature extraction confirmed
• Model training with 225 input features confirmed
• Model checkpoints saved and verified
• No critical errors or blockers encountered

Certified By: ________________ (Your name/ID) Date: 2025-10-__ Time: : (HH:MM)

---

📤 Deliverables

After successful validation, attach:

1. ✅ This completed checklist
2. ✅ Console output log (full training session)
3. ✅ Screenshot showing "225-dimensional feature vectors" line
4. ✅ ls -lh ml/trained_models/ output showing saved models
5. ⏳ (Optional) training_metrics.json from MAMBA-2 if Scenario 2 completed

---

🚀 Next Steps

After Scenario 1 PASS:

1. Proceed to production training (90-180 day datasets)
2. Integrate with ML Training Service (gRPC testing)
3. Deploy to cloud GPU (if local GPU is insufficient)
4. Test TLI commands: tli tune start --model PPO

After Scenario 2 PASS (optional):

1. Multi-model retraining (DQN, PPO, MAMBA-2, TFT)
2. Hyperparameter tuning with Optuna
3. Wave Comparison Backtest (Wave C vs Wave D)

After Scenario 3 PASS (optional):

1. Full production deployment
2. Live paper trading with 225-feature models
3. Performance monitoring and validation

---

END OF CHECKLIST - Good luck with validation! 🚀