8ce7c52586
- Fixed feature dimension mismatch in evaluate_dqn_main_orchestrator.rs - Updated all 5 occurrences: state_dim, input comments, feature vector type - Aligned with Wave 16D training (128 features: 125 market + 3 portfolio) Issue: Validation backtest reveals 100% HOLD action collapse - requires reward system investigation and redesign per latest RL research.
100 lines
3.3 KiB
Plaintext
100 lines
3.3 KiB
Plaintext
WAVE 16C SMOKE TEST - QUICK REFERENCE
|
|
=====================================
|
|
|
|
STATUS: ❌ NO-GO - Critical Integration Failures
|
|
|
|
FAILURES
|
|
--------
|
|
1. Feature Count Mismatch (CATASTROPHIC)
|
|
- Declared: 125 features (Wave 16A)
|
|
- Actual: 225 features extracted
|
|
- Model expects: 125 inputs
|
|
- Result: Shape mismatch → crash at first layer
|
|
- Impact: Cannot train, cannot validate Polyak averaging
|
|
|
|
2. Preprocessing Crash (CRITICAL)
|
|
- Error: "Failed to preprocess prices"
|
|
- Location: ml/src/trainers/dqn.rs:1195
|
|
- Details: Vague error, no diagnostic info
|
|
- Impact: Cannot use log returns + normalization
|
|
|
|
TRIAL OUTCOMES
|
|
--------------
|
|
Trial 1: CRASHED (0/3 completed, 0%)
|
|
- With preprocessing: Crashes immediately
|
|
- Without preprocessing: Shape mismatch crash
|
|
- Gradient norms: N/A (no training occurred)
|
|
- Duration: ~14 seconds (setup only, 0s training)
|
|
|
|
CONFIGURATION STATUS
|
|
-------------------
|
|
✅ Wave 16 startup logs: PASS (correct config displayed)
|
|
✅ Polyak averaging logs: PASS (tau=0.001, 692-step half-life)
|
|
❌ Feature extraction: FAIL (still extracting 225, not 125)
|
|
❌ Preprocessing: FAIL (crashes with vague error)
|
|
|
|
ROOT CAUSES
|
|
-----------
|
|
1. extract_full_features() still generates 225 features
|
|
- Location: ml/src/trainers/dqn.rs:1224
|
|
- Wave 16A feature reduction was declared but NOT implemented
|
|
- Model architecture updated to 125 inputs, data pipeline was not
|
|
|
|
2. preprocess_prices() crashes silently
|
|
- Location: ml/src/preprocessing.rs:340+
|
|
- Likely causes: NaN/Inf in data, zero division, tensor shape issues
|
|
- No diagnostic logging to identify root cause
|
|
|
|
REQUIRED FIXES
|
|
--------------
|
|
Priority 1: Fix Feature Count Mismatch (1-2h)
|
|
- Implement extract_reduced_features() with 125-feature subset
|
|
- Update 3 call sites in dqn.rs
|
|
- Test with --no-preprocessing to isolate preprocessing crash
|
|
|
|
Priority 2: Fix Preprocessing Crash (1-2h)
|
|
- Add detailed error logging to preprocessing.rs
|
|
- Test log returns + normalization in isolation
|
|
- Add input validation (NaN/Inf checks, positive price checks)
|
|
|
|
Priority 3: Rerun Smoke Test (15min)
|
|
- Execute 3-trial, 5-epoch smoke test
|
|
- Verify 1+ trials complete
|
|
- Validate gradient norms <500 (vs 1,742 baseline)
|
|
|
|
NEXT AGENT TASKS
|
|
----------------
|
|
Wave 16D Agent 40: Fix feature count mismatch (1-2h)
|
|
Wave 16D Agent 41: Fix preprocessing crash (1-2h)
|
|
Wave 16D Agent 42: Rerun smoke test (15min)
|
|
|
|
VALIDATION CRITERIA (POST-FIX)
|
|
------------------------------
|
|
GO to Wave 16E (10-trial full campaign) IF:
|
|
✅ Feature extraction logs show "125 features"
|
|
✅ No shape mismatch errors
|
|
✅ Preprocessing completes without crashes
|
|
✅ 1-3 trials complete all 5 epochs
|
|
✅ Average gradient norm <500
|
|
|
|
NO-GO (escalate) IF:
|
|
❌ Feature count still 225
|
|
❌ Preprocessing still crashes
|
|
❌ 0/3 trials complete AND gradient norms >1,000
|
|
|
|
SAVED OUTPUTS
|
|
-------------
|
|
Report: /home/jgrusewski/Work/foxhunt/WAVE_16C_SMOKE_TEST_REPORT.md
|
|
Logs: /tmp/dqn_wave16c_smoke_test_final.log
|
|
/tmp/dqn_wave16c_no_preproc.log
|
|
/tmp/dqn_wave16c_debug.log
|
|
|
|
KEY INSIGHT
|
|
-----------
|
|
Wave 16A-B implemented the ARCHITECTURE changes (model layer sizes,
|
|
Polyak tau, preprocessing config) but did NOT wire the feature
|
|
extraction changes into the data loading code. Classic integration
|
|
failure - components work in isolation but fail when combined.
|
|
|
|
ESTIMATED TIME TO FIX: 2-4 hours total (Priorities 1+2+3)
|