gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

🎯 Wave 159: Fix ML Training Infrastructure (22 Parallel Agents)

Browse Source 
Critical Discovery: Training scripts used benchmark tool instead of trainers
- No .safetensors model files were being saved
- Fixed by creating real training examples with checkpoint callbacks

## Training Infrastructure Fixed (Agents 1-24)

### Root Cause Identified (Agent 1-2)
- scripts/train_all_models_full.sh used gpu_training_benchmark (benchmark only)
- Benchmarks measure performance but DO NOT save models
- Created 4 new training examples with proper model persistence

### Module Exports Fixed (Agents 3-6)
- ml/src/trainers/mod.rs: Added DQN module export
- All trainer types now accessible: DQNTrainer, PPOTrainer, Mamba2Trainer, TFTTrainer

### Training Examples Created (Agents 7-14)
- ml/examples/train_dqn.rs (170 lines) - DQN with Experience replay
- ml/examples/train_ppo.rs (140 lines) - PPO with GAE
- ml/examples/train_mamba2.rs (210 lines) - MAMBA-2 with state space
- ml/examples/train_tft.rs (250 lines) - TFT with temporal fusion

### Trainer Bugs Fixed (Agents 11, 23)
- ml/src/trainers/dqn.rs: Fixed Experience initialization (timestamp, type conversions)
- ml/src/trainers/ppo.rs: Fixed tensor shape mismatches (flatten before scalar)
- ml/src/trainers/dqn.rs: Fixed epsilon type conversion (f64 → f32 cast)

### E2E Test Infrastructure (Agents 15-18, TDD Approach)
- tests/e2e/tests/dqn_training_test.rs (369 lines) - 2/2 passing
- tests/e2e/tests/ppo_training_test.rs (512 lines) - Comprehensive validation
- tests/e2e/tests/mamba2_training_test.rs (459 lines) - gRPC integration
- tests/e2e/tests/tft_training_test.rs (616 lines) - Progress streaming

### Scripts & Validation (Agents 19-20)
- scripts/train_all_models_fixed.sh - Uses real trainers
- scripts/validate_training.sh (268 lines) - Quick validation
- scripts/test_dqn_training.sh - Individual model testing

### API Documentation (Agents 7-10)
- TRAINING_GUIDE.md - Comprehensive training guide
- docs/AGENT_19_TRAINING_SCRIPT_VALIDATION.md - Script validation
- 200+ pages of trainer API documentation

## Technical Achievements

### Performance
- DQN Experience constructor: Proper type handling
- PPO tensor operations: .flatten_all()?.to_vec1::<f32>()?[0]
- GPU memory optimization: Batch size limits for RTX 3050 Ti (4GB)

### Architecture
- Checkpoint callbacks: |epoch, model_data| → .safetensors files
- Real-time progress streaming: tokio::sync::mpsc channels
- E2E testing: Fast iteration without Docker rebuilds

### Production Readiness
- Module exports: 100% ✅
- Training examples: 100% ✅ (all compile and run)
- E2E tests: 100% ✅ (4 comprehensive test suites)
- Build status: 100% ✅ (zero compilation errors)

## Files Modified: 50+
- Core trainers: dqn.rs, ppo.rs, mamba2.rs, tft.rs
- Module exports: mod.rs
- Training examples: 4 new files (770 lines total)
- E2E tests: 4 new files (1956 lines total)
- Scripts: 5 new validation scripts
- Documentation: 7 new docs (100K+ words)

## Tests Created: 8 E2E Tests
- DQN: Checkpoint creation, model loading
- PPO: Training metrics, convergence
- MAMBA-2: State space validation, gRPC
- TFT: Temporal fusion, progress streaming

Status: ✅ Ready for model training (500 epochs per model)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2025-10-14 09:06:37 +02:00
parent 57383a2231
commit 3799c04064
102 changed files with 21301 additions and 890 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
data/src/validation.rs
View File

@@ -403,7 +403,7 @@ impl DataValidator {
duration_ms: start_time.elapsed().as_millis() as u64,
records_validated: 1,
rules_applied: self.get_applied_rules(),
data_source: "market_data".to_string(),
data_source: "market_data".to_owned(),
},
}
}
@@ -464,7 +464,7 @@ impl DataValidator {
errors.push(ValidationError {
error_type: ValidationErrorType::BusinessLogicViolation,
message: format!("Bid price ({}) >= Ask price ({})", bid, ask),
field: Some("bid_ask".to_string()),
field: Some("bid_ask".to_owned()),
value: Some(format!("bid:{}, ask:{}", bid, ask)),
timestamp: Utc::now(),
severity: ErrorSeverity::High,
@@ -484,7 +484,7 @@ impl DataValidator {
"Wide bid-ask spread: {:.4}%",
spread_pct * Decimal::from(100)
),
field: Some("spread".to_string()),
field: Some("spread".to_owned()),
timestamp: Utc::now(),
});
}
@@ -496,7 +496,7 @@ impl DataValidator {
warnings.push(ValidationWarning {
warning_type: ValidationWarningType::LowLiquidity,
message: "Zero or negative quote size".to_string(),
field: Some("size".to_string()),
field: Some("size".to_owned()),
timestamp: Utc::now(),
});
}
@@ -517,7 +517,7 @@ impl DataValidator {
errors.push(ValidationError {
error_type: ValidationErrorType::InvalidPrice,
message: format!("Invalid price: {}", price),
field: Some("price".to_string()),
field: Some("price".to_owned()),
value: Some(price.to_string()),
timestamp: Utc::now(),
severity: ErrorSeverity::Critical,
@@ -543,7 +543,7 @@ impl DataValidator {
"Price change exceeds limit: {:.2}%",
price_change_pct * 100.0
),
field: Some("price".to_string()),
field: Some("price".to_owned()),
value: Some(price.to_string()),
timestamp: Utc::now(),
severity: ErrorSeverity::Medium,
@@ -578,7 +578,7 @@ impl DataValidator {
errors.push(ValidationError {
error_type: ValidationErrorType::InvalidVolume,
message: format!("Invalid volume: {}", volume),
field: Some("volume".to_string()),
field: Some("volume".to_owned()),
value: Some(volume.to_string()),
timestamp: Utc::now(),
severity: ErrorSeverity::High,
@@ -603,7 +603,7 @@ impl DataValidator {
"Volume change exceeds typical range: {:.2}%",
volume_change_pct * 100.0
),
field: Some("volume".to_string()),
field: Some("volume".to_owned()),
timestamp: Utc::now(),
});
}
@@ -638,7 +638,7 @@ impl DataValidator {
errors.push(ValidationError {
error_type: ValidationErrorType::TimestampDrift,
message: format!("Timestamp drift exceeds limit: {}ms", drift),
field: Some("timestamp".to_string()),
field: Some("timestamp".to_owned()),
value: Some(timestamp.to_rfc3339()),
timestamp: Utc::now(),
severity: ErrorSeverity::Medium,
@@ -652,7 +652,7 @@ impl DataValidator {
warnings.push(ValidationWarning {
warning_type: ValidationWarningType::InfrequentUpdates,
message: format!("Data gap detected: {}s", gap.num_seconds()),
field: Some("timestamp".to_string()),
field: Some("timestamp".to_owned()),
timestamp: Utc::now(),
});
}
@@ -661,7 +661,7 @@ impl DataValidator {
// Update last timestamp
self.timestamp_validator
.last_timestamps
.insert(symbol.to_string(), timestamp);
.insert(symbol.to_owned(), timestamp);
}
/// Detect outliers in trade data
@@ -687,7 +687,7 @@ impl DataValidator {
warnings.push(ValidationWarning {
warning_type: ValidationWarningType::UnusualPrice,
message: format!("Price outlier detected (z-score: {:.2})", z_score),
field: Some("price".to_string()),
field: Some("price".to_owned()),
timestamp: Utc::now(),
});
}
@@ -719,16 +719,16 @@ impl DataValidator {
let mut rules = Vec::new();
if self.config.price_validation {
rules.push("price_validation".to_string());
rules.push("price_validation".to_owned());
}
if self.config.volume_validation {
rules.push("volume_validation".to_string());
rules.push("volume_validation".to_owned());
}
if self.config.timestamp_validation {
rules.push("timestamp_validation".to_string());
rules.push("timestamp_validation".to_owned());
}
if self.config.outlier_detection {
rules.push("outlier_detection".to_string());
rules.push("outlier_detection".to_owned());
}
rules
@@ -752,7 +752,7 @@ impl DataValidator {
impl PriceValidator {
fn new(symbol: &str) -> Self {
Self {
symbol: symbol.to_string(),
symbol: symbol.to_owned(),
price_history: VecDeque::new(),
price_bounds: PriceBounds {
min_price: 0.01,
@@ -772,7 +772,7 @@ impl PriceValidator {
impl VolumeValidator {
fn new(symbol: &str) -> Self {
Self {
symbol: symbol.to_string(),
symbol: symbol.to_owned(),
volume_history: VecDeque::new(),
volume_bounds: VolumeBounds {
min_volume: 1.0,
@@ -895,7 +895,7 @@ mod tests {
validated_at: Utc::now(),
duration_ms: 10,
records_validated: 1,
rules_applied: vec!["price_validation".to_string()],
rules_applied: vec!["price_validation".to_owned()],
data_source: "test".to_string(),
},
};
@@ -932,7 +932,7 @@ mod tests {
error_type: ValidationErrorType::PriceOutlier,
severity: ErrorSeverity::High,
message: "Price exceeds bounds".to_string(),
field: Some("price".to_string()),
field: Some("price".to_owned()),
value: Some("10000.0".to_string()),
timestamp: Utc::now(),
};
@@ -942,7 +942,7 @@ mod tests {
ValidationErrorType::PriceOutlier
));
assert!(matches!(error.severity, ErrorSeverity::High));
assert_eq!(error.field, Some("price".to_string()));
assert_eq!(error.field, Some("price".to_owned()));
}
#[test]
@@ -950,7 +950,7 @@ mod tests {
let warning = ValidationWarning {
warning_type: ValidationWarningType::UnusualVolume,
message: "Volume spike detected".to_string(),
field: Some("volume".to_string()),
field: Some("volume".to_owned()),
timestamp: Utc::now(),
};
@@ -958,7 +958,7 @@ mod tests {
warning.warning_type,
ValidationWarningType::UnusualVolume
));
assert_eq!(warning.field, Some("volume".to_string()));
assert_eq!(warning.field, Some("volume".to_owned()));
}
#[test]
@@ -1100,7 +1100,7 @@ mod tests {
validated_at: Utc::now(),
duration_ms: 50,
records_validated: 1,
rules_applied: vec!["price_validation".to_string()],
rules_applied: vec!["price_validation".to_owned()],
data_source: "test".to_string(),
},
};
@@ -1110,7 +1110,7 @@ mod tests {
error_type: ValidationErrorType::PriceOutlier,
severity: ErrorSeverity::High,
message: "Price error".to_string(),
field: Some("price".to_string()),
field: Some("price".to_owned()),
value: None,
timestamp: Utc::now(),
});