31890df312
Wave 12 Group 3 Progress: ML Training Infrastructure Improvements ## Changes Summary ### Warning Fixes (W12-16B-WARNINGS: COMPLETE) - Fixed all actionable ML library warnings (0 warnings in ml/src/) - Fixed training example warnings (train_tft.rs, train_dqn.rs, train_ppo.rs, train_mamba2_dbn.rs) - Removed 900+ lines dead code (duplicate types, orphaned tests) - Enhanced metrics output with wall-clock timing Key fixes: - ml/examples/train_tft.rs: Changed 50→225 features, removed unused imports - ml/examples/train_tft_dbn.rs: Used training_duration and feature_config properly - ml/src/trainers/tft.rs: Fixed unused metadata, removed dead code methods - ml/src/dqn/: Deleted rainbow_types.rs (828 lines duplicate code) - ml/src/trainers/ppo.rs: Enhanced value pre-training metrics output ### Training Infrastructure - Added TFT Parquet support (ml/src/trainers/tft_parquet.rs) - Completed DQN training (30 epochs, 178 min) - Completed PPO training (30 epochs, production ready) - Completed MAMBA-2 retraining (20 epochs, best epoch 15) ### Test Data - Added 180-day Parquet files: ES.FUT, NQ.FUT, 6E.FUT, ZN.FUT - Added DBN validation examples - Added 225-feature validation examples ### Model Checkpoints - DQN: dqn_final_epoch30.safetensors (production ready) - PPO: ppo_actor/critic_epoch_30.safetensors (production ready) - MAMBA-2: best_model_epoch_15.safetensors (production ready) ## Remaining Work (W12-16B+) - Implement PPO Parquet support (4-6h) - Implement MAMBA-2 Parquet support (4-6h) - Wire gRPC orchestrator for Parquet training (2-3h) - Fix lazy loading implementation (8-12h) - Complete TFT training with 225 features 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
28 lines
1015 B
Rust
28 lines
1015 B
Rust
//! Quick test to inspect Parquet schema
|
|
use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
|
|
use std::fs::File;
|
|
|
|
fn main() -> Result<(), Box<dyn std::error::Error>> {
|
|
let file = File::open("test_data/ES_FUT_180d.parquet")?;
|
|
let builder = ParquetRecordBatchReaderBuilder::try_new(file)?;
|
|
let reader = builder.build()?;
|
|
|
|
// Get the first batch to inspect schema
|
|
if let Some(batch_result) = reader.into_iter().next() {
|
|
let batch = batch_result?;
|
|
println!("Schema: {:?}", batch.schema());
|
|
println!("\nColumn 0 (timestamp):");
|
|
println!(" Data type: {:?}", batch.column(0).data_type());
|
|
println!(" Null count: {}", batch.column(0).null_count());
|
|
println!(" Length: {}", batch.column(0).len());
|
|
|
|
// Try to print first few values
|
|
println!("\nFirst 3 values:");
|
|
use arrow::array::Array;
|
|
let col = batch.column(0);
|
|
println!(" Value type: {}", std::any::type_name_of_val(&col));
|
|
}
|
|
|
|
Ok(())
|
|
}
|