# data

Market data ingestion, broker integration, and feature extraction.

## Providers

- **Databento** -- Historical and real-time market data via DBN format
- **Benzinga** -- News and fundamentals feed

## Broker Integrations

- **IB TWS** -- Interactive Brokers TWS/Gateway socket connection
- **ICMarkets** -- FIX 4.4 protocol integration

## Key Modules

- **brokers** -- Broker adapters (IB TWS, ICMarkets FIX)
- **providers** -- Data provider clients (Databento, Benzinga)
- **dbn_uploader** -- DBN file upload and processing
- **parquet_persistence** -- Parquet read/write for tick and bar data
- **replay** -- Market data replay for backtesting
- **training_pipeline** -- Data preparation for ML model training
- **features** -- Technical indicator and feature computation
- **unified_feature_extractor** -- Normalized feature vectors across providers
- **validation** -- Data quality checks and schema validation

## Cargo Features

| Feature       | Default | Description                        |
|---------------|---------|------------------------------------|
| `databento`   | yes     | Databento provider support         |
| `benzinga`    | yes     | Benzinga provider support          |
| `icmarkets`   | yes     | ICMarkets FIX 4.4 integration      |
| `redis-cache` | no      | Redis caching layer                |
| `ib`          | no      | Interactive Brokers TWS adapter    |
| `mock`        | no      | Mock providers for testing         |

## Testing

```bash
SQLX_OFFLINE=true cargo test -p data --lib
```