# Imbalance Bars Default + Feature Flag Cleanup Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Make information-driven imbalance bars (López de Prado) the default data source for DQN training when MBP-10 data is available, remove 8 enable_* boolean flags from FeatureConfig, and make min_hold_bars configurable for A/B testing.

**Architecture:** Three independent subsystems: (1) Wire MBP-10 → ImbalanceBarSampler → existing feature pipeline in the DQN data loader, falling back to OHLCV when MBP-10 is absent. (2) Remove all enable_* flags from ml-features FeatureConfig — all feature groups always active. (3) Make min_hold_bars a TOML-configurable + CLI-overridable parameter.

**Tech Stack:** Rust (crates/ml-features, crates/ml, crates/ml-dqn), Databento DBN decoder, CUDA kernels unchanged

---

## File Structure

| File | Responsibility | Changes |
|------|---------------|---------|
| `crates/ml/src/trainers/dqn/data_loading.rs` | DQN training data pipeline | Add MBP-10 → imbalance bars path |
| `crates/ml-features/src/config.rs` | Feature group configuration | Remove 8 enable_* flags |
| `crates/ml-features/src/pipeline.rs` | Feature extraction pipeline | Remove conditional feature gating |
| `crates/ml-features/src/mbp10_loader.rs` | MBP-10 file loading | Already exists, may need tick extraction |
| `crates/ml-features/src/alternative_bars.rs` | ImbalanceBarSampler | Already exists, production-ready |
| `crates/ml/src/trainers/dqn/config.rs` | DQN hyperparameters | Add `imbalance_bar_threshold`, `min_hold_bars` |
| `crates/ml/src/training_profile.rs` | TOML deserialization | Wire new params |
| `crates/ml/examples/train_baseline_rl.rs` | Training CLI | Add `--min-hold-bars` |
| `crates/ml/examples/hyperopt_baseline_rl.rs` | Hyperopt CLI | Add `--min-hold-bars` |
| `config/training/dqn-production.toml` | Production config | Add imbalance_bar_threshold, min_hold_bars |
| `infra/k8s/argo/compile-and-train-template.yaml` | Argo workflow | Wire --min-hold-bars |

---

## Phase 1: Remove enable_* flags from FeatureConfig

### Task 1: Remove 8 enable_* boolean fields from FeatureConfig

The `crates/ml-features/src/config.rs` FeatureConfig struct has 8 `enable_*: bool` flags that violate the "no feature flags" rule. All feature groups should always be active. The conditional `if self.enable_X { count += N }` logic creates dead code paths and false impressions that features can be toggled.

**Files:**
- Modify: `crates/ml-features/src/config.rs` — remove 8 fields, collapse conditionals
- Modify: `crates/ml-features/src/pipeline.rs` — remove `if config.enable_X` guards
- Modify: `crates/ml-features/src/lib.rs` — update re-exports if needed

- [ ] **Step 1: Read the FeatureConfig struct and all its methods**

Read `crates/ml-features/src/config.rs` fully. Identify:
- The 8 `enable_*: bool` fields (lines ~219-247)
- The `feature_count()` method that conditionally sums features
- The `feature_indices()` method that conditionally assigns index ranges
- The `is_enabled()` method on FeatureGroup enum
- The 4 constructor fns: `default()`, `wave_b()`, `wave_c()`, `wave_d()`
- All tests that check `enable_*` values

- [ ] **Step 2: Remove the 8 boolean fields from the struct**

Delete these fields from `FeatureConfig`:
```
pub enable_ohlcv: bool,
pub enable_technical_indicators: bool,
pub enable_microstructure: bool,
pub enable_alternative_bars: bool,
pub enable_barrier_optimization: bool,
pub enable_fractional_diff: bool,
pub enable_regime_detection: bool,
pub enable_wave_d_regime: bool,
```

- [ ] **Step 3: Collapse feature_count() — always count all groups**

Remove all `if self.enable_X { count += N }` conditionals. The total feature count is the sum of ALL groups unconditionally.

- [ ] **Step 4: Collapse feature_indices() — always assign all ranges**

Remove all `if self.enable_X { indices.X = Some(...) }` conditionals. All index ranges always assigned.

- [ ] **Step 5: Collapse is_enabled() — always return true**

The `is_enabled(group: FeatureGroup) -> bool` method should just return `true` for all groups, or be removed entirely. If callers exist, replace with `true`.

- [ ] **Step 6: Simplify constructors**

Remove all `enable_X: true/false` from `default()`, `wave_b()`, `wave_c()`, `wave_d()`. The phased wave system no longer makes sense when all features are always active — consider collapsing to a single constructor.

- [ ] **Step 7: Fix pipeline.rs**

In `crates/ml-features/src/pipeline.rs`, remove any `if config.is_enabled(FeatureGroup::X)` guards. All feature extraction paths always execute.

- [ ] **Step 8: Fix all compilation errors**

```bash
SQLX_OFFLINE=true cargo check -p ml-features -p ml 2>&1 | head -80
```

Grep for remaining `enable_` references and fix them:
```bash
grep -rn 'enable_ohlcv\|enable_technical\|enable_microstructure\|enable_alternative\|enable_barrier\|enable_fractional\|enable_regime\|enable_wave_d' crates/ml-features/ crates/ml/
```

- [ ] **Step 9: Update tests**

Tests that assert `!config.enable_alternative_bars` or `config.enable_ohlcv` need to be deleted or rewritten. Tests that assert `feature_count()` need updated counts (now always the maximum).

- [ ] **Step 10: Compile check and commit**

```bash
cargo check --workspace 2>&1 | head -80
```

```bash
git add -u crates/ml-features/ crates/ml/
git commit -m "cleanup: remove 8 enable_* flags from FeatureConfig — all features always active"
```

---

## Phase 2: Wire MBP-10 → Imbalance Bars → DQN Pipeline

### Task 2: Add MBP-10 → tick extraction function

The `mbp10_loader.rs` loads `Mbp10Snapshot` records. We need a function that extracts trade ticks from MBP-10 snapshots (MBP-10 contains trades as actions with `action == Trade`).

**Files:**
- Modify: `crates/ml-features/src/mbp10_loader.rs` — add `extract_trades_from_mbp10()`

- [ ] **Step 1: Read mbp10_loader.rs to understand the Mbp10Snapshot struct**

Understand what fields are available: timestamp, bid/ask prices and sizes at 10 levels, action type.

- [ ] **Step 2: Add trade extraction function**

```rust
/// Extract trade ticks from MBP-10 snapshots.
/// Filters for action == Trade, returns (price, volume, timestamp, is_buy).
pub fn extract_trades_from_mbp10(
    snapshots: &[Mbp10Snapshot],
) -> Vec<(f64, f64, chrono::DateTime<chrono::Utc>, bool)> {
    // Filter for trade actions, extract price/volume/side
    // is_buy determined by comparing trade price to mid-price
}
```

- [ ] **Step 3: Compile check**

```bash
SQLX_OFFLINE=true cargo check -p ml-features 2>&1 | head -40
```

- [ ] **Step 4: Commit**

```bash
git add crates/ml-features/src/mbp10_loader.rs
git commit -m "feat: extract trade ticks from MBP-10 snapshots"
```

---

### Task 3: Add MBP-10 → imbalance bars pipeline function

Create a function that takes an MBP-10 data directory and produces `Vec<OHLCVBar>` via `ImbalanceBarSampler`.

**Files:**
- Modify: `crates/ml-features/src/mbp10_loader.rs` — add `mbp10_to_imbalance_bars()`

- [ ] **Step 1: Add the pipeline function**

```rust
/// Load MBP-10 files from a directory and convert to imbalance bars.
///
/// Pipeline: MBP-10 .dbn.zst → trade extraction → ImbalanceBarSampler → OHLCVBar
///
/// Falls back to empty Vec if directory doesn't exist or has no files.
pub fn mbp10_to_imbalance_bars(
    mbp10_dir: &Path,
    symbol: &str,
    threshold: f64,
    ewma_alpha: f64,
) -> Result<Vec<OHLCVBar>, MLError> {
    // 1. Find all .dbn.zst files in mbp10_dir/symbol/
    // 2. Load each file with load_mbp10_snapshots_sync()
    // 3. Extract trades from snapshots
    // 4. Feed trades into ImbalanceBarSampler::new_with_ewma()
    // 5. Collect emitted OHLCVBars
    // 6. Sort by timestamp
}
```

- [ ] **Step 2: Add unit test with test_data/mbp10/**

```rust
#[test]
fn test_mbp10_to_imbalance_bars() {
    let bars = mbp10_to_imbalance_bars(
        Path::new("test_data/mbp10"),
        "ES.FUT",
        100.0,  // threshold
        0.1,    // ewma_alpha
    ).unwrap();
    assert!(!bars.is_empty(), "Should produce imbalance bars from MBP-10 data");
    // Verify bars are time-ordered
    for w in bars.windows(2) {
        assert!(w[0].timestamp <= w[1].timestamp);
    }
}
```

- [ ] **Step 3: Compile and test**

```bash
SQLX_OFFLINE=true cargo test -p ml-features --lib -- mbp10_to_imbalance_bars
```

- [ ] **Step 4: Commit**

```bash
git add crates/ml-features/src/mbp10_loader.rs
git commit -m "feat: MBP-10 → imbalance bars pipeline via ImbalanceBarSampler"
```

---

### Task 4: Wire imbalance bars into DQN data loader

This is the critical integration. The DQN data loader currently only reads OHLCV `.dbn.zst` files. Add a path that tries MBP-10 first, falls back to OHLCV.

**Files:**
- Modify: `crates/ml/src/trainers/dqn/data_loading.rs` — add MBP-10 loading path
- Modify: `crates/ml/src/trainers/dqn/config.rs` — add `imbalance_bar_threshold: f64`

- [ ] **Step 1: Add hyperparameter**

In `crates/ml/src/trainers/dqn/config.rs`, add to `DQNHyperparameters`:
```rust
/// Imbalance bar threshold for MBP-10 → bar conversion.
/// Higher = fewer bars (less noise), lower = more bars (more signal).
/// Only used when mbp10_data_dir is set. Default: 100.0.
pub imbalance_bar_threshold: f64,

/// EWMA smoothing alpha for adaptive imbalance threshold.
/// 0.1 = slow adaptation, 0.5 = fast. Default: 0.1.
pub imbalance_bar_ewma_alpha: f64,
```

Default: `imbalance_bar_threshold: 100.0, imbalance_bar_ewma_alpha: 0.1,`

- [ ] **Step 2: Modify data loading to try MBP-10 first**

In `crates/ml/src/trainers/dqn/data_loading.rs`, find where OHLCV files are loaded (the `extract_ohlcv_bars_from_dbn` call). Before it, add:

```rust
// Try MBP-10 → imbalance bars first (information-driven sampling)
let ohlcv_bars = if let Some(ref mbp10_dir) = self.hyperparams.mbp10_data_dir {
    let mbp10_path = Path::new(mbp10_dir);
    if mbp10_path.exists() {
        info!("Loading MBP-10 data from {} → imbalance bars", mbp10_dir);
        let bars = ml_features::mbp10_loader::mbp10_to_imbalance_bars(
            mbp10_path,
            &symbol,
            self.hyperparams.imbalance_bar_threshold,
            self.hyperparams.imbalance_bar_ewma_alpha,
        )?;
        if !bars.is_empty() {
            info!("Produced {} imbalance bars from MBP-10 (vs ~{} 1-min OHLCV bars)",
                bars.len(), bars.len() / 5); // rough estimate
            bars
        } else {
            warn!("MBP-10 produced no bars, falling back to OHLCV");
            self.extract_ohlcv_bars_from_dbn(file_path)?
        }
    } else {
        self.extract_ohlcv_bars_from_dbn(file_path)?
    }
} else {
    self.extract_ohlcv_bars_from_dbn(file_path)?
};
```

IMPORTANT: Read the actual data loading flow first. The MBP-10 files are in a different directory (`mbp10_data_dir`) than the OHLCV files (`data_dir`). The function needs to scan the MBP-10 dir for files matching the symbol, not the OHLCV dir.

- [ ] **Step 3: Wire TOML deserialization**

In `crates/ml/src/training_profile.rs`, add to the relevant section:
```toml
imbalance_bar_threshold = 100.0
imbalance_bar_ewma_alpha = 0.1
```

- [ ] **Step 4: Add to TOML configs**

`config/training/dqn-production.toml` under `[experience]`:
```toml
imbalance_bar_threshold = 100.0
imbalance_bar_ewma_alpha = 0.1
```

Same for `dqn-smoketest.toml` and `dqn-localdev.toml`.

- [ ] **Step 5: Compile check**

```bash
cargo check --workspace 2>&1 | head -80
```

- [ ] **Step 6: Commit**

```bash
git add -u crates/ml/ config/training/
git commit -m "feat: wire MBP-10 → imbalance bars as default DQN data source"
```

---

### Task 5: Update Argo workflow to log bar type

**Files:**
- Modify: `infra/k8s/argo/compile-and-train-template.yaml` — add logging for data source

- [ ] **Step 1: Add data source logging to hyperopt and train steps**

In the hyperopt and train-best shell scripts, add after the data directory listing:
```bash
echo "Bar type: imbalance (MBP-10) if available, OHLCV 1-min fallback"
```

- [ ] **Step 2: Commit**

```bash
git add infra/k8s/argo/compile-and-train-template.yaml
git commit -m "docs: log bar type in Argo training workflow"
```

---

## Phase 3: Make min_hold_bars Configurable

### Task 6: Wire min_hold_bars through CLI and TOML

Currently `min_hold_bars` is hardcoded at 5 in the TOML and fixed in DQNHyperparameters. Make it overridable via CLI for A/B testing (3 vs 5).

**Files:**
- Modify: `crates/ml/examples/train_baseline_rl.rs` — add `--min-hold-bars` CLI arg
- Modify: `crates/ml/examples/hyperopt_baseline_rl.rs` — add `--min-hold-bars` CLI arg
- Modify: `infra/k8s/argo/compile-and-train-template.yaml` — add workflow parameter

- [ ] **Step 1: Add CLI arg to train_baseline_rl.rs**

```rust
/// Minimum bars to hold a position before allowing exit (churn prevention).
/// Lower = more trades (faster cycling), higher = fewer trades (longer holds).
/// Default: 5 (from TOML). Set to 3 for aggressive OFI-driven trading.
#[arg(long)]
min_hold_bars: Option<usize>,
```

Wire: `if let Some(mhb) = args.min_hold_bars { hyperparams.min_hold_bars = mhb; }`

- [ ] **Step 2: Add CLI arg to hyperopt_baseline_rl.rs**

Same pattern as train_baseline_rl.

- [ ] **Step 3: Add workflow parameter**

In `compile-and-train-template.yaml`, add parameter:
```yaml
- name: min-hold-bars
  value: "5"
```

Pass to hyperopt and train-best steps:
```bash
--min-hold-bars {{workflow.parameters.min-hold-bars}}
```

- [ ] **Step 4: Compile check**

```bash
cargo check -p ml --example train_baseline_rl --example hyperopt_baseline_rl 2>&1 | head -40
```

- [ ] **Step 5: Commit**

```bash
git add crates/ml/examples/ infra/k8s/argo/compile-and-train-template.yaml
git commit -m "feat: configurable --min-hold-bars for A/B testing (3 vs 5)"
```

---

## Verification Checklist

After all phases:

- [ ] `cargo check --workspace` — full workspace compiles
- [ ] `grep -rn 'enable_ohlcv\|enable_technical\|enable_microstructure\|enable_alternative\|enable_barrier\|enable_fractional\|enable_regime_detection\|enable_wave_d' crates/ml-features/src/config.rs` — zero matches
- [ ] `SQLX_OFFLINE=true cargo test -p ml-features --lib` — all feature tests pass
- [ ] `cargo test -p ml --lib -- smoke_tests --ignored --nocapture` — smoke tests pass (OHLCV fallback works when no MBP-10)
- [ ] `mbp10_to_imbalance_bars()` produces bars from `test_data/mbp10/`
- [ ] `dqn-production.toml` has `imbalance_bar_threshold`, `imbalance_bar_ewma_alpha`, `min_hold_bars`
- [ ] `train_baseline_rl --help` shows `--min-hold-bars` and `--initial-capital`
- [ ] All 3 Argo train templates pass `--min-hold-bars` parameter
