foxhunt/DQN_PORTFOLIO_FEATURES_DESIGN.md

# DQN Portfolio Features Bug - Design Solution

**Agent**: Wave 1 - Agent 4
**Date**: 2025-11-04
**Bug Location**: `ml/src/trainers/dqn.rs:1571-1580`
**Status**: Design Complete

---

## Executive Summary

The `feature_vector_to_state()` method in DQNTrainer creates `TradingState` objects with **empty portfolio_features**, causing P&L reward calculations to **always return 0.0**. This breaks the reward signal, making the DQN agent unable to learn profitable trading strategies.

**Impact**: CRITICAL - DQN cannot learn without P&L rewards

---

## Required Portfolio Features

### Analysis of `ml/src/dqn/reward.rs:145-200`

The reward calculation requires the following portfolio state fields:

| Index | Field | Type | Usage | Required By |
|-------|-------|------|-------|-------------|
| `[0]` | **Portfolio Value** | f32 | P&L calculation | `calculate_pnl_reward()` (lines 152-156) |
| `[1]` | **Position Size** | f32 | Risk penalty & transaction costs | `calculate_risk_penalty()` (line 172), `calculate_cost_penalty()` (line 193) |
| `[2]` | **Spread** | f32 | Transaction cost estimation | `calculate_cost_penalty()` (line 200) |

**Minimal Requirements**:
- **portfolio_features[0]**: Portfolio value (cash + positions)
- **portfolio_features[1]**: Current position size (signed: +Long, -Short, 0 for flat)
- **portfolio_features[2]**: Bid-ask spread (for transaction costs)

**Default values** if missing:
- Portfolio value: 10,000.0 (default initial capital)
- Position size: 0.0 (flat position)
- Spread: 0.0001 (1 basis point)

---

## Current Code Flow

```rust
// ml/src/trainers/dqn.rs:1554-1582
fn feature_vector_to_state(&self, feature_vec: &FeatureVector225) -> Result<TradingState> {
    let price_features: Vec<f32> = vec![
        feature_vec[0] as f32, // open log return
        feature_vec[1] as f32, // high log return
        feature_vec[2] as f32, // low log return
        feature_vec[3] as f32, // close log return
    ];

    let technical_indicators: Vec<f32> = feature_vec[4..].iter().map(|&v| v as f32).collect();

    let market_features = vec![];       // ❌ Empty!
    let portfolio_features = vec![];    // ❌ Empty! (BUG)

    Ok(TradingState::from_normalized(
        price_features,
        technical_indicators,
        market_features,
        portfolio_features,
    ))
}
```

**Problem**: No portfolio state tracking between time steps.

---

## Design Approaches

### Approach A: Track Portfolio State in DQNTrainer ✅ RECOMMENDED

**Architecture**:
```
┌──────────────────────────────────────────────────────────────┐
│                        DQNTrainer                             │
├──────────────────────────────────────────────────────────────┤
│  Fields (NEW):                                                │
│  - portfolio_tracker: PortfolioTracker                        │
│                                                                │
│  Methods (MODIFIED):                                          │
│  - feature_vector_to_state() → includes portfolio state       │
│  - process_training_sample() → updates portfolio after action│
│  - process_training_batch() → batched portfolio updates       │
└──────────────────────────────────────────────────────────────┘
                           │
                           │ uses
                           ▼
┌──────────────────────────────────────────────────────────────┐
│                    PortfolioTracker (NEW)                     │
├──────────────────────────────────────────────────────────────┤
│  Fields:                                                      │
│  - cash: f32                                                  │
│  - position_size: f32 (signed)                                │
│  - position_entry_price: f32                                  │
│  - initial_capital: f32                                       │
│  - avg_spread: f32                                            │
│                                                                │
│  Methods:                                                     │
│  - get_portfolio_features() → [value, size, spread]           │
│  - execute_action(action, price) → updates state              │
│  - get_portfolio_value(current_price) → cash + unrealized P&L │
│  - reset() → reinitialize for new episode                     │
└──────────────────────────────────────────────────────────────┘
```

**Data Flow**:
```
Training Loop (Epoch N, Sample i)
  │
  ├─> feature_vector_to_state(feature_vec)
  │     ├─> Extract price_features (OHLC)
  │     ├─> Extract technical_indicators (221 features)
  │     └─> portfolio_tracker.get_portfolio_features() ← NEW
  │           └─> Returns [value, position, spread]
  │
  ├─> select_action(state) → action
  │
  ├─> portfolio_tracker.execute_action(action, current_price) ← NEW
  │     ├─> BUY:  Open long, deduct cash
  │     ├─> SELL: Open short, add cash
  │     └─> HOLD: No change
  │
  ├─> next_state = feature_vector_to_state(next_feature_vec)
  │     └─> Uses UPDATED portfolio state
  │
  ├─> calculate_reward(action, state, next_state)
  │     ├─> calculate_pnl_reward() ← Uses portfolio_features[0]
  │     ├─> calculate_risk_penalty() ← Uses portfolio_features[1]
  │     └─> calculate_cost_penalty() ← Uses portfolio_features[1,2]
  │
  └─> store_experience(state, action, reward, next_state)
```

**Pros**:
- ✅ **Clean separation**: Portfolio logic in dedicated struct
- ✅ **Minimal changes**: Only modify DQNTrainer, no external APIs
- ✅ **Backwards compatible**: Existing tests don't break
- ✅ **Stateful tracking**: Portfolio persists across training steps
- ✅ **Easy testing**: PortfolioTracker can be unit tested independently

**Cons**:
- ⚠️ Requires synchronization between trainer and portfolio tracker
- ⚠️ Portfolio resets at episode boundaries (need explicit reset logic)

**Implementation Effort**: 2-3 hours
- 30 min: Implement `PortfolioTracker` struct (100 lines)
- 45 min: Modify `feature_vector_to_state()` to fetch portfolio state
- 45 min: Add `execute_action()` calls in `process_training_sample()` and `process_training_batch()`
- 30 min: Add reset logic at epoch boundaries
- 30 min: Unit tests for `PortfolioTracker`

---

### Approach B: Extend FeatureVector225

**Architecture**:
```
Current: FeatureVector225 = [f64; 225]
         ├─ [0..3]: OHLC price features
         └─ [4..224]: Technical indicators (221)

Proposed: FeatureVector228 = [f64; 228]  ← BREAKING CHANGE
          ├─ [0..3]: OHLC price features
          ├─ [4..224]: Technical indicators (221)
          └─ [225..227]: Portfolio features (3) ← NEW
               ├─ [225]: Portfolio value
               ├─ [226]: Position size
               └─ [227]: Spread
```

**Pros**:
- ✅ Self-contained state representation
- ✅ No external tracking needed
- ✅ Feature vector includes all information

**Cons**:
- ❌ **BREAKING CHANGE**: All feature extraction code must be updated
- ❌ **Circular dependency**: Features need portfolio state, but portfolio state depends on past features
- ❌ **Re-extraction overhead**: Must recompute features after each action
- ❌ **Backtesting integration**: DBN data loader must inject portfolio state
- ❌ **Neural network**: State dimension changes from 225 → 228 (retrain all models)

**Implementation Effort**: 8-12 hours (HIGH RISK)
- 2h: Update `FeatureVector225` → `FeatureVector228` across codebase
- 2h: Modify feature extraction pipeline to include portfolio state
- 2h: Update neural network configs (state_dim: 225 → 228)
- 2h: Update all DBN data loaders
- 2h: Retrain all DQN models (checkpoints incompatible)
- 2h: Update all tests

**Backwards Compatibility**: ❌ NONE - Breaks all existing checkpoints

---

### Approach C: Separate Portfolio State Tracker (Backtesting Integration)

**Architecture**:
```
┌──────────────────────────────────────────────────────────────┐
│                   BacktestingEngine (EXISTING)                │
│                  (ml/src/evaluation/engine.rs)                │
├──────────────────────────────────────────────────────────────┤
│  Fields:                                                      │
│  - current_position: Option<Position>                         │
│  - trades: Vec<Trade>                                         │
│  - initial_capital: f32                                       │
│  - action_counts: [usize; 3]                                  │
│                                                                │
│  Methods:                                                     │
│  - process_bar(bar, action) → updates position                │
│  - close_position() → calculates P&L                          │
│  - get_portfolio_features() → [value, size, spread] ← NEW     │
└──────────────────────────────────────────────────────────────┘
                           │
                           │ used by
                           ▼
┌──────────────────────────────────────────────────────────────┐
│                        DQNTrainer                             │
├──────────────────────────────────────────────────────────────┤
│  Fields (NEW):                                                │
│  - backtesting_engine: Option<BacktestingEngine>              │
│                                                                │
│  Methods (MODIFIED):                                          │
│  - feature_vector_to_state() → queries backtesting engine     │
│  - process_training_sample() → calls backtesting engine       │
└──────────────────────────────────────────────────────────────┘
```

**Pros**:
- ✅ Reuses existing `BacktestingEngine` from evaluation
- ✅ Unified portfolio tracking for training and evaluation
- ✅ Already has position management logic

**Cons**:
- ❌ **Tight coupling**: DQNTrainer depends on backtesting module
- ❌ **Circular dependency**: Backtesting designed for EVALUATION, not TRAINING
- ❌ **API mismatch**: BacktestingEngine expects `OHLCVBar`, trainer has `FeatureVector225`
- ❌ **Performance overhead**: BacktestingEngine tracks trade history (not needed during training)
- ❌ **Complexity**: Mixing training and evaluation concerns

**Implementation Effort**: 6-8 hours
- 3h: Refactor `BacktestingEngine` to support training mode
- 2h: Adapt API to work with `FeatureVector225` instead of `OHLCVBar`
- 2h: Integrate into DQNTrainer
- 1h: Unit tests

**Backwards Compatibility**: ⚠️ Requires refactoring existing evaluation code

---

## Recommended Approach: A (Track Portfolio State in DQNTrainer)

### Why Approach A?

| Criterion | Score | Justification |
|-----------|-------|---------------|
| **Simplicity** | ⭐⭐⭐⭐⭐ | Clean separation, minimal changes |
| **Backwards Compatibility** | ⭐⭐⭐⭐⭐ | No breaking changes |
| **Implementation Time** | ⭐⭐⭐⭐⭐ | 2-3 hours vs. 8-12h (B) or 6-8h (C) |
| **Performance** | ⭐⭐⭐⭐⭐ | No overhead, O(1) portfolio state access |
| **Testability** | ⭐⭐⭐⭐⭐ | PortfolioTracker is unit-testable |
| **Maintainability** | ⭐⭐⭐⭐ | Single-purpose struct |

**Decision**: Approach A provides **maximum value** with **minimum risk** and **fastest delivery**.

---

## Implementation Pseudo-Code (Approach A)

### 1. PortfolioTracker Struct

```rust
// NEW FILE: ml/src/dqn/portfolio_tracker.rs

#[derive(Debug, Clone)]
pub struct PortfolioTracker {
    /// Current cash balance
    cash: f32,
    /// Current position size (positive = long, negative = short, 0 = flat)
    position_size: f32,
    /// Entry price for current position
    position_entry_price: f32,
    /// Initial capital (for reset)
    initial_capital: f32,
    /// Average bid-ask spread (estimated from historical data)
    avg_spread: f32,
}

impl PortfolioTracker {
    pub fn new(initial_capital: f32, avg_spread: f32) -> Self {
        Self {
            cash: initial_capital,
            position_size: 0.0,
            position_entry_price: 0.0,
            initial_capital,
            avg_spread,
        }
    }

    /// Get portfolio features for TradingState
    pub fn get_portfolio_features(&self, current_price: f32) -> [f32; 3] {
        let portfolio_value = self.get_portfolio_value(current_price);
        [
            portfolio_value,    // [0] Portfolio value
            self.position_size, // [1] Position size (signed)
            self.avg_spread,    // [2] Spread
        ]
    }

    /// Execute trading action and update portfolio state
    pub fn execute_action(&mut self, action: TradingAction, price: f32, position_units: f32) {
        match action {
            TradingAction::Buy => {
                if self.position_size == 0.0 {
                    // Open long position
                    self.position_size = position_units;
                    self.position_entry_price = price;
                    self.cash -= position_units * price;
                } else if self.position_size < 0.0 {
                    // Close short position
                    let pnl = self.position_size * (self.position_entry_price - price);
                    self.cash += pnl;
                    self.position_size = 0.0;
                }
            }
            TradingAction::Sell => {
                if self.position_size == 0.0 {
                    // Open short position
                    self.position_size = -position_units;
                    self.position_entry_price = price;
                    self.cash += position_units * price;
                } else if self.position_size > 0.0 {
                    // Close long position
                    let pnl = self.position_size * (price - self.position_entry_price);
                    self.cash += pnl;
                    self.position_size = 0.0;
                }
            }
            TradingAction::Hold => {
                // No action - portfolio state unchanged
            }
        }
    }

    /// Calculate current portfolio value (cash + unrealized P&L)
    fn get_portfolio_value(&self, current_price: f32) -> f32 {
        if self.position_size == 0.0 {
            self.cash
        } else {
            let unrealized_pnl = if self.position_size > 0.0 {
                // Long position
                self.position_size * (current_price - self.position_entry_price)
            } else {
                // Short position
                self.position_size * (self.position_entry_price - current_price)
            };
            self.cash + unrealized_pnl
        }
    }

    /// Reset portfolio to initial state (for new episode)
    pub fn reset(&mut self) {
        self.cash = self.initial_capital;
        self.position_size = 0.0;
        self.position_entry_price = 0.0;
    }
}
```

### 2. DQNTrainer Modifications

```rust
// MODIFY: ml/src/trainers/dqn.rs

pub struct DQNTrainer {
    agent: Arc<RwLock<WorkingDQN>>,
    hyperparams: DQNHyperparameters,
    device: Device,
    metrics: Arc<RwLock<TrainingMetrics>>,
    // ... existing fields ...

    // NEW: Portfolio tracking
    portfolio_tracker: Arc<RwLock<PortfolioTracker>>,
}

impl DQNTrainer {
    pub fn new(hyperparams: DQNHyperparameters) -> Result<Self> {
        // ... existing initialization ...

        // Initialize portfolio tracker
        let portfolio_tracker = Arc::new(RwLock::new(PortfolioTracker::new(
            10_000.0, // Initial capital
            0.0001,   // 1 basis point spread
        )));

        Ok(Self {
            // ... existing fields ...
            portfolio_tracker,
        })
    }

    /// Convert 225-dim feature vector to TradingState (with portfolio features)
    async fn feature_vector_to_state(&self, feature_vec: &FeatureVector225, current_price: f32) -> Result<TradingState> {
        let price_features: Vec<f32> = vec![
            feature_vec[0] as f32, // open log return
            feature_vec[1] as f32, // high log return
            feature_vec[2] as f32, // low log return
            feature_vec[3] as f32, // close log return
        ];

        let technical_indicators: Vec<f32> = feature_vec[4..]
            .iter()
            .map(|&v| v as f32)
            .collect();

        let market_features = vec![];

        // NEW: Fetch portfolio features from tracker
        let portfolio_tracker = self.portfolio_tracker.read().await;
        let portfolio_features = portfolio_tracker.get_portfolio_features(current_price).to_vec();

        Ok(TradingState::from_normalized(
            price_features,
            technical_indicators,
            market_features,
            portfolio_features,
        ))
    }

    /// Process a single training sample (MODIFIED)
    async fn process_training_sample(
        &mut self,
        i: usize,
        feature_vec: &FeatureVector225,
        _target: &[f64],
        training_data: &[(FeatureVector225, Vec<f64>)],
    ) -> Result<Option<(f64, f64, f64)>> {
        // Extract current price (close price = feature_vec[3])
        let current_price = feature_vec[3] as f32;

        // Convert feature vector to state (includes portfolio)
        let state = self.feature_vector_to_state(feature_vec, current_price).await?;

        // Select action
        let action = self.select_action(&state).await?;

        // NEW: Execute action in portfolio tracker
        let mut portfolio_tracker = self.portfolio_tracker.write().await;
        portfolio_tracker.execute_action(action, current_price, 1.0); // 1 unit per trade
        drop(portfolio_tracker);

        // Get next state (with UPDATED portfolio)
        let next_price = if i + 1 < training_data.len() {
            training_data[i + 1].0[3] as f32
        } else {
            current_price
        };
        let next_state = self.feature_vector_to_state(&training_data[i + 1].0, next_price).await?;

        // Calculate reward (now uses populated portfolio_features)
        let reward = self.calculate_reward(action, &state, &next_state).await?;

        // ... rest of method unchanged ...
    }

    /// Train with data (MODIFIED - add reset at epoch boundaries)
    async fn train_with_data_full_loop<F>(
        &mut self,
        training_data: Vec<(FeatureVector225, Vec<f64>)>,
        mut checkpoint_callback: F,
    ) -> Result<TrainingMetrics> {
        // ... existing setup ...

        for epoch in 0..self.hyperparams.epochs {
            // NEW: Reset portfolio at start of each epoch
            let mut portfolio_tracker = self.portfolio_tracker.write().await;
            portfolio_tracker.reset();
            drop(portfolio_tracker);

            // ... rest of training loop unchanged ...
        }
    }
}
```

### 3. Integration Points

| Method | Change Required | Complexity |
|--------|----------------|------------|
| `feature_vector_to_state()` | Add `current_price` parameter, fetch portfolio features | LOW |
| `process_training_sample()` | Extract price, call `execute_action()` | LOW |
| `process_training_batch()` | Batch portfolio updates | MEDIUM |
| `train_with_data_full_loop()` | Reset portfolio at epoch boundaries | LOW |

---

## Potential Breaking Changes

### Approach A (RECOMMENDED):
✅ **NONE** - All changes are internal to `DQNTrainer`

### Approach B:
❌ **HIGH IMPACT**:
- Feature vector dimension changes: 225 → 228
- All neural networks must be retrained
- All checkpoints incompatible
- All feature extraction code must be updated

### Approach C:
⚠️ **MEDIUM IMPACT**:
- BacktestingEngine API changes
- Existing evaluation code may need refactoring

---

## Estimated Implementation Time

| Approach | Implementation | Testing | Total | Risk Level |
|----------|---------------|---------|-------|------------|
| **A** | 2 hours | 1 hour | **3 hours** | ✅ LOW |
| B | 8 hours | 4 hours | 12 hours | ❌ HIGH |
| C | 6 hours | 2 hours | 8 hours | ⚠️ MEDIUM |

---

## Test Plan (Approach A)

### Unit Tests

```rust
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_portfolio_tracker_initial_state() {
        let tracker = PortfolioTracker::new(10_000.0, 0.0001);
        let features = tracker.get_portfolio_features(100.0);

        assert_eq!(features[0], 10_000.0); // Portfolio value = cash
        assert_eq!(features[1], 0.0);      // No position
        assert_eq!(features[2], 0.0001);   // Spread
    }

    #[test]
    fn test_portfolio_tracker_buy_action() {
        let mut tracker = PortfolioTracker::new(10_000.0, 0.0001);
        tracker.execute_action(TradingAction::Buy, 100.0, 10.0);

        assert_eq!(tracker.position_size, 10.0);
        assert_eq!(tracker.position_entry_price, 100.0);
        assert_eq!(tracker.cash, 9_000.0); // 10_000 - (10 * 100)
    }

    #[test]
    fn test_portfolio_tracker_pnl_calculation() {
        let mut tracker = PortfolioTracker::new(10_000.0, 0.0001);
        tracker.execute_action(TradingAction::Buy, 100.0, 10.0);

        // Price rises to 110
        let features = tracker.get_portfolio_features(110.0);
        let expected_value = 9_000.0 + (10.0 * (110.0 - 100.0)); // 9000 + 100 = 9100
        assert_eq!(features[0], expected_value);
    }

    #[test]
    fn test_portfolio_tracker_reset() {
        let mut tracker = PortfolioTracker::new(10_000.0, 0.0001);
        tracker.execute_action(TradingAction::Buy, 100.0, 10.0);
        tracker.reset();

        assert_eq!(tracker.cash, 10_000.0);
        assert_eq!(tracker.position_size, 0.0);
    }
}
```

### Integration Tests

```rust
#[tokio::test]
async fn test_dqn_trainer_with_portfolio_features() {
    let hyperparams = DQNHyperparameters::default();
    let mut trainer = DQNTrainer::new(hyperparams).unwrap();

    // Create dummy feature vector
    let feature_vec = [0.0; 225];
    let state = trainer.feature_vector_to_state(&feature_vec, 100.0).await.unwrap();

    // Portfolio features should be populated
    assert!(!state.portfolio_features.is_empty());
    assert_eq!(state.portfolio_features.len(), 3);
    assert_eq!(state.portfolio_features[0], 10_000.0); // Initial capital
}

#[tokio::test]
async fn test_dqn_reward_calculation_with_portfolio() {
    let hyperparams = DQNHyperparameters::default();
    let mut trainer = DQNTrainer::new(hyperparams).unwrap();

    // Simulate two states with portfolio change
    let feature_vec1 = [0.0; 225];
    let feature_vec2 = [0.0; 225];

    let state1 = trainer.feature_vector_to_state(&feature_vec1, 100.0).await.unwrap();

    // Execute buy action
    trainer.portfolio_tracker.write().await.execute_action(
        TradingAction::Buy, 100.0, 10.0
    );

    let state2 = trainer.feature_vector_to_state(&feature_vec2, 110.0).await.unwrap();

    // Reward should be non-zero (profitable long position)
    let reward = trainer.calculate_reward(
        TradingAction::Buy, &state1, &state2
    ).await.unwrap();

    assert!(reward != 0.0); // Previously would be 0.0 due to bug
    assert!(reward > 0.0);  // Profitable trade should have positive reward
}
```

---

## Conclusion

**Recommendation**: Implement **Approach A** (Track Portfolio State in DQNTrainer)

**Justification**:
1. ✅ **Minimal risk**: No breaking changes
2. ✅ **Fast delivery**: 3 hours total implementation + testing
3. ✅ **Clean design**: Single-purpose `PortfolioTracker` struct
4. ✅ **Testable**: Independent unit tests for portfolio logic
5. ✅ **Performant**: O(1) portfolio state access

**Next Steps**:
1. Implement `PortfolioTracker` struct (1h)
2. Modify `DQNTrainer.feature_vector_to_state()` (30m)
3. Add `execute_action()` calls in training loop (45m)
4. Add reset logic at epoch boundaries (15m)
5. Write unit tests (30m)
6. Integration testing (30m)

**Total Estimated Time**: **3 hours**

---

**Files to Create**:
- `ml/src/dqn/portfolio_tracker.rs` (NEW)

**Files to Modify**:
- `ml/src/trainers/dqn.rs` (4 methods)
- `ml/src/dqn/mod.rs` (add `pub mod portfolio_tracker;`)

**Breaking Changes**: NONE ✅