foxhunt/DQN_PORTFOLIO_FEATURES_DESIGN.md

DQN Portfolio Features Bug - Design Solution

Agent: Wave 1 - Agent 4 Date: 2025-11-04 Bug Location: ml/src/trainers/dqn.rs:1571-1580 Status: Design Complete

---

Executive Summary

The feature_vector_to_state() method in DQNTrainer creates TradingState objects with empty portfolio_features, causing P&L reward calculations to always return 0.0. This breaks the reward signal, making the DQN agent unable to learn profitable trading strategies.

Impact: CRITICAL - DQN cannot learn without P&L rewards

---

Required Portfolio Features

Analysis of ml/src/dqn/reward.rs:145-200

The reward calculation requires the following portfolio state fields:

Index	Field	Type	Usage	Required By
[0]	Portfolio Value	f32	P&L calculation	calculate_pnl_reward() (lines 152-156)
[1]	Position Size	f32	Risk penalty & transaction costs	calculate_risk_penalty() (line 172), calculate_cost_penalty() (line 193)
[2]	Spread	f32	Transaction cost estimation	calculate_cost_penalty() (line 200)

Minimal Requirements:

• portfolio_features[0]: Portfolio value (cash + positions)
• portfolio_features[1]: Current position size (signed: +Long, -Short, 0 for flat)
• portfolio_features[2]: Bid-ask spread (for transaction costs)

Default values if missing:

• Portfolio value: 10,000.0 (default initial capital)
• Position size: 0.0 (flat position)
• Spread: 0.0001 (1 basis point)

---

Current Code Flow

// ml/src/trainers/dqn.rs:1554-1582
fn feature_vector_to_state(&self, feature_vec: &FeatureVector225) -> Result<TradingState> {
    let price_features: Vec<f32> = vec![
        feature_vec[0] as f32, // open log return
        feature_vec[1] as f32, // high log return
        feature_vec[2] as f32, // low log return
        feature_vec[3] as f32, // close log return
    ];

    let technical_indicators: Vec<f32> = feature_vec[4..].iter().map(|&v| v as f32).collect();

    let market_features = vec![];       // ❌ Empty!
    let portfolio_features = vec![];    // ❌ Empty! (BUG)

    Ok(TradingState::from_normalized(
        price_features,
        technical_indicators,
        market_features,
        portfolio_features,
    ))
}

Problem: No portfolio state tracking between time steps.

---

Design Approaches

Approach A: Track Portfolio State in DQNTrainer ✅ RECOMMENDED

Architecture:

┌──────────────────────────────────────────────────────────────┐
│                        DQNTrainer                             │
├──────────────────────────────────────────────────────────────┤
│  Fields (NEW):                                                │
│  - portfolio_tracker: PortfolioTracker                        │
│                                                                │
│  Methods (MODIFIED):                                          │
│  - feature_vector_to_state() → includes portfolio state       │
│  - process_training_sample() → updates portfolio after action│
│  - process_training_batch() → batched portfolio updates       │
└──────────────────────────────────────────────────────────────┘
                           │
                           │ uses
                           ▼
┌──────────────────────────────────────────────────────────────┐
│                    PortfolioTracker (NEW)                     │
├──────────────────────────────────────────────────────────────┤
│  Fields:                                                      │
│  - cash: f32                                                  │
│  - position_size: f32 (signed)                                │
│  - position_entry_price: f32                                  │
│  - initial_capital: f32                                       │
│  - avg_spread: f32                                            │
│                                                                │
│  Methods:                                                     │
│  - get_portfolio_features() → [value, size, spread]           │
│  - execute_action(action, price) → updates state              │
│  - get_portfolio_value(current_price) → cash + unrealized P&L │
│  - reset() → reinitialize for new episode                     │
└──────────────────────────────────────────────────────────────┘

Data Flow:

Training Loop (Epoch N, Sample i)
  │
  ├─> feature_vector_to_state(feature_vec)
  │     ├─> Extract price_features (OHLC)
  │     ├─> Extract technical_indicators (221 features)
  │     └─> portfolio_tracker.get_portfolio_features() ← NEW
  │           └─> Returns [value, position, spread]
  │
  ├─> select_action(state) → action
  │
  ├─> portfolio_tracker.execute_action(action, current_price) ← NEW
  │     ├─> BUY:  Open long, deduct cash
  │     ├─> SELL: Open short, add cash
  │     └─> HOLD: No change
  │
  ├─> next_state = feature_vector_to_state(next_feature_vec)
  │     └─> Uses UPDATED portfolio state
  │
  ├─> calculate_reward(action, state, next_state)
  │     ├─> calculate_pnl_reward() ← Uses portfolio_features[0]
  │     ├─> calculate_risk_penalty() ← Uses portfolio_features[1]
  │     └─> calculate_cost_penalty() ← Uses portfolio_features[1,2]
  │
  └─> store_experience(state, action, reward, next_state)

Pros:

• ✅ Clean separation: Portfolio logic in dedicated struct
• ✅ Minimal changes: Only modify DQNTrainer, no external APIs
• ✅ Backwards compatible: Existing tests don't break
• ✅ Stateful tracking: Portfolio persists across training steps
• ✅ Easy testing: PortfolioTracker can be unit tested independently

Cons:

• ⚠️ Requires synchronization between trainer and portfolio tracker
• ⚠️ Portfolio resets at episode boundaries (need explicit reset logic)

Implementation Effort: 2-3 hours

• 30 min: Implement PortfolioTracker struct (100 lines)
• 45 min: Modify feature_vector_to_state() to fetch portfolio state
• 45 min: Add execute_action() calls in process_training_sample() and process_training_batch()
• 30 min: Add reset logic at epoch boundaries
• 30 min: Unit tests for PortfolioTracker

---

Approach B: Extend FeatureVector225

Architecture:

Current: FeatureVector225 = [f64; 225]
         ├─ [0..3]: OHLC price features
         └─ [4..224]: Technical indicators (221)

Proposed: FeatureVector228 = [f64; 228]  ← BREAKING CHANGE
          ├─ [0..3]: OHLC price features
          ├─ [4..224]: Technical indicators (221)
          └─ [225..227]: Portfolio features (3) ← NEW
               ├─ [225]: Portfolio value
               ├─ [226]: Position size
               └─ [227]: Spread

Pros:

• ✅ Self-contained state representation
• ✅ No external tracking needed
• ✅ Feature vector includes all information

Cons:

• ❌ BREAKING CHANGE: All feature extraction code must be updated
• ❌ Circular dependency: Features need portfolio state, but portfolio state depends on past features
• ❌ Re-extraction overhead: Must recompute features after each action
• ❌ Backtesting integration: DBN data loader must inject portfolio state
• ❌ Neural network: State dimension changes from 225 → 228 (retrain all models)

Implementation Effort: 8-12 hours (HIGH RISK)

• 2h: Update FeatureVector225 → FeatureVector228 across codebase
• 2h: Modify feature extraction pipeline to include portfolio state
• 2h: Update neural network configs (state_dim: 225 → 228)
• 2h: Update all DBN data loaders
• 2h: Retrain all DQN models (checkpoints incompatible)
• 2h: Update all tests

Backwards Compatibility: ❌ NONE - Breaks all existing checkpoints

---

Approach C: Separate Portfolio State Tracker (Backtesting Integration)

Architecture:

┌──────────────────────────────────────────────────────────────┐
│                   BacktestingEngine (EXISTING)                │
│                  (ml/src/evaluation/engine.rs)                │
├──────────────────────────────────────────────────────────────┤
│  Fields:                                                      │
│  - current_position: Option<Position>                         │
│  - trades: Vec<Trade>                                         │
│  - initial_capital: f32                                       │
│  - action_counts: [usize; 3]                                  │
│                                                                │
│  Methods:                                                     │
│  - process_bar(bar, action) → updates position                │
│  - close_position() → calculates P&L                          │
│  - get_portfolio_features() → [value, size, spread] ← NEW     │
└──────────────────────────────────────────────────────────────┘
                           │
                           │ used by
                           ▼
┌──────────────────────────────────────────────────────────────┐
│                        DQNTrainer                             │
├──────────────────────────────────────────────────────────────┤
│  Fields (NEW):                                                │
│  - backtesting_engine: Option<BacktestingEngine>              │
│                                                                │
│  Methods (MODIFIED):                                          │
│  - feature_vector_to_state() → queries backtesting engine     │
│  - process_training_sample() → calls backtesting engine       │
└──────────────────────────────────────────────────────────────┘

Pros:

• ✅ Reuses existing BacktestingEngine from evaluation
• ✅ Unified portfolio tracking for training and evaluation
• ✅ Already has position management logic

Cons:

• ❌ Tight coupling: DQNTrainer depends on backtesting module
• ❌ Circular dependency: Backtesting designed for EVALUATION, not TRAINING
• ❌ API mismatch: BacktestingEngine expects OHLCVBar, trainer has FeatureVector225
• ❌ Performance overhead: BacktestingEngine tracks trade history (not needed during training)
• ❌ Complexity: Mixing training and evaluation concerns

Implementation Effort: 6-8 hours

• 3h: Refactor BacktestingEngine to support training mode
• 2h: Adapt API to work with FeatureVector225 instead of OHLCVBar
• 2h: Integrate into DQNTrainer
• 1h: Unit tests

Backwards Compatibility: ⚠️ Requires refactoring existing evaluation code

---

Recommended Approach: A (Track Portfolio State in DQNTrainer)

Why Approach A?

Criterion	Score	Justification
Simplicity	⭐⭐⭐⭐⭐	Clean separation, minimal changes
Backwards Compatibility	⭐⭐⭐⭐⭐	No breaking changes
Implementation Time	⭐⭐⭐⭐⭐	2-3 hours vs. 8-12h (B) or 6-8h (C)
Performance	⭐⭐⭐⭐⭐	No overhead, O(1) portfolio state access
Testability	⭐⭐⭐⭐⭐	PortfolioTracker is unit-testable
Maintainability	⭐⭐⭐⭐	Single-purpose struct

Decision: Approach A provides maximum value with minimum risk and fastest delivery.

---

Implementation Pseudo-Code (Approach A)

1. PortfolioTracker Struct

// NEW FILE: ml/src/dqn/portfolio_tracker.rs

#[derive(Debug, Clone)]
pub struct PortfolioTracker {
    /// Current cash balance
    cash: f32,
    /// Current position size (positive = long, negative = short, 0 = flat)
    position_size: f32,
    /// Entry price for current position
    position_entry_price: f32,
    /// Initial capital (for reset)
    initial_capital: f32,
    /// Average bid-ask spread (estimated from historical data)
    avg_spread: f32,
}

impl PortfolioTracker {
    pub fn new(initial_capital: f32, avg_spread: f32) -> Self {
        Self {
            cash: initial_capital,
            position_size: 0.0,
            position_entry_price: 0.0,
            initial_capital,
            avg_spread,
        }
    }

    /// Get portfolio features for TradingState
    pub fn get_portfolio_features(&self, current_price: f32) -> [f32; 3] {
        let portfolio_value = self.get_portfolio_value(current_price);
        [
            portfolio_value,    // [0] Portfolio value
            self.position_size, // [1] Position size (signed)
            self.avg_spread,    // [2] Spread
        ]
    }

    /// Execute trading action and update portfolio state
    pub fn execute_action(&mut self, action: TradingAction, price: f32, position_units: f32) {
        match action {
            TradingAction::Buy => {
                if self.position_size == 0.0 {
                    // Open long position
                    self.position_size = position_units;
                    self.position_entry_price = price;
                    self.cash -= position_units * price;
                } else if self.position_size < 0.0 {
                    // Close short position
                    let pnl = self.position_size * (self.position_entry_price - price);
                    self.cash += pnl;
                    self.position_size = 0.0;
                }
            }
            TradingAction::Sell => {
                if self.position_size == 0.0 {
                    // Open short position
                    self.position_size = -position_units;
                    self.position_entry_price = price;
                    self.cash += position_units * price;
                } else if self.position_size > 0.0 {
                    // Close long position
                    let pnl = self.position_size * (price - self.position_entry_price);
                    self.cash += pnl;
                    self.position_size = 0.0;
                }
            }
            TradingAction::Hold => {
                // No action - portfolio state unchanged
            }
        }
    }

    /// Calculate current portfolio value (cash + unrealized P&L)
    fn get_portfolio_value(&self, current_price: f32) -> f32 {
        if self.position_size == 0.0 {
            self.cash
        } else {
            let unrealized_pnl = if self.position_size > 0.0 {
                // Long position
                self.position_size * (current_price - self.position_entry_price)
            } else {
                // Short position
                self.position_size * (self.position_entry_price - current_price)
            };
            self.cash + unrealized_pnl
        }
    }

    /// Reset portfolio to initial state (for new episode)
    pub fn reset(&mut self) {
        self.cash = self.initial_capital;
        self.position_size = 0.0;
        self.position_entry_price = 0.0;
    }
}

2. DQNTrainer Modifications

// MODIFY: ml/src/trainers/dqn.rs

pub struct DQNTrainer {
    agent: Arc<RwLock<WorkingDQN>>,
    hyperparams: DQNHyperparameters,
    device: Device,
    metrics: Arc<RwLock<TrainingMetrics>>,
    // ... existing fields ...

    // NEW: Portfolio tracking
    portfolio_tracker: Arc<RwLock<PortfolioTracker>>,
}

impl DQNTrainer {
    pub fn new(hyperparams: DQNHyperparameters) -> Result<Self> {
        // ... existing initialization ...

        // Initialize portfolio tracker
        let portfolio_tracker = Arc::new(RwLock::new(PortfolioTracker::new(
            10_000.0, // Initial capital
            0.0001,   // 1 basis point spread
        )));

        Ok(Self {
            // ... existing fields ...
            portfolio_tracker,
        })
    }

    /// Convert 225-dim feature vector to TradingState (with portfolio features)
    async fn feature_vector_to_state(&self, feature_vec: &FeatureVector225, current_price: f32) -> Result<TradingState> {
        let price_features: Vec<f32> = vec![
            feature_vec[0] as f32, // open log return
            feature_vec[1] as f32, // high log return
            feature_vec[2] as f32, // low log return
            feature_vec[3] as f32, // close log return
        ];

        let technical_indicators: Vec<f32> = feature_vec[4..]
            .iter()
            .map(|&v| v as f32)
            .collect();

        let market_features = vec![];

        // NEW: Fetch portfolio features from tracker
        let portfolio_tracker = self.portfolio_tracker.read().await;
        let portfolio_features = portfolio_tracker.get_portfolio_features(current_price).to_vec();

        Ok(TradingState::from_normalized(
            price_features,
            technical_indicators,
            market_features,
            portfolio_features,
        ))
    }

    /// Process a single training sample (MODIFIED)
    async fn process_training_sample(
        &mut self,
        i: usize,
        feature_vec: &FeatureVector225,
        _target: &[f64],
        training_data: &[(FeatureVector225, Vec<f64>)],
    ) -> Result<Option<(f64, f64, f64)>> {
        // Extract current price (close price = feature_vec[3])
        let current_price = feature_vec[3] as f32;

        // Convert feature vector to state (includes portfolio)
        let state = self.feature_vector_to_state(feature_vec, current_price).await?;

        // Select action
        let action = self.select_action(&state).await?;

        // NEW: Execute action in portfolio tracker
        let mut portfolio_tracker = self.portfolio_tracker.write().await;
        portfolio_tracker.execute_action(action, current_price, 1.0); // 1 unit per trade
        drop(portfolio_tracker);

        // Get next state (with UPDATED portfolio)
        let next_price = if i + 1 < training_data.len() {
            training_data[i + 1].0[3] as f32
        } else {
            current_price
        };
        let next_state = self.feature_vector_to_state(&training_data[i + 1].0, next_price).await?;

        // Calculate reward (now uses populated portfolio_features)
        let reward = self.calculate_reward(action, &state, &next_state).await?;

        // ... rest of method unchanged ...
    }

    /// Train with data (MODIFIED - add reset at epoch boundaries)
    async fn train_with_data_full_loop<F>(
        &mut self,
        training_data: Vec<(FeatureVector225, Vec<f64>)>,
        mut checkpoint_callback: F,
    ) -> Result<TrainingMetrics> {
        // ... existing setup ...

        for epoch in 0..self.hyperparams.epochs {
            // NEW: Reset portfolio at start of each epoch
            let mut portfolio_tracker = self.portfolio_tracker.write().await;
            portfolio_tracker.reset();
            drop(portfolio_tracker);

            // ... rest of training loop unchanged ...
        }
    }
}

3. Integration Points

Method	Change Required	Complexity
feature_vector_to_state()	Add current_price parameter, fetch portfolio features	LOW
process_training_sample()	Extract price, call execute_action()	LOW
process_training_batch()	Batch portfolio updates	MEDIUM
train_with_data_full_loop()	Reset portfolio at epoch boundaries	LOW

---

Potential Breaking Changes

Approach A (RECOMMENDED):

✅ NONE - All changes are internal to DQNTrainer

Approach B:

❌ HIGH IMPACT:

• Feature vector dimension changes: 225 → 228
• All neural networks must be retrained
• All checkpoints incompatible
• All feature extraction code must be updated

Approach C:

⚠️ MEDIUM IMPACT:

• BacktestingEngine API changes
• Existing evaluation code may need refactoring

---

Estimated Implementation Time

Approach	Implementation	Testing	Total	Risk Level
A	2 hours	1 hour	3 hours	✅ LOW
B	8 hours	4 hours	12 hours	❌ HIGH
C	6 hours	2 hours	8 hours	⚠️ MEDIUM

---

Test Plan (Approach A)

Unit Tests

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_portfolio_tracker_initial_state() {
        let tracker = PortfolioTracker::new(10_000.0, 0.0001);
        let features = tracker.get_portfolio_features(100.0);

        assert_eq!(features[0], 10_000.0); // Portfolio value = cash
        assert_eq!(features[1], 0.0);      // No position
        assert_eq!(features[2], 0.0001);   // Spread
    }

    #[test]
    fn test_portfolio_tracker_buy_action() {
        let mut tracker = PortfolioTracker::new(10_000.0, 0.0001);
        tracker.execute_action(TradingAction::Buy, 100.0, 10.0);

        assert_eq!(tracker.position_size, 10.0);
        assert_eq!(tracker.position_entry_price, 100.0);
        assert_eq!(tracker.cash, 9_000.0); // 10_000 - (10 * 100)
    }

    #[test]
    fn test_portfolio_tracker_pnl_calculation() {
        let mut tracker = PortfolioTracker::new(10_000.0, 0.0001);
        tracker.execute_action(TradingAction::Buy, 100.0, 10.0);

        // Price rises to 110
        let features = tracker.get_portfolio_features(110.0);
        let expected_value = 9_000.0 + (10.0 * (110.0 - 100.0)); // 9000 + 100 = 9100
        assert_eq!(features[0], expected_value);
    }

    #[test]
    fn test_portfolio_tracker_reset() {
        let mut tracker = PortfolioTracker::new(10_000.0, 0.0001);
        tracker.execute_action(TradingAction::Buy, 100.0, 10.0);
        tracker.reset();

        assert_eq!(tracker.cash, 10_000.0);
        assert_eq!(tracker.position_size, 0.0);
    }
}

Integration Tests

#[tokio::test]
async fn test_dqn_trainer_with_portfolio_features() {
    let hyperparams = DQNHyperparameters::default();
    let mut trainer = DQNTrainer::new(hyperparams).unwrap();

    // Create dummy feature vector
    let feature_vec = [0.0; 225];
    let state = trainer.feature_vector_to_state(&feature_vec, 100.0).await.unwrap();

    // Portfolio features should be populated
    assert!(!state.portfolio_features.is_empty());
    assert_eq!(state.portfolio_features.len(), 3);
    assert_eq!(state.portfolio_features[0], 10_000.0); // Initial capital
}

#[tokio::test]
async fn test_dqn_reward_calculation_with_portfolio() {
    let hyperparams = DQNHyperparameters::default();
    let mut trainer = DQNTrainer::new(hyperparams).unwrap();

    // Simulate two states with portfolio change
    let feature_vec1 = [0.0; 225];
    let feature_vec2 = [0.0; 225];

    let state1 = trainer.feature_vector_to_state(&feature_vec1, 100.0).await.unwrap();

    // Execute buy action
    trainer.portfolio_tracker.write().await.execute_action(
        TradingAction::Buy, 100.0, 10.0
    );

    let state2 = trainer.feature_vector_to_state(&feature_vec2, 110.0).await.unwrap();

    // Reward should be non-zero (profitable long position)
    let reward = trainer.calculate_reward(
        TradingAction::Buy, &state1, &state2
    ).await.unwrap();

    assert!(reward != 0.0); // Previously would be 0.0 due to bug
    assert!(reward > 0.0);  // Profitable trade should have positive reward
}

---

Conclusion

Recommendation: Implement Approach A (Track Portfolio State in DQNTrainer)

Justification:

1. ✅ Minimal risk: No breaking changes
2. ✅ Fast delivery: 3 hours total implementation + testing
3. ✅ Clean design: Single-purpose PortfolioTracker struct
4. ✅ Testable: Independent unit tests for portfolio logic
5. ✅ Performant: O(1) portfolio state access

Next Steps:

1. Implement PortfolioTracker struct (1h)
2. Modify DQNTrainer.feature_vector_to_state() (30m)
3. Add execute_action() calls in training loop (45m)
4. Add reset logic at epoch boundaries (15m)
5. Write unit tests (30m)
6. Integration testing (30m)

Total Estimated Time: 3 hours

---

Files to Create:

• ml/src/dqn/portfolio_tracker.rs (NEW)

Files to Modify:

• ml/src/trainers/dqn.rs (4 methods)
• ml/src/dqn/mod.rs (add pub mod portfolio_tracker;)

Breaking Changes: NONE ✅