gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Wave 11: Rainbow DQN integration + 23/23 tests passing

Browse Source 
CRITICAL FINDINGS from 3-trial validation:
- 85,120 gradient clipping warnings (81.6% of logs) - REGRESSION
- Rainbow features DISABLED: use_dueling=false, use_distributional=false, use_noisy_nets=false
- Negative Q-values confirmed: HOLD -1000 to -3250
- Performance: Sharpe 0.29 (target 0.77)

Changes:
- Fixed N-Step compilation (7/7 tests passing)
- Fixed Distributional compilation (6/6 tests passing)
- Fixed Dueling CUDA errors (10/10 tests passing)
- Added TDD validation for state_dim=225
- Total: 23/23 Wave 11 tests passing (100%)

Issues requiring investigation:
1. Why are Dueling/Distributional/Noisy disabled in hyperopt?
2. Why gradient explosion despite previous fixes?
3. Test coverage gaps - unit tests pass but integration fails

🤖 Generated with Claude Code
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2025-11-18 13:53:59 +01:00
parent 3d3b5d32fe
commit c645e6222d
240 changed files with 69556 additions and 3838 deletions
Download Patch File Download Diff File Expand all files Collapse all files

550
ml/zen_generated.code
View File

@@ -1,402 +1,236 @@
<GENERATED-CODE>
Here are the step-by-step instructions to fix the training instability and improve reward scaling.
I will create the core flow matching loss function in a new, self-contained module.
1. **Update `ml/src/trainers/dqn.rs`**: Modify the training loop to break the positive feedback loop in the risk-adjusted reward calculation. We will update the PnL history tracker with the raw reward *before* applying the Sharpe ratio adjustment.
1. Create a new directory `ml/src/ppo/flow_policy/` to house components related to the flow-based policy.
2. Create a new module file `ml/src/ppo/flow_policy/mod.rs` to declare the `flow_matching` submodule.
3. Create the implementation file `ml/src/ppo/flow_policy/flow_matching.rs`, which contains the PPO clipped surrogate objective adapted for flow-based policies.
2. **Update `ml/src/dqn/reward.rs`**: Overhaul the reward logic to be more statistically robust. This involves three key changes:
* Refactor the P&L reward to be based on percentage portfolio change, which is a more stationary signal.
* Introduce a `RewardNormalizer` struct that uses Welford's algorithm to compute running mean and standard deviation.
* Integrate the normalizer into the `RewardFunction` to automatically scale all rewards before they are used for training.
<UPDATED_EXISTING_FILE: ml/src/trainers/dqn.rs>
<NEWFILE: ml/src/ppo/flow_policy/mod.rs>
```rust
// ... (code before line 1169)
let raw_reward = reward_decimal.to_string().parse::<f32>().unwrap_or(0.0) as f64;
//! # Flow-Based Policy Module
//!
//! This module contains components for implementing a flow-based policy within the PPO framework,
//! including the flow matching objective function.
// Calculate price return for volatility tracking
// Add safety check for division by zero
let price_return = if current_close.abs() > 1e-9 {
(next_close - current_close) / current_close
} else {
0.0
};
// CRITICAL FIX: Update adaptive risk trackers with the raw, unadjusted reward.
// This breaks the positive feedback loop where amplified rewards were used
// to calculate the next amplification factor (Sharpe ratio).
self.update_risk_trackers(raw_reward, price_return);
// WAVE 16S: Apply risk-adjusted rewards (Sharpe ratio) using history of raw rewards
let risk_adjusted_reward = self.calculate_risk_adjusted_reward(raw_reward);
// Convert back to f32 for experience storage
let reward = risk_adjusted_reward as f32;
// WAVE 16S: Log adaptive features periodically
if i % 100 == 0 && i > 0 {
// ... (code from line 1185 to 2677)
/// Update adaptive risk trackers with new market data
fn update_risk_trackers(&mut self, reward: f64, price_return: f64) {
// Update PnL history
self.pnl_history.push_back(reward);
if self.pnl_history.len() > 1000 {
// ... (rest of the file is unchanged)
pub mod flow_matching;
```
</UPDATED_EXISTING_FILE>
</NEWFILE>
<UPDATED_EXISTING_FILE: ml/src/dqn/reward.rs>
Next, here is the core implementation of the flow matching objective.
<NEWFILE: ml/src/ppo/flow_policy/flow_matching.rs>
```rust
//! Trading-specific reward functions for DQN
//! # Flow Matching Objective for Flow-Based Policy Optimization (FPO)
//!
//! This module implements the clipped surrogate objective for PPO using a flow-based policy.
//! Instead of using the ratio of likelihoods (log probabilities), it uses the ratio of
//! the determinants of the Jacobian of the flow transformation. This approach, detailed in
//! "Flow Matching Policy Optimization," helps mitigate gradient explosion issues common
//! with traditional policy gradient methods in continuous action spaces.
//!
//! The core idea is to replace the likelihood ratio `r(θ) = π_θ(a|s) / π_θ_old(a|s)` with
//! a flow ratio based on log-determinants: `r_flow = exp(log_det_new - log_det_old)`.
//! This ratio is then used in the standard PPO clipped surrogate objective.
// CANONICAL TYPE IMPORTS - Use common::Decimal
use serde::{Deserialize, Serialize};
// For Decimal::from_f64
use common::types::Price;
use rust_decimal::Decimal;
use candle_core::{Device, Tensor};
use super::action_space::FactoredAction;
use super::agent::{TradingAction, TradingState};
use crate::MLError;
use crate::{tensor_ops::TensorOps, MLError};
/// Configuration for reward function
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RewardConfig {
/// Weight for P&L component
pub pnl_weight: Decimal,
/// Weight for risk penalty
pub risk_weight: Decimal,
/// Weight for transaction cost penalty
pub cost_weight: Decimal,
/// Weight for hold reward (to reduce over-trading)
pub hold_reward: Decimal,
/// Price movement threshold for dynamic HOLD reward (as fraction, e.g., 0.02 = 2%)
pub movement_threshold: Decimal,
/// Weight for HOLD action penalty during high volatility (negative value applied)
pub hold_penalty_weight: Decimal,
/// Weight for diversity penalty (negative value to penalize low entropy, -0.1 default)
pub diversity_weight: Decimal,
/// Enable running normalization of rewards
pub enable_reward_normalization: bool,
/// Configuration for the Flow Matching loss function.
#[derive(Debug, Clone, Copy)]
pub struct FlowMatchingConfig {
/// The clipping parameter (epsilon) for the PPO surrogate objective.
/// Typically set to 0.2.
pub clip_epsilon: f32,
/// The minimum value for clipping the log flow ratio to prevent underflow in `exp()`.
pub log_ratio_clip_min: f32,
/// The maximum value for clipping the log flow ratio to prevent overflow in `exp()`.
pub log_ratio_clip_max: f32,
}
impl Default for RewardConfig {
impl Default for FlowMatchingConfig {
fn default() -> Self {
Self {
pnl_weight: Decimal::ONE,
risk_weight: Decimal::try_from(0.1).unwrap_or(Decimal::ZERO),
cost_weight: Decimal::try_from(0.05).unwrap_or(Decimal::ZERO),
hold_reward: Decimal::try_from(0.001).unwrap_or(Decimal::ZERO),
movement_threshold: Decimal::try_from(0.01).unwrap_or(Decimal::ZERO), // 1% matches data distribution
hold_penalty_weight: Decimal::try_from(0.01).unwrap_or(Decimal::ZERO), // 1% default penalty
diversity_weight: Decimal::try_from(-0.1).unwrap_or(Decimal::ZERO), // -0.1 default (100x stronger than hold_reward)
enable_reward_normalization: true, // Default to enabled for robustness
clip_epsilon: 0.2,
log_ratio_clip_min: -20.0,
log_ratio_clip_max: 20.0,
}
}
}
/// A running normalizer for rewards using Welford's online algorithm.
/// This adaptively scales rewards to have a mean of ~0 and stddev of ~1,
/// which stabilizes DQN training with non-stationary reward distributions.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RewardNormalizer {
count: u64,
mean: f64,
m2: f64, // Sum of squares of differences from the current mean
enabled: bool,
}
impl RewardNormalizer {
/// Creates a new `RewardNormalizer`.
pub fn new(enabled: bool) -> Self {
Self {
count: 0,
mean: 0.0,
m2: 0.0,
enabled,
}
}
/// Updates the running statistics with a new reward value using Welford's algorithm.
pub fn update(&mut self, reward: Decimal) {
if !self.enabled {
return;
}
if let Ok(reward_f64) = reward.try_into::<f64>() {
self.count += 1;
let delta = reward_f64 - self.mean;
self.mean += delta / self.count as f64;
let delta2 = reward_f64 - self.mean;
self.m2 += delta * delta2;
}
}
/// Normalizes a reward value based on the current running statistics.
pub fn normalize(&self, reward: Decimal) -> Decimal {
// Do not normalize until we have seen at least a few samples
if !self.enabled || self.count < 10 {
return reward;
}
let variance = self.m2 / self.count as f64;
let std_dev = variance.sqrt();
if let Ok(reward_f64) = reward.try_into::<f64>() {
// Clip std_dev to avoid division by zero or near-zero values, which can cause explosions.
let safe_std_dev = std_dev.max(1e-8);
let normalized_reward = (reward_f64 - self.mean) / safe_std_dev;
Decimal::try_from(normalized_reward).unwrap_or(reward)
} else {
reward
}
}
}
/// Risk metrics for trading state
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct RiskMetrics {
/// Value at Risk (95%)
pub var_95: Decimal,
/// Maximum drawdown
pub max_drawdown: Decimal,
/// Sharpe ratio
pub sharpe_ratio: Decimal,
/// Volatility
pub volatility: Decimal,
}
/// Market data snapshot
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct MarketData {
/// Current bid price
pub bid: Price,
/// Current ask price
pub ask: Price,
/// Bid-ask spread
pub spread: Price,
/// Volume
pub volume: Decimal,
}
/// Calculate Shannon entropy for action distribution
/// Computes the PPO clipped surrogate objective using the flow matching ratio.
///
/// This function is the core of the FPO update rule. It takes the log-determinants from the
/// current and old policies, computes the flow ratio, and then calculates the PPO loss.
///
/// # Arguments
/// * `recent_actions` - Sliding window of recent actions (last 100 actions)
/// * `new_log_dets` - A tensor of log-determinants from the current policy network for the actions in the batch. Shape: `[batch_size]`.
/// * `old_log_dets` - A tensor of log-determinants from the policy network used to collect the trajectory data. Shape: `[batch_size]`.
/// * `advantages` - A tensor of normalized advantages. Shape: `[batch_size]`.
/// * `config` - Configuration for the loss computation, including clipping parameters.
/// * `device` - The device on which to perform tensor operations.
///
/// # Returns
/// Shannon entropy H = -Σ(p_i * log2(p_i)) where p_i is frequency of action i
/// Range: [0.0, 1.585] (0 = all same action, 1.585 = perfectly balanced)
///
/// # Note
/// For FactoredAction, we convert to legacy TradingAction (Buy/Sell/Hold) for entropy calculation
/// to maintain backward compatibility with the original 3-action entropy range.
fn calculate_entropy(recent_actions: &[FactoredAction]) -> Decimal {
if recent_actions.is_empty() {
return Decimal::ONE; // Default to high entropy if no history
}
/// A scalar `Tensor` representing the final policy loss, ready for backpropagation.
/// The loss is negated because optimizers perform minimization, while PPO aims to maximize the objective.
pub fn compute_flow_matching_loss(
new_log_dets: &Tensor,
old_log_dets: &Tensor,
advantages: &Tensor,
config: &FlowMatchingConfig,
device: &Device,
) -> Result<Tensor, MLError> {
// 1. Compute the log of the flow ratio.
// log_flow_ratio = log(det_new / det_old) = log_det_new - log_det_old
let log_flow_ratio = (new_log_dets - old_log_dets)?;
// Convert to legacy actions and count
let mut counts = [0, 0, 0]; // BUY, SELL, HOLD
for action in recent_actions {
let legacy_action = action.to_legacy_action();
counts[legacy_action as usize] += 1;
}
// 2. Clip the log ratio to prevent numerical instability (overflow/underflow) when exponentiating.
// A range of [-20, 20] is a safe default, as exp(20) is large but manageable,
// while exp(>~80) can lead to f32 infinity. This is a critical step for stability.
let log_ratio_min = Tensor::full(config.log_ratio_clip_min, log_flow_ratio.dims(), device)
.map_err(|e| MLError::TrainingError(format!("Failed to create log_ratio min tensor: {}", e)))?;
let log_ratio_max = Tensor::full(config.log_ratio_clip_max, log_flow_ratio.dims(), device)
.map_err(|e| MLError::TrainingError(format!("Failed to create log_ratio max tensor: {}", e)))?;
let clipped_log_ratio = log_flow_ratio.clamp(&log_ratio_min, &log_ratio_max)?;
let total = recent_actions.len() as f64;
let mut entropy = Decimal::ZERO;
// 3. Compute the flow ratio.
let ratio = clipped_log_ratio.exp()?;
for &count in &counts {
if count > 0 {
let p = Decimal::try_from(count as f64 / total).unwrap_or(Decimal::ZERO);
// Shannon entropy: -p * log2(p)
// Use natural log and convert: log2(x) = ln(x) / ln(2)
if let Ok(p_f64) = TryInto::<f64>::try_into(p) {
if p_f64 > 0.0 {
let log2_p = p_f64.ln() / 2.0_f64.ln();
let term = p * Decimal::try_from(log2_p).unwrap_or(Decimal::ZERO);
entropy -= term;
}
}
}
}
// 4. Compute the clipped surrogate objective, as in standard PPO.
// surrogate1 = ratio * advantage
let surr1 = (&ratio * advantages)?;
entropy
// surrogate2 = clamp(ratio, 1 - ε, 1 + ε) * advantage
let clip_min = 1.0 - config.clip_epsilon;
let clip_max = 1.0 + config.clip_epsilon;
let clipped_ratio = ratio.clamp(clip_min, clip_max)?;
let surr2 = (&clipped_ratio * advantages)?;
// 5. The PPO objective is the minimum of the two surrogates.
let policy_loss_raw = TensorOps::elementwise_min(&surr1, &surr2)?;
// 6. The final loss is the negative mean of the objective function.
// We negate it because we want to maximize the objective via gradient ascent,
// which is equivalent to minimizing the negative objective.
let policy_loss_mean = policy_loss_raw.mean_all()?;
let final_loss = TensorOps::negate(&policy_loss_mean)?;
Ok(final_loss)
}
/// Reward function for `DQN` training
#[derive(Debug)]
pub struct RewardFunction {
/// Configuration
config: RewardConfig,
/// Previous rewards for tracking
reward_history: Vec<Decimal>,
/// Running normalizer for rewards
normalizer: RewardNormalizer,
}
#[cfg(test)]
mod tests {
use super::*;
use candle_core::{Device, Tensor};
impl RewardFunction {
/// Create a new reward function
pub fn new(config: RewardConfig) -> Self {
Self {
normalizer: RewardNormalizer::new(config.enable_reward_normalization),
config,
reward_history: Vec::new(),
}
}
#[test]
fn test_flow_matching_loss_computation() -> Result<(), MLError> {
let device = Device::Cpu;
let config = FlowMatchingConfig::default();
// Batch size of 4
let new_log_dets = Tensor::from_vec(vec![1.2, 0.8, -0.5, 2.0], 4, &device)?;
let old_log_dets = Tensor::from_vec(vec![1.0, 1.0, -0.4, 1.5], 4, &device)?;
let advantages = Tensor::from_vec(vec![1.5, -0.5, 2.0, 0.8], 4, &device)?;
// Expected log_flow_ratio = [0.2, -0.2, -0.1, 0.5]
// Expected ratio = [1.2214, 0.8187, 0.9048, 1.6487]
// Expected clipped_ratio (epsilon=0.2, range=[0.8, 1.2])
// clipped_ratio = [1.2, 0.8187, 0.9048, 1.2]
// surr1 = [1.8321, -0.4093, 1.8096, 1.3189]
// surr2 = [1.8, -0.4093, 1.8096, 0.96]
// min(surr1, surr2) = [1.8, -0.4093, 1.8096, 0.96]
// mean = (1.8 - 0.4093 + 1.8096 + 0.96) / 4 = 4.1603 / 4 = 1.040075
// final_loss = -1.040075
let loss =
compute_flow_matching_loss(&new_log_dets, &old_log_dets, &advantages, &config, &device)?;
let loss_val = loss.to_scalar::<f32>()?;
assert!((loss_val - (-1.040075)).abs() < 1e-4);
/// Validate that a TradingState has the required portfolio features
///
/// # Required Structure
/// Portfolio features must have length >= 3:
/// - [0]: Portfolio value (cash + unrealized P&L)
/// - [1]: Position size (signed: +Long, -Short, 0=flat)
/// - [2]: Bid-ask spread
///
/// # Returns
/// Ok(()) if valid, Err(MLError) with descriptive message if invalid
fn validate_portfolio_features(state: &TradingState) -> Result<(), MLError> {
if state.portfolio_features.len() < 3 {
return Err(MLError::InvalidInput(
format!(
"TradingState portfolio_features must have length >= 3 (got {}). Expected: [portfolio_value, position_size, spread]",
state.portfolio_features.len()
)
));
}
Ok(())
}
/// Calculate reward for a state transition
///
/// # Arguments
/// * `action` - Trading action taken (FactoredAction with exposure, order type, urgency)
/// * `current_state` - Current trading state (128-dim: 4 price + 121 technical + 3 portfolio)
/// * `next_state` - Next trading state after action (128-dim structure)
/// * `recent_actions` - Sliding window of last 100 actions for diversity penalty
///
/// # Validation
/// Validates that both states have proper portfolio_features (length >= 3).
/// Logs warnings if features are missing but continues with default values.
///
/// # Note
/// Converts FactoredAction to legacy TradingAction for reward calculation logic.
pub fn calculate_reward(
&mut self,
action: FactoredAction,
current_state: &TradingState,
next_state: &TradingState,
recent_actions: &[FactoredAction],
) -> Result<Decimal, MLError> {
// Validate portfolio features (non-fatal, logs warnings)
if let Err(e) = Self::validate_portfolio_features(current_state) {
tracing::warn!("Current state validation: {}", e);
}
if let Err(e) = Self::validate_portfolio_features(next_state) {
tracing::warn!("Next state validation: {}", e);
}
// Convert to legacy action for reward logic
let legacy_action = action.to_legacy_action();
let base_reward = match legacy_action {
TradingAction::Buy | TradingAction::Sell => {
// Calculate P&L-based reward
let pnl_reward = self.calculate_pnl_reward(current_state, next_state)?;
// Calculate risk penalty
let risk_penalty = self.calculate_risk_penalty(next_state);
// Calculate transaction cost penalty
let cost_penalty = self.calculate_cost_penalty(current_state, next_state);
self.config.pnl_weight * pnl_reward
- self.config.risk_weight * risk_penalty
- self.config.cost_weight * cost_penalty
},
TradingAction::Hold => {
// Dynamic HOLD reward based on price movement
self.calculate_hold_reward(current_state, next_state)?
},
#[test]
fn test_ratio_clipping() -> Result<(), MLError> {
let device = Device::Cpu;
let config = FlowMatchingConfig {
clip_epsilon: 0.2,
..Default::default()
};
// Calculate diversity bonus (in-training regularization)
// Entropy threshold: 0.5 (50% of max entropy for 3 actions = 1.585)
// Low entropy → action bias → apply penalty (-0.1)
let entropy = calculate_entropy(recent_actions);
let entropy_threshold = Decimal::try_from(0.5).unwrap_or(Decimal::ZERO);
let diversity_bonus = if entropy < entropy_threshold {
self.config.diversity_weight // -0.1 (penalty for low diversity)
} else {
Decimal::ZERO // No penalty for balanced actions
// This log_det difference will produce a large ratio that should be clipped
let new_log_dets = Tensor::from_vec(vec![2.0], 1, &device)?;
let old_log_dets = Tensor::from_vec(vec![0.0], 1, &device)?;
let advantages = Tensor::from_vec(vec![10.0], 1, &device)?; // Positive advantage
// log_flow_ratio = 2.0
// ratio = exp(2.0) ≈ 7.389
// clipped_ratio = clamp(7.389, 0.8, 1.2) = 1.2
// surr1 = 7.389 * 10 ≈ 73.89
// surr2 = 1.2 * 10 = 12.0
// min = 12.0
// loss = -12.0
let loss =
compute_flow_matching_loss(&new_log_dets, &old_log_dets, &advantages, &config, &device)?;
let loss_val = loss.to_scalar::<f32>()?;
assert!((loss_val - (-12.0)).abs() < 1e-4);
// Test with negative advantage
let advantages_neg = Tensor::from_vec(vec![-10.0], 1, &device)?;
// surr1 = 7.389 * -10 ≈ -73.89
// surr2 = 1.2 * -10 = -12.0
// min(surr1, surr2) is surr1 because we want to decrease the probability of this action
// min ≈ -73.89
// loss = -(-73.89) ≈ 73.89
let loss_neg_adv = compute_flow_matching_loss(
&new_log_dets,
&old_log_dets,
&advantages_neg,
&config,
&device,
)?;
let loss_val_neg_adv = loss_neg_adv.to_scalar::<f32>()?;
assert!((loss_val_neg_adv - 73.8905).abs() < 1e-4);
Ok(())
}
#[test]
fn test_log_ratio_clipping() -> Result<(), MLError> {
let device = Device::Cpu;
let config = FlowMatchingConfig {
log_ratio_clip_min: -1.0,
log_ratio_clip_max: 1.0,
..Default::default()
};
let final_reward = base_reward + diversity_bonus;
// This log_det difference is large and should be clipped before exp()
let new_log_dets = Tensor::from_vec(vec![10.0], 1, &device)?;
let old_log_dets = Tensor::from_vec(vec![0.0], 1, &device)?;
let advantages = Tensor::from_vec(vec![1.0], 1, &device)?;
// Update normalizer with the raw, unclamped reward
self.normalizer.update(final_reward);
// log_flow_ratio = 10.0
// clipped_log_ratio = 1.0
// ratio = exp(1.0) ≈ 2.718
// clipped_ratio = clamp(2.718, 0.8, 1.2) = 1.2
// surr1 ≈ 2.718
// surr2 = 1.2
// min = 1.2
// loss = -1.2
// Normalize the reward before clamping
let normalized_reward = self.normalizer.normalize(final_reward);
let loss =
compute_flow_matching_loss(&new_log_dets, &old_log_dets, &advantages, &config, &device)?;
let loss_val = loss.to_scalar::<f32>()?;
assert!((loss_val - (-1.2)).abs() < 1e-4);
// Clamp reward to prevent cumulative explosions. This is a final safeguard.
let clamped_reward = normalized_reward.clamp(Decimal::from(-1), Decimal::ONE);
// Store the original, unclamped reward for statistical analysis
self.reward_history.push(final_reward);
if self.reward_history.len() > 1000 {
self.reward_history.remove(0);
}
Ok(clamped_reward)
Ok(())
}
/// Calculate P&L-based reward component as a percentage of portfolio value.
///
/// This approach creates a more stationary reward signal, as the reward is
/// proportional to the percentage gain/loss rather than the absolute dollar amount.
/// A 0.5% gain yields a similar reward regardless of whether the portfolio is $100K or $1M.
fn calculate_pnl_reward(
&self,
current_state: &TradingState,
next_state: &TradingState,
) -> Result<Decimal, MLError> {
// Validate portfolio_features length (defensive check)
if current_state.portfolio_features.is_empty() {
tracing::warn!("Current state portfolio_features is empty, using 0.0 for P&L reward");
return Ok(Decimal::ZERO);
}
if next_state.portfolio_features.is_empty() {
tracing::warn!("Next state portfolio_features is empty, using 0.0 for P&L reward");
return Ok(Decimal::ZERO);
}
let current_value =
Decimal::try_from(*current_state.portfolio_features.get(0).unwrap_or(&100000.0) as f64)
.unwrap_or_else(|_| Decimal::from(100000));
let next_value =
Decimal::try_from(*next_state.portfolio_features.get(0).unwrap_or(&100000.0) as f64)
.unwrap_or(current_value);
// Avoid division by zero if portfolio value is somehow zero.
if current_value.is_zero() {
return Ok(Decimal::ZERO);
}
let pnl_change = next_value - current_value;
// Calculate P&L as a percentage of portfolio value to create a stationary reward signal.
let pnl_percentage = pnl_change / current_value;
// Scale percentage to a more intuitive reward magnitude.
// A 1% portfolio gain (pnl_percentage = 0.01) becomes a reward of 1.0.
// A 0.1% gain (0.001) becomes a reward of 0.1.
let scaled_pnl = pnl_percentage * Decimal::from(100);
Ok(scaled_pnl)
}
/// Calculate risk penalty
///
// ... (rest of the file is unchanged)
}
```
</UPDATED_EXISTING_FILE>
</GENERATED-CODE>