foxhunt/docs/HER_QUICK_REF.md

Hindsight Experience Replay (HER) - Quick Reference

What is HER?

HER relabels failed experiences with achieved goals to improve learning efficiency by 5-10x. Instead of discarding failures, it treats them as successes for different goals.

File Location

/home/jgrusewski/Work/foxhunt/ml/src/dqn/hindsight_replay.rs

Quick Start

use ml::dqn::{HindsightReplayBuffer, HindsightReplayConfig, HindsightStrategy};

// 1. Create buffer
let config = HindsightReplayConfig {
    goal_dim: 1,              // Goal size in state
    her_ratio: 0.5,           // 50% HER samples
    her_strategy: HindsightStrategy::Future,
    batch_size: 32,
    ..Default::default()
};

let buffer = HindsightReplayBuffer::new(config)?;

// 2. Add experiences
buffer.push(experience)?;
buffer.mark_episode_boundary(); // On episode end

// 3. Sample with HER
let (batch, weights, indices) = buffer.sample_with_hindsight(32)?;

// 4. Update priorities after training
buffer.update_priorities(&indices, &td_errors)?;

Strategies

Strategy	Description	Best For
Final	Use final achieved state	Episodic tasks
Future	Sample future achieved states	Long episodes
Episode	Sample from episode	Varied goals
Random	Random from buffer	General use

Configuration

HindsightReplayConfig {
    base_config: PrioritizedReplayConfig::default(),
    goal_dim: 1,        // Default: 1 (target return)
    her_ratio: 0.5,     // Default: 50% HER samples
    her_strategy: HindsightStrategy::Final,
    k_future: 4,        // Future goals to sample
    batch_size: 32,
}

API Methods

// Core operations
fn new(config: HindsightReplayConfig) -> Result<Self, MLError>
fn push(&self, experience: Experience) -> Result<(), MLError>
fn sample_with_hindsight(&self, batch_size: usize)
    -> Result<(ExperienceBatch, Vec<f32>, Vec<usize>), MLError>

// Priority updates
fn update_priorities(&self, indices: &[usize], priorities: &[f32]) -> Result<(), MLError>

// Episode tracking
fn mark_episode_boundary(&self)

// Stats
fn stats(&self) -> HindsightReplayStats
fn size(&self) -> usize
fn can_sample(&self) -> bool

Statistics

let stats = buffer.stats();
println!("Normal: {}, HER: {}, Relabeled: {}, Avg improvement: {:.3}",
    stats.normal_samples,
    stats.her_samples,
    stats.relabeled_count,
    stats.avg_reward_improvement
);

State Representation

Goal format: First goal_dim elements of state vector

state = [goal, feature1, feature2, ...]
         ^^^^  ^^^^^^^^^^^^^^^^^^^^^
         goal  non-goal features

Example (goal_dim=1):

state = [target_return, price, volume, indicators...]

Reward Relabeling

// Original experience: state=[0.5, ...], reward=-1.0 (failed to reach 0.5)
// Achieved state: [0.3, ...]

// HER relabels:
// new_state = [0.3, ...] (goal changed to achieved)
// new_reward = 1.0 (now succeeded at reaching 0.3)

Integration with DQN Trainer

// Replace ReplayBuffer with HindsightReplayBuffer
let her_buffer = HindsightReplayBuffer::new(config)?;

// Training loop
for episode in episodes {
    for step in episode {
        her_buffer.push(experience)?;
    }
    her_buffer.mark_episode_boundary();

    if her_buffer.can_sample() {
        let (batch, weights, indices) = her_buffer.sample_with_hindsight(32)?;

        // Train DQN
        let td_errors = train_step(batch, weights)?;

        // Update priorities
        her_buffer.update_priorities(&indices, &td_errors)?;
    }
}

Performance Benefits

• 5-10x sample efficiency
• Learn from failures
• Better generalization
• Faster convergence
• Sparse reward handling

Test Coverage

✅ 15 comprehensive unit tests

• Buffer creation & storage
• All 4 strategies
• Goal relabeling correctness
• HER ratio distribution
• Episode boundaries
• Statistics tracking
• Error handling

Thread Safety

• ✅ Concurrent reads
• ✅ Thread-safe sampling
• ✅ Atomic counters
• ✅ RwLock for buffers

Memory Overhead

• Minimal: Only episode boundaries tracking
• O(1) per experience
• Shares base PrioritizedReplayBuffer

When to Use

✅ Use HER when:

• Sparse reward environments
• Goal-conditioned tasks
• Learning from failures
• Sample efficiency critical

❌ Don't use when:

• Dense rewards (no benefit)
• Goal not in state
• No clear goal representation

Common Patterns

Trading strategy:

// Goal = target return
HindsightReplayConfig {
    goal_dim: 1,              // Return target
    her_ratio: 0.8,           // High HER ratio
    her_strategy: HindsightStrategy::Future,
    ..Default::default()
}

Multi-asset portfolio:

// Goal = portfolio allocation
HindsightReplayConfig {
    goal_dim: num_assets,     // Allocation per asset
    her_ratio: 0.5,
    her_strategy: HindsightStrategy::Episode,
    ..Default::default()
}

Troubleshooting

Issue: Low reward improvement

• Fix: Increase her_ratio (try 0.8)

Issue: Sampling errors

• Fix: Ensure goal_dim matches state size

Issue: Poor strategy performance

• Fix: Try different HindsightStrategy variants

References

• Paper: "Hindsight Experience Replay" (Andrychowicz et al., 2017)
• Implementation: /home/jgrusewski/Work/foxhunt/ml/src/dqn/hindsight_replay.rs
• Full report: /home/jgrusewski/Work/foxhunt/docs/WAVE26_P1.7_HER_IMPLEMENTATION.md