foxhunt/REGIME_TRACKING_QUICK_REFERENCE.md

Regime Tracking Database - Quick Reference

Status: ✅ Production Ready (13/13 tests passing) Location: Migration 045 (migrations/045_wave_d_regime_tracking.sql) Test File: common/tests/wave_d_regime_tracking_tests.rs

---

Quick Commands

Run Tests

# With database connected (generates SQLX cache)
SQLX_OFFLINE=false cargo test -p common --test wave_d_regime_tracking_tests --features database -- --test-threads=1

# With offline cache
cargo test -p common --test wave_d_regime_tracking_tests --features database -- --test-threads=1

Database Queries

# Connect to database
psql postgresql://foxhunt:foxhunt_dev_password@localhost:5432/foxhunt

# Get latest regime for symbol
SELECT * FROM get_latest_regime('ES.FUT');

# Get transition matrix (last 24 hours)
SELECT * FROM get_regime_transition_matrix('ES.FUT', 24);

# Get performance by regime
SELECT * FROM get_regime_performance('ES.FUT', 24);
SELECT * FROM get_regime_performance(NULL, 24);  -- All symbols

---

Tables at a Glance

regime_states (12 columns, 5 indexes)

Purpose: Current regime + CUSUM/ADX/stability metrics Key Fields: symbol, regime, confidence, cusum_s_plus, cusum_s_minus, adx, stability UPSERT Key: (symbol, event_timestamp) Valid Regimes: Normal, Trending, Ranging, Volatile, Crisis, Illiquid, Momentum

regime_transitions (10 columns, 4 indexes)

Purpose: Regime change history Key Fields: from_regime, to_regime, duration_bars, transition_probability, adx_at_transition Constraint: from_regime != to_regime (enforced by CHECK)

adaptive_strategy_metrics (12 columns, 5 indexes)

Purpose: Adaptive strategy performance per regime Key Fields: position_multiplier (0.0-2.0), stop_loss_multiplier (1.0-5.0), regime_sharpe, total_trades, winning_trades, total_pnl UPSERT Key: (symbol, event_timestamp, regime) Accumulation: total_trades, winning_trades, total_pnl (on conflict, adds values)

---

Rust API (DatabasePool methods)

use common::database::DatabasePool;

// Get latest regime
let regime = pool.get_latest_regime("ES.FUT").await?;
println!("Current regime: {} (confidence: {})", regime.regime, regime.confidence);

// Insert regime state (UPSERT)
pool.insert_regime_state(
    "ES.FUT",
    "Trending",
    0.85,
    chrono::Utc::now(),
    Some(2.5),  // cusum_s_plus
    Some(-1.2), // cusum_s_minus
    Some(45.0), // adx
    Some(0.92)  // stability
).await?;

// Insert regime transition
pool.insert_regime_transition(
    "ES.FUT",
    "Normal",
    "Trending",
    chrono::Utc::now(),
    Some(120),  // duration_bars
    Some(0.35), // transition_probability
    Some(48.5), // adx_at_transition
    true        // cusum_alert_triggered
).await?;

// Upsert adaptive strategy metrics (accumulates trades/PnL)
pool.upsert_adaptive_strategy_metrics(
    "ES.FUT",
    "Trending",
    chrono::Utc::now(),
    1.5,        // position_multiplier
    2.5,        // stop_loss_multiplier
    Some(2.1),  // regime_sharpe
    Some(0.80), // risk_budget_utilization
    15,         // total_trades
    12,         // winning_trades
    25000       // total_pnl
).await?;

// Get regime performance
let performance = pool.get_regime_performance(Some("ES.FUT"), 24).await?;
for perf in performance {
    println!("{}: {} trades, {:.2}% win rate, Sharpe {:.2}",
        perf.regime, perf.total_trades, perf.win_rate * 100.0,
        perf.avg_sharpe.unwrap_or(0.0));
}

// Get transition history
let transitions = pool.get_regime_transitions("ES.FUT", 10).await?;
for trans in transitions {
    println!("{} -> {} (duration: {} bars)",
        trans.from_regime, trans.to_regime,
        trans.duration_bars.unwrap_or(0));
}

---

Performance Targets vs. Actual

Operation	Target	Actual	Status
Regime State Insert	<10ms	0.07ms	✅ 143x faster
Latest Regime Query	<10ms	1.68ms	✅ 6x faster
Transition Insert	<10ms	0.86ms	✅ 12x faster
Transition Matrix	<50ms	1.47ms	✅ 34x faster
Performance Query	<50ms	1.02ms	✅ 49x faster

Average Improvement: 48.8x faster than targets

---

Production Recommendations

⚠️ TODO Before Production

1. Convert to TimescaleDB Hypertables (MEDIUM priority)

   • Enables automatic partitioning by time
   • Required for 100M+ rows
   • Reduces storage by 95% with compression
   • See migration example in migrations/022_create_ensemble_tables.sql

   SELECT create_hypertable('regime_states', 'event_timestamp',
       chunk_time_interval => INTERVAL '1 day',
       migrate_data => TRUE
   );
   ALTER TABLE regime_states SET (timescaledb.compress, timescaledb.compress_segmentby = 'symbol');
   SELECT add_compression_policy('regime_states', INTERVAL '7 days');
   

2. Add Test for get_regime_transitions() (LOW priority)

   • Currently untested in isolation
   • Covered indirectly via transition matrix function

3. Create Rollback Migration (LOW priority)

   • Add 045_wave_d_regime_tracking_down.sql with DROP statements

---

Example Queries

Find High-Confidence Trending Regimes

SELECT symbol, confidence, adx, event_timestamp
FROM regime_states
WHERE regime = 'Trending' AND confidence > 0.85
ORDER BY confidence DESC, adx DESC
LIMIT 10;

Most Common Regime Transitions

SELECT from_regime, to_regime, COUNT(*) AS count,
       AVG(duration_bars) AS avg_duration
FROM regime_transitions
WHERE event_timestamp >= NOW() - INTERVAL '7 days'
GROUP BY from_regime, to_regime
ORDER BY count DESC;

Best Performing Regime (by Sharpe)

SELECT regime, COUNT(*) AS occurrences,
       AVG(regime_sharpe) AS avg_sharpe,
       SUM(total_trades) AS total_trades,
       SUM(total_pnl) AS total_pnl
FROM adaptive_strategy_metrics
WHERE event_timestamp >= NOW() - INTERVAL '30 days'
  AND regime_sharpe IS NOT NULL
GROUP BY regime
ORDER BY avg_sharpe DESC;

Regime Stability Analysis

SELECT symbol, regime, AVG(stability) AS avg_stability,
       COUNT(*) AS observations
FROM regime_states
WHERE event_timestamp >= NOW() - INTERVAL '7 days'
GROUP BY symbol, regime
HAVING COUNT(*) > 10
ORDER BY avg_stability DESC;

---

SQLX Cache Files

Location: common/.sqlx/ Count: 6 files (generated by test run) Total Size: 7.2 KB Regenerate: Run tests with SQLX_OFFLINE=false

---

Testing Checklist

• Regime state insert (basic)
• Regime state UPSERT (conflict handling)
• Latest regime retrieval
• Regime constraints (7 valid values)
• Transition insert (basic)
• Transition constraints (from != to)
• Multiple transitions
• Transition matrix calculation
• Adaptive metrics UPSERT
• Adaptive metrics accumulation (trades/PnL)
• Performance aggregation
• End-to-end workflow
• Concurrent updates (5 parallel inserts)
• Direct get_regime_transitions() test (indirect coverage exists)

---

Related Files

• Migration: migrations/045_wave_d_regime_tracking.sql
• Test File: common/tests/wave_d_regime_tracking_tests.rs
• Database Module: common/src/database.rs:356-599 (regime tracking methods)
• Types: common/src/database.rs (RegimeState, RegimeTransition, RegimePerformance structs)
• Validation Report: AGENT_F10_REGIME_TRACKING_VALIDATION_REPORT.md

---

Last Updated: 2025-10-18 Status: ✅ Production Ready (with TimescaleDB hypertable conversion recommended)