- Rename node-dns-fix → node-bootstrap, fix nvidia gate race condition
(always create conf.d/ and write 50-registry.toml, don't gate on
99-nvidia.toml which doesn't exist on fresh autoscaled GPU nodes)
- Update busybox 1.36 → 1.37
- Fix fetch-binary: use PRIVATE-TOKEN/gitlab-pat (not DEPLOY-TOKEN)
- Fix upload-results: use gitlab-pat (gitlab-ci-token didn't exist)
- Add 'latest' rolling package version in both compile-services and
compile-training (re-upload after CalVer upload)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Service pods' initContainers need egress to gitlab-webservice-default:8181
to fetch release binaries from the Generic Package Registry. Without this,
curl hangs for 2+ minutes then times out (default-deny-all blocks it).
Covers all app.kubernetes.io/part-of=foxhunt pods in one policy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Argo Workflows can't resolve {{tasks.X.outputs}} inside template
bodies — only in DAG task arguments. Pass tag via arguments →
inputs.parameters in compile-services, compile-training,
upload-release, and deploy-services templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The hyperopt preload_data() created a loader with default hyperparams
(mbp10_data_dir=None), so MBP-10/trades data was never loaded. Each
trial then set mbp10_data_dir → state_dim=51, but the preloaded data
had no OFI features → shape mismatch [128,43] vs [51,1024].
- Pass mbp10_data_dir/trades_data_dir to preload hyperparams
- Extract ofi_features from loader after preload
- Store as preloaded_ofi_features: Option<Arc<Vec<[f64;8]>>>
- Inject into each trial's DQNTrainer before training
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace MinIO binary distribution with GitLab Generic Package Registry.
Every code push to main auto-creates a CalVer tag (vYYYY.MM.N),
compiles, uploads binaries to GitLab packages, creates a Release
with auto-generated notes, and deploys via deployment patching.
- New CI templates: create-tag, upload-release
- Modified: compile-services/training upload to GitLab packages
- Modified: deploy-services patches FOXHUNT_RELEASE on deployments
- All 7 service initContainers fetch from GitLab (curl, deploy token)
- Training job-template binary fetch from GitLab (data stays MinIO)
- MinIO retains: sccache, training data, model checkpoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU path (build_batch_states / build_state_tensor) only concatenated
market[40] + portfolio[3] = 43 dims, while state_dim was set to 51 when
OFI was enabled. This caused shape mismatch [128,43] vs [51,1024].
- Add ofi_features: Option<Tensor> field to DqnGpuData
- Add upload_ofi() method for MBP-10 order book features (8 dims/bar)
- Concatenate OFI in build_batch_states: [count,40]+[count,3]+[count,8]=[count,51]
- Concatenate OFI in build_state_tensor: [1,40]+[1,3]+[1,8]=[1,51]
- Wire trainer to call upload_ofi() after GPU data upload
- Fix CPU path to zero-pad regime_features when OFI enabled but data missing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The training-data PVC already has all data (OHLCV, MBP-10, trades).
Mount it read-only at /data in hyperopt/train/evaluate steps instead
of rcloning ~50GB from MinIO every step.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CI pipeline: add gpu-warmup step parallel to compile-training,
triggers GPU node autoscale so it's ready when training starts
- Training workflow: add mbp10-data-dir and trades-data-dir parameters,
download MBP-10 + trades data in all steps (hyperopt, train, evaluate)
- Use parameterized paths consistently (no hardcoded /tmp paths)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --mbp10-data-dir and --trades-data-dir CLI args to
hyperopt_baseline_rl binary so hyperopt trials can use real
order book and trade data for VPIN/Kyle's Lambda features.
- DQNTrainer: add mbp10_data_dir/trades_data_dir fields + with_ofi_data_dirs() builder
- DQNHyperparameters: pipe through from trainer instead of hardcoded None
- download-trades-job: fix nodeSelector to ci-compile-cpu (platform pool full)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Expand training-data-pvc from 10Gi → 200Gi (OHLCV + MBP-10 + headroom)
- Add data-sync-job: rclone sync from MinIO → PVC (delta-only, fast repeats)
- Add sync-training-data init container to training job template
(auto-syncs data from MinIO before each training run)
- Add training-job + data-sync-job NetworkPolicies (MinIO, Tempo, Pushgateway)
- Add app.kubernetes.io/part-of: foxhunt to training pod template
- download_baseline: add --parallel N flag for concurrent Databento downloads
- download_baseline: delete local files after MinIO upload (saves scratch space)
- Bump download job scratch volume to 100Gi
Architecture: Databento → MinIO (source of truth) → PVC (local cache).
First sync pulls everything; subsequent runs only sync deltas (seconds).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The download job pod was missing app.kubernetes.io/part-of: foxhunt,
so the default-deny-all egress policy blocked it and MinIO's ingress
policy rejected it. Also removed hostname pinning (was a misdiagnosis).
Added data-download-job NetworkPolicy allowing egress to MinIO:9000
and external HTTPS:443 (Databento API).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire 8 OFI features (OFI L1/L5, depth imbalance, VPIN, Kyle's lambda,
bid/ask slopes, trade imbalance) through the DQN training pipeline:
- Add mbp10_data_dir config field to DQNHyperparameters
- Dynamic state_dim: 43 (no OFI) or 51 (with OFI) based on config
- Compute OFI per bar during data loading, store on trainer
- Pass OFI features through regime_features slot in TradingState
- Configurable MBP-10 path with recursive .dbn/.dbn.zst discovery
- Add zstd auto-detection to DbnParser::parse_mbp10_file()
- Add --mbp10-data-dir CLI flag to train_baseline_rl
- Fix hardcoded [f64; 51] → FeatureVector51 ([f64; 40]) across
examples, walk_forward, GPU memory profile, and test fixtures
- Fix stale state_dim=51 in dqn_config_2025() and DQN tests
2747 tests pass, 0 failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- download_baseline now reads schema from universe TOML config
(was hardcoded to ohlcv-1m, now supports mbp-10 and other schemas)
- Add --rclone-dest flag for direct upload to MinIO after each download
- Add config/universe-es-mbp10.toml: ES.FUT MBP-10, same date range
- Add infra/k8s/training/download-mbp10-job.yaml: K8s Job to download
ES.FUT MBP-10 data in-cluster and upload to MinIO
Usage:
kubectl apply -f infra/k8s/training/download-mbp10-job.yaml
Prereqs: databento-credentials secret, download_baseline binary in MinIO
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
batch_greedy_actions, batch_softmax_actions, and
batch_hierarchical_softmax_actions all used self.forward() which routes
through the dist_dueling/dueling/standard Q-network. When use_iqn=true,
the training loss trains the IQN QuantileNetwork but inference never
consulted it — the trained IQN weights were ignored at evaluation time.
Add q_values_for_batch() helper that dispatches to the IQN network
(with CVaR or expected-Q reduction) when use_iqn=true, and wire all
three batch methods through it. select_action, select_action_with_confidence,
and select_action_inference already had correct IQN branches.
Add test_iqn_batch_greedy_actions_uses_iqn_network covering training,
batch greedy, batch softmax, and inference paths with IQN enabled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add RMSNorm (root mean square normalization) after each hidden layer
in the distributional-dueling Q-network: shared backbone layers, value
stream, and advantage stream. RMSNorm stabilizes activations and
gradients without the overhead of full LayerNorm (no mean centering),
making the network less sensitive to input scale during training.
Architecture per layer: Linear -> LeakyReLU -> RMSNorm
RMSNorm weights are automatically tracked in the existing VarMap since
they are created via VarBuilder::from_varmap with the same shared map.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove all hardcoded Y-axis min/max bounds (41 removals) — dynamic scaling
- Eliminate Hyperopt section: merge into Training Status + Run Summary
- Deduplicate trial progress (3 panels → 1 gauge + 1 stat)
- Combine ML Jobs + Errors into single panel
- Add Current Trial and Hyperopt Elapsed to status bar
- Add Best Objective Δ (deriv) trend indicator
- Smooth line interpolation + gradient fills on all timeseries
- Stacked area for Action Distribution, gradient fill for Replay Buffer
- Shared crosshair tooltips, consistent threshold colors
- Status bar: stat panels with sparklines, trial before epoch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Walk-forward windows: replaced 3 non-overlapping with sliding (50% overlap, ~5 windows).
Aggregation changed from mean-0.5*std to median-0.5*IQR for outlier robustness.
2. Composite score: tanh normalization prevents Calmar ratio scale dominance
(0.02% drawdowns → values in thousands drowning out Sharpe/Sortino).
3. Q-value overestimation: new Prometheus gauge foxhunt_training_q_overestimation_ratio,
warning log when ratio>10 or q_mean>5, adaptive tau doubles when Q-mean growth>0.5/epoch
(capped at 0.01), decays back when stable.
2742 tests pass, 0 failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
nvidia-smi is driver-mounted by the GPU operator, which may not be
ready when the warmup pod starts on a fresh autoscaled node. The
warmup's purpose is just triggering autoscale, not GPU validation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous fix removed ALL trailing backslashes from --s3-no-check-bucket,
but 3 of 6 occurrences need the continuation for --transfers=8 on the
next line. Restores \ on lines where --transfers follows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Same bug as service manifests — \\ after --s3-no-check-bucket caused
chmod to be parsed as rclone args. Fixed in 6 places.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both Dockerfiles already install mold v2.35.1 but the registry images
are stale builds without it. This change triggers rebuild-ci-builder
and rebuild-ci-builder-cpu pipeline steps via detect-changes.
.cargo/config.toml uses -fuse-ld=mold — without mold in the image,
linking falls back to the system default (slower).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace stub StalenessTracking trait with real StalenessTracker:
- Per-slot timestamp tracking (last_updated_step per experience)
- get_stale_indices() returns oldest-first for refresh
- Wired into PrioritizedReplayBuffer (mark_updated on priority update)
- Epoch-boundary refresh in DQN trainer: recomputes priorities for
stale entries (>500 steps old) via Q-value forward pass
- 7 staleness tests + 23 PER tests pass, 0 clippy
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- C1: Narrow V_min/V_max from [-15,+15] to [-2,+2] for C51 atom resolution
- C2: Remove exploration stacking (28D→26D), keep NoisyNet only
- C3: Fix entropy regularizer for 5-action space, remove 2x/3x multipliers
- H1: Narrow gamma to [0.88, 0.96] (avoid overnight gap discounting)
- H2: Raise tau lower bound to 0.005 (target net tracks in short trials)
- H3: Cap kelly_fractional at 0.75 (no full-Kelly ruin)
- M1: Enable QR-DQN when num_atoms > 100 (hyperopt-toggled)
- M3: Add Q-value gap logging (Q_best - Q_second_best) per epoch
2735 tests pass, 0 clippy warnings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The objective function was using buy/sell/hold percentages from training
metrics instead of the backtest. This caused the optimizer to receive stale
action distribution signals that diverged from actual backtest behavior.
Added buy_action_pct/sell_action_pct/hold_action_pct to BacktestMetrics,
counted during the multi-window backtest loop, and used in extract_objective
when backtest metrics are available.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Patience-based early stopping (WAVE 24) was not gated by early_stopping_enabled,
causing ALL hyperopt trials to terminate early and return penalty metrics with
backtest_metrics: None — no backtest ever ran, producing FALLBACK OBJECTIVE 44.6.
Also removes eval_softmax_temp from search space (29D→28D) since backtest uses
greedy argmax, widens gamma range (0.95-0.99→0.90-0.999) for better TPE signal,
and updates all tests to match.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Greedy argmax legitimately converges to 1-2 actions when the model
is confident. The hard gate (unique_actions < 3 → 8.0 penalty) was
blocking ALL trials from using real backtest metrics, forcing FALLBACK.
Changes:
- Remove hard diversity short-circuit gate (unique_actions < 3)
- Reduce soft diversity penalty from 3.0 to 0.8 max (mild TPE signal)
- Fix early_stopping_enabled: false (was self.epochs > 12)
- Update test to validate mono-action policies produce good objectives
Multi-window backtest (3 windows, mean - 0.5*std) already handles
phantom Sharpe from single-bucket flukes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
apply_position_mask() in factored_q_network.rs looped 0..45 against a
5-wide Q-values tensor — would panic at runtime. Changed to 0..5 using
ExposureLevel::from_index() directly. Updated stale "45 actions" comments
in 5 files (dqn.rs, reward.rs, hyperopt/adapters/dqn.rs, curriculum.rs).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
A: Switch backtest eval from Gumbel softmax to greedy argmax (batch_greedy_actions)
so hyperopt Sharpe reflects the agent's actual learned policy, not noisy sampling.
C: Disable reward normalization (enable_normalization=false). EMA normalizer with
±3.0 clipping was flattening the reward landscape, preventing the agent from
distinguishing large winners from scratch trades.
D: Wire tx_cost_bps (0.1 bps for IBKR ES) through to EvaluationEngine via
new_with_fee_rate(). Previously hardcoded at 15 bps (150x mismatch with actual
commission costs), massively penalizing every trade in backtest.
E: Scale PnL reward by agent's target exposure in calculate_pnl_reward().
Previously, a Short100 action received POSITIVE reward when market went up
(pct_return ignored position direction). Now: reward = pct_return × exposure.
2735 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Early stopping with 8-epoch trials returns penalty metrics (no backtest),
causing ALL trials to hit FALLBACK OBJECTIVE = 44.6 regardless of actual
model quality. The Sharpe was 1.36-1.61 but natural fluctuation triggered
"Sharpe worsening" at epoch 5, killing the backtest evaluation.
Fix: disable early stopping when epochs ≤ 12. With ~90s per trial, running
all 8 epochs is cheap. Long training runs (50 epochs) still use it.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The C2 fix from the previous session was too aggressive — it eliminated
ALL exploration mechanisms simultaneously:
1. epsilon forced to 0.0 when noisy nets active (select_action)
2. count bonus removed from Q-value computation (metrics only)
3. noisy_epsilon_floor config field declared but never read
This left noisy nets as the sole exploration mechanism, which produces
perturbations too small to overcome Q-value gaps (A4=0.12 vs others≈0.02).
Result: 20/20 hyperopt trials hit fallback objective with 1/5 diversity.
Fixes:
- select_action: use noisy_epsilon_floor (not 0.0) as effective_epsilon
when noisy nets active — guarantees minimum random action rate
- select_action: re-enable UCB count bonus on Q-values before argmax
(both IQN and standard paths) for directed exploration
- select_action_with_confidence: same fixes for consistency
- trainer: set epsilon to noisy_epsilon_floor (not 0.0) at init
- hyperopt: widen noisy_epsilon_floor range from [0.0, 0.05] to
[0.03, 0.15] with default 0.05
Exploration now has two complementary mechanisms:
- noisy_epsilon_floor: random actions feed diverse replay buffer
- count bonus: UCB term biases greedy selection toward under-visited actions
- noisy nets: weight perturbation adds stochasticity to Q-values
select_action_inference (production) is unchanged — pure exploitation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Critical bug: all 3 DQN action selection methods (select_action,
select_action_with_confidence, select_action_inference) used
FactoredAction::from_index() which maps indices 0-4 to exposure_idx=0
(Short100) via division by 9. This is the root cause of action
diversity collapse during both training and production inference.
Fix: ExposureLevel::from_index() + OrderRouter::route_default() in all
DQN paths. Also fixes hyperopt objective thresholds (<10 → <3 for
5-action degenerate detection), stale defaults/comments, integration
test configs.
Files: dqn.rs (3 methods), trainer.rs (validation + select_action),
hyperopt/adapters/dqn.rs (thresholds), dqn_model.rs (comments),
train_baseline_rl.rs (default), reward.rs (comment),
dqn_integration.rs + ensemble_integration.rs (num_actions).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- agent.rs: select_action_factored() now uses ExposureLevel::from_index()
+ OrderRouter::route_default() instead of FactoredAction::from_index().
Previously, indices 0-4 mapped to all-Short100 variants in the 45-action
space — now correctly maps to 5 distinct exposure levels.
- hyperopt: plateau_window .max(3) → .max(5) to prevent premature early
stopping with short trial epochs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: 45 factored actions (5 exposure × 3 order × 3 urgency) caused
reward degeneracy — 9 actions per exposure level produced nearly identical
rewards since order type/urgency had 1000-4000x weaker signal than PnL.
This collapsed action diversity as DQN couldn't differentiate actions.
Changes:
- DQN now outputs 5 Q-values (Short100, Short50, Flat, Long50, Long100)
- New OrderRouter deterministically maps exposure → (order_type, urgency)
based on spread and volatility microstructure signals
- PPO retains full 45-action space (separate CUDA constants DQN_NUM_ACTIONS
vs PPO_NUM_ACTIONS)
- CUDA kernels: DQN diversity entropy uses 5 categories, PPO keeps 45
- Phase B: pnl_history cleared per epoch so Sharpe reflects current epoch
(was accumulating across all epochs, causing frozen Sharpe metric)
24 files, 2728 tests pass, 0 clippy warnings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sharpe-based early stopping kills every hyperopt trial at epoch 4
because compute_epoch_financials() is deterministic (greedy argmax on
fixed validation data) — the model doesn't change enough in 8 short
epochs to shift any argmax decisions, making Sharpe bit-identical
across epochs and triggering plateau detection immediately.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kubernetes defaults to Always for :latest tags, forcing registry
round-trips that fail on fresh GPU nodes where containerd HTTP-only
registry config has a race condition with HTTPS fallback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Addresses eval/training mismatch (B1-B3), reward architecture (C1-C3),
and early stopping (C4). See design doc for full analysis.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous C4 fix re-enabled early stopping with adaptive plateau
window but still used val-loss as the stopping metric. Val-loss (TD
Bellman residual) can plateau while trading strategy still improves.
Now:
- Best-checkpoint saved when epoch Sharpe improves (not val-loss)
- Plateau detection checks sharpe_history (not val_loss_history)
- Patience-based EarlyStopping receives -Sharpe (negate for lower=better API)
- Per-epoch Sharpe extracted from compute_epoch_financials() (already computed)
This ensures early stopping and best-model selection track the metric
that actually matters for hyperopt: trading performance.
2720 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
