The C2 fix from the previous session was too aggressive — it eliminated
ALL exploration mechanisms simultaneously:
1. epsilon forced to 0.0 when noisy nets active (select_action)
2. count bonus removed from Q-value computation (metrics only)
3. noisy_epsilon_floor config field declared but never read
This left noisy nets as the sole exploration mechanism, which produces
perturbations too small to overcome Q-value gaps (A4=0.12 vs others≈0.02).
Result: 20/20 hyperopt trials hit fallback objective with 1/5 diversity.
Fixes:
- select_action: use noisy_epsilon_floor (not 0.0) as effective_epsilon
when noisy nets active — guarantees minimum random action rate
- select_action: re-enable UCB count bonus on Q-values before argmax
(both IQN and standard paths) for directed exploration
- select_action_with_confidence: same fixes for consistency
- trainer: set epsilon to noisy_epsilon_floor (not 0.0) at init
- hyperopt: widen noisy_epsilon_floor range from [0.0, 0.05] to
[0.03, 0.15] with default 0.05
Exploration now has two complementary mechanisms:
- noisy_epsilon_floor: random actions feed diverse replay buffer
- count bonus: UCB term biases greedy selection toward under-visited actions
- noisy nets: weight perturbation adds stochasticity to Q-values
select_action_inference (production) is unchanged — pure exploitation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Critical bug: all 3 DQN action selection methods (select_action,
select_action_with_confidence, select_action_inference) used
FactoredAction::from_index() which maps indices 0-4 to exposure_idx=0
(Short100) via division by 9. This is the root cause of action
diversity collapse during both training and production inference.
Fix: ExposureLevel::from_index() + OrderRouter::route_default() in all
DQN paths. Also fixes hyperopt objective thresholds (<10 → <3 for
5-action degenerate detection), stale defaults/comments, integration
test configs.
Files: dqn.rs (3 methods), trainer.rs (validation + select_action),
hyperopt/adapters/dqn.rs (thresholds), dqn_model.rs (comments),
train_baseline_rl.rs (default), reward.rs (comment),
dqn_integration.rs + ensemble_integration.rs (num_actions).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- agent.rs: select_action_factored() now uses ExposureLevel::from_index()
+ OrderRouter::route_default() instead of FactoredAction::from_index().
Previously, indices 0-4 mapped to all-Short100 variants in the 45-action
space — now correctly maps to 5 distinct exposure levels.
- hyperopt: plateau_window .max(3) → .max(5) to prevent premature early
stopping with short trial epochs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: 45 factored actions (5 exposure × 3 order × 3 urgency) caused
reward degeneracy — 9 actions per exposure level produced nearly identical
rewards since order type/urgency had 1000-4000x weaker signal than PnL.
This collapsed action diversity as DQN couldn't differentiate actions.
Changes:
- DQN now outputs 5 Q-values (Short100, Short50, Flat, Long50, Long100)
- New OrderRouter deterministically maps exposure → (order_type, urgency)
based on spread and volatility microstructure signals
- PPO retains full 45-action space (separate CUDA constants DQN_NUM_ACTIONS
vs PPO_NUM_ACTIONS)
- CUDA kernels: DQN diversity entropy uses 5 categories, PPO keeps 45
- Phase B: pnl_history cleared per epoch so Sharpe reflects current epoch
(was accumulating across all epochs, causing frozen Sharpe metric)
24 files, 2728 tests pass, 0 clippy warnings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous C4 fix re-enabled early stopping with adaptive plateau
window but still used val-loss as the stopping metric. Val-loss (TD
Bellman residual) can plateau while trading strategy still improves.
Now:
- Best-checkpoint saved when epoch Sharpe improves (not val-loss)
- Plateau detection checks sharpe_history (not val_loss_history)
- Patience-based EarlyStopping receives -Sharpe (negate for lower=better API)
- Per-epoch Sharpe extracted from compute_epoch_financials() (already computed)
This ensures early stopping and best-model selection track the metric
that actually matters for hyperopt: trading performance.
2720 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Multi-window backtest: splits validation data into 3 non-overlapping
windows and aggregates with mean(Sharpe) - 0.5*std(Sharpe), penalizing
inconsistency and reducing overfit to a single data segment.
Top-K ensemble: hyperopt now emits top_k_params (top 5 trials) in JSON
output. train_baseline_rl gains --ensemble-top-k flag to train multiple
models per fold from different hyperopt configs, saving checkpoints as
dqn_ensemble_{k}_fold_{n}.safetensors.
Workflow template: adds ensemble-top-k parameter (default 5) and passes
--ensemble-top-k to the train-best step.
2720 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 root causes of 45-action DQN collapsing to 1-6 actions:
1. Batch epsilon ignoring noisy_epsilon_floor: select_actions_batch()
and select_actions_batch_gpu() used get_epsilon() which returns 0.0
with noisy nets — zero random exploration in the training path.
Added get_effective_epsilon() that respects noisy_epsilon_floor.
2. Entropy coefficient too weak: default 0.05 with bounds (0.01, 0.2)
produced max ~0.19 bonus vs TD loss of 4+. Bumped default to 0.1,
widened bounds to (0.05, 0.5) for effective anti-collapse.
3. count_bonus_coefficient not in search space: was hardcoded at 0.1
in from_continuous(). Promoted to 31st search dimension with bounds
(0.05, 1.0) so PSO/TPE can optimize exploration strength.
4. Diversity penalty too coarse: objective had <10 unique actions
short-circuit but nothing for 10-20. Added graduated penalty that
linearly ramps from 3.0 (10 actions) to 0.0 (20 actions).
Also fixes pre-existing clippy impl_trait_in_params in optimizer.rs.
2720 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ephemeral Argo workflow pods terminate after training completes, causing
Prometheus to lose all scraped metrics. Add push_to_gateway() to POST
final metrics to the existing pushgateway service so they persist on the
Grafana training dashboard after pod completion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The batch training path (select_actions_batch → forward()) does NOT
increment DQN::total_steps. Only select_action() does. When
warmup_steps > 0, train_step() checks total_steps < warmup_steps
and returns (0.0, 0.0) — zero loss, zero gradients. The model
never trained; "results" were random initialization Q-values.
Fix: force warmup_steps=0 in hyperopt adapter and remove
warmup_ratio from the 31D→30D search space (saves a dimension
for TPE/PSO effectiveness).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runs nvidia-smi on GPU node in parallel with fetch-binary, triggering
H100 autoscale during compilation so the node is ready when hyperopt
starts. Exits immediately to free GPU resources.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace Silverman's bandwidth (h = 1.06σn^(-1/5)) with Scott's rule
(h = 0.7σn^(-1/(d+4))) for tighter kernels in high-D parameter spaces
- Add best-trial injection: always evaluate EI at best known point plus
5 small perturbations (±5%), preventing optimizer from forgetting peaks
- Scale n_candidates dynamically: max(256, 8*n_dims) instead of fixed 100
- Reduce gamma from 0.25 to 0.15 when trials < 50 for tighter exploitation
- Wire model_name through PSO/TPE paths for per-trial Prometheus metrics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The foxhunt-runtime image runs as non-root user 'foxhunt' and uses
/bin/sh (dash), not bash. Two issues:
1. &>/dev/null is bash-only — use >/dev/null 2>&1 for POSIX sh
2. Fallback download to /tmp (writable), not /usr/local/bin (root-only)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Re-export HardwareTimestamp through data crate instead of ml/risk
depending directly on trading_engine. Reduces coupling between
the ML pipeline and the trading engine.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The build-ci-image template used volumeClaimTemplates (PVC) which
weren't inherited when called via templateRef from ci-pipeline.
Restructured to single pod: init container (alpine/git) clones repo
into emptyDir, main container (kaniko) builds and pushes image from
the same volume. No PVC needed, works correctly via templateRef.
Tested: foxhunt-runtime image rebuilt successfully with kubectl.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
build-ci-image used volumeClaimTemplates (PVC) shared between a
git-clone pod and a kaniko-build pod in a DAG. When called via
templateRef from ci-pipeline, the VCT wasn't inherited, causing
"volume 'workspace' not found" errors.
Fixed by merging into a single pod: init container (git clone) +
main container (kaniko build) sharing an emptyDir volume. No PVC
needed, works correctly when called via templateRef.
Also removed stale compile-training-template.yaml reference from
kustomization (file doesn't exist, template is inline in ci-pipeline).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add kubectl v1.31.4 to foxhunt-runtime Dockerfile so deploy step
doesn't need to download it each run. Deploy step falls back to
curl download if kubectl not found (for current image version).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Deploy step: use foxhunt-runtime image (cached on platform node) +
curl kubectl, replacing bitnami/kubectl:1.31 which doesn't exist.
- Sensor: add podGC:OnPodCompletion and ttlStrategy to workflow spec
(not inherited from WorkflowTemplate via workflowTemplateRef).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two issues causing all 20 hyperopt trials to have identical f64::MAX
objective (TPE optimizer blind):
1. Val-loss plateau early stopping fired at epoch 5-6 of every 8-epoch
trial (plateau_window=5 too aggressive for short runs). Disabled
early_stopping_enabled for hyperopt; gradient-collapse patience
still active as safety net.
2. Penalty metrics used f64::MAX for gradient_norm/q_value_std which
produced ~3.6e+308 objective. Changed to 100.0 so TPE can still
differentiate between early-stopped trials by other metric fields.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three issues prevented services from being deployed after push:
1. No deploy step: pipeline compiled + uploaded binaries to MinIO but
never restarted service deployments. Added deploy-services step
(bitnami/kubectl) that runs rollout restart after compile-services.
2. compile-services couldn't schedule: targeted platform node (4 CPU,
6Gi allocatable) but requested 6Gi memory. Moved to ci-compile-cpu
(POP2-32C-128G) alongside compile-training — both fit simultaneously
(28/32 CPU, 48/128Gi).
3. No podGC: completed Argo pods held resources indefinitely, blocking
subsequent pipeline steps. Added podGC:OnPodCompletion.
Also: detect-changes now triggers service rebuild when CI pipeline
template or k8s service manifests change.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DaemonSet init container now:
- Detects GPU nodes (99-nvidia.toml in conf.d)
- Creates v3-compatible registry config drop-in
- Restarts containerd via chroot if config files changed
- Idempotent: skips restart if files already exist
Requires hostPID + privileged for chroot /proc/1/root.
On new nodes, first run creates files + restarts containerd
(kills pod), second run finds files unchanged (no restart).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Hyperopt: catch early stopping errors and return penalty metrics
instead of aborting entire run. Trials that stop early are scored
as poor (objective=-500) so optimizer avoids those configs.
- DaemonSet: detect GPU nodes (NVIDIA conf.d overlay) and create
v3-compatible registry config drop-in for containerd v2.1+.
Old grpc.v1.cri.registry path is silently ignored by containerd v2.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule nodes use nameserver 127.0.53.53 which resolves external DNS
but returns NXDOMAIN for .svc.cluster.local. Containerd uses node DNS
for image pulls, causing ErrImagePull for cluster-internal registry refs.
Deploy node-dns-fix DaemonSet (kube-system) that:
1. Points /etc/resolv.conf to kube-dns ClusterIP (10.32.0.10)
2. Creates containerd certs.d config for HTTP GitLab registry
3. Watches for Kapsule node reconciliation resets (every 30s)
4. Restores original resolv.conf on SIGTERM
Also bump compile-services memory: 6Gi→10Gi limit (was OOM-killed at 5.7GB).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add api_uptime_seconds gauge to API service (5s update interval),
matching the pattern used by trading/backtesting/ml-training services
- Remove obsolete Web Gateway Uptime and Active WebSocket Connections
panels from cockpit (web-gateway is not deployed)
- Add API Gateway Uptime (api_uptime_seconds) and API Auth Rate
(api_auth_requests_total) panels in their place
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes from deep investigation audit (LOW/MEDIUM priority):
1. CVaR penalty: hard cliff (0 or 10) → smooth ramp with gradient signal
for PSO. Formula: min(10, max(0, -cvar-0.05)*200).
2. Clip outliers leakage: data_loading.rs now computes clip bounds from
training portion only (first 80%), then applies to full series.
Log returns and windowed normalize are causal (no leakage).
3. Noisy sigma scheduler: hyperopt now matches conservative() defaults
(enabled, initial=0.8, final=0.4) so hyperopt-found params
generalize to train_best without scheduler mismatch.
4. evaluate_supervised.rs: NormStats fallback from test data (leakage)
replaced with bail! matching evaluate_baseline.rs behavior.
5. Doc comments: stale 27D references updated to 31D (4 locations).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4 bugs found by deep investigation agents:
1. HIGH: minimum_profit_factor (search dim 30) was never forwarded from
DQNHyperparameters to DQNConfig — trainer hardcoded 1.5, making the
entire dimension wasted. Added field to DQNHyperparameters, wired
through trainer.rs.
2. HIGH: Backtest EvaluationEngine used hardcoded $10K initial capital
while training used $35K (self.initial_capital). Returns/Sharpe were
3.5x distorted. Now uses self.initial_capital.
3. MEDIUM: calculate_hft_activity_score_wave10 multiplied already-100x
buy_pct/sell_pct by 100 again, making the diversity penalty threshold
(15%) unreachable (values were ~2700). Removed double multiplication.
4. MEDIUM: Sortino ratio returned 0.0 for all-positive returns (no
downside deviation), penalizing perfect strategies in the 40%-weighted
composite score. Now returns 100.0 (capped) when mean return > 0.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Join node metrics with node_uname_info to resolve instance IPs to
hostnames (e.g. scw-foxhunt-platform-694db...) in all 10 node-level
panels: CPU, Memory, Disk I/O, Network I/O, Load Average, Disk Usage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add __meta_kubernetes_pod_phase=Running filter to training-pods job.
Completed/failed Argo workflow step pods were showing as DOWN targets
because their containers are no longer running to serve /metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace api_gateway_ with api_ prefix to match actual metric exports (cockpit, observability)
- Change grpc_code!="OK" to grpc_code!="0" — gRPC uses numeric status codes (cockpit, trading)
- Update gRPC service filter to actual exported_service labels: trading-agent-service,
broker-gateway, backtesting-service, ml-training-service, data-acquisition-service (trading)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove duplicate annotated-pods and gitlab-annotated-pods scrape jobs
(all targets already covered by dedicated jobs)
- Drop Argo controller from foxhunt-services job (dedicated HTTPS job exists)
- Resolve datasource variable placeholders to actual UIDs in dashboard JSON
- Add Ingress rule to Prometheus netpol allowing Grafana on port 9090
Targets: 49 → 18 (zero duplicates)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Degenerate trials with <10/45 unique actions now short-circuit the objective
to a graduated penalty (8.0 for 1 action → 0 for 10+), ignoring composite
score entirely. Previously, phantom Sharpe=2317 drove objective to -369k,
trapping TPE in the low-temp region.
Combined with temp floor raise (0.1→0.5 from prior commit), this eliminates
both the supply (no low temps) and demand (no reward) for degenerate trials.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hyperopt TPE was stuck chasing phantom Sharpe=2317 from degenerate trials
with 2/45 unique actions (temp 0.15-0.20). Two fixes:
1. Raise eval_softmax_temp bounds from [0.1, 2.0] to [0.5, 2.0] — temps
below 0.3 consistently produce 1-3/45 actions regardless of model quality
2. Add diversity_penalty to objective: 50*(1 - unique/10) for <10/45 actions,
so degenerate trials score badly even if they accidentally reach the TPE
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Direct trading from the dashboard is not an option in our system.
Trading page is now read-only monitoring (positions, orders, order book).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite batch_hierarchical_softmax_actions to keep all sampling on GPU:
reshape [N,45]→[N,5,9], max-pool→[N,5] exposure Q, Gumbel-max over
exposures, parallel Gumbel-max over all sub-actions, gather chosen
exposure's sub-action. Only CPU transfer: N flat indices.
Add imagePullPolicy: Always to compile-services and compile-training
containers so rebuilt builder images are never stale-cached.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Flat Gumbel-max over 45 actions can collapse to a single exposure
bucket (e.g., 2/45 unique actions all in Long100) even with adequate
temperature, because order/urgency diversity doesn't produce trades.
Two-stage sampling: first Gumbel-max over 5 exposure levels (max Q
per level), then Gumbel-max over 9 order×urgency combos within the
chosen exposure. Guarantees exposure-level diversity proportional to
actual Q-value differences.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard's Config page was returning [unimplemented] because
trading-service had no ConfigService implementation. The API gateway's
ConfigServiceProxy forwards config RPCs to trading-service which now
handles them via direct SQL against the `configuration` table.
Implements GetConfiguration, UpdateConfiguration, DeleteConfiguration,
and ListCategories. Remaining 8 RPCs return UNIMPLEMENTED until needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Raise eval_softmax_temp lower bound from 0.01 to 0.1 (log scale) to
prevent near-greedy collapse in hyperopt walk-forward eval. Temps below
0.05 produced degenerate 1-trade trials even with softmax sampling.
- Mount MinIO CA cert in CI compile pods and append to system trust store
so sccache S3 backend can verify MinIO's self-signed TLS certificate.
- Add openssh-client to ci-builder-cpu Dockerfile (missing, broke git
clone over SSH).
- Bump MinIO memory limits from 512Mi to 2Gi (OOMKilled under load).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
grpc_pass only handles native gRPC (HTTP/2 frames). The web dashboard
uses gRPC-web via Connect protocol (HTTP/1.1 POST requests). Switch
api.fxhnt.ai nginx block to proxy_pass so tonic_web::GrpcWebLayer on
the API service can handle the protocol conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TypeScript 5.9 erasableSyntaxOnly rejects enum declarations in .ts
files (they emit runtime code). Switch protoc-gen-es from target=ts
to target=js+dts which generates .js runtime + .d.ts type declarations.
Also fix type mismatches in useApi.ts (OrderSide/OrderType enum types,
StartBacktestRequest field mapping) and MLDashboard signal defaulting.
Add web-dashboard network policy for MinIO egress + proxy ingress.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
containerd on Kapsule nodes uses host DNS which can't resolve
.svc.cluster.local names. Switch all image references from
gitlab-registry.foxhunt.svc.cluster.local:5000 to localhost:30500
(NodePort on the GitLab registry). Also make training runtime image
configurable via TRAINING_RUNTIME_IMAGE env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- nginx pod serves Vite-built static assets from MinIO S3
- initContainer fetches dist/ from s3://foxhunt-binaries/web-dashboard/
- SPA routing via try_files, aggressive caching for hashed assets
- dashboard.fxhnt.ai proxied to web-dashboard:80 via tailscale nginx
- CI pipeline: build-web-dashboard task (npm ci + build + rclone upload)
- detect-changes: needs-dashboard output for web-dashboard/ paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the combined grafana+dashboard nginx block so dashboard.fxhnt.ai
proxies to web-gateway:3000 (the actual trading dashboard) instead of
Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
