The remoteWrite block had placeholder values (COCKPIT_METRICS_PUSH_URL,
COCKPIT_PUSH_TOKEN) that were never replaced, causing Prometheus to
spam "Failed to send batch" warnings every minute. Cockpit was replaced
by self-hosted Grafana+Prometheus+Loki+Tempo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ibapi handshake: retry once after 1s on first failure (IB Gateway
needs time to release stale client_id slots after disconnection)
- test_load_nonexistent_config → test_load_config_succeeds: remove
default-value assertions that fail when ~/.foxhunt/config.toml exists
(defaults already tested by test_serde_defaults and test_default_config)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Match the actual K8s broker-gateway service port discovered during
live validation of fxt broker check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The IB Gateway JVM uses non-blocking I/O on port 4002, which causes
EAGAIN errors with ibapi's blocking Client::connect(). The gnzsnz
image provides socat on port 4004 as a stable proxy that bridges
blocking clients to the non-blocking gateway socket.
Port 4004 is the correct client-facing port for paper mode (4003 for
live). The K8s readiness probe stays on 4002 since TCP probes work
fine with non-blocking sockets.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Readiness/liveness probes were checking port 4004 (socat), which
always listens even when the gateway is stuck at login — masking
failures. Now probes check port 4002 (TWS API), which only opens
after successful IBKR authentication.
Deploy strategy changed to Recreate because IBKR allows only one
session per account. RollingUpdate starts the new pod before killing
the old one, causing both to fight over the session in a crash loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The gnzsnz/ib-gateway image renders its IBC config.ini from a template
using envsubst. The ExistingSessionDetectedAction field was rendering
as empty, causing IBC to show a GUI dialog when IBKR detected a
competing session — which nobody could dismiss in a headless pod.
Setting primaryoverride tells IBC to automatically take over any
existing session, which is the correct behavior for a K8s deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Broker variant to Commands enum with BrokerArgs (flatten),
route to execute_broker_command in match block (no JWT required),
exit(1) on check failure. Clone config.api_gateway_url to avoid
partial move. Add two CLI parsing tests for broker check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy broker_gateway.proto from broker_gateway_service into fxt/proto/
and wire it into build.rs compile_protos + lib.rs proto module so the
CLI can call BrokerGatewayService RPCs (health check, account state,
session status).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds IbkrConfig (host/port/client_id with defaults for paper trading)
and BrokerConfig (gateway_url + ibkr nested) to the TLI config file.
Both structs derive serde defaults so existing config files remain
backward-compatible.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-layer IBKR connectivity validation: direct ibapi handshake +
broker-gateway gRPC health check. Pure-client architecture using
ibapi as optional dep on fxt (not trading_engine).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule autoscaler already handles GPU node scale-down
(10min idle timeout). This CronJob only printed a message
and served no purpose.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Grafana now stores users, preferences, and state in our existing
PostgreSQL instance (dedicated 'grafana' database). This eliminates
the 2Gi PVC dependency — all persistent state lives in Postgres,
dashboards in ConfigMaps, datasources in Helm values.
Also adds Grafana to Postgres NetworkPolicy ingress allowlist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Grafana PVC was lost during cleanup (orphaned PV deletion).
Dashboards survived (ConfigMap-based), but user accounts and
datasources were lost.
This commit captures all Helm values previously set via --set:
- Sidecar config with foldersFromFilesStructure for folder organization
- Loki + Tempo datasources alongside Prometheus
- Platform node toleration for gitlab taint
Also fixed CI/CD folder annotation (slash caused nested directory).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ci-rl pool was permanently disabled (enable_ci_rl_pool=false), all
training consolidated on ci-training (L40S). foxhunt-binaries S3
bucket was never created — binaries now deploy via shared PVC.
Also: deleted orphaned infra/live/production/ci-runner/ directory,
cleaned up stale Grafana ReplicaSets, recreated missing grafana PVC,
deleted orphaned Released PV, synced runner Helm configmap.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI executor pods (label foxhunt-ci) were selected by allow-monitoring-scrape
(which uses matchExpressions In [foxhunt, foxhunt-ci]) making them
policy-controlled, but allow-dns only selected foxhunt pods via matchLabels.
This blocked DNS resolution → git clone failure in deploy jobs.
- Update allow-dns podSelector to matchExpressions In [foxhunt, foxhunt-ci]
- Add ci-egress policy granting broad egress for ephemeral CI pods
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DQN and PPO trainers now resolve hidden_dim_base from GPU VRAM when no
explicit override is given, so L4/L40S/H100 GPUs use proportionally
larger networks instead of being stuck at RTX 3050 Ti defaults (256).
- Add resolve_hidden_dim_base() tiered lookup (256/512/768/1024 by VRAM)
- Add network_param_count() for accurate model size estimation
- DQN: pre-compute hidden dims, use real param count for batch sizing
- PPO: add hidden_dim_base field, VRAM resolution in PpoTrainer::new()
- PPO hyperopt: raise hidden_dim_base ceiling from 2048 to 4096
- TFT hyperopt: expand hidden_sizes from [128,256,512] to 5 tiers
- Update stale estimates (DQN 50K→200K, PPO 100K→400K params)
- Fix pre-existing clippy lints in prefetch.rs and ppo.rs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
11-task step-by-step plan: create platform pool, move monitoring
(stateless), migrate databases (PVC migration with backups), scale
services pool to 1 node. Includes verification and rollback procedures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Isolate platform infra (postgres, redis, minio, monitoring) from foxhunt
app services on separate DEV1-L nodes. Same EUR 61/mo total cost, better
blast radius containment and burst headroom for trading services.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Infrastructure Cockpit:
- GPU gauges → side-by-side timeseries (GPU Utilization + GPU VRAM with total line)
- Added Memory by Service timeseries next to existing CPU by Service
- Cluster resource gauges shrunk to h=4 for tighter layout
Training Cockpit:
- GPU Utilization/Memory gauges → side-by-side timeseries matching infra cockpit
- GPU Temperature/Power widened from w=6 to w=12 in paired row
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6 stat panels (Ensemble Agreement, Prediction Confidence, Model Accuracy,
Ensemble Sharpe, Realized PnL, Unrealized PnL) showed red when ML metrics
don't exist yet. Added special null mapping → "Inactive" in dark-blue.
Panels will still show red/yellow/green when the trading system is active
and emitting metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The "Active CI Jobs" stat panel used sum(gitlab_ci_active_jobs_sum) which
is a histogram accumulator that only grows (was showing 982 red). Changed
to sum(increase(pipeline_processing_events_total[1h])) which shows actual
pipeline activity in the last hour. Threshold changed to blue=idle,
green=active.
Also fixed Grafana provisioning:
- Removed overlapping named providers (cockpits/operations/infrastructure/cicd)
that duplicated sidecarProvider, causing UID conflicts and write lockout
- Enabled foldersFromFilesStructure on sidecarProvider
- Unlabeled grafana-dashboards-foxhunt ConfigMap (duplicate of cockpits+operations)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Change training job tags from kapsule to kapsule-rl so they are
picked up exclusively by the RL runner (GPU-dedicated), not the
general runner (CPU compile pool).
- Add OTEL_EXPORTER_OTLP_ENDPOINT to both .train-rl-base and
.train-validate-base so training binaries export OTLP traces
to Tempo.
- Update allow-monitoring-scrape NetworkPolicy to include
foxhunt-ci pods so CI training pods can reach Tempo:4317.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI build pods have label app.kubernetes.io/part-of=foxhunt-ci,
not foxhunt. Add foxhunt-ci to the MinIO NetworkPolicy ingress
values so compile/upload jobs can reach MinIO S3 endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds 12 stat panels in a new row between Service Status and Resources:
- Infrastructure: postgres, redis, minio, questdb (UP/DOWN)
- GitLab: webservice status + link to gitlab-services dashboard
- Network: tailscale proxy
- Monitoring: grafana, loki, tempo, kube-state-metrics (UP/DOWN)
- Counts: CI runners (active), collectors (promtail + node-exporter)
Uses w=4 panels (6 per row) for better readability on small screens.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
