11-task step-by-step plan: create platform pool, move monitoring
(stateless), migrate databases (PVC migration with backups), scale
services pool to 1 node. Includes verification and rollback procedures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Isolate platform infra (postgres, redis, minio, monitoring) from foxhunt
app services on separate DEV1-L nodes. Same EUR 61/mo total cost, better
blast radius containment and burst headroom for trading services.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Infrastructure Cockpit:
- GPU gauges → side-by-side timeseries (GPU Utilization + GPU VRAM with total line)
- Added Memory by Service timeseries next to existing CPU by Service
- Cluster resource gauges shrunk to h=4 for tighter layout
Training Cockpit:
- GPU Utilization/Memory gauges → side-by-side timeseries matching infra cockpit
- GPU Temperature/Power widened from w=6 to w=12 in paired row
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6 stat panels (Ensemble Agreement, Prediction Confidence, Model Accuracy,
Ensemble Sharpe, Realized PnL, Unrealized PnL) showed red when ML metrics
don't exist yet. Added special null mapping → "Inactive" in dark-blue.
Panels will still show red/yellow/green when the trading system is active
and emitting metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The "Active CI Jobs" stat panel used sum(gitlab_ci_active_jobs_sum) which
is a histogram accumulator that only grows (was showing 982 red). Changed
to sum(increase(pipeline_processing_events_total[1h])) which shows actual
pipeline activity in the last hour. Threshold changed to blue=idle,
green=active.
Also fixed Grafana provisioning:
- Removed overlapping named providers (cockpits/operations/infrastructure/cicd)
that duplicated sidecarProvider, causing UID conflicts and write lockout
- Enabled foldersFromFilesStructure on sidecarProvider
- Unlabeled grafana-dashboards-foxhunt ConfigMap (duplicate of cockpits+operations)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Change training job tags from kapsule to kapsule-rl so they are
picked up exclusively by the RL runner (GPU-dedicated), not the
general runner (CPU compile pool).
- Add OTEL_EXPORTER_OTLP_ENDPOINT to both .train-rl-base and
.train-validate-base so training binaries export OTLP traces
to Tempo.
- Update allow-monitoring-scrape NetworkPolicy to include
foxhunt-ci pods so CI training pods can reach Tempo:4317.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI build pods have label app.kubernetes.io/part-of=foxhunt-ci,
not foxhunt. Add foxhunt-ci to the MinIO NetworkPolicy ingress
values so compile/upload jobs can reach MinIO S3 endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds 12 stat panels in a new row between Service Status and Resources:
- Infrastructure: postgres, redis, minio, questdb (UP/DOWN)
- GitLab: webservice status + link to gitlab-services dashboard
- Network: tailscale proxy
- Monitoring: grafana, loki, tempo, kube-state-metrics (UP/DOWN)
- Counts: CI runners (active), collectors (promtail + node-exporter)
Uses w=4 panels (6 per row) for better readability on small screens.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New row shows GitLab, CI Runners, Monitoring, Collectors, Tailscale,
and Platform Total stat panels, plus a pod status table, CPU/memory
timeseries, and container restart chart for all non-foxhunt pods.
Also fix minio-init-buckets job: remove part-of:foxhunt label (triggers
default-deny egress block) and rely on minio NetworkPolicy ingress
allowlist for minio-init pods instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The minio-init-buckets job pod has label app.kubernetes.io/name: minio-init
which doesn't match part-of: foxhunt (adding part-of would trigger
default-deny egress). Add explicit ingress allowlist entry for minio-init
pods on the minio NetworkPolicy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All per-service NetworkPolicies used bare 'app: postgres/redis/minio'
selectors but infrastructure pods use 'app.kubernetes.io/name'. This
blocked all egress to databases, causing initContainer hangs and
service connectivity failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pod-level runAsNonRoot:true conflicts with initContainer runAsUser:0.
Kubernetes rejects the container. Add explicit override so the
initContainer can still run as root to fetch binaries from S3.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab webservice/sidekiq share the foxhunt namespace but don't have
part-of=foxhunt label. Add release=gitlab selector to postgres and
redis NetworkPolicies so GitLab can reach its database.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Caught during pre-apply review:
- Scope default-deny/allow-dns/allow-monitoring to pods with
app.kubernetes.io/part-of=foxhunt (was podSelector:{} which would
block gitlab, grafana, loki sharing the namespace)
- Fix infrastructure.yaml: use app.kubernetes.io/name instead of
bare app label for postgres/redis/minio/questdb podSelectors
- Add app.kubernetes.io/part-of=foxhunt to all pod templates so
default-deny and infrastructure ingress rules match correctly
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous fix tried appending to /etc/ssl/certs/ca-certificates.crt
which fails on non-root containers (Permission denied). Instead,
copy system CAs to /tmp/ca-bundle.crt, append MinIO CA, and set
SSL_CERT_FILE for Go's crypto/tls.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RCLONE_CA_CERT env var does not apply to rclone's S3 backend — Go's
crypto/tls reads from the system CA bundle. Append the MinIO CA cert
to /etc/ssl/certs/ca-certificates.crt before rclone commands. Fixes
compile-services, compile-training, web-dashboard, write-manifest,
and both training base templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
db-credentials, jwt-secret, redis-credentials, s3-credentials.
Each service only references the secrets it needs.
Values are REPLACE_IN_DEPLOY markers — real values
injected by scripts/deploy-secrets.sh.
Also updated references in postgres.yaml, migrate.yaml,
and postgres-init.yaml to use db-credentials.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Least-privilege ingress/egress for all 8 services + IB gateway +
infrastructure pods (postgres, redis, minio, questdb). Each policy
restricts which pods can communicate on which ports, limiting lateral
movement in the cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Establishes zero-trust network baseline for the foxhunt namespace:
- default-deny-all: blocks all ingress/egress unless explicitly allowed
- allow-dns: permits CoreDNS (kube-system) egress on UDP/TCP 53
- allow-monitoring-scrape: allows Prometheus ingress on metrics ports
(9091-9098) and Tempo OTLP gRPC egress on 4317
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Minimal hardening plan for production deployment: NetworkPolicy
(default-deny + explicit allow), SecurityContext on all pods,
secret scoping (split monolithic foxhunt-secrets), and Trivy
image scanning in CI pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Emit set_eval_metrics (directional_accuracy, sharpe, profit_factor, return)
per fold for supervised model evaluation. Start metrics server on :9094.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Emit set_eval_metrics (win_rate, sharpe, profit_factor, return) per fold
for DQN/PPO evaluation. Start metrics server on :9094. Feeds training
cockpit Grafana dashboard eval panels.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace tracing_subscriber::fmt() with init_observability() which adds
JSON structured logging + optional OTLP export to Tempo. When
OTEL_EXPORTER_OTLP_ENDPOINT env var is unset, falls back to fmt-only.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Emit set_epoch, set_epoch_loss, set_validation_loss, set_iteration_seconds
after each DQN/PPO fold completes. Feeds training cockpit Grafana dashboard.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training binaries export OTLP traces to Tempo when this env var is set.
Traces appear in foxhunt-traces Grafana dashboard.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove unnecessary async from init_observability -- body was fully sync.
Change otlp_endpoint from &str to Option<&str> -- when None, OTLP layer
is skipped (fmt-only mode). Update all 8 service callers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Standalone .rs file with no Cargo.toml, not in workspace.
Cannot compile or run. 1,280 lines of dead code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add //! module-level doc comment to training_uploader/src/main.rs.
The other 6 crates (backtesting, ctrader-openapi, data, ml, risk,
trading_engine) already had doc comments and were skipped.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create 3 missing service READMEs (api_gateway, data_acquisition_service,
trading_agent_service). Create bin/fxt/README.md. Create
testing/service-integration/README.md. Update existing service and testing
READMEs to standard template. Delete 4 subdirectory READMEs from
testing/integration/ and testing/e2e/.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove stale test reports, quick-start guides, benchmark analyses,
profiling reports, and tool artifacts from across the workspace.
Keeps only root README.md per crate/service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete 92 scattered .md files, 137 plan docs, create/update all READMEs,
add lib.rs doc comments, clean stale worktrees.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DCGM exporter + dashboards already exist. Actual work: 3 tasks —
per-fold metrics in train_baseline_rl, OTLP tracing in all 6 binaries,
OTEL env var in K8s job template.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GPU metrics polling, per-epoch RL metrics, OTLP tracing in training
binaries, and three Grafana dashboards (training cockpit, traces,
infra cockpit trace row).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously train_baseline_rl.rs only passed commission (tx_cost_bps) to
DQN/PPO trainers, ignoring bid-ask spread slippage. Now computes per-fold
average spread via spread_cost_bps() (same as evaluate_baseline) and passes
total cost (commission + spread) to both trainers.
Removes #[allow(dead_code)] — function is now used by all 4 example binaries.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the liquidity heuristic in score_symbols with a real
GetEnsembleVote gRPC call to trading-service. Each symbol gets a
concurrent RPC; on failure the original liquidity*0.9+0.1 heuristic
is used as fallback. The ML client is lazily initialized via OnceCell
and the service URL is configurable via ML_SERVICE_URL env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
