- Argo workflow archive via PostgreSQL — persistent logs after pod GC
- Allow Argo controller to reach PostgreSQL via network policy
- Allow Argo server to reach MinIO for S3 log archival
- Replace foxhunt-runtime with lightweight images (alpine:3.21, curlimages/curl)
in orchestration steps: fetch-binary, upload-results, detect-changes, gpu-warmup
- Use ci-builder for training steps (hyperopt, train-best, evaluate) — nvcc required
- Fix --hyperopt-results → --hyperopt-params flag in train-best step
- Remove invalid podGCGracePeriod from Argo Helm values
- Archive RBAC, kustomization updates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stalwart v0.15.5 requires %{file:path}% macros to read cert/key
from disk — bare paths were treated as literal content, causing
"No certificates found" errors.
Added HTTPS egress to NetworkPolicy so Stalwart can download the
webadmin bundle and spam/geo databases from GitHub/CDN on boot.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Stalwart all-in-one mail server: SMTP (cluster-internal relay for
Mattermost), IMAP + HTTPS admin via Tailscale proxy. RocksDB storage,
self-signed TLS (stalwart-tls secret). No public MX, no outbound SMTP.
SCW TEM relay noted in NetworkPolicy comments for future use.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Mattermost Team Edition deployment (PVC, Service, NetworkPolicy),
DNS records (chat.fxhnt.ai, mail.fxhnt.ai), Tailscale nginx proxy
server block with WebSocket upgrade support, and Mattermost egress
rules to all 5 Argo workflow NetworkPolicies for exit handler webhooks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add QuestDB ILP sink for training metrics, update Prometheus scrape
configs, and fix network policies for monitoring stack connectivity.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Service pods' initContainers need egress to gitlab-webservice-default:8181
to fetch release binaries from the Generic Package Registry. Without this,
curl hangs for 2+ minutes then times out (default-deny-all blocks it).
Covers all app.kubernetes.io/part-of=foxhunt pods in one policy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Expand training-data-pvc from 10Gi → 200Gi (OHLCV + MBP-10 + headroom)
- Add data-sync-job: rclone sync from MinIO → PVC (delta-only, fast repeats)
- Add sync-training-data init container to training job template
(auto-syncs data from MinIO before each training run)
- Add training-job + data-sync-job NetworkPolicies (MinIO, Tempo, Pushgateway)
- Add app.kubernetes.io/part-of: foxhunt to training pod template
- download_baseline: add --parallel N flag for concurrent Databento downloads
- download_baseline: delete local files after MinIO upload (saves scratch space)
- Bump download job scratch volume to 100Gi
Architecture: Databento → MinIO (source of truth) → PVC (local cache).
First sync pulls everything; subsequent runs only sync deltas (seconds).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The download job pod was missing app.kubernetes.io/part-of: foxhunt,
so the default-deny-all egress policy blocked it and MinIO's ingress
policy rejected it. Also removed hostname pinning (was a misdiagnosis).
Added data-download-job NetworkPolicy allowing egress to MinIO:9000
and external HTTPS:443 (Databento API).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove duplicate annotated-pods and gitlab-annotated-pods scrape jobs
(all targets already covered by dedicated jobs)
- Drop Argo controller from foxhunt-services job (dedicated HTTPS job exists)
- Resolve datasource variable placeholders to actual UIDs in dashboard JSON
- Add Ingress rule to Prometheus netpol allowing Grafana on port 9090
Targets: 49 → 18 (zero duplicates)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TypeScript 5.9 erasableSyntaxOnly rejects enum declarations in .ts
files (they emit runtime code). Switch protoc-gen-es from target=ts
to target=js+dts which generates .js runtime + .d.ts type declarations.
Also fix type mismatches in useApi.ts (OrderSide/OrderType enum types,
StartBacktestRequest field mapping) and MLDashboard signal defaulting.
Add web-dashboard network policy for MinIO egress + proxy ingress.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Training cockpit sub-tabs (Live/History) with expandable per-epoch
detail panel showing financial metrics, RL diagnostics, and gradient health.
History tab fetches completed/failed runs from ListTrainingJobs RPC.
Add scrollable pod table to Pods cockpit and compact pods summary to
Overview bottom row. Fix K8s API egress in api-gateway NetworkPolicy
(monitoring-service→prometheus, apply existing K8s API CIDR rules).
Add 62 new unit tests across event_loop (51) and data_fetcher (11)
covering all StateUpdate variants, ring buffer eviction, epoch history
accumulation, pod scroll helpers, and navigation edge cases.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allow api-gateway to reach the Kubernetes API server for pod listing
(Pods cockpit). Two CIDR blocks are needed because Cilium applies
NetworkPolicy after DNAT:
- 10.32.0.0/12:443 — kubernetes.default.svc ClusterIP
- 172.16.0.0/16:6443 — real node IP after Cilium DNAT
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add query_cluster_resources() for real CPU/RAM from Prometheus
- Wire into build_response() replacing hardcoded zeros
- Fix NetworkPolicy: add ports 8080/9100/9400 for KSM/node-exporter/dcgm
- Add CiliumNetworkPolicy for hostNetwork pod scraping
- Add SubscribeClusterPods stub to unblock proto addition from Task 5
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove from workspace members, delete K8s deployment, remove nginx routing.
Training metrics and system health now served directly by API Gateway.
Changes:
- Cargo.toml: remove services/monitoring_service from workspace members
- services/monitoring_service/: deleted entirely (6 files)
- infra/k8s/services/monitoring-service.yaml: deleted
- infra/k8s/network-policies/monitoring-service.yaml: deleted
- infra/k8s/network-policies/api-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/network-policies/web-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/services/api-gateway.yaml: remove stale MONITORING_SERVICE_URL env var
- infra/k8s/gitlab/tailscale-proxy.yaml: redirect monitor.fxhnt.ai to api-gateway:50051
- .gitlab-ci.yml: remove monitoring_service from compile, copy, and deploy loops
- services/trading_service/src/services/monitoring.rs: update stale comments to
reference api_gateway (not monitoring_service) as training metrics host
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add file_descriptor_set_path for all proto compilations (enables gRPC reflection)
- Add BackendHealthState and ServiceHealthEntry for structured health tracking
- Export health types from grpc module
- Fix web-gateway network policy port
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Swap the stale trading_service monitoring proto (10 RPCs, never
implemented) for the real monitoring_service proto (2 RPCs:
GetLiveTrainingMetrics, StreamTrainingMetrics). Add a dedicated
MonitoringServiceProxy that forwards directly to monitoring-service
on port 50057, so `fxt watch` works end-to-end through the gateway.
- build.rs: compile monitoring_service/proto/monitoring.proto with server+client
- MonitoringServiceProxy: new zero-copy proxy (unary + streaming)
- TradingServiceProxy: remove monitoring_client, stub 6 stale methods
- main.rs: MONITORING_SERVICE_URL env, optional proxy with graceful degradation
- Network policies: api-gateway↔monitoring-service egress/ingress on 50057
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DQN/PPO trainers now record epoch, loss, val_loss, batch/s every epoch
- CI training jobs get Prometheus scrape annotations via runner overrides
- Allow Prometheus (foxhunt namespace) to scrape foxhunt-ci pods
- Skip no-model metrics in monitoring service session grouping
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add DNS record, nginx gRPC proxy block, network policy for Tailscale
ingress, and auto-detect fxhnt.ai in fxt monitor URL derivation.
Show GPU telemetry even when no training sessions are active.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
monitoring-service needs egress to Prometheus (port 80/9090) and ingress
from web-gateway (port 50057). web-gateway needs egress to monitoring-service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Grafana now stores users, preferences, and state in our existing
PostgreSQL instance (dedicated 'grafana' database). This eliminates
the 2Gi PVC dependency — all persistent state lives in Postgres,
dashboards in ConfigMaps, datasources in Helm values.
Also adds Grafana to Postgres NetworkPolicy ingress allowlist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI executor pods (label foxhunt-ci) were selected by allow-monitoring-scrape
(which uses matchExpressions In [foxhunt, foxhunt-ci]) making them
policy-controlled, but allow-dns only selected foxhunt pods via matchLabels.
This blocked DNS resolution → git clone failure in deploy jobs.
- Update allow-dns podSelector to matchExpressions In [foxhunt, foxhunt-ci]
- Add ci-egress policy granting broad egress for ephemeral CI pods
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Change training job tags from kapsule to kapsule-rl so they are
picked up exclusively by the RL runner (GPU-dedicated), not the
general runner (CPU compile pool).
- Add OTEL_EXPORTER_OTLP_ENDPOINT to both .train-rl-base and
.train-validate-base so training binaries export OTLP traces
to Tempo.
- Update allow-monitoring-scrape NetworkPolicy to include
foxhunt-ci pods so CI training pods can reach Tempo:4317.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI build pods have label app.kubernetes.io/part-of=foxhunt-ci,
not foxhunt. Add foxhunt-ci to the MinIO NetworkPolicy ingress
values so compile/upload jobs can reach MinIO S3 endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The minio-init-buckets job pod has label app.kubernetes.io/name: minio-init
which doesn't match part-of: foxhunt (adding part-of would trigger
default-deny egress). Add explicit ingress allowlist entry for minio-init
pods on the minio NetworkPolicy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All per-service NetworkPolicies used bare 'app: postgres/redis/minio'
selectors but infrastructure pods use 'app.kubernetes.io/name'. This
blocked all egress to databases, causing initContainer hangs and
service connectivity failures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab webservice/sidekiq share the foxhunt namespace but don't have
part-of=foxhunt label. Add release=gitlab selector to postgres and
redis NetworkPolicies so GitLab can reach its database.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Caught during pre-apply review:
- Scope default-deny/allow-dns/allow-monitoring to pods with
app.kubernetes.io/part-of=foxhunt (was podSelector:{} which would
block gitlab, grafana, loki sharing the namespace)
- Fix infrastructure.yaml: use app.kubernetes.io/name instead of
bare app label for postgres/redis/minio/questdb podSelectors
- Add app.kubernetes.io/part-of=foxhunt to all pod templates so
default-deny and infrastructure ingress rules match correctly
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Least-privilege ingress/egress for all 8 services + IB gateway +
infrastructure pods (postgres, redis, minio, questdb). Each policy
restricts which pods can communicate on which ports, limiting lateral
movement in the cluster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Establishes zero-trust network baseline for the foxhunt namespace:
- default-deny-all: blocks all ingress/egress unless explicitly allowed
- allow-dns: permits CoreDNS (kube-system) egress on UDP/TCP 53
- allow-monitoring-scrape: allows Prometheus ingress on metrics ports
(9091-9098) and Tempo OTLP gRPC egress on 4317
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
