- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename 7 service binaries from snake_case to kebab-case to match
K8s deployment names. Update Cargo.toml package/bin names, K8s
manifest S3 paths and commands, and cross-crate dependency keys.
- api_gateway → api-gateway
- trading_service → trading-service
- broker_gateway_service → broker-gateway
- ml_training_service → ml-training-service
- backtesting_service → backtesting-service
- trading_agent_service → trading-agent-service
- data_acquisition_service → data-acquisition-service
broker-gateway gets an explicit [lib] name = "broker_gateway_service"
since its new package name maps to broker_gateway (not the original
broker_gateway_service used in source code). All other services map
correctly with Rust's automatic hyphen-to-underscore conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
On Ampere+ GPUs the training dtype is BF16. Network forward passes
with BF16 inputs return BF16 tensors, but all loss-path arithmetic
(rewards, dones, gamma, PER weights, Huber constants, atoms, clip
epsilon) was created as F32 from Tensor::from_vec. Candle forbids
mixed-dtype binary ops, causing "dtype mismatch in {mul,sub,add}"
on every training step.
DQN fixes (7 mismatch sites):
- Cast current_q_values to F32 immediately after forward pass
- Cast all target network outputs (standard, dueling, C51, IQN) to F32
- Keep rewards/dones/gamma/weights/atoms/half/delta as F32 (remove
.to_dtype(training_dtype) casts — now use DType::F32 sentinel)
- Entropy regularization and CQL paths now match (F32 + F32)
PPO fixes (3 mismatch sites):
- Use DType::F32 for clip epsilon one_tensor (was training_dtype BF16)
- Cast critic.forward() output to F32 in both MLP and LSTM value loss
- Cast target_returns to F32 for symlog/normalization path
Infra: revert runner-h100 from SXM2 (zero Scaleway quota) back to
ci-training-h100 PCIe pool.
2704 ml lib tests pass, 260 PPO tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.
Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BUG #41 kept forward pass in F32 for autograd, but the target-side
tensors (reward, gamma, done, next_q) were cast to BF16 via `dtype`.
The `state_action_values.sub(&target_q_values)` then hit F32-vs-BF16
mismatch on Ampere+ GPUs, causing every training step to fail silently.
Fix: `.to_dtype(state_action_values.dtype())` on the detached target.
Safe because target is detached (no autograd graph to break).
Also: H100 runner → SXM2 pool, GPU availability checker script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add ServiceAccount, Role, and RoleBinding so the api-gateway pod can
list/get/watch pods and pod metrics in the foxhunt namespace. Also sets
serviceAccountName on the Deployment to bind the new SA.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allow api-gateway to reach the Kubernetes API server for pod listing
(Pods cockpit). Two CIDR blocks are needed because Cilium applies
NetworkPolicy after DNAT:
- 10.32.0.0/12:443 — kubernetes.default.svc ClusterIP
- 172.16.0.0/16:6443 — real node IP after Cilium DNAT
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add query_cluster_resources() for real CPU/RAM from Prometheus
- Wire into build_response() replacing hardcoded zeros
- Fix NetworkPolicy: add ports 8080/9100/9400 for KSM/node-exporter/dcgm
- Add CiliumNetworkPolicy for hostNetwork pod scraping
- Add SubscribeClusterPods stub to unblock proto addition from Task 5
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pool ci-training-h100x2 (H100-2-80G, 160GB VRAM total).
MultiGpuConfig::detect() in ml crate auto-enables data parallelism.
Tags: kapsule,h100x2. Autoscale 0→1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Clear separation: l40s tag → L40S pool, h100 tag → H100 pool.
Updated runner-rl-values.yaml and CI base templates to match.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create ci-training-h100 runner (runner-h100-values.yaml)
- Route hyperopt-dqn and hyperopt-ppo to H100 via kapsule,h100 tags
- H100 is ~2x faster and cheaper per run vs L40S for hyperopt workloads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add BROKER_GATEWAY_SERVICE_URL, DATA_ACQUISITION_SERVICE_URL, and
ML_SERVICE_URL env vars so the gateway can connect to all backend services.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove from workspace members, delete K8s deployment, remove nginx routing.
Training metrics and system health now served directly by API Gateway.
Changes:
- Cargo.toml: remove services/monitoring_service from workspace members
- services/monitoring_service/: deleted entirely (6 files)
- infra/k8s/services/monitoring-service.yaml: deleted
- infra/k8s/network-policies/monitoring-service.yaml: deleted
- infra/k8s/network-policies/api-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/network-policies/web-gateway.yaml: remove egress rule to monitoring-service
- infra/k8s/services/api-gateway.yaml: remove stale MONITORING_SERVICE_URL env var
- infra/k8s/gitlab/tailscale-proxy.yaml: redirect monitor.fxhnt.ai to api-gateway:50051
- .gitlab-ci.yml: remove monitoring_service from compile, copy, and deploy loops
- services/trading_service/src/services/monitoring.rs: update stale comments to
reference api_gateway (not monitoring_service) as training metrics host
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add file_descriptor_set_path for all proto compilations (enables gRPC reflection)
- Add BackendHealthState and ServiceHealthEntry for structured health tracking
- Export health types from grpc module
- Fix web-gateway network policy port
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Swap the stale trading_service monitoring proto (10 RPCs, never
implemented) for the real monitoring_service proto (2 RPCs:
GetLiveTrainingMetrics, StreamTrainingMetrics). Add a dedicated
MonitoringServiceProxy that forwards directly to monitoring-service
on port 50057, so `fxt watch` works end-to-end through the gateway.
- build.rs: compile monitoring_service/proto/monitoring.proto with server+client
- MonitoringServiceProxy: new zero-copy proxy (unary + streaming)
- TradingServiceProxy: remove monitoring_client, stub 6 stale methods
- main.rs: MONITORING_SERVICE_URL env, optional proxy with graceful degradation
- Network policies: api-gateway↔monitoring-service egress/ingress on 50057
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DQN/PPO trainers now record epoch, loss, val_loss, batch/s every epoch
- CI training jobs get Prometheus scrape annotations via runner overrides
- Allow Prometheus (foxhunt namespace) to scrape foxhunt-ci pods
- Skip no-model metrics in monitoring service session grouping
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add DNS record, nginx gRPC proxy block, network policy for Tailscale
ingress, and auto-detect fxhnt.ai in fxt monitor URL derivation.
Show GPU telemetry even when no training sessions are active.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
monitoring-service needs egress to Prometheus (port 80/9090) and ingress
from web-gateway (port 50057). web-gateway needs egress to monitoring-service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The S3→PVC migration had several issues causing deploy failures:
- Stale binary-writer pod from previous failed deploys blocked new ones
(terminated pods can't be updated via kubectl apply)
- Services were applied BEFORE binaries written to PVC, so first deploy
or empty PVC caused pods to crash (binary not found)
- GPU overlay files in services/ dir were auto-applied by kubectl apply,
referencing nonexistent S3/minio secrets and PVCs
- Training job template mounted foxhunt-binaries PVC but training runs
on ci-training node — RWO PVC is bound to foxhunt node
Fixes:
- Reorder deploy: write binaries → apply manifests → rollout restart
- Clean up stale writer pods before creating new ones
- Move GPU overlays to gpu-overlays/ (manual apply only), update to PVC
- Add training-binaries PVC for GPU node, best-effort population
- Training job template uses initContainer to copy from training PVC
- set -e for critical path, set +e for optional training binary copy
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runner #2 already has tags [kapsule, gpu] but CI jobs still used
kapsule-rl. Also removes stale minio-ca-cert volume from runner values.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
MinIO pod was removed but compile-training still uploaded to it via rclone,
causing 403 Forbidden failures. Now:
- compile-training: saves binaries as CI artifacts (3 day TTL)
- deploy stage: copies training binaries to foxhunt-binaries PVC via binary-writer
- CI training jobs: use compile-training CI artifacts directly
- job-template.yaml: mounts foxhunt-binaries PVC (no initContainer)
- Removed MinIO deploy from deploy stage (no longer needed)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The remoteWrite block had placeholder values (COCKPIT_METRICS_PUSH_URL,
COCKPIT_PUSH_TOKEN) that were never replaced, causing Prometheus to
spam "Failed to send batch" warnings every minute. Cockpit was replaced
by self-hosted Grafana+Prometheus+Loki+Tempo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Readiness/liveness probes were checking port 4004 (socat), which
always listens even when the gateway is stuck at login — masking
failures. Now probes check port 4002 (TWS API), which only opens
after successful IBKR authentication.
Deploy strategy changed to Recreate because IBKR allows only one
session per account. RollingUpdate starts the new pod before killing
the old one, causing both to fight over the session in a crash loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The gnzsnz/ib-gateway image renders its IBC config.ini from a template
using envsubst. The ExistingSessionDetectedAction field was rendering
as empty, causing IBC to show a GUI dialog when IBKR detected a
competing session — which nobody could dismiss in a headless pod.
Setting primaryoverride tells IBC to automatically take over any
existing session, which is the correct behavior for a K8s deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule autoscaler already handles GPU node scale-down
(10min idle timeout). This CronJob only printed a message
and served no purpose.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Grafana now stores users, preferences, and state in our existing
PostgreSQL instance (dedicated 'grafana' database). This eliminates
the 2Gi PVC dependency — all persistent state lives in Postgres,
dashboards in ConfigMaps, datasources in Helm values.
Also adds Grafana to Postgres NetworkPolicy ingress allowlist.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Grafana PVC was lost during cleanup (orphaned PV deletion).
Dashboards survived (ConfigMap-based), but user accounts and
datasources were lost.
This commit captures all Helm values previously set via --set:
- Sidecar config with foldersFromFilesStructure for folder organization
- Loki + Tempo datasources alongside Prometheus
- Platform node toleration for gitlab taint
Also fixed CI/CD folder annotation (slash caused nested directory).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ci-rl pool was permanently disabled (enable_ci_rl_pool=false), all
training consolidated on ci-training (L40S). foxhunt-binaries S3
bucket was never created — binaries now deploy via shared PVC.
Also: deleted orphaned infra/live/production/ci-runner/ directory,
cleaned up stale Grafana ReplicaSets, recreated missing grafana PVC,
deleted orphaned Released PV, synced runner Helm configmap.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI executor pods (label foxhunt-ci) were selected by allow-monitoring-scrape
(which uses matchExpressions In [foxhunt, foxhunt-ci]) making them
policy-controlled, but allow-dns only selected foxhunt pods via matchLabels.
This blocked DNS resolution → git clone failure in deploy jobs.
- Update allow-dns podSelector to matchExpressions In [foxhunt, foxhunt-ci]
- Add ci-egress policy granting broad egress for ephemeral CI pods
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
