Add Training cockpit sub-tabs (Live/History) with expandable per-epoch
detail panel showing financial metrics, RL diagnostics, and gradient health.
History tab fetches completed/failed runs from ListTrainingJobs RPC.
Add scrollable pod table to Pods cockpit and compact pods summary to
Overview bottom row. Fix K8s API egress in api-gateway NetworkPolicy
(monitoring-service→prometheus, apply existing K8s API CIDR rules).
Add 62 new unit tests across event_loop (51) and data_fetcher (11)
covering all StateUpdate variants, ring buffer eviction, epoch history
accumulation, pod scroll helpers, and navigation edge cases.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add nginx basic auth (htpasswd) to ci.fxhnt.ai proxy block and
switch Argo to server auth mode. Login: jgrusewski / Welcome01.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename DNS record from argo→ci and update nginx proxy server_name
to match. Applied via terragrunt and kubectl.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch Argo server from --auth-mode=server (no auth) to
--auth-mode=client (K8s ServiceAccount token). Created
jgrusewski-argo-token secret bound to argo-workflows-server SA
with admin RBAC for full workflow management.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hyperopt requires trials > n_initial. When running quick smoke tests
with few trials, auto-reduce n_initial to trials-1 so workflows don't
fail with "trials must be greater than n_initial".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU runners used runtime_class_name=nvidia for CUDA access but
didn't request nvidia.com/gpu as a K8s resource, allowing multiple
training pods to share a GPU without K8s awareness. Add pod_spec
patches to request GPU resources properly so K8s enforces mutual
exclusion between GitLab CI and Argo training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The default-deny-all NetworkPolicy blocks all egress for foxhunt-labeled
pods. Argo workflow pods need:
- Egress to MinIO (9000) for binary fetch and result upload
- Egress to K8s API (443/6443) for executor task result reporting
- Egress to Tempo (4317) for OTLP traces
- DNS already covered by allow-dns policy
Also fix PVC permission denied by setting fsGroup: 1000 to match the
foxhunt user (UID 1000) in the runtime images.
Smoke test fetch-binary step: Succeeded.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace monolithic deploy job with per-service deploy jobs that fire
as soon as each compile finishes. No more waiting for all 8 services
+ training to compile before any deployment starts.
Structure:
- .deploy-base: shared kubeconfig setup
- .deploy-service-base: upload binary + apply manifest + rollout restart
- deploy-<service> (×8): needs compile-<service>, same rules via YAML anchors
- deploy-training: uploads training binaries after compile-training
- deploy-web-dashboard: uploads web assets after build-web-dashboard
- deploy-infra: migrations, monitoring, dashboards, proxy (no compile dep)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When all 8 per-service compile jobs fire simultaneously (e.g. on
Cargo.toml change), the single POP2-32C-128G node is fully loaded.
Training compile pod can't schedule until service jobs finish.
KUBERNETES_POLL_TIMEOUT=1200 gives it time to wait.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full GitLab CI → Argo migration: Argo Events webhook trigger,
per-service compile WorkflowTemplates, selective deploy, test
workspace, training compile, and IaC templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each service gets its own compile job with dependency-aware changes:
filters so only affected services recompile on push. Deploy job
selectively restarts only services whose binary was compiled.
- Add .compile-service-base template (4 CPUs, 8Gi per job)
- 8 jobs: api-gateway, trading-service, ml-training-service,
backtesting-service, trading-agent-service, broker-gateway,
data-acquisition-service, web-gateway
- Deploy needs: all 8 compile jobs (optional: true, artifacts: true)
- Rollout restart loop checks build-out/services/ for actual binaries
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename 7 service binaries from snake_case to kebab-case to match
K8s deployment names. Update Cargo.toml package/bin names, K8s
manifest S3 paths and commands, and cross-crate dependency keys.
- api_gateway → api-gateway
- trading_service → trading-service
- broker_gateway_service → broker-gateway
- ml_training_service → ml-training-service
- backtesting_service → backtesting-service
- trading_agent_service → trading-agent-service
- data_acquisition_service → data-acquisition-service
broker-gateway gets an explicit [lib] name = "broker_gateway_service"
since its new package name maps to broker_gateway (not the original
broker_gateway_service used in source code). All other services map
correctly with Rust's automatic hyphen-to-underscore conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete stub rainbow_agent.rs (fake select_action, hardcoded train loss)
and promote rainbow_agent_impl.rs to rainbow_agent.rs. Unify on the
real 8-field RainbowAgentMetrics from rainbow_config.rs, replacing the
4-field stub version (epsilon→exploration_rate, average_loss→current_loss).
5 files changed: -667/+447 lines, 2698+32 tests pass, 0 clippy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change PercentileScaler, RewardNormalizer, CompositeReward, and
PPORewardShaper public APIs from f64 to f32 — matching what callers
actually pass. Internal EMA/percentile math stays f64 for precision.
Keep data_loader SMA/EMA/MACD accumulators in f64 throughout (was
f64→f32→f64 per step), cast to f32 only at output boundary.
Remove dead _transition computation in rainbow_agent_impl and unused
bigdecimal deps from broker_gateway_service and trading_service.
2704 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training job before_script now fetches from MinIO S3 when CI
artifacts are unavailable. Fixes hyperopt/train jobs failing with
"command not found" when triggered without compile-training artifacts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CPU request 14000m→10000m (burstable to 15500m), CARGO_BUILD_JOBS 14→10.
Three compile jobs now total 22000m on 31800m node (was 30000m).
Per-job resource_group prevents cross-pipeline contention.
interruptible: true lets newer pipelines cancel stale ones.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
On Ampere+ GPUs the training dtype is BF16. Network forward passes
with BF16 inputs return BF16 tensors, but all loss-path arithmetic
(rewards, dones, gamma, PER weights, Huber constants, atoms, clip
epsilon) was created as F32 from Tensor::from_vec. Candle forbids
mixed-dtype binary ops, causing "dtype mismatch in {mul,sub,add}"
on every training step.
DQN fixes (7 mismatch sites):
- Cast current_q_values to F32 immediately after forward pass
- Cast all target network outputs (standard, dueling, C51, IQN) to F32
- Keep rewards/dones/gamma/weights/atoms/half/delta as F32 (remove
.to_dtype(training_dtype) casts — now use DType::F32 sentinel)
- Entropy regularization and CQL paths now match (F32 + F32)
PPO fixes (3 mismatch sites):
- Use DType::F32 for clip epsilon one_tensor (was training_dtype BF16)
- Cast critic.forward() output to F32 in both MLP and LSTM value loss
- Cast target_returns to F32 for symlog/normalization path
Infra: revert runner-h100 from SXM2 (zero Scaleway quota) back to
ci-training-h100 PCIe pool.
2704 ml lib tests pass, 260 PPO tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.
Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
kube-rs Client::try_default() uses rustls for in-cluster TLS but no
CryptoProvider was installed, causing a panic on first pod list call.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BUG #41 kept forward pass in F32 for autograd, but the target-side
tensors (reward, gamma, done, next_q) were cast to BF16 via `dtype`.
The `state_action_values.sub(&target_q_values)` then hit F32-vs-BF16
mismatch on Ampere+ GPUs, causing every training step to fail silently.
Fix: `.to_dtype(state_action_values.dtype())` on the detached target.
Safe because target is detached (no autograd graph to break).
Also: H100 runner → SXM2 pool, GPU availability checker script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 32 MCP tool handlers now make real gRPC calls to the API Gateway
instead of returning stub responses. Covers: service monitoring,
training management, trading operations, ML inference, risk metrics,
broker status, data acquisition, cluster resources, agent control,
config management, and hyperopt tuning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
end_to_end_tests.rs and error_handling_tests.rs imported non-existent
modules (fxt::client, fxt::prelude, etc.) from a prior architecture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- StreamSystemStatus now queries real Prometheus data (was all zeros)
- StreamAlerts returns honest unimplemented (was sending empty events)
- Training epoch shows "N" not "N/0" (max_epochs not in proto)
- GPU power shows "300W" not "300W / 0W" when limit unavailable
- Services cockpit shows error reason when service is DOWN
- System status polling detects staleness after 3 consecutive failures
- Training cockpit header shows active K8s job count
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 30 MCP tool handlers now return structured JSON error with
isError=true instead of fake "stub: would ..." placeholder responses.
Tool registrations preserved so list_tools still works.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
config_service.proto (package foxhunt.config, 4 RPCs) was a subset of
config.proto (package config, 12 RPCs). Migrated api_gateway to implement
ConfigService from config.proto directly. Removed dead config_service()
method from fxt client.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
57 references updated across source, tests, benches, and config.
Proto package name foxhunt.tli kept for wire compatibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace three occurrences of String::new() used as "fetch all" filter
values in gRPC request construction with a named FILTER_ALL constant,
making the intent explicit.
The api_gateway monitoring handler was reviewed but its Prometheus URL
appears only once in tests, so no constant was extracted (single-use).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use em-dash (—) consistently instead of mix of - and — for "no data"
placeholders across all cockpits (data, trading, risk, services, pods)
- Collapse pipeline status section to single "not connected" message
instead of showing 4 misleading rows of dashes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Same class of bug as the DQN fix (4c88498b): network outputs were
being cast to F32 mid-pipeline, defeating tensor-core acceleration
on H100/L40S. Now the full training loop stays in training_dtype()
(BF16 on CUDA Ampere+, F32 on CPU), with F32 casts only at scalar
extraction boundaries (to_scalar, to_vec1).
Files fixed:
- ppo.rs: Actor/Critic forward, act_with_log_prob, compute_losses,
update_mlp, LSTM recurrent loop, predict method
- lstm_networks.rs: removed F32 output casts from both networks
- continuous_ppo.rs: one_tensor + scalar extractions
- hidden_state_manager.rs: zeros/ones use training_dtype()
- flow_policy/mod.rs: log_det accumulators + dummy log_std
- ensemble/adapters/ppo.rs + dqn.rs: F32 cast at extraction
2704 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove premature F32 output casts from network forward passes
(Sequential, NetworkLayers, DistributionalDueling) that were negating
tensor-core acceleration. Instead, cast F32 input tensors (rewards,
dones, gamma, weights, atoms) to training_dtype() at the boundary
and only escape to F32 at scalar extraction points (loss value,
TD errors, logging vectors).
465 DQN tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add ServiceAccount, Role, and RoleBinding so the api-gateway pod can
list/get/watch pods and pod metrics in the foxhunt namespace. Also sets
serviceAccountName on the Deployment to bind the new SA.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allow api-gateway to reach the Kubernetes API server for pod listing
(Pods cockpit). Two CIDR blocks are needed because Cilium applies
NetworkPolicy after DNAT:
- 10.32.0.0/12:443 — kubernetes.default.svc ClusterIP
- 172.16.0.0/16:6443 — real node IP after Cilium DNAT
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add kube 3.0 + k8s-openapi 0.27 dependencies and create pods_handler.rs
that queries the Kubernetes API for pod status. Replace the UNIMPLEMENTED
stub in monitoring_handler.rs with a real streaming implementation that
polls pods every N seconds and maps them to PodInfo proto messages.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
