Backend (Rust):
- Create proto/auth.proto with Login and RefreshToken RPCs
- Implement AuthGrpcService in services/api with JWT issuance
- Register as unauthenticated service in main.rs (pre-auth)
- Update JWT issuer from foxhunt-api-gateway to foxhunt-api
Dashboard (TypeScript):
- Install @connectrpc/connect-web + @bufbuild/protobuf
- Generate TypeScript proto clients via buf (src/gen/)
- Create grpc.ts transport with JWT interceptor
- Rewrite all useApi hooks to typed gRPC calls
- Replace WebSocket with useGrpcStream server-streaming hooks
- Migrate all 6 pages + 6 components to grpc-web
- Delete api.ts, websocket.ts, useWebSocket.ts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copied from api_gateway, removed REST handlers (port 8080),
added tonic-web + CORS for grpc-web browser access.
Binary renamed: api-gateway → api
Changes:
- Package name: api-gateway → api
- Deleted src/handlers/ (REST ML endpoints on port 8080)
- Added tonic-web 0.13 + tower-http CORS layer
- Server::builder().accept_http1(true) for grpc-web
- CORS_ORIGINS env var (default http://localhost:5173)
- Metrics server on port 9091 (axum) preserved
- All 95 lib tests pass, 0 clippy warnings
- Added services/api to workspace members
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unify both gateways into a single gRPC-only `services/api/` with tonic-web
for browser access. Drop REST+WebSocket, keep full 6-layer auth.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When use_noisy_nets=true (the conservative() default), epsilon never
decayed from 1.0 because (1) the trainer skipped update_epsilon() and
(2) DQNAgentType::set_epsilon() was a no-op for RegimeConditional agents.
This caused ALL training actions to be random — Q-values were learned but
never used for action selection.
Fix: set stored epsilon to 0.0 at training start when noisy nets are on.
Exploration is provided by NoisyLinear weight perturbation + the separate
noisy_epsilon_floor (5% safety floor for 45-action spaces).
Also fixes:
- Zstd-compressed .dbn file detection via magic bytes (0x28B52FFD)
- Test data path resolution using ancestors().find() for workspace root
- Test assertions updated for epsilon < 0.01 with noisy nets
2698 lib tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces monolithic compile-services with 8 per-service compile jobs,
each with dependency-aware changes: filters. Deploy job only restarts
services whose binary actually changed. Also renames service crates
from snake_case to kebab-case to match k8s deployment names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add live training metrics monitor CLI command (streaming & one-shot) using
the monitoring gRPC service. Update DQN tests to match post-fix defaults:
IQN disabled, CQL alpha=0.1, v_min/v_max widened, 26D search space.
- train.rs: `fxt train monitor [--once] [--model X] [--interval N]`
- Rewrite gradient collapse test for BF16 mixed precision awareness
- Update inference test config to match trainer defaults (IQN off, CQL on)
- Update production smoke test for 26D parameter space
- Add dqn_action_collapse_fix_test.rs verifying all 6 root cause fixes
- Add planning docs for monitoring service and epoch financial metrics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Expand search space from 25D to 26D with cql_alpha (0.0-0.5)
- v_min bounds: (-3,-1) → (-15,-3), v_max: (1,3) → (3,15)
- Default use_qr_dqn: true → false (IQN disabled)
- Wire cql_alpha into DQNHyperparameters construction
- Update all 18 adapter tests for new dimensions and ranges
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hold penalty was divided by 1000 (producing ~0.00001), negligible vs
transaction costs (0.05-0.15%). Now uses hold_penalty_weight directly
from hyperopt (0.01-2.0 range).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Training cockpit sub-tabs (Live/History) with expandable per-epoch
detail panel showing financial metrics, RL diagnostics, and gradient health.
History tab fetches completed/failed runs from ListTrainingJobs RPC.
Add scrollable pod table to Pods cockpit and compact pods summary to
Overview bottom row. Fix K8s API egress in api-gateway NetworkPolicy
(monitoring-service→prometheus, apply existing K8s API CIDR rules).
Add 62 new unit tests across event_loop (51) and data_fetcher (11)
covering all StateUpdate variants, ring buffer eviction, epoch history
accumulation, pod scroll helpers, and navigation edge cases.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add nginx basic auth (htpasswd) to ci.fxhnt.ai proxy block and
switch Argo to server auth mode. Login: jgrusewski / Welcome01.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename DNS record from argo→ci and update nginx proxy server_name
to match. Applied via terragrunt and kubectl.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch Argo server from --auth-mode=server (no auth) to
--auth-mode=client (K8s ServiceAccount token). Created
jgrusewski-argo-token secret bound to argo-workflows-server SA
with admin RBAC for full workflow management.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hyperopt requires trials > n_initial. When running quick smoke tests
with few trials, auto-reduce n_initial to trials-1 so workflows don't
fail with "trials must be greater than n_initial".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU runners used runtime_class_name=nvidia for CUDA access but
didn't request nvidia.com/gpu as a K8s resource, allowing multiple
training pods to share a GPU without K8s awareness. Add pod_spec
patches to request GPU resources properly so K8s enforces mutual
exclusion between GitLab CI and Argo training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The default-deny-all NetworkPolicy blocks all egress for foxhunt-labeled
pods. Argo workflow pods need:
- Egress to MinIO (9000) for binary fetch and result upload
- Egress to K8s API (443/6443) for executor task result reporting
- Egress to Tempo (4317) for OTLP traces
- DNS already covered by allow-dns policy
Also fix PVC permission denied by setting fsGroup: 1000 to match the
foxhunt user (UID 1000) in the runtime images.
Smoke test fetch-binary step: Succeeded.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace monolithic deploy job with per-service deploy jobs that fire
as soon as each compile finishes. No more waiting for all 8 services
+ training to compile before any deployment starts.
Structure:
- .deploy-base: shared kubeconfig setup
- .deploy-service-base: upload binary + apply manifest + rollout restart
- deploy-<service> (×8): needs compile-<service>, same rules via YAML anchors
- deploy-training: uploads training binaries after compile-training
- deploy-web-dashboard: uploads web assets after build-web-dashboard
- deploy-infra: migrations, monitoring, dashboards, proxy (no compile dep)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When all 8 per-service compile jobs fire simultaneously (e.g. on
Cargo.toml change), the single POP2-32C-128G node is fully loaded.
Training compile pod can't schedule until service jobs finish.
KUBERNETES_POLL_TIMEOUT=1200 gives it time to wait.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full GitLab CI → Argo migration: Argo Events webhook trigger,
per-service compile WorkflowTemplates, selective deploy, test
workspace, training compile, and IaC templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each service gets its own compile job with dependency-aware changes:
filters so only affected services recompile on push. Deploy job
selectively restarts only services whose binary was compiled.
- Add .compile-service-base template (4 CPUs, 8Gi per job)
- 8 jobs: api-gateway, trading-service, ml-training-service,
backtesting-service, trading-agent-service, broker-gateway,
data-acquisition-service, web-gateway
- Deploy needs: all 8 compile jobs (optional: true, artifacts: true)
- Rollout restart loop checks build-out/services/ for actual binaries
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename 7 service binaries from snake_case to kebab-case to match
K8s deployment names. Update Cargo.toml package/bin names, K8s
manifest S3 paths and commands, and cross-crate dependency keys.
- api_gateway → api-gateway
- trading_service → trading-service
- broker_gateway_service → broker-gateway
- ml_training_service → ml-training-service
- backtesting_service → backtesting-service
- trading_agent_service → trading-agent-service
- data_acquisition_service → data-acquisition-service
broker-gateway gets an explicit [lib] name = "broker_gateway_service"
since its new package name maps to broker_gateway (not the original
broker_gateway_service used in source code). All other services map
correctly with Rust's automatic hyphen-to-underscore conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete stub rainbow_agent.rs (fake select_action, hardcoded train loss)
and promote rainbow_agent_impl.rs to rainbow_agent.rs. Unify on the
real 8-field RainbowAgentMetrics from rainbow_config.rs, replacing the
4-field stub version (epsilon→exploration_rate, average_loss→current_loss).
5 files changed: -667/+447 lines, 2698+32 tests pass, 0 clippy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change PercentileScaler, RewardNormalizer, CompositeReward, and
PPORewardShaper public APIs from f64 to f32 — matching what callers
actually pass. Internal EMA/percentile math stays f64 for precision.
Keep data_loader SMA/EMA/MACD accumulators in f64 throughout (was
f64→f32→f64 per step), cast to f32 only at output boundary.
Remove dead _transition computation in rainbow_agent_impl and unused
bigdecimal deps from broker_gateway_service and trading_service.
2704 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training job before_script now fetches from MinIO S3 when CI
artifacts are unavailable. Fixes hyperopt/train jobs failing with
"command not found" when triggered without compile-training artifacts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CPU request 14000m→10000m (burstable to 15500m), CARGO_BUILD_JOBS 14→10.
Three compile jobs now total 22000m on 31800m node (was 30000m).
Per-job resource_group prevents cross-pipeline contention.
interruptible: true lets newer pipelines cancel stale ones.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
On Ampere+ GPUs the training dtype is BF16. Network forward passes
with BF16 inputs return BF16 tensors, but all loss-path arithmetic
(rewards, dones, gamma, PER weights, Huber constants, atoms, clip
epsilon) was created as F32 from Tensor::from_vec. Candle forbids
mixed-dtype binary ops, causing "dtype mismatch in {mul,sub,add}"
on every training step.
DQN fixes (7 mismatch sites):
- Cast current_q_values to F32 immediately after forward pass
- Cast all target network outputs (standard, dueling, C51, IQN) to F32
- Keep rewards/dones/gamma/weights/atoms/half/delta as F32 (remove
.to_dtype(training_dtype) casts — now use DType::F32 sentinel)
- Entropy regularization and CQL paths now match (F32 + F32)
PPO fixes (3 mismatch sites):
- Use DType::F32 for clip epsilon one_tensor (was training_dtype BF16)
- Cast critic.forward() output to F32 in both MLP and LSTM value loss
- Cast target_returns to F32 for symlog/normalization path
Infra: revert runner-h100 from SXM2 (zero Scaleway quota) back to
ci-training-h100 PCIe pool.
2704 ml lib tests pass, 260 PPO tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.
Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
kube-rs Client::try_default() uses rustls for in-cluster TLS but no
CryptoProvider was installed, causing a panic on first pod list call.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BUG #41 kept forward pass in F32 for autograd, but the target-side
tensors (reward, gamma, done, next_q) were cast to BF16 via `dtype`.
The `state_action_values.sub(&target_q_values)` then hit F32-vs-BF16
mismatch on Ampere+ GPUs, causing every training step to fail silently.
Fix: `.to_dtype(state_action_values.dtype())` on the detached target.
Safe because target is detached (no autograd graph to break).
Also: H100 runner → SXM2 pool, GPU availability checker script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 32 MCP tool handlers now make real gRPC calls to the API Gateway
instead of returning stub responses. Covers: service monitoring,
training management, trading operations, ML inference, risk metrics,
broker status, data acquisition, cluster resources, agent control,
config management, and hyperopt tuning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
end_to_end_tests.rs and error_handling_tests.rs imported non-existent
modules (fxt::client, fxt::prelude, etc.) from a prior architecture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- StreamSystemStatus now queries real Prometheus data (was all zeros)
- StreamAlerts returns honest unimplemented (was sending empty events)
- Training epoch shows "N" not "N/0" (max_epochs not in proto)
- GPU power shows "300W" not "300W / 0W" when limit unavailable
- Services cockpit shows error reason when service is DOWN
- System status polling detects staleness after 3 consecutive failures
- Training cockpit header shows active K8s job count
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 30 MCP tool handlers now return structured JSON error with
isError=true instead of fake "stub: would ..." placeholder responses.
Tool registrations preserved so list_tools still works.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
