Training data PVCs (ReadWriteOnce) don't work with ephemeral GPU nodes
that autoscale across availability zones. Replace with rclone download
from MinIO to /tmp at the start of each GPU step.
- Remove training-data-pvc parameter and volume mount
- Add rclone download from s3:foxhunt-training-data/futures-baseline/
to /tmp/futures-baseline/ in hyperopt, train-best, and evaluate steps
- Add MINIO_ACCESS_KEY/SECRET_KEY env vars to all GPU steps
- Add cuda-compute-cap parameter (default "90")
- Change data-dir default to /tmp/futures-baseline
- 36 .dbn.zst files (53MB total) uploaded to MinIO bucket
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add device_pool to DQN/PPO hyperopt trainers for round-robin GPU assignment
per trial. Binary detects all CUDA devices, scales VRAM budget by GPU count.
Single-GPU: no behavior change (pool of 1).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename ci-training → ci-training-l40s, add H100/H100x2/H100-SXM/H100-SXM-8
pool resources (all autoscaling 0→1). Remove dev pool (DevPod on platform).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
build-ci-image templateRef needs registry-auth volume defined in parent
workflow. Secret was also named wrong (scw-registry-credentials →
scw-registry) and needed key projection (.dockerconfigjson → config.json).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
detect-changes was falling back to Cargo.toml (rebuild-all) because jq
is not installed in foxhunt-runtime:latest. Replaced with grep+sed
extraction that needs no external dependencies. Also added jq to the
Dockerfile for future use.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
T=0.1 was too conservative for degenerate configs — trials with uniform
Q-values still collapsed to 1 unique action. Making temperature a
hyperopt parameter lets TPE learn the optimal exploration-exploitation
balance per config. Range [0.01, 2.0] log-scale.
Also sets training-workflow default gpu-pool to ci-training-h100.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Registers foxhunt-production agent with CI and user access
for in-cluster kubectl from GitLab pipelines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two bugs fixed:
1. jq parsed webhook as plain array but payload is {body:{commits:[...]}} —
fell back to Cargo.toml catch-all, triggering full recompile every push.
2. Single needs-compile flag gave no granularity — now split into
needs-services and needs-training with path-based detection.
crates/ml/ only → training + services (ml-training-service depends on ml)
services/ only → services only, skip training (saves ~5.5min CUDA compile)
infra/ only → skip both compiles entirely
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DQN hyperopt walk-forward eval collapsed to 1 trade when Q-values were
nearly uniform (greedy argmax always picked the same action). Replace
batch_greedy_actions with batch_softmax_actions using the Gumbel-max
trick (argmax(Q/T + Gumbel(0,1))) — fully GPU-resident, no CPU
softmax/sampling. Temperature 0.1: nearly greedy when Q-values are
separated, diverse when uniform. Logs unique_actions/45 per backtest.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The sensor passes GitLab commit data containing single quotes in
commit messages (e.g. Merge branch 'feature-name'). When embedded
directly in the shell script via Argo template substitution, this
breaks the shell syntax. Pass as a container env var instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When using templateRef, Argo only borrows the template definition — not
spec-level volumes or volumeClaimTemplates from the source WorkflowTemplate.
Add workspace VCT, sccache PVC, and git-ssh-key secret at the ci-pipeline
spec level so compile-training's sub-templates can access them.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace legacy Buy/Sell/Hold collapse with exposure-aware evaluation.
Long100→Long50 now generates a partial-close trade instead of being
a no-op. Combined with graduated trade penalty, PSO now gets gradient
signal across the entire 26D search space.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add calculate_trade_insufficiency_penalty() that produces monotonically
decreasing penalties for low trade counts (10.0 at 0 trades, 0.0 at 100+),
giving PSO gradient signal across the degenerate plateau where all trials
produce the same 1.25 objective.
Wire the penalty into extract_objective() with short-circuit for <10 trades
(skip composite score since it's all zeros anyway).
Include 4 unit tests verifying zero/one/graduated/no-plateau properties.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add factored evaluation support to EvaluationEngine with continuous
exposure tracking (-1.0 to +1.0), partial position changes, and
order-type-aware transaction costs for the 45-action factored space.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace simulated login in fxt CLI with real AuthServiceClient gRPC call
- Add auth proto to fxt build.rs compile list and lib.rs proto module
- Add api.access permission to JWT claims issued by AuthGrpcService
- Remove orphaned ci-sensor.yaml (replaced by cluster-applied version)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
argo-workflow SA needs workflows/create and workflowtemplates/get
permissions to submit CI pipelines from sensor triggers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unnecessary case map — service names already match Cargo package names
- Add sensor pod serviceAccountName: argo-workflow (RBAC fix)
- Add sensor config to events/ for version tracking
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace compile-api-gateway + compile-web-gateway with single compile-api task
- Update detect-changes script: services/api/ instead of services/api_gateway/
- Remove web-gateway from change detection and compile targets
- Fix templateRef parameter resolution (inputs.parameters instead of workflow.parameters)
- Add SSH key setup and cluster-internal GitLab host for compile-service
- Use ci-builder image (has ssh) instead of ci-builder-cpu for service compilation
- Add compile-training and build-ci-image templates to kustomization
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- JWT issuer now foxhunt-api across all 16 files (services, tests, config, docker-compose)
- Remove serde alias api_gateway_url from FxtConfig (no backwards compat)
- Remove api_gateway CLI alias from e2e orchestrator
- All services must deploy simultaneously for JWT validation to match
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend (Rust):
- Create proto/auth.proto with Login and RefreshToken RPCs
- Implement AuthGrpcService in services/api with JWT issuance
- Register as unauthenticated service in main.rs (pre-auth)
- Update JWT issuer from foxhunt-api-gateway to foxhunt-api
Dashboard (TypeScript):
- Install @connectrpc/connect-web + @bufbuild/protobuf
- Generate TypeScript proto clients via buf (src/gen/)
- Create grpc.ts transport with JWT interceptor
- Rewrite all useApi hooks to typed gRPC calls
- Replace WebSocket with useGrpcStream server-streaming hooks
- Migrate all 6 pages + 6 components to grpc-web
- Delete api.ts, websocket.ts, useWebSocket.ts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copied from api_gateway, removed REST handlers (port 8080),
added tonic-web + CORS for grpc-web browser access.
Binary renamed: api-gateway → api
Changes:
- Package name: api-gateway → api
- Deleted src/handlers/ (REST ML endpoints on port 8080)
- Added tonic-web 0.13 + tower-http CORS layer
- Server::builder().accept_http1(true) for grpc-web
- CORS_ORIGINS env var (default http://localhost:5173)
- Metrics server on port 9091 (axum) preserved
- All 95 lib tests pass, 0 clippy warnings
- Added services/api to workspace members
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unify both gateways into a single gRPC-only `services/api/` with tonic-web
for browser access. Drop REST+WebSocket, keep full 6-layer auth.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When use_noisy_nets=true (the conservative() default), epsilon never
decayed from 1.0 because (1) the trainer skipped update_epsilon() and
(2) DQNAgentType::set_epsilon() was a no-op for RegimeConditional agents.
This caused ALL training actions to be random — Q-values were learned but
never used for action selection.
Fix: set stored epsilon to 0.0 at training start when noisy nets are on.
Exploration is provided by NoisyLinear weight perturbation + the separate
noisy_epsilon_floor (5% safety floor for 45-action spaces).
Also fixes:
- Zstd-compressed .dbn file detection via magic bytes (0x28B52FFD)
- Test data path resolution using ancestors().find() for workspace root
- Test assertions updated for epsilon < 0.01 with noisy nets
2698 lib tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces monolithic compile-services with 8 per-service compile jobs,
each with dependency-aware changes: filters. Deploy job only restarts
services whose binary actually changed. Also renames service crates
from snake_case to kebab-case to match k8s deployment names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add live training metrics monitor CLI command (streaming & one-shot) using
the monitoring gRPC service. Update DQN tests to match post-fix defaults:
IQN disabled, CQL alpha=0.1, v_min/v_max widened, 26D search space.
- train.rs: `fxt train monitor [--once] [--model X] [--interval N]`
- Rewrite gradient collapse test for BF16 mixed precision awareness
- Update inference test config to match trainer defaults (IQN off, CQL on)
- Update production smoke test for 26D parameter space
- Add dqn_action_collapse_fix_test.rs verifying all 6 root cause fixes
- Add planning docs for monitoring service and epoch financial metrics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Expand search space from 25D to 26D with cql_alpha (0.0-0.5)
- v_min bounds: (-3,-1) → (-15,-3), v_max: (1,3) → (3,15)
- Default use_qr_dqn: true → false (IQN disabled)
- Wire cql_alpha into DQNHyperparameters construction
- Update all 18 adapter tests for new dimensions and ranges
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hold penalty was divided by 1000 (producing ~0.00001), negligible vs
transaction costs (0.05-0.15%). Now uses hold_penalty_weight directly
from hyperopt (0.01-2.0 range).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Training cockpit sub-tabs (Live/History) with expandable per-epoch
detail panel showing financial metrics, RL diagnostics, and gradient health.
History tab fetches completed/failed runs from ListTrainingJobs RPC.
Add scrollable pod table to Pods cockpit and compact pods summary to
Overview bottom row. Fix K8s API egress in api-gateway NetworkPolicy
(monitoring-service→prometheus, apply existing K8s API CIDR rules).
Add 62 new unit tests across event_loop (51) and data_fetcher (11)
covering all StateUpdate variants, ring buffer eviction, epoch history
accumulation, pod scroll helpers, and navigation edge cases.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add nginx basic auth (htpasswd) to ci.fxhnt.ai proxy block and
switch Argo to server auth mode. Login: jgrusewski / Welcome01.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename DNS record from argo→ci and update nginx proxy server_name
to match. Applied via terragrunt and kubectl.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch Argo server from --auth-mode=server (no auth) to
--auth-mode=client (K8s ServiceAccount token). Created
jgrusewski-argo-token secret bound to argo-workflows-server SA
with admin RBAC for full workflow management.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hyperopt requires trials > n_initial. When running quick smoke tests
with few trials, auto-reduce n_initial to trials-1 so workflows don't
fail with "trials must be greater than n_initial".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU runners used runtime_class_name=nvidia for CUDA access but
didn't request nvidia.com/gpu as a K8s resource, allowing multiple
training pods to share a GPU without K8s awareness. Add pod_spec
patches to request GPU resources properly so K8s enforces mutual
exclusion between GitLab CI and Argo training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The default-deny-all NetworkPolicy blocks all egress for foxhunt-labeled
pods. Argo workflow pods need:
- Egress to MinIO (9000) for binary fetch and result upload
- Egress to K8s API (443/6443) for executor task result reporting
- Egress to Tempo (4317) for OTLP traces
- DNS already covered by allow-dns policy
Also fix PVC permission denied by setting fsGroup: 1000 to match the
foxhunt user (UID 1000) in the runtime images.
Smoke test fetch-binary step: Succeeded.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Helm values (controller + server on platform node, MinIO artifact repo)
- WorkflowTemplate: parameterized 5-step DAG (fetch→hyperopt→train→eval→upload)
- Nginx proxy for argo.fxhnt.ai → Argo Server :2746
- DNS A record for argo.fxhnt.ai
- MinIO bucket foxhunt-training-results for Argo artifacts
- Kustomization for kubectl apply -k
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace monolithic deploy job with per-service deploy jobs that fire
as soon as each compile finishes. No more waiting for all 8 services
+ training to compile before any deployment starts.
Structure:
- .deploy-base: shared kubeconfig setup
- .deploy-service-base: upload binary + apply manifest + rollout restart
- deploy-<service> (×8): needs compile-<service>, same rules via YAML anchors
- deploy-training: uploads training binaries after compile-training
- deploy-web-dashboard: uploads web assets after build-web-dashboard
- deploy-infra: migrations, monitoring, dashboards, proxy (no compile dep)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When all 8 per-service compile jobs fire simultaneously (e.g. on
Cargo.toml change), the single POP2-32C-128G node is fully loaded.
Training compile pod can't schedule until service jobs finish.
KUBERNETES_POLL_TIMEOUT=1200 gives it time to wait.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Full GitLab CI → Argo migration: Argo Events webhook trigger,
per-service compile WorkflowTemplates, selective deploy, test
workspace, training compile, and IaC templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
