Degenerate trials with <10/45 unique actions now short-circuit the objective
to a graduated penalty (8.0 for 1 action → 0 for 10+), ignoring composite
score entirely. Previously, phantom Sharpe=2317 drove objective to -369k,
trapping TPE in the low-temp region.
Combined with temp floor raise (0.1→0.5 from prior commit), this eliminates
both the supply (no low temps) and demand (no reward) for degenerate trials.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hyperopt TPE was stuck chasing phantom Sharpe=2317 from degenerate trials
with 2/45 unique actions (temp 0.15-0.20). Two fixes:
1. Raise eval_softmax_temp bounds from [0.1, 2.0] to [0.5, 2.0] — temps
below 0.3 consistently produce 1-3/45 actions regardless of model quality
2. Add diversity_penalty to objective: 50*(1 - unique/10) for <10/45 actions,
so degenerate trials score badly even if they accidentally reach the TPE
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Direct trading from the dashboard is not an option in our system.
Trading page is now read-only monitoring (positions, orders, order book).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite batch_hierarchical_softmax_actions to keep all sampling on GPU:
reshape [N,45]→[N,5,9], max-pool→[N,5] exposure Q, Gumbel-max over
exposures, parallel Gumbel-max over all sub-actions, gather chosen
exposure's sub-action. Only CPU transfer: N flat indices.
Add imagePullPolicy: Always to compile-services and compile-training
containers so rebuilt builder images are never stale-cached.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Flat Gumbel-max over 45 actions can collapse to a single exposure
bucket (e.g., 2/45 unique actions all in Long100) even with adequate
temperature, because order/urgency diversity doesn't produce trades.
Two-stage sampling: first Gumbel-max over 5 exposure levels (max Q
per level), then Gumbel-max over 9 order×urgency combos within the
chosen exposure. Guarantees exposure-level diversity proportional to
actual Q-value differences.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard's Config page was returning [unimplemented] because
trading-service had no ConfigService implementation. The API gateway's
ConfigServiceProxy forwards config RPCs to trading-service which now
handles them via direct SQL against the `configuration` table.
Implements GetConfiguration, UpdateConfiguration, DeleteConfiguration,
and ListCategories. Remaining 8 RPCs return UNIMPLEMENTED until needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Raise eval_softmax_temp lower bound from 0.01 to 0.1 (log scale) to
prevent near-greedy collapse in hyperopt walk-forward eval. Temps below
0.05 produced degenerate 1-trade trials even with softmax sampling.
- Mount MinIO CA cert in CI compile pods and append to system trust store
so sccache S3 backend can verify MinIO's self-signed TLS certificate.
- Add openssh-client to ci-builder-cpu Dockerfile (missing, broke git
clone over SSH).
- Bump MinIO memory limits from 512Mi to 2Gi (OOMKilled under load).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
grpc_pass only handles native gRPC (HTTP/2 frames). The web dashboard
uses gRPC-web via Connect protocol (HTTP/1.1 POST requests). Switch
api.fxhnt.ai nginx block to proxy_pass so tonic_web::GrpcWebLayer on
the API service can handle the protocol conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TypeScript 5.9 erasableSyntaxOnly rejects enum declarations in .ts
files (they emit runtime code). Switch protoc-gen-es from target=ts
to target=js+dts which generates .js runtime + .d.ts type declarations.
Also fix type mismatches in useApi.ts (OrderSide/OrderType enum types,
StartBacktestRequest field mapping) and MLDashboard signal defaulting.
Add web-dashboard network policy for MinIO egress + proxy ingress.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
containerd on Kapsule nodes uses host DNS which can't resolve
.svc.cluster.local names. Switch all image references from
gitlab-registry.foxhunt.svc.cluster.local:5000 to localhost:30500
(NodePort on the GitLab registry). Also make training runtime image
configurable via TRAINING_RUNTIME_IMAGE env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- nginx pod serves Vite-built static assets from MinIO S3
- initContainer fetches dist/ from s3://foxhunt-binaries/web-dashboard/
- SPA routing via try_files, aggressive caching for hashed assets
- dashboard.fxhnt.ai proxied to web-dashboard:80 via tailscale nginx
- CI pipeline: build-web-dashboard task (npm ci + build + rclone upload)
- detect-changes: needs-dashboard output for web-dashboard/ paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the combined grafana+dashboard nginx block so dashboard.fxhnt.ai
proxies to web-gateway:3000 (the actual trading dashboard) instead of
Grafana.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training data PVCs (ReadWriteOnce) don't work with ephemeral GPU nodes
that autoscale across availability zones. Replace with rclone download
from MinIO to /tmp at the start of each GPU step.
- Remove training-data-pvc parameter and volume mount
- Add rclone download from s3:foxhunt-training-data/futures-baseline/
to /tmp/futures-baseline/ in hyperopt, train-best, and evaluate steps
- Add MINIO_ACCESS_KEY/SECRET_KEY env vars to all GPU steps
- Add cuda-compute-cap parameter (default "90")
- Change data-dir default to /tmp/futures-baseline
- 36 .dbn.zst files (53MB total) uploaded to MinIO bucket
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add device_pool to DQN/PPO hyperopt trainers for round-robin GPU assignment
per trial. Binary detects all CUDA devices, scales VRAM budget by GPU count.
Single-GPU: no behavior change (pool of 1).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename ci-training → ci-training-l40s, add H100/H100x2/H100-SXM/H100-SXM-8
pool resources (all autoscaling 0→1). Remove dev pool (DevPod on platform).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
build-ci-image templateRef needs registry-auth volume defined in parent
workflow. Secret was also named wrong (scw-registry-credentials →
scw-registry) and needed key projection (.dockerconfigjson → config.json).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
detect-changes was falling back to Cargo.toml (rebuild-all) because jq
is not installed in foxhunt-runtime:latest. Replaced with grep+sed
extraction that needs no external dependencies. Also added jq to the
Dockerfile for future use.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
T=0.1 was too conservative for degenerate configs — trials with uniform
Q-values still collapsed to 1 unique action. Making temperature a
hyperopt parameter lets TPE learn the optimal exploration-exploitation
balance per config. Range [0.01, 2.0] log-scale.
Also sets training-workflow default gpu-pool to ci-training-h100.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Registers foxhunt-production agent with CI and user access
for in-cluster kubectl from GitLab pipelines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two bugs fixed:
1. jq parsed webhook as plain array but payload is {body:{commits:[...]}} —
fell back to Cargo.toml catch-all, triggering full recompile every push.
2. Single needs-compile flag gave no granularity — now split into
needs-services and needs-training with path-based detection.
crates/ml/ only → training + services (ml-training-service depends on ml)
services/ only → services only, skip training (saves ~5.5min CUDA compile)
infra/ only → skip both compiles entirely
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DQN hyperopt walk-forward eval collapsed to 1 trade when Q-values were
nearly uniform (greedy argmax always picked the same action). Replace
batch_greedy_actions with batch_softmax_actions using the Gumbel-max
trick (argmax(Q/T + Gumbel(0,1))) — fully GPU-resident, no CPU
softmax/sampling. Temperature 0.1: nearly greedy when Q-values are
separated, diverse when uniform. Logs unique_actions/45 per backtest.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The sensor passes GitLab commit data containing single quotes in
commit messages (e.g. Merge branch 'feature-name'). When embedded
directly in the shell script via Argo template substitution, this
breaks the shell syntax. Pass as a container env var instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When using templateRef, Argo only borrows the template definition — not
spec-level volumes or volumeClaimTemplates from the source WorkflowTemplate.
Add workspace VCT, sccache PVC, and git-ssh-key secret at the ci-pipeline
spec level so compile-training's sub-templates can access them.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace legacy Buy/Sell/Hold collapse with exposure-aware evaluation.
Long100→Long50 now generates a partial-close trade instead of being
a no-op. Combined with graduated trade penalty, PSO now gets gradient
signal across the entire 26D search space.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add calculate_trade_insufficiency_penalty() that produces monotonically
decreasing penalties for low trade counts (10.0 at 0 trades, 0.0 at 100+),
giving PSO gradient signal across the degenerate plateau where all trials
produce the same 1.25 objective.
Wire the penalty into extract_objective() with short-circuit for <10 trades
(skip composite score since it's all zeros anyway).
Include 4 unit tests verifying zero/one/graduated/no-plateau properties.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add factored evaluation support to EvaluationEngine with continuous
exposure tracking (-1.0 to +1.0), partial position changes, and
order-type-aware transaction costs for the 45-action factored space.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace simulated login in fxt CLI with real AuthServiceClient gRPC call
- Add auth proto to fxt build.rs compile list and lib.rs proto module
- Add api.access permission to JWT claims issued by AuthGrpcService
- Remove orphaned ci-sensor.yaml (replaced by cluster-applied version)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
argo-workflow SA needs workflows/create and workflowtemplates/get
permissions to submit CI pipelines from sensor triggers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unnecessary case map — service names already match Cargo package names
- Add sensor pod serviceAccountName: argo-workflow (RBAC fix)
- Add sensor config to events/ for version tracking
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace compile-api-gateway + compile-web-gateway with single compile-api task
- Update detect-changes script: services/api/ instead of services/api_gateway/
- Remove web-gateway from change detection and compile targets
- Fix templateRef parameter resolution (inputs.parameters instead of workflow.parameters)
- Add SSH key setup and cluster-internal GitLab host for compile-service
- Use ci-builder image (has ssh) instead of ci-builder-cpu for service compilation
- Add compile-training and build-ci-image templates to kustomization
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- JWT issuer now foxhunt-api across all 16 files (services, tests, config, docker-compose)
- Remove serde alias api_gateway_url from FxtConfig (no backwards compat)
- Remove api_gateway CLI alias from e2e orchestrator
- All services must deploy simultaneously for JWT validation to match
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend (Rust):
- Create proto/auth.proto with Login and RefreshToken RPCs
- Implement AuthGrpcService in services/api with JWT issuance
- Register as unauthenticated service in main.rs (pre-auth)
- Update JWT issuer from foxhunt-api-gateway to foxhunt-api
Dashboard (TypeScript):
- Install @connectrpc/connect-web + @bufbuild/protobuf
- Generate TypeScript proto clients via buf (src/gen/)
- Create grpc.ts transport with JWT interceptor
- Rewrite all useApi hooks to typed gRPC calls
- Replace WebSocket with useGrpcStream server-streaming hooks
- Migrate all 6 pages + 6 components to grpc-web
- Delete api.ts, websocket.ts, useWebSocket.ts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copied from api_gateway, removed REST handlers (port 8080),
added tonic-web + CORS for grpc-web browser access.
Binary renamed: api-gateway → api
Changes:
- Package name: api-gateway → api
- Deleted src/handlers/ (REST ML endpoints on port 8080)
- Added tonic-web 0.13 + tower-http CORS layer
- Server::builder().accept_http1(true) for grpc-web
- CORS_ORIGINS env var (default http://localhost:5173)
- Metrics server on port 9091 (axum) preserved
- All 95 lib tests pass, 0 clippy warnings
- Added services/api to workspace members
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unify both gateways into a single gRPC-only `services/api/` with tonic-web
for browser access. Drop REST+WebSocket, keep full 6-layer auth.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When use_noisy_nets=true (the conservative() default), epsilon never
decayed from 1.0 because (1) the trainer skipped update_epsilon() and
(2) DQNAgentType::set_epsilon() was a no-op for RegimeConditional agents.
This caused ALL training actions to be random — Q-values were learned but
never used for action selection.
Fix: set stored epsilon to 0.0 at training start when noisy nets are on.
Exploration is provided by NoisyLinear weight perturbation + the separate
noisy_epsilon_floor (5% safety floor for 45-action spaces).
Also fixes:
- Zstd-compressed .dbn file detection via magic bytes (0x28B52FFD)
- Test data path resolution using ancestors().find() for workspace root
- Test assertions updated for epsilon < 0.01 with noisy nets
2698 lib tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replaces monolithic compile-services with 8 per-service compile jobs,
each with dependency-aware changes: filters. Deploy job only restarts
services whose binary actually changed. Also renames service crates
from snake_case to kebab-case to match k8s deployment names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add live training metrics monitor CLI command (streaming & one-shot) using
the monitoring gRPC service. Update DQN tests to match post-fix defaults:
IQN disabled, CQL alpha=0.1, v_min/v_max widened, 26D search space.
- train.rs: `fxt train monitor [--once] [--model X] [--interval N]`
- Rewrite gradient collapse test for BF16 mixed precision awareness
- Update inference test config to match trainer defaults (IQN off, CQL on)
- Update production smoke test for 26D parameter space
- Add dqn_action_collapse_fix_test.rs verifying all 6 root cause fixes
- Add planning docs for monitoring service and epoch financial metrics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Expand search space from 25D to 26D with cql_alpha (0.0-0.5)
- v_min bounds: (-3,-1) → (-15,-3), v_max: (1,3) → (3,15)
- Default use_qr_dqn: true → false (IQN disabled)
- Wire cql_alpha into DQNHyperparameters construction
- Update all 18 adapter tests for new dimensions and ranges
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
