Branch worktree-agent-aadd27f0, commit 9b27428de. Replaces the stub
{value:0.5, confidence:0.0} with dispatch via Arc<dyn
ModelInferenceAdapter>, wrapping any of the 10 concrete per-model
adapters in crates/ml/src/ensemble/adapters/. prediction_value
derived from inner direction∈[-1,1] via (d+1)/2 bullish probability.
confidence forwarded from inner adapter (softmax-max / quantile IQR
per model), clamped [0,1]. Returns Err (not fake 0-confidence) when
inner is_ready=false, inner predict fails, output non-finite, or
features empty.
build_production_strategy signature now Vec<(String, Arc<...>)>; sole
caller ml_strategy_engine.rs passes empty vec (not 10 ghost adapters)
with a note to wire real from_checkpoint when backtesting-side
checkpoint loading is ready.
# Conflicts:
# crates/ml/src/ensemble/model_adapter.rs
The adapter's `predict` returned
`{ prediction_value: 0.5, confidence: 0.0 }` as a "neutral" stub.
Downstream ensemble filtering dropped any 0-confidence prediction,
so the adapter silently contributed nothing -- a stub that survived
only because the caller discarded it.
Now: `EnsembleModelAdapter` holds an
`Arc<dyn ModelInferenceAdapter>` and dispatches `predict` to the
wrapped per-model GPU adapter (`DqnInferenceAdapter`,
`PpoInferenceAdapter`, `TftInferenceAdapter`, `Mamba2InferenceAdapter`,
`LiquidInferenceAdapter`, `KanInferenceAdapter`,
`XlstmInferenceAdapter`, `TggnInferenceAdapter`,
`TlobInferenceAdapter`, `DiffusionInferenceAdapter`). The inner
adapter already runs a full forward pass on its model and returns a
normalized `(direction, confidence)` pair; the bridge maps
`direction in [-1, 1]` -> `prediction_value in [0, 1]` via
`(direction + 1) / 2` to match `MLPrediction`'s bullish-probability
contract (> 0.5 = bullish), clamps confidence to [0, 1], and
propagates `metadata.latency_us` as the inference latency.
The bridge returns `Err` (never a faked 0-confidence success) when
the inner model reports `is_ready() == false`, the inner `predict`
fails, the output is non-finite, or the feature slice is empty.
`build_production_strategy` no longer fabricates ten zero-confidence
ghost adapters. It now accepts
`Vec<(String, Arc<dyn ModelInferenceAdapter>)>` -- the caller owns
real model construction (checkpoint loading, device selection). The
only current caller (backtesting service `MLPoweredStrategy::new`)
passes an empty vec; that yields an empty ensemble, which is an
honest "no models loaded" signal rather than ten stubs that exist
only to be filtered.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
services/trading_service/src/services/risk.rs compute_sharpe_ratio /
compute_sortino_ratio previously returned `0.0` as an
"insufficient-samples" fallback. Plain zero is indistinguishable
from a legitimate "Sharpe happened to equal 0" result; callers
gating on `sharpe > threshold` silently pass/fail on the fallback,
and downstream monitoring logs it as a real value.
Now: returns `Option<f64>` — `Some(value)` for valid samples,
`None` when:
* returns.len() < MIN_RETURN_OBSERVATIONS (threshold at line 30
unchanged — only the return-shape changes)
* daily_std < 1e-12 (degenerate zero-volatility series; ratio
mathematically undefined, not zero)
* downside_count < 2 (Sortino only; too few sub-risk-free
observations to estimate downside variance)
* downside_dev < 1e-12 (Sortino only)
Production caller (get_risk_metrics, the only call site) updated:
* On Some/Some: debug-log as before
* On either None: WARN-log with `[insufficient-samples]` tag,
formatting each ratio as "None" or "{:.4}" so operators can
distinguish missing data from a real zero
* Protobuf RiskMetrics { sharpe_ratio, sortino_ratio } is a
non-optional `double` in risk.proto — export `f64::NAN` rather
than a fake 0, so consumers can detect the undefined case
without silently failing threshold gates.
Tests: each function now has happy-path `Some(_)`,
insufficient-samples `None`, and empty-input `None` cases;
existing `_zero_volatility` / `_no_downside` tests re-asserted
against `None` (previously asserted against 0.0).
compute_volatility and compute_current_drawdown retain `f64`
return — their 0.0 outputs are legitimate (no position → 0%
drawdown; constant returns → 0% volatility).
Callers touched:
- services/trading_service/src/services/risk.rs:570-596
(get_risk_metrics — only production caller)
- services/trading_service/src/services/risk.rs:1585-1671
(tests)
Cross-file grep (compute_sharpe|compute_sortino|RiskServiceImpl::)
confirms no other callers in crates/, services/, bin/.
position_manager.rs line 772 references the function in a comment
only.
Compile: SQLX_OFFLINE=true CARGO_INCREMENTAL=0 cargo check
--workspace --tests — clean on risk.rs (pre-existing ml-crate
warnings unchanged).
Tests: cargo test -p trading-service --lib risk — 59 passed,
0 failed, including 2 new empty-input tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
migrations:
- 001_trading_events.sql, 003_audit_system.sql: the hard-coded
node_id literals (`trading-node-01`, `audit-node-01`,
`ml-node-01`, `system-node-01`, `change-tracker-01`) are
overridden per-deployment by later migrations rather than read
from the environment. Describe that in the inline comment.
- 004_compliance_views.sql: `generate_compliance_report` is a log
stub — actual report generation is performed by the compliance
service. Say so explicitly.
services:
- ml_training_service/tests/orchestrator_225_features_test.rs: the
empty `#[ignore]`d placeholder for the 225-feature orchestrator
loader has been removed; it held no assertions and only tracked
a TODO (feedback_no_stubs.md).
- trading_agent_service/src/service.rs: portfolio volatility uses
the diagonal-only approximation because cross-asset return
correlations are not maintained in this service. Document that.
- trading_service/src/services/risk.rs: `get_risk_metrics` uses
`calculate_marginal_var` + asset-class fallback; describe why
`calculate_comprehensive_var` is not wired at this boundary.
- trading_service/tests/auth_comprehensive.rs: delete the entire
commented-out legacy BackupCodeValidator test block — the old
`generate_backup_codes` / `store_backup_code` /
`verify_backup_code` surface no longer exists, and MFA
integration tests already cover the new API.
testing:
- harness/grpc_clients.rs: no BacktestingServiceClient proto
exists; reword the stale TODO import line.
- chaos/*: reword the family of "TODO: Implement ..." stubs as
"Currently a no-op / synthetic result" descriptions so readers
know exactly how much of the chaos framework is live.
- compliance_automation_tests.rs: delete the file; it was a giant
/* ... */ block referencing a nonexistent compliance module
and was not wired into any Cargo target.
- framework.rs: describe why `setup()` uses `println!` instead of
`tracing_subscriber` (tracing_subscriber is not a dep of this
integration crate).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove pub state_dim field from DQNConfig and GpuReplayBufferConfig; remove the
state_dim field from GpuExperienceCollector. Replace all reads with
ml_core::state_layout::STATE_DIM (and STATE_DIM_PADDED for cuBLAS-padded
strides). Checkpoint loading now validates saved state_dim against the
constant and hard-errors on mismatch. GpuAttentionConfig.state_dim is a
distinct attention-feature dim and is left untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
OFICalculator + MicrostructureState fed per-tick. Bar close snapshots
and broadcasts 20 features via tokio::watch channel. Intermediate
snapshots available for speculative compute between bars.
Databento live client integration deferred (requires crate dep).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove use_double_dqn, use_dueling, use_per, use_branching,
use_distributional, use_noisy_nets, use_huber_loss, and use_cql
from DQNConfig, DQNHyperparameters, and DqnParams structs.
These features are always enabled (Rainbow DQN standard). The boolean
flags were dead code — every constructor set them to true, and the
only code paths that set them to false were in tests that disabled
features for simplicity. With the fields removed, the features are
unconditionally active, eliminating ~490 lines of dead configuration.
Key changes:
- Struct field declarations removed from 3 core config structs
- Conditional branches (if use_X { ... } else { ... }) simplified:
dueling/branching/PER network creation is now unconditional
- Checkpoint metadata hardcodes "true" for backward compatibility
- Hyperopt search space index 11 (use_branching) fixed at 1.0
- TOML/YAML config files cleaned of removed fields
- Tests that toggled these flags updated or rewritten
45 files changed, -487 net lines. Zero new test failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Final cleanup:
- 61 test files + 5 example files: candle imports replaced
- 8 testing/integration files: migrated to cudarc/ml-core types
- 3 services/trading_service test files: migrated
- Root Cargo.toml: candle-core, candle-nn removed from [workspace.dependencies]
- crates/ml/Cargo.toml: candle-nn dependency removed
- testing/e2e/Cargo.toml: candle-core dependency removed
Zero active candle_core/candle_nn/candle_optimisers code references remain.
Zero candle dependency declarations in any Cargo.toml.
Remaining "candle" strings are exclusively in doc comments.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove ALL #[cfg(feature = "cuda")] guards (~400+ occurrences)
- Remove ALL #[cfg_attr(not(feature = "cuda"), ignore)] test annotations (~250)
- Make cuda default feature in 9 ML crates (ml, ml-core, ml-dqn, ml-ppo, etc.)
- Convert nvrtc JIT compilation to precompiled nvcc (searchsorted, prefix_sum)
- Move compile_ptx_for_device() to ml-core for shared access
- Delete dead CPU code: multi_step.rs, self_supervised_pretraining.rs,
training_guard_gpu_tests.rs, CPU PER buffer paths, CPU Q-diagnostics
- Replace unwrap_or(Device::Cpu) with hard errors everywhere
- Remove dead is_cuda() else branches in DQN/PPO/hyperopt trainers
- Change config defaults from "cpu" to "cuda" (rainbow, tlob, pipeline)
- Port IQL value network to GPU kernel (5 CUDA entry points)
- Port HER goal relabeling to GPU kernel (warp-per-sample)
- Wire DSR GPU-to-CPU sync in training loop
- cfg!(feature = "cuda") → true in inference_validator
Zero warnings, zero errors across entire workspace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ProductionFeatureExtractorAdapter was changed to produce 42 features
(40 base + 2 regime) but three test sites and the FEATURE_NAMES constant
still expected 51 (42 + 1 volatility_regime + 8 OFI placeholders).
- backtesting: strategy_runner test assertions 51→42
- trading-service: ensemble_coordinator test assertions 51→42
- trading-service: FEATURE_NAMES_51 → FEATURE_NAMES_42 (drop OFI
placeholders and volatility_regime, matching extraction.rs v2 layout)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move all inline type definitions from ml/src/lib.rs to ml-core:
MLError, MLResult, Trade, MarketRegime, HealthStatus, Features,
MLModel trait, ModelRegistry, ParallelExecutor, LatencyOptimizer,
TrainingMetrics, ValidationMetrics, InferenceResult, ModelMetadata.
Dedup: consolidate ConfigError{reason}/ConfigurationError(msg) into
single ConfigError(String) tuple variant (was 2 variants, 174 refs).
Cleanup: convert create_hft_* free functions to associated methods
(HFTPerformanceProfile::ultra_low_latency(), ParallelExecutor::hft()).
ml facade re-exports via `pub use ml_core::*`.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Critical bug: all 3 DQN action selection methods (select_action,
select_action_with_confidence, select_action_inference) used
FactoredAction::from_index() which maps indices 0-4 to exposure_idx=0
(Short100) via division by 9. This is the root cause of action
diversity collapse during both training and production inference.
Fix: ExposureLevel::from_index() + OrderRouter::route_default() in all
DQN paths. Also fixes hyperopt objective thresholds (<10 → <3 for
5-action degenerate detection), stale defaults/comments, integration
test configs.
Files: dqn.rs (3 methods), trainer.rs (validation + select_action),
hyperopt/adapters/dqn.rs (thresholds), dqn_model.rs (comments),
train_baseline_rl.rs (default), reward.rs (comment),
dqn_integration.rs + ensemble_integration.rs (num_actions).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add api_uptime_seconds gauge to API service (5s update interval),
matching the pattern used by trading/backtesting/ml-training services
- Remove obsolete Web Gateway Uptime and Active WebSocket Connections
panels from cockpit (web-gateway is not deployed)
- Add API Gateway Uptime (api_uptime_seconds) and API Auth Rate
(api_auth_requests_total) panels in their place
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The dashboard's Config page was returning [unimplemented] because
trading-service had no ConfigService implementation. The API gateway's
ConfigServiceProxy forwards config RPCs to trading-service which now
handles them via direct SQL against the `configuration` table.
Implements GetConfiguration, UpdateConfiguration, DeleteConfiguration,
and ListCategories. Remaining 8 RPCs return UNIMPLEMENTED until needed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
containerd on Kapsule nodes uses host DNS which can't resolve
.svc.cluster.local names. Switch all image references from
gitlab-registry.foxhunt.svc.cluster.local:5000 to localhost:30500
(NodePort on the GitLab registry). Also make training runtime image
configurable via TRAINING_RUNTIME_IMAGE env var.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace simulated login in fxt CLI with real AuthServiceClient gRPC call
- Add auth proto to fxt build.rs compile list and lib.rs proto module
- Add api.access permission to JWT claims issued by AuthGrpcService
- Remove orphaned ci-sensor.yaml (replaced by cluster-applied version)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- JWT issuer now foxhunt-api across all 16 files (services, tests, config, docker-compose)
- Remove serde alias api_gateway_url from FxtConfig (no backwards compat)
- Remove api_gateway CLI alias from e2e orchestrator
- All services must deploy simultaneously for JWT validation to match
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backend (Rust):
- Create proto/auth.proto with Login and RefreshToken RPCs
- Implement AuthGrpcService in services/api with JWT issuance
- Register as unauthenticated service in main.rs (pre-auth)
- Update JWT issuer from foxhunt-api-gateway to foxhunt-api
Dashboard (TypeScript):
- Install @connectrpc/connect-web + @bufbuild/protobuf
- Generate TypeScript proto clients via buf (src/gen/)
- Create grpc.ts transport with JWT interceptor
- Rewrite all useApi hooks to typed gRPC calls
- Replace WebSocket with useGrpcStream server-streaming hooks
- Migrate all 6 pages + 6 components to grpc-web
- Delete api.ts, websocket.ts, useWebSocket.ts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copied from api_gateway, removed REST handlers (port 8080),
added tonic-web + CORS for grpc-web browser access.
Binary renamed: api-gateway → api
Changes:
- Package name: api-gateway → api
- Deleted src/handlers/ (REST ML endpoints on port 8080)
- Added tonic-web 0.13 + tower-http CORS layer
- Server::builder().accept_http1(true) for grpc-web
- CORS_ORIGINS env var (default http://localhost:5173)
- Metrics server on port 9091 (axum) preserved
- All 95 lib tests pass, 0 clippy warnings
- Added services/api to workspace members
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename 7 service binaries from snake_case to kebab-case to match
K8s deployment names. Update Cargo.toml package/bin names, K8s
manifest S3 paths and commands, and cross-crate dependency keys.
- api_gateway → api-gateway
- trading_service → trading-service
- broker_gateway_service → broker-gateway
- ml_training_service → ml-training-service
- backtesting_service → backtesting-service
- trading_agent_service → trading-agent-service
- data_acquisition_service → data-acquisition-service
broker-gateway gets an explicit [lib] name = "broker_gateway_service"
since its new package name maps to broker_gateway (not the original
broker_gateway_service used in source code). All other services map
correctly with Rust's automatic hyphen-to-underscore conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change PercentileScaler, RewardNormalizer, CompositeReward, and
PPORewardShaper public APIs from f64 to f32 — matching what callers
actually pass. Internal EMA/percentile math stays f64 for precision.
Keep data_loader SMA/EMA/MACD accumulators in f64 throughout (was
f64→f32→f64 per step), cast to f32 only at output boundary.
Remove dead _transition computation in rainbow_agent_impl and unused
bigdecimal deps from broker_gateway_service and trading_service.
2704 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
kube-rs Client::try_default() uses rustls for in-cluster TLS but no
CryptoProvider was installed, causing a panic on first pod list call.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- StreamSystemStatus now queries real Prometheus data (was all zeros)
- StreamAlerts returns honest unimplemented (was sending empty events)
- Training epoch shows "N" not "N/0" (max_epochs not in proto)
- GPU power shows "300W" not "300W / 0W" when limit unavailable
- Services cockpit shows error reason when service is DOWN
- System status polling detects staleness after 3 consecutive failures
- Training cockpit header shows active K8s job count
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
config_service.proto (package foxhunt.config, 4 RPCs) was a subset of
config.proto (package config, 12 RPCs). Migrated api_gateway to implement
ConfigService from config.proto directly. Removed dead config_service()
method from fxt client.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add kube 3.0 + k8s-openapi 0.27 dependencies and create pods_handler.rs
that queries the Kubernetes API for pod status. Replace the UNIMPLEMENTED
stub in monitoring_handler.rs with a real streaming implementation that
polls pods every N seconds and maps them to PodInfo proto messages.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add query_cluster_resources() for real CPU/RAM from Prometheus
- Wire into build_response() replacing hardcoded zeros
- Fix NetworkPolicy: add ports 8080/9100/9400 for KSM/node-exporter/dcgm
- Add CiliumNetworkPolicy for hostNetwork pod scraping
- Add SubscribeClusterPods stub to unblock proto addition from Task 5
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Added select_action_inference(&self) — pure greedy forward pass with
no exploration mutations (epsilon, noisy nets, count bonus, step counter
are all skipped). Returns (FactoredAction, f32) where confidence is
softmax probability of the selected action, clamped [0.5, 0.95].
Also added select_action_with_confidence(&mut self) for training path
with the same confidence computation.
Production wrapper now uses read lock (was write lock), enabling
concurrent predictions. Confidence is real softmax probability
instead of hardcoded 0.85.
323 trading_service tests, 15 DQN core tests, 0 clippy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PPO now reads architecture config (state_dim, num_actions, hidden_dims,
clip_epsilon, entropy_coeff, max_grad_norm) from the _metadata.json
saved alongside the actor checkpoint. Falls back to hardcoded defaults
for legacy checkpoints without metadata.
Also fixes predict() to use self.feature_count instead of hardcoded 54,
matching the DQN pattern.
323 tests, 0 clippy.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DQN: disable all exploration for production (warmup=0, epsilon=0,
noisy_nets=false, count_bonus=false), read feature_count from
checkpoint metadata instead of hardcoding 54, add loaded guard
and NaN check on predict output.
PPO: fix confidence formula — use act_with_log_prob() instead of
act() which returns value_estimate not log_prob, add loaded guard
and NaN check, remove WorkingPPO alias.
Liquid: add loaded flag for is_ready()/predict() guards.
Remove EgoboxOptimizer/EgoboxOptimizerBuilder backward-compat
aliases — replaced with canonical ArgminOptimizer everywhere.
323 trading_service tests, 200 hyperopt tests, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Extract 5 model wrappers from enhanced_ml.rs into dedicated files:
dqn_model.rs, ppo_model.rs, tft_model.rs, mamba2_model.rs, liquid_model.rs
- Rename enhanced_ml.rs → ml.rs, EnhancedMLServiceImpl → MLService
- Drop legacy "Real" prefix: DQNModel, PPOModel, TFTModel, Mamba2Model, LiquidModel
- Delete dead ml.rs stub (was never wired in mod.rs)
- Move model tests into their respective modules with proper test helpers
- DQNModel uses clean DQN API with FactoredAction (45 actions), checkpoint-driven config
- 312 tests passing, 0 clippy warnings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace 3 stub RPCs with real implementations:
- get_system_status: fan-out gRPC health checks to all backends, reports SystemStatus
- get_health_check: fan-out checks with response time measurement, returns HealthCheck entries
- get_metrics: queries Prometheus for requested metric names
MonitoringServiceHandler now accepts backend URLs via with_backends() builder.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Read new env vars: BROKER_GATEWAY_SERVICE_URL, DATA_ACQUISITION_SERVICE_URL,
ML_SERVICE_URL, TRADING_AGENT_SERVICE_URL
- Create lazy channels: trading channel shared by trading/risk/config proxies,
separate channels for ML, broker, data_acq, trading_agent
- Register 7 new services on router with auth interceptor
- Update reflection service with all 6 new proto descriptors
- Add broker/data_acq/trading_agent to health check backend list
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
