Replace &Device::Cpu with &self.device for input and target tensor
creation in Mamba2 hyperopt adapter. Avoids unnecessary CPU→GPU
transfer during hyperparameter optimization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stream-aware ensemble that runs models on separate CUDA streams
for true GPU-level parallelism. Falls back to rayon on CPU.
Uses CudaStreamPool for synchronization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CUDA stream pool with CPU no-op fallback. Foundation for
StreamAwareEnsemble that runs models on separate CUDA streams.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use predict_raw() to collect raw GPU tensors from adapters. Stack,
sigmoid, weighted-sum on GPU before single extraction. Falls back
to CPU path for adapters without tensor output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Override predict_raw() in TGGN, TLOB, KAN, xLSTM, Diffusion adapters
to return raw GPU tensors. Enables GPU-side ensemble aggregation
instead of per-model CPU extraction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace clamped.to_vec1() CPU loop with GPU-native floor/ceil/frac
operations. Removes 3 tensor transfers per forward pass (N floats
down + 3*N up). All index computation now stays on GPU.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stack var_returns, var_residuals, mean_reward, var_reward into single
tensor before extraction. Uses broadcast_sub for GPU-native mean
centering instead of scalar round-trip.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backward-compatible trait extension. Default predict_raw() wraps
predict() result with tensor: None. Adapters can override to return
raw GPU tensors for GPU-side ensemble aggregation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The S3→PVC migration had several issues causing deploy failures:
- Stale binary-writer pod from previous failed deploys blocked new ones
(terminated pods can't be updated via kubectl apply)
- Services were applied BEFORE binaries written to PVC, so first deploy
or empty PVC caused pods to crash (binary not found)
- GPU overlay files in services/ dir were auto-applied by kubectl apply,
referencing nonexistent S3/minio secrets and PVCs
- Training job template mounted foxhunt-binaries PVC but training runs
on ci-training node — RWO PVC is bound to foxhunt node
Fixes:
- Reorder deploy: write binaries → apply manifests → rollout restart
- Clean up stale writer pods before creating new ones
- Move GPU overlays to gpu-overlays/ (manual apply only), update to PVC
- Add training-binaries PVC for GPU node, best-effort population
- Training job template uses initContainer to copy from training PVC
- set -e for critical path, set +e for optional training binary copy
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-param gradient norm extraction loop with batched
Tensor::stack pattern. Single GPU→CPU sync instead of one per
parameter tensor.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-param calculate_gradient_norm() loop with batched
Tensor::stack pattern. Single GPU→CPU sync instead of one per
parameter tensor.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-batch loss.to_vec0() in TFT training/validation loops
with GPU tensor accumulation. Single extraction per epoch + NaN guard
every 100 batches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-batch .to_dtype(F64).to_scalar() in Liquid train/validate
with GPU tensor accumulation. Single extraction per epoch + NaN guard
every 100 batches. Removes ~100-500 GPU→CPU syncs per epoch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-batch loss.to_scalar() in TLOB train_epoch/validate_epoch
with GPU tensor accumulation. Single extraction per epoch + NaN guard
every 100 batches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The before_script was copying training binaries to ${CI_PROJECT_DIR}/bin/
which collides with the git checkout bin/fxt/ directory. chmod +x on the
fxt subdirectory fails with "Operation not permitted", crashing the job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runner #2 already has tags [kapsule, gpu] but CI jobs still used
kapsule-rl. Also removes stale minio-ca-cert volume from runner values.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy stage needs compile-training artifacts to copy training binaries
to the foxhunt-binaries PVC alongside service binaries.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
MinIO pod was removed but compile-training still uploaded to it via rclone,
causing 403 Forbidden failures. Now:
- compile-training: saves binaries as CI artifacts (3 day TTL)
- deploy stage: copies training binaries to foxhunt-binaries PVC via binary-writer
- CI training jobs: use compile-training CI artifacts directly
- job-template.yaml: mounts foxhunt-binaries PVC (no initContainer)
- Removed MinIO deploy from deploy stage (no longer needed)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both DQN and PPO eval paths used old 3-action index matching (0=Buy,
1=Sell, 2=Hold). Now uses FactoredAction.target_exposure() for
exposure-weighted returns and order-type-specific transaction costs.
PPO path had .to_int() which doesn't exist on FactoredAction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
compute_reward_pnl() took raw action_idx (0=Buy,1=Sell,2=Hold) — wrong
with 45-action FactoredAction encoding. Now takes &FactoredAction and
uses order-type-specific transaction costs. Position tracking uses
action.exposure instead of old 3-way index match.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hyperopt adapter and PPO benchmark still had num_actions: 3, which would
produce misconfigured PPO models when used with the 45-action FactoredAction
sampling path. Found by spec compliance review.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use a purely linear network (no ReLU) for the completeness axiom test.
IG on linear functions is mathematically exact, so the test is
deterministic regardless of random weight initialization. Tighten
tolerance from 20% to 1% (f32 rounding only). Keep ReLU network in
the basic test for non-linear verification.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- trading_service: PPO predict() used TradingAction match but act() now
returns FactoredAction. Use target_exposure() mapped to 0-1 range.
- IG completeness axiom test: relax tolerance from 5% to 20% (random
weights with ReLU non-linearity and trapezoidal rule can exceed 5%).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Was using direction=idx/15 (3 groups of 15: Buy/Sell/Hold) — incompatible
with DQN's FactoredAction encoding. Now uses exposure_idx=idx/9 (5 groups
of 9: Short100/Short50/Flat/Long50/Long100) matching FactoredAction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sample_action(), act(), act_with_log_prob(), greedy_action() now return
FactoredAction instead of TradingAction. Fixes the architectural disconnect
where num_actions=45 output neurons were sampled through a 3-action bottleneck.
TrajectoryStep.action and TrajectoryBatch.actions now use FactoredAction.
Added FactoredAction::from_legacy() for backward compatibility in tests.
Updated all PPO consumers: trainers/ppo.rs, hyperopt/adapters/ppo.rs,
validation/ppo_adapter.rs, benchmark/ppo_benchmark.rs.
2487 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidates ExposureLevel, Urgency, FactoredAction from dqn/action_space.rs
and ppo/factored_action.rs into common/action.rs. Both dqn:: and ppo::
re-export for backward compatibility. Deletes ppo/factored_action.rs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Changed from Status::unavailable to Status::unimplemented with path
forward (ml_training_service forwarding). Fixed supports_feature_importance
from true to false.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
calculate_feature_importance() returned hardcoded fabricated scores with
wrong feature names on every inference. Replaced with honest empty map.
Real importance is computed on-demand via integrated gradients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements IntegratedGradients using Candle autograd (Var::from_tensor +
backward). Computes attributions by integrating input gradients along
interpolation path from baseline to input.
Verified via completeness axiom test: sum(attributions) ≈ F(x) - F(baseline).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The real_ prefix was misleading — there is no fake data loader.
Mechanical rename across 18 source files, no logic changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DQN PER: defer td_errors to_vec1() after loss.to_scalar() — piggyback on
existing pipeline flush instead of forcing premature GPU→CPU stall
- PPO trajectories: capacity-hint Vec allocations, extend_flat_states methods,
states_flat field on TrajectoryBatch for zero-copy GPU upload
- TGGN validate(): batch N per-sample losses on GPU → single to_scalar() sync
(was N GPU→CPU syncs)
- Liquid backward(): batch grad-norm per-param sqr().sum_all() on GPU → single
to_scalar() sync (was N GPU→CPU syncs per optimizer step)
- Liquid validate(): same N→1 GPU sync reduction as TGGN
- DQN trainer: restore EpochPrefetcher/DoubleBufferedLoader API (wrongly deleted)
- train_baseline_rl: wire DoubleBuffer GPU pre-upload — after CPU prefetch
completes, immediately upload next fold to GPU via DqnGpuData::upload() so
next fold starts with data already resident on GPU
2478 tests pass, 0 clippy warnings, 0 compile errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The remoteWrite block had placeholder values (COCKPIT_METRICS_PUSH_URL,
COCKPIT_PUSH_TOKEN) that were never replaced, causing Prometheus to
spam "Failed to send batch" warnings every minute. Cockpit was replaced
by self-hosted Grafana+Prometheus+Loki+Tempo.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ibapi handshake: retry once after 1s on first failure (IB Gateway
needs time to release stale client_id slots after disconnection)
- test_load_nonexistent_config → test_load_config_succeeds: remove
default-value assertions that fail when ~/.foxhunt/config.toml exists
(defaults already tested by test_serde_defaults and test_default_config)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Match the actual K8s broker-gateway service port discovered during
live validation of fxt broker check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The IB Gateway JVM uses non-blocking I/O on port 4002, which causes
EAGAIN errors with ibapi's blocking Client::connect(). The gnzsnz
image provides socat on port 4004 as a stable proxy that bridges
blocking clients to the non-blocking gateway socket.
Port 4004 is the correct client-facing port for paper mode (4003 for
live). The K8s readiness probe stays on 4002 since TCP probes work
fine with non-blocking sockets.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Readiness/liveness probes were checking port 4004 (socat), which
always listens even when the gateway is stuck at login — masking
failures. Now probes check port 4002 (TWS API), which only opens
after successful IBKR authentication.
Deploy strategy changed to Recreate because IBKR allows only one
session per account. RollingUpdate starts the new pod before killing
the old one, causing both to fight over the session in a crash loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The gnzsnz/ib-gateway image renders its IBC config.ini from a template
using envsubst. The ExistingSessionDetectedAction field was rendering
as empty, causing IBC to show a GUI dialog when IBKR detected a
competing session — which nobody could dismiss in a headless pod.
Setting primaryoverride tells IBC to automatically take over any
existing session, which is the correct behavior for a K8s deployment.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Broker variant to Commands enum with BrokerArgs (flatten),
route to execute_broker_command in match block (no JWT required),
exit(1) on check failure. Clone config.api_gateway_url to avoid
partial move. Add two CLI parsing tests for broker check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy broker_gateway.proto from broker_gateway_service into fxt/proto/
and wire it into build.rs compile_protos + lib.rs proto module so the
CLI can call BrokerGatewayService RPCs (health check, account state,
session status).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds IbkrConfig (host/port/client_id with defaults for paper trading)
and BrokerConfig (gateway_url + ibkr nested) to the TLI config file.
Both structs derive serde defaults so existing config files remain
backward-compatible.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two-layer IBKR connectivity validation: direct ibapi handshake +
broker-gateway gRPC health check. Pure-client architecture using
ibapi as optional dep on fxt (not trading_engine).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
