On Ampere+ GPUs the training dtype is BF16. Network forward passes
with BF16 inputs return BF16 tensors, but all loss-path arithmetic
(rewards, dones, gamma, PER weights, Huber constants, atoms, clip
epsilon) was created as F32 from Tensor::from_vec. Candle forbids
mixed-dtype binary ops, causing "dtype mismatch in {mul,sub,add}"
on every training step.
DQN fixes (7 mismatch sites):
- Cast current_q_values to F32 immediately after forward pass
- Cast all target network outputs (standard, dueling, C51, IQN) to F32
- Keep rewards/dones/gamma/weights/atoms/half/delta as F32 (remove
.to_dtype(training_dtype) casts — now use DType::F32 sentinel)
- Entropy regularization and CQL paths now match (F32 + F32)
PPO fixes (3 mismatch sites):
- Use DType::F32 for clip epsilon one_tensor (was training_dtype BF16)
- Cast critic.forward() output to F32 in both MLP and LSTM value loss
- Cast target_returns to F32 for symlog/normalization path
Infra: revert runner-h100 from SXM2 (zero Scaleway quota) back to
ci-training-h100 PCIe pool.
2704 ml lib tests pass, 260 PPO tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Eliminate foxhunt-binaries and training-binaries PVCs — services and
training jobs now fetch binaries from MinIO via rclone initContainers.
This removes L40S GPU autoscale-for-PVC-writes, removes RWO node
affinity constraints, and requires zero image rebuilds.
Changes:
- .gitlab-ci.yml: replace ~115 lines of binary-writer pod logic with
aws s3 cp to MinIO (~25 lines)
- 8 service YAMLs + 2 GPU overlays: add rclone initContainer + emptyDir
- Training job template: MinIO rclone fetch replaces PVC copy
- Delete foxhunt-binaries-pvc.yaml and training-binaries-pvc.yaml
- Add s3.fxhnt.ai DNS record (Terraform) and nginx proxy block
- Replace pod-writer-deploy skill with MinIO-based deploy skill
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
kube-rs Client::try_default() uses rustls for in-cluster TLS but no
CryptoProvider was installed, causing a panic on first pod list call.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BUG #41 kept forward pass in F32 for autograd, but the target-side
tensors (reward, gamma, done, next_q) were cast to BF16 via `dtype`.
The `state_action_values.sub(&target_q_values)` then hit F32-vs-BF16
mismatch on Ampere+ GPUs, causing every training step to fail silently.
Fix: `.to_dtype(state_action_values.dtype())` on the detached target.
Safe because target is detached (no autograd graph to break).
Also: H100 runner → SXM2 pool, GPU availability checker script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 32 MCP tool handlers now make real gRPC calls to the API Gateway
instead of returning stub responses. Covers: service monitoring,
training management, trading operations, ML inference, risk metrics,
broker status, data acquisition, cluster resources, agent control,
config management, and hyperopt tuning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
end_to_end_tests.rs and error_handling_tests.rs imported non-existent
modules (fxt::client, fxt::prelude, etc.) from a prior architecture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- StreamSystemStatus now queries real Prometheus data (was all zeros)
- StreamAlerts returns honest unimplemented (was sending empty events)
- Training epoch shows "N" not "N/0" (max_epochs not in proto)
- GPU power shows "300W" not "300W / 0W" when limit unavailable
- Services cockpit shows error reason when service is DOWN
- System status polling detects staleness after 3 consecutive failures
- Training cockpit header shows active K8s job count
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 30 MCP tool handlers now return structured JSON error with
isError=true instead of fake "stub: would ..." placeholder responses.
Tool registrations preserved so list_tools still works.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
config_service.proto (package foxhunt.config, 4 RPCs) was a subset of
config.proto (package config, 12 RPCs). Migrated api_gateway to implement
ConfigService from config.proto directly. Removed dead config_service()
method from fxt client.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
57 references updated across source, tests, benches, and config.
Proto package name foxhunt.tli kept for wire compatibility.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace three occurrences of String::new() used as "fetch all" filter
values in gRPC request construction with a named FILTER_ALL constant,
making the intent explicit.
The api_gateway monitoring handler was reviewed but its Prometheus URL
appears only once in tests, so no constant was extracted (single-use).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use em-dash (—) consistently instead of mix of - and — for "no data"
placeholders across all cockpits (data, trading, risk, services, pods)
- Collapse pipeline status section to single "not connected" message
instead of showing 4 misleading rows of dashes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Same class of bug as the DQN fix (4c88498b): network outputs were
being cast to F32 mid-pipeline, defeating tensor-core acceleration
on H100/L40S. Now the full training loop stays in training_dtype()
(BF16 on CUDA Ampere+, F32 on CPU), with F32 casts only at scalar
extraction boundaries (to_scalar, to_vec1).
Files fixed:
- ppo.rs: Actor/Critic forward, act_with_log_prob, compute_losses,
update_mlp, LSTM recurrent loop, predict method
- lstm_networks.rs: removed F32 output casts from both networks
- continuous_ppo.rs: one_tensor + scalar extractions
- hidden_state_manager.rs: zeros/ones use training_dtype()
- flow_policy/mod.rs: log_det accumulators + dummy log_std
- ensemble/adapters/ppo.rs + dqn.rs: F32 cast at extraction
2704 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove premature F32 output casts from network forward passes
(Sequential, NetworkLayers, DistributionalDueling) that were negating
tensor-core acceleration. Instead, cast F32 input tensors (rewards,
dones, gamma, weights, atoms) to training_dtype() at the boundary
and only escape to F32 at scalar extraction points (loss value,
TD errors, logging vectors).
465 DQN tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add ServiceAccount, Role, and RoleBinding so the api-gateway pod can
list/get/watch pods and pod metrics in the foxhunt namespace. Also sets
serviceAccountName on the Deployment to bind the new SA.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allow api-gateway to reach the Kubernetes API server for pod listing
(Pods cockpit). Two CIDR blocks are needed because Cilium applies
NetworkPolicy after DNAT:
- 10.32.0.0/12:443 — kubernetes.default.svc ClusterIP
- 172.16.0.0/16:6443 — real node IP after Cilium DNAT
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add kube 3.0 + k8s-openapi 0.27 dependencies and create pods_handler.rs
that queries the Kubernetes API for pod status. Replace the UNIMPLEMENTED
stub in monitoring_handler.rs with a real streaming implementation that
polls pods every N seconds and maps them to PodInfo proto messages.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a new gRPC stream spawner that subscribes to SubscribeClusterPods
and groups pods by service name for the pods cockpit view. Follows the
same backoff/reconnect pattern used by all other stream spawners.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add query_cluster_resources() for real CPU/RAM from Prometheus
- Wire into build_response() replacing hardcoded zeros
- Fix NetworkPolicy: add ports 8080/9100/9400 for KSM/node-exporter/dcgm
- Add CiliumNetworkPolicy for hostNetwork pod scraping
- Add SubscribeClusterPods stub to unblock proto addition from Task 5
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add K8s pod data structures to TUI state layer for the upcoming pods
cockpit view. PodData represents individual pods, PodGroupData groups
them by service with ready/total counts. The StateUpdate::Pods variant
and apply_update handler wire the data path from gRPC streams to state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
NoisyLinear layers have different parameter names (sigma_weight/sigma_bias)
than standard Linear layers. Hardcoding use_noisy_nets=false caused
checkpoint loading to silently fail when the training run used noisy nets,
resulting in empty folds in the eval report.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pool ci-training-h100x2 (H100-2-80G, 160GB VRAM total).
MultiGpuConfig::detect() in ml crate auto-enables data parallelism.
Tags: kapsule,h100x2. Autoscale 0→1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Clear separation: l40s tag → L40S pool, h100 tag → H100 pool.
Updated runner-rl-values.yaml and CI base templates to match.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create ci-training-h100 runner (runner-h100-values.yaml)
- Route hyperopt-dqn and hyperopt-ppo to H100 via kapsule,h100 tags
- H100 is ~2x faster and cheaper per run vs L40S for hyperopt workloads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
num_atoms, dueling_hidden_dim, v_min/v_max, gamma were using defaults
instead of hyperopt values — causing tensor shape mismatch on checkpoint
load (e.g. output layer 45×200=9000 vs 45×51=2295). Also fixed use_iqn
to read from use_qr_dqn key (matching trainer's field mapping).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
evaluate_baseline was using default hidden_dim_base=128 instead of the
hyperopt-optimized 4096, causing checkpoint loading to fail silently
(tensor shape mismatch). This produced empty folds: [] in eval reports.
Also adds consistent --tx-cost-bps/--tick-size/--spread-ticks to
train_baseline_rl and evaluate_baseline in the hyperopt-dqn job so all
three pipeline stages (hyperopt, train, eval) use the same cost model.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Hyperopt found hold_penalty_weight=1.818, curiosity_weight=0.402, etc.
but walk-forward training used tiny defaults (0.01, 0.1), causing the
agent to learn "holding is optimal" → 0 trades across all 50 epochs.
Now passes: hold_penalty_weight, max_position_absolute, huber_delta,
entropy_coefficient, curiosity_weight, weight_decay, kelly_fractional,
kelly_max_fraction from hyperopt JSON to DQNHyperparameters.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Walk-forward DQN training hardcoded epsilon_start=1.0 with noisy_nets=true,
forcing pure random exploration for all 50 epochs (0 trades, 0 Sharpe).
Now reads epsilon, PER, dueling, distributional, noisy nets, and QR-DQN
params from the hyperopt JSON. When noisy_nets=true, defaults epsilon to
0.05 instead of 1.0.
Evaluator now falls back to highest dqn_fold{N}_epoch{E}.safetensors when
_best.safetensors doesn't exist (early stopping saves epoch checkpoints).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
