TDD plan covering GpuReplayBuffer, proportional + rank-based GPU
sampling, priority scatter updates, async loss readback, trainer
integration, OOM fallback, and distribution correctness tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Approved design for moving Prioritized Experience Replay entirely to
GPU — flat priority array with parallel prefix-sum sampling, GPU-resident
ring buffer for experiences, async loss readback. Eliminates the last
major CPU bottleneck in the DQN training loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI training jobs run as GitLab runner pods, not K8s Jobs, so the
kube_job_status_active query always returned 0. Switch to
foxhunt_training_active_workers which is emitted by all training
binaries regardless of deployment method.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- DQN/PPO trainers now record epoch, loss, val_loss, batch/s every epoch
- CI training jobs get Prometheus scrape annotations via runner overrides
- Allow Prometheus (foxhunt namespace) to scrape foxhunt-ci pods
- Skip no-model metrics in monitoring service session grouping
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add DNS record, nginx gRPC proxy block, network policy for Tailscale
ingress, and auto-detect fxhnt.ai in fxt monitor URL derivation.
Show GPU telemetry even when no training sessions are active.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
monitoring-service needs egress to Prometheus (port 80/9090) and ingress
from web-gateway (port 50057). web-gateway needs egress to monitoring-service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The compile-services job already builds monitoring_service but the deploy
job was missing it from the PVC copy loop and rollout restart/wait loops.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU experience collector gate checked only `dqn.dueling_q_network`
(plain dueling), but with both `use_dueling: true` AND `use_distributional: true`
(the defaults), DQN creates hybrid `dist_dueling_q_network` instead, leaving
the plain dueling fields as None. This meant the GPU collector never initialized
despite curiosity being enabled.
Fix: Add else-if fallback to check `dist_dueling_q_network`/`dist_dueling_target_network`
when plain dueling fields are None. Same fix applied to the weight sync site.
Also fix test_train_with_empty_data_completes_gracefully: reduce to 5 epochs with
early stopping disabled. The debug-mode async state machine is large enough that
empty-data epochs run ~180ms each (vs ~3ms in release), triggering both plateau
and patience-based early stopping. The test purpose is crash-freedom, not timing.
2497 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire monitoring_service gRPC into web-gateway: proto compilation,
config/state/client plumbing, REST endpoint at /api/monitoring/live-metrics,
and a 3s polling bridge that broadcasts training_progress over WebSocket.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a new CLI subcommand that connects to the monitoring service via
gRPC to display live training metrics. Supports single-snapshot mode
(--once), model filtering (--model), and configurable streaming
interval. Renders per-model epoch/loss table, GPU telemetry, hyperopt
progress, and health counters.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Prometheus client queries training/hyperopt/GPU/K8s metrics, gRPC service
exposes unary + server-streaming RPCs, 4 unit tests, zero clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enable curiosity module by default (weight 0.0→0.1) to satisfy the
three-way gate (dueling + target + curiosity) that was blocking the
GPU experience collector CUDA kernel. This eliminates the 30-40% CPU
experience collection phase that was the main GPU idle bottleneck.
Additional changes:
- BatchSample API: train_step/compute_gradients now accept
Option<BatchSample> instead of Option<Vec<Experience>>, preserving
PER importance-sampling weights and indices through pre-sampling.
- Async PER pre-sampling: train_step_single_batch and
train_step_with_accumulation now pre-sample from the replay buffer
using a READ lock before acquiring the WRITE lock for GPU training.
This separates CPU sampling (~250μs) from GPU forward/backward (~3ms).
- Delete dead EpochPrefetcher: the binary (train_baseline_rl.rs) already
implements fold prefetching with background thread + mpsc channel +
GPU double-buffering, making the trainer's EpochPrefetcher redundant.
- Hyperopt DQN bounds: curiosity_weight min 0.0→0.01 so PSO can never
fully disable curiosity (which would re-gate the GPU collector).
10 files changed, -195 net lines. 2497 tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New standalone monitoring_service that bridges Prometheus → gRPC,
enabling `fxt train monitor` and web-gateway WS streaming for
all training jobs (CI and service-managed).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The auto-detect heuristic used cpus/2 ("smart cap"), designed for
CPU-bound workloads. DQN/PPO trials are GPU-bound — each rayon
thread submits CUDA kernels and waits on cudaDeviceSynchronize(),
using minimal CPU. cpus-1 is the correct cap.
On L40S-1-48G (8 vCPU): 3 threads → 7 threads (2.3× more trials).
Also bumps CI CPU limit 7500m→8000m to expose all 8 cores.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The batch span processor needs a tokio runtime for gRPC transport and
periodic flush. Async services already have one via #[tokio::main], but
sync training binaries (hyperopt, train, evaluate) don't.
Previous approach (making binaries async with #[tokio::main]) caused
"Cannot start a runtime from within a runtime" panics because the ML
crate's internal code creates its own tokio runtimes for block_on().
New approach: build_otel_tracer() detects runtime context via
Handle::try_current(). If absent, it creates a dedicated 1-worker
multi-thread runtime stored in a process-lifetime OnceLock. The worker
thread actively polls the OTLP batch export task.
Reverts training binaries to sync fn main() so internal runtime creation
(hyperopt adapters, DQN/PPO trainers) continues working as before.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All 6 training binaries (hyperopt_baseline_rl, hyperopt_baseline_supervised,
train_baseline_rl, train_baseline_supervised, evaluate_baseline,
evaluate_supervised) used sync fn main() but the OTLP batch exporter
requires a tokio runtime (tonic/hyper-util gRPC transport). This caused
an immediate panic on CI when OTEL_EXPORTER_OTLP_ENDPOINT was set.
Fix: #[tokio::main(flavor = "current_thread")] on all 6 binaries.
Also fix pre-existing clippy warnings (shadow, let_underscore_must_use,
doc_markdown, cognitive_complexity, integer_division, unsafe_code).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace CPU-side argmax (to_vec1 + iter enumerate max_by) with
GPU-native argmax(D). Single u32 scalar extraction instead of
full Q-value vector transfer.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace &Device::Cpu with &self.device for input and target tensor
creation in Mamba2 hyperopt adapter. Avoids unnecessary CPU→GPU
transfer during hyperparameter optimization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stream-aware ensemble that runs models on separate CUDA streams
for true GPU-level parallelism. Falls back to rayon on CPU.
Uses CudaStreamPool for synchronization.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CUDA stream pool with CPU no-op fallback. Foundation for
StreamAwareEnsemble that runs models on separate CUDA streams.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use predict_raw() to collect raw GPU tensors from adapters. Stack,
sigmoid, weighted-sum on GPU before single extraction. Falls back
to CPU path for adapters without tensor output.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Override predict_raw() in TGGN, TLOB, KAN, xLSTM, Diffusion adapters
to return raw GPU tensors. Enables GPU-side ensemble aggregation
instead of per-model CPU extraction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace clamped.to_vec1() CPU loop with GPU-native floor/ceil/frac
operations. Removes 3 tensor transfers per forward pass (N floats
down + 3*N up). All index computation now stays on GPU.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stack var_returns, var_residuals, mean_reward, var_reward into single
tensor before extraction. Uses broadcast_sub for GPU-native mean
centering instead of scalar round-trip.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backward-compatible trait extension. Default predict_raw() wraps
predict() result with tensor: None. Adapters can override to return
raw GPU tensors for GPU-side ensemble aggregation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The S3→PVC migration had several issues causing deploy failures:
- Stale binary-writer pod from previous failed deploys blocked new ones
(terminated pods can't be updated via kubectl apply)
- Services were applied BEFORE binaries written to PVC, so first deploy
or empty PVC caused pods to crash (binary not found)
- GPU overlay files in services/ dir were auto-applied by kubectl apply,
referencing nonexistent S3/minio secrets and PVCs
- Training job template mounted foxhunt-binaries PVC but training runs
on ci-training node — RWO PVC is bound to foxhunt node
Fixes:
- Reorder deploy: write binaries → apply manifests → rollout restart
- Clean up stale writer pods before creating new ones
- Move GPU overlays to gpu-overlays/ (manual apply only), update to PVC
- Add training-binaries PVC for GPU node, best-effort population
- Training job template uses initContainer to copy from training PVC
- set -e for critical path, set +e for optional training binary copy
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-param gradient norm extraction loop with batched
Tensor::stack pattern. Single GPU→CPU sync instead of one per
parameter tensor.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-param calculate_gradient_norm() loop with batched
Tensor::stack pattern. Single GPU→CPU sync instead of one per
parameter tensor.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-batch loss.to_vec0() in TFT training/validation loops
with GPU tensor accumulation. Single extraction per epoch + NaN guard
every 100 batches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-batch .to_dtype(F64).to_scalar() in Liquid train/validate
with GPU tensor accumulation. Single extraction per epoch + NaN guard
every 100 batches. Removes ~100-500 GPU→CPU syncs per epoch.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-batch loss.to_scalar() in TLOB train_epoch/validate_epoch
with GPU tensor accumulation. Single extraction per epoch + NaN guard
every 100 batches.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The before_script was copying training binaries to ${CI_PROJECT_DIR}/bin/
which collides with the git checkout bin/fxt/ directory. chmod +x on the
fxt subdirectory fails with "Operation not permitted", crashing the job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
