AutoBatchSizer was computing an "optimal" batch_size (128 on L4) and
overriding the PSO-chosen value. On an L4 24GB this meant 3.5% VRAM
utilization per trial. Now the trainer only clamps if the batch_size
would actually OOM, letting PSO explore the full [64, 4096] range.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Revert needs: [] back to needs: compile-training (optional). CI training
jobs should wait for fresh binaries when compile runs in the same
pipeline. Independent training via fxt CLI uses the gRPC path instead.
Keeps the writable bin dir fix (${CI_PROJECT_DIR}/bin/) and PATH export.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI runner pods run as non-root — can't write to /usr/local/bin/.
Download to ${CI_PROJECT_DIR}/bin/ and prepend to PATH instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training jobs fetch binaries from S3 via rclone, not CI artifacts.
Remove needs: compile-training so training jobs can be triggered
independently without waiting for a compile stage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The rclone --include "*.{rl,supervised,baseline,uploader}*" glob does
not expand brace patterns like bash — silently downloads zero files.
Replace with unfiltered sync (bucket only has training binaries) and
wildcard chmod.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- All services: maxSurge=0, maxUnavailable=1 (ReadWriteOnce PVCs
deadlock with maxSurge=1 — new pod can't attach volume while old
pod still holds it)
- ml-training-service: switch httpGet /health:8080 to tcpSocket:50053
(service crashes on log dir permission, never binds health port)
- GPU overlays: same strategy and probe fixes
- job-template.yaml: use generateName instead of placeholder name
(training-MODEL-TIMESTAMP fails k8s RFC 1123 validation)
- deploy job: increase rollout timeout 120s → 300s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
training-data-pvc is immutable (already bound with ReadWriteOnce),
and job-template.yaml has placeholder names (MODEL/TIMESTAMP) that
fail k8s validation. Only apply service deployments and binary cache
PVCs in the deploy job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The deploy job was using kubectl set image to stamp per-service Docker
images (api_gateway:<sha>, etc.) — these images no longer exist after
the S3 binary share migration. Replace with kubectl apply + rollout
restart, which picks up YAML changes and triggers initContainers to
fetch latest binaries from S3.
Delete redundant deploy-services job (merged into deploy).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaleway quota limits POP2-32C-128G to 1 instance. Reduce each compile
job from 28→14 CPUs so compile-services and compile-training run in
parallel on the same 32-vCPU node instead of sequentially.
Also fix data-acquisition-service health probe: service doesn't implement
gRPC health protocol, switch to tcpSocket probe on port 50057.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The foxhunt-runtime base image runs as uid 1000 (foxhunt user), but
new PVC volumes are owned by root. The initContainer needs root to
write the cached binary to the PVC. Main container still runs as
foxhunt (non-root) for security.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaleway block storage minimum volume size is 1Gi. The 500Mi PVCs
were stuck in Pending state, blocking all service pod scheduling.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Web dashboard build now uses node:22-slim with inline rclone install
instead of a custom Kaniko-built image. This removes 1 Kaniko build
from the prepare stage, reducing pod scheduling contention.
Only 2 runtime images needed: CPU (foxhunt-runtime) and GPU
(foxhunt-training-runtime). Binary selection is via K8s command.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Detailed step-by-step plan covering: S3 bucket creation, CI builder
updates, 3 base image Dockerfiles, CI pipeline rewiring, 8 cache PVCs,
all service deployment YAML updates, training job template, and cleanup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace per-service Docker images with generic base images + S3-based binary
distribution. CI compiles and uploads stripped binaries to S3. Pods fetch
binaries via initContainer with PVC cache fallback for trading resilience.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU experience collector only supported DQNAgentType::Standard, but
enable_regime_qnetwork defaults to true, creating RegimeConditional
agents. This silently fell back to CPU with a debug! message invisible
at INFO log level.
- Add primary_head() accessor to RegimeConditionalDQN (returns trending head)
- Extract weights from primary head for both collector init and weight sync
- Upgrade debug! to warn! for missing GPU collection prerequisites
- Add warn! to PPO for silent GPU skip when raw market data not uploaded
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The GPU experience collection kernels (dqn_experience_kernel.cu,
ppo_experience_kernel.cu) use cudarc::nvrtc::compile_ptx() at runtime.
Without libnvrtc.so the kernel compile fails silently and falls back to
CPU experience collection. Adding cuda-nvrtc-12-4 (~30MB) enables full
GPU-accelerated experience collection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both images now use identical cuda:12.4.1-cudnn-runtime base, so maintaining
two separate Dockerfiles and build jobs was wasteful. Single image contains
all 7 training binaries, halving registry storage and Kaniko build time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
candle's CUDA backend links against libcudnn.so.9 for kernel operations.
Switch training-rl base from cuda:12.4.1-runtime to cuda:12.4.1-cudnn-runtime.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These files were part of the Phase 2 cuda_pipeline but were untracked
and not included in previous commits. Required for compilation with
--features ml/cuda.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 3 of the CUDA pipeline: both trainers now take a GPU-first path
for experience collection (128×500 = 64K experiences per kernel launch,
zero CPU-GPU roundtrips per timestep) with automatic CPU fallback.
- DQN: upload features alongside targets, GPU collection branch before
CPU loop, gpu_batch_to_experiences() conversion into replay buffer
- PPO: set_raw_market_data() for CudaSlice upload, GPU collection
branch bypasses collect_rollouts + prepare_training_batch entirely,
gpu_batch_to_trajectory_batch() conversion with in-kernel GAE
- Configurable GPU batch sizes (gpu_n_episodes, gpu_timesteps_per_episode)
and trading params (initial_capital, avg_spread) via hyperparameters
- SAFETY training diagnostics downgraded from warn! to debug!
- 4 new batch index-math validation tests in cuda_pipeline
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move 10 shared device functions (gpu_random, leaky_relu, matvec_leaky_relu,
action_to_exposure, action_to_tx_cost, barrier_init/check/reset,
diversity_entropy, curiosity_inference) and shared constants from
dqn_experience_kernel.cu into a new common_device_functions.cuh header.
The DQN kernel now expects the common header to be prepended via NVRTC
source concatenation at compile time. DQN-specific functions
(q_forward_dueling, argmax_q) remain in the kernel file. This enables
the upcoming PPO kernel to reuse the same shared functions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Single monolithic kernel with in-kernel GAE, full portfolio simulation,
shared device functions header, 5-layer critic matching trainer exactly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
