Add nginx server block for api.fxhnt.ai with grpc_pass to the
api-gateway ClusterIP service. Add Scaleway DNS A record via Terraform
module. Fix fxt gRPC client to enable TLS when connecting to https://
endpoints (tonic requires explicit ClientTlsConfig).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --hyperopt, --trials, and --parallel flags to `fxt train start` so
users can dispatch PSO hyperparameter optimization jobs alongside
regular training. The mode=hyperopt tag propagates through the service
layer which selects the correct binary (hyperopt_baseline_rl for DQN/PPO,
hyperopt_baseline_supervised for all others) and builds hyperopt-specific
CLI arguments via a new extra_args field on TrainingJobParams.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove artificial swarm_size cap from plan_hyperopt() that limited
concurrent trials to n_particles (default 20). Now concurrency is
purely VRAM-driven with a hardware cap of 128 threads.
optimize_parallel() auto-scales n_particles to match GPU budget:
- L4 24GB: ~65 concurrent DQN trials (was 20)
- H100 80GB: 128 concurrent DQN trials (was 20)
- CPU/small GPU: falls back to configured n_particles
max_trials scales proportionally to ensure 3+ PSO iterations
for convergence with larger swarms.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Auto-detect now considers both CPU count and GPU VRAM when choosing
concurrent trial count. Prevents OOM on smaller GPUs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
optimize_parallel() now uses HardwareBudget::plan_hyperopt() to compute
max concurrent trials based on model VRAM footprint. Includes ramp-up
(start at half concurrency), VRAM watchdog (check free memory before
spawning), and graceful degradation to sequential.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy job now triggers on infra/k8s/** and .gitlab-ci.yml changes,
not just source code changes. Ensures K8s manifest updates get applied.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Computes optimal (concurrency, batch_size_bounds) based on model memory
footprint vs detected GPU VRAM. Scales from CPU-only (sequential) through
H100 80GB (full swarm parallel). 20% safety margin for CUDA allocator.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PSO now controls batch_size within [64, 4096] range. HardwareBudget
adjusts upper bound based on detected VRAM. The .min(160.0) clamp was
silently overriding PSO's choices on any GPU larger than 3050 Ti.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- init_observability now reads LOG_DIR env var for log directory (falls back
to relative logs/ path). Gracefully disables file logging if directory
cannot be created instead of failing the entire observability init.
- Add ServiceAccount, Role, and RoleBinding for ml-training-service to
create/get/list/watch/delete batch/v1 Jobs (K8s training dispatch).
- Add LOG_DIR=/app/logs env to ml-training-service, trading-service, and
backtesting-service deployment YAMLs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Parallel hyperopt trials, VRAM-aware batch bounds, 5-layer OOM protection.
Scales from L4 through H100.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align test assertions with the production job spec changes from b5fca4d1:
- Binary paths use /binaries/ prefix
- Args use --symbol=X format
- Config uses runtime_image and binaries_bucket fields
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace separate training/uploader images with single runtime_image
- Add fetch-binaries initContainer (rclone from S3 binaries bucket)
- Switch to emptyDir for output and binaries (no output PVC needed)
- Add Cilium CNI toleration for fresh scale-from-zero nodes
- Extract symbol from file_path last component in data_source
- Increase active_deadline_seconds to 6 hours for hyperopt jobs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Single model_memory_mb field now serves as the sole source of truth for
base model memory. Removes the redundant base_model_memory_mb field,
legacy dual-path branches in calculate_optimal_batch_size() and
max_safe_batch_size(), and the associated legacy test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
AutoBatchSizer was computing an "optimal" batch_size (128 on L4) and
overriding the PSO-chosen value. On an L4 24GB this meant 3.5% VRAM
utilization per trial. Now the trainer only clamps if the batch_size
would actually OOM, letting PSO explore the full [64, 4096] range.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Revert needs: [] back to needs: compile-training (optional). CI training
jobs should wait for fresh binaries when compile runs in the same
pipeline. Independent training via fxt CLI uses the gRPC path instead.
Keeps the writable bin dir fix (${CI_PROJECT_DIR}/bin/) and PATH export.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CI runner pods run as non-root — can't write to /usr/local/bin/.
Download to ${CI_PROJECT_DIR}/bin/ and prepend to PATH instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training jobs fetch binaries from S3 via rclone, not CI artifacts.
Remove needs: compile-training so training jobs can be triggered
independently without waiting for a compile stage.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The rclone --include "*.{rl,supervised,baseline,uploader}*" glob does
not expand brace patterns like bash — silently downloads zero files.
Replace with unfiltered sync (bucket only has training binaries) and
wildcard chmod.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- All services: maxSurge=0, maxUnavailable=1 (ReadWriteOnce PVCs
deadlock with maxSurge=1 — new pod can't attach volume while old
pod still holds it)
- ml-training-service: switch httpGet /health:8080 to tcpSocket:50053
(service crashes on log dir permission, never binds health port)
- GPU overlays: same strategy and probe fixes
- job-template.yaml: use generateName instead of placeholder name
(training-MODEL-TIMESTAMP fails k8s RFC 1123 validation)
- deploy job: increase rollout timeout 120s → 300s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
training-data-pvc is immutable (already bound with ReadWriteOnce),
and job-template.yaml has placeholder names (MODEL/TIMESTAMP) that
fail k8s validation. Only apply service deployments and binary cache
PVCs in the deploy job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The deploy job was using kubectl set image to stamp per-service Docker
images (api_gateway:<sha>, etc.) — these images no longer exist after
the S3 binary share migration. Replace with kubectl apply + rollout
restart, which picks up YAML changes and triggers initContainers to
fetch latest binaries from S3.
Delete redundant deploy-services job (merged into deploy).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaleway quota limits POP2-32C-128G to 1 instance. Reduce each compile
job from 28→14 CPUs so compile-services and compile-training run in
parallel on the same 32-vCPU node instead of sequentially.
Also fix data-acquisition-service health probe: service doesn't implement
gRPC health protocol, switch to tcpSocket probe on port 50057.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The foxhunt-runtime base image runs as uid 1000 (foxhunt user), but
new PVC volumes are owned by root. The initContainer needs root to
write the cached binary to the PVC. Main container still runs as
foxhunt (non-root) for security.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scaleway block storage minimum volume size is 1Gi. The 500Mi PVCs
were stuck in Pending state, blocking all service pod scheduling.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Web dashboard build now uses node:22-slim with inline rclone install
instead of a custom Kaniko-built image. This removes 1 Kaniko build
from the prepare stage, reducing pod scheduling contention.
Only 2 runtime images needed: CPU (foxhunt-runtime) and GPU
(foxhunt-training-runtime). Binary selection is via K8s command.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
