Aliased re-exports avoid collision with existing types::Price/Quantity/Money
during migration. Will rename to canonical names in Phase 6 cleanup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Ratio arithmetic (Add, Sub, Mul, Div) now clamps overflow to
f64::MAX/f64::MIN instead of producing Inf from finite inputs
- Cross-type Price*Quantity→Money and Money/Quantity→Price use
saturating i128→i64 conversion via clamp instead of silent truncation
- Added clamp_finite() and saturating_i128_to_i64() helper functions
- Added 9 edge-case tests: 89/89 passing, clippy clean
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
compile-training now needs compile-services. Both request 28 vCPUs but
the ci-compile-cpu pool (POP2-32C-128G) has only 32 vCPUs total.
Build-services jobs start as soon as compile-services finishes while
compile-training runs; no wall-clock penalty.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
compile-training was stuck on GPU node autoscale (L4 pool) but only
needs CUDA stubs for candle-kernels PTX generation, not a real GPU.
Switch to .rust-base-cpu (ci-compile-cpu pool) with ci-builder image
override for CUDA stubs. Unify pool labels — all compilation on CPU.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add Serialize/Deserialize to all four financial types
- Ratio::ln returns 0.0 for non-positive inputs (prevents NaN leak)
- Ratio::exp clamps to f64::MAX on overflow (prevents Inf leak)
- Ratio::powi clamps to f64::MAX/MIN on overflow
- Add 4 invariant-preservation tests (80 total)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 1 of GPU pipeline: eliminates per-bar/per-step heap allocations
by pre-uploading training data as GPU tensors at epoch start.
- New cuda_pipeline module with DqnGpuData + PpoGpuData structs
- DQN: bulk upload [N,51] features + [N,4] targets, narrow() slices
- PPO: bulk upload [N,state_dim] states, zero-copy per-step access
- VRAM safety guards (2GB limit) and estimation utilities
- 8 new tests, 2411 total ML tests passing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
12-worktree phased migration: foundation types → core crates →
ML boundary → services → peripheral → cleanup. Full code for
Phase 1 (Price/Quantity/Money/Ratio in common/financial_types),
migration patterns for Phases 2-6.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate competing Price/Quantity implementations into canonical
i64 fixed-point types in common/. Adds Money and Ratio newtypes.
Phased migration plan across 5 phases using parallel worktrees.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all `.to_string().parse::<f32/f64>()` patterns with
`num_traits::ToPrimitive` methods (`.to_f32()`, `.to_f64()`).
Each string roundtrip heap-allocated per conversion — fatal in
DQN hot loop (300K+ bars × epochs). Decimal stays as canonical
financial type; conversions happen at GPU/float boundaries only.
Also fixes blocking_read() in async context (risk_integration.rs).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
P2-C reversal/cash warns fire thousands of times per hyperopt trial
during backtest simulation. Also removes stale Kelly patch file (already
applied).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These fire every training step when portfolio isn't populated yet —
extremely noisy during hyperopt with parallel trials.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous "auto-detect" logic forced CPU which was wrong — GPU is
faster for forward/backward even with parallel trials (DQN/PPO use
<350MB of 24GB VRAM across 7 threads).
Changes:
- Remove --device flag, always require CUDA GPU
- Add DQNTrainer::new_with_device() to share CUDA context across trials
- Propagate hyperopt device to internal DQN trainer (was ignoring it)
- PPO/DQN adapters error on missing GPU instead of silent CPU fallback
- Downgrade batch-size clamping from warn to debug
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The compile-services stage outputs binaries to build-out/services/ but
the web-gateway Dockerfile still referenced the old build-out/ path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
POP2-32C-128G pool now allows up to 4 concurrent nodes for parallel
CI compilation (services + training + builder image builds).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add CI_BUILDER_CPU_IMAGE variable and build-ci-builder-cpu prepare job
- Add .rust-base-cpu template (no CUDA_COMPUTE_CAP)
- Split compile-services into compile-services (CPU) and compile-training (CUDA)
- Split .kaniko-base into .kaniko-service-base and .kaniko-training-base
- Service builds use build-out/services/, training uses build-out/training/
- Each compile job has conditional changes: rules for its source paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
infra-apply needs infra-plan, but infra-plan only runs when infra/**
changes. API/web-triggered pipelines fail to create because the
dependency doesn't exist. Adding optional: true allows infra-apply
to run without infra-plan for manual triggers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Based on rust:1.89-slim-bookworm (~2-3GB vs ~8GB CUDA devel).
Same toolchain: mold 2.35, protoc 28.3, sccache 0.10, clang, lld.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DQN/PPO networks are tiny (3 layers × 128 neurons). Running parallel
hyperopt on GPU wastes cores because CUDA context serializes across
threads — 5 trials on L4 only used 2000m of 6000m requested CPU.
Changes:
- Add --device flag to hyperopt_baseline_rl (auto/cpu/cuda)
- Auto mode forces CPU for parallel runs (no CUDA contention)
- CPU mode uses all available cores (no 2-core reserve)
- Add with_device() builder to DQN/PPO hyperopt trainers
- Downgrade "portfolio value <= 0" and GPU utilization warnings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Portfolio value <= 0 during early exploration is expected behavior,
not a warning. GPU memory utilization advisory is informational.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kapsule doesn't allow node-labels in kubelet_args. Instead, use the
auto-assigned k8s.scaleway.com/pool-name label that every node gets.
- Revert kubelet_args from Terraform (not supported)
- Switch runner node_selector: pool→k8s.scaleway.com/pool-name
- Update CI pipeline KUBERNETES_NODE_SELECTOR to match
- Helm upgraded both runners (CPU + GPU)
No more manual kubectl label after node scale-up.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Nodes from autoscaler now get pool=<name> labels automatically via
kubelet_args, eliminating the need to manually label nodes after
scale-up events.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace stub approve_promotion and reject_promotion gRPC handlers with
real implementations that call PromotionManager.approve() and .reject().
- approve_promotion: removes from pending, promotes to active model map
- reject_promotion: removes from pending with operator-supplied reason
- Input validation: empty model_id returns INVALID_ARGUMENT
- Error handling: nonexistent model_id returns success=false with message
- Add 4 tests covering approve, reject, and not-found error paths
- Add #[cfg(test)] active_models_for_test() accessor on PromotionManager
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the stub list_pending_promotions gRPC handler with a real
implementation that queries PromotionManager::list_pending() and maps
each PendingModel to the proto PendingPromotion message. Adds a unit
test verifying the field mapping from domain type to proto type.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the stub report_job_completion gRPC handler with a real
implementation that:
- Validates job_id as UUID upfront
- On failure: updates DB status to Failed (best-effort), returns "failed"
- On success: updates DB status to Completed, looks up model_type and
symbol from child_jobs table, registers with PromotionManager, and
returns the actual promotion status string
Also adds JobSpawner::get_job_by_id() to fetch individual child jobs
from PostgreSQL, and two new tests covering promotion status mapping
and the PromotionManager integration path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Changed from non-existent scw-bssd-nfs to scw-bssd (RWO).
Training output is single-pod, doesn't need ReadWriteMany.
PVC deployed to foxhunt namespace (WaitForFirstConsumer).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The original migration 046 only allowed DQN, PPO, MAMBA-2, TFT, TLOB.
Now includes TGGN, LIQUID, KAN, XLSTM, DIFFUSION.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Background tokio task polls JobSpawner.get_next_pending_job() every 5s,
marks found jobs as Running, builds TrainingJobParams, and dispatches
to K8s via K8sDispatcher. On dispatch failure the job is marked Failed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add JobSpawner as the highest-priority dispatch path in start_training.
When job_spawner is available, training requests are persisted to
PostgreSQL via spawn_batch() and a batch ID is returned immediately.
The queue consumer (Task 3) will later poll for pending jobs and
dispatch them to K8s.
Priority order: JobSpawner (DB) → K8sDispatcher (direct) → orchestrator (in-process).
Also adds test_start_training_model_binary_mapping unit test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL (DQN/PPO) stays on L4-1-24G — CPU-bound, low VRAM (<1GB).
Supervised stays on L40S-1-48G — GPU-bound, high VRAM.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All training jobs (RL + supervised) now use ci-training pool (L40S-1-48G)
instead of splitting RL→L4, supervised→L40S. With parallel hyperopt,
RL benefits from the L40S's more powerful GPU and same 8 vCPU.
Simplifies infrastructure: can potentially decommission L4 pool.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire JobSpawner into the gRPC service struct so that training jobs can
be persisted to PostgreSQL before being dispatched to K8s. This is the
first step toward a durable job queue that survives pod restarts.
- Add `job_spawner: Option<Arc<JobSpawner>>` to MLTrainingServiceImpl
- Extend `new()` constructor to accept the spawner parameter
- Add `DatabaseManager::pg_pool()` accessor for cheap PgPool cloning
- Construct JobSpawner in main.rs and pass `Some(job_spawner)` to service
- Add compile-time test verifying the field exists
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RL hyperopt runs parallel PSO trials across CPU cores. Previous 2000m
request underutilized the L4's 8 vCPUs. Bumped to 6000m/7800m so
hyperopt_baseline_rl --parallel 0 can saturate all cores.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runner defaults (16 CPU, 32Gi) are tuned for compile jobs but too high
for Kaniko image packaging. Only 2 build pods fit on the 32-core node,
queuing the remaining 7 with FailedScheduling: Insufficient cpu.
Set kaniko builds to 1000m/1Gi — all 9 can now run concurrently.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L and PRO2-L both have 0/0 quota in fr-par-2. POP2-HC-32C-64G had
insufficient root disk (20GB). POP2-32C-128G with 100GB SBS root works.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split runners into CPU (compile/build/deploy) and GPU (training):
- CPU runner: tags kapsule,rust,docker → ci-compile-cpu (32 vCPU, 64GB)
- GPU runner: tags kapsule,gpu → ci-compile (L4) or ci-training (L40S)
Changes:
- Add ci-compile-cpu Terraform pool (POP2-HC-32C-64G, autoscale 0-1)
- Set CARGO_BUILD_JOBS=30 to use all CPUs for Rust compilation
- Increase CPU request to 28000m/31000m (was 7000m/7800m on L4)
- Add docker tag to deploy/infra jobs for explicit CPU runner routing
- Update CI header with new pool routing documentation
Cost: EUR 0.85/h (vs EUR 1.1/h for L4) — cheaper AND 4x faster compile
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Compile only needs CUDA stubs (baked into ci-builder) + CUDA_COMPUTE_CAP
for candle-kernels PTX generation. No nvidia-smi or real GPU needed.
- Hardcode CUDA_COMPUTE_CAP=89 (L4) in .rust-base variables
- Remove KUBERNETES_RUNTIME_CLASS_NAME from .rust-base
- compile-services: skip LD_LIBRARY_PATH stripping and nvidia-smi
- test: graceful fallback when nvidia-smi unavailable
This unblocks compile on nodes without nvidia RuntimeClass handler
and opens the door to using CPU-only nodes for faster compilation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GitLab Runner 18.9 expects KUBERNETES_NODE_SELECTOR_* values in
"key=value" format. Changed from bare values to "pool=services"
and "pool=ci-training".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Runner default nodeSelector changed from k8s.scaleway.com/pool-name to
simple pool=<name> label. CI variables now use correct format:
KUBERNETES_NODE_SELECTOR_pool: <value> (lowercase suffix = label key).
Fixes deploy job scheduling failure (wrong label format caused pods to
land on ci-compile instead of services pool).
Also fixes .train-rl-base tag from kapsule-rl (no matching runner) to
kapsule+gpu (matches existing runner).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
