Pipeline #79 failed instantly (0 jobs created) - likely transient
GitLab issue. This empty commit triggers a clean pipeline #81.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The tensor-based log_probs() had an inverted sign on the squared
difference term: .sub(&(x * -0.5)) = +0.5*x² instead of -0.5*x².
This caused actions far from the mean to get higher log probabilities.
Also fix test assertions: continuous log probability densities CAN
be positive (when σ is small and action is near mean), unlike
discrete log probabilities which are always <= 0.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
K8s Job dispatch from ml_training_service with Rust sidecar uploader,
S3 artifact storage, and model promotion with fxt CLI approval gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy was using underscore names (trading_service) but K8s deployments
use hyphens (trading-service), causing 7/8 services to silently fail.
Uses explicit image_repo:deployment:container mapping table.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The deploy pod runs inside Kapsule but the kubeconfig points to the
external API URL (api.k8s.fr-par.scw.cloud:6443). Due to pod CIDR
overlap with the egress path, the connection is reset by peer. Sed
rewrites the server URL to kubernetes.default.svc (in-cluster endpoint)
while keeping the scw exec plugin for token auth.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The scw CLI used by the kubeconfig exec plugin (get-token) needs
SCW_ACCESS_KEY, SCW_SECRET_KEY, SCW_DEFAULT_PROJECT_ID, and
SCW_DEFAULT_ORGANIZATION_ID environment variables. Falls back to
project ID for org ID (common for single-org Scaleway accounts).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a parent/child GitLab CI pipeline for ML model training:
- Generator script produces per-model hyperopt/train/evaluate jobs
- Parent pipeline (.gitlab-ci-training.yml) with manual trigger
- NFS-backed ReadWriteMany PVC for shared training outputs
- Hyperopt params wired into training binaries (DQN, PPO, TFT, Mamba2)
- Shared DBN loader eliminates duplicate code across hyperopt adapters
- Supervised hyperopt unified to DBN data (was parquet-only)
Pipeline: hyperopt (4 models) → train (10 models) → evaluate ensemble
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy was silently failing — bitnami/kubectl image lacks scw CLI
needed by Kapsule kubeconfig credential plugin. Now uses infra-runner
(has scw), installs kubectl inline until image is rebuilt. Adds
cluster-info check before deploy and proper error reporting per service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training image was recompiling from scratch inside Kaniko (~16+ min)
without sccache. Now training binaries are built in compile-services
with --features ml/cuda and PVC sccache (warm cache from service
binaries), then packaged into a CUDA runtime image (~30s).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Delete 22 dead/placeholder/broken example files (-3,489 lines code)
- Delete 4 tracked CSV files (-1.1M lines, were accidentally committed)
- Move baseline training data default from data/cache/ to test_data/
- Update 5 unified binary defaults, gitignore, k8s upload comment, docs
- Consolidate all training data under test_data/futures-baseline/
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the build pipeline: one compile-services job builds all 8 service
binaries with PVC-backed sccache, saves as artifacts. Then 9 Kaniko jobs
just package pre-built binaries into slim runtime images (~30s each).
Before: 9 parallel Kaniko jobs each doing full cargo build --release
(~20min each, no sccache, 9x duplicated dep compilation)
After: 1 compile job with sccache (~5min cached) + 9 package jobs (~30s)
- Add compile stage between test and build
- Add Dockerfile.runtime (minimal debian + pre-built binary)
- Add Dockerfile.web-gateway-runtime (Node dashboard + pre-built binary)
- Keep Dockerfile.training via Kaniko (needs CUDA dev image for H100)
- Remove all SCCACHE_BUCKET build-args from service builds
- Use dir:// context for Kaniko (only sends build-out/ dir, not full repo)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The check stage ran cargo check + clippy (~6min) before tests, but
the test stage compiles the same workspace anyway. Skip check with
an echo pass-through to save ~6min per pipeline. Easy to uncomment
when clippy enforcement is needed (e.g., before releases).
Also adds sccache stats to the test job for cache monitoring.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The SCCACHE_BUCKET, SCCACHE_REGION, and SCCACHE_ENDPOINT CI variables
were deleted from GitLab settings. The .rust-base override is no longer
needed — sccache now uses SCCACHE_DIR (PVC) by default.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mass cleanup of crates/ml/:
- Delete 121 dead/broken/superseded example files (-44,098 lines)
- Delete 6 QAT test files (-4,201 lines) — QAT is disabled at runtime
(trainer.rs falls back to FP32 with warning)
- Delete 2 stale markdown files in examples/
- Delete orphaned src/bin/train_tft.rs (unimplemented stub)
- Clean up Cargo.toml: remove stale [[example]] entries, add missing ones
- Fix stale binary references in log_size_test.rs
- Add infra consistency test (35 checks across Dockerfile/train.sh/Cargo.toml)
Remaining examples (7): train_baseline_rl, train_baseline_supervised,
evaluate_baseline, hyperopt_baseline_rl, hyperopt_baseline_supervised,
download_baseline, cuda_test
All 2390 lib tests pass. All examples compile. 35/35 infra checks pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Widen observer checkpoint test bounds from 4σ to 5σ — random
calibration data can exceed 4.0 with 100 batches of normal(0,1)
- Override SCCACHE_BUCKET="" in .rust-base so check/test jobs use
PVC-backed SCCACHE_DIR instead of S3
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create dedicated ci-build node pool (L4-1-24G, €0.75/hr) replacing
H100 (€3.50+/hr) for CI builds. H100 reserved for training only.
- Add ci-build pool to Terraform (L4-1-24G, scale-to-zero, fr-par-2)
- Update GitLab Runner to target ci-build pool with L4-sized limits
- Change CUDA_COMPUTE_CAP from 90 (H100) to 89 (L4) in CI
- Dockerfile.training keeps CUDA_COMPUTE_CAP=90 for H100 training
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These two files survived the 20-file consolidation in 022036cb.
Both are now fully superseded:
- train_ppo.rs → train_baseline_rl --model ppo
- train_mamba2.rs → train_baseline_supervised --model mamba2
Also updates entrypoint-generic.sh usage examples to reference
the unified binaries (train_baseline_rl, train_baseline_supervised).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --cache=true --cache-repo to all 12 Kaniko builds
- Cache Docker layers in Scaleway CR (rg.fr-par.scw.cloud/foxhunt-ci/cache)
- Add Docker Hub auth to devcontainer + infra-runner prepare jobs
- First build populates cache; subsequent builds skip base image pulls
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create 20Gi sccache-pvc (scw-bssd) for fast local cache hits
- Mount at /mnt/sccache in CI build pods via runner helm values
- Use SCCACHE_DIR instead of SCCACHE_BUCKET for check/test jobs
- Kaniko builds keep S3 sccache (can't mount PVCs in Kaniko)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate 21 training examples into 2 binaries:
- train_baseline_rl (DQN, PPO with RL walk-forward)
- train_baseline_supervised (8 models via UnifiedTrainable)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add tx_cost_bps/tick_size/spread_ticks CLI args to tggn, xlstm, diffusion
- Subtract spread + commission from target returns during data prep
- Delete unused validate_model_parameters from train_mamba2_dbn
- Wire validate_training_batch into mamba2 pre-training validation
All 16 training examples now compile with zero warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add missing Mamba2Config fields (early_stopping_enabled, patience, min_delta, min_epochs)
- Replace unwrap_or on f64 (not Option) with direct field access
- Replace .unwrap() on path.to_str() with safe .ok_or_else()
- Fix DQN closure signature: 3 args (epoch, data, is_final), not 2
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Migrate tfstate backend from nl-ams to fr-par
- Add missing resources to TF: sccache bucket, foxhunt-ci registry,
grafana + prometheus DNS records
- Add infra-runner Dockerfile (terraform + terragrunt + scw + glab)
- Add CI jobs: infra-plan (MR), infra-apply (merge), drift-check (weekly)
- All IaC jobs run on gitlab pool (zero extra cost)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Artifact paths changed from /tmp/ to ${CI_PROJECT_DIR}/ for GitLab
k8s executor compatibility
- Drift check uses $? instead of PIPESTATUS (Alpine sh doesn't support
bash arrays)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Covers tfstate migration nl-ams→fr-par, resource import for
drift gaps (sccache bucket, foxhunt-ci registry, DNS records),
and GitLab CI/CD pipeline (plan on MR, apply on merge, weekly
drift detection).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sccache 0.8+ (opendal backend) requires AWS_REGION, not just
AWS_DEFAULT_REGION. Also set SCCACHE_REGION as belt-and-suspenders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scale-to-zero pools start with 0 nodes, so Terraform's default
wait_for_pool_ready=true always fails with "state warning, wants ready".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds one-click remote dev on Kapsule with Claude Code, persistent
storage, and autoscale-to-zero. GP1-L (16 vCPU, 64GB) dev pool,
50Gi PVC for home directory, devcontainer image on SCW registry.
- .devcontainer/ (Dockerfile, devcontainer.json, pod-template, init-home)
- Terraform gpu-dev pool (GP1-L, 0→1 autoscaling)
- CI job to build devcontainer image (Kaniko → SCW registry)
- scripts/devpod-setup.sh for one-time provider config
- "Open in DevPod" link: https://devpod.sh/open#ssh://git@100.90.76.85:2222/root/foxhunt.git
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The replace_all for --insecure-registry left blank lines in the YAML
multi-line scalars, splitting each kaniko command in two. GitLab only
executed the first half (--context + --dockerfile), missing --destination.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Builds .devcontainer/Dockerfile with Kaniko, pushes to internal
GitLab registry. Triggers on .devcontainer/ changes or manual run.
Authenticates to both internal registry and Scaleway CR (for base image pull).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Configures DevPod kubernetes provider, creates dev-home PVC,
verifies cluster access. Run once on developer laptop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GP1-L (16 vCPU, 64GB), autoscaling 0→1, dedicated for dev pods.
Separate from CI (H100) and inference (L4) pools.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
