TFT create_model used TFTConfig::default() values for num_known_features(10)
and num_unknown_features(210) totaling 220, but input_dim was 51 from the
feature extractor. Set both explicitly: known=0, unknown=feature_dim.
S3 uploader now uses path-style requests (required for Scaleway S3) and
explicitly passes AWS credentials from env vars instead of relying on the
instance metadata credential provider (unavailable on Kapsule).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Switch output volume from PVC to emptyDir (sidecar uploads to S3)
- Map s3-credentials secret keys to AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY
(object_store crate expects standard AWS env var names)
- Fix data-dir path to include /futures-baseline subdirectory
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a CronJob that runs on the 1st of every month at 03:00 UTC:
1. Reads current PAT from K8s Secret (gitlab-pat)
2. Calls GitLab rotate API (atomic: creates new, revokes old)
3. Updates K8s Secret with new token (3 retries)
4. Prints token to logs as recovery fallback if update fails
RBAC scoped to only get/patch the gitlab-pat secret.
Runs on gitlab node pool (zero GPU cost).
Manual trigger: kubectl -n foxhunt create job <name> --from=cronjob/gitlab-pat-rotation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The e2e test crate uses pre-generated proto code (no build.rs).
Missing stubs for: ReportJobCompletion, ListPendingPromotions,
ApprovePromotion, RejectPromotion + message types.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The CI runner pool has both L4 (sm_89) and H100 (sm_90) nodes.
Hardcoding CUDA_COMPUTE_CAP causes PTX mismatches when sccache
serves objects compiled for the wrong architecture.
- Detect compute capability via nvidia-smi at job start
- Partition sccache into /mnt/sccache/sm_89 and sm_90
- Each GPU type maintains its own warm cache
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sccache PVC holds CUDA kernel objects compiled for sm_89 (L4).
After moving the runner pool to H100 (sm_90), these produce
CUDA_ERROR_INVALID_PTX at runtime. Nuke the cache so candle
recompiles kernels for the correct compute capability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extend ml_training_service to dispatch GPU training jobs as K8s batch/v1
Jobs, collect results via a Rust sidecar uploader, and support model
promotion with operator approval via fxt CLI.
- K8s dispatcher creates Jobs on gpu-training pool with native sidecar
- training_uploader crate: watches DONE/FAILED marker, uploads to S3,
reports completion via ReportJobCompletion gRPC
- PromotionManager compares metrics, queues better models for approval
- 4 new proto RPCs: ReportJobCompletion, ListPendingPromotions,
ApprovePromotion, RejectPromotion
- fxt commands: train start, model list/approve/reject
- Training binaries write DONE/FAILED markers + metrics.json
- Dockerfile, K8s job template, and CI pipeline updated
- StartTraining gracefully falls back to in-process when outside K8s
- 27 new tests (16 service + 11 promotion), 141 total service tests pass
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pipeline #79 failed instantly (0 jobs created) - likely transient
GitLab issue. This empty commit triggers a clean pipeline #81.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The tensor-based log_probs() had an inverted sign on the squared
difference term: .sub(&(x * -0.5)) = +0.5*x² instead of -0.5*x².
This caused actions far from the mean to get higher log probabilities.
Also fix test assertions: continuous log probability densities CAN
be positive (when σ is small and action is near mean), unlike
discrete log probabilities which are always <= 0.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
K8s Job dispatch from ml_training_service with Rust sidecar uploader,
S3 artifact storage, and model promotion with fxt CLI approval gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy was using underscore names (trading_service) but K8s deployments
use hyphens (trading-service), causing 7/8 services to silently fail.
Uses explicit image_repo:deployment:container mapping table.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The deploy pod runs inside Kapsule but the kubeconfig points to the
external API URL (api.k8s.fr-par.scw.cloud:6443). Due to pod CIDR
overlap with the egress path, the connection is reset by peer. Sed
rewrites the server URL to kubernetes.default.svc (in-cluster endpoint)
while keeping the scw exec plugin for token auth.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The scw CLI used by the kubeconfig exec plugin (get-token) needs
SCW_ACCESS_KEY, SCW_SECRET_KEY, SCW_DEFAULT_PROJECT_ID, and
SCW_DEFAULT_ORGANIZATION_ID environment variables. Falls back to
project ID for org ID (common for single-org Scaleway accounts).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a parent/child GitLab CI pipeline for ML model training:
- Generator script produces per-model hyperopt/train/evaluate jobs
- Parent pipeline (.gitlab-ci-training.yml) with manual trigger
- NFS-backed ReadWriteMany PVC for shared training outputs
- Hyperopt params wired into training binaries (DQN, PPO, TFT, Mamba2)
- Shared DBN loader eliminates duplicate code across hyperopt adapters
- Supervised hyperopt unified to DBN data (was parquet-only)
Pipeline: hyperopt (4 models) → train (10 models) → evaluate ensemble
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Deploy was silently failing — bitnami/kubectl image lacks scw CLI
needed by Kapsule kubeconfig credential plugin. Now uses infra-runner
(has scw), installs kubectl inline until image is rebuilt. Adds
cluster-info check before deploy and proper error reporting per service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Training image was recompiling from scratch inside Kaniko (~16+ min)
without sccache. Now training binaries are built in compile-services
with --features ml/cuda and PVC sccache (warm cache from service
binaries), then packaged into a CUDA runtime image (~30s).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Delete 22 dead/placeholder/broken example files (-3,489 lines code)
- Delete 4 tracked CSV files (-1.1M lines, were accidentally committed)
- Move baseline training data default from data/cache/ to test_data/
- Update 5 unified binary defaults, gitignore, k8s upload comment, docs
- Consolidate all training data under test_data/futures-baseline/
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the build pipeline: one compile-services job builds all 8 service
binaries with PVC-backed sccache, saves as artifacts. Then 9 Kaniko jobs
just package pre-built binaries into slim runtime images (~30s each).
Before: 9 parallel Kaniko jobs each doing full cargo build --release
(~20min each, no sccache, 9x duplicated dep compilation)
After: 1 compile job with sccache (~5min cached) + 9 package jobs (~30s)
- Add compile stage between test and build
- Add Dockerfile.runtime (minimal debian + pre-built binary)
- Add Dockerfile.web-gateway-runtime (Node dashboard + pre-built binary)
- Keep Dockerfile.training via Kaniko (needs CUDA dev image for H100)
- Remove all SCCACHE_BUCKET build-args from service builds
- Use dir:// context for Kaniko (only sends build-out/ dir, not full repo)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The check stage ran cargo check + clippy (~6min) before tests, but
the test stage compiles the same workspace anyway. Skip check with
an echo pass-through to save ~6min per pipeline. Easy to uncomment
when clippy enforcement is needed (e.g., before releases).
Also adds sccache stats to the test job for cache monitoring.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The SCCACHE_BUCKET, SCCACHE_REGION, and SCCACHE_ENDPOINT CI variables
were deleted from GitLab settings. The .rust-base override is no longer
needed — sccache now uses SCCACHE_DIR (PVC) by default.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mass cleanup of crates/ml/:
- Delete 121 dead/broken/superseded example files (-44,098 lines)
- Delete 6 QAT test files (-4,201 lines) — QAT is disabled at runtime
(trainer.rs falls back to FP32 with warning)
- Delete 2 stale markdown files in examples/
- Delete orphaned src/bin/train_tft.rs (unimplemented stub)
- Clean up Cargo.toml: remove stale [[example]] entries, add missing ones
- Fix stale binary references in log_size_test.rs
- Add infra consistency test (35 checks across Dockerfile/train.sh/Cargo.toml)
Remaining examples (7): train_baseline_rl, train_baseline_supervised,
evaluate_baseline, hyperopt_baseline_rl, hyperopt_baseline_supervised,
download_baseline, cuda_test
All 2390 lib tests pass. All examples compile. 35/35 infra checks pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Widen observer checkpoint test bounds from 4σ to 5σ — random
calibration data can exceed 4.0 with 100 batches of normal(0,1)
- Override SCCACHE_BUCKET="" in .rust-base so check/test jobs use
PVC-backed SCCACHE_DIR instead of S3
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Create dedicated ci-build node pool (L4-1-24G, €0.75/hr) replacing
H100 (€3.50+/hr) for CI builds. H100 reserved for training only.
- Add ci-build pool to Terraform (L4-1-24G, scale-to-zero, fr-par-2)
- Update GitLab Runner to target ci-build pool with L4-sized limits
- Change CUDA_COMPUTE_CAP from 90 (H100) to 89 (L4) in CI
- Dockerfile.training keeps CUDA_COMPUTE_CAP=90 for H100 training
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These two files survived the 20-file consolidation in 022036cb.
Both are now fully superseded:
- train_ppo.rs → train_baseline_rl --model ppo
- train_mamba2.rs → train_baseline_supervised --model mamba2
Also updates entrypoint-generic.sh usage examples to reference
the unified binaries (train_baseline_rl, train_baseline_supervised).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --cache=true --cache-repo to all 12 Kaniko builds
- Cache Docker layers in Scaleway CR (rg.fr-par.scw.cloud/foxhunt-ci/cache)
- Add Docker Hub auth to devcontainer + infra-runner prepare jobs
- First build populates cache; subsequent builds skip base image pulls
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create 20Gi sccache-pvc (scw-bssd) for fast local cache hits
- Mount at /mnt/sccache in CI build pods via runner helm values
- Use SCCACHE_DIR instead of SCCACHE_BUCKET for check/test jobs
- Kaniko builds keep S3 sccache (can't mount PVCs in Kaniko)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate 21 training examples into 2 binaries:
- train_baseline_rl (DQN, PPO with RL walk-forward)
- train_baseline_supervised (8 models via UnifiedTrainable)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add tx_cost_bps/tick_size/spread_ticks CLI args to tggn, xlstm, diffusion
- Subtract spread + commission from target returns during data prep
- Delete unused validate_model_parameters from train_mamba2_dbn
- Wire validate_training_batch into mamba2 pre-training validation
All 16 training examples now compile with zero warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add missing Mamba2Config fields (early_stopping_enabled, patience, min_delta, min_epochs)
- Replace unwrap_or on f64 (not Option) with direct field access
- Replace .unwrap() on path.to_str() with safe .ok_or_else()
- Fix DQN closure signature: 3 args (epoch, data, is_final), not 2
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Migrate tfstate backend from nl-ams to fr-par
- Add missing resources to TF: sccache bucket, foxhunt-ci registry,
grafana + prometheus DNS records
- Add infra-runner Dockerfile (terraform + terragrunt + scw + glab)
- Add CI jobs: infra-plan (MR), infra-apply (merge), drift-check (weekly)
- All IaC jobs run on gitlab pool (zero extra cost)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Artifact paths changed from /tmp/ to ${CI_PROJECT_DIR}/ for GitLab
k8s executor compatibility
- Drift check uses $? instead of PIPESTATUS (Alpine sh doesn't support
bash arrays)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Covers tfstate migration nl-ams→fr-par, resource import for
drift gaps (sccache bucket, foxhunt-ci registry, DNS records),
and GitLab CI/CD pipeline (plan on MR, apply on merge, weekly
drift detection).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sccache 0.8+ (opendal backend) requires AWS_REGION, not just
AWS_DEFAULT_REGION. Also set SCCACHE_REGION as belt-and-suspenders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Scale-to-zero pools start with 0 nodes, so Terraform's default
wait_for_pool_ready=true always fails with "state warning, wants ready".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
