Files

jgrusewski 6e339316cf feat(ml): add manually-triggered GitLab CI training pipeline

Adds a parent/child GitLab CI pipeline for ML model training:

- Generator script produces per-model hyperopt/train/evaluate jobs
- Parent pipeline (.gitlab-ci-training.yml) with manual trigger
- NFS-backed ReadWriteMany PVC for shared training outputs
- Hyperopt params wired into training binaries (DQN, PPO, TFT, Mamba2)
- Shared DBN loader eliminates duplicate code across hyperopt adapters
- Supervised hyperopt unified to DBN data (was parquet-only)

Pipeline: hyperopt (4 models) → train (10 models) → evaluate ensemble

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-02-26 09:04:58 +01:00

5.1 KiB

Raw Blame History

ML Training Pipeline Design

Overview

Manually-triggered GitLab CI pipeline for training the 10-model ML ensemble on 730 days of OHLCV-1m futures data. Runs on H100 GPU pool with configurable parallelism.

Trigger Variables

Variable	Default	Description
`SYMBOLS`	`ES.FUT`	Comma-separated symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT)
`MODELS`	`all`	`all`, `rl`, `supervised`, or comma-separated model names
`PHASE`	`full`	`hyperopt`, `train`, `eval`, or `full` (all three)
`MAX_PARALLEL`	`10`	GPU concurrency cap
`EPOCHS`	`50`	Training epochs per walk-forward fold
`HYPEROPT_TRIALS`	`20`	PSO trials per model
`RUN_ID`	auto	YYYYMMDD-HHMMSS or manual override

Model Groups

RL (2): DQN, PPO — via train_baseline_rl / hyperopt_baseline_rl
Supervised (8): TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion — via train_baseline_supervised / hyperopt_baseline_supervised
Hyperopt support: DQN, PPO, TFT, Mamba2 (other 6 use default configs)

Architecture

Pipeline Structure

.gitlab-ci-training.yml (manual trigger, parent pipeline)
  ├── prepare: generate-training-pipeline.sh
  │     → writes .training-generated.yml as artifact
  └── trigger: launches child pipeline from artifact
        ├── STAGE: hyperopt (parallel, max MAX_PARALLEL)
        ├── STAGE: train (parallel, needs hyperopt)
        └── STAGE: evaluate (single job, needs all training)

Why Generated Child Pipeline

GitLab CI lacks dynamic matrix jobs. A prepare job generates exactly the needed jobs based on MODELS × SYMBOLS, avoiding dead "skipped" jobs. Adding a model = one line in the generator.

Infrastructure

Training image: rg.fr-par.scw.cloud/foxhunt-ci/training:latest (built by main CI)
GPU pool: gpu-training (H100), Kubernetes autoscaler handles node provisioning
Input PVC: training-data-pvc at /data (read-only, futures-baseline data)
Output PVC: training-output-pvc at /output (read-write, checkpoints + results)

Output Directory Convention

/output/<RUN_ID>/
├── hyperopt/<model>/<symbol>/results.json
├── models/<model>/<symbol>/*.safetensors
├── models/<model>/<symbol>/norm_stats.json
└── eval/ensemble_report.json

Stage Details

Stage 1: Hyperopt (optional)

Per model × symbol, runs PSO hyperparameter optimization.

RL: hyperopt_baseline_rl --model {dqn|ppo} --trials $HYPEROPT_TRIALS --epochs 10 --data-dir /data/$SYMBOL --output /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json
Supervised: hyperopt_baseline_supervised --model {tft|mamba2} --trials $HYPEROPT_TRIALS --data-dir /data/$SYMBOL --output /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json
Models without hyperopt adapters (TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion) skip this stage.

Stage 2: Train

Per model × symbol, runs walk-forward training.

RL: train_baseline_rl --model {dqn|ppo} --epochs $EPOCHS --data-dir /data/$SYMBOL --output-dir /output/$RUN_ID/models/$MODEL/$SYMBOL --params-file /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json
Supervised: train_baseline_supervised --model $MODEL --epochs $EPOCHS --data-dir /data/$SYMBOL --output-dir /output/$RUN_ID/models/$MODEL/$SYMBOL --params-file /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json

--params-file is optional — if the file doesn't exist (no hyperopt ran), uses compiled defaults. CLI args override JSON params.

Stage 3: Evaluate

Single job after all training completes.

evaluate_baseline --models-dir /output/$RUN_ID/models --data-dir /data --output /output/$RUN_ID/eval/ensemble_report.json

Runs inference on held-out test windows, computes per-model and ensemble metrics (Sharpe, MaxDD, win rate, profit factor).

Parameter Passing: Hyperopt → Training

Add --params-file <path> optional arg to train_baseline_rl and train_baseline_supervised. Logic:

Parse CLI args as normal (with defaults)
If --params-file provided and file exists, load JSON, override matching config fields
CLI args still take precedence over JSON (explicit override)

~20 lines per binary, using existing clap + serde_json.

Files to Create/Modify

New Files

.gitlab-ci-training.yml — parent pipeline definition
scripts/generate-training-pipeline.sh — child YAML generator
infra/k8s/training/training-output-pvc.yaml — output PVC

Modified Files

crates/ml/examples/train_baseline_rl.rs — add --params-file arg
crates/ml/examples/train_baseline_supervised.rs — add --params-file arg

Cost Estimate (H100 ~€3/hr)

Scenario	Jobs	Time	Cost
Single model, single symbol	3	~1hr	~€3
Full ensemble, single symbol	14	~1hr (parallel)	~€16
Full ensemble, 4 symbols	56	~2hr (parallel)	~€65

Concurrency

Jobs use Kubernetes resource requests (1 GPU each). The autoscaler provisions H100 nodes as needed. MAX_PARALLEL limits fan-out at the pipeline level. If cluster is busy, Kubernetes queues pending pods naturally.

5.1 KiB Raw Blame History Unescape Escape