6e339316cf
Adds a parent/child GitLab CI pipeline for ML model training: - Generator script produces per-model hyperopt/train/evaluate jobs - Parent pipeline (.gitlab-ci-training.yml) with manual trigger - NFS-backed ReadWriteMany PVC for shared training outputs - Hyperopt params wired into training binaries (DQN, PPO, TFT, Mamba2) - Shared DBN loader eliminates duplicate code across hyperopt adapters - Supervised hyperopt unified to DBN data (was parquet-only) Pipeline: hyperopt (4 models) → train (10 models) → evaluate ensemble Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
119 lines
5.1 KiB
Markdown
119 lines
5.1 KiB
Markdown
# ML Training Pipeline Design
|
||
|
||
## Overview
|
||
|
||
Manually-triggered GitLab CI pipeline for training the 10-model ML ensemble on 730 days of OHLCV-1m futures data. Runs on H100 GPU pool with configurable parallelism.
|
||
|
||
## Trigger Variables
|
||
|
||
| Variable | Default | Description |
|
||
|----------|---------|-------------|
|
||
| `SYMBOLS` | `ES.FUT` | Comma-separated symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT) |
|
||
| `MODELS` | `all` | `all`, `rl`, `supervised`, or comma-separated model names |
|
||
| `PHASE` | `full` | `hyperopt`, `train`, `eval`, or `full` (all three) |
|
||
| `MAX_PARALLEL` | `10` | GPU concurrency cap |
|
||
| `EPOCHS` | `50` | Training epochs per walk-forward fold |
|
||
| `HYPEROPT_TRIALS` | `20` | PSO trials per model |
|
||
| `RUN_ID` | auto | YYYYMMDD-HHMMSS or manual override |
|
||
|
||
## Model Groups
|
||
|
||
- **RL (2)**: DQN, PPO — via `train_baseline_rl` / `hyperopt_baseline_rl`
|
||
- **Supervised (8)**: TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion — via `train_baseline_supervised` / `hyperopt_baseline_supervised`
|
||
- **Hyperopt support**: DQN, PPO, TFT, Mamba2 (other 6 use default configs)
|
||
|
||
## Architecture
|
||
|
||
### Pipeline Structure
|
||
|
||
```
|
||
.gitlab-ci-training.yml (manual trigger, parent pipeline)
|
||
├── prepare: generate-training-pipeline.sh
|
||
│ → writes .training-generated.yml as artifact
|
||
└── trigger: launches child pipeline from artifact
|
||
├── STAGE: hyperopt (parallel, max MAX_PARALLEL)
|
||
├── STAGE: train (parallel, needs hyperopt)
|
||
└── STAGE: evaluate (single job, needs all training)
|
||
```
|
||
|
||
### Why Generated Child Pipeline
|
||
|
||
GitLab CI lacks dynamic matrix jobs. A prepare job generates exactly the needed jobs based on MODELS × SYMBOLS, avoiding dead "skipped" jobs. Adding a model = one line in the generator.
|
||
|
||
### Infrastructure
|
||
|
||
- **Training image**: `rg.fr-par.scw.cloud/foxhunt-ci/training:latest` (built by main CI)
|
||
- **GPU pool**: `gpu-training` (H100), Kubernetes autoscaler handles node provisioning
|
||
- **Input PVC**: `training-data-pvc` at `/data` (read-only, futures-baseline data)
|
||
- **Output PVC**: `training-output-pvc` at `/output` (read-write, checkpoints + results)
|
||
|
||
### Output Directory Convention
|
||
|
||
```
|
||
/output/<RUN_ID>/
|
||
├── hyperopt/<model>/<symbol>/results.json
|
||
├── models/<model>/<symbol>/*.safetensors
|
||
├── models/<model>/<symbol>/norm_stats.json
|
||
└── eval/ensemble_report.json
|
||
```
|
||
|
||
## Stage Details
|
||
|
||
### Stage 1: Hyperopt (optional)
|
||
|
||
Per model × symbol, runs PSO hyperparameter optimization.
|
||
|
||
- RL: `hyperopt_baseline_rl --model {dqn|ppo} --trials $HYPEROPT_TRIALS --epochs 10 --data-dir /data/$SYMBOL --output /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json`
|
||
- Supervised: `hyperopt_baseline_supervised --model {tft|mamba2} --trials $HYPEROPT_TRIALS --data-dir /data/$SYMBOL --output /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json`
|
||
- Models without hyperopt adapters (TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion) skip this stage.
|
||
|
||
### Stage 2: Train
|
||
|
||
Per model × symbol, runs walk-forward training.
|
||
|
||
- RL: `train_baseline_rl --model {dqn|ppo} --epochs $EPOCHS --data-dir /data/$SYMBOL --output-dir /output/$RUN_ID/models/$MODEL/$SYMBOL --params-file /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json`
|
||
- Supervised: `train_baseline_supervised --model $MODEL --epochs $EPOCHS --data-dir /data/$SYMBOL --output-dir /output/$RUN_ID/models/$MODEL/$SYMBOL --params-file /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json`
|
||
|
||
`--params-file` is optional — if the file doesn't exist (no hyperopt ran), uses compiled defaults. CLI args override JSON params.
|
||
|
||
### Stage 3: Evaluate
|
||
|
||
Single job after all training completes.
|
||
|
||
`evaluate_baseline --models-dir /output/$RUN_ID/models --data-dir /data --output /output/$RUN_ID/eval/ensemble_report.json`
|
||
|
||
Runs inference on held-out test windows, computes per-model and ensemble metrics (Sharpe, MaxDD, win rate, profit factor).
|
||
|
||
## Parameter Passing: Hyperopt → Training
|
||
|
||
Add `--params-file <path>` optional arg to `train_baseline_rl` and `train_baseline_supervised`. Logic:
|
||
|
||
1. Parse CLI args as normal (with defaults)
|
||
2. If `--params-file` provided and file exists, load JSON, override matching config fields
|
||
3. CLI args still take precedence over JSON (explicit override)
|
||
|
||
~20 lines per binary, using existing clap + serde_json.
|
||
|
||
## Files to Create/Modify
|
||
|
||
### New Files
|
||
- `.gitlab-ci-training.yml` — parent pipeline definition
|
||
- `scripts/generate-training-pipeline.sh` — child YAML generator
|
||
- `infra/k8s/training/training-output-pvc.yaml` — output PVC
|
||
|
||
### Modified Files
|
||
- `crates/ml/examples/train_baseline_rl.rs` — add `--params-file` arg
|
||
- `crates/ml/examples/train_baseline_supervised.rs` — add `--params-file` arg
|
||
|
||
## Cost Estimate (H100 ~€3/hr)
|
||
|
||
| Scenario | Jobs | Time | Cost |
|
||
|----------|------|------|------|
|
||
| Single model, single symbol | 3 | ~1hr | ~€3 |
|
||
| Full ensemble, single symbol | 14 | ~1hr (parallel) | ~€16 |
|
||
| Full ensemble, 4 symbols | 56 | ~2hr (parallel) | ~€65 |
|
||
|
||
## Concurrency
|
||
|
||
Jobs use Kubernetes resource requests (1 GPU each). The autoscaler provisions H100 nodes as needed. MAX_PARALLEL limits fan-out at the pipeline level. If cluster is busy, Kubernetes queues pending pods naturally.
|