# ML Training Pipeline Design

## Overview

Manually-triggered GitLab CI pipeline for training the 10-model ML ensemble on 730 days of OHLCV-1m futures data. Runs on H100 GPU pool with configurable parallelism.

## Trigger Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `SYMBOLS` | `ES.FUT` | Comma-separated symbols (ES.FUT, NQ.FUT, ZN.FUT, 6E.FUT) |
| `MODELS` | `all` | `all`, `rl`, `supervised`, or comma-separated model names |
| `PHASE` | `full` | `hyperopt`, `train`, `eval`, or `full` (all three) |
| `MAX_PARALLEL` | `10` | GPU concurrency cap |
| `EPOCHS` | `50` | Training epochs per walk-forward fold |
| `HYPEROPT_TRIALS` | `20` | PSO trials per model |
| `RUN_ID` | auto | YYYYMMDD-HHMMSS or manual override |

## Model Groups

- **RL (2)**: DQN, PPO — via `train_baseline_rl` / `hyperopt_baseline_rl`
- **Supervised (8)**: TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion — via `train_baseline_supervised` / `hyperopt_baseline_supervised`
- **Hyperopt support**: DQN, PPO, TFT, Mamba2 (other 6 use default configs)

## Architecture

### Pipeline Structure

```
.gitlab-ci-training.yml (manual trigger, parent pipeline)
  ├── prepare: generate-training-pipeline.sh
  │     → writes .training-generated.yml as artifact
  └── trigger: launches child pipeline from artifact
        ├── STAGE: hyperopt (parallel, max MAX_PARALLEL)
        ├── STAGE: train (parallel, needs hyperopt)
        └── STAGE: evaluate (single job, needs all training)
```

### Why Generated Child Pipeline

GitLab CI lacks dynamic matrix jobs. A prepare job generates exactly the needed jobs based on MODELS × SYMBOLS, avoiding dead "skipped" jobs. Adding a model = one line in the generator.

### Infrastructure

- **Training image**: `rg.fr-par.scw.cloud/foxhunt-ci/training:latest` (built by main CI)
- **GPU pool**: `gpu-training` (H100), Kubernetes autoscaler handles node provisioning
- **Input PVC**: `training-data-pvc` at `/data` (read-only, futures-baseline data)
- **Output PVC**: `training-output-pvc` at `/output` (read-write, checkpoints + results)

### Output Directory Convention

```
/output/<RUN_ID>/
├── hyperopt/<model>/<symbol>/results.json
├── models/<model>/<symbol>/*.safetensors
├── models/<model>/<symbol>/norm_stats.json
└── eval/ensemble_report.json
```

## Stage Details

### Stage 1: Hyperopt (optional)

Per model × symbol, runs PSO hyperparameter optimization.

- RL: `hyperopt_baseline_rl --model {dqn|ppo} --trials $HYPEROPT_TRIALS --epochs 10 --data-dir /data/$SYMBOL --output /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json`
- Supervised: `hyperopt_baseline_supervised --model {tft|mamba2} --trials $HYPEROPT_TRIALS --data-dir /data/$SYMBOL --output /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json`
- Models without hyperopt adapters (TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion) skip this stage.

### Stage 2: Train

Per model × symbol, runs walk-forward training.

- RL: `train_baseline_rl --model {dqn|ppo} --epochs $EPOCHS --data-dir /data/$SYMBOL --output-dir /output/$RUN_ID/models/$MODEL/$SYMBOL --params-file /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json`
- Supervised: `train_baseline_supervised --model $MODEL --epochs $EPOCHS --data-dir /data/$SYMBOL --output-dir /output/$RUN_ID/models/$MODEL/$SYMBOL --params-file /output/$RUN_ID/hyperopt/$MODEL/$SYMBOL/results.json`

`--params-file` is optional — if the file doesn't exist (no hyperopt ran), uses compiled defaults. CLI args override JSON params.

### Stage 3: Evaluate

Single job after all training completes.

`evaluate_baseline --models-dir /output/$RUN_ID/models --data-dir /data --output /output/$RUN_ID/eval/ensemble_report.json`

Runs inference on held-out test windows, computes per-model and ensemble metrics (Sharpe, MaxDD, win rate, profit factor).

## Parameter Passing: Hyperopt → Training

Add `--params-file <path>` optional arg to `train_baseline_rl` and `train_baseline_supervised`. Logic:

1. Parse CLI args as normal (with defaults)
2. If `--params-file` provided and file exists, load JSON, override matching config fields
3. CLI args still take precedence over JSON (explicit override)

~20 lines per binary, using existing clap + serde_json.

## Files to Create/Modify

### New Files
- `.gitlab-ci-training.yml` — parent pipeline definition
- `scripts/generate-training-pipeline.sh` — child YAML generator
- `infra/k8s/training/training-output-pvc.yaml` — output PVC

### Modified Files
- `crates/ml/examples/train_baseline_rl.rs` — add `--params-file` arg
- `crates/ml/examples/train_baseline_supervised.rs` — add `--params-file` arg

## Cost Estimate (H100 ~€3/hr)

| Scenario | Jobs | Time | Cost |
|----------|------|------|------|
| Single model, single symbol | 3 | ~1hr | ~€3 |
| Full ensemble, single symbol | 14 | ~1hr (parallel) | ~€16 |
| Full ensemble, 4 symbols | 56 | ~2hr (parallel) | ~€65 |

## Concurrency

Jobs use Kubernetes resource requests (1 GPU each). The autoscaler provisions H100 nodes as needed. MAX_PARALLEL limits fan-out at the pipeline level. If cluster is busy, Kubernetes queues pending pods naturally.
