feat: training pipeline for all 10 ML ensemble models

- Add standalone training binaries: TGGN, KAN, xLSTM, Diffusion (DBN data) - Update Dockerfile.training: 6 → 16 binaries (all 10 models + hyperopt + baseline) - Expand train.sh: 4 → 10 models, fix registry URL and GPU pool nodeSelector - Add GPU overlay manifests for trading-service and ml-training-service - Create training data PVC and upload pod manifests - Expand web-gateway model validation: 4 → 10 types (training + tune routes) - Extend dashboard: 10 model cards grouped by category (RL/Temporal/Graph/Generative) - Add training image build job to Gitea CI workflow - Update GPU taint controller to exclude inference pool from tainting - Fix job-template nodeSelector: gpu → gpu-training Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:47:50 +01:00
parent e45bda9412
commit 5fb84a02f0
19 changed files with 3602 additions and 35 deletions
--- a/docs/plans/2026-02-24-training-pipeline-validation-design.md
+++ b/docs/plans/2026-02-24-training-pipeline-validation-design.md
@@ -0,0 +1,171 @@
+# Training Pipeline Validation Design
+
+## Goal
+
+End-to-end smoke test of the GPU training pipeline for all 10 ML ensemble models, validating both the K8s Job path (standalone binaries) and the gRPC service path (TrainingOrchestrator via dashboard). Catch errors early before committing to full training runs on expensive GPU nodes.
+
+## Current State
+
+### What Works
+- 10/10 cluster pods healthy (7 services on CI, 3 DBs on always-on)
+- GPU pools (H100 training, L4 inference) provisioned, scaled to 0
+- GPU taint controller auto-taints training nodes
+- 36 OHLCV-1min `.dbn.zst` files in `data/cache/futures-baseline/` (ES, NQ, 6E, ZN × 9 quarters, 53MB total)
+- UnifiedTrainable adapters exist for all 10 models
+- ml_training_service runs gRPC on port 50053 with 15 RPCs
+- Web dashboard has training progress UI
+- `baseline_common` shared DBN loading module
+
+### Gaps
+
+| Gap | Impact |
+|-----|--------|
+| Dockerfile.training builds only 6 binaries | Can't run TGGN/KAN/xLSTM/Diffusion/Liquid/TFT-DBN as K8s Jobs |
+| train.sh maps only 4 models | Full-ensemble preset incomplete |
+| No standalone training binaries for TGGN, KAN, xLSTM, Diffusion | Need new `train_*.rs` examples using UnifiedTrainable adapters |
+| TLOB needs L2 orderbook data (MBP-10), not OHLCV | `download_l2_data.rs` example exists but hasn't been run |
+| Proto ModelType enum has 6 variants | Missing TGGN, KAN, xLSTM, Diffusion |
+| Web gateway validates 4 model types | Missing 6 models |
+| Dashboard shows 4 model cards | Missing 6 models |
+| PVC `training-data-pvc` not created | Training jobs can't mount data |
+| job-template nodeSelector was `gpu` (fixed to `gpu-training`) | Was targeting non-existent pool |
+| CUDA_COMPUTE_CAP=90 in Dockerfile | Correct for H100 training, need separate L4 build for inference |
+
+## Design
+
+### Phase 1: Data & Binary Readiness
+
+**1a. Download L2 orderbook data for TLOB**
+- Run `download_l2_data` example targeting ES.FUT with MBP-10 schema
+- Store in `data/cache/futures-baseline-l2/` following existing directory structure
+- Minimum: 1 quarter of ES.FUT L2 data for smoke test
+
+**1b. Write 4 missing standalone training binaries**
+
+Create `ml/examples/train_<model>_dbn.rs` for each missing model:
+
+| Binary | Model | Adapter | Data |
+|--------|-------|---------|------|
+| `train_tggn_dbn` | TGGN | TGGNTrainableAdapter | OHLCV |
+| `train_kan_dbn` | KAN | KANTrainableAdapter | OHLCV |
+| `train_xlstm_dbn` | xLSTM | XLSTMTrainableAdapter | OHLCV |
+| `train_diffusion_dbn` | Diffusion | DiffusionTrainableAdapter | OHLCV |
+
+Each binary follows the pattern of `train_dqn_es_fut.rs`:
+1. Load DBN data via `baseline_common`
+2. Construct model config with CLI args (symbol, epochs, batch-size, learning-rate, max-steps-per-epoch)
+3. Instantiate UnifiedTrainable adapter
+4. Train with epoch loop, print loss/metrics
+5. Save checkpoint to `--output-dir`
+
+TLOB already has `train_tlob.rs` — update it to use L2 data from `data/cache/futures-baseline-l2/`.
+
+**1c. Update Dockerfile.training**
+
+Add all 10 model binaries + evaluate/baseline tools:
+- train_dqn_es_fut (replaces train_dqn)
+- train_ppo_parquet (keep, works with converted data)
+- train_tft_dbn
+- train_mamba2_dbn
+- train_liquid_dbn
+- train_tggn_dbn (new)
+- train_kan_dbn (new)
+- train_xlstm_dbn (new)
+- train_diffusion_dbn (new)
+- train_tlob (L2 data)
+- train_baseline (walk-forward orchestrator)
+- evaluate_baseline
+
+**1d. Create PVC and upload data**
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: training-data-pvc
+  namespace: foxhunt
+spec:
+  accessModes: [ReadOnlyMany]
+  resources:
+    requests:
+      storage: 10Gi
+  storageClassName: scw-bssd
+```
+
+Upload data via a temporary pod with the PVC mounted:
+- `data/cache/futures-baseline/` → `/data/futures-baseline/`
+- `data/cache/futures-baseline-l2/` → `/data/futures-baseline-l2/`
+
+**1e. Update train.sh for all 10 models**
+
+Expand MODEL_BINARY map to include all 10 models. Update `full-ensemble` preset to train all 10 sequentially (parallel would exhaust GPU memory).
+
+**1f. Build, push, and smoke test**
+
+1. Build training Docker image with all binaries
+2. Push to `rg.fr-par.scw.cloud/foxhunt/training:latest`
+3. Scale gpu-training pool to 1 (H100)
+4. Submit quick-test K8s Job for each model (--max-steps=100)
+5. Verify: job completes, checkpoint written, logs show loss decreasing
+6. Scale gpu-training pool back to 0
+
+### Phase 2: gRPC Service Pipeline (Dashboard → Training)
+
+**2a. Extend proto ModelType enum**
+
+In `fxt/proto/ml_training.proto`, add missing model types:
+- TGGN, KAN, XLSTM, DIFFUSION
+
+Add corresponding Hyperparameters oneof variants.
+
+**2b. Update web-gateway validation**
+
+In `web-gateway/src/routes/training.rs`:
+- Expand VALID_MODEL_TYPES to all 10
+- Map new model names to proto enum values
+
+**2c. Update dashboard**
+
+In `web-dashboard/src/pages/MLDashboard.tsx`:
+- Extend model cards to show all 10 models (responsive grid)
+- Group by category: RL (DQN, PPO), Temporal (TFT, Mamba2, xLSTM), Graph/Structure (TGGN, TLOB, LNN), Generative (KAN, Diffusion)
+
+**2d. End-to-end test**
+
+1. Scale L4 pool to 1
+2. Apply GPU overlay: `kubectl apply -f infra/k8s/services/ml-training-service-gpu.yaml`
+3. Via dashboard: POST /training/jobs for each model
+4. Verify: job appears in dashboard, progress streams via WebSocket, completion/failure reported
+5. Revert to CPU manifest, scale L4 to 0
+
+### Phase 3: Gitea CI Pipeline
+
+**3a. Create `.gitea/workflows/ci.yml`**
+
+Triggers: push to main, pull request
+
+Steps:
+1. `cargo check --workspace` (SQLX_OFFLINE=true)
+2. `cargo test --workspace --lib` (SQLX_OFFLINE=true)
+3. `cargo clippy --workspace` (deny warnings)
+4. Build Docker images (all services + training)
+5. Push to Scaleway registry (only on main merge)
+
+This is CI only. Training is triggered manually or via trading_service, never by CI.
+
+## Cost Estimate
+
+| Activity | Duration | Cost |
+|----------|----------|------|
+| Phase 1 smoke test (H100) | ~30 min (10 models × 100 steps) | ~€1.37 |
+| Phase 2 gRPC test (L4) | ~15 min | ~€0.19 |
+| Idle between tests | CI node | €0.08/hr |
+
+## Success Criteria
+
+1. All 10 model binaries compile and run on H100 (loss decreases over 100 steps)
+2. Checkpoint files written to output directory
+3. Training job visible in dashboard with progress updates
+4. gRPC streaming delivers real-time metrics to WebSocket
+5. Gitea CI pipeline builds and pushes images on main merge
+6. GPU pools scale back to 0 after tests
--- a/docs/plans/2026-02-24-training-pipeline-validation.md
+++ b/docs/plans/2026-02-24-training-pipeline-validation.md
@@ -0,0 +1,819 @@
+# Training Pipeline Validation Implementation Plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
+
+**Goal:** Validate the end-to-end GPU training pipeline for all 10 ML ensemble models via K8s Jobs and gRPC service paths.
+
+**Architecture:** Standalone training binaries (one per model) load DBN market data via `baseline_common`, train through the `UnifiedTrainable` adapter interface, and save checkpoints. These run as K8s Jobs on the H100 GPU pool. The gRPC path goes dashboard → web-gateway → ml-training-service → TrainingOrchestrator. A Gitea Actions CI pipeline handles build/test/push (not training).
+
+**Tech Stack:** Rust (Candle v0.9.1, tonic gRPC), Databento DBN, Docker (CUDA 12.4.1), K8s (Scaleway Kapsule), React + TypeScript dashboard, Gitea Actions CI
+
+---
+
+## Phase 1: Data & Binary Readiness
+
+### Task 1: Download L2 Orderbook Data for TLOB
+
+TLOB requires Level 2 (MBP-10) data, not OHLCV. The `download_l2_data` example already exists.
+
+**Files:**
+- Run: `ml/examples/download_l2_data.rs`
+- Output: `data/cache/futures-baseline-l2/ES.FUT/`
+
+**Step 1: Check existing download example compiles**
+
+Run: `SQLX_OFFLINE=true cargo build -p ml --example download_l2_data --release`
+Expected: Successful build
+
+**Step 2: Run L2 data download (ES.FUT only, 90 days)**
+
+Run: `cargo run -p ml --example download_l2_data --release -- --symbols ES.FUT --days 90 --start-date 2024-01-01`
+Expected: MBP-10 `.dbn.zst` files saved to `data/cache/futures-baseline-l2/ES.FUT/`
+
+Note: Requires valid `DATABENTO_API_KEY` environment variable. If the API key is expired or has insufficient credits, skip this step and use synthetic L2 data for the smoke test (TLOB adapter's `generate_sample_batch` method).
+
+**Step 3: Verify downloaded files**
+
+Run: `find data/cache/futures-baseline-l2 -name "*.dbn.zst" | wc -l`
+Expected: At least 1 file
+
+**Step 4: Commit data manifest (not data itself)**
+
+Run:
+```bash
+echo "data/cache/futures-baseline-l2/" >> .gitignore
+git add .gitignore
+git commit -m "chore: add L2 data cache to gitignore"
+```
+
+---
+
+### Task 2: Write TGGN Standalone Training Binary
+
+**Files:**
+- Create: `ml/examples/train_tggn_dbn.rs`
+- Reference: `ml/src/tgnn/trainable_adapter.rs` (TGGNTrainableAdapter)
+- Reference: `ml/examples/baseline_common/mod.rs`
+
+**Step 1: Create the training binary**
+
+Create `ml/examples/train_tggn_dbn.rs`:
+
+```rust
+//! TGGN (Temporal Graph Gated Network) training on DBN market data.
+//!
+//! Usage:
+//!   cargo run -p ml --example train_tggn_dbn --release -- --data-dir data/cache/futures-baseline --symbol ES.FUT
+//!   cargo run -p ml --example train_tggn_dbn --release -- --epochs 5 --max-steps 100
+
+#[allow(unreachable_pub)]
+mod baseline_common;
+
+use anyhow::{Context, Result};
+use candle_core::Device;
+use clap::Parser;
+use ml::tgnn::trainable_adapter::TGGNTrainableAdapter;
+use ml::tgnn::TGGNConfig;
+use ml::training::unified_trainer::UnifiedTrainable;
+use std::path::PathBuf;
+use std::time::Instant;
+
+#[derive(Parser, Debug)]
+#[command(about = "Train TGGN model on DBN market data")]
+struct Args {
+    #[arg(short, long, default_value = "data/cache/futures-baseline")]
+    data_dir: String,
+    #[arg(short, long, default_value = "ES.FUT")]
+    symbol: String,
+    #[arg(short, long, default_value_t = 10)]
+    epochs: usize,
+    #[arg(short, long, default_value_t = 64)]
+    batch_size: usize,
+    #[arg(short, long, default_value_t = 0.001)]
+    learning_rate: f64,
+    #[arg(long, default_value_t = 2000)]
+    max_steps_per_epoch: usize,
+    #[arg(short, long, default_value = "checkpoints")]
+    output_dir: String,
+}
+
+#[tokio::main]
+async fn main() -> Result<()> {
+    tracing_subscriber::fmt().with_target(false).init();
+
+    let args = Args::parse();
+    println!("=== TGGN Training on {} ===", args.symbol);
+
+    // Load market data
+    let bars = baseline_common::load_all_bars(
+        &PathBuf::from(&args.data_dir),
+        &args.symbol,
+    )?;
+    println!("Loaded {} bars", bars.len());
+
+    // Configure TGGN
+    let config = TGGNConfig {
+        node_dim: 16,
+        hidden_dim: 64,
+        num_heads: 4,
+        num_layers: 2,
+        learning_rate: args.learning_rate as f32,
+        ..Default::default()
+    };
+
+    let device = Device::cuda_if_available(0).unwrap_or(Device::Cpu);
+    println!("Device: {:?}", device);
+
+    let mut adapter = TGGNTrainableAdapter::new(config, &device)
+        .context("Failed to create TGGN adapter")?;
+
+    // Training loop
+    std::fs::create_dir_all(&args.output_dir)?;
+    let start = Instant::now();
+
+    for epoch in 0..args.epochs {
+        let steps = args.max_steps_per_epoch.min(bars.len() / args.batch_size);
+        let mut epoch_loss = 0.0;
+
+        for step in 0..steps {
+            let batch = adapter.generate_sample_batch(args.batch_size, &device)?;
+            let loss = adapter.train_step(&batch)?;
+            epoch_loss += loss;
+        }
+
+        let avg_loss = if steps > 0 { epoch_loss / steps as f64 } else { 0.0 };
+        println!("Epoch {}/{}: loss={:.6} ({} steps)", epoch + 1, args.epochs, avg_loss, steps);
+
+        // Save checkpoint every 5 epochs
+        if (epoch + 1) % 5 == 0 || epoch + 1 == args.epochs {
+            let path = format!("{}/tggn_{}_epoch{}.safetensors", args.output_dir, args.symbol, epoch + 1);
+            adapter.save_checkpoint(&path)?;
+            println!("  Checkpoint: {}", path);
+        }
+    }
+
+    println!("Training complete in {:.1}s", start.elapsed().as_secs_f64());
+    Ok(())
+}
+```
+
+**Important:** The exact API (`generate_sample_batch`, `train_step`, `save_checkpoint`) may differ per adapter. Check the actual `UnifiedTrainable` trait in `ml/src/training/unified_trainer.rs` and adjust method names accordingly. The pattern is: construct adapter → loop (generate batch → train_step → log loss) → save checkpoint.
+
+**Step 2: Verify it compiles**
+
+Run: `SQLX_OFFLINE=true cargo build -p ml --example train_tggn_dbn --release 2>&1 | tail -5`
+Expected: Successful compilation (or compilation errors to fix — iterate until it compiles)
+
+**Step 3: Run quick smoke test (CPU, 2 epochs)**
+
+Run: `cargo run -p ml --example train_tggn_dbn --release -- --epochs 2 --max-steps-per-epoch 10 --output-dir /tmp/tggn_test`
+Expected: Prints loss values, creates checkpoint file
+
+**Step 4: Commit**
+
+```bash
+git add ml/examples/train_tggn_dbn.rs
+git commit -m "feat(ml): add TGGN standalone DBN training binary"
+```
+
+---
+
+### Task 3: Write KAN Standalone Training Binary
+
+Same pattern as Task 2 but for KAN (Kolmogorov-Arnold Network).
+
+**Files:**
+- Create: `ml/examples/train_kan_dbn.rs`
+- Reference: `ml/src/kan/trainable.rs` (KANTrainableAdapter)
+
+**Step 1: Create the training binary**
+
+Follow the exact same template as Task 2 but substitute:
+- `TGGNTrainableAdapter` → `KANTrainableAdapter`
+- `TGGNConfig` → `KANConfig`
+- Config fields: check `ml/src/kan/mod.rs` for the actual KANConfig struct fields (likely `input_dim`, `hidden_dim`, `output_dim`, `grid_size`, `spline_order`, `learning_rate`)
+- Checkpoint prefix: `kan_`
+
+**Step 2: Verify it compiles**
+
+Run: `SQLX_OFFLINE=true cargo build -p ml --example train_kan_dbn --release 2>&1 | tail -5`
+
+**Step 3: Run quick smoke test**
+
+Run: `cargo run -p ml --example train_kan_dbn --release -- --epochs 2 --max-steps-per-epoch 10 --output-dir /tmp/kan_test`
+
+**Step 4: Commit**
+
+```bash
+git add ml/examples/train_kan_dbn.rs
+git commit -m "feat(ml): add KAN standalone DBN training binary"
+```
+
+---
+
+### Task 4: Write xLSTM Standalone Training Binary
+
+Same pattern as Task 2 but for xLSTM.
+
+**Files:**
+- Create: `ml/examples/train_xlstm_dbn.rs`
+- Reference: `ml/src/xlstm/trainable.rs` (XLSTMTrainableAdapter)
+
+**Step 1: Create the training binary**
+
+Same template, substitute:
+- `XLSTMTrainableAdapter` + `XLSTMConfig`
+- Config fields: check `ml/src/xlstm/mod.rs` (likely `input_size`, `hidden_size`, `num_layers`, `learning_rate`, sLSTM + mLSTM mixing)
+- Checkpoint prefix: `xlstm_`
+
+**Step 2-4: Compile, smoke test, commit** (same pattern)
+
+```bash
+git add ml/examples/train_xlstm_dbn.rs
+git commit -m "feat(ml): add xLSTM standalone DBN training binary"
+```
+
+---
+
+### Task 5: Write Diffusion Standalone Training Binary
+
+Same pattern as Task 2 but for Diffusion model (DDPM/DDIM).
+
+**Files:**
+- Create: `ml/examples/train_diffusion_dbn.rs`
+- Reference: `ml/src/diffusion/trainable.rs` (DiffusionTrainableAdapter)
+
+**Step 1: Create the training binary**
+
+Same template, substitute:
+- `DiffusionTrainableAdapter` + `DiffusionConfig`
+- Config fields: check `ml/src/diffusion/mod.rs` (likely `hidden_dim`, `num_layers`, `num_timesteps`, `time_embed_dim`, `learning_rate`)
+- Checkpoint prefix: `diffusion_`
+
+Note: Diffusion models have a different training dynamic (predicting noise, not direct loss). The adapter handles this internally through `train_step`.
+
+**Step 2-4: Compile, smoke test, commit** (same pattern)
+
+```bash
+git add ml/examples/train_diffusion_dbn.rs
+git commit -m "feat(ml): add Diffusion standalone DBN training binary"
+```
+
+---
+
+### Task 6: Update Dockerfile.training for All 10 Models
+
+**Files:**
+- Modify: `infra/docker/Dockerfile.training`
+
+**Step 1: Update the cargo build command to include all training binaries**
+
+Replace the existing `RUN cargo build` block (lines 60-70) with:
+
+```dockerfile
+# Build all training example binaries with CUDA support
+RUN cargo build --release -p ml --features ml/cuda \
+    --example train_dqn_es_fut \
+    --example train_ppo_parquet \
+    --example train_tft_dbn \
+    --example train_mamba2_dbn \
+    --example train_liquid_dbn \
+    --example train_tggn_dbn \
+    --example train_kan_dbn \
+    --example train_xlstm_dbn \
+    --example train_diffusion_dbn \
+    --example train_tlob \
+    --example train_baseline \
+    --example evaluate_baseline \
+    --example hyperopt_dqn_demo \
+    --example hyperopt_ppo_demo \
+    --example hyperopt_tft_demo \
+    --example hyperopt_mamba2_demo \
+    && mkdir -p /build/out \
+    && for bin in \
+        train_dqn_es_fut train_ppo_parquet train_tft_dbn train_mamba2_dbn \
+        train_liquid_dbn train_tggn_dbn train_kan_dbn train_xlstm_dbn \
+        train_diffusion_dbn train_tlob train_baseline evaluate_baseline \
+        hyperopt_dqn_demo hyperopt_ppo_demo hyperopt_tft_demo hyperopt_mamba2_demo; do \
+        cp target/release/examples/${bin} /build/out/ && strip /build/out/${bin}; \
+    done
+```
+
+**Step 2: Verify Dockerfile syntax**
+
+Run: `docker build --check -f infra/docker/Dockerfile.training .` (or just syntax-check manually)
+
+**Step 3: Commit**
+
+```bash
+git add infra/docker/Dockerfile.training
+git commit -m "build: add all 10 model training binaries to Docker image"
+```
+
+---
+
+### Task 7: Update train.sh for All 10 Models
+
+**Files:**
+- Modify: `infra/scripts/train.sh`
+
+**Step 1: Update model-to-binary mapping and ALL_MODELS array**
+
+Replace lines 30-38:
+
+```bash
+ALL_MODELS=(dqn ppo tft mamba2 tggn tlob liquid kan xlstm diffusion)
+
+# --- Model-to-binary mapping ----------------------------------------------
+declare -A MODEL_BINARY=(
+  [dqn]=train_dqn_es_fut
+  [ppo]=train_ppo_parquet
+  [tft]=train_tft_dbn
+  [mamba2]=train_mamba2_dbn
+  [tggn]=train_tggn_dbn
+  [tlob]=train_tlob
+  [liquid]=train_liquid_dbn
+  [kan]=train_kan_dbn
+  [xlstm]=train_xlstm_dbn
+  [diffusion]=train_diffusion_dbn
+)
+```
+
+**Step 2: Update the full-ensemble preset to run sequentially (not parallel)**
+
+Replace lines 218-222 (the full-ensemble case):
+
+```bash
+  full-ensemble)
+    for m in "${ALL_MODELS[@]}"; do
+      submit_job "$m"
+      echo "  Waiting for ${m} to complete before next model (GPU memory)"
+      kubectl -n "${NAMESPACE}" wait --for=condition=complete "job/train-${m}-${TIMESTAMP}" --timeout="${TIMEOUT}s" 2>/dev/null || true
+    done
+    ;;
+```
+
+**Step 3: Update nodeSelector from `gpu` to `gpu-training` in the generated manifest**
+
+Replace line 155:
+```bash
+        k8s.scaleway.com/pool-name: gpu-training
+```
+
+**Step 4: Fix the image registry URL**
+
+Line 25 currently has `rg.nl-ams.scw.cloud` — should be `rg.fr-par.scw.cloud`:
+```bash
+IMAGE="rg.fr-par.scw.cloud/foxhunt/training:latest"
+```
+
+**Step 5: Update usage/help text to list all 10 models**
+
+**Step 6: Commit**
+
+```bash
+git add infra/scripts/train.sh
+git commit -m "feat: expand train.sh to support all 10 ensemble models"
+```
+
+---
+
+### Task 8: Create PVC and Data Upload Manifests
+
+**Files:**
+- Create: `infra/k8s/training/training-data-pvc.yaml`
+- Create: `infra/k8s/training/data-upload-job.yaml`
+
+**Step 1: Create PVC manifest**
+
+Create `infra/k8s/training/training-data-pvc.yaml`:
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: training-data-pvc
+  namespace: foxhunt
+  labels:
+    app.kubernetes.io/name: training-data
+    app.kubernetes.io/part-of: foxhunt
+spec:
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Gi
+  storageClassName: scw-bssd
+```
+
+Note: Scaleway Block Storage (scw-bssd) only supports ReadWriteOnce. This is fine since training jobs run sequentially on one node.
+
+**Step 2: Create data upload job**
+
+Create `infra/k8s/training/data-upload-job.yaml`:
+
+```yaml
+# Temporary pod for uploading training data to PVC.
+# Usage:
+#   1. kubectl apply -f infra/k8s/training/training-data-pvc.yaml
+#   2. kubectl apply -f infra/k8s/training/data-upload-job.yaml
+#   3. kubectl cp data/cache/futures-baseline foxhunt/data-upload:/data/futures-baseline
+#   4. kubectl cp data/cache/futures-baseline-l2 foxhunt/data-upload:/data/futures-baseline-l2  (if L2 data exists)
+#   5. kubectl delete pod data-upload -n foxhunt
+apiVersion: v1
+kind: Pod
+metadata:
+  name: data-upload
+  namespace: foxhunt
+spec:
+  containers:
+    - name: upload
+      image: busybox:1.36
+      command: ["sleep", "3600"]
+      volumeMounts:
+        - name: training-data
+          mountPath: /data
+  volumes:
+    - name: training-data
+      persistentVolumeClaim:
+        claimName: training-data-pvc
+  restartPolicy: Never
+```
+
+**Step 3: Commit**
+
+```bash
+git add infra/k8s/training/training-data-pvc.yaml infra/k8s/training/data-upload-job.yaml
+git commit -m "infra: add training data PVC and upload manifests"
+```
+
+---
+
+### Task 9: Build and Push Training Docker Image
+
+This task requires Docker build capabilities. Run from a machine with Docker installed.
+
+**Step 1: Build the training image**
+
+Run from repo root:
+```bash
+docker build -f infra/docker/Dockerfile.training -t rg.fr-par.scw.cloud/foxhunt/training:latest .
+```
+
+Expected: Multi-stage build succeeds, all 16 binaries compiled with CUDA support. This will take 15-30 minutes on first build.
+
+**Step 2: Push to Scaleway registry**
+
+```bash
+docker push rg.fr-par.scw.cloud/foxhunt/training:latest
+```
+
+**Step 3: Verify image in registry**
+
+```bash
+scw registry image list namespace-id=e8b31a6b-0dba-442f-8b34-23be2b3a3e52 | grep training
+```
+
+---
+
+### Task 10: Create PVC and Upload Data to Cluster
+
+**Step 1: Apply PVC manifest**
+
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl apply -f infra/k8s/training/training-data-pvc.yaml
+```
+
+**Step 2: Start upload pod**
+
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl apply -f infra/k8s/training/data-upload-job.yaml
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl wait --for=condition=ready pod/data-upload -n foxhunt --timeout=60s
+```
+
+**Step 3: Copy training data to PVC**
+
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl cp data/cache/futures-baseline foxhunt/data-upload:/data/futures-baseline
+```
+
+If L2 data exists:
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl cp data/cache/futures-baseline-l2 foxhunt/data-upload:/data/futures-baseline-l2
+```
+
+**Step 4: Verify data on PVC**
+
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl exec -n foxhunt data-upload -- find /data -name "*.dbn.zst" | wc -l
+```
+Expected: 36 (or more if L2 data included)
+
+**Step 5: Clean up upload pod**
+
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl delete pod data-upload -n foxhunt
+```
+
+---
+
+### Task 11: GPU Smoke Test — Run All 10 Models
+
+**Step 1: Scale up H100 training pool**
+
+```bash
+scw k8s pool update 4a6f18ea-aa28-42a5-82b8-83c3cdc2f987 size=1 region=fr-par-2
+```
+
+Wait for node to be Ready (~3-5 minutes):
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl get nodes --watch
+```
+
+**Step 2: Run quick-test for each model**
+
+For each model, run:
+```bash
+./infra/scripts/train.sh quick-test --model dqn --max-steps 100
+./infra/scripts/train.sh quick-test --model ppo --max-steps 100
+# ... repeat for all 10 models
+```
+
+Or use `full-ensemble` (sequential):
+```bash
+./infra/scripts/train.sh full-ensemble --max-steps 100
+```
+
+**Step 3: Monitor jobs**
+
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl get jobs -n foxhunt -l foxhunt/job-type=training
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl logs -n foxhunt -l foxhunt/job-type=training --tail=50
+```
+
+**Step 4: Verify success criteria**
+
+For each model:
+- Job status = Complete (not Failed)
+- Logs show decreasing loss over steps
+- No CUDA OOM errors
+
+**Step 5: Scale H100 pool back to 0**
+
+```bash
+scw k8s pool update 4a6f18ea-aa28-42a5-82b8-83c3cdc2f987 size=0 region=fr-par-2
+```
+
+**Step 6: Commit any fixes discovered during smoke test**
+
+---
+
+## Phase 2: gRPC Service Pipeline (Dashboard → Training)
+
+### Task 12: Expand Web Gateway Model Validation
+
+**Files:**
+- Modify: `web-gateway/src/routes/training.rs:56`
+
+**Step 1: Read the current validation**
+
+Read `web-gateway/src/routes/training.rs` and find the `VALID_MODEL_TYPES` constant.
+
+**Step 2: Expand VALID_MODEL_TYPES**
+
+Change:
+```rust
+const VALID_MODEL_TYPES: &[&str] = &["dqn", "ppo", "tft", "mamba2"];
+```
+
+To:
+```rust
+const VALID_MODEL_TYPES: &[&str] = &[
+    "dqn", "ppo", "tft", "mamba2", "tggn", "tlob", "liquid", "kan", "xlstm", "diffusion",
+];
+```
+
+**Step 3: Run existing tests to verify no regressions**
+
+Run: `SQLX_OFFLINE=true cargo test -p web-gateway --lib -- training`
+Expected: All existing tests pass
+
+**Step 4: Add test for new model types**
+
+Add a test that verifies POST /training/jobs accepts all 10 model types. Check existing test patterns in the file for the exact test structure.
+
+**Step 5: Run tests**
+
+Run: `SQLX_OFFLINE=true cargo test -p web-gateway --lib -- training`
+Expected: All tests pass including new ones
+
+**Step 6: Commit**
+
+```bash
+git add web-gateway/src/routes/training.rs
+git commit -m "feat(web-gateway): accept all 10 model types for training jobs"
+```
+
+---
+
+### Task 13: Extend Dashboard Model Cards
+
+**Files:**
+- Modify: `web-dashboard/src/pages/MLDashboard.tsx`
+
+**Step 1: Read current dashboard code**
+
+Read `web-dashboard/src/pages/MLDashboard.tsx` to understand the current ModelCard rendering.
+
+**Step 2: Extend model cards to show all 10 models grouped by category**
+
+Update the model cards section. The models should be grouped:
+- **RL**: DQN, PPO
+- **Temporal**: TFT, Mamba2, xLSTM
+- **Graph/Structure**: TGGN, TLOB, LNN (Liquid)
+- **Generative**: KAN, Diffusion
+
+Adjust the grid to accommodate 10 cards (e.g., responsive 2-5 columns).
+
+**Step 3: Build dashboard to verify**
+
+Run: `cd web-dashboard && npm run build`
+Expected: Build succeeds with no TypeScript errors
+
+**Step 4: Commit**
+
+```bash
+git add web-dashboard/src/pages/MLDashboard.tsx
+git commit -m "feat(dashboard): display all 10 ensemble model cards"
+```
+
+---
+
+### Task 14: End-to-End gRPC Service Test
+
+This is a manual integration test on the cluster.
+
+**Step 1: Scale L4 inference pool to 1**
+
+```bash
+scw k8s pool update 590e281c-95c4-4f8c-8565-ff7e8cd517a4 size=1 region=fr-par-2
+```
+
+**Step 2: Apply GPU overlay for ml-training-service**
+
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl apply -f infra/k8s/services/ml-training-service-gpu.yaml
+```
+
+Wait for pod to be ready on L4 node:
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl get pods -n foxhunt -l app.kubernetes.io/name=ml-training-service -o wide --watch
+```
+
+**Step 3: Test via web-gateway REST API**
+
+Port-forward web-gateway:
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl port-forward -n foxhunt svc/web-gateway 3000:3000 &
+```
+
+Submit a training job:
+```bash
+curl -X POST http://localhost:3000/training/jobs \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer <JWT_TOKEN>" \
+  -d '{"model_type": "dqn", "description": "smoke test", "use_gpu": true}'
+```
+
+List jobs:
+```bash
+curl http://localhost:3000/training/jobs -H "Authorization: Bearer <JWT_TOKEN>"
+```
+
+Expected: Job appears with status PENDING or RUNNING.
+
+**Step 4: Verify dashboard shows training progress**
+
+Open `http://localhost:3000` in browser. Navigate to ML Dashboard. Verify:
+- Training job appears in the Training Progress section
+- Status updates flow via WebSocket
+
+**Step 5: Revert to CPU manifest and scale down**
+
+```bash
+KUBECONFIG=~/.kube/foxhunt-fr-par.yaml kubectl apply -f infra/k8s/services/ml-training-service.yaml
+scw k8s pool update 590e281c-95c4-4f8c-8565-ff7e8cd517a4 size=0 region=fr-par-2
+```
+
+---
+
+## Phase 3: Gitea CI Pipeline
+
+### Task 15: Create Gitea Actions CI Workflow
+
+**Files:**
+- Create: `.gitea/workflows/ci.yml`
+
+**Step 1: Create the CI workflow**
+
+Create `.gitea/workflows/ci.yml`:
+
+```yaml
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+env:
+  SQLX_OFFLINE: "true"
+  CARGO_TERM_COLOR: always
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@stable
+        with:
+          components: clippy
+
+      - name: Install protoc
+        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
+
+      - name: Cache cargo
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
+
+      - name: Cargo check
+        run: cargo check --workspace
+
+      - name: Cargo clippy
+        run: cargo clippy --workspace -- -D warnings
+
+      - name: Cargo test
+        run: cargo test --workspace --lib
+
+  docker:
+    needs: check
+    if: github.ref == 'refs/heads/main'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Login to Scaleway Registry
+        run: echo "${{ secrets.SCW_SECRET_KEY }}" | docker login rg.fr-par.scw.cloud -u nologin --password-stdin
+
+      - name: Build and push service images
+        run: ./infra/scripts/build-and-push.sh --parallel
+
+      - name: Build and push training image
+        run: |
+          docker build -f infra/docker/Dockerfile.training -t rg.fr-par.scw.cloud/foxhunt/training:latest .
+          docker push rg.fr-par.scw.cloud/foxhunt/training:latest
+```
+
+Note: This workflow requires:
+- Gitea runner configured with Docker support
+- `SCW_SECRET_KEY` secret set in Gitea repository settings
+- Sufficient disk space for Rust compilation + Docker builds
+
+**Step 2: Verify YAML syntax**
+
+Run: `python3 -c "import yaml; yaml.safe_load(open('.gitea/workflows/ci.yml'))"`
+
+**Step 3: Commit**
+
+```bash
+git add .gitea/workflows/ci.yml
+git commit -m "ci: add Gitea Actions workflow for build, test, and push"
+```
+
+---
+
+## Summary of All Commits
+
+| # | Commit | Phase |
+|---|--------|-------|
+| 1 | `chore: add L2 data cache to gitignore` | 1 |
+| 2 | `feat(ml): add TGGN standalone DBN training binary` | 1 |
+| 3 | `feat(ml): add KAN standalone DBN training binary` | 1 |
+| 4 | `feat(ml): add xLSTM standalone DBN training binary` | 1 |
+| 5 | `feat(ml): add Diffusion standalone DBN training binary` | 1 |
+| 6 | `build: add all 10 model training binaries to Docker image` | 1 |
+| 7 | `feat: expand train.sh to support all 10 ensemble models` | 1 |
+| 8 | `infra: add training data PVC and upload manifests` | 1 |
+| 9 | `feat(web-gateway): accept all 10 model types for training jobs` | 2 |
+| 10 | `feat(dashboard): display all 10 ensemble model cards` | 2 |
+| 11 | `ci: add Gitea Actions workflow for build, test, and push` | 3 |
+
+Tasks 9-11 (build Docker, upload data, GPU smoke test) and Task 14 (gRPC e2e test) are operational steps, not code commits.