foxhunt/docs/plans/2026-02-27-training-deploy-design.md

# Training Pipeline & Model Deployment Design

**Date**: 2026-02-27
**Status**: Approved
**Tracks**: Two independent parallel tracks

## Overview

Wire the complete training-to-serving pipeline on Kapsule. Track 1 handles training job dispatch (fxt CLI → gRPC → PostgreSQL → K8s GPU Jobs). Track 2 handles model serving and promotion (S3 checkpoints → model loader → inference → operator-gated promotion).

Both tracks share S3 (`foxhunt-models/`) as the handoff point.

## Track 1: Training Pipeline

### Goal

`fxt train start tft ES.FUT --epochs 50` creates a real GPU training job on Kapsule.

### Flow

```
fxt train start tft ES.FUT --epochs 50
  → gRPC StartTraining (ml_training_service:50053)
  → service.rs handler → JobSpawner::spawn_batch() → PostgreSQL
  → In-process queue consumer (tokio::spawn, 5s poll)
  → JobSpawner::get_next_pending_job()
  → K8sDispatcher::dispatch() → batch/v1 Job
      ├── training container (GPU, train_baseline_rl/supervised)
      └── uploader sidecar (S3 → foxhunt-models/)
  → Job completion callback → update status in PostgreSQL
```

### Existing Code

| Component | File | Status |
|-----------|------|--------|
| fxt train start | `bin/fxt/src/commands/train/start.rs` | Done (sends gRPC) |
| fxt train status | `bin/fxt/src/commands/train/status.rs` | Done |
| fxt train list | `bin/fxt/src/commands/train/list.rs` | Done |
| K8sDispatcher | `services/ml_training_service/src/k8s_dispatcher.rs` | Done (builds Jobs) |
| JobSpawner | `services/ml_training_service/src/job_spawner.rs` | Done (PostgreSQL CRUD) |
| Job template | `infra/k8s/training/job-template.yaml` | Done (reference only) |

### Changes Needed

1. **Wire gRPC handler to JobSpawner**: `service.rs` StartTraining handler must call `JobSpawner::spawn_batch()` instead of returning a stub response.

2. **Add queue consumer loop**: In `main.rs`, `tokio::spawn` a loop that polls `get_next_pending_job()` every 5s and calls `K8sDispatcher::dispatch()`.

3. **Job completion callback**: The uploader sidecar already has `CALLBACK_ENDPOINT`. Add a gRPC or HTTP endpoint in ml_training_service that receives completion notifications and calls `JobSpawner::update_job_status()`.

4. **Deploy training-output-pvc**: Currently only `training-data-pvc` is deployed. Need `training-output-pvc` for job artifact staging.

5. **PostgreSQL schema**: Deploy `batch_jobs` and `child_jobs` tables. Add sqlx migration.

6. **Docker image**: Rebuild ml-training-service image with kube-rs K8sDispatcher compiled in.

### K8s Resources

- `ml-training-service` Deployment (exists, needs redeploy with new image)
- `training-data-pvc` (exists, 10Gi scw-bssd)
- `training-output-pvc` (new, needs creation)
- `s3-credentials` Secret (exists)
- `scw-registry` Secret (exists)
- `gpu-training` node pool (Scaleway, autoscales 0→N)

## Track 2: Model Serving & Validation

### Goal

Trained model checkpoints deploy to a serving endpoint with operator-gated promotion and automatic rollback.

### Flow

```
Training job completes → uploader sidecar → S3 (foxhunt-models/)
  → Callback to ml_training_service
  → PromotionManager::register_completion()
      ├── No active model → auto-register
      └── Compare metrics → PendingPromotion / NoImprovement
  → fxt model promote <model-id> (operator approval)
  → DeploymentPipeline::deploy()
      ├── Rolling update
      ├── Health check (inference smoke test)
      └── Auto-rollback on failure
```

### Existing Code

| Component | File | Status |
|-----------|------|--------|
| S3ModelLoader | `crates/model_loader/src/lib.rs` | Done (S3 + LRU cache + versioning) |
| PromotionManager | `services/ml_training_service/src/promotion_manager.rs` | Done (metric comparison, pending/active tracking) |
| DeploymentPipeline | `services/ml_training_service/src/deployment_pipeline.rs` | Done (rolling update, health check, rollback) |
| ValidationPipeline | `services/ml_training_service/src/validation_pipeline.rs` | Exists |

### Changes Needed

1. **fxt model commands**: Add `fxt model promote`, `fxt model list`, `fxt model status` CLI subcommands.

2. **Wire promotion callback**: Uploader sidecar completion → PromotionManager::register_completion(). Extract metrics from training output.

3. **Model serving deployment**: K8s Deployment that loads active model from S3 via S3ModelLoader and serves predictions via gRPC.

4. **Validation test**: Send sample market data to inference endpoint, verify prediction response shape and latency.

5. **Proto additions**: Add `PromoteModel`, `ListModels`, `GetModelStatus` RPCs to ml_training proto.

## Shared Infrastructure

| Resource | Status | Notes |
|----------|--------|-------|
| PostgreSQL | Running in foxhunt ns | Needs schema migration |
| S3 foxhunt-models | Exists | Bucket + credentials |
| training-data-pvc | Deployed | 10Gi, read-only for jobs |
| training-output-pvc | Needs creation | Staging for job artifacts |
| scw-registry secret | Deployed | Image pull auth |
| gpu-training pool | Configured | Autoscales 0→N |

## Decision Log

| Decision | Choice | Rationale |
|----------|--------|-----------|
| Job dispatch path | DB-backed (JobSpawner) | Full history, batch orchestration, rollback |
| Queue consumer | In-process tokio task | Zero additional infra, simple |
| Track split | Training vs Serving | Independent concerns, parallel development |
| Promotion gating | Operator approval | Production safety for HFT system |