foxhunt/docs/plans/2026-02-27-training-deploy-design.md

Training Pipeline & Model Deployment Design

Date: 2026-02-27 Status: Approved Tracks: Two independent parallel tracks

Overview

Wire the complete training-to-serving pipeline on Kapsule. Track 1 handles training job dispatch (fxt CLI → gRPC → PostgreSQL → K8s GPU Jobs). Track 2 handles model serving and promotion (S3 checkpoints → model loader → inference → operator-gated promotion).

Both tracks share S3 (foxhunt-models/) as the handoff point.

Track 1: Training Pipeline

Goal

fxt train start tft ES.FUT --epochs 50 creates a real GPU training job on Kapsule.

Flow

fxt train start tft ES.FUT --epochs 50
  → gRPC StartTraining (ml_training_service:50053)
  → service.rs handler → JobSpawner::spawn_batch() → PostgreSQL
  → In-process queue consumer (tokio::spawn, 5s poll)
  → JobSpawner::get_next_pending_job()
  → K8sDispatcher::dispatch() → batch/v1 Job
      ├── training container (GPU, train_baseline_rl/supervised)
      └── uploader sidecar (S3 → foxhunt-models/)
  → Job completion callback → update status in PostgreSQL

Existing Code

Component	File	Status
fxt train start	bin/fxt/src/commands/train/start.rs	Done (sends gRPC)
fxt train status	bin/fxt/src/commands/train/status.rs	Done
fxt train list	bin/fxt/src/commands/train/list.rs	Done
K8sDispatcher	services/ml_training_service/src/k8s_dispatcher.rs	Done (builds Jobs)
JobSpawner	services/ml_training_service/src/job_spawner.rs	Done (PostgreSQL CRUD)
Job template	infra/k8s/training/job-template.yaml	Done (reference only)

Changes Needed

1. Wire gRPC handler to JobSpawner: service.rs StartTraining handler must call JobSpawner::spawn_batch() instead of returning a stub response.

2. Add queue consumer loop: In main.rs, tokio::spawn a loop that polls get_next_pending_job() every 5s and calls K8sDispatcher::dispatch().

3. Job completion callback: The uploader sidecar already has CALLBACK_ENDPOINT. Add a gRPC or HTTP endpoint in ml_training_service that receives completion notifications and calls JobSpawner::update_job_status().

4. Deploy training-output-pvc: Currently only training-data-pvc is deployed. Need training-output-pvc for job artifact staging.

5. PostgreSQL schema: Deploy batch_jobs and child_jobs tables. Add sqlx migration.

6. Docker image: Rebuild ml-training-service image with kube-rs K8sDispatcher compiled in.

K8s Resources

• ml-training-service Deployment (exists, needs redeploy with new image)
• training-data-pvc (exists, 10Gi scw-bssd)
• training-output-pvc (new, needs creation)
• s3-credentials Secret (exists)
• scw-registry Secret (exists)
• gpu-training node pool (Scaleway, autoscales 0→N)

Track 2: Model Serving & Validation

Goal

Trained model checkpoints deploy to a serving endpoint with operator-gated promotion and automatic rollback.

Flow

Training job completes → uploader sidecar → S3 (foxhunt-models/)
  → Callback to ml_training_service
  → PromotionManager::register_completion()
      ├── No active model → auto-register
      └── Compare metrics → PendingPromotion / NoImprovement
  → fxt model promote <model-id> (operator approval)
  → DeploymentPipeline::deploy()
      ├── Rolling update
      ├── Health check (inference smoke test)
      └── Auto-rollback on failure

Existing Code

Component	File	Status
S3ModelLoader	crates/model_loader/src/lib.rs	Done (S3 + LRU cache + versioning)
PromotionManager	services/ml_training_service/src/promotion_manager.rs	Done (metric comparison, pending/active tracking)
DeploymentPipeline	services/ml_training_service/src/deployment_pipeline.rs	Done (rolling update, health check, rollback)
ValidationPipeline	services/ml_training_service/src/validation_pipeline.rs	Exists

Changes Needed

1. fxt model commands: Add fxt model promote, fxt model list, fxt model status CLI subcommands.

2. Wire promotion callback: Uploader sidecar completion → PromotionManager::register_completion(). Extract metrics from training output.

3. Model serving deployment: K8s Deployment that loads active model from S3 via S3ModelLoader and serves predictions via gRPC.

4. Validation test: Send sample market data to inference endpoint, verify prediction response shape and latency.

5. Proto additions: Add PromoteModel, ListModels, GetModelStatus RPCs to ml_training proto.

Shared Infrastructure

Resource	Status	Notes
PostgreSQL	Running in foxhunt ns	Needs schema migration
S3 foxhunt-models	Exists	Bucket + credentials
training-data-pvc	Deployed	10Gi, read-only for jobs
training-output-pvc	Needs creation	Staging for job artifacts
scw-registry secret	Deployed	Image pull auth
gpu-training pool	Configured	Autoscales 0→N

Decision Log

Decision	Choice	Rationale
Job dispatch path	DB-backed (JobSpawner)	Full history, batch orchestration, rollback
Queue consumer	In-process tokio task	Zero additional infra, simple
Track split	Training vs Serving	Independent concerns, parallel development
Promotion gating	Operator approval	Production safety for HFT system