foxhunt/docs/plans/2026-02-26-on-demand-training-design.md

On-Demand Training Dispatch via ml_training_service

Goal

Extend ml_training_service to dispatch GPU training jobs as K8s Jobs, collect results via a Rust sidecar uploader, and auto-promote models with operator approval via fxt.

Context

• GitLab CI pipeline handles scheduled/manual full-ensemble retraining (10 models × N symbols)
• ml_training_service handles on-demand single-model retraining via gRPC, dispatches K8s Jobs to the gpu-training pool
• Both coexist: CI for bulk runs, service for operational on-demand jobs

Architecture

fxt CLI / web-gateway
        │ gRPC
        ▼
ml_training_service (no GPU needed)
        │
        ├─ Creates K8s batch/v1 Job ──► gpu-training pool
        │     ├─ main: training binary (exits on completion)
        │     └─ sidecar: training-uploader (K8s 1.34 native sidecar)
        │           ├─ watches /output/DONE marker
        │           ├─ uploads artifacts to S3
        │           └─ calls ReportJobCompletion gRPC
        │
        ├─ Receives completion callback
        │     ├─ registers checkpoint in DB
        │     └─ compares metrics vs active model
        │
        └─ If better → "pending_promotion"
              └─ operator: fxt model approve <id>
                    └─ updates active model pointer

Components

1. K8s Job Dispatcher

New module in ml_training_service. Uses kube crate to create batch/v1 Jobs programmatically. Reuses existing infra/k8s/training/job-template.yaml structure.

Injects into Job spec:

• Model type, symbol, data dir, output dir
• S3 credentials (from K8s secret)
• Callback endpoint (service ClusterIP + port)

Watches Job status via K8s API as fallback if sidecar callback fails.

2. training-uploader (new crate)

Small Rust binary (~300 lines) in crates/training_uploader/.

Behavior:

1. Watches for /output/DONE (success) or /output/FAILED (error) marker
2. Reads /output/metrics.json for training metrics
3. Uploads /output/models/ directory to S3 via object_store crate
4. Calls ReportJobCompletion gRPC on ml_training_service
5. Exits (K8s native sidecar with restartPolicy: Always)

Uses same S3 stack as crates/storage (object_store crate, Scaleway S3-compatible).

3. ReportJobCompletion RPC

New unary RPC in ml_training.proto:

rpc ReportJobCompletion(JobCompletionReport) returns (JobCompletionAck);

message JobCompletionReport {
  string job_id = 1;
  string s3_path = 2;
  bool success = 3;
  string error_message = 4;
  map<string, double> metrics = 5;
}

message JobCompletionAck {
  bool accepted = 1;
  string promotion_status = 2;  // "pending_promotion", "no_improvement", "error"
}

On receipt:

• Registers checkpoint via existing CheckpointManager
• Compares metrics against active model version
• If better → sets status to pending_promotion
• Broadcasts update to SubscribeToTrainingStatus subscribers

4. Model Promotion Flow

New RPCs:

• ListPendingPromotions — models awaiting approval
• ApprovePromotion(model_id) — promote pending model to active
• RejectPromotion(model_id) — archive checkpoint, keep current

On approval: updates active model pointer in DB. model_loader picks up on next refresh cycle.

5. fxt CLI Commands

New subcommands:

• fxt train <model> <symbol> — trigger on-demand training
• fxt train status [job-id] — show running/recent jobs
• fxt model list — show models with active/pending status
• fxt model approve <id> — promote pending model
• fxt model reject <id> — reject pending model

Changes Summary

Component	Change
ml_training_service	K8s Job dispatcher, ReportJobCompletion RPC, promotion logic
ml_training.proto	4 new RPCs (ReportJobCompletion, ListPendingPromotions, Approve, Reject)
New crate training_uploader	S3 upload + gRPC callback binary
Dockerfile.training	Add training-uploader binary
K8s job template	Add native sidecar container
fxt CLI	train and model subcommands
Training binaries	Write DONE/FAILED marker + metrics.json on exit

Unchanged

• GitLab CI pipeline (bulk retraining)
• S3 storage layout and bucket config
• Existing training binaries (minimal change: marker file only)
• CheckpointManager and ModelStorageManager (reused)
• model_loader crate (reads active model pointer)

Not in Scope

• Automatic degradation detection triggering retraining (future — trading_service already tracks metrics in QuestDB)
• Multi-symbol batch dispatch (use CI pipeline)
• Hyperopt via service (use CI pipeline)

Infrastructure

• K8s: 1.34 (native sidecar support confirmed)
• GPU pool: gpu-training (L4)
• S3: Scaleway Object Storage (foxhunt-gitlab-artifacts bucket)
• Registry: rg.fr-par.scw.cloud/foxhunt-ci
• PVCs: training-data-pvc (RO input), training-output-pvc (RWX NFS output)