# On-Demand Training Dispatch via ml_training_service

## Goal

Extend `ml_training_service` to dispatch GPU training jobs as K8s Jobs,
collect results via a Rust sidecar uploader, and auto-promote models with
operator approval via `fxt`.

## Context

- **GitLab CI pipeline** handles scheduled/manual full-ensemble retraining
  (10 models × N symbols)
- **ml_training_service** handles on-demand single-model retraining via gRPC,
  dispatches K8s Jobs to the `gpu-training` pool
- Both coexist: CI for bulk runs, service for operational on-demand jobs

## Architecture

```
fxt CLI / web-gateway
        │ gRPC
        ▼
ml_training_service (no GPU needed)
        │
        ├─ Creates K8s batch/v1 Job ──► gpu-training pool
        │     ├─ main: training binary (exits on completion)
        │     └─ sidecar: training-uploader (K8s 1.34 native sidecar)
        │           ├─ watches /output/DONE marker
        │           ├─ uploads artifacts to S3
        │           └─ calls ReportJobCompletion gRPC
        │
        ├─ Receives completion callback
        │     ├─ registers checkpoint in DB
        │     └─ compares metrics vs active model
        │
        └─ If better → "pending_promotion"
              └─ operator: fxt model approve <id>
                    └─ updates active model pointer
```

## Components

### 1. K8s Job Dispatcher

New module in `ml_training_service`. Uses `kube` crate to create `batch/v1`
Jobs programmatically. Reuses existing `infra/k8s/training/job-template.yaml`
structure.

Injects into Job spec:
- Model type, symbol, data dir, output dir
- S3 credentials (from K8s secret)
- Callback endpoint (service ClusterIP + port)

Watches Job status via K8s API as fallback if sidecar callback fails.

### 2. training-uploader (new crate)

Small Rust binary (~300 lines) in `crates/training_uploader/`.

Behavior:
1. Watches for `/output/DONE` (success) or `/output/FAILED` (error) marker
2. Reads `/output/metrics.json` for training metrics
3. Uploads `/output/models/` directory to S3 via `object_store` crate
4. Calls `ReportJobCompletion` gRPC on `ml_training_service`
5. Exits (K8s native sidecar with `restartPolicy: Always`)

Uses same S3 stack as `crates/storage` (`object_store` crate, Scaleway S3-compatible).

### 3. ReportJobCompletion RPC

New unary RPC in `ml_training.proto`:

```protobuf
rpc ReportJobCompletion(JobCompletionReport) returns (JobCompletionAck);

message JobCompletionReport {
  string job_id = 1;
  string s3_path = 2;
  bool success = 3;
  string error_message = 4;
  map<string, double> metrics = 5;
}

message JobCompletionAck {
  bool accepted = 1;
  string promotion_status = 2;  // "pending_promotion", "no_improvement", "error"
}
```

On receipt:
- Registers checkpoint via existing `CheckpointManager`
- Compares metrics against active model version
- If better → sets status to `pending_promotion`
- Broadcasts update to `SubscribeToTrainingStatus` subscribers

### 4. Model Promotion Flow

New RPCs:
- `ListPendingPromotions` — models awaiting approval
- `ApprovePromotion(model_id)` — promote pending model to active
- `RejectPromotion(model_id)` — archive checkpoint, keep current

On approval: updates active model pointer in DB. `model_loader` picks up
on next refresh cycle.

### 5. fxt CLI Commands

New subcommands:
- `fxt train <model> <symbol>` — trigger on-demand training
- `fxt train status [job-id]` — show running/recent jobs
- `fxt model list` — show models with active/pending status
- `fxt model approve <id>` — promote pending model
- `fxt model reject <id>` — reject pending model

## Changes Summary

| Component | Change |
|-----------|--------|
| `ml_training_service` | K8s Job dispatcher, `ReportJobCompletion` RPC, promotion logic |
| `ml_training.proto` | 4 new RPCs (ReportJobCompletion, ListPendingPromotions, Approve, Reject) |
| New crate `training_uploader` | S3 upload + gRPC callback binary |
| `Dockerfile.training` | Add `training-uploader` binary |
| K8s job template | Add native sidecar container |
| `fxt` CLI | `train` and `model` subcommands |
| Training binaries | Write DONE/FAILED marker + metrics.json on exit |

## Unchanged

- GitLab CI pipeline (bulk retraining)
- S3 storage layout and bucket config
- Existing training binaries (minimal change: marker file only)
- `CheckpointManager` and `ModelStorageManager` (reused)
- `model_loader` crate (reads active model pointer)

## Not in Scope

- Automatic degradation detection triggering retraining (future —
  `trading_service` already tracks metrics in QuestDB)
- Multi-symbol batch dispatch (use CI pipeline)
- Hyperopt via service (use CI pipeline)

## Infrastructure

- K8s: 1.34 (native sidecar support confirmed)
- GPU pool: `gpu-training` (L4)
- S3: Scaleway Object Storage (`foxhunt-gitlab-artifacts` bucket)
- Registry: `rg.fr-par.scw.cloud/foxhunt-ci`
- PVCs: `training-data-pvc` (RO input), `training-output-pvc` (RWX NFS output)
