foxhunt/docs/plans/2026-02-26-on-demand-training-implementation.md

On-Demand Training Dispatch Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Enable ml_training_service to dispatch GPU training jobs as K8s Jobs with a Rust sidecar uploader, S3 artifact storage, and model promotion with fxt CLI approval.

Architecture: The service creates K8s batch/v1 Jobs programmatically via the kube crate. Each Job has a training container and a native sidecar (training-uploader) that watches for completion, uploads to S3, and calls back to the service via gRPC. The service compares metrics and queues models for promotion, approved via fxt model approve.

Tech Stack: Rust, tonic gRPC, kube-rs, object_store (S3), K8s 1.34 native sidecars, protobuf

---

Task 1: Add DONE/FAILED marker + metrics.json to training binaries

The training binaries currently exit with Ok(()) or Err. We need them to write a marker file and a metrics summary so the sidecar knows when training is done and what happened.

Files:

• Create: crates/ml/examples/baseline_common/completion.rs
• Modify: crates/ml/examples/baseline_common/mod.rs
• Modify: crates/ml/examples/train_baseline_supervised.rs
• Modify: crates/ml/examples/train_baseline_rl.rs

Step 1: Create the completion module

Create crates/ml/examples/baseline_common/completion.rs:

//! Training completion marker and metrics writer.
//!
//! Writes DONE/FAILED marker files and metrics.json to the output directory
//! so the training-uploader sidecar can detect completion and report results.

use std::path::Path;
use tracing::{error, info};

/// Metrics collected during training for the sidecar to report back.
#[derive(serde::Serialize)]
pub struct CompletionMetrics {
    pub model: String,
    pub symbol: String,
    pub best_val_loss: Option<f64>,
    pub sharpe_ratio: Option<f64>,
    pub epochs_completed: usize,
    pub folds_completed: usize,
}

/// Write a DONE marker and metrics.json on successful training completion.
pub fn write_success_marker(output_dir: &Path, metrics: &CompletionMetrics) {
    let metrics_path = output_dir.join("metrics.json");
    match serde_json::to_string_pretty(metrics) {
        Ok(json) => {
            if let Err(e) = std::fs::write(&metrics_path, &json) {
                error!("Failed to write metrics.json: {}", e);
            } else {
                info!("Wrote metrics to {}", metrics_path.display());
            }
        }
        Err(e) => error!("Failed to serialize metrics: {}", e),
    }

    let done_path = output_dir.join("DONE");
    if let Err(e) = std::fs::write(&done_path, "") {
        error!("Failed to write DONE marker: {}", e);
    } else {
        info!("Wrote DONE marker to {}", done_path.display());
    }
}

/// Write a FAILED marker with error details.
pub fn write_failure_marker(output_dir: &Path, error_msg: &str) {
    let failed_path = output_dir.join("FAILED");
    if let Err(e) = std::fs::write(&failed_path, error_msg) {
        error!("Failed to write FAILED marker: {}", e);
    } else {
        info!("Wrote FAILED marker to {}", failed_path.display());
    }
}

Step 2: Export from baseline_common/mod.rs

Add to crates/ml/examples/baseline_common/mod.rs:

pub mod completion;

Step 3: Wire into train_baseline_supervised.rs

At the end of main(), after the training loop completes, call write_success_marker with collected metrics. Wrap the training body in a closure or match so failures call write_failure_marker.

Key pattern — the --output-dir arg already exists. After the fold loop:

use baseline_common::completion::{write_success_marker, write_failure_marker, CompletionMetrics};

// After fold loop, before Ok(())
let metrics = CompletionMetrics {
    model: args.model.clone(),
    symbol: args.symbol.clone(),
    best_val_loss: fold_results.iter().map(|(_, l)| *l).reduce(f64::min),
    sharpe_ratio: None, // supervised models don't compute sharpe during training
    epochs_completed: args.epochs,
    folds_completed: fold_results.len(),
};
write_success_marker(&args.output_dir, &metrics);

Add a top-level catch for errors:

fn main() -> Result<()> {
    // ... existing setup ...
    let result = run_training(&args);
    if let Err(ref e) = result {
        write_failure_marker(&args.output_dir, &format!("{:#}", e));
    }
    result
}

This requires extracting the training body into a run_training() function.

Step 4: Wire into train_baseline_rl.rs

Same pattern. The RL binary already has --output-dir. Add marker writes at success/failure points.

Step 5: Verify

Run: SQLX_OFFLINE=true cargo check -p ml --example train_baseline_supervised --example train_baseline_rl Expected: compiles with zero errors

Step 6: Commit

git add crates/ml/examples/baseline_common/completion.rs crates/ml/examples/baseline_common/mod.rs \
  crates/ml/examples/train_baseline_supervised.rs crates/ml/examples/train_baseline_rl.rs
git commit -m "feat(ml): add DONE/FAILED marker + metrics.json to training binaries"

---

Task 2: Create training_uploader crate

New Rust binary that runs as a K8s native sidecar. Watches for DONE/FAILED marker, uploads model artifacts to S3, and reports back to ml_training_service via gRPC.

Files:

• Create: crates/training_uploader/Cargo.toml
• Create: crates/training_uploader/src/main.rs
• Modify: Cargo.toml (workspace members)

Step 1: Create Cargo.toml

Create crates/training_uploader/Cargo.toml:

[package]
name = "training_uploader"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
authors.workspace = true
license.workspace = true
description = "K8s sidecar: uploads training artifacts to S3 and reports completion via gRPC"

[dependencies]
tokio.workspace = true
anyhow.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true
serde.workspace = true
serde_json.workspace = true
clap.workspace = true
object_store = { workspace = true, features = ["aws"] }
bytes.workspace = true
tonic.workspace = true
tonic-prost.workspace = true
prost.workspace = true

[build-dependencies]
tonic-prost-build.workspace = true
prost-build.workspace = true

Step 2: Create build.rs

Create crates/training_uploader/build.rs:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_prost_build::configure()
        .build_server(false)
        .build_client(true)
        .compile_well_known_types(true)
        .extern_path(".google.protobuf", "::prost_types")
        .compile_protos(
            &["../../services/ml_training_service/proto/ml_training.proto"],
            &["../../services/ml_training_service/proto"],
        )?;
    println!("cargo:rerun-if-changed=../../services/ml_training_service/proto/ml_training.proto");
    Ok(())
}

Step 3: Create main.rs

Create crates/training_uploader/src/main.rs. This is the core binary (~250 lines):

//! Training Uploader — K8s native sidecar
//!
//! Watches for DONE/FAILED marker in the training output directory,
//! uploads model artifacts to S3, and reports completion to ml_training_service.

#![allow(unused_crate_dependencies)]

use std::path::{Path, PathBuf};
use std::time::Duration;

use anyhow::{Context, Result};
use bytes::Bytes;
use clap::Parser;
use object_store::aws::AmazonS3Builder;
use object_store::{ObjectStore, PutPayload};
use tracing::{error, info, warn};

mod proto {
    pub mod ml_training {
        tonic::include_proto!("ml_training");
    }
    pub use ml_training::*;
}

#[derive(Parser, Debug)]
#[command(name = "training-uploader")]
struct Args {
    /// Output directory to watch for DONE/FAILED marker
    #[arg(long, default_value = "/output")]
    output_dir: PathBuf,

    /// S3 bucket name
    #[arg(long, env = "S3_BUCKET")]
    s3_bucket: String,

    /// S3 endpoint URL (Scaleway)
    #[arg(long, env = "S3_ENDPOINT")]
    s3_endpoint: String,

    /// S3 region
    #[arg(long, env = "S3_REGION", default_value = "fr-par")]
    s3_region: String,

    /// S3 prefix path for this job's artifacts
    #[arg(long, env = "S3_PREFIX")]
    s3_prefix: String,

    /// gRPC endpoint of ml_training_service
    #[arg(long, env = "CALLBACK_ENDPOINT")]
    callback_endpoint: String,

    /// Job ID to report back
    #[arg(long, env = "JOB_ID")]
    job_id: String,

    /// Poll interval in seconds
    #[arg(long, default_value = "10")]
    poll_interval: u64,

    /// Maximum wait time in seconds (default: match Job activeDeadlineSeconds)
    #[arg(long, default_value = "3600")]
    max_wait: u64,
}

#[tokio::main]
async fn main() -> Result<()> {
    tracing_subscriber::fmt()
        .with_max_level(tracing::Level::INFO)
        .with_target(false)
        .init();

    let args = Args::parse();
    info!("Training uploader started for job {}", args.job_id);
    info!("Watching: {}", args.output_dir.display());
    info!("S3 target: s3://{}/{}", args.s3_bucket, args.s3_prefix);

    // Wait for completion marker
    let (success, error_msg) = wait_for_completion(&args.output_dir, args.poll_interval, args.max_wait).await;

    // Read metrics if available
    let metrics = read_metrics(&args.output_dir);

    // Upload artifacts to S3 (even on failure — partial results may be useful)
    let s3_path = match upload_artifacts(&args).await {
        Ok(path) => {
            info!("Uploaded artifacts to s3://{}/{}", args.s3_bucket, path);
            path
        }
        Err(e) => {
            error!("S3 upload failed: {:#}", e);
            String::new()
        }
    };

    // Report back to ml_training_service
    if let Err(e) = report_completion(&args, success, &s3_path, &error_msg, &metrics).await {
        error!("gRPC callback failed: {:#}", e);
        // Don't fail the sidecar — the service can poll K8s Job status as fallback
    }

    info!("Uploader complete (success={})", success);
    Ok(())
}

async fn wait_for_completion(dir: &Path, poll_secs: u64, max_wait: u64) -> (bool, String) {
    let done_path = dir.join("DONE");
    let failed_path = dir.join("FAILED");
    let interval = Duration::from_secs(poll_secs);
    let deadline = tokio::time::Instant::now() + Duration::from_secs(max_wait);

    loop {
        if done_path.exists() {
            info!("DONE marker found");
            return (true, String::new());
        }
        if failed_path.exists() {
            let msg = std::fs::read_to_string(&failed_path).unwrap_or_default();
            warn!("FAILED marker found: {}", msg);
            return (false, msg);
        }
        if tokio::time::Instant::now() > deadline {
            return (false, "Timeout waiting for training completion".to_owned());
        }
        tokio::time::sleep(interval).await;
    }
}

fn read_metrics(dir: &Path) -> std::collections::HashMap<String, f64> {
    let metrics_path = dir.join("metrics.json");
    if !metrics_path.exists() {
        return std::collections::HashMap::new();
    }
    match std::fs::read_to_string(&metrics_path) {
        Ok(content) => serde_json::from_str::<serde_json::Value>(&content)
            .ok()
            .and_then(|v| v.as_object().cloned())
            .map(|obj| {
                obj.into_iter()
                    .filter_map(|(k, v)| v.as_f64().map(|f| (k, f)))
                    .collect()
            })
            .unwrap_or_default(),
        Err(_) => std::collections::HashMap::new(),
    }
}

async fn upload_artifacts(args: &Args) -> Result<String> {
    let store = AmazonS3Builder::new()
        .with_bucket_name(&args.s3_bucket)
        .with_endpoint(&args.s3_endpoint)
        .with_region(&args.s3_region)
        .with_allow_http(false)
        .build()
        .context("Failed to build S3 client")?;

    // Walk output directory and upload all files
    let mut uploaded = 0usize;
    upload_dir_recursive(&store, &args.output_dir, &args.s3_prefix, &mut uploaded).await?;
    info!("Uploaded {} files to S3", uploaded);
    Ok(args.s3_prefix.clone())
}

#[async_recursion::async_recursion]
async fn upload_dir_recursive(
    store: &dyn ObjectStore,
    local_dir: &Path,
    s3_prefix: &str,
    count: &mut usize,
) -> Result<()> {
    let entries = std::fs::read_dir(local_dir)
        .with_context(|| format!("Failed to read dir {}", local_dir.display()))?;

    for entry in entries.flatten() {
        let path = entry.path();
        let name = entry.file_name().to_string_lossy().to_string();

        if path.is_dir() {
            let sub_prefix = format!("{}/{}", s3_prefix, name);
            upload_dir_recursive(store, &path, &sub_prefix, count).await?;
        } else {
            let s3_key = format!("{}/{}", s3_prefix, name);
            let data = std::fs::read(&path)
                .with_context(|| format!("Failed to read {}", path.display()))?;
            let s3_path = object_store::path::Path::from(s3_key.as_str());
            store
                .put(&s3_path, PutPayload::from(Bytes::from(data)))
                .await
                .with_context(|| format!("Failed to upload {}", s3_key))?;
            *count += 1;
        }
    }
    Ok(())
}

async fn report_completion(
    args: &Args,
    success: bool,
    s3_path: &str,
    error_msg: &str,
    metrics: &std::collections::HashMap<String, f64>,
) -> Result<()> {
    use proto::ml_training_service_client::MlTrainingServiceClient;
    use proto::JobCompletionReport;

    let mut client = MlTrainingServiceClient::connect(args.callback_endpoint.clone())
        .await
        .context("Failed to connect to ml_training_service")?;

    let report = JobCompletionReport {
        job_id: args.job_id.clone(),
        s3_path: s3_path.to_owned(),
        success,
        error_message: error_msg.to_owned(),
        metrics: metrics.iter().map(|(k, v)| (k.clone(), *v)).collect(),
    };

    let response = client.report_job_completion(report).await?;
    let ack = response.into_inner();
    info!(
        "Service acknowledged: accepted={}, promotion_status={}",
        ack.accepted, ack.promotion_status
    );
    Ok(())
}

Note: This references JobCompletionReport and JobCompletionAck proto messages that don't exist yet. They'll be added in Task 3 (proto update). The crate won't compile until then — that's expected.

Step 4: Add to workspace

In root Cargo.toml, add "crates/training_uploader" to the workspace members list.

Step 5: Add async-recursion dependency

Add async-recursion = "1.0" to crates/training_uploader/Cargo.toml dependencies.

Step 6: Commit (will not compile yet — proto messages needed from Task 3)

git add crates/training_uploader/ Cargo.toml
git commit -m "feat: add training-uploader sidecar crate (proto pending)"

---

Task 3: Add new RPCs to ml_training.proto

Add the ReportJobCompletion, ListPendingPromotions, ApprovePromotion, and RejectPromotion RPCs.

Files:

• Modify: services/ml_training_service/proto/ml_training.proto

Step 1: Add RPCs to the service definition

In the service MLTrainingService block, add after the batch tuning RPCs:

  // Job Completion Callback (called by training-uploader sidecar)
  rpc ReportJobCompletion(JobCompletionReport) returns (JobCompletionAck);

  // Model Promotion Management
  rpc ListPendingPromotions(ListPendingPromotionsRequest) returns (ListPendingPromotionsResponse);
  rpc ApprovePromotion(ApprovePromotionRequest) returns (ApprovePromotionResponse);
  rpc RejectPromotion(RejectPromotionRequest) returns (RejectPromotionResponse);

Step 2: Add the message definitions

Add at the end of the proto file:

// --- Job Completion Callback (from training-uploader sidecar) ---

message JobCompletionReport {
  string job_id = 1;
  string s3_path = 2;
  bool success = 3;
  string error_message = 4;
  map<string, double> metrics = 5;
}

message JobCompletionAck {
  bool accepted = 1;
  string promotion_status = 2;  // "pending_promotion", "no_improvement", "registered", "error"
}

// --- Model Promotion ---

message ListPendingPromotionsRequest {}

message ListPendingPromotionsResponse {
  repeated PendingPromotion promotions = 1;
}

message PendingPromotion {
  string model_id = 1;
  string model_type = 2;
  string symbol = 3;
  string s3_path = 4;
  map<string, double> new_metrics = 5;
  map<string, double> current_metrics = 6;
  int64 trained_at = 7;
  string job_id = 8;
}

message ApprovePromotionRequest {
  string model_id = 1;
}

message ApprovePromotionResponse {
  bool success = 1;
  string message = 2;
}

message RejectPromotionRequest {
  string model_id = 1;
  string reason = 2;
}

message RejectPromotionResponse {
  bool success = 1;
  string message = 2;
}

Step 3: Verify proto compiles

Run: SQLX_OFFLINE=true cargo check -p ml_training_service -p training_uploader Expected: compiles (the new RPCs won't have implementations yet — tonic generates default unimplemented!() stubs)

Step 4: Commit

git add services/ml_training_service/proto/ml_training.proto
git commit -m "feat(proto): add ReportJobCompletion and model promotion RPCs"

---

Task 4: Implement K8s Job Dispatcher

New module in ml_training_service that creates K8s batch/v1 Jobs using the kube crate.

Files:

• Create: services/ml_training_service/src/k8s_dispatcher.rs
• Modify: services/ml_training_service/Cargo.toml
• Modify: services/ml_training_service/src/main.rs (wire up)

Step 1: Add kube dependencies

Add to services/ml_training_service/Cargo.toml under [dependencies]:

# K8s API for job dispatch
kube = { version = "0.98", features = ["runtime", "client", "derive"] }
k8s-openapi = { version = "0.23", features = ["latest"] }

Also add to root Cargo.toml workspace dependencies if not already present.

Step 2: Create k8s_dispatcher.rs

Create services/ml_training_service/src/k8s_dispatcher.rs:

This module provides:

//! K8s Job Dispatcher
//!
//! Creates batch/v1 Jobs on the gpu-training pool for on-demand model training.
//! Each Job includes a training container and a native sidecar (training-uploader)
//! that handles S3 upload and gRPC completion callback.

use std::collections::BTreeMap;

use anyhow::{Context, Result};
use k8s_openapi::api::batch::v1::{Job, JobSpec};
use k8s_openapi::api::core::v1::{
    Container, EnvVar, PodSpec, PodTemplateSpec, ResourceRequirements,
    Volume, VolumeMount, PersistentVolumeClaimVolumeSource, EmptyDirVolumeSource,
};
use k8s_openapi::apimachinery::pkg::api::resource::Quantity;
use k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta;
use kube::{Api, Client};
use tracing::{info, error};
use uuid::Uuid;

/// Configuration for the K8s job dispatcher
#[derive(Debug, Clone)]
pub struct DispatcherConfig {
    pub namespace: String,
    pub training_image: String,
    pub uploader_image: String,
    pub data_pvc: String,
    pub output_pvc: String,
    pub s3_bucket: String,
    pub s3_endpoint: String,
    pub s3_region: String,
    pub callback_endpoint: String, // e.g. "http://ml-training-service.foxhunt:50053"
}

impl Default for DispatcherConfig {
    fn default() -> Self {
        Self {
            namespace: std::env::var("K8S_NAMESPACE").unwrap_or_else(|_| "foxhunt".to_owned()),
            training_image: std::env::var("TRAINING_IMAGE")
                .unwrap_or_else(|_| "rg.fr-par.scw.cloud/foxhunt-ci/training:latest".to_owned()),
            uploader_image: std::env::var("UPLOADER_IMAGE")
                .unwrap_or_else(|_| "rg.fr-par.scw.cloud/foxhunt-ci/training:latest".to_owned()),
            data_pvc: "training-data-pvc".to_owned(),
            output_pvc: "training-output-pvc".to_owned(),
            s3_bucket: std::env::var("S3_BUCKET").unwrap_or_else(|_| "foxhunt-models".to_owned()),
            s3_endpoint: std::env::var("S3_ENDPOINT")
                .unwrap_or_else(|_| "https://s3.fr-par.scw.cloud".to_owned()),
            s3_region: "fr-par".to_owned(),
            callback_endpoint: std::env::var("CALLBACK_ENDPOINT")
                .unwrap_or_else(|_| "http://ml-training-service.foxhunt.svc.cluster.local:50053".to_owned()),
        }
    }
}

/// Dispatch a training job to K8s
pub struct K8sDispatcher {
    client: Client,
    config: DispatcherConfig,
}

/// Parameters for a training job dispatch
#[derive(Debug, Clone)]
pub struct TrainingJobParams {
    pub job_id: Uuid,
    pub model_type: String,   // e.g. "dqn", "tft", "mamba2"
    pub symbol: String,       // e.g. "ES.FUT"
    pub epochs: u32,
    pub binary: String,       // e.g. "train_baseline_rl" or "train_baseline_supervised"
}

The K8sDispatcher should have:

• async fn new() -> Result<Self> — creates kube Client from in-cluster config
• async fn dispatch(&self, params: &TrainingJobParams) -> Result<String> — creates the Job, returns the K8s Job name
• fn build_job_spec(&self, params: &TrainingJobParams) -> Job — builds the Job manifest with training container + sidecar
• async fn get_job_status(&self, job_name: &str) -> Result<Option<String>> — polls Job status
• async fn delete_job(&self, job_name: &str) -> Result<()> — cleanup

The build_job_spec method should produce a Job matching the structure in infra/k8s/training/job-template.yaml but with the sidecar added:

• Main container: training — runs the specified binary with --model, --symbol, --data-dir=/data, --output-dir=/output/{job_id}
• Sidecar container: uploader — runs training-uploader with --output-dir=/output/{job_id}, S3 config, callback endpoint, job ID
• Sidecar uses restartPolicy: Always (K8s 1.28+ native sidecar)
• Shared volumes: training-data (PVC, RO), output (PVC, RW)
• Node selector: gpu-training pool
• GPU requests: 1 nvidia.com/gpu
• Labels: foxhunt/job-type: training, foxhunt/model: {model}, foxhunt/job-id: {uuid}

Step 3: Verify

Run: SQLX_OFFLINE=true cargo check -p ml_training_service Expected: compiles

Step 4: Commit

git add services/ml_training_service/src/k8s_dispatcher.rs services/ml_training_service/Cargo.toml Cargo.toml
git commit -m "feat(service): add K8s Job dispatcher for GPU training"

---

Task 5: Implement ReportJobCompletion + promotion logic

Wire the new ReportJobCompletion RPC in the service. On completion callback: register checkpoint, compare metrics, set promotion status.

Files:

• Create: services/ml_training_service/src/promotion_manager.rs
• Modify: services/ml_training_service/src/service.rs

Step 1: Create promotion_manager.rs

This module tracks models pending promotion and compares metrics:

//! Model Promotion Manager
//!
//! Tracks models awaiting operator approval for promotion to active status.
//! Compares new model metrics against currently active model to determine
//! if promotion should be offered.

use std::collections::HashMap;
use std::sync::Arc;

use chrono::{DateTime, Utc};
use tokio::sync::RwLock;
use tracing::info;
use uuid::Uuid;

#[derive(Debug, Clone)]
pub struct PendingModel {
    pub model_id: Uuid,
    pub model_type: String,
    pub symbol: String,
    pub s3_path: String,
    pub new_metrics: HashMap<String, f64>,
    pub current_metrics: HashMap<String, f64>,
    pub trained_at: DateTime<Utc>,
    pub job_id: String,
}

pub struct PromotionManager {
    pending: Arc<RwLock<HashMap<Uuid, PendingModel>>>,
    active_models: Arc<RwLock<HashMap<(String, String), ActiveModel>>>, // (model_type, symbol) -> active
}

#[derive(Debug, Clone)]
pub struct ActiveModel {
    pub model_id: Uuid,
    pub s3_path: String,
    pub metrics: HashMap<String, f64>,
    pub promoted_at: DateTime<Utc>,
}

Methods:

• new() — initialize empty
• register_completion(job_id, model_type, symbol, s3_path, metrics) -> PromotionStatus — compare against active, return status
• list_pending() -> Vec<PendingModel>
• approve(model_id) -> Result<()> — move to active, remove from pending
• reject(model_id, reason) -> Result<()> — remove from pending
• get_active(model_type, symbol) -> Option<ActiveModel>

Comparison logic: if new model has better best_val_loss (lower) or better sharpe_ratio (higher) than active, mark as pending_promotion. If no active model exists, auto-promote.

Step 2: Implement the gRPC handlers in service.rs

Add implementations for the 4 new RPCs in the MlTrainingService trait impl. Each delegates to PromotionManager.

Step 3: Wire PromotionManager into MLTrainingServiceImpl

Add promotion_manager: Arc<PromotionManager> field. Initialize in main.rs.

Step 4: Verify

Run: SQLX_OFFLINE=true cargo check -p ml_training_service

Step 5: Commit

git add services/ml_training_service/src/promotion_manager.rs services/ml_training_service/src/service.rs \
  services/ml_training_service/src/main.rs
git commit -m "feat(service): implement ReportJobCompletion and model promotion"

---

Task 6: Wire StartTraining to K8s dispatch

Modify the existing StartTraining RPC to dispatch a K8s Job instead of running training in-process.

Files:

• Modify: services/ml_training_service/src/service.rs
• Modify: services/ml_training_service/src/main.rs

Step 1: Add K8sDispatcher to MLTrainingServiceImpl

Add dispatcher: Arc<K8sDispatcher> field. Initialize in main.rs with K8sDispatcher::new().await.

Step 2: Modify StartTraining handler

The current start_training handler calls self.orchestrator.submit_training_job() which runs in-process. Change it to:

1. Determine the correct binary (RL vs supervised) from model_type
2. Create TrainingJobParams
3. Call self.dispatcher.dispatch(&params).await
4. Record the job in the orchestrator for status tracking
5. Return the job_id

The orchestrator's in-memory job tracking stays — it just no longer runs the training itself. Job status updates will come from the ReportJobCompletion callback instead.

Step 3: Verify

Run: SQLX_OFFLINE=true cargo check -p ml_training_service

Step 4: Commit

git add services/ml_training_service/src/service.rs services/ml_training_service/src/main.rs
git commit -m "feat(service): wire StartTraining to K8s Job dispatch"

---

Task 7: Add fxt model subcommand

Add fxt model list, fxt model approve, fxt model reject commands.

Files:

• Create: bin/fxt/src/commands/model/mod.rs
• Create: bin/fxt/src/commands/model/list.rs
• Create: bin/fxt/src/commands/model/approve.rs
• Create: bin/fxt/src/commands/model/reject.rs
• Modify: bin/fxt/src/commands/mod.rs
• Modify: bin/fxt/src/main.rs (add Model subcommand to CLI)

Step 1: Create model command module

bin/fxt/src/commands/model/mod.rs:

pub mod approve;
pub mod list;
pub mod reject;

use anyhow::Result;
use clap::Subcommand;

#[derive(Subcommand, Debug)]
pub enum ModelCommand {
    /// List models with active/pending status
    List,
    /// Approve a pending model promotion
    Approve {
        /// Model ID to approve
        model_id: String,
    },
    /// Reject a pending model promotion
    Reject {
        /// Model ID to reject
        model_id: String,
        /// Reason for rejection
        #[arg(long, default_value = "operator rejected")]
        reason: String,
    },
}

pub async fn execute_model_command(
    command: ModelCommand,
    api_gateway_url: &str,
    jwt_token: &str,
) -> Result<()> {
    match command {
        ModelCommand::List => list::run(api_gateway_url, jwt_token).await,
        ModelCommand::Approve { model_id } => approve::run(api_gateway_url, jwt_token, &model_id).await,
        ModelCommand::Reject { model_id, reason } => reject::run(api_gateway_url, jwt_token, &model_id, &reason).await,
    }
}

Step 2: Implement list, approve, reject

Each makes a gRPC call to the api_gateway (which proxies to ml_training_service). Follow the same pattern as the existing train/list.rs and train/status.rs commands.

• list.rs — calls ListPendingPromotions, displays table with model_type, symbol, metrics comparison, trained_at
• approve.rs — calls ApprovePromotion, prints success/failure
• reject.rs — calls RejectPromotion, prints success/failure

Step 3: Wire into commands/mod.rs and main.rs

Add pub mod model; to commands/mod.rs. Add Model variant to the main CLI enum in main.rs. Follow the pattern of the existing Train and Tune commands.

Step 4: Verify

Run: SQLX_OFFLINE=true cargo check -p fxt

Step 5: Commit

git add bin/fxt/src/commands/model/ bin/fxt/src/commands/mod.rs bin/fxt/src/main.rs
git commit -m "feat(fxt): add model list/approve/reject commands"

---

Task 8: Add fxt train start subcommand

The existing fxt train has list and status. Add start to trigger on-demand training.

Files:

• Create: bin/fxt/src/commands/train/start.rs
• Modify: bin/fxt/src/commands/train/mod.rs

Step 1: Create start.rs

//! fxt train start — trigger on-demand training job

use anyhow::Result;
use clap::Args;

#[derive(Args, Debug)]
pub struct StartCommand {
    /// Model type (dqn, ppo, tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion)
    pub model: String,

    /// Symbol to train on (e.g. ES.FUT)
    pub symbol: String,

    /// Number of training epochs
    #[arg(long, default_value = "50")]
    pub epochs: u32,
}

The run() method calls StartTraining gRPC, prints the job ID and a status URL.

Step 2: Wire into train/mod.rs

Add Start variant to TrainCommand enum. Add pub mod start;.

Step 3: Verify

Run: SQLX_OFFLINE=true cargo check -p fxt

Step 4: Commit

git add bin/fxt/src/commands/train/start.rs bin/fxt/src/commands/train/mod.rs
git commit -m "feat(fxt): add train start command for on-demand training"

---

Task 9: Update Dockerfile and K8s job template

Add training-uploader binary to the training Docker image and update the job template with the native sidecar.

Files:

• Modify: infra/docker/Dockerfile.training (add training-uploader to build)
• Modify: infra/docker/Dockerfile.training-runtime (copy uploader binary)
• Modify: infra/k8s/training/job-template.yaml (add sidecar)
• Modify: .gitlab-ci.yml (compile training_uploader in compile-services stage)

Step 1: Update Dockerfile.training-runtime

Add COPY --from=build training_uploader /usr/local/bin/training-uploader (or equivalent pattern matching the existing Dockerfile).

Step 2: Update job-template.yaml

Add the sidecar container alongside the training container:

      # Native sidecar (K8s 1.28+) — uploads artifacts on completion
      initContainers:
        - name: uploader
          image: rg.fr-par.scw.cloud/foxhunt-ci/training:latest
          restartPolicy: Always  # Makes this a native sidecar
          command: ["/usr/local/bin/training-uploader"]
          env:
            - name: JOB_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['foxhunt/job-id']
            - name: S3_BUCKET
              value: foxhunt-models
            - name: S3_ENDPOINT
              value: https://s3.fr-par.scw.cloud
            - name: S3_PREFIX
              value: "models/$(JOB_ID)"
            - name: CALLBACK_ENDPOINT
              value: http://ml-training-service.foxhunt.svc.cluster.local:50053
          envFrom:
            - secretRef:
                name: s3-credentials
          volumeMounts:
            - name: output
              mountPath: /output
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi

Step 3: Update .gitlab-ci.yml compile-services stage

Add training_uploader to the cargo build command that builds training binaries:

    - cargo build --release -p training_uploader

And copy it to build-out:

    cp target/release/training_uploader build-out/
    strip build-out/training_uploader

Step 4: Commit

git add infra/docker/ infra/k8s/training/job-template.yaml .gitlab-ci.yml
git commit -m "infra: add training-uploader sidecar to Docker image and K8s job template"

---

Task 10: Integration test

End-to-end test: verify the service can build a Job spec, the proto compiles, and the fxt commands parse correctly.

Files:

• Modify: services/ml_training_service/src/k8s_dispatcher.rs (add unit tests)

Step 1: Add unit tests for Job spec building

Test that build_job_spec() produces a valid Job with:

• Correct main container (binary, args, env)
• Sidecar container with correct env vars
• GPU resource requests
• Correct volume mounts
• Correct labels and node selector

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_build_job_spec() {
        let config = DispatcherConfig::default();
        let dispatcher = K8sDispatcher { client: todo!(), config };
        // ... test job spec fields
    }
}

Since kube::Client requires a cluster, extract build_job_spec as a pure function that takes &DispatcherConfig and &TrainingJobParams and returns Job. This makes it testable without a cluster.

Step 2: Test fxt CLI parsing

Run: SQLX_OFFLINE=true cargo check -p fxt

Verify fxt train start --help and fxt model --help parse correctly.

Step 3: Full workspace check

Run: SQLX_OFFLINE=true cargo check --workspace Expected: zero errors

Step 4: Commit

git add services/ml_training_service/src/k8s_dispatcher.rs
git commit -m "test(service): add K8s dispatcher unit tests"

---

Dependency Order

Task 1 (markers)  ──┐
Task 2 (uploader)  ──┼── Task 3 (proto) ── Task 5 (completion handler) ── Task 6 (wire StartTraining)
                     │                  └── Task 4 (dispatcher) ──────────┘
                     │
Task 7 (fxt model) ──┤
Task 8 (fxt train start) ─┤
                           └── Task 9 (Docker/K8s) ── Task 10 (integration test)

Tasks 1, 2, 7, 8 can start in parallel. Task 3 unblocks 4 and 5. Task 9 is last infra change.