gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(rl): R8 — alpha_rl_train CLI + Argo template + dispatcher

Browse Source 
Closes the rebuild plan's R8 scope: production runner shape for the
integrated RL trainer. Three artifacts wired end-to-end:

  1. `crates/ml-alpha/examples/alpha_rl_train.rs` — clap CLI driving
     `IntegratedTrainer::step_with_lobsim` against MBP-10 windows
     loaded via `MultiHorizonLoader::next_sequence_pair` (R2) for
     true `(s_t, s_{t+1})` adjacency. Per `feedback_mbp10_mandatory`,
     `--mbp10-data-dir` is required — no synthetic-data fallback in
     the production path.
  2. `infra/k8s/argo/alpha-rl-template.yaml` — WorkflowTemplate
     mirroring alpha-perception's DAG (check-cache → ensure-binary →
     train; warmup-gpu parallel). Binary cache slot is
     `/data/bin/<sha>/alpha_rl_train` (distinct from `alpha_train`
     so the two binaries coexist at the same SHA).
  3. `scripts/argo-alpha-rl.sh` — dispatcher with three rebuild-plan
     guards baked in.

## Dispatcher guards (per the rebuild plan's feedback list)

`feedback_default_to_l40s_pool` (2026-05-09): default `--gpu-pool`
is `ci-training-l40s` (sm_89). H100 (sm_90) is opt-in for production
scale-up only. Cubins must match the device, so the dispatcher
derives `cuda-compute-cap` from the pool name and threads it into
the workflow params.

`feedback_argo_template_must_apply` (2026-05-21 canonical incident):
`argo submit --from=wftmpl/<name>` reads the cluster CRD, NOT the
on-disk YAML; unknown `-p` parameters silently no-op without a prior
`kubectl apply`. Dispatcher applies the local template BEFORE every
submission (overrideable via `--skip-template-apply` for the rare
case where you've already applied manually).

`feedback_push_before_deploy` (2026-05-20 canonical incident): the
in-cluster `ensure-binary` pod fetches source from `origin/<branch>`,
NOT the local working tree. Submitting before `git push` deploys the
last-pushed SHA, which can lag local diff by N commits. Dispatcher
verifies `git rev-parse HEAD == git rev-parse origin/<branch>` and
hard-errors with the explicit push command otherwise. Bypass via
`--skip-push-check` (only when intentionally deploying a previously-
pushed SHA via `--sha`).

## CLI: gate G8 (NaN abort)

Per `feedback_stop_on_anomaly` + `feedback_kill_runs_on_anomaly_quickly`,
the CLI checks every per-head loss (l_bce / l_q / l_pi / l_v / l_aux
/ l_total) for finiteness after each `step_with_lobsim` call.
Non-finite at any step → write summary with `nan_abort_step` set →
`process::exit(2)`. R9's cluster smoke tail-watcher kills the
workflow on the non-zero exit code, satisfying gate G8 from the
rebuild plan.

## CLI: knobs that ARE on the CLI

Structural / boundary parameters only (per
`pearl_controller_anchors_isv_driven`: every adaptive knob lives in
ISV, not CLI flags):
  * `--mbp10-data-dir / --predecoded-dir / --out` — I/O paths.
  * `--n-steps` — wall-budget control (1000 R9 smoke / 50k+ prod).
  * `--seq-len / --n-backtests / --per-capacity` — structural
    sizing. seq_len threads into the loader's multi-resolution
    `1:<seq_len>` config; n_backtests into both LobSimCuda and
    PerceptionTrainerConfig.n_batch.
  * `--seed` — reproducibility per
    `pearl_scoped_init_seed_for_reproducibility` (forks deterministic
    sub-seeds for dqn / ppo / per).
  * `--instrument-mode` — MBP-10 filter (all / front-month / id=N).
  * `--gpu-idx` — CUDA device selection.

What's NOT on the CLI: γ / τ / ε / entropy_coef / per_α / reward_scale
/ per-head LRs — all live in ISV[400..417] and are driven by R5's
controllers from EMA-tracked diagnostics. Per the rebuild plan
A1: "every adaptive bound is signal-driven, not tuned."

## Cluster smoke entry point

```bash
# R9 validation smoke (after pre-cluster local CUDA tests green).
./scripts/argo-alpha-rl.sh --n-steps 1000 --instrument-mode front-month

# Production scale-up (gated by R9's multi-fold pass).
./scripts/argo-alpha-rl.sh --n-steps 50000 --n-backtests 32 \
  --per-capacity 100000  # GPU sum-tree R-future when capacity > 4096
```

## What's NOT in this commit

The R9 cluster smoke run itself is out of band — this commit ships
the entry points. R9 will execute the pre-cluster validation
checklist + first 1000-step smoke + multi-fold walk-forward G8 gate
per the rebuild plan §"Cluster smoke discipline".

The summary JSON's schema is intentionally narrow (final-step losses
+ replay len + completion state + NaN abort marker). R-future may
add per-epoch breakdowns + per-ISV-slot snapshots once the cluster
smoke tells us which diagnostics are actually load-bearing for kill
decisions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-23 13:30:10 +02:00
parent c7ccf0c301
commit 1168f3ea83
3 changed files with 966 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

356
crates/ml-alpha/examples/alpha_rl_train.rs Normal file
View File

@@ -0,0 +1,356 @@
//! Integrated RL trainer CLI (DQN + PPO on shared Mamba2 -> CfC encoder).
//!
//! Drives `IntegratedTrainer::step_with_lobsim` against a real `LobSimCuda`
//! backed by MBP-10 windows loaded via `MultiHorizonLoader::next_sequence_pair`
//! (R2) for true `(s_t, s_{t+1})` adjacency. The trainer combines:
//! * R3: GPU-resident EMA + advantage/return kernels.
//! * R4: GPU Thompson + Double-DQN argmax + log π_old kernels.
//! * R5: 7 RL controllers + target-net soft update.
//! * R6: GPU-pure env step (extract_realized_pnl_delta, apply_reward_scale,
//! actions_to_market_targets).
//! * R7a/R7b: host-lift of every per-step orchestration loop.
//! * R7c-data: true `h_{t+1}` / `V(s_{t+1})` / Bellman target on `h_{t+1}`.
//! * R7d: PER push/sample + off-policy DQN with stop-grad on encoder.
//!
//! Emits `alpha_rl_train_summary.json` on exit. Gate G8 (NaN abort): if any
//! per-head loss is non-finite at any step, the binary exits with code 2 so
//! the cluster smoke harness can terminate the workflow on first divergence.
//!
//! Usage:
//! alpha_rl_train \
//! --mbp10-data-dir /data/futures-baseline-mbp10/ES.FUT \
//! --predecoded-dir /feature-cache/predecoded \
//! --out /feature-cache/alpha-rl-runs/<sha> \
//! --n-steps 1000 \
//! --seed 16962 \
//! --instrument-mode front-month \
//! --gpu-idx 0
//!
//! Plan-doc reference: docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md
//! Section R8 (CLI binary + Argo template + dispatcher).
use anyhow::{Context, Result};
use clap::Parser;
use data::providers::databento::dbn_parser::InstrumentFilter;
use ml_alpha::data::loader::{
discover_mbp10_files_sorted, MultiHorizonLoader, MultiHorizonLoaderConfig,
DEFAULT_OUTCOME_LABEL_COST_ES,
};
use ml_alpha::heads::HORIZONS;
use ml_alpha::trainer::integrated::{
IntegratedStepStats, IntegratedTrainer, IntegratedTrainerConfig,
};
use ml_alpha::trainer::perception::PerceptionTrainerConfig;
use ml_backtesting::sim::LobSimCuda;
use ml_core::device::MlDevice;
use serde::Serialize;
use std::path::PathBuf;
use std::process;
#[derive(Parser)]
#[command(name = "alpha_rl_train")]
struct Cli {
/// Directory containing MBP-10 .dbn.zst files. Per
/// `feedback_mbp10_mandatory`, this argument is required — no
/// synthetic-data fallback in the production training path.
#[arg(long)]
mbp10_data_dir: PathBuf,
/// Directory holding predecoded sidecar caches (created on miss).
#[arg(long)]
predecoded_dir: PathBuf,
/// Output directory for summary JSON + per-step log artefacts.
#[arg(long)]
out: PathBuf,
/// Total `step_with_lobsim` calls to run. The validation smoke
/// (R9 prerequisite) ships with 1000; production scale-up runs
/// 50k+. Each call processes one `(s_t, s_{t+1})` sequence pair
/// across `n_backtests` parallel LobSim instances.
#[arg(long, default_value_t = 1000)]
n_steps: usize,
/// Encoder sequence length (snapshots per step). Must match
/// `PerceptionTrainerConfig::seq_len`. Default 32 matches the
/// validated single-scale `1:32` window from alpha_train.
#[arg(long, default_value_t = 32)]
seq_len: usize,
/// Batch dimension — number of parallel LobSim backtests run in
/// lockstep per step. Larger batches amortise the encoder forward
/// cost across more trajectories but push GPU memory pressure
/// linearly. Smoke at 1; production sweep at 32-64 (L40S 48GB).
#[arg(long, default_value_t = 1)]
n_backtests: usize,
/// PER buffer capacity (R7d). Per `replay.rs` doc, naive O(N)
/// sampling is acceptable up to ~4096. Production scale-up to
/// 100k+ needs a GPU sum-tree (R-future).
#[arg(long, default_value_t = 4096)]
per_capacity: usize,
/// Random seed pinning RNG + Xavier init + ISV controller bootstrap
/// state. Per `pearl_scoped_init_seed_for_reproducibility`.
#[arg(long, default_value_t = 16962)]
seed: u64,
/// MBP-10 instrument filter — `all`, `front-month`, or `id=<N>`.
/// `front-month` is the canonical setting for ES (auto-detects the
/// dominant contract per file, handles quarterly roll).
#[arg(long, default_value = "front-month", value_parser = parse_instrument_mode)]
instrument_mode: InstrumentFilter,
/// CUDA device index. Single-GPU dev (RTX 3050 Ti) uses 0; cluster
/// L40S/H100 also 0 (one GPU per pod per the alpha-rl template).
#[arg(long, default_value_t = 0)]
gpu_idx: usize,
/// Round-trip trade cost in price units for the D-style outcome
/// labels (consumed by the perception aux head, not by the RL
/// path directly — but the loader needs it). Default 0.5 = 2 ticks
/// × $0.25/tick for ES.
#[arg(long, default_value_t = DEFAULT_OUTCOME_LABEL_COST_ES)]
outcome_label_cost: f32,
/// How often (in steps) to flush a progress line to stderr. The
/// summary JSON is written ONCE at the end; this controls in-flight
/// visibility only.
#[arg(long, default_value_t = 100)]
log_every: usize,
}
fn parse_instrument_mode(s: &str) -> Result<InstrumentFilter, String> {
let trimmed = s.trim();
if trimmed.eq_ignore_ascii_case("all") {
return Ok(InstrumentFilter::All);
}
if trimmed.eq_ignore_ascii_case("front-month")
|| trimmed.eq_ignore_ascii_case("front_month")
{
return Ok(InstrumentFilter::FrontMonth);
}
if let Some(num) = trimmed.strip_prefix("id=") {
let id: u32 = num.parse().map_err(|e| {
format!("instrument-mode: expected `id=<u32>` but failed to parse '{num}': {e}")
})?;
return Ok(InstrumentFilter::Id(id));
}
Err(format!(
"instrument-mode: expected `all`, `front-month`, or `id=<N>`; got '{s}'"
))
}
#[derive(Serialize, Default)]
struct AlphaRlTrainSummary {
n_steps_planned: usize,
n_steps_completed: usize,
seq_len: usize,
n_backtests: usize,
per_capacity: usize,
seed: u64,
/// True iff `n_steps_completed == n_steps_planned` AND no NaN
/// abort fired. False if early-terminated for any reason (NaN,
/// loader EOF, panic — though panics also exit nonzero).
completed_clean: bool,
/// Final-step loss components (raw, not λ-weighted). All NaN if
/// trainer never returned a finite step.
final_l_bce: f32,
final_l_q: f32,
final_l_pi: f32,
final_l_v: f32,
final_l_aux: f32,
final_l_total: f32,
/// Replay buffer state at exit.
final_replay_len: usize,
/// Step at which the first NaN was observed, if any. -1 means
/// no NaN observed.
nan_abort_step: i64,
}
fn main() -> Result<()> {
tracing_subscriber::fmt()
.with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
.init();
let cli = Cli::parse();
std::fs::create_dir_all(&cli.out)
.with_context(|| format!("mkdir -p {}", cli.out.display()))?;
std::fs::create_dir_all(&cli.predecoded_dir)
.with_context(|| format!("mkdir -p {}", cli.predecoded_dir.display()))?;
let dev = MlDevice::cuda(cli.gpu_idx)
.with_context(|| format!("MlDevice::cuda({})", cli.gpu_idx))?;
eprintln!("CUDA device {} initialised", cli.gpu_idx);
// Loader: paired (s_t, s_{t+1}) sequences. inference_only=false so
// next_sequence_pair is available (the labels it preloads are
// unused by step_with_lobsim, but the API requires the label
// precomputation pass per `feedback_no_partial_refactor`).
let files = discover_mbp10_files_sorted(&cli.mbp10_data_dir)
.with_context(|| format!("discover_mbp10_files_sorted({})", cli.mbp10_data_dir.display()))?;
anyhow::ensure!(
!files.is_empty(),
"no MBP-10 .dbn.zst files under {}",
cli.mbp10_data_dir.display()
);
eprintln!("loader: {} MBP-10 files", files.len());
let multi_resolution: ml_alpha::data::aggregation::MultiResolutionConfig =
format!("1:{}", cli.seq_len)
.parse()
.with_context(|| format!("multi-resolution `1:{}`", cli.seq_len))?;
anyhow::ensure!(
multi_resolution.total_positions() == cli.seq_len,
"internal: single-scale 1:{} should produce total_positions={} but got {}",
cli.seq_len,
cli.seq_len,
multi_resolution.total_positions()
);
let loader_cfg = MultiHorizonLoaderConfig {
files,
predecoded_dir: cli.predecoded_dir.clone(),
multi_resolution,
horizons: HORIZONS,
// Soft cap above n_steps × 2 so next_sequence_pair never
// returns Ok(None) mid-run. Loader's per-file random anchor
// sampling reuses files indefinitely up to this cap.
n_max_sequences: cli.n_steps.saturating_mul(2).max(64),
seed: cli.seed,
inference_only: false,
outcome_label_cost: cli.outcome_label_cost,
instrument_filter: cli.instrument_mode,
};
let mut loader = MultiHorizonLoader::new(&loader_cfg).context("MultiHorizonLoader::new")?;
// Trainer + LobSim.
let perception_cfg = PerceptionTrainerConfig {
seq_len: cli.seq_len,
n_batch: cli.n_backtests,
..PerceptionTrainerConfig::default()
};
let trainer_cfg = IntegratedTrainerConfig {
perception: perception_cfg,
dqn_seed: cli.seed.wrapping_add(0xDA),
ppo_seed: cli.seed.wrapping_add(0xBE),
per_capacity: cli.per_capacity,
per_seed: cli.seed.wrapping_add(0x9E37_79B9),
};
let mut trainer =
IntegratedTrainer::new(&dev, trainer_cfg).context("IntegratedTrainer::new")?;
let mut sim = LobSimCuda::new(cli.n_backtests, &dev).context("LobSimCuda::new")?;
eprintln!(
"trainer + LobSimCuda initialised: seq_len={} n_backtests={} per_capacity={}",
cli.seq_len, cli.n_backtests, cli.per_capacity
);
let mut summary = AlphaRlTrainSummary {
n_steps_planned: cli.n_steps,
seq_len: cli.seq_len,
n_backtests: cli.n_backtests,
per_capacity: cli.per_capacity,
seed: cli.seed,
nan_abort_step: -1,
..Default::default()
};
let mut last_stats: Option<IntegratedStepStats> = None;
// ── Training loop. ───────────────────────────────────────────────
let t_start = std::time::Instant::now();
for step in 0..cli.n_steps {
let pair = loader
.next_sequence_pair()
.with_context(|| format!("next_sequence_pair at step {step}"))?;
let (s_t, s_tp1) = match pair {
Some(p) => p,
None => {
eprintln!(
"loader EOF at step {step} (n_max_sequences exhausted) — exiting early"
);
break;
}
};
let stats = trainer
.step_with_lobsim(&s_t.snapshots, &s_tp1.snapshots, &mut sim)
.with_context(|| format!("step_with_lobsim at step {step}"))?;
// Gate G8: NaN abort. Per `feedback_stop_on_anomaly` +
// `feedback_kill_runs_on_anomaly_quickly` the cluster smoke
// tail-watcher kills the workflow on non-zero exit; here we
// signal that with exit code 2 (distinct from clean exit 0
// and panic-derived exit codes).
let non_finite = !stats.l_total.is_finite()
|| !stats.l_bce.is_finite()
|| !stats.l_q.is_finite()
|| !stats.l_pi.is_finite()
|| !stats.l_v.is_finite()
|| !stats.l_aux.is_finite();
if non_finite {
eprintln!(
"G8 NaN ABORT at step {step}: l_bce={} l_q={} l_pi={} l_v={} l_aux={} l_total={}",
stats.l_bce, stats.l_q, stats.l_pi, stats.l_v, stats.l_aux, stats.l_total
);
summary.n_steps_completed = step;
summary.nan_abort_step = step as i64;
summary.completed_clean = false;
write_summary(&cli.out, &summary)?;
process::exit(2);
}
if step % cli.log_every == 0 || step == cli.n_steps - 1 {
eprintln!(
"step {:>6}/{}: l_q={:.4} l_pi={:.4} l_v={:.4} l_total={:.4} \
replay={} elapsed={:.1}s",
step,
cli.n_steps,
stats.l_q,
stats.l_pi,
stats.l_v,
stats.l_total,
trainer.replay.len(),
t_start.elapsed().as_secs_f32()
);
}
last_stats = Some(stats);
summary.n_steps_completed = step + 1;
}
// ── Compose + emit summary. ──────────────────────────────────────
if let Some(s) = last_stats {
summary.final_l_bce = s.l_bce;
summary.final_l_q = s.l_q;
summary.final_l_pi = s.l_pi;
summary.final_l_v = s.l_v;
summary.final_l_aux = s.l_aux;
summary.final_l_total = s.l_total;
}
summary.final_replay_len = trainer.replay.len();
summary.completed_clean = summary.n_steps_completed == summary.n_steps_planned;
write_summary(&cli.out, &summary)?;
eprintln!(
"alpha_rl_train complete: {} / {} steps in {:.1}s (replay len = {})",
summary.n_steps_completed,
summary.n_steps_planned,
t_start.elapsed().as_secs_f32(),
summary.final_replay_len
);
Ok(())
}
fn write_summary(out_dir: &std::path::Path, summary: &AlphaRlTrainSummary) -> Result<()> {
let path = out_dir.join("alpha_rl_train_summary.json");
let f = std::fs::File::create(&path)
.with_context(|| format!("create {}", path.display()))?;
serde_json::to_writer_pretty(f, summary)
.with_context(|| format!("write {}", path.display()))?;
eprintln!("summary written: {}", path.display());
Ok(())
}

416
infra/k8s/argo/alpha-rl-template.yaml Normal file
View File

@@ -0,0 +1,416 @@
# Integrated RL trainer workflow (Phase R8).
#
# Drives `IntegratedTrainer::step_with_lobsim` (DQN + PPO on shared
# Mamba2 -> CfC encoder) against MBP-10 from the training-data PVC,
# using LobSimCuda as the synthetic-book environment. Emits
# `alpha_rl_train_summary.json` to the feature-cache PVC with final-step
# loss components + replay buffer state for monitoring + G8 NaN-abort
# detection.
#
# DAG mirrors alpha-perception (check-cache → ensure-binary →
# warmup-gpu → train) so the alpine probe / sccache compile / autoscaler
# warmup behaviour stays a single source of truth. The only material
# difference is the binary built (`alpha_rl_train` vs `alpha_train`).
#
# Defaults match the R9 validation smoke (1000 steps, b_size=1,
# front-month ES). Production scale-up overrides via dispatcher flags.
#
# Usage:
# argo submit -n foxhunt --from=wftmpl/alpha-rl \
# -p commit-sha=HEAD -p git-branch=ml-alpha-phase-a
#
# Plan reference: docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: alpha-rl
namespace: foxhunt
labels:
app.kubernetes.io/name: alpha-rl
app.kubernetes.io/part-of: foxhunt
app.kubernetes.io/component: train
spec:
entrypoint: pipeline
serviceAccountName: argo-workflow
archiveLogs: true
podMetadata:
labels:
app.kubernetes.io/part-of: foxhunt
app.kubernetes.io/component: train
securityContext:
fsGroup: 0
ttlStrategy:
secondsAfterCompletion: 3600
activeDeadlineSeconds: 14400
arguments:
parameters:
- name: commit-sha
value: HEAD
- name: git-branch
value: ml-alpha-phase-a
- name: cuda-compute-cap
# Default sm_89 matches the L40S default pool per
# `feedback_default_to_l40s_pool`. Dispatcher derives this
# automatically from --gpu-pool.
value: "89"
- name: gpu-pool
# Per `feedback_default_to_l40s_pool`: ci-training-l40s is the
# default (NOT H100). Opt into ci-training-h100 explicitly for
# production scale-up sweeps that need the 80GB / sm_90 capacity.
value: ci-training-l40s
# RL-specific trainer knobs (mirror alpha_rl_train CLI defaults).
- name: n-steps
value: "1000"
- name: seq-len
value: "32"
- name: n-backtests
value: "1"
- name: per-capacity
value: "4096"
- name: seed
value: "16962"
- name: instrument-mode
value: "front-month"
volumes:
- name: git-ssh-key
secret:
secretName: argo-git-ssh-key
defaultMode: 256
- name: training-data
persistentVolumeClaim:
claimName: training-data-pvc
- name: cargo-target-cuda
persistentVolumeClaim:
claimName: cargo-target-cuda
- name: feature-cache
persistentVolumeClaim:
claimName: feature-cache-pvc
templates:
# ── DAG: check-cache → ensure-binary (when cache miss) → train ──
# warmup-gpu runs unconditionally in parallel to trigger L40S
# autoscaler scale-up so the node is warm by train-time, matching
# the alpha-perception pattern.
- name: pipeline
dag:
tasks:
- name: check-cache
template: check-cache
- name: ensure-binary
template: ensure-binary
dependencies: [check-cache]
when: "{{tasks.check-cache.outputs.parameters.cache}} == miss"
arguments:
parameters:
- name: sha
value: "{{tasks.check-cache.outputs.parameters.sha}}"
- name: warmup-gpu
template: warmup-gpu
- name: train
template: train
dependencies: [check-cache, ensure-binary]
arguments:
parameters:
- name: sha
value: "{{tasks.check-cache.outputs.parameters.sha}}"
# ── check-cache: probe training-data PVC for cached binary ──
# Same alpine probe pattern as alpha-perception. Caches at
# /data/bin/$SHORT_SHA/alpha_rl_train (distinct from alpha_train's
# cache slot so the two binaries can coexist at the same SHA).
- name: check-cache
outputs:
parameters:
- name: sha
valueFrom:
path: /tmp/sha
- name: cache
valueFrom:
path: /tmp/cache
nodeSelector:
k8s.scaleway.com/pool-name: platform
topology.kubernetes.io/zone: fr-par-2
tolerations:
- key: node.cilium.io/agent-not-ready
operator: Exists
effect: NoSchedule
container:
image: alpine:3.20
command: ["/bin/sh", "-c"]
resources:
requests:
cpu: "50m"
memory: 32Mi
limits:
cpu: "100m"
memory: 64Mi
volumeMounts:
- name: training-data
mountPath: /data
readOnly: true
args:
- |
set -e
SHA="{{workflow.parameters.commit-sha}}"
if [ "$SHA" = "HEAD" ]; then
echo "ERROR: commit-sha cannot be HEAD inside the workflow."
echo " Resolve via the submission script (scripts/argo-alpha-rl.sh)."
exit 1
fi
SHORT_SHA=$(echo "$SHA" | cut -c1-9)
echo "$SHORT_SHA" > /tmp/sha
BIN="/data/bin/$SHORT_SHA/alpha_rl_train"
if [ -x "$BIN" ]; then
SIZE=$(stat -c %s "$BIN")
echo "Cache HIT: $BIN ($SIZE bytes)"
echo "hit" > /tmp/cache
else
echo "Cache MISS: $BIN not present, ensure-binary will compile"
ls -lh "/data/bin/" 2>/dev/null | head -10 || echo " (no /data/bin/ directory yet)"
echo "miss" > /tmp/cache
fi
# ── ensure-binary: compile alpha_rl_train via sccache on miss ──
- name: ensure-binary
inputs:
parameters:
- name: sha
outputs:
parameters:
- name: sha
valueFrom:
path: /tmp/sha
nodeSelector:
k8s.scaleway.com/pool-name: ci-compile-cpu
topology.kubernetes.io/zone: fr-par-2
tolerations:
- key: node.cilium.io/agent-not-ready
operator: Exists
effect: NoSchedule
container:
image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
imagePullPolicy: Always
command: ["/bin/bash", "-c"]
env:
- name: SQLX_OFFLINE
value: "true"
- name: CARGO_TERM_COLOR
value: always
- name: CARGO_TARGET_DIR
value: /cargo-target
- name: CARGO_HOME
value: /cargo-target/cargo-home
- name: RUSTC_WRAPPER
value: sccache
- name: SCCACHE_DIR
value: /cargo-target/sccache
- name: SCCACHE_CACHE_SIZE
value: "40G"
- name: CARGO_INCREMENTAL
value: "0"
- name: CUDA_COMPUTE_CAP
value: "{{workflow.parameters.cuda-compute-cap}}"
resources:
requests:
cpu: "14"
memory: 32Gi
limits:
cpu: "30"
memory: 64Gi
volumeMounts:
- name: git-ssh-key
mountPath: /etc/git-ssh
readOnly: true
- name: cargo-target-cuda
mountPath: /cargo-target
- name: training-data
mountPath: /data
args:
- |
set -e
SHA="{{workflow.parameters.commit-sha}}"
BRANCH="{{workflow.parameters.git-branch}}"
mkdir -p ~/.ssh
cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
printf 'Host *\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n' > ~/.ssh/config
chmod 600 ~/.ssh/config
REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
BUILD="/cargo-target/src"
git config --global --add safe.directory "$BUILD"
if [ "$SHA" = "HEAD" ]; then
if [ -d "$BUILD/.git" ]; then
cd "$BUILD"; git fetch origin; SHA=$(git rev-parse "origin/$BRANCH"); cd /
else
git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD"
git checkout "origin/$BRANCH"; SHA=$(git rev-parse HEAD); cd /
fi
fi
SHORT_SHA=$(echo "$SHA" | cut -c1-9)
echo "Resolved SHA: $SHA (short: $SHORT_SHA)"
BIN_DIR="/data/bin/$SHORT_SHA"
if [ -x "$BIN_DIR/alpha_rl_train" ]; then
echo "=== Cache HIT: $BIN_DIR/alpha_rl_train ==="
ls -lh "$BIN_DIR/"
echo "$SHORT_SHA" > /tmp/sha
exit 0
fi
echo "=== Cache MISS: compiling alpha_rl_train for $SHORT_SHA ==="
if [ -d "$BUILD/.git" ]; then
cd "$BUILD"; git fetch origin
CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
if [ "$CURRENT" != "$SHA" ]; then
git checkout --force "$SHA"; git clean -fd
fi
else
git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD"; git checkout "$SHA"
fi
export PATH="${CARGO_HOME}/bin:${PATH}"
echo "Building alpha_rl_train (ml-alpha + ml-backtesting dev-dep)..."
cargo build --release -p ml-alpha --example alpha_rl_train
mkdir -p "$BIN_DIR"
cp "$CARGO_TARGET_DIR/release/examples/alpha_rl_train" "$BIN_DIR/"
strip "$BIN_DIR/alpha_rl_train"
ls -lh "$BIN_DIR/"
# Prune old cache (keep last 5 SHAs).
cd /data/bin
ls -1t | tail -n +6 | while read -r old; do
echo "Pruning old cache: $old"; rm -rf "$old"
done
echo "$SHORT_SHA" > /tmp/sha
# ── warmup-gpu: trigger autoscaler scale-up of the GPU pool ──
# Same pattern as alpha-perception — tiny alpine pod scheduled with
# the GPU pool's nodeSelector but no GPU resource request, so it
# doesn't compete with `train` for the single GPU.
- name: warmup-gpu
nodeSelector:
k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
topology.kubernetes.io/zone: fr-par-2
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
- key: node.cilium.io/agent-not-ready
operator: Exists
effect: NoSchedule
container:
image: alpine:3.20
command: ["/bin/sh", "-c"]
resources:
requests:
cpu: "50m"
memory: 32Mi
limits:
cpu: "100m"
memory: 64Mi
args:
- |
echo "GPU warmup pod scheduled on $(hostname) — autoscaler triggered, node in scaledown-grace window."
# ── train: run alpha_rl_train on L40S (default) or H100 (opt-in) ──
- name: train
inputs:
parameters:
- name: sha
nodeSelector:
k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
topology.kubernetes.io/zone: fr-par-2
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
- key: node.cilium.io/agent-not-ready
operator: Exists
effect: NoSchedule
container:
image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
imagePullPolicy: Always
command: ["/bin/bash", "-c"]
env:
- name: RUST_LOG
value: info
- name: SQLX_OFFLINE
value: "true"
- name: CUDA_COMPUTE_CAP
value: "{{workflow.parameters.cuda-compute-cap}}"
resources:
# L40S-1-48G has 8 vCPU and ~91Gi memory. Request <7800m so
# the pod fits alongside system daemonsets on the same node.
requests:
cpu: "6"
memory: 16Gi
nvidia.com/gpu: "1"
limits:
cpu: "7"
memory: 64Gi
nvidia.com/gpu: "1"
volumeMounts:
- name: training-data
mountPath: /data
- name: feature-cache
mountPath: /feature-cache
args:
- |
set -e
SHA="{{inputs.parameters.sha}}"
BIN="/data/bin/$SHA/alpha_rl_train"
MBP10_DIR="/data/futures-baseline-mbp10/ES.FUT"
PREDECODED_DIR="/feature-cache/predecoded"
OUT_DIR="/feature-cache/alpha-rl-runs/$SHA"
mkdir -p "$OUT_DIR" "$PREDECODED_DIR"
if [ ! -x "$BIN" ]; then
echo "ERROR: binary not found at $BIN"
ls -lh /data/bin/ || true
exit 1
fi
if [ ! -d "$MBP10_DIR" ]; then
echo "ERROR: MBP-10 data directory not found at $MBP10_DIR"
ls -lh /data/ || true
exit 1
fi
echo "Running alpha_rl_train (DQN + PPO on shared Mamba2→CfC encoder, R7d off-policy)"
echo " binary: $BIN"
echo " mbp10: $MBP10_DIR ($(find "$MBP10_DIR" -name '*.dbn.zst' | wc -l) files)"
echo " predecoded: $PREDECODED_DIR"
echo " out: $OUT_DIR"
echo " n-steps: {{workflow.parameters.n-steps}}"
echo " seq-len: {{workflow.parameters.seq-len}}"
echo " n-backtests: {{workflow.parameters.n-backtests}}"
echo " per-capacity: {{workflow.parameters.per-capacity}}"
echo " seed: {{workflow.parameters.seed}}"
echo " instrument: {{workflow.parameters.instrument-mode}}"
"$BIN" \
--mbp10-data-dir "$MBP10_DIR" \
--predecoded-dir "$PREDECODED_DIR" \
--out "$OUT_DIR" \
--n-steps {{workflow.parameters.n-steps}} \
--seq-len {{workflow.parameters.seq-len}} \
--n-backtests {{workflow.parameters.n-backtests}} \
--per-capacity {{workflow.parameters.per-capacity}} \
--seed {{workflow.parameters.seed}} \
--instrument-mode "{{workflow.parameters.instrument-mode}}"
RC=$?
echo "=== alpha_rl_train exited with code $RC ==="
if [ -f "$OUT_DIR/alpha_rl_train_summary.json" ]; then
echo "--- alpha_rl_train_summary.json ---"
cat "$OUT_DIR/alpha_rl_train_summary.json"
else
echo "WARNING: alpha_rl_train_summary.json not present in $OUT_DIR"
ls -lh "$OUT_DIR/" || true
fi
exit $RC

194
scripts/argo-alpha-rl.sh Executable file
View File

@@ -0,0 +1,194 @@
#!/usr/bin/env bash
# Submit the alpha-rl (integrated RL trainer) workflow.
#
# Drives `IntegratedTrainer::step_with_lobsim` (DQN + PPO on shared
# Mamba2 → CfC encoder, R7d off-policy replay) against MBP-10 from the
# training-data PVC, using LobSimCuda as the synthetic-book environment.
# Emits alpha_rl_train_summary.json to the feature-cache PVC.
#
# Guards (enforced by the rebuild plan A9 feedback list):
# * `feedback_default_to_l40s_pool` — defaults to ci-training-l40s;
# H100 (sm_90) is opt-in for scale-up sweeps only.
# * `feedback_push_before_deploy` — verifies local HEAD == origin
# HEAD before submission, refuses to submit otherwise.
# * `feedback_argo_template_must_apply` — kubectl-applies the local
# alpha-rl-template.yaml BEFORE `argo submit --from=wftmpl/alpha-rl`
# (the submit reads the cluster CRD, NOT the on-disk YAML, so
# unknown -p params silently no-op without the apply).
#
# Usage:
# ./scripts/argo-alpha-rl.sh # 1k step smoke on HEAD
# ./scripts/argo-alpha-rl.sh --branch main
# ./scripts/argo-alpha-rl.sh --n-steps 50000 --n-backtests 32 # production sweep
# ./scripts/argo-alpha-rl.sh --gpu-pool ci-training-h100 # opt into H100
# ./scripts/argo-alpha-rl.sh --watch
#
# Plan reference: docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md
set -euo pipefail
SHA=HEAD
BRANCH=$(git symbolic-ref --short HEAD 2>/dev/null || echo "ml-alpha-phase-a")
GPU_POOL=ci-training-l40s
N_STEPS=1000
SEQ_LEN=32
N_BACKTESTS=1
PER_CAPACITY=4096
SEED=16962
INSTRUMENT_MODE=front-month
WATCH=false
SKIP_PUSH_CHECK=false
SKIP_TEMPLATE_APPLY=false
usage() {
cat <<EOF
Usage: $0 [OPTIONS]
--sha <commit> Git SHA (default: HEAD on the current branch)
--branch <branch> Git branch (default: $BRANCH)
--gpu-pool <pool> GPU pool (default: $GPU_POOL).
Per feedback_default_to_l40s_pool: L40S is
the default. Opt into ci-training-h100 only
for scale-up runs that need sm_90 / 80GB.
--n-steps <n> step_with_lobsim count (default: $N_STEPS).
R9 smoke = 1000; production = 50000+.
--seq-len <n> Encoder sequence length (default: $SEQ_LEN).
--n-backtests <n> Parallel LobSim instances per step
(default: $N_BACKTESTS).
--per-capacity <n> PER buffer capacity (default: $PER_CAPACITY).
--seed <n> Random seed (default: $SEED).
--instrument-mode <m> all | front-month | id=<N> (default: $INSTRUMENT_MODE)
--watch Follow logs via argo watch
--skip-push-check Bypass the feedback_push_before_deploy guard
(rare; use only when submitting against a
previously-pushed SHA via --sha).
--skip-template-apply Bypass the feedback_argo_template_must_apply
guard (rare; only when you've manually
kubectl applied the template already).
-h, --help Show this help
EOF
}
while [[ $# -gt 0 ]]; do
case "$1" in
--sha) SHA="$2"; shift 2 ;;
--branch) BRANCH="$2"; shift 2 ;;
--gpu-pool) GPU_POOL="$2"; shift 2 ;;
--n-steps) N_STEPS="$2"; shift 2 ;;
--seq-len) SEQ_LEN="$2"; shift 2 ;;
--n-backtests) N_BACKTESTS="$2"; shift 2 ;;
--per-capacity) PER_CAPACITY="$2"; shift 2 ;;
--seed) SEED="$2"; shift 2 ;;
--instrument-mode) INSTRUMENT_MODE="$2"; shift 2 ;;
--watch) WATCH=true; shift ;;
--skip-push-check) SKIP_PUSH_CHECK=true; shift ;;
--skip-template-apply) SKIP_TEMPLATE_APPLY=true; shift ;;
-h|--help) usage; exit 0 ;;
*) echo "Unknown option: $1"; usage; exit 1 ;;
esac
done
# ── GPU pool → CUDA_COMPUTE_CAP ─────────────────────────────────────
# Per `feedback_default_to_l40s_pool` (2026-05-09): SP-chain training
# defaults to L40S (sm_89). H100 (sm_90) is opt-in for scale-up only.
# Cubins must match the device, so the dispatcher derives the cap from
# the pool name and threads it into the workflow params.
case "$GPU_POOL" in
ci-training-l40s|ci-training)
SM=89 ;;
ci-training-h100)
SM=90 ;;
*)
echo "ERROR: unknown gpu-pool: $GPU_POOL"
echo " Supported: ci-training-l40s, ci-training-h100"
exit 1 ;;
esac
# ── Guard 1: feedback_argo_template_must_apply ───────────────────────
# `argo submit --from=wftmpl/<name>` reads the cluster CRD, NOT the
# on-disk YAML. If the template has been edited locally but never
# `kubectl apply -f`'d, the submit silently uses the stale cluster
# version + drops any newly-added parameters. Pearl ref: 2026-05-21
# canonical incident.
TEMPLATE_FILE="infra/k8s/argo/alpha-rl-template.yaml"
if [[ ! -f "$TEMPLATE_FILE" ]]; then
echo "ERROR: template not found at $TEMPLATE_FILE (run from repo root)"
exit 1
fi
if [[ "$SKIP_TEMPLATE_APPLY" == "false" ]]; then
echo "Applying alpha-rl WorkflowTemplate to cluster..."
kubectl apply -n foxhunt -f "$TEMPLATE_FILE" >/dev/null
echo " applied: $TEMPLATE_FILE"
else
echo "WARNING: --skip-template-apply set — argo will use whatever's"
echo " already in the cluster CRD (may be stale)."
fi
# ── Guard 2: feedback_push_before_deploy ─────────────────────────────
# Per the 2026-05-20 incident: cluster pods pull source from origin
# `<BRANCH>`, NOT from the local working tree. Submitting before
# `git push` deploys whatever was last on origin, which can lag the
# local diff by N commits. Verify HEAD == origin/<branch> before
# submission unless explicitly bypassed.
if [[ "$SKIP_PUSH_CHECK" == "false" ]]; then
echo "Verifying local HEAD == origin/$BRANCH (feedback_push_before_deploy)..."
if ! git rev-parse --git-dir >/dev/null 2>&1; then
echo "ERROR: not in a git repo; cannot verify push state"
exit 1
fi
git fetch --quiet origin "$BRANCH" 2>/dev/null || true
LOCAL_HEAD=$(git rev-parse HEAD)
ORIGIN_HEAD=$(git rev-parse "origin/$BRANCH" 2>/dev/null || echo "MISSING")
if [[ "$ORIGIN_HEAD" == "MISSING" ]]; then
echo "ERROR: origin/$BRANCH does not exist — push the branch first:"
echo " git push -u origin $BRANCH"
exit 1
fi
if [[ "$LOCAL_HEAD" != "$ORIGIN_HEAD" ]]; then
echo "ERROR: local HEAD ($LOCAL_HEAD) != origin/$BRANCH ($ORIGIN_HEAD)"
echo " Push first:"
echo " git push origin $BRANCH"
echo " Or override with --skip-push-check if you're knowingly"
echo " deploying a previously-pushed SHA via --sha."
exit 1
fi
echo " HEAD verified: $LOCAL_HEAD == origin/$BRANCH"
fi
# Resolve SHA=HEAD to a concrete SHA before submission so the in-cluster
# check-cache pod doesn't need git access. Mirrors alpha-perception.
if [ "$SHA" = "HEAD" ]; then
SHA=$(git rev-parse "origin/$BRANCH" 2>/dev/null \
|| git rev-parse "$BRANCH" 2>/dev/null \
|| git rev-parse HEAD)
echo "Resolved HEAD → $SHA"
fi
echo ""
echo "Submitting alpha-rl workflow..."
echo " branch: $BRANCH"
echo " sha: $SHA"
echo " gpu-pool: $GPU_POOL (sm_$SM)"
echo " n-steps: $N_STEPS"
echo " seq-len: $SEQ_LEN"
echo " n-backtests: $N_BACKTESTS"
echo " per-capacity: $PER_CAPACITY"
echo " seed: $SEED"
echo " instrument: $INSTRUMENT_MODE"
WATCH_FLAG=""
if [[ "$WATCH" == "true" ]]; then
WATCH_FLAG="--watch"
fi
argo submit -n foxhunt --from=wftmpl/alpha-rl \
-p commit-sha="$SHA" \
-p git-branch="$BRANCH" \
-p cuda-compute-cap="$SM" \
-p gpu-pool="$GPU_POOL" \
-p n-steps="$N_STEPS" \
-p seq-len="$SEQ_LEN" \
-p n-backtests="$N_BACKTESTS" \
-p per-capacity="$PER_CAPACITY" \
-p seed="$SEED" \
-p instrument-mode="$INSTRUMENT_MODE" \
$WATCH_FLAG