From 1168f3ea83fe0d4438a58cbdd6ab62e03039f07d Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sat, 23 May 2026 13:30:10 +0200 Subject: [PATCH] =?UTF-8?q?feat(rl):=20R8=20=E2=80=94=20alpha=5Frl=5Ftrain?= =?UTF-8?q?=20CLI=20+=20Argo=20template=20+=20dispatcher?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the rebuild plan's R8 scope: production runner shape for the integrated RL trainer. Three artifacts wired end-to-end: 1. `crates/ml-alpha/examples/alpha_rl_train.rs` — clap CLI driving `IntegratedTrainer::step_with_lobsim` against MBP-10 windows loaded via `MultiHorizonLoader::next_sequence_pair` (R2) for true `(s_t, s_{t+1})` adjacency. Per `feedback_mbp10_mandatory`, `--mbp10-data-dir` is required — no synthetic-data fallback in the production path. 2. `infra/k8s/argo/alpha-rl-template.yaml` — WorkflowTemplate mirroring alpha-perception's DAG (check-cache → ensure-binary → train; warmup-gpu parallel). Binary cache slot is `/data/bin//alpha_rl_train` (distinct from `alpha_train` so the two binaries coexist at the same SHA). 3. `scripts/argo-alpha-rl.sh` — dispatcher with three rebuild-plan guards baked in. ## Dispatcher guards (per the rebuild plan's feedback list) `feedback_default_to_l40s_pool` (2026-05-09): default `--gpu-pool` is `ci-training-l40s` (sm_89). H100 (sm_90) is opt-in for production scale-up only. Cubins must match the device, so the dispatcher derives `cuda-compute-cap` from the pool name and threads it into the workflow params. `feedback_argo_template_must_apply` (2026-05-21 canonical incident): `argo submit --from=wftmpl/` reads the cluster CRD, NOT the on-disk YAML; unknown `-p` parameters silently no-op without a prior `kubectl apply`. Dispatcher applies the local template BEFORE every submission (overrideable via `--skip-template-apply` for the rare case where you've already applied manually). `feedback_push_before_deploy` (2026-05-20 canonical incident): the in-cluster `ensure-binary` pod fetches source from `origin/`, NOT the local working tree. Submitting before `git push` deploys the last-pushed SHA, which can lag local diff by N commits. Dispatcher verifies `git rev-parse HEAD == git rev-parse origin/` and hard-errors with the explicit push command otherwise. Bypass via `--skip-push-check` (only when intentionally deploying a previously- pushed SHA via `--sha`). ## CLI: gate G8 (NaN abort) Per `feedback_stop_on_anomaly` + `feedback_kill_runs_on_anomaly_quickly`, the CLI checks every per-head loss (l_bce / l_q / l_pi / l_v / l_aux / l_total) for finiteness after each `step_with_lobsim` call. Non-finite at any step → write summary with `nan_abort_step` set → `process::exit(2)`. R9's cluster smoke tail-watcher kills the workflow on the non-zero exit code, satisfying gate G8 from the rebuild plan. ## CLI: knobs that ARE on the CLI Structural / boundary parameters only (per `pearl_controller_anchors_isv_driven`: every adaptive knob lives in ISV, not CLI flags): * `--mbp10-data-dir / --predecoded-dir / --out` — I/O paths. * `--n-steps` — wall-budget control (1000 R9 smoke / 50k+ prod). * `--seq-len / --n-backtests / --per-capacity` — structural sizing. seq_len threads into the loader's multi-resolution `1:` config; n_backtests into both LobSimCuda and PerceptionTrainerConfig.n_batch. * `--seed` — reproducibility per `pearl_scoped_init_seed_for_reproducibility` (forks deterministic sub-seeds for dqn / ppo / per). * `--instrument-mode` — MBP-10 filter (all / front-month / id=N). * `--gpu-idx` — CUDA device selection. What's NOT on the CLI: γ / τ / ε / entropy_coef / per_α / reward_scale / per-head LRs — all live in ISV[400..417] and are driven by R5's controllers from EMA-tracked diagnostics. Per the rebuild plan A1: "every adaptive bound is signal-driven, not tuned." ## Cluster smoke entry point ```bash # R9 validation smoke (after pre-cluster local CUDA tests green). ./scripts/argo-alpha-rl.sh --n-steps 1000 --instrument-mode front-month # Production scale-up (gated by R9's multi-fold pass). ./scripts/argo-alpha-rl.sh --n-steps 50000 --n-backtests 32 \ --per-capacity 100000 # GPU sum-tree R-future when capacity > 4096 ``` ## What's NOT in this commit The R9 cluster smoke run itself is out of band — this commit ships the entry points. R9 will execute the pre-cluster validation checklist + first 1000-step smoke + multi-fold walk-forward G8 gate per the rebuild plan §"Cluster smoke discipline". The summary JSON's schema is intentionally narrow (final-step losses + replay len + completion state + NaN abort marker). R-future may add per-epoch breakdowns + per-ISV-slot snapshots once the cluster smoke tells us which diagnostics are actually load-bearing for kill decisions. Co-Authored-By: Claude Opus 4.7 --- crates/ml-alpha/examples/alpha_rl_train.rs | 356 ++++++++++++++++++ infra/k8s/argo/alpha-rl-template.yaml | 416 +++++++++++++++++++++ scripts/argo-alpha-rl.sh | 194 ++++++++++ 3 files changed, 966 insertions(+) create mode 100644 crates/ml-alpha/examples/alpha_rl_train.rs create mode 100644 infra/k8s/argo/alpha-rl-template.yaml create mode 100755 scripts/argo-alpha-rl.sh diff --git a/crates/ml-alpha/examples/alpha_rl_train.rs b/crates/ml-alpha/examples/alpha_rl_train.rs new file mode 100644 index 000000000..c75058eb6 --- /dev/null +++ b/crates/ml-alpha/examples/alpha_rl_train.rs @@ -0,0 +1,356 @@ +//! Integrated RL trainer CLI (DQN + PPO on shared Mamba2 -> CfC encoder). +//! +//! Drives `IntegratedTrainer::step_with_lobsim` against a real `LobSimCuda` +//! backed by MBP-10 windows loaded via `MultiHorizonLoader::next_sequence_pair` +//! (R2) for true `(s_t, s_{t+1})` adjacency. The trainer combines: +//! * R3: GPU-resident EMA + advantage/return kernels. +//! * R4: GPU Thompson + Double-DQN argmax + log π_old kernels. +//! * R5: 7 RL controllers + target-net soft update. +//! * R6: GPU-pure env step (extract_realized_pnl_delta, apply_reward_scale, +//! actions_to_market_targets). +//! * R7a/R7b: host-lift of every per-step orchestration loop. +//! * R7c-data: true `h_{t+1}` / `V(s_{t+1})` / Bellman target on `h_{t+1}`. +//! * R7d: PER push/sample + off-policy DQN with stop-grad on encoder. +//! +//! Emits `alpha_rl_train_summary.json` on exit. Gate G8 (NaN abort): if any +//! per-head loss is non-finite at any step, the binary exits with code 2 so +//! the cluster smoke harness can terminate the workflow on first divergence. +//! +//! Usage: +//! alpha_rl_train \ +//! --mbp10-data-dir /data/futures-baseline-mbp10/ES.FUT \ +//! --predecoded-dir /feature-cache/predecoded \ +//! --out /feature-cache/alpha-rl-runs/ \ +//! --n-steps 1000 \ +//! --seed 16962 \ +//! --instrument-mode front-month \ +//! --gpu-idx 0 +//! +//! Plan-doc reference: docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md +//! Section R8 (CLI binary + Argo template + dispatcher). + +use anyhow::{Context, Result}; +use clap::Parser; +use data::providers::databento::dbn_parser::InstrumentFilter; +use ml_alpha::data::loader::{ + discover_mbp10_files_sorted, MultiHorizonLoader, MultiHorizonLoaderConfig, + DEFAULT_OUTCOME_LABEL_COST_ES, +}; +use ml_alpha::heads::HORIZONS; +use ml_alpha::trainer::integrated::{ + IntegratedStepStats, IntegratedTrainer, IntegratedTrainerConfig, +}; +use ml_alpha::trainer::perception::PerceptionTrainerConfig; +use ml_backtesting::sim::LobSimCuda; +use ml_core::device::MlDevice; +use serde::Serialize; +use std::path::PathBuf; +use std::process; + +#[derive(Parser)] +#[command(name = "alpha_rl_train")] +struct Cli { + /// Directory containing MBP-10 .dbn.zst files. Per + /// `feedback_mbp10_mandatory`, this argument is required — no + /// synthetic-data fallback in the production training path. + #[arg(long)] + mbp10_data_dir: PathBuf, + + /// Directory holding predecoded sidecar caches (created on miss). + #[arg(long)] + predecoded_dir: PathBuf, + + /// Output directory for summary JSON + per-step log artefacts. + #[arg(long)] + out: PathBuf, + + /// Total `step_with_lobsim` calls to run. The validation smoke + /// (R9 prerequisite) ships with 1000; production scale-up runs + /// 50k+. Each call processes one `(s_t, s_{t+1})` sequence pair + /// across `n_backtests` parallel LobSim instances. + #[arg(long, default_value_t = 1000)] + n_steps: usize, + + /// Encoder sequence length (snapshots per step). Must match + /// `PerceptionTrainerConfig::seq_len`. Default 32 matches the + /// validated single-scale `1:32` window from alpha_train. + #[arg(long, default_value_t = 32)] + seq_len: usize, + + /// Batch dimension — number of parallel LobSim backtests run in + /// lockstep per step. Larger batches amortise the encoder forward + /// cost across more trajectories but push GPU memory pressure + /// linearly. Smoke at 1; production sweep at 32-64 (L40S 48GB). + #[arg(long, default_value_t = 1)] + n_backtests: usize, + + /// PER buffer capacity (R7d). Per `replay.rs` doc, naive O(N) + /// sampling is acceptable up to ~4096. Production scale-up to + /// 100k+ needs a GPU sum-tree (R-future). + #[arg(long, default_value_t = 4096)] + per_capacity: usize, + + /// Random seed pinning RNG + Xavier init + ISV controller bootstrap + /// state. Per `pearl_scoped_init_seed_for_reproducibility`. + #[arg(long, default_value_t = 16962)] + seed: u64, + + /// MBP-10 instrument filter — `all`, `front-month`, or `id=`. + /// `front-month` is the canonical setting for ES (auto-detects the + /// dominant contract per file, handles quarterly roll). + #[arg(long, default_value = "front-month", value_parser = parse_instrument_mode)] + instrument_mode: InstrumentFilter, + + /// CUDA device index. Single-GPU dev (RTX 3050 Ti) uses 0; cluster + /// L40S/H100 also 0 (one GPU per pod per the alpha-rl template). + #[arg(long, default_value_t = 0)] + gpu_idx: usize, + + /// Round-trip trade cost in price units for the D-style outcome + /// labels (consumed by the perception aux head, not by the RL + /// path directly — but the loader needs it). Default 0.5 = 2 ticks + /// × $0.25/tick for ES. + #[arg(long, default_value_t = DEFAULT_OUTCOME_LABEL_COST_ES)] + outcome_label_cost: f32, + + /// How often (in steps) to flush a progress line to stderr. The + /// summary JSON is written ONCE at the end; this controls in-flight + /// visibility only. + #[arg(long, default_value_t = 100)] + log_every: usize, +} + +fn parse_instrument_mode(s: &str) -> Result { + let trimmed = s.trim(); + if trimmed.eq_ignore_ascii_case("all") { + return Ok(InstrumentFilter::All); + } + if trimmed.eq_ignore_ascii_case("front-month") + || trimmed.eq_ignore_ascii_case("front_month") + { + return Ok(InstrumentFilter::FrontMonth); + } + if let Some(num) = trimmed.strip_prefix("id=") { + let id: u32 = num.parse().map_err(|e| { + format!("instrument-mode: expected `id=` but failed to parse '{num}': {e}") + })?; + return Ok(InstrumentFilter::Id(id)); + } + Err(format!( + "instrument-mode: expected `all`, `front-month`, or `id=`; got '{s}'" + )) +} + +#[derive(Serialize, Default)] +struct AlphaRlTrainSummary { + n_steps_planned: usize, + n_steps_completed: usize, + seq_len: usize, + n_backtests: usize, + per_capacity: usize, + seed: u64, + + /// True iff `n_steps_completed == n_steps_planned` AND no NaN + /// abort fired. False if early-terminated for any reason (NaN, + /// loader EOF, panic — though panics also exit nonzero). + completed_clean: bool, + + /// Final-step loss components (raw, not λ-weighted). All NaN if + /// trainer never returned a finite step. + final_l_bce: f32, + final_l_q: f32, + final_l_pi: f32, + final_l_v: f32, + final_l_aux: f32, + final_l_total: f32, + + /// Replay buffer state at exit. + final_replay_len: usize, + + /// Step at which the first NaN was observed, if any. -1 means + /// no NaN observed. + nan_abort_step: i64, +} + +fn main() -> Result<()> { + tracing_subscriber::fmt() + .with_env_filter(tracing_subscriber::EnvFilter::from_default_env()) + .init(); + + let cli = Cli::parse(); + + std::fs::create_dir_all(&cli.out) + .with_context(|| format!("mkdir -p {}", cli.out.display()))?; + std::fs::create_dir_all(&cli.predecoded_dir) + .with_context(|| format!("mkdir -p {}", cli.predecoded_dir.display()))?; + + let dev = MlDevice::cuda(cli.gpu_idx) + .with_context(|| format!("MlDevice::cuda({})", cli.gpu_idx))?; + eprintln!("CUDA device {} initialised", cli.gpu_idx); + + // Loader: paired (s_t, s_{t+1}) sequences. inference_only=false so + // next_sequence_pair is available (the labels it preloads are + // unused by step_with_lobsim, but the API requires the label + // precomputation pass per `feedback_no_partial_refactor`). + let files = discover_mbp10_files_sorted(&cli.mbp10_data_dir) + .with_context(|| format!("discover_mbp10_files_sorted({})", cli.mbp10_data_dir.display()))?; + anyhow::ensure!( + !files.is_empty(), + "no MBP-10 .dbn.zst files under {}", + cli.mbp10_data_dir.display() + ); + eprintln!("loader: {} MBP-10 files", files.len()); + + let multi_resolution: ml_alpha::data::aggregation::MultiResolutionConfig = + format!("1:{}", cli.seq_len) + .parse() + .with_context(|| format!("multi-resolution `1:{}`", cli.seq_len))?; + anyhow::ensure!( + multi_resolution.total_positions() == cli.seq_len, + "internal: single-scale 1:{} should produce total_positions={} but got {}", + cli.seq_len, + cli.seq_len, + multi_resolution.total_positions() + ); + + let loader_cfg = MultiHorizonLoaderConfig { + files, + predecoded_dir: cli.predecoded_dir.clone(), + multi_resolution, + horizons: HORIZONS, + // Soft cap above n_steps × 2 so next_sequence_pair never + // returns Ok(None) mid-run. Loader's per-file random anchor + // sampling reuses files indefinitely up to this cap. + n_max_sequences: cli.n_steps.saturating_mul(2).max(64), + seed: cli.seed, + inference_only: false, + outcome_label_cost: cli.outcome_label_cost, + instrument_filter: cli.instrument_mode, + }; + let mut loader = MultiHorizonLoader::new(&loader_cfg).context("MultiHorizonLoader::new")?; + + // Trainer + LobSim. + let perception_cfg = PerceptionTrainerConfig { + seq_len: cli.seq_len, + n_batch: cli.n_backtests, + ..PerceptionTrainerConfig::default() + }; + let trainer_cfg = IntegratedTrainerConfig { + perception: perception_cfg, + dqn_seed: cli.seed.wrapping_add(0xDA), + ppo_seed: cli.seed.wrapping_add(0xBE), + per_capacity: cli.per_capacity, + per_seed: cli.seed.wrapping_add(0x9E37_79B9), + }; + let mut trainer = + IntegratedTrainer::new(&dev, trainer_cfg).context("IntegratedTrainer::new")?; + let mut sim = LobSimCuda::new(cli.n_backtests, &dev).context("LobSimCuda::new")?; + eprintln!( + "trainer + LobSimCuda initialised: seq_len={} n_backtests={} per_capacity={}", + cli.seq_len, cli.n_backtests, cli.per_capacity + ); + + let mut summary = AlphaRlTrainSummary { + n_steps_planned: cli.n_steps, + seq_len: cli.seq_len, + n_backtests: cli.n_backtests, + per_capacity: cli.per_capacity, + seed: cli.seed, + nan_abort_step: -1, + ..Default::default() + }; + let mut last_stats: Option = None; + + // ── Training loop. ─────────────────────────────────────────────── + let t_start = std::time::Instant::now(); + for step in 0..cli.n_steps { + let pair = loader + .next_sequence_pair() + .with_context(|| format!("next_sequence_pair at step {step}"))?; + let (s_t, s_tp1) = match pair { + Some(p) => p, + None => { + eprintln!( + "loader EOF at step {step} (n_max_sequences exhausted) — exiting early" + ); + break; + } + }; + + let stats = trainer + .step_with_lobsim(&s_t.snapshots, &s_tp1.snapshots, &mut sim) + .with_context(|| format!("step_with_lobsim at step {step}"))?; + + // Gate G8: NaN abort. Per `feedback_stop_on_anomaly` + + // `feedback_kill_runs_on_anomaly_quickly` the cluster smoke + // tail-watcher kills the workflow on non-zero exit; here we + // signal that with exit code 2 (distinct from clean exit 0 + // and panic-derived exit codes). + let non_finite = !stats.l_total.is_finite() + || !stats.l_bce.is_finite() + || !stats.l_q.is_finite() + || !stats.l_pi.is_finite() + || !stats.l_v.is_finite() + || !stats.l_aux.is_finite(); + if non_finite { + eprintln!( + "G8 NaN ABORT at step {step}: l_bce={} l_q={} l_pi={} l_v={} l_aux={} l_total={}", + stats.l_bce, stats.l_q, stats.l_pi, stats.l_v, stats.l_aux, stats.l_total + ); + summary.n_steps_completed = step; + summary.nan_abort_step = step as i64; + summary.completed_clean = false; + write_summary(&cli.out, &summary)?; + process::exit(2); + } + + if step % cli.log_every == 0 || step == cli.n_steps - 1 { + eprintln!( + "step {:>6}/{}: l_q={:.4} l_pi={:.4} l_v={:.4} l_total={:.4} \ + replay={} elapsed={:.1}s", + step, + cli.n_steps, + stats.l_q, + stats.l_pi, + stats.l_v, + stats.l_total, + trainer.replay.len(), + t_start.elapsed().as_secs_f32() + ); + } + last_stats = Some(stats); + summary.n_steps_completed = step + 1; + } + + // ── Compose + emit summary. ────────────────────────────────────── + if let Some(s) = last_stats { + summary.final_l_bce = s.l_bce; + summary.final_l_q = s.l_q; + summary.final_l_pi = s.l_pi; + summary.final_l_v = s.l_v; + summary.final_l_aux = s.l_aux; + summary.final_l_total = s.l_total; + } + summary.final_replay_len = trainer.replay.len(); + summary.completed_clean = summary.n_steps_completed == summary.n_steps_planned; + write_summary(&cli.out, &summary)?; + + eprintln!( + "alpha_rl_train complete: {} / {} steps in {:.1}s (replay len = {})", + summary.n_steps_completed, + summary.n_steps_planned, + t_start.elapsed().as_secs_f32(), + summary.final_replay_len + ); + Ok(()) +} + +fn write_summary(out_dir: &std::path::Path, summary: &AlphaRlTrainSummary) -> Result<()> { + let path = out_dir.join("alpha_rl_train_summary.json"); + let f = std::fs::File::create(&path) + .with_context(|| format!("create {}", path.display()))?; + serde_json::to_writer_pretty(f, summary) + .with_context(|| format!("write {}", path.display()))?; + eprintln!("summary written: {}", path.display()); + Ok(()) +} diff --git a/infra/k8s/argo/alpha-rl-template.yaml b/infra/k8s/argo/alpha-rl-template.yaml new file mode 100644 index 000000000..80ad64adc --- /dev/null +++ b/infra/k8s/argo/alpha-rl-template.yaml @@ -0,0 +1,416 @@ +# Integrated RL trainer workflow (Phase R8). +# +# Drives `IntegratedTrainer::step_with_lobsim` (DQN + PPO on shared +# Mamba2 -> CfC encoder) against MBP-10 from the training-data PVC, +# using LobSimCuda as the synthetic-book environment. Emits +# `alpha_rl_train_summary.json` to the feature-cache PVC with final-step +# loss components + replay buffer state for monitoring + G8 NaN-abort +# detection. +# +# DAG mirrors alpha-perception (check-cache → ensure-binary → +# warmup-gpu → train) so the alpine probe / sccache compile / autoscaler +# warmup behaviour stays a single source of truth. The only material +# difference is the binary built (`alpha_rl_train` vs `alpha_train`). +# +# Defaults match the R9 validation smoke (1000 steps, b_size=1, +# front-month ES). Production scale-up overrides via dispatcher flags. +# +# Usage: +# argo submit -n foxhunt --from=wftmpl/alpha-rl \ +# -p commit-sha=HEAD -p git-branch=ml-alpha-phase-a +# +# Plan reference: docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md +apiVersion: argoproj.io/v1alpha1 +kind: WorkflowTemplate +metadata: + name: alpha-rl + namespace: foxhunt + labels: + app.kubernetes.io/name: alpha-rl + app.kubernetes.io/part-of: foxhunt + app.kubernetes.io/component: train +spec: + entrypoint: pipeline + serviceAccountName: argo-workflow + archiveLogs: true + podMetadata: + labels: + app.kubernetes.io/part-of: foxhunt + app.kubernetes.io/component: train + securityContext: + fsGroup: 0 + ttlStrategy: + secondsAfterCompletion: 3600 + activeDeadlineSeconds: 14400 + + arguments: + parameters: + - name: commit-sha + value: HEAD + - name: git-branch + value: ml-alpha-phase-a + - name: cuda-compute-cap + # Default sm_89 matches the L40S default pool per + # `feedback_default_to_l40s_pool`. Dispatcher derives this + # automatically from --gpu-pool. + value: "89" + - name: gpu-pool + # Per `feedback_default_to_l40s_pool`: ci-training-l40s is the + # default (NOT H100). Opt into ci-training-h100 explicitly for + # production scale-up sweeps that need the 80GB / sm_90 capacity. + value: ci-training-l40s + # RL-specific trainer knobs (mirror alpha_rl_train CLI defaults). + - name: n-steps + value: "1000" + - name: seq-len + value: "32" + - name: n-backtests + value: "1" + - name: per-capacity + value: "4096" + - name: seed + value: "16962" + - name: instrument-mode + value: "front-month" + + volumes: + - name: git-ssh-key + secret: + secretName: argo-git-ssh-key + defaultMode: 256 + - name: training-data + persistentVolumeClaim: + claimName: training-data-pvc + - name: cargo-target-cuda + persistentVolumeClaim: + claimName: cargo-target-cuda + - name: feature-cache + persistentVolumeClaim: + claimName: feature-cache-pvc + + templates: + # ── DAG: check-cache → ensure-binary (when cache miss) → train ── + # warmup-gpu runs unconditionally in parallel to trigger L40S + # autoscaler scale-up so the node is warm by train-time, matching + # the alpha-perception pattern. + - name: pipeline + dag: + tasks: + - name: check-cache + template: check-cache + - name: ensure-binary + template: ensure-binary + dependencies: [check-cache] + when: "{{tasks.check-cache.outputs.parameters.cache}} == miss" + arguments: + parameters: + - name: sha + value: "{{tasks.check-cache.outputs.parameters.sha}}" + - name: warmup-gpu + template: warmup-gpu + - name: train + template: train + dependencies: [check-cache, ensure-binary] + arguments: + parameters: + - name: sha + value: "{{tasks.check-cache.outputs.parameters.sha}}" + + # ── check-cache: probe training-data PVC for cached binary ── + # Same alpine probe pattern as alpha-perception. Caches at + # /data/bin/$SHORT_SHA/alpha_rl_train (distinct from alpha_train's + # cache slot so the two binaries can coexist at the same SHA). + - name: check-cache + outputs: + parameters: + - name: sha + valueFrom: + path: /tmp/sha + - name: cache + valueFrom: + path: /tmp/cache + nodeSelector: + k8s.scaleway.com/pool-name: platform + topology.kubernetes.io/zone: fr-par-2 + tolerations: + - key: node.cilium.io/agent-not-ready + operator: Exists + effect: NoSchedule + container: + image: alpine:3.20 + command: ["/bin/sh", "-c"] + resources: + requests: + cpu: "50m" + memory: 32Mi + limits: + cpu: "100m" + memory: 64Mi + volumeMounts: + - name: training-data + mountPath: /data + readOnly: true + args: + - | + set -e + SHA="{{workflow.parameters.commit-sha}}" + if [ "$SHA" = "HEAD" ]; then + echo "ERROR: commit-sha cannot be HEAD inside the workflow." + echo " Resolve via the submission script (scripts/argo-alpha-rl.sh)." + exit 1 + fi + SHORT_SHA=$(echo "$SHA" | cut -c1-9) + echo "$SHORT_SHA" > /tmp/sha + + BIN="/data/bin/$SHORT_SHA/alpha_rl_train" + if [ -x "$BIN" ]; then + SIZE=$(stat -c %s "$BIN") + echo "Cache HIT: $BIN ($SIZE bytes)" + echo "hit" > /tmp/cache + else + echo "Cache MISS: $BIN not present, ensure-binary will compile" + ls -lh "/data/bin/" 2>/dev/null | head -10 || echo " (no /data/bin/ directory yet)" + echo "miss" > /tmp/cache + fi + + # ── ensure-binary: compile alpha_rl_train via sccache on miss ── + - name: ensure-binary + inputs: + parameters: + - name: sha + outputs: + parameters: + - name: sha + valueFrom: + path: /tmp/sha + nodeSelector: + k8s.scaleway.com/pool-name: ci-compile-cpu + topology.kubernetes.io/zone: fr-par-2 + tolerations: + - key: node.cilium.io/agent-not-ready + operator: Exists + effect: NoSchedule + container: + image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest + imagePullPolicy: Always + command: ["/bin/bash", "-c"] + env: + - name: SQLX_OFFLINE + value: "true" + - name: CARGO_TERM_COLOR + value: always + - name: CARGO_TARGET_DIR + value: /cargo-target + - name: CARGO_HOME + value: /cargo-target/cargo-home + - name: RUSTC_WRAPPER + value: sccache + - name: SCCACHE_DIR + value: /cargo-target/sccache + - name: SCCACHE_CACHE_SIZE + value: "40G" + - name: CARGO_INCREMENTAL + value: "0" + - name: CUDA_COMPUTE_CAP + value: "{{workflow.parameters.cuda-compute-cap}}" + resources: + requests: + cpu: "14" + memory: 32Gi + limits: + cpu: "30" + memory: 64Gi + volumeMounts: + - name: git-ssh-key + mountPath: /etc/git-ssh + readOnly: true + - name: cargo-target-cuda + mountPath: /cargo-target + - name: training-data + mountPath: /data + args: + - | + set -e + SHA="{{workflow.parameters.commit-sha}}" + BRANCH="{{workflow.parameters.git-branch}}" + mkdir -p ~/.ssh + cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519 + chmod 600 ~/.ssh/id_ed25519 + printf 'Host *\n StrictHostKeyChecking no\n UserKnownHostsFile /dev/null\n' > ~/.ssh/config + chmod 600 ~/.ssh/config + REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git" + BUILD="/cargo-target/src" + git config --global --add safe.directory "$BUILD" + + if [ "$SHA" = "HEAD" ]; then + if [ -d "$BUILD/.git" ]; then + cd "$BUILD"; git fetch origin; SHA=$(git rev-parse "origin/$BRANCH"); cd / + else + git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD" + git checkout "origin/$BRANCH"; SHA=$(git rev-parse HEAD); cd / + fi + fi + SHORT_SHA=$(echo "$SHA" | cut -c1-9) + echo "Resolved SHA: $SHA (short: $SHORT_SHA)" + + BIN_DIR="/data/bin/$SHORT_SHA" + if [ -x "$BIN_DIR/alpha_rl_train" ]; then + echo "=== Cache HIT: $BIN_DIR/alpha_rl_train ===" + ls -lh "$BIN_DIR/" + echo "$SHORT_SHA" > /tmp/sha + exit 0 + fi + echo "=== Cache MISS: compiling alpha_rl_train for $SHORT_SHA ===" + + if [ -d "$BUILD/.git" ]; then + cd "$BUILD"; git fetch origin + CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none") + if [ "$CURRENT" != "$SHA" ]; then + git checkout --force "$SHA"; git clean -fd + fi + else + git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD"; git checkout "$SHA" + fi + + export PATH="${CARGO_HOME}/bin:${PATH}" + + echo "Building alpha_rl_train (ml-alpha + ml-backtesting dev-dep)..." + cargo build --release -p ml-alpha --example alpha_rl_train + + mkdir -p "$BIN_DIR" + cp "$CARGO_TARGET_DIR/release/examples/alpha_rl_train" "$BIN_DIR/" + strip "$BIN_DIR/alpha_rl_train" + ls -lh "$BIN_DIR/" + + # Prune old cache (keep last 5 SHAs). + cd /data/bin + ls -1t | tail -n +6 | while read -r old; do + echo "Pruning old cache: $old"; rm -rf "$old" + done + echo "$SHORT_SHA" > /tmp/sha + + # ── warmup-gpu: trigger autoscaler scale-up of the GPU pool ── + # Same pattern as alpha-perception — tiny alpine pod scheduled with + # the GPU pool's nodeSelector but no GPU resource request, so it + # doesn't compete with `train` for the single GPU. + - name: warmup-gpu + nodeSelector: + k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}" + topology.kubernetes.io/zone: fr-par-2 + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + - key: node.cilium.io/agent-not-ready + operator: Exists + effect: NoSchedule + container: + image: alpine:3.20 + command: ["/bin/sh", "-c"] + resources: + requests: + cpu: "50m" + memory: 32Mi + limits: + cpu: "100m" + memory: 64Mi + args: + - | + echo "GPU warmup pod scheduled on $(hostname) — autoscaler triggered, node in scaledown-grace window." + + # ── train: run alpha_rl_train on L40S (default) or H100 (opt-in) ── + - name: train + inputs: + parameters: + - name: sha + nodeSelector: + k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}" + topology.kubernetes.io/zone: fr-par-2 + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + - key: node.cilium.io/agent-not-ready + operator: Exists + effect: NoSchedule + container: + image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest + imagePullPolicy: Always + command: ["/bin/bash", "-c"] + env: + - name: RUST_LOG + value: info + - name: SQLX_OFFLINE + value: "true" + - name: CUDA_COMPUTE_CAP + value: "{{workflow.parameters.cuda-compute-cap}}" + resources: + # L40S-1-48G has 8 vCPU and ~91Gi memory. Request <7800m so + # the pod fits alongside system daemonsets on the same node. + requests: + cpu: "6" + memory: 16Gi + nvidia.com/gpu: "1" + limits: + cpu: "7" + memory: 64Gi + nvidia.com/gpu: "1" + volumeMounts: + - name: training-data + mountPath: /data + - name: feature-cache + mountPath: /feature-cache + args: + - | + set -e + SHA="{{inputs.parameters.sha}}" + BIN="/data/bin/$SHA/alpha_rl_train" + MBP10_DIR="/data/futures-baseline-mbp10/ES.FUT" + PREDECODED_DIR="/feature-cache/predecoded" + OUT_DIR="/feature-cache/alpha-rl-runs/$SHA" + mkdir -p "$OUT_DIR" "$PREDECODED_DIR" + + if [ ! -x "$BIN" ]; then + echo "ERROR: binary not found at $BIN" + ls -lh /data/bin/ || true + exit 1 + fi + if [ ! -d "$MBP10_DIR" ]; then + echo "ERROR: MBP-10 data directory not found at $MBP10_DIR" + ls -lh /data/ || true + exit 1 + fi + + echo "Running alpha_rl_train (DQN + PPO on shared Mamba2→CfC encoder, R7d off-policy)" + echo " binary: $BIN" + echo " mbp10: $MBP10_DIR ($(find "$MBP10_DIR" -name '*.dbn.zst' | wc -l) files)" + echo " predecoded: $PREDECODED_DIR" + echo " out: $OUT_DIR" + echo " n-steps: {{workflow.parameters.n-steps}}" + echo " seq-len: {{workflow.parameters.seq-len}}" + echo " n-backtests: {{workflow.parameters.n-backtests}}" + echo " per-capacity: {{workflow.parameters.per-capacity}}" + echo " seed: {{workflow.parameters.seed}}" + echo " instrument: {{workflow.parameters.instrument-mode}}" + + "$BIN" \ + --mbp10-data-dir "$MBP10_DIR" \ + --predecoded-dir "$PREDECODED_DIR" \ + --out "$OUT_DIR" \ + --n-steps {{workflow.parameters.n-steps}} \ + --seq-len {{workflow.parameters.seq-len}} \ + --n-backtests {{workflow.parameters.n-backtests}} \ + --per-capacity {{workflow.parameters.per-capacity}} \ + --seed {{workflow.parameters.seed}} \ + --instrument-mode "{{workflow.parameters.instrument-mode}}" + RC=$? + + echo "=== alpha_rl_train exited with code $RC ===" + if [ -f "$OUT_DIR/alpha_rl_train_summary.json" ]; then + echo "--- alpha_rl_train_summary.json ---" + cat "$OUT_DIR/alpha_rl_train_summary.json" + else + echo "WARNING: alpha_rl_train_summary.json not present in $OUT_DIR" + ls -lh "$OUT_DIR/" || true + fi + exit $RC diff --git a/scripts/argo-alpha-rl.sh b/scripts/argo-alpha-rl.sh new file mode 100755 index 000000000..e7a80a0d3 --- /dev/null +++ b/scripts/argo-alpha-rl.sh @@ -0,0 +1,194 @@ +#!/usr/bin/env bash +# Submit the alpha-rl (integrated RL trainer) workflow. +# +# Drives `IntegratedTrainer::step_with_lobsim` (DQN + PPO on shared +# Mamba2 → CfC encoder, R7d off-policy replay) against MBP-10 from the +# training-data PVC, using LobSimCuda as the synthetic-book environment. +# Emits alpha_rl_train_summary.json to the feature-cache PVC. +# +# Guards (enforced by the rebuild plan A9 feedback list): +# * `feedback_default_to_l40s_pool` — defaults to ci-training-l40s; +# H100 (sm_90) is opt-in for scale-up sweeps only. +# * `feedback_push_before_deploy` — verifies local HEAD == origin +# HEAD before submission, refuses to submit otherwise. +# * `feedback_argo_template_must_apply` — kubectl-applies the local +# alpha-rl-template.yaml BEFORE `argo submit --from=wftmpl/alpha-rl` +# (the submit reads the cluster CRD, NOT the on-disk YAML, so +# unknown -p params silently no-op without the apply). +# +# Usage: +# ./scripts/argo-alpha-rl.sh # 1k step smoke on HEAD +# ./scripts/argo-alpha-rl.sh --branch main +# ./scripts/argo-alpha-rl.sh --n-steps 50000 --n-backtests 32 # production sweep +# ./scripts/argo-alpha-rl.sh --gpu-pool ci-training-h100 # opt into H100 +# ./scripts/argo-alpha-rl.sh --watch +# +# Plan reference: docs/superpowers/plans/2026-05-23-integrated-rl-trainer-rebuild.md + +set -euo pipefail + +SHA=HEAD +BRANCH=$(git symbolic-ref --short HEAD 2>/dev/null || echo "ml-alpha-phase-a") +GPU_POOL=ci-training-l40s +N_STEPS=1000 +SEQ_LEN=32 +N_BACKTESTS=1 +PER_CAPACITY=4096 +SEED=16962 +INSTRUMENT_MODE=front-month +WATCH=false +SKIP_PUSH_CHECK=false +SKIP_TEMPLATE_APPLY=false + +usage() { + cat < Git SHA (default: HEAD on the current branch) + --branch Git branch (default: $BRANCH) + --gpu-pool GPU pool (default: $GPU_POOL). + Per feedback_default_to_l40s_pool: L40S is + the default. Opt into ci-training-h100 only + for scale-up runs that need sm_90 / 80GB. + --n-steps step_with_lobsim count (default: $N_STEPS). + R9 smoke = 1000; production = 50000+. + --seq-len Encoder sequence length (default: $SEQ_LEN). + --n-backtests Parallel LobSim instances per step + (default: $N_BACKTESTS). + --per-capacity PER buffer capacity (default: $PER_CAPACITY). + --seed Random seed (default: $SEED). + --instrument-mode all | front-month | id= (default: $INSTRUMENT_MODE) + --watch Follow logs via argo watch + --skip-push-check Bypass the feedback_push_before_deploy guard + (rare; use only when submitting against a + previously-pushed SHA via --sha). + --skip-template-apply Bypass the feedback_argo_template_must_apply + guard (rare; only when you've manually + kubectl applied the template already). + -h, --help Show this help +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --sha) SHA="$2"; shift 2 ;; + --branch) BRANCH="$2"; shift 2 ;; + --gpu-pool) GPU_POOL="$2"; shift 2 ;; + --n-steps) N_STEPS="$2"; shift 2 ;; + --seq-len) SEQ_LEN="$2"; shift 2 ;; + --n-backtests) N_BACKTESTS="$2"; shift 2 ;; + --per-capacity) PER_CAPACITY="$2"; shift 2 ;; + --seed) SEED="$2"; shift 2 ;; + --instrument-mode) INSTRUMENT_MODE="$2"; shift 2 ;; + --watch) WATCH=true; shift ;; + --skip-push-check) SKIP_PUSH_CHECK=true; shift ;; + --skip-template-apply) SKIP_TEMPLATE_APPLY=true; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "Unknown option: $1"; usage; exit 1 ;; + esac +done + +# ── GPU pool → CUDA_COMPUTE_CAP ───────────────────────────────────── +# Per `feedback_default_to_l40s_pool` (2026-05-09): SP-chain training +# defaults to L40S (sm_89). H100 (sm_90) is opt-in for scale-up only. +# Cubins must match the device, so the dispatcher derives the cap from +# the pool name and threads it into the workflow params. +case "$GPU_POOL" in + ci-training-l40s|ci-training) + SM=89 ;; + ci-training-h100) + SM=90 ;; + *) + echo "ERROR: unknown gpu-pool: $GPU_POOL" + echo " Supported: ci-training-l40s, ci-training-h100" + exit 1 ;; +esac + +# ── Guard 1: feedback_argo_template_must_apply ─────────────────────── +# `argo submit --from=wftmpl/` reads the cluster CRD, NOT the +# on-disk YAML. If the template has been edited locally but never +# `kubectl apply -f`'d, the submit silently uses the stale cluster +# version + drops any newly-added parameters. Pearl ref: 2026-05-21 +# canonical incident. +TEMPLATE_FILE="infra/k8s/argo/alpha-rl-template.yaml" +if [[ ! -f "$TEMPLATE_FILE" ]]; then + echo "ERROR: template not found at $TEMPLATE_FILE (run from repo root)" + exit 1 +fi +if [[ "$SKIP_TEMPLATE_APPLY" == "false" ]]; then + echo "Applying alpha-rl WorkflowTemplate to cluster..." + kubectl apply -n foxhunt -f "$TEMPLATE_FILE" >/dev/null + echo " applied: $TEMPLATE_FILE" +else + echo "WARNING: --skip-template-apply set — argo will use whatever's" + echo " already in the cluster CRD (may be stale)." +fi + +# ── Guard 2: feedback_push_before_deploy ───────────────────────────── +# Per the 2026-05-20 incident: cluster pods pull source from origin +# ``, NOT from the local working tree. Submitting before +# `git push` deploys whatever was last on origin, which can lag the +# local diff by N commits. Verify HEAD == origin/ before +# submission unless explicitly bypassed. +if [[ "$SKIP_PUSH_CHECK" == "false" ]]; then + echo "Verifying local HEAD == origin/$BRANCH (feedback_push_before_deploy)..." + if ! git rev-parse --git-dir >/dev/null 2>&1; then + echo "ERROR: not in a git repo; cannot verify push state" + exit 1 + fi + git fetch --quiet origin "$BRANCH" 2>/dev/null || true + LOCAL_HEAD=$(git rev-parse HEAD) + ORIGIN_HEAD=$(git rev-parse "origin/$BRANCH" 2>/dev/null || echo "MISSING") + if [[ "$ORIGIN_HEAD" == "MISSING" ]]; then + echo "ERROR: origin/$BRANCH does not exist — push the branch first:" + echo " git push -u origin $BRANCH" + exit 1 + fi + if [[ "$LOCAL_HEAD" != "$ORIGIN_HEAD" ]]; then + echo "ERROR: local HEAD ($LOCAL_HEAD) != origin/$BRANCH ($ORIGIN_HEAD)" + echo " Push first:" + echo " git push origin $BRANCH" + echo " Or override with --skip-push-check if you're knowingly" + echo " deploying a previously-pushed SHA via --sha." + exit 1 + fi + echo " HEAD verified: $LOCAL_HEAD == origin/$BRANCH" +fi + +# Resolve SHA=HEAD to a concrete SHA before submission so the in-cluster +# check-cache pod doesn't need git access. Mirrors alpha-perception. +if [ "$SHA" = "HEAD" ]; then + SHA=$(git rev-parse "origin/$BRANCH" 2>/dev/null \ + || git rev-parse "$BRANCH" 2>/dev/null \ + || git rev-parse HEAD) + echo "Resolved HEAD → $SHA" +fi + +echo "" +echo "Submitting alpha-rl workflow..." +echo " branch: $BRANCH" +echo " sha: $SHA" +echo " gpu-pool: $GPU_POOL (sm_$SM)" +echo " n-steps: $N_STEPS" +echo " seq-len: $SEQ_LEN" +echo " n-backtests: $N_BACKTESTS" +echo " per-capacity: $PER_CAPACITY" +echo " seed: $SEED" +echo " instrument: $INSTRUMENT_MODE" + +WATCH_FLAG="" +if [[ "$WATCH" == "true" ]]; then + WATCH_FLAG="--watch" +fi + +argo submit -n foxhunt --from=wftmpl/alpha-rl \ + -p commit-sha="$SHA" \ + -p git-branch="$BRANCH" \ + -p cuda-compute-cap="$SM" \ + -p gpu-pool="$GPU_POOL" \ + -p n-steps="$N_STEPS" \ + -p seq-len="$SEQ_LEN" \ + -p n-backtests="$N_BACKTESTS" \ + -p per-capacity="$PER_CAPACITY" \ + -p seed="$SEED" \ + -p instrument-mode="$INSTRUMENT_MODE" \ + $WATCH_FLAG