Merge: relocate platform IaC to fxhnt (Phase 2D)

Platform terraform (live+modules) + k8s manifests moved to fxhnt (now the IaC home).
Rust-only remnants kept: docker build images, gpu-taint.yaml, Rust scripts. foxhunt
being archived; history retained. Live platform IaC now in the fxhnt repo.

.gitignore

@@ -204,3 +204,4 @@ crates/ml/ml/
 # Foxhunt audit hook dedup state (cleared at SessionStart)
 .claude/.foxhunt-audit-state
 /config/ml/alpha_logits_cache.bin
+data/surfer/

docs/superpowers/plans/2026-06-05-surfer-phase0-floor-implementation.md

@@ -0,0 +1,192 @@
+# Surfer Phase 0 — Diversified-Trend Floor + Validation (CUDA/Rust) — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:executing-plans (or subagent-driven-development).
+> Steps use checkbox (`- [ ]`). TDD throughout — **GPU-oracle tests only, NO CPU reference oracle**
+> (`feedback_no_cpu_test_fallbacks`). **CUDA-only, CPU read-only** (`feedback_cpu_is_read_only`,
+> `pearl_cold_path_no_exception_to_gpu_drives`): every formula is a CUDA kernel; CPU reads only final gate scalars
+> from mapped-pinned buffers and only enumerates split index-sets as control flow.
+
+**Goal:** Build the deterministic diversified-trend FLOOR and the CPCV/PBO/Deflated-Sharpe validation, **entirely as
+CUDA kernels + Rust orchestration in `crates/ml-alpha`**, then measure whether the floor has a real OOS edge after
+costs. The cheap, falsifiable gate before any ML (Phases 1-3). Runs on the local GPU (data is tiny: ~30 instruments
+× ~5000 days = ~150k floats).
+
+**Architecture:** Reuse ml-alpha's CUDA pipeline — `build.rs` cubin precompile (`KERNELS` list), `pinned_mem.rs`
+mapped-pinned buffers, cudarc `load_cubin`/`launch_builder`, the determinism foundation (`FOXHUNT_DETERMINISTIC`,
+single-thread-per-series sequential sweeps, no `atomicAdd`, no nvrtc). GPU-drives / CPU-reads, uniformly.
+
+**Tech Stack:** Rust 1.85 (`crates/ml-alpha`), CUDA 12.4 (pre-compiled cubins via build.rs, NO nvrtc), cudarc,
+mapped-pinned host buffers. Data via `crates/data` dbn decode (the L0-fixed `dbn_parser`). NO Python in the built
+system. NO GPU↔CPU roundtrips except the final gate-scalar readback.
+
+**Spec:** `docs/superpowers/specs/2026-06-05-surfer-uncorrelated-universe-design.md`
+
+---
+
+## GPU-contract compliance (apply to EVERY task)
+- **No `atomicAdd`** (`feedback_no_atomicadd`) → reductions are tree-reduce or single-thread-per-series sequential.
+- **Mapped-pinned only** for any host visibility (`feedback_no_htod_htoh_only_mapped_pinned`); no raw HtoD/DtoH for
+  compute results. Data UPLOAD uses the existing mapped-pinned dataset path.
+- **Determinism**: per-series sequential sweeps (EWMA vol, portfolio carry) are single-thread-per-instrument with a
+  sequential time loop — mirror `gae_backward_sweep.cu` (deterministic, no inter-thread race). Verify with
+  `determinism-check`-style same-seed bit-equality.
+- **No CPU compute**: signals, vol, sizing, returns, Sharpe, PBO, Deflated-Sharpe are kernels. CPU may only:
+  (a) enumerate combinatorial split index-sets (control flow, like `epoch_idx`), uploading them as int arrays;
+  (b) read the FINAL gate scalars from mapped-pinned at the end.
+- **No CPU test oracle**: tests assert kernel outputs against HAND-DERIVED analytic constants read from mapped-pinned
+  (e.g. "rising series → TSMOM signal == +1.0"), never a CPU re-implementation of the algorithm.
+
+## File structure
+| File | Change | Responsibility |
+|---|---|---|
+| `crates/ml-alpha/cuda/surfer_continuous_adjust.cu` | new | element-wise backward ratio-adjust of stitched closes |
+| `crates/ml-alpha/cuda/surfer_tsmom_signal.cu` | new | per (instrument,day) 1/3/12-mo sign-ensemble signal |
+| `crates/ml-alpha/cuda/surfer_ewma_vol.cu` | new | per-instrument EWMA vol (single-thread-per-instrument seq) |
+| `crates/ml-alpha/cuda/surfer_floor_portfolio.cu` | new | inverse-vol size + vol-target + no-trade band → daily port. returns |
+| `crates/ml-alpha/cuda/surfer_cpcv_sharpe.cu` | new | per (path,config) Sharpe over returns matrix (parallel) |
+| `crates/ml-alpha/cuda/surfer_overfit_stats.cu` | new | PBO + Deflated-Sharpe scalars on GPU (erff cdf) |
+| `crates/ml-alpha/build.rs` | edit `KERNELS` | register the 6 cubins |
+| `crates/ml-alpha/src/surfer/mod.rs` | new | Rust orchestration (load→upload→launch→read gates) |
+| `crates/ml-alpha/src/surfer/continuous.rs` | new | roll detection + stitch (data-assembly; decode via `data` crate) |
+| `crates/ml-alpha/examples/surfer_phase0.rs` | new | the verdict-run binary |
+| `crates/ml-alpha/tests/surfer_floor_invariants.rs` | new | GPU-oracle invariant tests |
+
+---
+
+## Task A: Continuous-contract series (data-assembly + ratio-adjust kernel)
+
+**Files:** `crates/ml-alpha/src/surfer/continuous.rs`; `crates/ml-alpha/cuda/surfer_continuous_adjust.cu`; build.rs.
+
+- [ ] **Step 1:** Add `"surfer_continuous_adjust"` to `KERNELS` in `crates/ml-alpha/build.rs`.
+- [ ] **Step 2: kernel** `surfer_continuous_adjust.cu` — given stitched raw closes `[N_days]` and a roll-ratio array
+  `ratio[N_days]` (cumulative back-adjust factor per day, =1.0 after the last roll), output `adj[i]=raw[i]*ratio[i]`.
+  Element-wise, one thread per day. (Roll DETECTION — which contract is active per day by max volume, and the per-roll
+  ratio — is data-assembly control flow in `continuous.rs`: it decides WHAT was loaded, not a compute-over-results.
+  It reads per-expiry volume from the decoded dbn, picks argmax-volume contract per day, and builds the cumulative
+  `ratio` array, which is then UPLOADED mapped-pinned; the multiply is the kernel.)
+- [ ] **Step 3: `continuous.rs`** — `pub fn build_continuous(root, decoder) -> ContinuousSeries { ts, raw_close_d,
+  adj_close_d, volume… }` decoding per-expiry OHLCV via the `data` crate, picking active contract per day by volume,
+  computing cumulative ratio, uploading raw+ratio mapped-pinned, launching the adjust kernel → `adj_close_d` on GPU.
+  Keep BOTH `raw_close_d` (fills/reward USD) and `adj_close_d` (signals/returns). Per spec §4: NEVER Panama; reward on raw.
+- [ ] **Step 4: GPU-oracle test** (`surfer_floor_invariants.rs`): upload raw=[100,101,102,202,204], roll at day3,
+  ratio=[r,r,r,1,1] with r=202/102; launch; read `adj` mapped-pinned; assert `adj[3]==202`, `adj[2]==102*r`, and the
+  seam log-return `ln(adj[3]/adj[2])≈0`. (Analytic constants, no CPU re-impl.)
+- [ ] **Step 5:** `SQLX_OFFLINE=true cargo test -p ml-alpha --test surfer_floor_invariants continuous` → PASS.
+- [ ] **Step 6: Commit** — `feat(surfer): continuous-contract assembly + ratio-adjust kernel`.
+
+---
+
+## Task B: Floor kernels (TSMOM signal, EWMA vol, portfolio backtest)
+
+**Files:** `surfer_tsmom_signal.cu`, `surfer_ewma_vol.cu`, `surfer_floor_portfolio.cu`; `src/surfer/mod.rs`; build.rs.
+
+- [ ] **Step 1:** Register the 3 kernels in build.rs `KERNELS`.
+- [ ] **Step 2: `surfer_tsmom_signal.cu`** — input `logc[N_inst × N_days]`; per (inst, day) output
+  `sig = clip( mean_{L∈{21,63,252}} sign(logc[end]-logc[end-L]), -1, 1)`, where for L=252 `end = day-21` (skip recent),
+  else `end=day`; 0 before warmup. Grid over (inst×day), branch-free. (Sign-inversion for yield-based rates handled
+  by a per-instrument `sign_flip[inst]` multiplier uploaded as config.)
+- [ ] **Step 3: `surfer_ewma_vol.cu`** — per instrument, single thread, sequential over days:
+  `v[t] = a·r[t]^2 + (1-a)·v[t-1]`, `a=1-exp(ln0.5/halflife)`, `sigma[t]=sqrt(v[t]·252)`. One thread per instrument
+  (deterministic sequential carry — mirror `gae_backward_sweep.cu`'s single-thread-per-series pattern). No atomics.
+- [ ] **Step 4: `surfer_floor_portfolio.cu`** — per day (sequential single-thread for the portfolio vol-target carry;
+  per-instrument inner loop): target weight `w_i = sig_i · (risk_budget / sigma_i)`; apply no-trade band (skip change
+  if |w_i - w_i_prev| < band); portfolio gross scaled to `vol_target/realized_port_vol` (realized from a trailing
+  window kernel-side); daily portfolio return `R[t] = Σ_i w_i[t-1]·r_i[t]`. Output `port_ret_d[N_days]`,
+  `turnover_d[N_days]` on GPU.
+- [ ] **Step 5: `src/surfer/mod.rs`** — `pub struct Floor` loads the 3 cubins; `pub fn run_floor(&mut self,
+  series: &[ContinuousSeries], cfg: FloorCfg) -> FloorOutput` launches signal→vol→portfolio, returns GPU handles
+  (`port_ret_d`, `turnover_d`). No host math.
+- [ ] **Step 6: GPU-oracle tests** (analytic): (a) monotonic-rising logc → `tsmom_signal` last value == +1.0;
+  monotonic-falling → −1.0; (b) constant-vol synthetic → `ewma_vol` converges to the known σ (hand value); (c) a
+  2-instrument hand-built case → assert `port_ret_d[t]` equals the hand-derived `Σ w·r` for one step. Read all via
+  mapped-pinned; no CPU re-impl of the formulas.
+- [ ] **Step 7:** Run tests → PASS. **Smoke**: `run_floor` on the local 4 instruments (6E/ES/NQ/ZN), read
+  `port_ret_d` mapped-pinned, print annualized Sharpe (expect noisy/underpowered on 2y — proves the machinery).
+- [ ] **Step 8: Commit** — `feat(surfer): floor kernels (TSMOM signal + EWMA vol + portfolio backtest)`.
+
+---
+
+## Task C: Validation kernels (CPCV Sharpe, PBO, Deflated Sharpe) + cost
+
+**Files:** `surfer_cpcv_sharpe.cu`, `surfer_overfit_stats.cu`; `src/surfer/validation.rs`; build.rs.
+
+- [ ] **Step 1:** Register both kernels in build.rs `KERNELS`.
+- [ ] **Step 2: cost in the portfolio kernel** — extend `surfer_floor_portfolio.cu` to subtract per-rebalance cost:
+  `cost[t] = turnover[t]·(per_contract_usd/notional + half_spread_bps/1e4)` (+ roll cost on roll days), producing a
+  NET `port_ret_d`. Costs are config scalars uploaded mapped-pinned. (No host arithmetic.)
+- [ ] **Step 3: `surfer_cpcv_sharpe.cu`** — inputs: net `port_ret_d[N_days]` (or `[N_cfg × N_days]` if sweeping
+  configs), and CPU-enumerated `test_mask[N_path × N_days]` (uint8, uploaded). Per (path[, cfg]) compute mean/std →
+  Sharpe over the path's OOS days via tree-reduce (no atomics). Output `oos_sharpe[N_path(× N_cfg)]` on GPU.
+- [ ] **Step 4: `surfer_overfit_stats.cu`** — on GPU: (a) Deflated Sharpe `DSR = Φ((SR−SR0)√(T−1)/√(1−γ3·SR+
+  (γ4−1)/4·SR²))` with `Φ` via `0.5·erfcf(-x/√2)`, `SR0` from `n_trials` (Euler-γ order-statistic formula), `n_trials`
+  uploaded; (b) PBO from the per-split IS-argmax / OOS-rank arrays (the CSCV split sets enumerated CPU-side, the
+  per-split argmax+rank reduced GPU-side). Output `{dsr, pbo, oos_sharpe_5pct, rank_consistency}` to a mapped-pinned
+  gate-scalar struct.
+- [ ] **Step 5: `src/surfer/validation.rs`** — `pub fn validate(net_ret_d, cfg_grid) -> Gates`: CPU ENUMERATES the
+  CPCV combinations and CSCV splits (control flow), uploads index masks mapped-pinned, launches the two kernels,
+  reads the final `Gates` struct from mapped-pinned. The ONLY host readback.
+- [ ] **Step 6: GPU-oracle tests**: (a) feed a returns series with known mean/std → assert `oos_sharpe` equals the
+  hand-derived value; (b) Deflated-Sharpe: same SR, n_trials=1 vs 50 → assert `dsr` decreases (monotonic, hand-checked
+  direction); (c) PBO: construct IS-best=OOS-worst returns → assert `pbo > 0.5`. All via mapped-pinned; no CPU oracle.
+- [ ] **Step 7:** Run tests → PASS. **Determinism**: run `validate` twice same inputs → gate scalars bit-equal.
+- [ ] **Step 8: Commit** — `feat(surfer): CUDA validation (CPCV Sharpe + PBO + Deflated Sharpe + cost)`.
+
+---
+
+## Task D: Acquire broad-universe deep-history data
+
+**Files:** reuse `services/data_acquisition` / the existing Databento pipeline; cache decoded series.
+
+- [ ] **Step 1:** Target ≥15 FULL-SIZE roots across classes (equity ES/NQ/YM/RTY; rates ZN/ZB/ZF/ZT; FX 6E/6J/6B/6A/6C;
+  metals GC/SI/HG; energy CL/NG/RB; ags ZC/ZS/ZW), max history via Databento `GLBX.MDP3 ohlcv-1d`. (Edge is identical to
+  micros; micros are a deployment-sizing concern only.)
+- [ ] **Step 2:** Fetch per-expiry daily OHLCV (existing data pipeline), land on the PVC/local; the surfer's
+  `continuous.rs` builds continuous series at load.
+- [ ] **Step 3:** Validate each root: ≥10y, decode clean via the L0-fixed `dbn_parser`, ratio seam-returns sane.
+- [ ] **STOP-if:** Databento history/cost prohibitive → document and proceed with the largest free continuous set,
+  noting the data-quality caveat in the verdict.
+
+---
+
+## Task E: Phase-0 verdict run (the decisive gate)
+
+**Files:** `crates/ml-alpha/examples/surfer_phase0.rs`.
+
+- [ ] **Step 1:** Wire: load all continuous series → upload → `run_floor` (net of cost) → `validate(net_ret_d, grid)`
+  over the lookback/vol-target config grid → read the `Gates` struct (mapped-pinned, the only readback) → print a
+  verdict block (per `tier1_5_verdict.py` STYLE, but the binary is Rust reading GPU scalars):
+  - SV-G1: CPCV (≥50 paths) 5th-pct OOS Sharpe after costs > 0.
+  - SV-G3: Deflated Sharpe > 0.95, `n_trials` = ALL trials ever (record the count: 64 commits + session harnesses +
+    every floor lookback/vol-target variant in the grid).
+  - SV-G4: IS↔OOS config rank-consistency > 0.
+  - SV-G5: edge survives 2× cost + roll (re-run with doubled cost scalars).
+- [ ] **Step 2:** Determinism: same-inputs run twice → identical verdict.
+- [ ] **Step 3:** Record verdict in spec §9 + a memory pearl.
+- [ ] **Step 4: Commit.**
+
+**VERDICT BRANCH:**
+- **PASS** (floor clears SV-G1/G3/G5 OOS net of costs) → diversified-trend premium is real & capturable on this
+  universe → proceed to Phase 1 (regime overlay). The floor becomes the live benchmark.
+- **FAIL** → even the floor has no powered OOS edge → do NOT build ML. Reassess universe/history/thesis. Cost: a few
+  days of local-GPU dev, $0 cluster.
+
+---
+
+## Self-review
+- **CUDA-only / CPU-read-only**: every formula is a kernel (Tasks A-C); CPU only enumerates split masks + reads the
+  final `Gates` struct (Task C Step 5, Task E Step 1). Complies with `feedback_cpu_is_read_only`,
+  `pearl_cold_path_no_exception_to_gpu_drives`. ✓
+- **No CPU test oracle**: all tests assert kernel outputs vs hand-derived analytic constants read from mapped-pinned
+  (`feedback_no_cpu_test_fallbacks`). ✓
+- **GPU-contract**: no atomicAdd (tree-reduce / single-thread-per-series), mapped-pinned only, cubins via build.rs
+  (no nvrtc), determinism via sequential-per-series sweeps + same-input bit-equality. ✓
+- **Spec coverage**: floor §3.1 → Task B ✓ | validation gates §5 → Task C+E ✓ | universe+pipeline §4 → Task A+D ✓ |
+  Phase-0 STOP §6 → Task E verdict ✓ | edge-validation on full-size (not micros) → Task D ✓ | reuse ml-alpha infra §3.5 → file structure ✓.
+- **Placeholders**: kernel bodies are given as signature + exact formula + determinism pattern + the existing kernel to
+  mirror (gae_backward_sweep / ema_update_per_step / the reduce kernels) — a competent engineer following ml-alpha's
+  `cuda/` conventions writes them directly. Not silent gaps.
+
+## Execution note
+Tasks A-C run on the local GPU against the 4 local instruments to prove the kernels + machinery; D-E need the broad
+universe. No cluster, no Python in the built system. The floor + CUDA validation are the reusable core the ML surfer
+(Phases 1-3) sits on.

docs/superpowers/plans/2026-06-21-decommission-gitlab.md

@@ -0,0 +1,191 @@
+# Decommission GitLab (compute) — Implementation Plan (Phase 2C)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
+> **Do NOT run subagent-driven** — irreversible helm uninstall + a proxy edit that can take down all
+> `*.fxhnt.ai` if mis-ordered; each step needs a human-confirmed gate. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Remove GitLab's compute (2 helm releases, dead Argo events, the `:5050` proxy block, the values
+manifest) while retaining all GitLab data as a cold backup.
+
+**Architecture:** Proxy `:5050` block removed FIRST (defuses the nginx-startup landmine), then helm
+uninstall, then dead-event + manifest cleanup. gitaly PVC + GitLab buckets + secrets are left untouched.
+
+**Tech Stack:** helm, kubectl, nginx+socat Tailscale proxy, Argo Events.
+
+**Spec:** `docs/superpowers/specs/2026-06-21-decommission-gitlab-design.md`
+
+---
+
+## Task 0: Branch
+- [ ] **Step 1**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git checkout -b chore/decommission-gitlab
+```
+
+---
+
+## Task 1: Pre-flight — confirm no live GitLab consumers + baseline
+
+- [ ] **Step 1: Confirm nothing pulls the GitLab image / uses the GitLab backend anymore**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+echo "pods on gitlab-registry image (expect none):"
+kubectl get pods -n foxhunt -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' 2>/dev/null | grep -c "gitlab-registry.foxhunt.svc.cluster.local:5000" | xargs echo
+echo "git remotes (expect git.fxhnt.ai = Gitea, not :2222/root):"
+git remote get-url origin; (cd /home/jgrusewski/Work/fxhnt && git remote get-url origin)
+```
+Expected: `0` pods on the GitLab image; both remotes `ssh://git@git.fxhnt.ai/gitadmin/...`. **STOP if any
+pod still references the GitLab registry.**
+
+- [ ] **Step 2: Baseline RAM + pod count**
+```bash
+kubectl top pods -n foxhunt 2>/dev/null | grep -i gitlab | awk '{s+=$3} END{print NR" gitlab pods, "s" Mi"}'
+```
+Expected: ~12 pods, ~4.6 GiB (this is what gets reclaimed).
+
+---
+
+## Task 2: Remove the `:5050` registry block from the proxy (FIRST — safety-critical)
+
+**Files:** Modify `infra/k8s/gitlab/tailscale-proxy.yaml`
+
+- [ ] **Step 1: Delete `containerPort: 5050`**
+
+In `infra/k8s/gitlab/tailscale-proxy.yaml`, remove the line:
+```
+            - containerPort: 5050
+```
+(leave `- containerPort: 443`).
+
+- [ ] **Step 2: Delete the entire `# Container Registry — git.fxhnt.ai:5050` server block**
+
+Remove the whole block (the `# Container Registry — git.fxhnt.ai:5050` comment + its `server { listen 5050
+ssl; ... }`, i.e. the `proxy_pass http://gitlab-registry.foxhunt.svc.cluster.local:5000;` server). Leave
+the `git.fxhnt.ai:443` (Gitea) block and all other server blocks intact.
+
+- [ ] **Step 3: Confirm no GitLab upstream remains in the proxy**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+grep -nE "gitlab-registry|gitlab-webservice|gitlab-gitlab-shell|listen 5050" infra/k8s/gitlab/tailscale-proxy.yaml || echo "OK: no gitlab upstreams left"
+```
+Expected: `OK: no gitlab upstreams left`.
+
+- [ ] **Step 4: Apply + restart proxy**
+```bash
+kubectl apply -f infra/k8s/gitlab/tailscale-proxy.yaml
+kubectl rollout restart deploy/tailscale-gitlab-proxy -n foxhunt
+kubectl rollout status deploy/tailscale-gitlab-proxy -n foxhunt --timeout=120s
+```
+
+- [ ] **Step 5: GATE — proxy still serves all `*.fxhnt.ai`**
+```bash
+for h in git dashboard grafana minio; do curl -sk -m 10 -o /dev/null -w "$h.fxhnt.ai: %{http_code}\n" https://$h.fxhnt.ai/ 2>&1; done
+curl -sk -m 10 https://git.fxhnt.ai/api/healthz 2>/dev/null | grep -o '"status": "pass"'
+```
+Expected: `git`/`dashboard` 200 (others may 200/302/401 — just not 000). `"status": "pass"` for Gitea.
+**STOP + revert (`git checkout` the file + re-apply) if git/dashboard go 000.**
+
+---
+
+## Task 3: Helm-uninstall GitLab + agent
+
+- [ ] **Step 1: Uninstall both releases**
+```bash
+helm uninstall gitlab -n foxhunt 2>&1 | tail -1
+helm uninstall foxhunt-agent -n foxhunt 2>&1 | tail -1
+```
+Expected: both `uninstalled`.
+
+- [ ] **Step 2: Wait for GitLab pods to terminate + confirm RAM reclaimed**
+```bash
+i=0; until [ -z "$(kubectl get pods -n foxhunt 2>/dev/null | grep -iE 'gitlab|gitaly' | grep -v Terminating)" ] || [ $i -ge 24 ]; do sleep 5; i=$((i+1)); done
+echo "remaining gitlab pods:"; kubectl get pods -n foxhunt 2>/dev/null | grep -iE "gitlab|gitaly" || echo "  (none)"
+```
+Expected: no GitLab pods (or only `Terminating` briefly). The `gitaly` PVC remains (cold backup).
+
+---
+
+## Task 4: Delete dead Argo Events
+
+- [ ] **Step 1: Delete the GitLab CI eventsource + sensor**
+```bash
+kubectl delete eventsource gitlab-push -n foxhunt 2>&1 | tail -1
+kubectl delete sensor ci-pipeline-trigger -n foxhunt 2>&1 | tail -1
+echo "=== remaining events (gitea-push + gitea-deploy must stay) ==="
+kubectl get eventsource,sensor -n foxhunt 2>/dev/null
+```
+Expected: both deleted; `gitea-push` eventsource + `gitea-deploy` sensor still present (+ workflow-trigger).
+
+---
+
+## Task 5: Verify cold backup intact + remove values manifest + commit
+
+- [ ] **Step 1: Confirm the cold backup is untouched**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+echo "gitaly PVC (must remain):"; kubectl get pvc -n foxhunt 2>/dev/null | grep gitaly || echo "  MISSING!"
+echo "gitlab secrets (sample, must remain):"; kubectl get secret -n foxhunt 2>/dev/null | grep -c gitlab | xargs echo "  gitlab-* secrets:"
+export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
+echo "Scaleway gitlab buckets (must remain):"; scw object bucket list 2>/dev/null | grep -i gitlab
+```
+Expected: gitaly PVC present; gitlab secrets count > 0; both Scaleway gitlab buckets present. **STOP if the
+gitaly PVC is gone** (helm uninstall should not have removed a StatefulSet PVC — investigate).
+
+- [ ] **Step 2: Remove the GitLab Helm values manifest**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git rm infra/k8s/gitlab/values.yaml 2>&1 | tail -1
+ls infra/k8s/gitlab/   # tailscale-proxy.yaml must remain
+```
+Expected: `values.yaml` removed; `tailscale-proxy.yaml` still listed.
+
+- [ ] **Step 3: Commit**
+```bash
+git add -A infra/k8s/gitlab/
+git commit -m "chore(infra): decommission GitLab compute — keep data as cold backup (Phase 2C)
+
+helm uninstall gitlab + foxhunt-agent (~4.6GiB RAM / 12 pods freed); removed :5050
+registry block from the (now Gitea) proxy; deleted dead gitlab-push/ci-pipeline-trigger
+Argo events; removed gitlab values.yaml. RETAINED (cold backup): gitaly PVC,
+gitlab MinIO/Scaleway buckets, gitlab-* secrets. tailscale-proxy.yaml kept (serves Gitea).
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 6: Final verification
+
+- [ ] **Step 1: Releases gone + platform healthy + auto-deploy still works**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+echo "=== helm (no gitlab/agent) ==="; helm list -n foxhunt | grep -iE "gitlab" || echo "  (none — removed)"
+echo "=== platform health ==="
+curl -sk -m 10 -o /dev/null -w "git.fxhnt.ai: %{http_code}\n" https://git.fxhnt.ai/api/healthz
+curl -sk -m 10 -o /dev/null -w "dashboard.fxhnt.ai: %{http_code}\n" https://dashboard.fxhnt.ai
+echo "=== auto-deploy still wired (push test) ==="
+BEFORE=$(kubectl get wf -n foxhunt 2>/dev/null | grep -c fxhnt-cockpit)
+( cd /home/jgrusewski/Work/fxhnt && git commit --allow-empty -m "test: post-GitLab-removal auto-deploy" -q && GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin master 2>&1 | tail -1 )
+i=0; until [ "$(kubectl get wf -n foxhunt 2>/dev/null | grep -c fxhnt-cockpit)" -gt "$BEFORE" ] || [ $i -ge 12 ]; do sleep 5; i=$((i+1)); done
+echo "cockpit wfs: $BEFORE -> $(kubectl get wf -n foxhunt 2>/dev/null | grep -c fxhnt-cockpit)"
+```
+Expected: no gitlab helm releases; `git`/`dashboard` 200; cockpit wf count incremented (webhook→Argo intact).
+
+---
+
+## Rollback
+- **Proxy (Task 2):** `git checkout infra/k8s/gitlab/tailscale-proxy.yaml` + `kubectl apply` + rollout restart.
+- **GitLab (Task 3):** data is retained (gitaly PVC + buckets + secrets) — `helm install gitlab gitlab/gitlab
+  --version 9.9.3 -f <values>` would re-attach the gitaly PVC. (Primary going forward is Gitea/Scaleway.)
+
+---
+
+## Acceptance criteria (from spec)
+- No `gitlab`/`foxhunt-agent` helm releases; GitLab pods gone (~4.6 GiB reclaimed). ✅ Task 3, Task 6
+- `git.fxhnt.ai` + `dashboard.fxhnt.ai` 200; push auto-deploys via Gitea→Argo. ✅ Task 6
+- `gitlab-push`/`ci-pipeline-trigger` deleted; `gitea-push`/`gitea-deploy` kept. ✅ Task 4
+- `:5050` block gone; proxy healthy. ✅ Task 2
+- Cold backup intact: gitaly PVC + GitLab buckets + `gitlab-*` secrets. ✅ Task 5
+- `values.yaml` removed; `tailscale-proxy.yaml` kept; committed. ✅ Task 5

docs/superpowers/plans/2026-06-21-decommission-rust-infra.md

@@ -0,0 +1,263 @@
+# Decommission Dead Rust Infra — Implementation Plan (Phase 1)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
+> **Do NOT run this subagent-driven** — every task performs irreversible production deletes that need
+> human confirmation at each gate. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Remove the now-unused Rust-specific infra (GPU pools, GPU CI runners, build-cache PVCs, Rust
+artifact buckets, training/GPU manifests) — preserving ALL `.dbn` market data and every fxhnt/platform resource.
+
+**Architecture:** Verify-then-delete, gated. Terraform destroys the GPU/precompute pools (plan reviewed
+for exactly-3); helm uninstalls the GPU runners; kubectl deletes the unmounted build-cache PVCs; mc deletes
+the no-`.dbn` Rust buckets; repo + cluster Argo templates cleaned up. Cockpit/dagster/fund verified healthy.
+
+**Tech Stack:** terragrunt + OpenTofu (Scaleway provider), helm, kubectl, mc (MinIO), Scaleway Kapsule.
+
+**Spec:** `docs/superpowers/specs/2026-06-21-decommission-rust-infra-design.md`
+
+---
+
+## Shared env (export at the start of each shell session that runs terragrunt/scw)
+```bash
+cd /home/jgrusewski/Work/foxhunt
+export TG_TF_PATH=tofu TERRAGRUNT_TFPATH=tofu
+export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
+export SCW_DEFAULT_REGION=fr-par SCW_DEFAULT_ZONE=fr-par-2
+export TF_HTTP_USERNAME=root TF_HTTP_PASSWORD=$(kubectl get secret gitlab-pat -n foxhunt -o jsonpath='{.data.token}'|base64 -d)
+```
+
+---
+
+## Task 0: Branch
+- [ ] **Step 1**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git checkout -b chore/decommission-rust-infra
+```
+
+---
+
+## Task 1: Destroy the GPU + precompute pools (Terraform)
+
+**Files:** Modify `infra/live/production/kapsule/terragrunt.hcl`
+
+- [ ] **Step 1: Set the three enable flags to false**
+
+In `infra/live/production/kapsule/terragrunt.hcl`, change:
+```
+  enable_ci_compile_cpu_hm_pool = true
+```
+to `false`; and
+```
+  enable_ci_training_l40s_pool = true
+```
+to `false`; and
+```
+  enable_ci_training_h100_pool = true
+```
+to `false`. (Leave `enable_ci_compile_cpu_pool = true` — Python cockpit builds use it.)
+
+- [ ] **Step 2: Plan and GATE on exactly-3-destroys**
+
+Run (with shared env, in `infra/live/production/kapsule`):
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/kapsule
+terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "^Plan:|will be destroyed|# scaleway"
+```
+Expected: `Plan: 0 to add, 0 to change, 3 to destroy.` and the 3 destroyed are EXACTLY
+`scaleway_k8s_pool.ci_compile_cpu_hm[0]`, `scaleway_k8s_pool.ci_training_l40s[0]`,
+`scaleway_k8s_pool.ci_training_h100[0]`.
+**STOP if the count ≠ 3 or any other resource is destroyed.**
+
+- [ ] **Step 3: Apply**
+```bash
+terragrunt apply -input=false -no-color -auto-approve 2>&1 | sed 's/.*tofu: //' | grep -iE "Apply complete|Destroy complete|Error"
+```
+Expected: `Apply complete! Resources: 0 added, 0 changed, 3 destroyed.`
+
+- [ ] **Step 4: Verify pools gone**
+```bash
+scw k8s pool list cluster-id=34a1e3c4-ac35-48c8-ab49-5f6ec4df32c1 -o json 2>/dev/null | python3 -c "import sys,json;print([p['name'] for p in json.load(sys.stdin)])"
+```
+Expected: `['ci-compile-cpu', 'platform']` (no l40s/h100/hm).
+
+- [ ] **Step 5: Commit**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git add infra/live/production/kapsule/terragrunt.hcl
+git commit -m "chore(infra): destroy unused Rust GPU + precompute pools (L40S/H100/hm)
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 2: Uninstall the GPU CI runners (helm)
+
+- [ ] **Step 1: Uninstall the 3 GPU runners**
+```bash
+for r in gitlab-runner-h100 gitlab-runner-h100-sxm gitlab-runner-h100x2; do
+  helm uninstall "$r" -n foxhunt 2>&1 | tail -1
+done
+```
+Expected: `release "<r>" uninstalled` for each.
+
+- [ ] **Step 2: Decide the main `gitlab-runner`**
+
+Check whether anything still uses it (fxhnt has no `.gitlab-ci.yml`; cockpit deploys via Argo):
+```bash
+kubectl get pods -n foxhunt | grep gitlab-runner   # expect: no running runner pods
+# Inspect what the main runner is registered for (manual judgement):
+helm get values gitlab-runner -n foxhunt 2>/dev/null | grep -iE "tags|runUntagged|description" | head
+```
+- [ ] **Step 3: If confirmed dead, uninstall it; else keep (document the decision):**
+```bash
+helm uninstall gitlab-runner -n foxhunt 2>&1 | tail -1   # ONLY if Step 2 confirms no consumer
+```
+
+- [ ] **Step 4: Verify**
+```bash
+helm list -n foxhunt | grep -i runner   # expect: none (or only a deliberately-kept one)
+```
+
+---
+
+## Task 3: Delete the build-cache PVCs (re-verify unmounted first)
+
+- [ ] **Step 1: Re-confirm each PVC is unmounted (safety)**
+```bash
+for p in cargo-target-cpu cargo-target-cuda cargo-target-cuda-test sccache-cpu sccache-cuda; do
+  echo -n "$p: "; kubectl describe pvc $p -n foxhunt 2>/dev/null | grep -A1 "Used By" | tr '\n' ' '; echo
+done
+```
+Expected: every line shows `Used By: <none>`. **STOP on any PVC that lists a pod.**
+
+- [ ] **Step 2: Delete them**
+```bash
+kubectl delete pvc cargo-target-cpu cargo-target-cuda cargo-target-cuda-test sccache-cpu sccache-cuda -n foxhunt 2>&1 | tail
+```
+Expected: `persistentvolumeclaim "<name>" deleted` ×5.
+
+- [ ] **Step 3: Verify reclaim**
+```bash
+kubectl get pvc -n foxhunt | grep -E "cargo-target|sccache" || echo "all build-cache PVCs gone (~175Gi reclaimed)"
+```
+
+---
+
+## Task 4: Delete the Rust artifact buckets (re-verify 0 `.dbn` first)
+
+- [ ] **Step 1: Start a MinIO port-forward + mc alias**
+```bash
+pkill -f "port-forward svc/minio" 2>/dev/null; sleep 1
+kubectl port-forward svc/minio -n foxhunt 19000:9000 >/tmp/minio-pf.log 2>&1 &
+sleep 4
+ak=$(kubectl get secret minio-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+sk=$(kubectl get secret minio-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+mc alias set fxl http://localhost:19000 "$ak" "$sk"
+```
+
+- [ ] **Step 2: Re-verify the 3 target buckets contain ZERO `.dbn`/market data, and the KEPT one does**
+```bash
+for b in foxhunt-binaries foxhunt-training-results foxhunt-models; do
+  echo -n "$b dbn-count="; mc ls --recursive fxl/$b 2>/dev/null | grep -icE "\.dbn|\.zst|databento|mbp"
+done
+echo -n "KEEP foxhunt-training-data dbn-count="; mc ls --recursive fxl/foxhunt-training-data 2>/dev/null | grep -icE "\.dbn|\.zst|databento|mbp"
+```
+Expected: the 3 targets show `0`; `foxhunt-training-data` shows `>0`. **STOP if any target shows >0.**
+
+- [ ] **Step 3: Delete the 3 Rust buckets**
+```bash
+for b in foxhunt-binaries foxhunt-training-results foxhunt-models; do
+  mc rb --force fxl/$b 2>&1 | tail -1
+done
+```
+Expected: `Removed '<bucket>' successfully.` ×3.
+
+- [ ] **Step 4: Verify market data intact**
+```bash
+mc du fxl/foxhunt-training-data   # expect: still ~38GiB, 54+ objects
+kubectl get pvc training-data-pvc test-data-pvc feature-cache-pvc -n foxhunt   # expect: all 3 still Bound
+pkill -f "port-forward svc/minio" 2>/dev/null
+```
+
+---
+
+## Task 5: Remove dead manifests + Argo templates (repo + cluster)
+
+**Files:** Remove `infra/k8s/training/`, `infra/k8s/gpu-overlays/`, Rust Argo templates, the databento job, GPU-runner helm values.
+
+- [ ] **Step 1: Identify the Rust Argo WorkflowTemplates in the cluster (keep fxhnt-cockpit!)**
+```bash
+kubectl get wftmpl -n foxhunt -o name 2>/dev/null
+```
+Note the Rust ones (e.g. train-multi-seed, lob-backtest-sweep, ci-pipeline, alpha-rl-*). **Do NOT touch `fxhnt-cockpit`.**
+
+- [ ] **Step 2: Delete the dead WorkflowTemplates from the cluster** (substitute the exact names from Step 1):
+```bash
+# example — replace with the actual Rust template names from Step 1:
+kubectl delete wftmpl -n foxhunt <rust-template-1> <rust-template-2> ... 2>&1 | tail
+```
+
+- [ ] **Step 3: Remove dead manifests + the now-unused Terraform pool blocks from the repo**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git rm -r infra/k8s/training infra/k8s/gpu-overlays 2>/dev/null
+git rm infra/k8s/jobs/download-trades-job.yaml infra/k8s/argo/train-multi-seed-template.yaml infra/k8s/argo/lob-backtest-sweep-template.yaml infra/k8s/argo/ci-pipeline-template.yaml 2>/dev/null
+# (also git rm any GPU-runner helm value files + alpha-rl argo templates found under infra/)
+ls infra/k8s/argo infra/k8s/jobs   # eyeball what remains; keep cockpit/fxhnt/fund + cert-manager etc.
+```
+- [ ] **Step 4: Remove the dead pool resource blocks + variables from the kapsule module** (optional tidy):
+in `infra/modules/kapsule/main.tf` remove the `scaleway_k8s_pool.ci_training_l40s`,
+`...ci_training_h100`, `...ci_compile_cpu_hm` resource blocks (now count=0); in `variables.tf` remove
+their `enable_*`/`*_type`/`*_max_size` vars; in `terragrunt.hcl` remove the 3 dead `enable_*`/`*_type`
+lines. Then re-plan to confirm still clean:
+```bash
+cd infra/live/production/kapsule && terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "^Plan:|No changes"
+```
+Expected: `No changes` (removing count=0 blocks is a no-op against the cluster).
+
+- [ ] **Step 5: Commit**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git add -A infra
+git commit -m "chore(infra): remove dead Rust training/GPU manifests, Argo templates, pool config
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+```
+
+---
+
+## Task 6: Final verification (cluster + fund still healthy)
+
+- [ ] **Step 1: Platform/fund health**
+```bash
+curl -sS -m 12 -o /dev/null -w "dashboard HTTP %{http_code}\n" https://dashboard.fxhnt.ai/
+kubectl get pods -n foxhunt | grep -E "dagster|fxhnt-dashboard|gitlab-webservice" | grep -vE "Completed"
+kubectl get pvc training-data-pvc test-data-pvc feature-cache-pvc fxhnt-surfer-data fxhnt-backtest-data -n foxhunt
+```
+Expected: dashboard 200; dagster/dashboard/gitlab Running; all kept PVCs Bound.
+
+- [ ] **Step 2: Terraform clean**
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/kapsule && terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "^Plan:|No changes"
+```
+Expected: `No changes`.
+
+- [ ] **Step 3: Merge to main**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git checkout main && git merge --ff-only chore/decommission-rust-infra && git branch -d chore/decommission-rust-infra
+git push origin main
+```
+
+---
+
+## Notes for the executor
+- **Data safety is paramount:** never delete `training-data-pvc`, `test-data-pvc`, `feature-cache-pvc`, the
+  `foxhunt-training-data` bucket, or any fxhnt/platform resource. Re-verify (`Used By`, `.dbn` count) at each
+  delete gate; STOP on any surprise.
+- **Terraform gate:** Task 1 Step 2 must show exactly the 3 GPU/precompute pools destroyed — stop otherwise.
+- `pkill -f "port-forward svc/minio"` after Task 4 to clean up the forward.
+- If the main `gitlab-runner` is ambiguous, KEEP it (a kept idle runner is harmless; a wrongly-removed one breaks CI).

docs/superpowers/plans/2026-06-21-gitea-replace-gitlab.md

@@ -0,0 +1,472 @@
+# Gitea Replaces GitLab + Cockpit Registry to Scaleway — Implementation Plan (Phase 2B)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
+> **Do NOT run subagent-driven** — this touches live git access, the container registry, and the cockpit
+> deploy; each cutover step needs a human-confirmed gate. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Stand up Gitea (chart + existing Postgres) as the git host behind the unchanged `git.fxhnt.ai`,
+migrate both repos, and move the cockpit image to Scaleway Container Registry — so GitLab can be removed in 2C.
+
+**Architecture:** Gitea comes up internal-only; repos migrate + validate over a port-forward; the cockpit
+image moves to Scaleway registry and is build+rollout tested; THEN `git.fxhnt.ai` (HTTPS + SSH `:2222`)
+cuts over to Gitea in one switch; finally a Gitea webhook → Argo Events auto-deploys the cockpit. GitLab
+stays running (unaddressed) as the rollback net until 2C.
+
+**Tech Stack:** Gitea Helm chart (`gitea-charts/gitea`), existing in-cluster PostgreSQL, nginx+socat
+Tailscale proxy, Scaleway Container Registry, Argo Events, kaniko.
+
+**Spec:** `docs/superpowers/specs/2026-06-21-gitea-replace-gitlab-design.md`
+
+**TWO REPOS:** infra changes → `foxhunt` (`~/Work/foxhunt`); cockpit build/deploy → `fxhnt`
+(`~/Work/fxhnt`). Each gets its own branch + commits.
+
+---
+
+## Shared env
+```bash
+export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
+export SCW_DEFAULT_REGION=fr-par SCW_DEFAULT_ZONE=fr-par-2
+```
+
+---
+
+## Task 0: Branches
+- [ ] **Step 1**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git checkout -b chore/gitea-replace-gitlab
+cd /home/jgrusewski/Work/fxhnt   && git checkout -b chore/gitea-cockpit-registry
+```
+
+---
+
+## Task 1: Postgres `gitea` DB + secrets
+
+**Files:** none committed (cluster secrets). Record secret names only.
+
+- [ ] **Step 1: Read existing postgres superuser creds**
+```bash
+PGU=$(kubectl get deploy postgres -n foxhunt -o jsonpath='{.spec.template.spec.containers[0].env[?(@.name=="POSTGRES_USER")].value}')
+PGPOD=$(kubectl get pod -n foxhunt -l app=postgres -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || kubectl get pod -n foxhunt | grep '^postgres-' | grep -v backup | awk '{print $1}' | head -1)
+echo "postgres user=$PGU pod=$PGPOD"
+```
+Expected: a username + a running pod name. (If `POSTGRES_USER` is via secretRef not value, read it from the referenced secret instead.)
+
+- [ ] **Step 2: Generate a Gitea DB password + create role/db**
+```bash
+GITEA_DB_PW=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
+kubectl exec -n foxhunt "$PGPOD" -- psql -U "$PGU" -v ON_ERROR_STOP=1 -c \
+  "CREATE ROLE gitea LOGIN PASSWORD '$GITEA_DB_PW';" -c \
+  "CREATE DATABASE gitea OWNER gitea ENCODING 'UTF8';" 2>&1 | grep -iE "CREATE|already exists|Error"
+echo "$GITEA_DB_PW" # capture for the next step
+```
+Expected: `CREATE ROLE` + `CREATE DATABASE` (or `already exists` if re-run).
+
+- [ ] **Step 3: Create the k8s secrets Gitea needs** (DB creds + admin)
+```bash
+GITEA_ADMIN_PW=$(openssl rand -base64 24 | tr -d '/+=' | head -c 24)
+kubectl create secret generic gitea-db -n foxhunt \
+  --from-literal=password="$GITEA_DB_PW" --dry-run=client -o yaml | kubectl apply -f -
+kubectl create secret generic gitea-admin -n foxhunt \
+  --from-literal=username=gitadmin --from-literal=password="$GITEA_ADMIN_PW" \
+  --from-literal=email=jeroen@bizworx.nl --dry-run=client -o yaml | kubectl apply -f -
+echo "ADMIN PW (save in your password manager): $GITEA_ADMIN_PW"
+```
+Expected: both secrets `created`/`configured`. **Save the admin password** — needed for Gitea login.
+
+---
+
+## Task 2: Gitea Helm values + install (internal-only)
+
+**Files:** Create `infra/k8s/gitea/values.yaml` (foxhunt repo).
+
+- [ ] **Step 1: Add the Gitea Helm repo**
+```bash
+helm repo add gitea-charts https://dl.gitea.com/charts/ && helm repo update gitea-charts
+helm search repo gitea-charts/gitea --versions | head -3
+```
+Expected: lists chart versions (use the latest stable in Step 3).
+
+- [ ] **Step 2: Write `infra/k8s/gitea/values.yaml`**
+```yaml
+# Gitea — lightweight git host replacing GitLab (Phase 2B). External Postgres (existing in-cluster
+# `postgres`), no bundled DB/redis/memcached, Actions off (Argo does CI). Internal-only until cutover.
+replicaCount: 1
+image:
+  rootless: true
+
+# Disable all bundled subcharts — reuse the existing in-cluster postgres
+postgresql:
+  enabled: false
+postgresql-ha:
+  enabled: false
+redis-cluster:
+  enabled: false
+redis:
+  enabled: false
+
+persistence:
+  enabled: true
+  size: 5Gi
+  storageClass: sbs-default-retain
+
+resources:
+  requests: { cpu: 100m, memory: 128Mi }
+  limits:   { cpu: "1",  memory: 512Mi }
+
+service:
+  http: { type: ClusterIP, port: 3000 }
+  ssh:  { type: ClusterIP, port: 22 }
+
+actions:
+  enabled: false
+
+gitea:
+  admin:
+    existingSecret: gitea-admin
+  config:
+    server:
+      ROOT_URL: https://git.fxhnt.ai/
+      DOMAIN: git.fxhnt.ai
+      SSH_DOMAIN: git.fxhnt.ai
+      SSH_PORT: "22"            # clean git@git.fxhnt.ai clone URLs. Port 22 is free (nothing host-SSHes
+                               # the git node); :2222 retired, local remotes updated in Task 5 Step 2b.
+      DISABLE_SSH: "false"
+    database:
+      DB_TYPE: postgres
+      HOST: postgres.foxhunt.svc.cluster.local:5432
+      NAME: gitea
+      USER: gitea
+    service:
+      DISABLE_REGISTRATION: "true"
+    cache:
+      ADAPTER: memory
+  additionalConfigFromEnvs:
+    - name: GITEA__database__PASSWD
+      valueFrom:
+        secretKeyRef: { name: gitea-db, key: password }
+```
+
+- [ ] **Step 3: Install (internal-only — no public hostname yet)**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+helm install gitea gitea-charts/gitea -n foxhunt -f infra/k8s/gitea/values.yaml --wait --timeout 5m 2>&1 | tail -5
+```
+Expected: `STATUS: deployed`. (If `--wait` times out, continue; verify pod in Step 4.)
+
+- [ ] **Step 4: Verify Gitea is healthy**
+```bash
+kubectl get pods -n foxhunt | grep gitea
+kubectl run gitea-probe --rm -i --restart=Never -n foxhunt --image=curlimages/curl -- \
+  curl -s http://gitea-http.foxhunt.svc.cluster.local:3000/api/healthz
+```
+Expected: a JSON health blob with `"status": "pass"`. Pod `Running`.
+
+- [ ] **Step 5: Commit values (foxhunt repo)**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git add infra/k8s/gitea/values.yaml
+git commit -m "feat(infra): Gitea Helm values (external postgres, internal-only) — Phase 2B"
+```
+
+---
+
+## Task 3: Migrate both repos via port-forward + validate
+
+**Files:** none committed (git data operations).
+
+- [ ] **Step 1: Port-forward Gitea + create the repos via API**
+```bash
+kubectl port-forward -n foxhunt svc/gitea-http 3000:3000 >/tmp/gitea-pf.log 2>&1 & echo $! >/tmp/gitea-pf.pid
+sleep 3
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+for r in fxhnt foxhunt; do
+  curl -s -u "$GA:$GP" -X POST http://localhost:3000/api/v1/user/repos \
+    -H 'Content-Type: application/json' -d "{\"name\":\"$r\",\"private\":true}" -o /dev/null -w "$r: %{http_code}\n"
+done
+```
+Expected: `fxhnt: 201` and `foxhunt: 201`.
+
+- [ ] **Step 2: Mirror-push both repos from local clones**
+```bash
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+( cd /home/jgrusewski/Work/fxhnt   && git push --mirror "http://$GA:$GP@localhost:3000/$GA/fxhnt.git" 2>&1 | tail -2 )
+( cd /home/jgrusewski/Work/foxhunt && git push --mirror "http://$GA:$GP@localhost:3000/$GA/foxhunt.git" 2>&1 | tail -2 )
+```
+Expected: both report refs pushed (no error). foxhunt is 876 MB — may take a minute.
+
+- [ ] **Step 3: VALIDATE — clone back + diff top SHAs vs GitLab**
+```bash
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+for r in fxhnt foxhunt; do
+  gitea_sha=$(git ls-remote "http://$GA:$GP@localhost:3000/$GA/$r.git" HEAD | awk '{print $1}')
+  gitlab_sha=$(cd /home/jgrusewski/Work/$r && git ls-remote origin HEAD | awk '{print $1}')
+  [ "$gitea_sha" = "$gitlab_sha" ] && echo "$r: MATCH ($gitea_sha)" || echo "$r: MISMATCH gitea=$gitea_sha gitlab=$gitlab_sha"
+done
+```
+Expected: `fxhnt: MATCH` and `foxhunt: MATCH`. **STOP if any MISMATCH.**
+
+- [ ] **Step 4: Mark foxhunt archived (read-only)**
+```bash
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+curl -s -u "$GA:$GP" -X PATCH "http://localhost:3000/api/v1/repos/$GA/foxhunt" \
+  -H 'Content-Type: application/json' -d '{"archived":true}' -o /dev/null -w "archive foxhunt: %{http_code}\n"
+kill $(cat /tmp/gitea-pf.pid) 2>/dev/null
+```
+Expected: `archive foxhunt: 200`.
+
+---
+
+## Task 4: Move cockpit image to Scaleway Container Registry
+
+**Files:** Modify `~/Work/fxhnt/infra/argo/cockpit-build-deploy.yaml`,
+`~/Work/fxhnt/infra/k8s/orchestration/dagster.yaml` (+ dashboard manifest if separate).
+
+- [ ] **Step 1: Create the Scaleway registry pull/push secret**
+```bash
+kubectl create secret docker-registry scw-registry -n foxhunt \
+  --docker-server=rg.fr-par.scw.cloud \
+  --docker-username=nologin --docker-password="$SCW_SECRET_KEY" \
+  --dry-run=client -o yaml | kubectl apply -f -
+```
+Expected: secret `created`/`configured`. (Scaleway registry auth: username `nologin`, password = a
+Scaleway secret key.)
+
+- [ ] **Step 2: Repoint kaniko build → Scaleway registry**
+
+In `~/Work/fxhnt/infra/argo/cockpit-build-deploy.yaml`, change the kaniko args:
+```
+              --destination=rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest \
+              --cache-repo=rg.fr-par.scw.cloud/bizworx/cache \
+```
+Remove the `--insecure-registry=…gitlab-registry…:5000` and `--skip-tls-verify-registry=…` lines
+(Scaleway is TLS). Mount the `scw-registry` dockerconfig for kaniko (replace the `gitlab-registry`
+dockerconfig volume): set the kaniko `DOCKER_CONFIG`/volume to the `scw-registry` secret at
+`/kaniko/.docker/config.json`.
+
+- [ ] **Step 3: Repoint the deploy image refs**
+
+In `~/Work/fxhnt/infra/k8s/orchestration/dagster.yaml` (and the dashboard manifest), change every
+`image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/fxhnt-cockpit:latest` →
+`image: rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest`, and add under each pod spec:
+```yaml
+      imagePullSecrets:
+        - name: scw-registry
+```
+
+- [ ] **Step 4: Run one cockpit build → Scaleway + verify the image exists**
+```bash
+cd /home/jgrusewski/Work/fxhnt && ./scripts/argo-deploy-cockpit.sh 2>&1 | tail -8
+# verify the pushed image
+scw registry image list namespace-id=6561a90a-f4ba-4f44-b43f-be6e27cbdfca 2>/dev/null | grep -i fxhnt-cockpit || echo "(verify via Scaleway console)"
+```
+Expected: workflow `Succeeded`; `fxhnt-cockpit` image listed in the `bizworx` namespace.
+
+- [ ] **Step 5: Verify a rollout pulls from Scaleway**
+```bash
+kubectl rollout restart deploy/dagster deploy/fxhnt-dashboard -n foxhunt 2>/dev/null
+kubectl rollout status deploy/dagster -n foxhunt --timeout=180s
+kubectl get pods -n foxhunt -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | grep -i "rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit" | head
+```
+Expected: rollout succeeds; pods now reference the `rg.fr-par.scw.cloud/bizworx/...` image (no `ImagePullBackOff`).
+
+- [ ] **Step 6: Commit (fxhnt repo)**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git add infra/argo/cockpit-build-deploy.yaml infra/k8s/orchestration/dagster.yaml
+git commit -m "feat(infra): cockpit image -> Scaleway Container Registry (Phase 2B)"
+```
+
+---
+
+## Task 5: Cut `git.fxhnt.ai` + SSH `:2222` over to Gitea
+
+**Files:** Modify `infra/k8s/gitlab/tailscale-proxy.yaml` (foxhunt repo). **CHECKPOINT before applying.**
+
+- [ ] **Step 1: Repoint nginx `git.fxhnt.ai:443` → Gitea**
+
+In `infra/k8s/gitlab/tailscale-proxy.yaml`, in the `server_name git.fxhnt.ai;` (`listen 443`) block,
+change:
+```
+            proxy_pass http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181;
+```
+to:
+```
+            proxy_pass http://gitea-http.foxhunt.svc.cluster.local:3000;
+```
+Delete the entire `listen 5050 ssl; server_name git.fxhnt.ai;` registry block (Scaleway registry is
+external — not proxied).
+
+- [ ] **Step 2: Repoint the socat SSH proxy to Gitea SSH on `:22` (drop `:2222`)**
+
+Nothing host-SSHes the git node, so use the standard port `22` and retire `:2222` entirely (clean
+`git@git.fxhnt.ai` URLs). In the `ssh-proxy` container args, change the listen port to `22` and the target
+to Gitea:
+```
+            - "TCP-LISTEN:22,fork,reuseaddr,sndbuf=1048576,rcvbuf=1048576"
+            - "TCP:gitea-ssh.foxhunt.svc.cluster.local:22,sndbuf=1048576,rcvbuf=1048576"
+```
+Update the container `ports:` entry from `containerPort: 2222` → `containerPort: 22`. Ensure the Tailscale
+node/proxy advertises port `22` (update the tailnet sidecar's exposed ports / `serve` config: `22`,
+replacing `2222`).
+
+- [ ] **Step 2b: Update local git remotes from `:2222` → default (port 22)**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git remote set-url origin ssh://git@git.fxhnt.ai/gitadmin/foxhunt.git
+cd /home/jgrusewski/Work/fxhnt   && git remote set-url origin ssh://git@git.fxhnt.ai/gitadmin/fxhnt.git
+cd /home/jgrusewski/Work/foxhunt && git remote -v | grep fetch
+```
+Expected: remotes now show `ssh://git@git.fxhnt.ai/...` (no `:2222`).
+
+- [ ] **Step 3: Apply + restart the proxy**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+kubectl apply -f infra/k8s/gitlab/tailscale-proxy.yaml
+kubectl rollout restart deploy/tailscale-gitlab-proxy -n foxhunt
+kubectl rollout status deploy/tailscale-gitlab-proxy -n foxhunt --timeout=120s
+```
+Expected: rollout complete.
+
+- [ ] **Step 4: GATE — verify `git.fxhnt.ai` serves Gitea**
+```bash
+curl -sk https://git.fxhnt.ai/api/healthz | head -c 200; echo
+echo "--- SSH on :22 (git@git.fxhnt.ai) ---"; git ls-remote ssh://git@git.fxhnt.ai/gitadmin/fxhnt.git HEAD 2>&1 | head -1
+```
+Expected: Gitea healthz JSON; the `:22` ssh ls-remote returns the HEAD sha (proves Gitea web + SSH on the
+canonical name). **STOP + rollback (revert this file, re-apply) if either fails.**
+
+- [ ] **Step 5: Commit (foxhunt repo)**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git add infra/k8s/gitlab/tailscale-proxy.yaml
+git commit -m "feat(infra): cut git.fxhnt.ai + SSH :2222 over to Gitea; drop :5050 registry (Phase 2B)"
+```
+
+---
+
+## Task 6: Wire Gitea push webhook → Argo Events auto-deploy
+
+**Files:** Create `infra/k8s/argo/events/gitea-push-eventsource.yaml` +
+`infra/k8s/argo/events/gitea-deploy-sensor.yaml` (foxhunt repo).
+
+- [ ] **Step 1: Create the webhook eventsource**
+
+Create `infra/k8s/argo/events/gitea-push-eventsource.yaml`:
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: EventSource
+metadata:
+  name: gitea-push
+  namespace: foxhunt
+spec:
+  service:
+    ports:
+      - port: 12000
+        targetPort: 12000
+  webhook:
+    fxhnt-push:
+      port: "12000"
+      endpoint: /push
+      method: POST
+```
+
+- [ ] **Step 2: Create the sensor that submits the cockpit workflow**
+
+Create `infra/k8s/argo/events/gitea-deploy-sensor.yaml`:
+```yaml
+apiVersion: argoproj.io/v1alpha1
+kind: Sensor
+metadata:
+  name: gitea-deploy
+  namespace: foxhunt
+spec:
+  dependencies:
+    - name: push
+      eventSourceName: gitea-push
+      eventName: fxhnt-push
+      filters:
+        data:
+          - path: body.ref
+            type: string
+            value: ["refs/heads/main", "refs/heads/master"]
+  triggers:
+    - template:
+        name: submit-cockpit
+        argoWorkflow:
+          operation: submit
+          source:
+            resource:
+              apiVersion: argoproj.io/v1alpha1
+              kind: Workflow
+              metadata:
+                generateName: fxhnt-cockpit-
+                namespace: foxhunt
+              spec:
+                workflowTemplateRef:
+                  name: fxhnt-cockpit
+```
+
+- [ ] **Step 3: Apply + register the webhook in Gitea**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+kubectl apply -f infra/k8s/argo/events/gitea-push-eventsource.yaml -f infra/k8s/argo/events/gitea-deploy-sensor.yaml
+kubectl get pods -n foxhunt | grep -iE "gitea-push|gitea-deploy"
+# register webhook (in-cluster URL) on the fxhnt repo
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+kubectl port-forward -n foxhunt svc/gitea-http 3000:3000 >/tmp/gitea-pf.log 2>&1 & echo $! >/tmp/gitea-pf.pid; sleep 3
+curl -s -u "$GA:$GP" -X POST "http://localhost:3000/api/v1/repos/$GA/fxhnt/hooks" -H 'Content-Type: application/json' -d '{
+  "type":"gitea","active":true,"events":["push"],
+  "config":{"url":"http://gitea-push-eventsource-svc.foxhunt.svc.cluster.local:12000/push","content_type":"json"}
+}' -o /dev/null -w "hook: %{http_code}\n"
+kill $(cat /tmp/gitea-pf.pid) 2>/dev/null
+```
+Expected: eventsource + sensor pods `Running`; `hook: 201`.
+
+- [ ] **Step 4: GATE — test push triggers a deploy**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git commit --allow-empty -m "test: trigger Gitea->Argo cockpit deploy"
+git push origin main   # origin already = git.fxhnt.ai (now Gitea)
+sleep 10
+kubectl get wf -n foxhunt --sort-by=.metadata.creationTimestamp 2>/dev/null | grep fxhnt-cockpit | tail -2
+```
+Expected: a new `fxhnt-cockpit-*` workflow appears (triggered by the push). **If none fires**, check the
+eventsource pod logs; fallback is the manual `./scripts/argo-deploy-cockpit.sh` (still works).
+
+- [ ] **Step 5: Commit (foxhunt repo)**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git add infra/k8s/argo/events/gitea-push-eventsource.yaml infra/k8s/argo/events/gitea-deploy-sensor.yaml
+git commit -m "feat(infra): Gitea push webhook -> Argo Events cockpit auto-deploy (Phase 2B)"
+```
+
+---
+
+## Task 7: Final verification
+
+- [ ] **Step 1: Footprint + health summary**
+```bash
+echo "=== Gitea (new) ==="; kubectl top pods -n foxhunt 2>/dev/null | grep -i gitea
+echo "=== cockpit healthy on Scaleway image ==="; curl -sk -o /dev/null -w "dashboard.fxhnt.ai: %{http_code}\n" https://dashboard.fxhnt.ai
+echo "=== git.fxhnt.ai serves Gitea ==="; curl -sk https://git.fxhnt.ai/api/healthz | head -c 80; echo
+echo "=== GitLab still running (fallback for 2C) ==="; kubectl get pods -n foxhunt | grep -c gitlab | xargs echo "gitlab pods:"
+```
+Expected: Gitea ≤~300 MiB; dashboard 200; git.fxhnt.ai healthz pass; GitLab pods still present.
+
+---
+
+## Rollback
+- **Cutover (Task 5):** `git checkout infra/k8s/gitlab/tailscale-proxy.yaml` + `kubectl apply` + rollout
+  restart → `git.fxhnt.ai`/SSH back to GitLab.
+- **Registry (Task 4):** revert the fxhnt manifests + rollout restart → pods pull the GitLab image again
+  (still present until 2C).
+- **Gitea:** `helm uninstall gitea -n foxhunt`; drop the `gitea` DB. GitLab was never touched.
+
+---
+
+## Acceptance criteria (from spec)
+- Gitea running on existing postgres, healthz pass, ≤~300 MiB. ✅ Task 2, Task 7
+- Both repos in Gitea, SHAs match GitLab, foxhunt archived. ✅ Task 3
+- Cockpit builds to + deploys from Scaleway registry; dashboard 200. ✅ Task 4, Task 7
+- `git.fxhnt.ai` (HTTPS + SSH :2222) serves Gitea; remotes unchanged. ✅ Task 5
+- Push to fxhnt main auto-triggers a cockpit deploy. ✅ Task 6
+- GitLab still running as fallback. ✅ Task 7
+```

docs/superpowers/plans/2026-06-21-relocate-iac-to-fxhnt.md

@@ -0,0 +1,206 @@
+# Relocate Platform IaC foxhunt → fxhnt + Archive foxhunt — Implementation Plan (Phase 2D)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
+> Spans TWO repos + a production Terraform state gate; the `terragrunt plan = No changes` check is the
+> safety pin. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Move active platform IaC (terraform `live`+`modules`, platform k8s manifests) from `foxhunt`
+into `fxhnt`, preserving TF state keys (no migration), then archive `foxhunt`.
+
+**Architecture:** Copy into fxhnt preserving the `live/production/<module>` structure → state keys stay
+identical → `terragrunt plan = No changes` proves it. Then `git rm` from foxhunt, commit/push both,
+archive foxhunt. Cluster untouched throughout (source-of-truth move only).
+
+**Tech Stack:** terragrunt + OpenTofu (Scaleway s3 backend), kubectl manifests, git, Gitea.
+
+**Repos:** source `~/Work/foxhunt`, dest `~/Work/fxhnt`. **Spec:**
+`docs/superpowers/specs/2026-06-21-relocate-iac-to-fxhnt-design.md`
+
+---
+
+## Shared env (terragrunt steps)
+```bash
+export TG_TF_PATH=tofu TERRAGRUNT_TFPATH=tofu
+export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
+export SCW_DEFAULT_REGION=fr-par SCW_DEFAULT_ZONE=fr-par-2
+export AWS_ACCESS_KEY_ID="$SCW_ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SCW_SECRET_KEY" AWS_REGION=fr-par
+```
+
+---
+
+## Task 0: Branches
+```bash
+cd /home/jgrusewski/Work/fxhnt   && git checkout master && git checkout -b chore/relocate-iac
+cd /home/jgrusewski/Work/foxhunt && git checkout main   && git checkout -b chore/relocate-iac-remove
+```
+
+---
+
+## Task 1: Collision pre-flight (must be empty)
+```bash
+cd /home/jgrusewski/Work/foxhunt
+for d in k8s/network-policies k8s/services k8s/jobs k8s/storage; do
+  echo "--- $d ---"; comm -12 <(ls infra/$d 2>/dev/null|sort) <(ls ~/Work/fxhnt/infra/$d 2>/dev/null|sort)
+done
+```
+Expected: no filenames printed. **STOP + resolve if any collision appears.**
+
+---
+
+## Task 2: Copy Terraform (live + modules) into fxhnt, structure preserved
+
+- [ ] **Step 1: rsync source files (exclude generated/cached, keep .terraform.lock.hcl)**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+rsync -a --exclude '.terraform/' --exclude 'backend.tf' --exclude 'provider.tf' --exclude '*.tfstate*' \
+  infra/live infra/modules /home/jgrusewski/Work/fxhnt/infra/
+find /home/jgrusewski/Work/fxhnt/infra/live /home/jgrusewski/Work/fxhnt/infra/modules -name '*.hcl' -o -name '*.tf' | sort
+```
+Expected: `fxhnt/infra/live/production/{root.hcl,kapsule,public-gateway,dns,block-storage}` +
+`fxhnt/infra/modules/{kapsule,public-gateway,dns,block-storage}` present.
+
+- [ ] **Step 2: Ensure fxhnt .gitignore covers terragrunt-generated files**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+grep -qE '^backend\.tf' .gitignore || printf '\n# terragrunt-generated\n**/.terraform/\n**/backend.tf\n**/provider.tf\n*.tfstate\n*.tfstate.*\n' >> .gitignore
+tail -7 .gitignore
+```
+
+- [ ] **Step 3: GATE — terragrunt plan = No changes for all 4 modules (from fxhnt)**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+for m in kapsule public-gateway dns block-storage; do
+  echo "=== $m ==="
+  ( cd infra/live/production/$m && terragrunt init -reconfigure -input=false -no-color >/dev/null 2>&1 && \
+    terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head -1 )
+done
+```
+Expected: every module prints `No changes.` — proves state keys are unchanged (no migration). **STOP if any
+module shows a plan diff or Error** (structure not preserved → do not proceed).
+
+---
+
+## Task 3: Copy platform k8s manifests + the one platform script into fxhnt
+
+- [ ] **Step 1: rsync platform k8s dirs (everything except gpu-taint.yaml) + namespace.yaml**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+for d in argo cert-manager databases dns gitea gitlab jobs mattermost minio monitoring network-policies secrets services stalwart tailscale training storage; do
+  rsync -a "infra/k8s/$d" /home/jgrusewski/Work/fxhnt/infra/k8s/
+done
+cp infra/k8s/namespace.yaml /home/jgrusewski/Work/fxhnt/infra/k8s/
+mkdir -p /home/jgrusewski/Work/fxhnt/infra/scripts && cp infra/scripts/generate-minio-tls.sh /home/jgrusewski/Work/fxhnt/infra/scripts/
+echo "=== fxhnt infra/k8s now ==="; ls /home/jgrusewski/Work/fxhnt/infra/k8s/
+```
+Expected: fxhnt/infra/k8s/ now contains the platform dirs (merged with fxhnt's existing
+jobs/network-policies/services/storage/orchestration) + namespace.yaml. `gpu-taint.yaml` NOT copied.
+
+- [ ] **Step 2: Sanity — no remaining hardcoded foxhunt-repo paths in moved manifests**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+grep -rIn "Work/foxhunt\|foxhunt/infra\|root/foxhunt" infra/k8s/ infra/live/ infra/modules/ 2>/dev/null | grep -v "foxhunt.svc.cluster.local\|namespace: foxhunt\|name: foxhunt\|cluster_name\|foxhunt-tfstate\|foxhunt-gitlab\|foxhunt-training\|foxhunt-backups\|foxhunt-sccache\|foxhunt-binaries\|foxhunt-agent" | head
+echo "(above should be empty — 'foxhunt' as the k8s namespace / SCW resource names is expected + correct)"
+```
+Expected: empty (the namespace `foxhunt` + Scaleway resource names legitimately stay).
+
+---
+
+## Task 4: Verify cluster unaffected + cockpit still deploys from fxhnt
+
+- [ ] **Step 1: terragrunt still clean from fxhnt + platform health**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+curl -sk -m 10 -o /dev/null -w "dashboard.fxhnt.ai: %{http_code}\n" https://dashboard.fxhnt.ai
+curl -sk -m 10 https://git.fxhnt.ai/api/healthz 2>/dev/null | grep -o '"status": "pass"'
+```
+Expected: dashboard 200; gitea pass. (No cluster change was made — this confirms nothing regressed.)
+
+---
+
+## Task 5: Commit fxhnt (additions) + push
+
+- [ ] **Step 1**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git add infra/ .gitignore
+git commit -m "feat(infra): adopt platform IaC from foxhunt — terraform live+modules + k8s manifests (Phase 2D)
+
+fxhnt is now the single home for platform IaC. Terraform state keys unchanged
+(live/production structure preserved) — terragrunt plan = No changes, no migration.
+Includes k8s platform manifests (cert-manager, databases, dns, gitea, minio, monitoring,
+network-policies, tailscale, stalwart, mattermost, services, storage, jobs, training,
+secrets, argo, gitlab proxy) + generate-minio-tls.sh.
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin chore/relocate-iac 2>&1 | tail -2
+```
+
+---
+
+## Task 6: Remove the moved paths from foxhunt + commit + push
+
+- [ ] **Step 1: git rm the relocated IaC from foxhunt (keep gpu-taint.yaml, docker, Rust scripts)**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git rm -r -q infra/live infra/modules
+for d in argo cert-manager databases dns gitea gitlab jobs mattermost minio monitoring network-policies secrets services stalwart tailscale training storage; do git rm -r -q "infra/k8s/$d"; done
+git rm -q infra/k8s/namespace.yaml infra/scripts/generate-minio-tls.sh
+echo "=== foxhunt infra/ remnant (Rust-only) ==="; find infra -maxdepth 2 | sort | head -30
+```
+Expected: `infra/k8s/gpu-taint.yaml` + `infra/docker/*` + `infra/scripts/{build-and-push,harden,smoke-test,train,...}` remain; live/modules + platform k8s gone.
+
+- [ ] **Step 2: Commit + push foxhunt**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git commit -q -m "chore(infra): relocate platform IaC to fxhnt (Phase 2D)
+
+Platform terraform (live+modules) + k8s manifests moved to the fxhnt repo (now the IaC
+home). Rust-only remnants kept here (docker build images, gpu-taint, Rust scripts).
+foxhunt is being archived; history retained. See fxhnt for live platform IaC."
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin chore/relocate-iac-remove 2>&1 | tail -2
+```
+
+---
+
+## Task 7: Merge both branches, then archive foxhunt
+
+- [ ] **Step 1: Merge fxhnt branch → master + push**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+git checkout master && git merge --no-ff chore/relocate-iac -m "Merge: adopt platform IaC from foxhunt (Phase 2D)" 2>&1 | tail -2
+git branch -d chore/relocate-iac
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin master 2>&1 | tail -1
+```
+
+- [ ] **Step 2: Merge foxhunt branch → main + push**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git checkout main && git merge --no-ff chore/relocate-iac-remove -m "Merge: relocate platform IaC to fxhnt (Phase 2D)" 2>&1 | tail -2
+git branch -d chore/relocate-iac-remove
+GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin main 2>&1 | tail -1
+```
+
+- [ ] **Step 3: Final re-verify terragrunt from fxhnt (post-merge) + archive foxhunt in Gitea**
+```bash
+cd /home/jgrusewski/Work/fxhnt
+( cd infra/live/production/kapsule && terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:" | head -1 )
+GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
+curl -sk -u "$GA:$GP" -X PATCH "https://git.fxhnt.ai/api/v1/repos/$GA/foxhunt" -H 'Content-Type: application/json' -d '{"archived":true}' -o /dev/null -w "archive foxhunt: %{http_code}\n"
+```
+Expected: kapsule `No changes`; `archive foxhunt: 200`.
+
+---
+
+## Rollback
+- Before Task 6 (foxhunt rm): just delete `fxhnt/infra/{live,modules}` + the copied k8s dirs; nothing lost
+  (foxhunt still authoritative).
+- After: both repos retain history; un-archive foxhunt + `git revert` if needed. TF state never moved.
+
+## Acceptance criteria (from spec)
+- `fxhnt/infra/{live,modules}` present; `terragrunt plan = No changes` all 4 modules from fxhnt. ✅ T2, T7
+- Platform k8s manifests + generate-minio-tls.sh in fxhnt; no collisions lost. ✅ T1, T3
+- Moved paths `git rm`'d from foxhunt; both repos committed + pushed. ✅ T5, T6, T7
+- dashboard 200 + cockpit deploy unaffected. ✅ T4
+- foxhunt archived in Gitea; Rust code + history intact. ✅ T7

docs/superpowers/plans/2026-06-21-tfstate-to-scaleway.md

@@ -0,0 +1,264 @@
+# Migrate Terraform State to Scaleway Object Storage — Implementation Plan (Phase 2A)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
+> **Do NOT run subagent-driven** — state migration is irreversible-adjacent and each module needs a
+> human-confirmed `plan = No changes` gate. Steps use checkbox (`- [ ]`) syntax.
+
+**Goal:** Move the 4 production modules' Terraform state from the self-hosted GitLab http backend to
+external Scaleway Object Storage (bucket `foxhunt-tfstate`), with native S3 lockfile locking.
+
+**Architecture:** Back up all 4 states from GitLab first; flip the single `remote_state` block in
+`infra/live/production/root.hcl` from `http`→`s3`; migrate each module in-place with `terragrunt init
+-migrate-state`; gate each on `terragrunt plan = No changes`. GitLab state is left intact as a fallback
+(deleted later in 2C).
+
+**Tech Stack:** terragrunt v0.77.20 + OpenTofu v1.11.5, Scaleway Object Storage (S3-compatible), AWS s3
+backend.
+
+**Spec:** `docs/superpowers/specs/2026-06-21-tfstate-to-scaleway-design.md`
+
+---
+
+## Shared env (export in EVERY shell that runs terragrunt during this plan)
+The s3 backend reads **`AWS_*`** env vars; the Scaleway provider reads **`SCW_*`**; reading the OLD
+GitLab state during `-migrate-state` needs **`TF_HTTP_*`**. Export all three sets:
+```bash
+cd /home/jgrusewski/Work/foxhunt
+export TG_TF_PATH=tofu TERRAGRUNT_TFPATH=tofu
+# Scaleway provider creds
+export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
+export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
+export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
+export SCW_DEFAULT_REGION=fr-par SCW_DEFAULT_ZONE=fr-par-2
+# s3 backend creds (SAME Scaleway keys, AWS-style names)
+export AWS_ACCESS_KEY_ID="$SCW_ACCESS_KEY"
+export AWS_SECRET_ACCESS_KEY="$SCW_SECRET_KEY"
+export AWS_REGION=fr-par
+# OLD GitLab http backend creds (needed to READ existing state during migrate)
+export TF_HTTP_USERNAME=root
+export TF_HTTP_PASSWORD=$(kubectl get secret gitlab-pat -n foxhunt -o jsonpath='{.data.token}'|base64 -d)
+echo "env: SCW=${SCW_ACCESS_KEY:0:6}… AWS=${AWS_ACCESS_KEY_ID:0:6}… PAT=${TF_HTTP_PASSWORD:0:4}…"
+```
+
+---
+
+## Task 0: Branch
+- [ ] **Step 1**
+```bash
+cd /home/jgrusewski/Work/foxhunt && git checkout -b chore/tfstate-to-scaleway
+```
+
+---
+
+## Task 1: Verify prerequisites + back up all 4 GitLab states
+
+**Files:** none (read-only + /tmp backups)
+
+- [ ] **Step 1: Confirm the target bucket exists + is empty-ish**
+```bash
+scw object bucket list 2>/dev/null | grep foxhunt-tfstate || echo "MISSING — create: scw object bucket create name=foxhunt-tfstate region=fr-par"
+```
+Expected: a line containing `foxhunt-tfstate`. If MISSING, run the create shown, then re-check.
+
+- [ ] **Step 2: Confirm tooling versions**
+```bash
+tofu version | head -1; terragrunt --version | head -1
+```
+Expected: OpenTofu `v1.11.x` (≥1.10 for `use_lockfile`), terragrunt `v0.77.x`.
+
+- [ ] **Step 3: Back up each module's CURRENT (GitLab) state** — run with shared env exported
+```bash
+for m in kapsule public-gateway dns block-storage; do
+  ( cd infra/live/production/$m && terragrunt state pull > /tmp/tfstate-backup-$m.tfstate 2>/dev/null )
+  echo "$m: $(python3 -c "import json;d=json.load(open('/tmp/tfstate-backup-$m.tfstate'));print('serial',d.get('serial'),'resources',len(d.get('resources',[])))" 2>/dev/null || echo 'EMPTY/ERROR')"
+done
+```
+Expected: each module prints a non-zero `resources` count (e.g. kapsule ~5+, dns/public-gateway/block-storage ≥1). **STOP if any backup is EMPTY/ERROR** — that module's GitLab state isn't readable; do not migrate it blind.
+
+- [ ] **Step 4: Record current state serials for post-migration comparison**
+```bash
+grep -H '"serial"' /tmp/tfstate-backup-*.tfstate 2>/dev/null || for m in kapsule public-gateway dns block-storage; do echo "$m $(python3 -c "import json;print(json.load(open('/tmp/tfstate-backup-$m.tfstate'))['serial'])")"; done
+```
+Expected: a serial number per module. Keep this output — the migrated state should have the same resource set.
+
+---
+
+## Task 2: Switch the backend in root.hcl (http → s3)
+
+**Files:** Modify `infra/live/production/root.hcl`
+
+- [ ] **Step 1: Replace the `remote_state` block**
+
+In `infra/live/production/root.hcl`, replace the entire existing `remote_state { ... }` block (the `backend = "http"`
+GitLab block) with:
+```hcl
+# Remote state in Scaleway Object Storage (bucket foxhunt-tfstate, fr-par) — migrated off GitLab 2026-06-21.
+# s3 backend reads AWS_* env vars (set to the Scaleway access/secret keys); SCW_* drives the provider.
+remote_state {
+  backend = "s3"
+  generate = {
+    path      = "backend.tf"
+    if_exists = "overwrite"
+  }
+  config = {
+    bucket = "foxhunt-tfstate"
+    key    = "${path_relative_to_include()}/terraform.tfstate"
+    region = "fr-par"
+
+    endpoints = {
+      s3 = "https://s3.fr-par.scw.cloud"
+    }
+
+    # Scaleway S3-compat: skip AWS-specific preflight calls
+    skip_credentials_validation = true
+    skip_region_validation      = true
+    skip_requesting_account_id  = true
+    skip_metadata_api_check     = true
+
+    # OpenTofu-native lock (no DynamoDB)
+    use_lockfile = true
+  }
+}
+```
+Leave the `generate "provider"` block and `inputs`/`locals` untouched.
+
+- [ ] **Step 2: Sanity-check the file parses**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+grep -A2 'backend = "s3"' infra/live/production/root.hcl && grep -c 'backend = "http"' infra/live/production/root.hcl
+```
+Expected: shows the `s3` backend line; the `backend = "http"` count is `0`.
+
+---
+
+## Task 3: Migrate `kapsule` (canary) + lock smoke test
+
+**Files:** none (state operation)
+
+- [ ] **Step 1: Migrate state (with shared env exported)**
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/kapsule
+terragrunt init -migrate-state -force-copy -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "Successfully configured|migrat|Error|s3" | head
+```
+Expected: a line like `Terraform has been successfully migrated to the "s3" backend!` (or `Successfully
+configured the backend "s3"`). **If it errors on checksum** (Scaleway quirk), add `skip_s3_checksum = true`
+to the `config` block in `infra/live/production/root.hcl` and re-run this step.
+
+- [ ] **Step 2: GATE — plan must show No changes**
+```bash
+terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head
+```
+Expected: `No changes. Your infrastructure matches the configuration.`
+**STOP if it shows any add/change/destroy** — the migrated state doesn't match reality. Revert (Task 7
+rollback) and investigate before continuing.
+
+- [ ] **Step 3: Confirm the state object landed in the bucket**
+```bash
+scw object bucket list 2>/dev/null >/dev/null  # ensure scw configured
+aws --endpoint-url https://s3.fr-par.scw.cloud s3 ls s3://foxhunt-tfstate/kapsule/ 2>/dev/null || \
+  echo "(aws cli n/a — verify via: mc ls on a scw alias, or scw object)"
+```
+Expected: `terraform.tfstate` listed under `kapsule/`. (If `aws` CLI is absent, this is best-effort;
+the `plan = No changes` in Step 2 already proves the s3 backend is the live source.)
+
+- [ ] **Step 4: Lock smoke test** — confirm `use_lockfile` works on Scaleway
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/kapsule
+# A refresh acquires + releases the lock; success + no leftover .tflock proves locking works
+terragrunt apply -refresh-only -auto-approve -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "Apply complete|Error|lock" | head
+aws --endpoint-url https://s3.fr-par.scw.cloud s3 ls s3://foxhunt-tfstate/kapsule/ 2>/dev/null | grep -i tflock && echo "WARN: orphan lockfile" || echo "OK: no orphan lockfile"
+```
+Expected: `Apply complete!` and `OK: no orphan lockfile`. **If locking errors**, fall back to
+`use_lockfile = false` (single-maintainer is safe) — document the change in the commit.
+
+---
+
+## Task 4: Migrate the remaining 3 modules (`public-gateway`, `dns`, `block-storage`)
+
+**Files:** none (state operations). Same procedure as Task 3 Steps 1-2, per module, each gated.
+
+- [ ] **Step 1: Migrate + gate `public-gateway`**
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/public-gateway
+terragrunt init -migrate-state -force-copy -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "migrat|successfully|Error" | head
+terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head
+```
+Expected: migration success line, then `No changes`. **STOP on any drift.**
+
+- [ ] **Step 2: Migrate + gate `dns`**
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/dns
+terragrunt init -migrate-state -force-copy -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "migrat|successfully|Error" | head
+terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head
+```
+Expected: migration success line, then `No changes`. **STOP on any drift.**
+
+- [ ] **Step 3: Migrate + gate `block-storage`**
+```bash
+cd /home/jgrusewski/Work/foxhunt/infra/live/production/block-storage
+terragrunt init -migrate-state -force-copy -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "migrat|successfully|Error" | head
+terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head
+```
+Expected: migration success line, then `No changes`. **STOP on any drift.**
+
+---
+
+## Task 5: Final verification + commit
+
+**Files:** Modify (commit) `infra/live/production/root.hcl`; also commit generated `backend.tf` only if not gitignored.
+
+- [ ] **Step 1: Re-plan all 4 modules from a clean shell** (proves the s3 backend is authoritative)
+```bash
+cd /home/jgrusewski/Work/foxhunt
+for m in kapsule public-gateway dns block-storage; do
+  echo "=== $m ==="
+  ( cd infra/live/production/$m && terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head -1 )
+done
+```
+Expected: every module prints `No changes.`
+
+- [ ] **Step 2: Confirm all 4 state objects exist in the bucket**
+```bash
+for m in kapsule public-gateway dns block-storage; do
+  aws --endpoint-url https://s3.fr-par.scw.cloud s3 ls s3://foxhunt-tfstate/$m/ 2>/dev/null | grep -q terraform.tfstate && echo "$m: state present" || echo "$m: NOT FOUND (verify manually)"
+done
+```
+Expected: `state present` for all 4 (or manual confirmation if `aws` CLI absent — `plan = No changes` is the real proof).
+
+- [ ] **Step 3: Verify GitLab state still intact (fallback preserved)** — should NOT be deleted in 2A
+```bash
+echo "GitLab TF state is intentionally left in place as a 2A fallback; deleted in Phase 2C."
+```
+
+- [ ] **Step 4: Commit**
+```bash
+cd /home/jgrusewski/Work/foxhunt
+git add infra/live/production/root.hcl
+# backend.tf is terragrunt-generated; add only if tracked (usually gitignored)
+git status --porcelain infra/live/production/*/backend.tf 2>/dev/null
+git commit -m "chore(infra): migrate Terraform state GitLab -> Scaleway Object Storage (Phase 2A)
+
+All 4 modules (kapsule/public-gateway/dns/block-storage) migrated to s3 backend
+(bucket foxhunt-tfstate, fr-par, use_lockfile). plan=No changes verified per module.
+GitLab http state left intact as fallback (removed in 2C).
+
+Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
+```
+
+---
+
+## Rollback (if any module's post-migration plan shows drift)
+1. Restore the `http` backend block in `infra/live/production/root.hcl` (git checkout the file).
+2. Re-attach GitLab state: in each affected module, `terragrunt init -migrate-state -force-copy` (copies
+   back s3→http) OR `terragrunt init -reconfigure` (the GitLab state was never deleted).
+3. If state is corrupted, restore from `/tmp/tfstate-backup-<module>.tfstate` via `terragrunt state push`.
+4. The GitLab copy is untouched throughout 2A, so reverting is always possible.
+
+---
+
+## Acceptance criteria (from spec)
+- `infra/live/production/root.hcl` uses the Scaleway s3 backend; committed. ✅ Task 2 + Task 5.4
+- All 4 modules: `terragrunt plan = No changes` on the new backend. ✅ Task 3.2, Task 4, Task 5.1
+- State objects at `foxhunt-tfstate/<module>/terraform.tfstate`. ✅ Task 3.3, Task 5.2
+- Lock smoke test passes. ✅ Task 3.4
+- GitLab http state intact (fallback). ✅ Task 5.3

docs/superpowers/specs/2026-06-05-surfer-uncorrelated-universe-design.md

@@ -0,0 +1,144 @@
+# The Surfer over an Uncorrelated Universe — Design Spec
+
+> **For agentic workers:** This spec defines WHY and WHAT for foxhunt's strategic pivot from
+> single-instrument seconds-horizon order-book RL to a diversified, days-to-weeks, multi-asset
+> ML trend system. It does NOT implement; an implementation plan follows (Phase 0 first).
+
+**Status:** Draft 1 (2026-06-05)
+**Supersedes (direction):** the seconds-horizon ES order-book RL thesis (measured structurally unprofitable).
+**Branch target:** new branch `surfer-universe` off the current branch (keeps the session's validation harnesses).
+**Capital stance:** $35k is a PILOT to prove the pipeline + edge (paper / tiny live); real book targets ~$100k.
+
+**Linked pearls (load-bearing):**
+- `pearl_surfer_thesis_is_uncorrelated_trend_recognition` — the thesis
+- `pearl_surfer_universe_build_synthesis` — the 8-agent research synthesis behind every choice here
+- `pearl_ofi_edge_uncapturable_by_crossing` / `pearl_passive_mm_knife_edge_ofi_conditioning_promising` — why seconds-horizon ES is dead
+- `pearl_lowfreq_no_robust_edge_on_2y_es` — why breadth/universe is the bottleneck; validation discipline
+- `pearl_omnisearch_synthesis_signal_first_and_mtm_reward` — dense-MtM → portfolio Differential Sharpe reward
+
+---
+
+## §1. Why this, why now
+
+Measured this session (zero cluster runs, ~5 cheap harnesses): on ES, seconds-horizon crossing edge is ~100× smaller
+than the spread; passive MM is adversely-selected negative (winner's curse); intraday has no IC; daily/cross-asset
+edges sign-flip IS↔OOS (overfit). **The bottleneck was the configuration — single instrument × seconds-horizon ×
+order-book microstructure — not the model.** That is the worst possible config for a non-colocated ML participant:
+most-efficient market, most speed-dependent horizon, zero diversification.
+
+The fix is to invert all three: **many uncorrelated instruments × days-to-weeks horizon × trend/cross-sectional
+recognition.** This is also foxhunt's actual "surfer" thesis ("no wave is the same") finally pointed at an ocean
+with enough waves to ride. Breadth = opportunity sourcing: across many uncorrelated markets, some subset always
+has a rideable trend, and uncorrelated rides don't sink together.
+
+## §2. The thesis
+
+A fixed trend rule applies identical logic to every wave. The ML "surfer" reads each unique wave — is it a real
+trend or chop? building or exhausting? what's the sea-state (regime)? — and adjusts. That recognition is the
+legitimate ML edge ON TOP of the trend risk premium. **Breadth gives the floor; ML earns its keep above it.**
+
+## §3. Architecture
+
+**Core design principle (the eval-collapse cure):** the ML surfer outputs **bounded DEVIATIONS from a deterministic
+diversified-trend floor**: `w_i = floor_i + clamp(ML_delta_i, ±δ)`. The model can only add value above a proven
+baseline and degrades gracefully to the floor when it has no edge. It structurally cannot run off a cliff.
+
+### §3.1 The floor (deterministic)
+Diversified time-series momentum: `signal_i = mean over h∈{1,3,12}mo of sign(return_{i,h})` (12mo skips last ~21d);
+inverse-vol size with 63-day EWMA vol; portfolio vol-target 10% annual (rescale weekly on realized portfolio vol);
+weekly rebalance + ≥10%-change no-trade band. Realistic net Sharpe 0.4–0.5 (small universe), DD 15–25%.
+
+### §3.2 The ML surfer (bounded deltas), in confidence order
+1. **Regime / change-point overlay (build FIRST)** — recognizes trend-vs-chop, throttles gross, cuts drawdowns.
+   ML's most defensible edge (DD reduction, NOT return prediction). Lowest overfit surface.
+2. **Cross-sectional tilt (panel)** — ONE shared encoder over all instruments (N×T samples) + per-instrument
+   embedding → continuous dollar-neutral tilt on the floor; gated on rank-IC / ICIR. NOT discrete ranks (a <20-name
+   universe is statistically starved; breadth is recovered through time + pooling).
+3. **Direct return prediction → DEFERRED** (the 64-commit trap; cost-cliff at 2–3 bps = turnover noise).
+
+### §3.3 Reward / objective
+Portfolio **Differential Sharpe Ratio** (Moody-Saffell) + drawdown penalty + turnover/cost penalty — the
+portfolio-level, risk-adjusted generalization of the dense per-step mark-to-market reward (Φ=unrealized-PnL).
+
+### §3.4 Risk / sizing
+Vol-target 10%; inverse-vol within asset class + ERC across classes; fractional Kelly (half/quarter, reuse the
+existing Kelly controller clamped [0,0.5]); drawdown throttle θ; reuse the CMDP overlay as the tail kill-switch.
+Discreteness is binding at micro size: target <0.5 contract → hold 0, carry the residual; $35k sustains ~3–5
+simultaneous 1-lot positions.
+
+### §3.5 Reuse of existing foxhunt assets + architecture discipline
+CfC/Mamba2 encoder (shared across instruments + instrument embedding + cross-sectional attention head); RL stack;
+Kelly + CMDP controllers; the determinism foundation; ml-alpha's CUDA build pipeline (build.rs cubins, mapped-pinned
+buffers, cudarc launch). **All compute is CUDA-only, CPU read-only** (`feedback_cpu_is_read_only`,
+`pearl_cold_path_no_exception_to_gpu_drives`, `feedback_no_cpu_test_fallbacks`): every formula — signals, vol,
+sizing, backtest returns, and the validation statistics (CPCV per-path Sharpe, PBO, Deflated Sharpe) — is a CUDA
+kernel; CPU only reads final gate scalars from mapped-pinned buffers and only enumerates split index-sets as control
+flow. No CPU compute, no cold-path exception, no CPU test oracle. The session's Python `scripts/measure_*.py` were
+one-off edge AUDITS, not part of the built system; the system is Rust+CUDA. The new surfer lives in `crates/ml-alpha`
+(reuses its build.rs/cuda/mapped-pinned/determinism infra and, in Phases 1-3, its encoder + RL).
+
+## §4. Universe + data pipeline
+
+**Universe (~8 micros, maximal asset-class diversification):** MES, M2K (equity); 10Y micro-yield (rates — note
+YIELD-based = sign inversion vs price); M6E, M6A (FX); MGC (metals); MCL (energy); MBT (crypto).
+**Data:** Databento `GLBX.MDP3`, `ohlcv-1d` (+ `ohlcv-1h`). **Continuous contracts built in-house:** volume-roll
+(validate vs OI) + **backward RATIO adjustment for the encoder's returns**; keep an **UNADJUSTED** series for
+actual fills + USD reward. NEVER Panama-adjust for a trend learner; NEVER compute reward on adjusted prices.
+
+## §5. Validation gates (the guardrail — non-negotiable)
+
+| Gate | Metric | Pass |
+|---|---|---|
+| SV-G0 | Leakage audit: purge+embargo covers full hold + feature lookback | verified |
+| SV-G1 | CPCV (≥50 paths): 5th-percentile OOS pnl after costs | > 0 |
+| SV-G2 | PBO (CSCV) | < 0.20 |
+| SV-G3 | Deflated Sharpe, deflated by N = ALL trials ever (64-commit + session + this build) | > 0.95 |
+| SV-G4 | IS↔OOS config rank-consistency (Spearman) — the sign-flip detector | > 0 |
+| SV-G5 | Edge survives 2× cost + roll cost | net OOS pnl > 0 |
+| SV-G6 | Capacity/turnover: cost × turnover < fixed fraction of gross | pass |
+
+Every config / seed / hyperparameter is a logged trial feeding N. The 64 failed commits prove N is large.
+
+## §6. Phases (signal-first, cheapest-first, zero cluster until validated)
+
+- **Phase 0 — Floor + validation harness (CUDA kernels + Rust orchestration, CPU read-only, no ML, no cluster — per foxhunt GPU-drives discipline; `feedback_cpu_is_read_only`, `pearl_cold_path_no_exception_to_gpu_drives`). Runs on the local GPU. FULLY SPECIFIED in the plan.**
+  Build the deterministic floor on real Databento micro data; build the CPCV/PBO/DSR harness; measure whether the
+  floor itself clears the gates OOS after costs. **STOP-if:** floor fails SV-G1/G3/G5 → the universe has no
+  capturable trend premium at this scale; reassess (more instruments / more history) before any ML.
+- **Phase 1 — Regime overlay.** Must beat the floor OOS (DD reduction first). Gates SV-G1..G6 vs floor.
+- **Phase 2 — Cross-sectional panel tilt** (shared encoder). Must beat floor+regime OOS via rank-IC/ICIR.
+- **Phase 3 — Full RL surfer** (floor + bounded delta, DSR reward). Cluster only here, gated by all above.
+
+## §7. Scope boundary
+
+**In (this spec):** the floor, the bounded-delta architecture, the 3 ML layers (regime → cross-sectional → RL),
+the universe + continuous-contract pipeline, the validation gates, the phased build. **Phase 0 is the immediate
+deliverable.**
+**Out:** seconds-horizon / order-book microstructure (measured dead); direct return-prediction sequence models
+(deferred until floor+regime+cross-sectional prove out); live-capital deployment beyond a pilot (until ~$100k and
+gates pass); broker/execution automation (separate spec once a validated strategy exists).
+
+## §8. Risks
+
+1. **Capital ($35k below viable minimum)** — under-diversified → trend's drawdowns without its smoothing; micro-cost
+   drag eats Sharpe. MITIGATION: treat as a pilot to prove pipeline/edge; scale to ~$100k for the real book.
+2. **ML adds nothing over the floor** — entirely possible; the bounded-delta design makes that a graceful
+   no-op (you keep the floor), and the gates catch it before any spend.
+3. **Floor itself has no edge at this scale/universe** — Phase 0 STOP catches this cheaply.
+4. **Re-overfitting (commit #65)** — the SV gates + deflation-by-all-trials are the explicit defense; honor them.
+5. **Roll / continuous-contract artifacts** — ratio-adjust + volume-roll + reward-on-unadjusted; validated in Phase 0.
+
+## §9. Decision log
+
+```
+Decision: Pivot foxhunt to "the surfer over an uncorrelated universe" — diversified days-to-weeks ML trend across
+~8 uncorrelated micro futures, ML as bounded deviations from a deterministic trend floor, gated by CPCV/PBO/Deflated-
+Sharpe. Phase 0 (build+validate the floor, CPU-only) is the immediate, decisive, near-free test.
+Rejected: continuing seconds-horizon ES order-book RL (measured structurally unprofitable for a non-colocated ML
+participant — crossing edge 100× < spread; passive MM adversely-selected; daily/cross-asset edges sign-flip).
+Capital: $35k = pilot to prove the approach; real book ~$100k (Carver retail floor; under-diversification = main risk).
+ML scope: regime overlay first (DD reduction = ML's defensible edge); cross-sectional tilt second; direct return
+prediction deferred (the 64-commit trap).
+Date: 2026-06-05
+Falsification (Phase 0): floor fails SV-G1/G3/G5 OOS → no capturable trend premium at this scale; reassess before ML.
+```

docs/superpowers/specs/2026-06-07-adaptive-multistrat-book-deployable-spec.md

@@ -0,0 +1,226 @@
+# Adaptive Multi-Strat Book — Deployable Spec
+
+**Date:** 2026-06-07
+**Status:** Validated design, pre-deployment (paper-forward running)
+**Branch:** surfer-universe
+**Engine:** `scripts/surfer/multistrat_book_v3.py` (validated), `sixtyforty_paper.py` (baseline, live)
+
+---
+
+## 1. Executive summary
+
+A **diversified, edge-decay-adaptive, adaptively-risk-managed, unlevered** multi-asset book — the
+honest endpoint of an exhaustive search. It is *not* alpha; it is **premium harvesting with superior
+risk management** (the foxhunt engine's genuine, validated strength). Realistic **~0.5–0.7 Sharpe,
+~−7% max drawdown**, ~$2–3k/yr on $35k, scales linearly with capital.
+
+**Why this and not the alternatives** (all tested to OOS exhaustion, all in memory):
+- Single predictive edges (equities, futures, ML, AI4Finance, PEAD): no OOS alpha for non-colocated retail.
+- Crypto funding (single-venue): hedgeability-blocked. Cross-venue: marginal/breakeven OOS (the +14.9 Sharpe was a max-min artifact; honest fixed-pair = ~breakeven).
+- Leverage: **doesn't pay at retail financing** (6–7% margin vs prime-brokerage SOFR+1–2%) — adaptive 1x Sharpe +0.48 vs 2x +0.14. The hedge-fund moat is *cheap financing*, not the strategy.
+
+**What we keep** (validated, measurable): the **edge-decay-adaptive allocation** (+0.11 Sharpe, lower
+DD vs static risk-parity) and the **adaptive risk layer** (beat the static one decisively; halves
+drawdown). The engine's value is *risk management + adaptive allocation*, and that is real.
+
+---
+
+## 2. Strategy overview
+
+Hold a small set of **uncorrelated return streams**, each vol-normalized to equal risk, weighted by
+a **per-stream edge-health trust** (down-weight decaying streams, resurrect recovered ones), combined
+and run through an **adaptive risk-management layer**, **unlevered (~1x)**, rebalanced on a schedule.
+
+Return source = the structural premia (equity, term, gold/inflation, commodity, trend/crisis-alpha,
+a small crypto sleeve). The *alpha* is the disciplined, adaptive risk allocation — not prediction.
+
+---
+
+## 3. Instruments (deployable at $35k via fractional-share ETFs + small crypto)
+
+| Stream | ETF | Role |
+|---|---|---|
+| Equity | **SPY** (or VTI) | equity risk premium |
+| Bonds | **IEF** (7–10y Treasuries) | term premium, equity diversifier |
+| Gold | **GLD** (or IAU) | inflation/crisis diversifier |
+| Commodity | **PDBC** (or DBC) | inflation/commodity premium |
+| Trend / managed futures | **DBMF** (or KMLM) | crisis-alpha, uncorrelated (use the *real* CTA ETF — our DIY trend was ~0 Sharpe) |
+| Crypto (small) | **BTC spot** (or IBIT) | high-return uncorrelated sleeve; trust-layer auto-caps it (currently down-weighted in deleverage) |
+
+All ETFs are marginable, fractional-shareable, ~zero commission. **Unlevered** (cash account is fine;
+no margin needed). Crypto held separately (spot/IBIT), sized small.
+
+---
+
+## 4. Allocation engine — edge-decay-adaptive trust (foxhunt idea, validated)
+
+For each stream `i`, maintain a **trust** `θᵢ ∈ [0.1, 1]`:
+- Compute trailing-126d risk-adjusted return (Sharpe) of the stream.
+- Map: Sharpe ≥ +0.5 → θ=1.0; ≤ −0.5 → θ=0.1 (floored — **resurrection-capable**); linear between.
+- EMA-smooth (α≈0.06) so it adapts gradually, not jumpily.
+
+Weight each (vol-normalized) stream by `θᵢ`, normalize to sum 1. This **down-weights decaying premia
+and re-weights recovered ones** automatically — currently: equity 0.73, commod 1.00, trend 0.79
+(healthy) vs bond 0.11, crypto 0.10 (correctly de-emphasized). Validated: +0.11 Sharpe, lower DD vs
+static equal-risk. Source: `pearl_edge_decay_detection_is_a_missing_abstraction_layer`,
+`pearl_dead_signal_resurrection_discipline`.
+
+---
+
+## 5. Risk-management layer — adaptive controllers (the foxhunt port; the load-bearing piece)
+
+Static thresholds **crush returns** (one-way latch, `pearl_cmdp_consec_loss_counter_is_one_way_latch`).
+The adaptive layer (validated: beat static +0.03→+0.14 Sharpe, maxDD −18.7%→−14.5%). Daily leverage
+`L = clip( min(L_vol, L_kelly) × dd_mult × corr_mult , LEV_FLOOR, MAXLEV )`, applied to the combination:
+
+1. **EMA online vol** (α≈0.03) → `L_vol = TARGET_VOL / realized_vol` (responsive, no window edges).
+2. **Kelly with floor + bootstrap** → `L_kelly = clip(EMA_mean·252 / EMA_var·252, KELLY_FLOOR=0.5, MAXLEV)` — adapts to edge strength, **never dies** (`pearl_bootstrap_must_respect_clamp_range`).
+3. **Drawdown de-lever, CONTINUOUS + self-recovering** → `dd_mult = clip(1 − 3·max(0, −dd − 0.05), 0.40, 1.0)` — scales down as drawdown deepens, **recovers immediately as it heals** (NOT a latch — this is the fix).
+4. **Correlation de-risk, z-scored** → when avg cross-stream correlation is unusually high vs its own trailing distribution (diversification breaking in a crisis), reduce: `corr_mult = clip(1 − 0.2·max(0, z), 0.5, 1.0)`.
+5. **Leverage floor** `LEV_FLOOR=0.3`, **MAXLEV=1.0** (unlevered — see §8).
+
+Defaults: `TARGET_VOL=10%`. The whole layer is the engine's true job — *survive and adapt*, not maximize.
+
+---
+
+## 6. Rebalance & execution
+
+- **Weekly rebalance** (fixed-phase): recompute vol-normalization, trust weights, and `L`; trade to target.
+- **Hysteresis / no-churn:** only trade a stream if its target weight moved > ~2% (cut turnover/cost).
+- **Cost budget:** ETFs ~0 commission; the only friction is spread/slippage (minimal on SPY/IEF/GLD/DBMF). Crypto sleeve: small, spot.
+- **Cash account, unlevered** — no financing cost, no liquidation risk.
+
+---
+
+## 7. Expected performance (honest)
+
+**Validated** on the real ETFs (SPY/IEF/GLD/PDBC/DBMF + BTC) over 2019-05..2026-06, exact live
+pipeline (`multistrat_etf_backtest.py`), multiple regimes (COVID, 2022 bear, bulls):
+
+| Metric | Backtest 2019–26 | Realistic forward |
+|---|---|---|
+| Sharpe (unlevered) | **+1.20** | ~0.8–1.0 (haircut: favorable period + low-vol-flatter) |
+| Annual return | +6.1% | ~5–7% |
+| Realized vol | 5.1% | ~5–8% |
+| Max drawdown | **−5.4%** | ~−8 to −12% |
+| Per-year | **positive every year** incl 2022 (+0.1) | — |
+| vs 60/40 (SPY/IEF) | +0.87 Sharpe, −21% DD | book wins on Sharpe AND drawdown |
+| vs equity buy-hold | +0.85 Sharpe, −34% DD | — |
+| On $35k | ~$2.1k/yr, very smooth (−5% DD) | — |
+| On $500k | ~$30k/yr (Sharpe scale-invariant — capital is the lever) | — |
+
+The robust signals are the **−5.4% max drawdown and every-year-positive** (not just the low-vol
+Sharpe). The book genuinely beats 60/40 on risk-adjusted terms with ~1/4 the drawdown.
+
+Honest caveats: it's **long beta** (falls in everything-down 2022-style, though trend + adaptive
+de-lever cushion it); returns are *modest* in dollars at small capital (the lever is capital, not
+Sharpe); crypto sleeve validated on one window — keep it small and let the trust layer cap it.
+
+---
+
+## 8. Why unlevered (the key finding)
+
+Leverage scales return but **at retail financing (6–7%) it lowers risk-adjusted return**: adaptive 1x
+Sharpe +0.48 vs 2x +0.14 — the financing drag eats the leverage benefit, while drawdown grows. Hedge
+funds lever ~0.7-Sharpe books profitably **only because prime brokerage finances at ~SOFR+1–2%.** That
+financing access is the institutional moat, and it's structural — not replicable at retail. **So: run
+it 1x.** Re-evaluate leverage only if you ever access institutional-rate financing.
+
+---
+
+## 9. Validation status
+
+**Validated:** edge-decay-adaptive allocation (>static), adaptive risk layer (>static, halves DD),
+combination Sharpe ~0.5–0.72 across windows, the unlevered-is-best financing finding. Scripts:
+`multistrat.py`, `multistrat_book.py`, `multistrat_book_v2.py`, `multistrat_book_v3.py`.
+**Live baseline:** `sixtyforty_paper.py` paper-forward running (the 60/40 core).
+**Not yet:** live execution of the full 6-stream adaptive book (paper-forward harness for it = next),
+real ETF fills/spreads (minor), the crypto sleeve operationally.
+
+---
+
+## 10. Phased rollout
+
+1. **Paper-forward (4–8 wk):** extend the live harness to the full 6-stream adaptive book (ETF closes
+   via Yahoo + BTC), logging target weights + realized — confirm the design forward. (60/40 core
+   already runs via `sixtyforty_paper.py`.)
+2. **Micro-live (small):** deploy the ETF book unlevered at small size; validate rebalance discipline,
+   spreads, the trust/risk layer operating live.
+3. **Scale:** full capital, same unlevered book. Returns scale with capital, Sharpe unchanged.
+
+Gate each phase: forward Sharpe ~consistent, drawdown controlled, trust/risk layer behaving.
+
+---
+
+## 11. Go / no-go + risk limits
+
+- **Go:** you accept a modest (~0.5–0.7 Sharpe, ~$2–3k/yr on $35k) but *robust, well-risk-managed,
+  unlevered* diversified book whose return scales with capital — and that the engine's role is risk
+  management, not alpha.
+- **No-go / reconsider:** if you need higher absolute return at $35k (then the lever is *more capital*,
+  not more strategy), or if you want market-neutral (this is long-beta — accept the equity-correlated
+  drawdowns).
+- **Risk limits:** unlevered (MAXLEV 1.0); crypto sleeve ≤ ~15% target (trust-capped); weekly rebalance;
+  the adaptive layer de-risks automatically in vol spikes / correlation crises / drawdowns.
+
+**The honest bottom line:** after exhausting every alpha avenue (incl. the engine's own advanced
+ideas), this is what's real and deployable — a simple, diversified, adaptively-risk-managed harvest.
+The engine earns its keep as the risk/allocation brain, not as an oracle. The path to *meaningful*
+money is capital, not a better signal.
+
+---
+
+## Appendix A — Micro-live runbook (Phase 2, concrete)
+
+**Validation so far (historical, exhaustive):** in-sample Sharpe +1.20; holdout-OOS +1.41; 20-year
+(2006–26, conservative no-DBMF/crypto version) +0.96 surviving 2008/2020/2022 with −10% maxDD;
+bootstrap CI p5 +0.62 / median +0.99, P(Sharpe>0.5)=99%. The *only* thing history can't give is
+forward/live confirmation — that's what micro-live provides.
+
+**Objective:** validate *execution* — fills, spreads, the weekly/monthly rebalance, the live adaptive
+weighting — and start the real-money clock. **NOT to make money** (on $1k the P&L is pennies). It
+de-risks the scale step; that's its whole job.
+
+**Capital:** $500–1,000, fully at-risk, **unlevered**. Worst case ≈ the book's maxDD (~−7%) ≈ −$70.
+Trivial by design — that's the point of micro.
+
+**Instruments (all in ONE brokerage, fractional shares):**
+| Stream | Ticker | Note |
+|---|---|---|
+| equity | SPY | |
+| bonds | IEF | |
+| gold | GLD | |
+| commodity | PDBC | |
+| trend / CTA | DBMF | the real managed-futures ETF (our DIY trend was ~0) |
+| crypto | **IBIT** | iShares Bitcoin ETF — use instead of BTC spot: in-brokerage, fractional, **NO crypto-exchange counterparty tail** |
+
+Broker: any fractional-share, ~zero-commission (IBKR / Fidelity / Schwab / Robinhood). IBKR or
+Fidelity recommended (clean fractional fills).
+
+**Step 1 — today's target:**
+```
+python3 scripts/surfer/multistrat_paper.py weights 1000
+```
+→ exact % and $ per instrument (the adaptive trust + risk-layer output). Map the crypto line → IBIT.
+
+**Step 2 — place orders manually** (micro scale → eyeball the fills). Marketable-limit or market on
+these liquid ETFs. Record actual fill prices.
+
+**Step 3 — rebalance MONTHLY (not weekly) at micro scale.** At $1k, weekly deltas are ~$3 —
+rounding-dominated and operationally silly. Rebalance monthly, and only trade an instrument if its
+target weight drifted >~5% (hysteresis). The adaptive layer's value shows over months, not days.
+
+**Step 4 — track actual vs intended.** Each rebalance, log the harness's intended weights vs your
+actual fills → measure (a) tracking error, (b) realized cost (spread/slippage), (c) live-vs-paper match.
+
+**Gate to scale (~4–8 weeks):**
+- Live cumulative ≈ paper-forward cumulative (small tracking error).
+- Realized cost ≤ ~10–20 bp/rebalance (should be tiny on these ETFs).
+- Rebalance executes cleanly; adaptive weights behave sensibly; no operational surprises.
+- → all green: **scale to full capital, same unlevered book** (Sharpe is scale-invariant; capital is the lever).
+- → tracking error large / cost high: diagnose before scaling.
+
+**Risk limits:** unlevered (no margin); $500–1k at-risk; adaptive layer auto-de-risks in vol/DD;
+IBIT removes the counterparty tail. **What it does NOT do:** make meaningful money — it proves the
+machine runs cleanly with real fills before you commit real capital. The honest bridge from
+validated backtest to scaled deployment.

docs/superpowers/specs/2026-06-07-crypto-funding-harvest-deployable-spec.md

@@ -0,0 +1,183 @@
+# Crypto Funding Harvest — Deployable Strategy Spec
+
+**Date:** 2026-06-07
+**Status:** Validated (4 gates passed), pre-deployment
+**Branch:** surfer-universe
+**Author:** research campaign synthesis
+
+---
+
+## 1. Executive summary
+
+A **delta-neutral crypto perpetual-funding harvest**: hold `long spot + short perp` on coins
+whose funding rate is reliably positive, collecting the funding payment as carry with **no price
+exposure**. No prediction, no ML — a pure cross-sectional carry harvest gated by a regime filter.
+
+**Why this and not everything else:** a multi-week, multi-market search (efficient equities/futures
+= no alpha, leak-free ML IC 0.004; simple 60/40 = ~0.7 Sharpe ceiling; energy = capital not
+algorithm) found this is the **only edge that breaks the ~0.7 retail Sharpe ceiling**, and it does
+so because crypto funding is structurally inefficient (retail perp longs over-pay) and the harvest
+is *latency-insensitive* (no colocation needed — a carry trade, not a race).
+
+**Validated metrics (this dataset, 135 coins, 2019–2026):**
+
+| Gate | Result |
+|---|---|
+| Structural | funding positive 75.5% of the time; 78.8% of coins pay carry on an avg day |
+| Delta-neutral | worst monthly bucket −1.5% (price risk genuinely hedged) |
+| Liveness | 2025–26 weakness is *regime* (deleverage), not decay; positive-funding yield intact (~3bp, = 2019/2023 levels) |
+| **Clean OOS** | filter chosen on 2019–24, applied blind to 2025–26 → **OOS Sharpe +5.1 raw** (IS-best); 9/12 strong-IS filters also OOS-positive; naive no-filter FAILS OOS (−5.8) |
+
+**Honest expected performance (realistic, after basis-vol haircut ÷~2.5):**
+- **Sharpe ~2–3.6** (raw funding-only model shows 6–9; realistic accounts for basis/tracking-error vol we cannot model with single-price data)
+- **APR ~15–22%** on deployed capital (sensible-hurdle filter), regime-dependent
+- **Max drawdown ~−7 to −15%** (price-neutral; drawdowns are funding-flip + churn, not crashes)
+- **The Sharpe does NOT show the real risk: counterparty/exchange failure (−100% tail).**
+
+---
+
+## 2. The edge (why it exists, why it persists)
+
+- **Perpetual futures funding** is the mechanism that tethers perp price to spot. When perps trade
+  at a premium (bullish retail leverage), **longs pay shorts** a periodic funding rate (typically
+  8-hourly on Binance/Bybit/OKX).
+- **Net structural bias is positive**: retail over-leverages long → funding is positive ~75% of
+  the time. A `long spot + short perp` position is **delta-neutral** (spot and perp price moves
+  cancel) and **collects the funding** the short perp leg receives.
+- **Why it persists** (not arbitraged to zero): capturing it requires capital *pre-positioned on
+  each venue*, active management, and acceptance of counterparty risk — operational/risk frictions,
+  not informational ones. It is the crypto analogue of an insurance premium.
+- **Latency-insensitive:** funding accrues over 8h periods; seconds of execution delay are
+  immaterial. This is the single reason it is reachable by a non-colocated retail trader, unlike
+  cross-exchange *latency* arb (which needs colocation and is NOT this strategy).
+
+---
+
+## 3. Strategy rules (precise, implementable)
+
+### 3.1 Universe
+- Perpetual contracts on a **tier-1 venue** (see §4) that offers both spot and perp for the coin.
+- **Liquidity floor:** trailing-30d mean quote volume **> $5M/day** (validated threshold).
+- Exclude stablecoin-pair anomalies, delisting candidates, and coins without a spot leg.
+
+### 3.2 Signal (the regime filter — the load-bearing piece)
+- For each eligible coin, compute **trailing-30-day mean funding rate** `tf30`.
+- **Qualify a coin iff `tf30 > 5 bp/day`** (≈ persistently well-paid carry). This is the validated
+  filter: strong both IS (+7.9) and OOS (+9.1), and it sits the book out of the deleverage regime
+  that whipsaws the naive "harvest anything positive" version (which FAILED OOS).
+- Causality: position decisions use **yesterday's** funding (`tf30` through t−1) to harvest day t.
+  Funding is persistent, so this is both realistic and leak-free.
+
+### 3.3 Positioning
+- For each qualifying coin: **long spot notional X + short perp notional X** (delta-neutral).
+- **Equal-weight** across qualifying coins (validated; funding-weighting did not improve risk-adj).
+- **Cash when nothing qualifies** — in a deleverage regime few/no coins clear the 5bp hurdle; the
+  book correctly de-risks to stablecoin (do NOT force-harvest a dead regime).
+
+### 3.4 Rebalance & cost
+- **Daily** check: enter coins crossing above the hurdle, exit coins falling below.
+- Cost budget: strategy validated net of **10 bp round-trip** (perp + spot, both legs); survives to
+  ~20 bp. Use **maker/limit orders** on entry/exit where possible to stay inside budget. Avoid
+  rebalancing on marginal hurdle-crossings (add hysteresis: enter >5bp, exit <3bp, to cut churn).
+
+### 3.5 Sizing
+- Per-coin notional = (deployed capital) / (number of qualifying coins), capped by §4 per-venue
+  limits.
+- **Leverage:** the short-perp leg uses exchange margin; keep effective leverage low (≤2–3×) so a
+  funding-flip + basis move cannot trigger liquidation. The spot leg is the hedge — never let the
+  perp get liquidated while holding spot (that converts delta-neutral into naked long).
+
+---
+
+## 4. Risk management — counterparty is THE risk (read this twice)
+
+Delta-neutrality removes *price* risk. It does **not** remove **counterparty/exchange risk**, which
+is the actual way this strategy produces a −100% (FTX, Mt. Gox, QuadrigaCX). The Sharpe ratio is
+blind to it. **This section matters more than the backtest.**
+
+1. **Venue selection:** only tier-1 venues with **proof-of-reserves**, deep liquidity, and a
+   solvency track record. No yield-farming protocols, no obscure CEXs chasing higher funding.
+2. **Collateral spreading:** spread capital across **≥2–3 venues** so no single failure is fatal.
+   *Caveat at $35k:* small capital makes spreading hard (per-venue minimums + cost). Below ~$50k,
+   counterparty concentration is unavoidable — treat the whole strategy as at-risk capital you can
+   lose entirely, and start tiny (§7).
+3. **Withdrawal discipline:** keep only working collateral on exchanges; sweep profits to
+   self-custody on a schedule. Never let the on-exchange balance grow unmonitored.
+4. **Liquidation guard:** low leverage (≤2–3×), automated margin-top-up alerts; the spot hedge must
+   always survive a perp-leg margin call.
+5. **Deleverage kill-switch:** if fleet-wide funding goes broadly negative (the 2022/2025-26 signal:
+   `frac_pos < 0.5` or `mean_all < 0`), **go fully to cash** — the filter does this automatically,
+   but add a hard override.
+6. **Stablecoin risk:** the cash leg sits in stablecoins (USDT/USDC) — itself a (smaller)
+   counterparty/depeg risk; diversify stable holdings.
+7. **Position cap per venue:** no more than (venue risk budget) on any one exchange.
+
+---
+
+## 5. Expected performance (honest)
+
+| Metric | Raw model (funding-only) | Realistic (after basis vol) |
+|---|---|---|
+| Sharpe | 6–9 | **~2–3.6** |
+| APR (deployed) | ~17–22% | ~15–22% |
+| Max drawdown | −7 to −15% | similar (price-neutral) |
+| Worst month | −1.5% | basis noise adds some |
+| Regime behavior | cash in deleverage | cash in deleverage |
+
+- **The realistic Sharpe (~2–3.6) is still 3–5× the 0.7 retail ceiling** for everything else found.
+- **Dollar reality at $35k:** ~15–20% APR ≈ **$5–7k/yr** expected, market-neutral — but with full
+  counterparty-loss tail. Scales linearly with capital (Sharpe is scale-invariant; capacity is far
+  above retail size).
+- **Unmodeled / open:** basis-vol (need spot+perp tick pairs to measure precisely), and the
+  counterparty tail (un-backtestable — managed via §4, not measured).
+
+---
+
+## 6. Validation status
+
+**Passed:**
+- Structural (funding positive 75.5%), delta-neutral (worst month −1.5%), liveness (regime not
+  decay), clean OOS (IS-chosen filter holds OOS +5.1, 9/12 robust, naive no-filter fails OOS).
+- Scripts: `scripts/surfer/crypto_funding_harvest.py`, `crypto_funding_liveness.py`,
+  `crypto_funding_oos.py`.
+
+**Not yet validated (must close before scaling):**
+- **Basis vol** — current Sharpe is funding-only; get spot+perp price pairs and remodel realized P&L
+  including tracking error → confirm realistic Sharpe.
+- **OOS length** — only ~1.5y OOS (single deleverage regime); held through the *hard* case, but
+  extend as data accrues.
+- **Counterparty** — un-backtestable; validated only by §4 design discipline.
+- **Live execution** — fills, slippage, funding-timestamp capture, margin mechanics on the real
+  venue (the paper-forward test, §7).
+
+---
+
+## 7. Phased rollout (gate each phase before the next)
+
+1. **Paper-forward (4–8 weeks):** run the daily algorithm against *live* funding feeds, record
+   intended positions + realized funding, **no capital**. Confirms the edge persists on truly
+   unseen forward data (mirrors the surfer PoC's cron paper-test). Gate: forward funding-APR
+   positive and consistent with backtest.
+2. **Micro-live ($1–3k, 1 venue):** smallest real size to validate execution, fills, funding
+   capture, margin mechanics, cost budget. Gate: realized net ≈ paper, cost ≤ 15bp round-trip.
+3. **Small-live ($5–15k, 2 venues):** add collateral spreading, confirm counterparty controls and
+   withdrawal discipline operate. Gate: clean operation through a funding-flip / mini-deleverage.
+4. **Scale ($35k+):** deploy full capital across ≥3 venues, full §4 risk stack engaged.
+
+**Stop at any gate that fails.** This is the same diagnostic-first discipline that caught every
+false positive in the search (the inflated 5.68 Sharpe, the carry trap, the equity factors).
+
+---
+
+## 8. Go / no-go
+
+**Go** if: paper-forward (§7.1) confirms positive forward funding-APR AND you accept that this is
+**at-risk crypto capital with a counterparty −100% tail** in exchange for a realistic ~2–3.6 Sharpe
+market-neutral return.
+
+**No-go** if: you cannot accept the counterparty tail, OR paper-forward shows the edge has decayed
+(funding compression to ~cost), OR you cannot spread across ≥2 reputable venues at your capital.
+
+**The honest framing:** this is the one validated way past 0.7 the whole search found. It is real,
+alive, and reachable — and the price of admission is crypto + counterparty risk, not market risk.
+That trade-off is a values decision, now made with complete information.

docs/superpowers/specs/2026-06-08-cagr-reference-plan.md

@@ -0,0 +1,86 @@
+# CAGR-Maximizer — Honest Reference Plan
+
+**Date:** 2026-06-08
+**Purpose:** the honest answer to "how do we build the most wealth?" after exhaustively testing strategies,
+the adaptive multi-strat book, leverage, and risk-management. Comparison tool: `scripts/surfer/strategy_compare.py`.
+
+---
+
+## The core truth (measured, not assumed)
+
+1. **CAGR builds terminal wealth, not Sharpe.** Over a long horizon with contributions, the highest-CAGR
+   asset wins — even at lower Sharpe. (SPY 0.64 Sharpe but 12.4% CAGR beats the book's 0.96 Sharpe / 5.1% CAGR on money.)
+2. **More CAGR = more risk. Always. No exception.** Every risk-managed variant (the book, the overlay,
+   blends) gives up CAGR for lower drawdown. Risk management is *insurance you pay for*, not a free improvement.
+3. **The only "free" CAGR is drag reduction** — taxes, fees, and (biggest) not selling at the bottom.
+4. **The book's apparent Sharpe edge (0.96 vs 0.64) was an rf=0 artifact.** Measured as excess-over-financing
+   (what matters for leverage), equities (0.49) actually beat the diversified book (0.40) in this era —
+   so levering the book does NOT beat buy-hold equity. The institutional risk-parity edge needs the book's
+   *excess* Sharpe > equity's, which it wasn't.
+5. **The leverage moat is cheap financing.** Futures/box ≈ SOFR+spread (~3%); retail margin ~6.5% destroys it.
+
+---
+
+## The candidates (measured 2006–2026, incl. 2008 −55%)
+
+| Strategy | CAGR | vol | Sharpe | maxDD | $360k+$8k/mo → 20y |
+|---|---|---|---|---|---|
+| **SPY buy-hold (1.0x)** | ~10.6% | 19% | 0.64 | **−55%** | ~$7.8M |
+| SPY 1.2–1.3x (cheap futures) | ~12–13% | 23–25% | ~0.6 | **−63 to −67%** | ~$8.8–9.2M |
+| SPY 1.5x | ~14.5% | 28% | — | −73% | ~$10.1M |
+| 60/40 | ~8% | 14% | 0.73 | −20 to −30% | ~$6.5M |
+| 70/30 SPY+book | ~10% | 14% | 0.73 | −40% | ~$9.0M |
+| SPY + adaptive overlay | ~8% | 11% | 0.77 | −18% | ~$6.7M |
+| **Adaptive multi-strat book** | ~5–6% | 5% | **0.96** | **−10%** | ~$4.3–4.9M |
+
+(SPY ≥2x: CAGR peaks ~2x then volatility-drag falls; −84%+ drawdown = ruin/margin-call territory. Not viable.)
+
+---
+
+## CAGR levers, ranked by sense
+
+| Lever | Effect | Honesty |
+|---|---|---|
+| **Max equity allocation, held** | base ~10–11%/yr | highest-return asset; reliable |
+| **Reduce drag** (tax-advantaged account, cheap index funds, never sell, stay invested) | **+1–3%/yr** | **free + reliable — most people leave this on the table** |
+| **Time + contributions ($8k/mo)** | dominant terminal driver | your strongest card |
+| **Modest cheap leverage (~1.2–1.3x via futures)** | +~1.5–2%/yr (+$1–1.4M/20y) | amplifies drawdown to −63–67% + margin-call risk |
+| Factor tilts (small-cap value, momentum, quality) | hist. +1–2%/yr | less reliable forward |
+| ~~Risk-managed book / overlay~~ | **−4%/yr CAGR** | a risk *reducer*, not a CAGR maximizer |
+
+---
+
+## The leverage reality (the only knob with real upside — handle with care)
+
+- 1.2–1.3x boosts CAGR ~+1.5–2%/yr (~$1–1.4M over 20y) but turns the −55% crash into **−63–67%**.
+- **−67% on a $2M pot = $660k at the 2009 bottom**, while still contributing into the abyss.
+- **Margin calls:** fixed-notional futures in a −55% crash force liquidation at the bottom → *realized* ruin
+  (the backtest "survives" only because it's daily-rebalanced math). Lived experience is worse than backtest.
+- Above ~1.5x: reckless (−73%+); above ~2x: ruin.
+- **The real constraint is not the math — it's whether you survive a −65% drawdown + margin without forced/panic selling.**
+
+---
+
+## Recommendation
+
+For a disciplined accumulator with a strong savings rate (~$8k/mo), long horizon, and stable income:
+
+1. **Max equity (100%), in a tax-advantaged account, cheap index funds, contribute monthly, never sell.**
+   Base case ~$7.8M over 20y. Simple, cheap, the most for the disciplined.
+2. **Optional ~1.2–1.3x via futures (cheap financing)** — only if you can survive (financially + emotionally)
+   a −65% drawdown + margin calls without forced selling. Adds ~$1–1.4M over 20y.
+3. **Always: reduce drag** (tax-efficiency, low fees, stay invested) — the free CAGR.
+4. **The adaptive multi-strat book is NOT the wealth-maximizer** — it's the *capital-preservation / low-stress / withdrawal-phase* tool. Use it if you'd panic-sell equities, are near withdrawal, or value sleep over ~4%/yr.
+
+**The decision hinges on one honest question: do you sell at the bottom of a crash, or hold?**
+- Hold → high equity (±1.3x), simplest, most money.
+- Unsure → 70/30 or the overlay (pay return for a tolerable ride).
+- Need capital preservation → the book.
+
+---
+
+## What's deployed
+
+- **Adaptive multi-strat book:** live on IBKR paper via the K8s CronJob (autonomous, daily eval) — the *low-drawdown* reference.
+- **Comparison tool:** `scripts/surfer/strategy_compare.py` — re-runnable side-by-side of all candidates (backtest + trajectory).
+- **Honest bottom line:** there is no secret signal and no clever leverage trick that beats "high-return assets, cheap, held long, funded heavily." We tested them all. The wealth levers are savings rate, time, low costs, and the discipline to not sell.

docs/superpowers/specs/2026-06-21-decommission-gitlab-design.md

@@ -0,0 +1,69 @@
+# Decommission GitLab (compute) — Design (Phase 2C)
+
+**Date:** 2026-06-21
+**Status:** Design (approved)
+**Repo:** foxhunt
+**Part of:** Phase 2 ([[project_phase2_gitlab_to_gitea]]). 2A (TF state→Scaleway) + 2B (Gitea + Scaleway
+registry + Argo webhook) DONE. This is **2C**. 2D (IaC relocation foxhunt→fxhnt) follows.
+
+## Motivation
+
+GitLab is fully replaced: git→Gitea (`git.fxhnt.ai`, SHA-verified), Terraform state→Scaleway Object
+Storage, container images→Scaleway Container Registry, CI/CD→Argo (Gitea webhook). It has **zero live
+consumers** but still runs 12 pods / ~4.6 GiB RAM. 2C removes GitLab's compute and retains its data as a
+cold backup (per the data-safety preference — migrated copies exist and are verified, but keep the
+originals through a grace period).
+
+## Scope — remove (compute only)
+
+1. **Helm uninstall** (foxhunt ns): `gitlab` (chart gitlab-9.9.3, rev 25) + `foxhunt-agent`
+   (gitlab-agent). Frees ~4.6 GiB RAM / 12 pods.
+2. **Dead Argo Events**: `kubectl delete eventsource gitlab-push` + `sensor ci-pipeline-trigger`
+   (Rust-era GitLab CI triggers; the gitea-push/gitea-deploy pair replaced them in 2B).
+3. **nginx proxy `:5050` registry block** + `containerPort: 5050` in `infra/k8s/gitlab/tailscale-proxy.yaml`
+   (the GitLab container registry face; images are on Scaleway now).
+4. **Repo manifest** `infra/k8s/gitlab/values.yaml` (GitLab Helm values — chart no longer deployed).
+
+## 🛑 PRESERVE (cold backup — delete in a later grace-period pass)
+- **gitaly PVC** `repo-data-gitlab-gitaly-0` (50 GiB, old git repos). StatefulSet PVCs survive
+  `helm uninstall` by default — no action needed to keep it.
+- **MinIO buckets**: `foxhunt-gitlab-artifacts` (holds the superseded TF-state fallback),
+  `foxhunt-gitlab-packages`, `foxhunt-gitlab-registry`.
+- **Scaleway buckets**: `foxhunt-gitlab-artifacts`, `foxhunt-gitlab-registry`.
+- **All `gitlab-*` secrets** (e.g. `gitlab-pat`, `gitlab-s3-credentials`, `gitlab-tls-cert`).
+- **`infra/k8s/gitlab/tailscale-proxy.yaml`** — KEEP: it is now the **Gitea** proxy (serves
+  `git.fxhnt.ai` + all `*.fxhnt.ai`). Only its `:5050` block is removed.
+
+## Execution order (safety-critical)
+
+1. **Proxy first:** remove the `:5050` server block + `containerPort: 5050` from `tailscale-proxy.yaml`;
+   `kubectl apply` + rollout restart; **verify `git.fxhnt.ai` + `dashboard.fxhnt.ai` still 200**.
+   *Rationale:* nginx resolves `proxy_pass` upstream hostnames at startup. If GitLab is uninstalled while
+   the `:5050` block still references `gitlab-registry.foxhunt.svc`, nginx fails to start on its next
+   restart → **all `*.fxhnt.ai` go down**. Removing the dead upstream first defuses this.
+2. **helm uninstall** `gitlab` then `foxhunt-agent`.
+3. **Delete dead Argo Events** (`gitlab-push`, `ci-pipeline-trigger`).
+4. **Remove `values.yaml`** from the repo; commit.
+5. **Verify** (see acceptance).
+
+## Risks / safety
+
+- **Proxy outage** if `:5050` not removed before uninstall → mitigated by step 1 + the 200-check gate.
+- **Irreversible**: helm uninstall. Mitigated: all data retained (gitaly PVC + buckets + secrets), so a
+  reinstall could re-attach. Migrated copies (Gitea/Scaleway) are the primary going forward.
+- **Shared resource**: confirm the `gitlab` release owns only GitLab objects (it does — namespaced release).
+  The `postgres`/`minio`/`tailscale-proxy` are separate and stay.
+- **No `.dbn`/market data** in any GitLab bucket (git artifacts/registry/packages only) — unaffected.
+
+## Out of scope
+- Deleting the retained GitLab data (gitaly PVC + buckets + secrets) — a later grace-period pass.
+- 2D IaC relocation foxhunt→fxhnt (separate spec); re-archiving the foxhunt repo happens there.
+
+## Acceptance criteria
+- `helm list -n foxhunt` shows no `gitlab` / `foxhunt-agent`; GitLab pods gone (~4.6 GiB RAM reclaimed).
+- `git.fxhnt.ai` + `dashboard.fxhnt.ai` return 200; a push to fxhnt master still auto-deploys via
+  Gitea→Argo.
+- `gitlab-push` eventsource + `ci-pipeline-trigger` sensor deleted; `gitea-push`/`gitea-deploy` still present.
+- `:5050` block gone from the proxy; proxy healthy.
+- **Cold backup intact:** gitaly PVC + all GitLab MinIO/Scaleway buckets + `gitlab-*` secrets still present.
+- `infra/k8s/gitlab/values.yaml` removed; `tailscale-proxy.yaml` kept; committed.

docs/superpowers/specs/2026-06-21-decommission-rust-infra-design.md

@@ -0,0 +1,80 @@
+# Decommission Dead Rust Infra — Design (Phase 1 of infra consolidation)
+
+**Date:** 2026-06-21
+**Status:** Design (pending approval)
+**Repo:** foxhunt (where the platform IaC currently lives)
+
+## Motivation
+
+The active trading work is now the Python **fxhnt** fund; the Rust **foxhunt** ML/HFT system is dormant.
+Its dedicated infra (GPU training pools, Rust build/training caches, GPU CI runners, training artifact
+buckets, training manifests) is unused but still declared/provisioned. Phase 1 decommissions it — for
+cost/clutter savings and to shrink the surface before Phase 2 (relocating the remaining *platform* IaC
+into `fxhnt/infra`, a separate spec).
+
+**Verified unused (2026-06-21):** L40S + H100 pools have 0 nodes; `cargo-target-{cpu,cuda,cuda-test}` +
+`feature-cache-pvc` are `Used By: <none>`; no gitlab-runner pods running.
+
+## Scope — what gets removed (all verified unused)
+
+1. **Terraform (kapsule module + terragrunt.hcl):** set to `false` →
+   - `enable_ci_training_l40s_pool` (L40S-1-48G GPU pool)
+   - `enable_ci_training_h100_pool` (H100-1-80G GPU pool)
+   - `enable_ci_compile_cpu_hm_pool` (POP2-HM-32C-256G precompute pool — Rust `precompute_features` only)
+   `terragrunt apply` destroys exactly these 3 pools. Also delete their now-dead var blocks/resources.
+2. **Helm uninstall** (foxhunt ns): `gitlab-runner-h100`, `gitlab-runner-h100-sxm`, `gitlab-runner-h100x2`
+   (Rust GPU CI runners). **Main `gitlab-runner`:** verify nothing non-Rust uses it (fxhnt has no
+   `.gitlab-ci.yml`; cockpit deploys via Argo) → uninstall if confirmed dead, else keep. (confirm step)
+3. **PVCs delete** (foxhunt ns — PURE BUILD CACHES only, unmounted-verified, ~175 GB reclaimed):
+   `cargo-target-cpu` (60Gi), `cargo-target-cuda` (45Gi), `cargo-target-cuda-test` (30Gi),
+   `sccache-cpu` (20Gi), `sccache-cuda` (20Gi). These hold zero data — regenerated on any build.
+4. **MinIO buckets delete** (NO market data — verified 0 `.dbn` objects): `foxhunt-binaries` (2.3 GiB
+   compiled binaries), `foxhunt-training-results` (4.4 GiB run logs/outputs), `foxhunt-models` (empty).
+5. **Repo manifests/templates remove** (foxhunt repo): `infra/k8s/training/`, `infra/k8s/gpu-overlays/`,
+   the Rust Argo workflow templates (`train-multi-seed-template.yaml`, `lob-backtest-sweep-template.yaml`,
+   `ci-pipeline-template.yaml`, alpha-rl train templates), `infra/k8s/jobs/download-trades-job.yaml`
+   (databento), the 3 GPU-runner helm value files. Also remove the dead Argo `WorkflowTemplate`s from the
+   cluster (`kubectl delete wftmpl`) for the Rust train/backtest pipelines.
+
+## 🛑 EXPLICITLY PRESERVE (do NOT delete — market data + reusable assets)
+- **`training-data-pvc` (500 GiB)** — the raw Databento MBP-10 `.dbn` market data. KEEP.
+- **`foxhunt-training-data` bucket (38 GiB, 54 `.dbn`/`.zst`)** — Databento market data (expensive to
+  re-acquire; the fund's `.dbn` backtests read it). KEEP.
+- **`test-data-pvc` (50 GiB)** — `.dbn` test subsets (tier-1.5 smoke uses test_data). KEEP (verify, don't delete).
+- **`feature-cache-pvc` (100 GiB)** — derived ML features; regenerable but compute-costly. KEEP (conservative).
+- All fxhnt/platform data: `fxhnt-backtest-data`, `fxhnt-surfer-data`, `multistrat-state`, `questdb-pvc`,
+  `tempo-data`, `netbird-data`, `foxhunt-gitlab-*` + `foxhunt-backups` buckets.
+General rule: delete only **pure compute/build artifacts** (caches, compiled binaries, run logs, GPU pools);
+**never** anything holding `.dbn`/market data or any potentially-reusable dataset.
+
+## Out of scope (KEEP — shared platform / Python fund)
+`ci-compile-cpu` pool (Python cockpit builds), platform pool, GitLab, MinIO, monitoring, Mattermost,
+Stalwart, Kanidm, NetBird, DNS, databases, cert-manager, tailscale proxy, public-gateway, dagster/cockpit,
+the fxhnt forward-track jobs. Phase 2 (IaC relocation to `fxhnt/infra` + TF-state move) is a separate spec.
+
+## Execution order (each destructive step verified + confirmed)
+1. **Terraform first**: edit terragrunt.hcl (3 `enable_*=false`), `terragrunt plan` → **STOP unless the
+   plan shows ONLY the 3 pools destroyed + 0 other destroys**; then `apply`.
+2. **Helm uninstalls** (GPU runners; main runner only after the use-check confirms dead).
+3. **PVC deletes** (re-confirm `Used By: <none>` immediately before each delete — irreversible).
+4. **Bucket deletes** (list contents first; irreversible — explicit confirm; `mc rb --force` via the
+   port-forward + minio creds).
+5. **Repo cleanup**: remove the dead manifests/templates + module var blocks, `kubectl delete wftmpl` the
+   dead templates, commit.
+
+## Risks / safety
+- **Irreversible**: PVC + bucket deletes, GPU-pool destroys. Mitigation: verify-unused immediately before
+  each; Terraform plan reviewed for 0-unexpected-destroys; explicit confirm on PVC/bucket deletes.
+- **Mis-scope risk**: a kept resource accidentally listed. Mitigation: the Terraform plan gate (only the 3
+  pools) + the `Used By` re-check + bucket content listing before delete.
+- **Main gitlab-runner**: don't uninstall without confirming no active consumer (could break a CI path).
+- Cluster health unaffected: nothing here touches the cockpit/dagster/fund/platform services.
+
+## Acceptance criteria
+- `terragrunt plan` (kapsule) shows the 3 GPU/precompute pools gone, then clean (no changes).
+- GPU-runner helm releases uninstalled; main runner resolved (kept or uninstalled per check).
+- The 5 build-cache PVCs deleted (cargo×3 + sccache×2, ~175 GB reclaimed); 3 Rust buckets deleted
+  (binaries, training-results, models). **`.dbn` market data untouched** (training-data-pvc + the
+  foxhunt-training-data bucket + test-data-pvc still present, byte-for-byte).
+- Dead training/gpu manifests + Argo templates removed from repo + cluster; committed.
+- Cockpit (`dashboard.fxhnt.ai` 200), dagster, and the fund tracks still healthy post-cleanup.

docs/superpowers/specs/2026-06-21-gitea-replace-gitlab-design.md

@@ -0,0 +1,114 @@
+# Stand up Gitea + migrate repos + move cockpit registry — Design (Phase 2B)
+
+**Date:** 2026-06-21
+**Status:** Design (approved)
+**Repo:** foxhunt (IaC current home; relocation to fxhnt is Phase 2D)
+**Part of:** Phase 2 (GitLab → Gitea consolidation, [[project_phase2_gitlab_to_gitea]]). 2A (TF state →
+Scaleway) DONE. This is **2B**. 2C (GitLab removal) and 2D (IaC relocation) follow.
+
+## Motivation
+
+Self-hosted GitLab uses ~6.5 GiB RAM across 17 pods (webservice 1.96 GiB + 3× sidekiq ~3 GiB + gitaly +
+registry + prometheus + …). The active work is the Python `fxhnt` fund; GitLab is Rust-era overkill.
+Replace it with **Gitea** (~150–250 MiB, chart + existing Postgres) for git hosting, and move the
+**cockpit container image** to **Scaleway Container Registry** so GitLab can be removed in 2C without
+breaking the live cockpit deploy.
+
+## Decisions (from brainstorming)
+
+- **Gitea install:** `gitea/gitea` Helm chart. Subcharts `postgresql`/`redis-cluster`/`redis`/`memcached`
+  **disabled**. Built-in **Actions disabled** (Argo does CI/CD). Cache = memory (single replica).
+- **Database:** reuse the **existing in-cluster `postgres`** (svc `postgres:5432`, already has a backup
+  cronjob). Create a dedicated `gitea` database + role.
+- **Repos:** migrate **both** — `fxhnt` (active) and `foxhunt` (876 MB, flagged **archived/read-only**) —
+  via `git push --mirror` from existing local clones.
+- **Canonical hostname stays `git.fxhnt.ai`** (no permanent `gitea.fxhnt.ai`, no remote-URL churn).
+  Gitea comes up internal-only; validate over a port-forward; then cut `git.fxhnt.ai` (HTTPS) + SSH over
+  to Gitea in one switch. **SSH on `:22`** (nothing host-SSHes the git node, so the standard port is free);
+  clone URLs become the clean `git@git.fxhnt.ai` and `:2222` is retired (local remotes updated once).
+  GitLab keeps running (unaddressed) as the 2C-removal fallback.
+- **Container registry:** cockpit image → **Scaleway Container Registry** (`rg.fr-par.scw.cloud/bizworx/`,
+  namespace exists). Retire the in-cluster GitLab registry. External/managed, no registry pod.
+- **Argo integration:** Gitea push webhook → Argo Events webhook eventsource → sensor → submit
+  `wftmpl/fxhnt-cockpit`. Replaces the old `gitlab-push-eventsource`.
+
+## Components
+
+1. **`infra/k8s/gitea/` (new):** Helm values `values.yaml` + a thin install (release `gitea`, ns
+   `foxhunt`). Storage: 1 PVC `sbs-default-retain` ~5 Gi (foxhunt 876 MB + fxhnt + headroom). Resources:
+   requests 128 Mi/100m, limits 512 Mi. Admin user + secrets from a k8s secret (NOT committed).
+   Gitea config: `server.SSH_PORT`/`SSH_LISTEN_PORT`, `server.ROOT_URL=https://git.fxhnt.ai/`,
+   `server.DISABLE_SSH=false`, DB type `postgres` → `postgres:5432/gitea`.
+2. **Postgres prep:** one-time `psql` on the existing `postgres` pod — `CREATE ROLE gitea …; CREATE
+   DATABASE gitea OWNER gitea;`. Credentials in a k8s secret consumed by the Gitea release.
+3. **Exposure:**
+   - HTTPS: extend the nginx `*.fxhnt.ai` proxy (`infra/k8s/gitlab/tailscale-proxy.yaml`) — at cutover,
+     repoint the `git.fxhnt.ai` server block `proxy_pass` → `gitea-http.foxhunt.svc:3000`.
+   - SSH: expose Gitea SSH on `:22` via the Tailscale proxy (socat) at cutover; retire `:2222`; update the
+     two local remotes to `ssh://git@git.fxhnt.ai/...`.
+   - Registry `:5050` server block: removed (Scaleway registry is external, not proxied here).
+4. **Registry migration:**
+   - `fxhnt/infra/argo/cockpit-build-deploy.yaml`: kaniko `--destination` + `--cache-repo` →
+     `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest` (+ cache repo); drop the `--insecure-registry`/
+     `--skip-tls-verify` GitLab flags (Scaleway is TLS); auth via a Scaleway registry dockerconfig secret.
+   - `fxhnt/infra/k8s/orchestration/dagster.yaml` + dashboard: image ref →
+     `rg.fr-par.scw.cloud/bizworx/fxhnt-cockpit:latest`; `imagePullSecrets` → Scaleway registry secret.
+   - Create the Scaleway registry pull/push secret (dockerconfigjson) in `foxhunt` ns.
+5. **Argo Events wiring:** `infra/k8s/argo/events/` — a Gitea webhook eventsource + sensor that submits
+   `fxhnt-cockpit` on push to `fxhnt` default branch (reuses the surviving eventbus + workflow-trigger
+   machinery). Webhook secret in a k8s secret; configured in Gitea repo settings.
+6. **Deploy script:** `fxhnt/scripts/argo-deploy-cockpit.sh` + the `fxhnt-cockpit` wftmpl clone URL →
+   Gitea (`git.fxhnt.ai`). After cutover the remote name is unchanged, so this is a clone-source confirm.
+
+## Data flow
+
+Dev pushes to `git.fxhnt.ai` (Gitea) → webhook → Argo Events sensor → `fxhnt-cockpit` workflow → kaniko
+builds + pushes to Scaleway registry → `kubectl apply` + rollout → dagster/dashboard pull the new image
+from Scaleway. Terraform unaffected (state external since 2A).
+
+## Migration & cutover sequence (each step verified)
+
+1. Create `gitea` DB on existing postgres + secrets.
+2. `helm install gitea` (internal svc only). Verify `/api/healthz` 200, admin login via port-forward.
+3. Port-forward Gitea; `git push --mirror` `fxhnt` and `foxhunt` to it; mark `foxhunt` archived.
+4. **Validate (pre-cutover):** clone both repos back from the port-forward, diff top commit SHAs vs
+   GitLab — must match.
+5. **Registry migration:** create Scaleway registry secret; update kaniko build (+cache) + deploy image
+   refs; run one cockpit build → confirm image in Scaleway registry + a test rollout pulls it OK.
+6. **Cutover:** repoint nginx `git.fxhnt.ai` → Gitea + SSH `:2222` → Gitea; remove the `:5050` block.
+   Verify `git.fxhnt.ai` serves Gitea (web + clone + SSH push).
+7. **Wire Argo Events:** add Gitea webhook → sensor; test push to `fxhnt` main fires a cockpit deploy.
+
+## Safety / reversibility
+
+- GitLab stays fully running throughout 2B (just unaddressed after cutover) — rollback = repoint nginx/SSH
+  back to GitLab. Removal is **2C** only.
+- The Scaleway registry change is additive until the deploy refs flip; the GitLab registry image stays as
+  fallback until 2C.
+- All repo data validated (SHA diff) before the public name moves.
+- Secrets (DB creds, admin, webhook, registry dockerconfig) are k8s secrets — **never committed**.
+
+## Risks
+
+- **Registry auth:** Scaleway registry needs a valid API-key dockerconfig for both kaniko push and pod
+  pull. Mitigation: test the build+rollout (step 5) before cutover.
+- **Gitea↔Postgres coupling:** Gitea shares the existing `postgres`. If that instance is later removed,
+  Gitea breaks. Acceptable (postgres is a kept platform service with backups); noted for 2C/2D.
+- **SSH port collision at cutover:** GitLab + Gitea can't both own `:2222` on the Tailscale proxy. Cutover
+  flips it atomically; brief push unavailability during the switch (single maintainer — fine).
+- **Webhook reachability:** Gitea (in-cluster) must reach the Argo Events webhook svc in-cluster — same
+  ns, no Tailscale hop needed.
+
+## Out of scope
+
+GitLab helm uninstall + GitLab buckets/state cleanup (**2C**); `git mv` IaC foxhunt → fxhnt (**2D**).
+
+## Acceptance criteria
+
+- Gitea running (1 pod, existing postgres), `/api/healthz` 200, ~≤300 MiB.
+- Both repos in Gitea; clone SHAs match GitLab; `foxhunt` archived.
+- Cockpit image builds to + deploys from Scaleway Container Registry; `dashboard.fxhnt.ai` 200 on the
+  Scaleway-sourced image.
+- `git.fxhnt.ai` (HTTPS + SSH `:2222`) serves Gitea; git remotes unchanged.
+- Push to `fxhnt` main auto-triggers a cockpit deploy via Argo Events.
+- GitLab still running as fallback (removed in 2C).

docs/superpowers/specs/2026-06-21-relocate-iac-to-fxhnt-design.md

@@ -0,0 +1,85 @@
+# Relocate Platform IaC foxhunt → fxhnt + Archive foxhunt — Design (Phase 2D)
+
+**Date:** 2026-06-21
+**Status:** Design (approved)
+**Repos:** foxhunt (source), fxhnt (destination)
+**Part of:** Phase 2 ([[project_phase2_gitlab_to_gitea]]). 2A/2B/2C DONE. This is the final sub-project, 2D.
+
+## Motivation
+
+The active work is the Python `fxhnt` fund; `foxhunt` is the dormant Rust system that still happens to
+hold the platform IaC. Consolidate the active platform IaC into `fxhnt` so it is the single home, then
+archive `foxhunt` (keeping its Rust code + history as a read-only snapshot).
+
+## Decision (keystone) — no Terraform state migration
+
+State keys are `${path_relative_to_include()}/terraform.tfstate` (relative to `root.hcl`). Moving
+`infra/live/` + `infra/modules/` into `fxhnt/infra/` **with byte-identical internal structure**
+(`fxhnt/infra/live/production/root.hcl`, `.../<module>/terragrunt.hcl`, module `source = "../../../modules/<m>"`)
+keeps every state key unchanged (`kapsule/`, `public-gateway/`, `dns/`, `block-storage/`). **No state
+migration** — proven by `terragrunt plan = No changes` from the new location. Move method: **plain copy**
+(git history stays in the archived foxhunt).
+
+## Scope — MOVE to `fxhnt/infra/` (active platform)
+
+- **Terraform:** `infra/live/` (terragrunt) + `infra/modules/` (kapsule, public-gateway, dns,
+  block-storage) — structure preserved exactly.
+- **k8s platform manifests** (`infra/k8s/`): `cert-manager`, `databases`, `dns`, `gitea`, `mattermost`,
+  `minio`, `monitoring`, `network-policies`, `secrets`, `services`, `stalwart`, `tailscale`, `storage`,
+  `argo` (surviving gitea events + rbac + netpol + values + databento-download-template + feature-cache-pvc
+  + kustomization), `gitlab` (only `tailscale-proxy.yaml` + `grafana-values.yaml`), `jobs`, `training`
+  (the kept `.dbn` data-acquisition tooling), `namespace.yaml`.
+- **Platform scripts:** the deploy/ops scripts that `kubectl apply`/terragrunt the above (enumerated in the
+  plan after a content scan). `fxhnt/scripts/argo-deploy-cockpit.sh` already lives in fxhnt.
+
+## Scope — STAYS in `foxhunt` (archived, Rust-dead)
+
+Rust source (`crates/`, `services/`, `bin/`, `proto/`, `migrations/`, etc.), Rust scripts (`argo-alpha-*`,
+`argo-compile-deploy`, `argo-sanitizer`, `argo-nsys`, `argo-precompute`, `argo-lob-sweep`, `argo-smoke`,
+`argo-download`), Rust docker build images (ci-builder/training-runtime), `infra/k8s/gpu-taint.yaml`, and
+all foxhunt git history.
+
+## Merge strategy (fxhnt already has `infra/{argo,docker,k8s/{jobs,network-policies,services,storage,orchestration}}`)
+
+Same-named dirs merge **file-by-file**. Pre-flight: scan for filename collisions between foxhunt's and
+fxhnt's `infra/k8s/{network-policies,services,jobs,storage}` (+ `docker`); resolve any collision explicitly
+before copying (expected: none — different resources). Foxhunt's `infra/k8s/argo` lands at
+`fxhnt/infra/k8s/argo` (fxhnt's existing argo is `fxhnt/infra/argo` — different path, no collision).
+
+## Execution order (verify-gated)
+
+1. **Collision pre-flight** — list overlaps; resolve/confirm none.
+2. **Copy Terraform** (`live/` + `modules/`) into `fxhnt/infra/` preserving structure; **`terragrunt
+   init -reconfigure && plan` from `fxhnt/infra/live/production/<each module>` → must be `No changes`**
+   (the no-migration proof). STOP on any diff.
+3. **Copy platform k8s manifests + scripts** into `fxhnt/infra/`; update any in-script `infra/...` paths.
+4. **Verify** manifests still apply + a cockpit deploy still works from fxhnt (it already does).
+5. **`git rm` the moved paths from foxhunt**; commit both repos; push.
+6. **Archive `foxhunt`** in Gitea (read-only) once 2-5 verified.
+
+## Safety / reversibility
+
+- Plain copy first; foxhunt files removed only after fxhnt copy + terragrunt-No-changes verified.
+- TF state untouched (same keys); the Scaleway backend doesn't care which repo runs terragrunt.
+- foxhunt archive (not delete) — full history + Rust code remain restorable.
+- No live workload changes — this is a source-of-truth move, not a cluster change. The running cluster is
+  unaffected (manifests are already applied; terraform state already external).
+
+## Risks
+- **Path drift** breaking TF state → mitigated by the structure-preserve rule + the `No changes` gate.
+- **Merge collision** silently overwriting a fxhnt file → mitigated by the collision pre-flight.
+- **A moved script with a stale `infra/` path** → grep + fix in step 3; verify in step 4.
+- **Premature archive** (foxhunt still referenced) → archive last, after all verification.
+
+## Out of scope
+- Deleting GitLab cold-backup data (separate grace-period pass).
+- Pruning the Rust codebase itself (foxhunt is archived as-is).
+- Phase 2E observability (Grafana/Prometheus).
+
+## Acceptance criteria
+- `fxhnt/infra/{live,modules}` present; `terragrunt plan` = `No changes` for all 4 modules from fxhnt
+  (state keys unchanged — no migration).
+- Platform k8s manifests + scripts present in `fxhnt/infra/`; no filename collisions lost.
+- Moved paths `git rm`'d from foxhunt; both repos committed + pushed to Gitea.
+- Cockpit deploy + `dashboard.fxhnt.ai` 200 still work (cluster unaffected).
+- `foxhunt` repo archived in Gitea; Rust code + history intact.

docs/superpowers/specs/2026-06-21-tfstate-to-scaleway-design.md

@@ -0,0 +1,94 @@
+# Migrate Terraform State to Scaleway Object Storage — Design (Phase 2A)
+
+**Date:** 2026-06-21
+**Status:** Design (approved)
+**Repo:** foxhunt (IaC current home; relocation to fxhnt is Phase 2D)
+**Part of:** Phase 2 (GitLab → Gitea consolidation). Sub-projects: **2A state migration (this)**, 2B Gitea
+stand-up + repo migration + Argo wiring, 2C GitLab decommission, 2D IaC relocation into fxhnt.
+
+## Motivation
+
+Terraform state for the whole production cluster currently lives in the **self-hosted GitLab** http
+backend (project ID=1), whose object data sits in **in-cluster MinIO**. That is a chicken-and-egg
+fragility: the state describing the cluster lives *inside* the cluster it manages. GitLab is also slated
+for decommission (Phase 2C, replaced by lightweight Gitea). Moving state to **external Scaleway Object
+Storage** decouples it from both GitLab and the cluster — a prerequisite for removing GitLab and a
+durability win on its own (state survives cluster loss).
+
+## Decision (keystone)
+
+**Target backend: Scaleway Object Storage (native S3), bucket `foxhunt-tfstate`** (already exists,
+fr-par, created ~3 months ago but never wired). Reuse it (no rename — avoids a needless create; the
+`foxhunt-` prefix is cosmetic and harmless). Locking via **OpenTofu native S3 lockfile** (`use_lockfile`,
+OpenTofu v1.11.5 present) — no DynamoDB.
+
+## Scope — what changes
+
+1. **`infra/live/production/root.hcl`** — replace the single `remote_state` block: `backend = "http"` (GitLab) →
+   `backend = "s3"` (Scaleway). New config:
+   ```hcl
+   remote_state {
+     backend = "s3"
+     generate = { path = "backend.tf", if_exists = "overwrite" }
+     config = {
+       bucket = "foxhunt-tfstate"
+       key    = "${path_relative_to_include()}/terraform.tfstate"
+       region = "fr-par"
+       endpoints = { s3 = "https://s3.fr-par.scw.cloud" }
+       skip_credentials_validation = true
+       skip_region_validation      = true
+       skip_requesting_account_id  = true
+       skip_metadata_api_check     = true
+       use_lockfile                = true
+     }
+   }
+   ```
+   Credentials: Scaleway access/secret keys exported as `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`
+   (the s3 backend reads AWS-style env vars) from the existing `scaleway-credentials` k8s secret.
+2. **State migration, per module, sequential** (`kapsule`, `public-gateway`, `dns`, `block-storage`):
+   - Back up first: `terragrunt state pull > /tmp/tfstate-backup-<module>.tfstate` (from GitLab, BEFORE
+     changing the backend for that module).
+   - `terragrunt init -migrate-state -force-copy` → copies GitLab state → Scaleway S3.
+   - `terragrunt plan` → **must report `No changes`** before proceeding to the next module.
+3. **Lock smoke test** (once, on one module): confirm `use_lockfile` works against Scaleway — a lock
+   object is created during apply/plan-with-lock and released after (verify no orphan `.tflock` left).
+
+## Migration order & gates
+
+`kapsule` → `public-gateway` → `dns` → `block-storage`. After EACH: backup taken, `-migrate-state`
+succeeded, `plan = No changes` confirmed. **STOP and revert if any module's post-migration plan shows
+drift** (revert = restore `http` backend in root.hcl + `terragrunt init` to re-attach GitLab state; the
+GitLab copy is never deleted in 2A).
+
+## Safety / reversibility
+
+- **Old GitLab state is NOT deleted in 2A** — it stays as a live fallback until GitLab is removed in 2C.
+  So the entire migration is reversible by flipping `root.hcl` back to the http backend.
+- `/tmp` state backups taken before each module (belt-and-suspenders; never committed — contains
+  resource IDs, not secrets, but treat as sensitive).
+- One module at a time with a `plan = No changes` gate prevents a bad backend config from cascading.
+- `endpoints.s3` (not deprecated top-level `endpoint`) + the four `skip_*` flags are required for the
+  AWS s3 backend to talk to Scaleway's S3-compatible API without AWS-specific preflight calls.
+
+## Risks
+
+- **Scaleway S3 quirks:** some provider/backend versions need `skip_s3_checksum = true` for Scaleway. If
+  `-migrate-state` errors on checksum, add it. (Documented as a known fallback in the plan.)
+- **Lock semantics:** Scaleway Object Storage must honor conditional-write for `use_lockfile`. The lock
+  smoke test catches this; fallback is to run with locking disabled (single maintainer) if unsupported.
+- **Credential confusion:** the s3 backend reads `AWS_*` env vars, NOT `SCW_*`. Plan must export both
+  (SCW_* for the provider, AWS_* for the backend) — a common foot-gun.
+
+## Out of scope (later sub-projects)
+
+Gitea stand-up + repo migration + Argo wiring (2B); GitLab helm removal + GitLab state/bucket cleanup
+(2C); `git mv` of IaC from foxhunt → fxhnt + path updates (2D). Files stay in foxhunt for 2A; only the
+backend changes.
+
+## Acceptance criteria
+
+- `infra/live/production/root.hcl` uses the Scaleway s3 backend; committed.
+- All 4 modules migrated: each shows `terragrunt plan = No changes` against the new backend.
+- State objects present in `foxhunt-tfstate` at `<module>/terraform.tfstate` (verified via `scw`/`mc`).
+- Lock smoke test passes (lock acquired + released, no orphan lockfile).
+- GitLab http state still intact (untouched fallback) — not yet deleted.

infra/k8s/argo/README-deps-cache.md

@@ -1,161 +0,0 @@
-# Deps Cache Image — Cold-start Compile Acceleration
-
-This README documents the **`ci-builder-cpu-with-deps:nightly`** image
-and the Argo plumbing that uses it. Created as part of the CI cold-start
-optimization sweep (April 2026).
-
-## What it is
-
-A Docker image that carries a pre-built `target/` directory at
-`/cargo-target-prebuilt/` containing the workspace's third-party
-dependency rlibs. It's pulled by the `compile-services` step of
-`compile-and-deploy-template.yaml` as an **initContainer** that rsyncs
-its payload into the `cargo-target-cpu` PVC.
-
-## Why it exists
-
-On a fresh node (autoscaler scaled up, fresh PVC, or after a PVC purge),
-`cargo build --release` would otherwise spend 3-5 minutes compiling
-~1500 third-party crates before touching workspace member code. With
-the deps cache image, the cold-start compile path is:
-
-1. **initContainer pulls image** from in-cluster registry (~30-60s)
-2. **rsync prebuilt rlibs into PVC** (~30-60s, ~6-8 GB transfer over local FS)
-3. **cargo build delta-compiles only workspace member code** (~1-2 min)
-
-Net cold-start saving: **2-4 min per compile pod**.
-
-## Files
-
-| File | Role |
-|------|------|
-| `infra/docker/Dockerfile.ci-deps-cache` | Builds the deps-cache image (multi-stage). Stage 1 atop ci-builder-cpu, runs `cargo build --release --workspace`. Stage 2 thin Ubuntu + rsync layer carrying just `/cargo-target-prebuilt/`. |
-| `infra/k8s/argo/refresh-deps-cache-template.yaml` | WorkflowTemplate that drives a Kaniko build of the Dockerfile + push to GitLab registry. Includes a CronWorkflow that runs nightly at 03:00 UTC (suspended by default). |
-| `infra/k8s/argo/compile-and-deploy-template.yaml` | The `compile-services` step has a `seed-deps-cache` initContainer that rsyncs `/cargo-target-prebuilt/` into the PVC, gated by a stamp file. |
-
-## How to refresh manually
-
-```bash
-# Trigger a one-shot rebuild against current main HEAD.
-argo submit --from workflowtemplate/refresh-deps-cache -n foxhunt
-```
-
-To target a specific commit or bump the cache version (forces re-seed
-on every compile pod):
-
-```bash
-argo submit --from workflowtemplate/refresh-deps-cache -n foxhunt \
-  -p commit-sha=$(git rev-parse main) \
-  -p deps-version=2 \
-  -p image-tag=nightly
-```
-
-## Enabling the nightly cron
-
-The CronWorkflow ships **suspended** so you can validate manually first.
-To enable:
-
-```bash
-kubectl -n foxhunt patch cronworkflow refresh-deps-cache-nightly \
-  --type=merge -p '{"spec":{"suspend":false}}'
-```
-
-To disable (return to manual-only):
-
-```bash
-kubectl -n foxhunt patch cronworkflow refresh-deps-cache-nightly \
-  --type=merge -p '{"spec":{"suspend":true}}'
-```
-
-## Bumping `DEPS_VERSION`
-
-The compile-services initContainer skips re-seeding when
-`/cargo-target/cpu_deps_v${DEPS_VERSION}.stamp` exists on the PVC.
-Bump `DEPS_VERSION` to force every running compile pod to re-seed
-its PVC from a fresh image. Bump in two places (must match):
-
-1. `refresh-deps-cache-template.yaml` workflow `deps-version` parameter
-   default value
-2. `compile-and-deploy-template.yaml` initContainer `DEPS_VERSION` env var
-
-Bump when:
-- Cargo.lock has churned significantly (most prebuilt rlibs
-  fingerprint-mismatch cargo's freshness check)
-- The Rust toolchain version changes (e.g. 1.89 -> 1.90)
-- A workspace member is renamed or split (rare)
-
-You do **not** need to bump for routine commits — cargo's own
-fingerprint check will discard stale rlibs and rebuild as needed.
-
-## Debugging
-
-**Symptom: compile-services is still slow on a fresh PVC.**
-
-Check the seed-deps-cache initContainer logs first:
-
-```bash
-argo logs -n foxhunt <workflow-name> -c seed-deps-cache
-```
-
-Look for:
-- `WARN: /cargo-target-prebuilt missing in image` — the image was built
-  but the snapshot dir is empty. The inner `cargo build --workspace`
-  failed in the deps-cache image build. Check the
-  `refresh-deps-cache` workflow logs.
-- `PVC already seeded for deps v1 (stamp present), skipping.` —
-  expected steady-state on warm pods.
-- `=== Seeding cargo-target-cpu PVC from prebuilt deps cache ===` +
-  `=== Seed complete ===` — first run on this PVC, working as designed.
-
-**Symptom: cargo recompiles many third-party deps anyway despite the
-seed completing.**
-
-Cargo's fingerprint check is sensitive to: rustc version, the entire
-target.rustflags list, `RUSTFLAGS` env var, source code mtimes (on
-build.rs), feature flags. The commit-SHA used to build the deps cache
-image must match the live compile workflow's flag set. The most common
-cause of full-rebuild is a `RUSTFLAGS` mismatch between the deps-cache
-image build and the compile pod (e.g. mold vs wild linker swap).
-
-To diagnose, look at the cargo build output: lines like
-`Compiling tokio v1.x.x` for crates that were obviously in the cache
-indicate fingerprint mismatch.
-
-**Symptom: PVC fills up faster than before.**
-
-Each seed adds ~6-8 GB to the PVC. The compile-services step has a
-prune-at-25GB guard that wipes `release/` and `debug/` if usage
-exceeds 25GB — when that fires, the next run re-seeds (stamp file is
-in PVC root, survives the prune). To raise the threshold, edit the
-`TARGET_SIZE_MB -gt 25000` check in compile-and-deploy-template.yaml.
-
-## Assumptions that break the cache
-
-- **rustc toolchain version mismatch** between deps-cache image and
-  compile pod — cargo will full-rebuild. Resolution: rebuild the
-  deps-cache image (refresh-deps-cache).
-- **Different `RUSTFLAGS` / `target.<triple>.rustflags`** between
-  image build and live compile (e.g. switching CPU target features)
-  — full rebuild. Resolution: rebuild deps-cache image after the
-  rustflags change is committed.
-- **Cargo.lock SHA divergence** > a few weeks old — most rlibs no
-  longer match. Resolution: rebuild deps-cache image (refresh nightly,
-  or trigger manually).
-- **Linker swap (e.g. mold -> wild)** — link-output is cached at the
-  fingerprint level; changing linker doesn't trigger rebuild but may
-  produce different final binaries. Not a cache invalidation, just a
-  link-time difference.
-
-## When to delete this entirely
-
-If the underlying compile time drops to <30s in cold-start (e.g. via a
-much faster registry or sccache covering 100% of deps), the
-deps-cache complexity is no longer worth maintaining. To remove:
-
-1. Remove the `seed-deps-cache` initContainer block from
-   `compile-and-deploy-template.yaml` (the compile script tolerates
-   an unseeded PVC — cargo just compiles deps from scratch).
-2. Delete `infra/docker/Dockerfile.ci-deps-cache` and
-   `infra/k8s/argo/refresh-deps-cache-template.yaml`.
-3. Delete the published image:
-   `crane delete gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu-with-deps:nightly`.

infra/k8s/argo/alpha-cv-template.yaml

@@ -1,495 +0,0 @@
-# Alpha walk-forward CV workflow.
-#
-# Trains the Mamba2+stacker alpha logits cache on the full 9Q fxcache,
-# then runs `alpha_baseline` 9 times across disjoint 1.9M-bar windows
-# (one per quarter). Each fold trains its own execution-policy DQN
-# from scratch on the front of the window and backtests on the back
-# (purged train/eval split via `--train-frac`).
-#
-# Sequential by design: 9 folds chain in the DAG so a single L40S node
-# stays warm across the run. With --decision-stride 200 and --horizon
-# 1200, each fold's DQN training is ~5-10 min on L40S; total wall ~60-90
-# min for stacker + 9 folds.
-#
-# DAG:
-#   ensure-binary ──> ensure-fxcache ──> stacker-train ──> fold-0 ──> ... ──> fold-8
-#
-# Usage:
-#   argo submit -n foxhunt --from=wftmpl/alpha-cv \
-#     -p commit-sha=HEAD -p git-branch=main -p decision-stride=200
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: alpha-cv
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: alpha-cv
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train
-spec:
-  entrypoint: pipeline
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 28800
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: git-branch
-        value: main
-      - name: cuda-compute-cap
-        value: "89"
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: symbol
-        value: ES.FUT
-      # Match the fxcache build params (cluster's 9Q fxcache was built
-      # with these on 2026-05-16; see commit e7ce4395e).
-      - name: imbalance-bar-threshold
-        value: "20.0"
-      - name: imbalance-bar-ewma-alpha
-        value: "0.1"
-      - name: volume-bar-size
-        value: "100"
-      - name: data-source
-        value: "mbp10"
-      # Stacker training params (mirror local 2Q smoke).
-      - name: stacker-horizon
-        value: "6000"
-      - name: stacker-seq-len
-        value: "32"
-      - name: stacker-hidden-dim
-        value: "64"
-      - name: stacker-state-dim
-        value: "16"
-      - name: stacker-epochs
-        value: "5"
-      - name: stacker-batch-size
-        value: "128"
-      - name: stacker-lr
-        value: "3e-3"
-      - name: stacker-train-frac
-        value: "0.8"
-      # CV / alpha_baseline params — defaults match the validated 2Q
-      # config that produced +1.78 Sharpe at quarter-tick cost.
-      - name: decision-stride
-        value: "200"
-      - name: fold-window
-        value: "1900000"
-      - name: train-frac
-        value: "0.6"
-      - name: window-k
-        value: "16"
-      - name: horizon
-        value: "1200"
-      - name: n-train-par
-        value: "25"
-      - name: n-train-episodes
-        value: "8000"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  templates:
-    # ── DAG ──
-    - name: pipeline
-      dag:
-        tasks:
-          - name: ensure-binary
-            template: ensure-binary
-          - name: stacker-train
-            template: stacker-train
-            dependencies: [ensure-binary]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          - name: fold-0
-            template: alpha-cv-fold
-            dependencies: [stacker-train]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "0"
-                - name: offset
-                  value: "0"
-          - name: fold-1
-            template: alpha-cv-fold
-            dependencies: [fold-0]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "1"
-                - name: offset
-                  value: "1900000"
-          - name: fold-2
-            template: alpha-cv-fold
-            dependencies: [fold-1]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "2"
-                - name: offset
-                  value: "3800000"
-          - name: fold-3
-            template: alpha-cv-fold
-            dependencies: [fold-2]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "3"
-                - name: offset
-                  value: "5700000"
-          - name: fold-4
-            template: alpha-cv-fold
-            dependencies: [fold-3]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "4"
-                - name: offset
-                  value: "7600000"
-          - name: fold-5
-            template: alpha-cv-fold
-            dependencies: [fold-4]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "5"
-                - name: offset
-                  value: "9500000"
-          - name: fold-6
-            template: alpha-cv-fold
-            dependencies: [fold-5]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "6"
-                - name: offset
-                  value: "11400000"
-          - name: fold-7
-            template: alpha-cv-fold
-            dependencies: [fold-6]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "7"
-                - name: offset
-                  value: "13300000"
-          - name: fold-8
-            template: alpha-cv-fold
-            dependencies: [fold-7]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-                - name: fold-idx
-                  value: "8"
-                - name: offset
-                  value: "15200000"
-
-    # ── ensure-binary: compile alpha_train_stacker + alpha_baseline ──
-    - name: ensure-binary
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: RUSTC_WRAPPER
-            value: sccache
-          - name: SCCACHE_DIR
-            value: /cargo-target/sccache
-          - name: SCCACHE_CACHE_SIZE
-            value: "40G"
-          - name: CARGO_INCREMENTAL
-            value: "0"
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: "14"
-            memory: 32Gi
-          limits:
-            cpu: "30"
-            memory: 64Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            BRANCH="{{workflow.parameters.git-branch}}"
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            if [ "$SHA" = "HEAD" ]; then
-              if [ -d "$BUILD/.git" ]; then
-                cd "$BUILD"; git fetch origin; SHA=$(git rev-parse "origin/$BRANCH"); cd /
-              else
-                git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD"
-                git checkout "origin/$BRANCH"; SHA=$(git rev-parse HEAD); cd /
-              fi
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "Resolved SHA: $SHA (short: $SHORT_SHA)"
-
-            BIN_DIR="/data/bin/$SHORT_SHA"
-            BINARIES="alpha_train_stacker alpha_baseline"
-            ALL_CACHED=true
-            for bin in $BINARIES; do
-              [ ! -x "$BIN_DIR/$bin" ] && ALL_CACHED=false && break
-            done
-
-            if [ "$ALL_CACHED" = "true" ]; then
-              echo "=== Cache HIT ==="
-              ls -lh "$BIN_DIR/"
-              echo "$SHORT_SHA" > /tmp/sha
-              exit 0
-            fi
-            echo "=== Cache MISS: compiling alpha binaries for $SHORT_SHA ==="
-
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"; git fetch origin
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$SHA" ]; then
-                git checkout --force "$SHA"; git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD"; git checkout "$SHA"
-            fi
-
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            # alpha_train_stacker lives in ml-alpha, alpha_baseline in ml.
-            echo "Building alpha_train_stacker (ml-alpha)..."
-            cargo build --release -p ml-alpha --example alpha_train_stacker
-            echo "Building alpha_baseline (ml)..."
-            cargo build --release -p ml --features ml/cuda --example alpha_baseline
-
-            mkdir -p "$BIN_DIR"
-            cp "$CARGO_TARGET_DIR/release/examples/alpha_train_stacker" "$BIN_DIR/"
-            cp "$CARGO_TARGET_DIR/release/examples/alpha_baseline" "$BIN_DIR/"
-            strip "$BIN_DIR/"*
-            # alpha_baseline reads config/ml/alpha_fill_coeffs.json as a
-            # required --fill-coeffs input. The source tree's copy travels
-            # with the binaries so the GPU fold pods don't need to clone.
-            cp "$BUILD/config/ml/alpha_fill_coeffs.json" "$BIN_DIR/" || true
-            ls -lh "$BIN_DIR/"
-
-            cd /data/bin
-            ls -1t | tail -n +6 | while read -r old; do
-              echo "Pruning old cache: $old"; rm -rf "$old"
-            done
-            echo "$SHORT_SHA" > /tmp/sha
-
-    # ── stacker-train: train Mamba2+stacker on the 9Q fxcache ──
-    - name: stacker-train
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-        resources:
-          requests:
-            cpu: "8"
-            memory: 32Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "16"
-            memory: 80Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BIN="/data/bin/$SHA/alpha_train_stacker"
-            FXCACHE=$(ls -t /feature-cache/*.fxcache | head -1)
-            ALPHA_OUT="/feature-cache/alpha_logits_cache_9q.bin"
-            echo "Using fxcache: $FXCACHE"
-            echo "Output: $ALPHA_OUT"
-
-            "$BIN" \
-              --fxcache-path "$FXCACHE" \
-              --horizon {{workflow.parameters.stacker-horizon}} \
-              --seq-len {{workflow.parameters.stacker-seq-len}} \
-              --hidden-dim {{workflow.parameters.stacker-hidden-dim}} \
-              --state-dim {{workflow.parameters.stacker-state-dim}} \
-              --epochs {{workflow.parameters.stacker-epochs}} \
-              --batch-size {{workflow.parameters.stacker-batch-size}} \
-              --lr {{workflow.parameters.stacker-lr}} \
-              --train-frac {{workflow.parameters.stacker-train-frac}} \
-              --alpha-cache-out "$ALPHA_OUT"
-            ls -lh "$ALPHA_OUT"
-
-    # ── alpha-cv-fold: run alpha_baseline on one window ──
-    - name: alpha-cv-fold
-      inputs:
-        parameters:
-          - name: sha
-          - name: fold-idx
-          - name: offset
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-        resources:
-          requests:
-            cpu: "8"
-            memory: 32Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "16"
-            memory: 80Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            FOLD={{inputs.parameters.fold-idx}}
-            OFFSET={{inputs.parameters.offset}}
-            BIN="/data/bin/$SHA/alpha_baseline"
-            FXCACHE=$(ls -t /feature-cache/*.fxcache | head -1)
-            ALPHA="/feature-cache/alpha_logits_cache_9q.bin"
-            # alpha_fill_coeffs.json travels with the source tree; the
-            # binary needs it next to itself or via explicit --fill-coeffs.
-            # The ci-builder image bakes it under /opt/foxhunt/config/ml.
-            FILL="/data/bin/$SHA/alpha_fill_coeffs.json"
-            if [ ! -f "$FILL" ]; then
-              # Fall back to copying from the cloned source on the
-              # cargo-target PVC mounted in ensure-binary; we don't have
-              # that PVC here, so embed via fxcache PVC pre-stage.
-              FILL="/feature-cache/alpha_fill_coeffs.json"
-            fi
-            OUT="/feature-cache/cv_results_9fold/fold_${FOLD}.json"
-            mkdir -p /feature-cache/cv_results_9fold
-            echo "Fold $FOLD: offset=$OFFSET out=$OUT"
-
-            "$BIN" \
-              --fxcache-path "$FXCACHE" \
-              --alpha-cache "$ALPHA" \
-              --fill-coeffs "$FILL" \
-              --data-start-offset $OFFSET \
-              --max-snapshots {{workflow.parameters.fold-window}} \
-              --train-frac {{workflow.parameters.train-frac}} \
-              --window-k {{workflow.parameters.window-k}} \
-              --decision-stride {{workflow.parameters.decision-stride}} \
-              --horizon {{workflow.parameters.horizon}} \
-              --n-train-par {{workflow.parameters.n-train-par}} \
-              --n-train-episodes {{workflow.parameters.n-train-episodes}} \
-              --out-path "$OUT"
-            ls -lh "$OUT"

infra/k8s/argo/alpha-perception-template.yaml

@@ -1,484 +0,0 @@
-# Alpha perception trainer workflow.
-#
-# Stacked Mamba2 -> CfC -> heads supervised pretrain on MBP-10 from the
-# training-data PVC. Emits `alpha_train_summary.json` to MinIO-backed
-# feature-cache PVC with per-horizon val AUC for monitoring. Runs on
-# a single L40S; ~30-90 min wall depending on
-# n-train-seqs.
-#
-# DAG:
-#   ensure-binary ──> train
-#
-# Usage:
-#   argo submit -n foxhunt --from=wftmpl/alpha-perception \
-#     -p commit-sha=HEAD -p git-branch=ml-alpha-phase-a
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: alpha-perception
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: alpha-perception
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train
-spec:
-  entrypoint: pipeline
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 14400
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: git-branch
-        value: ml-alpha-phase-a
-      - name: cuda-compute-cap
-        value: "89"
-      - name: gpu-pool
-        value: ci-training-l40s
-      # Trainer hyperparameters (mirror the validated smoke config).
-      - name: epochs
-        value: "5"
-      - name: multi-resolution
-        value: "1:32"
-      - name: mamba2-state-dim
-        value: "16"
-      - name: lr-cfc
-        value: "3e-3"
-      - name: lr-mamba2
-        value: "1e-3"
-      - name: n-train-seqs
-        value: "8000"
-      - name: n-val-seqs
-        value: "1000"
-      - name: seed
-        value: "16962"
-      - name: batch-size
-        value: "1"
-      - name: auto-horizon-weights
-        value: "false"
-      - name: instrument-mode
-        value: "all"
-      - name: early-stop-metric
-        value: "mean_auc"
-      - name: early-stop-patience
-        value: "5"
-      - name: cv-fold
-        value: "0"
-      - name: cv-n-folds
-        value: "1"
-      - name: cv-train-window
-        value: "0"
-      - name: smoothness-base-lambda
-        value: "0.0"
-      - name: kernel-step-trace-enable
-        value: "false"
-      - name: kernel-step-trace-path
-        value: ""
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  templates:
-    # ── DAG ──
-    #
-    # check-cache (tiny alpine pod on platform pool, ~3 sec) inspects
-    # the training-data PVC for /data/bin/$SHA/alpha_train. If present
-    # (cache hit), ensure-binary is SKIPPED via `when:` — the
-    # ~4.8GB ci-builder image pull + sccache compile cycle are avoided
-    # entirely. train still runs, sourcing the binary path from
-    # check-cache's SHA output.
-    #
-    # warmup-gpu fires unconditionally in parallel — triggers the L40S
-    # autoscaler so the node is warm by the time train needs it,
-    # whether or not we compile.
-    - name: pipeline
-      dag:
-        tasks:
-          - name: check-cache
-            template: check-cache
-          - name: ensure-binary
-            template: ensure-binary
-            dependencies: [check-cache]
-            when: "{{tasks.check-cache.outputs.parameters.cache}} == miss"
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.check-cache.outputs.parameters.sha}}"
-          - name: warmup-gpu
-            template: warmup-gpu
-          - name: train
-            template: train
-            dependencies: [check-cache, ensure-binary]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.check-cache.outputs.parameters.sha}}"
-
-    # ── check-cache: probe the training-data PVC for an existing binary ──
-    #
-    # Runs on the platform pool (always up, no autoscaler delay). Tiny
-    # alpine pod, ~3 sec end-to-end. Emits two output parameters:
-    #   - sha   = the short SHA used for binary cache keying
-    #   - cache = "hit" if /data/bin/$SHA/alpha_train exists else "miss"
-    #
-    # Resolves `commit-sha=HEAD` by reading /tmp/head-sha from a
-    # ConfigMap... actually no, simpler: the submission script
-    # (scripts/argo-alpha-perception.sh) pre-resolves HEAD via local
-    # git so commit-sha is always a real SHA in the workflow. This
-    # pod just trims to short SHA and stats the binary.
-    - name: check-cache
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-          - name: cache
-            valueFrom:
-              path: /tmp/cache
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: alpine:3.20
-        command: ["/bin/sh", "-c"]
-        resources:
-          requests:
-            cpu: "50m"
-            memory: 32Mi
-          limits:
-            cpu: "100m"
-            memory: 64Mi
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            if [ "$SHA" = "HEAD" ]; then
-              echo "ERROR: commit-sha cannot be HEAD inside the workflow."
-              echo "       Resolve via the submission script (scripts/argo-alpha-perception.sh)."
-              exit 1
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "$SHORT_SHA" > /tmp/sha
-
-            BIN="/data/bin/$SHORT_SHA/alpha_train"
-            if [ -x "$BIN" ]; then
-              SIZE=$(stat -c %s "$BIN")
-              echo "Cache HIT: $BIN ($SIZE bytes)"
-              echo "hit" > /tmp/cache
-            else
-              echo "Cache MISS: $BIN not present, ensure-binary will compile"
-              ls -lh "/data/bin/" 2>/dev/null | head -10 || echo "  (no /data/bin/ directory yet)"
-              echo "miss" > /tmp/cache
-            fi
-
-    # ── ensure-binary: compile `alpha_train` example via sccache ──
-    # Only runs when check-cache returned cache=miss. Outputs the SHA
-    # for consistency, but the DAG sources SHA from check-cache directly.
-    - name: ensure-binary
-      inputs:
-        parameters:
-          - name: sha
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: RUSTC_WRAPPER
-            value: sccache
-          - name: SCCACHE_DIR
-            value: /cargo-target/sccache
-          - name: SCCACHE_CACHE_SIZE
-            value: "40G"
-          - name: CARGO_INCREMENTAL
-            value: "0"
-          - name: CUDA_COMPUTE_CAP
-            value: "{{workflow.parameters.cuda-compute-cap}}"
-        resources:
-          requests:
-            cpu: "14"
-            memory: 32Gi
-          limits:
-            cpu: "30"
-            memory: 64Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            BRANCH="{{workflow.parameters.git-branch}}"
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            if [ "$SHA" = "HEAD" ]; then
-              if [ -d "$BUILD/.git" ]; then
-                cd "$BUILD"; git fetch origin; SHA=$(git rev-parse "origin/$BRANCH"); cd /
-              else
-                git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD"
-                git checkout "origin/$BRANCH"; SHA=$(git rev-parse HEAD); cd /
-              fi
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "Resolved SHA: $SHA (short: $SHORT_SHA)"
-
-            BIN_DIR="/data/bin/$SHORT_SHA"
-            if [ -x "$BIN_DIR/alpha_train" ]; then
-              echo "=== Cache HIT: $BIN_DIR/alpha_train ==="
-              ls -lh "$BIN_DIR/"
-              echo "$SHORT_SHA" > /tmp/sha
-              exit 0
-            fi
-            echo "=== Cache MISS: compiling alpha_train for $SHORT_SHA ==="
-
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"; git fetch origin
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$SHA" ]; then
-                git checkout --force "$SHA"; git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"; cd "$BUILD"; git checkout "$SHA"
-            fi
-
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-
-            FEATURES_FLAG=""
-            if [ "{{workflow.parameters.kernel-step-trace-enable}}" = "true" ]; then
-              FEATURES_FLAG="--features kernel-step-trace"
-              echo "Building with --features kernel-step-trace"
-              # Bust cache: feature-on binary differs from feature-off at same SHA.
-              if [ -x "$BIN_DIR/alpha_train" ]; then
-                echo "Removing cached binary (feature change requires rebuild)"
-                rm -f "$BIN_DIR/alpha_train"
-              fi
-            fi
-
-            echo "Building alpha_train (ml-alpha)..."
-            cargo build --release $FEATURES_FLAG -p ml-alpha --example alpha_train
-
-            mkdir -p "$BIN_DIR"
-            cp "$CARGO_TARGET_DIR/release/examples/alpha_train" "$BIN_DIR/"
-            strip "$BIN_DIR/alpha_train"
-            ls -lh "$BIN_DIR/"
-
-            # Prune old cache (keep last 5)
-            cd /data/bin
-            ls -1t | tail -n +6 | while read -r old; do
-              echo "Pruning old cache: $old"; rm -rf "$old"
-            done
-            echo "$SHORT_SHA" > /tmp/sha
-
-    # ── warmup-gpu: trigger L40S autoscaler scale-up in parallel ──
-    #
-    # Schedules a tiny CPU-only pod on the gpu-pool's nodeSelector.
-    # Kubernetes sees an unschedulable pod (no node currently in the
-    # pool), autoscaler scales the pool 0 → 1. The pod exits the
-    # instant it lands; the node enters scaledown-grace.
-    #
-    # Cluster autoscaler config (see infra/modules/kapsule/main.tf):
-    #   scale_down_delay_after_add = "10m"  -- 10m before any scaledown consideration
-    #   scale_down_unneeded_time   = "10m"  -- 10m empty before action
-    # Effective grace window is up to ~20m, which always spans
-    # ensure-binary compile (sccache-hit ~10s through cold ~15m).
-    # train lands on a hot node with zero provisioning latency.
-    #
-    # No GPU resource request — that would compete with `train` for
-    # the single GPU and serialise the steps. The nodeSelector +
-    # nvidia.com/gpu toleration are enough to force placement on the
-    # right pool.
-    - name: warmup-gpu
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: alpine:3.20
-        command: ["/bin/sh", "-c"]
-        resources:
-          requests:
-            cpu: "50m"
-            memory: 32Mi
-          limits:
-            cpu: "100m"
-            memory: 64Mi
-        args:
-          - |
-            echo "GPU warmup pod scheduled on $(hostname) — autoscaler scale-up triggered, node now in scaledown-grace window."
-
-    # ── train: run alpha_train on L40S ──
-    - name: train
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CUDA_COMPUTE_CAP
-            value: "{{workflow.parameters.cuda-compute-cap}}"
-        resources:
-          # L40S-1-48G has 8 vCPU (7800m allocatable, after kubelet
-          # overhead) and ~91Gi memory. Request <7800m so the pod
-          # fits alongside the system daemonsets (cilium, csi-node,
-          # nvidia drivers/dcgm/feature-discovery) on the same node.
-          requests:
-            cpu: "6"
-            memory: 16Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "7"
-            memory: 64Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BIN="/data/bin/$SHA/alpha_train"
-            MBP10_DIR="/data/futures-baseline-mbp10/ES.FUT"
-            PREDECODED_DIR="/feature-cache/predecoded"
-            OUT_DIR="/feature-cache/alpha-perception-runs/$SHA"
-            mkdir -p "$OUT_DIR" "$PREDECODED_DIR"
-
-            if [ ! -x "$BIN" ]; then
-              echo "ERROR: binary not found at $BIN"
-              ls -lh /data/bin/ || true
-              exit 1
-            fi
-            if [ ! -d "$MBP10_DIR" ]; then
-              echo "ERROR: MBP-10 data directory not found at $MBP10_DIR"
-              ls -lh /data/ || true
-              exit 1
-            fi
-            echo "Running alpha_train (stacked Mamba2 -> CfC -> heads)"
-            echo "  binary:        $BIN"
-            echo "  mbp10:         $MBP10_DIR ($(find "$MBP10_DIR" -name '*.dbn.zst' | wc -l) files)"
-            echo "  predecoded:    $PREDECODED_DIR"
-            echo "  out:           $OUT_DIR"
-
-            EXTRA_FLAGS=""
-            if [ "{{workflow.parameters.auto-horizon-weights}}" = "true" ]; then
-              EXTRA_FLAGS="$EXTRA_FLAGS --auto-horizon-weights"
-            fi
-            if [ -n "{{workflow.parameters.instrument-mode}}" ]; then
-              EXTRA_FLAGS="$EXTRA_FLAGS --instrument-mode {{workflow.parameters.instrument-mode}}"
-            fi
-
-            TRACE_FLAG=""
-            if [ -n "{{workflow.parameters.kernel-step-trace-path}}" ]; then
-              TRACE_FLAG="--kernel-step-trace {{workflow.parameters.kernel-step-trace-path}}"
-            fi
-
-            "$BIN" \
-              --mbp10-data-dir   "$MBP10_DIR" \
-              --predecoded-dir   "$PREDECODED_DIR" \
-              --out              "$OUT_DIR" \
-              --epochs           {{workflow.parameters.epochs}} \
-              --multi-resolution "{{workflow.parameters.multi-resolution}}" \
-              --mamba2-state-dim {{workflow.parameters.mamba2-state-dim}} \
-              --lr-cfc           {{workflow.parameters.lr-cfc}} \
-              --lr-mamba2        {{workflow.parameters.lr-mamba2}} \
-              --n-train-seqs     {{workflow.parameters.n-train-seqs}} \
-              --n-val-seqs       {{workflow.parameters.n-val-seqs}} \
-              --seed             {{workflow.parameters.seed}} \
-              --batch-size       {{workflow.parameters.batch-size}} \
-              --early-stop-metric   {{workflow.parameters.early-stop-metric}} \
-              --early-stop-patience {{workflow.parameters.early-stop-patience}} \
-              --cv-fold             {{workflow.parameters.cv-fold}} \
-              --cv-n-folds          {{workflow.parameters.cv-n-folds}} \
-              --cv-train-window     {{workflow.parameters.cv-train-window}} \
-              --smoothness-base-lambda {{workflow.parameters.smoothness-base-lambda}} \
-              ${TRACE_FLAG} \
-              $EXTRA_FLAGS
-
-            echo "=== Training complete ==="
-            ls -lh "$OUT_DIR/"
-            echo "--- alpha_train_summary.json ---"
-            cat "$OUT_DIR/alpha_train_summary.json"

infra/k8s/argo/alpha-rl-template.yaml

@@ -1,207 +0,0 @@
-# alpha-rl: single-pod compile + train on GPU.
-#
-# Compiles alpha_rl_train incrementally on the GPU node (~3s warm,
-# ~90s cold) using the cargo-target-cuda PVC, then runs training
-# immediately. No separate compile node, no binary transfer, no
-# fxcache step. Predecoded MBP-10 sidecars live on feature-cache PVC.
-#
-# Usage:
-#   argo submit -n foxhunt --from=wftmpl/alpha-rl \
-#     -p git-branch=ml-alpha-phase-a
-#   argo submit -n foxhunt --from=wftmpl/alpha-rl \
-#     -p git-branch=ml-alpha-phase-a -p n-steps=50000 -p n-backtests=128
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: alpha-rl
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: alpha-rl
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train
-spec:
-  entrypoint: compile-and-train
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 14400
-
-  arguments:
-    parameters:
-      - name: git-branch
-        value: ml-alpha-phase-a
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: n-steps
-        value: "50000"
-      - name: n-backtests
-        value: "128"
-      - name: seed
-        value: "16962"
-      - name: seq-len
-        value: "32"
-      - name: per-capacity
-        value: "32768"
-      - name: instrument-mode
-        value: "all"
-      - name: log-every
-        value: "5000"
-      - name: fold-idx
-        value: "0"
-      - name: n-folds
-        value: "1"
-      - name: n-eval-steps
-        value: "0"
-      - name: nsys-profile
-        value: "false"
-      - name: use-multi-head-policy
-        value: "0"
-      - name: band-enabled
-        value: "0"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  templates:
-    - name: compile-and-train
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CARGO_INCREMENTAL
-            value: "1"
-          - name: FOXHUNT_USE_MULTI_HEAD_POLICY
-            value: "{{workflow.parameters.use-multi-head-policy}}"
-          - name: FOXHUNT_BAND_ENABLED
-            value: "{{workflow.parameters.band-enabled}}"
-        resources:
-          requests:
-            cpu: "4"
-            memory: 16Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "7"
-            memory: 40Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-        args:
-          - |
-            set -e
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
-
-            # SSH
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            nvidia-smi
-
-            # Git
-            BRANCH="{{workflow.parameters.git-branch}}"
-            BUILD="/cargo-target/src"
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            git config --global --add safe.directory "$BUILD"
-
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              git fetch origin
-              git checkout --force "origin/$BRANCH"
-              git clean -fd
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git checkout "origin/$BRANCH"
-            fi
-            SHA=$(git rev-parse --short=9 HEAD)
-            echo "=== Branch: $BRANCH  SHA: $SHA ==="
-
-            # Auto-detect GPU compute capability for all crates' build.rs
-            export CUDA_COMPUTE_CAP=$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader | head -1 | tr -d '.')
-            echo "=== GPU arch: sm_${CUDA_COMPUTE_CAP} ==="
-
-            # Clear stale build artifacts when arch changes
-            rm -rf ${CARGO_TARGET_DIR}/build/ml-alpha-* ${CARGO_TARGET_DIR}/build/ml-core-*
-
-            # Compile (~90s cold, ~3s warm after first run)
-            echo "=== Compile ==="
-            time cargo build --release -p ml-alpha --example alpha_rl_train 2>&1
-
-            # Train
-            OUT="/feature-cache/alpha-rl-runs/$SHA/fold{{workflow.parameters.fold-idx}}"
-            mkdir -p "$OUT"
-            echo "=== Train on $(nvidia-smi --query-gpu=name --format=csv,noheader) ==="
-            TRAIN_BIN="${CARGO_TARGET_DIR}/release/examples/alpha_rl_train"
-            if [ "{{workflow.parameters.nsys-profile}}" = "true" ]; then
-              TRAIN_CMD="nsys profile -o $OUT/nsys_trace --stats=true --force-overwrite=true $TRAIN_BIN"
-            else
-              TRAIN_CMD="stdbuf -oL $TRAIN_BIN"
-            fi
-            $TRAIN_CMD \
-              --mbp10-data-dir /data/futures-baseline-mbp10/ES.FUT \
-              --predecoded-dir /feature-cache/predecoded \
-              --out "$OUT" \
-              --n-steps {{workflow.parameters.n-steps}} \
-              --seq-len {{workflow.parameters.seq-len}} \
-              --n-backtests {{workflow.parameters.n-backtests}} \
-              --per-capacity {{workflow.parameters.per-capacity}} \
-              --seed {{workflow.parameters.seed}} \
-              --instrument-mode "{{workflow.parameters.instrument-mode}}" \
-              --fold-idx {{workflow.parameters.fold-idx}} \
-              --n-folds {{workflow.parameters.n-folds}} \
-              --n-eval-steps {{workflow.parameters.n-eval-steps}} \
-              --log-every {{workflow.parameters.log-every}}
-
-            echo "=== Complete: $OUT ==="
-            ls -lh "$OUT/"
-            if [ -f "$OUT/alpha_rl_train_summary.json" ]; then
-              cat "$OUT/alpha_rl_train_summary.json"
-            fi

infra/k8s/argo/archive-rbac.yaml

@@ -1,34 +0,0 @@
-# RBAC for Argo workflow archive — grants controller and server access
-# to the PostgreSQL archive secret and MinIO credentials.
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: argo-archive-secrets
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: argo-workflows
-rules:
-- apiGroups: [""]
-  resources: ["secrets"]
-  resourceNames: ["argo-postgres-config", "minio-credentials"]
-  verbs: ["get"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: argo-archive-secrets
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: argo-workflows
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: argo-archive-secrets
-subjects:
-- kind: ServiceAccount
-  name: argo-workflows-workflow-controller
-  namespace: foxhunt
-- kind: ServiceAccount
-  name: argo-workflows-server
-  namespace: foxhunt

infra/k8s/argo/argo-workflow-netpol.yaml

@@ -1,518 +0,0 @@
-# Base egress for ALL Argo workflow pods: DNS, K8s API, MinIO (logging), Mattermost (notifications).
-# Applies to every pod with app.kubernetes.io/part-of: foxhunt.
-# Component-specific policies below add extra rules (GitLab SSH, external HTTPS, etc.).
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-base-egress
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/part-of: foxhunt
-  policyTypes:
-    - Egress
-  egress:
-    # DNS
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor reports task results
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # MinIO — log archival (Argo wait sidecar)
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
----
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-training-workflow
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: training-workflow
-  policyTypes:
-    - Egress
-  egress:
-    # DNS — required for hostname resolution (minio, gitlab, etc.)
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor reports task results
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # HTTPS (external registries, etc.)
-    - ports:
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git clone
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # GitLab API — binary fetch from packages
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice
-    # MinIO — log archival + result upload
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Pushgateway — Prometheus metrics
-    - ports:
-        - port: 9091
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: pushgateway
-    # Tempo — OTLP traces
-    - ports:
-        - port: 4317
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: tempo
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
-    # Container registry — image pulls
-    - ports:
-        - port: 5000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: registry
----
-# Compile workflow pods: need GitLab SSH, MinIO, k8s API, external HTTPS (cargo deps)
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-compile-workflow
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: compile
-  policyTypes:
-    - Egress
-  egress:
-    # DNS — resolve external hostnames (crates.io, github.com)
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # External HTTPS — cargo fetches crates.io, github.com (candle git dep)
-    - ports:
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git clone (Tailscale IP + cluster service)
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # MinIO — binary upload
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
----
-# Compile-and-deploy workflow pods: need GitLab HTTP (tagging/releases), MinIO,
-# k8s API (deploy rollouts + Argo executor), DNS, GitLab SSH, external HTTPS.
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-compile-and-deploy-workflow
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: compile-and-deploy
-  policyTypes:
-    - Egress
-  egress:
-    # DNS
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor + kubectl rollout
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # External HTTPS — cargo fetches, kubectl download
-    - ports:
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git clone
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # GitLab webservice — CalVer tagging, releases, package uploads
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice
-    # MinIO — binary upload + log archival
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
----
-# Compile-and-train workflow pods: compile step needs GitLab SSH + external HTTPS + DNS,
-# training steps need GitLab HTTP (package fetch/upload), MinIO, Tempo, k8s API.
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-train-workflow
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: train
-  policyTypes:
-    - Egress
-  egress:
-    # DNS
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # External HTTPS — cargo fetches crates.io, github.com (candle git dep)
-    - ports:
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git clone
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # GitLab webservice — package upload/download
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice
-    # MinIO — binary upload
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Tempo — OTLP traces
-    - ports:
-        - port: 4317
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: tempo
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
----
-# CI pipeline pods (detect-changes + compile-service inline templates)
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-ci-pipeline
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: ci-pipeline
-  policyTypes:
-    - Egress
-  egress:
-    # DNS — resolve external hostnames
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API — Argo executor
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # External HTTP/HTTPS — Kaniko base image pulls, apt-get, cargo fetches
-    - ports:
-        - port: 80
-          protocol: TCP
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git clone (compile-service uses ssh://git@git.fxhnt.ai)
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # MinIO — binary upload from compile-service
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # GitLab registry — Kaniko image push
-    - ports:
-        - port: 5000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: registry
-    # GitLab webservice — JWT auth for registry tokens
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice
-    # Mattermost — exit handler webhook notifications
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
----
-# ── sanitizer-test + nsys-test + smoke-test workflows: same egress as ci-pipeline ──
-# (git fetch over SSH, MinIO log archive + nsys-rep upload, registry image pull,
-# external HTTPS for mc download)
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-sanitizer-nsys-workflow
-  namespace: foxhunt
-spec:
-  podSelector:
-    matchExpressions:
-      - key: app.kubernetes.io/component
-        operator: In
-        values:
-          - sanitizer-test
-          - nsys-test
-          - smoke-test
-  policyTypes:
-    - Egress
-  egress:
-    # DNS
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # External HTTPS — mc binary download from dl.min.io
-    - ports:
-        - port: 80
-          protocol: TCP
-        - port: 443
-          protocol: TCP
-    # GitLab SSH — git fetch
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # MinIO — log archive + nsys profile upload
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # GitLab registry — ci-builder image pull
-    - ports:
-        - port: 5000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: registry
-    # GitLab webservice — JWT auth for registry tokens
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice

infra/k8s/argo/build-ci-image-template.yaml

@@ -1,121 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: build-ci-image
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: build-ci-image
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  activeDeadlineSeconds: 3600
-  serviceAccountName: argo-workflow
-  entrypoint: build
-  podMetadata:
-    labels:
-      app.kubernetes.io/component: ci-pipeline
-      app.kubernetes.io/part-of: foxhunt
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: dockerfile
-        value: Dockerfile.ci-builder
-      - name: image-name
-        value: ci-builder
-
-  templates:
-    # Single-pod build: init container clones repo, kaniko builds image.
-    # Uses emptyDir — no PVC needed, works when called via templateRef.
-    - name: build
-      inputs:
-        parameters:
-          - name: commit-sha
-            value: "{{workflow.parameters.commit-sha}}"
-          - name: dockerfile
-            value: "{{workflow.parameters.dockerfile}}"
-          - name: image-name
-            value: "{{workflow.parameters.image-name}}"
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      volumes:
-        - name: workspace
-          emptyDir: {}
-        - name: git-ssh-key
-          secret:
-            secretName: argo-git-ssh-key
-            defaultMode: 256
-        - name: registry-auth
-          secret:
-            secretName: gitlab-registry
-            items:
-              - key: .dockerconfigjson
-                path: config.json
-      initContainers:
-        - name: git-clone
-          image: alpine/git:latest
-          command: ["/bin/sh", "-c"]
-          resources:
-            requests:
-              cpu: 200m
-              memory: 256Mi
-            limits:
-              cpu: "1"
-              memory: 512Mi
-          volumeMounts:
-            - name: git-ssh-key
-              mountPath: /etc/git-ssh
-              readOnly: true
-            - name: workspace
-              mountPath: /workspace
-          args:
-            - |
-              set -ex
-              mkdir -p /root/.ssh
-              cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
-              chmod 600 /root/.ssh/id_ed25519
-              printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > /root/.ssh/config
-              chmod 600 /root/.ssh/config
-
-              SHA="{{inputs.parameters.commit-sha}}"
-              REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-              git clone --no-checkout --filter=blob:none "$REPO" /workspace/src
-              cd /workspace/src
-              git checkout "$SHA"
-              echo "Checked out $(git rev-parse --short HEAD)"
-      container:
-        image: gcr.io/kaniko-project/executor:debug
-        command: ["/busybox/sh", "-c"]
-        env:
-          - name: DOCKER_CONFIG
-            value: /kaniko/.docker
-        resources:
-          requests:
-            cpu: "4"
-            memory: 8Gi
-          limits:
-            cpu: "8"
-            memory: 16Gi
-        volumeMounts:
-          - name: registry-auth
-            mountPath: /kaniko/.docker
-            readOnly: true
-          - name: workspace
-            mountPath: /workspace
-        args:
-          - |
-            /kaniko/executor \
-              --context=/workspace/src \
-              --dockerfile=/workspace/src/infra/docker/{{inputs.parameters.dockerfile}} \
-              --destination=gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/{{inputs.parameters.image-name}}:latest \
-              --insecure-registry=gitlab-registry.foxhunt.svc.cluster.local:5000 \
-              --skip-tls-verify-registry=gitlab-registry.foxhunt.svc.cluster.local:5000 \
-              --cache=true \
-              --cache-repo=gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/cache \
-              --snapshot-mode=redo

infra/k8s/argo/cargo-target-cuda-test-pvc.yaml

@@ -1,16 +0,0 @@
-# infra/k8s/argo/cargo-target-cuda-test-pvc.yaml
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: cargo-target-cuda-test
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: cargo-target
-    app.kubernetes.io/component: ci-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 30Gi

infra/k8s/argo/cargo-target-pvcs.yaml

@@ -1,43 +0,0 @@
-# Persistent cargo target directories for incremental compilation.
-#
-# Why: sccache can't cache workspace rlib crates (109 non-cacheable per build).
-# Persisting target/ lets cargo's incremental compilation skip unchanged crates,
-# reducing typical CI builds from ~20 min (full rebuild) to ~2-3 min.
-#
-# Two separate PVCs because compile-services (no cuda feature) and
-# compile-training (cuda feature) produce incompatible artifacts.
-#
-# NOTE: Only one workflow should use each PVC at a time. ci-pipeline and
-# compile-and-train must not run their compile steps concurrently.
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: cargo-target-cpu
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: cargo-target
-    app.kubernetes.io/component: ci-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 30Gi
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: cargo-target-cuda
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: cargo-target
-    app.kubernetes.io/component: ci-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 30Gi

infra/k8s/argo/ci-deploy-rbac.yaml

@@ -1,40 +0,0 @@
-# RBAC for CI pipeline — deploy step + self-apply Argo templates
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: ci-deploy
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-rules:
-  - apiGroups: ["apps"]
-    resources: [deployments]
-    verbs: [get, list, patch]
-  - apiGroups: ["argoproj.io"]
-    resources: [workflowtemplates, eventsources, sensors, eventbus]
-    verbs: [get, list, create, update, patch]
-  - apiGroups: [""]
-    resources: [services, configmaps]
-    verbs: [get, list, create, update, patch]
-  - apiGroups: ["networking.k8s.io"]
-    resources: [networkpolicies]
-    verbs: [get, list, create, update, patch]
-  - apiGroups: ["rbac.authorization.k8s.io"]
-    resources: [roles, rolebindings]
-    verbs: [get, list, create, update, patch, bind, escalate]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: ci-deploy
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: Role
-  name: ci-deploy
-subjects:
-  - kind: ServiceAccount
-    name: argo-workflow
-    namespace: foxhunt

infra/k8s/argo/ci-pipeline-template.yaml

@@ -1,685 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: ci-pipeline
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: ci-pipeline
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  activeDeadlineSeconds: 7200
-  serviceAccountName: argo-workflow
-  entrypoint: pipeline
-  onExit: notify-result
-  podMetadata:
-    labels:
-      app.kubernetes.io/component: ci-pipeline
-      app.kubernetes.io/part-of: foxhunt
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: registry-auth
-      secret:
-        secretName: gitlab-registry
-        optional: true
-        items:
-          - key: .dockerconfigjson
-            path: config.json
-    - name: cargo-target-cpu
-      persistentVolumeClaim:
-        claimName: cargo-target-cpu
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: commits-json
-        value: "[]"
-  templates:
-    # ── DAG: orchestrate pipeline ──
-    # Policy (user directive): push-triggered runs are limited to infrastructure
-    # sync work — docker image rebuilds + argo-template apply + terragrunt apply.
-    # These run automatically when the relevant paths change:
-    #   - rebuild-ci-builder* / rebuild-runtime / rebuild-training-runtime: gated on
-    #     `detect-changes.docker-images == true` (infra/docker/)
-    #   - apply-argo-templates: gated on `detect-changes.needs-argo-templates` (infra/k8s/argo/)
-    #   - terragrunt-apply:     gated on `detect-changes.needs-infra` (infra/live/, infra/modules/)
-    # All other tasks (test-gate, build-web-dashboard, gpu-test) are `when: "false"` —
-    # they run via manual script invocation (`argo submit` / `./scripts/argo-*.sh`)
-    # or the manual `/compile-deploy` webhook. Training + compile/deploy workflows
-    # are separate templates, always manual.
-    - name: pipeline
-      dag:
-        tasks:
-          - name: detect-changes
-            template: detect-changes
-
-          - name: build-web-dashboard
-            depends: "detect-changes"
-            template: build-web-dashboard
-            # Policy: push-triggered runs are docker image rebuilds only.
-            # Dashboard builds run via manual `argo submit` (scripts).
-            when: "false"
-
-          - name: rebuild-ci-builder
-            depends: "detect-changes"
-            templateRef:
-              name: build-ci-image
-              template: build
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-                - name: dockerfile
-                  value: Dockerfile.ci-builder
-                - name: image-name
-                  value: ci-builder
-            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
-
-          - name: rebuild-ci-builder-cpu
-            depends: "detect-changes"
-            templateRef:
-              name: build-ci-image
-              template: build
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-                - name: dockerfile
-                  value: Dockerfile.ci-builder-cpu
-                - name: image-name
-                  value: ci-builder-cpu
-            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
-
-          - name: rebuild-runtime
-            depends: "detect-changes"
-            templateRef:
-              name: build-ci-image
-              template: build
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-                - name: dockerfile
-                  value: Dockerfile.foxhunt-runtime
-                - name: image-name
-                  value: foxhunt-runtime
-            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
-
-          - name: rebuild-training-runtime
-            depends: "detect-changes"
-            templateRef:
-              name: build-ci-image
-              template: build
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-                - name: dockerfile
-                  value: Dockerfile.foxhunt-training-runtime
-                - name: image-name
-                  value: foxhunt-training-runtime
-            when: "{{tasks.detect-changes.outputs.parameters.docker-images}} == true"
-
-          - name: test-gate
-            depends: "detect-changes && (rebuild-ci-builder-cpu.Succeeded || rebuild-ci-builder-cpu.Skipped)"
-            template: test-gate
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-            # Policy: push-triggered runs are docker image rebuilds only.
-            # Cargo test + clippy quality gate runs via manual `argo submit`
-            # (scripts/argo-test.sh or CI-driven `argo submit --from=wftmpl/ci-pipeline`).
-            when: "false"
-
-          - name: apply-argo-templates
-            depends: "detect-changes.Succeeded && (rebuild-ci-builder-cpu.Succeeded || rebuild-ci-builder-cpu.Skipped)"
-            template: apply-argo-templates
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-            when: "{{tasks.detect-changes.outputs.parameters.needs-argo-templates}} == true"
-
-          - name: terragrunt-apply
-            depends: "detect-changes.Succeeded && (rebuild-ci-builder-cpu.Succeeded || rebuild-ci-builder-cpu.Skipped)"
-            template: terragrunt-apply
-            arguments:
-              parameters:
-                - name: commit-sha
-                  value: "{{workflow.parameters.commit-sha}}"
-            when: "{{tasks.detect-changes.outputs.parameters.needs-infra}} == true"
-
-          # GPU tests: manual only (argo submit --from=wftmpl/gpu-test-pipeline).
-          # Auto-trigger disabled — H100 nodes are expensive, run on demand.
-          - name: gpu-test
-            depends: "detect-changes"
-            template: submit-gpu-test
-            when: "false"
-
-    # ── detect-changes: classify changed files to gate downstream steps ──
-    - name: detect-changes
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: COMMITS_JSON
-            value: "{{workflow.parameters.commits-json}}"
-        resources:
-          requests:
-            cpu: 100m
-            memory: 128Mi
-          limits:
-            cpu: 500m
-            memory: 256Mi
-        args:
-          - |
-            set -e
-            cat <<'SCRIPT' > /tmp/detect.sh
-            #!/bin/sh
-            set -e
-
-            # Extract changed file paths from webhook JSON using grep+sed (no jq dependency).
-            CHANGED_FILES=$(echo "$COMMITS_JSON" \
-              | grep -oE '"(added|modified|removed)":\[[^]]*\]' \
-              | sed 's/"added"://;s/"modified"://;s/"removed"://' \
-              | tr ',' '\n' | tr -d '[]"' | sed '/^$/d' | sort -u \
-              || echo "")
-
-            if [ -z "$CHANGED_FILES" ]; then
-              echo "No changed files detected — triggering full rebuild"
-              CHANGED_FILES="Cargo.toml"
-            fi
-
-            echo "=== Changed files ==="
-            echo "$CHANGED_FILES"
-            echo "===================="
-
-            # --- Helper: check if any changed file matches a set of path prefixes ---
-            check_paths() {
-              local patterns="$1"
-              for file in $CHANGED_FILES; do
-                for pattern in $patterns; do
-                  case "$file" in
-                    ${pattern}*) echo "true"; return ;;
-                  esac
-                done
-              done
-              echo "false"
-            }
-
-            # --- Boolean gates for downstream DAG tasks ---
-            NEEDS_CODE=$(check_paths "Cargo.toml Cargo.lock crates/ services/ bin/fxt/")
-            NEEDS_DASHBOARD=$(check_paths "web-dashboard/")
-            DOCKER_IMAGES=$(check_paths "infra/docker/")
-            NEEDS_INFRA=$(check_paths "infra/live/ infra/modules/")
-            NEEDS_ARGO_TEMPLATES=$(check_paths "infra/k8s/argo/")
-            ML_CHANGED=$(check_paths "crates/ml crates/ml-dqn crates/ml-core crates/ml-regime crates/ml-features crates/ml-supervised crates/ml-ppo crates/ml-ensemble crates/ml-hyperopt crates/ml-labeling config/training config/gpu")
-
-            echo "=== Build decisions ==="
-            echo "needs-code:        $NEEDS_CODE"
-            echo "needs-dashboard:   $NEEDS_DASHBOARD"
-            echo "docker-images:     $DOCKER_IMAGES"
-            echo "needs-infra:       $NEEDS_INFRA"
-            echo "needs-argo:        $NEEDS_ARGO_TEMPLATES"
-            echo "ml-changed:        $ML_CHANGED"
-            echo "======================"
-
-            mkdir -p /tmp/outputs
-            echo -n "$NEEDS_CODE"        > /tmp/outputs/needs-code
-            echo -n "$NEEDS_DASHBOARD"   > /tmp/outputs/needs-dashboard
-            echo -n "$DOCKER_IMAGES"     > /tmp/outputs/docker-images
-            echo -n "$NEEDS_INFRA"       > /tmp/outputs/needs-infra
-            echo -n "$NEEDS_ARGO_TEMPLATES" > /tmp/outputs/needs-argo-templates
-            echo -n "$ML_CHANGED"          > /tmp/outputs/ml-changed
-            SCRIPT
-            chmod +x /tmp/detect.sh
-            /tmp/detect.sh
-      outputs:
-        parameters:
-          - name: needs-code
-            valueFrom:
-              path: /tmp/outputs/needs-code
-          - name: needs-dashboard
-            valueFrom:
-              path: /tmp/outputs/needs-dashboard
-          - name: docker-images
-            valueFrom:
-              path: /tmp/outputs/docker-images
-          - name: needs-infra
-            valueFrom:
-              path: /tmp/outputs/needs-infra
-          - name: needs-argo-templates
-            valueFrom:
-              path: /tmp/outputs/needs-argo-templates
-          - name: ml-changed
-            valueFrom:
-              path: /tmp/outputs/ml-changed
-
-    # ── test-gate: clippy + cargo test quality gate ──
-    # Uses ci-builder (with CUDA toolkit) because cudarc 0.19 requires nvcc at build time.
-    - name: test-gate
-      inputs:
-        parameters:
-          - name: commit-sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: k8s.scaleway.com/pool-name
-          operator: Equal
-          value: ci-compile-cpu
-          effect: NoSchedule
-      sidecars:
-        - name: redis
-          image: redis:7-alpine
-          command: ["redis-server", "--save", "", "--appendonly", "no"]
-          ports:
-            - containerPort: 6379
-          resources:
-            requests:
-              cpu: 100m
-              memory: 64Mi
-            limits:
-              cpu: 500m
-              memory: 128Mi
-          readinessProbe:
-            tcpSocket:
-              port: 6379
-            initialDelaySeconds: 1
-            periodSeconds: 1
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: REDIS_URL
-            value: "redis://localhost:6379"
-        resources:
-          requests:
-            cpu: "14"
-            memory: 16Gi
-          limits:
-            cpu: "30"
-            memory: 32Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cpu
-            mountPath: /cargo-target
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.commit-sha}}"
-
-            # SSH setup
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            WORKSPACE="/cargo-target/src"
-
-            git config --global --add safe.directory "$WORKSPACE"
-
-            if [ -d "$WORKSPACE/.git" ]; then
-              cd "$WORKSPACE"
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" = "$SHA" ]; then
-                echo "=== Already at $SHA ==="
-              else
-                git fetch origin
-                git checkout --force "$SHA"
-                git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$WORKSPACE"
-              cd "$WORKSPACE"
-              git checkout "$SHA"
-            fi
-
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-
-            echo "=== Waiting for Redis sidecar ==="
-            for i in 1 2 3 4 5; do
-              if printf 'PING\r\n' | nc -w1 127.0.0.1 6379 2>/dev/null | grep -q PONG; then
-                echo "Redis ready"
-                break
-              fi
-              sleep 1
-            done
-
-            echo "=== Running clippy (lib targets) ==="
-            cargo clippy --workspace --lib -- -D warnings 2>&1 | tee /cargo-target/clippy.log
-
-            echo "=== Running tests (lib only, no integration) ==="
-            set +e
-            cargo test --workspace --lib 2>&1 | tee /cargo-target/test-output.log
-            TEST_EXIT=$?
-            set -e
-            if [ "$TEST_EXIT" -ne 0 ]; then
-              echo "=== TEST FAILURES ==="
-              grep -A5 'FAILED\|panicked\|test result: FAILED\|error\[' /cargo-target/test-output.log || true
-              exit "$TEST_EXIT"
-            fi
-
-            echo "=== Test gate passed ==="
-
-    # ── build-web-dashboard: npm build + upload to MinIO ──
-    - name: build-web-dashboard
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: node:22-alpine
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: MINIO_ACCESS_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: access-key
-          - name: MINIO_SECRET_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: secret-key
-        resources:
-          requests:
-            cpu: "1"
-            memory: 1Gi
-          limits:
-            cpu: "2"
-            memory: 2Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-
-            # Git clone
-            apk add --no-cache git openssh rclone
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            WORKSPACE="/tmp/workspace"
-            git clone --no-checkout --filter=blob:none "$REPO" "$WORKSPACE"
-            cd "$WORKSPACE"
-            git checkout "$SHA"
-
-            cd "$WORKSPACE/web-dashboard"
-            echo "=== Building web dashboard ==="
-            npm ci
-            npm run build
-
-            echo "=== Uploading to MinIO ==="
-            rclone copy dist/ :s3:foxhunt-binaries/web-dashboard/ \
-              --s3-provider=Minio \
-              --s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
-              --s3-access-key-id="${MINIO_ACCESS_KEY}" \
-              --s3-secret-access-key="${MINIO_SECRET_KEY}" \
-              --s3-no-check-bucket
-
-            echo "=== Web dashboard build + upload done ==="
-
-    # ── terragrunt-apply: apply infra changes on main push ──
-    - name: terragrunt-apply
-      inputs:
-        parameters:
-          - name: commit-sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      serviceAccountName: argo-workflow
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu:latest
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: GITLAB_TOKEN
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-          - name: TF_HTTP_USERNAME
-            value: root
-          - name: TF_HTTP_PASSWORD
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-          - name: SCW_ACCESS_KEY
-            valueFrom:
-              secretKeyRef:
-                name: scaleway-credentials
-                key: access-key
-          - name: SCW_SECRET_KEY
-            valueFrom:
-              secretKeyRef:
-                name: scaleway-credentials
-                key: secret-key
-          - name: SCW_DEFAULT_PROJECT_ID
-            valueFrom:
-              secretKeyRef:
-                name: scaleway-credentials
-                key: project-id
-          - name: GITLAB_TF_STATE_URL
-            value: "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-        resources:
-          requests:
-            cpu: 200m
-            memory: 256Mi
-          limits:
-            cpu: "1"
-            memory: 512Mi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-        args:
-          - |
-            set -e
-
-            # Clone repo
-            mkdir -p /root/.ssh
-            cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
-            chmod 600 /root/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > /root/.ssh/config
-            chmod 600 /root/.ssh/config
-
-            SHA="{{inputs.parameters.commit-sha}}"
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            git clone --no-checkout --filter=blob:none "$REPO" /workspace/src
-            cd /workspace/src
-            git checkout "$SHA"
-            echo "Checked out $(git rev-parse --short HEAD)"
-
-            tofu --version
-            terragrunt --version
-
-            # Apply each terragrunt module
-            for module in block-storage dns public-gateway kapsule; do
-              MODULE_DIR="infra/live/production/${module}"
-              if [ -d "$MODULE_DIR" ]; then
-                echo "=== Terragrunt plan: ${module} ==="
-                cd "/workspace/src/${MODULE_DIR}"
-                terragrunt init --non-interactive -reconfigure
-                OUTPUT=$(terragrunt plan --non-interactive -detailed-exitcode 2>&1) || EXITCODE=$?
-                EXITCODE=${EXITCODE:-0}
-
-                if [ "$EXITCODE" -eq 2 ]; then
-                  echo "=== Terragrunt plan output: ${module} ==="
-                  echo "$OUTPUT"
-                  echo "=== Terragrunt apply: ${module} ==="
-                  terragrunt apply --non-interactive -auto-approve
-                elif [ "$EXITCODE" -eq 0 ]; then
-                  echo "=== No changes for ${module} ==="
-                else
-                  echo "=== ERROR planning ${module} ==="
-                  echo "$OUTPUT"
-                  exit 1
-                fi
-                cd /workspace/src
-              fi
-            done
-
-            echo "=== Terragrunt apply complete ==="
-
-    # ── apply-argo-templates: self-apply WorkflowTemplates, EventSources, Sensors ──
-    - name: apply-argo-templates
-      inputs:
-        parameters:
-          - name: commit-sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      serviceAccountName: argo-workflow
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu:latest
-        command: ["/bin/sh", "-c"]
-        resources:
-          requests:
-            cpu: 100m
-            memory: 128Mi
-          limits:
-            cpu: 500m
-            memory: 256Mi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-        args:
-          - |
-            set -e
-
-            # Clone repo at commit
-            mkdir -p /root/.ssh
-            cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
-            chmod 600 /root/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > /root/.ssh/config
-            chmod 600 /root/.ssh/config
-
-            SHA="{{inputs.parameters.commit-sha}}"
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            git clone --no-checkout --filter=blob:none "$REPO" /workspace/src
-            cd /workspace/src
-            git checkout "$SHA"
-            echo "Checked out $(git rev-parse --short HEAD)"
-
-            # Apply all Argo WorkflowTemplates
-            for f in infra/k8s/argo/*-template.yaml; do
-              if [ -f "$f" ]; then
-                echo "=== Applying $(basename $f) ==="
-                kubectl -n foxhunt apply -f "$f"
-              fi
-            done
-
-            # Apply EventSources and Sensors
-            for f in infra/k8s/argo/events/*.yaml; do
-              if [ -f "$f" ]; then
-                echo "=== Applying $(basename $f) ==="
-                kubectl -n foxhunt apply -f "$f"
-              fi
-            done
-
-            # Apply RBAC and NetworkPolicies
-            kubectl -n foxhunt apply -f infra/k8s/argo/ci-deploy-rbac.yaml
-            kubectl -n foxhunt apply -f infra/k8s/argo/argo-workflow-netpol.yaml
-
-            echo "=== Argo templates applied ==="
-
-    # ── submit-gpu-test: launch GPU test workflow as a child workflow ──
-    - name: submit-gpu-test
-      # Argo executor (main + wait) needs 256Mi to track large child workflow status JSON.
-      # Default 64Mi causes OOMKilled when child workflow has many parameters.
-      podSpecPatch: '{"containers":[{"name":"main","resources":{"requests":{"memory":"128Mi"},"limits":{"memory":"256Mi"}}},{"name":"wait","resources":{"requests":{"memory":"128Mi"},"limits":{"memory":"256Mi"}}}]}'
-      resource:
-        action: create
-        setOwnerReference: true
-        successCondition: status.phase == Succeeded
-        failureCondition: status.phase in (Failed, Error)
-        manifest: |
-          apiVersion: argoproj.io/v1alpha1
-          kind: Workflow
-          metadata:
-            generateName: gpu-test-on-push-
-            namespace: foxhunt
-          spec:
-            workflowTemplateRef:
-              name: gpu-test-pipeline
-            arguments:
-              parameters:
-                - name: commit-ref
-                  value: "{{workflow.parameters.commit-sha}}"
-                - name: models
-                  value: "dqn,ppo,tft"
-
-    # ── notify-result: post workflow outcome to Mattermost ──
-    - name: notify-result
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: WEBHOOK_URL
-            valueFrom:
-              secretKeyRef:
-                name: notification-webhook
-                key: webhook-url
-                optional: true
-        resources:
-          requests:
-            cpu: 50m
-            memory: 32Mi
-          limits:
-            cpu: 100m
-            memory: 64Mi
-        args:
-          - |
-            STATUS="{{workflow.status}}"
-            NAME="{{workflow.name}}"
-
-            if [ -z "$WEBHOOK_URL" ] || echo "$WEBHOOK_URL" | grep -q "PLACEHOLDER"; then
-              echo "No webhook configured, skipping notification"
-              exit 0
-            fi
-
-            if [ "$STATUS" = "Succeeded" ]; then
-              EMOJI=":white_check_mark:"
-            else
-              EMOJI=":x:"
-            fi
-
-            PAYLOAD="{\"username\":\"Argo CI\",\"text\":\"${EMOJI} **${NAME}** — ${STATUS} ({{workflow.duration}}s)\"}"
-
-            curl -sf -X POST -H 'Content-Type: application/json' \
-              -d "$PAYLOAD" "$WEBHOOK_URL" || echo "WARN: webhook post failed"

infra/k8s/argo/compile-and-deploy-template.yaml

@@ -1,614 +0,0 @@
-# Compile-and-Deploy: manual workflow for building and deploying service binaries.
-#
-# DAG:
-#   create-tag ──> compile-services ──> upload-release ──> deploy-services
-#
-# Image builds handled by ci-pipeline, not here.
-# Training compilation lives in compile-and-train-template.yaml.
----
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: compile-and-deploy
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: compile-and-deploy
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: pipeline
-  onExit: notify-result
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: compile-and-deploy
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 3600  # 1 hour (compile + deploy)
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: service-packages
-        value: "api trading-service backtesting-service trading-agent-service broker-gateway data-acquisition-service"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: cargo-target-cpu
-      persistentVolumeClaim:
-        claimName: cargo-target-cpu
-    - name: sccache
-      persistentVolumeClaim:
-        claimName: sccache-cpu
-
-  templates:
-    - name: pipeline
-      dag:
-        tasks:
-          - name: create-tag
-            template: create-tag
-          - name: compile-services
-            depends: "create-tag"
-            template: compile-services
-            arguments:
-              parameters:
-                - name: tag
-                  value: "{{tasks.create-tag.outputs.parameters.tag}}"
-                - name: service-packages
-                  value: "{{workflow.parameters.service-packages}}"
-          - name: upload-release
-            depends: "compile-services"
-            template: upload-release
-            arguments:
-              parameters:
-                - name: tag
-                  value: "{{tasks.create-tag.outputs.parameters.tag}}"
-          - name: deploy-services
-            depends: "upload-release"
-            template: deploy-services
-            arguments:
-              parameters:
-                - name: tag
-                  value: "{{tasks.create-tag.outputs.parameters.tag}}"
-                - name: deploy-list
-                  value: "{{workflow.parameters.service-packages}}"
-
-    # ── create-tag: CalVer auto-tag on code changes ──
-    - name: create-tag
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            cpu: 200m
-            memory: 128Mi
-        args:
-          - |
-            set -e
-            GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-            PROJECT_ID=1
-            SHA="{{workflow.parameters.commit-sha}}"
-
-            # Compute CalVer prefix: vYYYY.MM
-            PREFIX="v$(date +%Y.%m)"
-
-            # Query existing tags for this month
-            TAGS=$(curl -sf \
-              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags?search=${PREFIX}" \
-              || echo "[]")
-
-            # Parse highest N from vYYYY.MM.N tags
-            LAST_N=$(echo "$TAGS" | grep -oP "\"name\":\"${PREFIX}\.\K[0-9]+" | sort -n | tail -1)
-            if [ -z "$LAST_N" ]; then
-              NEXT_N=1
-            else
-              NEXT_N=$((LAST_N + 1))
-            fi
-
-            TAG="${PREFIX}.${NEXT_N}"
-            echo "Creating tag: ${TAG} at ${SHA}"
-
-            # Create the tag (tolerate failure if tag already exists)
-            HTTP_CODE=$(curl -s -o /tmp/tag_response -w "%{http_code}" -X POST \
-              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-              -d "tag_name=${TAG}" \
-              -d "ref=${SHA}" \
-              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags")
-
-            RESULT=$(cat /tmp/tag_response 2>/dev/null || echo "{}")
-
-            if [ "$HTTP_CODE" = "201" ]; then
-              echo "Tag ${TAG} created successfully"
-            elif [ "$HTTP_CODE" = "400" ] && echo "$RESULT" | grep -q "already exists"; then
-              echo "Tag ${TAG} already exists — reusing"
-            else
-              echo "WARN: Tag creation returned HTTP ${HTTP_CODE}: ${RESULT}"
-              echo "Falling back to dev tag"
-              TAG="dev-$(echo $SHA | cut -c1-8)"
-            fi
-
-            mkdir -p /tmp/outputs
-            echo -n "$TAG" > /tmp/outputs/tag
-      outputs:
-        parameters:
-          - name: tag
-            valueFrom:
-              path: /tmp/outputs/tag
-
-    # ── compile-services: selective per-binary build, incremental on local RWO PVC ──
-    - name: compile-services
-      metadata:
-        labels:
-          app.kubernetes.io/component: compile
-      inputs:
-        parameters:
-          - name: tag
-          - name: service-packages
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      # ── seed-deps-cache: rsync prebuilt third-party rlibs into the PVC ──
-      # On a true cold PVC (e.g. fresh node, first run, or after a purge),
-      # this saves 2-4 min that would otherwise be spent compiling the
-      # ~1500 third-party crates in the workspace dep graph. The image is
-      # rebuilt nightly by refresh-deps-cache (see
-      # refresh-deps-cache-template.yaml). On a warm PVC, the stamp file
-      # short-circuits the whole step in ~50ms.
-      #
-      # Why rsync --ignore-existing instead of cp/overwrite: a warmer PVC
-      # may already have NEWER artifacts from a prior compile; we don't
-      # want to clobber them with potentially-stale prebuilt rlibs. Cargo
-      # will discard rlibs whose fingerprint mismatches anyway.
-      #
-      # If the deps-cache image isn't published yet (first deploy of this
-      # template), set imagePullPolicy: IfNotPresent + the initContainer
-      # is non-blocking-on-failure (`|| true`) so the compile-services
-      # step still runs without the cache prelude.
-      initContainers:
-        - name: seed-deps-cache
-          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu-with-deps:nightly
-          imagePullPolicy: IfNotPresent
-          env:
-            - name: DEPS_VERSION
-              value: "1"
-          resources:
-            requests:
-              cpu: 500m
-              memory: 512Mi
-            limits:
-              cpu: "2"
-              memory: 1Gi
-          volumeMounts:
-            - name: cargo-target-cpu
-              mountPath: /cargo-target
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              STAMP="/cargo-target/cpu_deps_v${DEPS_VERSION}.stamp"
-              if [ -f "$STAMP" ]; then
-                echo "PVC already seeded for deps v${DEPS_VERSION} (stamp present), skipping."
-                exit 0
-              fi
-              if [ ! -d /cargo-target-prebuilt ]; then
-                echo "WARN: /cargo-target-prebuilt missing in image — image not yet published?"
-                echo "      Compile will run without prebuilt deps cache (slower cold-start)."
-                exit 0
-              fi
-              echo "=== Seeding cargo-target-cpu PVC from prebuilt deps cache (v${DEPS_VERSION}) ==="
-              du -sh /cargo-target-prebuilt 2>/dev/null || true
-              # --ignore-existing: never clobber newer artifacts already on PVC.
-              # -a: preserve mtimes, perms, links — cargo's freshness check needs accurate mtimes.
-              rsync -a --ignore-existing /cargo-target-prebuilt/ /cargo-target/
-              touch "$STAMP"
-              echo "=== Seed complete ==="
-              du -sh /cargo-target/release 2>/dev/null || true
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu:latest
-        imagePullPolicy: Always
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          # sccache: rustc wrapper that content-hashes compile inputs and reuses
-          # object output across pods/commits via the sccache-cpu PVC. Survives
-          # cargo's incremental-cache invalidation from git checkout mtime touches.
-          - name: RUSTC_WRAPPER
-            value: sccache
-          - name: SCCACHE_DIR
-            value: /sccache-cache
-          - name: SCCACHE_CACHE_SIZE
-            value: 20G
-          # Required: workspace `.cargo/config.toml` sets [build] incremental = true,
-          # which forces incremental compilation even on --release.  sccache cannot
-          # cache incremental rustc output (documented limitation).  Override here
-          # so sccache actually catches.  Local dev keeps incremental via config.toml.
-          - name: CARGO_INCREMENTAL
-            value: "0"
-          # Match the cgroup cpu limit below (limits.cpu: "30").
-          # Without this, cargo asks num_cpus::get_physical() — the host's
-          # whole CPU count — and over-subscribes vs the cgroup throttle,
-          # producing scheduling waste under load.
-          - name: CARGO_BUILD_JOBS
-            value: "30"
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: "14"
-            memory: 16Gi
-          limits:
-            cpu: "30"
-            memory: 32Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cpu
-            mountPath: /cargo-target
-          - name: sccache
-            mountPath: /sccache-cache
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            SERVICE_PKGS="{{inputs.parameters.service-packages}}"
-
-            # SSH setup
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            WORKSPACE="/cargo-target/src"
-
-            # PVC may be owned by a different UID from a previous run
-            git config --global --add safe.directory "$WORKSPACE"
-
-            # Persistent checkout on PVC — only changed files get new mtimes,
-            # so cargo skips recompiling unchanged workspace crates.
-            if [ -d "$WORKSPACE/.git" ]; then
-              cd "$WORKSPACE"
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" = "$SHA" ]; then
-                echo "=== Already at $SHA, skipping checkout ==="
-              else
-                echo "=== Updating checkout: $(echo $CURRENT | cut -c1-8) -> $(echo $SHA | cut -c1-8) ==="
-                git fetch origin
-                git checkout --force "$SHA"
-                git clean -fd
-              fi
-            else
-              echo "=== Initial clone (first run) ==="
-              git clone --filter=blob:none "$REPO" "$WORKSPACE"
-              cd "$WORKSPACE"
-              git checkout "$SHA"
-            fi
-            echo "Checked out $(git rev-parse --short HEAD)"
-
-            # Ensure cargo home registry is on PVC
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-
-            export FOXHUNT_BUILD_VERSION="{{inputs.parameters.tag}}"
-
-            # Prune build artifacts if PVC exceeds 25GB (prevents unbounded growth)
-            TARGET_SIZE_MB=$(du -sm "$CARGO_TARGET_DIR" 2>/dev/null | cut -f1 || echo 0)
-            echo "PVC usage: ${TARGET_SIZE_MB}MB"
-            if [ "$TARGET_SIZE_MB" -gt 25000 ]; then
-              echo "PVC exceeds 25GB limit, pruning build artifacts..."
-              rm -rf "$CARGO_TARGET_DIR/release" "$CARGO_TARGET_DIR/debug"
-            fi
-
-            # Guard: empty package list would build entire workspace
-            if [ -z "$SERVICE_PKGS" ]; then
-              echo "ERROR: service-packages is empty, refusing to build entire workspace"
-              exit 1
-            fi
-
-            # Build only the affected service packages (incremental via persistent target dir)
-            CARGO_ARGS=""
-            for pkg in $SERVICE_PKGS; do
-              CARGO_ARGS="$CARGO_ARGS -p $pkg"
-            done
-
-            # CI-only linker swap: prefer wild over mold. wild is a Rust-native
-            # linker, typically 10-30% faster than mold on release-LTO links;
-            # we have ~5 service binaries each doing release-LTO, so this is
-            # a meaningful 30-90 sec wall-time saving. Both linkers ship in
-            # Dockerfile.ci-builder-cpu — to revert, drop this sed and rebuild.
-            # Sed is idempotent (no-op if wild already substituted from a prior
-            # run on the same PVC checkout) and surgical (single line in
-            # .cargo/config.toml). If wild is missing for any reason, the build
-            # falls back to mold once we revert this hunk.
-            if command -v wild >/dev/null 2>&1; then
-              echo "=== Swapping linker mold -> wild for this CI run ==="
-              sed -i 's|-fuse-ld=mold|-fuse-ld=wild|' .cargo/config.toml
-              grep -n 'fuse-ld' .cargo/config.toml || true
-            else
-              echo "WARN: wild not on PATH, sticking with mold"
-            fi
-
-            echo "=== Building service binaries: $SERVICE_PKGS (incremental) ==="
-            # --locked: skip Cargo.lock resolver work (~5-15s saved) and fail fast
-            # if the lock has drifted (catches accidental Cargo.toml edits without
-            # a re-resolve commit).
-            cargo build --locked --release $CARGO_ARGS
-
-            # Collect built binaries
-            mkdir -p "$WORKSPACE/bin/services"
-            for pkg in $SERVICE_PKGS; do
-              bin_name=$(echo "$pkg" | tr '-' '_')
-              cp "$CARGO_TARGET_DIR/release/$pkg" "$WORKSPACE/bin/services/" 2>/dev/null \
-                || cp "$CARGO_TARGET_DIR/release/$bin_name" "$WORKSPACE/bin/services/" 2>/dev/null \
-                || { echo "Binary not found for $pkg"; ls "$CARGO_TARGET_DIR/release/"; exit 1; }
-            done
-            strip "$WORKSPACE/bin/services/"*
-
-            echo "=== Service binaries ==="
-            ls -lh "$WORKSPACE/bin/services/"
-
-            echo "=== Uploading service binaries to GitLab packages ==="
-            GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-            TAG="${FOXHUNT_BUILD_VERSION}"
-            for bin in "$WORKSPACE/bin/services/"*; do
-              BIN_NAME=$(basename "$bin")
-              echo "Uploading ${BIN_NAME} (${TAG})..."
-              curl -f --upload-file "$bin" \
-                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                "${GITLAB}/api/v4/projects/1/packages/generic/foxhunt-services/${TAG}/${BIN_NAME}"
-            done
-
-            # Update 'latest' per-file (preserves unbuilt binaries from prior runs)
-            echo "=== Updating 'latest' package ==="
-            LATEST_PKG=$(curl -sf -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-              "${GITLAB}/api/v4/projects/1/packages?package_name=foxhunt-services&package_version=latest" \
-              | grep -oP '"id":\K[0-9]+' | head -1)
-            for bin in "$WORKSPACE/bin/services/"*; do
-              BIN_NAME=$(basename "$bin")
-              # Delete existing file by name before uploading replacement
-              if [ -n "$LATEST_PKG" ]; then
-                FILE_ID=$(curl -sf -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                  "${GITLAB}/api/v4/projects/1/packages/${LATEST_PKG}/package_files" \
-                  | grep -oP "\"id\":([0-9]+),\"package_id\":${LATEST_PKG}[^}]*\"file_name\":\"${BIN_NAME}\"" \
-                  | grep -oP '"id":\K[0-9]+' | head -1)
-                if [ -n "$FILE_ID" ]; then
-                  curl -sf -X DELETE -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                    "${GITLAB}/api/v4/projects/1/packages/${LATEST_PKG}/package_files/${FILE_ID}" || true
-                fi
-              fi
-              curl -f --upload-file "$bin" \
-                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                "${GITLAB}/api/v4/projects/1/packages/generic/foxhunt-services/latest/${BIN_NAME}" || true
-            done
-
-            echo "=== sccache stats ==="
-            sccache --show-stats || true
-
-            echo "=== Service compile + upload done ($SERVICE_PKGS) ==="
-
-    # ── upload-release: create GitLab Release with package links ──
-    - name: upload-release
-      inputs:
-        parameters:
-          - name: tag
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            cpu: 200m
-            memory: 128Mi
-        args:
-          - |
-            set -e
-            GITLAB="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-            PROJECT_ID=1
-            TAG="{{inputs.parameters.tag}}"
-
-            echo "=== Creating GitLab Release ${TAG} ==="
-
-            # Get commits since previous tag for release notes
-            PREV_TAG=$(curl -sf \
-              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-              "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/tags?per_page=2" \
-              | grep -oP '"name":"\K[^"]+' | sed -n '2p')
-
-            if [ -n "$PREV_TAG" ]; then
-              COMMITS=$(curl -sf \
-                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                "${GITLAB}/api/v4/projects/${PROJECT_ID}/repository/compare?from=${PREV_TAG}&to=${TAG}" \
-                | grep -oP '"title":"\K[^"]+' | head -20 \
-                | sed 's/^/- /' || echo "- Release ${TAG}")
-              DESCRIPTION="## Changes since ${PREV_TAG}\n\n${COMMITS}"
-            else
-              DESCRIPTION="## Initial release\n\nFirst CalVer release."
-            fi
-
-            # Create the release
-            RESPONSE=$(curl -sf -X POST \
-              -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-              -H "Content-Type: application/json" \
-              -d "{
-                \"tag_name\": \"${TAG}\",
-                \"name\": \"${TAG}\",
-                \"description\": \"$(printf '%s' "$DESCRIPTION" | sed 's/"/\\"/g')\"
-              }" \
-              "${GITLAB}/api/v4/projects/${PROJECT_ID}/releases") || {
-              echo "WARN: Release creation failed (may already exist)"
-            }
-
-            echo "Release ${TAG} created"
-            echo "$RESPONSE" | head -5
-
-            # Add package links as release assets
-            for pkg_name in foxhunt-services foxhunt-training; do
-              PKG_CHECK=$(curl -sf \
-                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                "${GITLAB}/api/v4/projects/${PROJECT_ID}/packages?package_name=${pkg_name}&package_version=${TAG}" \
-                || echo "[]")
-
-              if echo "$PKG_CHECK" | grep -q "$TAG"; then
-                curl -sf -X POST \
-                  -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                  -H "Content-Type: application/json" \
-                  -d "{
-                    \"name\": \"${pkg_name}\",
-                    \"url\": \"${GITLAB}/-/packages?type=generic&search=${pkg_name}&version=${TAG}\",
-                    \"link_type\": \"package\"
-                  }" \
-                  "${GITLAB}/api/v4/projects/${PROJECT_ID}/releases/${TAG}/assets/links" || true
-                echo "Linked ${pkg_name} package to release"
-              fi
-            done
-
-            echo "=== Release ${TAG} complete ==="
-
-    # ── deploy-services: selective rolling restart for affected deployments ──
-    - name: deploy-services
-      inputs:
-        parameters:
-          - name: tag
-          - name: deploy-list
-      serviceAccountName: argo-workflow
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        resources:
-          requests:
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            cpu: 200m
-            memory: 128Mi
-        args:
-          - |
-            set -e
-            if ! command -v kubectl >/dev/null 2>&1; then
-              echo "=== Installing kubectl ==="
-              curl -sLo /tmp/kubectl "https://dl.k8s.io/release/v1.31.4/bin/linux/amd64/kubectl"
-              chmod +x /tmp/kubectl
-              export PATH="/tmp:$PATH"
-            fi
-
-            TAG="{{inputs.parameters.tag}}"
-            DEPLOY_LIST="{{inputs.parameters.deploy-list}}"
-
-            if [ -z "$DEPLOY_LIST" ]; then
-              echo "=== No services to deploy ==="
-              exit 0
-            fi
-
-            echo "=== Deploying release ${TAG}: $DEPLOY_LIST ==="
-
-            for svc in $DEPLOY_LIST; do
-              echo "Patching $svc with FOXHUNT_RELEASE=${TAG}..."
-              kubectl -n foxhunt patch deployment "$svc" -p "{
-                \"spec\":{\"template\":{
-                  \"metadata\":{\"annotations\":{\"foxhunt.io/release\":\"${TAG}\"}},
-                  \"spec\":{\"initContainers\":[{
-                    \"name\":\"fetch-binary\",
-                    \"env\":[{\"name\":\"FOXHUNT_RELEASE\",\"value\":\"${TAG}\"}]
-                  }]}
-                }}
-              }" || echo "WARN: $svc patch failed (may not exist)"
-            done
-
-            echo "=== Waiting for rollouts ==="
-            for svc in $DEPLOY_LIST; do
-              kubectl -n foxhunt rollout status deployment "$svc" --timeout=120s || echo "WARN: $svc rollout timeout"
-            done
-
-            echo "=== Deploy ${TAG} complete ($DEPLOY_LIST) ==="
-
-    # ── notify-result: post workflow outcome to Mattermost ──
-    - name: notify-result
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: WEBHOOK_URL
-            valueFrom:
-              secretKeyRef:
-                name: notification-webhook
-                key: webhook-url
-                optional: true
-        resources:
-          requests:
-            cpu: 50m
-            memory: 32Mi
-          limits:
-            cpu: 100m
-            memory: 64Mi
-        args:
-          - |
-            STATUS="{{workflow.status}}"
-            NAME="{{workflow.name}}"
-
-            if [ -z "$WEBHOOK_URL" ] || echo "$WEBHOOK_URL" | grep -q "PLACEHOLDER"; then
-              echo "No webhook configured, skipping notification"
-              exit 0
-            fi
-
-            if [ "$STATUS" = "Succeeded" ]; then
-              EMOJI=":white_check_mark:"
-            else
-              EMOJI=":x:"
-            fi
-
-            PAYLOAD="{\"username\":\"Argo CI\",\"text\":\"${EMOJI} **${NAME}** — ${STATUS} ({{workflow.duration}}s)\"}"
-
-            curl -sf -X POST -H 'Content-Type: application/json' \
-              -d "$PAYLOAD" "$WEBHOOK_URL" || echo "WARN: webhook post failed"

infra/k8s/argo/databento-download-template.yaml

@@ -1,135 +0,0 @@
-# Databento data download via streaming API (get_range).
-#
-# Parameterized Argo WorkflowTemplate — replaces the hardcoded K8s Job.
-# Binary pre-uploaded to /data/bin/download_baseline on training-data-pvc.
-# Universe configs loaded from configmap databento-universe-all.
-#
-# Usage:
-#   argo submit -n foxhunt --from=wftmpl/databento-download
-#   argo submit -n foxhunt --from=wftmpl/databento-download -p schema=trades -p output-dir=/data/futures-baseline-trades
-#   ./scripts/argo-download.sh mbp-10 --parallel 9 --pool ci-compile-cpu
-#
-# Databento rate limits: 100 concurrent connections per IP, 100 req/s.
----
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: databento-download
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: databento-download
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: download
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: databento-download
-  securityContext:
-    fsGroup: 1000
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 86400  # 24 hours
-
-  arguments:
-    parameters:
-      - name: schema
-        value: mbp-10
-        description: "Databento schema: ohlcv-1m, ohlcv-1s, trades, mbp-10"
-      - name: output-dir
-        value: /data/futures-baseline-mbp10
-        description: "Output directory on training-data PVC"
-      - name: parallel
-        value: "9"
-        description: "Number of parallel quarterly downloads (max ~100 per Databento rate limit)"
-      - name: node-pool
-        value: ci-compile-cpu
-        description: "K8s node pool: ci-compile-cpu (32c/64G) or platform (4c/8G)"
-
-  volumes:
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: universe-configs
-      configMap:
-        name: databento-universe-all
-
-  templates:
-    - name: download
-      inputs:
-        parameters:
-          - name: schema
-          - name: output-dir
-          - name: parallel
-          - name: node-pool
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{inputs.parameters.node-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: ubuntu:24.04
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: DATABENTO_API_KEY
-            valueFrom:
-              secretKeyRef:
-                name: databento-credentials
-                key: api-key
-        resources:
-          requests:
-            cpu: 500m
-            memory: 512Mi
-          limits:
-            cpu: "2"
-            memory: 4Gi
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-          - name: universe-configs
-            mountPath: /configs
-        args:
-          - |
-            set -e
-            apt-get update -qq > /dev/null 2>&1 && apt-get install -y -qq ca-certificates > /dev/null 2>&1
-
-            BINARY=/data/bin/download_baseline
-            chmod +x "$BINARY"
-
-            SCHEMA="{{inputs.parameters.schema}}"
-            OUTPUT_DIR="{{inputs.parameters.output-dir}}"
-            PARALLEL="{{inputs.parameters.parallel}}"
-
-            # Map schema name to configmap key
-            CONFIG="/configs/${SCHEMA}.toml"
-            if [ ! -f "$CONFIG" ]; then
-              echo "ERROR: Config not found: $CONFIG"
-              echo "Available configs:"
-              ls /configs/
-              exit 1
-            fi
-
-            mkdir -p "$OUTPUT_DIR"
-
-            echo "=== Databento Download ==="
-            echo "Schema:    $SCHEMA"
-            echo "Config:    $CONFIG"
-            echo "Output:    $OUTPUT_DIR"
-            echo "Parallel:  $PARALLEL"
-            echo "Node:      $(hostname), CPUs: $(nproc), RAM: $(free -g | awk '/Mem:/{print $2}')G"
-            echo ""
-
-            $BINARY \
-              --universe-config "$CONFIG" \
-              --output-dir "$OUTPUT_DIR" \
-              --parallel "$PARALLEL" \
-              --yes
-
-            echo ""
-            echo "=== Download complete ==="
-            echo "Files:"
-            find "$OUTPUT_DIR" -name '*.dbn.zst' -exec ls -lh {} \;
-            echo ""
-            du -sh "$OUTPUT_DIR"
-            df -h /data

infra/k8s/argo/dns-bootstrap-policy.yaml

@@ -1,25 +0,0 @@
-# DNS Bootstrap Prevention — Prevent CoreDNS chicken-and-egg deadlock
-#
-# ROOT CAUSE (2026-03-18 incident):
-# 1. Rolling replacement of ALL platform nodes simultaneously
-# 2. New nodes boot with /etc/resolv.conf → 10.32.0.10 (kube-dns only)
-# 3. CoreDNS pods need to pull images → need DNS → need CoreDNS
-# 4. Cilium BPF socket LB returns EPERM when kube-dns has no endpoints
-# 5. Complete cluster DNS deadlock — nothing can pull images
-#
-# PREVENTION:
-# 1. VPC DefaultRoutePropagation MUST be enabled (API, not Terraform yet)
-# 2. NEVER replace ALL nodes at once — one at a time, wait for CoreDNS
-# 3. Do NOT create CiliumClusterwideNetworkPolicy with egress rules
-#    (enables default deny, makes the problem worse)
-# 4. bpf-lb-sock MUST remain "true" — disabling breaks ALL service routing
-# 5. Keep bastion enabled on VPC gateway for emergency SSH access
-#
-# RECOVERY PROCEDURE:
-# 1. SSH via bastion: ssh -o ProxyCommand="ssh -W %h:%p -p 61000 bastion@<gw-public-ip>" root@<node-private-ip>
-# 2. On the node: /usr/local/bin/crictl pull <coredns-image>
-# 3. kubectl delete pods -l k8s-app=kube-dns --force
-# 4. CoreDNS starts from cached image → DNS recovers → cascade
-#
-# DATA PROTECTION:
-# All PVs set to Retain reclaim policy — data survives cluster deletion

infra/k8s/argo/events/auto-compile-configmap.yaml

@@ -1,12 +0,0 @@
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: auto-compile-config
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: auto-compile-config
-    app.kubernetes.io/part-of: foxhunt
-data:
-  # Set to "true" to enable auto-compile on push to main.
-  # kubectl -n foxhunt patch configmap auto-compile-config -p '{"data":{"enabled":"true"}}'
-  enabled: "false"

infra/k8s/argo/events/ci-pipeline-sensor.yaml

@@ -1,70 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: Sensor
-metadata:
-  name: ci-pipeline-trigger
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: ci-pipeline-trigger
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  template:
-    serviceAccountName: argo-workflow
-    nodeSelector:
-      k8s.scaleway.com/pool-name: platform
-  eventBusName: default
-  dependencies:
-    - name: gitlab-push-dep
-      eventSourceName: gitlab-push
-      eventName: gitlab-push
-      filters:
-        data:
-          - path: body.ref
-            type: string
-            value:
-              - "refs/heads/main"
-  # Policy (user directive): push-triggered runs are infra-sync only.
-  # The ci-pipeline DAG runs three conditional task groups on push:
-  #   - docker image rebuilds    (gated on infra/docker/ changes)
-  #   - apply-argo-templates     (gated on infra/k8s/argo/ changes)
-  #   - terragrunt-apply         (gated on infra/live/ or infra/modules/ changes)
-  # Non-infra tasks (test-gate, build-web-dashboard, gpu-test) have `when: "false"`
-  # in the WorkflowTemplate. Training + compile/deploy are separate templates,
-  # always manual.
-  #
-  # All non-push workflows (cargo tests, training, compile/deploy, GPU tests)
-  # run via scripts/manual `argo submit`.
-
-  triggers:
-    - template:
-        name: ci-pipeline
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: ci-pipeline-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: ci-pipeline
-                arguments:
-                  parameters:
-                    - name: commit-sha
-                    - name: commits-json
-          parameters:
-            - src:
-                dependencyName: gitlab-push-dep
-                dataKey: body.checkout_sha
-              dest: spec.arguments.parameters.0.value
-            - src:
-                dependencyName: gitlab-push-dep
-                dataKey: body.commits
-                dataTemplate: "{{ toJson .Input }}"
-              dest: spec.arguments.parameters.1.value

infra/k8s/argo/events/eventbus.yaml

@@ -1,10 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: EventBus
-metadata:
-  name: default
-  namespace: foxhunt
-spec:
-  nats:
-    native:
-      replicas: 3
-      auth: token

infra/k8s/argo/events/eventsource-svc.yaml

@@ -1,16 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: gitlab-push-eventsource-svc
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: gitlab-push-eventsource
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  type: ClusterIP
-  ports:
-    - port: 12000
-      targetPort: 12000
-      protocol: TCP
-  selector:
-    eventsource-name: gitlab-push

infra/k8s/argo/events/gitlab-push-eventsource.yaml

@@ -1,15 +0,0 @@
-apiVersion: argoproj.io/v1alpha1
-kind: EventSource
-metadata:
-  name: gitlab-push
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: gitlab-push-eventsource
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  eventBusName: default
-  webhook:
-    gitlab-push:
-      port: "12000"
-      endpoint: /push
-      method: POST

infra/k8s/argo/events/rbac.yaml

@@ -1,29 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: argo-sensor-submit
-  namespace: foxhunt
-rules:
-  - apiGroups: ["argoproj.io"]
-    resources: ["workflows"]
-    verbs: ["create", "get", "list", "watch"]
-  - apiGroups: ["argoproj.io"]
-    resources: ["workflowtemplates"]
-    verbs: ["get", "list"]
-  - apiGroups: ["argoproj.io"]
-    resources: ["workflowtaskresults"]
-    verbs: ["create", "patch"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: argo-sensor-submit
-  namespace: foxhunt
-subjects:
-  - kind: ServiceAccount
-    name: argo-workflow
-    namespace: foxhunt
-roleRef:
-  kind: Role
-  name: argo-sensor-submit
-  apiGroup: rbac.authorization.k8s.io

infra/k8s/argo/events/workflow-trigger-eventsource.yaml

@@ -1,51 +0,0 @@
-# Manual workflow trigger EventSource.
-#
-# Webhook endpoints for triggering compile/deploy, training, and image builds.
-# Each endpoint accepts a JSON POST body with workflow-specific parameters.
-#
-# Endpoints (all on port 12001):
-#   POST /compile-deploy    — {"commit_sha": "HEAD"}
-#   POST /train/dqn         — {"binary_tag": "latest"}
-#   POST /train/ppo         — {"binary_tag": "latest"}
-#   POST /train/supervised  — {"model": "tft", "binary_tag": "latest"}
-#   POST /build-image       — {"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}
-#
-# Example:
-#   curl -X POST http://workflow-trigger-eventsource-svc.foxhunt:12001/build-image \
-#     -H 'Content-Type: application/json' \
-#     -d '{"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}'
----
-apiVersion: argoproj.io/v1alpha1
-kind: EventSource
-metadata:
-  name: workflow-trigger
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: workflow-trigger
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  eventBusName: default
-  template:
-    nodeSelector:
-      k8s.scaleway.com/pool-name: platform
-  webhook:
-    compile-deploy:
-      port: "12001"
-      endpoint: /compile-deploy
-      method: POST
-    train-dqn:
-      port: "12001"
-      endpoint: /train/dqn
-      method: POST
-    train-ppo:
-      port: "12001"
-      endpoint: /train/ppo
-      method: POST
-    train-supervised:
-      port: "12001"
-      endpoint: /train/supervised
-      method: POST
-    build-image:
-      port: "12001"
-      endpoint: /build-image
-      method: POST

infra/k8s/argo/events/workflow-trigger-sensor.yaml

@@ -1,217 +0,0 @@
-# Sensor for manual workflow triggers.
-#
-# Listens on workflow-trigger EventSource and submits the appropriate workflow
-# when a webhook endpoint is hit. Each trigger maps JSON body fields to
-# workflow parameters.
-#
-# Trigger reference:
-#   compile-deploy    — POST /compile-deploy    {"commit_sha": "..."}
-#   train-dqn         — POST /train/dqn         {"binary_tag": "..."}
-#   train-ppo         — POST /train/ppo         {"binary_tag": "..."}
-#   train-supervised  — POST /train/supervised  {"model": "...", "binary_tag": "..."}
-#   build-image       — POST /build-image       {"dockerfile": "...", "image_name": "...", "commit_sha": "..."}
----
-apiVersion: argoproj.io/v1alpha1
-kind: Sensor
-metadata:
-  name: workflow-trigger
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: workflow-trigger
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  template:
-    serviceAccountName: argo-workflow
-    nodeSelector:
-      k8s.scaleway.com/pool-name: platform
-  eventBusName: default
-
-  dependencies:
-    - name: compile-deploy-dep
-      eventSourceName: workflow-trigger
-      eventName: compile-deploy
-    - name: train-dqn-dep
-      eventSourceName: workflow-trigger
-      eventName: train-dqn
-    - name: train-ppo-dep
-      eventSourceName: workflow-trigger
-      eventName: train-ppo
-    - name: train-supervised-dep
-      eventSourceName: workflow-trigger
-      eventName: train-supervised
-    - name: build-image-dep
-      eventSourceName: workflow-trigger
-      eventName: build-image
-
-  triggers:
-    # ── Compile + Deploy ──
-    # POST /compile-deploy {"commit_sha": "abc1234"}
-    # Omit service_packages to build all 7 services (default).
-    - template:
-        name: compile-deploy
-        conditions: compile-deploy-dep
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: compile-deploy-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: compile-and-deploy
-                arguments:
-                  parameters:
-                    - name: commit-sha
-          parameters:
-            - src:
-                dependencyName: compile-deploy-dep
-                dataKey: body.commit_sha
-              dest: spec.arguments.parameters.0.value
-
-    # ── Train DQN ──
-    # POST /train/dqn {"binary_tag": "latest"}
-    - template:
-        name: train-dqn
-        conditions: train-dqn-dep
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: train-dqn-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: train-dqn
-                arguments:
-                  parameters:
-                    - name: binary-tag
-          parameters:
-            - src:
-                dependencyName: train-dqn-dep
-                dataKey: body.binary_tag
-              dest: spec.arguments.parameters.0.value
-
-    # ── Train PPO ──
-    # POST /train/ppo {"binary_tag": "latest"}
-    - template:
-        name: train-ppo
-        conditions: train-ppo-dep
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: train-ppo-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: train-ppo
-                arguments:
-                  parameters:
-                    - name: binary-tag
-          parameters:
-            - src:
-                dependencyName: train-ppo-dep
-                dataKey: body.binary_tag
-              dest: spec.arguments.parameters.0.value
-
-    # ── Train Supervised ──
-    # POST /train/supervised {"model": "tft", "binary_tag": "latest"}
-    # model: one of tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion
-    - template:
-        name: train-supervised
-        conditions: train-supervised-dep
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: train-supervised-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: train-supervised
-                arguments:
-                  parameters:
-                    - name: model
-                    - name: binary-tag
-          parameters:
-            - src:
-                dependencyName: train-supervised-dep
-                dataKey: body.model
-              dest: spec.arguments.parameters.0.value
-            - src:
-                dependencyName: train-supervised-dep
-                dataKey: body.binary_tag
-              dest: spec.arguments.parameters.1.value
-
-    # ── Build Image ──
-    # POST /build-image {"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}
-    - template:
-        name: build-image
-        conditions: build-image-dep
-        argoWorkflow:
-          operation: submit
-          source:
-            resource:
-              apiVersion: argoproj.io/v1alpha1
-              kind: Workflow
-              metadata:
-                generateName: build-image-
-                namespace: foxhunt
-              spec:
-                serviceAccountName: argo-workflow
-                podGC:
-                  strategy: OnPodCompletion
-                ttlStrategy:
-                  secondsAfterCompletion: 3600
-                workflowTemplateRef:
-                  name: build-ci-image
-                arguments:
-                  parameters:
-                    - name: commit-sha
-                    - name: dockerfile
-                    - name: image-name
-          parameters:
-            - src:
-                dependencyName: build-image-dep
-                dataKey: body.commit_sha
-              dest: spec.arguments.parameters.0.value
-            - src:
-                dependencyName: build-image-dep
-                dataKey: body.dockerfile
-              dest: spec.arguments.parameters.1.value
-            - src:
-                dependencyName: build-image-dep
-                dataKey: body.image_name
-              dest: spec.arguments.parameters.2.value

infra/k8s/argo/events/workflow-trigger-svc.yaml

@@ -1,16 +0,0 @@
-apiVersion: v1
-kind: Service
-metadata:
-  name: workflow-trigger-eventsource-svc
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: workflow-trigger
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  type: ClusterIP
-  ports:
-    - port: 12001
-      targetPort: 12001
-      protocol: TCP
-  selector:
-    eventsource-name: workflow-trigger

infra/k8s/argo/feature-cache-pvc.yaml

@@ -1,18 +0,0 @@
-# Dedicated PVC for precomputed .fxcache files.
-# Survives workflow completions (unlike workspace volumeClaimTemplate).
-# Precompute writes here, all training steps read from here.
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: feature-cache-pvc
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: feature-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 10Gi

infra/k8s/argo/gpu-test-network-policy.yaml

@@ -1,93 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: argo-gpu-test-workflow
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/component: gpu-test
-  policyTypes:
-    - Egress
-  egress:
-    # DNS
-    - ports:
-        - port: 53
-          protocol: UDP
-        - port: 53
-          protocol: TCP
-    # Kubernetes API + internal services (service CIDR)
-    - ports:
-        - port: 443
-          protocol: TCP
-        - port: 6443
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 10.32.0.0/16
-        - ipBlock:
-            cidr: 172.16.0.4/32
-    # HTTPS egress (crates.io, etc.)
-    - ports:
-        - port: 443
-          protocol: TCP
-    # Git SSH (GitLab)
-    - ports:
-        - port: 2222
-          protocol: TCP
-      to:
-        - ipBlock:
-            cidr: 100.90.76.85/32
-        - podSelector:
-            matchLabels:
-              app: gitlab-shell
-    # GitLab webservice (git-http)
-    - ports:
-        - port: 8181
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: webservice
-    # MinIO (artifact/log storage)
-    - ports:
-        - port: 9000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: minio
-    # Pushgateway (Prometheus metrics)
-    - ports:
-        - port: 9091
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: pushgateway
-    # OTLP (Tempo)
-    - ports:
-        - port: 4317
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: tempo
-    # Mattermost (notifications)
-    - ports:
-        - port: 8065
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app.kubernetes.io/name: mattermost
-    # GitLab container registry
-    - ports:
-        - port: 5000
-          protocol: TCP
-      to:
-        - podSelector:
-            matchLabels:
-              app: registry

infra/k8s/argo/gpu-test-nightly-cron.yaml

@@ -1,25 +0,0 @@
-# infra/k8s/argo/gpu-test-nightly-cron.yaml
-apiVersion: argoproj.io/v1alpha1
-kind: CronWorkflow
-metadata:
-  name: gpu-test-nightly
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: gpu-test-nightly
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  schedule: "0 2 * * *"
-  timezone: "UTC"
-  suspend: true  # Enable when ready: kubectl patch cronworkflow gpu-test-nightly -n foxhunt -p '{"spec":{"suspend":false}}'
-  concurrencyPolicy: Replace
-  successfulJobsHistoryLimit: 3
-  failedJobsHistoryLimit: 5
-  workflowSpec:
-    workflowTemplateRef:
-      name: gpu-test-pipeline
-    arguments:
-      parameters:
-        - name: models
-          value: "dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion"
-        - name: commit-ref
-          value: "main"

infra/k8s/argo/gpu-test-pipeline-template.yaml

@@ -1,529 +0,0 @@
-# infra/k8s/argo/gpu-test-pipeline-template.yaml
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: gpu-test-pipeline
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: gpu-test-pipeline
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: pipeline
-  onExit: notify-result
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: gpu-test
-  securityContext:
-    fsGroup: 0
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 7200
-  arguments:
-    parameters:
-      - name: commit-ref
-        value: HEAD
-      - name: models
-        value: "dqn,ppo,tft"
-      - name: test-scope
-        value: all
-      - name: gpu-pool
-        value: ci-training-h100
-      - name: cuda-compute-cap
-        value: "90"
-      - name: notify
-        value: "true"
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: cargo-target
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda-test
-    - name: test-data
-      persistentVolumeClaim:
-        claimName: test-data-pvc
-        readOnly: true
-
-  templates:
-    # ── pipeline: DAG entrypoint ──
-    # compile-and-test runs on GPU node (RWO PVC can't be shared cross-node).
-    # gpu-warmup ensures H100 is scaled up before compile starts.
-    - name: pipeline
-      dag:
-        tasks:
-          - name: gpu-warmup
-            template: gpu-warmup
-          - name: compile-and-test
-            template: compile-and-test
-            dependencies: [gpu-warmup]
-          - name: perf-benchmark
-            template: perf-benchmark
-            dependencies: [compile-and-test]
-
-    # ── gpu-warmup: trigger H100 autoscale ──
-    # Requests GPU to force autoscaler to provision node, then releases it.
-    - name: gpu-warmup
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: alpine:3.21
-        command: ["/bin/sh", "-c"]
-        args:
-          - |
-            echo "GPU warmup: triggering node autoscale..."
-            nvidia-smi --query-gpu=name,memory.total --format=csv,noheader || true
-            echo "GPU node ready, releasing for compile-and-test"
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: 200m
-            memory: 128Mi
-
-    # ── compile-and-test: compile + run GPU tests in single H100 pod ──
-    - name: compile-and-test
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      outputs:
-        parameters:
-          - name: results
-            valueFrom:
-              path: /tmp/outputs/results
-              default: "unknown:FAIL"
-          - name: failures
-            valueFrom:
-              path: /tmp/outputs/failures
-              default: "1"
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CUDA_VISIBLE_DEVICES
-            value: "0"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: TEST_DATA_DIR
-            value: /data/test-data
-          - name: RUST_LOG
-            value: info
-          - name: LD_LIBRARY_PATH
-            value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
-          - name: CARGO_PROFILE_TEST_OPT_LEVEL
-            value: "2"
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target
-            mountPath: /cargo-target
-          - name: test-data
-            mountPath: /data/test-data
-            readOnly: true
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        args:
-          - |
-            set -e
-
-            REF="{{workflow.parameters.commit-ref}}"
-            MODELS="{{workflow.parameters.models}}"
-            SCOPE="{{workflow.parameters.test-scope}}"
-
-            # --- SSH setup (same as compile-and-train) ---
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            # --- Persistent checkout on PVC ---
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              echo "=== Fetching latest refs ==="
-              git fetch origin
-              # Resolve REF after fetch so we always get the latest commit.
-              # Try origin/$REF (branch), then $REF directly (tag or SHA).
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                TARGET=$(git rev-parse "origin/$REF")
-              else
-                TARGET=$(git rev-parse --verify "$REF" 2>/dev/null || echo "$REF")
-              fi
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" = "$TARGET" ]; then
-                echo "=== Already at $REF ($TARGET) ==="
-              else
-                echo "=== Updating checkout to $REF ($TARGET) ==="
-                git checkout --force --detach "$TARGET"
-                git clean -fd
-              fi
-            else
-              echo "=== Initial clone ==="
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                git checkout --force --detach "origin/$REF"
-              else
-                git checkout "$REF"
-              fi
-            fi
-            echo "Checked out $(git rev-parse --short HEAD)"
-
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            # --- PTX cache invalidation ---
-            # Purge stale cached PTX if any CUDA kernel source changed since last run.
-            bash scripts/ptx-cache-invalidate.sh "${CARGO_TARGET_DIR}/.ptx_cache"
-
-            # --- PVC size guard ---
-            TARGET_SIZE_MB=$(du -sm "$CARGO_TARGET_DIR" 2>/dev/null | cut -f1 || echo 0)
-            echo "PVC usage: ${TARGET_SIZE_MB}MB"
-            if [ "$TARGET_SIZE_MB" -gt 25000 ]; then
-              echo "PVC exceeds 25GB, pruning..."
-              rm -rf "$CARGO_TARGET_DIR/release" "$CARGO_TARGET_DIR/debug"
-            fi
-
-            # --- Compile ---
-            echo "=== Compiling test binaries (--features cuda) ==="
-            cargo test -p ml -p ml-dqn -p ml-core --features cuda --no-run 2>&1 | tee /cargo-target/gpu-compile.log
-            echo "=== Compilation done ==="
-
-            # --- Expand "all" ---
-            if [ "$MODELS" = "all" ]; then
-              MODELS="dqn,ppo,tft,mamba2,tggn,tlob,liquid,kan,xlstm,diffusion"
-            fi
-
-            # --- Test runner ---
-            RESULTS=""
-            FAILURES=0
-
-            # Reset CUDA context between test binaries to prevent cuBLAS
-            # CUBLAS_STATUS_NOT_INITIALIZED cascades. Each test binary creates
-            # and destroys hundreds of cuBLAS handles; without a reset, the
-            # driver fails to re-init for the next binary.
-            gpu_context_drain() {
-              nvidia-smi -rgc >/dev/null 2>&1 || true
-              sleep 1
-            }
-
-            run_tests() {
-              local NAME="$1"; shift
-              echo ""
-              echo "========================================"
-              echo "  TEST: $NAME"
-              echo "========================================"
-              # --nocapture is a test-binary flag, must come after --
-              # If args already contain --, append after it; otherwise add -- first
-              local HAS_SEP=false
-              for arg in "$@"; do
-                [ "$arg" = "--" ] && HAS_SEP=true && break
-              done
-              set +e
-              if $HAS_SEP; then
-                "$@" --nocapture 2>&1
-              else
-                "$@" -- --nocapture 2>&1
-              fi
-              EXIT=$?
-              set -e
-              if [ $EXIT -eq 0 ]; then
-                RESULTS="${RESULTS}${NAME}:PASS\n"
-              else
-                RESULTS="${RESULTS}${NAME}:FAIL\n"
-                FAILURES=$((FAILURES + 1))
-              fi
-              # Drain CUDA context after each GPU test suite
-              gpu_context_drain
-            }
-
-            # --- Core tests (always run) ---
-            # --test-threads=1 for all GPU lib tests: prevents concurrent cuBLAS
-            # handle creation that causes CUBLAS_STATUS_NOT_INITIALIZED cascades.
-            run_tests "core-lib" cargo test -p ml-core --features cuda --lib -- --test-threads=1
-            run_tests "bayesian" cargo test -p ml --features cuda --test bayesian_changepoint_test -- --test-threads=1
-
-            # --- Per-model tests ---
-            IFS=',' read -ra MODEL_LIST <<< "$MODELS"
-            for MODEL in "${MODEL_LIST[@]}"; do
-              case "$MODEL" in
-                dqn)
-                  if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then
-                    # --test-threads=1: GPU lib tests must run serially — each test
-                    # creates a cuBLAS handle via Device::new_cuda(0). Under parallel
-                    # execution, concurrent cuBLAS init races cause
-                    # CUBLAS_STATUS_NOT_INITIALIZED failures (51 test cascade).
-                    run_tests "dqn-lib" cargo test -p ml-dqn --features cuda --lib -- --test-threads=1
-                    run_tests "dqn-ml-lib" cargo test -p ml --features cuda --lib -- --test-threads=1 dqn
-                  fi
-                  if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then
-                    # --test-threads=1 for ALL GPU integration tests: parallel execution
-                    # corrupts the CUDA primary context (cuDevicePrimaryCtxRetain fails
-                    # when multiple threads race on context init/teardown).
-                    run_tests "dqn-smoke" cargo test -p ml --features cuda --test smoke_test_real_data -- --test-threads=1
-                    # Run each pipeline test in its own cargo test process.
-                    # CUDA Graph capture corrupts the async memory pool, making
-                    # cuMemAllocAsync fail with CUDA_ERROR_INVALID_VALUE in
-                    # subsequent DQNTrainer instances within the same process.
-                    run_tests "dqn-pipeline-train" cargo test -p ml --features cuda --test dqn_training_pipeline_test test_dqn_trains_on_es_fut -- --test-threads=1 --exact
-                    run_tests "dqn-pipeline-loss" cargo test -p ml --features cuda --test dqn_training_pipeline_test test_dqn_loss_decreases -- --test-threads=1 --exact
-                    run_tests "dqn-pipeline-ckpt" cargo test -p ml --features cuda --test dqn_training_pipeline_test test_dqn_checkpoint_save_load -- --test-threads=1 --exact
-                    run_tests "dqn-pipeline-qval" cargo test -p ml --features cuda --test dqn_training_pipeline_test test_dqn_q_value_predictions -- --test-threads=1 --exact
-                    run_tests "dqn-pipeline-eps" cargo test -p ml --features cuda --test dqn_training_pipeline_test test_dqn_epsilon_greedy -- --test-threads=1 --exact
-                    run_tests "dqn-smoke-train" cargo test -p ml --features cuda --test dqn_training_smoke_test -- --test-threads=1
-                    run_tests "dqn-early-stop" cargo test -p ml --features cuda --test dqn_early_stopping_termination_test -- --test-threads=1
-                    run_tests "dqn-collapse" cargo test -p ml --features cuda --test dqn_action_collapse_fix_test -- --test-threads=1
-                  fi
-                  ;;
-                ppo)
-                  if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then
-                    run_tests "ppo-lib" cargo test -p ml --features cuda --lib -- --test-threads=1 ppo
-                  fi
-                  if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then
-                    run_tests "ppo-barrier" cargo test -p ml --features cuda --test barrier_optimization_test -- --test-threads=1
-                  fi
-                  ;;
-                *)
-                  # Supervised models (TFT, Mamba2, TGGN, TLOB, Liquid, KAN, xLSTM, Diffusion)
-                  # Map model names to Rust module names where they differ
-                  LIB_FILTER="$MODEL"
-                  [ "$MODEL" = "tggn" ] && LIB_FILTER="tgnn"
-                  if [ "$SCOPE" = "lib" ] || [ "$SCOPE" = "all" ]; then
-                    run_tests "${MODEL}-lib" cargo test -p ml --features cuda --lib -- --test-threads=1 "$LIB_FILTER"
-                  fi
-                  if [ "$SCOPE" = "integration" ] || [ "$SCOPE" = "all" ]; then
-                    run_tests "${MODEL}-gpu" cargo test -p ml --features cuda --test supervised_gpu_smoke_test -- --test-threads=1 "test_${MODEL}_gpu_smoke"
-                    # Also run model-specific integration tests if they exist
-                    if cargo test -p ml --features cuda --test "${MODEL}_integration" --no-run 2>/dev/null; then
-                      run_tests "${MODEL}-integ" cargo test -p ml --features cuda --test "${MODEL}_integration" -- --test-threads=1
-                    fi
-                  fi
-                  ;;
-              esac
-            done
-
-            # --- Summary ---
-            echo ""
-            echo "========================================"
-            echo "  GPU TEST RESULTS"
-            echo "========================================"
-            printf "$RESULTS" | while IFS=: read -r name status; do
-              [ -z "$name" ] && continue
-              printf "  %-25s %s\n" "$name" "$status"
-            done
-            echo "========================================"
-            echo "  Total failures: $FAILURES"
-            echo "========================================"
-
-            # Write results for notification step
-            mkdir -p /tmp/outputs
-            printf "$RESULTS" > /tmp/outputs/results
-            echo "$FAILURES" > /tmp/outputs/failures
-
-            [ "$FAILURES" -gt 0 ] && exit 1 || exit 0
-
-    # ── perf-benchmark: DQN epoch/s on 3Q data (performance regression guard) ──
-    # Runs after tests pass. Trains DQN on 3Q of ES.FUT data and reports epoch time.
-    # Uses the same binary compiled by compile-and-test (shared cargo-target PVC).
-    # Fails the pipeline if epoch time exceeds 500ms (regression threshold for H100).
-    - name: perf-benchmark
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      outputs:
-        parameters:
-          - name: epoch-ms
-            valueFrom:
-              path: /tmp/outputs/epoch-ms
-              default: "9999"
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CUDA_VISIBLE_DEVICES
-            value: "0"
-          - name: LD_LIBRARY_PATH
-            value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
-        volumeMounts:
-          - name: cargo-target
-            mountPath: /cargo-target
-          - name: test-data
-            mountPath: /data/test-data
-            readOnly: true
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 16Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 32Gi
-        args:
-          - |
-            set -e
-            cd /cargo-target/src
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            echo "========================================"
-            echo "  PERF BENCHMARK: DQN epoch/s (3Q ES.FUT)"
-            echo "========================================"
-
-            # Run 5 epochs × 100 steps on 3Q data (--train-months 3 fits in 1 quarter)
-            # Use --step-months 3 to avoid walk-forward window splits
-            OUTPUT=$(cargo run --release --example train_baseline_rl -p ml -- \
-              --model dqn \
-              --data-dir /data/test-data/ohlcv \
-              --mbp10-data-dir /data/test-data/mbp10 \
-              --trades-data-dir /data/test-data/trades \
-              --symbol ES.FUT \
-              --epochs 5 \
-              --train-months 3 --val-months 1 --test-months 1 --step-months 3 \
-              2>&1)
-
-            # Extract epoch times (skip epoch 1 which includes init)
-            EPOCH_TIMES=$(echo "$OUTPUT" | grep "phase breakdown" | grep -v "Epoch 1/" | \
-              sed 's/.*total=\([0-9]*\)ms.*/\1/' | head -4)
-
-            if [ -z "$EPOCH_TIMES" ]; then
-              echo "ERROR: No phase breakdown output found"
-              echo "$OUTPUT" | tail -20
-              mkdir -p /tmp/outputs
-              echo "9999" > /tmp/outputs/epoch-ms
-              exit 1
-            fi
-
-            # Compute average epoch time (epochs 2-5)
-            SUM=0
-            COUNT=0
-            for T in $EPOCH_TIMES; do
-              SUM=$((SUM + T))
-              COUNT=$((COUNT + 1))
-            done
-            AVG=$((SUM / COUNT))
-
-            echo ""
-            echo "  Epoch times (ms, excl. epoch 1): $EPOCH_TIMES"
-            echo "  Average: ${AVG}ms"
-            echo ""
-
-            mkdir -p /tmp/outputs
-            echo "$AVG" > /tmp/outputs/epoch-ms
-
-            # Regression guard: fail if avg epoch > 500ms on H100
-            THRESHOLD=500
-            if [ "$AVG" -gt "$THRESHOLD" ]; then
-              echo "PERF REGRESSION: ${AVG}ms > ${THRESHOLD}ms threshold"
-              echo "========================================"
-              exit 1
-            fi
-
-            echo "  PASS: ${AVG}ms <= ${THRESHOLD}ms threshold"
-            echo "========================================"
-
-    # ── notify-result: post test outcome to Mattermost (onExit) ──
-    - name: notify-result
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: WEBHOOK_URL
-            valueFrom:
-              secretKeyRef:
-                name: notification-webhook
-                key: webhook-url
-                optional: true
-        resources:
-          requests:
-            cpu: 50m
-            memory: 32Mi
-          limits:
-            cpu: 100m
-            memory: 64Mi
-        args:
-          - |
-            NOTIFY="{{workflow.parameters.notify}}"
-            if [ "$NOTIFY" != "true" ]; then
-              echo "Notifications disabled, skipping"
-              exit 0
-            fi
-
-            STATUS="{{workflow.status}}"
-            NAME="{{workflow.name}}"
-
-            if [ -z "$WEBHOOK_URL" ] || echo "$WEBHOOK_URL" | grep -q "PLACEHOLDER"; then
-              echo "No webhook configured, skipping notification"
-              exit 0
-            fi
-
-            if [ "$STATUS" = "Succeeded" ]; then
-              EMOJI=":white_check_mark:"
-            else
-              EMOJI=":x:"
-            fi
-
-            PAYLOAD="{\"username\":\"Argo CI\",\"text\":\"${EMOJI} **GPU Tests** ${NAME} — ${STATUS} ({{workflow.duration}}s)\"}"
-
-            curl -sf -X POST -H 'Content-Type: application/json' \
-              -d "$PAYLOAD" "$WEBHOOK_URL" || echo "WARN: webhook post failed"

infra/k8s/argo/kustomization.yaml

@@ -1,15 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - sccache-pvcs.yaml
-  - ci-pipeline-template.yaml
-  - compile-and-deploy-template.yaml
-  - train-template.yaml
-  - build-ci-image-template.yaml
-  - sanitizer-test-template.yaml
-  - nsys-test-template.yaml
-  - smoke-test-template.yaml
-  - refresh-deps-cache-template.yaml
-  - argo-workflow-netpol.yaml
-  - ci-deploy-rbac.yaml
-  - archive-rbac.yaml

infra/k8s/argo/lob-backtest-sweep-template.yaml

@@ -1,459 +0,0 @@
-# Real-LOB backtest sweep workflow.
-#
-# Fans out a parameter grid across N parallel GPU pods, each running
-# `fxt-backtest run` against the same MBP-10 data + checkpoint with one
-# cell's worth of overrides. A final CPU pod runs `fxt-backtest aggregate`
-# to produce aggregate.parquet + pareto_frontier.json at the sweep root.
-#
-# The `# __SWEEP_CELLS__` marker on the dag.tasks line is replaced by
-# scripts/argo-lob-sweep.sh with N generated WorkflowTask stanzas before
-# submission. Same convention as train-multi-seed-template.yaml's
-# # __MATRIX_TASKS__ marker.
-#
-# DAG:
-#   ensure-binary ──> [N parallel run-cell-<i> tasks on ci-training-l40s,
-#                       each with its own outputDir on the shared PVC]
-#                     │
-#                     └──> aggregate (CPU node, runs fxt-backtest aggregate
-#                                     against the sweep root)
-#
-# Per `feedback_default_to_l40s_pool.md` (2026-05-09): defaults to
-# ci-training-l40s + sm_89. Override via --gpu-pool ci-training-h100
-# for sm_90 / 80 GB.
-#
-# Usage:
-#   ./scripts/argo-lob-sweep.sh --grid config/ml/sweep_decision_stride_example.yaml
-#   ./scripts/argo-lob-sweep.sh --grid <path> --sha abc1234 --gpu-pool ci-training-h100
-#   ./scripts/argo-lob-sweep.sh --grid <path> --watch
-#   ./scripts/argo-lob-sweep.sh --grid <path> --dry-run > /tmp/wf.yaml
-
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: lob-backtest-sweep
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: lob-backtest-sweep
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train          # reuses argo-train-workflow NetworkPolicy egress (port 2222 to gitlab-shell)
-spec:
-  entrypoint: sweep-matrix
-  onExit: notify-result
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  # LOB sweep cells are much faster than training (~minutes vs hours);
-  # 2h walltime is generous even for N=128 cells.
-  activeDeadlineSeconds: 7200
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: git-branch
-        value: main
-      - name: cuda-compute-cap
-        value: "89"
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: data-root
-        value: /mnt/training-data/futures-baseline/ES.FUT
-      - name: predecoded-dir
-        value: /mnt/training-data/futures-baseline/ES.FUT
-      - name: checkpoint
-        value: ""               # empty = trunk runs with --seed (random init);
-                                # set to a path on the data PVC for trained weights
-      - name: sweep-root
-        value: /mnt/training-data/sweeps/lob-backtest
-      - name: sweep-tag
-        value: default          # subdirectory: <sweep-root>/<sweep-tag>/
-      # Operational knobs. argo-lob-sweep.sh sets these per-grid.
-      - name: n-parallel
-        value: "1"
-      - name: latency-ns
-        value: "100000000"
-      - name: target-annual-vol-units
-        value: "50.0"
-      - name: annualisation-factor
-        value: "825.0"
-      - name: max-lots
-        value: "5"
-      - name: max-events
-        value: "0"
-      # Per-step kernel-state JSONL trace path. Empty (default) = disabled:
-      # ensure-binary compiles fxt-backtest WITHOUT --features kernel-step-trace
-      # and run-cell omits the --kernel-step-trace flag. When set to a
-      # non-empty path, ensure-binary rebuilds with the feature enabled
-      # and run-cell passes --kernel-step-trace <path> through to the
-      # binary. Trace is written to that absolute path on the pod (must
-      # resolve to a mounted PVC — usually /feature-cache or /mnt/training-data).
-      - name: kernel-step-trace
-        value: ""
-
-  volumes:
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 0400
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  templates:
-    # ── DAG entrypoint ──────────────────────────────────────────────
-    - name: sweep-matrix
-      dag:
-        tasks:
-          - name: ensure-binary
-            template: ensure-binary
-          # __SWEEP_CELLS__ — replaced by argo-lob-sweep.sh
-          - name: aggregate
-            template: aggregate
-            dependencies: [ensure-binary]    # plus all run-cell-* via shell-injected list
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-
-    # ── ensure-binary: cache-or-compile fxt-backtest by commit SHA ─
-    # Same shape as train-multi-seed-template.yaml's ensure-binary but
-    # for the fxt-backtest binary only.
-    - name: ensure-binary
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: RUSTC_WRAPPER
-            value: sccache
-          - name: SCCACHE_DIR
-            value: /cargo-target/sccache
-          - name: SCCACHE_CACHE_SIZE
-            value: "40G"
-          - name: CARGO_INCREMENTAL
-            value: "0"
-          - name: FOXHUNT_CUDA_ARCH
-            value: "sm_{{workflow.parameters.cuda-compute-cap}}"
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: "14"
-            memory: 32Gi
-          limits:
-            cpu: "30"
-            memory: 64Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /mnt/training-data
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            BRANCH="{{workflow.parameters.git-branch}}"
-            KERNEL_TRACE="{{workflow.parameters.kernel-step-trace}}"
-
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            if [ "$SHA" = "HEAD" ]; then
-              if [ -d "$BUILD/.git" ]; then
-                cd "$BUILD"; git fetch origin
-                SHA=$(git rev-parse "origin/$BRANCH"); cd /
-              else
-                git clone --filter=blob:none "$REPO" "$BUILD"
-                cd "$BUILD"; git checkout "origin/$BRANCH"
-                SHA=$(git rev-parse HEAD); cd /
-              fi
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "Resolved SHA: $SHA (short: $SHORT_SHA)"
-
-            # Feature variant: when kernel-step-trace is enabled, build
-            # with the Cargo feature and cache under a distinct subdir
-            # so the default + diagnostic binaries don't clobber each other.
-            FEATURE_FLAGS=""
-            VARIANT="default"
-            if [ -n "$KERNEL_TRACE" ]; then
-              FEATURE_FLAGS="--features kernel-step-trace"
-              VARIANT="kstrace"
-            fi
-            BIN_DIR="/mnt/training-data/bin/$SHORT_SHA-$VARIANT"
-            mkdir -p "$BIN_DIR"
-            if [ -x "$BIN_DIR/fxt-backtest" ]; then
-              echo "=== Cache HIT: fxt-backtest ($VARIANT) present in $BIN_DIR ==="
-              ls -lh "$BIN_DIR/fxt-backtest"
-              echo "$SHORT_SHA-$VARIANT" > /tmp/sha
-              exit 0
-            fi
-
-            echo "=== Cache MISS: compiling fxt-backtest ($VARIANT) for $SHORT_SHA ==="
-            if [ -d "$BUILD/.git" ]; then
-              # Same pattern as alpha-perception-template.yaml: prior
-              # cargo build mutates Cargo.lock (and sometimes other
-              # generated files); --force checkout overwrites them,
-              # `git clean -fd` drops untracked. Without this, the
-              # second SHA's checkout fails on dirty working tree.
-              cd "$BUILD"; git fetch origin
-              git checkout --force "$SHA"; git clean -fd
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"; git checkout "$SHA"
-            fi
-
-            cargo build -p fxt-backtest --release $FEATURE_FLAGS
-            cp "$CARGO_TARGET_DIR/release/fxt-backtest" "$BIN_DIR/fxt-backtest"
-            echo "$SHORT_SHA-$VARIANT" > /tmp/sha
-            ls -lh "$BIN_DIR/fxt-backtest"
-
-    # ── run-cell: one sweep cell against a GPU pod ─────────────────
-    # Inputs: cell name + every Run arg that varies across the grid.
-    # The shell-rendered DAG tasks supply these per-cell.
-    - name: run-cell
-      inputs:
-        parameters:
-          - name: sha
-          - name: cell-name
-          - name: decision-stride
-            value: "4"
-          - name: latency-ns
-            value: "{{workflow.parameters.latency-ns}}"
-          - name: target-annual-vol-units
-            value: "{{workflow.parameters.target-annual-vol-units}}"
-          - name: annualisation-factor
-            value: "{{workflow.parameters.annualisation-factor}}"
-          - name: max-lots
-            value: "{{workflow.parameters.max-lots}}"
-          - name: max-events
-            value: "{{workflow.parameters.max-events}}"
-          - name: seed
-            value: "0xC0FFEE"
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        resources:
-          requests:
-            cpu: "4"
-            memory: 16Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "8"
-            memory: 32Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /mnt/training-data
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            CELL="{{inputs.parameters.cell-name}}"
-            BIN="/mnt/training-data/bin/$SHA/fxt-backtest"
-            CKPT="{{workflow.parameters.checkpoint}}"
-            KERNEL_TRACE="{{workflow.parameters.kernel-step-trace}}"
-            OUT="{{workflow.parameters.sweep-root}}/{{workflow.parameters.sweep-tag}}/$CELL"
-            mkdir -p "$OUT"
-            echo "=== sweep cell $CELL on $(hostname) → $OUT ==="
-
-            CKPT_FLAG=""
-            if [ -n "$CKPT" ]; then
-              CKPT_FLAG="--checkpoint $CKPT"
-            fi
-            KERNEL_TRACE_FLAG=""
-            if [ -n "$KERNEL_TRACE" ]; then
-              # Per-cell trace path: append cell name so concurrent run-cell
-              # pods don't clobber each other's JSONL files. Operators get
-              # one trace per cell; aggregate by reading <root>/<sweep>/<cell>/
-              # at analysis time.
-              KERNEL_TRACE_FLAG="--kernel-step-trace $OUT/kernel_step_trace.jsonl"
-            fi
-
-            "$BIN" run \
-              --data "{{workflow.parameters.data-root}}" \
-              --predecoded-dir "{{workflow.parameters.predecoded-dir}}" \
-              --n-parallel "{{workflow.parameters.n-parallel}}" \
-              --decision-stride "{{inputs.parameters.decision-stride}}" \
-              --latency-ns "{{inputs.parameters.latency-ns}}" \
-              --target-annual-vol-units "{{inputs.parameters.target-annual-vol-units}}" \
-              --annualisation-factor "{{inputs.parameters.annualisation-factor}}" \
-              --max-lots "{{inputs.parameters.max-lots}}" \
-              --max-events "{{inputs.parameters.max-events}}" \
-              --seed "{{inputs.parameters.seed}}" \
-              $CKPT_FLAG \
-              $KERNEL_TRACE_FLAG \
-              --out "$OUT"
-
-            echo "=== cell $CELL done ==="
-            ls -lh "$OUT"
-
-    # ── run-sweep: P6 batched flow. Invokes `fxt-backtest sweep` ───
-    # against the FULL grid YAML inside a single GPU pod. When the YAML
-    # carries sim_variants, each cell expands into n_parallel=variants.len()
-    # backtests sharing one forward pass. argo-lob-sweep.sh emits ONE
-    # run-sweep task (instead of N fan-out run-cell tasks) when --batched
-    # is set or sim_variants is detected in the YAML.
-    - name: run-sweep
-      inputs:
-        parameters:
-          - name: sha
-          - name: grid-yaml-b64        # base64-encoded grid YAML, written to a file in-pod
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        resources:
-          requests:
-            cpu: "4"
-            memory: 16Gi
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "8"
-            memory: 32Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /mnt/training-data
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BIN="/mnt/training-data/bin/$SHA/fxt-backtest"
-            OUT="{{workflow.parameters.sweep-root}}/{{workflow.parameters.sweep-tag}}"
-            mkdir -p "$OUT"
-            # Write the grid YAML to a known path. base64 in/out keeps Argo
-            # parameter encoding stable across YAML special characters
-            # (curly braces from data_template + sim_variants).
-            GRID_YAML=/tmp/sweep-grid.yaml
-            echo "{{inputs.parameters.grid-yaml-b64}}" | base64 -d > "$GRID_YAML"
-            echo "=== running fxt-backtest sweep on $(hostname) → $OUT ==="
-            echo "=== grid yaml: ==="
-            head -20 "$GRID_YAML"
-            echo "=== ... ==="
-
-            "$BIN" sweep --grid "$GRID_YAML" --out "$OUT"
-
-            echo "=== sweep complete ==="
-            ls -lh "$OUT"
-
-    # ── aggregate: runs fxt-backtest aggregate on the GPU pool. ───
-    # The binary is dynamically linked against libcuda.so.1, which the
-    # ci-compile-cpu pool's host doesn't expose; the aggregate workload
-    # itself is CPU-only (~seconds) but the binary needs the driver libs.
-    # Reusing the GPU pool is cheap (single fast pod) and avoids needing
-    # a separate CPU-only binary.
-    - name: aggregate
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-training-runtime:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        resources:
-          requests:
-            cpu: "2"
-            memory: 4Gi
-            # Request a GPU only to make the Scaleway L40S device plugin
-            # mount libcuda.so.1 into the container — aggregate logic is
-            # CPU-only (~seconds). Without the request, the binary's
-            # dynamic loader fails before main(). Cheapest fix; the GPU
-            # is held for ~5s of CPU work, then released.
-            nvidia.com/gpu: "1"
-          limits:
-            cpu: "4"
-            memory: 8Gi
-            nvidia.com/gpu: "1"
-        volumeMounts:
-          - name: training-data
-            mountPath: /mnt/training-data
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BIN="/mnt/training-data/bin/$SHA/fxt-backtest"
-            SWEEP_DIR="{{workflow.parameters.sweep-root}}/{{workflow.parameters.sweep-tag}}"
-            echo "=== aggregate sweep at $SWEEP_DIR ==="
-            "$BIN" aggregate "$SWEEP_DIR"
-            ls -lh "$SWEEP_DIR"
-
-    # ── notify-result: exit hook (placeholder; real impl emits to Slack/MinIO) ─
-    - name: notify-result
-      container:
-        image: alpine:3.20
-        command: ["sh", "-c"]
-        args:
-          - |
-            echo "lob-backtest-sweep workflow {{workflow.name}} finished with status {{workflow.status}}"
-            echo "results under {{workflow.parameters.sweep-root}}/{{workflow.parameters.sweep-tag}}"

infra/k8s/argo/notification-webhook.yaml

@@ -1,11 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-  name: notification-webhook
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-type: Opaque
-stringData:
-  # Mattermost incoming webhook URL — update after Mattermost is deployed
-  webhook-url: "http://mattermost.foxhunt.svc.cluster.local:8065/hooks/PLACEHOLDER"

infra/k8s/argo/nsys-test-template.yaml

@@ -1,318 +0,0 @@
-# infra/k8s/argo/nsys-test-template.yaml
-#
-# One-shot Nsight Systems profiling run on L40S. Wraps a smoke test under
-# `nsys profile` and uploads the .nsys-rep to MinIO at
-# foxhunt-training-results/profiles/smoke/<short-sha>/.
-#
-# Mirrors the nsys integration in train-multi-seed-template (Plan 5 Task 3,
-# A.4.1) but for smoke tests. Use this for rapid kernel-timing iteration
-# (per-step bottleneck hunts, SOL %, occupancy diffs) without the full
-# multi-seed Argo cycle.
-#
-# Why L40S not local: laptop RTX 3050 Ti has 4 GB VRAM; nsys on L40S (48 GB)
-# avoids any contention and runs at near-native speed (typical ~5-10%
-# slowdown vs uninstrumented).
-#
-# Usage:
-#   argo submit --watch -n foxhunt nsys-test-template.yaml \
-#     -p commit-ref=main \
-#     -p test-name=performance::test_real_data_single_epoch
-#
-# Or via the wrapper script: scripts/argo-nsys.sh
-
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: nsys-test
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: nsys-test
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: nsys-run
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: nsys-test
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 7200
-  activeDeadlineSeconds: 3600  # nsys overhead is small — 1h cap is generous
-  arguments:
-    parameters:
-      - name: commit-ref
-        value: HEAD
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: cuda-compute-cap
-        value: "89"
-      - name: test-name
-        value: "performance::test_real_data_single_epoch"
-      # nsys capture options. Defaults capture the full kernel timeline.
-      # GPU metrics counters (--gpu-metrics-devices) require elevated CUDA
-      # performance-counter privileges (NVGPUCTRPERM) which the container
-      # lacks; CUDA/NVTX/osrt traces alone still give per-kernel timing and
-      # call-graph context — sufficient for the bottleneck-hunt use case.
-      # Override via --extra-args if running on a node with relaxed perms.
-      - name: nsys-extra-args
-        value: "--trace=cuda,nvtx,osrt"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: cargo-target
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda-test
-    # Use the same training-data-pvc as production training (full 27 months
-    # of OHLCV + MBP-10 + trades) instead of the smaller test-data-pvc.
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-        readOnly: true
-
-  templates:
-    - name: nsys-run
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CUDA_VISIBLE_DEVICES
-            value: "0"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_TEST_DATA
-            value: /data/futures-baseline
-          - name: TEST_DATA_DIR
-            value: /data/futures-baseline
-          # MBP-10 + trades are mandatory per feedback_mbp10_mandatory.md
-          # (OFI features are part of the production state vector).
-          - name: FOXHUNT_MBP10_DATA
-            value: /data/futures-baseline-mbp10
-          - name: FOXHUNT_TRADES_DATA
-            value: /data/futures-baseline-trades
-          - name: RUST_LOG
-            value: info
-          - name: LD_LIBRARY_PATH
-            value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
-          # MinIO upload — same secret + optional pattern as train-multi-seed.
-          # Both refs are `optional: true` so env mount succeeds on clusters
-          # that do not pre-create `minio-credentials` (upload then warn-fails).
-          - name: MINIO_ACCESS_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: access-key
-                optional: true
-          - name: MINIO_SECRET_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: secret-key
-                optional: true
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        args:
-          - |
-            set -euo pipefail
-
-            REF="{{workflow.parameters.commit-ref}}"
-            TEST_NAME="{{workflow.parameters.test-name}}"
-            EXTRA_ARGS="{{workflow.parameters.nsys-extra-args}}"
-
-            echo "==================================="
-            echo "  nsys profile L40S run"
-            echo "==================================="
-            echo "  Ref:      $REF"
-            echo "  Test:     $TEST_NAME"
-            echo "  Extra:    $EXTRA_ARGS"
-            echo "  Pool:     {{workflow.parameters.gpu-pool}}"
-            echo "==================================="
-
-            # --- SSH setup ---
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            # --- Persistent checkout on PVC ---
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                TARGET=$(git rev-parse "origin/$REF")
-              else
-                TARGET=$(git rev-parse --verify "$REF" 2>/dev/null || echo "$REF")
-              fi
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$TARGET" ]; then
-                echo "=== Updating checkout to $REF ($TARGET) ==="
-                git checkout --force --detach "$TARGET"
-                git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                git checkout --force --detach "origin/$REF"
-              else
-                git checkout "$REF"
-              fi
-            fi
-            SHA=$(git rev-parse --short HEAD)
-            echo "Checked out $SHA"
-
-            export PATH="${CARGO_HOME}/bin:/usr/local/cuda/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            bash scripts/ptx-cache-invalidate.sh "${CARGO_TARGET_DIR}/.ptx_cache"
-
-            # --- Verify nsys ---
-            NSYS_BIN=$(which nsys 2>/dev/null || echo /usr/local/cuda/bin/nsys)
-            if [ ! -x "$NSYS_BIN" ]; then
-              echo "ERROR: nsys not found at $NSYS_BIN"
-              exit 2
-            fi
-            echo "=== nsys version ==="
-            "$NSYS_BIN" --version | head -3
-
-            # --- Compile test binary + train_baseline_rl ---
-            echo "=== Compiling (--release --no-run + train_baseline_rl) ==="
-            cargo test -p ml --release --lib --no-run 2>&1 | tee /cargo-target/nsys-compile.log
-            cargo build -p ml --release --example train_baseline_rl 2>&1 | tee -a /cargo-target/nsys-compile.log
-
-            # Honour CARGO_TARGET_DIR — outputs go to ${CARGO_TARGET_DIR}/release/deps,
-            # NOT the repo-relative target/. (CI workflow sets CARGO_TARGET_DIR=/cargo-target.)
-            # Subshell + || true wraps the head -1 SIGPIPE so pipefail doesn't fire.
-            TARGET_DIR="${CARGO_TARGET_DIR:-target}"
-            TEST_BIN=$( (ls -t "$TARGET_DIR/release/deps/ml-"* 2>/dev/null \
-              | grep -vE '\.(d|rcgu|rmeta|rlib|so|json)$' \
-              | head -1) || true )
-            if [ -z "$TEST_BIN" ] || [ ! -x "$TEST_BIN" ]; then
-              echo "ERROR: could not locate compiled test binary in $TARGET_DIR/release/deps/"
-              ls -la "$TARGET_DIR/release/deps/" 2>&1 | head -20
-              exit 3
-            fi
-            echo "=== Test binary: $TEST_BIN ==="
-
-            # grep -q + pipefail = SIGPIPE on early exit. Capture once, grep
-            # without -q so the consumer reads all input.
-            TEST_LIST=$("$TEST_BIN" --list 2>&1 || true)
-            if ! echo "$TEST_LIST" | grep -F "$TEST_NAME" > /dev/null; then
-              echo "ERROR: test '$TEST_NAME' not found"
-              echo "$TEST_LIST" | grep smoke_tests | (head -30 || true)
-              exit 4
-            fi
-
-            # --- Run under nsys profile ---
-            mkdir -p /tmp/nsys-output
-            POD_NAME="${HOSTNAME:-pod}"
-            NSYS_OUT="/tmp/nsys-output/profile-${POD_NAME}.nsys-rep"
-
-            echo ""
-            echo "==================================="
-            echo "  Launching nsys profile..."
-            echo "==================================="
-            START_TS=$(date +%s)
-            set +e
-            "$NSYS_BIN" profile \
-              --output="$NSYS_OUT" \
-              --force-overwrite=true \
-              --stats=true \
-              ${EXTRA_ARGS} \
-              -- "$TEST_BIN" "$TEST_NAME" --ignored --nocapture
-            EXIT_CODE=$?
-            set -e
-            END_TS=$(date +%s)
-            ELAPSED=$((END_TS - START_TS))
-
-            echo ""
-            echo "==================================="
-            echo "  nsys run complete"
-            echo "==================================="
-            echo "  Exit code:  $EXIT_CODE"
-            echo "  Elapsed:    ${ELAPSED}s"
-            ls -la "$NSYS_OUT" 2>/dev/null && \
-              echo "  Profile:    $(du -h "$NSYS_OUT" | cut -f1)"
-            echo ""
-
-            # --- Upload to MinIO (same pattern as train-multi-seed) ---
-            if [ -f "$NSYS_OUT" ]; then
-              echo "=== Uploading nsys profile to MinIO ==="
-              MC_BIN=$(which mc 2>/dev/null || echo "")
-              if [ -z "$MC_BIN" ]; then
-                MC_BIN=/tmp/mc
-                curl -fsSL https://dl.min.io/client/mc/release/linux-amd64/mc -o "$MC_BIN" || {
-                  echo "WARN: failed to download mc — skipping upload"
-                  echo "Profile available locally at: $NSYS_OUT (PVC: cargo-target)"
-                  exit 0
-                }
-                chmod +x "$MC_BIN"
-              fi
-              "$MC_BIN" alias set foxhunt http://minio.foxhunt.svc.cluster.local:9000 \
-                "${MINIO_ACCESS_KEY:-}" "${MINIO_SECRET_KEY:-}" 2>/dev/null || true
-              # Smoke profiles go to a separate prefix to keep the
-              # production training profile bucket clean.
-              UPLOAD_PATH="foxhunt/foxhunt-training-results/profiles/smoke/${SHA}/profile-${POD_NAME}.nsys-rep"
-              if "$MC_BIN" cp "$NSYS_OUT" "$UPLOAD_PATH"; then
-                echo "=== nsys profile uploaded to ${UPLOAD_PATH} ==="
-                echo ""
-                echo "Download with:"
-                echo "  mc cp ${UPLOAD_PATH} ./profile.nsys-rep"
-                echo "Open with: nsys-ui profile.nsys-rep"
-              else
-                echo "WARN: nsys upload failed — profile remains on PVC at $NSYS_OUT"
-              fi
-            fi
-
-            if [ "$EXIT_CODE" -ne 0 ]; then
-              echo ""
-              echo "FAIL: test exited with code $EXIT_CODE"
-              exit "$EXIT_CODE"
-            fi
-            echo ""
-            echo "PASS: nsys profile captured, test passed."

infra/k8s/argo/refresh-deps-cache-template.yaml

@@ -1,181 +0,0 @@
-# refresh-deps-cache: rebuild ci-builder-cpu-with-deps:nightly image
-#
-# Triggered:
-#   - nightly via CronWorkflow (see CronWorkflow at the bottom of this file)
-#   - manually via:  argo submit --from workflowtemplate/refresh-deps-cache -n foxhunt
-#
-# What it does: clones the repo at HEAD of main, then runs Kaniko to build
-# infra/docker/Dockerfile.ci-deps-cache and pushes to
-# gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu-with-deps:nightly.
-#
-# The compile-and-deploy workflow's seed-deps-cache initContainer pulls
-# this image and rsyncs its /cargo-target-prebuilt/ into the cargo-target-cpu PVC.
----
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: refresh-deps-cache
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: refresh-deps-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  activeDeadlineSeconds: 7200  # 2 hours; full workspace cargo build is slow
-  serviceAccountName: argo-workflow
-  entrypoint: build
-  podMetadata:
-    labels:
-      # Reuse ci-pipeline label so the pod inherits argo-ci-pipeline netpol egress
-      # clone, gitlab-registry:5000 for image push, crates.io HTTPS for cargo
-      # NetworkPolicy clone — same egress targets, no functional difference.
-      app.kubernetes.io/component: ci-pipeline
-      app.kubernetes.io/part-of: foxhunt
-  ttlStrategy:
-    secondsAfterCompletion: 7200
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: deps-version
-        value: "1"
-      - name: image-tag
-        value: nightly
-
-  templates:
-    - name: build
-      inputs:
-        parameters:
-          - name: commit-sha
-            value: "{{workflow.parameters.commit-sha}}"
-          - name: deps-version
-            value: "{{workflow.parameters.deps-version}}"
-          - name: image-tag
-            value: "{{workflow.parameters.image-tag}}"
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      volumes:
-        - name: workspace
-          emptyDir:
-            sizeLimit: 30Gi  # full workspace + target/release ~10GB
-        - name: git-ssh-key
-          secret:
-            secretName: argo-git-ssh-key
-            defaultMode: 256
-        - name: registry-auth
-          secret:
-            secretName: gitlab-registry
-            items:
-              - key: .dockerconfigjson
-                path: config.json
-      initContainers:
-        - name: git-clone
-          image: alpine/git:latest
-          command: ["/bin/sh", "-c"]
-          resources:
-            requests:
-              cpu: 200m
-              memory: 256Mi
-            limits:
-              cpu: "1"
-              memory: 512Mi
-          volumeMounts:
-            - name: git-ssh-key
-              mountPath: /etc/git-ssh
-              readOnly: true
-            - name: workspace
-              mountPath: /workspace
-          args:
-            - |
-              set -ex
-              mkdir -p /root/.ssh
-              cp /etc/git-ssh/ssh-privatekey /root/.ssh/id_ed25519
-              chmod 600 /root/.ssh/id_ed25519
-              printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > /root/.ssh/config
-              chmod 600 /root/.ssh/config
-
-              SHA="{{inputs.parameters.commit-sha}}"
-              REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-              git clone --no-checkout --filter=blob:none "$REPO" /workspace/src
-              cd /workspace/src
-              git checkout "$SHA"
-              echo "Checked out $(git rev-parse --short HEAD)"
-      container:
-        image: gcr.io/kaniko-project/executor:debug
-        command: ["/busybox/sh", "-c"]
-        env:
-          - name: DOCKER_CONFIG
-            value: /kaniko/.docker
-        # Larger than build-ci-image because the inner cargo build is heavy.
-        resources:
-          requests:
-            cpu: "8"
-            memory: 16Gi
-          limits:
-            cpu: "16"
-            memory: 32Gi
-        volumeMounts:
-          - name: registry-auth
-            mountPath: /kaniko/.docker
-            readOnly: true
-          - name: workspace
-            mountPath: /workspace
-        args:
-          - |
-            /kaniko/executor \
-              --context=/workspace/src \
-              --dockerfile=/workspace/src/infra/docker/Dockerfile.ci-deps-cache \
-              --destination=gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder-cpu-with-deps:{{inputs.parameters.image-tag}} \
-              --build-arg=DEPS_VERSION={{inputs.parameters.deps-version}} \
-              --build-arg=BASE_TAG=latest \
-              --insecure-registry=gitlab-registry.foxhunt.svc.cluster.local:5000 \
-              --skip-tls-verify-registry=gitlab-registry.foxhunt.svc.cluster.local:5000 \
-              --cache=true \
-              --cache-repo=gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/cache \
-              --snapshot-mode=redo
-
----
-# Nightly cron — rebuilds the deps cache image at 03:00 UTC.
-# Suspended by default; enable with:
-#   kubectl -n foxhunt patch cronworkflow refresh-deps-cache-nightly \
-#     -p '{"spec":{"suspend":false}}' --type=merge
-apiVersion: argoproj.io/v1alpha1
-kind: CronWorkflow
-metadata:
-  name: refresh-deps-cache-nightly
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: refresh-deps-cache-nightly
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  schedules:
-    - "0 3 * * *"
-  timezone: UTC
-  concurrencyPolicy: Forbid
-  successfulJobsHistoryLimit: 2
-  failedJobsHistoryLimit: 3
-  suspend: true
-  workflowSpec:
-    entrypoint: trigger
-    serviceAccountName: argo-workflow
-    ttlStrategy:
-      secondsAfterCompletion: 7200
-    templates:
-      - name: trigger
-        steps:
-          - - name: refresh
-              templateRef:
-                name: refresh-deps-cache
-                template: build
-              arguments:
-                parameters:
-                  - name: commit-sha
-                    value: HEAD
-                  - name: deps-version
-                    value: "1"
-                  - name: image-tag
-                    value: nightly

infra/k8s/argo/sanitizer-test-template.yaml

@@ -1,311 +0,0 @@
-# infra/k8s/argo/sanitizer-test-template.yaml
-#
-# One-shot compute-sanitizer run on L40S. Wraps a smoke test under
-# `compute-sanitizer --tool memcheck` to validate no CUDA memory errors
-# (out-of-bounds, leaks, sync violations, race conditions).
-#
-# Why L40S not local: the laptop RTX 3050 Ti (4 GB VRAM) cannot fit
-# compute-sanitizer's instrumentation metadata alongside the production
-# training workload — the sanitizer falls back to "didn't track the
-# launch" with 60k+ internal-allocation errors. L40S (48 GB) has ample
-# headroom for memcheck's 2-3× shadow-memory overhead.
-#
-# Usage:
-#   argo submit --watch -n foxhunt sanitizer-test-template.yaml \
-#     -p commit-ref=main \
-#     -p test-name=iqn_quantile_monotonicity::iqn_multi_quantile_heads_produce_monotonic_estimates \
-#     -p sanitizer-tool=memcheck
-#
-# Or via the wrapper script: scripts/argo-sanitizer.sh
-
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: sanitizer-test
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: sanitizer-test
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: sanitizer-run
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: sanitizer-test
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 7200
-  activeDeadlineSeconds: 7200  # 2h cap — sanitizer is slow but bounded
-  arguments:
-    parameters:
-      - name: commit-ref
-        value: HEAD
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: cuda-compute-cap
-        value: "89"  # L40S Ada Lovelace
-      - name: test-name
-        value: "iqn_quantile_monotonicity::iqn_multi_quantile_heads_produce_monotonic_estimates"
-      - name: sanitizer-tool
-        value: memcheck  # memcheck | racecheck | synccheck | initcheck
-      - name: sanitizer-extra-args
-        value: ""
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: cargo-target
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda-test
-    # Use the same training-data-pvc as production training (full 27 months
-    # of OHLCV + MBP-10 + trades) instead of the smaller test-data-pvc which
-    # only has 3-4 months of MBP-10 — walk-forward needs ≥10 months.
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-        readOnly: true
-
-  templates:
-    - name: sanitizer-run
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CUDA_VISIBLE_DEVICES
-            value: "0"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_TEST_DATA
-            value: /data/futures-baseline
-          - name: TEST_DATA_DIR
-            value: /data/futures-baseline
-          # MBP-10 + trades are mandatory per feedback_mbp10_mandatory.md
-          # (OFI features are part of the production state vector).
-          - name: FOXHUNT_MBP10_DATA
-            value: /data/futures-baseline-mbp10
-          - name: FOXHUNT_TRADES_DATA
-            value: /data/futures-baseline-trades
-          - name: RUST_LOG
-            value: info
-          - name: LD_LIBRARY_PATH
-            value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        args:
-          - |
-            set -euo pipefail
-
-            REF="{{workflow.parameters.commit-ref}}"
-            TEST_NAME="{{workflow.parameters.test-name}}"
-            SANITIZER_TOOL="{{workflow.parameters.sanitizer-tool}}"
-            EXTRA_ARGS="{{workflow.parameters.sanitizer-extra-args}}"
-
-            echo "==================================="
-            echo "  compute-sanitizer L40S run"
-            echo "==================================="
-            echo "  Ref:      $REF"
-            echo "  Test:     $TEST_NAME"
-            echo "  Tool:     $SANITIZER_TOOL"
-            echo "  Pool:     {{workflow.parameters.gpu-pool}}"
-            echo "  Compute:  {{workflow.parameters.cuda-compute-cap}}"
-            echo "==================================="
-
-            # --- SSH setup (same as gpu-test-pipeline) ---
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            # --- Persistent checkout on PVC ---
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                TARGET=$(git rev-parse "origin/$REF")
-              else
-                TARGET=$(git rev-parse --verify "$REF" 2>/dev/null || echo "$REF")
-              fi
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$TARGET" ]; then
-                echo "=== Updating checkout to $REF ($TARGET) ==="
-                git checkout --force --detach "$TARGET"
-                git clean -fd
-              fi
-            else
-              echo "=== Initial clone ==="
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                git checkout --force --detach "origin/$REF"
-              else
-                git checkout "$REF"
-              fi
-            fi
-            echo "Checked out $(git rev-parse --short HEAD)"
-
-            export PATH="${CARGO_HOME}/bin:/usr/local/cuda/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            # --- PTX cache invalidation ---
-            bash scripts/ptx-cache-invalidate.sh "${CARGO_TARGET_DIR}/.ptx_cache"
-
-            # --- Verify compute-sanitizer is available ---
-            if ! command -v compute-sanitizer >/dev/null 2>&1; then
-              echo "ERROR: compute-sanitizer not found in PATH. CUDA toolkit incomplete."
-              exit 2
-            fi
-            echo "=== compute-sanitizer version ==="
-            compute-sanitizer --version | head -3
-
-            # --- Compile test binary + train_baseline_rl example ---
-            # train_baseline_rl is needed for multi_fold_convergence test which
-            # spawns it as a subprocess via `cargo run --example`. Pre-building
-            # avoids cargo doing it inside the sanitizer-instrumented run.
-            echo "=== Compiling test binary + train_baseline_rl (--release) ==="
-            cargo test -p ml --release --lib --no-run 2>&1 | tee /cargo-target/sanitizer-compile.log
-            cargo build -p ml --release --example train_baseline_rl 2>&1 | tee -a /cargo-target/sanitizer-compile.log
-
-            # Locate the freshest ml-* test binary
-            # Honour CARGO_TARGET_DIR — outputs go to ${CARGO_TARGET_DIR}/release/deps,
-            # NOT the repo-relative target/. (CI workflow sets CARGO_TARGET_DIR=/cargo-target.)
-            # Subshell + || true wraps the head -1 SIGPIPE so pipefail doesn't fire.
-            TARGET_DIR="${CARGO_TARGET_DIR:-target}"
-            TEST_BIN=$( (ls -t "$TARGET_DIR/release/deps/ml-"* 2>/dev/null \
-              | grep -vE '\.(d|rcgu|rmeta|rlib|so|json)$' \
-              | head -1) || true )
-            if [ -z "$TEST_BIN" ] || [ ! -x "$TEST_BIN" ]; then
-              echo "ERROR: could not locate compiled test binary in $TARGET_DIR/release/deps/"
-              ls -la "$TARGET_DIR/release/deps/" 2>&1 | head -20
-              exit 3
-            fi
-            echo "=== Test binary: $TEST_BIN ==="
-
-            # --- Verify the test exists in the binary ---
-            # grep -q + pipefail = SIGPIPE on early exit. Capture once, grep
-            # without -q so the consumer reads all input. Same below for nsys.
-            TEST_LIST=$("$TEST_BIN" --list 2>&1 || true)
-            if ! echo "$TEST_LIST" | grep -F "$TEST_NAME" > /dev/null; then
-              echo "ERROR: test '$TEST_NAME' not found in binary."
-              echo "Available smoke tests:"
-              echo "$TEST_LIST" | grep smoke_tests | (head -30 || true)
-              exit 4
-            fi
-
-            # --- Run under compute-sanitizer ---
-            mkdir -p /tmp/sanitizer-output
-            LOG_FILE=/tmp/sanitizer-output/${SANITIZER_TOOL}.log
-
-            echo ""
-            echo "==================================="
-            echo "  Launching compute-sanitizer..."
-            echo "==================================="
-            START_TS=$(date +%s)
-            # --target-processes all so multi_fold_convergence's spawned
-            # `train_baseline_rl` subprocess is also instrumented (the test
-            # itself does no GPU work — the child binary does).
-            set +e
-            compute-sanitizer \
-              --tool "$SANITIZER_TOOL" \
-              --target-processes all \
-              --launch-timeout 1200 \
-              --error-exitcode 99 \
-              --print-limit 200 \
-              --log-file "$LOG_FILE" \
-              ${EXTRA_ARGS} \
-              -- "$TEST_BIN" "$TEST_NAME" --ignored --nocapture
-            EXIT_CODE=$?
-            set -e
-            END_TS=$(date +%s)
-            ELAPSED=$((END_TS - START_TS))
-
-            echo ""
-            echo "==================================="
-            echo "  Sanitizer run complete"
-            echo "==================================="
-            echo "  Exit code:  $EXIT_CODE"
-            echo "  Elapsed:    ${ELAPSED}s"
-            echo "  Log:        $LOG_FILE ($(wc -l <"$LOG_FILE") lines)"
-            echo ""
-
-            # --- Triage report ---
-            echo "=== Sanitizer error summary ==="
-            grep -E "ERROR SUMMARY|Internal Sanitizer Error" "$LOG_FILE" | tail -5 || echo "  (no summary line)"
-            echo ""
-
-            # Categorise errors. compute-sanitizer prints "ERROR SUMMARY: N errors" at the
-            # end. "Internal Sanitizer Error" is sanitizer-internal (allocation failure,
-            # tracking gap) — NOT a real bug in the application code.
-            INTERNAL_ERRORS=$(grep -c "Internal Sanitizer Error" "$LOG_FILE" || true)
-            REAL_ERROR_LINE=$(grep "ERROR SUMMARY:" "$LOG_FILE" | tail -1 || echo "")
-            REAL_ERRORS=$(echo "$REAL_ERROR_LINE" | grep -oE "[0-9]+ errors" | head -1 | grep -oE "[0-9]+" || echo "0")
-
-            echo "=== Triage ==="
-            echo "  Internal Sanitizer Errors (instrumentation gaps): $INTERNAL_ERRORS"
-            echo "  Reported errors total:                            $REAL_ERRORS"
-            echo "  Real (non-internal) errors:                       $((REAL_ERRORS - INTERNAL_ERRORS))"
-            echo ""
-
-            # Show first 50 real (non-internal) errors with context
-            echo "=== First non-internal error excerpts ==="
-            grep -vE "Internal Sanitizer Error|^=========\s*$" "$LOG_FILE" \
-              | grep -E "^=========" \
-              | head -50 || echo "  (none)"
-
-            # Final pass/fail
-            if [ "$EXIT_CODE" -eq 99 ]; then
-              echo ""
-              echo "FAIL: compute-sanitizer detected real memory errors (--error-exitcode 99 fired)."
-              exit 1
-            elif [ "$EXIT_CODE" -ne 0 ]; then
-              echo ""
-              echo "FAIL: test process exited with code $EXIT_CODE (test failure or sanitizer abort)."
-              exit "$EXIT_CODE"
-            else
-              echo ""
-              echo "PASS: zero real memory errors detected, test passed."
-            fi

infra/k8s/argo/sccache-pvcs.yaml

@@ -1,32 +0,0 @@
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: sccache-cpu
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: sccache
-    app.kubernetes.io/component: ci-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 20Gi
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: sccache-cuda
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: sccache
-    app.kubernetes.io/component: ci-cache
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd-retain
-  resources:
-    requests:
-      storage: 20Gi

infra/k8s/argo/smoke-test-template.yaml

@@ -1,308 +0,0 @@
-# infra/k8s/argo/smoke-test-template.yaml
-#
-# Plain smoke-test runner on L40S — no compute-sanitizer, no nsys profile,
-# no MinIO artefact upload. Mirrors the compile + checkout + GPU + data-mount
-# layout of nsys-test-template.yaml but executes the test binary directly.
-#
-# Use this for fast multi-fold / single-epoch validation runs against the
-# full training-data PVC (27 months) when local laptop data is too short
-# (laptop only has 1 quarter of MBP-10/trades).
-#
-# Why a separate template: the nsys/sanitizer wrappers add overhead and
-# extract artefacts not needed for a straight "did the test pass on the
-# real dataset" gate. Keeping the smoke variant lean keeps the iteration
-# cycle short (no .nsys-rep upload, no sanitizer instrumentation slowdown).
-#
-# Usage:
-#   argo submit --watch -n foxhunt --from=wftmpl/smoke-test \
-#     -p commit-ref=main \
-#     -p test-name=multi_fold_convergence::test_multi_fold_convergence
-#
-# Or via the wrapper script: scripts/argo-smoke.sh
-
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: smoke-test
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: smoke-test
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  entrypoint: smoke-run
-  serviceAccountName: argo-workflow
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: smoke-test
-  archiveLogs: true
-  podGC:
-    strategy: OnPodCompletion
-  ttlStrategy:
-    secondsAfterCompletion: 7200
-  activeDeadlineSeconds: 3600
-  arguments:
-    parameters:
-      - name: commit-ref
-        value: HEAD
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: cuda-compute-cap
-        value: "89"
-      - name: test-name
-        value: "multi_fold_convergence::test_multi_fold_convergence"
-      # Wipe `/cargo-target/release` for the ml + ml-dqn crates before
-      # compile so the smoke binary is built from a known-clean state. The
-      # PVC's persistent target dir accumulates rmeta/object artefacts
-      # across probes; file deletions (e.g., `regime_conditional.rs` in
-      # ff00af68a) can leave dangling references that perturb downstream
-      # codegen between bisect runs. Defaults to "false" — only set when
-      # bisecting suspected build-cache contamination.
-      - name: clean-cache
-        value: "false"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: cargo-target
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda-test
-    # Same training-data-pvc as production training (full 27 months of
-    # OHLCV + MBP-10 + trades) — laptop's 1-quarter MBP-10 truncates the
-    # fxcache below the smoke's 10-month minimum.
-    #
-    # RW (not readOnly) so the smoke can populate `/data/bin/$SHORT_SHA/`
-    # with the train binaries it just built, letting the next train run's
-    # `ensure-binary` cache check (train-multi-seed-template.yaml:235-246)
-    # hit and skip the 6-min compile. Read-side (fxcache + raw market data)
-    # is unchanged.
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-
-  templates:
-    - name: smoke-run
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CUDA_VISIBLE_DEVICES
-            value: "0"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_TEST_DATA
-            value: /data/futures-baseline
-          - name: TEST_DATA_DIR
-            value: /data/futures-baseline
-          # MBP-10 + trades are mandatory per feedback_mbp10_mandatory.md
-          # (OFI features are part of the production state vector).
-          - name: FOXHUNT_MBP10_DATA
-            value: /data/futures-baseline-mbp10
-          - name: FOXHUNT_TRADES_DATA
-            value: /data/futures-baseline-trades
-          - name: RUST_LOG
-            value: info
-          - name: LD_LIBRARY_PATH
-            value: /usr/local/nvidia/lib64:/usr/local/cuda/lib64
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-            # RW so the post-PASS cache-population step can write to
-            # /data/bin/$SHORT_SHA/. Read-side paths (fxcache, MBP-10,
-            # trades) are unaffected.
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        args:
-          - |
-            set -euo pipefail
-
-            REF="{{workflow.parameters.commit-ref}}"
-            TEST_NAME="{{workflow.parameters.test-name}}"
-
-            echo "==================================="
-            echo "  Plain smoke L40S run"
-            echo "==================================="
-            echo "  Ref:      $REF"
-            echo "  Test:     $TEST_NAME"
-            echo "  Pool:     {{workflow.parameters.gpu-pool}}"
-            echo "==================================="
-
-            # --- SSH setup ---
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            # --- Persistent checkout on PVC ---
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                TARGET=$(git rev-parse "origin/$REF")
-              else
-                TARGET=$(git rev-parse --verify "$REF" 2>/dev/null || echo "$REF")
-              fi
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$TARGET" ]; then
-                echo "=== Updating checkout to $REF ($TARGET) ==="
-                git checkout --force --detach "$TARGET"
-                git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git fetch origin
-              if git rev-parse --verify "origin/$REF" >/dev/null 2>&1; then
-                git checkout --force --detach "origin/$REF"
-              else
-                git checkout "$REF"
-              fi
-            fi
-            SHA=$(git rev-parse --short HEAD)
-            echo "Checked out $SHA"
-
-            export PATH="${CARGO_HOME}/bin:/usr/local/cuda/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            bash scripts/ptx-cache-invalidate.sh "${CARGO_TARGET_DIR}/.ptx_cache"
-
-            # --- Optional clean: ml + ml-dqn before compile ---
-            CLEAN_CACHE="{{workflow.parameters.clean-cache}}"
-            if [ "$CLEAN_CACHE" = "true" ]; then
-              echo "=== clean-cache=true — wiping ml + ml-dqn build artefacts ==="
-              cargo clean -p ml --release 2>&1 | tee /cargo-target/smoke-clean.log
-              cargo clean -p ml-dqn --release 2>&1 | tee -a /cargo-target/smoke-clean.log
-              echo "=== Cleaned. Forced fresh compile of ml/ml-dqn (deps stay cached) ==="
-            fi
-
-            # --- Compile test binary + the 3 train binaries ---
-            #
-            # The 3 examples mirror exactly what `ensure-binary` produces
-            # (see train-multi-seed-template.yaml line ~268). Compiling
-            # them here lets the post-PASS step populate the
-            # `/data/bin/$SHORT_SHA/` cache so the next train run at the
-            # same SHA skips its own ensure-binary compile.
-            echo "=== Compiling (--release --no-run + train binaries) ==="
-            cargo test -p ml --release --lib --no-run 2>&1 | tee /cargo-target/smoke-compile.log
-            cargo build -p ml --release \
-              --example train_baseline_rl \
-              --example evaluate_baseline \
-              --example precompute_features \
-              2>&1 | tee -a /cargo-target/smoke-compile.log
-
-            # Honour CARGO_TARGET_DIR — outputs go to ${CARGO_TARGET_DIR}/release/deps,
-            # NOT the repo-relative target/. Subshell + || true wraps the head -1
-            # SIGPIPE so pipefail doesn't fire.
-            TARGET_DIR="${CARGO_TARGET_DIR:-target}"
-            TEST_BIN=$( (ls -t "$TARGET_DIR/release/deps/ml-"* 2>/dev/null \
-              | grep -vE '\.(d|rcgu|rmeta|rlib|so|json)$' \
-              | head -1) || true )
-            if [ -z "$TEST_BIN" ] || [ ! -x "$TEST_BIN" ]; then
-              echo "ERROR: could not locate compiled test binary in $TARGET_DIR/release/deps/"
-              ls -la "$TARGET_DIR/release/deps/" 2>&1 | head -20
-              exit 3
-            fi
-            echo "=== Test binary: $TEST_BIN ==="
-
-            # grep -q + pipefail = SIGPIPE on early exit. Capture once, grep
-            # without -q so the consumer reads all input.
-            TEST_LIST=$("$TEST_BIN" --list 2>&1 || true)
-            if ! echo "$TEST_LIST" | grep -F "$TEST_NAME" > /dev/null; then
-              echo "ERROR: test '$TEST_NAME' not found"
-              echo "$TEST_LIST" | grep smoke_tests | (head -30 || true)
-              exit 4
-            fi
-
-            # --- Run the test ---
-            echo ""
-            echo "==================================="
-            echo "  Launching test..."
-            echo "==================================="
-            START_TS=$(date +%s)
-            set +e
-            "$TEST_BIN" "$TEST_NAME" --ignored --nocapture
-            EXIT_CODE=$?
-            set -e
-            END_TS=$(date +%s)
-            ELAPSED=$((END_TS - START_TS))
-
-            echo ""
-            echo "==================================="
-            echo "  Smoke run complete"
-            echo "==================================="
-            echo "  Exit code:  $EXIT_CODE"
-            echo "  Elapsed:    ${ELAPSED}s"
-            echo "  Commit:     $SHA"
-            echo "  Test:       $TEST_NAME"
-            echo "==================================="
-
-            if [ "$EXIT_CODE" -ne 0 ]; then
-              echo ""
-              echo "FAIL: test exited with code $EXIT_CODE"
-              exit "$EXIT_CODE"
-            fi
-            echo ""
-            echo "PASS: smoke completed."
-
-            # --- Populate /data/bin/$SHORT_SHA/ for ensure-binary cache ---
-            # On PASS only — broken binaries must not be cached. Mirrors
-            # train-multi-seed-template.yaml:270-274 exactly: same
-            # destination layout, same strip step. Idempotent: writes are
-            # to a SHA-keyed dir so concurrent smokes at different SHAs
-            # don't collide; same-SHA writes are deterministic compile
-            # output and overwrite-safe.
-            FULL_SHA=$(git rev-parse HEAD)
-            SHORT_SHA=$(echo "$FULL_SHA" | cut -c1-9)
-            BIN_DIR="/data/bin/$SHORT_SHA"
-            BINARIES="train_baseline_rl evaluate_baseline precompute_features"
-            echo ""
-            echo "=== Populating ensure-binary cache at $BIN_DIR ==="
-            mkdir -p "$BIN_DIR"
-            for bin in $BINARIES; do
-              src="${CARGO_TARGET_DIR}/release/examples/$bin"
-              if [ ! -x "$src" ]; then
-                echo "WARN: $src not found (cache pop skipped for $bin)"
-                continue
-              fi
-              cp "$src" "$BIN_DIR/"
-              strip "$BIN_DIR/$bin" 2>/dev/null || true
-            done
-            ls -lh "$BIN_DIR/"
-            echo "=== Cache populated; next train@$SHORT_SHA hits ensure-binary cache ==="

infra/k8s/argo/train-multi-seed-template.yaml

@@ -1,605 +0,0 @@
-# Multi-seed training workflow — Plan 5 Task 5 Phase B (one-job-per-seed).
-#
-# Renders an Argo DAG that fans out N seeds into N parallel `train-single` task
-# instances. Each task receives `seed` via inputs.parameters and runs a
-# walk-forward training that internally sweeps all K folds via the binary's
-# `--max-folds K` arg. Per-job runtime is K× longer than the original (seed,
-# fold) matrix but fanout drops from N*K to N — a better fit for the L40S pool
-# (5-GPU capacity vs 30 jobs queueing) and a simpler binary contract
-# (`train_baseline_rl` is a multi-fold walk-forward executor; it does NOT
-# accept `--fold K`).
-#
-# The `# __MATRIX_TASKS__` marker on the dag.tasks line is replaced by
-# scripts/argo-train.sh with N generated WorkflowTask stanzas before
-# submission. This avoids hand-writing the matrix and keeps the template
-# human-readable.
-#
-# Usage:
-#   ./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --tag plan5-final
-#
-# DAG (per task):
-#   ensure-binary ──┐
-#   gpu-warmup ─────┼──> ensure-fxcache ──> [N parallel train-single tasks,
-#                                              one per seed, each runs all K folds]
-#                                              │
-#                                              └──> aggregate (manual via
-#                                                   scripts/gather-multi-seed-metrics.sh)
----
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: train-multi-seed
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: train-multi-seed
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train
-spec:
-  entrypoint: multi-seed-matrix
-  onExit: notify-result
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  # Multi-seed runs: N jobs in parallel (one per seed), each running all K folds
-  # in walk-forward sequence. Allow 12h walltime — 6-fold runs are ~6× longer
-  # than the original per-(seed,fold) jobs but easily fit in 12h on L40S.
-  activeDeadlineSeconds: 43200
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: git-branch
-        value: main
-      # Default sm_89 / ci-training-l40s as of 2026-05-14 per
-      # `feedback_default_to_l40s_pool.md`. argo-train.sh's
-      # CUDA_COMPUTE_CAP derivation passes "89" by default to match;
-      # explicit --gpu-pool ci-training-h100 overrides both back to
-      # sm_90 + H100.
-      - name: cuda-compute-cap
-        value: "89"
-      - name: model
-        value: dqn
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: hyperopt-trials
-        value: "0"  # Multi-seed runs typically skip hyperopt (re-use baseline params)
-      - name: hyperopt-epochs
-        value: "8"
-      - name: train-epochs
-        value: "50"
-      - name: symbol
-        value: ES.FUT
-      - name: initial-capital
-        value: "35000"
-      - name: tx-cost-bps
-        value: "0.1"
-      - name: tick-size
-        value: "0.25"
-      - name: spread-ticks
-        value: "1.0"
-      - name: sanitizer
-        value: "none"
-      - name: multi-seed
-        value: "5"
-      - name: folds
-        value: "6"
-      # Plan 5 Task 3 (A.4.1): nsys profile harness toggle. When "true", each
-      # train-single job wraps the training binary under `nsys profile` and
-      # uploads the resulting .nsys-rep to MinIO bucket
-      # foxhunt-training-artifacts/profiles/<short-sha>/. Default off.
-      - name: profile
-        value: "false"
-      # Bar formation params — passed to BOTH precompute_features (when
-      # building fxcache) AND train_baseline_rl (when looking up fxcache).
-      # MUST match between the two for fxcache HIT (cache key includes them
-      # as of 2026-05-09 architectural fix). Defaults match dqn-production.toml.
-      #
-      # imbalance-bar-threshold default: 20.0 (set 2026-05-10). At threshold=0.5
-      # the sampler produced 209M bars from 209M trade ticks (1:1 ratio,
-      # essentially per-tick) on workflow f5wnd, near-OOM in feature extraction
-      # (54Gi/56Gi limit). To match the volume-bar density baseline (~5.74M
-      # bars at 100 contracts), threshold ~20 is the right scale. Override via
-      # argo-train.sh --imbalance-bar-threshold for resolution-sweep experiments.
-      - name: imbalance-bar-threshold
-        value: "20.0"
-      - name: imbalance-bar-ewma-alpha
-        value: "0.1"
-      # Volume bar size (contracts/bar). Used when data-source != "mbp10".
-      # Default 100 matches DEFAULT_VOLUME_BAR_SIZE. Cache key includes this.
-      - name: volume-bar-size
-        value: "100"
-      # Data source mode: "mbp10" (imbalance bars from MBP-10) or "ohlcv"
-      # (volume bars from trades). Threaded into both precompute_features and
-      # the trainer so cache keys align.
-      - name: data-source
-        value: "mbp10"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  volumeClaimTemplates:
-    - metadata:
-        name: workspace
-      spec:
-        accessModes: ["ReadWriteOnce"]
-        storageClassName: scw-bssd
-        resources:
-          requests:
-            storage: 5Gi
-
-  templates:
-    # ── DAG: fan out to N train-single tasks (one per seed) ──
-    # The `# __MATRIX_TASKS__` marker is replaced by argo-train.sh with the
-    # generated per-seed WorkflowTask stanzas. Each task sweeps all K folds
-    # via the binary's `--max-folds {{workflow.parameters.folds}}` argument.
-    # The marker MUST stay on its own line for the awk substitution to work.
-    - name: multi-seed-matrix
-      dag:
-        tasks:
-          - name: ensure-binary
-            template: ensure-binary
-          - name: gpu-warmup
-            template: gpu-warmup
-          - name: ensure-fxcache
-            template: ensure-fxcache
-            dependencies: [ensure-binary]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          # __MATRIX_TASKS__
-
-    # ── ensure-binary: cache-or-compile training binaries by commit SHA ──
-    # (Identical to train-template.yaml. Kept inline rather than templated
-    # via wftmpl-cross-ref to avoid Argo's reluctance to chase template
-    # references across WorkflowTemplates at submission time.)
-    - name: ensure-binary
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          - name: CARGO_INCREMENTAL
-            value: "1"
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: "14"
-            memory: 32Gi
-          limits:
-            cpu: "30"
-            memory: 64Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            BRANCH="{{workflow.parameters.git-branch}}"
-            MODEL="{{workflow.parameters.model}}"
-
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-            git config --global --add safe.directory "$BUILD"
-
-            if [ "$SHA" = "HEAD" ]; then
-              if [ -d "$BUILD/.git" ]; then
-                cd "$BUILD"; git fetch origin
-                SHA=$(git rev-parse "origin/$BRANCH"); cd /
-              else
-                git clone --filter=blob:none "$REPO" "$BUILD"
-                cd "$BUILD"; git checkout "origin/$BRANCH"
-                SHA=$(git rev-parse HEAD); cd /
-              fi
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "Resolved SHA: $SHA (short: $SHORT_SHA)"
-
-            case "$MODEL" in
-              alpha-rl) BINARIES="alpha_rl_train" ;;
-              dqn|ppo) BINARIES="train_baseline_rl evaluate_baseline precompute_features" ;;
-              *)       BINARIES="train_baseline_supervised evaluate_supervised precompute_features" ;;
-            esac
-
-            BIN_DIR="/data/bin/$SHORT_SHA"
-            ALL_CACHED=true
-            for bin in $BINARIES; do
-              if [ ! -x "$BIN_DIR/$bin" ]; then ALL_CACHED=false; break; fi
-            done
-
-            if [ "$ALL_CACHED" = "true" ]; then
-              echo "=== Cache HIT: all binaries present in $BIN_DIR ==="
-              ls -lh "$BIN_DIR/"
-              echo "$SHORT_SHA" > /tmp/sha
-              exit 0
-            fi
-
-            echo "=== Cache MISS: compiling binaries for $SHORT_SHA ==="
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"; git fetch origin
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$SHA" ]; then
-                git checkout --force "$SHA"; git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"; git checkout "$SHA"
-            fi
-
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            if [ "$MODEL" = "alpha-rl" ]; then
-              cargo build --release -p ml-alpha --example alpha_rl_train
-            else
-              ML_EXAMPLE_ARGS=""
-              for ex in $BINARIES; do
-                ML_EXAMPLE_ARGS="$ML_EXAMPLE_ARGS --example $ex"
-              done
-              cargo build --release -p ml --features ml/cuda $ML_EXAMPLE_ARGS
-            fi
-
-            mkdir -p "$BIN_DIR"
-            for bin in $BINARIES; do
-              cp "$CARGO_TARGET_DIR/release/examples/$bin" "$BIN_DIR/"
-            done
-            strip "$BIN_DIR/"*
-            echo "$SHORT_SHA" > /tmp/sha
-
-    # ── gpu-warmup: trigger GPU node autoscale during compile ──
-    - name: gpu-warmup
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: busybox:1.37
-        command: ["/bin/sh", "-c"]
-        args:
-          - |
-            echo "GPU warmup: triggering node autoscale..."
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: 200m
-            memory: 128Mi
-
-    # ── ensure-fxcache: precompute feature cache (shared across all seeds/folds) ──
-    - name: ensure-fxcache
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: ubuntu:24.04
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-        resources:
-          requests:
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            cpu: "28"
-            memory: 96Gi
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-            readOnly: false
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BINARY="/data/bin/$SHA/precompute_features"
-            if [ ! -x "$BINARY" ]; then
-              echo "ERROR: precompute_features not found at $BINARY"; exit 1
-            fi
-            export RAYON_NUM_THREADS=20
-            if $BINARY \
-              --data-dir /data/futures-baseline \
-              --mbp10-data-dir /data/futures-baseline-mbp10 \
-              --trades-data-dir /data/futures-baseline-trades \
-              --output-dir /feature-cache \
-              --symbol {{workflow.parameters.symbol}} \
-              --data-source {{workflow.parameters.data-source}} \
-              --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-              --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-              --volume-bar-size {{workflow.parameters.volume-bar-size}} \
-              --yes; then
-              echo "=== Feature cache ready ==="
-            else
-              echo "=== Cache stale or missing — regenerating ==="
-              # NOTE: don't blanket-rm /feature-cache/*.fxcache anymore. With
-              # bar-params now in the cache key, multiple valid caches can
-              # coexist (different threshold experiments). The early-exit
-              # check above already validates the specific key — fall through
-              # to regen only the one we need.
-              $BINARY \
-                --data-dir /data/futures-baseline \
-                --mbp10-data-dir /data/futures-baseline-mbp10 \
-                --trades-data-dir /data/futures-baseline-trades \
-                --output-dir /feature-cache \
-                --symbol {{workflow.parameters.symbol}} \
-                --data-source {{workflow.parameters.data-source}} \
-                --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-                --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-                --volume-bar-size {{workflow.parameters.volume-bar-size}} \
-                --yes
-            fi
-
-    # ── train-single: one seed, all folds (training instance) ──
-    # Invoked once per matrix entry. Reads SEED from inputs.parameters and
-    # forwards it via `--seed`; the binary's `--max-folds K` arg drives the
-    # walk-forward sweep over all K folds inside this single process.
-    # Plan 5 Task 5 Phase B pivot: was per-(seed, fold) on the failed deploy
-    # because train_baseline_rl does not accept `--fold N` (it is a multi-fold
-    # executor, not a single-fold one). One-job-per-seed matches the binary's
-    # actual contract and the L40S pool's capacity.
-    - name: train-single
-      inputs:
-        parameters:
-          - name: seed
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: IfNotPresent
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_FEATURE_CACHE_DIR
-            value: /feature-cache
-          - name: SEED
-            value: "{{inputs.parameters.seed}}"
-          # Plan 5 Task 3 (A.4.1): MinIO creds for the optional `mc cp` of
-          # the .nsys-rep artefact at the end of train-single. Both refs are
-          # `optional: true` so the env mount succeeds on clusters that do
-          # not pre-create `minio-credentials` (the upload then warn-fails
-          # gracefully — training itself doesn't depend on these).
-          - name: MINIO_ACCESS_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: access-key
-                optional: true
-          - name: MINIO_SECRET_KEY
-            valueFrom:
-              secretKeyRef:
-                name: minio-credentials
-                key: secret-key
-                optional: true
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        volumeMounts:
-          - name: workspace
-            mountPath: /workspace
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-            readOnly: true
-        args:
-          - |
-            set -e
-            # Resolve SHA via the binary cache directory layout the
-            # ensure-binary task established. multi-seed-matrix passes the
-            # SHA implicitly via the workflow-scoped /data/bin tree.
-            SHA=$(ls -1t /data/bin | head -1)
-            export PATH="/data/bin/$SHA:$PATH"
-            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
-            nvidia-smi
-
-            MODEL="{{workflow.parameters.model}}"
-            case "$MODEL" in
-              alpha-rl) BINARY=alpha_rl_train ;;
-              dqn|ppo) BINARY=train_baseline_rl ;;
-              *)       BINARY=train_baseline_supervised ;;
-            esac
-
-            mkdir -p /workspace/output
-
-            # Plan 5 Task 3 (A.4.1): optional nsys profile wrapper.
-            # When workflow.parameters.profile=="true", wrap the training
-            # binary under `nsys profile --capture-range=cudaProfilerApi`.
-            # The .nsys-rep is uploaded to MinIO bucket
-            # foxhunt-training-artifacts/profiles/<short-sha>/ in the
-            # post-training upload block below.
-            PROFILE="{{workflow.parameters.profile}}"
-            POD_NAME="${HOSTNAME:-pod}"
-            NSYS_OUT="/workspace/output/profile-${POD_NAME}.nsys-rep"
-            NSYS_PREFIX=""
-            if [ "$PROFILE" = "true" ]; then
-              NSYS_BIN=$(which nsys 2>/dev/null || echo "/usr/local/cuda/bin/nsys")
-              if [ -x "$NSYS_BIN" ]; then
-                NSYS_PREFIX="$NSYS_BIN profile --capture-range=cudaProfilerApi --output=${NSYS_OUT} --force-overwrite=true"
-                echo "=== nsys profile enabled: ${NSYS_OUT} ==="
-              else
-                echo "WARN: --profile requested but nsys not found at $NSYS_BIN — running without"
-              fi
-            fi
-
-            echo "=== Training: $MODEL seed=$SEED folds={{workflow.parameters.folds}} ==="
-            if [ "$MODEL" = "alpha-rl" ]; then
-              stdbuf -oL $NSYS_PREFIX ${BINARY} \
-                --mbp10-data-dir /data/futures-baseline-mbp10/{{workflow.parameters.symbol}} \
-                --predecoded-dir /feature-cache/predecoded \
-                --out /workspace/output \
-                --n-steps 50000 \
-                --n-backtests 16 \
-                --seed "$SEED" \
-                --instrument-mode all
-            else
-              stdbuf -oL $NSYS_PREFIX ${BINARY} \
-                --model "$MODEL" \
-                --symbol {{workflow.parameters.symbol}} \
-                --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
-                --tick-size {{workflow.parameters.tick-size}} \
-                --spread-ticks {{workflow.parameters.spread-ticks}} \
-                --initial-capital {{workflow.parameters.initial-capital}} \
-                --data-dir /data/futures-baseline \
-                --mbp10-data-dir /data/futures-baseline-mbp10 \
-                --trades-data-dir /data/futures-baseline-trades \
-                --output-dir /workspace/output \
-                --epochs {{workflow.parameters.train-epochs}} \
-                --seed "$SEED" \
-                --max-folds {{workflow.parameters.folds}} \
-                --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-                --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-                --volume-bar-size {{workflow.parameters.volume-bar-size}}
-            fi
-
-            echo "=== Training complete: seed=$SEED folds={{workflow.parameters.folds}} ==="
-
-            # Plan 5 Task 3 (A.4.1): upload .nsys-rep to MinIO if profile run.
-            # mc is fetched on-demand (~25 MB single static binary) — the
-            # ci-builder training image does not bundle it. Bucket:
-            # foxhunt-training-artifacts/profiles/<short-sha>/.
-            if [ "$PROFILE" = "true" ] && [ -f "$NSYS_OUT" ]; then
-              echo "=== Uploading nsys profile to MinIO ==="
-              MC_BIN=$(which mc 2>/dev/null || echo "")
-              if [ -z "$MC_BIN" ]; then
-                MC_BIN=/tmp/mc
-                curl -fsSL https://dl.min.io/client/mc/release/linux-amd64/mc -o "$MC_BIN" || {
-                  echo "WARN: failed to download mc — skipping upload"
-                  exit 0
-                }
-                chmod +x "$MC_BIN"
-              fi
-              "$MC_BIN" alias set foxhunt http://minio.foxhunt.svc.cluster.local:9000 \
-                "$MINIO_ACCESS_KEY" "$MINIO_SECRET_KEY" 2>/dev/null || true
-              "$MC_BIN" cp "$NSYS_OUT" \
-                "foxhunt/foxhunt-training-artifacts/profiles/$SHA/profile-seed${SEED}-${POD_NAME}.nsys-rep" || \
-                echo "WARN: nsys upload failed for seed=$SEED"
-              echo "=== nsys profile upload complete ==="
-            fi
-
-    # ── notify-result: post workflow outcome to Mattermost (onExit) ──
-    - name: notify-result
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: WEBHOOK_URL
-            valueFrom:
-              secretKeyRef:
-                name: notification-webhook
-                key: webhook-url
-                optional: true
-        resources:
-          requests:
-            cpu: 50m
-            memory: 32Mi
-          limits:
-            cpu: 100m
-            memory: 64Mi
-        args:
-          - |
-            STATUS="{{workflow.status}}"
-            NAME="{{workflow.name}}"
-            if [ -z "$WEBHOOK_URL" ] || echo "$WEBHOOK_URL" | grep -q "PLACEHOLDER"; then
-              echo "No webhook configured, skipping notification"; exit 0
-            fi
-            EMOJI=":x:"
-            [ "$STATUS" = "Succeeded" ] && EMOJI=":white_check_mark:"
-            PAYLOAD="{\"username\":\"Argo CI\",\"text\":\"${EMOJI} **${NAME}** (multi-seed) — ${STATUS}\"}"
-            curl -sf -X POST -H 'Content-Type: application/json' \
-              -d "$PAYLOAD" "$WEBHOOK_URL" || echo "WARN: webhook post failed"

infra/k8s/argo/train-template.yaml

@@ -1,838 +0,0 @@
-# Unified training workflow — replaces all model-specific training templates.
-#
-# Key innovation: binary caching by commit SHA on PVC.
-# Check /data/bin/$SHA first; compile only on cache miss.
-#
-# DAG:
-#   ensure-binary ──┐
-#   gpu-warmup ─────┼──> ensure-fxcache ──> hyperopt ──> train-best ──> evaluate ──> upload-results
-#
-# When hyperopt-trials=0 (the `scripts/argo-precompute.sh` path),
-# gpu-warmup / hyperopt / train-best / evaluate all skip and the
-# workflow runs ensure-binary → ensure-fxcache only — no GPU node
-# gets provisioned for nothing.
-#
-# Usage:
-#   argo submit -n foxhunt --from=wftmpl/train
-#   argo submit -n foxhunt --from=wftmpl/train -p model=ppo -p train-epochs=100
-#   argo submit -n foxhunt --from=wftmpl/train -p hyperopt-trials=0   # fxcache-only path
-apiVersion: argoproj.io/v1alpha1
-kind: WorkflowTemplate
-metadata:
-  name: train
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: train
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: train
-spec:
-  entrypoint: pipeline
-  onExit: notify-result
-  serviceAccountName: argo-workflow
-  archiveLogs: true
-  podMetadata:
-    labels:
-      app.kubernetes.io/part-of: foxhunt
-      app.kubernetes.io/component: train
-  securityContext:
-    fsGroup: 0
-  ttlStrategy:
-    secondsAfterCompletion: 3600
-  activeDeadlineSeconds: 28800  # 8 hours
-
-  arguments:
-    parameters:
-      - name: commit-sha
-        value: HEAD
-      - name: git-branch
-        value: main
-      - name: cuda-compute-cap
-        value: "90"
-      - name: model
-        value: dqn
-      # gpu-pool default: ci-training-l40s (set 2026-05-14 per
-      # `feedback_default_to_l40s_pool.md` — SP-chain training has been
-      # standardising on L40S since 2026-05-09; the prior ci-training-h100
-      # default required every SP-run invocation to pass an explicit
-      # `--gpu-pool ci-training-l40s` override. H100 remains opt-in via
-      # `--gpu-pool ci-training-h100` for runs that genuinely need
-      # 80 GB VRAM or sm_90 features). Compute-cap derivation in
-      # `argo-train.sh` matches this default to sm_89 (Ada Lovelace).
-      - name: gpu-pool
-        value: ci-training-l40s
-      - name: hyperopt-trials
-        value: "20"
-      - name: hyperopt-epochs
-        value: "8"
-      - name: train-epochs
-        value: "50"
-      - name: symbol
-        value: ES.FUT
-      - name: initial-capital
-        value: "35000"
-      - name: tx-cost-bps
-        value: "0.1"
-      - name: tick-size
-        value: "0.25"
-      - name: spread-ticks
-        value: "1.0"
-      - name: sanitizer
-        value: "none"  # "none", "memcheck", "racecheck", "synccheck"
-      # Bar formation params — passed to BOTH precompute_features (when
-      # building fxcache) AND train_baseline_rl (when looking up fxcache).
-      # MUST match between the two for fxcache HIT (cache key includes them
-      # as of 2026-05-09 architectural fix).
-      #
-      # imbalance-bar-threshold default: 20.0 (set 2026-05-10). At threshold=0.5
-      # the sampler produced 209M bars from 209M trade ticks (1:1 ratio,
-      # essentially per-tick) on workflow f5wnd, near-OOM in feature extraction.
-      # threshold=20 produces ~5-6M bars matching the volume-bar density baseline.
-      - name: imbalance-bar-threshold
-        value: "20.0"
-      - name: imbalance-bar-ewma-alpha
-        value: "0.1"
-      - name: volume-bar-size
-        value: "100"
-      - name: data-source
-        value: "mbp10"
-
-  volumes:
-    - name: git-ssh-key
-      secret:
-        secretName: argo-git-ssh-key
-        defaultMode: 256
-    - name: training-data
-      persistentVolumeClaim:
-        claimName: training-data-pvc
-    - name: cargo-target-cuda
-      persistentVolumeClaim:
-        claimName: cargo-target-cuda
-    - name: feature-cache
-      persistentVolumeClaim:
-        claimName: feature-cache-pvc
-
-  volumeClaimTemplates:
-    - metadata:
-        name: workspace
-      spec:
-        accessModes: ["ReadWriteOnce"]
-        storageClassName: scw-bssd
-        resources:
-          requests:
-            storage: 5Gi
-
-  templates:
-    # ── DAG: orchestrate all steps ──
-    - name: pipeline
-      dag:
-        tasks:
-          - name: ensure-binary
-            template: ensure-binary
-          - name: gpu-warmup
-            template: gpu-warmup
-            # gpu-warmup pre-provisions an L40S node so hyperopt /
-            # train-best don't pay autoscaler latency at start. When
-            # hyperopt-trials==0 (precompute-only path used by
-            # `scripts/argo-precompute.sh`), both downstream consumers
-            # are skipped and the GPU node would sit idle until the
-            # workflow ends. Skip the warmup in that case to avoid
-            # provisioning expensive L40S capacity for nothing.
-            when: "{{workflow.parameters.hyperopt-trials}} != 0"
-          - name: ensure-fxcache
-            template: ensure-fxcache
-            dependencies: [ensure-binary]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          - name: hyperopt
-            template: hyperopt
-            dependencies: [ensure-binary, gpu-warmup, ensure-fxcache]
-            when: "{{workflow.parameters.hyperopt-trials}} != 0"
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          - name: train-best
-            template: train-best
-            dependencies: [ensure-binary, gpu-warmup, ensure-fxcache, hyperopt]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          - name: evaluate
-            template: evaluate
-            dependencies: [train-best]
-            arguments:
-              parameters:
-                - name: sha
-                  value: "{{tasks.ensure-binary.outputs.parameters.sha}}"
-          - name: upload-results
-            template: upload-results
-            dependencies: [evaluate]
-
-    # ── ensure-binary: cache-or-compile training binaries by commit SHA ──
-    - name: ensure-binary
-      outputs:
-        parameters:
-          - name: sha
-            valueFrom:
-              path: /tmp/sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: Always
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CARGO_TERM_COLOR
-            value: always
-          - name: CARGO_TARGET_DIR
-            value: /cargo-target
-          - name: CARGO_HOME
-            value: /cargo-target/cargo-home
-          # sccache: per-crate compile-output cache. Disk-local on the same
-          # cargo-target-cuda PVC so no network cost on hit; survives across
-          # pods and commits (cargo's incremental cache is invalidated by
-          # git checkout mtime touches — sccache is content-hash keyed so it
-          # isn't). Binary assumed present in ci-builder image.
-          #
-          # CARGO_INCREMENTAL=0 is required to make sccache effective:
-          # the project's Cargo.toml / .cargo/config.toml set
-          # `incremental = true`, which emits per-query save-analysis
-          # artifacts that sccache does not cache. Turning incremental off
-          # lets rustc emit pure object output that sccache hashes
-          # uniformly — same source + same args → cache hit.
-          - name: RUSTC_WRAPPER
-            value: sccache
-          - name: SCCACHE_DIR
-            value: /cargo-target/sccache
-          - name: SCCACHE_CACHE_SIZE
-            value: "40G"
-          - name: CARGO_INCREMENTAL
-            value: "0"
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: "14"
-            memory: 32Gi
-          limits:
-            cpu: "30"
-            memory: 64Gi
-        volumeMounts:
-          - name: git-ssh-key
-            mountPath: /etc/git-ssh
-            readOnly: true
-          - name: cargo-target-cuda
-            mountPath: /cargo-target
-          - name: training-data
-            mountPath: /data
-        args:
-          - |
-            set -e
-            SHA="{{workflow.parameters.commit-sha}}"
-            BRANCH="{{workflow.parameters.git-branch}}"
-            MODEL="{{workflow.parameters.model}}"
-
-            # ── SSH setup ──
-            mkdir -p ~/.ssh
-            cp /etc/git-ssh/ssh-privatekey ~/.ssh/id_ed25519
-            chmod 600 ~/.ssh/id_ed25519
-            printf 'Host *\n  StrictHostKeyChecking no\n  UserKnownHostsFile /dev/null\n' > ~/.ssh/config
-            chmod 600 ~/.ssh/config
-
-            REPO="ssh://git@gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222/root/foxhunt.git"
-            BUILD="/cargo-target/src"
-
-            git config --global --add safe.directory "$BUILD"
-
-            # ── Resolve SHA ──
-            if [ "$SHA" = "HEAD" ]; then
-              if [ -d "$BUILD/.git" ]; then
-                cd "$BUILD"
-                git fetch origin
-                SHA=$(git rev-parse "origin/$BRANCH")
-                cd /
-              else
-                git clone --filter=blob:none "$REPO" "$BUILD"
-                cd "$BUILD"
-                git checkout "origin/$BRANCH"
-                SHA=$(git rev-parse HEAD)
-                cd /
-              fi
-            fi
-            SHORT_SHA=$(echo "$SHA" | cut -c1-9)
-            echo "Resolved SHA: $SHA (short: $SHORT_SHA)"
-
-            # ── Derive needed binaries from model ──
-            case "$MODEL" in
-              dqn|ppo)
-                BINARIES="hyperopt_baseline_rl train_baseline_rl evaluate_baseline precompute_features"
-                ;;
-              *)
-                BINARIES="hyperopt_baseline_supervised train_baseline_supervised evaluate_supervised precompute_features"
-                ;;
-            esac
-
-            # ── Cache check ──
-            BIN_DIR="/data/bin/$SHORT_SHA"
-            ALL_CACHED=true
-            for bin in $BINARIES; do
-              if [ ! -x "$BIN_DIR/$bin" ]; then
-                ALL_CACHED=false
-                break
-              fi
-            done
-
-            if [ "$ALL_CACHED" = "true" ]; then
-              echo "=== Cache HIT: all binaries present in $BIN_DIR ==="
-              ls -lh "$BIN_DIR/"
-              echo "$SHORT_SHA" > /tmp/sha
-              exit 0
-            fi
-
-            echo "=== Cache MISS: compiling binaries for $SHORT_SHA ==="
-
-            # ── Clone / checkout ──
-            if [ -d "$BUILD/.git" ]; then
-              cd "$BUILD"
-              git fetch origin
-              CURRENT=$(git rev-parse HEAD 2>/dev/null || echo "none")
-              if [ "$CURRENT" != "$SHA" ]; then
-                echo "Updating checkout: $(echo $CURRENT | cut -c1-8) -> $SHORT_SHA"
-                git checkout --force "$SHA"
-                git clean -fd
-              fi
-            else
-              git clone --filter=blob:none "$REPO" "$BUILD"
-              cd "$BUILD"
-              git checkout "$SHA"
-            fi
-
-            # ── Build ──
-            export PATH="${CARGO_HOME}/bin:${PATH}"
-            export CUDA_COMPUTE_CAP={{workflow.parameters.cuda-compute-cap}}
-
-            ML_EXAMPLE_ARGS=""
-            for ex in $BINARIES; do
-              ML_EXAMPLE_ARGS="$ML_EXAMPLE_ARGS --example $ex"
-            done
-
-            echo "Building: $BINARIES"
-            echo "  CUDA_COMPUTE_CAP=$CUDA_COMPUTE_CAP"
-            cargo build --release -p ml --features ml/cuda $ML_EXAMPLE_ARGS
-
-            # ── Install to cache dir ──
-            mkdir -p "$BIN_DIR"
-            for bin in $BINARIES; do
-              cp "$CARGO_TARGET_DIR/release/examples/$bin" "$BIN_DIR/"
-            done
-            strip "$BIN_DIR/"*
-
-            echo "=== Cached binaries ==="
-            ls -lh "$BIN_DIR/"
-
-            # ── Prune old SHAs: keep last 5 ──
-            cd /data/bin
-            ls -1t | tail -n +6 | while read -r old; do
-              echo "Pruning old cache: $old"
-              rm -rf "$old"
-            done
-
-            echo "$SHORT_SHA" > /tmp/sha
-
-    # ── ensure-fxcache: precompute feature cache if needed ──
-    #
-    # Pinned to the high-memory ci-compile-cpu-hm pool (POP2-HM-32C-256G,
-    # 32 vCPU + 256GB RAM, min_size=0). The 9-quarter precompute_features
-    # peaks ~50-60GB during the post-OFI alpha_trades conversion which
-    # OOM-killed the standard 64GB ci-compile-cpu pool twice on
-    # 2026-05-16 (workflows train-wq8b8 + train-2l6p4 both died at
-    # exitCode 137 right after "OFI computed"). The HM pool autoscales
-    # to zero when idle so this pin only costs anything during an
-    # actual fxcache rebuild.
-    - name: ensure-fxcache
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: ci-compile-cpu-hm
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: ubuntu:24.04
-        command: ["/bin/bash", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-        resources:
-          requests:
-            cpu: "4"
-            memory: 16Gi
-          limits:
-            cpu: "28"
-            memory: 200Gi
-        volumeMounts:
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            BINARY="/data/bin/$SHA/precompute_features"
-
-            if [ ! -x "$BINARY" ]; then
-              echo "ERROR: precompute_features not found at $BINARY"
-              exit 1
-            fi
-
-            # fxcache has built-in version validation (FXCACHE_VERSION in header).
-            # The precompute binary checks existing cache — if version matches,
-            # it skips regeneration. If version mismatches, it fails and we
-            # delete + regenerate. No unconditional rm — cache is reused when valid.
-            echo "=== Running precompute_features (SHA: $SHA) ==="
-            if $BINARY \
-              --data-dir /data/futures-baseline \
-              --mbp10-data-dir /data/futures-baseline-mbp10 \
-              --trades-data-dir /data/futures-baseline-trades \
-              --output-dir /feature-cache \
-              --symbol {{workflow.parameters.symbol}} \
-              --data-source {{workflow.parameters.data-source}} \
-              --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-              --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-              --volume-bar-size {{workflow.parameters.volume-bar-size}} \
-              --yes; then
-              echo "=== Feature cache ready ==="
-            else
-              echo "=== Cache stale or missing — regenerating ==="
-              # NOTE: don't blanket-rm /feature-cache/*.fxcache anymore. With
-              # bar-params now in the cache key, multiple valid caches can
-              # coexist (different threshold experiments).
-              $BINARY \
-                --data-dir /data/futures-baseline \
-                --mbp10-data-dir /data/futures-baseline-mbp10 \
-                --trades-data-dir /data/futures-baseline-trades \
-                --output-dir /feature-cache \
-                --symbol {{workflow.parameters.symbol}} \
-                --data-source {{workflow.parameters.data-source}} \
-                --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-                --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-                --volume-bar-size {{workflow.parameters.volume-bar-size}} \
-                --yes
-              echo "=== Feature cache regenerated ==="
-            fi
-
-    # ── gpu-warmup: trigger GPU node autoscale during compile ──
-    - name: gpu-warmup
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: busybox:1.37
-        command: ["/bin/sh", "-c"]
-        args:
-          - |
-            echo "GPU warmup: triggering node autoscale..."
-            echo "GPU node scheduled, exiting to free resources"
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: 100m
-            memory: 64Mi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: 200m
-            memory: 128Mi
-
-    # ── hyperopt: PSO/TPE hyperparameter optimization on GPU ──
-    - name: hyperopt
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: IfNotPresent
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_FEATURE_CACHE_DIR
-            value: /feature-cache
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        volumeMounts:
-          - name: workspace
-            mountPath: /workspace
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-            readOnly: true
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            export PATH="/data/bin/$SHA:$PATH"
-            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
-            nvidia-smi
-
-            mkdir -p /workspace/output/hyperopt
-
-            MODEL="{{workflow.parameters.model}}"
-            case "$MODEL" in
-              dqn|ppo) BINARY=hyperopt_baseline_rl ;;
-              *)       BINARY=hyperopt_baseline_supervised ;;
-            esac
-
-            echo "=== Running hyperopt: $MODEL ($BINARY) ==="
-            echo "  Trials: {{workflow.parameters.hyperopt-trials}}, Epochs: {{workflow.parameters.hyperopt-epochs}}"
-
-            ${BINARY} \
-              --model "$MODEL" \
-              --phase fast \
-              --trials {{workflow.parameters.hyperopt-trials}} \
-              --epochs {{workflow.parameters.hyperopt-epochs}} \
-              --symbol {{workflow.parameters.symbol}} \
-              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
-              --tick-size {{workflow.parameters.tick-size}} \
-              --spread-ticks {{workflow.parameters.spread-ticks}} \
-              --initial-capital {{workflow.parameters.initial-capital}} \
-              --data-dir /data/futures-baseline \
-              --mbp10-data-dir /data/futures-baseline-mbp10 \
-              --trades-data-dir /data/futures-baseline-trades \
-              --base-dir /workspace/output/hyperopt \
-              --output /workspace/output/${MODEL}_hyperopt_results.json
-
-            echo "=== Hyperopt complete ==="
-            cat /workspace/output/${MODEL}_hyperopt_results.json 2>/dev/null || echo "No results file"
-
-    # ── train-best: full training with best hyperparams ──
-    - name: train-best
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: IfNotPresent
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-          - name: CUBLAS_WORKSPACE_CONFIG
-            value: ":4096:8"
-          - name: FOXHUNT_FEATURE_CACHE_DIR
-            value: /feature-cache
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "8"
-            memory: 64Gi
-        volumeMounts:
-          - name: workspace
-            mountPath: /workspace
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-            readOnly: true
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            export PATH="/data/bin/$SHA:$PATH"
-            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
-            nvidia-smi
-
-            MODEL="{{workflow.parameters.model}}"
-            case "$MODEL" in
-              dqn|ppo) BINARY=train_baseline_rl ;;
-              *)       BINARY=train_baseline_supervised ;;
-            esac
-
-            HYPEROPT_FLAG=""
-            if [ -f "/workspace/output/${MODEL}_hyperopt_results.json" ]; then
-              HYPEROPT_FLAG="--hyperopt-params /workspace/output/${MODEL}_hyperopt_results.json"
-              echo "  Using hyperopt results: ${MODEL}_hyperopt_results.json"
-            else
-              echo "  No hyperopt results — training with default hyperparams"
-            fi
-
-            echo "=== Training: $MODEL ({{workflow.parameters.train-epochs}} epochs) ==="
-
-            # compute-sanitizer / nsys: optional GPU debugging/profiling
-            SANITIZER="{{workflow.parameters.sanitizer}}"
-            SANITIZER_PREFIX=""
-            if [ "$SANITIZER" = "nsys" ]; then
-              NSYS_BIN=$(which nsys 2>/dev/null || echo "/usr/local/cuda/bin/nsys")
-              if [ -x "$NSYS_BIN" ]; then
-                NSYS_OUT="/feature-cache/nsys_$(date +%Y%m%d_%H%M%S)"
-                mkdir -p /feature-cache
-                SANITIZER_PREFIX="$NSYS_BIN profile -o $NSYS_OUT --cuda-graph-trace=node --stats=true --show-output=true -f true --duration=60"
-                echo "  nsys profiling enabled — output: ${NSYS_OUT}.nsys-rep (persistent PVC)"
-                echo "  --cuda-graph-trace=node: per-kernel timing inside parent graph"
-                echo "  --cudabacktrace=kernel: kernel source attribution"
-                echo "  --capture-range=cudaProfilerApi: use cudaProfilerStart/Stop to limit capture"
-                echo "  WARNING: ~2x slower — use with 1-2 epochs only"
-              else
-                echo "  WARNING: nsys not found at $NSYS_BIN — running without"
-              fi
-            elif [ "$SANITIZER" != "none" ] && [ -n "$SANITIZER" ]; then
-              SANITIZER_BIN=$(which compute-sanitizer 2>/dev/null || echo "/usr/local/cuda/bin/compute-sanitizer")
-              if [ -x "$SANITIZER_BIN" ]; then
-                SANITIZER_PREFIX="$SANITIZER_BIN --tool $SANITIZER --print-limit 20 --error-exitcode 1"
-                echo "  compute-sanitizer enabled: --tool $SANITIZER"
-                echo "  WARNING: 10-100x slower — use with 1-2 epochs only"
-              else
-                echo "  WARNING: compute-sanitizer not found at $SANITIZER_BIN — running without"
-              fi
-            fi
-
-            stdbuf -oL $SANITIZER_PREFIX ${BINARY} \
-              --model "$MODEL" \
-              --symbol {{workflow.parameters.symbol}} \
-              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
-              --tick-size {{workflow.parameters.tick-size}} \
-              --spread-ticks {{workflow.parameters.spread-ticks}} \
-              --initial-capital {{workflow.parameters.initial-capital}} \
-              --data-dir /data/futures-baseline \
-              --mbp10-data-dir /data/futures-baseline-mbp10 \
-              --trades-data-dir /data/futures-baseline-trades \
-              --output-dir /workspace/output \
-              --epochs {{workflow.parameters.train-epochs}} \
-              --imbalance-bar-threshold {{workflow.parameters.imbalance-bar-threshold}} \
-              --imbalance-bar-ewma-alpha {{workflow.parameters.imbalance-bar-ewma-alpha}} \
-              --volume-bar-size {{workflow.parameters.volume-bar-size}} \
-              $HYPEROPT_FLAG
-
-            echo "=== Training complete ==="
-
-    # ── evaluate: run evaluation on trained model ──
-    - name: evaluate
-      inputs:
-        parameters:
-          - name: sha
-      nodeSelector:
-        k8s.scaleway.com/pool-name: "{{workflow.parameters.gpu-pool}}"
-        topology.kubernetes.io/zone: fr-par-2
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-        - key: node.cilium.io/agent-not-ready
-          operator: Exists
-          effect: NoSchedule
-      container:
-        image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/ci-builder:latest
-        imagePullPolicy: IfNotPresent
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: RUST_LOG
-            value: info
-          - name: SQLX_OFFLINE
-            value: "true"
-        resources:
-          requests:
-            nvidia.com/gpu: "1"
-            cpu: "2"
-            memory: 16Gi
-          limits:
-            nvidia.com/gpu: "1"
-            cpu: "4"
-            memory: 32Gi
-        volumeMounts:
-          - name: workspace
-            mountPath: /workspace
-          - name: training-data
-            mountPath: /data
-            readOnly: true
-          - name: feature-cache
-            mountPath: /feature-cache
-            readOnly: true
-        args:
-          - |
-            set -e
-            SHA="{{inputs.parameters.sha}}"
-            export PATH="/data/bin/$SHA:$PATH"
-            export LD_LIBRARY_PATH=$(echo "$LD_LIBRARY_PATH" | tr ':' '\n' | grep -v stubs | tr '\n' ':' | sed 's/:$//')
-
-            MODEL="{{workflow.parameters.model}}"
-            case "$MODEL" in
-              dqn|ppo) BINARY=evaluate_baseline ;;
-              *)       BINARY=evaluate_supervised ;;
-            esac
-
-            echo "=== Evaluating: $MODEL ==="
-            # Args must match evaluate_baseline's Args struct — see crates/ml/examples/evaluate_baseline.rs
-            # `--models-dir` (not --checkpoint-dir), `--output` is a FILE path (not --output-dir).
-            mkdir -p /workspace/output/eval
-            ${BINARY} \
-              --model "$MODEL" \
-              --symbol {{workflow.parameters.symbol}} \
-              --data-dir /data/futures-baseline \
-              --models-dir /workspace/output \
-              --output /workspace/output/eval/evaluation_report.json \
-              --initial-capital {{workflow.parameters.initial-capital}} \
-              --tx-cost-bps {{workflow.parameters.tx-cost-bps}} \
-              --tick-size {{workflow.parameters.tick-size}} \
-              --spread-ticks {{workflow.parameters.spread-ticks}} || {
-              echo "WARN: Evaluation failed, continuing"
-            }
-
-            echo "=== Evaluation complete ==="
-
-    # ── upload-results: push artifacts to GitLab packages ──
-    - name: upload-results
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: GITLAB_PAT
-            valueFrom:
-              secretKeyRef:
-                name: gitlab-pat
-                key: token
-        resources:
-          requests:
-            cpu: 100m
-            memory: 128Mi
-          limits:
-            cpu: 500m
-            memory: 256Mi
-        volumeMounts:
-          - name: workspace
-            mountPath: /workspace
-        args:
-          - |
-            set -e
-            MODEL="{{workflow.parameters.model}}"
-            SYMBOL="{{workflow.parameters.symbol}}"
-            TIMESTAMP=$(date +%Y%m%d-%H%M%S)
-            GITLAB_API="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181/api/v4"
-            PKG_NAME="foxhunt-training-results"
-            PKG_VERSION="${MODEL}-${SYMBOL}-${TIMESTAMP}"
-
-            echo "Uploading training artifacts: ${PKG_NAME}/${PKG_VERSION}"
-
-            UPLOADED=0
-            find /workspace/output -type f | while read -r file; do
-              REL_PATH="${file#/workspace/output/}"
-              SAFE_NAME=$(echo "$REL_PATH" | tr '/' '--')
-              echo "  Uploading ${REL_PATH} as ${SAFE_NAME}..."
-              curl -f --upload-file "$file" \
-                -H "PRIVATE-TOKEN: ${GITLAB_PAT}" \
-                "${GITLAB_API}/projects/1/packages/generic/${PKG_NAME}/${PKG_VERSION}/${SAFE_NAME}" && \
-                UPLOADED=$((UPLOADED + 1)) || \
-                echo "  WARN: Failed to upload ${REL_PATH}"
-            done
-
-            echo "=== Upload complete (${UPLOADED} files) ==="
-            echo "Package: ${PKG_NAME}/${PKG_VERSION}"
-
-    # ── notify-result: post workflow outcome to Mattermost (onExit) ──
-    - name: notify-result
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-        topology.kubernetes.io/zone: fr-par-2
-      container:
-        image: curlimages/curl:8.12.1
-        command: ["/bin/sh", "-c"]
-        env:
-          - name: WEBHOOK_URL
-            valueFrom:
-              secretKeyRef:
-                name: notification-webhook
-                key: webhook-url
-                optional: true
-        resources:
-          requests:
-            cpu: 50m
-            memory: 32Mi
-          limits:
-            cpu: 100m
-            memory: 64Mi
-        args:
-          - |
-            STATUS="{{workflow.status}}"
-            NAME="{{workflow.name}}"
-
-            if [ -z "$WEBHOOK_URL" ] || echo "$WEBHOOK_URL" | grep -q "PLACEHOLDER"; then
-              echo "No webhook configured, skipping notification"
-              exit 0
-            fi
-
-            if [ "$STATUS" = "Succeeded" ]; then
-              EMOJI=":white_check_mark:"
-            else
-              EMOJI=":x:"
-            fi
-
-            PAYLOAD="{\"username\":\"Argo CI\",\"text\":\"${EMOJI} **${NAME}** — ${STATUS} ({{workflow.duration}}s)\"}"
-
-            curl -sf -X POST -H 'Content-Type: application/json' \
-              -d "$PAYLOAD" "$WEBHOOK_URL" || echo "WARN: webhook post failed"

infra/k8s/argo/values.yaml

@@ -1,80 +0,0 @@
-# Argo Workflows Helm values — foxhunt cluster
-# Chart: argo/argo-workflows v1.0.2 (Argo v4.0.2)
-# Install: helm upgrade argo-workflows argo/argo-workflows -n foxhunt -f infra/k8s/argo/values.yaml
-
-controller:
-  nodeSelector:
-    k8s.scaleway.com/pool-name: platform
-  resources:
-    requests:
-      cpu: 50m
-      memory: 128Mi
-    limits:
-      cpu: 200m
-      memory: 256Mi
-  # Only watch foxhunt namespace (not cluster-wide)
-  workflowNamespaces:
-    - foxhunt
-  # Workflow archive — persists workflow state + logs to PostgreSQL after pod GC
-  persistence:
-    archive: true
-    postgresql:
-      host: postgres.foxhunt.svc.cluster.local
-      port: 5432
-      database: argo_archive
-      tableName: argo_workflows
-      userNameSecret:
-        name: argo-postgres-config
-        key: username
-      passwordSecret:
-        name: argo-postgres-config
-        key: password
-      ssl: false
-      sslMode: disable
-
-server:
-  nodeSelector:
-    k8s.scaleway.com/pool-name: platform
-  resources:
-    requests:
-      cpu: 25m
-      memory: 64Mi
-    limits:
-      cpu: 100m
-      memory: 128Mi
-  ingress:
-    enabled: false
-  extraArgs:
-    - --auth-mode=server
-    - --secure=false
-
-# Default artifact repository — MinIO (in-cluster)
-artifactRepository:
-  archiveLogs: true
-  s3:
-    endpoint: minio.foxhunt.svc.cluster.local:9000
-    insecure: true
-    bucket: argo-logs
-    accessKeySecret:
-      name: minio-credentials
-      key: access-key
-    secretKeySecret:
-      name: minio-credentials
-      key: secret-key
-
-executor:
-  resources:
-    requests:
-      cpu: 10m
-      memory: 32Mi
-    limits:
-      cpu: 100m
-      memory: 64Mi
-
-workflow:
-  serviceAccount:
-    create: true
-    name: argo-workflow
-
-crds:
-  install: true

infra/k8s/databases/postgres-backup.yaml

@@ -1,249 +0,0 @@
-# Postgres backup — daily pg_dump to MinIO S3
-# Uses initContainer (postgres:16-alpine) for pg_dump, then main container
-# (python:3.12-alpine, already cached) uploads to MinIO via S3 API.
-# Retains 7 daily + 4 weekly backups.
-# Apply: kubectl apply -f postgres-backup.yaml
----
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: postgres-backup-script
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: postgres-backup
-    app.kubernetes.io/part-of: foxhunt
-data:
-  upload.py: |
-    #!/usr/bin/env python3
-    """Upload pg_dump backup to MinIO, prune old backups."""
-    import hashlib, hmac, os, sys, glob, urllib.request, urllib.error
-    from datetime import datetime, timezone
-
-    ENDPOINT = os.environ["MINIO_ENDPOINT"]  # http://minio.foxhunt...:9000
-    ACCESS_KEY = os.environ["MINIO_ACCESS_KEY"]
-    SECRET_KEY = os.environ["MINIO_SECRET_KEY"]
-    BUCKET = "foxhunt-backups"
-    BACKUP_DIR = "/backup"
-
-    def s3_sign(method, path, headers, payload_hash):
-        """AWS Signature V4 for MinIO."""
-        now = datetime.now(timezone.utc)
-        date_stamp = now.strftime("%Y%m%d")
-        amz_date = now.strftime("%Y%m%dT%H%M%SZ")
-        region = "us-east-1"
-        service = "s3"
-        scope = f"{date_stamp}/{region}/{service}/aws4_request"
-
-        headers["x-amz-date"] = amz_date
-        headers["x-amz-content-sha256"] = payload_hash
-
-        signed_headers = ";".join(sorted(headers.keys()))
-        canonical_headers = "".join(f"{k}:{headers[k]}\n" for k in sorted(headers.keys()))
-        canonical_request = f"{method}\n{path}\n\n{canonical_headers}\n{signed_headers}\n{payload_hash}"
-
-        string_to_sign = f"AWS4-HMAC-SHA256\n{amz_date}\n{scope}\n{hashlib.sha256(canonical_request.encode()).hexdigest()}"
-
-        def sign(key, msg):
-            return hmac.new(key, msg.encode(), hashlib.sha256).digest()
-
-        signing_key = sign(sign(sign(sign(
-            f"AWS4{SECRET_KEY}".encode(), date_stamp), region), service), "aws4_request")
-        signature = hmac.new(signing_key, string_to_sign.encode(), hashlib.sha256).hexdigest()
-
-        headers["Authorization"] = (
-            f"AWS4-HMAC-SHA256 Credential={ACCESS_KEY}/{scope}, "
-            f"SignedHeaders={signed_headers}, Signature={signature}"
-        )
-        return headers
-
-    def s3_put(key, data):
-        path = f"/{BUCKET}/{key}"
-        url = f"{ENDPOINT}{path}"
-        payload_hash = hashlib.sha256(data).hexdigest()
-        host = ENDPOINT.split("//")[1]
-        headers = {"host": host, "content-length": str(len(data))}
-        headers = s3_sign("PUT", path, headers, payload_hash)
-        req = urllib.request.Request(url, data=data, method="PUT", headers=headers)
-        urllib.request.urlopen(req, timeout=120)
-
-    def s3_list(prefix):
-        path = f"/{BUCKET}?prefix={prefix}&list-type=2"
-        url = f"{ENDPOINT}{path}"
-        host = ENDPOINT.split("//")[1]
-        headers = {"host": host}
-        headers = s3_sign("GET", f"/{BUCKET}", headers, hashlib.sha256(b"").hexdigest())
-        req = urllib.request.Request(f"{url}", method="GET", headers=headers)
-        try:
-            resp = urllib.request.urlopen(req, timeout=30).read().decode()
-            import re
-            return sorted(re.findall(r"<Key>([^<]+)</Key>", resp))
-        except Exception:
-            return []
-
-    def s3_delete(key):
-        path = f"/{BUCKET}/{key}"
-        url = f"{ENDPOINT}{path}"
-        host = ENDPOINT.split("//")[1]
-        headers = {"host": host}
-        headers = s3_sign("DELETE", path, headers, hashlib.sha256(b"").hexdigest())
-        req = urllib.request.Request(url, method="DELETE", headers=headers)
-        urllib.request.urlopen(req, timeout=30)
-
-    def s3_ensure_bucket():
-        path = f"/{BUCKET}"
-        url = f"{ENDPOINT}{path}"
-        host = ENDPOINT.split("//")[1]
-        headers = {"host": host}
-        headers = s3_sign("PUT", path, headers, hashlib.sha256(b"").hexdigest())
-        req = urllib.request.Request(url, method="PUT", headers=headers)
-        try:
-            urllib.request.urlopen(req, timeout=30)
-        except urllib.error.HTTPError as e:
-            if e.code not in (409, 200):  # 409 = bucket exists
-                raise
-
-    # Find the backup file
-    files = glob.glob(f"{BACKUP_DIR}/*.sql.gz")
-    if not files:
-        print("ERROR: No backup file found in /backup/")
-        sys.exit(1)
-
-    backup_file = files[0]
-    filename = os.path.basename(backup_file)
-    size_mb = os.path.getsize(backup_file) / (1024 * 1024)
-    print(f"Uploading {filename} ({size_mb:.1f} MB)")
-
-    s3_ensure_bucket()
-
-    with open(backup_file, "rb") as f:
-        data = f.read()
-
-    # Upload daily
-    s3_put(f"daily/{filename}", data)
-    print(f"  Uploaded daily/{filename}")
-
-    # On Sundays, also upload weekly
-    if datetime.now(timezone.utc).isoweekday() == 7:
-        s3_put(f"weekly/{filename}", data)
-        print(f"  Uploaded weekly/{filename}")
-
-    # Prune old dailies (keep 7)
-    dailies = s3_list("daily/")
-    if len(dailies) > 7:
-        for old in dailies[:len(dailies) - 7]:
-            s3_delete(old)
-            print(f"  Pruned {old}")
-
-    # Prune old weeklies (keep 4)
-    weeklies = s3_list("weekly/")
-    if len(weeklies) > 4:
-        for old in weeklies[:len(weeklies) - 4]:
-            s3_delete(old)
-            print(f"  Pruned {old}")
-
-    print("=== Backup upload complete ===")
----
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: postgres-backup
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: postgres-backup
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  schedule: "30 2 * * *"
-  successfulJobsHistoryLimit: 1
-  failedJobsHistoryLimit: 3
-  jobTemplate:
-    spec:
-      backoffLimit: 2
-      activeDeadlineSeconds: 600
-      template:
-        metadata:
-          labels:
-            app: postgres-backup
-            app.kubernetes.io/part-of: foxhunt
-        spec:
-          nodeSelector:
-            k8s.scaleway.com/pool-name: platform
-          restartPolicy: OnFailure
-          # Step 1: pg_dump in initContainer → shared /backup volume
-          initContainers:
-            - name: pg-dump
-              image: postgres:16-alpine
-              command:
-                - sh
-                - -c
-                - |
-                  TIMESTAMP=$(date -u +%Y%m%d-%H%M%S)
-                  echo "=== Postgres Backup ${TIMESTAMP} ==="
-                  PGPASSWORD="${POSTGRES_PASSWORD}" pg_dumpall \
-                    -h "${POSTGRES_HOST}" \
-                    -p 5432 \
-                    -U "${POSTGRES_USER}" \
-                    --clean --if-exists \
-                    | gzip > "/backup/foxhunt-${TIMESTAMP}.sql.gz"
-                  ls -lh /backup/
-                  echo "Dump complete"
-              env:
-                - name: POSTGRES_HOST
-                  value: "postgres.foxhunt.svc.cluster.local"
-                - name: POSTGRES_USER
-                  value: "foxhunt"
-                - name: POSTGRES_PASSWORD
-                  valueFrom:
-                    secretKeyRef:
-                      name: db-credentials
-                      key: password
-              volumeMounts:
-                - name: backup
-                  mountPath: /backup
-              resources:
-                requests:
-                  cpu: 50m
-                  memory: 128Mi
-                limits:
-                  cpu: 500m
-                  memory: 512Mi
-          # Step 2: Upload to MinIO in main container (python cached on nodes)
-          containers:
-            - name: upload
-              image: python:3.12-alpine
-              command: ["python3", "-u", "/scripts/upload.py"]
-              env:
-                - name: MINIO_ENDPOINT
-                  value: "http://minio.foxhunt.svc.cluster.local:9000"
-                - name: MINIO_ACCESS_KEY
-                  valueFrom:
-                    secretKeyRef:
-                      name: minio-credentials
-                      key: access-key
-                - name: MINIO_SECRET_KEY
-                  valueFrom:
-                    secretKeyRef:
-                      name: minio-credentials
-                      key: secret-key
-              volumeMounts:
-                - name: backup
-                  mountPath: /backup
-                - name: script
-                  mountPath: /scripts
-              resources:
-                requests:
-                  cpu: 10m
-                  memory: 64Mi
-                limits:
-                  cpu: 100m
-                  memory: 128Mi
-          volumes:
-            - name: backup
-              emptyDir:
-                sizeLimit: 2Gi
-            - name: script
-              configMap:
-                name: postgres-backup-script
-                items:
-                  - key: upload.py
-                    path: upload.py
-                defaultMode: 0755

infra/k8s/databases/postgres.yaml

@@ -1,96 +0,0 @@
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: postgres-pvc
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: postgres
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes:
-    - ReadWriteOnce
-  resources:
-    requests:
-      storage: 20Gi
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: postgres
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: postgres
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: postgres
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: postgres
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: infra
-      containers:
-        - name: postgres
-          image: timescale/timescaledb:latest-pg16
-          ports:
-            - containerPort: 5432
-              name: postgres
-          args:
-            - -c
-            - max_connections=100
-          env:
-            - name: POSTGRES_DB
-              value: foxhunt
-            - name: POSTGRES_USER
-              value: foxhunt
-            - name: POSTGRES_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-          volumeMounts:
-            - name: postgres-data
-              mountPath: /var/lib/postgresql/data
-              subPath: pgdata
-          readinessProbe:
-            exec:
-              command:
-                - pg_isready
-                - -U
-                - foxhunt
-            initialDelaySeconds: 10
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              cpu: 500m
-              memory: 1Gi
-      volumes:
-        - name: postgres-data
-          persistentVolumeClaim:
-            claimName: postgres-pvc
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: postgres
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: postgres
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  selector:
-    app.kubernetes.io/name: postgres
-  ports:
-    - port: 5432
-      targetPort: 5432
-      name: postgres

infra/k8s/databases/questdb.yaml

@@ -1,95 +0,0 @@
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: questdb-pvc
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: questdb
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes:
-    - ReadWriteOnce
-  resources:
-    requests:
-      storage: 10Gi
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: questdb
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: questdb
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: questdb
-  strategy:
-    type: Recreate
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: questdb
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-      containers:
-        - name: questdb
-          image: questdb/questdb:8.2.3
-          ports:
-            - containerPort: 9009
-              name: ilp
-            - containerPort: 8812
-              name: pg
-            - containerPort: 9003
-              name: metrics
-          env:
-            - name: QDB_PG_ENABLED
-              value: "true"
-            - name: QDB_METRICS_ENABLED
-              value: "TRUE"
-          volumeMounts:
-            - name: questdb-data
-              mountPath: /var/lib/questdb
-          readinessProbe:
-            httpGet:
-              path: /exec?query=SELECT%201
-              port: 9003
-            initialDelaySeconds: 10
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              cpu: 500m
-              memory: 1Gi
-      volumes:
-        - name: questdb-data
-          persistentVolumeClaim:
-            claimName: questdb-pvc
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: questdb
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: questdb
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  selector:
-    app.kubernetes.io/name: questdb
-  ports:
-    - port: 9009
-      targetPort: 9009
-      name: ilp
-    - port: 8812
-      targetPort: 8812
-      name: pg
-    - port: 9003
-      targetPort: 9003
-      name: metrics

infra/k8s/databases/redis.yaml

@@ -1,63 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: redis
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: redis
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: redis
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: redis
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: infra
-      containers:
-        - name: redis
-          image: redis:7-alpine
-          command:
-            - redis-server
-            - --maxmemory
-            - 512mb
-            - --maxmemory-policy
-            - allkeys-lru
-          ports:
-            - containerPort: 6379
-              name: redis
-          readinessProbe:
-            exec:
-              command:
-                - redis-cli
-                - ping
-            initialDelaySeconds: 5
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              cpu: 250m
-              memory: 640Mi
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: redis
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: redis
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  selector:
-    app.kubernetes.io/name: redis
-  ports:
-    - port: 6379
-      targetPort: 6379
-      name: redis

infra/k8s/dns/node-local-dns.yaml

@@ -1,182 +0,0 @@
-# Node Bootstrap DaemonSet for Kapsule
-#
-# Two problems on Kapsule nodes:
-# 1. Nodes have nameserver 127.0.53.53 which can't resolve .svc.cluster.local
-# 2. Containerd defaults to HTTPS for non-localhost registries
-#
-# Fixes:
-# 1. Point /etc/resolv.conf to kube-dns ClusterIP (10.32.0.10)
-# 2. Create /etc/containerd/certs.d/ config for HTTP GitLab registry
-# 3. On GPU nodes (containerd v3 with conf.d/), add registry drop-in
-#
-# Init container sets up containerd registry config (one-time).
-# Main container manages resolv.conf (watches for Kapsule resets).
----
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: node-bootstrap
-  namespace: kube-system
-  labels:
-    app.kubernetes.io/name: node-bootstrap
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: node-bootstrap
-  updateStrategy:
-    type: RollingUpdate
-    rollingUpdate:
-      maxUnavailable: 1
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: node-bootstrap
-    spec:
-      hostNetwork: true
-      dnsPolicy: Default
-      priorityClassName: system-node-critical
-      tolerations:
-        - operator: Exists
-      hostPID: true
-      initContainers:
-        # Configure containerd to use HTTP for the GitLab registry.
-        # On GPU nodes (NVIDIA overlay), also creates a v3-compatible conf.d drop-in
-        # and restarts containerd so it picks up the new config.
-        - name: setup-registry
-          image: alpine:3.19
-          securityContext:
-            privileged: true
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              NEEDS_RESTART=false
-
-              # 1. Create per-host containerd registry config (HTTP for GitLab)
-              REGISTRY_DIR="/host-etc/containerd/certs.d/gitlab-registry.foxhunt.svc.cluster.local:5000"
-              mkdir -p "$REGISTRY_DIR"
-              DESIRED='server = "http://gitlab-registry.foxhunt.svc.cluster.local:5000"
-
-              [host."http://gitlab-registry.foxhunt.svc.cluster.local:5000"]
-                capabilities = ["pull", "resolve"]'
-              DESIRED=$(printf '%s' "$DESIRED" | sed 's/^              //')
-              if [ ! -f "$REGISTRY_DIR/hosts.toml" ] || [ "$(cat "$REGISTRY_DIR/hosts.toml")" != "$DESIRED" ]; then
-                printf '%s\n' "$DESIRED" > "$REGISTRY_DIR/hosts.toml"
-                echo "Created hosts.toml for HTTP gitlab-registry"
-                NEEDS_RESTART=true
-              else
-                echo "hosts.toml already up to date"
-              fi
-
-              # 2. Ensure conf.d/ exists and always write the v3 registry drop-in.
-              #    containerd 2.x may ignore the v2-style grpc.v1.cri.registry.config_path
-              #    even when it's set in config.toml. The v1 images drop-in is the reliable path.
-              #    Create conf.d/ if missing (base config already has imports = ["conf.d/*.toml"]).
-              CONF_D="/host-etc/containerd/conf.d"
-              mkdir -p "$CONF_D"
-              if [ -d "$CONF_D" ]; then
-                DROPIN='version = 3
-
-              [plugins."io.containerd.cri.v1.images".registry]
-                config_path = "/etc/containerd/certs.d"'
-                DROPIN=$(printf '%s' "$DROPIN" | sed 's/^              //')
-                if [ ! -f "$CONF_D/50-registry.toml" ] || [ "$(cat "$CONF_D/50-registry.toml")" != "$DROPIN" ]; then
-                  printf '%s\n' "$DROPIN" > "$CONF_D/50-registry.toml"
-                  echo "GPU node: created v3 registry config drop-in"
-                  NEEDS_RESTART=true
-                else
-                  echo "GPU node: v3 drop-in already up to date"
-                fi
-              fi
-
-              # 3. Restart containerd only if we changed config files.
-              #    Uses /proc/1/root chroot to access host systemctl (requires hostPID + privileged).
-              #    On a new node this will kill the DaemonSet pod; kubelet recreates it,
-              #    second run finds files unchanged (NEEDS_RESTART=false), proceeds normally.
-              if [ "$NEEDS_RESTART" = "true" ]; then
-                echo "Restarting containerd to pick up config changes..."
-                chroot /proc/1/root systemctl restart containerd || \
-                  echo "WARNING: containerd restart failed (node may need manual restart)"
-                echo "containerd restart issued"
-              fi
-          resources:
-            requests:
-              cpu: 10m
-              memory: 8Mi
-            limits:
-              cpu: 50m
-              memory: 16Mi
-          volumeMounts:
-            - name: host-etc
-              mountPath: /host-etc
-      containers:
-        - name: resolv-manager
-          image: busybox:1.37
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              RESOLV="/host-etc/resolv.conf"
-              BACKUP="/host-etc/resolv.conf.pre-kube-dns"
-              KUBE_DNS="10.32.0.10"
-
-              restore() {
-                if [ -f "$BACKUP" ]; then
-                  cp "$BACKUP" "$RESOLV"
-                  echo "$(date): Restored original resolv.conf"
-                fi
-                exit 0
-              }
-              trap restore TERM INT
-
-              # Save original
-              if [ ! -f "$BACKUP" ]; then
-                cp "$RESOLV" "$BACKUP"
-                echo "Saved original: $(cat "$BACKUP")"
-              fi
-
-              apply() {
-                # CRITICAL: keep VPC upstream resolver as FALLBACK after kube-dns.
-                # If resolv.conf ONLY has 10.32.0.10, node DNS is dead when CoreDNS
-                # is down — containerd can't pull images, CoreDNS can't start,
-                # entire cluster deadlocks (incident 2026-03-18, 2026-04-12).
-                # Scaleway VPC gateway pushes 172.16.0.13 as DNS via DHCP.
-                UPSTREAM=""
-                if [ -f "$BACKUP" ]; then
-                  UPSTREAM=$(grep "^nameserver" "$BACKUP" | grep -v "127.0.53.53\|$KUBE_DNS" | head -1 | awk '{print $2}')
-                fi
-                if [ -z "$UPSTREAM" ]; then
-                  UPSTREAM="169.254.169.254"  # Scaleway metadata DNS fallback
-                fi
-                printf 'nameserver %s\nnameserver %s\nsearch foxhunt.svc.cluster.local svc.cluster.local cluster.local\noptions ndots:5 timeout:2 attempts:3\n' "$KUBE_DNS" "$UPSTREAM" > "$RESOLV"
-                echo "$(date): resolv.conf → $KUBE_DNS + $UPSTREAM (fallback)"
-              }
-
-              # Initial apply
-              apply
-
-              # Watch for resets
-              while true; do
-                sleep 30 &
-                wait $!
-                if ! grep -q "$KUBE_DNS" "$RESOLV" 2>/dev/null; then
-                  echo "$(date): resolv.conf was reset, re-applying"
-                  apply
-                fi
-              done
-          resources:
-            requests:
-              cpu: 5m
-              memory: 8Mi
-            limits:
-              cpu: 10m
-              memory: 16Mi
-          volumeMounts:
-            - name: host-etc
-              mountPath: /host-etc
-      terminationGracePeriodSeconds: 10
-      volumes:
-        - name: host-etc
-          hostPath:
-            path: /etc
-            type: Directory

infra/k8s/gitlab/grafana-values.yaml

@@ -1,120 +0,0 @@
-# Grafana — Foxhunt monitoring dashboard
-# Accessed via Tailscale proxy on port 3000
-
-replicas: 1
-
-nodeSelector:
-  k8s.scaleway.com/pool-name: platform
-
-resources:
-  requests:
-    cpu: 50m
-    memory: 128Mi
-  limits:
-    cpu: 200m
-    memory: 256Mi
-
-persistence:
-  enabled: false
-
-admin:
-  existingSecret: grafana-admin
-  userKey: admin-user
-  passwordKey: admin-password
-
-envFromSecrets:
-  - name: db-credentials
-
-grafana.ini:
-  server:
-    root_url: https://grafana.fxhnt.ai
-  unified_alerting:
-    enabled: true
-  alerting:
-    enabled: false
-  database:
-    type: postgres
-    host: postgres:5432
-    name: grafana
-    user: foxhunt
-    password: $__env{password}
-    ssl_mode: disable
-
-datasources:
-  datasources.yaml:
-    apiVersion: 1
-    datasources:
-      - name: Prometheus
-        type: prometheus
-        url: http://prometheus-stack-kube-prom-prometheus.foxhunt.svc.cluster.local:9090
-        access: proxy
-        isDefault: true
-      - name: Alertmanager
-        type: alertmanager
-        uid: alertmanager
-        access: proxy
-        url: http://prometheus-stack-kube-prom-alertmanager.foxhunt.svc.cluster.local:9093
-        isDefault: false
-        jsonData:
-          implementation: prometheus
-          handleGrafanaManagedAlerts: true
-      - name: Loki
-        type: loki
-        access: proxy
-        url: http://loki.foxhunt.svc.cluster.local:3100
-        isDefault: false
-        jsonData:
-          derivedFields:
-            - datasourceUid: tempo
-              matcherRegex: '"trace_id":"(\w+)"'
-              name: TraceID
-              url: "$${__value.raw}"
-      - name: Tempo
-        type: tempo
-        uid: tempo
-        access: proxy
-        url: http://tempo.foxhunt.svc.cluster.local:3200
-        isDefault: false
-        jsonData:
-          tracesToLogsV2:
-            datasourceUid: loki
-            filterByTraceID: true
-          nodeGraph:
-            enabled: true
-          serviceMap:
-            datasourceUid: prometheus
-
-sidecar:
-  dashboards:
-    enabled: true
-    folder: /tmp/dashboards
-    folderAnnotation: grafana_folder
-    label: grafana_dashboard
-    labelValue: "1"
-    searchNamespace: foxhunt
-    provider:
-      foldersFromFilesStructure: true
-  datasources:
-    enabled: false
-    label: grafana_datasource
-    labelValue: ""
-
-dashboardProviders:
-  dashboardproviders.yaml:
-    apiVersion: 1
-    providers:
-      - name: default
-        orgId: 1
-        folder: Foxhunt
-        type: file
-        disableDeletion: false
-        editable: true
-        options:
-          path: /var/lib/grafana/dashboards/default
-
-dashboards:
-  default:
-    node-exporter:
-      gnetId: 1860
-      revision: 37
-      datasource: Prometheus

infra/k8s/gitlab/pat-rotation.yaml

@@ -1,253 +0,0 @@
-# GitLab PAT auto-rotation CronJob
-# Rotates the personal access token monthly, storing the new token in a K8s Secret.
-# Uses GitLab's atomic rotate API: creates new token + revokes old in one call.
-#
-# The rotation runs on the 1st of every month. Additionally, a weekly expiry check
-# runs every Monday and logs a warning if the token expires within 14 days, giving
-# time to investigate rotation failures before CI breaks.
-#
-# Initial setup:
-#   kubectl -n foxhunt create secret generic gitlab-pat \
-#     --from-literal=token=glpat-<initial-token>
-#
-# Manual rotation:
-#   kubectl -n foxhunt create job pat-rotate-manual --from=cronjob/gitlab-pat-rotation
-#
-# Manual expiry check:
-#   kubectl -n foxhunt create job pat-check-manual --from=cronjob/gitlab-pat-expiry-check
-#
-# Sync local token:
-#   export GITLAB_TOKEN=$(kubectl -n foxhunt get secret gitlab-pat -o jsonpath='{.data.token}' | base64 -d)
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: gitlab-pat-rotator
-  namespace: foxhunt
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: gitlab-pat-rotator
-  namespace: foxhunt
-rules:
-  - apiGroups: [""]
-    resources: ["secrets"]
-    resourceNames: ["gitlab-pat"]
-    verbs: ["get", "patch"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: gitlab-pat-rotator
-  namespace: foxhunt
-subjects:
-  - kind: ServiceAccount
-    name: gitlab-pat-rotator
-    namespace: foxhunt
-roleRef:
-  kind: Role
-  name: gitlab-pat-rotator
-  apiGroup: rbac.authorization.k8s.io
----
-# Monthly rotation — creates new token, revokes old, updates K8s secret
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: gitlab-pat-rotation
-  namespace: foxhunt
-spec:
-  schedule: "0 3 1 * *"  # 1st of every month at 03:00 UTC
-  successfulJobsHistoryLimit: 3
-  failedJobsHistoryLimit: 5
-  concurrencyPolicy: Forbid
-  jobTemplate:
-    spec:
-      backoffLimit: 3
-      activeDeadlineSeconds: 180
-      template:
-        metadata:
-          labels:
-            app: gitlab-pat-rotation
-        spec:
-          serviceAccountName: gitlab-pat-rotator
-          restartPolicy: Never
-          nodeSelector:
-            k8s.scaleway.com/pool-name: platform
-          containers:
-            - name: rotate
-              image: alpine:3.21
-              command: ["/bin/sh", "-c"]
-              args:
-                - |
-                  set -e
-
-                  apk add --no-cache -q curl jq
-
-                  CURRENT_PAT=$(cat /etc/gitlab-pat/token)
-                  GITLAB_URL="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-
-                  # 1. Verify current token is valid
-                  TOKEN_INFO=$(curl -sf "${GITLAB_URL}/api/v4/personal_access_tokens/self" \
-                    -H "PRIVATE-TOKEN: ${CURRENT_PAT}") || {
-                    echo "CRITICAL: Current PAT is invalid or GitLab unreachable."
-                    echo "Manual recovery required: create a new PAT in GitLab UI and run:"
-                    echo "  kubectl -n foxhunt create secret generic gitlab-pat --from-literal=token=<new-pat> --dry-run=client -o yaml | kubectl apply -f -"
-                    exit 1
-                  }
-                  OLD_EXPIRY=$(echo "$TOKEN_INFO" | jq -r '.expires_at')
-                  TOKEN_NAME=$(echo "$TOKEN_INFO" | jq -r '.name')
-                  echo "Current PAT '${TOKEN_NAME}' expires: ${OLD_EXPIRY}"
-
-                  # 2. Rotate: new expiry = 1 year from now
-                  EXPIRES=$(date -d "@$(($(date +%s) + 31536000))" +%Y-%m-%d)
-                  RESPONSE=$(curl -sf -X POST \
-                    "${GITLAB_URL}/api/v4/personal_access_tokens/self/rotate" \
-                    -H "PRIVATE-TOKEN: ${CURRENT_PAT}" \
-                    -d "expires_at=${EXPIRES}") || {
-                    echo "ERROR: Rotation API call failed. Token may still be valid."
-                    exit 1
-                  }
-
-                  NEW_PAT=$(echo "$RESPONSE" | jq -r '.token')
-                  if [ "$NEW_PAT" = "null" ] || [ -z "$NEW_PAT" ]; then
-                    echo "ERROR: No token in rotation response: $RESPONSE"
-                    exit 1
-                  fi
-
-                  echo "PAT rotated successfully. New expiry: ${EXPIRES}"
-
-                  # 3. Update K8s secret (old token is already revoked at this point)
-                  K8S_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
-                  K8S_CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
-                  K8S_NS=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
-                  K8S_API="https://kubernetes.default.svc"
-
-                  NEW_PAT_B64=$(printf '%s' "$NEW_PAT" | base64)
-
-                  for attempt in 1 2 3 4 5; do
-                    if curl -sf -X PATCH \
-                      "${K8S_API}/api/v1/namespaces/${K8S_NS}/secrets/gitlab-pat" \
-                      --cacert "$K8S_CA" \
-                      -H "Authorization: Bearer ${K8S_TOKEN}" \
-                      -H "Content-Type: application/strategic-merge-patch+json" \
-                      -d "{\"data\":{\"token\":\"${NEW_PAT_B64}\"}}"; then
-                      echo ""
-                      echo "K8s secret updated on attempt ${attempt}."
-
-                      # 4. Verify the new token works
-                      VERIFY=$(curl -sf "${GITLAB_URL}/api/v4/personal_access_tokens/self" \
-                        -H "PRIVATE-TOKEN: ${NEW_PAT}" | jq -r '.expires_at') || true
-                      if [ "$VERIFY" = "$EXPIRES" ]; then
-                        echo "Verification passed: new token valid until ${VERIFY}"
-                      else
-                        echo "WARNING: Verification returned unexpected expiry: ${VERIFY}"
-                      fi
-                      exit 0
-                    fi
-                    echo "Secret update attempt ${attempt} failed, retrying in 3s..."
-                    sleep 3
-                  done
-
-                  # If we get here, rotation succeeded but secret update failed.
-                  # Print token so it can be recovered from job logs.
-                  echo "CRITICAL: Secret update failed after 5 attempts!"
-                  echo "New PAT (recover manually): ${NEW_PAT}"
-                  echo "Run: kubectl -n foxhunt create secret generic gitlab-pat --from-literal=token=<above> --dry-run=client -o yaml | kubectl apply -f -"
-                  exit 1
-              volumeMounts:
-                - name: gitlab-pat
-                  mountPath: /etc/gitlab-pat
-                  readOnly: true
-              resources:
-                requests:
-                  cpu: 50m
-                  memory: 32Mi
-                limits:
-                  cpu: 200m
-                  memory: 64Mi
-          volumes:
-            - name: gitlab-pat
-              secret:
-                secretName: gitlab-pat
----
-# Weekly expiry check — warns if token expires within 14 days
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: gitlab-pat-expiry-check
-  namespace: foxhunt
-spec:
-  schedule: "0 8 * * 1"  # Every Monday at 08:00 UTC
-  successfulJobsHistoryLimit: 1
-  failedJobsHistoryLimit: 3
-  concurrencyPolicy: Forbid
-  jobTemplate:
-    spec:
-      backoffLimit: 1
-      activeDeadlineSeconds: 60
-      template:
-        metadata:
-          labels:
-            app: gitlab-pat-expiry-check
-        spec:
-          restartPolicy: Never
-          nodeSelector:
-            k8s.scaleway.com/pool-name: platform
-          containers:
-            - name: check
-              image: alpine:3.21
-              command: ["/bin/sh", "-c"]
-              args:
-                - |
-                  set -e
-
-                  apk add --no-cache -q curl jq
-
-                  CURRENT_PAT=$(cat /etc/gitlab-pat/token)
-                  GITLAB_URL="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-
-                  TOKEN_INFO=$(curl -sf "${GITLAB_URL}/api/v4/personal_access_tokens/self" \
-                    -H "PRIVATE-TOKEN: ${CURRENT_PAT}") || {
-                    echo "CRITICAL: PAT is INVALID. CI will fail!"
-                    echo "Create a new PAT in GitLab UI and run:"
-                    echo "  kubectl -n foxhunt create secret generic gitlab-pat --from-literal=token=<new-pat> --dry-run=client -o yaml | kubectl apply -f -"
-                    exit 1
-                  }
-
-                  EXPIRY=$(echo "$TOKEN_INFO" | jq -r '.expires_at')
-                  TOKEN_NAME=$(echo "$TOKEN_INFO" | jq -r '.name')
-                  SCOPES=$(echo "$TOKEN_INFO" | jq -r '.scopes | join(",")')
-
-                  # Calculate days until expiry
-                  EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s)
-                  NOW_EPOCH=$(date +%s)
-                  DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))
-
-                  echo "PAT '${TOKEN_NAME}': expires=${EXPIRY} scopes=${SCOPES} days_left=${DAYS_LEFT}"
-
-                  if [ "$DAYS_LEFT" -lt 0 ]; then
-                    echo "CRITICAL: PAT has EXPIRED ${DAYS_LEFT} days ago!"
-                    exit 1
-                  elif [ "$DAYS_LEFT" -lt 14 ]; then
-                    echo "WARNING: PAT expires in ${DAYS_LEFT} days. Rotation may have failed."
-                    echo "Manual rotation: kubectl -n foxhunt create job pat-rotate-manual --from=cronjob/gitlab-pat-rotation"
-                    exit 1
-                  else
-                    echo "OK: PAT valid for ${DAYS_LEFT} more days."
-                  fi
-              volumeMounts:
-                - name: gitlab-pat
-                  mountPath: /etc/gitlab-pat
-                  readOnly: true
-              resources:
-                requests:
-                  cpu: 10m
-                  memory: 32Mi
-                limits:
-                  cpu: 100m
-                  memory: 64Mi
-          volumes:
-            - name: gitlab-pat
-              secret:
-                secretName: gitlab-pat

infra/k8s/gitlab/postgres-init.yaml

@@ -1,63 +0,0 @@
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: gitlab-postgres-init
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: gitlab-postgres-init
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  ttlSecondsAfterFinished: 300
-  backoffLimit: 3
-  template:
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: infra
-      restartPolicy: Never
-      containers:
-        - name: init
-          image: timescale/timescaledb:latest-pg16
-          command:
-            - bash
-            - -c
-            - |
-              set -euo pipefail
-              export PGPASSWORD="$POSTGRES_PASSWORD"
-
-              until pg_isready -h postgres -U foxhunt; do
-                echo "Waiting for postgres..."
-                sleep 2
-              done
-
-              echo "Creating gitlab user..."
-              psql -h postgres -U foxhunt -d foxhunt -tc \
-                "SELECT 1 FROM pg_roles WHERE rolname='gitlab'" | grep -q 1 || \
-                psql -h postgres -U foxhunt -d foxhunt -c \
-                  "CREATE ROLE gitlab WITH LOGIN PASSWORD '${GITLAB_DB_PASSWORD}'"
-
-              echo "Creating gitlab database..."
-              psql -h postgres -U foxhunt -tc \
-                "SELECT 1 FROM pg_database WHERE datname='gitlab'" | grep -q 1 || \
-                psql -h postgres -U foxhunt -c \
-                  "CREATE DATABASE gitlab OWNER gitlab"
-
-              echo "Creating extensions..."
-              psql -h postgres -U foxhunt -d gitlab -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
-              psql -h postgres -U foxhunt -d gitlab -c "CREATE EXTENSION IF NOT EXISTS btree_gist;"
-
-              echo "Granting permissions..."
-              psql -h postgres -U foxhunt -d gitlab -c "GRANT ALL PRIVILEGES ON DATABASE gitlab TO gitlab;"
-              psql -h postgres -U foxhunt -d gitlab -c "GRANT ALL ON SCHEMA public TO gitlab;"
-
-              echo "GitLab database initialized successfully."
-          env:
-            - name: POSTGRES_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-            - name: GITLAB_DB_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: gitlab-secrets
-                  key: db-password

infra/k8s/gitlab/runner-h100-values.yaml

@@ -1,73 +0,0 @@
-# GitLab Runner — H100 GPU training workloads (hyperopt, walk-forward)
-# Runs on ci-training-h100 pool (H100 PCIe 1x80GB)
-# SXM pools have zero Scaleway quota — use PCIe until quota is granted.
-
-gitlabUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-# runnerToken set via --set at install time
-
-replicas: 1
-
-# Reuse the existing gitlab-runner SA (has pods/secrets/configmaps RBAC)
-rbac:
-  create: false
-serviceAccount:
-  create: false
-  name: gitlab-runner
-
-nodeSelector:
-  k8s.scaleway.com/pool-name: infra
-
-runners:
-  cloneUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-  config: |
-    [[runners]]
-      clone_url = "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-      tag_list = ["kapsule", "h100"]
-      [runners.kubernetes]
-        namespace = "foxhunt"
-        service_account = "gitlab-runner"
-        image = "rust:1.89-slim"
-        privileged = false
-        node_selector_overwrite_allowed = ".*"
-        cpu_request_overwrite_max_allowed = "24000m"
-        cpu_limit_overwrite_max_allowed = "24000m"
-        memory_request_overwrite_max_allowed = "200Gi"
-        memory_limit_overwrite_max_allowed = "200Gi"
-        poll_timeout = 600
-        runtime_class_name = "nvidia"
-        pod_annotations_overwrite_allowed = ".*"
-        # Default resources for H100 training
-        cpu_request = "2000m"
-        cpu_limit = "3800m"
-        memory_request = "4Gi"
-        memory_limit = "8Gi"
-        helper_cpu_request = "100m"
-        helper_cpu_limit = "500m"
-        helper_memory_request = "128Mi"
-        helper_memory_limit = "512Mi"
-        image_pull_secrets = ["gitlab-registry"]
-      [runners.kubernetes.node_selector]
-        "k8s.scaleway.com/pool-name" = "ci-training-h100"
-      [runners.kubernetes.node_tolerations]
-        "nvidia.com/gpu" = "NoSchedule"
-        "node.cilium.io/agent-not-ready" = "NoSchedule"
-      [runners.kubernetes.pod_labels]
-        "app.kubernetes.io/part-of" = "foxhunt-ci"
-      # Request GPU via K8s scheduler so only one training pod runs per GPU
-      [[runners.kubernetes.pod_spec]]
-        name = "build"
-        patch_type = "strategic"
-        patch = '{"containers":[{"name":"build","resources":{"requests":{"nvidia.com/gpu":"1"},"limits":{"nvidia.com/gpu":"1"}}}]}'
-      # H100 PCIe PVCs (separate from L40S to avoid RWO conflicts)
-      [[runners.kubernetes.volumes.pvc]]
-        name = "training-data-h100-pvc"
-        mount_path = "/mnt/training-data"
-        read_only = true
-      [[runners.kubernetes.volumes.pvc]]
-        name = "sccache-h100-pvc"
-        mount_path = "/mnt/sccache"
-        read_only = false
-
-  tags: "kapsule,h100"
-
-concurrent: 2

infra/k8s/gitlab/runner-h100x2-values.yaml

@@ -1,69 +0,0 @@
-# GitLab Runner — 2×H100 GPU training workloads (multi-GPU hyperopt)
-# Runs on ci-training-h100x2 pool (2× H100 PCIe: 160GB total VRAM)
-# Multi-GPU auto-detected by MultiGpuConfig::detect() in ml crate.
-
-gitlabUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-
-replicas: 1
-
-rbac:
-  create: false
-serviceAccount:
-  create: false
-  name: gitlab-runner
-
-nodeSelector:
-  k8s.scaleway.com/pool-name: infra
-
-runners:
-  cloneUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-  config: |
-    [[runners]]
-      clone_url = "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-      tag_list = ["kapsule", "h100x2"]
-      [runners.kubernetes]
-        namespace = "foxhunt"
-        service_account = "gitlab-runner"
-        image = "rust:1.89-slim"
-        privileged = false
-        node_selector_overwrite_allowed = ".*"
-        cpu_request_overwrite_max_allowed = "16000m"
-        cpu_limit_overwrite_max_allowed = "16000m"
-        memory_request_overwrite_max_allowed = "192Gi"
-        memory_limit_overwrite_max_allowed = "192Gi"
-        poll_timeout = 600
-        runtime_class_name = "nvidia"
-        pod_annotations_overwrite_allowed = ".*"
-        cpu_request = "4000m"
-        cpu_limit = "8000m"
-        memory_request = "8Gi"
-        memory_limit = "16Gi"
-        helper_cpu_request = "100m"
-        helper_cpu_limit = "500m"
-        helper_memory_request = "128Mi"
-        helper_memory_limit = "512Mi"
-        image_pull_secrets = ["gitlab-registry"]
-      [runners.kubernetes.node_selector]
-        "k8s.scaleway.com/pool-name" = "ci-training-h100x2"
-      [runners.kubernetes.node_tolerations]
-        "nvidia.com/gpu" = "NoSchedule"
-        "node.cilium.io/agent-not-ready" = "NoSchedule"
-      [runners.kubernetes.pod_labels]
-        "app.kubernetes.io/part-of" = "foxhunt-ci"
-      # Request both GPUs via K8s scheduler for exclusive access
-      [[runners.kubernetes.pod_spec]]
-        name = "build"
-        patch_type = "strategic"
-        patch = '{"containers":[{"name":"build","resources":{"requests":{"nvidia.com/gpu":"2"},"limits":{"nvidia.com/gpu":"2"}}}]}'
-      [[runners.kubernetes.volumes.pvc]]
-        name = "training-data-h100x2-pvc"
-        mount_path = "/mnt/training-data"
-        read_only = true
-      [[runners.kubernetes.volumes.pvc]]
-        name = "sccache-h100x2-pvc"
-        mount_path = "/mnt/sccache"
-        read_only = false
-
-  tags: "kapsule,h100x2"
-
-concurrent: 2

infra/k8s/gitlab/runner-rl-values.yaml

@@ -1,76 +0,0 @@
-# GitLab Runner — RL training workloads (DQN, PPO)
-# Runs on ci-training pool (L40S: 48GB VRAM)
-# Mounts separate PVCs to avoid RWO conflicts with main runner
-
-gitlabUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-# runnerToken set via --set at install time
-
-replicas: 1
-
-# Reuse the existing gitlab-runner SA (has pods/secrets/configmaps RBAC)
-rbac:
-  create: false
-serviceAccount:
-  create: false
-  name: gitlab-runner
-
-nodeSelector:
-  k8s.scaleway.com/pool-name: infra
-
-runners:
-  cloneUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-  config: |
-    [[runners]]
-      clone_url = "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-      tag_list = ["kapsule", "l40s"]
-      [runners.kubernetes]
-        namespace = "foxhunt"
-        service_account = "gitlab-runner"
-        image = "rust:1.89-slim"
-        privileged = false
-        node_selector_overwrite_allowed = ".*"
-        # Hyperopt RL jobs need 6000m CPU + 20Gi mem (PSO parallel trials)
-        cpu_request_overwrite_max_allowed = "8000m"
-        cpu_limit_overwrite_max_allowed = "8000m"
-        memory_request_overwrite_max_allowed = "48Gi"
-        memory_limit_overwrite_max_allowed = "48Gi"
-        poll_timeout = 600
-        # All RL runner jobs target GPU nodes → set nvidia runtime globally
-        runtime_class_name = "nvidia"
-        # Allow CI jobs to set pod annotations (training jobs expose Prometheus metrics)
-        pod_annotations_overwrite_allowed = ".*"
-        # Default resources for RL training on L40S
-        cpu_request = "2000m"
-        cpu_limit = "3800m"
-        memory_request = "4Gi"
-        memory_limit = "8Gi"
-        helper_cpu_request = "100m"
-        helper_cpu_limit = "500m"
-        helper_memory_request = "128Mi"
-        helper_memory_limit = "512Mi"
-        image_pull_secrets = ["gitlab-registry"]
-      [runners.kubernetes.node_selector]
-        "k8s.scaleway.com/pool-name" = "ci-training"
-      [runners.kubernetes.node_tolerations]
-        "nvidia.com/gpu" = "NoSchedule"
-        "node.cilium.io/agent-not-ready" = "NoSchedule"
-      [runners.kubernetes.pod_labels]
-        "app.kubernetes.io/part-of" = "foxhunt-ci"
-      # Request GPU via K8s scheduler so only one training pod runs per GPU
-      [[runners.kubernetes.pod_spec]]
-        name = "build"
-        patch_type = "strategic"
-        patch = '{"containers":[{"name":"build","resources":{"requests":{"nvidia.com/gpu":"1"},"limits":{"nvidia.com/gpu":"1"}}}]}'
-      # Training PVCs — separate from main runner to avoid RWO Multi-Attach errors
-      [[runners.kubernetes.volumes.pvc]]
-        name = "training-data-l4-pvc"
-        mount_path = "/mnt/training-data"
-        read_only = true
-      [[runners.kubernetes.volumes.pvc]]
-        name = "sccache-l4-pvc"
-        mount_path = "/mnt/sccache"
-        read_only = false
-
-  tags: "kapsule,l40s"
-
-concurrent: 2

infra/k8s/gitlab/runner-values.yaml

@@ -1,86 +0,0 @@
-# GitLab Runner — Kubernetes executor
-# Runner manager pod lives on gitlab node pool
-# Default: build pods spawn on ci-compile-cpu pool (POP2-32C-128G)
-# CI jobs override node_selector for their target pool
-
-gitlabUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-# runnerToken set via --set at install time
-
-replicas: 1
-
-nodeSelector:
-  k8s.scaleway.com/pool-name: platform
-
-runners:
-  # Override clone URL to internal service (pods can't reach Tailscale IPs)
-  cloneUrl: http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181
-  config: |
-    [[runners]]
-      clone_url = "http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
-      tag_list = ["kapsule", "rust", "docker", "gpu"]
-      [runners.kubernetes]
-        namespace = "foxhunt"
-        image = "rust:1.89-slim"
-        privileged = false
-        # Allow CI jobs to override node selector via KUBERNETES_NODE_SELECTOR_* vars
-        # Format: KUBERNETES_NODE_SELECTOR_<LABEL>: "key=value"
-        node_selector_overwrite_allowed = ".*"
-        # Allow CI jobs to override resources via KUBERNETES_*_{REQUEST,LIMIT} vars
-        # Values are maximums (not regexes) — training can request up to 80Gi
-        cpu_request_overwrite_max_allowed = "24000m"
-        cpu_limit_overwrite_max_allowed = "24000m"
-        memory_request_overwrite_max_allowed = "80Gi"
-        memory_limit_overwrite_max_allowed = "80Gi"
-        # Scale-to-zero pools need ~3-5 min to provision; default 180s times out
-        poll_timeout = 600
-        # Allow CI jobs to override runtimeClassName via KUBERNETES_RUNTIME_CLASS_NAME
-        # GPU jobs set "nvidia"; compile/kaniko jobs leave unset (default runc)
-        runtime_class_name_overwrite_allowed = ".*"
-        # Allow CI jobs to set pod annotations (training jobs expose Prometheus metrics)
-        pod_annotations_overwrite_allowed = ".*"
-        # Default resource limits for ci-compile-cpu (POP2-32C-128G)
-        cpu_request = "3500m"
-        cpu_limit = "7800m"
-        memory_request = "12Gi"
-        memory_limit = "28Gi"
-        helper_cpu_request = "100m"
-        helper_cpu_limit = "500m"
-        helper_memory_request = "128Mi"
-        helper_memory_limit = "512Mi"
-        image_pull_secrets = ["gitlab-registry"]
-      # Sub-tables must come AFTER all scalar values (TOML rule)
-      [runners.kubernetes.node_selector]
-        "k8s.scaleway.com/pool-name" = "ci-compile-cpu"
-      [runners.kubernetes.node_tolerations]
-        "nvidia.com/gpu" = "NoSchedule"
-        # Cilium CNI agent takes ~30s to initialize on fresh scale-from-zero nodes
-        "node.cilium.io/agent-not-ready" = "NoSchedule"
-      [runners.kubernetes.pod_labels]
-        "app.kubernetes.io/part-of" = "foxhunt-ci"
-      # Mount training data PVC (Databento futures .dbn.zst files)
-      # Path must NOT be under /data — Redis image's WORKDIR is /data and entrypoint chowns it
-      [[runners.kubernetes.volumes.pvc]]
-        name = "training-data-pvc"
-        mount_path = "/mnt/training-data"
-        read_only = true
-
-  # Runner tags for job matching
-  tags: "kapsule,rust,docker,gpu"
-
-# Concurrency — 10 allows all 9 Kaniko builds + test to run in parallel
-concurrent: 10
-
-# RBAC for runner to spawn pods
-rbac:
-  create: true
-  rules:
-    - apiGroups: [""]
-      resources: ["pods", "pods/exec", "pods/log", "secrets", "configmaps"]
-      verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
-    - apiGroups: [""]
-      resources: ["pods/attach"]
-      verbs: ["create", "get"]
-
-serviceAccount:
-  create: true
-  name: gitlab-runner

infra/k8s/gitlab/tailscale-proxy.yaml

@@ -1,348 +0,0 @@
----
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  name: tailscale-gitlab
-  namespace: foxhunt
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  name: tailscale-gitlab
-  namespace: foxhunt
-rules:
-  - apiGroups: [""]
-    resources: ["secrets"]
-    verbs: ["create", "get", "update", "patch"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: RoleBinding
-metadata:
-  name: tailscale-gitlab
-  namespace: foxhunt
-subjects:
-  - kind: ServiceAccount
-    name: tailscale-gitlab
-    namespace: foxhunt
-roleRef:
-  kind: Role
-  name: tailscale-gitlab
-  apiGroup: rbac.authorization.k8s.io
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: tailscale-gitlab-proxy
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: tailscale-gitlab-proxy
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: tailscale-gitlab-proxy
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: tailscale-gitlab-proxy
-    spec:
-      serviceAccountName: tailscale-gitlab
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-      containers:
-        - name: nginx
-          image: nginx:alpine
-          ports:
-            - containerPort: 80
-            - containerPort: 443
-            - containerPort: 5050
-          volumeMounts:
-            - name: nginx-conf
-              mountPath: /etc/nginx/conf.d
-            - name: tls-certs
-              mountPath: /etc/nginx/certs
-              readOnly: true
-          resources:
-            requests:
-              cpu: 25m
-              memory: 32Mi
-            limits:
-              cpu: 100m
-              memory: 64Mi
-        - name: ssh-proxy
-          image: alpine/socat:latest
-          args:
-            - "TCP-LISTEN:2222,fork,reuseaddr,sndbuf=1048576,rcvbuf=1048576"
-            - "TCP:gitlab-gitlab-shell.foxhunt.svc.cluster.local:2222,sndbuf=1048576,rcvbuf=1048576"
-          ports:
-            - containerPort: 2222
-          resources:
-            requests:
-              cpu: 25m
-              memory: 64Mi
-            limits:
-              cpu: 200m
-              memory: 128Mi
-        - name: imap-proxy
-          image: alpine/socat:latest
-          args:
-            - "TCP-LISTEN:993,fork,reuseaddr"
-            - "TCP:stalwart.foxhunt.svc.cluster.local:993"
-          ports:
-            - containerPort: 993
-          resources:
-            requests:
-              cpu: 10m
-              memory: 16Mi
-            limits:
-              cpu: 50m
-              memory: 32Mi
-        - name: tailscale
-          image: ghcr.io/tailscale/tailscale:latest
-          env:
-            - name: TS_AUTHKEY
-              valueFrom:
-                secretKeyRef:
-                  name: tailscale-auth
-                  key: TS_AUTHKEY
-            - name: TS_KUBE_SECRET
-              value: tailscale-gitlab-state
-            - name: TS_USERSPACE
-              value: "false"
-            - name: TS_HOSTNAME
-              value: "foxhunt-gitlab"
-            - name: TS_ACCEPT_DNS
-              value: "false"
-            - name: TS_EXTRA_ARGS
-              value: ""
-          resources:
-            requests:
-              cpu: 25m
-              memory: 32Mi
-            limits:
-              cpu: 100m
-              memory: 64Mi
-          securityContext:
-            capabilities:
-              add:
-                - NET_ADMIN
-                - NET_RAW
-      volumes:
-        - name: nginx-conf
-          configMap:
-            name: tailscale-gitlab-nginx
-        - name: tls-certs
-          secret:
-            secretName: gitlab-tls-cert
----
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: tailscale-gitlab-nginx
-  namespace: foxhunt
-data:
-  default.conf: |
-    # HTTP → HTTPS redirect for all subdomains
-    server {
-        listen 80;
-        server_name *.fxhnt.ai;
-        return 301 https://$host$request_uri;
-    }
-
-    # GitLab — git.fxhnt.ai
-    server {
-        listen 443 ssl;
-        server_name git.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        client_max_body_size 0;
-
-        location / {
-            proxy_pass http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-
-            proxy_buffering off;
-            proxy_request_buffering off;
-            proxy_http_version 1.1;
-            proxy_read_timeout 3600s;
-            proxy_send_timeout 3600s;
-            proxy_connect_timeout 300s;
-        }
-    }
-
-    # Container Registry — git.fxhnt.ai:5050
-    server {
-        listen 5050 ssl;
-        server_name git.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        client_max_body_size 0;
-
-        location / {
-            proxy_pass http://gitlab-registry.foxhunt.svc.cluster.local:5000;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-
-            proxy_buffering off;
-            proxy_request_buffering off;
-            proxy_http_version 1.1;
-            proxy_read_timeout 3600s;
-            proxy_send_timeout 3600s;
-        }
-    }
-
-    # Grafana — grafana.fxhnt.ai
-    server {
-        listen 443 ssl;
-        server_name grafana.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        location / {
-            proxy_pass http://grafana.foxhunt.svc.cluster.local:80;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-            proxy_http_version 1.1;
-            proxy_set_header Upgrade $http_upgrade;
-            proxy_set_header Connection "upgrade";
-        }
-    }
-
-    # Web Dashboard — dashboard.fxhnt.ai
-    server {
-        listen 443 ssl;
-        server_name dashboard.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        location / {
-            proxy_pass http://web-dashboard.foxhunt.svc.cluster.local:80;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-            proxy_http_version 1.1;
-            proxy_set_header Upgrade $http_upgrade;
-            proxy_set_header Connection "upgrade";
-        }
-    }
-
-    # API Gateway (gRPC-Web + gRPC) — api.fxhnt.ai
-    # Browser dashboard uses gRPC-web (HTTP/1.1 POST), so proxy_pass not grpc_pass
-    server {
-        listen 443 ssl;
-        http2 on;
-        server_name api.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        location / {
-            proxy_pass http://api.foxhunt.svc.cluster.local:50051;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-            proxy_http_version 1.1;
-            proxy_read_timeout 3600s;
-            proxy_send_timeout 3600s;
-            proxy_connect_timeout 30s;
-        }
-    }
-
-    # Stalwart Admin — mail.fxhnt.ai
-    server {
-        listen 443 ssl;
-        server_name mail.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        location / {
-            proxy_pass https://stalwart.foxhunt.svc.cluster.local:443;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-            proxy_http_version 1.1;
-            proxy_ssl_verify off;
-        }
-    }
-
-    # Mattermost — chat.fxhnt.ai
-    server {
-        listen 443 ssl;
-        server_name chat.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        client_max_body_size 50m;
-
-        location / {
-            proxy_pass http://mattermost.foxhunt.svc.cluster.local:8065;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-            proxy_http_version 1.1;
-            proxy_set_header Upgrade $http_upgrade;
-            proxy_set_header Connection "upgrade";
-            proxy_read_timeout 3600s;
-            proxy_send_timeout 3600s;
-        }
-    }
-
-    # MinIO S3 API — minio.fxhnt.ai
-    server {
-        listen 443 ssl;
-        server_name minio.fxhnt.ai;
-
-        ssl_certificate /etc/nginx/certs/tls.crt;
-        ssl_certificate_key /etc/nginx/certs/tls.key;
-        ssl_protocols TLSv1.2 TLSv1.3;
-
-        client_max_body_size 0;
-
-        location / {
-            proxy_pass http://minio.foxhunt.svc.cluster.local:9000;
-            proxy_set_header Host $http_host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto https;
-
-            proxy_buffering off;
-            proxy_request_buffering off;
-            proxy_http_version 1.1;
-            proxy_read_timeout 3600s;
-            proxy_send_timeout 3600s;
-            proxy_connect_timeout 300s;
-
-            # MinIO uses self-signed TLS internally
-            proxy_ssl_verify off;
-        }
-    }
-

infra/k8s/gitlab/values.yaml

@@ -1,196 +0,0 @@
-# GitLab CE - Foxhunt deployment
-# Runs on 'infra' node pool alongside platform services
-# External Postgres + Redis on same pool
-
-global:
-  edition: ce
-  kas:
-    enabled: false
-  hosts:
-    domain: fxhnt.ai
-    gitlab:
-      name: git.fxhnt.ai
-      https: true
-    registry:
-      name: git.fxhnt.ai
-    https: true
-  ingress:
-    enabled: false
-    configureCertmanager: false
-  shell:
-    port: 2222
-  psql:
-    host: postgres.foxhunt.svc.cluster.local
-    port: 5432
-    database: gitlab
-    username: gitlab
-    password:
-      secret: gitlab-secrets
-      key: db-password
-  redis:
-    host: redis.foxhunt.svc.cluster.local
-    port: 6379
-    auth:
-      enabled: false
-  minio:
-    enabled: false
-  registry:
-    bucket: foxhunt-gitlab-registry
-  appConfig:
-    lfs:
-      bucket: foxhunt-gitlab-artifacts
-      connection:
-        secret: gitlab-s3-credentials
-        key: connection
-    artifacts:
-      bucket: foxhunt-gitlab-artifacts
-      connection:
-        secret: gitlab-s3-credentials
-        key: connection
-    uploads:
-      bucket: foxhunt-gitlab-artifacts
-      connection:
-        secret: gitlab-s3-credentials
-        key: connection
-    packages:
-      bucket: foxhunt-gitlab-artifacts
-      connection:
-        secret: gitlab-s3-credentials
-        key: connection
-    gitlab_kas:
-      enabled: false
-
-# Disable bundled databases — use external
-postgresql:
-  install: false
-redis:
-  install: false
-
-# Disable components we don't need
-installCertmanager: false
-certmanager:
-  installCRDs: false
-nginx-ingress:
-  enabled: false
-prometheus:
-  install: true
-  rbac:
-    create: true
-  alertmanager:
-    enabled: false
-  server:
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-    persistentVolume:
-      enabled: false
-    resources:
-      requests:
-        cpu: 100m
-        memory: 512Mi
-      limits:
-        cpu: 500m
-        memory: 1Gi
-gitlab-runner:
-  install: false
-
-# Registry — S3-backed, NodePort for kubelet image pulls
-# Token auth disabled (cluster-internal only, secured by network policies)
-# Pods pull via gitlab-registry.foxhunt.svc.cluster.local:5000 (internal DNS)
-registry:
-  replicaCount: 1
-  hpa:
-    minReplicas: 1
-    maxReplicas: 1
-  service:
-    type: NodePort
-    nodePort: 30500
-  storage:
-    secret: gitlab-s3-credentials
-    key: registry
-    redirect:
-      disable: true
-  nodeSelector:
-    k8s.scaleway.com/pool-name: infra
-
-# GitLab core components — all on infra node pool
-gitlab:
-  webservice:
-    replicaCount: 2
-    hpa:
-      minReplicas: 2
-      maxReplicas: 2
-    workerProcesses: 2
-    workhorse:
-      extraArgs: "-apiLimit 0 -apiQueueLimit 0"
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-    resources:
-      requests:
-        cpu: 500m
-        memory: 2Gi
-      limits:
-        cpu: 2000m
-        memory: 4Gi
-    extraEnv:
-      REDIS_URL: "redis://redis.foxhunt.svc.cluster.local:6379/8"
-
-  sidekiq:
-    replicas: 1
-    hpa:
-      minReplicas: 1
-      maxReplicas: 1
-    concurrency: 10
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-    resources:
-      requests:
-        cpu: 250m
-        memory: 1Gi
-      limits:
-        cpu: 1000m
-        memory: 2Gi
-    extraEnv:
-      REDIS_URL: "redis://redis.foxhunt.svc.cluster.local:6379/8"
-
-  gitaly:
-    persistence:
-      enabled: true
-      size: 50Gi
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-    resources:
-      requests:
-        cpu: 200m
-        memory: 512Mi
-      limits:
-        cpu: 1000m
-        memory: 2Gi
-
-  gitlab-shell:
-    replicaCount: 1
-    hpa:
-      minReplicas: 1
-      maxReplicas: 1
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-    service:
-      type: NodePort
-      nodePort: 32222
-
-  toolbox:
-    backups:
-      objectStorage:
-        config:
-          secret: gitlab-s3-credentials
-          key: connection
-    nodeSelector:
-      k8s.scaleway.com/pool-name: infra
-
-  kas:
-    enabled: false
-    hpa:
-      minReplicas: 0
-      maxReplicas: 0
-
-  gitlab-exporter:
-    enabled: false

infra/k8s/gpu-overlays/ml-training-service-gpu.yaml

@@ -1,169 +0,0 @@
-# GPU-enabled overlay for ml-training-service
-# Apply manually: kubectl apply -f infra/k8s/gpu-overlays/ml-training-service-gpu.yaml
-# Revert to CPU: kubectl apply -f infra/k8s/services/ml-training-service.yaml
-#
-# Binary fetched from MinIO at pod startup — works on any node pool.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: ml-training-service
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: ml-training-service
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  strategy:
-    type: RollingUpdate
-    rollingUpdate:
-      maxSurge: 0
-      maxUnavailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: ml-training-service
-  template:
-    metadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "9094"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: ml-training-service
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      serviceAccountName: ml-training-service
-      securityContext:
-        runAsNonRoot: true
-        runAsUser: 1000
-        runAsGroup: 1000
-        fsGroup: 1000
-        seccompProfile:
-          type: RuntimeDefault
-      nodeSelector:
-        k8s.scaleway.com/pool-name: gpu-inference
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-      imagePullSecrets:
-        - name: gitlab-registry
-      initContainers:
-        - name: fetch-binary
-          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              rclone copyto \
-                ":s3:foxhunt-binaries/services/ml-training-service" \
-                "/binaries/ml-training-service" \
-                --s3-provider=Minio \
-                --s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
-                --s3-access-key-id="${MINIO_ACCESS_KEY}" \
-                --s3-secret-access-key="${MINIO_SECRET_KEY}" \
-                --s3-no-check-bucket
-              chmod +x /binaries/ml-training-service
-              echo "Fetched ml-training-service ($(stat -c%s /binaries/ml-training-service) bytes)"
-          env:
-            - name: MINIO_ACCESS_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: access-key
-            - name: MINIO_SECRET_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: secret-key
-          volumeMounts:
-            - name: binaries
-              mountPath: /binaries
-          resources:
-            requests:
-              cpu: 100m
-              memory: 64Mi
-            limits:
-              cpu: 500m
-              memory: 128Mi
-      containers:
-        - name: ml-training-service
-          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-          command: ["/binaries/ml-training-service", "serve"]
-          ports:
-            - containerPort: 50053
-              name: grpc
-            - containerPort: 9094
-              name: metrics
-          env:
-            - name: DATABASE_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-            - name: DATABASE_URL
-              value: "postgresql://foxhunt:$(DATABASE_PASSWORD)@postgres:5432/foxhunt"
-            - name: REDIS_URL
-              value: "redis://redis:6379"
-            - name: JWT_SECRET
-              valueFrom:
-                secretKeyRef:
-                  name: jwt-secret
-                  key: secret
-            - name: JWT_ISSUER
-              value: foxhunt-api
-            - name: JWT_AUDIENCE
-              value: foxhunt-services
-            - name: S3_ENDPOINT
-              value: "http://minio.foxhunt.svc.cluster.local:9000"
-            - name: S3_BUCKET
-              value: foxhunt-models
-            - name: ENABLE_GPU
-              value: "true"
-            - name: RUST_LOG
-              value: info
-            - name: OTEL_EXPORTER_OTLP_ENDPOINT
-              value: "http://tempo.foxhunt.svc.cluster.local:4317"
-          volumeMounts:
-            - name: binaries
-              mountPath: /binaries
-              readOnly: true
-            - name: tls-certs
-              mountPath: /app/certs/ml-training-service
-              readOnly: true
-            - name: tmp
-              mountPath: /tmp
-          readinessProbe:
-            tcpSocket:
-              port: 50053
-            initialDelaySeconds: 15
-            periodSeconds: 10
-          livenessProbe:
-            tcpSocket:
-              port: 50053
-            initialDelaySeconds: 30
-            periodSeconds: 15
-            failureThreshold: 5
-          resources:
-            requests:
-              nvidia.com/gpu: "1"
-              cpu: "1"
-              memory: 2Gi
-            limits:
-              nvidia.com/gpu: "1"
-              cpu: "4"
-              memory: 8Gi
-      volumes:
-        - name: binaries
-          emptyDir:
-            sizeLimit: 200Mi
-        - name: tls-certs
-          secret:
-            secretName: ml-training-tls
-        - name: tmp
-          emptyDir:
-            sizeLimit: 50Mi

infra/k8s/gpu-overlays/trading-service-gpu.yaml

@@ -1,156 +0,0 @@
-# GPU-enabled overlay for trading-service
-# Apply manually: kubectl apply -f infra/k8s/gpu-overlays/trading-service-gpu.yaml
-# Revert to CPU: kubectl apply -f infra/k8s/services/trading-service.yaml
-#
-# Binary fetched from MinIO at pod startup — works on any node pool.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: trading-service
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: trading-service
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  strategy:
-    type: RollingUpdate
-    rollingUpdate:
-      maxSurge: 0
-      maxUnavailable: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: trading-service
-  template:
-    metadata:
-      annotations:
-        prometheus.io/scrape: "true"
-        prometheus.io/port: "9092"
-        prometheus.io/path: "/metrics"
-      labels:
-        app.kubernetes.io/name: trading-service
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      securityContext:
-        runAsNonRoot: true
-        runAsUser: 1000
-        runAsGroup: 1000
-        fsGroup: 1000
-        seccompProfile:
-          type: RuntimeDefault
-      nodeSelector:
-        k8s.scaleway.com/pool-name: l40s
-      tolerations:
-        - key: nvidia.com/gpu
-          operator: Exists
-          effect: NoSchedule
-      imagePullSecrets:
-        - name: gitlab-registry
-      initContainers:
-        - name: fetch-binary
-          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              rclone copyto \
-                ":s3:foxhunt-binaries/services/trading-service" \
-                "/binaries/trading-service" \
-                --s3-provider=Minio \
-                --s3-endpoint=http://minio.foxhunt.svc.cluster.local:9000 \
-                --s3-access-key-id="${MINIO_ACCESS_KEY}" \
-                --s3-secret-access-key="${MINIO_SECRET_KEY}" \
-                --s3-no-check-bucket
-              chmod +x /binaries/trading-service
-              echo "Fetched trading-service ($(stat -c%s /binaries/trading-service) bytes)"
-          env:
-            - name: MINIO_ACCESS_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: access-key
-            - name: MINIO_SECRET_KEY
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: secret-key
-          volumeMounts:
-            - name: binaries
-              mountPath: /binaries
-          resources:
-            requests:
-              cpu: 100m
-              memory: 64Mi
-            limits:
-              cpu: 500m
-              memory: 128Mi
-      containers:
-        - name: trading-service
-          image: gitlab-registry.foxhunt.svc.cluster.local:5000/root/foxhunt/foxhunt-runtime:latest
-          securityContext:
-            allowPrivilegeEscalation: false
-            readOnlyRootFilesystem: true
-            capabilities:
-              drop: ["ALL"]
-          command: ["/binaries/trading-service"]
-          ports:
-            - containerPort: 50051
-              name: grpc
-            - containerPort: 9092
-              name: metrics
-          env:
-            - name: DATABASE_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-            - name: DATABASE_URL
-              value: "postgresql://foxhunt:$(DATABASE_PASSWORD)@postgres:5432/foxhunt"
-            - name: REDIS_URL
-              value: "redis://redis:6379"
-            - name: JWT_SECRET
-              valueFrom:
-                secretKeyRef:
-                  name: jwt-secret
-                  key: secret
-            - name: JWT_ISSUER
-              value: foxhunt-api
-            - name: JWT_AUDIENCE
-              value: foxhunt-services
-            - name: QUESTDB_ILP_HOST
-              value: "questdb:9009"
-            - name: GRPC_PORT
-              value: "50051"
-            - name: ENABLE_GPU_INFERENCE
-              value: "true"
-            - name: RUST_LOG
-              value: info
-          volumeMounts:
-            - name: binaries
-              mountPath: /binaries
-              readOnly: true
-            - name: tmp
-              mountPath: /tmp
-          readinessProbe:
-            exec:
-              command:
-                - grpc_health_probe
-                - -addr=localhost:50051
-            initialDelaySeconds: 10
-            periodSeconds: 10
-          resources:
-            requests:
-              nvidia.com/gpu: "1"
-              cpu: "1"
-              memory: 2Gi
-            limits:
-              nvidia.com/gpu: "1"
-              cpu: "4"
-              memory: 8Gi
-      volumes:
-        - name: binaries
-          emptyDir:
-            sizeLimit: 200Mi
-        - name: tmp
-          emptyDir:
-            sizeLimit: 50Mi

infra/k8s/jobs/migrate.yaml

@@ -1,34 +0,0 @@
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: foxhunt-migrate
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/part-of: foxhunt
-    app.kubernetes.io/component: migration
-spec:
-  backoffLimit: 3
-  template:
-    spec:
-      restartPolicy: Never
-      imagePullSecrets:
-        - name: gitlab-registry
-      containers:
-        - name: migrate
-          image: rg.fr-par.scw.cloud/foxhunt/api:latest
-          command: ["sqlx", "migrate", "run"]
-          env:
-            - name: DB_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-            - name: DATABASE_URL
-              value: "postgresql://foxhunt:$(DB_PASSWORD)@postgres:5432/foxhunt"
-          resources:
-            requests:
-              memory: "64Mi"
-              cpu: "50m"
-            limits:
-              memory: "256Mi"
-              cpu: "200m"

infra/k8s/mattermost/mattermost.yaml

@@ -1,121 +0,0 @@
-# Mattermost Team Edition — self-hosted Slack alternative for foxhunt.
-# Runs on platform nodes. PostgreSQL is the shared foxhunt instance.
-# Access: https://chat.fxhnt.ai (Tailscale only)
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: mattermost-data
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: mattermost
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes: [ReadWriteOnce]
-  storageClassName: scw-bssd
-  resources:
-    requests:
-      storage: 5Gi
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: mattermost
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: mattermost
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: mattermost
-  strategy:
-    type: Recreate  # RWO PVC
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: mattermost
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-      containers:
-        - name: mattermost
-          image: mattermost/mattermost-team-edition:10.5
-          ports:
-            - containerPort: 8065
-              name: http
-          env:
-            - name: DB_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: db-credentials
-                  key: password
-            - name: MM_SQLSETTINGS_DRIVERNAME
-              value: postgres
-            - name: MM_SQLSETTINGS_DATASOURCE
-              value: "postgres://foxhunt:$(DB_PASSWORD)@postgres.foxhunt.svc.cluster.local:5432/mattermost?sslmode=disable"
-            - name: MM_SERVICESETTINGS_SITEURL
-              value: "https://chat.fxhnt.ai"
-            - name: MM_SERVICESETTINGS_LISTENADDRESS
-              value: ":8065"
-            - name: MM_SERVICESETTINGS_ENABLEINCOMINGWEBHOOKS
-              value: "true"
-            - name: MM_SERVICESETTINGS_ENABLEPOSTUSERNAMEOVERRIDE
-              value: "true"
-            - name: MM_SERVICESETTINGS_ENABLEPOSTICONOVERRIDE
-              value: "true"
-            - name: MM_LOGSETTINGS_ENABLECONSOLE
-              value: "true"
-            - name: MM_LOGSETTINGS_CONSOLELEVEL
-              value: INFO
-            - name: MM_LOGSETTINGS_ENABLEFILE
-              value: "false"
-            - name: MM_FILESETTINGS_DRIVERNAME
-              value: local
-            - name: MM_FILESETTINGS_DIRECTORY
-              value: /mattermost/data
-            # Email via Stalwart — uncomment after Stalwart is deployed:
-            # - name: MM_EMAILSETTINGS_SMTPSERVER
-            #   value: stalwart.foxhunt.svc.cluster.local
-            # - name: MM_EMAILSETTINGS_SMTPPORT
-            #   value: "25"
-            # - name: MM_EMAILSETTINGS_SENDEMAILNOTIFICATIONS
-            #   value: "true"
-          volumeMounts:
-            - name: mattermost-data
-              mountPath: /mattermost/data
-          readinessProbe:
-            httpGet:
-              path: /api/v4/system/ping
-              port: 8065
-            initialDelaySeconds: 15
-            periodSeconds: 10
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              cpu: "1"
-              memory: 1Gi
-      volumes:
-        - name: mattermost-data
-          persistentVolumeClaim:
-            claimName: mattermost-data
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: mattermost
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: mattermost
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  selector:
-    app.kubernetes.io/name: mattermost
-  ports:
-    - port: 8065
-      targetPort: 8065
-      name: http

infra/k8s/minio/kustomization.yaml

@@ -1,4 +0,0 @@
-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-resources:
-  - minio.yaml

infra/k8s/minio/minio.yaml

@@ -1,188 +0,0 @@
-# MinIO — in-cluster S3 for compiled binaries and trained models
-# Replaces Scaleway public S3 to keep proprietary code/IP in-cluster
-# Plain HTTP only — internal cluster traffic, no TLS needed
-apiVersion: v1
-kind: Secret
-metadata:
-  name: minio-credentials
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: minio
-    app.kubernetes.io/part-of: foxhunt
-type: Opaque
-stringData:
-  root-user: foxhunt-minio
-  root-password: CHANGE_ME_BEFORE_DEPLOY
-  # rclone-compatible keys (same values, different key names for initContainers)
-  access-key: foxhunt-minio
-  secret-key: CHANGE_ME_BEFORE_DEPLOY
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: minio-data-new
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: minio
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  accessModes:
-    - ReadWriteOnce
-  storageClassName: scw-bssd
-  resources:
-    requests:
-      storage: 150Gi
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: minio
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: minio
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate  # single-node MinIO — no rolling update
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: minio
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: minio
-        app.kubernetes.io/part-of: foxhunt
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-      containers:
-        - name: minio
-          image: minio/minio:latest
-          args: ["server", "/data", "--console-address", ":9001"]
-          ports:
-            - containerPort: 9000
-              name: api
-            - containerPort: 9001
-              name: console
-          env:
-            - name: MINIO_ROOT_USER
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: root-user
-            - name: MINIO_ROOT_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: root-password
-          volumeMounts:
-            - name: data
-              mountPath: /data
-          readinessProbe:
-            httpGet:
-              path: /minio/health/live
-              port: 9000
-            initialDelaySeconds: 5
-            periodSeconds: 10
-          livenessProbe:
-            httpGet:
-              path: /minio/health/live
-              port: 9000
-            initialDelaySeconds: 10
-            periodSeconds: 15
-            failureThreshold: 5
-          resources:
-            requests:
-              cpu: 200m
-              memory: 512Mi
-            limits:
-              cpu: "1"
-              memory: 2Gi
-      volumes:
-        - name: data
-          persistentVolumeClaim:
-            claimName: minio-data-new
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: minio
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: minio
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  selector:
-    app.kubernetes.io/name: minio
-  ports:
-    - port: 9000
-      targetPort: 9000
-      name: api
-    - port: 9001
-      targetPort: 9001
-      name: console
----
-# Create buckets on first deploy
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: minio-init-buckets
-  namespace: foxhunt
-  labels:
-    app.kubernetes.io/name: minio
-    app.kubernetes.io/part-of: foxhunt
-spec:
-  backoffLimit: 5
-  ttlSecondsAfterFinished: 300
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: minio-init
-    spec:
-      nodeSelector:
-        k8s.scaleway.com/pool-name: platform
-      restartPolicy: OnFailure
-      containers:
-        - name: mc
-          image: minio/mc:latest
-          command: ["/bin/sh", "-c"]
-          args:
-            - |
-              set -e
-              echo "Waiting for MinIO to be ready..."
-              until mc alias set foxhunt http://minio.foxhunt.svc.cluster.local:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" 2>/dev/null; do
-                echo "MinIO not ready, retrying in 3s..."
-                sleep 3
-              done
-              echo "Creating buckets..."
-              mc mb --ignore-existing foxhunt/foxhunt-binaries
-              mc mb --ignore-existing foxhunt/foxhunt-models
-              mc mb --ignore-existing foxhunt/foxhunt-training-results
-              mc mb --ignore-existing foxhunt/argo-logs
-              mc mb --ignore-existing foxhunt/foxhunt-sccache
-              mc mb --ignore-existing foxhunt/foxhunt-training-data
-              # Plan 5 Task 3 (A.4.1): nsys profile artefacts bucket.
-              # Stores .nsys-rep files keyed by commit SHA under
-              # profiles/<short-sha>/profile-seed*-fold*-*.nsys-rep.
-              mc mb --ignore-existing foxhunt/foxhunt-training-artifacts
-              echo "Buckets created:"
-              mc ls foxhunt/
-          env:
-            - name: MINIO_ROOT_USER
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: root-user
-            - name: MINIO_ROOT_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: minio-credentials
-                  key: root-password
-          resources:
-            requests:
-              cpu: 50m
-              memory: 32Mi
-            limits:
-              cpu: 200m
-              memory: 64Mi

infra/k8s/monitoring/APPLY-ORDER.md

@@ -1,104 +0,0 @@
-# Cockpit + NAT Gateway — Apply Order
-
-## Prerequisites
-- `scw` CLI authenticated
-- `terragrunt` installed
-- `kubectl` access to foxhunt cluster
-- `helm` installed
-
-## Phase 1: Cockpit (metrics backend)
-
-1. Apply Cockpit Terraform module:
-   ```bash
-   cd infra/live/production/cockpit
-   terragrunt apply
-   ```
-
-2. Note the outputs:
-   ```bash
-   terragrunt output metrics_push_url
-   terragrunt output -raw push_token
-   terragrunt output grafana_url
-   terragrunt output -raw grafana_password
-   ```
-
-3. Create the K8s secret for the push token:
-   ```bash
-   kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
-   kubectl -n monitoring create secret generic cockpit-push-token \
-     --from-literal=token=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
-   ```
-
-4. Update placeholders in `infra/k8s/monitoring/alloy-values.yaml`:
-   - Replace `COCKPIT_METRICS_PUSH_URL` with the `metrics_push_url` output
-
-5. Deploy DCGM Exporter:
-   ```bash
-   kubectl apply -f infra/k8s/monitoring/dcgm-exporter.yaml
-   ```
-
-6. Deploy Grafana Alloy (k8s-monitoring):
-   ```bash
-   helm repo add grafana https://grafana.github.io/helm-charts
-   helm repo update
-   helm install k8s-monitoring grafana/k8s-monitoring \
-     --namespace monitoring \
-     -f infra/k8s/monitoring/alloy-values.yaml
-   ```
-
-7. Verify metrics flow:
-   - Open Cockpit Grafana (URL from step 2)
-   - Login with foxhunt-admin / (password from step 2)
-   - Check "Kubernetes / Compute Resources / Cluster" dashboard
-
-## Phase 2: GitLab Prometheus remote_write
-
-1. Update placeholders in `infra/k8s/gitlab/values.yaml`:
-   - Replace `COCKPIT_METRICS_PUSH_URL` and `COCKPIT_PUSH_TOKEN`
-
-2. Upgrade GitLab Helm release:
-   ```bash
-   helm upgrade gitlab gitlab/gitlab \
-     --namespace foxhunt \
-     -f infra/k8s/gitlab/values.yaml
-   ```
-
-## Phase 3: NAT Gateway
-
-1. Apply Public Gateway module:
-   ```bash
-   cd infra/live/production/public-gateway
-   terragrunt apply
-   ```
-   This creates the gateway and attaches it to the private network.
-   Existing nodes still have public IPs — nothing breaks yet.
-
-2. Apply Kapsule pool changes (disables public IPs):
-   ```bash
-   cd infra/live/production/kapsule
-   terragrunt apply
-   ```
-   **WARNING**: This may recreate nodes. For zero-downtime:
-
-3. Zero-downtime node migration for services pool:
-   ```bash
-   # Temporarily allow 2 nodes
-   scw k8s pool update <services-pool-id> max-size=2 size=2
-   # Wait for new node (private IP) to be Ready
-   kubectl get nodes -w
-   # Drain old node
-   kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
-   # Scale back
-   scw k8s pool update <services-pool-id> max-size=1 size=1
-   ```
-
-4. Repeat for gitlab pool (if not autoscaling, same process).
-
-5. Autoscaling pools (ci-training, gpu-dev):
-   - No action needed — next scale-up automatically uses private IP.
-
-6. Verify Tailscale still works:
-   ```bash
-   tailscale status  # should show foxhunt-kapsule and foxhunt-gitlab nodes
-   curl -k https://git.fxhnt.ai  # should still resolve via Tailscale
-   ```

infra/k8s/monitoring/alloy-values.yaml

@@ -1,88 +0,0 @@
-# Grafana k8s-monitoring Helm chart values (v1.x format)
-# Chart: grafana/k8s-monitoring --version 1.6.50
-# Deploys: Alloy (DaemonSet) + node-exporter + kube-state-metrics
-#
-# Install:
-#   helm repo add grafana https://grafana.github.io/helm-charts
-#   PUSH_TOKEN=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
-#   helm install k8s-monitoring grafana/k8s-monitoring \
-#     --namespace monitoring --version 1.6.50 \
-#     -f infra/k8s/monitoring/alloy-values.yaml \
-#     --set "externalServices.prometheus.extraHeaders.X-Token=${PUSH_TOKEN}"
-
-cluster:
-  name: foxhunt
-
-externalServices:
-  prometheus:
-    host: "https://d8cd4c5d-6d3b-4258-9f92-2ad0eaa36ba0.metrics.cockpit.fr-par.scw.cloud"
-    writeEndpoint: "/api/v1/push"
-    authMode: none
-    # X-Token header passed via --set to keep secret out of git
-  # Loki host required by chart validation even with logs disabled
-  loki:
-    host: "https://noop.localhost"
-    authMode: none
-
-metrics:
-  enabled: true
-  cost:
-    enabled: false
-  node-exporter:
-    enabled: true
-  kube-state-metrics:
-    enabled: true
-  kubelet:
-    enabled: true
-  cadvisor:
-    enabled: true
-  apiserver:
-    enabled: true
-
-# Alloy DaemonSet — runs on every node
-alloy-metrics:
-  alloy:
-    resources:
-      requests:
-        cpu: 50m
-        memory: 128Mi
-      limits:
-        cpu: 200m
-        memory: 256Mi
-
-# node-exporter — host metrics (CPU, memory, disk, network)
-prometheus-node-exporter:
-  resources:
-    requests:
-      cpu: 25m
-      memory: 32Mi
-    limits:
-      cpu: 100m
-      memory: 64Mi
-
-# kube-state-metrics — K8s object state
-kube-state-metrics:
-  resources:
-    requests:
-      cpu: 25m
-      memory: 64Mi
-    limits:
-      cpu: 100m
-      memory: 128Mi
-  # Scheduled on gitlab node (GP1-XS 16GB) — services (DEV1-M 4GB) too small
-
-# Disable components we don't need
-logs:
-  enabled: false
-  cluster_events:
-    enabled: false
-traces:
-  enabled: false
-opencost:
-  enabled: false
-
-# Disable log/event collectors
-alloy-logs:
-  enabled: false
-alloy-events:
-  enabled: false

infra/k8s/monitoring/dashboards/foxhunt-ci-logs.json

@@ -1,809 +0,0 @@
-{
-  "description": "Loki-powered CI pipeline log viewer \u2014 test gate, clippy, builds, errors",
-  "editable": true,
-  "graphTooltip": 2,
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "title": "Pipeline Overview",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "red",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0",
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "orientation": "auto",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=\"ERROR\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "yellow",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0",
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 4,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "orientation": "auto",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=\"WARN\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Warnings",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "blue",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 8,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "orientation": "auto",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Total Log Lines",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisLabel": "lines/interval",
-            "drawStyle": "bars",
-            "fillOpacity": 80,
-            "lineWidth": 0,
-            "pointSize": 5,
-            "showPoints": "never",
-            "stacking": {
-              "mode": "normal"
-            }
-          }
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "ERROR"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "red",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          },
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "WARN"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "yellow",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          },
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "INFO"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "green",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          },
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "DEBUG"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "blue",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 6,
-        "w": 12,
-        "x": 12,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum by (level) (count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__interval]))",
-          "legendFormat": "{{level}}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Volume by Level",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "drawStyle": "bars",
-            "fillOpacity": 60,
-            "lineWidth": 0,
-            "showPoints": "never",
-            "stacking": {
-              "mode": "normal"
-            }
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 12,
-        "x": 0,
-        "y": 5
-      },
-      "id": 6,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum by (container) (count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__interval]))",
-          "legendFormat": "{{container}}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Volume by Container",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 9
-      },
-      "id": 10,
-      "title": "Full Pipeline Log Stream",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 14,
-        "w": 24,
-        "x": 0,
-        "y": 10
-      },
-      "id": 11,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "{namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=~\"$level\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg ${search:pipe}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Pipeline Logs \u2014 $pipeline",
-      "type": "logs"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 24
-      },
-      "id": 20,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 25
-          },
-          "id": 21,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": false,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"ci-pipeline-.*\", container=\"main\"} |~ \"(clippy|cargo test|warning\\\\[|error\\\\[|FAILED|test result|running \\\\d+ test|failures:)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Clippy + Test Output",
-          "type": "logs"
-        }
-      ],
-      "title": "Test Gate (clippy + cargo test)",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 25
-      },
-      "id": 30,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 8,
-            "w": 24,
-            "x": 0,
-            "y": 26
-          },
-          "id": 31,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": false,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"ci-pipeline-.*\", container=\"redis\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Redis Sidecar Logs",
-          "type": "logs"
-        }
-      ],
-      "title": "Redis Sidecar",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 26
-      },
-      "id": 40,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 27
-          },
-          "id": 41,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} |~ \"(?i)(error|failed|failure|panic|fatal|SIGABRT|SIGSEGV|thread .* panicked)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "All Errors & Failures",
-          "type": "logs"
-        }
-      ],
-      "title": "Errors & Failures",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 27
-      },
-      "id": 50,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 10,
-            "w": 12,
-            "x": 0,
-            "y": 28
-          },
-          "id": 51,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": false,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"(ci-pipeline|compile-and-deploy|compile-and-train)-.*\", container=\"main\"} |~ \"(Compiling|Finished|warning|error|Downloading|Building|cargo build|rclone)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Compile Logs",
-          "type": "logs"
-        },
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 10,
-            "w": 12,
-            "x": 12,
-            "y": 28
-          },
-          "id": 52,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": false,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"(ci-pipeline|compile-and-deploy)-.*\", container=\"main\"} |~ \"(kubectl|apply|rollout|deploy|configured|unchanged|created|image|kaniko)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Deploy & Argo Template Logs",
-          "type": "logs"
-        }
-      ],
-      "title": "Build & Deploy",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 28
-      },
-      "id": 60,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 29
-          },
-          "id": 61,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"gpu-test-.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "GPU Test Logs (DQN/PPO/TFT + Supervised)",
-          "type": "logs"
-        }
-      ],
-      "title": "GPU Test Pipeline",
-      "type": "row"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "foxhunt",
-    "ci",
-    "loki",
-    "logs"
-  ],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": true,
-          "text": "ci-pipeline",
-          "value": "ci-pipeline"
-        },
-        "label": "Pipeline",
-        "name": "pipeline",
-        "options": [],
-        "query": "ci-pipeline,compile-and-deploy,compile-and-train,gpu-test,gpu-diag,gpu-walltime,build-ci-image,build-training-runtime,evaluate",
-        "type": "custom"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "selected": true,
-          "text": "All",
-          "value": "$__all"
-        },
-        "includeAll": true,
-        "label": "Level",
-        "multi": true,
-        "name": "level",
-        "options": [
-          {
-            "selected": true,
-            "text": "All",
-            "value": "$__all"
-          },
-          {
-            "selected": false,
-            "text": "ERROR",
-            "value": "ERROR"
-          },
-          {
-            "selected": false,
-            "text": "WARN",
-            "value": "WARN"
-          },
-          {
-            "selected": false,
-            "text": "INFO",
-            "value": "INFO"
-          },
-          {
-            "selected": false,
-            "text": "DEBUG",
-            "value": "DEBUG"
-          },
-          {
-            "selected": false,
-            "text": "TRACE",
-            "value": "TRACE"
-          }
-        ],
-        "query": "ERROR,WARN,INFO,DEBUG,TRACE",
-        "type": "custom"
-      },
-      {
-        "current": {
-          "selected": false,
-          "text": "",
-          "value": ""
-        },
-        "label": "Search",
-        "name": "search",
-        "type": "textbox"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-6h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "browser",
-  "title": "CI Pipeline Logs",
-  "uid": "foxhunt-ci-logs",
-  "__inputs": [
-    {
-      "name": "DS_PROMETHEUS",
-      "type": "datasource",
-      "pluginId": "prometheus"
-    },
-    {
-      "name": "DS_LOKI",
-      "type": "datasource",
-      "pluginId": "loki"
-    },
-    {
-      "name": "DS_TEMPO",
-      "type": "datasource",
-      "pluginId": "${DS_TEMPO}"
-    }
-  ]
-}

infra/k8s/monitoring/dashboards/foxhunt-training-logs.json

@@ -1,835 +0,0 @@
-{
-  "description": "Loki-powered training log viewer \u2014 hyperopt trials, epoch progress, GPU status, errors",
-  "editable": true,
-  "graphTooltip": 2,
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "title": "Training Overview",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "red",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0",
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\", level=\"ERROR\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "yellow",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0",
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 4,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\", level=\"WARN\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Warnings",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "fixedColor": "blue",
-            "mode": "fixed"
-          },
-          "mappings": [],
-          "noValue": "0"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 4,
-        "x": 8,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "colorMode": "background",
-        "graphMode": "none",
-        "justifyMode": "center",
-        "reduceOptions": {
-          "calcs": [
-            "sum"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "textMode": "auto"
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__range]))",
-          "queryType": "instant",
-          "refId": "A"
-        }
-      ],
-      "title": "Total Log Lines",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisLabel": "lines/interval",
-            "drawStyle": "bars",
-            "fillOpacity": 80,
-            "lineWidth": 0,
-            "showPoints": "never",
-            "stacking": {
-              "mode": "normal"
-            }
-          }
-        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "ERROR"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "red",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          },
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "WARN"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "yellow",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          },
-          {
-            "matcher": {
-              "id": "byName",
-              "options": "INFO"
-            },
-            "properties": [
-              {
-                "id": "color",
-                "value": {
-                  "fixedColor": "green",
-                  "mode": "fixed"
-                }
-              }
-            ]
-          }
-        ]
-      },
-      "gridPos": {
-        "h": 6,
-        "w": 12,
-        "x": 12,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum by (level) (count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__interval]))",
-          "legendFormat": "{{level}}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Volume by Level",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "drawStyle": "bars",
-            "fillOpacity": 60,
-            "lineWidth": 0,
-            "showPoints": "never",
-            "stacking": {
-              "mode": "normal"
-            }
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 12,
-        "x": 0,
-        "y": 5
-      },
-      "id": 6,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "right"
-        },
-        "tooltip": {
-          "mode": "multi"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "sum by (pod) (count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__interval]))",
-          "legendFormat": "{{pod}}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Volume by Pod",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 9
-      },
-      "id": 10,
-      "title": "Full Training Log Stream",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 14,
-        "w": 24,
-        "x": 0,
-        "y": 10
-      },
-      "id": 11,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "editorMode": "code",
-          "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\", level=~\"$level\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg ${search:pipe}",
-          "queryType": "range",
-          "refId": "A"
-        }
-      ],
-      "title": "Training Logs \u2014 $job",
-      "type": "logs"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 24
-      },
-      "id": 20,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 25
-          },
-          "id": 21,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(epoch|loss|reward|sharpe|accuracy|val_loss|train_loss|best_trial|trial|fold|window|learning_rate|lr=|MaxDD|win.?rate|drawdown)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Epoch / Loss / Reward / Sharpe",
-          "type": "logs"
-        }
-      ],
-      "title": "Epoch Progress & Loss",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 25
-      },
-      "id": 30,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 26
-          },
-          "id": 31,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"(hyperopt|train)-.*\"} |~ \"(?i)(trial|best|cost|param|learning_rate|hidden_dim|batch_size|buffer_size|gamma|epsilon|clip|PSO|iteration|objective|optimal)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Hyperopt Trial Results",
-          "type": "logs"
-        }
-      ],
-      "title": "Hyperopt Trials",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 26
-      },
-      "id": 40,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 27
-          },
-          "id": 41,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(GPU|CUDA|VRAM|cuda|gpu|vram|OOM|out.of.memory|nvrtc|tensor.core|BF16|bf16|SM|occupancy|kernel|device|nvidia)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "GPU / CUDA / VRAM Logs",
-          "type": "logs"
-        }
-      ],
-      "title": "GPU / CUDA / VRAM",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 27
-      },
-      "id": 50,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 10,
-            "w": 24,
-            "x": 0,
-            "y": 28
-          },
-          "id": 51,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(walk.?forward|fold|backtest|window|train_start|train_end|val_start|val_end|test_start|test_end|embargo|purge|bars|features)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Walk-Forward / Fold / Backtest",
-          "type": "logs"
-        }
-      ],
-      "title": "Walk-Forward Windows",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 28
-      },
-      "id": 60,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 12,
-            "w": 24,
-            "x": 0,
-            "y": 29
-          },
-          "id": 61,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(error|failed|failure|panic|fatal|SIGABRT|SIGSEGV|thread .* panicked|OOM|NaN|inf|diverge)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "All Training Errors & Failures",
-          "type": "logs"
-        }
-      ],
-      "title": "Errors & Failures",
-      "type": "row"
-    },
-    {
-      "collapsed": true,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 29
-      },
-      "id": 70,
-      "panels": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "gridPos": {
-            "h": 10,
-            "w": 24,
-            "x": 0,
-            "y": 30
-          },
-          "id": 71,
-          "options": {
-            "dedupStrategy": "none",
-            "enableLogDetails": true,
-            "showLabels": true,
-            "showTime": true,
-            "sortOrder": "Descending",
-            "wrapLogMessage": true
-          },
-          "targets": [
-            {
-              "datasource": {
-                "type": "loki",
-                "uid": "${DS_LOKI}"
-              },
-              "editorMode": "code",
-              "expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(loading|loaded|dbn|ofi|features|bars|MBP.?10|OHLCV|symbol|normalize|norm_stats|state_dim|rclone|S3|download)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-              "queryType": "range",
-              "refId": "A"
-            }
-          ],
-          "title": "Data Loading / OFI / DBN / Features",
-          "type": "logs"
-        }
-      ],
-      "title": "Data Loading & OFI",
-      "type": "row"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "foxhunt",
-    "training",
-    "loki",
-    "logs",
-    "gpu"
-  ],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": true,
-          "text": "All Training",
-          "value": "(train|hyperopt|compile-and-train)"
-        },
-        "label": "Job Type",
-        "name": "job",
-        "options": [
-          {
-            "selected": true,
-            "text": "All Training",
-            "value": "(train|hyperopt|compile-and-train)"
-          },
-          {
-            "selected": false,
-            "text": "DQN",
-            "value": "train-dqn"
-          },
-          {
-            "selected": false,
-            "text": "PPO",
-            "value": "train-ppo"
-          },
-          {
-            "selected": false,
-            "text": "Supervised",
-            "value": "train-supervised"
-          },
-          {
-            "selected": false,
-            "text": "Hyperopt",
-            "value": "hyperopt"
-          },
-          {
-            "selected": false,
-            "text": "Compile+Train",
-            "value": "compile-and-train"
-          }
-        ],
-        "query": "All Training : (train|hyperopt|compile-and-train), DQN : train-dqn, PPO : train-ppo, Supervised : train-supervised, Hyperopt : hyperopt, Compile+Train : compile-and-train",
-        "type": "custom"
-      },
-      {
-        "allValue": ".*",
-        "current": {
-          "selected": true,
-          "text": "All",
-          "value": "$__all"
-        },
-        "includeAll": true,
-        "label": "Level",
-        "multi": true,
-        "name": "level",
-        "options": [
-          {
-            "selected": true,
-            "text": "All",
-            "value": "$__all"
-          },
-          {
-            "selected": false,
-            "text": "ERROR",
-            "value": "ERROR"
-          },
-          {
-            "selected": false,
-            "text": "WARN",
-            "value": "WARN"
-          },
-          {
-            "selected": false,
-            "text": "INFO",
-            "value": "INFO"
-          },
-          {
-            "selected": false,
-            "text": "DEBUG",
-            "value": "DEBUG"
-          },
-          {
-            "selected": false,
-            "text": "TRACE",
-            "value": "TRACE"
-          }
-        ],
-        "query": "ERROR,WARN,INFO,DEBUG,TRACE",
-        "type": "custom"
-      },
-      {
-        "current": {
-          "selected": false,
-          "text": "",
-          "value": ""
-        },
-        "label": "Search",
-        "name": "search",
-        "type": "textbox"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-12h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "browser",
-  "title": "Training Pipeline Logs",
-  "uid": "foxhunt-training-logs",
-  "__inputs": [
-    {
-      "name": "DS_PROMETHEUS",
-      "type": "datasource",
-      "pluginId": "prometheus"
-    },
-    {
-      "name": "DS_LOKI",
-      "type": "datasource",
-      "pluginId": "loki"
-    },
-    {
-      "name": "DS_TEMPO",
-      "type": "datasource",
-      "pluginId": "${DS_TEMPO}"
-    }
-  ]
-}

infra/k8s/monitoring/dashboards/import.sh

@@ -1,212 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-# Import all Grafana dashboards via API (Postgres-backed, no ConfigMaps).
-#
-# Dashboards use ${DS_PROMETHEUS}, ${DS_LOKI}, ${DS_TEMPO} variables.
-# This script auto-detects datasource UIDs, resolves variables, and
-# imports via the Grafana API. Dashboards are stored in Postgres.
-#
-# Required environment variables:
-#   GRAFANA_PASS - Password for admin user
-#
-# Optional overrides:
-#   GRAFANA_URL       - Grafana base URL (default: https://grafana.fxhnt.ai)
-#   GRAFANA_USER      - Grafana user (default: admin)
-#   GRAFANA_PROM_UID  - Override Prometheus datasource UID
-#   GRAFANA_LOKI_UID  - Override Loki datasource UID
-#   GRAFANA_TEMPO_UID - Override Tempo datasource UID
-#
-# Usage:
-#   GRAFANA_PASS="xxx" ./import.sh
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-GRAFANA_USER="${GRAFANA_USER:-admin}"
-GRAFANA_URL="${GRAFANA_URL:-https://grafana.fxhnt.ai}"
-GRAFANA_URL="${GRAFANA_URL%/}"
-
-if [[ -z "${GRAFANA_PASS:-}" ]]; then
-  echo "ERROR: GRAFANA_PASS is not set" >&2
-  exit 1
-fi
-
-# --- Datasource auto-detection ---
-
-detect_datasource_uid() {
-  local ds_type="$1"
-  curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" "${GRAFANA_URL}/api/datasources" 2>/dev/null | \
-    python3 -c "
-import sys, json
-for ds in json.load(sys.stdin):
-    if ds.get('type') == '${ds_type}':
-        print(ds['uid'])
-        break
-" 2>/dev/null || true
-}
-
-echo "Detecting datasource UIDs..."
-PROM_UID="${GRAFANA_PROM_UID:-$(detect_datasource_uid prometheus)}"
-LOKI_UID="${GRAFANA_LOKI_UID:-$(detect_datasource_uid loki)}"
-TEMPO_UID="${GRAFANA_TEMPO_UID:-$(detect_datasource_uid tempo)}"
-
-echo "  Prometheus: ${PROM_UID:-NOT FOUND}"
-echo "  Loki:       ${LOKI_UID:-NOT FOUND}"
-echo "  Tempo:      ${TEMPO_UID:-NOT FOUND}"
-
-if [[ -z "$PROM_UID" ]]; then
-  echo "ERROR: Could not detect Prometheus datasource. Set GRAFANA_PROM_UID." >&2
-  exit 1
-fi
-
-# --- Resolve datasource variables ---
-
-resolve_dashboard() {
-  local file="$1"
-  python3 -c "
-import sys, json
-
-with open('${file}') as f:
-    d = json.load(f)
-
-d.pop('__inputs', None)
-d.pop('__requires', None)
-
-raw = json.dumps(d)
-raw = raw.replace('\${DS_PROMETHEUS}', '${PROM_UID}')
-raw = raw.replace('\${DS_LOKI}', '${LOKI_UID}')
-raw = raw.replace('\${DS_TEMPO}', '${TEMPO_UID}')
-
-print(raw)
-"
-}
-
-# --- Ensure folders exist ---
-
-ensure_folder() {
-  local title="$1"
-  local uid
-  uid=$(curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" "${GRAFANA_URL}/api/folders" 2>/dev/null | \
-    python3 -c "
-import sys, json
-for f in json.load(sys.stdin):
-    if f['title'] == '${title}':
-        print(f['uid'])
-        break
-" 2>/dev/null || true)
-
-  if [[ -z "$uid" ]]; then
-    uid=$(curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" \
-      -H "Content-Type: application/json" \
-      -d "{\"title\": \"${title}\"}" \
-      "${GRAFANA_URL}/api/folders" 2>/dev/null | \
-      python3 -c "import sys,json; print(json.load(sys.stdin)['uid'])" 2>/dev/null)
-    echo "  Created folder: ${title} (${uid})"
-  fi
-  echo "$uid"
-}
-
-echo ""
-echo "Ensuring folders..."
-FOLDER_CI=$(ensure_folder "CI")
-FOLDER_INFRA=$(ensure_folder "Infrastructure")
-FOLDER_OPS=$(ensure_folder "Operations")
-FOLDER_TRADING=$(ensure_folder "Trading")
-FOLDER_TRAINING=$(ensure_folder "Training")
-FOLDER_SERVICES=$(ensure_folder "Services")
-FOLDER_PLATFORM=$(ensure_folder "Platform")
-
-# --- Import a single dashboard to a folder ---
-
-import_dashboard() {
-  local file="$1"
-  local folder_uid="$2"
-  local name
-  name="$(basename "$file" .json)"
-
-  local resolved
-  resolved=$(resolve_dashboard "$file")
-
-  local response
-  response=$(curl -s -w "\n%{http_code}" \
-    -u "${GRAFANA_USER}:${GRAFANA_PASS}" \
-    -H "Content-Type: application/json" \
-    -d "{\"dashboard\": ${resolved}, \"folderUid\": \"${folder_uid}\", \"overwrite\": true}" \
-    "${GRAFANA_URL}/api/dashboards/db")
-
-  local http_code
-  http_code=$(echo "$response" | tail -1)
-
-  if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then
-    echo "  OK   ${name}"
-  else
-    local body
-    body=$(echo "$response" | sed '$d')
-    echo "  FAIL ${name} (${http_code}): ${body}" >&2
-    return 1
-  fi
-}
-
-# --- Dashboard → Folder mapping ---
-
-declare -A DASHBOARD_FOLDERS=(
-  # Original dashboards
-  [foxhunt-cockpit]="${FOLDER_OPS}"
-  [foxhunt-trading]="${FOLDER_TRADING}"
-  [foxhunt-training]="${FOLDER_TRAINING}"
-  [foxhunt-infrastructure]="${FOLDER_INFRA}"
-  [foxhunt-observability]="${FOLDER_INFRA}"
-  [foxhunt-ci-logs]="${FOLDER_CI}"
-  [foxhunt-training-logs]="${FOLDER_TRAINING}"
-  [foxhunt-training-deep-dive]="${FOLDER_TRAINING}"
-  # Application services
-  [svc-api-gateway]="${FOLDER_SERVICES}"
-  [svc-trading-service]="${FOLDER_SERVICES}"
-  [svc-backtesting]="${FOLDER_SERVICES}"
-  [svc-data-acquisition]="${FOLDER_SERVICES}"
-  [svc-ml-training]="${FOLDER_SERVICES}"
-  [svc-trading-agent]="${FOLDER_SERVICES}"
-  [svc-web-gateway]="${FOLDER_SERVICES}"
-  # Platform services
-  [platform-prometheus]="${FOLDER_PLATFORM}"
-  [platform-grafana]="${FOLDER_PLATFORM}"
-  [platform-loki]="${FOLDER_PLATFORM}"
-  [platform-tempo]="${FOLDER_PLATFORM}"
-  [platform-redis]="${FOLDER_PLATFORM}"
-  [platform-postgres]="${FOLDER_PLATFORM}"
-  [platform-argo]="${FOLDER_PLATFORM}"
-  [platform-questdb]="${FOLDER_PLATFORM}"
-  [platform-costs]="${FOLDER_PLATFORM}"
-  [platform-scaleway]="${FOLDER_PLATFORM}"
-)
-
-# --- Deploy all dashboards ---
-
-echo ""
-echo "Importing dashboards..."
-
-failed=0
-total=0
-
-for dashboard in "${SCRIPT_DIR}"/*.json; do
-  [[ -f "$dashboard" ]] || continue
-  name="$(basename "$dashboard" .json)"
-  folder_uid="${DASHBOARD_FOLDERS[$name]:-}"
-
-  if [[ -z "$folder_uid" ]]; then
-    echo "  SKIP ${name} (no folder mapping)"
-    continue
-  fi
-
-  total=$((total + 1))
-  if ! import_dashboard "$dashboard" "$folder_uid"; then
-    failed=$((failed + 1))
-  fi
-done
-
-echo ""
-if [[ $failed -eq 0 ]]; then
-  echo "All ${total} dashboards deployed successfully."
-else
-  echo "ERROR: ${failed}/${total} dashboard(s) failed." >&2
-  exit 1
-fi

infra/k8s/monitoring/dashboards/platform-grafana.json

@@ -1,802 +0,0 @@
-{
-  "annotations": {
-    "list": [
-      {
-        "builtIn": 1,
-        "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
-        },
-        "enable": true,
-        "hide": true,
-        "iconColor": "rgba(0, 211, 255, 1)",
-        "name": "Annotations & Alerts",
-        "type": "dashboard"
-      }
-    ]
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "links": [],
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "panels": [],
-      "title": "Grafana Health",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "red",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"grafana.*\", phase=\"Running\"}",
-          "instant": true,
-          "legendFormat": "Running",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Pod Status",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 3
-              },
-              {
-                "color": "red",
-                "value": 10
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 3,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"grafana.*\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Restarts",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 6,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[$interval]))",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 9,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"grafana.*\", resource=\"memory\"})",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 12,
-        "y": 1
-      },
-      "id": 6,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Memory (bytes)",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 15,
-        "y": 1
-      },
-      "id": 7,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Cores",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 18,
-        "y": 1
-      },
-      "id": 8,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network RX",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 21,
-        "y": 1
-      },
-      "id": 9,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network TX",
-      "type": "stat"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 5
-      },
-      "id": 10,
-      "panels": [],
-      "title": "Resources",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 6
-      },
-      "id": 11,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[$interval])",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 6
-      },
-      "id": 12,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 6
-      },
-      "id": 13,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[$interval])",
-          "legendFormat": "RX {{pod}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[$interval])",
-          "legendFormat": "TX {{pod}}",
-          "refId": "B"
-        }
-      ],
-      "title": "Network I/O",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 14
-      },
-      "id": 14,
-      "panels": [],
-      "title": "Logs",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 15
-      },
-      "id": 15,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"grafana.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Stream",
-      "type": "logs"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 25
-      },
-      "id": 16,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"grafana.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors & Warnings",
-      "type": "logs"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "platform",
-    "grafana"
-  ],
-  "templating": {
-    "list": [
-      {
-        "auto": false,
-        "current": {
-          "text": "5m",
-          "value": "5m"
-        },
-        "name": "interval",
-        "options": [
-          {
-            "selected": false,
-            "text": "1m",
-            "value": "1m"
-          },
-          {
-            "selected": true,
-            "text": "5m",
-            "value": "5m"
-          },
-          {
-            "selected": false,
-            "text": "15m",
-            "value": "15m"
-          },
-          {
-            "selected": false,
-            "text": "30m",
-            "value": "30m"
-          },
-          {
-            "selected": false,
-            "text": "1h",
-            "value": "1h"
-          }
-        ],
-        "query": "1m,5m,15m,30m,1h",
-        "type": "interval"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "browser",
-  "title": "Grafana",
-  "uid": "plat-grafana"
-}

infra/k8s/monitoring/dashboards/platform-prometheus.json

@@ -1,946 +0,0 @@
-{
-  "annotations": {
-    "list": [
-      {
-        "builtIn": 1,
-        "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
-        },
-        "enable": true,
-        "hide": true,
-        "iconColor": "rgba(0, 211, 255, 1)",
-        "name": "Annotations & Alerts",
-        "type": "dashboard"
-      }
-    ]
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "links": [],
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "panels": [],
-      "title": "Prometheus Health",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "count(up{job!=\"\"} == 1)",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Targets Up",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 3,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "count(up{job!=\"\"} == 0) or vector(0)",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Targets Down",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 6,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "prometheus_tsdb_head_series",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Head Series",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 9,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(prometheus_tsdb_head_samples_appended_total[5m])",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Ingestion Rate",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 12,
-        "y": 1
-      },
-      "id": 6,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "prometheus_tsdb_wal_storage_size_bytes",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "WAL Size",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 15,
-        "y": 1
-      },
-      "id": 7,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "prometheus_tsdb_head_chunks",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Head Chunks",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 18,
-        "y": 1
-      },
-      "id": 8,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "prometheus_tsdb_compactions_total",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Compactions",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 21,
-        "y": 1
-      },
-      "id": 9,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "prometheus_tsdb_storage_blocks_bytes",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Storage Size",
-      "type": "stat"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 5
-      },
-      "id": 10,
-      "panels": [],
-      "title": "Resources",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 6
-      },
-      "id": 11,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"prometheus.*\", container!=\"\"}[$interval])",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 6
-      },
-      "id": 12,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"prometheus.*\", container!=\"\"}",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 6
-      },
-      "id": 13,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"prometheus.*\"}[$interval])",
-          "legendFormat": "RX {{pod}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"prometheus.*\"}[$interval])",
-          "legendFormat": "TX {{pod}}",
-          "refId": "B"
-        }
-      ],
-      "title": "Network I/O",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 14
-      },
-      "id": 14,
-      "panels": [],
-      "title": "Scrape Performance",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "s"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 15
-      },
-      "id": 15,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "scrape_duration_seconds",
-          "legendFormat": "{{job}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Scrape Duration",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 15
-      },
-      "id": 16,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(prometheus_tsdb_head_samples_appended_total[$interval])",
-          "legendFormat": "samples/s",
-          "refId": "A"
-        }
-      ],
-      "title": "Samples Ingested",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 23
-      },
-      "id": 17,
-      "panels": [],
-      "title": "Target Status",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {},
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 24
-      },
-      "id": 18,
-      "options": {
-        "showHeader": true,
-        "sortBy": []
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "up",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Target Health",
-      "type": "table"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 32
-      },
-      "id": 19,
-      "panels": [],
-      "title": "Logs",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 33
-      },
-      "id": 20,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"prometheus.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Stream",
-      "type": "logs"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 43
-      },
-      "id": 21,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"prometheus.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors & Warnings",
-      "type": "logs"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "platform",
-    "prometheus"
-  ],
-  "templating": {
-    "list": [
-      {
-        "auto": false,
-        "current": {
-          "text": "5m",
-          "value": "5m"
-        },
-        "name": "interval",
-        "options": [
-          {
-            "selected": false,
-            "text": "1m",
-            "value": "1m"
-          },
-          {
-            "selected": true,
-            "text": "5m",
-            "value": "5m"
-          },
-          {
-            "selected": false,
-            "text": "15m",
-            "value": "15m"
-          },
-          {
-            "selected": false,
-            "text": "30m",
-            "value": "30m"
-          },
-          {
-            "selected": false,
-            "text": "1h",
-            "value": "1h"
-          }
-        ],
-        "query": "1m,5m,15m,30m,1h",
-        "type": "interval"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "browser",
-  "title": "Prometheus",
-  "uid": "plat-prometheus"
-}

infra/k8s/monitoring/dashboards/platform-questdb.json

@@ -1,802 +0,0 @@
-{
-  "annotations": {
-    "list": [
-      {
-        "builtIn": 1,
-        "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
-        },
-        "enable": true,
-        "hide": true,
-        "iconColor": "rgba(0, 211, 255, 1)",
-        "name": "Annotations & Alerts",
-        "type": "dashboard"
-      }
-    ]
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "links": [],
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "panels": [],
-      "title": "QuestDB Health",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "red",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"questdb.*\", phase=\"Running\"}",
-          "instant": true,
-          "legendFormat": "Running",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Pod Status",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 3
-              },
-              {
-                "color": "red",
-                "value": 10
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 3,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"questdb.*\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Restarts",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 6,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[$interval]))",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 9,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"questdb.*\", resource=\"memory\"})",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 12,
-        "y": 1
-      },
-      "id": 6,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Memory (bytes)",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 15,
-        "y": 1
-      },
-      "id": 7,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Cores",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 18,
-        "y": 1
-      },
-      "id": 8,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network RX",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 21,
-        "y": 1
-      },
-      "id": 9,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network TX",
-      "type": "stat"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 5
-      },
-      "id": 10,
-      "panels": [],
-      "title": "Resources",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 6
-      },
-      "id": 11,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[$interval])",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 6
-      },
-      "id": 12,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 6
-      },
-      "id": 13,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[$interval])",
-          "legendFormat": "RX {{pod}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[$interval])",
-          "legendFormat": "TX {{pod}}",
-          "refId": "B"
-        }
-      ],
-      "title": "Network I/O",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 14
-      },
-      "id": 14,
-      "panels": [],
-      "title": "Logs",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 15
-      },
-      "id": 15,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"questdb.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Stream",
-      "type": "logs"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 25
-      },
-      "id": 16,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"questdb.*\"} | json | level =~ \"ERROR|WARN|FATAL\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors & Warnings",
-      "type": "logs"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "platform",
-    "questdb"
-  ],
-  "templating": {
-    "list": [
-      {
-        "auto": false,
-        "current": {
-          "text": "5m",
-          "value": "5m"
-        },
-        "name": "interval",
-        "options": [
-          {
-            "selected": false,
-            "text": "1m",
-            "value": "1m"
-          },
-          {
-            "selected": true,
-            "text": "5m",
-            "value": "5m"
-          },
-          {
-            "selected": false,
-            "text": "15m",
-            "value": "15m"
-          },
-          {
-            "selected": false,
-            "text": "30m",
-            "value": "30m"
-          },
-          {
-            "selected": false,
-            "text": "1h",
-            "value": "1h"
-          }
-        ],
-        "query": "1m,5m,15m,30m,1h",
-        "type": "interval"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "browser",
-  "title": "QuestDB",
-  "uid": "plat-questdb"
-}

infra/k8s/monitoring/dashboards/platform-tempo.json

@@ -1,802 +0,0 @@
-{
-  "annotations": {
-    "list": [
-      {
-        "builtIn": 1,
-        "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
-        },
-        "enable": true,
-        "hide": true,
-        "iconColor": "rgba(0, 211, 255, 1)",
-        "name": "Annotations & Alerts",
-        "type": "dashboard"
-      }
-    ]
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "links": [],
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "panels": [],
-      "title": "Tempo Health",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "red",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", phase=\"Running\"}",
-          "instant": true,
-          "legendFormat": "Running",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Pod Status",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 3
-              },
-              {
-                "color": "red",
-                "value": 10
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 3,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Restarts",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 6,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[$interval]))",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 9,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", resource=\"memory\"})",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 12,
-        "y": 1
-      },
-      "id": 6,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Memory (bytes)",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 15,
-        "y": 1
-      },
-      "id": 7,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Cores",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 18,
-        "y": 1
-      },
-      "id": 8,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network RX",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 21,
-        "y": 1
-      },
-      "id": 9,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network TX",
-      "type": "stat"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 5
-      },
-      "id": 10,
-      "panels": [],
-      "title": "Resources",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 6
-      },
-      "id": 11,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[$interval])",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 6
-      },
-      "id": 12,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 6
-      },
-      "id": 13,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[$interval])",
-          "legendFormat": "RX {{pod}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[$interval])",
-          "legendFormat": "TX {{pod}}",
-          "refId": "B"
-        }
-      ],
-      "title": "Network I/O",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 14
-      },
-      "id": 14,
-      "panels": [],
-      "title": "Logs",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 15
-      },
-      "id": 15,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Stream",
-      "type": "logs"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 25
-      },
-      "id": 16,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors & Warnings",
-      "type": "logs"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "platform",
-    "${DS_TEMPO}"
-  ],
-  "templating": {
-    "list": [
-      {
-        "auto": false,
-        "current": {
-          "text": "5m",
-          "value": "5m"
-        },
-        "name": "interval",
-        "options": [
-          {
-            "selected": false,
-            "text": "1m",
-            "value": "1m"
-          },
-          {
-            "selected": true,
-            "text": "5m",
-            "value": "5m"
-          },
-          {
-            "selected": false,
-            "text": "15m",
-            "value": "15m"
-          },
-          {
-            "selected": false,
-            "text": "30m",
-            "value": "30m"
-          },
-          {
-            "selected": false,
-            "text": "1h",
-            "value": "1h"
-          }
-        ],
-        "query": "1m,5m,15m,30m,1h",
-        "type": "interval"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "browser",
-  "title": "Tempo",
-  "uid": "plat-${DS_TEMPO}"
-}

infra/k8s/monitoring/dashboards/svc-api-gateway.json

@@ -1,963 +0,0 @@
-{
-  "annotations": {
-    "list": [
-      {
-        "builtIn": 1,
-        "datasource": {
-          "type": "grafana",
-          "uid": "-- Grafana --"
-        },
-        "enable": true,
-        "hide": true,
-        "iconColor": "rgba(0, 211, 255, 1)",
-        "name": "Annotations & Alerts",
-        "type": "dashboard"
-      }
-    ]
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "links": [],
-  "panels": [
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "panels": [],
-      "title": "Service Health",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "red",
-                "value": null
-              },
-              {
-                "color": "green",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 0,
-        "y": 1
-      },
-      "id": 2,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"api.*\", phase=\"Running\"}",
-          "instant": true,
-          "legendFormat": "Running",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Pod Status",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 3
-              },
-              {
-                "color": "red",
-                "value": 10
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 3,
-        "y": 1
-      },
-      "id": 3,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"api.*\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Restarts",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 6,
-        "y": 1
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[$interval]))",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "max": 1,
-          "min": 0,
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.7
-              },
-              {
-                "color": "red",
-                "value": 0.9
-              }
-            ]
-          },
-          "unit": "percentunit"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 5,
-        "w": 3,
-        "x": 9,
-        "y": 1
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"api.*\", resource=\"memory\"})",
-          "legendFormat": "",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "gauge"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 12,
-        "y": 1
-      },
-      "id": 6,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"})",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Memory (bytes)",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 15,
-        "y": 1
-      },
-      "id": 7,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Cores",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 18,
-        "y": 1
-      },
-      "id": 8,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network RX",
-      "type": "stat"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "color": {
-            "mode": "thresholds"
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 4,
-        "w": 3,
-        "x": 21,
-        "y": 1
-      },
-      "id": 9,
-      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ]
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[5m]))",
-          "instant": true,
-          "legendFormat": "",
-          "range": false,
-          "refId": "A"
-        }
-      ],
-      "title": "Network TX",
-      "type": "stat"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 5
-      },
-      "id": 10,
-      "panels": [],
-      "title": "Resources",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "short"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 6
-      },
-      "id": 11,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[$interval])",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "CPU Usage",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "bytes"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 6
-      },
-      "id": 12,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}",
-          "legendFormat": "{{container}}",
-          "refId": "A"
-        }
-      ],
-      "title": "Memory",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "Bps"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 6
-      },
-      "id": 13,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[$interval])",
-          "legendFormat": "RX {{pod}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[$interval])",
-          "legendFormat": "TX {{pod}}",
-          "refId": "B"
-        }
-      ],
-      "title": "Network I/O",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 14
-      },
-      "id": 14,
-      "panels": [],
-      "title": "gRPC Metrics",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 15
-      },
-      "id": 15,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(grpc_server_handled_total{grpc_service=~\"foxhunt.*\", pod=~\"api.*\"}[$interval])",
-          "legendFormat": "{{grpc_method}}",
-          "refId": "A"
-        }
-      ],
-      "title": "gRPC Request Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          },
-          "unit": "s"
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 15
-      },
-      "id": 16,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, rate(grpc_server_handling_seconds_bucket{pod=~\"api.*\"}[$interval]))",
-          "legendFormat": "p99",
-          "refId": "A"
-        }
-      ],
-      "title": "gRPC Latency (p99)",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "drawStyle": "line",
-            "fillOpacity": 10,
-            "lineWidth": 1,
-            "stacking": {
-              "mode": "none"
-            }
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 15
-      },
-      "id": 17,
-      "options": {
-        "legend": {
-          "displayMode": "list",
-          "placement": "bottom"
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rate(grpc_server_handled_total{grpc_code!=\"OK\", pod=~\"api.*\"}[$interval])",
-          "legendFormat": "{{grpc_code}}",
-          "refId": "A"
-        }
-      ],
-      "title": "gRPC Errors",
-      "type": "timeseries"
-    },
-    {
-      "collapsed": false,
-      "gridPos": {
-        "h": 1,
-        "w": 24,
-        "x": 0,
-        "y": 23
-      },
-      "id": 18,
-      "panels": [],
-      "title": "Logs",
-      "type": "row"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 24
-      },
-      "id": 19,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"api.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Log Stream",
-      "type": "logs"
-    },
-    {
-      "datasource": {
-        "type": "loki",
-        "uid": "${DS_LOKI}"
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 34
-      },
-      "id": 20,
-      "options": {
-        "dedupStrategy": "none",
-        "enableLogDetails": true,
-        "prettifyLogMessage": false,
-        "showCommonLabels": false,
-        "showLabels": true,
-        "showTime": true,
-        "sortOrder": "Descending",
-        "wrapLogMessage": true
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "${DS_LOKI}"
-          },
-          "expr": "{namespace=\"foxhunt\", pod=~\"api.*\"} | json | level =~ \"ERROR|WARN|FATAL\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
-          "refId": "A"
-        }
-      ],
-      "title": "Errors & Warnings",
-      "type": "logs"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 39,
-  "tags": [
-    "service",
-    "api"
-  ],
-  "templating": {
-    "list": [
-      {
-        "auto": false,
-        "current": {
-          "text": "5m",
-          "value": "5m"
-        },
-        "name": "interval",
-        "options": [
-          {
-            "selected": false,
-            "text": "1m",
-            "value": "1m"
-          },
-          {
-            "selected": true,
-            "text": "5m",
-            "value": "5m"
-          },
-          {
-            "selected": false,
-            "text": "15m",
-            "value": "15m"
-          },
-          {
-            "selected": false,
-            "text": "30m",
-            "value": "30m"
-          },
-          {
-            "selected": false,
-            "text": "1h",
-            "value": "1h"
-          }
-        ],
-        "query": "1m,5m,15m,30m,1h",
-        "type": "interval"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "browser",
-  "title": "API Gateway",
-  "uid": "svc-api-gateway"
-}