The multi-seed path already does `kubectl apply` before `argo submit`,
so cluster template stays in sync with source. The single-job path used
`argo submit --from=wftmpl/train` directly, expecting the cluster's
template to already match — which silently drifts when defaults change.
Caused workflow train-jpxvn (2026-05-10) to dispatch with stale
imbalance-bar-threshold=0.5 default (the cluster's old value) when the
source had been bumped to 20.0. Triggered near-OOM in feature extraction.
One-line fix: apply the template before submission. Mirrors what
multi-seed already does. No behavior change for users who pass explicit
flags; just makes implicit defaults track source.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Two architectural cleanups, both surfaced by the wgdc8 experiment
### Part 1: volume_bar_size in cache key
Mirrors the imbalance_bar_threshold/ewma_alpha fix from `f7718b376`. The
volume bar size constant (100 contracts/bar) was previously hardcoded and
not in the fxcache key. Tuning it would have hit the same fossilization
bug as imbalance_bar_threshold did pre-fix.
Changes:
- `Hyperparams.volume_bar_size: u64` field added (default 100, matches
`DEFAULT_VOLUME_BAR_SIZE` for backwards compat).
- TrainingProfile loader reads `volume_bar_size` TOML key.
- `calculate_dbn_cache_key_full` signature 7 → 8 args. Hashed via
`to_le_bytes()`. Test `test_cache_key_includes_volume_bar_size`
added; passes alongside the 5 existing tests.
- 4 callers updated atomically (per `feedback_no_partial_refactor`):
`discover_and_load`, `data_loading.rs:146`, `train_baseline_rl.rs:599`,
`precompute_features.rs:259,720`.
- `data_loading.rs:279` now passes `self.hyperparams.volume_bar_size`
to `build_volume_bars` instead of the hardcoded `DEFAULT_VOLUME_BAR_SIZE`.
- New `--volume-bar-size` CLI arg on both binaries (default 100).
- New Argo workflow params `volume-bar-size` (default "100") and
`data-source` (default "mbp10") on both `train-template.yaml` and
`train-multi-seed-template.yaml`. Threaded into precompute + trainer
invocations.
- `scripts/argo-train.sh` exposes `--volume-bar-size <n>` and
`--data-source <s>` for ad-hoc overrides.
### Part 2: OFI front-month filter (latent bug fix)
`crates/ml/examples/precompute_features.rs:539-557` (the OFI/VPIN/Kyle's
Lambda computation branch when MBP-10 + trades data is available) was
loading trades unfiltered for per-bar microstructure feature computation.
The volume bar formation path filters front-month per-file (line 354), but
the OFI path did not.
Effect pre-fix: during contract rollover windows (e.g., ESZ24 → ESH25),
OFI per-bar microstructure features included trades from BOTH contracts
simultaneously, distorting VPIN, Kyle's Lambda, and trade imbalance
signals. Severity in production: small (front-month dominates ES.FUT
volume by 10-100×) but real and present in every prior MBP-10+trades
production run.
Fix: mirror the per-file `filter_front_month` call from the volume bar
path. Volume bar formation and OFI computation now both see the same
in-month trade tape. Added log line shows raw vs filtered count per file
for transparency.
## Why bundled
Both fixes touch trade-data plumbing in `precompute_features.rs` and the
fxcache key contract. Per `feedback_no_partial_refactor`, related
architectural cleanups land atomically. Both surfaced from the same
wgdc8 audit; bundling avoids two cache-key-invalidating commits in
sequence (each would force full fxcache regen).
## Compatibility
- `volume_bar_size` defaults to 100 → existing wgdc7-equivalent runs
reproduce, but with a *new* fxcache key (the f7718b376-era cache file
is unreachable; harmless, can GC manually).
- OFI fix is strictly more correct; no opt-out needed. Existing models
trained on contaminated OFI features may show slight feature
distribution drift on first cache regen — expected, not a regression.
- `data_source = "ohlcv"` Argo param now possible; routes precompute
through volume bar branch directly. wgdc8 experiment uses this to test
bar resolution sensitivity at volume_bar_size=500 (5× DEFAULT).
Tests: 6/6 feature_cache tests pass. Workspace + examples compile clean.
Audit-doc: `docs/dqn-wire-up-audit.md` updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## The bug (audit 2026-05-09)
`crates/ml/src/feature_cache.rs:calculate_dbn_cache_key_full` hashed only
`(symbol, data_source, dbn_filenames+sizes)` — NOT `imbalance_bar_threshold`
or `imbalance_bar_ewma_alpha`. Combined with `precompute_features.rs:346`
unconditionally calling `build_volume_bars` regardless of `data_source`,
this meant:
1. 14 audited production runs (Apr 11–May 8) all collided on the same
fxcache key (`a3f933aa...` / `c07c960a...`) regardless of TOML
`imbalance_bar_threshold` value
2. The imbalance-bar code path was reachable only via fxcache MISS, which
never happens in production because `ensure-fxcache` always populates
first
3. Every "tuning" of `imbalance_bar_threshold` across 16+ SP runs was a
silent no-op — the system was actually running volume bars at
DEFAULT_VOLUME_BAR_SIZE (100 contracts/bar)
## The fix (this commit)
**Part A — cache key includes bar formation params:**
- `calculate_dbn_cache_key_full` signature: 5 args → 7 args. Two new f64
params hashed via `to_le_bytes()`.
- 4 callers updated atomically (per `feedback_no_partial_refactor`).
- 2 new unit tests (`test_cache_key_includes_bar_threshold`,
`test_cache_key_includes_bar_alpha`) pin the contract.
**Part B — precompute_features actually USES data_source:**
- New CLI args `--imbalance-bar-threshold` (default 0.5) and
`--imbalance-bar-ewma-alpha` (default 0.1) on both train_baseline_rl
and precompute_features.
- `precompute_features.rs:346` now branches: when
`data_source == "mbp10"` AND `mbp10_data_dir.is_some()`, calls
`mbp10_to_imbalance_bars` instead of `build_volume_bars`.
**Argo plumbing:**
- `train-template.yaml` + `train-multi-seed-template.yaml`: new workflow
parameters threaded into BOTH precompute and trainer invocations so
both compute the same fxcache key.
- `scripts/argo-train.sh`: new CLI flags for ad-hoc overrides.
- ensure-fxcache regen path: removed `rm -f /feature-cache/*.fxcache`
(with bar-params now in key, parallel experiments coexist).
## Effects going forward
- Tuning `imbalance_bar_threshold` actually changes bar density
- Configuring `data_source = "mbp10"` actually produces imbalance bars
- Multiple parallel experiments at different thresholds coexist on PVC
- `dqn-production.toml: imbalance_bar_threshold = 0.5` no longer ignored
Default values match prior production behavior → existing wgdc7-equivalent
runs reproduce, just with a *new* fxcache key (the old volume-bar cache
file is still on disk but won't be hit; harmless, can GC manually).
Audit-doc: `docs/dqn-wire-up-audit.md` updated with full context.
Tests: 5/5 feature_cache tests pass, full workspace + examples compile clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 Tasks 1.1 + 1.2 of docs/superpowers/plans/2026-05-08-sp18-reward-shape-hold-attractor.md.
Per the plan's audit-first design: surface every live reference to the
SP13/SP16 Hold-cost-scale chain (D-leg) and the TD(λ) `q_next = rewards`
self-bootstrap (B-leg) BEFORE atomic deletion. This catches the SP17-style
"missed quantile_q_select consumer" failure mode where a sweeping refactor
left a dangling reference that compiled but broke at runtime.
What lands
- `scripts/audit_sp18_consumers.sh` — 17-section grep that walks every
consumer pattern from the plan's locked checklist (D-leg slot 380,
461, [462..468) HCS_*, hold_cost_scale_update kernel, hold_rate_observer
kernel retained chain, build.rs cubin manifest, state_layout.cuh
mirror constants, state_reset_registry entries, plus B-leg
td_lambda_kernel launch sites, q_next origin, rewards_out consumers,
PER priority sites, replay buffer schema, c51_loss target-Q origin,
target_params_buf consumers, PopArt slot 63 references). Three modes:
default (full grep output), `--fingerprint` (per-section per-file hit
count for diff-able snapshots), `--check` (diff fingerprint vs locked
snapshot in docs/sp18-wireup-audit.md, exit 1 on drift).
- `docs/sp18-wireup-audit.md` — Phase 1 Task 1.1 outcome with two
sections of import:
1. The plan's locked checklist (8 entries the plan author explicitly
identified as deletion targets).
2. The 10 ADDITIONAL CONSUMERS surfaced by the audit (A1-A10) that
the plan-author missed. Per the task input's halt-on-drift
directive, Phase 1 Tasks 1.3-1.5 (atomic deletion) are HALTED
pending human review of the expanded scope. The audit doc is the
spec-amendment record.
3. A locked fingerprint snapshot the pre-commit hook uses for drift
detection on subsequent commits.
B-leg verification confirms B-DD4 (no PER migration) + B-DD1 (target_
params_buf reusable) + the single q_next bootstrap site at
gpu_experience_collector.rs:4143.
- `scripts/pre-commit-hook.sh` — Invariant 7 list extended with the new
audit doc; new `check_sp18_consumer_audit` step runs the audit script
in `--check` mode whenever a commit touches the chain files
(experience_kernels.cu, hold_*_kernel.cu, state_layout.cuh, sp1[3-8]_
isv_slots.rs, gpu_dqn_trainer.rs, gpu_aux_trunk.rs, gpu_experience_
collector.rs, state_reset_registry.rs, training_loop.rs, build.rs,
sp1[3-8]_oracle_tests.rs). Drift triggers a hook failure with a
pointer to the regeneration command. This is a generalisation of
Invariant 7 and addresses Open Q-B from the spec (audit-as-pre-commit
hook for SP-chain consumer drift).
Findings (HALT trigger)
The audit found 10 consumers NOT in the plan's locked 8-site checklist:
A1 — gpu_aux_trunk.rs:1240-1323 HoldCostScaleUpdateOps struct + impl
(84 lines; the plan said launcher was in gpu_dqn_trainer but the
real struct/impl lives in gpu_aux_trunk; trainer just has a thin
forwarding method)
A2 — sp14_oracle_tests.rs:2174-2920 11 GPU oracle tests (~750 lines)
directly exercising the deleted kernel via include_bytes!
(sp16_phase2_hold_cost_scale_climbs_with_overrun + 10 others)
A3 — training_loop.rs:8971-9043 7 dispatch arms in reset_named_state
(slot 461 + HCS_* slots 462-467); contract test
every_fold_and_soft_reset_entry_has_dispatch_arm requires
atomic deletion alongside registry entries
A4 — gpu_dqn_trainer.rs:23200-23216 constructor block writing
HOLD_COST_BASE to slot 380 (RETAINED per DD7c, comment requires
update)
A5 — gpu_dqn_trainer.rs:617, 2245-2256 doc-block prose
A6 — sp13_isv_slots.rs:75-77 HOLD_COST_CONTROLLER_GAIN/FLOOR/CEIL
constants (HOLD_COST_BASE retained for A4)
A7 — gpu_experience_collector.rs:5579-5585 stale comment doc-ref
A8 — sp5_isv_slots.rs:325-327 comment doc-ref
A9 — state_reset_registry.rs:1138-1271 7 already-RETIRED entries
promote to FULL DELETION
A10 — state_reset_registry.rs:2256-2308 lock_sp18_v2_pp4_retired_chain
contract test asserts retired entries STILL EXIST; must be
deleted/rewritten when entries are removed
None of these are architectural surprises — they're straight extensions
of the atomic-deletion scope. But per the task input's halt-on-drift
directive ("If your audit surfaces ANY additional consumer, halt and
report — do NOT proceed with deletion ad-hoc"), Phase 1 Tasks 1.3-1.5
HALT pending human sign-off on the expanded scope.
Path forward (next phase)
Either expand the atomic-deletion commit scope to cover all 18 sites
(8 plan + 10 audit-surfaced) — recommended per `feedback_no_partial_
refactor`; the audit doc serves as the spec-amendment record — or
human-review the audit doc and explicitly approve/reject each A* entry.
The audit doc + script + hook are the pre-condition for either path
and ship together as Tasks 1.1 + 1.2 of Phase 1. Tasks 1.3-1.6 await
human go-ahead.
Branch is at INTERIM STATE: NOT runnable for L40S smoke between this
commit and the post-Phase-1.6 close-out. The interim is purely-additive
(audit script + hook + doc); the actual deletion that creates the
"3 reward sites missing Hold cost" interim from the plan's Task 1.5
has NOT yet been performed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
While P0b smoke (train-sw4ws on bdc5cb8bb) runs, two prep items:
(A) scripts/argo-train.sh — add `ci-training` to the L40S sm_89 case
arm. Bare `ci-training` is an L40S pool alias in some clusters;
previously defaulted to sm_90 (Hopper), causing train-mnpf7 to
deploy with wrong-arch cubins (terminated + resubmitted manually).
Now both `*l40s*` and bare `ci-training` resolve correctly.
(C) docs/superpowers/plans/...sp13...md — Layer B section expanded
with concrete codebase locations discovered during P0a:
- aux_heads_kernel.cu, aux_heads_loss_ema_kernel.cu locations
- aux_nb_label_buf populated as column 0 of next_states (log_return)
- F1/F2 regression history note (don't alias the label buffer)
- aux_pred_to_isv_tanh_kernel.cu is a P0a placeholder per its own
header — Layer B should rewrite to read softmax logit-diff
- dir_acc kernel + oracle tests need softmax-read updates
- Layer B + P0b combined rationale post-P0a empirics
Saves the next implementer ~30 min of re-investigation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related changes installing the structural guard against the SP6 Pearl 5
IQN τ failure mode (root cause fixed at facbf76eb for that one site) and
removing the only remaining orphan callers of the broken pattern.
The bug class. The mapped_pinned::{upload,clone_to_device}_{f32,i32}_via_pinned
helpers are named to suggest "no HtoD per feedback_no_htod_htoh_only_mapped_pinned"
but their bodies do MappedXBuffer::new() + memcpy_dtod_async() +
stream.synchronize(). The DtoD copy and synchronize are both forbidden
inside CUDA Graph capture (CUDA_ERROR_STREAM_CAPTURE_INVALIDATED) and add a
host stall otherwise. The canonical pattern is MappedXBuffer stored directly
+ write_from_slice + kernel reads via .dev_ptr, used by SP4 portfolio_state
and SP6 IQN τ at facbf76eb.
Guard. New check_no_dtod_via_pinned in pre-commit-hook.sh rejects any
staged .rs file calling upload_(f32|i32)_via_pinned or
clone_to_device_(f32|i32)_via_pinned, except mapped_pinned.rs itself. Per
feedback_no_hiding: no suppression marker. Also fixes a pre-existing
silent-skip bug: the gpu-hotpath-guard.sh invocation used
$(cd "$(dirname "$0")" && pwd) which resolved to .git/hooks/ (the symlink's
directory) instead of scripts/, so the guard never ran. Replaced with
readlink -f "$0" + an explicit "guard missing" error branch — silent skip
is worse than no guard.
Orphan deletion. gpu_her.rs carried legacy relabel_batch, generate_random_donors
(CPU), HerBatch, slice_clone_f32, slice_clone_i32 — zero production callers
(verified via grep). Production uses relabel_batch_with_strategy +
generate_random_donors_gpu. The orphan held the only upload_i32_via_pinned
callers in the codebase; per feedback_no_hiding the right fix is delete.
Scope. Eliminates 2 of 47 production *_via_pinned call sites. Remaining 45
across 14 files are cold-path init — graph-capture-fragile and host-stalling
but not breaking operationally. Guard enforces no new calls; existing 45
migrate in subsequent atomic per-buffer commits. After all 45 are converted,
the four helpers themselves get deleted from mapped_pinned.rs.
Validation. Smoke smoke-test-82fjk at facbf76eb succeeded — magnitude
differentiation restored (q_full=0.462 > q_half=0.409 > q_quarter=0.350 vs
baseline frozen Pascal-triangle 0.225/0.280/0.495), eval distribution
unfrozen (eq=0.596, eh=0.404, ef=0.000 vs baseline single-action collapse).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The L40S smoke template uses CARGO_TARGET_DIR=/cargo-target on a
persistent PVC, so every smoke probe builds incrementally on top of
artefacts left by prior probes. File deletions (e.g., regime_conditional.rs
in ff00af68a) can leave dangling rmeta/object references that perturb
downstream codegen between bisect runs — making the bisect result
contingent on which order probes were submitted in, not on the source.
Adds an optional `clean-cache` parameter (default `false`) which runs
`cargo clean -p ml -p ml-dqn --release` before the compile step. Other
crate artefacts (ml-core, ml-supervised, etc.) stay cached so the wipe
is bounded — fresh ml/ml-dqn compile in ~3-5 min vs ~30+ min full clean.
Use case: re-running a52d99613 + ff00af68a with `--clean-cache` to
verify the bisect under controlled build conditions. If the
broken/clean status flips with clean cache, the regression is
build-state-dependent rather than source-line; if it reproduces, the
source regression is real and bisect is definitive.
scripts/argo-smoke.sh exposes `--clean-cache` flag passing through
to the workflow parameter.
Adds a no-wrapper L40S smoke runner alongside the nsys/sanitizer
templates. Same compile + checkout + data-mount layout, no profiler
binary, no sanitizer instrumentation, no MinIO artefact upload — just
a plain test execution against the full training-data PVC.
Use case: validate a smoke test passes on the real 27-month dataset
when local data is too short (laptop only has 1 quarter of MBP-10/
trades, fxcache truncates below the 10-month minimum).
- infra/k8s/argo/smoke-test-template.yaml: WorkflowTemplate
smoke-test, entrypoint smoke-run, default test
multi_fold_convergence::test_multi_fold_convergence.
- infra/k8s/argo/kustomization.yaml: register the new template.
- infra/k8s/argo/argo-workflow-netpol.yaml: extend the
sanitizer/nsys NetworkPolicy podSelector to include smoke-test
(identical egress requirements: git fetch, ci-builder pull,
training-data PVC).
- scripts/argo-smoke.sh: thin wrapper mirroring argo-nsys.sh /
argo-sanitizer.sh (--multi-fold default, --test, --ref, --watch).
Verified: kustomize dry-run clean, kubectl apply -k creates the
template + reconfigures the netpol live in foxhunt namespace.
The original target 'foxhunt-training-artifacts' (cloned from
train-multi-seed-template) does not exist on the cluster MinIO.
Available production buckets: foxhunt-models, foxhunt-training-data,
foxhunt-training-results, foxhunt-binaries, foxhunt-backups,
foxhunt-gitlab-{artifacts,packages,registry}.
foxhunt-training-results is the right home for profiling artefacts
(model evaluation outputs already land there). Upload path is now
foxhunt-training-results/profiles/smoke/<short-sha>/profile-<pod>.nsys-rep.
Note: train-multi-seed-template has the same bucket-name typo —
production --profile uploads have been silently failing. Out of scope
for this commit, separate fix needed there.
Two new one-shot WorkflowTemplates for validating smoke tests under
GPU-instrumentation tools that don't fit in the laptop's 4 GB VRAM:
- `sanitizer-test`: wraps cargo test smoke under `compute-sanitizer
--tool=memcheck|racecheck|synccheck|initcheck` with --target-processes all
so multi_fold_convergence's spawned `train_baseline_rl` subprocess is also
instrumented. Pre-builds train_baseline_rl example to avoid sanitizer
instrumenting cargo/rustc on the inner spawn. Triages internal-vs-real
errors and exits non-zero only on real bugs.
- `nsys-test`: wraps cargo test smoke under `nsys profile`. Captures CUDA +
NVTX + osrt traces with GPU metrics (ga10x set). Uploads .nsys-rep to
MinIO at foxhunt-training-artifacts/profiles/smoke/<short-sha>/, mirroring
the existing Plan 5 Task 3 production-training profile pattern in
train-multi-seed-template.
Wrapper scripts:
- scripts/argo-sanitizer.sh — `argo submit` wrapper, supports --multi-fold
shortcut for fold-boundary code paths (IQN sync, aux Adam reset,
iqn_readiness reset, MSE clamp) that single-fold tests cannot exercise.
- scripts/argo-nsys.sh — same shape as argo-sanitizer.sh, default test is
performance::test_real_data_single_epoch for broad coverage.
Why L40S: laptop RTX 3050 Ti's 4 GB cannot fit compute-sanitizer's
instrumentation metadata (~2-3x app VRAM) — sanitizer falls back to "didn't
track the launch" with 60k+ internal-allocation errors. L40S 48 GB has
ample headroom for both memcheck and nsys overhead.
Both templates compile cargo test --release --lib --no-run plus cargo build
--release --example train_baseline_rl on the cargo-target PVC. Compile time
dominated by sccache hit rate (production training image: 100% C/C++ cache,
~75% Rust cache after warmup).
Templates registered in kustomization.yaml — apply with `kubectl apply -k
infra/k8s/argo` before first submit.
The first L40S deploy attempt (workflow `train-multi-seed-z2llf`, terminated)
failed at startup with `error: unexpected argument '--fold' found` on every
job: `train_baseline_rl` is a multi-fold walk-forward executor that accepts
`--max-folds K`, NOT `--fold N`. The original P5T1 harness assumed the
opposite and fanned out N seeds × K folds = N*K jobs, each invoking the
binary with `--seed N --fold K`.
User chose Path B: pivot to one job per seed (each runs all K folds via the
existing `--max-folds` mechanism). Per-job runtime is K× longer, but fanout
drops from N*K=30 → N=5 (matches L40S pool capacity better) and the binary
contract becomes the one the binary actually has.
4 surface changes:
1. crates/ml/examples/train_baseline_rl.rs — add `--seed N` CLI arg
(default 42 — historic implicit value). Sets `FOXHUNT_SEED` env var at
startup BEFORE any CUDA module spins up. Logs the seed value at the
training start banner.
2. crates/ml/src/cuda_pipeline/mod.rs — add `global_seed()` (reads
`FOXHUNT_SEED`, default 42) + `mix_seed(base)` (SplitMix64 avalanche
so adjacent global seeds produce uncorrelated module seeds). Six call
sites updated to mix the global seed into their previously-hardcoded
constants:
- trainer/action.rs: GpuActionSelector seed (0xDEAD_BEEF_CAFE) + the
epsilon-greedy fallback StdRng (0xAC7_DEF0).
- cuda_pipeline/gpu_iqn_head.rs: IQN Xavier-init RNG (0x1CA_1234).
- cuda_pipeline/gpu_iql_trainer.rs: V(s) Xavier-init RNG (0x1C1_9ABC).
- cuda_pipeline/gpu_her.rs: random-donor RNG (0x4E4_5678).
- cuda_pipeline/gpu_ppo_collector.rs: rng_seeds Vec for PPO
experience-collector init + reset (0xAA0_5EED).
- trainer/training_loop.rs: per-epoch regime_dropout_seed.
3. infra/k8s/argo/train-multi-seed-template.yaml — drop `fold` parameter
from `train-single` template; binary invoked as `--seed "$SEED"
--max-folds {{workflow.parameters.folds}}` so the walk-forward sweep
happens inside the single training process. Drop `FOLD` env var. Update
the nsys-rep upload filename to drop the fold suffix. Update banners /
doc comments to reflect "one-job-per-seed" semantics.
4. scripts/argo-train.sh — matrix generator drops the inner fold loop.
Each emitted task carries only `seed=${s}` and depends on the same
ensure-fxcache + gpu-warmup. The dry-run synthetic marker switches from
`seed=${s} fold=${f}` to `seed=${s} max_folds=${FOLDS}` so test harnesses
count the new shape correctly.
5. scripts/tests/test_multi_seed_harness.sh — assertions updated:
- `--multi-seed 3 --folds 2` produces 3 tasks (was 6).
- Rendered binary command must include `--max-folds
{{workflow.parameters.folds}}` placeholder.
- Rendered template must declare `folds` workflow parameter (so
`argo submit -p folds=K` overrides the default).
- Rendered binary command must NOT contain any per-fold flag — this
catches the failure mode that broke the first L40S deploy.
- Backward-compat: `--multi-seed 1 --folds 1` preserves the existing
single-template path (no DAG matrix tasks emitted).
6. docs/dqn-wire-up-audit.md — adds 1 Wired row documenting the pivot,
the new `--seed`/`mix_seed` plumbing, all 6 RNG call sites, and the
end-to-end seed-variation verification result.
Validation:
cargo check --workspace clean at 11 warnings (workspace baseline preserved).
cargo build --release --example train_baseline_rl succeeds; --help shows
the new --seed flag with documented default 42.
Seed-variation end-to-end test on RTX 3050 Ti (1 fold × 2 epochs each):
--seed 42 → F0 best Sharpe = -9.7831, best_val_metric = 1.957244,
epoch-2 train Sharpe = -16.12, val_Sharpe = +1.11.
--seed 999 → F0 best Sharpe = +92.9341, best_val_metric = 2.161012,
epoch-2 train Sharpe = +92.93, val_Sharpe = -0.25.
Different best Sharpe / best_val_metric / epoch-2 train + val Sharpe
across seeds proves the seed actually propagates through the RNG init
paths and is not just accepted-and-ignored. The seed=42 numbers match
the prompt's "deterministic baseline" expectation (F0 = -9.7831 was
bit-identical pre-pivot because no global-seed plumbing existed).
./scripts/argo-train.sh dqn --multi-seed 5 --folds 6 --dry-run produces
exactly 5 WorkflowTask markers (train-s0..train-s4), each with
`--max-folds {{workflow.parameters.folds}}` in the binary invocation.
All 3 harness tests PASS:
- test_multi_seed_harness.sh: 5 PASS lines, exit 0.
- test_nsys_harness.sh: 4 PASS lines + ALL PASS, exit 0.
- test_tier_checks.sh: PASS overall (good-fixture passes, bad-fixture
surfaces expected check rejections), exit 0.
Backward compat: existing single-job `argo-train.sh` callers (no
`--multi-seed`, no `--folds`) route to the original `train-template.yaml`
unchanged. `--seed 42` is a no-op offset for the SplitMix64 mix at the call
sites — the trajectory shifts only when the user passes `--seed` explicitly,
matching the prompt's "default 42 (historic implicit value)" requirement.
L40S pool: argo-train.sh defaults `--gpu-pool ci-training-h100`; user passes
`--gpu-pool ci-training-l40s` at deploy time. No script default change
(per constraint 5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan 5 Task 4 left every tier-2/tier-3 check failing with "metric missing
from aggregate" because the existing 'Validation backtest:' free-form log
line was not parseable by the aggregate-multi-seed-metrics.py block-keyed
parser. Phase A closes that gap end-to-end (CPU-only, no kernel touch):
* metrics.rs::compute_validation_loss — emit a new
HEALTH_DIAG[<epoch>]: val [sharpe=… sortino=… win_rate=…
max_drawdown=… trade_count=… calmar=…
omega_ratio=… total_pnl=… var_95=… cvar_95=…
trades_per_bar=… active_frac=… dir_entropy=…
sharpe_annualised=… profit_factor=…
window_bars=…]
block immediately after the existing 'Validation backtest:' line. All
16 keys derive from the existing GpuBacktestEvaluator WindowMetrics
reduction (no new GPU work):
- sharpe / sortino / win_rate / max_drawdown / total_trades /
calmar / omega_ratio / total_pnl / var_95 / cvar_95 / buy_count /
sell_count / hold_count come straight from m.*
- window_bars = buy + sell + hold (kernel tallies one direction
per bar)
- trades_per_bar = total_trades / window_bars
- active_frac = (buy + sell) / window_bars (kernel folds Hold AND
Flat into hold_count, so 'active' = bars where the policy chose
Short or Long — meets the Tier-2 'not always Hold' intent)
- dir_entropy = -Σ p ln p over the 3-bucket {short, hold-or-flat,
long} distribution. Documented limitation: max log(3) ≈ 1.099
vs spec's 4-bucket 0.8·log(4) ≈ 1.109 ceiling — tier2 dir_entropy
threshold is unreachable from this 3-bucket distribution; resolution
tracked in audit row.
- sharpe_annualised = m.sharpe alias (kernel already multiplies by
sqrt(bars_per_day · 252) at backtest_metrics_kernel:266)
- profit_factor = m.omega_ratio alias (kernel's omega computes
gain_sum/loss_sum at threshold 0, equivalent to per-step PF;
trade-level PF deferred — needs boundary-aware kernel work)
* mod.rs — adds last_val_metrics: Option<[f32; 14]> on DQNTrainer to
snapshot the WindowMetrics-derived values for downstream consumers
(smoke tests, future telemetry).
* constructor.rs — initialises the new field to None.
* aggregate-multi-seed-metrics.py — switches the block→key joiner from
'__' to '_' so 'val [sharpe=…]' surfaces as the bare 'val_sharpe'
aggregate key the tier check scripts and synthetic test fixtures
already expect. The pre-existing '__' joiner was an oversight in
Plan 5 Task 1B that was never validated against actual aggregator
output (the aggregator emitted 90 'block__key' metrics that nothing
consumed; the synthetic good_tier1.json / bad_tier1.json fixtures
were always shaped as 'val_sharpe', confirming the single-underscore
convention was intended). Renaming the 90 existing keys is safe — no
consumers had locked in on the '__' form.
* docs/dqn-wire-up-audit.md — updates Plan 5 Task 4 row to reference
the now-landed wiring and adds a new row documenting the val [...]
HEALTH_DIAG block pipeline + aggregator joiner change + the deferred
4-bucket dir_dist + trade-level PF caveats.
Validation:
cargo check --workspace clean at 11 warnings.
multi_fold_convergence smoke (629s, 3 folds × 5 epochs on RTX 3050 Ti)
PASSES with 3/3 fold checkpoints. Per-fold best Sharpe: -9.78 / 42.46 /
88.18 (within smoke noise band — no perturbation from the additive
CPU-only HEALTH_DIAG line).
scripts/aggregate-multi-seed-metrics.py against /tmp/p5t5a-smoke.log
produces 3 streams (one per fold), 16 val_* keys all present:
val_sharpe, val_sharpe_annualised, val_sortino, val_win_rate,
val_max_drawdown, val_trade_count, val_calmar, val_omega_ratio,
val_total_pnl, val_var_95, val_cvar_95, val_trades_per_bar,
val_active_frac, val_dir_entropy, val_profit_factor, val_window_bars.
check_tier2.py / check_tier3.py rejection messages are now substantive
(threshold-based) rather than "missing key":
Tier 2: trades_per_bar PASS @ 0.0127; active_frac FAIL @ 0.058 (model
mostly Hold on 5-epoch smoke); dir_entropy FAIL @ 0.18 (within
documented 3-bucket vs 4-bucket caveat).
Tier 3: sharpe_annualised FAIL @ -0.25 (5-epoch smoke not converged);
win_rate skipped (192 trades ≤ 500 noise gate); profit_factor
FAIL @ 0.18 (untrained policy).
Real validation pass requires the L40S 60-epoch run (Phase C).
Deferred (out of T5 Phase A scope):
- val_dir_dist_{short,hold,long,flat} per-direction breakdown — kernel
intentionally collapses Hold+Flat for trade-cycle counting; Tier-2's
log(4) threshold needs either a kernel-level split or a 3-bucket
threshold tweak in check_tier2.py.
- Trade-level profit_factor (sum-winner-PnL / sum-loser-PnL) vs the
per-step omega-equivalent emitted here.
- avg_q_value bare-key aggregation — the metric is logged via separate
Prometheus + tracing paths but not inside any HEALTH_DIAG block; out
of T5 Phase A scope and pre-existing.
Creates scripts/validation/ with per-tier exit checks consuming the
aggregate JSON from scripts/aggregate-multi-seed-metrics.py (P5T1B):
check_tier1.py — convergence (std/mean ≤ 0.15 on val_sharpe /
avg_q_value / train_loss; avg_q_value max ≤ 500 fold-1 explosion
guard; placeholders for Q-saturation + hot-path-DtoH per spec).
check_tier2.py — behavioural (val_trades_per_bar ≥ 0.005,
val_active_frac > 0.2, dir argmax entropy > 0.8·log4 with
val_dir_entropy primary + val_dir_dist_* fallback).
check_tier3.py — profitability (val_sharpe_annualised > 1.0 with
val_sharpe per-bar fallback, val_win_rate ≥ 0.52 gated on
>500 trades, val_profit_factor mean ≥ 1.1 AND cross-seed std < 0.3).
check_all_tiers.py — subprocess wrapper, exits 0 only if all pass.
Stdlib-only (statistics / argparse / json / subprocess) — no new deps.
Defensive missing-metric handling: each check FAILs with an explanatory
message when its required aggregate key is absent rather than silently
passing, so missing HEALTH_DIAG metrics are surfaced loudly.
Test harness scripts/validation/tests/test_tier_checks.sh exercises
good + bad fixtures across all four scripts and against the wrapper.
Audit row added to docs/dqn-wire-up-audit.md documenting the suite +
the deferred metrics list (val_trades_per_bar, val_active_frac,
val_dir_entropy/_dist_*, val_sharpe_annualised, val_win_rate,
val_profit_factor, val_trade_count) that HEALTH_DIAG must emit before
tiers 2/3 can ever PASS on real data — tracked for Plan 5 Task 5
pre-flight wire-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- argo-train.sh: --profile flag forces multi-seed render path so the
nsys wrapper + foxhunt-training-artifacts upload step are visible in
--dry-run YAML without cluster contact (test surface).
- train-multi-seed-template.yaml: new `profile` parameter (default
"false") gates the per-(seed, fold) `nsys profile
--capture-range=cudaProfilerApi` wrapper and the `mc cp` upload to
foxhunt-training-artifacts/profiles/<sha>/. mc binary fetched
on-demand (ci-builder image lacks it). MinIO creds optional —
upload warn-skips if absent.
- Dockerfile.foxhunt-training-runtime: install nsight-systems-cli
unpinned (pinning the stale 2024.4.1.61-1 from earlier plans
breaks builds when apt index advances).
- minio.yaml: add foxhunt-training-artifacts bucket to minio-init.
- compare-nsys-profiles.py: V0 regression detector — compares
cuda_gpu_kern_sum total_ns / epoch_count between two profiles;
exits 1 on >20% slowdown. NVTX per-epoch ranges deferred to T5.
- tests/test_nsys_harness.sh: dry-run grep test — verifies both
required strings appear when --profile is set, and that the
default (no --profile) path keeps profile=false in the rendered
template.
- dqn-wire-up-audit.md: Plan 5 Task 3 row added documenting the
harness + the baseline-capture deferral to T5.
Backward compat: test_multi_seed_harness.sh from P5T1 still PASS.
Adds the convergence guardrail: every per-epoch HEALTH_DIAG metric is
checked against the bands in config/metric-bands.toml; N consecutive
warn-band epochs emit a tracing::warn; 2N consecutive error-band epochs
return Err(CommonError::RegressionDetected{...}) cleanly from the
training loop, which propagates to the train_baseline_rl subprocess
exit code (no libc::raise — clean Rust error path).
Wire-points:
- New module: crates/ml/src/trainers/dqn/trainer/monitoring.rs
- MetricBands {warn_low, warn_high, error_low, error_high}
- BandSettings {consecutive_epochs_for_warn, consecutive_epochs_for_error}
- MetricBandsRegistry: load_from_toml + update_and_check
- TerminationReason {RegressionWarn, RegressionError}
- NaN treated as out-of-band (consecutive++; never resets streak)
- Unknown metrics return None (silent OK per Invariant 7 audit)
- crates/common/src/error.rs: new CommonError::RegressionDetected variant
carrying {metric, value, band, consecutive}
- crates/ml/src/trainers/dqn/trainer/constructor.rs: load
config/metric-bands.toml at trainer init; warn-only on missing file
(backward compat for environments without the config)
- crates/ml/src/trainers/dqn/trainer/training_loop.rs: harvest per-epoch
metrics (parallel emit alongside HEALTH_DIAG), feed each through
registry.update_and_check; on Some(TerminationReason::RegressionError)
emit final HEALTH_DIAG[N]: TERMINATED_BY_REGRESSION line and return Err
- services/trading_service/src/error.rs: minimal handler for the new
CommonError variant (existing pattern)
Validation:
- 8 unit tests in monitoring::tests pass (band logic, NaN, warn-only
behaviour, error-streak threshold, unknown-metric, invalid TOML)
- regression_detection GPU smoke (3.19s): trainer with intentionally
narrow train_loss error band [0, 1e-9] self-terminates at epoch 5
after 6 consecutive error-band epochs; final HEALTH_DIAG line emits
TERMINATED_BY_REGRESSION with metric/value/consecutive/band fields
- multi_fold_convergence smoke (650s, --release): all 3 folds train
to completion, all 3 checkpoints saved, no false-positive
termination on the populated metric bands. Per-fold best train
Sharpe: F0=-9.7831 (bit-baseline), F1=25.8272, F2=39.2687. F1/F2
on the lower end of observed noise distribution
({74.56, 61.10, 71.53, 25.83} for F1; {88.20, 61.57, 65.96, 39.27}
for F2) but training healthy throughout: aux clauses fire every
epoch, sharpe_ema recovers from F0 collapse (-9.78 → +14.8 by start
of F2), no regression detection trips.
config/metric-bands.toml populated for the metrics emitted by
HEALTH_DIAG today (avg_q_value, train_loss, val_sharpe, train_sharpe,
aux_next_bar_mse, aux_regime_ce, isv_* slot EMAs, sharpe_ema, etc.).
Bands derived from current cleanroom smoke + permissive defaults
where only one sample exists; populate-metric-bands-from-runs.py will
tighten them after Plan 5 Task 5's multi-seed pass produces real
distributions.
Constraints honoured: GPU-only in hot path (band check is CPU-side
post-HEALTH_DIAG, off the captured graph); no atomicAdd; no stubs;
no // ok: band-aids; no tuned constants beyond the toml-loaded bands;
no .unwrap() introduced; cargo check clean at 11 warnings (workspace
baseline preserved, plus ml-dqn pre-existing 1 warning).
Audit doc: new row added documenting monitoring.rs module, the
CommonError variant, the training_loop wire-point, and the design
choice that band-checks run AFTER HEALTH_DIAG emit (not before) so
the diag log already reflects the metric values that triggered any
termination.
Plan 5 T1 (multi-seed harness) landed at c6634254e+47c8b783c; T2
(this) gives the regression hard-stop that the multi-seed final
pass (T5) consumes to bail out early on bad seeds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion to Plan 5 Task 1A (multi-seed Argo DAG). Adds the post-run
aggregation pipeline:
- scripts/gather-multi-seed-metrics.sh: thin wrapper that pulls Argo logs
for every workflow tagged foxhunt-tag=<tag>, concatenates them into
/tmp/all-logs-<tag>.txt, then dispatches to the Python aggregator.
- scripts/aggregate-multi-seed-metrics.py: stdlib-only HEALTH_DIAG parser.
Recognises both bare and JSON-envelope-wrapped HEALTH_DIAG[<epoch>]
lines, parses every <block>[<key=val> ...] segment, and emits per-
(epoch, metric_name) mean / std / median / min / max across streams
(where each stream is one (seed, fold) training run, attributed by
pod-name prefix when present, else by epoch-rewind detection — naive
epoch=0 trigger over-segments the multi-line per-epoch HEALTH_DIAG
output, fixed by requiring epoch < last_epoch to start a new stream).
Output JSON schema matches the plan example (top-level: tag,
multi_seed, folds, warmup_end_epoch, streams_seen, aggregates;
per-entry: epoch, mean, std, median, min, max, n_samples).
- scripts/aggregate-norm-stats.py: stdlib-only merger for
norm_stats_foldN_seedM.json files. Per-fold output collapses N seeds
into mean/median/std/std_dispersion arrays consumed by the
`evaluate` step's inference normaliser.
- scripts/requirements.txt: declares numpy/scipy/matplotlib for downstream
Plan 5 T4-T5 tier-validation/plotting scripts. The aggregators
themselves are stdlib-only (statistics module).
Smoke-validated on /tmp/p4t6-cleanroom-smoke.log: detects 3 streams
(matching the 3 fold runs in that log), 90 unique metrics, n_samples=3
per (epoch, metric), schema matches plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the orchestration surface for Plan 5 Task 1 — Multi-Seed × Multi-Fold
Validation Harness:
- scripts/argo-train.sh: new --multi-seed N, --folds K, --tag T, --dry-run
flags. Default --multi-seed 1 --folds 1 routes to the existing
train-template.yaml (backward compat — existing callers unchanged). When
N>1 or K>1 the script renders train-multi-seed-template.yaml with an
inline-generated (seed, fold) matrix and either prints the YAML
(--dry-run) or applies + submits it.
- infra/k8s/argo/train-multi-seed-template.yaml: new WorkflowTemplate with
entrypoint multi-seed-matrix → ensure-binary, gpu-warmup, ensure-fxcache,
then N*K parallel train-single instances. Each train-single receives
seed/fold via inputs.parameters and forwards them to the training binary
via --seed/--fold CLI args + SEED/FOLD env vars. The dag.tasks placeholder
`# __MATRIX_TASKS__` is substituted programmatically by argo-train.sh
(awk) — no hand-written 30-task matrix.
- scripts/tests/test_multi_seed_harness.sh: dry-run regression test.
Asserts --multi-seed 3 --folds 2 emits 6 WorkflowTask markers AND
--multi-seed 1 --folds 1 emits zero (single-template path preserved).
Validation:
- argo lint --offline passes on both the source template and the rendered
3x2 / 5x6 outputs.
- test_multi_seed_harness.sh passes locally.
- Single-job dry-run still produces the unchanged train-template YAML.
Note: plan Step 0.1 pre-plan check expects ISV_TOTAL_DIM=72 and seven
ATTN_*_FOCUS_EMA_INDEX slots — both stale (Plan 4 landed
ISV_TOTAL_DIM=117 and the VSN_MAG_EMA / VSN_DIR_EMA / MAMBA2_RETENTION_EMA
slots instead). Plan 4 validation doc never landed (T8 deferred → Plan 5
T5). T1 is pure infrastructure that builds the harness consumed by T5,
so the stale pre-plan expectations do not block this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements spec §4.A.2 structural layout fingerprint with tail placement
rather than head placement (spec alternative: §4.A.2 Step 5.3 alt).
Head placement (ISV[0..2)) was rejected because isv_signals[0] and [1]
are actively written by the isv_signal_update kernel (Q-drift EMA and
gradient-norm EMA). Shifting those would require updating every literal
reference in experience_kernels.cu — a larger change than warranted for
pure contract enforcement. Tail placement leaves all existing indices
intact, touches zero kernel .cu files, and fulfils the same design contract.
Key changes:
- ISV_LAYOUT_FINGERPRINT_LO_INDEX = 37, HI_INDEX = 38 (u64 across 2×f32).
- LAYOUT_FINGERPRINT_CURRENT: u64 = FNV-1a of slot-list seed bytes.
Value: 0x85d4d76b578a7c17. Any slot change updates seed bytes,
which updates the hash automatically.
- Constructor writes fingerprint after zero-init; calls
check_layout_fingerprint() to self-verify before returning.
- check_layout_fingerprint(): reads pinned slots [37..39), recomposes u64,
fails-fast on mismatch with "retrain required" message.
- Error message does NOT mention migration as an option.
- Pre-commit hook rejects `fn migrate_isv|upgrade_isv` names — makes the
no-migration rule structurally enforced (check_no_isv_migrations).
- ISV_TOTAL_DIM: 37 → 39.
- Zero existing index shifts (no kernel literal sites affected).
- StateResetRegistry entry renamed ISV_SCHEMA_VERSION → ISV_LAYOUT_FINGERPRINT.
- ResetCategory::SchemaContract docstring updated to remove "migration" framing.
- docs/isv-slots.md: updated table + design note for tail placement.
Tests: state_reset_registry 3 unit tests pass with renamed entry.
cargo check -p ml clean (pre-existing warnings only).
Plan 1 Task 5. Spec §4.A.2 (tail-placement alternative).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan 1 Task 1. Creates the five audit docs plus config/metric-bands.toml
that track Invariants 2, 7, 8 per the DQN v2 spec, and extends the
pre-commit hook with two checks:
- component-adding commits must touch an audit doc (Invariant 7)
- added code may not contain TODO/FIXME/XXX/HACK/TBD/unimplemented!/
todo! markers (Invariant 9)
Tests: manually verified by staging a TODO-marked file; commit
rejected with the correct error message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Script rewritten to use argo submit --from=wftmpl/train with
train-epochs=0 — runs ensure-binary + ensure-fxcache only.
Local test fxcache regenerated with FXCACHE_VERSION=2, OFI_DIM=20.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add scripts/cuda-kernel-guard.sh — catches CPU/host leaks in .cu files:
printf, cudaMalloc/Free, host API calls, double-precision math (exp/log
instead of expf/logf), assert, file I/O. Filters comments and trailing
/* */ annotations. Zero false positives across 38 production kernels.
Extend gpu-hotpath-hook.sh to route .cu/.cuh files to the new guard.
Configure ruflo hooks in .claude/settings.json: pre-edit context loading,
post-edit pattern learning, post-command metrics, session-end persistence.
Fires for subagents too via Claude Code hook system.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The script only hashed .cu/.cuh in crates/ml/. Missed:
- build.rs changes (removing --use_fast_math → stale cubins cached)
- crates/ml-dqn/src/*.cu (replay buffer, seg tree kernels)
- crates/ml-core/src/cuda_autograd/*.cu
- crates/ml-ppo/src/cuda_nn/*.cu
Now hashes ALL kernel sources + ALL build.rs files. Any change to
nvcc flags or kernel source invalidates the entire PTX cache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added scaleway_vpc_public_gateway + DHCP + gateway_network to TF
(was manually created, now codified with push_default_route=true)
- Added scripts/safe-node-replace.sh — one-at-a-time with DNS verification
- Added dns-bootstrap-policy.yaml — incident documentation + recovery procedure
- Bastion enabled (port 61000) for emergency SSH access
Prevention: NEVER replace all nodes at once. Use safe-node-replace.sh.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: shmem_max_in_dim only included trunk dims (state_dim,
shared_h1, shared_h2) but not head dims (value_h, adv_h). When
hidden_dim_base=32 made the trunk narrow while heads stayed at 128,
the BF16 weight tile for branch output (255×128=32640 BF16 elements)
overflowed the shared memory region (12288 BF16 elements). On H100
the overflow landed in unused-but-mapped hardware shmem (silent
corruption). On RTX 3050 (48KB physical shmem) it hit unmapped
memory → CUDA_ERROR_ILLEGAL_ADDRESS.
Changes:
- gpu_dqn_trainer.rs: shmem_max_in_dim includes value_h/adv_h
- Remove all #[ignore] from smoke tests (feature_coverage,
training_stability, gpu_residency)
- Smoke tests use real .dbn data from test_data/ (hard error if missing)
- Remove synthetic_data() fallback — no fake data in tests
- GPU-direct DtoD training path (train_step_gpu, FusedTrainScalars)
- GPU-native PER priority update kernel (zero CPU readback)
- IQN dual-head integration (gpu_iqn_head.rs)
- BF16 dtype fixes across 6 model adapters
- Hyperopt 30D→31D (iqn_lambda)
- portfolio_transformer: unconditional BF16 (remove dead CPU branches)
- liquid/adapter: all tests use Cuda(0) directly
- Fix pre-existing gpu_kernel_parity_test.rs (stale args)
- Fix pre-existing evaluate_baseline.rs (removed fields)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After removing ensure_training_dtype(), forward methods that receive F32
inputs from tests/callers now fail with dtype mismatch against BF16 weights.
Add to_dtype(BF16) at forward entry of xLSTM (slstm, mlstm, block, network),
CfC cell, and diffusion time embedding. Fix quantization to accept BF16
tensors by casting to F32 before INT8 conversion. Update guard to catch
#[cfg(not(feature = "cuda"))] dead code. Bump DQN emergency_safe_defaults
replay_buffer_capacity from 1000 to 2048 (GPU PER floor = 1024).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unit tests need scalar readbacks for assertions — .to_scalar() is not
flagged but .to_vec1() in test modules would generate false positives.
Guard now correctly excludes inline #[cfg(test)] modules while checking
all production code including ensemble adapters.
Full --all scan reveals 31 pre-existing production violations across
ml-dqn, ml-ppo, ml-supervised, and ensemble adapters — separate cleanup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace closure-based evaluate() with evaluate_dqn_graphed() for non-OFI
walk-forward backtest path. Extracts DuelingWeightSet from VarMap (branching
or standard dueling) and runs hand-written warp-cooperative CUDA forward
kernel with CUDA Graph capture — zero Candle dispatch overhead per step.
Key changes:
- GpuBacktestEvaluator::stream() getter for weight extraction on eval stream
- DQNAgentType::is_using_branching() / network_dims() for CUDA kernel config
- Hyperopt evaluate_gpu() non-OFI path: extract_dueling_weights_branching()
→ evaluate_dqn_graphed() (CUDA Graph accelerated)
- OFI path: retains Candle closure for state permutation (gather kernel
layout mismatch — future CUDA permutation kernel)
- 66+ GPU hot-path violations hardened to hard errors across DQN/PPO/supervised
- Stripped all gpu-ok suppression comments
- Proper #[cfg(feature = "cuda")] gating for CUDA-only code paths
77 files, 0 errors, 0 warnings across workspace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Restore 45-action factored space via Branching DQN (Tavakoli 2018),
outputting 11 Q-values (5+3+3) instead of 45. This was reduced to 5
exposure-only actions during debugging and was never intended as permanent.
- Enable use_branching: true by default in DQNConfig and DQNHyperparameters
- Add branching paths to select_action_with_confidence and select_action_inference
- Update agent.rs select_action_factored for branching-aware selection
- Expand CountBonus to per-branch tracking with bonuses_branched()
- Add order_type + urgency distribution tracking in monitoring
- Add DQN_ORDER_ACTIONS=3, DQN_URGENCY_ACTIONS=3, DQN_TOTAL_ACTIONS=45 to CUDA header
- Fix 7 pre-existing clippy doc_markdown errors in regime_conditional.rs
- Fix pre-existing cognitive_complexity in replay_buffer_type.rs (extract helpers)
- Fix flaky GPU test OOM under parallel execution (CPU fallback + test VRAM safety)
- Delete unused flash_attention submodules (block_sparse, causal_masking, etc.)
- Add GPU hot-path guard scripts and ensemble/hyperopt adapter improvements
Tests: ml-dqn 416/0, ml 905/0, clippy 0 errors on both crates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BUG #41 kept forward pass in F32 for autograd, but the target-side
tensors (reward, gamma, done, next_q) were cast to BF16 via `dtype`.
The `state_action_values.sub(&target_q_values)` then hit F32-vs-BF16
mismatch on Ampere+ GPUs, causing every training step to fail silently.
Fix: `.to_dtype(state_action_values.dtype())` on the detached target.
Safe because target is detached (no autograd graph to break).
Also: H100 runner → SXM2 pool, GPU availability checker script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a parent/child GitLab CI pipeline for ML model training:
- Generator script produces per-model hyperopt/train/evaluate jobs
- Parent pipeline (.gitlab-ci-training.yml) with manual trigger
- NFS-backed ReadWriteMany PVC for shared training outputs
- Hyperopt params wired into training binaries (DQN, PPO, TFT, Mamba2)
- Shared DBN loader eliminates duplicate code across hyperopt adapters
- Supervised hyperopt unified to DBN data (was parquet-only)
Pipeline: hyperopt (4 models) → train (10 models) → evaluate ensemble
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the build pipeline: one compile-services job builds all 8 service
binaries with PVC-backed sccache, saves as artifacts. Then 9 Kaniko jobs
just package pre-built binaries into slim runtime images (~30s each).
Before: 9 parallel Kaniko jobs each doing full cargo build --release
(~20min each, no sccache, 9x duplicated dep compilation)
After: 1 compile job with sccache (~5min cached) + 9 package jobs (~30s)
- Add compile stage between test and build
- Add Dockerfile.runtime (minimal debian + pre-built binary)
- Add Dockerfile.web-gateway-runtime (Node dashboard + pre-built binary)
- Keep Dockerfile.training via Kaniko (needs CUDA dev image for H100)
- Remove all SCCACHE_BUCKET build-args from service builds
- Use dir:// context for Kaniko (only sends build-out/ dir, not full repo)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
