Implements spec §4.A.2 structural layout fingerprint with tail placement
rather than head placement (spec alternative: §4.A.2 Step 5.3 alt).
Head placement (ISV[0..2)) was rejected because isv_signals[0] and [1]
are actively written by the isv_signal_update kernel (Q-drift EMA and
gradient-norm EMA). Shifting those would require updating every literal
reference in experience_kernels.cu — a larger change than warranted for
pure contract enforcement. Tail placement leaves all existing indices
intact, touches zero kernel .cu files, and fulfils the same design contract.
Key changes:
- ISV_LAYOUT_FINGERPRINT_LO_INDEX = 37, HI_INDEX = 38 (u64 across 2×f32).
- LAYOUT_FINGERPRINT_CURRENT: u64 = FNV-1a of slot-list seed bytes.
Value: 0x85d4d76b578a7c17. Any slot change updates seed bytes,
which updates the hash automatically.
- Constructor writes fingerprint after zero-init; calls
check_layout_fingerprint() to self-verify before returning.
- check_layout_fingerprint(): reads pinned slots [37..39), recomposes u64,
fails-fast on mismatch with "retrain required" message.
- Error message does NOT mention migration as an option.
- Pre-commit hook rejects `fn migrate_isv|upgrade_isv` names — makes the
no-migration rule structurally enforced (check_no_isv_migrations).
- ISV_TOTAL_DIM: 37 → 39.
- Zero existing index shifts (no kernel literal sites affected).
- StateResetRegistry entry renamed ISV_SCHEMA_VERSION → ISV_LAYOUT_FINGERPRINT.
- ResetCategory::SchemaContract docstring updated to remove "migration" framing.
- docs/isv-slots.md: updated table + design note for tail placement.
Tests: state_reset_registry 3 unit tests pass with renamed entry.
cargo check -p ml clean (pre-existing warnings only).
Plan 1 Task 5. Spec §4.A.2 (tail-placement alternative).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan 1 Task 1. Creates the five audit docs plus config/metric-bands.toml
that track Invariants 2, 7, 8 per the DQN v2 spec, and extends the
pre-commit hook with two checks:
- component-adding commits must touch an audit doc (Invariant 7)
- added code may not contain TODO/FIXME/XXX/HACK/TBD/unimplemented!/
todo! markers (Invariant 9)
Tests: manually verified by staging a TODO-marked file; commit
rejected with the correct error message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Script rewritten to use argo submit --from=wftmpl/train with
train-epochs=0 — runs ensure-binary + ensure-fxcache only.
Local test fxcache regenerated with FXCACHE_VERSION=2, OFI_DIM=20.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add scripts/cuda-kernel-guard.sh — catches CPU/host leaks in .cu files:
printf, cudaMalloc/Free, host API calls, double-precision math (exp/log
instead of expf/logf), assert, file I/O. Filters comments and trailing
/* */ annotations. Zero false positives across 38 production kernels.
Extend gpu-hotpath-hook.sh to route .cu/.cuh files to the new guard.
Configure ruflo hooks in .claude/settings.json: pre-edit context loading,
post-edit pattern learning, post-command metrics, session-end persistence.
Fires for subagents too via Claude Code hook system.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The script only hashed .cu/.cuh in crates/ml/. Missed:
- build.rs changes (removing --use_fast_math → stale cubins cached)
- crates/ml-dqn/src/*.cu (replay buffer, seg tree kernels)
- crates/ml-core/src/cuda_autograd/*.cu
- crates/ml-ppo/src/cuda_nn/*.cu
Now hashes ALL kernel sources + ALL build.rs files. Any change to
nvcc flags or kernel source invalidates the entire PTX cache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added scaleway_vpc_public_gateway + DHCP + gateway_network to TF
(was manually created, now codified with push_default_route=true)
- Added scripts/safe-node-replace.sh — one-at-a-time with DNS verification
- Added dns-bootstrap-policy.yaml — incident documentation + recovery procedure
- Bastion enabled (port 61000) for emergency SSH access
Prevention: NEVER replace all nodes at once. Use safe-node-replace.sh.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: shmem_max_in_dim only included trunk dims (state_dim,
shared_h1, shared_h2) but not head dims (value_h, adv_h). When
hidden_dim_base=32 made the trunk narrow while heads stayed at 128,
the BF16 weight tile for branch output (255×128=32640 BF16 elements)
overflowed the shared memory region (12288 BF16 elements). On H100
the overflow landed in unused-but-mapped hardware shmem (silent
corruption). On RTX 3050 (48KB physical shmem) it hit unmapped
memory → CUDA_ERROR_ILLEGAL_ADDRESS.
Changes:
- gpu_dqn_trainer.rs: shmem_max_in_dim includes value_h/adv_h
- Remove all #[ignore] from smoke tests (feature_coverage,
training_stability, gpu_residency)
- Smoke tests use real .dbn data from test_data/ (hard error if missing)
- Remove synthetic_data() fallback — no fake data in tests
- GPU-direct DtoD training path (train_step_gpu, FusedTrainScalars)
- GPU-native PER priority update kernel (zero CPU readback)
- IQN dual-head integration (gpu_iqn_head.rs)
- BF16 dtype fixes across 6 model adapters
- Hyperopt 30D→31D (iqn_lambda)
- portfolio_transformer: unconditional BF16 (remove dead CPU branches)
- liquid/adapter: all tests use Cuda(0) directly
- Fix pre-existing gpu_kernel_parity_test.rs (stale args)
- Fix pre-existing evaluate_baseline.rs (removed fields)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After removing ensure_training_dtype(), forward methods that receive F32
inputs from tests/callers now fail with dtype mismatch against BF16 weights.
Add to_dtype(BF16) at forward entry of xLSTM (slstm, mlstm, block, network),
CfC cell, and diffusion time embedding. Fix quantization to accept BF16
tensors by casting to F32 before INT8 conversion. Update guard to catch
#[cfg(not(feature = "cuda"))] dead code. Bump DQN emergency_safe_defaults
replay_buffer_capacity from 1000 to 2048 (GPU PER floor = 1024).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unit tests need scalar readbacks for assertions — .to_scalar() is not
flagged but .to_vec1() in test modules would generate false positives.
Guard now correctly excludes inline #[cfg(test)] modules while checking
all production code including ensemble adapters.
Full --all scan reveals 31 pre-existing production violations across
ml-dqn, ml-ppo, ml-supervised, and ensemble adapters — separate cleanup.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace closure-based evaluate() with evaluate_dqn_graphed() for non-OFI
walk-forward backtest path. Extracts DuelingWeightSet from VarMap (branching
or standard dueling) and runs hand-written warp-cooperative CUDA forward
kernel with CUDA Graph capture — zero Candle dispatch overhead per step.
Key changes:
- GpuBacktestEvaluator::stream() getter for weight extraction on eval stream
- DQNAgentType::is_using_branching() / network_dims() for CUDA kernel config
- Hyperopt evaluate_gpu() non-OFI path: extract_dueling_weights_branching()
→ evaluate_dqn_graphed() (CUDA Graph accelerated)
- OFI path: retains Candle closure for state permutation (gather kernel
layout mismatch — future CUDA permutation kernel)
- 66+ GPU hot-path violations hardened to hard errors across DQN/PPO/supervised
- Stripped all gpu-ok suppression comments
- Proper #[cfg(feature = "cuda")] gating for CUDA-only code paths
77 files, 0 errors, 0 warnings across workspace.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Restore 45-action factored space via Branching DQN (Tavakoli 2018),
outputting 11 Q-values (5+3+3) instead of 45. This was reduced to 5
exposure-only actions during debugging and was never intended as permanent.
- Enable use_branching: true by default in DQNConfig and DQNHyperparameters
- Add branching paths to select_action_with_confidence and select_action_inference
- Update agent.rs select_action_factored for branching-aware selection
- Expand CountBonus to per-branch tracking with bonuses_branched()
- Add order_type + urgency distribution tracking in monitoring
- Add DQN_ORDER_ACTIONS=3, DQN_URGENCY_ACTIONS=3, DQN_TOTAL_ACTIONS=45 to CUDA header
- Fix 7 pre-existing clippy doc_markdown errors in regime_conditional.rs
- Fix pre-existing cognitive_complexity in replay_buffer_type.rs (extract helpers)
- Fix flaky GPU test OOM under parallel execution (CPU fallback + test VRAM safety)
- Delete unused flash_attention submodules (block_sparse, causal_masking, etc.)
- Add GPU hot-path guard scripts and ensemble/hyperopt adapter improvements
Tests: ml-dqn 416/0, ml 905/0, clippy 0 errors on both crates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BUG #41 kept forward pass in F32 for autograd, but the target-side
tensors (reward, gamma, done, next_q) were cast to BF16 via `dtype`.
The `state_action_values.sub(&target_q_values)` then hit F32-vs-BF16
mismatch on Ampere+ GPUs, causing every training step to fail silently.
Fix: `.to_dtype(state_action_values.dtype())` on the detached target.
Safe because target is detached (no autograd graph to break).
Also: H100 runner → SXM2 pool, GPU availability checker script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a parent/child GitLab CI pipeline for ML model training:
- Generator script produces per-model hyperopt/train/evaluate jobs
- Parent pipeline (.gitlab-ci-training.yml) with manual trigger
- NFS-backed ReadWriteMany PVC for shared training outputs
- Hyperopt params wired into training binaries (DQN, PPO, TFT, Mamba2)
- Shared DBN loader eliminates duplicate code across hyperopt adapters
- Supervised hyperopt unified to DBN data (was parquet-only)
Pipeline: hyperopt (4 models) → train (10 models) → evaluate ensemble
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split the build pipeline: one compile-services job builds all 8 service
binaries with PVC-backed sccache, saves as artifacts. Then 9 Kaniko jobs
just package pre-built binaries into slim runtime images (~30s each).
Before: 9 parallel Kaniko jobs each doing full cargo build --release
(~20min each, no sccache, 9x duplicated dep compilation)
After: 1 compile job with sccache (~5min cached) + 9 package jobs (~30s)
- Add compile stage between test and build
- Add Dockerfile.runtime (minimal debian + pre-built binary)
- Add Dockerfile.web-gateway-runtime (Node dashboard + pre-built binary)
- Keep Dockerfile.training via Kaniko (needs CUDA dev image for H100)
- Remove all SCCACHE_BUCKET build-args from service builds
- Use dir:// context for Kaniko (only sends build-out/ dir, not full repo)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These two files survived the 20-file consolidation in 022036cb.
Both are now fully superseded:
- train_ppo.rs → train_baseline_rl --model ppo
- train_mamba2.rs → train_baseline_supervised --model mamba2
Also updates entrypoint-generic.sh usage examples to reference
the unified binaries (train_baseline_rl, train_baseline_supervised).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add --cache=true --cache-repo to all 12 Kaniko builds
- Cache Docker layers in Scaleway CR (rg.fr-par.scw.cloud/foxhunt-ci/cache)
- Add Docker Hub auth to devcontainer + infra-runner prepare jobs
- First build populates cache; subsequent builds skip base image pulls
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Configures DevPod kubernetes provider, creates dev-home PVC,
verifies cluster access. Run once on developer laptop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Backs up the .git/hooks/pre-commit hook to a tracked file.
Includes stub detection (hardcoded returns, marker strings).
Cargo check removed — agents validate before commit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add train_launcher.sh that detects local GPU VRAM and routes training
to local or Scaleway cloud. Auto-selects batch size per model based on
available VRAM tier. Maps model names to actual ml/examples/train_*.rs
cargo targets. Document Scaleway GPU instance types, setup procedure,
batch size tables, and cost estimates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
