gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(sp21): T2.2 Phase 8.3+9 — eval pipeline GPU-only, hard-fail, delete CPU path (atomic)

Browse Source 
Combined Phase 8.3 (visibility + hard-fail) and Phase 9 (CPU path
removal) per pearl_no_deferrals_for_complementary_fixes. Surfaced by
v6 smoke (train-x4m96) where:

  [DQN GPU] Fold 0 GPU eval failed: GpuBacktestEvaluator::evaluate failed
    for fold 0. Falling back to CPU path.
  [DQN] Fold 0 evaluation failed: Failed to create DQN state tensor for
    bar 0: Dimension mismatch: expected 54, got 45

Both fold 0 and fold 1 hit this; workflow exited 0, masking eval
failure for every smoke run since STATE_DIM grew beyond legacy 54.

Root cause (silent GPU failure):
  evaluate_dqn_fold_gpu calls evaluator.evaluate(closure, portfolio_dim: 3)
  but GpuBacktestEvaluator initialises portfolio_dim = PORTFOLIO_BASE_DIM (8)
  + MTF_DIM (16) = 24 per canonical state layout. gather_states asserts
  match → returns MLError::ConfigError. Caller wraps with .with_context()
  + warn!("... {}", e) — `{}` strips anyhow chain, hiding root cause.
  The CPU fallback runs with a separate stale 45-dim state builder
  (42 from extract_ml_features + 3 portfolio) → fails at GpuTensor::
  from_host shape validation against the model's 54-feature default.

Fixes (all atomic):

  1. portfolio_dim: 3 → 24 at all 4 call sites (DQN x2, PPO, supervised).
     The GPU evaluator's gather_kernel handles full 128-dim state
     assembly (Market 42 + OFI 32 + TLOB 16 + MTF 16 + Portfolio 8 +
     PlanISV 7 + Padding 7); caller just declares correct portfolio_dim.

  2. Surface anyhow chain: {} → {:#} in error messages.

  3. Hard-fail on GPU eval failure: anyhow::bail! (no CPU fallback).
     Per user directive: "hard fail on gpu panic, cpu path strictly
     forbidden should be removed entirely!"

  4. DELETE CPU DQN eval path entirely:
     - fn evaluate_dqn_fold
     - fn build_chunk_states (stale 45-dim state builder)
     - fn simulate_chunk_trades
     - fn compute_metrics + struct ComputedMetrics
     - struct PortfolioState

  5. DELETE coupled surrogate-noise machinery:
     - struct SurrogateSampler + impl
     - fn load_surrogate_marginals
     - fn compute_pooled_sharpe
     - Surrogate init blocks in main
     - ACTION_MARGINALS / POOLED_SHARPE emission blocks

  6. DELETE coupled CLI flags:
     - --gpu-eval / --no-gpu-eval (GPU mandatory)
     - --surrogate-mode, --surrogate-seed, --surrogate-marginals
     - --emit-action-marginals, --emit-pooled-sharpe

  7. Collapse `if args.gpu_eval { ... }` blocks to direct calls; cleaner
     control flow, no gpu_handled tracking.

Pearls honoured:
  - feedback_no_cpu_test_fallbacks: GPU oracle only
  - feedback_no_partial_refactor: stale CPU layout from pre-STATE_DIM=128 era
  - feedback_no_hiding: error chain now visible via {:#}
  - feedback_no_legacy_aliases: no deprecated --no-gpu-eval wrapper
  - pearl_no_deferrals_for_complementary_fixes: 8.3+9 combined

Files changed:
  - crates/ml/examples/evaluate_baseline.rs:
      −892 net lines (1066 del, 174 ins; 2817 → 1925)
  - docs/dqn-wire-up-audit.md: 2026-05-12 audit entry

Verification:
  - cargo check -p ml --example evaluate_baseline --features cuda  # clean
  - cargo check --workspace --features cuda                        # clean
  - cargo test -p ml --lib --features cuda financials              # 7/7

Note: OFI/TLOB/MTF feature-set fidelity is a separate concern. The GPU
gather_kernel handles state assembly; caller currently provides zeroed
OFI (LobBar.ofi = 0.0) and no MTF data. Eval will run, but on degraded
features. Faithful feature wiring deferred to a later Phase once
eval-runs-at-all is validated by v7 smoke.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
jgrusewski
2026-05-12 08:47:05 +02:00
parent 79d0c53034
commit 23b89a90e9
2 changed files with 303 additions and 1067 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1242
crates/ml/examples/evaluate_baseline.rs
View File

File diff suppressed because it is too large Load Diff

128
docs/dqn-wire-up-audit.md
View File

@@ -15534,3 +15534,131 @@ to v1/v4 train-side stats):
The downstream PER alpha boost (E6+E7+E8 → `per_update_pa` and
`per_insert_pa`) will now have non-zero scalars to operate on,
finally exercising the Phase 5/6/7 wiring end-to-end.
## 2026-05-12 — SP21 T2.2 Phase 8.3+9: eval pipeline — fix GPU + hard-fail + delete CPU path (atomic)
### Scope (atomic single commit)
Combined Phase 8.3 (visibility + hard-fail) and Phase 9 (CPU fallback
removal) into one atomic change per `pearl_no_deferrals_for_complementary_fixes`.
Surfaced during v6 smoke (train-x4m96) eval phase, where:
```
[DQN GPU] Fold 0 GPU eval failed: GpuBacktestEvaluator::evaluate failed for fold 0. Falling back to CPU path.
[DQN] Fold 0 evaluation failed: Failed to create DQN state tensor for bar 0: Dimension mismatch: expected 54, got 45
```
Workflow exited 0 (eval pod swallowed both failures), giving the
illusion of success for every smoke run since STATE_DIM grew beyond
the legacy 54.
### Root cause (silent GPU eval failure)
`evaluate_dqn_fold_gpu` calls `evaluator.evaluate(closure, portfolio_dim: 3)`
but `GpuBacktestEvaluator::new` initialises with
`portfolio_dim = PORTFOLIO_BASE_DIM (8) + MTF_DIM (16) = 24` per the
canonical state layout. `gather_states` asserts:
```rust
if portfolio_dim != self.portfolio_dim {
return Err(MLError::ConfigError(format!(
"gather_states: portfolio_dim={portfolio_dim} != expected {}",
self.portfolio_dim
)));
}
```
→ every GPU eval fold returned `MLError::ConfigError("portfolio_dim=3 != expected 24")`.
The caller wrapped with `.with_context("GpuBacktestEvaluator::evaluate failed for fold {fold}")`
and logged via `warn!("... {}", e)` — `{}` strips anyhow's chain, hiding
the actual `ConfigError`. The CPU fallback then ran with a *separate*
state-builder using a 45-dim layout (42 from `extract_ml_features` + 3
portfolio), which failed at `GpuTensor::from_host` shape validation
against the model's expected 54-feature input (CLI default).
### Fixes (all atomic)
**1. Fix the actual GPU eval bug** — `portfolio_dim: 3 → 24` at all four
call sites (DQN closure x2, PPO, supervised). The GPU evaluator's
gather-kernel handles 128-dim state assembly internally (Market 42 +
OFI 32 + TLOB 16 + MTF 16 + Portfolio 8 + PlanISV 7 + Padding 7); the
caller just needs to declare the correct portfolio_dim.
**2. Surface the anyhow chain** on GPU eval failures — `{}` → `{:#}`
in error messages, so future regressions print the full root cause.
**3. Hard-fail on GPU eval failure (no CPU fallback)** — DQN/PPO/
supervised all now `anyhow::bail!` on GPU error per the user's
"hard fail on gpu panic, cpu path strictly forbidden" directive.
**4. DELETE the entire CPU DQN eval path** — removed:
- `fn evaluate_dqn_fold` (CPU-fallback DQN evaluator)
- `fn build_chunk_states` (stale 45-dim state builder)
- `fn simulate_chunk_trades` (CPU trade simulator)
- `fn compute_metrics` + `struct ComputedMetrics` (CPU metric aggregator)
- `struct PortfolioState` (CPU-side portfolio tracker)
**5. DELETE coupled surrogate-noise machinery** — removed:
- `struct SurrogateSampler` + impl (random-action override)
- `fn load_surrogate_marginals` (CDF loader)
- `fn compute_pooled_sharpe` (cross-fold pooled Sharpe)
- Surrogate init blocks in main
- ACTION_MARGINALS / POOLED_SHARPE emission blocks
**6. DELETE coupled CLI flags** — removed:
- `--gpu-eval` / `--no-gpu-eval` (GPU is mandatory)
- `--surrogate-mode`, `--surrogate-seed`, `--surrogate-marginals`
- `--emit-action-marginals`, `--emit-pooled-sharpe`
**7. Collapse `if args.gpu_eval { ... }` blocks** to direct calls
(GPU is the only path). Cleaner control flow, no gpu_handled tracking.
### Pearls + invariants honoured
- **feedback_no_cpu_test_fallbacks**: CPU fallback eliminated; GPU
oracle only, no CPU reference impl
- **feedback_no_partial_refactor**: stale 45-dim CPU path was a
partial-refactor leftover from pre-STATE_DIM=128 era; deleted, not
patched
- **feedback_no_hiding**: GPU error chain now visible via `{:#}`;
no swallowed errors via warn-and-continue
- **feedback_no_legacy_aliases**: `--no-gpu-eval` removed; no
deprecated wrappers
- **feedback_no_functionality_removal**: surrogate-noise is technically
a feature loss — derivative removal because its sole runtime (CPU
eval) was removed per the primary directive; reintroducing it
belongs in a separate plan (would need GPU closure-based
reimplementation per the comment trail)
- **pearl_no_deferrals_for_complementary_fixes**: 8.3 (visibility)
and 9 (CPU removal) share non-overlapping refactor scope; combined
into one commit rather than sequencing through two smoke cycles
### Files changed
- `crates/ml/examples/evaluate_baseline.rs`:
−892 net lines (1066 deleted, 174 inserted; 2817 → 1925 total)
- `docs/dqn-wire-up-audit.md`: this entry
### Verification
```
cargo check -p ml --example evaluate_baseline --features cuda # clean, 0 warnings
cargo check --workspace --features cuda # clean
cargo test -p ml --lib --features cuda financials # 7/7
```
### Expected v7 smoke (next dispatch)
The eval phase should now run cleanly — `GpuBacktestEvaluator::evaluate`
receives correct `portfolio_dim=24` and runs the full 128-dim state
gather/forward/env-step loop. Real DQN test-window metrics (Sharpe,
WinRate, MaxDD, PF, etc.) should appear in the eval log for all 3
folds.
OFI/TLOB/MTF features feed into the gather kernel via `lob_bars` (OFI),
the model's TLOB attention sub-net (TLOB), and pre-computed MTF buffers
(MTF) — all already wired in the evaluator's state-assembly kernel
(`backtest_state_gather`). The caller has been providing zeroed
OFI (`LobBar.ofi = 0.0` placeholder) and no MTF data; this is a
separate "feature-set fidelity" concern, deferred to a future
Phase 10 once eval-runs-at-all is validated.