spec(sp19+20): patch 7 review issues
P1 (must-fix bugs/gaps): 1. ASYM_RATIO_INDEX → LOSS_CAP_INDEX with explicit formula in §4.1 and §4.5 (was double source-of-truth — formula in §4.1, ISV slot orphaned) 2. Replay buffer schema change (per-bar aux_conf) added to §8 implementation footprint (~50 LoC additional, was invisible in original spec) 3. hold_baseline_buffer size specified = LOOKAHEAD_HORIZON_MAX = 30 bars (§4.2) 4. "4-tier gate" typo → "5-tier gate" in §7 P2 (clarifications): 5. alpha_ema centering documented as load-bearing for cold-start learning (§4.1) — not just for Q-target stability. EV is slightly negative at WR=46% with uninformed SP19 label; advantage-style centering rescues cold-start. 6. Q-scale asymmetry between Hold (uncentered) and trade (alpha_ema centered) documented as INTENTIONAL design choice (§4.2) — produces marginal Q(trade) > Q(Hold) preference that counteracts the Q(Hold) attractor. Behavioral test sp20_pure_noise validates the no-signal Hold default still works. P3 (tightening): 7. Behavioral test thresholds tightened (§4.6): - sp20_pure_trend: WR > 70% → > 90%, PF > 2.0 → > 3.0 - sp20_pure_noise: Hold% > 80% → > 95%, trades < 50 → < 20
This commit is contained in:
jgrusewski
@@ -71,8 +71,8 @@ This spec re-grades the model on **closed-trade directional correctness** with *
|
||||
spread R_used over trade_duration bars │
|
||||
│ │
|
||||
└──────────► Replay buffer ◄───────────┘
|
||||
│ (aux_conf at trade open
|
||||
│ stored alongside reward)
|
||||
│ (aux_conf stored per bar,
|
||||
│ read at the sampled state)
|
||||
▼
|
||||
[Bellman target — Component 4]
|
||||
Q_target = aux_gate × full_target +
|
||||
@@ -88,7 +88,7 @@ This spec re-grades the model on **closed-trade directional correctness** with *
|
||||
fires every bar Hold is held; aux_conf high → cost high
|
||||
|
||||
──── Adaptive Curriculum (ISV) — Component 5 ────
|
||||
asym_ratio ← ramp(wr_ema)
|
||||
loss_cap ← ramp(wr_ema)
|
||||
aux_threshold ← adapt(aux_dir_acc_ema)
|
||||
n_step ← adapt(trade_duration_ema)
|
||||
cost_scale ← adapt(hold_pct_ema)
|
||||
@@ -108,8 +108,9 @@ Six components, each independently testable, all shipping atomically per `feedba
|
||||
```
|
||||
At trade close (is_close=1):
|
||||
win_cap = +1.0
|
||||
loss_cap = -1.0 - clamp((wr_ema - 0.50) / 0.05, 0, 1)
|
||||
# = -1 below WR=50%, ramps to -2 by WR=55%
|
||||
loss_cap = ISV[LOSS_CAP_INDEX] # produced by Kernel 2 from wr_ema
|
||||
# value: -1.0 - clamp((wr_ema - 0.50) / 0.05, 0, 1)
|
||||
# = -1 below WR=50%, ramps to -2 by WR=55%
|
||||
|
||||
label_at_open = sign(SP19_blended_label[trade_open_bar])
|
||||
dir_match = (sign(close_PnL) == label_at_open)
|
||||
@@ -133,13 +134,15 @@ At trade close (is_close=1):
|
||||
→ distributed n-step credit by Component 3
|
||||
```
|
||||
|
||||
**alpha_ema centering is load-bearing for cold-start learning, not just Q-target stability.** At WR=46% with the 4-quadrant table and uninformed SP19 label, raw EV ≈ −0.06 (model would prefer not to trade). The `R_used = alpha − alpha_ema` subtraction transforms absolute EV into advantage-style relative-to-mean. Even when absolute alpha is negative, the model still learns "this trade was BETTER than my average trade → do more like this". Without this centering, the model collapses to Hold during cold start before it can learn directional discrimination. Do not drop this term during implementation as an "optimization".
|
||||
|
||||
**Implementation:**
|
||||
- New device function `sp20_compute_event_reward()` in `experience_kernels.cu` (~80 LoC)
|
||||
- Replaces SP18 D-leg `compute_sp18_hold_opportunity_cost` (returned 0 throughout 5rbml — empirically null)
|
||||
- Replaces SP12 v3 reward block (`compute_sp12_reward_with_cost` and per-bar additions)
|
||||
|
||||
**ISV slots (3 new):**
|
||||
- `ASYM_RATIO_INDEX` — produced by Kernel 2 from `wr_ema`
|
||||
- `LOSS_CAP_INDEX` — produced by Kernel 2 from `wr_ema` (was `ASYM_RATIO_INDEX` — renamed for clarity; stores the actual loss_cap value, not a ratio)
|
||||
- `ALPHA_EMA_INDEX` — produced by Kernel 1 (Wiener-α EMA)
|
||||
- `WR_EMA_INDEX` — produced by Kernel 1 (per-trade EMA)
|
||||
|
||||
@@ -180,6 +183,10 @@ elif hold_pct_ema < target_hold_pct - 0.05:
|
||||
- Aux untrained (warmup): `aux_conf_p50_ema ≈ 0.05` → `target_hold_pct ≈ 0.725` → controller pushes `cost_scale` toward floor 0.01 → per-bar cost effectively zero. **No explicit warmup gate needed.**
|
||||
- Flat action: treated as not-Hold (no cost). Flat is intentional close-all, conceptually different from Hold.
|
||||
|
||||
**Buffer sizing:** `hold_baseline_buffer` is a circular buffer of size `LOOKAHEAD_HORIZON_MAX = 30` bars (matches SP19's longest horizon and bounds max trade duration). Bars older than 30 are overwritten — fine, since trades > 30 bars don't sample beyond their last 30 bars.
|
||||
|
||||
**Q-scale design note (intentional asymmetry between Hold and trade rewards):** Hold per-bar reward is **uncentered** (raw `-aux_conf × cost_scale`), whereas trade reward is centered via `R_used = alpha - alpha_ema`. This asymmetry is **deliberate**: it produces a marginal Q(trade) > Q(Hold) preference (Q(Hold) accumulates a small negative drift, Q(trade) is zero-mean), which counteracts the Q(Hold) attractor we are trying to break. Centering Hold rewards too would symmetrize the bias and remove this engineered preference. Behavioral test `sp20_pure_noise` validates the Hold attractor is still correctly induced when there is no signal — so the marginal trade preference doesn't override the no-signal Hold default.
|
||||
|
||||
**Implementation:**
|
||||
- 1 new producer kernel `hold_cost_scale_compute_kernel.cu` (~50 LoC, follows SP4 Pearls A+D)
|
||||
- Modification at `experience_kernels.cu` per-bar reward path: add `R_per_bar_i` term, write to `hold_baseline_buffer`
|
||||
@@ -281,7 +288,7 @@ Produces (Wiener-α EMAs with floor 0.4):
|
||||
|
||||
Consumes: above EMAs.
|
||||
Produces controller outputs:
|
||||
- `ASYM_RATIO_INDEX` = ramp from wr_ema (Section 4.1)
|
||||
- `LOSS_CAP_INDEX` = -1.0 - clamp((wr_ema - 0.50) / 0.05, 0, 1) (Section 4.1)
|
||||
- `N_STEP_INDEX` = round(trade_duration_ema), clamp [1, 30]
|
||||
- `AUX_CONF_THRESHOLD` = clamp(aux_dir_acc_ema − 0.50, 0.01, 0.20)
|
||||
- `AUX_GATE_TEMP` = aux_conf_std_ema (with floor)
|
||||
@@ -302,8 +309,8 @@ Uses `sp4_histogram_p99.cuh` pattern for p50 (median, k=128).
|
||||
|
||||
| Test | Synthetic market | Pass criterion | Proves |
|
||||
|---|---|---|---|
|
||||
| `sp20_pure_trend` | Strong monotonic trend, low noise | WR > 70%, PF > 2.0 | Model CAN learn directional skill when signal exists |
|
||||
| `sp20_pure_noise` | i.i.d. zero-mean returns | Hold% > 80%, trades < 50/run | Model doesn't gamble — Hold attractor correctly induced when no edge exists |
|
||||
| `sp20_pure_trend` | Strong monotonic trend, low noise | WR > 90%, PF > 3.0 | Model CAN learn directional skill when signal exists. Tightened: pure-trend with low noise is near-deterministic; loose threshold hides real failures. |
|
||||
| `sp20_pure_noise` | i.i.d. zero-mean returns | Hold% > 95%, trades < 20/run | Model doesn't gamble — Hold attractor correctly induced when no edge exists. Tightened for the same reason. |
|
||||
| `sp20_confidence_correlation` | Mixed (50% trend + 50% noise bars) | corr(aux_conf, trade_WR) > 0.4 | Aux gate is functional |
|
||||
| `sp20_asymmetry_no_game` | Trending market, hard symmetric clamp | PF > 1.0 (small but positive) | WR optimization isn't an artifact of asymmetric clamp |
|
||||
| `sp20_n_step_distribution` | 3-bar trade, known close reward = +0.6, n_step = 3 | Each of 3 bars receives credit ≈ 0.2 | n-step distributor is correct |
|
||||
@@ -359,7 +366,7 @@ Verified-natural cases (no explicit handling needed):
|
||||
|
||||
## 7. Testing strategy
|
||||
|
||||
4-tier gate, in order. Each tier blocks the next:
|
||||
5-tier gate, in order. Each tier blocks the next:
|
||||
|
||||
1. **Per-kernel unit tests** (RTX 3050 Ti, ~50 tests)
|
||||
- Each kernel: oracle test, sentinel-bootstrap test, ISV signal independence test, reset behavior test
|
||||
@@ -382,10 +389,12 @@ Approximate sizing per component (atomic ship per `feedback_no_partial_refactor`
|
||||
| 3: n-step distributor | `gpu_experience_collector.rs` | `n_step_credit_kernel.cu` | 110 |
|
||||
| 4: Aux→Q gate | `gpu_dqn_trainer.rs` | — | 40 |
|
||||
| 5: Adaptive curriculum | `sp14_isv_slots.rs`, `mod.rs`, `build.rs` | `sp20_emas_compute_kernel.cu`, `sp20_controllers_compute_kernel.cu`, `sp20_stats_compute_kernel.cu` | 280 |
|
||||
| Replay buffer schema | `gpu_experience_collector.rs`, `GpuBatchPtrs` struct, callers | — | 50 |
|
||||
| (per-bar `aux_conf` field for Component 4 gate consumption — propagated through experience tuple, batch ptrs, replay path)| | | |
|
||||
| 6: Behavioral tests | — | `sp19_20_behavioral_tests.rs` | 250 |
|
||||
| Per-component unit tests | — | various | 600 |
|
||||
| Audit doc + memory pearls | `audit/sp20_wireup.md` (new) | — | 100 |
|
||||
| **Total** | | | **~1,550** |
|
||||
| **Total** | | | **~1,600** |
|
||||
|
||||
Comparable in scope to SP18 (~2,500 LoC). Smaller than SP15 (~5,000 LoC).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user