diff --git a/docs/superpowers/specs/2026-05-25-q-learning-improvements.md b/docs/superpowers/specs/2026-05-25-q-learning-improvements.md index 34d4cc144..7cf464d3a 100644 --- a/docs/superpowers/specs/2026-05-25-q-learning-improvements.md +++ b/docs/superpowers/specs/2026-05-25-q-learning-improvements.md @@ -12,6 +12,59 @@ --- +## Mandatory Constraints + +Every line of code in this spec MUST follow these project rules. Violations are rejected at review. + +### GPU-drives-everything (CPU is read-only) +- **No CPU compute.** No CPU forwards, no CPU reference impls, no GPU↔CPU roundtrips in the hot path. Per `feedback_cpu_is_read_only`. +- **No host branches in captured graph.** Per `pearl_no_host_branches_in_captured_graph`. +- **No atomicAdd.** Block tree-reduce only. Per `feedback_no_atomicadd`. +- **Mapped-pinned memory only** for CPU↔GPU transfers. No `stream.memcpy_htod` / `memcpy_dtoh`. Use `write_slice_f32_d_pub` / `read_slice_d_pub` helpers. Per `feedback_no_htod_htoh_only_mapped_pinned`. +- **No nvrtc.** Pre-compiled cubins only via `build.rs`. Per `feedback_no_nvrtc`. +- **`build.rs` rerun-if-env-changed** for every `std::env::var()`. Per `pearl_build_rs_rerun_if_env_changed`. + +### ISV-driven (no hardcoded constants) +- **Every tunable parameter lives in ISV.** Thresholds, learning rates, n-step count, ensemble alpha, noise scale — all ISV slots with bootstrap defaults in the `isv_constants` array. Per `feedback_isv_for_adaptive_bounds`. +- **Structural dims reference consts, never literals.** `N_ACTIONS`, `Q_N_ATOMS`, `HIDDEN_DIM` etc. Per `feedback_use_consts_not_literals_for_structural_dims`. +- **CUDA kernels define ISV slot indices as `#define`s** matching the Rust `isv_slots.rs` constants exactly. + +### Memory and buffers +- **First-observation bootstrap.** Sentinel = 0; first observation replaces directly. Per `pearl_first_observation_bootstrap`. +- **Bilateral clamp on all bounded outputs.** `fmaxf(lo, fminf(x, hi))`. Per `pearl_symmetric_clamp_audit`. +- **Raw reward stored in replay.** Re-apply current `reward_scale` at sample time. Per `pearl_replay_reward_renormalization_at_sample_time`. +- **f64→f32 cast explicit** when calling float kernels via cudarc. Per `feedback_cudarc_f64_f32_abi`. + +### Code quality +- **No stubs.** Wire it for real or delete. Per `feedback_no_stubs`. +- **No TODO/FIXME/XXX.** Per `feedback_no_todo_fixme`. +- **No `_` hiding or `#[allow]`.** Per `feedback_no_hiding`. +- **No feature flags.** All features unconditional. Per `feedback_no_feature_flags`. +- **No SP/version prefixes** in file names or code identifiers. Per `feedback_no_sp_or_version_prefixes_in_file_names`. +- **Every new kernel wired in the same commit.** Per `feedback_wire_everything_up`. +- **State reset registry entries need matching dispatch arms.** Per `feedback_registry_entries_need_dispatch_arms`. + +### Testing +- **GPU oracle tests only.** No CPU reference implementation. Analytical invariants (sign symmetry, positional). Per `feedback_no_cpu_test_fallbacks`. +- **Local smoke before cluster.** 1k-step RTX 3050 Ti smoke catches CUDA arg-ABI, shared-mem, trait gaps. Per `feedback_local_smoke_before_cluster`. +- **Validate wiring with W=0 BEFORE deploying non-zero structural priors.** Per `feedback_smoke_validation_before_structural_priors`. + +### Training discipline +- **PER always enabled.** Per `feedback_always_per`. +- **Wiener-α floor 0.4 for non-stationary control loops.** Per `pearl_wiener_alpha_floor_for_nonstationary`. +- **Per-step ±2% clamp on reward_scale movement.** Per `pearl_reward_scale_per_step_clamp`. +- **Asymmetric controller decay** for coupling variables (ramp fast, decay 100× slower). Per `pearl_asymmetric_controller_decay_for_coupling`. +- **Gate warmup** before activation (ISV-driven steps). Per implemented warmup pattern. + +### Surfer philosophy +- **Entry cost** on every new position ($15 ISV-driven). +- **Min-hold** hard constraint (100 steps ISV-driven). Trail stops exempt at heat-cap level. +- **Asymmetric trail** — losers auto-tighten 0.995×/step, winners ratchet 50% of profit. +- **Session risk** circuit breaker at -$50 EMA. +- **Hold bonus** multiplicative on winning long rides. + +--- + ## Context ### Current Q path in `step_synthetic` (integrated.rs)