feat(sp14/sp22): enlarge aux head + trunk capacity (8x head, 2x trunk-H2)
After Path C confirmed aux head's 28% accuracy at H=60 is the H6 Phase 3 bottleneck (not the mechanism itself), enlarge aux capacity: - AUX_HIDDEN_DIM: 32 -> 256 (8x, matches input dim, removes bottleneck) - AUX_TRUNK_H2: 128 -> 256 (2x, uniform trunk width) Architecture changes: - Aux head: 256 -> Linear -> 256 -> ELU -> Linear -> 2 (was 256->32->2) - Aux trunk: 256 -> 256 -> 256 -> 256 (was 256 -> 256 -> 128 -> 256) - +160K params total (mostly aux_nb_w1 + aux_rg_w1: [256, 256] each) Side effects: - Checkpoint fingerprint change (intentional) - Thread utilisation improves: AUX_BLOCK=256 threads x H=256 = 1:1 (vs 8:1 at H=32 — most threads idle previously) Phase 3 mechanism stays DORMANT (W=0, beta=0) for this validation smoke. Verdict criteria: aux_dir_acc improves from 0.28 toward 0.50+ with the larger capacity. If yes, re-activate Phase 3 priors. If no, the bottleneck is signal/horizon, not capacity. Cargo build clean (full nvcc rebuild). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -64,7 +64,9 @@
|
||||
/* AUX_HIDDEN_DIM kept as a compile-time literal so kernel launch configs
|
||||
* don't need to thread `H` through. Both heads share the same hidden width. */
|
||||
#ifndef AUX_HIDDEN_DIM
|
||||
#define AUX_HIDDEN_DIM 32
|
||||
/* 2026-05-13: enlarged 32 → 256 (matches input h_s2_aux dim). See
|
||||
* `gpu_aux_heads.rs::AUX_HIDDEN_DIM` for rationale and side effects. */
|
||||
#define AUX_HIDDEN_DIM 256
|
||||
#endif
|
||||
|
||||
/* Block dim used by reduction-shaped launches (per-tensor backward, label
|
||||
|
||||
@@ -49,7 +49,24 @@ use super::gpu_dqn_trainer::{AUX_HEADS_CUBIN, AUX_HEADS_LOSS_EMA_CUBIN};
|
||||
/// Hidden dim shared by both auxiliary heads (Linear_1 output width).
|
||||
/// Kept in sync with the `AUX_HIDDEN_DIM` macro at the top of
|
||||
/// `aux_heads_kernel.cu`.
|
||||
pub(crate) const AUX_HIDDEN_DIM: usize = 32;
|
||||
///
|
||||
/// **Enlarged 32 → 256 (2026-05-13)** as part of the aux-head capacity
|
||||
/// upgrade after Path C investigation revealed the aux head's 28%
|
||||
/// accuracy at H=60 bars. The prior 32-unit bottleneck was severely
|
||||
/// under-capacity for a 256-dim input → 2-way classification task. The
|
||||
/// new 256 hidden dim matches the input dimension exactly (no
|
||||
/// dimensionality reduction in Layer 1) and gives the head full
|
||||
/// capacity to learn an arbitrary mapping. Architecture:
|
||||
/// 256 (h_s2_aux input) → Linear → 256 (hidden) → ELU → Linear → 2 (logits)
|
||||
///
|
||||
/// Side effects:
|
||||
/// - Param buffer grows: aux_nb_w1/aux_rg_w1 went from [32, 256] to
|
||||
/// [256, 256] each → +64K params per head → +128K params total.
|
||||
/// - Fingerprint changes → checkpoint incompatibility (intentional,
|
||||
/// fresh-start training).
|
||||
/// - Thread utilisation improves: AUX_BLOCK=256 threads × H=256 hidden
|
||||
/// = 1 thread per hidden unit (vs 8 threads/unit at H=32).
|
||||
pub(crate) const AUX_HIDDEN_DIM: usize = 256;
|
||||
|
||||
/// Output cardinality of the next-bar direction classification head
|
||||
/// (SP13 B1.1a, 2026-05-05). Flipped 1 → 2: the head was a K=1 MSE
|
||||
|
||||
@@ -29,11 +29,12 @@
|
||||
//! - `h_aux2 [B, H2]` — Layer-2 post-ELU activation
|
||||
//! - `h_s2_aux [B, AUX_HIDDEN_DIM]` — final linear output (no activation)
|
||||
//!
|
||||
//! Production topology (verified C.2):
|
||||
//! Production topology (verified C.2; widths uplifted 2026-05-13):
|
||||
//! - `ENCODER_OUT_DIM = config.shared_h1 = 256`
|
||||
//! - `H1 = AUX_TRUNK_H1 = 256`
|
||||
//! - `H2 = AUX_TRUNK_H2 = 128`
|
||||
//! - `AUX_HIDDEN_DIM = config.shared_h2 = 256` (matches existing aux head input dim)
|
||||
//! - `H2 = AUX_TRUNK_H2 = 256` (was 128 pre-2026-05-13 — bottleneck removed)
|
||||
//! - `AUX_HIDDEN_DIM = config.shared_h2 = 256` (matches existing aux head input dim;
|
||||
//! internal aux head hidden layer ALSO enlarged 32 → 256 in same commit)
|
||||
//!
|
||||
//! # Pearls applied
|
||||
//!
|
||||
@@ -65,10 +66,16 @@ use super::gpu_dqn_trainer::{
|
||||
pub(crate) const AUX_TRUNK_H1: usize = 256;
|
||||
|
||||
/// Hidden width of the aux trunk's second internal layer (Linear_2 → ELU
|
||||
/// output). Smaller than H1 for a mild bottleneck — encourages the aux
|
||||
/// trunk to compress its representation before lifting back up to
|
||||
/// `AUX_HIDDEN_DIM` in Layer 3.
|
||||
pub(crate) const AUX_TRUNK_H2: usize = 128;
|
||||
/// output).
|
||||
///
|
||||
/// **Enlarged 128 → 256 (2026-05-13)** alongside `AUX_HIDDEN_DIM 32 → 256`
|
||||
/// to remove the aux trunk's internal bottleneck. After the Path C
|
||||
/// investigation found the aux head's 28% accuracy at H=60 bars, the
|
||||
/// hypothesis is that capacity (not signal) was the bottleneck. The
|
||||
/// trunk now has uniform 256 width throughout: 256 (encoder) → 256 (L1)
|
||||
/// → 256 (L2) → 256 (L3 = h_s2_aux). This gives the trunk maximum
|
||||
/// expressive power before the aux head's classification layer.
|
||||
pub(crate) const AUX_TRUNK_H2: usize = 256;
|
||||
|
||||
/// Block dim used by the per-sample forward kernel. Mirrors
|
||||
/// `aux_heads_kernel.cu`'s `AUX_BLOCK = 256`.
|
||||
|
||||
@@ -17521,3 +17521,27 @@ To re-activate the mechanism, restore W=[-0.5, 0, +0.5, 0] in `aux_w_prior_init_
|
||||
6. **Use aux as confidence gate only**: keep state[121] = recentered aux p_up, but DON'T use it as direction signal. Use it as a "trust this decision" gate that scales action magnitude. Useful even if direction is non-predictable.
|
||||
|
||||
Cargo check clean.
|
||||
|
||||
#### Aux head + trunk capacity uplift (2026-05-13)
|
||||
|
||||
After Path C confirmed the aux head's 28% accuracy at H=60 bars is the actual H6 Phase 3 bottleneck, this commit enlarges aux capacity 8× on the head + 2× on the trunk's middle layer:
|
||||
|
||||
| Component | Before | After |
|
||||
|-----------|--------|-------|
|
||||
| `AUX_HIDDEN_DIM` (aux head internal hidden) | 32 | **256** |
|
||||
| `AUX_TRUNK_H2` (aux trunk middle layer) | 128 | **256** |
|
||||
|
||||
**Aux head architecture** (now): `256 (h_s2_aux) → Linear → 256 (hidden) → ELU → Linear → 2 (softmax)`. Removes the 32-unit bottleneck that compressed 256 input dims into 32 features before classification. Param count grows: aux_nb_w1 + aux_rg_w1 each `[32, 256]` → `[256, 256]` = +128K params total.
|
||||
|
||||
**Aux trunk architecture** (now): `256 → 256 → 256 → 256` (uniform width). Removes the H2 bottleneck. Param count for `aux_trunk_w2 [H1, H2]` grows `[256, 128]` → `[256, 256]` = +32K params.
|
||||
|
||||
**Hypothesis under test**: aux head's 28% accuracy was a **capacity bottleneck**, not a fundamental signal/horizon limitation. With 8× hidden dim, the head should learn at least majority-class prediction (≥50% accuracy on a 17/83 split) and ideally better.
|
||||
|
||||
**Side effects**:
|
||||
- Param buffer + tensor layout fingerprint change → checkpoint incompatibility
|
||||
- Modest compute increase per forward/backward (memory-bandwidth-bound at this scale, negligible wall-time impact on L40S)
|
||||
- Thread utilisation IMPROVES: AUX_BLOCK=256 threads × H=256 hidden = 1:1 mapping (vs 256:32 ≈ 8:1 idle ratio at H=32)
|
||||
|
||||
**Phase 3 mechanism stays DORMANT** for this smoke (W=0, β=0). If aux head accuracy improves to ≥55% with the larger capacity, the next step is to re-activate Phase 3 (W=[-0.5, 0, +0.5, 0], β=0.5) and re-validate WR.
|
||||
|
||||
Cargo build clean (1m12s — full nvcc rebuild of aux kernels).
|
||||
|
||||
Reference in New Issue
Block a user