feat(sp14/sp22): enlarge aux head + trunk capacity (8x head, 2x trunk-H2)
After Path C confirmed aux head's 28% accuracy at H=60 is the H6 Phase 3 bottleneck (not the mechanism itself), enlarge aux capacity: - AUX_HIDDEN_DIM: 32 -> 256 (8x, matches input dim, removes bottleneck) - AUX_TRUNK_H2: 128 -> 256 (2x, uniform trunk width) Architecture changes: - Aux head: 256 -> Linear -> 256 -> ELU -> Linear -> 2 (was 256->32->2) - Aux trunk: 256 -> 256 -> 256 -> 256 (was 256 -> 256 -> 128 -> 256) - +160K params total (mostly aux_nb_w1 + aux_rg_w1: [256, 256] each) Side effects: - Checkpoint fingerprint change (intentional) - Thread utilisation improves: AUX_BLOCK=256 threads x H=256 = 1:1 (vs 8:1 at H=32 — most threads idle previously) Phase 3 mechanism stays DORMANT (W=0, beta=0) for this validation smoke. Verdict criteria: aux_dir_acc improves from 0.28 toward 0.50+ with the larger capacity. If yes, re-activate Phase 3 priors. If no, the bottleneck is signal/horizon, not capacity. Cargo build clean (full nvcc rebuild). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -64,7 +64,9 @@
|
||||
/* AUX_HIDDEN_DIM kept as a compile-time literal so kernel launch configs
|
||||
* don't need to thread `H` through. Both heads share the same hidden width. */
|
||||
#ifndef AUX_HIDDEN_DIM
|
||||
#define AUX_HIDDEN_DIM 32
|
||||
/* 2026-05-13: enlarged 32 → 256 (matches input h_s2_aux dim). See
|
||||
* `gpu_aux_heads.rs::AUX_HIDDEN_DIM` for rationale and side effects. */
|
||||
#define AUX_HIDDEN_DIM 256
|
||||
#endif
|
||||
|
||||
/* Block dim used by reduction-shaped launches (per-tensor backward, label
|
||||
|
||||
@@ -49,7 +49,24 @@ use super::gpu_dqn_trainer::{AUX_HEADS_CUBIN, AUX_HEADS_LOSS_EMA_CUBIN};
|
||||
/// Hidden dim shared by both auxiliary heads (Linear_1 output width).
|
||||
/// Kept in sync with the `AUX_HIDDEN_DIM` macro at the top of
|
||||
/// `aux_heads_kernel.cu`.
|
||||
pub(crate) const AUX_HIDDEN_DIM: usize = 32;
|
||||
///
|
||||
/// **Enlarged 32 → 256 (2026-05-13)** as part of the aux-head capacity
|
||||
/// upgrade after Path C investigation revealed the aux head's 28%
|
||||
/// accuracy at H=60 bars. The prior 32-unit bottleneck was severely
|
||||
/// under-capacity for a 256-dim input → 2-way classification task. The
|
||||
/// new 256 hidden dim matches the input dimension exactly (no
|
||||
/// dimensionality reduction in Layer 1) and gives the head full
|
||||
/// capacity to learn an arbitrary mapping. Architecture:
|
||||
/// 256 (h_s2_aux input) → Linear → 256 (hidden) → ELU → Linear → 2 (logits)
|
||||
///
|
||||
/// Side effects:
|
||||
/// - Param buffer grows: aux_nb_w1/aux_rg_w1 went from [32, 256] to
|
||||
/// [256, 256] each → +64K params per head → +128K params total.
|
||||
/// - Fingerprint changes → checkpoint incompatibility (intentional,
|
||||
/// fresh-start training).
|
||||
/// - Thread utilisation improves: AUX_BLOCK=256 threads × H=256 hidden
|
||||
/// = 1 thread per hidden unit (vs 8 threads/unit at H=32).
|
||||
pub(crate) const AUX_HIDDEN_DIM: usize = 256;
|
||||
|
||||
/// Output cardinality of the next-bar direction classification head
|
||||
/// (SP13 B1.1a, 2026-05-05). Flipped 1 → 2: the head was a K=1 MSE
|
||||
|
||||
@@ -29,11 +29,12 @@
|
||||
//! - `h_aux2 [B, H2]` — Layer-2 post-ELU activation
|
||||
//! - `h_s2_aux [B, AUX_HIDDEN_DIM]` — final linear output (no activation)
|
||||
//!
|
||||
//! Production topology (verified C.2):
|
||||
//! Production topology (verified C.2; widths uplifted 2026-05-13):
|
||||
//! - `ENCODER_OUT_DIM = config.shared_h1 = 256`
|
||||
//! - `H1 = AUX_TRUNK_H1 = 256`
|
||||
//! - `H2 = AUX_TRUNK_H2 = 128`
|
||||
//! - `AUX_HIDDEN_DIM = config.shared_h2 = 256` (matches existing aux head input dim)
|
||||
//! - `H2 = AUX_TRUNK_H2 = 256` (was 128 pre-2026-05-13 — bottleneck removed)
|
||||
//! - `AUX_HIDDEN_DIM = config.shared_h2 = 256` (matches existing aux head input dim;
|
||||
//! internal aux head hidden layer ALSO enlarged 32 → 256 in same commit)
|
||||
//!
|
||||
//! # Pearls applied
|
||||
//!
|
||||
@@ -65,10 +66,16 @@ use super::gpu_dqn_trainer::{
|
||||
pub(crate) const AUX_TRUNK_H1: usize = 256;
|
||||
|
||||
/// Hidden width of the aux trunk's second internal layer (Linear_2 → ELU
|
||||
/// output). Smaller than H1 for a mild bottleneck — encourages the aux
|
||||
/// trunk to compress its representation before lifting back up to
|
||||
/// `AUX_HIDDEN_DIM` in Layer 3.
|
||||
pub(crate) const AUX_TRUNK_H2: usize = 128;
|
||||
/// output).
|
||||
///
|
||||
/// **Enlarged 128 → 256 (2026-05-13)** alongside `AUX_HIDDEN_DIM 32 → 256`
|
||||
/// to remove the aux trunk's internal bottleneck. After the Path C
|
||||
/// investigation found the aux head's 28% accuracy at H=60 bars, the
|
||||
/// hypothesis is that capacity (not signal) was the bottleneck. The
|
||||
/// trunk now has uniform 256 width throughout: 256 (encoder) → 256 (L1)
|
||||
/// → 256 (L2) → 256 (L3 = h_s2_aux). This gives the trunk maximum
|
||||
/// expressive power before the aux head's classification layer.
|
||||
pub(crate) const AUX_TRUNK_H2: usize = 256;
|
||||
|
||||
/// Block dim used by the per-sample forward kernel. Mirrors
|
||||
/// `aux_heads_kernel.cu`'s `AUX_BLOCK = 256`.
|
||||
|
||||
Reference in New Issue
Block a user