Add to_varstore() compatibility shims on DuelingQNetwork and
DistributionalDuelingQNetwork so test/example code can rebuild a
GpuVarStore snapshot when needed. Delete dead tests that referenced
removed DQNAgent, PrioritizedReplayBuffer, and ReplayBufferType.
Fix action index references (action_19/21 -> action_28/30) and
type annotation issues (sin ambiguity, remainder operator).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DuelingWeightSet/BranchingWeightSet now created via from_flat_buffer()
pointing directly into GpuDqnTrainer's params_buf. No intermediate
VarStore extraction, no D2D copies, no flatten/unflatten cycle for
the primary weight sets.
bottleneck_dim=16 now works — smoke test passes. The dimension mismatch
is resolved because params_buf is allocated at the correct reduced
dimension, and weight sets point directly into it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added OwnedGpuLinear to ml-core (self-contained linear layer with owned
weight/bias CudaSlice, forward_with_slices bypasses GpuVarStore lookup)
- Removed GpuVarStore from all 12 ml-dqn modules: Sequential, branching,
distributional_dueling, dueling, network, rmsnorm, residual, curiosity,
attention, iql, quantile_regression, regime_conditional
- Removed get_q_network_vars/vars/store accessors from DQN, branching,
distributional_dueling, dueling, network, quantile_regression
- Replaced GpuLinear+GpuVarStore with OwnedGpuLinear in all cold-path modules
- Made load_from_safetensors a no-op (flat buffer path handles checkpoints)
- Made sync_to_varmap a no-op (flat buffer is source of truth)
- Moved branching->VarStore conversion to gpu_weights::branching_to_varstore
- Re-enabled bottleneck_dim=16 in config defaults + all 3 TOML configs
- Removed flatten_online_weights bottleneck skip workaround
- Zero GpuVarStore references in crates/ml-dqn/src + crates/ml/src/trainers/dqn
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DuelingWeightSet and BranchingWeightSet fields changed from owned
CudaSlice<f32> to raw u64 device pointers + usize element counts.
This enables zero-copy views directly into the flat params_buf for
the training hot path (no D2D copies, no shadow allocations).
Key changes:
- Weight sets store u64 + usize pairs instead of CudaSlice<f32>
- from_flat_buffer() creates zero-copy views into params_buf
- from_slices() creates pointer views from owned CudaSlice allocations
- DuelingWeightBacking/BranchingWeightBacking hold owned CudaSlice
arrays for callers that need independent allocations (experience
collector, ensemble heads, tests)
- flatten/unflatten become no-ops when weight sets point into params_buf
- extract functions return (Backing, WeightSet) tuples
- KernelWeightPack::build() uses u64 fields directly (no raw_device_ptr)
- All sync functions updated for pointer-based signatures
- Ensemble clone functions return backing + pointer view pairs
- gradient_budget tests updated (7/7 pass)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
DuelingWeightSet becomes raw u64 pointers into params_buf, not separate
CudaSlice allocations. Eliminates ALL D2D copies between weight
representations. flatten/unflatten cycle deleted entirely.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VarStore is a CPU-trip corrupting data + degrading performance. Must be
replaced with direct pointer buffers (flat params_buf is already the
source of truth). Checkpoint, weight extraction, and polyak updates
all need porting to direct CudaSlice operations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bottleneck_dim=16 causes w_s1 size mismatch between VarStore (full state_dim)
and flat params_buf (reduced bottleneck_concat_dim). EMA target sync crashes
with CUDA_ERROR_ILLEGAL_ADDRESS.
Root cause: GpuVarStore allocates weights at full state_dim but GpuDqnTrainer's
flat buffer uses bottleneck-reduced dimensions. These two weight sources are
fundamentally incompatible when bottleneck is active.
Fix: eliminate GpuVarStore entirely (next task), then re-enable bottleneck.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Launch ensemble_aggregate_kernel per training step to populate
ensemble_var_q_buf with per-atom C51 logit variance across K heads.
Expose via ensemble_var_ptr() on FusedTrainingContext. Add
ensemble_exploration_beta config (default 0.2). Wire into
set_ensemble_variance_bonus() on GpuExperienceCollector which computes
beta * sqrt(mean_var) and tiles it as a constant SAXPY bonus over all
Q-values before Boltzmann action selection. No-op when ensemble_count <= 1.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IQR (Q75-Q25) from IQN's 32 quantile estimates is computed per-action
after each training step and added to Q-values before Boltzmann action
selection. High-IQR actions (uncertain outcomes, typically large-magnitude
positions) get an exploration boost, countering C51's structural bias
toward low-variance Small actions that causes magnitude collapse.
Pipeline: IQN train -> compute_iqr() -> set_iqr_bonus() -> SAXPY on
q_values [N, q_stride] each experience timestep. Alpha default 0.3.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Even timesteps use the existing directional mirror (Short<->Long, negated
reward). Odd timesteps substitute a different magnitude and linearly scale
the reward by alt_mag_frac / actual_mag_frac, giving the magnitude branch
direct gradient signal about what bigger or smaller positions would have
earned. Falls back to directional mirror when reward is near-zero (scaling
uninformative). No extra buffer needed -- reuses the existing N*L
counterfactual slot.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set c51_warmup_epochs default to 0. The ramp logic in training_loop.rs
already returns alpha_max immediately when c51_warmup_epochs==0, so all
epochs now see c51_alpha=0.5 from the first step. Stable loss landscape,
no CUDA graph recaptures from alpha transitions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ShortSmall=0, ShortHalf=1, ShortFull=2, Flat=3, LongSmall=4, LongHalf=5,
LongFull=6. Updates all array sizes, bounds checks, names, and distribution
logic across gpu_monitoring, monitoring, training_loop, metrics, and financials.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Task 3: Enable PopArt reward normalization (default true, all 3 TOML configs,
wiring confirmed in fused_training.rs submit_forward_ops_main()).
Task 4: Couple PopArt variance with tau reset — add prev_popart_var to
FusedTrainingCtx, read_popart_var() to GpuDqnTrainer, read_popart_variance()
+ should_reset_tau() to FusedTrainingCtx, tau reset injected at epoch boundary
in training_loop.rs after log_phase_timing().
Task 5: Add best_sharpe/best_epoch/best_val_loss reset to reset_for_fold() so
each walk-forward fold competes independently. Also wire v8 reward fields
(popart_enabled, micro_reward_scale, td_lambda, max_trace_length,
hindsight_fraction, hindsight_lookahead) through training_profile.rs apply_to().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The 4-branch DQN (direction x magnitude) had 3 degenerate variants
(Short25, Flat, Long25) that all mapped to 0.0 target exposure when
direction=Flat, causing 82% Flat collapse. Collapse these into a
single Flat variant, giving 7 levels (ShortSmall/Half/Full, Flat,
LongSmall/Half/Full) and 63 total factored actions (7x3x3).
- ExposureLevel enum: 9 variants -> 7 (add direction/magnitude/from_dir_mag)
- FactoredAction: 81 -> 63 total actions, from_index/to_index updated
- DQN epsilon-greedy: use from_dir_mag() instead of dir*3+mag indexing
- DQN config: num_actions default 9 -> 7
- PPO action space: 45 -> 63 actions, action masking updated
- Signal adapter CUDA kernel: 5-bin -> 7-bin exposure aggregation
- All tests updated for new variant names and index ranges
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
10-component design to fix 82% Flat/Small action concentration:
1. Collapse degenerate 9→7 level exposure space
2. Per-branch reward decomposition (direction/magnitude/order/urgency)
3. Hindsight action relabeling for magnitude
4. Position-aware temporal exploration with per-branch epsilon
5. IQN quantile spread as anti-collapse signal
6. Activate PopArt + couple with tau reset
7. Feed the gradient vaccine with diverse batches
8. Per-branch bottleneck widths
9. Ensemble variance at inference
10. Clear PER between folds
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fuses GEMM + bias-add + ReLU into a single cublasLtMatmul kernel via
CUBLASLT_EPILOGUE_RELU_BIAS. Eliminates 7 separate add_bias_relu
kernel launches per forward pass (3 trunk + 4 branch hidden layers).
Creates separate cached descriptors with epilogue enabled at init time.
Falls back to separate kernels if the epilogue heuristic isn't available.
Bias pointer set dynamically per-call via set_matmul_desc_attribute.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Performance:
- Cache cuBLAS descriptors: pre-create matmul_desc + layouts + algo at init.
requestedAlgoCount=3 for better algorithm selection. Zero per-GEMM overhead.
- Per-branch workspace: 4 × 32MB separate workspace buffers for multi-stream
branch dispatch. Eliminates workspace contention on parallel execution.
Training stability:
- Remove hardcoded shrink-perturb that fired every epoch on short runs (3-5 epochs).
With epochs=5, interval = epochs/4 = 1 → fired every epoch, destroying epoch 1
learned weights. This caused Sharpe to collapse from +0.60 to -0.29 after epoch 1.
- Phase 3 shrink-perturb now uses config values (was hardcoded alpha=0.9, sigma=0.01).
- The config-defined shrink_perturb_interval=20 now controls all shrink-perturb timing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The backward GEMM heuristic on H100 with CUDA 13.0 requires 32MB
workspace for TF32 algorithm selection. 16MB causes
CUBLAS_STATUS_NOT_SUPPORTED on dW_only (m=256,n=128,k=16384).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IQN was recomputing target h_s2 via iqn_trunk_forward_kernel using the
same target weights on the same next_states that DQN Pass 2 already
computed into tg_h_s2_buf. Now IQN reads from the pre-computed buffer
via a DtoD copy (negligible) instead of re-running 2 cuBLAS GEMMs.
Expected: -15ms GPU time on H100 (batch=16384, shared_h2=256).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Events recorded during CUDA graph capture cannot be queried with
cuEventElapsedTime after graph replay — the graph creates internal
copies. Reverted to original mega-graph capture structure.
Sub-graph timing fields kept in PhaseEvents for future use with
split-graph diagnostic mode or nsys profiling.
Also kept submit_loss_and_grad_ops() extraction and pub(crate) visibility
on launch_cublas_forward/backward for future per-phase graph splitting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Split fwd_bwd timing into 5 sub-phases: spectral, forward, loss,
backward, aux. Events recorded during graph capture replay with the
graph on every step.
Also extract submit_loss_and_grad_ops() from submit_forward_ops_main()
and make launch_cublas_forward/backward pub(crate) for sub-graph timing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CI builder uses CUDA 13.0 cudarc which exports _v2 suffix.
Local CUDA 12.x has both names but CI only resolves _v2.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: cuStreamSynchronize blocked the thread, preventing tokio
single-threaded runtime from progressing RwLock .write().await calls.
- Remove cuStreamSynchronize from training loop (stalls pipeline)
- Move log_phase_timing() after process_epoch_boundary (DtoH syncs stream)
- compute_epoch_q_diagnostics takes &mut DQNAgentType param (no re-lock)
- Disable Q-value gap diagnostics (deadlock in single-threaded runtime)
- smoke_trainer helpers: always set fxcache dir
- test_no_hang_single_epoch: use fxcache direct loading
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Delete 7 dead files: prioritized_replay.rs, prioritized_replay_staleness.rs,
replay_buffer.rs, hindsight_replay.rs, rainbow_agent.rs, checkpoint.rs,
strategy_dqn_bridge.rs (-4,760 lines)
- Rewrite replay_buffer_type.rs: ReplayBufferType enum → StagedGpuBuffer struct
(GPU PER is the only replay path, no CPU fallbacks)
- Strip agent.rs to data types only (TradingState + AgentMetrics)
- Convert DQNAgentType from enum to struct wrapping RegimeConditionalDQN
(Standard variant was never constructed, 43 dead match arms removed)
- Remove dead CPU methods: store_experience, fused_post_step,
fused_post_step_no_ema, adaptive_buffer_resize, refresh_stale_per_priorities
- Add always-on CUDA event per-phase profiling (8 events, 4 phases:
upload/fwd_bwd/adam/per_update) with Drop cleanup and error checking
to identify 329ms/batch H100 bottleneck
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: weight tensors packed sequentially in the flat params buffer
had non-aligned start offsets when preceding tensors had odd element counts
(e.g. bias of 51 atoms = 204 bytes, 204 % 16 = 12). cublasLtMatmul with
CUBLAS_COMPUTE_32F_FAST_TF32 requires 16-byte aligned buffer pointers.
Fix: pad each tensor to 4-element boundary (16 bytes) in both
f32_weight_ptrs_from_base (pointer computation) and compute_total_params
(buffer allocation). Added align4() and padded_byte_offset() helpers,
fixed shrink_perturb skip range and bottleneck gradient offset.
Switched compute type: CUBLAS_COMPUTE_32F → CUBLAS_COMPUTE_32F_FAST_TF32
(forward + backward). Explicit TF32 tensor core path, required by cuBLAS
13.0 on H100 SM90.
Deleted dead bf16_weight_ptrs function.
19/19 smoke tests pass on RTX 3050.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Argo requires task output parameters to flow through DAG arguments →
template inputs, not direct {{tasks.X.outputs}} in container args.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Single train template replaces 7 overlapping workflows. Binary cache
on PVC by commit SHA, fxcache skip-if-exists. Delete 5 dead templates.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
