19f6cce510
Phase 3.5.5 (69b8fdb61) landed dd_trajectory_kernel writing
ISV[DD_TRAJECTORY_DECREASING_INDEX=439] per-step but DEFERRED the PER
sampler integration AND the production launch. Wave 4.3 wires both
atomically per feedback_wire_everything_up.
Investigation finding — Case B (existing TD-error-driven priority
sampler): the GPU PER architecture in
crates/ml-dqn/src/gpu_replay_buffer.rs already weights replay by
per-transition priority (raw priority + priorities_pa = priority^alpha
feeding the prefix-sum sampler). Recovery boost slots in cleanly as a
multiplier on priorities[idx] at insert time — no architectural
refactor, no new sampling-path machinery.
Recovery-oversample formula:
effective = base_priority × (1.0 + ISV[440] × ISV[439])
priorities[idx] = effective
priorities_pa[idx] = effective^alpha
When a transition is in a recovery (DD shrinking from non-trivial DD,
ISV[439]=1.0), it lands in the buffer at 1 + ω(2.0) × δ(1.0) = 3.0×
the baseline max_priority, then sampled 3.0^0.6 ≈ 1.93× more often
than baseline (alpha=0.6 PER convention) until per_update_pa
overwrites priority based on TD-error and normal PER takes over.
Recovery transitions get amplified gradient signal — the agent learns
recovery dynamics over typical "average-DD" Bellman noise.
Atomic landings (per feedback_no_partial_refactor):
* crates/ml-dqn/src/per_kernels.cu — per_insert_pa kernel signature
extended (+priorities, +isv pointers); kernel writes both columns
of the (priority, priority^alpha) row pair; nullable ISV ptr falls
back to boost_factor=1.0
* crates/ml-dqn/src/gpu_replay_buffer.rs — new isv_signals_dev_ptr
field + setter + accessor + priorities_pa_slice accessor; redundant
pre-3.5.5.b scatter_insert_f32 broadcast of max_priority REMOVED
(now subsumed by per_insert_pa's priorities[idx]=effective store)
* crates/ml/src/cuda_pipeline/gpu_experience_collector.rs — new
sp15_dd_trajectory_prev_dd_dev_ptr field + setter; per-step
launch_sp15_dd_trajectory_decreasing invocation gated on both
ISV ptr + prev_dd ptr being non-zero, placed immediately after
launch_sp15_dd_state in the env-step loop
* crates/ml/src/trainers/dqn/trainer/training_loop.rs — two new
wiring blocks for the collector's prev_dd ptr and the replay
buffer's ISV ptr, mirroring the existing
set_isv_signals_ptr / set_sp15_alpha_warm_count_ptr plumbing
* crates/ml/tests/sp15_phase1_oracle_tests.rs — new oracle test
per_sampler_weights_recovery_transitions_higher verifying both
columns of the priority-buffer write equal exactly the
boosted/baseline values (3.0 vs 1.0; 3.0^0.6 vs 1.0^0.6) and the
boosted/baseline ratio = 3.0 within 1e-5 — the recovery oversample
factor by construction
Phase 3.5.5 deferred consumer eliminated per
feedback_wire_everything_up. This closes the SP15 Phase 3.5
recovery-dynamics chain (3.5.2 + 3.5.3 + 3.5.4 + 3.5.4.b + 3.5.5 +
3.5.5.b all wired). DD_TRAJECTORY_FLOOR (slot 441) and ISV-driven
RECOVERY_OVERSAMPLE_WEIGHT (slot 440) producers remain documented
follow-ups per feedback_isv_for_adaptive_bounds — kernel reads from
slots rather than literals so consumer migration is a no-op when the
producers land.
Verified on RTX 3050 Ti (CUDA 12.9, sm_86):
* cargo check -p ml-dqn / -p ml --features cuda clean
* 3 of 3 dd_trajectory oracle tests still green
* 1 of 1 new per_sampler oracle test green (3.0 vs 1.0 priority
bias, ratio = 3.000... within 1e-5)
* 8 of 8 (1 ignored) gpu_residency replay-buffer + adamw smoke
tests green
* 4 of 4 PER smoke tests green
* Full ml lib suite IMPROVES Wave 4.2 baseline: 947 pass / 12 fail
(was 946 pass / 13 fail; the previously-failing
test_dqn_checkpoint_round_trip now passes — likely the redundant
pre-3.5.5.b scatter_insert_f32 was racing with the immediate
per_insert_pa overwrite under particular timing conditions; the
merged single-kernel write closes that race)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ml
10-model ML ensemble for the Foxhunt HFT system, built on Candle v0.9.1.
Models
- DQN (Rainbow) — deep Q-network with prioritized replay, dueling heads, noisy nets
- PPO — proximal policy optimization with GAE, LSTM policies, clip-higher
- TFT — temporal fusion transformer for multi-horizon forecasting
- Mamba2 — state space model for sequence prediction
- Liquid Networks — biologically inspired networks for non-stationary data
- TLOB — transformer-based limit order book analysis
- KAN — Kolmogorov-Arnold networks
- xLSTM — extended LSTM architecture
- TGGN — temporal graph neural network
- Diffusion — diffusion-based generative model
Key Modules
ensemble— model ensemble coordination and confidence aggregationhyperopt— PSO-based hyperparameter optimization with per-model adapterstrainers— unified training loops (DQN, PPO, supervised)inference—InferenceAdaptertrait for predictioncheckpoint— model checkpointing and restorationevaluation— walk-forward evaluation pipeline
Usage
use ml::dqn::DQN;
use ml::ppo::PpoTrainer;