The GPU runners used runtime_class_name=nvidia for CUDA access but
didn't request nvidia.com/gpu as a K8s resource, allowing multiple
training pods to share a GPU without K8s awareness. Add pod_spec
patches to request GPU resources properly so K8s enforces mutual
exclusion between GitLab CI and Argo training jobs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
On Ampere+ GPUs the training dtype is BF16. Network forward passes
with BF16 inputs return BF16 tensors, but all loss-path arithmetic
(rewards, dones, gamma, PER weights, Huber constants, atoms, clip
epsilon) was created as F32 from Tensor::from_vec. Candle forbids
mixed-dtype binary ops, causing "dtype mismatch in {mul,sub,add}"
on every training step.
DQN fixes (7 mismatch sites):
- Cast current_q_values to F32 immediately after forward pass
- Cast all target network outputs (standard, dueling, C51, IQN) to F32
- Keep rewards/dones/gamma/weights/atoms/half/delta as F32 (remove
.to_dtype(training_dtype) casts — now use DType::F32 sentinel)
- Entropy regularization and CQL paths now match (F32 + F32)
PPO fixes (3 mismatch sites):
- Use DType::F32 for clip epsilon one_tensor (was training_dtype BF16)
- Cast critic.forward() output to F32 in both MLP and LSTM value loss
- Cast target_returns to F32 for symlog/normalization path
Infra: revert runner-h100 from SXM2 (zero Scaleway quota) back to
ci-training-h100 PCIe pool.
2704 ml lib tests pass, 260 PPO tests pass, 0 clippy warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
BUG #41 kept forward pass in F32 for autograd, but the target-side
tensors (reward, gamma, done, next_q) were cast to BF16 via `dtype`.
The `state_action_values.sub(&target_q_values)` then hit F32-vs-BF16
mismatch on Ampere+ GPUs, causing every training step to fail silently.
Fix: `.to_dtype(state_action_values.dtype())` on the detached target.
Safe because target is detached (no autograd graph to break).
Also: H100 runner → SXM2 pool, GPU availability checker script.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create ci-training-h100 runner (runner-h100-values.yaml)
- Route hyperopt-dqn and hyperopt-ppo to H100 via kapsule,h100 tags
- H100 is ~2x faster and cheaper per run vs L40S for hyperopt workloads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
