foxhunt/docs/superpowers/specs/2026-03-21-training-profiles-design.md

# Training Profile Configuration System

## Goal

Replace hardcoded hyperparameters across DQN, PPO, and supervised trainers with TOML configuration profiles following the same pattern as the existing GPU profiles (`config/gpu/*.toml`). All training behavior is driven by config, not code.

## Architecture

Per-model-type TOML profiles with 3-tier loading (env var > filesystem > embedded defaults). CLI args always override TOML values. GPU profile hardware constraints (batch_size, num_atoms) take precedence over training profile values.

## Directory Structure

```
config/training/
├── dqn-production.toml
├── dqn-smoketest.toml
├── dqn-hyperopt.toml
├── ppo-production.toml
├── ppo-smoketest.toml
├── supervised-production.toml
├── supervised-smoketest.toml
└── walk-forward.toml
```

## Loading Priority

1. `$FOXHUNT_TRAINING_PROFILE` env var (exact file path, highest priority)
2. `config/training/<profile>.toml` relative to workspace root
3. Embedded compile-time defaults via `include_str!`

CLI args always override TOML. TOML overrides struct defaults.

## TOML Schema

### DQN Profile (`dqn-production.toml`)

```toml
[training]
epochs = 100
batch_size = 1024
learning_rate = 0.0003
gamma = 0.99
tau = 0.005
warmup_steps = 1000
gradient_clip_norm = 1.0
weight_decay = 0.0001
max_steps_per_epoch = 2000

[exploration]
epsilon_start = 1.0
epsilon_end = 0.01
epsilon_decay = 0.995

[replay_buffer]
buffer_size = 500000
min_replay_size = 1000
use_per = true
per_alpha = 0.6
per_beta_start = 0.4

[distributional]
use_distributional = true
num_atoms = 51
v_min = -25.0
v_max = 25.0

[branching]
use_branching = true
branch_0_size = 5
branch_1_size = 3
branch_2_size = 3

[advanced]
use_double_dqn = true
use_dueling = true
use_noisy_nets = true
noisy_sigma_init = 0.5
use_cql = true
cql_alpha = 0.1
gradient_accumulation_steps = 1

[risk]
enable_kelly_sizing = true
kelly_fractional = 0.5
kelly_max_fraction = 0.25
enable_action_masking = true
max_position_absolute = 2.0
enable_circuit_breaker = true

[early_stopping]
enabled = true
patience = 20
min_epochs_before_stopping = 10
min_loss_improvement_pct = 0.1

[experience]
initial_capital = 100000.0
tx_cost_multiplier = 0.0001
hold_reward = 0.001
```

### DQN Smoke Test Profile (`dqn-smoketest.toml`)

```toml
[training]
epochs = 1
batch_size = 16
learning_rate = 0.001
warmup_steps = 0
max_steps_per_epoch = 50

[replay_buffer]
buffer_size = 1024
min_replay_size = 32

[early_stopping]
enabled = false

[experience]
gpu_n_episodes = 2
gpu_timesteps_per_episode = 50
```

### Walk-Forward Profile (`walk-forward.toml`)

```toml
[walk_forward]
train_months = 12
val_months = 3
test_months = 3
step_months = 3
expanding_window = true
```

### Hyperopt Profile (`dqn-hyperopt.toml`)

```toml
[search_space]
learning_rate = [0.00001, 0.01]
gamma = [0.95, 0.999]
tau = [0.001, 0.05]
batch_size = [32, 512]
buffer_size = [10000, 500000]
per_alpha = [0.4, 0.8]
per_beta_start = [0.2, 0.6]
epsilon_decay = [0.99, 0.999]
cql_alpha = [0.01, 0.5]
gradient_accumulation_steps = [1, 4]

[fixed]
use_branching = true
use_double_dqn = true
use_per = true
use_distributional = true

[pso]
swarm_size = 20
max_iterations = 50
inertia = 0.7
cognitive = 1.5
social = 1.5
```

## Rust Implementation

### Profile Loader (`crates/ml/src/training_profile.rs`)

```rust
use serde::Deserialize;

#[derive(Debug, Clone, Deserialize, Default)]
pub struct DqnTrainingProfile {
    pub training: Option<TrainingSection>,
    pub exploration: Option<ExplorationSection>,
    pub replay_buffer: Option<ReplayBufferSection>,
    pub distributional: Option<DistributionalSection>,
    pub branching: Option<BranchingSection>,
    pub advanced: Option<AdvancedSection>,
    pub risk: Option<RiskSection>,
    pub early_stopping: Option<EarlyStoppingSection>,
    pub experience: Option<ExperienceSection>,
}

impl DqnTrainingProfile {
    pub fn load(profile_name: &str) -> Self { ... }
    pub fn apply_to(&self, hp: &mut DQNHyperparameters) { ... }
}
```

Every field in every section is `Option<T>` — unset fields keep the struct default. `apply_to()` applies only the `Some` values, leaving the rest untouched. This enables sparse profiles (smoketest only overrides 8 fields, production sets 40+).

### Integration Points

1. **`train_baseline_rl.rs`**: Add `--training-profile` CLI arg. Load profile, apply to hyperparams, then apply CLI overrides on top.

2. **Smoke test helpers**: `smoke_params()` loads `dqn-smoketest.toml` instead of hardcoding values.

3. **Hyperopt adapters**: Load `dqn-hyperopt.toml` for search space bounds instead of hardcoded arrays.

4. **K8s job template**: Add `TRAINING_PROFILE` env var alongside `TRAINING_BINARY`.

### Merge Priority (highest wins)

```
CLI args > Training profile TOML > GPU profile TOML > Struct defaults
```

GPU profile `batch_size` and `num_atoms` cap the training profile values (hardware constraint).

## Scope

### In scope
- DQN training profiles (production, smoketest, hyperopt)
- PPO training profiles (production, smoketest)
- Supervised training profile (shared across 8 models)
- Walk-forward profile
- Profile loader with 3-tier resolution
- CLI `--training-profile` arg
- Smoke test helper migration to TOML
- K8s job template update

### Also in scope
- **PVC deployment**: Training profile TOML files uploaded to the training-data PVC so K8s jobs can read them. Update `populate-test-data-job.yaml` to copy `config/training/*.toml` to `/data/config/training/`.
- **Full CLI coverage**: Every field in the TOML profiles must also be settable via CLI arg on `train_baseline_rl` and `train_baseline_supervised`. No config-only or CLI-only fields — all values accessible from both paths.
- **`fxt` CLI integration**: `fxt train` subcommand accepts `--training-profile` to select profile by name.

### Out of scope
- Per-symbol profiles (ES.FUT vs NQ.FUT — same config, different data)
- Runtime profile switching (profiles are loaded once at startup)
- Web UI for profile editing
- Profile versioning / migration