foxhunt/docs/superpowers/specs/2026-04-16-homeostatic-regularization-design.md

Homeostatic Regularization — Design Spec

Goal: Unified self-regulating framework that maintains healthy training dynamics by adapting penalty strengths based on deviation from homeostatic targets. Replaces ad-hoc fixed lambdas (Q-mean drift=0.01, branch independence=0.01, temporal consistency=0.005) with a single principled mechanism borrowed from biological neural homeostasis.

Problem: Multiple training observables can drift or collapse (Q-mean, atom utilization, branch correlation, trade frequency, Q-gap). Each has its own fixed lambda that was hand-tuned. When one observable drifts faster than expected (Q-mean reached +4.89 by epoch 10 with lambda=0.01), the fixed penalty is too weak. When it's stable, the penalty wastes gradient budget. No coordinated response — each penalty operates independently.

Architecture: One CUDA kernel homeostatic_regularizer reads N observable scalars from pinned device-mapped memory, compares to N adaptive targets, produces N penalty gradients scaled by deviation². Targets self-calibrate from the first 5 epochs' healthy range. Penalties coordinate — if multiple observables drift simultaneously, total penalty budget is capped to prevent gradient domination.

---

The Mechanism

Biological neurons maintain homeostasis through intrinsic plasticity: fire too much → excitability decreases. Fire too little → excitability increases. The system converges to a target firing rate without external tuning.

Applied to DQN training:

For each observable k (Q-mean, atom_util, branch_corr, trade_freq, Q-gap):
    error_k = (observed_k - target_k) / max(|target_k|, eps)   // normalized error
    penalty_k = lambda_base * error_k^2 * sign(error_k)        // quadratic, signed
    
    // Adaptive target: EMA of healthy range (epochs 1-5)
    if epoch <= 5:
        target_k = EMA(observed_k, alpha=0.3)    // calibrate
    // else: target frozen, penalties active
    
    // Budget cap: total penalty gradient <= 10% of C51 gradient
    total_penalty = sum(|penalty_k|)
    if total_penalty > budget:
        scale = budget / total_penalty
        penalty_k *= scale

Observables and Targets

Observable	Target	Why	Source
Q-mean	0.0	Prevents bootstrapping drift	q_mean_scratch_pinned
Atom utilization	0.85	Keeps C51 resolution healthy	q_stats_buf[6]
Max branch correlation	0.1	Forces orthogonal branch features	branch_indep_penalty_buf
Trade frequency	calibrated	Prevents churn (too high) or inactivity (too low)	GPU epoch summary
Q-gap	calibrated	Prevents collapse (too low) or explosion (too high)	per_branch_q_gaps
Q-variance	calibrated	Prevents overconfidence (too low) or chaos (too high)	q_stats_buf[4]

Implementation

CUDA Kernel

extern "C" __global__ void homeostatic_regularizer(
    const float* __restrict__ observables,   /* [N_OBS] current values (pinned) */
    const float* __restrict__ targets,       /* [N_OBS] homeostatic targets (pinned) */
    float* __restrict__ penalties,           /* [N_OBS] output penalty gradients */
    float* __restrict__ total_penalty,       /* [1] sum of |penalty_k| for budget cap */
    int n_obs,
    float lambda_base,                       /* 0.01 */
    float budget_fraction                    /* 0.10 = max 10% of C51 gradient */
) {
    // Single-block kernel (n_obs <= 8)
    int k = threadIdx.x;
    if (k >= n_obs) return;
    
    float obs = observables[k];
    float tgt = targets[k];
    float denom = fmaxf(fabsf(tgt), 1e-6f);
    float error = (obs - tgt) / denom;
    
    // Quadratic signed penalty
    float pen = lambda_base * error * fabsf(error);
    penalties[k] = pen;
    
    // Atomic sum for budget cap
    atomicAdd(total_penalty, fabsf(pen));
}

Rust Integration

struct HomeostaticState {
    observables_pinned: *mut f32,    // [N_OBS] pinned device-mapped
    observables_dev_ptr: u64,
    targets_pinned: *mut f32,        // [N_OBS] pinned device-mapped  
    targets_dev_ptr: u64,
    penalties_buf: CudaSlice<f32>,   // [N_OBS]
    total_penalty_buf: CudaSlice<f32>, // [1]
    kernel: CudaFunction,
    calibration_epoch: usize,        // epoch at which targets freeze (5)
    n_obs: usize,                    // number of observables (6)
}

Observable Update (every training step)

fn update_observables(&self) {
    unsafe {
        let obs = self.observables_pinned;
        *obs.add(0) = *self.q_mean_scratch_pinned;      // Q-mean
        *obs.add(1) = self.utilization_ema;               // atom util
        *obs.add(2) = 0.0; // branch corr (read from penalty buf after compute)
        *obs.add(3) = self.epoch_trade_count as f32;     // trade freq
        *obs.add(4) = self.last_q_gap;                   // Q-gap
        *obs.add(5) = self.last_q_var;                   // Q-variance
    }
}

Target Calibration (epochs 1-5)

fn calibrate_targets(&self, epoch: usize) {
    if epoch > self.calibration_epoch { return; }
    let alpha = 0.3_f32;
    unsafe {
        for k in 0..self.n_obs {
            let obs = *self.observables_pinned.add(k);
            let old = *self.targets_pinned.add(k);
            *self.targets_pinned.add(k) = (1.0 - alpha) * old + alpha * obs;
        }
    }
    // Override Q-mean target to always be 0.0 (no drift is always the goal)
    unsafe { *self.targets_pinned.add(0) = 0.0; }
}

Penalty Application

The penalties feed into the C51 gradient (same path as the current Q-mean drift penalty). The homeostatic_regularizer kernel runs after Q-stats, and the penalty for observable 0 (Q-mean) replaces the current hardcoded drift penalty in c51_grad_kernel.cu.

For branch correlation (observable 2) and Q-gap (observable 4), the penalties feed into the respective auxiliary loss accumulator buffers.

Interaction with Existing Components

• Replaces: Q-mean drift penalty (G9d), fixed branch independence lambda (G6), fixed temporal consistency lambda (G10)
• Coordinates with: gamma annealing (G4 uses atom_util, homeostatic also targets atom_util — the two mechanisms reinforce each other)
• Enhances: cost curriculum (G2) — the sigmoid center (currently hardcoded epoch 10) should be driven by model readiness: center = epoch where training_Sharpe_ema first exceeds 0.3. If the model finds edges quickly, costs ramp sooner. If it struggles, costs stay low longer. This replaces the hardcoded epoch 10 with an adaptive trigger.

Success Criteria

Metric	Current (fixed lambda)	Target (homeostatic)
Q-mean at epoch 10	+4.89 (drifting)	< ±1.0 (contained)
Atom util at epoch 10	87% (healthy)	> 85% (maintained)
Training Sharpe oscillation	±0.3	< ±0.15 (damped)
Number of hand-tuned lambdas	5+	1 (lambda_base)

Risks

Risk	Mitigation
Calibration period too short	5 epochs × 88s = 7 min. Sufficient for healthy range. Extend to 8 if needed.
Budget cap too aggressive	10% of C51 gradient. Increase to 20% if penalties never fire.
Targets miscalibrated from bad early epochs	Q-mean target hardcoded to 0.0. Others have sensible fallbacks.
Penalty coordination masks individual issues	Log each penalty_k separately for diagnosis.