Platform terraform (live+modules) + k8s manifests moved to fxhnt (now the IaC home).
Rust-only remnants kept: docker build images, gpu-taint.yaml, Rust scripts. foxhunt
being archived; history retained. Live platform IaC now in the fxhnt repo.
- gitea-sshd svc targetPort 22->2222 (rootless gitea binds non-privileged 2222)
- netpol ingress ssh port 22->2222 (NetworkPolicy ports are POD ports, not svc ports)
SSH over ssh://git@git.fxhnt.ai now works (port 22 via proxy socat -> 2222).
- gitea-web/gitea-sshd ClusterIP services: the chart's gitea-http/ssh are HEADLESS
(pod IP 100.64.x) which COLLIDES with the Tailscale CGNAT range 100.64.0.0/10, so the
tailscale-sidecar proxy swallowed the traffic. ClusterIP (10.32.x) fixes it.
- tailscale-proxy: nginx git.fxhnt.ai -> gitea-web:3000; socat :22 -> gitea-sshd:22
(drop :2222, GitLab); GitLab webservice/registry blocks left for Task 4/2C.
- dns git_ip -> 100.95.225.27 (live proxy node); stale 100.90.76.85 duplicate A records
removed across api/chat/dashboard/git/grafana/mail/minio (were causing platform-wide flakiness).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
4 modules migrated off GitLab http backend to s3 (foxhunt-tfstate). State now
external to the cluster + GitLab — prerequisite for GitLab decommission (2C).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
All 4 modules (kapsule/public-gateway/dns/block-storage) migrated to s3 backend
(bucket foxhunt-tfstate, fr-par, use_lockfile=true). plan=No changes verified per
module + final re-plan; all 4 state objects present in bucket; lock smoke test passed
(refresh-only apply, no orphan lockfile). GitLab http state left intact as fallback
(removed in Phase 2C). Also fixed root.hcl path in spec/plan docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The real gateway foxhunt-gw (42653780) was an unmanaged orphan: kapsule state never had a gateway,
public-gateway state referenced a deleted gateway (7a7ffaa5) -> BOTH modules' terragrunt apply would
have created a duplicate gateway (risking cluster egress). Fix:
- kapsule module: removed the legacy gateway/dhcp/gateway_network resources (pre-refactor duplicate,
never in kapsule state; cluster only needs the private_network which stays). kapsule plan now clean.
- public-gateway module (canonical, IPAM mode): state rm the stale 7a7ffaa5 refs + imported the real
gateway/gateway_network/IP (42653780 / 661c8543 / dc419a10); set bastion_enabled=true to match the
live gw. public-gateway plan now 'No changes'. Live gateway untouched throughout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause: global.appConfig.terraformState was missing entirely from the GitLab Helm values, so the
chart rendered terraform_state.enabled=false + object_store disabled -> every /terraform/state API call
403'd ('Object Storage is not enabled for Terraform::StateUploader'), blocking terragrunt. State files
were intact in MinIO (foxhunt-gitlab-artifacts) the whole time. Added terraformState{enabled,bucket,
connection} mirroring artifacts/lfs/packages; helm upgrade -> revision 25 -> TF-state API now 200.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of the stuck cockpit deploy: ci-compile-cpu (32C/64G POP2-HC) hit a fr-par-2 zone stock-out
('Resource POP2-HC-32C-64G is out of stock') -> autoscaler ScaleUpTimedOut -> kaniko build Pending forever.
The big node was sized for the now-dormant Rust cargo compile + precompute_features; the Python fxhnt
cockpit build only needs ~2-6 vCPU. Downsized to POP2-4C-16G (in stock in fr-par-2, far cheaper, scale-to-zero).
Apply blocked in-session: cluster GitLab tokens lack 'api' scope for the TF state backend (403).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The fxhnt-cockpit deploy step applies manifests containing ServiceAccounts (dashboard/dagster), a PVC
(forward track), and a CronJob — none of which the ci-deploy Role permitted, so kubectl apply failed
partway. Also added 'watch' on deployments for 'kubectl rollout status'. Surfaced once the cockpit deploy
finally ran (it previously died at the MinIO log sink first). Rule: this Role must gain a grant for every
new resource KIND a deployed manifest introduces.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the web-dashboard upstream. Proxies peer-to-peer over the tailnet (the kernel-mode proxy can't
reach platform-pool ClusterIPs due to the 100.64/15 pod-CIDR ↔ Tailscale CGNAT overlap, but reaches tailnet
nodes fine). Live CM already patched; this keeps the repo in sync.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Daily 16y all-expiry Databento refetch was ~$6/day (~$190/mo) for the 17% hedge sleeve. Now: crypto_pit
fetched daily (free, the 83% alpha), futures fetched only on Sundays (~$25/mo, within free credits), and
forward-track runs daily (books the combined book's new days). The alpha is validated daily-accurate; the
slow trend hedge weekly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fetch_daily.py: END now clamps to the GLBX dataset range-end (was hardcoded 2026-06-05; clamp handles the
~1-3d exchange lag — END=today failed get_cost). scripts/surfer/fxhnt_forward_cron.sh: daily wrapper —
fetch crypto_pit (free) + futures (Databento, cost-capped) + 'fxhnt forward-track' to book the new paper-
NAV day of the validated combined book (crypto X-sec momentum alpha + futures trend hedge). Crontab: daily
23:30 UTC -> data/surfer/fxhnt_forward.log. Sources DATABENTO key from ~/.secrets (never printed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full drawdown profile of the deployable book (depth/frequency/duration/recovery/ulcer) vs 60/40.
Analysis tool, not part of the deployed pipeline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(1) IBKR paper reports summary fields (cash/GrossPositionValue) inconsistent with actual positions
(found: $141k/13% gap, fills+positions are ground truth). Bot now cross-checks NLV vs cash+position-
market-value, WARNs on >2% gap, sizes on the CONSERVATIVE NLV (never inflates), and a gate refuses to
trade if the gap exceeds MULTISTRAT_DATA_TOL (default 25%).
(2) build() included today's INCOMPLETE intraday bar when run mid-session -> spiked vol estimate ->
halved the leverage (1.0->0.5) -> would have sold half the book as churn at the 14:35 cron. Now drops
today's partial bar; uses completed closes only (matches the daily-close backtest). Verified live:
leverage back to 1.0, book recognized on-target (in_band), no spurious orders.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Works exactly like multistrat_paper.py (daily cron, status/run/weights, idempotent, forward-start,
catch-up) but tracks the GD strategy: 70% QQQ + 30% multi-strat book, continuously rebalanced. NAV
simulation (no orders) running alongside the live IBKR-paper book to compare forward: GD = higher
CAGR, deeper drawdown by design. Cron 12/15/18 UTC. GD_EQUITY_WEIGHT configurable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
User's '10x boring' idea tested. GD-70 (70%QQQ/15%trend/15%book) has highest Sharpe-excess (0.67-0.84
vs QQQ 0.65, book 0.39); trend sleeve = genuine crisis-alpha. GD@1.3x cheap-lev ~= QQQ return with
shallower drawdown (-48 vs -53%). VRP(PUTW) does NOT add (dilutes); carry no clean retail ETF. Clean
stackable premia = equity-core + trend-sleeve + execution-alpha (behavior gap/tax/cheap-lev, on top,
not in backtest). GD genuinely beats both the over-conservative book and panic-prone raw QQQ.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch from weekly to daily (weekday) evaluation for faster risk response — the leverage (vol-target
+ drawdown CB + corr de-risk) re-evaluates each day so a vol spike de-levers within a day vs a week.
Hysteresis (3% NLV) still gates actual trades, so turnover stays low (in-band days = no-ops). No
return impact; tighter risk only. Weekends excluded (US market closed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Eyes-on EXECUTE=true validation caught two bugs: (1) fractional qty -> all 6 orders cancelled (10243);
round to whole shares (skip if 0). (2) last_rebalance was set even when all orders were rejected,
which would block retry; now only marked when >=1 order is accepted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After full validation (in-cluster connect via 4004, paper account DU9600528, gates pass, clean logs
post NaN-fix), arm the CronJob to place paper orders. Triple-guarded against real money: paper
account + Gateway paper-mode + bot refuses non-DU without MULTISTRAT_ALLOW_LIVE_CONFIRMED.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
np.corrcoef on a window where a vol-normalized stream is constant (warmup / non-moving stream)
divided by zero std -> RuntimeWarning + NaN in the corr matrix -> ac=NaN -> propagated via chist
into the z-score and could poison leverage L and book with NaN (silent corruption). Fix: correlate
only streams with std>0 in the window, nansum over the actual valid count, fall back to ac=0 when
<2 vary. Verified: no RuntimeWarning (tested with warnings-as-errors), no NaN, identical L/weights.
Affects all scripts importing book_series + the cluster ConfigMap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pod-to-pod on 4002 reaches IB Gateway with the pod IP as source -> not in the trusted-IP allowlist
(default 127.0.0.1) -> TCP connects but API handshake times out. The gnzsnz image's socat on 4004
relays via localhost so IB Gateway trusts it. (Local port-forward worked on 4002 because it enters
the pod netns = localhost.) Egress netpol now allows 4002+4004; IB_PORT=4004.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Weekly in-cluster CronJob running multistrat_bot.py on IBKR paper via the existing ib-gateway
(connects to ib-gateway:4002 directly, no port-forward). Lean: code via ConfigMap (no image build),
python:3.12-slim + runtime pip. Includes PVC (state/HWM persistence across runs), egress
NetworkPolicy (ib-gateway:4002 + 443 for pip/Yahoo + DNS), and ib-gateway ingress patched to allow
the rebalancer. Schedule Mon 14:35 UTC (~1h after US open). MULTISTRAT_EXECUTE=false by default
(cluster dry-run); flip to true after validating logs. Drawdown CB + leverage cap + idempotency +
paper-guard enforced by the bot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
'mini' mode: confirms account is paper (DU prefix; aborts on live U-prefix), places BUY 1 SPY,
reports fill/routing. Validated against cluster ib-gateway: account DU9600528 [PAPER], order routed
(PreSubmitted, queued for next open — market closed on weekend). Confirms the full chain works:
cluster Gateway -> port-forward 4002 -> bot -> paper account.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
