docs: add infrastructure security hardening design

Minimal hardening plan for production deployment: NetworkPolicy
(default-deny + explicit allow), SecurityContext on all pods,
secret scoping (split monolithic foxhunt-secrets), and Trivy
image scanning in CI pipeline.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

docs/plans/2026-03-01-infra-security-hardening-design.md

@@ -0,0 +1,144 @@
+# Infrastructure Security Hardening — Design
+
+**Date**: 2026-03-01
+**Scope**: Minimal hardening for production deployment prep
+**Approach**: No new infrastructure dependencies (no ESO, no cert-manager, no service mesh)
+
+## Problem
+
+Infrastructure audit found 4 critical gaps:
+- Zero NetworkPolicy files — all pods communicate freely
+- No securityContext on main service containers — run as root
+- Single monolithic secret (foxhunt-secrets) shared by all services
+- No container image vulnerability scanning in CI
+
+Application-level security is strong (mTLS, JWT, RBAC, AES-256-GCM, rate limiting).
+This plan addresses infrastructure-layer gaps only.
+
+## 1. NetworkPolicy — Default-Deny + Explicit Allow
+
+Create `infra/k8s/network-policies/` directory.
+
+### 1a. Default deny-all (`default-deny.yaml`)
+
+Deny all ingress and egress for namespace `foxhunt`. Forces every service to
+declare what it needs.
+
+### 1b. Per-service allow rules (one file per service)
+
+| Service | Ingress From | Egress To |
+|---------|-------------|-----------|
+| api-gateway | Tailscale sidecar (any pod with `app: tailscale`) | postgres:5432, redis:6379, web-gateway:3000 |
+| web-gateway | api-gateway:3000 | api-gateway:50051 (gRPC upstream) |
+| trading-service | api-gateway, trading-agent | postgres, redis, ib-gateway:4002 |
+| backtesting-service | api-gateway | postgres, redis, minio:9000 |
+| ml-training-service | api-gateway | postgres, minio:9000, gpu nodes |
+| trading-agent-service | api-gateway | trading-service, postgres |
+| broker-gateway | trading-service | ib-gateway:4002, postgres |
+| data-acquisition-service | api-gateway | postgres, external HTTPS (443) |
+| ib-gateway | broker-gateway, trading-service | external (IBKR servers) |
+
+### 1c. Monitoring exemption
+
+Prometheus scrape allowed from `monitoring` namespace to all metrics ports (909x).
+
+### 1d. DNS exemption
+
+All pods get egress to `kube-system` on port 53 (CoreDNS).
+
+All policies use `podSelector` with `app` labels already present on deployments.
+
+## 2. SecurityContext — Pod Hardening
+
+### 2a. Pod-level securityContext (every deployment)
+
+```yaml
+securityContext:
+  runAsNonRoot: true
+  runAsUser: 1000
+  runAsGroup: 1000
+  fsGroup: 1000
+  seccompProfile:
+    type: RuntimeDefault
+```
+
+### 2b. Main container securityContext
+
+```yaml
+securityContext:
+  allowPrivilegeEscalation: false
+  readOnlyRootFilesystem: true
+  capabilities:
+    drop: ["ALL"]
+```
+
+### 2c. InitContainer stays privileged
+
+`runAsUser: 0` kept for `fetch-binary` initContainer (needs to write to emptyDir).
+
+### 2d. Writable tmpfs
+
+Add `emptyDir` mount at `/tmp` for services needing temp writes.
+
+### 2e. Exceptions
+
+- **ib-gateway**: Third-party Java image, skip `readOnlyRootFilesystem`.
+- **Tailscale sidecar**: Keeps `NET_ADMIN`, `NET_RAW` (required for VPN).
+
+Applies to all 9 service deployments + both GPU overlay variants.
+
+## 3. Secret Scoping — Split Monolithic Secret
+
+### 3a. Per-concern secrets
+
+| Secret Name | Keys | Used By |
+|------------|------|---------|
+| `db-credentials` | `password` | api-gw, trading, trading-agent, backtesting, broker-gw, data-acq, ml-training |
+| `jwt-secret` | `secret` | api-gw, web-gw, trading, trading-agent, backtesting, broker-gw |
+| `redis-credentials` | `password` | api-gw, trading |
+| `s3-credentials` | `access-key`, `secret-key` | ml-training, backtesting |
+| `ibkr-credentials` | (unchanged) | trading, broker-gw, ib-gateway |
+| `ml-training-tls` | (unchanged) | ml-training |
+| `minio-credentials` | (unchanged) | all (initContainer fetch-binary) |
+
+### 3b. Update secretKeyRef
+
+Each deployment updated to reference new secret names instead of foxhunt-secrets.
+
+### 3c. Remove placeholder values
+
+Secret manifests contain `REPLACE_IN_DEPLOY` markers. Real values injected by deploy script.
+
+### 3d. deploy-secrets.sh
+
+Script reads from Scaleway Secrets Manager CLI and creates K8s secrets.
+Replaces manually editing YAML with placeholder values.
+
+## 4. CI Image Scanning — Trivy
+
+### 4a. New `scan` stage
+
+Between `prepare` and `test` in `.gitlab-ci.yml`.
+
+### 4b. Scope
+
+Scan 2 runtime images only: foxhunt-runtime, foxhunt-training-runtime.
+CI builder images are internal-only.
+
+### 4c. Fail policy
+
+`--exit-code 1` on CRITICAL and HIGH with `--ignore-unfixed`.
+Pipeline blocks on fixable critical/high CVEs.
+
+### 4d. Cache
+
+Trivy DB cache persisted via CI cache key.
+
+## Risk
+
+Low-medium. NetworkPolicy and securityContext changes could break connectivity
+or file access if misconfigured. Mitigation: deploy one service at a time,
+verify health checks pass before proceeding.
+
+Secret scoping is a rename operation — same values, different secret objects.
+CI scanning is additive (new stage, no existing stages modified).