foxhunt/docs/plans/2026-03-01-infra-security-hardening-design.md

Infrastructure Security Hardening — Design

Date: 2026-03-01 Scope: Minimal hardening for production deployment prep Approach: No new infrastructure dependencies (no ESO, no cert-manager, no service mesh)

Problem

Infrastructure audit found 4 critical gaps:

• Zero NetworkPolicy files — all pods communicate freely
• No securityContext on main service containers — run as root
• Single monolithic secret (foxhunt-secrets) shared by all services
• No container image vulnerability scanning in CI

Application-level security is strong (mTLS, JWT, RBAC, AES-256-GCM, rate limiting). This plan addresses infrastructure-layer gaps only.

1. NetworkPolicy — Default-Deny + Explicit Allow

Create infra/k8s/network-policies/ directory.

1a. Default deny-all (default-deny.yaml)

Deny all ingress and egress for namespace foxhunt. Forces every service to declare what it needs.

1b. Per-service allow rules (one file per service)

Service	Ingress From	Egress To
api-gateway	Tailscale sidecar (any pod with app: tailscale)	postgres:5432, redis:6379, web-gateway:3000
web-gateway	api-gateway:3000	api-gateway:50051 (gRPC upstream)
trading-service	api-gateway, trading-agent	postgres, redis, ib-gateway:4002
backtesting-service	api-gateway	postgres, redis, minio:9000
ml-training-service	api-gateway	postgres, minio:9000, gpu nodes
trading-agent-service	api-gateway	trading-service, postgres
broker-gateway	trading-service	ib-gateway:4002, postgres
data-acquisition-service	api-gateway	postgres, external HTTPS (443)
ib-gateway	broker-gateway, trading-service	external (IBKR servers)

1c. Monitoring exemption

Prometheus scrape allowed from monitoring namespace to all metrics ports (909x).

1d. DNS exemption

All pods get egress to kube-system on port 53 (CoreDNS).

All policies use podSelector with app labels already present on deployments.

2. SecurityContext — Pod Hardening

2a. Pod-level securityContext (every deployment)

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault

2b. Main container securityContext

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]

2c. InitContainer stays privileged

runAsUser: 0 kept for fetch-binary initContainer (needs to write to emptyDir).

2d. Writable tmpfs

Add emptyDir mount at /tmp for services needing temp writes.

2e. Exceptions

• ib-gateway: Third-party Java image, skip readOnlyRootFilesystem.
• Tailscale sidecar: Keeps NET_ADMIN, NET_RAW (required for VPN).

Applies to all 9 service deployments + both GPU overlay variants.

3. Secret Scoping — Split Monolithic Secret

3a. Per-concern secrets

Secret Name	Keys	Used By
db-credentials	password	api-gw, trading, trading-agent, backtesting, broker-gw, data-acq, ml-training
jwt-secret	secret	api-gw, web-gw, trading, trading-agent, backtesting, broker-gw
redis-credentials	password	api-gw, trading
s3-credentials	access-key, secret-key	ml-training, backtesting
ibkr-credentials	(unchanged)	trading, broker-gw, ib-gateway
ml-training-tls	(unchanged)	ml-training
minio-credentials	(unchanged)	all (initContainer fetch-binary)

3b. Update secretKeyRef

Each deployment updated to reference new secret names instead of foxhunt-secrets.

3c. Remove placeholder values

Secret manifests contain REPLACE_IN_DEPLOY markers. Real values injected by deploy script.

3d. deploy-secrets.sh

Script reads from Scaleway Secrets Manager CLI and creates K8s secrets. Replaces manually editing YAML with placeholder values.

4. CI Image Scanning — Trivy

4a. New scan stage

Between prepare and test in .gitlab-ci.yml.

4b. Scope

Scan 2 runtime images only: foxhunt-runtime, foxhunt-training-runtime. CI builder images are internal-only.

4c. Fail policy

--exit-code 1 on CRITICAL and HIGH with --ignore-unfixed. Pipeline blocks on fixable critical/high CVEs.

4d. Cache

Trivy DB cache persisted via CI cache key.

Risk

Low-medium. NetworkPolicy and securityContext changes could break connectivity or file access if misconfigured. Mitigation: deploy one service at a time, verify health checks pass before proceeding.

Secret scoping is a rename operation — same values, different secret objects. CI scanning is additive (new stage, no existing stages modified).