From a5e0558e08ff5ac9c5629dbb2c9eb3fe36abc26a Mon Sep 17 00:00:00 2001 From: jgrusewski Date: Sun, 1 Mar 2026 23:13:13 +0100 Subject: [PATCH] docs: add infrastructure security hardening design Minimal hardening plan for production deployment: NetworkPolicy (default-deny + explicit allow), SecurityContext on all pods, secret scoping (split monolithic foxhunt-secrets), and Trivy image scanning in CI pipeline. Co-Authored-By: Claude Opus 4.6 --- ...6-03-01-infra-security-hardening-design.md | 144 ++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 docs/plans/2026-03-01-infra-security-hardening-design.md diff --git a/docs/plans/2026-03-01-infra-security-hardening-design.md b/docs/plans/2026-03-01-infra-security-hardening-design.md new file mode 100644 index 000000000..1c4f20a41 --- /dev/null +++ b/docs/plans/2026-03-01-infra-security-hardening-design.md @@ -0,0 +1,144 @@ +# Infrastructure Security Hardening — Design + +**Date**: 2026-03-01 +**Scope**: Minimal hardening for production deployment prep +**Approach**: No new infrastructure dependencies (no ESO, no cert-manager, no service mesh) + +## Problem + +Infrastructure audit found 4 critical gaps: +- Zero NetworkPolicy files — all pods communicate freely +- No securityContext on main service containers — run as root +- Single monolithic secret (foxhunt-secrets) shared by all services +- No container image vulnerability scanning in CI + +Application-level security is strong (mTLS, JWT, RBAC, AES-256-GCM, rate limiting). +This plan addresses infrastructure-layer gaps only. + +## 1. NetworkPolicy — Default-Deny + Explicit Allow + +Create `infra/k8s/network-policies/` directory. + +### 1a. Default deny-all (`default-deny.yaml`) + +Deny all ingress and egress for namespace `foxhunt`. Forces every service to +declare what it needs. + +### 1b. Per-service allow rules (one file per service) + +| Service | Ingress From | Egress To | +|---------|-------------|-----------| +| api-gateway | Tailscale sidecar (any pod with `app: tailscale`) | postgres:5432, redis:6379, web-gateway:3000 | +| web-gateway | api-gateway:3000 | api-gateway:50051 (gRPC upstream) | +| trading-service | api-gateway, trading-agent | postgres, redis, ib-gateway:4002 | +| backtesting-service | api-gateway | postgres, redis, minio:9000 | +| ml-training-service | api-gateway | postgres, minio:9000, gpu nodes | +| trading-agent-service | api-gateway | trading-service, postgres | +| broker-gateway | trading-service | ib-gateway:4002, postgres | +| data-acquisition-service | api-gateway | postgres, external HTTPS (443) | +| ib-gateway | broker-gateway, trading-service | external (IBKR servers) | + +### 1c. Monitoring exemption + +Prometheus scrape allowed from `monitoring` namespace to all metrics ports (909x). + +### 1d. DNS exemption + +All pods get egress to `kube-system` on port 53 (CoreDNS). + +All policies use `podSelector` with `app` labels already present on deployments. + +## 2. SecurityContext — Pod Hardening + +### 2a. Pod-level securityContext (every deployment) + +```yaml +securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + seccompProfile: + type: RuntimeDefault +``` + +### 2b. Main container securityContext + +```yaml +securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] +``` + +### 2c. InitContainer stays privileged + +`runAsUser: 0` kept for `fetch-binary` initContainer (needs to write to emptyDir). + +### 2d. Writable tmpfs + +Add `emptyDir` mount at `/tmp` for services needing temp writes. + +### 2e. Exceptions + +- **ib-gateway**: Third-party Java image, skip `readOnlyRootFilesystem`. +- **Tailscale sidecar**: Keeps `NET_ADMIN`, `NET_RAW` (required for VPN). + +Applies to all 9 service deployments + both GPU overlay variants. + +## 3. Secret Scoping — Split Monolithic Secret + +### 3a. Per-concern secrets + +| Secret Name | Keys | Used By | +|------------|------|---------| +| `db-credentials` | `password` | api-gw, trading, trading-agent, backtesting, broker-gw, data-acq, ml-training | +| `jwt-secret` | `secret` | api-gw, web-gw, trading, trading-agent, backtesting, broker-gw | +| `redis-credentials` | `password` | api-gw, trading | +| `s3-credentials` | `access-key`, `secret-key` | ml-training, backtesting | +| `ibkr-credentials` | (unchanged) | trading, broker-gw, ib-gateway | +| `ml-training-tls` | (unchanged) | ml-training | +| `minio-credentials` | (unchanged) | all (initContainer fetch-binary) | + +### 3b. Update secretKeyRef + +Each deployment updated to reference new secret names instead of foxhunt-secrets. + +### 3c. Remove placeholder values + +Secret manifests contain `REPLACE_IN_DEPLOY` markers. Real values injected by deploy script. + +### 3d. deploy-secrets.sh + +Script reads from Scaleway Secrets Manager CLI and creates K8s secrets. +Replaces manually editing YAML with placeholder values. + +## 4. CI Image Scanning — Trivy + +### 4a. New `scan` stage + +Between `prepare` and `test` in `.gitlab-ci.yml`. + +### 4b. Scope + +Scan 2 runtime images only: foxhunt-runtime, foxhunt-training-runtime. +CI builder images are internal-only. + +### 4c. Fail policy + +`--exit-code 1` on CRITICAL and HIGH with `--ignore-unfixed`. +Pipeline blocks on fixable critical/high CVEs. + +### 4d. Cache + +Trivy DB cache persisted via CI cache key. + +## Risk + +Low-medium. NetworkPolicy and securityContext changes could break connectivity +or file access if misconfigured. Mitigation: deploy one service at a time, +verify health checks pass before proceeding. + +Secret scoping is a rename operation — same values, different secret objects. +CI scanning is additive (new stage, no existing stages modified).