gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
9c081ac7ce52fe480d3ecce495c94dea3d3bc22b
foxhunt/docs/plans/2026-03-01-infra-security-hardening-implementation.md
jgrusewski 3799981321 docs: add infrastructure security hardening implementation plan 
7-task plan: NetworkPolicy (default-deny + per-service), SecurityContext
(pod + container hardening), secret scoping (split foxhunt-secrets),
deploy-secrets.sh (Scaleway integration), Trivy CI scanning.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:16:44 +01:00

1390 lines
35 KiB
Markdown
Raw Blame History

# Infrastructure Security Hardening — Implementation Plan
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
**Goal:** Harden K8s infrastructure for production: NetworkPolicy, SecurityContext, secret scoping, CI image scanning.
**Architecture:** 4 independent hardening layers applied to existing manifests in `infra/k8s/`. NetworkPolicy files are new; SecurityContext + secret scoping modify existing deployment YAMLs; Trivy scanning adds a new CI stage. No Rust code changes.
**Tech Stack:** Kubernetes NetworkPolicy, Pod SecurityContext, Scaleway Secrets Manager CLI (`scw`), Trivy container scanner, GitLab CI YAML.
---
### Task 1: NetworkPolicy — Default Deny + DNS Exemption
**Files:**
- Create: `infra/k8s/network-policies/default-deny.yaml`
- Create: `infra/k8s/network-policies/allow-dns.yaml`
- Create: `infra/k8s/network-policies/allow-monitoring.yaml`
**Context:** Currently zero NetworkPolicy files exist. All pods in namespace `foxhunt` communicate freely. We start with the foundation: deny-all + DNS + monitoring exemptions. Services will be blocked until Task 2 adds per-service rules.
**Step 1: Create the network-policies directory**
```bash
mkdir -p infra/k8s/network-policies
```
**Step 2: Create default-deny.yaml**
Create `infra/k8s/network-policies/default-deny.yaml`:
```yaml
# Default deny all ingress and egress in foxhunt namespace.
# Every service must declare its own NetworkPolicy to communicate.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: foxhunt
labels:
app.kubernetes.io/part-of: foxhunt
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
```
**Step 3: Create allow-dns.yaml**
Create `infra/k8s/network-policies/allow-dns.yaml`:
```yaml
# Allow all pods to reach CoreDNS (kube-system) for name resolution.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: foxhunt
labels:
app.kubernetes.io/part-of: foxhunt
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
```
**Step 4: Create allow-monitoring.yaml**
Create `infra/k8s/network-policies/allow-monitoring.yaml`:
```yaml
# Allow Prometheus (monitoring namespace) to scrape metrics ports on all foxhunt pods.
# Also allow all foxhunt pods to push traces to Tempo.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-monitoring-scrape
namespace: foxhunt
labels:
app.kubernetes.io/part-of: foxhunt
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
ports:
- protocol: TCP
port: 9091
- protocol: TCP
port: 9092
- protocol: TCP
port: 9093
- protocol: TCP
port: 9094
- protocol: TCP
port: 9095
- protocol: TCP
port: 9096
- protocol: TCP
port: 9097
- protocol: TCP
port: 9098
egress:
# Tempo OTLP gRPC
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: foxhunt
podSelector:
matchLabels:
app.kubernetes.io/name: tempo
ports:
- protocol: TCP
port: 4317
```
**Step 5: Verify YAML is valid**
```bash
# Quick syntax check — kubectl dry-run (no cluster needed)
for f in infra/k8s/network-policies/*.yaml; do
echo "--- $f ---"
cat "$f" | python3 -c "import sys, yaml; yaml.safe_load(sys.stdin.read()); print('OK')"
done
```
Expected: all 3 files print `OK`.
**Step 6: Commit**
```bash
git add infra/k8s/network-policies/
git commit -m "infra: add default-deny NetworkPolicy + DNS + monitoring exemptions"
```
---
### Task 2: NetworkPolicy — Per-Service Allow Rules
**Files:**
- Create: `infra/k8s/network-policies/api-gateway.yaml`
- Create: `infra/k8s/network-policies/web-gateway.yaml`
- Create: `infra/k8s/network-policies/trading-service.yaml`
- Create: `infra/k8s/network-policies/backtesting-service.yaml`
- Create: `infra/k8s/network-policies/ml-training-service.yaml`
- Create: `infra/k8s/network-policies/trading-agent-service.yaml`
- Create: `infra/k8s/network-policies/broker-gateway.yaml`
- Create: `infra/k8s/network-policies/data-acquisition-service.yaml`
- Create: `infra/k8s/network-policies/ib-gateway.yaml`
- Create: `infra/k8s/network-policies/infrastructure.yaml`
**Context:** Each service needs ingress (who can call it) and egress (what it can call). Labels used: `app.kubernetes.io/name` (most services) and `app: ib-gateway` (ib-gateway uses different label scheme). All initContainers need egress to minio:9000. Infrastructure pods (postgres, redis, minio, questdb) need ingress from services.
**Important label details** (read from existing manifests):
- Most services use `app.kubernetes.io/name: <service-name>` as their pod label
- ib-gateway uses `app: ib-gateway` as its pod label
- Postgres, Redis, Minio, QuestDB — assume labels `app: postgres`, `app: redis`, `app: minio`, `app: questdb`
**Step 1: Create api-gateway.yaml**
Create `infra/k8s/network-policies/api-gateway.yaml`:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-gateway
namespace: foxhunt
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
policyTypes:
- Ingress
- Egress
ingress:
# Tailscale ingress proxy
- from:
- podSelector:
matchLabels:
app: tailscale
ports:
- protocol: TCP
port: 50051
egress:
# Postgres
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
# Redis
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
# Web gateway
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: web-gateway
ports:
- protocol: TCP
port: 3000
# Downstream gRPC services (for proxied requests)
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: trading-service
ports:
- protocol: TCP
port: 50051
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: backtesting-service
ports:
- protocol: TCP
port: 50053
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: ml-training-service
ports:
- protocol: TCP
port: 50053
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: trading-agent-service
ports:
- protocol: TCP
port: 50055
# Minio (initContainer binary fetch)
- to:
- podSelector:
matchLabels:
app: minio
ports:
- protocol: TCP
port: 9000
```
**Step 2: Create web-gateway.yaml**
Create `infra/k8s/network-policies/web-gateway.yaml`:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: web-gateway
namespace: foxhunt
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: web-gateway
policyTypes:
- Ingress
- Egress
ingress:
# api-gateway proxies HTTP requests
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
ports:
- protocol: TCP
port: 3000
# Tailscale direct access (dashboard)
- from:
- podSelector:
matchLabels:
app: tailscale
ports:
- protocol: TCP
port: 3000
egress:
# api-gateway gRPC upstream
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
ports:
- protocol: TCP
port: 50051
# Direct gRPC to downstream services (web-gateway env vars show these)
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: backtesting-service
ports:
- protocol: TCP
port: 50053
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: ml-training-service
ports:
- protocol: TCP
port: 50053
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: trading-agent-service
ports:
- protocol: TCP
port: 50055
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: broker-gateway
ports:
- protocol: TCP
port: 50056
# Minio (initContainer binary fetch)
- to:
- podSelector:
matchLabels:
app: minio
ports:
- protocol: TCP
port: 9000
```
**Step 3: Create trading-service.yaml**
Create `infra/k8s/network-policies/trading-service.yaml`:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: trading-service
namespace: foxhunt
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: trading-service
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
- podSelector:
matchLabels:
app.kubernetes.io/name: trading-agent-service
ports:
- protocol: TCP
port: 50051
egress:
# Postgres
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
# Redis
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
# IB Gateway TWS API
- to:
- podSelector:
matchLabels:
app: ib-gateway
ports:
- protocol: TCP
port: 4002
# QuestDB ILP
- to:
- podSelector:
matchLabels:
app: questdb
ports:
- protocol: TCP
port: 9009
# Minio (initContainer)
- to:
- podSelector:
matchLabels:
app: minio
ports:
- protocol: TCP
port: 9000
```
**Step 4: Create backtesting-service.yaml**
Create `infra/k8s/network-policies/backtesting-service.yaml`:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: backtesting-service
namespace: foxhunt
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: backtesting-service
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
ports:
- protocol: TCP
port: 50053
egress:
# Postgres
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
# Redis
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
# Minio (data + initContainer)
- to:
- podSelector:
matchLabels:
app: minio
ports:
- protocol: TCP
port: 9000
```
**Step 5: Create ml-training-service.yaml**
Create `infra/k8s/network-policies/ml-training-service.yaml`:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: ml-training-service
namespace: foxhunt
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: ml-training-service
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
ports:
- protocol: TCP
port: 50053
egress:
# Postgres
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
# Redis
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
# Minio (model storage + initContainer)
- to:
- podSelector:
matchLabels:
app: minio
ports:
- protocol: TCP
port: 9000
# Kubernetes API (for job creation — ml-training-service has RBAC for batch/jobs)
- to:
- ipBlock:
cidr: 0.0.0.0/0
ports:
- protocol: TCP
port: 443
```
**Step 6: Create trading-agent-service.yaml**
Create `infra/k8s/network-policies/trading-agent-service.yaml`:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: trading-agent-service
namespace: foxhunt
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: trading-agent-service
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
ports:
- protocol: TCP
port: 50055
egress:
# Trading service
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: trading-service
ports:
- protocol: TCP
port: 50051
# Postgres
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
# Redis
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
# Minio (initContainer)
- to:
- podSelector:
matchLabels:
app: minio
ports:
- protocol: TCP
port: 9000
```
**Step 7: Create broker-gateway.yaml**
Create `infra/k8s/network-policies/broker-gateway.yaml`:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: broker-gateway
namespace: foxhunt
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: broker-gateway
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: trading-service
- podSelector:
matchLabels:
app.kubernetes.io/name: web-gateway
ports:
- protocol: TCP
port: 50056
egress:
# IB Gateway TWS API
- to:
- podSelector:
matchLabels:
app: ib-gateway
ports:
- protocol: TCP
port: 4002
# Postgres
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
# Redis
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
# Minio (initContainer)
- to:
- podSelector:
matchLabels:
app: minio
ports:
- protocol: TCP
port: 9000
```
**Step 8: Create data-acquisition-service.yaml**
Create `infra/k8s/network-policies/data-acquisition-service.yaml`:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: data-acquisition-service
namespace: foxhunt
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: data-acquisition-service
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
ports:
- protocol: TCP
port: 50057
egress:
# Postgres
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- protocol: TCP
port: 5432
# Redis
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
# External HTTPS (Databento, Benzinga APIs)
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- protocol: TCP
port: 443
# Minio (initContainer)
- to:
- podSelector:
matchLabels:
app: minio
ports:
- protocol: TCP
port: 9000
```
**Step 9: Create ib-gateway.yaml**
Create `infra/k8s/network-policies/ib-gateway.yaml`:
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: ib-gateway
namespace: foxhunt
spec:
podSelector:
matchLabels:
app: ib-gateway
policyTypes:
- Ingress
- Egress
ingress:
# Trading service + broker gateway connect to TWS API
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: trading-service
- podSelector:
matchLabels:
app.kubernetes.io/name: broker-gateway
ports:
- protocol: TCP
port: 4002
- protocol: TCP
port: 4004
egress:
# IBKR external servers (market data, order routing)
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- protocol: TCP
port: 443
- protocol: TCP
port: 4000
- protocol: TCP
port: 4001
```
**Step 10: Create infrastructure.yaml (postgres, redis, minio, questdb)**
Create `infra/k8s/network-policies/infrastructure.yaml`:
```yaml
# Postgres: accepts connections from all foxhunt app pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: postgres
namespace: foxhunt
spec:
podSelector:
matchLabels:
app: postgres
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchExpressions:
- key: app.kubernetes.io/part-of
operator: In
values: ["foxhunt"]
ports:
- protocol: TCP
port: 5432
---
# Redis: accepts connections from foxhunt app pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: redis
namespace: foxhunt
spec:
podSelector:
matchLabels:
app: redis
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchExpressions:
- key: app.kubernetes.io/part-of
operator: In
values: ["foxhunt"]
ports:
- protocol: TCP
port: 6379
---
# Minio: accepts connections from foxhunt app pods (S3 API)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: minio
namespace: foxhunt
spec:
podSelector:
matchLabels:
app: minio
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchExpressions:
- key: app.kubernetes.io/part-of
operator: In
values: ["foxhunt"]
ports:
- protocol: TCP
port: 9000
---
# QuestDB: accepts ILP writes from trading-service
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: questdb
namespace: foxhunt
spec:
podSelector:
matchLabels:
app: questdb
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: trading-service
ports:
- protocol: TCP
port: 9009
```
**Step 11: Validate all YAML files**
```bash
for f in infra/k8s/network-policies/*.yaml; do
echo "--- $f ---"
cat "$f" | python3 -c "import sys, yaml; [yaml.safe_load(d) for d in sys.stdin.read().split('---') if d.strip()]; print('OK')"
done
```
Expected: all 10 files print `OK`.
**Step 12: Commit**
```bash
git add infra/k8s/network-policies/
git commit -m "infra: add per-service NetworkPolicy rules
9 service policies + infrastructure ingress rules.
Each service declares explicit ingress/egress.
External access limited to data-acquisition (HTTPS)
and ib-gateway (IBKR servers)."
```
---
### Task 3: SecurityContext — Harden All Service Deployments
**Files:**
- Modify: `infra/k8s/services/api-gateway.yaml`
- Modify: `infra/k8s/services/web-gateway.yaml`
- Modify: `infra/k8s/services/trading-service.yaml`
- Modify: `infra/k8s/services/backtesting-service.yaml`
- Modify: `infra/k8s/services/ml-training-service.yaml`
- Modify: `infra/k8s/services/trading-agent-service.yaml`
- Modify: `infra/k8s/services/broker-gateway.yaml`
- Modify: `infra/k8s/services/data-acquisition-service.yaml`
- Modify: `infra/k8s/services/ib-gateway.yaml`
- Modify: `infra/k8s/services/trading-service-gpu.yaml`
- Modify: `infra/k8s/services/ml-training-service-gpu.yaml`
**Context:** All 9 service deployments + 2 GPU overlays need pod-level and container-level securityContext. The Dockerfiles already create user `foxhunt` with UID 1000. InitContainers keep `runAsUser: 0` (binary fetch needs root). ib-gateway is third-party (Java) and cannot use `readOnlyRootFilesystem`.
**For each of the 9 CPU service manifests** (api-gateway, web-gateway, trading-service, backtesting-service, ml-training-service, trading-agent-service, broker-gateway, data-acquisition-service), apply these changes to the Deployment spec:
**Step 1: Add pod-level securityContext**
In each deployment's `spec.template.spec`, add immediately after `nodeSelector` (or after `serviceAccountName` for ml-training-service):
```yaml
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
```
**Step 2: Add container-level securityContext to main container**
In each deployment's main container (the one under `containers:`), add a `securityContext` block:
```yaml
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
```
**Step 3: Add /tmp emptyDir volume**
Add to the `volumes:` section of each deployment:
```yaml
- name: tmp
emptyDir:
sizeLimit: 50Mi
```
And add to the main container's `volumeMounts:`:
```yaml
- name: tmp
mountPath: /tmp
```
**Step 4: Handle ib-gateway differently**
For `infra/k8s/services/ib-gateway.yaml`, add pod-level securityContext but **without** `runAsNonRoot` (third-party image manages its own user), and **without** `readOnlyRootFilesystem` (Java needs writes):
```yaml
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
containers:
- name: ib-gateway
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
```
**Step 5: Apply same changes to GPU overlay files**
For `infra/k8s/services/trading-service-gpu.yaml` and `infra/k8s/services/ml-training-service-gpu.yaml`, apply the same pod-level and container-level securityContext as their CPU counterparts.
**Step 6: Validate YAML syntax**
```bash
for f in infra/k8s/services/*.yaml; do
echo "--- $f ---"
cat "$f" | python3 -c "import sys, yaml; [yaml.safe_load(d) for d in sys.stdin.read().split('---') if d.strip()]; print('OK')"
done
```
Expected: all 11 files print `OK`.
**Step 7: Commit**
```bash
git add infra/k8s/services/
git commit -m "infra: add SecurityContext to all service deployments
Pod-level: runAsNonRoot, runAsUser/Group 1000, seccomp RuntimeDefault.
Container-level: no privilege escalation, read-only rootfs, drop ALL caps.
Exception: ib-gateway skips runAsNonRoot and readOnlyRootFilesystem.
InitContainers keep runAsUser: 0 for binary fetch."
```
---
### Task 4: Secret Scoping — Split foxhunt-secrets
**Files:**
- Modify: `infra/k8s/secrets/foxhunt-secrets.yaml` → replace with 4 scoped secrets
- Modify: `infra/k8s/services/api-gateway.yaml` (secretKeyRef names)
- Modify: `infra/k8s/services/web-gateway.yaml`
- Modify: `infra/k8s/services/trading-service.yaml`
- Modify: `infra/k8s/services/backtesting-service.yaml`
- Modify: `infra/k8s/services/ml-training-service.yaml`
- Modify: `infra/k8s/services/trading-agent-service.yaml`
- Modify: `infra/k8s/services/broker-gateway.yaml`
- Modify: `infra/k8s/services/data-acquisition-service.yaml`
- Modify: `infra/k8s/services/trading-service-gpu.yaml`
- Modify: `infra/k8s/services/ml-training-service-gpu.yaml`
**Context:** Currently one `foxhunt-secrets` Opaque secret holds db-password, jwt-secret, redis-password, s3-access-key, s3-secret-key. All services reference it even if they only need 1-2 keys. Splitting into per-concern secrets reduces blast radius.
**Step 1: Replace foxhunt-secrets.yaml with scoped secrets**
Replace the contents of `infra/k8s/secrets/foxhunt-secrets.yaml` with:
```yaml
# Database credentials — used by services that connect to PostgreSQL.
# Real values injected by scripts/deploy-secrets.sh from Scaleway Secrets Manager.
apiVersion: v1
kind: Secret
metadata:
name: db-credentials
namespace: foxhunt
labels:
app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
password: "REPLACE_IN_DEPLOY"
---
# JWT signing secret — used by services that validate/issue JWTs.
apiVersion: v1
kind: Secret
metadata:
name: jwt-secret
namespace: foxhunt
labels:
app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
secret: "REPLACE_IN_DEPLOY"
---
# Redis credentials — used by services with Redis connections.
apiVersion: v1
kind: Secret
metadata:
name: redis-credentials
namespace: foxhunt
labels:
app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
password: "REPLACE_IN_DEPLOY"
---
# S3/Minio credentials for application-level access (model storage, data).
# NOT the same as minio-credentials (used by initContainers for binary fetch).
apiVersion: v1
kind: Secret
metadata:
name: s3-credentials
namespace: foxhunt
labels:
app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
access-key: "REPLACE_IN_DEPLOY"
secret-key: "REPLACE_IN_DEPLOY"
---
# Registry pull secret for Scaleway Container Registry
# Create with:
# kubectl create secret docker-registry scw-registry \
# --namespace foxhunt \
# --docker-server=rg.fr-par.scw.cloud \
# --docker-username=foxhunt \
# --docker-password=<SCW_SECRET_KEY>
```
**Step 2: Update secretKeyRef in all service deployments**
For every service deployment, change `foxhunt-secrets` references to the new secret names:
**db-password references** (`name: foxhunt-secrets, key: db-password` → `name: db-credentials, key: password`):
- api-gateway.yaml
- trading-service.yaml (+ GPU overlay)
- backtesting-service.yaml
- ml-training-service.yaml (+ GPU overlay)
- trading-agent-service.yaml
- broker-gateway.yaml
- data-acquisition-service.yaml
**jwt-secret references** (`name: foxhunt-secrets, key: jwt-secret` → `name: jwt-secret, key: secret`):
- api-gateway.yaml
- web-gateway.yaml
- trading-service.yaml (+ GPU overlay)
- backtesting-service.yaml
- ml-training-service.yaml (+ GPU overlay)
- trading-agent-service.yaml
- broker-gateway.yaml
**Step 3: Validate YAML syntax**
```bash
for f in infra/k8s/secrets/foxhunt-secrets.yaml infra/k8s/services/*.yaml; do
echo "--- $f ---"
cat "$f" | python3 -c "import sys, yaml; [yaml.safe_load(d) for d in sys.stdin.read().split('---') if d.strip()]; print('OK')"
done
```
Expected: all files print `OK`.
**Step 4: Commit**
```bash
git add infra/k8s/secrets/foxhunt-secrets.yaml infra/k8s/services/
git commit -m "infra: split foxhunt-secrets into per-concern secrets
db-credentials, jwt-secret, redis-credentials, s3-credentials.
Each service only references the secrets it needs.
Values are REPLACE_IN_DEPLOY markers — real values
injected by scripts/deploy-secrets.sh."
```
---
### Task 5: deploy-secrets.sh — Scaleway Secrets Manager Integration
**Files:**
- Create: `scripts/deploy-secrets.sh`
**Context:** The split secrets from Task 4 need real values at deploy time. This script reads from Scaleway Secrets Manager and creates K8s secrets. The Scaleway CLI (`scw`) can access secrets via `scw secret version access-by-path`.
**Step 1: Create scripts/deploy-secrets.sh**
Create `scripts/deploy-secrets.sh`:
```bash
#!/usr/bin/env bash
# deploy-secrets.sh — Populate K8s secrets from Scaleway Secrets Manager.
# Usage: ./scripts/deploy-secrets.sh [--dry-run]
#
# Prerequisites:
# - scw CLI configured (SCW_ACCESS_KEY, SCW_SECRET_KEY, SCW_DEFAULT_PROJECT_ID)
# - kubectl configured for target cluster
# - Scaleway secrets exist at paths: foxhunt/db-password, foxhunt/jwt-secret, etc.
#
# Scaleway secret paths (create these in SCW console or CLI):
# foxhunt/db-password
# foxhunt/jwt-secret
# foxhunt/redis-password
# foxhunt/s3-access-key
# foxhunt/s3-secret-key
set -euo pipefail
NAMESPACE="foxhunt"
DRY_RUN=""
if [[ "${1:-}" == "--dry-run" ]]; then
DRY_RUN="--dry-run=client"
echo "DRY RUN — no changes will be applied"
fi
# Fetch a secret value from Scaleway Secrets Manager.
# Usage: scw_secret <path>
scw_secret() {
local path="$1"
scw secret version access-by-path name="$path" field=data --output json 2>/dev/null \
| python3 -c "import sys,json; print(json.load(sys.stdin)['data'])" \
|| { echo "ERROR: Failed to fetch secret '$path' from Scaleway" >&2; exit 1; }
}
echo "Fetching secrets from Scaleway Secrets Manager..."
DB_PASSWORD=$(scw_secret "foxhunt/db-password")
JWT_SECRET=$(scw_secret "foxhunt/jwt-secret")
REDIS_PASSWORD=$(scw_secret "foxhunt/redis-password")
S3_ACCESS_KEY=$(scw_secret "foxhunt/s3-access-key")
S3_SECRET_KEY=$(scw_secret "foxhunt/s3-secret-key")
echo "Creating K8s secrets in namespace $NAMESPACE..."
# db-credentials
kubectl create secret generic db-credentials \
--namespace="$NAMESPACE" \
--from-literal=password="$DB_PASSWORD" \
--save-config $DRY_RUN \
-o yaml | kubectl apply -f - $DRY_RUN
# jwt-secret
kubectl create secret generic jwt-secret \
--namespace="$NAMESPACE" \
--from-literal=secret="$JWT_SECRET" \
--save-config $DRY_RUN \
-o yaml | kubectl apply -f - $DRY_RUN
# redis-credentials
kubectl create secret generic redis-credentials \
--namespace="$NAMESPACE" \
--from-literal=password="$REDIS_PASSWORD" \
--save-config $DRY_RUN \
-o yaml | kubectl apply -f - $DRY_RUN
# s3-credentials
kubectl create secret generic s3-credentials \
--namespace="$NAMESPACE" \
--from-literal=access-key="$S3_ACCESS_KEY" \
--from-literal=secret-key="$S3_SECRET_KEY" \
--save-config $DRY_RUN \
-o yaml | kubectl apply -f - $DRY_RUN
echo "Done. Secrets created/updated in namespace $NAMESPACE."
echo ""
echo "Verify with: kubectl get secrets -n $NAMESPACE"
```
**Step 2: Make executable**
```bash
chmod +x scripts/deploy-secrets.sh
```
**Step 3: Commit**
```bash
git add scripts/deploy-secrets.sh
git commit -m "scripts: add deploy-secrets.sh for Scaleway Secrets Manager integration
Reads secrets from SCW and creates/updates K8s secrets.
Supports --dry-run for preview. Replaces manual YAML editing."
```
---
### Task 6: CI Image Scanning — Trivy in GitLab CI
**Files:**
- Modify: `.gitlab-ci.yml`
**Context:** Currently 5 stages: prepare → test → compile → train → deploy. Add `scan` between `prepare` and `test`. Scan the 2 runtime images (foxhunt-runtime, foxhunt-training-runtime) that all service pods use. CI builder images are internal-only and don't need blocking scans.
**Step 1: Add `scan` stage to stages list**
In `.gitlab-ci.yml`, change the `stages:` block from:
```yaml
stages:
- prepare
- test
- compile
- train
- deploy
```
to:
```yaml
stages:
- prepare
- scan
- test
- compile
- train
- deploy
```
**Step 2: Add scan-runtime-images job**
Add this job after the last `prepare` stage job (after `build-foxhunt-training-runtime`) and before the `.rust-base` template:
```yaml
# --------------------------------------------------------------------------
# Stage 0.5: Scan runtime images for vulnerabilities (Trivy)
# --------------------------------------------------------------------------
scan-runtime-images:
stage: scan
image:
name: aquasec/trivy:latest
entrypoint: [""]
tags:
- kapsule
- docker
variables:
KUBERNETES_CPU_REQUEST: "200m"
KUBERNETES_CPU_LIMIT: "500m"
KUBERNETES_MEMORY_REQUEST: "256Mi"
KUBERNETES_MEMORY_LIMIT: "512Mi"
needs:
- job: build-foxhunt-runtime
optional: true
- job: build-foxhunt-training-runtime
optional: true
rules:
- if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
- if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
cache:
key: trivy-db
paths:
- .trivy-cache/
before_script:
# Authenticate to Scaleway Container Registry for image pull
- export TRIVY_USERNAME=nologin
- export TRIVY_PASSWORD=${SCW_SECRET_KEY}
script:
- echo "Scanning runtime images for CRITICAL/HIGH vulnerabilities..."
- trivy image
--cache-dir .trivy-cache
--exit-code 1
--severity CRITICAL,HIGH
--ignore-unfixed
--no-progress
"${REGISTRY}/foxhunt-runtime:latest"
- trivy image
--cache-dir .trivy-cache
--exit-code 1
--severity CRITICAL,HIGH
--ignore-unfixed
--no-progress
"${REGISTRY}/foxhunt-training-runtime:latest"
- echo "Image scan passed — no fixable CRITICAL/HIGH vulnerabilities."
allow_failure: false
```
**Step 3: Validate CI YAML syntax**
```bash
python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml')); print('OK')"
```
Expected: `OK`.
**Step 4: Commit**
```bash
git add .gitlab-ci.yml
git commit -m "ci: add Trivy image scanning for runtime images
New 'scan' stage between prepare and test.
Scans foxhunt-runtime and foxhunt-training-runtime
for CRITICAL/HIGH fixable CVEs. Pipeline blocks on findings.
Trivy DB cached between runs."
```
---
### Task 7: Verify Everything
**Step 1: Check all new files exist**
```bash
echo "=== NetworkPolicy files ==="
ls -la infra/k8s/network-policies/
echo ""
echo "=== Secret manifests ==="
ls -la infra/k8s/secrets/
echo ""
echo "=== deploy-secrets.sh ==="
ls -la scripts/deploy-secrets.sh
```
Expected: 13 network policy files, 3 secret files, deploy-secrets.sh is executable.
**Step 2: Validate all YAML syntax**
```bash
for f in $(find infra/k8s -name '*.yaml'); do
cat "$f" | python3 -c "
import sys, yaml
docs = [d for d in sys.stdin.read().split('---') if d.strip()]
for d in docs:
yaml.safe_load(d)
print('OK: ' + '$f')
"
done
```
Expected: all files print `OK`.
**Step 3: Count changes**
```bash
git diff --stat HEAD~6..HEAD
```
Expected: ~20 new files, ~11 modified files. No Rust code changes.
**Step 4: Verify CI stages**
```bash
grep -A 10 '^stages:' .gitlab-ci.yml
```
Expected: prepare, scan, test, compile, train, deploy (6 stages).
Reference in New Issue View Git Blame Copy Permalink