foxhunt/docs/plans/2026-03-01-infra-security-hardening-implementation.md

Infrastructure Security Hardening — Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Harden K8s infrastructure for production: NetworkPolicy, SecurityContext, secret scoping, CI image scanning.

Architecture: 4 independent hardening layers applied to existing manifests in infra/k8s/. NetworkPolicy files are new; SecurityContext + secret scoping modify existing deployment YAMLs; Trivy scanning adds a new CI stage. No Rust code changes.

Tech Stack: Kubernetes NetworkPolicy, Pod SecurityContext, Scaleway Secrets Manager CLI (scw), Trivy container scanner, GitLab CI YAML.

---

Task 1: NetworkPolicy — Default Deny + DNS Exemption

Files:

• Create: infra/k8s/network-policies/default-deny.yaml
• Create: infra/k8s/network-policies/allow-dns.yaml
• Create: infra/k8s/network-policies/allow-monitoring.yaml

Context: Currently zero NetworkPolicy files exist. All pods in namespace foxhunt communicate freely. We start with the foundation: deny-all + DNS + monitoring exemptions. Services will be blocked until Task 2 adds per-service rules.

Step 1: Create the network-policies directory

mkdir -p infra/k8s/network-policies

Step 2: Create default-deny.yaml

Create infra/k8s/network-policies/default-deny.yaml:

# Default deny all ingress and egress in foxhunt namespace.
# Every service must declare its own NetworkPolicy to communicate.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Step 3: Create allow-dns.yaml

Create infra/k8s/network-policies/allow-dns.yaml:

# Allow all pods to reach CoreDNS (kube-system) for name resolution.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Step 4: Create allow-monitoring.yaml

Create infra/k8s/network-policies/allow-monitoring.yaml:

# Allow Prometheus (monitoring namespace) to scrape metrics ports on all foxhunt pods.
# Also allow all foxhunt pods to push traces to Tempo.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-monitoring-scrape
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - protocol: TCP
          port: 9091
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
        - protocol: TCP
          port: 9094
        - protocol: TCP
          port: 9095
        - protocol: TCP
          port: 9096
        - protocol: TCP
          port: 9097
        - protocol: TCP
          port: 9098
  egress:
    # Tempo OTLP gRPC
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: foxhunt
          podSelector:
            matchLabels:
              app.kubernetes.io/name: tempo
      ports:
        - protocol: TCP
          port: 4317

Step 5: Verify YAML is valid

# Quick syntax check — kubectl dry-run (no cluster needed)
for f in infra/k8s/network-policies/*.yaml; do
  echo "--- $f ---"
  cat "$f" | python3 -c "import sys, yaml; yaml.safe_load(sys.stdin.read()); print('OK')"
done

Expected: all 3 files print OK.

Step 6: Commit

git add infra/k8s/network-policies/
git commit -m "infra: add default-deny NetworkPolicy + DNS + monitoring exemptions"

---

Task 2: NetworkPolicy — Per-Service Allow Rules

Files:

• Create: infra/k8s/network-policies/api-gateway.yaml
• Create: infra/k8s/network-policies/web-gateway.yaml
• Create: infra/k8s/network-policies/trading-service.yaml
• Create: infra/k8s/network-policies/backtesting-service.yaml
• Create: infra/k8s/network-policies/ml-training-service.yaml
• Create: infra/k8s/network-policies/trading-agent-service.yaml
• Create: infra/k8s/network-policies/broker-gateway.yaml
• Create: infra/k8s/network-policies/data-acquisition-service.yaml
• Create: infra/k8s/network-policies/ib-gateway.yaml
• Create: infra/k8s/network-policies/infrastructure.yaml

Context: Each service needs ingress (who can call it) and egress (what it can call). Labels used: app.kubernetes.io/name (most services) and app: ib-gateway (ib-gateway uses different label scheme). All initContainers need egress to minio:9000. Infrastructure pods (postgres, redis, minio, questdb) need ingress from services.

Important label details (read from existing manifests):

• Most services use app.kubernetes.io/name: <service-name> as their pod label
• ib-gateway uses app: ib-gateway as its pod label
• Postgres, Redis, Minio, QuestDB — assume labels app: postgres, app: redis, app: minio, app: questdb

Step 1: Create api-gateway.yaml

Create infra/k8s/network-policies/api-gateway.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-gateway
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: api-gateway
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Tailscale ingress proxy
    - from:
        - podSelector:
            matchLabels:
              app: tailscale
      ports:
        - protocol: TCP
          port: 50051
  egress:
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # Web gateway
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: web-gateway
      ports:
        - protocol: TCP
          port: 3000
    # Downstream gRPC services (for proxied requests)
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-service
      ports:
        - protocol: TCP
          port: 50051
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: backtesting-service
      ports:
        - protocol: TCP
          port: 50053
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: ml-training-service
      ports:
        - protocol: TCP
          port: 50053
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-agent-service
      ports:
        - protocol: TCP
          port: 50055
    # Minio (initContainer binary fetch)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000

Step 2: Create web-gateway.yaml

Create infra/k8s/network-policies/web-gateway.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-gateway
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: web-gateway
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # api-gateway proxies HTTP requests
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 3000
    # Tailscale direct access (dashboard)
    - from:
        - podSelector:
            matchLabels:
              app: tailscale
      ports:
        - protocol: TCP
          port: 3000
  egress:
    # api-gateway gRPC upstream
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 50051
    # Direct gRPC to downstream services (web-gateway env vars show these)
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: backtesting-service
      ports:
        - protocol: TCP
          port: 50053
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: ml-training-service
      ports:
        - protocol: TCP
          port: 50053
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-agent-service
      ports:
        - protocol: TCP
          port: 50055
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: broker-gateway
      ports:
        - protocol: TCP
          port: 50056
    # Minio (initContainer binary fetch)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000

Step 3: Create trading-service.yaml

Create infra/k8s/network-policies/trading-service.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: trading-service
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: trading-service
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-agent-service
      ports:
        - protocol: TCP
          port: 50051
  egress:
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # IB Gateway TWS API
    - to:
        - podSelector:
            matchLabels:
              app: ib-gateway
      ports:
        - protocol: TCP
          port: 4002
    # QuestDB ILP
    - to:
        - podSelector:
            matchLabels:
              app: questdb
      ports:
        - protocol: TCP
          port: 9009
    # Minio (initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000

Step 4: Create backtesting-service.yaml

Create infra/k8s/network-policies/backtesting-service.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backtesting-service
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: backtesting-service
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 50053
  egress:
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # Minio (data + initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000

Step 5: Create ml-training-service.yaml

Create infra/k8s/network-policies/ml-training-service.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ml-training-service
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: ml-training-service
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 50053
  egress:
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # Minio (model storage + initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000
    # Kubernetes API (for job creation — ml-training-service has RBAC for batch/jobs)
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
      ports:
        - protocol: TCP
          port: 443

Step 6: Create trading-agent-service.yaml

Create infra/k8s/network-policies/trading-agent-service.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: trading-agent-service
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: trading-agent-service
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 50055
  egress:
    # Trading service
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-service
      ports:
        - protocol: TCP
          port: 50051
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # Minio (initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000

Step 7: Create broker-gateway.yaml

Create infra/k8s/network-policies/broker-gateway.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: broker-gateway
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: broker-gateway
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-service
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: web-gateway
      ports:
        - protocol: TCP
          port: 50056
  egress:
    # IB Gateway TWS API
    - to:
        - podSelector:
            matchLabels:
              app: ib-gateway
      ports:
        - protocol: TCP
          port: 4002
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # Minio (initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000

Step 8: Create data-acquisition-service.yaml

Create infra/k8s/network-policies/data-acquisition-service.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: data-acquisition-service
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: data-acquisition-service
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 50057
  egress:
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # External HTTPS (Databento, Benzinga APIs)
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - protocol: TCP
          port: 443
    # Minio (initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000

Step 9: Create ib-gateway.yaml

Create infra/k8s/network-policies/ib-gateway.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ib-gateway
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app: ib-gateway
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Trading service + broker gateway connect to TWS API
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-service
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: broker-gateway
      ports:
        - protocol: TCP
          port: 4002
        - protocol: TCP
          port: 4004
  egress:
    # IBKR external servers (market data, order routing)
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - protocol: TCP
          port: 443
        - protocol: TCP
          port: 4000
        - protocol: TCP
          port: 4001

Step 10: Create infrastructure.yaml (postgres, redis, minio, questdb)

Create infra/k8s/network-policies/infrastructure.yaml:

# Postgres: accepts connections from all foxhunt app pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: app.kubernetes.io/part-of
                operator: In
                values: ["foxhunt"]
      ports:
        - protocol: TCP
          port: 5432
---
# Redis: accepts connections from foxhunt app pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: redis
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app: redis
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: app.kubernetes.io/part-of
                operator: In
                values: ["foxhunt"]
      ports:
        - protocol: TCP
          port: 6379
---
# Minio: accepts connections from foxhunt app pods (S3 API)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: minio
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app: minio
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: app.kubernetes.io/part-of
                operator: In
                values: ["foxhunt"]
      ports:
        - protocol: TCP
          port: 9000
---
# QuestDB: accepts ILP writes from trading-service
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: questdb
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app: questdb
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-service
      ports:
        - protocol: TCP
          port: 9009

Step 11: Validate all YAML files

for f in infra/k8s/network-policies/*.yaml; do
  echo "--- $f ---"
  cat "$f" | python3 -c "import sys, yaml; [yaml.safe_load(d) for d in sys.stdin.read().split('---') if d.strip()]; print('OK')"
done

Expected: all 10 files print OK.

Step 12: Commit

git add infra/k8s/network-policies/
git commit -m "infra: add per-service NetworkPolicy rules

9 service policies + infrastructure ingress rules.
Each service declares explicit ingress/egress.
External access limited to data-acquisition (HTTPS)
and ib-gateway (IBKR servers)."

---

Task 3: SecurityContext — Harden All Service Deployments

Files:

• Modify: infra/k8s/services/api-gateway.yaml
• Modify: infra/k8s/services/web-gateway.yaml
• Modify: infra/k8s/services/trading-service.yaml
• Modify: infra/k8s/services/backtesting-service.yaml
• Modify: infra/k8s/services/ml-training-service.yaml
• Modify: infra/k8s/services/trading-agent-service.yaml
• Modify: infra/k8s/services/broker-gateway.yaml
• Modify: infra/k8s/services/data-acquisition-service.yaml
• Modify: infra/k8s/services/ib-gateway.yaml
• Modify: infra/k8s/services/trading-service-gpu.yaml
• Modify: infra/k8s/services/ml-training-service-gpu.yaml

Context: All 9 service deployments + 2 GPU overlays need pod-level and container-level securityContext. The Dockerfiles already create user foxhunt with UID 1000. InitContainers keep runAsUser: 0 (binary fetch needs root). ib-gateway is third-party (Java) and cannot use readOnlyRootFilesystem.

For each of the 9 CPU service manifests (api-gateway, web-gateway, trading-service, backtesting-service, ml-training-service, trading-agent-service, broker-gateway, data-acquisition-service), apply these changes to the Deployment spec:

Step 1: Add pod-level securityContext

In each deployment's spec.template.spec, add immediately after nodeSelector (or after serviceAccountName for ml-training-service):

      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault

Step 2: Add container-level securityContext to main container

In each deployment's main container (the one under containers:), add a securityContext block:

          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

Step 3: Add /tmp emptyDir volume

Add to the volumes: section of each deployment:

        - name: tmp
          emptyDir:
            sizeLimit: 50Mi

And add to the main container's volumeMounts::

            - name: tmp
              mountPath: /tmp

Step 4: Handle ib-gateway differently

For infra/k8s/services/ib-gateway.yaml, add pod-level securityContext but without runAsNonRoot (third-party image manages its own user), and without readOnlyRootFilesystem (Java needs writes):

    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: ib-gateway
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]

Step 5: Apply same changes to GPU overlay files

For infra/k8s/services/trading-service-gpu.yaml and infra/k8s/services/ml-training-service-gpu.yaml, apply the same pod-level and container-level securityContext as their CPU counterparts.

Step 6: Validate YAML syntax

for f in infra/k8s/services/*.yaml; do
  echo "--- $f ---"
  cat "$f" | python3 -c "import sys, yaml; [yaml.safe_load(d) for d in sys.stdin.read().split('---') if d.strip()]; print('OK')"
done

Expected: all 11 files print OK.

Step 7: Commit

git add infra/k8s/services/
git commit -m "infra: add SecurityContext to all service deployments

Pod-level: runAsNonRoot, runAsUser/Group 1000, seccomp RuntimeDefault.
Container-level: no privilege escalation, read-only rootfs, drop ALL caps.
Exception: ib-gateway skips runAsNonRoot and readOnlyRootFilesystem.
InitContainers keep runAsUser: 0 for binary fetch."

---

Task 4: Secret Scoping — Split foxhunt-secrets

Files:

• Modify: infra/k8s/secrets/foxhunt-secrets.yaml → replace with 4 scoped secrets
• Modify: infra/k8s/services/api-gateway.yaml (secretKeyRef names)
• Modify: infra/k8s/services/web-gateway.yaml
• Modify: infra/k8s/services/trading-service.yaml
• Modify: infra/k8s/services/backtesting-service.yaml
• Modify: infra/k8s/services/ml-training-service.yaml
• Modify: infra/k8s/services/trading-agent-service.yaml
• Modify: infra/k8s/services/broker-gateway.yaml
• Modify: infra/k8s/services/data-acquisition-service.yaml
• Modify: infra/k8s/services/trading-service-gpu.yaml
• Modify: infra/k8s/services/ml-training-service-gpu.yaml

Context: Currently one foxhunt-secrets Opaque secret holds db-password, jwt-secret, redis-password, s3-access-key, s3-secret-key. All services reference it even if they only need 1-2 keys. Splitting into per-concern secrets reduces blast radius.

Step 1: Replace foxhunt-secrets.yaml with scoped secrets

Replace the contents of infra/k8s/secrets/foxhunt-secrets.yaml with:

# Database credentials — used by services that connect to PostgreSQL.
# Real values injected by scripts/deploy-secrets.sh from Scaleway Secrets Manager.
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
  password: "REPLACE_IN_DEPLOY"
---
# JWT signing secret — used by services that validate/issue JWTs.
apiVersion: v1
kind: Secret
metadata:
  name: jwt-secret
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
  secret: "REPLACE_IN_DEPLOY"
---
# Redis credentials — used by services with Redis connections.
apiVersion: v1
kind: Secret
metadata:
  name: redis-credentials
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
  password: "REPLACE_IN_DEPLOY"
---
# S3/Minio credentials for application-level access (model storage, data).
# NOT the same as minio-credentials (used by initContainers for binary fetch).
apiVersion: v1
kind: Secret
metadata:
  name: s3-credentials
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
  access-key: "REPLACE_IN_DEPLOY"
  secret-key: "REPLACE_IN_DEPLOY"
---
# Registry pull secret for Scaleway Container Registry
# Create with:
#   kubectl create secret docker-registry scw-registry \
#     --namespace foxhunt \
#     --docker-server=rg.fr-par.scw.cloud \
#     --docker-username=foxhunt \
#     --docker-password=<SCW_SECRET_KEY>

Step 2: Update secretKeyRef in all service deployments

For every service deployment, change foxhunt-secrets references to the new secret names:

db-password references (name: foxhunt-secrets, key: db-password → name: db-credentials, key: password):

• api-gateway.yaml
• trading-service.yaml (+ GPU overlay)
• backtesting-service.yaml
• ml-training-service.yaml (+ GPU overlay)
• trading-agent-service.yaml
• broker-gateway.yaml
• data-acquisition-service.yaml

jwt-secret references (name: foxhunt-secrets, key: jwt-secret → name: jwt-secret, key: secret):

• api-gateway.yaml
• web-gateway.yaml
• trading-service.yaml (+ GPU overlay)
• backtesting-service.yaml
• ml-training-service.yaml (+ GPU overlay)
• trading-agent-service.yaml
• broker-gateway.yaml

Step 3: Validate YAML syntax

for f in infra/k8s/secrets/foxhunt-secrets.yaml infra/k8s/services/*.yaml; do
  echo "--- $f ---"
  cat "$f" | python3 -c "import sys, yaml; [yaml.safe_load(d) for d in sys.stdin.read().split('---') if d.strip()]; print('OK')"
done

Expected: all files print OK.

Step 4: Commit

git add infra/k8s/secrets/foxhunt-secrets.yaml infra/k8s/services/
git commit -m "infra: split foxhunt-secrets into per-concern secrets

db-credentials, jwt-secret, redis-credentials, s3-credentials.
Each service only references the secrets it needs.
Values are REPLACE_IN_DEPLOY markers — real values
injected by scripts/deploy-secrets.sh."

---

Task 5: deploy-secrets.sh — Scaleway Secrets Manager Integration

Files:

• Create: scripts/deploy-secrets.sh

Context: The split secrets from Task 4 need real values at deploy time. This script reads from Scaleway Secrets Manager and creates K8s secrets. The Scaleway CLI (scw) can access secrets via scw secret version access-by-path.

Step 1: Create scripts/deploy-secrets.sh

Create scripts/deploy-secrets.sh:

#!/usr/bin/env bash
# deploy-secrets.sh — Populate K8s secrets from Scaleway Secrets Manager.
# Usage: ./scripts/deploy-secrets.sh [--dry-run]
#
# Prerequisites:
#   - scw CLI configured (SCW_ACCESS_KEY, SCW_SECRET_KEY, SCW_DEFAULT_PROJECT_ID)
#   - kubectl configured for target cluster
#   - Scaleway secrets exist at paths: foxhunt/db-password, foxhunt/jwt-secret, etc.
#
# Scaleway secret paths (create these in SCW console or CLI):
#   foxhunt/db-password
#   foxhunt/jwt-secret
#   foxhunt/redis-password
#   foxhunt/s3-access-key
#   foxhunt/s3-secret-key

set -euo pipefail

NAMESPACE="foxhunt"
DRY_RUN=""

if [[ "${1:-}" == "--dry-run" ]]; then
  DRY_RUN="--dry-run=client"
  echo "DRY RUN — no changes will be applied"
fi

# Fetch a secret value from Scaleway Secrets Manager.
# Usage: scw_secret <path>
scw_secret() {
  local path="$1"
  scw secret version access-by-path name="$path" field=data --output json 2>/dev/null \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['data'])" \
    || { echo "ERROR: Failed to fetch secret '$path' from Scaleway" >&2; exit 1; }
}

echo "Fetching secrets from Scaleway Secrets Manager..."

DB_PASSWORD=$(scw_secret "foxhunt/db-password")
JWT_SECRET=$(scw_secret "foxhunt/jwt-secret")
REDIS_PASSWORD=$(scw_secret "foxhunt/redis-password")
S3_ACCESS_KEY=$(scw_secret "foxhunt/s3-access-key")
S3_SECRET_KEY=$(scw_secret "foxhunt/s3-secret-key")

echo "Creating K8s secrets in namespace $NAMESPACE..."

# db-credentials
kubectl create secret generic db-credentials \
  --namespace="$NAMESPACE" \
  --from-literal=password="$DB_PASSWORD" \
  --save-config $DRY_RUN \
  -o yaml | kubectl apply -f - $DRY_RUN

# jwt-secret
kubectl create secret generic jwt-secret \
  --namespace="$NAMESPACE" \
  --from-literal=secret="$JWT_SECRET" \
  --save-config $DRY_RUN \
  -o yaml | kubectl apply -f - $DRY_RUN

# redis-credentials
kubectl create secret generic redis-credentials \
  --namespace="$NAMESPACE" \
  --from-literal=password="$REDIS_PASSWORD" \
  --save-config $DRY_RUN \
  -o yaml | kubectl apply -f - $DRY_RUN

# s3-credentials
kubectl create secret generic s3-credentials \
  --namespace="$NAMESPACE" \
  --from-literal=access-key="$S3_ACCESS_KEY" \
  --from-literal=secret-key="$S3_SECRET_KEY" \
  --save-config $DRY_RUN \
  -o yaml | kubectl apply -f - $DRY_RUN

echo "Done. Secrets created/updated in namespace $NAMESPACE."
echo ""
echo "Verify with: kubectl get secrets -n $NAMESPACE"

Step 2: Make executable

chmod +x scripts/deploy-secrets.sh

Step 3: Commit

git add scripts/deploy-secrets.sh
git commit -m "scripts: add deploy-secrets.sh for Scaleway Secrets Manager integration

Reads secrets from SCW and creates/updates K8s secrets.
Supports --dry-run for preview. Replaces manual YAML editing."

---

Task 6: CI Image Scanning — Trivy in GitLab CI

Files:

• Modify: .gitlab-ci.yml

Context: Currently 5 stages: prepare → test → compile → train → deploy. Add scan between prepare and test. Scan the 2 runtime images (foxhunt-runtime, foxhunt-training-runtime) that all service pods use. CI builder images are internal-only and don't need blocking scans.

Step 1: Add scan stage to stages list

In .gitlab-ci.yml, change the stages: block from:

stages:
  - prepare
  - test
  - compile
  - train
  - deploy

to:

stages:
  - prepare
  - scan
  - test
  - compile
  - train
  - deploy

Step 2: Add scan-runtime-images job

Add this job after the last prepare stage job (after build-foxhunt-training-runtime) and before the .rust-base template:

# --------------------------------------------------------------------------
# Stage 0.5: Scan runtime images for vulnerabilities (Trivy)
# --------------------------------------------------------------------------
scan-runtime-images:
  stage: scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  tags:
    - kapsule
    - docker
  variables:
    KUBERNETES_CPU_REQUEST: "200m"
    KUBERNETES_CPU_LIMIT: "500m"
    KUBERNETES_MEMORY_REQUEST: "256Mi"
    KUBERNETES_MEMORY_LIMIT: "512Mi"
  needs:
    - job: build-foxhunt-runtime
      optional: true
    - job: build-foxhunt-training-runtime
      optional: true
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
  cache:
    key: trivy-db
    paths:
      - .trivy-cache/
  before_script:
    # Authenticate to Scaleway Container Registry for image pull
    - export TRIVY_USERNAME=nologin
    - export TRIVY_PASSWORD=${SCW_SECRET_KEY}
  script:
    - echo "Scanning runtime images for CRITICAL/HIGH vulnerabilities..."
    - trivy image
      --cache-dir .trivy-cache
      --exit-code 1
      --severity CRITICAL,HIGH
      --ignore-unfixed
      --no-progress
      "${REGISTRY}/foxhunt-runtime:latest"
    - trivy image
      --cache-dir .trivy-cache
      --exit-code 1
      --severity CRITICAL,HIGH
      --ignore-unfixed
      --no-progress
      "${REGISTRY}/foxhunt-training-runtime:latest"
    - echo "Image scan passed — no fixable CRITICAL/HIGH vulnerabilities."
  allow_failure: false

Step 3: Validate CI YAML syntax

python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml')); print('OK')"

Expected: OK.

Step 4: Commit

git add .gitlab-ci.yml
git commit -m "ci: add Trivy image scanning for runtime images

New 'scan' stage between prepare and test.
Scans foxhunt-runtime and foxhunt-training-runtime
for CRITICAL/HIGH fixable CVEs. Pipeline blocks on findings.
Trivy DB cached between runs."

---

Task 7: Verify Everything

Step 1: Check all new files exist

echo "=== NetworkPolicy files ==="
ls -la infra/k8s/network-policies/
echo ""
echo "=== Secret manifests ==="
ls -la infra/k8s/secrets/
echo ""
echo "=== deploy-secrets.sh ==="
ls -la scripts/deploy-secrets.sh

Expected: 13 network policy files, 3 secret files, deploy-secrets.sh is executable.

Step 2: Validate all YAML syntax

for f in $(find infra/k8s -name '*.yaml'); do
  cat "$f" | python3 -c "
import sys, yaml
docs = [d for d in sys.stdin.read().split('---') if d.strip()]
for d in docs:
    yaml.safe_load(d)
print('OK: ' + '$f')
"
done

Expected: all files print OK.

Step 3: Count changes

git diff --stat HEAD~6..HEAD

Expected: ~20 new files, ~11 modified files. No Rust code changes.

Step 4: Verify CI stages

grep -A 10 '^stages:' .gitlab-ci.yml

Expected: prepare, scan, test, compile, train, deploy (6 stages).