# Infrastructure Security Hardening — Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Harden K8s infrastructure for production: NetworkPolicy, SecurityContext, secret scoping, CI image scanning.

**Architecture:** 4 independent hardening layers applied to existing manifests in `infra/k8s/`. NetworkPolicy files are new; SecurityContext + secret scoping modify existing deployment YAMLs; Trivy scanning adds a new CI stage. No Rust code changes.

**Tech Stack:** Kubernetes NetworkPolicy, Pod SecurityContext, Scaleway Secrets Manager CLI (`scw`), Trivy container scanner, GitLab CI YAML.

---

### Task 1: NetworkPolicy — Default Deny + DNS Exemption

**Files:**
- Create: `infra/k8s/network-policies/default-deny.yaml`
- Create: `infra/k8s/network-policies/allow-dns.yaml`
- Create: `infra/k8s/network-policies/allow-monitoring.yaml`

**Context:** Currently zero NetworkPolicy files exist. All pods in namespace `foxhunt` communicate freely. We start with the foundation: deny-all + DNS + monitoring exemptions. Services will be blocked until Task 2 adds per-service rules.

**Step 1: Create the network-policies directory**

```bash
mkdir -p infra/k8s/network-policies
```

**Step 2: Create default-deny.yaml**

Create `infra/k8s/network-policies/default-deny.yaml`:

```yaml
# Default deny all ingress and egress in foxhunt namespace.
# Every service must declare its own NetworkPolicy to communicate.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
```

**Step 3: Create allow-dns.yaml**

Create `infra/k8s/network-policies/allow-dns.yaml`:

```yaml
# Allow all pods to reach CoreDNS (kube-system) for name resolution.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
```

**Step 4: Create allow-monitoring.yaml**

Create `infra/k8s/network-policies/allow-monitoring.yaml`:

```yaml
# Allow Prometheus (monitoring namespace) to scrape metrics ports on all foxhunt pods.
# Also allow all foxhunt pods to push traces to Tempo.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-monitoring-scrape
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - protocol: TCP
          port: 9091
        - protocol: TCP
          port: 9092
        - protocol: TCP
          port: 9093
        - protocol: TCP
          port: 9094
        - protocol: TCP
          port: 9095
        - protocol: TCP
          port: 9096
        - protocol: TCP
          port: 9097
        - protocol: TCP
          port: 9098
  egress:
    # Tempo OTLP gRPC
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: foxhunt
          podSelector:
            matchLabels:
              app.kubernetes.io/name: tempo
      ports:
        - protocol: TCP
          port: 4317
```

**Step 5: Verify YAML is valid**

```bash
# Quick syntax check — kubectl dry-run (no cluster needed)
for f in infra/k8s/network-policies/*.yaml; do
  echo "--- $f ---"
  cat "$f" | python3 -c "import sys, yaml; yaml.safe_load(sys.stdin.read()); print('OK')"
done
```

Expected: all 3 files print `OK`.

**Step 6: Commit**

```bash
git add infra/k8s/network-policies/
git commit -m "infra: add default-deny NetworkPolicy + DNS + monitoring exemptions"
```

---

### Task 2: NetworkPolicy — Per-Service Allow Rules

**Files:**
- Create: `infra/k8s/network-policies/api-gateway.yaml`
- Create: `infra/k8s/network-policies/web-gateway.yaml`
- Create: `infra/k8s/network-policies/trading-service.yaml`
- Create: `infra/k8s/network-policies/backtesting-service.yaml`
- Create: `infra/k8s/network-policies/ml-training-service.yaml`
- Create: `infra/k8s/network-policies/trading-agent-service.yaml`
- Create: `infra/k8s/network-policies/broker-gateway.yaml`
- Create: `infra/k8s/network-policies/data-acquisition-service.yaml`
- Create: `infra/k8s/network-policies/ib-gateway.yaml`
- Create: `infra/k8s/network-policies/infrastructure.yaml`

**Context:** Each service needs ingress (who can call it) and egress (what it can call). Labels used: `app.kubernetes.io/name` (most services) and `app: ib-gateway` (ib-gateway uses different label scheme). All initContainers need egress to minio:9000. Infrastructure pods (postgres, redis, minio, questdb) need ingress from services.

**Important label details** (read from existing manifests):
- Most services use `app.kubernetes.io/name: <service-name>` as their pod label
- ib-gateway uses `app: ib-gateway` as its pod label
- Postgres, Redis, Minio, QuestDB — assume labels `app: postgres`, `app: redis`, `app: minio`, `app: questdb`

**Step 1: Create api-gateway.yaml**

Create `infra/k8s/network-policies/api-gateway.yaml`:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-gateway
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: api-gateway
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Tailscale ingress proxy
    - from:
        - podSelector:
            matchLabels:
              app: tailscale
      ports:
        - protocol: TCP
          port: 50051
  egress:
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # Web gateway
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: web-gateway
      ports:
        - protocol: TCP
          port: 3000
    # Downstream gRPC services (for proxied requests)
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-service
      ports:
        - protocol: TCP
          port: 50051
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: backtesting-service
      ports:
        - protocol: TCP
          port: 50053
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: ml-training-service
      ports:
        - protocol: TCP
          port: 50053
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-agent-service
      ports:
        - protocol: TCP
          port: 50055
    # Minio (initContainer binary fetch)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000
```

**Step 2: Create web-gateway.yaml**

Create `infra/k8s/network-policies/web-gateway.yaml`:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: web-gateway
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: web-gateway
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # api-gateway proxies HTTP requests
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 3000
    # Tailscale direct access (dashboard)
    - from:
        - podSelector:
            matchLabels:
              app: tailscale
      ports:
        - protocol: TCP
          port: 3000
  egress:
    # api-gateway gRPC upstream
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 50051
    # Direct gRPC to downstream services (web-gateway env vars show these)
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: backtesting-service
      ports:
        - protocol: TCP
          port: 50053
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: ml-training-service
      ports:
        - protocol: TCP
          port: 50053
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-agent-service
      ports:
        - protocol: TCP
          port: 50055
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: broker-gateway
      ports:
        - protocol: TCP
          port: 50056
    # Minio (initContainer binary fetch)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000
```

**Step 3: Create trading-service.yaml**

Create `infra/k8s/network-policies/trading-service.yaml`:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: trading-service
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: trading-service
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-agent-service
      ports:
        - protocol: TCP
          port: 50051
  egress:
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # IB Gateway TWS API
    - to:
        - podSelector:
            matchLabels:
              app: ib-gateway
      ports:
        - protocol: TCP
          port: 4002
    # QuestDB ILP
    - to:
        - podSelector:
            matchLabels:
              app: questdb
      ports:
        - protocol: TCP
          port: 9009
    # Minio (initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000
```

**Step 4: Create backtesting-service.yaml**

Create `infra/k8s/network-policies/backtesting-service.yaml`:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backtesting-service
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: backtesting-service
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 50053
  egress:
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # Minio (data + initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000
```

**Step 5: Create ml-training-service.yaml**

Create `infra/k8s/network-policies/ml-training-service.yaml`:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ml-training-service
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: ml-training-service
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 50053
  egress:
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # Minio (model storage + initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000
    # Kubernetes API (for job creation — ml-training-service has RBAC for batch/jobs)
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
      ports:
        - protocol: TCP
          port: 443
```

**Step 6: Create trading-agent-service.yaml**

Create `infra/k8s/network-policies/trading-agent-service.yaml`:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: trading-agent-service
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: trading-agent-service
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 50055
  egress:
    # Trading service
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-service
      ports:
        - protocol: TCP
          port: 50051
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # Minio (initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000
```

**Step 7: Create broker-gateway.yaml**

Create `infra/k8s/network-policies/broker-gateway.yaml`:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: broker-gateway
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: broker-gateway
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-service
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: web-gateway
      ports:
        - protocol: TCP
          port: 50056
  egress:
    # IB Gateway TWS API
    - to:
        - podSelector:
            matchLabels:
              app: ib-gateway
      ports:
        - protocol: TCP
          port: 4002
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # Minio (initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000
```

**Step 8: Create data-acquisition-service.yaml**

Create `infra/k8s/network-policies/data-acquisition-service.yaml`:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: data-acquisition-service
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: data-acquisition-service
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: api-gateway
      ports:
        - protocol: TCP
          port: 50057
  egress:
    # Postgres
    - to:
        - podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    # Redis
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - protocol: TCP
          port: 6379
    # External HTTPS (Databento, Benzinga APIs)
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - protocol: TCP
          port: 443
    # Minio (initContainer)
    - to:
        - podSelector:
            matchLabels:
              app: minio
      ports:
        - protocol: TCP
          port: 9000
```

**Step 9: Create ib-gateway.yaml**

Create `infra/k8s/network-policies/ib-gateway.yaml`:

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ib-gateway
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app: ib-gateway
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Trading service + broker gateway connect to TWS API
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-service
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: broker-gateway
      ports:
        - protocol: TCP
          port: 4002
        - protocol: TCP
          port: 4004
  egress:
    # IBKR external servers (market data, order routing)
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - protocol: TCP
          port: 443
        - protocol: TCP
          port: 4000
        - protocol: TCP
          port: 4001
```

**Step 10: Create infrastructure.yaml (postgres, redis, minio, questdb)**

Create `infra/k8s/network-policies/infrastructure.yaml`:

```yaml
# Postgres: accepts connections from all foxhunt app pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: app.kubernetes.io/part-of
                operator: In
                values: ["foxhunt"]
      ports:
        - protocol: TCP
          port: 5432
---
# Redis: accepts connections from foxhunt app pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: redis
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app: redis
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: app.kubernetes.io/part-of
                operator: In
                values: ["foxhunt"]
      ports:
        - protocol: TCP
          port: 6379
---
# Minio: accepts connections from foxhunt app pods (S3 API)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: minio
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app: minio
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: app.kubernetes.io/part-of
                operator: In
                values: ["foxhunt"]
      ports:
        - protocol: TCP
          port: 9000
---
# QuestDB: accepts ILP writes from trading-service
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: questdb
  namespace: foxhunt
spec:
  podSelector:
    matchLabels:
      app: questdb
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: trading-service
      ports:
        - protocol: TCP
          port: 9009
```

**Step 11: Validate all YAML files**

```bash
for f in infra/k8s/network-policies/*.yaml; do
  echo "--- $f ---"
  cat "$f" | python3 -c "import sys, yaml; [yaml.safe_load(d) for d in sys.stdin.read().split('---') if d.strip()]; print('OK')"
done
```

Expected: all 10 files print `OK`.

**Step 12: Commit**

```bash
git add infra/k8s/network-policies/
git commit -m "infra: add per-service NetworkPolicy rules

9 service policies + infrastructure ingress rules.
Each service declares explicit ingress/egress.
External access limited to data-acquisition (HTTPS)
and ib-gateway (IBKR servers)."
```

---

### Task 3: SecurityContext — Harden All Service Deployments

**Files:**
- Modify: `infra/k8s/services/api-gateway.yaml`
- Modify: `infra/k8s/services/web-gateway.yaml`
- Modify: `infra/k8s/services/trading-service.yaml`
- Modify: `infra/k8s/services/backtesting-service.yaml`
- Modify: `infra/k8s/services/ml-training-service.yaml`
- Modify: `infra/k8s/services/trading-agent-service.yaml`
- Modify: `infra/k8s/services/broker-gateway.yaml`
- Modify: `infra/k8s/services/data-acquisition-service.yaml`
- Modify: `infra/k8s/services/ib-gateway.yaml`
- Modify: `infra/k8s/services/trading-service-gpu.yaml`
- Modify: `infra/k8s/services/ml-training-service-gpu.yaml`

**Context:** All 9 service deployments + 2 GPU overlays need pod-level and container-level securityContext. The Dockerfiles already create user `foxhunt` with UID 1000. InitContainers keep `runAsUser: 0` (binary fetch needs root). ib-gateway is third-party (Java) and cannot use `readOnlyRootFilesystem`.

**For each of the 9 CPU service manifests** (api-gateway, web-gateway, trading-service, backtesting-service, ml-training-service, trading-agent-service, broker-gateway, data-acquisition-service), apply these changes to the Deployment spec:

**Step 1: Add pod-level securityContext**

In each deployment's `spec.template.spec`, add immediately after `nodeSelector` (or after `serviceAccountName` for ml-training-service):

```yaml
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
```

**Step 2: Add container-level securityContext to main container**

In each deployment's main container (the one under `containers:`), add a `securityContext` block:

```yaml
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
```

**Step 3: Add /tmp emptyDir volume**

Add to the `volumes:` section of each deployment:

```yaml
        - name: tmp
          emptyDir:
            sizeLimit: 50Mi
```

And add to the main container's `volumeMounts:`:

```yaml
            - name: tmp
              mountPath: /tmp
```

**Step 4: Handle ib-gateway differently**

For `infra/k8s/services/ib-gateway.yaml`, add pod-level securityContext but **without** `runAsNonRoot` (third-party image manages its own user), and **without** `readOnlyRootFilesystem` (Java needs writes):

```yaml
    spec:
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: ib-gateway
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
```

**Step 5: Apply same changes to GPU overlay files**

For `infra/k8s/services/trading-service-gpu.yaml` and `infra/k8s/services/ml-training-service-gpu.yaml`, apply the same pod-level and container-level securityContext as their CPU counterparts.

**Step 6: Validate YAML syntax**

```bash
for f in infra/k8s/services/*.yaml; do
  echo "--- $f ---"
  cat "$f" | python3 -c "import sys, yaml; [yaml.safe_load(d) for d in sys.stdin.read().split('---') if d.strip()]; print('OK')"
done
```

Expected: all 11 files print `OK`.

**Step 7: Commit**

```bash
git add infra/k8s/services/
git commit -m "infra: add SecurityContext to all service deployments

Pod-level: runAsNonRoot, runAsUser/Group 1000, seccomp RuntimeDefault.
Container-level: no privilege escalation, read-only rootfs, drop ALL caps.
Exception: ib-gateway skips runAsNonRoot and readOnlyRootFilesystem.
InitContainers keep runAsUser: 0 for binary fetch."
```

---

### Task 4: Secret Scoping — Split foxhunt-secrets

**Files:**
- Modify: `infra/k8s/secrets/foxhunt-secrets.yaml` → replace with 4 scoped secrets
- Modify: `infra/k8s/services/api-gateway.yaml` (secretKeyRef names)
- Modify: `infra/k8s/services/web-gateway.yaml`
- Modify: `infra/k8s/services/trading-service.yaml`
- Modify: `infra/k8s/services/backtesting-service.yaml`
- Modify: `infra/k8s/services/ml-training-service.yaml`
- Modify: `infra/k8s/services/trading-agent-service.yaml`
- Modify: `infra/k8s/services/broker-gateway.yaml`
- Modify: `infra/k8s/services/data-acquisition-service.yaml`
- Modify: `infra/k8s/services/trading-service-gpu.yaml`
- Modify: `infra/k8s/services/ml-training-service-gpu.yaml`

**Context:** Currently one `foxhunt-secrets` Opaque secret holds db-password, jwt-secret, redis-password, s3-access-key, s3-secret-key. All services reference it even if they only need 1-2 keys. Splitting into per-concern secrets reduces blast radius.

**Step 1: Replace foxhunt-secrets.yaml with scoped secrets**

Replace the contents of `infra/k8s/secrets/foxhunt-secrets.yaml` with:

```yaml
# Database credentials — used by services that connect to PostgreSQL.
# Real values injected by scripts/deploy-secrets.sh from Scaleway Secrets Manager.
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
  password: "REPLACE_IN_DEPLOY"
---
# JWT signing secret — used by services that validate/issue JWTs.
apiVersion: v1
kind: Secret
metadata:
  name: jwt-secret
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
  secret: "REPLACE_IN_DEPLOY"
---
# Redis credentials — used by services with Redis connections.
apiVersion: v1
kind: Secret
metadata:
  name: redis-credentials
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
  password: "REPLACE_IN_DEPLOY"
---
# S3/Minio credentials for application-level access (model storage, data).
# NOT the same as minio-credentials (used by initContainers for binary fetch).
apiVersion: v1
kind: Secret
metadata:
  name: s3-credentials
  namespace: foxhunt
  labels:
    app.kubernetes.io/part-of: foxhunt
type: Opaque
stringData:
  access-key: "REPLACE_IN_DEPLOY"
  secret-key: "REPLACE_IN_DEPLOY"
---
# Registry pull secret for Scaleway Container Registry
# Create with:
#   kubectl create secret docker-registry scw-registry \
#     --namespace foxhunt \
#     --docker-server=rg.fr-par.scw.cloud \
#     --docker-username=foxhunt \
#     --docker-password=<SCW_SECRET_KEY>
```

**Step 2: Update secretKeyRef in all service deployments**

For every service deployment, change `foxhunt-secrets` references to the new secret names:

**db-password references** (`name: foxhunt-secrets, key: db-password` → `name: db-credentials, key: password`):
- api-gateway.yaml
- trading-service.yaml (+ GPU overlay)
- backtesting-service.yaml
- ml-training-service.yaml (+ GPU overlay)
- trading-agent-service.yaml
- broker-gateway.yaml
- data-acquisition-service.yaml

**jwt-secret references** (`name: foxhunt-secrets, key: jwt-secret` → `name: jwt-secret, key: secret`):
- api-gateway.yaml
- web-gateway.yaml
- trading-service.yaml (+ GPU overlay)
- backtesting-service.yaml
- ml-training-service.yaml (+ GPU overlay)
- trading-agent-service.yaml
- broker-gateway.yaml

**Step 3: Validate YAML syntax**

```bash
for f in infra/k8s/secrets/foxhunt-secrets.yaml infra/k8s/services/*.yaml; do
  echo "--- $f ---"
  cat "$f" | python3 -c "import sys, yaml; [yaml.safe_load(d) for d in sys.stdin.read().split('---') if d.strip()]; print('OK')"
done
```

Expected: all files print `OK`.

**Step 4: Commit**

```bash
git add infra/k8s/secrets/foxhunt-secrets.yaml infra/k8s/services/
git commit -m "infra: split foxhunt-secrets into per-concern secrets

db-credentials, jwt-secret, redis-credentials, s3-credentials.
Each service only references the secrets it needs.
Values are REPLACE_IN_DEPLOY markers — real values
injected by scripts/deploy-secrets.sh."
```

---

### Task 5: deploy-secrets.sh — Scaleway Secrets Manager Integration

**Files:**
- Create: `scripts/deploy-secrets.sh`

**Context:** The split secrets from Task 4 need real values at deploy time. This script reads from Scaleway Secrets Manager and creates K8s secrets. The Scaleway CLI (`scw`) can access secrets via `scw secret version access-by-path`.

**Step 1: Create scripts/deploy-secrets.sh**

Create `scripts/deploy-secrets.sh`:

```bash
#!/usr/bin/env bash
# deploy-secrets.sh — Populate K8s secrets from Scaleway Secrets Manager.
# Usage: ./scripts/deploy-secrets.sh [--dry-run]
#
# Prerequisites:
#   - scw CLI configured (SCW_ACCESS_KEY, SCW_SECRET_KEY, SCW_DEFAULT_PROJECT_ID)
#   - kubectl configured for target cluster
#   - Scaleway secrets exist at paths: foxhunt/db-password, foxhunt/jwt-secret, etc.
#
# Scaleway secret paths (create these in SCW console or CLI):
#   foxhunt/db-password
#   foxhunt/jwt-secret
#   foxhunt/redis-password
#   foxhunt/s3-access-key
#   foxhunt/s3-secret-key

set -euo pipefail

NAMESPACE="foxhunt"
DRY_RUN=""

if [[ "${1:-}" == "--dry-run" ]]; then
  DRY_RUN="--dry-run=client"
  echo "DRY RUN — no changes will be applied"
fi

# Fetch a secret value from Scaleway Secrets Manager.
# Usage: scw_secret <path>
scw_secret() {
  local path="$1"
  scw secret version access-by-path name="$path" field=data --output json 2>/dev/null \
    | python3 -c "import sys,json; print(json.load(sys.stdin)['data'])" \
    || { echo "ERROR: Failed to fetch secret '$path' from Scaleway" >&2; exit 1; }
}

echo "Fetching secrets from Scaleway Secrets Manager..."

DB_PASSWORD=$(scw_secret "foxhunt/db-password")
JWT_SECRET=$(scw_secret "foxhunt/jwt-secret")
REDIS_PASSWORD=$(scw_secret "foxhunt/redis-password")
S3_ACCESS_KEY=$(scw_secret "foxhunt/s3-access-key")
S3_SECRET_KEY=$(scw_secret "foxhunt/s3-secret-key")

echo "Creating K8s secrets in namespace $NAMESPACE..."

# db-credentials
kubectl create secret generic db-credentials \
  --namespace="$NAMESPACE" \
  --from-literal=password="$DB_PASSWORD" \
  --save-config $DRY_RUN \
  -o yaml | kubectl apply -f - $DRY_RUN

# jwt-secret
kubectl create secret generic jwt-secret \
  --namespace="$NAMESPACE" \
  --from-literal=secret="$JWT_SECRET" \
  --save-config $DRY_RUN \
  -o yaml | kubectl apply -f - $DRY_RUN

# redis-credentials
kubectl create secret generic redis-credentials \
  --namespace="$NAMESPACE" \
  --from-literal=password="$REDIS_PASSWORD" \
  --save-config $DRY_RUN \
  -o yaml | kubectl apply -f - $DRY_RUN

# s3-credentials
kubectl create secret generic s3-credentials \
  --namespace="$NAMESPACE" \
  --from-literal=access-key="$S3_ACCESS_KEY" \
  --from-literal=secret-key="$S3_SECRET_KEY" \
  --save-config $DRY_RUN \
  -o yaml | kubectl apply -f - $DRY_RUN

echo "Done. Secrets created/updated in namespace $NAMESPACE."
echo ""
echo "Verify with: kubectl get secrets -n $NAMESPACE"
```

**Step 2: Make executable**

```bash
chmod +x scripts/deploy-secrets.sh
```

**Step 3: Commit**

```bash
git add scripts/deploy-secrets.sh
git commit -m "scripts: add deploy-secrets.sh for Scaleway Secrets Manager integration

Reads secrets from SCW and creates/updates K8s secrets.
Supports --dry-run for preview. Replaces manual YAML editing."
```

---

### Task 6: CI Image Scanning — Trivy in GitLab CI

**Files:**
- Modify: `.gitlab-ci.yml`

**Context:** Currently 5 stages: prepare → test → compile → train → deploy. Add `scan` between `prepare` and `test`. Scan the 2 runtime images (foxhunt-runtime, foxhunt-training-runtime) that all service pods use. CI builder images are internal-only and don't need blocking scans.

**Step 1: Add `scan` stage to stages list**

In `.gitlab-ci.yml`, change the `stages:` block from:

```yaml
stages:
  - prepare
  - test
  - compile
  - train
  - deploy
```

to:

```yaml
stages:
  - prepare
  - scan
  - test
  - compile
  - train
  - deploy
```

**Step 2: Add scan-runtime-images job**

Add this job after the last `prepare` stage job (after `build-foxhunt-training-runtime`) and before the `.rust-base` template:

```yaml
# --------------------------------------------------------------------------
# Stage 0.5: Scan runtime images for vulnerabilities (Trivy)
# --------------------------------------------------------------------------
scan-runtime-images:
  stage: scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  tags:
    - kapsule
    - docker
  variables:
    KUBERNETES_CPU_REQUEST: "200m"
    KUBERNETES_CPU_LIMIT: "500m"
    KUBERNETES_MEMORY_REQUEST: "256Mi"
    KUBERNETES_MEMORY_LIMIT: "512Mi"
  needs:
    - job: build-foxhunt-runtime
      optional: true
    - job: build-foxhunt-training-runtime
      optional: true
  rules:
    - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
    - if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "api"
  cache:
    key: trivy-db
    paths:
      - .trivy-cache/
  before_script:
    # Authenticate to Scaleway Container Registry for image pull
    - export TRIVY_USERNAME=nologin
    - export TRIVY_PASSWORD=${SCW_SECRET_KEY}
  script:
    - echo "Scanning runtime images for CRITICAL/HIGH vulnerabilities..."
    - trivy image
      --cache-dir .trivy-cache
      --exit-code 1
      --severity CRITICAL,HIGH
      --ignore-unfixed
      --no-progress
      "${REGISTRY}/foxhunt-runtime:latest"
    - trivy image
      --cache-dir .trivy-cache
      --exit-code 1
      --severity CRITICAL,HIGH
      --ignore-unfixed
      --no-progress
      "${REGISTRY}/foxhunt-training-runtime:latest"
    - echo "Image scan passed — no fixable CRITICAL/HIGH vulnerabilities."
  allow_failure: false
```

**Step 3: Validate CI YAML syntax**

```bash
python3 -c "import yaml; yaml.safe_load(open('.gitlab-ci.yml')); print('OK')"
```

Expected: `OK`.

**Step 4: Commit**

```bash
git add .gitlab-ci.yml
git commit -m "ci: add Trivy image scanning for runtime images

New 'scan' stage between prepare and test.
Scans foxhunt-runtime and foxhunt-training-runtime
for CRITICAL/HIGH fixable CVEs. Pipeline blocks on findings.
Trivy DB cached between runs."
```

---

### Task 7: Verify Everything

**Step 1: Check all new files exist**

```bash
echo "=== NetworkPolicy files ==="
ls -la infra/k8s/network-policies/
echo ""
echo "=== Secret manifests ==="
ls -la infra/k8s/secrets/
echo ""
echo "=== deploy-secrets.sh ==="
ls -la scripts/deploy-secrets.sh
```

Expected: 13 network policy files, 3 secret files, deploy-secrets.sh is executable.

**Step 2: Validate all YAML syntax**

```bash
for f in $(find infra/k8s -name '*.yaml'); do
  cat "$f" | python3 -c "
import sys, yaml
docs = [d for d in sys.stdin.read().split('---') if d.strip()]
for d in docs:
    yaml.safe_load(d)
print('OK: ' + '$f')
"
done
```

Expected: all files print `OK`.

**Step 3: Count changes**

```bash
git diff --stat HEAD~6..HEAD
```

Expected: ~20 new files, ~11 modified files. No Rust code changes.

**Step 4: Verify CI stages**

```bash
grep -A 10 '^stages:' .gitlab-ci.yml
```

Expected: prepare, scan, test, compile, train, deploy (6 stages).
