foxhunt/docs/plans/2026-03-02-node-pool-split-design.md

# 3-Pool Node Split Design

## Problem

All foxhunt microservices, infrastructure databases, and monitoring share two DEV1-L nodes
in a single `services` pool. At idle, CPU utilization is 4-5% and memory 22-33%. There is
no isolation between platform infrastructure (postgres, redis) and application services
(trading-service, ib-gateway). A misbehaving foxhunt service could starve the database.

Meanwhile, monitoring (grafana, loki, tempo) sits on the gitlab node, consuming ~1 GiB RAM
that GitLab could use for Gitaly caching.

## Decision

Split into 3 purpose-specific node pools. Same total cost (EUR 61/mo), better isolation.

## Layout

| Pool | Instance | vCPU | RAM | Cost/mo | Workload |
|------|----------|------|-----|---------|----------|
| `gitlab` | GP1-XS | 4 | 16G | EUR 27 | GitLab CE (webservice, sidekiq ×2, gitaly, shell ×2, registry, toolbox, prometheus, runners ×2, pat-rotation) |
| `platform` | DEV1-L | 4 | 8G | EUR 17 | Postgres, Redis, MinIO, QuestDB, Grafana, Loki, Tempo, kube-state-metrics, tailscale-proxy |
| `foxhunt` | DEV1-L | 4 | 8G | EUR 17 | api-gateway, trading-service, broker-gateway, ib-gateway, ml-training-service, trading-agent-service, backtesting-service, data-acquisition-service, web-gateway |

CI pools (ci-compile-cpu, ci-training) are unchanged — autoscale 0→1.

## Resource Budget

### Platform node (DEV1-L: 3800m CPU / 6.4 GiB allocatable)

| Pod | CPU req | MEM req | MEM actual |
|-----|---------|---------|------------|
| postgres | 200m | 512Mi | ~800Mi |
| questdb | 200m | 512Mi | ~300Mi |
| redis | 100m | 256Mi | ~100Mi |
| minio | 100m | 256Mi | ~200Mi |
| grafana | — | — | ~190Mi |
| loki | 100m | 128Mi | ~140Mi |
| tempo | 100m | 256Mi | ~630Mi |
| kube-state-metrics | 25m | 64Mi | ~17Mi |
| tailscale-proxy | 25m | 32Mi | ~33Mi |
| **Total** | **~850m** | **~2 GiB** | **~2.4 GiB** |

Headroom: 3 GiB CPU, 4 GiB memory — comfortable.

### Foxhunt node (DEV1-L: 3800m CPU / 6.4 GiB allocatable)

| Pod | CPU req | MEM req | MEM actual |
|-----|---------|---------|------------|
| ib-gateway | 500m | 1Gi | ~500Mi |
| api-gateway | 200m | 256Mi | ~200Mi |
| trading-service | 200m | 256Mi | ~300Mi |
| broker-gateway | 200m | 256Mi | ~150Mi |
| backtesting-service | 200m | 256Mi | ~200Mi |
| trading-agent-service | 200m | 256Mi | ~200Mi |
| ml-training-service | 200m | 256Mi | ~200Mi |
| data-acquisition-service | 100m | 256Mi | ~150Mi |
| web-gateway | 100m | 128Mi | ~100Mi |
| **Total** | **~1.9G** | **~2.9 GiB** | **~2 GiB** |

Headroom: 1.9 GiB CPU, 3.5 GiB memory — room for trading burst.

## Migration Strategy

Brief downtime acceptable (no live trading). Order matters — databases first, then dependents.

### Phase 1: Create `platform` node pool
- Scaleway console or Terraform: DEV1-L, min=1, max=1, autoscale=off
- Wait for node to join cluster and become Ready

### Phase 2: Move stateless monitoring to `platform`
- Change nodeSelector from `gitlab` to `platform` for: grafana, loki, tempo, kube-state-metrics, tailscale-proxy
- `kubectl rollout restart` each — they reschedule to the platform node
- Verify Grafana accessible, Loki ingesting, Tempo receiving traces

### Phase 3: Move stateful infrastructure to `platform`
For each of postgres, questdb, minio (in this order):
1. Scale deployment to 0
2. Take backup (pg_dump for postgres, mc mirror for minio, questdb backup)
3. Delete old PVC (bound to old services node)
4. Create new PVC (will bind to platform node)
5. Restore data to new PVC via a temporary pod
6. Update deployment nodeSelector to `platform`
7. Scale back to 1
8. Verify service healthy

Redis: no persistent data that matters — just change nodeSelector and restart.

### Phase 4: Move foxhunt services
- Change nodeSelector from `services` to `foxhunt` for all 9 foxhunt deployments
- Delete binary-cache PVCs (disposable — services re-fetch from S3 on next init)
- `kubectl rollout restart` all services
- Verify all pods Running, health checks passing

### Phase 5: Rename/reconfigure services pool
- Option A: Rename existing `services` pool to `foxhunt` in Scaleway, scale to 1 node
- Option B: Create new `foxhunt` pool, drain old `services` pool, delete it
- Delete extra services node (was 2 nodes, now need 1)

### Phase 6: Verify
- All pods Running on correct nodes
- Grafana dashboards loading
- Loki queries working
- Network policies still enforced
- GitLab CI pipeline runs successfully

## Network Policies

No changes needed. All network policies use pod label selectors (`app.kubernetes.io/name`,
`app.kubernetes.io/part-of`), not node selectors. Cross-node pod communication works
transparently via Kapsule CNI (Cilium).

## PVC Handling

| PVC | Action |
|-----|--------|
| postgres-pvc (10Gi) | Migrate: pg_dump → new PVC → pg_restore |
| questdb-pvc (10Gi) | Migrate: copy data dir via temp pod |
| minio-data (20Gi) | Migrate: mc mirror to new PVC |
| grafana (2Gi) | Migrate: copy data dir (dashboards are provisioned, so loss is acceptable) |
| binary-cache-* (1Gi each, ×8) | Delete and recreate — populated from S3 on pod init |
| sccache/training PVCs | Unchanged — bound to CI/training pools |

## Risks

| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| PVC data loss | Low | High | pg_dump + mc mirror before any PVC deletion |
| Pod stuck Pending | Low | Medium | Verify node Ready before migration |
| Memory pressure on platform | Low | Medium | Tempo is the heaviest at 630Mi — can add limits |
| Cross-node latency | Negligible | Low | All nodes in same datacenter, <1ms |
| Scaleway pool creation fails | Low | None | Can abort, everything stays on old nodes |

## Rollback

If anything goes wrong mid-migration:
- Stateful services: restore from backup to original PVCs
- Stateless services: revert nodeSelector to `services`
- Delete the new `platform` pool
- Everything returns to pre-migration state