Compare commits
13 Commits
chore/gite
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8213f29a5b | ||
|
|
bf9a6d20ee | ||
|
|
ffc83da643 | ||
|
|
b017daf04b | ||
|
|
b566b2c194 | ||
|
|
c6d0d2ab38 | ||
|
|
e86384ff99 | ||
|
|
35e4078989 | ||
|
|
9a7707c749 | ||
|
|
f0c4de66ad | ||
|
|
ad3637303f | ||
|
|
c284158dfb | ||
|
|
bf9558539a |
191
docs/superpowers/plans/2026-06-21-decommission-gitlab.md
Normal file
191
docs/superpowers/plans/2026-06-21-decommission-gitlab.md
Normal file
@@ -0,0 +1,191 @@
|
||||
# Decommission GitLab (compute) — Implementation Plan (Phase 2C)
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
|
||||
> **Do NOT run subagent-driven** — irreversible helm uninstall + a proxy edit that can take down all
|
||||
> `*.fxhnt.ai` if mis-ordered; each step needs a human-confirmed gate. Steps use checkbox (`- [ ]`) syntax.
|
||||
|
||||
**Goal:** Remove GitLab's compute (2 helm releases, dead Argo events, the `:5050` proxy block, the values
|
||||
manifest) while retaining all GitLab data as a cold backup.
|
||||
|
||||
**Architecture:** Proxy `:5050` block removed FIRST (defuses the nginx-startup landmine), then helm
|
||||
uninstall, then dead-event + manifest cleanup. gitaly PVC + GitLab buckets + secrets are left untouched.
|
||||
|
||||
**Tech Stack:** helm, kubectl, nginx+socat Tailscale proxy, Argo Events.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-06-21-decommission-gitlab-design.md`
|
||||
|
||||
---
|
||||
|
||||
## Task 0: Branch
|
||||
- [ ] **Step 1**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt && git checkout -b chore/decommission-gitlab
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Pre-flight — confirm no live GitLab consumers + baseline
|
||||
|
||||
- [ ] **Step 1: Confirm nothing pulls the GitLab image / uses the GitLab backend anymore**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
echo "pods on gitlab-registry image (expect none):"
|
||||
kubectl get pods -n foxhunt -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' 2>/dev/null | grep -c "gitlab-registry.foxhunt.svc.cluster.local:5000" | xargs echo
|
||||
echo "git remotes (expect git.fxhnt.ai = Gitea, not :2222/root):"
|
||||
git remote get-url origin; (cd /home/jgrusewski/Work/fxhnt && git remote get-url origin)
|
||||
```
|
||||
Expected: `0` pods on the GitLab image; both remotes `ssh://git@git.fxhnt.ai/gitadmin/...`. **STOP if any
|
||||
pod still references the GitLab registry.**
|
||||
|
||||
- [ ] **Step 2: Baseline RAM + pod count**
|
||||
```bash
|
||||
kubectl top pods -n foxhunt 2>/dev/null | grep -i gitlab | awk '{s+=$3} END{print NR" gitlab pods, "s" Mi"}'
|
||||
```
|
||||
Expected: ~12 pods, ~4.6 GiB (this is what gets reclaimed).
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Remove the `:5050` registry block from the proxy (FIRST — safety-critical)
|
||||
|
||||
**Files:** Modify `infra/k8s/gitlab/tailscale-proxy.yaml`
|
||||
|
||||
- [ ] **Step 1: Delete `containerPort: 5050`**
|
||||
|
||||
In `infra/k8s/gitlab/tailscale-proxy.yaml`, remove the line:
|
||||
```
|
||||
- containerPort: 5050
|
||||
```
|
||||
(leave `- containerPort: 443`).
|
||||
|
||||
- [ ] **Step 2: Delete the entire `# Container Registry — git.fxhnt.ai:5050` server block**
|
||||
|
||||
Remove the whole block (the `# Container Registry — git.fxhnt.ai:5050` comment + its `server { listen 5050
|
||||
ssl; ... }`, i.e. the `proxy_pass http://gitlab-registry.foxhunt.svc.cluster.local:5000;` server). Leave
|
||||
the `git.fxhnt.ai:443` (Gitea) block and all other server blocks intact.
|
||||
|
||||
- [ ] **Step 3: Confirm no GitLab upstream remains in the proxy**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
grep -nE "gitlab-registry|gitlab-webservice|gitlab-gitlab-shell|listen 5050" infra/k8s/gitlab/tailscale-proxy.yaml || echo "OK: no gitlab upstreams left"
|
||||
```
|
||||
Expected: `OK: no gitlab upstreams left`.
|
||||
|
||||
- [ ] **Step 4: Apply + restart proxy**
|
||||
```bash
|
||||
kubectl apply -f infra/k8s/gitlab/tailscale-proxy.yaml
|
||||
kubectl rollout restart deploy/tailscale-gitlab-proxy -n foxhunt
|
||||
kubectl rollout status deploy/tailscale-gitlab-proxy -n foxhunt --timeout=120s
|
||||
```
|
||||
|
||||
- [ ] **Step 5: GATE — proxy still serves all `*.fxhnt.ai`**
|
||||
```bash
|
||||
for h in git dashboard grafana minio; do curl -sk -m 10 -o /dev/null -w "$h.fxhnt.ai: %{http_code}\n" https://$h.fxhnt.ai/ 2>&1; done
|
||||
curl -sk -m 10 https://git.fxhnt.ai/api/healthz 2>/dev/null | grep -o '"status": "pass"'
|
||||
```
|
||||
Expected: `git`/`dashboard` 200 (others may 200/302/401 — just not 000). `"status": "pass"` for Gitea.
|
||||
**STOP + revert (`git checkout` the file + re-apply) if git/dashboard go 000.**
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Helm-uninstall GitLab + agent
|
||||
|
||||
- [ ] **Step 1: Uninstall both releases**
|
||||
```bash
|
||||
helm uninstall gitlab -n foxhunt 2>&1 | tail -1
|
||||
helm uninstall foxhunt-agent -n foxhunt 2>&1 | tail -1
|
||||
```
|
||||
Expected: both `uninstalled`.
|
||||
|
||||
- [ ] **Step 2: Wait for GitLab pods to terminate + confirm RAM reclaimed**
|
||||
```bash
|
||||
i=0; until [ -z "$(kubectl get pods -n foxhunt 2>/dev/null | grep -iE 'gitlab|gitaly' | grep -v Terminating)" ] || [ $i -ge 24 ]; do sleep 5; i=$((i+1)); done
|
||||
echo "remaining gitlab pods:"; kubectl get pods -n foxhunt 2>/dev/null | grep -iE "gitlab|gitaly" || echo " (none)"
|
||||
```
|
||||
Expected: no GitLab pods (or only `Terminating` briefly). The `gitaly` PVC remains (cold backup).
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Delete dead Argo Events
|
||||
|
||||
- [ ] **Step 1: Delete the GitLab CI eventsource + sensor**
|
||||
```bash
|
||||
kubectl delete eventsource gitlab-push -n foxhunt 2>&1 | tail -1
|
||||
kubectl delete sensor ci-pipeline-trigger -n foxhunt 2>&1 | tail -1
|
||||
echo "=== remaining events (gitea-push + gitea-deploy must stay) ==="
|
||||
kubectl get eventsource,sensor -n foxhunt 2>/dev/null
|
||||
```
|
||||
Expected: both deleted; `gitea-push` eventsource + `gitea-deploy` sensor still present (+ workflow-trigger).
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Verify cold backup intact + remove values manifest + commit
|
||||
|
||||
- [ ] **Step 1: Confirm the cold backup is untouched**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
echo "gitaly PVC (must remain):"; kubectl get pvc -n foxhunt 2>/dev/null | grep gitaly || echo " MISSING!"
|
||||
echo "gitlab secrets (sample, must remain):"; kubectl get secret -n foxhunt 2>/dev/null | grep -c gitlab | xargs echo " gitlab-* secrets:"
|
||||
export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
|
||||
export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
|
||||
export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
|
||||
echo "Scaleway gitlab buckets (must remain):"; scw object bucket list 2>/dev/null | grep -i gitlab
|
||||
```
|
||||
Expected: gitaly PVC present; gitlab secrets count > 0; both Scaleway gitlab buckets present. **STOP if the
|
||||
gitaly PVC is gone** (helm uninstall should not have removed a StatefulSet PVC — investigate).
|
||||
|
||||
- [ ] **Step 2: Remove the GitLab Helm values manifest**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
git rm infra/k8s/gitlab/values.yaml 2>&1 | tail -1
|
||||
ls infra/k8s/gitlab/ # tailscale-proxy.yaml must remain
|
||||
```
|
||||
Expected: `values.yaml` removed; `tailscale-proxy.yaml` still listed.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
```bash
|
||||
git add -A infra/k8s/gitlab/
|
||||
git commit -m "chore(infra): decommission GitLab compute — keep data as cold backup (Phase 2C)
|
||||
|
||||
helm uninstall gitlab + foxhunt-agent (~4.6GiB RAM / 12 pods freed); removed :5050
|
||||
registry block from the (now Gitea) proxy; deleted dead gitlab-push/ci-pipeline-trigger
|
||||
Argo events; removed gitlab values.yaml. RETAINED (cold backup): gitaly PVC,
|
||||
gitlab MinIO/Scaleway buckets, gitlab-* secrets. tailscale-proxy.yaml kept (serves Gitea).
|
||||
|
||||
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Final verification
|
||||
|
||||
- [ ] **Step 1: Releases gone + platform healthy + auto-deploy still works**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
echo "=== helm (no gitlab/agent) ==="; helm list -n foxhunt | grep -iE "gitlab" || echo " (none — removed)"
|
||||
echo "=== platform health ==="
|
||||
curl -sk -m 10 -o /dev/null -w "git.fxhnt.ai: %{http_code}\n" https://git.fxhnt.ai/api/healthz
|
||||
curl -sk -m 10 -o /dev/null -w "dashboard.fxhnt.ai: %{http_code}\n" https://dashboard.fxhnt.ai
|
||||
echo "=== auto-deploy still wired (push test) ==="
|
||||
BEFORE=$(kubectl get wf -n foxhunt 2>/dev/null | grep -c fxhnt-cockpit)
|
||||
( cd /home/jgrusewski/Work/fxhnt && git commit --allow-empty -m "test: post-GitLab-removal auto-deploy" -q && GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin master 2>&1 | tail -1 )
|
||||
i=0; until [ "$(kubectl get wf -n foxhunt 2>/dev/null | grep -c fxhnt-cockpit)" -gt "$BEFORE" ] || [ $i -ge 12 ]; do sleep 5; i=$((i+1)); done
|
||||
echo "cockpit wfs: $BEFORE -> $(kubectl get wf -n foxhunt 2>/dev/null | grep -c fxhnt-cockpit)"
|
||||
```
|
||||
Expected: no gitlab helm releases; `git`/`dashboard` 200; cockpit wf count incremented (webhook→Argo intact).
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
- **Proxy (Task 2):** `git checkout infra/k8s/gitlab/tailscale-proxy.yaml` + `kubectl apply` + rollout restart.
|
||||
- **GitLab (Task 3):** data is retained (gitaly PVC + buckets + secrets) — `helm install gitlab gitlab/gitlab
|
||||
--version 9.9.3 -f <values>` would re-attach the gitaly PVC. (Primary going forward is Gitea/Scaleway.)
|
||||
|
||||
---
|
||||
|
||||
## Acceptance criteria (from spec)
|
||||
- No `gitlab`/`foxhunt-agent` helm releases; GitLab pods gone (~4.6 GiB reclaimed). ✅ Task 3, Task 6
|
||||
- `git.fxhnt.ai` + `dashboard.fxhnt.ai` 200; push auto-deploys via Gitea→Argo. ✅ Task 6
|
||||
- `gitlab-push`/`ci-pipeline-trigger` deleted; `gitea-push`/`gitea-deploy` kept. ✅ Task 4
|
||||
- `:5050` block gone; proxy healthy. ✅ Task 2
|
||||
- Cold backup intact: gitaly PVC + GitLab buckets + `gitlab-*` secrets. ✅ Task 5
|
||||
- `values.yaml` removed; `tailscale-proxy.yaml` kept; committed. ✅ Task 5
|
||||
206
docs/superpowers/plans/2026-06-21-relocate-iac-to-fxhnt.md
Normal file
206
docs/superpowers/plans/2026-06-21-relocate-iac-to-fxhnt.md
Normal file
@@ -0,0 +1,206 @@
|
||||
# Relocate Platform IaC foxhunt → fxhnt + Archive foxhunt — Implementation Plan (Phase 2D)
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans (INLINE, with checkpoints).
|
||||
> Spans TWO repos + a production Terraform state gate; the `terragrunt plan = No changes` check is the
|
||||
> safety pin. Steps use checkbox (`- [ ]`) syntax.
|
||||
|
||||
**Goal:** Move active platform IaC (terraform `live`+`modules`, platform k8s manifests) from `foxhunt`
|
||||
into `fxhnt`, preserving TF state keys (no migration), then archive `foxhunt`.
|
||||
|
||||
**Architecture:** Copy into fxhnt preserving the `live/production/<module>` structure → state keys stay
|
||||
identical → `terragrunt plan = No changes` proves it. Then `git rm` from foxhunt, commit/push both,
|
||||
archive foxhunt. Cluster untouched throughout (source-of-truth move only).
|
||||
|
||||
**Tech Stack:** terragrunt + OpenTofu (Scaleway s3 backend), kubectl manifests, git, Gitea.
|
||||
|
||||
**Repos:** source `~/Work/foxhunt`, dest `~/Work/fxhnt`. **Spec:**
|
||||
`docs/superpowers/specs/2026-06-21-relocate-iac-to-fxhnt-design.md`
|
||||
|
||||
---
|
||||
|
||||
## Shared env (terragrunt steps)
|
||||
```bash
|
||||
export TG_TF_PATH=tofu TERRAGRUNT_TFPATH=tofu
|
||||
export SCW_ACCESS_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.access-key}'|base64 -d)
|
||||
export SCW_SECRET_KEY=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.secret-key}'|base64 -d)
|
||||
export SCW_DEFAULT_PROJECT_ID=$(kubectl get secret scaleway-credentials -n foxhunt -o jsonpath='{.data.project-id}'|base64 -d)
|
||||
export SCW_DEFAULT_REGION=fr-par SCW_DEFAULT_ZONE=fr-par-2
|
||||
export AWS_ACCESS_KEY_ID="$SCW_ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SCW_SECRET_KEY" AWS_REGION=fr-par
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 0: Branches
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/fxhnt && git checkout master && git checkout -b chore/relocate-iac
|
||||
cd /home/jgrusewski/Work/foxhunt && git checkout main && git checkout -b chore/relocate-iac-remove
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Collision pre-flight (must be empty)
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
for d in k8s/network-policies k8s/services k8s/jobs k8s/storage; do
|
||||
echo "--- $d ---"; comm -12 <(ls infra/$d 2>/dev/null|sort) <(ls ~/Work/fxhnt/infra/$d 2>/dev/null|sort)
|
||||
done
|
||||
```
|
||||
Expected: no filenames printed. **STOP + resolve if any collision appears.**
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Copy Terraform (live + modules) into fxhnt, structure preserved
|
||||
|
||||
- [ ] **Step 1: rsync source files (exclude generated/cached, keep .terraform.lock.hcl)**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
rsync -a --exclude '.terraform/' --exclude 'backend.tf' --exclude 'provider.tf' --exclude '*.tfstate*' \
|
||||
infra/live infra/modules /home/jgrusewski/Work/fxhnt/infra/
|
||||
find /home/jgrusewski/Work/fxhnt/infra/live /home/jgrusewski/Work/fxhnt/infra/modules -name '*.hcl' -o -name '*.tf' | sort
|
||||
```
|
||||
Expected: `fxhnt/infra/live/production/{root.hcl,kapsule,public-gateway,dns,block-storage}` +
|
||||
`fxhnt/infra/modules/{kapsule,public-gateway,dns,block-storage}` present.
|
||||
|
||||
- [ ] **Step 2: Ensure fxhnt .gitignore covers terragrunt-generated files**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/fxhnt
|
||||
grep -qE '^backend\.tf' .gitignore || printf '\n# terragrunt-generated\n**/.terraform/\n**/backend.tf\n**/provider.tf\n*.tfstate\n*.tfstate.*\n' >> .gitignore
|
||||
tail -7 .gitignore
|
||||
```
|
||||
|
||||
- [ ] **Step 3: GATE — terragrunt plan = No changes for all 4 modules (from fxhnt)**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/fxhnt
|
||||
for m in kapsule public-gateway dns block-storage; do
|
||||
echo "=== $m ==="
|
||||
( cd infra/live/production/$m && terragrunt init -reconfigure -input=false -no-color >/dev/null 2>&1 && \
|
||||
terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:|Error" | head -1 )
|
||||
done
|
||||
```
|
||||
Expected: every module prints `No changes.` — proves state keys are unchanged (no migration). **STOP if any
|
||||
module shows a plan diff or Error** (structure not preserved → do not proceed).
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Copy platform k8s manifests + the one platform script into fxhnt
|
||||
|
||||
- [ ] **Step 1: rsync platform k8s dirs (everything except gpu-taint.yaml) + namespace.yaml**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
for d in argo cert-manager databases dns gitea gitlab jobs mattermost minio monitoring network-policies secrets services stalwart tailscale training storage; do
|
||||
rsync -a "infra/k8s/$d" /home/jgrusewski/Work/fxhnt/infra/k8s/
|
||||
done
|
||||
cp infra/k8s/namespace.yaml /home/jgrusewski/Work/fxhnt/infra/k8s/
|
||||
mkdir -p /home/jgrusewski/Work/fxhnt/infra/scripts && cp infra/scripts/generate-minio-tls.sh /home/jgrusewski/Work/fxhnt/infra/scripts/
|
||||
echo "=== fxhnt infra/k8s now ==="; ls /home/jgrusewski/Work/fxhnt/infra/k8s/
|
||||
```
|
||||
Expected: fxhnt/infra/k8s/ now contains the platform dirs (merged with fxhnt's existing
|
||||
jobs/network-policies/services/storage/orchestration) + namespace.yaml. `gpu-taint.yaml` NOT copied.
|
||||
|
||||
- [ ] **Step 2: Sanity — no remaining hardcoded foxhunt-repo paths in moved manifests**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/fxhnt
|
||||
grep -rIn "Work/foxhunt\|foxhunt/infra\|root/foxhunt" infra/k8s/ infra/live/ infra/modules/ 2>/dev/null | grep -v "foxhunt.svc.cluster.local\|namespace: foxhunt\|name: foxhunt\|cluster_name\|foxhunt-tfstate\|foxhunt-gitlab\|foxhunt-training\|foxhunt-backups\|foxhunt-sccache\|foxhunt-binaries\|foxhunt-agent" | head
|
||||
echo "(above should be empty — 'foxhunt' as the k8s namespace / SCW resource names is expected + correct)"
|
||||
```
|
||||
Expected: empty (the namespace `foxhunt` + Scaleway resource names legitimately stay).
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Verify cluster unaffected + cockpit still deploys from fxhnt
|
||||
|
||||
- [ ] **Step 1: terragrunt still clean from fxhnt + platform health**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/fxhnt
|
||||
curl -sk -m 10 -o /dev/null -w "dashboard.fxhnt.ai: %{http_code}\n" https://dashboard.fxhnt.ai
|
||||
curl -sk -m 10 https://git.fxhnt.ai/api/healthz 2>/dev/null | grep -o '"status": "pass"'
|
||||
```
|
||||
Expected: dashboard 200; gitea pass. (No cluster change was made — this confirms nothing regressed.)
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Commit fxhnt (additions) + push
|
||||
|
||||
- [ ] **Step 1**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/fxhnt
|
||||
git add infra/ .gitignore
|
||||
git commit -m "feat(infra): adopt platform IaC from foxhunt — terraform live+modules + k8s manifests (Phase 2D)
|
||||
|
||||
fxhnt is now the single home for platform IaC. Terraform state keys unchanged
|
||||
(live/production structure preserved) — terragrunt plan = No changes, no migration.
|
||||
Includes k8s platform manifests (cert-manager, databases, dns, gitea, minio, monitoring,
|
||||
network-policies, tailscale, stalwart, mattermost, services, storage, jobs, training,
|
||||
secrets, argo, gitlab proxy) + generate-minio-tls.sh.
|
||||
|
||||
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>"
|
||||
GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin chore/relocate-iac 2>&1 | tail -2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Remove the moved paths from foxhunt + commit + push
|
||||
|
||||
- [ ] **Step 1: git rm the relocated IaC from foxhunt (keep gpu-taint.yaml, docker, Rust scripts)**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
git rm -r -q infra/live infra/modules
|
||||
for d in argo cert-manager databases dns gitea gitlab jobs mattermost minio monitoring network-policies secrets services stalwart tailscale training storage; do git rm -r -q "infra/k8s/$d"; done
|
||||
git rm -q infra/k8s/namespace.yaml infra/scripts/generate-minio-tls.sh
|
||||
echo "=== foxhunt infra/ remnant (Rust-only) ==="; find infra -maxdepth 2 | sort | head -30
|
||||
```
|
||||
Expected: `infra/k8s/gpu-taint.yaml` + `infra/docker/*` + `infra/scripts/{build-and-push,harden,smoke-test,train,...}` remain; live/modules + platform k8s gone.
|
||||
|
||||
- [ ] **Step 2: Commit + push foxhunt**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
git commit -q -m "chore(infra): relocate platform IaC to fxhnt (Phase 2D)
|
||||
|
||||
Platform terraform (live+modules) + k8s manifests moved to the fxhnt repo (now the IaC
|
||||
home). Rust-only remnants kept here (docker build images, gpu-taint, Rust scripts).
|
||||
foxhunt is being archived; history retained. See fxhnt for live platform IaC."
|
||||
GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin chore/relocate-iac-remove 2>&1 | tail -2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Merge both branches, then archive foxhunt
|
||||
|
||||
- [ ] **Step 1: Merge fxhnt branch → master + push**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/fxhnt
|
||||
git checkout master && git merge --no-ff chore/relocate-iac -m "Merge: adopt platform IaC from foxhunt (Phase 2D)" 2>&1 | tail -2
|
||||
git branch -d chore/relocate-iac
|
||||
GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin master 2>&1 | tail -1
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Merge foxhunt branch → main + push**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/foxhunt
|
||||
git checkout main && git merge --no-ff chore/relocate-iac-remove -m "Merge: relocate platform IaC to fxhnt (Phase 2D)" 2>&1 | tail -2
|
||||
git branch -d chore/relocate-iac-remove
|
||||
GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=accept-new" git push origin main 2>&1 | tail -1
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Final re-verify terragrunt from fxhnt (post-merge) + archive foxhunt in Gitea**
|
||||
```bash
|
||||
cd /home/jgrusewski/Work/fxhnt
|
||||
( cd infra/live/production/kapsule && terragrunt plan -input=false -no-color 2>&1 | sed 's/.*tofu: //' | grep -iE "No changes|^Plan:" | head -1 )
|
||||
GA=gitadmin; GP=$(kubectl get secret gitea-admin -n foxhunt -o jsonpath='{.data.password}'|base64 -d)
|
||||
curl -sk -u "$GA:$GP" -X PATCH "https://git.fxhnt.ai/api/v1/repos/$GA/foxhunt" -H 'Content-Type: application/json' -d '{"archived":true}' -o /dev/null -w "archive foxhunt: %{http_code}\n"
|
||||
```
|
||||
Expected: kapsule `No changes`; `archive foxhunt: 200`.
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
- Before Task 6 (foxhunt rm): just delete `fxhnt/infra/{live,modules}` + the copied k8s dirs; nothing lost
|
||||
(foxhunt still authoritative).
|
||||
- After: both repos retain history; un-archive foxhunt + `git revert` if needed. TF state never moved.
|
||||
|
||||
## Acceptance criteria (from spec)
|
||||
- `fxhnt/infra/{live,modules}` present; `terragrunt plan = No changes` all 4 modules from fxhnt. ✅ T2, T7
|
||||
- Platform k8s manifests + generate-minio-tls.sh in fxhnt; no collisions lost. ✅ T1, T3
|
||||
- Moved paths `git rm`'d from foxhunt; both repos committed + pushed. ✅ T5, T6, T7
|
||||
- dashboard 200 + cockpit deploy unaffected. ✅ T4
|
||||
- foxhunt archived in Gitea; Rust code + history intact. ✅ T7
|
||||
@@ -0,0 +1,69 @@
|
||||
# Decommission GitLab (compute) — Design (Phase 2C)
|
||||
|
||||
**Date:** 2026-06-21
|
||||
**Status:** Design (approved)
|
||||
**Repo:** foxhunt
|
||||
**Part of:** Phase 2 ([[project_phase2_gitlab_to_gitea]]). 2A (TF state→Scaleway) + 2B (Gitea + Scaleway
|
||||
registry + Argo webhook) DONE. This is **2C**. 2D (IaC relocation foxhunt→fxhnt) follows.
|
||||
|
||||
## Motivation
|
||||
|
||||
GitLab is fully replaced: git→Gitea (`git.fxhnt.ai`, SHA-verified), Terraform state→Scaleway Object
|
||||
Storage, container images→Scaleway Container Registry, CI/CD→Argo (Gitea webhook). It has **zero live
|
||||
consumers** but still runs 12 pods / ~4.6 GiB RAM. 2C removes GitLab's compute and retains its data as a
|
||||
cold backup (per the data-safety preference — migrated copies exist and are verified, but keep the
|
||||
originals through a grace period).
|
||||
|
||||
## Scope — remove (compute only)
|
||||
|
||||
1. **Helm uninstall** (foxhunt ns): `gitlab` (chart gitlab-9.9.3, rev 25) + `foxhunt-agent`
|
||||
(gitlab-agent). Frees ~4.6 GiB RAM / 12 pods.
|
||||
2. **Dead Argo Events**: `kubectl delete eventsource gitlab-push` + `sensor ci-pipeline-trigger`
|
||||
(Rust-era GitLab CI triggers; the gitea-push/gitea-deploy pair replaced them in 2B).
|
||||
3. **nginx proxy `:5050` registry block** + `containerPort: 5050` in `infra/k8s/gitlab/tailscale-proxy.yaml`
|
||||
(the GitLab container registry face; images are on Scaleway now).
|
||||
4. **Repo manifest** `infra/k8s/gitlab/values.yaml` (GitLab Helm values — chart no longer deployed).
|
||||
|
||||
## 🛑 PRESERVE (cold backup — delete in a later grace-period pass)
|
||||
- **gitaly PVC** `repo-data-gitlab-gitaly-0` (50 GiB, old git repos). StatefulSet PVCs survive
|
||||
`helm uninstall` by default — no action needed to keep it.
|
||||
- **MinIO buckets**: `foxhunt-gitlab-artifacts` (holds the superseded TF-state fallback),
|
||||
`foxhunt-gitlab-packages`, `foxhunt-gitlab-registry`.
|
||||
- **Scaleway buckets**: `foxhunt-gitlab-artifacts`, `foxhunt-gitlab-registry`.
|
||||
- **All `gitlab-*` secrets** (e.g. `gitlab-pat`, `gitlab-s3-credentials`, `gitlab-tls-cert`).
|
||||
- **`infra/k8s/gitlab/tailscale-proxy.yaml`** — KEEP: it is now the **Gitea** proxy (serves
|
||||
`git.fxhnt.ai` + all `*.fxhnt.ai`). Only its `:5050` block is removed.
|
||||
|
||||
## Execution order (safety-critical)
|
||||
|
||||
1. **Proxy first:** remove the `:5050` server block + `containerPort: 5050` from `tailscale-proxy.yaml`;
|
||||
`kubectl apply` + rollout restart; **verify `git.fxhnt.ai` + `dashboard.fxhnt.ai` still 200**.
|
||||
*Rationale:* nginx resolves `proxy_pass` upstream hostnames at startup. If GitLab is uninstalled while
|
||||
the `:5050` block still references `gitlab-registry.foxhunt.svc`, nginx fails to start on its next
|
||||
restart → **all `*.fxhnt.ai` go down**. Removing the dead upstream first defuses this.
|
||||
2. **helm uninstall** `gitlab` then `foxhunt-agent`.
|
||||
3. **Delete dead Argo Events** (`gitlab-push`, `ci-pipeline-trigger`).
|
||||
4. **Remove `values.yaml`** from the repo; commit.
|
||||
5. **Verify** (see acceptance).
|
||||
|
||||
## Risks / safety
|
||||
|
||||
- **Proxy outage** if `:5050` not removed before uninstall → mitigated by step 1 + the 200-check gate.
|
||||
- **Irreversible**: helm uninstall. Mitigated: all data retained (gitaly PVC + buckets + secrets), so a
|
||||
reinstall could re-attach. Migrated copies (Gitea/Scaleway) are the primary going forward.
|
||||
- **Shared resource**: confirm the `gitlab` release owns only GitLab objects (it does — namespaced release).
|
||||
The `postgres`/`minio`/`tailscale-proxy` are separate and stay.
|
||||
- **No `.dbn`/market data** in any GitLab bucket (git artifacts/registry/packages only) — unaffected.
|
||||
|
||||
## Out of scope
|
||||
- Deleting the retained GitLab data (gitaly PVC + buckets + secrets) — a later grace-period pass.
|
||||
- 2D IaC relocation foxhunt→fxhnt (separate spec); re-archiving the foxhunt repo happens there.
|
||||
|
||||
## Acceptance criteria
|
||||
- `helm list -n foxhunt` shows no `gitlab` / `foxhunt-agent`; GitLab pods gone (~4.6 GiB RAM reclaimed).
|
||||
- `git.fxhnt.ai` + `dashboard.fxhnt.ai` return 200; a push to fxhnt master still auto-deploys via
|
||||
Gitea→Argo.
|
||||
- `gitlab-push` eventsource + `ci-pipeline-trigger` sensor deleted; `gitea-push`/`gitea-deploy` still present.
|
||||
- `:5050` block gone from the proxy; proxy healthy.
|
||||
- **Cold backup intact:** gitaly PVC + all GitLab MinIO/Scaleway buckets + `gitlab-*` secrets still present.
|
||||
- `infra/k8s/gitlab/values.yaml` removed; `tailscale-proxy.yaml` kept; committed.
|
||||
@@ -0,0 +1,85 @@
|
||||
# Relocate Platform IaC foxhunt → fxhnt + Archive foxhunt — Design (Phase 2D)
|
||||
|
||||
**Date:** 2026-06-21
|
||||
**Status:** Design (approved)
|
||||
**Repos:** foxhunt (source), fxhnt (destination)
|
||||
**Part of:** Phase 2 ([[project_phase2_gitlab_to_gitea]]). 2A/2B/2C DONE. This is the final sub-project, 2D.
|
||||
|
||||
## Motivation
|
||||
|
||||
The active work is the Python `fxhnt` fund; `foxhunt` is the dormant Rust system that still happens to
|
||||
hold the platform IaC. Consolidate the active platform IaC into `fxhnt` so it is the single home, then
|
||||
archive `foxhunt` (keeping its Rust code + history as a read-only snapshot).
|
||||
|
||||
## Decision (keystone) — no Terraform state migration
|
||||
|
||||
State keys are `${path_relative_to_include()}/terraform.tfstate` (relative to `root.hcl`). Moving
|
||||
`infra/live/` + `infra/modules/` into `fxhnt/infra/` **with byte-identical internal structure**
|
||||
(`fxhnt/infra/live/production/root.hcl`, `.../<module>/terragrunt.hcl`, module `source = "../../../modules/<m>"`)
|
||||
keeps every state key unchanged (`kapsule/`, `public-gateway/`, `dns/`, `block-storage/`). **No state
|
||||
migration** — proven by `terragrunt plan = No changes` from the new location. Move method: **plain copy**
|
||||
(git history stays in the archived foxhunt).
|
||||
|
||||
## Scope — MOVE to `fxhnt/infra/` (active platform)
|
||||
|
||||
- **Terraform:** `infra/live/` (terragrunt) + `infra/modules/` (kapsule, public-gateway, dns,
|
||||
block-storage) — structure preserved exactly.
|
||||
- **k8s platform manifests** (`infra/k8s/`): `cert-manager`, `databases`, `dns`, `gitea`, `mattermost`,
|
||||
`minio`, `monitoring`, `network-policies`, `secrets`, `services`, `stalwart`, `tailscale`, `storage`,
|
||||
`argo` (surviving gitea events + rbac + netpol + values + databento-download-template + feature-cache-pvc
|
||||
+ kustomization), `gitlab` (only `tailscale-proxy.yaml` + `grafana-values.yaml`), `jobs`, `training`
|
||||
(the kept `.dbn` data-acquisition tooling), `namespace.yaml`.
|
||||
- **Platform scripts:** the deploy/ops scripts that `kubectl apply`/terragrunt the above (enumerated in the
|
||||
plan after a content scan). `fxhnt/scripts/argo-deploy-cockpit.sh` already lives in fxhnt.
|
||||
|
||||
## Scope — STAYS in `foxhunt` (archived, Rust-dead)
|
||||
|
||||
Rust source (`crates/`, `services/`, `bin/`, `proto/`, `migrations/`, etc.), Rust scripts (`argo-alpha-*`,
|
||||
`argo-compile-deploy`, `argo-sanitizer`, `argo-nsys`, `argo-precompute`, `argo-lob-sweep`, `argo-smoke`,
|
||||
`argo-download`), Rust docker build images (ci-builder/training-runtime), `infra/k8s/gpu-taint.yaml`, and
|
||||
all foxhunt git history.
|
||||
|
||||
## Merge strategy (fxhnt already has `infra/{argo,docker,k8s/{jobs,network-policies,services,storage,orchestration}}`)
|
||||
|
||||
Same-named dirs merge **file-by-file**. Pre-flight: scan for filename collisions between foxhunt's and
|
||||
fxhnt's `infra/k8s/{network-policies,services,jobs,storage}` (+ `docker`); resolve any collision explicitly
|
||||
before copying (expected: none — different resources). Foxhunt's `infra/k8s/argo` lands at
|
||||
`fxhnt/infra/k8s/argo` (fxhnt's existing argo is `fxhnt/infra/argo` — different path, no collision).
|
||||
|
||||
## Execution order (verify-gated)
|
||||
|
||||
1. **Collision pre-flight** — list overlaps; resolve/confirm none.
|
||||
2. **Copy Terraform** (`live/` + `modules/`) into `fxhnt/infra/` preserving structure; **`terragrunt
|
||||
init -reconfigure && plan` from `fxhnt/infra/live/production/<each module>` → must be `No changes`**
|
||||
(the no-migration proof). STOP on any diff.
|
||||
3. **Copy platform k8s manifests + scripts** into `fxhnt/infra/`; update any in-script `infra/...` paths.
|
||||
4. **Verify** manifests still apply + a cockpit deploy still works from fxhnt (it already does).
|
||||
5. **`git rm` the moved paths from foxhunt**; commit both repos; push.
|
||||
6. **Archive `foxhunt`** in Gitea (read-only) once 2-5 verified.
|
||||
|
||||
## Safety / reversibility
|
||||
|
||||
- Plain copy first; foxhunt files removed only after fxhnt copy + terragrunt-No-changes verified.
|
||||
- TF state untouched (same keys); the Scaleway backend doesn't care which repo runs terragrunt.
|
||||
- foxhunt archive (not delete) — full history + Rust code remain restorable.
|
||||
- No live workload changes — this is a source-of-truth move, not a cluster change. The running cluster is
|
||||
unaffected (manifests are already applied; terraform state already external).
|
||||
|
||||
## Risks
|
||||
- **Path drift** breaking TF state → mitigated by the structure-preserve rule + the `No changes` gate.
|
||||
- **Merge collision** silently overwriting a fxhnt file → mitigated by the collision pre-flight.
|
||||
- **A moved script with a stale `infra/` path** → grep + fix in step 3; verify in step 4.
|
||||
- **Premature archive** (foxhunt still referenced) → archive last, after all verification.
|
||||
|
||||
## Out of scope
|
||||
- Deleting GitLab cold-backup data (separate grace-period pass).
|
||||
- Pruning the Rust codebase itself (foxhunt is archived as-is).
|
||||
- Phase 2E observability (Grafana/Prometheus).
|
||||
|
||||
## Acceptance criteria
|
||||
- `fxhnt/infra/{live,modules}` present; `terragrunt plan` = `No changes` for all 4 modules from fxhnt
|
||||
(state keys unchanged — no migration).
|
||||
- Platform k8s manifests + scripts present in `fxhnt/infra/`; no filename collisions lost.
|
||||
- Moved paths `git rm`'d from foxhunt; both repos committed + pushed to Gitea.
|
||||
- Cockpit deploy + `dashboard.fxhnt.ai` 200 still work (cluster unaffected).
|
||||
- `foxhunt` repo archived in Gitea; Rust code + history intact.
|
||||
@@ -1,34 +0,0 @@
|
||||
# RBAC for Argo workflow archive — grants controller and server access
|
||||
# to the PostgreSQL archive secret and MinIO credentials.
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: argo-archive-secrets
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: argo-workflows
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
resourceNames: ["argo-postgres-config", "minio-credentials"]
|
||||
verbs: ["get"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: argo-archive-secrets
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: argo-workflows
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: argo-archive-secrets
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: argo-workflows-workflow-controller
|
||||
namespace: foxhunt
|
||||
- kind: ServiceAccount
|
||||
name: argo-workflows-server
|
||||
namespace: foxhunt
|
||||
@@ -1,518 +0,0 @@
|
||||
# Base egress for ALL Argo workflow pods: DNS, K8s API, MinIO (logging), Mattermost (notifications).
|
||||
# Applies to every pod with app.kubernetes.io/part-of: foxhunt.
|
||||
# Component-specific policies below add extra rules (GitLab SSH, external HTTPS, etc.).
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: argo-base-egress
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# DNS
|
||||
- ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
# Kubernetes API — Argo executor reports task results
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
- port: 6443
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 10.32.0.0/16
|
||||
- ipBlock:
|
||||
cidr: 172.16.0.4/32
|
||||
# MinIO — log archival (Argo wait sidecar)
|
||||
- ports:
|
||||
- port: 9000
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
# Mattermost — exit handler webhook notifications
|
||||
- ports:
|
||||
- port: 8065
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
---
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: argo-training-workflow
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/component: training-workflow
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# DNS — required for hostname resolution (minio, gitlab, etc.)
|
||||
- ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
# Kubernetes API — Argo executor reports task results
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
- port: 6443
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 10.32.0.0/16
|
||||
- ipBlock:
|
||||
cidr: 172.16.0.4/32
|
||||
# HTTPS (external registries, etc.)
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
# GitLab SSH — git clone
|
||||
- ports:
|
||||
- port: 2222
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 100.90.76.85/32
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: gitlab-shell
|
||||
# GitLab API — binary fetch from packages
|
||||
- ports:
|
||||
- port: 8181
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: webservice
|
||||
# MinIO — log archival + result upload
|
||||
- ports:
|
||||
- port: 9000
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
# Pushgateway — Prometheus metrics
|
||||
- ports:
|
||||
- port: 9091
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: pushgateway
|
||||
# Tempo — OTLP traces
|
||||
- ports:
|
||||
- port: 4317
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tempo
|
||||
# Mattermost — exit handler webhook notifications
|
||||
- ports:
|
||||
- port: 8065
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
# Container registry — image pulls
|
||||
- ports:
|
||||
- port: 5000
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: registry
|
||||
---
|
||||
# Compile workflow pods: need GitLab SSH, MinIO, k8s API, external HTTPS (cargo deps)
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: argo-compile-workflow
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/component: compile
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# DNS — resolve external hostnames (crates.io, github.com)
|
||||
- ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
# Kubernetes API — Argo executor
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
- port: 6443
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 10.32.0.0/16
|
||||
- ipBlock:
|
||||
cidr: 172.16.0.4/32
|
||||
# External HTTPS — cargo fetches crates.io, github.com (candle git dep)
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
# GitLab SSH — git clone (Tailscale IP + cluster service)
|
||||
- ports:
|
||||
- port: 2222
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 100.90.76.85/32
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: gitlab-shell
|
||||
# MinIO — binary upload
|
||||
- ports:
|
||||
- port: 9000
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
# Mattermost — exit handler webhook notifications
|
||||
- ports:
|
||||
- port: 8065
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
---
|
||||
# Compile-and-deploy workflow pods: need GitLab HTTP (tagging/releases), MinIO,
|
||||
# k8s API (deploy rollouts + Argo executor), DNS, GitLab SSH, external HTTPS.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: argo-compile-and-deploy-workflow
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/component: compile-and-deploy
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# DNS
|
||||
- ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
# Kubernetes API — Argo executor + kubectl rollout
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
- port: 6443
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 10.32.0.0/16
|
||||
- ipBlock:
|
||||
cidr: 172.16.0.4/32
|
||||
# External HTTPS — cargo fetches, kubectl download
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
# GitLab SSH — git clone
|
||||
- ports:
|
||||
- port: 2222
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 100.90.76.85/32
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: gitlab-shell
|
||||
# GitLab webservice — CalVer tagging, releases, package uploads
|
||||
- ports:
|
||||
- port: 8181
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: webservice
|
||||
# MinIO — binary upload + log archival
|
||||
- ports:
|
||||
- port: 9000
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
# Mattermost — exit handler webhook notifications
|
||||
- ports:
|
||||
- port: 8065
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
---
|
||||
# Compile-and-train workflow pods: compile step needs GitLab SSH + external HTTPS + DNS,
|
||||
# training steps need GitLab HTTP (package fetch/upload), MinIO, Tempo, k8s API.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: argo-train-workflow
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/component: train
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# DNS
|
||||
- ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
# Kubernetes API — Argo executor
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
- port: 6443
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 10.32.0.0/16
|
||||
- ipBlock:
|
||||
cidr: 172.16.0.4/32
|
||||
# External HTTPS — cargo fetches crates.io, github.com (candle git dep)
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
# GitLab SSH — git clone
|
||||
- ports:
|
||||
- port: 2222
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 100.90.76.85/32
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: gitlab-shell
|
||||
# GitLab webservice — package upload/download
|
||||
- ports:
|
||||
- port: 8181
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: webservice
|
||||
# MinIO — binary upload
|
||||
- ports:
|
||||
- port: 9000
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
# Tempo — OTLP traces
|
||||
- ports:
|
||||
- port: 4317
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tempo
|
||||
# Mattermost — exit handler webhook notifications
|
||||
- ports:
|
||||
- port: 8065
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
---
|
||||
# CI pipeline pods (detect-changes + compile-service inline templates)
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: argo-ci-pipeline
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/component: ci-pipeline
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# DNS — resolve external hostnames
|
||||
- ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
# Kubernetes API — Argo executor
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
- port: 6443
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 10.32.0.0/16
|
||||
- ipBlock:
|
||||
cidr: 172.16.0.4/32
|
||||
# External HTTP/HTTPS — Kaniko base image pulls, apt-get, cargo fetches
|
||||
- ports:
|
||||
- port: 80
|
||||
protocol: TCP
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
# GitLab SSH — git clone (compile-service uses ssh://git@git.fxhnt.ai)
|
||||
- ports:
|
||||
- port: 2222
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 100.90.76.85/32
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: gitlab-shell
|
||||
# MinIO — binary upload from compile-service
|
||||
- ports:
|
||||
- port: 9000
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
# GitLab registry — Kaniko image push
|
||||
- ports:
|
||||
- port: 5000
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: registry
|
||||
# GitLab webservice — JWT auth for registry tokens
|
||||
- ports:
|
||||
- port: 8181
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: webservice
|
||||
# Mattermost — exit handler webhook notifications
|
||||
- ports:
|
||||
- port: 8065
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
---
|
||||
# ── sanitizer-test + nsys-test + smoke-test workflows: same egress as ci-pipeline ──
|
||||
# (git fetch over SSH, MinIO log archive + nsys-rep upload, registry image pull,
|
||||
# external HTTPS for mc download)
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: argo-sanitizer-nsys-workflow
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/component
|
||||
operator: In
|
||||
values:
|
||||
- sanitizer-test
|
||||
- nsys-test
|
||||
- smoke-test
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# DNS
|
||||
- ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
# Kubernetes API
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
- port: 6443
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 10.32.0.0/16
|
||||
- ipBlock:
|
||||
cidr: 172.16.0.4/32
|
||||
# External HTTPS — mc binary download from dl.min.io
|
||||
- ports:
|
||||
- port: 80
|
||||
protocol: TCP
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
# GitLab SSH — git fetch
|
||||
- ports:
|
||||
- port: 2222
|
||||
protocol: TCP
|
||||
to:
|
||||
- ipBlock:
|
||||
cidr: 100.90.76.85/32
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: gitlab-shell
|
||||
# MinIO — log archive + nsys profile upload
|
||||
- ports:
|
||||
- port: 9000
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
# GitLab registry — ci-builder image pull
|
||||
- ports:
|
||||
- port: 5000
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: registry
|
||||
# GitLab webservice — JWT auth for registry tokens
|
||||
- ports:
|
||||
- port: 8181
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: webservice
|
||||
@@ -1,47 +0,0 @@
|
||||
# RBAC for CI pipeline — deploy step + self-apply Argo templates
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: ci-deploy
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
rules:
|
||||
# apps: roll + apply Deployments. watch is required by `kubectl rollout status`.
|
||||
- apiGroups: ["apps"]
|
||||
resources: [deployments]
|
||||
verbs: [get, list, watch, patch]
|
||||
- apiGroups: ["argoproj.io"]
|
||||
resources: [workflowtemplates, eventsources, sensors, eventbus]
|
||||
verbs: [get, list, create, update, patch]
|
||||
# core: services/configmaps, plus serviceaccounts (tailscale-dashboard, dagster) and the forward-track PVC
|
||||
# — added so the fxhnt-cockpit deploy can `kubectl apply` its full manifest set without a partial-apply failure.
|
||||
- apiGroups: [""]
|
||||
resources: [services, configmaps, serviceaccounts, persistentvolumeclaims]
|
||||
verbs: [get, list, create, update, patch]
|
||||
- apiGroups: ["networking.k8s.io"]
|
||||
resources: [networkpolicies]
|
||||
verbs: [get, list, create, update, patch]
|
||||
# batch: the fxhnt-forward CronJob
|
||||
- apiGroups: ["batch"]
|
||||
resources: [cronjobs]
|
||||
verbs: [get, list, create, update, patch]
|
||||
- apiGroups: ["rbac.authorization.k8s.io"]
|
||||
resources: [roles, rolebindings]
|
||||
verbs: [get, list, create, update, patch, bind, escalate]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: ci-deploy
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: Role
|
||||
name: ci-deploy
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: argo-workflow
|
||||
namespace: foxhunt
|
||||
@@ -1,135 +0,0 @@
|
||||
# Databento data download via streaming API (get_range).
|
||||
#
|
||||
# Parameterized Argo WorkflowTemplate — replaces the hardcoded K8s Job.
|
||||
# Binary pre-uploaded to /data/bin/download_baseline on training-data-pvc.
|
||||
# Universe configs loaded from configmap databento-universe-all.
|
||||
#
|
||||
# Usage:
|
||||
# argo submit -n foxhunt --from=wftmpl/databento-download
|
||||
# argo submit -n foxhunt --from=wftmpl/databento-download -p schema=trades -p output-dir=/data/futures-baseline-trades
|
||||
# ./scripts/argo-download.sh mbp-10 --parallel 9 --pool ci-compile-cpu
|
||||
#
|
||||
# Databento rate limits: 100 concurrent connections per IP, 100 req/s.
|
||||
---
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: WorkflowTemplate
|
||||
metadata:
|
||||
name: databento-download
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: databento-download
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
entrypoint: download
|
||||
serviceAccountName: argo-workflow
|
||||
podMetadata:
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
app.kubernetes.io/component: databento-download
|
||||
securityContext:
|
||||
fsGroup: 1000
|
||||
ttlStrategy:
|
||||
secondsAfterCompletion: 3600
|
||||
activeDeadlineSeconds: 86400 # 24 hours
|
||||
|
||||
arguments:
|
||||
parameters:
|
||||
- name: schema
|
||||
value: mbp-10
|
||||
description: "Databento schema: ohlcv-1m, ohlcv-1s, trades, mbp-10"
|
||||
- name: output-dir
|
||||
value: /data/futures-baseline-mbp10
|
||||
description: "Output directory on training-data PVC"
|
||||
- name: parallel
|
||||
value: "9"
|
||||
description: "Number of parallel quarterly downloads (max ~100 per Databento rate limit)"
|
||||
- name: node-pool
|
||||
value: ci-compile-cpu
|
||||
description: "K8s node pool: ci-compile-cpu (32c/64G) or platform (4c/8G)"
|
||||
|
||||
volumes:
|
||||
- name: training-data
|
||||
persistentVolumeClaim:
|
||||
claimName: training-data-pvc
|
||||
- name: universe-configs
|
||||
configMap:
|
||||
name: databento-universe-all
|
||||
|
||||
templates:
|
||||
- name: download
|
||||
inputs:
|
||||
parameters:
|
||||
- name: schema
|
||||
- name: output-dir
|
||||
- name: parallel
|
||||
- name: node-pool
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: "{{inputs.parameters.node-pool}}"
|
||||
topology.kubernetes.io/zone: fr-par-2
|
||||
container:
|
||||
image: ubuntu:24.04
|
||||
command: ["/bin/bash", "-c"]
|
||||
env:
|
||||
- name: RUST_LOG
|
||||
value: info
|
||||
- name: DATABENTO_API_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: databento-credentials
|
||||
key: api-key
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: "2"
|
||||
memory: 4Gi
|
||||
volumeMounts:
|
||||
- name: training-data
|
||||
mountPath: /data
|
||||
- name: universe-configs
|
||||
mountPath: /configs
|
||||
args:
|
||||
- |
|
||||
set -e
|
||||
apt-get update -qq > /dev/null 2>&1 && apt-get install -y -qq ca-certificates > /dev/null 2>&1
|
||||
|
||||
BINARY=/data/bin/download_baseline
|
||||
chmod +x "$BINARY"
|
||||
|
||||
SCHEMA="{{inputs.parameters.schema}}"
|
||||
OUTPUT_DIR="{{inputs.parameters.output-dir}}"
|
||||
PARALLEL="{{inputs.parameters.parallel}}"
|
||||
|
||||
# Map schema name to configmap key
|
||||
CONFIG="/configs/${SCHEMA}.toml"
|
||||
if [ ! -f "$CONFIG" ]; then
|
||||
echo "ERROR: Config not found: $CONFIG"
|
||||
echo "Available configs:"
|
||||
ls /configs/
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
|
||||
echo "=== Databento Download ==="
|
||||
echo "Schema: $SCHEMA"
|
||||
echo "Config: $CONFIG"
|
||||
echo "Output: $OUTPUT_DIR"
|
||||
echo "Parallel: $PARALLEL"
|
||||
echo "Node: $(hostname), CPUs: $(nproc), RAM: $(free -g | awk '/Mem:/{print $2}')G"
|
||||
echo ""
|
||||
|
||||
$BINARY \
|
||||
--universe-config "$CONFIG" \
|
||||
--output-dir "$OUTPUT_DIR" \
|
||||
--parallel "$PARALLEL" \
|
||||
--yes
|
||||
|
||||
echo ""
|
||||
echo "=== Download complete ==="
|
||||
echo "Files:"
|
||||
find "$OUTPUT_DIR" -name '*.dbn.zst' -exec ls -lh {} \;
|
||||
echo ""
|
||||
du -sh "$OUTPUT_DIR"
|
||||
df -h /data
|
||||
@@ -1,25 +0,0 @@
|
||||
# DNS Bootstrap Prevention — Prevent CoreDNS chicken-and-egg deadlock
|
||||
#
|
||||
# ROOT CAUSE (2026-03-18 incident):
|
||||
# 1. Rolling replacement of ALL platform nodes simultaneously
|
||||
# 2. New nodes boot with /etc/resolv.conf → 10.32.0.10 (kube-dns only)
|
||||
# 3. CoreDNS pods need to pull images → need DNS → need CoreDNS
|
||||
# 4. Cilium BPF socket LB returns EPERM when kube-dns has no endpoints
|
||||
# 5. Complete cluster DNS deadlock — nothing can pull images
|
||||
#
|
||||
# PREVENTION:
|
||||
# 1. VPC DefaultRoutePropagation MUST be enabled (API, not Terraform yet)
|
||||
# 2. NEVER replace ALL nodes at once — one at a time, wait for CoreDNS
|
||||
# 3. Do NOT create CiliumClusterwideNetworkPolicy with egress rules
|
||||
# (enables default deny, makes the problem worse)
|
||||
# 4. bpf-lb-sock MUST remain "true" — disabling breaks ALL service routing
|
||||
# 5. Keep bastion enabled on VPC gateway for emergency SSH access
|
||||
#
|
||||
# RECOVERY PROCEDURE:
|
||||
# 1. SSH via bastion: ssh -o ProxyCommand="ssh -W %h:%p -p 61000 bastion@<gw-public-ip>" root@<node-private-ip>
|
||||
# 2. On the node: /usr/local/bin/crictl pull <coredns-image>
|
||||
# 3. kubectl delete pods -l k8s-app=kube-dns --force
|
||||
# 4. CoreDNS starts from cached image → DNS recovers → cascade
|
||||
#
|
||||
# DATA PROTECTION:
|
||||
# All PVs set to Retain reclaim policy — data survives cluster deletion
|
||||
@@ -1,10 +0,0 @@
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: EventBus
|
||||
metadata:
|
||||
name: default
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
nats:
|
||||
native:
|
||||
replicas: 3
|
||||
auth: token
|
||||
@@ -1,16 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: gitlab-push-eventsource-svc
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: gitlab-push-eventsource
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
type: ClusterIP
|
||||
ports:
|
||||
- port: 12000
|
||||
targetPort: 12000
|
||||
protocol: TCP
|
||||
selector:
|
||||
eventsource-name: gitlab-push
|
||||
@@ -1,29 +0,0 @@
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: argo-sensor-submit
|
||||
namespace: foxhunt
|
||||
rules:
|
||||
- apiGroups: ["argoproj.io"]
|
||||
resources: ["workflows"]
|
||||
verbs: ["create", "get", "list", "watch"]
|
||||
- apiGroups: ["argoproj.io"]
|
||||
resources: ["workflowtemplates"]
|
||||
verbs: ["get", "list"]
|
||||
- apiGroups: ["argoproj.io"]
|
||||
resources: ["workflowtaskresults"]
|
||||
verbs: ["create", "patch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: argo-sensor-submit
|
||||
namespace: foxhunt
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: argo-workflow
|
||||
namespace: foxhunt
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: argo-sensor-submit
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
@@ -1,51 +0,0 @@
|
||||
# Manual workflow trigger EventSource.
|
||||
#
|
||||
# Webhook endpoints for triggering compile/deploy, training, and image builds.
|
||||
# Each endpoint accepts a JSON POST body with workflow-specific parameters.
|
||||
#
|
||||
# Endpoints (all on port 12001):
|
||||
# POST /compile-deploy — {"commit_sha": "HEAD"}
|
||||
# POST /train/dqn — {"binary_tag": "latest"}
|
||||
# POST /train/ppo — {"binary_tag": "latest"}
|
||||
# POST /train/supervised — {"model": "tft", "binary_tag": "latest"}
|
||||
# POST /build-image — {"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}
|
||||
#
|
||||
# Example:
|
||||
# curl -X POST http://workflow-trigger-eventsource-svc.foxhunt:12001/build-image \
|
||||
# -H 'Content-Type: application/json' \
|
||||
# -d '{"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}'
|
||||
---
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: EventSource
|
||||
metadata:
|
||||
name: workflow-trigger
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: workflow-trigger
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
eventBusName: default
|
||||
template:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
webhook:
|
||||
compile-deploy:
|
||||
port: "12001"
|
||||
endpoint: /compile-deploy
|
||||
method: POST
|
||||
train-dqn:
|
||||
port: "12001"
|
||||
endpoint: /train/dqn
|
||||
method: POST
|
||||
train-ppo:
|
||||
port: "12001"
|
||||
endpoint: /train/ppo
|
||||
method: POST
|
||||
train-supervised:
|
||||
port: "12001"
|
||||
endpoint: /train/supervised
|
||||
method: POST
|
||||
build-image:
|
||||
port: "12001"
|
||||
endpoint: /build-image
|
||||
method: POST
|
||||
@@ -1,217 +0,0 @@
|
||||
# Sensor for manual workflow triggers.
|
||||
#
|
||||
# Listens on workflow-trigger EventSource and submits the appropriate workflow
|
||||
# when a webhook endpoint is hit. Each trigger maps JSON body fields to
|
||||
# workflow parameters.
|
||||
#
|
||||
# Trigger reference:
|
||||
# compile-deploy — POST /compile-deploy {"commit_sha": "..."}
|
||||
# train-dqn — POST /train/dqn {"binary_tag": "..."}
|
||||
# train-ppo — POST /train/ppo {"binary_tag": "..."}
|
||||
# train-supervised — POST /train/supervised {"model": "...", "binary_tag": "..."}
|
||||
# build-image — POST /build-image {"dockerfile": "...", "image_name": "...", "commit_sha": "..."}
|
||||
---
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Sensor
|
||||
metadata:
|
||||
name: workflow-trigger
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: workflow-trigger
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
template:
|
||||
serviceAccountName: argo-workflow
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
eventBusName: default
|
||||
|
||||
dependencies:
|
||||
- name: compile-deploy-dep
|
||||
eventSourceName: workflow-trigger
|
||||
eventName: compile-deploy
|
||||
- name: train-dqn-dep
|
||||
eventSourceName: workflow-trigger
|
||||
eventName: train-dqn
|
||||
- name: train-ppo-dep
|
||||
eventSourceName: workflow-trigger
|
||||
eventName: train-ppo
|
||||
- name: train-supervised-dep
|
||||
eventSourceName: workflow-trigger
|
||||
eventName: train-supervised
|
||||
- name: build-image-dep
|
||||
eventSourceName: workflow-trigger
|
||||
eventName: build-image
|
||||
|
||||
triggers:
|
||||
# ── Compile + Deploy ──
|
||||
# POST /compile-deploy {"commit_sha": "abc1234"}
|
||||
# Omit service_packages to build all 7 services (default).
|
||||
- template:
|
||||
name: compile-deploy
|
||||
conditions: compile-deploy-dep
|
||||
argoWorkflow:
|
||||
operation: submit
|
||||
source:
|
||||
resource:
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Workflow
|
||||
metadata:
|
||||
generateName: compile-deploy-
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
serviceAccountName: argo-workflow
|
||||
podGC:
|
||||
strategy: OnPodCompletion
|
||||
ttlStrategy:
|
||||
secondsAfterCompletion: 3600
|
||||
workflowTemplateRef:
|
||||
name: compile-and-deploy
|
||||
arguments:
|
||||
parameters:
|
||||
- name: commit-sha
|
||||
parameters:
|
||||
- src:
|
||||
dependencyName: compile-deploy-dep
|
||||
dataKey: body.commit_sha
|
||||
dest: spec.arguments.parameters.0.value
|
||||
|
||||
# ── Train DQN ──
|
||||
# POST /train/dqn {"binary_tag": "latest"}
|
||||
- template:
|
||||
name: train-dqn
|
||||
conditions: train-dqn-dep
|
||||
argoWorkflow:
|
||||
operation: submit
|
||||
source:
|
||||
resource:
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Workflow
|
||||
metadata:
|
||||
generateName: train-dqn-
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
serviceAccountName: argo-workflow
|
||||
podGC:
|
||||
strategy: OnPodCompletion
|
||||
ttlStrategy:
|
||||
secondsAfterCompletion: 3600
|
||||
workflowTemplateRef:
|
||||
name: train-dqn
|
||||
arguments:
|
||||
parameters:
|
||||
- name: binary-tag
|
||||
parameters:
|
||||
- src:
|
||||
dependencyName: train-dqn-dep
|
||||
dataKey: body.binary_tag
|
||||
dest: spec.arguments.parameters.0.value
|
||||
|
||||
# ── Train PPO ──
|
||||
# POST /train/ppo {"binary_tag": "latest"}
|
||||
- template:
|
||||
name: train-ppo
|
||||
conditions: train-ppo-dep
|
||||
argoWorkflow:
|
||||
operation: submit
|
||||
source:
|
||||
resource:
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Workflow
|
||||
metadata:
|
||||
generateName: train-ppo-
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
serviceAccountName: argo-workflow
|
||||
podGC:
|
||||
strategy: OnPodCompletion
|
||||
ttlStrategy:
|
||||
secondsAfterCompletion: 3600
|
||||
workflowTemplateRef:
|
||||
name: train-ppo
|
||||
arguments:
|
||||
parameters:
|
||||
- name: binary-tag
|
||||
parameters:
|
||||
- src:
|
||||
dependencyName: train-ppo-dep
|
||||
dataKey: body.binary_tag
|
||||
dest: spec.arguments.parameters.0.value
|
||||
|
||||
# ── Train Supervised ──
|
||||
# POST /train/supervised {"model": "tft", "binary_tag": "latest"}
|
||||
# model: one of tft, mamba2, tggn, tlob, liquid, kan, xlstm, diffusion
|
||||
- template:
|
||||
name: train-supervised
|
||||
conditions: train-supervised-dep
|
||||
argoWorkflow:
|
||||
operation: submit
|
||||
source:
|
||||
resource:
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Workflow
|
||||
metadata:
|
||||
generateName: train-supervised-
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
serviceAccountName: argo-workflow
|
||||
podGC:
|
||||
strategy: OnPodCompletion
|
||||
ttlStrategy:
|
||||
secondsAfterCompletion: 3600
|
||||
workflowTemplateRef:
|
||||
name: train-supervised
|
||||
arguments:
|
||||
parameters:
|
||||
- name: model
|
||||
- name: binary-tag
|
||||
parameters:
|
||||
- src:
|
||||
dependencyName: train-supervised-dep
|
||||
dataKey: body.model
|
||||
dest: spec.arguments.parameters.0.value
|
||||
- src:
|
||||
dependencyName: train-supervised-dep
|
||||
dataKey: body.binary_tag
|
||||
dest: spec.arguments.parameters.1.value
|
||||
|
||||
# ── Build Image ──
|
||||
# POST /build-image {"dockerfile": "Dockerfile.ci-builder", "image_name": "ci-builder", "commit_sha": "HEAD"}
|
||||
- template:
|
||||
name: build-image
|
||||
conditions: build-image-dep
|
||||
argoWorkflow:
|
||||
operation: submit
|
||||
source:
|
||||
resource:
|
||||
apiVersion: argoproj.io/v1alpha1
|
||||
kind: Workflow
|
||||
metadata:
|
||||
generateName: build-image-
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
serviceAccountName: argo-workflow
|
||||
podGC:
|
||||
strategy: OnPodCompletion
|
||||
ttlStrategy:
|
||||
secondsAfterCompletion: 3600
|
||||
workflowTemplateRef:
|
||||
name: build-ci-image
|
||||
arguments:
|
||||
parameters:
|
||||
- name: commit-sha
|
||||
- name: dockerfile
|
||||
- name: image-name
|
||||
parameters:
|
||||
- src:
|
||||
dependencyName: build-image-dep
|
||||
dataKey: body.commit_sha
|
||||
dest: spec.arguments.parameters.0.value
|
||||
- src:
|
||||
dependencyName: build-image-dep
|
||||
dataKey: body.dockerfile
|
||||
dest: spec.arguments.parameters.1.value
|
||||
- src:
|
||||
dependencyName: build-image-dep
|
||||
dataKey: body.image_name
|
||||
dest: spec.arguments.parameters.2.value
|
||||
@@ -1,16 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: workflow-trigger-eventsource-svc
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: workflow-trigger
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
type: ClusterIP
|
||||
ports:
|
||||
- port: 12001
|
||||
targetPort: 12001
|
||||
protocol: TCP
|
||||
selector:
|
||||
eventsource-name: workflow-trigger
|
||||
@@ -1,18 +0,0 @@
|
||||
# Dedicated PVC for precomputed .fxcache files.
|
||||
# Survives workflow completions (unlike workspace volumeClaimTemplate).
|
||||
# Precompute writes here, all training steps read from here.
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: feature-cache-pvc
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: feature-cache
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: scw-bssd-retain
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
@@ -1,8 +0,0 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
# Rust train/CI/cache templates removed 2026-06-21 (decommission-rust-infra).
|
||||
# Remaining: platform RBAC + network policy used by the active fxhnt-cockpit deploys.
|
||||
resources:
|
||||
- argo-workflow-netpol.yaml
|
||||
- ci-deploy-rbac.yaml
|
||||
- archive-rbac.yaml
|
||||
@@ -1,11 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: notification-webhook
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
type: Opaque
|
||||
stringData:
|
||||
# Mattermost incoming webhook URL — update after Mattermost is deployed
|
||||
webhook-url: "http://mattermost.foxhunt.svc.cluster.local:8065/hooks/PLACEHOLDER"
|
||||
@@ -1,80 +0,0 @@
|
||||
# Argo Workflows Helm values — foxhunt cluster
|
||||
# Chart: argo/argo-workflows v1.0.2 (Argo v4.0.2)
|
||||
# Install: helm upgrade argo-workflows argo/argo-workflows -n foxhunt -f infra/k8s/argo/values.yaml
|
||||
|
||||
controller:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 256Mi
|
||||
# Only watch foxhunt namespace (not cluster-wide)
|
||||
workflowNamespaces:
|
||||
- foxhunt
|
||||
# Workflow archive — persists workflow state + logs to PostgreSQL after pod GC
|
||||
persistence:
|
||||
archive: true
|
||||
postgresql:
|
||||
host: postgres.foxhunt.svc.cluster.local
|
||||
port: 5432
|
||||
database: argo_archive
|
||||
tableName: argo_workflows
|
||||
userNameSecret:
|
||||
name: argo-postgres-config
|
||||
key: username
|
||||
passwordSecret:
|
||||
name: argo-postgres-config
|
||||
key: password
|
||||
ssl: false
|
||||
sslMode: disable
|
||||
|
||||
server:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
ingress:
|
||||
enabled: false
|
||||
extraArgs:
|
||||
- --auth-mode=server
|
||||
- --secure=false
|
||||
|
||||
# Default artifact repository — MinIO (in-cluster)
|
||||
artifactRepository:
|
||||
archiveLogs: true
|
||||
s3:
|
||||
endpoint: minio.foxhunt.svc.cluster.local:9000
|
||||
insecure: true
|
||||
bucket: argo-logs
|
||||
accessKeySecret:
|
||||
name: minio-credentials
|
||||
key: access-key
|
||||
secretKeySecret:
|
||||
name: minio-credentials
|
||||
key: secret-key
|
||||
|
||||
executor:
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
|
||||
workflow:
|
||||
serviceAccount:
|
||||
create: true
|
||||
name: argo-workflow
|
||||
|
||||
crds:
|
||||
install: true
|
||||
@@ -1,72 +0,0 @@
|
||||
# ACME (Let's Encrypt) wildcard cert for *.fxhnt.ai via cert-manager + Scaleway DNS-01.
|
||||
# Replaces the prior MANUAL, unrenewed LE cert that expired 2026-05-26 (the tailscale-proxy
|
||||
# fell back to the GitLab self-signed cert). DNS-01 is required: the services are tailnet-only
|
||||
# (*.fxhnt.ai -> 100.x Tailscale CGNAT) so HTTP-01 is unreachable, and wildcards need DNS-01.
|
||||
#
|
||||
# Solver creds: cert-manager/scaleway-dns-credentials (SCW_ACCESS_KEY/SCW_SECRET_KEY, copied from
|
||||
# foxhunt/scaleway-credentials). Webhook: scaleway-certmanager-webhook (helm, cert-manager ns).
|
||||
# The issued secret gitlab-tls-cert is mounted by infra/k8s/gitlab/tailscale-proxy.yaml.
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: letsencrypt-staging
|
||||
spec:
|
||||
acme:
|
||||
email: jeroen@grusewski.nl
|
||||
server: https://acme-staging-v02.api.letsencrypt.org/directory
|
||||
privateKeySecretRef:
|
||||
name: letsencrypt-staging-account-key
|
||||
solvers:
|
||||
- dns01:
|
||||
webhook:
|
||||
groupName: acme.scaleway.com
|
||||
solverName: scaleway
|
||||
config:
|
||||
accessKeySecretRef:
|
||||
name: scaleway-dns-credentials
|
||||
key: SCW_ACCESS_KEY
|
||||
secretKeySecretRef:
|
||||
name: scaleway-dns-credentials
|
||||
key: SCW_SECRET_KEY
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: letsencrypt-prod
|
||||
spec:
|
||||
acme:
|
||||
email: jeroen@grusewski.nl
|
||||
server: https://acme-v02.api.letsencrypt.org/directory
|
||||
privateKeySecretRef:
|
||||
name: letsencrypt-prod-account-key
|
||||
solvers:
|
||||
- dns01:
|
||||
webhook:
|
||||
groupName: acme.scaleway.com
|
||||
solverName: scaleway
|
||||
config:
|
||||
accessKeySecretRef:
|
||||
name: scaleway-dns-credentials
|
||||
key: SCW_ACCESS_KEY
|
||||
secretKeySecretRef:
|
||||
name: scaleway-dns-credentials
|
||||
key: SCW_SECRET_KEY
|
||||
---
|
||||
# Wildcard cert for the tailscale-proxy. issuerRef flips staging->prod after staging validates;
|
||||
# secretName flips to gitlab-tls-cert (the secret the proxy mounts) for the prod issuance.
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: fxhnt-wildcard
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
secretName: gitlab-tls-cert # takes over the secret the tailscale-proxy mounts
|
||||
privateKey:
|
||||
rotationPolicy: Always # old manual key had a mismatching algorithm; regenerate on issue/renew
|
||||
dnsNames:
|
||||
- fxhnt.ai
|
||||
- "*.fxhnt.ai"
|
||||
issuerRef:
|
||||
name: letsencrypt-prod
|
||||
kind: ClusterIssuer
|
||||
@@ -1,249 +0,0 @@
|
||||
# Postgres backup — daily pg_dump to MinIO S3
|
||||
# Uses initContainer (postgres:16-alpine) for pg_dump, then main container
|
||||
# (python:3.12-alpine, already cached) uploads to MinIO via S3 API.
|
||||
# Retains 7 daily + 4 weekly backups.
|
||||
# Apply: kubectl apply -f postgres-backup.yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: postgres-backup-script
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: postgres-backup
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
data:
|
||||
upload.py: |
|
||||
#!/usr/bin/env python3
|
||||
"""Upload pg_dump backup to MinIO, prune old backups."""
|
||||
import hashlib, hmac, os, sys, glob, urllib.request, urllib.error
|
||||
from datetime import datetime, timezone
|
||||
|
||||
ENDPOINT = os.environ["MINIO_ENDPOINT"] # http://minio.foxhunt...:9000
|
||||
ACCESS_KEY = os.environ["MINIO_ACCESS_KEY"]
|
||||
SECRET_KEY = os.environ["MINIO_SECRET_KEY"]
|
||||
BUCKET = "foxhunt-backups"
|
||||
BACKUP_DIR = "/backup"
|
||||
|
||||
def s3_sign(method, path, headers, payload_hash):
|
||||
"""AWS Signature V4 for MinIO."""
|
||||
now = datetime.now(timezone.utc)
|
||||
date_stamp = now.strftime("%Y%m%d")
|
||||
amz_date = now.strftime("%Y%m%dT%H%M%SZ")
|
||||
region = "us-east-1"
|
||||
service = "s3"
|
||||
scope = f"{date_stamp}/{region}/{service}/aws4_request"
|
||||
|
||||
headers["x-amz-date"] = amz_date
|
||||
headers["x-amz-content-sha256"] = payload_hash
|
||||
|
||||
signed_headers = ";".join(sorted(headers.keys()))
|
||||
canonical_headers = "".join(f"{k}:{headers[k]}\n" for k in sorted(headers.keys()))
|
||||
canonical_request = f"{method}\n{path}\n\n{canonical_headers}\n{signed_headers}\n{payload_hash}"
|
||||
|
||||
string_to_sign = f"AWS4-HMAC-SHA256\n{amz_date}\n{scope}\n{hashlib.sha256(canonical_request.encode()).hexdigest()}"
|
||||
|
||||
def sign(key, msg):
|
||||
return hmac.new(key, msg.encode(), hashlib.sha256).digest()
|
||||
|
||||
signing_key = sign(sign(sign(sign(
|
||||
f"AWS4{SECRET_KEY}".encode(), date_stamp), region), service), "aws4_request")
|
||||
signature = hmac.new(signing_key, string_to_sign.encode(), hashlib.sha256).hexdigest()
|
||||
|
||||
headers["Authorization"] = (
|
||||
f"AWS4-HMAC-SHA256 Credential={ACCESS_KEY}/{scope}, "
|
||||
f"SignedHeaders={signed_headers}, Signature={signature}"
|
||||
)
|
||||
return headers
|
||||
|
||||
def s3_put(key, data):
|
||||
path = f"/{BUCKET}/{key}"
|
||||
url = f"{ENDPOINT}{path}"
|
||||
payload_hash = hashlib.sha256(data).hexdigest()
|
||||
host = ENDPOINT.split("//")[1]
|
||||
headers = {"host": host, "content-length": str(len(data))}
|
||||
headers = s3_sign("PUT", path, headers, payload_hash)
|
||||
req = urllib.request.Request(url, data=data, method="PUT", headers=headers)
|
||||
urllib.request.urlopen(req, timeout=120)
|
||||
|
||||
def s3_list(prefix):
|
||||
path = f"/{BUCKET}?prefix={prefix}&list-type=2"
|
||||
url = f"{ENDPOINT}{path}"
|
||||
host = ENDPOINT.split("//")[1]
|
||||
headers = {"host": host}
|
||||
headers = s3_sign("GET", f"/{BUCKET}", headers, hashlib.sha256(b"").hexdigest())
|
||||
req = urllib.request.Request(f"{url}", method="GET", headers=headers)
|
||||
try:
|
||||
resp = urllib.request.urlopen(req, timeout=30).read().decode()
|
||||
import re
|
||||
return sorted(re.findall(r"<Key>([^<]+)</Key>", resp))
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
def s3_delete(key):
|
||||
path = f"/{BUCKET}/{key}"
|
||||
url = f"{ENDPOINT}{path}"
|
||||
host = ENDPOINT.split("//")[1]
|
||||
headers = {"host": host}
|
||||
headers = s3_sign("DELETE", path, headers, hashlib.sha256(b"").hexdigest())
|
||||
req = urllib.request.Request(url, method="DELETE", headers=headers)
|
||||
urllib.request.urlopen(req, timeout=30)
|
||||
|
||||
def s3_ensure_bucket():
|
||||
path = f"/{BUCKET}"
|
||||
url = f"{ENDPOINT}{path}"
|
||||
host = ENDPOINT.split("//")[1]
|
||||
headers = {"host": host}
|
||||
headers = s3_sign("PUT", path, headers, hashlib.sha256(b"").hexdigest())
|
||||
req = urllib.request.Request(url, method="PUT", headers=headers)
|
||||
try:
|
||||
urllib.request.urlopen(req, timeout=30)
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code not in (409, 200): # 409 = bucket exists
|
||||
raise
|
||||
|
||||
# Find the backup file
|
||||
files = glob.glob(f"{BACKUP_DIR}/*.sql.gz")
|
||||
if not files:
|
||||
print("ERROR: No backup file found in /backup/")
|
||||
sys.exit(1)
|
||||
|
||||
backup_file = files[0]
|
||||
filename = os.path.basename(backup_file)
|
||||
size_mb = os.path.getsize(backup_file) / (1024 * 1024)
|
||||
print(f"Uploading {filename} ({size_mb:.1f} MB)")
|
||||
|
||||
s3_ensure_bucket()
|
||||
|
||||
with open(backup_file, "rb") as f:
|
||||
data = f.read()
|
||||
|
||||
# Upload daily
|
||||
s3_put(f"daily/{filename}", data)
|
||||
print(f" Uploaded daily/{filename}")
|
||||
|
||||
# On Sundays, also upload weekly
|
||||
if datetime.now(timezone.utc).isoweekday() == 7:
|
||||
s3_put(f"weekly/{filename}", data)
|
||||
print(f" Uploaded weekly/{filename}")
|
||||
|
||||
# Prune old dailies (keep 7)
|
||||
dailies = s3_list("daily/")
|
||||
if len(dailies) > 7:
|
||||
for old in dailies[:len(dailies) - 7]:
|
||||
s3_delete(old)
|
||||
print(f" Pruned {old}")
|
||||
|
||||
# Prune old weeklies (keep 4)
|
||||
weeklies = s3_list("weekly/")
|
||||
if len(weeklies) > 4:
|
||||
for old in weeklies[:len(weeklies) - 4]:
|
||||
s3_delete(old)
|
||||
print(f" Pruned {old}")
|
||||
|
||||
print("=== Backup upload complete ===")
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: postgres-backup
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: postgres-backup
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
schedule: "30 2 * * *"
|
||||
successfulJobsHistoryLimit: 1
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
activeDeadlineSeconds: 600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: postgres-backup
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
restartPolicy: OnFailure
|
||||
# Step 1: pg_dump in initContainer → shared /backup volume
|
||||
initContainers:
|
||||
- name: pg-dump
|
||||
image: postgres:16-alpine
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
TIMESTAMP=$(date -u +%Y%m%d-%H%M%S)
|
||||
echo "=== Postgres Backup ${TIMESTAMP} ==="
|
||||
PGPASSWORD="${POSTGRES_PASSWORD}" pg_dumpall \
|
||||
-h "${POSTGRES_HOST}" \
|
||||
-p 5432 \
|
||||
-U "${POSTGRES_USER}" \
|
||||
--clean --if-exists \
|
||||
| gzip > "/backup/foxhunt-${TIMESTAMP}.sql.gz"
|
||||
ls -lh /backup/
|
||||
echo "Dump complete"
|
||||
env:
|
||||
- name: POSTGRES_HOST
|
||||
value: "postgres.foxhunt.svc.cluster.local"
|
||||
- name: POSTGRES_USER
|
||||
value: "foxhunt"
|
||||
- name: POSTGRES_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: db-credentials
|
||||
key: password
|
||||
volumeMounts:
|
||||
- name: backup
|
||||
mountPath: /backup
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
# Step 2: Upload to MinIO in main container (python cached on nodes)
|
||||
containers:
|
||||
- name: upload
|
||||
image: python:3.12-alpine
|
||||
command: ["python3", "-u", "/scripts/upload.py"]
|
||||
env:
|
||||
- name: MINIO_ENDPOINT
|
||||
value: "http://minio.foxhunt.svc.cluster.local:9000"
|
||||
- name: MINIO_ACCESS_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: minio-credentials
|
||||
key: access-key
|
||||
- name: MINIO_SECRET_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: minio-credentials
|
||||
key: secret-key
|
||||
volumeMounts:
|
||||
- name: backup
|
||||
mountPath: /backup
|
||||
- name: script
|
||||
mountPath: /scripts
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
volumes:
|
||||
- name: backup
|
||||
emptyDir:
|
||||
sizeLimit: 2Gi
|
||||
- name: script
|
||||
configMap:
|
||||
name: postgres-backup-script
|
||||
items:
|
||||
- key: upload.py
|
||||
path: upload.py
|
||||
defaultMode: 0755
|
||||
@@ -1,96 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: postgres-pvc
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: postgres
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 20Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: postgres
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: postgres
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: postgres
|
||||
strategy:
|
||||
type: Recreate
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: postgres
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
containers:
|
||||
- name: postgres
|
||||
image: timescale/timescaledb:latest-pg16
|
||||
ports:
|
||||
- containerPort: 5432
|
||||
name: postgres
|
||||
args:
|
||||
- -c
|
||||
- max_connections=100
|
||||
env:
|
||||
- name: POSTGRES_DB
|
||||
value: foxhunt
|
||||
- name: POSTGRES_USER
|
||||
value: foxhunt
|
||||
- name: POSTGRES_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: db-credentials
|
||||
key: password
|
||||
volumeMounts:
|
||||
- name: postgres-data
|
||||
mountPath: /var/lib/postgresql/data
|
||||
subPath: pgdata
|
||||
readinessProbe:
|
||||
exec:
|
||||
command:
|
||||
- pg_isready
|
||||
- -U
|
||||
- foxhunt
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 1Gi
|
||||
volumes:
|
||||
- name: postgres-data
|
||||
persistentVolumeClaim:
|
||||
claimName: postgres-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: postgres
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: postgres
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: postgres
|
||||
ports:
|
||||
- port: 5432
|
||||
targetPort: 5432
|
||||
name: postgres
|
||||
@@ -1,95 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: questdb-pvc
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: questdb
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: questdb
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: questdb
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: questdb
|
||||
strategy:
|
||||
type: Recreate
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: questdb
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
containers:
|
||||
- name: questdb
|
||||
image: questdb/questdb:8.2.3
|
||||
ports:
|
||||
- containerPort: 9009
|
||||
name: ilp
|
||||
- containerPort: 8812
|
||||
name: pg
|
||||
- containerPort: 9003
|
||||
name: metrics
|
||||
env:
|
||||
- name: QDB_PG_ENABLED
|
||||
value: "true"
|
||||
- name: QDB_METRICS_ENABLED
|
||||
value: "TRUE"
|
||||
volumeMounts:
|
||||
- name: questdb-data
|
||||
mountPath: /var/lib/questdb
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /exec?query=SELECT%201
|
||||
port: 9003
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 1Gi
|
||||
volumes:
|
||||
- name: questdb-data
|
||||
persistentVolumeClaim:
|
||||
claimName: questdb-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: questdb
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: questdb
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: questdb
|
||||
ports:
|
||||
- port: 9009
|
||||
targetPort: 9009
|
||||
name: ilp
|
||||
- port: 8812
|
||||
targetPort: 8812
|
||||
name: pg
|
||||
- port: 9003
|
||||
targetPort: 9003
|
||||
name: metrics
|
||||
@@ -1,63 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: redis
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: redis
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
containers:
|
||||
- name: redis
|
||||
image: redis:7-alpine
|
||||
command:
|
||||
- redis-server
|
||||
- --maxmemory
|
||||
- 512mb
|
||||
- --maxmemory-policy
|
||||
- allkeys-lru
|
||||
ports:
|
||||
- containerPort: 6379
|
||||
name: redis
|
||||
readinessProbe:
|
||||
exec:
|
||||
command:
|
||||
- redis-cli
|
||||
- ping
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: 250m
|
||||
memory: 640Mi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: redis
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: redis
|
||||
ports:
|
||||
- port: 6379
|
||||
targetPort: 6379
|
||||
name: redis
|
||||
@@ -1,182 +0,0 @@
|
||||
# Node Bootstrap DaemonSet for Kapsule
|
||||
#
|
||||
# Two problems on Kapsule nodes:
|
||||
# 1. Nodes have nameserver 127.0.53.53 which can't resolve .svc.cluster.local
|
||||
# 2. Containerd defaults to HTTPS for non-localhost registries
|
||||
#
|
||||
# Fixes:
|
||||
# 1. Point /etc/resolv.conf to kube-dns ClusterIP (10.32.0.10)
|
||||
# 2. Create /etc/containerd/certs.d/ config for HTTP GitLab registry
|
||||
# 3. On GPU nodes (containerd v3 with conf.d/), add registry drop-in
|
||||
#
|
||||
# Init container sets up containerd registry config (one-time).
|
||||
# Main container manages resolv.conf (watches for Kapsule resets).
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: node-bootstrap
|
||||
namespace: kube-system
|
||||
labels:
|
||||
app.kubernetes.io/name: node-bootstrap
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: node-bootstrap
|
||||
updateStrategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxUnavailable: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: node-bootstrap
|
||||
spec:
|
||||
hostNetwork: true
|
||||
dnsPolicy: Default
|
||||
priorityClassName: system-node-critical
|
||||
tolerations:
|
||||
- operator: Exists
|
||||
hostPID: true
|
||||
initContainers:
|
||||
# Configure containerd to use HTTP for the GitLab registry.
|
||||
# On GPU nodes (NVIDIA overlay), also creates a v3-compatible conf.d drop-in
|
||||
# and restarts containerd so it picks up the new config.
|
||||
- name: setup-registry
|
||||
image: alpine:3.19
|
||||
securityContext:
|
||||
privileged: true
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -e
|
||||
NEEDS_RESTART=false
|
||||
|
||||
# 1. Create per-host containerd registry config (HTTP for GitLab)
|
||||
REGISTRY_DIR="/host-etc/containerd/certs.d/gitlab-registry.foxhunt.svc.cluster.local:5000"
|
||||
mkdir -p "$REGISTRY_DIR"
|
||||
DESIRED='server = "http://gitlab-registry.foxhunt.svc.cluster.local:5000"
|
||||
|
||||
[host."http://gitlab-registry.foxhunt.svc.cluster.local:5000"]
|
||||
capabilities = ["pull", "resolve"]'
|
||||
DESIRED=$(printf '%s' "$DESIRED" | sed 's/^ //')
|
||||
if [ ! -f "$REGISTRY_DIR/hosts.toml" ] || [ "$(cat "$REGISTRY_DIR/hosts.toml")" != "$DESIRED" ]; then
|
||||
printf '%s\n' "$DESIRED" > "$REGISTRY_DIR/hosts.toml"
|
||||
echo "Created hosts.toml for HTTP gitlab-registry"
|
||||
NEEDS_RESTART=true
|
||||
else
|
||||
echo "hosts.toml already up to date"
|
||||
fi
|
||||
|
||||
# 2. Ensure conf.d/ exists and always write the v3 registry drop-in.
|
||||
# containerd 2.x may ignore the v2-style grpc.v1.cri.registry.config_path
|
||||
# even when it's set in config.toml. The v1 images drop-in is the reliable path.
|
||||
# Create conf.d/ if missing (base config already has imports = ["conf.d/*.toml"]).
|
||||
CONF_D="/host-etc/containerd/conf.d"
|
||||
mkdir -p "$CONF_D"
|
||||
if [ -d "$CONF_D" ]; then
|
||||
DROPIN='version = 3
|
||||
|
||||
[plugins."io.containerd.cri.v1.images".registry]
|
||||
config_path = "/etc/containerd/certs.d"'
|
||||
DROPIN=$(printf '%s' "$DROPIN" | sed 's/^ //')
|
||||
if [ ! -f "$CONF_D/50-registry.toml" ] || [ "$(cat "$CONF_D/50-registry.toml")" != "$DROPIN" ]; then
|
||||
printf '%s\n' "$DROPIN" > "$CONF_D/50-registry.toml"
|
||||
echo "GPU node: created v3 registry config drop-in"
|
||||
NEEDS_RESTART=true
|
||||
else
|
||||
echo "GPU node: v3 drop-in already up to date"
|
||||
fi
|
||||
fi
|
||||
|
||||
# 3. Restart containerd only if we changed config files.
|
||||
# Uses /proc/1/root chroot to access host systemctl (requires hostPID + privileged).
|
||||
# On a new node this will kill the DaemonSet pod; kubelet recreates it,
|
||||
# second run finds files unchanged (NEEDS_RESTART=false), proceeds normally.
|
||||
if [ "$NEEDS_RESTART" = "true" ]; then
|
||||
echo "Restarting containerd to pick up config changes..."
|
||||
chroot /proc/1/root systemctl restart containerd || \
|
||||
echo "WARNING: containerd restart failed (node may need manual restart)"
|
||||
echo "containerd restart issued"
|
||||
fi
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 8Mi
|
||||
limits:
|
||||
cpu: 50m
|
||||
memory: 16Mi
|
||||
volumeMounts:
|
||||
- name: host-etc
|
||||
mountPath: /host-etc
|
||||
containers:
|
||||
- name: resolv-manager
|
||||
image: busybox:1.37
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -e
|
||||
RESOLV="/host-etc/resolv.conf"
|
||||
BACKUP="/host-etc/resolv.conf.pre-kube-dns"
|
||||
KUBE_DNS="10.32.0.10"
|
||||
|
||||
restore() {
|
||||
if [ -f "$BACKUP" ]; then
|
||||
cp "$BACKUP" "$RESOLV"
|
||||
echo "$(date): Restored original resolv.conf"
|
||||
fi
|
||||
exit 0
|
||||
}
|
||||
trap restore TERM INT
|
||||
|
||||
# Save original
|
||||
if [ ! -f "$BACKUP" ]; then
|
||||
cp "$RESOLV" "$BACKUP"
|
||||
echo "Saved original: $(cat "$BACKUP")"
|
||||
fi
|
||||
|
||||
apply() {
|
||||
# CRITICAL: keep VPC upstream resolver as FALLBACK after kube-dns.
|
||||
# If resolv.conf ONLY has 10.32.0.10, node DNS is dead when CoreDNS
|
||||
# is down — containerd can't pull images, CoreDNS can't start,
|
||||
# entire cluster deadlocks (incident 2026-03-18, 2026-04-12).
|
||||
# Scaleway VPC gateway pushes 172.16.0.13 as DNS via DHCP.
|
||||
UPSTREAM=""
|
||||
if [ -f "$BACKUP" ]; then
|
||||
UPSTREAM=$(grep "^nameserver" "$BACKUP" | grep -v "127.0.53.53\|$KUBE_DNS" | head -1 | awk '{print $2}')
|
||||
fi
|
||||
if [ -z "$UPSTREAM" ]; then
|
||||
UPSTREAM="169.254.169.254" # Scaleway metadata DNS fallback
|
||||
fi
|
||||
printf 'nameserver %s\nnameserver %s\nsearch foxhunt.svc.cluster.local svc.cluster.local cluster.local\noptions ndots:5 timeout:2 attempts:3\n' "$KUBE_DNS" "$UPSTREAM" > "$RESOLV"
|
||||
echo "$(date): resolv.conf → $KUBE_DNS + $UPSTREAM (fallback)"
|
||||
}
|
||||
|
||||
# Initial apply
|
||||
apply
|
||||
|
||||
# Watch for resets
|
||||
while true; do
|
||||
sleep 30 &
|
||||
wait $!
|
||||
if ! grep -q "$KUBE_DNS" "$RESOLV" 2>/dev/null; then
|
||||
echo "$(date): resolv.conf was reset, re-applying"
|
||||
apply
|
||||
fi
|
||||
done
|
||||
resources:
|
||||
requests:
|
||||
cpu: 5m
|
||||
memory: 8Mi
|
||||
limits:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
volumeMounts:
|
||||
- name: host-etc
|
||||
mountPath: /host-etc
|
||||
terminationGracePeriodSeconds: 10
|
||||
volumes:
|
||||
- name: host-etc
|
||||
hostPath:
|
||||
path: /etc
|
||||
type: Directory
|
||||
@@ -1,40 +0,0 @@
|
||||
# The gitea chart's gitea-http/gitea-ssh services are HEADLESS (clusterIP: None), so they resolve to the
|
||||
# pod IP (100.64.x.x). That range OVERLAPS the Tailscale CGNAT range (100.64.0.0/10), so the tailscale
|
||||
# sidecar in the tailscale-gitlab-proxy pod swallows traffic to the pod IP. These normal ClusterIP services
|
||||
# give Gitea a service IP in the service CIDR (10.32.x.x, outside the tailscale range) — the proxy nginx +
|
||||
# socat target these instead. (GitLab worked because its webservice svc was a normal ClusterIP.)
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: gitea-web
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
ports:
|
||||
- name: http
|
||||
port: 3000
|
||||
targetPort: 3000
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: gitea-sshd
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app.kubernetes.io/name: gitea
|
||||
app.kubernetes.io/instance: gitea
|
||||
ports:
|
||||
- name: ssh
|
||||
port: 22
|
||||
targetPort: 22
|
||||
protocol: TCP
|
||||
@@ -1,61 +0,0 @@
|
||||
# Gitea NetworkPolicy (foxhunt ns has default-deny-all). Egress: postgres + DNS + in-cluster webhook
|
||||
# target + general HTTPS. Ingress: from platform/argo pods + the tailscale proxy → :3000 (web) and :22 (ssh).
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: gitea
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
# web (3000) + ssh (22) from platform + argo-workflow pods (cockpit build clones over HTTP)
|
||||
- from:
|
||||
- podSelector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/part-of
|
||||
operator: In
|
||||
values: ["foxhunt", "argo-workflows"]
|
||||
ports:
|
||||
- { port: 3000, protocol: TCP }
|
||||
- { port: 22, protocol: TCP }
|
||||
# web + ssh from the tailscale proxy (nginx + socat) that fronts git.fxhnt.ai
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tailscale-gitlab-proxy
|
||||
ports:
|
||||
- { port: 3000, protocol: TCP }
|
||||
- { port: 22, protocol: TCP }
|
||||
egress:
|
||||
# PostgreSQL
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: postgres
|
||||
ports:
|
||||
- { port: 5432, protocol: TCP }
|
||||
# DNS
|
||||
- ports:
|
||||
- { port: 53, protocol: UDP }
|
||||
- { port: 53, protocol: TCP }
|
||||
# Argo Events webhook target (gitea push hook → eventsource :12000, in-cluster)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/part-of
|
||||
operator: In
|
||||
values: ["foxhunt", "argo-workflows"]
|
||||
ports:
|
||||
- { port: 12000, protocol: TCP }
|
||||
# general HTTPS (avatars, external clones) — external only, exclude cluster CIDRs
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except: ["10.32.0.0/16", "172.16.0.0/16"]
|
||||
ports:
|
||||
- { port: 443, protocol: TCP }
|
||||
@@ -1,64 +0,0 @@
|
||||
# Gitea — lightweight git host replacing GitLab (Phase 2B). External Postgres (existing in-cluster
|
||||
# `postgres`), no bundled DB/redis/memcached, Actions off (Argo does CI). Internal-only until cutover.
|
||||
replicaCount: 1
|
||||
image:
|
||||
rootless: true
|
||||
|
||||
# Disable all bundled subcharts — reuse the existing in-cluster postgres, no redis/memcached.
|
||||
postgresql:
|
||||
enabled: false
|
||||
postgresql-ha:
|
||||
enabled: false
|
||||
redis-cluster:
|
||||
enabled: false
|
||||
redis:
|
||||
enabled: false
|
||||
# chart 12.x replaced redis with valkey — disable both, gitea uses embedded queue/cache/session
|
||||
valkey-cluster:
|
||||
enabled: false
|
||||
valkey:
|
||||
enabled: false
|
||||
|
||||
persistence:
|
||||
enabled: true
|
||||
size: 5Gi
|
||||
storageClass: sbs-default-retain
|
||||
|
||||
resources:
|
||||
requests: { cpu: 100m, memory: 128Mi }
|
||||
limits: { cpu: "1", memory: 512Mi }
|
||||
|
||||
service:
|
||||
http: { type: ClusterIP, port: 3000 }
|
||||
ssh: { type: ClusterIP, port: 22 }
|
||||
|
||||
gitea:
|
||||
admin:
|
||||
existingSecret: gitea-admin
|
||||
config:
|
||||
actions:
|
||||
ENABLED: "false" # Argo does CI/CD; Gitea Actions off (subchart removed in chart 12.x)
|
||||
server:
|
||||
ROOT_URL: https://git.fxhnt.ai/
|
||||
DOMAIN: git.fxhnt.ai
|
||||
SSH_DOMAIN: git.fxhnt.ai
|
||||
SSH_PORT: "22" # clean git@git.fxhnt.ai clone URLs; :2222 retired (Task 5).
|
||||
DISABLE_SSH: "false"
|
||||
database:
|
||||
DB_TYPE: postgres
|
||||
HOST: postgres.foxhunt.svc.cluster.local:5432
|
||||
NAME: gitea
|
||||
USER: gitea
|
||||
service:
|
||||
DISABLE_REGISTRATION: "true"
|
||||
# No redis/memcached — use embedded adapters
|
||||
cache:
|
||||
ADAPTER: memory
|
||||
session:
|
||||
PROVIDER: db
|
||||
queue:
|
||||
TYPE: level
|
||||
additionalConfigFromEnvs:
|
||||
- name: GITEA__database__PASSWD
|
||||
valueFrom:
|
||||
secretKeyRef: { name: gitea-db, key: password }
|
||||
@@ -1,120 +0,0 @@
|
||||
# Grafana — Foxhunt monitoring dashboard
|
||||
# Accessed via Tailscale proxy on port 3000
|
||||
|
||||
replicas: 1
|
||||
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 256Mi
|
||||
|
||||
persistence:
|
||||
enabled: false
|
||||
|
||||
admin:
|
||||
existingSecret: grafana-admin
|
||||
userKey: admin-user
|
||||
passwordKey: admin-password
|
||||
|
||||
envFromSecrets:
|
||||
- name: db-credentials
|
||||
|
||||
grafana.ini:
|
||||
server:
|
||||
root_url: https://grafana.fxhnt.ai
|
||||
unified_alerting:
|
||||
enabled: true
|
||||
alerting:
|
||||
enabled: false
|
||||
database:
|
||||
type: postgres
|
||||
host: postgres:5432
|
||||
name: grafana
|
||||
user: foxhunt
|
||||
password: $__env{password}
|
||||
ssl_mode: disable
|
||||
|
||||
datasources:
|
||||
datasources.yaml:
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
url: http://prometheus-stack-kube-prom-prometheus.foxhunt.svc.cluster.local:9090
|
||||
access: proxy
|
||||
isDefault: true
|
||||
- name: Alertmanager
|
||||
type: alertmanager
|
||||
uid: alertmanager
|
||||
access: proxy
|
||||
url: http://prometheus-stack-kube-prom-alertmanager.foxhunt.svc.cluster.local:9093
|
||||
isDefault: false
|
||||
jsonData:
|
||||
implementation: prometheus
|
||||
handleGrafanaManagedAlerts: true
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki.foxhunt.svc.cluster.local:3100
|
||||
isDefault: false
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- datasourceUid: tempo
|
||||
matcherRegex: '"trace_id":"(\w+)"'
|
||||
name: TraceID
|
||||
url: "$${__value.raw}"
|
||||
- name: Tempo
|
||||
type: tempo
|
||||
uid: tempo
|
||||
access: proxy
|
||||
url: http://tempo.foxhunt.svc.cluster.local:3200
|
||||
isDefault: false
|
||||
jsonData:
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
filterByTraceID: true
|
||||
nodeGraph:
|
||||
enabled: true
|
||||
serviceMap:
|
||||
datasourceUid: prometheus
|
||||
|
||||
sidecar:
|
||||
dashboards:
|
||||
enabled: true
|
||||
folder: /tmp/dashboards
|
||||
folderAnnotation: grafana_folder
|
||||
label: grafana_dashboard
|
||||
labelValue: "1"
|
||||
searchNamespace: foxhunt
|
||||
provider:
|
||||
foldersFromFilesStructure: true
|
||||
datasources:
|
||||
enabled: false
|
||||
label: grafana_datasource
|
||||
labelValue: ""
|
||||
|
||||
dashboardProviders:
|
||||
dashboardproviders.yaml:
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: default
|
||||
orgId: 1
|
||||
folder: Foxhunt
|
||||
type: file
|
||||
disableDeletion: false
|
||||
editable: true
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards/default
|
||||
|
||||
dashboards:
|
||||
default:
|
||||
node-exporter:
|
||||
gnetId: 1860
|
||||
revision: 37
|
||||
datasource: Prometheus
|
||||
@@ -1,253 +0,0 @@
|
||||
# GitLab PAT auto-rotation CronJob
|
||||
# Rotates the personal access token monthly, storing the new token in a K8s Secret.
|
||||
# Uses GitLab's atomic rotate API: creates new token + revokes old in one call.
|
||||
#
|
||||
# The rotation runs on the 1st of every month. Additionally, a weekly expiry check
|
||||
# runs every Monday and logs a warning if the token expires within 14 days, giving
|
||||
# time to investigate rotation failures before CI breaks.
|
||||
#
|
||||
# Initial setup:
|
||||
# kubectl -n foxhunt create secret generic gitlab-pat \
|
||||
# --from-literal=token=glpat-<initial-token>
|
||||
#
|
||||
# Manual rotation:
|
||||
# kubectl -n foxhunt create job pat-rotate-manual --from=cronjob/gitlab-pat-rotation
|
||||
#
|
||||
# Manual expiry check:
|
||||
# kubectl -n foxhunt create job pat-check-manual --from=cronjob/gitlab-pat-expiry-check
|
||||
#
|
||||
# Sync local token:
|
||||
# export GITLAB_TOKEN=$(kubectl -n foxhunt get secret gitlab-pat -o jsonpath='{.data.token}' | base64 -d)
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: gitlab-pat-rotator
|
||||
namespace: foxhunt
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: gitlab-pat-rotator
|
||||
namespace: foxhunt
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
resourceNames: ["gitlab-pat"]
|
||||
verbs: ["get", "patch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: gitlab-pat-rotator
|
||||
namespace: foxhunt
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: gitlab-pat-rotator
|
||||
namespace: foxhunt
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: gitlab-pat-rotator
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
---
|
||||
# Monthly rotation — creates new token, revokes old, updates K8s secret
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: gitlab-pat-rotation
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
schedule: "0 3 1 * *" # 1st of every month at 03:00 UTC
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 5
|
||||
concurrencyPolicy: Forbid
|
||||
jobTemplate:
|
||||
spec:
|
||||
backoffLimit: 3
|
||||
activeDeadlineSeconds: 180
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: gitlab-pat-rotation
|
||||
spec:
|
||||
serviceAccountName: gitlab-pat-rotator
|
||||
restartPolicy: Never
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
containers:
|
||||
- name: rotate
|
||||
image: alpine:3.21
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -e
|
||||
|
||||
apk add --no-cache -q curl jq
|
||||
|
||||
CURRENT_PAT=$(cat /etc/gitlab-pat/token)
|
||||
GITLAB_URL="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
|
||||
|
||||
# 1. Verify current token is valid
|
||||
TOKEN_INFO=$(curl -sf "${GITLAB_URL}/api/v4/personal_access_tokens/self" \
|
||||
-H "PRIVATE-TOKEN: ${CURRENT_PAT}") || {
|
||||
echo "CRITICAL: Current PAT is invalid or GitLab unreachable."
|
||||
echo "Manual recovery required: create a new PAT in GitLab UI and run:"
|
||||
echo " kubectl -n foxhunt create secret generic gitlab-pat --from-literal=token=<new-pat> --dry-run=client -o yaml | kubectl apply -f -"
|
||||
exit 1
|
||||
}
|
||||
OLD_EXPIRY=$(echo "$TOKEN_INFO" | jq -r '.expires_at')
|
||||
TOKEN_NAME=$(echo "$TOKEN_INFO" | jq -r '.name')
|
||||
echo "Current PAT '${TOKEN_NAME}' expires: ${OLD_EXPIRY}"
|
||||
|
||||
# 2. Rotate: new expiry = 1 year from now
|
||||
EXPIRES=$(date -d "@$(($(date +%s) + 31536000))" +%Y-%m-%d)
|
||||
RESPONSE=$(curl -sf -X POST \
|
||||
"${GITLAB_URL}/api/v4/personal_access_tokens/self/rotate" \
|
||||
-H "PRIVATE-TOKEN: ${CURRENT_PAT}" \
|
||||
-d "expires_at=${EXPIRES}") || {
|
||||
echo "ERROR: Rotation API call failed. Token may still be valid."
|
||||
exit 1
|
||||
}
|
||||
|
||||
NEW_PAT=$(echo "$RESPONSE" | jq -r '.token')
|
||||
if [ "$NEW_PAT" = "null" ] || [ -z "$NEW_PAT" ]; then
|
||||
echo "ERROR: No token in rotation response: $RESPONSE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "PAT rotated successfully. New expiry: ${EXPIRES}"
|
||||
|
||||
# 3. Update K8s secret (old token is already revoked at this point)
|
||||
K8S_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
|
||||
K8S_CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
K8S_NS=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
|
||||
K8S_API="https://kubernetes.default.svc"
|
||||
|
||||
NEW_PAT_B64=$(printf '%s' "$NEW_PAT" | base64)
|
||||
|
||||
for attempt in 1 2 3 4 5; do
|
||||
if curl -sf -X PATCH \
|
||||
"${K8S_API}/api/v1/namespaces/${K8S_NS}/secrets/gitlab-pat" \
|
||||
--cacert "$K8S_CA" \
|
||||
-H "Authorization: Bearer ${K8S_TOKEN}" \
|
||||
-H "Content-Type: application/strategic-merge-patch+json" \
|
||||
-d "{\"data\":{\"token\":\"${NEW_PAT_B64}\"}}"; then
|
||||
echo ""
|
||||
echo "K8s secret updated on attempt ${attempt}."
|
||||
|
||||
# 4. Verify the new token works
|
||||
VERIFY=$(curl -sf "${GITLAB_URL}/api/v4/personal_access_tokens/self" \
|
||||
-H "PRIVATE-TOKEN: ${NEW_PAT}" | jq -r '.expires_at') || true
|
||||
if [ "$VERIFY" = "$EXPIRES" ]; then
|
||||
echo "Verification passed: new token valid until ${VERIFY}"
|
||||
else
|
||||
echo "WARNING: Verification returned unexpected expiry: ${VERIFY}"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
echo "Secret update attempt ${attempt} failed, retrying in 3s..."
|
||||
sleep 3
|
||||
done
|
||||
|
||||
# If we get here, rotation succeeded but secret update failed.
|
||||
# Print token so it can be recovered from job logs.
|
||||
echo "CRITICAL: Secret update failed after 5 attempts!"
|
||||
echo "New PAT (recover manually): ${NEW_PAT}"
|
||||
echo "Run: kubectl -n foxhunt create secret generic gitlab-pat --from-literal=token=<above> --dry-run=client -o yaml | kubectl apply -f -"
|
||||
exit 1
|
||||
volumeMounts:
|
||||
- name: gitlab-pat
|
||||
mountPath: /etc/gitlab-pat
|
||||
readOnly: true
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 64Mi
|
||||
volumes:
|
||||
- name: gitlab-pat
|
||||
secret:
|
||||
secretName: gitlab-pat
|
||||
---
|
||||
# Weekly expiry check — warns if token expires within 14 days
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: gitlab-pat-expiry-check
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
schedule: "0 8 * * 1" # Every Monday at 08:00 UTC
|
||||
successfulJobsHistoryLimit: 1
|
||||
failedJobsHistoryLimit: 3
|
||||
concurrencyPolicy: Forbid
|
||||
jobTemplate:
|
||||
spec:
|
||||
backoffLimit: 1
|
||||
activeDeadlineSeconds: 60
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: gitlab-pat-expiry-check
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
containers:
|
||||
- name: check
|
||||
image: alpine:3.21
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -e
|
||||
|
||||
apk add --no-cache -q curl jq
|
||||
|
||||
CURRENT_PAT=$(cat /etc/gitlab-pat/token)
|
||||
GITLAB_URL="http://gitlab-webservice-default.foxhunt.svc.cluster.local:8181"
|
||||
|
||||
TOKEN_INFO=$(curl -sf "${GITLAB_URL}/api/v4/personal_access_tokens/self" \
|
||||
-H "PRIVATE-TOKEN: ${CURRENT_PAT}") || {
|
||||
echo "CRITICAL: PAT is INVALID. CI will fail!"
|
||||
echo "Create a new PAT in GitLab UI and run:"
|
||||
echo " kubectl -n foxhunt create secret generic gitlab-pat --from-literal=token=<new-pat> --dry-run=client -o yaml | kubectl apply -f -"
|
||||
exit 1
|
||||
}
|
||||
|
||||
EXPIRY=$(echo "$TOKEN_INFO" | jq -r '.expires_at')
|
||||
TOKEN_NAME=$(echo "$TOKEN_INFO" | jq -r '.name')
|
||||
SCOPES=$(echo "$TOKEN_INFO" | jq -r '.scopes | join(",")')
|
||||
|
||||
# Calculate days until expiry
|
||||
EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s)
|
||||
NOW_EPOCH=$(date +%s)
|
||||
DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))
|
||||
|
||||
echo "PAT '${TOKEN_NAME}': expires=${EXPIRY} scopes=${SCOPES} days_left=${DAYS_LEFT}"
|
||||
|
||||
if [ "$DAYS_LEFT" -lt 0 ]; then
|
||||
echo "CRITICAL: PAT has EXPIRED ${DAYS_LEFT} days ago!"
|
||||
exit 1
|
||||
elif [ "$DAYS_LEFT" -lt 14 ]; then
|
||||
echo "WARNING: PAT expires in ${DAYS_LEFT} days. Rotation may have failed."
|
||||
echo "Manual rotation: kubectl -n foxhunt create job pat-rotate-manual --from=cronjob/gitlab-pat-rotation"
|
||||
exit 1
|
||||
else
|
||||
echo "OK: PAT valid for ${DAYS_LEFT} more days."
|
||||
fi
|
||||
volumeMounts:
|
||||
- name: gitlab-pat
|
||||
mountPath: /etc/gitlab-pat
|
||||
readOnly: true
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
volumes:
|
||||
- name: gitlab-pat
|
||||
secret:
|
||||
secretName: gitlab-pat
|
||||
@@ -1,63 +0,0 @@
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: gitlab-postgres-init
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: gitlab-postgres-init
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
ttlSecondsAfterFinished: 300
|
||||
backoffLimit: 3
|
||||
template:
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: init
|
||||
image: timescale/timescaledb:latest-pg16
|
||||
command:
|
||||
- bash
|
||||
- -c
|
||||
- |
|
||||
set -euo pipefail
|
||||
export PGPASSWORD="$POSTGRES_PASSWORD"
|
||||
|
||||
until pg_isready -h postgres -U foxhunt; do
|
||||
echo "Waiting for postgres..."
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "Creating gitlab user..."
|
||||
psql -h postgres -U foxhunt -d foxhunt -tc \
|
||||
"SELECT 1 FROM pg_roles WHERE rolname='gitlab'" | grep -q 1 || \
|
||||
psql -h postgres -U foxhunt -d foxhunt -c \
|
||||
"CREATE ROLE gitlab WITH LOGIN PASSWORD '${GITLAB_DB_PASSWORD}'"
|
||||
|
||||
echo "Creating gitlab database..."
|
||||
psql -h postgres -U foxhunt -tc \
|
||||
"SELECT 1 FROM pg_database WHERE datname='gitlab'" | grep -q 1 || \
|
||||
psql -h postgres -U foxhunt -c \
|
||||
"CREATE DATABASE gitlab OWNER gitlab"
|
||||
|
||||
echo "Creating extensions..."
|
||||
psql -h postgres -U foxhunt -d gitlab -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;"
|
||||
psql -h postgres -U foxhunt -d gitlab -c "CREATE EXTENSION IF NOT EXISTS btree_gist;"
|
||||
|
||||
echo "Granting permissions..."
|
||||
psql -h postgres -U foxhunt -d gitlab -c "GRANT ALL PRIVILEGES ON DATABASE gitlab TO gitlab;"
|
||||
psql -h postgres -U foxhunt -d gitlab -c "GRANT ALL ON SCHEMA public TO gitlab;"
|
||||
|
||||
echo "GitLab database initialized successfully."
|
||||
env:
|
||||
- name: POSTGRES_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: db-credentials
|
||||
key: password
|
||||
- name: GITLAB_DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: gitlab-secrets
|
||||
key: db-password
|
||||
@@ -1,352 +0,0 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: tailscale-gitlab
|
||||
namespace: foxhunt
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: tailscale-gitlab
|
||||
namespace: foxhunt
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["secrets"]
|
||||
verbs: ["create", "get", "update", "patch"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: tailscale-gitlab
|
||||
namespace: foxhunt
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: tailscale-gitlab
|
||||
namespace: foxhunt
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: tailscale-gitlab
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: tailscale-gitlab-proxy
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: tailscale-gitlab-proxy
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tailscale-gitlab-proxy
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: tailscale-gitlab-proxy
|
||||
spec:
|
||||
serviceAccountName: tailscale-gitlab
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
containers:
|
||||
- name: nginx
|
||||
image: nginx:alpine
|
||||
ports:
|
||||
- containerPort: 80
|
||||
- containerPort: 443
|
||||
- containerPort: 5050
|
||||
volumeMounts:
|
||||
- name: nginx-conf
|
||||
mountPath: /etc/nginx/conf.d
|
||||
- name: tls-certs
|
||||
mountPath: /etc/nginx/certs
|
||||
readOnly: true
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
- name: ssh-proxy
|
||||
image: alpine/socat:latest
|
||||
args:
|
||||
- "TCP-LISTEN:22,fork,reuseaddr,sndbuf=1048576,rcvbuf=1048576"
|
||||
- "TCP:gitea-sshd.foxhunt.svc.cluster.local:22,sndbuf=1048576,rcvbuf=1048576"
|
||||
ports:
|
||||
- containerPort: 22
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 128Mi
|
||||
- name: imap-proxy
|
||||
image: alpine/socat:latest
|
||||
args:
|
||||
- "TCP-LISTEN:993,fork,reuseaddr"
|
||||
- "TCP:stalwart.foxhunt.svc.cluster.local:993"
|
||||
ports:
|
||||
- containerPort: 993
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
cpu: 50m
|
||||
memory: 32Mi
|
||||
- name: tailscale
|
||||
image: ghcr.io/tailscale/tailscale:latest
|
||||
env:
|
||||
- name: TS_AUTHKEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: tailscale-auth
|
||||
key: TS_AUTHKEY
|
||||
- name: TS_KUBE_SECRET
|
||||
value: tailscale-gitlab-state
|
||||
- name: TS_USERSPACE
|
||||
value: "false"
|
||||
- name: TS_HOSTNAME
|
||||
value: "foxhunt-gitlab"
|
||||
- name: TS_ACCEPT_DNS
|
||||
value: "false"
|
||||
- name: TS_EXTRA_ARGS
|
||||
value: ""
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
securityContext:
|
||||
capabilities:
|
||||
add:
|
||||
- NET_ADMIN
|
||||
- NET_RAW
|
||||
volumes:
|
||||
- name: nginx-conf
|
||||
configMap:
|
||||
name: tailscale-gitlab-nginx
|
||||
- name: tls-certs
|
||||
secret:
|
||||
secretName: gitlab-tls-cert
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: tailscale-gitlab-nginx
|
||||
namespace: foxhunt
|
||||
data:
|
||||
default.conf: |
|
||||
# HTTP → HTTPS redirect for all subdomains
|
||||
server {
|
||||
listen 80;
|
||||
server_name *.fxhnt.ai;
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
|
||||
# Gitea — git.fxhnt.ai (replaced GitLab, Phase 2B cutover)
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name git.fxhnt.ai;
|
||||
|
||||
ssl_certificate /etc/nginx/certs/tls.crt;
|
||||
ssl_certificate_key /etc/nginx/certs/tls.key;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
client_max_body_size 0;
|
||||
|
||||
location / {
|
||||
proxy_pass http://gitea-web.foxhunt.svc.cluster.local:3000;
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
|
||||
proxy_buffering off;
|
||||
proxy_request_buffering off;
|
||||
proxy_http_version 1.1;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
proxy_connect_timeout 300s;
|
||||
}
|
||||
}
|
||||
|
||||
# Container Registry — git.fxhnt.ai:5050
|
||||
server {
|
||||
listen 5050 ssl;
|
||||
server_name git.fxhnt.ai;
|
||||
|
||||
ssl_certificate /etc/nginx/certs/tls.crt;
|
||||
ssl_certificate_key /etc/nginx/certs/tls.key;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
client_max_body_size 0;
|
||||
|
||||
location / {
|
||||
proxy_pass http://gitlab-registry.foxhunt.svc.cluster.local:5000;
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
|
||||
proxy_buffering off;
|
||||
proxy_request_buffering off;
|
||||
proxy_http_version 1.1;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
}
|
||||
|
||||
# Grafana — grafana.fxhnt.ai
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name grafana.fxhnt.ai;
|
||||
|
||||
ssl_certificate /etc/nginx/certs/tls.crt;
|
||||
ssl_certificate_key /etc/nginx/certs/tls.key;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
location / {
|
||||
proxy_pass http://grafana.foxhunt.svc.cluster.local:80;
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
}
|
||||
}
|
||||
|
||||
# fxhnt cockpit — dashboard.fxhnt.ai
|
||||
# Proxies to the cockpit's Tailscale node (peer-to-peer over the tailnet) rather than the cluster Service:
|
||||
# the pod CIDR 100.64.0.0/15 overlaps Tailscale CGNAT, so this kernel-mode proxy can't reach platform-pool
|
||||
# pods via ClusterIP, but it CAN reach another tailnet node. IP is stable while the cockpit's TS state
|
||||
# secret (fxhnt-dashboard-ts-state) persists. (Was web-dashboard.foxhunt.svc — replaced by the fxhnt cockpit.)
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name dashboard.fxhnt.ai;
|
||||
|
||||
ssl_certificate /etc/nginx/certs/tls.crt;
|
||||
ssl_certificate_key /etc/nginx/certs/tls.key;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
location / {
|
||||
proxy_pass http://100.81.150.18:80; # fxhnt-dashboard tailnet node (TCP:80 -> cockpit :8080)
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
}
|
||||
}
|
||||
|
||||
# API Gateway (gRPC-Web + gRPC) — api.fxhnt.ai
|
||||
# Browser dashboard uses gRPC-web (HTTP/1.1 POST), so proxy_pass not grpc_pass
|
||||
server {
|
||||
listen 443 ssl;
|
||||
http2 on;
|
||||
server_name api.fxhnt.ai;
|
||||
|
||||
ssl_certificate /etc/nginx/certs/tls.crt;
|
||||
ssl_certificate_key /etc/nginx/certs/tls.key;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
location / {
|
||||
proxy_pass http://api.foxhunt.svc.cluster.local:50051;
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_http_version 1.1;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
proxy_connect_timeout 30s;
|
||||
}
|
||||
}
|
||||
|
||||
# Stalwart Admin — mail.fxhnt.ai
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name mail.fxhnt.ai;
|
||||
|
||||
ssl_certificate /etc/nginx/certs/tls.crt;
|
||||
ssl_certificate_key /etc/nginx/certs/tls.key;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
location / {
|
||||
proxy_pass https://stalwart.foxhunt.svc.cluster.local:443;
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_http_version 1.1;
|
||||
proxy_ssl_verify off;
|
||||
}
|
||||
}
|
||||
|
||||
# Mattermost — chat.fxhnt.ai
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name chat.fxhnt.ai;
|
||||
|
||||
ssl_certificate /etc/nginx/certs/tls.crt;
|
||||
ssl_certificate_key /etc/nginx/certs/tls.key;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
client_max_body_size 50m;
|
||||
|
||||
location / {
|
||||
proxy_pass http://mattermost.foxhunt.svc.cluster.local:8065;
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
}
|
||||
|
||||
# MinIO S3 API — minio.fxhnt.ai
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name minio.fxhnt.ai;
|
||||
|
||||
ssl_certificate /etc/nginx/certs/tls.crt;
|
||||
ssl_certificate_key /etc/nginx/certs/tls.key;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
|
||||
client_max_body_size 0;
|
||||
|
||||
location / {
|
||||
proxy_pass http://minio.foxhunt.svc.cluster.local:9000;
|
||||
proxy_set_header Host $http_host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto https;
|
||||
|
||||
proxy_buffering off;
|
||||
proxy_request_buffering off;
|
||||
proxy_http_version 1.1;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
proxy_connect_timeout 300s;
|
||||
|
||||
# MinIO uses self-signed TLS internally
|
||||
proxy_ssl_verify off;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,206 +0,0 @@
|
||||
# GitLab CE - Foxhunt deployment
|
||||
# Runs on 'infra' node pool alongside platform services
|
||||
# External Postgres + Redis on same pool
|
||||
|
||||
global:
|
||||
edition: ce
|
||||
kas:
|
||||
enabled: false
|
||||
hosts:
|
||||
domain: fxhnt.ai
|
||||
gitlab:
|
||||
name: git.fxhnt.ai
|
||||
https: true
|
||||
registry:
|
||||
name: git.fxhnt.ai
|
||||
https: true
|
||||
ingress:
|
||||
enabled: false
|
||||
configureCertmanager: false
|
||||
shell:
|
||||
port: 2222
|
||||
psql:
|
||||
host: postgres.foxhunt.svc.cluster.local
|
||||
port: 5432
|
||||
database: gitlab
|
||||
username: gitlab
|
||||
password:
|
||||
secret: gitlab-secrets
|
||||
key: db-password
|
||||
redis:
|
||||
host: redis.foxhunt.svc.cluster.local
|
||||
port: 6379
|
||||
auth:
|
||||
enabled: false
|
||||
minio:
|
||||
enabled: false
|
||||
registry:
|
||||
bucket: foxhunt-gitlab-registry
|
||||
appConfig:
|
||||
lfs:
|
||||
bucket: foxhunt-gitlab-artifacts
|
||||
connection:
|
||||
secret: gitlab-s3-credentials
|
||||
key: connection
|
||||
artifacts:
|
||||
bucket: foxhunt-gitlab-artifacts
|
||||
connection:
|
||||
secret: gitlab-s3-credentials
|
||||
key: connection
|
||||
uploads:
|
||||
bucket: foxhunt-gitlab-artifacts
|
||||
connection:
|
||||
secret: gitlab-s3-credentials
|
||||
key: connection
|
||||
packages:
|
||||
bucket: foxhunt-gitlab-artifacts
|
||||
connection:
|
||||
secret: gitlab-s3-credentials
|
||||
key: connection
|
||||
# Terraform/OpenTofu state object storage. MUST be present — without it the
|
||||
# Terraform::StateUploader has no object store and every TF-state API call 403s
|
||||
# ("Object Storage is not enabled for Terraform::StateUploader"). State files already
|
||||
# live in foxhunt-gitlab-artifacts (6b/86/<sha256(project_id)>/<state>/<ver>.tfstate).
|
||||
terraformState:
|
||||
enabled: true
|
||||
bucket: foxhunt-gitlab-artifacts
|
||||
connection:
|
||||
secret: gitlab-s3-credentials
|
||||
key: connection
|
||||
gitlab_kas:
|
||||
enabled: false
|
||||
|
||||
# Disable bundled databases — use external
|
||||
postgresql:
|
||||
install: false
|
||||
redis:
|
||||
install: false
|
||||
|
||||
# Disable components we don't need
|
||||
installCertmanager: false
|
||||
certmanager:
|
||||
installCRDs: false
|
||||
nginx-ingress:
|
||||
enabled: false
|
||||
prometheus:
|
||||
install: true
|
||||
rbac:
|
||||
create: true
|
||||
alertmanager:
|
||||
enabled: false
|
||||
server:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
persistentVolume:
|
||||
enabled: false
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 1Gi
|
||||
gitlab-runner:
|
||||
install: false
|
||||
|
||||
# Registry — S3-backed, NodePort for kubelet image pulls
|
||||
# Token auth disabled (cluster-internal only, secured by network policies)
|
||||
# Pods pull via gitlab-registry.foxhunt.svc.cluster.local:5000 (internal DNS)
|
||||
registry:
|
||||
replicaCount: 1
|
||||
hpa:
|
||||
minReplicas: 1
|
||||
maxReplicas: 1
|
||||
service:
|
||||
type: NodePort
|
||||
nodePort: 30500
|
||||
storage:
|
||||
secret: gitlab-s3-credentials
|
||||
key: registry
|
||||
redirect:
|
||||
disable: true
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
|
||||
# GitLab core components — all on infra node pool
|
||||
gitlab:
|
||||
webservice:
|
||||
replicaCount: 2
|
||||
hpa:
|
||||
minReplicas: 2
|
||||
maxReplicas: 2
|
||||
workerProcesses: 2
|
||||
workhorse:
|
||||
extraArgs: "-apiLimit 0 -apiQueueLimit 0"
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
resources:
|
||||
requests:
|
||||
cpu: 500m
|
||||
memory: 2Gi
|
||||
limits:
|
||||
cpu: 2000m
|
||||
memory: 4Gi
|
||||
extraEnv:
|
||||
REDIS_URL: "redis://redis.foxhunt.svc.cluster.local:6379/8"
|
||||
|
||||
sidekiq:
|
||||
replicas: 1
|
||||
hpa:
|
||||
minReplicas: 1
|
||||
maxReplicas: 1
|
||||
concurrency: 10
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
resources:
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 1Gi
|
||||
limits:
|
||||
cpu: 1000m
|
||||
memory: 2Gi
|
||||
extraEnv:
|
||||
REDIS_URL: "redis://redis.foxhunt.svc.cluster.local:6379/8"
|
||||
|
||||
gitaly:
|
||||
persistence:
|
||||
enabled: true
|
||||
size: 50Gi
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
resources:
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: 1000m
|
||||
memory: 2Gi
|
||||
|
||||
gitlab-shell:
|
||||
replicaCount: 1
|
||||
hpa:
|
||||
minReplicas: 1
|
||||
maxReplicas: 1
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
service:
|
||||
type: NodePort
|
||||
nodePort: 32222
|
||||
|
||||
toolbox:
|
||||
backups:
|
||||
objectStorage:
|
||||
config:
|
||||
secret: gitlab-s3-credentials
|
||||
key: connection
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
|
||||
kas:
|
||||
enabled: false
|
||||
hpa:
|
||||
minReplicas: 0
|
||||
maxReplicas: 0
|
||||
|
||||
gitlab-exporter:
|
||||
enabled: false
|
||||
@@ -1,34 +0,0 @@
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: foxhunt-migrate
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
app.kubernetes.io/component: migration
|
||||
spec:
|
||||
backoffLimit: 3
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
imagePullSecrets:
|
||||
- name: gitlab-registry
|
||||
containers:
|
||||
- name: migrate
|
||||
image: rg.fr-par.scw.cloud/foxhunt/api:latest
|
||||
command: ["sqlx", "migrate", "run"]
|
||||
env:
|
||||
- name: DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: db-credentials
|
||||
key: password
|
||||
- name: DATABASE_URL
|
||||
value: "postgresql://foxhunt:$(DB_PASSWORD)@postgres:5432/foxhunt"
|
||||
resources:
|
||||
requests:
|
||||
memory: "64Mi"
|
||||
cpu: "50m"
|
||||
limits:
|
||||
memory: "256Mi"
|
||||
cpu: "200m"
|
||||
@@ -1,121 +0,0 @@
|
||||
# Mattermost Team Edition — self-hosted Slack alternative for foxhunt.
|
||||
# Runs on platform nodes. PostgreSQL is the shared foxhunt instance.
|
||||
# Access: https://chat.fxhnt.ai (Tailscale only)
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: mattermost-data
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: mattermost
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
strategy:
|
||||
type: Recreate # RWO PVC
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
containers:
|
||||
- name: mattermost
|
||||
image: mattermost/mattermost-team-edition:10.5
|
||||
ports:
|
||||
- containerPort: 8065
|
||||
name: http
|
||||
env:
|
||||
- name: DB_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: db-credentials
|
||||
key: password
|
||||
- name: MM_SQLSETTINGS_DRIVERNAME
|
||||
value: postgres
|
||||
- name: MM_SQLSETTINGS_DATASOURCE
|
||||
value: "postgres://foxhunt:$(DB_PASSWORD)@postgres.foxhunt.svc.cluster.local:5432/mattermost?sslmode=disable"
|
||||
- name: MM_SERVICESETTINGS_SITEURL
|
||||
value: "https://chat.fxhnt.ai"
|
||||
- name: MM_SERVICESETTINGS_LISTENADDRESS
|
||||
value: ":8065"
|
||||
- name: MM_SERVICESETTINGS_ENABLEINCOMINGWEBHOOKS
|
||||
value: "true"
|
||||
- name: MM_SERVICESETTINGS_ENABLEPOSTUSERNAMEOVERRIDE
|
||||
value: "true"
|
||||
- name: MM_SERVICESETTINGS_ENABLEPOSTICONOVERRIDE
|
||||
value: "true"
|
||||
- name: MM_LOGSETTINGS_ENABLECONSOLE
|
||||
value: "true"
|
||||
- name: MM_LOGSETTINGS_CONSOLELEVEL
|
||||
value: INFO
|
||||
- name: MM_LOGSETTINGS_ENABLEFILE
|
||||
value: "false"
|
||||
- name: MM_FILESETTINGS_DRIVERNAME
|
||||
value: local
|
||||
- name: MM_FILESETTINGS_DIRECTORY
|
||||
value: /mattermost/data
|
||||
# Email via Stalwart — uncomment after Stalwart is deployed:
|
||||
# - name: MM_EMAILSETTINGS_SMTPSERVER
|
||||
# value: stalwart.foxhunt.svc.cluster.local
|
||||
# - name: MM_EMAILSETTINGS_SMTPPORT
|
||||
# value: "25"
|
||||
# - name: MM_EMAILSETTINGS_SENDEMAILNOTIFICATIONS
|
||||
# value: "true"
|
||||
volumeMounts:
|
||||
- name: mattermost-data
|
||||
mountPath: /mattermost/data
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /api/v4/system/ping
|
||||
port: 8065
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 1Gi
|
||||
volumes:
|
||||
- name: mattermost-data
|
||||
persistentVolumeClaim:
|
||||
claimName: mattermost-data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: mattermost
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: mattermost
|
||||
ports:
|
||||
- port: 8065
|
||||
targetPort: 8065
|
||||
name: http
|
||||
@@ -1,4 +0,0 @@
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- minio.yaml
|
||||
@@ -1,188 +0,0 @@
|
||||
# MinIO — in-cluster S3 for compiled binaries and trained models
|
||||
# Replaces Scaleway public S3 to keep proprietary code/IP in-cluster
|
||||
# Plain HTTP only — internal cluster traffic, no TLS needed
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: minio-credentials
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: minio
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
type: Opaque
|
||||
stringData:
|
||||
root-user: foxhunt-minio
|
||||
root-password: CHANGE_ME_BEFORE_DEPLOY
|
||||
# rclone-compatible keys (same values, different key names for initContainers)
|
||||
access-key: foxhunt-minio
|
||||
secret-key: CHANGE_ME_BEFORE_DEPLOY
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: minio-data-new
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: minio
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 150Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: minio
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: minio
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate # single-node MinIO — no rolling update
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: minio
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
containers:
|
||||
- name: minio
|
||||
image: minio/minio:latest
|
||||
args: ["server", "/data", "--console-address", ":9001"]
|
||||
ports:
|
||||
- containerPort: 9000
|
||||
name: api
|
||||
- containerPort: 9001
|
||||
name: console
|
||||
env:
|
||||
- name: MINIO_ROOT_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: minio-credentials
|
||||
key: root-user
|
||||
- name: MINIO_ROOT_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: minio-credentials
|
||||
key: root-password
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /minio/health/live
|
||||
port: 9000
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /minio/health/live
|
||||
port: 9000
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 15
|
||||
failureThreshold: 5
|
||||
resources:
|
||||
requests:
|
||||
cpu: 200m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: "1"
|
||||
memory: 2Gi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: minio-data-new
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: minio
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: minio
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: minio
|
||||
ports:
|
||||
- port: 9000
|
||||
targetPort: 9000
|
||||
name: api
|
||||
- port: 9001
|
||||
targetPort: 9001
|
||||
name: console
|
||||
---
|
||||
# Create buckets on first deploy
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: minio-init-buckets
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: minio
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
backoffLimit: 5
|
||||
ttlSecondsAfterFinished: 300
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: minio-init
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
restartPolicy: OnFailure
|
||||
containers:
|
||||
- name: mc
|
||||
image: minio/mc:latest
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -e
|
||||
echo "Waiting for MinIO to be ready..."
|
||||
until mc alias set foxhunt http://minio.foxhunt.svc.cluster.local:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" 2>/dev/null; do
|
||||
echo "MinIO not ready, retrying in 3s..."
|
||||
sleep 3
|
||||
done
|
||||
echo "Creating buckets..."
|
||||
mc mb --ignore-existing foxhunt/foxhunt-binaries
|
||||
mc mb --ignore-existing foxhunt/foxhunt-models
|
||||
mc mb --ignore-existing foxhunt/foxhunt-training-results
|
||||
mc mb --ignore-existing foxhunt/argo-logs
|
||||
mc mb --ignore-existing foxhunt/foxhunt-sccache
|
||||
mc mb --ignore-existing foxhunt/foxhunt-training-data
|
||||
# Plan 5 Task 3 (A.4.1): nsys profile artefacts bucket.
|
||||
# Stores .nsys-rep files keyed by commit SHA under
|
||||
# profiles/<short-sha>/profile-seed*-fold*-*.nsys-rep.
|
||||
mc mb --ignore-existing foxhunt/foxhunt-training-artifacts
|
||||
echo "Buckets created:"
|
||||
mc ls foxhunt/
|
||||
env:
|
||||
- name: MINIO_ROOT_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: minio-credentials
|
||||
key: root-user
|
||||
- name: MINIO_ROOT_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: minio-credentials
|
||||
key: root-password
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 64Mi
|
||||
@@ -1,104 +0,0 @@
|
||||
# Cockpit + NAT Gateway — Apply Order
|
||||
|
||||
## Prerequisites
|
||||
- `scw` CLI authenticated
|
||||
- `terragrunt` installed
|
||||
- `kubectl` access to foxhunt cluster
|
||||
- `helm` installed
|
||||
|
||||
## Phase 1: Cockpit (metrics backend)
|
||||
|
||||
1. Apply Cockpit Terraform module:
|
||||
```bash
|
||||
cd infra/live/production/cockpit
|
||||
terragrunt apply
|
||||
```
|
||||
|
||||
2. Note the outputs:
|
||||
```bash
|
||||
terragrunt output metrics_push_url
|
||||
terragrunt output -raw push_token
|
||||
terragrunt output grafana_url
|
||||
terragrunt output -raw grafana_password
|
||||
```
|
||||
|
||||
3. Create the K8s secret for the push token:
|
||||
```bash
|
||||
kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -
|
||||
kubectl -n monitoring create secret generic cockpit-push-token \
|
||||
--from-literal=token=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
|
||||
```
|
||||
|
||||
4. Update placeholders in `infra/k8s/monitoring/alloy-values.yaml`:
|
||||
- Replace `COCKPIT_METRICS_PUSH_URL` with the `metrics_push_url` output
|
||||
|
||||
5. Deploy DCGM Exporter:
|
||||
```bash
|
||||
kubectl apply -f infra/k8s/monitoring/dcgm-exporter.yaml
|
||||
```
|
||||
|
||||
6. Deploy Grafana Alloy (k8s-monitoring):
|
||||
```bash
|
||||
helm repo add grafana https://grafana.github.io/helm-charts
|
||||
helm repo update
|
||||
helm install k8s-monitoring grafana/k8s-monitoring \
|
||||
--namespace monitoring \
|
||||
-f infra/k8s/monitoring/alloy-values.yaml
|
||||
```
|
||||
|
||||
7. Verify metrics flow:
|
||||
- Open Cockpit Grafana (URL from step 2)
|
||||
- Login with foxhunt-admin / (password from step 2)
|
||||
- Check "Kubernetes / Compute Resources / Cluster" dashboard
|
||||
|
||||
## Phase 2: GitLab Prometheus remote_write
|
||||
|
||||
1. Update placeholders in `infra/k8s/gitlab/values.yaml`:
|
||||
- Replace `COCKPIT_METRICS_PUSH_URL` and `COCKPIT_PUSH_TOKEN`
|
||||
|
||||
2. Upgrade GitLab Helm release:
|
||||
```bash
|
||||
helm upgrade gitlab gitlab/gitlab \
|
||||
--namespace foxhunt \
|
||||
-f infra/k8s/gitlab/values.yaml
|
||||
```
|
||||
|
||||
## Phase 3: NAT Gateway
|
||||
|
||||
1. Apply Public Gateway module:
|
||||
```bash
|
||||
cd infra/live/production/public-gateway
|
||||
terragrunt apply
|
||||
```
|
||||
This creates the gateway and attaches it to the private network.
|
||||
Existing nodes still have public IPs — nothing breaks yet.
|
||||
|
||||
2. Apply Kapsule pool changes (disables public IPs):
|
||||
```bash
|
||||
cd infra/live/production/kapsule
|
||||
terragrunt apply
|
||||
```
|
||||
**WARNING**: This may recreate nodes. For zero-downtime:
|
||||
|
||||
3. Zero-downtime node migration for services pool:
|
||||
```bash
|
||||
# Temporarily allow 2 nodes
|
||||
scw k8s pool update <services-pool-id> max-size=2 size=2
|
||||
# Wait for new node (private IP) to be Ready
|
||||
kubectl get nodes -w
|
||||
# Drain old node
|
||||
kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
|
||||
# Scale back
|
||||
scw k8s pool update <services-pool-id> max-size=1 size=1
|
||||
```
|
||||
|
||||
4. Repeat for gitlab pool (if not autoscaling, same process).
|
||||
|
||||
5. Autoscaling pools (ci-training, gpu-dev):
|
||||
- No action needed — next scale-up automatically uses private IP.
|
||||
|
||||
6. Verify Tailscale still works:
|
||||
```bash
|
||||
tailscale status # should show foxhunt-kapsule and foxhunt-gitlab nodes
|
||||
curl -k https://git.fxhnt.ai # should still resolve via Tailscale
|
||||
```
|
||||
@@ -1,88 +0,0 @@
|
||||
# Grafana k8s-monitoring Helm chart values (v1.x format)
|
||||
# Chart: grafana/k8s-monitoring --version 1.6.50
|
||||
# Deploys: Alloy (DaemonSet) + node-exporter + kube-state-metrics
|
||||
#
|
||||
# Install:
|
||||
# helm repo add grafana https://grafana.github.io/helm-charts
|
||||
# PUSH_TOKEN=$(cd infra/live/production/cockpit && terragrunt output -raw push_token)
|
||||
# helm install k8s-monitoring grafana/k8s-monitoring \
|
||||
# --namespace monitoring --version 1.6.50 \
|
||||
# -f infra/k8s/monitoring/alloy-values.yaml \
|
||||
# --set "externalServices.prometheus.extraHeaders.X-Token=${PUSH_TOKEN}"
|
||||
|
||||
cluster:
|
||||
name: foxhunt
|
||||
|
||||
externalServices:
|
||||
prometheus:
|
||||
host: "https://d8cd4c5d-6d3b-4258-9f92-2ad0eaa36ba0.metrics.cockpit.fr-par.scw.cloud"
|
||||
writeEndpoint: "/api/v1/push"
|
||||
authMode: none
|
||||
# X-Token header passed via --set to keep secret out of git
|
||||
# Loki host required by chart validation even with logs disabled
|
||||
loki:
|
||||
host: "https://noop.localhost"
|
||||
authMode: none
|
||||
|
||||
metrics:
|
||||
enabled: true
|
||||
cost:
|
||||
enabled: false
|
||||
node-exporter:
|
||||
enabled: true
|
||||
kube-state-metrics:
|
||||
enabled: true
|
||||
kubelet:
|
||||
enabled: true
|
||||
cadvisor:
|
||||
enabled: true
|
||||
apiserver:
|
||||
enabled: true
|
||||
|
||||
# Alloy DaemonSet — runs on every node
|
||||
alloy-metrics:
|
||||
alloy:
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 256Mi
|
||||
|
||||
# node-exporter — host metrics (CPU, memory, disk, network)
|
||||
prometheus-node-exporter:
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
|
||||
# kube-state-metrics — K8s object state
|
||||
kube-state-metrics:
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
# Scheduled on gitlab node (GP1-XS 16GB) — services (DEV1-M 4GB) too small
|
||||
|
||||
# Disable components we don't need
|
||||
logs:
|
||||
enabled: false
|
||||
cluster_events:
|
||||
enabled: false
|
||||
traces:
|
||||
enabled: false
|
||||
opencost:
|
||||
enabled: false
|
||||
|
||||
# Disable log/event collectors
|
||||
alloy-logs:
|
||||
enabled: false
|
||||
alloy-events:
|
||||
enabled: false
|
||||
@@ -1,809 +0,0 @@
|
||||
{
|
||||
"description": "Loki-powered CI pipeline log viewer \u2014 test gate, clippy, builds, errors",
|
||||
"editable": true,
|
||||
"graphTooltip": 2,
|
||||
"panels": [
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"title": "Pipeline Overview",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"fixedColor": "red",
|
||||
"mode": "fixed"
|
||||
},
|
||||
"mappings": [],
|
||||
"noValue": "0",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 0,
|
||||
"y": 1
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"sum"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "auto"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=\"ERROR\"} [$__range]))",
|
||||
"queryType": "instant",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Errors",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"fixedColor": "yellow",
|
||||
"mode": "fixed"
|
||||
},
|
||||
"mappings": [],
|
||||
"noValue": "0",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 4,
|
||||
"y": 1
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"sum"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "auto"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=\"WARN\"} [$__range]))",
|
||||
"queryType": "instant",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Warnings",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"fixedColor": "blue",
|
||||
"mode": "fixed"
|
||||
},
|
||||
"mappings": [],
|
||||
"noValue": "0"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 8,
|
||||
"y": 1
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"sum"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "auto"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__range]))",
|
||||
"queryType": "instant",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Total Log Lines",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisLabel": "lines/interval",
|
||||
"drawStyle": "bars",
|
||||
"fillOpacity": 80,
|
||||
"lineWidth": 0,
|
||||
"pointSize": 5,
|
||||
"showPoints": "never",
|
||||
"stacking": {
|
||||
"mode": "normal"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "ERROR"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"fixedColor": "red",
|
||||
"mode": "fixed"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "WARN"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"fixedColor": "yellow",
|
||||
"mode": "fixed"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "INFO"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"fixedColor": "green",
|
||||
"mode": "fixed"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "DEBUG"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"fixedColor": "blue",
|
||||
"mode": "fixed"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 6,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 1
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (level) (count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__interval]))",
|
||||
"legendFormat": "{{level}}",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Log Volume by Level",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"drawStyle": "bars",
|
||||
"fillOpacity": 60,
|
||||
"lineWidth": 0,
|
||||
"showPoints": "never",
|
||||
"stacking": {
|
||||
"mode": "normal"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 5
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "right"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (container) (count_over_time({namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} [$__interval]))",
|
||||
"legendFormat": "{{container}}",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Log Volume by Container",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 9
|
||||
},
|
||||
"id": 10,
|
||||
"title": "Full Pipeline Log Stream",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 14,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 10
|
||||
},
|
||||
"id": 11,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"$pipeline-.*\", level=~\"$level\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg ${search:pipe}",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Pipeline Logs \u2014 $pipeline",
|
||||
"type": "logs"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 24
|
||||
},
|
||||
"id": 20,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 25
|
||||
},
|
||||
"id": 21,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": false,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"ci-pipeline-.*\", container=\"main\"} |~ \"(clippy|cargo test|warning\\\\[|error\\\\[|FAILED|test result|running \\\\d+ test|failures:)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Clippy + Test Output",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"title": "Test Gate (clippy + cargo test)",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 25
|
||||
},
|
||||
"id": 30,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 26
|
||||
},
|
||||
"id": 31,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": false,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"ci-pipeline-.*\", container=\"redis\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Redis Sidecar Logs",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"title": "Redis Sidecar",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 26
|
||||
},
|
||||
"id": 40,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 27
|
||||
},
|
||||
"id": 41,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"$pipeline-.*\"} |~ \"(?i)(error|failed|failure|panic|fatal|SIGABRT|SIGSEGV|thread .* panicked)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "All Errors & Failures",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"title": "Errors & Failures",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 27
|
||||
},
|
||||
"id": 50,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 28
|
||||
},
|
||||
"id": 51,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": false,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"(ci-pipeline|compile-and-deploy|compile-and-train)-.*\", container=\"main\"} |~ \"(Compiling|Finished|warning|error|Downloading|Building|cargo build|rclone)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Compile Logs",
|
||||
"type": "logs"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 28
|
||||
},
|
||||
"id": 52,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": false,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"(ci-pipeline|compile-and-deploy)-.*\", container=\"main\"} |~ \"(kubectl|apply|rollout|deploy|configured|unchanged|created|image|kaniko)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Deploy & Argo Template Logs",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"title": "Build & Deploy",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 28
|
||||
},
|
||||
"id": 60,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 29
|
||||
},
|
||||
"id": 61,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"gpu-test-.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "GPU Test Logs (DQN/PPO/TFT + Supervised)",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"title": "GPU Test Pipeline",
|
||||
"type": "row"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 39,
|
||||
"tags": [
|
||||
"foxhunt",
|
||||
"ci",
|
||||
"loki",
|
||||
"logs"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": true,
|
||||
"text": "ci-pipeline",
|
||||
"value": "ci-pipeline"
|
||||
},
|
||||
"label": "Pipeline",
|
||||
"name": "pipeline",
|
||||
"options": [],
|
||||
"query": "ci-pipeline,compile-and-deploy,compile-and-train,gpu-test,gpu-diag,gpu-walltime,build-ci-image,build-training-runtime,evaluate",
|
||||
"type": "custom"
|
||||
},
|
||||
{
|
||||
"allValue": ".*",
|
||||
"current": {
|
||||
"selected": true,
|
||||
"text": "All",
|
||||
"value": "$__all"
|
||||
},
|
||||
"includeAll": true,
|
||||
"label": "Level",
|
||||
"multi": true,
|
||||
"name": "level",
|
||||
"options": [
|
||||
{
|
||||
"selected": true,
|
||||
"text": "All",
|
||||
"value": "$__all"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "ERROR",
|
||||
"value": "ERROR"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "WARN",
|
||||
"value": "WARN"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "INFO",
|
||||
"value": "INFO"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "DEBUG",
|
||||
"value": "DEBUG"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "TRACE",
|
||||
"value": "TRACE"
|
||||
}
|
||||
],
|
||||
"query": "ERROR,WARN,INFO,DEBUG,TRACE",
|
||||
"type": "custom"
|
||||
},
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "",
|
||||
"value": ""
|
||||
},
|
||||
"label": "Search",
|
||||
"name": "search",
|
||||
"type": "textbox"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-6h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "CI Pipeline Logs",
|
||||
"uid": "foxhunt-ci-logs",
|
||||
"__inputs": [
|
||||
{
|
||||
"name": "DS_PROMETHEUS",
|
||||
"type": "datasource",
|
||||
"pluginId": "prometheus"
|
||||
},
|
||||
{
|
||||
"name": "DS_LOKI",
|
||||
"type": "datasource",
|
||||
"pluginId": "loki"
|
||||
},
|
||||
{
|
||||
"name": "DS_TEMPO",
|
||||
"type": "datasource",
|
||||
"pluginId": "${DS_TEMPO}"
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,835 +0,0 @@
|
||||
{
|
||||
"description": "Loki-powered training log viewer \u2014 hyperopt trials, epoch progress, GPU status, errors",
|
||||
"editable": true,
|
||||
"graphTooltip": 2,
|
||||
"panels": [
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"title": "Training Overview",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"fixedColor": "red",
|
||||
"mode": "fixed"
|
||||
},
|
||||
"mappings": [],
|
||||
"noValue": "0",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 0,
|
||||
"y": 1
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"sum"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "auto"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\", level=\"ERROR\"} [$__range]))",
|
||||
"queryType": "instant",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Errors",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"fixedColor": "yellow",
|
||||
"mode": "fixed"
|
||||
},
|
||||
"mappings": [],
|
||||
"noValue": "0",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 4,
|
||||
"y": 1
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"sum"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "auto"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\", level=\"WARN\"} [$__range]))",
|
||||
"queryType": "instant",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Warnings",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"fixedColor": "blue",
|
||||
"mode": "fixed"
|
||||
},
|
||||
"mappings": [],
|
||||
"noValue": "0"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 4,
|
||||
"x": 8,
|
||||
"y": 1
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"colorMode": "background",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"sum"
|
||||
],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"textMode": "auto"
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum(count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__range]))",
|
||||
"queryType": "instant",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Total Log Lines",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisLabel": "lines/interval",
|
||||
"drawStyle": "bars",
|
||||
"fillOpacity": 80,
|
||||
"lineWidth": 0,
|
||||
"showPoints": "never",
|
||||
"stacking": {
|
||||
"mode": "normal"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": [
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "ERROR"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"fixedColor": "red",
|
||||
"mode": "fixed"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "WARN"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"fixedColor": "yellow",
|
||||
"mode": "fixed"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "INFO"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "color",
|
||||
"value": {
|
||||
"fixedColor": "green",
|
||||
"mode": "fixed"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 6,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 1
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (level) (count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__interval]))",
|
||||
"legendFormat": "{{level}}",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Log Volume by Level",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"drawStyle": "bars",
|
||||
"fillOpacity": 60,
|
||||
"lineWidth": 0,
|
||||
"showPoints": "never",
|
||||
"stacking": {
|
||||
"mode": "normal"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 5
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "right"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "sum by (pod) (count_over_time({namespace=\"foxhunt\", pod=~\"$job-.*\"} [$__interval]))",
|
||||
"legendFormat": "{{pod}}",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Log Volume by Pod",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 9
|
||||
},
|
||||
"id": 10,
|
||||
"title": "Full Training Log Stream",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 14,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 10
|
||||
},
|
||||
"id": 11,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\", level=~\"$level\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg ${search:pipe}",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Training Logs \u2014 $job",
|
||||
"type": "logs"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 24
|
||||
},
|
||||
"id": 20,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 25
|
||||
},
|
||||
"id": 21,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(epoch|loss|reward|sharpe|accuracy|val_loss|train_loss|best_trial|trial|fold|window|learning_rate|lr=|MaxDD|win.?rate|drawdown)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Epoch / Loss / Reward / Sharpe",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"title": "Epoch Progress & Loss",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 25
|
||||
},
|
||||
"id": 30,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 26
|
||||
},
|
||||
"id": 31,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"(hyperopt|train)-.*\"} |~ \"(?i)(trial|best|cost|param|learning_rate|hidden_dim|batch_size|buffer_size|gamma|epsilon|clip|PSO|iteration|objective|optimal)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Hyperopt Trial Results",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"title": "Hyperopt Trials",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 26
|
||||
},
|
||||
"id": 40,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 27
|
||||
},
|
||||
"id": 41,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(GPU|CUDA|VRAM|cuda|gpu|vram|OOM|out.of.memory|nvrtc|tensor.core|BF16|bf16|SM|occupancy|kernel|device|nvidia)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "GPU / CUDA / VRAM Logs",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"title": "GPU / CUDA / VRAM",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 27
|
||||
},
|
||||
"id": 50,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 28
|
||||
},
|
||||
"id": 51,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(walk.?forward|fold|backtest|window|train_start|train_end|val_start|val_end|test_start|test_end|embargo|purge|bars|features)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Walk-Forward / Fold / Backtest",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"title": "Walk-Forward Windows",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 28
|
||||
},
|
||||
"id": 60,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 29
|
||||
},
|
||||
"id": 61,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(error|failed|failure|panic|fatal|SIGABRT|SIGSEGV|thread .* panicked|OOM|NaN|inf|diverge)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "All Training Errors & Failures",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"title": "Errors & Failures",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"collapsed": true,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 29
|
||||
},
|
||||
"id": 70,
|
||||
"panels": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 30
|
||||
},
|
||||
"id": 71,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"editorMode": "code",
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"$job-.*\"} |~ \"(?i)(loading|loaded|dbn|ofi|features|bars|MBP.?10|OHLCV|symbol|normalize|norm_stats|state_dim|rclone|S3|download)\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"queryType": "range",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Data Loading / OFI / DBN / Features",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"title": "Data Loading & OFI",
|
||||
"type": "row"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 39,
|
||||
"tags": [
|
||||
"foxhunt",
|
||||
"training",
|
||||
"loki",
|
||||
"logs",
|
||||
"gpu"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"current": {
|
||||
"selected": true,
|
||||
"text": "All Training",
|
||||
"value": "(train|hyperopt|compile-and-train)"
|
||||
},
|
||||
"label": "Job Type",
|
||||
"name": "job",
|
||||
"options": [
|
||||
{
|
||||
"selected": true,
|
||||
"text": "All Training",
|
||||
"value": "(train|hyperopt|compile-and-train)"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "DQN",
|
||||
"value": "train-dqn"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "PPO",
|
||||
"value": "train-ppo"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "Supervised",
|
||||
"value": "train-supervised"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "Hyperopt",
|
||||
"value": "hyperopt"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "Compile+Train",
|
||||
"value": "compile-and-train"
|
||||
}
|
||||
],
|
||||
"query": "All Training : (train|hyperopt|compile-and-train), DQN : train-dqn, PPO : train-ppo, Supervised : train-supervised, Hyperopt : hyperopt, Compile+Train : compile-and-train",
|
||||
"type": "custom"
|
||||
},
|
||||
{
|
||||
"allValue": ".*",
|
||||
"current": {
|
||||
"selected": true,
|
||||
"text": "All",
|
||||
"value": "$__all"
|
||||
},
|
||||
"includeAll": true,
|
||||
"label": "Level",
|
||||
"multi": true,
|
||||
"name": "level",
|
||||
"options": [
|
||||
{
|
||||
"selected": true,
|
||||
"text": "All",
|
||||
"value": "$__all"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "ERROR",
|
||||
"value": "ERROR"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "WARN",
|
||||
"value": "WARN"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "INFO",
|
||||
"value": "INFO"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "DEBUG",
|
||||
"value": "DEBUG"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "TRACE",
|
||||
"value": "TRACE"
|
||||
}
|
||||
],
|
||||
"query": "ERROR,WARN,INFO,DEBUG,TRACE",
|
||||
"type": "custom"
|
||||
},
|
||||
{
|
||||
"current": {
|
||||
"selected": false,
|
||||
"text": "",
|
||||
"value": ""
|
||||
},
|
||||
"label": "Search",
|
||||
"name": "search",
|
||||
"type": "textbox"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-12h",
|
||||
"to": "now"
|
||||
},
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "Training Pipeline Logs",
|
||||
"uid": "foxhunt-training-logs",
|
||||
"__inputs": [
|
||||
{
|
||||
"name": "DS_PROMETHEUS",
|
||||
"type": "datasource",
|
||||
"pluginId": "prometheus"
|
||||
},
|
||||
{
|
||||
"name": "DS_LOKI",
|
||||
"type": "datasource",
|
||||
"pluginId": "loki"
|
||||
},
|
||||
{
|
||||
"name": "DS_TEMPO",
|
||||
"type": "datasource",
|
||||
"pluginId": "${DS_TEMPO}"
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,212 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Import all Grafana dashboards via API (Postgres-backed, no ConfigMaps).
|
||||
#
|
||||
# Dashboards use ${DS_PROMETHEUS}, ${DS_LOKI}, ${DS_TEMPO} variables.
|
||||
# This script auto-detects datasource UIDs, resolves variables, and
|
||||
# imports via the Grafana API. Dashboards are stored in Postgres.
|
||||
#
|
||||
# Required environment variables:
|
||||
# GRAFANA_PASS - Password for admin user
|
||||
#
|
||||
# Optional overrides:
|
||||
# GRAFANA_URL - Grafana base URL (default: https://grafana.fxhnt.ai)
|
||||
# GRAFANA_USER - Grafana user (default: admin)
|
||||
# GRAFANA_PROM_UID - Override Prometheus datasource UID
|
||||
# GRAFANA_LOKI_UID - Override Loki datasource UID
|
||||
# GRAFANA_TEMPO_UID - Override Tempo datasource UID
|
||||
#
|
||||
# Usage:
|
||||
# GRAFANA_PASS="xxx" ./import.sh
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
GRAFANA_USER="${GRAFANA_USER:-admin}"
|
||||
GRAFANA_URL="${GRAFANA_URL:-https://grafana.fxhnt.ai}"
|
||||
GRAFANA_URL="${GRAFANA_URL%/}"
|
||||
|
||||
if [[ -z "${GRAFANA_PASS:-}" ]]; then
|
||||
echo "ERROR: GRAFANA_PASS is not set" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- Datasource auto-detection ---
|
||||
|
||||
detect_datasource_uid() {
|
||||
local ds_type="$1"
|
||||
curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" "${GRAFANA_URL}/api/datasources" 2>/dev/null | \
|
||||
python3 -c "
|
||||
import sys, json
|
||||
for ds in json.load(sys.stdin):
|
||||
if ds.get('type') == '${ds_type}':
|
||||
print(ds['uid'])
|
||||
break
|
||||
" 2>/dev/null || true
|
||||
}
|
||||
|
||||
echo "Detecting datasource UIDs..."
|
||||
PROM_UID="${GRAFANA_PROM_UID:-$(detect_datasource_uid prometheus)}"
|
||||
LOKI_UID="${GRAFANA_LOKI_UID:-$(detect_datasource_uid loki)}"
|
||||
TEMPO_UID="${GRAFANA_TEMPO_UID:-$(detect_datasource_uid tempo)}"
|
||||
|
||||
echo " Prometheus: ${PROM_UID:-NOT FOUND}"
|
||||
echo " Loki: ${LOKI_UID:-NOT FOUND}"
|
||||
echo " Tempo: ${TEMPO_UID:-NOT FOUND}"
|
||||
|
||||
if [[ -z "$PROM_UID" ]]; then
|
||||
echo "ERROR: Could not detect Prometheus datasource. Set GRAFANA_PROM_UID." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- Resolve datasource variables ---
|
||||
|
||||
resolve_dashboard() {
|
||||
local file="$1"
|
||||
python3 -c "
|
||||
import sys, json
|
||||
|
||||
with open('${file}') as f:
|
||||
d = json.load(f)
|
||||
|
||||
d.pop('__inputs', None)
|
||||
d.pop('__requires', None)
|
||||
|
||||
raw = json.dumps(d)
|
||||
raw = raw.replace('\${DS_PROMETHEUS}', '${PROM_UID}')
|
||||
raw = raw.replace('\${DS_LOKI}', '${LOKI_UID}')
|
||||
raw = raw.replace('\${DS_TEMPO}', '${TEMPO_UID}')
|
||||
|
||||
print(raw)
|
||||
"
|
||||
}
|
||||
|
||||
# --- Ensure folders exist ---
|
||||
|
||||
ensure_folder() {
|
||||
local title="$1"
|
||||
local uid
|
||||
uid=$(curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" "${GRAFANA_URL}/api/folders" 2>/dev/null | \
|
||||
python3 -c "
|
||||
import sys, json
|
||||
for f in json.load(sys.stdin):
|
||||
if f['title'] == '${title}':
|
||||
print(f['uid'])
|
||||
break
|
||||
" 2>/dev/null || true)
|
||||
|
||||
if [[ -z "$uid" ]]; then
|
||||
uid=$(curl -sf -u "${GRAFANA_USER}:${GRAFANA_PASS}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"title\": \"${title}\"}" \
|
||||
"${GRAFANA_URL}/api/folders" 2>/dev/null | \
|
||||
python3 -c "import sys,json; print(json.load(sys.stdin)['uid'])" 2>/dev/null)
|
||||
echo " Created folder: ${title} (${uid})"
|
||||
fi
|
||||
echo "$uid"
|
||||
}
|
||||
|
||||
echo ""
|
||||
echo "Ensuring folders..."
|
||||
FOLDER_CI=$(ensure_folder "CI")
|
||||
FOLDER_INFRA=$(ensure_folder "Infrastructure")
|
||||
FOLDER_OPS=$(ensure_folder "Operations")
|
||||
FOLDER_TRADING=$(ensure_folder "Trading")
|
||||
FOLDER_TRAINING=$(ensure_folder "Training")
|
||||
FOLDER_SERVICES=$(ensure_folder "Services")
|
||||
FOLDER_PLATFORM=$(ensure_folder "Platform")
|
||||
|
||||
# --- Import a single dashboard to a folder ---
|
||||
|
||||
import_dashboard() {
|
||||
local file="$1"
|
||||
local folder_uid="$2"
|
||||
local name
|
||||
name="$(basename "$file" .json)"
|
||||
|
||||
local resolved
|
||||
resolved=$(resolve_dashboard "$file")
|
||||
|
||||
local response
|
||||
response=$(curl -s -w "\n%{http_code}" \
|
||||
-u "${GRAFANA_USER}:${GRAFANA_PASS}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"dashboard\": ${resolved}, \"folderUid\": \"${folder_uid}\", \"overwrite\": true}" \
|
||||
"${GRAFANA_URL}/api/dashboards/db")
|
||||
|
||||
local http_code
|
||||
http_code=$(echo "$response" | tail -1)
|
||||
|
||||
if [[ "$http_code" -ge 200 && "$http_code" -lt 300 ]]; then
|
||||
echo " OK ${name}"
|
||||
else
|
||||
local body
|
||||
body=$(echo "$response" | sed '$d')
|
||||
echo " FAIL ${name} (${http_code}): ${body}" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Dashboard → Folder mapping ---
|
||||
|
||||
declare -A DASHBOARD_FOLDERS=(
|
||||
# Original dashboards
|
||||
[foxhunt-cockpit]="${FOLDER_OPS}"
|
||||
[foxhunt-trading]="${FOLDER_TRADING}"
|
||||
[foxhunt-training]="${FOLDER_TRAINING}"
|
||||
[foxhunt-infrastructure]="${FOLDER_INFRA}"
|
||||
[foxhunt-observability]="${FOLDER_INFRA}"
|
||||
[foxhunt-ci-logs]="${FOLDER_CI}"
|
||||
[foxhunt-training-logs]="${FOLDER_TRAINING}"
|
||||
[foxhunt-training-deep-dive]="${FOLDER_TRAINING}"
|
||||
# Application services
|
||||
[svc-api-gateway]="${FOLDER_SERVICES}"
|
||||
[svc-trading-service]="${FOLDER_SERVICES}"
|
||||
[svc-backtesting]="${FOLDER_SERVICES}"
|
||||
[svc-data-acquisition]="${FOLDER_SERVICES}"
|
||||
[svc-ml-training]="${FOLDER_SERVICES}"
|
||||
[svc-trading-agent]="${FOLDER_SERVICES}"
|
||||
[svc-web-gateway]="${FOLDER_SERVICES}"
|
||||
# Platform services
|
||||
[platform-prometheus]="${FOLDER_PLATFORM}"
|
||||
[platform-grafana]="${FOLDER_PLATFORM}"
|
||||
[platform-loki]="${FOLDER_PLATFORM}"
|
||||
[platform-tempo]="${FOLDER_PLATFORM}"
|
||||
[platform-redis]="${FOLDER_PLATFORM}"
|
||||
[platform-postgres]="${FOLDER_PLATFORM}"
|
||||
[platform-argo]="${FOLDER_PLATFORM}"
|
||||
[platform-questdb]="${FOLDER_PLATFORM}"
|
||||
[platform-costs]="${FOLDER_PLATFORM}"
|
||||
[platform-scaleway]="${FOLDER_PLATFORM}"
|
||||
)
|
||||
|
||||
# --- Deploy all dashboards ---
|
||||
|
||||
echo ""
|
||||
echo "Importing dashboards..."
|
||||
|
||||
failed=0
|
||||
total=0
|
||||
|
||||
for dashboard in "${SCRIPT_DIR}"/*.json; do
|
||||
[[ -f "$dashboard" ]] || continue
|
||||
name="$(basename "$dashboard" .json)"
|
||||
folder_uid="${DASHBOARD_FOLDERS[$name]:-}"
|
||||
|
||||
if [[ -z "$folder_uid" ]]; then
|
||||
echo " SKIP ${name} (no folder mapping)"
|
||||
continue
|
||||
fi
|
||||
|
||||
total=$((total + 1))
|
||||
if ! import_dashboard "$dashboard" "$folder_uid"; then
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
if [[ $failed -eq 0 ]]; then
|
||||
echo "All ${total} dashboards deployed successfully."
|
||||
else
|
||||
echo "ERROR: ${failed}/${total} dashboard(s) failed." >&2
|
||||
exit 1
|
||||
fi
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,802 +0,0 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"panels": [],
|
||||
"title": "Grafana Health",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 0,
|
||||
"y": 1
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"grafana.*\", phase=\"Running\"}",
|
||||
"instant": true,
|
||||
"legendFormat": "Running",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Pod Status",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 3
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 3,
|
||||
"y": 1
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"grafana.*\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Restarts",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 0.7
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.9
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 6,
|
||||
"y": 1
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[$interval]))",
|
||||
"legendFormat": "",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Usage",
|
||||
"type": "gauge"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 0.7
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.9
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 9,
|
||||
"y": 1
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"grafana.*\", resource=\"memory\"})",
|
||||
"legendFormat": "",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory",
|
||||
"type": "gauge"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 12,
|
||||
"y": 1
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory (bytes)",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 15,
|
||||
"y": 1
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Cores",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 18,
|
||||
"y": 1
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Network RX",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 21,
|
||||
"y": 1
|
||||
},
|
||||
"id": 9,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Network TX",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 5
|
||||
},
|
||||
"id": 10,
|
||||
"panels": [],
|
||||
"title": "Resources",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 6
|
||||
},
|
||||
"id": 11,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}[$interval])",
|
||||
"legendFormat": "{{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Usage",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 6
|
||||
},
|
||||
"id": 12,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"grafana.*\", container!=\"\"}",
|
||||
"legendFormat": "{{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 6
|
||||
},
|
||||
"id": 13,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[$interval])",
|
||||
"legendFormat": "RX {{pod}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"grafana.*\"}[$interval])",
|
||||
"legendFormat": "TX {{pod}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Network I/O",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 14
|
||||
},
|
||||
"id": 14,
|
||||
"panels": [],
|
||||
"title": "Logs",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 15
|
||||
},
|
||||
"id": 15,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"grafana.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Log Stream",
|
||||
"type": "logs"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 25
|
||||
},
|
||||
"id": 16,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"grafana.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Errors & Warnings",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 39,
|
||||
"tags": [
|
||||
"platform",
|
||||
"grafana"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"auto": false,
|
||||
"current": {
|
||||
"text": "5m",
|
||||
"value": "5m"
|
||||
},
|
||||
"name": "interval",
|
||||
"options": [
|
||||
{
|
||||
"selected": false,
|
||||
"text": "1m",
|
||||
"value": "1m"
|
||||
},
|
||||
{
|
||||
"selected": true,
|
||||
"text": "5m",
|
||||
"value": "5m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "15m",
|
||||
"value": "15m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "30m",
|
||||
"value": "30m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "1h",
|
||||
"value": "1h"
|
||||
}
|
||||
],
|
||||
"query": "1m,5m,15m,30m,1h",
|
||||
"type": "interval"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timezone": "browser",
|
||||
"title": "Grafana",
|
||||
"uid": "plat-grafana"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,946 +0,0 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"panels": [],
|
||||
"title": "Prometheus Health",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 0,
|
||||
"y": 1
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "count(up{job!=\"\"} == 1)",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Targets Up",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 3,
|
||||
"y": 1
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "count(up{job!=\"\"} == 0) or vector(0)",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Targets Down",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 6,
|
||||
"y": 1
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "prometheus_tsdb_head_series",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Head Series",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 9,
|
||||
"y": 1
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(prometheus_tsdb_head_samples_appended_total[5m])",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Ingestion Rate",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 12,
|
||||
"y": 1
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "prometheus_tsdb_wal_storage_size_bytes",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "WAL Size",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 15,
|
||||
"y": 1
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "prometheus_tsdb_head_chunks",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Head Chunks",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 18,
|
||||
"y": 1
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "prometheus_tsdb_compactions_total",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Compactions",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 21,
|
||||
"y": 1
|
||||
},
|
||||
"id": 9,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "prometheus_tsdb_storage_blocks_bytes",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Storage Size",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 5
|
||||
},
|
||||
"id": 10,
|
||||
"panels": [],
|
||||
"title": "Resources",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 6
|
||||
},
|
||||
"id": 11,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"prometheus.*\", container!=\"\"}[$interval])",
|
||||
"legendFormat": "{{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Usage",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 6
|
||||
},
|
||||
"id": 12,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"prometheus.*\", container!=\"\"}",
|
||||
"legendFormat": "{{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 6
|
||||
},
|
||||
"id": 13,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"prometheus.*\"}[$interval])",
|
||||
"legendFormat": "RX {{pod}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"prometheus.*\"}[$interval])",
|
||||
"legendFormat": "TX {{pod}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Network I/O",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 14
|
||||
},
|
||||
"id": 14,
|
||||
"panels": [],
|
||||
"title": "Scrape Performance",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 15
|
||||
},
|
||||
"id": 15,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "scrape_duration_seconds",
|
||||
"legendFormat": "{{job}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Scrape Duration",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 15
|
||||
},
|
||||
"id": 16,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(prometheus_tsdb_head_samples_appended_total[$interval])",
|
||||
"legendFormat": "samples/s",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Samples Ingested",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 23
|
||||
},
|
||||
"id": 17,
|
||||
"panels": [],
|
||||
"title": "Target Status",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 24
|
||||
},
|
||||
"id": 18,
|
||||
"options": {
|
||||
"showHeader": true,
|
||||
"sortBy": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "up",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Target Health",
|
||||
"type": "table"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 32
|
||||
},
|
||||
"id": 19,
|
||||
"panels": [],
|
||||
"title": "Logs",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 33
|
||||
},
|
||||
"id": 20,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"prometheus.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Log Stream",
|
||||
"type": "logs"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 43
|
||||
},
|
||||
"id": 21,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"prometheus.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Errors & Warnings",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 39,
|
||||
"tags": [
|
||||
"platform",
|
||||
"prometheus"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"auto": false,
|
||||
"current": {
|
||||
"text": "5m",
|
||||
"value": "5m"
|
||||
},
|
||||
"name": "interval",
|
||||
"options": [
|
||||
{
|
||||
"selected": false,
|
||||
"text": "1m",
|
||||
"value": "1m"
|
||||
},
|
||||
{
|
||||
"selected": true,
|
||||
"text": "5m",
|
||||
"value": "5m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "15m",
|
||||
"value": "15m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "30m",
|
||||
"value": "30m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "1h",
|
||||
"value": "1h"
|
||||
}
|
||||
],
|
||||
"query": "1m,5m,15m,30m,1h",
|
||||
"type": "interval"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timezone": "browser",
|
||||
"title": "Prometheus",
|
||||
"uid": "plat-prometheus"
|
||||
}
|
||||
@@ -1,802 +0,0 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"panels": [],
|
||||
"title": "QuestDB Health",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 0,
|
||||
"y": 1
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"questdb.*\", phase=\"Running\"}",
|
||||
"instant": true,
|
||||
"legendFormat": "Running",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Pod Status",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 3
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 3,
|
||||
"y": 1
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"questdb.*\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Restarts",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 0.7
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.9
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 6,
|
||||
"y": 1
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[$interval]))",
|
||||
"legendFormat": "",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Usage",
|
||||
"type": "gauge"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 0.7
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.9
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 9,
|
||||
"y": 1
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"questdb.*\", resource=\"memory\"})",
|
||||
"legendFormat": "",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory",
|
||||
"type": "gauge"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 12,
|
||||
"y": 1
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory (bytes)",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 15,
|
||||
"y": 1
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Cores",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 18,
|
||||
"y": 1
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Network RX",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 21,
|
||||
"y": 1
|
||||
},
|
||||
"id": 9,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Network TX",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 5
|
||||
},
|
||||
"id": 10,
|
||||
"panels": [],
|
||||
"title": "Resources",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 6
|
||||
},
|
||||
"id": 11,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}[$interval])",
|
||||
"legendFormat": "{{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Usage",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 6
|
||||
},
|
||||
"id": 12,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"questdb.*\", container!=\"\"}",
|
||||
"legendFormat": "{{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 6
|
||||
},
|
||||
"id": 13,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[$interval])",
|
||||
"legendFormat": "RX {{pod}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"questdb.*\"}[$interval])",
|
||||
"legendFormat": "TX {{pod}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Network I/O",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 14
|
||||
},
|
||||
"id": 14,
|
||||
"panels": [],
|
||||
"title": "Logs",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 15
|
||||
},
|
||||
"id": 15,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"questdb.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Log Stream",
|
||||
"type": "logs"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 25
|
||||
},
|
||||
"id": 16,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"questdb.*\"} | json | level =~ \"ERROR|WARN|FATAL\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Errors & Warnings",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 39,
|
||||
"tags": [
|
||||
"platform",
|
||||
"questdb"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"auto": false,
|
||||
"current": {
|
||||
"text": "5m",
|
||||
"value": "5m"
|
||||
},
|
||||
"name": "interval",
|
||||
"options": [
|
||||
{
|
||||
"selected": false,
|
||||
"text": "1m",
|
||||
"value": "1m"
|
||||
},
|
||||
{
|
||||
"selected": true,
|
||||
"text": "5m",
|
||||
"value": "5m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "15m",
|
||||
"value": "15m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "30m",
|
||||
"value": "30m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "1h",
|
||||
"value": "1h"
|
||||
}
|
||||
],
|
||||
"query": "1m,5m,15m,30m,1h",
|
||||
"type": "interval"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timezone": "browser",
|
||||
"title": "QuestDB",
|
||||
"uid": "plat-questdb"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,802 +0,0 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"panels": [],
|
||||
"title": "Tempo Health",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 0,
|
||||
"y": 1
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", phase=\"Running\"}",
|
||||
"instant": true,
|
||||
"legendFormat": "Running",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Pod Status",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 3
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 3,
|
||||
"y": 1
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Restarts",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 0.7
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.9
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 6,
|
||||
"y": 1
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[$interval]))",
|
||||
"legendFormat": "",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Usage",
|
||||
"type": "gauge"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 0.7
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.9
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 9,
|
||||
"y": 1
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", resource=\"memory\"})",
|
||||
"legendFormat": "",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory",
|
||||
"type": "gauge"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 12,
|
||||
"y": 1
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory (bytes)",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 15,
|
||||
"y": 1
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Cores",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 18,
|
||||
"y": 1
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Network RX",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 21,
|
||||
"y": 1
|
||||
},
|
||||
"id": 9,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Network TX",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 5
|
||||
},
|
||||
"id": 10,
|
||||
"panels": [],
|
||||
"title": "Resources",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 6
|
||||
},
|
||||
"id": 11,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}[$interval])",
|
||||
"legendFormat": "{{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Usage",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 6
|
||||
},
|
||||
"id": 12,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\", container!=\"\"}",
|
||||
"legendFormat": "{{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 6
|
||||
},
|
||||
"id": 13,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[$interval])",
|
||||
"legendFormat": "RX {{pod}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"}[$interval])",
|
||||
"legendFormat": "TX {{pod}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Network I/O",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 14
|
||||
},
|
||||
"id": 14,
|
||||
"panels": [],
|
||||
"title": "Logs",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 15
|
||||
},
|
||||
"id": 15,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Log Stream",
|
||||
"type": "logs"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 25
|
||||
},
|
||||
"id": 16,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"${DS_TEMPO}.*\"} | logfmt | level =~ \"error|warn|fatal\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Errors & Warnings",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 39,
|
||||
"tags": [
|
||||
"platform",
|
||||
"${DS_TEMPO}"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"auto": false,
|
||||
"current": {
|
||||
"text": "5m",
|
||||
"value": "5m"
|
||||
},
|
||||
"name": "interval",
|
||||
"options": [
|
||||
{
|
||||
"selected": false,
|
||||
"text": "1m",
|
||||
"value": "1m"
|
||||
},
|
||||
{
|
||||
"selected": true,
|
||||
"text": "5m",
|
||||
"value": "5m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "15m",
|
||||
"value": "15m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "30m",
|
||||
"value": "30m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "1h",
|
||||
"value": "1h"
|
||||
}
|
||||
],
|
||||
"query": "1m,5m,15m,30m,1h",
|
||||
"type": "interval"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timezone": "browser",
|
||||
"title": "Tempo",
|
||||
"uid": "plat-${DS_TEMPO}"
|
||||
}
|
||||
@@ -1,963 +0,0 @@
|
||||
{
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"builtIn": 1,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"enable": true,
|
||||
"hide": true,
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"name": "Annotations & Alerts",
|
||||
"type": "dashboard"
|
||||
}
|
||||
]
|
||||
},
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"id": 1,
|
||||
"panels": [],
|
||||
"title": "Service Health",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 0,
|
||||
"y": 1
|
||||
},
|
||||
"id": 2,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "kube_pod_status_phase{namespace=\"foxhunt\", pod=~\"api.*\", phase=\"Running\"}",
|
||||
"instant": true,
|
||||
"legendFormat": "Running",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Pod Status",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 3
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 10
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 3,
|
||||
"y": 1
|
||||
},
|
||||
"id": 3,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(kube_pod_container_status_restarts_total{namespace=\"foxhunt\", pod=~\"api.*\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Restarts",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 0.7
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.9
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 6,
|
||||
"y": 1
|
||||
},
|
||||
"id": 4,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[$interval]))",
|
||||
"legendFormat": "",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Usage",
|
||||
"type": "gauge"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"max": 1,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 0.7
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.9
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 5,
|
||||
"w": 3,
|
||||
"x": 9,
|
||||
"y": 1
|
||||
},
|
||||
"id": 5,
|
||||
"options": {
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}) / sum(kube_pod_container_resource_limits{namespace=\"foxhunt\", pod=~\"api.*\", resource=\"memory\"})",
|
||||
"legendFormat": "",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory",
|
||||
"type": "gauge"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 12,
|
||||
"y": 1
|
||||
},
|
||||
"id": 6,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"})",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory (bytes)",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 15,
|
||||
"y": 1
|
||||
},
|
||||
"id": 7,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Cores",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 18,
|
||||
"y": 1
|
||||
},
|
||||
"id": 8,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Network RX",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 4,
|
||||
"w": 3,
|
||||
"x": 21,
|
||||
"y": 1
|
||||
},
|
||||
"id": 9,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "area",
|
||||
"reduceOptions": {
|
||||
"calcs": [
|
||||
"lastNotNull"
|
||||
]
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "sum(rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[5m]))",
|
||||
"instant": true,
|
||||
"legendFormat": "",
|
||||
"range": false,
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Network TX",
|
||||
"type": "stat"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 5
|
||||
},
|
||||
"id": 10,
|
||||
"panels": [],
|
||||
"title": "Resources",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 6
|
||||
},
|
||||
"id": 11,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_cpu_usage_seconds_total{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}[$interval])",
|
||||
"legendFormat": "{{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "CPU Usage",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 6
|
||||
},
|
||||
"id": 12,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "container_memory_working_set_bytes{namespace=\"foxhunt\", pod=~\"api.*\", container!=\"\"}",
|
||||
"legendFormat": "{{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Memory",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "Bps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 6
|
||||
},
|
||||
"id": 13,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_network_receive_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[$interval])",
|
||||
"legendFormat": "RX {{pod}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(container_network_transmit_bytes_total{namespace=\"foxhunt\", pod=~\"api.*\"}[$interval])",
|
||||
"legendFormat": "TX {{pod}}",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Network I/O",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 14
|
||||
},
|
||||
"id": 14,
|
||||
"panels": [],
|
||||
"title": "gRPC Metrics",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 15
|
||||
},
|
||||
"id": 15,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(grpc_server_handled_total{grpc_service=~\"foxhunt.*\", pod=~\"api.*\"}[$interval])",
|
||||
"legendFormat": "{{grpc_method}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "gRPC Request Rate",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
},
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 15
|
||||
},
|
||||
"id": 16,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "histogram_quantile(0.99, rate(grpc_server_handling_seconds_bucket{pod=~\"api.*\"}[$interval]))",
|
||||
"legendFormat": "p99",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "gRPC Latency (p99)",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 1,
|
||||
"stacking": {
|
||||
"mode": "none"
|
||||
}
|
||||
}
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 15
|
||||
},
|
||||
"id": 17,
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
},
|
||||
"tooltip": {
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "rate(grpc_server_handled_total{grpc_code!=\"OK\", pod=~\"api.*\"}[$interval])",
|
||||
"legendFormat": "{{grpc_code}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "gRPC Errors",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": {
|
||||
"h": 1,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 23
|
||||
},
|
||||
"id": 18,
|
||||
"panels": [],
|
||||
"title": "Logs",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 24
|
||||
},
|
||||
"id": 19,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"api.*\"} | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Log Stream",
|
||||
"type": "logs"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 34
|
||||
},
|
||||
"id": 20,
|
||||
"options": {
|
||||
"dedupStrategy": "none",
|
||||
"enableLogDetails": true,
|
||||
"prettifyLogMessage": false,
|
||||
"showCommonLabels": false,
|
||||
"showLabels": true,
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"wrapLogMessage": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "${DS_LOKI}"
|
||||
},
|
||||
"expr": "{namespace=\"foxhunt\", pod=~\"api.*\"} | json | level =~ \"ERROR|WARN|FATAL\" | pattern `<_> <_> <_> <content>` | regexp `msg=\"(?P<clean_msg>[^\"]*)\"` | line_format `{{if .clean_msg}}{{.clean_msg}}{{else}}{{.content}}{{end}}` | drop content, clean_msg",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Errors & Warnings",
|
||||
"type": "logs"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 39,
|
||||
"tags": [
|
||||
"service",
|
||||
"api"
|
||||
],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"auto": false,
|
||||
"current": {
|
||||
"text": "5m",
|
||||
"value": "5m"
|
||||
},
|
||||
"name": "interval",
|
||||
"options": [
|
||||
{
|
||||
"selected": false,
|
||||
"text": "1m",
|
||||
"value": "1m"
|
||||
},
|
||||
{
|
||||
"selected": true,
|
||||
"text": "5m",
|
||||
"value": "5m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "15m",
|
||||
"value": "15m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "30m",
|
||||
"value": "30m"
|
||||
},
|
||||
{
|
||||
"selected": false,
|
||||
"text": "1h",
|
||||
"value": "1h"
|
||||
}
|
||||
],
|
||||
"query": "1m,5m,15m,30m,1h",
|
||||
"type": "interval"
|
||||
}
|
||||
]
|
||||
},
|
||||
"time": {
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"timezone": "browser",
|
||||
"title": "API Gateway",
|
||||
"uid": "svc-api-gateway"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,125 +0,0 @@
|
||||
# NVIDIA DCGM Exporter — GPU metrics for Cockpit
|
||||
# Runs on all GPU node pools (ci-training, gpu-dev)
|
||||
# Alloy auto-discovers and scrapes these pods via annotations
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: dcgm-counters
|
||||
namespace: monitoring
|
||||
data:
|
||||
counters.csv: |
|
||||
# Basic GPU metrics (no DCP/profiling — avoids SYS_ADMIN requirement)
|
||||
dcgm_sm_clock, gauge, SM clock frequency (in MHz).
|
||||
dcgm_memory_clock, gauge, Memory clock frequency (in MHz).
|
||||
dcgm_memory_temp, gauge, Memory temperature (in C).
|
||||
dcgm_gpu_temp, gauge, GPU temperature (in C).
|
||||
dcgm_power_usage, gauge, Power draw (in W).
|
||||
dcgm_total_energy_consumption, counter, Total energy consumption since boot (in mJ).
|
||||
dcgm_gpu_utilization, gauge, GPU utilization (in %).
|
||||
dcgm_mem_copy_utilization, gauge, Memory utilization (in %).
|
||||
dcgm_enc_utilization, gauge, Encoder utilization (in %).
|
||||
dcgm_dec_utilization, gauge, Decoder utilization (in %).
|
||||
dcgm_xid_errors, gauge, Value of the last XID error encountered.
|
||||
dcgm_fb_free, gauge, Framebuffer memory free (in MiB).
|
||||
dcgm_fb_used, gauge, Framebuffer memory used (in MiB).
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: dcgm-exporter
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/name: dcgm-exporter
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: dcgm-exporter
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: dcgm-exporter
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "9400"
|
||||
prometheus.io/path: "/metrics"
|
||||
spec:
|
||||
# Only schedule on nodes that have GPUs
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: nvidia.com/gpu.present
|
||||
operator: In
|
||||
values:
|
||||
- "true"
|
||||
# NOTE: runtimeClassName: nvidia removed — Scaleway Kapsule GPU pools use
|
||||
# the nvidia container runtime as the default containerd handler. The
|
||||
# RuntimeClass CRD is only created by the full NVIDIA GPU Operator, not
|
||||
# the device plugin alone, so requiring it prevents pod scheduling.
|
||||
tolerations:
|
||||
# Tolerate GPU taints so the exporter can run on GPU nodes
|
||||
- operator: Exists
|
||||
effect: NoSchedule
|
||||
containers:
|
||||
- name: dcgm-exporter
|
||||
image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubuntu22.04
|
||||
ports:
|
||||
- name: metrics
|
||||
containerPort: 9400
|
||||
args:
|
||||
- "-f"
|
||||
- "/etc/dcgm-exporter/custom/counters.csv"
|
||||
env:
|
||||
- name: DCGM_EXPORTER_KUBERNETES
|
||||
value: "true"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 1Gi
|
||||
securityContext:
|
||||
runAsNonRoot: false
|
||||
runAsUser: 0
|
||||
volumeMounts:
|
||||
- name: device-metrics
|
||||
mountPath: /var/lib/dcgm
|
||||
- name: custom-counters
|
||||
mountPath: /etc/dcgm-exporter/custom
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: device-metrics
|
||||
emptyDir: {}
|
||||
- name: custom-counters
|
||||
configMap:
|
||||
name: dcgm-counters
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: dcgm-exporter
|
||||
namespace: monitoring
|
||||
labels:
|
||||
app.kubernetes.io/name: dcgm-exporter
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "9400"
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app.kubernetes.io/name: dcgm-exporter
|
||||
ports:
|
||||
- name: metrics
|
||||
port: 9400
|
||||
targetPort: 9400
|
||||
@@ -1,57 +0,0 @@
|
||||
# Grafana datasource provisioning for Loki, Tempo, and QuestDB
|
||||
# Apply via: kubectl apply -f grafana-datasources.yaml && kubectl rollout restart deploy/grafana -n foxhunt
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: grafana-datasources-extra
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: grafana
|
||||
grafana_datasource: "1"
|
||||
data:
|
||||
loki-tempo.yaml: |
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki.foxhunt.svc.cluster.local:3100
|
||||
isDefault: false
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- datasourceUid: tempo
|
||||
matcherRegex: '"trace_id":"(\w+)"'
|
||||
name: TraceID
|
||||
url: "$${__value.raw}"
|
||||
- name: Tempo
|
||||
type: tempo
|
||||
uid: tempo
|
||||
access: proxy
|
||||
url: http://tempo.foxhunt.svc.cluster.local:3200
|
||||
isDefault: false
|
||||
jsonData:
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
filterByTraceID: true
|
||||
nodeGraph:
|
||||
enabled: true
|
||||
serviceMap:
|
||||
datasourceUid: prometheus
|
||||
questdb.yaml: |
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: QuestDB
|
||||
type: postgres
|
||||
uid: questdb
|
||||
access: proxy
|
||||
url: questdb.foxhunt.svc.cluster.local:8812
|
||||
user: admin
|
||||
database: qdb
|
||||
isDefault: false
|
||||
jsonData:
|
||||
sslmode: disable
|
||||
maxOpenConns: 5
|
||||
postgresVersion: 1200
|
||||
timescaledb: false
|
||||
secureJsonData:
|
||||
password: quest
|
||||
@@ -1,122 +0,0 @@
|
||||
# Grafana Loki — lightweight log aggregation (single-binary mode)
|
||||
# Runs on gitlab node pool alongside Grafana and Prometheus
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: loki-config
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: loki
|
||||
data:
|
||||
loki.yaml: |
|
||||
auth_enabled: false
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
grpc_listen_port: 9096
|
||||
common:
|
||||
path_prefix: /loki
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /loki/chunks
|
||||
rules_directory: /loki/rules
|
||||
replication_factor: 1
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
schema_config:
|
||||
configs:
|
||||
- from: "2024-01-01"
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
limits_config:
|
||||
retention_period: 168h # 7 days
|
||||
max_query_series: 500
|
||||
compactor:
|
||||
working_directory: /loki/compactor
|
||||
compaction_interval: 10m
|
||||
retention_enabled: true
|
||||
delete_request_store: filesystem
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: loki
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: loki
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: loki
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: loki
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
containers:
|
||||
- name: loki
|
||||
image: grafana/loki:3.4.2
|
||||
args: ["-config.file=/etc/loki/loki.yaml"]
|
||||
ports:
|
||||
- containerPort: 3100
|
||||
name: http
|
||||
- containerPort: 9096
|
||||
name: grpc
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/loki
|
||||
- name: data
|
||||
mountPath: /loki
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: 3100
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: 3100
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: loki-config
|
||||
- name: data
|
||||
emptyDir:
|
||||
sizeLimit: 5Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: loki
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: loki
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: loki
|
||||
ports:
|
||||
- port: 3100
|
||||
targetPort: 3100
|
||||
name: http
|
||||
- port: 9096
|
||||
targetPort: 9096
|
||||
name: grpc
|
||||
@@ -1,169 +0,0 @@
|
||||
# Broker Gateway alert rules — discovered automatically by Prometheus Operator
|
||||
# Apply: kubectl apply -f prometheus-rules-broker.yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: foxhunt-broker-alerts
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
role: alert-rules
|
||||
spec:
|
||||
groups:
|
||||
- name: broker_gateway_fix_session
|
||||
interval: 30s
|
||||
rules:
|
||||
# FIX session disconnected >60s
|
||||
- alert: BrokerGatewayFIXSessionDisconnected
|
||||
expr: broker_gateway_fix_session_status == 0
|
||||
for: 60s
|
||||
labels:
|
||||
severity: critical
|
||||
service: broker_gateway_service
|
||||
component: fix_session
|
||||
impact: trading_halted
|
||||
annotations:
|
||||
summary: "FIX session {{ $labels.session_id }} DISCONNECTED"
|
||||
description: "FIX session down >60s. All order routing halted."
|
||||
action: "Check FIX engine logs, verify network to AMP Futures gateway"
|
||||
|
||||
# Order latency P95 > 100ms
|
||||
- alert: BrokerGatewayHighOrderLatency
|
||||
expr: |
|
||||
histogram_quantile(0.95,
|
||||
sum by(le, order_type) (rate(broker_gateway_order_latency_seconds_bucket[5m]))
|
||||
) * 1000 > 100
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
service: broker_gateway_service
|
||||
component: order_routing
|
||||
impact: performance_degraded
|
||||
annotations:
|
||||
summary: "Order latency P95 > 100ms for {{ $labels.order_type }}"
|
||||
|
||||
# Error rate > 5%
|
||||
- alert: BrokerGatewayHighErrorRate
|
||||
expr: |
|
||||
(
|
||||
sum by(severity) (rate(broker_gateway_error_total[5m]))
|
||||
/
|
||||
(sum(rate(broker_gateway_orders_submitted_total[5m])) + 0.001)
|
||||
) > 0.05
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
service: broker_gateway_service
|
||||
component: error_handling
|
||||
impact: reliability_degraded
|
||||
annotations:
|
||||
summary: "Error rate {{ $value | humanizePercentage }} ({{ $labels.severity }})"
|
||||
|
||||
# No orders in 10m during market hours
|
||||
- alert: BrokerGatewayNoOrderActivity
|
||||
expr: |
|
||||
(time() - broker_gateway_last_order_time) > 600
|
||||
and
|
||||
(hour() >= 9 and hour() < 16)
|
||||
for: 1m
|
||||
labels:
|
||||
severity: warning
|
||||
service: broker_gateway_service
|
||||
component: order_flow
|
||||
impact: trading_inactive
|
||||
annotations:
|
||||
summary: "No order activity for 10+ minutes during market hours"
|
||||
|
||||
# Position mismatch > $10K
|
||||
- alert: BrokerGatewayPositionMismatch
|
||||
expr: |
|
||||
abs(
|
||||
broker_gateway_position_value_usd
|
||||
-
|
||||
on(symbol, account_id) group_left()
|
||||
trading_service_position_value_usd
|
||||
) > 10000
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
service: broker_gateway_service
|
||||
component: position_management
|
||||
impact: data_integrity
|
||||
annotations:
|
||||
summary: "Position mismatch ${{ $value | humanize }} on {{ $labels.symbol }}"
|
||||
action: "HALT trading on affected symbol, reconcile with broker"
|
||||
|
||||
# FIX sequence number gap
|
||||
- alert: BrokerGatewayFIXSequenceGap
|
||||
expr: rate(broker_gateway_sequence_number_gap_total[5m]) > 0
|
||||
for: 1m
|
||||
labels:
|
||||
severity: warning
|
||||
service: broker_gateway_service
|
||||
component: fix_session
|
||||
impact: message_loss
|
||||
annotations:
|
||||
summary: "FIX sequence gap on {{ $labels.session_id }} ({{ $value | humanize }} gaps/sec)"
|
||||
|
||||
- name: broker_gateway_performance
|
||||
interval: 60s
|
||||
rules:
|
||||
# High FIX heartbeat RTT
|
||||
- alert: BrokerGatewayHighFIXHeartbeatRTT
|
||||
expr: broker_gateway_fix_heartbeat_rtt_ms > 50
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
service: broker_gateway_service
|
||||
component: fix_session
|
||||
impact: network_latency
|
||||
annotations:
|
||||
summary: "FIX heartbeat RTT {{ $value }}ms on {{ $labels.session_id }}"
|
||||
|
||||
# High order rejection rate per symbol
|
||||
- alert: BrokerGatewayHighOrderRejectionRate
|
||||
expr: |
|
||||
(
|
||||
sum by(symbol) (rate(broker_gateway_orders_rejected_total[5m]))
|
||||
/
|
||||
(sum by(symbol) (rate(broker_gateway_orders_submitted_total[5m])) + 0.001)
|
||||
) > 0.10
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
service: broker_gateway_service
|
||||
component: order_validation
|
||||
impact: order_flow_disrupted
|
||||
annotations:
|
||||
summary: "Order rejection rate {{ $value | humanizePercentage }} for {{ $labels.symbol }}"
|
||||
|
||||
# Slow database queries
|
||||
- alert: BrokerGatewaySlowDatabaseQueries
|
||||
expr: |
|
||||
histogram_quantile(0.95,
|
||||
sum by(le) (rate(database_query_duration_seconds_bucket{service="broker_gateway_service"}[5m]))
|
||||
) > 0.100
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
service: broker_gateway_service
|
||||
component: database
|
||||
impact: performance_degraded
|
||||
annotations:
|
||||
summary: "Database query P95 > 100ms"
|
||||
|
||||
# High margin usage
|
||||
- alert: BrokerGatewayHighMarginUsage
|
||||
expr: |
|
||||
(
|
||||
broker_gateway_margin_used_usd
|
||||
/ (broker_gateway_cash_balance_usd + 0.001)
|
||||
) > 0.80
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
service: broker_gateway_service
|
||||
component: risk_management
|
||||
impact: margin_pressure
|
||||
annotations:
|
||||
summary: "Margin usage {{ $value | humanizePercentage }} for {{ $labels.account_id }}"
|
||||
@@ -1,328 +0,0 @@
|
||||
# HFT alert rules — discovered automatically by Prometheus Operator
|
||||
# Apply: kubectl apply -f prometheus-rules-hft.yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: foxhunt-hft-alerts
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
role: alert-rules
|
||||
spec:
|
||||
groups:
|
||||
# ═══════════════════════════════
|
||||
# CRITICAL HFT ALERTS
|
||||
# ═══════════════════════════════
|
||||
- name: hft_critical
|
||||
rules:
|
||||
- alert: TradingServiceDown
|
||||
expr: up{job="foxhunt-services", service="trading-service"} == 0
|
||||
for: 5s
|
||||
labels:
|
||||
severity: critical
|
||||
component: trading_engine
|
||||
impact: trading_halt
|
||||
annotations:
|
||||
summary: "Trading Service is DOWN"
|
||||
description: "Trading Service has been down for >5s. All trading operations halted."
|
||||
action: "Immediate intervention required"
|
||||
|
||||
- alert: DataAcquisitionDown
|
||||
expr: up{job="foxhunt-services", service="data-acquisition-service"} == 0
|
||||
for: 10s
|
||||
labels:
|
||||
severity: critical
|
||||
component: market_data
|
||||
impact: blind_trading
|
||||
annotations:
|
||||
summary: "Data Acquisition Service is DOWN"
|
||||
description: "Market data feed down >10s. Trading without data is dangerous."
|
||||
action: "Stop all trading immediately"
|
||||
|
||||
- alert: BrokerGatewayDown
|
||||
expr: up{job="foxhunt-services", service="broker-gateway-service"} == 0
|
||||
for: 5s
|
||||
labels:
|
||||
severity: critical
|
||||
component: risk_management
|
||||
impact: uncontrolled_risk
|
||||
annotations:
|
||||
summary: "Broker Gateway (risk controls) is DOWN"
|
||||
description: "Broker gateway down >5s. Order routing and risk controls offline."
|
||||
action: "Emergency trading halt"
|
||||
|
||||
- alert: TradingLatencyHigh
|
||||
expr: histogram_quantile(0.95, foxhunt_trading_latency_seconds_bucket{operation="order_placement"}) > 0.001
|
||||
for: 30s
|
||||
labels:
|
||||
severity: critical
|
||||
component: trading_engine
|
||||
impact: competitive_disadvantage
|
||||
annotations:
|
||||
summary: "Trading latency P95 > 1ms"
|
||||
description: "95th percentile order placement latency is {{ $value }}s. HFT advantage compromised."
|
||||
action: "Investigate performance bottlenecks"
|
||||
|
||||
- alert: PositionLimitBreach
|
||||
expr: foxhunt_position_size_usd / foxhunt_position_limit_usd > 0.95
|
||||
for: 0s
|
||||
labels:
|
||||
severity: critical
|
||||
component: risk_management
|
||||
impact: regulatory_breach
|
||||
annotations:
|
||||
summary: "Position limit nearly breached ({{ $value | humanizePercentage }})"
|
||||
description: "Current position at {{ $value | humanizePercentage }} of limit."
|
||||
action: "Reduce positions immediately"
|
||||
|
||||
- alert: LossLimitBreach
|
||||
expr: foxhunt_daily_pnl_usd < foxhunt_loss_limit_usd
|
||||
for: 0s
|
||||
labels:
|
||||
severity: critical
|
||||
component: risk_management
|
||||
impact: financial_loss
|
||||
annotations:
|
||||
summary: "Daily loss limit breached"
|
||||
description: "Daily P&L ${{ $value }} breaches loss limit. Trading must halt."
|
||||
action: "Emergency trading halt"
|
||||
|
||||
# ═══════════════════════════════
|
||||
# HIGH PRIORITY ALERTS
|
||||
# ═══════════════════════════════
|
||||
- name: hft_high
|
||||
rules:
|
||||
- alert: DatabaseConnectionHigh
|
||||
expr: foxhunt_database_connections_active / foxhunt_database_connections_max > 0.85
|
||||
for: 1m
|
||||
labels:
|
||||
severity: high
|
||||
component: database
|
||||
impact: performance_degradation
|
||||
annotations:
|
||||
summary: "Database connection pool at {{ $value | humanizePercentage }}"
|
||||
|
||||
- alert: MemoryUsageHigh
|
||||
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
|
||||
for: 2m
|
||||
labels:
|
||||
severity: high
|
||||
component: system
|
||||
impact: performance_degradation
|
||||
annotations:
|
||||
summary: "Memory usage {{ $value | humanizePercentage }} on {{ $labels.instance }}"
|
||||
|
||||
- alert: CPUUsageHigh
|
||||
expr: 100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
|
||||
for: 5m
|
||||
labels:
|
||||
severity: high
|
||||
component: system
|
||||
impact: performance_degradation
|
||||
annotations:
|
||||
summary: "CPU usage {{ $value | printf \"%.1f\" }}% on {{ $labels.instance }}"
|
||||
|
||||
- alert: DiskUsageHigh
|
||||
expr: (1 - (node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"})) > 0.85
|
||||
for: 5m
|
||||
labels:
|
||||
severity: high
|
||||
component: system
|
||||
impact: storage_failure
|
||||
annotations:
|
||||
summary: "Disk usage {{ $value | humanizePercentage }} on {{ $labels.instance }} {{ $labels.mountpoint }}"
|
||||
|
||||
- alert: OrderRejectionRateHigh
|
||||
expr: rate(foxhunt_orders_rejected_total[5m]) / rate(foxhunt_orders_total[5m]) > 0.05
|
||||
for: 2m
|
||||
labels:
|
||||
severity: high
|
||||
component: trading_engine
|
||||
impact: trading_inefficiency
|
||||
annotations:
|
||||
summary: "Order rejection rate {{ $value | humanizePercentage }}"
|
||||
|
||||
- alert: MarketDataLag
|
||||
expr: foxhunt_market_data_lag_seconds > 0.1
|
||||
for: 1m
|
||||
labels:
|
||||
severity: high
|
||||
component: market_data
|
||||
impact: stale_data
|
||||
annotations:
|
||||
summary: "Market data lag {{ $value }}s — trading on stale data"
|
||||
|
||||
- alert: DrawdownHigh
|
||||
expr: foxhunt_drawdown_percent > 0.05
|
||||
for: 15m
|
||||
labels:
|
||||
severity: high
|
||||
component: risk_management
|
||||
impact: financial_risk
|
||||
annotations:
|
||||
summary: "Portfolio drawdown {{ $value | humanizePercentage }}"
|
||||
|
||||
# ═══════════════════════════════
|
||||
# MEDIUM / WARNING ALERTS
|
||||
# ═══════════════════════════════
|
||||
- name: hft_medium
|
||||
rules:
|
||||
- alert: ServiceRestarted
|
||||
expr: increase(process_start_time_seconds[10m]) > 0
|
||||
for: 0s
|
||||
labels:
|
||||
severity: medium
|
||||
component: service
|
||||
impact: disruption
|
||||
annotations:
|
||||
summary: "Service {{ $labels.job }} restarted on {{ $labels.instance }}"
|
||||
|
||||
- alert: NetworkLatencyHigh
|
||||
expr: foxhunt_network_latency_seconds > 0.01
|
||||
for: 5m
|
||||
labels:
|
||||
severity: medium
|
||||
component: network
|
||||
impact: performance_impact
|
||||
annotations:
|
||||
summary: "Network latency {{ $value }}s"
|
||||
|
||||
- alert: BackupFailed
|
||||
expr: time() - foxhunt_last_backup_timestamp > 86400
|
||||
for: 1h
|
||||
labels:
|
||||
severity: medium
|
||||
component: backup
|
||||
impact: data_risk
|
||||
annotations:
|
||||
summary: "Last backup {{ $value | humanizeDuration }} ago"
|
||||
|
||||
- alert: SSLCertificateExpiringSoon
|
||||
expr: (ssl_certificate_expiry_timestamp - time()) / 86400 < 30
|
||||
for: 1h
|
||||
labels:
|
||||
severity: medium
|
||||
component: security
|
||||
impact: service_disruption
|
||||
annotations:
|
||||
summary: "SSL cert expires in {{ $value }} days for {{ $labels.instance }}"
|
||||
|
||||
# ═══════════════════════════════
|
||||
# BUSINESS LOGIC ALERTS
|
||||
# ═══════════════════════════════
|
||||
- name: hft_business
|
||||
rules:
|
||||
- alert: TradingVolumeAnomalyHigh
|
||||
expr: rate(foxhunt_trades_total[5m]) > (avg_over_time(rate(foxhunt_trades_total[5m])[1h:5m]) * 3)
|
||||
for: 2m
|
||||
labels:
|
||||
severity: medium
|
||||
component: trading_engine
|
||||
impact: business_anomaly
|
||||
annotations:
|
||||
summary: "Trading volume 3x hourly average ({{ $value }} trades/sec)"
|
||||
|
||||
- alert: TradingVolumeAnomalyLow
|
||||
expr: rate(foxhunt_trades_total[5m]) < (avg_over_time(rate(foxhunt_trades_total[5m])[1h:5m]) * 0.1)
|
||||
for: 10m
|
||||
labels:
|
||||
severity: medium
|
||||
component: trading_engine
|
||||
impact: business_anomaly
|
||||
annotations:
|
||||
summary: "Trading volume 10% of hourly average ({{ $value }} trades/sec)"
|
||||
|
||||
- alert: PnLVolatilityHigh
|
||||
expr: stddev_over_time(foxhunt_pnl_usd[1h]) > 10000
|
||||
for: 30m
|
||||
labels:
|
||||
severity: medium
|
||||
component: risk_management
|
||||
impact: financial_risk
|
||||
annotations:
|
||||
summary: "P&L std dev ${{ $value }} over 1h"
|
||||
|
||||
# ═══════════════════════════════
|
||||
# INFRASTRUCTURE ALERTS
|
||||
# ═══════════════════════════════
|
||||
- name: hft_infrastructure
|
||||
rules:
|
||||
- alert: RedisConnectionFailed
|
||||
expr: redis_connected_clients{job="redis"} == 0
|
||||
for: 30s
|
||||
labels:
|
||||
severity: critical
|
||||
component: redis
|
||||
impact: cache_failure
|
||||
annotations:
|
||||
summary: "Redis has 0 connected clients"
|
||||
|
||||
- alert: PostgreSQLDown
|
||||
expr: pg_up{job="postgres"} == 0
|
||||
for: 30s
|
||||
labels:
|
||||
severity: critical
|
||||
component: postgresql
|
||||
impact: data_unavailable
|
||||
annotations:
|
||||
summary: "PostgreSQL is not responding"
|
||||
|
||||
# ═══════════════════════════════
|
||||
# SECURITY ALERTS
|
||||
# ═══════════════════════════════
|
||||
- name: hft_security
|
||||
rules:
|
||||
- alert: AuthenticationFailuresHigh
|
||||
expr: rate(foxhunt_auth_failures_total[5m]) > 5
|
||||
for: 2m
|
||||
labels:
|
||||
severity: high
|
||||
component: security
|
||||
impact: security_breach
|
||||
annotations:
|
||||
summary: "{{ $value }} auth failures/sec"
|
||||
|
||||
- alert: UnauthorizedAccessAttempts
|
||||
expr: rate(foxhunt_unauthorized_requests_total[5m]) > 1
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
component: security
|
||||
impact: security_breach
|
||||
annotations:
|
||||
summary: "{{ $value }} unauthorized requests/sec from {{ $labels.source_ip }}"
|
||||
|
||||
- alert: APIRateLimitExceeded
|
||||
expr: rate(foxhunt_api_rate_limit_exceeded_total[5m]) > 0.1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: medium
|
||||
component: api
|
||||
impact: service_degradation
|
||||
annotations:
|
||||
summary: "API rate limits exceeded {{ $value }}/sec"
|
||||
|
||||
# ═══════════════════════════════
|
||||
# MONITORING HEALTH
|
||||
# ═══════════════════════════════
|
||||
- name: hft_monitoring
|
||||
rules:
|
||||
- alert: PrometheusTargetDown
|
||||
expr: up == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: medium
|
||||
component: monitoring
|
||||
impact: observability_loss
|
||||
annotations:
|
||||
summary: "{{ $labels.job }} target {{ $labels.instance }} down >2m"
|
||||
|
||||
- alert: HighAlertRate
|
||||
expr: rate(prometheus_notifications_total[5m]) > 10
|
||||
for: 10m
|
||||
labels:
|
||||
severity: medium
|
||||
component: monitoring
|
||||
impact: alert_fatigue
|
||||
annotations:
|
||||
summary: "{{ $value }} alerts/sec — check for alert storm"
|
||||
@@ -1,56 +0,0 @@
|
||||
# Storage alert rules — volume usage warnings before auto-expand kicks in
|
||||
# Apply: kubectl apply -f prometheus-rules-storage.yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: foxhunt-storage-alerts
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
role: alert-rules
|
||||
spec:
|
||||
groups:
|
||||
- name: storage_volume_usage
|
||||
interval: 60s
|
||||
rules:
|
||||
# PVC usage > 75% — warning (auto-expand triggers at 85%)
|
||||
- alert: PVCUsageHigh
|
||||
expr: |
|
||||
100 * kubelet_volume_stats_used_bytes{namespace="foxhunt"}
|
||||
/ kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 75
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
component: storage
|
||||
annotations:
|
||||
summary: "PVC {{ $labels.persistentvolumeclaim }} at {{ $value | printf \"%.0f\" }}% usage"
|
||||
description: "Volume {{ $labels.persistentvolumeclaim }} is {{ $value | printf \"%.0f\" }}% full. Auto-expand triggers at 85%."
|
||||
|
||||
# PVC usage > 90% — critical (auto-expand should have already acted)
|
||||
- alert: PVCUsageCritical
|
||||
expr: |
|
||||
100 * kubelet_volume_stats_used_bytes{namespace="foxhunt"}
|
||||
/ kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 90
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
component: storage
|
||||
annotations:
|
||||
summary: "PVC {{ $labels.persistentvolumeclaim }} CRITICAL at {{ $value | printf \"%.0f\" }}%"
|
||||
description: "Volume {{ $labels.persistentvolumeclaim }} is {{ $value | printf \"%.0f\" }}% full despite auto-expand. Investigate immediately."
|
||||
action: "Check pvc-autoscaler CronJob logs. Manual expand: kubectl patch pvc {{ $labels.persistentvolumeclaim }} -n foxhunt --type merge -p '{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"XXGi\"}}}}'"
|
||||
|
||||
# PVC approaching max auto-expand limit (>80Gi of 100Gi max)
|
||||
- alert: PVCApproachingMaxSize
|
||||
expr: |
|
||||
kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 80 * 1024 * 1024 * 1024
|
||||
and
|
||||
100 * kubelet_volume_stats_used_bytes{namespace="foxhunt"}
|
||||
/ kubelet_volume_stats_capacity_bytes{namespace="foxhunt"} > 70
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
component: storage
|
||||
annotations:
|
||||
summary: "PVC {{ $labels.persistentvolumeclaim }} is large and growing"
|
||||
description: "Volume is >80Gi and >70% full. Auto-expand max is 100Gi. Consider manual intervention or data cleanup."
|
||||
@@ -1,400 +0,0 @@
|
||||
# kube-prometheus-stack Helm values — Foxhunt production monitoring
|
||||
#
|
||||
# Install:
|
||||
# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
|
||||
# helm repo update
|
||||
# helm install prometheus-stack prometheus-community/kube-prometheus-stack \
|
||||
# --namespace foxhunt \
|
||||
# -f infra/k8s/monitoring/prometheus-stack-values.yaml
|
||||
#
|
||||
# Upgrade:
|
||||
# helm upgrade prometheus-stack prometheus-community/kube-prometheus-stack \
|
||||
# --namespace foxhunt \
|
||||
# -f infra/k8s/monitoring/prometheus-stack-values.yaml
|
||||
#
|
||||
# Prerequisites:
|
||||
# kubectl -n foxhunt create secret generic alertmanager-slack \
|
||||
# --from-literal=slack-webhook-url='https://hooks.slack.com/services/...'
|
||||
#
|
||||
# After install, delete old standalone resources:
|
||||
# kubectl -n foxhunt delete deploy prometheus
|
||||
# kubectl -n foxhunt delete svc prometheus
|
||||
# kubectl -n foxhunt delete cm prometheus-config
|
||||
# kubectl -n foxhunt delete pvc prometheus-data
|
||||
# kubectl -n foxhunt delete sa prometheus
|
||||
# kubectl delete clusterrole foxhunt-prometheus
|
||||
# kubectl delete clusterrolebinding foxhunt-prometheus
|
||||
# kubectl -n foxhunt delete deploy node-exporter
|
||||
# kubectl -n foxhunt delete ds node-exporter # if DaemonSet
|
||||
# kubectl -n foxhunt delete deploy kube-state-metrics
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# Prometheus Operator
|
||||
# ─────────────────────────────────────────────
|
||||
prometheusOperator:
|
||||
enabled: true
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 128Mi
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# Prometheus
|
||||
# ─────────────────────────────────────────────
|
||||
prometheus:
|
||||
prometheusSpec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
podMetadata:
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
retention: 14d
|
||||
retentionSize: 8GB
|
||||
replicas: 1
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
storageSpec:
|
||||
volumeClaimTemplate:
|
||||
spec:
|
||||
storageClassName: scw-bssd
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
|
||||
# Discover ALL ServiceMonitors/PodMonitors/PrometheusRules in the cluster,
|
||||
# not just those matching Helm release labels.
|
||||
serviceMonitorSelectorNilUsesHelmValues: false
|
||||
podMonitorSelectorNilUsesHelmValues: false
|
||||
ruleSelectorNilUsesHelmValues: false
|
||||
|
||||
# Scrape configs for targets that don't fit ServiceMonitor/PodMonitor CRDs
|
||||
additionalScrapeConfigs:
|
||||
# Argo Workflows controller (HTTPS with self-signed cert)
|
||||
- job_name: argo-workflows
|
||||
scheme: https
|
||||
tls_config:
|
||||
insecure_skip_verify: true
|
||||
static_configs:
|
||||
- targets: ['argo-workflows-workflow-controller-metrics.foxhunt.svc.cluster.local:9090']
|
||||
relabel_configs:
|
||||
- target_label: service
|
||||
replacement: argo-workflows
|
||||
|
||||
# Training workflow pods (ephemeral Argo pods on port 9094, Running only)
|
||||
- job_name: training-pods
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
namespaces:
|
||||
names: [foxhunt]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
|
||||
action: keep
|
||||
regex: training-workflow|compile-and-train
|
||||
- source_labels: [__meta_kubernetes_pod_phase]
|
||||
action: keep
|
||||
regex: Running
|
||||
- source_labels: [__meta_kubernetes_pod_ip]
|
||||
target_label: __address__
|
||||
replacement: ${1}:9094
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: pod
|
||||
- source_labels: [__meta_kubernetes_pod_label_workflows_argoproj_io_workflow]
|
||||
target_label: workflow
|
||||
|
||||
# QuestDB (ML metrics store)
|
||||
- job_name: questdb
|
||||
metrics_path: /metrics
|
||||
static_configs:
|
||||
- targets: ['questdb.foxhunt.svc.cluster.local:9003']
|
||||
relabel_configs:
|
||||
- target_label: service
|
||||
replacement: questdb
|
||||
|
||||
# Pushgateway (training job push metrics)
|
||||
- job_name: pushgateway
|
||||
honor_labels: true
|
||||
static_configs:
|
||||
- targets: ['pushgateway.foxhunt.svc.cluster.local:9091']
|
||||
|
||||
# Loki metrics
|
||||
- job_name: loki
|
||||
static_configs:
|
||||
- targets: ['loki.foxhunt.svc.cluster.local:3100']
|
||||
|
||||
# Tempo metrics
|
||||
- job_name: tempo
|
||||
static_configs:
|
||||
- targets: ['tempo.foxhunt.svc.cluster.local:3200']
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# AlertManager
|
||||
# ─────────────────────────────────────────────
|
||||
alertmanager:
|
||||
alertmanagerSpec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
replicas: 1
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
# Mount K8s Secrets into AlertManager at /etc/alertmanager/secrets/<name>/
|
||||
secrets:
|
||||
- alertmanager-mattermost
|
||||
|
||||
config:
|
||||
global:
|
||||
# Mattermost incoming webhook (Slack-compatible API).
|
||||
# Reads URL from K8s Secret: alertmanager-mattermost/webhook-url
|
||||
# Create with:
|
||||
# kubectl -n foxhunt create secret generic alertmanager-mattermost \
|
||||
# --from-literal=webhook-url='http://mattermost.foxhunt.svc.cluster.local:8065/hooks/<HOOK_ID>'
|
||||
slack_api_url_file: /etc/alertmanager/secrets/alertmanager-mattermost/webhook-url
|
||||
|
||||
route:
|
||||
receiver: default
|
||||
group_by: ['alertname', 'severity', 'service']
|
||||
group_wait: 5s
|
||||
group_interval: 30s
|
||||
repeat_interval: 4h
|
||||
|
||||
routes:
|
||||
# CRITICAL trading/risk → immediate Slack + PagerDuty
|
||||
- matchers:
|
||||
- severity="critical"
|
||||
- component=~"trading_engine|risk_management|trading|risk"
|
||||
receiver: critical-trading
|
||||
group_wait: 0s
|
||||
group_interval: 5s
|
||||
repeat_interval: 5m
|
||||
continue: true
|
||||
|
||||
# CRITICAL infrastructure → Slack critical channel
|
||||
- matchers:
|
||||
- severity="critical"
|
||||
receiver: critical-infra
|
||||
group_wait: 0s
|
||||
group_interval: 10s
|
||||
repeat_interval: 15m
|
||||
|
||||
# HIGH priority → Slack alerts channel
|
||||
- matchers:
|
||||
- severity="high"
|
||||
receiver: high-priority
|
||||
group_wait: 10s
|
||||
group_interval: 1m
|
||||
repeat_interval: 30m
|
||||
|
||||
# WARNING / MEDIUM → Slack digest
|
||||
- matchers:
|
||||
- severity=~"warning|WARNING|medium"
|
||||
receiver: warning-alerts
|
||||
group_wait: 30s
|
||||
group_interval: 5m
|
||||
repeat_interval: 4h
|
||||
|
||||
receivers:
|
||||
- name: default
|
||||
slack_configs:
|
||||
- channel: '#foxhunt-alerts'
|
||||
title: '{{ .GroupLabels.alertname }}'
|
||||
text: >-
|
||||
{{ range .Alerts }}*{{ .Labels.severity | toUpper }}*: {{ .Annotations.summary }}
|
||||
{{ end }}
|
||||
send_resolved: true
|
||||
|
||||
- name: critical-trading
|
||||
slack_configs:
|
||||
- channel: '#foxhunt-critical'
|
||||
title: 'CRITICAL: {{ .GroupLabels.alertname }}'
|
||||
text: >-
|
||||
{{ range .Alerts }}
|
||||
*Alert*: {{ .Annotations.summary }}
|
||||
*Impact*: {{ .Annotations.impact }}
|
||||
*Action*: {{ .Annotations.action }}
|
||||
{{ end }}
|
||||
color: 'danger'
|
||||
send_resolved: true
|
||||
|
||||
- name: critical-infra
|
||||
slack_configs:
|
||||
- channel: '#foxhunt-critical'
|
||||
title: 'INFRA CRITICAL: {{ .GroupLabels.alertname }}'
|
||||
text: >-
|
||||
{{ range .Alerts }}{{ .Annotations.summary }}
|
||||
{{ end }}
|
||||
color: 'danger'
|
||||
send_resolved: true
|
||||
|
||||
- name: high-priority
|
||||
slack_configs:
|
||||
- channel: '#foxhunt-alerts'
|
||||
title: 'HIGH: {{ .GroupLabels.alertname }}'
|
||||
text: >-
|
||||
{{ range .Alerts }}{{ .Annotations.summary }}
|
||||
{{ end }}
|
||||
color: 'warning'
|
||||
send_resolved: true
|
||||
|
||||
- name: warning-alerts
|
||||
slack_configs:
|
||||
- channel: '#foxhunt-alerts'
|
||||
title: 'WARNING: {{ .GroupLabels.alertname }}'
|
||||
text: >-
|
||||
{{ range .Alerts }}{{ .Annotations.summary }}
|
||||
{{ end }}
|
||||
color: '#ff9500'
|
||||
send_resolved: true
|
||||
|
||||
inhibit_rules:
|
||||
# Trading engine down → suppress all trading alerts
|
||||
- source_matchers:
|
||||
- alertname="TradingEngineDown"
|
||||
target_matchers:
|
||||
- component=~"trading_engine|trading"
|
||||
equal: ['service']
|
||||
|
||||
# Node down → suppress service-level alerts on that node
|
||||
- source_matchers:
|
||||
- alertname=~"NodeDown|KubeNodeNotReady"
|
||||
target_matchers:
|
||||
- severity=~"high|warning|medium"
|
||||
equal: ['instance']
|
||||
|
||||
# PostgreSQL down → suppress database alerts
|
||||
- source_matchers:
|
||||
- alertname="PostgreSQLDown"
|
||||
target_matchers:
|
||||
- component="database"
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# ServiceMonitors
|
||||
# ─────────────────────────────────────────────
|
||||
additionalServiceMonitors:
|
||||
# All foxhunt microservices (7 services, all expose named port "metrics")
|
||||
- name: foxhunt-services
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
namespaceSelector:
|
||||
matchNames: [foxhunt]
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 15s
|
||||
|
||||
# DCGM Exporter (GPU metrics from kube-system)
|
||||
- name: dcgm-exporter
|
||||
selector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/name
|
||||
operator: In
|
||||
values: [dcgm-exporter]
|
||||
namespaceSelector:
|
||||
matchNames: [kube-system, monitoring, foxhunt]
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 15s
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# Grafana (disabled — managed separately via grafana Helm chart)
|
||||
# ─────────────────────────────────────────────
|
||||
grafana:
|
||||
enabled: false
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# Node Exporter (replaces standalone node-exporter.yaml)
|
||||
# ─────────────────────────────────────────────
|
||||
prometheus-node-exporter:
|
||||
tolerations:
|
||||
- effect: NoSchedule
|
||||
operator: Exists
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
cpu: 50m
|
||||
memory: 32Mi
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# kube-state-metrics (replaces standalone kube-state-metrics.yaml)
|
||||
# ─────────────────────────────────────────────
|
||||
kube-state-metrics:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# Default rules (Kubernetes cluster health alerts included out-of-box)
|
||||
# ─────────────────────────────────────────────
|
||||
defaultRules:
|
||||
create: true
|
||||
rules:
|
||||
alertmanager: true
|
||||
etcd: false # managed by Scaleway Kapsule
|
||||
configReloaders: true
|
||||
general: true
|
||||
k8sContainerCpuUsageSecondsTotal: true
|
||||
k8sContainerMemoryCache: true
|
||||
k8sContainerMemoryRss: true
|
||||
k8sContainerMemorySwap: true
|
||||
k8sContainerMemoryWorkingSetBytes: true
|
||||
k8sPodOwner: true
|
||||
kubeApiserverAvailability: false # managed by Scaleway
|
||||
kubeApiserverBurnrate: false
|
||||
kubeApiserverHistogram: false
|
||||
kubeApiserverSlos: false
|
||||
kubeControllerManager: false # managed by Scaleway
|
||||
kubelet: false # kubelet metrics not accessible on Kapsule
|
||||
kubeProxy: false # managed by Scaleway
|
||||
kubePrometheusGeneral: true
|
||||
kubePrometheusNodeRecording: true
|
||||
kubernetesAbsent: true
|
||||
kubernetesApps: true
|
||||
kubernetesResources: true
|
||||
kubernetesStorage: true
|
||||
kubernetesSystem: true
|
||||
kubeScheduler: false # managed by Scaleway
|
||||
kubeStateMetrics: true
|
||||
network: true
|
||||
node: true
|
||||
nodeExporterAlerting: true
|
||||
nodeExporterRecording: true
|
||||
prometheus: true
|
||||
prometheusOperator: true
|
||||
|
||||
# ─────────────────────────────────────────────
|
||||
# Disable components managed by Scaleway Kapsule
|
||||
# ─────────────────────────────────────────────
|
||||
kubeApiServer:
|
||||
enabled: false
|
||||
kubeControllerManager:
|
||||
enabled: false
|
||||
kubeScheduler:
|
||||
enabled: false
|
||||
kubeProxy:
|
||||
enabled: false
|
||||
kubelet:
|
||||
enabled: false # kubelet metrics not accessible on Kapsule
|
||||
kubeEtcd:
|
||||
enabled: false
|
||||
@@ -1,172 +0,0 @@
|
||||
# Prometheus override values for GitLab Helm chart
|
||||
# Removes Cockpit remote_write, adds comprehensive cluster scrape configs
|
||||
#
|
||||
# Apply with:
|
||||
# helm upgrade gitlab gitlab/gitlab -n foxhunt --reuse-values \
|
||||
# -f infra/k8s/monitoring/prometheus-values.yaml
|
||||
prometheus:
|
||||
server:
|
||||
# Remove Cockpit remote_write — all metrics stay local
|
||||
remoteWrite: []
|
||||
|
||||
# Extra scrape configs for full cluster observability
|
||||
# NOTE: this is at prometheus.extraScrapeConfigs (chart root), NOT prometheus.server.extraScrapeConfigs
|
||||
extraScrapeConfigs: |
|
||||
# --- Host metrics from node-exporter ---
|
||||
- job_name: node-exporter
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
namespaces:
|
||||
names: [foxhunt]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
|
||||
action: keep
|
||||
regex: node-exporter
|
||||
- source_labels: [__meta_kubernetes_endpoints_name]
|
||||
action: keep
|
||||
regex: node-exporter
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
|
||||
# --- Kubernetes object state ---
|
||||
- job_name: kube-state-metrics
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
namespaces:
|
||||
names: [foxhunt]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]
|
||||
action: keep
|
||||
regex: kube-state-metrics
|
||||
- source_labels: [__meta_kubernetes_endpoint_port_name]
|
||||
action: keep
|
||||
regex: http
|
||||
|
||||
# --- Kubelet metrics (container resource usage) ---
|
||||
- job_name: kubelet
|
||||
scheme: https
|
||||
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||||
tls_config:
|
||||
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
insecure_skip_verify: true
|
||||
kubernetes_sd_configs:
|
||||
- role: node
|
||||
relabel_configs:
|
||||
- target_label: __address__
|
||||
replacement: kubernetes.default.svc:443
|
||||
- source_labels: [__meta_kubernetes_node_name]
|
||||
regex: (.+)
|
||||
target_label: __metrics_path__
|
||||
replacement: /api/v1/nodes/$1/proxy/metrics
|
||||
|
||||
# --- cAdvisor (container CPU, memory, network, filesystem) ---
|
||||
- job_name: cadvisor
|
||||
scheme: https
|
||||
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
||||
tls_config:
|
||||
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
||||
insecure_skip_verify: true
|
||||
kubernetes_sd_configs:
|
||||
- role: node
|
||||
relabel_configs:
|
||||
- target_label: __address__
|
||||
replacement: kubernetes.default.svc:443
|
||||
- source_labels: [__meta_kubernetes_node_name]
|
||||
regex: (.+)
|
||||
target_label: __metrics_path__
|
||||
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
|
||||
metric_relabel_configs:
|
||||
- source_labels: [namespace]
|
||||
action: keep
|
||||
regex: (foxhunt|monitoring|kube-system)
|
||||
|
||||
# --- GPU metrics from NVIDIA DCGM Exporter ---
|
||||
- job_name: dcgm-exporter
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
namespaces:
|
||||
names: [kube-system, monitoring]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_service_name]
|
||||
action: keep
|
||||
regex: .*dcgm.*
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
|
||||
# --- Foxhunt services (app metrics ports) ---
|
||||
- job_name: foxhunt-services
|
||||
kubernetes_sd_configs:
|
||||
- role: endpoints
|
||||
namespaces:
|
||||
names: [foxhunt]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_part_of]
|
||||
action: keep
|
||||
regex: foxhunt
|
||||
- source_labels: [__meta_kubernetes_endpoint_port_name]
|
||||
action: keep
|
||||
regex: metrics
|
||||
- source_labels: [__meta_kubernetes_service_name]
|
||||
target_label: service
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
target_label: namespace
|
||||
|
||||
# --- General prometheus.io annotation-based scraping ---
|
||||
- job_name: prometheus-annotated-pods
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
|
||||
action: keep
|
||||
regex: "true"
|
||||
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
|
||||
action: replace
|
||||
regex: (https?)
|
||||
target_label: __scheme__
|
||||
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
|
||||
action: replace
|
||||
regex: (.+)
|
||||
target_label: __metrics_path__
|
||||
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
|
||||
action: replace
|
||||
regex: ([^:]+)(?::\d+)?;(\d+)
|
||||
replacement: $1:$2
|
||||
target_label: __address__
|
||||
- action: labelmap
|
||||
regex: __meta_kubernetes_pod_label_(.+)
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
target_label: namespace
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: pod
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
|
||||
# --- Loki metrics ---
|
||||
- job_name: loki
|
||||
static_configs:
|
||||
- targets: ['loki.foxhunt.svc.cluster.local:3100']
|
||||
|
||||
# --- Tempo metrics ---
|
||||
- job_name: tempo
|
||||
static_configs:
|
||||
- targets: ['tempo.foxhunt.svc.cluster.local:3200']
|
||||
|
||||
# --- Promtail metrics ---
|
||||
- job_name: promtail
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
namespaces:
|
||||
names: [foxhunt]
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
|
||||
action: keep
|
||||
regex: promtail
|
||||
- source_labels: [__meta_kubernetes_pod_ip]
|
||||
target_label: __address__
|
||||
replacement: $1:9080
|
||||
|
||||
# --- Pushgateway (training job metrics) ---
|
||||
- job_name: pushgateway
|
||||
honor_labels: true
|
||||
static_configs:
|
||||
- targets: ['pushgateway.foxhunt.svc.cluster.local:9091']
|
||||
@@ -1,265 +0,0 @@
|
||||
# Promtail — ships container logs to Loki (DaemonSet on all nodes)
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: promtail
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: promtail
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: promtail
|
||||
labels:
|
||||
app.kubernetes.io/name: promtail
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["nodes", "nodes/proxy", "services", "endpoints", "pods"]
|
||||
verbs: ["get", "watch", "list"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: promtail
|
||||
labels:
|
||||
app.kubernetes.io/name: promtail
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: promtail
|
||||
namespace: foxhunt
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: promtail
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: promtail-config
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: promtail
|
||||
data:
|
||||
promtail.yaml: |
|
||||
server:
|
||||
http_listen_port: 9080
|
||||
positions:
|
||||
filename: /run/promtail/positions.yaml
|
||||
clients:
|
||||
- url: http://loki.foxhunt.svc.cluster.local:3100/loki/api/v1/push
|
||||
scrape_configs:
|
||||
# Foxhunt application pods (structured Rust logs)
|
||||
- job_name: foxhunt-pods
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
action: keep
|
||||
regex: foxhunt
|
||||
# Drop noisy infra pods (gitlab internals, prometheus, grafana)
|
||||
- source_labels: [__meta_kubernetes_pod_label_app]
|
||||
action: drop
|
||||
regex: (gitlab|prometheus|grafana|promtail)
|
||||
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
|
||||
action: drop
|
||||
regex: (promtail|node-exporter|kube-state-metrics)
|
||||
# Map to containerd log path
|
||||
- source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
|
||||
separator: /
|
||||
target_label: __path__
|
||||
replacement: /var/log/pods/*$1/*.log
|
||||
# Standard K8s metadata labels
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
target_label: namespace
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: pod
|
||||
- source_labels: [__meta_kubernetes_pod_container_name]
|
||||
target_label: container
|
||||
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
|
||||
target_label: app
|
||||
- source_labels: [__meta_kubernetes_pod_label_foxhunt_job__type]
|
||||
target_label: job_type
|
||||
- source_labels: [__meta_kubernetes_pod_label_foxhunt_model]
|
||||
target_label: model
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
pipeline_stages:
|
||||
- json:
|
||||
expressions:
|
||||
level: level
|
||||
message: fields.message
|
||||
target: target
|
||||
timestamp: timestamp
|
||||
- labels:
|
||||
level:
|
||||
target:
|
||||
- match:
|
||||
selector: '{level=""}'
|
||||
stages:
|
||||
- regex:
|
||||
expression: '(?P<level>TRACE|DEBUG|INFO|WARN|ERROR)'
|
||||
- labels:
|
||||
level:
|
||||
|
||||
# Kapsule system pods (kube-system, kube-node-lease, kube-public)
|
||||
- job_name: kapsule-system
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
action: keep
|
||||
regex: (kube-system|kube-node-lease|kube-public)
|
||||
# Map to containerd log path
|
||||
- source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
|
||||
separator: /
|
||||
target_label: __path__
|
||||
replacement: /var/log/pods/*$1/*.log
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
target_label: namespace
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: pod
|
||||
- source_labels: [__meta_kubernetes_pod_container_name]
|
||||
target_label: container
|
||||
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
|
||||
target_label: app
|
||||
- source_labels: [__meta_kubernetes_pod_label_app]
|
||||
target_label: kube_app
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
pipeline_stages:
|
||||
- match:
|
||||
selector: '{namespace="kube-system"}'
|
||||
stages:
|
||||
- regex:
|
||||
expression: '(?P<level>TRACE|DEBUG|INFO|WARN|ERROR|warning|error|fatal)'
|
||||
- labels:
|
||||
level:
|
||||
|
||||
# GitLab CI runner logs (foxhunt namespace, runner pods)
|
||||
- job_name: gitlab-runners
|
||||
kubernetes_sd_configs:
|
||||
- role: pod
|
||||
relabel_configs:
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
action: keep
|
||||
regex: foxhunt
|
||||
- source_labels: [__meta_kubernetes_pod_label_app]
|
||||
action: keep
|
||||
regex: gitlab-runner
|
||||
- source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
|
||||
separator: /
|
||||
target_label: __path__
|
||||
replacement: /var/log/pods/*$1/*.log
|
||||
- source_labels: [__meta_kubernetes_namespace]
|
||||
target_label: namespace
|
||||
- source_labels: [__meta_kubernetes_pod_name]
|
||||
target_label: pod
|
||||
- source_labels: [__meta_kubernetes_pod_container_name]
|
||||
target_label: container
|
||||
- replacement: gitlab-runner
|
||||
target_label: app
|
||||
- source_labels: [__meta_kubernetes_pod_node_name]
|
||||
target_label: node
|
||||
pipeline_stages:
|
||||
- regex:
|
||||
expression: '(?P<level>TRACE|DEBUG|INFO|WARN|ERROR|WARNING)'
|
||||
- labels:
|
||||
level:
|
||||
|
||||
# Kubernetes node-level logs (kubelet, containerd)
|
||||
- job_name: kapsule-node-logs
|
||||
static_configs:
|
||||
- targets: [localhost]
|
||||
labels:
|
||||
job: kapsule-nodes
|
||||
__path__: /var/log/syslog
|
||||
- targets: [localhost]
|
||||
labels:
|
||||
job: kapsule-nodes
|
||||
__path__: /var/log/kern.log
|
||||
- targets: [localhost]
|
||||
labels:
|
||||
job: kapsule-nodes
|
||||
__path__: /var/log/kubelet.log
|
||||
relabel_configs:
|
||||
- source_labels: [__path__]
|
||||
regex: .*/(.+)\.log
|
||||
target_label: log_type
|
||||
pipeline_stages:
|
||||
- regex:
|
||||
expression: '(?P<level>TRACE|DEBUG|INFO|WARN|ERROR|warning|error|fatal)'
|
||||
- labels:
|
||||
level:
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
name: promtail
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: promtail
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: promtail
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: promtail
|
||||
spec:
|
||||
serviceAccountName: promtail
|
||||
tolerations:
|
||||
- effect: NoSchedule
|
||||
operator: Exists
|
||||
containers:
|
||||
- name: promtail
|
||||
image: grafana/promtail:3.4.2
|
||||
args: ["-config.file=/etc/promtail/promtail.yaml"]
|
||||
ports:
|
||||
- containerPort: 9080
|
||||
name: http
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/promtail
|
||||
- name: varlog
|
||||
mountPath: /var/log
|
||||
readOnly: true
|
||||
- name: containers
|
||||
mountPath: /var/lib/docker/containers
|
||||
readOnly: true
|
||||
- name: pods
|
||||
mountPath: /var/log/pods
|
||||
readOnly: true
|
||||
- name: run
|
||||
mountPath: /run/promtail
|
||||
env:
|
||||
- name: HOSTNAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: spec.nodeName
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 256Mi
|
||||
securityContext:
|
||||
readOnlyRootFilesystem: true
|
||||
allowPrivilegeEscalation: false
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: promtail-config
|
||||
- name: varlog
|
||||
hostPath:
|
||||
path: /var/log
|
||||
- name: containers
|
||||
hostPath:
|
||||
path: /var/lib/docker/containers
|
||||
- name: pods
|
||||
hostPath:
|
||||
path: /var/log/pods
|
||||
- name: run
|
||||
emptyDir: {}
|
||||
@@ -1,61 +0,0 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: pushgateway
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: pushgateway
|
||||
app.kubernetes.io/part-of: monitoring
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: pushgateway
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: pushgateway
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: infra
|
||||
containers:
|
||||
- name: pushgateway
|
||||
image: prom/pushgateway:v1.11.0
|
||||
args:
|
||||
- "--persistence.interval=5m"
|
||||
ports:
|
||||
- containerPort: 9091
|
||||
name: http
|
||||
resources:
|
||||
requests:
|
||||
cpu: 50m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 128Mi
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /-/healthy
|
||||
port: 9091
|
||||
initialDelaySeconds: 5
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /-/ready
|
||||
port: 9091
|
||||
initialDelaySeconds: 5
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: pushgateway
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: pushgateway
|
||||
spec:
|
||||
type: ClusterIP
|
||||
ports:
|
||||
- port: 9091
|
||||
targetPort: 9091
|
||||
name: http
|
||||
selector:
|
||||
app.kubernetes.io/name: pushgateway
|
||||
@@ -1,250 +0,0 @@
|
||||
# PVC Auto-Expand — expands PVCs automatically when usage exceeds 85%
|
||||
# Uses K8s API to list PVCs and exec df inside pods to check usage.
|
||||
# Runs every 15 minutes. Apply: kubectl apply -f pvc-autoscaler.yaml
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: pvc-autoscaler
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: pvc-autoscaler
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: Role
|
||||
metadata:
|
||||
name: pvc-autoscaler
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: pvc-autoscaler
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["persistentvolumeclaims"]
|
||||
verbs: ["get", "list", "patch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods"]
|
||||
verbs: ["get", "list"]
|
||||
- apiGroups: ["apps"]
|
||||
resources: ["deployments", "statefulsets"]
|
||||
verbs: ["get", "list"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: pvc-autoscaler-nodes
|
||||
labels:
|
||||
app.kubernetes.io/name: pvc-autoscaler
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["nodes"]
|
||||
verbs: ["list"]
|
||||
- apiGroups: [""]
|
||||
resources: ["nodes/proxy"]
|
||||
verbs: ["get"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: RoleBinding
|
||||
metadata:
|
||||
name: pvc-autoscaler
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: pvc-autoscaler
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: pvc-autoscaler
|
||||
namespace: foxhunt
|
||||
roleRef:
|
||||
kind: Role
|
||||
name: pvc-autoscaler
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: pvc-autoscaler-nodes
|
||||
labels:
|
||||
app.kubernetes.io/name: pvc-autoscaler
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: pvc-autoscaler
|
||||
namespace: foxhunt
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: pvc-autoscaler-nodes
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: pvc-autoscaler-script
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: pvc-autoscaler
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
data:
|
||||
autoscale.py: |
|
||||
#!/usr/bin/env python3
|
||||
"""PVC auto-expand via kubelet stats API (no Prometheus dependency)."""
|
||||
import json, os, sys, ssl, urllib.request
|
||||
from datetime import datetime, timezone
|
||||
|
||||
THRESHOLD = int(os.environ.get("PVC_USAGE_THRESHOLD", "85"))
|
||||
GROW_PERCENT = int(os.environ.get("PVC_GROW_PERCENT", "50"))
|
||||
MAX_SIZE_GI = int(os.environ.get("PVC_MAX_SIZE_GI", "100"))
|
||||
NAMESPACE = os.environ.get("NAMESPACE", "foxhunt")
|
||||
|
||||
TOKEN_PATH = "/var/run/secrets/kubernetes.io/serviceaccount/token"
|
||||
CA_PATH = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
|
||||
K8S_API = "https://kubernetes.default.svc"
|
||||
|
||||
def k8s(path, method="GET", data=None):
|
||||
token = open(TOKEN_PATH).read().strip()
|
||||
ctx = ssl.create_default_context(cafile=CA_PATH)
|
||||
body = json.dumps(data).encode() if data else None
|
||||
req = urllib.request.Request(f"{K8S_API}{path}", data=body, method=method)
|
||||
req.add_header("Authorization", f"Bearer {token}")
|
||||
if data:
|
||||
req.add_header("Content-Type", "application/merge-patch+json")
|
||||
return json.loads(urllib.request.urlopen(req, context=ctx, timeout=30).read())
|
||||
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
print(f"=== PVC Auto-Expand {now} ===")
|
||||
print(f"Threshold: {THRESHOLD}% | Grow: +{GROW_PERCENT}% | Max: {MAX_SIZE_GI}Gi")
|
||||
sys.stdout.flush()
|
||||
|
||||
# Step 1: Get all nodes and their kubelet stats via K8s proxy API
|
||||
nodes = k8s("/api/v1/nodes")
|
||||
pvc_usage = {} # pvc_name -> (used_bytes, capacity_bytes)
|
||||
|
||||
for node in nodes["items"]:
|
||||
node_name = node["metadata"]["name"]
|
||||
try:
|
||||
stats = k8s(f"/api/v1/nodes/{node_name}/proxy/stats/summary")
|
||||
for pod in stats.get("pods", []):
|
||||
if pod.get("podRef", {}).get("namespace") != NAMESPACE:
|
||||
continue
|
||||
for vol in pod.get("volume", []):
|
||||
pvc_ref = vol.get("pvcRef")
|
||||
if not pvc_ref:
|
||||
continue
|
||||
pvc_name = pvc_ref["name"]
|
||||
used = vol.get("usedBytes", 0)
|
||||
cap = vol.get("capacityBytes", 1)
|
||||
pvc_usage[pvc_name] = (used, cap)
|
||||
except Exception as e:
|
||||
print(f" WARN: could not get stats from node {node_name}: {e}")
|
||||
sys.stdout.flush()
|
||||
|
||||
if not pvc_usage:
|
||||
print(" No PVC stats found")
|
||||
sys.exit(0)
|
||||
|
||||
# Step 2: Check each PVC and expand if needed
|
||||
expanded = 0
|
||||
for pvc_name, (used, cap) in sorted(pvc_usage.items()):
|
||||
pct = 100 * used / cap if cap > 0 else 0
|
||||
used_gi = used / (1024**3)
|
||||
cap_gi = cap / (1024**3)
|
||||
print(f" {pvc_name}: {pct:.1f}% ({used_gi:.1f}G / {cap_gi:.1f}G)")
|
||||
|
||||
if pct >= THRESHOLD:
|
||||
print(f" >> {pvc_name} exceeds {THRESHOLD}%")
|
||||
try:
|
||||
pvc_obj = k8s(f"/api/v1/namespaces/{NAMESPACE}/persistentvolumeclaims/{pvc_name}")
|
||||
sc = pvc_obj.get("spec", {}).get("storageClassName", "")
|
||||
# Only expand if StorageClass supports it
|
||||
if sc:
|
||||
try:
|
||||
sc_obj = k8s(f"/apis/storage.k8s.io/v1/storageclasses/{sc}")
|
||||
if not sc_obj.get("allowVolumeExpansion"):
|
||||
print(f" SKIP: StorageClass {sc} does not allow expansion")
|
||||
continue
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
status_storage = pvc_obj.get("status", {}).get("capacity", {}).get("storage", "0Gi")
|
||||
current_gi = int(status_storage.replace("Gi", ""))
|
||||
spec_storage = pvc_obj["spec"]["resources"]["requests"]["storage"]
|
||||
spec_gi = int(spec_storage.replace("Gi", ""))
|
||||
|
||||
new_gi = current_gi + max(current_gi * GROW_PERCENT // 100, 1)
|
||||
if new_gi > MAX_SIZE_GI:
|
||||
print(f" SKIP: target {new_gi}Gi > max {MAX_SIZE_GI}Gi")
|
||||
continue
|
||||
if spec_gi >= new_gi:
|
||||
print(f" SKIP: spec={spec_gi}Gi >= target={new_gi}Gi (resize pending)")
|
||||
continue
|
||||
|
||||
patch = {"spec": {"resources": {"requests": {"storage": f"{new_gi}Gi"}}}}
|
||||
k8s(f"/api/v1/namespaces/{NAMESPACE}/persistentvolumeclaims/{pvc_name}",
|
||||
method="PATCH", data=patch)
|
||||
print(f" EXPANDED: {current_gi}Gi -> {new_gi}Gi")
|
||||
expanded += 1
|
||||
except Exception as e:
|
||||
print(f" FAILED: {e}", file=sys.stderr)
|
||||
sys.stdout.flush()
|
||||
|
||||
print(f"=== Done ({expanded} expanded) ===")
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: pvc-autoscaler
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: pvc-autoscaler
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
schedule: "*/15 * * * *"
|
||||
successfulJobsHistoryLimit: 1
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
activeDeadlineSeconds: 120
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: pvc-autoscaler
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
serviceAccountName: pvc-autoscaler
|
||||
restartPolicy: OnFailure
|
||||
containers:
|
||||
- name: autoscaler
|
||||
image: python:3.12-alpine
|
||||
command: ["python3", "-u", "/scripts/autoscale.py"]
|
||||
env:
|
||||
- name: PVC_USAGE_THRESHOLD
|
||||
value: "85"
|
||||
- name: PVC_GROW_PERCENT
|
||||
value: "50"
|
||||
- name: PVC_MAX_SIZE_GI
|
||||
value: "100"
|
||||
- name: NAMESPACE
|
||||
value: "foxhunt"
|
||||
volumeMounts:
|
||||
- name: script
|
||||
mountPath: /scripts
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
volumes:
|
||||
- name: script
|
||||
configMap:
|
||||
name: pvc-autoscaler-script
|
||||
items:
|
||||
- key: autoscale.py
|
||||
path: autoscale.py
|
||||
defaultMode: 0755
|
||||
@@ -1,239 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: scaleway-billing
|
||||
namespace: foxhunt
|
||||
type: Opaque
|
||||
stringData:
|
||||
SCW_SECRET_KEY: "84fa878b-81c2-401a-8aef-22996d30e9ea"
|
||||
SCW_ORG_ID: "a55098a3-2cf6-4f73-b011-fff5a299ed94"
|
||||
SCW_PROJECT_ID: "c293eb98-228d-427d-9b16-f0941f3f2adb"
|
||||
SCW_CLUSTER_ID: "34a1e3c4-ac35-48c8-ab49-5f6ec4df32c1"
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: scaleway-billing-script
|
||||
namespace: foxhunt
|
||||
data:
|
||||
scrape.py: |
|
||||
#!/usr/bin/env python3
|
||||
"""Scaleway infrastructure scraper → Prometheus pushgateway."""
|
||||
import json, os, sys, urllib.request, urllib.error, re
|
||||
|
||||
def sanitize(s):
|
||||
"""Sanitize label values for Prometheus text format."""
|
||||
return str(s).replace("\\", "\\\\").replace('"', '\\"').replace("\n", "\\n")
|
||||
|
||||
PUSHGATEWAY = os.environ.get("PUSHGATEWAY_URL", "http://pushgateway.foxhunt.svc.cluster.local:9091")
|
||||
SECRET = os.environ["SCW_SECRET_KEY"]
|
||||
ORG = os.environ["SCW_ORG_ID"]
|
||||
PROJECT = os.environ["SCW_PROJECT_ID"]
|
||||
CLUSTER = os.environ["SCW_CLUSTER_ID"]
|
||||
|
||||
SCW_BILLING = "https://api.scaleway.com/billing/v2beta1"
|
||||
SCW_INSTANCE = "https://api.scaleway.com/instance/v1/zones/fr-par-2"
|
||||
SCW_K8S = "https://api.scaleway.com/k8s/v1/regions/fr-par"
|
||||
SCW_BLOCK = "https://api.scaleway.com/block/v1alpha1/zones/fr-par-2"
|
||||
SCW_IPAM = "https://api.scaleway.com/ipam/v1/regions/fr-par"
|
||||
|
||||
def fetch(url):
|
||||
req = urllib.request.Request(url)
|
||||
req.add_header("X-Auth-Token", SECRET)
|
||||
try:
|
||||
return json.loads(urllib.request.urlopen(req, timeout=30).read())
|
||||
except Exception as e:
|
||||
print(f"WARN: {url}: {e}", file=sys.stderr)
|
||||
return {}
|
||||
|
||||
# ── Fetch all data (dynamic — auto-discovers any infra changes) ──
|
||||
billing = fetch(f"{SCW_BILLING}/consumptions?organization_id={ORG}")
|
||||
instances = fetch(f"{SCW_INSTANCE}/servers?project={PROJECT}&per_page=50")
|
||||
pools = fetch(f"{SCW_K8S}/clusters/{CLUSTER}/pools")
|
||||
volumes = fetch(f"{SCW_BLOCK}/volumes?project_id={PROJECT}&per_page=100")
|
||||
ips = fetch(f"{SCW_IPAM}/ips?project_id={PROJECT}&is_ipv6=false&per_page=100")
|
||||
|
||||
L = []
|
||||
|
||||
# ── Billing ──
|
||||
total = 0
|
||||
cats = {}
|
||||
products = {}
|
||||
for c in billing.get("consumptions", []):
|
||||
cat = c.get("category_name", "other").lower().replace(" ", "_").replace("&", "and")
|
||||
product = c.get("product_name", "unknown").lower().replace(" ", "_").replace("-", "_")
|
||||
sku = c.get("sku", "")
|
||||
val = c.get("value", {})
|
||||
amount = int(val.get("units", 0) or 0) + int(val.get("nanos", 0) or 0) / 1e9
|
||||
total += amount
|
||||
cats[cat] = cats.get(cat, 0) + amount
|
||||
products[(cat, product, sku)] = products.get((cat, product, sku), 0) + amount
|
||||
|
||||
L.append("# HELP scaleway_billing_total_eur Total current month billing in EUR")
|
||||
L.append("# TYPE scaleway_billing_total_eur gauge")
|
||||
L.append(f"scaleway_billing_total_eur {total:.4f}")
|
||||
L.append("# HELP scaleway_billing_consumption_eur Billing by category in EUR")
|
||||
L.append("# TYPE scaleway_billing_consumption_eur gauge")
|
||||
for cat, amt in sorted(cats.items(), key=lambda x: -x[1]):
|
||||
L.append(f'scaleway_billing_consumption_eur{{category="{sanitize(cat)}"}} {amt:.4f}')
|
||||
L.append("# HELP scaleway_billing_product_eur Billing by product in EUR")
|
||||
L.append("# TYPE scaleway_billing_product_eur gauge")
|
||||
for (cat, product, sku), amt in sorted(products.items(), key=lambda x: -x[1]):
|
||||
if amt >= 0.01:
|
||||
L.append(f'scaleway_billing_product_eur{{category="{sanitize(cat)}",product="{sanitize(product)}",sku="{sanitize(sku)}"}} {amt:.4f}')
|
||||
|
||||
# ── Instances (dynamic — auto-discovers new/removed instances) ──
|
||||
L.append("# HELP scaleway_instance_info Instance metadata")
|
||||
L.append("# TYPE scaleway_instance_info gauge")
|
||||
L.append("# HELP scaleway_instance_running Instance running state (1=running)")
|
||||
L.append("# TYPE scaleway_instance_running gauge")
|
||||
type_counts = {}
|
||||
for s in instances.get("servers", []):
|
||||
name = s.get("name", "?")
|
||||
itype = s.get("commercial_type", "?")
|
||||
state = s.get("state", "?")
|
||||
tags = dict(t.split("=", 1) for t in s.get("tags", []) if "=" in t)
|
||||
pool = tags.get("pool-name", "unknown")
|
||||
L.append(f'scaleway_instance_info{{name="{sanitize(name)}",type="{sanitize(itype)}",pool="{sanitize(pool)}",state="{sanitize(state)}"}} 1')
|
||||
L.append(f'scaleway_instance_running{{name="{sanitize(name)}",type="{sanitize(itype)}",pool="{sanitize(pool)}"}} {1 if state == "running" else 0}')
|
||||
type_counts[itype] = type_counts.get(itype, 0) + 1
|
||||
L.append("# HELP scaleway_instances_total Total instances by type")
|
||||
L.append("# TYPE scaleway_instances_total gauge")
|
||||
for t, c in type_counts.items():
|
||||
L.append(f'scaleway_instances_total{{type="{sanitize(t)}"}} {c}')
|
||||
|
||||
# ── K8s Pools (dynamic — auto-discovers pool changes/autoscaler config) ──
|
||||
L.append("# HELP scaleway_pool_size Current pool node count")
|
||||
L.append("# TYPE scaleway_pool_size gauge")
|
||||
L.append("# HELP scaleway_pool_min Minimum pool size")
|
||||
L.append("# TYPE scaleway_pool_min gauge")
|
||||
L.append("# HELP scaleway_pool_max Maximum pool size")
|
||||
L.append("# TYPE scaleway_pool_max gauge")
|
||||
L.append("# HELP scaleway_pool_autoscaling Pool autoscaling enabled")
|
||||
L.append("# TYPE scaleway_pool_autoscaling gauge")
|
||||
L.append("# HELP scaleway_pool_root_volume_bytes Pool root volume size")
|
||||
L.append("# TYPE scaleway_pool_root_volume_bytes gauge")
|
||||
for p in pools.get("pools", []):
|
||||
name = p.get("name", "?")
|
||||
ntype = p.get("node_type", "?")
|
||||
status = p.get("status", "?")
|
||||
lb = f'pool="{sanitize(name)}",type="{sanitize(ntype)}",status="{sanitize(status)}"'
|
||||
L.append(f"scaleway_pool_size{{{lb}}} {p.get('size', 0)}")
|
||||
L.append(f"scaleway_pool_min{{{lb}}} {p.get('min_size', 0)}")
|
||||
L.append(f"scaleway_pool_max{{{lb}}} {p.get('max_size', 0)}")
|
||||
L.append(f"scaleway_pool_autoscaling{{{lb}}} {1 if p.get('autoscaling') else 0}")
|
||||
L.append(f"scaleway_pool_root_volume_bytes{{{lb}}} {p.get('root_volume_size', 0)}")
|
||||
|
||||
# ── Block Storage (dynamic — auto-discovers volume additions/removals) ──
|
||||
L.append("# HELP scaleway_volume_size_bytes Volume size in bytes")
|
||||
L.append("# TYPE scaleway_volume_size_bytes gauge")
|
||||
L.append("# HELP scaleway_volume_attached Volume attachment state (1=in_use, 0=available)")
|
||||
L.append("# TYPE scaleway_volume_attached gauge")
|
||||
total_bytes = attached_bytes = detached_bytes = 0
|
||||
vol_count = attached_count = detached_count = 0
|
||||
for v in volumes.get("volumes", []):
|
||||
vid = v.get("id", "?")[:36]
|
||||
name = v.get("name", "?")[:60]
|
||||
size = v.get("size", 0)
|
||||
state = v.get("status", "unknown")
|
||||
attached = 1 if state == "in_use" else 0
|
||||
total_bytes += size; vol_count += 1
|
||||
if attached:
|
||||
attached_bytes += size; attached_count += 1
|
||||
else:
|
||||
detached_bytes += size; detached_count += 1
|
||||
L.append(f'scaleway_volume_size_bytes{{id="{sanitize(vid)}",name="{sanitize(name)}",state="{sanitize(state)}"}} {size}')
|
||||
L.append(f'scaleway_volume_attached{{id="{sanitize(vid)}",name="{sanitize(name)}"}} {attached}')
|
||||
L.append("# HELP scaleway_volumes_total_bytes Total block storage bytes")
|
||||
L.append("# TYPE scaleway_volumes_total_bytes gauge")
|
||||
L.append(f"scaleway_volumes_total_bytes {total_bytes}")
|
||||
L.append("# HELP scaleway_volumes_attached_bytes Attached volume bytes")
|
||||
L.append("# TYPE scaleway_volumes_attached_bytes gauge")
|
||||
L.append(f"scaleway_volumes_attached_bytes {attached_bytes}")
|
||||
L.append("# HELP scaleway_volumes_detached_bytes Detached (wasted) volume bytes")
|
||||
L.append("# TYPE scaleway_volumes_detached_bytes gauge")
|
||||
L.append(f"scaleway_volumes_detached_bytes {detached_bytes}")
|
||||
L.append("# HELP scaleway_volumes_count Volume counts")
|
||||
L.append("# TYPE scaleway_volumes_count gauge")
|
||||
L.append(f'scaleway_volumes_count{{state="total"}} {vol_count}')
|
||||
L.append(f'scaleway_volumes_count{{state="attached"}} {attached_count}')
|
||||
L.append(f'scaleway_volumes_count{{state="detached"}} {detached_count}')
|
||||
|
||||
# ── IPs (dynamic) ──
|
||||
L.append("# HELP scaleway_ips_total Total allocated IPs")
|
||||
L.append("# TYPE scaleway_ips_total gauge")
|
||||
L.append(f'scaleway_ips_total {len(ips.get("ips", []))}')
|
||||
|
||||
# ── Push to pushgateway ──
|
||||
metrics = "\n".join(L) + "\n"
|
||||
count = sum(1 for line in L if line.startswith("scaleway_"))
|
||||
print(f"Generated {count} metrics ({len(metrics)} bytes)", file=sys.stderr)
|
||||
|
||||
# Validate: no empty lines between metrics (pushgateway is strict)
|
||||
metrics = re.sub(r'\n{2,}', '\n', metrics)
|
||||
|
||||
url = f"{PUSHGATEWAY}/metrics/job/scaleway_billing"
|
||||
req = urllib.request.Request(url, data=metrics.encode(), method="PUT")
|
||||
req.add_header("Content-Type", "text/plain; version=0.0.4")
|
||||
req.add_header("Connection", "close")
|
||||
try:
|
||||
resp = urllib.request.urlopen(req, timeout=30)
|
||||
print(f"OK: pushed {count} metrics (HTTP {resp.status})")
|
||||
except Exception as e:
|
||||
print(f"WARN: PUT failed ({e}), trying POST...", file=sys.stderr)
|
||||
req2 = urllib.request.Request(url, data=metrics.encode(), method="POST")
|
||||
req2.add_header("Content-Type", "text/plain; version=0.0.4")
|
||||
req2.add_header("Connection", "close")
|
||||
try:
|
||||
resp = urllib.request.urlopen(req2, timeout=30)
|
||||
print(f"OK: pushed {count} metrics via POST (HTTP {resp.status})")
|
||||
except Exception as e2:
|
||||
print(f"ERROR: push failed: PUT={e}, POST={e2}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: scaleway-billing-scraper
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
schedule: "0 * * * *"
|
||||
successfulJobsHistoryLimit: 1
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
activeDeadlineSeconds: 120
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: scaleway-billing
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
restartPolicy: OnFailure
|
||||
containers:
|
||||
- name: scraper
|
||||
image: python:3.12-alpine
|
||||
command: ["python3", "/scripts/scrape.py"]
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: scaleway-billing
|
||||
volumeMounts:
|
||||
- name: script
|
||||
mountPath: /scripts
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 64Mi
|
||||
volumes:
|
||||
- name: script
|
||||
configMap:
|
||||
name: scaleway-billing-script
|
||||
items:
|
||||
- key: scrape.py
|
||||
path: scrape.py
|
||||
defaultMode: 0755
|
||||
@@ -1,43 +0,0 @@
|
||||
# ServiceMonitors for Foxhunt services and GPU metrics
|
||||
# Applied separately from the Helm chart (additionalServiceMonitors not rendered by chart)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: foxhunt-services
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
namespaceSelector:
|
||||
matchNames: [foxhunt]
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 15s
|
||||
# Override job label to "foxhunt-services" (matching old standalone config)
|
||||
# so all existing dashboard queries continue to work
|
||||
relabelings:
|
||||
- targetLabel: job
|
||||
replacement: foxhunt-services
|
||||
---
|
||||
# DCGM Exporter (GPU metrics from nvidia-gpu-operator DaemonSet)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: dcgm-exporter
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
selector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/name
|
||||
operator: In
|
||||
values: [dcgm-exporter]
|
||||
namespaceSelector:
|
||||
matchNames: [kube-system, monitoring, foxhunt]
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 15s
|
||||
@@ -1,139 +0,0 @@
|
||||
# Grafana Tempo — lightweight distributed tracing (single-binary mode)
|
||||
# Receives OTLP traces from services, queryable via Grafana
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: tempo-config
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: tempo
|
||||
data:
|
||||
tempo.yaml: |
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: "0.0.0.0:4317"
|
||||
http:
|
||||
endpoint: "0.0.0.0:4318"
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
local:
|
||||
path: /var/tempo/traces
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 168h # 7 days
|
||||
metrics_generator:
|
||||
registry:
|
||||
external_labels:
|
||||
source: tempo
|
||||
cluster: foxhunt
|
||||
storage:
|
||||
path: /var/tempo/generator/wal
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: tempo
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: tempo
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tempo
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: tempo
|
||||
spec:
|
||||
nodeSelector:
|
||||
k8s.scaleway.com/pool-name: platform
|
||||
securityContext:
|
||||
fsGroup: 10001
|
||||
runAsUser: 10001
|
||||
runAsNonRoot: true
|
||||
containers:
|
||||
- name: tempo
|
||||
image: grafana/tempo:2.7.1
|
||||
args: ["-config.file=/etc/tempo/tempo.yaml"]
|
||||
ports:
|
||||
- containerPort: 3200
|
||||
name: http
|
||||
- containerPort: 4317
|
||||
name: otlp-grpc
|
||||
- containerPort: 4318
|
||||
name: otlp-http
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/tempo
|
||||
- name: data
|
||||
mountPath: /var/tempo
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 512Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 2Gi
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: 3200
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 10
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: 3200
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
name: tempo-config
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: tempo-data
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: tempo-data
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: scw-bssd
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: tempo
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/name: tempo
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: tempo
|
||||
ports:
|
||||
- port: 3200
|
||||
targetPort: 3200
|
||||
name: http
|
||||
- port: 4317
|
||||
targetPort: 4317
|
||||
name: otlp-grpc
|
||||
- port: 4318
|
||||
targetPort: 4318
|
||||
name: otlp-http
|
||||
@@ -1,7 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
app.kubernetes.io/managed-by: kubectl
|
||||
@@ -1,64 +0,0 @@
|
||||
# AlertManager network policy
|
||||
# Ingress: Prometheus (alert delivery), Grafana (alert UI queries)
|
||||
# Egress: Mattermost webhooks (cluster-internal), DNS
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: alertmanager
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alertmanager
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
# Prometheus pushes alerts and scrapes metrics (9093 API, 8080 config-reloader)
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9093
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
# Grafana queries AlertManager API for alert state/silences
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9093
|
||||
egress:
|
||||
# DNS resolution
|
||||
- to:
|
||||
- namespaceSelector: {}
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
# Mattermost incoming webhook (cluster-internal)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8065
|
||||
# External HTTPS (PagerDuty, email, future integrations)
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
@@ -1,26 +0,0 @@
|
||||
# Allow all foxhunt + CI pods to reach CoreDNS (kube-system) for name resolution.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: allow-dns
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/part-of
|
||||
operator: In
|
||||
values: [foxhunt, foxhunt-ci]
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
@@ -1,76 +0,0 @@
|
||||
# Allow Prometheus to scrape metrics ports on all foxhunt + CI pods.
|
||||
# Also allow all foxhunt + CI pods to push traces to Tempo.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: allow-monitoring-scrape
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/part-of
|
||||
operator: In
|
||||
values: ["foxhunt", "foxhunt-ci"]
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
# External monitoring namespace (if any)
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: monitoring
|
||||
# GitLab Prometheus in foxhunt namespace
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- protocol: TCP
|
||||
port: 9003
|
||||
- protocol: TCP
|
||||
port: 9091
|
||||
- protocol: TCP
|
||||
port: 9092
|
||||
- protocol: TCP
|
||||
port: 9093
|
||||
- protocol: TCP
|
||||
port: 9094
|
||||
- protocol: TCP
|
||||
port: 9095
|
||||
- protocol: TCP
|
||||
port: 9096
|
||||
- protocol: TCP
|
||||
port: 9097
|
||||
- protocol: TCP
|
||||
port: 9098
|
||||
egress:
|
||||
# Tempo OTLP gRPC
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: foxhunt
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tempo
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 4317
|
||||
# Kubernetes API (for kube-state-metrics list/watch)
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 10.32.0.0/16
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- protocol: TCP
|
||||
port: 6443
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 172.16.0.0/16
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6443
|
||||
@@ -1,110 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: api
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: api
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
# Tailscale ingress proxy
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tailscale-gitlab-proxy
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50051
|
||||
egress:
|
||||
# Postgres
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: postgres
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
# Redis
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6379
|
||||
# Downstream gRPC services
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: trading-service
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50051
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: backtesting-service
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50053
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: ml-training-service
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50053
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: trading-agent-service
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50055
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: broker-gateway
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50056
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: data-acquisition-service
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50057
|
||||
# Prometheus (metric scraping from monitoring handler)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
# Minio (initContainer binary fetch)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
# Kubernetes API (pod listing for Pods cockpit)
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 10.32.0.0/12
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 172.16.0.0/16
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6443
|
||||
@@ -1,42 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: backtesting-service
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: backtesting-service
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: api
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50053
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: postgres
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6379
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
@@ -1,52 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: broker-gateway
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: broker-gateway
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: api
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: trading-service
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50056
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: ib-gateway
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 4002
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: postgres
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6379
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
@@ -1,18 +0,0 @@
|
||||
# CI executor pods need broad egress: git clone (GitLab webservice), container
|
||||
# registries, kubectl (K8s API), MinIO S3, and external tool downloads.
|
||||
# These pods are ephemeral (minutes) — fine-grained egress adds little value.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: ci-egress
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/part-of: foxhunt-ci
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- {}
|
||||
@@ -1,53 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: data-acquisition-service
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: data-acquisition-service
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: api
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50057
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: postgres
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6379
|
||||
# External HTTPS (Databento, Benzinga APIs)
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
@@ -1,60 +0,0 @@
|
||||
# Egress policy for data-download jobs (MBP-10 download, etc.)
|
||||
# Allows: MinIO (upload), Databento API (HTTPS download)
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: data-download-job
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
foxhunt/job-type: data-download
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# MinIO — binary fetch + data upload
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
# Databento API (HTTPS)
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
- 100.64.0.0/10
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
---
|
||||
# Egress policy for data-sync jobs (MinIO → PVC sync)
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: data-sync-job
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
foxhunt/job-type: data-sync
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# MinIO — data sync source
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
@@ -1,16 +0,0 @@
|
||||
# Default deny all ingress and egress in foxhunt namespace.
|
||||
# Every service must declare its own NetworkPolicy to communicate.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny-all
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
@@ -1,24 +0,0 @@
|
||||
# Allow all foxhunt pods to fetch binaries from GitLab Generic Package Registry.
|
||||
# initContainers in services and training jobs use curl to download release binaries
|
||||
# from gitlab-webservice-default:8181 (GitLab API).
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: gitlab-packages
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: webservice
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8181
|
||||
@@ -1,44 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: ib-gateway
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: ib-gateway
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: trading-service
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: broker-gateway
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: multistrat-rebalancer
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 4002
|
||||
- protocol: TCP
|
||||
port: 4004
|
||||
egress:
|
||||
# IBKR external servers
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
except:
|
||||
- 10.0.0.0/8
|
||||
- 172.16.0.0/12
|
||||
- 192.168.0.0/16
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- protocol: TCP
|
||||
port: 4000
|
||||
- protocol: TCP
|
||||
port: 4001
|
||||
@@ -1,124 +0,0 @@
|
||||
# Postgres: accepts connections from foxhunt app pods, GitLab, Grafana, and Gitea
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: postgres
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: postgres
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/part-of
|
||||
operator: In
|
||||
values: ["foxhunt", "argo-workflows"]
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
release: gitlab
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
- podSelector: # Gitea (Phase 2B) reuses the in-cluster postgres
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
---
|
||||
# Redis: accepts connections from foxhunt app pods + GitLab (Helm release)
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: redis
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/part-of
|
||||
operator: In
|
||||
values: ["foxhunt"]
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
release: gitlab
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6379
|
||||
---
|
||||
# Minio: accepts connections from foxhunt app pods, GitLab, and init jobs
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: minio
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/part-of
|
||||
operator: In
|
||||
values: ["foxhunt", "foxhunt-ci", "argo-workflows"]
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
release: gitlab
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio-init
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
---
|
||||
# QuestDB: accepts ILP writes from trading-service + Prometheus metrics scrape
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: questdb
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: questdb
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: trading-service
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
foxhunt/job-type: training
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9009
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9003
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8812
|
||||
@@ -1,67 +0,0 @@
|
||||
# Network policies for maintenance CronJobs (postgres-backup, pvc-autoscaler)
|
||||
# These jobs need egress to internal services but no internet access.
|
||||
# Apply: kubectl apply -f maintenance-jobs.yaml
|
||||
---
|
||||
# Postgres backup job: egress to postgres (5432) + minio (9000)
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: postgres-backup
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: postgres-backup
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# DNS
|
||||
- ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
# Postgres
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: postgres
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
# MinIO (backup upload)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
---
|
||||
# PVC autoscaler job: egress to K8s API + kubelet stats
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: pvc-autoscaler
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: pvc-autoscaler
|
||||
policyTypes:
|
||||
- Egress
|
||||
egress:
|
||||
# DNS
|
||||
- ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
# Kubernetes API server (HTTPS)
|
||||
- ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
# Kubelet stats API (port 10250)
|
||||
- ports:
|
||||
- protocol: TCP
|
||||
port: 10250
|
||||
@@ -1,42 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: mattermost
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
# HTTP from Tailscale proxy + Argo workflow pods (webhook notifications)
|
||||
- ports:
|
||||
- port: 8065
|
||||
protocol: TCP
|
||||
egress:
|
||||
# DNS
|
||||
- ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
# PostgreSQL
|
||||
- ports:
|
||||
- port: 5432
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: postgres
|
||||
# Stalwart SMTP (for email notifications, wired later)
|
||||
- ports:
|
||||
- port: 25
|
||||
protocol: TCP
|
||||
to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: stalwart
|
||||
@@ -1,49 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: ml-training-service
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: ml-training-service
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: api
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50053
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: postgres
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6379
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
# Kubernetes API (for job creation — ml-training-service has RBAC for batch/jobs)
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 0.0.0.0/0
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
@@ -1,23 +0,0 @@
|
||||
# CiliumNetworkPolicy: allow Prometheus to reach hostNetwork pods (node-exporter).
|
||||
#
|
||||
# Vanilla K8s NetworkPolicy ipBlock rules don't match traffic to hostNetwork
|
||||
# pods because Cilium assigns them the 'host'/'remote-node' identity, not 'world'.
|
||||
# This CiliumNetworkPolicy uses toEntities to cover that gap.
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: prometheus-host-access
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
egress:
|
||||
- toEntities:
|
||||
- host
|
||||
- remote-node
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: "9100"
|
||||
protocol: TCP
|
||||
@@ -1,188 +0,0 @@
|
||||
# Prometheus egress: K8s API (service discovery), foxhunt services, node-exporter,
|
||||
# kube-state-metrics, DCGM exporter, Argo workflow controller, pushgateway,
|
||||
# Loki, Tempo, QuestDB, and AlertManager.
|
||||
# Applies to operator-managed Prometheus (kube-prometheus-stack).
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: prometheus
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: prometheus
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
# Grafana queries Prometheus on port 9090
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: grafana
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
egress:
|
||||
# DNS resolution (required for additionalScrapeConfigs with hostnames)
|
||||
- to:
|
||||
- namespaceSelector: {}
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
|
||||
# Kubernetes API server (required for kubernetes_sd_configs service discovery)
|
||||
# ClusterIP 10.32.0.1:443 NATs to node IP 172.16.0.0/16:6443
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 10.32.0.0/16
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
- protocol: TCP
|
||||
port: 6443
|
||||
|
||||
# Scrape foxhunt service metrics ports (9003, 9090-9098)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/part-of
|
||||
operator: In
|
||||
values: ["foxhunt", "foxhunt-ci"]
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9003
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
- protocol: TCP
|
||||
port: 9091
|
||||
- protocol: TCP
|
||||
port: 9092
|
||||
- protocol: TCP
|
||||
port: 9093
|
||||
- protocol: TCP
|
||||
port: 9094
|
||||
- protocol: TCP
|
||||
port: 9095
|
||||
- protocol: TCP
|
||||
port: 9096
|
||||
- protocol: TCP
|
||||
port: 9097
|
||||
- protocol: TCP
|
||||
port: 9098
|
||||
|
||||
# kube-state-metrics (Helm sub-chart, port 8080)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: kube-state-metrics
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
|
||||
# Prometheus operator (container port 10250)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: kube-prometheus-stack-prometheus-operator
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 10250
|
||||
|
||||
# Alertmanager (port 9093 for API, port 8080 for config-reloader sidecar)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: alertmanager
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9093
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
|
||||
# Loki metrics (port 3100) and Tempo metrics (port 3200)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: loki
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3100
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: tempo
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 3200
|
||||
|
||||
# QuestDB metrics
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: questdb
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9003
|
||||
|
||||
# Node-exporter (hostNetwork), kubelet, and K8s API on node IPs
|
||||
- to:
|
||||
- ipBlock:
|
||||
cidr: 172.16.0.0/16
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6443
|
||||
- protocol: TCP
|
||||
port: 9100
|
||||
- protocol: TCP
|
||||
port: 9400
|
||||
- protocol: TCP
|
||||
port: 10250 # kubelet metrics (cAdvisor, resource usage)
|
||||
|
||||
# kube-system: CoreDNS metrics, kube-state-metrics, node-exporter, DCGM
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8080
|
||||
- protocol: TCP
|
||||
port: 8443
|
||||
- protocol: TCP
|
||||
port: 9100
|
||||
- protocol: TCP
|
||||
port: 9153 # CoreDNS metrics
|
||||
- protocol: TCP
|
||||
port: 9400
|
||||
|
||||
# monitoring namespace: DCGM exporter (GPU metrics)
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: monitoring
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9400
|
||||
|
||||
# Pushgateway (training pods push metrics here)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: pushgateway
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9091
|
||||
|
||||
# Argo Workflows controller metrics
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: argo-workflows-workflow-controller
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9090
|
||||
@@ -1,49 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: stalwart
|
||||
namespace: foxhunt
|
||||
labels:
|
||||
app.kubernetes.io/part-of: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: stalwart
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
# SMTP from cluster (Mattermost internal relay)
|
||||
- ports:
|
||||
- port: 25
|
||||
protocol: TCP
|
||||
from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: mattermost
|
||||
# IMAPS from Tailscale proxy
|
||||
- ports:
|
||||
- port: 993
|
||||
protocol: TCP
|
||||
# HTTPS admin panel from Tailscale proxy
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
egress:
|
||||
# DNS
|
||||
- ports:
|
||||
- port: 53
|
||||
protocol: UDP
|
||||
- port: 53
|
||||
protocol: TCP
|
||||
# HTTPS out — webadmin bundle + spam model downloads from GitHub
|
||||
- ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
# No outbound SMTP — internal-only for now.
|
||||
# To add SCW TEM relay later:
|
||||
# - ports:
|
||||
# - port: 465
|
||||
# protocol: TCP
|
||||
# - port: 587
|
||||
# protocol: TCP
|
||||
@@ -1,49 +0,0 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: trading-agent-service
|
||||
namespace: foxhunt
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: trading-agent-service
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
- from:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: api
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50055
|
||||
egress:
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: trading-service
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50051
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: postgres
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: redis
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 6379
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: minio
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user