1ad79493dc
- Expand training-data-pvc from 10Gi → 200Gi (OHLCV + MBP-10 + headroom) - Add data-sync-job: rclone sync from MinIO → PVC (delta-only, fast repeats) - Add sync-training-data init container to training job template (auto-syncs data from MinIO before each training run) - Add training-job + data-sync-job NetworkPolicies (MinIO, Tempo, Pushgateway) - Add app.kubernetes.io/part-of: foxhunt to training pod template - download_baseline: add --parallel N flag for concurrent Databento downloads - download_baseline: delete local files after MinIO upload (saves scratch space) - Bump download job scratch volume to 100Gi Architecture: Databento → MinIO (source of truth) → PVC (local cache). First sync pulls everything; subsequent runs only sync deltas (seconds). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
16 lines
309 B
YAML
16 lines
309 B
YAML
apiVersion: v1
|
|
kind: PersistentVolumeClaim
|
|
metadata:
|
|
name: training-data-pvc
|
|
namespace: foxhunt
|
|
labels:
|
|
app.kubernetes.io/name: training-data
|
|
app.kubernetes.io/part-of: foxhunt
|
|
spec:
|
|
accessModes:
|
|
- ReadWriteOnce
|
|
resources:
|
|
requests:
|
|
storage: 200Gi
|
|
storageClassName: scw-bssd
|