---
name: "MinIO Binary Deploy"
description: "Deploy Rust service/training binaries to the Kapsule cluster via MinIO S3. Use when deploying any foxhunt binary (trading_service, api_gateway, train_baseline_supervised, etc.) to the Scaleway Kapsule cluster."
---

# MinIO S3 Binary Deployment

## What This Skill Does

Deploys locally-built Rust binaries to MinIO S3 (`foxhunt-binaries` bucket), then triggers a pod rollout so service initContainers fetch the new binary. No Docker image builds, no PVCs, no binary-writer pods.

## When to Use

- Deploying any service or training binary after local code changes
- User says "deploy", "push to cluster", "update the service", "minio deploy"
- After building a binary in release mode

## Architecture

All foxhunt services and training jobs use a shared deployment pattern:
- **Base images**: `foxhunt-runtime:latest` (services) / `foxhunt-training-runtime:latest` (training)
- **Binary source**: MinIO S3 bucket `foxhunt-binaries` at `minio.foxhunt.svc.cluster.local:9000`
- **Fetch method**: rclone initContainer fetches binary to emptyDir at pod startup
- **External access**: `s3.fxhnt.ai` via tailscale proxy (for external GPU nodes)

## Service → Binary → Deployment Mapping

| Service (Cargo crate)          | Binary name                  | K8s Deployment name         | S3 path                              |
|-------------------------------|------------------------------|-----------------------------|---------------------------------------|
| `trading_service`             | `trading_service`            | `trading-service`           | `services/trading_service`            |
| `api_gateway`                 | `api_gateway`                | `api-gateway`               | `services/api_gateway`               |
| `broker_gateway_service`      | `broker_gateway_service`     | `broker-gateway`            | `services/broker_gateway_service`    |
| `ml_training_service`         | `ml_training_service`        | `ml-training-service`       | `services/ml_training_service`       |
| `backtesting_service`         | `backtesting_service`        | `backtesting-service`       | `services/backtesting_service`       |
| `trading_agent_service`       | `trading_agent_service`      | `trading-agent-service`     | `services/trading_agent_service`     |
| `data_acquisition_service`    | `data_acquisition_service`   | `data-acquisition-service`  | `services/data_acquisition_service`  |
| `web-gateway`                 | `web-gateway`                | `web-gateway`               | `services/web-gateway`               |

### Training Binaries

| Binary                            | S3 path                                  |
|-----------------------------------|------------------------------------------|
| `train_baseline_rl`              | `training/train_baseline_rl`             |
| `train_baseline_supervised`      | `training/train_baseline_supervised`     |
| `evaluate_baseline`              | `training/evaluate_baseline`             |
| `hyperopt_baseline_rl`           | `training/hyperopt_baseline_rl`          |
| `hyperopt_baseline_supervised`   | `training/hyperopt_baseline_supervised`  |

**Note**: `web-gateway` is in `crates/web-gateway/`, all others are in `services/`.

## Procedure

### Step 1: Build the binary (release mode)

```bash
SQLX_OFFLINE=true cargo build --release -p <crate_name>
```

The binary will be at `target/release/<binary_name>`.

### Step 2: Upload to MinIO via rclone

```bash
# Service binary:
rclone copyto target/release/<binary_name> \
  :s3:foxhunt-binaries/services/<binary_name> \
  --s3-provider=Minio \
  --s3-endpoint=https://s3.fxhnt.ai \
  --s3-access-key-id="${MINIO_ACCESS_KEY}" \
  --s3-secret-access-key="${MINIO_SECRET_KEY}" \
  --s3-no-check-bucket

# Training binary:
rclone copyto target/release/<binary_name> \
  :s3:foxhunt-binaries/training/<binary_name> \
  --s3-provider=Minio \
  --s3-endpoint=https://s3.fxhnt.ai \
  --s3-access-key-id="${MINIO_ACCESS_KEY}" \
  --s3-secret-access-key="${MINIO_SECRET_KEY}" \
  --s3-no-check-bucket
```

**Alternative using aws-cli:**

```bash
export AWS_ACCESS_KEY_ID="${MINIO_ACCESS_KEY}"
export AWS_SECRET_ACCESS_KEY="${MINIO_SECRET_KEY}"

# Service binary:
aws s3 cp target/release/<binary_name> \
  s3://foxhunt-binaries/services/<binary_name> \
  --endpoint-url https://s3.fxhnt.ai

# Training binary:
aws s3 cp target/release/<binary_name> \
  s3://foxhunt-binaries/training/<binary_name> \
  --endpoint-url https://s3.fxhnt.ai
```

### Step 3: Restart the deployment to fetch new binary

```bash
# Service:
kubectl rollout restart deployment/<deployment_name> -n foxhunt
kubectl rollout status deployment/<deployment_name> -n foxhunt --timeout=120s

# Training jobs pick up the new binary automatically on next job creation.
```

## Deploying Multiple Services at Once

```bash
# Build all changed services
SQLX_OFFLINE=true cargo build --release -p trading_service -p api_gateway

# Upload all
export AWS_ACCESS_KEY_ID="${MINIO_ACCESS_KEY}"
export AWS_SECRET_ACCESS_KEY="${MINIO_SECRET_KEY}"
S3="aws s3 --endpoint-url https://s3.fxhnt.ai"
$S3 cp target/release/trading_service s3://foxhunt-binaries/services/trading_service
$S3 cp target/release/api_gateway s3://foxhunt-binaries/services/api_gateway

# Restart all affected deployments
kubectl rollout restart deployment/trading-service deployment/api-gateway -n foxhunt
```

## Web Dashboard Static Assets

```bash
cd web-dashboard && npm run build
aws s3 cp dist/ s3://foxhunt-binaries/static/ --recursive \
  --endpoint-url https://s3.fxhnt.ai
kubectl rollout restart deployment/web-gateway -n foxhunt
```

## Troubleshooting

### InitContainer fails with connection error
- MinIO must be running: `kubectl get pod -n foxhunt -l app.kubernetes.io/name=minio`
- Check MinIO CA cert: `kubectl get configmap minio-ca-cert -n foxhunt`
- From inside cluster, `--no-check-certificate` bypasses self-signed CA

### Service CrashLoopBackOff after deploy
- Binary may have been built for the wrong target. All cluster nodes are x86_64 Linux.
- Check logs: `kubectl logs -n foxhunt deployment/<deployment_name> -c fetch-binary`
- Check main container: `kubectl logs -n foxhunt deployment/<deployment_name> --tail=50`

### Upload fails from local machine
- Ensure `s3.fxhnt.ai` resolves (DNS via Scaleway, proxied through tailscale)
- Ensure you're on the tailnet: `tailscale status`
- Verify credentials: `aws s3 ls --endpoint-url https://s3.fxhnt.ai s3://foxhunt-binaries/`

## Important Constraints

- **No Docker builds needed**: The runtime images are generic and pre-built.
- **Namespace**: Always `foxhunt`.
- **MinIO bucket**: `foxhunt-binaries` (services/ and training/ prefixes).
- **External endpoint**: `s3.fxhnt.ai` (via tailscale proxy).
- **Internal endpoint**: `minio.foxhunt.svc.cluster.local:9000` (used by initContainers).
- **SQLX_OFFLINE=true**: Required for builds (no PostgreSQL running locally).
- **Credentials**: `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` (same as in k8s secret `minio-credentials`).
