foxhunt/docs/guides/LOCAL_CI_PIPELINE_GUIDE.md

# Local CI/CD Pipeline Testing Guide

**Created**: 2025-10-29
**Purpose**: Simulate GitLab CI/CD pipeline locally before deployment
**Script**: `scripts/local_ci_pipeline.sh`

---

## Overview

The local CI/CD pipeline simulator replicates GitLab CI/CD stages for testing Docker image builds and deployments before pushing to GitLab. This ensures:

1. **Build validation**: Docker image builds successfully with CUDA 12.4.1 + cuDNN 9
2. **Test validation**: GLIBC 2.35, CUDA libraries, entrypoints work correctly
3. **Push readiness**: Image can be pushed to Docker Hub registry

---

## Quick Start

### 1. Full Pipeline (Build + Test + Push)
```bash
./scripts/local_ci_pipeline.sh
```

### 2. Test Build Only (Skip Push)
```bash
./scripts/local_ci_pipeline.sh --skip-push
```

### 3. Dry-Run (Show Commands)
```bash
./scripts/local_ci_pipeline.sh --dry-run
```

### 4. Verbose Output
```bash
./scripts/local_ci_pipeline.sh --verbose --skip-push
```

---

## Pipeline Stages

### Stage 0: Pre-Flight Checks (🔍)
**Duration**: ~3 seconds
**Validates**:
- Docker daemon running
- Docker BuildKit available
- Docker Hub authentication (if pushing)
- Dockerfile exists (`Dockerfile.runpod`)
- Git repository status

**Output**:
```
✓ All required commands available
✓ Docker daemon running
✓ Docker Hub authenticated
✓ Dockerfile found: Dockerfile.runpod
✓ Git repository validated
```

### Stage 1: Build (🔨)
**Duration**: ~2-3 minutes
**Executes**:
```bash
docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .
```

**Validates**:
- Dockerfile syntax
- CUDA 12.4.1 + cuDNN 9 base image pull
- Image layers build successfully
- Final image exists

**Output**:
```
✓ Docker image built successfully: 4.80 GB
```

### Stage 2: Test (🧪)
**Duration**: ~10-20 seconds
**Validates**:

#### Test 1: GLIBC Version
```bash
docker run --rm jgrusewski/foxhunt:latest ldd --version
```
**Expected**: GLIBC 2.35 (Ubuntu 22.04)

#### Test 2: CUDA Libraries
**Required Libraries**:
- `libcuda.so.1` (CUDA driver)
- `libcurand.so.10` (CUDA random)
- `libcublas.so.12` (CUDA BLAS)
- `libcublasLt.so.12` (CUDA BLAS Light)
- `libcudnn.so.9` (cuDNN 9)

#### Test 3: nvidia-smi (Optional)
**Checks**: GPU driver availability on host
**Note**: Warns if not available (normal for CI/CD without GPU)

#### Test 4: Binary GLIBC Dependencies
**Validates**: System binaries link against GLIBC 2.35

#### Test 5: Entrypoint Scripts
**Checks**:
- `/entrypoint.sh` exists and executable
- `/entrypoint-generic.sh` exists and executable
- `--help` command works

**Output**:
```
✓ GLIBC 2.35 validated
✓ CUDA libraries validated
✓ nvidia-smi available, driver version: 550.120
✓ Binary GLIBC dependencies validated
✓ Entrypoint script exists and is executable
```

### Stage 3: Push (🚀)
**Duration**: ~1-5 minutes (depends on network)
**Executes**:
```bash
docker push jgrusewski/foxhunt:latest
```

**Validates**:
- Docker Hub authentication
- Image push succeeds
- Warns to set repository to PRIVATE

**Output**:
```
✓ Image pushed successfully: jgrusewski/foxhunt:latest
⚠ WARNING: Set Docker Hub repository to PRIVATE if not already
```

---

## Command Options

| Option | Description | Use Case |
|--------|-------------|----------|
| `--dry-run` | Show commands without executing | Preview pipeline actions |
| `--skip-push` | Skip push stage | Local testing only |
| `--verbose` | Enable verbose output | Debugging build issues |
| `--help` | Show help message | View usage instructions |

---

## Exit Codes

| Code | Meaning | Action |
|------|---------|--------|
| 0 | Success | All stages passed |
| 1 | Stage failure | Check error output and logs |

---

## Troubleshooting

### Error: Docker daemon not running
```bash
# Start Docker daemon
sudo systemctl start docker

# Verify
docker info
```

### Error: Docker Hub authentication failed
```bash
# Login to Docker Hub
docker login

# Enter credentials for jgrusewski account
```

### Error: Docker build failed
```bash
# Check build logs
cat /tmp/docker_build.log

# Verify Dockerfile exists
ls -la Dockerfile.runpod

# Check disk space
df -h
```

### Error: GLIBC version mismatch
```bash
# Expected: GLIBC 2.35 (Ubuntu 22.04)
# If mismatch, check Dockerfile base image
docker run --rm jgrusewski/foxhunt:latest ldd --version
```

### Error: CUDA libraries missing
```bash
# Check CUDA installation in image
docker run --rm jgrusewski/foxhunt:latest sh -c "ls /usr/local/cuda/lib64/"

# Verify cuDNN
docker run --rm jgrusewski/foxhunt:latest sh -c "ldconfig -p | grep cudnn"
```

### Warning: nvidia-smi not available
```
⚠ WARNING: This is normal for CI/CD environments without GPU
⚠ WARNING: GPU will be available in Runpod deployment
```

---

## Pipeline Metrics

### Typical Run Times

| Stage | Duration | Size |
|-------|----------|------|
| Pre-flight | ~3s | - |
| Build | ~2-3 min | 4.8 GB |
| Test | ~10-20s | - |
| Push | ~1-5 min | 4.8 GB |
| **Total** | **~4-9 min** | **4.8 GB** |

### Resource Requirements

| Resource | Minimum | Recommended |
|----------|---------|-------------|
| Disk Space | 10 GB | 20 GB |
| RAM | 4 GB | 8 GB |
| Docker | 20.10+ | 24.0+ |

---

## Integration with GitLab CI/CD

### .gitlab-ci.yml Template

```yaml
stages:
  - build
  - test
  - push

variables:
  DOCKER_IMAGE: jgrusewski/foxhunt:latest
  DOCKERFILE: Dockerfile.runpod

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker build -f $DOCKERFILE -t $DOCKER_IMAGE .
  only:
    - main

test:
  stage: test
  image: docker:latest
  services:
    - docker:dind
  script:
    # GLIBC validation
    - docker run --rm $DOCKER_IMAGE ldd --version | grep "2.35"

    # CUDA library checks
    - docker run --rm $DOCKER_IMAGE sh -c "ldconfig -p | grep libcuda.so.1"
    - docker run --rm $DOCKER_IMAGE sh -c "ldconfig -p | grep libcurand.so.10"
    - docker run --rm $DOCKER_IMAGE sh -c "ldconfig -p | grep libcublas.so.12"
    - docker run --rm $DOCKER_IMAGE sh -c "ldconfig -p | grep libcublasLt.so.12"
    - docker run --rm $DOCKER_IMAGE sh -c "ldconfig -p | grep libcudnn.so.9"

    # Entrypoint validation
    - docker run --rm $DOCKER_IMAGE sh -c "[ -x /entrypoint.sh ]"
    - docker run --rm $DOCKER_IMAGE sh -c "[ -x /entrypoint-generic.sh ]"
  only:
    - main

push:
  stage: push
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker login -u $DOCKER_HUB_USER -p $DOCKER_HUB_TOKEN
    - docker push $DOCKER_IMAGE
  only:
    - main
```

### GitLab CI/CD Variables

Add these variables to GitLab project settings:

| Variable | Type | Value | Protected |
|----------|------|-------|-----------|
| `DOCKER_HUB_USER` | Variable | `jgrusewski` | Yes |
| `DOCKER_HUB_TOKEN` | Variable | `<token>` | Yes |

---

## Next Steps

After successful local pipeline run:

1. **Verify Docker Hub**: Check image at https://hub.docker.com/r/jgrusewski/foxhunt
2. **Set PRIVATE**: Update repository visibility in Docker Hub settings
3. **Test Runpod**: Deploy to Runpod with volume mount
4. **Validate Training**: Run hyperopt demos with GPU
5. **Push to GitLab**: Commit `.gitlab-ci.yml` and trigger CI/CD

---

## Additional Resources

- **Dockerfile**: `/home/jgrusewski/Work/foxhunt/Dockerfile.runpod`
- **Entrypoint**: `/home/jgrusewski/Work/foxhunt/entrypoint.sh`
- **Build Logs**: `/tmp/docker_build.log`
- **Push Logs**: `/tmp/docker_push.log`
- **Docker Hub**: https://hub.docker.com/r/jgrusewski/foxhunt
- **Runpod**: https://www.runpod.io/console/pods

---

## Example Output

### Successful Pipeline Run

```
========================================
🚀 LOCAL CI/CD PIPELINE SIMULATOR
========================================

ℹ Simulating GitLab CI/CD pipeline locally
ℹ Image: jgrusewski/foxhunt:latest
ℹ Dockerfile: Dockerfile.runpod

========================================
🔍 STAGE 0: PRE-FLIGHT CHECKS
========================================

✓ All required commands available
✓ Docker daemon running
✓ Docker Hub authenticated
✓ Dockerfile found: Dockerfile.runpod
✓ Git repository validated
⏱ Pre-flight checks completed in 0m 3s

========================================
🔨 STAGE 1: BUILD
========================================

✓ Docker image built successfully: 4.80 GB
⏱ Build completed in 2m 34s

========================================
🧪 STAGE 2: TEST
========================================

✓ GLIBC 2.35 validated
✓ CUDA libraries validated
✓ nvidia-smi available, driver version: 550.120
✓ Binary GLIBC dependencies validated
✓ Entrypoint script exists and is executable
⏱ Test completed in 0m 18s

========================================
🚀 STAGE 3: PUSH
========================================

✓ Image pushed successfully: jgrusewski/foxhunt:latest
⚠ WARNING: Set Docker Hub repository to PRIVATE if not already
⏱ Push completed in 3m 12s

========================================
✅ PIPELINE COMPLETE
========================================

ℹ Pipeline Summary:
  • Image: jgrusewski/foxhunt:latest
  • Dockerfile: Dockerfile.runpod
  • Mode: Full execution
  • Push: Completed

✓ Total pipeline time: 6m 7s

ℹ GitLab CI/CD readiness: ✅
  • Build stage: Validated
  • Test stage: Validated
  • Push stage: Validated
```

---

## Summary

The `local_ci_pipeline.sh` script provides:

✅ **Complete CI/CD simulation** - All GitLab stages tested locally
✅ **Fast iteration** - Catch issues before GitLab deployment
✅ **GLIBC validation** - Ensures 2.35 compatibility
✅ **CUDA validation** - Verifies CUDA 12.4.1 + cuDNN 9
✅ **Entrypoint testing** - Validates volume mount architecture
✅ **Color-coded output** - Easy to read success/error states
✅ **Stage timing** - Performance metrics for each stage
✅ **Error handling** - Exit on first failure (CI/CD behavior)

**Total Time**: 4-9 minutes (vs. 10-15 min on GitLab CI/CD)
**Cost**: $0 (vs. GitLab CI/CD minutes)
**Reliability**: 100% local control before cloud deployment