foxhunt/docs/guides/LOCAL_CI_PIPELINE_GUIDE.md

Local CI/CD Pipeline Testing Guide

Created: 2025-10-29 Purpose: Simulate GitLab CI/CD pipeline locally before deployment Script: scripts/local_ci_pipeline.sh

---

Overview

The local CI/CD pipeline simulator replicates GitLab CI/CD stages for testing Docker image builds and deployments before pushing to GitLab. This ensures:

1. Build validation: Docker image builds successfully with CUDA 12.4.1 + cuDNN 9
2. Test validation: GLIBC 2.35, CUDA libraries, entrypoints work correctly
3. Push readiness: Image can be pushed to Docker Hub registry

---

Quick Start

1. Full Pipeline (Build + Test + Push)

./scripts/local_ci_pipeline.sh

2. Test Build Only (Skip Push)

./scripts/local_ci_pipeline.sh --skip-push

3. Dry-Run (Show Commands)

./scripts/local_ci_pipeline.sh --dry-run

4. Verbose Output

./scripts/local_ci_pipeline.sh --verbose --skip-push

---

Pipeline Stages

Stage 0: Pre-Flight Checks (🔍)

Duration: ~3 seconds Validates:

• Docker daemon running
• Docker BuildKit available
• Docker Hub authentication (if pushing)
• Dockerfile exists (Dockerfile.runpod)
• Git repository status

Output:

✓ All required commands available
✓ Docker daemon running
✓ Docker Hub authenticated
✓ Dockerfile found: Dockerfile.runpod
✓ Git repository validated

Stage 1: Build (🔨)

Duration: ~2-3 minutes Executes:

docker build -f Dockerfile.runpod -t jgrusewski/foxhunt:latest .

Validates:

• Dockerfile syntax
• CUDA 12.4.1 + cuDNN 9 base image pull
• Image layers build successfully
• Final image exists

Output:

✓ Docker image built successfully: 4.80 GB

Stage 2: Test (🧪)

Duration: ~10-20 seconds Validates:

Test 1: GLIBC Version

docker run --rm jgrusewski/foxhunt:latest ldd --version

Expected: GLIBC 2.35 (Ubuntu 22.04)

Test 2: CUDA Libraries

Required Libraries:

• libcuda.so.1 (CUDA driver)
• libcurand.so.10 (CUDA random)
• libcublas.so.12 (CUDA BLAS)
• libcublasLt.so.12 (CUDA BLAS Light)
• libcudnn.so.9 (cuDNN 9)

Test 3: nvidia-smi (Optional)

Checks: GPU driver availability on host Note: Warns if not available (normal for CI/CD without GPU)

Test 4: Binary GLIBC Dependencies

Validates: System binaries link against GLIBC 2.35

Test 5: Entrypoint Scripts

Checks:

• /entrypoint.sh exists and executable
• /entrypoint-generic.sh exists and executable
• --help command works

Output:

✓ GLIBC 2.35 validated
✓ CUDA libraries validated
✓ nvidia-smi available, driver version: 550.120
✓ Binary GLIBC dependencies validated
✓ Entrypoint script exists and is executable

Stage 3: Push (🚀)

Duration: ~1-5 minutes (depends on network) Executes:

docker push jgrusewski/foxhunt:latest

Validates:

• Docker Hub authentication
• Image push succeeds
• Warns to set repository to PRIVATE

Output:

✓ Image pushed successfully: jgrusewski/foxhunt:latest
⚠ WARNING: Set Docker Hub repository to PRIVATE if not already

---

Command Options

Option	Description	Use Case
--dry-run	Show commands without executing	Preview pipeline actions
--skip-push	Skip push stage	Local testing only
--verbose	Enable verbose output	Debugging build issues
--help	Show help message	View usage instructions

---

Exit Codes

Code	Meaning	Action
0	Success	All stages passed
1	Stage failure	Check error output and logs

---

Troubleshooting

Error: Docker daemon not running

# Start Docker daemon
sudo systemctl start docker

# Verify
docker info

Error: Docker Hub authentication failed

# Login to Docker Hub
docker login

# Enter credentials for jgrusewski account

Error: Docker build failed

# Check build logs
cat /tmp/docker_build.log

# Verify Dockerfile exists
ls -la Dockerfile.runpod

# Check disk space
df -h

Error: GLIBC version mismatch

# Expected: GLIBC 2.35 (Ubuntu 22.04)
# If mismatch, check Dockerfile base image
docker run --rm jgrusewski/foxhunt:latest ldd --version

Error: CUDA libraries missing

# Check CUDA installation in image
docker run --rm jgrusewski/foxhunt:latest sh -c "ls /usr/local/cuda/lib64/"

# Verify cuDNN
docker run --rm jgrusewski/foxhunt:latest sh -c "ldconfig -p | grep cudnn"

Warning: nvidia-smi not available

⚠ WARNING: This is normal for CI/CD environments without GPU
⚠ WARNING: GPU will be available in Runpod deployment

---

Pipeline Metrics

Typical Run Times

Stage	Duration	Size
Pre-flight	~3s	-
Build	~2-3 min	4.8 GB
Test	~10-20s	-
Push	~1-5 min	4.8 GB
Total	~4-9 min	4.8 GB

Resource Requirements

Resource	Minimum	Recommended
Disk Space	10 GB	20 GB
RAM	4 GB	8 GB
Docker	20.10+	24.0+

---

Integration with GitLab CI/CD

.gitlab-ci.yml Template

stages:
  - build
  - test
  - push

variables:
  DOCKER_IMAGE: jgrusewski/foxhunt:latest
  DOCKERFILE: Dockerfile.runpod

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker build -f $DOCKERFILE -t $DOCKER_IMAGE .
  only:
    - main

test:
  stage: test
  image: docker:latest
  services:
    - docker:dind
  script:
    # GLIBC validation
    - docker run --rm $DOCKER_IMAGE ldd --version | grep "2.35"

    # CUDA library checks
    - docker run --rm $DOCKER_IMAGE sh -c "ldconfig -p | grep libcuda.so.1"
    - docker run --rm $DOCKER_IMAGE sh -c "ldconfig -p | grep libcurand.so.10"
    - docker run --rm $DOCKER_IMAGE sh -c "ldconfig -p | grep libcublas.so.12"
    - docker run --rm $DOCKER_IMAGE sh -c "ldconfig -p | grep libcublasLt.so.12"
    - docker run --rm $DOCKER_IMAGE sh -c "ldconfig -p | grep libcudnn.so.9"

    # Entrypoint validation
    - docker run --rm $DOCKER_IMAGE sh -c "[ -x /entrypoint.sh ]"
    - docker run --rm $DOCKER_IMAGE sh -c "[ -x /entrypoint-generic.sh ]"
  only:
    - main

push:
  stage: push
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker login -u $DOCKER_HUB_USER -p $DOCKER_HUB_TOKEN
    - docker push $DOCKER_IMAGE
  only:
    - main

GitLab CI/CD Variables

Add these variables to GitLab project settings:

Variable	Type	Value	Protected
DOCKER_HUB_USER	Variable	jgrusewski	Yes
DOCKER_HUB_TOKEN	Variable	<token>	Yes

---

Next Steps

After successful local pipeline run:

1. Verify Docker Hub: Check image at https://hub.docker.com/r/jgrusewski/foxhunt
2. Set PRIVATE: Update repository visibility in Docker Hub settings
3. Test Runpod: Deploy to Runpod with volume mount
4. Validate Training: Run hyperopt demos with GPU
5. Push to GitLab: Commit .gitlab-ci.yml and trigger CI/CD

---

Additional Resources

• Dockerfile: /home/jgrusewski/Work/foxhunt/Dockerfile.runpod
• Entrypoint: /home/jgrusewski/Work/foxhunt/entrypoint.sh
• Build Logs: /tmp/docker_build.log
• Push Logs: /tmp/docker_push.log
• Docker Hub: https://hub.docker.com/r/jgrusewski/foxhunt
• Runpod: https://www.runpod.io/console/pods

---

Example Output

Successful Pipeline Run

========================================
🚀 LOCAL CI/CD PIPELINE SIMULATOR
========================================

ℹ Simulating GitLab CI/CD pipeline locally
ℹ Image: jgrusewski/foxhunt:latest
ℹ Dockerfile: Dockerfile.runpod

========================================
🔍 STAGE 0: PRE-FLIGHT CHECKS
========================================

✓ All required commands available
✓ Docker daemon running
✓ Docker Hub authenticated
✓ Dockerfile found: Dockerfile.runpod
✓ Git repository validated
⏱ Pre-flight checks completed in 0m 3s

========================================
🔨 STAGE 1: BUILD
========================================

✓ Docker image built successfully: 4.80 GB
⏱ Build completed in 2m 34s

========================================
🧪 STAGE 2: TEST
========================================

✓ GLIBC 2.35 validated
✓ CUDA libraries validated
✓ nvidia-smi available, driver version: 550.120
✓ Binary GLIBC dependencies validated
✓ Entrypoint script exists and is executable
⏱ Test completed in 0m 18s

========================================
🚀 STAGE 3: PUSH
========================================

✓ Image pushed successfully: jgrusewski/foxhunt:latest
⚠ WARNING: Set Docker Hub repository to PRIVATE if not already
⏱ Push completed in 3m 12s

========================================
✅ PIPELINE COMPLETE
========================================

ℹ Pipeline Summary:
  • Image: jgrusewski/foxhunt:latest
  • Dockerfile: Dockerfile.runpod
  • Mode: Full execution
  • Push: Completed

✓ Total pipeline time: 6m 7s

ℹ GitLab CI/CD readiness: ✅
  • Build stage: Validated
  • Test stage: Validated
  • Push stage: Validated

---

Summary

The local_ci_pipeline.sh script provides:

✅ Complete CI/CD simulation - All GitLab stages tested locally ✅ Fast iteration - Catch issues before GitLab deployment ✅ GLIBC validation - Ensures 2.35 compatibility ✅ CUDA validation - Verifies CUDA 12.4.1 + cuDNN 9 ✅ Entrypoint testing - Validates volume mount architecture ✅ Color-coded output - Easy to read success/error states ✅ Stage timing - Performance metrics for each stage ✅ Error handling - Exit on first failure (CI/CD behavior)

Total Time: 4-9 minutes (vs. 10-15 min on GitLab CI/CD) Cost: $0 (vs. GitLab CI/CD minutes) Reliability: 100% local control before cloud deployment