foxhunt/DOCKER_ROOT_FILES_ANALYSIS.md

Docker Root Files Investigation Report

Date: 2025-10-30 Repository: foxhunt (Rust HFT Trading System) Current Status: Production Certified ✅

---

Executive Summary

Key Findings:

1. 3 main Docker files in root (all actively used)
2. 1 obsolete docker-compose override file (should be removed)
3. Multiple service Dockerfiles properly organized in /services
4. Separate monitoring stack in /monitoring/docker-compose.yml
5. Multiple script references to deprecated Dockerfile.runpod (removed in cleanup)

Recommendation:

• REMOVE: docker-compose.override.yml.disabled (backup/obsolete)
• KEEP in root: All other Docker files (active production use)
• Consider: Moving monitoring stack compose to docker/ for organization (optional)

---

Detailed File Analysis

ROOT LEVEL DOCKER FILES

1. Dockerfile.foxhunt-build (6.0K, 160 lines)

Status: ✅ ACTIVE - PRODUCTION BUILD Purpose: Multi-stage build for Runpod GPU deployment Usage: Referenced in .gitlab-ci.yml for CI/CD pipeline Details:

• 5-stage build with cargo-chef dependency caching
• CUDA 12.4.1 + cuDNN 9 (Ubuntu 22.04, GLIBC 2.35)
• Builds 4 hyperopt binaries: mamba2, dqn, ppo, tft
• Minimal runtime image (~2.6GB)
• BuildKit layer caching optimized

Location: /home/jgrusewski/Work/foxhunt/Dockerfile.foxhunt-build

---

2. .dockerignore (5.0K, 76 lines)

Status: ✅ ACTIVE - BUILD OPTIMIZATION Purpose: Reduce Docker build context size Usage: Automatically used by Docker daemon during build Details:

• Excludes: target/, migrations/, test data, docs, logs
• Includes specific script exception: !scripts/runpod_deploy.py
• Reduces build context significantly (improves BuildKit caching)

Location: /home/jgrusewski/Work/foxhunt/.dockerignore

---

3. docker-compose.yml (9.0K, 487 lines)

Status: ✅ ACTIVE - PRIMARY ORCHESTRATION Purpose: Full local development and testing stack Usage: Main compose file for docker-compose up -d Services (10 total):

• Infrastructure: PostgreSQL 16 (TimescaleDB), Redis, InfluxDB, Vault, Prometheus, Grafana, MinIO
• Microservices: API Gateway, Trading Service, Backtesting Service, ML Training Service, Trading Agent Service

Key Features:

• Health checks on all services
• Volume mounts for data persistence
• Network isolation (foxhunt-network bridge)
• Environment variables from .env file
• GPU support for ML Training Service (nvidia runtime)

Location: /home/jgrusewski/Work/foxhunt/docker-compose.yml

---

4. docker-compose.override.yml (5.0K, 18 lines)

Status: ✅ ACTIVE - DEVELOPMENT OVERRIDES Purpose: Auto-loaded by docker-compose for development setup Usage: Automatically merged with docker-compose.yml when running locally Services Modified:

• trading_service: Adds .dev dockerfile, enables live code reload
• backtesting_service: Adds .dev dockerfile, volume mounts for source
• ml_training_service: Adds .dev dockerfile, source code volumes
• api_gateway: Sets debug logging (RUST_LOG=debug, RUST_BACKTRACE=full)

Key Features:

• Dockerfile overrides for dev builds (enables compilation in container)
• Source code volume mounts (RO) for hot-reload capability
• Enhanced debugging output
• Minimal changes (~18 lines)

Location: /home/jgrusewski/Work/foxhunt/docker-compose.override.yml

Note: Docker automatically loads docker-compose.override.yml if present. This is the CURRENT/ACTIVE development config.

---

5. docker-compose.override.yml.disabled (5.0K, 47 lines) ❌ OBSOLETE

Status: ❌ OBSOLETE - BACKUP/DISABLED Purpose: Backup of old override configuration Usage: NONE - Explicitly disabled with .disabled suffix Content:

• References to old Dockerfile.dev files (no longer used)
• Same structure as #4 above but outdated

Recommendation: REMOVE THIS FILE

• It's a backup file with .disabled suffix indicating it's not in use
• Clutters root directory
• References deprecated Dockerfile.dev pattern
• Can be recovered from git history if needed

Location: /home/jgrusewski/Work/foxhunt/docker-compose.override.yml.disabled

---

MONITORING STACK (SEPARATE)

6. monitoring/docker-compose.yml (5.0K, 126 lines)

Status: ✅ ACTIVE - STANDALONE MONITORING Purpose: Separate monitoring stack (Prometheus, Grafana, AlertManager) Usage: Run separately: docker-compose -f monitoring/docker-compose.yml up -d Services (6 total):

• Prometheus v2.48.0
• Grafana 10.2.2
• AlertManager v0.26.0
• PostgreSQL Exporter
• Redis Exporter
• Node Exporter

Network: foxhunt-monitoring (separate from main foxhunt-network) Integration: Cross-network connection to main postgres/redis services

Location: /home/jgrusewski/Work/foxhunt/monitoring/docker-compose.yml

Note: Can be run in parallel with main stack or consolidated if desired.

---

SERVICE-LEVEL DOCKERFILES (Proper Location)

Located in /services/*/ - Correctly organized:

1. /services/api_gateway/Dockerfile
2. /services/backtesting_service/Dockerfile
3. /services/ml_training_service/Dockerfile
4. /services/trading_service/Dockerfile
5. /services/trading_agent_service/Dockerfile

Status: ✅ All ACTIVE

Each service has its own Dockerfile referenced in docker-compose.yml:

build:
  context: .
  dockerfile: services/trading_service/Dockerfile

---

TEST-LEVEL DOCKER COMPOSE FILES

Proper organization in test directories:

1. /services/api_gateway/tests/docker-compose.yml - Test harness
2. /services/ml_training_service/tests/docker/docker-compose.test.yml - Test harness

Status: ✅ All ACTIVE - Used for integration testing

---

Historical Context: Dockerfile.runpod

What Happened:

• Previous file: Dockerfile.runpod (now removed)
• Replacement: Dockerfile.foxhunt-build (current production build)
• Cleanup commit: 8ea5a650 (Oct 29, 2025) - "chore: Major codebase cleanup"

Current References to Removed File:

Several scripts still reference the removed Dockerfile.runpod:

1. scripts/build_docker_images.sh - Usage example comment
2. scripts/deploy_debug_image.sh - DOCKERFILE="Dockerfile.runpod.debug"
3. scripts/verify_dockerfile_updates.sh - Old verification script
4. scripts/LOCAL_CI_QUICK_REF.md - Documentation
5. scripts/README.md - Documentation
6. scripts/upload_to_runpod_s3.sh - References old dockerfile

Status:

These references are in archived/deprecated scripts and can be cleaned up, but are not blocking any active workflows.

---

Summary Table

File	Size	Lines	Status	Usage	Location	Action
Dockerfile.foxhunt-build	6.0K	160	✅ Active	CI/CD Pipeline	Root	KEEP
.dockerignore	5.0K	76	✅ Active	Build Context	Root	KEEP
docker-compose.yml	9.0K	487	✅ Active	Main Orchestration	Root	KEEP
docker-compose.override.yml	5.0K	18	✅ Active	Dev Overrides	Root	KEEP
docker-compose.override.yml.disabled	5.0K	47	❌ Obsolete	None	Root	REMOVE
monitoring/docker-compose.yml	5.0K	126	✅ Active	Monitoring Stack	monitoring/	KEEP
services/*/Dockerfile (5 files)	N/A	N/A	✅ Active	Service Build	services/	KEEP

---

Recommendations

Priority 1: Remove Obsolete Files (IMMEDIATE)

# Remove backup override file
rm docker-compose.override.yml.disabled
git add docker-compose.override.yml.disabled
git commit -m "chore: Remove obsolete docker-compose.override.yml.disabled backup file"

Priority 2: Clean Up Script References (OPTIONAL)

Update/clean these archived scripts that reference removed Dockerfile.runpod:

• scripts/LOCAL_CI_QUICK_REF.md - Update examples
• scripts/README.md - Update references
• scripts/build_docker_images.sh - Update usage examples
• scripts/verify_dockerfile_updates.sh - Either archive or update

Note: Only if these scripts are actively maintained. Archived scripts can remain as-is for historical reference.

Priority 3: Organization (OPTIONAL - NICE TO HAVE)

If you want stricter organization, could create /docker/ directory:

docker/
  ├── Dockerfile.foxhunt-build
  ├── docker-compose.yml
  ├── docker-compose.override.yml
  ├── .dockerignore
  └── monitoring/
      └── docker-compose.yml

Trade-off:

• Pro: Cleaner root directory
• Con: Requires updating .dockerignore path references, CI/CD config, scripts

Recommendation: Keep in root (current location) - these are central to the project and CLI tools expect them at root level.

Priority 4: CI/CD Verification

The .gitlab-ci.yml currently references Dockerfile.foxhunt-build:

-f Dockerfile.foxhunt-build \

This is correct. No changes needed.

---

Docker Build Flow Diagram

Development Workflow:
┌─────────────────────────────────────────┐
│ docker-compose up -d                    │
├─────────────────────────────────────────┤
│ 1. Load docker-compose.yml (base)       │
│ 2. Merge docker-compose.override.yml    │
│ 3. Build services using their          │
│    Dockerfiles (services/*/Dockerfile) │
│ 4. Start 10 services with healthchecks │
│ 5. Enable live code reload (override)   │
└─────────────────────────────────────────┘

CI/CD Pipeline (GitLab):
┌─────────────────────────────────────────┐
│ .gitlab-ci.yml triggered on push        │
├─────────────────────────────────────────┤
│ 1. Build using Dockerfile.foxhunt-build │
│ 2. Multi-stage with cargo-chef caching  │
│ 3. BuildKit optimization enabled        │
│ 4. Push to Docker Hub registry          │
│ 5. Deploy to Runpod (manual approval)   │
└─────────────────────────────────────────┘

Monitoring Stack (Optional/Separate):
┌─────────────────────────────────────────┐
│ docker-compose -f monitoring/           │
│  docker-compose.yml up -d               │
├─────────────────────────────────────────┤
│ Runs on foxhunt-monitoring network      │
│ Connects to main stack for scraping     │
└─────────────────────────────────────────┘

---

Conclusion

Current State Assessment: ✅ WELL ORGANIZED

Root Level Files (3 active + 1 backup):

• ✅ Properly configured for production CI/CD
• ✅ Development overrides correctly set up
• ✅ Build context optimization in place
• ❌ One obsolete backup file needs removal

Service Organization (5 services):

• ✅ Correctly placed in /services/*/
• ✅ Properly referenced in main compose file
• ✅ Independent and reusable

Monitoring Stack (separate):

• ✅ Properly isolated in /monitoring/
• ✅ Can run independently or in parallel
• ✅ Clean network separation

Overall: System is production-ready. Only action needed is removing one obsolete backup file.

---

File Locations Summary

Root Level:

• /home/jgrusewski/Work/foxhunt/Dockerfile.foxhunt-build
• /home/jgrusewski/Work/foxhunt/.dockerignore
• /home/jgrusewski/Work/foxhunt/docker-compose.yml
• /home/jgrusewski/Work/foxhunt/docker-compose.override.yml
• /home/jgrusewski/Work/foxhunt/docker-compose.override.yml.disabled (REMOVE)

Services:

• /home/jgrusewski/Work/foxhunt/services/api_gateway/Dockerfile
• /home/jgrusewski/Work/foxhunt/services/backtesting_service/Dockerfile
• /home/jgrusewski/Work/foxhunt/services/ml_training_service/Dockerfile
• /home/jgrusewski/Work/foxhunt/services/trading_service/Dockerfile
• /home/jgrusewski/Work/foxhunt/services/trading_agent_service/Dockerfile

Monitoring:

• /home/jgrusewski/Work/foxhunt/monitoring/docker-compose.yml

Tests:

• /home/jgrusewski/Work/foxhunt/services/api_gateway/tests/docker-compose.yml
• /home/jgrusewski/Work/foxhunt/services/ml_training_service/tests/docker/docker-compose.test.yml