gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e61e8f54da130c2c6972e7f679add72e2b3a0959
foxhunt/AGENT_4_DOCUMENTATION_INDEX.md
jgrusewski 6da9d262db feat(ml): MAMBA-2 P0 fixes + hyperparameter optimization (13 params) 
CRITICAL P0 FIXES (Validated - Loss 0.87 → 0.07):
- Add sigmoid activation to inference and training (ml/src/mamba/mod.rs:798, 1538)
- Fix config.total_decay_steps (was hardcoded 10000) (ml/src/mamba/mod.rs:2271)
- Update d_state: 16→64, 32→64 (Mamba-2 spec) (ml/src/mamba/mod.rs:178, 730)

HYPERPARAMETER OPTIMIZATION:
- Implement 13-parameter Bayesian optimization with argmin
- Add async data loading with 3-batch prefetch (+20-30% speedup)
- Create hyperopt adapter: ml/src/hyperopt/adapters/mamba2.rs
- Add example: ml/examples/hyperopt_mamba2_demo.rs

VALIDATION:
- Local test: Loss 0.07 vs 0.87 (12× improvement)
- Val loss: 0.04-0.14 vs 1.2 (27× improvement)
- Accuracy: 12-30% vs 1-5% (3-6× improvement)
- All binaries rebuilt and uploaded to Runpod S3

DEPLOYMENT:
- RTX 4090 pod active (n0fq2ikt4uk0zy)
- Training: 10 trials × 50 epochs, batch_size=256
- Expected: 1.3 days, $10.41 cost

Fixes #P0-sigmoid #P0-decay-steps #hyperopt-mamba2
2025-10-28 14:11:18 +01:00

12 KiB
Raw Blame History

AGENT 4: Documentation Index - CUDA Version Enforcement

Date: 2025-10-27 Status: Complete - Ready for Implementation Total Documentation: 4 files (~3,000 lines)

Quick Navigation

For Immediate Action

For Understanding

Document Summaries

1. Quick Start Guide

File: CUDA_VERSION_ENFORCEMENT_QUICK_START.md Size: ~500 lines Read Time: 5 minutes Purpose: Get started immediately

What's Inside:

  • 30-second problem summary
  • 4-phase implementation steps (75 min total)
  • Testing checklist (8 tests)
  • Rollback plan (2 min)
  • Quick reference commands
  • Expected error messages
  • Cost & timeline summary

When to Use:

  • You want to start immediately
  • You need quick reference commands
  • You want to see error messages
  • You need cost/timeline estimates

Read This If: You're ready to implement NOW

2. Implementation Plan

File: AGENT_4_CUDA_VERSION_ENFORCEMENT_PLAN.md Size: ~1,500 lines Read Time: 15 minutes Purpose: Complete technical specification

What's Inside:

  • Executive summary (problem + solution)
  • 6 implementation components (with full code)
  • Testing strategy (8 tests, 3 phases)
  • Rollback plan (2 scenarios)
  • Timeline & cost breakdown
  • Risk assessment (3 levels)
  • Long-term maintenance plan
  • Appendices (file summary, error messages, commands)

When to Use:

  • You need complete implementation details
  • You want to understand the code changes
  • You need testing procedures
  • You want risk analysis
  • You need rollback procedures

Read This If: You need complete technical details

3. Synthesis Summary

File: AGENT_4_SYNTHESIS_SUMMARY.md Size: ~1,000 lines Read Time: 10 minutes Purpose: Understand how we got here

What's Inside:

  • Synthesis of 3 agents' findings
  • Root cause analysis (with diagrams)
  • Why AGENT K3's fix was wrong
  • Component breakdown (6 components)
  • Testing strategy
  • Success criteria
  • Key takeaways and lessons learned

When to Use:

  • You want to understand the problem history
  • You need to know why certain decisions were made
  • You want to learn from past mistakes
  • You need to explain to others

Read This If: You want complete context and reasoning

4. Documentation Index

File: AGENT_4_DOCUMENTATION_INDEX.md (This File) Size: ~200 lines Read Time: 2 minutes Purpose: Navigate all documentation

What's Inside:

  • Document summaries (what's in each file)
  • Navigation guidance (which file to read when)
  • Related documentation (from other agents)
  • Reading paths (different user scenarios)

When to Use:

  • You're new to this investigation
  • You're not sure which file to read
  • You want an overview before diving in

Read This If: You're starting from scratch

Reading Paths by Role

Path 1: Developer (Immediate Implementation)

Goal: Implement CUDA version enforcement NOW

Reading Order:

  1. Quick Start Guide (5 min) - Get overview + immediate steps
  2. Implementation Plan - Phase 1 (10 min) - Core enforcement details
  3. Execute Phase 1 (30 min) - Update build.rs, create validation script, test
  4. Implementation Plan - Phase 2 (10 min) - Deployment integration details
  5. Execute Phase 2 (20 min) - Enhance deploy script, verify Docker
  6. Continue through Phase 3 & 4 (25 min) - Documentation + validation

Total Time: 100 minutes (25 min reading + 75 min implementation)

Path 2: Project Manager (Oversight)

Goal: Understand problem, solution, cost, timeline, risk

Reading Order:

  1. Synthesis Summary - Executive Summary (2 min) - Problem overview
  2. Quick Start Guide - Cost & Timeline (1 min) - Budget impact
  3. Implementation Plan - Risk Assessment (3 min) - Risk evaluation
  4. Synthesis Summary - Key Takeaways (2 min) - Lessons learned

Total Time: 8 minutes

Path 3: Architect (Technical Review)

Goal: Validate solution design, assess technical decisions

Reading Order:

  1. Synthesis Summary - Root Cause (5 min) - Problem analysis
  2. Synthesis Summary - Why AGENT K3's Fix Was Wrong (3 min) - Critical review
  3. Implementation Plan - Solution Architecture (5 min) - Design principles
  4. Implementation Plan - Component Breakdown (10 min) - Technical details
  5. Implementation Plan - Testing Strategy (5 min) - Validation approach

Total Time: 28 minutes

Path 4: DevOps Engineer (Deployment Focus)

Goal: Understand deployment changes, CI/CD integration

Reading Order:

  1. Quick Start Guide - Phase 2 (2 min) - Deployment integration
  2. Implementation Plan - Component 4 (5 min) - Pre-deploy validation
  3. Implementation Plan - Component 6 (5 min) - CI/CD workflow
  4. Implementation Plan - Testing Phase 2 (3 min) - Deployment tests
  5. Quick Start Guide - Quick Commands (2 min) - Reference commands

Total Time: 17 minutes

Path 5: Maintainer (Long-Term Perspective)

Goal: Understand maintenance requirements, future updates

Reading Order:

  1. Synthesis Summary - Key Takeaways (5 min) - Lessons learned
  2. Implementation Plan - Long-Term Maintenance (5 min) - Update process
  3. Implementation Plan - Monitoring (2 min) - Metrics to track
  4. Synthesis Summary - Why This Solution Is Right (3 min) - Design rationale

Total Time: 15 minutes

Related Documentation (From Other Agents)

Problem Discovery

AGENT 1: Binary Build Timeline

  • File: AGENT_1_BINARY_BUILD_TIMELINE_REPORT.md
  • Found: Binary staleness, CUDA 13.0 compilation
  • Relevance: Identified when CUDA 13.0 binary was created

AGENT K3: Docker CUDA 13.0 Fix

  • File: AGENT_K3_CUDA13_DOCKER_FIX.md
  • Attempted: Upgrade Docker to CUDA 13.0
  • Relevance: Incorrect fix (violates Runpod driver 550 constraint)

CUDA Version Mismatch Analysis

  • File: CUDA_VERSION_MISMATCH_ANALYSIS.md
  • Found: Root cause (local CUDA 13.0, Docker CUDA 12.9.1)
  • Relevance: Comprehensive diagnosis, solution options

Error Analysis

CUDA PTX Fix Complete

  • File: CUDA_PTX_FIX_COMPLETE.md
  • Found: PTX version mismatch error details
  • Relevance: Runtime error symptoms, local fix attempts

CUDA PTX Version Fix

  • File: CUDA_PTX_VERSION_FIX.md
  • Found: Local environment analysis
  • Relevance: CUDA 12.9 vs. 13.0 comparison, fix options

System Documentation

CLAUDE.md

  • File: CLAUDE.md
  • Section: "☁️ Runpod GPU Deployment" (line ~350)
  • Relevance: Design decision (CUDA 12.9 for driver 550 compatibility)
  • UPDATE REQUIRED: Add CUDA version requirements section

ML Training Parquet Guide

  • File: ML_TRAINING_PARQUET_GUIDE.md
  • Section: Build Prerequisites (early in file)
  • Relevance: ML training setup instructions
  • UPDATE REQUIRED: Add CUDA validation section

Key Files to Modify

Phase 1: Core Enforcement (30 min)

  1. ml/build.rs (MODIFY)

    • Current: 21 lines (minimal CUDA check)
    • New: ~120 lines (full CUDA version enforcement)
    • Change: Detect & reject CUDA 13.0+
    • Risk: Low (easy rollback)

  2. scripts/validate_cuda_env.sh (CREATE)

    • Current: N/A (doesn't exist)
    • New: ~130 lines (standalone validation)
    • Change: Bash script for CI/CD
    • Risk: Low (no dependencies)

  3. Dockerfile.runpod (REVERT)

    • Current: Line 24 uses CUDA 13.0 (AGENT K3's change)
    • New: Line 24 uses CUDA 12.9.1 (revert to original)
    • Change: Revert AGENT K3's incorrect fix
    • Risk: Low (known good state)

Phase 2: Deployment Integration (20 min)

  1. scripts/runpod_deploy.py (ENHANCE)
    • Current: No binary validation
    • New: +85 lines (validation functions + call)
    • Change: Pre-deploy binary linkage check
    • Risk: Low (only blocks invalid binaries)

Phase 3: Documentation (15 min)

  1. CLAUDE.md (UPDATE)

    • Section: "☁️ Runpod GPU Deployment"
    • New: +14 lines (CUDA requirements)
    • Change: Add CUDA version clarification
    • Risk: None (documentation only)

  2. ML_TRAINING_PARQUET_GUIDE.md (UPDATE)

    • Section: Build Prerequisites
    • New: +20 lines (CUDA validation section)
    • Change: Add pre-build validation steps
    • Risk: None (documentation only)

Phase 4: CI/CD Integration (Optional, 15 min)

  1. .github/workflows/build-binaries.yml (CREATE)
    • Current: N/A (doesn't exist)
    • New: ~60 lines (GitHub Actions workflow)
    • Change: Automate CUDA validation in CI/CD
    • Risk: Low (optional enhancement)

Success Checklist

After Implementation, Verify:

  • ./scripts/validate_cuda_env.sh exits 0 with CUDA 12.9
  • ./scripts/validate_cuda_env.sh exits 1 with CUDA 13.0
  • Build with CUDA 13.0 fails with clear error message
  • Build with CUDA 12.9 succeeds with "✅ CUDA 12.9 detected"
  • ldd shows libcublas.so.12 (not .so.13)
  • Deployment script validates binaries pre-upload
  • Docker image has CUDA 12.9.1 (not 13.0)
  • Runpod pod trains successfully (NO PTX errors)

All 8 must pass for successful implementation.

Timeline Summary

Phase Tasks Time Total
Reading Review documentation 5-15 min 5-15 min
Phase 1 Core enforcement 30 min 30 min
Phase 2 Deployment integration 20 min 50 min
Phase 3 Documentation 15 min 65 min
Phase 4 Validation & deploy 10 min 75 min
TOTAL - - 80-90 min

Cost: $0.15 (testing + validation on Runpod)

Quick Reference

Essential Commands

Check CUDA Version:

nvcc --version
ls -la /usr/local/cuda

Switch to CUDA 12.9:

sudo rm /etc/alternatives/cuda
sudo ln -s /usr/local/cuda-12.9 /etc/alternatives/cuda
nvcc --version  # Verify

Validate Environment:

./scripts/validate_cuda_env.sh

Build with Validation:

cargo clean
cargo build -p ml --release --features cuda

Verify Binary:

ldd target/release/examples/train_tft_parquet | grep cublas
# Expected: libcublas.so.12

Deploy to Runpod:

python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"

Contact & Support

Questions?

  • CUDA version issues: See Quick Start Guide - Error Messages section
  • Build failures: Check Implementation Plan - Rollback section
  • Deployment failures: Check Synthesis Summary - Testing Strategy
  • General questions: Start with Synthesis Summary - Executive Summary

Document Status

Document Status Last Updated Lines
Implementation Plan ✅ Complete 2025-10-27 ~1,500
Quick Start Guide ✅ Complete 2025-10-27 ~500
Synthesis Summary ✅ Complete 2025-10-27 ~1,000
Documentation Index ✅ Complete 2025-10-27 ~200
TOTAL - - ~3,200

Next Steps

  1. Choose your reading path (see "Reading Paths by Role" above)
  2. Read relevant documentation (5-15 minutes)
  3. Execute Phase 1 (Core Enforcement) - 30 minutes
  4. Execute Phase 2 (Deployment Integration) - 20 minutes
  5. Execute Phase 3 (Documentation) - 15 minutes
  6. Execute Phase 4 (Validation & Deploy) - 10 minutes
  7. Verify success (8-test checklist above)

Status: ✅ COMPLETE - READY FOR IMPLEMENTATION Confidence: 95% (high confidence, low risk) Priority: P0 (blocks Runpod deployment) Recommendation: Start with Quick Start Guide, execute Phase 1 immediately

END OF DOCUMENTATION INDEX

Reference in New Issue View Git Blame Copy Permalink