e393a8af89
## Summary Third major cleanup wave after investigating 287 remaining root files. Archived historical reports, organized documentation, removed regeneratable artifacts, and fixed critical security issue. ## Files Cleaned (119 total) - Archived: 78 files (7 WAVE reports + 71 summaries) → docs/archive/ - Archived: 7 build logs → docs/archive/build_logs/ - Organized: 10 markdown files → docs/guides/ + docs/checklists/ - Deleted: 17 test/coverage artifacts (regeneratable) - Deleted: 7 empty/obsolete files (docker override, clippy baselines) - Deleted: 3 large files (119MB - .venv, ppo_hyperopt_output.txt, backup) ## Space Recovered - Total: ~120.7 MB - Large files: 119.25 MB (.venv, ppo_hyperopt_output.txt) - Archives: 1.04 MB (summaries + build logs) - Test artifacts: 980 KB ## Security Fix (CRITICAL) - Fixed: certs/security.env removed from git tracking (contained JWT secrets) - Updated: .gitignore to prevent future tracking of sensitive cert files - Removed: 4 files from git history (security.env, production.env.template, *.serial) ## Documentation Organization - Created: docs/archive/ (wave_reports/, summaries/, build_logs/) - Created: docs/guides/ (7 detailed implementation guides) - Created: docs/checklists/ (3 operational checklists) - Retained: 30 essential .md files in root (quick refs, CLAUDE.md) ## Investigation Reports Created - MARKDOWN_ORGANIZATION_REPORT.md - TXT_FILES_INVENTORY_AND_ARCHIVAL_PLAN.md - ROOT_CONFIG_FILES_ANALYSIS_REPORT.md - DOCKER_ROOT_FILES_ANALYSIS.md - DATABASE_INITIALIZATION_AND_SETUP_ANALYSIS.md - (6 additional investigation/index files) ## Cleanup Wave Progress - Wave 1: 899 files deleted (1,071,884 lines) - Wave 2: 543 files archived/deleted (~34GB) - Wave 3: 119 files archived/deleted/organized (~121MB) - Total: 1,561 files cleaned, ~35.1GB space recovered ## Result Root directory: 287 files → ~180 files (excluding investigation reports) Clean, organized, production-ready structure maintained. Related: Second cleanup wave (previous commit)
68 lines
2.4 KiB
Plaintext
68 lines
2.4 KiB
Plaintext
╔══════════════════════════════════════════════════════════════════════════════╗
|
||
║ MAMBA-2 VRAM ANALYSIS - QUICK SUMMARY ║
|
||
╚══════════════════════════════════════════════════════════════════════════════╝
|
||
|
||
🔍 ROOT CAUSE FOUND: Data duplication (2.74GB CPU + 2.74GB GPU = 5.48GB wasted)
|
||
|
||
📊 OLD FORMULA (WRONG):
|
||
VRAM = 529MB + 88MB × batch_size
|
||
@ BS=144: Predicted 13.2GB, Actual 7.0GB (88% ERROR!)
|
||
|
||
✅ NEW FORMULA (CORRECT):
|
||
VRAM = 6,474MB + 7.0MB × batch_size
|
||
@ BS=144: Predicted 7.5GB, Actual 7.0GB (4.4% error)
|
||
|
||
🎯 IMMEDIATE ACTION (30 min):
|
||
Update batch_size_max: 96 → 180
|
||
Files:
|
||
- ml/examples/hyperopt_mamba2_demo.rs:69
|
||
- ml/src/hyperopt/adapters/mamba2.rs:118
|
||
|
||
Impact: +25% training speed, -$0.20 per run
|
||
|
||
🔧 SHORT-TERM FIX (2-4 hours):
|
||
Fix data duplication bug
|
||
Change: Create tensors on GPU device (not CPU)
|
||
Files:
|
||
- ml/src/hyperopt/adapters/mamba2.rs:496-526
|
||
- ml/src/mamba/mod.rs:1295-1296
|
||
|
||
Savings: 2.74GB VRAM, max batch_size: 180 → 250
|
||
|
||
📈 BATCH SIZE RECOMMENDATIONS:
|
||
|
||
Current: BS=144 → 7.5GB VRAM (46% of 16GB)
|
||
Recommended: BS=180 → 7.7GB VRAM (48% of 16GB) ← SAFE
|
||
Aggressive: BS=220 → 8.0GB VRAM (50% of 16GB) ← RISKY
|
||
Maximum: BS=250 → 8.2GB VRAM (after data fix)
|
||
|
||
💾 MEMORY BREAKDOWN (7.0 GB @ BS=144):
|
||
|
||
Data (CPU): 2.74 GB (37%) ← BUG!
|
||
Data (GPU): 2.74 GB (37%) ← BUG!
|
||
CUDA context/overhead: 0.82 GB (11%)
|
||
Activations (BS=144): 0.67 GB (9%)
|
||
Gradient temporaries: 0.28 GB (4%)
|
||
Model/grads/optimizer: 0.04 GB (1%)
|
||
Memory fragmentation: 0.15 GB (2%)
|
||
|
||
🚀 NEXT STEPS:
|
||
|
||
1. TODAY (30 min):
|
||
□ Update batch_size_max to 180
|
||
□ Test 1 trial @ BS=180
|
||
□ Measure VRAM (expect ~7.7GB)
|
||
|
||
2. THIS WEEK (4 hours):
|
||
□ Fix data duplication bug
|
||
□ Test 1 trial @ BS=250
|
||
□ Run 10-trial hyperopt validation
|
||
|
||
3. NEXT SPRINT (2-3 days):
|
||
□ Implement mixed precision FP16
|
||
□ Expected: 2× speed, 40% memory savings
|
||
□ New max batch_size: 500
|
||
|
||
📚 FULL REPORT: AGENT_R3_A5_VRAM_ANALYSIS.md
|
||
|