Files
foxhunt/scripts/SCRIPTS_INVENTORY.md
jgrusewski 433af5c25d chore: Major codebase cleanup - remove deprecated files and organize structure
- Docker: Delete 23 deprecated Dockerfiles, fix CI/CD to use Dockerfile.foxhunt-build
- Config: Remove 36 .env files, keep 4 essential, delete config/environments/
- Docs: Archive 614 Wave D files to docs/archive/wave_d/, 95% reduction in root
- Scripts: Delete 56 deprecated scripts, keep 58 production-critical (49% reduction)
- Python: Organize 37 scripts into scripts/python/ subdirectories, delete ml/python/
- Build: Remove 1GB artifacts, delete old venvs, clean Python cache from git
- Migrations: Delete deprecated directory (4,432 lines), remove duplicate database/migrations/
- Infrastructure: Delete deployment/ (61 files), docs/scripts/ (8 files)

Total impact: ~2,500 files cleaned, 750MB+ space freed, zero production impact
All deleted scripts backed up to archives. runpod/ and tests/runpod/ preserved.
data_acquisition_service retained per user request.
2025-10-30 01:02:34 +01:00

7.5 KiB

Scripts Inventory - Active Scripts Only

Last Updated: 2025-10-30 (Post-Cleanup) Total Scripts: 58


Quick Reference by Purpose

🚀 Deployment & Infrastructure (8 scripts)

Script Purpose Usage
runpod_deploy.py Deploy GPU pod to Runpod python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"
upload_binary.py Upload binaries to Runpod S3 python3 scripts/upload_binary.py <binary_path>
monitor_logs.py Monitor pod logs python3 scripts/monitor_logs.py <pod_id>
build_docker_images.sh Build multi-stage Docker images ./scripts/build_docker_images.sh
build_hyperopt_docker.sh Build hyperopt Docker ./scripts/build_hyperopt_docker.sh
local_ci_pipeline.sh Run local CI/CD ./scripts/local_ci_pipeline.sh
start_foxhunt.sh Start all services ./scripts/start_foxhunt.sh
stop_foxhunt.sh Stop all services ./scripts/stop_foxhunt.sh

Validation & Testing (16 scripts)

Script Validates Usage
validate_auth_enabled.sh Auth enabled ./scripts/validate_auth_enabled.sh
validate_binary.sh Binary integrity ./scripts/validate_binary.sh
validate_clippy.sh Clippy warnings ./scripts/validate_clippy.sh
validate_data_quality.sh Data quality ./scripts/validate_data_quality.sh
validate_dqn_performance.sh DQN performance ./scripts/validate_dqn_performance.sh
validate_grpc_endpoints.sh gRPC endpoints ./scripts/validate_grpc_endpoints.sh
validate_h5_alerting.sh H5 alerting ./scripts/validate_h5_alerting.sh
validate_jwt_config.sh JWT config ./scripts/validate_jwt_config.sh
validate_ml_monitoring_metrics.sh ML monitoring ./scripts/validate_ml_monitoring_metrics.sh
validate-monitoring-performance.sh Monitoring performance ./scripts/validate-monitoring-performance.sh
validate-performance.py System performance python3 scripts/validate-performance.py
validate_ppo_fix.sh PPO fixes ./scripts/validate_ppo_fix.sh
validate_tft_configs.py TFT configs python3 scripts/validate_tft_configs.py
validate_tls_setup.sh TLS setup ./scripts/validate_tls_setup.sh
validate_training.sh Training pipeline ./scripts/validate_training.sh
validate_train_script.sh Train scripts ./scripts/validate_train_script.sh

🏥 Health & Diagnostics (5 scripts)

Script Purpose Usage
health_check.sh Quick health check ./scripts/health_check.sh
smoke_test.sh Smoke testing ./scripts/smoke_test.sh
comprehensive_health_check.sh Full system health ./scripts/comprehensive_health_check.sh
quick_status.sh Quick status ./scripts/quick_status.sh
system_resource_monitor.sh Resource monitoring ./scripts/system_resource_monitor.sh

🤖 ML Training & Analysis (7 scripts)

Script Purpose Usage
train_all_models_fixed.sh Train all models ./scripts/train_all_models_fixed.sh
train_tft_production.py TFT production training python3 scripts/train_tft_production.py
monitor_hyperopt.sh Monitor hyperopt ./scripts/monitor_hyperopt.sh
analyze_checkpoints_simple.py Checkpoint analysis python3 scripts/analyze_checkpoints_simple.py
compare_checkpoints.py Compare checkpoints python3 scripts/compare_checkpoints.py
extract_best_hyperparameters.py Extract best params python3 scripts/extract_best_hyperparameters.py
quarterly_retrain.sh Quarterly retraining ./scripts/quarterly_retrain.sh

🔒 Security & Production (7 scripts)

Script Purpose Usage
generate-production-secrets.sh Generate secrets ./scripts/generate-production-secrets.sh
production-security-hardening.sh Harden security ./scripts/production-security-hardening.sh
security-hardening.sh Security config ./scripts/security-hardening.sh
setup-docker-secrets.sh Docker secrets ./scripts/setup-docker-secrets.sh
setup_production_passwords.sh Password setup ./scripts/setup_production_passwords.sh
export_vault_passwords.sh Export Vault passwords ./scripts/export_vault_passwords.sh
generate-compliance-report.py Compliance report python3 scripts/generate-compliance-report.py

🔍 Verification & Setup (7 scripts)

Script Purpose Usage
verify_ci_setup.sh Verify CI/CD ./scripts/verify_ci_setup.sh
verify_ml_dependencies.sh Verify ML deps ./scripts/verify_ml_dependencies.sh
verify_vault_setup.sh Verify Vault ./scripts/verify_vault_setup.sh
check_service_binaries.sh Check service bins ./scripts/check_service_binaries.sh
check_all_available_gpus.py Check all GPUs python3 scripts/check_all_available_gpus.py
check_gpu_availability.py Check GPU status python3 scripts/check_gpu_availability.py
watch_gpu_availability.sh Watch GPU ./scripts/watch_gpu_availability.sh

🛠️ Utilities (8 scripts)

Script Purpose Usage
convert_csv_to_parquet.py CSV → Parquet python3 scripts/convert_csv_to_parquet.py
plan_databento_download.sh Plan data download ./scripts/plan_databento_download.sh
enforce_coverage.sh Enforce coverage ./scripts/enforce_coverage.sh
run-coverage.sh Run coverage ./scripts/run-coverage.sh
run_comprehensive_tests.sh Comprehensive tests ./scripts/run_comprehensive_tests.sh
offline_service_validation.sh Offline validation ./scripts/offline_service_validation.sh
install_cron.sh Install cron jobs ./scripts/install_cron.sh
cleanup_deprecated_scripts.sh Cleanup scripts ./scripts/cleanup_deprecated_scripts.sh

Common Workflows

Deploy to Runpod

# 1. Build Docker image
./scripts/build_docker_images.sh

# 2. Deploy pod
python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"

# 3. Monitor logs
python3 scripts/monitor_logs.py <pod_id>

Local Development

# 1. Start services
./scripts/start_foxhunt.sh

# 2. Run health checks
./scripts/health_check.sh

# 3. Run smoke tests
./scripts/smoke_test.sh

# 4. Stop services
./scripts/stop_foxhunt.sh

CI/CD Validation

# 1. Run local CI pipeline
./scripts/local_ci_pipeline.sh

# 2. Verify setup
./scripts/verify_ci_setup.sh

# 3. Validate performance
python3 scripts/validate-performance.py

ML Training

# 1. Verify ML dependencies
./scripts/verify_ml_dependencies.sh

# 2. Check GPU availability
python3 scripts/check_gpu_availability.py

# 3. Train all models
./scripts/train_all_models_fixed.sh

# 4. Monitor training
./scripts/monitor_hyperopt.sh

Deprecated Scripts (Backed Up)

All deprecated scripts moved to:

/home/jgrusewski/Work/foxhunt/scripts/archive/cleanup_2025_10_30/

Total Deleted: 56 scripts (49% reduction)

See /home/jgrusewski/Work/foxhunt/SCRIPTS_CLEANUP_REPORT_2025_10_30.md for details.


Maintenance

Adding New Scripts

  1. Create script in /scripts/
  2. Make executable: chmod +x scripts/<script>.sh
  3. Update this inventory
  4. Document in scripts/README.md

Deprecating Scripts

  1. Move to scripts/archive/deprecated_<date>/
  2. Update this inventory
  3. Remove from CI/CD references
  4. Update documentation

Last Updated: 2025-10-30 Maintained By: Foxhunt Development Team