gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1934367bfaf8dcccd31ec80fe2d3034da94c515a
foxhunt/scripts/SCRIPTS_INVENTORY.md
jgrusewski 433af5c25d chore: Major codebase cleanup - remove deprecated files and organize structure 
- Docker: Delete 23 deprecated Dockerfiles, fix CI/CD to use Dockerfile.foxhunt-build
- Config: Remove 36 .env files, keep 4 essential, delete config/environments/
- Docs: Archive 614 Wave D files to docs/archive/wave_d/, 95% reduction in root
- Scripts: Delete 56 deprecated scripts, keep 58 production-critical (49% reduction)
- Python: Organize 37 scripts into scripts/python/ subdirectories, delete ml/python/
- Build: Remove 1GB artifacts, delete old venvs, clean Python cache from git
- Migrations: Delete deprecated directory (4,432 lines), remove duplicate database/migrations/
- Infrastructure: Delete deployment/ (61 files), docs/scripts/ (8 files)

Total impact: ~2,500 files cleaned, 750MB+ space freed, zero production impact
All deleted scripts backed up to archives. runpod/ and tests/runpod/ preserved.
data_acquisition_service retained per user request.
2025-10-30 01:02:34 +01:00

198 lines
7.5 KiB
Markdown
Raw Blame History

# Scripts Inventory - Active Scripts Only
**Last Updated**: 2025-10-30 (Post-Cleanup)
**Total Scripts**: 58
---
## Quick Reference by Purpose
### 🚀 Deployment & Infrastructure (8 scripts)
| Script | Purpose | Usage |
|--------|---------|-------|
| `runpod_deploy.py` | Deploy GPU pod to Runpod | `python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"` |
| `upload_binary.py` | Upload binaries to Runpod S3 | `python3 scripts/upload_binary.py <binary_path>` |
| `monitor_logs.py` | Monitor pod logs | `python3 scripts/monitor_logs.py <pod_id>` |
| `build_docker_images.sh` | Build multi-stage Docker images | `./scripts/build_docker_images.sh` |
| `build_hyperopt_docker.sh` | Build hyperopt Docker | `./scripts/build_hyperopt_docker.sh` |
| `local_ci_pipeline.sh` | Run local CI/CD | `./scripts/local_ci_pipeline.sh` |
| `start_foxhunt.sh` | Start all services | `./scripts/start_foxhunt.sh` |
| `stop_foxhunt.sh` | Stop all services | `./scripts/stop_foxhunt.sh` |
---
### ✅ Validation & Testing (16 scripts)
| Script | Validates | Usage |
|--------|-----------|-------|
| `validate_auth_enabled.sh` | Auth enabled | `./scripts/validate_auth_enabled.sh` |
| `validate_binary.sh` | Binary integrity | `./scripts/validate_binary.sh` |
| `validate_clippy.sh` | Clippy warnings | `./scripts/validate_clippy.sh` |
| `validate_data_quality.sh` | Data quality | `./scripts/validate_data_quality.sh` |
| `validate_dqn_performance.sh` | DQN performance | `./scripts/validate_dqn_performance.sh` |
| `validate_grpc_endpoints.sh` | gRPC endpoints | `./scripts/validate_grpc_endpoints.sh` |
| `validate_h5_alerting.sh` | H5 alerting | `./scripts/validate_h5_alerting.sh` |
| `validate_jwt_config.sh` | JWT config | `./scripts/validate_jwt_config.sh` |
| `validate_ml_monitoring_metrics.sh` | ML monitoring | `./scripts/validate_ml_monitoring_metrics.sh` |
| `validate-monitoring-performance.sh` | Monitoring performance | `./scripts/validate-monitoring-performance.sh` |
| `validate-performance.py` | System performance | `python3 scripts/validate-performance.py` |
| `validate_ppo_fix.sh` | PPO fixes | `./scripts/validate_ppo_fix.sh` |
| `validate_tft_configs.py` | TFT configs | `python3 scripts/validate_tft_configs.py` |
| `validate_tls_setup.sh` | TLS setup | `./scripts/validate_tls_setup.sh` |
| `validate_training.sh` | Training pipeline | `./scripts/validate_training.sh` |
| `validate_train_script.sh` | Train scripts | `./scripts/validate_train_script.sh` |
---
### 🏥 Health & Diagnostics (5 scripts)
| Script | Purpose | Usage |
|--------|---------|-------|
| `health_check.sh` | Quick health check | `./scripts/health_check.sh` |
| `smoke_test.sh` | Smoke testing | `./scripts/smoke_test.sh` |
| `comprehensive_health_check.sh` | Full system health | `./scripts/comprehensive_health_check.sh` |
| `quick_status.sh` | Quick status | `./scripts/quick_status.sh` |
| `system_resource_monitor.sh` | Resource monitoring | `./scripts/system_resource_monitor.sh` |
---
### 🤖 ML Training & Analysis (7 scripts)
| Script | Purpose | Usage |
|--------|---------|-------|
| `train_all_models_fixed.sh` | Train all models | `./scripts/train_all_models_fixed.sh` |
| `train_tft_production.py` | TFT production training | `python3 scripts/train_tft_production.py` |
| `monitor_hyperopt.sh` | Monitor hyperopt | `./scripts/monitor_hyperopt.sh` |
| `analyze_checkpoints_simple.py` | Checkpoint analysis | `python3 scripts/analyze_checkpoints_simple.py` |
| `compare_checkpoints.py` | Compare checkpoints | `python3 scripts/compare_checkpoints.py` |
| `extract_best_hyperparameters.py` | Extract best params | `python3 scripts/extract_best_hyperparameters.py` |
| `quarterly_retrain.sh` | Quarterly retraining | `./scripts/quarterly_retrain.sh` |
---
### 🔒 Security & Production (7 scripts)
| Script | Purpose | Usage |
|--------|---------|-------|
| `generate-production-secrets.sh` | Generate secrets | `./scripts/generate-production-secrets.sh` |
| `production-security-hardening.sh` | Harden security | `./scripts/production-security-hardening.sh` |
| `security-hardening.sh` | Security config | `./scripts/security-hardening.sh` |
| `setup-docker-secrets.sh` | Docker secrets | `./scripts/setup-docker-secrets.sh` |
| `setup_production_passwords.sh` | Password setup | `./scripts/setup_production_passwords.sh` |
| `export_vault_passwords.sh` | Export Vault passwords | `./scripts/export_vault_passwords.sh` |
| `generate-compliance-report.py` | Compliance report | `python3 scripts/generate-compliance-report.py` |
---
### 🔍 Verification & Setup (7 scripts)
| Script | Purpose | Usage |
|--------|---------|-------|
| `verify_ci_setup.sh` | Verify CI/CD | `./scripts/verify_ci_setup.sh` |
| `verify_ml_dependencies.sh` | Verify ML deps | `./scripts/verify_ml_dependencies.sh` |
| `verify_vault_setup.sh` | Verify Vault | `./scripts/verify_vault_setup.sh` |
| `check_service_binaries.sh` | Check service bins | `./scripts/check_service_binaries.sh` |
| `check_all_available_gpus.py` | Check all GPUs | `python3 scripts/check_all_available_gpus.py` |
| `check_gpu_availability.py` | Check GPU status | `python3 scripts/check_gpu_availability.py` |
| `watch_gpu_availability.sh` | Watch GPU | `./scripts/watch_gpu_availability.sh` |
---
### 🛠️ Utilities (8 scripts)
| Script | Purpose | Usage |
|--------|---------|-------|
| `convert_csv_to_parquet.py` | CSV → Parquet | `python3 scripts/convert_csv_to_parquet.py` |
| `plan_databento_download.sh` | Plan data download | `./scripts/plan_databento_download.sh` |
| `enforce_coverage.sh` | Enforce coverage | `./scripts/enforce_coverage.sh` |
| `run-coverage.sh` | Run coverage | `./scripts/run-coverage.sh` |
| `run_comprehensive_tests.sh` | Comprehensive tests | `./scripts/run_comprehensive_tests.sh` |
| `offline_service_validation.sh` | Offline validation | `./scripts/offline_service_validation.sh` |
| `install_cron.sh` | Install cron jobs | `./scripts/install_cron.sh` |
| `cleanup_deprecated_scripts.sh` | Cleanup scripts | `./scripts/cleanup_deprecated_scripts.sh` |
---
## Common Workflows
### Deploy to Runpod
```bash
# 1. Build Docker image
./scripts/build_docker_images.sh
# 2. Deploy pod
python3 scripts/runpod_deploy.py --gpu-type "RTX A4000"
# 3. Monitor logs
python3 scripts/monitor_logs.py <pod_id>
```
### Local Development
```bash
# 1. Start services
./scripts/start_foxhunt.sh
# 2. Run health checks
./scripts/health_check.sh
# 3. Run smoke tests
./scripts/smoke_test.sh
# 4. Stop services
./scripts/stop_foxhunt.sh
```
### CI/CD Validation
```bash
# 1. Run local CI pipeline
./scripts/local_ci_pipeline.sh
# 2. Verify setup
./scripts/verify_ci_setup.sh
# 3. Validate performance
python3 scripts/validate-performance.py
```
### ML Training
```bash
# 1. Verify ML dependencies
./scripts/verify_ml_dependencies.sh
# 2. Check GPU availability
python3 scripts/check_gpu_availability.py
# 3. Train all models
./scripts/train_all_models_fixed.sh
# 4. Monitor training
./scripts/monitor_hyperopt.sh
```
---
## Deprecated Scripts (Backed Up)
All deprecated scripts moved to:
```
/home/jgrusewski/Work/foxhunt/scripts/archive/cleanup_2025_10_30/
```
**Total Deleted**: 56 scripts (49% reduction)
See `/home/jgrusewski/Work/foxhunt/SCRIPTS_CLEANUP_REPORT_2025_10_30.md` for details.
---
## Maintenance
### Adding New Scripts
1. Create script in `/scripts/`
2. Make executable: `chmod +x scripts/<script>.sh`
3. Update this inventory
4. Document in `scripts/README.md`
### Deprecating Scripts
1. Move to `scripts/archive/deprecated_<date>/`
2. Update this inventory
3. Remove from CI/CD references
4. Update documentation
---
**Last Updated**: 2025-10-30
**Maintained By**: Foxhunt Development Team
Reference in New Issue View Git Blame Copy Permalink