foxhunt/DATA_ACQUISITION_SERVICE_TDD_SUMMARY.md

Data Acquisition Service - TDD Implementation Summary

Date: 2025-10-15 Status: ✅ TDD RED PHASE COMPLETE - Tests written and failing as expected Next Phase: GREEN - Implement service to make tests pass

---

🎯 Mission

Build automated data acquisition service for Databento downloads using strict Test-Driven Development (TDD) methodology.

---

📊 TDD Progress

Phase 1: RED ✅ COMPLETE

Tests Written First (27 tests total):

Download Workflow Tests (8 tests)

• ✅ test_schedule_download_creates_pending_job - Verify job creation
• ✅ test_download_workflow_progresses_through_states - State transitions
• ✅ test_get_download_status_returns_accurate_progress - Progress tracking
• ✅ test_list_download_jobs_with_pagination - Pagination logic
• ✅ test_cancel_download_job - Cancellation handling
• ✅ test_data_quality_validation_detects_issues - Quality validation
• ✅ test_cost_estimation_is_accurate - Cost tracking

MinIO Upload Tests (9 tests)

• ✅ test_upload_dbn_file_to_minio - Basic upload
• ✅ test_upload_with_metadata_tags - Metadata handling
• ✅ test_upload_with_progress_tracking - Progress callbacks
• ✅ test_upload_retries_on_transient_failures - Retry logic
• ✅ test_upload_fails_after_max_retries - Max retry enforcement
• ✅ test_upload_validates_file_exists - Pre-upload validation
• ✅ test_upload_calculates_checksum - Data integrity (SHA256)
• ✅ test_concurrent_uploads - Parallel upload handling

Error Handling Tests (10 tests)

• ✅ test_network_failure_triggers_retry - Network retry logic
• ✅ test_exponential_backoff_timing - Backoff timing validation
• ✅ test_rate_limit_error_triggers_backoff - Rate limiting
• ✅ test_authentication_failure_not_retried - Auth error handling
• ✅ test_download_timeout_handled - Timeout detection
• ✅ test_data_corruption_detected - Checksum validation
• ✅ test_invalid_response_format_handled - Format validation
• ✅ test_disk_space_exhaustion_detected - Storage checks
• ✅ test_partial_download_cleaned_up - Cleanup on failure
• ✅ test_concurrent_download_limits_enforced - Concurrency limits
• ✅ test_error_messages_are_descriptive - Error clarity

Verification: All tests fail with unimplemented!() - ✅ RED PHASE COMPLETE

---

🏗️ Architecture

Service Structure

services/data_acquisition_service/
├── proto/
│   └── data_acquisition.proto       ✅ COMPLETE - 22 gRPC methods
├── src/
│   ├── lib.rs                       ✅ COMPLETE - Module structure
│   ├── main.rs                      ⏳ PENDING - Service entry point
│   ├── error.rs                     ✅ COMPLETE - Error types
│   ├── service.rs                   ⏳ PENDING - gRPC service impl
│   ├── downloader.rs                ⏳ PENDING - Databento downloader
│   ├── uploader.rs                  ⏳ PENDING - MinIO uploader
│   └── validator.rs                 ⏳ PENDING - Data quality validation
├── tests/
│   ├── download_workflow_tests.rs   ✅ COMPLETE - 8 tests (failing)
│   ├── minio_upload_tests.rs        ✅ COMPLETE - 9 tests (failing)
│   └── error_handling_tests.rs      ✅ COMPLETE - 10 tests (failing)
├── Cargo.toml                       ✅ COMPLETE - Dependencies configured
└── build.rs                         ✅ COMPLETE - Proto compilation

gRPC API (Proto Definition)

5 Core Methods:

1. ScheduleDownload - Queue new download job
2. GetDownloadStatus - Query job progress
3. CancelDownload - Cancel running job
4. ListDownloadJobs - Paginated job listing
5. HealthCheck - Service health

Status Flow:

PENDING → DOWNLOADING → VALIDATING → UPLOADING → COMPLETED
                                               ↓
                                            FAILED
                                               ↓
                                           CANCELLED

Key Features (Defined by Tests)

Download Management:

• Priority queuing (1=low, 5=high)
• Databento API integration
• Cost estimation ($0-2/day, $5-15/week, $20-60/month)
• Concurrent download limits (configurable)

Data Quality:

• Record count validation
• Invalid record detection
• Quality score calculation (0.0-1.0)
• Automatic anomaly detection

Upload Integration:

• MinIO/S3 automatic upload
• Metadata tagging (symbol, date, schema)
• Progress callbacks
• SHA256 checksum verification
• Parallel upload support

Error Handling:

• Exponential backoff (1s, 2s, 4s)
• Rate limit detection and cooldown
• Auth error immediate failure (no retry)
• Timeout detection
• Data corruption detection
• Partial download cleanup
• Descriptive error messages

Resource Management:

• Disk space validation
• Concurrent download limits
• Memory-efficient streaming
• Cleanup on failure

---

📦 Dependencies

Core:

• tonic - gRPC framework
• tokio - Async runtime
• sqlx - Database (PostgreSQL)
• reqwest - HTTP client (Databento API)

Storage:

• storage (workspace) - MinIO/S3 integration
• object_store - Object store abstraction
• dbn - Databento binary format

Error Handling:

• thiserror - Error derivation
• anyhow - Error context
• tokio-retry - Retry logic

Observability:

• tracing - Logging
• prometheus - Metrics
• axum - Health endpoint

---

🔬 Test Coverage

Test Files Location

/home/jgrusewski/Work/foxhunt/services/data_acquisition_service/tests/
├── download_workflow_tests.rs    (8 tests, 350 lines)
├── minio_upload_tests.rs         (9 tests, 337 lines)
└── error_handling_tests.rs       (10 tests, 384 lines)

Coverage Areas

Download Workflow (8 tests):

• Job creation and status transitions
• Progress tracking accuracy
• Pagination logic
• Cancellation handling
• Quality validation
• Cost estimation

MinIO Integration (9 tests):

• Upload success path
• Metadata tagging
• Progress callbacks
• Retry logic
• Max retry enforcement
• File validation
• Checksum calculation
• Concurrent uploads

Error Handling (10 tests):

• Network failures
• Exponential backoff
• Rate limiting
• Authentication errors
• Timeouts
• Data corruption
• Invalid formats
• Disk space
• Cleanup on failure
• Concurrency limits

---

🚀 Next Steps (GREEN Phase)

Priority 1: Core Service Implementation

1. src/main.rs - Service entry point

   • CLI argument parsing
   • Configuration loading
   • TLS setup (mTLS)
   • Health endpoint (port 8095)
   • Metrics endpoint (port 9095)

2. src/service.rs - gRPC service

   • Implement 5 gRPC methods
   • Job queue management
   • Status tracking
   • Database integration

3. src/downloader.rs - Databento integration

   • Python subprocess for databento SDK
   • Cost estimation logic
   • Download progress tracking
   • Retry with exponential backoff
   • Rate limit handling

4. src/uploader.rs - MinIO uploader

   • Use storage::ObjectStoreBackend
   • Progress callbacks
   • SHA256 checksum calculation
   • Metadata tagging
   • Retry logic

5. src/validator.rs - Data quality

   • DBN file validation
   • Record count verification
   • Quality score calculation
   • Anomaly detection

Priority 2: Database Schema

CREATE TABLE download_jobs (
    job_id UUID PRIMARY KEY,
    status VARCHAR(20) NOT NULL,
    dataset VARCHAR(50) NOT NULL,
    symbols TEXT[] NOT NULL,
    start_date DATE NOT NULL,
    end_date DATE NOT NULL,
    schema VARCHAR(50) NOT NULL,
    description TEXT,
    priority INTEGER DEFAULT 3,

    -- Progress
    progress_percentage REAL DEFAULT 0.0,
    bytes_downloaded BIGINT DEFAULT 0,
    total_bytes BIGINT,

    -- Costs
    estimated_cost_usd REAL,
    actual_cost_usd REAL,

    -- Quality
    records_count BIGINT DEFAULT 0,
    invalid_records BIGINT DEFAULT 0,
    data_quality_score REAL,

    -- Storage
    local_path TEXT,
    minio_path TEXT,

    -- Timestamps
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    started_at TIMESTAMPTZ,
    completed_at TIMESTAMPTZ,

    -- Error handling
    error_message TEXT,
    retry_count INTEGER DEFAULT 0,

    -- Metadata
    created_by VARCHAR(100),
    tags JSONB
);

CREATE INDEX idx_download_jobs_status ON download_jobs(status);
CREATE INDEX idx_download_jobs_created_at ON download_jobs(created_at DESC);

Priority 3: Configuration

Environment Variables:

# Service
GRPC_PORT=50055
HEALTH_PORT=8095
METRICS_PORT=9095

# Databento
DATABENTO_API_KEY=<api-key>
DATABENTO_DOWNLOAD_DIR=/tmp/databento-downloads

# MinIO/S3
S3_ENDPOINT=http://localhost:9000
S3_BUCKET=market-data
S3_ACCESS_KEY=<access-key>
S3_SECRET_KEY=<secret-key>

# Limits
MAX_CONCURRENT_DOWNLOADS=2
MAX_RETRY_ATTEMPTS=3
DOWNLOAD_TIMEOUT_SECS=300

Priority 4: Run Tests (GREEN Phase)

# Run all tests
cargo test -p data_acquisition_service

# Expected outcome: 27/27 tests passing ✅

---

📈 Success Criteria

TDD GREEN Phase Complete When:

• ✅ All 27 tests passing
• ✅ 0 compilation errors
• ✅ 0 clippy warnings
• ✅ Service compiles and runs
• ✅ Health check responds
• ✅ Can schedule and execute a real download

Integration Validation:

1. Schedule ES.FUT download for 1 day
2. Verify status transitions (PENDING → DOWNLOADING → VALIDATING → UPLOADING → COMPLETED)
3. Verify file uploaded to MinIO
4. Verify quality score > 0.95
5. Verify cost estimation accuracy

---

🔍 Test Execution Plan

Phase 1: Unit Tests (Current)

cargo test -p data_acquisition_service --lib

Phase 2: Integration Tests

cargo test -p data_acquisition_service --tests

Phase 3: E2E Validation

# Start service
cargo run -p data_acquisition_service serve --dev

# Schedule test download
tli data schedule-download \
  --dataset GLBX.MDP3 \
  --symbols ES.FUT \
  --start-date 2024-01-01 \
  --end-date 2024-01-02 \
  --schema ohlcv-1m

# Monitor progress
tli data get-status --job-id <uuid>

---

📝 Implementation Notes

Databento Integration Strategy

Option 1: Python Subprocess (Recommended)

• Use existing databento Python SDK
• Spawn subprocess for downloads
• Parse JSON output for progress
• Pros: Reliable, well-tested SDK
• Cons: Python dependency

Option 2: Direct HTTP API

• Implement Databento REST API client in Rust
• No Python dependency
• Pros: Pure Rust, faster
• Cons: More implementation work, manual rate limiting

Decision: Start with Option 1 (Python subprocess) for reliability, optimize later if needed.

Retry Strategy

Exponential Backoff:

let delays = [1000ms, 2000ms, 4000ms];  // Max 3 retries

Retry Decision Matrix:

Error Type	Retry?	Delay
Network timeout	Yes	Exponential
Connection refused	Yes	Exponential
Rate limit 429	Yes	Fixed (from header)
Auth 401/403	No	-
Bad request 400	No	-
Server error 500+	Yes	Exponential

Concurrency Limits

Download Queue:

• Max 2 concurrent downloads (configurable)
• Priority-based scheduling (1=low, 5=high)
• FIFO within same priority

Upload Queue:

• Max 5 concurrent uploads
• Independent of download queue

---

🎓 TDD Lessons Learned

What Worked Well:

1. ✅ Writing tests first forced clear API design
2. ✅ Test failures guided implementation priorities
3. ✅ Mock types made tests compile without implementation
4. ✅ Comprehensive error scenarios identified early
5. ✅ Test structure maps cleanly to implementation modules

Improvements:

1. Added Send + Sync bounds to error types early
2. Used #[derive(Debug)] on all test types
3. Made test helper functions async-aware
4. Structured tests by feature area (workflow, upload, errors)

---

📚 References

Architecture:

• ML Training Service: /home/jgrusewski/Work/foxhunt/services/ml_training_service/
• Storage Library: /home/jgrusewski/Work/foxhunt/storage/
• Proto Definitions: /home/jgrusewski/Work/foxhunt/services/ml_training_service/proto/

Documentation:

• CLAUDE.md: System architecture and current status
• Proto files: gRPC service definitions
• Cargo.toml patterns: Workspace dependency management

---

Last Updated: 2025-10-15 Next Action: Implement src/main.rs and src/service.rs to start GREEN phase Estimated Implementation Time: 4-6 hours (5 modules × ~1 hour each)