gitadmin/foxhunt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3db41edf704bd1663309d2bb4dd3eee263e14e77
foxhunt/database/migrations/NOTIFY_ARCHITECTURE_DIAGRAM.md
jgrusewski 18944be360 📊 Wave 73: Production Validation (12 parallel agents) 
All 12 validation agents complete:
- Agent 1: E2E auth testing (11/11 tests pass, 8-layer validation)
- Agent 2: Load testing framework ready (4 scenarios documented)
- Agent 3: Docker deployment (6/6 infra services healthy)
- Agent 4: Database integration (4 migrations, 6 NOTIFY channels, RBAC)
- Agent 5: TLI client integration (JWT auth, OS keyring, API Gateway)
- Agent 6: Performance profiling (978ns pipeline, 3 optimization recommendations)
- Agent 7: Security penetration testing (OWASP Top 10, 3 critical findings)
- Agent 8: gRPC proxy testing (3 proxies, 100% test pass, 5-8μs overhead)
- Agent 9: Monitoring validation (Prometheus + Grafana, 5 issues identified)
- Agent 10: Rate limiting stress test (8/8 tests pass, 99% attack mitigation)
- Agent 11: Production readiness (7/9 criteria, 2 P0 blockers identified)
- Agent 12: Documentation audit (92% complete, A- grade, production ready)

Deliverables:
- 30+ validation reports created (150+ KB documentation)
- All 5 Dockerfiles updated with complete workspace
- Redis/PostgreSQL integration tests operational
- Comprehensive performance profiling completed
- Security vulnerabilities documented with remediation

🔴 CRITICAL P0 BLOCKERS IDENTIFIED:
1. Audit trail persistence (trading_engine/src/compliance/audit_trails.rs:857)
   - Impact: SOX/MiFID II compliance violation
   - Status: Events not saved to database (only printed)

2. Test suite validation timeout
   - Historical: 1,919/1,919 tests passing (100%)
   - Current: Timeout after 2 minutes
   - Impact: Cannot certify regression-free state

⚠️ CRITICAL SECURITY VULNERABILITIES:
1. Authentication DISABLED (services/trading_service/src/main.rs:298-302)
2. Execution engine PANICS (execution_engine.rs:661,667,674)
3. Audit trail persistence (covered above)

Production Decision: CONDITIONAL GO
- Must fix 2 P0 blockers before production deployment
- 7/9 production criteria met (78%)
- SOX: 87.5% compliant, MiFID II: 87.5% compliant
- Documentation: 92% complete (4,329 production lines)

Next Wave: Address P0 blockers + performance optimization
2025-10-03 13:35:14 +02:00

336 lines
18 KiB
Markdown
Raw Blame History

# NOTIFY/LISTEN ARCHITECTURE DIAGRAM
## Channel Routing Flow
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ PostgreSQL NOTIFY/LISTEN │
│ Hot-Reload Configuration System │
└─────────────────────────────────────────────────────────────────────────────┘
┌──────────────┐
│ config_ │
│ settings │
│ table │
└──────┬───────┘
│
┌────────────────┴────────────────┐
│ INSERT / UPDATE / DELETE │
│ (tr_config_settings_notify) │
└────────────────┬────────────────┘
│
▼
┌────────────────────────┐
│ notify_config_change() │
│ (TRIGGER FUNCTION) │
└────────┬───────────────┘
│
┌──────────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
Extract category Build JSON Determine
from config_key payload service name
(split on '.') (CASE logic)
│ │ │
└──────────────────┴──────────────────┘
│
▼
┌──────────────────────┐
│ Route to Channel(s) │
└──────────┬───────────┘
│
┌──────────────────────────────┼──────────────────────────────┐
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ config_changed│ │ config_changed│ │ config_changed│
│ _trading │ │ _backtesting │ │ _ml_training │
└───────┬───────┘ └───────┬───────┘ └───────┬───────┘
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ Trading │ │ Backtesting │ │ ML Training │
│ Service │ │ Service │ │ Service │
│ (Rust) │ │ (Rust) │ │ (Rust) │
└───────────────┘ └───────────────┘ └───────────────┘
- Risk mgmt - Strategy sim - Model training
- Execution - Backtest runs - Inference
- Compliance - Model cache
│ ▼ │
▼ ┌───────────────┐ ▼
┌───────────────┐ │ config_changed│ ┌───────────────┐
│ config_changed│ │ _api_gateway │ │ config_changed│
│ _global │ └───────┬───────┘ │ _global │
└───────┬───────┘ │ └───────┬───────┘
│ ▼ │
│ ┌───────────────┐ │
│ │ API Gateway │ │
│ │ Service │ │
│ │ (Rust) │ │
│ └───────────────┘ │
│ - Auth/JWT │
│ - Rate limiting │
│ - RBAC │
│ │
└────────────────────────┬───────────────────────────────┘
│
▼
┌───────────────┐
│ Monitoring │
│ Dashboard │
│ (All changes) │
└───────────────┘
```
---
## RBAC NOTIFY Flow
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ RBAC Permission Change Events │
└─────────────────────────────────────────────────────────────────────────────┘
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ roles │ │ permissions │ │ role_ │ │ user_roles │
│ table │ │ table │ │ permissions │ │ table │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │ │
│ INSERT/UPDATE/ │ INSERT/UPDATE/ │ INSERT/UPDATE/ │ INSERT/UPDATE/
│ DELETE │ DELETE │ DELETE │ DELETE
│ │ │ │
└──────────────────┴──────────────────┴──────────────────┘
│
▼
┌─────────────────────────┐
│ notify_permission_ │
│ change() │
│ (TRIGGER FUNCTION) │
└─────────┬───────────────┘
│
│ Build JSON payload:
│ - operation (INSERT/UPDATE/DELETE)
│ - table name
│ - role_id, permission_id, user_id
│ - timestamp
│
▼
┌─────────────────────────┐
│ pg_notify() │
│ 'permissions_changed' │
└─────────┬───────────────┘
│
┌───────────────────┴───────────────────┐
│ │
▼ ▼
┌───────────────┐ ┌───────────────┐
│ API Gateway │ │ config_changed│
│ Service │ │ _global │
└───────┬───────┘ └───────┬───────┘
│ │
│ Actions: │
│ 1. Invalidate permission cache │
│ 2. Reload RBAC rules │
│ 3. Update in-memory ACL │
│ ▼
│ ┌───────────────┐
│ │ Monitoring │
│ │ Dashboard │
└──────────────────────────────┴───────────────┘
│
▼
┌──────────────────┐
│ Audit Log │
│ - Who changed │
│ - What changed │
│ - When changed │
└──────────────────┘
```
---
## Channel Routing Logic
### Config Key → Channel Mapping
| Config Key Prefix | Channel | Service | Example Keys |
|-------------------|---------|---------|--------------|
| `risk.*` | `config_changed_trading` | Trading Service | `risk.max_daily_loss` |
| `compliance.*` | `config_changed_trading` | Trading Service | `compliance.sox_enabled` |
| `execution.*` | `config_changed_trading` | Trading Service | `execution.slippage_tolerance` |
| `circuit.*` | `config_changed_trading` | Trading Service | `circuit.activation_threshold` |
| `strategy.*` | `config_changed_backtesting` | Backtesting Service | `strategy.lookback_days` |
| `backtest.*` | `config_changed_backtesting` | Backtesting Service | `backtest.data_window` |
| `ml.*` | `config_changed_ml_training` | ML Training Service | `ml.model_cache_ttl` |
| `training.*` | `config_changed_ml_training` | ML Training Service | `training.batch_size` |
| `model.*` | `config_changed_ml_training` | ML Training Service | `model.mamba2.lr` |
| `auth.*` | `config_changed_api_gateway` | API Gateway | `auth.jwt_expiry` |
| `api.*` | `config_changed_api_gateway` | API Gateway | `api.rate_limit_rpm` |
| `mfa.*` | `config_changed_api_gateway` | API Gateway | `mfa.enforce_for_roles` |
| `system.*` | `config_changed_global` | All Services | `system.latency_target_ns` |
| `s3.*` | `config_changed_global` | All Services | `s3.bucket_name` |
| `*` (unknown) | `config_changed_global` | All Services | Fallback for unknown |
---
## Payload Structure Examples
### Config Change Payload
```json
{
"operation": "UPDATE",
"table": "config_settings",
"key": "risk.max_daily_loss",
"value": "100000",
"old_value": "50000",
"category": "risk",
"timestamp": 1730000000.123,
"id": "550e8400-e29b-41d4-a716-446655440000"
}
```
### Permission Change Payload
```json
{
"operation": "INSERT",
"table": "role_permissions",
"timestamp": 1730000000.123,
"role_id": "550e8400-e29b-41d4-a716-446655440001",
"permission_id": "550e8400-e29b-41d4-a716-446655440002",
"user_id": null
}
```
---
## Multi-Service Notification Example
### Model Configuration Change
```
┌──────────────┐
│ model_config │
│ table │
└──────┬───────┘
│ UPDATE is_active = true
│ WHERE name = 'mamba2'
│
▼
┌────────────────────────┐
│ notify_model_config_ │
│ change() │
└────────┬───────────────┘
│
├──────────────────────────────────┐
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ config_changed_ │ │ config_changed_ │
│ ml_training │ │ trading │
└─────────┬───────────┘ └─────────┬───────────┘
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ ML Training Service │ │ Trading Service │
│ - Update model list │ │ - Reload model │
│ - Mark as active │ │ - Invalidate cache │
└─────────────────────┘ └─────────────────────┘
```
---
## Service Integration Points
### Trading Service
**Listens On**: `config_changed_trading`, `config_changed_global`
**Reloads**:
- Risk limits (max_daily_loss, position_limit_pct)
- Execution settings (slippage_tolerance, order_timeout)
- Compliance rules (sox_enabled, mifid2_enabled)
- Circuit breaker thresholds
### Backtesting Service
**Listens On**: `config_changed_backtesting`, `config_changed_global`
**Reloads**:
- Strategy parameters (lookback_days, indicators)
- Historical data windows
- Simulation settings (commission, slippage)
### ML Training Service
**Listens On**: `config_changed_ml_training`, `config_changed_global`
**Reloads**:
- Model configurations (learning_rate, batch_size)
- Training parameters (epochs, early_stopping)
- Model cache TTL
- Inference timeouts
### API Gateway
**Listens On**: `config_changed_api_gateway`, `permissions_changed`, `config_changed_global`
**Reloads**:
- JWT expiry and signing keys
- Rate limits (per-minute, per-hour)
- MFA enforcement rules
- RBAC permission cache
---
## Performance Characteristics
| Metric | Value | Notes |
|--------|-------|-------|
| **NOTIFY Latency** | <10ms | PostgreSQL internal |
| **Service Receive** | <50ms | Network + deserialization |
| **Cache Invalidation** | <1ms | In-memory operation |
| **Config Reload** | <100ms | Parse + apply new config |
| **Total Hot-Reload Time** | <200ms | End-to-end |
---
## Monitoring and Debugging
### Global Channel Monitoring
All configuration changes are sent to `config_changed_global` for centralized monitoring:
```sql
-- Terminal: Listen to all config changes
LISTEN config_changed_global;
-- Payload received for any config update:
-- {operation, table, key, value, old_value, category, timestamp, id}
```
### Permission Change Monitoring
```sql
-- Terminal: Listen to permission changes
LISTEN permissions_changed;
-- Payload received for RBAC updates:
-- {operation, table, timestamp, role_id, permission_id, user_id}
```
---
## Security Considerations
1. **NOTIFY payload size**: Limited to 8KB
- Config values should be reasonably sized
- Large configs should use references
2. **No authentication on NOTIFY**:
- All database users can receive notifications
- Services must validate config before applying
3. **Idempotency**:
- Services should handle duplicate NOTIFY events
- Use timestamp/version for de-duplication
4. **Graceful degradation**:
- Services cache last-known-good config
- Continue operating if NOTIFY fails
- Periodic config refresh as fallback
---
**Diagram Version**: 1.0
**Last Updated**: 2025-10-03
**Wave**: 73 Agent 4
Reference in New Issue View Git Blame Copy Permalink