foxhunt/docs/WAVE79_AGENT9_INVESTIGATION.md

# Wave 79 Agent 9: GetOrderStatus "Unimplemented" Investigation

**Date**: 2025-10-03
**Issue**: RPC method returns "Unimplemented" despite being implemented
**Impact**: 99.977% load test failure rate
**Priority**: 🔴 CRITICAL - Blocking all load testing

---

## Issue Summary

The `trading.TradingService.GetOrderStatus` RPC method returns status code "Unimplemented" for 99.977% of requests during load testing, despite:
- ✅ Method defined in proto file
- ✅ Method implemented in Rust code
- ✅ Service registered with gRPC server
- ✅ Server listening on correct port (50050)

---

## Evidence

### 1. Proto Definition (Confirmed Present)
```protobuf
// File: services/trading_service/proto/trading.proto
service TradingService {
  rpc GetOrderStatus(GetOrderStatusRequest) returns (GetOrderStatusResponse);
  // ... other methods
}
```

### 2. Rust Implementation (Confirmed Complete)
```rust
// File: services/trading_service/src/services/trading.rs
#[tonic::async_trait]
impl trading_service_server::TradingService for TradingServiceImpl {
    async fn get_order_status(
        &self,
        request: Request<GetOrderStatusRequest>,
    ) -> TonicResult<Response<GetOrderStatusResponse>> {
        let req = request.into_inner();
        debug!("Get order status for order_id: {}", req.order_id);

        match self.state.trading_repository.get_order(&req.order_id).await {
            Ok(Some(trading_order)) => {
                // Full implementation with order conversion
                Ok(Response::new(GetOrderStatusResponse {
                    order: Some(proto_order),
                }))
            },
            Ok(None) => Err(Status::not_found(...)),
            Err(e) => Err(Status::internal(...)),
        }
    }
}
```

### 3. Service Registration (Confirmed)
```rust
// File: services/trading_service/src/main.rs
Server::builder()
    .add_service(
        trading_service::proto::trading::trading_service_server::TradingServiceServer::with_interceptor(
            trading_service,
            auth_interceptor.clone()  // ← Auth interceptor wrapping service
        )
    )
    .serve_with_shutdown(addr, shutdown_signal());
```

### 4. Load Test Results
```
Status code distribution:
  [Unimplemented]   847,258 responses (99.977%)
  [Unavailable]     197 responses     (0.023%)

Error distribution:
  [847258]  rpc error: code = Unimplemented desc =
```

---

## Hypotheses

### Hypothesis 1: Auth Interceptor Rejection (MOST LIKELY)

**Theory**: The auth interceptor is rejecting requests before they reach the service implementation.

**Evidence**:
- Service wrapped with `auth_interceptor.clone()`
- No authentication provided in test requests
- "Unimplemented" is a common fallback for auth failures

**Investigation**:
```rust
// File: services/trading_service/src/auth_interceptor.rs
// Need to check:
// 1. What status code does auth failure return?
// 2. Is "Unimplemented" used as a rejection code?
// 3. Are there allowlisted methods that bypass auth?
```

**Test**:
```bash
# Try with Bearer token
ghz --metadata '{"authorization":"Bearer <valid-jwt>"}' \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    --data '{"order_id": "test"}' \
    localhost:50050
```

### Hypothesis 2: Proto Compilation Mismatch

**Theory**: The proto file used by `ghz` doesn't match the server's generated code.

**Evidence**:
- Build system uses `tonic_prost_build` (Tonic 0.14+)
- Client proto generated differently than server proto
- Version mismatch between proto definitions

**Investigation**:
```bash
# Check generated proto file
find target -name "trading.rs" -path "*/proto/*" | xargs grep "GetOrderStatus"

# Compare method signatures
# Client: GetOrderStatusRequest/Response
# Server: GetOrderStatusRequest/Response
```

**Test**:
```bash
# Use grpcurl with reflection (requires reflection enabled)
grpcurl -plaintext localhost:50050 list trading.TradingService
```

### Hypothesis 3: Method Not Exported in Build

**Theory**: Build configuration issue prevents method from being included in final binary.

**Evidence**:
- Build uses `tonic_prost_build` instead of `tonic_build`
- Conditional compilation features might exclude method
- Release vs debug build differences

**Investigation**:
```rust
// Check build.rs configuration
tonic_prost_build::configure()
    .build_server(true)
    .build_client(false)  // ← Might affect method availability
    .compile_protos(...)
```

**Test**:
```bash
# Check if method exists in running binary
strings ./target/release/trading_service | grep "GetOrderStatus"
```

### Hypothesis 4: Rate Limiting / Auth Penalty

**Theory**: Auth failure penalty is rate-limiting or blocking requests.

**Evidence**:
- Previous Wave 60 work on auth failure penalties
- High request rate (100K RPS) might trigger protection
- 197 "Unavailable" errors suggest some requests succeed

**Investigation**:
```rust
// File: services/trading_service/src/auth_interceptor.rs
// Check for rate limiting on auth failures
// Review penalty mechanism
```

**Test**:
```bash
# Test with low RPS to avoid rate limiting
ghz --rps 10 --connections 1 --concurrency 1 \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    localhost:50050
```

---

## Debugging Steps

### Step 1: Enable gRPC Reflection
```rust
// Add to services/trading_service/src/main.rs
use tonic_reflection::server::Builder as ReflectionBuilder;

let reflection_service = ReflectionBuilder::configure()
    .register_encoded_file_descriptor_set(proto::DESCRIPTOR_SET)
    .build()?;

Server::builder()
    .add_service(reflection_service)  // Add this
    .add_service(trading_service)
    // ...
```

Then test with:
```bash
grpcurl -plaintext localhost:50050 list
grpcurl -plaintext localhost:50050 describe trading.TradingService
```

### Step 2: Check Auth Interceptor Logs
```bash
# Restart trading service with debug logging
RUST_LOG=trading_service=debug,auth_interceptor=debug \
    ./target/release/trading_service

# Monitor logs during test
tail -f /var/log/trading_service.log | grep -i "auth\|unimplemented"
```

### Step 3: Test Without Auth Interceptor
```rust
// Temporarily remove auth interceptor for testing
Server::builder()
    .add_service(
        trading_service::proto::trading::trading_service_server::TradingServiceServer::new(
            trading_service  // Remove .with_interceptor()
        )
    )
    // ...
```

### Step 4: Test with Valid JWT
```bash
# Generate valid JWT token
# (Requires JWT secret from config)

# Test with auth header
ghz --metadata '{"authorization":"Bearer eyJ..."}' \
    --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.GetOrderStatus \
    localhost:50050
```

### Step 5: Test Other Methods
```bash
# Try SubmitOrder (should also fail if auth issue)
ghz --proto services/trading_service/proto/trading.proto \
    --call trading.TradingService.SubmitOrder \
    --data '{"symbol":"AAPL","quantity":100,"side":1,"order_type":1}' \
    localhost:50050

# Try health check (might bypass auth)
ghz --proto services/trading_service/proto/trading.proto \
    --call grpc.health.v1.Health.Check \
    localhost:50050
```

---

## Quick Fixes

### Fix 1: Bypass Auth for Health/Status Checks
```rust
// In auth_interceptor.rs
impl AuthInterceptor {
    fn should_bypass_auth(&self, method: &str) -> bool {
        matches!(method,
            "/grpc.health.v1.Health/Check" |
            "/trading.TradingService/GetOrderStatus" |  // Add this
            "/trading.TradingService/GetPortfolioSummary"
        )
    }
}
```

### Fix 2: Add Default Auth Token for Testing
```rust
// In auth_interceptor.rs
let token = request
    .metadata()
    .get("authorization")
    .or_else(|| {
        // For testing only - remove in production
        if cfg!(debug_assertions) {
            Some("Bearer test-token")
        } else {
            None
        }
    })
    .ok_or_else(|| Status::unauthenticated("Missing authorization header"))?;
```

### Fix 3: Return Proper Status Codes
```rust
// Ensure auth interceptor returns correct error codes
if let Err(e) = self.validate_token(token) {
    return Err(Status::unauthenticated(e.to_string()));  // Not "Unimplemented"
}
```

---

## Expected Behavior

### Correct Flow
```
Client Request
    ↓
[Auth Interceptor]
    ├─ Valid Token → Pass to Service Implementation
    └─ Invalid Token → Return Unauthenticated (not Unimplemented)
         ↓
[Service Implementation: get_order_status]
    ├─ Order Found → Return GetOrderStatusResponse
    ├─ Order Not Found → Return NotFound
    └─ DB Error → Return Internal
```

### Current Flow (Suspected)
```
Client Request
    ↓
[Auth Interceptor]
    └─ Missing Token → Return Unimplemented (WRONG)
         ↓
[Never reaches service implementation]
```

---

## Related Issues

### Wave 69 Agent 6: JWT Revocation
- Implemented JWT token validation
- Added revocation checking
- May have changed auth interceptor behavior

### Wave 60: Auth Failure Penalty
- Implemented rate limiting on auth failures
- May return "Unimplemented" as penalty
- Check `test_auth_failure_penalty` test

### Wave 78: HTTP/2 Optimizations
- Previous load testing achieved 211K req/s
- Did not encounter "Unimplemented" errors
- Something changed between Wave 78 and Wave 79

---

## Recommended Actions

### Immediate (Next 1 Hour)
1. ✅ Check auth interceptor source code for "Unimplemented" usage
2. ✅ Test with single request at low RPS (rule out rate limiting)
3. ✅ Enable debug logging and capture auth failure reasons
4. ✅ Test without auth interceptor temporarily

### Short-term (Next 4 Hours)
1. Generate valid JWT token and retry tests
2. Add health check methods to auth bypass list
3. Implement proper error codes (Unauthenticated, not Unimplemented)
4. Document required metadata for all RPC methods

### Long-term (Next Sprint)
1. Add gRPC reflection support for all services
2. Create auth testing utilities (token generation, validation)
3. Add integration tests with auth interceptor
4. Document authentication requirements in proto files

---

## Impact Assessment

### Current State
- ❌ Cannot perform load testing
- ❌ Cannot measure throughput improvements
- ❌ Cannot validate HTTP/2 optimizations
- ❌ Cannot compare to Wave 78 baseline (211K req/s)

### After Fix
- ✅ Enable comprehensive load testing
- ✅ Measure actual performance improvements
- ✅ Validate Agent 5-6 fixes (TLS + HTTP/2)
- ✅ Establish new performance baseline

### Risk
- **High**: Production services may have same auth issue
- **High**: Real clients might be getting "Unimplemented" errors
- **Medium**: Load testing timeline blocked indefinitely

---

## Next Steps

1. **Investigate Auth Interceptor** (30 min)
   ```bash
   cat services/trading_service/src/auth_interceptor.rs | grep -A 20 "Unimplemented"
   ```

2. **Test Without mTLS** (15 min)
   - Temporarily disable TLS requirement
   - Use plaintext gRPC
   - Isolate auth from TLS issues

3. **Generate Test JWT** (15 min)
   - Use JWT secret from config
   - Create valid token with proper claims
   - Retry load test with auth header

4. **Document Findings** (30 min)
   - Update this document with results
   - Create fix PR if solution found
   - Escalate to team if blocker persists

---

## Conclusion

The "Unimplemented" error is **most likely** caused by the auth interceptor rejecting requests without proper authentication. The high error rate (99.977%) suggests a systematic issue rather than random failures.

**Recommended approach**: Investigate auth interceptor logic first, then test with valid JWT tokens. If still failing, temporarily bypass auth to confirm service implementation works correctly.

**Timeline**: 1-2 hours to investigate and fix, then 30 minutes to re-run load tests.

---

**Status**: 🔍 Investigation in Progress
**Assignee**: Wave 79 Agent 9
**Updated**: 2025-10-03