foxhunt/SYMBOL_MIGRATION_GUIDE.md

# Symbol Type Migration Guide

**Last Updated**: 2025-10-16
**Audience**: Foxhunt service developers
**Status**: Production guidelines for Symbol type usage

---

## Quick Reference

### When to Use Symbol vs String

| Context | Use Type | Example |
|---------|----------|---------|
| **API Boundaries** (gRPC, REST) | `Symbol` | Validate at entry point |
| **Database Columns** | `VARCHAR(50)` → `Symbol` | Use `.as_str()` in queries |
| **Internal Logic** | `String` or `Symbol` | Either works, prefer Symbol |
| **CLI Arguments** | `String` | Convert to Symbol after parsing |
| **Proto Definitions** | `string` | Codegen handles conversion |
| **Test Fixtures** | `Symbol::from("ES.FUT")` | Use `From<&str>` trait |

---

## 1. The Symbol Type

### 1.1 What is Symbol?

```rust
// Location: common/src/types.rs:3568
#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[cfg_attr(feature = "database", derive(sqlx::Type))]
pub struct Symbol {
    value: String,  // Private field for encapsulation
}
```

**Purpose**: Type-safe wrapper around `String` for trading instrument identifiers with validation.

**Features**:
- ✅ **Validation**: Rejects empty/invalid symbols
- ✅ **Type Safety**: Prevents mixing with other string types
- ✅ **Zero-Cost**: No runtime overhead vs String
- ✅ **Database**: Works seamlessly with SQLX
- ✅ **Serialization**: Transparent JSON/Binary encoding

### 1.2 Validation Rules

```rust
// Valid symbols
Symbol::new_validated("ES.FUT".to_string()).unwrap();   // ✅ Futures
Symbol::new_validated("AAPL".to_string()).unwrap();     // ✅ Equity
Symbol::new_validated("BTC-USD".to_string()).unwrap();  // ✅ Crypto
Symbol::new_validated("EUR_USD".to_string()).unwrap();  // ✅ Forex

// Invalid symbols
Symbol::new_validated("".to_string()).unwrap_err();           // ❌ Empty
Symbol::new_validated("ES FUT".to_string()).unwrap_err();     // ❌ Space
Symbol::new_validated("ES/FUT".to_string()).unwrap_err();     // ❌ Slash
Symbol::new_validated("A".repeat(51)).unwrap_err();           // ❌ Too long (>50 chars)
```

**Validation Checklist**:
- ✅ Length: 1-50 characters
- ✅ Charset: Alphanumeric + dot (`.`) + hyphen (`-`) + underscore (`_`)
- ✅ No whitespace (leading, trailing, or internal)
- ✅ Case-sensitive (ES.FUT ≠ es.fut)

---

## 2. Creating Symbols

### 2.1 Permissive Constructor (Internal Use)

```rust
use common::types::Symbol;

// No validation - use for trusted sources
let symbol = Symbol::new("ES.FUT".to_string());
let symbol = Symbol::from("ES.FUT");  // Shorthand via From trait
```

**When to use**: Internal logic where symbols are already validated.

### 2.2 Validated Constructor (API Boundaries)

```rust
use common::types::Symbol;

// With validation - use for user input
match Symbol::new_validated(user_input) {
    Ok(symbol) => {
        // Symbol is valid, proceed with business logic
        process_order(symbol);
    }
    Err(e) => {
        // Symbol is invalid, return error to user
        return Err(format!("Invalid symbol: {}", e));
    }
}
```

**When to use**: API handlers, CLI parsing, external data ingestion.

### 2.3 From Trait Conversions

```rust
// String → Symbol (zero-cost)
let s = "ES.FUT".to_string();
let symbol: Symbol = s.into();

// &str → Symbol (allocates String)
let symbol: Symbol = "ES.FUT".into();

// Symbol → String (clones inner value)
let symbol = Symbol::from("ES.FUT");
let s: String = symbol.into();
```

---

## 3. Using Symbols

### 3.1 String Operations

```rust
let symbol = Symbol::from("ES.FUT");

// Get as string slice (zero-cost)
let s: &str = symbol.as_str();       // Primary method
let s: &str = symbol.value();        // Alias
let s: &str = symbol.as_ref();       // Via AsRef trait

// Get as owned String (clones)
let s: String = symbol.to_string();  // Via Display trait
let s: String = symbol.into();       // Via Into trait

// String manipulation
let upper = symbol.to_uppercase();   // "ES.FUT"
let replaced = symbol.replace(".", "_");  // "ES_FUT"
let contains = symbol.contains("ES");     // true
```

### 3.2 Comparison Operations

```rust
let symbol = Symbol::from("ES.FUT");

// Compare with other Symbols
assert_eq!(symbol, Symbol::from("ES.FUT"));

// Compare with &str (no allocation)
assert_eq!(symbol, "ES.FUT");
assert_ne!(symbol, "NQ.FUT");

// Compare with String
let s = "ES.FUT".to_string();
assert_eq!(symbol, s);

// Pattern matching
match symbol.as_str() {
    "ES.FUT" => println!("E-mini S&P 500"),
    "NQ.FUT" => println!("Nasdaq futures"),
    _ => println!("Unknown symbol"),
}
```

### 3.3 Special Helpers

```rust
// Check if empty
let symbol = Symbol::default();  // Empty symbol
assert!(symbol.is_empty());

// Create "NONE" sentinel value
let none = Symbol::none();
assert_eq!(none, "NONE");

// Get as bytes (for hashing, binary protocols)
let bytes = symbol.as_bytes();
```

---

## 4. Database Integration

### 4.1 SQLX Queries

```rust
use common::types::Symbol;
use sqlx::PgPool;

// SELECT query
pub async fn get_order(pool: &PgPool, symbol: &Symbol) -> Result<Order> {
    let order = sqlx::query_as!(
        Order,
        r#"
        SELECT order_id, symbol, quantity, price
        FROM orders
        WHERE symbol = $1
        "#,
        symbol.as_str()  // Use .as_str() for query parameters
    )
    .fetch_one(pool)
    .await?;

    Ok(order)
}

// INSERT query
pub async fn insert_order(pool: &PgPool, order: &Order) -> Result<()> {
    sqlx::query!(
        r#"
        INSERT INTO orders (order_id, symbol, quantity, price)
        VALUES ($1, $2, $3, $4)
        "#,
        order.id,
        order.symbol.as_str(),  // Convert Symbol to &str
        order.quantity,
        order.price
    )
    .execute(pool)
    .await?;

    Ok(())
}
```

### 4.2 Database Schema

```sql
-- Standard symbol column definition
CREATE TABLE orders (
    order_id UUID PRIMARY KEY,
    symbol VARCHAR(50) NOT NULL,
    quantity DECIMAL(20, 8) NOT NULL,
    price DECIMAL(20, 8),
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Add validation constraints (optional)
ALTER TABLE orders
ADD CONSTRAINT symbol_length CHECK (LENGTH(symbol) BETWEEN 1 AND 50);

ALTER TABLE orders
ADD CONSTRAINT symbol_charset CHECK (symbol ~ '^[A-Za-z0-9._-]+$');

-- Index for symbol lookups
CREATE INDEX idx_orders_symbol ON orders(symbol);
```

---

## 5. gRPC/Proto Integration

### 5.1 Proto Definition

```protobuf
// File: trading.proto
message SubmitOrderRequest {
  string symbol = 1;  // ✅ Use string type in proto (not Symbol)
  double quantity = 2;
  double price = 3;
}
```

**Rule**: Always use `string` in proto definitions. Rust codegen handles conversion.

### 5.2 gRPC Handler (Server)

```rust
use common::types::Symbol;
use tonic::{Request, Response, Status};

pub async fn submit_order(
    &self,
    request: Request<SubmitOrderRequest>,
) -> Result<Response<SubmitOrderResponse>, Status> {
    let req = request.into_inner();

    // Validate symbol at API boundary
    let symbol = Symbol::new_validated(req.symbol)
        .map_err(|e| {
            Status::invalid_argument(format!("Invalid symbol: {}", e))
        })?;

    // Use validated Symbol in business logic
    let order = self.service.create_order(symbol, req.quantity, req.price).await
        .map_err(|e| Status::internal(e.to_string()))?;

    Ok(Response::new(SubmitOrderResponse {
        order_id: order.id,
        symbol: order.symbol.to_string(),  // Convert back to String for proto
        ...
    }))
}
```

### 5.3 gRPC Client

```rust
use common::types::Symbol;

pub async fn submit_order(&mut self, symbol: Symbol) -> Result<String> {
    let request = SubmitOrderRequest {
        symbol: symbol.to_string(),  // Convert Symbol → String for proto
        quantity: 100.0,
        price: 4500.0,
    };

    let response = self.client.submit_order(request).await?;

    Ok(response.into_inner().order_id)
}
```

---

## 6. Serialization Examples

### 6.1 JSON Serialization

```rust
use serde::{Deserialize, Serialize};
use common::types::Symbol;

#[derive(Serialize, Deserialize)]
struct Order {
    order_id: String,
    symbol: Symbol,  // Serializes as JSON string
    quantity: f64,
}

// Serialize to JSON
let order = Order {
    order_id: "ord_123".to_string(),
    symbol: Symbol::from("ES.FUT"),
    quantity: 100.0,
};

let json = serde_json::to_string(&order)?;
// Output: {"order_id":"ord_123","symbol":"ES.FUT","quantity":100.0}

// Deserialize from JSON
let json = r#"{"order_id":"ord_123","symbol":"ES.FUT","quantity":100.0}"#;
let order: Order = serde_json::from_str(json)?;
assert_eq!(order.symbol, "ES.FUT");
```

### 6.2 Binary Serialization (Bincode)

```rust
use bincode;
use common::types::Symbol;

let symbol = Symbol::from("ES.FUT");

// Serialize to bytes
let bytes = bincode::serialize(&symbol)?;

// Deserialize from bytes
let symbol2: Symbol = bincode::deserialize(&bytes)?;
assert_eq!(symbol, symbol2);
```

---

## 7. Common Patterns

### 7.1 API Boundary Pattern

```rust
// API Gateway handler
pub async fn handle_request(
    symbol_str: String,  // Raw input from user
) -> Result<Response> {
    // 1. Validate at boundary
    let symbol = Symbol::new_validated(symbol_str)
        .map_err(|e| ApiError::InvalidSymbol(e.to_string()))?;

    // 2. Pass validated Symbol to business logic
    let result = business_logic(symbol).await?;

    // 3. Convert back to String for response
    Ok(Response {
        symbol: result.symbol.to_string(),
        ...
    })
}
```

### 7.2 Service Layer Pattern

```rust
// Business logic accepts Symbol (already validated)
pub async fn create_order(
    &self,
    symbol: Symbol,  // Type-safe, validated input
    quantity: f64,
    price: f64,
) -> Result<Order> {
    // Use symbol directly (no validation needed)
    let order = Order {
        id: Uuid::new_v4().to_string(),
        symbol,  // Direct assignment
        quantity,
        price,
        status: OrderStatus::New,
    };

    // Persist to database
    self.repository.insert_order(&order).await?;

    Ok(order)
}
```

### 7.3 Test Fixture Pattern

```rust
#[cfg(test)]
mod tests {
    use common::types::Symbol;

    fn create_test_symbol() -> Symbol {
        Symbol::from("TEST.SYM")  // Use From trait for brevity
    }

    #[test]
    fn test_order_creation() {
        let symbol = create_test_symbol();
        let order = Order::new(symbol, 100.0, 4500.0);

        assert_eq!(order.symbol, "TEST.SYM");
    }

    #[test]
    fn test_with_real_symbols() {
        // Test with actual DBN symbols
        let symbols = vec![
            Symbol::from("ES.FUT"),
            Symbol::from("NQ.FUT"),
            Symbol::from("ZN.FUT"),
        ];

        for symbol in symbols {
            assert!(symbol.as_str().ends_with(".FUT"));
        }
    }
}
```

---

## 8. Performance Considerations

### 8.1 Zero-Cost Abstractions

```rust
// String storage
let s = "ES.FUT".to_string();  // Heap: 6 bytes, Stack: 24 bytes

// Symbol storage
let sym = Symbol::from("ES.FUT");  // Heap: 6 bytes, Stack: 24 bytes

// Same memory layout! Symbol is a newtype wrapper around String.
```

### 8.2 Conversion Costs

| Operation | Cost | Notes |
|-----------|------|-------|
| `String` → `Symbol` | Zero | Moves pointer, no reallocation |
| `Symbol.as_str()` | Zero | Inlined by compiler |
| `Symbol.to_string()` | Clone | Allocates new String |
| `Symbol::new_validated()` | ~50ns | Length + charset validation |

### 8.3 Best Practices

```rust
// ✅ GOOD: Pass by reference
fn process_order(symbol: &Symbol) {
    // Use .as_str() for string operations
    if symbol.as_str().starts_with("ES") {
        // Process futures order
    }
}

// ✅ GOOD: Take ownership if needed
fn store_order(symbol: Symbol) -> Order {
    Order { symbol, ... }  // Moves Symbol into Order
}

// ❌ BAD: Unnecessary clones
fn process_order(symbol: Symbol) {
    let s = symbol.clone();  // Unnecessary clone
    // Use symbol directly or pass by reference
}

// ❌ BAD: Repeated to_string() calls
fn format_order(symbol: &Symbol) -> String {
    format!("{} {} {}", symbol.to_string(), symbol.to_string(), symbol.to_string())
    // Use symbol directly (implements Display)
    format!("{} {} {}", symbol, symbol, symbol)  // Better
}
```

---

## 9. Migration Checklist

### 9.1 For New Code

- [ ] Use `Symbol` type in struct fields (not `String`)
- [ ] Validate symbols at API boundaries with `new_validated()`
- [ ] Use `.as_str()` for database queries
- [ ] Use `.to_string()` only when returning to external APIs
- [ ] Write tests with real symbols (ES.FUT, NQ.FUT, etc.)

### 9.2 For Existing Code

- [ ] Identify `symbol: String` fields in structs
- [ ] Replace with `symbol: Symbol`
- [ ] Update constructors to accept `Symbol`
- [ ] Update database queries to use `.as_str()`
- [ ] Update gRPC handlers to validate at boundaries
- [ ] Run tests to verify no regressions

### 9.3 Validation Checklist

When adding Symbol validation:

- [ ] Check input is not empty
- [ ] Check length is 1-50 characters
- [ ] Check charset is alphanumeric + `.` + `-` + `_`
- [ ] Check no leading/trailing whitespace
- [ ] Return clear error message on failure
- [ ] Test with real market symbols (ES.FUT, AAPL, BTC-USD)

---

## 10. Troubleshooting

### 10.1 Compilation Errors

**Error**: `cannot convert String to Symbol`

```rust
// ❌ Wrong
let symbol: Symbol = "ES.FUT".to_string();

// ✅ Correct
let symbol: Symbol = Symbol::from("ES.FUT");
let symbol = Symbol::from("ES.FUT".to_string());
```

**Error**: `expected &str, found Symbol`

```rust
// ❌ Wrong
my_function(symbol);  // Expected &str

// ✅ Correct
my_function(symbol.as_str());
```

**Error**: `Symbol does not implement Display`

```rust
// Symbol DOES implement Display - check imports
use std::fmt;

// ✅ Should work
println!("Symbol: {}", symbol);
```

### 10.2 Runtime Errors

**Error**: `ValidationError: Symbol cannot be empty`

```rust
// User provided empty string
let result = Symbol::new_validated("".to_string());
// Handle error gracefully
match result {
    Ok(sym) => process_symbol(sym),
    Err(e) => return Err(ApiError::InvalidInput(e.to_string())),
}
```

**Error**: `ValidationError: Symbol contains invalid characters`

```rust
// User provided symbol with spaces
let result = Symbol::new_validated("ES FUT".to_string());
// Suggest correct format to user
match result {
    Err(e) => {
        return Err(ApiError::InvalidInput(
            format!("{} - Use 'ES.FUT' format", e)
        ));
    }
    Ok(sym) => process_symbol(sym),
}
```

### 10.3 Database Errors

**Error**: `symbol value too long for type character varying(20)`

```sql
-- Database column is too small
ALTER TABLE orders ALTER COLUMN symbol TYPE VARCHAR(50);

-- Or use TEXT for unlimited length (less efficient)
ALTER TABLE orders ALTER COLUMN symbol TYPE TEXT;
```

**Error**: `invalid input syntax for type character varying`

```rust
// Ensure symbol.as_str() is used, not symbol directly
sqlx::query!("INSERT INTO orders (symbol) VALUES ($1)", symbol.as_str())
```

---

## 11. Examples by Service

### 11.1 Trading Service

```rust
// File: services/trading_service/src/lib.rs
use common::types::Symbol;

pub struct TradingService {
    db_pool: PgPool,
}

impl TradingService {
    pub async fn submit_order(
        &self,
        symbol: Symbol,  // Accept validated Symbol
        side: OrderSide,
        quantity: f64,
    ) -> Result<Order> {
        // Use symbol directly (no validation needed)
        let order = Order {
            id: Uuid::new_v4().to_string(),
            symbol,
            side,
            quantity,
            status: OrderStatus::New,
            created_at: Utc::now(),
        };

        // Persist to database
        sqlx::query!(
            "INSERT INTO orders (id, symbol, side, quantity, status) VALUES ($1, $2, $3, $4, $5)",
            order.id,
            order.symbol.as_str(),  // Convert to &str for DB
            order.side as i32,
            order.quantity,
            order.status as i32
        )
        .execute(&self.db_pool)
        .await?;

        Ok(order)
    }
}
```

### 11.2 Risk Service

```rust
// File: risk/src/position_tracker.rs
use common::types::Symbol;
use std::collections::HashMap;

pub struct PositionTracker {
    positions: HashMap<Symbol, Position>,  // Use Symbol as HashMap key
}

impl PositionTracker {
    pub fn update_position(&mut self, symbol: Symbol, quantity: f64) {
        self.positions
            .entry(symbol)  // Symbol implements Hash + Eq
            .and_modify(|pos| pos.quantity += quantity)
            .or_insert(Position { quantity, ..Default::default() });
    }

    pub fn get_position(&self, symbol: &Symbol) -> Option<&Position> {
        self.positions.get(symbol)  // HashMap lookup by Symbol
    }
}
```

### 11.3 ML Service

```rust
// File: ml/src/ensemble/decision.rs
use common::types::Symbol;

pub struct EnsemblePredictor {
    models: Vec<Box<dyn MLModel>>,
}

impl EnsemblePredictor {
    pub async fn predict(&self, symbol: Symbol) -> Result<Prediction> {
        // Generate predictions from all models
        let predictions = futures::future::try_join_all(
            self.models.iter().map(|model| model.predict(&symbol))
        ).await?;

        // Ensemble voting
        let action = self.vote(predictions);

        Ok(Prediction {
            symbol,  // Move Symbol into result
            action,
            confidence: 0.85,
            timestamp: Utc::now(),
        })
    }
}
```

---

## 12. FAQ

**Q: Why not just use String everywhere?**

A: Symbol provides:
- Type safety (prevents mixing with other string types)
- Validation (rejects invalid symbols at API boundaries)
- Documentation (self-documenting code)
- Zero-cost abstraction (no performance penalty)

**Q: When should I use `new()` vs `new_validated()`?**

A: Use `new_validated()` at API boundaries (user input, external data). Use `new()` for internal logic where symbols are already validated.

**Q: Does Symbol work with HashMap/HashSet?**

A: Yes! Symbol implements `Hash` and `Eq`, so it works as a HashMap key or HashSet member.

**Q: Can I compare Symbol with &str directly?**

A: Yes! Symbol implements `PartialEq<&str>`, so `symbol == "ES.FUT"` works without conversion.

**Q: What's the performance impact of Symbol vs String?**

A: Zero. Symbol is a newtype wrapper around String with identical memory layout and zero-cost abstractions.

**Q: How do I test with real market symbols?**

A: Use the From trait: `Symbol::from("ES.FUT")` in tests. See real DBN symbols in `test_data/real/dbn/`.

**Q: Should I migrate existing String fields to Symbol?**

A: Not necessarily. Prioritize API boundaries and new code. Internal String usage is fine if it works.

**Q: Can I use Symbol in proto definitions?**

A: No. Proto definitions must use `string` type. Rust codegen handles conversion to/from Symbol at API boundaries.

---

## 13. References

- **Symbol Implementation**: `/home/jgrusewski/Work/foxhunt/common/src/types.rs:3568`
- **Symbol Tests**: `/home/jgrusewski/Work/foxhunt/common/src/types.rs:4638-4695`
- **Database Schemas**: `/home/jgrusewski/Work/foxhunt/migrations/`
- **Proto Definitions**: `/home/jgrusewski/Work/foxhunt/tli/proto/trading.proto`
- **Real Market Data**: `/home/jgrusewski/Work/foxhunt/test_data/real/dbn/`

---

**Last Updated**: 2025-10-16
**Version**: 1.0
**Status**: Production-ready guidelines