foxhunt/POSTGRESQL_PARTITION_ROUTING_INSIGHTS.md

# PostgreSQL Partition Routing: Critical Insights from Wave 128

**Date**: 2025-10-09  
**Context**: Wave 128 Agent 19 - E2E validation revealed partition routing failures

---

## Problem Summary

Database triggers were inserting rows into partitioned tables WITHOUT the partition key column, causing partition routing failures.

**Error Message**:
```
ERROR: no partition of relation "trading_events" found for row
DETAIL: Partition key of the failing row contains (event_date) = (null)
```

---

## Root Cause: PostgreSQL Constraint Ordering

### Key Discovery: Constraint Evaluation Order

PostgreSQL evaluates constraints in this order:
1. **NOT NULL constraints** ← Evaluated FIRST
2. **CHECK constraints**
3. **BEFORE INSERT triggers** ← Evaluated AFTER NOT NULL check
4. **AFTER INSERT triggers**

**Critical Implication**: You CANNOT rely on a BEFORE INSERT trigger to populate a NOT NULL partition key column.

### Why This Failed

Our initial approach:
```sql
-- Table definition
CREATE TABLE trading_events (
    event_date DATE NOT NULL  -- Partition key with NOT NULL constraint
) PARTITION BY RANGE (event_date);

-- BEFORE INSERT trigger (designed to auto-populate event_date)
CREATE TRIGGER tg_set_trading_event_date
    BEFORE INSERT ON trading_events
    FOR EACH ROW
    EXECUTE FUNCTION set_trading_event_date();

-- Function that sets event_date from event_timestamp
CREATE FUNCTION set_trading_event_date() RETURNS trigger AS $$
BEGIN
    NEW.event_date := DATE(TO_TIMESTAMP(NEW.event_timestamp / 1000000000.0));
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;
```

**What We Expected**:
1. INSERT statement runs without `event_date`
2. BEFORE INSERT trigger populates `event_date`
3. Partition routing uses `event_date` to route row

**What Actually Happened**:
1. INSERT statement runs without `event_date`
2. PostgreSQL checks NOT NULL constraint on `event_date` → **NULL value detected** ❌
3. **ERROR: NOT NULL constraint violation** (before trigger ever runs)

---

## Solutions

### Solution 1: Explicit Column in INSERT (Recommended)

Compute the partition key value BEFORE the INSERT and include it in the VALUES clause:

```sql
CREATE OR REPLACE FUNCTION public.generate_order_event()
RETURNS trigger AS $$
DECLARE
    event_ts ns_timestamp;
    computed_event_date DATE;  -- NEW: Pre-compute the partition key
BEGIN
    event_ts := EXTRACT(EPOCH FROM NOW()) * 1000000000;
    computed_event_date := DATE(TO_TIMESTAMP(event_ts / 1000000000.0));  -- NEW
    
    -- Include event_date in the INSERT
    INSERT INTO trading_events (
        correlation_id, 
        event_timestamp, 
        received_timestamp, 
        processing_timestamp,
        event_type, 
        event_source, 
        symbol, 
        event_data,
        event_date  -- ← EXPLICIT COLUMN
    ) VALUES (
        COALESCE(NEW.id, OLD.id),
        event_ts,
        event_ts,
        event_ts,
        event_type_val,
        'order_management',
        COALESCE(NEW.symbol, OLD.symbol),
        jsonb_build_object(...),
        computed_event_date  -- ← EXPLICIT VALUE
    );
    
    RETURN COALESCE(NEW, OLD);
END;
$$ LANGUAGE plpgsql;
```

**Why This Works**:
- Partition key value is explicitly provided in INSERT
- NOT NULL constraint is satisfied immediately
- No dependency on trigger execution order

### Solution 2: DEFAULT Expression

Use a DEFAULT expression instead of a trigger:

```sql
-- Table definition with DEFAULT
CREATE TABLE trading_events (
    event_date DATE NOT NULL DEFAULT DATE(TO_TIMESTAMP(
        (EXTRACT(EPOCH FROM NOW()) * 1000000000) / 1000000000.0
    ))
) PARTITION BY RANGE (event_date);

-- Now INSERT without event_date works
INSERT INTO trading_events (
    correlation_id, 
    event_timestamp, 
    ...
) VALUES (
    gen_random_uuid(),
    EXTRACT(EPOCH FROM NOW()) * 1000000000,
    ...
);  -- event_date automatically populated by DEFAULT
```

**Why This Works**:
- DEFAULT expression evaluated BEFORE constraint checking
- No trigger dependency
- Simpler than trigger approach

**Caveat**: DEFAULT expression must be deterministic or use `CURRENT_DATE` (not `NOW()` with computation)

### Solution 3: Generated Column (PostgreSQL 12+)

Use a GENERATED ALWAYS column:

```sql
CREATE TABLE trading_events (
    event_timestamp ns_timestamp NOT NULL,
    event_date DATE GENERATED ALWAYS AS (
        DATE(TO_TIMESTAMP(event_timestamp / 1000000000.0))
    ) STORED
) PARTITION BY RANGE (event_date);
```

**Why This Works**:
- Generated column computed automatically from `event_timestamp`
- No trigger or DEFAULT needed
- Partition key always consistent with timestamp

**Caveat**: Cannot manually override generated value

---

## Partition Management

### Manual Partition Creation

We created 31 daily partitions for `change_tracking` table:

```sql
DO $$
DECLARE
    partition_date DATE;
    partition_name TEXT;
BEGIN
    FOR i IN 0..30 LOOP
        partition_date := CURRENT_DATE + (i || ' days')::INTERVAL;
        partition_name := 'change_tracking_' || to_char(partition_date, 'YYYY_MM_DD');
        
        EXECUTE format(
            'CREATE TABLE IF NOT EXISTS %I PARTITION OF change_tracking
             FOR VALUES FROM (%L) TO (%L)',
            partition_name,
            partition_date,
            partition_date + INTERVAL '1 day'
        );
    END LOOP;
END $$;
```

**Result**: Partitions from `change_tracking_2025_10_09` to `change_tracking_2025_11_08`

### Automated Partition Management (Recommended for Production)

**Option 1: pg_partman Extension**
```sql
-- Install extension
CREATE EXTENSION pg_partman;

-- Configure automatic partition creation
SELECT partman.create_parent(
    p_parent_table := 'public.trading_events',
    p_control := 'event_date',
    p_type := 'native',
    p_interval := 'daily',
    p_premake := 7  -- Create 7 days ahead
);

-- Schedule maintenance (run daily)
SELECT partman.run_maintenance_proc();
```

**Option 2: Custom Maintenance Job**
```sql
CREATE OR REPLACE FUNCTION maintain_partitions()
RETURNS void AS $$
DECLARE
    table_name TEXT;
    partition_column TEXT;
BEGIN
    -- Loop through partitioned tables
    FOR table_name, partition_column IN
        SELECT tablename, 'event_date' FROM pg_tables WHERE tablename IN ('trading_events', 'change_tracking')
    LOOP
        -- Create partitions 30 days ahead
        PERFORM create_future_partitions(table_name, partition_column, 30);
        
        -- Drop partitions older than 90 days
        PERFORM drop_old_partitions(table_name, partition_column, 90);
    END LOOP;
END;
$$ LANGUAGE plpgsql;

-- Schedule via cron or pg_cron
SELECT cron.schedule('maintain-partitions', '0 2 * * *', 'SELECT maintain_partitions()');
```

---

## Verification Queries

### Check Partition Key Population

```sql
-- Verify all rows have partition key populated
SELECT 
  COUNT(*) as total_rows,
  COUNT(*) FILTER (WHERE event_date IS NOT NULL) as rows_with_date,
  COUNT(*) FILTER (WHERE event_date IS NULL) as rows_without_date,
  MIN(event_date) as earliest_date,
  MAX(event_date) as latest_date
FROM trading_events;
```

**Expected Result**:
```
total_rows | rows_with_date | rows_without_date | earliest_date | latest_date
-----------+----------------+-------------------+---------------+-------------
        35 |             35 |                 0 | 2025-10-09    | 2025-10-09
```

### Check Partition Distribution

```sql
-- See how rows are distributed across partitions
SELECT 
  schemaname,
  tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size,
  (SELECT count(*) FROM pg_catalog.pg_class c WHERE c.relname = tablename) as row_count
FROM pg_tables
WHERE tablename LIKE 'trading_events_%'
ORDER BY tablename;
```

### List All Partitions

```sql
-- List all partitions for a table
SELECT 
  inhrelid::regclass AS partition_name,
  pg_get_expr(c.relpartbound, c.oid) AS partition_range
FROM pg_inherits i
JOIN pg_class c ON c.oid = i.inhrelid
WHERE i.inhparent = 'trading_events'::regclass
ORDER BY partition_name;
```

---

## Testing Partition Routing

### Test Insert Without Partition Key

```sql
-- This will FAIL with partition routing error
INSERT INTO trading_events (
    correlation_id,
    event_timestamp,
    event_type,
    event_source,
    symbol
) VALUES (
    gen_random_uuid(),
    EXTRACT(EPOCH FROM NOW()) * 1000000000,
    'order_submitted',
    'test',
    'BTC/USD'
    -- Missing event_date!
);
```

**Error**:
```
ERROR: no partition of relation "trading_events" found for row
DETAIL: Partition key of the failing row contains (event_date) = (null)
```

### Test Insert With Partition Key

```sql
-- This will SUCCEED
INSERT INTO trading_events (
    correlation_id,
    event_timestamp,
    event_type,
    event_source,
    symbol,
    event_date  -- EXPLICIT partition key
) VALUES (
    gen_random_uuid(),
    EXTRACT(EPOCH FROM NOW()) * 1000000000,
    'order_submitted',
    'test',
    'BTC/USD',
    CURRENT_DATE  -- EXPLICIT value
);
```

**Success**: Row routed to `trading_events_2025_10_09` partition

---

## Lessons Learned

### 1. **Never Rely on Triggers for NOT NULL Partition Keys**

❌ **Bad**:
```sql
CREATE TABLE events (
    event_date DATE NOT NULL  -- NOT NULL + partition key
) PARTITION BY RANGE (event_date);

CREATE TRIGGER set_date BEFORE INSERT ...  -- Will fail!
```

✅ **Good**:
```sql
-- Option A: Explicit in INSERT
INSERT INTO events (..., event_date) VALUES (..., CURRENT_DATE);

-- Option B: DEFAULT expression
CREATE TABLE events (
    event_date DATE DEFAULT CURRENT_DATE
) PARTITION BY RANGE (event_date);

-- Option C: Generated column
CREATE TABLE events (
    ts TIMESTAMP,
    event_date DATE GENERATED ALWAYS AS (ts::DATE) STORED
) PARTITION BY RANGE (event_date);
```

### 2. **PostgreSQL Constraint Order Matters**

Execution order:
1. NOT NULL → checked FIRST (before triggers)
2. CHECK constraints
3. BEFORE INSERT triggers
4. Foreign key constraints
5. AFTER INSERT triggers

**Implication**: Plan your constraint strategy around this order

### 3. **Partition Key Must Be Explicitly Provided or Computed**

You cannot "inject" partition key values after INSERT starts. The value must be:
- In the INSERT VALUES clause, OR
- Computed by a DEFAULT expression, OR
- Computed by a GENERATED column

### 4. **Test Partition Routing in E2E Tests**

Unit tests may not catch partition routing failures. Always test:
- INSERT without partition key (should fail gracefully)
- INSERT with partition key (should succeed)
- Verify row lands in correct partition
- Check partition key NULL count = 0

---

## Migration Strategy for Existing Systems

If you have existing partitioned tables with trigger-based partition key population:

### Step 1: Identify Affected Tables

```sql
SELECT 
  c.relname as table_name,
  pg_get_partkeydef(c.oid) as partition_key
FROM pg_class c
WHERE c.relkind = 'p'  -- Partitioned tables
AND EXISTS (
  SELECT 1 FROM pg_attribute a
  WHERE a.attrelid = c.oid
  AND a.attnotnull = true
  AND a.attname IN (
    SELECT unnest(string_to_array(
      regexp_replace(pg_get_partkeydef(c.oid), '[^a-z_]', '', 'gi'),
      ','
    ))
  )
);
```

### Step 2: Audit Triggers

```sql
SELECT 
  tgname as trigger_name,
  tgrelid::regclass as table_name,
  pg_get_triggerdef(oid) as trigger_definition
FROM pg_trigger
WHERE tgrelid IN (
  SELECT oid FROM pg_class WHERE relkind = 'p'
)
AND tgname LIKE '%date%';
```

### Step 3: Update Insert Logic

For each affected trigger:
1. Extract partition key computation logic
2. Move computation BEFORE INSERT
3. Include partition key in INSERT statement
4. Test with E2E integration tests

### Step 4: Remove Unnecessary Triggers

```sql
-- After verifying INSERT includes partition key
DROP TRIGGER IF EXISTS tg_set_trading_event_date ON trading_events;
DROP FUNCTION IF EXISTS set_trading_event_date();
```

---

## References

- PostgreSQL Documentation: [Table Partitioning](https://www.postgresql.org/docs/current/ddl-partitioning.html)
- PostgreSQL Documentation: [Constraint Evaluation](https://www.postgresql.org/docs/current/ddl-constraints.html)
- PostgreSQL Documentation: [Generated Columns](https://www.postgresql.org/docs/current/ddl-generated-columns.html)
- pg_partman: [Partition Management Extension](https://github.com/pgpartman/pg_partman)

---

**Document Created**: 2025-10-09  
**Wave**: 128 Agent 19  
**Status**: Production-validated insights from E2E testing