foxhunt/POSTGRESQL_PARTITION_ROUTING_INSIGHTS.md

PostgreSQL Partition Routing: Critical Insights from Wave 128

Date: 2025-10-09
Context: Wave 128 Agent 19 - E2E validation revealed partition routing failures

---

Problem Summary

Database triggers were inserting rows into partitioned tables WITHOUT the partition key column, causing partition routing failures.

Error Message:

ERROR: no partition of relation "trading_events" found for row
DETAIL: Partition key of the failing row contains (event_date) = (null)

---

Root Cause: PostgreSQL Constraint Ordering

Key Discovery: Constraint Evaluation Order

PostgreSQL evaluates constraints in this order:

1. NOT NULL constraints ← Evaluated FIRST
2. CHECK constraints
3. BEFORE INSERT triggers ← Evaluated AFTER NOT NULL check
4. AFTER INSERT triggers

Critical Implication: You CANNOT rely on a BEFORE INSERT trigger to populate a NOT NULL partition key column.

Why This Failed

Our initial approach:

-- Table definition
CREATE TABLE trading_events (
    event_date DATE NOT NULL  -- Partition key with NOT NULL constraint
) PARTITION BY RANGE (event_date);

-- BEFORE INSERT trigger (designed to auto-populate event_date)
CREATE TRIGGER tg_set_trading_event_date
    BEFORE INSERT ON trading_events
    FOR EACH ROW
    EXECUTE FUNCTION set_trading_event_date();

-- Function that sets event_date from event_timestamp
CREATE FUNCTION set_trading_event_date() RETURNS trigger AS $$
BEGIN
    NEW.event_date := DATE(TO_TIMESTAMP(NEW.event_timestamp / 1000000000.0));
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

What We Expected:

1. INSERT statement runs without event_date
2. BEFORE INSERT trigger populates event_date
3. Partition routing uses event_date to route row

What Actually Happened:

1. INSERT statement runs without event_date
2. PostgreSQL checks NOT NULL constraint on event_date → NULL value detected ❌
3. ERROR: NOT NULL constraint violation (before trigger ever runs)

---

Solutions

Solution 1: Explicit Column in INSERT (Recommended)

Compute the partition key value BEFORE the INSERT and include it in the VALUES clause:

CREATE OR REPLACE FUNCTION public.generate_order_event()
RETURNS trigger AS $$
DECLARE
    event_ts ns_timestamp;
    computed_event_date DATE;  -- NEW: Pre-compute the partition key
BEGIN
    event_ts := EXTRACT(EPOCH FROM NOW()) * 1000000000;
    computed_event_date := DATE(TO_TIMESTAMP(event_ts / 1000000000.0));  -- NEW
    
    -- Include event_date in the INSERT
    INSERT INTO trading_events (
        correlation_id, 
        event_timestamp, 
        received_timestamp, 
        processing_timestamp,
        event_type, 
        event_source, 
        symbol, 
        event_data,
        event_date  -- ← EXPLICIT COLUMN
    ) VALUES (
        COALESCE(NEW.id, OLD.id),
        event_ts,
        event_ts,
        event_ts,
        event_type_val,
        'order_management',
        COALESCE(NEW.symbol, OLD.symbol),
        jsonb_build_object(...),
        computed_event_date  -- ← EXPLICIT VALUE
    );
    
    RETURN COALESCE(NEW, OLD);
END;
$$ LANGUAGE plpgsql;

Why This Works:

• Partition key value is explicitly provided in INSERT
• NOT NULL constraint is satisfied immediately
• No dependency on trigger execution order

Solution 2: DEFAULT Expression

Use a DEFAULT expression instead of a trigger:

-- Table definition with DEFAULT
CREATE TABLE trading_events (
    event_date DATE NOT NULL DEFAULT DATE(TO_TIMESTAMP(
        (EXTRACT(EPOCH FROM NOW()) * 1000000000) / 1000000000.0
    ))
) PARTITION BY RANGE (event_date);

-- Now INSERT without event_date works
INSERT INTO trading_events (
    correlation_id, 
    event_timestamp, 
    ...
) VALUES (
    gen_random_uuid(),
    EXTRACT(EPOCH FROM NOW()) * 1000000000,
    ...
);  -- event_date automatically populated by DEFAULT

Why This Works:

• DEFAULT expression evaluated BEFORE constraint checking
• No trigger dependency
• Simpler than trigger approach

Caveat: DEFAULT expression must be deterministic or use CURRENT_DATE (not NOW() with computation)

Solution 3: Generated Column (PostgreSQL 12+)

Use a GENERATED ALWAYS column:

CREATE TABLE trading_events (
    event_timestamp ns_timestamp NOT NULL,
    event_date DATE GENERATED ALWAYS AS (
        DATE(TO_TIMESTAMP(event_timestamp / 1000000000.0))
    ) STORED
) PARTITION BY RANGE (event_date);

Why This Works:

• Generated column computed automatically from event_timestamp
• No trigger or DEFAULT needed
• Partition key always consistent with timestamp

Caveat: Cannot manually override generated value

---

Partition Management

Manual Partition Creation

We created 31 daily partitions for change_tracking table:

DO $$
DECLARE
    partition_date DATE;
    partition_name TEXT;
BEGIN
    FOR i IN 0..30 LOOP
        partition_date := CURRENT_DATE + (i || ' days')::INTERVAL;
        partition_name := 'change_tracking_' || to_char(partition_date, 'YYYY_MM_DD');
        
        EXECUTE format(
            'CREATE TABLE IF NOT EXISTS %I PARTITION OF change_tracking
             FOR VALUES FROM (%L) TO (%L)',
            partition_name,
            partition_date,
            partition_date + INTERVAL '1 day'
        );
    END LOOP;
END $$;

Result: Partitions from change_tracking_2025_10_09 to change_tracking_2025_11_08

Automated Partition Management (Recommended for Production)

Option 1: pg_partman Extension

-- Install extension
CREATE EXTENSION pg_partman;

-- Configure automatic partition creation
SELECT partman.create_parent(
    p_parent_table := 'public.trading_events',
    p_control := 'event_date',
    p_type := 'native',
    p_interval := 'daily',
    p_premake := 7  -- Create 7 days ahead
);

-- Schedule maintenance (run daily)
SELECT partman.run_maintenance_proc();

Option 2: Custom Maintenance Job

CREATE OR REPLACE FUNCTION maintain_partitions()
RETURNS void AS $$
DECLARE
    table_name TEXT;
    partition_column TEXT;
BEGIN
    -- Loop through partitioned tables
    FOR table_name, partition_column IN
        SELECT tablename, 'event_date' FROM pg_tables WHERE tablename IN ('trading_events', 'change_tracking')
    LOOP
        -- Create partitions 30 days ahead
        PERFORM create_future_partitions(table_name, partition_column, 30);
        
        -- Drop partitions older than 90 days
        PERFORM drop_old_partitions(table_name, partition_column, 90);
    END LOOP;
END;
$$ LANGUAGE plpgsql;

-- Schedule via cron or pg_cron
SELECT cron.schedule('maintain-partitions', '0 2 * * *', 'SELECT maintain_partitions()');

---

Verification Queries

Check Partition Key Population

-- Verify all rows have partition key populated
SELECT 
  COUNT(*) as total_rows,
  COUNT(*) FILTER (WHERE event_date IS NOT NULL) as rows_with_date,
  COUNT(*) FILTER (WHERE event_date IS NULL) as rows_without_date,
  MIN(event_date) as earliest_date,
  MAX(event_date) as latest_date
FROM trading_events;

Expected Result:

total_rows | rows_with_date | rows_without_date | earliest_date | latest_date
-----------+----------------+-------------------+---------------+-------------
        35 |             35 |                 0 | 2025-10-09    | 2025-10-09

Check Partition Distribution

-- See how rows are distributed across partitions
SELECT 
  schemaname,
  tablename,
  pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size,
  (SELECT count(*) FROM pg_catalog.pg_class c WHERE c.relname = tablename) as row_count
FROM pg_tables
WHERE tablename LIKE 'trading_events_%'
ORDER BY tablename;

List All Partitions

-- List all partitions for a table
SELECT 
  inhrelid::regclass AS partition_name,
  pg_get_expr(c.relpartbound, c.oid) AS partition_range
FROM pg_inherits i
JOIN pg_class c ON c.oid = i.inhrelid
WHERE i.inhparent = 'trading_events'::regclass
ORDER BY partition_name;

---

Testing Partition Routing

Test Insert Without Partition Key

-- This will FAIL with partition routing error
INSERT INTO trading_events (
    correlation_id,
    event_timestamp,
    event_type,
    event_source,
    symbol
) VALUES (
    gen_random_uuid(),
    EXTRACT(EPOCH FROM NOW()) * 1000000000,
    'order_submitted',
    'test',
    'BTC/USD'
    -- Missing event_date!
);

Error:

ERROR: no partition of relation "trading_events" found for row
DETAIL: Partition key of the failing row contains (event_date) = (null)

Test Insert With Partition Key

-- This will SUCCEED
INSERT INTO trading_events (
    correlation_id,
    event_timestamp,
    event_type,
    event_source,
    symbol,
    event_date  -- EXPLICIT partition key
) VALUES (
    gen_random_uuid(),
    EXTRACT(EPOCH FROM NOW()) * 1000000000,
    'order_submitted',
    'test',
    'BTC/USD',
    CURRENT_DATE  -- EXPLICIT value
);

Success: Row routed to trading_events_2025_10_09 partition

---

Lessons Learned

1. Never Rely on Triggers for NOT NULL Partition Keys

❌ Bad:

CREATE TABLE events (
    event_date DATE NOT NULL  -- NOT NULL + partition key
) PARTITION BY RANGE (event_date);

CREATE TRIGGER set_date BEFORE INSERT ...  -- Will fail!

✅ Good:

-- Option A: Explicit in INSERT
INSERT INTO events (..., event_date) VALUES (..., CURRENT_DATE);

-- Option B: DEFAULT expression
CREATE TABLE events (
    event_date DATE DEFAULT CURRENT_DATE
) PARTITION BY RANGE (event_date);

-- Option C: Generated column
CREATE TABLE events (
    ts TIMESTAMP,
    event_date DATE GENERATED ALWAYS AS (ts::DATE) STORED
) PARTITION BY RANGE (event_date);

2. PostgreSQL Constraint Order Matters

Execution order:

1. NOT NULL → checked FIRST (before triggers)
2. CHECK constraints
3. BEFORE INSERT triggers
4. Foreign key constraints
5. AFTER INSERT triggers

Implication: Plan your constraint strategy around this order

3. Partition Key Must Be Explicitly Provided or Computed

You cannot "inject" partition key values after INSERT starts. The value must be:

• In the INSERT VALUES clause, OR
• Computed by a DEFAULT expression, OR
• Computed by a GENERATED column

4. Test Partition Routing in E2E Tests

Unit tests may not catch partition routing failures. Always test:

• INSERT without partition key (should fail gracefully)
• INSERT with partition key (should succeed)
• Verify row lands in correct partition
• Check partition key NULL count = 0

---

Migration Strategy for Existing Systems

If you have existing partitioned tables with trigger-based partition key population:

Step 1: Identify Affected Tables

SELECT 
  c.relname as table_name,
  pg_get_partkeydef(c.oid) as partition_key
FROM pg_class c
WHERE c.relkind = 'p'  -- Partitioned tables
AND EXISTS (
  SELECT 1 FROM pg_attribute a
  WHERE a.attrelid = c.oid
  AND a.attnotnull = true
  AND a.attname IN (
    SELECT unnest(string_to_array(
      regexp_replace(pg_get_partkeydef(c.oid), '[^a-z_]', '', 'gi'),
      ','
    ))
  )
);

Step 2: Audit Triggers

SELECT 
  tgname as trigger_name,
  tgrelid::regclass as table_name,
  pg_get_triggerdef(oid) as trigger_definition
FROM pg_trigger
WHERE tgrelid IN (
  SELECT oid FROM pg_class WHERE relkind = 'p'
)
AND tgname LIKE '%date%';

Step 3: Update Insert Logic

For each affected trigger:

1. Extract partition key computation logic
2. Move computation BEFORE INSERT
3. Include partition key in INSERT statement
4. Test with E2E integration tests

Step 4: Remove Unnecessary Triggers

-- After verifying INSERT includes partition key
DROP TRIGGER IF EXISTS tg_set_trading_event_date ON trading_events;
DROP FUNCTION IF EXISTS set_trading_event_date();

---

References

• PostgreSQL Documentation: Table Partitioning
• PostgreSQL Documentation: Constraint Evaluation
• PostgreSQL Documentation: Generated Columns
• pg_partman: Partition Management Extension

---

Document Created: 2025-10-09
Wave: 128 Agent 19
Status: Production-validated insights from E2E testing