metabuilder/deployment/docker/postgres/README.md

Email Service PostgreSQL Container

PostgreSQL 16 Docker container configured for MetaBuilder email client with IMAP/SMTP support. Provides multi-tenant database with row-level security, comprehensive indexing, and production-ready configuration.

Features

Database Design

• 4 Core Tables: email_accounts, email_folders, email_messages, email_attachments
• Multi-Tenant: Tenant isolation via Row Level Security (RLS) policies
• RFC 5322 Compliant: Full email header support with threading
• Soft Deletes: Audit trail via is_deleted flags and deleted_at timestamps
• IMAP Sync: Message-ID and IMAP UID tracking for incremental sync

Security

• Row Level Security: All tables use RLS for tenant isolation
• pgcrypto Extension: UUID generation and password hashing
• Application User: Separate email_service user with limited privileges
• Encryption Support: TLS/STARTTLS configuration for remote connections

Performance

• 30+ Indexes: Optimized for common queries (sync, search, threading)
• Connection Pooling: Supports 200 concurrent connections
• Full-Text Search: GIN index for subject and body search
• Query Optimization: Partial indexes for active/unread messages

High Availability

• Health Checks: Automatic liveness probe on port 5432
• Data Persistence: Docker volumes for recovery across container restarts
• Incremental Backups: WAL configuration for point-in-time recovery
• Replication Ready: Configuration for primary/standby setup

Quick Start

Development

# Start PostgreSQL container
docker-compose -f deployment/docker/docker-compose.email-service.yml up email-postgres

# Connect with psql
psql postgresql://email_service:postgres@localhost:5434/email_service

# View database structure
\dt email_*
\di idx_email_*

# Check connection pool status
SELECT * FROM pg_stat_activity;

Docker Compose

version: '3.8'

services:
  email-postgres:
    build:
      context: deployment/docker/postgres
      dockerfile: Dockerfile
    container_name: email-postgres
    environment:
      POSTGRES_DB: email_service
      POSTGRES_USER: email_service
      POSTGRES_PASSWORD: secure_password
    volumes:
      - email_postgres_data:/var/lib/postgresql/data
    ports:
      - "5434:5432"
    networks:
      - email-service-network
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U email_service -d email_service"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  email_postgres_data:
    driver: local

networks:
  email-service-network:
    driver: bridge

Database Schema

email_accounts

Email account configuration (IMAP/POP3/SMTP).

CREATE TABLE email_accounts (
  id UUID PRIMARY KEY,
  tenant_id UUID NOT NULL,
  user_id UUID NOT NULL,
  account_name VARCHAR(255),
  email_address VARCHAR(255) UNIQUE,
  protocol VARCHAR(20) -- 'imap', 'pop3'
  hostname VARCHAR(255),
  port INTEGER,
  encryption VARCHAR(20) -- 'none', 'tls', 'starttls'
  username VARCHAR(255),
  credential_id UUID,
  is_sync_enabled BOOLEAN DEFAULT TRUE,
  sync_interval INTEGER DEFAULT 300,
  last_sync_at BIGINT,
  is_syncing BOOLEAN DEFAULT FALSE,
  is_enabled BOOLEAN DEFAULT TRUE,
  created_at BIGINT,
  updated_at BIGINT
);

Indexes:
- idx_email_accounts_tenant_user (tenant_id, user_id)
- idx_email_accounts_email_address (email_address)
- idx_email_accounts_sync_enabled (tenant_id, is_sync_enabled, last_sync_at)

email_folders

Email folder hierarchy (Inbox, Sent, Drafts, custom).

CREATE TABLE email_folders (
  id UUID PRIMARY KEY,
  tenant_id UUID NOT NULL,
  account_id UUID NOT NULL REFERENCES email_accounts,
  name VARCHAR(255),
  type VARCHAR(20) -- 'inbox', 'sent', 'drafts', 'trash', 'spam', 'archive', 'custom'
  unread_count INTEGER DEFAULT 0,
  total_count INTEGER DEFAULT 0,
  sync_token TEXT,
  is_selectable BOOLEAN DEFAULT TRUE,
  created_at BIGINT,
  updated_at BIGINT
);

Indexes:
- idx_email_folders_account_type (account_id, type)
- idx_email_folders_tenant_account (tenant_id, account_id)
- idx_email_folders_type (type) WHERE type IN ('inbox', 'sent', ...)

email_messages

Email messages with RFC 5322 headers and threading support.

CREATE TABLE email_messages (
  id UUID PRIMARY KEY,
  tenant_id UUID NOT NULL,
  account_id UUID NOT NULL REFERENCES email_accounts,
  folder_id UUID NOT NULL REFERENCES email_folders,
  message_id VARCHAR(500) NOT NULL,
  imap_uid VARCHAR(100),
  in_reply_to VARCHAR(500),
  from VARCHAR(500),
  to JSONB,              -- ['user@example.com', ...]
  cc JSONB,
  bcc JSONB,
  reply_to VARCHAR(255),
  subject VARCHAR(500),
  text_body TEXT,
  html_body TEXT,
  is_read BOOLEAN DEFAULT FALSE,
  is_flagged BOOLEAN DEFAULT FALSE,
  is_draft BOOLEAN DEFAULT FALSE,
  thread_id UUID,
  sent_at BIGINT,
  received_at BIGINT,
  created_at BIGINT,
  updated_at BIGINT
);

Indexes:
- idx_email_messages_folder_received (folder_id, received_at DESC)
- idx_email_messages_thread_id (thread_id)
- idx_email_messages_account_read (account_id, is_read)
- idx_email_messages_message_id (message_id, account_id) UNIQUE
- idx_email_messages_imap_uid (imap_uid, folder_id) UNIQUE
- idx_email_messages_text_search (full-text search on subject + body)

email_attachments

File attachments with S3/blob storage references.

CREATE TABLE email_attachments (
  id UUID PRIMARY KEY,
  tenant_id UUID NOT NULL,
  message_id UUID NOT NULL REFERENCES email_messages,
  filename VARCHAR(500),
  mime_type VARCHAR(100),
  size_bytes INTEGER,
  storage_path VARCHAR(1000),
  storage_type VARCHAR(20) -- 'blob', 's3', 'local'
  blob_id UUID,
  content_hash VARCHAR(64),
  created_at BIGINT
);

Indexes:
- idx_email_attachments_message_id (message_id)
- idx_email_attachments_tenant (tenant_id)
- idx_email_attachments_storage_path (storage_path)
- idx_email_attachments_content_hash (content_hash)

Configuration

Connection Pooling (Port 5432)

max_connections = 200

For external pooling with PgBouncer:
  max_client_conn = 1000     (clients)
  max_db_connections = 25    (per database)
  default_pool_size = 25     (per user)
  reserve_pool_size = 5      (reserve)

Memory Settings

shared_buffers = 256MB          # PostgreSQL page cache
effective_cache_size = 1GB      # Query planner hint
work_mem = 1310kB               # Per-operation sorting/hashing
maintenance_work_mem = 64MB     # CREATE INDEX, VACUUM, ALTER

WAL (Write-Ahead Logging)

wal_buffers = 16MB                       # WAL page cache
min_wal_size = 1GB                       # Minimum WAL files
max_wal_size = 4GB                       # Maximum WAL files before checkpoint
checkpoint_completion_target = 0.9       # Spread I/O over time

Query Planning

random_page_cost = 1.1                   # SSD-optimized
effective_io_concurrency = 200           # NVMe-optimized
default_statistics_target = 100          # Column statistics

Initialization Scripts

Executed automatically on container startup in order:

1. init-email-service.sql

• Creates email_service database
• Creates 4 core tables with constraints
• Enables extensions (uuid-ossp, pgcrypto, pg_trgm)
• Sets up RLS policies for multi-tenancy
• Creates audit table (email_audit_log)

2. init-indexes.sql

• Creates 30+ indexes for query optimization
• Partial indexes for active/unread messages
• BRIN indexes for timestamp columns
• Analyzes statistics for query planner

3. init-connection-pooling.sql

• Configures connection pooling parameters
• Creates monitoring views (pg_connection_stats, pg_long_queries)
• Provides helper functions (pg_table_bloat_status)
• Documents configuration guidelines

Multi-Tenant Isolation

Row Level Security (RLS) policies ensure data isolation:

-- RLS Policy: email_messages - tenant isolation
CREATE POLICY email_messages_tenant_policy ON email_messages
  USING (tenant_id = current_setting('app.current_tenant_id')::UUID)
  WITH CHECK (tenant_id = current_setting('app.current_tenant_id')::UUID);

Application must set tenant context before queries:

// Set tenant for this connection
await db.raw(`SET LOCAL app.current_tenant_id = '${tenantId}'`)

// All subsequent queries automatically filtered by tenant_id
const messages = await db.email_messages.select() // Only this tenant's messages

Monitoring

Connection Usage

-- Current connections by user/database
SELECT * FROM pg_connection_stats;

-- Long-running queries
SELECT * FROM pg_long_queries;

Table Statistics

-- Table bloat
SELECT * FROM pg_table_bloat_status();

-- Index usage
SELECT * FROM pg_stat_user_indexes ORDER BY idx_scan DESC;

-- Unused indexes
SELECT * FROM pg_stat_user_indexes WHERE idx_scan = 0;

Query Performance

-- Cache hit ratio (should be > 95%)
SELECT
  sum(heap_blks_read) as heap_read,
  sum(heap_blks_hit) as heap_hit,
  sum(heap_blks_hit) / (sum(heap_blks_hit) + sum(heap_blks_read)) as ratio
FROM pg_statio_user_tables;

-- Slow queries (> 1 second)
SELECT query, calls, mean_exec_time FROM pg_stat_statements
WHERE mean_exec_time > 1000
ORDER BY mean_exec_time DESC;

Troubleshooting

PostgreSQL won't start

# Check logs
docker logs metabuilder-email-postgres

# Verify data directory permissions
docker exec metabuilder-email-postgres ls -la /var/lib/postgresql

# Reset volume
docker volume rm email_postgres_data
docker-compose up email-postgres

Health check failing

# Check if PostgreSQL is responsive
docker exec metabuilder-email-postgres pg_isready -U email_service

# View last 50 lines of logs
docker logs --tail 50 metabuilder-email-postgres

# Connect directly
docker exec -it metabuilder-email-postgres psql -U email_service -d email_service

Connection pool exhausted

-- Check active connections
SELECT datname, usename, COUNT(*) FROM pg_stat_activity GROUP BY datname, usename;

-- Kill idle connections (over 30 minutes)
SELECT pg_terminate_backend(pid) FROM pg_stat_activity
WHERE state = 'idle' AND query_start < NOW() - INTERVAL '30 minutes';

-- Increase max_connections (requires restart)
ALTER SYSTEM SET max_connections = 300;
SELECT pg_reload_conf();

Slow queries

-- Enable slow query logging
ALTER SYSTEM SET log_min_duration_statement = 1000;  -- 1 second
SELECT pg_reload_conf();

-- View slow query log
SELECT query_start, query, query_time FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 10;

Performance Tuning

For production (8GB+ RAM)

ALTER SYSTEM SET shared_buffers = '2GB';
ALTER SYSTEM SET effective_cache_size = '6GB';
ALTER SYSTEM SET work_mem = '10MB';
ALTER SYSTEM SET maintenance_work_mem = '512MB';
SELECT pg_reload_conf();

For high-throughput email (1000+ messages/sec)

ALTER SYSTEM SET wal_level = 'minimal';  # Disable replication logging
ALTER SYSTEM SET synchronous_commit = 'off';  # Async WAL writes
ALTER SYSTEM SET max_wal_size = '16GB';
SELECT pg_reload_conf();

For read-heavy workloads

ALTER SYSTEM SET random_page_cost = 1.01;  # Favor index scans
ALTER SYSTEM SET effective_io_concurrency = 500;
ALTER SYSTEM SET effective_cache_size = '8GB';
SELECT pg_reload_conf();

Backup & Recovery

Manual backup

# Backup entire database
docker exec metabuilder-email-postgres pg_dump -U email_service email_service > backup.sql

# Backup with compression
docker exec metabuilder-email-postgres pg_dump -U email_service -Fc email_service > backup.dump

# Restore from backup
docker exec -i metabuilder-email-postgres psql -U email_service email_service < backup.sql

Point-in-time recovery

-- Enable WAL archiving
ALTER SYSTEM SET archive_mode = 'on';
ALTER SYSTEM SET archive_command = 'cp %p /mnt/archive/%f';
SELECT pg_reload_conf();

-- Recover to specific timestamp
SELECT pg_create_restore_point('pre_migration');

-- Restore using backup + WAL files

Docker Compose Integration

Development Environment

# Start all services (PostgreSQL + email service + Redis)
docker-compose -f deployment/docker/docker-compose.email-service.yml up

# View logs
docker-compose -f deployment/docker/docker-compose.email-service.yml logs -f email-postgres

# Stop services
docker-compose -f deployment/docker/docker-compose.email-service.yml down

Production Environment

# Start with production overrides
docker-compose \
  -f deployment/docker/docker-compose.email-service.yml \
  -f deployment/docker/docker-compose.email-service.prod.yml \
  up -d

# View status
docker-compose -f deployment/docker/docker-compose.email-service.yml ps

Environment Variables

Create .env file in deployment/docker/:

# Database
POSTGRES_PASSWORD=secure_password_here
POSTGRES_PORT=5434

# Email service (future)
FLASK_ENV=production
LOG_LEVEL=INFO
IMAP_POLL_INTERVAL=300
SMTP_TIMEOUT=30

References

• PostgreSQL Documentation
• PostgreSQL Performance Tuning
• Row Level Security
• Email Schemas