git/metabuilder

Fork 0

mirror of https://github.com/johndoe6345789/metabuilder.git synced 2026-05-06 03:29:35 +00:00

Files

T

git 46f8daebb9 stuff

2026-01-24 00:25:09 +00:00

11 KiB

Raw Blame History

Celery Worker Container - Complete Implementation

Phase 8: Email Service Background Task Processing

Location: /deployment/docker/celery-worker/

Files Created

1. Dockerfile (2.8 KB)

Multi-stage Docker image for Celery worker container.

Features:

Python 3.11-slim base image (minimal footprint)
Build stage: Compiles dependencies with gcc/g++
Runtime stage: Only runtime dependencies (ca-certificates, libpq5, curl)
Non-root user (celeryworker UID 1000) for security
Health check via celery inspect ping (30s interval, 15s startup)
Environment variables for Redis broker and result backend
Logs volume: /app/logs/ for persistent logging

CMD:

celery -A tasks.celery_app worker \
  --loglevel=info \
  --concurrency=4 \
  --time-limit=300 \
  --soft-time-limit=280 \
  --pool=prefork \
  --queues=sync,send,delete,spam,periodic \
  --hostname=celery-worker@%h \
  --logfile=/app/logs/celery-worker.log

2. docker-compose.yml (6.7 KB)

Docker Compose service definitions for Celery ecosystem.

Services:

celery-worker - Main task processor
- Image: Custom (from Dockerfile)
- Container: metabuilder-celery-worker
- Concurrency: 4 worker processes
- Timeout: 300s hard, 280s soft
- Health check: celery inspect ping (30s)
- Resource limits: 2 CPU / 512 MB memory
- Logs: JSON-file driver (10MB / 3 files)
- Volumes: celery_worker_logs:/app/logs
celery-beat - Task scheduler
- Image: Custom (same Dockerfile)
- Container: metabuilder-celery-beat
- Scheduler: PersistentScheduler (database-backed)
- Schedule DB: /app/logs/celery-beat-schedule.db
- Tasks:
  - sync-emails-every-5min - Periodic email sync
  - cleanup-stale-tasks-hourly - Redis cleanup
- Health check: ps aux (process monitor)
- Depends on: redis, postgres, celery-worker
celery-flower - Web monitoring dashboard
- Image: mher/flower:2.0.1 (official)
- Container: metabuilder-celery-flower
- Port: 5556:5555
- URL: http://localhost:5556
- Database: /data/flower.db (persistent SQLite)
- Features: Task history, worker stats, real-time graphs
- Health check: curl http://localhost:5555/health
- Depends on: redis, celery-worker

Volumes:

celery_worker_logs - tmpfs (100MB, in-memory) for logs
celery_flower_data - local (persistent) for Flower database

Network: metabuilder-dev-network (external, from main compose)

3. README.md (13 KB)

Complete user guide for Celery worker operations.

Sections:

Quick start guide (3 commands to get running)
Architecture overview with diagrams
Multi-tenant safety validation
Queue types and priorities (sync, send, delete, spam, periodic)
Task timeout configuration (hard/soft limits)
Deployment instructions (Docker Compose)
Configuration via environment variables
Resource limits and tuning
Monitoring with Flower dashboard
Health checks (worker, beat, flower)
Logs inspection and troubleshooting
Task management (list, revoke, cancel, trigger)
Queue management (depth, status, flush)
Production checklist

4. SETUP.md (14 KB)

Step-by-step setup and operational guide.

Sections:

Quick start (5-step guide)
Environment setup (create .env, configure)
Docker setup (build images, network, volumes)
Database setup (PostgreSQL verification)
Redis setup (broker/result backend)
Configuration guide (concurrency, timeouts, memory, logging)
Operational tasks (start, stop, restart, monitor)
Monitoring commands (Flower, CLI tools, logs)
Task management (list, check status, cancel, trigger)
Troubleshooting (not starting, crashes, not processing, memory, timeouts, connectivity)
Performance tuning (queue optimization, connection pooling)
Production deployment (checklist, environment, build, Kubernetes)

5. ARCHITECTURE.md (19 KB)

Technical architecture documentation.

Sections:

System overview with component diagrams
Service architecture (workers, beat, flower, infrastructure)
Component details (concurrency, queues, health checks)
Data flow (email sync, email send)
Multi-tenant safety (validation pattern, data isolation)
Retry and error handling (exponential backoff)
Health checks (worker, beat, flower, startup)
Resource management (memory, CPU, connection pooling)
Volume and persistence (tmpfs vs local)
Security (task validation, credential handling, network)
Monitoring and observability (metrics, logging, Flower UI)
References (Celery docs, Flower docs, Redis docs)

6. .env.example (6.8 KB)

Environment variable configuration template.

Sections:

Redis configuration (broker, result backend, SSL/TLS)
Database configuration (PostgreSQL)
Worker configuration (concurrency, timeouts, backoff)
Celery beat scheduler configuration
Logging configuration
Email service configuration
Task-specific settings (sync batch size, SMTP timeout)
Security configuration (encryption keys, SSL verification)
Monitoring and observability (Flower, Prometheus)
Resource limits (Docker)
Deployment mode (environment variable)

61 configurable settings with descriptions and sensible defaults.

7. manage.sh (15 KB)

Management script for lifecycle and monitoring commands.

Command Groups:

Lifecycle:

up - Start all services
down - Stop all services
restart - Restart all services
rebuild - Rebuild Docker images

Monitoring:

logs [service] - Show logs (worker, beat, flower, redis, all)
stats - Worker statistics
health - Health check all services
ps - Show running containers

Task Management:

tasks [state] - List tasks (active, reserved, registered, failed)
task:status <id> - Check task status
task:revoke <id> - Cancel running task
task:purge - Clear all pending tasks

Queue Management:

queue:status - Show queue statistics
queue:list - List all queues

Worker Operations:

worker:ping - Check if worker responsive
worker:info - Show worker information

Flower:

flower:open - Open Flower in browser

Development:

dev:logs - Follow all logs
dev:shell - Open Python shell
dev:test - Run test tasks

Maintenance:

clean:logs - Clear log files
clean:redis - Flush Redis
clean:all - Clean all data

Help:

help - Show usage guide
version - Show version

Quick Start

1. Build and Start

cd /path/to/metabuilder

# Start with main compose + Celery
docker-compose -f deployment/docker/docker-compose.development.yml \
               -f deployment/docker/celery-worker/docker-compose.yml \
               up -d

2. Verify

# Check containers
docker ps | grep metabuilder

# Check health
docker-compose exec celery-worker celery -A tasks.celery_app inspect ping

3. Monitor

# Open Flower dashboard
http://localhost:5556

# Check logs
docker-compose logs -f celery-worker

Configuration Summary

Default Settings

concurrency: 4                          # 4 concurrent worker processes
task_timeout: 300 seconds               # 5 minutes hard limit
task_soft_timeout: 280 seconds          # 4m 40s soft limit
redis_broker: redis://redis:6379/0     # Task queue
redis_result: redis://redis:6379/1     # Task results
database: postgresql://metabuilder:password@postgres:5432/metabuilder_dev
pool: prefork                          # Process-based (not threaded)
queues: sync,send,delete,spam,periodic # 5 task queue types

Memory Overhead

Per worker process: 100-150 MB
4 workers base: 400-600 MB
Docker limit: 512 MB
Recommended: Reserve 256 MB

Total stack (postgres, redis, workers, beat, flower):
~1.5-2 GB with all services running

Files Modified

None. All files are new in /deployment/docker/celery-worker/ directory.

Dependencies

External Services

PostgreSQL 16: Database (metadata, credentials, email messages)
Redis 7: Broker (task queue DB 0) + Result backend (DB 1)
Docker: Container runtime
docker-compose: Service orchestration

Python Packages (from services/email_service/requirements.txt)

celery[redis]==5.3.4 - Task queue framework
redis==5.0.1 - Redis client
kombu==5.3.4 - Message broking library
sqlalchemy==2.0.23 - Database ORM
psycopg2-binary==2.9.9 - PostgreSQL driver
celery-beat==2.5.0 - Task scheduler
flower==2.0.1 - Monitoring dashboard (in separate image)

Base Images

python:3.11-slim - Worker and beat containers (385 MB)
mher/flower:2.0.1 - Flower monitoring (Official, lightweight)

Security Considerations

Multi-Tenant

Every task validates tenant_id and user_id parameters
Cannot operate across tenant boundaries
ACL checks before task execution

Credentials

Email passwords encrypted in Credential entity
Decrypted at runtime, never logged
Encryption: SHA-512 + salt

Network

All services on internal Docker network
Only Flower (port 5556) exposed for monitoring
Worker and Beat isolated from external access

User

Non-root user celeryworker (UID 1000) in container
Cannot write to system directories
Limited filesystem access

Performance Characteristics

Throughput

4 concurrent workers × multiple tasks = High throughput
Prefork pool: No GIL (Global Interpreter Lock) blocking
Expected: 100-1000 tasks/hour depending on task complexity

Latency

Queue to execution: <100ms (typical)
Email sync: 5-30 seconds per mailbox
Email send: 5-10 seconds per message

Scalability

Horizontal: Run multiple worker containers
Vertical: Increase concurrency per container
Load balancing: Auto via Redis broker

Monitoring Integration

Built-in

Flower web UI: http://localhost:5556
Health checks: 30-second intervals
Logging: Structured JSON format
Task tracking: Result storage (1 hour TTL)

Optional

Prometheus metrics: Can be added via prometheus-client
Log aggregation: Use docker json-file driver with log aggregator
APM: Integrate with DataDog, New Relic, etc.

Next Steps

Start Services: docker-compose up -d
Access Dashboard: http://localhost:5556
Monitor Tasks: docker-compose logs -f celery-worker
Review Code: services/email_service/tasks/celery_app.py
Test Manually: docker exec ... celery tasks
Production: See SETUP.md Production Deployment section

Support

Documentation: README.md, SETUP.md, ARCHITECTURE.md
Logs: /app/logs/celery-worker.log, /app/logs/celery-beat.log
Monitoring: Flower UI at http://localhost:5556
Email Service Code: services/email_service/
Email Client Plan: docs/plans/2026-01-23-email-client-implementation.md

Version Information

Phase: Phase 8 (Email Service Background Tasks)
Created: January 24, 2026
Python: 3.11
Celery: 5.3.4
Docker: Compose format 3.8
Status: Production-ready

11 KiB Raw Blame History Unescape Escape

Celery Worker Container - Complete Implementation

Files Created

1. Dockerfile (2.8 KB)

2. docker-compose.yml (6.7 KB)

3. README.md (13 KB)

4. SETUP.md (14 KB)

5. ARCHITECTURE.md (19 KB)

6. .env.example (6.8 KB)

7. manage.sh (15 KB)

Quick Start

1. Build and Start

2. Verify

3. Monitor

Configuration Summary

Default Settings

Memory Overhead

Files Modified

Dependencies

External Services

Python Packages (from services/email_service/requirements.txt)

Base Images

Security Considerations

Multi-Tenant

Credentials

Network

User

Performance Characteristics

Throughput

Latency

Scalability

Monitoring Integration

Built-in

Optional

Next Steps

Support

Version Information

11 KiB

Raw Blame History