metabuilder/README.md

MetaBuilder

An ultra-generic, data-driven platform where everything flows through the database.

No hardcoded routes, no component imports, no assumptions. The entire application structure lives in the database, rendered by a generic JSON-to-React engine.

---

Table of Contents

1. Core Concept
2. Quick Start
3. Architecture
4. Routing System
5. Package System
6. Permission System
7. Deployment
8. Database
9. Development
10. Project Structure

---

Core Concept

The Mental Model

Browser URL → Database Query → JSON Component → Generic Renderer → React → User

Zero hardcoded connections. Everything is a database lookup:

// ❌ Traditional way (hardcoded)
import HomePage from './HomePage'
<Route path="/" component={HomePage} />

// ✅ MetaBuilder way (data-driven)
const route = await db.query('PageConfig', { path: '/' })
const component = await loadPackage(route.packageId)
return renderJSONComponent(component)

Key Principles

1. No Hardcoded Routes: Routes live in PageConfig table
2. No Component Imports: Components are JSON definitions in package files
3. No Direct Database Access: Everything goes through DBAL (Database Abstraction Layer)
4. Complete Loose Coupling: Frontend knows nothing about packages

---

Quick Start

One-Command Deployment

# Deploy everything (PostgreSQL, DBAL, Next.js, Media daemon, Redis, Nginx)
./deployment/deploy.sh all --bootstrap

This spins up:

• PostgreSQL database
• C++ DBAL daemon (database abstraction)
• Next.js frontend
• C++ media processing daemon
• Redis cache
• Nginx reverse proxy
• Monitoring stack (optional)

Then visit: http://localhost:3000

Development Setup

# Clone and install
git clone <repo>
cd metabuilder

# Option 1: Run from root (recommended for quick setup)
npm install
npm run db:generate
npm run db:push

# Option 2: Run from frontends/nextjs
cd frontends/nextjs
npm install
npm run db:generate
npm run db:push

# Bootstrap seed data
cd ../../deployment
./scripts/bootstrap-system.sh

# Start development (from root)
cd ..
npm run dev

# Or from frontends/nextjs
cd frontends/nextjs
npm run dev

Prerequisites

• Docker & Docker Compose
• Node.js 18+
• Build tools: cmake, ninja, g++/clang (for C++ daemons)

---

Architecture

System Components

┌─────────────┐
│   Browser   │
└──────┬──────┘
       │ HTTP
┌──────▼──────┐     ┌───────────┐
│   Nginx     │────▶│   Redis   │
└──────┬──────┘     └───────────┘
       │
┌──────▼──────┐     ┌───────────┐
│  Next.js    │────▶│   DBAL    │
│  Frontend   │     │  Daemon   │
└──────┬──────┘     │  (C++)    │
       │            └─────┬─────┘
       │                  │
       │            ┌─────▼─────┐
       │            │ PostgreSQL│
       │            └───────────┘
       │
┌──────▼──────┐     ┌───────────┐
│   Media     │────▶│    HLS    │
│  Daemon     │     │  Streams  │
│  (C++)      │     └───────────┘
└─────────────┘

Data Flow

1. User visits URL (/, /dashboard, /about)
2. Next.js queries PageConfig table for that path
3. Database returns packageId + component reference
4. Frontend loads JSON package from /packages/{packageId}/
5. Generic renderer converts JSON → React elements
6. User sees the rendered page

Zero hardcoded assumptions!

---

Routing System

Priority-Based Routing

Routes have two priority levels:

Priority 1: God Panel Routes (PageConfig table)

• User-configurable through admin UI
• God/supergod users can add/remove/modify routes
• Highest priority, overrides everything

Priority 2: Package Routes (InstalledPackage.config)

• Package-defined defaults
• Set during package installation
• Fallback when god panel has no override

Example: Root Route

Database seed (seed/database/installed_packages.yaml:44-54):

- packageId: ui_home
  config: |
    {
      "defaultRoute": "/",
      "publicAccess": true
    }

Frontend code (frontends/nextjs/src/app/page.tsx):

// Check god panel routes first
const godRoute = await db.query('PageConfig', { path: '/', isPublished: true })
if (godRoute) {
  return renderComponent(godRoute.packageId, godRoute.component)
}

// Fall back to package default routes
const pkg = await db.query('InstalledPackage', { 'config.defaultRoute': '/' })
return renderComponent(pkg.packageId, 'HomePage')

God Panel Override

Supergod users can remap any route:

-- Remap "/" to dashboard instead of ui_home
INSERT INTO "PageConfig" (
  id, path, packageId, component, level, requiresAuth, isPublished
) VALUES (
  'root_route', '/', 'dashboard', 'DashboardPage', 1, true, true
);

No code changes required!

Route Schema

model PageConfig {
  path          String   // URL: "/", "/dashboard", "/about"
  packageId     String?  // Which package: "ui_home", "dashboard"
  component     String?  // Which component: "HomePage", "DashboardPage"
  componentTree String   // Or full JSON component tree
  level         Int      // Permission level (0-5)
  requiresAuth  Boolean
  isPublished   Boolean
}

---

Package System

Package Structure

Packages are completely self-contained JSON + optional static files:

packages/{packageId}/
├── package.json           # Metadata
├── components/
│   └── ui.json           # Declarative JSON components
├── styles/
│   └── tokens.json       # Design tokens (colors, spacing, shadows, etc.)
└── static/
    └── assets/           # Images, fonts, etc.

JSON Component Example

From packages/ui_home/components/ui.json:

{
  "components": [
    {
      "id": "home_page",
      "name": "HomePage",
      "description": "Main home page layout",
      "render": {
        "template": {
          "type": "Box",
          "component": "main",
          "className": "home-page",
          "children": [
            { "$ref": "hero_section" },
            { "$ref": "features_section" }
          ]
        }
      }
    },
    {
      "id": "hero_section",
      "name": "HeroSection",
      "props": [
        { "name": "title", "type": "string", "default": "Build Anything, Visually" }
      ],
      "render": {
        "template": {
          "type": "Box",
          "component": "section",
          "className": "hero-section",
          "children": [
            {
              "type": "Text",
              "variant": "h1",
              "children": "{{title}}"
            }
          ]
        }
      }
    }
  ]
}

Generic Renderer

The frontend never imports package components. It uses a generic renderer:

import { renderJSONComponent } from '@/lib/packages/json/render-json-component'

// Load component from JSON
const pkg = await loadJSONPackage('/packages/ui_home')
const component = pkg.components.find(c => c.id === 'home_page')

// Render using generic renderer (no hardcoded React components!)
return renderJSONComponent(component, {}, {})

Available Packages

Package ID	Description
ui_home	Landing page with hero, features, about sections
ui_header	Application header/navbar
ui_footer	Application footer
ui_auth	Authentication UI components
ui_login	Login page
dashboard	User dashboard
user_manager	User management (admin)
package_manager	Package installation UI (god)
database_manager	Database management (supergod)
schema_editor	Schema editor (supergod)

---

Permission System

6-Level Hierarchy

Level	Role	Access	Example Route
0	Public	Unauthenticated	/ (landing page)
1	User	Personal dashboard	/dashboard
2	Moderator	Content moderation	/moderator
3	Admin	User management	/admin
4	God	Package installation, workflows	/builder
5	Supergod	Full system control	/supergod

Each level inherits all permissions from levels below.

Permission Checks

Database records control access:

# seed/database/package_permissions.yaml
- packageId: ui_home
  role: public        # Level 0 - anyone
  permission: read
  granted: true

- packageId: dashboard
  role: user          # Level 1 - authenticated
  permission: read
  granted: true

- packageId: database_manager
  role: supergod      # Level 5 - full control
  permission: admin
  granted: true

---

Deployment

One-Command Deploy

# Full stack with bootstrap
./deployment/deploy.sh all --bootstrap

# Individual stacks
./deployment/deploy.sh production    # Main services only
./deployment/deploy.sh development   # Dev environment
./deployment/deploy.sh monitoring    # Monitoring stack

What Gets Deployed

Production Stack (docker-compose.production.yml):

• PostgreSQL 16
• C++ DBAL daemon (Conan + CMake)
• Next.js app (Node 18)
• C++ Media daemon (FFmpeg, ImageMagick)
• Redis 7
• Nginx (SSL, caching)

Monitoring Stack (docker-compose.monitoring.yml):

• Prometheus (metrics)
• Grafana (dashboards)
• Loki (logs)
• Promtail (log collector)
• Exporters (PostgreSQL, Redis, Node, Nginx)

Bootstrap Process

The bootstrap-system.sh script runs in 7 phases:

1. Wait for database to be ready
2. Run Prisma migrations (schema setup)
3. Check bootstrap status (idempotent)
4. Seed database from /seed/database/*.yaml
5. Install core packages (12 packages in priority order)
6. Verify installation (health checks)
7. Run post-hooks (custom initialization)

Core Packages Bootstrap Order

Phase 1: package_manager (required first) Phase 2: ui_header, ui_footer, ui_home, ui_auth, ui_login Phase 3: dashboard Phase 4: user_manager, role_editor Phase 5: admin_dialog, database_manager, schema_editor

See seed/packages/core-packages.yaml for full configuration.

Environment Variables

# Database
POSTGRES_PASSWORD=changeme_prod_password
DATABASE_URL=postgresql://metabuilder:password@postgres:5432/metabuilder

# DBAL
DBAL_API_KEY=your_api_key_here

# Media Daemon
ICECAST_PASSWORD=hackme
REDIS_PASSWORD=changeme_redis_password

Docker Build Process

C++ Components (DBAL, Media daemon):

• Multi-stage builds (builder + runtime)
• Conan for dependency management
• CMake + Ninja build system
• Optimized production images (~200MB final)

Next.js:

• Node 18 Alpine base
• Production build with output standalone
• Minimal runtime dependencies

---

Database

Schema Overview

Core Tables:

• User - User accounts, permission levels
• Tenant - Multi-tenant isolation
• InstalledPackage - Installed packages, config
• PackagePermission - Package access control
• PageConfig - God panel route definitions
• Workflow - Workflow definitions

Example: Installed Package

model InstalledPackage {
  packageId    String   @id
  tenantId     String?
  version      String
  enabled      Boolean
  config       String   @db.Text  // JSON config
  installedAt  BigInt?
}

Seed data (seed/database/installed_packages.yaml:44-54):

- packageId: ui_home
  config: |
    {
      "systemPackage": true,
      "defaultRoute": "/",       # Maps root URL to this package
      "publicAccess": true        # Level 0 permission
    }

DBAL (Database Abstraction Layer)

Why? Credentials never touch the frontend. All database operations go through the DBAL daemon.

Architecture:

• TypeScript SDK (dbal/development/) - Development, fast iteration
• C++ Daemon (dbal/production/) - Production, credential protection

Usage:

import { getAdapter } from '@/lib/db/core/dbal-client'

const adapter = getAdapter()
const result = await adapter.list('InstalledPackage', {
  filters: { enabled: true }
})

Database Commands

# Development (can be run from root or frontends/nextjs)
npm run db:generate   # Generate Prisma client
npm run db:push       # Apply schema changes
npm run db:migrate    # Create migration

# From root directory
npm run db:generate   # Delegates to frontends/nextjs

# From frontends/nextjs directory
npm run db:generate   # Uses --schema=../../prisma/schema.prisma

# Production (via Docker)
docker exec -it metabuilder-postgres-prod psql -U metabuilder

---

Development

Prerequisites

# Install Node.js dependencies
cd frontends/nextjs
npm install

# Install C++ build tools (for DBAL/Media daemons)
# Ubuntu/Debian:
sudo apt install cmake ninja-build g++ libssl-dev libpq-dev

# macOS:
brew install cmake ninja conan

Development Workflow

# Terminal 1: Frontend
cd frontends/nextjs
npm run dev

# Terminal 2: DBAL daemon (optional, TypeScript version)
cd dbal/development
npm install
npm run dev

# Terminal 3: PostgreSQL (via Docker)
docker run -p 5432:5432 -e POSTGRES_PASSWORD=dev postgres:16

Code Conventions

CRITICAL RULES:

• ❌ NO hardcoded routes - use PageConfig
• ❌ NO component imports from packages - use generic renderer
• ❌ NO direct database access - use DBAL adapter
• ✅ Material-UI only for UI components
• ✅ Absolute imports with @/ path
• ✅ One lambda per file pattern

Testing

# From frontends/nextjs/
npm run test:unit              # Vitest unit tests
npm run test:unit -- --run     # Run once (no watch)
npm run test:e2e               # Playwright E2E tests
npm run lint                   # ESLint
npm run typecheck              # TypeScript validation
npm run build                  # Production build

---

Project Structure

metabuilder/
├── frontends/nextjs/          # Next.js 14 App Router frontend
│   ├── src/
│   │   ├── app/               # App routes (minimal, generic)
│   │   │   └── page.tsx      # Root route (database-driven)
│   │   ├── components/        # Generic renderers only
│   │   ├── lib/
│   │   │   ├── db/           # DBAL client
│   │   │   ├── packages/     # Package loaders
│   │   │   └── rendering/    # JSON→React renderer
│   │   └── hooks/
│   └── package.json
│
├── packages/                  # Self-contained packages (JSON)
│   ├── ui_home/
│   │   ├── package.json
│   │   ├── components/ui.json
│   │   └── styles/
│   ├── ui_header/
│   ├── dashboard/
│   └── ... (12+ packages)
│
├── seed/                      # Bootstrap data
│   ├── packages/
│   │   └── core-packages.yaml
│   └── database/
│       ├── installed_packages.yaml
│       └── package_permissions.yaml
│
├── dbal/                      # Database Abstraction Layer
│   ├── development/           # TypeScript (dev)
│   └── production/            # C++ daemon (prod)
│       ├── src/
│       ├── include/
│       └── build-config/
│
├── services/
│   └── media_daemon/          # C++ media processing (FFmpeg, HLS)
│       ├── src/
│       ├── include/
│       └── Dockerfile
│
├── deployment/                # Docker deployment
│   ├── docker/
│   │   ├── docker-compose.production.yml
│   │   ├── docker-compose.development.yml
│   │   └── docker-compose.monitoring.yml
│   ├── scripts/
│   │   ├── deploy.sh
│   │   ├── bootstrap-system.sh
│   │   └── backup-database.sh
│   └── config/                # Service configs (Nginx, Prometheus, etc.)
│
├── prisma/
│   └── schema.prisma          # Database schema
│
└── docs/                      # Documentation

---

Key Concepts Recap

1. Ultra-Generic Routing

Traditional:

// Hardcoded route mapping
<Route path="/" component={HomePage} />
<Route path="/dashboard" component={Dashboard} />

MetaBuilder:

// Database-driven routing
const route = await db.query('PageConfig', { path: req.url })
const pkg = await loadPackage(route.packageId)
const component = pkg.components.find(c => c.id === route.component)
return renderJSONComponent(component)

2. Zero Hardcoded Imports

Traditional:

import HomePage from '@/components/HomePage'
import Dashboard from '@/components/Dashboard'

MetaBuilder:

// All components loaded from JSON
const pkg = await loadJSONPackage('/packages/ui_home')
const component = pkg.components.find(c => c.name === 'HomePage')

3. God Panel Control

Supergod users can remap the entire application without touching code:

-- Remap root page
UPDATE "InstalledPackage"
SET config = '{"defaultRoute": "/"}'
WHERE "packageId" = 'dashboard';

-- Override with god panel
INSERT INTO "PageConfig" (path, packageId, component, ...)
VALUES ('/', 'custom_package', 'CustomPage', ...);

4. Complete Loose Coupling

Database (Source of Truth)
    ↓
JSON Packages (Data)
    ↓
Generic Renderer (Engine)
    ↓
React (Output)

No layer knows about the layers above or below.

---

Trade-offs

Pros

✅ Infinite flexibility - remap anything via database ✅ Zero coupling - no hardcoded dependencies ✅ God users control everything - no code deploys needed ✅ Package isolation - packages don't know about each other ✅ Multi-tenant ready - tenant-specific route overrides

Cons

❌ Steep learning curve - "where does this render?" ❌ Harder debugging - trace through database → JSON → renderer ❌ Performance overhead - runtime JSON parsing ❌ Type safety loss - JSON components aren't type-checked ❌ Tooling challenges - no IDE autocomplete for package components

The trade-off is flexibility vs. simplicity.

---

Troubleshooting

Common Issues

Problem	Solution
"Page not found" at /	Check InstalledPackage has defaultRoute: "/" config
Package not rendering	Verify package installed: SELECT * FROM "InstalledPackage"
Permission denied	Check PackagePermission table for correct role
Database connection failed	Check DATABASE_URL env var, run npm run db:push
DBAL daemon won't start	Check PostgreSQL is running, credentials correct

Debug Steps

# 1. Check database
docker exec -it metabuilder-postgres-prod psql -U metabuilder -c "SELECT * FROM \"InstalledPackage\" WHERE \"packageId\" = 'ui_home';"

# 2. Check package files exist
ls -la packages/ui_home/components/

# 3. Check bootstrap logs
docker logs metabuilder-app-prod | grep bootstrap

# 4. View all routes
curl http://localhost:8080/api/list?entity=PageConfig

---

License

MIT License - See LICENSE file

---

Learn More

• Deployment Guide - Detailed deployment documentation
• Package Structure - How to create packages
• DBAL Documentation - Database abstraction layer
• Prisma Schema - Database schema reference

---

Built with the philosophy: Everything is data, nothing is hardcoded.