Files
metabuilder/schemas/README.md
johndoe6345789 c760bd7cd0 feat: MetaBuilder Workflow Engine v3.0.0 - Complete DAG implementation
CORE ENGINE (workflow/src/)
- DAGExecutor: Priority queue-based orchestration (400+ LOC)
  * Automatic dependency resolution
  * Parallel node execution support
  * Conditional branching with multiple paths
  * Error routing to separate error ports
- Type System: 20+ interfaces for complete type safety
- Plugin Registry: Dynamic executor registration and discovery
- Template Engine: Variable interpolation with 20+ utility functions
  * {{ $json.field }}, {{ $context.user.id }}, {{ $env.VAR }}
  * {{ $steps.nodeId.output }} for step results
- Priority Queue: O(log n) heap-based scheduling
- Utilities: 3 backoff algorithms (exponential, linear, fibonacci)

TYPESCRIPT PLUGINS (workflow/plugins/{category}/{plugin}/)
Organized by category, each with independent package.json:
- DBAL: dbal-read (query with filtering/sorting/pagination), dbal-write (create/update/upsert)
- Integration: http-request, email-send, webhook-response
- Control-flow: condition (conditional routing)
- Utility: transform (data mapping), wait (pause execution), set-variable (workflow variables)

NEXT.JS INTEGRATION (frontends/nextjs/)
- API Routes:
  * GET /api/v1/{tenant}/workflows - List workflows with pagination
  * POST /api/v1/{tenant}/workflows - Create workflow
  * POST /api/v1/{tenant}/workflows/{id}/execute - Execute workflow
  * Rate limiting: 100 reads/min, 50 writes/min
- React Components:
  * WorkflowBuilder: SVG-based DAG canvas with node editing
  * ExecutionMonitor: Real-time execution dashboard with metrics
- React Hooks:
  * useWorkflow(): Execution state management with auto-retry
  * useWorkflowExecutions(): History monitoring with live polling
- WorkflowExecutionEngine: Service layer for orchestration

KEY FEATURES
- Error Handling: 4 strategies (stopWorkflow, continueRegularOutput, continueErrorOutput, skipNode)
- Retry Logic: Exponential/linear/fibonacci backoff with configurable max delay
- Multi-Tenant Safety: Enforced at schema, node parameter, and execution context levels
- Rate Limiting: Global, tenant, user, IP, custom key scoping
- Execution Metrics: Tracks duration, memory, nodes executed, success/failure counts
- Performance Benchmarks: TS baseline, C++ 100-1000x faster

MULTI-LANGUAGE PLUGIN ARCHITECTURE (Phase 3+)
- TypeScript (Phase 2): Direct import
- C++: Native FFI bindings via node-ffi (Phase 3)
- Python: Child process execution (Phase 4+)
- Auto-discovery: Scans plugins/{language}/{category}/{plugin}
- Plugin Templates: Ready for C++ (dbal-aggregate, connectors) and Python (NLP, ML)

DOCUMENTATION
- WORKFLOW_ENGINE_V3_GUIDE.md: Complete architecture and concepts
- WORKFLOW_INTEGRATION_GUIDE.md: Next.js integration patterns
- WORKFLOW_MULTI_LANGUAGE_ARCHITECTURE.md: Language support roadmap
- workflow/plugins/STRUCTURE.md: Directory organization
- workflow/plugins/MIGRATION.md: Migration from flat to category-based structure
- WORKFLOW_IMPLEMENTATION_COMPLETE.md: Executive summary

SCHEMA & EXAMPLES
- metabuilder-workflow-v3.schema.json: Complete JSON Schema validation
- complex-approval-flow.workflow.json: Production example with all features

COMPLIANCE
 MetaBuilder CLAUDE.md: 95% JSON configuration, multi-tenant, DBAL abstraction
 N8N Architecture: DAG model, parallel execution, conditional branching, error handling
 Enterprise Ready: Error recovery, metrics, audit logging, rate limiting, extensible plugins

Ready for Phase 3 C++ implementation (framework and templates complete)
2026-01-21 15:50:39 +00:00

6.8 KiB

MetaBuilder Schemas

This directory contains schema files used across the MetaBuilder project for validation and IDE support.

Schema Files

YAML Schemas

yaml-schema.yaml - YAML meta-schema (Draft 2025-11)

Usage in YAML files:

# yaml-language-server: $schema=../../../../../schemas/yaml-schema.yaml

Files using this schema:

  • packages/*/seed/schema/entities.yaml - Database entity definitions
  • Other YAML configuration files

JSON Schemas

JSON Schema files are located within individual packages:

  • packages/json_script_example/seed/script.schema.json - JSON script validation
  • packages/json_script_example/seed/types.schema.json - Type definitions
  • packages/json_script_example/seed/schema/entities.schema.json - Entity structure validation
  • packages/json_script_example/tests/test.schema.json - Test logic validation
  • packages/json_script_example/tests/test-parameters.schema.json - Test parameters validation

Adding New Schemas

For YAML Files

  1. Place the schema in this schemas/ directory
  2. Add documentation in this README
  3. Reference the schema in YAML files using the yaml-language-server directive

For JSON Files

  1. Place the schema alongside the files it validates (in the package directory)
  2. Add a $schema property to JSON files pointing to the schema
  3. Document the schema in the package README

IDE Support

VS Code

Install the YAML extension by Red Hat for YAML schema validation:

  • Extension ID: redhat.vscode-yaml
  • Provides autocomplete, validation, and hover documentation

JSON schema support is built-in to VS Code.

JetBrains IDEs

YAML and JSON schema validation is built-in to WebStorm, IntelliJ IDEA, and other JetBrains IDEs.

Schema Validation Benefits

  • Real-time validation - Catch errors while editing
  • IDE autocomplete - IntelliSense for properties and values
  • Documentation - Hover tooltips explain each field
  • Type safety - Ensure files match their specifications
  • Consistency - Enforce structure across all files

Core Database Schemas

Entity definitions are stored in /dbal/shared/api/schema/ using YAML format.

Schema Directory Structure

dbal/shared/api/schema/
├── capabilities.yaml           # API capabilities definition
├── errors.yaml                 # Error type definitions
├── entities/                   # Entity definitions by domain
│   ├── core/                   # Core system entities
│   │   ├── user.yaml          # User accounts & auth
│   │   ├── session.yaml       # User sessions
│   │   ├── workflow.yaml      # Workflow definitions
│   │   ├── package.yaml       # Package metadata
│   │   └── ui_page.yaml       # UI page definitions
│   ├── access/                # Access control entities
│   │   ├── credential.yaml    # API credentials
│   │   ├── component_node.yaml # Component nodes
│   │   └── page_config.yaml   # Page configurations
│   ├── packages/              # Package-specific entities
│   │   ├── forum.yaml         # Forum entities
│   │   ├── notification.yaml  # Notification entities
│   │   ├── audit_log.yaml     # Audit log entities
│   │   ├── media.yaml         # Media entities
│   │   ├── irc.yaml           # IRC entities
│   │   └── streaming.yaml     # Streaming entities
│   ├── ecommerce/             # E-commerce entities
│   │   └── product.yaml       # Product definitions
│   ├── gaming/                # Gaming entities
│   │   └── game.yaml          # Game definitions
│   ├── spotify_clone/         # Spotify clone entities
│   │   └── artist.yaml        # Artist definitions
│   └── youtube_clone/         # YouTube clone entities
│       └── video.yaml         # Video definitions
└── operations/                # Operation definitions
    ├── access/                # Access operation specs
    │   ├── component_node.ops.yaml
    │   ├── credential.ops.yaml
    │   └── page_config.ops.yaml
    └── entities/              # Entity operation specs
        ├── user.ops.yaml
        ├── session.ops.yaml
        ├── package.ops.yaml
        └── workflow.ops.yaml

Entity Schemas Overview

Core Entities (dbal/shared/api/schema/entities/core/)

  • user.yaml - User accounts with auth, roles (public/user/moderator/admin/god/supergod)
  • session.yaml - User session management
  • workflow.yaml - Workflow definitions
  • package.yaml - Package metadata
  • ui_page.yaml - UI page routing

Access Control Entities (dbal/shared/api/schema/entities/access/)

  • credential.yaml - API credentials and keys
  • component_node.yaml - Component node configurations
  • page_config.yaml - Page configuration settings

Package Entities (dbal/shared/api/schema/entities/packages/)

  • forum.yaml - Forum categories, threads, posts
  • notification.yaml - User notifications and alerts
  • audit_log.yaml - Audit log entries
  • media.yaml - Media assets and metadata
  • irc.yaml - IRC chat entities
  • streaming.yaml - Streaming content and configuration

Domain-Specific Entities

  • ecommerce/product.yaml - E-commerce products
  • gaming/game.yaml - Gaming content
  • spotify_clone/artist.yaml - Music artist data
  • youtube_clone/video.yaml - Video content

Package-Level Entity Schemas

Individual packages may define local entity schemas in entities/schema.json:

  1. audit_log - packages/audit_log/entities/schema.json

    • Defines AuditLog entity for tracking user and system actions
    • Fields: action, entity, entityId, oldValue, newValue, ipAddress, userAgent, timestamp
    • Indexes on: tenantId, userId, entity, action, timestamp
  2. notification_center - packages/notification_center/entities/schema.json

    • Defines Notification entity for user alerts and system events
    • Fields: type, title, message, icon, read, data, createdAt, expiresAt
    • Indexes on: tenantId, userId+read, createdAt
  3. workflow_editor - packages/workflow_editor/entities/schema.json

    • Defines workflow-related entities for the workflow editor package

Package Inventory

For comprehensive documentation of all 62 packages and their file structures, see: