mirror of
https://github.com/johndoe6345789/metabuilder.git
synced 2026-05-01 01:05:00 +00:00
CORE ENGINE (workflow/src/)
- DAGExecutor: Priority queue-based orchestration (400+ LOC)
* Automatic dependency resolution
* Parallel node execution support
* Conditional branching with multiple paths
* Error routing to separate error ports
- Type System: 20+ interfaces for complete type safety
- Plugin Registry: Dynamic executor registration and discovery
- Template Engine: Variable interpolation with 20+ utility functions
* {{ $json.field }}, {{ $context.user.id }}, {{ $env.VAR }}
* {{ $steps.nodeId.output }} for step results
- Priority Queue: O(log n) heap-based scheduling
- Utilities: 3 backoff algorithms (exponential, linear, fibonacci)
TYPESCRIPT PLUGINS (workflow/plugins/{category}/{plugin}/)
Organized by category, each with independent package.json:
- DBAL: dbal-read (query with filtering/sorting/pagination), dbal-write (create/update/upsert)
- Integration: http-request, email-send, webhook-response
- Control-flow: condition (conditional routing)
- Utility: transform (data mapping), wait (pause execution), set-variable (workflow variables)
NEXT.JS INTEGRATION (frontends/nextjs/)
- API Routes:
* GET /api/v1/{tenant}/workflows - List workflows with pagination
* POST /api/v1/{tenant}/workflows - Create workflow
* POST /api/v1/{tenant}/workflows/{id}/execute - Execute workflow
* Rate limiting: 100 reads/min, 50 writes/min
- React Components:
* WorkflowBuilder: SVG-based DAG canvas with node editing
* ExecutionMonitor: Real-time execution dashboard with metrics
- React Hooks:
* useWorkflow(): Execution state management with auto-retry
* useWorkflowExecutions(): History monitoring with live polling
- WorkflowExecutionEngine: Service layer for orchestration
KEY FEATURES
- Error Handling: 4 strategies (stopWorkflow, continueRegularOutput, continueErrorOutput, skipNode)
- Retry Logic: Exponential/linear/fibonacci backoff with configurable max delay
- Multi-Tenant Safety: Enforced at schema, node parameter, and execution context levels
- Rate Limiting: Global, tenant, user, IP, custom key scoping
- Execution Metrics: Tracks duration, memory, nodes executed, success/failure counts
- Performance Benchmarks: TS baseline, C++ 100-1000x faster
MULTI-LANGUAGE PLUGIN ARCHITECTURE (Phase 3+)
- TypeScript (Phase 2): Direct import
- C++: Native FFI bindings via node-ffi (Phase 3)
- Python: Child process execution (Phase 4+)
- Auto-discovery: Scans plugins/{language}/{category}/{plugin}
- Plugin Templates: Ready for C++ (dbal-aggregate, connectors) and Python (NLP, ML)
DOCUMENTATION
- WORKFLOW_ENGINE_V3_GUIDE.md: Complete architecture and concepts
- WORKFLOW_INTEGRATION_GUIDE.md: Next.js integration patterns
- WORKFLOW_MULTI_LANGUAGE_ARCHITECTURE.md: Language support roadmap
- workflow/plugins/STRUCTURE.md: Directory organization
- workflow/plugins/MIGRATION.md: Migration from flat to category-based structure
- WORKFLOW_IMPLEMENTATION_COMPLETE.md: Executive summary
SCHEMA & EXAMPLES
- metabuilder-workflow-v3.schema.json: Complete JSON Schema validation
- complex-approval-flow.workflow.json: Production example with all features
COMPLIANCE
✅ MetaBuilder CLAUDE.md: 95% JSON configuration, multi-tenant, DBAL abstraction
✅ N8N Architecture: DAG model, parallel execution, conditional branching, error handling
✅ Enterprise Ready: Error recovery, metrics, audit logging, rate limiting, extensible plugins
Ready for Phase 3 C++ implementation (framework and templates complete)
Set Variable Node Plugin
Set workflow variables for use in subsequent nodes.
Installation
npm install @metabuilder/workflow-plugin-set-variable
Usage
{
"id": "set-vars",
"type": "operation",
"nodeType": "set-variable",
"parameters": {
"variables": {
"userId": "{{ $json.id }}",
"userEmail": "{{ $json.email }}",
"processDate": "{{ new Date().toISOString() }}"
},
"mode": "merge"
}
}
Operations
Set Simple Variables
Store string and number values:
{
"variables": {
"count": 0,
"name": "John Doe",
"active": true
}
}
Set Variables from Input Data
Reference input data with template expressions:
{
"variables": {
"userId": "{{ $json.id }}",
"userName": "{{ $json.name }}",
"userStatus": "{{ $json.status }}"
}
}
Set Computed Variables
Use expressions to compute values:
{
"variables": {
"fullName": "{{ $json.firstName + ' ' + $json.lastName }}",
"isActive": "{{ $json.status === 'active' }}",
"count": "{{ $json.items.length }}"
}
}
Set Complex Variables
Store objects and arrays:
{
"variables": {
"userData": {
"id": "{{ $json.id }}",
"name": "{{ $json.name }}",
"email": "{{ $json.email }}"
},
"tags": ["{{ $json.tag1 }}", "{{ $json.tag2 }}"]
}
}
Merge vs Replace Mode
Merge adds to existing variables (default):
{
"variables": { "newVar": "value" },
"mode": "merge"
}
Replace clears all previous variables:
{
"variables": { "var1": "value" },
"mode": "replace"
}
Parameters
variables(required): Object with variable definitions- Keys must be valid variable names (alphanumeric + underscore)
- Values can be strings, numbers, booleans, objects, or arrays
- Supports template expressions in strings
- Recursively interpolates nested objects
mode(optional): How to handle existing variablesmerge(default): Add new variables to existing setreplace: Clear existing variables firstappend: Append to existing array variables
Variable Names
Valid variable names:
- Must start with letter (a-z, A-Z) or underscore (_)
- Can contain letters, numbers (0-9), and underscores
- Are case-sensitive
- Examples:
userId,_private,tempData123
Invalid variable names:
123var(starts with number)user-id(contains hyphen)user.id(contains dot)user@id(contains special char)
Template Expressions
Variable values support template interpolation:
{{ $json.fieldName }}- Access input field{{ $json.field1 + $json.field2 }}- Concatenate/add{{ $json.price * 1.1 }}- Calculate{{ $json.items.length }}- Array length{{ $json.active ? 'yes' : 'no' }}- Conditionals{{ new Date().toISOString() }}- Current timestamp{{ $env.API_KEY }}- Environment variables{{ $context.tenantId }}- Context values
Reserved Names
These names conflict with built-in variables and should be avoided:
contextstatejsonenvutils$json$context$state$env
Using reserved names will trigger a validation warning.
Features
- Template expression interpolation in variable values
- Nested object and array support
- Multiple variable setting in single node
- Mode selection (merge/replace/append)
- Variable name validation
- Reserved name detection
- Type preservation (strings, numbers, booleans, objects)
- Recursive object interpolation
Examples
Set User Information
{
"id": "set-user-vars",
"nodeType": "set-variable",
"parameters": {
"variables": {
"userId": "{{ $json.id }}",
"userName": "{{ $json.name }}",
"userEmail": "{{ $json.email }}",
"isAdmin": "{{ $json.role === 'admin' }}"
}
}
}
Set Computed Values
{
"id": "set-computed",
"nodeType": "set-variable",
"parameters": {
"variables": {
"total": "{{ $json.subtotal + $json.tax }}",
"discount": "{{ $json.total * 0.1 }}",
"finalPrice": "{{ ($json.subtotal + $json.tax) - ($json.total * 0.1) }}",
"timestamp": "{{ new Date().toISOString() }}"
}
}
}
Set Conditional Variables
{
"id": "set-conditional",
"nodeType": "set-variable",
"parameters": {
"variables": {
"status": "{{ $json.amount > 1000 ? 'high' : 'normal' }}",
"requiresApproval": "{{ $json.amount > 5000 }}",
"priority": "{{ $json.urgent ? 1 : 3 }}"
}
}
}
Set Complex Data Structure
{
"id": "set-complex",
"nodeType": "set-variable",
"parameters": {
"variables": {
"order": {
"id": "{{ $json.id }}",
"customer": {
"name": "{{ $json.customerName }}",
"email": "{{ $json.customerEmail }}"
},
"items": "{{ $json.items }}",
"totals": {
"subtotal": "{{ $json.subtotal }}",
"tax": "{{ $json.tax }}",
"total": "{{ $json.total }}"
}
}
}
}
}
Replace All Variables
{
"id": "reset-variables",
"nodeType": "set-variable",
"parameters": {
"variables": {
"step": 1,
"status": "processing"
},
"mode": "replace"
}
}
Build from Previous Variable
{
"id": "build-from-var",
"nodeType": "set-variable",
"parameters": {
"variables": {
"processedUserId": "{{ $context.variables.userId }}_processed",
"incrementedCount": "{{ parseInt($context.variables.count) + 1 }}"
}
}
}
Accessing Variables in Subsequent Nodes
Once set, variables are available in other nodes:
{
"id": "next-node",
"nodeType": "http-request",
"parameters": {
"url": "{{ $context.variables.apiUrl }}",
"method": "POST",
"body": {
"userId": "{{ $context.variables.userId }}",
"email": "{{ $context.variables.userEmail }}"
}
}
}
Variable Scope
Variables are scoped to workflow execution:
- Available to all subsequent nodes in same execution
- Not persisted across executions
- Can be overwritten by later Set Variable nodes
- Lost when workflow completes
License
MIT