Files
metabuilder/workflowui
johndoe6345789 bb45bb5c0f feat(workflowui): add SQLAlchemy database layer with persistent storage
Implemented SQLAlchemy ORM for database persistence:

**Models**:
- Workflow: Stores workflow definitions with JSON fields for nodes/connections
- Execution: Tracks workflow execution runs with status, timing, and results
- NodeType: Caches available node types for registry lookups
- AuditLog: Tracks all changes for compliance and debugging

**Database Features**:
- Proper indexing for efficient queries (tenant_id, workflow_id, status, etc.)
- Foreign key relationships with cascade deletes
- JSON fields for flexible node/connection storage
- Audit trail for all workflow changes

**Updated Flask Server** (server_sqlalchemy.py):
- Uses Flask-SQLAlchemy for ORM
- Database connection pooling
- Proper error handling and transactions
- Audit logging on all changes
- Pagination support for list endpoints

**Configuration**:
- Support for SQLite (development) and PostgreSQL (production)
- Environment-based database URL
- Automatic table creation
- Transaction management

**Endpoints Enhanced**:
- Pagination (limit, offset parameters)
- Better error handling with proper HTTP status codes
- Total count in list responses
- Change tracking in audit logs

**Performance Optimizations**:
- Indexed queries for common filters
- Database constraints to prevent invalid states
- Efficient JSON storage for flexible fields
- Connection pooling for concurrent requests

Architecture:
- SQLAlchemy models separate from Flask routes
- ORM handles all database interactions
- JSON serialization/deserialization for flexibility
- Backward compatible API responses

Migration Path:
- In-memory server (server.py) still available for simple deployments
- Production server uses SQLAlchemy (server_sqlalchemy.py)
- Database-first schema allows easy schema migration tools

Next: Update Flask app to use server_sqlalchemy.py by default

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-01-23 02:10:43 +00:00
..

WorkflowUI - Visual Workflow Editor

Modern, production-grade visual workflow editor inspired by n8n, built for MetaBuilder's DAG executor system.

🎯 Overview

WorkflowUI is a full-featured workflow editor with:

  • Visual DAG Editor: Drag-and-drop node-based workflow construction with React Flow
  • Real-Time Collaboration: Live updates via Redux and WebSockets
  • Offline-First: IndexedDB for local workflow storage and sync
  • Plugin Ecosystem: Support for custom node types and extensions
  • Type-Safe: Full TypeScript with strict mode enabled

🏗️ Architecture

Frontend (Next.js + React)
  ├─ UI Components (FakeMUI)
  ├─ Redux Store (state management)
  ├─ React Flow (DAG visualization)
  └─ IndexedDB (offline storage)
       │
       ▼
Backend (Flask + Python)
  ├─ Workflow execution
  ├─ Plugin management
  ├─ Database persistence
  └─ WebSocket server
       │
       ▼
MetaBuilder DAG Executor
  ├─ Node execution
  ├─ Error recovery
  ├─ Multi-tenant support
  └─ Plugin registry

📦 Tech Stack

Layer Technology Purpose
Frontend Next.js 14 Server-side rendering, API routes
UI Framework FakeMUI Material UI compatible components
State Redux + Redux Toolkit Centralized state management
Flow Editor React Flow DAG visualization and manipulation
Styling SCSS Component scoped styles
Storage IndexedDB (Dexie) Offline workflow persistence
HTTP Axios API client with interceptors
Backend Flask Python workflow execution
Validation n8n JSON Schema Workflow and node validation

🚀 Quick Start

Prerequisites

Node.js 18+
Python 3.11+
npm or yarn

Installation

# Install frontend dependencies
npm install

# Install backend dependencies
pip install -r backend/requirements.txt

# Initialize IndexedDB
npm run db:init

Development

# Start both frontend and backend
npm run dev:all

# Or separately:
npm run dev          # Frontend on http://localhost:3000
npm run backend      # Backend on http://localhost:5000

Build

npm run build
npm start

📂 Folder Structure

workflowui/
├── src/
│   ├── app/                    # Next.js app directory
│   │   ├── layout.tsx          # Root layout
│   │   ├── page.tsx            # Dashboard page
│   │   ├── editor/[id].tsx     # Workflow editor
│   │   └── api/                # Next.js API routes
│   │
│   ├── components/             # React components
│   │   ├── Editor/             # Workflow editor components
│   │   │   ├── Canvas.tsx      # React Flow canvas
│   │   │   ├── Toolbar.tsx     # Editor toolbar
│   │   │   ├── NodePanel.tsx   # Node configuration panel
│   │   │   └── Properties.tsx  # Node properties panel
│   │   ├── Nodes/              # Custom node types
│   │   │   ├── BaseNode.tsx    # Base node wrapper
│   │   │   ├── PlaywrightNode.tsx
│   │   │   ├── StorybookNode.tsx
│   │   │   └── CustomNode.tsx
│   │   ├── UI/                 # Shared UI components
│   │   │   ├── Button.tsx
│   │   │   ├── Modal.tsx
│   │   │   ├── Sidebar.tsx
│   │   │   └── Toolbar.tsx
│   │   └── Layout/             # Layout components
│   │       ├── Header.tsx
│   │       └── MainLayout.tsx
│   │
│   ├── store/                  # Redux store
│   │   ├── store.ts            # Store configuration
│   │   ├── slices/
│   │   │   ├── workflowSlice.ts
│   │   │   ├── editorSlice.ts
│   │   │   ├── nodesSlice.ts
│   │   │   ├── connectionSlice.ts
│   │   │   └── uiSlice.ts
│   │   └── types.ts            # Store type definitions
│   │
│   ├── services/               # API/backend services
│   │   ├── workflowService.ts  # Workflow API calls
│   │   ├── nodeService.ts      # Node registry API
│   │   ├── executionService.ts # Workflow execution
│   │   └── storageService.ts   # IndexedDB persistence
│   │
│   ├── db/                     # IndexedDB schema & operations
│   │   ├── schema.ts           # Dexie schema
│   │   ├── workflows.ts        # Workflow queries
│   │   ├── nodes.ts            # Node cache
│   │   └── cache.ts            # Cache operations
│   │
│   ├── hooks/                  # React hooks
│   │   ├── useWorkflow.ts      # Workflow state hook
│   │   ├── useEditor.ts        # Editor state hook
│   │   ├── useNodes.ts         # Node operations hook
│   │   ├── useConnection.ts    # Connection hook
│   │   └── useStorage.ts       # IndexedDB hook
│   │
│   ├── types/                  # TypeScript types
│   │   ├── workflow.ts         # Workflow types
│   │   ├── node.ts             # Node types
│   │   ├── connection.ts       # Connection types
│   │   └── index.ts            # Export all types
│   │
│   ├── utils/                  # Utility functions
│   │   ├── dagValidation.ts    # DAG validation
│   │   ├── nodeFactory.ts      # Node creation factory
│   │   ├── layoutEngine.ts     # Auto-layout algorithm
│   │   ├── jsonValidator.ts    # JSON Schema validation
│   │   └── transformers.ts     # n8n format converters
│   │
│   └── styles/                 # Global styles
│       ├── globals.scss        # Global styles
│       ├── variables.scss      # Design tokens
│       └── mixins.scss         # SCSS mixins
│
├── backend/                    # Flask backend
│   ├── server.py               # Flask app
│   ├── requirements.txt        # Python dependencies
│   ├── routes/
│   │   ├── workflows.py        # Workflow endpoints
│   │   ├── execution.py        # Execution endpoints
│   │   └── nodes.py            # Node registry
│   ├── models/
│   │   ├── workflow.py         # Workflow model
│   │   └── execution.py        # Execution model
│   ├── services/
│   │   ├── executor.py         # Workflow execution
│   │   └── storage.py          # Persistence
│   └── utils/
│       ├── validation.py       # Workflow validation
│       └── converters.py       # Format converters
│
├── workflows/                  # Sample workflows
│   ├── e2e-testing.json
│   ├── documentation.json
│   └── complex-pipeline.json
│
├── stories/                    # Storybook stories
│   ├── Editor.stories.tsx
│   ├── Nodes.stories.tsx
│   └── UI.stories.tsx
│
├── scripts/                    # Build/setup scripts
│   ├── init-db.js              # Initialize IndexedDB
│   └── migrate-db.js           # Database migrations
│
├── public/                     # Static assets
│   └── icons/
│
├── tsconfig.json               # TypeScript config
├── next.config.js              # Next.js config
├── jest.config.js              # Jest config
├── .storybook/                 # Storybook config
└── README.md

🎨 Key Components

1. Canvas (React Flow)

// Drag-and-drop DAG editor with zoom, pan, selection
<WorkflowCanvas>
  <ReactFlowProvider>
    <ReactFlow nodes={nodes} edges={edges} onConnect={onConnect}>
      <Background />
      <Controls />
      <MiniMap />
    </ReactFlow>
  </ReactFlowProvider>
</WorkflowCanvas>

2. Node Types

// Playwright node example
<PlaywrightNode
  id="test_chromium"
  data={{
    browser: "chromium",
    baseUrl: "http://localhost:3000",
    headless: true
  }}
/>

3. Redux Store

// Central state management
store/slices:
  - workflowSlice: Current workflow, metadata
  - editorSlice: Canvas zoom, pan, selection
  - nodesSlice: Node registry, templates
  - connectionSlice: Edge creation state
  - uiSlice: Modals, panels, notifications

4. IndexedDB Storage

// Offline-first storage with Dexie
db.workflows.put(workflow)
db.workflows.get(id)
db.workflows.toArray()
db.syncWithServer() // Auto-sync when online

5. Backend API

# Flask endpoints
POST   /api/workflows              # Create workflow
GET    /api/workflows/<id>         # Get workflow
PUT    /api/workflows/<id>         # Update workflow
DELETE /api/workflows/<id>         # Delete workflow
POST   /api/workflows/<id>/execute # Execute workflow
GET    /api/nodes                  # Get node registry

🔌 Plugin System

Custom node types can be added:

// Define custom node
const CustomNode: NodeType = {
  id: 'my.custom',
  name: 'My Custom Node',
  category: 'custom',
  parameters: {
    field1: { type: 'string', required: true },
    field2: { type: 'number', required: false }
  }
};

// Register in node registry
registerNodeType(CustomNode);

📊 Features

Visual Editing

  • Drag-and-drop node creation
  • Connection drawing with validation
  • Node selection and multi-select
  • Undo/redo support
  • Auto-layout (horizontal, vertical, hierarchical)
  • Zoom and pan controls
  • Minimap navigation

Workflow Management

  • Create/edit/delete workflows
  • Version control integration
  • Workflow templates
  • Import/export (JSON, n8n format)
  • Workflow validation (DAG constraints)
  • Multi-tenant support

Execution

  • Dry-run execution
  • Execution history
  • Result visualization
  • Error reporting
  • Step-by-step debugging
  • Performance metrics

Developer Experience

  • Hot reload
  • Redux DevTools integration
  • Storybook for component development
  • TypeScript strict mode
  • Comprehensive error messages
  • Debug mode

🧪 Testing

# Run tests
npm test

# Watch mode
npm test:watch

# Component development
npm run storybook

📚 Documentation

🚀 Deployment

# Build for production
npm run build

# Start production server
npm start

Deployed with:

  • Vercel (Next.js frontend)
  • Heroku/Railway (Flask backend)
  • MongoDB Atlas (persistent storage)

🤝 Contributing

  1. Create feature branch: git checkout -b feature/name
  2. Make changes with tests
  3. Submit PR with description

📄 License

Part of MetaBuilder project