metabuilder/workflow/plugins/MIGRATION.md

Plugin Migration to Multi-Language Structure

Current State → New State

Old Structure (Flat)

workflow/plugins/
├── dbal-read/
├── dbal-write/
├── http-request/
├── email-send/
├── webhook-response/
├── condition/
├── transform/
├── wait/
└── set-variable/

New Structure (Language + Category)

workflow/plugins/
├── ts/                    # TypeScript (Phase 2)
│   ├── dbal/
│   │   ├── dbal-read/
│   │   └── dbal-write/
│   ├── integration/
│   │   ├── http-request/
│   │   ├── email-send/
│   │   └── webhook-response/
│   ├── control-flow/
│   │   └── condition/
│   └── utility/
│       ├── transform/
│       ├── wait/
│       └── set-variable/
├── cpp/                   # C++ (Phase 3 - prepare now)
│   ├── dbal/              # High-perf aggregations
│   ├── integration/       # Native connectors (S3, Redis, Kafka)
│   └── performance/       # Bulk operations
└── python/                # Python (Phase 4+)
    ├── ai/                # ML, NLP
    └── data-science/      # Statistical analysis

Migration Steps

1. Reorganize Existing TypeScript Plugins

# Backup current plugins
cp -r workflow/plugins workflow/plugins.backup

# Create language structure
mkdir -p workflow/plugins/ts/{dbal,integration,control-flow,utility}
mkdir -p workflow/plugins/{cpp,python}

# Move plugins
mv workflow/plugins/dbal-read workflow/plugins/ts/dbal/
mv workflow/plugins/dbal-write workflow/plugins/ts/dbal/
mv workflow/plugins/http-request workflow/plugins/ts/integration/
mv workflow/plugins/email-send workflow/plugins/ts/integration/
mv workflow/plugins/webhook-response workflow/plugins/ts/integration/
mv workflow/plugins/condition workflow/plugins/ts/control-flow/
mv workflow/plugins/transform workflow/plugins/ts/utility/
mv workflow/plugins/wait workflow/plugins/ts/utility/
mv workflow/plugins/set-variable workflow/plugins/ts/utility/

# Clean up old flat structure
rm -rf workflow/plugins/dbal-read workflow/plugins/dbal-write # etc (if different paths)

2. Update package.json Names

For each plugin, update the name to include language and category:

Before:

{
  "name": "@metabuilder/workflow-plugin-dbal-read"
}

After:

{
  "name": "@metabuilder/workflow-plugin-ts-dbal-read",
  "language": "typescript",
  "category": "dbal",
  "nodeType": "dbal-read"
}

3. Update TypeScript Imports

The imports from @metabuilder/workflow remain the same, but the plugin paths change:

Before:

import { dbalReadExecutor } from '@metabuilder/workflow-plugin-dbal-read';

After:

import { dbalReadExecutor } from '@metabuilder/workflow-plugin-ts-dbal-read';

Or use the new registry auto-loader:

await registry.loadAllPlugins('workflow/plugins');

4. Create C++ Plugin Template (for Phase 3)

mkdir -p workflow/plugins/cpp/dbal/dbal-aggregate/{src,build}
mkdir -p workflow/plugins/cpp/integration/{s3-upload,redis-cache,kafka-producer}

# Create C++ plugin template files
cat > workflow/plugins/cpp/dbal/dbal-aggregate/package.json << 'EOF'
{
  "name": "@metabuilder/workflow-plugin-cpp-dbal-aggregate",
  "version": "1.0.0",
  "language": "c++",
  "category": "dbal",
  "nodeType": "dbal-aggregate",
  "main": "build/libaggregate.so",
  "build": "cmake",
  "bindings": "node-ffi"
}
EOF

cat > workflow/plugins/cpp/dbal/dbal-aggregate/CMakeLists.txt << 'EOF'
cmake_minimum_required(VERSION 3.10)
project(workflow-plugin-dbal-aggregate)

set(CMAKE_CXX_STANDARD 17)

add_library(aggregate SHARED
  src/aggregate.cpp
)

target_include_directories(aggregate PRIVATE include)
EOF

5. Prepare Python Plugin Template (for Phase 4+)

mkdir -p workflow/plugins/python/ai/{nlp-process,sentiment-analyze}
mkdir -p workflow/plugins/python/data-science/statistical-analysis

cat > workflow/plugins/python/ai/nlp-process/package.json << 'EOF'
{
  "name": "@metabuilder/workflow-plugin-python-nlp-process",
  "version": "1.0.0",
  "language": "python",
  "category": "ai",
  "nodeType": "nlp-process",
  "runtime": "python3.11",
  "bindings": "child-process"
}
EOF

cat > workflow/plugins/python/ai/nlp-process/requirements.txt << 'EOF'
transformers>=4.30.0
torch>=2.0.0
numpy>=1.24.0
EOF

Package Publishing Strategy

TypeScript Plugins

cd workflow/plugins/ts/dbal/dbal-read
npm publish --access public

C++ Plugins (with prebuilt binaries)

cd workflow/plugins/cpp/dbal/dbal-aggregate
npm run build:native
npm publish --access public

Python Plugins (via pip)

cd workflow/plugins/python/ai/nlp-process
python -m build
python -m twine upload dist/*

Registry Auto-Discovery

New loader that scans directory structure:

async function loadAllPlugins(baseDir: string): Promise<void> {
  const languages = await fs.readdir(path.join(baseDir, 'plugins'));

  for (const lang of languages) {
    if (lang === 'STRUCTURE.md' || lang === 'MIGRATION.md') continue;

    const langPath = path.join(baseDir, 'plugins', lang);
    const categories = await fs.readdir(langPath);

    for (const category of categories) {
      const categoryPath = path.join(langPath, category);
      const plugins = await fs.readdir(categoryPath);

      for (const pluginName of plugins) {
        const pluginPath = path.join(categoryPath, pluginName);
        const packageJson = JSON.parse(
          await fs.readFile(path.join(pluginPath, 'package.json'), 'utf-8')
        );

        // Load based on language
        if (lang === 'ts') {
          const module = require(pluginPath);
          registry.register(packageJson.nodeType, module[`${pluginName}Executor`]);
        } else if (lang === 'cpp') {
          const binding = require(path.join(pluginPath, packageJson.main));
          registry.register(packageJson.nodeType, new binding.Executor());
        } else if (lang === 'python') {
          const executor = new PythonProcessExecutor(
            path.join(pluginPath, 'src/executor.py')
          );
          registry.register(packageJson.nodeType, executor);
        }
      }
    }
  }
}

Breaking Changes

For Users

If publishing plugins to npm:

{
  "old": "@metabuilder/workflow-plugin-dbal-read",
  "new": "@metabuilder/workflow-plugin-ts-dbal-read"
}

Update imports:

// Old
import { dbalReadExecutor } from '@metabuilder/workflow-plugin-dbal-read';

// New
import { dbalReadExecutor } from '@metabuilder/workflow-plugin-ts-dbal-read';

For Plugin Developers

New naming convention:

@metabuilder/workflow-plugin-{language}-{category}-{plugin}

Examples:
  @metabuilder/workflow-plugin-ts-dbal-read
  @metabuilder/workflow-plugin-cpp-dbal-aggregate
  @metabuilder/workflow-plugin-python-ai-nlp

Backward Compatibility

To maintain backward compatibility, create alias packages:

// @metabuilder/workflow-plugin-dbal-read/package.json
{
  "name": "@metabuilder/workflow-plugin-dbal-read",
  "version": "1.0.0",
  "deprecated": "Use @metabuilder/workflow-plugin-ts-dbal-read instead",
  "main": "node_modules/@metabuilder/workflow-plugin-ts-dbal-read/dist/index.js"
}

Timeline

Week 1-2: Phase 2 (Current)

• ✅ Reorganize existing TypeScript plugins
• ✅ Update package.json with language/category
• ✅ Update imports in Next.js integration
• ✅ Test all plugins work in new structure

Week 3-4: Phase 3 Prep

• Create C++ plugin templates
• Set up build system (CMake)
• Document C++ plugin development
• Create example C++ plugin

Week 5+: Phase 3 & Beyond

• Implement C++ plugins (dbal-aggregate, connectors)
• Performance testing and benchmarks
• Python plugin templates
• Multi-language execution testing