Implement automatic plugin discovery and fix package.json files

Co-authored-by: johndoe6345789 <224850594+johndoe6345789@users.noreply.github.com>

PACKAGE_JSON_GUIDE.md

@@ -0,0 +1,255 @@
+# Package.json Files in AutoMetabuilder
+
+This document explains the purpose and location of package.json files throughout the AutoMetabuilder project to make them easy to find and understand.
+
+## Overview
+
+AutoMetabuilder uses `package.json` files in two main contexts:
+
+1. **Workflow Plugin Packages** - Define individual workflow plugins
+2. **Workflow Template Packages** - Define complete workflow templates
+
+## Workflow Plugin Packages
+
+### Location
+```
+backend/autometabuilder/workflow/plugins/<category>/<plugin_name>/package.json
+```
+
+### Purpose
+Each workflow plugin has a `package.json` that defines:
+- Plugin name and type
+- Entry point (Python file)
+- Metadata and categorization
+
+### Structure
+```json
+{
+  "name": "@autometabuilder/plugin_name",
+  "version": "1.0.0",
+  "description": "Plugin description",
+  "main": "plugin_file.py",
+  "author": "AutoMetabuilder",
+  "license": "MIT",
+  "keywords": ["category", "keyword"],
+  "metadata": {
+    "plugin_type": "category.plugin_name",
+    "category": "category"
+  }
+}
+```
+
+### Key Fields
+- **`name`**: NPM-style package name (e.g., `@autometabuilder/web_api_navigation`)
+- **`main`**: Python file containing the `run()` function
+- **`metadata.plugin_type`**: The actual plugin identifier used in workflows (e.g., `web.api_navigation`)
+- **`metadata.category`**: Plugin category for organization
+
+### Plugin Discovery
+Plugins are **automatically discovered** by scanning for package.json files in the plugins directory. No manual registration required!
+
+### Categories
+- `backend/` - Backend initialization plugins
+- `core/` - Core workflow operations
+- `web/` - Web/Flask server plugins
+- `control/` - Control flow plugins
+- `logic/` - Logical operations
+- `math/` - Mathematical operations
+- `string/` - String manipulation
+- `list/` - List operations
+- `dict/` - Dictionary operations
+- `convert/` - Type conversion
+- `utils/` - Utility functions
+
+### Finding All Plugin package.json Files
+```bash
+# Find all plugin package.json files
+find backend/autometabuilder/workflow/plugins -name "package.json"
+
+# Count plugins by category
+find backend/autometabuilder/workflow/plugins -name "package.json" | \
+  cut -d'/' -f5 | sort | uniq -c
+```
+
+## Workflow Template Packages
+
+### Location
+```
+backend/autometabuilder/packages/<workflow_name>/package.json
+```
+
+### Purpose
+Workflow packages define complete, reusable workflow templates that can be selected and executed.
+
+### Structure
+```json
+{
+  "name": "workflow_name",
+  "version": "1.0.0",
+  "description": "Workflow description",
+  "main": "workflow.json",
+  "author": "AutoMetabuilder",
+  "metadata": {
+    "label": "Human Readable Name",
+    "tags": ["tag1", "tag2"],
+    "icon": "icon_name",
+    "category": "templates"
+  }
+}
+```
+
+### Key Fields
+- **`name`**: Workflow identifier (used as `id` in the system)
+- **`main`**: Workflow JSON file (usually `workflow.json`)
+- **`metadata.label`**: Display name in UI
+- **`metadata.tags`**: Tags for filtering/searching
+- **`metadata.category`**: Organization category
+
+### Available Workflows
+```bash
+# List all workflow packages
+ls -1 backend/autometabuilder/packages/
+
+# Find all workflow package.json files
+find backend/autometabuilder/packages -name "package.json" -maxdepth 2
+```
+
+## Example: Creating a New Plugin
+
+### 1. Create Plugin Directory
+```bash
+mkdir -p backend/autometabuilder/workflow/plugins/web/web_my_plugin
+```
+
+### 2. Create package.json
+```json
+{
+  "name": "@autometabuilder/web_my_plugin",
+  "version": "1.0.0",
+  "description": "My custom plugin",
+  "main": "web_my_plugin.py",
+  "author": "Your Name",
+  "license": "MIT",
+  "keywords": ["web", "custom"],
+  "metadata": {
+    "plugin_type": "web.my_plugin",
+    "category": "web"
+  }
+}
+```
+
+### 3. Create Plugin Python File
+```python
+# web_my_plugin.py
+def run(runtime, inputs):
+    """Plugin implementation."""
+    return {"result": "success"}
+```
+
+### 4. Use in Workflow
+The plugin will be **automatically discovered** and can be used immediately:
+```json
+{
+  "id": "my_node",
+  "name": "My Node",
+  "type": "web.my_plugin",
+  "parameters": {}
+}
+```
+
+## Example: Creating a New Workflow Package
+
+### 1. Create Workflow Directory
+```bash
+mkdir -p backend/autometabuilder/packages/my_workflow
+```
+
+### 2. Create package.json
+```json
+{
+  "name": "my_workflow",
+  "version": "1.0.0",
+  "description": "My custom workflow",
+  "main": "workflow.json",
+  "author": "Your Name",
+  "metadata": {
+    "label": "My Custom Workflow",
+    "tags": ["custom", "example"],
+    "icon": "workflow",
+    "category": "templates"
+  }
+}
+```
+
+### 3. Create workflow.json
+Create an n8n-style workflow JSON with nodes and connections.
+
+## Quick Reference
+
+### Find All package.json Files
+```bash
+# All package.json in the project
+find backend -name "package.json" -type f
+
+# Only plugin packages
+find backend/autometabuilder/workflow/plugins -name "package.json"
+
+# Only workflow packages
+find backend/autometabuilder/packages -name "package.json" -maxdepth 2
+
+# Count total
+find backend -name "package.json" -type f | wc -l
+```
+
+### Validate package.json Files
+```bash
+# Check for valid JSON
+find backend -name "package.json" -exec python3 -m json.tool {} \; > /dev/null
+
+# Check for required fields in plugin packages
+find backend/autometabuilder/workflow/plugins -name "package.json" -exec \
+  python3 -c "import json, sys; \
+  data = json.load(open(sys.argv[1])); \
+  assert 'metadata' in data and 'plugin_type' in data['metadata'], \
+  f'{sys.argv[1]} missing metadata.plugin_type'" {} \;
+```
+
+## Key Differences
+
+| Aspect | Plugin Package | Workflow Package |
+|--------|---------------|------------------|
+| **Location** | `workflow/plugins/<category>/<name>/` | `packages/<name>/` |
+| **Purpose** | Single reusable operation | Complete workflow template |
+| **Main File** | Python file with `run()` function | workflow.json |
+| **Identifier** | `metadata.plugin_type` | `name` field |
+| **Discovery** | Automatic scanning | Loaded via `web.load_workflow_packages` |
+| **Usage** | Referenced in workflow nodes | Selected as workflow template |
+
+## Notes
+
+- **No manual registration**: Plugins are automatically discovered by scanning
+- **package.json is mandatory**: Every plugin and workflow must have one
+- **Consistent naming**: Use `@autometabuilder/` prefix for plugin names
+- **Plugin type vs name**: `metadata.plugin_type` is used in workflows, not `name`
+- **Case sensitivity**: Plugin types are case-sensitive (e.g., `web.api_navigation`)
+
+## Troubleshooting
+
+### Plugin not found
+1. Check `package.json` exists
+2. Verify `metadata.plugin_type` field is set
+3. Ensure Python file has `run()` function
+4. Check Python file name matches `main` field (without .py)
+
+### Workflow package not loading
+1. Check `package.json` exists in workflow directory
+2. Verify `workflow.json` exists
+3. Check `main` field points to correct file
+4. Validate JSON syntax
+
+## Resources
+
+- Plugin registry: `backend/autometabuilder/workflow/plugin_registry.py`
+- Package loader: `backend/autometabuilder/workflow/plugins/web/web_load_workflow_packages/`
+- Example plugins: `backend/autometabuilder/workflow/plugins/*/`
+- Example workflows: `backend/autometabuilder/packages/*/`

backend/autometabuilder/workflow/plugin_registry.py

@@ -1,24 +1,97 @@
-"""Workflow plugin registry."""
+"""Workflow plugin registry with automatic plugin discovery."""
 import json
 import logging
 import os
+from pathlib import Path
 from .plugin_loader import load_plugin_callable
 
 logger = logging.getLogger("autometabuilder")
 
 
+def scan_plugins() -> dict:
+    """
+    Automatically scan and discover workflow plugins.
+    
+    Scans the plugins directory and subdirectories, looking for package.json files
+    that define plugins. Returns a map of plugin_name -> callable_path.
+    
+    Plugin structure:
+    - Each plugin is in its own directory with a package.json file
+    - Plugin name can be in "metadata.plugin_type" (preferred) or "name" field
+    - package.json must have a "main" field pointing to the Python file
+    - The Python file must have a "run" function
+    """
+    plugin_map = {}
+    plugins_base = Path(__file__).parent / "plugins"
+    
+    if not plugins_base.exists():
+        logger.warning("Plugins directory not found: %s", plugins_base)
+        return plugin_map
+    
+    # Scan all subdirectories for package.json files
+    for package_json_path in plugins_base.rglob("package.json"):
+        try:
+            # Read package.json
+            with open(package_json_path, "r", encoding="utf-8") as f:
+                package_data = json.load(f)
+            
+            # Try metadata.plugin_type first (preferred), then fall back to name
+            metadata = package_data.get("metadata", {})
+            plugin_name = metadata.get("plugin_type") or package_data.get("name")
+            main_file = package_data.get("main")
+            
+            if not plugin_name or not main_file:
+                logger.debug("Skipping %s: missing 'plugin_type'/'name' or 'main' field", package_json_path)
+                continue
+            
+            # Build the Python module path
+            plugin_dir = package_json_path.parent
+            main_file_stem = Path(main_file).stem  # Remove .py extension
+            
+            # Calculate relative path from plugins directory
+            rel_path = plugin_dir.relative_to(plugins_base)
+            
+            # Build module path: autometabuilder.workflow.plugins.<category>.<plugin_dir>.<main_file>.run
+            parts = ["autometabuilder", "workflow", "plugins"] + list(rel_path.parts) + [main_file_stem, "run"]
+            callable_path = ".".join(parts)
+            
+            plugin_map[plugin_name] = callable_path
+            logger.debug("Discovered plugin %s -> %s", plugin_name, callable_path)
+            
+        except json.JSONDecodeError:
+            logger.warning("Invalid JSON in %s", package_json_path)
+        except Exception as error:  # pylint: disable=broad-exception-caught
+            logger.debug("Error scanning %s: %s", package_json_path, error)
+    
+    logger.info("Discovered %d plugins via scanning", len(plugin_map))
+    return plugin_map
+
+
 def load_plugin_map() -> dict:
-    """Load workflow plugin map JSON."""
+    """
+    Load workflow plugin map.
+    
+    This function now uses automatic plugin discovery by scanning the plugins
+    directory instead of reading from a static plugin_map.json file.
+    
+    Falls back to plugin_map.json if it exists (for backwards compatibility).
+    """
+    # Try scanning first
+    plugin_map = scan_plugins()
+    
+    # If no plugins found, try legacy plugin_map.json as fallback
+    if not plugin_map:
         map_path = os.path.join(os.path.dirname(__file__), "plugin_map.json")
-    if not os.path.exists(map_path):
-        return {}
+        if os.path.exists(map_path):
+            logger.info("Using legacy plugin_map.json")
             try:
                 with open(map_path, "r", encoding="utf-8") as f:
                     data = json.load(f)
+                    plugin_map = data if isinstance(data, dict) else {}
             except json.JSONDecodeError:
                 logger.error("Invalid workflow plugin map JSON.")
-        return {}
-    return data if isinstance(data, dict) else {}
+    
+    return plugin_map
 
 
 class PluginRegistry:

backend/autometabuilder/workflow/plugins/web/web_api_navigation/package.json

@@ -1,8 +1,18 @@
 {
-  "name": "web.api_navigation",
+  "name": "@autometabuilder/web_api_navigation",
   "version": "1.0.0",
   "description": "Handle /api/navigation endpoint",
   "main": "web_api_navigation.py",
   "author": "AutoMetabuilder",
+  "license": "MIT",
+  "keywords": [
+    "web",
+    "api",
+    "navigation",
+    "plugin"
+  ],
+  "metadata": {
+    "plugin_type": "web.api_navigation",
     "category": "web"
+  }
 }

backend/autometabuilder/workflow/plugins/web/web_api_translation_options/package.json

@@ -1,8 +1,19 @@
 {
-  "name": "web.api_translation_options",
+  "name": "@autometabuilder/web_api_translation_options",
   "version": "1.0.0",
   "description": "Handle /api/translation-options endpoint",
   "main": "web_api_translation_options.py",
   "author": "AutoMetabuilder",
+  "license": "MIT",
+  "keywords": [
+    "web",
+    "api",
+    "translation",
+    "i18n",
+    "plugin"
+  ],
+  "metadata": {
+    "plugin_type": "web.api_translation_options",
     "category": "web"
+  }
 }

backend/autometabuilder/workflow/plugins/web/web_api_workflow_graph/package.json

@@ -1,8 +1,19 @@
 {
-  "name": "web.api_workflow_graph",
+  "name": "@autometabuilder/web_api_workflow_graph",
   "version": "1.0.0",
   "description": "Handle /api/workflow/graph endpoint",
   "main": "web_api_workflow_graph.py",
   "author": "AutoMetabuilder",
+  "license": "MIT",
+  "keywords": [
+    "web",
+    "api",
+    "workflow",
+    "graph",
+    "plugin"
+  ],
+  "metadata": {
+    "plugin_type": "web.api_workflow_graph",
     "category": "web"
+  }
 }

backend/autometabuilder/workflow/plugins/web/web_api_workflow_packages/package.json

@@ -1,8 +1,19 @@
 {
-  "name": "web.api_workflow_packages",
+  "name": "@autometabuilder/web_api_workflow_packages",
   "version": "1.0.0",
   "description": "Handle /api/workflow/packages endpoint",
   "main": "web_api_workflow_packages.py",
   "author": "AutoMetabuilder",
+  "license": "MIT",
+  "keywords": [
+    "web",
+    "api",
+    "workflow",
+    "packages",
+    "plugin"
+  ],
+  "metadata": {
+    "plugin_type": "web.api_workflow_packages",
     "category": "web"
+  }
 }

backend/autometabuilder/workflow/plugins/web/web_api_workflow_plugins/package.json

@@ -1,8 +1,19 @@
 {
-  "name": "web.api_workflow_plugins",
+  "name": "@autometabuilder/web_api_workflow_plugins",
   "version": "1.0.0",
   "description": "Handle /api/workflow/plugins endpoint",
   "main": "web_api_workflow_plugins.py",
   "author": "AutoMetabuilder",
+  "license": "MIT",
+  "keywords": [
+    "web",
+    "api",
+    "workflow",
+    "plugins",
+    "plugin"
+  ],
+  "metadata": {
+    "plugin_type": "web.api_workflow_plugins",
     "category": "web"
+  }
 }

backend/autometabuilder/workflow/plugins/web/web_register_routes/package.json

@@ -1,8 +1,18 @@
 {
-  "name": "web.register_routes",
+  "name": "@autometabuilder/web_register_routes",
   "version": "1.0.0",
   "description": "Register Flask routes from JSON configuration",
   "main": "web_register_routes.py",
   "author": "AutoMetabuilder",
+  "license": "MIT",
+  "keywords": [
+    "web",
+    "flask",
+    "routes",
+    "plugin"
+  ],
+  "metadata": {
+    "plugin_type": "web.register_routes",
     "category": "web"
+  }
 }