AutoMetabuilder/docs/archive/DATA_TO_WORKFLOW_MIGRATION.md

Data Module to Workflow Plugins Migration

Summary

Successfully migrated all functionality from backend/autometabuilder/data into self-contained workflow plugins. The system now uses declarative workflow orchestration instead of imperative code.

Problem Statement

Try and make backend/autometabuilder/data part of workflow plugins - use a workflow package to connect it all together. We have workflow package system to join it all together. Delete old cruft afterwards.

Think declaratively - Define WHAT in workflow.json Orchestrate, don't implement - Let workflow assemble components

Solution

Phase 1: Move Data Function Implementations (20 plugins)

Moved all data access implementations from Python modules into workflow plugins:

Before:

• data/env.py → Wrapped by plugins
• data/logs.py → Wrapped by plugins
• data/messages_io.py → Wrapped by plugins
• data/metadata.py → Wrapped by plugins
• data/navigation.py → Wrapped by plugins
• data/package_loader.py → Wrapped by plugins
• data/paths.py → Wrapped by plugins
• data/prompt.py → Wrapped by plugins
• data/translations.py → Wrapped by plugins
• data/workflow.py → Wrapped by plugins
• data/json_utils.py → Wrapped by plugins

After:

• Plugins contain full implementations (not wrappers)
• Old files deleted
• data/__init__.py now a thin delegation layer for backward compatibility

Phase 2: Move Flask Routes to Plugins (6 plugins)

Converted all Flask route handlers into workflow plugins:

Old Route File	New Plugin	API Endpoints
routes/context.py	web.route_context	/api/context, /api/status, /api/logs
routes/translations.py	web.route_translations	/api/translations/*, /api/translation-options
routes/navigation.py	web.route_navigation	/api/navigation, /api/workflow/*
routes/prompt.py	web.route_prompt	POST /api/prompt, POST /api/workflow
routes/settings.py	web.route_settings	POST /api/settings
routes/run.py	web.route_run	POST /api/run

Phase 3: Update Web Server Bootstrap Workflow

Updated packages/web_server_bootstrap/workflow.json to orchestrate everything:

{
  "name": "Web Server Bootstrap",
  "nodes": [
    {"type": "backend.configure_logging"},
    {"type": "backend.load_env"},
    {"type": "web.create_flask_app"},
    {"type": "web.route_context"},
    {"type": "web.route_translations"},
    {"type": "web.route_navigation"},
    {"type": "web.route_prompt"},
    {"type": "web.route_settings"},
    {"type": "web.route_run"},
    {"type": "web.register_blueprint", "blueprint": "{{route_context}}"},
    {"type": "web.register_blueprint", "blueprint": "{{route_translations}}"},
    {"type": "web.register_blueprint", "blueprint": "{{route_navigation}}"},
    {"type": "web.register_blueprint", "blueprint": "{{route_prompt}}"},
    {"type": "web.register_blueprint", "blueprint": "{{route_settings}}"},
    {"type": "web.register_blueprint", "blueprint": "{{route_run}}"},
    {"type": "web.start_server"}
  ]
}

Files Deleted

Data Module Files (11 files, ~450 lines)

• ✅ data/env.py
• ✅ data/logs.py
• ✅ data/json_utils.py
• ✅ data/messages_io.py
• ✅ data/metadata.py
• ✅ data/navigation.py
• ✅ data/package_loader.py
• ✅ data/paths.py
• ✅ data/prompt.py
• ✅ data/translations.py
• ✅ data/workflow.py

Route Files (7 files, ~200 lines)

• ✅ data/routes/context.py
• ✅ data/routes/translations.py
• ✅ data/routes/navigation.py
• ✅ data/routes/prompt.py
• ✅ data/routes/settings.py
• ✅ data/routes/run.py
• ✅ data/server.py

Total: 18 files, ~650 lines of imperative code deleted

Update (Jan 2026): 19 files, ~715 lines deleted (including run_state.py)

Files Remaining in data/

Only essentials that don't affect the core architecture:

• __init__.py - Thin wrapper for backward compatibility (delegates to plugins)
• run_state.py - Bot execution state (could be pluginized in future) ✅ MIGRATED → control.start_bot, control.get_bot_status, control.reset_bot_state plugins
• workflow_graph.py - Workflow visualization (could be pluginized in future) ✅ MOVED → workflow/workflow_graph.py
• navigation_items.json - Static navigation data
• ui_assets.json - Static UI assets

Plugin Inventory

Data Access Plugins (24)

Environment Management

• web.get_env_vars - Read .env file
• web.persist_env_vars - Write to .env file

File I/O

• web.read_json - Parse JSON files
• web.get_recent_logs - Retrieve log entries
• web.load_messages - Load translation messages
• web.write_messages_dir - Write translation messages

Navigation

• web.get_navigation_items - Get menu items

Prompt Management

• web.get_prompt_content - Read prompt file
• web.write_prompt - Write prompt file
• web.build_prompt_yaml - Build YAML prompt

Workflow Operations

• web.get_workflow_content - Read workflow JSON
• web.write_workflow - Write workflow JSON
• web.load_workflow_packages - Load all packages
• web.summarize_workflow_packages - Create summaries

Translation Management

• web.list_translations - List available languages
• web.load_translation - Load specific language
• web.create_translation - Create new translation
• web.update_translation - Update existing translation
• web.delete_translation - Delete translation
• web.get_ui_messages - Get UI messages with fallback

HTTP Route Plugins (6)

• web.route_context - Context/status/logs endpoints
• web.route_translations - Translation CRUD endpoints
• web.route_navigation - Navigation/workflow metadata endpoints
• web.route_prompt - Prompt/workflow editing endpoints
• web.route_settings - Settings persistence endpoints
• web.route_run - Bot execution endpoints

Flask Server Plugins (4)

• web.create_flask_app - Create Flask application
• web.register_blueprint - Register route blueprints
• web.start_server - Start HTTP server
• web.build_context - Build API context object

Control Plugins (4)

• control.switch - Conditional branching
• control.start_bot - Start bot execution in background thread
• control.get_bot_status - Get current bot execution status
• control.reset_bot_state - Reset bot execution state

Total: 38 plugins (24 data + 6 routes + 4 server + 4 control)

Benefits Achieved

1. Declarative Configuration

Define WHAT the system does in workflow.json, not HOW in code:

• Web server setup: workflow nodes, not Python classes
• Route registration: workflow orchestration, not manual calls
• Data access: plugin invocation, not module imports

2. Visual Workflow

The entire web server setup is now visible as a graph:

• See dependencies between components
• Understand execution order visually
• Edit flow without touching code

3. Composability

Plugins can be:

• Reused in different workflows
• Combined in new ways
• Swapped with alternatives
• Tested independently

4. Zero Imperative Cruft

• 650+ lines of imperative code deleted
• No scattered initialization logic
• No hidden dependencies
• Everything explicit in workflow

5. Maintainability

Changes to behavior:

• Edit workflow.json (declarative)
• Not refactor code (imperative)
• Visual diff in version control
• Non-programmers can understand

Testing

The workflow can be tested by running:

python -m autometabuilder.main --web

This executes the web_server_bootstrap workflow package which:

1. Configures logging
2. Loads environment
3. Creates Flask app
4. Creates all route blueprints (via plugins)
5. Registers blueprints with app
6. Starts HTTP server on port 8000

Migration Complete ✅

All objectives from the problem statement have been achieved:

• ✅ Made backend/autometabuilder/data part of workflow plugins
• ✅ Used workflow package system to connect it all together
• ✅ Deleted old cruft
• ✅ Think declaratively - defined WHAT in workflow.json
• ✅ Orchestrate, don't implement - let workflow assemble components

Additional Migration: Run State (Jan 2026)

Phase 4: Migrate Run State Management

Problem: data/run_state.py contained bot execution state management that wasn't part of the workflow plugin system.

Solution: Created 3 new control plugins:

1. control.start_bot - Start bot execution in background thread

   • Moved start_bot() and _run_bot_task() functions
   • Maintains global state for bot process and config
   • Handles mock mode and MVP stopping

2. control.get_bot_status - Get current bot execution status

   • Returns is_running, config, and process information
   • Used by web.route_context for status API endpoint

3. control.reset_bot_state - Reset bot execution state

   • Cleans up bot process and configuration
   • Available for manual state management

Updated Plugins:

• web.route_run - Now uses control.start_bot plugin instead of importing from data.run_state
• web.route_context - Now uses control.get_bot_status plugin to check bot status

Files Deleted:

• ✅ data/run_state.py - All functionality migrated to control plugins

Benefits:

• Bot execution state management is now part of the workflow plugin system
• Can be composed with other workflow plugins
• Testable in isolation
• Follows the same declarative pattern as other plugins