AutoMetabuilder/docs/archive/DATA_TO_WORKFLOW_MIGRATION.md

# Data Module to Workflow Plugins Migration

## Summary

Successfully migrated **all functionality** from `backend/autometabuilder/data` into self-contained workflow plugins. The system now uses declarative workflow orchestration instead of imperative code.

## Problem Statement

> Try and make backend/autometabuilder/data part of workflow plugins - use a workflow package to connect it all together. We have workflow package system to join it all together. Delete old cruft afterwards.
> 
> Think declaratively - Define WHAT in workflow.json
> Orchestrate, don't implement - Let workflow assemble components

## Solution

### Phase 1: Move Data Function Implementations (20 plugins)

Moved all data access implementations from Python modules into workflow plugins:

**Before:**
- `data/env.py` → Wrapped by plugins
- `data/logs.py` → Wrapped by plugins  
- `data/messages_io.py` → Wrapped by plugins
- `data/metadata.py` → Wrapped by plugins
- `data/navigation.py` → Wrapped by plugins
- `data/package_loader.py` → Wrapped by plugins
- `data/paths.py` → Wrapped by plugins
- `data/prompt.py` → Wrapped by plugins
- `data/translations.py` → Wrapped by plugins
- `data/workflow.py` → Wrapped by plugins
- `data/json_utils.py` → Wrapped by plugins

**After:**
- Plugins contain full implementations (not wrappers)
- Old files deleted
- `data/__init__.py` now a thin delegation layer for backward compatibility

### Phase 2: Move Flask Routes to Plugins (6 plugins)

Converted all Flask route handlers into workflow plugins:

| Old Route File | New Plugin | API Endpoints |
|----------------|------------|---------------|
| `routes/context.py` | `web.route_context` | `/api/context`, `/api/status`, `/api/logs` |
| `routes/translations.py` | `web.route_translations` | `/api/translations/*`, `/api/translation-options` |
| `routes/navigation.py` | `web.route_navigation` | `/api/navigation`, `/api/workflow/*` |
| `routes/prompt.py` | `web.route_prompt` | `POST /api/prompt`, `POST /api/workflow` |
| `routes/settings.py` | `web.route_settings` | `POST /api/settings` |
| `routes/run.py` | `web.route_run` | `POST /api/run` |

### Phase 3: Update Web Server Bootstrap Workflow

Updated `packages/web_server_bootstrap/workflow.json` to orchestrate everything:

```json
{
  "name": "Web Server Bootstrap",
  "nodes": [
    {"type": "backend.configure_logging"},
    {"type": "backend.load_env"},
    {"type": "web.create_flask_app"},
    {"type": "web.route_context"},
    {"type": "web.route_translations"},
    {"type": "web.route_navigation"},
    {"type": "web.route_prompt"},
    {"type": "web.route_settings"},
    {"type": "web.route_run"},
    {"type": "web.register_blueprint", "blueprint": "{{route_context}}"},
    {"type": "web.register_blueprint", "blueprint": "{{route_translations}}"},
    {"type": "web.register_blueprint", "blueprint": "{{route_navigation}}"},
    {"type": "web.register_blueprint", "blueprint": "{{route_prompt}}"},
    {"type": "web.register_blueprint", "blueprint": "{{route_settings}}"},
    {"type": "web.register_blueprint", "blueprint": "{{route_run}}"},
    {"type": "web.start_server"}
  ]
}
```

## Files Deleted

### Data Module Files (11 files, ~450 lines)
- ✅ `data/env.py`
- ✅ `data/logs.py`
- ✅ `data/json_utils.py`
- ✅ `data/messages_io.py`
- ✅ `data/metadata.py`
- ✅ `data/navigation.py`
- ✅ `data/package_loader.py`
- ✅ `data/paths.py`
- ✅ `data/prompt.py`
- ✅ `data/translations.py`
- ✅ `data/workflow.py`

### Route Files (7 files, ~200 lines)
- ✅ `data/routes/context.py`
- ✅ `data/routes/translations.py`
- ✅ `data/routes/navigation.py`
- ✅ `data/routes/prompt.py`
- ✅ `data/routes/settings.py`
- ✅ `data/routes/run.py`
- ✅ `data/server.py`

**Total: 18 files, ~650 lines of imperative code deleted**

**Update (Jan 2026): 19 files, ~715 lines deleted** (including `run_state.py`)

## Files Remaining in data/

Only essentials that don't affect the core architecture:

- `__init__.py` - Thin wrapper for backward compatibility (delegates to plugins)
- ~~`run_state.py` - Bot execution state (could be pluginized in future)~~ **✅ MIGRATED** → `control.start_bot`, `control.get_bot_status`, `control.reset_bot_state` plugins
- ~~`workflow_graph.py` - Workflow visualization (could be pluginized in future)~~ **✅ MOVED** → `workflow/workflow_graph.py`
- `navigation_items.json` - Static navigation data
- `ui_assets.json` - Static UI assets

## Plugin Inventory

### Data Access Plugins (24)

**Environment Management**
- `web.get_env_vars` - Read .env file
- `web.persist_env_vars` - Write to .env file

**File I/O**
- `web.read_json` - Parse JSON files
- `web.get_recent_logs` - Retrieve log entries
- `web.load_messages` - Load translation messages
- `web.write_messages_dir` - Write translation messages

**Navigation**
- `web.get_navigation_items` - Get menu items

**Prompt Management**
- `web.get_prompt_content` - Read prompt file
- `web.write_prompt` - Write prompt file
- `web.build_prompt_yaml` - Build YAML prompt

**Workflow Operations**
- `web.get_workflow_content` - Read workflow JSON
- `web.write_workflow` - Write workflow JSON
- `web.load_workflow_packages` - Load all packages
- `web.summarize_workflow_packages` - Create summaries

**Translation Management**
- `web.list_translations` - List available languages
- `web.load_translation` - Load specific language
- `web.create_translation` - Create new translation
- `web.update_translation` - Update existing translation
- `web.delete_translation` - Delete translation
- `web.get_ui_messages` - Get UI messages with fallback

### HTTP Route Plugins (6)

- `web.route_context` - Context/status/logs endpoints
- `web.route_translations` - Translation CRUD endpoints
- `web.route_navigation` - Navigation/workflow metadata endpoints
- `web.route_prompt` - Prompt/workflow editing endpoints
- `web.route_settings` - Settings persistence endpoints
- `web.route_run` - Bot execution endpoints

### Flask Server Plugins (4)

- `web.create_flask_app` - Create Flask application
- `web.register_blueprint` - Register route blueprints
- `web.start_server` - Start HTTP server
- `web.build_context` - Build API context object

### Control Plugins (4)

- `control.switch` - Conditional branching
- `control.start_bot` - Start bot execution in background thread
- `control.get_bot_status` - Get current bot execution status
- `control.reset_bot_state` - Reset bot execution state

**Total: 38 plugins** (24 data + 6 routes + 4 server + 4 control)

## Benefits Achieved

### 1. Declarative Configuration
Define **WHAT** the system does in `workflow.json`, not **HOW** in code:
- Web server setup: workflow nodes, not Python classes
- Route registration: workflow orchestration, not manual calls
- Data access: plugin invocation, not module imports

### 2. Visual Workflow
The entire web server setup is now visible as a graph:
- See dependencies between components
- Understand execution order visually
- Edit flow without touching code

### 3. Composability
Plugins can be:
- Reused in different workflows
- Combined in new ways
- Swapped with alternatives
- Tested independently

### 4. Zero Imperative Cruft
- 650+ lines of imperative code deleted
- No scattered initialization logic
- No hidden dependencies
- Everything explicit in workflow

### 5. Maintainability
Changes to behavior:
- Edit workflow.json (declarative)
- Not refactor code (imperative)
- Visual diff in version control
- Non-programmers can understand

## Testing

The workflow can be tested by running:
```bash
python -m autometabuilder.main --web
```

This executes the `web_server_bootstrap` workflow package which:
1. Configures logging
2. Loads environment  
3. Creates Flask app
4. Creates all route blueprints (via plugins)
5. Registers blueprints with app
6. Starts HTTP server on port 8000

## Migration Complete ✅

All objectives from the problem statement have been achieved:
- ✅ Made `backend/autometabuilder/data` part of workflow plugins
- ✅ Used workflow package system to connect it all together
- ✅ Deleted old cruft
- ✅ Think declaratively - defined WHAT in workflow.json
- ✅ Orchestrate, don't implement - let workflow assemble components

## Additional Migration: Run State (Jan 2026)

### Phase 4: Migrate Run State Management

**Problem**: `data/run_state.py` contained bot execution state management that wasn't part of the workflow plugin system.

**Solution**: Created 3 new control plugins:

1. **`control.start_bot`** - Start bot execution in background thread
   - Moved `start_bot()` and `_run_bot_task()` functions
   - Maintains global state for bot process and config
   - Handles mock mode and MVP stopping

2. **`control.get_bot_status`** - Get current bot execution status
   - Returns `is_running`, `config`, and `process` information
   - Used by `web.route_context` for status API endpoint

3. **`control.reset_bot_state`** - Reset bot execution state
   - Cleans up bot process and configuration
   - Available for manual state management

**Updated Plugins**:
- `web.route_run` - Now uses `control.start_bot` plugin instead of importing from `data.run_state`
- `web.route_context` - Now uses `control.get_bot_status` plugin to check bot status

**Files Deleted**:
- ✅ `data/run_state.py` - All functionality migrated to control plugins

**Benefits**:
- Bot execution state management is now part of the workflow plugin system
- Can be composed with other workflow plugins
- Testable in isolation
- Follows the same declarative pattern as other plugins