AutoMetabuilder/E2E_TESTING.md

End-to-End Testing for AutoMetabuilder Backend

This document explains how to run and understand the E2E tests for the AutoMetabuilder backend after the migration to workflows.

Overview

The E2E tests verify that the backend API works correctly after the major migration to workflow-based architecture. These tests use Flask's test client to verify API endpoints without needing to start an actual server.

Test File

Location: backend/tests/test_backend_e2e.py

Running the Tests

Run All E2E Tests

PYTHONPATH=backend pytest backend/tests/test_backend_e2e.py -v

Run Specific Test Class

# Test workflow endpoints only
PYTHONPATH=backend pytest backend/tests/test_backend_e2e.py::TestWorkflowEndpoints -v

# Test navigation and translation endpoints
PYTHONPATH=backend pytest backend/tests/test_backend_e2e.py::TestNavigationAndTranslation -v

Run Single Test

PYTHONPATH=backend pytest backend/tests/test_backend_e2e.py::TestWorkflowEndpoints::test_workflow_graph -v

Test Coverage

TestWorkflowEndpoints

Tests workflow-related API endpoints:

• test_workflow_graph - GET /api/workflow/graph
• test_workflow_plugins - GET /api/workflow/plugins
• test_workflow_packages - GET /api/workflow/packages

TestNavigationAndTranslation

Tests navigation and i18n endpoints:

• test_navigation - GET /api/navigation
• test_translation_options - GET /api/translation-options

TestBasicFunctionality

Basic functionality tests:

• test_json_response_format - Verifies JSON response format

What Makes These Tests E2E

These tests verify the complete workflow system from end to end:

1. Workflow Package Loading - Tests load the web_server_json_routes workflow package
2. Workflow Execution - Executes the complete workflow to build the Flask app
3. Route Registration - Routes are registered via the web.register_routes plugin
4. API Handler Plugins - Each route calls a specific plugin handler
5. Data Layer - Plugins use the data access layer
6. Response Validation - Full request/response cycle is tested

This validates the entire architecture works together.

Key Features Tested

JSON-Based Route Definitions

Routes are defined declaratively in workflow JSON:

{
  "type": "web.register_routes",
  "parameters": {
    "routes": [
      {
        "path": "/api/navigation",
        "handler": "web.api_navigation"
      }
    ]
  }
}

Automatic Plugin Discovery

Plugins are discovered automatically by scanning package.json files:

• No manual plugin map maintenance
• 135+ plugins discovered automatically
• Plugins can be added without registration

Workflow-Based Server

The Flask server is built through workflow execution:

• Logging configuration
• Environment loading
• App creation
• Route registration
• All configured via JSON workflow

Expected Output

Successful Run

============================= test session starts ==============================
...
backend/tests/test_backend_e2e.py::TestWorkflowEndpoints::test_workflow_graph PASSED
backend/tests/test_backend_e2e.py::TestWorkflowEndpoints::test_workflow_plugins PASSED
backend/tests/test_backend_e2e.py::TestWorkflowEndpoints::test_workflow_packages PASSED
backend/tests/test_backend_e2e.py::TestNavigationAndTranslation::test_navigation PASSED
backend/tests/test_backend_e2e.py::TestNavigationAndTranslation::test_translation_options PASSED
backend/tests/test_backend_e2e.py::TestBasicFunctionality::test_json_response_format PASSED
============================== 6 passed in 1.27s ===============================

Test Failures

If tests fail, check:

1. Plugin errors - Some plugins may fail to load (this is expected, they're logged as warnings)
2. Missing files - metadata.json or other files may not exist (tests handle this gracefully)
3. Import errors - Ensure PYTHONPATH is set correctly

Common Issues

Plugin Registration Warnings

You may see warnings like:

ERROR Failed to register plugin utils.map_list: No module named 'value_helpers'

These are expected and don't affect the tests. These plugins have import issues but aren't needed for the web server functionality.

Metadata Not Found

Some endpoints may return 500 if metadata.json doesn't exist. Tests handle this gracefully as these files are optional.

Dependencies

The tests require:

pip install pytest flask requests pyyaml python-dotenv

Or use the full project dependencies:

pip install -r requirements.txt  # if exists
# or
pip install pytest flask PyGithub openai python-dotenv tenacity slack-sdk discord.py

Test Architecture

Fixtures

flask_app fixture:

• Loads web_server_json_routes workflow package
• Removes start_server node to prevent blocking
• Executes workflow to build Flask app
• Returns configured Flask app

client fixture:

• Creates Flask test client
• Used to make test requests
• No actual server needed

Workflow Used

The tests use the web_server_json_routes workflow package, which demonstrates:

• JSON-based route definitions
• Plugin-based request handlers
• Workflow-driven server configuration

Location: backend/autometabuilder/packages/web_server_json_routes/

Comparison with Other Tests

vs test_ajax_contracts.py

• test_ajax_contracts.py: Uses old route structure with Python blueprints
• test_backend_e2e.py: Uses new JSON route structure

vs Integration Tests

• Integration tests focus on individual plugins
• E2E tests verify the complete workflow system

Continuous Integration

These tests should be run as part of CI/CD:

# Example GitHub Actions
- name: Run E2E Tests
  run: |
    PYTHONPATH=backend pytest backend/tests/test_backend_e2e.py -v

Future Enhancements

Potential additions to E2E tests:

• Test POST/PUT/DELETE endpoints
• Test error handling and validation
• Test authentication/authorization
• Test with real database
• Performance/load testing
• Test all workflow packages

Related Documentation

• PACKAGE_JSON_GUIDE.md - Understanding package.json files
• MIGRATION_SUMMARY.md - Details of the workflow migration
• backend/tests/README.md - Overview of all tests

Questions?

If tests fail unexpectedly:

1. Check the test output for specific error messages
2. Verify PYTHONPATH is set: PYTHONPATH=backend
3. Ensure dependencies are installed
4. Check that workflow packages exist: ls backend/autometabuilder/packages/
5. Verify plugins can be discovered: PYTHONPATH=backend python3 -c "from autometabuilder.workflow.plugin_registry import scan_plugins; print(len(scan_plugins()))"

The E2E tests confirm that the backend works correctly after the major migration to workflows!