metabuilder/frontends/codegen/PERSISTENCE_MIDDLEWARE_DOCS.md

Redux Persistence Middleware Documentation

Overview

The Redux Persistence Middleware system provides automatic synchronization between Redux state, IndexedDB (local storage), and Flask API (remote storage). This system ensures data consistency, provides offline-first capabilities, and enables seamless sync operations.

Architecture

┌─────────────────┐
│  Redux Actions  │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────┐
│  Persistence Middleware     │
│  • Debouncing (300ms)       │
│  • Batch Operations         │
│  • Queue Management         │
└────────┬────────────────────┘
         │
         ├──────────────────┬──────────────────┐
         ▼                  ▼                  ▼
┌──────────────┐    ┌──────────────┐   ┌──────────────┐
│  IndexedDB   │    │  Flask API   │   │   Metrics    │
│  (Local)     │    │  (Remote)    │   │  Tracking    │
└──────────────┘    └──────────────┘   └──────────────┘

Middleware Components

1. Persistence Middleware

Purpose: Automatically persists Redux state changes to IndexedDB and optionally syncs to Flask API.

Features:

• Automatic persistence with configurable debouncing (default: 300ms)
• Batch operations for improved performance
• Per-slice configuration
• Support for both PUT and DELETE operations
• Queue management to prevent overwhelming storage systems

Configuration:

const config = {
  storeName: 'files',      // IndexedDB store name
  enabled: true,           // Enable/disable persistence
  syncToFlask: true,       // Sync to Flask API
  debounceMs: 300,         // Debounce time in milliseconds
  batchSize: 10,           // Max operations per batch
}

Supported Actions:

• addItem, updateItem, removeItem
• saveFile, saveModel, saveComponent, etc.
• setFiles, setModels, setComponents, etc.
• deleteFile, deleteModel, deleteComponent, etc.

2. Sync Monitor Middleware

Purpose: Tracks metrics and performance of sync operations.

Metrics Tracked:

• Total operations count
• Successful operations count
• Failed operations count
• Last operation timestamp
• Average operation duration
• Operation time history (last 100 operations)

Usage:

import { getSyncMetrics, subscribeSyncMetrics } from '@/store/middleware'

// Get current metrics
const metrics = getSyncMetrics()

// Subscribe to metric updates
const unsubscribe = subscribeSyncMetrics((newMetrics) => {
  console.log('Sync metrics updated:', newMetrics)
})

3. Auto-Sync Middleware

Purpose: Provides automatic periodic synchronization with Flask API.

Features:

• Configurable sync interval (default: 30 seconds)
• Optional sync-on-change mode
• Change tracking with configurable queue size
• Automatic connection health checks
• Manual sync trigger support

Configuration:

import { configureAutoSync } from '@/store/middleware'

configureAutoSync({
  enabled: true,           // Enable auto-sync
  intervalMs: 30000,       // Sync every 30 seconds
  syncOnChange: true,      // Sync immediately when changes detected
  maxQueueSize: 50,        // Max changes before forcing sync
})

React Hooks

usePersistence()

A comprehensive hook for interacting with the persistence system.

Returns:

{
  status: {
    enabled: boolean
    lastSyncTime: number | null
    syncStatus: 'idle' | 'syncing' | 'success' | 'error'
    error: string | null
    flaskConnected: boolean
  },
  metrics: {
    totalOperations: number
    successfulOperations: number
    failedOperations: number
    lastOperationTime: number
    averageOperationTime: number
  },
  autoSyncStatus: {
    enabled: boolean
    lastSyncTime: number
    changeCounter: number
    nextSyncIn: number | null
  },
  flush: () => Promise<void>
  configure: (sliceName: string, config: any) => void
  enable: (sliceName: string) => void
  disable: (sliceName: string) => void
  resetMetrics: () => void
  configureAutoSync: (config: any) => void
  syncNow: () => Promise<void>
}

Example:

import { usePersistence } from '@/hooks/use-persistence'

function MyComponent() {
  const { status, metrics, syncNow } = usePersistence()

  return (
    <div>
      <p>Status: {status.syncStatus}</p>
      <p>Success Rate: {(metrics.successfulOperations / metrics.totalOperations * 100)}%</p>
      <button onClick={syncNow}>Sync Now</button>
    </div>
  )
}

Persistence Dashboard

A visual dashboard component for monitoring and controlling the persistence system.

Features:

• Real-time connection status
• Sync metrics visualization
• Auto-sync configuration
• Manual sync operations
• Error reporting

Usage:

import { PersistenceDashboard } from '@/components/PersistenceDashboard'

function App() {
  return <PersistenceDashboard />
}

API Reference

Persistence Control

// Flush all pending operations immediately
await flushPersistence()

// Configure a specific slice
configurePersistence('files', {
  debounceMs: 500,
  syncToFlask: false,
})

// Enable/disable persistence for a slice
enablePersistence('models')
disablePersistence('theme')

Sync Operations

import { syncToFlaskBulk, syncFromFlaskBulk } from '@/store/slices/syncSlice'

// Push all data to Flask
dispatch(syncToFlaskBulk())

// Pull all data from Flask
dispatch(syncFromFlaskBulk())

// Check Flask connection
dispatch(checkFlaskConnection())

Metrics Management

import { getSyncMetrics, resetSyncMetrics } from '@/store/middleware'

// Get current metrics
const metrics = getSyncMetrics()

// Reset all metrics
resetSyncMetrics()

Data Flow Examples

Create Operation

User creates file
  → Redux action dispatched (files/addItem)
  → Persistence middleware intercepts action
  → Item added to persistence queue with 300ms debounce
  → After debounce, item saved to IndexedDB
  → If syncToFlask enabled, item synced to Flask API
  → Sync monitor tracks operation
  → Metrics updated

Update Operation

User edits file
  → Redux action dispatched (files/updateItem)
  → Persistence middleware intercepts
  → Existing debounce timer cleared
  → New debounce timer started (300ms)
  → After debounce, updated item saved to IndexedDB
  → Item synced to Flask API
  → Auto-sync change counter incremented

Auto-Sync Operation

Auto-sync timer fires (every 30s)
  → Check if changes pending
  → If changes > 0, trigger bulk sync
  → Collect all items from IndexedDB
  → Send bulk request to Flask API
  → Update last sync timestamp
  → Reset change counter
  → Update connection status

Performance Considerations

Debouncing

• Default 300ms debounce prevents excessive writes
• Each new action resets the timer for that item
• Independent timers per item (files:123, models:456)

Batching

• Operations batched for IndexedDB efficiency
• Maximum 10 operations per batch by default
• Failed operations don't block successful ones

Queue Management

• FIFO queue ensures order
• Latest operation for a key overwrites previous
• Automatic flush on component unmount

Error Handling

Connection Failures

• Automatically detected via health checks
• System gracefully degrades to local-only mode
• Queued operations retained for next successful connection

Operation Failures

• Failed operations logged to console
• Metrics track failure count
• Error messages displayed in dashboard
• Toast notifications for user feedback

Data Conflicts

• Detected during sync operations
• Timestamp comparison used for detection
• Manual resolution via conflict resolution UI
• Multiple resolution strategies available

Best Practices

1. Enable Auto-Sync for Production

configureAutoSync({
  enabled: true,
  intervalMs: 30000,
  syncOnChange: true,
})

2. Monitor Metrics Regularly

useEffect(() => {
  const unsubscribe = subscribeSyncMetrics((metrics) => {
    if (metrics.failedOperations > 10) {
      console.warn('High failure rate detected')
    }
  })
  return unsubscribe
}, [])

3. Flush on Critical Operations

async function saveProject() {
  // ... save project data ...
  await flushPersistence()
  // Ensure all data is persisted before continuing
}

4. Configure Per-Slice Settings

// Disable Flask sync for local-only data
configurePersistence('settings', {
  syncToFlask: false,
})

// Increase debounce for frequently updated data
configurePersistence('theme', {
  debounceMs: 1000,
})

Troubleshooting

Problem: Data not syncing to Flask

Solution: Check Flask connection status in dashboard. Verify Flask API is running and accessible.

Problem: High failure rate

Solution: Check network connection. Verify IndexedDB quota not exceeded. Check console for detailed error messages.

Problem: Slow performance

Solution: Increase debounce time. Reduce sync frequency. Disable Flask sync for non-critical data.

Problem: Auto-sync not working

Solution: Verify auto-sync is enabled in settings. Check that Flask is connected. Verify change counter is incrementing.

Future Enhancements

• Optimistic updates with rollback
• Selective sync by data type
• Compression for large payloads
• Differential sync (only changed fields)
• WebSocket real-time sync
• Multi-device sync with conflict resolution
• Offline queue with retry logic
• Sync progress indicators