metabuilder/codegen/docs/STORAGE_MIGRATION.md

Storage Migration Guide

Overview

CodeForge has migrated from Spark KV-based storage to IndexedDB as the primary local database solution. This provides:

• ✅ Better Performance: Structured queries with indexes
• ✅ More Storage: No 10MB limit like LocalStorage
• ✅ Structured Data: Organized collections instead of flat key-value
• ✅ Offline First: Robust offline capabilities
• ✅ Backward Compatible: Spark KV still available as fallback

Architecture

Storage Layers

┌─────────────────────────────────────┐
│     Application Layer               │
│  (React Components & Hooks)         │
└─────────────┬───────────────────────┘
              │
┌─────────────▼───────────────────────┐
│     Hybrid Storage API              │
│  • Prefers IndexedDB                │
│  • Falls back to Spark KV           │
│  • Automatic sync/migration         │
└─────────────┬───────────────────────┘
              │
      ┌───────┴────────┐
      │                │
┌─────▼──────┐  ┌─────▼──────┐
│ IndexedDB  │  │  Spark KV  │
│ (Primary)  │  │ (Fallback) │
└────────────┘  └────────────┘

IndexedDB Schema

{
  projects: {
    id: string,
    name: string,
    files: any[],
    models: any[],
    // ... full project data
  },
  files: {
    id: string,
    name: string,
    content: string,
    language: string,
  },
  models: {
    id: string,
    name: string,
    fields: any[],
  },
  components: {
    id: string,
    name: string,
    code: string,
  },
  workflows: {
    id: string,
    name: string,
    nodes: any[],
    edges: any[],
  },
  settings: {
    key: string,
    value: any,
  }
}

Usage

Basic Storage Hook

Replace useKV from Spark with useStorage:

// ❌ Old way (Spark KV only)
import { useKV } from '@github/spark/hooks'
const [todos, setTodos] = useKV('todos', [])

// ✅ New way (IndexedDB + Spark KV fallback)
import { useStorage } from '@/hooks/use-storage'
const [todos, setTodos] = useStorage('todos', [])

Collection-Based Storage

For structured data collections:

import { useIndexedDB } from '@/hooks/use-indexed-db'

// Single item by ID
const [project, updateProject, deleteProject, loading] = 
  useIndexedDB('projects', projectId, defaultProject)

// All items in collection
const [allProjects, refresh, loading] = 
  useIndexedDBCollection('projects')

Direct Database Access

For advanced queries:

import { db } from '@/lib/db'

// Get by ID
const project = await db.get('projects', 'proj-123')

// Get all
const allProjects = await db.getAll('projects')

// Query by index
const recentProjects = await db.query(
  'projects',
  'updatedAt',
  IDBKeyRange.lowerBound(Date.now() - 7 * 24 * 60 * 60 * 1000)
)

// Save
await db.put('projects', {
  id: 'proj-123',
  name: 'My Project',
  // ...
})

// Delete
await db.delete('projects', 'proj-123')

Migration

Automatic Migration

The hybrid storage system automatically handles migration:

1. On first read, checks IndexedDB
2. If not found, checks Spark KV
3. On write, saves to both (if enabled)

Manual Migration

Use the Storage Settings page or programmatically:

import { storage } from '@/lib/storage'

// Migrate all data from Spark KV to IndexedDB
const { migrated, failed } = await storage.migrateFromSparkKV()
console.log(`Migrated ${migrated} items, ${failed} failed`)

// Sync IndexedDB back to Spark KV (backup)
const { synced, failed } = await storage.syncToSparkKV()
console.log(`Synced ${synced} items, ${failed} failed`)

Storage Settings UI

Access via Settings → Storage Management:

• View Statistics: See item counts in each storage
• Migrate Data: One-click migration from Spark KV
• Sync to Cloud: Backup IndexedDB to Spark KV
• Clear Data: Emergency data reset

Configuration

Storage Options

import { HybridStorage } from '@/lib/storage'

// Custom configuration
const customStorage = new HybridStorage({
  useIndexedDB: true,      // Enable IndexedDB
  useSparkKV: true,        // Enable Spark KV fallback
  preferIndexedDB: true,   // Try IndexedDB first
})

Pre-configured Instances

import { 
  storage,              // Default: IndexedDB preferred, Spark KV fallback
  indexedDBOnlyStorage, // IndexedDB only
  sparkKVOnlyStorage    // Spark KV only
} from '@/lib/storage'

Best Practices

1. Use Functional Updates

Always use functional updates for concurrent-safe operations:

// ❌ Wrong - stale closure
setTodos([...todos, newTodo])

// ✅ Correct - always current
setTodos((current) => [...current, newTodo])

2. Structured Data in Collections

Store structured data in typed collections:

// ❌ Wrong - flat key-value
await storage.set('project-123', projectData)

// ✅ Correct - structured collection
await db.put('projects', {
  id: '123',
  ...projectData
})

3. Error Handling

Always handle storage errors gracefully:

try {
  await updateProject(newData)
} catch (error) {
  console.error('Failed to save project:', error)
  toast.error('Save failed. Please try again.')
}

4. Periodic Backups

Regularly sync to Spark KV for backup:

// Backup on significant changes
useEffect(() => {
  if (hasUnsavedChanges) {
    storage.syncToSparkKV().catch(console.error)
  }
}, [hasUnsavedChanges])

Performance Benefits

Before (Spark KV Only)

• 🐌 Linear search through all keys
• 🐌 No indexes or structured queries
• 🐌 Serialization overhead on every access
• ⚠️ 10MB storage limit

After (IndexedDB Primary)

• ⚡ Indexed queries (O(log n))
• ⚡ Structured collections with schemas
• ⚡ Efficient binary storage
• ✅ ~1GB+ storage (browser dependent)

Browser Support

IndexedDB is supported in all modern browsers:

• ✅ Chrome 24+
• ✅ Firefox 16+
• ✅ Safari 10+
• ✅ Edge 12+
• ✅ Mobile browsers

Spark KV automatically serves as fallback if IndexedDB is unavailable.

Troubleshooting

"Database not initialized" Error

// Ensure init is called before use
await db.init()
const data = await db.get('projects', 'proj-123')

Storage Quota Exceeded

// Check available storage
if (navigator.storage && navigator.storage.estimate) {
  const { usage, quota } = await navigator.storage.estimate()
  console.log(`Using ${usage} of ${quota} bytes`)
}

Data Migration Issues

1. Check browser console for specific errors
2. Verify Spark KV data exists: window.spark.kv.keys()
3. Clear IndexedDB and retry migration
4. Use Storage Settings UI for guided migration

Future Enhancements

• Remote Sync: Sync to cloud database
• Compression: Compress large datasets
• Encryption: Encrypt sensitive data at rest
• Import/Export: JSON export for portability
• Version Control: Track data changes over time

Summary

The migration to IndexedDB provides:

1. Better Performance: Structured queries with indexes
2. More Capacity: Gigabytes instead of megabytes
3. Backward Compatible: Spark KV still works
4. Easy Migration: One-click data transfer
5. Flexible: Use IndexedDB, Spark KV, or both

The hybrid storage system ensures your data is always accessible while providing the performance benefits of a proper database.