diff --git a/.github/COPILOT_ANALYSIS.md b/.github/COPILOT_ANALYSIS.md index 7c19a3ac2..a8b3376e4 100644 --- a/.github/COPILOT_ANALYSIS.md +++ b/.github/COPILOT_ANALYSIS.md @@ -8,7 +8,7 @@ ### Analysis Approach 1. **Examined existing instructions** - - `dbal/AGENTS.md` (605 lines) - DBAL-specific agent development guide + - `dbal/docs/AGENTS.md` (605 lines) - DBAL-specific agent development guide - `.github/copilot-instructions.md` (existing) - Original generic guidance 2. **Analyzed codebase patterns** through: @@ -116,7 +116,7 @@ Instructions now reference: | File | Purpose | Why Referenced | |------|---------|-----------------| -| `dbal/AGENTS.md` | DBAL development guide | Critical for DBAL changes | +| `dbal/docs/AGENTS.md` | DBAL development guide | Critical for DBAL changes | | `src/lib/database.ts` | Database operations | 1200+ LOC utility wrapper, required for all DB access | | `src/components/RenderComponent.tsx` | Generic renderer | 221 LOC example of declarative UI pattern | | `src/lib/schema-utils.test.ts` | Test examples | 63 tests showing parameterized pattern | @@ -159,7 +159,7 @@ Instructions now reference: ### Adding a new database entity 1. Read: API-First DBAL Development pattern 2. Check: DBAL-Specific Guidance (YAML → Types → Adapters) -3. Reference: `dbal/AGENTS.md` for detailed workflow +3. Reference: `dbal/docs/AGENTS.md` for detailed workflow ### Creating a new component feature 1. Read: Generic Component Rendering pattern @@ -192,7 +192,7 @@ Agents should prioritize these when onboarding: 1. **Start**: `docs/architecture/5-level-system.md` (understand permissions) 2. **Then**: `docs/architecture/packages.md` (understand modularity) 3. **Then**: `src/lib/database.ts` (understand DB pattern) -4. **Then**: `dbal/AGENTS.md` (if working on DBAL) +4. **Then**: `dbal/docs/AGENTS.md` (if working on DBAL) 5. **Always**: `FUNCTION_TEST_COVERAGE.md` (for test requirements) --- diff --git a/.github/TEMPLATES.md b/.github/TEMPLATES.md index 10302ae0f..e50ce893b 100644 --- a/.github/TEMPLATES.md +++ b/.github/TEMPLATES.md @@ -338,6 +338,6 @@ Please submit an issue with the "documentation" template to suggest improvements - **Workflow Guide**: `.github/prompts/0-kickstart.md` - **Contributing**: `README.md` → Contributing section - **Architecture**: `docs/architecture/` -- **DBAL Guide**: `dbal/AGENTS.md` +- **DBAL Guide**: `dbal/docs/AGENTS.md` - **UI Standards**: `UI_STANDARDS.md` - **Copilot Instructions**: `.github/copilot-instructions.md` diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 2f5f0cb69..195e1f833 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -217,7 +217,7 @@ If fixing a DBAL bug: - **Database**: `src/lib/database.ts` (all DB operations), `prisma/schema.prisma` (schema) - **Packages**: `src/lib/package-loader.ts` (initialization), `packages/*/seed/` (definitions) - **Tests**: `src/lib/schema-utils.test.ts` (parameterized pattern), `FUNCTION_TEST_COVERAGE.md` (auto-generated report) -- **DBAL**: `dbal/AGENTS.md` (detailed DBAL agent guide), `api/schema/` (YAML contracts) +- **DBAL**: `dbal/docs/AGENTS.md` (detailed DBAL agent guide), `api/schema/` (YAML contracts) ## Questions to Ask diff --git a/.github/prompts/workflow/0-kickstart.md b/.github/prompts/workflow/0-kickstart.md index 649c6b246..b33daa973 100644 --- a/.github/prompts/workflow/0-kickstart.md +++ b/.github/prompts/workflow/0-kickstart.md @@ -4,7 +4,7 @@ Use this as the default workflow when starting work in this repo. ## Workflow 1. Skim `docs/START_HERE.md` (if new), `docs/INDEX.md`, and relevant items in `docs/todo/`. -2. Check for scoped rules in nested `AGENTS.md` files (e.g. `dbal/AGENTS.md`) before editing those areas. +2. Check for scoped rules in nested `AGENTS.md` files (e.g. `dbal/docs/AGENTS.md`) before editing those areas. 3. Use the prompts in `.github/prompts/` as needed: - Plan: `1-plan-feature.prompt.md` - Design: `2-design-component.prompt.md` diff --git a/AGENTS.md b/AGENTS.md index 4ff9f6756..728e5e354 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -45,5 +45,5 @@ DBAL library workflows live in `dbal/development/` (`npm run build`, `npm run te ## Agent-Specific Notes -- Check for scoped rules in nested `AGENTS.md` files (e.g., `dbal/AGENTS.md`) before editing those areas. +- Check for scoped rules in nested `AGENTS.md` files (e.g., `dbal/docs/AGENTS.md`) before editing those areas. - Keep changes focused, avoid dependency churn, and follow existing patterns/config in `config/` and `frontends/nextjs/`. diff --git a/dbal/README.md b/dbal/README.md index 129ff36c1..446bd3af8 100644 --- a/dbal/README.md +++ b/dbal/README.md @@ -1,437 +1,46 @@ -# Database Abstraction Layer (DBAL) +# DBAL - Database Abstraction Layer -A language-agnostic database abstraction layer that provides a secure interface between client applications and database backends. The DBAL uses TypeScript for rapid development and testing, with a C++ production layer for enhanced security and performance. +A language-agnostic database abstraction layer that provides a secure interface between client applications and database backends. -## Architecture Overview - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ Client Application (Spark) │ -│ (TypeScript/React) │ -└────────────────────────────────┬────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ DBAL Client │ -│ (TypeScript Dev / C++ Production) │ -│ ┌────────────────────┬──────────────────┬────────────────────┐ │ -│ │ Query Builder │ Validation │ Error Handling │ │ -│ └────────────────────┴──────────────────┴────────────────────┘ │ -└────────────────────────────────┬────────────────────────────────┘ - │ - ┌────────────┴────────────┐ - │ IPC/RPC Bridge │ - │ (gRPC/WebSocket) │ - └────────────┬────────────┘ - │ -┌─────────────────────────────────────────────────────────────────┐ -│ DBAL Daemon (C++) │ -│ [Production Only - Sandboxed] │ -│ ┌────────────────────┬──────────────────┬────────────────────┐ │ -│ │ Auth/ACL │ Query Executor │ Connection Pool │ │ -│ └────────────────────┴──────────────────┴────────────────────┘ │ -└────────────────────────────────┬────────────────────────────────┘ - │ - ┌────────────┴────────────┐ - │ │ - ▼ ▼ - ┌────────────────┐ ┌────────────────┐ - │ Prisma Client │ │ SQLite Direct │ - │ (Server-side) │ │ (Embedded) │ - └────────────────┘ └────────────────┘ - │ │ - ▼ ▼ - ┌────────────────┐ ┌────────────────┐ - │ PostgreSQL │ │ SQLite DB │ - │ MySQL │ │ │ - │ SQL Server │ │ │ - └────────────────┘ └────────────────┘ -``` - -## Supported Databases - -The Prisma adapter behind DBAL already targets the databases you care about: PostgreSQL, MySQL, SQLite, and any other engine Prisma supports (SQL Server, CockroachDB, MongoDB, etc.). Switch between them by pointing `DATABASE_URL` at the desired backend and regenerating the Prisma client for your schema. - -The TypeScript client exposes three Prisma-based adapters: `PrismaAdapter`, `PostgresAdapter`, and `MySQLAdapter`. Setting `config.adapter` to `'postgres'` or `'mysql'` constructs the dialect-specific adapter, which keeps the shared Prisma logic but tweaks the capabilities metadata (e.g., enabling full-text search where supported) and leaves the rest of the stack focused on validation, ACLs, and audit logging. - -```bash -# PostgreSQL -export DATABASE_URL="postgresql://user:pass@db:5432/metabuilder" - -# MySQL -export DATABASE_URL="mysql://user:pass@db:3306/metabuilder" - -npx prisma generate -``` - -With `config.adapter = 'prisma'`, DBAL sends every request through `PrismaAdapter`, and Prisma handles dialect differences, migrations, and connection pooling defined in `prisma/schema.prisma` and `prisma/migrations/`. That keeps DBAL focused on validation, ACLs, and audit logging while it can still drive PostgreSQL, MySQL, or any other Prisma-supported store. - -The C++ daemon still resides in Phase 3—the current implementation is backed by the in-memory store described in `dbal/production/docs/PHASE3_DAEMON.md`, so Postgres/MySQL adapters for the daemon are still future work. - -### Native Prisma bridge - -The Phase 3 daemon can still leverage Prisma without bundling Node by calling `NativePrismaAdapter`. Each SQL plan is serialized as a JSON payload with the `$n` or `?` placeholders plus parameters and sent to `/api/native-prisma` on the Next.js server. The API route validates `DBAL_NATIVE_PRISMA_TOKEN`, reconstructs a `Prisma.sql` template, executes the query through the shared Prisma client, and returns rows or affected counts so the daemon sees the same `SqlRow`/`int` values as a regular SQL adapter. Set the same `DBAL_NATIVE_PRISMA_TOKEN` (mirrored in `frontends/nextjs/.env.example`) when running the daemon so the bridge rejects unauthorized callers. - -## Design Principles - -1. **Language Agnostic**: API contracts defined in YAML/Proto, not tied to any language -2. **Security First**: C++ daemon sandboxes all database access with ACL enforcement -3. **Development Speed**: TypeScript implementation for rapid iteration -4. **Zero Trust**: User code never touches database credentials or raw connections -5. **Capability-based**: Adapters declare what they support (transactions, joins, TTL, etc.) -6. **Testable**: Shared test vectors ensure both implementations behave identically - -## Repository Structure +## Structure ``` dbal/ -├── api/ # Language-agnostic contracts (source of truth) -│ ├── schema/ # Entity and operation definitions -│ ├── idl/ # Optional: Proto/FlatBuffers schemas -│ └── versioning/ # Compatibility rules -├── common/ # Shared test vectors and fixtures -├── ts/ # TypeScript implementation (development) -├── cpp/ # C++ implementation (production) -├── backends/ # Backend-specific assets -├── tools/ # Code generation and build tools -└── scripts/ # Cross-platform build scripts +├── development/ # TypeScript implementation (fast iteration) +├── production/ # C++ implementation (security & performance) +├── shared/ # Shared resources (API specs, tools, etc.) +└── docs/ # Documentation ``` -## Quick Start +## Quick Links -### Development Mode (TypeScript) +- 📖 **[Full Documentation](docs/README.md)** - Complete project documentation +- 🚀 **[Quick Start](shared/docs/QUICK_START.md)** - Get started in 5 minutes +- 🏗️ **[Architecture](docs/PROJECT.md)** - System architecture and design +- 🤖 **[Agent Guide](docs/AGENTS.md)** - AI development guidelines +- 📋 **[Restructure Info](docs/RESTRUCTURE_SUMMARY.md)** - Recent organizational changes +## Development + +### TypeScript (Development) ```bash -cd dbal/development +cd development npm install npm run build npm test ``` -### Production Mode (C++ Daemon) - +### C++ (Production) ```bash -cd dbal/production -mkdir build && cd build -cmake .. -make -./dbal_daemon --config=../config/prod.yaml +cd production +# See production/docs/ for C++ build instructions ``` -### GitHub Spark Integration - -For GitHub Spark deployments, the DBAL daemon runs as a sidecar service: - -```yaml -# In your Spark deployment config -services: - dbal: - image: your-org/dbal-daemon:latest - ports: - - "50051:50051" # gRPC endpoint - environment: - - DBAL_MODE=production - - DBAL_SANDBOX=strict -``` - -## Monitoring & Daemon UI - -`frontends/dbal` is a dedicated Next.js mini-app that showcases the C++ daemon's architecture, deployment readiness, and the `ServerStatusPanel`. The main `frontends/nextjs` app re-exports the `@dbal-ui` component at `/dbal-daemon`, and the panel polls `/api/status` (the shared feed lives in `frontends/dbal/src/status.ts`). Keep this page covered with `frontends/nextjs/e2e/dbal-daemon/daemon.spec.ts` and `playwright.dbal-daemon.config.ts`, or run `npm run test:e2e:dbal-daemon` after touching the UI. - -## Security Model - -### Sandboxing Strategy - -1. **Process Isolation**: Daemon runs in separate process with restricted permissions -2. **Capability-based Security**: Each request checked against user ACL -3. **Query Validation**: All queries parsed and validated before execution -4. **Credential Protection**: DB credentials never exposed to client code -5. **Audit Logging**: All operations logged for security review - -### ACL System - -```yaml -user: "user_123" -role: "editor" -permissions: - - entity: "posts" - operations: [create, read, update] - filters: - author_id: "$user.id" # Row-level security - - entity: "comments" - operations: [create, read] -``` - -## API Contract Example - -### HTTP Utilities - -For outbound integrations the daemon can use the new requests-inspired helper `runtime::RequestsClient`. It wraps the `cpr` HTTP helpers, exposes `get`/`post` helpers, parses JSON responses, and throws clean timeouts so code paths stay predictable. - -Native Prisma calls route through `NativePrismaAdapter`, which currently POSTs to the `/api/native-prisma` Next.js API and returns the raw JSON rows or affected count using that helper. When the daemon calls `runQuery`/`runNonQuery`, the response is mapped back into `SqlRow` results so the rest of the stack stays unaware of the HTTP transport. - -```cpp -using namespace dbal::runtime; - -RequestsClient http("https://api.prisma.example"); -auto response = http.post("/rpc/execute", jsonPayload.dump(), {{"Authorization", "Bearer ..."}}); -if (response.statusCode == 200) { - const auto result = response.json["result"]; - // handle Prisma response -} -``` - -### Entity Definition (YAML) - -```yaml -# api/schema/entities/post.yaml -entity: Post -version: "1.0" -fields: - id: - type: uuid - primary: true - generated: true - title: - type: string - required: true - max_length: 200 - content: - type: text - required: true - author_id: - type: uuid - required: true - foreign_key: - entity: User - field: id - created_at: - type: datetime - generated: true - updated_at: - type: datetime - auto_update: true -``` - -### Operations (YAML) - -```yaml -# api/schema/operations/post.ops.yaml -operations: - create: - input: [title, content, author_id] - output: Post - acl_required: ["post:create"] - - read: - input: [id] - output: Post - acl_required: ["post:read"] - - update: - input: [id, title?, content?] - output: Post - acl_required: ["post:update"] - row_level_check: "author_id = $user.id" - - delete: - input: [id] - output: boolean - acl_required: ["post:delete"] - row_level_check: "author_id = $user.id OR $user.role = 'admin'" - - list: - input: [filter?, sort?, page?, limit?] - output: Post[] - acl_required: ["post:read"] -``` - -## Client Usage - -### TypeScript Client - -```typescript -import { DBALClient } from '@metabuilder/dbal' - -const client = new DBALClient({ - mode: 'development', // or 'production' - endpoint: 'localhost:50051', - auth: { - user: currentUser, - session: currentSession - } -}) - -// CRUD operations -const post = await client.posts.create({ - title: 'Hello World', - content: 'This is my first post', - author_id: user.id -}) - -const posts = await client.posts.list({ - filter: { author_id: user.id }, - sort: { created_at: 'desc' }, - limit: 10 -}) - -const updated = await client.posts.update(post.id, { - title: 'Updated Title' -}) - -await client.posts.delete(post.id) -``` - -## Development Workflow - -1. **Define Schema**: Edit YAML files in `api/schema/` -2. **Generate Code**: `python tools/codegen/gen_types.py` -3. **Implement Adapter**: Add backend support in `ts/src/adapters/` -4. **Write Tests**: Create conformance tests in `common/fixtures/` -5. **Run Tests**: `npm run test:conformance` -6. **Build C++ Daemon**: `cd cpp && cmake --build build` -7. **Deploy**: Use Docker/Kubernetes to deploy daemon - -## Testing - -### Conformance Testing - -The DBAL includes comprehensive conformance tests that ensure both TypeScript and C++ implementations behave identically: - -```bash -# Run all conformance tests -python tools/conformance/run_all.py - -# Run TS tests only -cd ts && npm run test:conformance - -# Run C++ tests only -cd cpp && ./build/tests/conformance_tests -``` - -### Test Vectors - -Shared test vectors in `common/fixtures/` ensure consistency: - -```yaml -# common/contracts/conformance_cases.yaml -- name: "Create and read post" - operations: - - action: create - entity: Post - input: - title: "Test Post" - content: "Test content" - author_id: "user_123" - expected: - status: success - output: - id: "" - title: "Test Post" - - action: read - entity: Post - input: - id: "$prev.id" - expected: - status: success - output: - title: "Test Post" -``` - -## Migration from Current System - -### Phase 1: Development Mode (Complete) -- Use TypeScript DBAL client in development -- Direct Prisma access (no daemon) -- Validates API contract compliance - -### Phase 2: Hybrid Mode (Current Implementation) -- Complete TypeScript DBAL client with Prisma adapter -- WebSocket bridge for remote daemon communication (prepared for C++) -- ACL enforcement and audit logging in TypeScript -- Runs entirely in GitHub Spark environment -- Prepares architecture for C++ daemon migration - -### Phase 3: Full Production (Future) -- All environments use C++ daemon -- TypeScript client communicates via WebSocket/gRPC -- Maximum security and performance -- Requires infrastructure beyond GitHub Spark - -## Capabilities System - -Different backends support different features: - -```yaml -# api/schema/capabilities.yaml -adapters: - prisma: - transactions: true - joins: true - full_text_search: false - ttl: false - json_queries: true - - sqlite: - transactions: true - joins: true - full_text_search: true - ttl: false - json_queries: true - - mongodb: - transactions: true - joins: false - full_text_search: true - ttl: true - json_queries: true -``` - -Client code can check capabilities: - -```typescript -if (await client.capabilities.hasJoins()) { - // Use join query -} else { - // Fall back to multiple queries -} -``` - -## Error Handling - -Standardized errors across all implementations: - -```yaml -# api/schema/errors.yaml -errors: - NOT_FOUND: - code: 404 - message: "Entity not found" - - CONFLICT: - code: 409 - message: "Entity already exists" - - UNAUTHORIZED: - code: 401 - message: "Authentication required" - - FORBIDDEN: - code: 403 - message: "Insufficient permissions" - - VALIDATION_ERROR: - code: 422 - message: "Validation failed" - fields: - - field: string - error: string -``` - -## Contributing - -See [CONTRIBUTING.md](../docs/CONTRIBUTING.md) for development guidelines. +### Shared Resources +- **API Schemas**: `shared/api/schema/` +- **Tools**: `shared/tools/` (codegen, build assistant) +- **Scripts**: `shared/scripts/` (build, test) ## License -MIT License - see [LICENSE](LICENSE) +MIT - See [LICENSE](LICENSE) file. diff --git a/dbal/AGENTS.md b/dbal/docs/AGENTS.md similarity index 100% rename from dbal/AGENTS.md rename to dbal/docs/AGENTS.md diff --git a/dbal/BEFORE_AFTER.md b/dbal/docs/BEFORE_AFTER.md similarity index 100% rename from dbal/BEFORE_AFTER.md rename to dbal/docs/BEFORE_AFTER.md diff --git a/dbal/PROJECT.md b/dbal/docs/PROJECT.md similarity index 85% rename from dbal/PROJECT.md rename to dbal/docs/PROJECT.md index 9998f1399..cc667a81f 100644 --- a/dbal/PROJECT.md +++ b/dbal/docs/PROJECT.md @@ -4,21 +4,28 @@ This directory contains the Database Abstraction Layer for MetaBuilder. ## Quick Links -- [Main README](README.md) - Overview and architecture -- [Agent Guide](AGENTS.md) - For AI agents and automated tools -- [Spark Integration](shared/docs/SPARK_INTEGRATION.md) - GitHub Spark deployment guide -- [TypeScript Implementation](development/README.md) - TS development guide -- [C++ Implementation](production/README.md) - C++ production guide +- [Main README](../README.md) - Overview and getting started +- [Agent Guide](./AGENTS.md) - For AI agents and automated tools +- [Spark Integration](../shared/docs/SPARK_INTEGRATION.md) - GitHub Spark deployment guide +- [TypeScript Implementation](../development/README.md) - TS development guide +- [C++ Implementation](../production/README.md) - C++ production guide ## Directory Structure ``` dbal/ -├── README.md # Main documentation +├── README.md # Quick reference ├── LICENSE # MIT License -├── AGENTS.md # Agent development guide ├── .gitignore # Git ignore rules │ +├── docs/ # Documentation +│ ├── README.md # Main project documentation +│ ├── README_INDEX.md # Documentation index +│ ├── PROJECT.md # Complete project reference +│ ├── AGENTS.md # Agent development guide +│ ├── RESTRUCTURE_SUMMARY.md # Restructure details +│ └── BEFORE_AFTER.md # Before/after comparison +│ ├── development/ # TypeScript implementation (fast iteration) │ ├── src/ # Source code │ ├── tests/ # Test suites diff --git a/dbal/docs/README.md b/dbal/docs/README.md new file mode 100644 index 000000000..129ff36c1 --- /dev/null +++ b/dbal/docs/README.md @@ -0,0 +1,437 @@ +# Database Abstraction Layer (DBAL) + +A language-agnostic database abstraction layer that provides a secure interface between client applications and database backends. The DBAL uses TypeScript for rapid development and testing, with a C++ production layer for enhanced security and performance. + +## Architecture Overview + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Client Application (Spark) │ +│ (TypeScript/React) │ +└────────────────────────────────┬────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ DBAL Client │ +│ (TypeScript Dev / C++ Production) │ +│ ┌────────────────────┬──────────────────┬────────────────────┐ │ +│ │ Query Builder │ Validation │ Error Handling │ │ +│ └────────────────────┴──────────────────┴────────────────────┘ │ +└────────────────────────────────┬────────────────────────────────┘ + │ + ┌────────────┴────────────┐ + │ IPC/RPC Bridge │ + │ (gRPC/WebSocket) │ + └────────────┬────────────┘ + │ +┌─────────────────────────────────────────────────────────────────┐ +│ DBAL Daemon (C++) │ +│ [Production Only - Sandboxed] │ +│ ┌────────────────────┬──────────────────┬────────────────────┐ │ +│ │ Auth/ACL │ Query Executor │ Connection Pool │ │ +│ └────────────────────┴──────────────────┴────────────────────┘ │ +└────────────────────────────────┬────────────────────────────────┘ + │ + ┌────────────┴────────────┐ + │ │ + ▼ ▼ + ┌────────────────┐ ┌────────────────┐ + │ Prisma Client │ │ SQLite Direct │ + │ (Server-side) │ │ (Embedded) │ + └────────────────┘ └────────────────┘ + │ │ + ▼ ▼ + ┌────────────────┐ ┌────────────────┐ + │ PostgreSQL │ │ SQLite DB │ + │ MySQL │ │ │ + │ SQL Server │ │ │ + └────────────────┘ └────────────────┘ +``` + +## Supported Databases + +The Prisma adapter behind DBAL already targets the databases you care about: PostgreSQL, MySQL, SQLite, and any other engine Prisma supports (SQL Server, CockroachDB, MongoDB, etc.). Switch between them by pointing `DATABASE_URL` at the desired backend and regenerating the Prisma client for your schema. + +The TypeScript client exposes three Prisma-based adapters: `PrismaAdapter`, `PostgresAdapter`, and `MySQLAdapter`. Setting `config.adapter` to `'postgres'` or `'mysql'` constructs the dialect-specific adapter, which keeps the shared Prisma logic but tweaks the capabilities metadata (e.g., enabling full-text search where supported) and leaves the rest of the stack focused on validation, ACLs, and audit logging. + +```bash +# PostgreSQL +export DATABASE_URL="postgresql://user:pass@db:5432/metabuilder" + +# MySQL +export DATABASE_URL="mysql://user:pass@db:3306/metabuilder" + +npx prisma generate +``` + +With `config.adapter = 'prisma'`, DBAL sends every request through `PrismaAdapter`, and Prisma handles dialect differences, migrations, and connection pooling defined in `prisma/schema.prisma` and `prisma/migrations/`. That keeps DBAL focused on validation, ACLs, and audit logging while it can still drive PostgreSQL, MySQL, or any other Prisma-supported store. + +The C++ daemon still resides in Phase 3—the current implementation is backed by the in-memory store described in `dbal/production/docs/PHASE3_DAEMON.md`, so Postgres/MySQL adapters for the daemon are still future work. + +### Native Prisma bridge + +The Phase 3 daemon can still leverage Prisma without bundling Node by calling `NativePrismaAdapter`. Each SQL plan is serialized as a JSON payload with the `$n` or `?` placeholders plus parameters and sent to `/api/native-prisma` on the Next.js server. The API route validates `DBAL_NATIVE_PRISMA_TOKEN`, reconstructs a `Prisma.sql` template, executes the query through the shared Prisma client, and returns rows or affected counts so the daemon sees the same `SqlRow`/`int` values as a regular SQL adapter. Set the same `DBAL_NATIVE_PRISMA_TOKEN` (mirrored in `frontends/nextjs/.env.example`) when running the daemon so the bridge rejects unauthorized callers. + +## Design Principles + +1. **Language Agnostic**: API contracts defined in YAML/Proto, not tied to any language +2. **Security First**: C++ daemon sandboxes all database access with ACL enforcement +3. **Development Speed**: TypeScript implementation for rapid iteration +4. **Zero Trust**: User code never touches database credentials or raw connections +5. **Capability-based**: Adapters declare what they support (transactions, joins, TTL, etc.) +6. **Testable**: Shared test vectors ensure both implementations behave identically + +## Repository Structure + +``` +dbal/ +├── api/ # Language-agnostic contracts (source of truth) +│ ├── schema/ # Entity and operation definitions +│ ├── idl/ # Optional: Proto/FlatBuffers schemas +│ └── versioning/ # Compatibility rules +├── common/ # Shared test vectors and fixtures +├── ts/ # TypeScript implementation (development) +├── cpp/ # C++ implementation (production) +├── backends/ # Backend-specific assets +├── tools/ # Code generation and build tools +└── scripts/ # Cross-platform build scripts +``` + +## Quick Start + +### Development Mode (TypeScript) + +```bash +cd dbal/development +npm install +npm run build +npm test +``` + +### Production Mode (C++ Daemon) + +```bash +cd dbal/production +mkdir build && cd build +cmake .. +make +./dbal_daemon --config=../config/prod.yaml +``` + +### GitHub Spark Integration + +For GitHub Spark deployments, the DBAL daemon runs as a sidecar service: + +```yaml +# In your Spark deployment config +services: + dbal: + image: your-org/dbal-daemon:latest + ports: + - "50051:50051" # gRPC endpoint + environment: + - DBAL_MODE=production + - DBAL_SANDBOX=strict +``` + +## Monitoring & Daemon UI + +`frontends/dbal` is a dedicated Next.js mini-app that showcases the C++ daemon's architecture, deployment readiness, and the `ServerStatusPanel`. The main `frontends/nextjs` app re-exports the `@dbal-ui` component at `/dbal-daemon`, and the panel polls `/api/status` (the shared feed lives in `frontends/dbal/src/status.ts`). Keep this page covered with `frontends/nextjs/e2e/dbal-daemon/daemon.spec.ts` and `playwright.dbal-daemon.config.ts`, or run `npm run test:e2e:dbal-daemon` after touching the UI. + +## Security Model + +### Sandboxing Strategy + +1. **Process Isolation**: Daemon runs in separate process with restricted permissions +2. **Capability-based Security**: Each request checked against user ACL +3. **Query Validation**: All queries parsed and validated before execution +4. **Credential Protection**: DB credentials never exposed to client code +5. **Audit Logging**: All operations logged for security review + +### ACL System + +```yaml +user: "user_123" +role: "editor" +permissions: + - entity: "posts" + operations: [create, read, update] + filters: + author_id: "$user.id" # Row-level security + - entity: "comments" + operations: [create, read] +``` + +## API Contract Example + +### HTTP Utilities + +For outbound integrations the daemon can use the new requests-inspired helper `runtime::RequestsClient`. It wraps the `cpr` HTTP helpers, exposes `get`/`post` helpers, parses JSON responses, and throws clean timeouts so code paths stay predictable. + +Native Prisma calls route through `NativePrismaAdapter`, which currently POSTs to the `/api/native-prisma` Next.js API and returns the raw JSON rows or affected count using that helper. When the daemon calls `runQuery`/`runNonQuery`, the response is mapped back into `SqlRow` results so the rest of the stack stays unaware of the HTTP transport. + +```cpp +using namespace dbal::runtime; + +RequestsClient http("https://api.prisma.example"); +auto response = http.post("/rpc/execute", jsonPayload.dump(), {{"Authorization", "Bearer ..."}}); +if (response.statusCode == 200) { + const auto result = response.json["result"]; + // handle Prisma response +} +``` + +### Entity Definition (YAML) + +```yaml +# api/schema/entities/post.yaml +entity: Post +version: "1.0" +fields: + id: + type: uuid + primary: true + generated: true + title: + type: string + required: true + max_length: 200 + content: + type: text + required: true + author_id: + type: uuid + required: true + foreign_key: + entity: User + field: id + created_at: + type: datetime + generated: true + updated_at: + type: datetime + auto_update: true +``` + +### Operations (YAML) + +```yaml +# api/schema/operations/post.ops.yaml +operations: + create: + input: [title, content, author_id] + output: Post + acl_required: ["post:create"] + + read: + input: [id] + output: Post + acl_required: ["post:read"] + + update: + input: [id, title?, content?] + output: Post + acl_required: ["post:update"] + row_level_check: "author_id = $user.id" + + delete: + input: [id] + output: boolean + acl_required: ["post:delete"] + row_level_check: "author_id = $user.id OR $user.role = 'admin'" + + list: + input: [filter?, sort?, page?, limit?] + output: Post[] + acl_required: ["post:read"] +``` + +## Client Usage + +### TypeScript Client + +```typescript +import { DBALClient } from '@metabuilder/dbal' + +const client = new DBALClient({ + mode: 'development', // or 'production' + endpoint: 'localhost:50051', + auth: { + user: currentUser, + session: currentSession + } +}) + +// CRUD operations +const post = await client.posts.create({ + title: 'Hello World', + content: 'This is my first post', + author_id: user.id +}) + +const posts = await client.posts.list({ + filter: { author_id: user.id }, + sort: { created_at: 'desc' }, + limit: 10 +}) + +const updated = await client.posts.update(post.id, { + title: 'Updated Title' +}) + +await client.posts.delete(post.id) +``` + +## Development Workflow + +1. **Define Schema**: Edit YAML files in `api/schema/` +2. **Generate Code**: `python tools/codegen/gen_types.py` +3. **Implement Adapter**: Add backend support in `ts/src/adapters/` +4. **Write Tests**: Create conformance tests in `common/fixtures/` +5. **Run Tests**: `npm run test:conformance` +6. **Build C++ Daemon**: `cd cpp && cmake --build build` +7. **Deploy**: Use Docker/Kubernetes to deploy daemon + +## Testing + +### Conformance Testing + +The DBAL includes comprehensive conformance tests that ensure both TypeScript and C++ implementations behave identically: + +```bash +# Run all conformance tests +python tools/conformance/run_all.py + +# Run TS tests only +cd ts && npm run test:conformance + +# Run C++ tests only +cd cpp && ./build/tests/conformance_tests +``` + +### Test Vectors + +Shared test vectors in `common/fixtures/` ensure consistency: + +```yaml +# common/contracts/conformance_cases.yaml +- name: "Create and read post" + operations: + - action: create + entity: Post + input: + title: "Test Post" + content: "Test content" + author_id: "user_123" + expected: + status: success + output: + id: "" + title: "Test Post" + - action: read + entity: Post + input: + id: "$prev.id" + expected: + status: success + output: + title: "Test Post" +``` + +## Migration from Current System + +### Phase 1: Development Mode (Complete) +- Use TypeScript DBAL client in development +- Direct Prisma access (no daemon) +- Validates API contract compliance + +### Phase 2: Hybrid Mode (Current Implementation) +- Complete TypeScript DBAL client with Prisma adapter +- WebSocket bridge for remote daemon communication (prepared for C++) +- ACL enforcement and audit logging in TypeScript +- Runs entirely in GitHub Spark environment +- Prepares architecture for C++ daemon migration + +### Phase 3: Full Production (Future) +- All environments use C++ daemon +- TypeScript client communicates via WebSocket/gRPC +- Maximum security and performance +- Requires infrastructure beyond GitHub Spark + +## Capabilities System + +Different backends support different features: + +```yaml +# api/schema/capabilities.yaml +adapters: + prisma: + transactions: true + joins: true + full_text_search: false + ttl: false + json_queries: true + + sqlite: + transactions: true + joins: true + full_text_search: true + ttl: false + json_queries: true + + mongodb: + transactions: true + joins: false + full_text_search: true + ttl: true + json_queries: true +``` + +Client code can check capabilities: + +```typescript +if (await client.capabilities.hasJoins()) { + // Use join query +} else { + // Fall back to multiple queries +} +``` + +## Error Handling + +Standardized errors across all implementations: + +```yaml +# api/schema/errors.yaml +errors: + NOT_FOUND: + code: 404 + message: "Entity not found" + + CONFLICT: + code: 409 + message: "Entity already exists" + + UNAUTHORIZED: + code: 401 + message: "Authentication required" + + FORBIDDEN: + code: 403 + message: "Insufficient permissions" + + VALIDATION_ERROR: + code: 422 + message: "Validation failed" + fields: + - field: string + error: string +``` + +## Contributing + +See [CONTRIBUTING.md](../docs/CONTRIBUTING.md) for development guidelines. + +## License + +MIT License - see [LICENSE](LICENSE) diff --git a/dbal/README_INDEX.md b/dbal/docs/README_INDEX.md similarity index 73% rename from dbal/README_INDEX.md rename to dbal/docs/README_INDEX.md index 5bd4016bd..11fa9f710 100644 --- a/dbal/README_INDEX.md +++ b/dbal/docs/README_INDEX.md @@ -6,14 +6,14 @@ The DBAL (Data Bus Abstraction Layer) provides a comprehensive implementation gu ### Getting Started -- [Quick Start Guide](./shared/docs/QUICK_START.md) - Setup and first steps +- [Quick Start Guide](../shared/docs/QUICK_START.md) - Setup and first steps - [README](./README.md) - Project overview ### Implementation Guides -- [Phase 2 Implementation](./shared/docs/PHASE2_IMPLEMENTATION.md) - Version 2 features and design -- [Phase 2 Complete](./shared/docs/PHASE2_COMPLETE.md) - Implementation completion status -- [Implementation Summary](./shared/docs/IMPLEMENTATION_SUMMARY.md) - Feature overview +- [Phase 2 Implementation](../shared/docs/PHASE2_IMPLEMENTATION.md) - Version 2 features and design +- [Phase 2 Complete](../shared/docs/PHASE2_COMPLETE.md) - Implementation completion status +- [Implementation Summary](../shared/docs/IMPLEMENTATION_SUMMARY.md) - Feature overview ### Architecture @@ -24,12 +24,16 @@ The DBAL (Data Bus Abstraction Layer) provides a comprehensive implementation gu ``` dbal/ -├── README.md # Project overview -├── README_INDEX.md # Documentation index -├── PROJECT.md # Complete documentation -├── AGENTS.md # AI development guidelines +├── README.md # Quick reference ├── LICENSE # MIT License ├── .gitignore # Git ignore rules +├── docs/ # Documentation +│ ├── README.md # Main project documentation +│ ├── README_INDEX.md # Documentation index +│ ├── PROJECT.md # Complete project reference +│ ├── AGENTS.md # AI development guidelines +│ ├── RESTRUCTURE_SUMMARY.md # Restructure details +│ └── BEFORE_AFTER.md # Before/after comparison ├── development/ # TypeScript implementation (fast iteration) ├── production/ # C++ implementation (security & performance) └── shared/ # Shared resources diff --git a/dbal/RESTRUCTURE_SUMMARY.md b/dbal/docs/RESTRUCTURE_SUMMARY.md similarity index 100% rename from dbal/RESTRUCTURE_SUMMARY.md rename to dbal/docs/RESTRUCTURE_SUMMARY.md diff --git a/docs/archive/PHASE2_SUMMARY.md b/docs/archive/PHASE2_SUMMARY.md index 53ce42dd8..137293eb3 100644 --- a/docs/archive/PHASE2_SUMMARY.md +++ b/docs/archive/PHASE2_SUMMARY.md @@ -108,7 +108,7 @@ MetaBuilder App 📖 **Start Here**: `dbal/QUICK_START.md` 📚 **Full Guide**: `dbal/PHASE2_IMPLEMENTATION.md` -🏗️ **Architecture**: `dbal/README.md` +🏗️ **Architecture**: `dbal/docs/README.md` 🚀 **Future**: `dbal/production/PHASE3_DAEMON.md` ## Performance diff --git a/docs/archive/organization/DOCS_ORGANIZATION_COMPLETE.md b/docs/archive/organization/DOCS_ORGANIZATION_COMPLETE.md index 612b93dcd..f2763dfc4 100644 --- a/docs/archive/organization/DOCS_ORGANIZATION_COMPLETE.md +++ b/docs/archive/organization/DOCS_ORGANIZATION_COMPLETE.md @@ -160,7 +160,7 @@ Documentation is now: - ✅ `docs/ORGANIZATION.md` - New documentation organization guide - ✅ `docs/guides/README.md` - Development guides index -- ✅ `docs/dbal/README.md` - Database abstraction layer overview +- ✅ `docs/dbal/docs/README.md` - Database abstraction layer overview - ✅ `docs/packages/README.md` - Package system overview - ✅ `docs/development/README.md` - Development resources - ✅ `docs/deployments/README.md` - Infrastructure guide diff --git a/docs/guides/PRIORITY_ACTION_PLAN.md b/docs/guides/PRIORITY_ACTION_PLAN.md index e38d93c6e..3cd1fe580 100644 --- a/docs/guides/PRIORITY_ACTION_PLAN.md +++ b/docs/guides/PRIORITY_ACTION_PLAN.md @@ -236,7 +236,7 @@ interface StateAnalysis { - ❌ Import/export missing UI - ❌ No pre-built packages -**Read:** `/dbal/AGENTS.md` for package architecture guidance +**Read:** `/dbal/docs/AGENTS.md` for package architecture guidance --- @@ -762,10 +762,10 @@ npx tsx scripts/enforce-size-limits.ts **Package System:** - [packages/README.md](./packages/README.md) - [docs/architecture/packages.md](./docs/architecture/packages.md) -- [dbal/AGENTS.md](./dbal/AGENTS.md) - Package guidance +- [dbal/docs/AGENTS.md](./dbal/docs/AGENTS.md) - Package guidance **DBAL:** -- [dbal/README.md](./dbal/README.md) +- [dbal/docs/README.md](./dbal/docs/README.md) - [dbal/QUICK_START.md](./dbal/QUICK_START.md) --- diff --git a/docs/implementation/roadmap/IMPROVEMENT_ROADMAP_INDEX.md b/docs/implementation/roadmap/IMPROVEMENT_ROADMAP_INDEX.md index c8e7e8a22..4d7c583a1 100644 --- a/docs/implementation/roadmap/IMPROVEMENT_ROADMAP_INDEX.md +++ b/docs/implementation/roadmap/IMPROVEMENT_ROADMAP_INDEX.md @@ -363,7 +363,7 @@ Documentation: Consolidated, indexed, comprehensive ### Package System - [packages/README.md](./packages/README.md) -- [DBAL Architecture](./dbal/README.md) +- [DBAL Architecture](./dbal/docs/README.md) - [Package Development Guide](./PACKAGE_SYSTEM_COMPLETION.md) --- diff --git a/docs/navigation/NAVIGATION.md b/docs/navigation/NAVIGATION.md index f465d637b..cc3bf0d3b 100644 --- a/docs/navigation/NAVIGATION.md +++ b/docs/navigation/NAVIGATION.md @@ -89,7 +89,7 @@ API reference and integration: - [api/README.md](./api/README.md) - API overview - [api/platform-guide.md](./api/platform-guide.md) - Platform guide - [api/quick-reference.md](./api/quick-reference.md) - Quick reference -- [dbal/README.md](./dbal/README.md) - DBAL overview +- [dbal/docs/README.md](./dbal/docs/README.md) - DBAL overview ### 🚢 Deployment & DevOps Deployment and infrastructure: @@ -148,7 +148,7 @@ TODO: docs/src/ is missing; add the folder or update/remove the src links below. - [src/types/](./src/types/) - Type definitions ### 🏁 DBAL (Database Abstraction Layer) -- [dbal/README.md](./dbal/README.md) - Overview +- [dbal/docs/README.md](./dbal/docs/README.md) - Overview - [dbal/shared/api/README.md](./dbal/shared/api/README.md) - API docs - [dbal/shared/backends/README.md](./dbal/shared/backends/README.md) - Backends - [dbal/production/README.md](./dbal/production/README.md) - C++ implementation diff --git a/docs/reference/indexes/DOCUMENTATION_FINDINGS.md b/docs/reference/indexes/DOCUMENTATION_FINDINGS.md index 6e80193bf..770f16688 100644 --- a/docs/reference/indexes/DOCUMENTATION_FINDINGS.md +++ b/docs/reference/indexes/DOCUMENTATION_FINDINGS.md @@ -632,11 +632,11 @@ TODO: Core doc links below point to docs/reference; update to correct paths unde | File | Purpose | |------|---------| -| [dbal/README.md](../dbal/README.md) | DBAL architecture and overview | +| [dbal/docs/README.md](../dbal/docs/README.md) | DBAL architecture and overview | | [dbal/QUICK_START.md](../dbal/QUICK_START.md) | DBAL 5-minute quick start | | [dbal/PROJECT.md](../dbal/PROJECT.md) | DBAL project structure | | [dbal/IMPLEMENTATION_SUMMARY.md](../dbal/IMPLEMENTATION_SUMMARY.md) | Phase 2 DBAL implementation summary | -| [dbal/AGENTS.md](../dbal/AGENTS.md) | Agent development guide for DBAL | +| [dbal/docs/AGENTS.md](../dbal/docs/AGENTS.md) | Agent development guide for DBAL | | [dbal/PHASE2_COMPLETE.md](../dbal/PHASE2_COMPLETE.md) | Phase 2 completion status | ### Development Guides