git/metabuilder
1
0
Fork 0
mirror of https://github.com/johndoe6345789/metabuilder.git synced 2026-04-24 13:54:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

Move documentation files to docs folder

Browse Source 
Co-authored-by: johndoe6345789 <224850594+johndoe6345789@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2025-12-27 13:36:33 +00:00
parent 6658c6af0d
commit 082c687325
18 changed files with 506 additions and 449 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
.github/COPILOT_ANALYSIS.md vendored
View File

@@ -8,7 +8,7 @@
### Analysis Approach
1. **Examined existing instructions**
- `dbal/AGENTS.md` (605 lines) - DBAL-specific agent development guide
- `dbal/docs/AGENTS.md` (605 lines) - DBAL-specific agent development guide
- `.github/copilot-instructions.md` (existing) - Original generic guidance
2. **Analyzed codebase patterns** through:
@@ -116,7 +116,7 @@ Instructions now reference:
| File | Purpose | Why Referenced |
|------|---------|-----------------|
| `dbal/AGENTS.md` | DBAL development guide | Critical for DBAL changes |
| `dbal/docs/AGENTS.md` | DBAL development guide | Critical for DBAL changes |
| `src/lib/database.ts` | Database operations | 1200+ LOC utility wrapper, required for all DB access |
| `src/components/RenderComponent.tsx` | Generic renderer | 221 LOC example of declarative UI pattern |
| `src/lib/schema-utils.test.ts` | Test examples | 63 tests showing parameterized pattern |
@@ -159,7 +159,7 @@ Instructions now reference:
### Adding a new database entity
1. Read: API-First DBAL Development pattern
2. Check: DBAL-Specific Guidance (YAML → Types → Adapters)
3. Reference: `dbal/AGENTS.md` for detailed workflow
3. Reference: `dbal/docs/AGENTS.md` for detailed workflow
### Creating a new component feature
1. Read: Generic Component Rendering pattern
@@ -192,7 +192,7 @@ Agents should prioritize these when onboarding:
1. **Start**: `docs/architecture/5-level-system.md` (understand permissions)
2. **Then**: `docs/architecture/packages.md` (understand modularity)
3. **Then**: `src/lib/database.ts` (understand DB pattern)
4. **Then**: `dbal/AGENTS.md` (if working on DBAL)
4. **Then**: `dbal/docs/AGENTS.md` (if working on DBAL)
5. **Always**: `FUNCTION_TEST_COVERAGE.md` (for test requirements)
---

2
.github/TEMPLATES.md vendored
View File

@@ -338,6 +338,6 @@ Please submit an issue with the "documentation" template to suggest improvements
- **Workflow Guide**: `.github/prompts/0-kickstart.md`
- **Contributing**: `README.md` → Contributing section
- **Architecture**: `docs/architecture/`
- **DBAL Guide**: `dbal/AGENTS.md`
- **DBAL Guide**: `dbal/docs/AGENTS.md`
- **UI Standards**: `UI_STANDARDS.md`
- **Copilot Instructions**: `.github/copilot-instructions.md`

2
.github/copilot-instructions.md vendored
View File

@@ -217,7 +217,7 @@ If fixing a DBAL bug:
- **Database**: `src/lib/database.ts` (all DB operations), `prisma/schema.prisma` (schema)
- **Packages**: `src/lib/package-loader.ts` (initialization), `packages/*/seed/` (definitions)
- **Tests**: `src/lib/schema-utils.test.ts` (parameterized pattern), `FUNCTION_TEST_COVERAGE.md` (auto-generated report)
- **DBAL**: `dbal/AGENTS.md` (detailed DBAL agent guide), `api/schema/` (YAML contracts)
- **DBAL**: `dbal/docs/AGENTS.md` (detailed DBAL agent guide), `api/schema/` (YAML contracts)
## Questions to Ask

2
.github/prompts/workflow/0-kickstart.md vendored
View File

@@ -4,7 +4,7 @@ Use this as the default workflow when starting work in this repo.
## Workflow
1. Skim `docs/START_HERE.md` (if new), `docs/INDEX.md`, and relevant items in `docs/todo/`.
2. Check for scoped rules in nested `AGENTS.md` files (e.g. `dbal/AGENTS.md`) before editing those areas.
2. Check for scoped rules in nested `AGENTS.md` files (e.g. `dbal/docs/AGENTS.md`) before editing those areas.
3. Use the prompts in `.github/prompts/` as needed:
- Plan: `1-plan-feature.prompt.md`
- Design: `2-design-component.prompt.md`

2
AGENTS.md
View File

@@ -45,5 +45,5 @@ DBAL library workflows live in `dbal/development/` (`npm run build`, `npm run te
## Agent-Specific Notes
- Check for scoped rules in nested `AGENTS.md` files (e.g., `dbal/AGENTS.md`) before editing those areas.
- Check for scoped rules in nested `AGENTS.md` files (e.g., `dbal/docs/AGENTS.md`) before editing those areas.
- Keep changes focused, avoid dependency churn, and follow existing patterns/config in `config/` and `frontends/nextjs/`.

441
dbal/README.md
View File

@@ -1,437 +1,46 @@
# Database Abstraction Layer (DBAL)
# DBAL - Database Abstraction Layer
A language-agnostic database abstraction layer that provides a secure interface between client applications and database backends. The DBAL uses TypeScript for rapid development and testing, with a C++ production layer for enhanced security and performance.
A language-agnostic database abstraction layer that provides a secure interface between client applications and database backends.
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ Client Application (Spark) │
│ (TypeScript/React) │
└────────────────────────────────┬────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ DBAL Client │
│ (TypeScript Dev / C++ Production) │
│ ┌────────────────────┬──────────────────┬────────────────────┐ │
│ │ Query Builder │ Validation │ Error Handling │ │
│ └────────────────────┴──────────────────┴────────────────────┘ │
└────────────────────────────────┬────────────────────────────────┘
│
┌────────────┴────────────┐
│ IPC/RPC Bridge │
│ (gRPC/WebSocket) │
└────────────┬────────────┘
│
┌─────────────────────────────────────────────────────────────────┐
│ DBAL Daemon (C++) │
│ [Production Only - Sandboxed] │
│ ┌────────────────────┬──────────────────┬────────────────────┐ │
│ │ Auth/ACL │ Query Executor │ Connection Pool │ │
│ └────────────────────┴──────────────────┴────────────────────┘ │
└────────────────────────────────┬────────────────────────────────┘
│
┌────────────┴────────────┐
│ │
▼ ▼
┌────────────────┐ ┌────────────────┐
│ Prisma Client │ │ SQLite Direct │
│ (Server-side) │ │ (Embedded) │
└────────────────┘ └────────────────┘
│ │
▼ ▼
┌────────────────┐ ┌────────────────┐
│ PostgreSQL │ │ SQLite DB │
│ MySQL │ │ │
│ SQL Server │ │ │
└────────────────┘ └────────────────┘
```
## Supported Databases
The Prisma adapter behind DBAL already targets the databases you care about: PostgreSQL, MySQL, SQLite, and any other engine Prisma supports (SQL Server, CockroachDB, MongoDB, etc.). Switch between them by pointing `DATABASE_URL` at the desired backend and regenerating the Prisma client for your schema.
The TypeScript client exposes three Prisma-based adapters: `PrismaAdapter`, `PostgresAdapter`, and `MySQLAdapter`. Setting `config.adapter` to `'postgres'` or `'mysql'` constructs the dialect-specific adapter, which keeps the shared Prisma logic but tweaks the capabilities metadata (e.g., enabling full-text search where supported) and leaves the rest of the stack focused on validation, ACLs, and audit logging.
```bash
# PostgreSQL
export DATABASE_URL="postgresql://user:pass@db:5432/metabuilder"
# MySQL
export DATABASE_URL="mysql://user:pass@db:3306/metabuilder"
npx prisma generate
```
With `config.adapter = 'prisma'`, DBAL sends every request through `PrismaAdapter`, and Prisma handles dialect differences, migrations, and connection pooling defined in `prisma/schema.prisma` and `prisma/migrations/`. That keeps DBAL focused on validation, ACLs, and audit logging while it can still drive PostgreSQL, MySQL, or any other Prisma-supported store.
The C++ daemon still resides in Phase 3—the current implementation is backed by the in-memory store described in `dbal/production/docs/PHASE3_DAEMON.md`, so Postgres/MySQL adapters for the daemon are still future work.
### Native Prisma bridge
The Phase 3 daemon can still leverage Prisma without bundling Node by calling `NativePrismaAdapter`. Each SQL plan is serialized as a JSON payload with the `$n` or `?` placeholders plus parameters and sent to `/api/native-prisma` on the Next.js server. The API route validates `DBAL_NATIVE_PRISMA_TOKEN`, reconstructs a `Prisma.sql` template, executes the query through the shared Prisma client, and returns rows or affected counts so the daemon sees the same `SqlRow`/`int` values as a regular SQL adapter. Set the same `DBAL_NATIVE_PRISMA_TOKEN` (mirrored in `frontends/nextjs/.env.example`) when running the daemon so the bridge rejects unauthorized callers.
## Design Principles
1. **Language Agnostic**: API contracts defined in YAML/Proto, not tied to any language
2. **Security First**: C++ daemon sandboxes all database access with ACL enforcement
3. **Development Speed**: TypeScript implementation for rapid iteration
4. **Zero Trust**: User code never touches database credentials or raw connections
5. **Capability-based**: Adapters declare what they support (transactions, joins, TTL, etc.)
6. **Testable**: Shared test vectors ensure both implementations behave identically
## Repository Structure
## Structure
```
dbal/
├── api/ # Language-agnostic contracts (source of truth)
│ ├── schema/ # Entity and operation definitions
│ ├── idl/ # Optional: Proto/FlatBuffers schemas
│ └── versioning/ # Compatibility rules
├── common/ # Shared test vectors and fixtures
├── ts/ # TypeScript implementation (development)
├── cpp/ # C++ implementation (production)
├── backends/ # Backend-specific assets
├── tools/ # Code generation and build tools
└── scripts/ # Cross-platform build scripts
├── development/ # TypeScript implementation (fast iteration)
├── production/ # C++ implementation (security & performance)
├── shared/ # Shared resources (API specs, tools, etc.)
└── docs/ # Documentation
```
## Quick Start
## Quick Links
### Development Mode (TypeScript)
- 📖 **[Full Documentation](docs/README.md)** - Complete project documentation
- 🚀 **[Quick Start](shared/docs/QUICK_START.md)** - Get started in 5 minutes
- 🏗️ **[Architecture](docs/PROJECT.md)** - System architecture and design
- 🤖 **[Agent Guide](docs/AGENTS.md)** - AI development guidelines
- 📋 **[Restructure Info](docs/RESTRUCTURE_SUMMARY.md)** - Recent organizational changes
## Development
### TypeScript (Development)
```bash
cd dbal/development
cd development
npm install
npm run build
npm test
```
### Production Mode (C++ Daemon)
### C++ (Production)
```bash
cd dbal/production
mkdir build && cd build
cmake ..
make
./dbal_daemon --config=../config/prod.yaml
cd production
# See production/docs/ for C++ build instructions
```
### GitHub Spark Integration
For GitHub Spark deployments, the DBAL daemon runs as a sidecar service:
```yaml
# In your Spark deployment config
services:
dbal:
image: your-org/dbal-daemon:latest
ports:
- "50051:50051" # gRPC endpoint
environment:
- DBAL_MODE=production
- DBAL_SANDBOX=strict
```
## Monitoring & Daemon UI
`frontends/dbal` is a dedicated Next.js mini-app that showcases the C++ daemon's architecture, deployment readiness, and the `ServerStatusPanel`. The main `frontends/nextjs` app re-exports the `@dbal-ui` component at `/dbal-daemon`, and the panel polls `/api/status` (the shared feed lives in `frontends/dbal/src/status.ts`). Keep this page covered with `frontends/nextjs/e2e/dbal-daemon/daemon.spec.ts` and `playwright.dbal-daemon.config.ts`, or run `npm run test:e2e:dbal-daemon` after touching the UI.
## Security Model
### Sandboxing Strategy
1. **Process Isolation**: Daemon runs in separate process with restricted permissions
2. **Capability-based Security**: Each request checked against user ACL
3. **Query Validation**: All queries parsed and validated before execution
4. **Credential Protection**: DB credentials never exposed to client code
5. **Audit Logging**: All operations logged for security review
### ACL System
```yaml
user: "user_123"
role: "editor"
permissions:
- entity: "posts"
operations: [create, read, update]
filters:
author_id: "$user.id" # Row-level security
- entity: "comments"
operations: [create, read]
```
## API Contract Example
### HTTP Utilities
For outbound integrations the daemon can use the new requests-inspired helper `runtime::RequestsClient`. It wraps the `cpr` HTTP helpers, exposes `get`/`post` helpers, parses JSON responses, and throws clean timeouts so code paths stay predictable.
Native Prisma calls route through `NativePrismaAdapter`, which currently POSTs to the `/api/native-prisma` Next.js API and returns the raw JSON rows or affected count using that helper. When the daemon calls `runQuery`/`runNonQuery`, the response is mapped back into `SqlRow` results so the rest of the stack stays unaware of the HTTP transport.
```cpp
using namespace dbal::runtime;
RequestsClient http("https://api.prisma.example");
auto response = http.post("/rpc/execute", jsonPayload.dump(), {{"Authorization", "Bearer ..."}});
if (response.statusCode == 200) {
const auto result = response.json["result"];
// handle Prisma response
}
```
### Entity Definition (YAML)
```yaml
# api/schema/entities/post.yaml
entity: Post
version: "1.0"
fields:
id:
type: uuid
primary: true
generated: true
title:
type: string
required: true
max_length: 200
content:
type: text
required: true
author_id:
type: uuid
required: true
foreign_key:
entity: User
field: id
created_at:
type: datetime
generated: true
updated_at:
type: datetime
auto_update: true
```
### Operations (YAML)
```yaml
# api/schema/operations/post.ops.yaml
operations:
create:
input: [title, content, author_id]
output: Post
acl_required: ["post:create"]
read:
input: [id]
output: Post
acl_required: ["post:read"]
update:
input: [id, title?, content?]
output: Post
acl_required: ["post:update"]
row_level_check: "author_id = $user.id"
delete:
input: [id]
output: boolean
acl_required: ["post:delete"]
row_level_check: "author_id = $user.id OR $user.role = 'admin'"
list:
input: [filter?, sort?, page?, limit?]
output: Post[]
acl_required: ["post:read"]
```
## Client Usage
### TypeScript Client
```typescript
import { DBALClient } from '@metabuilder/dbal'
const client = new DBALClient({
mode: 'development', // or 'production'
endpoint: 'localhost:50051',
auth: {
user: currentUser,
session: currentSession
}
})
// CRUD operations
const post = await client.posts.create({
title: 'Hello World',
content: 'This is my first post',
author_id: user.id
})
const posts = await client.posts.list({
filter: { author_id: user.id },
sort: { created_at: 'desc' },
limit: 10
})
const updated = await client.posts.update(post.id, {
title: 'Updated Title'
})
await client.posts.delete(post.id)
```
## Development Workflow
1. **Define Schema**: Edit YAML files in `api/schema/`
2. **Generate Code**: `python tools/codegen/gen_types.py`
3. **Implement Adapter**: Add backend support in `ts/src/adapters/`
4. **Write Tests**: Create conformance tests in `common/fixtures/`
5. **Run Tests**: `npm run test:conformance`
6. **Build C++ Daemon**: `cd cpp && cmake --build build`
7. **Deploy**: Use Docker/Kubernetes to deploy daemon
## Testing
### Conformance Testing
The DBAL includes comprehensive conformance tests that ensure both TypeScript and C++ implementations behave identically:
```bash
# Run all conformance tests
python tools/conformance/run_all.py
# Run TS tests only
cd ts && npm run test:conformance
# Run C++ tests only
cd cpp && ./build/tests/conformance_tests
```
### Test Vectors
Shared test vectors in `common/fixtures/` ensure consistency:
```yaml
# common/contracts/conformance_cases.yaml
- name: "Create and read post"
operations:
- action: create
entity: Post
input:
title: "Test Post"
content: "Test content"
author_id: "user_123"
expected:
status: success
output:
id: "<uuid>"
title: "Test Post"
- action: read
entity: Post
input:
id: "$prev.id"
expected:
status: success
output:
title: "Test Post"
```
## Migration from Current System
### Phase 1: Development Mode (Complete)
- Use TypeScript DBAL client in development
- Direct Prisma access (no daemon)
- Validates API contract compliance
### Phase 2: Hybrid Mode (Current Implementation)
- Complete TypeScript DBAL client with Prisma adapter
- WebSocket bridge for remote daemon communication (prepared for C++)
- ACL enforcement and audit logging in TypeScript
- Runs entirely in GitHub Spark environment
- Prepares architecture for C++ daemon migration
### Phase 3: Full Production (Future)
- All environments use C++ daemon
- TypeScript client communicates via WebSocket/gRPC
- Maximum security and performance
- Requires infrastructure beyond GitHub Spark
## Capabilities System
Different backends support different features:
```yaml
# api/schema/capabilities.yaml
adapters:
prisma:
transactions: true
joins: true
full_text_search: false
ttl: false
json_queries: true
sqlite:
transactions: true
joins: true
full_text_search: true
ttl: false
json_queries: true
mongodb:
transactions: true
joins: false
full_text_search: true
ttl: true
json_queries: true
```
Client code can check capabilities:
```typescript
if (await client.capabilities.hasJoins()) {
// Use join query
} else {
// Fall back to multiple queries
}
```
## Error Handling
Standardized errors across all implementations:
```yaml
# api/schema/errors.yaml
errors:
NOT_FOUND:
code: 404
message: "Entity not found"
CONFLICT:
code: 409
message: "Entity already exists"
UNAUTHORIZED:
code: 401
message: "Authentication required"
FORBIDDEN:
code: 403
message: "Insufficient permissions"
VALIDATION_ERROR:
code: 422
message: "Validation failed"
fields:
- field: string
error: string
```
## Contributing
See [CONTRIBUTING.md](../docs/CONTRIBUTING.md) for development guidelines.
### Shared Resources
- **API Schemas**: `shared/api/schema/`
- **Tools**: `shared/tools/` (codegen, build assistant)
- **Scripts**: `shared/scripts/` (build, test)
## License
MIT License - see [LICENSE](LICENSE)
MIT - See [LICENSE](LICENSE) file.

0
dbal/AGENTS.md → dbal/docs/AGENTS.md
View File

0
dbal/BEFORE_AFTER.md → dbal/docs/BEFORE_AFTER.md
View File

21
dbal/PROJECT.md → dbal/docs/PROJECT.md
View File

@@ -4,21 +4,28 @@ This directory contains the Database Abstraction Layer for MetaBuilder.
## Quick Links
- [Main README](README.md) - Overview and architecture
- [Agent Guide](AGENTS.md) - For AI agents and automated tools
- [Spark Integration](shared/docs/SPARK_INTEGRATION.md) - GitHub Spark deployment guide
- [TypeScript Implementation](development/README.md) - TS development guide
- [C++ Implementation](production/README.md) - C++ production guide
- [Main README](../README.md) - Overview and getting started
- [Agent Guide](./AGENTS.md) - For AI agents and automated tools
- [Spark Integration](../shared/docs/SPARK_INTEGRATION.md) - GitHub Spark deployment guide
- [TypeScript Implementation](../development/README.md) - TS development guide
- [C++ Implementation](../production/README.md) - C++ production guide
## Directory Structure
```
dbal/
├── README.md # Main documentation
├── README.md # Quick reference
├── LICENSE # MIT License
├── AGENTS.md # Agent development guide
├── .gitignore # Git ignore rules
│
├── docs/ # Documentation
│ ├── README.md # Main project documentation
│ ├── README_INDEX.md # Documentation index
│ ├── PROJECT.md # Complete project reference
│ ├── AGENTS.md # Agent development guide
│ ├── RESTRUCTURE_SUMMARY.md # Restructure details
│ └── BEFORE_AFTER.md # Before/after comparison
│
├── development/ # TypeScript implementation (fast iteration)
│ ├── src/ # Source code
│ ├── tests/ # Test suites

437
dbal/docs/README.md Normal file
View File

@@ -0,0 +1,437 @@
# Database Abstraction Layer (DBAL)
A language-agnostic database abstraction layer that provides a secure interface between client applications and database backends. The DBAL uses TypeScript for rapid development and testing, with a C++ production layer for enhanced security and performance.
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ Client Application (Spark) │
│ (TypeScript/React) │
└────────────────────────────────┬────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ DBAL Client │
│ (TypeScript Dev / C++ Production) │
│ ┌────────────────────┬──────────────────┬────────────────────┐ │
│ │ Query Builder │ Validation │ Error Handling │ │
│ └────────────────────┴──────────────────┴────────────────────┘ │
└────────────────────────────────┬────────────────────────────────┘
│
┌────────────┴────────────┐
│ IPC/RPC Bridge │
│ (gRPC/WebSocket) │
└────────────┬────────────┘
│
┌─────────────────────────────────────────────────────────────────┐
│ DBAL Daemon (C++) │
│ [Production Only - Sandboxed] │
│ ┌────────────────────┬──────────────────┬────────────────────┐ │
│ │ Auth/ACL │ Query Executor │ Connection Pool │ │
│ └────────────────────┴──────────────────┴────────────────────┘ │
└────────────────────────────────┬────────────────────────────────┘
│
┌────────────┴────────────┐
│ │
▼ ▼
┌────────────────┐ ┌────────────────┐
│ Prisma Client │ │ SQLite Direct │
│ (Server-side) │ │ (Embedded) │
└────────────────┘ └────────────────┘
│ │
▼ ▼
┌────────────────┐ ┌────────────────┐
│ PostgreSQL │ │ SQLite DB │
│ MySQL │ │ │
│ SQL Server │ │ │
└────────────────┘ └────────────────┘
```
## Supported Databases
The Prisma adapter behind DBAL already targets the databases you care about: PostgreSQL, MySQL, SQLite, and any other engine Prisma supports (SQL Server, CockroachDB, MongoDB, etc.). Switch between them by pointing `DATABASE_URL` at the desired backend and regenerating the Prisma client for your schema.
The TypeScript client exposes three Prisma-based adapters: `PrismaAdapter`, `PostgresAdapter`, and `MySQLAdapter`. Setting `config.adapter` to `'postgres'` or `'mysql'` constructs the dialect-specific adapter, which keeps the shared Prisma logic but tweaks the capabilities metadata (e.g., enabling full-text search where supported) and leaves the rest of the stack focused on validation, ACLs, and audit logging.
```bash
# PostgreSQL
export DATABASE_URL="postgresql://user:pass@db:5432/metabuilder"
# MySQL
export DATABASE_URL="mysql://user:pass@db:3306/metabuilder"
npx prisma generate
```
With `config.adapter = 'prisma'`, DBAL sends every request through `PrismaAdapter`, and Prisma handles dialect differences, migrations, and connection pooling defined in `prisma/schema.prisma` and `prisma/migrations/`. That keeps DBAL focused on validation, ACLs, and audit logging while it can still drive PostgreSQL, MySQL, or any other Prisma-supported store.
The C++ daemon still resides in Phase 3—the current implementation is backed by the in-memory store described in `dbal/production/docs/PHASE3_DAEMON.md`, so Postgres/MySQL adapters for the daemon are still future work.
### Native Prisma bridge
The Phase 3 daemon can still leverage Prisma without bundling Node by calling `NativePrismaAdapter`. Each SQL plan is serialized as a JSON payload with the `$n` or `?` placeholders plus parameters and sent to `/api/native-prisma` on the Next.js server. The API route validates `DBAL_NATIVE_PRISMA_TOKEN`, reconstructs a `Prisma.sql` template, executes the query through the shared Prisma client, and returns rows or affected counts so the daemon sees the same `SqlRow`/`int` values as a regular SQL adapter. Set the same `DBAL_NATIVE_PRISMA_TOKEN` (mirrored in `frontends/nextjs/.env.example`) when running the daemon so the bridge rejects unauthorized callers.
## Design Principles
1. **Language Agnostic**: API contracts defined in YAML/Proto, not tied to any language
2. **Security First**: C++ daemon sandboxes all database access with ACL enforcement
3. **Development Speed**: TypeScript implementation for rapid iteration
4. **Zero Trust**: User code never touches database credentials or raw connections
5. **Capability-based**: Adapters declare what they support (transactions, joins, TTL, etc.)
6. **Testable**: Shared test vectors ensure both implementations behave identically
## Repository Structure
```
dbal/
├── api/ # Language-agnostic contracts (source of truth)
│ ├── schema/ # Entity and operation definitions
│ ├── idl/ # Optional: Proto/FlatBuffers schemas
│ └── versioning/ # Compatibility rules
├── common/ # Shared test vectors and fixtures
├── ts/ # TypeScript implementation (development)
├── cpp/ # C++ implementation (production)
├── backends/ # Backend-specific assets
├── tools/ # Code generation and build tools
└── scripts/ # Cross-platform build scripts
```
## Quick Start
### Development Mode (TypeScript)
```bash
cd dbal/development
npm install
npm run build
npm test
```
### Production Mode (C++ Daemon)
```bash
cd dbal/production
mkdir build && cd build
cmake ..
make
./dbal_daemon --config=../config/prod.yaml
```
### GitHub Spark Integration
For GitHub Spark deployments, the DBAL daemon runs as a sidecar service:
```yaml
# In your Spark deployment config
services:
dbal:
image: your-org/dbal-daemon:latest
ports:
- "50051:50051" # gRPC endpoint
environment:
- DBAL_MODE=production
- DBAL_SANDBOX=strict
```
## Monitoring & Daemon UI
`frontends/dbal` is a dedicated Next.js mini-app that showcases the C++ daemon's architecture, deployment readiness, and the `ServerStatusPanel`. The main `frontends/nextjs` app re-exports the `@dbal-ui` component at `/dbal-daemon`, and the panel polls `/api/status` (the shared feed lives in `frontends/dbal/src/status.ts`). Keep this page covered with `frontends/nextjs/e2e/dbal-daemon/daemon.spec.ts` and `playwright.dbal-daemon.config.ts`, or run `npm run test:e2e:dbal-daemon` after touching the UI.
## Security Model
### Sandboxing Strategy
1. **Process Isolation**: Daemon runs in separate process with restricted permissions
2. **Capability-based Security**: Each request checked against user ACL
3. **Query Validation**: All queries parsed and validated before execution
4. **Credential Protection**: DB credentials never exposed to client code
5. **Audit Logging**: All operations logged for security review
### ACL System
```yaml
user: "user_123"
role: "editor"
permissions:
- entity: "posts"
operations: [create, read, update]
filters:
author_id: "$user.id" # Row-level security
- entity: "comments"
operations: [create, read]
```
## API Contract Example
### HTTP Utilities
For outbound integrations the daemon can use the new requests-inspired helper `runtime::RequestsClient`. It wraps the `cpr` HTTP helpers, exposes `get`/`post` helpers, parses JSON responses, and throws clean timeouts so code paths stay predictable.
Native Prisma calls route through `NativePrismaAdapter`, which currently POSTs to the `/api/native-prisma` Next.js API and returns the raw JSON rows or affected count using that helper. When the daemon calls `runQuery`/`runNonQuery`, the response is mapped back into `SqlRow` results so the rest of the stack stays unaware of the HTTP transport.
```cpp
using namespace dbal::runtime;
RequestsClient http("https://api.prisma.example");
auto response = http.post("/rpc/execute", jsonPayload.dump(), {{"Authorization", "Bearer ..."}});
if (response.statusCode == 200) {
const auto result = response.json["result"];
// handle Prisma response
}
```
### Entity Definition (YAML)
```yaml
# api/schema/entities/post.yaml
entity: Post
version: "1.0"
fields:
id:
type: uuid
primary: true
generated: true
title:
type: string
required: true
max_length: 200
content:
type: text
required: true
author_id:
type: uuid
required: true
foreign_key:
entity: User
field: id
created_at:
type: datetime
generated: true
updated_at:
type: datetime
auto_update: true
```
### Operations (YAML)
```yaml
# api/schema/operations/post.ops.yaml
operations:
create:
input: [title, content, author_id]
output: Post
acl_required: ["post:create"]
read:
input: [id]
output: Post
acl_required: ["post:read"]
update:
input: [id, title?, content?]
output: Post
acl_required: ["post:update"]
row_level_check: "author_id = $user.id"
delete:
input: [id]
output: boolean
acl_required: ["post:delete"]
row_level_check: "author_id = $user.id OR $user.role = 'admin'"
list:
input: [filter?, sort?, page?, limit?]
output: Post[]
acl_required: ["post:read"]
```
## Client Usage
### TypeScript Client
```typescript
import { DBALClient } from '@metabuilder/dbal'
const client = new DBALClient({
mode: 'development', // or 'production'
endpoint: 'localhost:50051',
auth: {
user: currentUser,
session: currentSession
}
})
// CRUD operations
const post = await client.posts.create({
title: 'Hello World',
content: 'This is my first post',
author_id: user.id
})
const posts = await client.posts.list({
filter: { author_id: user.id },
sort: { created_at: 'desc' },
limit: 10
})
const updated = await client.posts.update(post.id, {
title: 'Updated Title'
})
await client.posts.delete(post.id)
```
## Development Workflow
1. **Define Schema**: Edit YAML files in `api/schema/`
2. **Generate Code**: `python tools/codegen/gen_types.py`
3. **Implement Adapter**: Add backend support in `ts/src/adapters/`
4. **Write Tests**: Create conformance tests in `common/fixtures/`
5. **Run Tests**: `npm run test:conformance`
6. **Build C++ Daemon**: `cd cpp && cmake --build build`
7. **Deploy**: Use Docker/Kubernetes to deploy daemon
## Testing
### Conformance Testing
The DBAL includes comprehensive conformance tests that ensure both TypeScript and C++ implementations behave identically:
```bash
# Run all conformance tests
python tools/conformance/run_all.py
# Run TS tests only
cd ts && npm run test:conformance
# Run C++ tests only
cd cpp && ./build/tests/conformance_tests
```
### Test Vectors
Shared test vectors in `common/fixtures/` ensure consistency:
```yaml
# common/contracts/conformance_cases.yaml
- name: "Create and read post"
operations:
- action: create
entity: Post
input:
title: "Test Post"
content: "Test content"
author_id: "user_123"
expected:
status: success
output:
id: "<uuid>"
title: "Test Post"
- action: read
entity: Post
input:
id: "$prev.id"
expected:
status: success
output:
title: "Test Post"
```
## Migration from Current System
### Phase 1: Development Mode (Complete)
- Use TypeScript DBAL client in development
- Direct Prisma access (no daemon)
- Validates API contract compliance
### Phase 2: Hybrid Mode (Current Implementation)
- Complete TypeScript DBAL client with Prisma adapter
- WebSocket bridge for remote daemon communication (prepared for C++)
- ACL enforcement and audit logging in TypeScript
- Runs entirely in GitHub Spark environment
- Prepares architecture for C++ daemon migration
### Phase 3: Full Production (Future)
- All environments use C++ daemon
- TypeScript client communicates via WebSocket/gRPC
- Maximum security and performance
- Requires infrastructure beyond GitHub Spark
## Capabilities System
Different backends support different features:
```yaml
# api/schema/capabilities.yaml
adapters:
prisma:
transactions: true
joins: true
full_text_search: false
ttl: false
json_queries: true
sqlite:
transactions: true
joins: true
full_text_search: true
ttl: false
json_queries: true
mongodb:
transactions: true
joins: false
full_text_search: true
ttl: true
json_queries: true
```
Client code can check capabilities:
```typescript
if (await client.capabilities.hasJoins()) {
// Use join query
} else {
// Fall back to multiple queries
}
```
## Error Handling
Standardized errors across all implementations:
```yaml
# api/schema/errors.yaml
errors:
NOT_FOUND:
code: 404
message: "Entity not found"
CONFLICT:
code: 409
message: "Entity already exists"
UNAUTHORIZED:
code: 401
message: "Authentication required"
FORBIDDEN:
code: 403
message: "Insufficient permissions"
VALIDATION_ERROR:
code: 422
message: "Validation failed"
fields:
- field: string
error: string
```
## Contributing
See [CONTRIBUTING.md](../docs/CONTRIBUTING.md) for development guidelines.
## License
MIT License - see [LICENSE](LICENSE)

20
dbal/README_INDEX.md → dbal/docs/README_INDEX.md
View File

@@ -6,14 +6,14 @@ The DBAL (Data Bus Abstraction Layer) provides a comprehensive implementation gu
### Getting Started
- [Quick Start Guide](./shared/docs/QUICK_START.md) - Setup and first steps
- [Quick Start Guide](../shared/docs/QUICK_START.md) - Setup and first steps
- [README](./README.md) - Project overview
### Implementation Guides
- [Phase 2 Implementation](./shared/docs/PHASE2_IMPLEMENTATION.md) - Version 2 features and design
- [Phase 2 Complete](./shared/docs/PHASE2_COMPLETE.md) - Implementation completion status
- [Implementation Summary](./shared/docs/IMPLEMENTATION_SUMMARY.md) - Feature overview
- [Phase 2 Implementation](../shared/docs/PHASE2_IMPLEMENTATION.md) - Version 2 features and design
- [Phase 2 Complete](../shared/docs/PHASE2_COMPLETE.md) - Implementation completion status
- [Implementation Summary](../shared/docs/IMPLEMENTATION_SUMMARY.md) - Feature overview
### Architecture
@@ -24,12 +24,16 @@ The DBAL (Data Bus Abstraction Layer) provides a comprehensive implementation gu
```
dbal/
├── README.md # Project overview
├── README_INDEX.md # Documentation index
├── PROJECT.md # Complete documentation
├── AGENTS.md # AI development guidelines
├── README.md # Quick reference
├── LICENSE # MIT License
├── .gitignore # Git ignore rules
├── docs/ # Documentation
│ ├── README.md # Main project documentation
│ ├── README_INDEX.md # Documentation index
│ ├── PROJECT.md # Complete project reference
│ ├── AGENTS.md # AI development guidelines
│ ├── RESTRUCTURE_SUMMARY.md # Restructure details
│ └── BEFORE_AFTER.md # Before/after comparison
├── development/ # TypeScript implementation (fast iteration)
├── production/ # C++ implementation (security & performance)
└── shared/ # Shared resources

0
dbal/RESTRUCTURE_SUMMARY.md → dbal/docs/RESTRUCTURE_SUMMARY.md
View File

2
docs/archive/PHASE2_SUMMARY.md
View File

@@ -108,7 +108,7 @@ MetaBuilder App
📖 **Start Here**: `dbal/QUICK_START.md`
📚 **Full Guide**: `dbal/PHASE2_IMPLEMENTATION.md`
🏗️ **Architecture**: `dbal/README.md`
🏗️ **Architecture**: `dbal/docs/README.md`
🚀 **Future**: `dbal/production/PHASE3_DAEMON.md`
## Performance

2
docs/archive/organization/DOCS_ORGANIZATION_COMPLETE.md
View File

@@ -160,7 +160,7 @@ Documentation is now:
- ✅ `docs/ORGANIZATION.md` - New documentation organization guide
- ✅ `docs/guides/README.md` - Development guides index
- ✅ `docs/dbal/README.md` - Database abstraction layer overview
- ✅ `docs/dbal/docs/README.md` - Database abstraction layer overview
- ✅ `docs/packages/README.md` - Package system overview
- ✅ `docs/development/README.md` - Development resources
- ✅ `docs/deployments/README.md` - Infrastructure guide

6
docs/guides/PRIORITY_ACTION_PLAN.md
View File

@@ -236,7 +236,7 @@ interface StateAnalysis {
- ❌ Import/export missing UI
- ❌ No pre-built packages
**Read:** `/dbal/AGENTS.md` for package architecture guidance
**Read:** `/dbal/docs/AGENTS.md` for package architecture guidance
---
@@ -762,10 +762,10 @@ npx tsx scripts/enforce-size-limits.ts
**Package System:**
- [packages/README.md](./packages/README.md)
- [docs/architecture/packages.md](./docs/architecture/packages.md)
- [dbal/AGENTS.md](./dbal/AGENTS.md) - Package guidance
- [dbal/docs/AGENTS.md](./dbal/docs/AGENTS.md) - Package guidance
**DBAL:**
- [dbal/README.md](./dbal/README.md)
- [dbal/docs/README.md](./dbal/docs/README.md)
- [dbal/QUICK_START.md](./dbal/QUICK_START.md)
---

2
docs/implementation/roadmap/IMPROVEMENT_ROADMAP_INDEX.md
View File

@@ -363,7 +363,7 @@ Documentation: Consolidated, indexed, comprehensive
### Package System
- [packages/README.md](./packages/README.md)
- [DBAL Architecture](./dbal/README.md)
- [DBAL Architecture](./dbal/docs/README.md)
- [Package Development Guide](./PACKAGE_SYSTEM_COMPLETION.md)
---

4
docs/navigation/NAVIGATION.md
View File

@@ -89,7 +89,7 @@ API reference and integration:
- [api/README.md](./api/README.md) - API overview
- [api/platform-guide.md](./api/platform-guide.md) - Platform guide
- [api/quick-reference.md](./api/quick-reference.md) - Quick reference
- [dbal/README.md](./dbal/README.md) - DBAL overview
- [dbal/docs/README.md](./dbal/docs/README.md) - DBAL overview
### 🚢 Deployment & DevOps
Deployment and infrastructure:
@@ -148,7 +148,7 @@ TODO: docs/src/ is missing; add the folder or update/remove the src links below.
- [src/types/](./src/types/) - Type definitions
### 🏁 DBAL (Database Abstraction Layer)
- [dbal/README.md](./dbal/README.md) - Overview
- [dbal/docs/README.md](./dbal/docs/README.md) - Overview
- [dbal/shared/api/README.md](./dbal/shared/api/README.md) - API docs
- [dbal/shared/backends/README.md](./dbal/shared/backends/README.md) - Backends
- [dbal/production/README.md](./dbal/production/README.md) - C++ implementation

4
docs/reference/indexes/DOCUMENTATION_FINDINGS.md
View File

@@ -632,11 +632,11 @@ TODO: Core doc links below point to docs/reference; update to correct paths unde
| File | Purpose |
|------|---------|
| [dbal/README.md](../dbal/README.md) | DBAL architecture and overview |
| [dbal/docs/README.md](../dbal/docs/README.md) | DBAL architecture and overview |
| [dbal/QUICK_START.md](../dbal/QUICK_START.md) | DBAL 5-minute quick start |
| [dbal/PROJECT.md](../dbal/PROJECT.md) | DBAL project structure |
| [dbal/IMPLEMENTATION_SUMMARY.md](../dbal/IMPLEMENTATION_SUMMARY.md) | Phase 2 DBAL implementation summary |
| [dbal/AGENTS.md](../dbal/AGENTS.md) | Agent development guide for DBAL |
| [dbal/docs/AGENTS.md](../dbal/docs/AGENTS.md) | Agent development guide for DBAL |
| [dbal/PHASE2_COMPLETE.md](../dbal/PHASE2_COMPLETE.md) | Phase 2 completion status |
### Development Guides