git/metabuilder
1
0
Fork 0
mirror of https://github.com/johndoe6345789/metabuilder.git synced 2026-04-24 13:54:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into copilot/create-issue-and-pr-templates

Browse Source
This commit is contained in:
johndoe6345789
2025-12-27 03:58:28 +00:00
committed by GitHub
parent 75e4cea3cd 666f548088
commit 8a35a71d7d
7 changed files with 1385 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

196
KANBAN_READY.md Normal file
View File

@@ -0,0 +1,196 @@
# 🎯 READY TO POPULATE KANBAN
## ✅ Implementation Complete
All tools and documentation are ready to populate your GitHub kanban board at:
**https://github.com/users/johndoe6345789/projects/2**
---
## 📦 What's Been Created
### Scripts
- ✅ **`tools/project-management/populate-kanban.py`** - Main script (775 TODO items ready)
### Documentation
- ✅ **`docs/guides/POPULATE_KANBAN.md`** - Step-by-step user guide
- ✅ **`docs/guides/KANBAN_IMPLEMENTATION_SUMMARY.md`** - Complete overview
- ✅ **`tools/project-management/README.md`** - Detailed script reference
- ✅ **`tools/README.md`** - Updated with project management section
---
## 🚀 Quick Start (3 Steps)
### Step 1: Authenticate with GitHub CLI
```bash
gh auth login
```
Choose:
- GitHub.com
- HTTPS protocol
- Login with web browser
### Step 2: Preview Issues (Recommended)
```bash
cd /path/to/metabuilder
python3 tools/project-management/populate-kanban.py --dry-run --limit 10
```
This shows you what the first 10 issues will look like.
### Step 3: Populate the Kanban
**⚠️ Warning**: This will create 775 issues and take 15-20 minutes.
```bash
python3 tools/project-management/populate-kanban.py --create --project-id 2
```
---
## 📊 What Gets Created
### Statistics
- **Total Issues**: 775
- **By Priority**:
- 🔴 Critical: 40 (5%)
- 🟠 High: 386 (50%)
- 🟡 Medium: 269 (35%)
- 🟢 Low: 80 (10%)
### Top Categories
1. **feature** (292) - New features
2. **workflow** (182) - SDLC improvements
3. **core** (182) - Core functionality
4. **enhancement** (160) - Improvements
5. **infrastructure** (141) - DevOps
### Example Issue
**Title**: `npm run typecheck`
**Body**:
```markdown
**File:** `docs/todo/core/0-kickstart.md`
**Section:** 15-Minute Local Sanity Check (Frontend)
**Line:** 33
**Task:** `npm run typecheck`
```
**Labels**: `workflow`, `core`, `🟠 High`
---
## 📚 Documentation Guide
### For Quick Start
👉 Read: **`docs/guides/POPULATE_KANBAN.md`**
### For Detailed Reference
👉 Read: **`tools/project-management/README.md`**
### For Complete Overview
👉 Read: **`docs/guides/KANBAN_IMPLEMENTATION_SUMMARY.md`**
---
## ⚙️ Advanced Options
### Export to JSON First (Recommended)
```bash
python3 tools/project-management/populate-kanban.py --output issues.json
# Review the JSON, then create
python3 tools/project-management/populate-kanban.py --create
```
### Create Only Critical Issues
```bash
python3 tools/project-management/populate-kanban.py --output all.json
cat all.json | jq '[.[] | select(.priority == "🔴 Critical")]' > critical.json
# Then manually create from critical.json (40 issues)
```
### Create in Batches
```bash
# First 50
python3 tools/project-management/populate-kanban.py --create --limit 50
# Wait, then run again (note: will create duplicates, so use limit carefully)
```
---
## ✅ Verification
Test the script is working:
```bash
# 1. Check help
python3 tools/project-management/populate-kanban.py --help
# 2. Dry run with 3 issues
python3 tools/project-management/populate-kanban.py --dry-run --limit 3
# 3. Export sample to JSON
python3 tools/project-management/populate-kanban.py --output /tmp/test.json --limit 5
cat /tmp/test.json | jq '.[0]'
```
All tests should complete successfully! ✅
---
## 🔧 Troubleshooting
### Not Authenticated?
```bash
gh auth status
gh auth login
```
### Project Not Found?
```bash
# List your projects
gh project list --owner johndoe6345789
# Use the correct ID
python3 populate-kanban.py --create --project-id <correct-id>
```
### Rate Limited?
The script includes automatic pausing. If you still hit limits:
- Wait 15-30 minutes
- Use `--limit` to create fewer at once
---
## 📋 Next Steps After Population
Once issues are created:
1. **Organize** - Use project board columns (Backlog, In Progress, Done)
2. **Triage** - Review and adjust priorities as needed
3. **Assign** - Assign issues to team members
4. **Milestone** - Group issues for releases
5. **Labels** - Add custom labels (bug, etc.) if needed
---
## 🎉 You're Ready!
All tools are tested and working. The kanban board is ready to be populated with 775 issues organized by priority and category.
**Need help?** Check the documentation files listed above.
**Ready to go?** Run the 3 steps in "Quick Start" above! 🚀
---
**Status**: ✅ READY TO USE
**Issues Ready**: 775
**Target Board**: https://github.com/users/johndoe6345789/projects/2
**Estimated Time**: 15-20 minutes

2
README.md
View File

@@ -2,6 +2,8 @@
A **data-driven, multi-tenant platform** where 95% of functionality lives in JSON/Lua, not TypeScript. Build enterprise applications declaratively with a 6-level permission system.
> 📋 **Ready to populate the kanban?** See [KANBAN_READY.md](KANBAN_READY.md) for a quick guide to populate the GitHub project board with 775 TODO items.
---
## Table of Contents

242
docs/guides/KANBAN_IMPLEMENTATION_SUMMARY.md Normal file
View File

@@ -0,0 +1,242 @@
# Kanban Population - Implementation Summary
## Overview
This implementation provides a complete solution for populating the MetaBuilder GitHub project kanban board (https://github.com/users/johndoe6345789/projects/2) with issues extracted from the TODO markdown files.
## What Was Created
### 1. Main Script: `tools/project-management/populate-kanban.py`
A comprehensive Python script that:
- Parses all TODO markdown files in `docs/todo/`
- Extracts 775 unchecked TODO items
- Categorizes them with appropriate labels
- Assigns priorities based on file mappings
- Creates GitHub issues via GitHub CLI
- Adds issues to the specified project board
**Key Features:**
- ✅ Dry-run mode for previewing issues
- ✅ JSON export for review before creation
- ✅ Automatic label categorization (14+ label types)
- ✅ Priority assignment (Critical, High, Medium, Low)
- ✅ Rate limiting protection
- ✅ Batch processing support
- ✅ Context preservation (file, section, line number)
### 2. Documentation
**User Guide**: `docs/guides/POPULATE_KANBAN.md`
- Step-by-step instructions
- Quick start (3 steps)
- Alternative approaches
- Troubleshooting guide
- Cleanup instructions
**Script Documentation**: `tools/project-management/README.md`
- Detailed feature list
- Usage examples
- Option reference
- Label categories
- Priority mapping
- Development guide
**Tools README**: `tools/README.md`
- Updated with project management section
- Quick reference for common tasks
## Statistics
### TODO Items Parsed
- **Total**: 775 unchecked items
- **Source files**: 20 TODO markdown files
### By Priority
- 🔴 **Critical**: 40 items (5.2%)
- 🟠 **High**: 386 items (49.8%)
- 🟡 **Medium**: 269 items (34.7%)
- 🟢 **Low**: 80 items (10.3%)
### By Category
| Label | Count | Description |
|-------|-------|-------------|
| feature | 292 | New features to implement |
| workflow | 182 | SDLC and workflow improvements |
| core | 182 | Core functionality |
| enhancement | 160 | Improvements to existing features |
| infrastructure | 141 | Infrastructure and DevOps |
| sdlc | 127 | Software development lifecycle |
| database | 61 | Database and schema work |
| documentation | 59 | Documentation improvements |
| packages | 50 | Package system work |
| ui | 45 | User interface changes |
## How to Use
### Prerequisites
1. Python 3.7+
2. GitHub CLI (`gh`) installed
3. GitHub authentication
### Quick Start
```bash
# Step 1: Authenticate (first time only)
gh auth login
# Step 2: Preview issues (recommended)
cd /path/to/metabuilder
python3 tools/project-management/populate-kanban.py --dry-run --limit 10
# Step 3: Create all issues
python3 tools/project-management/populate-kanban.py --create --project-id 2
```
**Estimated time**: 15-20 minutes for all 775 issues
### Options
```bash
# Dry run (no changes)
python3 populate-kanban.py --dry-run
# Export to JSON for review
python3 populate-kanban.py --output issues.json
# Create with limit
python3 populate-kanban.py --create --limit 50
# Specify different project
python3 populate-kanban.py --create --project-id <id>
```
## Verification
To verify the script is working:
```bash
# 1. Check help
python3 tools/project-management/populate-kanban.py --help
# 2. Run dry-run with small limit
python3 tools/project-management/populate-kanban.py --dry-run --limit 3
# 3. Export to JSON and inspect
python3 tools/project-management/populate-kanban.py --output /tmp/test.json --limit 5
cat /tmp/test.json | jq '.[0]'
```
## Example Issue
When created, issues will look like:
**Title**: `npm run typecheck`
**Body**:
```markdown
**File:** `docs/todo/core/0-kickstart.md`
**Section:** 15-Minute Local Sanity Check (Frontend)
**Line:** 33
**Task:** `npm run typecheck`
```
**Labels**: `workflow`, `core`, `🟠 High`
## Label Mapping
The script automatically assigns labels based on:
### Directory Structure
- `docs/todo/core/` → `core`, `workflow`
- `docs/todo/infrastructure/` → `infrastructure`
- `docs/todo/features/` → `feature`
- `docs/todo/improvements/` → `enhancement`
### Filename Patterns
- Contains `dbal` → `dbal`
- Contains `frontend` → `frontend`
- Contains `security` → `security`
- Contains `test` → `testing`
- Contains `database` → `database`
- Contains `deploy` → `deployment`
- Contains `doc` → `documentation`
- And more...
### Priority Levels
Based on README.md priority table and filename patterns:
- 🔴 Critical: security issues, build fixes
- 🟠 High: dbal, frontend, database, sdlc
- 🟡 Medium: general features
- 🟢 Low: future features, copilot
## Files Modified/Created
```
tools/project-management/
├── populate-kanban.py # Main script (NEW)
└── README.md # Script documentation (NEW)
docs/guides/
└── POPULATE_KANBAN.md # User guide (NEW)
tools/
└── README.md # Updated with project mgmt section
```
## Next Steps
1. **Authenticate**: Run `gh auth login` if not already authenticated
2. **Review**: Run with `--dry-run` to preview issues
3. **Create**: Run with `--create --project-id 2` to populate the kanban
4. **Organize**: Use GitHub project board to organize issues into columns
5. **Triage**: Review and adjust priorities/labels as needed
6. **Assign**: Assign issues to team members
7. **Milestone**: Group issues into milestones for releases
## Troubleshooting
### Authentication Issues
```bash
gh auth status
gh auth login
```
### Rate Limiting
The script includes automatic pausing. If still hitting limits:
- Use `--limit` to create fewer issues at once
- Wait 15-30 minutes between batches
### Project Not Found
Verify project ID:
```bash
gh project list --owner johndoe6345789
```
## Future Enhancements
Possible improvements for future versions:
- [ ] Skip already-created issues (check existing by title)
- [ ] Support for updating existing issues
- [ ] Batch creation with offset/skip support
- [ ] Custom label mapping configuration file
- [ ] Integration with GitHub Projects v2 API
- [ ] Progress bar for long-running operations
- [ ] Parallel issue creation (with rate limiting)
- [ ] Issue templates support
## References
- [GitHub Project Board](https://github.com/users/johndoe6345789/projects/2)
- [TODO System](../../docs/todo/README.md)
- [TODO Status](../../docs/todo/TODO_STATUS.md)
- [Script Documentation](../../tools/project-management/README.md)
- [User Guide](../../docs/guides/POPULATE_KANBAN.md)
---
**Created**: 2025-12-27
**Script Version**: 1.0
**Issues Ready**: 775
**Status**: ✅ Ready to use

199
docs/guides/POPULATE_KANBAN.md Normal file
View File

@@ -0,0 +1,199 @@
# Populating the GitHub Kanban Board
This guide shows how to populate the MetaBuilder GitHub project board (https://github.com/users/johndoe6345789/projects/2) with issues from the TODO files.
## Quick Start (3 Steps)
### Step 1: Authenticate with GitHub
```bash
gh auth login
```
Follow the prompts to authenticate. Choose:
- GitHub.com (not GitHub Enterprise)
- HTTPS protocol
- Login with a web browser (recommended)
### Step 2: Preview Issues (Recommended)
Before creating hundreds of issues, do a dry run:
```bash
cd /path/to/metabuilder
python3 tools/project-management/populate-kanban.py --dry-run --limit 10
```
This shows you what the first 10 issues would look like.
### Step 3: Create All Issues
**Warning**: This creates 775 issues and will take 15-20 minutes.
```bash
python3 tools/project-management/populate-kanban.py --create --project-id 2
```
The script will:
1. Parse all TODO files in `docs/todo/`
2. Create a GitHub issue for each unchecked TODO item
3. Add appropriate labels (core, infrastructure, feature, etc.)
4. Add priority labels (🔴 Critical, 🟠 High, 🟡 Medium, 🟢 Low)
5. Add each issue to project #2
6. Pause periodically to avoid rate limiting
## Alternative Approaches
### Create Issues in Batches
If you want more control, create issues in smaller batches:
```bash
# Export all issues to JSON first
python3 tools/project-management/populate-kanban.py --output all-issues.json
# Then create in batches using the --limit flag
# First 100 issues
python3 tools/project-management/populate-kanban.py --create --limit 100
# Wait a bit, then continue...
# (Note: You'd need to modify the script to support offset/skip)
```
### Create Only High Priority Issues
```bash
# First export to JSON
python3 tools/project-management/populate-kanban.py --output all-issues.json
# Filter for critical and high priority
cat all-issues.json | jq '[.[] | select(.priority == "🔴 Critical" or .priority == "🟠 High")]' > high-priority.json
# This gives you 426 high-priority items
# You'll need to write a small script to create from this filtered JSON
```
### Manual Review Before Creating
```bash
# Export to JSON
python3 tools/project-management/populate-kanban.py --output issues.json
# Review the issues
cat issues.json | jq '.[] | {title, priority, labels, file}' | less
# If satisfied, create them
python3 tools/project-management/populate-kanban.py --create
```
## What Gets Created
Each TODO item from the markdown files becomes a GitHub issue with:
**Title**: The TODO text (truncated to 100 chars if needed)
**Body**: Contains:
- Source file path
- Section within the file
- Line number
- Context from surrounding items
- The full task description
**Labels**: Automatically assigned based on:
- File location (core, infrastructure, features, improvements)
- Filename content (dbal, frontend, security, testing, etc.)
- Priority level from README
**Example Issue**:
```markdown
Title: Run `npm run act:diagnose` and `npm run act` to validate local GitHub Actions testing
Body:
**File:** `docs/todo/core/1-TODO.md`
**Section:** Quick Wins
**Line:** 7
**Task:** Run `npm run act:diagnose` and `npm run act` to validate local GitHub Actions testing
Labels: workflow, core, 🟠 High
```
## Statistics
The script will create approximately:
- **Total issues**: 775
- **By priority**:
- 🔴 Critical: 40 issues
- 🟠 High: 386 issues
- 🟡 Medium: 269 issues
- 🟢 Low: 80 issues
- **By category**:
- Feature: 292 issues
- Workflow/Core: 182 issues
- Enhancement: 160 issues
- Infrastructure: 141 issues
- SDLC: 127 issues
- And more...
## Troubleshooting
### "You are not logged into any GitHub hosts"
**Solution**: Run `gh auth login` first
### "Rate limit exceeded"
**Solution**: The script includes automatic pausing. If you still hit limits:
1. Wait 15-30 minutes
2. Use `--limit` to create fewer issues at once
### "Project not found"
**Solution**: Verify the project ID:
```bash
gh project list --owner johndoe6345789
```
Then use the correct ID:
```bash
python3 tools/project-management/populate-kanban.py --create --project-id <correct-id>
```
### Want to test first?
Always safe to use `--dry-run`:
```bash
python3 tools/project-management/populate-kanban.py --dry-run
```
## Cleanup (If Needed)
If you need to remove issues:
```bash
# List all open issues
gh issue list --repo johndoe6345789/metabuilder --limit 1000
# Close issues in bulk (requires jq)
gh issue list --repo johndoe6345789/metabuilder --limit 1000 --json number \
| jq -r '.[] | .number' \
| xargs -I {} gh issue close {} --repo johndoe6345789/metabuilder
```
## Next Steps After Population
Once issues are created:
1. **Triage**: Review and adjust priorities/labels as needed
2. **Milestones**: Group issues into milestones for releases
3. **Assignments**: Assign issues to team members
4. **Project board**: Organize columns (Backlog, In Progress, Done)
5. **Labels**: Add custom labels if needed (bug, documentation, etc.)
## See Also
- [tools/project-management/README.md](../tools/project-management/README.md) - Detailed script documentation
- [docs/todo/README.md](../docs/todo/README.md) - TODO system overview
- [GitHub Project](https://github.com/users/johndoe6345789/projects/2) - Target kanban board

17
tools/README.md
View File

@@ -28,8 +28,25 @@ Scripts organized by function:
- **`diagnose-workflows.sh`** - Diagnose workflow issues
- **`capture-screenshot.ts`** - Capture UI screenshots
### Project Management
- **`project-management/populate-kanban.py`** - Populate GitHub project board from TODO files
## 🚀 Common Tasks
### Populate GitHub Project Kanban
```bash
# Preview what issues would be created
python3 tools/project-management/populate-kanban.py --dry-run --limit 10
# Create all issues and add to project board
gh auth login # First time only
python3 tools/project-management/populate-kanban.py --create --project-id 2
```
See [project-management/README.md](project-management/README.md) for detailed documentation.
### Check Documentation Quality
```bash

244
tools/project-management/README.md Normal file
View File

@@ -0,0 +1,244 @@
# Project Management Tools
Tools for managing MetaBuilder's GitHub project board and issues.
## populate-kanban.py
Automatically populate the GitHub project kanban board from TODO markdown files.
### Features
- **Parse TODO files**: Extracts unchecked TODO items from all markdown files in `docs/todo/`
- **Categorize items**: Automatically assigns labels based on file location and content
- **Priority assignment**: Maps TODO files to priority levels (Critical, High, Medium, Low)
- **GitHub integration**: Creates issues directly via GitHub CLI
- **Project board support**: Automatically adds issues to specified GitHub project
- **Flexible output**: Export to JSON or create issues directly
### Quick Start
```bash
# Preview what issues would be created (dry run)
python3 tools/project-management/populate-kanban.py --dry-run --limit 10
# Export all issues to JSON file
python3 tools/project-management/populate-kanban.py --output issues.json
# Create issues on GitHub (requires authentication)
gh auth login
python3 tools/project-management/populate-kanban.py --create
# Create issues and add to project board
python3 tools/project-management/populate-kanban.py --create --project-id 2
```
### Prerequisites
- **Python 3.7+**: Script requires Python 3.7 or higher
- **GitHub CLI**: Install `gh` for creating issues
```bash
# macOS
brew install gh
# Ubuntu/Debian
sudo apt install gh
# Windows
winget install GitHub.cli
```
- **Authentication**: Run `gh auth login` before creating issues
### Usage
#### Options
| Option | Description | Default |
|--------|-------------|---------|
| `--dry-run` | Preview issues without creating them | - |
| `--output FILE` | Export issues to JSON file | - |
| `--create` | Create issues on GitHub | - |
| `--project-id ID` | Add issues to GitHub project | 2 |
| `--repo OWNER/NAME` | Target repository | johndoe6345789/metabuilder |
| `--todo-dir PATH` | Path to TODO directory | auto-detect |
| `--limit N` | Limit number of issues to create | no limit |
#### Examples
**1. Test with small subset (recommended first)**
```bash
python3 populate-kanban.py --dry-run --limit 5
```
**2. Export all issues to review before creating**
```bash
python3 populate-kanban.py --output /tmp/issues.json
# Review the JSON file
cat /tmp/issues.json | jq '.[0]'
```
**3. Create all issues on GitHub**
```bash
# Authenticate first
gh auth login
# Create issues (this will take a while - 775 items!)
python3 populate-kanban.py --create
```
**4. Create issues and add to specific project**
```bash
python3 populate-kanban.py --create --project-id 2
```
**5. Create only high priority issues**
```bash
# First export to JSON
python3 populate-kanban.py --output all-issues.json
# Filter and create (requires jq)
cat all-issues.json | jq '[.[] | select(.priority == "🟠 High")]' > high-priority.json
# Then manually create from filtered JSON
```
### Output Format
The script generates the following statistics:
```
Total TODO items found: 775
Breakdown by priority:
🔴 Critical: 40
🟠 High: 386
🟡 Medium: 269
🟢 Low: 80
Breakdown by label:
feature: 292
workflow: 182
core: 182
enhancement: 160
infrastructure: 141
```
### Label Categories
Labels are automatically assigned based on:
1. **Directory structure**: `core/`, `infrastructure/`, `features/`, `improvements/`
2. **Filename patterns**: `dbal`, `frontend`, `security`, `testing`, etc.
3. **Priority levels**: 🔴 Critical, 🟠 High, 🟡 Medium, 🟢 Low
#### Standard Labels
| Label | Description | Source |
|-------|-------------|--------|
| `core` | Core functionality | `docs/todo/core/` |
| `infrastructure` | Infrastructure & DevOps | `docs/todo/infrastructure/` |
| `feature` | New features | `docs/todo/features/` |
| `enhancement` | Improvements | `docs/todo/improvements/` |
| `workflow` | SDLC & workflows | Filename contains workflow/sdlc |
| `dbal` | Database abstraction | Filename contains dbal |
| `frontend` | Frontend work | Filename contains frontend |
| `security` | Security issues | Filename contains security |
| `testing` | Test infrastructure | Filename contains test |
| `database` | Database work | Filename contains database/db |
| `deployment` | Deployment tasks | Filename contains deploy |
| `documentation` | Documentation | Filename contains doc |
### Priority Mapping
Priorities are assigned based on:
1. **README.md**: Quick Reference table priorities
2. **Filename patterns**:
- 🔴 **Critical**: security, build-fixes
- 🟠 **High**: dbal, frontend, database, sdlc
- 🟡 **Medium**: most feature work
- 🟢 **Low**: future features, copilot
### Issue Format
Each created issue includes:
```markdown
**File:** `docs/todo/core/1-TODO.md`
**Section:** Quick Wins
**Line:** 7
**Task:** Run `npm run act:diagnose` and `npm run act` to validate local GitHub Actions testing
```
### Troubleshooting
#### GitHub CLI not authenticated
```bash
gh auth status
# If not authenticated:
gh auth login
```
#### Rate limiting
The script includes automatic rate limiting (pauses every 10 issues). If you hit GitHub's rate limits:
1. Wait 15-30 minutes
2. Use `--limit` to process in smaller batches
3. Export to JSON first, then create issues in batches
#### Duplicate issues
The script does not check for existing issues. Before running `--create`, either:
1. Use `--dry-run` to preview
2. Export to JSON and review
3. Clear existing issues from the project first
### Development
#### Adding new label categories
Edit the `_categorize_file()` method in `TodoParser` class:
```python
if 'mypattern' in filename:
labels.append('mylabel')
```
#### Adjusting priority mapping
Edit the `_get_priority()` method in `TodoParser` class:
```python
if 'myfile' in filename.lower():
return '🔴 Critical'
```
#### Testing changes
```bash
# Always test with dry-run first
python3 populate-kanban.py --dry-run --limit 3
# Check JSON export
python3 populate-kanban.py --output /tmp/test.json --limit 5
cat /tmp/test.json | jq '.'
```
### Statistics
From the last full parse:
- **Total TODO files**: 20
- **Total TODO items**: 775 (845 with completed items)
- **Critical priority**: 40 items
- **High priority**: 386 items
- **Medium priority**: 269 items
- **Low priority**: 80 items
### See Also
- [docs/todo/README.md](../../docs/todo/README.md) - TODO system overview
- [docs/todo/TODO_STATUS.md](../../docs/todo/TODO_STATUS.md) - Current status
- [GitHub Projects](https://github.com/users/johndoe6345789/projects/2) - Target kanban board

485
tools/project-management/populate-kanban.py Executable file
View File

@@ -0,0 +1,485 @@
#!/usr/bin/env python3
"""
Populate GitHub Project Kanban from TODO files.
This script:
1. Parses all TODO markdown files in docs/todo/
2. Extracts unchecked TODO items
3. Generates GitHub issues with appropriate labels and metadata
4. Can output to JSON or directly create issues via GitHub CLI
Usage:
python3 populate-kanban.py --dry-run # Preview issues
python3 populate-kanban.py --output issues.json # Export to JSON
python3 populate-kanban.py --create # Create issues (requires gh auth)
python3 populate-kanban.py --project-id 2 # Add to specific project
"""
import argparse
import json
import os
import re
import subprocess
import sys
from pathlib import Path
from typing import Dict, List, Optional, Tuple
from dataclasses import dataclass, asdict
@dataclass
class TodoItem:
"""Represents a single TODO item."""
title: str
body: str
file: str
section: str
labels: List[str]
priority: str
line_number: int
class TodoParser:
"""Parse TODO markdown files and extract items."""
# Map TODO files to label categories
LABEL_MAP = {
'core': ['core', 'workflow'],
'infrastructure': ['infrastructure'],
'features': ['feature'],
'improvements': ['enhancement'],
}
# Priority mapping based on README
PRIORITY_MAP = {
'critical': '🔴 Critical',
'high': '🟠 High',
'medium': '🟡 Medium',
'low': '🟢 Low',
}
def __init__(self, todo_dir: Path):
self.todo_dir = todo_dir
self.readme_priorities = self._parse_readme_priorities()
def _parse_readme_priorities(self) -> Dict[str, str]:
"""Parse priority information from README.md."""
priorities = {}
readme_path = self.todo_dir / 'README.md'
if not readme_path.exists():
return priorities
with open(readme_path, 'r', encoding='utf-8') as f:
content = f.read()
# Extract priority from the Quick Reference table
pattern = r'\|\s*\[([^\]]+)\]\([^\)]+\)\s*\|[^|]*\|\s*(Critical|High|Medium|Low)\s*\|'
matches = re.findall(pattern, content, re.MULTILINE)
for filename, priority in matches:
priorities[filename] = priority.lower()
return priorities
def _categorize_file(self, filepath: Path) -> List[str]:
"""Determine labels based on file location and name."""
labels = []
# Get category from directory structure
relative = filepath.relative_to(self.todo_dir)
parts = relative.parts
if len(parts) > 1:
category = parts[0] # e.g., 'core', 'infrastructure', etc.
if category in self.LABEL_MAP:
labels.extend(self.LABEL_MAP[category])
# Add specific labels based on filename
filename = filepath.stem.lower()
if 'dbal' in filename:
labels.append('dbal')
if 'frontend' in filename:
labels.append('frontend')
if 'backend' in filename:
labels.append('backend')
if 'test' in filename:
labels.append('testing')
if 'security' in filename:
labels.append('security')
if 'database' in filename or 'db' in filename:
labels.append('database')
if 'deploy' in filename:
labels.append('deployment')
if 'doc' in filename:
labels.append('documentation')
if 'package' in filename:
labels.append('packages')
if 'lua' in filename:
labels.append('lua')
if 'multi-tenant' in filename:
labels.append('multi-tenant')
if 'ui' in filename or 'declarative' in filename:
labels.append('ui')
if 'accessibility' in filename or 'a11y' in filename:
labels.append('accessibility')
if 'performance' in filename:
labels.append('performance')
if 'copilot' in filename:
labels.append('copilot')
if 'sdlc' in filename:
labels.append('sdlc')
return list(set(labels)) # Remove duplicates
def _get_priority(self, filepath: Path) -> str:
"""Get priority level for a TODO file."""
filename = filepath.name
# Check if priority is defined in README
if filename in self.readme_priorities:
priority = self.readme_priorities[filename]
return self.PRIORITY_MAP.get(priority, '🟡 Medium')
# Default priorities based on patterns
if 'critical' in filename.lower() or 'security' in filename.lower():
return '🔴 Critical'
elif any(x in filename.lower() for x in ['build', 'dbal', 'frontend', 'database', 'sdlc']):
return '🟠 High'
elif 'future' in filename.lower() or 'copilot' in filename.lower():
return '🟢 Low'
else:
return '🟡 Medium'
def _parse_file(self, filepath: Path) -> List[TodoItem]:
"""Parse a single TODO markdown file."""
items = []
with open(filepath, 'r', encoding='utf-8') as f:
lines = f.readlines()
current_section = "General"
labels = self._categorize_file(filepath)
priority = self._get_priority(filepath)
for i, line in enumerate(lines, start=1):
# Track section headers
if line.startswith('#'):
# Extract section name
section_match = re.match(r'^#+\s+(.+)$', line.strip())
if section_match:
current_section = section_match.group(1)
continue
# Look for unchecked TODO items
todo_match = re.match(r'^(\s*)- \[ \]\s+(.+)$', line)
if todo_match:
indent, content = todo_match.groups()
# Skip if it's empty or just a placeholder
if not content.strip() or len(content.strip()) < 5:
continue
# Build context body
context_lines = []
# Look back for context (list items, sub-items)
for j in range(max(0, i-5), i):
prev_line = lines[j].strip()
if prev_line and not prev_line.startswith('#'):
if prev_line.startswith('- ['):
# Include related checked items for context
context_lines.append(prev_line)
elif prev_line.startswith('>'):
# Include blockquotes (often have important notes)
context_lines.append(prev_line)
# Build body with file reference and section
body_parts = [
f"**File:** `{filepath.relative_to(self.todo_dir.parent.parent)}`",
f"**Section:** {current_section}",
f"**Line:** {i}",
"",
]
if context_lines:
body_parts.append("**Context:**")
body_parts.extend(context_lines)
body_parts.append("")
body_parts.append(f"**Task:** {content}")
item = TodoItem(
title=content[:100] + ('...' if len(content) > 100 else ''),
body='\n'.join(body_parts),
file=str(filepath.relative_to(self.todo_dir.parent.parent)),
section=current_section,
labels=labels,
priority=priority,
line_number=i
)
items.append(item)
return items
def parse_all(self) -> List[TodoItem]:
"""Parse all TODO files in the directory."""
all_items = []
# Get all .md files except README, STATUS, and SCAN reports
todo_files_set = set()
for pattern in ['**/*TODO*.md', '**/[0-9]*.md']:
todo_files_set.update(self.todo_dir.glob(pattern))
# Filter out special files
exclude_patterns = ['README', 'STATUS', 'SCAN', 'REFACTOR']
todo_files = [
f for f in todo_files_set
if not any(pattern in f.name.upper() for pattern in exclude_patterns)
]
print(f"Found {len(todo_files)} TODO files to parse")
for filepath in sorted(todo_files):
print(f" Parsing {filepath.relative_to(self.todo_dir.parent.parent)}...")
items = self._parse_file(filepath)
all_items.extend(items)
print(f" Found {len(items)} items")
return all_items
class GitHubIssueCreator:
"""Create GitHub issues from TODO items."""
def __init__(self, repo: str, project_id: Optional[int] = None):
self.repo = repo
self.project_id = project_id
def _check_gh_auth(self) -> bool:
"""Check if GitHub CLI is authenticated."""
try:
result = subprocess.run(
['gh', 'auth', 'status'],
capture_output=True,
text=True
)
return result.returncode == 0
except FileNotFoundError:
print("ERROR: GitHub CLI (gh) not found. Please install it first.")
return False
def create_issue(self, item: TodoItem, dry_run: bool = False) -> Optional[str]:
"""Create a single GitHub issue."""
if dry_run:
print(f"\n[DRY RUN] Would create issue:")
print(f" Title: {item.title}")
print(f" Labels: {', '.join(item.labels + [item.priority])}")
print(f" Body preview: {item.body[:100]}...")
return None
# Build gh issue create command
cmd = [
'gh', 'issue', 'create',
'--repo', self.repo,
'--title', item.title,
'--body', item.body,
]
# Add labels
all_labels = item.labels + [item.priority]
for label in all_labels:
cmd.extend(['--label', label])
try:
result = subprocess.run(
cmd,
capture_output=True,
text=True,
check=True
)
issue_url = result.stdout.strip()
print(f" ✓ Created: {issue_url}")
# Add to project if specified
if self.project_id and issue_url:
self._add_to_project(issue_url)
return issue_url
except subprocess.CalledProcessError as e:
print(f" ✗ Failed to create issue: {e.stderr}")
return None
def _add_to_project(self, issue_url: str):
"""Add an issue to a GitHub project."""
try:
subprocess.run(
[
'gh', 'project', 'item-add', str(self.project_id),
'--owner', self.repo.split('/')[0],
'--url', issue_url
],
capture_output=True,
text=True,
check=True
)
print(f" ✓ Added to project {self.project_id}")
except subprocess.CalledProcessError as e:
print(f" ✗ Failed to add to project: {e.stderr}")
def create_all(self, items: List[TodoItem], dry_run: bool = False,
batch_size: int = 10) -> List[str]:
"""Create issues for all TODO items."""
if not dry_run and not self._check_gh_auth():
print("ERROR: GitHub CLI is not authenticated. Run 'gh auth login' first.")
return []
print(f"\nCreating {len(items)} issues...")
if dry_run:
print("\n[DRY RUN MODE - No issues will be created]")
created_urls = []
for i, item in enumerate(items, start=1):
print(f"\n[{i}/{len(items)}] {item.title[:60]}...")
url = self.create_issue(item, dry_run=dry_run)
if url:
created_urls.append(url)
# Pause after batch to avoid rate limiting
if not dry_run and i % batch_size == 0 and i < len(items):
print(f"\n Pausing after {batch_size} issues (rate limiting)...")
import time
time.sleep(2)
return created_urls
def main():
parser = argparse.ArgumentParser(
description='Populate GitHub Project Kanban from TODO files',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__
)
parser.add_argument(
'--dry-run',
action='store_true',
help='Preview issues without creating them'
)
parser.add_argument(
'--output',
type=str,
help='Export issues to JSON file instead of creating them'
)
parser.add_argument(
'--create',
action='store_true',
help='Create issues on GitHub (requires gh auth)'
)
parser.add_argument(
'--project-id',
type=int,
default=2,
help='GitHub project ID to add issues to (default: 2)'
)
parser.add_argument(
'--repo',
type=str,
default='johndoe6345789/metabuilder',
help='GitHub repository (default: johndoe6345789/metabuilder)'
)
parser.add_argument(
'--todo-dir',
type=Path,
help='Path to docs/todo directory (default: auto-detect)'
)
parser.add_argument(
'--limit',
type=int,
help='Limit number of issues to create (for testing)'
)
args = parser.parse_args()
# Auto-detect todo directory if not specified
if args.todo_dir is None:
script_dir = Path(__file__).parent
repo_root = script_dir.parent.parent
args.todo_dir = repo_root / 'docs' / 'todo'
if not args.todo_dir.exists():
print(f"ERROR: TODO directory not found: {args.todo_dir}")
sys.exit(1)
print(f"Parsing TODO files from: {args.todo_dir}")
# Parse TODO files
parser_obj = TodoParser(args.todo_dir)
items = parser_obj.parse_all()
if args.limit:
items = items[:args.limit]
print(f"\nLimited to first {args.limit} items")
print(f"\n{'='*60}")
print(f"Total TODO items found: {len(items)}")
print(f"{'='*60}")
# Output summary by priority
priority_counts = {}
for item in items:
priority_counts[item.priority] = priority_counts.get(item.priority, 0) + 1
print("\nBreakdown by priority:")
for priority in ['🔴 Critical', '🟠 High', '🟡 Medium', '🟢 Low']:
count = priority_counts.get(priority, 0)
if count > 0:
print(f" {priority}: {count}")
# Output summary by label
label_counts = {}
for item in items:
for label in item.labels:
label_counts[label] = label_counts.get(label, 0) + 1
print("\nBreakdown by label:")
for label, count in sorted(label_counts.items(), key=lambda x: -x[1])[:10]:
print(f" {label}: {count}")
# Export to JSON if requested
if args.output:
output_path = Path(args.output)
with open(output_path, 'w', encoding='utf-8') as f:
json.dump([asdict(item) for item in items], f, indent=2)
print(f"\n✓ Exported {len(items)} items to {output_path}")
return
# Create issues if requested
if args.create or args.dry_run:
creator = GitHubIssueCreator(args.repo, args.project_id)
created = creator.create_all(items, dry_run=args.dry_run)
if not args.dry_run:
print(f"\n{'='*60}")
print(f"Successfully created {len(created)} issues")
print(f"{'='*60}")
else:
print("\nTo create issues, run with --create or --dry-run")
print("Example: python3 populate-kanban.py --dry-run")
if __name__ == '__main__':
main()