Add npm scripts and comprehensive documentation for TODO to issues conversion

Co-authored-by: johndoe6345789 <224850594+johndoe6345789@users.noreply.github.com>
2026-04-24 13:54:57 +00:00 · 2025-12-27 04:05:57 +00:00
parent e79ea8564a
commit 08513ab8a3
3 changed files with 389 additions and 6 deletions
--- a/docs/guides/TODO_TO_ISSUES.md
+++ b/docs/guides/TODO_TO_ISSUES.md
@@ -0,0 +1,300 @@
+# Converting TODO Items to GitHub Issues
+
+This guide explains how to convert TODO items from `docs/todo/` markdown files into GitHub issues.
+
+## Overview
+
+The MetaBuilder repository contains 775+ TODO items organized across 20+ markdown files in `docs/todo/`. The `populate-kanban.py` script can parse these files and create GitHub issues automatically.
+
+## Quick Start
+
+### Using npm Scripts (Recommended)
+
+From the repository root:
+
+```bash
+# Preview first 10 issues that would be created
+npm run todos:preview
+
+# Run tests to verify the script works
+npm run todos:test
+
+# Export all TODOs to JSON for review
+npm run todos:export
+
+# Export only critical priority items
+npm run todos:export-critical
+
+# Export with checklist items excluded
+npm run todos:export-filtered
+
+# Show all available options
+npm run todos:help
+```
+
+### Creating Issues on GitHub
+
+**⚠️ Warning**: This will create 775 issues (or fewer if filtered). Make sure you're ready!
+
+```bash
+# Authenticate with GitHub CLI first
+gh auth login
+
+# Preview what will be created (dry-run)
+python3 tools/project-management/populate-kanban.py --dry-run --limit 10
+
+# Create all issues (takes 15-20 minutes)
+npm run todos:create
+
+# Or create with filters
+python3 tools/project-management/populate-kanban.py --create --filter-priority critical
+python3 tools/project-management/populate-kanban.py --create --filter-label security --limit 20
+python3 tools/project-management/populate-kanban.py --create --exclude-checklist
+```
+
+## Features
+
+### Filtering Options
+
+#### By Priority
+
+```bash
+# Critical items only (40 items)
+python3 tools/project-management/populate-kanban.py --filter-priority critical --output critical.json
+
+# High priority items (386 items)
+python3 tools/project-management/populate-kanban.py --filter-priority high --output high.json
+
+# Medium priority items (269 items)
+python3 tools/project-management/populate-kanban.py --filter-priority medium --output medium.json
+
+# Low priority items (80 items)
+python3 tools/project-management/populate-kanban.py --filter-priority low --output low.json
+```
+
+#### By Label
+
+```bash
+# Security-related items
+python3 tools/project-management/populate-kanban.py --filter-label security --output security.json
+
+# DBAL items
+python3 tools/project-management/populate-kanban.py --filter-label dbal --output dbal.json
+
+# Frontend items
+python3 tools/project-management/populate-kanban.py --filter-label frontend --output frontend.json
+```
+
+#### Exclude Checklist Items
+
+Some TODO files contain checklist items like "Done Criteria" that are more like templates than actual tasks. Exclude them:
+
+```bash
+# Excludes items from sections: Done Criteria, Quick Wins, Sanity Check, Checklist
+python3 tools/project-management/populate-kanban.py --exclude-checklist --output filtered.json
+# This reduces 775 items to ~763 items
+```
+
+### Combining Filters
+
+```bash
+# Critical security items only
+python3 tools/project-management/populate-kanban.py \
+  --filter-priority critical \
+  --filter-label security \
+  --output critical-security.json
+
+# High priority frontend items, excluding checklists
+python3 tools/project-management/populate-kanban.py \
+  --filter-priority high \
+  --filter-label frontend \
+  --exclude-checklist \
+  --output high-frontend.json
+```
+
+## What Gets Created
+
+Each GitHub issue includes:
+
+- **Title**: First 100 characters of the TODO item
+- **Body**: 
+  - File path where TODO is located
+  - Section within that file
+  - Line number
+  - Context (nearby TODO items)
+  - The full TODO text
+- **Labels**: Automatically assigned based on file location and name
+  - Category labels: `core`, `infrastructure`, `feature`, `enhancement`
+  - Domain labels: `dbal`, `frontend`, `backend`, `security`, `database`, etc.
+  - Priority label: `🔴 Critical`, `🟠 High`, `🟡 Medium`, or `🟢 Low`
+
+### Example Issue
+
+**Title**: `Add password strength requirements`
+
+**Body**:
+```markdown
+**File:** `docs/todo/infrastructure/10-SECURITY-TODO.md`
+**Section:** Authentication
+**Line:** 11
+
+**Context:**
+- [x] Add unit tests for security-scanner.ts ✅ (24 parameterized tests)
+- [ ] Implement secure password hashing (verify SHA-512 implementation)
+- [ ] Add password strength requirements
+
+**Task:** Add password strength requirements
+```
+
+**Labels**: `security`, `infrastructure`, `🔴 Critical`
+
+## Statistics
+
+Total items by category:
+- **Total**: 775 items
+- **Critical**: 40 items (5%)
+- **High**: 386 items (50%)
+- **Medium**: 269 items (35%)
+- **Low**: 80 items (10%)
+
+Top labels:
+1. `feature` (292) - New features
+2. `workflow` (182) - SDLC improvements
+3. `core` (182) - Core functionality
+4. `enhancement` (160) - Improvements
+5. `infrastructure` (141) - DevOps
+
+## Testing
+
+Run the test suite to verify everything works:
+
+```bash
+npm run todos:test
+```
+
+This runs 15 unit tests covering:
+- Parsing TODO items from markdown
+- Priority assignment
+- Label categorization
+- Filtering logic
+- File exclusion rules
+- Context extraction
+
+## Advanced Usage
+
+### Export to JSON for Manual Review
+
+```bash
+# Export all items
+npm run todos:export
+
+# Review the JSON
+cat todos.json | jq '.[0]'
+
+# Count items by priority
+cat todos.json | jq '[.[] | .priority] | group_by(.) | map({priority: .[0], count: length})'
+
+# Filter in JSON with jq
+cat todos.json | jq '[.[] | select(.priority == "🔴 Critical")]' > critical-only.json
+```
+
+### Batch Creation
+
+To avoid rate limiting, create issues in batches:
+
+```bash
+# First 50 items
+python3 tools/project-management/populate-kanban.py --create --limit 50
+
+# Wait a few minutes, then continue with next batch
+# Note: Will create duplicates of first 50, so track carefully!
+```
+
+Better approach - create filtered sets:
+
+```bash
+# Step 1: Create critical items
+python3 tools/project-management/populate-kanban.py --create --filter-priority critical
+
+# Step 2: Create high priority items
+python3 tools/project-management/populate-kanban.py --create --filter-priority high
+
+# And so on...
+```
+
+### Add to GitHub Project
+
+If you have a GitHub project board:
+
+```bash
+# Find your project ID
+gh project list --owner johndoe6345789
+
+# Create issues and add to project
+python3 tools/project-management/populate-kanban.py --create --project-id 2
+```
+
+## Troubleshooting
+
+### GitHub CLI Not Authenticated
+
+```bash
+gh auth status
+# If not authenticated:
+gh auth login
+```
+
+### Rate Limiting
+
+GitHub has rate limits. If you hit them:
+- Wait 15-30 minutes
+- Use `--limit` to create fewer issues at once
+- Use filters to create smaller batches
+
+### Duplicate Issues
+
+If you accidentally create duplicates:
+```bash
+# List recent issues
+gh issue list --limit 100
+
+# Close duplicates
+gh issue close 123 --reason "duplicate"
+```
+
+### Testing Without Creating
+
+Always use `--dry-run` first:
+```bash
+python3 tools/project-management/populate-kanban.py --dry-run --limit 5
+```
+
+## Updating TODOs After Creating Issues
+
+After creating GitHub issues, you can:
+
+1. **Mark TODOs as done** with issue reference:
+   ```markdown
+   - [x] Add password strength requirements (#123)
+   ```
+
+2. **Update TODO with issue link**:
+   ```markdown
+   - [ ] Add password strength requirements (see issue #123)
+   ```
+
+3. **Remove TODO** (since it's now tracked as an issue):
+   - Delete the line from the TODO file
+   - Run `npm run todos:scan` to update reports
+
+## Related Documentation
+
+- [KANBAN_READY.md](/KANBAN_READY.md) - Original implementation documentation
+- [tools/project-management/README.md](/tools/project-management/README.md) - Script technical reference
+- [docs/todo/README.md](/docs/todo/README.md) - TODO organization guide
+
+## See Also
+
+- [GitHub CLI documentation](https://cli.github.com/manual/)
+- [GitHub Projects documentation](https://docs.github.com/en/issues/planning-and-tracking-with-projects)
+- [Markdown checklist syntax](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/about-task-lists)
--- a/package.json
+++ b/package.json
@@ -1,4 +1,14 @@
 {
+  "scripts": {
+    "todos:scan": "python3 docs/todo/scans/scan-project-todos.py",
+    "todos:preview": "python3 tools/project-management/populate-kanban.py --dry-run --limit 10",
+    "todos:export": "python3 tools/project-management/populate-kanban.py --output todos.json",
+    "todos:export-critical": "python3 tools/project-management/populate-kanban.py --output todos-critical.json --filter-priority critical",
+    "todos:export-filtered": "python3 tools/project-management/populate-kanban.py --output todos-filtered.json --exclude-checklist",
+    "todos:test": "python3 tools/project-management/test_populate_kanban.py",
+    "todos:create": "python3 tools/project-management/populate-kanban.py --create",
+    "todos:help": "python3 tools/project-management/populate-kanban.py --help"
+  },
  "devDependencies": {
    "prisma": "^7.2.0"
  },
--- a/tools/project-management/README.md
+++ b/tools/project-management/README.md
@@ -61,6 +61,9 @@ python3 tools/project-management/populate-kanban.py --create --project-id 2
 | `--repo OWNER/NAME` | Target repository | johndoe6345789/metabuilder |
 | `--todo-dir PATH` | Path to TODO directory | auto-detect |
 | `--limit N` | Limit number of issues to create | no limit |
+| `--filter-priority LEVEL` | Filter by priority: critical, high, medium, low | - |
+| `--filter-label LABEL` | Filter by label (e.g., security, frontend) | - |
+| `--exclude-checklist` | Exclude checklist items from "Done Criteria" sections | - |

 #### Examples

@@ -90,14 +93,37 @@ python3 populate-kanban.py --create
 python3 populate-kanban.py --create --project-id 2
 ```

-**5. Create only high priority issues**
+**5. Filter by priority level**
 ```bash
-# First export to JSON
-python3 populate-kanban.py --output all-issues.json
+# Create only critical priority issues (40 items)
+python3 populate-kanban.py --create --filter-priority critical

-# Filter and create (requires jq)
-cat all-issues.json | jq '[.[] | select(.priority == "🟠 High")]' > high-priority.json
-# Then manually create from filtered JSON
+# Create only high priority issues (386 items)
+python3 populate-kanban.py --create --filter-priority high
+```
+
+**6. Filter by label**
+```bash
+# Create only security-related issues
+python3 populate-kanban.py --create --filter-label security
+
+# Create only frontend issues
+python3 populate-kanban.py --create --filter-label frontend
+```
+
+**7. Exclude checklist items**
+```bash
+# Exclude "Done Criteria" and similar checklist sections (reduces to ~763 items)
+python3 populate-kanban.py --create --exclude-checklist
+```
+
+**8. Combine filters**
+```bash
+# Critical security issues only
+python3 populate-kanban.py --create --filter-priority critical --filter-label security
+
+# High priority frontend issues, excluding checklists
+python3 populate-kanban.py --create --filter-priority high --filter-label frontend --exclude-checklist
 ```

 ### Output Format
@@ -197,6 +223,53 @@ The script does not check for existing issues. Before running `--create`, either

 ### Development

+#### Running Tests
+
+The script includes comprehensive unit tests:
+
+```bash
+# Run all tests
+python3 test_populate_kanban.py
+
+# Or use npm script (from repository root)
+npm run todos:test
+```
+
+Tests cover:
+- Parsing TODO items from markdown files
+- Priority assignment logic
+- Label categorization
+- Filtering functionality
+- Context extraction
+- Edge cases and error handling
+
+#### NPM Scripts
+
+For convenience, add these to your workflow (from repository root):
+
+```bash
+# Preview issues
+npm run todos:preview
+
+# Run tests
+npm run todos:test
+
+# Export all TODOs
+npm run todos:export
+
+# Export only critical items
+npm run todos:export-critical
+
+# Export with filters
+npm run todos:export-filtered
+
+# Create issues on GitHub
+npm run todos:create
+
+# Show help
+npm run todos:help
+```
+
 #### Adding new label categories

 Edit the `_categorize_file()` method in `TodoParser` class: