diff --git a/deployment/README.md b/deployment/README.md new file mode 100644 index 000000000..cdbdde854 --- /dev/null +++ b/deployment/README.md @@ -0,0 +1,123 @@ +# MetaBuilder Deployment + +Build and deploy the full MetaBuilder stack locally using Docker. + +## Prerequisites + +- Docker Desktop with BuildKit enabled +- Bash 4+ (macOS: `brew install bash`) +- Add `localhost:5050` to Docker Desktop insecure registries: + Settings → Docker Engine → `"insecure-registries": ["localhost:5050"]` + +## Build & Deploy Order + +### Step 1 — Start Local Registries (Nexus + Artifactory) + +All base image builds pull dependencies through local registries. **Start these first.** + +```bash +cd deployment +docker compose -f docker-compose.nexus.yml up -d +``` + +Wait ~2 minutes for init containers to finish, then populate: + +```bash +./push-to-nexus.sh # Docker images → Nexus +./publish-npm-patches.sh # Patched npm packages → Nexus +conan remote add artifactory http://localhost:8092/artifactory/api/conan/conan-local +``` + +| Service | URL | Credentials | +|-------------|----------------------------------|--------------------| +| Nexus UI | http://localhost:8091 | admin / nexus | +| Artifactory | http://localhost:8092 | admin / password | +| npm group | http://localhost:8091/repository/npm-group/ | — | +| Conan2 | http://localhost:8092/artifactory/api/conan/conan-local | — | +| Docker repo | localhost:5050 | — | + +### Step 2 — Build Base Images + +```bash +./build-base-images.sh # Build all (skips existing) +./build-base-images.sh --force # Rebuild all +./build-base-images.sh node-deps # Build a specific image +./build-base-images.sh --list # List available images +``` + +Build order (dependencies respected automatically): + +1. `base-apt` — system packages (no deps) +2. `base-conan-deps` — C++ dependencies (needs base-apt) +3. `base-android-sdk` — Android SDK (needs base-apt) +4. `base-node-deps` — npm workspace dependencies (standalone, needs Nexus running) +5. `base-pip-deps` — Python dependencies (standalone) +6. `devcontainer` — full dev environment (needs all above) + +### Step 3 — Build App Images + +```bash +./build-apps.sh # Build all (skips existing) +./build-apps.sh --force # Rebuild all +./build-apps.sh workflowui # Build specific app +./build-apps.sh --sequential # Lower RAM usage +``` + +### Step 4 — Start the Stack + +```bash +./start-stack.sh # Core services +./start-stack.sh --monitoring # + Prometheus, Grafana, Loki +./start-stack.sh --media # + Media daemon, Icecast, HLS +./start-stack.sh --all # Everything +``` + +Portal: http://localhost (nginx welcome page with links to all apps) + +### Quick Deploy (rebuild + restart specific apps) + +```bash +./deploy.sh codegen # Build and deploy codegen +./deploy.sh codegen pastebin # Multiple apps +./deploy.sh --all # All apps +``` + +## Compose Files + +| File | Purpose | +|------|---------| +| `docker-compose.nexus.yml` | Local registries (Nexus + Artifactory) | +| `docker-compose.stack.yml` | Full application stack | +| `docker-compose.test.yml` | Integration test services | +| `docker-compose.smoke.yml` | Smoke test environment | + +## Scripts Reference + +| Script | Purpose | +|--------|---------| +| `build-base-images.sh` | Build base Docker images | +| `build-apps.sh` | Build app Docker images | +| `build-testcontainers.sh` | Build test container images | +| `start-stack.sh` | Start the full stack | +| `deploy.sh` | Quick build + deploy for specific apps | +| `push-to-nexus.sh` | Push Docker images to Nexus | +| `publish-npm-patches.sh` | Publish patched npm packages to Nexus | +| `populate-nexus.sh` | Populate Nexus with all artifacts | +| `nexus-init.sh` | Nexus repository setup (runs automatically) | +| `nexus-ci-init.sh` | Nexus setup for CI environments | +| `artifactory-init.sh` | Artifactory repository setup (runs automatically) | +| `release.sh` | Release workflow | + +## Troubleshooting + +**npm install fails with "proxy" error during base-node-deps build** +→ Nexus/Verdaccio isn't running. The `.npmrc` references `localhost:4873` for `@esbuild-kit` scoped packages. Start registries first (Step 1) or comment out the scoped registry in `.npmrc`. + +**Build takes 20+ minutes then fails on npm install** +→ Same as above. The Dockerfile now has a pre-flight registry check that fails fast with actionable instructions instead of retrying for 20 minutes. + +**Docker image push rejected** +→ Add `localhost:5050` to Docker Desktop insecure registries and restart Docker Desktop. + +**Nexus not ready after `docker compose up -d`** +→ Nexus takes ~2 minutes to start. The `nexus-init` container waits for the healthcheck automatically. Check with: `docker compose -f docker-compose.nexus.yml logs -f nexus-init` diff --git a/deployment/base-images/Dockerfile.node-deps b/deployment/base-images/Dockerfile.node-deps index bd2cb1526..43be7db47 100644 --- a/deployment/base-images/Dockerfile.node-deps +++ b/deployment/base-images/Dockerfile.node-deps @@ -55,9 +55,41 @@ COPY workflow/package.json ./workflow/ COPY scripts/patch-bundled-deps.sh ./scripts/ # Install all workspace deps (generates lock file from package.json manifests) +# +# Pre-flight: verify every registry in .npmrc is reachable before burning +# 20+ minutes on retries that will never succeed. RUN npm config set fetch-retries 5 \ && npm config set fetch-retry-mintimeout 20000 \ && npm config set maxsockets 5 \ + && echo "==> Checking npm registry connectivity..." \ + && registries=$(grep -E '^\s*(@[^:]+:)?registry=' .npmrc 2>/dev/null | sed 's/.*registry=//' | sort -u) \ + && for reg in $registries; do \ + printf " %-50s " "$reg"; \ + if wget -q --spider --timeout=5 "$reg" 2>/dev/null \ + || curl -sf --connect-timeout 5 "$reg" >/dev/null 2>&1; then \ + echo "OK"; \ + else \ + echo "UNREACHABLE"; \ + echo ""; \ + echo "========================================================"; \ + echo "ERROR: Cannot reach npm registry: $reg"; \ + echo "========================================================"; \ + echo ""; \ + echo "If this is a local registry (Verdaccio/Nexus/Artifactory),"; \ + echo "make sure it is running BEFORE building this image:"; \ + echo ""; \ + echo " Verdaccio: npx verdaccio --config deployment/verdaccio.yaml"; \ + echo " Nexus: cd deployment && docker compose -f docker-compose.nexus.yml up -d"; \ + echo ""; \ + echo "Then rebuild with --network=host so the build can reach localhost:"; \ + echo " docker build --network=host -f Dockerfile.node-deps ..."; \ + echo ""; \ + echo "Or remove/comment out the unreachable registry in .npmrc"; \ + echo "========================================================"; \ + exit 1; \ + fi; \ + done \ + && echo "==> All registries reachable, running npm install..." \ && for i in 1 2 3 4 5; do \ npm install 2>&1 && break; \ [ "$i" = "5" ] && echo "npm install failed after 5 attempts" && exit 1; \