Merge pull request #8 from johndoe6345789/copilot/refine-docs-readme

Refactor README: Extract GPU details to dedicated doc, add deps management structure
This commit is contained in:
2025-12-28 19:13:11 +00:00
committed by GitHub
7 changed files with 521 additions and 61 deletions

View File

@@ -19,72 +19,19 @@ This OS exists solely to run **one QT6 application** on **AMD64 + Radeon RX 6600
**Creative freedom** - Not bound by POSIX or tradition
**Precise drivers** - Hardware code follows specs exactly
## GPU Implementation Strategy
1) Reality check: where the bloat really lives (RDNA2)
MetalOS leverages Mesa RADV (userspace Vulkan driver) with a minimal kernel-side GPU API to achieve high performance without excessive complexity. The strategy focuses on implementing only the essential kernel interfaces that RADV requires:
On Navi 23, you will not get good performance without:
• GPU firmware blobs (various dimgrey_cavefish_*.bin files; Navi 23s codename is “dimgrey cavefish”, and Linux systems load firmware files with that prefix).
• A real memory manager (VRAM/GTT, page tables, buffer objects)
Command submission (rings/queues) + fences/semaphores
• A Vulkan driver implementation (or reuse one)
- **Firmware loading** and ASIC initialization for Navi 23
- **Buffer objects** (VRAM/GTT management)
- **Virtual memory** (GPU page tables)
- **Command submission** (rings/queues) and synchronization primitives
So the “least bloat” strategy is: reuse a Vulkan implementation (Mesa RADV is the obvious candidate), but avoid importing a whole Unix stack by giving it a very small kernel/userspace interface tailored to your OS.
This approach keeps the OS non-POSIX while avoiding the complexity of writing a Vulkan driver from scratch.
RADV is explicitly a userspace Vulkan driver for modern AMD GPUs.
For detailed implementation notes, see [docs/GPU_IMPLEMENTATION.md](docs/GPU_IMPLEMENTATION.md).
2) The best “toy OS but fast” plan: RADV + a tiny amdgpu-shaped shim
Why this is the sweet spot
• You keep your OS non-POSIX.
• You avoid writing a Vulkan driver from scratch (the truly hard part).
• You implement only the kernel-facing parts RADV needs: a buffer object + VM + submit + sync API.
Shape of the stack
MetalOS kernel
• PCIe enumeration, BAR mapping
• interrupts (MSI/MSI-X)
• DMA mapping (or identity-map if youre being reckless)
• a GPU kernel driver that exposes a small ioctl-like API
Userspace
• gpu-service (optional but recommended for structure)
• libradv-metal (a minimal libdrm-like bridge)
• Mesa RADV compiled against your bridge (not Linux libdrm)
This is “Unix-like internally” only in the sense of interfaces, not user experience.
3) Minimal kernel GPU API (the smallest set that still performs)
Think in terms of four pillars:
A) Firmware load + ASIC init
• gpu_load_firmware(name, blob)
• gpu_init() → returns chip info (gfx1032, VRAM size, doorbells, etc.)
You will need those Navi23 firmware blobs (again: dimgrey_cavefish_*.bin family is the practical breadcrumb).
B) Buffer objects (BOs)
• bo_create(size, domain=VRAM|GTT, flags)
• bo_map(bo) / bo_unmap(bo) (CPU mapping)
• bo_export_handle(bo) (so Vulkan can bind memory)
C) Virtual memory (GPU page tables)
• vm_create()
• vm_map(vm, bo, gpu_va, size, perms)
• vm_unmap(vm, gpu_va, size)
D) Submission + synchronization
• queue_create(type=GFX|COMPUTE|DMA)
• queue_submit(queue, cs_buffer, fence_out)
• fence_wait(fence, timeout)
• timeline_semaphore_* (optional, but hugely useful)
If you implement these correctly, you get real GPU throughput.
## What We Cut
@@ -138,9 +85,21 @@ make qemu-uefi-test # Test UEFI firmware setup
See [docs/BUILD.md](docs/BUILD.md) for detailed build instructions and [docs/TESTING.md](docs/TESTING.md) for testing guide.
## Dependencies
MetalOS manages third-party dependencies in-house for reproducibility and offline development:
- **GPU Firmware** - AMD Navi 23 firmware blobs (dimgrey_cavefish_*.bin)
- **Mesa RADV** - Vulkan driver for AMD GPUs
- **QT6** - Application framework (minimal static build)
- **OVMF** - UEFI firmware for QEMU testing
See [deps/README.md](deps/README.md) for detailed dependency management instructions.
## Documentation
- [ARCHITECTURE.md](docs/ARCHITECTURE.md) - System architecture and design
- [GPU_IMPLEMENTATION.md](docs/GPU_IMPLEMENTATION.md) - GPU driver strategy and implementation
- [MINIMALISM.md](docs/MINIMALISM.md) - Extreme minimalism philosophy
- [ROADMAP.md](docs/ROADMAP.md) - Development phases and milestones
- [BUILD.md](docs/BUILD.md) - Build system and toolchain

156
deps/README.md vendored Normal file
View File

@@ -0,0 +1,156 @@
# MetalOS Dependencies
This directory contains third-party dependencies managed in-house for MetalOS development.
## Directory Structure
```
deps/
├── firmware/ # AMD GPU firmware blobs
├── mesa-radv/ # Mesa RADV Vulkan driver source
├── qt6/ # QT6 framework (minimal build)
└── ovmf/ # UEFI firmware for testing
```
## Firmware Blobs
**Location**: `deps/firmware/`
AMD Radeon RX 6600 (Navi 23) requires several firmware files:
- `dimgrey_cavefish_ce.bin` - Command Engine firmware
- `dimgrey_cavefish_me.bin` - MicroEngine firmware
- `dimgrey_cavefish_mec.bin` - MEC (Compute) firmware
- `dimgrey_cavefish_pfp.bin` - Pre-Fetch Parser firmware
- `dimgrey_cavefish_rlc.bin` - Run List Controller firmware
- `dimgrey_cavefish_sdma.bin` - SDMA engine firmware
- `dimgrey_cavefish_vcn.bin` - Video Core Next firmware
### Obtaining Firmware
These firmware files are available from the Linux firmware repository:
```bash
# Clone linux-firmware repository
git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git /tmp/linux-firmware
# Copy Navi 23 firmware files
cp /tmp/linux-firmware/amdgpu/dimgrey_cavefish_*.bin deps/firmware/
```
**License**: These files are redistributable under the AMD GPU firmware license (see firmware files for details).
## Mesa RADV
**Location**: `deps/mesa-radv/`
Mesa RADV is the userspace Vulkan driver for AMD GPUs. MetalOS uses RADV to avoid implementing a Vulkan driver from scratch.
### Setup
```bash
# Clone Mesa repository (or download specific release)
git clone https://gitlab.freedesktop.org/mesa/mesa.git /tmp/mesa
cd /tmp/mesa
# Checkout stable version (e.g., 24.0)
git checkout mesa-24.0.0
# Copy RADV driver sources to deps
cp -r src/amd/vulkan ../MetalOS/deps/mesa-radv/
```
We will compile RADV against our custom `libradv-metal` bridge instead of Linux's libdrm.
**License**: MIT License (see Mesa source for full license text)
## QT6 Framework
**Location**: `deps/qt6/`
QT6 is the application framework used by the single MetalOS application.
### Setup
```bash
# Download QT6 source (minimal modules only)
# We only need QtCore, QtGui, and QtWidgets
wget https://download.qt.io/official_releases/qt/6.5/6.5.3/single/qt-everywhere-src-6.5.3.tar.xz
# Extract and configure minimal build
tar xf qt-everywhere-src-6.5.3.tar.xz
cd qt-everywhere-src-6.5.3
# Configure for MetalOS (minimal, static linking)
./configure -static -no-dbus -no-ssl -no-cups -no-fontconfig \
-prefix $PWD/../deps/qt6
```
**Note**: Adjust the path as needed based on your directory structure. The example above assumes you extracted QT source alongside the MetalOS repository.
**License**: LGPL v3 / GPL v2 / Commercial (see QT documentation)
## OVMF (EDK II)
**Location**: `deps/ovmf/`
OVMF provides UEFI firmware for QEMU testing.
### Setup
```bash
# Download pre-built OVMF binaries
mkdir -p deps/ovmf
wget https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-0-20230524.209.gf0064ac3af.EOL.no.nore.updates.noarch.rpm
# Extract OVMF files
rpm2cpio edk2.git-ovmf-*.rpm | cpio -idmv
cp usr/share/edk2/ovmf/OVMF_CODE.fd deps/ovmf/
cp usr/share/edk2/ovmf/OVMF_VARS.fd deps/ovmf/
```
Alternatively, build from source:
```bash
git clone https://github.com/tianocore/edk2.git
cd edk2
git submodule update --init
make -C BaseTools
source edksetup.sh
build -a X64 -t GCC5 -p OvmfPkg/OvmfPkgX64.dsc
```
**License**: BSD-2-Clause (see EDK II license)
## Version Management
Each dependency should be tracked with a specific version/commit:
- **Firmware**: Linux firmware commit hash
- **Mesa RADV**: Mesa version tag (e.g., `mesa-24.0.0`)
- **QT6**: QT version (e.g., `6.5.3`)
- **OVMF**: EDK II commit hash or release tag
Create a `VERSION` file in each subdirectory documenting the exact version in use.
## Build Integration
The main Makefile will be updated to:
1. Check for dependencies in `deps/`
2. Use in-house versions when available
3. Fall back to system versions if needed
4. Provide targets to download/build dependencies
## Rationale
Managing dependencies in-house provides:
- **Reproducibility**: Exact versions are locked and available
- **Offline development**: No external downloads required
- **Control**: We can patch or customize dependencies as needed
- **Simplicity**: Single repository contains everything needed to build
## Size Considerations
The deps directory may become large (~500MB-1GB). Consider:
- Using `.gitattributes` for Git LFS on large binary files
- Documenting exact download/build steps instead of committing binaries
- Selective inclusion based on development phase
For now, we document the structure and process. Actual population of dependencies will occur as needed during implementation phases.

40
deps/firmware/README.md vendored Normal file
View File

@@ -0,0 +1,40 @@
# AMD GPU Firmware Blobs
This directory contains firmware files for the AMD Radeon RX 6600 (Navi 23 / "dimgrey cavefish").
## Required Files
- `dimgrey_cavefish_ce.bin` - Command Engine firmware
- `dimgrey_cavefish_me.bin` - MicroEngine firmware
- `dimgrey_cavefish_mec.bin` - MEC (Compute) firmware
- `dimgrey_cavefish_pfp.bin` - Pre-Fetch Parser firmware
- `dimgrey_cavefish_rlc.bin` - Run List Controller firmware
- `dimgrey_cavefish_sdma.bin` - SDMA engine firmware
- `dimgrey_cavefish_vcn.bin` - Video Core Next firmware
## Source
These files are available from the [linux-firmware repository](https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git).
## Obtaining Firmware
```bash
# From repository root
git clone --depth 1 https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git /tmp/linux-firmware
cp /tmp/linux-firmware/amdgpu/dimgrey_cavefish_*.bin deps/firmware/
```
## License
These firmware files are redistributable under the AMD GPU firmware license included with each file.
## Version Tracking
Create a `VERSION` file documenting:
- Linux firmware repository commit hash
- Date obtained
- File checksums (SHA256)
## Status
⚠️ **Not yet populated** - Firmware files will be added during Phase 4 (Hardware Support)

52
deps/mesa-radv/README.md vendored Normal file
View File

@@ -0,0 +1,52 @@
# Mesa RADV Driver
This directory will contain the Mesa RADV (AMD Vulkan driver) source code.
## Overview
RADV is a userspace Vulkan driver for AMD GPUs. MetalOS uses RADV to provide Vulkan support without implementing a driver from scratch.
## Source
Mesa RADV is part of the [Mesa 3D Graphics Library](https://gitlab.freedesktop.org/mesa/mesa).
## Setup Instructions
```bash
# Clone Mesa repository
git clone https://gitlab.freedesktop.org/mesa/mesa.git /tmp/mesa
cd /tmp/mesa
# Checkout stable version
git checkout mesa-24.0.0
# Copy RADV driver sources to deps
# Assuming you're in the mesa source directory and MetalOS repo is in parent
cp -r src/amd/vulkan/* ../MetalOS/deps/mesa-radv/
```
## Integration with MetalOS
MetalOS will compile RADV against a custom `libradv-metal` bridge that provides:
- Buffer object management
- Virtual memory (GPU page tables)
- Command submission
- Synchronization primitives
This replaces the standard Linux libdrm dependency.
## License
MIT License - See Mesa source for full license text.
## Version Tracking
Create a `VERSION` file documenting:
- Mesa version/tag (e.g., `mesa-24.0.0`)
- Git commit hash
- Date obtained
- Any local modifications
## Status
⚠️ **Not yet populated** - RADV sources will be added during Phase 7 (QT6 Port)

87
deps/ovmf/README.md vendored Normal file
View File

@@ -0,0 +1,87 @@
# OVMF UEFI Firmware
This directory contains OVMF (Open Virtual Machine Firmware) for QEMU testing.
## Overview
OVMF provides UEFI firmware implementation for QEMU, allowing us to test MetalOS in a UEFI environment.
## Required Files
- `OVMF_CODE.fd` - UEFI firmware code
- `OVMF_VARS.fd` - UEFI variables template
## Source
OVMF is part of the [EDK II project](https://github.com/tianocore/edk2).
## Obtaining OVMF
### Option 1: Use System-Installed OVMF (Recommended)
Many distributions provide OVMF packages:
```bash
# Ubuntu/Debian
sudo apt-get install ovmf
# Copy to deps folder
cp /usr/share/OVMF/OVMF_CODE.fd deps/ovmf/
cp /usr/share/OVMF/OVMF_VARS.fd deps/ovmf/
```
### Option 2: Download Pre-built Binaries
Pre-built OVMF binaries are available from various sources:
```bash
# Visit https://www.kraxel.org/repos/jenkins/edk2/ for current builds
# Or use direct links from distributions' mirror sites
# Example (adjust URL to current available version):
wget https://www.kraxel.org/repos/jenkins/edk2/edk2.git-ovmf-x64-<VERSION>.rpm
rpm2cpio edk2.git-ovmf-*.rpm | cpio -idmv
cp usr/share/edk2/ovmf/OVMF_CODE.fd deps/ovmf/
cp usr/share/edk2/ovmf/OVMF_VARS.fd deps/ovmf/
```
**Note**: Pre-built binaries may become unavailable. System packages (Option 1) or building from source (Option 3) are more reliable long-term.
### Option 3: Build from Source
```bash
git clone https://github.com/tianocore/edk2.git
cd edk2
git submodule update --init
make -C BaseTools
source edksetup.sh
build -a X64 -t GCC5 -p OvmfPkg/OvmfPkgX64.dsc
# Files will be in Build/OvmfX64/RELEASE_GCC5/FV/
cp Build/OvmfX64/RELEASE_GCC5/FV/OVMF_CODE.fd ../MetalOS/deps/ovmf/
cp Build/OvmfX64/RELEASE_GCC5/FV/OVMF_VARS.fd ../MetalOS/deps/ovmf/
```
## Usage with QEMU
```bash
qemu-system-x86_64 \
-bios deps/ovmf/OVMF_CODE.fd \
-drive file=metalos.img,format=raw \
-m 512M
```
## License
BSD-2-Clause - See EDK II license for full details.
## Version Tracking
Create a `VERSION` file documenting:
- EDK II commit hash or version
- Build date
- Source (pre-built package, jenkins, or self-built)
- File checksums (SHA256)
## Status
⚠️ **Optional** - Can use system-installed OVMF or this in-house copy. The Makefile already detects system OVMF paths.

66
deps/qt6/README.md vendored Normal file
View File

@@ -0,0 +1,66 @@
# QT6 Framework
This directory will contain a minimal QT6 build for MetalOS.
## Overview
QT6 provides the application framework for the single MetalOS application. We use a minimal, statically-linked build with only essential modules.
## Required Modules
- QtCore - Core non-GUI functionality
- QtGui - GUI components
- QtWidgets - Widget toolkit
## Source
QT6 is available from the [Qt Project](https://download.qt.io/official_releases/qt/).
## Setup Instructions
```bash
# Download QT6 source
wget https://download.qt.io/official_releases/qt/6.5/6.5.3/single/qt-everywhere-src-6.5.3.tar.xz
tar xf qt-everywhere-src-6.5.3.tar.xz
cd qt-everywhere-src-6.5.3
# Configure minimal build for MetalOS
./configure -static -release \
-no-dbus -no-ssl -no-cups -no-fontconfig \
-no-feature-network -no-feature-sql \
-prefix $PWD/../deps/qt6 \
-opensource -confirm-license
# Build (this will take a while)
make -j$(nproc)
make install
```
## MetalOS-Specific Configuration
We configure QT6 with:
- Static linking (no shared libraries)
- Minimal feature set
- No networking, database, or printing support
- Custom platform plugin for MetalOS framebuffer
## License
QT6 is available under:
- LGPL v3
- GPL v2
- Commercial License
For MetalOS, we use LGPL v3. See QT documentation for license details.
## Version Tracking
Create a `VERSION` file documenting:
- QT version (e.g., `6.5.3`)
- Build configuration options
- Date built
- Any local patches
## Status
⚠️ **Not yet populated** - QT6 will be added during Phase 7 (QT6 Port)

100
docs/GPU_IMPLEMENTATION.md Normal file
View File

@@ -0,0 +1,100 @@
# GPU Implementation Strategy
## Overview
This document outlines the GPU implementation strategy for MetalOS targeting the AMD Radeon RX 6600 (RDNA2 / Navi 23 architecture).
## Reality Check: Where the Bloat Really Lives (RDNA2)
On Navi 23, you will not get good performance without:
- GPU firmware blobs (various `dimgrey_cavefish_*.bin` files; Navi 23's codename is "dimgrey cavefish", and Linux systems load firmware files with that prefix)
- A real memory manager (VRAM/GTT, page tables, buffer objects)
- Command submission (rings/queues) + fences/semaphores
- A Vulkan driver implementation (or reuse one)
So the "least bloat" strategy is: reuse a Vulkan implementation (Mesa RADV is the obvious candidate), but avoid importing a whole Unix stack by giving it a very small kernel/userspace interface tailored to your OS.
RADV is explicitly a userspace Vulkan driver for modern AMD GPUs.
---
## The Best "Toy OS but Fast" Plan: RADV + a Tiny amdgpu-shaped Shim
### Why This is the Sweet Spot
- You keep your OS non-POSIX
- You avoid writing a Vulkan driver from scratch (the truly hard part)
- You implement only the kernel-facing parts RADV needs: a buffer object + VM + submit + sync API
### Shape of the Stack
**MetalOS Kernel:**
- PCIe enumeration, BAR mapping
- Interrupts (MSI/MSI-X)
- DMA mapping (or identity-map if you're being reckless)
- A GPU kernel driver that exposes a small ioctl-like API
**Userspace:**
- `gpu-service` (optional but recommended for structure)
- `libradv-metal` (a minimal libdrm-like bridge)
- Mesa RADV compiled against your bridge (not Linux libdrm)
This is "Unix-like internally" only in the sense of interfaces, not user experience.
---
## Minimal Kernel GPU API (The Smallest Set That Still Performs)
Think in terms of four pillars:
### A) Firmware Load + ASIC Init
```c
gpu_load_firmware(name, blob)
gpu_init() returns chip info (gfx1032, VRAM size, doorbells, etc.)
```
You will need those Navi23 firmware blobs (again: `dimgrey_cavefish_*.bin` family is the practical breadcrumb).
### B) Buffer Objects (BOs)
```c
bo_create(size, domain=VRAM|GTT, flags)
bo_map(bo) / bo_unmap(bo) // CPU mapping
bo_export_handle(bo) // so Vulkan can bind memory
```
### C) Virtual Memory (GPU Page Tables)
```c
vm_create()
vm_map(vm, bo, gpu_va, size, perms)
vm_unmap(vm, gpu_va, size)
```
### D) Submission + Synchronization
```c
queue_create(type=GFX|COMPUTE|DMA)
queue_submit(queue, cs_buffer, fence_out)
fence_wait(fence, timeout)
timeline_semaphore_* // optional, but hugely useful
```
If you implement these correctly, you get real GPU throughput.
---
## Implementation Notes
- Focus on the minimal API surface that RADV requires
- Firmware blobs are non-negotiable for Navi 23 performance
- Memory management (VRAM/GTT) is critical for proper GPU operation
- Command submission infrastructure must be solid for reliability
- Synchronization primitives (fences/semaphores) enable proper GPU-CPU coordination
## References
- Mesa RADV driver source code
- AMD GPU specifications for RDNA2 architecture
- Linux amdgpu kernel driver for reference implementation patterns