208 lines
11 KiB
Markdown
208 lines
11 KiB
Markdown
# Porting the Kernel to a New Architecture
|
|
|
|
The kernel is written so that everything outside `kernel/src/Arch/` is
|
|
architecture-neutral. A port is one directory, `kernel/src/Arch/<arch>/`,
|
|
implementing the interfaces declared in `kernel/src/Arch/*.hpp` (plus a few
|
|
interface headers elsewhere, listed below). The build compiles the generic
|
|
tree plus `Arch/$(ARCH)/` only.
|
|
|
|
`scripts/check-arch-neutral.sh` enforces the boundary and should pass before
|
|
every commit that touches the kernel:
|
|
|
|
- no inline assembly, assembly sources or architecture `#if`s outside
|
|
`src/Arch/` (the Limine adapter is the one exception: `limine.h` has
|
|
per-architecture SMP structures);
|
|
- no generic file includes a port's private headers (`<Arch/x86_64/...>`);
|
|
- with clang installed, the kernel compiles for `aarch64` and every symbol
|
|
the generic code references is defined by the port.
|
|
|
|
## Layout
|
|
|
|
```
|
|
kernel/src/Arch/
|
|
Arch.hpp target selection (MONTAUK_ARCH_*), MONTAUK_ARCH_NAME
|
|
Cpu.hpp inline CPU primitives (interrupt state, halt, barriers,
|
|
timestamp counter, TLB, page-table root)
|
|
Interrupts.hpp IRQ slots, handler registration, legacy line routing,
|
|
MSI message composition, IPIs
|
|
Smp.hpp Smp::CpuData (generic fields + arch sub-structs),
|
|
per-CPU base access, InitBsp / BootAPs
|
|
Context.hpp thread context: FPU area, initial kernel stack,
|
|
SwitchContext, ActivateThread, EnterUserMode
|
|
Abi.hpp user ABI: ELF machine, relocation kinds, TLS variant,
|
|
exit stub, initial user stack, EFI calling convention
|
|
Syscall.hpp SyscallFrame contract and entry-point installation
|
|
Timer.hpp per-CPU tick timer hardware
|
|
Platform.hpp boot-sequence hooks, platform device list, reboot
|
|
CpuPower.hpp frequency scaling / thermal governor (may be no-ops)
|
|
x86_64/ the complete port
|
|
aarch64/ the port in progress (see below)
|
|
```
|
|
|
|
Other interfaces a port implements:
|
|
|
|
| Header | What the port provides |
|
|
|---|---|
|
|
| `Memory/Paging.hpp` | the whole `Memory::VMM::Paging` class: page-table format, kernel and user mappings, address-space creation and teardown |
|
|
| `Io/IoPort.hpp` | port I/O (`Io::In8`...), or the absent-device fallback with `HasPortIo = false` |
|
|
| `ACPI/AcpiSleep.hpp` | `Initialize` / `IsS3Available` / `Suspend` (S3 via the FACS waking vector is an x86 mechanism) |
|
|
| `Common/Panic.hpp` | `Arch::PanicFrame` and `Arch::PrintPanicFrame` |
|
|
|
|
Interface headers that need inline definitions include the port's header of
|
|
the same name at their end (`Arch/Cpu.hpp` -> `Arch/<arch>/Cpu.hpp`); those
|
|
are `Cpu.hpp`, `IoPort.hpp`, `PerCpu.hpp`, `Context.hpp`, `Abi.hpp` and
|
|
`SyscallFrame.hpp`. Everything else is an ordinary out-of-line function.
|
|
|
|
## Boot sequence
|
|
|
|
`kmain` (`Boot/Main.cpp`) owns the order and calls the port at fixed points:
|
|
|
|
1. `Arch::EarlyInitialize()` - descriptor tables, FPU/SIMD enable. No allocator.
|
|
2. Frame allocator and heap come up (generic).
|
|
3. `Arch::InitializeTraps()` - exception vectors.
|
|
4. `Paging::Init()` switches to the kernel's own page tables, then
|
|
`Arch::InitializeMemoryAttributes()` (x86: PAT for write-combining).
|
|
5. ACPI tables are parsed (generic). `Arch::InitializeInterrupts(xsdt)` brings
|
|
up interrupt controllers, calls `Smp::InitBsp()`, idle and power
|
|
management; kmain then calls `Arch::EnableInterrupts()`.
|
|
6. PCI, AML, drivers (generic); `Timekeeping::TimerInitialize()` registers
|
|
`IRQ_TIMER` and calls `Arch::Timer::InitializeBsp(1)`.
|
|
7. `Arch::InitializePlatformDevices()` (x86: PS/2).
|
|
8. `Arch::PrepareUserMode()`, `montauk::abi::InitializeSyscalls()` (calls
|
|
`Arch::InitializeSyscallEntry()`), scheduler, IPC.
|
|
9. `Smp::BootAPs()`, `Arch::LateInitialize()`, first process, idle loop.
|
|
|
|
## Assumptions the generic kernel makes
|
|
|
|
- 64-bit, little-endian, 4 KiB pages.
|
|
- A higher-half direct map of physical memory (`Memory::HHDM`), as Limine
|
|
provides on every architecture.
|
|
- User space in the lower half of each address space, kernel in the upper
|
|
half; an address space is identified by the physical address of its root
|
|
table (on ARM, the TTBR0 table; the kernel half lives in TTBR1).
|
|
- Firmware tables via ACPI (`Hal::ACPI`); device-tree-only platforms would
|
|
need a discovery path first.
|
|
- Interrupts are numbered IRQ slots (`Arch::IRQ_COUNT`); slot numbers for
|
|
MSI-capable drivers are fixed in the drivers.
|
|
|
|
## The aarch64 port
|
|
|
|
Real: CPU primitives (DAIF, WFI, `CNTVCT_EL0`, TLBI, TTBR0), per-CPU data via
|
|
`TPIDR_EL1`, the context switch and EL0 entry (`Context.S`), TLS through
|
|
`TPIDR_EL0`, the ELF/TLS/exit-stub ABI (syscall number in x8), the IRQ slot
|
|
dispatch table (`ArchIrqDispatch`), panic-frame printing, and:
|
|
|
|
- Exception vectors (`Vectors.S`, installed in `EarlyInitialize`). Every
|
|
exception saves one 288-byte frame that is both `Arch::PanicFrame` and
|
|
`Arch::SyscallFrame`. EL0 `svc` goes to `SyscallDispatch`; other
|
|
synchronous exceptions go to `ArchSyncException` (`Exceptions.cpp`), which
|
|
grows user stacks and lazy anonymous memory, kills faulting processes with
|
|
a crash report, and panics on kernel faults.
|
|
- Translation tables (`Paging.cpp`): 4 KiB granule, 48-bit VAs. The kernel
|
|
half lives in its own TTBR1 table; `KernelRoot()` is the TTBR0 table of
|
|
kernel contexts (EFI identity mappings only, so null pointers fault). The
|
|
HHDM maps only the memory map's RAM and firmware regions, never the holes
|
|
between them (device space on ARM). MAIR: 0 Normal WB, 1 Normal NC
|
|
(`MapWC`), 2 Device-nGnRE (`MapMMIO`).
|
|
- GICv2 (`Interrupts.cpp`, from the MADT): IRQ slots map onto INTIDs --
|
|
`IRQ_TIMER` onto the generic timer's PPI, the MSI slots (24-45) onto the
|
|
SPIs of a GICv2m MSI frame (`ComposeMsi`), the IPI slots onto SGIs 0/1,
|
|
and the platform lines (1-23) onto the SPIs `AssignPlatformLine` gives
|
|
them (see "Interrupt lines" below).
|
|
Interrupts are completed (EOI) before the handler runs, as on x86; a
|
|
level-triggered source drops its line first in an ack hook.
|
|
- Generic timer tick (`Timer.cpp`): the EL1 virtual timer at
|
|
`CNTFRQ_EL0`, its PPI from the GTDT, re-armed from the ack hook.
|
|
- PSCI `SYSTEM_RESET` / `SYSTEM_OFF` (`Arch::Reboot` / `Arch::PowerOff`)
|
|
through the conduit the FADT's ARM_BOOT_ARCH names. Hardware-reduced
|
|
ACPI has no PM1 block, so `AcpiShutdown` leaves power-off to the port.
|
|
- `Arch::SyncInstructionCache` after the ELF loader writes code: ARM
|
|
instruction fetch does not snoop data writes (QEMU hides this; real
|
|
cores do not). EL0 may maintain caches itself (`SCTLR_EL1.UCI`).
|
|
|
|
- SMP (`Smp.cpp`): Limine wakes each secondary CPU (PSCI `CPU_ON`
|
|
underneath) on the bootloader's tables; it programs its own system
|
|
registers and the kernel translation regime (`Bringup.hpp`), its banked
|
|
GIC CPU interface and timer PPI, then idles until a reschedule SGI.
|
|
TLB maintenance is broadcast (`tlbi ...is`).
|
|
|
|
The generic kernel is compiled `-fsigned-char` for AArch64: it was written
|
|
for x86's signed plain `char`.
|
|
|
|
Building it needs `clang` and `ld.lld`; without a system `ld.lld` the
|
|
kernel makefile uses the one rustup ships
|
|
(`~/.rustup/toolchains/*/lib/rustlib/*/bin/gcc-ld/ld.lld`), and `KERNEL_LD=`
|
|
overrides both. `make run ARCH=aarch64` boots the full image (desktop, apps,
|
|
USB keyboard and mouse) on QEMU `virt` with a GICv2 and Cortex-A72 cores, the
|
|
Raspberry Pi 4/400's configuration; it needs `qemu-system-aarch64` and
|
|
`qemu-efi-aarch64`, and the userspace compiler
|
|
(`./toolchain/build-montauk-toolchain.sh aarch64`). `KERNEL_ONLY=1` builds an
|
|
image without userspace.
|
|
|
|
The kernel console is mirrored to the serial port the ACPI SPCR table names
|
|
(PL011, SBSA UART, 16550 or the BCM2835 mini UART; `Drivers/Serial/Uart.cpp`),
|
|
including the log written before the port was found. On QEMU that is
|
|
`-serial stdio`.
|
|
|
|
## Interrupt lines
|
|
|
|
Drivers for PCI functions without MSI read the Interrupt Line register and
|
|
pass it to `Arch::EnableLegacyIrq`. On x86 the firmware fills that register
|
|
with the function's I/O APIC input (`Arch::PlatformLinesAreFixed()`). On
|
|
other platforms `Pci::AssignInterruptLines` (`Pci/PciAcpi.cpp`) fills it
|
|
instead: it finds each pin's global system interrupt in the root bridge's
|
|
`_PRT` (swizzling through PCI-to-PCI bridges, resolving PNP0C0F link
|
|
devices through `_CRS`), binds the GSI to a platform line with
|
|
`Hal::GsiRouting::LineForGsi` and writes the line number into the register.
|
|
Devices that ACPI describes directly (platform devices) get their line from
|
|
`LineForGsi` too.
|
|
|
|
## DMA
|
|
|
|
`Memory/Dma.hpp` serves devices whose DMA does not snoop the CPU caches
|
|
(ACPI `_CCA = 0`): pages for rings and descriptors come uncached, and
|
|
`SyncForDevice` / `SyncForCpu` clean and invalidate streaming buffers
|
|
around a transfer (`Arch::CleanDataCache` / `InvalidateDataCache`). For
|
|
coherent devices every call is the plain allocation or a no-op. The xHCI
|
|
driver uses it; it also orders its TRB writes with `Arch::DmaWriteFence`
|
|
(the cycle bit last, a control TD's Setup TRB after its Data and Status
|
|
TRBs) and its event reads with `Arch::DmaReadFence`. Those fences cost
|
|
nothing on x86 and are required on ARM, where stores to normal memory and
|
|
device registers may otherwise be observed out of order. The other PCI
|
|
drivers (AHCI, NVMe, e1000, HDA) do not have them yet: they work on QEMU,
|
|
which does not reorder, but need them before they meet real ARM hardware.
|
|
|
|
## Raspberry Pi 4 / 400
|
|
|
|
See `docs/raspberry-pi.md`: `make ARCH=aarch64 rpi4-sd`, the firmware it
|
|
expects and how the Pi's hardware is reached.
|
|
|
|
## Userspace
|
|
|
|
The programs are architecture-neutral C/C++ built by `<arch>-montauk-gcc`
|
|
(see `toolchain/README.md`). `programs/mk/arch.mk` holds everything
|
|
architecture-specific in the build: the toolchain prefix, FP/SIMD flag sets
|
|
(`ARCH_FP_FLAGS` for GUI apps, `ARCH_NOFP_FLAGS` for system programs) and the
|
|
output layout. x86_64 keeps `bin/`, `obj/` and libraries next to their
|
|
sources; other architectures build beside them into `bin-<arch>/`,
|
|
`obj-<arch>/` and `lib/<name>/build-<arch>/`, so both can be built from one
|
|
tree. The few architecture-specific sources are the raw system-call
|
|
wrappers (`#if defined(__x86_64__)` / `__aarch64__` blocks in
|
|
`montauk/syscall.h`, `libc/montauk.h`, `lib/libc/libc.c` and
|
|
`lib/libc/crt/crt1.c`), `lib/libc/arch/setjmp-<arch>.S` and the scanout
|
|
store fence in `gui/framebuffer.hpp`.
|
|
|
|
Not built for AArch64: tcc (it generates x86 code), the on-OS SDK and the
|
|
Rust programs (their `x86_64-unknown-montauk` target).
|
|
|
|
## Adding another architecture
|
|
|
|
1. Create `kernel/src/Arch/<arch>/` with the six inline headers listed above
|
|
and stubs for the out-of-line functions (copy `aarch64/` as a template).
|
|
2. Add the target to `Arch/Arch.hpp` and to the `#if` chains at the end of
|
|
the dispatching interface headers.
|
|
3. Add `KernelStartSymbol` / `KernelEndSymbol` to `kernel/linker-<arch>.ld`,
|
|
and the `ld.lld` emulation to `LD_EMULATION_<arch>` in `kernel/GNUmakefile`.
|
|
4. Teach `Boot/Protocols/LimineProtocol.cpp` the architecture's MP fields.
|
|
5. Run `./scripts/check-arch-neutral.sh <arch>` until it passes.
|