diff --git a/docs/building.md b/docs/building.md index 7e00278..2ec5cfe 100644 --- a/docs/building.md +++ b/docs/building.md @@ -70,10 +70,16 @@ sudo apt install build-essential nasm git curl xorriso mtools gdisk qemu-system- ### Cross-Architecture Builds -To build for a different architecture, pass `ARCH=`: +The kernel is split into architecture-neutral code and per-architecture +ports under `kernel/src/Arch/` (see [porting.md](porting.md)). x86_64 is the +only complete port. `aarch64` is a skeleton: the whole kernel compiles and +links for it, but paging, exception vectors, the GIC and the timer are not +written yet, so it does not boot. Userspace is x86_64-only. + +Non-x86 kernels are built with LLVM (`clang`, plus `ld.lld` for the link): ```bash -make ARCH=aarch64 -make ARCH=riscv64 -make ARCH=loongarch64 +make kernel ARCH=aarch64 # kernel/bin-aarch64/kernel +make kernel ARCH=aarch64 KERNEL_LD=/path/to/ld.lld +./scripts/check-arch-neutral.sh # boundary check + aarch64 compile ``` diff --git a/docs/porting.md b/docs/porting.md new file mode 100644 index 0000000..e4053e8 --- /dev/null +++ b/docs/porting.md @@ -0,0 +1,117 @@ +# Porting the Kernel to a New Architecture + +The kernel is written so that everything outside `kernel/src/Arch/` is +architecture-neutral. A port is one directory, `kernel/src/Arch//`, +implementing the interfaces declared in `kernel/src/Arch/*.hpp` (plus a few +interface headers elsewhere, listed below). The build compiles the generic +tree plus `Arch/$(ARCH)/` only. + +`scripts/check-arch-neutral.sh` enforces the boundary and should pass before +every commit that touches the kernel: + +- no inline assembly, assembly sources or architecture `#if`s outside + `src/Arch/` (the Limine adapter is the one exception: `limine.h` has + per-architecture SMP structures); +- no generic file includes a port's private headers (``); +- with clang installed, the kernel compiles for `aarch64` and every symbol + the generic code references is defined by the port. + +## Layout + +``` +kernel/src/Arch/ + Arch.hpp target selection (MONTAUK_ARCH_*), MONTAUK_ARCH_NAME + Cpu.hpp inline CPU primitives (interrupt state, halt, barriers, + timestamp counter, TLB, page-table root) + Interrupts.hpp IRQ slots, handler registration, legacy line routing, + MSI message composition, IPIs + Smp.hpp Smp::CpuData (generic fields + arch sub-structs), + per-CPU base access, InitBsp / BootAPs + Context.hpp thread context: FPU area, initial kernel stack, + SwitchContext, ActivateThread, EnterUserMode + Abi.hpp user ABI: ELF machine, relocation kinds, TLS variant, + exit stub, initial user stack, EFI calling convention + Syscall.hpp SyscallFrame contract and entry-point installation + Timer.hpp per-CPU tick timer hardware + Platform.hpp boot-sequence hooks, platform device list, reboot + CpuPower.hpp frequency scaling / thermal governor (may be no-ops) + x86_64/ the complete port + aarch64/ skeleton port +``` + +Other interfaces a port implements: + +| Header | What the port provides | +|---|---| +| `Memory/Paging.hpp` | the whole `Memory::VMM::Paging` class: page-table format, kernel and user mappings, address-space creation and teardown | +| `Io/IoPort.hpp` | port I/O (`Io::In8`...), or the absent-device fallback with `HasPortIo = false` | +| `ACPI/AcpiSleep.hpp` | `Initialize` / `IsS3Available` / `Suspend` (S3 via the FACS waking vector is an x86 mechanism) | +| `Common/Panic.hpp` | `Arch::PanicFrame` and `Arch::PrintPanicFrame` | + +Interface headers that need inline definitions include the port's header of +the same name at their end (`Arch/Cpu.hpp` -> `Arch//Cpu.hpp`); those +are `Cpu.hpp`, `IoPort.hpp`, `PerCpu.hpp`, `Context.hpp`, `Abi.hpp` and +`SyscallFrame.hpp`. Everything else is an ordinary out-of-line function. + +## Boot sequence + +`kmain` (`Boot/Main.cpp`) owns the order and calls the port at fixed points: + +1. `Arch::EarlyInitialize()` - descriptor tables, FPU/SIMD enable. No allocator. +2. Frame allocator and heap come up (generic). +3. `Arch::InitializeTraps()` - exception vectors. +4. `Paging::Init()` switches to the kernel's own page tables, then + `Arch::InitializeMemoryAttributes()` (x86: PAT for write-combining). +5. ACPI tables are parsed (generic). `Arch::InitializeInterrupts(xsdt)` brings + up interrupt controllers, calls `Smp::InitBsp()`, idle and power + management; kmain then calls `Arch::EnableInterrupts()`. +6. PCI, AML, drivers (generic); `Timekeeping::TimerInitialize()` registers + `IRQ_TIMER` and calls `Arch::Timer::InitializeBsp(1)`. +7. `Arch::InitializePlatformDevices()` (x86: PS/2). +8. `Arch::PrepareUserMode()`, `montauk::abi::InitializeSyscalls()` (calls + `Arch::InitializeSyscallEntry()`), scheduler, IPC. +9. `Smp::BootAPs()`, `Arch::LateInitialize()`, first process, idle loop. + +## Assumptions the generic kernel makes + +- 64-bit, little-endian, 4 KiB pages. +- A higher-half direct map of physical memory (`Memory::HHDM`), as Limine + provides on every architecture. +- User space in the lower half of each address space, kernel in the upper + half; an address space is identified by the physical address of its root + table (on ARM, the TTBR0 table; the kernel half lives in TTBR1). +- Firmware tables via ACPI (`Hal::ACPI`); device-tree-only platforms would + need a discovery path first. +- Interrupts are numbered IRQ slots (`Arch::IRQ_COUNT`); slot numbers for + MSI-capable drivers are fixed in the drivers. + +## The aarch64 skeleton + +Real: CPU primitives (DAIF, WFI, `CNTVCT_EL0`, TLBI, TTBR0), per-CPU data via +`TPIDR_EL1`, the context switch and EL0 entry (`Context.S`), TLS through +`TPIDR_EL0`, the ELF/TLS/exit-stub ABI (syscall number in x8), the IRQ slot +dispatch table (`ArchIrqDispatch`), panic-frame printing. + +Not yet written (each is marked `TODO(aarch64)`): + +- `Paging.cpp`: translation tables and MAIR (every method panics). +- Exception vectors (`VBAR_EL1`): synchronous faults, `svc`, IRQs. +- GIC driver: SPI routing, SGIs for IPIs, MSIs via ITS/GICv2m + (`ComposeMsi` returns false, so drivers fall back to INTx/polling). +- Generic timer tick (`Arch::Timer::InitializeBsp` returns false). +- Secondary CPU start-up, PSCI reset and suspend. + +Building it needs `clang` and `ld.lld` (`make kernel ARCH=aarch64`; +`KERNEL_LD=` points at a different `ld.lld`, e.g. rustup's +`lib/rustlib/x86_64-unknown-linux-gnu/bin/gcc-ld/ld.lld`). + +## Adding another architecture + +1. Create `kernel/src/Arch//` with the six inline headers listed above + and stubs for the out-of-line functions (copy `aarch64/` as a template). +2. Add the target to `Arch/Arch.hpp` and to the `#if` chains at the end of + the dispatching interface headers. +3. Add `KernelStartSymbol` / `KernelEndSymbol` to `kernel/linker-.ld`, + and the `ld.lld` emulation to `LD_EMULATION_` in `kernel/GNUmakefile`. +4. Teach `Boot/Protocols/LimineProtocol.cpp` the architecture's MP fields. +5. Run `./scripts/check-arch-neutral.sh ` until it passes. diff --git a/kernel/GNUmakefile b/kernel/GNUmakefile index df8b312..5a9de63 100644 --- a/kernel/GNUmakefile +++ b/kernel/GNUmakefile @@ -27,6 +27,20 @@ ifeq ($(ARCH),x86_64) endif endif +# The other architectures have no in-tree cross GCC: default to LLVM, which +# cross-compiles to every target (linking needs ld.lld; see docs/porting.md). +ifneq ($(ARCH),x86_64) + ifeq ($(filter command line environment,$(origin CC)),) + CC := clang + endif + ifeq ($(filter command line environment,$(origin CXX)),) + CXX := clang++ + endif + ifeq ($(filter command line environment,$(origin AR)),) + AR := ar + endif +endif + # User controllable C compiler command. CC ?= cc @@ -197,8 +211,10 @@ override CXXFLAGS += \ -std=gnu++20 \ -fno-rtti \ -fno-exceptions \ - -fmodules-ts \ $(SHARED_FLAGS) +ifneq ($(CXX_IS_CLANG),1) + override CXXFLAGS += -fmodules-ts +endif # Internal linker flags that should not be changed by the user. override LDFLAGS += \ @@ -212,6 +228,9 @@ override LDFLAGS += \ # Use "find" to glob all *.c, *.cpp, *.S, and *.asm files in the tree and obtain the # object and header dependency file names. override SRCFILES := $(shell cd src && find -L * -type f | LC_ALL=C sort) +# Architecture-specific code lives in src/Arch//; only the target's +# directory is built. Everything else in src/ is architecture-neutral. +override SRCFILES := $(filter-out Arch/%,$(SRCFILES)) $(filter Arch/$(ARCH)/%,$(SRCFILES)) override CFILES := $(filter %.c,$(SRCFILES)) override CXXFILES := $(filter %.cpp,$(SRCFILES)) override OBJCFILES := $(filter %.mm,$(SRCFILES)) @@ -253,6 +272,11 @@ $(BUILDNO_HDR): $(filter-out $(BUILDNO_OBJ),$(OBJ)) $(PROG_SRC) # on a clean build (where no .d dependency exists yet). $(BUILDNO_OBJ): $(BUILDNO_HDR) +# Compile everything without linking (used by scripts/check-arch-neutral.sh +# to verify a port's objects before a cross linker is available). +.PHONY: objects +objects: $(OBJ) cc-runtime-$(ARCH)/cc-runtime.a + # Link rules for building the C compiler runtime. cc-runtime-$(ARCH)/cc-runtime.a: GNUmakefile cc-runtime/* rm -rf cc-runtime-$(ARCH) @@ -263,10 +287,33 @@ cc-runtime-$(ARCH)/cc-runtime.a: GNUmakefile cc-runtime/* CFLAGS="$(CFLAGS)" \ CPPFLAGS='-isystem ../freestnd-c-hdrs/$(ARCH)/include -DCC_RUNTIME_NO_FLOAT' +# Clang's generic-ELF driver links cross targets through the host's GCC, +# so a clang-built non-x86 kernel is linked by ld.lld directly. +ifneq ($(ARCH),x86_64) + ifeq ($(CXX_IS_CLANG),1) + KERNEL_LD ?= ld.lld + override LD_EMULATION_aarch64 := aarch64elf + override LD_EMULATION_riscv64 := elf64lriscv + override LD_EMULATION_loongarch64 := elf64loongarch + override KERNEL_LDFLAGS := \ + -m $(LD_EMULATION_$(ARCH)) \ + --build-id=none \ + --no-relax \ + -static \ + -z max-page-size=0x1000 \ + --gc-sections \ + -T linker-$(ARCH).ld + endif +endif + # Link rules for the final executable. bin-$(ARCH)/$(OUTPUT): GNUmakefile linker-$(ARCH).ld $(OBJ) cc-runtime-$(ARCH)/cc-runtime.a mkdir -p "$$(dirname $@)" +ifeq ($(KERNEL_LD),) $(CXX) $(CXXFLAGS) $(LDFLAGS) $(OBJ) cc-runtime-$(ARCH)/cc-runtime.a -o $@ +else + $(KERNEL_LD) $(KERNEL_LDFLAGS) $(OBJ) cc-runtime-$(ARCH)/cc-runtime.a -o $@ +endif # Compilation rules for *.c files. obj-$(ARCH)/%.c.o: src/%.c GNUmakefile diff --git a/kernel/linker-aarch64.ld b/kernel/linker-aarch64.ld index 00ad9ec..3135402 100644 --- a/kernel/linker-aarch64.ld +++ b/kernel/linker-aarch64.ld @@ -22,6 +22,7 @@ SECTIONS /* Any address in this region will do, but often 0xffffffff80000000 is chosen as */ /* that is the beginning of the region. */ . = 0xffffffff80000000; + KernelStartSymbol = .; /* Define a section to contain the Limine requests and assign it to its own PHDR */ .limine_requests : { @@ -74,4 +75,6 @@ SECTIONS *(.eh_frame*) *(.note .note.*) } + + KernelEndSymbol = .; } diff --git a/kernel/src/ACPI/AML/AmlInterpreter.cpp b/kernel/src/ACPI/AML/AmlInterpreter.cpp index e88ec55..7520164 100644 --- a/kernel/src/ACPI/AML/AmlInterpreter.cpp +++ b/kernel/src/ACPI/AML/AmlInterpreter.cpp @@ -16,7 +16,7 @@ #include #include #include -#include +#include using namespace Kt; diff --git a/kernel/src/ACPI/AcpiEvents.cpp b/kernel/src/ACPI/AcpiEvents.cpp index bf09ffe..3611023 100644 --- a/kernel/src/ACPI/AcpiEvents.cpp +++ b/kernel/src/ACPI/AcpiEvents.cpp @@ -7,9 +7,7 @@ #include "AcpiEvents.hpp" #include #include -#include -#include -#include +#include #include #include @@ -191,24 +189,21 @@ namespace Hal { // routed. Clear stale status only after the enables are down. DisableAndClearGpes(); - // Route SCI to an IRQ vector. The SCI is level-triggered, + // Route the SCI to its IRQ slot. The SCI is level-triggered, // active-low (ACPI spec requirement). The MADT may have an // Interrupt Source Override for this IRQ that already sets - // these flags; RouteIrq applies overrides automatically. - uint8_t sciVector = IRQ_VECTOR_BASE + (uint8_t)g_sciIrq; - uint8_t bspApicId = (uint8_t)LocalApic::GetId(); + // these flags; RouteLegacyIrq applies overrides automatically. // Register the handler before routing to avoid missing events - RegisterIrqHandler((uint8_t)g_sciIrq, SciHandler); + Arch::RegisterIrqHandler((uint8_t)g_sciIrq, SciHandler); - // Route via IoApic (applies MADT overrides for polarity/trigger) - IoApic::RouteIrq((uint8_t)g_sciIrq, sciVector, bspApicId); + // Route to the boot CPU (applies firmware polarity/trigger overrides) + Arch::RouteLegacyIrq((uint8_t)g_sciIrq); g_initialized = true; KernelLogStream(OK, "ACPI") << "ACPI events initialized (SCI IRQ " - << base::dec << (uint64_t)g_sciIrq << ", vector " - << (uint64_t)sciVector << ")"; + << base::dec << (uint64_t)g_sciIrq << ")"; } void Reinitialize() { diff --git a/kernel/src/ACPI/AcpiShutdown.cpp b/kernel/src/ACPI/AcpiShutdown.cpp index 2de8253..d3990f5 100644 --- a/kernel/src/ACPI/AcpiShutdown.cpp +++ b/kernel/src/ACPI/AcpiShutdown.cpp @@ -5,6 +5,7 @@ */ #include "AcpiShutdown.hpp" +#include #include "AcpiSleep.hpp" #include "AcpiEvents.hpp" #include @@ -95,7 +96,7 @@ namespace Hal { } } - asm volatile("cli"); + Arch::DisableInterrupts(); // Phase 1: Write SLP_TYP to PM1x_CNT without SLP_EN, // preserving existing register bits @@ -110,7 +111,7 @@ namespace Hal { } // Flush all caches before entering sleep state - asm volatile("wbinvd"); + Arch::WritebackInvalidateCaches(); // Phase 2: Assert SLP_EN to trigger the transition Io::Out16(pm1a | SLP_EN, (uint16_t)g_pm1aControlBlock); @@ -122,7 +123,7 @@ namespace Hal { // Halt and wait for the hardware to power off for (;;) { - asm volatile("hlt"); + Arch::Halt(); } } }; diff --git a/kernel/src/ACPI/AcpiSleep.hpp b/kernel/src/ACPI/AcpiSleep.hpp index 386cfd2..54b6d51 100644 --- a/kernel/src/ACPI/AcpiSleep.hpp +++ b/kernel/src/ACPI/AcpiSleep.hpp @@ -75,21 +75,6 @@ namespace Hal { // CPU State (saved across S3) - // ============================================================================ - // Layout must match S3Wake.asm offsets exactly. - // No FPU/SSE state — the kernel is compiled with -mno-sse. - struct CpuState { - uint64_t Rax, Rbx, Rcx, Rdx; // 0x00 - 0x18 - uint64_t Rsi, Rdi, Rbp, Rsp; // 0x20 - 0x38 - uint64_t R8, R9, R10, R11; // 0x40 - 0x58 - uint64_t R12, R13, R14, R15; // 0x60 - 0x78 - uint64_t Rflags; // 0x80 - uint64_t Rip; // 0x88 - uint64_t Cr3; // 0x90 - uint64_t Cr0, Cr4; // 0x98, 0xA0 - uint8_t GdtPtr[10]; // 0xA8 - uint8_t IdtPtr[10]; // 0xB2 - } __attribute__((aligned(16))); // ============================================================================ @@ -109,5 +94,9 @@ namespace Hal { // Returns 0 on successful resume, or -1 if S3 is unavailable. int Suspend(); + // Initialize/IsS3Available/Suspend are implemented by the + // architecture (the FACS waking-vector path is an x86 platform + // mechanism); other targets report S3 as unavailable. + }; }; diff --git a/kernel/src/ACPI/Battery.cpp b/kernel/src/ACPI/Battery.cpp index 16b5468..c5c0957 100644 --- a/kernel/src/ACPI/Battery.cpp +++ b/kernel/src/ACPI/Battery.cpp @@ -9,7 +9,7 @@ #include "AcpiDevices.hpp" #include #include -#include +#include #include #include diff --git a/kernel/src/Api/Device.hpp b/kernel/src/Api/Device.hpp index 0ad9906..7c8af51 100644 --- a/kernel/src/Api/Device.hpp +++ b/kernel/src/Api/Device.hpp @@ -6,8 +6,6 @@ #pragma once #include -#include -#include #include #include #include @@ -18,6 +16,9 @@ #include #include #include +#include +#include +#include #include "Syscall.hpp" #include @@ -69,28 +70,21 @@ namespace montauk::abi { }; // CPU cores (category 0) - int cpuCount = Hal::GetDetectedCpuCount(); + int cpuCount = Smp::GetDetectedCpuCount(); if (cpuCount > 0) { char detail[48]; int p = 0; - p = dl_append(detail, p, "x86_64, ", 48); + p = dl_append(detail, p, MONTAUK_ARCH_NAME ", ", 48); p = dl_append_dec(detail, p, cpuCount, 48); p = dl_append(detail, p, " core(s)", 48); add(0, "Processor", detail); } - // Interrupt controllers (category 1) - add(1, "Local APIC", ""); - add(1, "I/O APIC", ""); - - // Timer (category 2) - add(2, "LAPIC Timer", "Local APIC periodic timer"); - - // PS/2 Input (category 3) - add(3, "PS/2 Keyboard", "IRQ 1"); - if (Drivers::PS2::IsDualChannel()) { - add(3, "PS/2 Mouse", "IRQ 12"); - } + // Interrupt controllers (category 1), timer (2), platform input (3) + Arch::ListPlatformDevices([](void* ctx, uint8_t category, + const char* name, const char* detail) { + (*(decltype(add)*)ctx)(category, name, detail); + }, &add); // USB devices (category 4) if (Drivers::USB::Xhci::IsInitialized()) { diff --git a/kernel/src/Api/Filesystem.hpp b/kernel/src/Api/Filesystem.hpp index 42627fd..7cd8964 100644 --- a/kernel/src/Api/Filesystem.hpp +++ b/kernel/src/Api/Filesystem.hpp @@ -94,14 +94,14 @@ namespace montauk::abi { proc->readdirCursor = (slot + 1) % Sched::UserReadDirSlots; uint64_t userVa = Sched::UserReadDirBase + (uint64_t)slot * 0x1000ULL; - uint64_t physAddr = Memory::VMM::Paging::GetPhysAddr(proc->pml4Phys, userVa); + uint64_t physAddr = Memory::VMM::Paging::GetPhysAddr(proc->pageTableRoot, userVa); uint8_t* pageBuf = nullptr; if (physAddr == 0) { void* page = Memory::g_pfa->AllocateZeroed(); if (page == nullptr) return OutOfMemory; physAddr = Memory::SubHHDM((uint64_t)page); - if (!Memory::VMM::Paging::MapUserIn(proc->pml4Phys, physAddr, userVa)) { + if (!Memory::VMM::Paging::MapUserIn(proc->pageTableRoot, physAddr, userVa)) { Memory::g_pfa->Free(page); return OutOfMemory; } diff --git a/kernel/src/Api/Graphics.hpp b/kernel/src/Api/Graphics.hpp index 7731807..4deff9e 100644 --- a/kernel/src/Api/Graphics.hpp +++ b/kernel/src/Api/Graphics.hpp @@ -83,7 +83,7 @@ namespace montauk::abi { for (uint64_t i = 0; i < numPages; i++) { if (!Memory::VMM::Paging::MapUserInWC( - proc->pml4Phys, + proc->pageTableRoot, fbPhys + i * 0x1000, userVa + i * 0x1000 )) { @@ -98,7 +98,7 @@ namespace montauk::abi { uint64_t buf1Phys = Drivers::Graphics::IntelGPU::GetBufferPhys(1); for (uint64_t i = 0; i < numPages; i++) { if (!Memory::VMM::Paging::MapUserInWC( - proc->pml4Phys, + proc->pageTableRoot, buf1Phys + i * 0x1000, userVa + (numPages + i) * 0x1000 )) { diff --git a/kernel/src/Api/Heap.hpp b/kernel/src/Api/Heap.hpp index 4d8ae64..f1a0a72 100644 --- a/kernel/src/Api/Heap.hpp +++ b/kernel/src/Api/Heap.hpp @@ -107,7 +107,7 @@ namespace montauk::abi { if (page == nullptr) break; uint64_t phys = Memory::SubHHDM((uint64_t)page); if (!Memory::VMM::Paging::MapUserInPermissions( - proc->pml4Phys, phys, pageVa, writable, executable)) { + proc->pageTableRoot, phys, pageVa, writable, executable)) { Memory::g_pfa->Free(page); break; } @@ -131,7 +131,7 @@ namespace montauk::abi { // Reset heap allocation tracking for a process slot. // The actual physical pages are freed by Paging::FreeUserHalf() during process cleanup. - inline void CleanupHeapForSlot(int slot, uint64_t /*pml4Phys*/) { + inline void CleanupHeapForSlot(int slot, uint64_t /*pageTableRoot*/) { if (slot < 0 || slot >= Sched::MaxProcesses) return; g_heapLocks[slot].Acquire(); HeapAlloc* area = g_heapAllocs[slot]; @@ -182,7 +182,7 @@ namespace montauk::abi { } for (uint64_t i = 0; i < area->numPages; i++) if (Memory::VMM::Paging::GetPhysAddr( - proc->pml4Phys, area->va + i * 0x1000ULL) != 0) + proc->pageTableRoot, area->va + i * 0x1000ULL) != 0) resident++; *link = area->next; area->next = released; @@ -194,7 +194,7 @@ namespace montauk::abi { // user TLB entry can otherwise corrupt the frame's next owner. while (released != nullptr) { HeapAlloc* next = released->next; - Memory::UnmapAndFreeUserRange(proc->pml4Phys, released->va, + Memory::UnmapAndFreeUserRange(proc->pageTableRoot, released->va, released->numPages); Sched::ReleaseUserHeapRange(slot, released->va, released->numPages * 0x1000ULL); @@ -253,11 +253,11 @@ namespace montauk::abi { uint64_t resident = 0; for (uint64_t i = 0; i < pages; i++) - if (Memory::VMM::Paging::GetPhysAddr(proc->pml4Phys, addr + i * 0x1000ULL) != 0) + if (Memory::VMM::Paging::GetPhysAddr(proc->pageTableRoot, addr + i * 0x1000ULL) != 0) resident++; g_heapLocks[slot].Release(); - Memory::UnmapAndFreeUserRange(proc->pml4Phys, addr, pages); + Memory::UnmapAndFreeUserRange(proc->pageTableRoot, addr, pages); Sched::ReleaseUserHeapRange(slot, addr, size); g_heapLocks[slot].Acquire(); Sched::g_allocatedPages[slot] -= resident; @@ -336,10 +336,10 @@ namespace montauk::abi { bool writable = (prot & VmProtWrite) != 0; bool executable = (prot & VmProtExec) != 0; for (uint64_t i = 0; i < pages; i++) { - if (Memory::VMM::Paging::GetPhysAddr(proc->pml4Phys, + if (Memory::VMM::Paging::GetPhysAddr(proc->pageTableRoot, addr + i * 0x1000ULL) == 0) continue; - if (!Memory::VMM::Paging::ProtectUserIn(proc->pml4Phys, + if (!Memory::VMM::Paging::ProtectUserIn(proc->pageTableRoot, addr + i * 0x1000ULL, writable, executable)) { g_heapLocks[slot].Release(); @@ -347,17 +347,17 @@ namespace montauk::abi { } } g_heapLocks[slot].Release(); - Memory::ShootdownUserRange(proc->pml4Phys, addr, (uint32_t)pages); + Memory::ShootdownUserRange(proc->pageTableRoot, addr, (uint32_t)pages); return Success; } // Resolve a non-present access inside an anonymous VMA. This is used for // both ring-3 faults and kernel copies to untouched userspace buffers. - inline bool TryHandleAnonymousPageFault(uint64_t faultAddr, uint64_t errorCode) { + inline bool TryHandleAnonymousPageFault(uint64_t faultAddr, const Memory::VMM::PageFault& fault) { auto* proc = Sched::GetCurrentProcessPtr(); int slot = GetCurrentSlot(); if (proc == nullptr || slot < 0 || faultAddr < Sched::UserHeapBase || - faultAddr >= Sched::UserHeapLimit || (errorCode & 1) != 0) return false; + faultAddr >= Sched::UserHeapLimit || fault.present) return false; uint64_t pageVa = faultAddr & ~0xFFFULL; g_heapLocks[slot].Acquire(); @@ -370,14 +370,14 @@ namespace montauk::abi { } } if (found == nullptr || - ((errorCode & (1ULL << 1)) != 0 && (found->prot & VmProtWrite) == 0) || - ((errorCode & (1ULL << 4)) != 0 && (found->prot & VmProtExec) == 0)) { + (fault.write && (found->prot & VmProtWrite) == 0) || + (fault.execute && (found->prot & VmProtExec) == 0)) { g_heapLocks[slot].Release(); return false; } // A sibling may have resolved the same fault while this CPU waited. - if (Memory::VMM::Paging::GetPhysAddr(proc->pml4Phys, pageVa) != 0) { + if (Memory::VMM::Paging::GetPhysAddr(proc->pageTableRoot, pageVa) != 0) { g_heapLocks[slot].Release(); return true; } @@ -389,7 +389,7 @@ namespace montauk::abi { } uint64_t phys = Memory::SubHHDM((uint64_t)page); bool mapped = Memory::VMM::Paging::MapUserInPermissions( - proc->pml4Phys, phys, pageVa, + proc->pageTableRoot, phys, pageVa, (found->prot & VmProtWrite) != 0, (found->prot & VmProtExec) != 0); if (!mapped) Memory::g_pfa->Free(page); else Sched::g_allocatedPages[slot]++; diff --git a/kernel/src/Api/LibSyscall.hpp b/kernel/src/Api/LibSyscall.hpp index 19d4853..8aad684 100644 --- a/kernel/src/Api/LibSyscall.hpp +++ b/kernel/src/Api/LibSyscall.hpp @@ -91,7 +91,7 @@ namespace montauk::abi { } // Load the library into the process's address space - uint64_t loadBias = Sched::ElfLoadLib(path, proc->pml4Phys, libSlot); + uint64_t loadBias = Sched::ElfLoadLib(path, proc->pageTableRoot, libSlot); if (loadBias == 0) { Kt::KernelLogStream(Kt::ERROR, "Lib") << "Failed to load library: " << path; g_libLocks[slot].Release(); @@ -134,7 +134,7 @@ namespace montauk::abi { auto* proc = Sched::GetCurrentProcessPtr(); if (proc != nullptr) { - Memory::UnmapAndFreeUserRange(proc->pml4Phys, libBase, + Memory::UnmapAndFreeUserRange(proc->pageTableRoot, libBase, (libEnd - libBase) / 0x1000ULL); } @@ -203,7 +203,7 @@ namespace montauk::abi { uint64_t libBase = GetLibSlotBase(i); uint64_t libEnd = libBase + Sched::LIB_MAX_SIZE; - Memory::UnmapAndFreeUserRange(proc->pml4Phys, libBase, + Memory::UnmapAndFreeUserRange(proc->pageTableRoot, libBase, (libEnd - libBase) / 0x1000ULL); g_libTable[slot][i].inUse = false; diff --git a/kernel/src/Api/Net.hpp b/kernel/src/Api/Net.hpp index 37a3bc6..cbf9818 100644 --- a/kernel/src/Api/Net.hpp +++ b/kernel/src/Api/Net.hpp @@ -9,7 +9,7 @@ #pragma once #include #include -#include +#include #include #include #include diff --git a/kernel/src/Api/Power.hpp b/kernel/src/Api/Power.hpp index a9995d9..d170117 100644 --- a/kernel/src/Api/Power.hpp +++ b/kernel/src/Api/Power.hpp @@ -8,13 +8,13 @@ #include #include #include -#include #include #include #include -#include #include -#include +#include +#include +#include #include #include #include @@ -56,15 +56,13 @@ namespace montauk::abi { Kt::KernelLogStream(Kt::INFO, "Power") << "pid " << kcp::dec << (uint64_t)Sched::GetCurrentPid() << " entered final reboot stage"; if (Efi::g_ResetSystem) { - /* Switch to kernel PML4 which has identity-mapped UEFI runtime regions */ - Memory::VMM::LoadCR3(Memory::VMM::g_paging->PML4); + /* Switch to the kernel address space, which has identity-mapped UEFI runtime regions */ + Memory::VMM::ActivateKernelAddressSpace(); Efi::g_ResetSystem(Efi::EfiResetCold, 0, 0, nullptr); } - /* Fallback: triple fault via null IDT */ - struct [[gnu::packed]] { uint16_t limit; uint64_t base; } nullIdt = {0, 0}; - asm volatile("lidt %0; int $0x03" :: "m"(nullIdt)); - __builtin_unreachable(); + /* Fallback: architecture-specific reset */ + Arch::Reboot(); } static void Sys_Shutdown() { @@ -77,13 +75,12 @@ namespace montauk::abi { /* Fallback: EFI ResetSystem */ if (Efi::g_ResetSystem) { - Memory::VMM::LoadCR3(Memory::VMM::g_paging->PML4); + Memory::VMM::ActivateKernelAddressSpace(); Efi::g_ResetSystem(Efi::EfiResetShutdown, 0, 0, nullptr); } /* Last resort: halt the CPU */ - asm volatile("cli; hlt"); - __builtin_unreachable(); + Arch::HaltForever(); } static int64_t Sys_Suspend() { @@ -96,8 +93,8 @@ namespace montauk::abi { // SYS_POWERINFO. Fills a CPU power/thermal snapshot. Returns -1 when the // CPU exposes no power management features (e.g. under QEMU). static int64_t Sys_PowerInfo(PowerInfo* out) { - Hal::CpuPower::Snapshot s{}; - if (!Hal::CpuPower::GetSnapshot(s)) { + Arch::CpuPower::Snapshot s{}; + if (!Arch::CpuPower::GetSnapshot(s)) { return NoDevice; } @@ -112,7 +109,7 @@ namespace montauk::abi { out->baseMHz = s.BaseMHz; out->maxMHz = s.MaxMHz; out->effMHz = s.EffMHz; - out->apIdleHint = Hal::CpuIdle::GetApIdleHint(); + out->apIdleHint = s.ApIdleHint; return Success; } diff --git a/kernel/src/Api/Process.hpp b/kernel/src/Api/Process.hpp index 6009de5..2cf4c76 100644 --- a/kernel/src/Api/Process.hpp +++ b/kernel/src/Api/Process.hpp @@ -7,7 +7,7 @@ #pragma once #include -#include +#include #include #include #include diff --git a/kernel/src/Api/Random.hpp b/kernel/src/Api/Random.hpp index 7655459..f906d4c 100644 --- a/kernel/src/Api/Random.hpp +++ b/kernel/src/Api/Random.hpp @@ -6,23 +6,23 @@ #pragma once #include +#include namespace montauk::abi { // ---- Random number generation ---- - // Uses RDTSC mixed with xorshift64* PRNG for entropy. - // RDRAND is intentionally avoided: some firmware disables the RDRAND - // hardware unit while CPUID still advertises support (bit 30 of ECX), - // causing #UD on real hardware. RDTSC-based entropy is sufficient for - // seeding BearSSL's PRNG for TLS session keys. + // Uses the CPU timestamp counter mixed with xorshift64* PRNG for entropy. + // RDRAND is intentionally avoided on x86: some firmware disables the + // RDRAND hardware unit while CPUID still advertises support (bit 30 of + // ECX), causing #UD on real hardware. Timestamp-based entropy is + // sufficient for seeding BearSSL's PRNG for TLS session keys. static int64_t Sys_GetRandom(uint8_t* buf, uint64_t len) { - uint64_t tsc; - asm volatile("rdtsc; shl $32, %%rdx; or %%rdx, %%rax" : "=a"(tsc) :: "rdx"); + uint64_t tsc = Arch::ReadTimestamp(); uint64_t state = tsc; for (uint64_t i = 0; i < len; i += 8) { - asm volatile("rdtsc; shl $32, %%rdx; or %%rdx, %%rax" : "=a"(tsc) :: "rdx"); + tsc = Arch::ReadTimestamp(); state ^= tsc; state ^= state >> 12; state ^= state << 25; diff --git a/kernel/src/Api/Syscall.cpp b/kernel/src/Api/Syscall.cpp index 67db2a6..756bbe6 100644 --- a/kernel/src/Api/Syscall.cpp +++ b/kernel/src/Api/Syscall.cpp @@ -6,8 +6,6 @@ #include "Syscall.hpp" #include -#include -#include /* For common functions used by multiple syscall implementations*/ #include "Common.hpp" @@ -42,9 +40,6 @@ #include "Clipboard.hpp" // SYS_CLIPBOARD_* (text and multi-format) #include "UserMemory.hpp" -// Assembly entry point -extern "C" void SyscallEntry(); - namespace montauk::abi { static constexpr uint64_t kMaxPrintableStringBytes = 4096; @@ -796,29 +791,13 @@ namespace montauk::abi { } } - // ---- SYSCALL MSR initialization ---- + // ---- Entry point initialization ---- void InitializeSyscalls() { - // Enable SYSCALL/SYSRET and no-execute page permissions in EFER. - uint64_t efer = Hal::ReadMSR(Hal::IA32_EFER); - efer |= 1 | (1ULL << 11); // SCE | NXE - Hal::WriteMSR(Hal::IA32_EFER, efer); + Arch::InitializeSyscallEntry(); - // STAR: kernel CS in [47:32], sysret base in [63:48] - // SYSCALL: CS=0x08, SS=0x10 - // SYSRET: CS=0x10+16=0x20|RPL3=0x23, SS=0x10+8=0x18|RPL3=0x1B - uint64_t star = (0x0010ULL << 48) | (0x0008ULL << 32); - Hal::WriteMSR(Hal::IA32_STAR, star); - - // LSTAR: SYSCALL entry point - Hal::WriteMSR(Hal::IA32_LSTAR, (uint64_t)SyscallEntry); - - // FMASK: mask IF on SYSCALL entry (bit 9 = IF) - Hal::WriteMSR(Hal::IA32_FMASK, 0x200); - - Kt::KernelLogStream(Kt::OK, "Syscall") << "SYSCALL/SYSRET initialized (LSTAR=" - << kcp::hex << (uint64_t)SyscallEntry << kcp::dec << ", " - << (SYS_SPAWN_REDIR_CAPS + 1) << " syscall slots)"; + Kt::KernelLogStream(Kt::OK, "Syscall") << "System call entry installed (" + << kcp::dec << (SYS_SPAWN_REDIR_CAPS + 1) << " syscall slots)"; } } diff --git a/kernel/src/Api/Syscall.hpp b/kernel/src/Api/Syscall.hpp index dd3b592..02e6e13 100644 --- a/kernel/src/Api/Syscall.hpp +++ b/kernel/src/Api/Syscall.hpp @@ -8,6 +8,7 @@ #include #include #include +#include namespace montauk::abi { @@ -1229,15 +1230,11 @@ namespace montauk::abi { uint64_t timestampTick; }; - // Stack frame pushed by SyscallEntry.asm - struct SyscallFrame { - uint64_t r15, r14, r13, r12, rbp, rbx; // callee-saved - uint64_t arg6, arg5, arg4, arg3, arg2, arg1; - uint64_t syscall_nr; - uint64_t user_rflags, user_rip, user_rsp; - }; + // Saved user context of a system call; its layout is the + // architecture's (see Arch/Syscall.hpp). + using SyscallFrame = Arch::SyscallFrame; - // Kernel-only: set up SYSCALL MSRs and initialize dispatch + // Kernel-only: install the system-call entry point and initialize dispatch void InitializeSyscalls(); } diff --git a/kernel/src/Api/Time.hpp b/kernel/src/Api/Time.hpp index 880d272..c334833 100644 --- a/kernel/src/Api/Time.hpp +++ b/kernel/src/Api/Time.hpp @@ -5,7 +5,7 @@ */ #pragma once -#include +#include #include #include "Syscall.hpp" diff --git a/kernel/src/Api/UserMemory.hpp b/kernel/src/Api/UserMemory.hpp index 8c67b32..fd5b8d2 100644 --- a/kernel/src/Api/UserMemory.hpp +++ b/kernel/src/Api/UserMemory.hpp @@ -19,9 +19,9 @@ namespace montauk::abi::UserMemory { return addr != 0 && addr < USER_SPACE_END; } - inline uint64_t CurrentPml4() { + inline uint64_t CurrentRoot() { auto* proc = Sched::GetCurrentProcessPtr(); - return proc ? proc->pml4Phys : 0; + return proc ? proc->pageTableRoot : 0; } inline bool Range(uint64_t addr, uint64_t size, bool requireWrite = false) { @@ -30,10 +30,10 @@ namespace montauk::abi::UserMemory { if (addr + size < addr) return false; if (addr + size > USER_SPACE_END) return false; - uint64_t pml4Phys = CurrentPml4(); - if (pml4Phys == 0) return false; + uint64_t pageTableRoot = CurrentRoot(); + if (pageTableRoot == 0) return false; - if (Memory::VMM::Paging::IsUserRangeAccessible(pml4Phys, addr, size, requireWrite)) + if (Memory::VMM::Paging::IsUserRangeAccessible(pageTableRoot, addr, size, requireWrite)) return true; // Anonymous VMAs are demand-committed. Validation precedes the actual @@ -42,13 +42,13 @@ namespace montauk::abi::UserMemory { uint64_t page = addr & ~0xFFFULL; uint64_t last = (addr + size - 1) & ~0xFFFULL; for (;;) { - if (!Memory::VMM::Paging::IsUserRangeAccessible(pml4Phys, page, 1, requireWrite) && - !TryHandleAnonymousPageFault(page, requireWrite ? 2ULL : 0ULL)) + if (!Memory::VMM::Paging::IsUserRangeAccessible(pageTableRoot, page, 1, requireWrite) && + !TryHandleAnonymousPageFault(page, {false, requireWrite, false})) return false; if (page == last) break; page += 0x1000ULL; } - return Memory::VMM::Paging::IsUserRangeAccessible(pml4Phys, addr, size, requireWrite); + return Memory::VMM::Paging::IsUserRangeAccessible(pageTableRoot, addr, size, requireWrite); } inline bool OptionalRange(uint64_t addr, uint64_t size, bool requireWrite = false) { @@ -68,13 +68,13 @@ namespace montauk::abi::UserMemory { inline bool String(uint64_t addr, uint64_t maxLen) { if (!ValidUserPtr(addr) || maxLen == 0) return false; - uint64_t pml4Phys = CurrentPml4(); - if (pml4Phys == 0) return false; + uint64_t pageTableRoot = CurrentRoot(); + if (pageTableRoot == 0) return false; const char* str = (const char*)addr; for (uint64_t i = 0; i < maxLen; i++) { - if (!Memory::VMM::Paging::IsUserRangeAccessible(pml4Phys, addr + i, 1, false)) { - if (!TryHandleAnonymousPageFault(addr + i, 0)) return false; + if (!Memory::VMM::Paging::IsUserRangeAccessible(pageTableRoot, addr + i, 1, false)) { + if (!TryHandleAnonymousPageFault(addr + i, {false, false, false})) return false; } if (str[i] == '\0') return true; } diff --git a/kernel/src/Api/WinServer.cpp b/kernel/src/Api/WinServer.cpp index 8909ae6..34eb073 100644 --- a/kernel/src/Api/WinServer.cpp +++ b/kernel/src/Api/WinServer.cpp @@ -54,13 +54,13 @@ namespace WinServer { Ipc::ReleaseSurface(surface); } - static int UnmapRetiredSnapshotsLocked(int windowId, int callerPid, uint64_t callerPml4) { + static int UnmapRetiredSnapshotsLocked(int windowId, int callerPid, uint64_t callerRoot) { int result = InvalidArgument; for (int i = 0; i < MaxRetiredSnapshots; i++) { if (!g_retiredSnapshots[i].used || g_retiredSnapshots[i].windowId != windowId) continue; if (g_retiredSnapshots[i].surface != nullptr) { - if (Ipc::UnmapSurfaceForPid(g_retiredSnapshots[i].surface, callerPid, callerPml4) == 0) { + if (Ipc::UnmapSurfaceForPid(g_retiredSnapshots[i].surface, callerPid, callerRoot) == 0) { result = Success; } Ipc::ReleaseSurface(g_retiredSnapshots[i].surface); @@ -84,7 +84,7 @@ namespace WinServer { if (unmapOwner && slot.liveSurface != nullptr) { auto* owner = Sched::GetProcessByPid(slot.ownerPid); if (owner != nullptr) { - Ipc::UnmapSurfaceForPid(slot.liveSurface, slot.ownerPid, owner->pml4Phys); + Ipc::UnmapSurfaceForPid(slot.liveSurface, slot.ownerPid, owner->pageTableRoot); } } @@ -157,7 +157,7 @@ namespace WinServer { return rc > 0 ? Success : Busy; } - int Create(int ownerPid, uint64_t ownerPml4, const char* title, int w, int h, + int Create(int ownerPid, uint64_t ownerRoot, const char* title, int w, int h, uint64_t& heapNext, uint64_t& outVa) { WsGuard guard; @@ -194,7 +194,7 @@ namespace WinServer { FillSurfaceLocked(snapshot, w, h, InitialWindowPixel); uint64_t userVa = 0; - if (Ipc::MapSurfaceForPid(live, ownerPid, ownerPml4, heapNext, userVa, false) != 0) { + if (Ipc::MapSurfaceForPid(live, ownerPid, ownerRoot, heapNext, userVa, false) != 0) { Ipc::ReleaseMailbox(mailbox, true, true); Ipc::ReleaseSurface(snapshot); Ipc::ReleaseSurface(live); @@ -287,7 +287,7 @@ namespace WinServer { return count; } - uint64_t Map(int windowId, int callerPid, uint64_t callerPml4, uint64_t& heapNext) { + uint64_t Map(int windowId, int callerPid, uint64_t callerRoot, uint64_t& heapNext) { WsGuard guard; if (windowId < 0 || windowId >= MaxWindows) return 0; @@ -295,23 +295,23 @@ namespace WinServer { if (!slot.used || slot.snapshotSurface == nullptr) return 0; uint64_t outVa = 0; - if (Ipc::MapSurfaceForPid(slot.snapshotSurface, callerPid, callerPml4, heapNext, outVa, false) != 0) { + if (Ipc::MapSurfaceForPid(slot.snapshotSurface, callerPid, callerRoot, heapNext, outVa, false) != 0) { return 0; } return outVa; } - int Unmap(int windowId, int callerPid, uint64_t callerPml4) { + int Unmap(int windowId, int callerPid, uint64_t callerRoot) { WsGuard guard; if (windowId < 0 || windowId >= MaxWindows) { - return UnmapRetiredSnapshotsLocked(windowId, callerPid, callerPml4); + return UnmapRetiredSnapshotsLocked(windowId, callerPid, callerRoot); } - int result = UnmapRetiredSnapshotsLocked(windowId, callerPid, callerPml4); + int result = UnmapRetiredSnapshotsLocked(windowId, callerPid, callerRoot); WindowSlot& slot = g_slots[windowId]; if (!slot.used || slot.snapshotSurface == nullptr) return result; - int current = Ipc::UnmapSurfaceForPid(slot.snapshotSurface, callerPid, callerPml4); + int current = Ipc::UnmapSurfaceForPid(slot.snapshotSurface, callerPid, callerRoot); if (current == 0) return Success; return result; } @@ -321,7 +321,7 @@ namespace WinServer { return SendEventLocked(windowId, event); } - int Resize(int windowId, int callerPid, uint64_t ownerPml4, int newW, int newH, + int Resize(int windowId, int callerPid, uint64_t ownerRoot, int newW, int newH, uint64_t& heapNext, uint64_t& outVa) { WsGuard guard; if (windowId < 0 || windowId >= MaxWindows) return InvalidArgument; @@ -348,7 +348,7 @@ namespace WinServer { Ipc::RetainSurface(newSnapshot); uint64_t newOwnerVa = 0; - if (Ipc::MapSurfaceForPid(newLive, callerPid, ownerPml4, heapNext, newOwnerVa, false) != 0) { + if (Ipc::MapSurfaceForPid(newLive, callerPid, ownerRoot, heapNext, newOwnerVa, false) != 0) { Ipc::ReleaseSurface(newSnapshot); Ipc::ReleaseSurface(newLive); return GenericFailure; @@ -366,7 +366,7 @@ namespace WinServer { } if (slot.liveSurface != nullptr) { - Ipc::UnmapSurfaceForPid(slot.liveSurface, callerPid, ownerPml4); + Ipc::UnmapSurfaceForPid(slot.liveSurface, callerPid, ownerRoot); Ipc::ReleaseSurface(slot.liveSurface); } if (slot.snapshotSurface != nullptr) { diff --git a/kernel/src/Api/WinServer.hpp b/kernel/src/Api/WinServer.hpp index db87e96..fe7d79a 100644 --- a/kernel/src/Api/WinServer.hpp +++ b/kernel/src/Api/WinServer.hpp @@ -28,16 +28,16 @@ namespace WinServer { uint32_t flags; }; - int Create(int ownerPid, uint64_t ownerPml4, const char* title, int w, int h, + int Create(int ownerPid, uint64_t ownerRoot, const char* title, int w, int h, uint64_t& heapNext, uint64_t& outVa); int Destroy(int windowId, int callerPid); int Present(int windowId, int callerPid); int Poll(int windowId, int callerPid, montauk::abi::WinEvent* outEvent); int Enumerate(montauk::abi::WinInfo* outArray, int maxCount); - uint64_t Map(int windowId, int callerPid, uint64_t callerPml4, uint64_t& heapNext); - int Unmap(int windowId, int callerPid, uint64_t callerPml4); + uint64_t Map(int windowId, int callerPid, uint64_t callerRoot, uint64_t& heapNext); + int Unmap(int windowId, int callerPid, uint64_t callerRoot); int SendEvent(int windowId, const montauk::abi::WinEvent* event); - int Resize(int windowId, int callerPid, uint64_t ownerPml4, int newW, int newH, + int Resize(int windowId, int callerPid, uint64_t ownerRoot, int newW, int newH, uint64_t& heapNext, uint64_t& outVa); void CleanupProcess(int pid); int SetCursor(int windowId, int callerPid, int cursor); diff --git a/kernel/src/Api/Window.hpp b/kernel/src/Api/Window.hpp index 2518ea7..d9012b7 100644 --- a/kernel/src/Api/Window.hpp +++ b/kernel/src/Api/Window.hpp @@ -22,7 +22,7 @@ namespace montauk::abi { if (proc == nullptr) return GenericFailure; uint64_t outVa = 0; - int id = WinServer::Create(proc->pid, proc->pml4Phys, title, w, h, + int id = WinServer::Create(proc->pid, proc->pageTableRoot, title, w, h, proc->heapNext, outVa); result->id = id; result->pixelVa = (id >= 0) ? outVa : 0; @@ -55,13 +55,13 @@ namespace montauk::abi { if (!Sched::HasCapability(CAP_COMPOSITOR)) return 0; auto* proc = Sched::GetCurrentProcessPtr(); if (proc == nullptr) return 0; - return WinServer::Map(windowId, proc->pid, proc->pml4Phys, proc->heapNext); + return WinServer::Map(windowId, proc->pid, proc->pageTableRoot, proc->heapNext); } static int Sys_WinUnmap(int windowId) { auto* proc = Sched::GetCurrentProcessPtr(); if (proc == nullptr) return GenericFailure; - return WinServer::Unmap(windowId, proc->pid, proc->pml4Phys); + return WinServer::Unmap(windowId, proc->pid, proc->pageTableRoot); } static int Sys_WinSendEvent(int windowId, const WinEvent* event) { @@ -74,7 +74,7 @@ namespace montauk::abi { auto* proc = Sched::GetCurrentProcessPtr(); if (proc == nullptr) return 0; uint64_t outVa = 0; - int r = WinServer::Resize(windowId, proc->pid, proc->pml4Phys, newW, newH, + int r = WinServer::Resize(windowId, proc->pid, proc->pageTableRoot, newW, newH, proc->heapNext, outVa); return (r == 0) ? outVa : 0; } diff --git a/kernel/src/Arch/Abi.hpp b/kernel/src/Arch/Abi.hpp new file mode 100644 index 0000000..20b9e32 --- /dev/null +++ b/kernel/src/Arch/Abi.hpp @@ -0,0 +1,64 @@ +/* + * Abi.hpp + * Architecture-specific parts of the user-mode ABI + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include +#include + +// What the ELF loader and the process/thread setup code need to know about +// the user-mode binary interface of the target. Defined inline by +// Arch//Abi.hpp. + +namespace Arch { + + // ELF e_machine value accepted for user programs and libraries, and a + // name for diagnostics. + // constexpr uint16_t ElfMachine; + // constexpr const char* ElfMachineName; + + // Calling-convention attribute for UEFI runtime services: + // #define ARCH_EFIAPI + + // How the loader must treat a dynamic relocation type. + enum class RelocationKind { + None, // no-op (type 0) + Relative, // *target = loadBias + addend + Symbol, // *target = symbol address + addend (64-bit absolute) + Unsupported, + }; + inline RelocationKind ClassifyRelocation(uint32_t type); + + // Geometry of one thread's TLS block, as offsets from its page-aligned + // base. The variant (I: TLS above the thread pointer, II: below it) is + // fixed by the architecture's ELF TLS ABI. + struct TlsLayout { + uint64_t templateOffset; // where .tdata/.tbss start + uint64_t tpOffset; // thread pointer; [TP] holds the self-pointer + uint64_t totalSize; // bytes to map, incl. the TCB + }; + inline TlsLayout ComputeTlsLayout(uint64_t memSize, uint64_t pAlign); + + // Emit the user-mode code that calls SYS_EXIT(0). A process's entry + // function returns into it. + inline void WriteExitStub(uint8_t* page); + + // Prepare the main thread's initial user stack so that returning from + // the entry function lands at `returnAddress`. `topPage` is the kernel + // mapping of the page just below `stackTop`. Returns the initial user SP. + inline uint64_t SetupUserEntryStack(uint8_t* topPage, uint64_t stackTop, + uint64_t returnAddress); + + // Initial user SP for a sibling thread whose stack ends at `stackTop` + // (the thread trampoline never returns, so no return address is set). + inline uint64_t ThreadEntryStackPointer(uint64_t stackTop); + +} + +#if MONTAUK_ARCH_X86_64 +#include +#elif MONTAUK_ARCH_AARCH64 +#include +#endif diff --git a/kernel/src/Arch/Arch.hpp b/kernel/src/Arch/Arch.hpp new file mode 100644 index 0000000..007b476 --- /dev/null +++ b/kernel/src/Arch/Arch.hpp @@ -0,0 +1,25 @@ +/* + * Arch.hpp + * Target architecture selection + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once + +// Everything outside src/Arch/ is architecture-neutral and talks to the CPU +// only through the interfaces in src/Arch/*.hpp. Each supported architecture +// implements them under src/Arch//; the build compiles only the +// directory matching ARCH. See docs/porting.md. +// +// Interface headers that need inline (per-arch) definitions include the +// matching implementation header at their end, selected below. + +#if defined(__x86_64__) +#define MONTAUK_ARCH_X86_64 1 +#define MONTAUK_ARCH_NAME "x86_64" +#elif defined(__aarch64__) +#define MONTAUK_ARCH_AARCH64 1 +#define MONTAUK_ARCH_NAME "aarch64" +#else +#error "MontaukOS: unsupported target architecture (add one under src/Arch/)" +#endif diff --git a/kernel/src/Arch/Context.hpp b/kernel/src/Arch/Context.hpp new file mode 100644 index 0000000..f3ab59d --- /dev/null +++ b/kernel/src/Arch/Context.hpp @@ -0,0 +1,49 @@ +/* + * Context.hpp + * Architecture-neutral thread context interface + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include +#include +#include + +// The scheduler's view of a thread's CPU context: a kernel stack pointer +// saved by SwitchContext, an FPU/SIMD save area, and a page-table root. + +#if MONTAUK_ARCH_X86_64 +#include +#elif MONTAUK_ARCH_AARCH64 +#include +#endif + +// The architecture provides (in Arch//Context.hpp): +// constexpr uint64_t Arch::FpuStateSize, Arch::FpuStateAlign + +namespace Arch { + + // Reset an FPU/SIMD save area to the state a new thread starts with. + void InitFpuState(uint8_t* area); + + // Build the initial kernel stack of a new thread so that the first + // SwitchContext to it "returns" into `entry`. Returns the saved SP. + uint64_t PrepareThreadStack(uint64_t kernelStackTop, void (*entry)()); + + // Save the running context (stack pointer to *oldSp, FPU state to + // oldFpu when non-null), switch to page-table root `newRoot` if it is + // not already active, and resume the context saved at newSp (restoring + // newFpu when non-null). Returns when the old context is resumed. + void SwitchContext(uint64_t* oldSp, uint64_t newSp, uint64_t newRoot, + uint8_t* oldFpu, uint8_t* newFpu); + + // Point this CPU at the thread about to run: the kernel stack used when + // it enters the kernel from user mode, and its TLS thread pointer. + void ActivateThread(Smp::CpuData* cpu, uint64_t kernelStackTop, + uint64_t threadPointer); + + // Drop to user mode at `entry` with stack `userSp`; `arg` is the first + // argument register. Interrupts are enabled in user mode. + [[noreturn]] void EnterUserMode(uint64_t entry, uint64_t userSp, uint64_t arg); + +} diff --git a/kernel/src/Arch/Cpu.hpp b/kernel/src/Arch/Cpu.hpp new file mode 100644 index 0000000..55b83be --- /dev/null +++ b/kernel/src/Arch/Cpu.hpp @@ -0,0 +1,100 @@ +/* + * Cpu.hpp + * Architecture-neutral CPU primitives + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include +#include + +// Every function here is defined inline by the architecture +// (Arch//Cpu.hpp), so the primitives cost what the raw instruction +// costs. Generic kernel code must use these instead of inline assembly. + +namespace Arch { + + // Saved interrupt-enable state, as returned by SaveAndDisableInterrupts. + // Opaque to generic code: only hand it back to RestoreInterrupts. + using IrqFlags = uint64_t; + + // ---- Spinning and barriers ---- + + // Spin-loop hint (x86 PAUSE, ARM YIELD). + inline void Pause(); + + // Stop the compiler from caching memory across this point. Emits no + // instruction; use it in polling loops on device-written memory. + inline void CompilerBarrier() { + asm volatile("" ::: "memory"); + } + + // Order all earlier stores before all later stores, including stores to + // write-combining memory (framebuffers) that must become visible. + inline void StoreFence(); + + // Full read/write memory barrier. + inline void MemoryFence(); + + // ---- Interrupt state ---- + + inline void EnableInterrupts(); + inline void DisableInterrupts(); + inline bool InterruptsEnabled(); + + // Disable interrupts and return the previous state. + inline IrqFlags SaveAndDisableInterrupts(); + + // Restore an interrupt state returned by SaveAndDisableInterrupts. + inline void RestoreInterrupts(IrqFlags flags); + + // ---- Halting ---- + + // Wait for the next interrupt. Interrupt state is left as it was; with + // interrupts disabled this only returns on an NMI-class event. + inline void Halt(); + + // Called with interrupts disabled: atomically enable them and wait for + // an interrupt, then disable them again before returning. The atomicity + // closes the race where a wakeup IRQ lands between the readiness check + // and the wait, leaving the CPU asleep until the next timer deadline. + inline void HaltWithInterruptsDisabled(); + + // Disable interrupts and stop this CPU for good. + [[noreturn]] inline void HaltForever(); + + // ---- Time and delays ---- + + // A free-running, monotonically increasing cycle counter (x86 TSC, ARM + // CNTVCT). Its rate is architecture-defined; use it for entropy and + // relative timing, not wall-clock time. + inline uint64_t ReadTimestamp(); + + // Delay for at least about one microsecond. Works with interrupts + // disabled and before any clock is calibrated; errs long. + inline void IoDelay(); + + // ---- Caches and translation ---- + + // Write back and invalidate all CPU caches (before power state changes). + inline void WritebackInvalidateCaches(); + + // Invalidate this CPU's translation for one virtual page. + inline void FlushTlbPage(uint64_t virtualAddress); + + // Invalidate this CPU's non-global translations. + inline void FlushTlbAll(); + + // Physical address of the active page-table root (x86 CR3, ARM TTBR0). + inline uint64_t ReadPageTableRoot(); + + // Switch to the page-table root at the given physical address. + inline void WritePageTableRoot(uint64_t rootPhys); + +} + +#if MONTAUK_ARCH_X86_64 +#include +#elif MONTAUK_ARCH_AARCH64 +#include +#endif diff --git a/kernel/src/Hal/CpuPower.hpp b/kernel/src/Arch/CpuPower.hpp similarity index 83% rename from kernel/src/Hal/CpuPower.hpp rename to kernel/src/Arch/CpuPower.hpp index 15face5..042f5f3 100644 --- a/kernel/src/Hal/CpuPower.hpp +++ b/kernel/src/Arch/CpuPower.hpp @@ -1,14 +1,18 @@ /* * CpuPower.hpp - * Intel CPU power management: HWP frequency scaling, C1E promotion, - * and the package thermal governor. + * Architecture-neutral CPU power/thermal management interface * Copyright (c) 2026 Daniel Hammer */ #pragma once #include -namespace Hal { +// Frequency scaling, idle-state policy and the thermal governor are CPU +// specific (x86_64: Intel HWP, C1E promotion and a package-temperature +// governor driven through MSRs). An architecture without them implements +// these as no-ops and returns false from GetSnapshot. + +namespace Arch { namespace CpuPower { // Snapshot of the power/thermal state for SYS_POWERINFO and logging. @@ -24,6 +28,7 @@ namespace Hal { uint32_t BaseMHz; // CPUID base frequency (0 = unknown) uint32_t MaxMHz; // CPUID max turbo frequency uint32_t EffMHz; // measured average active frequency (BSP) + uint8_t ApIdleHint; // idle state secondary CPUs enter (arch-defined) }; // BSP: detect features, enable HWP with a balanced energy preference, diff --git a/kernel/src/Arch/Interrupts.hpp b/kernel/src/Arch/Interrupts.hpp new file mode 100644 index 0000000..938e461 --- /dev/null +++ b/kernel/src/Arch/Interrupts.hpp @@ -0,0 +1,70 @@ +/* + * Interrupts.hpp + * Architecture-neutral hardware interrupt interface + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +// Devices and kernel services see interrupts as numbered IRQ slots. The +// architecture maps each slot onto its interrupt controller (x86: an IDT +// vector fed by the I/O APIC, MSIs and local APIC IPIs; ARM: a GIC INTID) +// and calls the registered handler. + +namespace Arch { + + // IRQ handler function type. The parameter is the IRQ slot number. + // `fromUser` is derived directly from the interrupted context. Do not + // cache it in per-CPU data: an IRQ handler may context-switch and later + // resume on another CPU, making per-CPU saved-origin state stale. + using IrqHandler = void(*)(uint8_t irq, bool fromUser); + + // Number of IRQ slots: 0-23 platform interrupt lines, 24-45 message- + // signalled interrupts (one slot per MSI-capable driver), 46-47 IPIs. + constexpr int IRQ_COUNT = 48; + + // The architecture's periodic scheduler/timekeeping timer. + constexpr uint8_t IRQ_TIMER = 0; + + // Platform interrupt lines with PC/ISA meaning. Only drivers for PC + // devices (PS/2, legacy PCI INTx through the I/O APIC) use these. + constexpr uint8_t IRQ_KEYBOARD = 1; + constexpr uint8_t IRQ_CASCADE = 2; + constexpr uint8_t IRQ_COM2 = 3; + constexpr uint8_t IRQ_COM1 = 4; + constexpr uint8_t IRQ_FLOPPY = 6; + constexpr uint8_t IRQ_RTC = 8; + constexpr uint8_t IRQ_MOUSE = 12; + constexpr uint8_t IRQ_ATA1 = 14; + constexpr uint8_t IRQ_ATA2 = 15; + + // Inter-processor interrupts. + constexpr uint8_t IRQ_TLB_SHOOTDOWN = 46; + constexpr uint8_t IRQ_RESCHEDULE = 47; + + // Register the handler for an IRQ slot. + void RegisterIrqHandler(uint8_t irq, IrqHandler handler); + + // Route platform interrupt line `irq` to IRQ slot `irq` on the boot CPU, + // applying the firmware's polarity/trigger overrides, and unmask it. + void RouteLegacyIrq(uint8_t irq); + + // Unmask platform interrupt line `irq` (already routed at boot). + void EnableLegacyIrq(uint8_t irq); + + // A message-signalled interrupt targeting IRQ slot `irq` on the boot + // CPU: the device writes `data` to `address` to raise it. Program these + // into a PCI MSI capability or MSI-X table entry. Returns false when the + // platform cannot receive MSIs (drivers then fall back to INTx/polling). + struct MsiMessage { + uint64_t address; + uint32_t data; + }; + bool ComposeMsi(uint8_t irq, MsiMessage& out); + + // Raise IPI slot `irq` (IRQ_RESCHEDULE / IRQ_TLB_SHOOTDOWN) on the CPU + // with the given Smp index. Returns false if delivery did not complete. + bool SendIpi(int cpuIndex, uint8_t irq); + +} diff --git a/kernel/src/Arch/Platform.hpp b/kernel/src/Arch/Platform.hpp new file mode 100644 index 0000000..2a421fd --- /dev/null +++ b/kernel/src/Arch/Platform.hpp @@ -0,0 +1,54 @@ +/* + * Platform.hpp + * Architecture hooks for kernel bring-up and power-off paths + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include +#include + +// kmain (Boot/Main.cpp) owns the boot sequence and calls into the +// architecture at these points, in this order. + +namespace Arch { + + // First thing after the boot console exists: CPU descriptor tables, + // FPU/SIMD enablement. No memory allocation is available yet. + void EarlyInitialize(); + + // The frame allocator and heap exist: install exception/trap vectors. + void InitializeTraps(); + + // The kernel page tables are live: program memory-type attributes that + // MapWC/MapMMIO rely on (x86 PAT). + void InitializeMemoryAttributes(); + + // ACPI tables are available: bring up interrupt controllers, the boot + // CPU's per-CPU data (Smp::InitBsp), idle-state and power management. + // Interrupts stay disabled; kmain enables them afterwards. + void InitializeInterrupts(Hal::ACPI::CommonSDTHeader* xsdt); + + // Platform devices that are not discovered on PCI (x86: the PS/2 + // controller, keyboard and mouse). Called once the timer is running. + void InitializePlatformDevices(); + + // Right before the first user process is created: anything user-mode + // entry needs (x86: load the task register for ring-3 interrupt stacks). + void PrepareUserMode(); + + // Secondary CPUs are up and the first process is about to be spawned. + void LateInitialize(); + + // Report devices that belong to the platform rather than a bus (for + // SYS_DEVLIST): interrupt controllers (category 1), timers (2) and + // platform input devices (3). + using PlatformDeviceSink = void (*)(void* context, uint8_t category, + const char* name, const char* detail); + void ListPlatformDevices(PlatformDeviceSink sink, void* context); + + // Reset the machine without firmware help (last resort after EFI + // ResetSystem). Never returns. + [[noreturn]] void Reboot(); + +} diff --git a/kernel/src/Arch/Smp.hpp b/kernel/src/Arch/Smp.hpp new file mode 100644 index 0000000..b962cff --- /dev/null +++ b/kernel/src/Arch/Smp.hpp @@ -0,0 +1,94 @@ +/* + * Smp.hpp + * Architecture-neutral per-CPU data and SMP bring-up interface + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include +#include +#include + +#if MONTAUK_ARCH_X86_64 +#include +#elif MONTAUK_ARCH_AARCH64 +#include +#endif + +// The architecture provides (in Arch//PerCpu.hpp): +// struct Arch::CpuEntryState -- fields kernel-entry assembly reads at a +// fixed offset (8) from the per-CPU base +// struct Arch::CpuState -- everything else the port keeps per CPU +// void* Arch::CurrentCpuLocal() -- this CPU's CpuData (fast path) +// void* Arch::TryCurrentCpuLocal() -- same, or nullptr before InitBsp + +namespace Smp { + + static constexpr int MaxCPUs = 64; + + struct CpuData { + // === Fixed layout: read by low-level entry code === + uint64_t selfPtr; // offset 0: pointer to self + Arch::CpuEntryState entry; // offset 8: arch kernel-entry fields + + // === Generic fields === + int32_t currentSlot; // process table slot, -1 = idle + int cpuIndex; // 0-based CPU index (0 = boot CPU) + uint64_t idleSavedSp; // idle context's stack pointer while a process runs + volatile bool started; // set by an AP once its init is complete + + // When set, the scheduler tick must NOT pull this (idle) CPU away to + // run a ready process: its idle context is busy with long-running + // kernel work whose wall-clock-bounded waits would spuriously expire + // if the context only ran when nothing else is ready (observed: the + // deferred Bluetooth firmware download racing login-screen startup). + // Other CPUs schedule normally while this is held. + volatile bool reservedForKernelWork = false; + + // Set by the BSP timer when an IRQ bottom half needs process-safe + // execution even though runnable user processes keep this CPU busy. + // Schedule() consumes it by yielding once to the idle kernel context. + bool kernelWorkPending = false; + + // === Architecture-private state === + Arch::CpuState arch; + }; + + static_assert(__builtin_offsetof(CpuData, selfPtr) == 0, + "CpuData: per-CPU base must point at selfPtr"); + static_assert(__builtin_offsetof(CpuData, entry) == 8, + "CpuData: entry-assembly fields must start at offset 8"); + + // The current CPU's data. + inline CpuData* GetCurrentCpuData() { + return (CpuData*)Arch::CurrentCpuLocal(); + } + + // Exception paths may run before InitBsp() has installed the per-CPU + // base. This variant returns nullptr in that state instead of faulting + // recursively on a bogus base. + inline CpuData* TryGetCurrentCpuData() { + return (CpuData*)Arch::TryCurrentCpuLocal(); + } + + // Get CPU data by index + CpuData* GetCpuData(int index); + + // Number of online CPUs + int GetCpuCount(); + + // Number of CPUs the firmware describes (available once interrupts are + // initialized; may exceed GetCpuCount if some failed to start). + // Implemented by the architecture. + int GetDetectedCpuCount(); + + // Initialize the boot CPU's per-CPU data (call before interrupts are + // enabled). Implemented by the architecture. + void InitBsp(); + + // Boot all secondary CPUs (call after all subsystems are ready). + // `smp` is the boot contract's SMP block; if FeatureSmp was not + // advertised the caller may still pass it (cpuCount<=1 is a no-op). + // Implemented by the architecture. + void BootAPs(const montauk::boot::SmpInfo& smp); +} diff --git a/kernel/src/Arch/Syscall.hpp b/kernel/src/Arch/Syscall.hpp new file mode 100644 index 0000000..c9ef104 --- /dev/null +++ b/kernel/src/Arch/Syscall.hpp @@ -0,0 +1,38 @@ +/* + * Syscall.hpp + * Architecture-neutral system-call entry interface + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include +#include + +#if MONTAUK_ARCH_X86_64 +#include +#elif MONTAUK_ARCH_AARCH64 +#include +#endif + +// The architecture's system-call entry code saves the user context into an +// Arch::SyscallFrame on the thread's kernel stack (Arch// +// SyscallFrame.hpp defines it, exposing at least `syscall_nr` and `arg1` .. +// `arg6`), enables interrupts, and calls +// +// extern "C" int64_t SyscallDispatch(Arch::SyscallFrame* frame); +// +// whose result it returns to user mode. Before returning it calls +// +// extern "C" void SchedExitIfKilled(); +// +// so a kill that arrived during the syscall takes effect at a point where +// the syscall holds no locks. + +namespace Arch { + + // Install the system-call entry point on the calling CPU. Called on the + // boot CPU by montauk::abi::InitializeSyscalls and by every secondary + // CPU during its bring-up. + void InitializeSyscallEntry(); + +} diff --git a/kernel/src/Arch/Timer.hpp b/kernel/src/Arch/Timer.hpp new file mode 100644 index 0000000..dbbade1 --- /dev/null +++ b/kernel/src/Arch/Timer.hpp @@ -0,0 +1,32 @@ +/* + * Timer.hpp + * Architecture-neutral per-CPU tick timer interface + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +// The hardware behind Timekeeping (Timekeeping/Timer.hpp): a per-CPU timer +// that raises Arch::IRQ_TIMER periodically. Timekeeping registers the IRQ +// handler and owns all tick accounting; the architecture only calibrates +// and programs the hardware (x86: local APIC timer calibrated against the +// PIT; ARM: the generic timer). + +namespace Arch::Timer { + + // Calibrate the timer and start the calling (boot) CPU's periodic tick + // with the given period. Returns false if no usable timer was found. + bool InitializeBsp(uint32_t periodMs); + + // (Re)start the calling CPU's periodic tick, reusing the boot CPU's + // calibration: secondary CPU bring-up, leaving idle, and the boot CPU + // after S3 resume (timer state does not survive suspend). Only called + // after InitializeBsp succeeded. + void StartPeriodic(uint32_t periodMs); + + // Mask the calling CPU's tick (idle secondary CPUs rely on IPIs). + // Cheap when it is already masked. + void Stop(); + +} diff --git a/kernel/src/Arch/aarch64/Abi.hpp b/kernel/src/Arch/aarch64/Abi.hpp new file mode 100644 index 0000000..173ab4a --- /dev/null +++ b/kernel/src/Arch/aarch64/Abi.hpp @@ -0,0 +1,55 @@ +/* + * Abi.hpp + * aarch64 user-mode ABI details (implements Arch/Abi.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +// UEFI on AArch64 uses the standard AAPCS64 calling convention. +#define ARCH_EFIAPI + +namespace Arch { + + constexpr uint16_t ElfMachine = 183; // EM_AARCH64 + constexpr const char* ElfMachineName = "aarch64"; + + inline RelocationKind ClassifyRelocation(uint32_t type) { + switch (type) { + case 0: return RelocationKind::None; + case 1027: return RelocationKind::Relative; // R_AARCH64_RELATIVE + case 257: // R_AARCH64_ABS64 + case 1025: // R_AARCH64_GLOB_DAT + case 1026: return RelocationKind::Symbol; // R_AARCH64_JUMP_SLOT + default: return RelocationKind::Unsupported; + } + } + + // Variant I: TP points at a 16-byte TCB and the TLS block follows it, + // starting at the next p_align boundary. + inline TlsLayout ComputeTlsLayout(uint64_t memSize, uint64_t pAlign) { + uint64_t align = pAlign ? pAlign : 1; + uint64_t templateOffset = (16 + align - 1) & ~(align - 1); + return {templateOffset, 0, templateOffset + memSize}; + } + + // System calls: number in x8, arguments in x0-x5, `svc #0`. + inline void WriteExitStub(uint8_t* page) { + uint32_t* code = (uint32_t*)page; + code[0] = 0xD2800000; // mov x0, #0 (exit code 0) + code[1] = 0xD2800008; // mov x8, #0 (SYS_EXIT = 0) + code[2] = 0xD4000001; // svc #0 + } + + // The return address travels in x30, not on the stack: EnterUserMode + // points x30 at the exit stub. The stack only needs 16-byte alignment. + inline uint64_t SetupUserEntryStack(uint8_t*, uint64_t stackTop, uint64_t) { + return stackTop & ~0xFULL; + } + + inline uint64_t ThreadEntryStackPointer(uint64_t stackTop) { + return stackTop & ~0xFULL; + } + +} diff --git a/kernel/src/Arch/aarch64/AcpiSleep.cpp b/kernel/src/Arch/aarch64/AcpiSleep.cpp new file mode 100644 index 0000000..aa9438e --- /dev/null +++ b/kernel/src/Arch/aarch64/AcpiSleep.cpp @@ -0,0 +1,26 @@ +/* + * AcpiSleep.cpp + * aarch64: no ACPI S3 (implements ACPI/AcpiSleep.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#include +#include + +// Suspend on ARM goes through PSCI (SYSTEM_SUSPEND), not the FACS waking +// vector; until that exists, S3 is reported as unavailable. + +namespace Hal::AcpiSleep { + + void Initialize(ACPI::CommonSDTHeader*) { + } + + bool IsS3Available() { + return false; + } + + int Suspend() { + return -1; + } + +} diff --git a/kernel/src/Arch/aarch64/Context.S b/kernel/src/Arch/aarch64/Context.S new file mode 100644 index 0000000..5319714 --- /dev/null +++ b/kernel/src/Arch/aarch64/Context.S @@ -0,0 +1,144 @@ +/* + Context.S + aarch64 context switch and first entry to EL0 (see Arch/Context.hpp) + Copyright (c) 2026 Daniel Hammer +*/ + + /* The kernel is built -mgeneral-regs-only; this file alone touches + the FP/SIMD registers, to save and restore user state. */ + .arch_extension fp + .arch_extension simd + + .text + +/* + * void ArchContextSwitch(uint64_t* oldSp, uint64_t newSp, uint64_t newRoot, + * uint8_t* oldFpu, uint8_t* newFpu) + * x0 = where to save the old SP x1 = SP to resume + * x2 = TTBR0 root for the new side x3/x4 = FP/SIMD save areas (may be 0) + * + * Frame on the kernel stack (Context.cpp's PrepareThreadStack builds the + * same shape): x19..x28, x29, x30 -- 96 bytes. The kernel is built with + * -mgeneral-regs-only, so q0-q31/FPCR/FPSR only ever hold user state. + */ + .global ArchContextSwitch + .type ArchContextSwitch, %function +ArchContextSwitch: + cbz x3, 1f + stp q0, q1, [x3, #0] + stp q2, q3, [x3, #32] + stp q4, q5, [x3, #64] + stp q6, q7, [x3, #96] + stp q8, q9, [x3, #128] + stp q10, q11, [x3, #160] + stp q12, q13, [x3, #192] + stp q14, q15, [x3, #224] + stp q16, q17, [x3, #256] + stp q18, q19, [x3, #288] + stp q20, q21, [x3, #320] + stp q22, q23, [x3, #352] + stp q24, q25, [x3, #384] + stp q26, q27, [x3, #416] + stp q28, q29, [x3, #448] + stp q30, q31, [x3, #480] + mrs x9, fpcr + mrs x10, fpsr + add x11, x3, #512 + stp x9, x10, [x11] +1: + sub sp, sp, #96 + stp x19, x20, [sp, #0] + stp x21, x22, [sp, #16] + stp x23, x24, [sp, #32] + stp x25, x26, [sp, #48] + stp x27, x28, [sp, #64] + stp x29, x30, [sp, #80] + mov x9, sp + str x9, [x0] + + mov sp, x1 + + /* Switch the process half only when it differs (no ASIDs yet, so a + switch invalidates this CPU's TLB). */ + mrs x9, ttbr0_el1 + and x9, x9, #0x0000fffffffffffe + cmp x9, x2 + b.eq 2f + dsb ishst + msr ttbr0_el1, x2 + isb + tlbi vmalle1 + dsb nsh + isb +2: + ldp x19, x20, [sp, #0] + ldp x21, x22, [sp, #16] + ldp x23, x24, [sp, #32] + ldp x25, x26, [sp, #48] + ldp x27, x28, [sp, #64] + ldp x29, x30, [sp, #80] + add sp, sp, #96 + + cbz x4, 3f + ldp q0, q1, [x4, #0] + ldp q2, q3, [x4, #32] + ldp q4, q5, [x4, #64] + ldp q6, q7, [x4, #96] + ldp q8, q9, [x4, #128] + ldp q10, q11, [x4, #160] + ldp q12, q13, [x4, #192] + ldp q14, q15, [x4, #224] + ldp q16, q17, [x4, #256] + ldp q18, q19, [x4, #288] + ldp q20, q21, [x4, #320] + ldp q22, q23, [x4, #352] + ldp q24, q25, [x4, #384] + ldp q26, q27, [x4, #416] + ldp q28, q29, [x4, #448] + ldp q30, q31, [x4, #480] + add x11, x4, #512 + ldp x9, x10, [x11] + msr fpcr, x9 + msr fpsr, x10 +3: + ret + .size ArchContextSwitch, . - ArchContextSwitch + +/* + * void ArchEnterUserMode(uint64_t entry, uint64_t userSp, uint64_t arg, + * uint64_t returnAddress, uint64_t kernelStackTop) + * + * Drops to EL0t with interrupts unmasked. The kernel stack is reset to its + * top: SP_EL1 is what the next exception from EL0 runs on. + */ + .global ArchEnterUserMode + .type ArchEnterUserMode, %function +ArchEnterUserMode: + msr daifset, #0xf + mov sp, x4 + msr elr_el1, x0 + msr sp_el0, x1 + msr spsr_el1, xzr /* EL0t, DAIF clear */ + mov x0, x2 + mov x30, x3 + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + mov x12, xzr + mov x13, xzr + mov x14, xzr + mov x15, xzr + mov x16, xzr + mov x17, xzr + mov x18, xzr + mov x29, xzr + eret + .size ArchEnterUserMode, . - ArchEnterUserMode diff --git a/kernel/src/Arch/aarch64/Context.cpp b/kernel/src/Arch/aarch64/Context.cpp new file mode 100644 index 0000000..f75b8a1 --- /dev/null +++ b/kernel/src/Arch/aarch64/Context.cpp @@ -0,0 +1,53 @@ +/* + * Context.cpp + * aarch64 thread context (implements Arch/Context.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#include +#include +#include + +// Context.S +extern "C" void ArchContextSwitch(uint64_t* oldSp, uint64_t newSp, uint64_t newRoot, + uint8_t* oldFpu, uint8_t* newFpu); +extern "C" [[noreturn]] void ArchEnterUserMode(uint64_t entry, uint64_t userSp, uint64_t arg, + uint64_t returnAddress, uint64_t kernelStackTop); + +namespace Arch { + + void InitFpuState(uint8_t* area) { + // All-zero is the architectural reset state: round-to-nearest, no + // exceptions trapped, no flags raised. + memset(area, 0, FpuStateSize); + } + + uint64_t PrepareThreadStack(uint64_t kernelStackTop, void (*entry)()) { + // The frame Context.S pops: x19..x28, x29, x30 (= return address). + uint64_t* sp = (uint64_t*)kernelStackTop - 12; + for (int i = 0; i < 11; i++) { + sp[i] = 0; + } + sp[11] = (uint64_t)entry; + return (uint64_t)sp; + } + + void SwitchContext(uint64_t* oldSp, uint64_t newSp, uint64_t newRoot, + uint8_t* oldFpu, uint8_t* newFpu) { + ArchContextSwitch(oldSp, newSp, newRoot, oldFpu, newFpu); + } + + void ActivateThread(Smp::CpuData* cpu, uint64_t kernelStackTop, + uint64_t threadPointer) { + cpu->entry.kernelStackTop = kernelStackTop; + asm volatile("msr tpidr_el0, %0" :: "r"(threadPointer)); + } + + void EnterUserMode(uint64_t entry, uint64_t userSp, uint64_t arg) { + // Returning from the entry function lands in the exit stub (see + // SetupUserEntryStack in Abi.hpp). + ArchEnterUserMode(entry, userSp, arg, Sched::ExitStubAddr, + Smp::GetCurrentCpuData()->entry.kernelStackTop); + } + +} diff --git a/kernel/src/Arch/aarch64/Context.hpp b/kernel/src/Arch/aarch64/Context.hpp new file mode 100644 index 0000000..356d4cf --- /dev/null +++ b/kernel/src/Arch/aarch64/Context.hpp @@ -0,0 +1,14 @@ +/* + * Context.hpp + * aarch64 thread context constants (see Arch/Context.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +namespace Arch { + // q0-q31 followed by FPCR and FPSR (Context.S). + constexpr uint64_t FpuStateSize = 32 * 16 + 16; + constexpr uint64_t FpuStateAlign = 16; +} diff --git a/kernel/src/Arch/aarch64/Cpu.hpp b/kernel/src/Arch/aarch64/Cpu.hpp new file mode 100644 index 0000000..99fb158 --- /dev/null +++ b/kernel/src/Arch/aarch64/Cpu.hpp @@ -0,0 +1,120 @@ +/* + * Cpu.hpp + * aarch64 implementation of the Arch/Cpu.hpp primitives + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +namespace Arch { + + // PSTATE.I in the DAIF view + static constexpr uint64_t DaifIrqMask = 1ULL << 7; + + inline void Pause() { + asm volatile("yield" ::: "memory"); + } + + // Device-nGnRE / Normal-NC (write-combining) stores complete in order + // once a DSB has been issued. + inline void StoreFence() { + asm volatile("dsb st" ::: "memory"); + } + + inline void MemoryFence() { + asm volatile("dsb sy" ::: "memory"); + } + + inline void EnableInterrupts() { + asm volatile("msr daifclr, #2" ::: "memory"); + } + + inline void DisableInterrupts() { + asm volatile("msr daifset, #2" ::: "memory"); + } + + inline bool InterruptsEnabled() { + uint64_t daif; + asm volatile("mrs %0, daif" : "=r"(daif)); + return (daif & DaifIrqMask) == 0; + } + + inline IrqFlags SaveAndDisableInterrupts() { + uint64_t daif; + asm volatile("mrs %0, daif\n\tmsr daifset, #2" : "=r"(daif) :: "memory"); + return daif; + } + + inline void RestoreInterrupts(IrqFlags flags) { + asm volatile("msr daif, %0" :: "r"(flags) : "memory"); + } + + inline void Halt() { + asm volatile("wfi" ::: "memory"); + } + + // WFI wakes on a pending interrupt even while PSTATE.I masks it, so + // checking for work and then waiting with IRQs masked cannot lose a + // wakeup. Unmask briefly afterwards so the pending IRQ is taken. + inline void HaltWithInterruptsDisabled() { + asm volatile("wfi\n\tmsr daifclr, #2\n\tisb\n\tmsr daifset, #2" ::: "memory"); + } + + [[noreturn]] inline void HaltForever() { + for (;;) { + asm volatile("msr daifset, #0xf\n\twfi" ::: "memory"); + } + } + + inline uint64_t ReadTimestamp() { + uint64_t value; + asm volatile("isb\n\tmrs %0, cntvct_el0" : "=r"(value) :: "memory"); + return value; + } + + // The generic timer runs from boot at CNTFRQ_EL0 Hz, so it times short + // delays without calibration. + inline void IoDelay() { + uint64_t frequency; + asm volatile("mrs %0, cntfrq_el0" : "=r"(frequency)); + uint64_t ticks = frequency / 1000000 + 1; + uint64_t start = ReadTimestamp(); + while (ReadTimestamp() - start < ticks) { + Pause(); + } + } + + // ARMv8 has no "clean the whole cache" instruction: set/way maintenance + // is only safe from firmware, which PSCI performs on power transitions. + // Order outstanding accesses so the firmware call sees them. + inline void WritebackInvalidateCaches() { + asm volatile("dsb sy\n\tisb" ::: "memory"); + } + + inline void FlushTlbPage(uint64_t virtualAddress) { + asm volatile("dsb ishst\n\ttlbi vaae1is, %0\n\tdsb ish\n\tisb" + :: "r"(virtualAddress >> 12) : "memory"); + } + + inline void FlushTlbAll() { + asm volatile("dsb ishst\n\ttlbi vmalle1is\n\tdsb ish\n\tisb" ::: "memory"); + } + + // The process (lower) half of the address space is translated through + // TTBR0_EL1; the kernel half through TTBR1_EL1, which never changes. + static constexpr uint64_t TtbrAddressMask = 0x0000FFFFFFFFFFFEULL; + + inline uint64_t ReadPageTableRoot() { + uint64_t ttbr; + asm volatile("mrs %0, ttbr0_el1" : "=r"(ttbr)); + return ttbr & TtbrAddressMask; + } + + // No ASIDs yet: every switch invalidates the old process's entries. + inline void WritePageTableRoot(uint64_t rootPhys) { + asm volatile("dsb ishst\n\tmsr ttbr0_el1, %0\n\tisb\n\ttlbi vmalle1\n\tdsb nsh\n\tisb" + :: "r"(rootPhys) : "memory"); + } + +} diff --git a/kernel/src/Arch/aarch64/CpuPower.cpp b/kernel/src/Arch/aarch64/CpuPower.cpp new file mode 100644 index 0000000..c1448aa --- /dev/null +++ b/kernel/src/Arch/aarch64/CpuPower.cpp @@ -0,0 +1,26 @@ +/* + * CpuPower.cpp + * aarch64 CPU power management (implements Arch/CpuPower.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#include + +// No frequency scaling or thermal governor yet (on ARM these come from +// ACPI CPPC / SCMI rather than CPU registers). + +namespace Arch::CpuPower { + + void InitializeBsp() {} + void InitializeAp() {} + void ApplyPolicyIfChanged() {} + void ThermalTick(uint64_t) {} + void ServiceDeferredDiagnostics() {} + void ReapplyAfterWake() {} + + bool GetSnapshot(Snapshot& out) { + out = {}; + return false; + } + +} diff --git a/kernel/src/Arch/aarch64/InterruptFrame.hpp b/kernel/src/Arch/aarch64/InterruptFrame.hpp new file mode 100644 index 0000000..6a785c7 --- /dev/null +++ b/kernel/src/Arch/aarch64/InterruptFrame.hpp @@ -0,0 +1,22 @@ +/* + * InterruptFrame.hpp + * aarch64 exception frame + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +namespace Arch { + + // What the exception vectors save before calling into C++. + struct PanicFrame { + uint64_t x[31]; // x0-x30 + uint64_t sp; // interrupted SP (SP_EL0 for EL0, else SP_EL1) + uint64_t elr; // ELR_EL1: faulting/preferred-return PC + uint64_t spsr; // SPSR_EL1 + uint64_t esr; // ESR_EL1: exception syndrome + uint64_t far; // FAR_EL1: faulting address + }; + +} diff --git a/kernel/src/Arch/aarch64/Interrupts.cpp b/kernel/src/Arch/aarch64/Interrupts.cpp new file mode 100644 index 0000000..0f099e5 --- /dev/null +++ b/kernel/src/Arch/aarch64/Interrupts.cpp @@ -0,0 +1,52 @@ +/* + * Interrupts.cpp + * aarch64 IRQ slot dispatch (implements Arch/Interrupts.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#include +#include +#include + +// IRQ slots are delivered by the interrupt controller driver (GICv2/v3, +// not written yet), which maps INTIDs onto slots and calls ArchIrqDispatch +// from the IRQ exception vector after acknowledging the interrupt. + +namespace Arch { + + static IrqHandler g_irqHandlers[IRQ_COUNT] = {}; + + void RegisterIrqHandler(uint8_t irq, IrqHandler handler) { + if (irq >= IRQ_COUNT) return; + g_irqHandlers[irq] = handler; + Kt::KernelLogStream(Kt::DEBUG, "IRQ") << "Registered handler for IRQ " + << base::dec << (uint64_t)irq; + } + + // TODO(aarch64): GIC SPI routing (ACPI MADT GICD / interrupt overrides). + void RouteLegacyIrq(uint8_t irq) { + Kt::KernelLogStream(Kt::WARNING, "IRQ") << "aarch64: cannot route IRQ " + << base::dec << (uint64_t)irq << " (no GIC driver)"; + } + + void EnableLegacyIrq(uint8_t irq) { + RouteLegacyIrq(irq); + } + + // TODO(aarch64): GICv3 ITS / GICv2m doorbell. + bool ComposeMsi(uint8_t, MsiMessage&) { + return false; + } + + // TODO(aarch64): GIC software-generated interrupts. + bool SendIpi(int, uint8_t) { + return false; + } + +} + +extern "C" void ArchIrqDispatch(uint64_t irq, bool fromUser) { + if (irq < Arch::IRQ_COUNT && Arch::g_irqHandlers[irq] != nullptr) { + Arch::g_irqHandlers[irq]((uint8_t)irq, fromUser); + } +} diff --git a/kernel/src/Arch/aarch64/IoPort.hpp b/kernel/src/Arch/aarch64/IoPort.hpp new file mode 100644 index 0000000..4e0a32a --- /dev/null +++ b/kernel/src/Arch/aarch64/IoPort.hpp @@ -0,0 +1,28 @@ +/* + * IoPort.hpp + * aarch64 port I/O (implements Io/IoPort.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include +#include + +// ARM has no I/O address space. Until a PCI host bridge driver maps its I/O +// aperture, port accesses behave like accesses to an absent device. + +namespace Io { + static constexpr bool HasPortIo = false; + + inline void Out8(uint8_t, uint16_t) {} + inline void Out16(uint16_t, uint16_t) {} + inline void Out32(uint32_t, uint16_t) {} + + inline uint8_t In8(uint16_t) { return 0xFF; } + inline uint16_t In16(uint16_t) { return 0xFFFF; } + inline uint32_t In32(uint16_t) { return 0xFFFFFFFF; } + + inline void IoPortWait() { + Arch::IoDelay(); + } +}; diff --git a/kernel/src/Arch/aarch64/Paging.cpp b/kernel/src/Arch/aarch64/Paging.cpp new file mode 100644 index 0000000..43edb46 --- /dev/null +++ b/kernel/src/Arch/aarch64/Paging.cpp @@ -0,0 +1,110 @@ +/* + * Paging.cpp + * aarch64 virtual memory (implements Memory/Paging.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#include +#include +#include +#include + +// TODO(aarch64): 4 KiB-granule, 48-bit VA translation tables. The kernel +// half lives in TTBR1_EL1 and is never switched; "address space roots" +// handed to the generic kernel are TTBR0_EL1 tables, so +// CreateUserAddressSpace has no kernel half to copy. MAIR_EL1 needs +// Normal-WB, Normal-NC (MapWC) and Device-nGnRE (MapMMIO) indices. + +namespace Memory::VMM { + + Paging* g_paging = nullptr; + + static kcp::Mutex pagingLock; + + void LockUserPaging() { pagingLock.Acquire(); } + void UnlockUserPaging() { pagingLock.Release(); } + + [[noreturn]] static void NotImplemented(const char* what) { + Panic(what, nullptr); + } + + void ActivateKernelAddressSpace() { + Arch::WritePageTableRoot(g_paging->KernelRoot()); + } + + void FlushTLB() { + Arch::FlushTlbAll(); + } + + Paging::Paging() { + NotImplemented("aarch64: virtual memory not implemented"); + } + + void Paging::Init(std::uint64_t, std::uint64_t, const montauk::boot::MemoryMap&, + const montauk::boot::Framebuffer&) { + NotImplemented("aarch64: Paging::Init not implemented"); + } + + std::uint64_t Paging::HhdmLimit() { return 0; } + + void Paging::Map(std::uint64_t, std::uint64_t) { + NotImplemented("aarch64: Paging::Map not implemented"); + } + + void Paging::MapMMIO(std::uint64_t, std::uint64_t) { + NotImplemented("aarch64: Paging::MapMMIO not implemented"); + } + + void Paging::MapWC(std::uint64_t, std::uint64_t) { + NotImplemented("aarch64: Paging::MapWC not implemented"); + } + + void Paging::MapUser(std::uint64_t, std::uint64_t) { + NotImplemented("aarch64: Paging::MapUser not implemented"); + } + + std::uint64_t Paging::GetPhysAddr(std::uint64_t, std::uint64_t) { + NotImplemented("aarch64: Paging::GetPhysAddr not implemented"); + } + + std::uint64_t Paging::GetPhysAddr(std::uint64_t virtualAddress) { + return GetPhysAddr(rootPhys, virtualAddress); + } + + std::uint64_t Paging::CreateUserAddressSpace() { + NotImplemented("aarch64: Paging::CreateUserAddressSpace not implemented"); + } + + bool Paging::MapUserIn(std::uint64_t, std::uint64_t, std::uint64_t) { + NotImplemented("aarch64: Paging::MapUserIn not implemented"); + } + + bool Paging::MapUserInPermissions(std::uint64_t, std::uint64_t, std::uint64_t, bool, bool) { + NotImplemented("aarch64: Paging::MapUserInPermissions not implemented"); + } + + bool Paging::ProtectUserIn(std::uint64_t, std::uint64_t, bool, bool) { + NotImplemented("aarch64: Paging::ProtectUserIn not implemented"); + } + + bool Paging::MapUserInWC(std::uint64_t, std::uint64_t, std::uint64_t) { + NotImplemented("aarch64: Paging::MapUserInWC not implemented"); + } + + void Paging::UnmapUserIn(std::uint64_t, std::uint64_t) { + NotImplemented("aarch64: Paging::UnmapUserIn not implemented"); + } + + void Paging::FreeUserHalf(std::uint64_t) { + NotImplemented("aarch64: Paging::FreeUserHalf not implemented"); + } + + bool Paging::IsUserRangeAccessible(std::uint64_t, std::uint64_t, std::uint64_t, bool) { + NotImplemented("aarch64: Paging::IsUserRangeAccessible not implemented"); + } + + void Paging::MapEfiRuntime(const montauk::boot::EfiInfo&) { + NotImplemented("aarch64: Paging::MapEfiRuntime not implemented"); + } + +} diff --git a/kernel/src/Arch/aarch64/PanicFrame.cpp b/kernel/src/Arch/aarch64/PanicFrame.cpp new file mode 100644 index 0000000..7b19ca9 --- /dev/null +++ b/kernel/src/Arch/aarch64/PanicFrame.cpp @@ -0,0 +1,28 @@ +/* + * PanicFrame.cpp + * aarch64 CPU state for the panic screen + * Copyright (c) 2026 Daniel Hammer +*/ + +#include +#include +#include + +static constexpr int BoxWidth = PanicBoxWidth; + +namespace Arch { + void PrintPanicFrame(PanicFrame* frame) { + PrintBoxedSeparator(kerr, BoxWidth); + PrintBoxedLine(kerr, "CPU State:", BoxWidth, true); + PrintBoxedHex(kerr, "Exception Class (ESR)", frame->esr >> 26, BoxWidth); + PrintBoxedHex(kerr, "Syndrome (ESR)", frame->esr, BoxWidth); + PrintBoxedHex(kerr, "Fault Address (FAR)", frame->far, BoxWidth); + + PrintBoxedSeparator(kerr, BoxWidth); + PrintBoxedLine(kerr, "Registers:", BoxWidth, true); + PrintBoxedHex(kerr, "Program Counter (ELR)", frame->elr, BoxWidth); + PrintBoxedHex(kerr, "PSTATE (SPSR)", frame->spsr, BoxWidth); + PrintBoxedHex(kerr, "Stack Pointer", frame->sp, BoxWidth); + PrintBoxedHex(kerr, "Link Register (x30)", frame->x[30], BoxWidth); + } +} diff --git a/kernel/src/Arch/aarch64/PerCpu.hpp b/kernel/src/Arch/aarch64/PerCpu.hpp new file mode 100644 index 0000000..8514e10 --- /dev/null +++ b/kernel/src/Arch/aarch64/PerCpu.hpp @@ -0,0 +1,34 @@ +/* + * PerCpu.hpp + * aarch64 per-CPU state (see Arch/Smp.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +namespace Arch { + + // Read by the exception vectors through TPIDR_EL1. + struct CpuEntryState { + uint64_t kernelStackTop; // SP_EL1 to use on entry from EL0 + uint64_t scratch; // vector-private scratch slot + }; + + struct CpuState { + uint64_t mpidr; // MPIDR_EL1 affinity of this CPU + }; + + // TPIDR_EL1 holds the CpuData pointer (zeroed in EarlyInitialize until + // Smp::InitBsp installs it). + inline void* CurrentCpuLocal() { + void* ptr; + asm volatile("mrs %0, tpidr_el1" : "=r"(ptr)); + return ptr; + } + + inline void* TryCurrentCpuLocal() { + return CurrentCpuLocal(); + } + +} diff --git a/kernel/src/Arch/aarch64/Platform.cpp b/kernel/src/Arch/aarch64/Platform.cpp new file mode 100644 index 0000000..345ca9b --- /dev/null +++ b/kernel/src/Arch/aarch64/Platform.cpp @@ -0,0 +1,64 @@ +/* + * Platform.cpp + * aarch64 bring-up hooks (implements Arch/Platform.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#include +#include +#include +#include +#include + +using namespace Kt; + +namespace Arch { + + void EarlyInitialize() { + // No per-CPU data yet: Smp::TryGetCurrentCpuData must see null. + asm volatile("msr tpidr_el1, xzr"); + + // CPACR_EL1.FPEN = 0b11: FP/SIMD usable at EL0 and EL1 (the kernel + // itself is built -mgeneral-regs-only; Context.S saves user state). + uint64_t cpacr; + asm volatile("mrs %0, cpacr_el1" : "=r"(cpacr)); + cpacr |= 3ULL << 20; + asm volatile("msr cpacr_el1, %0\n\tisb" :: "r"(cpacr)); + } + + // TODO(aarch64): install the VBAR_EL1 vector table (EL1/EL0 synchronous + // exceptions -> panic / SyscallDispatch, IRQ -> GIC -> ArchIrqDispatch). + void InitializeTraps() { + KernelLogStream(WARNING, "Arch") << "aarch64: exception vectors not implemented"; + } + + // MAIR_EL1 attribute indices are chosen together with the page-table + // format (Paging.cpp), so there is nothing to program separately. + void InitializeMemoryAttributes() { + } + + // TODO(aarch64): GIC distributor/redistributor from the ACPI MADT. + void InitializeInterrupts(Hal::ACPI::CommonSDTHeader*) { + Smp::InitBsp(); + KernelLogStream(WARNING, "Arch") << "aarch64: no interrupt controller driver"; + } + + void InitializePlatformDevices() { + } + + // EL0 entry needs no global state beyond Arch::ActivateThread. + void PrepareUserMode() { + } + + void LateInitialize() { + } + + void ListPlatformDevices(PlatformDeviceSink, void*) { + } + + // TODO(aarch64): PSCI SYSTEM_RESET through the conduit the FADT names. + void Reboot() { + HaltForever(); + } + +} diff --git a/kernel/src/Arch/aarch64/Smp.cpp b/kernel/src/Arch/aarch64/Smp.cpp new file mode 100644 index 0000000..9eec433 --- /dev/null +++ b/kernel/src/Arch/aarch64/Smp.cpp @@ -0,0 +1,60 @@ +/* + * Smp.cpp + * aarch64 per-CPU data (implements the arch half of Arch/Smp.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#include +#include +#include +#include + +using namespace Kt; + +namespace Smp { + + static CpuData g_cpus[MaxCPUs]; + static int g_cpuCount = 0; + static int g_detectedCpuCount = 1; + + CpuData* GetCpuData(int index) { + if (index < 0 || index >= g_cpuCount) return nullptr; + return &g_cpus[index]; + } + + int GetCpuCount() { + return g_cpuCount; + } + + int GetDetectedCpuCount() { + return g_detectedCpuCount; + } + + void InitBsp() { + memset(g_cpus, 0, sizeof(g_cpus)); + + CpuData& bsp = g_cpus[0]; + bsp.selfPtr = (uint64_t)&bsp; + bsp.cpuIndex = 0; + bsp.currentSlot = -1; + bsp.started = true; + asm volatile("mrs %0, mpidr_el1" : "=r"(bsp.arch.mpidr)); + + asm volatile("msr tpidr_el1, %0" :: "r"(&bsp)); + g_cpuCount = 1; + + KernelLogStream(OK, "SMP") << "BSP initialized (MPIDR " << base::hex + << bsp.arch.mpidr << ")"; + } + + void BootAPs(const montauk::boot::SmpInfo& smp) { + if (smp.cpuCount > 1) { + g_detectedCpuCount = (int)smp.cpuCount; + } + + // TODO(aarch64): start secondaries through smp.startCpu once the GIC + // (reschedule/TLB-shootdown SGIs) and the per-CPU generic timer work. + KernelLogStream(WARNING, "SMP") << "aarch64: secondary CPUs not started (" + << base::dec << smp.cpuCount << " reported)"; + } +} diff --git a/kernel/src/Arch/aarch64/Syscall.cpp b/kernel/src/Arch/aarch64/Syscall.cpp new file mode 100644 index 0000000..52da1be --- /dev/null +++ b/kernel/src/Arch/aarch64/Syscall.cpp @@ -0,0 +1,16 @@ +/* + * Syscall.cpp + * aarch64 system-call entry (implements Arch/Syscall.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#include + +namespace Arch { + + // `svc #0` arrives through the EL0 synchronous vector installed by + // InitializeTraps; there is no per-CPU register to program. + void InitializeSyscallEntry() { + } + +} diff --git a/kernel/src/Arch/aarch64/SyscallFrame.hpp b/kernel/src/Arch/aarch64/SyscallFrame.hpp new file mode 100644 index 0000000..d9c1940 --- /dev/null +++ b/kernel/src/Arch/aarch64/SyscallFrame.hpp @@ -0,0 +1,21 @@ +/* + * SyscallFrame.hpp + * aarch64 system-call frame (see Arch/Syscall.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +namespace Arch { + + // Saved by the EL0 synchronous-exception vector for `svc #0`: + // x0-x5 carry the arguments and x8 the system-call number. + struct SyscallFrame { + uint64_t arg1, arg2, arg3, arg4, arg5, arg6; // x0-x5 + uint64_t syscall_nr; // x8 + uint64_t elr, spsr, sp_el0; // user PC, PSTATE, SP + uint64_t x19_x30[12]; // callee-saved + LR + }; + +} diff --git a/kernel/src/Arch/aarch64/Timer.cpp b/kernel/src/Arch/aarch64/Timer.cpp new file mode 100644 index 0000000..7f1d767 --- /dev/null +++ b/kernel/src/Arch/aarch64/Timer.cpp @@ -0,0 +1,24 @@ +/* + * Timer.cpp + * aarch64 tick timer (implements Arch/Timer.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#include + +// TODO(aarch64): drive the EL1 virtual timer (CNTV_TVAL/CNTV_CTL) from +// CNTFRQ_EL0, re-arming on each tick, once the GIC delivers its PPI. + +namespace Arch::Timer { + + bool InitializeBsp(uint32_t) { + return false; + } + + void StartPeriodic(uint32_t) { + } + + void Stop() { + } + +} diff --git a/kernel/src/Arch/x86_64/Abi.hpp b/kernel/src/Arch/x86_64/Abi.hpp new file mode 100644 index 0000000..7ee2c54 --- /dev/null +++ b/kernel/src/Arch/x86_64/Abi.hpp @@ -0,0 +1,62 @@ +/* + * Abi.hpp + * x86_64 user-mode ABI details (implements Arch/Abi.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +// UEFI uses the Microsoft x64 calling convention. +#define ARCH_EFIAPI __attribute__((__ms_abi__)) + +namespace Arch { + + constexpr uint16_t ElfMachine = 62; // EM_X86_64 + constexpr const char* ElfMachineName = "x86_64"; + + inline RelocationKind ClassifyRelocation(uint32_t type) { + switch (type) { + case 0: return RelocationKind::None; + case 8: return RelocationKind::Relative; // R_X86_64_RELATIVE + case 1: // R_X86_64_64 + case 6: // R_X86_64_GLOB_DAT + case 7: return RelocationKind::Symbol; // R_X86_64_JUMP_SLOT + default: return RelocationKind::Unsupported; + } + } + + // Variant II. The linker addresses TLS variables at + // TP - align_up(memSize, p_align) + offset, so the template must start + // exactly that far below TP; rounding p_align up first (as an older + // version did) shifts every variable away from its initial value. + // TP itself is kept at least 16-byte aligned, which the gap between + // base and the template absorbs. The 16 bytes above TP are the TCB. + inline TlsLayout ComputeTlsLayout(uint64_t memSize, uint64_t pAlign) { + uint64_t align = pAlign ? pAlign : 1; + uint64_t block = (memSize + align - 1) & ~(align - 1); + uint64_t tpAlign = align < 16 ? 16 : align; + uint64_t tp = (block + tpAlign - 1) & ~(tpAlign - 1); + return {tp - block, tp, tp + 16}; + } + + inline void WriteExitStub(uint8_t* stub) { + stub[0] = 0x31; stub[1] = 0xFF; // xor edi, edi (exit code 0) + stub[2] = 0x31; stub[3] = 0xC0; // xor eax, eax (SYS_EXIT = 0) + stub[4] = 0x0F; stub[5] = 0x05; // syscall + } + + // The entry function is entered as if CALLed: the return address sits + // at [RSP], with RSP = 16n - 8. + inline uint64_t SetupUserEntryStack(uint8_t* topPage, uint64_t stackTop, + uint64_t returnAddress) { + *(uint64_t*)(topPage + 0xFF8) = returnAddress; + return stackTop - 8; + } + + // SystemV alignment as if CALLed (16n - 8); the slot is never read. + inline uint64_t ThreadEntryStackPointer(uint64_t stackTop) { + return (stackTop & ~0xFULL) - 8; + } + +} diff --git a/kernel/src/ACPI/AcpiSleep.cpp b/kernel/src/Arch/x86_64/Acpi/AcpiSleep.cpp similarity index 96% rename from kernel/src/ACPI/AcpiSleep.cpp rename to kernel/src/Arch/x86_64/Acpi/AcpiSleep.cpp index f8a8b35..8a24a0e 100644 --- a/kernel/src/ACPI/AcpiSleep.cpp +++ b/kernel/src/Arch/x86_64/Acpi/AcpiSleep.cpp @@ -4,8 +4,10 @@ * Copyright (c) 2026 Daniel Hammer */ -#include "AcpiSleep.hpp" -#include "AcpiEvents.hpp" +#include +#include +#include +#include #include #include #include @@ -14,19 +16,19 @@ #include #include #include -#include -#include -#include -#include -#include -#include +#include +#include +#include +#include +#include +#include #include #include #include #include #include -#include -#include +#include +#include #include using namespace Kt; @@ -186,7 +188,7 @@ namespace Hal { << base::hex << (uint64_t)g_s3SlpTypA << " SLP_TYPb=" << base::hex << (uint64_t)g_s3SlpTypB << ")"; KernelLogStream(INFO, "S3") << "Kernel PML4 phys = " << base::hex - << (uint64_t)Memory::VMM::g_paging->PML4; + << Memory::VMM::g_paging->KernelRoot(); KernelLogStream(INFO, "S3") << "FACS at phys " << base::hex << (uint64_t)fadt.FacsAddress << " version=" << base::dec << (uint64_t)g_facs->Version << " flags=" << base::hex @@ -314,7 +316,7 @@ namespace Hal { // shadow copy of the PML4 at a fixed low-memory address. // Only the PML4 page itself needs to be below 4GB — its entries // contain full 52-bit physical addresses for sub-tables. - uint64_t kernelPml4Phys = (uint64_t)Memory::VMM::g_paging->PML4; + uint64_t kernelPml4Phys = Memory::VMM::g_paging->KernelRoot(); uint8_t* kernelPml4Virt = (uint8_t*)Memory::HHDM(kernelPml4Phys); uint8_t* shadowPml4Virt = (uint8_t*)Memory::HHDM((uint64_t)SHADOW_PML4_PHYS); memcpy(shadowPml4Virt, kernelPml4Virt, 4096); @@ -439,7 +441,7 @@ namespace Hal { Hal::InitializePAT(); // HWP enable and POWER_CTL reset across S3; re-apply them and // bump the policy epoch so APs re-apply on their next tick. - Hal::CpuPower::ReapplyAfterWake(); + Arch::CpuPower::ReapplyAfterWake(); reinitProgress(0x05); // PIC Hal::DisableLegacyPic(); reinitProgress(0x06); // Local APIC @@ -447,7 +449,7 @@ namespace Hal { reinitProgress(0x07); // IO APIC Hal::IoApic::Reinitialize(); reinitProgress(0x08); // APIC Timer - Timekeeping::ApicTimerReinitialize(); + Timekeeping::TimerReinitialize(); reinitProgress(0x09); // PM1 clear ClearPM1Status(); reinitProgress(0x0A); // _WAK @@ -463,7 +465,7 @@ namespace Hal { reinitProgress(0x0D); // discrete GPU back off Drivers::Graphics::DgpuPower::Resume(); reinitProgress(0x0E); // sti - asm volatile("sti"); + Arch::EnableInterrupts(); KernelLogStream(OK, "S3") << "Resumed from S3 suspend"; @@ -517,7 +519,7 @@ namespace Hal { Io::Out8(0x00, 0x71); // 3. Disable interrupts - asm volatile("cli"); + Arch::DisableInterrupts(); // 4. Identity-map the trampoline page so the CPU can execute it // after paging is re-enabled with our CR3 during wake. @@ -525,7 +527,7 @@ namespace Hal { Memory::VMM::g_paging->Map(TRAMPOLINE_PHYS, TRAMPOLINE_PHYS); // 5. Flush all caches - asm volatile("wbinvd"); + Arch::WritebackInvalidateCaches(); // 6. Set the waking vector in FACS SetWakingVector(); @@ -548,7 +550,7 @@ namespace Hal { } // 10. Flush caches again - asm volatile("wbinvd"); + Arch::WritebackInvalidateCaches(); // 11. Assert SLP_EN — this triggers S3 entry Io::Out16(pm1a | PM1_SLP_EN, (uint16_t)g_pm1aControlBlock); @@ -563,7 +565,7 @@ namespace Hal { // on QEMU/KVM the vCPU must reach HLT before QEMU finalizes S3. // On resume, firmware jumps to our waking vector. for (;;) { - asm volatile("hlt"); + Arch::Halt(); // If we wake from HLT (spurious or real wake), check WAK_STS if (g_pm1aEventBlock != 0) { uint16_t sts = Io::In16((uint16_t)g_pm1aEventBlock); @@ -574,7 +576,7 @@ namespace Hal { // If we get here, either S3 failed or we woke immediately. // The normal wake path goes through AcpiResumeLongMode instead. // Re-enable interrupts as a safety measure. - asm volatile("sti"); + Arch::EnableInterrupts(); KernelLogStream(WARNING, "S3") << "S3 entry may have failed (fell through)"; return 0; diff --git a/kernel/src/ACPI/CpuIdle.cpp b/kernel/src/Arch/x86_64/Acpi/CpuIdle.cpp similarity index 99% rename from kernel/src/ACPI/CpuIdle.cpp rename to kernel/src/Arch/x86_64/Acpi/CpuIdle.cpp index 9ad326a..90a93cd 100644 --- a/kernel/src/ACPI/CpuIdle.cpp +++ b/kernel/src/Arch/x86_64/Acpi/CpuIdle.cpp @@ -5,10 +5,11 @@ */ #include "CpuIdle.hpp" +#include #include #include -#include -#include +#include +#include #include #include #include @@ -480,7 +481,7 @@ namespace Hal { if (hasMwait) { Hal::IdleWaitWithInterruptsDisabled(addr, CurrentIdleHint()); } else { - Hal::HaltWithInterruptsDisabled(); + Arch::HaltWithInterruptsDisabled(); } } diff --git a/kernel/src/ACPI/CpuIdle.hpp b/kernel/src/Arch/x86_64/Acpi/CpuIdle.hpp similarity index 100% rename from kernel/src/ACPI/CpuIdle.hpp rename to kernel/src/Arch/x86_64/Acpi/CpuIdle.hpp diff --git a/kernel/src/Arch/x86_64/Acpi/S3State.hpp b/kernel/src/Arch/x86_64/Acpi/S3State.hpp new file mode 100644 index 0000000..2fb1d43 --- /dev/null +++ b/kernel/src/Arch/x86_64/Acpi/S3State.hpp @@ -0,0 +1,30 @@ +/* + * S3State.hpp + * CPU state saved across S3 by S3Wake.asm + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +namespace Hal { + namespace AcpiSleep { + + // ============================================================================ + // Layout must match S3Wake.asm offsets exactly. + // No FPU/SSE state — the kernel is compiled with -mno-sse. + struct CpuState { + uint64_t Rax, Rbx, Rcx, Rdx; // 0x00 - 0x18 + uint64_t Rsi, Rdi, Rbp, Rsp; // 0x20 - 0x38 + uint64_t R8, R9, R10, R11; // 0x40 - 0x58 + uint64_t R12, R13, R14, R15; // 0x60 - 0x78 + uint64_t Rflags; // 0x80 + uint64_t Rip; // 0x88 + uint64_t Cr3; // 0x90 + uint64_t Cr0, Cr4; // 0x98, 0xA0 + uint8_t GdtPtr[10]; // 0xA8 + uint8_t IdtPtr[10]; // 0xB2 + } __attribute__((aligned(16))); + + }; +}; diff --git a/kernel/src/ACPI/S3Trampoline.asm b/kernel/src/Arch/x86_64/Acpi/S3Trampoline.asm similarity index 100% rename from kernel/src/ACPI/S3Trampoline.asm rename to kernel/src/Arch/x86_64/Acpi/S3Trampoline.asm diff --git a/kernel/src/ACPI/S3Wake.asm b/kernel/src/Arch/x86_64/Acpi/S3Wake.asm similarity index 100% rename from kernel/src/ACPI/S3Wake.asm rename to kernel/src/Arch/x86_64/Acpi/S3Wake.asm diff --git a/kernel/src/Hal/Apic/Apic.cpp b/kernel/src/Arch/x86_64/Apic/Apic.cpp similarity index 100% rename from kernel/src/Hal/Apic/Apic.cpp rename to kernel/src/Arch/x86_64/Apic/Apic.cpp diff --git a/kernel/src/Hal/Apic/Apic.hpp b/kernel/src/Arch/x86_64/Apic/Apic.hpp similarity index 100% rename from kernel/src/Hal/Apic/Apic.hpp rename to kernel/src/Arch/x86_64/Apic/Apic.hpp diff --git a/kernel/src/Hal/Apic/ApicInit.cpp b/kernel/src/Arch/x86_64/Apic/ApicInit.cpp similarity index 91% rename from kernel/src/Hal/Apic/ApicInit.cpp rename to kernel/src/Arch/x86_64/Apic/ApicInit.cpp index b1fd688..dc4c76e 100644 --- a/kernel/src/Hal/Apic/ApicInit.cpp +++ b/kernel/src/Arch/x86_64/Apic/ApicInit.cpp @@ -10,7 +10,7 @@ #include "IoApic.hpp" #include "Interrupts.hpp" #include -#include +#include #include #include #include @@ -67,8 +67,8 @@ namespace Hal { // Step 7: Route keyboard (IRQ1) and mouse (IRQ12) to BSP uint8_t bspApicId = (uint8_t)LocalApic::GetId(); - IoApic::RouteIrq(IRQ_KEYBOARD, IRQ_VECTOR_BASE + IRQ_KEYBOARD, bspApicId); - IoApic::RouteIrq(IRQ_MOUSE, IRQ_VECTOR_BASE + IRQ_MOUSE, bspApicId); + IoApic::RouteIrq(Arch::IRQ_KEYBOARD, IRQ_VECTOR_BASE + Arch::IRQ_KEYBOARD, bspApicId); + IoApic::RouteIrq(Arch::IRQ_MOUSE, IRQ_VECTOR_BASE + Arch::IRQ_MOUSE, bspApicId); // Note: sti is NOT called here. The caller (Main.cpp) enables // interrupts after setting up per-CPU GS base for SMP/SWAPGS. diff --git a/kernel/src/Hal/Apic/ApicInit.hpp b/kernel/src/Arch/x86_64/Apic/ApicInit.hpp similarity index 100% rename from kernel/src/Hal/Apic/ApicInit.hpp rename to kernel/src/Arch/x86_64/Apic/ApicInit.hpp diff --git a/kernel/src/Hal/Apic/Interrupts.cpp b/kernel/src/Arch/x86_64/Apic/Interrupts.cpp similarity index 57% rename from kernel/src/Hal/Apic/Interrupts.cpp rename to kernel/src/Arch/x86_64/Apic/Interrupts.cpp index 7806645..48c7f9c 100644 --- a/kernel/src/Hal/Apic/Interrupts.cpp +++ b/kernel/src/Arch/x86_64/Apic/Interrupts.cpp @@ -6,7 +6,9 @@ #include "Interrupts.hpp" #include "Apic.hpp" -#include +#include "IoApic.hpp" +#include +#include #include #include @@ -14,19 +16,15 @@ extern "C" void* IrqStubTable[Hal::IRQ_COUNT]; extern "C" void IrqStubSpurious(); +// Local APIC message address range; destination APIC ID in bits 19:12. +static constexpr uint64_t MsiAddressBase = 0xFEE00000; + namespace Hal { // Dispatch table: one handler per IRQ - static IrqHandler g_irqHandlers[IRQ_COUNT] = {}; - - void RegisterIrqHandler(uint8_t irq, IrqHandler handler) { - if (irq >= IRQ_COUNT) return; - g_irqHandlers[irq] = handler; - Kt::KernelLogStream(Kt::DEBUG, "IRQ") << "Registered handler for IRQ " << base::dec << (uint64_t)irq - << " (vector " << (uint64_t)(IRQ_VECTOR_BASE + irq) << ")"; - } + static Arch::IrqHandler g_irqHandlers[IRQ_COUNT] = {}; void InitializeIrqHandlers() { - // Install IRQ stubs into IDT vectors 32..55 + // Install IRQ stubs into IDT vectors 32..79 for (int i = 0; i < IRQ_COUNT; i++) { IDTEncodeInterrupt(IRQ_VECTOR_BASE + i, IrqStubTable[i], 0x8E); } @@ -41,6 +39,41 @@ namespace Hal { }; +namespace Arch { + void RegisterIrqHandler(uint8_t irq, IrqHandler handler) { + if (irq >= IRQ_COUNT) return; + Hal::g_irqHandlers[irq] = handler; + Kt::KernelLogStream(Kt::DEBUG, "IRQ") << "Registered handler for IRQ " << base::dec << (uint64_t)irq + << " (vector " << (uint64_t)(Hal::IRQ_VECTOR_BASE + irq) << ")"; + } + + void RouteLegacyIrq(uint8_t irq) { + // RouteIrq applies the MADT's Interrupt Source Overrides (polarity, + // trigger mode, GSI remapping). + Hal::IoApic::RouteIrq(irq, Hal::IRQ_VECTOR_BASE + irq, + (uint8_t)Hal::LocalApic::GetId()); + } + + void EnableLegacyIrq(uint8_t irq) { + Hal::IoApic::UnmaskIrq(Hal::IoApic::GetGsiForIrq(irq)); + } + + bool ComposeMsi(uint8_t irq, MsiMessage& out) { + // Physical destination mode, fixed delivery, edge triggered, to the + // BSP (APIC ID 0 -- the destination field is left zero, as every + // driver has always programmed it). + out = MsiMessage{ MsiAddressBase, (uint32_t)(Hal::IRQ_VECTOR_BASE + irq) }; + return true; + } + + bool SendIpi(int cpuIndex, uint8_t irq) { + auto* cpu = Smp::GetCpuData(cpuIndex); + if (cpu == nullptr) return false; + return Hal::LocalApic::SendFixedIpi(cpu->arch.lapicId, + Hal::IRQ_VECTOR_BASE + irq); + } +}; + // C linkage dispatch function called from assembly stubs extern "C" void HalIrqDispatch(uint64_t irqNumber, uint64_t savedCs) { // Send EOI BEFORE calling the handler. The handler may context-switch diff --git a/kernel/src/Arch/x86_64/Apic/Interrupts.hpp b/kernel/src/Arch/x86_64/Apic/Interrupts.hpp new file mode 100644 index 0000000..d2789d9 --- /dev/null +++ b/kernel/src/Arch/x86_64/Apic/Interrupts.hpp @@ -0,0 +1,20 @@ +/* + * Interrupts.hpp + * x86_64 IRQ slot <-> IDT vector mapping + * Copyright (c) 2025 Daniel Hammer +*/ + +#pragma once +#include +#include + +namespace Hal { + using Arch::IRQ_COUNT; + + // IRQ vector base: IRQ slot N is delivered on IDT vector 32 + N. + constexpr uint8_t IRQ_VECTOR_BASE = 32; + + // Install IRQ stubs into the IDT and set up the dispatch table + void InitializeIrqHandlers(); + +}; diff --git a/kernel/src/Hal/Apic/IoApic.cpp b/kernel/src/Arch/x86_64/Apic/IoApic.cpp similarity index 100% rename from kernel/src/Hal/Apic/IoApic.cpp rename to kernel/src/Arch/x86_64/Apic/IoApic.cpp diff --git a/kernel/src/Hal/Apic/IoApic.hpp b/kernel/src/Arch/x86_64/Apic/IoApic.hpp similarity index 100% rename from kernel/src/Hal/Apic/IoApic.hpp rename to kernel/src/Arch/x86_64/Apic/IoApic.hpp diff --git a/kernel/src/Hal/Apic/IsrStubs.asm b/kernel/src/Arch/x86_64/Apic/IsrStubs.asm similarity index 100% rename from kernel/src/Hal/Apic/IsrStubs.asm rename to kernel/src/Arch/x86_64/Apic/IsrStubs.asm diff --git a/kernel/src/Hal/Apic/Pic.cpp b/kernel/src/Arch/x86_64/Apic/Pic.cpp similarity index 100% rename from kernel/src/Hal/Apic/Pic.cpp rename to kernel/src/Arch/x86_64/Apic/Pic.cpp diff --git a/kernel/src/Hal/Apic/Pic.hpp b/kernel/src/Arch/x86_64/Apic/Pic.hpp similarity index 100% rename from kernel/src/Hal/Apic/Pic.hpp rename to kernel/src/Arch/x86_64/Apic/Pic.hpp diff --git a/kernel/src/Arch/x86_64/ApicTimer.cpp b/kernel/src/Arch/x86_64/ApicTimer.cpp new file mode 100644 index 0000000..b496fb8 --- /dev/null +++ b/kernel/src/Arch/x86_64/ApicTimer.cpp @@ -0,0 +1,133 @@ +/* + * ApicTimer.cpp + * Local APIC timer, PIT-calibrated (implements Arch/Timer.hpp) + * Copyright (c) 2025 Daniel Hammer +*/ + +#include +#include +#include +#include +#include +#include +#include + +using namespace Kt; + +namespace Arch::Timer { + // PIT constants + static constexpr uint32_t PIT_FREQUENCY = 1193182; // PIT oscillator frequency in Hz + static constexpr uint16_t PIT_CHANNEL2_DATA = 0x42; + static constexpr uint16_t PIT_COMMAND = 0x43; + static constexpr uint16_t PIT_GATE_PORT = 0x61; + + // APIC timer LVT mode bits + static constexpr uint32_t LVT_MASKED = (1 << 16); + static constexpr uint32_t LVT_PERIODIC = (1 << 17); + + // APIC timer divide configuration values + static constexpr uint32_t DIVIDE_BY_16 = 0x03; + + static uint32_t g_ticksPerMs = 0; + + static uint32_t CountForIntervalMs(uint32_t intervalMs) { + return g_ticksPerMs * intervalMs; + } + + static void ProgramTimer(bool periodic, uint32_t intervalMs) { + uint32_t lvt = (Hal::IRQ_VECTOR_BASE + Arch::IRQ_TIMER); + if (periodic) { + lvt |= LVT_PERIODIC; + } + + Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_DIVIDE, DIVIDE_BY_16); + Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_LVT, lvt); + Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_INITIAL, + CountForIntervalMs(intervalMs)); + } + + // Use PIT channel 2 to create a precise delay for calibration. + // Returns the number of APIC timer ticks that elapsed during ~10ms. + static uint32_t CalibratePit() { + // We want to measure ~10ms worth of APIC timer ticks. + // PIT count for 10ms: PIT_FREQUENCY / 100 = 11931 + uint16_t pitCount = PIT_FREQUENCY / 100; + + // Set up PIT channel 2 in one-shot mode (mode 0), lobyte/hibyte + // Command: channel 2 (bits 7-6 = 10), lobyte/hibyte (bits 5-4 = 11), + // mode 0 (bits 3-1 = 000), binary (bit 0 = 0) => 0xB0 + Io::Out8(0xB0, PIT_COMMAND); + + // Disable PIT channel 2 gate to prepare + uint8_t gate = Io::In8(PIT_GATE_PORT); + gate &= ~0x01; // Disable gate (bit 0 = 0) + gate &= ~0x02; // Disable speaker (bit 1 = 0) + Io::Out8(gate, PIT_GATE_PORT); + + // Load the count value (low byte first, then high byte) + Io::Out8((uint8_t)(pitCount & 0xFF), PIT_CHANNEL2_DATA); + Io::IoPortWait(); + Io::Out8((uint8_t)(pitCount >> 8), PIT_CHANNEL2_DATA); + + // Set APIC timer to max count with divide-by-16, one-shot (masked so no IRQ) + Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_DIVIDE, DIVIDE_BY_16); + Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_LVT, LVT_MASKED); + Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_INITIAL, 0xFFFFFFFF); + + // Enable PIT channel 2 gate to start counting + gate = Io::In8(PIT_GATE_PORT); + gate |= 0x01; // Enable gate (bit 0 = 1) + Io::Out8(gate, PIT_GATE_PORT); + + // Wait for PIT channel 2 output to go high (bit 5 of port 0x61) + while (!(Io::In8(PIT_GATE_PORT) & 0x20)) { + Arch::Pause(); + } + + // Read how many APIC timer ticks elapsed + uint32_t currentCount = Hal::LocalApic::ReadRegister(Hal::LocalApic::REG_TIMER_CURRENT); + uint32_t elapsed = 0xFFFFFFFF - currentCount; + + // Stop the APIC timer + Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_INITIAL, 0); + + return elapsed; + } + + bool InitializeBsp(uint32_t periodMs) { + KernelLogStream(INFO, "Timer") << "Calibrating APIC timer using PIT channel 2"; + + // Calibrate: measure APIC ticks over ~10ms + uint32_t ticksIn10ms = CalibratePit(); + g_ticksPerMs = ticksIn10ms / 10; + + if (g_ticksPerMs == 0) { + KernelLogStream(ERROR, "Timer") << "APIC timer calibration failed (0 ticks/ms)"; + return false; + } + + uint64_t timerFreqHz = (uint64_t)g_ticksPerMs * 1000; + + KernelLogStream(OK, "Timer") << "APIC timer: " << base::dec << (uint64_t)g_ticksPerMs + << " ticks/ms (" << timerFreqHz << " Hz, divide-by-16)"; + + // Periodic mode on vector IRQ_VECTOR_BASE + IRQ_TIMER + ProgramTimer(true, periodMs); + + KernelLogStream(OK, "Timer") << "APIC timer started: initial count=" + << base::dec << (uint64_t)CountForIntervalMs(periodMs); + return true; + } + + void StartPeriodic(uint32_t periodMs) { + ProgramTimer(true, periodMs); + } + + void Stop() { + uint32_t lvt = Hal::LocalApic::ReadRegister(Hal::LocalApic::REG_TIMER_LVT); + if ((lvt & LVT_MASKED) == 0) { + Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_LVT, + lvt | LVT_MASKED); + } + } +}; diff --git a/kernel/src/Sched/Context.asm b/kernel/src/Arch/x86_64/Context.asm similarity index 100% rename from kernel/src/Sched/Context.asm rename to kernel/src/Arch/x86_64/Context.asm diff --git a/kernel/src/Arch/x86_64/Context.cpp b/kernel/src/Arch/x86_64/Context.cpp new file mode 100644 index 0000000..8eedeae --- /dev/null +++ b/kernel/src/Arch/x86_64/Context.cpp @@ -0,0 +1,70 @@ +/* + * Context.cpp + * x86_64 thread context (implements Arch/Context.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#include +#include +#include + +// Context.asm: save callee-saved registers + RSP + FPU, switch CR3, restore. +extern "C" void SchedContextSwitch(uint64_t* oldRsp, uint64_t newRsp, uint64_t newCR3, + uint8_t* oldFpuArea, uint8_t* newFpuArea); + +// SyscallEntry.asm: jump to user mode via IRETQ. +// `arg` is delivered as the user-mode RDI (SystemV first argument). +extern "C" [[noreturn]] void JumpToUserMode(uint64_t rip, uint64_t rsp, uint64_t arg); + +namespace Arch { + + static constexpr uint32_t IA32_FS_BASE = 0xC0000100; + + void InitFpuState(uint8_t* area) { + memset(area, 0, FpuStateSize); + *(uint16_t*)&area[0] = 0x037F; // FCW: default x87 control word + *(uint32_t*)&area[24] = 0x1F80; // MXCSR: default SSE control/status + } + + uint64_t PrepareThreadStack(uint64_t kernelStackTop, void (*entry)()) { + // The frame Context.asm pops: r15, r14, r13, r12, rbx, rbp, then the + // return address it RETs to. + uint64_t* sp = (uint64_t*)kernelStackTop; + *(--sp) = (uint64_t)entry; // return addr + *(--sp) = 0; // rbp + *(--sp) = 0; // rbx + *(--sp) = 0; // r12 + *(--sp) = 0; // r13 + *(--sp) = 0; // r14 + *(--sp) = 0; // r15 + return (uint64_t)sp; + } + + void SwitchContext(uint64_t* oldSp, uint64_t newSp, uint64_t newRoot, + uint8_t* oldFpu, uint8_t* newFpu) { + SchedContextSwitch(oldSp, newSp, newRoot, oldFpu, newFpu); + } + + void ActivateThread(Smp::CpuData* cpu, uint64_t kernelStackTop, + uint64_t threadPointer) { + // Kernel RSP for SYSCALL entry + cpu->entry.kernelRsp = kernelStackTop; + + // TSS RSP0 for hardware interrupts from ring 3 + cpu->arch.tss->rsp0 = kernelStackTop; + + // The TLS thread pointer is IA32_FS_BASE. Cached per CPU: the MSR + // write is skipped when unchanged, so TLS-free processes (pointer 0) + // cost one write per CPU ever. Userspace cannot move FS base itself + // (CR4.FSGSBASE is off), so the cache cannot go stale. + if (cpu->arch.currentFsBase != threadPointer) { + cpu->arch.currentFsBase = threadPointer; + Hal::WriteMSR(IA32_FS_BASE, threadPointer); + } + } + + void EnterUserMode(uint64_t entry, uint64_t userSp, uint64_t arg) { + JumpToUserMode(entry, userSp, arg); + } + +} diff --git a/kernel/src/Arch/x86_64/Context.hpp b/kernel/src/Arch/x86_64/Context.hpp new file mode 100644 index 0000000..af7e184 --- /dev/null +++ b/kernel/src/Arch/x86_64/Context.hpp @@ -0,0 +1,14 @@ +/* + * Context.hpp + * x86_64 thread context constants (see Arch/Context.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +namespace Arch { + // FXSAVE image. + constexpr uint64_t FpuStateSize = 512; + constexpr uint64_t FpuStateAlign = 16; +} diff --git a/kernel/src/Arch/x86_64/Cpu.hpp b/kernel/src/Arch/x86_64/Cpu.hpp new file mode 100644 index 0000000..e8cbbf4 --- /dev/null +++ b/kernel/src/Arch/x86_64/Cpu.hpp @@ -0,0 +1,100 @@ +/* + * Cpu.hpp + * x86_64 implementation of the Arch/Cpu.hpp primitives + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include + +namespace Arch { + + // RFLAGS.IF + static constexpr uint64_t RflagsInterruptEnable = 1ULL << 9; + + inline void Pause() { + asm volatile("pause" ::: "memory"); + } + + inline void StoreFence() { + asm volatile("sfence" ::: "memory"); + } + + inline void MemoryFence() { + asm volatile("mfence" ::: "memory"); + } + + inline void EnableInterrupts() { + asm volatile("sti" ::: "memory"); + } + + inline void DisableInterrupts() { + asm volatile("cli" ::: "memory"); + } + + inline bool InterruptsEnabled() { + uint64_t flags; + asm volatile("pushfq; pop %0" : "=r"(flags)); + return (flags & RflagsInterruptEnable) != 0; + } + + inline IrqFlags SaveAndDisableInterrupts() { + uint64_t flags; + asm volatile("pushfq; pop %0; cli" : "=r"(flags) :: "memory"); + return flags; + } + + inline void RestoreInterrupts(IrqFlags flags) { + asm volatile("push %0; popfq" :: "r"(flags) : "memory", "cc"); + } + + inline void Halt() { + asm volatile("hlt" ::: "memory"); + } + + // STI's one-instruction interrupt shadow makes STI;HLT atomic. + inline void HaltWithInterruptsDisabled() { + asm volatile("sti\n\thlt\n\tcli" ::: "memory"); + } + + [[noreturn]] inline void HaltForever() { + for (;;) { + asm volatile("cli; hlt" ::: "memory"); + } + } + + inline uint64_t ReadTimestamp() { + uint32_t lo, hi; + asm volatile("rdtsc" : "=a"(lo), "=d"(hi)); + return ((uint64_t)hi << 32) | lo; + } + + // A write to the POST diagnostic port 0x80 is an ISA-bus cycle of roughly + // a microsecond on every chipset since the 486, and never shorter. + inline void IoDelay() { + asm volatile("outb %%al, $0x80" :: "a"(0) : "memory"); + } + + inline void WritebackInvalidateCaches() { + asm volatile("wbinvd" ::: "memory"); + } + + inline void FlushTlbPage(uint64_t virtualAddress) { + asm volatile("invlpg (%0)" :: "r"(virtualAddress) : "memory"); + } + + inline void FlushTlbAll() { + asm volatile("mov %%cr3, %%rax; mov %%rax, %%cr3" ::: "rax", "memory"); + } + + inline uint64_t ReadPageTableRoot() { + uint64_t cr3; + asm volatile("mov %%cr3, %0" : "=r"(cr3)); + return cr3; + } + + inline void WritePageTableRoot(uint64_t rootPhys) { + asm volatile("mov %0, %%cr3" :: "r"(rootPhys) : "memory"); + } + +} diff --git a/kernel/src/Hal/Cpu.hpp b/kernel/src/Arch/x86_64/CpuFeatures.hpp similarity index 83% rename from kernel/src/Hal/Cpu.hpp rename to kernel/src/Arch/x86_64/CpuFeatures.hpp index fba637a..2f0509e 100644 --- a/kernel/src/Hal/Cpu.hpp +++ b/kernel/src/Arch/x86_64/CpuFeatures.hpp @@ -1,6 +1,6 @@ /* - * Cpu.hpp - * CPU feature enablement helpers + * CpuFeatures.hpp + * x86_64 CPU feature enablement and MWAIT idle helpers * Copyright (c) 2025 Daniel Hammer */ @@ -32,14 +32,11 @@ namespace Hal { asm volatile("mwait" :: "a"(hint), "c"(0)); } - // Atomic idle entry for paths that have already disabled interrupts. - // Keeping STI immediately adjacent to HLT/MWAIT avoids the classic race - // where an IRQ is handled between enabling interrupts and entering idle, - // after which the CPU sleeps until the next timer deadline. - inline void HaltWithInterruptsDisabled() { - asm volatile("sti\n\thlt\n\tcli" ::: "memory"); - } - + // Atomic idle entry for paths that have already disabled interrupts + // (the MWAIT counterpart of Arch::HaltWithInterruptsDisabled). Keeping + // STI immediately adjacent to MWAIT avoids the classic race where an IRQ + // is handled between enabling interrupts and entering idle, after which + // the CPU sleeps until the next timer deadline. inline void IdleWaitWithInterruptsDisabled(volatile uint64_t* monitorAddr, uint32_t hint = 0) { asm volatile("monitor" :: "a"(monitorAddr), "c"(0), "d"(0) : "memory"); diff --git a/kernel/src/Hal/CpuPower.cpp b/kernel/src/Arch/x86_64/CpuPower.cpp similarity index 98% rename from kernel/src/Hal/CpuPower.cpp rename to kernel/src/Arch/x86_64/CpuPower.cpp index 4220541..d94a1bf 100644 --- a/kernel/src/Hal/CpuPower.cpp +++ b/kernel/src/Arch/x86_64/CpuPower.cpp @@ -30,16 +30,19 @@ * Copyright (c) 2026 Daniel Hammer */ -#include "CpuPower.hpp" -#include "MSR.hpp" -#include "SmpBoot.hpp" +#include +#include +#include +#include #include #include using namespace Kt; -namespace Hal { +namespace Arch { namespace CpuPower { + using Hal::ReadMSR; + using Hal::WriteMSR; // ============================================================ // MSR addresses (power/thermal; used only in this module) @@ -399,6 +402,7 @@ namespace Hal { out.BaseMHz = g_baseMHz; out.MaxMHz = g_maxMHz; out.EffMHz = g_effMHz.load(std::memory_order_relaxed); + out.ApIdleHint = Hal::CpuIdle::GetApIdleHint(); return true; } diff --git a/kernel/src/Hal/GDT.asm b/kernel/src/Arch/x86_64/GDT.asm similarity index 100% rename from kernel/src/Hal/GDT.asm rename to kernel/src/Arch/x86_64/GDT.asm diff --git a/kernel/src/Hal/GDT.cpp b/kernel/src/Arch/x86_64/GDT.cpp similarity index 98% rename from kernel/src/Hal/GDT.cpp rename to kernel/src/Arch/x86_64/GDT.cpp index a40aba4..d09e5c1 100644 --- a/kernel/src/Hal/GDT.cpp +++ b/kernel/src/Arch/x86_64/GDT.cpp @@ -5,7 +5,7 @@ */ #include "GDT.hpp" -#include "../Terminal/Terminal.hpp" +#include #include namespace Hal { diff --git a/kernel/src/Hal/GDT.hpp b/kernel/src/Arch/x86_64/GDT.hpp similarity index 100% rename from kernel/src/Hal/GDT.hpp rename to kernel/src/Arch/x86_64/GDT.hpp diff --git a/kernel/src/Hal/IDT.cpp b/kernel/src/Arch/x86_64/IDT.cpp similarity index 85% rename from kernel/src/Hal/IDT.cpp rename to kernel/src/Arch/x86_64/IDT.cpp index 0c58057..50603c5 100644 --- a/kernel/src/Hal/IDT.cpp +++ b/kernel/src/Arch/x86_64/IDT.cpp @@ -9,14 +9,14 @@ #include #include #include -#include +#include #include #include #include #include #include -#include -#include +#include +#include namespace Hal { constexpr auto InterruptGate = 0x8E; @@ -59,6 +59,20 @@ namespace Hal { "Reserved" }; + // Termination code for a user-mode CPU exception. The exact vector is + // kept in the crash report; this is only the category. + static TerminationCodes TerminationFromVector(uint8_t vector) { + switch (vector) { + case 0x06: return TerminatedIllegalInstruction; // #UD + case 0x01: // #DB + case 0x03: return TerminatedBreakpoint; // #BP + case 0x00: // #DE + case 0x10: // #MF + case 0x13: return TerminatedArithmeticError; // #XM + default: return TerminatedAccessViolation; // #PF, #GP, #SS, #NP, #AC, #OF, #BR, #CP, ... + } + } + // Exceptions that push a hardware error code before IP/CS/FLAGS/SP/SS static bool ExceptionHasErrorCode(uint8_t vector) { return vector == 8 || (vector >= 10 && vector <= 14) @@ -67,16 +81,16 @@ namespace Hal { // Extract CS from the interrupt frame. For error-code exceptions, the // error code sits at offset 0, shifting IP to +8 and CS to +16. - static uint64_t GetExceptionCS(uint8_t vector, System::PanicFrame* frame) { + static uint64_t GetExceptionCS(uint8_t vector, Arch::PanicFrame* frame) { if (ExceptionHasErrorCode(vector)) { return *(uint64_t*)((uint8_t*)frame + 16); } return frame->CS; } - static System::PanicFrame* GetExceptionRegs(uint8_t vector, System::PanicFrame* frame) { + static Arch::PanicFrame* GetExceptionRegs(uint8_t vector, Arch::PanicFrame* frame) { if (ExceptionHasErrorCode(vector)) { - return (System::PanicFrame*)((uint8_t*)frame + sizeof(uint64_t)); + return (Arch::PanicFrame*)((uint8_t*)frame + sizeof(uint64_t)); } return frame; } @@ -84,7 +98,7 @@ namespace Hal { // Shared fatal-exception path: kill the faulting user process (with a // crash report) or panic the kernel. `frame` is the RAW interrupt frame // (error code at offset 0 for vectors that push one). Never returns. - static void HandleFatalException(uint8_t i, System::PanicFrame* frame) + static void HandleFatalException(uint8_t i, Arch::PanicFrame* frame) { uint64_t cs = GetExceptionCS(i, frame); bool fromUser = (cs & 3) == 3; @@ -136,7 +150,7 @@ namespace Hal { constexpr uint64_t snapshotBytes = CrashReport::StackSnapshotWords * sizeof(uint64_t); if (Memory::VMM::Paging::IsUserRangeAccessible( - proc->pml4Phys, regs->SP, snapshotBytes, false)) { + proc->pageTableRoot, regs->SP, snapshotBytes, false)) { auto* userStack = (const volatile uint64_t*)regs->SP; for (int word = 0; word < CrashReport::StackSnapshotWords; word++) rep.stackWords[word] = userStack[word]; @@ -146,7 +160,7 @@ namespace Hal { if (i == 0x0E) { // Page fault: read CR2 for faulting address asm volatile("mov %%cr2, %0" : "=r"(rep.faultingAddress)); - auto* pf = (System::PageFaultPanicFrame*)frame; + auto* pf = (Arch::PageFaultPanicFrame*)frame; rep.pfPresent = pf->PageFaultError.Present; rep.pfWrite = pf->PageFaultError.Write; rep.pfUser = pf->PageFaultError.User; @@ -201,7 +215,7 @@ namespace Hal { } template - __attribute__((interrupt)) void ExceptionHandler(System::PanicFrame* frame) + __attribute__((interrupt)) void ExceptionHandler(Arch::PanicFrame* frame) { HandleFatalException(i, frame); } @@ -212,7 +226,7 @@ namespace Hal { // user stack growth region maps a fresh zeroed page and retries the // faulting instruction. With the two-argument form, `frame` points past // the error code, directly at the saved IP. - __attribute__((interrupt)) void PageFaultHandler(System::PanicFrame* frame, uint64_t errorCode) + __attribute__((interrupt)) void PageFaultHandler(Arch::PanicFrame* frame, uint64_t errorCode) { bool fromUser = (frame->CS & 3) == 3; if (fromUser) asm volatile("swapgs"); @@ -226,7 +240,10 @@ namespace Hal { // user stack buffers passed into syscalls. if ((errorCode & 1) == 0 && cpu != nullptr && cpu->currentSlot >= 0 && Sched::GetCurrentPid() >= 0) { - if (montauk::abi::TryHandleAnonymousPageFault(cr2, errorCode) || + // Error code bit 1 = write, bit 4 = instruction fetch. + Memory::VMM::PageFault fault{false, (errorCode & (1ULL << 1)) != 0, + (errorCode & (1ULL << 4)) != 0}; + if (montauk::abi::TryHandleAnonymousPageFault(cr2, fault) || Sched::TryGrowUserStack(cr2)) { if (fromUser) asm volatile("swapgs"); return; @@ -236,7 +253,7 @@ namespace Hal { // Not a growable fault. Hand the RAW frame (error code at offset 0) // to the fatal path, which re-derives fromUser and swaps GS itself. if (fromUser) asm volatile("swapgs"); - HandleFatalException(0x0E, (System::PanicFrame*)((uint8_t*)frame - 8)); + HandleFatalException(0x0E, (Arch::PanicFrame*)((uint8_t*)frame - 8)); } void LoadIDT(IDTRStruct& idtr) { diff --git a/kernel/src/Hal/IDT.hpp b/kernel/src/Arch/x86_64/IDT.hpp similarity index 100% rename from kernel/src/Hal/IDT.hpp rename to kernel/src/Arch/x86_64/IDT.hpp diff --git a/kernel/src/Platform/Registers.hpp b/kernel/src/Arch/x86_64/InterruptFrame.hpp similarity index 88% rename from kernel/src/Platform/Registers.hpp rename to kernel/src/Arch/x86_64/InterruptFrame.hpp index 6b6020a..725b091 100644 --- a/kernel/src/Platform/Registers.hpp +++ b/kernel/src/Arch/x86_64/InterruptFrame.hpp @@ -1,10 +1,15 @@ +/* + * InterruptFrame.hpp + * x86_64 exception frames as pushed by the CPU + * Copyright (c) 2025 Daniel Hammer +*/ + #pragma once #include using namespace std; -#if defined (__x86_64__) -namespace System { +namespace Arch { struct PanicFrame { uint64_t IP; uint64_t CS; @@ -51,4 +56,3 @@ namespace System { }__attribute__((packed)); }; -#endif \ No newline at end of file diff --git a/kernel/src/Arch/x86_64/IoPort.hpp b/kernel/src/Arch/x86_64/IoPort.hpp new file mode 100644 index 0000000..753060d --- /dev/null +++ b/kernel/src/Arch/x86_64/IoPort.hpp @@ -0,0 +1,49 @@ +/* + * IoPort.hpp + * x86_64 port I/O (implements Io/IoPort.hpp) + * Copyright (c) 2025 Daniel Hammer +*/ + +#pragma once +#include + +namespace Io { + static constexpr bool HasPortIo = true; + + // OSDev website has the parameters for out(b/w/l) in the opposite order in their example, + // (port, then value), but the manpage on GNU ('man outb') has it in this order. + + inline void Out8(uint8_t value, uint16_t port) { + asm ("outb %0, %1" : : "a"(value), "Nd"(port) : "memory"); + } + + inline void Out16(uint16_t value, uint16_t port) { + asm ("outw %0, %1" : : "a"(value), "Nd"(port) : "memory"); + } + + inline void Out32(uint32_t value, uint16_t port) { + asm ("outl %0, %1" : : "a"(value), "Nd"(port) : "memory"); + } + + inline uint8_t In8(uint16_t port) { + uint8_t result; + asm volatile("inb %1, %0" : "=a"(result) : "Nd"(port) : "memory"); + return result; + } + + inline uint16_t In16(uint16_t port) { + uint16_t result; + asm volatile("inw %1, %0" : "=a"(result) : "Nd"(port) : "memory"); + return result; + } + + inline uint32_t In32(uint16_t port) { + uint32_t result; + asm volatile("inl %1, %0" : "=a"(result) : "Nd"(port) : "memory"); + return result; + } + + inline void IoPortWait() { + Out8(0, 0x80); + } +}; \ No newline at end of file diff --git a/kernel/src/Hal/MSR.hpp b/kernel/src/Arch/x86_64/MSR.hpp similarity index 100% rename from kernel/src/Hal/MSR.hpp rename to kernel/src/Arch/x86_64/MSR.hpp diff --git a/kernel/src/Arch/x86_64/PageTable.hpp b/kernel/src/Arch/x86_64/PageTable.hpp new file mode 100644 index 0000000..12c5d8d --- /dev/null +++ b/kernel/src/Arch/x86_64/PageTable.hpp @@ -0,0 +1,87 @@ +/* + * PageTable.hpp + * x86_64 4-level page-table structures + * Copyright (c) 2025 Daniel Hammer +*/ + +#pragma once +#include +#include + +namespace Memory::VMM { + struct PageTableEntry { + std::uint8_t Present : 1; + std::uint8_t Writable : 1; + std::uint8_t Supervisor : 1; + std::uint8_t WriteThrough : 1; + std::uint8_t CacheDisabled : 1; + std::uint8_t Accessed : 1; + std::uint8_t Ignore : 1; + std::uint8_t LargerPages : 1; + std::uint8_t PageSize : 1; + std::uint8_t Available : 3; + std::uint64_t Address : 52; + }; + + struct PageTableEntry40Bit { + std::uint8_t Present : 1; + std::uint8_t Writable : 1; + std::uint8_t Supervisor : 1; + std::uint8_t WriteThrough : 1; + std::uint8_t CacheDisabled : 1; + std::uint8_t Accessed : 1; + std::uint8_t Ignore : 1; + std::uint8_t LargerPages : 1; + std::uint8_t PageSize : 1; + std::uint8_t Available : 3; + std::uint64_t Address : 40; + std::uint8_t AvailableHigh : 7; + std::uint8_t PK : 4; + std::uint8_t NX : 1; + }; + + + struct PageTable { + PageTableEntry entries[512]; + } __attribute__((packed)) __attribute__((aligned(0x1000))); + + struct VirtualAddress { + std::uint64_t address; + + // Page-table index extraction works for any byte address. Callers that + // truly require page alignment validate that explicitly before mapping. + VirtualAddress(std::uint64_t newAddress) : address(newAddress) {} + + uint64_t GetL4Index() { + return (address >> 39) & 0x1ff; + } + + uint64_t GetL3Index() { + return (address >> 30) & 0x1ff; + } + + uint64_t GetL2Index() { + return (address >> 21) & 0x1ff; + } + + uint64_t GetPageIndex() { + return (address >> 12) & 0x1ff; + } + + uint64_t GetIndex(size_t level) { + if (level == 4) + return GetL4Index(); + + else if (level == 3) + return GetL3Index(); + + else if (level == 2) + return GetL2Index(); + + else if (level == 1) + return GetPageIndex(); + + return 0; + } + }; +}; diff --git a/kernel/src/Memory/Paging.cpp b/kernel/src/Arch/x86_64/Paging.cpp similarity index 88% rename from kernel/src/Memory/Paging.cpp rename to kernel/src/Arch/x86_64/Paging.cpp index 27e4a9d..c4aa9b5 100644 --- a/kernel/src/Memory/Paging.cpp +++ b/kernel/src/Arch/x86_64/Paging.cpp @@ -1,4 +1,12 @@ -#include "Paging.hpp" +/* + * Paging.cpp + * x86_64 4-level paging (implements Memory/Paging.hpp) + * Copyright (c) 2025 Daniel Hammer +*/ + +#include +#include +#include #include #include #include @@ -51,12 +59,22 @@ namespace Memory::VMM { extern "C" uint64_t KernelStartSymbol; + static std::uint64_t GetPhysAddrImpl(std::uint64_t pml4, std::uint64_t virtualAddress, bool use40BitL1); + std::uint64_t GetPhysKernelAddress(std::uint64_t virtualAddress) { - return Paging::GetPhysAddr(GetCR3(), (std::uint64_t)virtualAddress, true); + return GetPhysAddrImpl(Arch::ReadPageTableRoot(), (std::uint64_t)virtualAddress, true); + } + + void ActivateKernelAddressSpace() { + Arch::WritePageTableRoot(g_paging->KernelRoot()); + } + + void FlushTLB() { + Arch::FlushTlbAll(); } Paging::Paging() { - PML4 = (PageTable*)SubHHDM((PageTable*)Memory::g_pfa->AllocateZeroed()); + rootPhys = SubHHDM(Memory::g_pfa->AllocateZeroed()); } void Paging::Init(std::uint64_t kernelBaseVirt, std::uint64_t kernelSize, const montauk::boot::MemoryMap& memMap, @@ -120,11 +138,11 @@ namespace Memory::VMM { } } - LoadCR3(PML4); + Arch::WritePageTableRoot(rootPhys); Kt::KernelLogStream(Kt::OK, "VMM") << "Switched CR3"; } - PageTable* Paging::HandleLevel(VirtualAddress virtualAddress, PageTable* table, const size_t level) { + static PageTable* HandleLevel(VirtualAddress virtualAddress, PageTable* table, const size_t level) { PageTableEntry* entry = (PageTableEntry*)Memory::HHDM(&table->entries[virtualAddress.GetIndex(level)]); if (!entry->Present) { @@ -144,7 +162,7 @@ namespace Memory::VMM { } } - PageTable* Paging::HandleLevelUser(VirtualAddress virtualAddress, PageTable* table, const size_t level) { + static PageTable* HandleLevelUser(VirtualAddress virtualAddress, PageTable* table, const size_t level) { PageTableEntry* entry = (PageTableEntry*)Memory::HHDM(&table->entries[virtualAddress.GetIndex(level)]); if (!entry->Present) { @@ -174,7 +192,7 @@ namespace Memory::VMM { VirtualAddress virtualAddressObj(virtualAddress); - auto PML3 = HandleLevel(virtualAddressObj, PML4, 4); + auto PML3 = HandleLevel(virtualAddressObj, (PageTable*)rootPhys, 4); auto PML2 = HandleLevel(virtualAddressObj, PML3, 3); auto PML1 = HandleLevel(virtualAddressObj, PML2, 2); @@ -190,7 +208,7 @@ namespace Memory::VMM { VirtualAddress virtualAddressObj(virtualAddress); - auto PML3 = HandleLevel(virtualAddressObj, PML4, 4); + auto PML3 = HandleLevel(virtualAddressObj, (PageTable*)rootPhys, 4); auto PML2 = HandleLevel(virtualAddressObj, PML3, 3); auto PML1 = HandleLevel(virtualAddressObj, PML2, 2); @@ -206,7 +224,7 @@ namespace Memory::VMM { VirtualAddress virtualAddressObj(virtualAddress); - auto PML3 = HandleLevel(virtualAddressObj, PML4, 4); + auto PML3 = HandleLevel(virtualAddressObj, (PageTable*)rootPhys, 4); auto PML2 = HandleLevel(virtualAddressObj, PML3, 3); auto PML1 = HandleLevel(virtualAddressObj, PML2, 2); @@ -216,7 +234,7 @@ namespace Memory::VMM { // The page may have been mapped before with other cache attributes // (an HHDM page remapped uncached): drop any stale translation. - asm volatile("invlpg (%0)" :: "r"(virtualAddress) : "memory"); + Arch::FlushTlbPage(virtualAddress); } void Paging::MapUser(std::uint64_t physicalAddress, std::uint64_t virtualAddress) { @@ -226,7 +244,7 @@ namespace Memory::VMM { VirtualAddress virtualAddressObj(virtualAddress); - auto PML3 = HandleLevelUser(virtualAddressObj, PML4, 4); + auto PML3 = HandleLevelUser(virtualAddressObj, (PageTable*)rootPhys, 4); auto PML2 = HandleLevelUser(virtualAddressObj, PML3, 3); auto PML1 = HandleLevelUser(virtualAddressObj, PML2, 2); @@ -235,14 +253,14 @@ namespace Memory::VMM { SetLeafPte(pageEntry, physicalAddress, true, false, false); } - std::uint64_t Paging::CreateUserPML4() { + std::uint64_t Paging::CreateUserAddressSpace() { // Allocate a new PML4 void* newPage = Memory::g_pfa->AllocateZeroed(); uint64_t newPml4Phys = Memory::SubHHDM((uint64_t)newPage); PageTable* newPml4 = (PageTable*)newPage; // HHDM virtual address // Copy kernel-half entries (256-511) from the global PML4 - PageTable* kernelPml4 = (PageTable*)Memory::HHDM((uint64_t)g_paging->PML4); + PageTable* kernelPml4 = (PageTable*)Memory::HHDM(g_paging->KernelRoot()); for (int i = 256; i < 512; i++) { newPml4->entries[i] = kernelPml4->entries[i]; } @@ -250,11 +268,11 @@ namespace Memory::VMM { return newPml4Phys; } - bool Paging::MapUserIn(std::uint64_t pml4Phys, std::uint64_t physicalAddress, std::uint64_t virtualAddress) { - return MapUserInPermissions(pml4Phys, physicalAddress, virtualAddress, true, true); + bool Paging::MapUserIn(std::uint64_t pageTableRoot, std::uint64_t physicalAddress, std::uint64_t virtualAddress) { + return MapUserInPermissions(pageTableRoot, physicalAddress, virtualAddress, true, true); } - bool Paging::MapUserInPermissions(std::uint64_t pml4Phys, std::uint64_t physicalAddress, + bool Paging::MapUserInPermissions(std::uint64_t pageTableRoot, std::uint64_t physicalAddress, std::uint64_t virtualAddress, bool writable, bool executable) { pagingLock.Acquire(); @@ -284,7 +302,7 @@ namespace Memory::VMM { } }; - PageTable* pml4 = (PageTable*)pml4Phys; + PageTable* pml4 = (PageTable*)pageTableRoot; auto pml3 = walkLevel(pml4, va.GetL4Index()); if (!pml3) { pagingLock.Release(); return false; } auto pml2 = walkLevel(pml3, va.GetL3Index()); @@ -298,7 +316,7 @@ namespace Memory::VMM { return true; } - bool Paging::ProtectUserIn(std::uint64_t pml4Phys, std::uint64_t virtualAddress, + bool Paging::ProtectUserIn(std::uint64_t pageTableRoot, std::uint64_t virtualAddress, bool writable, bool executable) { if ((virtualAddress & 0xFFFULL) != 0) return false; pagingLock.Acquire(); @@ -310,7 +328,7 @@ namespace Memory::VMM { return (PageTable*)(entry->Address << 12); }; - PageTable* pml4 = (PageTable*)pml4Phys; + PageTable* pml4 = (PageTable*)pageTableRoot; auto pml3 = walkRead(pml4, va.GetL4Index()); if (!pml3) { pagingLock.Release(); return false; } auto pml2 = walkRead(pml3, va.GetL3Index()); @@ -329,12 +347,12 @@ namespace Memory::VMM { if (executable) raw &= ~LeafNoExecute; else raw |= LeafNoExecute; *(volatile uint64_t*)pte = raw; - asm volatile("invlpg (%0)" :: "r"(virtualAddress) : "memory"); + Arch::FlushTlbPage(virtualAddress); pagingLock.Release(); return true; } - bool Paging::MapUserInWC(std::uint64_t pml4Phys, std::uint64_t physicalAddress, std::uint64_t virtualAddress) { + bool Paging::MapUserInWC(std::uint64_t pageTableRoot, std::uint64_t physicalAddress, std::uint64_t virtualAddress) { pagingLock.Acquire(); if (virtualAddress % 0x1000 != 0 || physicalAddress % 0x1000 != 0) { pagingLock.Release(); @@ -360,7 +378,7 @@ namespace Memory::VMM { } }; - PageTable* pml4 = (PageTable*)pml4Phys; + PageTable* pml4 = (PageTable*)pageTableRoot; auto pml3 = walkLevel(pml4, va.GetL4Index()); if (!pml3) { pagingLock.Release(); return false; } auto pml2 = walkLevel(pml3, va.GetL3Index()); @@ -374,7 +392,7 @@ namespace Memory::VMM { return true; } - void Paging::UnmapUserIn(std::uint64_t pml4Phys, std::uint64_t virtualAddress) { + void Paging::UnmapUserIn(std::uint64_t pageTableRoot, std::uint64_t virtualAddress) { pagingLock.Acquire(); VirtualAddress va(virtualAddress); @@ -385,7 +403,7 @@ namespace Memory::VMM { return (PageTable*)(entry->Address << 12); }; - PageTable* pml4 = (PageTable*)pml4Phys; + PageTable* pml4 = (PageTable*)pageTableRoot; auto pml3 = walkRead(pml4, va.GetL4Index()); if (!pml3) { pagingLock.Release(); return; } auto pml2 = walkRead(pml3, va.GetL3Index()); @@ -400,14 +418,14 @@ namespace Memory::VMM { *(uint64_t*)pageEntry = 0; // Invalidate TLB for this virtual address - asm volatile("invlpg (%0)" :: "r"(virtualAddress) : "memory"); + Arch::FlushTlbPage(virtualAddress); pagingLock.Release(); } - void Paging::FreeUserHalf(std::uint64_t pml4Phys) { + void Paging::FreeUserHalf(std::uint64_t pageTableRoot) { // PageTable* pointers store PHYSICAL addresses (same convention as MapUserIn/UnmapUserIn). // Each individual entry access goes through HHDM() to get a valid virtual pointer. - PageTable* pml4 = (PageTable*)pml4Phys; + PageTable* pml4 = (PageTable*)pageTableRoot; // Walk user-half entries (0-255); kernel-half (256-511) is shared and must not be touched. for (int i4 = 0; i4 < 256; i4++) { @@ -459,7 +477,7 @@ namespace Memory::VMM { } } - bool Paging::IsUserRangeAccessible(std::uint64_t pml4Phys, std::uint64_t virtualAddress, + bool Paging::IsUserRangeAccessible(std::uint64_t pageTableRoot, std::uint64_t virtualAddress, std::uint64_t size, bool requireWrite) { static constexpr uint64_t USER_SPACE_END = 0x0000800000000000ULL; @@ -470,7 +488,7 @@ namespace Memory::VMM { auto pageAccessible = [&](uint64_t pageVa) -> bool { VirtualAddress va(pageVa); - PageTable* pml4 = (PageTable*)pml4Phys; + PageTable* pml4 = (PageTable*)pageTableRoot; PageTableEntry* pml4e = (PageTableEntry*)Memory::HHDM(&pml4->entries[va.GetL4Index()]); if (!pml4e->Present || !pml4e->Supervisor) return false; @@ -506,7 +524,13 @@ namespace Memory::VMM { return true; } - std::uint64_t Paging::GetPhysAddr(std::uint64_t pml4, std::uint64_t virtualAddress, bool use40BitL1) { + std::uint64_t Paging::GetPhysAddr(std::uint64_t root, std::uint64_t virtualAddress) { + return GetPhysAddrImpl(root, virtualAddress, false); + } + + // use40BitL1 reads the leaf through the narrower historical layout; the + // boot-time kernel-image walk over the bootloader's tables relies on it. + static std::uint64_t GetPhysAddrImpl(std::uint64_t pml4, std::uint64_t virtualAddress, bool use40BitL1) { VirtualAddress virtualAddressObj(virtualAddress); PageTable* pml4Virt = (PageTable*)HHDM(pml4); @@ -548,7 +572,7 @@ namespace Memory::VMM { } std::uint64_t Paging::GetPhysAddr(std::uint64_t virtualAddress) { - return GetPhysAddr((std::uint64_t)PML4, virtualAddress, false); + return GetPhysAddrImpl(rootPhys, virtualAddress, false); } void Paging::MapEfiRuntime(const montauk::boot::EfiInfo& efi) { diff --git a/kernel/src/Arch/x86_64/PanicFrame.cpp b/kernel/src/Arch/x86_64/PanicFrame.cpp new file mode 100644 index 0000000..05150ac --- /dev/null +++ b/kernel/src/Arch/x86_64/PanicFrame.cpp @@ -0,0 +1,74 @@ +/* + * PanicFrame.cpp + * x86_64 CPU state for the panic screen + * Copyright (c) 2025 Daniel Hammer +*/ + +#include +#include +#include + +static constexpr int BoxWidth = PanicBoxWidth; + +static void PrintPageFaultInfo(Arch::PanicFrame*& frame) { + auto* pf = (Arch::PageFaultPanicFrame*)frame; + frame = (Arch::PanicFrame*)&pf->IP; + + uint64_t cr2; + asm volatile("mov %%cr2, %0" : "=r"(cr2)); + PrintBoxedHex(kerr, "Faulting Address (CR2)", cr2, BoxWidth); + + PrintBoxedLine(kerr, "Page Fault Error:", BoxWidth, true); + PrintBoxedDec(kerr, "Present", pf->PageFaultError.Present, BoxWidth); + PrintBoxedDec(kerr, "Write", pf->PageFaultError.Write, BoxWidth); + PrintBoxedDec(kerr, "User", pf->PageFaultError.User, BoxWidth); + PrintBoxedDec(kerr, "Reserved Write", pf->PageFaultError.ReservedWrite, BoxWidth); + PrintBoxedDec(kerr, "Instruction Fetch",pf->PageFaultError.InstructionFetch, BoxWidth); + PrintBoxedDec(kerr, "Protection Key", pf->PageFaultError.ProtectionKey, BoxWidth); + PrintBoxedDec(kerr, "Shadow Stack", pf->PageFaultError.ShadowStack, BoxWidth); + PrintBoxedDec(kerr, "SGX", pf->PageFaultError.SGX, BoxWidth); +} + +static void PrintGPFInfo(Arch::PanicFrame*& frame) { + auto* gpf = (Arch::GPFPanicFrame*)frame; + frame = (Arch::PanicFrame*)&gpf->IP; + + PrintBoxedLine(kerr, "General Protection Fault:", BoxWidth, true); + PrintBoxedDec(kerr, "Error Code", gpf->GeneralProtectionFaultError, BoxWidth); +} + +// Link-time base of the kernel image (see linker-x86_64.ld). +static constexpr uint64_t KernelBaseVirt = 0xFFFFFFFF80000000ULL; + +static void PrintRegisters(Arch::PanicFrame* frame) { + PrintBoxedSeparator(kerr, BoxWidth); + PrintBoxedLine(kerr, "Registers:", BoxWidth, true); + PrintBoxedHex(kerr, "Instruction Pointer", frame->IP, BoxWidth); + + // The absolute address means nothing without the link map. Its offset from + // the kernel's load base is what `nm bin-x86_64/kernel` can be searched + // for directly, so print that next to it. + if (frame->IP >= KernelBaseVirt) + PrintBoxedHex(kerr, " kernel+", frame->IP - KernelBaseVirt, BoxWidth); + + + PrintBoxedHex(kerr, "Code Segment", frame->CS, BoxWidth); + PrintBoxedHex(kerr, "Flags", frame->Flags, BoxWidth); + PrintBoxedHex(kerr, "Stack Pointer", frame->SP, BoxWidth); + PrintBoxedHex(kerr, "Stack Segment", frame->SS, BoxWidth); +} + +namespace Arch { + void PrintPanicFrame(PanicFrame* frame) { + PrintBoxedSeparator(kerr, BoxWidth); + PrintBoxedLine(kerr, "CPU State:", BoxWidth, true); + PrintBoxedHex(kerr, "Interrupt Vector", frame->InterruptVector, BoxWidth); + + if (frame->InterruptVector == 0xE) + PrintPageFaultInfo(frame); + else if (frame->InterruptVector == 0xD) + PrintGPFInfo(frame); + + PrintRegisters(frame); + } +} diff --git a/kernel/src/Arch/x86_64/PerCpu.hpp b/kernel/src/Arch/x86_64/PerCpu.hpp new file mode 100644 index 0000000..b4d8b11 --- /dev/null +++ b/kernel/src/Arch/x86_64/PerCpu.hpp @@ -0,0 +1,57 @@ +/* + * PerCpu.hpp + * x86_64 per-CPU state (see Arch/Smp.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#pragma once +#include +#include + +// ==================================================================== +// Assembly-visible offsets into Smp::CpuData +// These MUST match the struct layout. Verified by static_assert in +// SmpBoot.cpp; SyscallEntry.asm repeats them. +// ==================================================================== +#define CPUDATA_SELF_PTR 0 +#define CPUDATA_KERNEL_RSP 8 +#define CPUDATA_USER_RSP 16 + +namespace Arch { + + // Read by SyscallEntry.asm through GS. + struct CpuEntryState { + uint64_t kernelRsp; // kernel stack top for SYSCALL + uint64_t userRspScratch; // scratch for saving user RSP + }; + + struct CpuState { + uint32_t lapicId; // local APIC ID + bool hasMwait; // CPU supports MONITOR/MWAIT + uint64_t currentFsBase = ~0ULL; // last IA32_FS_BASE written on this CPU + + Hal::TSS64* tss; // this CPU's TSS (BSP: the global one) + + // Per-CPU GDT and TSS (APs use these; BSP uses globals) + Hal::BasicGDT cpuGdt __attribute__((aligned(16))); + Hal::TSS64 cpuTss __attribute__((aligned(16))); + }; + + // GS base holds the CpuData pointer in kernel mode; gs:0 is its selfPtr. + inline void* CurrentCpuLocal() { + void* ptr; + asm volatile("mov %%gs:0, %0" : "=r"(ptr)); + return ptr; + } + + // Read IA32_GS_BASE directly rather than gs:0, which would fault before + // InitBsp() has installed a GS base and turn the original, diagnosable + // exception into a triple fault. + inline void* TryCurrentCpuLocal() { + uint32_t low; + uint32_t high; + asm volatile("rdmsr" : "=a"(low), "=d"(high) : "c"(0xC0000101u)); + return (void*)(((uint64_t)high << 32) | low); + } + +} diff --git a/kernel/src/Arch/x86_64/Platform.cpp b/kernel/src/Arch/x86_64/Platform.cpp new file mode 100644 index 0000000..f0dbaf3 --- /dev/null +++ b/kernel/src/Arch/x86_64/Platform.cpp @@ -0,0 +1,80 @@ +/* + * Platform.cpp + * x86_64 bring-up hooks (implements Arch/Platform.hpp) + * Copyright (c) 2026 Daniel Hammer +*/ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace Arch { + + void EarlyInitialize() { + Hal::PrepareGDT(); + Hal::BridgeLoadGDT(); + + Hal::EnableSSE(); + } + + void InitializeTraps() { + Hal::IDTInitialize(); + } + + void InitializeMemoryAttributes() { + Hal::InitializePAT(); + } + + void InitializeInterrupts(Hal::ACPI::CommonSDTHeader* xsdt) { + Hal::CpuIdle::Initialize(xsdt); + + Hal::ApicInitialize(xsdt); + + Smp::InitBsp(); + + CpuPower::InitializeBsp(); + } + + void InitializePlatformDevices() { + Drivers::PS2::Initialize(); + Drivers::PS2::Keyboard::Initialize(); + Drivers::PS2::Mouse::Initialize(); + } + + void PrepareUserMode() { + Hal::LoadTSS(); + } + + void LateInitialize() { + Drivers::PS2::Mouse::FlushState(); + } + + void ListPlatformDevices(PlatformDeviceSink sink, void* context) { + sink(context, 1, "Local APIC", ""); + sink(context, 1, "I/O APIC", ""); + + sink(context, 2, "LAPIC Timer", "Local APIC periodic timer"); + + sink(context, 3, "PS/2 Keyboard", "IRQ 1"); + if (Drivers::PS2::IsDualChannel()) { + sink(context, 3, "PS/2 Mouse", "IRQ 12"); + } + } + + void Reboot() { + // Triple fault: load an empty IDT and raise an exception. + struct [[gnu::packed]] { uint16_t limit; uint64_t base; } nullIdt = {0, 0}; + asm volatile("lidt %0; int $0x03" :: "m"(nullIdt)); + __builtin_unreachable(); + } + +} diff --git a/kernel/src/Hal/SmpBoot.cpp b/kernel/src/Arch/x86_64/SmpBoot.cpp similarity index 73% rename from kernel/src/Hal/SmpBoot.cpp rename to kernel/src/Arch/x86_64/SmpBoot.cpp index 0e19663..e1f1a04 100644 --- a/kernel/src/Hal/SmpBoot.cpp +++ b/kernel/src/Arch/x86_64/SmpBoot.cpp @@ -4,30 +4,30 @@ * Copyright (c) 2026 Daniel Hammer */ -#include "SmpBoot.hpp" -#include -#include -#include -#include -#include -#include -#include +#include +#include +#include +#include +#include +#include +#include +#include +#include #include +#include +#include #include #include #include #include -#include +#include #include #include // Verify assembly offsets match struct layout -static_assert(__builtin_offsetof(Smp::CpuData, selfPtr) == CPUDATA_SELF_PTR, "CpuData offset mismatch: selfPtr"); -static_assert(__builtin_offsetof(Smp::CpuData, kernelRsp) == CPUDATA_KERNEL_RSP, "CpuData offset mismatch: kernelRsp"); -static_assert(__builtin_offsetof(Smp::CpuData, userRspScratch)== CPUDATA_USER_RSP, "CpuData offset mismatch: userRspScratch"); -static_assert(__builtin_offsetof(Smp::CpuData, currentSlot) == CPUDATA_CURRENT_SLOT, "CpuData offset mismatch: currentSlot"); - -extern "C" void SyscallEntry(); +static_assert(__builtin_offsetof(Smp::CpuData, selfPtr) == CPUDATA_SELF_PTR, "CpuData offset mismatch: selfPtr"); +static_assert(__builtin_offsetof(Smp::CpuData, entry.kernelRsp) == CPUDATA_KERNEL_RSP, "CpuData offset mismatch: kernelRsp"); +static_assert(__builtin_offsetof(Smp::CpuData, entry.userRspScratch) == CPUDATA_USER_RSP, "CpuData offset mismatch: userRspScratch"); // Assembly helpers (GDT.asm) extern "C" void LoadGDT(Hal::GDTPointer* ptr); @@ -50,25 +50,29 @@ namespace Smp { return g_cpuCount; } + int GetDetectedCpuCount() { + return Hal::GetDetectedCpuCount(); + } + // ==================================================================== // Per-CPU GDT/TSS setup // ==================================================================== static void SetupPerCpuGdtTss(CpuData& cpu) { // Zero the TSS - memset(&cpu.cpuTss, 0, sizeof(Hal::TSS64)); - cpu.cpuTss.iopbOffset = sizeof(Hal::TSS64); + memset(&cpu.arch.cpuTss, 0, sizeof(Hal::TSS64)); + cpu.arch.cpuTss.iopbOffset = sizeof(Hal::TSS64); // Allocate a 4KB IST1 stack for Double Fault and NMI. // These exceptions need a known-good stack to avoid triple // faults when the normal kernel stack overflows. void* istPage = Memory::g_pfa->AllocateZeroed(); if (istPage) { - cpu.cpuTss.ist1 = (uint64_t)istPage + 0x1000; // top of stack + cpu.arch.cpuTss.ist1 = (uint64_t)istPage + 0x1000; // top of stack } // Copy the standard GDT layout - cpu.cpuGdt = { + cpu.arch.cpuGdt = { {0xFFFF, 0, 0, 0x00, 0x00, 0}, // 0x00 Null {0xFFFF, 0, 0, 0x9A, 0xA0, 0}, // 0x08 KernelCode {0xFFFF, 0, 0, 0x92, 0xA0, 0}, // 0x10 KernelData @@ -79,27 +83,27 @@ namespace Smp { }; // Encode the 16-byte TSS descriptor - uint64_t base = (uint64_t)&cpu.cpuTss; + uint64_t base = (uint64_t)&cpu.arch.cpuTss; uint32_t limit = sizeof(Hal::TSS64) - 1; // Low 8 bytes - cpu.cpuGdt.TSS.LimitLow = limit & 0xFFFF; - cpu.cpuGdt.TSS.BaseLow = base & 0xFFFF; - cpu.cpuGdt.TSS.BaseMiddle = (base >> 16) & 0xFF; - cpu.cpuGdt.TSS.AccessByte = 0x89; // Present, 64-bit TSS Available - cpu.cpuGdt.TSS.GranularityByte = (limit >> 16) & 0x0F; - cpu.cpuGdt.TSS.BaseHigh = (base >> 24) & 0xFF; + cpu.arch.cpuGdt.TSS.LimitLow = limit & 0xFFFF; + cpu.arch.cpuGdt.TSS.BaseLow = base & 0xFFFF; + cpu.arch.cpuGdt.TSS.BaseMiddle = (base >> 16) & 0xFF; + cpu.arch.cpuGdt.TSS.AccessByte = 0x89; // Present, 64-bit TSS Available + cpu.arch.cpuGdt.TSS.GranularityByte = (limit >> 16) & 0x0F; + cpu.arch.cpuGdt.TSS.BaseHigh = (base >> 24) & 0xFF; // High 8 bytes (base[63:32]) uint32_t baseUpper = (uint32_t)(base >> 32); - cpu.cpuGdt.TSSHigh.LimitLow = baseUpper & 0xFFFF; - cpu.cpuGdt.TSSHigh.BaseLow = (baseUpper >> 16) & 0xFFFF; - cpu.cpuGdt.TSSHigh.BaseMiddle = 0; - cpu.cpuGdt.TSSHigh.AccessByte = 0; - cpu.cpuGdt.TSSHigh.GranularityByte = 0; - cpu.cpuGdt.TSSHigh.BaseHigh = 0; + cpu.arch.cpuGdt.TSSHigh.LimitLow = baseUpper & 0xFFFF; + cpu.arch.cpuGdt.TSSHigh.BaseLow = (baseUpper >> 16) & 0xFFFF; + cpu.arch.cpuGdt.TSSHigh.BaseMiddle = 0; + cpu.arch.cpuGdt.TSSHigh.AccessByte = 0; + cpu.arch.cpuGdt.TSSHigh.GranularityByte = 0; + cpu.arch.cpuGdt.TSSHigh.BaseHigh = 0; - cpu.tss = &cpu.cpuTss; + cpu.arch.tss = &cpu.arch.cpuTss; } // ==================================================================== @@ -126,13 +130,13 @@ namespace Smp { CpuData& bsp = g_cpus[0]; bsp.selfPtr = (uint64_t)&bsp; bsp.cpuIndex = 0; - bsp.lapicId = Hal::LocalApic::GetId(); + bsp.arch.lapicId = Hal::LocalApic::GetId(); bsp.currentSlot = -1; bsp.started = true; - bsp.hasMwait = Hal::HasMwait(); + bsp.arch.hasMwait = Hal::HasMwait(); // BSP uses the global TSS (already set up in PrepareGDT) - bsp.tss = &Hal::g_tss; + bsp.arch.tss = &Hal::g_tss; // Allocate IST1 stack for the BSP (for Double Fault / NMI) void* bspIstPage = Memory::g_pfa->AllocateZeroed(); @@ -145,7 +149,7 @@ namespace Smp { g_cpuCount = 1; - KernelLogStream(OK, "SMP") << "BSP initialized (LAPIC ID " << (uint64_t)bsp.lapicId << ")"; + KernelLogStream(OK, "SMP") << "BSP initialized (LAPIC ID " << (uint64_t)bsp.arch.lapicId << ")"; } // ==================================================================== @@ -163,7 +167,7 @@ namespace Smp { // --- Load per-CPU GDT --- Hal::GDTPointer gdtPtr { .Size = sizeof(Hal::BasicGDT) - 1, - .GDTAddress = (uint64_t)&cpu->cpuGdt, + .GDTAddress = (uint64_t)&cpu->arch.cpuGdt, }; LoadGDT(&gdtPtr); ReloadSegments(); @@ -175,7 +179,7 @@ namespace Smp { Hal::IDTReload(); // --- Switch to kernel page tables --- - Memory::VMM::LoadCR3(Memory::VMM::g_paging->PML4); + Memory::VMM::ActivateKernelAddressSpace(); // --- Enable SSE --- Hal::EnableSSE(); @@ -187,45 +191,38 @@ namespace Smp { Hal::LocalApic::InitializeAP(); // --- Program SYSCALL MSRs --- - uint64_t efer = Hal::ReadMSR(Hal::IA32_EFER); - efer |= 1 | (1ULL << 11); // SCE | NXE - Hal::WriteMSR(Hal::IA32_EFER, efer); - - uint64_t star = (0x0010ULL << 48) | (0x0008ULL << 32); - Hal::WriteMSR(Hal::IA32_STAR, star); - Hal::WriteMSR(Hal::IA32_LSTAR, (uint64_t)SyscallEntry); - Hal::WriteMSR(Hal::IA32_FMASK, 0x200); + Arch::InitializeSyscallEntry(); // --- Program PAT (entry 1 = WC) --- Hal::InitializePAT(); // --- Calibrate and start APIC timer --- - Timekeeping::ApicTimerInitializeAP(); + Timekeeping::TimerInitializeAP(); // --- Check MWAIT support --- - cpu->hasMwait = Hal::HasMwait(); + cpu->arch.hasMwait = Hal::HasMwait(); // --- Per-CPU power setup (C1E promotion, HWP request) --- - Hal::CpuPower::InitializeAp(); + Arch::CpuPower::InitializeAp(); // --- Signal that we are online --- cpu->started = true; // --- Enable interrupts and enter idle loop --- - asm volatile("sti"); + Arch::EnableInterrupts(); static volatile uint64_t s_idleMonitor = 0; for (;;) { // Pick up thermal-governor frequency changes decided by the BSP. - Hal::CpuPower::ApplyPolicyIfChanged(); + Arch::CpuPower::ApplyPolicyIfChanged(); // Runnable work sends this AP a reschedule IPI. Keep its periodic // scheduler tick masked for the entire idle-context pass so long // firmware waits do not keep generating useless timer interrupts. - Timekeeping::ApicTimerEnterApIdle(); + Timekeeping::TimerEnterApIdle(); // Any idle core may run bounded USB/NIC bottom halves. Preserve // the AP's ACPI/MWAIT idle selection after servicing them. Timekeeping::ServiceDeferredWork(); - Hal::CpuIdle::Wait(10, cpu->hasMwait, &s_idleMonitor); + Hal::CpuIdle::Wait(10, cpu->arch.hasMwait, &s_idleMonitor); } } @@ -242,7 +239,7 @@ namespace Smp { uint64_t cpuCount = smp.cpuCount; KernelLogStream(INFO, "SMP") << "Bootloader reports " << cpuCount << " CPU(s), BSP LAPIC ID " - << (uint64_t)smp.bspLapicId; + << smp.bspHardwareId; if (cpuCount <= 1) { KernelLogStream(INFO, "SMP") << "Single CPU system - no APs to boot"; @@ -262,13 +259,13 @@ namespace Smp { for (uint64_t i = 0; i < cpuCount; i++) { montauk::boot::BootCpu& info = smp.cpus[i]; - if (info.lapicId == smp.bspLapicId) continue; + if (info.hardwareId == smp.bspHardwareId) continue; if (apIndex >= MaxCPUs) break; CpuData& ap = g_cpus[apIndex]; ap.selfPtr = (uint64_t)≈ ap.cpuIndex = apIndex; - ap.lapicId = info.lapicId; + ap.arch.lapicId = (uint32_t)info.hardwareId; ap.currentSlot = -1; ap.started = false; @@ -290,13 +287,13 @@ namespace Smp { volatile bool* flag = &g_cpus[i].started; uint64_t timeout = 100000000; while (!*flag && timeout > 0) { - asm volatile("pause"); + Arch::Pause(); timeout--; } if (!*flag) { KernelLogStream(ERROR, "SMP") << "AP " << i << " (LAPIC " - << (uint64_t)g_cpus[i].lapicId << ") failed to start"; + << (uint64_t)g_cpus[i].arch.lapicId << ") failed to start"; } } diff --git a/kernel/src/Arch/x86_64/Syscall.cpp b/kernel/src/Arch/x86_64/Syscall.cpp new file mode 100644 index 0000000..200baae --- /dev/null +++ b/kernel/src/Arch/x86_64/Syscall.cpp @@ -0,0 +1,34 @@ +/* + * Syscall.cpp + * x86_64 SYSCALL/SYSRET setup (implements Arch/Syscall.hpp) + * Copyright (c) 2025 Daniel Hammer +*/ + +#include +#include + +// SyscallEntry.asm +extern "C" void SyscallEntry(); + +namespace Arch { + + void InitializeSyscallEntry() { + // Enable SYSCALL/SYSRET and no-execute page permissions in EFER. + uint64_t efer = Hal::ReadMSR(Hal::IA32_EFER); + efer |= 1 | (1ULL << 11); // SCE | NXE + Hal::WriteMSR(Hal::IA32_EFER, efer); + + // STAR: kernel CS in [47:32], sysret base in [63:48] + // SYSCALL: CS=0x08, SS=0x10 + // SYSRET: CS=0x10+16=0x20|RPL3=0x23, SS=0x10+8=0x18|RPL3=0x1B + uint64_t star = (0x0010ULL << 48) | (0x0008ULL << 32); + Hal::WriteMSR(Hal::IA32_STAR, star); + + // LSTAR: SYSCALL entry point + Hal::WriteMSR(Hal::IA32_LSTAR, (uint64_t)SyscallEntry); + + // FMASK: mask IF on SYSCALL entry (bit 9 = IF) + Hal::WriteMSR(Hal::IA32_FMASK, 0x200); + } + +} diff --git a/kernel/src/Hal/SyscallEntry.asm b/kernel/src/Arch/x86_64/SyscallEntry.asm similarity index 100% rename from kernel/src/Hal/SyscallEntry.asm rename to kernel/src/Arch/x86_64/SyscallEntry.asm diff --git a/kernel/src/Arch/x86_64/SyscallFrame.hpp b/kernel/src/Arch/x86_64/SyscallFrame.hpp new file mode 100644 index 0000000..7035679 --- /dev/null +++ b/kernel/src/Arch/x86_64/SyscallFrame.hpp @@ -0,0 +1,20 @@ +/* + * SyscallFrame.hpp + * x86_64 system-call frame (see Arch/Syscall.hpp) + * Copyright (c) 2025 Daniel Hammer +*/ + +#pragma once +#include + +namespace Arch { + + // Stack frame pushed by SyscallEntry.asm + struct SyscallFrame { + uint64_t r15, r14, r13, r12, rbp, rbx; // callee-saved + uint64_t arg6, arg5, arg4, arg3, arg2, arg1; + uint64_t syscall_nr; + uint64_t user_rflags, user_rip, user_rsp; + }; + +} diff --git a/kernel/src/Boot/BootInfo.hpp b/kernel/src/Boot/BootInfo.hpp index 4aa5a56..f04f05a 100644 --- a/kernel/src/Boot/BootInfo.hpp +++ b/kernel/src/Boot/BootInfo.hpp @@ -187,17 +187,18 @@ namespace montauk::boot { // to bring up, then calls SmpInfo::startCpu. The bootloader resumes // the parked AP such that it begins executing `entry(cpu)` on a stack // the bootloader provides, with the same handoff machine state the BSP - // received (long mode, paging on, kernel mapped). `entry` recovers its + // received (64-bit mode, paging on, kernel mapped). `entry` recovers its // per-CPU context from `cpu->extraArgument`. struct BootCpu; struct SmpInfo { uint64_t cpuCount; // total CPUs reported (including the BSP) - uint32_t bspLapicId; // local-APIC ID of the bootstrap processor + uint64_t bspHardwareId; // hardware ID of the bootstrap processor BootCpu* cpus; // HHDM/direct-mapped virtual; `cpuCount` entries. - // The entry whose lapicId == bspLapicId is the BSP - // and must NOT be started (it is already running). + // The entry whose hardwareId == bspHardwareId is + // the BSP and must NOT be started (it is already + // running). // Wake `cpu`, causing it to begin executing `entry(cpu)`. Returns // true if the start request was accepted. `entry` runs on a @@ -207,7 +208,8 @@ namespace montauk::boot { }; struct BootCpu { - uint32_t lapicId; // local-APIC ID of this CPU + uint64_t hardwareId; // architectural CPU ID: x86 local-APIC ID, + // ARM MPIDR uint64_t extraArgument; // free for the kernel: stash a per-CPU // pointer here before calling startCpu; it is // handed back to `entry` via `cpu->extraArgument`. diff --git a/kernel/src/Boot/Main.cpp b/kernel/src/Boot/Main.cpp index 9885c2d..1d98a47 100644 --- a/kernel/src/Boot/Main.cpp +++ b/kernel/src/Boot/Main.cpp @@ -10,40 +10,31 @@ #include #include #include -#include #include #include #include #include #include #include -#include -#include #include #include #include -#include #include #include #include #include #include -#include #include -#include -#include -#include -#include +#include #include #include -#include -#include -#include +#include +#include #include #include #include #include -#include +#include using namespace Kt; namespace Memory { @@ -69,7 +60,7 @@ extern "C" void kmain() { } if (!montauk::boot::Initialize()) { - Hal::Halt(); + Arch::HaltForever(); } const montauk::boot::BootInfo& boot = montauk::boot::Info(); const montauk::boot::Framebuffer& framebuffer = boot.framebuffer; @@ -88,12 +79,7 @@ extern "C" void kmain() { ); -#if defined (__x86_64__) - Hal::PrepareGDT(); - Hal::BridgeLoadGDT(); - - Hal::EnableSSE(); -#endif + Arch::EarlyInitialize(); Memory::HHDMBase = boot.hhdmBase; @@ -112,26 +98,19 @@ extern "C" void kmain() { heap.Walk(); -#if defined (__x86_64__) - Hal::IDTInitialize(); + Arch::InitializeTraps(); Memory::VMM::Paging g_paging{}; Memory::VMM::g_paging = &g_paging; g_paging.Init((uint64_t)&KernelStartSymbol, ((uint64_t)&KernelEndSymbol - (uint64_t)&KernelStartSymbol), boot.memoryMap, framebuffer); - Hal::InitializePAT(); - -#endif + Arch::InitializeMemoryAttributes(); Graphics::Framebuffer::Initialize(framebuffer); - -#if defined (__x86_64__) Graphics::Framebuffer::MapWriteCombining(); -#endif Hal::ACPI g_acpi((Hal::ACPI::XSDP*)Memory::HHDM(boot.rsdpPhysical)); -#if defined (__x86_64__) if (g_acpi.GetXSDT() != nullptr) { Hal::AcpiShutdown::Initialize(g_acpi.GetXSDT()); @@ -140,15 +119,11 @@ extern "C" void kmain() { Hal::EmbeddedController::Initialize(g_acpi.GetXSDT()); Hal::Battery::Initialize(); - Hal::CpuIdle::Initialize(g_acpi.GetXSDT()); + // Interrupt controllers, this CPU's per-CPU data, idle and power + // management. + Arch::InitializeInterrupts(g_acpi.GetXSDT()); - Hal::ApicInitialize(g_acpi.GetXSDT()); - - Smp::InitBsp(); - - Hal::CpuPower::InitializeBsp(); - - asm volatile("sti"); + Arch::EnableInterrupts(); Hal::AcpiEvents::Initialize(g_acpi.GetXSDT()); @@ -163,11 +138,9 @@ extern "C" void kmain() { Drivers::ProbeEarly(); Drivers::InitializeGraphics(); - Timekeeping::ApicTimerInitialize(); + Timekeeping::TimerInitialize(); - Drivers::PS2::Initialize(); - Drivers::PS2::Keyboard::Initialize(); - Drivers::PS2::Mouse::Initialize(); + Arch::InitializePlatformDevices(); // Hybrid graphics: switch off a discrete GPU nothing drives before // the normal probe can claim its USB-C/audio functions. @@ -178,7 +151,6 @@ extern "C" void kmain() { Drivers::InitializeStorage(); Drivers::InitializeAudio(); } -#endif // UEFI runtime services are optional (absent on legacy-BIOS boots). if (boot.has(montauk::boot::FeatureEfiSystemTable)) { @@ -188,9 +160,7 @@ extern "C" void kmain() { Fs::InitializeBootFilesystems(boot.modules); -#if defined (__x86_64__) - Hal::LoadTSS(); -#endif + Arch::PrepareUserMode(); montauk::abi::InitializeSyscalls(); @@ -199,36 +169,16 @@ extern "C" void kmain() { Fs::LogProtectedPaths(); Ipc::Initialize(); -#if defined (__x86_64__) Smp::BootAPs(boot.smp); -#endif -#if defined (__x86_64__) - Drivers::PS2::Mouse::FlushState(); -#endif + Arch::LateInitialize(); Kt::SuppressKernelLog(); Sched::Spawn("0:/os/init.elf"); Timekeeping::EnableSchedulerTick(); -#if defined (__x86_64__) - // Use MWAIT for deeper C-states if available, otherwise HLT. - auto* bspCpu = Smp::GetCpuData(0); - - if (bspCpu && bspCpu->hasMwait) { - static volatile uint64_t s_bspIdleMonitor = 0; - for (;;) { - Timekeeping::IdleOnce(true, &s_bspIdleMonitor); - } - } else { - for (;;) { - Timekeeping::IdleOnce(false); - } - } -#else for (;;) { - Timekeeping::IdleOnce(false); + Timekeeping::IdleOnce(); } -#endif } diff --git a/kernel/src/Boot/Protocols/LimineProtocol.cpp b/kernel/src/Boot/Protocols/LimineProtocol.cpp index 4a5de11..cf25ccc 100644 --- a/kernel/src/Boot/Protocols/LimineProtocol.cpp +++ b/kernel/src/Boot/Protocols/LimineProtocol.cpp @@ -280,13 +280,21 @@ namespace montauk::boot { if (count > kMaxCpus) count = kMaxCpus; for (uint64_t i = 0; i < count; i++) { limine_mp_info* info = resp->cpus[i]; - g_cpus[i].lapicId = info->lapic_id; +#if defined(__x86_64__) + g_cpus[i].hardwareId = info->lapic_id; +#elif defined(__aarch64__) + g_cpus[i].hardwareId = info->mpidr; +#endif g_cpus[i].extraArgument = 0; g_cpus[i].entry = nullptr; g_cpus[i].native = info; } out.smp.cpuCount = count; - out.smp.bspLapicId = resp->bsp_lapic_id; +#if defined(__x86_64__) + out.smp.bspHardwareId = resp->bsp_lapic_id; +#elif defined(__aarch64__) + out.smp.bspHardwareId = resp->bsp_mpidr; +#endif out.smp.cpus = g_cpus; out.smp.startCpu = &LimineStartCpu; out.features |= FeatureSmp; diff --git a/kernel/src/Common/Panic.cpp b/kernel/src/Common/Panic.cpp index 7b398e0..aed155d 100644 --- a/kernel/src/Common/Panic.cpp +++ b/kernel/src/Common/Panic.cpp @@ -4,10 +4,11 @@ */ #include "Panic.hpp" -#include "../CppLib/BoxUI.hpp" -#include "../Drivers/Graphics/IntelGPU.hpp" +#include +#include +#include -static constexpr int BoxWidth = 72; +static constexpr int BoxWidth = PanicBoxWidth; static void PrintHorizontalEdge(const char* left, const char* right) { kerr << left; @@ -15,68 +16,7 @@ static void PrintHorizontalEdge(const char* left, const char* right) { kerr << right << "\n"; } -static void PrintPageFaultInfo(System::PanicFrame*& frame) { - auto* pf = (System::PageFaultPanicFrame*)frame; - frame = (System::PanicFrame*)&pf->IP; - - uint64_t cr2; - asm volatile("mov %%cr2, %0" : "=r"(cr2)); - PrintBoxedHex(kerr, "Faulting Address (CR2)", cr2, BoxWidth); - - PrintBoxedLine(kerr, "Page Fault Error:", BoxWidth, true); - PrintBoxedDec(kerr, "Present", pf->PageFaultError.Present, BoxWidth); - PrintBoxedDec(kerr, "Write", pf->PageFaultError.Write, BoxWidth); - PrintBoxedDec(kerr, "User", pf->PageFaultError.User, BoxWidth); - PrintBoxedDec(kerr, "Reserved Write", pf->PageFaultError.ReservedWrite, BoxWidth); - PrintBoxedDec(kerr, "Instruction Fetch",pf->PageFaultError.InstructionFetch, BoxWidth); - PrintBoxedDec(kerr, "Protection Key", pf->PageFaultError.ProtectionKey, BoxWidth); - PrintBoxedDec(kerr, "Shadow Stack", pf->PageFaultError.ShadowStack, BoxWidth); - PrintBoxedDec(kerr, "SGX", pf->PageFaultError.SGX, BoxWidth); -} - -static void PrintGPFInfo(System::PanicFrame*& frame) { - auto* gpf = (System::GPFPanicFrame*)frame; - frame = (System::PanicFrame*)&gpf->IP; - - PrintBoxedLine(kerr, "General Protection Fault:", BoxWidth, true); - PrintBoxedDec(kerr, "Error Code", gpf->GeneralProtectionFaultError, BoxWidth); -} - -// Link-time base of the kernel image (see linker-x86_64.ld). -static constexpr uint64_t KernelBaseVirt = 0xFFFFFFFF80000000ULL; - -static void PrintRegisters(System::PanicFrame* frame) { - PrintBoxedSeparator(kerr, BoxWidth); - PrintBoxedLine(kerr, "Registers:", BoxWidth, true); - PrintBoxedHex(kerr, "Instruction Pointer", frame->IP, BoxWidth); - - // The absolute address means nothing without the link map. Its offset from - // the kernel's load base is what `nm bin-x86_64/kernel` can be searched - // for directly, so print that next to it. - if (frame->IP >= KernelBaseVirt) - PrintBoxedHex(kerr, " kernel+", frame->IP - KernelBaseVirt, BoxWidth); - - - PrintBoxedHex(kerr, "Code Segment", frame->CS, BoxWidth); - PrintBoxedHex(kerr, "Flags", frame->Flags, BoxWidth); - PrintBoxedHex(kerr, "Stack Pointer", frame->SP, BoxWidth); - PrintBoxedHex(kerr, "Stack Segment", frame->SS, BoxWidth); -} - -[[noreturn]] static void Halt() { - while (true) { -#if defined (__x86_64__) - asm ("cli"); - asm ("hlt"); -#elif defined (__aarch64__) || defined (__riscv) - asm ("wfi"); -#elif defined (__loongarch64) - asm ("idle 0"); -#endif - } -} - -void Panic(const char *meditationString, System::PanicFrame* frame) { +void Panic(const char *meditationString, Arch::PanicFrame* frame) { // Bring scanout back to buffer 0 first: if a page-flipped client was // showing the other buffer, the panic box would land on an invisible // surface. Safe no-op when the GPU driver is inactive. @@ -96,32 +36,19 @@ void Panic(const char *meditationString, System::PanicFrame* frame) { PrintBoxedLine(kerr, meditationString, BoxWidth); PrintBoxedLine(kerr, "", BoxWidth); - // CPU state (x86_64 only) -#if defined (__x86_64__) + // CPU state at the faulting exception if (frame != nullptr) { - PrintBoxedSeparator(kerr, BoxWidth); - PrintBoxedLine(kerr, "CPU State:", BoxWidth, true); - PrintBoxedHex(kerr, "Interrupt Vector", frame->InterruptVector, BoxWidth); - - if (frame->InterruptVector == 0xE) - PrintPageFaultInfo(frame); - else if (frame->InterruptVector == 0xD) - PrintGPFInfo(frame); - - PrintRegisters(frame); + Arch::PrintPanicFrame(frame); } -#endif // Footer PrintBoxedLine(kerr, "", BoxWidth); PrintHorizontalEdge(BOXUI_BL, BOXUI_BR); kerr << BOXUI_ANSI_RESET; -#if defined (__x86_64__) // The framebuffer is write-combining. Drain the final panic text before // halting all forward progress so it cannot remain in a CPU write buffer. - asm volatile("sfence" ::: "memory"); -#endif + Arch::StoreFence(); - Halt(); + Arch::HaltForever(); } diff --git a/kernel/src/Common/Panic.hpp b/kernel/src/Common/Panic.hpp index 0b77ce9..767dec2 100644 --- a/kernel/src/Common/Panic.hpp +++ b/kernel/src/Common/Panic.hpp @@ -1,5 +1,18 @@ #pragma once -#include #include -void Panic(const char *meditationString, System::PanicFrame* frame); \ No newline at end of file +namespace Arch { + // The CPU state an exception handler captured; its layout belongs to + // the architecture (Arch//), generic code only passes it along. + struct PanicFrame; + + // Print the frame's CPU state into the panic box. Implemented by the + // architecture. + void PrintPanicFrame(PanicFrame* frame); +} + +// Width of the panic screen's box, shared with Arch::PrintPanicFrame. +static constexpr int PanicBoxWidth = 72; + +// Print the panic screen and halt this CPU. `frame` may be nullptr. +[[noreturn]] void Panic(const char *meditationString, Arch::PanicFrame* frame); diff --git a/kernel/src/Common/TerminationCodes.hpp b/kernel/src/Common/TerminationCodes.hpp index 671aaaa..bed29c4 100644 --- a/kernel/src/Common/TerminationCodes.hpp +++ b/kernel/src/Common/TerminationCodes.hpp @@ -21,17 +21,3 @@ enum TerminationCodes { TerminatedKilled = TerminationBase + 9, // SIGKILL TerminatedAccessViolation = TerminationBase + 11 // SIGSEGV }; - -// Termination code for a user-mode CPU exception. The exact vector is -// kept in the crash report; this is only the category. -inline TerminationCodes TerminationFromVector(uint8_t vector) { - switch (vector) { - case 0x06: return TerminatedIllegalInstruction; // #UD - case 0x01: // #DB - case 0x03: return TerminatedBreakpoint; // #BP - case 0x00: // #DE - case 0x10: // #MF - case 0x13: return TerminatedArithmeticError; // #XM - default: return TerminatedAccessViolation; // #PF, #GP, #SS, #NP, #AC, #OF, #BR, #CP, ... - } -} diff --git a/kernel/src/Platform/ABI.cpp b/kernel/src/CppLib/ABI.cpp similarity index 84% rename from kernel/src/Platform/ABI.cpp rename to kernel/src/CppLib/ABI.cpp index 5f67128..0a0abb3 100644 --- a/kernel/src/Platform/ABI.cpp +++ b/kernel/src/CppLib/ABI.cpp @@ -1,10 +1,10 @@ /* - * Itanium.cpp + * ABI.cpp * Definitions for the Itanium C++ ABI * Copyright (c) Daniel Hammer, Limine Contributors (via Limine C++ example) */ -#include +#include // The following stubs are required by the Itanium C++ ABI (the one we use, // regardless of the "Itanium" nomenclature). @@ -12,6 +12,6 @@ // but should not be removed, unless you know what you are doing. extern "C" { int __cxa_atexit(void (*)(void *), void *, void *) { return 0; } - void __cxa_pure_virtual() { Hal::Halt(); } + void __cxa_pure_virtual() { Arch::HaltForever(); } void *__dso_handle; } diff --git a/kernel/src/CppLib/Spinlock.cpp b/kernel/src/CppLib/Spinlock.cpp index e311a0f..e6b4a0a 100644 --- a/kernel/src/CppLib/Spinlock.cpp +++ b/kernel/src/CppLib/Spinlock.cpp @@ -5,20 +5,20 @@ */ #include "Spinlock.hpp" +#include namespace kcp { uint64_t Spinlock::AcquireIrqSave() { - uint64_t flags; - asm volatile("pushfq; pop %0; cli" : "=r"(flags) :: "memory"); + uint64_t flags = Arch::SaveAndDisableInterrupts(); while (atomic_flag.test_and_set(std::memory_order_acquire)) { - asm volatile("pause"); + Arch::Pause(); } return flags; } void Spinlock::ReleaseIrqRestore(uint64_t flags) { atomic_flag.clear(std::memory_order_release); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); } void Spinlock::ReleaseIrqNoRestore() { diff --git a/kernel/src/CppLib/Spinlock.hpp b/kernel/src/CppLib/Spinlock.hpp index 6ef80f9..0760605 100644 --- a/kernel/src/CppLib/Spinlock.hpp +++ b/kernel/src/CppLib/Spinlock.hpp @@ -6,6 +6,7 @@ #pragma once #include +#include namespace kcp { class Spinlock { @@ -32,7 +33,7 @@ namespace kcp { public: void Acquire() { while (flag.test_and_set(std::memory_order_acquire)) { - asm volatile("pause"); + Arch::Pause(); } } void Release() { diff --git a/kernel/src/Drivers/Audio/IntelHda.cpp b/kernel/src/Drivers/Audio/IntelHda.cpp index 7eaedf5..7d643b7 100644 --- a/kernel/src/Drivers/Audio/IntelHda.cpp +++ b/kernel/src/Drivers/Audio/IntelHda.cpp @@ -5,12 +5,12 @@ */ #include "IntelHda.hpp" +#include #include "Mixer.hpp" #include #include #include -#include -#include +#include #include #include #include @@ -173,7 +173,7 @@ namespace Drivers::Audio::IntelHda { static void MicroDelay(int us) { // Simple busy-wait; us is approximate for (volatile int i = 0; i < us * 100; i++) { - asm volatile("pause"); + Arch::Pause(); } } @@ -851,13 +851,9 @@ namespace Drivers::Audio::IntelHda { uint16_t msgCtrl = Pci::LegacyRead16(bus, dev, func, cap + 2); bool is64bit = (msgCtrl & (1 << 7)) != 0; - Pci::LegacyWrite32(bus, dev, func, cap + 4, MSI_ADDR_BASE); - - if (is64bit) { - Pci::LegacyWrite32(bus, dev, func, cap + 8, 0); - Pci::LegacyWrite16(bus, dev, func, cap + 12, MSI_VECTOR); - } else { - Pci::LegacyWrite16(bus, dev, func, cap + 8, MSI_VECTOR); + // Message Address/Data: IRQ slot MSI_IRQ on the boot CPU + if (!Pci::WriteMsiMessage(bus, dev, func, cap, MSI_IRQ)) { + return false; } msgCtrl &= ~(0x70); @@ -868,10 +864,9 @@ namespace Drivers::Audio::IntelHda { pciCmd |= Pci::PCI_CMD_INTX_DISABLE; Pci::LegacyWrite16(bus, dev, func, (uint8_t)Pci::PCI_REG_COMMAND, pciCmd); - Hal::RegisterIrqHandler(MSI_IRQ, HandleInterrupt); + Arch::RegisterIrqHandler(MSI_IRQ, HandleInterrupt); - KernelLogStream(OK, "HDA") << "MSI enabled: vector " << base::dec << (uint64_t)MSI_VECTOR - << " (IRQ slot " << (uint64_t)MSI_IRQ << ")" << (is64bit ? " [64-bit]" : " [32-bit]"); + KernelLogStream(OK, "HDA") << "MSI enabled: IRQ slot " << base::dec << (uint64_t)MSI_IRQ << (is64bit ? " [64-bit]" : " [32-bit]"); return true; } @@ -1023,8 +1018,8 @@ namespace Drivers::Audio::IntelHda { (uint8_t)Pci::PCI_REG_COMMAND); Pci::LegacyWrite16(g_bus, g_dev, g_func, (uint8_t)Pci::PCI_REG_COMMAND, command & ~Pci::PCI_CMD_INTX_DISABLE); - Hal::RegisterIrqHandler(irqLine, HandleInterrupt); - Hal::IoApic::UnmaskIrq(Hal::IoApic::GetGsiForIrq(irqLine)); + Arch::RegisterIrqHandler(irqLine, HandleInterrupt); + Arch::EnableLegacyIrq(irqLine); } else { KernelLogStream(WARNING, "HDA") << "No interrupt available, polling only"; } diff --git a/kernel/src/Drivers/Audio/IntelHda.hpp b/kernel/src/Drivers/Audio/IntelHda.hpp index 8117d3d..1c49cbd 100644 --- a/kernel/src/Drivers/Audio/IntelHda.hpp +++ b/kernel/src/Drivers/Audio/IntelHda.hpp @@ -235,9 +235,7 @@ namespace Drivers::Audio::IntelHda { // MSI configuration // ========================================================================= - constexpr uint8_t MSI_IRQ = 27; // IRQ slot 27 = vector 59 - constexpr uint32_t MSI_VECTOR = 59; - constexpr uint32_t MSI_ADDR_BASE = 0xFEE00000; + constexpr uint8_t MSI_IRQ = 27; // IRQ slot 27 // ========================================================================= // Audio stream state (one active output stream) diff --git a/kernel/src/Drivers/Graphics/DgpuPower.cpp b/kernel/src/Drivers/Graphics/DgpuPower.cpp index ba311a3..55633e7 100644 --- a/kernel/src/Drivers/Graphics/DgpuPower.cpp +++ b/kernel/src/Drivers/Graphics/DgpuPower.cpp @@ -10,7 +10,7 @@ #include #include #include -#include +#include #include #include diff --git a/kernel/src/Drivers/Graphics/IntelGPU.cpp b/kernel/src/Drivers/Graphics/IntelGPU.cpp index 436a048..780f408 100644 --- a/kernel/src/Drivers/Graphics/IntelGPU.cpp +++ b/kernel/src/Drivers/Graphics/IntelGPU.cpp @@ -7,6 +7,7 @@ */ #include "IntelGPU.hpp" +#include #include #include #include @@ -17,7 +18,7 @@ #include #include #include -#include +#include using namespace Kt; @@ -500,7 +501,7 @@ namespace Drivers::Graphics::IntelGPU { int busyWait = 100000; while (((g_auxLastStatus = ReadReg(g_auxControlReg)) & DP_AUX_SEND_BUSY) && --busyWait) - asm volatile("pause"); + Arch::Pause(); if (!busyWait) return -1; for (int i = 0; i < sendBytes; i += 4) { @@ -523,7 +524,7 @@ namespace Drivers::Graphics::IntelGPU { status = ReadReg(g_auxControlReg); g_auxLastStatus = status; if (!(status & DP_AUX_SEND_BUSY)) break; - asm volatile("pause"); + Arch::Pause(); } while (--completeWait); if (!completeWait) return -1; @@ -568,7 +569,7 @@ namespace Drivers::Graphics::IntelGPU { return true; if (nativeReply != 2 && i2cReply != 8 && result >= 0) return false; for (int delay = 0; delay < 40000; delay++) - asm volatile("pause"); + Arch::Pause(); } return false; } @@ -1012,7 +1013,7 @@ namespace Drivers::Graphics::IntelGPU { } static void ShortSpin() { - for (int i = 0; i < 20000; i++) asm volatile("pause"); + for (int i = 0; i < 20000; i++) Arch::Pause(); } // ---- Panel PWM ---------------------------------------------------------- @@ -1210,7 +1211,7 @@ namespace Drivers::Graphics::IntelGPU { if (status & GMBUS_SATOER) return false; if (status & bit) return true; if (Timekeeping::GetMilliseconds() - start > 50) return false; - asm volatile("pause"); + Arch::Pause(); } } @@ -1218,7 +1219,7 @@ namespace Drivers::Graphics::IntelGPU { uint64_t start = Timekeeping::GetMilliseconds(); while (ReadReg(GMBUS2) & GMBUS_ACTIVE) { if (Timekeeping::GetMilliseconds() - start > 50) return false; - asm volatile("pause"); + Arch::Pause(); } return true; } @@ -1706,7 +1707,7 @@ namespace Drivers::Graphics::IntelGPU { previous = current; if ((i & 0x3FF) == 0 && Timekeeping::GetMilliseconds() - started >= 100) return false; - asm volatile("pause"); + Arch::Pause(); } return false; } @@ -2098,7 +2099,7 @@ namespace Drivers::Graphics::IntelGPU { for (int i = 0; i < 200000; i++) { bool state = (ReadReg(g_transConfReg) & PIPECONF_STATE) != 0; if (state == enabled) return true; - asm volatile("pause"); + Arch::Pause(); } return false; } @@ -2366,7 +2367,7 @@ namespace Drivers::Graphics::IntelGPU { // Scanout memory is exposed through WC mappings. Drain userspace's // completed frame before programming the display-plane surface. - asm volatile("sfence" ::: "memory"); + Arch::StoreFence(); uint64_t gttOff = (index == 0) ? g_fbGttOffsetA : g_buf1GttOffset; WriteReg(DSPASURF, (uint32_t)gttOff); (void)ReadReg(DSPASURF); @@ -2511,7 +2512,7 @@ namespace Drivers::Graphics::IntelGPU { for (int i = 0; i < 100000; i++) { if (ReadReg(PIPEACONF) & PIPECONF_STATE) break; - asm volatile("pause"); + Arch::Pause(); } if (!(ReadReg(PIPEACONF) & PIPECONF_STATE)) { diff --git a/kernel/src/Drivers/Input/InputEvents.cpp b/kernel/src/Drivers/Input/InputEvents.cpp index 0f7be14..c8cbabb 100644 --- a/kernel/src/Drivers/Input/InputEvents.cpp +++ b/kernel/src/Drivers/Input/InputEvents.cpp @@ -7,7 +7,7 @@ #include "InputEvents.hpp" #include #include -#include +#include namespace Drivers::InputEvents { diff --git a/kernel/src/Drivers/Net/E1000.cpp b/kernel/src/Drivers/Net/E1000.cpp index 2b987d1..0eace78 100644 --- a/kernel/src/Drivers/Net/E1000.cpp +++ b/kernel/src/Drivers/Net/E1000.cpp @@ -12,8 +12,7 @@ #include #include #include -#include -#include +#include #include #include @@ -340,8 +339,8 @@ namespace Drivers::Net::E1000 { SetupRx(); SetupTx(); - Hal::RegisterIrqHandler(g_irqLine, HandleInterrupt); - Hal::IoApic::UnmaskIrq(Hal::IoApic::GetGsiForIrq(g_irqLine)); + Arch::RegisterIrqHandler(g_irqLine, HandleInterrupt); + Arch::EnableLegacyIrq(g_irqLine); WriteReg(REG_IMS, RX_INTERRUPT_MASK | ICR_LSC); g_initialized = true; @@ -450,8 +449,8 @@ namespace Drivers::Net::E1000 { SetupTx(); // Register interrupt handler - Hal::RegisterIrqHandler(g_irqLine, HandleInterrupt); - Hal::IoApic::UnmaskIrq(Hal::IoApic::GetGsiForIrq(g_irqLine)); + Arch::RegisterIrqHandler(g_irqLine, HandleInterrupt); + Arch::EnableLegacyIrq(g_irqLine); // Enable RX and link-change only. TX completion is polled by descriptor. WriteReg(REG_IMS, RX_INTERRUPT_MASK | ICR_LSC); diff --git a/kernel/src/Drivers/Net/E1000E.cpp b/kernel/src/Drivers/Net/E1000E.cpp index 085c94f..67cdee3 100644 --- a/kernel/src/Drivers/Net/E1000E.cpp +++ b/kernel/src/Drivers/Net/E1000E.cpp @@ -5,6 +5,7 @@ */ #include "E1000E.hpp" +#include #include #include #include @@ -12,8 +13,7 @@ #include #include #include -#include -#include +#include #include #include @@ -80,9 +80,7 @@ namespace Drivers::Net::E1000E { static constexpr uint32_t g_deviceTableSize = sizeof(g_deviceTable) / sizeof(g_deviceTable[0]); // MSI configuration - static constexpr uint8_t MSI_IRQ = 24; // IRQ slot 24 = vector 56 (first MSI slot) - static constexpr uint32_t MSI_VECTOR = 56; // IRQ_VECTOR_BASE + MSI_IRQ - static constexpr uint32_t MSI_ADDR_BASE = 0xFEE00000; // Local APIC message address range + static constexpr uint8_t MSI_IRQ = 24; // first MSI IRQ slot // Driver state static bool g_initialized = false; @@ -434,17 +432,9 @@ namespace Drivers::Net::E1000E { uint16_t msgCtrl = Pci::LegacyRead16(bus, dev, func, cap + 2); bool is64bit = (msgCtrl & (1 << 7)) != 0; - // Write Message Address (cap+4): BSP APIC ID 0, physical destination, fixed delivery - Pci::LegacyWrite32(bus, dev, func, cap + 4, MSI_ADDR_BASE); - - // Write Message Data (vector number, edge-triggered, fixed delivery) - if (is64bit) { - // 64-bit: Upper Address at cap+8, Data at cap+12 - Pci::LegacyWrite32(bus, dev, func, cap + 8, 0); - Pci::LegacyWrite16(bus, dev, func, cap + 12, MSI_VECTOR); - } else { - // 32-bit: Data at cap+8 - Pci::LegacyWrite16(bus, dev, func, cap + 8, MSI_VECTOR); + // Message Address/Data: IRQ slot MSI_IRQ on the boot CPU + if (!Pci::WriteMsiMessage(bus, dev, func, cap, MSI_IRQ)) { + return false; } // Enable MSI: set bit 0 (MSI Enable), clear bits 6:4 (single message) @@ -458,10 +448,9 @@ namespace Drivers::Net::E1000E { Pci::LegacyWrite16(bus, dev, func, (uint8_t)Pci::PCI_REG_COMMAND, pciCmd); // Register the interrupt handler for MSI vector - Hal::RegisterIrqHandler(MSI_IRQ, HandleInterrupt); + Arch::RegisterIrqHandler(MSI_IRQ, HandleInterrupt); - KernelLogStream(OK, "E1000E") << "MSI enabled: vector " << base::dec << (uint64_t)MSI_VECTOR - << " (IRQ slot " << (uint64_t)MSI_IRQ << ")" << (is64bit ? " [64-bit]" : " [32-bit]"); + KernelLogStream(OK, "E1000E") << "MSI enabled: IRQ slot " << base::dec << (uint64_t)MSI_IRQ << (is64bit ? " [64-bit]" : " [32-bit]"); return true; } @@ -545,7 +534,7 @@ namespace Drivers::Net::E1000E { WriteReg(REG_CTRL, ctrl | CTRL_RST); for (int i = 0; i < 100000; i++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } for (int i = 0; i < 10000; i++) { if (!(ReadReg(REG_CTRL) & CTRL_RST)) break; @@ -592,8 +581,8 @@ namespace Drivers::Net::E1000E { (uint8_t)Pci::PCI_REG_COMMAND); Pci::LegacyWrite16(bus, device, function, (uint8_t)Pci::PCI_REG_COMMAND, command & ~Pci::PCI_CMD_INTX_DISABLE); - Hal::RegisterIrqHandler(g_irqLine, HandleInterrupt); - Hal::IoApic::UnmaskIrq(Hal::IoApic::GetGsiForIrq(g_irqLine)); + Arch::RegisterIrqHandler(g_irqLine, HandleInterrupt); + Arch::EnableLegacyIrq(g_irqLine); WriteReg(REG_IMS, RX_INTERRUPT_MASK | ICR_LSC); } else { KernelLogStream(WARNING, "E1000E") << "No MSI or legacy IRQ available, using polling mode"; diff --git a/kernel/src/Drivers/Net/Wifi/IwxConnect.cpp b/kernel/src/Drivers/Net/Wifi/IwxConnect.cpp index 83f0785..a32054d 100644 --- a/kernel/src/Drivers/Net/Wifi/IwxConnect.cpp +++ b/kernel/src/Drivers/Net/Wifi/IwxConnect.cpp @@ -26,7 +26,7 @@ #include #include #include -#include +#include using namespace Kt; diff --git a/kernel/src/Drivers/Net/Wifi/IwxFw.cpp b/kernel/src/Drivers/Net/Wifi/IwxFw.cpp index 8612cbf..c27e71b 100644 --- a/kernel/src/Drivers/Net/Wifi/IwxFw.cpp +++ b/kernel/src/Drivers/Net/Wifi/IwxFw.cpp @@ -12,6 +12,7 @@ */ #include "Iwx.hpp" +#include #include #include #include @@ -342,7 +343,7 @@ namespace Drivers::Net::Wifi { } Fs::Vfs::ReadBackendFile(file, buf, 0, size); Fs::Vfs::CloseBackendFile(file); - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); *outSize = size; return buf; diff --git a/kernel/src/Drivers/Net/Wifi/IwxMvm.cpp b/kernel/src/Drivers/Net/Wifi/IwxMvm.cpp index 330a9af..63c1739 100644 --- a/kernel/src/Drivers/Net/Wifi/IwxMvm.cpp +++ b/kernel/src/Drivers/Net/Wifi/IwxMvm.cpp @@ -11,7 +11,7 @@ #include #include #include -#include +#include using namespace Kt; diff --git a/kernel/src/Drivers/Net/Wifi/IwxTrans.cpp b/kernel/src/Drivers/Net/Wifi/IwxTrans.cpp index f5745b2..c48bef1 100644 --- a/kernel/src/Drivers/Net/Wifi/IwxTrans.cpp +++ b/kernel/src/Drivers/Net/Wifi/IwxTrans.cpp @@ -17,6 +17,7 @@ */ #include "Iwx.hpp" +#include #include "Ieee80211.hpp" #include #include @@ -26,8 +27,8 @@ #include #include #include -#include -#include +#include +#include using namespace Kt; @@ -38,8 +39,6 @@ namespace Drivers::Net::Wifi { // MSI-X/MSI interrupt slot. Slots in use elsewhere: 24 (E1000E), // 27 (Intel HDA), 28 (GPU vblank), 29 (xHCI). static constexpr uint8_t MSI_IRQ = 25; - static constexpr uint32_t MSI_VECTOR = 57; // IRQ_VECTOR_BASE + MSI_IRQ - static constexpr uint32_t MSI_ADDR_BASE = 0xFEE00000; // Cached MSI-X masks (a cause is ENABLED when its mask bit is clear). static uint32_t g_fhInitMask = 0; @@ -76,14 +75,14 @@ namespace Drivers::Net::Wifi { // exposed here; the loop count is calibrated the same way IntelHda does it. void IwxDelayUs(uint32_t us) { for (volatile uint32_t i = 0; i < us * 100u; i++) { - asm volatile("pause" ::: "memory"); + Arch::Pause(); } } void IwxDelayMs(uint32_t ms) { uint64_t start = Timekeeping::GetMilliseconds(); while (Timekeeping::GetMilliseconds() - start < ms) { - asm volatile("pause" ::: "memory"); + Arch::Pause(); } } @@ -144,13 +143,13 @@ namespace Drivers::Net::Wifi { uint32_t IwxReadPrph(uint32_t addr) { IwxWrite32(IWX_HBUS_TARG_PRPH_RADDR, PrphAddr(addr)); - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); return IwxRead32(IWX_HBUS_TARG_PRPH_RDAT); } void IwxWritePrph(uint32_t addr, uint32_t val) { IwxWrite32(IWX_HBUS_TARG_PRPH_WADDR, PrphAddr(addr)); - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); IwxWrite32(IWX_HBUS_TARG_PRPH_WDAT, val); } @@ -457,7 +456,7 @@ namespace Drivers::Net::Wifi { descs[i].rbid = (uint16_t)(i & 0xffff); descs[i].addr = g_iwx.RxQ.BufPhys[i]; } - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } // `stageSlots` reserves one page per concurrently queued frame; pass 0 for @@ -652,7 +651,7 @@ namespace Drivers::Net::Wifi { ci->mcr_size = (uint16_t)IWX_RX_QUEUE_CB_SIZE(IWX_RX_MQ_RING_COUNT); memcpy(g_iwx.ImlDma.Virt, g_iwx.Fw.Iml, g_iwx.Fw.ImlLen); - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); uint64_t paddr = g_iwx.CtxtInfo.Phys; IwxWrite32(IWX_CSR_CTXT_INFO_ADDR, (uint32_t)(paddr & 0xffffffff)); @@ -829,7 +828,7 @@ namespace Drivers::Net::Wifi { auto* scratch = (IwxPrphScratch*)g_iwx.PrphScratch.Virt; scratch->ctrl_cfg.pnvm_cfg.pnvm_base_addr = g_iwx.PnvmDma.Phys; scratch->ctrl_cfg.pnvm_cfg.pnvm_size = g_iwx.PnvmSize; - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } if (!IwxNicLock()) return false; @@ -1006,7 +1005,7 @@ namespace Drivers::Net::Wifi { } else { desc->num_tbs = 1; } - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); g_iwx.CmdDone = false; g_iwx.LastCmdId = code; @@ -1337,7 +1336,7 @@ namespace Drivers::Net::Wifi { auto* bc = (IwxGen3BcTblEntry*)ring.BcTbl.Virt; bc[idx].tfd_offset = (uint16_t)((hdrLen + payloadLen) | (chunks << 14)); - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); ring.Queued++; ring.Cur = (ring.Cur + 1) % IWX_TX_RING_COUNT; @@ -1488,7 +1487,7 @@ namespace Drivers::Net::Wifi { static bool IwxReadMem(uint32_t addr, uint32_t* out, int dwords) { if (!IwxNicLock()) return false; IwxWrite32(IWX_HBUS_TARG_MEM_RADDR, addr); - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); for (int i = 0; i < dwords; i++) out[i] = IwxRead32(IWX_HBUS_TARG_MEM_RDAT); IwxNicUnlock(); @@ -1786,11 +1785,17 @@ namespace Drivers::Net::Wifi { return false; } + // Message for our IRQ slot on the boot CPU + Arch::MsiMessage msi; + if (!Arch::ComposeMsi(MSI_IRQ, msi)) { + return false; + } + // Program entry 0: all causes are mapped to vector 0 by IwxConfMsixHw. volatile uint32_t* entry = (volatile uint32_t*)(g_iwx.Mmio + tableOff); - entry[0] = MSI_ADDR_BASE; // message address (BSP, physical, fixed) - entry[1] = 0; // upper address - entry[2] = MSI_VECTOR; // message data + entry[0] = (uint32_t)msi.address; // message address + entry[1] = (uint32_t)(msi.address >> 32); // upper address + entry[2] = msi.data; // message data entry[3] = 0; // vector control: unmasked msgCtrl |= (1 << 15); // MSI-X enable @@ -1801,9 +1806,8 @@ namespace Drivers::Net::Wifi { pciCmd |= Pci::PCI_CMD_INTX_DISABLE; Pci::LegacyWrite16(bus, dev, func, (uint8_t)Pci::PCI_REG_COMMAND, pciCmd); - Hal::RegisterIrqHandler(MSI_IRQ, HandleInterrupt); - KernelLogStream(OK, "WiFi") << "MSI-X enabled: vector " - << (uint64_t)MSI_VECTOR << " (IRQ slot " << (uint64_t)MSI_IRQ << ")"; + Arch::RegisterIrqHandler(MSI_IRQ, HandleInterrupt); + KernelLogStream(OK, "WiFi") << "MSI-X enabled: IRQ slot " << (uint64_t)MSI_IRQ; return true; } @@ -1814,14 +1818,10 @@ namespace Drivers::Net::Wifi { Pci::DisableInterruptDelivery(bus, dev, func); uint16_t msgCtrl = Pci::LegacyRead16(bus, dev, func, cap + 2); - bool is64 = (msgCtrl & (1 << 7)) != 0; - Pci::LegacyWrite32(bus, dev, func, cap + 4, MSI_ADDR_BASE); - if (is64) { - Pci::LegacyWrite32(bus, dev, func, cap + 8, 0); - Pci::LegacyWrite16(bus, dev, func, cap + 12, MSI_VECTOR); - } else { - Pci::LegacyWrite16(bus, dev, func, cap + 8, MSI_VECTOR); + // Message Address/Data: IRQ slot MSI_IRQ on the boot CPU + if (!Pci::WriteMsiMessage(bus, dev, func, cap, MSI_IRQ)) { + return false; } msgCtrl &= ~0x70; // single message @@ -1832,9 +1832,8 @@ namespace Drivers::Net::Wifi { pciCmd |= Pci::PCI_CMD_INTX_DISABLE; Pci::LegacyWrite16(bus, dev, func, (uint8_t)Pci::PCI_REG_COMMAND, pciCmd); - Hal::RegisterIrqHandler(MSI_IRQ, HandleInterrupt); - KernelLogStream(OK, "WiFi") << "MSI enabled: vector " - << (uint64_t)MSI_VECTOR << " (IRQ slot " << (uint64_t)MSI_IRQ << ")"; + Arch::RegisterIrqHandler(MSI_IRQ, HandleInterrupt); + KernelLogStream(OK, "WiFi") << "MSI enabled: IRQ slot " << (uint64_t)MSI_IRQ; return true; } diff --git a/kernel/src/Drivers/Net/Wifi/Wifi.cpp b/kernel/src/Drivers/Net/Wifi/Wifi.cpp index 344bd43..ed45c6e 100644 --- a/kernel/src/Drivers/Net/Wifi/Wifi.cpp +++ b/kernel/src/Drivers/Net/Wifi/Wifi.cpp @@ -12,6 +12,7 @@ */ #include "Wifi.hpp" +#include #include "Iwx.hpp" #include "Wpa.hpp" #include @@ -19,10 +20,9 @@ #include #include #include -#include -#include -#include -#include +#include +#include +#include #include #include @@ -444,8 +444,7 @@ namespace Drivers::Net::Wifi { auto* t = Smp::GetCpuData(i); if (!t || !t->started) continue; if (t->currentSlot >= 0 || t->reservedForKernelWork) continue; - if (Hal::LocalApic::SendFixedIpi(t->lapicId, - Hal::IRQ_VECTOR_BASE + Hal::IRQ_RESCHEDULE)) { + if (Arch::SendIpi(t->cpuIndex, Arch::IRQ_RESCHEDULE)) { g_lastApKickMs = now; return true; } @@ -624,7 +623,7 @@ namespace Drivers::Net::Wifi { uint64_t start = Timekeeping::GetMilliseconds(); while (g_iwx.ScanActive && Timekeeping::GetMilliseconds() - start < timeoutMs) { IwxProcessEvents(); - for (int i = 0; i < 200; i++) asm volatile("pause" ::: "memory"); + for (int i = 0; i < 200; i++) Arch::Pause(); } if (g_iwx.ScanActive) { diff --git a/kernel/src/Drivers/Net/Wifi/Wpa.cpp b/kernel/src/Drivers/Net/Wifi/Wpa.cpp index 7581bcb..4757197 100644 --- a/kernel/src/Drivers/Net/Wifi/Wpa.cpp +++ b/kernel/src/Drivers/Net/Wifi/Wpa.cpp @@ -17,12 +17,13 @@ */ #include "Wpa.hpp" +#include #include "Ieee80211.hpp" #include #include #include #include -#include +#include using namespace Kt; @@ -131,12 +132,9 @@ namespace Drivers::Net::Wifi { } pool; for (int i = 0; i < 8; i++) { - uint64_t tsc; - asm volatile("rdtsc; shl $32, %%rdx; or %%rdx, %%rax" - : "=a"(tsc) :: "rdx"); - pool.tsc[i] = tsc; + pool.tsc[i] = Arch::ReadTimestamp(); // Space the samples out so the low bits differ between them. - for (int k = 0; k < 64; k++) asm volatile("pause" ::: "memory"); + for (int k = 0; k < 64; k++) Arch::Pause(); } pool.counter = ++counter; memcpy(pool.mac, g_cfg.OwnMac, 6); diff --git a/kernel/src/Drivers/PS2/Keyboard.cpp b/kernel/src/Drivers/PS2/Keyboard.cpp index 64cbe85..54b2db0 100644 --- a/kernel/src/Drivers/PS2/Keyboard.cpp +++ b/kernel/src/Drivers/PS2/Keyboard.cpp @@ -5,14 +5,14 @@ */ #include "Keyboard.hpp" +#include #include "PS2Controller.hpp" #include #include #include #include -#include -#include +#include #include namespace Drivers::PS2::Keyboard { @@ -145,8 +145,8 @@ namespace Drivers::PS2::Keyboard { g_ExtendedScancode = false; // Register IRQ handler and unmask IRQ1 on the IOAPIC - Hal::RegisterIrqHandler(Hal::IRQ_KEYBOARD, HandleIRQ); - Hal::IoApic::UnmaskIrq(Hal::IoApic::GetGsiForIrq(Hal::IRQ_KEYBOARD)); + Arch::RegisterIrqHandler(Arch::IRQ_KEYBOARD, HandleIRQ); + Arch::EnableLegacyIrq(Arch::IRQ_KEYBOARD); Kt::KernelLogStream(Kt::OK, "PS2/KB") << "Keyboard driver initialized"; } @@ -299,7 +299,7 @@ namespace Drivers::PS2::Keyboard { } } // Yield the CPU while waiting - asm volatile("hlt"); + Arch::Halt(); } } diff --git a/kernel/src/Drivers/PS2/Mouse.cpp b/kernel/src/Drivers/PS2/Mouse.cpp index 131c513..5bf62ba 100644 --- a/kernel/src/Drivers/PS2/Mouse.cpp +++ b/kernel/src/Drivers/PS2/Mouse.cpp @@ -11,8 +11,7 @@ #include #include #include -#include -#include +#include #include namespace Drivers::PS2::Mouse { @@ -103,8 +102,8 @@ namespace Drivers::PS2::Mouse { g_PacketIndex = 0; // Register IRQ handler and unmask IRQ12 on the IOAPIC - Hal::RegisterIrqHandler(Hal::IRQ_MOUSE, HandleIRQ); - Hal::IoApic::UnmaskIrq(Hal::IoApic::GetGsiForIrq(Hal::IRQ_MOUSE)); + Arch::RegisterIrqHandler(Arch::IRQ_MOUSE, HandleIRQ); + Arch::EnableLegacyIrq(Arch::IRQ_MOUSE); Kt::KernelLogStream(Kt::OK, "PS2/Mouse") << "Mouse driver initialized"; } diff --git a/kernel/src/Drivers/Storage/Ahci.cpp b/kernel/src/Drivers/Storage/Ahci.cpp index b51954a..d5b4cc3 100644 --- a/kernel/src/Drivers/Storage/Ahci.cpp +++ b/kernel/src/Drivers/Storage/Ahci.cpp @@ -5,6 +5,7 @@ */ #include "Ahci.hpp" +#include #include "BlockDevice.hpp" #include #include @@ -91,14 +92,14 @@ namespace Drivers::Storage::Ahci { KernelLogStream(OK, "AHCI") << "BIOS handoff complete"; return; } - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } // If BIOS Busy, wait another 2 seconds if (ReadReg(REG_BOHC) & BOHC_BB) { for (int i = 0; i < 2000000; i++) { if (!(ReadReg(REG_BOHC) & BOHC_BB)) break; - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } @@ -130,7 +131,7 @@ namespace Drivers::Storage::Ahci { if (!(cmd & PORT_CMD_CR) && !(cmd & PORT_CMD_FR)) { return; } - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } KernelLogStream(WARNING, "AHCI") << "Port " << port << " stop timed out"; @@ -140,7 +141,7 @@ namespace Drivers::Storage::Ahci { // Wait for CR to clear before starting for (int i = 0; i < 500000; i++) { if (!(ReadPortReg(port, PORT_CMD) & PORT_CMD_CR)) break; - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } uint32_t cmd = ReadPortReg(port, PORT_CMD); @@ -170,7 +171,7 @@ namespace Drivers::Storage::Ahci { for (int i = 0; i < 1000000; i++) { uint32_t tfd = ReadPortReg(port, PORT_TFD); if (!(tfd & PORT_TFD_BSY)) break; - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } uint32_t sig = ReadPortReg(port, PORT_SIG); @@ -248,7 +249,7 @@ namespace Drivers::Storage::Ahci { // Wait for the slot to be free for (int i = 0; i < 500000; i++) { if (!(ReadPortReg(port, PORT_CI) & (1u << slot))) break; - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } // Check that the port isn't in error @@ -258,7 +259,7 @@ namespace Drivers::Storage::Ahci { for (int i = 0; i < 1000000; i++) { tfd = ReadPortReg(port, PORT_TFD); if (!(tfd & (PORT_TFD_BSY | PORT_TFD_DRQ))) break; - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } if (tfd & (PORT_TFD_BSY | PORT_TFD_DRQ)) { KernelLogStream(ERROR, "AHCI") << "Port " << port << " device busy before command issue"; @@ -294,7 +295,7 @@ namespace Drivers::Storage::Ahci { return false; } - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } KernelLogStream(ERROR, "AHCI") << "Port " << port << " command timeout"; @@ -554,7 +555,7 @@ namespace Drivers::Storage::Ahci { WriteReg(REG_GHC, ghc); return true; } - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } KernelLogStream(ERROR, "AHCI") << "HBA reset timed out"; diff --git a/kernel/src/Drivers/Storage/Gpt.cpp b/kernel/src/Drivers/Storage/Gpt.cpp index a7a2428..d164f6e 100644 --- a/kernel/src/Drivers/Storage/Gpt.cpp +++ b/kernel/src/Drivers/Storage/Gpt.cpp @@ -5,6 +5,7 @@ */ #include "Gpt.hpp" +#include #include "BlockDevice.hpp" #include #include @@ -565,13 +566,11 @@ namespace Drivers::Storage::Gpt { } // ------------------------------------------------------------------------- - // Simple GUID generation from RDTSC + mixing + // Simple GUID generation from the CPU timestamp counter + mixing // ------------------------------------------------------------------------- static uint64_t SimpleRand64() { - uint32_t lo, hi; - asm volatile ("rdtsc" : "=a"(lo), "=d"(hi)); - uint64_t val = ((uint64_t)hi << 32) | lo; + uint64_t val = Arch::ReadTimestamp(); // xorshift64 val ^= val << 13; val ^= val >> 7; diff --git a/kernel/src/Drivers/Storage/Nvme.cpp b/kernel/src/Drivers/Storage/Nvme.cpp index 2fef021..2a02b25 100644 --- a/kernel/src/Drivers/Storage/Nvme.cpp +++ b/kernel/src/Drivers/Storage/Nvme.cpp @@ -5,6 +5,7 @@ */ #include "Nvme.hpp" +#include #include "BlockDevice.hpp" #include #include @@ -13,7 +14,7 @@ #include #include #include -#include +#include using namespace Kt; @@ -191,7 +192,7 @@ namespace Drivers::Storage::Nvme { if ((spins & 0x3FFF) == 0 && Timekeeping::GetMilliseconds() - start >= kTimeoutMs) break; if (spins >= kMaxSpins) break; - asm volatile("pause" ::: "memory"); + Arch::Pause(); } KernelLogStream(ERROR, "NVMe") << "Command " << base::dec << (uint64_t)cmdId @@ -254,7 +255,7 @@ namespace Drivers::Storage::Nvme { if (!(csts & CSTS_RDY)) { return true; } - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } KernelLogStream(ERROR, "NVMe") << "Controller disable timeout"; @@ -280,7 +281,7 @@ namespace Drivers::Storage::Nvme { if (csts & CSTS_RDY) { return true; } - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } KernelLogStream(ERROR, "NVMe") << "Controller enable timeout"; diff --git a/kernel/src/Drivers/USB/Bluetooth/A2dp.cpp b/kernel/src/Drivers/USB/Bluetooth/A2dp.cpp index 03b05ac..e773236 100644 --- a/kernel/src/Drivers/USB/Bluetooth/A2dp.cpp +++ b/kernel/src/Drivers/USB/Bluetooth/A2dp.cpp @@ -5,6 +5,7 @@ */ #include "A2dp.hpp" +#include #include "Sbc.hpp" #include "L2cap.hpp" #include "Hci.hpp" @@ -12,7 +13,7 @@ #include #include #include -#include +#include #include using namespace Kt; @@ -232,7 +233,7 @@ namespace Drivers::USB::Bluetooth::A2dp { return true; } for (int j = 0; j < 100; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } g_expectLabel.store(0xFF, std::memory_order_release); @@ -1166,7 +1167,7 @@ namespace Drivers::USB::Bluetooth::A2dp { answered = true; break; } - for (int j = 0; j < 100; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 100; j++) Arch::CompilerBarrier(); } // Transaction over: CLOSE the client channel (FreeChannel sends the @@ -1200,7 +1201,7 @@ namespace Drivers::USB::Bluetooth::A2dp { Hci::ProcessPendingCommands(); auto* c = Hci::GetConnection(g_aclHandle.load(std::memory_order_acquire)); if (c && c->Encrypted) return true; - for (int j = 0; j < 100; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 100; j++) Arch::CompilerBarrier(); } return false; } @@ -1236,7 +1237,7 @@ namespace Drivers::USB::Bluetooth::A2dp { && Timekeeping::GetMilliseconds() - waitStart < timeoutMs) { Xhci::PollEvents(); Hci::DrainEvents(); - for (int j = 0; j < 100; j++) asm volatile("pause" ::: "memory"); + for (int j = 0; j < 100; j++) Arch::Pause(); } return IsReady(); } @@ -1268,7 +1269,7 @@ namespace Drivers::USB::Bluetooth::A2dp { Hci::DrainEvents(); Hci::ProcessPendingCommands(); if (g_sigCid.load(std::memory_order_acquire) != 0) break; - for (int j = 0; j < 100; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 100; j++) Arch::CompilerBarrier(); } } if (g_sigCid.load(std::memory_order_acquire) != 0) { @@ -1327,7 +1328,7 @@ namespace Drivers::USB::Bluetooth::A2dp { // timer. if (L2cap::ConnRspResult(sig) == L2cap::CONN_PENDING) waitBudgetMs = timeoutMs; - for (int j = 0; j < 100; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 100; j++) Arch::CompilerBarrier(); } uint16_t readySig = g_sigCid.load(std::memory_order_acquire); if (readySig != 0) { @@ -1364,7 +1365,7 @@ namespace Drivers::USB::Bluetooth::A2dp { while (Timekeeping::GetMilliseconds() - st < 400) { Xhci::PollEvents(); Hci::DrainEvents(); - for (int j = 0; j < 200; j++) asm volatile("pause" ::: "memory"); + for (int j = 0; j < 200; j++) Arch::Pause(); } } @@ -1387,7 +1388,7 @@ namespace Drivers::USB::Bluetooth::A2dp { Hci::DrainEvents(); State s = g_state.load(std::memory_order_acquire); if (s == State::Open || s == State::Streaming) break; - for (int j = 0; j < 100; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 100; j++) Arch::CompilerBarrier(); } State s = g_state.load(std::memory_order_acquire); if (s != State::Open && s != State::Streaming) { @@ -1495,7 +1496,7 @@ namespace Drivers::USB::Bluetooth::A2dp { g_mediaCid.store(other, std::memory_order_release); break; } - for (int j = 0; j < 100; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 100; j++) Arch::CompilerBarrier(); } if (g_mediaCid.load(std::memory_order_acquire) == 0) { @@ -1879,7 +1880,7 @@ namespace Drivers::USB::Bluetooth::A2dp { if (g_serviceActive.compare_exchange_weak(expected, true, std::memory_order_acquire)) return true; - asm volatile("pause" ::: "memory"); + Arch::Pause(); } return false; } diff --git a/kernel/src/Drivers/USB/Bluetooth/Bluetooth.cpp b/kernel/src/Drivers/USB/Bluetooth/Bluetooth.cpp index 14a6434..4917cde 100644 --- a/kernel/src/Drivers/USB/Bluetooth/Bluetooth.cpp +++ b/kernel/src/Drivers/USB/Bluetooth/Bluetooth.cpp @@ -5,6 +5,7 @@ */ #include "Bluetooth.hpp" +#include #include "Hci.hpp" #include "A2dp.hpp" #include "IntelFirmware.hpp" @@ -15,8 +16,8 @@ #include #include #include -#include -#include +#include +#include #include using namespace Kt; @@ -223,7 +224,7 @@ namespace Drivers::USB::Bluetooth { uint64_t start = Timekeeping::GetMilliseconds(); while (Timekeeping::GetMilliseconds() - start < 200) { Xhci::PollEvents(); - asm volatile("pause" ::: "memory"); + Arch::Pause(); } // Start the event pipe BEFORE sending any HCI commands. @@ -498,7 +499,7 @@ namespace Drivers::USB::Bluetooth { auto* conn = Hci::GetConnectionByIndex(i); if (conn && conn->Active) { anyActive = true; break; } } - for (int k = 0; k < 200; k++) asm volatile("pause" ::: "memory"); + for (int k = 0; k < 200; k++) Arch::Pause(); } if (anyActive) return false; } @@ -551,7 +552,7 @@ namespace Drivers::USB::Bluetooth { Hci::DrainEvents(); for (int j = 0; j < 200; j++) { - asm volatile("pause" ::: "memory"); + Arch::Pause(); } } @@ -568,7 +569,7 @@ namespace Drivers::USB::Bluetooth { Xhci::PollEvents(); Hci::DrainEvents(); for (int j = 0; j < 100; j++) - asm volatile("pause" ::: "memory"); + Arch::Pause(); } if (Hci::IsInquiryActive() && !Hci::CancelInquiry()) return -1; } @@ -615,7 +616,7 @@ namespace Drivers::USB::Bluetooth { Hci::DrainEvents(); target = FindAclConnection(bdAddr); for (int j = 0; j < 200; j++) { - asm volatile("pause" ::: "memory"); + Arch::Pause(); } } if (!target) return -1; @@ -635,7 +636,7 @@ namespace Drivers::USB::Bluetooth { Hci::ProcessPendingCommands(); if (!target->Active || !SameAddress(target->BdAddr, bdAddr)) break; if (target->Encrypted) break; - for (int k = 0; k < 200; k++) asm volatile("pause" ::: "memory"); + for (int k = 0; k < 200; k++) Arch::Pause(); } } diff --git a/kernel/src/Drivers/USB/Bluetooth/Hci.cpp b/kernel/src/Drivers/USB/Bluetooth/Hci.cpp index 44c863b..b4ae308 100644 --- a/kernel/src/Drivers/USB/Bluetooth/Hci.cpp +++ b/kernel/src/Drivers/USB/Bluetooth/Hci.cpp @@ -5,6 +5,7 @@ */ #include "Hci.hpp" +#include #include "L2cap.hpp" #include "A2dp.hpp" #include @@ -17,7 +18,7 @@ #include #include #include -#include +#include using namespace Kt; @@ -632,7 +633,7 @@ namespace Drivers::USB::Bluetooth::Hci { static void BusyWaitMs(uint64_t ms) { uint64_t start = Timekeeping::GetMilliseconds(); while (Timekeeping::GetMilliseconds() - start < ms) { - asm volatile("pause" ::: "memory"); + Arch::Pause(); } } @@ -643,7 +644,7 @@ namespace Drivers::USB::Bluetooth::Hci { Xhci::PollEvents(); DrainEvents(); for (int j = 0; j < 100; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } } @@ -835,7 +836,7 @@ namespace Drivers::USB::Bluetooth::Hci { } for (int j = 0; j < 100; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } @@ -884,7 +885,7 @@ namespace Drivers::USB::Bluetooth::Hci { } for (int j = 0; j < 100; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } @@ -1644,7 +1645,7 @@ namespace Drivers::USB::Bluetooth::Hci { } } - for (int j = 0; j < 100; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 100; j++) Arch::CompilerBarrier(); } // Distinguish "pipe dead" (0 completions) from "responses arriving but @@ -1734,7 +1735,7 @@ namespace Drivers::USB::Bluetooth::Hci { if (outStatus) *outStatus = g_secureStatus; return true; } - for (int j = 0; j < 100; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 100; j++) Arch::CompilerBarrier(); } return false; } @@ -1809,7 +1810,7 @@ namespace Drivers::USB::Bluetooth::Hci { << " byte " << g_ssBytesSent; return false; } - asm volatile("pause" ::: "memory"); + Arch::Pause(); } g_aclTxLock.Acquire(); @@ -1846,7 +1847,7 @@ namespace Drivers::USB::Bluetooth::Hci { DrainEvents(); if (g_aclTxTransportError.load(std::memory_order_acquire)) return false; if (Timekeeping::GetMilliseconds() - start > timeoutMs) return false; - asm volatile("pause" ::: "memory"); + Arch::Pause(); } return !g_aclTxTransportError.load(std::memory_order_acquire); } @@ -1874,7 +1875,7 @@ namespace Drivers::USB::Bluetooth::Hci { Xhci::PollEvents(); DrainEvents(); if (g_intelBootup) return true; - for (int j = 0; j < 100; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 100; j++) Arch::CompilerBarrier(); } KernelLogStream(ERROR, "BT-HCI") << "Timed out waiting for Intel bootup event"; diff --git a/kernel/src/Drivers/USB/Bluetooth/IntelFirmware.cpp b/kernel/src/Drivers/USB/Bluetooth/IntelFirmware.cpp index 8fa3cf5..4c9e076 100644 --- a/kernel/src/Drivers/USB/Bluetooth/IntelFirmware.cpp +++ b/kernel/src/Drivers/USB/Bluetooth/IntelFirmware.cpp @@ -12,13 +12,14 @@ */ #include "IntelFirmware.hpp" +#include #include "Hci.hpp" #include #include #include #include #include -#include +#include using namespace Kt; @@ -475,7 +476,7 @@ namespace Drivers::USB::Bluetooth { } Fs::Vfs::ReadBackendFile(file, fw, 0, size); Fs::Vfs::CloseBackendFile(file); - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); // 4. Run the secure download + boot sequence. bool ok = DownloadImage(fw, size, ver, cnvi, cnvr); diff --git a/kernel/src/Drivers/USB/Bluetooth/L2cap.cpp b/kernel/src/Drivers/USB/Bluetooth/L2cap.cpp index e3a5fb7..348af65 100644 --- a/kernel/src/Drivers/USB/Bluetooth/L2cap.cpp +++ b/kernel/src/Drivers/USB/Bluetooth/L2cap.cpp @@ -5,6 +5,7 @@ */ #include "L2cap.hpp" +#include #include "Hci.hpp" #include "A2dp.hpp" #include "Avrcp.hpp" @@ -12,7 +13,7 @@ #include #include #include -#include +#include using namespace Kt; @@ -554,7 +555,7 @@ namespace Drivers::USB::Bluetooth::L2cap { if (ch && ch->Configured) return true; for (int j = 0; j < 100; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } return false; diff --git a/kernel/src/Drivers/USB/HidKeyboard.cpp b/kernel/src/Drivers/USB/HidKeyboard.cpp index 860ae63..afeedcb 100644 --- a/kernel/src/Drivers/USB/HidKeyboard.cpp +++ b/kernel/src/Drivers/USB/HidKeyboard.cpp @@ -7,7 +7,7 @@ #include "HidKeyboard.hpp" #include #include -#include +#include #include using namespace Kt; diff --git a/kernel/src/Drivers/USB/MassStorage.cpp b/kernel/src/Drivers/USB/MassStorage.cpp index 6471aa5..ed97158 100644 --- a/kernel/src/Drivers/USB/MassStorage.cpp +++ b/kernel/src/Drivers/USB/MassStorage.cpp @@ -5,6 +5,7 @@ */ #include "MassStorage.hpp" +#include #include "Xhci.hpp" #include "UsbDevice.hpp" #include @@ -15,7 +16,7 @@ #include #include #include -#include +#include #include using namespace Kt; @@ -130,12 +131,10 @@ namespace Drivers::USB::MassStorage { } static void PollWaitMs(uint32_t ms) { - uint64_t flags; - asm volatile("pushfq; pop %0" : "=r"(flags)); - if ((flags & (1 << 9)) == 0) { + if (!Arch::InterruptsEnabled()) { for (uint64_t i = 0; i < (uint64_t)ms * 1000; i++) { Xhci::PollEvents(); - asm volatile("outb %%al, $0x80" ::: "memory"); + Arch::IoDelay(); } return; } @@ -143,7 +142,7 @@ namespace Drivers::USB::MassStorage { uint64_t start = Timekeeping::GetMilliseconds(); while (Timekeeping::GetMilliseconds() - start < ms) { Xhci::PollEvents(); - asm volatile("pause" ::: "memory"); + Arch::Pause(); } } @@ -221,7 +220,7 @@ namespace Drivers::USB::MassStorage { std::memory_order_acquire, std::memory_order_relaxed)) { expected = false; - asm volatile("pause" ::: "memory"); + Arch::Pause(); } } diff --git a/kernel/src/Drivers/USB/UsbDevice.cpp b/kernel/src/Drivers/USB/UsbDevice.cpp index 3ae2f8c..ea25a85 100644 --- a/kernel/src/Drivers/USB/UsbDevice.cpp +++ b/kernel/src/Drivers/USB/UsbDevice.cpp @@ -5,6 +5,7 @@ */ #include "UsbDevice.hpp" +#include #include "Xhci.hpp" #include "HidKeyboard.hpp" #include "HidMouse.hpp" @@ -15,24 +16,22 @@ #include #include #include -#include +#include using namespace Kt; static void BusyWaitMs(uint64_t ms) { - uint64_t flags; - asm volatile("pushfq; pop %0" : "=r"(flags)); - if (flags & (1 << 9)) { + if (Arch::InterruptsEnabled()) { // Interrupts enabled — use timer-based delay uint64_t start = Timekeeping::GetMilliseconds(); while (Timekeeping::GetMilliseconds() - start < ms) { - asm volatile("pause" ::: "memory"); + Arch::Pause(); } } else { - // Interrupts disabled (e.g. timer tick context) — use I/O port delay - // Each outb to port 0x80 takes ~1µs on x86 + // Interrupts disabled (e.g. timer tick context) — the tick is not + // advancing here, so count ~1 µs delays instead for (uint64_t i = 0; i < ms * 1000; i++) { - asm volatile("outb %%al, $0x80" ::: "memory"); + Arch::IoDelay(); } } } diff --git a/kernel/src/Drivers/USB/Xhci.cpp b/kernel/src/Drivers/USB/Xhci.cpp index 2491379..1885520 100644 --- a/kernel/src/Drivers/USB/Xhci.cpp +++ b/kernel/src/Drivers/USB/Xhci.cpp @@ -5,6 +5,7 @@ */ #include "Xhci.hpp" +#include #include #include "UsbDevice.hpp" #include "HidKeyboard.hpp" @@ -18,28 +19,26 @@ #include #include #include -#include -#include -#include +#include +#include +#include #include #include using namespace Kt; static void BusyWaitMs(uint64_t ms) { - uint64_t flags; - asm volatile("pushfq; pop %0" : "=r"(flags)); - if (flags & (1 << 9)) { + if (Arch::InterruptsEnabled()) { // Interrupts enabled — use timer-based delay uint64_t start = Timekeeping::GetMilliseconds(); while (Timekeeping::GetMilliseconds() - start < ms) { - asm volatile("pause" ::: "memory"); + Arch::Pause(); } } else { - // Interrupts disabled (e.g. timer tick context) — use I/O port delay - // Each outb to port 0x80 takes ~1µs on x86 + // Interrupts disabled (e.g. timer tick context) — the tick is not + // advancing here, so count ~1 µs delays instead for (uint64_t i = 0; i < ms * 1000; i++) { - asm volatile("outb %%al, $0x80" ::: "memory"); + Arch::IoDelay(); } } } @@ -429,17 +428,9 @@ namespace Drivers::USB::Xhci { uint16_t msgCtrl = Pci::LegacyRead16(bus, dev, func, cap + 2); bool is64bit = (msgCtrl & (1 << 7)) != 0; - // Write Message Address (cap+4): BSP APIC ID 0, physical destination, fixed delivery - Pci::LegacyWrite32(bus, dev, func, cap + 4, MSI_ADDR_BASE); - - // Write Message Data (vector number, edge-triggered, fixed delivery) - if (is64bit) { - // 64-bit: Upper Address at cap+8, Data at cap+12 - Pci::LegacyWrite32(bus, dev, func, cap + 8, 0); - Pci::LegacyWrite16(bus, dev, func, cap + 12, MSI_VECTOR); - } else { - // 32-bit: Data at cap+8 - Pci::LegacyWrite16(bus, dev, func, cap + 8, MSI_VECTOR); + // Message Address/Data: IRQ slot MSI_IRQ on the boot CPU + if (!Pci::WriteMsiMessage(bus, dev, func, cap, MSI_IRQ)) { + return false; } // Enable MSI: set bit 0 (MSI Enable), clear bits 6:4 (single message) @@ -453,10 +444,9 @@ namespace Drivers::USB::Xhci { Pci::LegacyWrite16(bus, dev, func, (uint8_t)Pci::PCI_REG_COMMAND, pciCmd); // Register the interrupt handler for MSI vector - Hal::RegisterIrqHandler(MSI_IRQ, HandleInterrupt); + Arch::RegisterIrqHandler(MSI_IRQ, HandleInterrupt); - KernelLogStream(OK, "xHCI") << "MSI enabled: vector " << base::dec << (uint64_t)MSI_VECTOR - << " (IRQ slot " << (uint64_t)MSI_IRQ << ")" << (is64bit ? " [64-bit]" : " [32-bit]"); + KernelLogStream(OK, "xHCI") << "MSI enabled: IRQ slot " << base::dec << (uint64_t)MSI_IRQ << (is64bit ? " [64-bit]" : " [32-bit]"); return true; } @@ -814,7 +804,7 @@ namespace Drivers::USB::Xhci { } // Small delay for (int j = 0; j < 100; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } @@ -936,7 +926,7 @@ namespace Drivers::USB::Xhci { return g_xferCompletionCode[slotId]; } for (int j = 0; j < 100; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } @@ -1358,7 +1348,7 @@ namespace Drivers::USB::Xhci { } for (int j = 0; j < 100; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } @@ -1552,7 +1542,7 @@ namespace Drivers::USB::Xhci { break; } for (int j = 0; j < 100; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } @@ -1665,7 +1655,7 @@ namespace Drivers::USB::Xhci { for (uint32_t i = 0; i < 100000; i++) { if (ReadOp(OP_USBSTS) & USBSTS_HCH) break; - for (int j = 0; j < 10; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 10; j++) Arch::CompilerBarrier(); } if (!(ReadOp(OP_USBSTS) & USBSTS_HCH)) { @@ -1677,11 +1667,11 @@ namespace Drivers::USB::Xhci { WriteOp(OP_USBCMD, USBCMD_HCRST); for (uint32_t i = 0; i < 100000; i++) { if (!(ReadOp(OP_USBCMD) & USBCMD_HCRST)) break; - for (int j = 0; j < 10; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 10; j++) Arch::CompilerBarrier(); } for (uint32_t i = 0; i < 100000; i++) { if (!(ReadOp(OP_USBSTS) & USBSTS_CNR)) break; - for (int j = 0; j < 10; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 10; j++) Arch::CompilerBarrier(); } if (ReadOp(OP_USBSTS) & USBSTS_CNR) { KernelLogStream(WARNING, "xHCI") << "Controller not ready after reset"; @@ -1752,7 +1742,7 @@ namespace Drivers::USB::Xhci { WriteOp(OP_USBCMD, USBCMD_RS | USBCMD_INTE | USBCMD_HSEE); for (uint32_t i = 0; i < 100000; i++) { if (!(ReadOp(OP_USBSTS) & USBSTS_HCH)) break; - for (int j = 0; j < 10; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 10; j++) Arch::CompilerBarrier(); } KernelLogStream(OK, "xHCI") << "Controller started"; g_initialized = true; @@ -1783,7 +1773,7 @@ namespace Drivers::USB::Xhci { PollEvents(); uint32_t ps = ReadOp(OP_PORTSC_BASE + port * OP_PORTSC_STRIDE); if (ps & PORTSC_PRC) { resetDone = true; break; } - for (int j = 0; j < 100; j++) asm volatile("" ::: "memory"); + for (int j = 0; j < 100; j++) Arch::CompilerBarrier(); } if (!resetDone) { @@ -1924,7 +1914,7 @@ namespace Drivers::USB::Xhci { break; } for (int j = 0; j < 10; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } @@ -1945,7 +1935,7 @@ namespace Drivers::USB::Xhci { break; } for (int j = 0; j < 10; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } @@ -1955,7 +1945,7 @@ namespace Drivers::USB::Xhci { break; } for (int j = 0; j < 10; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } @@ -2074,7 +2064,7 @@ namespace Drivers::USB::Xhci { break; } for (int j = 0; j < 10; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } @@ -2123,7 +2113,7 @@ namespace Drivers::USB::Xhci { break; } for (int j = 0; j < 100; j++) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); } } diff --git a/kernel/src/Drivers/USB/Xhci.hpp b/kernel/src/Drivers/USB/Xhci.hpp index 9304e42..a43dff6 100644 --- a/kernel/src/Drivers/USB/Xhci.hpp +++ b/kernel/src/Drivers/USB/Xhci.hpp @@ -24,8 +24,6 @@ namespace Drivers::USB::Xhci { // the IRQ dispatcher has one handler per slot, so sharing would replace // one driver's handler and eventually strand USB or storage completions. constexpr uint8_t MSI_IRQ = 29; - constexpr uint32_t MSI_VECTOR = 61; - constexpr uint32_t MSI_ADDR_BASE = 0xFEE00000; // PCI class/subclass/progif for xHCI constexpr uint8_t PCI_CLASS_SERIAL = 0x0C; diff --git a/kernel/src/Efi/UEFI.hpp b/kernel/src/Efi/UEFI.hpp index 0de859d..f51d802 100644 --- a/kernel/src/Efi/UEFI.hpp +++ b/kernel/src/Efi/UEFI.hpp @@ -5,6 +5,7 @@ #pragma once #include +#include #include #include #include @@ -62,7 +63,8 @@ namespace Efi { /// typedef UINT64 EFI_VIRTUAL_ADDRESS; - #define EFIAPI __attribute__((__ms_abi__)) + // Firmware calling convention (Microsoft x64 ABI on x86_64, AAPCS on ARM) + #define EFIAPI ARCH_EFIAPI /* EFI structs */ typedef struct { diff --git a/kernel/src/Fs/Ext2.cpp b/kernel/src/Fs/Ext2.cpp index aa1baa9..d03417c 100644 --- a/kernel/src/Fs/Ext2.cpp +++ b/kernel/src/Fs/Ext2.cpp @@ -5,6 +5,7 @@ */ #include "Ext2.hpp" +#include #include "FsProbe.hpp" #include "BlockCache.hpp" #include @@ -2214,10 +2215,9 @@ namespace Fs::Ext2 { sb->s_block_group_nr = 0; sb->s_feature_incompat = 0x0002; // FILETYPE - // Generate UUID from RDTSC - uint32_t lo, hi; - asm volatile ("rdtsc" : "=a"(lo), "=d"(hi)); - uint32_t seed = lo ^ hi; + // Generate UUID from the CPU timestamp counter + uint64_t tsc = Arch::ReadTimestamp(); + uint32_t seed = (uint32_t)tsc ^ (uint32_t)(tsc >> 32); for (int i = 0; i < 16; i++) { seed = seed * 1103515245 + 12345; sb->s_uuid[i] = (uint8_t)(seed >> 16); diff --git a/kernel/src/Fs/Fat32.cpp b/kernel/src/Fs/Fat32.cpp index 60d048c..8c53179 100644 --- a/kernel/src/Fs/Fat32.cpp +++ b/kernel/src/Fs/Fat32.cpp @@ -5,6 +5,7 @@ */ #include "Fat32.hpp" +#include #include "FsProbe.hpp" #include #include @@ -2295,10 +2296,9 @@ namespace Fs::Fat32 { // Extended boot record bpb[64] = 0x80; // drive number bpb[66] = 0x29; // extended boot signature - // Volume serial (from RDTSC) - uint32_t lo, hi; - asm volatile ("rdtsc" : "=a"(lo), "=d"(hi)); - uint32_t serial = lo ^ hi; + // Volume serial (from the CPU timestamp counter) + uint64_t tsc = Arch::ReadTimestamp(); + uint32_t serial = (uint32_t)tsc ^ (uint32_t)(tsc >> 32); memcpy(bpb + 67, &serial, 4); // Volume label (11 bytes, space-padded) diff --git a/kernel/src/Hal/Apic/Interrupts.hpp b/kernel/src/Hal/Apic/Interrupts.hpp deleted file mode 100644 index 983ec39..0000000 --- a/kernel/src/Hal/Apic/Interrupts.hpp +++ /dev/null @@ -1,44 +0,0 @@ -/* - * Interrupts.hpp - * Hardware interrupt registration and dispatch - * Copyright (c) 2025 Daniel Hammer -*/ - -#pragma once -#include - -namespace Hal { - // IRQ handler function type. The parameter is the IRQ number (0-47). - // `fromUser` is derived directly from the saved interrupt-frame CS. Do not - // cache it in CpuData: an IRQ handler may context-switch and later resume - // on another CPU, making per-CPU saved-origin state stale. - using IrqHandler = void(*)(uint8_t irq, bool fromUser); - - // Number of IRQ slots supported (0-23: legacy ISA via IOAPIC, - // 24-45: MSI, 46-47: kernel IPIs) - constexpr int IRQ_COUNT = 48; - - // IRQ vector base: hardware IRQs start at IDT vector 32 - constexpr uint8_t IRQ_VECTOR_BASE = 32; - - // Well-known ISA IRQ assignments - constexpr uint8_t IRQ_TIMER = 0; - constexpr uint8_t IRQ_KEYBOARD = 1; - constexpr uint8_t IRQ_CASCADE = 2; - constexpr uint8_t IRQ_COM2 = 3; - constexpr uint8_t IRQ_COM1 = 4; - constexpr uint8_t IRQ_FLOPPY = 6; - constexpr uint8_t IRQ_RTC = 8; - constexpr uint8_t IRQ_MOUSE = 12; - constexpr uint8_t IRQ_ATA1 = 14; - constexpr uint8_t IRQ_ATA2 = 15; - constexpr uint8_t IRQ_TLB_SHOOTDOWN = 46; - constexpr uint8_t IRQ_RESCHEDULE = 47; - - // Register a handler for the given IRQ number (0-47) - void RegisterIrqHandler(uint8_t irq, IrqHandler handler); - - // Install IRQ stubs into the IDT and set up the dispatch table - void InitializeIrqHandlers(); - -}; diff --git a/kernel/src/Hal/SmpBoot.hpp b/kernel/src/Hal/SmpBoot.hpp deleted file mode 100644 index 82208f4..0000000 --- a/kernel/src/Hal/SmpBoot.hpp +++ /dev/null @@ -1,93 +0,0 @@ -/* - * SmpBoot.hpp - * Symmetric Multiprocessing bootstrap and per-CPU data - * Copyright (c) 2026 Daniel Hammer -*/ - -#pragma once -#include -#include -#include - -// ==================================================================== -// Assembly-visible offsets into CpuData -// These MUST match the struct layout below. Verified by static_assert -// in SmpBoot.cpp. -// ==================================================================== -#define CPUDATA_SELF_PTR 0 -#define CPUDATA_KERNEL_RSP 8 -#define CPUDATA_USER_RSP 16 -#define CPUDATA_CURRENT_SLOT 24 - -namespace Smp { - - static constexpr int MaxCPUs = 64; - - struct CpuData { - // === Fields accessed by assembly (offsets defined above) === - uint64_t selfPtr; // offset 0: pointer to self - uint64_t kernelRsp; // offset 8: kernel stack top for SYSCALL - uint64_t userRspScratch; // offset 16: scratch for saving user RSP - int32_t currentSlot; // offset 24: process table slot, -1 = idle - int32_t _pad0; // offset 28: padding - - // === C++-only fields === - int cpuIndex; // 0-based CPU index - uint32_t lapicId; // local APIC ID - uint64_t idleSavedRsp; // RSP saved when switching from idle to process - uint64_t currentFsBase = ~0ULL; // last IA32_FS_BASE written on this CPU - volatile bool started; // set by AP after init is complete - - Hal::TSS64* tss; // pointer to this CPU's TSS - bool hasMwait; // CPU supports MONITOR/MWAIT - - // When set, the scheduler tick must NOT pull this (idle) CPU away to - // run a ready process: its idle context is busy with long-running - // kernel work whose wall-clock-bounded waits would spuriously expire - // if the context only ran when nothing else is ready (observed: the - // deferred Bluetooth firmware download racing login-screen startup). - // Other CPUs schedule normally while this is held. - volatile bool reservedForKernelWork = false; - - // Set by the BSP timer when an IRQ bottom half needs process-safe - // execution even though runnable user processes keep this CPU busy. - // Schedule() consumes it by yielding once to the idle kernel context. - bool kernelWorkPending = false; - - // Per-CPU GDT and TSS (APs use these; BSP uses globals) - Hal::BasicGDT cpuGdt __attribute__((aligned(16))); - Hal::TSS64 cpuTss __attribute__((aligned(16))); - }; - - // Get the current CPU's data (reads gs:0 which holds the self-pointer) - inline CpuData* GetCurrentCpuData() { - CpuData* ptr; - asm volatile("mov %%gs:0, %0" : "=r"(ptr)); - return ptr; - } - - // Exception paths may run before InitBsp() has installed a GS base. Do - // not read gs:0 in that state: address zero would fault recursively and - // turn the original, diagnosable exception into a triple fault. - inline CpuData* TryGetCurrentCpuData() { - uint32_t low; - uint32_t high; - asm volatile("rdmsr" : "=a"(low), "=d"(high) : "c"(0xC0000101u)); - uint64_t base = ((uint64_t)high << 32) | low; - return (CpuData*)base; - } - - // Get CPU data by index - CpuData* GetCpuData(int index); - - // Number of online CPUs - int GetCpuCount(); - - // Initialize BSP per-CPU data (call before interrupts are enabled) - void InitBsp(); - - // Boot all Application Processors (call after all subsystems ready). - // `smp` is the boot contract's SMP block; if FeatureSmp was not - // advertised the caller may still pass it (cpuCount<=1 is a no-op). - void BootAPs(const montauk::boot::SmpInfo& smp); -} diff --git a/kernel/src/Io/IoPort.hpp b/kernel/src/Io/IoPort.hpp index e8327c8..3e1b144 100644 --- a/kernel/src/Io/IoPort.hpp +++ b/kernel/src/Io/IoPort.hpp @@ -1,47 +1,38 @@ /* - * PortIo.hpp - * Inline Intel port IO functions - * Copyright (c) 2025 Daniel Hammer + * IoPort.hpp + * Architecture-neutral port I/O interface + * Copyright (c) 2026 Daniel Hammer */ #pragma once #include +#include + +// Port I/O is a separate address space on x86. ACPI ("SystemIO" registers), +// legacy PCI configuration access and a few PC devices are defined in terms +// of it, so the accessors are part of the arch interface. Architectures +// without an I/O space either map it through a memory window (a PCI host +// bridge's I/O aperture) or report HasPortIo == false, in which case reads +// return all ones -- what a read from an absent device looks like -- and +// writes are dropped. namespace Io { - // OSDev website has the parameters for out(b/w/l) in the opposite order in their example, - // (port, then value), but the manpage on GNU ('man outb') has it in this order. - inline void Out8(uint8_t value, uint16_t port) { - asm ("outb %0, %1" : : "a"(value), "Nd"(port) : "memory"); - } + inline void Out8(uint8_t value, uint16_t port); + inline void Out16(uint16_t value, uint16_t port); + inline void Out32(uint32_t value, uint16_t port); - inline void Out16(uint16_t value, uint16_t port) { - asm ("outw %0, %1" : : "a"(value), "Nd"(port) : "memory"); - } + inline uint8_t In8(uint16_t port); + inline uint16_t In16(uint16_t port); + inline uint32_t In32(uint16_t port); - inline void Out32(uint32_t value, uint16_t port) { - asm ("outl %0, %1" : : "a"(value), "Nd"(port) : "memory"); - } + // Short (~1 µs) delay between accesses to slow legacy devices. + inline void IoPortWait(); - inline uint8_t In8(uint16_t port) { - uint8_t result; - asm volatile("inb %1, %0" : "=a"(result) : "Nd"(port) : "memory"); - return result; - } +} - inline uint16_t In16(uint16_t port) { - uint16_t result; - asm volatile("inw %1, %0" : "=a"(result) : "Nd"(port) : "memory"); - return result; - } - - inline uint32_t In32(uint16_t port) { - uint32_t result; - asm volatile("inl %1, %0" : "=a"(result) : "Nd"(port) : "memory"); - return result; - } - - inline void IoPortWait() { - Out8(0, 0x80); - } -}; \ No newline at end of file +#if MONTAUK_ARCH_X86_64 +#include +#elif MONTAUK_ARCH_AARCH64 +#include +#endif diff --git a/kernel/src/Ipc/Ipc.cpp b/kernel/src/Ipc/Ipc.cpp index 546552b..43fe503 100644 --- a/kernel/src/Ipc/Ipc.cpp +++ b/kernel/src/Ipc/Ipc.cpp @@ -5,6 +5,7 @@ */ #include "Ipc.hpp" +#include #include #include @@ -17,10 +18,9 @@ #include #include #include -#include -#include -#include -#include +#include +#include +#include #include #include #include @@ -328,7 +328,7 @@ namespace Ipc { break; } waitset->entriesLock.Release(); - asm volatile("pause"); + Arch::Pause(); } ReleaseRawObject(target); @@ -1648,8 +1648,8 @@ namespace Ipc { auto* proc = Sched::GetProcessSlot(s); if (proc == nullptr) continue; if (proc->state == Sched::ProcessState::Free) continue; - uint64_t pml4 = proc->pml4Phys; - if (pml4 == 0) continue; + uint64_t pageRoot = proc->pageTableRoot; + if (pageRoot == 0) continue; g_surfaceMapLocks[s].Acquire(); for (int i = 0; i < MaxSurfaceMapsPerProcess; i++) { @@ -1660,9 +1660,9 @@ namespace Ipc { uint64_t startVa = m.va + (uint64_t)m.numPages * 0x1000ULL; uint32_t rollbackPages = newPages - m.numPages; for (uint32_t p = m.numPages; p < newPages; p++) { - Memory::VMM::Paging::UnmapUserIn(pml4, m.va + (uint64_t)p * 0x1000ULL); + Memory::VMM::Paging::UnmapUserIn(pageRoot, m.va + (uint64_t)p * 0x1000ULL); } - Memory::ShootdownUserRange(pml4, startVa, rollbackPages); + Memory::ShootdownUserRange(pageRoot, startVa, rollbackPages); } g_surfaceMapLocks[s].Release(); } @@ -1673,8 +1673,8 @@ namespace Ipc { auto* proc = Sched::GetProcessSlot(s); if (proc == nullptr) continue; if (proc->state == Sched::ProcessState::Free) continue; - uint64_t pml4 = proc->pml4Phys; - if (pml4 == 0) continue; + uint64_t pageRoot = proc->pageTableRoot; + if (pageRoot == 0) continue; bool failed = false; g_surfaceMapLocks[s].Acquire(); @@ -1685,7 +1685,7 @@ namespace Ipc { for (uint32_t p = m.numPages; p < newPages; p++) { uint64_t va = m.va + (uint64_t)p * 0x1000ULL; - if (!Memory::VMM::Paging::MapUserIn(pml4, scratch[p], va)) { + if (!Memory::VMM::Paging::MapUserIn(pageRoot, scratch[p], va)) { failed = true; break; } @@ -1782,8 +1782,8 @@ namespace Ipc { auto* proc = Sched::GetProcessSlot(s); if (proc == nullptr) continue; if (proc->state == Sched::ProcessState::Free) continue; - uint64_t pml4 = proc->pml4Phys; - if (pml4 == 0) continue; + uint64_t pageRoot = proc->pageTableRoot; + if (pageRoot == 0) continue; g_surfaceMapLocks[s].Acquire(); for (int i = 0; i < MaxSurfaceMapsPerProcess; i++) { @@ -1804,17 +1804,17 @@ namespace Ipc { // recoverable error. for (uint32_t p = 0; p < common; p++) { uint64_t va = baseVa + (uint64_t)p * 0x1000ULL; - if (!Memory::VMM::Paging::MapUserIn(pml4, surface->physPages[p], va)) { + if (!Memory::VMM::Paging::MapUserIn(pageRoot, surface->physPages[p], va)) { Panic("ResizeSurface: MapUserIn failed refreshing existing PTE", nullptr); } } // Pages trimmed off the end on shrink: unmap them entirely. for (uint32_t p = newPages; p < mapPages; p++) { uint64_t va = baseVa + (uint64_t)p * 0x1000ULL; - Memory::VMM::Paging::UnmapUserIn(pml4, va); + Memory::VMM::Paging::UnmapUserIn(pageRoot, va); } - Memory::ShootdownUserRange(pml4, baseVa, flushPages); + Memory::ShootdownUserRange(pageRoot, baseVa, flushPages); m.numPages = newPages; } g_surfaceMapLocks[s].Release(); @@ -1841,7 +1841,7 @@ namespace Ipc { if (snapshot.type != HandleType::Surface || (snapshot.rights & RightMap) == 0) return 0; uint64_t va = 0; - if (MapSurfaceForPid((Surface*)snapshot.object, proc->pid, proc->pml4Phys, proc->heapNext, va) < 0) { + if (MapSurfaceForPid((Surface*)snapshot.object, proc->pid, proc->pageTableRoot, proc->heapNext, va) < 0) { return 0; } return va; @@ -1929,7 +1929,7 @@ namespace Ipc { return 0; } - int MapSurfaceForPid(Surface* surface, int pid, uint64_t pml4Phys, uint64_t& heapNext, + int MapSurfaceForPid(Surface* surface, int pid, uint64_t pageTableRoot, uint64_t& heapNext, uint64_t& outVa, bool reserveGrowthRange) { if (surface == nullptr) return -1; @@ -1979,10 +1979,10 @@ namespace Ipc { uint64_t baseVa = Sched::UserSurfaceBase + (uint64_t)mapIdx * Sched::UserSurfaceSlotSize; for (uint32_t i = 0; i < numPages; i++) { - if (!Memory::VMM::Paging::MapUserIn(pml4Phys, surface->physPages[i], + if (!Memory::VMM::Paging::MapUserIn(pageTableRoot, surface->physPages[i], baseVa + (uint64_t)i * 0x1000ULL)) { for (uint32_t j = 0; j < i; j++) { - Memory::VMM::Paging::UnmapUserIn(pml4Phys, baseVa + (uint64_t)j * 0x1000ULL); + Memory::VMM::Paging::UnmapUserIn(pageTableRoot, baseVa + (uint64_t)j * 0x1000ULL); } g_surfaceMapLocks[slot].Release(); surface->lock.Release(); @@ -2002,7 +2002,7 @@ namespace Ipc { return 0; } - int UnmapSurfaceForPid(Surface* surface, int pid, uint64_t pml4Phys) { + int UnmapSurfaceForPid(Surface* surface, int pid, uint64_t pageTableRoot) { if (surface == nullptr) return -1; int slot = SlotForPid(pid); @@ -2016,7 +2016,7 @@ namespace Ipc { uint64_t baseVa = g_surfaceMaps[slot][i].va; uint32_t numPages = g_surfaceMaps[slot][i].numPages; for (uint32_t p = 0; p < numPages; p++) { - Memory::VMM::Paging::UnmapUserIn(pml4Phys, + Memory::VMM::Paging::UnmapUserIn(pageTableRoot, baseVa + (uint64_t)p * 0x1000ULL); } @@ -2024,8 +2024,8 @@ namespace Ipc { // in this same process may still have one of these PTEs cached; // releasing the mapping reference can then destroy the surface // and recycle its frames while that sibling writes through its - // stale TLB entry. Quiesce every CPU using this PML4 first. - Memory::ShootdownUserRange(pml4Phys, baseVa, numPages); + // stale TLB entry. Quiesce every CPU using this address space first. + Memory::ShootdownUserRange(pageTableRoot, baseVa, numPages); g_surfaceMaps[slot][i].used = false; g_surfaceMaps[slot][i].surface = nullptr; @@ -2399,7 +2399,7 @@ namespace Ipc { break; } waitset->entriesLock.Release(); - asm volatile("pause"); + Arch::Pause(); } ReleaseRawObject(targetObject); @@ -2441,7 +2441,7 @@ namespace Ipc { // Teardown and init // ========================================================================== - void CleanupProcessSlot(int slot, int /*pid*/, uint64_t pml4Phys) { + void CleanupProcessSlot(int slot, int /*pid*/, uint64_t pageTableRoot) { if (slot < 0 || slot >= Sched::MaxProcesses) return; for (int h = 0; h < MaxHandlesPerProcess; h++) { @@ -2460,7 +2460,7 @@ namespace Ipc { } for (uint32_t p = 0; p < map.numPages; p++) { - Memory::VMM::Paging::UnmapUserIn(pml4Phys, map.va + (uint64_t)p * 0x1000ULL); + Memory::VMM::Paging::UnmapUserIn(pageTableRoot, map.va + (uint64_t)p * 0x1000ULL); } Object* surfaceObject = (Object*)map.surface; map.used = false; diff --git a/kernel/src/Ipc/Ipc.hpp b/kernel/src/Ipc/Ipc.hpp index 5041c12..8d6f70f 100644 --- a/kernel/src/Ipc/Ipc.hpp +++ b/kernel/src/Ipc/Ipc.hpp @@ -151,9 +151,9 @@ namespace Ipc { uint32_t fillPixel); uint64_t MapSurfaceHandle(int handle); int ResizeSurfaceHandle(int handle, uint64_t newSize); - int MapSurfaceForPid(Surface* surface, int pid, uint64_t pml4Phys, uint64_t& heapNext, + int MapSurfaceForPid(Surface* surface, int pid, uint64_t pageTableRoot, uint64_t& heapNext, uint64_t& outVa, bool reserveGrowthRange = true); - int UnmapSurfaceForPid(Surface* surface, int pid, uint64_t pml4Phys); + int UnmapSurfaceForPid(Surface* surface, int pid, uint64_t pageTableRoot); int OpenProcessHandle(int pid); void ProcessStartedInSlot(int slot, int pid); @@ -171,6 +171,6 @@ namespace Ipc { int WaitsetWaitHandle(int waitsetHandle, WaitsetReady* outReady, uint64_t timeoutMs); void NotifyObjectChanged(Object* object); - void CleanupProcessSlot(int slot, int pid, uint64_t pml4Phys); + void CleanupProcessSlot(int slot, int pid, uint64_t pageTableRoot); } diff --git a/kernel/src/Memory/Heap.cpp b/kernel/src/Memory/Heap.cpp index d155470..e9fe0b3 100644 --- a/kernel/src/Memory/Heap.cpp +++ b/kernel/src/Memory/Heap.cpp @@ -11,7 +11,6 @@ #include #include #include -#include #include #include @@ -21,7 +20,7 @@ namespace Memory { HeapAllocator::Header* HeapAllocator::GetHeader(void* block) { - return (Header*)(block - sizeof(Header)); + return (Header*)((uint8_t*)block - sizeof(Header)); } size_t HeapAllocator::GetAllocatedBlockSize(void* ptr) { diff --git a/kernel/src/Memory/PageFrameAllocator.cpp b/kernel/src/Memory/PageFrameAllocator.cpp index 442d6e1..fa3f09c 100644 --- a/kernel/src/Memory/PageFrameAllocator.cpp +++ b/kernel/src/Memory/PageFrameAllocator.cpp @@ -49,7 +49,7 @@ namespace Memory { current->size -= 0x1000; Lock.Release(); - return (void*)current_end_addr - 0x1000; + return (void*)((uint8_t*)current_end_addr - 0x1000); } } diff --git a/kernel/src/Memory/Paging.asm b/kernel/src/Memory/Paging.asm deleted file mode 100644 index c151296..0000000 --- a/kernel/src/Memory/Paging.asm +++ /dev/null @@ -1,12 +0,0 @@ -[bits 64] -section .text -global GetCR3 -global LoadCR3 - -GetCR3: - mov rax, cr3 - ret - -LoadCR3: - mov cr3, rdi - ret \ No newline at end of file diff --git a/kernel/src/Memory/Paging.hpp b/kernel/src/Memory/Paging.hpp index 8d82d52..0722992 100644 --- a/kernel/src/Memory/Paging.hpp +++ b/kernel/src/Memory/Paging.hpp @@ -1,143 +1,87 @@ +/* + * Paging.hpp + * Architecture-neutral virtual memory interface + * Copyright (c) 2025 Daniel Hammer +*/ + #pragma once #include #include #include #include +// An address space is identified by the physical address of its root page +// table (x86 PML4, ARM translation table base). The lower half of every +// address space belongs to its process; the upper half is the kernel's and +// is shared by all of them. The page-table format itself is private to the +// architecture (Arch//Paging.cpp implements this class). + namespace Memory::VMM { - struct PageTableEntry { - std::uint8_t Present : 1; - std::uint8_t Writable : 1; - std::uint8_t Supervisor : 1; - std::uint8_t WriteThrough : 1; - std::uint8_t CacheDisabled : 1; - std::uint8_t Accessed : 1; - std::uint8_t Ignore : 1; - std::uint8_t LargerPages : 1; - std::uint8_t PageSize : 1; - std::uint8_t Available : 3; - std::uint64_t Address : 52; - }; - - struct PageTableEntry40Bit { - std::uint8_t Present : 1; - std::uint8_t Writable : 1; - std::uint8_t Supervisor : 1; - std::uint8_t WriteThrough : 1; - std::uint8_t CacheDisabled : 1; - std::uint8_t Accessed : 1; - std::uint8_t Ignore : 1; - std::uint8_t LargerPages : 1; - std::uint8_t PageSize : 1; - std::uint8_t Available : 3; - std::uint64_t Address : 40; - std::uint8_t AvailableHigh : 7; - std::uint8_t PK : 4; - std::uint8_t NX : 1; - }; - - - struct PageTable { - PageTableEntry entries[512]; - } __attribute__((packed)) __attribute__((aligned(0x1000))); - - struct VirtualAddress { - std::uint64_t address; - - // Page-table index extraction works for any byte address. Callers that - // truly require page alignment validate that explicitly before mapping. - VirtualAddress(std::uint64_t newAddress) : address(newAddress) {} - - uint64_t GetL4Index() { - return (address >> 39) & 0x1ff; - } - - uint64_t GetL3Index() { - return (address >> 30) & 0x1ff; - } - - uint64_t GetL2Index() { - return (address >> 21) & 0x1ff; - } - - uint64_t GetPageIndex() { - return (address >> 12) & 0x1ff; - } - - uint64_t GetIndex(size_t level) { - if (level == 4) - return GetL4Index(); - - else if (level == 3) - return GetL3Index(); - - else if (level == 2) - return GetL2Index(); - - else if (level == 1) - return GetPageIndex(); - - return 0; - } - }; class Paging { - PageTable* HandleLevel(VirtualAddress virtualAddress, PageTable* table, size_t level); - PageTable* HandleLevelUser(VirtualAddress virtualAddress, PageTable* table, size_t level); + std::uint64_t rootPhys{}; public: - PageTable* PML4{}; - Paging(); void Init(std::uint64_t kernelBaseVirt, std::uint64_t kernelSize, const montauk::boot::MemoryMap& memMap, const montauk::boot::Framebuffer& framebuffer); + + // Physical address of the kernel's root page table (the address space + // idle contexts and early boot run in). + std::uint64_t KernelRoot() const { return rootPhys; } + // Top of the contiguous HHDM mapping built by Init(). A physical // address at or above this is NOT reachable at phys+hhdmBase, and - // dereferencing it there faults (non-canonical addresses raise #GP - // rather than #PF). Anything that turns firmware-supplied numbers into - // HHDM pointers -- the AML interpreter's SystemMemory regions above all - // -- must check against this first. + // dereferencing it there faults (on x86 non-canonical addresses raise + // #GP rather than #PF). Anything that turns firmware-supplied numbers + // into HHDM pointers -- the AML interpreter's SystemMemory regions + // above all -- must check against this first. static std::uint64_t HhdmLimit(); + // Kernel mappings: normal memory, uncached device memory, and + // write-combining memory (framebuffers). void Map(std::uint64_t physicalAddress, std::uint64_t virtualAddress); void MapMMIO(std::uint64_t physicalAddress, std::uint64_t virtualAddress); void MapWC(std::uint64_t physicalAddress, std::uint64_t virtualAddress); void MapUser(std::uint64_t physicalAddress, std::uint64_t virtualAddress); - static std::uint64_t GetPhysAddr(std::uint64_t PML4, std::uint64_t virtualAddress, bool use40BitL1 = false); + + // Translate through an arbitrary address space (0 if unmapped). + static std::uint64_t GetPhysAddr(std::uint64_t root, std::uint64_t virtualAddress); std::uint64_t GetPhysAddr(std::uint64_t virtualAddress); - // Create a new PML4 with kernel-half (entries 256-511) copied from g_paging. - // Returns the physical address of the new PML4. - static std::uint64_t CreateUserPML4(); + // Create a new address space with the kernel half shared with + // g_paging. Returns the physical address of its root table. + static std::uint64_t CreateUserAddressSpace(); - // Map a page into an arbitrary PML4 (specified by physical address) with User bit set. - static bool MapUserIn(std::uint64_t pml4Phys, std::uint64_t physicalAddress, std::uint64_t virtualAddress); + // Map a user-accessible page into an arbitrary address space. + static bool MapUserIn(std::uint64_t root, std::uint64_t physicalAddress, std::uint64_t virtualAddress); - // Map ordinary user memory with explicit leaf permissions. Read access - // is implied by x86 paging; writable and executable are independent. - static bool MapUserInPermissions(std::uint64_t pml4Phys, + // Map ordinary user memory with explicit leaf permissions. Read + // access is always implied; writable and executable are independent. + static bool MapUserInPermissions(std::uint64_t root, std::uint64_t physicalAddress, std::uint64_t virtualAddress, bool writable, bool executable); // Change leaf permissions on an existing user page. - static bool ProtectUserIn(std::uint64_t pml4Phys, std::uint64_t virtualAddress, + static bool ProtectUserIn(std::uint64_t root, std::uint64_t virtualAddress, bool writable, bool executable); - // Map a page into an arbitrary PML4 with User + Write-Combining attributes. - static bool MapUserInWC(std::uint64_t pml4Phys, std::uint64_t physicalAddress, std::uint64_t virtualAddress); + // Map a user page with write-combining attributes. + static bool MapUserInWC(std::uint64_t root, std::uint64_t physicalAddress, std::uint64_t virtualAddress); - // Unmap a single page from an arbitrary PML4 (clears PTE + invalidates TLB). - static void UnmapUserIn(std::uint64_t pml4Phys, std::uint64_t virtualAddress); + // Unmap a single page from an arbitrary address space (clears the + // leaf and invalidates this CPU's TLB entry). + static void UnmapUserIn(std::uint64_t root, std::uint64_t virtualAddress); - // Free all user-half page table structures and physical pages (PML4 entries 0-255). - // Does NOT free the PML4 page itself (caller handles that). - // Skips MMIO/WC pages (WriteThrough or CacheDisabled set in PTE). - static void FreeUserHalf(std::uint64_t pml4Phys); + // Free all user-half page table structures and physical pages. + // Does NOT free the root table itself (caller handles that). + // Skips MMIO/WC pages (they are not owned by the process). + static void FreeUserHalf(std::uint64_t root); // Validate that every page in the given user range is mapped with // user permissions. If requireWrite is true, all pages must also be // writable. - static bool IsUserRangeAccessible(std::uint64_t pml4Phys, std::uint64_t virtualAddress, + static bool IsUserRangeAccessible(std::uint64_t root, std::uint64_t virtualAddress, std::uint64_t size, bool requireWrite = false); // Identity-map EFI runtime service regions so firmware code can @@ -147,6 +91,14 @@ public: extern Paging* g_paging; + // Cause of a page fault, decoded from the architecture's fault syndrome + // (x86 error code, ARM ESR) by its exception handler. + struct PageFault { + bool present; // the page was mapped: a protection violation + bool write; // the access was a write + bool execute; // the access was an instruction fetch + }; + // Protects user page table modifications from concurrent access. // Required for SMP: e.g. WinServer::Resize unmaps pages from the // desktop's page tables while the desktop may be allocating pages @@ -154,10 +106,10 @@ public: void LockUserPaging(); void UnlockUserPaging(); - extern "C" uint64_t GetCR3(); - extern "C" void LoadCR3(PageTable* PML4); + // Switch this CPU to the kernel address space (e.g. before calling + // firmware that relies on the identity-mapped EFI runtime regions). + void ActivateKernelAddressSpace(); - inline void FlushTLB() { - asm volatile("mov %%cr3, %%rax; mov %%rax, %%cr3" ::: "rax", "memory"); - } + // Invalidate this CPU's non-global TLB entries. + void FlushTLB(); }; diff --git a/kernel/src/Memory/UserRange.cpp b/kernel/src/Memory/UserRange.cpp index 7aaa8b9..f60a40c 100644 --- a/kernel/src/Memory/UserRange.cpp +++ b/kernel/src/Memory/UserRange.cpp @@ -8,15 +8,15 @@ */ #include "UserRange.hpp" +#include #include #include #include #include #include -#include -#include -#include +#include +#include #include namespace Memory { @@ -30,53 +30,53 @@ namespace Memory { // like a hardware fault rather than a lock-type regression. static kcp::Mutex g_tlbShootdownLock; static volatile uint64_t g_tlbShootdownSeq = 0; - static volatile uint64_t g_tlbShootdownPml4 = 0; + static volatile uint64_t g_tlbShootdownRoot = 0; static volatile uint64_t g_tlbShootdownStartVa = 0; static volatile uint32_t g_tlbShootdownPages = 0; static volatile uint64_t g_tlbShootdownDone[Smp::MaxCPUs] = {}; - static bool CpuCurrentlyUsesPml4(Smp::CpuData* cpu, uint64_t pml4Phys) { - if (cpu == nullptr || pml4Phys == 0 || cpu->currentSlot < 0) return false; + static bool CpuCurrentlyUsesRoot(Smp::CpuData* cpu, uint64_t pageTableRoot) { + if (cpu == nullptr || pageTableRoot == 0 || cpu->currentSlot < 0) return false; Sched::Process* proc = Sched::GetProcessSlot(cpu->currentSlot); if (proc == nullptr) return false; if (proc->state == Sched::ProcessState::Free) return false; - return proc->pml4Phys == pml4Phys; + return proc->pageTableRoot == pageTableRoot; } static void InvalidateLocalUserRange(uint64_t startVa, uint32_t pages) { if (pages == 0) return; if (pages > 1024) { - Memory::VMM::FlushTLB(); + Arch::FlushTlbAll(); return; } for (uint32_t p = 0; p < pages; p++) { uint64_t va = startVa + (uint64_t)p * 0x1000ULL; - asm volatile("invlpg (%0)" :: "r"(va) : "memory"); + Arch::FlushTlbPage(va); } } static void TlbShootdownIpiHandler(uint8_t, bool) { Smp::CpuData* cpu = Smp::GetCurrentCpuData(); uint64_t seq = g_tlbShootdownSeq; - uint64_t pml4 = g_tlbShootdownPml4; + uint64_t pageRoot = g_tlbShootdownRoot; uint64_t startVa = g_tlbShootdownStartVa; uint32_t pages = g_tlbShootdownPages; - if (CpuCurrentlyUsesPml4(cpu, pml4)) { + if (CpuCurrentlyUsesRoot(cpu, pageRoot)) { InvalidateLocalUserRange(startVa, pages); } if (cpu != nullptr && cpu->cpuIndex >= 0 && cpu->cpuIndex < Smp::MaxCPUs) { - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); g_tlbShootdownDone[cpu->cpuIndex] = seq; } } - void ShootdownUserRange(uint64_t pml4Phys, uint64_t startVa, uint32_t pages) { - if (pml4Phys == 0 || pages == 0) return; + void ShootdownUserRange(uint64_t pageTableRoot, uint64_t startVa, uint32_t pages) { + if (pageTableRoot == 0 || pages == 0) return; bool targets[Smp::MaxCPUs] = {}; Smp::CpuData* currentCpu = Smp::GetCurrentCpuData(); @@ -84,10 +84,10 @@ namespace Memory { g_tlbShootdownLock.Acquire(); uint64_t seq = g_tlbShootdownSeq + 1; - g_tlbShootdownPml4 = pml4Phys; + g_tlbShootdownRoot = pageTableRoot; g_tlbShootdownStartVa = startVa; g_tlbShootdownPages = pages; - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); g_tlbShootdownSeq = seq; for (int i = 0; i < Smp::GetCpuCount(); i++) { @@ -95,21 +95,20 @@ namespace Memory { if (cpu == nullptr || !cpu->started) continue; if (i == currentCpuIndex) { - if (CpuCurrentlyUsesPml4(cpu, pml4Phys)) { + if (CpuCurrentlyUsesRoot(cpu, pageTableRoot)) { InvalidateLocalUserRange(startVa, pages); } g_tlbShootdownDone[i] = seq; continue; } - if (!CpuCurrentlyUsesPml4(cpu, pml4Phys)) { + if (!CpuCurrentlyUsesRoot(cpu, pageTableRoot)) { g_tlbShootdownDone[i] = seq; continue; } targets[i] = true; - (void)Hal::LocalApic::SendFixedIpi(cpu->lapicId, - Hal::IRQ_VECTOR_BASE + Hal::IRQ_TLB_SHOOTDOWN); + (void)Arch::SendIpi(cpu->cpuIndex, Arch::IRQ_TLB_SHOOTDOWN); } for (int i = 0; i < Smp::GetCpuCount(); i++) { @@ -117,7 +116,7 @@ namespace Memory { uint32_t spins = 0; uint32_t retries = 0; while (g_tlbShootdownDone[i] != seq) { - asm volatile("pause"); + Arch::Pause(); if (++spins < 1000000) continue; // Delivery normally completes in a handful of cycles. Retry a @@ -132,8 +131,7 @@ namespace Memory { } Smp::CpuData* cpu = Smp::GetCpuData(i); if (cpu != nullptr && cpu->started) { - (void)Hal::LocalApic::SendFixedIpi(cpu->lapicId, - Hal::IRQ_VECTOR_BASE + Hal::IRQ_TLB_SHOOTDOWN); + (void)Arch::SendIpi(cpu->cpuIndex, Arch::IRQ_TLB_SHOOTDOWN); } } } @@ -141,7 +139,7 @@ namespace Memory { g_tlbShootdownLock.Release(); } - void UnmapAndFreeUserRange(uint64_t pml4Phys, uint64_t startVa, uint64_t pages) { + void UnmapAndFreeUserRange(uint64_t pageTableRoot, uint64_t startVa, uint64_t pages) { static constexpr uint32_t PagesPerChunk = 64; uint64_t physPages[PagesPerChunk]; @@ -151,11 +149,11 @@ namespace Memory { for (uint32_t i = 0; i < count; i++) { uint64_t pageVa = startVa + (base + i) * 0x1000ULL; - physPages[i] = Memory::VMM::Paging::GetPhysAddr(pml4Phys, pageVa); - Memory::VMM::Paging::UnmapUserIn(pml4Phys, pageVa); + physPages[i] = Memory::VMM::Paging::GetPhysAddr(pageTableRoot, pageVa); + Memory::VMM::Paging::UnmapUserIn(pageTableRoot, pageVa); } - ShootdownUserRange(pml4Phys, startVa + base * 0x1000ULL, count); + ShootdownUserRange(pageTableRoot, startVa + base * 0x1000ULL, count); for (uint32_t i = 0; i < count; i++) { if (physPages[i] != 0) { @@ -166,7 +164,7 @@ namespace Memory { } void InitUserRange() { - Hal::RegisterIrqHandler(Hal::IRQ_TLB_SHOOTDOWN, TlbShootdownIpiHandler); + Arch::RegisterIrqHandler(Arch::IRQ_TLB_SHOOTDOWN, TlbShootdownIpiHandler); } } diff --git a/kernel/src/Memory/UserRange.hpp b/kernel/src/Memory/UserRange.hpp index e03216f..d8c7adb 100644 --- a/kernel/src/Memory/UserRange.hpp +++ b/kernel/src/Memory/UserRange.hpp @@ -15,9 +15,9 @@ namespace Memory { // Invalidate a user range on every CPU currently running the address // space. Call this after removing PTEs and before releasing their frames. - void ShootdownUserRange(uint64_t pml4Phys, uint64_t startVa, uint32_t pages); + void ShootdownUserRange(uint64_t pageTableRoot, uint64_t startVa, uint32_t pages); // Safely remove ordinary PFA-backed user mappings and release their frames. - void UnmapAndFreeUserRange(uint64_t pml4Phys, uint64_t startVa, uint64_t pages); + void UnmapAndFreeUserRange(uint64_t pageTableRoot, uint64_t startVa, uint64_t pages); } diff --git a/kernel/src/Net/Arp.cpp b/kernel/src/Net/Arp.cpp index cce221a..456f285 100644 --- a/kernel/src/Net/Arp.cpp +++ b/kernel/src/Net/Arp.cpp @@ -14,7 +14,7 @@ #include #include #include -#include +#include #include #include diff --git a/kernel/src/Net/Dns.cpp b/kernel/src/Net/Dns.cpp index 9d293ce..c5d3644 100644 --- a/kernel/src/Net/Dns.cpp +++ b/kernel/src/Net/Dns.cpp @@ -5,13 +5,14 @@ */ #include "Dns.hpp" +#include #include #include #include #include #include #include -#include +#include #include #include #include @@ -300,7 +301,7 @@ namespace Net::Dns { if (length > sizeof(query.response)) length = sizeof(query.response); memcpy(query.response, data, length); query.responseLen = length; - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); query.gotResponse = true; return; } @@ -432,7 +433,7 @@ namespace Net::Dns { Net::Udp::Unbind(query->localPort); // Parse the response - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); DnsAnswer answer = ParseResponse(txId, query->response, query->responseLen); query->active = false; if (!answer.found) return 0; diff --git a/kernel/src/Net/Ipv4.cpp b/kernel/src/Net/Ipv4.cpp index 5444f97..941daf8 100644 --- a/kernel/src/Net/Ipv4.cpp +++ b/kernel/src/Net/Ipv4.cpp @@ -16,7 +16,7 @@ #include #include #include -#include +#include using namespace Kt; diff --git a/kernel/src/Net/Tcp.cpp b/kernel/src/Net/Tcp.cpp index 7d1bab1..3cfa481 100644 --- a/kernel/src/Net/Tcp.cpp +++ b/kernel/src/Net/Tcp.cpp @@ -5,6 +5,7 @@ */ #include "Tcp.hpp" +#include #include #include #include @@ -13,7 +14,7 @@ #include #include #include -#include +#include #include using namespace Kt; @@ -641,31 +642,31 @@ namespace Net::Tcp { while (true) { uint64_t connWake = Sched::ObserveObjectWake(conn); uint64_t flags; - asm volatile("pushfq; pop %0; cli" : "=r"(flags) :: "memory"); + flags = Arch::SaveAndDisableInterrupts(); conn->Lock.Acquire(); if (conn->CurrentState != State::Established) { conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); return -1; } if (!conn->SendBusy) { conn->SendBusy = true; conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); break; } conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); Sched::BlockOnObjectSince(conn, 50, connWake); } auto finishSend = [&]() { uint64_t flags; - asm volatile("pushfq; pop %0; cli" : "=r"(flags) :: "memory"); + flags = Arch::SaveAndDisableInterrupts(); conn->Lock.Acquire(); conn->SendBusy = false; conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); Sched::WakeObjectWaiters(conn); Ipc::NotifyTcpConnectionChanged(conn); }; @@ -680,12 +681,12 @@ namespace Net::Tcp { uint32_t segSeq = 0; uint32_t expectedAck = 0; uint64_t flags; - asm volatile("pushfq; pop %0; cli" : "=r"(flags) :: "memory"); + flags = Arch::SaveAndDisableInterrupts(); conn->Lock.Acquire(); if (conn->CurrentState != State::Established) { conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); int result = sent > 0 ? (int)sent : -1; finishSend(); return result; @@ -697,7 +698,7 @@ namespace Net::Tcp { bool ok = SendSegment(conn, FLAG_ACK | FLAG_PSH, data + sent, segLen); if (!ok) { conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); int result = sent > 0 ? (int)sent : -1; finishSend(); return result; @@ -713,17 +714,17 @@ namespace Net::Tcp { } conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); while (true) { uint64_t connWake = Sched::ObserveObjectWake(conn); - asm volatile("pushfq; pop %0; cli" : "=r"(flags) :: "memory"); + flags = Arch::SaveAndDisableInterrupts(); conn->Lock.Acquire(); if (SeqLessOrEqual(expectedAck, conn->SendUnack)) { conn->RetransmitLen = 0; conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); sent += segLen; break; } @@ -732,7 +733,7 @@ namespace Net::Tcp { conn->SendNext = conn->SendUnack; conn->RetransmitLen = 0; conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); int result = sent > 0 ? (int)sent : -1; finishSend(); return result; @@ -748,7 +749,7 @@ namespace Net::Tcp { conn->SendNext = conn->SendUnack; conn->RetransmitLen = 0; conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); int result = sent > 0 ? (int)sent : -1; finishSend(); return result; @@ -764,7 +765,7 @@ namespace Net::Tcp { conn->SendNext = conn->SendUnack; conn->RetransmitLen = 0; conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); int result = sent > 0 ? (int)sent : -1; finishSend(); return result; @@ -779,7 +780,7 @@ namespace Net::Tcp { } conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); Sched::BlockOnObjectSince(conn, waitMs, connWake); } } @@ -797,7 +798,7 @@ namespace Net::Tcp { while (true) { uint64_t connWake = Sched::ObserveObjectWake(conn); uint64_t flags; - asm volatile("pushfq; pop %0; cli" : "=r"(flags) :: "memory"); + flags = Arch::SaveAndDisableInterrupts(); conn->Lock.Acquire(); if (conn->RecvCount > 0) { @@ -818,7 +819,7 @@ namespace Net::Tcp { conn->CurrentState == State::Established; conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); if (windowReopened) { SendSegment(conn, FLAG_ACK, nullptr, 0); } @@ -829,12 +830,12 @@ namespace Net::Tcp { conn->CurrentState == State::Closed || conn->CurrentState == State::TimeWait) { conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); return 0; // Connection closed } conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); Sched::BlockOnObjectSince(conn, 0, connWake); } } @@ -847,7 +848,7 @@ namespace Net::Tcp { // Disable interrupts while holding the lock to prevent deadlock // with OnPacketReceived (called from the network interrupt handler) uint64_t flags; - asm volatile("pushfq; pop %0; cli" : "=r"(flags) :: "memory"); + flags = Arch::SaveAndDisableInterrupts(); conn->Lock.Acquire(); @@ -881,7 +882,7 @@ namespace Net::Tcp { } conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); if (windowReopened) { // Window-update ACK so the peer resumes without waiting for a @@ -898,7 +899,7 @@ namespace Net::Tcp { } uint64_t flags; - asm volatile("pushfq; pop %0; cli" : "=r"(flags) :: "memory"); + flags = Arch::SaveAndDisableInterrupts(); conn->Lock.Acquire(); switch (conn->CurrentState) { @@ -907,7 +908,7 @@ namespace Net::Tcp { SendSegment(conn, FLAG_FIN | FLAG_ACK, nullptr, 0); conn->SendNext++; conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); // Wait for close to complete for (int i = 0; i < 100; i++) { @@ -929,7 +930,7 @@ namespace Net::Tcp { SendSegment(conn, FLAG_FIN | FLAG_ACK, nullptr, 0); conn->SendNext++; conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); // Wait for final ACK for (int i = 0; i < 100; i++) { @@ -949,7 +950,7 @@ namespace Net::Tcp { case State::SynSent: { conn->CurrentState = State::Closed; conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); Sched::WakeObjectWaiters(conn); Ipc::NotifyTcpConnectionChanged(conn); ReleaseConnection(conn); @@ -958,7 +959,7 @@ namespace Net::Tcp { default: conn->Lock.Release(); - asm volatile("push %0; popfq" :: "r"(flags) : "memory"); + Arch::RestoreInterrupts(flags); Sched::WakeObjectWaiters(conn); Ipc::NotifyTcpConnectionChanged(conn); ReleaseConnection(conn); diff --git a/kernel/src/Pci/Pci.cpp b/kernel/src/Pci/Pci.cpp index f43be69..86cbb7f 100644 --- a/kernel/src/Pci/Pci.cpp +++ b/kernel/src/Pci/Pci.cpp @@ -5,13 +5,14 @@ */ #include "Pci.hpp" +#include #include #include #include #include #include #include -#include +#include using namespace Kt; @@ -330,6 +331,25 @@ namespace Pci { return 0; } + bool WriteMsiMessage(uint8_t bus, uint8_t device, uint8_t function, uint8_t cap, uint8_t irq) { + Arch::MsiMessage msi; + if (!Arch::ComposeMsi(irq, msi)) { + return false; + } + uint16_t msgCtrl = LegacyRead16(bus, device, function, cap + 2); + + LegacyWrite32(bus, device, function, cap + 4, (uint32_t)msi.address); + if (msgCtrl & (1 << 7)) { + // 64-bit: Upper Address at cap+8, Data at cap+12 + LegacyWrite32(bus, device, function, cap + 8, (uint32_t)(msi.address >> 32)); + LegacyWrite16(bus, device, function, cap + 12, (uint16_t)msi.data); + } else { + // 32-bit: Data at cap+8 + LegacyWrite16(bus, device, function, cap + 8, (uint16_t)msi.data); + } + return true; + } + bool DisableInterruptDelivery(uint8_t bus, uint8_t device, uint8_t function) { uint16_t command = LegacyRead16(bus, device, function, (uint8_t)PCI_REG_COMMAND); diff --git a/kernel/src/Pci/Pci.hpp b/kernel/src/Pci/Pci.hpp index 48cf656..ee853db 100644 --- a/kernel/src/Pci/Pci.hpp +++ b/kernel/src/Pci/Pci.hpp @@ -138,6 +138,12 @@ namespace Pci { // when message-signalled delivery had been enabled and was turned off. bool DisableInterruptDelivery(uint8_t bus, uint8_t device, uint8_t function); + // Program the MSI capability at `cap` with the message for IRQ slot `irq` + // (Arch::ComposeMsi), using the 32- or 64-bit address layout the + // capability declares. Does not enable MSI. Returns false (writing + // nothing) when the platform cannot receive MSIs. + bool WriteMsiMessage(uint8_t bus, uint8_t device, uint8_t function, uint8_t cap, uint8_t irq); + // ========================================================================= // PCI driver matching // ========================================================================= diff --git a/kernel/src/Platform/Util.cpp b/kernel/src/Platform/Util.cpp deleted file mode 100644 index bb741b6..0000000 --- a/kernel/src/Platform/Util.cpp +++ /dev/null @@ -1,22 +0,0 @@ -/* - * Util.cpp - * Platform utility routines - * Copyright (c) Daniel Hammer, Limine Contributors (via Limine C++ example) -*/ - -// Halt and catch fire function. -namespace Hal { - __attribute__((noreturn)) void Halt() - { - for (;;) - { - #if defined(__x86_64__) - asm("hlt"); - #elif defined(__aarch64__) || defined(__riscv) - asm("wfi"); - #elif defined(__loongarch64) - asm("idle 0"); - #endif - } - } -}; \ No newline at end of file diff --git a/kernel/src/Platform/Util.hpp b/kernel/src/Platform/Util.hpp deleted file mode 100644 index 9aa9ca1..0000000 --- a/kernel/src/Platform/Util.hpp +++ /dev/null @@ -1,12 +0,0 @@ -/* - * Util.hpp - * Declarations of platform utility routines - * Copyright (c) Daniel Hammer, Limine Contributors (via Limine C++ example) -*/ - -#pragma once - -// Halt and catch fire function. -namespace Hal { - __attribute__((noreturn)) void Halt(); -}; \ No newline at end of file diff --git a/kernel/src/Sched/ElfLoader.cpp b/kernel/src/Sched/ElfLoader.cpp index f56baed..6faf8e5 100644 --- a/kernel/src/Sched/ElfLoader.cpp +++ b/kernel/src/Sched/ElfLoader.cpp @@ -5,6 +5,7 @@ */ #include "ElfLoader.hpp" +#include #include #include #include @@ -66,11 +67,6 @@ namespace Sched { static constexpr uint64_t DT_PLTREL = 20; static constexpr uint64_t DT_JMPREL = 23; - static constexpr uint32_t R_X86_64_64 = 1; - static constexpr uint32_t R_X86_64_GLOB_DAT = 6; - static constexpr uint32_t R_X86_64_JUMP_SLOT = 7; - static constexpr uint32_t R_X86_64_RELATIVE = 8; - static constexpr uint16_t SHN_UNDEF = 0; static constexpr uint8_t STB_WEAK = 2; @@ -143,8 +139,9 @@ namespace Sched { return false; } - if (hdr->e_machine != EM_X86_64) { - Kt::KernelLogStream(Kt::ERROR, "ELF") << "Not x86_64 (machine=" << (uint64_t)hdr->e_machine << ")"; + if (hdr->e_machine != Arch::ElfMachine) { + Kt::KernelLogStream(Kt::ERROR, "ELF") << "Not " << Arch::ElfMachineName + << " (machine=" << (uint64_t)hdr->e_machine << ")"; return false; } @@ -181,8 +178,8 @@ namespace Sched { return false; } - static bool WriteUser64(uint64_t pml4Phys, uint64_t virtualAddress, uint64_t value) { - uint64_t physBase = Memory::VMM::Paging::GetPhysAddr(pml4Phys, virtualAddress); + static bool WriteUser64(uint64_t pageTableRoot, uint64_t virtualAddress, uint64_t value) { + uint64_t physBase = Memory::VMM::Paging::GetPhysAddr(pageTableRoot, virtualAddress); if (physBase == 0) { Kt::KernelLogStream(Kt::ERROR, "ELF") << "Relocation target not mapped"; return false; @@ -193,7 +190,7 @@ namespace Sched { return true; } - static bool ApplyRelaBlock(uint64_t pml4Phys, uint64_t loadBias, + static bool ApplyRelaBlock(uint64_t pageTableRoot, uint64_t loadBias, const Elf64Rela* relas, uint64_t relaCount, const Elf64Sym* dynsym, uint64_t dynsymCount, const char* dynstr, uint64_t dynstrSize) { @@ -204,17 +201,15 @@ namespace Sched { uint64_t targetVA = loadBias + rela.r_offset; uint64_t value = 0; - switch (type) { - case 0: + switch (Arch::ClassifyRelocation(type)) { + case Arch::RelocationKind::None: continue; - case R_X86_64_RELATIVE: + case Arch::RelocationKind::Relative: value = AddSigned(loadBias, rela.r_addend); break; - case R_X86_64_GLOB_DAT: - case R_X86_64_JUMP_SLOT: - case R_X86_64_64: { + case Arch::RelocationKind::Symbol: { if (symIndex >= dynsymCount) { Kt::KernelLogStream(Kt::ERROR, "ELF") << "Relocation symbol index out of range"; return false; @@ -240,12 +235,12 @@ namespace Sched { break; } - default: + case Arch::RelocationKind::Unsupported: Kt::KernelLogStream(Kt::ERROR, "ELF") << "Unsupported relocation type=" << (uint64_t)type; return false; } - if (!WriteUser64(pml4Phys, targetVA, value)) { + if (!WriteUser64(pageTableRoot, targetVA, value)) { return false; } } @@ -254,7 +249,7 @@ namespace Sched { } static bool ApplyDynamicRelocations(const uint8_t* fileData, uint64_t fileSize, - const Elf64Header* hdr, uint64_t pml4Phys, + const Elf64Header* hdr, uint64_t pageTableRoot, uint64_t loadBias) { const Elf64ProgramHeader* dynamicPhdr = nullptr; @@ -351,7 +346,7 @@ namespace Sched { } auto* relas = (const Elf64Rela*)(fileData + relaOffset); - if (!ApplyRelaBlock(pml4Phys, loadBias, relas, info.relaSize / sizeof(Elf64Rela), + if (!ApplyRelaBlock(pageTableRoot, loadBias, relas, info.relaSize / sizeof(Elf64Rela), dynsym, dynsymCount, dynstr, info.strtabSize)) { return false; } @@ -369,7 +364,7 @@ namespace Sched { } auto* pltRelas = (const Elf64Rela*)(fileData + jmprelOffset); - if (!ApplyRelaBlock(pml4Phys, loadBias, pltRelas, info.pltrelSize / sizeof(Elf64Rela), + if (!ApplyRelaBlock(pageTableRoot, loadBias, pltRelas, info.pltrelSize / sizeof(Elf64Rela), dynsym, dynsymCount, dynstr, info.strtabSize)) { return false; } @@ -405,17 +400,18 @@ namespace Sched { return false; } - if (hdr->e_machine != EM_X86_64) { - Kt::KernelLogStream(Kt::ERROR, "ELF") << "Not x86_64 (machine=" << (uint64_t)hdr->e_machine << ")"; + if (hdr->e_machine != Arch::ElfMachine) { + Kt::KernelLogStream(Kt::ERROR, "ELF") << "Not " << Arch::ElfMachineName + << " (machine=" << (uint64_t)hdr->e_machine << ")"; return false; } return true; } - uint64_t ElfLoad(const char* vfsPath, uint64_t pml4Phys, TlsInfo* outTls) { + uint64_t ElfLoad(const char* vfsPath, uint64_t pageTableRoot, TlsInfo* outTls) { if (outTls != nullptr) { - outTls->fsBase = 0; + outTls->threadPointer = 0; outTls->templateVaddr = 0; outTls->fileSize = 0; outTls->memSize = 0; @@ -503,7 +499,7 @@ namespace Sched { maxVaddrEnd = phdr->p_vaddr + phdr->p_memsz; } - // Allocate pages and map them in the process PML4 with User bit + // Allocate pages and map them in the process address space with User bit uint64_t segBase = phdr->p_vaddr & ~0xFFFULL; uint64_t segEnd = (phdr->p_vaddr + phdr->p_memsz + 0xFFF) & ~0xFFFULL; uint64_t numPages = (segEnd - segBase) / 0x1000; @@ -519,8 +515,8 @@ namespace Sched { uint64_t physAddr = Memory::SubHHDM((uint64_t)page); uint64_t virtAddr = segBase + p * 0x1000; - // Map into the process's PML4 with User bit set - if (!Memory::VMM::Paging::MapUserIn(pml4Phys, physAddr, virtAddr)) { + // Map into the process's address space as user-accessible + if (!Memory::VMM::Paging::MapUserIn(pageTableRoot, physAddr, virtAddr)) { Kt::KernelLogStream(Kt::ERROR, "ELF") << "Failed to map page"; Fs::Vfs::CloseBackendFile(file); return 0; @@ -535,9 +531,9 @@ namespace Sched { } } - // Build the main thread's TLS block (x86-64 variant II): the block - // lives below the thread pointer (see ComputeTlsLayout), template - // data first, .tbss zeroed, and [TP] holds the ABI self-pointer. + // Build the main thread's TLS block, laid out by the architecture's + // TLS variant (Arch::ComputeTlsLayout): template data first, .tbss + // zeroed, and [TP] holds the ABI self-pointer. if (tlsPhdr != nullptr && outTls != nullptr) { TlsLayout layout = ComputeTlsLayout(tlsPhdr->p_memsz, tlsPhdr->p_align); uint64_t base = (maxVaddrEnd + 0x1FFFULL) & ~0xFFFULL; // page-aligned, 1 guard page gap @@ -553,7 +549,7 @@ namespace Sched { } uint64_t physAddr = Memory::SubHHDM((uint64_t)page); uint64_t virtAddr = base + p * 0x1000; - if (!Memory::VMM::Paging::MapUserIn(pml4Phys, physAddr, virtAddr)) { + if (!Memory::VMM::Paging::MapUserIn(pageTableRoot, physAddr, virtAddr)) { ok = false; break; } @@ -572,7 +568,7 @@ namespace Sched { } if (ok) { - outTls->fsBase = tp; + outTls->threadPointer = tp; outTls->templateVaddr = tlsPhdr->p_vaddr; outTls->fileSize = tlsPhdr->p_filesz; outTls->memSize = tlsPhdr->p_memsz; @@ -590,7 +586,7 @@ namespace Sched { return entryPoint; } - uint64_t ElfLoadLib(const char* vfsPath, uint64_t pml4Phys, int slot) { + uint64_t ElfLoadLib(const char* vfsPath, uint64_t pageTableRoot, int slot) { Fs::Vfs::BackendFile file = {-1, -1, 0}; if (Fs::Vfs::OpenBackendFile(vfsPath, file) < 0) { return 0; @@ -614,7 +610,7 @@ namespace Sched { Fs::Vfs::ReadBackendFile(file, fileData, 0, fileSize); Fs::Vfs::CloseBackendFile(file); - asm volatile("" ::: "memory"); + Arch::CompilerBarrier(); Elf64Header* hdr = (Elf64Header*)fileData; @@ -701,7 +697,7 @@ namespace Sched { return 0; } - if (!Memory::VMM::Paging::MapUserIn(pml4Phys, physAddr, pageStart)) { + if (!Memory::VMM::Paging::MapUserIn(pageTableRoot, physAddr, pageStart)) { Kt::KernelLogStream(Kt::ERROR, "ELF") << "Failed to map page"; Memory::g_heap->Free(fileData); return 0; @@ -726,7 +722,7 @@ namespace Sched { } } - if (!ApplyDynamicRelocations(fileData, fileSize, hdr, pml4Phys, loadBias)) { + if (!ApplyDynamicRelocations(fileData, fileSize, hdr, pageTableRoot, loadBias)) { Memory::g_heap->Free(fileData); return 0; } diff --git a/kernel/src/Sched/ElfLoader.hpp b/kernel/src/Sched/ElfLoader.hpp index c2094d3..fe3f7a3 100644 --- a/kernel/src/Sched/ElfLoader.hpp +++ b/kernel/src/Sched/ElfLoader.hpp @@ -6,6 +6,7 @@ #pragma once #include +#include namespace Sched { @@ -42,52 +43,34 @@ namespace Sched { static constexpr uint32_t PT_TLS = 7; static constexpr uint16_t ET_EXEC = 2; static constexpr uint16_t ET_DYN = 3; - static constexpr uint16_t EM_X86_64 = 62; // Fixed base address for shared libraries. static constexpr uint64_t LIB_BASE = 0x60000000ULL; static constexpr uint64_t LIB_MAX_SIZE = 0x200000ULL; // 2MB per library // Load an ELF64 binary into a per-process address space. - // pml4Phys = physical address of the process's PML4. + // pageTableRoot = physical address of the process's root page table. // Returns the entry point address, or 0 on failure. - // Thread-local storage description recovered from PT_TLS. fsBase is - // the main thread's thread pointer (x86-64 variant II: the TLS block - // sits below it, [fsBase] holds a self-pointer). The template fields + // Thread-local storage description recovered from PT_TLS. threadPointer is + // the main thread's TLS thread pointer (laid out by + // Arch::ComputeTlsLayout; [threadPointer] holds a self-pointer). The template fields // let SpawnThread build per-thread copies from the loaded image. struct TlsInfo { - uint64_t fsBase; // 0 when the binary has no PT_TLS + uint64_t threadPointer; // 0 when the binary has no PT_TLS uint64_t templateVaddr; // source of .tdata in the loaded image uint64_t fileSize; // initialized bytes uint64_t memSize; // total bytes (incl. .tbss) uint64_t align; // PT_TLS p_align, NOT rounded up }; - // Geometry of one variant II TLS block, as offsets from its - // page-aligned base. The linker addresses TLS variables at - // TP - align_up(memSize, p_align) + offset, so the template must start - // exactly that far below TP; rounding p_align up first (as an older - // version did) shifts every variable away from its initial value. - // TP itself is kept at least 16-byte aligned, which the gap between - // base and the template absorbs. - struct TlsLayout { - uint64_t templateOffset; // where .tdata/.tbss start - uint64_t tpOffset; // thread pointer; [TP] holds the self-pointer - uint64_t totalSize; // bytes to map, incl. the 16-byte TCB - }; + // Per-thread TLS block geometry; the variant is the architecture's. + using Arch::TlsLayout; + using Arch::ComputeTlsLayout; - inline TlsLayout ComputeTlsLayout(uint64_t memSize, uint64_t pAlign) { - uint64_t align = pAlign ? pAlign : 1; - uint64_t block = (memSize + align - 1) & ~(align - 1); - uint64_t tpAlign = align < 16 ? 16 : align; - uint64_t tp = (block + tpAlign - 1) & ~(tpAlign - 1); - return {tp - block, tp, tp + 16}; - } - - uint64_t ElfLoad(const char* vfsPath, uint64_t pml4Phys, TlsInfo* outTls); + uint64_t ElfLoad(const char* vfsPath, uint64_t pageTableRoot, TlsInfo* outTls); // Load a shared library into a fixed address range. // Returns the relocation base / load bias on success, or 0 on failure. - uint64_t ElfLoadLib(const char* vfsPath, uint64_t pml4Phys, int slot); + uint64_t ElfLoadLib(const char* vfsPath, uint64_t pageTableRoot, int slot); } diff --git a/kernel/src/Sched/Scheduler.cpp b/kernel/src/Sched/Scheduler.cpp index b27f2d3..300e899 100644 --- a/kernel/src/Sched/Scheduler.cpp +++ b/kernel/src/Sched/Scheduler.cpp @@ -18,12 +18,13 @@ #include #include #include -#include -#include -#include -#include -#include -#include +#include +#include +#include +#include +#include +#include +#include #include #include #include @@ -32,32 +33,6 @@ #include #include -// Assembly: context switch with CR3 and FPU state parameters -extern "C" void SchedContextSwitch(uint64_t* oldRsp, uint64_t newRsp, uint64_t newCR3, - uint8_t* oldFpuArea, uint8_t* newFpuArea); - -// Load a user thread's FS base (TLS thread pointer) before dispatching it. -// Cached per CPU: the MSR write is skipped when unchanged, so TLS-free -// processes (fsBase 0) cost one write per CPU ever. Userspace cannot move -// FS base itself (CR4.FSGSBASE is off), so the cache cannot go stale. -static inline void LoadUserFsBase(Smp::CpuData* cpu, uint64_t fsBase) { - if (cpu->currentFsBase == fsBase) { - return; - } - cpu->currentFsBase = fsBase; - asm volatile("wrmsr" - : - : "c"(0xC0000100u), /* IA32_FS_BASE */ - "a"((uint32_t)fsBase), - "d"((uint32_t)(fsBase >> 32))); -} - -// Assembly: jump to user mode via IRETQ. -// `arg` is delivered as the user-mode RDI (SystemV first argument). -// For freshly spawned processes this is 0; for SpawnThread it is the -// user-supplied entry argument. -extern "C" void JumpToUserMode(uint64_t rip, uint64_t rsp, uint64_t arg); - namespace Sched { static Process processTable[MaxProcesses]; @@ -74,7 +49,7 @@ namespace Sched { // The scheduler lock MUST be a Spinlock (interrupt-disabling). // It is held ACROSS context switches to prevent the race where - // another CPU picks up a process whose RSP hasn't been saved yet. + // another CPU picks up a process whose stack pointer hasn't been saved yet. // The resumed process releases it. // schedLock is a baton: the context that acquires it switches stacks and // the context being resumed releases it. A Spinlock's ordinary shared @@ -99,7 +74,7 @@ namespace Sched { } void Release() { - uint64_t flags = 0; // Fresh startup remains IF=0 until JumpToUserMode's IRETQ. + uint64_t flags = 0; // Fresh startup keeps interrupts off until EnterUserMode. auto* cpu = Smp::GetCurrentCpuData(); if (cpu != nullptr && cpu->currentSlot >= 0 && cpu->currentSlot < MaxProcesses) { flags = processFlags[cpu->currentSlot]; @@ -149,9 +124,9 @@ namespace Sched { return (uint32_t)(value >> 32); } - // The idle loop runs in the kernel PML4 - static uint64_t GetKernelCR3() { - return (uint64_t)Memory::VMM::g_paging->PML4; + // The idle loop runs in the kernel address space + static uint64_t GetKernelRoot() { + return Memory::VMM::g_paging->KernelRoot(); } static void RescheduleIpiHandler(uint8_t, bool interruptedUser) { @@ -192,8 +167,7 @@ namespace Sched { if (target->currentSlot >= 0) return false; if (target->reservedForKernelWork) return false; - return Hal::LocalApic::SendFixedIpi(target->lapicId, - Hal::IRQ_VECTOR_BASE + Hal::IRQ_RESCHEDULE); + return Arch::SendIpi(target->cpuIndex, Arch::IRQ_RESCHEDULE); }; if (tryKick(Smp::GetCpuData(0))) { @@ -230,30 +204,28 @@ namespace Sched { processTable[next].runningOnCpu = cpu->cpuIndex; processTable[next].sliceRemaining = TimeSliceMs; - cpu->kernelRsp = processTable[next].kernelStackTop; - cpu->tss->rsp0 = processTable[next].kernelStackTop; - - LoadUserFsBase(cpu, processTable[next].fsBase); - SchedContextSwitch(&processTable[slot].savedRsp, processTable[next].savedRsp, - processTable[next].pml4Phys, + Arch::ActivateThread(cpu, processTable[next].kernelStackTop, + processTable[next].threadPointer); + Arch::SwitchContext(&processTable[slot].savedSp, processTable[next].savedSp, + processTable[next].pageTableRoot, processTable[slot].fpuState, processTable[next].fpuState); schedLock.Release(); return; } cpu->currentSlot = -1; - SchedContextSwitch(&processTable[slot].savedRsp, cpu->idleSavedRsp, - GetKernelCR3(), processTable[slot].fpuState, nullptr); + Arch::SwitchContext(&processTable[slot].savedSp, cpu->idleSavedSp, + GetKernelRoot(), processTable[slot].fpuState, nullptr); schedLock.Release(); } // Startup function for newly spawned processes. - // SchedContextSwitch "returns" here on first schedule. + // Arch::SwitchContext "returns" here on first schedule. // The schedLock is held (acquired by the switching-from CPU's Schedule). static void ProcessStartup() { // Release the schedLock that the switching-from CPU held // without consulting this brand-new slot's (intentionally empty) - // saved interrupt state. JumpToUserMode's IRETQ enables interrupts. + // saved interrupt state. EnterUserMode enables interrupts. schedLock.ReleaseForStartup(); auto* cpu = Smp::GetCurrentCpuData(); @@ -262,23 +234,18 @@ namespace Sched { if (slot >= 0) { Process& proc = processTable[slot]; - // Set up per-CPU kernel RSP for SYSCALL entry - cpu->kernelRsp = proc.kernelStackTop; - - // Set up per-CPU TSS RSP0 for hardware interrupts from ring 3 - cpu->tss->rsp0 = proc.kernelStackTop; - - LoadUserFsBase(cpu, proc.fsBase); + // Kernel entry stack and TLS pointer for this thread + Arch::ActivateThread(cpu, proc.kernelStackTop, proc.threadPointer); // Jump to user mode (never returns). - // For main threads threadArg is 0 (libc _start ignores RDI); + // For main threads threadArg is 0 (libc _start ignores it); // for sibling threads it carries the user-supplied argument. - JumpToUserMode(proc.entryPoint, proc.userStackTop, proc.threadArg); + Arch::EnterUserMode(proc.entryPoint, proc.userStackTop, proc.threadArg); } ExitProcess(); for (;;) { - asm volatile("hlt"); + Arch::Halt(); } } @@ -287,12 +254,12 @@ namespace Sched { processTable[i].pid = i; processTable[i].state = ProcessState::Free; processTable[i].name[0] = '\0'; - processTable[i].savedRsp = 0; + processTable[i].savedSp = 0; processTable[i].stackBase = 0; processTable[i].entryPoint = 0; processTable[i].sliceRemaining = 0; processTable[i].cpuTimeMs = 0; - processTable[i].pml4Phys = 0; + processTable[i].pageTableRoot = 0; processTable[i].kernelStackTop = 0; processTable[i].userStackTop = 0; processTable[i].heapNext = 0; @@ -344,7 +311,7 @@ namespace Sched { exitLedgerCode[i] = 0; } exitLedgerHead = 0; - Hal::RegisterIrqHandler(Hal::IRQ_RESCHEDULE, RescheduleIpiHandler); + Arch::RegisterIrqHandler(Arch::IRQ_RESCHEDULE, RescheduleIpiHandler); Kt::KernelLogStream(Kt::OK, "Sched") << "Initialized (" << MaxProcesses << " process slots, " << (uint64_t)TimeSliceMs << " ms time slice)"; @@ -392,15 +359,15 @@ namespace Sched { processTable[slot].reapReady = false; schedLock.Release(); - // Create per-process PML4 with kernel-half copied - uint64_t pml4Phys = Memory::VMM::Paging::CreateUserPML4(); + // Create the process address space (kernel half shared) + uint64_t pageTableRoot = Memory::VMM::Paging::CreateUserAddressSpace(); // Load ELF into the process's address space TlsInfo tls{}; - uint64_t entry = ElfLoad(vfsPath, pml4Phys, &tls); + uint64_t entry = ElfLoad(vfsPath, pageTableRoot, &tls); if (entry == 0) { - Memory::VMM::Paging::FreeUserHalf(pml4Phys); - Memory::g_pfa->Free((void*)Memory::HHDM(pml4Phys)); + Memory::VMM::Paging::FreeUserHalf(pageTableRoot); + Memory::g_pfa->Free((void*)Memory::HHDM(pageTableRoot)); schedLock.Acquire(); processTable[slot].state = ProcessState::Free; schedLock.Release(); @@ -415,8 +382,8 @@ namespace Sched { void* stackMem = Memory::g_pfa->ReallocConsecutive(nullptr, StackPages); if (stackMem == nullptr) { Kt::KernelLogStream(Kt::ERROR, "Sched") << "Failed to allocate contiguous kernel stack"; - Memory::VMM::Paging::FreeUserHalf(pml4Phys); - Memory::g_pfa->Free((void*)Memory::HHDM(pml4Phys)); + Memory::VMM::Paging::FreeUserHalf(pageTableRoot); + Memory::g_pfa->Free((void*)Memory::HHDM(pageTableRoot)); schedLock.Acquire(); processTable[slot].state = ProcessState::Free; schedLock.Release(); @@ -429,15 +396,15 @@ namespace Sched { // Helper to clean up all resources allocated so far on failure auto cleanupOnFail = [&]() { - Memory::VMM::Paging::FreeUserHalf(pml4Phys); - Memory::g_pfa->Free((void*)Memory::HHDM(pml4Phys)); + Memory::VMM::Paging::FreeUserHalf(pageTableRoot); + Memory::g_pfa->Free((void*)Memory::HHDM(pageTableRoot)); Memory::g_pfa->Free(stackMem, StackPages); schedLock.Acquire(); processTable[slot].state = ProcessState::Free; schedLock.Release(); }; - // Allocate user stack pages and map them in the process PML4 + // Allocate user stack pages and map them in the process address space uint64_t userStackBase = UserStackTop - UserStackSize; uint64_t topStackPagePhys = 0; for (uint64_t i = 0; i < UserStackPages; i++) { @@ -448,7 +415,7 @@ namespace Sched { return -1; } uint64_t physAddr = Memory::SubHHDM((uint64_t)page); - if (!Memory::VMM::Paging::MapUserIn(pml4Phys, physAddr, userStackBase + i * 0x1000)) { + if (!Memory::VMM::Paging::MapUserIn(pageTableRoot, physAddr, userStackBase + i * 0x1000)) { Kt::KernelLogStream(Kt::ERROR, "Sched") << "Failed to map user stack page"; Memory::g_pfa->Free(page); cleanupOnFail(); @@ -466,37 +433,24 @@ namespace Sched { return -1; } uint64_t stubPhys = Memory::SubHHDM((uint64_t)stubPage); - if (!Memory::VMM::Paging::MapUserIn(pml4Phys, stubPhys, ExitStubAddr)) { + if (!Memory::VMM::Paging::MapUserIn(pageTableRoot, stubPhys, ExitStubAddr)) { Kt::KernelLogStream(Kt::ERROR, "Sched") << "Failed to map exit stub"; Memory::g_pfa->Free(stubPage); cleanupOnFail(); return -1; } - // Write: xor edi, edi; xor eax, eax; syscall - uint8_t* stub = (uint8_t*)stubPage; - stub[0] = 0x31; stub[1] = 0xFF; // xor edi, edi (exit code 0) - stub[2] = 0x31; stub[3] = 0xC0; // xor eax, eax (SYS_EXIT = 0) - stub[4] = 0x0F; stub[5] = 0x05; // syscall + // User code that calls SYS_EXIT(0) + Arch::WriteExitStub((uint8_t*)stubPage); } - // Push exit stub address as the return address on the user stack. - { - uint8_t* topPage = (uint8_t*)Memory::HHDM(topStackPagePhys); - *(uint64_t*)(topPage + 0xFF8) = ExitStubAddr; - } + // Returning from the entry function lands in the exit stub. + uint64_t initialUserSp = Arch::SetupUserEntryStack( + (uint8_t*)Memory::HHDM(topStackPagePhys), UserStackTop, ExitStubAddr); - // Set up the initial kernel stack frame so that SchedContextSwitch + // Set up the initial kernel stack frame so that Arch::SwitchContext // "returns" into ProcessStartup - uint64_t* sp = (uint64_t*)kernelStackTop; - - *(--sp) = (uint64_t)ProcessStartup; // return addr - *(--sp) = 0; // rbp - *(--sp) = 0; // rbx - *(--sp) = 0; // r12 - *(--sp) = 0; // r13 - *(--sp) = 0; // r14 - *(--sp) = 0; // r15 + uint64_t initialKernelSp = Arch::PrepareThreadStack(kernelStackTop, ProcessStartup); schedLock.Acquire(); @@ -516,18 +470,18 @@ namespace Sched { proc.joinable = false; // main thread is reaped by BSP, not joined proc.futexAddr = 0; proc.futexWoken = false; - proc.savedRsp = (uint64_t)sp; + proc.savedSp = initialKernelSp; proc.stackBase = (uint64_t)kernelStackBase; proc.entryPoint = entry; proc.sliceRemaining = TimeSliceMs; proc.cpuTimeMs = 0; - proc.pml4Phys = pml4Phys; + proc.pageTableRoot = pageTableRoot; proc.kernelStackTop = kernelStackTop; - proc.userStackTop = UserStackTop - 8; + proc.userStackTop = initialUserSp; proc.heapNext = UserHeapBase; for (uint64_t i = 0; i < UserHeapBitmapWords; i++) g_userHeapPageMap[slot][i] = 0; - proc.fsBase = tls.fsBase; + proc.threadPointer = tls.threadPointer; proc.tlsTemplateVaddr = tls.templateVaddr; proc.tlsFileSize = tls.fileSize; proc.tlsMemSize = tls.memSize; @@ -667,10 +621,7 @@ namespace Sched { proc.ioKeyHandle = -1; proc.ioWaitsetHandle = -1; - // Initialize FPU state: zero out, then set default FCW and MXCSR - memset(proc.fpuState, 0, 512); - *(uint16_t*)&proc.fpuState[0] = 0x037F; // FCW: default x87 control word - *(uint32_t*)&proc.fpuState[24] = 0x1F80; // MXCSR: default SSE control/status + Arch::InitFpuState(proc.fpuState); Ipc::ProcessStartedInSlot(slot, proc.pid); @@ -686,8 +637,8 @@ namespace Sched { // Threading // // Every thread is a slot in processTable. The main thread's primarySlot - // points to itself and owns process-level state (PML4, IPC handles, - // cwd/user/args, redirected I/O). A sibling thread copies pml4Phys from + // points to itself and owns process-level state (page tables, IPC handles, + // cwd/user/args, redirected I/O). A sibling thread copies pageTableRoot from // the primary so context-switch is cheap, but per-process getters use // primarySlot to reach the canonical state. // @@ -706,16 +657,16 @@ namespace Sched { int callerSlot = cpu->currentSlot; int primarySlot_ = processTable[callerSlot].primarySlot; if (primarySlot_ < 0) primarySlot_ = callerSlot; - uint64_t sharedPml4 = processTable[primarySlot_].pml4Phys; + uint64_t sharedRoot = processTable[primarySlot_].pageTableRoot; int processPid = processTable[primarySlot_].pid; - if (sharedPml4 == 0) return -1; + if (sharedRoot == 0) return -1; // Give the new thread its own TLS block when the image has one. // We are executing inside the process's address space, so the // freshly mapped pages and the template are directly addressable. // The block uses process-heap virtual space. Its physical pages are // reclaimed when the sibling exits (or if creation fails). - uint64_t threadFsBase = 0; + uint64_t threadTp = 0; uint64_t threadTlsBase = 0; uint64_t threadTlsPages = 0; Process& primary = processTable[primarySlot_]; @@ -739,7 +690,7 @@ namespace Sched { break; } uint64_t physAddr = Memory::SubHHDM((uint64_t)page); - if (!Memory::VMM::Paging::MapUserIn(sharedPml4, physAddr, + if (!Memory::VMM::Paging::MapUserIn(sharedRoot, physAddr, base + p * 0x1000)) { Memory::g_pfa->Free(page); ok = false; @@ -748,7 +699,7 @@ namespace Sched { mappedPages++; } if (!ok) { - Memory::UnmapAndFreeUserRange(sharedPml4, base, mappedPages); + Memory::UnmapAndFreeUserRange(sharedRoot, base, mappedPages); ReleaseUserHeapRange(primarySlot_, base, numPages * 0x1000ULL); Kt::KernelLogStream(Kt::ERROR, "Sched") << "Thread TLS allocation failed"; @@ -759,7 +710,7 @@ namespace Sched { memcpy((void*)(base + layout.templateOffset), (const void*)primary.tlsTemplateVaddr, primary.tlsFileSize); *(uint64_t*)tp = tp; /* ABI self-pointer */ - threadFsBase = tp; + threadTp = tp; threadTlsBase = base; threadTlsPages = numPages; } @@ -768,14 +719,14 @@ namespace Sched { // require validating the user VA against the process page tables). // Userspace is responsible for ensuring the thread entry calls // SYS_THREAD_EXIT; the libc trampoline does this. The kernel only - // enforces SystemV alignment by handing the entry RSP = 16n - 8. - uint64_t userRsp = (userStackTop & ~0xFULL) - 8; + // enforces the ABI's entry stack alignment. + uint64_t userSp = Arch::ThreadEntryStackPointer(userStackTop); // Allocate kernel stack. void* stackMem = Memory::g_pfa->ReallocConsecutive(nullptr, StackPages); if (stackMem == nullptr) { if (threadTlsPages != 0) { - Memory::UnmapAndFreeUserRange(sharedPml4, threadTlsBase, threadTlsPages); + Memory::UnmapAndFreeUserRange(sharedRoot, threadTlsBase, threadTlsPages); ReleaseUserHeapRange(primarySlot_, threadTlsBase, threadTlsPages * 0x1000ULL); } @@ -785,11 +736,8 @@ namespace Sched { uint8_t* kernelStackBase = (uint8_t*)stackMem; uint64_t kernelStackTop = (uint64_t)kernelStackBase + StackSize; - // Initial kernel frame -- SchedContextSwitch "returns" into ProcessStartup. - uint64_t* sp = (uint64_t*)kernelStackTop; - *(--sp) = (uint64_t)ProcessStartup; - *(--sp) = 0; *(--sp) = 0; *(--sp) = 0; - *(--sp) = 0; *(--sp) = 0; *(--sp) = 0; + // Initial kernel frame -- Arch::SwitchContext "returns" into ProcessStartup. + uint64_t initialKernelSp = Arch::PrepareThreadStack(kernelStackTop, ProcessStartup); schedLock.Acquire(); @@ -801,7 +749,7 @@ namespace Sched { schedLock.Release(); Memory::g_pfa->Free(stackMem, StackPages); if (threadTlsPages != 0) { - Memory::UnmapAndFreeUserRange(sharedPml4, threadTlsBase, threadTlsPages); + Memory::UnmapAndFreeUserRange(sharedRoot, threadTlsBase, threadTlsPages); ReleaseUserHeapRange(primarySlot_, threadTlsBase, threadTlsPages * 0x1000ULL); } @@ -826,12 +774,12 @@ namespace Sched { thr.threadArg = arg; thr.futexAddr = 0; thr.futexWoken = false; - thr.savedRsp = (uint64_t)sp; + thr.savedSp = initialKernelSp; thr.stackBase = (uint64_t)kernelStackBase; thr.entryPoint = entry; - thr.userStackTop = userRsp; + thr.userStackTop = userSp; thr.kernelStackTop = kernelStackTop; - thr.pml4Phys = sharedPml4; // shared with primary, NOT owned + thr.pageTableRoot = sharedRoot; // shared with primary, NOT owned thr.sliceRemaining = TimeSliceMs; thr.cpuTimeMs = 0; thr.heapNext = 0; @@ -846,7 +794,7 @@ namespace Sched { thr.ioOutHandle = -1; thr.ioInHandle = -1; thr.ioKeyHandle = -1; thr.ioWaitsetHandle = -1; thr.args[0] = '\0'; thr.user[0] = '\0'; thr.cwd[0] = '\0'; - thr.fsBase = threadFsBase; + thr.threadPointer = threadTp; thr.tlsTemplateVaddr = 0; thr.tlsFileSize = 0; thr.tlsMemSize = 0; @@ -862,9 +810,7 @@ namespace Sched { thr.name[i] = '\0'; } - memset(thr.fpuState, 0, 512); - *(uint16_t*)&thr.fpuState[0] = 0x037F; - *(uint32_t*)&thr.fpuState[24] = 0x1F80; + Arch::InitFpuState(thr.fpuState); int tid = thr.pid; schedLock.Release(); @@ -933,25 +879,25 @@ namespace Sched { static void FreeSiblingThreadTls(Process& thr) { int primarySlot_ = thr.primarySlot; if (primarySlot_ < 0 || primarySlot_ >= MaxProcesses || - primarySlot_ == (int)(&thr - processTable) || thr.fsBase == 0) { + primarySlot_ == (int)(&thr - processTable) || thr.threadPointer == 0) { return; } Process& primary = processTable[primarySlot_]; - if (primary.tlsMemSize == 0 || primary.pml4Phys == 0) { - thr.fsBase = 0; + if (primary.tlsMemSize == 0 || primary.pageTableRoot == 0) { + thr.threadPointer = 0; return; } TlsLayout layout = ComputeTlsLayout(primary.tlsMemSize, primary.tlsAlign); - if (layout.tpOffset > thr.fsBase || layout.totalSize < layout.tpOffset) { + if (layout.tpOffset > thr.threadPointer || layout.totalSize < layout.tpOffset) { Panic("Invalid sibling TLS metadata during thread teardown", nullptr); } - uint64_t base = thr.fsBase - layout.tpOffset; + uint64_t base = thr.threadPointer - layout.tpOffset; uint64_t pages = (layout.totalSize + 0xFFF) / 0x1000; - thr.fsBase = 0; - Memory::UnmapAndFreeUserRange(primary.pml4Phys, base, pages); + thr.threadPointer = 0; + Memory::UnmapAndFreeUserRange(primary.pageTableRoot, base, pages); ReleaseUserHeapRange(primarySlot_, base, pages * 0x1000ULL); } @@ -971,19 +917,18 @@ namespace Sched { readyCount--; processTable[next].runningOnCpu = cpu->cpuIndex; processTable[next].sliceRemaining = TimeSliceMs; - cpu->kernelRsp = processTable[next].kernelStackTop; - cpu->tss->rsp0 = processTable[next].kernelStackTop; if (readyCount > 0) { KickOneIdleCpu(cpu->cpuIndex); } - LoadUserFsBase(cpu, processTable[next].fsBase); - SchedContextSwitch(&thr.savedRsp, processTable[next].savedRsp, - processTable[next].pml4Phys, + Arch::ActivateThread(cpu, processTable[next].kernelStackTop, + processTable[next].threadPointer); + Arch::SwitchContext(&thr.savedSp, processTable[next].savedSp, + processTable[next].pageTableRoot, thr.fpuState, processTable[next].fpuState); } else { cpu->currentSlot = -1; - SchedContextSwitch(&thr.savedRsp, cpu->idleSavedRsp, - GetKernelCR3(), + Arch::SwitchContext(&thr.savedSp, cpu->idleSavedSp, + GetKernelRoot(), thr.fpuState, nullptr); } schedLock.Release(); @@ -994,7 +939,7 @@ namespace Sched { auto* cpu = Smp::GetCurrentCpuData(); int slot = cpu->currentSlot; if (slot < 0) { - for (;;) asm volatile("hlt"); + for (;;) Arch::Halt(); } Process& thr = processTable[slot]; @@ -1042,7 +987,7 @@ namespace Sched { SwitchAwayFromExitingThreadLocked(slot, thr); // Unreachable: the slot is Terminated, nobody resumes us. - for (;;) asm volatile("hlt"); + for (;;) Arch::Halt(); } int JoinThread(int tid, int* outExitCode) { @@ -1075,7 +1020,7 @@ namespace Sched { int code = processTable[target].exitCode; void* stackBase = (void*)processTable[target].stackBase; processTable[target].stackBase = 0; - processTable[target].pml4Phys = 0; + processTable[target].pageTableRoot = 0; processTable[target].joinerSlot = -1; processTable[target].primarySlot = -1; processTable[target].state = ProcessState::Free; @@ -1099,7 +1044,7 @@ namespace Sched { int code = processTable[target].exitCode; void* stackBase = (void*)processTable[target].stackBase; processTable[target].stackBase = 0; - processTable[target].pml4Phys = 0; + processTable[target].pageTableRoot = 0; processTable[target].joinerSlot = -1; processTable[target].primarySlot = -1; processTable[target].state = ProcessState::Free; @@ -1140,9 +1085,9 @@ namespace Sched { // Schedule -- core context switch logic // // The schedLock is held ACROSS the context switch. This is critical: - // setting the old process to Ready and saving its RSP must be atomic + // setting the old process to Ready and saving its stack pointer must be atomic // with respect to other CPUs. If we released the lock before saving - // RSP, another CPU could pick up the process with a stale savedRsp. + // the stack pointer, another CPU could pick up the process with a stale savedSp. // // The RESUMED process releases the lock (it was acquired by whatever // CPU called Schedule() and switched away from that process). @@ -1166,10 +1111,10 @@ namespace Sched { // Grab the pointers, mark Free, then free memory after releasing lock void* stackBase = (processTable[i].stackBase != 0) ? (void*)processTable[i].stackBase : nullptr; - void* pml4 = (processTable[i].pml4Phys != 0) - ? (void*)Memory::HHDM(processTable[i].pml4Phys) : nullptr; + void* pageRoot = (processTable[i].pageTableRoot != 0) + ? (void*)Memory::HHDM(processTable[i].pageTableRoot) : nullptr; processTable[i].stackBase = 0; - processTable[i].pml4Phys = 0; + processTable[i].pageTableRoot = 0; processTable[i].reapReady = false; processTable[i].startPending = false; processTable[i].state = ProcessState::Free; @@ -1177,7 +1122,7 @@ namespace Sched { // Release lock during PFA::Free to minimize hold time schedLock.Release(); if (stackBase) Memory::g_pfa->Free(stackBase, StackPages); - if (pml4) Memory::g_pfa->Free(pml4); + if (pageRoot) Memory::g_pfa->Free(pageRoot); schedLock.Acquire(); } } @@ -1233,7 +1178,7 @@ namespace Sched { // Thermal governor step (rate-limited internally to 500 ms). // The timer requests this idle-context pass even under full load. - Hal::CpuPower::ThermalTick(Timekeeping::GetMilliseconds()); + Arch::CpuPower::ThermalTick(Timekeeping::GetMilliseconds()); } uint64_t GetNextDeadlineTick() { @@ -1304,10 +1249,10 @@ namespace Sched { // a process from idle will release it via the resumed // process path. But idle is special: we release here // because idle doesn't go through ProcessStartup. - // The RSP save is safe because interrupts are disabled + // The SP save is safe because interrupts are disabled // (Spinlock), so no timer can fire between Ready and save. - SchedContextSwitch(&processTable[oldSlot].savedRsp, cpu->idleSavedRsp, - GetKernelCR3(), processTable[oldSlot].fpuState, nullptr); + Arch::SwitchContext(&processTable[oldSlot].savedSp, cpu->idleSavedSp, + GetKernelRoot(), processTable[oldSlot].fpuState, nullptr); // Resumed from idle -- lock was held by whoever switched to us schedLock.Release(); } else { @@ -1324,16 +1269,16 @@ namespace Sched { } // Prepare the context switch - uint64_t* oldRspPtr; + uint64_t* oldSpPtr; int oldSlot = cpu->currentSlot; if (oldSlot >= 0) { processTable[oldSlot].state = ProcessState::Ready; readyCount++; processTable[oldSlot].runningOnCpu = -1; - oldRspPtr = &processTable[oldSlot].savedRsp; + oldSpPtr = &processTable[oldSlot].savedSp; } else { - oldRspPtr = &cpu->idleSavedRsp; + oldSpPtr = &cpu->idleSavedSp; } cpu->currentSlot = next; @@ -1342,11 +1287,7 @@ namespace Sched { processTable[next].runningOnCpu = cpu->cpuIndex; processTable[next].sliceRemaining = TimeSliceMs; - uint64_t newCR3 = processTable[next].pml4Phys; - - // Update per-CPU kernel RSP and TSS RSP0 - cpu->kernelRsp = processTable[next].kernelStackTop; - cpu->tss->rsp0 = processTable[next].kernelStackTop; + uint64_t newRoot = processTable[next].pageTableRoot; uint8_t* oldFpu = (oldSlot >= 0) ? processTable[oldSlot].fpuState : nullptr; uint8_t* newFpu = processTable[next].fpuState; @@ -1356,16 +1297,18 @@ namespace Sched { // dispatching user work so preemption resumes with the process. // Also pick up a thermal-governor policy epoch that may have // changed while this CPU remained asleep without timer ticks. - Hal::CpuPower::ApplyPolicyIfChanged(); - Timekeeping::ApicTimerLeaveApIdle(); + Arch::CpuPower::ApplyPolicyIfChanged(); + Timekeeping::TimerLeaveApIdle(); } - LoadUserFsBase(cpu, processTable[next].fsBase); + // Kernel entry stack and TLS pointer for the incoming thread + Arch::ActivateThread(cpu, processTable[next].kernelStackTop, + processTable[next].threadPointer); // DO NOT release schedLock here! It is held across the context - // switch so that setting Ready + saving RSP is atomic. The + // switch so that setting Ready + saving the SP is atomic. The // resumed process releases it. - SchedContextSwitch(oldRspPtr, processTable[next].savedRsp, newCR3, oldFpu, newFpu); + Arch::SwitchContext(oldSpPtr, processTable[next].savedSp, newRoot, oldFpu, newFpu); // We reach here when this process is resumed by another CPU's // Schedule(). That CPU held the lock across its context switch. @@ -1379,7 +1322,7 @@ namespace Sched { // Pick up thermal-governor frequency changes (HWP requests are // per-core MSRs, so every CPU applies them itself). No-op unless // the governor bumped the policy epoch since our last tick. - Hal::CpuPower::ApplyPolicyIfChanged(); + Arch::CpuPower::ApplyPolicyIfChanged(); // BSP: expire timed waits in the bounded IRQ-safe pass. Queue heavier // reclamation/policy work for the idle kernel context. @@ -1514,7 +1457,7 @@ namespace Sched { stackGrowLock.Acquire(); // Another thread may have mapped this page between the fault and here. - if (Memory::VMM::Paging::IsUserRangeAccessible(proc->pml4Phys, pageVa, 0x1000, true)) { + if (Memory::VMM::Paging::IsUserRangeAccessible(proc->pageTableRoot, pageVa, 0x1000, true)) { stackGrowLock.Release(); return true; } @@ -1527,14 +1470,14 @@ namespace Sched { return false; } uint64_t physAddr = Memory::SubHHDM((uint64_t)page); - if (!Memory::VMM::Paging::MapUserIn(proc->pml4Phys, physAddr, pageVa)) { + if (!Memory::VMM::Paging::MapUserIn(proc->pageTableRoot, physAddr, pageVa)) { stackGrowLock.Release(); Memory::g_pfa->Free(page); return false; } stackGrowLock.Release(); - asm volatile("invlpg (%0)" :: "r"(pageVa) : "memory"); + Arch::FlushTlbPage(pageVa); return true; } @@ -1562,7 +1505,7 @@ namespace Sched { int exitingPid = proc.pid; int primarySlot_ = slot; - // Drain sibling threads: any thread sharing our PML4 must stop using + // Drain sibling threads: any thread sharing our address space must stop using // it before we call FreeUserHalf. Ready/Blocked siblings can be // terminated immediately; siblings running on other CPUs are tagged // killPending and we spin until their tick handler routes them @@ -1626,8 +1569,7 @@ namespace Sched { if (!targetCpus[i] || i == cpu->cpuIndex) continue; Smp::CpuData* target = Smp::GetCpuData(i); if (target != nullptr && target->started) { - (void)Hal::LocalApic::SendFixedIpi(target->lapicId, - Hal::IRQ_VECTOR_BASE + Hal::IRQ_RESCHEDULE); + (void)Arch::SendIpi(target->cpuIndex, Arch::IRQ_RESCHEDULE); } } } @@ -1644,7 +1586,7 @@ namespace Sched { // Wait for the sibling CPU(s) to observe killPending on their // next timer tick. Briefly busy-wait; this path is rare. for (int spin = 0; spin < 1024; spin++) { - asm volatile("pause"); + Arch::Pause(); } } @@ -1656,11 +1598,11 @@ namespace Sched { if (processTable[i].state != ProcessState::Terminated) continue; // Ready/blocked siblings terminated inline above never ran their // ordinary ExitCurrentThread cleanup. Running siblings normally - // arrive here with fsBase already cleared, making this a no-op. + // arrive here with threadPointer already cleared, making this a no-op. FreeSiblingThreadTls(processTable[i]); void* stackBase = (void*)processTable[i].stackBase; processTable[i].stackBase = 0; - processTable[i].pml4Phys = 0; + processTable[i].pageTableRoot = 0; processTable[i].primarySlot = -1; processTable[i].joinerSlot = -1; processTable[i].state = ProcessState::Free; @@ -1689,8 +1631,8 @@ namespace Sched { Drivers::USB::UserUsb::ReleaseAllForPid(exitingPid); // Release process-scoped IPC handles/mappings before tearing down the address space. - Ipc::CleanupProcessSlot(slot, exitingPid, proc.pml4Phys); - montauk::abi::CleanupHeapForSlot(slot, proc.pml4Phys); + Ipc::CleanupProcessSlot(slot, exitingPid, proc.pageTableRoot); + montauk::abi::CleanupHeapForSlot(slot, proc.pageTableRoot); montauk::abi::CleanupLibTable(slot); proc.waitingForPid = -1; @@ -1718,7 +1660,7 @@ namespace Sched { Ipc::ProcessExitedInSlot(slot, exitingPid); // Free all user-space physical pages and page table structures - Memory::VMM::Paging::FreeUserHalf(proc.pml4Phys); + Memory::VMM::Paging::FreeUserHalf(proc.pageTableRoot); schedLock.Acquire(); @@ -1765,26 +1707,25 @@ namespace Sched { KickOneIdleCpu(cpu->cpuIndex); } - uint64_t newCR3 = processTable[next].pml4Phys; - cpu->kernelRsp = processTable[next].kernelStackTop; - cpu->tss->rsp0 = processTable[next].kernelStackTop; - LoadUserFsBase(cpu, processTable[next].fsBase); + uint64_t newRoot = processTable[next].pageTableRoot; + Arch::ActivateThread(cpu, processTable[next].kernelStackTop, + processTable[next].threadPointer); // Lock held across context switch -- resumed process releases it - SchedContextSwitch(&processTable[slot].savedRsp, processTable[next].savedRsp, newCR3, + Arch::SwitchContext(&processTable[slot].savedSp, processTable[next].savedSp, newRoot, processTable[slot].fpuState, processTable[next].fpuState); schedLock.Release(); } else { cpu->currentSlot = -1; // Switch to idle -- release after resuming from idle - SchedContextSwitch(&processTable[slot].savedRsp, cpu->idleSavedRsp, GetKernelCR3(), + Arch::SwitchContext(&processTable[slot].savedSp, cpu->idleSavedSp, GetKernelRoot(), processTable[slot].fpuState, nullptr); schedLock.Release(); } for (;;) { - asm volatile("hlt"); + Arch::Halt(); } } @@ -2150,7 +2091,7 @@ namespace Sched { // it must not fault: the caller validated (and demand-committed) the // page, and the page walk below re-confirms it is still mapped. An // aligned 32-bit word never straddles a page. - if (!Memory::VMM::Paging::IsUserRangeAccessible(self.pml4Phys, uaddr, 4, false)) { + if (!Memory::VMM::Paging::IsUserRangeAccessible(self.pageTableRoot, uaddr, 4, false)) { schedLock.Release(); return BadPointer; } diff --git a/kernel/src/Sched/Scheduler.hpp b/kernel/src/Sched/Scheduler.hpp index a19f827..ed78fd2 100644 --- a/kernel/src/Sched/Scheduler.hpp +++ b/kernel/src/Sched/Scheduler.hpp @@ -9,11 +9,12 @@ #include #include #include +#include namespace Sched { // Process and thread slots share the same table: the main thread of a - // process owns the per-process resources (PML4, IPC handles, args/cwd/user, + // process owns the per-process resources (page tables, IPC handles, args/cwd/user, // redirected I/O), while sibling threads occupy additional slots whose // `primarySlot` points back to the main thread. The scheduler treats // every slot uniformly -- the primary/sibling distinction matters only @@ -78,13 +79,13 @@ namespace Sched { uint64_t sleepUntilTick; // Tick deadline for sleep/object wait timeout (0 = none) void* waitingOnObject; // IPC/scheduler object this process is blocked on (nullptr if none) char name[64]; - uint64_t savedRsp; + uint64_t savedSp; // Kernel SP saved by Arch::SwitchContext uint64_t stackBase; // Bottom of allocated kernel stack (lowest address) uint64_t entryPoint; uint64_t sliceRemaining; // Milliseconds left in current time slice uint64_t cpuTimeMs; // Accumulated scheduler runtime in milliseconds - uint64_t pml4Phys; // Physical address of per-process PML4 - uint64_t kernelStackTop; // Top of kernel stack (for TSS RSP0 / SYSCALL) + uint64_t pageTableRoot; // Physical address of the process's root page table + uint64_t kernelStackTop; // Top of kernel stack (kernel entry from user mode) uint64_t userStackTop; // User-space stack top uint64_t heapNext; // High-water mark of the user-heap address space uint32_t readdirCursor; // Next SYS_READDIR scratch slot @@ -102,11 +103,12 @@ namespace Sched { uint64_t consentDenied; // caps refused for the rest of this process's life char imagePath[256]; // full resolved image path; name[] truncates - // Thread-local storage. fsBase is loaded into IA32_FS_BASE when - // this thread is dispatched (0 = no TLS). The tls* template fields + // Thread-local storage. threadPointer is the TLS thread pointer + // installed by Arch::ActivateThread when this thread is dispatched + // (0 = no TLS). The tls* template fields // live on the primary slot and describe PT_TLS so sibling threads // can get their own block copies. - uint64_t fsBase = 0; + uint64_t threadPointer = 0; uint64_t tlsTemplateVaddr = 0; uint64_t tlsFileSize = 0; uint64_t tlsMemSize = 0; @@ -119,7 +121,7 @@ namespace Sched { // Threading. Every slot is a schedulable thread. The "main thread" of a // process has primarySlot == own slot and owns per-process resources - // (pml4Phys, IPC handles, libs, heap, redirected I/O, cwd/user/args). + // (pageTableRoot, IPC handles, libs, heap, redirected I/O, cwd/user/args). // Sibling threads share those resources via primarySlot. int primarySlot = -1; // Index into processTable; -1 if Free int joinerSlot = -1; // Slot blocked in JoinThread on this slot @@ -162,8 +164,8 @@ namespace Sched { int ioKeyHandle = -1; int ioWaitsetHandle = -1; - // FPU/SSE state (FXSAVE format, must be 16-byte aligned) - uint8_t fpuState[512] __attribute__((aligned(16))); + // FPU/SIMD state, in the architecture's save format + alignas(Arch::FpuStateAlign) uint8_t fpuState[Arch::FpuStateSize]; }; void Initialize(); @@ -220,7 +222,7 @@ namespace Sched { // ==================================================================== // Spawn a sibling thread inside the current process. - // `entry` is a user-mode RIP, `arg` is delivered to entry in %rdi, and + // `entry` is a user-mode PC, `arg` is delivered as its first argument, and // `userStackTop` points to the top-of-stack the caller (libc) allocated // out of the user heap. Returns the new TID, or -1 on failure. int SpawnThread(uint64_t entry, uint64_t arg, uint64_t userStackTop); diff --git a/kernel/src/Timekeeping/ApicTimer.hpp b/kernel/src/Timekeeping/ApicTimer.hpp deleted file mode 100644 index 6e3c98d..0000000 --- a/kernel/src/Timekeeping/ApicTimer.hpp +++ /dev/null @@ -1,54 +0,0 @@ -/* - * ApicTimer.hpp - * Local APIC timer for periodic tick interrupts and timekeeping - * Copyright (c) 2025 Daniel Hammer -*/ - -#pragma once -#include - -namespace Timekeeping { - // Initialize the APIC timer: calibrate against PIT, start periodic interrupts - void ApicTimerInitialize(); - - // Initialize the scheduler timer on an AP using the BSP calibration. - void ApicTimerInitializeAP(); - - // Idle APs are woken for runnable work by the reschedule IPI, so their - // periodic scheduler timer can remain masked until a process is dispatched. - void ApicTimerEnterApIdle(); - void ApicTimerLeaveApIdle(); - - // Reinitialize the APIC timer after S3 resume using the previously - // calibrated tick rate. Skips PIT calibration and IRQ registration - // (both survive in RAM). Only reprograms the timer hardware registers. - void ApicTimerReinitialize(); - - // Get the monotonic tick count (increments on each timer interrupt) - uint64_t GetTicks(); - - // Get elapsed milliseconds since timer initialization - uint64_t GetMilliseconds(); - - // Enable scheduler tick (called after scheduler is initialized) - void EnableSchedulerTick(); - - // Run bounded process-context bottom halves from an idle CPU. - void ServiceDeferredWork(); - - // Enter one idle wait cycle for the current CPU. Uses HLT and relies on - // the boot-time periodic LAPIC timer to wake -- one-shot wake doesn't - // survive deep C-states on some laptop CPUs. - void IdleOnce(bool hasMwait, volatile uint64_t* monitorAddr = nullptr); - - // Busy-wait sleep for the given number of milliseconds - void Sleep(uint64_t ms); - - // Spin for at least `ms` milliseconds without descheduling. Usable from - // boot code, before the scheduler runs, and with interrupts disabled - // (falls back to timed I/O port writes when the tick is not advancing). - void BusyWaitMs(uint64_t ms); - - // Spin for at least `us` microseconds. Port-0x80 timed, so it errs long. - void BusyWaitUs(uint64_t us); -}; diff --git a/kernel/src/Timekeeping/Time.cpp b/kernel/src/Timekeeping/Time.cpp index 8ad8e9d..bb823b6 100644 --- a/kernel/src/Timekeeping/Time.cpp +++ b/kernel/src/Timekeeping/Time.cpp @@ -4,7 +4,7 @@ */ #include "Time.hpp" -#include "ApicTimer.hpp" +#include "Timer.hpp" #include #include diff --git a/kernel/src/Timekeeping/ApicTimer.cpp b/kernel/src/Timekeeping/Timer.cpp similarity index 57% rename from kernel/src/Timekeeping/ApicTimer.cpp rename to kernel/src/Timekeeping/Timer.cpp index f724cfc..f7fc208 100644 --- a/kernel/src/Timekeeping/ApicTimer.cpp +++ b/kernel/src/Timekeeping/Timer.cpp @@ -1,18 +1,16 @@ /* - * ApicTimer.cpp - * Local APIC timer: PIT-calibrated periodic tick for timekeeping + * Timer.cpp + * Periodic tick, timekeeping and idle-loop servicing * Copyright (c) 2025 Daniel Hammer */ -#include "ApicTimer.hpp" +#include "Timer.hpp" #include -#include -#include -#include -#include -#include -#include -#include +#include +#include +#include +#include +#include #include #include #include @@ -27,19 +25,6 @@ using namespace Kt; namespace Timekeeping { - // PIT constants - static constexpr uint32_t PIT_FREQUENCY = 1193182; // PIT oscillator frequency in Hz - static constexpr uint16_t PIT_CHANNEL2_DATA = 0x42; - static constexpr uint16_t PIT_COMMAND = 0x43; - static constexpr uint16_t PIT_GATE_PORT = 0x61; - - // APIC timer LVT mode bits - static constexpr uint32_t LVT_MASKED = (1 << 16); - static constexpr uint32_t LVT_PERIODIC = (1 << 17); - - // APIC timer divide configuration values - static constexpr uint32_t DIVIDE_BY_16 = 0x03; - // The BSP keeps a 1 ms tick for timekeeping and sleep deadlines. // Running APs use a 10 ms scheduler tick. Idle APs mask it entirely and // rely on reschedule IPIs, avoiding periodic package wakeups. @@ -49,30 +34,12 @@ namespace Timekeeping { // Global state static std::atomic g_tickCount{0}; - static uint32_t g_ticksPerMs = 0; + + // Set once Arch::Timer is calibrated and the BSP tick is running. + static bool g_timerReady = false; static std::atomic g_schedEnabled{false}; - static uint32_t CountForIntervalMs(uint32_t intervalMs) { - return g_ticksPerMs * intervalMs; - } - - static void ProgramTimer(bool periodic, uint32_t intervalMs) { - uint32_t lvt = (Hal::IRQ_VECTOR_BASE + Hal::IRQ_TIMER); - if (periodic) { - lvt |= LVT_PERIODIC; - } - - Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_DIVIDE, DIVIDE_BY_16); - Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_LVT, lvt); - Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_INITIAL, - CountForIntervalMs(intervalMs)); - } - - static void ProgramBspPeriodicTimer() { - ProgramTimer(true, BSP_TICK_INTERVAL_MS); - } - // Timer IRQ handler: BSP runs timekeeping + scheduler accounting; APs // run scheduler accounting only. static void TimerHandler(uint8_t, bool interruptedUser) { @@ -112,94 +79,31 @@ namespace Timekeeping { } } - // Use PIT channel 2 to create a precise delay for calibration. - // Returns the number of APIC timer ticks that elapsed during ~10ms. - static uint32_t CalibratePit() { - // We want to measure ~10ms worth of APIC timer ticks. - // PIT count for 10ms: PIT_FREQUENCY / 100 = 11931 - uint16_t pitCount = PIT_FREQUENCY / 100; + void TimerInitialize() { + // Register before the hardware starts so the first tick is counted. + Arch::RegisterIrqHandler(Arch::IRQ_TIMER, TimerHandler); - // Set up PIT channel 2 in one-shot mode (mode 0), lobyte/hibyte - // Command: channel 2 (bits 7-6 = 10), lobyte/hibyte (bits 5-4 = 11), - // mode 0 (bits 3-1 = 000), binary (bit 0 = 0) => 0xB0 - Io::Out8(0xB0, PIT_COMMAND); - - // Disable PIT channel 2 gate to prepare - uint8_t gate = Io::In8(PIT_GATE_PORT); - gate &= ~0x01; // Disable gate (bit 0 = 0) - gate &= ~0x02; // Disable speaker (bit 1 = 0) - Io::Out8(gate, PIT_GATE_PORT); - - // Load the count value (low byte first, then high byte) - Io::Out8((uint8_t)(pitCount & 0xFF), PIT_CHANNEL2_DATA); - Io::IoPortWait(); - Io::Out8((uint8_t)(pitCount >> 8), PIT_CHANNEL2_DATA); - - // Set APIC timer to max count with divide-by-16, one-shot (masked so no IRQ) - Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_DIVIDE, DIVIDE_BY_16); - Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_LVT, LVT_MASKED); - Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_INITIAL, 0xFFFFFFFF); - - // Enable PIT channel 2 gate to start counting - gate = Io::In8(PIT_GATE_PORT); - gate |= 0x01; // Enable gate (bit 0 = 1) - Io::Out8(gate, PIT_GATE_PORT); - - // Wait for PIT channel 2 output to go high (bit 5 of port 0x61) - while (!(Io::In8(PIT_GATE_PORT) & 0x20)) { - asm volatile("pause"); + if (!Arch::Timer::InitializeBsp(BSP_TICK_INTERVAL_MS)) { + KernelLogStream(ERROR, "Timer") << "No usable tick timer"; + return; } + g_timerReady = true; - // Read how many APIC timer ticks elapsed - uint32_t currentCount = Hal::LocalApic::ReadRegister(Hal::LocalApic::REG_TIMER_CURRENT); - uint32_t elapsed = 0xFFFFFFFF - currentCount; - - // Stop the APIC timer - Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_INITIAL, 0); - - return elapsed; + KernelLogStream(OK, "Timer") << "Tick started: BSP " << base::dec + << (uint64_t)BSP_TIMER_HZ << " Hz periodic"; } - void ApicTimerInitialize() { - KernelLogStream(INFO, "Timer") << "Calibrating APIC timer using PIT channel 2"; - - // Calibrate: measure APIC ticks over ~10ms - uint32_t ticksIn10ms = CalibratePit(); - g_ticksPerMs = ticksIn10ms / 10; - - if (g_ticksPerMs == 0) { - KernelLogStream(ERROR, "Timer") << "APIC timer calibration failed (0 ticks/ms)"; + void TimerReinitialize() { + if (!g_timerReady) { + KernelLogStream(ERROR, "Timer") << "Cannot reinit timer - not calibrated"; return; } - uint64_t timerFreqHz = (uint64_t)g_ticksPerMs * 1000; + // Reprogram the timer hardware (its state was lost during S3). + // The calibration and the IRQ handler registration survive in RAM. + Arch::Timer::StartPeriodic(BSP_TICK_INTERVAL_MS); - KernelLogStream(OK, "Timer") << "APIC timer: " << base::dec << (uint64_t)g_ticksPerMs - << " ticks/ms (" << timerFreqHz << " Hz, divide-by-16)"; - - // Register IRQ handler for timer (IRQ 0 = vector 32) - Hal::RegisterIrqHandler(Hal::IRQ_TIMER, TimerHandler); - - // Configure APIC timer: periodic mode, vector 32 - ProgramBspPeriodicTimer(); - - KernelLogStream(OK, "Timer") << "APIC timer started: BSP " << base::dec - << (uint64_t)BSP_TIMER_HZ << " Hz periodic, initial count=" - << (uint64_t)CountForIntervalMs(BSP_TICK_INTERVAL_MS); - } - - void ApicTimerReinitialize() { - if (g_ticksPerMs == 0) { - KernelLogStream(ERROR, "Timer") << "Cannot reinit APIC timer - not calibrated"; - return; - } - - // Reprogram the APIC timer registers (they were lost during S3). - // The calibrated g_ticksPerMs value is still valid (it's in RAM). - // The IRQ handler registration also survives (it's a function pointer array in RAM). - ProgramBspPeriodicTimer(); - - KernelLogStream(OK, "Timer") << "APIC timer restarted after S3 resume"; + KernelLogStream(OK, "Timer") << "Timer restarted after S3 resume"; } uint64_t GetTicks() { @@ -214,35 +118,30 @@ namespace Timekeeping { g_schedEnabled.store(true, std::memory_order_release); } - void ApicTimerInitializeAP() { - // Use the BSP's calibrated ticks-per-ms value directly. - // All cores share the same bus clock, so the APIC timer rate is - // identical. This avoids PIT contention during AP boot. - if (g_ticksPerMs == 0) return; + void TimerInitializeAP() { + // Use the BSP's calibration directly: all cores share the same + // timer clock, and this avoids calibration contention during AP boot. + if (!g_timerReady) return; // Configure the 10 ms scheduler timer for running APs. Their idle loop // masks it after initialization and rearms it when dispatching work. - ProgramTimer(true, AP_TICK_INTERVAL_MS); + Arch::Timer::StartPeriodic(AP_TICK_INTERVAL_MS); } - void ApicTimerEnterApIdle() { + void TimerEnterApIdle() { auto* cpu = Smp::GetCurrentCpuData(); - if (cpu == nullptr || cpu->cpuIndex == 0 || g_ticksPerMs == 0) return; + if (cpu == nullptr || cpu->cpuIndex == 0 || !g_timerReady) return; - uint32_t lvt = Hal::LocalApic::ReadRegister(Hal::LocalApic::REG_TIMER_LVT); - if ((lvt & LVT_MASKED) == 0) { - Hal::LocalApic::WriteRegister(Hal::LocalApic::REG_TIMER_LVT, - lvt | LVT_MASKED); - } + Arch::Timer::Stop(); } - void ApicTimerLeaveApIdle() { + void TimerLeaveApIdle() { auto* cpu = Smp::GetCurrentCpuData(); - if (cpu == nullptr || cpu->cpuIndex == 0 || g_ticksPerMs == 0) return; + if (cpu == nullptr || cpu->cpuIndex == 0 || !g_timerReady) return; - // Reprogram the initial count as well as unmasking. A deep idle state + // Reprogram the period as well as unmasking. A deep idle state // may have stopped the local timer at an arbitrary point in its period. - ProgramTimer(true, AP_TICK_INTERVAL_MS); + Arch::Timer::StartPeriodic(AP_TICK_INTERVAL_MS); } void ServiceDeferredWork() { @@ -310,34 +209,32 @@ namespace Timekeeping { // Thermal policy records transitions during BSP maintenance; print // them from this explicitly non-interrupt idle path. - Hal::CpuPower::ServiceDeferredDiagnostics(); + Arch::CpuPower::ServiceDeferredDiagnostics(); cpu->reservedForKernelWork = wasReserved; } - void IdleOnce(bool hasMwait, volatile uint64_t* monitorAddr) { - (void)hasMwait; - (void)monitorAddr; + void IdleOnce() { auto* cpu = Smp::GetCurrentCpuData(); // HaltWithInterruptsDisabled() deliberately returns with IF clear so - // the next readiness check can lead into an atomic STI+HLT. Deferred + // the next readiness check can lead into an atomic enable+halt. Deferred // work cannot inherit that state: xHCI/Bluetooth service contains // wall-clock waits and depends on the BSP timer and device MSIs. If // it runs with IF=0, GetMilliseconds() never advances on the BSP and // the completion interrupt cannot arrive, producing a permanent idle- // loop freeze. Keep IRQs enabled for all active idle-context work and - // disable them again only immediately before entering HLT below. - asm volatile("sti" ::: "memory"); + // disable them again only immediately before halting below. + Arch::EnableInterrupts(); ServiceDeferredWork(); - if (cpu == nullptr || cpu->cpuIndex != 0 || !g_schedEnabled || g_ticksPerMs == 0) { + if (cpu == nullptr || cpu->cpuIndex != 0 || !g_schedEnabled || !g_timerReady) { // Non-BSP or pre-scheduler fallback: plain HLT keeps the LAPIC // timer running in C1 on hardware where MWAIT promotes to deeper // states that freeze the timer (no ARAT in practice). - asm volatile("cli" ::: "memory"); - Hal::HaltWithInterruptsDisabled(); + Arch::DisableInterrupts(); + Arch::HaltWithInterruptsDisabled(); return; } @@ -361,36 +258,32 @@ namespace Timekeeping { // HLT (vs MWAIT) is the second half: MWAIT EAX=0 is a *hint* and // the CPU may go deeper than C1; HLT generally stays in C1, which // is the deepest state guaranteed to keep the LAPIC timer alive. - asm volatile("cli" ::: "memory"); - Hal::HaltWithInterruptsDisabled(); + Arch::DisableInterrupts(); + Arch::HaltWithInterruptsDisabled(); } void BusyWaitUs(uint64_t us) { - // An I/O write to port 0x80 is an ISA-bus cycle of roughly a - // microsecond on every chipset since the 486, and never shorter. for (uint64_t i = 0; i < us; i++) - Io::IoPortWait(); + Arch::IoDelay(); } void BusyWaitMs(uint64_t ms) { if (ms == 0) return; - uint64_t flags; - asm volatile("pushfq; pop %0" : "=r"(flags)); - bool ticking = g_ticksPerMs != 0 && (flags & (1 << 9)); + bool ticking = g_timerReady && Arch::InterruptsEnabled(); // Either clock alone is long enough, so stop at whichever says so - // first. The tick is the accurate one; the port writes bound the wait + // first. The tick is the accurate one; the IoDelay spins bound the wait // when the tick is not running yet (or is masked), so this can never // hang on a timer that does not fire. One extra tick of slack because // the first may already be nearly over. uint64_t start = g_tickCount.load(std::memory_order_relaxed); - uint64_t portBudget = (ms + 1) * 1000; - for (uint64_t spins = 0; spins < portBudget; spins++) { + uint64_t delayBudget = (ms + 1) * 1000; + for (uint64_t spins = 0; spins < delayBudget; spins++) { if (ticking && g_tickCount.load(std::memory_order_relaxed) - start >= ms + 1) return; - Io::IoPortWait(); + Arch::IoDelay(); } } diff --git a/kernel/src/Timekeeping/Timer.hpp b/kernel/src/Timekeeping/Timer.hpp new file mode 100644 index 0000000..1442733 --- /dev/null +++ b/kernel/src/Timekeeping/Timer.hpp @@ -0,0 +1,55 @@ +/* + * Timer.hpp + * Periodic tick, timekeeping and idle-loop servicing + * Copyright (c) 2025 Daniel Hammer +*/ + +#pragma once +#include + +namespace Timekeeping { + // Start the boot CPU's periodic tick (Arch::Timer) and take over + // Arch::IRQ_TIMER for timekeeping and scheduling. + void TimerInitialize(); + + // Start the scheduler tick on a secondary CPU using the BSP calibration. + void TimerInitializeAP(); + + // Idle APs are woken for runnable work by the reschedule IPI, so their + // periodic scheduler timer can remain masked until a process is dispatched. + void TimerEnterApIdle(); + void TimerLeaveApIdle(); + + // Restart the boot CPU's tick after S3 resume using the previously + // calibrated rate. Skips calibration and IRQ registration (both survive + // in RAM); only reprograms the timer hardware. + void TimerReinitialize(); + + // Get the monotonic tick count (increments on each timer interrupt) + uint64_t GetTicks(); + + // Get elapsed milliseconds since timer initialization + uint64_t GetMilliseconds(); + + // Enable scheduler tick (called after scheduler is initialized) + void EnableSchedulerTick(); + + // Run bounded process-context bottom halves from an idle CPU. + void ServiceDeferredWork(); + + // Enter one idle wait cycle for the current CPU. Halts until the next + // interrupt and relies on the boot-time periodic timer to wake -- a + // one-shot wake doesn't survive deep C-states on some laptop CPUs. + void IdleOnce(); + + // Busy-wait sleep for the given number of milliseconds + void Sleep(uint64_t ms); + + // Spin for at least `ms` milliseconds without descheduling. Usable from + // boot code, before the scheduler runs, and with interrupts disabled + // (falls back to Arch::IoDelay when the tick is not advancing). + void BusyWaitMs(uint64_t ms); + + // Spin for at least `us` microseconds (Arch::IoDelay timed; errs long). + void BusyWaitUs(uint64_t us); +}; diff --git a/scripts/check-arch-neutral.sh b/scripts/check-arch-neutral.sh new file mode 100755 index 0000000..ab9bd6d --- /dev/null +++ b/scripts/check-arch-neutral.sh @@ -0,0 +1,101 @@ +#!/bin/sh +# check-arch-neutral.sh - Verify the kernel's architecture boundary. +# +# 1. Static rules: outside kernel/src/Arch/, kernel code may not contain +# inline assembly, assembly sources, architecture #ifdefs, or includes of +# a specific port's headers (it must use the interfaces in Arch/*.hpp). +# 2. If clang is installed, compile the whole kernel for a second +# architecture (default aarch64) and report any symbol the generic code +# needs that the port does not define. Links too when ld.lld is found. +# +# Usage: ./scripts/check-arch-neutral.sh [arch] +# +# Copyright (c) 2026 Daniel Hammer + +set -e + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +SRC="$ROOT/kernel/src" +ARCH="${1:-aarch64}" +status=0 + +cd "$SRC" + +# Files allowed to name an architecture: the Limine adapter reads the +# per-architecture MP structures limine.h defines, and Io/IoPort.hpp is an +# interface header that dispatches to the port like Arch/*.hpp do. +ALLOW='^(limine\.h|Boot/Protocols/LimineProtocol\.cpp|Io/IoPort\.hpp)$' + +generic=$(find . -type f \( -name '*.c' -o -name '*.cpp' -o -name '*.h' -o -name '*.hpp' \) \ + ! -path './Arch/*' | sed 's|^\./||' | LC_ALL=C sort) + +report() { + echo "check-arch-neutral: $1" >&2 + echo "$2" | sed 's/^/ /' >&2 + status=1 +} + +hits=$(grep -nE '(^|[^_[:alnum:]])(asm|__asm__)[[:space:]]*(volatile|\()' $generic || true) +[ -n "$hits" ] && report "inline assembly outside src/Arch/:" "$hits" + +hits=$(find . -type f \( -name '*.asm' -o -name '*.S' -o -name '*.s' \) ! -path './Arch/*') +[ -n "$hits" ] && report "assembly sources outside src/Arch/:" "$hits" + +hits=$(echo "$generic" | grep -vE "$ALLOW" | xargs grep -nE \ + '__(x86_64|i386|aarch64|arm|riscv|loongarch[0-9]*)__|MONTAUK_ARCH_(X86|AARCH|RISCV|LOONG)[A-Z0-9_]*' || true) +[ -n "$hits" ] && report "architecture conditionals outside src/Arch/:" "$hits" + +hits=$(echo "$generic" | grep -vE "$ALLOW" | xargs grep -nE \ + '#[[:space:]]*include[[:space:]]*[<"]Arch/[^/>"]+/' || true) +[ -n "$hits" ] && report "includes of a port's private headers:" "$hits" + +if [ $status -eq 0 ]; then + echo "check-arch-neutral: static rules OK" +fi + +# ---- Cross-compile check ---- +if ! command -v clang >/dev/null 2>&1; then + echo "check-arch-neutral: clang not found, skipping the $ARCH build" + exit $status +fi +if [ ! -d "Arch/$ARCH" ]; then + echo "check-arch-neutral: no src/Arch/$ARCH port" >&2 + exit 1 +fi + +echo "check-arch-neutral: compiling the kernel for $ARCH" +make -C "$ROOT/kernel" ARCH="$ARCH" objects -j"$(nproc 2>/dev/null || echo 4)" \ + >/dev/null 2>"$ROOT/kernel/check-arch-$ARCH.log" || { + echo "check-arch-neutral: $ARCH compile failed (kernel/check-arch-$ARCH.log):" >&2 + grep -E 'error' "$ROOT/kernel/check-arch-$ARCH.log" | head -20 >&2 + exit 1 +} +rm -f "$ROOT/kernel/check-arch-$ARCH.log" + +NM="$(clang -print-prog-name=llvm-nm)" +command -v "$NM" >/dev/null 2>&1 || NM=nm +CXXFILT="$(clang -print-prog-name=llvm-cxxfilt)" +command -v "$CXXFILT" >/dev/null 2>&1 || CXXFILT=cat + +cd "$ROOT/kernel" +objs=$(find "obj-$ARCH" -name '*.o') +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT +"$NM" --defined-only -j $objs "cc-runtime-$ARCH/cc-runtime.a" 2>/dev/null | LC_ALL=C sort -u > "$tmp/def" +"$NM" --undefined-only -j $objs | LC_ALL=C sort -u > "$tmp/undef" +# Provided by the linker script. +printf '%s\n' KernelStartSymbol KernelEndSymbol __init_array __init_array_end >> "$tmp/def" +LC_ALL=C sort -u -o "$tmp/def" "$tmp/def" +missing=$(LC_ALL=C comm -23 "$tmp/undef" "$tmp/def" | "$CXXFILT") +if [ -n "$missing" ]; then + report "symbols the $ARCH port must still provide:" "$missing" + exit 1 +fi +echo "check-arch-neutral: $ARCH objects compile and every symbol resolves" + +if command -v "${KERNEL_LD:-ld.lld}" >/dev/null 2>&1; then + make ARCH="$ARCH" >/dev/null + echo "check-arch-neutral: linked kernel/bin-$ARCH/kernel" +fi + +exit $status diff --git a/tests/wifi/shim/Timekeeping/ApicTimer.hpp b/tests/wifi/shim/Timekeeping/Timer.hpp similarity index 100% rename from tests/wifi/shim/Timekeeping/ApicTimer.hpp rename to tests/wifi/shim/Timekeeping/Timer.hpp