345 lines
19 KiB
Markdown
345 lines
19 KiB
Markdown
# Raspberry Pi 4 / 400
|
|
|
|
MontaukOS runs on the Raspberry Pi 4 and 400 through the Pi Firmware Task
|
|
Force's UEFI firmware ([pftf/RPi4](https://github.com/pftf/RPi4), an EDK2
|
|
build) in ACPI mode, booting Limine like any other UEFI machine.
|
|
|
|
## Building and writing the SD card
|
|
|
|
```bash
|
|
./toolchain/build-montauk-toolchain.sh aarch64 # once
|
|
make ARCH=aarch64 rpi4-sd # -> montauk-rpi4.img
|
|
sudo dd if=montauk-rpi4.img of=/dev/sdX bs=4M conv=fsync
|
|
```
|
|
|
|
Or use the imager, which also updates a card in place:
|
|
|
|
```bash
|
|
make rpi4-imager # = ./scripts/montauk-imager.py
|
|
```
|
|
|
|
### The imager
|
|
|
|
`scripts/montauk-imager.py` is a Tk window (`python3-tk`) that lists the SD
|
|
cards and USB drives attached (never the host's own disks) and what is on
|
|
them, and either writes a fresh image or updates the MontaukOS on a card. It
|
|
runs as the user and does the writing by starting itself again through
|
|
`pkexec` (it asks for the password each time). The same operations work from
|
|
a terminal: `list`, and as root `flash`, `update` and `expand`
|
|
(`--help` for the options). Images can be raw or `.gz`, `.xz` or `.zip`
|
|
compressed; **Build** runs `make ARCH=aarch64 rpi4-sd`.
|
|
|
|
- **Write a fresh image** erases the card, writes the image, clears the last
|
|
MiB of the card (a backup GPT from an earlier OS) and reads the card back to
|
|
verify it. *Expand* grows the root to fill the card right away instead of
|
|
leaving it to `firstboot.elf`.
|
|
- **Update** leaves the partition table alone and finds both partitions by
|
|
label, so a root that has grown to fill the card (or was resized by hand)
|
|
stays as it is. On the ESP every file is written except `RPI_EFI.fd`, which
|
|
holds the firmware's own settings (*Update the UEFI firmware* replaces it);
|
|
a changed `config.txt` is kept as `config.txt.bak`. On the root, `os/`,
|
|
`apps/`, `sdk/`, `boot/` get the image's new and changed files; `config/`
|
|
and `www/` only get files the card lacks, `users/` and `state/` are not
|
|
touched. `init.toml` becomes the image's, keeping each service's `enabled`
|
|
from the card and appending services only the card has; `capabilities.toml`
|
|
becomes the image's plus grants only the card has (old copies: `.old`).
|
|
`os/firstboot.elf` is only written if the card still has it, so a grown
|
|
card does not grow again. *Remove system files the new image no longer
|
|
ships* prunes `os/`, `apps/`, `sdk/` and `boot/` (keeping `os/btkeys.bin`,
|
|
the Bluetooth bonds). Unchanged files are skipped and each file is
|
|
replaced by a rename; the volume is checked with `e2fsck` before and after,
|
|
and its features against what `Fs/Ext2.cpp` mounts.
|
|
- **Expand** (a host-side first boot) grows the root partition to the end of
|
|
the card, at most 1 TiB, and the volume with `resize2fs`; `firstboot.elf`
|
|
then finds nothing to do and deletes itself.
|
|
|
|
While it works, a runtime udev rule keeps the desktop from automounting the
|
|
card's partitions (Linux writes to an ext2 volume as soon as it mounts it).
|
|
|
|
`scripts/mk-rpi4-sd.sh` downloads the firmware release once into
|
|
`rpi4-firmware/` (`PFTF_VERSION=v1.53` by default) and builds an MBR image:
|
|
|
|
| Partition | Type | Contents |
|
|
|---|---|---|
|
|
| 1 | 0xEF, FAT32 | the firmware files, Limine (`EFI/BOOT/BOOTAA64.EFI`), `boot/kernel` |
|
|
| 2 | 0x83, ext2, label `MONTAUKROOT` | the system tree (the contents of `ramdisk.tar`) |
|
|
|
|
Limine loads only the kernel; the `limine.conf` on the card has no ramdisk
|
|
module. Finding no ramdisk, the kernel mounts the partition labelled
|
|
`MONTAUKROOT` as `0:/` (`Fs/Boot.cpp`), read through the SD driver
|
|
(`Drivers/Storage/Sdhci`), and the ESP after it. Files load as they are
|
|
used instead of the firmware reading the whole 80 MB tree before boot.
|
|
The card can also boot from a USB card reader (or the image from a USB
|
|
stick): the root then comes through `Drivers/USB/MassStorage.cpp`. A USB
|
|
root can register after the boot scan (a SuperSpeed link still training, a
|
|
reader still bringing its card up), so a boot that finds no `MONTAUKROOT`
|
|
waits up to 10 s for one, servicing USB hot-plug meanwhile.
|
|
Unlike the ramdisk, the root is persistent: changes to `0:/` survive a
|
|
reboot, so shut down from the desktop rather than pulling power.
|
|
The image is sized to its contents (`ROOT_FREE_MIB`, default 64, sets the
|
|
free space on the root) and grows to fill the card on first boot.
|
|
|
|
### First boot: growing the root
|
|
|
|
Only the SD image ships `0:/os/firstboot.elf` (`programs/src/firstboot`,
|
|
built into `programs/bin-aarch64-rpi/`, which the script lays over the
|
|
system tree), and the script adds its init service, ahead of all others,
|
|
and its capability grant (`storage_admin`, `system_image`) to the image's
|
|
`init.toml` and `capabilities.toml`. On first boot it shows a status card
|
|
in the login screen's style while it calls `SYS_FS_GROW` on drive 0 in a
|
|
loop:
|
|
|
|
1. The kernel grows the MBR entry of the root partition as far as the
|
|
filesystem can use (to the end of the card, or 1 TiB), on a 1 MiB
|
|
boundary (`Gpt::ResizeMbrPartition`).
|
|
2. The ext2 driver grows the mounted volume (`Fs/Ext2.cpp`, "Online
|
|
grow"): each call writes the bitmaps and zeroed inode tables of a few
|
|
new block groups past the old end, then the last call commits the
|
|
bitmaps, descriptor table and superblock. The descriptor table grows
|
|
into the blocks `mke2fs -O resize_inode -E resize=2^28` reserved; the
|
|
leftover reservation is freed and `resize_inode` cleared, so the volume
|
|
grows once.
|
|
|
|
The new groups' inode tables are the bulk of the writes (0.1% of the card,
|
|
one inode per 256 KiB), written as long multi-block runs. Once the volume has grown, `firstboot.elf` deletes
|
|
itself and init skips its (optional) service from then on. If it fails it
|
|
stays and the next boot tries again; an interrupted grow is safe, since the
|
|
new groups are only referenced once the superblock is written, and a
|
|
partition larger than its filesystem is valid.
|
|
|
|
`make ARCH=aarch64 rpi4-sd ROOTFS=ramdisk` builds the old layout instead:
|
|
one FAT32 partition with `boot/ramdisk.tar`, which Limine loads through the
|
|
firmware. It boots slowly but does not depend on the kernel's SD driver.
|
|
|
|
`make ARCH=aarch64 run-rpi4-sd` boots the ext2 image on QEMU with the card
|
|
on `sdhci-pci`, so the root comes through the same driver. QEMU's UEFI has
|
|
no SD driver, so the card's ESP is also attached on its own over virtio for
|
|
the firmware to load Limine from. The card is padded to
|
|
`RPI4_QEMU_CARD_MIB` (default 4096), and every run starts from a fresh
|
|
copy, so every run is a first boot.
|
|
|
|
The image's `RPI_EFI.fd` comes with a **MontaukOS** boot entry first in the
|
|
firmware's boot order (`scripts/rpi4-efi-bootorder.py` writes it into the
|
|
release's empty variable store), so the card boots on its own instead of
|
|
falling behind the PXE/HTTP entries the firmware creates. The entry names the
|
|
ESP by the image's MBR disk ID and partition number, so it works from the SD
|
|
slot or a USB drive. A card whose `RPI_EFI.fd` predates it keeps its old
|
|
boot order until it is updated with *Update the UEFI firmware*
|
|
(`--firmware`), which also copies the image's disk ID to the card.
|
|
|
|
The firmware defaults are what MontaukOS needs; leave them:
|
|
|
|
- **System Table Mode: ACPI.** The kernel reads ACPI only.
|
|
- **Limit RAM to 3 GB: enabled.** The BCM2711's PCIe can only reach the
|
|
first 3 GB (its tables say so in `_DMA`). The kernel warns at boot when
|
|
RAM extends past a device's `_DMA` range.
|
|
|
|
## Serial console
|
|
|
|
The kernel console (the whole boot log, including what was printed before
|
|
the port was found) is mirrored to the UART the firmware's SPCR table names.
|
|
The firmware's `config.txt` applies the `miniuart-bt` overlay, which puts the
|
|
PL011 on GPIO 14/15: header pins 8 (TX) and 10 (RX), ground on pin 6, 115200
|
|
8N1, 3.3 V levels. On the Pi 400 the header is on the back.
|
|
|
|
## How the hardware is reached
|
|
|
|
| Part | Source | Driver |
|
|
|---|---|---|
|
|
| 4x Cortex-A72 | MADT, PSCI (TF-A, SMC) | `Arch/aarch64` SMP via Limine MP |
|
|
| GIC-400 (GICv2) | MADT | `Arch/aarch64/Interrupts.cpp` |
|
|
| Generic timer | GTDT | `Arch/aarch64/Timer.cpp` |
|
|
| Display | UEFI GOP (VideoCore) | Limine framebuffer |
|
|
| Keyboard (built in), USB ports | VL805 xHCI as ACPI `\_SB.SCB0.XHC0` (`PNP0D10`) | `Drivers/USB/Xhci.cpp` platform probe |
|
|
| USB 2 devices (built-in keyboard too) | VIA hub on the VL805's USB 2 root port | `Drivers/USB/Hub.cpp` |
|
|
| Reset, power-off | PSCI | `Arch::Reboot`, `Arch::PowerOff` |
|
|
| Bluetooth (CYW43455 on the 4B, CYW43456 on the 400) | `BCM2EA6` (`\_SB.GDV0.BTH0`) on the mini UART (`BCM2836`) | `Drivers/Bluetooth/Broadcom.cpp`, `Drivers/Serial/BtUart.cpp` |
|
|
| VideoCore firmware mailbox (radio power, clocks, board MAC) | `BCM2849` (`RPIQ`) | `Drivers/Platform/RpiFirmware.cpp` |
|
|
| GPIO pin functions | `BCM2845` (`GPI0`) | `Drivers/Platform/RpiGpio.cpp` |
|
|
| Ethernet (GENET v5 MAC, BCM54213PE PHY) | `BCM6E4E` (`\_SB.ETH0`) | `Drivers/Net/Genet.cpp` |
|
|
| Wi-Fi (the same CYW4345x, SDIO) | Arasan SD controller `BCM2847`, GPIO 34-39 | `Drivers/Net/Wifi/Brcmf*.cpp` |
|
|
|
|
The VL805 sits behind the BCM2711's PCIe, whose config space is not ECAM
|
|
(the root bridge's `_DSD` says `linux-ecam-quirk-id = "bcm2711"`), so the
|
|
kernel does not enumerate PCI there. The firmware has already brought up the
|
|
link and the VL805 and describes the controller as a platform device: a
|
|
register block at `0x600000000`, GSI 175, and `_CCA = 0`, i.e. its DMA does
|
|
not snoop the CPU caches. `XHC0._INI` (run by the AML interpreter) enables
|
|
the controller's PCI memory decoding and bus mastering. The xHCI driver then
|
|
keeps its rings and contexts in uncached memory, bounces control-transfer
|
|
data through an uncached page, and cleans/invalidates the caller buffers of
|
|
bulk transfers (`Memory/Dma.hpp`). It also watches the event ring from the
|
|
timer tick, so input keeps working if the INTx line never fires.
|
|
|
|
The VL805 has one USB 2 root port (port 1) and four USB 3 ones. An on-board
|
|
VIA hub (`2109:3431`) sits on the USB 2 port, and the USB 2 lines of every
|
|
socket, and the Pi 400's internal keyboard (`04d9:0007`), hang off it: a
|
|
keyboard or mouse is always a low/full-speed device behind a high-speed hub,
|
|
reached through that hub's transaction translator.
|
|
|
|
## Bluetooth
|
|
|
|
The radio's Bluetooth half speaks HCI over a UART (H4) on GPIO 30-33 (CTS,
|
|
RTS, TXD, RXD); its power switch, `BT_ON`, is line 0 of the VideoCore
|
|
firmware's GPIO expander, reached through the mailbox. The firmware tables
|
|
describe it as `BTH0`, whose `UARTSerialBus` names the UART it uses: with
|
|
the `miniuart-bt` overlay the PL011 is the console, so Bluetooth has the mini
|
|
UART. The kernel refuses to take a UART that is the SPCR console.
|
|
|
|
At boot the driver probe routes the pins (`ALT5` for the mini UART, `ALT3`
|
|
for the PL011), starts the UART (interrupt driven, 16 KiB receive and
|
|
transmit rings) and the H4 transport under the shared HCI layer. Once the
|
|
root is mounted the idle loop finishes, as for USB adapters:
|
|
|
|
1. power-cycle the controller through `BT_ON`, HCI Reset at 115200 baud;
|
|
2. Read Local Version: the ROM's LMP subversion picks the patch file,
|
|
`0:/os/firmware/brcm/BCM4345C0.hcd` (0x6119, 4B) or `BCM4345C5.hcd`
|
|
(0x6606, 400), shipped unmodified from Raspberry Pi OS's bluez-firmware;
|
|
3. raise both ends to the operating speed (vendor `0xFC18`);
|
|
4. Download Minidriver (`0xFC2E`), then every record of the `.hcd` file
|
|
(Write RAM, then Launch RAM), as Linux's `btbcm` does;
|
|
5. the patched firmware restarts at 115200: reset, raise the speed again.
|
|
|
|
The mini UART's FIFO is 8 bytes deep. Its hardware RTS is tried first; if
|
|
the controller answers with it on, the link runs at 921600 baud, otherwise
|
|
at 460800 without flow control (what Raspberry Pi OS uses), and at 115200
|
|
if the UART has no interrupt. The baud rate derives from the VPU core
|
|
clock, so `mk-rpi4-sd.sh` pins it (`core_freq=500`, `core_freq_min=500` in
|
|
`config.txt`), as the overlay's documentation requires. A failed fast
|
|
bring-up is retried entirely at 115200.
|
|
|
|
The CYW43456 reports a chip-wide default address (`43:45:C5:00:1F:AC`);
|
|
the driver replaces it with the board's Ethernet MAC from the firmware.
|
|
`0:/config/bluetooth.toml` still overrides both (Broadcom `0xFC01`).
|
|
|
|
## Ethernet
|
|
|
|
The BCM2711's own Gigabit MAC, GENET v5 (a UniMAC behind a DMA engine),
|
|
talks RGMII to a BCM54213PE PHY at MDIO address 1. The firmware describes
|
|
it as `BCM6E4E`: a 64 KiB register block, two interrupts (the driver uses
|
|
the first, which carries the default queue), `_CCA = 0`, and a `_DSD` with
|
|
`phy-mode` (`rgmii-rxid`: the PHY delays the receive clock, the MAC the
|
|
transmit clock) and `brcm,max-dma-burst-size`.
|
|
|
|
The driver uses only the default queue (ring 16), with all 256 descriptors
|
|
of each direction. The descriptors live in the MAC's own SRAM, so only the
|
|
2 KiB packet buffers need cache maintenance. Received frames start two
|
|
bytes into their buffer (`RBUF_ALIGN_2B`); neither direction uses the
|
|
64-byte status blocks, so there is no checksum offload. The UniMAC accepts
|
|
broadcast and its own address only (the MDF filter).
|
|
|
|
The interrupt only acknowledges and masks; frames are dispatched from the
|
|
idle bottom half, which also checks the ring itself, so receive works if
|
|
the interrupt never arrives. The PHY is polled once a second (its interrupt
|
|
pin is not described); a link change reprograms the RGMII block and the
|
|
UniMAC's speed and duplex. A link the firmware already negotiated is kept,
|
|
otherwise the PHY is reset and advertises 10/100 and 1000 full duplex.
|
|
|
|
The MAC address is the board's, read from OTP through the firmware mailbox
|
|
(the one the Bluetooth driver uses too); the address the firmware's own
|
|
driver programmed is the fallback. `dhcp.elf` waits up to 10 s for a wired
|
|
link, since autonegotiation takes about 3 s after the PHY reset.
|
|
|
|
## Wi-Fi
|
|
|
|
The radio's WLAN half is an SDIO card on the BCM2711's older SD controller,
|
|
the Arasan-derived EMMC (`BCM2847`, `0x7E300000`); the card slot is on
|
|
EMMC2, so the two never share a bus. Its pins are GPIO 34-39 (ALT3; clock
|
|
without a pull, the rest pulled up, as Linux's `sdio_pins`) and its power
|
|
switch, `WL_ON`, is line 1 of the firmware's GPIO expander. The probe
|
|
claims the controller only while the slot is on EMMC2 (`BRCME88C` present);
|
|
if the tables do not describe the Arasan, its address is taken as EMMC2's
|
|
less 256 KiB.
|
|
|
|
It is a FullMAC radio: the firmware does the 802.11 MLME itself and the host
|
|
sees Ethernet frames. The driver follows Linux's brcmfmac (ISC license,
|
|
Broadcom; its notice is in `NOTICES.txt`):
|
|
|
|
| File | Part |
|
|
|---|---|
|
|
| `BrcmfSdio.cpp` | the Arasan as an SDIO host: CMD5/3/7, 4-bit at 25 MHz, CMD52/CMD53, polled PIO, 32-bit register access |
|
|
| `BrcmfChip.cpp` | backplane window (function 1), clocks, the enumeration ROM's core list, firmware + NVRAM download, function 2, the firmware console |
|
|
| `BrcmfProto.cpp` | SDPCM framing and transmit credit, CDC ioctls, BDC data, events, the receive pump, the transmit queue |
|
|
| `BrcmfConnect.cpp` | preinit and CLM, escan, join, keys, link state |
|
|
|
|
Once the root is mounted the idle loop:
|
|
|
|
1. routes the pins, switches the EMMC clock on, pulses `WL_ON`, enumerates
|
|
the card (function 1 blocks of 64 bytes, function 2 of 512);
|
|
2. requests the ALP clock, reads the chip ID: `0x4345` revision 9 is the
|
|
CYW43456 (Pi 400), any other revision the CYW43455 (Pi 4B), which picks
|
|
`0:/os/firmware/brcm/brcmfmac43456-sdio.*` or `brcmfmac43455-sdio.*`
|
|
(shipped unmodified from Raspberry Pi OS's firmware-nonfree);
|
|
3. halts the ARM Cortex-R4, holds the 802.11 core in reset, sizes the RAM
|
|
from the CR4's banks, writes the firmware at `0x198000` and the NVRAM
|
|
(stripped to `key=value\0` pairs and a length token) at the top of RAM,
|
|
reading both back to verify, puts the reset vector at address 0 and
|
|
releases the CPU;
|
|
4. waits for the HT clock, sends the SDPCM protocol version through the
|
|
mailbox, enables function 2, the F2 watermark and MES busy signalling of
|
|
Linux's CYW43455 settings, save/restore if the chip has it, and the card
|
|
interrupt;
|
|
5. reads `cur_etheraddr` (the NVRAM's shared placeholder address
|
|
`b8:27:eb:74:f2:6c` is replaced with the board address + 1, as on
|
|
Raspberry Pi OS), loads the CLM blob (`clmload`, 1400-byte chunks: these
|
|
firmware builds carry no regulatory data of their own), turns off glom,
|
|
roaming, ARP/ND offload and power save, enables the events it handles and
|
|
brings the interface up (`WLC_UP`).
|
|
|
|
Nothing uses the controller's interrupt: the bus is swept from the idle
|
|
loop (and from the timer tick when the cores never idle), every 10 ms while
|
|
a scan or a connection is active and every 250 ms otherwise, and at once when
|
|
the controller shows the card interrupt. Every sweep reads the SDIO core's
|
|
interrupt status, then every frame waiting in function 2 (a 4-byte header
|
|
read, then the rest). Ioctls wait for transmit credit and their answer by
|
|
sweeping the bus themselves. Frames are delivered with the bus unlocked, and
|
|
transmitting never waits for the bus: data frames go into a 32-frame queue
|
|
that whoever holds the bus, or the next sweep, sends.
|
|
|
|
Scans are escans (all channels, active, firmware dwell times); each result
|
|
event carries a `brcmf_bss_info_le` whose information elements go into the
|
|
same scan table the Intel driver fills from beacons. A join sets `wpaie`
|
|
(the RSN element our supplicant builds), `wpa_auth`, `auth` and `wsec`, then
|
|
`join` with the BSSID and the chanspec escan reported. The 4-way handshake
|
|
runs in the host's supplicant (`Wpa.cpp`) as for the Intel adapter: EAPOL
|
|
frames are ordinary Ethernet frames here, and the keys go to the firmware
|
|
with `wsec_key` once message 4 has left the transmit queue. The radio does
|
|
AES-CCMP only: GCMP networks are refused, as are WPA3-only (SAE), WEP and
|
|
TKIP ones.
|
|
|
|
Unanswered ioctls (a leaky bucket of three), a halt reported through
|
|
the mailbox, or repeated bus errors mark the radio failed: the firmware's
|
|
console and any trap record are printed to the kernel log (`WiFi-FW`), and
|
|
the facade's recovery powers the radio off and repeats the bring-up (at
|
|
most three times).
|
|
|
|
`tests/wifi/run.sh` drives `BrcmfProto.cpp` and `BrcmfConnect.cpp` with the
|
|
real supplicant against a simulated firmware: preinit and the CLM chunks,
|
|
escan parsing, open and WPA2 joins (an independent authenticator checks
|
|
messages 2 and 4 and the installed keys), the wrong passphrase, refusals,
|
|
join timeouts and a firmware that stops answering.
|
|
|
|
## Not supported yet
|
|
|
|
- SD high-speed mode: the driver transfers by polled PIO at 25 MHz on a
|
|
4-bit bus (multi-block CMD18/CMD25 with Auto CMD12), at most 12.5 MB/s;
|
|
switching the card to 50 MHz (CMD6) would double that.
|
|
- Wi-Fi beyond 802.11 station mode on one interface: no access point mode,
|
|
no SDIO high speed (50 MHz), no glom, no firmware supplicant.
|
|
- Audio (HDMI via VideoCore, the 3.5 mm jack via PWM).
|
|
|
|
## Status
|
|
|
|
Bluetooth works on a Pi 400 (CYW43456), including A2DP music to
|
|
headphones. Ethernet has not run yet: QEMU has no GENET model. Wi-Fi has
|
|
not run on hardware yet either (QEMU has no model of the radio); only its
|
|
protocol and connection layers are tested, in the host harness. It is untested on a Pi 4B (CYW43455), and QEMU has no model of
|
|
the radio.
|
|
|
|
Everything else above is verified on QEMU only, including the Pi-specific path:
|
|
the firmware's own XHC0 table, retargeted at QEMU's xHCI and loaded with the
|
|
`bcm2711` quirk on QEMU's root bridge, boots to the desktop with non-coherent
|
|
DMA. Real hardware has not been tried yet. The ext2 root on the SD card boots on QEMU (`run-rpi4-sd`); the
|
|
first-boot grow has been tested only against the kernel's ext2 code in a
|
|
host harness (images grown, then checked with `e2fsck`).
|