// Język: English // Rodzaj aktywności: Prelekcja // Poziom zaawansowania: Średnio-zaawansowany // Czas trwania [min]: 60 // Kategorie tematyczne: Administracja systemami, Chmura, wirtualizacja i kontenery, Systemy embedded i mobile // Autorzy // 1. Imię i nazwisko: Zbigniew Jęrzejewski-Szmek // Opis autora: systemd developer, Fedora Linux maintainer // Tytuł zgłoszenia: Image-based Linux with systemd // Krótki opis // The majority of development effort in systemd upstream is directed at allowing image-based, immutable, cryptographically-verified "installations" of Linux. In theory, it is all possible and even easy, but in practice it turns out that doing this for real installations is hard. We need to take care of efficient building of the images, delivery of updates, secure extensions and configuration changes to those immutable systems, failure recovery, boot measurements predictions, a whole slew of engineering problems. I'll talk about the work happening in systemd, but also the sister project "mkosi", which we use to build installation images, initrds, and extensions from distribution packages. // Cel propozycji // Chciałbym opowiedzieć o głównych celech rozwoju w projekcie systemd, mkosi, particleos. Jest to obszar rozwoju Linuksa, a nawet współzawodnictwa pomiędzy różnymi grupami i podejściami. Mamy co najmniej rpm-ostree/bootc ze strony Red Hata, Ubuntu Core i snapy ze strony Canonicala, powyższe podejście ze strony "upstream systemd", oraz oczywiście różne systemy dostosowane do installacji embedded/IoT. Wydaje mi się, że jest to ciekawy temat. // Grupa docelowa/widownia // Zaawansowani użytkownicy Linuksa, osoby zajmujące się budową albo deploymentem Linuksa. // Szczegółowy abstrakt // Our final goal is a self-updating installation of the typical Linux software (kernel, systemd, libraries, GNOME/KDE/other graphical environment, various services). The whole boot sequence should be protected by cryptographic signatures and Secure Boot, and the access to "secrets" like SSH keys or data should be only allowed if the booted systems passes attestation. On the other hand, the whole system must be configurable and flexible. We don't want to do everything from scratch: Linux distributions are great at providing packages of whatever software, so the question is how to efficiently take those packages and build images from them, and how to build and deliver updates. // Plan propozycji // In the presentation, I want to start with the presentation of the goal (like in the detailed abstract above), and then go through various tools that we're building to make this possible: mkosi, adjustment to packages in Fedora and other distributions, various systemd components like sysexts, support for PCR measurements, etc. In the summary, I want to compare those approaches to e.g. rpm-ostree/bootc and the wider ecosystem and why we're trying this approach. // Get Polylux from the official package repository #import "@preview/polylux:0.4.0": * #enable-handout-mode(true) // Make the paper dimensions fit for a presentation and the text larger #set page(paper: "presentation-4-3") #set text(size: 25pt, font: "Lato") #set text(hyphenate: false) #show "DEMO!": name => [ #text(fill: red)[ #name ] ] #show "MKOSI": name => [#text(fill: red)[m]a#text(fill: red)[k]e #text(fill: red)[o]perating #text(fill: red)[s]ystem #text(fill: red)[i]mage] #set document(title: [Image-based Linux with systemd]) #slide[ #set align(center) #show title: smallcaps #title[#context document.title] #v(1fr); #image("images/blue-pacman-192x192.png", width: 2em) Zbigniew Jędrzejewski-Szmek \ #image("images/Logo-redhat-color-375.png", width: 8em) #v(1fr); #set text(size: 0.8em) Jesień Linuksowa, 25 października 2025, Rybnik ] #set page(numbering: (..nums) => [ #set text(size: 0.6em) #h(1fr); #toolbox.slide-number / #toolbox.last-slide-number ]) #slide[ == What have systemd developers been up to? #set align(center) #only(2)[ #image("images/commits_over_the_years.png", width: 80%) ] #only(3)[ #image("images/commits_over_the_years_recent.png", width: 80%) ] ] #slide[ == What have systemd developers been up to? - few new features directed at "traditional system adminstration" - a lot of new code for "image-based systems" ] #slide[ == Linux systems in 2025 #show: later always connected #show: later , \ untrusted physical environment _and_ untrusted payloads \ #show: later => always exposed #show: later traditional installations: mutable package-based system #show: later - each system is a snowflake - changes are non-atomic - modifications are persistent ] // #slide[ // == Linux security // Linux security: \ // - layers of additions, bits and pieces to close holes // #show: later // - complexity works against security // #show: later // - "that your software will be vulnerable is inevitable" // ] #slide[ == Why image-based systems? #show: later want to verify authenticity _at runtime_ \ => signed \ => immutable \ => image-based ] #slide[ == What have systemd developers been up to? ] #slide[ == New(-ish) components #set text(hyphenate: false) #box[`systemd-pcrextend`], #box[`systemd-creds`], #box[`systemd-homed`], #box[`systemd-cryptenroll`], #box[`systemd-cryptsetup`], #box[`systemd-repart`], #box[`systemd-measure`], #box[`systemd-pcrlock`], #box[`systemd-dissect`], #box[`systemd-tpm2-setup`], #box[`systemd-tpm2-clear`], #box[`systemd-veritysetup`], #box[`systemd-gpt-auto-generator`], #box[`systemd-boot`], #box[`systemd-stub`], #box[`systemd-sysupdate`], #box[`ukify`], #box[`mkosi`] ] #slide[ == What needs to be done? 1. build an image 2. install image to device 3. when booting: verify before running 4. when booting: measure what is run 5. when booting: unlock secrets if trusted 6. download new update && install 7. protect user data 8. customize the installation 9. reset system to factory defaults ] #slide[ == Building images: mkosi "MKOSI" see #link("https://media.ccc.de/v/all-systems-go-2023-190-mkosi-building-bespoke-operating-system-images")[ mkosi: Building Bespoke Operating System Images ] \@ ASG 2023 \ #link("https://media.ccc.de/v/all-systems-go-2023-191-systemd-repart-building-discoverable-disk-images")[ systemd-repart: Building Discoverable Disk Images ] \@ ASG 2023 ] #slide[ == Building images: why mkosi? - builds images from packages - developed in tandem with systemd // TODO?: insert example of systemd checking mkosi version - lightweight: heavy lifting is delegated to `systemd-repart`, `dnf`/`apt`/`pacman`/`apk`, `mkfs.btrfs`/`mkfs.ext4`/`mkfs.xfs`, `systemd-firstboot` and other tools - systemd-style configuration (ini files, drop-ins, match sections) - fully offline (no VMs, no loopback devices, plain file IO) - #text(size: 0.8em)[ Fedora/Debian/Kali/Ubuntu/PostmarketOS/ArchLinux/OpenSUSE/Mageia/CentOS/RHEL/RHEL-UBI/OpenMandriva/Rocky/Alma/AzureLinux ] ] #slide[ == Why mkosi?, part II mkosi-initrd: "build initrds from distro packages" We reuse the same code for the initrd, the OS, the exitrd. #v(1fr); #show: later DEMO! ] #slide[ == mkosi + OBS (Open Build Service \@ SUSE) mkosi is "local first" goal: IBaaS #show: later (image builds as a service) #show: later - customization via declarative config #show: later - multiple distros #show: later - multiple architectures #show: later - automated rebuilds #show: later - secure signing ] #slide[ == What needs to be done? 1. #strike(stroke:3pt + red)[build an image] 2. install image to device 3. when booting: verify before running 4. when booting: measure what is run 5. when booting: unlock secrets if trusted 6. download new update && install 7. protect user data 8. customize the installation 9. reset system to factory defaults ] #slide[ == How to install? - the system is "immutable" - … but `dd` is not enough! \ (GPT sector size, sector alignment, partition UUIDs) ] #slide[ == `systemd-repart` "online and offline partitioning based on declarative data" #show: later #align(center)[ #image("images/particle-os-repart.png", width: 110%) ] ] #slide[ == How to install? Our answer: copy partition _contents_, initialize the rest on the fly - `systemd-repart` with `CopyBlocks=auto`: copy read-only partitions, dm-verity and signatures \ initialize partition structure for system A/B updates, encrypted storage - `systemd-cryptsetup`: lock the encrypted storage to TPM ] #slide[ == How to install? Installers used to be very complex => better to do configuration in the live system // Push for configuration in the live system, after a reboot // (installer env is "special", // the running system needs to have a way to configure this too, // it's easier to blit the image, let end user do configuration, // makes factory-reset easier) #link("https://fedoraproject.org/wiki/Features/InitialExperience")[Features/InitialExperience] #h(1fr);(Fedora 19) \ #link("https://fedoraproject.org/wiki/Changes/ReduceInitialSetupRedundancy")[Changes/ReduceInitialSetupRedundancy] #h(1fr);(Fedora 28) \ #link("https://fedoraproject.org/wiki/Changes/Unified_KDE_OOBE")[Changes/Unified_KDE_OOBE] #h(1fr);(Fedora 44) \ `systemd-firstboot` #v(1fr); #show: later DEMO! ] #slide[ == How to install? Do we *need* to reboot? #link("https://cfp.all-systems-go.io/all-systems-go-2025/talk/QRJVL3/")[GNOME OS' prêt-à-booter image] \@ ASG2025 #image("images/gnome-os-style.png", width: 60%) // #image("images/gnome-os-disk-layout.png", width: 60%) (#link("https://www.youtube.com/watch?v=KLZJ9NhBPD")[youtube]) ] #slide[ == What needs to be done? 1. #strike(stroke:3pt + red)[build an image] 2. #strike(stroke:3pt + red)[install image to device] 3. when booting: verify before running 4. when booting: measure what is run 5. when booting: unlock secrets if trusted 6. download new update && install 7. protect user data 8. customize the installation 9. reset system to factory defaults ] #slide[ == Verify before running Traditional SecureBoot with shim which checks `vmlinuz` but not the initrd or the kernel command line is security theater. #show: later UKI — Unified Kernel Image \ => a single file with the kernel, initrd, command line, device tree, microcode \ signed as a whole #show: later `ukify` builds UKI images — `objcopy` on steroids ] #slide[ == SecureBoot and measured boot are available to anyone - real deployements would use a "proper" key - for development, we make our own #show: later `systemd-boot` supports autoenrollment - put files in `/loader/keys/auto/` (`KEK.auth`, `PK.auth`, `db.auth`, `dbx.auth`) - put machine in SB setup mode - set `secure-boot-enroll if-safe` - set `secure-boot-enroll-action reboot` ] #slide[ == SecureBoot vs. measured boot SecureBoot: hardware manufacturers embed their own key \ -> those keys sign the main Microsoft key \ -> that signs … many shims for Linux \ -> that trusts distro artifacts #show: later - a *very* wide tent // #show: later // The goal is to limit what can run on the machine at all. // #show: later // - can we enroll just our own key? (no) #show: later measured boot: TOFU, unlock secrets if state is good, establish trust #show: later tl;dr: generate own key, use `ukify` to build signed UKIs, enroll key on first boot ] #slide[ == What needs to be done? 1. #strike(stroke:3pt + red)[build an image] 2. #strike(stroke:3pt + red)[install image to device] 3. #strike(stroke:3pt + red)[when booting: verify before running] 4. when booting: measure what is run 5. when booting: unlock secrets if trusted 6. download new update && install 7. protect user data 8. customize the installation 9. reset system to factory defaults ] #slide[ == Measure what is run #show: later UEFI measures shim, \ shim measures the boot loader, \ the boot loader measures the UKI #show: later `systemd-pcrextend` `--machine-id`/`--file-system=`/`$phase` #show: later systemd devides runtime into phases: \ "enter-initrd", "leave-initrd", "sysinit", "ready" ] #slide[ == What needs to be done? 1. #strike(stroke:3pt + red)[build an image] 2. #strike(stroke:3pt + red)[install image to device] 3. #strike(stroke:3pt + red)[when booting: verify before running] 4. #strike(stroke:3pt + red)[when booting: measure what is run] 5. when booting: unlock secrets if trusted 6. download new update && install 7. protect user data 8. customize the installation 9. reset system to factory defaults ] #slide[ == Binding secrets to TPM PCR state #show: later TPM PCRs reflect system state: firmware hash, SecureBoot database hash, boot loader hash, UKI hash… \ `systemd-pcrextend` records more state #show: later two major approaches: #show: later 1. bind secrets to a TPM PCR digest #show: later 2. bind secrets to a public certificate hash signing a TPM PCR digest ] #slide[ == Binding secrets to TPM PCR state #show: later `systemd-measure` — precalculate PCR measurements for UKI & phase, sign a policy digest with a private key #show: later `ukify` — gather those signed digests and embed them in the UKI #show: later `systemd-cryptenroll` & `systemd-cryptsetup` — enroll and decrypt a disk via via public key infra \ `systemd-creds` — the same for credentials ] #slide[ == What needs to be done? 1. #strike(stroke:3pt + red)[build an image] 2. #strike(stroke:3pt + red)[install image to device] 3. #strike(stroke:3pt + red)[when booting: verify before running] 4. #strike(stroke:3pt + red)[when booting: measure what is run] 5. #strike(stroke:3pt + red)[when booting: unlock secrets if trusted] 6. download new update && install 7. protect user data 8. customize the installation 9. reset system to factory defaults ] #slide[ == Downloading images #show: later (`mkosi` to build the new image) #show: later `systemd-sysupdate` implements a downloader that does A/B updates \ `systemd-sysupdated` + `updatectl` a daemon+client service for updates ] #slide[ == What needs to be done? 1. #strike(stroke:3pt + red)[build an image] 2. #strike(stroke:3pt + red)[install image to device] 3. #strike(stroke:3pt + red)[when booting: verify before running] 4. #strike(stroke:3pt + red)[when booting: measure what is run] 5. #strike(stroke:3pt + red)[when booting: unlock secrets if trusted] 6. #strike(stroke:3pt + red)[download new update && install] 7. protect user data 8. customize the installation 9. reset system to factory defaults ] #slide[ == Protect user data `systemd-homed` manages home directories with per-user encryption `homectl`, `pam_systemd_home.so` ] #slide[ == What needs to be done? 1. #strike(stroke:3pt + red)[build an image] 2. #strike(stroke:3pt + red)[install image to device] 3. #strike(stroke:3pt + red)[when booting: verify before running] 4. #strike(stroke:3pt + red)[when booting: measure what is run] 5. #strike(stroke:3pt + red)[when booting: unlock secrets if trusted] 6. #strike(stroke:3pt + red)[download new update && install] 7. #strike(stroke:3pt + red)[protect user data] 8. customize the installation 9. reset system to factory defaults ] #slide[ == How to make changes to the shiny new _immutable_ system? #show: later First answer: don't \ #show: later flatpaks for apps \ #show: later custom images #show: later Second answer: verified extension mechanisms #show: later - initrd variants — multi-profile UKIs #show: later - "addons" → checked via SecureBoot db / shim #show: later - systemd-sysexts - systemd-confexts → checked via kernel keyring #show: later - credentials → encrypted via TPM|local key ] #slide[ == Credentials #set text(size: 0.8em) - `systemd.hostname` #h(1fr); #uncover("2-")[— set basic host information] - `firstboot.locale|keymap|timezone` #h(1fr); #uncover("2-")[— set basic host information] - `passwd.hashed-password.$user` / `passwd.shell.$user` #h(1fr); #uncover("3-")[— like `usermod`] - `fsck.mode` #h(1fr); #uncover("4-")[— new `/forcefsck` and `/fastboot`] - `quotacheck.mode` #h(1fr); #uncover("5-")[— new `/forcequotacheck`] - `home.add-signing-key.*` #h(1fr); #uncover("6-")[ — provision homed user record signing keys] - `home.register.*` #h(1fr); #uncover("7-")[ — create user automatically at boot] - `systemd.unit-dropin.*` #h(1fr); #uncover("8-")[ — tweak various units] - `vmm.notify_socket` #h(1fr); #uncover("9-")[ — establish notification channel to the host] - `ssh.listen` #h(1fr); #uncover("10-")[ — establish ssh access] - `ssh.authorized_keys.root` #h(1fr); #uncover("11-")[— backdoor sshd] - `ssh.ephemeral-authorized_keys-all` #h(1fr); #uncover("12-")[— backdoor sshd harder] #uncover("13-")[ #link("https://systemd.io/CREDENTIALS/") ] ] #slide[ == What needs to be done? 1. #strike(stroke:3pt + red)[build an image] 2. #strike(stroke:3pt + red)[install image to device] 3. #strike(stroke:3pt + red)[when booting: verify before running] 4. #strike(stroke:3pt + red)[when booting: measure what is run] 5. #strike(stroke:3pt + red)[when booting: unlock secrets if trusted] 6. #strike(stroke:3pt + red)[download new update && install] 7. #strike(stroke:3pt + red)[protect user data] 8. #strike(stroke:3pt + red)[customize the installation] 9. reset system to factory defaults ] #slide[ == Factory reset #show: later "all state, user data, and configuration is wiped" #show: later - "the system" is in `/usr` — everything else can go #show: later - `systemd-factory-reset` #show: later - `systemd-repart --factory-reset` #show: later - `systemd-tpm2-clear` ] #slide[ == What needs to be done? 1. #strike(stroke:3pt + red)[build an image] 2. #strike(stroke:3pt + red)[install image to device] 3. #strike(stroke:3pt + red)[when booting: verify before running] 4. #strike(stroke:3pt + red)[when booting: measure what is run] 5. #strike(stroke:3pt + red)[when booting: unlock secrets if trusted] 6. #strike(stroke:3pt + red)[download new update && install] 7. #strike(stroke:3pt + red)[protect user data] 8. #strike(stroke:3pt + red)[customize the installation] 9. #strike(stroke:3pt + red)[reset system to factory defaults] questions? ]