Building a Diskless Linux Boot System with ZFS, iSCSI, and PXE

For many developers, the tension between a stable gaming or primary OS and a flexible development environment is a constant struggle. Whether it is avoiding the "pollution" of a Windows installation with various toolchains or preventing Windows Updates from overwriting GRUB entries, the desire for an isolated, yet high-performance environment is common.

While USB drives are a frequent go-to for live Linux environments, they are easily misplaced or wiped. A more robust, professional-grade solution is diskless booting. By leveraging PXE (Preboot Execution Environment), iSCSI, and ZFS, it is possible to boot a full Linux installation from a remote NAS, treating a network-attached block device as if it were a local hard drive.

The Architecture

This setup relies on a client-server model where the server provides the boot infrastructure and the storage, and the client (the gaming PC or workstation) provides the compute power.

The Server Stack

• Netboot.xyz: A versatile tool that provides a menu-driven interface for booting various operating systems and installers over the network.
• TFTP (Trivial File Transfer Protocol): Used to deliver the initial bootloader binaries to the client.
• DNSMasq: Acts as the DHCP server, directing the client to the TFTP server based on its architecture (BIOS vs. UEFI).
• ZFS ZVol: A block device created within a ZFS pool, providing the raw storage for the virtual disk.
• iSCSI Target (targetcli): Exports the ZFS ZVol as a block device over the network, allowing the client to mount it as a local SCSI disk.

Implementation Guide

1. Setting Up the Boot Infrastructure

The first step is installing netboot.xyz on a Debian-based server (e.g., Proxmox). This involves installing a web server (Apache), TFTP, and Ansible to automate the deployment of the netboot assets.

To support custom boot targets, you must create an .ipxe script. This script tells the client to attempt a sanboot (SCSI network boot) using the iSCSI server's IP and the specific Target IQN. If the boot fails (which it will during the initial installation), the script is configured to fall back to the Debian installer.

2. Network Configuration

Your DHCP server (e.g., a router running DNSMasq) must be configured to handle different client types. A typical configuration involves:

• BIOS Clients: Directed to netboot.xyz-undionly.kpxe.
• UEFI Clients: Directed to netboot.xyz-snp.efi.
• iPXE Clients: Once the initial bootloader is running, subsequent requests are redirected to the menu.ipxe hosted on the web server.

3. Storage and iSCSI Configuration

Using ZFS, you can create a ZVol to act as the disk image:

zpool create tank /dev/disk/by-id/${DISK_ID}
zfs create -V 32G tank/debian-disk-12700k

This ZVol is then exported via targetcli. The critical steps here include creating the backstore, defining the iSCSI target, and setting up Access Control Lists (ACLs) with mutual authentication (username/password for both initiator and target) to ensure the network disk isn't exposed to unauthorized clients.

4. Installing the OS

Installing Debian onto an iSCSI target requires a few manual interventions during the installation process:

1. Initiator Configuration: Since the installer may not automatically detect the iSCSI target, you must switch to a TTY (Super+F2) and edit /etc/iscsi/initiatorname.iscsi to match the IQN configured on the server.
2. Restarting iscsid: The iscsid daemon must be restarted to recognize the new initiator name.
3. Target Login: Back in the installer, you can then provide the portal address and authentication credentials to map the remote ZVol as a local disk.

Once the installation is complete and GRUB is installed to the iSCSI disk, the system can be rebooted. The sanboot command in the iPXE script will now find a bootable OS and launch it.

Technical Considerations and Trade-offs

Performance and Latency

As noted in the implementation, network booting is inherently slower than local NVMe storage. While RAM disks can mitigate some of this for the OS, the underlying network speed is the primary bottleneck.

Community insights suggest that for production-grade diskless setups, 10Gbps Ethernet is almost mandatory. Furthermore, iSCSI can be sensitive to network congestion. To optimize performance, it is recommended to:

• Use a dedicated VLAN for iSCSI traffic.
• Implement Quality of Service (QoS) settings on switches to prioritize storage traffic.
• Explore NVMe over TCP as a modern, higher-performance alternative to iSCSI.

Stability and Maintenance

One of the primary advantages of this setup is the centralization of the OS. Because the bootloader (GRUB) resides on the remote drive, local Windows updates cannot break the Linux boot sequence.

However, some users suggest alternatives to GRUB, such as rEFInd, which can simplify EFI management by using a single text configuration file, reducing the complexity of maintaining UEFI entries across kernel updates.

Comparison with Other Methods

While iSCSI provides a block-level interface, other developers prefer NFS (Network File System) for diskless booting. NFS is often simpler to set up but operates at the file level. iSCSI is generally preferred when the guest OS needs full control over the disk partitioning and filesystem (such as installing a filesystem that the server doesn't natively support).

"I've been running iSCSI volumes for OrangePI boards which had only SD card support for years. Performance was much better on Gigabit Ethernet vs the SD card."

Summary Table: Local vs. Diskless Boot

Feature	Local NVMe	Diskless (iSCSI/PXE)
Boot Speed	Ultra Fast	Network Dependent
Isolation	Shared Disk/Partition	Complete Physical Isolation
Maintenance	Manual Partitioning	Centralized Image Management
Risk	Bootloader Overwrites	Network Outage = System Down
Flexibility	Fixed Disk Size	Dynamic ZVol Resizing