This could be an interesting setup for booting off a NAS like Synology or QNAP. I haven't really used iSCSI, it's intimidating how much prep this takes...
iSCSI seems intentionally obscure. One of the improvements I made to NBD was invent a simple, standardized URI format so that you can specify servers easily, eg:
Nice. I'm extra fond of ZFS backed network root filesystem, because it lets you put an OS on ZFS without needing to deal with ZFS support in that OS. (One of these days I want to try OpenBSD with its root on NFS on ZFS, either from Linux or FreeBSD.)
I don’t have direct experience, but when I looked into it my takeaway that NBD was unable to reliably deal with network interruptions as well as iscsi.
> UEFI fixes that to some extent, but it’s a pain to maintain the UEFI entries manually and change them every time the kernel updates.
… you don't have to update the UEFI entries every time the kernel updates. (I guess you might if you do like a kernel w/ CONFIG_EFI_STUB, and you place the new kernel under a different filename than what the UEFI boot entry point to then you might … but I was under the impression that that'd be kind of an unusual setup, and I thought most of us booting w/ EFI were doing so with Grub.)
I've done a lot of headless/diskless stuff. I haven't done much for years, because my NAS only has gigabit Ethernet ports. I can cascade them and get four Gbps downstream, but it's still painful.
I have recently upgraded my house to 10Gbps Ethernet, with only one room still stuck at gigabit, and unfortunately, it's my main office. I'm working on getting the drop there now (literally, just taking a break here).
Even once I'm done, accessing an iSCSI drive over 10GbE will be 4-8 times slower than a local NVMe drive, but it will sure be a lot better than it was!
Ideally, I could run VMs on the NAS and have great performance, but that's another hardware upgrade...
Really I wonder how this turns out to be diskless while you're clearly accessing a disk/drive over the network. Shouldn't we refer to this as network boot?
Pretty cool! You could also boot into an ephemeral minimal initrd that displays a selection menu instead of doing it in iPXE. That would grab the new kernel and initrd from the network and kexecs it without reboot.
I used similar ipxe setup for robotic cluster - every robot booted from the same thing, then kubernetes managed the containe orchestration. it was fun.
NFS diskless was easier for me to setup when I was doing it.
THe caveat was, you needed readonly root, so that meant freezing the OS, anything that needed changing was either stored in a ram disk (that you need to setup) or a per host nfs area (kinda like overlayfs, but not)
You can download the rootfs, extract it to a ramdisk, and just run in memory. This is fast for everything. Unfortunately, memory just got super expensive. Fortunately, Linux requires ~no memory to do many useful things.
You might find it worth upgrading to 10gbps if you continue to go down this road. The Mikrotik CRS-309 has served me well, and a couple Intel X520-DA2s. I believe those NICs can do iSCSI natively, and pass the session to the operating system with iBFT.
SFP28 might be cheap enough now too, I'm not sure...
This could be an interesting setup for booting off a NAS like Synology or QNAP. I haven't really used iSCSI, it's intimidating how much prep this takes...
iSCSI seems intentionally obscure. One of the improvements I made to NBD was invent a simple, standardized URI format so that you can specify servers easily, eg:
https://github.com/NetworkBlockDevice/nbd/blob/master/doc/ur...The 'target' moves slow so once you learn it, it all stays relevant forever.
... And it's very, very fun.
Nice. I'm extra fond of ZFS backed network root filesystem, because it lets you put an OS on ZFS without needing to deal with ZFS support in that OS. (One of these days I want to try OpenBSD with its root on NFS on ZFS, either from Linux or FreeBSD.)
Does anyone have an opinion on iSCSI vs NBD?
I don’t have direct experience, but when I looked into it my takeaway that NBD was unable to reliably deal with network interruptions as well as iscsi.
https://forums.gentoo.org/viewtopic.php?p=4895771&sid=f9b7ac...
https://github.com/NetworkBlockDevice/nbd/issues/93
Whether that’s the case with the latest version, I don’t know, but it’s something you might test if you choose to try it.
You might like https://smolbsd.org/
Well yes, I do like that:), but I don't see the connection to this thread?
> UEFI fixes that to some extent, but it’s a pain to maintain the UEFI entries manually and change them every time the kernel updates.
… you don't have to update the UEFI entries every time the kernel updates. (I guess you might if you do like a kernel w/ CONFIG_EFI_STUB, and you place the new kernel under a different filename than what the UEFI boot entry point to then you might … but I was under the impression that that'd be kind of an unusual setup, and I thought most of us booting w/ EFI were doing so with Grub.)
Even if you do CONFIG_EFI_STUB, there should be a post-update hook to automatically call efibootmgr.
or just copy the latest kernel to something like /vmlinux and /initramfs
I've done a lot of headless/diskless stuff. I haven't done much for years, because my NAS only has gigabit Ethernet ports. I can cascade them and get four Gbps downstream, but it's still painful.
I have recently upgraded my house to 10Gbps Ethernet, with only one room still stuck at gigabit, and unfortunately, it's my main office. I'm working on getting the drop there now (literally, just taking a break here).
Even once I'm done, accessing an iSCSI drive over 10GbE will be 4-8 times slower than a local NVMe drive, but it will sure be a lot better than it was!
Ideally, I could run VMs on the NAS and have great performance, but that's another hardware upgrade...
Really I wonder how this turns out to be diskless while you're clearly accessing a disk/drive over the network. Shouldn't we refer to this as network boot?
It's diskless from the point of view of the device being booted.
something worth mentioning here is that iSCSI is quite unhappy on congested networks or packet loss caused by incast traffic.
to make this actually work well, consider modifying your switches QoS settings to carve out a priority VLAN for iSCSI traffic
or a north-south/east-west architecture, so there's an entirely separate network just for iSCSI. Control plane vs data plane.
Pretty cool! You could also boot into an ephemeral minimal initrd that displays a selection menu instead of doing it in iPXE. That would grab the new kernel and initrd from the network and kexecs it without reboot.
I used similar ipxe setup for robotic cluster - every robot booted from the same thing, then kubernetes managed the containe orchestration. it was fun.
NFS diskless is the more common approach I've used but this is very cool.
NFS diskless was easier for me to setup when I was doing it.
THe caveat was, you needed readonly root, so that meant freezing the OS, anything that needed changing was either stored in a ram disk (that you need to setup) or a per host nfs area (kinda like overlayfs, but not)
Why would you need a read-only root? Do you mean to share across multiple machines?
When I tried root-on-nfs I had a lot of issues. The Redhat and Arch package managers don't seem to like it (presumably a sqlite thing?).
You can download the rootfs, extract it to a ramdisk, and just run in memory. This is fast for everything. Unfortunately, memory just got super expensive. Fortunately, Linux requires ~no memory to do many useful things.
You might find it worth upgrading to 10gbps if you continue to go down this road. The Mikrotik CRS-309 has served me well, and a couple Intel X520-DA2s. I believe those NICs can do iSCSI natively, and pass the session to the operating system with iBFT.
SFP28 might be cheap enough now too, I'm not sure...
what i want to play with is rdma and having a bcache block device with the remote as a backing and a small local nvme as a write-through cache
I would probably recommend to look into NVMe over TCP over iSCSI, especially for fast NVMe drives.