Enhanced PXE booting with iPXE

While doing more reading on PXE as a followup to Setting up a PXE environment for OS installations, I ran into iPXE, which has some interesting features which can simplify PXE booting. The main feature that caught my interest was HTTP support - meaning it can fetch various modules, kernels, etc from a web server, which in general is much more customizable and configurable than stock TFTP servers, and also much faster.

It took a while to figure out where it all fits into a boot stack, I thought I'd share what I've roughly figured out and gotten to work.

Stock PXE - a review

This diagram shows where we ended previously previously with installing Ubuntu over a network:

Stock PXE stack

The PXE stack would obtain from a DHCP server the IP address of a TFTP server and the name of a Network Boot Program (NBP) such as pxelinux.0

pxelinux.0 would then fetch a config file, and the config file fetched a menu module. Depending on what was selected from the menu, a Linux kernel and initrd could be fetched and booted, and then the Ubuntu installer presumably in the initrd would fetch additional install packages from an HTTP server.

Let's see what iPXE can do for us.

iPXE - replacement ROM

I didn't actually try this because it doesn't seem practical on a large scale, but you can on many NICs replace the stock PXE ROM with iPXE, giving you an arrangement like this:

iPXE ROM

A few things to note here:

  • We can now use HTTP to load our NBPs - it's just a matter of changing the DHCP config to say something like: filename "http://10.0.0.1/whatever"; (the next-server clause in the DHCP config doesn't really matter now)

  • We can use the same NBPs such as PXELINUX that worked in the stock PXE setup.

  • There's now an interactive console that lets you manually change settings and load modules including Linux kernels and ramdisks directly - so iPXE is also a bootloader itself and you don't necessarily need something like PXELINUX.

  • The same commands you can use interactively can be saved in script files, so essentially it's like having a NBP that's a text file instead of having to be a compiled binary.

As I mentioned, I don't think it's practical to go around flashing various NICs and motherboards, but there's another way to use iPXE....

Booting iPXE with PXE

There's a version of iPXE that can be chainloaded, so that you're using the stock PXE on your machine to bring in the enhanced iPXE stack. Here's the big picture first:

PXE->iPXE

One of the APIs a PXE stack makes available is UNDI (Universal Network Device Interface), giving a simple device driver for the NIC that an NBP can use. We can configure DHCP to load undionly.kpxe as our initial NBP, it will use the UNDI part of the stock PXE ROM (so it doesn't need to be configured for any particular NIC), and it will start the PXE cycle again (querying the DHCP server for an IP address, the name of an NBP, etc...), but with all the extra features of iPXE available.

We need to use a bit of logic in the DHCP config so that the iPXE stack isn't also told to load undionly.kpxe (basically causing a loop). This can be done in ISC dhcpd with an if statement:

if exists user-class and option user-class = "iPXE" {
    filename "http://10.0.0.1/pxelinux.0";
    } 
else {
    filename "undionly.kpxe";
    }

So that plain PXE is told to (TFTP) load undionly.kpxe and iPXE is told to load http://10.0.0.1/pxelinux.0

One cool thing about this setup is that we only need to make one file available through TFTP, undionly.kpxe. Everything else can be served up by an HTTP server like Nginx. This is great because HTTP servers are generally more configurable than a stock TFTP server, plus you aren't limited to serving up static files - some of these requests from iPXE could be handled by CGI scripts or other webapps for more of a dynamic behavior.

PXELINUX on iPXE

If iPXE is the active PXE stack, then PXELINUX gains the ability to use HTTP urls for module names, for example, an Ubuntu install might point directly into a mounted ISO image shared on the web like:

LABEL ubuntu-11.04-server-amd64-install             
    MENU LABEL Ubuntu 11.04 Server AMD64 Install
    kernel http://10.0.0.1/ubuntu-11.04-server-amd64/install/netboot/ubuntu-installer/amd64/linux
    append vga=788 initrd=http://10.0.0.1/ubuntu-11.04-server-amd64/install/netboot/ubuntu-installer/amd64/initrd.gz url=http://10.0.0.1/ubuntu-11.04-server-amd64.txt

Conclusion

So, drop iPXE's undionly.kpxe into your TFTP server, configure DHCP to give it out as an NBP only if a non-iPXE stack is asking, and you can use HTTP for everything else.