OrangePi Zero Netbooting

When you are hacking an OrangePi image or you have a few Oranges in your home, some of them not very acessible, booting them off your NAS or your desktop computer where you build the images could be a big boost to your developer experience. This article sums up what you have to do to make your Orange boot from network. I will not explain the netboot basics, there are a lot of sources about it over the Internet.

This guide is probably applicable to other small ARM computers also. Try it yourself. At least I can say it works with Nanopi Neo also (because it's almost identical to OrangePi Zero).

Even when netbooting, we will use bootloader on an SD card. It's possible to place it in the eMMC but I will not cover it here because not every Orange has one.

WARNING: This guide is intended for people who know what they are doing. You can destroy all your computer data just by using wrong /dev/sdX device. Proceed at your own risk.

Before You Start

I suppose you have an already working Armbian system to copy the needed files from. Of course you can build them yourself, copy them from other OSes or use Buildroot to completely automate the whole build process but let's make this your homework, I will keep the guide as short and straightforward as possible.

All the things you will need:

  1. OrangePi already running Armbian
  2. Some way to connect to Orange's serial console - this is absolutelly essential. With netbooting, many things can go wrong and you really need to know what's happening
  3. U-Boot image u-boot-sunxi-with-spl.bin - of course you should compile U-Boot yourself. But if you are lazy, you can just download this binary from me.
  4. DHCP server which let you set the serveraddress DHCP option. All but the cheapest routers should be ok. If you don't have one, you will have to (temporarily?) run DHCP server on your desktop computer.
  5. TFTP server - i.e. some NAS, intelligent router or again: just your computer with proper software installed and configured.
  6. NFS server - dtto

OrangePi Boot Sequence in a Nutshell

There's a (very limited) boot ROM in Orange which just loads something called "SPL (loader)" from a fixed location on the SD card (or other places I do not cover here).

The prominenent SPL implementation is U-Boot. It's a little bit smarter then boot ROM, can access different devices like USB or Ethernet, has a built-in shell and can interpret a simple sh-like scripts. The main purpose of U-Boot is similar to Grub or other bootloaders:

  1. to load a kernel image into the memory
  2. (optionally) to load a initial ramdisk image into the memory
  3. to start the kernel with given kernel parameters

If you are developing a simple non-persistent one-purpose OS, the ramdisk can contain your whole OS. But that is probably not the case. You probably want to boot something like Armbian. In this case, the ramdisk contains only the basic initialization scripts and kernel modules needed to mount the rest of the system (a.k.a. "rootfs").

Because in this stage you already have a (minimal) Linux system booted, you have many choices where to mount rootfs from. We will go the simplest way possible: we will use rootfs on NFS, mounted read-write. You can experiment with other possibilities like RO NFS + writtable overlay, image downloaded using HTTP... It's up to you and your use case. As you have a standard Linux booted in this stage, it's not very hard to implement even very crazy solutions.

So to conclude, the boot sequence (that we will use) is:

  1. boot ROM loads U-Boot from the SD card
  2. U-Boot initializes network interface and obtains IP address and the adress of the TFTP server from DHCP server
  3. U-Boot finds the configuration file on the TFTP server
  4. U-Boot loads kernel and initial ramdisk from the TFTP server (from paths configured in the file loaded in the previous step)
  5. U-Boot starts the kernel with the parameters given in the same configuration file
  6. kernel boots and runs initscripts from the initial ramdisk
  7. initscripts mount the root from NFS server and since then the boot continues as usual

U-Boot

At first, we will create an SD card with the bootloader hidden somewhere of an unpartitioned space and (optionally) one extfs partition where we will put the U-Boot script files (you can use FAT if you wish).

It's best to start with a clean SD card. Zero-out at least a first few megabytes of the card. You know how. If you don't know what I'm talking about, do not continue reading or you could harm yourself, your computer or some uninvolved cat. You have been warned!

don't do any harm!

Or even better, use wipefs to remove all metadata of all preexisting partitions and the partition table itself and then zero-out the beginning of the SD card.

We will start by creating a partition table. I'm using MBR here but you can also use GPT as long as you respect that the first partition start is at >=1MB as described in the SD card layout.

## your SD card device
SD=/dev/sdX

parted $SD mklabel msdos

Let's write the bootloader into the right place:

dd if=u-boot-sunxi-with-spl.bin of=$SD bs=1024 seek=8

Now you have a basic bootable SD card. All magic have taken place and since now we will work with good ol' files only.

(optional) U-Boot scripts

In Armbian (and various other distros), U-Boot is configured with a scripts placed in /boot. It's not needed for netbooting but if you want it for some reason, you must create a partition where you can place the scripts. As an example, we will copy the script files from an already running Armbian.

parted $SD mkpart primary ext4 1MiB 8MiB
mkfs.ext4 ${SD}1
mount ${SD}1 /mnt
scp root@YOUR_OTHER_PI:/boot/\{armbianEnv.txt,boot.cmd,boot.scr\} /mnt
umount /mnt

Please note: the boot.cmd file is here just for future refercence. In fact, only boot.scr is used by U-Boot and it's a compiled version of boot.cmd. See e.g. chapter "boot.scr support" here on how to prepare a custom script.

DHCP server

In contrast with a classic x86 PXE booting, you don't need to set filename option in your DHCP server (we already have bootloader loaded). You only need to advertise your TFTP server IP using serveraddress option. Consult your DHCP server manual on how to set this option.

NFS server

I will not explain how to create a NFS server, consult your OS manual. Let's just say that you should create a new (RW) NFS export and copy files from running Armbian (or other OS of your choice) to it. Something like this:

## where to put your Orange NFS root
nfs_root=/srv/nfs/orange

rsync -avx root@YOUR_OTHER_PI:/ $nfs_root/

Now you should have Armbian root at $nfs_root. Eg. the file $nfs_root/bin/dash should exist.

TFTP server

U-Boot netbooting resembles pxelinux a bit. It looks for its configuration file at different paths on the TFTP server - from the most specific one like /pxe/pxelinux.cfg/01-02-03-04-05-06-07 (the number being it's MAC address) upto the most general /pxe/pxelinux.cfg/default. This mechanism allows you to create a special configuration for one specific device or general configuration for a whole family of them. You can see all the paths tried on a serial console.

Ok, we will use a path that every Orange will look at and that will not break our x86 machines netboot - /pxe/pxelinux.cfg/default-arm-sunxi. Let's create the configuration file:

## your TFTP root
tftp_root=/var/tftpboot

mkdir -p $tftp_root/pxe/pxelinux.cfg

cat >$tftp_root/pxe/pxelinux.cfg/default-arm-sunxi <<EOF
default test

label test
  kernel arm/4.19.13-sunxi/vmlinuz-4.19.13-sunxi
  initrd arm/4.19.13-sunxi/uInitrd-4.19.13-sunxi
  fdt arm/4.19.13-sunxi/sun8i-h2-plus-orangepi-zero.dtb
  append root=/dev/nfs nfsroot=1.2.3.4:/srv/nfs/orange console=ttyS0,115200 hdmi.audio=EDID:0 panic=10 consoleblank=0 loglevel=1 sunxi_ve_mem_reserve=0 sunxi_g2d_mem_reserve=0 sunxi_fb_mem_reserve=16 cgroup_enable=memory swapaccount=1
EOF

Looks familiar, doesn't it? Good ol' days of LILO without any of the Grub2 insanities! Of course you must put valid IP adress of your NFS server and export path into nfsroot variable.

Now we will copy the three mentioned files from our running Orange:

mkdir -p $tftp_root/pxe/arm/4.19.13-sunxi

scp
root@YOUR_OTHER_PI:/boot/\{vmlinuz-4.19.13-sunxi,uInitrd-4.19.13-sunxi,dtb-4.9.13-sunxi/sun8i-h2-plus-orangepi-zero.dtb\} \
  $tftp_root/pxe/arm/4.19.13-sunxi/

Summary

And that's it! Now you should have everything in place. Put your SD card in your Orange, connect the ethernet, serial console and try your new, netbooting Orange!

If anything goes wrong (which is not unusual), look again at the Boot Sequence Overview and try to figure out which step has failed.

After you have successfully booted your Orange for the first time, you can experiment with other rootfs options, you can prepare your own ramdisk or even build your own fully customized OS with Buildroot. And this all without touching your SD card even once! You can probably fix it in its slot with a bubble gum! (or better try to figure out how to place the bootloader into eMMC ;) )

Of course there are also downsides:

  1. Mounting rootfs over NFS requires support in the initial ramdisk's scripts. I've seen distros which simply did not work when booted off NFS. You must try your beloved distro yourself. Suprisingly, systemd-based distro should work out of the box because the initialization logic is built-in into the systemd (they will probably break in some other creepy way but let's not start a flamewar...)
  2. Your kernel, ramdisk and DTB is now loaded from other location (TFTP) than where your root resides (NFS). This can cause serious problems after OS upgrades. There's no silver bullet for this. At least try to keep TFTP files and $rootfs/boot/* files in sync. And don't blame me if things get broken after upgrade. это жизнь.
comments powered by Disqus