Add network boot testing and monitoring documentation

Enhance the NETWORK-BOOT-LXC.md documentation with detailed steps for testing network boot functionality, including prerequisites, expected outcomes, and quick testing methods. Introduce a new section on monitoring network boot status on the LXC, outlining commands to check DHCP leases, dnsmasq status, and registered devices. Update the initramfs scripts to support a rescue mode for devices stuck in network-only boot, allowing users to change boot order settings. Include a new rescue script for eMMC management in the build process.

emmc-provisioning/network-boot-initramfs/README.md

@@ -65,6 +65,10 @@ provisioning_server=http://10.20.50.1:5000
 
 The init script reads `provisioning_server=` from `/proc/cmdline` and exports `PROVISIONING_SERVER` for the client.
 
+### Rescue mode (stuck in network-only boot)
+
+If the device has **BOOT_ORDER=0x2** (network only), it never boots from eMMC. To get a shell and change boot order using the eMMC’s **rpi-eeprom-config**, add **provisioning_rescue=1** to the kernel cmdline (e.g. in the TFTP-served `cmdline.txt` for that device). The initramfs will then start an interactive shell instead of the provisioning client. Run **/rescue-eeprom.sh** to mount eMMC and chroot to run `rpi-eeprom-config --edit`; set `BOOT_ORDER=0x1` or `0x21`, save, then `reboot`. See **docs/NETWORK-BOOT-TROUBLESHOOTING.md** (“Stuck in network-only boot”) for full steps.
+
 ## Flow summary
 
 1. Pi does DHCP → gets IP and TFTP server (e.g. 10.20.50.1).

emmc-provisioning/network-boot-initramfs/build.sh

@@ -15,10 +15,11 @@ trap "rm -rf $BUILD_DIR" EXIT
 echo "Build dir: $BUILD_DIR"
 
 # Layout: /init, /provisioning-client.sh, /bin/busybox, /bin/sh, /usr/bin/curl, /lib/*.so
-mkdir -p "$BUILD_DIR"/{bin,usr/bin,proc,sys,dev,dev/pts,lib}
+mkdir -p "$BUILD_DIR"/{bin,usr/bin,proc,sys,dev,dev/pts,lib,mnt}
 cp "$SCRIPT_DIR/init" "$BUILD_DIR/init"
 cp "$SCRIPT_DIR/provisioning-client.sh" "$BUILD_DIR/provisioning-client.sh"
-chmod +x "$BUILD_DIR/init" "$BUILD_DIR/provisioning-client.sh"
+cp "$SCRIPT_DIR/rescue-eeprom.sh" "$BUILD_DIR/rescue-eeprom.sh"
+chmod +x "$BUILD_DIR/init" "$BUILD_DIR/provisioning-client.sh" "$BUILD_DIR/rescue-eeprom.sh"
 
 ARCH=$(uname -m 2>/dev/null)
 if [ "$ARCH" = "aarch64" ] || [ "$ARCH" = "arm64" ] || [ "$ARCH" = "armv8l" ]; then

emmc-provisioning/network-boot-initramfs/init

@@ -21,15 +21,24 @@ if ! ip addr show | grep -q 'inet .* scope global'; then
   udhcpc -f -q -i eth0 -n 2>/dev/null || true
 fi
 
-# Allow kernel cmdline to override: provisioning_server=http://10.20.50.1:5000
+# Allow kernel cmdline to override: provisioning_server=... and rescue mode
+RESCUE=0
 for arg in $(cat /proc/cmdline); do
   case "$arg" in
     provisioning_server=*) export PROVISIONING_SERVER="${arg#*=}"; ;;
+    provisioning_rescue=1) RESCUE=1; ;;
   esac
 done
 PROVISIONING_SERVER="${PROVISIONING_SERVER:-http://10.20.50.1:5000}"
 export PROVISIONING_SERVER
 
+if [ "$RESCUE" -eq 1 ]; then
+  echo "=== RESCUE MODE (provisioning_rescue=1) ==="
+  echo "Run /rescue-eeprom.sh to mount eMMC and change boot order (rpi-eeprom-config), then reboot."
+  echo "Or run /bin/sh for a shell."
+  exec /bin/sh -i
+fi
+
 echo "Provisioning server: $PROVISIONING_SERVER"
 echo "Running provisioning client..."
 exec /bin/sh /provisioning-client.sh

emmc-provisioning/network-boot-initramfs/rescue-eeprom.sh

@@ -0,0 +1,36 @@
+#!/bin/sh
+# Rescue script: mount eMMC root and chroot to run rpi-eeprom-config.
+# Use this when stuck in network-only boot (BOOT_ORDER=0x2) to set BOOT_ORDER=0x1 or 0x21.
+# Run from the initramfs rescue shell (after booting with provisioning_rescue=1 in cmdline).
+
+set -e
+ROOT="/mnt/emmc"
+BOOT="$ROOT/boot/firmware"
+[ -d "$ROOT/boot" ] && [ ! -d "$BOOT" ] && BOOT="$ROOT/boot"
+
+echo "=== Mounting eMMC for EEPROM config ==="
+# CM4 / reTerminal: eMMC is usually mmcblk0, p1=boot (FAT), p2=root (ext4)
+if [ ! -b /dev/mmcblk0p2 ]; then
+  echo "No /dev/mmcblk0p2 found. Try: ls /dev/mmcblk*"
+  exit 1
+fi
+
+mkdir -p "$ROOT"
+mount /dev/mmcblk0p2 "$ROOT" || { echo "Mount root failed"; exit 1; }
+if [ -b /dev/mmcblk0p1 ]; then
+  mkdir -p "$BOOT"
+  mount /dev/mmcblk0p1 "$BOOT" 2>/dev/null || true
+fi
+mount -t proc none "$ROOT/proc"
+mount -t sysfs none "$ROOT/sys"
+mount --bind /dev "$ROOT/dev"
+mount --bind /dev/pts "$ROOT/dev/pts" 2>/dev/null || true
+
+if [ -x "$ROOT/usr/bin/rpi-eeprom-config" ]; then
+  echo "Chroot to eMMC and run: rpi-eeprom-config --edit"
+  echo "Set BOOT_ORDER=0x1 (eMMC only) or 0x21 (network first, then eMMC), save, then exit and run: reboot"
+  chroot "$ROOT" /usr/bin/rpi-eeprom-config --edit
+else
+  echo "rpi-eeprom-config not found in eMMC. Chrooting anyway; run: apt install rpi-eeprom && rpi-eeprom-config --edit"
+  chroot "$ROOT" /bin/sh -i
+fi