From 10c200f9947f56242e22ab2c334ad7456c734e0a Mon Sep 17 00:00:00 2001 From: nearxos Date: Wed, 4 Mar 2026 19:28:53 +0200 Subject: [PATCH] Enhance network boot provisioning with support for extra LAN IPs and VLAN configuration Update documentation and scripts to include configuration for extra LAN IPs on eth1 and VLAN interface eth1.40, allowing the LXC to serve multiple subnets and provide NAT for internet access. Modify nftables NAT configuration to accommodate these changes and ensure proper DHCP and DNS setup on eth1. This improves the overall network boot functionality and user experience for the CM4 eMMC provisioning service. --- emmc-provisioning/docs/NETWORK-BOOT-LXC.md | 15 ++ emmc-provisioning/lxc/70-cm4-extra-lan | 17 ++ emmc-provisioning/lxc/README.md | 4 +- emmc-provisioning/lxc/nft-nat-lan.conf | 8 +- .../scripts/setup-network-boot-on-lxc.sh | 154 +++++++++++------- 5 files changed, 135 insertions(+), 63 deletions(-) create mode 100644 emmc-provisioning/lxc/70-cm4-extra-lan diff --git a/emmc-provisioning/docs/NETWORK-BOOT-LXC.md b/emmc-provisioning/docs/NETWORK-BOOT-LXC.md index fb2dc37..d1b6790 100644 --- a/emmc-provisioning/docs/NETWORK-BOOT-LXC.md +++ b/emmc-provisioning/docs/NETWORK-BOOT-LXC.md @@ -88,6 +88,21 @@ So changing `DEPLOY_LXC_LAN_SUBNET` and **re-running the deploy script** updates Then run **toggle enable** again if you use network boot: `ssh root@ /opt/cm4-provisioning/toggle-network-boot-dhcp.sh enable` +### Extra LAN IPs and VLAN (eth1.40) + +The setup script also configures **extra IPs on eth1** and a **VLAN interface** so the LXC can serve multiple subnets and provide internet (NAT) to all of them: + +| Address / interface | Purpose | +|--------------------|--------| +| **Primary** (e.g. `10.20.50.1/24`) | Set at deploy; used by dnsmasq for DHCP/TFTP/DNS | +| **192.168.30.1/24** | Extra LAN on eth1 | +| **192.168.127.1/24** | Extra LAN on eth1 | +| **eth1.40** **192.168.0.1/24** | VLAN 40 on eth1 | + +- Config is persisted in **`/etc/network/interfaces.d/70-cm4-extra-lan`** (installed when you run `setup-network-boot-on-lxc.sh`). +- **NAT** is applied to all four: primary LAN, 192.168.30.0/24, 192.168.127.0/24, and 192.168.0.0/24 (VLAN 40), so clients on any of these subnets get internet via eth0. +- For **VLAN 40** to receive tagged traffic, the Proxmox bridge connected to eth1 (e.g. vmbr1) must either be a trunk that passes VLAN 40, or you use a dedicated bridge (e.g. vmbr1.40) and attach the container to it as a second interface; the script creates eth1.40 inside the LXC for the in-container VLAN case. + ## After setup: reTerminal network boot 1. Set the reTerminal **boot order** to try eMMC first, then network (e.g. `BOOT_ORDER=0xf21`): use the dashboard **Update EEPROM** when the device is connected via USB boot, or set manually (usbboot recovery / `rpi-eeprom-config` on device). Not set by first-boot. diff --git a/emmc-provisioning/lxc/70-cm4-extra-lan b/emmc-provisioning/lxc/70-cm4-extra-lan new file mode 100644 index 0000000..16a36cd --- /dev/null +++ b/emmc-provisioning/lxc/70-cm4-extra-lan @@ -0,0 +1,17 @@ +# Extra LAN IPs on eth1 and VLAN 40 on eth1. +# Primary eth1 address is set by Proxmox/deploy (used by dnsmasq DHCP). +# Installed by setup-network-boot-on-lxc.sh; ensure /etc/network/interfaces +# includes: source-directory /etc/network/interfaces.d + +# Secondary addresses on eth1 (192.168.30.1, 192.168.127.1) +iface eth1 inet static + address 192.168.30.1/24 +iface eth1 inet static + address 192.168.127.1/24 + +# VLAN 40 on eth1 — 192.168.0.0/24 (gateway 192.168.0.1) +# Requires: apt install vlan; host bridge must pass VLAN 40 if using tagged uplink +auto eth1.40 +iface eth1.40 inet static + address 192.168.0.1/24 + vlan-raw-device eth1 diff --git a/emmc-provisioning/lxc/README.md b/emmc-provisioning/lxc/README.md index f6c9870..ed0f039 100644 --- a/emmc-provisioning/lxc/README.md +++ b/emmc-provisioning/lxc/README.md @@ -7,7 +7,9 @@ Config files for the **provisioning LXC** when using **eth1** as a provisioning | File | Purpose | |------|--------| | **dnsmasq-network-boot.conf** | Template: dnsmasq DHCP + TFTP on eth1. Setup script writes `/etc/dnsmasq.d/network-boot.conf` using values from `lan-subnet.conf`. | -| **nft-nat-lan.conf** | Template: nftables NAT for LAN→WAN. Setup script writes `/etc/nftables.d/nat-lan.conf` using `LAN_CIDR` from `lan-subnet.conf`. | +| **nft-nat-lan.conf** | Template: nftables NAT for LAN→WAN (primary + extra subnets + VLAN 40). Setup script writes `/etc/nftables.d/nat-lan.conf`. | +| **70-cm4-extra-lan** | Extra LAN IPs on eth1 (192.168.30.1, 192.168.127.1) and VLAN eth1.40 (192.168.0.1/24). Installed to `/etc/network/interfaces.d/` by setup script. | +| **toggle-network-boot-dhcp.sh** | Enable/disable PXE (TFTP) on the LXC; copied to `/opt/cm4-provisioning/` by setup script. | Setup is done by running (from your machine): diff --git a/emmc-provisioning/lxc/nft-nat-lan.conf b/emmc-provisioning/lxc/nft-nat-lan.conf index ae1a555..42b3b4c 100644 --- a/emmc-provisioning/lxc/nft-nat-lan.conf +++ b/emmc-provisioning/lxc/nft-nat-lan.conf @@ -1,10 +1,14 @@ -# nftables: NAT for LAN (eth1) so clients use WAN (eth0) for internet. +# nftables: NAT for LAN (eth1 + extra IPs + eth1.40) so clients use WAN (eth0) for internet. # Load with: nft -f /etc/nftables.d/nat-lan.conf -# When using setup-network-boot-on-lxc.sh, the subnet is taken from /opt/cm4-provisioning/lan-subnet.conf (LAN_CIDR). +# When using setup-network-boot-on-lxc.sh, the primary subnet is from lan-subnet.conf (LAN_CIDR). +# Extra subnets: 192.168.30.0/24, 192.168.127.0/24 (eth1), 192.168.0.0/24 (eth1.40 VLAN). table ip nat { chain postrouting { type nat hook postrouting priority srcnat; policy accept; ip saddr 10.20.50.0/24 oifname "eth0" masquerade + ip saddr 192.168.30.0/24 oifname "eth0" masquerade + ip saddr 192.168.127.0/24 oifname "eth0" masquerade + ip saddr 192.168.0.0/24 oifname "eth0" masquerade } } diff --git a/emmc-provisioning/scripts/setup-network-boot-on-lxc.sh b/emmc-provisioning/scripts/setup-network-boot-on-lxc.sh index dcd42ec..0f96f72 100755 --- a/emmc-provisioning/scripts/setup-network-boot-on-lxc.sh +++ b/emmc-provisioning/scripts/setup-network-boot-on-lxc.sh @@ -1,5 +1,6 @@ #!/usr/bin/env bash -# Setup network boot on the provisioning LXC: DHCP + TFTP on eth1, NAT so LAN uses eth0 for internet. +# Setup provisioning LXC: DHCP + DNS on eth1, extra LAN IPs, VLAN eth1.40, NAT so LAN uses eth0 for internet. +# Network boot (TFTP, PXE, initrd) sections are commented out; uncomment if you need PXE/netboot. # Run inside the LXC (as root), or from your machine: ./setup-network-boot-on-lxc.sh root@10.130.60.141 [SUBNET] # SUBNET optional: A.B.C.D/PREFIX (e.g. 10.100.1.1/24). When run with ssh target, writes lan-subnet.conf on LXC if SUBNET given. # When run with ssh target, rsyncs lxc/ and runs this script inside the container. Subnet is read from /opt/cm4-provisioning/lan-subnet.conf. @@ -14,12 +15,13 @@ if [[ -n "$TARGET" ]]; then REPO_DIR="$(dirname "$SCRIPT_DIR")" echo "Syncing lxc config and script to $TARGET ..." rsync -a "$REPO_DIR/lxc/" "$TARGET:/tmp/cm4-network-boot-lxc/" --exclude='.git' - if [[ -f "$REPO_DIR/network-boot-initramfs/initrd.img" ]]; then - echo "Copying initrd.img to $TARGET ..." - scp "$REPO_DIR/network-boot-initramfs/initrd.img" "$TARGET:/tmp/cm4-network-boot-lxc/initrd.img" - else - echo "Note: network-boot-initramfs/initrd.img not found (run build.sh first); skipping." - fi + # Network boot: copy initrd.img for TFTP + # if [[ -f "$REPO_DIR/network-boot-initramfs/initrd.img" ]]; then + # echo "Copying initrd.img to $TARGET ..." + # scp "$REPO_DIR/network-boot-initramfs/initrd.img" "$TARGET:/tmp/cm4-network-boot-lxc/initrd.img" + # else + # echo "Note: network-boot-initramfs/initrd.img not found (run build.sh first); skipping." + # fi scp "$SCRIPT_DIR/setup-network-boot-on-lxc.sh" "$TARGET:/tmp/cm4-network-boot-lxc/setup.sh" # If SUBNET_ARG given, write lan-subnet.conf on LXC so inner script uses the set subnet if [[ -n "$SUBNET_ARG" ]]; then @@ -67,18 +69,21 @@ else DHCP_RANGE_END="10.20.50.200" echo "No lan-subnet.conf and no SUBNET argument; using defaults: $LAN_CIDR." fi -echo "Configuring network boot (DHCP + TFTP on eth1, NAT via eth0) — LAN $LAN_CIDR (gateway $LAN_GW), DHCP ${DHCP_RANGE_START}-${DHCP_RANGE_END} ..." +echo "Configuring LAN (DHCP + DNS on eth1, NAT via eth0) — LAN $LAN_CIDR (gateway $LAN_GW), DHCP ${DHCP_RANGE_START}-${DHCP_RANGE_END} ..." -# 1) Install dnsmasq +# 1) Install dnsmasq and vlan (for eth1.40) if ! command -v dnsmasq >/dev/null 2>&1; then apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq dnsmasq fi +if ! command -v vconfig >/dev/null 2>&1; then + apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq vlan +fi -# 2) dnsmasq config for eth1 only (DHCP + TFTP + DNS); PXE options in network-boot-pxe.conf (toggle with toggle-network-boot-dhcp.sh) +# 2) dnsmasq config for eth1 only (DHCP + DNS). PXE/TFTP in network-boot-pxe.conf when needed (toggle-network-boot-dhcp.sh) mkdir -p /etc/dnsmasq.d cat > /etc/dnsmasq.d/network-boot.conf << DNSMASQ # DHCP + DNS on eth1 only (provisioning LAN) -# TFTP and PXE options in network-boot-pxe.conf, controlled by toggle-network-boot-dhcp.sh +# (TFTP/PXE options in network-boot-pxe.conf when network boot is enabled) interface=eth1 bind-interfaces dhcp-range=${DHCP_RANGE_START},${DHCP_RANGE_END},12h @@ -91,51 +96,72 @@ log-dhcp log-queries DNSMASQ mkdir -p /opt/cm4-provisioning -if [ -f /tmp/cm4-network-boot-lxc/toggle-network-boot-dhcp.sh ]; then - cp /tmp/cm4-network-boot-lxc/toggle-network-boot-dhcp.sh /opt/cm4-provisioning/ - chmod +x /opt/cm4-provisioning/toggle-network-boot-dhcp.sh - /opt/cm4-provisioning/toggle-network-boot-dhcp.sh enable +# Network boot: install toggle script and enable PXE/TFTP +# if [ -f /tmp/cm4-network-boot-lxc/toggle-network-boot-dhcp.sh ]; then +# cp /tmp/cm4-network-boot-lxc/toggle-network-boot-dhcp.sh /opt/cm4-provisioning/ +# chmod +x /opt/cm4-provisioning/toggle-network-boot-dhcp.sh +# /opt/cm4-provisioning/toggle-network-boot-dhcp.sh enable +# fi + +# 3) Network boot: TFTP root — fetch Raspberry Pi 4 boot files from GitHub if missing +# mkdir -p /srv/tftpboot +# if [[ ! -f /srv/tftpboot/start4cd.elf ]]; then +# echo "Fetching Raspberry Pi firmware boot files from GitHub ..." +# if ! command -v curl >/dev/null 2>&1 && ! command -v wget >/dev/null 2>&1; then +# apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq curl +# fi +# tmpdir=$(mktemp -d) +# trap "rm -rf $tmpdir" EXIT +# if command -v curl >/dev/null 2>&1; then +# curl -sL "https://github.com/raspberrypi/firmware/archive/refs/heads/master.tar.gz" -o "$tmpdir/firmware.tar.gz" +# else +# wget -q -O "$tmpdir/firmware.tar.gz" "https://github.com/raspberrypi/firmware/archive/refs/heads/master.tar.gz" +# fi +# tar xzf "$tmpdir/firmware.tar.gz" -C "$tmpdir" +# cp -a "$tmpdir/firmware-master/boot/." /srv/tftpboot/ +# rm -rf "$tmpdir" +# echo "Copied RPi boot files to /srv/tftpboot" +# else +# echo "TFTP root already has boot files (start4cd.elf present), skipping fetch." +# fi + +# 3b) Network boot: copy provisioning initrd.img to TFTP root if provided +# if [[ -f /tmp/cm4-network-boot-lxc/initrd.img ]]; then +# cp /tmp/cm4-network-boot-lxc/initrd.img /srv/tftpboot/initrd.img +# echo "Copied initrd.img to /srv/tftpboot" +# if [[ -f /srv/tftpboot/config.txt ]] && ! grep -q 'initramfs initrd.img' /srv/tftpboot/config.txt 2>/dev/null; then +# echo "" >> /srv/tftpboot/config.txt +# echo "# Provisioning initramfs (network-boot-initramfs)" >> /srv/tftpboot/config.txt +# echo "initramfs initrd.img followkernel" >> /srv/tftpboot/config.txt +# echo "Added initramfs line to config.txt" +# fi +# fi + +# 4) Extra LAN IPs on eth1 (192.168.30.1, 192.168.127.1) and VLAN eth1.40 (192.168.0.1/24) +EXTRA_LAN_CONF="/etc/network/interfaces.d/70-cm4-extra-lan" +if [[ -f /tmp/cm4-network-boot-lxc/70-cm4-extra-lan ]]; then + mkdir -p /etc/network/interfaces.d + cp /tmp/cm4-network-boot-lxc/70-cm4-extra-lan "$EXTRA_LAN_CONF" + # Ensure main interfaces includes interfaces.d (Debian) + if [[ -f /etc/network/interfaces ]] && ! grep -q 'interfaces.d' /etc/network/interfaces 2>/dev/null; then + echo 'source /etc/network/interfaces.d/*' >> /etc/network/interfaces + fi + # Apply immediately: secondary IPs on eth1 + ip addr add 192.168.30.1/24 dev eth1 2>/dev/null || true + ip addr add 192.168.127.1/24 dev eth1 2>/dev/null || true + # Create and bring up eth1.40 (VLAN 40) + ip link add link eth1 name eth1.40 type vlan id 40 2>/dev/null || true + ip addr add 192.168.0.1/24 dev eth1.40 2>/dev/null || true + ip link set eth1.40 up 2>/dev/null || true + echo "Extra LAN: eth1 + 192.168.30.1/24, 192.168.127.1/24; eth1.40 192.168.0.1/24 (persisted in $EXTRA_LAN_CONF)" fi -# 3) TFTP root: fetch Raspberry Pi 4 boot files from GitHub if missing -mkdir -p /srv/tftpboot -if [[ ! -f /srv/tftpboot/start4cd.elf ]]; then - echo "Fetching Raspberry Pi firmware boot files from GitHub ..." - if ! command -v curl >/dev/null 2>&1 && ! command -v wget >/dev/null 2>&1; then - apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq curl - fi - tmpdir=$(mktemp -d) - trap "rm -rf $tmpdir" EXIT - if command -v curl >/dev/null 2>&1; then - curl -sL "https://github.com/raspberrypi/firmware/archive/refs/heads/master.tar.gz" -o "$tmpdir/firmware.tar.gz" - else - wget -q -O "$tmpdir/firmware.tar.gz" "https://github.com/raspberrypi/firmware/archive/refs/heads/master.tar.gz" - fi - tar xzf "$tmpdir/firmware.tar.gz" -C "$tmpdir" - cp -a "$tmpdir/firmware-master/boot/." /srv/tftpboot/ - rm -rf "$tmpdir" - echo "Copied RPi boot files to /srv/tftpboot" -else - echo "TFTP root already has boot files (start4cd.elf present), skipping fetch." -fi - -# 3b) Copy provisioning initrd.img to TFTP root if provided -if [[ -f /tmp/cm4-network-boot-lxc/initrd.img ]]; then - cp /tmp/cm4-network-boot-lxc/initrd.img /srv/tftpboot/initrd.img - echo "Copied initrd.img to /srv/tftpboot" - if [[ -f /srv/tftpboot/config.txt ]] && ! grep -q 'initramfs initrd.img' /srv/tftpboot/config.txt 2>/dev/null; then - echo "" >> /srv/tftpboot/config.txt - echo "# Provisioning initramfs (network-boot-initramfs)" >> /srv/tftpboot/config.txt - echo "initramfs initrd.img followkernel" >> /srv/tftpboot/config.txt - echo "Added initramfs line to config.txt" - fi -fi - -# 4) IP forwarding (LAN clients use WAN) +# 5) IP forwarding (LAN clients use WAN) echo 'net.ipv4.ip_forward=1' > /etc/sysctl.d/99-cm4-network-boot.conf sysctl -p /etc/sysctl.d/99-cm4-network-boot.conf 2>/dev/null || sysctl -w net.ipv4.ip_forward=1 -# 5) NAT: LAN subnet -> eth0 (masquerade) +# 6) NAT: primary LAN + extra LAN subnets + VLAN 40 -> eth0 (masquerade) +# Subnets: primary (from lan-subnet.conf), 192.168.30.0/24, 192.168.127.0/24, 192.168.0.0/24 (eth1.40) if command -v nft >/dev/null 2>&1; then mkdir -p /etc/nftables.d cat > /etc/nftables.d/nat-lan.conf << NFT @@ -143,29 +169,37 @@ table ip nat { chain postrouting { type nat hook postrouting priority srcnat; policy accept; ip saddr ${LAN_CIDR} oifname "eth0" masquerade + ip saddr 192.168.30.0/24 oifname "eth0" masquerade + ip saddr 192.168.127.0/24 oifname "eth0" masquerade + ip saddr 192.168.0.0/24 oifname "eth0" masquerade } } NFT if ! nft list table ip nat 2>/dev/null | grep -q postrouting; then nft -f /etc/nftables.d/nat-lan.conf + else + nft -f /etc/nftables.d/nat-lan.conf fi # Ensure main config includes our drop-in (Debian) if [[ -f /etc/nftables.conf ]] && ! grep -q 'nftables.d/nat-lan' /etc/nftables.conf 2>/dev/null; then echo 'include "/etc/nftables.d/nat-lan.conf"' >> /etc/nftables.conf fi - echo "NAT rule added (nftables) and saved to /etc/nftables.d/nat-lan.conf" + echo "NAT rules added (nftables): ${LAN_CIDR}, 192.168.30.0/24, 192.168.127.0/24, 192.168.0.0/24 -> eth0" else # Fallback iptables - iptables -t nat -C POSTROUTING -s "${LAN_CIDR}" -o eth0 -j MASQUERADE 2>/dev/null || \ - iptables -t nat -A POSTROUTING -s "${LAN_CIDR}" -o eth0 -j MASQUERADE - echo "NAT rule added (iptables)." + for cidr in "${LAN_CIDR}" 192.168.30.0/24 192.168.127.0/24 192.168.0.0/24; do + iptables -t nat -C POSTROUTING -s "$cidr" -o eth0 -j MASQUERADE 2>/dev/null || \ + iptables -t nat -A POSTROUTING -s "$cidr" -o eth0 -j MASQUERADE + done + echo "NAT rules added (iptables) for primary LAN and extra subnets." fi -# 6) Enable and start dnsmasq +# 7) Enable and start dnsmasq systemctl enable dnsmasq systemctl restart dnsmasq -echo "Network boot setup done." -echo " - DHCP + TFTP on eth1 ($LAN_GW), range ${DHCP_RANGE_START}-${DHCP_RANGE_END}" -echo " - NAT: ${LAN_CIDR} -> eth0 (internet)" -echo " - TFTP root: /srv/tftpboot (RPi boot files; initrd.img if provided)" +echo "Setup done." +echo " - DHCP + DNS on eth1 ($LAN_GW), range ${DHCP_RANGE_START}-${DHCP_RANGE_END}" +echo " - NAT: ${LAN_CIDR}, 192.168.30.0/24, 192.168.127.0/24, 192.168.0.0/24 -> eth0 (internet)" +echo " - Extra LAN: eth1 also 192.168.30.1, 192.168.127.1; eth1.40 192.168.0.1/24 (VLAN 40)" +# echo " - TFTP root: /srv/tftpboot (RPi boot files; initrd.img if provided)" # when network boot enabled