nearxos/reterminal-dm4
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c91cf6dd05417fe369ae081b7380c27f516740b4
reterminal-dm4/emmc-provisioning/docs/PROXMOX-LXC-DEPLOYMENT.md

16 KiB
Raw Blame History

CM4 eMMC provisioning on Proxmox (LXC + host)

The auto-flash runs on the Proxmox host (where the USB device appears). The LXC holds the dashboard and shares the golden image directory with the host.

One-command deploy

From your repo, a single run deploys all host and LXC files (scripts, systemd units, udev, dashboard):

./emmc-provisioning/scripts/deploy-to-proxmox.sh root@YOUR_PROXMOX_HOST

Optional env: CM4_BACKUPS_HOST_PATH=/path, DEPLOY_ROOTFS_STORAGE=local-lvm, DEPLOY_LXC_ROOT_PASSWORD=secret (set root password in LXC and enable SSH), DEPLOY_LXC_SSH_KEY=/path/to/pub (default: ~/.ssh/id_ed25519.pub or id_rsa.pub — copied to LXC root so you can ssh root@<LXC-IP>).

The script finds the container by hostname cm4-provisioning (any existing ID). If none exists, it creates a new LXC with the next available ID. So you can redeploy repeatedly without assuming a fixed ID like 201.

What is deployed

Where What
Proxmox host udev rule, trigger script, flash script, build-cloudinit and run-shrink scripts, systemd path units (build + shrink), /var/lib/cm4-provisioning/, /etc/cm4-provisioning/enabled
LXC (hostname cm4-provisioning) Dashboard (Flask) in /opt/cm4-provisioning/dashboard/; /var/lib/cm4-provisioning/ is a bind mount from the host (shared storage for golden image and backups)

When you plug the reTerminal in boot mode into the host, udev on the host runs the flash (rpiboot + dd). The golden image is read from /var/lib/cm4-provisioning/golden.img on the host (same path visible in the LXC).

Deployment layout (after running the deploy script)

  1. LXC (hostname cm4-provisioning, ID = found by hostname or next free):

    • Debian 12, 1 GB RAM, 8 GB rootfs
    • Bind mount: host /var/lib/cm4-provisioning → container /var/lib/cm4-provisioning
    • Optional second mount: CM4_BACKUPS_HOST_PATH → container /var/lib/cm4-provisioning/backups

  2. On the host:

    • /opt/cm4-provisioning/flash-emmc-on-connect.sh – flash script
    • /opt/cm4-provisioning/build-cloudinit-image.sh – build cloud-init image (triggered by path unit)
    • /opt/cm4-provisioning/run-shrink-on-host.sh – PiShrink for dashboard Shrink/Compress
    • /usr/local/bin/cm4-flash-trigger.sh – started by udev
    • /etc/udev/rules.d/90-cm4-boot-mode.rules – run trigger when USB vendor 2b8e is added
    • /opt/cm4-provisioning/env – GOLDEN_IMAGE, RPIBOOT_DIR, EMMC_SIZE_BYTES (and BACKUPS_DIR if CM4_BACKUPS_HOST_PATH set)
    • /etc/cm4-provisioning/enabled – safety switch (remove to disable auto-flash)
    • systemd: cm4-build-cloudinit.path + .service, cm4-shrink.path + .service
    • /opt/cm4-provisioning/fix-gadget-bootcode-on-host.sh – used by install-usbboot-on-host.sh after building usbboot (fixes "rpiboot gadget empty" when gadget has broken symlinks)

  3. Inside the LXC (use pct exec <CTID> -- ... where <CTID> is the ID of the container with hostname cm4-provisioning; get it with pct list):

    • Dashboard: Flask app in /opt/cm4-provisioning/dashboard/ (monitor deployment, backup list, build cloud-init, set golden).
    • Golden image path: /var/lib/cm4-provisioning/golden.img (bind-mounted from host).

  4. usbboot (rpiboot) is not installed by the deploy script. Install it when the host has internet (see below).

What you need to do

1. Build and install rpiboot on the Proxmox host (when it has internet)

On your machine (repo already synced to the host):

# From your repo
scp emmc-provisioning/scripts/install-usbboot-on-host.sh root@10.130.60.224:/tmp/
ssh root@10.130.60.224 "bash /tmp/install-usbboot-on-host.sh"

Or on the host (if the deploy folder is still there):

ssh root@10.130.60.224
bash /tmp/emmc-provisioning-deploy/scripts/install-usbboot-on-host.sh

This installs dependencies, clones usbboot, builds it, and copies rpiboot to /opt/usbboot/. It then runs fix-gadget-bootcode-on-host.sh if present (from deploy), so the gadget has valid boot files and rpiboot does not fail with "No bootcode files found".

2. Enable root SSH and add your SSH key to the LXC

If you deployed with DEPLOY_LXC_ROOT_PASSWORD and/or a default SSH key (~/.ssh/id_ed25519.pub or id_rsa.pub), the LXC already has SSH enabled, root password set, and your key in /root/.ssh/authorized_keys — you can skip to ssh root@<LXC-IP> (get IP from deploy output or pct exec <CTID> -- hostname -I).

Otherwise, to enable root SSH and add a key:

  • Option A – Use the setup script (recommended): From your machine (with SSH key and optional password):

    # Add your default SSH key (~/.ssh/id_ed25519.pub or id_rsa.pub) and enable root SSH
./emmc-provisioning/scripts/setup-lxc-ssh.sh root@10.130.60.224

# Or specify key file and set root password
ROOT_PASSWORD='YourPassword' ./emmc-provisioning/scripts/setup-lxc-ssh.sh root@10.130.60.224 ~/.ssh/id_ed25519.pub

    Then connect with ssh root@<LXC-IP> (script prints the IP). To get the LXC IP:
    ssh root@HOST "CID=\$(pct list -no-header -output vmid,name | awk '\''\$2==\"cm4-provisioning\"{print \$1}'\''); pct exec \$CID -- hostname -I"

  • Option B – Manual:
    ssh root@HOST, then pct exec <CTID> -- bash (use the container ID from pct list for hostname cm4-provisioning). Install openssh-server, set PermitRootLogin yes, set root password, add your key, restart ssh.

3. (Optional) Store backup images on a host directory

To keep backup images on a specific host path (e.g. a large disk or NFS mount) instead of under /var/lib/cm4-provisioning/backups, deploy with CM4_BACKUPS_HOST_PATH set. That directory is created on the host, bind-mounted into the LXC at /var/lib/cm4-provisioning/backups, and the host flash script is configured to write backups there. The dashboard in the LXC then lists and serves those same files.

Deploy with a host backup path:

CM4_BACKUPS_HOST_PATH=/mnt/storage/cm4-backups ./emmc-provisioning/scripts/deploy-to-proxmox.sh root@10.130.60.224

Create /mnt/storage/cm4-backups (or your path) on the host first if it doesn’t exist; the deploy script will create it if possible. To add or change the backup mount, set CM4_BACKUPS_HOST_PATH and run the deploy script again (it reuses the container by hostname and updates the bind mount).

4. Put the golden image on the host (or in the LXC)

The image must be at /var/lib/cm4-provisioning/golden.img on the host. Because that directory is bind-mounted into the LXC, you can use either:

  • From the host:

    scp your-golden.img root@10.130.60.224:/var/lib/cm4-provisioning/golden.img

  • From the LXC (e.g. after copying the image into the container elsewhere first):

    pct exec <CTID> -- ls -la /var/lib/cm4-provisioning/
# Replace <CTID> with the ID of the cm4-provisioning container (pct list).

5. Run the provisioning dashboard (optional)

The dashboard shows connection steps and live deployment status (idle / connecting / flashing / done / error) and a recent flash log. It reads the same status.json and flash.log that the host’s flash script writes (via the bind-mounted /var/lib/cm4-provisioning).

Inside the LXC (pct exec -- bash):

# Copy dashboard into the container (from host, if you have the repo there)
# Or from your workstation:
#   rsync -a emmc-provisioning/dashboard/ root@10.130.60.224:/tmp/dashboard/
#   Or re-run deploy-to-proxmox.sh to push the latest dashboard files.

# Inside the LXC (pct exec <CTID> -- bash):
apt-get update && apt-get install -y python3-flask
mkdir -p /opt/cm4-provisioning/dashboard/templates
# Copy app.py, templates/index.html, cm4-dashboard.service into the container (see dashboard/README.md)

cp /opt/cm4-provisioning/dashboard/cm4-dashboard.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now cm4-dashboard

Then open http://<LXC-IP>:5000 (get the IP with pct exec <CTID> -- hostname -I). If the LXC is on a private network, set up port forwarding on the Proxmox host or use a reverse proxy.

6. Optional: disable or enable auto-flash

  • Disable:
    ssh root@10.130.60.224 "rm /etc/cm4-provisioning/enabled"

  • Enable again:
    ssh root@10.130.60.224 "touch /etc/cm4-provisioning/enabled"

Usage

  1. Place the reTerminal in boot mode (eMMC disable jumper).
  2. Connect its USB slave port to the Proxmox host (not to the LXC).
  3. Power the reTerminal (or connect after power).
  4. On the host, udev will run the trigger and then the flash script (rpiboot, then dd). Watch logs: 
    ssh root@10.130.60.224 "journalctl -u cm4-flash-once -f"
# or
ssh root@10.130.60.224 "journalctl -t cm4-flash -f"
  5. When flashing finishes, remove the jumper and power cycle the reTerminal so it boots from eMMC.

Monitoring from the host

From the Proxmox host you can monitor:

What How
USB device lsusb — CM4 in boot mode shows as 2b8e (RPi) or 0a5c:2711 (Broadcom BCM2711)
Live status cat /var/lib/cm4-provisioning/status.json — same JSON the dashboard shows (phase, message, error)
Flash log tail -f /var/lib/cm4-provisioning/flash.log — script log (rpiboot, dd, errors)
Flash job systemctl status cm4-flash-once — whether the udev-triggered job is running/failed
Journal journalctl -u cm4-flash-once -f or journalctl -t cm4-flash -f — systemd/log output
Block devices lsblk — after rpiboot, the eMMC appears as a new disk (e.g. /dev/sdb)
Backups ls /var/lib/cm4-provisioning/backups/ — backup images (on host; if you used CM4_BACKUPS_HOST_PATH they are under that path on the host, bind-mounted into the LXC). To shrink automatically, set SHRINK_BACKUP=1 in /opt/cm4-provisioning/env — see EMMC-PROVISIONING-GUIDE.md § Shrinking backup and golden images.
Config cat /opt/cm4-provisioning/env — GOLDEN_IMAGE, RPIBOOT_DIR, EMMC_SIZE_BYTES

One-command snapshot:

# From your machine (stream script to host):
ssh root@10.130.60.224 'bash -s' < emmc-provisioning/scripts/monitor-from-host.sh

Or copy scripts/monitor-from-host.sh to the host and run ./monitor-from-host.sh for a full status dump (USB, status.json, flash unit, last log lines, block devices, config).

Troubleshooting: device connected but not shown in portal

  1. Host has old flash script – The script must not exit when the golden image is missing (so you can use Backup first). Update the host:

    scp emmc-provisioning/host/flash-emmc-on-connect.sh root@10.130.60.224:/opt/cm4-provisioning/
ssh root@10.130.60.224 "chmod +x /opt/cm4-provisioning/flash-emmc-on-connect.sh"

  2. Unplug and replug the USB – udev runs the trigger only when the device is added. Unplug the reTerminal USB (keep it in boot mode), then plug it back in. The trigger will run the script and rpiboot; when the eMMC is exposed, the portal shows "Device connected" with Backup/Deploy.

  3. If rpiboot fails – Check on the host: ssh root@10.130.60.224 'tail -30 /var/lib/cm4-provisioning/flash.log' (rpiboot stderr is appended there). Try unplug/replug again. To see the exact rpiboot error: ssh root@10.130.60.224 '/opt/usbboot/rpiboot -d /opt/usbboot/mass-storage-gadget64' (device connected; Ctrl+C to stop). Run scripts/monitor-from-host.sh for a full snapshot.

  4. "No 'bootcode' files found in mass-storage-gadget64" – Usually because bootfiles.bin is a broken symlink (e.g. -> ../firmware/bootfiles.bin) and that target doesn’t exist. Fix on host: run scripts/fix-gadget-bootcode-on-host.sh on the host (it removes the symlink and extracts bootcode4.bin from the installed rpiboot binary). From your machine: ssh root@10.130.60.224 'bash -s' < scripts/fix-gadget-bootcode-on-host.sh. Alternative: repopulate the gadget dir with ./scripts/populate-gadget-on-host.sh root@10.130.60.224, or full reinstall with ./scripts/build-and-deploy-usbboot-to-host.sh root@10.130.60.224. Then verify: ls -la /opt/usbboot/mass-storage-gadget64/ (should list a real bootcode4.bin or bootfiles.bin, plus boot.img, config.txt).

  5. Clear stuck error in portal – If the portal shows an old error (e.g. "Golden image not found" or "rpiboot failed"), click Clear message in the dashboard, or: ssh root@10.130.60.224 "echo '{\"phase\":\"idle\",\"message\":\"Waiting for reTerminal in boot mode or network.\",\"progress\":null}' > /var/lib/cm4-provisioning/status.json". Then unplug/replug the device.

  6. "PiShrink not installed" when clicking Shrink/Compress – Shrink and Compress run on the host, not in the LXC. Install PiShrink on the host: ssh root@HOST 'bash -s' < emmc-provisioning/scripts/install-pishrink-on-host.sh. Ensure deploy has been run so the host has run-shrink-on-host.sh and cm4-shrink.path enabled (systemctl status cm4-shrink.path).

  7. "Download failed" when building cloud-init image – The download runs on the host (not the LXC). Check: (1) Host can reach the internet: ssh root@HOST 'curl -sI https://downloads.raspberrypi.com/'. (2) Build status now shows curl’s error (e.g. "Could not resolve host", "Connection timed out"); check the dashboard error text. (3) If you use a proxy or custom CA, set in /opt/cm4-provisioning/env: CURL_INSECURE=1 to skip SSL verify (only if you understand the risk), then rerun the build.

  8. Backup stops before finishing – If backup or shrink appears to stop partway (e.g. dashboard stuck on "Creating backup…" or "Shrinking…"), the service may have been killed by systemd. The cm4-flash.service unit uses TimeoutStartSec=7200 (2 hours); if you deployed an older version with 15 minutes, redeploy so the host gets the updated unit, then on the host run systemctl daemon-reload so the next backup has enough time to complete.

  9. Trigger now runs the flash script in the background (not via systemd-run) so it can access the USB device; a 2s delay gives the device time to enumerate before rpiboot runs.

Redeploy / update scripts

From your repo (e.g. after changing scripts):

./emmc-provisioning/scripts/deploy-to-proxmox.sh root@10.130.60.224

That script syncs the repo to the host and reinstalls scripts on both the host and LXC 201. It does not overwrite /opt/cm4-provisioning/env or /etc/cm4-provisioning/enabled if you’ve changed them; adjust the script if you want that. It also does not build usbboot; run install-usbboot-on-host.sh on the host when needed.

Summary

Item Location
LXC Hostname cm4-provisioning (ID from pct list), on your Proxmox host
Golden image /var/lib/cm4-provisioning/golden.img (host and LXC see the same file)
Flash runs on Proxmox host (udev + rpiboot + dd)
Build rpiboot on host Run scripts/install-usbboot-on-host.sh on the host when it has internet
Dashboard Flask app in LXC at http://<LXC-IP>:5000; switch Flash/Backup mode, list and download backups; see dashboard/README.md and section 3 above
Backups Saved under /var/lib/cm4-provisioning/backups/ (optionally a host path bind-mounted into the LXC — set CM4_BACKUPS_HOST_PATH at deploy). When a device is detected, choose Backup or Deploy in the dashboard.
Network deploy/backup Network-booted devices run network-client/provisioning-client.sh and register with the dashboard; they then appear under "Device detected (Network)" and you choose Backup or Deploy. See network-client/README.md.
Network boot (DHCP + TFTP on eth1) If the LXC has a second interface eth1 as provisioning LAN (e.g. 10.20.50.1/24), run scripts/setup-network-boot-on-lxc.sh root@<LXC-IP> to install dnsmasq (DHCP+TFTP) on eth1 and NAT so LAN clients get internet via eth0. See docs/NETWORK-BOOT-LXC.md.