Enhance eMMC provisioning scripts and documentation: add troubleshooting section for device connection issues, update flash script to prevent concurrent runs, and improve logging. Adjust deployment scripts to verify presence of boot files in mass-storage-gadget.

chromium-setup/emmc-provisioning/docs/PROXMOX-LXC-DEPLOYMENT.md

@@ -167,6 +167,26 @@ Or copy `scripts/monitor-from-host.sh` to the host and run `./monitor-from-host.
 
 ---
 
+## Troubleshooting: device connected but not shown in portal
+
+1. **Host has old flash script** – The script must *not* exit when the golden image is missing (so you can use Backup first). Update the host:
+   ```bash
+   scp chromium-setup/emmc-provisioning/host/flash-emmc-on-connect.sh root@10.130.60.224:/opt/cm4-provisioning/
+   ssh root@10.130.60.224 "chmod +x /opt/cm4-provisioning/flash-emmc-on-connect.sh"
+   ```
+
+2. **Unplug and replug the USB** – udev runs the trigger only when the device is *added*. Unplug the reTerminal USB (keep it in boot mode), then plug it back in. The trigger will run the script and rpiboot; when the eMMC is exposed, the portal shows "Device connected" with Backup/Deploy.
+
+3. **If rpiboot fails** – Check on the host: `ssh root@10.130.60.224 'tail -30 /var/lib/cm4-provisioning/flash.log'` (rpiboot stderr is appended there). Try unplug/replug again. To see the exact rpiboot error: `ssh root@10.130.60.224 '/opt/usbboot/rpiboot -d /opt/usbboot/mass-storage-gadget64'` (device connected; Ctrl+C to stop). Run `scripts/monitor-from-host.sh` for a full snapshot.
+
+4. **"No 'bootcode' files found in mass-storage-gadget64"** – Usually because `bootfiles.bin` is a **broken symlink** (e.g. `-> ../firmware/bootfiles.bin`) and that target doesn’t exist. **Fix on host:** run `scripts/fix-gadget-bootcode-on-host.sh` on the host (it removes the symlink and extracts `bootcode4.bin` from the installed rpiboot binary). From your machine: `ssh root@10.130.60.224 'bash -s' < scripts/fix-gadget-bootcode-on-host.sh`. **Alternative:** repopulate the gadget dir with `./scripts/populate-gadget-on-host.sh root@10.130.60.224`, or full reinstall with `./scripts/build-and-deploy-usbboot-to-host.sh root@10.130.60.224`. Then verify: `ls -la /opt/usbboot/mass-storage-gadget64/` (should list a real `bootcode4.bin` or `bootfiles.bin`, plus `boot.img`, `config.txt`).
+
+4. **Clear stuck error in portal** – If the portal shows an old error (e.g. "Golden image not found" or "rpiboot failed"), click **Clear message** in the dashboard, or: `ssh root@10.130.60.224 "echo '{\"phase\":\"idle\",\"message\":\"Waiting for reTerminal in boot mode or network.\",\"progress\":null}' > /var/lib/cm4-provisioning/status.json"`. Then unplug/replug the device.
+
+5. **Trigger now runs the flash script in the background** (not via systemd-run) so it can access the USB device; a 2s delay gives the device time to enumerate before rpiboot runs.
+
+---
+
 ## Redeploy / update scripts
 
 From your repo (e.g. after changing scripts):

chromium-setup/emmc-provisioning/host/89-cm4-boot-mode-permissions.rules

@@ -0,0 +1,4 @@
+# Ensure CM4 boot-mode USB device is accessible to rpiboot (libusb).
+# Load before 90-cm4-boot-mode.rules so permissions are set before the trigger runs.
+SUBSYSTEM=="usb", ATTR{idVendor}=="2b8e", MODE="0666"
+SUBSYSTEM=="usb", ATTR{idVendor}=="0a5c", ATTR{idProduct}=="2711", MODE="0666"

chromium-setup/emmc-provisioning/host/cm4-flash-trigger.sh

@@ -1,6 +1,7 @@
 #!/usr/bin/env bash
-# Called by udev when CM4 in boot mode is connected. Starts the provisioning script in the
-# background (rpiboot + wait for portal Backup/Deploy choice). Install to /usr/local/bin/cm4-flash-trigger.sh
+# Called by udev when CM4 in boot mode (0a5c:2711 or 2b8e) is connected.
+# Start the flash service via systemd so it runs under systemd, not udev —
+# otherwise udev kills the process when the device re-enumerates (first→second stage).
+# --no-block: return immediately so udev doesn't wait; the service runs in the background.
 
-FLASH_SCRIPT="${CM4_FLASH_SCRIPT:-/opt/cm4-provisioning/flash-emmc-on-connect.sh}"
-exec systemd-run --no-block --unit=cm4-flash-once "$FLASH_SCRIPT"
+systemctl --no-block start cm4-flash.service

chromium-setup/emmc-provisioning/host/cm4-flash.service

@@ -0,0 +1,21 @@
+[Unit]
+Description=CM4 eMMC provisioning (rpiboot + backup/deploy)
+# Run after udev has settled; do not block boot
+After=systemd-udevd.service
+DefaultDependencies=yes
+
+[Service]
+Type=oneshot
+# Delay so USB device is enumerated and udev permissions applied before we run
+ExecStartPre=/bin/sleep 5
+ExecStart=/opt/cm4-provisioning/flash-emmc-on-connect.sh
+# Run as root; flash script logs to /var/lib/cm4-provisioning/flash.log
+User=root
+StandardOutput=journal
+StandardError=journal
+# Allow long run (rpiboot + device scan + wait for user choice)
+TimeoutStartSec=900
+
+[Install]
+# Only started by udev trigger, not at boot
+WantedBy=multi-user.target

chromium-setup/emmc-provisioning/host/flash-emmc-on-connect.sh

@@ -4,7 +4,11 @@
 # to choose Backup or Deploy in the portal — no auto-flash; action runs only after portal choice.
 # Run this from udev or a systemd service. Requires: usbboot (rpiboot) built, golden image for deploy.
 
-set -e
+# Do NOT use set -e: rpiboot and intermediate commands may return non-zero without being fatal
+
+# Redirect all output to log file (stdout + stderr) so nothing is lost to nohup buffering
+LOG_FILE="${LOG_FILE:-/var/lib/cm4-provisioning/flash.log}"
+exec >> "$LOG_FILE" 2>&1
 
 # Load overrides from env file if present
 [[ -f /opt/cm4-provisioning/env ]] && source /opt/cm4-provisioning/env
@@ -12,13 +16,14 @@ set -e
 # Configuration - adjust paths and size for your setup
 RPIBOOT_DIR="${RPIBOOT_DIR:-/opt/usbboot}"
 GOLDEN_IMAGE="${GOLDEN_IMAGE:-/var/lib/cm4-provisioning/golden.img}"
-# Expected eMMC size in bytes (reTerminal CM4 often 8GB or 16GB). Used to identify the correct block device.
-EMMC_SIZE_BYTES="${EMMC_SIZE_BYTES:-$(( 8 * 1024 * 1024 * 1024 ))}"
+# Expected eMMC size in bytes. reTerminal DM (CM4) has 32 GB eMMC (~31268536320 bytes).
+EMMC_SIZE_BYTES="${EMMC_SIZE_BYTES:-$(( 32 * 1024 * 1024 * 1024 ))}"
 LOG_TAG="cm4-flash"
 STATUS_FILE="${STATUS_FILE:-/var/lib/cm4-provisioning/status.json}"
 LOG_FILE="${LOG_FILE:-/var/lib/cm4-provisioning/flash.log}"
+LOCK_FILE="${LOCK_FILE:-/var/lib/cm4-provisioning/flash.lock}"
 
-log() { echo "[$LOG_TAG] $*"; logger -t "$LOG_TAG" "$*"; echo "[$(date -Iseconds)] $*" >> "$LOG_FILE" 2>/dev/null || true; }
+log() { echo "[$(date -Iseconds)] [$LOG_TAG] $*"; logger -t "$LOG_TAG" "$*" 2>/dev/null || true; }
 
 write_status() {
   local phase="$1" message="$2" progress="${3:-null}" error="${4:-}"
@@ -33,6 +38,14 @@ write_status() {
   fi
 }
 
+# Prevent concurrent runs (udev fires multiple times during rpiboot re-enumeration)
+exec 9>"$LOCK_FILE"
+if ! flock -n 9; then
+  echo "[$(date -Iseconds)] Another flash-emmc instance is already running; exiting."
+  exit 0
+fi
+trap 'rm -f "$LOCK_FILE" "$CURRENT_DEVICE_FILE" "$DEVICE_SOURCE_FILE" 2>/dev/null; exec 9>&-' EXIT
+
 # Optional: only run if this file exists (safety)
 ENABLE_FILE="${ENABLE_FILE:-/etc/cm4-provisioning/enabled}"
 if [[ -n "$ENABLE_FILE" && ! -f "$ENABLE_FILE" ]]; then
@@ -72,6 +85,12 @@ if [[ -z "$RPIBOOT_GADGET" ]]; then
   write_status "error" "rpiboot gadget missing" "null" "Copy mass-storage-gadget(64) to $RPIBOOT_DIR"
   exit 1
 fi
+# rpiboot requires bootfiles.bin or one of bootcode*.bin in the gadget dir; empty dir causes "No 'bootcode' files found"
+if [[ ! -f "$RPIBOOT_GADGET/bootfiles.bin" && ! -f "$RPIBOOT_GADGET/bootcode.bin" && ! -f "$RPIBOOT_GADGET/bootcode4.bin" && ! -f "$RPIBOOT_GADGET/bootcode5.bin" ]]; then
+  log "rpiboot gadget dir has no boot files: $RPIBOOT_GADGET (reinstall usbboot)"
+  write_status "error" "rpiboot gadget empty" "null" "No boot files in $RPIBOOT_GADGET. Reinstall usbboot: run install-usbboot-on-host.sh on the host or build-and-deploy-usbboot-to-host.sh from your machine."
+  exit 1
+fi
 
 # Ensure status dir exists and start with running state
 mkdir -p "$(dirname "$STATUS_FILE")" "$BACKUPS_DIR" 2>/dev/null || true
@@ -81,35 +100,47 @@ write_status "rpiboot" "Connecting to CM4 in boot mode…" "0"
 before_devs=$(lsblk -nd -o NAME 2>/dev/null | sort)
 
 log "Starting rpiboot to expose CM4 eMMC as mass storage..."
-if ! "$RPIBOOT_BIN" -d "$RPIBOOT_GADGET"; then
-  log "rpiboot failed or no device connected"
-  write_status "error" "rpiboot failed" "null" "rpiboot failed or no device connected"
+# Run rpiboot with 90s timeout so we don't hang if it doesn't exit cleanly when device switches to mass storage
+rpiboot_exit=0
+timeout 90 "$RPIBOOT_BIN" -d "$RPIBOOT_GADGET" || rpiboot_exit=$?
+# timeout returns 124 if killed by timeout; 0 or other if rpiboot exited on its own
+if [[ "$rpiboot_exit" -eq 124 ]]; then
+  log "rpiboot timed out after 90s (device may have switched to mass storage)"
+elif [[ "$rpiboot_exit" -ne 0 ]]; then
+  log "rpiboot exited with code $rpiboot_exit"
+  write_status "error" "rpiboot failed" "null" "rpiboot failed or no device connected. Check flash.log on host. Try unplug/replug USB."
   exit 1
 fi
 
-# rpiboot exits when mass storage appears; give udev a moment to create /dev/sdX
-sleep 3
+echo "[$(date -Iseconds)] rpiboot finished (exit=$rpiboot_exit); starting device scan"
+log "rpiboot completed; waiting for block device..."
+write_status "rpiboot" "rpiboot done, waiting for block device…" "10"
 
-# Find new block device (prefer one matching expected eMMC size)
+# rpiboot exits when device switches to mass storage; udev may need several seconds to create /dev/sdX
+# Poll for new block device for up to 30s (device switch can be slow)
 target_dev=""
-for dev in /dev/sd[a-z] /dev/sd[a-z][a-z]; do
-  [[ -b "$dev" ]] || continue
-  # Skip partitions
-  [[ "$dev" =~ [0-9]$ ]] && continue
-  size=$(blockdev --getsize64 "$dev" 2>/dev/null || true)
-  if [[ -n "$size" ]]; then
-    # Allow 5% tolerance on size
-    if (( size >= EMMC_SIZE_BYTES * 95 / 100 && size <= EMMC_SIZE_BYTES * 105 / 100 )); then
-      target_dev=$dev
-      break
+for wait_sec in $(seq 2 2 10) $(seq 12 2 30); do
+  sleep 2
+  for dev in /dev/sd[a-z] /dev/sd[a-z][a-z]; do
+    [[ -b "$dev" ]] || continue
+    [[ "$dev" =~ [0-9]$ ]] && continue
+    size=$(blockdev --getsize64 "$dev" 2>/dev/null || true)
+    if [[ -n "$size" ]]; then
+      if (( size >= EMMC_SIZE_BYTES * 95 / 100 && size <= EMMC_SIZE_BYTES * 105 / 100 )); then
+        target_dev=$dev
+        break 2
+      fi
+      if [[ -z "$target_dev" && "$before_devs" != *"${dev#/dev/}"* ]]; then
+        target_dev=$dev
+      fi
     fi
-    # Otherwise take first new disk that appeared (fallback)
-    if [[ -z "$target_dev" && "$before_devs" != *"${dev#/dev/}"* ]]; then
-      target_dev=$dev
-    fi
-  fi
+  done
+  [[ -n "$target_dev" ]] && break
+  log "Waiting for block device... ${wait_sec}s"
+  write_status "rpiboot" "Waiting for eMMC block device… (${wait_sec}s)" "10"
 done
 
+log "Device scan complete. before_devs=[$before_devs] target_dev=[$target_dev]"
 if [[ -z "$target_dev" ]]; then
   log "No suitable block device found after rpiboot (expected ~${EMMC_SIZE_BYTES} bytes)"
   write_status "error" "No eMMC device found" "null" "No suitable block device after rpiboot"
@@ -131,7 +162,7 @@ for (( i = 0; i < WAIT_TIMEOUT; i += 2 )); do
       backup_path="$BACKUPS_DIR/$backup_name"
       write_status "backup" "Creating backup…" "null"
       log "Backing up $target_dev to $backup_path..."
-      if dd if="$target_dev" of="$backup_path" bs=4M status=progress conv=fsync 2>>"$LOG_FILE"; then
+      if dd if="$target_dev" of="$backup_path" bs=4M status=progress conv=fsync; then
         log "Backup complete: $backup_path"
         write_status "done" "Backup complete: $backup_name" "100"
       else
@@ -141,12 +172,11 @@ for (( i = 0; i < WAIT_TIMEOUT; i += 2 )); do
       if [[ ! -f "$GOLDEN_IMAGE" ]]; then
         log "Golden image not found; cannot deploy."
         write_status "error" "Deploy unavailable" "null" "Golden image not found. Add golden.img to /var/lib/cm4-provisioning/ for deploy."
-        rm -f "$CURRENT_DEVICE_FILE" "$DEVICE_SOURCE_FILE" 2>/dev/null || true
         exit 1
       fi
       write_status "flashing" "Writing golden image…" "null"
       log "Flashing $GOLDEN_IMAGE to $target_dev..."
-      if dd if="$GOLDEN_IMAGE" of="$target_dev" bs=4M status=progress conv=fsync 2>>"$LOG_FILE"; then
+      if dd if="$GOLDEN_IMAGE" of="$target_dev" bs=4M status=progress conv=fsync; then
         log "Flash complete. Remove eMMC disable jumper and power cycle the reTerminal."
         write_status "done" "Flash complete. Remove eMMC disable jumper and power cycle the reTerminal." "100"
       else
@@ -155,11 +185,9 @@ for (( i = 0; i < WAIT_TIMEOUT; i += 2 )); do
     else
       write_status "error" "Unknown action" "null" "action_request must be 'backup' or 'deploy'"
     fi
-    rm -f "$CURRENT_DEVICE_FILE" "$DEVICE_SOURCE_FILE" 2>/dev/null || true
     exit 0
   fi
 done
 log "Timeout waiting for user choice"
 write_status "idle" "Timeout waiting for choice. Connect the device again to retry." "null"
-rm -f "$CURRENT_DEVICE_FILE" "$DEVICE_SOURCE_FILE" 2>/dev/null || true
 exit 0

chromium-setup/emmc-provisioning/scripts/build-and-deploy-usbboot-to-host.sh

@@ -41,11 +41,16 @@ rsync -a "$BUILD_DIR/usbboot/rpiboot" "$PROXMOX:/opt/usbboot/"
 # CM4 needs mass-storage-gadget (64-bit); copy whichever exists
 for dir in mass-storage-gadget64 mass-storage-gadget; do
   if [[ -d "$BUILD_DIR/usbboot/$dir" ]]; then
-    rsync -a "$BUILD_DIR/usbboot/$dir" "$PROXMOX:/opt/usbboot/"
+    rsync -a "$BUILD_DIR/usbboot/$dir/" "$PROXMOX:/opt/usbboot/$dir/"
     echo "  Copied $dir/"
   fi
 done
 ssh "$PROXMOX" "chmod +x /opt/usbboot/rpiboot"
 
+# Verify gadget has boot files (rpiboot needs bootfiles.bin or bootcode*.bin)
+if ! ssh "$PROXMOX" "test -f /opt/usbboot/mass-storage-gadget64/bootfiles.bin || test -f /opt/usbboot/mass-storage-gadget64/boot.img" 2>/dev/null; then
+  echo "Warning: mass-storage-gadget64 may be missing boot files. If rpiboot fails with 'No bootcode files found', run: ./populate-gadget-on-host.sh $PROXMOX"
+fi
+
 echo "[$(date -Iseconds)] usbboot deployed to $PROXMOX:/opt/usbboot"
 echo "Ensure the host flash script runs rpiboot with: -d /opt/usbboot/mass-storage-gadget64 (or mass-storage-gadget)."

chromium-setup/emmc-provisioning/scripts/deploy-to-proxmox.sh

@@ -52,6 +52,9 @@ cp "$DEPLOY/host/flash-emmc-on-connect.sh" /opt/cm4-provisioning/
 chmod +x /opt/cm4-provisioning/flash-emmc-on-connect.sh
 cp "$DEPLOY/host/cm4-flash-trigger.sh" /usr/local/bin/
 chmod +x /usr/local/bin/cm4-flash-trigger.sh
+cp "$DEPLOY/host/cm4-flash.service" /etc/systemd/system/
+systemctl daemon-reload
+cp "$DEPLOY/host/89-cm4-boot-mode-permissions.rules" /etc/udev/rules.d/ 2>/dev/null || true
 cp "$DEPLOY/host/90-cm4-boot-mode.rules" /etc/udev/rules.d/
 udevadm control --reload-rules
 log "Host: env and dirs ..."

chromium-setup/emmc-provisioning/scripts/fix-gadget-bootcode-on-host.sh

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+# Run on the Proxmox host (as root) when rpiboot fails with "No 'bootcode' files found".
+# Cause: mass-storage-gadget64/bootfiles.bin is a broken symlink (-> ../firmware/bootfiles.bin).
+# This script removes the symlink and extracts bootcode4.bin from the installed rpiboot binary.
+#
+# On host: bash fix-gadget-bootcode-on-host.sh
+# From your machine: ssh root@10.130.60.224 'bash -s' < scripts/fix-gadget-bootcode-on-host.sh
+
+set -e
+GADGET="${1:-/opt/usbboot/mass-storage-gadget64}"
+RPIBOOT="${2:-/opt/usbboot/rpiboot}"
+
+[[ -d "$GADGET" ]] || { echo "Error: $GADGET not found"; exit 1; }
+[[ -x "$RPIBOOT" ]] || { echo "Error: $RPIBOOT not found"; exit 1; }
+
+# Remove broken bootfiles.bin symlink if present
+if [[ -L "$GADGET/bootfiles.bin" ]] && ! [[ -f "$GADGET/bootfiles.bin" ]]; then
+  rm -f "$GADGET/bootfiles.bin"
+  echo "Removed broken symlink $GADGET/bootfiles.bin"
+fi
+
+# If we already have a valid boot file, done
+if [[ -f "$GADGET/bootcode4.bin" ]] || [[ -f "$GADGET/bootfiles.bin" ]]; then
+  echo "Already has bootcode4.bin or valid bootfiles.bin. OK."
+  exit 0
+fi
+
+# Get .data section file offset and address from ELF
+readelf -S "$RPIBOOT" | grep -q "\.data" || { echo "No .data section"; exit 1; }
+DATA_OFF=$(readelf -S "$RPIBOOT" | awk '/\.data\s/ { print "0x"$5; exit }')
+DATA_ADDR=$(readelf -S "$RPIBOOT" | awk '/\.data\s/ { print "0x"$4; exit }')
+LEN_SYM=$(objdump -t "$RPIBOOT" | awk '/msd_bootcode4_bin_len/ { print $1; exit }')
+BIN_SYM=$(objdump -t "$RPIBOOT" | awk '/msd_bootcode4_bin\s/ { print $1; exit }')
+[[ -n "$LEN_SYM" && -n "$BIN_SYM" ]] || { echo "msd_bootcode4_bin symbols not found in rpiboot"; exit 1; }
+
+LEN_ADDR=$((0x$LEN_SYM))
+BIN_ADDR=$((0x$BIN_SYM))
+LEN_OFF=$((DATA_OFF + LEN_ADDR - DATA_ADDR))
+BIN_OFF=$((DATA_OFF + BIN_ADDR - DATA_ADDR))
+LEN=$(dd if="$RPIBOOT" bs=1 skip=$LEN_OFF count=4 2>/dev/null | od -An -t u4 | tr -d ' ')
+dd if="$RPIBOOT" of="$GADGET/bootcode4.bin" bs=1 skip=$BIN_OFF count=$LEN 2>/dev/null
+echo "Wrote $GADGET/bootcode4.bin ($LEN bytes). Verify: $RPIBOOT -d $GADGET -V"

chromium-setup/emmc-provisioning/scripts/populate-gadget-on-host.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# Populate /opt/usbboot/mass-storage-gadget64 on the Proxmox host (no rpiboot build).
+# Use when rpiboot works but you see "No 'bootcode' files found" — the gadget dir is empty.
+# Usage: ./populate-gadget-on-host.sh [proxmox_host]
+# Example: ./populate-gadget-on-host.sh root@10.130.60.224
+
+set -e
+PROXMOX="${1:-root@10.130.60.224}"
+BUILD_DIR="/tmp/usbboot-gadget-$$"
+cleanup() { rm -rf "$BUILD_DIR"; }
+trap cleanup EXIT
+
+echo "[$(date -Iseconds)] Cloning usbboot to fetch mass-storage-gadget64 ..."
+mkdir -p "$BUILD_DIR"
+git clone --depth=1 https://github.com/raspberrypi/usbboot "$BUILD_DIR/usbboot"
+
+GADGET="$BUILD_DIR/usbboot/mass-storage-gadget64"
+if [[ ! -d "$GADGET" ]]; then
+  echo "Error: mass-storage-gadget64 not found in clone."
+  exit 1
+fi
+# rpiboot needs at least bootfiles.bin or bootcode*.bin
+if [[ ! -f "$GADGET/bootfiles.bin" && ! -f "$GADGET/boot.img" ]]; then
+  echo "Warning: clone has no bootfiles.bin or boot.img. Trying git lfs pull..."
+  (cd "$BUILD_DIR/usbboot" && git lfs pull 2>/dev/null) || true
+fi
+if [[ ! -f "$GADGET/bootfiles.bin" && ! -f "$GADGET/boot.img" ]]; then
+  echo "Error: mass-storage-gadget64 still has no boot files. Install git-lfs and retry, or run build-and-deploy-usbboot-to-host.sh for a full deploy."
+  exit 1
+fi
+
+echo "[$(date -Iseconds)] Syncing mass-storage-gadget64 to $PROXMOX:/opt/usbboot/ ..."
+ssh "$PROXMOX" "mkdir -p /opt/usbboot"
+rsync -a "$GADGET/" "$PROXMOX:/opt/usbboot/mass-storage-gadget64/"
+echo "Done. Verify with: ssh $PROXMOX 'ls -la /opt/usbboot/mass-storage-gadget64/'"