# CM4 eMMC provisioning on Proxmox (LXC + host)

The auto-flash **runs on the Proxmox host** (where the USB device appears). The **LXC** holds the dashboard and shares the **golden image** directory with the host.

**New deployment:** For a clear step-by-step guide to deploy on a **new** Proxmox instance, see **[DEPLOY-NEW-PROXMOX.md](DEPLOY-NEW-PROXMOX.md)**.

## One-command deploy

From your repo, a single run deploys **all** host and LXC files (scripts, systemd units, udev, dashboard):

```bash
./emmc-provisioning/scripts/deploy-to-proxmox.sh root@YOUR_PROXMOX_HOST
```

Optional env: `CM4_BACKUPS_HOST_PATH=/path`, `DEPLOY_ROOTFS_STORAGE=local-lvm`, `DEPLOY_LXC_ROOT_PASSWORD=secret` (set root password in LXC and enable SSH), `DEPLOY_LXC_SSH_KEY=/path/to/pub` (default: `~/.ssh/id_ed25519.pub` or `id_rsa.pub` — copied to LXC root so you can `ssh root@<LXC-IP>`).

**Network (WAN / LAN):** `DEPLOY_LXC_WAN_BRIDGE=vmbr0` (default), `DEPLOY_LXC_WAN_IP=dhcp` (default or e.g. `192.168.1.10/24`), `DEPLOY_LXC_LAN_BRIDGE=vmbr1` (if set, add eth1 as LAN), `DEPLOY_LXC_LAN_SUBNET=10.20.50.1/24` (LXC IP on LAN; default `10.20.50.1/24` when LAN bridge is set). Example: `DEPLOY_LXC_WAN_BRIDGE=vmbr0 DEPLOY_LXC_LAN_BRIDGE=vmbr1 DEPLOY_LXC_LAN_SUBNET=10.20.50.1/24 ./scripts/deploy-to-proxmox.sh root@host`.

The script **finds the container by hostname `cm4-provisioning`** (any existing ID). If none exists, it **creates a new LXC with the next available ID**. So you can redeploy repeatedly without assuming a fixed ID like 201.

## What is deployed

| Where | What |
|-------|-----|
| **Proxmox host** | udev rule, trigger script, flash script, build-cloudinit and run-shrink scripts, systemd path units (build + shrink), `/var/lib/cm4-provisioning/`, `/etc/cm4-provisioning/enabled` |
| **LXC (hostname cm4-provisioning)** | Dashboard (Flask) in `/opt/cm4-provisioning/dashboard/`; `/var/lib/cm4-provisioning/` is a **bind mount** from the host (shared storage for golden image and backups) |

When you plug the reTerminal in boot mode into the **host**, udev on the host runs the flash (rpiboot + dd). The golden image is read from `/var/lib/cm4-provisioning/golden.img` on the host (same path visible in the LXC).

---

## Deployment layout (after running the deploy script)

1. **LXC** (hostname `cm4-provisioning`, ID = found by hostname or next free):
   - Debian 12, 1 GB RAM, 8 GB rootfs
   - Bind mount: host `/var/lib/cm4-provisioning` → container `/var/lib/cm4-provisioning`
   - Optional second mount: `CM4_BACKUPS_HOST_PATH` → container `/var/lib/cm4-provisioning/backups`

2. **On the host**:
   - `/opt/cm4-provisioning/flash-emmc-on-connect.sh` – flash script
   - `/opt/cm4-provisioning/build-cloudinit-image.sh` – build cloud-init image (triggered by path unit)
   - `/opt/cm4-provisioning/run-shrink-on-host.sh` – PiShrink for dashboard Shrink/Compress
   - `/usr/local/bin/cm4-flash-trigger.sh` – started by udev
   - `/etc/udev/rules.d/90-cm4-boot-mode.rules` – run trigger when USB vendor `2b8e` is added
   - `/opt/cm4-provisioning/env` – `GOLDEN_IMAGE`, `RPIBOOT_DIR`, `EMMC_SIZE_BYTES` (and `BACKUPS_DIR` if `CM4_BACKUPS_HOST_PATH` set)
   - `/etc/cm4-provisioning/enabled` – safety switch (remove to disable auto-flash)
   - systemd: `cm4-build-cloudinit.path` + `.service`, `cm4-shrink.path` + `.service`
   - `/opt/cm4-provisioning/fix-gadget-bootcode-on-host.sh` – used by `install-usbboot-on-host.sh` after building usbboot (fixes "rpiboot gadget empty" when gadget has broken symlinks)

3. **Inside the LXC** (use `pct exec <CTID> -- ...` where `<CTID>` is the ID of the container with hostname `cm4-provisioning`; get it with `pct list`):
   - Dashboard: Flask app in `/opt/cm4-provisioning/dashboard/` (monitor deployment, backup list, build cloud-init, set golden).
   - Golden image path: `/var/lib/cm4-provisioning/golden.img` (bind-mounted from host).

4. **usbboot (rpiboot)** is **not** installed by the deploy script. Install it when the host has internet (see below).

---

## What you need to do

### 1. Build and install rpiboot on the Proxmox host (when it has internet)

On your machine (repo already synced to the host):

```bash
# From your repo
scp emmc-provisioning/scripts/install-usbboot-on-host.sh root@10.130.60.224:/tmp/
ssh root@10.130.60.224 "bash /tmp/install-usbboot-on-host.sh"
```

Or on the host (if the deploy folder is still there):

```bash
ssh root@10.130.60.224
bash /tmp/emmc-provisioning-deploy/scripts/install-usbboot-on-host.sh
```

This installs dependencies, clones usbboot, builds it, and copies `rpiboot` to `/opt/usbboot/`. It then runs **fix-gadget-bootcode-on-host.sh** if present (from deploy), so the gadget has valid boot files and rpiboot does not fail with "No bootcode files found".

### 2. Enable root SSH and add your SSH key to the LXC

If you deployed with **`DEPLOY_LXC_ROOT_PASSWORD`** and/or a default SSH key (**`~/.ssh/id_ed25519.pub`** or **`id_rsa.pub`**), the LXC already has SSH enabled, root password set, and your key in `/root/.ssh/authorized_keys` — you can **skip** to `ssh root@<LXC-IP>` (get IP from deploy output or `pct exec <CTID> -- hostname -I`).

Otherwise, to enable root SSH and add a key:

- **Option A – Use the setup script (recommended):** From your machine (with SSH key and optional password):

  ```bash
  # Add your default SSH key (~/.ssh/id_ed25519.pub or id_rsa.pub) and enable root SSH
  ./emmc-provisioning/scripts/setup-lxc-ssh.sh root@10.130.60.224

  # Or specify key file and set root password
  ROOT_PASSWORD='YourPassword' ./emmc-provisioning/scripts/setup-lxc-ssh.sh root@10.130.60.224 ~/.ssh/id_ed25519.pub
  ```

  Then connect with `ssh root@<LXC-IP>` (script prints the IP). To get the LXC IP:  
  `ssh root@HOST "CID=\$(pct list -no-header -output vmid,name | awk '\''\$2==\"cm4-provisioning\"{print \$1}'\''); pct exec \$CID -- hostname -I"`

- **Option B – Manual:**  
  `ssh root@HOST`, then `pct exec <CTID> -- bash` (use the container ID from `pct list` for hostname cm4-provisioning). Install openssh-server, set `PermitRootLogin yes`, set root password, add your key, restart ssh.

### 3. (Optional) Store backup images on a host directory

To keep backup images on a specific host path (e.g. a large disk or NFS mount) instead of under `/var/lib/cm4-provisioning/backups`, deploy with **`CM4_BACKUPS_HOST_PATH`** set. That directory is created on the host, bind-mounted into the LXC at `/var/lib/cm4-provisioning/backups`, and the host flash script is configured to write backups there. The dashboard in the LXC then lists and serves those same files.

**Deploy with a host backup path:**

```bash
CM4_BACKUPS_HOST_PATH=/mnt/storage/cm4-backups ./emmc-provisioning/scripts/deploy-to-proxmox.sh root@10.130.60.224
```

Create `/mnt/storage/cm4-backups` (or your path) on the host first if it doesn’t exist; the deploy script will create it if possible. To add or change the backup mount, set `CM4_BACKUPS_HOST_PATH` and run the deploy script again (it reuses the container by hostname and updates the bind mount).

### 4. Put the golden image on the host (or in the LXC)

The image must be at **`/var/lib/cm4-provisioning/golden.img`** on the **host**. Because that directory is bind-mounted into the LXC, you can use either:

- **From the host:**
  ```bash
  scp your-golden.img root@10.130.60.224:/var/lib/cm4-provisioning/golden.img
  ```

- **From the LXC** (e.g. after copying the image into the container elsewhere first):
  ```bash
  pct exec <CTID> -- ls -la /var/lib/cm4-provisioning/
  # Replace <CTID> with the ID of the cm4-provisioning container (pct list).
  ```

### 5. Run the provisioning dashboard (optional)

The dashboard shows **connection steps** and **live deployment status** (idle / connecting / flashing / done / error) and a recent flash log. It reads the same `status.json` and `flash.log` that the host’s flash script writes (via the bind-mounted `/var/lib/cm4-provisioning`).

**Inside the LXC (pct exec <CTID> -- bash):**

```bash
# Copy dashboard into the container (from host, if you have the repo there)
# Or from your workstation:
#   rsync -a emmc-provisioning/dashboard/ root@10.130.60.224:/tmp/dashboard/
#   Or re-run deploy-to-proxmox.sh to push the latest dashboard files.

# Inside the LXC (pct exec <CTID> -- bash):
apt-get update && apt-get install -y python3-flask
mkdir -p /opt/cm4-provisioning/dashboard/templates
# Copy app.py, templates/index.html, cm4-dashboard.service into the container (see dashboard/README.md)

cp /opt/cm4-provisioning/dashboard/cm4-dashboard.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now cm4-dashboard
```

Then open **http://&lt;LXC-IP&gt;:5000** (get the IP with `pct exec <CTID> -- hostname -I`). If the LXC is on a private network, set up port forwarding on the Proxmox host or use a reverse proxy.

### 6. Optional: disable or enable auto-flash

- **Disable:**  
  `ssh root@10.130.60.224 "rm /etc/cm4-provisioning/enabled"`

- **Enable again:**  
  `ssh root@10.130.60.224 "touch /etc/cm4-provisioning/enabled"`

---

## Usage

1. Place the reTerminal in **boot mode** (eMMC disable jumper).
2. Connect its **USB slave** port to the **Proxmox host** (not to the LXC).
3. Power the reTerminal (or connect after power).
4. On the host, udev will run the trigger and then the flash script (rpiboot, then dd). Watch logs:
   ```bash
   ssh root@10.130.60.224 "journalctl -u cm4-flash-once -f"
   # or
   ssh root@10.130.60.224 "journalctl -t cm4-flash -f"
   ```
5. When flashing finishes, remove the jumper and power cycle the reTerminal so it boots from eMMC.

---

## Monitoring from the host

From the **Proxmox host** you can monitor:

| What | How |
|------|-----|
| **USB device** | `lsusb` — CM4 in boot mode shows as **2b8e** (RPi) or **0a5c:2711** (Broadcom BCM2711) |
| **Live status** | `cat /var/lib/cm4-provisioning/status.json` — same JSON the dashboard shows (phase, message, error) |
| **Flash log** | `tail -f /var/lib/cm4-provisioning/flash.log` — script log (rpiboot, dd, errors) |
| **Flash job** | `systemctl status cm4-flash-once` — whether the udev-triggered job is running/failed |
| **Journal** | `journalctl -u cm4-flash-once -f` or `journalctl -t cm4-flash -f` — systemd/log output |
| **Block devices** | `lsblk` — after rpiboot, the eMMC appears as a new disk (e.g. `/dev/sdb`) |
| **Backups** | `ls /var/lib/cm4-provisioning/backups/` — backup images (on host; if you used `CM4_BACKUPS_HOST_PATH` they are under that path on the host, bind-mounted into the LXC). To shrink automatically, set `SHRINK_BACKUP=1` in `/opt/cm4-provisioning/env` — see **EMMC-PROVISIONING-GUIDE.md** § Shrinking backup and golden images. |
| **Config** | `cat /opt/cm4-provisioning/env` — GOLDEN_IMAGE, RPIBOOT_DIR, EMMC_SIZE_BYTES |

**One-command snapshot:**

```bash
# From your machine (stream script to host):
ssh root@10.130.60.224 'bash -s' < emmc-provisioning/scripts/monitor-from-host.sh
```

Or copy `scripts/monitor-from-host.sh` to the host and run `./monitor-from-host.sh` for a full status dump (USB, status.json, flash unit, last log lines, block devices, config).

---

## Troubleshooting: device connected but not shown in portal

1. **Host has old flash script** – The script must *not* exit when the golden image is missing (so you can use Backup first). Update the host:
   ```bash
   scp emmc-provisioning/host/flash-emmc-on-connect.sh root@10.130.60.224:/opt/cm4-provisioning/
   ssh root@10.130.60.224 "chmod +x /opt/cm4-provisioning/flash-emmc-on-connect.sh"
   ```

2. **Unplug and replug the USB** – udev runs the trigger only when the device is *added*. Unplug the reTerminal USB (keep it in boot mode), then plug it back in. The trigger will run the script and rpiboot; when the eMMC is exposed, the portal shows "Device connected" with Backup/Deploy.

3. **If rpiboot fails** ("rpiboot failed or no device connected"):
   - **Check flash.log on the host** (rpiboot runs there; the log has the real error):  
     `ssh root@YOUR_PROXMOX_HOST 'tail -50 /var/lib/cm4-provisioning/flash.log'`
   - **Quick diagnostic:**  
     `ssh root@YOUR_PROXMOX_HOST 'bash -s' < emmc-provisioning/scripts/check-usb-on-host.sh`  
     This shows whether the device is seen (lsusb), status, and last lines of flash.log.
   - **Ensure device is in boot mode:** eMMC disable jumper set, use the **USB slave** port (not host port). Unplug and replug after setting jumper.
   - **Run rpiboot manually** (device connected; Ctrl+C to stop):  
     `ssh root@YOUR_PROXMOX_HOST '/opt/usbboot/rpiboot -v -d /opt/usbboot/mass-storage-gadget64'`  
     The last line before exit is usually the error (e.g. "No device found", "Unable to open device").
   - Run `scripts/monitor-from-host.sh root@YOUR_PROXMOX_HOST` for a full snapshot.

**Manual usbboot test** (to verify rpiboot and the CM4 without the full flash flow):

1. **On the Proxmox host** (where USB is connected), with the reTerminal in **boot mode** (eMMC disable jumper set, USB slave port connected):
   ```bash
   /opt/usbboot/rpiboot -v -d /opt/usbboot/mass-storage-gadget64
   ```
2. You should see lines like: `Device located successfully`, `Loading: .../bootcode4.bin`, `Sending bootcode.bin`. When the device switches to mass storage, rpiboot exits and a new `/dev/sdX` may appear (check with `lsblk` in another terminal). Press **Ctrl+C** to stop rpiboot at any time.
3. **From your machine** (with device already connected to the host):
   ```bash
   ./emmc-provisioning/scripts/test-usbboot-on-host.sh root@YOUR_PROXMOX_HOST
   ```
   Optional: run with a timeout so it doesn’t wait forever: `TIMEOUT=60 ./emmc-provisioning/scripts/test-usbboot-on-host.sh root@YOUR_PROXMOX_HOST`

**"libusb_bulk_transfer sent 0 bytes; returned -7" / "Failed to write correct length"** (device found, then transfer fails in a loop):

This is a known USB timing/controller issue with CM4 and rpiboot ([raspberrypi/usbboot#36](https://github.com/raspberrypi/usbboot/issues/36)). Try in order:

1. **Use a USB 2.0 port** on the Proxmox host (not USB 3.0). Many reports say USB 2.0 is more reliable for rpiboot.
2. **Avoid udev fighting with manual rpiboot:** Temporarily disable the provisioning trigger so only your manual rpiboot runs:  
   `ssh root@HOST 'mv /etc/cm4-provisioning/enabled /etc/cm4-provisioning/enabled.bak'`  
   Run rpiboot, then re-enable:  
   `ssh root@HOST 'mv /etc/cm4-provisioning/enabled.bak /etc/cm4-provisioning/enabled'`
3. **Try a different USB port and cable** on the host; unplug/replug and retry.
4. **Add a delay** so the device is ready before transfer:  
   `rpiboot -v -d /opt/usbboot/mass-storage-gadget64 -m 2000`  
   (`-m` is microseconds between device checks; 2000 = 2 ms.)
5. **Reboot the Proxmox host** and try again (USB controller state can get stuck).
6. **Find a USB 2.0 port:** On the host run `lsusb -t` — look for the BCM2711 device; the tree shows which controller (e.g. "xHCI" = USB 3, "ehci" = USB 2). Try a port that is under an **ehci** or **ohci** controller, or a black (non‑blue) physical port.
7. **Try a powered USB 2.0 hub** between host and reTerminal (some hosts work only through a hub).
8. If you have the **working host** (10.130.60.224), try the same reTerminal there; if it works there, the difference is host USB controller or port.

3b. **"No suitable block device after rpiboot"** – rpiboot ran but no new block device was seen. Detection is **dynamic**: any block device that appears after rpiboot (not present before) is used, so 8/16/32 GB CM4 work without setting eMMC size. **Check on host:** `tail -80 /var/lib/cm4-provisioning/flash.log` — at the end you’ll see "Current block devices" and each `/dev/sdX` with size. **Causes:** (1) Device didn’t switch to mass storage (try unplug/replug, keep eMMC disable jumper set). (2) udev slow — try again; the script waits up to 90s for the device.

4. **"No 'bootcode' files found in mass-storage-gadget64"** – Usually because `bootfiles.bin` is a **broken symlink** (e.g. `-> ../firmware/bootfiles.bin`) and that target doesn’t exist. **Fix on host:** run `scripts/fix-gadget-bootcode-on-host.sh` on the host (it removes the symlink and extracts `bootcode4.bin` from the installed rpiboot binary). From your machine: `ssh root@10.130.60.224 'bash -s' < scripts/fix-gadget-bootcode-on-host.sh`. **Alternative:** repopulate the gadget dir with `./scripts/populate-gadget-on-host.sh root@10.130.60.224`, or full reinstall with `./scripts/build-and-deploy-usbboot-to-host.sh root@10.130.60.224`. Then verify: `ls -la /opt/usbboot/mass-storage-gadget64/` (should list a real `bootcode4.bin` or `bootfiles.bin`, plus `boot.img`, `config.txt`).

4. **Clear stuck error in portal** – If the portal shows an old error (e.g. "Golden image not found" or "rpiboot failed"), click **Clear message** in the dashboard, or: `ssh root@10.130.60.224 "echo '{\"phase\":\"idle\",\"message\":\"Waiting for Device in USB boot mode.\",\"progress\":null}' > /var/lib/cm4-provisioning/status.json"`. Then unplug/replug the device.

5. **"PiShrink not installed" when clicking Shrink/Compress** – Shrink and Compress run **on the host**, not in the LXC. Install PiShrink on the host: `ssh root@HOST 'bash -s' < emmc-provisioning/scripts/install-pishrink-on-host.sh`. Ensure deploy has been run so the host has `run-shrink-on-host.sh` and `cm4-shrink.path` enabled (`systemctl status cm4-shrink.path`).

6. **"Download failed" when building cloud-init image** – The download runs on the **host** (not the LXC). Check: (1) Host can reach the internet: `ssh root@HOST 'curl -sI https://downloads.raspberrypi.com/'`. (2) Build status now shows curl’s error (e.g. "Could not resolve host", "Connection timed out"); check the dashboard error text. (3) If you use a proxy or custom CA, set in `/opt/cm4-provisioning/env`: `CURL_INSECURE=1` to skip SSL verify (only if you understand the risk), then rerun the build.

7. **Backup stops before finishing** – If backup or shrink appears to stop partway (e.g. dashboard stuck on "Creating backup…" or "Shrinking…"), the service may have been killed by systemd. The `cm4-flash.service` unit uses `TimeoutStartSec=7200` (2 hours); if you deployed an older version with 15 minutes, redeploy so the host gets the updated unit, then on the host run `systemctl daemon-reload` so the next backup has enough time to complete.

8. **Trigger now runs the flash script in the background** (not via systemd-run) so it can access the USB device; a 2s delay gives the device time to enumerate before rpiboot runs.

---

## Redeploy / update scripts

From your repo (e.g. after changing scripts):

```bash
./emmc-provisioning/scripts/deploy-to-proxmox.sh root@10.130.60.224
```

That script syncs the repo to the host and reinstalls scripts on both the host and LXC 201. It does **not** overwrite `/opt/cm4-provisioning/env` or `/etc/cm4-provisioning/enabled` if you’ve changed them; adjust the script if you want that. It also does **not** build usbboot; run `install-usbboot-on-host.sh` on the host when needed.

---

## Summary

| Item | Location |
|------|----------|
| LXC | Hostname `cm4-provisioning` (ID from `pct list`), on your Proxmox host |
| Golden image | `/var/lib/cm4-provisioning/golden.img` (host and LXC see the same file) |
| Flash runs on | Proxmox **host** (udev + rpiboot + dd) |
| Build rpiboot on host | Run `scripts/install-usbboot-on-host.sh` on the host when it has internet |
| Dashboard | Flask app in LXC at `http://<LXC-IP>:5000`; switch Flash/Backup mode, list and download backups; see **dashboard/README.md** and section 3 above |
| Backups | Saved under `/var/lib/cm4-provisioning/backups/` (optionally a host path bind-mounted into the LXC — set `CM4_BACKUPS_HOST_PATH` at deploy). When a device is detected, choose **Backup** or **Deploy** in the dashboard. |
| Network deploy/backup | Network-booted devices run **network-client/provisioning-client.sh** and register with the dashboard; they then appear under "Device detected (Network)" and you choose Backup or Deploy. See **network-client/README.md**. |
| Network boot (DHCP + TFTP on eth1) | If the LXC has a second interface **eth1** as provisioning LAN (e.g. 10.20.50.1/24), run **scripts/setup-network-boot-on-lxc.sh root@&lt;LXC-IP&gt;** to install dnsmasq (DHCP+TFTP) on eth1 and NAT so LAN clients get internet via eth0. See **docs/NETWORK-BOOT-LXC.md**. |
