249 lines
16 KiB
Markdown
249 lines
16 KiB
Markdown
# CM4 eMMC provisioning on Proxmox (LXC + host)
|
||
|
||
The auto-flash **runs on the Proxmox host** (where the USB device appears). The **LXC** holds the dashboard and shares the **golden image** directory with the host.
|
||
|
||
## One-command deploy
|
||
|
||
From your repo, a single run deploys **all** host and LXC files (scripts, systemd units, udev, dashboard):
|
||
|
||
```bash
|
||
./emmc-provisioning/scripts/deploy-to-proxmox.sh root@YOUR_PROXMOX_HOST
|
||
```
|
||
|
||
Optional env: `CM4_BACKUPS_HOST_PATH=/path`, `DEPLOY_ROOTFS_STORAGE=local-lvm`, `DEPLOY_LXC_ROOT_PASSWORD=secret` (set root password in LXC and enable SSH), `DEPLOY_LXC_SSH_KEY=/path/to/pub` (default: `~/.ssh/id_ed25519.pub` or `id_rsa.pub` — copied to LXC root so you can `ssh root@<LXC-IP>`).
|
||
|
||
The script **finds the container by hostname `cm4-provisioning`** (any existing ID). If none exists, it **creates a new LXC with the next available ID**. So you can redeploy repeatedly without assuming a fixed ID like 201.
|
||
|
||
## What is deployed
|
||
|
||
| Where | What |
|
||
|-------|-----|
|
||
| **Proxmox host** | udev rule, trigger script, flash script, build-cloudinit and run-shrink scripts, systemd path units (build + shrink), `/var/lib/cm4-provisioning/`, `/etc/cm4-provisioning/enabled` |
|
||
| **LXC (hostname cm4-provisioning)** | Dashboard (Flask) in `/opt/cm4-provisioning/dashboard/`; `/var/lib/cm4-provisioning/` is a **bind mount** from the host (shared storage for golden image and backups) |
|
||
|
||
When you plug the reTerminal in boot mode into the **host**, udev on the host runs the flash (rpiboot + dd). The golden image is read from `/var/lib/cm4-provisioning/golden.img` on the host (same path visible in the LXC).
|
||
|
||
---
|
||
|
||
## Deployment layout (after running the deploy script)
|
||
|
||
1. **LXC** (hostname `cm4-provisioning`, ID = found by hostname or next free):
|
||
- Debian 12, 1 GB RAM, 8 GB rootfs
|
||
- Bind mount: host `/var/lib/cm4-provisioning` → container `/var/lib/cm4-provisioning`
|
||
- Optional second mount: `CM4_BACKUPS_HOST_PATH` → container `/var/lib/cm4-provisioning/backups`
|
||
|
||
2. **On the host**:
|
||
- `/opt/cm4-provisioning/flash-emmc-on-connect.sh` – flash script
|
||
- `/opt/cm4-provisioning/build-cloudinit-image.sh` – build cloud-init image (triggered by path unit)
|
||
- `/opt/cm4-provisioning/run-shrink-on-host.sh` – PiShrink for dashboard Shrink/Compress
|
||
- `/usr/local/bin/cm4-flash-trigger.sh` – started by udev
|
||
- `/etc/udev/rules.d/90-cm4-boot-mode.rules` – run trigger when USB vendor `2b8e` is added
|
||
- `/opt/cm4-provisioning/env` – `GOLDEN_IMAGE`, `RPIBOOT_DIR`, `EMMC_SIZE_BYTES` (and `BACKUPS_DIR` if `CM4_BACKUPS_HOST_PATH` set)
|
||
- `/etc/cm4-provisioning/enabled` – safety switch (remove to disable auto-flash)
|
||
- systemd: `cm4-build-cloudinit.path` + `.service`, `cm4-shrink.path` + `.service`
|
||
- `/opt/cm4-provisioning/fix-gadget-bootcode-on-host.sh` – used by `install-usbboot-on-host.sh` after building usbboot (fixes "rpiboot gadget empty" when gadget has broken symlinks)
|
||
|
||
3. **Inside the LXC** (use `pct exec <CTID> -- ...` where `<CTID>` is the ID of the container with hostname `cm4-provisioning`; get it with `pct list`):
|
||
- Dashboard: Flask app in `/opt/cm4-provisioning/dashboard/` (monitor deployment, backup list, build cloud-init, set golden).
|
||
- Golden image path: `/var/lib/cm4-provisioning/golden.img` (bind-mounted from host).
|
||
|
||
4. **usbboot (rpiboot)** is **not** installed by the deploy script. Install it when the host has internet (see below).
|
||
|
||
---
|
||
|
||
## What you need to do
|
||
|
||
### 1. Build and install rpiboot on the Proxmox host (when it has internet)
|
||
|
||
On your machine (repo already synced to the host):
|
||
|
||
```bash
|
||
# From your repo
|
||
scp emmc-provisioning/scripts/install-usbboot-on-host.sh root@10.130.60.224:/tmp/
|
||
ssh root@10.130.60.224 "bash /tmp/install-usbboot-on-host.sh"
|
||
```
|
||
|
||
Or on the host (if the deploy folder is still there):
|
||
|
||
```bash
|
||
ssh root@10.130.60.224
|
||
bash /tmp/emmc-provisioning-deploy/scripts/install-usbboot-on-host.sh
|
||
```
|
||
|
||
This installs dependencies, clones usbboot, builds it, and copies `rpiboot` to `/opt/usbboot/`. It then runs **fix-gadget-bootcode-on-host.sh** if present (from deploy), so the gadget has valid boot files and rpiboot does not fail with "No bootcode files found".
|
||
|
||
### 2. Enable root SSH and add your SSH key to the LXC
|
||
|
||
If you deployed with **`DEPLOY_LXC_ROOT_PASSWORD`** and/or a default SSH key (**`~/.ssh/id_ed25519.pub`** or **`id_rsa.pub`**), the LXC already has SSH enabled, root password set, and your key in `/root/.ssh/authorized_keys` — you can **skip** to `ssh root@<LXC-IP>` (get IP from deploy output or `pct exec <CTID> -- hostname -I`).
|
||
|
||
Otherwise, to enable root SSH and add a key:
|
||
|
||
- **Option A – Use the setup script (recommended):** From your machine (with SSH key and optional password):
|
||
|
||
```bash
|
||
# Add your default SSH key (~/.ssh/id_ed25519.pub or id_rsa.pub) and enable root SSH
|
||
./emmc-provisioning/scripts/setup-lxc-ssh.sh root@10.130.60.224
|
||
|
||
# Or specify key file and set root password
|
||
ROOT_PASSWORD='YourPassword' ./emmc-provisioning/scripts/setup-lxc-ssh.sh root@10.130.60.224 ~/.ssh/id_ed25519.pub
|
||
```
|
||
|
||
Then connect with `ssh root@<LXC-IP>` (script prints the IP). To get the LXC IP:
|
||
`ssh root@HOST "CID=\$(pct list -no-header -output vmid,name | awk '\''\$2==\"cm4-provisioning\"{print \$1}'\''); pct exec \$CID -- hostname -I"`
|
||
|
||
- **Option B – Manual:**
|
||
`ssh root@HOST`, then `pct exec <CTID> -- bash` (use the container ID from `pct list` for hostname cm4-provisioning). Install openssh-server, set `PermitRootLogin yes`, set root password, add your key, restart ssh.
|
||
|
||
### 3. (Optional) Store backup images on a host directory
|
||
|
||
To keep backup images on a specific host path (e.g. a large disk or NFS mount) instead of under `/var/lib/cm4-provisioning/backups`, deploy with **`CM4_BACKUPS_HOST_PATH`** set. That directory is created on the host, bind-mounted into the LXC at `/var/lib/cm4-provisioning/backups`, and the host flash script is configured to write backups there. The dashboard in the LXC then lists and serves those same files.
|
||
|
||
**Deploy with a host backup path:**
|
||
|
||
```bash
|
||
CM4_BACKUPS_HOST_PATH=/mnt/storage/cm4-backups ./emmc-provisioning/scripts/deploy-to-proxmox.sh root@10.130.60.224
|
||
```
|
||
|
||
Create `/mnt/storage/cm4-backups` (or your path) on the host first if it doesn’t exist; the deploy script will create it if possible. To add or change the backup mount, set `CM4_BACKUPS_HOST_PATH` and run the deploy script again (it reuses the container by hostname and updates the bind mount).
|
||
|
||
### 4. Put the golden image on the host (or in the LXC)
|
||
|
||
The image must be at **`/var/lib/cm4-provisioning/golden.img`** on the **host**. Because that directory is bind-mounted into the LXC, you can use either:
|
||
|
||
- **From the host:**
|
||
```bash
|
||
scp your-golden.img root@10.130.60.224:/var/lib/cm4-provisioning/golden.img
|
||
```
|
||
|
||
- **From the LXC** (e.g. after copying the image into the container elsewhere first):
|
||
```bash
|
||
pct exec <CTID> -- ls -la /var/lib/cm4-provisioning/
|
||
# Replace <CTID> with the ID of the cm4-provisioning container (pct list).
|
||
```
|
||
|
||
### 5. Run the provisioning dashboard (optional)
|
||
|
||
The dashboard shows **connection steps** and **live deployment status** (idle / connecting / flashing / done / error) and a recent flash log. It reads the same `status.json` and `flash.log` that the host’s flash script writes (via the bind-mounted `/var/lib/cm4-provisioning`).
|
||
|
||
**Inside the LXC (pct exec <CTID> -- bash):**
|
||
|
||
```bash
|
||
# Copy dashboard into the container (from host, if you have the repo there)
|
||
# Or from your workstation:
|
||
# rsync -a emmc-provisioning/dashboard/ root@10.130.60.224:/tmp/dashboard/
|
||
# Or re-run deploy-to-proxmox.sh to push the latest dashboard files.
|
||
|
||
# Inside the LXC (pct exec <CTID> -- bash):
|
||
apt-get update && apt-get install -y python3-flask
|
||
mkdir -p /opt/cm4-provisioning/dashboard/templates
|
||
# Copy app.py, templates/index.html, cm4-dashboard.service into the container (see dashboard/README.md)
|
||
|
||
cp /opt/cm4-provisioning/dashboard/cm4-dashboard.service /etc/systemd/system/
|
||
systemctl daemon-reload
|
||
systemctl enable --now cm4-dashboard
|
||
```
|
||
|
||
Then open **http://<LXC-IP>:5000** (get the IP with `pct exec <CTID> -- hostname -I`). If the LXC is on a private network, set up port forwarding on the Proxmox host or use a reverse proxy.
|
||
|
||
### 6. Optional: disable or enable auto-flash
|
||
|
||
- **Disable:**
|
||
`ssh root@10.130.60.224 "rm /etc/cm4-provisioning/enabled"`
|
||
|
||
- **Enable again:**
|
||
`ssh root@10.130.60.224 "touch /etc/cm4-provisioning/enabled"`
|
||
|
||
---
|
||
|
||
## Usage
|
||
|
||
1. Place the reTerminal in **boot mode** (eMMC disable jumper).
|
||
2. Connect its **USB slave** port to the **Proxmox host** (not to the LXC).
|
||
3. Power the reTerminal (or connect after power).
|
||
4. On the host, udev will run the trigger and then the flash script (rpiboot, then dd). Watch logs:
|
||
```bash
|
||
ssh root@10.130.60.224 "journalctl -u cm4-flash-once -f"
|
||
# or
|
||
ssh root@10.130.60.224 "journalctl -t cm4-flash -f"
|
||
```
|
||
5. When flashing finishes, remove the jumper and power cycle the reTerminal so it boots from eMMC.
|
||
|
||
---
|
||
|
||
## Monitoring from the host
|
||
|
||
From the **Proxmox host** you can monitor:
|
||
|
||
| What | How |
|
||
|------|-----|
|
||
| **USB device** | `lsusb` — CM4 in boot mode shows as **2b8e** (RPi) or **0a5c:2711** (Broadcom BCM2711) |
|
||
| **Live status** | `cat /var/lib/cm4-provisioning/status.json` — same JSON the dashboard shows (phase, message, error) |
|
||
| **Flash log** | `tail -f /var/lib/cm4-provisioning/flash.log` — script log (rpiboot, dd, errors) |
|
||
| **Flash job** | `systemctl status cm4-flash-once` — whether the udev-triggered job is running/failed |
|
||
| **Journal** | `journalctl -u cm4-flash-once -f` or `journalctl -t cm4-flash -f` — systemd/log output |
|
||
| **Block devices** | `lsblk` — after rpiboot, the eMMC appears as a new disk (e.g. `/dev/sdb`) |
|
||
| **Backups** | `ls /var/lib/cm4-provisioning/backups/` — backup images (on host; if you used `CM4_BACKUPS_HOST_PATH` they are under that path on the host, bind-mounted into the LXC). To shrink automatically, set `SHRINK_BACKUP=1` in `/opt/cm4-provisioning/env` — see **EMMC-PROVISIONING-GUIDE.md** § Shrinking backup and golden images. |
|
||
| **Config** | `cat /opt/cm4-provisioning/env` — GOLDEN_IMAGE, RPIBOOT_DIR, EMMC_SIZE_BYTES |
|
||
|
||
**One-command snapshot:**
|
||
|
||
```bash
|
||
# From your machine (stream script to host):
|
||
ssh root@10.130.60.224 'bash -s' < emmc-provisioning/scripts/monitor-from-host.sh
|
||
```
|
||
|
||
Or copy `scripts/monitor-from-host.sh` to the host and run `./monitor-from-host.sh` for a full status dump (USB, status.json, flash unit, last log lines, block devices, config).
|
||
|
||
---
|
||
|
||
## Troubleshooting: device connected but not shown in portal
|
||
|
||
1. **Host has old flash script** – The script must *not* exit when the golden image is missing (so you can use Backup first). Update the host:
|
||
```bash
|
||
scp emmc-provisioning/host/flash-emmc-on-connect.sh root@10.130.60.224:/opt/cm4-provisioning/
|
||
ssh root@10.130.60.224 "chmod +x /opt/cm4-provisioning/flash-emmc-on-connect.sh"
|
||
```
|
||
|
||
2. **Unplug and replug the USB** – udev runs the trigger only when the device is *added*. Unplug the reTerminal USB (keep it in boot mode), then plug it back in. The trigger will run the script and rpiboot; when the eMMC is exposed, the portal shows "Device connected" with Backup/Deploy.
|
||
|
||
3. **If rpiboot fails** – Check on the host: `ssh root@10.130.60.224 'tail -30 /var/lib/cm4-provisioning/flash.log'` (rpiboot stderr is appended there). Try unplug/replug again. To see the exact rpiboot error: `ssh root@10.130.60.224 '/opt/usbboot/rpiboot -d /opt/usbboot/mass-storage-gadget64'` (device connected; Ctrl+C to stop). Run `scripts/monitor-from-host.sh` for a full snapshot.
|
||
|
||
4. **"No 'bootcode' files found in mass-storage-gadget64"** – Usually because `bootfiles.bin` is a **broken symlink** (e.g. `-> ../firmware/bootfiles.bin`) and that target doesn’t exist. **Fix on host:** run `scripts/fix-gadget-bootcode-on-host.sh` on the host (it removes the symlink and extracts `bootcode4.bin` from the installed rpiboot binary). From your machine: `ssh root@10.130.60.224 'bash -s' < scripts/fix-gadget-bootcode-on-host.sh`. **Alternative:** repopulate the gadget dir with `./scripts/populate-gadget-on-host.sh root@10.130.60.224`, or full reinstall with `./scripts/build-and-deploy-usbboot-to-host.sh root@10.130.60.224`. Then verify: `ls -la /opt/usbboot/mass-storage-gadget64/` (should list a real `bootcode4.bin` or `bootfiles.bin`, plus `boot.img`, `config.txt`).
|
||
|
||
4. **Clear stuck error in portal** – If the portal shows an old error (e.g. "Golden image not found" or "rpiboot failed"), click **Clear message** in the dashboard, or: `ssh root@10.130.60.224 "echo '{\"phase\":\"idle\",\"message\":\"Waiting for reTerminal in boot mode or network.\",\"progress\":null}' > /var/lib/cm4-provisioning/status.json"`. Then unplug/replug the device.
|
||
|
||
5. **"PiShrink not installed" when clicking Shrink/Compress** – Shrink and Compress run **on the host**, not in the LXC. Install PiShrink on the host: `ssh root@HOST 'bash -s' < emmc-provisioning/scripts/install-pishrink-on-host.sh`. Ensure deploy has been run so the host has `run-shrink-on-host.sh` and `cm4-shrink.path` enabled (`systemctl status cm4-shrink.path`).
|
||
|
||
6. **"Download failed" when building cloud-init image** – The download runs on the **host** (not the LXC). Check: (1) Host can reach the internet: `ssh root@HOST 'curl -sI https://downloads.raspberrypi.com/'`. (2) Build status now shows curl’s error (e.g. "Could not resolve host", "Connection timed out"); check the dashboard error text. (3) If you use a proxy or custom CA, set in `/opt/cm4-provisioning/env`: `CURL_INSECURE=1` to skip SSL verify (only if you understand the risk), then rerun the build.
|
||
|
||
7. **Backup stops before finishing** – If backup or shrink appears to stop partway (e.g. dashboard stuck on "Creating backup…" or "Shrinking…"), the service may have been killed by systemd. The `cm4-flash.service` unit uses `TimeoutStartSec=7200` (2 hours); if you deployed an older version with 15 minutes, redeploy so the host gets the updated unit, then on the host run `systemctl daemon-reload` so the next backup has enough time to complete.
|
||
|
||
8. **Trigger now runs the flash script in the background** (not via systemd-run) so it can access the USB device; a 2s delay gives the device time to enumerate before rpiboot runs.
|
||
|
||
---
|
||
|
||
## Redeploy / update scripts
|
||
|
||
From your repo (e.g. after changing scripts):
|
||
|
||
```bash
|
||
./emmc-provisioning/scripts/deploy-to-proxmox.sh root@10.130.60.224
|
||
```
|
||
|
||
That script syncs the repo to the host and reinstalls scripts on both the host and LXC 201. It does **not** overwrite `/opt/cm4-provisioning/env` or `/etc/cm4-provisioning/enabled` if you’ve changed them; adjust the script if you want that. It also does **not** build usbboot; run `install-usbboot-on-host.sh` on the host when needed.
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
| Item | Location |
|
||
|------|----------|
|
||
| LXC | Hostname `cm4-provisioning` (ID from `pct list`), on your Proxmox host |
|
||
| Golden image | `/var/lib/cm4-provisioning/golden.img` (host and LXC see the same file) |
|
||
| Flash runs on | Proxmox **host** (udev + rpiboot + dd) |
|
||
| Build rpiboot on host | Run `scripts/install-usbboot-on-host.sh` on the host when it has internet |
|
||
| Dashboard | Flask app in LXC at `http://<LXC-IP>:5000`; switch Flash/Backup mode, list and download backups; see **dashboard/README.md** and section 3 above |
|
||
| Backups | Saved under `/var/lib/cm4-provisioning/backups/` (optionally a host path bind-mounted into the LXC — set `CM4_BACKUPS_HOST_PATH` at deploy). When a device is detected, choose **Backup** or **Deploy** in the dashboard. |
|
||
| Network deploy/backup | Network-booted devices run **network-client/provisioning-client.sh** and register with the dashboard; they then appear under "Device detected (Network)" and you choose Backup or Deploy. See **network-client/README.md**. |
|
||
| Network boot (DHCP + TFTP on eth1) | If the LXC has a second interface **eth1** as provisioning LAN (e.g. 10.20.50.1/24), run **scripts/setup-network-boot-on-lxc.sh root@<LXC-IP>** to install dnsmasq (DHCP+TFTP) on eth1 and NAT so LAN clients get internet via eth0. See **docs/NETWORK-BOOT-LXC.md**. |
|