reterminal-dm4/emmc-provisioning/docs/EMMC-PROVISIONING-GUIDE.md

# eMMC provisioning for reTerminal DM4 (CM4)

This guide covers:

1. **USB boot mode**: When the reTerminal is in boot mode (eMMC disable jumper) and connected via USB, the host runs **rpiboot** to expose the eMMC, then the **dashboard** shows "Device connected (USB)". You choose **Backup** or **Deploy** in the portal — there is no auto-flash; the action runs only after your choice.
2. **Network boot**: If the device boots over the network and runs the **provisioning client** (see `network-client/`), it registers with the dashboard and appears as "Device (Network)"; you then choose Backup or Deploy. Deploy streams the golden image to the device; Backup uploads the device eMMC to the server.
3. **Cloud-init**: The golden image can include cloud-init so each device configures itself on first boot (hostname, network, packages, kiosk setup).

---

## Part 1: USB boot mode — detect device, choose Backup or Deploy in portal

### How it works

- reTerminal has an **eMMC disable** jumper (see reTerminal docs; often “J2” or “nRPIBOOT”). When the jumper is fitted, the CM4 boots in **USB device mode** and waits for `rpiboot` from the host.
- You connect the reTerminal’s **USB slave** port to a **provisioning PC** (Linux).
- **udev** detects the Raspberry Pi Foundation USB device (vendor `2b8e`) and runs a trigger script.
- The trigger starts the **provisioning script** that:
  1. Runs **rpiboot** (from the `usbboot` project). The CM4 then exposes its eMMC as a USB mass-storage device.
  2. Finds the new block device (eMMC) and writes status so the **dashboard** shows "Device connected (USB boot mode). Choose Backup or Deploy in the dashboard."
  3. **Waits for your choice in the portal** — no automatic flash. When you click **Backup** or **Deploy** in the dashboard, the script runs that action (dd backup or dd deploy).
- You remove the jumper and power cycle; the reTerminal boots from eMMC and can run **cloud-init** on first boot.

### Provisioning host setup (Linux)

#### 1. Build and install usbboot (rpiboot)

```bash
sudo apt-get install -y libusb-1.0-0-dev
git clone --depth=1 https://github.com/raspberrypi/usbboot
cd usbboot
make
sudo mkdir -p /opt/usbboot
sudo cp rpiboot /opt/usbboot/
```

#### 2. Create golden image and config directory

- Build your golden image (see Part 2) and place it where the script will find it, e.g.:

  ```bash
  sudo mkdir -p /var/lib/cm4-provisioning
  sudo cp /path/to/your/golden-reterminal.img /var/lib/cm4-provisioning/golden.img
  ```

- Or use a different path and set `GOLDEN_IMAGE` when installing the script (see below). On Proxmox with LXC, you can store backup images on a host directory by setting `CM4_BACKUPS_HOST_PATH` at deploy so that folder is bind-mounted into the LXC — see **PROXMOX-LXC-DEPLOYMENT.md** § Store backup images on a host directory.

#### 3. Install the provisioning script and trigger

```bash
# From this repo (emmc-provisioning/host/)
cd emmc-provisioning/host

sudo mkdir -p /opt/cm4-provisioning
sudo cp flash-emmc-on-connect.sh /opt/cm4-provisioning/
sudo chmod +x /opt/cm4-provisioning/flash-emmc-on-connect.sh

# Optional: override paths via environment (create env file)
echo 'GOLDEN_IMAGE=/var/lib/cm4-provisioning/golden.img' | sudo tee /opt/cm4-provisioning/env
echo 'RPIBOOT_DIR=/opt/usbboot' | sudo tee -a /opt/cm4-provisioning/env
echo 'EMMC_SIZE_BYTES=8589934592' | sudo tee -a /opt/cm4-provisioning/env   # 8GB; use 17179869184 for 16GB
# Optional: shrink backups after dd (requires PiShrink; see "Shrinking backup and golden images" below)
# echo 'SHRINK_BACKUP=1' | sudo tee -a /opt/cm4-provisioning/env
# echo 'PISHRINK_COMPRESS=gz' | sudo tee -a /opt/cm4-provisioning/env   # or xz

sudo cp cm4-flash-trigger.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/cm4-flash-trigger.sh
```

If your golden image path or rpiboot path is different, set `GOLDEN_IMAGE`, `RPIBOOT_DIR`, and optionally `EMMC_SIZE_BYTES` in `/opt/cm4-provisioning/env` and source it from the script, or pass them into the systemd-run call in the trigger (e.g. by making the trigger source the env file and export variables before `systemd-run`).

#### 4. Install udev rule

```bash
# From emmc-provisioning/host/
sudo cp 90-cm4-boot-mode.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules
sudo udevadm trigger
```

#### 5. Enable provisioning (safety)

Provisioning runs only if the “enabled” file exists:

```bash
sudo mkdir -p /etc/cm4-provisioning
sudo touch /etc/cm4-provisioning/enabled
```

To disable provisioning (no device detection), remove that file: `sudo rm /etc/cm4-provisioning/enabled`.

#### 6. Optional: pass environment into the provisioning job

If you use `/opt/cm4-provisioning/env`, update the trigger so the flash script sees those variables. For example change `/usr/local/bin/cm4-flash-trigger.sh` to:

```bash
#!/usr/bin/env bash
set -a
[[ -f /opt/cm4-provisioning/env ]] && source /opt/cm4-provisioning/env
set +a
export GOLDEN_IMAGE RPIBOOT_DIR EMMC_SIZE_BYTES
FLASH_SCRIPT="${CM4_FLASH_SCRIPT:-/opt/cm4-provisioning/flash-emmc-on-connect.sh}"
exec systemd-run --no-block --unit=cm4-flash-once --property=Environment="GOLDEN_IMAGE=$GOLDEN_IMAGE" ...
```

Or keep it simple and edit the defaults inside `flash-emmc-on-connect.sh` (e.g. `GOLDEN_IMAGE`, `RPIBOOT_DIR`, `EMMC_SIZE_BYTES`).

### Usage

1. Fit the **eMMC disable** jumper on the reTerminal.
2. Connect the reTerminal **USB slave** port to the provisioning PC.
3. Power the reTerminal (or apply power after USB).
4. On the host, `rpiboot` will run automatically; when it exits, the script will `dd` the golden image to the eMMC. Watch logs: `journalctl -u cm4-flash-once -f` or `journalctl -t cm4-flash -f`.
5. When done, remove the jumper and power cycle the reTerminal. It will boot from eMMC; cloud-init will run on first boot.

---

## Part 2: Golden image with cloud-init

Raspberry Pi OS (recent versions) supports **cloud-init** using the **NoCloud** datasource: it reads `user-data`, `meta-data`, and optionally `network-config` from the **boot** (FAT32) partition.

### Steps to prepare a cloud-init image for Raspberry Pi OS

1. **Obtain Raspberry Pi OS**
   - Download from [raspberrypi.com/software](https://www.raspberrypi.com/software/) (desktop or Lite), or use **Raspberry Pi Imager** with "Edit settings" for locale/SSH. Recent Raspberry Pi OS has cloud-init built in.

2. **Flash the image**
   - Flash to a spare SD card, or to a loop file for building without physical media: `cp raspios.img golden-work.img`

3. **Mount the boot partition**
   - **From an image file**: `sudo losetup -fP golden-work.img` then e.g. `sudo mount /dev/loop0p1 /mnt/boot`
   - **From SD**: mount the first (FAT32) partition at `/mnt/boot`. On a running Pi, boot is often `/boot/firmware`.

4. **Add NoCloud files on the boot partition** (root of FAT32, same level as `config.txt`):
   - `user-data`, `meta-data`, `network-config`
   - From this repo: `cp emmc-provisioning/cloud-init/{user-data,meta-data,network-config} /mnt/boot/`

5. **Customise** `user-data` and `network-config`. Use the **remote bootstrap script** pattern (below) to avoid rebuilding the image when you change first-boot commands.

6. **Unmount and create golden image**: `sudo umount /mnt/boot`, then copy the image to your `GOLDEN_IMAGE` path (e.g. `/var/lib/cm4-provisioning/golden.img`).

### Remote bootstrap script (no image rebuild for script changes)

You can keep the golden image **fixed** and have cloud-init **download a script from a file server** and run it on first boot. When you change the script on the server, the next device gets the new commands without rebuilding the image.

1. **Host the script** on an HTTP/HTTPS file server (e.g. nginx, or `python3 -m http.server`) at a URL the Pi can reach, e.g. `http://192.168.1.10/provisioning/bootstrap.sh`.

2. **In `user-data`, use `runcmd` to download and run it** (runcmd runs after packages and network are up):

   ```yaml
   #cloud-config
   package_update: true
   package_upgrade: false
   packages:
     - curl   # or wget

   runcmd:
     - curl -fsSL "http://YOUR_FILE_SERVER/provisioning/bootstrap.sh" -o /tmp/bootstrap.sh
     - chmod +x /tmp/bootstrap.sh
     - /tmp/bootstrap.sh
   ```

   With **wget**: `wget -q -O /tmp/bootstrap.sh "http://.../bootstrap.sh"` then `chmod +x /tmp/bootstrap.sh` and `/tmp/bootstrap.sh`.

3. **Bootstrap script**: use a normal shell script (e.g. `#!/bin/bash` and `set -e`). It can install packages, configure kiosk, set hostname, register with a dashboard, etc.

4. **Notes**: Ensure `network-config` (or DHCP) gives the Pi an IP before runcmd. For HTTPS, add `ca-certificates` to `packages` if needed. You can use one script for all devices or have the script call your server with serial/MAC for device-specific config.

### Creating the golden image

1. **Flash Raspberry Pi OS** (or your base image) to a spare SD card or a loop file.
2. **Mount the boot partition** (first partition, FAT32). On the image file it might be at an offset; use `losetup -P` or mount the SD’s partition.
3. **Add cloud-init NoCloud files** on the boot partition (same level as `config.txt`, not in a subfolder for default NoCloud):
   - `user-data` – main config (packages, runcmd, etc.)
   - `meta-data` – optional (instance-id, local-hostname)
   - `network-config` – optional (network config in netplan format)

You can use the examples in this repo:

```bash
# After mounting boot partition at e.g. /mnt/boot
# (On Raspberry Pi OS, boot is often /boot/firmware on the running system, or the first FAT partition of the image)
cp emmc-provisioning/cloud-init/user-data /mnt/boot/
cp emmc-provisioning/cloud-init/meta-data /mnt/boot/
cp emmc-provisioning/cloud-init/network-config /mnt/boot/
```

4. **Customise** `user-data` and `network-config` (hostname, WiFi, packages, Chromium kiosk, or a single runcmd that downloads and runs a remote script—see "Remote bootstrap script" above).
5. **Copy your kiosk/Chromium scripts** into the image rootfs only if you are not using a remote script; otherwise the remote script can pull what it needs.
6. **Unmount**, then create a **golden image** from the SD or loop device (e.g. `dd` or copy of the whole block device). Use that as `golden.img` on the provisioning host. Optionally shrink it with PiShrink (see below) to save space and speed up deploy.

### Shrinking backup and golden images (PiShrink)

Raw full-disk backups and golden images are the full size of the eMMC (e.g. 32 GB). [PiShrink](https://github.com/Drewsif/PiShrink) shrinks the **last partition** (must be ext2/3/4) to its minimum size and truncates the image file; on first boot the rootfs can expand back to fill the device. This reduces backup/golden image size (often to a few GB) and improves transfer times.

**On the provisioning host:**

1. **Install PiShrink and dependencies** (run as root on the host, or via `ssh root@HOST 'bash -s' < scripts/install-pishrink-on-host.sh`):

   ```bash
   # From this repo (on the host or via ssh)
   bash emmc-provisioning/scripts/install-pishrink-on-host.sh
   ```

   This installs `parted`, `e2fsprogs`, `gzip`, `pigz`, `xz-utils` and downloads `pishrink.sh` to `/usr/local/bin/pishrink.sh`.

2. **Shrink backups automatically** after each Backup in the dashboard: add to `/opt/cm4-provisioning/env` and ensure the trigger sources it:

   ```bash
   echo 'SHRINK_BACKUP=1' | sudo tee -a /opt/cm4-provisioning/env
   # Optional: compress after shrinking (smaller file; must decompress before using as golden image)
   # echo 'PISHRINK_COMPRESS=gz' | sudo tee -a /opt/cm4-provisioning/env   # good balance
   echo 'PISHRINK_COMPRESS=xz' | sudo tee -a /opt/cm4-provisioning/env   # minimum size (slower)
   ```

   With `SHRINK_BACKUP=1`, once a backup finishes, the script runs PiShrink on the `.img` file. Use **`PISHRINK_COMPRESS=xz`** for **minimum size** (smallest file, slower); or **`gz`** for a good balance. The file becomes `.img.xz` or `.img.gz` and must be decompressed before deploy (e.g. `xz -dk backup.img.xz` then copy to `golden.img`).

3. **Shrink a golden image manually** (e.g. after building from Raspberry Pi OS):

   ```bash
   sudo pishrink.sh -n /path/to/large.img /var/lib/cm4-provisioning/golden.img
   ```

   `-n` disables update check. Omit the second argument to shrink in place. The shrunk image will expand the rootfs on first boot when deployed to eMMC.

**Notes:**

- PiShrink only shrinks the **last** partition; it must be ext2/3/4 (standard Raspberry Pi OS root is ext4).
- Compressed backups (`.img.gz` / `.img.xz`) are for archival; to use as golden image, decompress first (e.g. `gunzip -k backup.img.gz` then copy to `golden.img`).
- **Dashboard "Shrink" / "Compress"** buttons run PiShrink **on the host** (not in the LXC). The dashboard writes a request file; the host runs `run-shrink-on-host.sh` when `cm4-shrink.path` sees it. Ensure PiShrink is installed on the host (see above) and that deploy has installed `run-shrink-on-host.sh` and enabled `cm4-shrink.path`.

### Cloud-init file locations on the Pi

- **NoCloud**: Boot partition root – `user-data`, `meta-data`, `network-config`.
- Some images expect them in a subfolder `cloud-init/` or on a separate vfat partition labeled `cidata`; check your OS docs. Standard Raspberry Pi OS NoCloud uses the boot partition root.

### Per-device config (optional)

NoCloud can also use a **seed** partition or **config drive**. For per-device hostname/settings you can:

- Use **meta-data** `instance-id` and `local-hostname` and generate different `meta-data` per device when imaging (e.g. script that writes `meta-data` before flashing), or
- Use a first-boot script that calls a provisioning server (e.g. by serial number) and applies device-specific config; cloud-init can launch that script from `runcmd`.

---

## Summary

| Step | Action |
|------|--------|
| 1 | Build `usbboot`, install `rpiboot` on provisioning host. |
| 2 | Create golden image with cloud-init `user-data`, `meta-data`, `network-config` on boot partition. |
| 3 | Install `flash-emmc-on-connect.sh`, `cm4-flash-trigger.sh`, and udev rule; set `GOLDEN_IMAGE` and enable file. |
| 4 | Put reTerminal in boot mode (jumper), connect USB to host; image is written automatically. |
| 5 | Remove jumper, power cycle; device boots from eMMC and cloud-init runs on first boot. |

This gives you automatic deployment of the golden image to eMMC when the reTerminal is in boot mode, plus first-boot configuration via cloud-init.