Add backup shrinking functionality to eMMC provisioning dashboard: implement API for shrinking raw .img backups using PiShrink, update UI to support shrink option after backup, and enhance documentation for backup image handling and storage options on Proxmox host.
This commit is contained in:
nearxos
@@ -43,7 +43,7 @@ sudo cp rpiboot /opt/usbboot/
|
||||
sudo cp /path/to/your/golden-reterminal.img /var/lib/cm4-provisioning/golden.img
|
||||
```
|
||||
|
||||
- Or use a different path and set `GOLDEN_IMAGE` when installing the script (see below).
|
||||
- Or use a different path and set `GOLDEN_IMAGE` when installing the script (see below). On Proxmox with LXC, you can store backup images on a host directory by setting `CM4_BACKUPS_HOST_PATH` at deploy so that folder is bind-mounted into the LXC — see **PROXMOX-LXC-DEPLOYMENT.md** § Store backup images on a host directory.
|
||||
|
||||
#### 3. Install the provisioning script and trigger
|
||||
|
||||
@@ -59,6 +59,9 @@ sudo chmod +x /opt/cm4-provisioning/flash-emmc-on-connect.sh
|
||||
echo 'GOLDEN_IMAGE=/var/lib/cm4-provisioning/golden.img' | sudo tee /opt/cm4-provisioning/env
|
||||
echo 'RPIBOOT_DIR=/opt/usbboot' | sudo tee -a /opt/cm4-provisioning/env
|
||||
echo 'EMMC_SIZE_BYTES=8589934592' | sudo tee -a /opt/cm4-provisioning/env # 8GB; use 17179869184 for 16GB
|
||||
# Optional: shrink backups after dd (requires PiShrink; see "Shrinking backup and golden images" below)
|
||||
# echo 'SHRINK_BACKUP=1' | sudo tee -a /opt/cm4-provisioning/env
|
||||
# echo 'PISHRINK_COMPRESS=gz' | sudo tee -a /opt/cm4-provisioning/env # or xz
|
||||
|
||||
sudo cp cm4-flash-trigger.sh /usr/local/bin/
|
||||
sudo chmod +x /usr/local/bin/cm4-flash-trigger.sh
|
||||
@@ -116,6 +119,53 @@ Or keep it simple and edit the defaults inside `flash-emmc-on-connect.sh` (e.g.
|
||||
|
||||
Raspberry Pi OS (recent versions) supports **cloud-init** using the **NoCloud** datasource: it reads `user-data`, `meta-data`, and optionally `network-config` from the **boot** (FAT32) partition.
|
||||
|
||||
### Steps to prepare a cloud-init image for Raspberry Pi OS
|
||||
|
||||
1. **Obtain Raspberry Pi OS**
|
||||
- Download from [raspberrypi.com/software](https://www.raspberrypi.com/software/) (desktop or Lite), or use **Raspberry Pi Imager** with "Edit settings" for locale/SSH. Recent Raspberry Pi OS has cloud-init built in.
|
||||
|
||||
2. **Flash the image**
|
||||
- Flash to a spare SD card, or to a loop file for building without physical media: `cp raspios.img golden-work.img`
|
||||
|
||||
3. **Mount the boot partition**
|
||||
- **From an image file**: `sudo losetup -fP golden-work.img` then e.g. `sudo mount /dev/loop0p1 /mnt/boot`
|
||||
- **From SD**: mount the first (FAT32) partition at `/mnt/boot`. On a running Pi, boot is often `/boot/firmware`.
|
||||
|
||||
4. **Add NoCloud files on the boot partition** (root of FAT32, same level as `config.txt`):
|
||||
- `user-data`, `meta-data`, `network-config`
|
||||
- From this repo: `cp emmc-provisioning/cloud-init/{user-data,meta-data,network-config} /mnt/boot/`
|
||||
|
||||
5. **Customise** `user-data` and `network-config`. Use the **remote bootstrap script** pattern (below) to avoid rebuilding the image when you change first-boot commands.
|
||||
|
||||
6. **Unmount and create golden image**: `sudo umount /mnt/boot`, then copy the image to your `GOLDEN_IMAGE` path (e.g. `/var/lib/cm4-provisioning/golden.img`).
|
||||
|
||||
### Remote bootstrap script (no image rebuild for script changes)
|
||||
|
||||
You can keep the golden image **fixed** and have cloud-init **download a script from a file server** and run it on first boot. When you change the script on the server, the next device gets the new commands without rebuilding the image.
|
||||
|
||||
1. **Host the script** on an HTTP/HTTPS file server (e.g. nginx, or `python3 -m http.server`) at a URL the Pi can reach, e.g. `http://192.168.1.10/provisioning/bootstrap.sh`.
|
||||
|
||||
2. **In `user-data`, use `runcmd` to download and run it** (runcmd runs after packages and network are up):
|
||||
|
||||
```yaml
|
||||
#cloud-config
|
||||
package_update: true
|
||||
package_upgrade: false
|
||||
packages:
|
||||
- curl # or wget
|
||||
|
||||
runcmd:
|
||||
- curl -fsSL "http://YOUR_FILE_SERVER/provisioning/bootstrap.sh" -o /tmp/bootstrap.sh
|
||||
- chmod +x /tmp/bootstrap.sh
|
||||
- /tmp/bootstrap.sh
|
||||
```
|
||||
|
||||
With **wget**: `wget -q -O /tmp/bootstrap.sh "http://.../bootstrap.sh"` then `chmod +x /tmp/bootstrap.sh` and `/tmp/bootstrap.sh`.
|
||||
|
||||
3. **Bootstrap script**: use a normal shell script (e.g. `#!/bin/bash` and `set -e`). It can install packages, configure kiosk, set hostname, register with a dashboard, etc.
|
||||
|
||||
4. **Notes**: Ensure `network-config` (or DHCP) gives the Pi an IP before runcmd. For HTTPS, add `ca-certificates` to `packages` if needed. You can use one script for all devices or have the script call your server with serial/MAC for device-specific config.
|
||||
|
||||
### Creating the golden image
|
||||
|
||||
1. **Flash Raspberry Pi OS** (or your base image) to a spare SD card or a loop file.
|
||||
@@ -135,9 +185,47 @@ cp emmc-provisioning/cloud-init/meta-data /mnt/boot/
|
||||
cp emmc-provisioning/cloud-init/network-config /mnt/boot/
|
||||
```
|
||||
|
||||
4. **Customise** `user-data` and `network-config` (hostname, WiFi, packages, Chromium kiosk, etc.).
|
||||
5. **Copy your kiosk/Chromium scripts** into the image rootfs if needed (e.g. under `/home/pi/` or `/opt/`) and reference them from `user-data` `runcmd` or a systemd unit.
|
||||
6. **Unmount**, then create a **golden image** from the SD or loop device (e.g. `dd` or `dd` of the whole block device). Use that as `golden.img` on the provisioning host.
|
||||
4. **Customise** `user-data` and `network-config` (hostname, WiFi, packages, Chromium kiosk, or a single runcmd that downloads and runs a remote script—see "Remote bootstrap script" above).
|
||||
5. **Copy your kiosk/Chromium scripts** into the image rootfs only if you are not using a remote script; otherwise the remote script can pull what it needs.
|
||||
6. **Unmount**, then create a **golden image** from the SD or loop device (e.g. `dd` or copy of the whole block device). Use that as `golden.img` on the provisioning host. Optionally shrink it with PiShrink (see below) to save space and speed up deploy.
|
||||
|
||||
### Shrinking backup and golden images (PiShrink)
|
||||
|
||||
Raw full-disk backups and golden images are the full size of the eMMC (e.g. 32 GB). [PiShrink](https://github.com/Drewsif/PiShrink) shrinks the **last partition** (must be ext2/3/4) to its minimum size and truncates the image file; on first boot the rootfs can expand back to fill the device. This reduces backup/golden image size (often to a few GB) and improves transfer times.
|
||||
|
||||
**On the provisioning host:**
|
||||
|
||||
1. **Install PiShrink and dependencies** (run as root on the host, or via `ssh root@HOST 'bash -s' < scripts/install-pishrink-on-host.sh`):
|
||||
|
||||
```bash
|
||||
# From this repo (on the host or via ssh)
|
||||
bash chromium-setup/emmc-provisioning/scripts/install-pishrink-on-host.sh
|
||||
```
|
||||
|
||||
This installs `parted`, `e2fsprogs`, `gzip`, `pigz`, `xz-utils` and downloads `pishrink.sh` to `/usr/local/bin/pishrink.sh`.
|
||||
|
||||
2. **Shrink backups automatically** after each Backup in the dashboard: add to `/opt/cm4-provisioning/env` and ensure the trigger sources it:
|
||||
|
||||
```bash
|
||||
echo 'SHRINK_BACKUP=1' | sudo tee -a /opt/cm4-provisioning/env
|
||||
# Optional: compress after shrinking (smaller file, but must decompress before using as golden image)
|
||||
# echo 'PISHRINK_COMPRESS=gz' | sudo tee -a /opt/cm4-provisioning/env # or xz
|
||||
```
|
||||
|
||||
With `SHRINK_BACKUP=1`, once a backup finishes, the script runs PiShrink on the `.img` file. Use `PISHRINK_COMPRESS=gz` or `xz` for maximum size reduction; the file becomes `.img.gz` or `.img.xz` and must be decompressed before deploy (e.g. `gunzip -c backup.img.gz > golden.img`).
|
||||
|
||||
3. **Shrink a golden image manually** (e.g. after building from Raspberry Pi OS):
|
||||
|
||||
```bash
|
||||
sudo pishrink.sh -n /path/to/large.img /var/lib/cm4-provisioning/golden.img
|
||||
```
|
||||
|
||||
`-n` disables update check. Omit the second argument to shrink in place. The shrunk image will expand the rootfs on first boot when deployed to eMMC.
|
||||
|
||||
**Notes:**
|
||||
|
||||
- PiShrink only shrinks the **last** partition; it must be ext2/3/4 (standard Raspberry Pi OS root is ext4).
|
||||
- Compressed backups (`.img.gz` / `.img.xz`) are for archival; to use as golden image, decompress first (e.g. `gunzip -k backup.img.gz` then copy to `golden.img`).
|
||||
|
||||
### Cloud-init file locations on the Pi
|
||||
|
||||
|
||||
Reference in New Issue
Block a user