Enhance provisioning documentation and scripts for improved network boot and DNS management</message>
<message>Add new documentation files for device DNS management via DHCP and dnsmasq configuration. Update cloud-init scripts to ensure proper handling of /etc/resolv.conf and DNS settings, allowing for seamless integration with file.server. Modify existing scripts to support dynamic LAN subnet configuration and improve overall network boot functionality. These changes enhance user experience and streamline the setup process for the CM4 eMMC provisioning service.
This commit is contained in:
nearxos
@@ -21,7 +21,7 @@ Step-by-step guide to deploy the provisioning service (host + LXC) on a **new**
|
||||
- `CM4_BACKUPS_HOST_PATH=/mnt/storage/cm4-backups` — Store backups on this host path (create the directory on the host if needed).
|
||||
- **Network (WAN/LAN):**
|
||||
`DEPLOY_LXC_WAN_BRIDGE=vmbr0` (default), `DEPLOY_LXC_WAN_IP=dhcp` (default),
|
||||
`DEPLOY_LXC_LAN_BRIDGE=vmbr1`, `DEPLOY_LXC_LAN_SUBNET=10.20.50.1/24` — To add eth1 as provisioning LAN with a custom subnet.
|
||||
`DEPLOY_LXC_LAN_BRIDGE=vmbr1`, `DEPLOY_LXC_LAN_SUBNET=10.20.50.1/24` — To add eth1 as provisioning LAN. **Set these if you want the portal reachable from the LAN** (e.g. http://10.20.50.1:5000); the dashboard listens on all interfaces.
|
||||
|
||||
---
|
||||
|
||||
@@ -48,8 +48,7 @@ DEPLOY_LXC_LAN_SUBNET=10.20.50.1/24 \
|
||||
```
|
||||
|
||||
- On **first run**, the script will ask you to choose LXC rootfs storage (unless `DEPLOY_ROOTFS_STORAGE` is set). It then creates the LXC, installs host scripts, udev, systemd units, and the dashboard in the LXC.
|
||||
- The script prints the **LXC IP** at the end. Note it for the next steps (or get it with:
|
||||
`ssh root@YOUR_PROXMOX_HOST "pct exec \$(pct list -no-header -output vmid,name | awk '\''\$2==\"cm4-provisioning\"{print \$1}'\'') -- hostname -I"`).
|
||||
- The script prints **LXC IP (WAN)** and, if you set `DEPLOY_LXC_LAN_BRIDGE`, **LXC IP (LAN)**. The portal is reachable at `http://<IP>:5000` on both; use the LAN IP from devices on the provisioning LAN.
|
||||
|
||||
---
|
||||
|
||||
@@ -90,6 +89,16 @@ scp /path/to/your-golden.img root@YOUR_PROXMOX_HOST:/var/lib/cm4-provisioning/go
|
||||
|
||||
---
|
||||
|
||||
## Accessing the portal from the LAN
|
||||
|
||||
The dashboard listens on **all interfaces** (`0.0.0.0:5000`), so it is reachable on both WAN and LAN IPs when the LXC has two networks.
|
||||
|
||||
- **Deploy with a LAN interface:** set `DEPLOY_LXC_LAN_BRIDGE=vmbr1` (and optionally `DEPLOY_LXC_LAN_SUBNET=10.20.50.1/24`) when running the deploy script. The LXC will get eth1 with the LAN IP (e.g. 10.20.50.1).
|
||||
- **From the provisioning LAN:** open **http://<LAN-IP>:5000** (e.g. http://10.20.50.1:5000). Devices on that subnet can use the portal without going through WAN.
|
||||
- If you did not set a LAN bridge at deploy time, you only have one IP (WAN); use that for the portal. To add LAN later you would need to add eth1 to the container and reconfigure (see PROXMOX-LXC-DEPLOYMENT.md).
|
||||
|
||||
---
|
||||
|
||||
## Step 4: (Optional) SSH into the LXC
|
||||
|
||||
If you set `DEPLOY_LXC_ROOT_PASSWORD` or had a default SSH key, you can already run:
|
||||
@@ -144,9 +153,11 @@ Or from your machine (stream the script): use the same pattern as in [PROXMOX-LX
|
||||
|
||||
**After deployment:**
|
||||
|
||||
- **Dashboard:** http://<LXC-IP>:5000
|
||||
- **Dashboard:** http://<LXC-IP>:5000 (WAN). If you set `DEPLOY_LXC_LAN_BRIDGE`, also **http://<LAN-IP>:5000** (e.g. http://10.20.50.1:5000) from the LAN.
|
||||
- **Golden image path (host and LXC):** `/var/lib/cm4-provisioning/golden.img`
|
||||
- **Disable auto-flash:** `ssh root@YOUR_PROXMOX_HOST "rm /etc/cm4-provisioning/enabled"`
|
||||
- **Enable again:** `ssh root@YOUR_PROXMOX_HOST "touch /etc/cm4-provisioning/enabled"`
|
||||
|
||||
**If you see "rpiboot failed or no device connected":** The error is from the **Proxmox host** (where USB is connected). On the host run: `tail -50 /var/lib/cm4-provisioning/flash.log` to see the real rpiboot message. Ensure the reTerminal is in **boot mode** (eMMC disable jumper, USB slave port), then unplug/replug. See [PROXMOX-LXC-DEPLOYMENT.md](PROXMOX-LXC-DEPLOYMENT.md) § "If rpiboot fails" for full steps.
|
||||
|
||||
Full reference: [PROXMOX-LXC-DEPLOYMENT.md](PROXMOX-LXC-DEPLOYMENT.md).
|
||||
|
||||
90
emmc-provisioning/docs/DEVICE-DNS-DHCP-RESOLVCONF.md
Normal file
90
emmc-provisioning/docs/DEVICE-DNS-DHCP-RESOLVCONF.md
Normal file
@@ -0,0 +1,90 @@
|
||||
# Device DNS from DHCP and /etc/resolv.conf
|
||||
|
||||
This document describes how to configure provisioned devices (e.g. Raspberry Pi / reTerminal) so they **use DNS from DHCP** and do **not** have a fixed nameserver in `/etc/resolv.conf`. That way the LXC’s dnsmasq (option 6) is used, **file.server** resolves, and scripts can use `http://file.server/...` without hardcoding IPs.
|
||||
|
||||
## Summary of changes (what we did on the Pi)
|
||||
|
||||
1. **Do not overwrite `/etc/resolv.conf`**
|
||||
No script (e.g. bootstrap or first-boot) should write a fixed nameserver (e.g. `8.8.8.8`) into `/etc/resolv.conf`. DNS should come from DHCP.
|
||||
|
||||
2. **LXC sends DHCP option 6 (DNS server)**
|
||||
dnsmasq on the LXC must send the LXC’s eth1 IP as the DNS server so clients use it and get **file.server** resolution. See [DNSMASQ-DNS-FILESERVER.md](DNSMASQ-DNS-FILESERVER.md) and `scripts/setup-network-boot-on-lxc.sh` (`dhcp-option=6,${LAN_GW}`).
|
||||
|
||||
3. **Let NetworkManager manage `/etc/resolv.conf`**
|
||||
So that the nameserver in `/etc/resolv.conf` is the one from DHCP (option 6), either:
|
||||
- **Option A (recommended for full cloud-init):** Use **systemd-resolved** and make `/etc/resolv.conf` a symlink to the resolved stub; resolved gets DNS from NetworkManager.
|
||||
- **Option B (minimal cloud-init):** Use **NetworkManager** to manage `/etc/resolv.conf` via a symlink: add `rc-manager=symlink` in NetworkManager config so `/etc/resolv.conf` points to `/run/NetworkManager/resolv.conf`, which NM fills with the DHCP DNS.
|
||||
|
||||
4. **Ensure connection uses DHCP DNS**
|
||||
The NetworkManager connection should have `ipv4.ignore-auto-dns: no` (default) so it accepts option 6 from DHCP. No fixed `ipv4.dns` in the connection.
|
||||
|
||||
## What to change in cloud-init
|
||||
|
||||
### Option A: user-data.bootstrap (systemd-resolved)
|
||||
|
||||
**File:** `cloud-init/user-data.bootstrap`
|
||||
|
||||
- **manage_resolv_conf: false** — already set; cloud-init must not overwrite resolv.conf.
|
||||
- **systemd-resolved** — runcmd enables/starts resolved and makes `/etc/resolv.conf` a symlink to `stub-resolv.conf`. Resolved gets DNS from NetworkManager (and from the hooks in write_files).
|
||||
- **NetworkManager** — `99-use-resolved.conf` has `dns=systemd-resolved` and `rc-manager=unmanaged` so NM doesn’t write resolv.conf; resolved does.
|
||||
- **Bootstrap script** — must **not** write `nameserver 8.8.8.8` (or any fixed server) into `/etc/resolv.conf`. Our `bootstrap.sh` no longer does that.
|
||||
|
||||
No extra changes needed if you use `user-data.bootstrap` as-is; just ensure your bootstrap script does not touch resolv.conf.
|
||||
|
||||
### Option B: Minimal user-data (first-boot or bootstrap-only, no systemd-resolved)
|
||||
|
||||
If your user-data only runs a remote script (e.g. `first-boot.sh` or `bootstrap.sh`) and does **not** enable systemd-resolved, add the following so the device uses DNS from DHCP and NM manages resolv.conf:
|
||||
|
||||
1. **Set in user-data (cloud-config):**
|
||||
```yaml
|
||||
manage_resolv_conf: false
|
||||
```
|
||||
|
||||
2. **Add a write_files entry** so NetworkManager manages resolv.conf with the DHCP-provided DNS:
|
||||
```yaml
|
||||
write_files:
|
||||
# ... your other write_files ...
|
||||
- path: /etc/NetworkManager/conf.d/99-resolv-dhcp.conf
|
||||
content: |
|
||||
[main]
|
||||
rc-manager=symlink
|
||||
permissions: '0644'
|
||||
```
|
||||
|
||||
3. **In your bootstrap/first-boot script:**
|
||||
Do **not** write a fixed nameserver to `/etc/resolv.conf` (e.g. remove any line like `echo "nameserver 8.8.8.8" > /etc/resolv.conf`).
|
||||
|
||||
4. **Optional runcmd** (if you want a clean state on first boot):
|
||||
Remove any existing static resolv.conf so NM can create its symlink and write DHCP DNS:
|
||||
```yaml
|
||||
runcmd:
|
||||
- rm -f /etc/resolv.conf
|
||||
- systemctl restart NetworkManager
|
||||
# ... then your download and run of bootstrap.sh or first-boot.sh ...
|
||||
```
|
||||
|
||||
After first boot, devices will get DNS from DHCP (LXC option 6), and **file.server** will resolve to the LXC’s eth1 IP.
|
||||
|
||||
## Verification on the device
|
||||
|
||||
```bash
|
||||
# Should show the LXC as nameserver (e.g. 10.20.40.1), not 8.8.8.8
|
||||
cat /etc/resolv.conf
|
||||
|
||||
# Should resolve to LXC eth1
|
||||
getent hosts file.server
|
||||
```
|
||||
|
||||
## Reference: manual fix on an already-provisioned device
|
||||
|
||||
If a device was provisioned before these changes and still has a fixed DNS (e.g. 8.8.8.8):
|
||||
|
||||
1. **LXC:** Ensure dnsmasq sends option 6 (see [DNSMASQ-DNS-FILESERVER.md](DNSMASQ-DNS-FILESERVER.md)); re-run `setup-network-boot-on-lxc.sh` if needed.
|
||||
2. **On the device:**
|
||||
- Add NetworkManager config:
|
||||
`echo -e '[main]\nrc-manager=symlink' | sudo tee /etc/NetworkManager/conf.d/99-resolv-dhcp.conf`
|
||||
- Remove static resolv.conf and restart NM:
|
||||
`sudo rm -f /etc/resolv.conf && sudo systemctl restart NetworkManager`
|
||||
- Renew DHCP so the device gets option 6:
|
||||
`sudo nmcli con down "Wired connection 1"; sudo nmcli con up "Wired connection 1"`
|
||||
3. Check: `cat /etc/resolv.conf` and `getent hosts file.server`.
|
||||
101
emmc-provisioning/docs/DNSMASQ-DNS-FILESERVER.md
Normal file
101
emmc-provisioning/docs/DNSMASQ-DNS-FILESERVER.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# dnsmasq DNS and file.server on the LXC
|
||||
|
||||
This document describes the dnsmasq DNS configuration on the provisioning LXC and the static hostname **file.server** used for the fileserver.
|
||||
|
||||
## What was changed
|
||||
|
||||
### 1. dnsmasq now provides DNS on eth1
|
||||
|
||||
Previously, dnsmasq on the LXC was configured with **`port=0`**, which disabled DNS and provided only DHCP and TFTP on the provisioning interface (eth1).
|
||||
|
||||
**Change:** `port=0` was removed so dnsmasq also acts as a DNS server on eth1 (port 53). Clients that receive DHCP from dnsmasq will use the LXC as their DNS server for the provisioning LAN.
|
||||
|
||||
- **DHCP** on eth1 — unchanged (range from `lan-subnet.conf`, e.g. 10.20.40.100–10.20.40.200).
|
||||
- **TFTP/PXE** on eth1 — unchanged (toggle with `/opt/cm4-provisioning/toggle-network-boot-dhcp.sh`).
|
||||
- **DNS** on eth1 — **new**: local static records (e.g. `file.server`) plus forwarding of other queries via the LXC’s `/etc/resolv.conf`.
|
||||
|
||||
### 2. Static DNS record: file.server → eth1 IP
|
||||
|
||||
A static A record was added so the hostname **file.server** resolves to the LXC’s eth1 address (the provisioning LAN gateway). That IP is taken from **`/opt/cm4-provisioning/lan-subnet.conf`** as **`LAN_GW`** (e.g. `10.20.40.1`).
|
||||
|
||||
**dnsmasq config (written by `setup-network-boot-on-lxc.sh`):**
|
||||
|
||||
```text
|
||||
address=/file.server/${LAN_GW}
|
||||
```
|
||||
|
||||
So scripts and devices on the provisioning LAN can use **`http://file.server/...`** (or `file.server` in general) without hardcoding the LXC’s IP. The IP stays correct even if the LAN subnet is changed and the setup script is re-run.
|
||||
|
||||
### 3. Files modified in the repo
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| **scripts/setup-network-boot-on-lxc.sh** | Removed `port=0`; added `address=/file.server/${LAN_GW}` and comments in the generated `/etc/dnsmasq.d/network-boot.conf`. |
|
||||
| **lxc/dnsmasq-network-boot.conf** | Template updated: removed `port=0`, added comment for DNS and `file.server` (commented example). |
|
||||
|
||||
### 4. Applied on LXC (root@10.20.40.1)
|
||||
|
||||
On **2025-03-04** the setup script was run against **root@10.20.40.1**:
|
||||
|
||||
```bash
|
||||
./emmc-provisioning/scripts/setup-network-boot-on-lxc.sh root@10.20.40.1
|
||||
```
|
||||
|
||||
Result on that LXC:
|
||||
|
||||
- **LAN:** 10.20.40.0/24, gateway 10.20.40.1 (from existing `lan-subnet.conf`).
|
||||
- **DHCP:** 10.20.40.100–10.20.40.200 on eth1.
|
||||
- **DNS:** Enabled on eth1; **file.server** → **10.20.40.1**.
|
||||
- dnsmasq and NAT were (re)configured; TFTP root and network boot toggle unchanged.
|
||||
|
||||
So on the provisioning LAN, **file.server** resolves to **10.20.40.1** (the LXC’s eth1).
|
||||
|
||||
## How to use file.server in scripts
|
||||
|
||||
On devices that get DHCP (and thus DNS) from the LXC on the provisioning LAN:
|
||||
|
||||
- Use **`http://file.server/...`** (or `file.server` as hostname) instead of `http://10.20.40.1/...`.
|
||||
- No need to hardcode the LXC IP; if you change the subnet and re-run the setup script, **file.server** will still point at the correct gateway.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
curl -O http://file.server/cloud-init/seed.img
|
||||
```
|
||||
|
||||
## Adding more static DNS entries
|
||||
|
||||
To add more names (e.g. `fileserver` or another hostname), add more **`address=/name/${LAN_GW}`** lines in the heredoc in **scripts/setup-network-boot-on-lxc.sh** (where `network-boot.conf` is generated), or add a separate file under `/etc/dnsmasq.d/` on the LXC with the same format. Then restart dnsmasq:
|
||||
|
||||
```bash
|
||||
systemctl restart dnsmasq
|
||||
```
|
||||
|
||||
## Re-applying on another or existing LXC
|
||||
|
||||
To apply or refresh this configuration on any LXC:
|
||||
|
||||
```bash
|
||||
./emmc-provisioning/scripts/setup-network-boot-on-lxc.sh root@<LXC-IP> [SUBNET]
|
||||
```
|
||||
|
||||
Example with explicit subnet:
|
||||
|
||||
```bash
|
||||
./emmc-provisioning/scripts/setup-network-boot-on-lxc.sh root@10.20.40.1 10.20.40.1/24
|
||||
```
|
||||
|
||||
This rewrites `/etc/dnsmasq.d/network-boot.conf` (including `address=/file.server/${LAN_GW}`) and restarts dnsmasq.
|
||||
|
||||
## Verification on the LXC
|
||||
|
||||
```bash
|
||||
# DNS and file.server
|
||||
grep -E 'address=|port=' /etc/dnsmasq.d/network-boot.conf
|
||||
|
||||
# Resolve file.server (from a client on the provisioning LAN, or from LXC with server 127.0.0.1)
|
||||
getent hosts file.server
|
||||
# or: dig @10.20.40.1 file.server
|
||||
```
|
||||
|
||||
Expected: **file.server** resolves to the LAN gateway (e.g. 10.20.40.1).
|
||||
@@ -18,8 +18,9 @@ Devices plugged into the same network as **eth1** (e.g. reTerminals with network
|
||||
## What you need on the LXC
|
||||
|
||||
1. **DHCP server** on eth1 only (e.g. **dnsmasq**), handing out addresses in e.g. `10.20.50.100`–`10.20.50.200` and advertising the TFTP server (next-server = LXC’s eth1 IP).
|
||||
2. **TFTP server** (dnsmasq can provide this) with **TFTP root** containing Raspberry Pi 4 / CM4 boot files.
|
||||
3. **IP forwarding** and **NAT** (nftables or iptables) so traffic from `10.20.50.0/24` is masqueraded out **eth0**.
|
||||
2. **DNS server** on eth1 (dnsmasq): static name **file.server** → eth1 IP so scripts can use `http://file.server/...`; other queries forwarded upstream. See [DNSMASQ-DNS-FILESERVER.md](DNSMASQ-DNS-FILESERVER.md).
|
||||
3. **TFTP server** (dnsmasq can provide this) with **TFTP root** containing Raspberry Pi 4 / CM4 boot files.
|
||||
4. **IP forwarding** and **NAT** (nftables or iptables) so traffic from `10.20.50.0/24` is masqueraded out **eth0**.
|
||||
|
||||
## One-time setup (inside the LXC)
|
||||
|
||||
@@ -40,8 +41,8 @@ bash /path/to/setup-network-boot-on-lxc.sh
|
||||
|
||||
The script will:
|
||||
|
||||
- Install **dnsmasq** (DHCP + TFTP).
|
||||
- Configure dnsmasq to listen only on **eth1**, with a DHCP range and TFTP root.
|
||||
- Install **dnsmasq** (DHCP + TFTP + DNS).
|
||||
- Configure dnsmasq to listen only on **eth1**, with a DHCP range, TFTP root, and DNS (including **file.server** → eth1).
|
||||
- Create `/srv/tftpboot` and **fetch Raspberry Pi 4 boot files from GitHub** (raspberrypi/firmware, `boot/` folder) if not already present.
|
||||
- Enable **IPv4 forwarding** and **NAT** (nftables) so clients on eth1 use eth0 for internet.
|
||||
- Enable and start the **dnsmasq** service.
|
||||
|
||||
63
emmc-provisioning/docs/PROXMOX-HOST-COMPARISON.md
Normal file
63
emmc-provisioning/docs/PROXMOX-HOST-COMPARISON.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# Proxmox host comparison: working vs new
|
||||
|
||||
Quick reference from comparing **working** (10.130.60.224) and **new** (100.106.128.36) Proxmox hosts.
|
||||
|
||||
## Same on both
|
||||
|
||||
| Item | Status |
|
||||
|------|--------|
|
||||
| `/opt/cm4-provisioning/env` | Same (GOLDEN_IMAGE, RPIBOOT_DIR, EMMC_SIZE_BYTES=8589934592, SHRINK_BACKUP, etc.) |
|
||||
| `/etc/cm4-provisioning/enabled` | Present (provisioning enabled) |
|
||||
| udev rules `89-cm4-boot-mode-permissions.rules` | Same (MODE="0666" for 2b8e, 0a5c:2711) |
|
||||
| udev rules `90-cm4-boot-mode.rules` | Same (trigger cm4-flash-trigger.sh on add) |
|
||||
| `/usr/local/bin/cm4-flash-trigger.sh` | Same (starts cm4-flash.service) |
|
||||
| `cm4-flash.service` | Same (ExecStartPre=sleep 5, TimeoutStartSec=7200) |
|
||||
| `cm4-build-cloudinit.path`, `cm4-shrink.path` + services | Same |
|
||||
| `build-cloudinit-image.sh`, `run-shrink-on-host.sh`, `fix-gadget-bootcode-on-host.sh` | Same (md5 match) |
|
||||
|
||||
## Differences
|
||||
|
||||
### 1. `/opt/usbboot/mass-storage-gadget64/bootcode4.bin` — **fixed**
|
||||
|
||||
- **Working:** Present (105984 bytes). rpiboot needs this or it prints "No 'bootcode' files found".
|
||||
- **New:** Was missing; fixed by copying from working host. Verify: `ls -la /opt/usbboot/mass-storage-gadget64/bootcode4.bin` on new.
|
||||
|
||||
### 2. `/opt/cm4-provisioning/flash-emmc-on-connect.sh`
|
||||
|
||||
- **Working:** Older version (14421 bytes, md5 `dbac0bc2...`).
|
||||
- **New:** Newer version (15449 bytes, md5 `6081bda7...`) with longer device wait (60s/90s), relaxed size check (50–120%), and extra diagnostics on "No suitable block device".
|
||||
|
||||
**Recommendation:** Copy the repo script to the **working** host so both use the same version:
|
||||
|
||||
```bash
|
||||
scp emmc-provisioning/host/flash-emmc-on-connect.sh root@10.130.60.224:/opt/cm4-provisioning/
|
||||
```
|
||||
|
||||
### 3. `/var/lib/cm4-provisioning/golden.img`
|
||||
|
||||
- **Working:** Present (symlink to a backup image). Required for **Deploy** (writing image to device).
|
||||
- **New:** Missing. Backup works; Deploy will show "Golden image not found" until you set one (dashboard “Set as golden” or `scp` an image to the host as `golden.img`).
|
||||
|
||||
### 4. `/opt/cm4-provisioning/dashboard/` (host only)
|
||||
|
||||
- **Working:** Directory exists (owner 1000:1000). Dashboard normally runs in the **LXC**, so this may be leftover or unused on the host.
|
||||
- **New:** No dashboard dir on host. No action needed unless you run the dashboard on the host.
|
||||
|
||||
### 5. Extra files in `/var/lib/cm4-provisioning/` on working
|
||||
|
||||
- **Working:** `build_cloudinit_status.json`, `cloudinit_templates.json`, `first_boot_status.json` (written by dashboard/LXC when using cloud-init build).
|
||||
- **New:** Not present yet. They appear when you use the dashboard from the LXC; not required for flash/backup.
|
||||
|
||||
## Checklist for new host (100.106.128.36)
|
||||
|
||||
- [x] `bootcode4.bin` in `/opt/usbboot/mass-storage-gadget64/` (copied from working)
|
||||
- [x] Same udev rules, trigger, and systemd units
|
||||
- [x] Same env. `EMMC_SIZE_BYTES` is optional; device detection is dynamic (any new block device after rpiboot is accepted for 8/16/32 GB CM4).
|
||||
- [ ] Set `golden.img` for Deploy (copy image or use dashboard “Set as golden” from a backup)
|
||||
- [x] `flash-emmc-on-connect.sh` is the updated version (longer wait, diagnostics)
|
||||
|
||||
## If flash still fails on new host
|
||||
|
||||
1. Check flash.log: `ssh root@100.106.128.36 'tail -80 /var/lib/cm4-provisioning/flash.log'` — the script now logs "Current block devices" and sizes when no device is found.
|
||||
2. Ensure eMMC disable jumper is set and you use the USB slave port; unplug and replug.
|
||||
3. Optional: increase udev/systemd delay (e.g. `ExecStartPre=/bin/sleep 10` in `cm4-flash.service`) if the device is slow to enumerate.
|
||||
@@ -210,7 +210,50 @@ Or copy `scripts/monitor-from-host.sh` to the host and run `./monitor-from-host.
|
||||
|
||||
2. **Unplug and replug the USB** – udev runs the trigger only when the device is *added*. Unplug the reTerminal USB (keep it in boot mode), then plug it back in. The trigger will run the script and rpiboot; when the eMMC is exposed, the portal shows "Device connected" with Backup/Deploy.
|
||||
|
||||
3. **If rpiboot fails** – Check on the host: `ssh root@10.130.60.224 'tail -30 /var/lib/cm4-provisioning/flash.log'` (rpiboot stderr is appended there). Try unplug/replug again. To see the exact rpiboot error: `ssh root@10.130.60.224 '/opt/usbboot/rpiboot -d /opt/usbboot/mass-storage-gadget64'` (device connected; Ctrl+C to stop). Run `scripts/monitor-from-host.sh` for a full snapshot.
|
||||
3. **If rpiboot fails** ("rpiboot failed or no device connected"):
|
||||
- **Check flash.log on the host** (rpiboot runs there; the log has the real error):
|
||||
`ssh root@YOUR_PROXMOX_HOST 'tail -50 /var/lib/cm4-provisioning/flash.log'`
|
||||
- **Quick diagnostic:**
|
||||
`ssh root@YOUR_PROXMOX_HOST 'bash -s' < emmc-provisioning/scripts/check-usb-on-host.sh`
|
||||
This shows whether the device is seen (lsusb), status, and last lines of flash.log.
|
||||
- **Ensure device is in boot mode:** eMMC disable jumper set, use the **USB slave** port (not host port). Unplug and replug after setting jumper.
|
||||
- **Run rpiboot manually** (device connected; Ctrl+C to stop):
|
||||
`ssh root@YOUR_PROXMOX_HOST '/opt/usbboot/rpiboot -v -d /opt/usbboot/mass-storage-gadget64'`
|
||||
The last line before exit is usually the error (e.g. "No device found", "Unable to open device").
|
||||
- Run `scripts/monitor-from-host.sh root@YOUR_PROXMOX_HOST` for a full snapshot.
|
||||
|
||||
**Manual usbboot test** (to verify rpiboot and the CM4 without the full flash flow):
|
||||
|
||||
1. **On the Proxmox host** (where USB is connected), with the reTerminal in **boot mode** (eMMC disable jumper set, USB slave port connected):
|
||||
```bash
|
||||
/opt/usbboot/rpiboot -v -d /opt/usbboot/mass-storage-gadget64
|
||||
```
|
||||
2. You should see lines like: `Device located successfully`, `Loading: .../bootcode4.bin`, `Sending bootcode.bin`. When the device switches to mass storage, rpiboot exits and a new `/dev/sdX` may appear (check with `lsblk` in another terminal). Press **Ctrl+C** to stop rpiboot at any time.
|
||||
3. **From your machine** (with device already connected to the host):
|
||||
```bash
|
||||
./emmc-provisioning/scripts/test-usbboot-on-host.sh root@YOUR_PROXMOX_HOST
|
||||
```
|
||||
Optional: run with a timeout so it doesn’t wait forever: `TIMEOUT=60 ./emmc-provisioning/scripts/test-usbboot-on-host.sh root@YOUR_PROXMOX_HOST`
|
||||
|
||||
**"libusb_bulk_transfer sent 0 bytes; returned -7" / "Failed to write correct length"** (device found, then transfer fails in a loop):
|
||||
|
||||
This is a known USB timing/controller issue with CM4 and rpiboot ([raspberrypi/usbboot#36](https://github.com/raspberrypi/usbboot/issues/36)). Try in order:
|
||||
|
||||
1. **Use a USB 2.0 port** on the Proxmox host (not USB 3.0). Many reports say USB 2.0 is more reliable for rpiboot.
|
||||
2. **Avoid udev fighting with manual rpiboot:** Temporarily disable the provisioning trigger so only your manual rpiboot runs:
|
||||
`ssh root@HOST 'mv /etc/cm4-provisioning/enabled /etc/cm4-provisioning/enabled.bak'`
|
||||
Run rpiboot, then re-enable:
|
||||
`ssh root@HOST 'mv /etc/cm4-provisioning/enabled.bak /etc/cm4-provisioning/enabled'`
|
||||
3. **Try a different USB port and cable** on the host; unplug/replug and retry.
|
||||
4. **Add a delay** so the device is ready before transfer:
|
||||
`rpiboot -v -d /opt/usbboot/mass-storage-gadget64 -m 2000`
|
||||
(`-m` is microseconds between device checks; 2000 = 2 ms.)
|
||||
5. **Reboot the Proxmox host** and try again (USB controller state can get stuck).
|
||||
6. **Find a USB 2.0 port:** On the host run `lsusb -t` — look for the BCM2711 device; the tree shows which controller (e.g. "xHCI" = USB 3, "ehci" = USB 2). Try a port that is under an **ehci** or **ohci** controller, or a black (non‑blue) physical port.
|
||||
7. **Try a powered USB 2.0 hub** between host and reTerminal (some hosts work only through a hub).
|
||||
8. If you have the **working host** (10.130.60.224), try the same reTerminal there; if it works there, the difference is host USB controller or port.
|
||||
|
||||
3b. **"No suitable block device after rpiboot"** – rpiboot ran but no new block device was seen. Detection is **dynamic**: any block device that appears after rpiboot (not present before) is used, so 8/16/32 GB CM4 work without setting eMMC size. **Check on host:** `tail -80 /var/lib/cm4-provisioning/flash.log` — at the end you’ll see "Current block devices" and each `/dev/sdX` with size. **Causes:** (1) Device didn’t switch to mass storage (try unplug/replug, keep eMMC disable jumper set). (2) udev slow — try again; the script waits up to 90s for the device.
|
||||
|
||||
4. **"No 'bootcode' files found in mass-storage-gadget64"** – Usually because `bootfiles.bin` is a **broken symlink** (e.g. `-> ../firmware/bootfiles.bin`) and that target doesn’t exist. **Fix on host:** run `scripts/fix-gadget-bootcode-on-host.sh` on the host (it removes the symlink and extracts `bootcode4.bin` from the installed rpiboot binary). From your machine: `ssh root@10.130.60.224 'bash -s' < scripts/fix-gadget-bootcode-on-host.sh`. **Alternative:** repopulate the gadget dir with `./scripts/populate-gadget-on-host.sh root@10.130.60.224`, or full reinstall with `./scripts/build-and-deploy-usbboot-to-host.sh root@10.130.60.224`. Then verify: `ls -la /opt/usbboot/mass-storage-gadget64/` (should list a real `bootcode4.bin` or `bootfiles.bin`, plus `boot.img`, `config.txt`).
|
||||
|
||||
|
||||
Reference in New Issue
Block a user