reterminal-dm4/emmc-provisioning/docs/DEVICE-DNS-DHCP-RESOLVCONF.md

Device DNS from DHCP and /etc/resolv.conf

This document describes how to configure provisioned devices (e.g. Raspberry Pi / reTerminal) so they use DNS from DHCP and do not have a fixed nameserver in /etc/resolv.conf. That way the LXC’s dnsmasq (option 6) is used, file.server resolves, and scripts can use http://file.server/... without hardcoding IPs.

Summary of changes (what we did on the Pi)

1. Do not overwrite /etc/resolv.conf
   No script (e.g. bootstrap or first-boot) should write a fixed nameserver (e.g. 8.8.8.8) into /etc/resolv.conf. DNS should come from DHCP.

2. LXC sends DHCP option 6 (DNS server)
   dnsmasq on the LXC must send the LXC’s eth1 IP as the DNS server so clients use it and get file.server resolution. See DNSMASQ-DNS-FILESERVER.md and scripts/setup-network-boot-on-lxc.sh (dhcp-option=6,${LAN_GW}).

3. Let NetworkManager manage /etc/resolv.conf
   So that the nameserver in /etc/resolv.conf is the one from DHCP (option 6), either:

   • Option A (recommended for full cloud-init): Use systemd-resolved and make /etc/resolv.conf a symlink to the resolved stub; resolved gets DNS from NetworkManager.
   • Option B (minimal cloud-init): Use NetworkManager to manage /etc/resolv.conf via a symlink: add rc-manager=symlink in NetworkManager config so /etc/resolv.conf points to /run/NetworkManager/resolv.conf, which NM fills with the DHCP DNS.

4. Ensure connection uses DHCP DNS
   The NetworkManager connection should have ipv4.ignore-auto-dns: no (default) so it accepts option 6 from DHCP. No fixed ipv4.dns in the connection.

What to change in cloud-init

Option A: user-data.bootstrap (uses Option B for RPi OS)

File: cloud-init/user-data.bootstrap

• manage_resolv_conf: false — already set; cloud-init must not overwrite resolv.conf.
• NetworkManager — 99-resolv-dhcp.conf has rc-manager=symlink so NM creates /etc/resolv.conf with DHCP DNS. RPi OS does not use systemd-resolved by default.
• runcmd — removes static resolv.conf and restarts NM so it creates the symlink with DHCP option 6.
• Bootstrap script — must not write nameserver 8.8.8.8 (or any fixed server) into /etc/resolv.conf. Our bootstrap.sh no longer does that.

No extra changes needed if you use user-data.bootstrap as-is; just ensure your bootstrap script does not touch resolv.conf.

Option B: Minimal user-data (first-boot or bootstrap-only, no systemd-resolved)

If your user-data only runs a remote script (e.g. first-boot.sh or bootstrap.sh) and does not enable systemd-resolved, add the following so the device uses DNS from DHCP and NM manages resolv.conf:

1. Set in user-data (cloud-config):

   manage_resolv_conf: false
   

2. Add a write_files entry so NetworkManager manages resolv.conf with the DHCP-provided DNS:

   write_files:
     # ... your other write_files ...
     - path: /etc/NetworkManager/conf.d/99-resolv-dhcp.conf
       content: |
         [main]
         rc-manager=symlink
       permissions: '0644'
   

3. In your bootstrap/first-boot script:
   Do not write a fixed nameserver to /etc/resolv.conf (e.g. remove any line like echo "nameserver 8.8.8.8" > /etc/resolv.conf).

4. Optional runcmd (if you want a clean state on first boot):
   Remove any existing static resolv.conf so NM can create its symlink and write DHCP DNS:

   runcmd:
     - rm -f /etc/resolv.conf
     - systemctl restart NetworkManager
     # ... then your download and run of bootstrap.sh or first-boot.sh ...
   

After first boot, devices will get DNS from DHCP (LXC option 6), and file.server will resolve to the LXC’s eth1 IP.

Verification on the device

# Should show the LXC as nameserver (e.g. 10.20.40.1), not 8.8.8.8
cat /etc/resolv.conf

# Should resolve to LXC eth1
getent hosts file.server

Reference: manual fix on an already-provisioned device

If a device was provisioned before these changes and still has a fixed DNS (e.g. 8.8.8.8):

1. LXC: Ensure dnsmasq sends option 6 (see DNSMASQ-DNS-FILESERVER.md); re-run setup-network-boot-on-lxc.sh if needed.
2. On the device:
   • Add NetworkManager config:
     echo -e '[main]\nrc-manager=symlink' | sudo tee /etc/NetworkManager/conf.d/99-resolv-dhcp.conf
   • Remove static resolv.conf and restart NM:
     sudo rm -f /etc/resolv.conf && sudo systemctl restart NetworkManager
   • Renew DHCP so the device gets option 6:
     sudo nmcli con down "Wired connection 1"; sudo nmcli con up "Wired connection 1"
3. Check: cat /etc/resolv.conf and getent hosts file.server.