homelab-blueprint/docs/proxmox/nfs-lxc-boot-dependency.md
2026-06-28 13:40:08 +02:00

4.4 KiB

LXC Boot Dependencies — Network Mounts and NVIDIA

Two things must be ready on the Proxmox host before LXC 101 (arr-stack) can start: the UNAS NFS mount and the NVIDIA device nodes. Both are confirmed to have caused silent startup failures.


Problem 1 — NFS not ready (ENODEV, status 19)

All UNAS NFS mounts in /etc/fstab use x-systemd.automount — they are lazy-mounted on first access, not eagerly at boot. When Proxmox starts an LXC container with onboot: 1, its pre-start hook (lxc-pve-prestart-hook) tries to bind-mount every mp entry. If the NFS share isn't mounted yet (UNAS still booting after a power cycle), the automount times out and the hook exits with ENODEV (status 19) — killing the container startup silently.

Confirmed: LXC 101 (arr-stack) failed this way on 2026-06-21.

Global fix — fstab timeout

/etc/fstab — mount timeout increased from 10 s → 120 s for all UNAS shares, giving UNAS two minutes to come up before mounts fail.

x-systemd.mount-timeout=120s

Problem 2 — NVIDIA device nodes missing (nvml insufficient permissions)

The NVIDIA kernel modules (nvidia, nvidia_modeset, nvidia_drm) load at boot, but the character device nodes (/dev/nvidia0, /dev/nvidiactl, /dev/nvidia-caps/*) are only created when something first calls into the driver. Without these nodes the LXC bind-mounts them as empty files (optional), and nvidia-container-cli inside the container fails with nvml error: insufficient permissions — so emby cannot start.

Confirmed: emby exited (128) on 2026-06-21 for exactly this reason.

Fix — nvidia-init.service

/etc/systemd/system/nvidia-init.service runs nvidia-smi once at boot, which initialises all device nodes before any container starts.

[Unit]
Description=Initialize NVIDIA device nodes
Before=pve-container@101.service
After=systemd-udev-settle.service

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Enable with: systemctl enable nvidia-init.service


LXC 101 drop-in (both dependencies)

/etc/systemd/system/pve-container@101.service.d/nfs-deps.conf wires up both requirements so the container won't start until NFS is mounted and NVIDIA devices exist:

[Unit]
After=mnt-unas-arr_data.mount nvidia-init.service
Requires=mnt-unas-arr_data.mount nvidia-init.service

After any change: systemctl daemon-reload


Per-container network-mount fix (future containers)

Every LXC that bind-mounts a UNAS NFS or SMB share via mp and has onboot: 1 needs its own drop-in. Run these commands on Proxmox right after creating the container (substitute the real CT ID and mount name):

CTID=<id>
MOUNT_UNIT=mnt-unas-<mountname>.mount   # see table below

mkdir -p /etc/systemd/system/pve-container@${CTID}.service.d
cat > /etc/systemd/system/pve-container@${CTID}.service.d/nfs-deps.conf << EOF
[Unit]
After=${MOUNT_UNIT}
Requires=${MOUNT_UNIT}
EOF
systemctl daemon-reload

Requires= means: if the network mount fails (UNAS is genuinely down), systemd will not start the container — which is correct, since the container's data lives on that mount.

Container checklist

Container Protocol Network mount path systemd unit Drop-in status
LXC 101 — arr-stack NFS /mnt/unas/arr_data mnt-unas-arr_data.mount done
LXC ??? — immich NFS /mnt/unas/immich_data mnt-unas-immich_data.mount apply when created
LXC ??? — gramps NFS /mnt/unas/gramps_data mnt-unas-gramps_data.mount apply when created
LXC ??? — paperless NFS /mnt/unas/paperless_data mnt-unas-paperless_data.mount apply when created
LXC 103 — ytd SMB /mnt/unas/youtube_downloads mnt-unas-youtube_downloads.mount done

LXC 100 (PBS) uses a local disk — no NFS or NVIDIA dependency.


Verify everything is wired up

# NFS + NVIDIA deps on LXC 101
systemctl show pve-container@101.service | grep -E 'After|Requires'

# NVIDIA init service is enabled
systemctl is-enabled nvidia-init.service

# All device nodes exist
ls /dev/nvidia0 /dev/nvidiactl /dev/nvidia-uvm /dev/nvidia-caps/

Systemd unit name derivation

systemd escapes the mount path to generate the unit name: /mnt/unas/<name>mnt-unas-<name>.mount (slashes become dashes, leading slash dropped).