homelab-blueprint/docs/troubleshooting.md

# Docker Stack Troubleshooting Guide

## Quick diagnostics

Run these first:

```bash
docker ps -a
docker stats --no-stream
docker network ls
docker-compose -f docker-compose.arr-stack.yml config
```

## Common issues

### 1. Variables not substituted

Symptoms:

- Generated file still contains `{{VAR}}`
- Container fails due to missing env values

Fix:

```bash
./substitute_env.sh docker-compose-files/arr-stack/template.yaml docker-compose.arr-stack.yml .env
```

The script prints any unresolved placeholders. Add missing keys to `.env` (or provide defaults in templates).

### 2. Recyclarr errors (`base_url must start with http`)

Use service names, not `localhost`:

```yaml
sonarr:
  sonarr-main:
    base_url: http://sonarr:8989

radarr:
  radarr-main:
    base_url: http://radarr:7878
```

Generate your file from this repo template:

```bash
./substitute_env.sh docker-compose-files/arr-stack/config/recyclarr_template.yml /opt/arr-stack/appdata/recyclarr/recyclarr.yml .env
```

### 2b. Recyclarr errors (`Unable to find include template ...`)

Symptoms:

- `Unable to find include template with name 'radarr-quality-definition-movie'`
- Similar errors for other `include: - template:` entries

Cause:

- You are using an older pre-v8 `recyclarr.yml` that still has `include: - template:` entries.
- Recyclarr v8 removed official include templates from the default provider.

Fix (v8 path used by this repository):

```bash
# 1) Regenerate recyclarr config from the v8 template
./substitute_env.sh docker-compose-files/arr-stack/config/recyclarr_template.yml /opt/arr-stack/appdata/recyclarr/recyclarr.yml .env

# 2) Regenerate compose and recreate recyclarr (v8.6.0 pin)
./substitute_env.sh docker-compose-files/arr-stack/template.yaml docker-compose.arr-stack.yml .env
docker-compose -f docker-compose.arr-stack.yml pull recyclarr
docker-compose -f docker-compose.arr-stack.yml up -d recyclarr
```

### 3. qBittorrent/SAB not using VPN IP

In this template, only download clients are expected behind VPN tunnel.

- Behind VPN: `gluetun`, `qbittorrent`
- Not behind VPN by default: `radarr`, `sonarr`, `prowlarr`, `bazarr`, `emby`, `seerr`

Checks:

```bash
docker logs gluetun | grep -i connected
docker exec qbittorrent curl -s ifconfig.me
```

### 4. Dependency/healthcheck startup failures

Symptoms:

- `dependency failed to start`
- Services wait forever on `service_healthy`

Checks:

```bash
docker inspect gluetun | grep -A 20 Health
docker inspect prowlarr | grep -A 20 Health
docker logs <container_name>
```

Fixes:

- Increase `start_period` for slow-start services.
- Test healthcheck command inside the container.
- Validate generated compose YAML syntax with `docker-compose ... config`.

### 5. Network communication failures

Checks:

```bash
docker network inspect vpn_network
docker exec radarr wget -qO- http://prowlarr:9696/ping
```

Fixes:

- Ensure relevant services are on `vpn-network` in generated compose.
- Recreate stack networks:

```bash
docker-compose -f docker-compose.arr-stack.yml down
docker network prune
docker-compose -f docker-compose.arr-stack.yml up -d
```

### 5b. qBittorrent stuck in "Created" / `network service:gluetun not found`

Symptoms:

- `qbittorrent` does not start after updates and remains in `Created`
- Recreate shows: `Error response from daemon: network service:gluetun not found`

Cause:

- `network_mode: service:gluetun` can fail during single-service recreate/update flows on some Docker setups.

Fix:

```bash
# Ensure template uses container mode for qbittorrent:
# network_mode: "container:gluetun"

# Re-render compose and recreate the pair
./substitute_env.sh docker-compose-files/arr-stack/template.yaml docker-compose.arr-stack.yml .env
docker compose -f docker-compose.arr-stack.yml up -d --force-recreate gluetun qbittorrent
```

Verify:

```bash
docker compose -f docker-compose.arr-stack.yml ps gluetun qbittorrent
docker compose -f docker-compose.arr-stack.yml logs --tail 100 qbittorrent
```

### 5c. qBittorrent fails with `joining network namespace of container: No such container`

Symptoms:

- `docker compose up -d qbittorrent` exits with:
  `Error response from daemon: joining network namespace of container: No such container: <hash>`
- Happens after gluetun was recreated (e.g., you ran `docker compose up -d` and
  gluetun got a new container ID because its config changed).

Cause:

- qbittorrent stores the gluetun container ID at start time. When gluetun is
  recreated its container ID changes, but the old qbittorrent container still
  holds a reference to the obsolete ID. `docker compose up -d` leaves an already-
  running qbittorrent alone, so the stale reference is never cleared.

Fix — force-remove the stale container and recreate it:

```bash
docker rm -f qbittorrent
docker compose -f docker-compose.arr-stack.yml up -d qbittorrent
```

Prevention: any time you explicitly recreate gluetun, recreate qbittorrent in
the same command so it picks up the new container ID:

```bash
docker compose -f docker-compose.arr-stack.yml up -d --force-recreate gluetun qbittorrent
```

### 5d. qBittorrent HTTPS `502` after stack updates

Symptoms:

- qBittorrent opens via HTTPS reverse proxy URL and returns `502` after stack updates/recreates.

Common causes:

- `gluetun` and `qbittorrent` are out of sync after recreate/update and need a paired restart.
- qBittorrent WebUI can be left in HTTPS mode while your reverse proxy still targets HTTP upstream.

Fix:

```bash
# 1) Recreate the VPN + torrent pair together
docker-compose -f docker-compose.arr-stack.yml up -d --force-recreate gluetun qbittorrent

# 2) If 502 persists, force qBittorrent WebUI back to HTTP
CONF="$(find /opt/arr-stack/appdata/qbittorrent -type f -name 'qBittorrent.conf' | head -n1)"
echo "Using config: $CONF"
sudo sed -i 's#^WebUI\\HTTPS\\Enabled=.*#WebUI\\HTTPS\\Enabled=false#' "$CONF"
sudo sed -i 's#^WebUI\\HTTPS\\CertificatePath=.*#WebUI\\HTTPS\\CertificatePath=#' "$CONF"
sudo sed -i 's#^WebUI\\HTTPS\\KeyPath=.*#WebUI\\HTTPS\\KeyPath=#' "$CONF"
docker-compose -f docker-compose.arr-stack.yml restart qbittorrent
```

Verify:

```bash
docker-compose -f docker-compose.arr-stack.yml ps gluetun qbittorrent
docker-compose -f docker-compose.arr-stack.yml logs --tail 120 qbittorrent
```

### 5e. Radarr/Sonarr reports a missing qBittorrent download path

Symptoms:

- Health reports that qBittorrent places downloads in a path that does not
  exist inside Radarr or Sonarr.
- qBittorrent logs contain `Permission denied` for paths such as
  `/data/.torrents/...`.
- The same torrent repeatedly rechecks or appears to restart because qBittorrent
  first tries the stale path, fails the move, then resumes against the corrected
  incomplete path.

Checks:

```bash
grep -E 'DefaultSavePath|TempPath' /opt/arr-stack/appdata/qbittorrent/config/qBittorrent.conf
docker exec radarr ls -ld /data/torrents /data/torrents/movies /data/torrents/tv
docker exec sonarr ls -ld /data/torrents /data/torrents/movies /data/torrents/tv
```

Required qBittorrent settings:

```text
Default save path: /data/torrents
Incomplete path:   /data/torrents/incomplete
movies category:  /data/torrents/movies
tv category:      /data/torrents/tv
```

Do not use `/data/.torrents`, and do not add remote path mappings when all
containers use the documented `/data` layout. After correcting an existing
torrent's location, force recheck and resume it, then rerun the Radarr/Sonarr
health check.

### 5f. No indexers available with RSS sync enabled

Cause:

- Prowlarr indexers use an app profile such as `Automatic Search` that has RSS
  disabled. Prowlarr synchronizes that disabled flag to Radarr and Sonarr.

Fix:

1. In Prowlarr, edit each affected indexer.
2. Set **App Profile** to `Standard`, or another profile with RSS, automatic
   search, and interactive search enabled.
3. Open **Settings > Apps** and run **Sync App Indexers** for Radarr and
   Sonarr.
4. Confirm the synchronized indexers show RSS enabled in both applications.

### 5g. Library Import leaves unmatched movie or series folders

Radarr expects one movie per folder. Split trilogy, franchise, and collection
packs into individual `Movie Title (Year)` folders before importing them.
Sample clips inside a movie folder can be mistaken for the feature; move or
remove samples before rescanning.

Sonarr permits only one root path per series. If duplicate series folders
exist, compare their episode coverage and quality, merge any unique episodes
into the preferred folder, update the series path, and verify the episode-file
count before removing the obsolete folder.

Do not delete unmatched folders solely because Library Import skipped them.
First verify whether they contain unique media, higher-quality replacements,
or files that were matched to the wrong title. Use exact TMDB/TVDB matches for
ambiguous titles.

### 5h. Download appears to restart mid-progress (qBittorrent OOM-killed)

Symptoms:

- A torrent regresses to a lower progress percentage mid-download (commonly at
  60–80 % on large files).
- `qbittorrent.log` shows a new `qBittorrent v... started.` line with no preceding
  error, roughly 10–15 minutes after the previous start.
- `docker inspect qbittorrent --format '{{.State.OOMKilled}}'` returns `true`.
- `docker stats --no-stream qbittorrent` shows memory near or at its limit.

Cause:

- qBittorrent buffers piece data in memory during download and hash-verification.
  For large torrents (20 GB+) this can exceed the 1 GiB container limit, causing
  the cgroup OOM killer to terminate the process. s6-overlay restarts qBittorrent
  inside the same container (Docker restart count stays at 0), which triggers a
  full piece re-verify from disk, visually resetting progress.

Fix — raise the limit live without restarting the container or losing progress:

```bash
docker update --memory 3g --memory-swap 6g qbittorrent
```

Then persist the change in the compose file and re-render from the template
(the template already sets 3 GiB):

```bash
./substitute_env.sh docker-compose-files/arr-stack/template.yaml docker-compose.arr-stack.yml .env
```

Do not lower `memory` below `2G` for a stack that downloads files larger than 10 GB.

### 5j. Download appears to restart mid-progress (Decluttarr auth ban loop)

Symptoms:

- A torrent that was at, say, 60 % progress appears to restart from a lower
  percentage after qBittorrent logs show a process restart.
- `qbittorrent.log` contains a long run of:
  ```
  (W) WebAPI login failure. Reason: IP has been banned, IP: ::ffff:172.18.0.12
  ```
  followed by `qBittorrent v... started.`
- `docker logs decluttarr` shows repeated `401`/`403` errors against
  `http://gluetun:8080/api/v2/auth/login`.

Cause:

- Decluttarr has blank (or wrong) `QBITTORRENT_USERNAME` / `QBITTORRENT_PASSWORD`
  in `.env`. It polls qBittorrent every 30 seconds; after several failures
  qBittorrent bans `172.18.0.0/16` (the Docker bridge), which forces a qBittorrent
  process restart. After restart, qBittorrent re-verifies pieces on disk, which
  can drop the reported progress before it climbs back.

Fix — whitelist the Docker bridge in qBittorrent so internal containers bypass
authentication:

```bash
# Run from inside the qbittorrent container (LocalHostAuth=false means no creds needed from localhost)
docker exec qbittorrent curl -s -X POST 'http://localhost:8080/api/v2/app/setPreferences' \
  -d 'json={"bypass_auth_subnet_whitelist":"10.0.10.0/24\n172.18.0.0/16","bypass_auth_subnet_whitelist_enabled":true}'

# Verify
docker exec qbittorrent curl -s 'http://localhost:8080/api/v2/app/preferences' \
  | python3 -c "import json,sys; p=json.load(sys.stdin); print(p.get('bypass_auth_subnet_whitelist'))"
```

qBittorrent saves this to its persisted config on the volume so it survives
container restarts. Alternatively, fill in `QBITTORRENT_USERNAME` and
`QBITTORRENT_PASSWORD` in `.env`, re-render the compose, and recreate the stack.

Verify fix:

```bash
docker restart decluttarr
docker logs decluttarr --tail 20
# Should show: INFO | OK | qBittorrent (http://gluetun:8080)
```

### 5k. Sonarr/Radarr warning: "Download client qBittorrent is set to remove completed downloads"

Symptoms:

- System status in Sonarr or Radarr shows:
  `Download client qBittorrent is set to remove completed downloads. This can
  result in downloads being removed from your client before Sonarr can import them.`

Cause:

- `Session\ShareLimitAction=Remove` in `qBittorrent.conf`. When seeding limits
  are reached (ratio or time), qBittorrent removes the torrent entry before
  Sonarr/Radarr can import it.
- The qBittorrent API `setPreferences` call does **not** persist `max_ratio_act`
  to disk in the hotio image. The value must be changed directly in the config
  file.

Fix — edit the config file and recreate the container:

```bash
# 1. Edit the config
sed -i 's/Session\\ShareLimitAction=Remove/Session\\ShareLimitAction=Pause/' \
  /opt/arr-stack/appdata/qbittorrent/config/qBittorrent.conf

# Verify
grep 'ShareLimitAction' /opt/arr-stack/appdata/qbittorrent/config/qBittorrent.conf
# Expected: Session\ShareLimitAction=Pause

# 2. Recreate qBittorrent (must use rm + up, not restart, due to gluetun network_mode)
cd /opt/homelab-blueprint
docker rm -f qbittorrent
docker compose -f docker-compose.arr-stack.yml up -d qbittorrent
```

With `Pause` set, qBittorrent pauses the torrent when limits are hit instead of
removing it. Sonarr and Radarr then import the file and signal qBittorrent to
remove it themselves.

Verify the warning clears in Sonarr/Radarr **System → Status** (may need a
minute for the health check to re-run).

> **Note:** The default seeding limits in this stack are `GlobalMaxRatio=0.1`
> and `GlobalMaxSeedingMinutes=20`. These are intentionally aggressive for
> public trackers. Raise them in qBittorrent Settings → BitTorrent if you use
> private trackers, which typically require a minimum ratio of 1.0.

### 5l. Sonarr "Downloaded - Waiting to Import" — Episode has a TBA title and recently aired

Symptoms:

- A completed download sits in the Sonarr queue under **Activity → Queue** with
  status `Downloaded - Waiting to Import`.
- The status message reads: `Episode has a TBA title and recently aired`.
- The episode title in TVDB is still `TBA` (common for hot new episodes within
  24–48 hours of airing).

Cause:

- Sonarr's **Episode Title Required** setting (Settings → Media Management) is
  set to `Always`. When TVDB hasn't published the real title yet, Sonarr holds
  the download rather than importing it with "TBA" in the filename.

Fix — change the setting to allow single-episode imports even with a TBA title:

```bash
curl -s -X PUT 'http://localhost:8989/api/v3/config/mediamanagement' \
  -H 'X-Api-Key: <your-sonarr-api-key>' \
  -H 'Content-Type: application/json' \
  -d '{"episodeTitleRequired": "bulkSeasonReleases", ... <rest of current config>}'
```

Or via the UI: **Settings → Media Management → Episode Title Required → Bulk
Season Releases**.

| Value | Behaviour |
|---|---|
| `Always` | Block import if title is TBA — safest naming, but you wait |
| `Bulk Season Releases` (recommended) | Only block for full-season dumps; allow single-episode imports even if TBA |
| `Never` | Always import regardless of title |

After changing the setting, Sonarr will automatically process any held queue
items within its next scan cycle.

**If you already have a TBA file imported and want to rename it:**

Once TVDB publishes the real title, go to **Series → Refresh & Scan** in Sonarr
and it will rename the file automatically. To rename immediately using a title
from an external source (e.g., Overseerr):

```bash
# On the Proxmox host, via pct exec into the LXC:
OLD='/data/media/tv/Series Name/Season 03/Series Name (Year) - S03E01 - TBA [...].mkv'
NEW='/data/media/tv/Series Name/Season 03/Series Name (Year) - S03E01 - Real Title [...].mkv'
pct exec 101 -- mv "$OLD" "$NEW"

# Then trigger a Sonarr rescan to update the database path
curl -s -X POST 'http://10.0.10.20:8989/api/v3/command' \
  -H 'X-Api-Key: <your-sonarr-api-key>' \
  -H 'Content-Type: application/json' \
  -d '{"name": "RescanSeries", "seriesId": <id>}'
```

### 5m. Prowlarr indexer recommendations and usenet retention gaps

When usenet articles expire (typical retention: 12–14 months), SABnzbd logs
`Article ... unavailable on all servers, discarding` and the download shows an
ETA of hundreds of days. Public torrent indexers serve as a fallback for
older content.

**Active indexer list (as of 2026-06-22):**

| Indexer | Type | Priority | Notes |
|---|---|---|---|
| NZBgeek | Usenet | 1 | Primary usenet source |
| Nzb.life | Usenet | 2 | Secondary usenet source |
| altHUB | Usenet | 3 | Tertiary usenet source |
| 1337x | Torrent | 20 | Requires FlareSolverr (Cloudflare-protected) |
| EZTV | Torrent | 10 | TV-specific; requires FlareSolverr (primary URL eztvx.to is Cloudflare-protected) |
| Knaben | Torrent | 40 | Aggregator; good coverage of older content |
| The Pirate Bay | Torrent | 30 | Broad fallback; magnet-based |

FlareSolverr must be running for 1337x **and EZTV** — both use eztvx.to/Cloudflare-protected
primary URLs. FlareSolverr is included in the arr-stack compose and is configured
in Prowlarr under **Settings → Indexers → FlareSolverr Proxy** (tag ID 1).
Assign this tag to any Cardigann indexer whose primary URL returns a Cloudflare challenge.

**To add a missing indexer via the Prowlarr API:**

```bash
# 1. Get the schema for the indexer you want to add
curl -s 'http://10.0.10.20:9696/api/v1/indexer/schema' \
  -H 'X-Api-Key: <prowlarr-api-key>' | jq '[.[] | {id: .id, name: .name}]'

# 2. Pull the specific schema
curl -s 'http://10.0.10.20:9696/api/v1/indexer/schema' \
  -H 'X-Api-Key: <prowlarr-api-key>' \
  | jq '.[] | select(.name == "EZTV")'

# 3. POST it (set enable: true and appProfileId: 1)
curl -s -X POST 'http://10.0.10.20:9696/api/v1/indexer' \
  -H 'X-Api-Key: <prowlarr-api-key>' \
  -H 'Content-Type: application/json' \
  -d '<schema JSON with enable: true>'

# 4. Sync to Sonarr and Radarr
curl -s -X POST 'http://10.0.10.20:9696/api/v1/applications/sync' \
  -H 'X-Api-Key: <prowlarr-api-key>'
```

**If EZTV shows "Indexer Disabled / Initial Failure":**

EZTV sometimes rate-limits or returns errors on first test. The 1-minute
disabled-until window is automatic. To re-enable:

1. In Prowlarr, go to **Indexers** and click the EZTV pencil icon.
2. Click **Test** — if it passes, save and the indexer re-enables.
3. If the test still fails, check that `eztv.re` is reachable from inside
   gluetun (VPN IP may be blocked by EZTV). In that case, try the alternate
   URL `eztvx.to` in the indexer base URL field.

**Private trackers (upgrade path):**

If you gain access to BTN (BroadcastTheNet) or TorrentLeech, add them via
Prowlarr — they have far better TV coverage and ratio requirements over
EZTV/1337x for current-season content.

### 6. Data/config persistence problems

Checks:

```bash
docker inspect radarr | grep -A 20 Mounts
ls -la /opt/arr-stack/appdata/radarr
```

Fixes:

```bash
mkdir -p /opt/arr-stack/appdata/radarr
chown -R <PUID>:<PGID> /opt/arr-stack/appdata/radarr
```

For stacks that use named volumes, inspect them with `docker volume ls` rather
than changing ownership on an old host appdata directory.

### 7. Caddy reverse proxy issues

If Caddy refuses to start, validate syntax first:

```bash
./substitute_env.sh docker-compose-files/caddy/Caddyfile_template docker-compose-files/caddy/Caddyfile .env
./substitute_env.sh docker-compose-files/caddy/template.yaml docker-compose.caddy.yml .env
chmod 644 docker-compose-files/caddy/Caddyfile
docker compose -f docker-compose.caddy.yml run --rm caddy \
  caddy validate --config /etc/caddy/Caddyfile
```

If validation reports `open /etc/caddy/Caddyfile: permission denied`, the
template renderer created the generated file with mode `600`, and the
container's mapped user cannot read it. Apply the `chmod 644` command above.
The generated Caddyfile does not contain the Cloudflare token. Do not apply
that mode to `docker-compose.caddy.yml`, which does contain the rendered
token.

If logs report `mkdir /data/caddy: permission denied` or
`mkdir /config/caddy: permission denied`, the generated Compose file still
uses host bind mounts that are not writable through Docker's user-namespace
mapping. Re-render it from the current Caddy template, which uses named
volumes, then recreate the container:

```bash
./substitute_env.sh docker-compose-files/caddy/template.yaml docker-compose.caddy.yml .env
docker compose -f docker-compose.caddy.yml down
docker compose -f docker-compose.caddy.yml up -d --build
docker compose -f docker-compose.caddy.yml logs --tail=100 caddy
```

The same risk applies to any writable bind mount when Docker uses rootless or
user-namespace mapping. The Caddy, Beszel Agent, PostgreSQL/pgAdmin,
Technitium, Gramps Web, Home Assistant, and Mosquitto templates use
named volumes for application state to avoid this issue.

Arr Stack, Gramps media, Vault config/TLS, and other explicit external storage
mounts intentionally remain host-managed. Verify their ownership and access on
the target Docker installation; do not replace those mounts with named volumes
without planning configuration edits, backups, and existing-data migration.

Before changing an existing bind-mount deployment to named volumes, stop the
stack and copy its current data. Starting against a new empty named volume does
not import the old bind-mounted data automatically.

If certificate issuance fails with errors like:

- `could not determine zone for domain ...` / `expected 1 zone, got 0`
- `dial tcp 1.0.0.1:53: i/o timeout`

these are two distinct failure modes with the same underlying cause: Caddy's DNS-01 resolver
cannot reach `1.1.1.1`/`1.0.0.1` on port 53.

**Distinguish the two errors:**

| Error | Meaning |
|---|---|
| `expected 1 zone, got 0` | Caddy reached a resolver, but it was Technitium (split-DNS). NAT exception misconfigured or alias IP wrong. |
| `dial tcp 1.0.0.1:53: i/o timeout` | Caddy's packets are being dropped. A firewall block rule on OPT2 is above the pass rule, or the NAT exception is not set up at all. |

Check DNS reachability from the Caddy host:

```bash
# Must return Cloudflare SOA, not Technitium SOA (dns01.home.example.com)
dig +short SOA home.example.com @1.1.1.1

# Distinguish firewall drop from redirect:
dig @1.1.1.1 +tcp +time=5 google.com   # timeout = firewall drop
dig @1.1.1.1 +notcp +time=5 google.com # timeout = same; success = TCP-specific block
```

Keep public resolvers in the Caddy TLS block — do not switch to Technitium as a workaround,
as it causes `expected 1 zone, got 0` because Technitium is authoritative for the internal zone:

```caddy
tls you@example.com {
  dns cloudflare {env.CLOUDFLARE_DNS_TOKEN}
  resolvers 1.1.1.1 1.0.0.1
}
```

If your network forces DNS redirection, exclude Caddy from that redirect and
allow it to query public resolvers on TCP and UDP port `53`.

If a proxied app fails over HTTPS with self-signed backend certs, ensure the target block uses:

```caddy
transport http {
  tls_insecure_skip_verify
}
```

## Update flow (safe)

1. Generate compose files from templates.
2. Validate with `docker-compose -f <file> config`.
3. Pull images.
4. Recreate services.
5. Review logs after startup.

```bash
docker-compose -f docker-compose.arr-stack.yml pull
docker-compose -f docker-compose.arr-stack.yml up -d
```

---

**Last Updated:** 2026-06-22