matt/homelab-docker

Watch

Fork 0

Komodo Periphery<->Core websocket flaps every ~60-90s on Dispatcharr LXC 113 #194

New issue

Open

opened 2026-07-07 02:19:37 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-07 02:19:37 +00:00

Collaborator

Context

Found 2026-07-06/07 while debugging why Komodo silently failed to redeploy several stacks after PR merges (see #36 comment history and the Komodo procedure-split fix that same day).

Symptom

docker logs komodo-periphery on PVE LXC 113 (Dispatcharr, 192.168.1.254) shows a continuous reconnect loop, roughly every 60-90 seconds, indefinitely:

INFO Logged in to Komodo Core 192.168.1.226:9120 websocket as Server dispatcharr
WARN Timed out waiting for Ping: deadline has elapsed
INFO Logged in to Komodo Core 192.168.1.226:9120 websocket as Server dispatcharr
WARN Timed out waiting for Ping: deadline has elapsed
...

Occasionally escalates to a harder failure:

WARN Failed to connect to websocket | url: ws://192.168.1.226:9120/ws/periphery?server=dispatcharr: IO error: Connection timed out (os error 110)

This is the ONLY Periphery connection that crosses a real network hop (Nexus <-> PVE LXC over the LAN) — every other Komodo-managed stack runs Periphery locally on Nexus itself (effectively loopback), so none of them show this pattern. That's why this specific LXC's flakiness was able to silently block deploys for unrelated stacks until the procedure was split (see below).

What's been ruled out

Not the 7/3-7/4 DHCP conflict (tracked in #36): converted LXC 113 from DHCP to a static IP (192.168.1.254/24) on 2026-07-07 and rebooted the container. The flapping continued unchanged immediately after reboot (reconnected at 02:10:58, timed out again by 02:11:45). So the original duplicate-IP conflict and this ongoing flapping are two separate problems.
Not basic ICMP-level connectivity: ping -c 10 from Nexus to 192.168.1.254 showed 0% packet loss, ~0.3-1ms latency, no jitter.
Not ARP conflict (at least not currently observable): ip neigh / /proc/net/arp on Nexus shows a single, stable, consistently-REACHABLE MAC for 192.168.1.254.
Not CPU/backlog overload: LXC 113 has 2 cores allocated, load average was ~0.2 (idle) during the flapping. /proc/net/softnet_stat inside the container showed 0 drops — so it's not a receive-queue overflow either.
Interface RX drops on the LXC's veth (~7.9% of packets) are likely a red herring — the drop rate is disproportionate to the bridge (~0.7%) and physical NIC (~0.02%), but with 0 softnet drops and low CPU, this is probably just normal broadcast/multicast noise being filtered (the bridge sees 1.1M+ multicast packets), not evidence of a real problem.

What hasn't been tried yet

Packet capture during a live disconnect (tcpdump on both ends, or at minimum on the PVE host's vmbr0/physical NIC) to see whether the websocket connection is getting an actual TCP RST, a silent timeout with no response, or something being dropped by a firewall/conntrack table entry mid-connection.
Check Komodo Core's ping interval/timeout configuration — it's possible the timeout is just too aggressive for a connection that crosses a physical network hop with even minor latency variance, and other (loopback) Periphery connections never exercise that code path enough to reveal it.
Check switch port stats for the physical port LXC 113/PVE traffic goes over (per CLAUDE.md, homelab network is currently flat/unsegmented; there may be relevant switch-level diagnostics, e.g. on the TP-Link switch).
Try increasing/adjusting Periphery's reconnect and ping-timeout settings if configurable (check periphery.yml config options and/or Komodo Core's server-side ping interval).

Impact

Currently contained, not urgent: as of 2026-07-06, Dispatcharr was split out of the shared "Deploy Stacks" Komodo procedure into its own dedicated "Deploy Dispatcharr" procedure/webhook. This flapping can therefore no longer cause silent deploy failures for unrelated stacks (Home Assistant, Tandoor, etc.) — worst case now is Dispatcharr's own deploys/git-pulls occasionally failing and needing a manual retry.

Suggested next step

Start with a packet capture during an active disconnect window (the flapping is frequent enough that a 2-3 minute capture on both Nexus and the PVE host should catch several cycles), then check Komodo's ping/timeout config as a lower-effort first attempt.

## Context Found 2026-07-06/07 while debugging why Komodo silently failed to redeploy several stacks after PR merges (see #36 comment history and the Komodo procedure-split fix that same day). ## Symptom `docker logs komodo-periphery` on PVE LXC 113 (Dispatcharr, 192.168.1.254) shows a continuous reconnect loop, roughly every 60-90 seconds, indefinitely: ``` INFO Logged in to Komodo Core 192.168.1.226:9120 websocket as Server dispatcharr WARN Timed out waiting for Ping: deadline has elapsed INFO Logged in to Komodo Core 192.168.1.226:9120 websocket as Server dispatcharr WARN Timed out waiting for Ping: deadline has elapsed ... ``` Occasionally escalates to a harder failure: ``` WARN Failed to connect to websocket | url: ws://192.168.1.226:9120/ws/periphery?server=dispatcharr: IO error: Connection timed out (os error 110) ``` This is the ONLY Periphery connection that crosses a real network hop (Nexus <-> PVE LXC over the LAN) — every other Komodo-managed stack runs Periphery locally on Nexus itself (effectively loopback), so none of them show this pattern. That's why this specific LXC's flakiness was able to silently block deploys for unrelated stacks until the procedure was split (see below). ## What's been ruled out - **Not the 7/3-7/4 DHCP conflict** (tracked in #36): converted LXC 113 from DHCP to a static IP (192.168.1.254/24) on 2026-07-07 and rebooted the container. The flapping continued unchanged immediately after reboot (reconnected at 02:10:58, timed out again by 02:11:45). So the original duplicate-IP conflict and this ongoing flapping are two separate problems. - **Not basic ICMP-level connectivity**: `ping -c 10` from Nexus to 192.168.1.254 showed 0% packet loss, ~0.3-1ms latency, no jitter. - **Not ARP conflict** (at least not currently observable): `ip neigh` / `/proc/net/arp` on Nexus shows a single, stable, consistently-REACHABLE MAC for 192.168.1.254. - **Not CPU/backlog overload**: LXC 113 has 2 cores allocated, load average was ~0.2 (idle) during the flapping. `/proc/net/softnet_stat` inside the container showed 0 drops — so it's not a receive-queue overflow either. - **Interface RX drops on the LXC's veth (~7.9% of packets) are likely a red herring** — the drop rate is disproportionate to the bridge (~0.7%) and physical NIC (~0.02%), but with 0 softnet drops and low CPU, this is probably just normal broadcast/multicast noise being filtered (the bridge sees 1.1M+ multicast packets), not evidence of a real problem. ## What hasn't been tried yet - **Packet capture during a live disconnect** (tcpdump on both ends, or at minimum on the PVE host's vmbr0/physical NIC) to see whether the websocket connection is getting an actual TCP RST, a silent timeout with no response, or something being dropped by a firewall/conntrack table entry mid-connection. - **Check Komodo Core's ping interval/timeout configuration** — it's possible the timeout is just too aggressive for a connection that crosses a physical network hop with even minor latency variance, and other (loopback) Periphery connections never exercise that code path enough to reveal it. - **Check switch port stats** for the physical port LXC 113/PVE traffic goes over (per CLAUDE.md, homelab network is currently flat/unsegmented; there may be relevant switch-level diagnostics, e.g. on the TP-Link switch). - **Try increasing/adjusting Periphery's reconnect and ping-timeout settings** if configurable (check periphery.yml config options and/or Komodo Core's server-side ping interval). ## Impact Currently contained, not urgent: as of 2026-07-06, Dispatcharr was split out of the shared "Deploy Stacks" Komodo procedure into its own dedicated "Deploy Dispatcharr" procedure/webhook. This flapping can therefore no longer cause silent deploy failures for unrelated stacks (Home Assistant, Tandoor, etc.) — worst case now is Dispatcharr's own deploys/git-pulls occasionally failing and needing a manual retry. ## Suggested next step Start with a packet capture during an active disconnect window (the flapping is frequent enough that a 2-3 minute capture on both Nexus and the PVE host should catch several cycles), then check Komodo's ping/timeout config as a lower-effort first attempt.

claude-bot referenced this issue

2026-07-07 02:20:14 +00:00

Verify Jellyfin Live TV is actually pulling channels/EPG from Dispatcharr #196

claude-bot commented

2026-07-07 02:48:21 +00:00

Author

Collaborator

Root cause found (packet capture, 2026-07-07)

There is still an IP conflict on 192.168.1.254 — a Netgear device (MAC 38:94:ed:52:73:92) is also using this address. The #36 fix (static .254) didn't help because .254 itself was already occupied. Confirmed via curl to the device while Nexus's ARP entry pointed at it: lighttpd, TLS cert CN=www.routerlogin.net, O=Netgear — a Netgear router/AP/extender, not documented anywhere in the homelab docs.

Mechanism (from 4-min tcpdump on PVE `veth113i0`, saved at `/root/issue194.pcap` on PVE)

The Netgear broadcasts ARP requests with sender IP .254 (e.g. who-has 192.168.1.1 tell 192.168.1.254) on a ~66s cycle (22:26:39.5, 22:27:45.9 EDT).
Linux updates existing ARP entries from any observed ARP — so Nexus's entry for .254 flips to the Netgear MAC. Caught live: ip neigh show 192.168.1.254 on Nexus → 38:94:ed:52:73:92 REACHABLE (LXC 113's real MAC is bc:24:11:0a:3b:15).
All Nexus→LXC traffic (Komodo Core pings, ACKs, SYN-ACKs) then goes to the Netgear, which even ARPs for .226 milliseconds after receiving each misdelivered packet (to send RSTs back). LXC→Nexus still works — the blackhole is one-way, which is why it looked so weird.
Periphery's hardcoded 10s receive deadline (lib/transport/src/websocket/mod.rs, not configurable) trips → Timed out waiting for Ping. Nexus's neighbor-probing recovers the correct MAC after ~30-60s, the websocket relogs in, then the next Netgear ARP re-poisons. The ~60-90s flap period is the Netgear's ARP refresh cycle.

This also explains all prior observations: ICMP tests passed (1s ping keeps the neighbor entry fresh / hit clean windows), ip neigh looked stable when checked (point-in-time), and only this Periphery flaps (all others are local to Nexus — no ARP involved).

Mitigation applied (2026-07-07 02:37 UTC)

Static ARP pin on Nexus: ip neigh replace 192.168.1.254 lladdr bc:24:11:0a:3b:15 dev eno1 nud permanent

Result: websocket stable, zero flaps in 10 minutes (previous max uptime ~75s).
Runtime-only — does not survive a Nexus reboot. Revert with ip neigh del 192.168.1.254 dev eno1.
Scope: fixes Komodo Core↔Periphery only. Other hosts (Atlas, the router, IPTV clients) still get poisoned and will see intermittent .254 blackouts of their own.

Permanent fix — needs a decision

Option A (recommended): renumber LXC 113 to a verified-free static IP. Known references to update: README.md (2 lines) and dashy/conf.yml (2 lines) in this repo, homelab docs/memory, and any IPTV clients pointed at .254:9191. Then remove the ARP pin on Nexus.

Option B: find and reconfigure/remove the Netgear device. It answers HTTP/HTTPS at .254 during poisoned windows (routerlogin.net cert). If it's a forgotten extender, unplugging it ends the conflict and .254 stays as-is. If it's an active AP, renumber it or the LXC.

Either way the ARP pin should be treated as temporary.

## Root cause found (packet capture, 2026-07-07) **There is still an IP conflict on 192.168.1.254 — a Netgear device (MAC `38:94:ed:52:73:92`) is also using this address.** The #36 fix (static .254) didn't help because .254 itself was already occupied. Confirmed via `curl` to the device while Nexus's ARP entry pointed at it: lighttpd, TLS cert `CN=www.routerlogin.net, O=Netgear` — a Netgear router/AP/extender, not documented anywhere in the homelab docs. ### Mechanism (from 4-min tcpdump on PVE `veth113i0`, saved at `/root/issue194.pcap` on PVE) 1. The Netgear broadcasts ARP requests with sender IP .254 (e.g. `who-has 192.168.1.1 tell 192.168.1.254`) on a **~66s cycle** (22:26:39.5, 22:27:45.9 EDT). 2. Linux updates existing ARP entries from any observed ARP — so Nexus's entry for .254 flips to the Netgear MAC. Caught live: `ip neigh show 192.168.1.254` on Nexus → `38:94:ed:52:73:92 REACHABLE` (LXC 113's real MAC is `bc:24:11:0a:3b:15`). 3. All Nexus→LXC traffic (Komodo Core pings, ACKs, SYN-ACKs) then goes to the Netgear, which even ARPs for .226 milliseconds after receiving each misdelivered packet (to send RSTs back). LXC→Nexus still works — the blackhole is one-way, which is why it looked so weird. 4. Periphery's hardcoded 10s receive deadline (`lib/transport/src/websocket/mod.rs`, not configurable) trips → `Timed out waiting for Ping`. Nexus's neighbor-probing recovers the correct MAC after ~30-60s, the websocket relogs in, then the next Netgear ARP re-poisons. **The ~60-90s flap period is the Netgear's ARP refresh cycle.** This also explains all prior observations: ICMP tests passed (1s ping keeps the neighbor entry fresh / hit clean windows), `ip neigh` looked stable when checked (point-in-time), and only this Periphery flaps (all others are local to Nexus — no ARP involved). ### Mitigation applied (2026-07-07 02:37 UTC) Static ARP pin on Nexus: `ip neigh replace 192.168.1.254 lladdr bc:24:11:0a:3b:15 dev eno1 nud permanent` - Result: websocket stable, **zero flaps in 10 minutes** (previous max uptime ~75s). - **Runtime-only — does not survive a Nexus reboot.** Revert with `ip neigh del 192.168.1.254 dev eno1`. - Scope: fixes Komodo Core↔Periphery only. Other hosts (Atlas, the router, IPTV clients) still get poisoned and will see intermittent .254 blackouts of their own. ### Permanent fix — needs a decision **Option A (recommended): renumber LXC 113** to a verified-free static IP. Known references to update: `README.md` (2 lines) and `dashy/conf.yml` (2 lines) in this repo, homelab docs/memory, and any IPTV clients pointed at `.254:9191`. Then remove the ARP pin on Nexus. **Option B: find and reconfigure/remove the Netgear device.** It answers HTTP/HTTPS at .254 during poisoned windows (`routerlogin.net` cert). If it's a forgotten extender, unplugging it ends the conflict and .254 stays as-is. If it's an active AP, renumber it or the LXC. Either way the ARP pin should be treated as temporary.

No labels

blocked

needs-manual-review

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

matt/homelab-docker#194

No description provided.

Rows
Columns