89 lines
3.7 KiB
Markdown
89 lines
3.7 KiB
Markdown
---
|
|
id: "bug-1975"
|
|
status: "fixed"
|
|
status_updated: "2026-10-27"
|
|
status_note: ""
|
|
component: "controller"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-10-27T03:40:05Z"
|
|
last_seen: "2026-10-27T03:40:05Z"
|
|
occurrences: 1
|
|
files: ["src/esd/src/esd.c", "src/lib/client.c", "src/lib/client_internal.h"]
|
|
tags: ["esd", "external-license", "reconnect", "E2"]
|
|
related: ["bug-1760", "bug-1973", "bug-1974"]
|
|
fix_commits: ["2fcacac2"]
|
|
summary: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
|
|
title: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
|
|
---
|
|
|
|
# bug-1975: xceesd idle on dead connection after controller restart (stable value: no re-push)
|
|
|
|
## Description
|
|
**Observed:**
|
|
Old xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after
|
|
the controller was restarted:
|
|
|
|
```
|
|
(no socket() attempts; strace showed only collector forks + epoll_wait;
|
|
/proc/net socket table had no trace of the controller connection)
|
|
```
|
|
|
|
Last log line was the boot-time `esd full re-push: 0/1 source(s)` — no
|
|
further activity.
|
|
|
|
**Expected:** within one poll cycle of the controller's restart the daemon
|
|
must re-establish the connection and re-push its value (the new primary has
|
|
no record of the daemon or its value).
|
|
|
|
**Impact:** after any controller restart/failover with a stable license
|
|
value, the controller restarts with extlic read 0 (seed skipped for EXTERNAL
|
|
by design) and no daemon re-push — every rusage[extlic] job PENDs forever
|
|
with NO_RESOURCE until the license value happens to change.
|
|
|
|
**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).
|
|
|
|
## Environment
|
|
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
|
|
- libc: glibc 2.42 (de-nix deployment closure)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: debug (build.dbg)
|
|
- Code: feature/license-resources @ ba42fdd8-dirty
|
|
- Deployment: t1 single-node; xceesd extres daemon
|
|
|
|
## Reproduction
|
|
1. xceesd running, pushing a value the collector keeps returning unchanged
|
|
2. Restart (or fail over) the controller
|
|
3. Observe the daemon: no reconnect attempts, pushes never resume
|
|
- Frequency: deterministic whenever the value is stable across the restart
|
|
- Evidence: strace captures /tmp/esd-strace*.out (t1, 2026-10-26); log silence quoted above
|
|
|
|
## Root cause
|
|
Three mechanisms conspired:
|
|
1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip
|
|
push) — with a stable value it pushes NOTHING.
|
|
2. A reconnect inside the client library only fires while INSIDE a push's
|
|
failover call — no pushes, no reconnect.
|
|
3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate;
|
|
with no push attempt it never runs.
|
|
|
|
## Fix
|
|
`esd_run()`'s main loop now checks `xcecli_is_connection_broken()`
|
|
(bug-1760's pure predicate — the transport's RDHUP handler sets
|
|
XCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches.
|
|
The write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the
|
|
full re-push contract for the failover path.
|
|
|
|
## Verification
|
|
- Test: covered by the live contract (client-side predicate already unit
|
|
tested by bug-1760's suite); esd-side loop is integration-shaped (the
|
|
trigger is a transport callback), so live E2E is the proof.
|
|
- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
|
|
- Live: t1 — controller killed and restarted; esd logged
|
|
`connection broken -- forcing a full re-push to reconnect`, rotated to the
|
|
new primary, re-pushed; `xcecli resources distribution list` shows extlic=4
|
|
and a fresh rusage[extlic=3] job RUNS.
|
|
|
|
## References
|
|
- Related: bug-1760 (the pure broken-connection predicate this reuses),
|
|
E2 spec design.md §3.5 E2E checklist (daemon kill/restart + controller
|
|
failover items)
|