myxce/.wolf/bugs/bug-1975.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	summary	title
bug-1975	fixed	2026-10-27		controller	wrong-result	2026-10-27T03:40:05Z	2026-10-27T03:40:05Z	1	src/esd/src/esd.c src/lib/client.c src/lib/client_internal.h	esd external-license reconnect E2	bug-1760 bug-1973 bug-1974	2fcacac2	xceesd idle on dead connection after controller restart (stable value: no re-push)	xceesd idle on dead connection after controller restart (stable value: no re-push)

bug-1975: xceesd idle on dead connection after controller restart (stable value: no re-push)

Description

Observed: Old xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after the controller was restarted:

(no socket() attempts; strace showed only collector forks + epoll_wait;
 /proc/net socket table had no trace of the controller connection)

Last log line was the boot-time esd full re-push: 0/1 source(s) — no further activity.

Expected: within one poll cycle of the controller's restart the daemon must re-establish the connection and re-push its value (the new primary has no record of the daemon or its value).

Impact: after any controller restart/failover with a stable license value, the controller restarts with extlic read 0 (seed skipped for EXTERNAL by design) and no daemon re-push — every rusage[extlic] job PENDs forever with NO_RESOURCE until the license value happens to change.

Found by: E2 Phase 1 live E2E on t1 (controller restart step).

Environment

• OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
• libc: glibc 2.42 (de-nix deployment closure)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: debug (build.dbg)
• Code: feature/license-resources @ ba42fdd8-dirty
• Deployment: t1 single-node; xceesd extres daemon

Reproduction

1. xceesd running, pushing a value the collector keeps returning unchanged
2. Restart (or fail over) the controller
3. Observe the daemon: no reconnect attempts, pushes never resume

• Frequency: deterministic whenever the value is stable across the restart
• Evidence: strace captures /tmp/esd-strace*.out (t1, 2026-10-26); log silence quoted above

Root cause

Three mechanisms conspired:

1. esd_poll_round() debounces unchanged values (last_value == v → skip push) — with a stable value it pushes NOTHING.
2. A reconnect inside the client library only fires while INSIDE a push's failover call — no pushes, no reconnect.
3. esd_on_reconnect (the rejoin hook) only runs on a successful rotate; with no push attempt it never runs.

Fix

esd_run()'s main loop now checks xcecli_is_connection_broken() (bug-1760's pure predicate — the transport's RDHUP handler sets XCE_CTX_ERR asynchronously) and forces one esd_push_all() when it latches. The write attempt drives the rotate/reconnect; esd_on_reconnect keeps the full re-push contract for the failover path.

Verification

• Test: covered by the live contract (client-side predicate already unit tested by bug-1760's suite); esd-side loop is integration-shaped (the trigger is a transport callback), so live E2E is the proof.
• Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
• Live: t1 — controller killed and restarted; esd logged connection broken -- forcing a full re-push to reconnect, rotated to the new primary, re-pushed; xcecli resources distribution list shows extlic=4 and a fresh rusage[extlic=3] job RUNS.

References

• Related: bug-1760 (the pure broken-connection predicate this reuses), E2 spec design.md §3.5 E2E checklist (daemon kill/restart + controller failover items)