myxce/.wolf/bugs/bug-1978.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits
bug-1978	fixed	2026-10-28		controller	wrong-result	2026-10-28T12:47:59Z	2026-10-28T12:47:59Z	3	src/esd/src/esd.c	esd reconnect failover	bug-1760	72301fd5

bug-1978: esd reconnect re-pushes without re-join -- pushes -EPERM forever

Description

Observed:

t2 /tmp/xceesd-d.log:
[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1)
[2026-10-28 09:29:49,019] INFO  esd full re-push: 0/1 source(s)
[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1)
... (every 10s poll, forever)

After any controller restart the daemon's pushes fail rc=-1 (-EPERM) on every attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED, the poll loop runs, the collector script fires every 10s, the status file refreshes with stale: false -- only pushes_fail climbs.

Expected: after the reconnect the daemon re-joins (HostRole_extres) and pushes resume (pushes_ok advances, xce resources distribution list shows extlic).

Impact: the entire external-license value chain dies at the first controller restart: extlic never reaches c->sharedinst, the scheduler never hears a RESVALS_DELTA, every rusage[extlic=...] job PENDs forever (admission sees free=0). Found during the license-resource regression: the regression config push (gen 19) succeeded but resources distribution list showed NO extlic row -- daemon健康、值链断。

Found by: regression test -- live t1/t2 E2E (2026-10-28). Timeline evidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller restarts in /opt/xce/logs/xce_controller.log, zero extres joins after 10:58, and esd strace showing zero network sends (debounce suppressed -- no rotate ever triggered again).

Environment

• OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
• libc: glibc 2.39 (t2 host)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build) -- failing binary was the de-nix deploy /opt/xce-denix/bin/xceesd (Oct 27 10:51, non-ASAN deploy build)
• Code: feature/license-resources @ 30c51ea1 (esd) vs controller b38ef6de
• Deployment: 3-node t1/t2/t3; xceesd on t2, controller+scheduler on t1

Reproduction

1. Start xceesd on t2 (/opt/xce-denix/bin/xceesd -c /tmp/e2e/esd.yaml ...); observe pushes_ok advance (the boot join works).
2. Restart the t1 controller (or kill it long enough for the connection to drop and transparently reconnect).
3. Watch /tmp/xceesd-d.log: every subsequent esd push ... failed (rc=-1). Controller log logs no new extres join.

• Frequency: deterministic (3 sightings: 09:29, 10:58 controller restarts
  • the Oct-26 original "unknown role 5" cousin)
• Evidence: t2:/tmp/xceesd-d.log, t1:/opt/xce/logs/xce_controller.log, t2:/tmp/e2e/esd-status.json (pushes_ok frozen at 4, pushes_fail growing)

Root cause

esd.c's reconnect contract was "reconnect = full re-PUSH" (esd_on_reconnect sets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a re-JOIN: the fresh controller endpoint is created by handle_connection() (cluster.c) with priv->role = HostRole_client, and only OPCode_join sets priv->role = HostRole_extres (join.c handle_join_). The transparent reconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket but never re-runs the role join, and the client library's rejoin hook (xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the un-joined endpoint's resval update falls through BOTH branches of do_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and returns rc = -EPERM == -1 forever. (The worker solved this exact shape with the needs_rejoin contract, controller_connector.c; the scheduler likewise. The esd -- written later -- re-used the flag machinery but dropped the re-join itself.)

Fix

src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and g_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing xcecli_join(g_esd.apictx, HostRole_extres, NULL) on the main thread (outside the library's failover lock -- the xcecd_on_reconnect no-I/O constraint), and only then does the full re-push. Join failure keeps the flag set (next tick retries -- the worker's self-healing cadence).

Verification

• Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push
  • test_reconnect_flags_recover_from_any_previous_state (src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED with the old esd.c (compile error: g_needs_rejoin did not exist -- the re-join state was not represented at all), GREEN with the fix: 13/13.
• Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build rc=0.
• Live: pending -- fix binary redeploy + controller-restart drill on t1/t2 (restart controller, observe "esd: re-joined after reconnect" + pushes_ok advancing + extlic row in resources distribution list).

References

• Related: bug-1760 (the connection-broken predicate that makes the debounce-stall recovery work), worker needs_rejoin precedent (controller_connector.c), bug-1522 (register-after-start ordering).