fix(esd): re-join after transparent reconnect -- bug-1978

A controller restart dropped xceesd's resval pushes forever: the fresh
endpoint is created by handle_connection() with priv->role =
HostRole_client, and only OPCode_join sets HostRole_extres -- but the
old esd_on_reconnect cleared only g_needs_full_push, so the daemon
re-PUSHED onto an un-joined endpoint and every push landed in
do_update_resvals's neither-branch, refused -EPERM forever (live
sighting: pushes_ok frozen, rc=-1 every 10s).

Mirror the worker's needs_rejoin contract: the reconnect handler now
sets BOTH g_needs_rejoin and g_needs_full_push; the main loop consumes
the re-join first (xcecli_join(HostRole_extres) on the main thread,
outside the failover lock -- the xcecd_on_reconnect no-I/O constraint),
retrying next tick on failure, then does the full re-push.

Live t1/t2 proof: controller restart -> "esd: re-joined after
reconnect (rc=0)" + "full re-push: 1/1 source(s)" + extlic back in
`xcecli resources distribution list`. Two new white-box unit tests pin
the both-flags contract (13/13 in the esd suite).

Co-Authored-By: Claude Code <noreply@anthropic.com>

.wolf/bugs/bug-1978.md

@@ -0,0 +1,110 @@
+---
+id: "bug-1978"
+status: "fixed"
+status_updated: "2026-10-28"
+status_note: ""
+component: "controller"
+severity: "wrong-result"
+timestamp: "2026-10-28T12:47:59Z"
+last_seen: "2026-10-28T12:47:59Z"
+occurrences: 3
+files: ["src/esd/src/esd.c"]
+tags: ["esd", "reconnect", "failover"]
+related: ["bug-1760"]
+fix_commits: ["72301fd5"]
+---
+
+# bug-1978: esd reconnect re-pushes without re-join -- pushes -EPERM forever
+
+## Description
+**Observed:**
+```
+t2 /tmp/xceesd-d.log:
+[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1)
+[2026-10-28 09:29:49,019] INFO  esd full re-push: 0/1 source(s)
+[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1)
+... (every 10s poll, forever)
+```
+After any controller restart the daemon's pushes fail rc=-1 (-EPERM) on
+every attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED,
+the poll loop runs, the collector script fires every 10s, the status file
+refreshes with `stale: false` -- only pushes_fail climbs.
+
+**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and
+pushes resume (pushes_ok advances, `xce resources distribution list` shows
+extlic).
+
+**Impact:** the entire external-license value chain dies at the first
+controller restart: extlic never reaches c->sharedinst, the scheduler never
+hears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever
+(admission sees free=0). Found during the license-resource regression: the
+regression config push (gen 19) succeeded but `resources distribution list`
+showed NO extlic row -- daemon健康、值链断。
+
+**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline
+evidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller
+restarts in /opt/xce/logs/xce_controller.log, zero extres joins after
+10:58, and esd strace showing zero network sends (debounce suppressed -- no
+rotate ever triggered again).
+
+## Environment
+- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
+- libc: glibc 2.39 (t2 host)
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: asan (xce.build) -- failing binary was the de-nix deploy
+  /opt/xce-denix/bin/xceesd (Oct 27 10:51, non-ASAN deploy build)
+- Code: feature/license-resources @ 30c51ea1 (esd) vs controller b38ef6de
+- Deployment: 3-node t1/t2/t3; xceesd on t2, controller+scheduler on t1
+
+## Reproduction
+1. Start xceesd on t2 (`/opt/xce-denix/bin/xceesd -c /tmp/e2e/esd.yaml ...`);
+   observe pushes_ok advance (the boot join works).
+2. Restart the t1 controller (or kill it long enough for the connection to
+   drop and transparently reconnect).
+3. Watch /tmp/xceesd-d.log: every subsequent `esd push ... failed (rc=-1)`.
+   Controller log logs no new extres join.
+- Frequency: deterministic (3 sightings: 09:29, 10:58 controller restarts
+  + the Oct-26 original "unknown role 5" cousin)
+- Evidence: t2:/tmp/xceesd-d.log, t1:/opt/xce/logs/xce_controller.log,
+  t2:/tmp/e2e/esd-status.json (pushes_ok frozen at 4, pushes_fail growing)
+
+## Root cause
+esd.c's reconnect contract was "reconnect = full re-PUSH" (esd_on_reconnect
+sets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a
+re-JOIN: the fresh controller endpoint is created by handle_connection()
+(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets
+priv->role = HostRole_extres (join.c handle_join_). The transparent
+reconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket
+but never re-runs the role join, and the client library's rejoin hook
+(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the
+un-joined endpoint's resval update falls through BOTH branches of
+do_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and
+returns rc = -EPERM == -1 forever. (The worker solved this exact shape with
+the needs_rejoin contract, controller_connector.c; the scheduler likewise.
+The esd -- written later -- re-used the flag machinery but dropped the
+re-join itself.)
+
+## Fix
+src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and
+g_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing
+`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread
+(outside the library's failover lock -- the xcecd_on_reconnect no-I/O
+constraint), and only then does the full re-push. Join failure keeps the
+flag set (next tick retries -- the worker's self-healing cadence).
+
+## Verification
+- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push
+  + test_reconnect_flags_recover_from_any_previous_state
+  (src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED
+  with the old esd.c (compile error: g_needs_rejoin did not exist -- the
+  re-join state was not represented at all), GREEN with the fix: 13/13.
+- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build
+  rc=0.
+- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2
+  (restart controller, observe "esd: re-joined after reconnect" + pushes_ok
+  advancing + extlic row in `resources distribution list`).
+
+## References
+- Related: bug-1760 (the connection-broken predicate that makes the
+  debounce-stall recovery work), worker needs_rejoin precedent
+  (controller_connector.c), bug-1522 (register-after-start ordering).