---
id: "bug-1978"
status: "fixed"
status_updated: "2026-10-28"
status_note: ""
component: "controller"
severity: "wrong-result"
timestamp: "2026-10-28T12:47:59Z"
last_seen: "2026-10-28T12:47:59Z"
occurrences: 3
files: ["src/esd/src/esd.c"]
tags: ["esd", "reconnect", "failover"]
related: ["bug-1760"]
fix_commits: ["72301fd5"]
---

# bug-1978: esd reconnect re-pushes without re-join -- pushes -EPERM forever

## Description
**Observed:**
```
t2 /tmp/xceesd-d.log:
[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1)
[2026-10-28 09:29:49,019] INFO  esd full re-push: 0/1 source(s)
[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1)
... (every 10s poll, forever)
```
After any controller restart the daemon's pushes fail rc=-1 (-EPERM) on
every attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED,
the poll loop runs, the collector script fires every 10s, the status file
refreshes with `stale: false` -- only pushes_fail climbs.

**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and
pushes resume (pushes_ok advances, `xce resources distribution list` shows
extlic).

**Impact:** the entire external-license value chain dies at the first
controller restart: extlic never reaches c->sharedinst, the scheduler never
hears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever
(admission sees free=0). Found during the license-resource regression: the
regression config push (gen 19) succeeded but `resources distribution list`
showed NO extlic row -- daemon健康、值链断。

**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline
evidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller
restarts in /opt/xce/logs/xce_controller.log, zero extres joins after
10:58, and esd strace showing zero network sends (debounce suppressed -- no
rotate ever triggered again).

## Environment
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
- libc: glibc 2.39 (t2 host)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: asan (xce.build) -- failing binary was the de-nix deploy
  /opt/xce-denix/bin/xceesd (Oct 27 10:51, non-ASAN deploy build)
- Code: feature/license-resources @ 30c51ea1 (esd) vs controller b38ef6de
- Deployment: 3-node t1/t2/t3; xceesd on t2, controller+scheduler on t1

## Reproduction
1. Start xceesd on t2 (`/opt/xce-denix/bin/xceesd -c /tmp/e2e/esd.yaml ...`);
   observe pushes_ok advance (the boot join works).
2. Restart the t1 controller (or kill it long enough for the connection to
   drop and transparently reconnect).
3. Watch /tmp/xceesd-d.log: every subsequent `esd push ... failed (rc=-1)`.
   Controller log logs no new extres join.
- Frequency: deterministic (3 sightings: 09:29, 10:58 controller restarts
  + the Oct-26 original "unknown role 5" cousin)
- Evidence: t2:/tmp/xceesd-d.log, t1:/opt/xce/logs/xce_controller.log,
  t2:/tmp/e2e/esd-status.json (pushes_ok frozen at 4, pushes_fail growing)

## Root cause
esd.c's reconnect contract was "reconnect = full re-PUSH" (esd_on_reconnect
sets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a
re-JOIN: the fresh controller endpoint is created by handle_connection()
(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets
priv->role = HostRole_extres (join.c handle_join_). The transparent
reconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket
but never re-runs the role join, and the client library's rejoin hook
(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the
un-joined endpoint's resval update falls through BOTH branches of
do_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and
returns rc = -EPERM == -1 forever. (The worker solved this exact shape with
the needs_rejoin contract, controller_connector.c; the scheduler likewise.
The esd -- written later -- re-used the flag machinery but dropped the
re-join itself.)

## Fix
src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and
g_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing
`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread
(outside the library's failover lock -- the xcecd_on_reconnect no-I/O
constraint), and only then does the full re-push. Join failure keeps the
flag set (next tick retries -- the worker's self-healing cadence).

## Verification
- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push
  + test_reconnect_flags_recover_from_any_previous_state
  (src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED
  with the old esd.c (compile error: g_needs_rejoin did not exist -- the
  re-join state was not represented at all), GREEN with the fix: 13/13.
- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build
  rc=0.
- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2
  (restart controller, observe "esd: re-joined after reconnect" + pushes_ok
  advancing + extlic row in `resources distribution list`).

## References
- Related: bug-1760 (the connection-broken predicate that makes the
  debounce-stall recovery work), worker needs_rejoin precedent
  (controller_connector.c), bug-1522 (register-after-start ordering).