b471ce31ca
A controller restart dropped xceesd's resval pushes forever: the fresh endpoint is created by handle_connection() with priv->role = HostRole_client, and only OPCode_join sets HostRole_extres -- but the old esd_on_reconnect cleared only g_needs_full_push, so the daemon re-PUSHED onto an un-joined endpoint and every push landed in do_update_resvals's neither-branch, refused -EPERM forever (live sighting: pushes_ok frozen, rc=-1 every 10s). Mirror the worker's needs_rejoin contract: the reconnect handler now sets BOTH g_needs_rejoin and g_needs_full_push; the main loop consumes the re-join first (xcecli_join(HostRole_extres) on the main thread, outside the failover lock -- the xcecd_on_reconnect no-I/O constraint), retrying next tick on failure, then does the full re-push. Live t1/t2 proof: controller restart -> "esd: re-joined after reconnect (rc=0)" + "full re-push: 1/1 source(s)" + extlic back in `xcecli resources distribution list`. Two new white-box unit tests pin the both-flags contract (13/13 in the esd suite). Co-Authored-By: Claude Code <noreply@anthropic.com>
110 lines
5.1 KiB
Markdown
110 lines
5.1 KiB
Markdown
---
|
|
id: "bug-1978"
|
|
status: "fixed"
|
|
status_updated: "2026-10-28"
|
|
status_note: ""
|
|
component: "controller"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-10-28T12:47:59Z"
|
|
last_seen: "2026-10-28T12:47:59Z"
|
|
occurrences: 3
|
|
files: ["src/esd/src/esd.c"]
|
|
tags: ["esd", "reconnect", "failover"]
|
|
related: ["bug-1760"]
|
|
fix_commits: ["72301fd5"]
|
|
---
|
|
|
|
# bug-1978: esd reconnect re-pushes without re-join -- pushes -EPERM forever
|
|
|
|
## Description
|
|
**Observed:**
|
|
```
|
|
t2 /tmp/xceesd-d.log:
|
|
[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1)
|
|
[2026-10-28 09:29:49,019] INFO esd full re-push: 0/1 source(s)
|
|
[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1)
|
|
... (every 10s poll, forever)
|
|
```
|
|
After any controller restart the daemon's pushes fail rc=-1 (-EPERM) on
|
|
every attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED,
|
|
the poll loop runs, the collector script fires every 10s, the status file
|
|
refreshes with `stale: false` -- only pushes_fail climbs.
|
|
|
|
**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and
|
|
pushes resume (pushes_ok advances, `xce resources distribution list` shows
|
|
extlic).
|
|
|
|
**Impact:** the entire external-license value chain dies at the first
|
|
controller restart: extlic never reaches c->sharedinst, the scheduler never
|
|
hears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever
|
|
(admission sees free=0). Found during the license-resource regression: the
|
|
regression config push (gen 19) succeeded but `resources distribution list`
|
|
showed NO extlic row -- daemon健康、值链断。
|
|
|
|
**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline
|
|
evidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller
|
|
restarts in /opt/xce/logs/xce_controller.log, zero extres joins after
|
|
10:58, and esd strace showing zero network sends (debounce suppressed -- no
|
|
rotate ever triggered again).
|
|
|
|
## Environment
|
|
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
|
|
- libc: glibc 2.39 (t2 host)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build) -- failing binary was the de-nix deploy
|
|
/opt/xce-denix/bin/xceesd (Oct 27 10:51, non-ASAN deploy build)
|
|
- Code: feature/license-resources @ 30c51ea1 (esd) vs controller b38ef6de
|
|
- Deployment: 3-node t1/t2/t3; xceesd on t2, controller+scheduler on t1
|
|
|
|
## Reproduction
|
|
1. Start xceesd on t2 (`/opt/xce-denix/bin/xceesd -c /tmp/e2e/esd.yaml ...`);
|
|
observe pushes_ok advance (the boot join works).
|
|
2. Restart the t1 controller (or kill it long enough for the connection to
|
|
drop and transparently reconnect).
|
|
3. Watch /tmp/xceesd-d.log: every subsequent `esd push ... failed (rc=-1)`.
|
|
Controller log logs no new extres join.
|
|
- Frequency: deterministic (3 sightings: 09:29, 10:58 controller restarts
|
|
+ the Oct-26 original "unknown role 5" cousin)
|
|
- Evidence: t2:/tmp/xceesd-d.log, t1:/opt/xce/logs/xce_controller.log,
|
|
t2:/tmp/e2e/esd-status.json (pushes_ok frozen at 4, pushes_fail growing)
|
|
|
|
## Root cause
|
|
esd.c's reconnect contract was "reconnect = full re-PUSH" (esd_on_reconnect
|
|
sets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a
|
|
re-JOIN: the fresh controller endpoint is created by handle_connection()
|
|
(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets
|
|
priv->role = HostRole_extres (join.c handle_join_). The transparent
|
|
reconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket
|
|
but never re-runs the role join, and the client library's rejoin hook
|
|
(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the
|
|
un-joined endpoint's resval update falls through BOTH branches of
|
|
do_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and
|
|
returns rc = -EPERM == -1 forever. (The worker solved this exact shape with
|
|
the needs_rejoin contract, controller_connector.c; the scheduler likewise.
|
|
The esd -- written later -- re-used the flag machinery but dropped the
|
|
re-join itself.)
|
|
|
|
## Fix
|
|
src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and
|
|
g_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing
|
|
`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread
|
|
(outside the library's failover lock -- the xcecd_on_reconnect no-I/O
|
|
constraint), and only then does the full re-push. Join failure keeps the
|
|
flag set (next tick retries -- the worker's self-healing cadence).
|
|
|
|
## Verification
|
|
- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push
|
|
+ test_reconnect_flags_recover_from_any_previous_state
|
|
(src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED
|
|
with the old esd.c (compile error: g_needs_rejoin did not exist -- the
|
|
re-join state was not represented at all), GREEN with the fix: 13/13.
|
|
- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build
|
|
rc=0.
|
|
- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2
|
|
(restart controller, observe "esd: re-joined after reconnect" + pushes_ok
|
|
advancing + extlic row in `resources distribution list`).
|
|
|
|
## References
|
|
- Related: bug-1760 (the connection-broken predicate that makes the
|
|
debounce-stall recovery work), worker needs_rejoin precedent
|
|
(controller_connector.c), bug-1522 (register-after-start ordering).
|