From b471ce31cab1167fcfb5f3a07a98aae39e0b8e77 Mon Sep 17 00:00:00 2001 From: "lei.chen" Date: Wed, 28 Oct 2026 14:07:28 +0000 Subject: [PATCH] fix(esd): re-join after transparent reconnect -- bug-1978 A controller restart dropped xceesd's resval pushes forever: the fresh endpoint is created by handle_connection() with priv->role = HostRole_client, and only OPCode_join sets HostRole_extres -- but the old esd_on_reconnect cleared only g_needs_full_push, so the daemon re-PUSHED onto an un-joined endpoint and every push landed in do_update_resvals's neither-branch, refused -EPERM forever (live sighting: pushes_ok frozen, rc=-1 every 10s). Mirror the worker's needs_rejoin contract: the reconnect handler now sets BOTH g_needs_rejoin and g_needs_full_push; the main loop consumes the re-join first (xcecli_join(HostRole_extres) on the main thread, outside the failover lock -- the xcecd_on_reconnect no-I/O constraint), retrying next tick on failure, then does the full re-push. Live t1/t2 proof: controller restart -> "esd: re-joined after reconnect (rc=0)" + "full re-push: 1/1 source(s)" + extlic back in `xcecli resources distribution list`. Two new white-box unit tests pin the both-flags contract (13/13 in the esd suite). Co-Authored-By: Claude Code --- .wolf/buglog.json | 27 ++++++++ .wolf/bugs/bug-1978.md | 110 ++++++++++++++++++++++++++++++ src/esd/meson.build | 8 +-- src/esd/src/esd.c | 40 ++++++++++- src/esd/testsuite/test_esd_core.c | 41 +++++++++++ 5 files changed, 219 insertions(+), 7 deletions(-) create mode 100644 .wolf/bugs/bug-1978.md diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 95143cc9..118a9a92 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -192,6 +192,33 @@ "8c1995c2" ], "record": ".wolf/bugs/bug-1977.md" + }, + { + "id": "bug-1978", + "status": "fixed", + "status_updated": "2026-10-28", + "status_note": "", + "component": "controller", + "severity": "wrong-result", + "timestamp": "2026-10-28T12:47:59Z", + "last_seen": "2026-10-28T12:47:59Z", + "occurrences": 3, + "error_message": "esd reconnect re-pushes without re-join -- pushes -EPERM forever\n\n**Observed:**\n```\nt2 /tmp/xceesd-d.log:\n[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1)\n[2026-10-28 09:29:49,019] INFO esd full re-push: 0/1 source(s)\n[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1)\n... (every 10s poll, forever)\n```\nAfter any controller restart the daemon's pushes fail rc=-1 (-EPERM) on\nevery attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED,\nthe poll loop runs, the collector script fires every 10s, the status file\nrefreshes with `stale: false` -- only pushes_fail climbs.\n\n**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and\npushes resume (pushes_ok advances, `xce resources distribution list` shows\nextlic).\n\n**Impact:** the entire external-license value chain dies at the first\ncontroller restart: extlic never reaches c->sharedinst, the scheduler never\nhears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever\n(admission sees free=0). Found during the license-resource regression: the\nregression config push (gen 19) succeeded but `resources distribution list`\nshowed NO extlic row -- daemon健康、值链断。\n\n**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline\nevidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller\nrestarts in /opt/xce/logs/xce_controller.log, zero extres joins after\n10:58, and esd strace showing zero network sends (debounce suppressed -- no\nrotate ever triggered again).", + "file": "src/esd/src/esd.c", + "root_cause": "esd.c's reconnect contract was \"reconnect = full re-PUSH\" (esd_on_reconnect\nsets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a\nre-JOIN: the fresh controller endpoint is created by handle_connection()\n(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets\npriv->role = HostRole_extres (join.c handle_join_). The transparent\nreconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket\nbut never re-runs the role join, and the client library's rejoin hook\n(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the\nun-joined endpoint's resval update falls through BOTH branches of\ndo_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and\nreturns rc = -EPERM == -1 forever. (The worker solved this exact shape with\nthe needs_rejoin contract, controller_connector.c; the scheduler likewise.\nThe esd -- written later -- re-used the flag machinery but dropped the\nre-join itself.)", + "fix": "src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and\ng_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing\n`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread\n(outside the library's failover lock -- the xcecd_on_reconnect no-I/O\nconstraint), and only then does the full re-push. Join failure keeps the\nflag set (next tick retries -- the worker's self-healing cadence).\n\nVerification:\n- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push\n + test_reconnect_flags_recover_from_any_previous_state\n (src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED\n with the old esd.c (compile error: g_needs_rejoin did not exist -- the\n re-join state was not represented at all), GREEN with the fix: 13/13.\n- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build\n rc=0.\n- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2\n (restart controller, observe \"esd: re-joined after reconnect\" + pushes_ok\n advancing + extlic row in `resources distribution list`).", + "tags": [ + "esd", + "reconnect", + "failover" + ], + "related_bugs": [ + "bug-1760" + ], + "fix_commits": [ + "72301fd5" + ], + "record": ".wolf/bugs/bug-1978.md" } ] } diff --git a/.wolf/bugs/bug-1978.md b/.wolf/bugs/bug-1978.md new file mode 100644 index 00000000..f763d290 --- /dev/null +++ b/.wolf/bugs/bug-1978.md @@ -0,0 +1,110 @@ +--- +id: "bug-1978" +status: "fixed" +status_updated: "2026-10-28" +status_note: "" +component: "controller" +severity: "wrong-result" +timestamp: "2026-10-28T12:47:59Z" +last_seen: "2026-10-28T12:47:59Z" +occurrences: 3 +files: ["src/esd/src/esd.c"] +tags: ["esd", "reconnect", "failover"] +related: ["bug-1760"] +fix_commits: ["72301fd5"] +--- + +# bug-1978: esd reconnect re-pushes without re-join -- pushes -EPERM forever + +## Description +**Observed:** +``` +t2 /tmp/xceesd-d.log: +[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1) +[2026-10-28 09:29:49,019] INFO esd full re-push: 0/1 source(s) +[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1) +... (every 10s poll, forever) +``` +After any controller restart the daemon's pushes fail rc=-1 (-EPERM) on +every attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED, +the poll loop runs, the collector script fires every 10s, the status file +refreshes with `stale: false` -- only pushes_fail climbs. + +**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and +pushes resume (pushes_ok advances, `xce resources distribution list` shows +extlic). + +**Impact:** the entire external-license value chain dies at the first +controller restart: extlic never reaches c->sharedinst, the scheduler never +hears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever +(admission sees free=0). Found during the license-resource regression: the +regression config push (gen 19) succeeded but `resources distribution list` +showed NO extlic row -- daemon健康、值链断。 + +**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline +evidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller +restarts in /opt/xce/logs/xce_controller.log, zero extres joins after +10:58, and esd strace showing zero network sends (debounce suppressed -- no +rotate ever triggered again). + +## Environment +- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64 +- libc: glibc 2.39 (t2 host) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build) -- failing binary was the de-nix deploy + /opt/xce-denix/bin/xceesd (Oct 27 10:51, non-ASAN deploy build) +- Code: feature/license-resources @ 30c51ea1 (esd) vs controller b38ef6de +- Deployment: 3-node t1/t2/t3; xceesd on t2, controller+scheduler on t1 + +## Reproduction +1. Start xceesd on t2 (`/opt/xce-denix/bin/xceesd -c /tmp/e2e/esd.yaml ...`); + observe pushes_ok advance (the boot join works). +2. Restart the t1 controller (or kill it long enough for the connection to + drop and transparently reconnect). +3. Watch /tmp/xceesd-d.log: every subsequent `esd push ... failed (rc=-1)`. + Controller log logs no new extres join. +- Frequency: deterministic (3 sightings: 09:29, 10:58 controller restarts + + the Oct-26 original "unknown role 5" cousin) +- Evidence: t2:/tmp/xceesd-d.log, t1:/opt/xce/logs/xce_controller.log, + t2:/tmp/e2e/esd-status.json (pushes_ok frozen at 4, pushes_fail growing) + +## Root cause +esd.c's reconnect contract was "reconnect = full re-PUSH" (esd_on_reconnect +sets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a +re-JOIN: the fresh controller endpoint is created by handle_connection() +(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets +priv->role = HostRole_extres (join.c handle_join_). The transparent +reconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket +but never re-runs the role join, and the client library's rejoin hook +(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the +un-joined endpoint's resval update falls through BOTH branches of +do_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and +returns rc = -EPERM == -1 forever. (The worker solved this exact shape with +the needs_rejoin contract, controller_connector.c; the scheduler likewise. +The esd -- written later -- re-used the flag machinery but dropped the +re-join itself.) + +## Fix +src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and +g_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing +`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread +(outside the library's failover lock -- the xcecd_on_reconnect no-I/O +constraint), and only then does the full re-push. Join failure keeps the +flag set (next tick retries -- the worker's self-healing cadence). + +## Verification +- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push + + test_reconnect_flags_recover_from_any_previous_state + (src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED + with the old esd.c (compile error: g_needs_rejoin did not exist -- the + re-join state was not represented at all), GREEN with the fix: 13/13. +- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build + rc=0. +- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2 + (restart controller, observe "esd: re-joined after reconnect" + pushes_ok + advancing + extlic row in `resources distribution list`). + +## References +- Related: bug-1760 (the connection-broken predicate that makes the + debounce-stall recovery work), worker needs_rejoin precedent + (controller_connector.c), bug-1522 (register-after-start ordering). diff --git a/src/esd/meson.build b/src/esd/meson.build index 445af1ed..9bf4a993 100644 --- a/src/esd/meson.build +++ b/src/esd/meson.build @@ -23,10 +23,10 @@ xceesd = executable('xceesd', dependencies: esd_deps, install: true) -# Unit tests: white-box #include of the pure-core sources (config parse, -# script-output parse) -- same convention as the worker's -# resource_sync_test. The join/reconnect wiring (esd.c) drives the live -# client library and is covered by the t1 E2E, not unit tests. +# Unit tests: white-box #include of the core sources (config parse, +# script-output parse, status writer, and -- since bug-1978 -- esd.c's +# reconnect-flag contract; esd_run's live-join sequence is still covered by +# the t1 E2E) -- same convention as the worker's resource_sync_test. esd_test_incdir = include_directories('include', '../include', '../lib', '.') esd_test_deps = [libxce_dep, yaml_deps, tinylog_dep, unity_dep] diff --git a/src/esd/src/esd.c b/src/esd/src/esd.c index 364c8f81..26996f70 100644 --- a/src/esd/src/esd.c +++ b/src/esd/src/esd.c @@ -159,14 +159,24 @@ int esd_push_all(void) /* --rejoin hook (client library contract): the library calls this while * holding its failover lock -- no I/O, no xcecli_* calls (would self- * deadlock, the worker's xcecd_on_reconnect lesson, controller_connector.c). - * We only set a flag; the main loop consumes it after the lock is free. */ + * We only set flags; the main loop consumes them after the lock is free. */ static volatile int g_needs_full_push = 0; +static volatile int g_needs_rejoin = 0; static int esd_on_reconnect(xcecli_ctx_t ctx, void* arg) { (void)ctx; (void)arg; + /* bug-1978: a re-PUSH alone is not enough -- the fresh controller + * endpoint was created by handle_connection() with priv->role = + * HostRole_client, and only OPCode_join sets it to HostRole_extres + * (join.c). Without the re-join every resval push lands in + * do_update_resvals's neither-branch and is refused -EPERM forever + * (the live sighting: pushes_ok frozen, rc=-1 on every attempt after + * a controller restart). Mirror the worker's needs_rejoin contract: + * the main loop re-issues the role join, then does the full re-push. */ + g_needs_rejoin = 1; g_needs_full_push = 1; return 0; @@ -250,6 +260,28 @@ int esd_run(void) while (g_esd.running) { int i; + if (g_needs_rejoin) { + /* bug-1978: a transparent reconnect just landed -- the fresh + * controller endpoint defaults to HostRole_client, so the role + * join must be re-issued BEFORE any push (an un-joined push is + * refused -EPERM). This runs on the main loop, outside the + * library's failover lock (the xcecd_on_reconnect constraint). + * Failure keeps the flag set: the next tick retries, mirroring + * the worker's needs_rejoin self-healing cadence. rc >= 0 is + * success for the same reason as the worker's rejoin (the + * controller's reply.status for a plain extres join is 0, but + * the failover wrapper surfaces its own codes on success). */ + int rc = xcecli_join(g_esd.apictx, HostRole_extres, NULL); + if (rc >= 0) { + g_needs_rejoin = 0; + INFO("esd: re-joined after reconnect (rc=%d)", rc); + } else { + WARN("esd: re-join after reconnect failed (rc=%d) -- " + "retrying next tick", + rc); + } + } + if (g_needs_full_push) { /* A transparent reconnect just landed (new primary after * failover, or the old controller restarted): it has no @@ -272,8 +304,10 @@ int esd_run(void) * The broken-connection flag is set asynchronously by the * transport's RDHUP handler (bug-1760's pure predicate): when it * latches, force one full re-push -- the write attempt triggers the - * rotate/reconnect, and esd_on_reconnect's flag keeps the full - * re-push contract for the failover path too. */ + * rotate/reconnect, and esd_on_reconnect's flags keep the + * re-join + full re-push contract for the failover path too + * (bug-1978: the re-push alone was refused -EPERM on the un-joined + * fresh endpoint). */ if (g_esd.cfg.nsrc > 0 && xcecli_is_connection_broken(g_esd.apictx)) { INFO("esd: connection broken -- forcing a full re-push to " "reconnect"); diff --git a/src/esd/testsuite/test_esd_core.c b/src/esd/testsuite/test_esd_core.c index b12a1f0b..2ad03a85 100644 --- a/src/esd/testsuite/test_esd_core.c +++ b/src/esd/testsuite/test_esd_core.c @@ -26,6 +26,13 @@ /* White-box: the stale predicate and the status-file writer (Phase 2, * design.md 3.5 "运维面"). */ #include "../src/esd_status.c" +/* White-box: the reconnect-flag contract (bug-1978). esd.c's main loop + * drives the live client library, but the reconnect handler's flag contract + * -- set BOTH rejoin and full-push, and the loop consuming rejoin first -- + * is unit-testable here without a controller: esd_run() itself is not + * linked (its join sequence would need a live primary); we drive + * esd_on_reconnect and read the loop's guard statics directly. */ +#include "../src/esd.c" static char g_tmpdir[256]; @@ -373,6 +380,38 @@ static void test_status_write_marks_stale(void) free(cfg.src); } +/* ── bug-1978: the reconnect-flag contract ───────────────────────────── */ + +static void test_reconnect_sets_both_rejoin_and_full_push(void) +{ + g_needs_rejoin = 0; + g_needs_full_push = 0; + + TEST_ASSERT_EQUAL_INT(0, esd_on_reconnect((xcecli_ctx_t)0x1, NULL)); + + /* BOTH flags: the re-join must happen before any re-push (an un-joined + * push is refused -EPERM on the fresh endpoint), and the re-push must + * still happen (the fresh controller knows no values). The bug-1978 + * defect cleared only g_needs_full_push -- a daemon that re-pushed onto + * an un-joined HostRole_client endpoint, every push -EPERM forever. */ + TEST_ASSERT_EQUAL_INT(1, g_needs_rejoin); + TEST_ASSERT_EQUAL_INT(1, g_needs_full_push); +} + +static void test_reconnect_flags_recover_from_any_previous_state(void) +{ + /* The handler is level-set, not level-toggle: regardless of what a + * previous (already consumed) reconnect left behind, a new callback + * invocation re-arms both flags. */ + g_needs_rejoin = 0; + g_needs_full_push = 1; /* stale leftover from an unconsumed cycle */ + + TEST_ASSERT_EQUAL_INT(0, esd_on_reconnect((xcecli_ctx_t)0x2, NULL)); + + TEST_ASSERT_EQUAL_INT(1, g_needs_rejoin); + TEST_ASSERT_EQUAL_INT(1, g_needs_full_push); +} + int main(void) { UNITY_BEGIN(); @@ -388,6 +427,8 @@ int main(void) RUN_TEST(test_stale_predicate); RUN_TEST(test_status_write_basic); RUN_TEST(test_status_write_marks_stale); + RUN_TEST(test_reconnect_sets_both_rejoin_and_full_push); + RUN_TEST(test_reconnect_flags_recover_from_any_previous_state); return UNITY_END(); }