fix(esd): re-join after transparent reconnect -- bug-1978

A controller restart dropped xceesd's resval pushes forever: the fresh
endpoint is created by handle_connection() with priv->role =
HostRole_client, and only OPCode_join sets HostRole_extres -- but the
old esd_on_reconnect cleared only g_needs_full_push, so the daemon
re-PUSHED onto an un-joined endpoint and every push landed in
do_update_resvals's neither-branch, refused -EPERM forever (live
sighting: pushes_ok frozen, rc=-1 every 10s).

Mirror the worker's needs_rejoin contract: the reconnect handler now
sets BOTH g_needs_rejoin and g_needs_full_push; the main loop consumes
the re-join first (xcecli_join(HostRole_extres) on the main thread,
outside the failover lock -- the xcecd_on_reconnect no-I/O constraint),
retrying next tick on failure, then does the full re-push.

Live t1/t2 proof: controller restart -> "esd: re-joined after
reconnect (rc=0)" + "full re-push: 1/1 source(s)" + extlic back in
`xcecli resources distribution list`. Two new white-box unit tests pin
the both-flags contract (13/13 in the esd suite).

Co-Authored-By: Claude Code <noreply@anthropic.com>

.wolf/buglog.json

@@ -192,6 +192,33 @@
         "8c1995c2"
       ],
       "record": ".wolf/bugs/bug-1977.md"
+    },
+    {
+      "id": "bug-1978",
+      "status": "fixed",
+      "status_updated": "2026-10-28",
+      "status_note": "",
+      "component": "controller",
+      "severity": "wrong-result",
+      "timestamp": "2026-10-28T12:47:59Z",
+      "last_seen": "2026-10-28T12:47:59Z",
+      "occurrences": 3,
+      "error_message": "esd reconnect re-pushes without re-join -- pushes -EPERM forever\n\n**Observed:**\n```\nt2 /tmp/xceesd-d.log:\n[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1)\n[2026-10-28 09:29:49,019] INFO  esd full re-push: 0/1 source(s)\n[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1)\n... (every 10s poll, forever)\n```\nAfter any controller restart the daemon's pushes fail rc=-1 (-EPERM) on\nevery attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED,\nthe poll loop runs, the collector script fires every 10s, the status file\nrefreshes with `stale: false` -- only pushes_fail climbs.\n\n**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and\npushes resume (pushes_ok advances, `xce resources distribution list` shows\nextlic).\n\n**Impact:** the entire external-license value chain dies at the first\ncontroller restart: extlic never reaches c->sharedinst, the scheduler never\nhears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever\n(admission sees free=0). Found during the license-resource regression: the\nregression config push (gen 19) succeeded but `resources distribution list`\nshowed NO extlic row -- daemon健康、值链断。\n\n**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline\nevidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller\nrestarts in /opt/xce/logs/xce_controller.log, zero extres joins after\n10:58, and esd strace showing zero network sends (debounce suppressed -- no\nrotate ever triggered again).",
+      "file": "src/esd/src/esd.c",
+      "root_cause": "esd.c's reconnect contract was \"reconnect = full re-PUSH\" (esd_on_reconnect\nsets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a\nre-JOIN: the fresh controller endpoint is created by handle_connection()\n(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets\npriv->role = HostRole_extres (join.c handle_join_). The transparent\nreconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket\nbut never re-runs the role join, and the client library's rejoin hook\n(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the\nun-joined endpoint's resval update falls through BOTH branches of\ndo_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and\nreturns rc = -EPERM == -1 forever. (The worker solved this exact shape with\nthe needs_rejoin contract, controller_connector.c; the scheduler likewise.\nThe esd -- written later -- re-used the flag machinery but dropped the\nre-join itself.)",
+      "fix": "src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and\ng_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing\n`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread\n(outside the library's failover lock -- the xcecd_on_reconnect no-I/O\nconstraint), and only then does the full re-push. Join failure keeps the\nflag set (next tick retries -- the worker's self-healing cadence).\n\nVerification:\n- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push\n  + test_reconnect_flags_recover_from_any_previous_state\n  (src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED\n  with the old esd.c (compile error: g_needs_rejoin did not exist -- the\n  re-join state was not represented at all), GREEN with the fix: 13/13.\n- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build\n  rc=0.\n- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2\n  (restart controller, observe \"esd: re-joined after reconnect\" + pushes_ok\n  advancing + extlic row in `resources distribution list`).",
+      "tags": [
+        "esd",
+        "reconnect",
+        "failover"
+      ],
+      "related_bugs": [
+        "bug-1760"
+      ],
+      "fix_commits": [
+        "72301fd5"
+      ],
+      "record": ".wolf/bugs/bug-1978.md"
     }
   ]
 }

.wolf/bugs/bug-1978.md

@@ -0,0 +1,110 @@
+---
+id: "bug-1978"
+status: "fixed"
+status_updated: "2026-10-28"
+status_note: ""
+component: "controller"
+severity: "wrong-result"
+timestamp: "2026-10-28T12:47:59Z"
+last_seen: "2026-10-28T12:47:59Z"
+occurrences: 3
+files: ["src/esd/src/esd.c"]
+tags: ["esd", "reconnect", "failover"]
+related: ["bug-1760"]
+fix_commits: ["72301fd5"]
+---
+
+# bug-1978: esd reconnect re-pushes without re-join -- pushes -EPERM forever
+
+## Description
+**Observed:**
+```
+t2 /tmp/xceesd-d.log:
+[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1)
+[2026-10-28 09:29:49,019] INFO  esd full re-push: 0/1 source(s)
+[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1)
+... (every 10s poll, forever)
+```
+After any controller restart the daemon's pushes fail rc=-1 (-EPERM) on
+every attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED,
+the poll loop runs, the collector script fires every 10s, the status file
+refreshes with `stale: false` -- only pushes_fail climbs.
+
+**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and
+pushes resume (pushes_ok advances, `xce resources distribution list` shows
+extlic).
+
+**Impact:** the entire external-license value chain dies at the first
+controller restart: extlic never reaches c->sharedinst, the scheduler never
+hears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever
+(admission sees free=0). Found during the license-resource regression: the
+regression config push (gen 19) succeeded but `resources distribution list`
+showed NO extlic row -- daemon健康、值链断。
+
+**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline
+evidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller
+restarts in /opt/xce/logs/xce_controller.log, zero extres joins after
+10:58, and esd strace showing zero network sends (debounce suppressed -- no
+rotate ever triggered again).
+
+## Environment
+- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
+- libc: glibc 2.39 (t2 host)
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: asan (xce.build) -- failing binary was the de-nix deploy
+  /opt/xce-denix/bin/xceesd (Oct 27 10:51, non-ASAN deploy build)
+- Code: feature/license-resources @ 30c51ea1 (esd) vs controller b38ef6de
+- Deployment: 3-node t1/t2/t3; xceesd on t2, controller+scheduler on t1
+
+## Reproduction
+1. Start xceesd on t2 (`/opt/xce-denix/bin/xceesd -c /tmp/e2e/esd.yaml ...`);
+   observe pushes_ok advance (the boot join works).
+2. Restart the t1 controller (or kill it long enough for the connection to
+   drop and transparently reconnect).
+3. Watch /tmp/xceesd-d.log: every subsequent `esd push ... failed (rc=-1)`.
+   Controller log logs no new extres join.
+- Frequency: deterministic (3 sightings: 09:29, 10:58 controller restarts
+  + the Oct-26 original "unknown role 5" cousin)
+- Evidence: t2:/tmp/xceesd-d.log, t1:/opt/xce/logs/xce_controller.log,
+  t2:/tmp/e2e/esd-status.json (pushes_ok frozen at 4, pushes_fail growing)
+
+## Root cause
+esd.c's reconnect contract was "reconnect = full re-PUSH" (esd_on_reconnect
+sets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a
+re-JOIN: the fresh controller endpoint is created by handle_connection()
+(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets
+priv->role = HostRole_extres (join.c handle_join_). The transparent
+reconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket
+but never re-runs the role join, and the client library's rejoin hook
+(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the
+un-joined endpoint's resval update falls through BOTH branches of
+do_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and
+returns rc = -EPERM == -1 forever. (The worker solved this exact shape with
+the needs_rejoin contract, controller_connector.c; the scheduler likewise.
+The esd -- written later -- re-used the flag machinery but dropped the
+re-join itself.)
+
+## Fix
+src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and
+g_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing
+`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread
+(outside the library's failover lock -- the xcecd_on_reconnect no-I/O
+constraint), and only then does the full re-push. Join failure keeps the
+flag set (next tick retries -- the worker's self-healing cadence).
+
+## Verification
+- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push
+  + test_reconnect_flags_recover_from_any_previous_state
+  (src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED
+  with the old esd.c (compile error: g_needs_rejoin did not exist -- the
+  re-join state was not represented at all), GREEN with the fix: 13/13.
+- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build
+  rc=0.
+- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2
+  (restart controller, observe "esd: re-joined after reconnect" + pushes_ok
+  advancing + extlic row in `resources distribution list`).
+
+## References
+- Related: bug-1760 (the connection-broken predicate that makes the
+  debounce-stall recovery work), worker needs_rejoin precedent
+  (controller_connector.c), bug-1522 (register-after-start ordering).

src/esd/meson.build

@@ -23,10 +23,10 @@ xceesd = executable('xceesd',
                     dependencies: esd_deps,
                     install: true)
 
-# Unit tests: white-box #include of the pure-core sources (config parse,
+# Unit tests: white-box #include of the core sources (config parse,
-# script-output parse) -- same convention as the worker's
+# script-output parse, status writer, and -- since bug-1978 -- esd.c's
-# resource_sync_test. The join/reconnect wiring (esd.c) drives the live
+# reconnect-flag contract; esd_run's live-join sequence is still covered by
-# client library and is covered by the t1 E2E, not unit tests.
+# the t1 E2E) -- same convention as the worker's resource_sync_test.
 esd_test_incdir = include_directories('include', '../include', '../lib', '.')
 esd_test_deps = [libxce_dep, yaml_deps, tinylog_dep, unity_dep]
 

src/esd/src/esd.c

@@ -159,14 +159,24 @@ int esd_push_all(void)
 /* --rejoin hook (client library contract): the library calls this while
  * holding its failover lock -- no I/O, no xcecli_* calls (would self-
  * deadlock, the worker's xcecd_on_reconnect lesson, controller_connector.c).
- * We only set a flag; the main loop consumes it after the lock is free. */
+ * We only set flags; the main loop consumes them after the lock is free. */
 static volatile int g_needs_full_push = 0;
+static volatile int g_needs_rejoin    = 0;
 
 static int esd_on_reconnect(xcecli_ctx_t ctx, void* arg)
 {
     (void)ctx;
     (void)arg;
 
+    /* bug-1978: a re-PUSH alone is not enough -- the fresh controller
+     * endpoint was created by handle_connection() with priv->role =
+     * HostRole_client, and only OPCode_join sets it to HostRole_extres
+     * (join.c). Without the re-join every resval push lands in
+     * do_update_resvals's neither-branch and is refused -EPERM forever
+     * (the live sighting: pushes_ok frozen, rc=-1 on every attempt after
+     * a controller restart). Mirror the worker's needs_rejoin contract:
+     * the main loop re-issues the role join, then does the full re-push. */
+    g_needs_rejoin    = 1;
     g_needs_full_push = 1;
 
     return 0;
@@ -250,6 +260,28 @@ int esd_run(void)
     while (g_esd.running) {
         int i;
 
+        if (g_needs_rejoin) {
+            /* bug-1978: a transparent reconnect just landed -- the fresh
+             * controller endpoint defaults to HostRole_client, so the role
+             * join must be re-issued BEFORE any push (an un-joined push is
+             * refused -EPERM). This runs on the main loop, outside the
+             * library's failover lock (the xcecd_on_reconnect constraint).
+             * Failure keeps the flag set: the next tick retries, mirroring
+             * the worker's needs_rejoin self-healing cadence. rc >= 0 is
+             * success for the same reason as the worker's rejoin (the
+             * controller's reply.status for a plain extres join is 0, but
+             * the failover wrapper surfaces its own codes on success). */
+            int rc = xcecli_join(g_esd.apictx, HostRole_extres, NULL);
+            if (rc >= 0) {
+                g_needs_rejoin = 0;
+                INFO("esd: re-joined after reconnect (rc=%d)", rc);
+            } else {
+                WARN("esd: re-join after reconnect failed (rc=%d) -- "
+                     "retrying next tick",
+                     rc);
+            }
+        }
+
         if (g_needs_full_push) {
             /* A transparent reconnect just landed (new primary after
              * failover, or the old controller restarted): it has no
@@ -272,8 +304,10 @@ int esd_run(void)
          * The broken-connection flag is set asynchronously by the
          * transport's RDHUP handler (bug-1760's pure predicate): when it
          * latches, force one full re-push -- the write attempt triggers the
-         * rotate/reconnect, and esd_on_reconnect's flag keeps the full
+         * rotate/reconnect, and esd_on_reconnect's flags keep the
-         * re-push contract for the failover path too. */
+         * re-join + full re-push contract for the failover path too
+         * (bug-1978: the re-push alone was refused -EPERM on the un-joined
+         * fresh endpoint). */
         if (g_esd.cfg.nsrc > 0 && xcecli_is_connection_broken(g_esd.apictx)) {
             INFO("esd: connection broken -- forcing a full re-push to "
                  "reconnect");

src/esd/testsuite/test_esd_core.c

@@ -26,6 +26,13 @@
 /* White-box: the stale predicate and the status-file writer (Phase 2,
  * design.md 3.5 "运维面"). */
 #include "../src/esd_status.c"
+/* White-box: the reconnect-flag contract (bug-1978). esd.c's main loop
+ * drives the live client library, but the reconnect handler's flag contract
+ * -- set BOTH rejoin and full-push, and the loop consuming rejoin first --
+ * is unit-testable here without a controller: esd_run() itself is not
+ * linked (its join sequence would need a live primary); we drive
+ * esd_on_reconnect and read the loop's guard statics directly. */
+#include "../src/esd.c"
 
 static char g_tmpdir[256];
 
@@ -373,6 +380,38 @@ static void test_status_write_marks_stale(void)
     free(cfg.src);
 }
 
+/* ── bug-1978: the reconnect-flag contract ───────────────────────────── */
+
+static void test_reconnect_sets_both_rejoin_and_full_push(void)
+{
+    g_needs_rejoin    = 0;
+    g_needs_full_push = 0;
+
+    TEST_ASSERT_EQUAL_INT(0, esd_on_reconnect((xcecli_ctx_t)0x1, NULL));
+
+    /* BOTH flags: the re-join must happen before any re-push (an un-joined
+     * push is refused -EPERM on the fresh endpoint), and the re-push must
+     * still happen (the fresh controller knows no values). The bug-1978
+     * defect cleared only g_needs_full_push -- a daemon that re-pushed onto
+     * an un-joined HostRole_client endpoint, every push -EPERM forever. */
+    TEST_ASSERT_EQUAL_INT(1, g_needs_rejoin);
+    TEST_ASSERT_EQUAL_INT(1, g_needs_full_push);
+}
+
+static void test_reconnect_flags_recover_from_any_previous_state(void)
+{
+    /* The handler is level-set, not level-toggle: regardless of what a
+     * previous (already consumed) reconnect left behind, a new callback
+     * invocation re-arms both flags. */
+    g_needs_rejoin    = 0;
+    g_needs_full_push = 1; /* stale leftover from an unconsumed cycle */
+
+    TEST_ASSERT_EQUAL_INT(0, esd_on_reconnect((xcecli_ctx_t)0x2, NULL));
+
+    TEST_ASSERT_EQUAL_INT(1, g_needs_rejoin);
+    TEST_ASSERT_EQUAL_INT(1, g_needs_full_push);
+}
+
 int main(void)
 {
     UNITY_BEGIN();
@@ -388,6 +427,8 @@ int main(void)
     RUN_TEST(test_stale_predicate);
     RUN_TEST(test_status_write_basic);
     RUN_TEST(test_status_write_marks_stale);
+    RUN_TEST(test_reconnect_sets_both_rejoin_and_full_push);
+    RUN_TEST(test_reconnect_flags_recover_from_any_previous_state);
 
     return UNITY_END();
 }