fix(esd): re-join after transparent reconnect -- bug-1978

A controller restart dropped xceesd's resval pushes forever: the fresh
endpoint is created by handle_connection() with priv->role =
HostRole_client, and only OPCode_join sets HostRole_extres -- but the
old esd_on_reconnect cleared only g_needs_full_push, so the daemon
re-PUSHED onto an un-joined endpoint and every push landed in
do_update_resvals's neither-branch, refused -EPERM forever (live
sighting: pushes_ok frozen, rc=-1 every 10s).

Mirror the worker's needs_rejoin contract: the reconnect handler now
sets BOTH g_needs_rejoin and g_needs_full_push; the main loop consumes
the re-join first (xcecli_join(HostRole_extres) on the main thread,
outside the failover lock -- the xcecd_on_reconnect no-I/O constraint),
retrying next tick on failure, then does the full re-push.

Live t1/t2 proof: controller restart -> "esd: re-joined after
reconnect (rc=0)" + "full re-push: 1/1 source(s)" + extlic back in
`xcecli resources distribution list`. Two new white-box unit tests pin
the both-flags contract (13/13 in the esd suite).

Co-Authored-By: Claude Code <noreply@anthropic.com>

.wolf/buglog.json

@@ -192,6 +192,33 @@
         "8c1995c2"
       ],
       "record": ".wolf/bugs/bug-1977.md"
+    },
+    {
+      "id": "bug-1978",
+      "status": "fixed",
+      "status_updated": "2026-10-28",
+      "status_note": "",
+      "component": "controller",
+      "severity": "wrong-result",
+      "timestamp": "2026-10-28T12:47:59Z",
+      "last_seen": "2026-10-28T12:47:59Z",
+      "occurrences": 3,
+      "error_message": "esd reconnect re-pushes without re-join -- pushes -EPERM forever\n\n**Observed:**\n```\nt2 /tmp/xceesd-d.log:\n[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1)\n[2026-10-28 09:29:49,019] INFO  esd full re-push: 0/1 source(s)\n[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1)\n... (every 10s poll, forever)\n```\nAfter any controller restart the daemon's pushes fail rc=-1 (-EPERM) on\nevery attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED,\nthe poll loop runs, the collector script fires every 10s, the status file\nrefreshes with `stale: false` -- only pushes_fail climbs.\n\n**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and\npushes resume (pushes_ok advances, `xce resources distribution list` shows\nextlic).\n\n**Impact:** the entire external-license value chain dies at the first\ncontroller restart: extlic never reaches c->sharedinst, the scheduler never\nhears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever\n(admission sees free=0). Found during the license-resource regression: the\nregression config push (gen 19) succeeded but `resources distribution list`\nshowed NO extlic row -- daemon健康、值链断。\n\n**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline\nevidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller\nrestarts in /opt/xce/logs/xce_controller.log, zero extres joins after\n10:58, and esd strace showing zero network sends (debounce suppressed -- no\nrotate ever triggered again).",
+      "file": "src/esd/src/esd.c",
+      "root_cause": "esd.c's reconnect contract was \"reconnect = full re-PUSH\" (esd_on_reconnect\nsets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a\nre-JOIN: the fresh controller endpoint is created by handle_connection()\n(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets\npriv->role = HostRole_extres (join.c handle_join_). The transparent\nreconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket\nbut never re-runs the role join, and the client library's rejoin hook\n(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the\nun-joined endpoint's resval update falls through BOTH branches of\ndo_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and\nreturns rc = -EPERM == -1 forever. (The worker solved this exact shape with\nthe needs_rejoin contract, controller_connector.c; the scheduler likewise.\nThe esd -- written later -- re-used the flag machinery but dropped the\nre-join itself.)",
+      "fix": "src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and\ng_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing\n`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread\n(outside the library's failover lock -- the xcecd_on_reconnect no-I/O\nconstraint), and only then does the full re-push. Join failure keeps the\nflag set (next tick retries -- the worker's self-healing cadence).\n\nVerification:\n- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push\n  + test_reconnect_flags_recover_from_any_previous_state\n  (src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED\n  with the old esd.c (compile error: g_needs_rejoin did not exist -- the\n  re-join state was not represented at all), GREEN with the fix: 13/13.\n- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build\n  rc=0.\n- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2\n  (restart controller, observe \"esd: re-joined after reconnect\" + pushes_ok\n  advancing + extlic row in `resources distribution list`).",
+      "tags": [
+        "esd",
+        "reconnect",
+        "failover"
+      ],
+      "related_bugs": [
+        "bug-1760"
+      ],
+      "fix_commits": [
+        "72301fd5"
+      ],
+      "record": ".wolf/bugs/bug-1978.md"
     }
   ]
 }