E2 Phase 1: external-license-resources — xceesd daemon, extres role, EXTERNAL admission

Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.

Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
  leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
  update_resval_list(c, NULL, ...) — host=="" entries route through
  update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
  non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
  — EEXIST-tolerant so definitions added by an activation register on a
  RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
  fields (valtype/ordtype/flags/interval) in place so a corrected config
  heals a stale registration (bug-1974); seed_sharedinst_from_config skips
  EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
  capacity while the daemon is down)

Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
  in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
  snapshot_load's host=="" precedent — previously WARN-skipped as unknown
  worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
  (F >= uncomit + wanted; F is already net of our checkouts)

xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
  fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
  nothing and reconnects only fire inside a push — after a controller
  restart the daemon sat on a dead connection forever; the main loop now
  forces a full re-push when xcecli_is_connection_broken() (bug-1760
  predicate) latches

Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
  xceadmin tests

Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.

Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.

Co-Authored-By: Claude Code <noreply@anthropic.com>

.wolf/bugs/bug-1975.md

@@ -0,0 +1,88 @@
+---
+id: "bug-1975"
+status: "fixed"
+status_updated: "2026-10-27"
+status_note: ""
+component: "controller"
+severity: "wrong-result"
+timestamp: "2026-10-27T03:40:05Z"
+last_seen: "2026-10-27T03:40:05Z"
+occurrences: 1
+files: ["src/esd/src/esd.c", "src/lib/client.c", "src/lib/client_internal.h"]
+tags: ["esd", "external-license", "reconnect", "E2"]
+related: ["bug-1760", "bug-1973", "bug-1974"]
+summary: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
+title: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
+---
+
+# bug-1975: xceesd idle on dead connection after controller restart (stable value: no re-push)
+
+## Description
+**Observed:**
+Old xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after
+the controller was restarted:
+
+```
+(no socket() attempts; strace showed only collector forks + epoll_wait;
+ /proc/net socket table had no trace of the controller connection)
+```
+
+Last log line was the boot-time `esd full re-push: 0/1 source(s)` — no
+further activity.
+
+**Expected:** within one poll cycle of the controller's restart the daemon
+must re-establish the connection and re-push its value (the new primary has
+no record of the daemon or its value).
+
+**Impact:** after any controller restart/failover with a stable license
+value, the controller restarts with extlic read 0 (seed skipped for EXTERNAL
+by design) and no daemon re-push — every rusage[extlic] job PENDs forever
+with NO_RESOURCE until the license value happens to change.
+
+**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).
+
+## Environment
+- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
+- libc: glibc 2.42 (de-nix deployment closure)
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: debug (build.dbg)
+- Code: feature/license-resources @ ba42fdd8-dirty
+- Deployment: t1 single-node; xceesd extres daemon
+
+## Reproduction
+1. xceesd running, pushing a value the collector keeps returning unchanged
+2. Restart (or fail over) the controller
+3. Observe the daemon: no reconnect attempts, pushes never resume
+- Frequency: deterministic whenever the value is stable across the restart
+- Evidence: strace captures /tmp/esd-strace*.out (t1, 2026-10-26); log silence quoted above
+
+## Root cause
+Three mechanisms conspired:
+1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip
+   push) — with a stable value it pushes NOTHING.
+2. A reconnect inside the client library only fires while INSIDE a push's
+   failover call — no pushes, no reconnect.
+3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate;
+   with no push attempt it never runs.
+
+## Fix
+`esd_run()`'s main loop now checks `xcecli_is_connection_broken()`
+(bug-1760's pure predicate — the transport's RDHUP handler sets
+XCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches.
+The write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the
+full re-push contract for the failover path.
+
+## Verification
+- Test: covered by the live contract (client-side predicate already unit
+  tested by bug-1760's suite); esd-side loop is integration-shaped (the
+  trigger is a transport callback), so live E2E is the proof.
+- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
+- Live: t1 — controller killed and restarted; esd logged
+  `connection broken -- forcing a full re-push to reconnect`, rotated to the
+  new primary, re-pushed; `xcecli resources distribution list` shows extlic=4
+  and a fresh rusage[extlic=3] job RUNS.
+
+## References
+- Related: bug-1760 (the pure broken-connection predicate this reuses),
+  E2 spec design.md §3.5 E2E checklist (daemon kill/restart + controller
+  failover items)