E2 Phase 1: external-license-resources — xceesd daemon, extres role, EXTERNAL admission

Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.

Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
  leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
  update_resval_list(c, NULL, ...) — host=="" entries route through
  update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
  non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
  — EEXIST-tolerant so definitions added by an activation register on a
  RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
  fields (valtype/ordtype/flags/interval) in place so a corrected config
  heals a stale registration (bug-1974); seed_sharedinst_from_config skips
  EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
  capacity while the daemon is down)

Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
  in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
  snapshot_load's host=="" precedent — previously WARN-skipped as unknown
  worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
  (F >= uncomit + wanted; F is already net of our checkouts)

xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
  fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
  nothing and reconnects only fire inside a push — after a controller
  restart the daemon sat on a dead connection forever; the main loop now
  forces a full re-push when xcecli_is_connection_broken() (bug-1760
  predicate) latches

Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
  xceadmin tests

Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.

Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.

Co-Authored-By: Claude Code <noreply@anthropic.com>

.wolf/bugs/bug-1973.md

@@ -0,0 +1,85 @@
+---
+id: "bug-1973"
+status: "fixed"
+status_updated: "2026-10-27"
+status_note: ""
+component: "controller"
+severity: "wrong-result"
+timestamp: "2026-10-27T03:27:47Z"
+last_seen: "2026-10-27T03:27:47Z"
+occurrences: 1
+files: ["src/controller/cluster.c", "src/controller/testsuite/test_config_resource_reconcile.c"]
+tags: ["config", "reconcile", "E2", "external-license"]
+related: ["bug-1854"]
+summary: "cluster_build_resources aborted at first EEXIST: new resource defs never registered on a running controller"
+---
+
+# bug-1973: cluster_build_resources aborted whole loop on first EEXIST so new resource definitions never registered on a running controller
+
+## Description
+**Observed:**
+Controller log on t1, after pushing a cluster.bin whose resources section ADDED
+the new `extlic` definition (external-license-resources E2 Phase 1):
+
+```
+[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory
+```
+
+The xceesd daemon joined and pushed extlic values, but the controller had
+no `extlic` registration at all (update_resval key_find → -ENOENT).
+
+**Expected:** a config activation that adds a new resource definition must
+register it in the running controller's resmgr, so daemon pushes land.
+
+**Impact:** every newly-added resource definition was silently inert until
+the controller process was restarted (boot-time registration path did run).
+For E2 this meant the whole extres-daemon chain was dead on a live controller.
+
+**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).
+
+## Environment
+- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
+- libc: glibc 2.42 (de-nix deployment closure)
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: debug (build.dbg), deployed with patchelf interpreter+RUNPATH
+- Code: feature/license-resources @ ba42fdd8-dirty
+- Deployment: t1 single-node (controller+scheduler+apiserver+worker), xceesd extres daemon
+
+## Reproduction
+1. Controller running with a cluster.bin containing resources {slots, ...}
+2. `xceadmin config build && xceadmin config push` with a bin that ADDS `extlic`
+3. Start xceesd; it joins and pushes extlic values
+4. Controller: `update_resval 'extlic': No such file or directory` (-ENOENT)
+- Frequency: deterministic
+- Evidence: controller log lines quoted above (t1, 2026-10-26 17:40)
+
+## Root cause
+`cluster_build_resources()` looped over the decoded config's resource list
+calling `makeResource()` for each. `makeResource_` (src/lib/resobj.c:244-247)
+returns NULL with errno=EEXIST when the name is already registered — which is
+the COMMON case at activation (boot already registered some definitions). The
+old loop treated that rc as fatal and aborted the whole loop, so every
+definition AFTER the first already-registered one was never registered.
+
+## Fix
+Rewrite the loop on top of bug-1854's `resourceApplyDef()` (lookup+add under
+one write lock, EEXIST-tolerant): RES_APPLY_ADDED/SAME proceed normally;
+RES_APPLY_CHANGED updates the registered object's definition fields
+(valtype/ordtype/flags/interval) IN PLACE — none of them touches the
+registration slot (idx) or accumulated accounting; mirrors
+reconcile_config_queues' in-place refresh precedent (second live bug, see
+bug-1974 — kept here as one loop). Negative rc still propagates.
+
+## Verification
+- Test: `test_reconcile_resources_registers_new_defs_past_eexist` +
+  `test_reconcile_resources_hot_applies_changed_definition`
+  (src/controller/testsuite/test_config_resource_reconcile.c); RED before the
+  fix (lic1 unregistered / flags stayed 0x00), GREEN after.
+- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
+- Live: t1 E2E — config gen=15 push with extlic; xceesd push landed in
+  sharedinst (extlic=4), rusage[extlic=3] job RUNNED; second job PENDed at
+  free=1; release re-admitted it.
+
+## References
+- Related: bug-1854 (resourceApplyDef change-detection contract this loop is
+  built on), E2 spec .kiro/specs/external-license-resources/design.md §3.2

.wolf/bugs/bug-1974.md

@@ -0,0 +1,84 @@
+---
+id: "bug-1974"
+status: "fixed"
+status_updated: "2026-10-27"
+status_note: ""
+component: "controller"
+severity: "wrong-result"
+timestamp: "2026-10-27T03:35:12Z"
+last_seen: "2026-10-27T03:35:12Z"
+occurrences: 1
+files: ["src/controller/cluster.c", "src/controller/worker.c", "src/xceadmin/src/xcecommon/mapping.py"]
+tags: ["config", "flags", "E2", "external-license"]
+related: ["bug-1854", "bug-1973"]
+---
+
+# bug-1974: config activation kept stale resource definition (flags=0): daemon pushes refused with -EPERM
+
+## Description
+**Observed:**
+Controller log on t1 after re-pushing a corrected cluster.bin (extlic mapped
+with `location: [external]` → flags SHARED|GLOBAL|EXTERNAL):
+
+```
+[ERROR] [log_update_resval_failure] update_resval `extlic': Operation not permitted
+```
+
+**Expected:** update_val_res' SHARED branch routes a host=="" resval for a
+SHARED-flagged resource into c->sharedinst; with correct flags the push lands.
+
+**Impact:** the extres daemon's pushes were permanently refused on a controller
+whose registration predated the flag fix — the stale shape was frozen for the
+process lifetime, so re-pushing a corrected config did not heal it.
+
+**Found by:** E2 Phase 1 live E2E on t1.
+
+## Environment
+- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
+- libc: glibc 2.42 (de-nix deployment closure)
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: debug (build.dbg)
+- Code: feature/license-resources @ ba42fdd8-dirty
+- Deployment: t1 single-node; xceesd extres daemon
+
+## Reproduction
+1. Register extlic with flags=0 (see contributing factor below)
+2. Push a corrected cluster.bin where extlic has SHARED|GLOBAL|EXTERNAL
+3. xceesd pushes extlic=4 (host=="")
+4. Controller: `update_resval 'extlic': Operation not permitted` (-EPERM)
+- Frequency: deterministic (persists until controller restart or the bug-1973 fix)
+- Evidence: controller log lines quoted above (t1, 2026-10-26 18:40/18:45)
+
+## Root cause
+Two compounding factors:
+1. **Producer**: the DEPLOYED xceadmin venv (/opt/xce-denix/share/xce/
+   xceadmin-venv) carries a stale mapping.py with NO resourcemap handling
+   (83-line diff vs tree) — cluster bins it builds carry flags=0 for every
+   resource. Ops issue, fixed by building with the tree venv
+   (src/build.dbg/xceadmin/.venv/bin/xceadmin); the deployed venv still needs
+   redeployment.
+2. **Controller**: even with a corrected bin, `cluster_build_resources()`
+   treated an already-registered name as "nothing to do" (bug-1854's
+   RES_APPLY_CHANGED was reported but never applied), so the stale flags=0
+   shape survived every activation.
+
+## Fix
+`cluster_build_resources()` now applies RES_APPLY_CHANGED by updating the
+registered object's definition fields (valtype/ordtype/flags/interval) in
+place under the resmgr write lock — none of them touches the registration
+slot (idx) or accumulated accounting; mirrors reconcile_config_queues'
+"updated by configuration activation" in-place refresh precedent. (Same loop
+as bug-1973's fix.) Producer side tracked separately as ops follow-up:
+redeploy the xceadmin venv.
+
+## Verification
+- Test: `test_reconcile_resources_hot_applies_changed_definition`
+  (src/controller/testsuite/test_config_resource_reconcile.c); RED before
+  (flags stayed 0x00), GREEN after (0x1C, idx preserved).
+- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
+- Live: t1 — bin rebuilt with tree venv (gen=15) + fix deployed; xceesd push
+  landed (extlic=4 in `resources distribution list`), rusage[extlic=3] job RUN.
+
+## References
+- Related: bug-1854 (resourceApplyDef change-detection), bug-1973 (same loop's
+  EEXIST abort), worker.c update_val_res guard (the -EPERM surface)

.wolf/bugs/bug-1975.md

@@ -0,0 +1,88 @@
+---
+id: "bug-1975"
+status: "fixed"
+status_updated: "2026-10-27"
+status_note: ""
+component: "controller"
+severity: "wrong-result"
+timestamp: "2026-10-27T03:40:05Z"
+last_seen: "2026-10-27T03:40:05Z"
+occurrences: 1
+files: ["src/esd/src/esd.c", "src/lib/client.c", "src/lib/client_internal.h"]
+tags: ["esd", "external-license", "reconnect", "E2"]
+related: ["bug-1760", "bug-1973", "bug-1974"]
+summary: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
+title: "xceesd idle on dead connection after controller restart (stable value: no re-push)"
+---
+
+# bug-1975: xceesd idle on dead connection after controller restart (stable value: no re-push)
+
+## Description
+**Observed:**
+Old xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after
+the controller was restarted:
+
+```
+(no socket() attempts; strace showed only collector forks + epoll_wait;
+ /proc/net socket table had no trace of the controller connection)
+```
+
+Last log line was the boot-time `esd full re-push: 0/1 source(s)` — no
+further activity.
+
+**Expected:** within one poll cycle of the controller's restart the daemon
+must re-establish the connection and re-push its value (the new primary has
+no record of the daemon or its value).
+
+**Impact:** after any controller restart/failover with a stable license
+value, the controller restarts with extlic read 0 (seed skipped for EXTERNAL
+by design) and no daemon re-push — every rusage[extlic] job PENDs forever
+with NO_RESOURCE until the license value happens to change.
+
+**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).
+
+## Environment
+- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
+- libc: glibc 2.42 (de-nix deployment closure)
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: debug (build.dbg)
+- Code: feature/license-resources @ ba42fdd8-dirty
+- Deployment: t1 single-node; xceesd extres daemon
+
+## Reproduction
+1. xceesd running, pushing a value the collector keeps returning unchanged
+2. Restart (or fail over) the controller
+3. Observe the daemon: no reconnect attempts, pushes never resume
+- Frequency: deterministic whenever the value is stable across the restart
+- Evidence: strace captures /tmp/esd-strace*.out (t1, 2026-10-26); log silence quoted above
+
+## Root cause
+Three mechanisms conspired:
+1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip
+   push) — with a stable value it pushes NOTHING.
+2. A reconnect inside the client library only fires while INSIDE a push's
+   failover call — no pushes, no reconnect.
+3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate;
+   with no push attempt it never runs.
+
+## Fix
+`esd_run()`'s main loop now checks `xcecli_is_connection_broken()`
+(bug-1760's pure predicate — the transport's RDHUP handler sets
+XCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches.
+The write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the
+full re-push contract for the failover path.
+
+## Verification
+- Test: covered by the live contract (client-side predicate already unit
+  tested by bug-1760's suite); esd-side loop is integration-shaped (the
+  trigger is a transport callback), so live E2E is the proof.
+- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
+- Live: t1 — controller killed and restarted; esd logged
+  `connection broken -- forcing a full re-push to reconnect`, rotated to the
+  new primary, re-pushed; `xcecli resources distribution list` shows extlic=4
+  and a fresh rusage[extlic=3] job RUNS.
+
+## References
+- Related: bug-1760 (the pure broken-connection predicate this reuses),
+  E2 spec design.md §3.5 E2E checklist (daemon kill/restart + controller
+  failover items)