E2 Phase 1: external-license-resources — xceesd daemon, extres role, EXTERNAL admission

Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.

Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
  leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
  update_resval_list(c, NULL, ...) — host=="" entries route through
  update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
  non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
  — EEXIST-tolerant so definitions added by an activation register on a
  RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
  fields (valtype/ordtype/flags/interval) in place so a corrected config
  heals a stale registration (bug-1974); seed_sharedinst_from_config skips
  EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
  capacity while the daemon is down)

Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
  in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
  snapshot_load's host=="" precedent — previously WARN-skipped as unknown
  worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
  (F >= uncomit + wanted; F is already net of our checkouts)

xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
  fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
  nothing and reconnects only fire inside a push — after a controller
  restart the daemon sat on a dead connection forever; the main loop now
  forces a full re-push when xcecli_is_connection_broken() (bug-1760
  predicate) latches

Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
  xceadmin tests

Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.

Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.

Co-Authored-By: Claude Code <noreply@anthropic.com>

.wolf/buglog.json

@@ -52,6 +52,87 @@
         "f780a166"
       ],
       "record": ".wolf/bugs/bug-1972.md"
+    },
+    {
+      "id": "bug-1973",
+      "status": "fixed",
+      "status_updated": "2026-10-27",
+      "status_note": "",
+      "component": "controller",
+      "severity": "wrong-result",
+      "timestamp": "2026-10-27T03:27:47Z",
+      "last_seen": "2026-10-27T03:27:47Z",
+      "occurrences": 1,
+      "error_message": "cluster_build_resources aborted whole loop on first EEXIST so new resource definitions never registered on a running controller\n\n**Observed:**\nController log on t1, after pushing a cluster.bin whose resources section ADDED\nthe new `extlic` definition (external-license-resources E2 Phase 1):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory\n```\n\nThe xceesd daemon joined and pushed extlic values, but the controller had\nno `extlic` registration at all (update_resval key_find → -ENOENT).\n\n**Expected:** a config activation that adds a new resource definition must\nregister it in the running controller's resmgr, so daemon pushes land.\n\n**Impact:** every newly-added resource definition was silently inert until\nthe controller process was restarted (boot-time registration path did run).\nFor E2 this meant the whole extres-daemon chain was dead on a live controller.\n\n**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).",
+      "file": "src/controller/cluster.c, src/controller/testsuite/test_config_resource_reconcile.c",
+      "root_cause": "`cluster_build_resources()` looped over the decoded config's resource list\ncalling `makeResource()` for each. `makeResource_` (src/lib/resobj.c:244-247)\nreturns NULL with errno=EEXIST when the name is already registered — which is\nthe COMMON case at activation (boot already registered some definitions). The\nold loop treated that rc as fatal and aborted the whole loop, so every\ndefinition AFTER the first already-registered one was never registered.",
+      "fix": "Rewrite the loop on top of bug-1854's `resourceApplyDef()` (lookup+add under\none write lock, EEXIST-tolerant): RES_APPLY_ADDED/SAME proceed normally;\nRES_APPLY_CHANGED updates the registered object's definition fields\n(valtype/ordtype/flags/interval) IN PLACE — none of them touches the\nregistration slot (idx) or accumulated accounting; mirrors\nreconcile_config_queues' in-place refresh precedent (second live bug, see\nbug-1974 — kept here as one loop). Negative rc still propagates.\n\nVerification:\n- Test: `test_reconcile_resources_registers_new_defs_past_eexist` +\n  `test_reconcile_resources_hot_applies_changed_definition`\n  (src/controller/testsuite/test_config_resource_reconcile.c); RED before the\n  fix (lic1 unregistered / flags stayed 0x00), GREEN after.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 E2E — config gen=15 push with extlic; xceesd push landed in\n  sharedinst (extlic=4), rusage[extlic=3] job RUNNED; second job PENDed at\n  free=1; release re-admitted it.",
+      "tags": [
+        "config",
+        "reconcile",
+        "E2",
+        "external-license"
+      ],
+      "related_bugs": [
+        "bug-1854"
+      ],
+      "fix_commits": [],
+      "record": ".wolf/bugs/bug-1973.md"
+    },
+    {
+      "id": "bug-1974",
+      "status": "fixed",
+      "status_updated": "2026-10-27",
+      "status_note": "",
+      "component": "controller",
+      "severity": "wrong-result",
+      "timestamp": "2026-10-27T03:35:12Z",
+      "last_seen": "2026-10-27T03:35:12Z",
+      "occurrences": 1,
+      "error_message": "config activation kept stale resource definition (flags=0): daemon pushes refused with -EPERM\n\n**Observed:**\nController log on t1 after re-pushing a corrected cluster.bin (extlic mapped\nwith `location: [external]` → flags SHARED|GLOBAL|EXTERNAL):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': Operation not permitted\n```\n\n**Expected:** update_val_res' SHARED branch routes a host==\"\" resval for a\nSHARED-flagged resource into c->sharedinst; with correct flags the push lands.\n\n**Impact:** the extres daemon's pushes were permanently refused on a controller\nwhose registration predated the flag fix — the stale shape was frozen for the\nprocess lifetime, so re-pushing a corrected config did not heal it.\n\n**Found by:** E2 Phase 1 live E2E on t1.",
+      "file": "src/controller/cluster.c, src/controller/worker.c, src/xceadmin/src/xcecommon/mapping.py",
+      "root_cause": "Two compounding factors:\n1. **Producer**: the DEPLOYED xceadmin venv (/opt/xce-denix/share/xce/\n   xceadmin-venv) carries a stale mapping.py with NO resourcemap handling\n   (83-line diff vs tree) — cluster bins it builds carry flags=0 for every\n   resource. Ops issue, fixed by building with the tree venv\n   (src/build.dbg/xceadmin/.venv/bin/xceadmin); the deployed venv still needs\n   redeployment.\n2. **Controller**: even with a corrected bin, `cluster_build_resources()`\n   treated an already-registered name as \"nothing to do\" (bug-1854's\n   RES_APPLY_CHANGED was reported but never applied), so the stale flags=0\n   shape survived every activation.",
+      "fix": "`cluster_build_resources()` now applies RES_APPLY_CHANGED by updating the\nregistered object's definition fields (valtype/ordtype/flags/interval) in\nplace under the resmgr write lock — none of them touches the registration\nslot (idx) or accumulated accounting; mirrors reconcile_config_queues'\n\"updated by configuration activation\" in-place refresh precedent. (Same loop\nas bug-1973's fix.) Producer side tracked separately as ops follow-up:\nredeploy the xceadmin venv.\n\nVerification:\n- Test: `test_reconcile_resources_hot_applies_changed_definition`\n  (src/controller/testsuite/test_config_resource_reconcile.c); RED before\n  (flags stayed 0x00), GREEN after (0x1C, idx preserved).\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 — bin rebuilt with tree venv (gen=15) + fix deployed; xceesd push\n  landed (extlic=4 in `resources distribution list`), rusage[extlic=3] job RUN.",
+      "tags": [
+        "config",
+        "flags",
+        "E2",
+        "external-license"
+      ],
+      "related_bugs": [
+        "bug-1854",
+        "bug-1973"
+      ],
+      "fix_commits": [],
+      "record": ".wolf/bugs/bug-1974.md"
+    },
+    {
+      "id": "bug-1975",
+      "status": "fixed",
+      "status_updated": "2026-10-27",
+      "status_note": "",
+      "component": "controller",
+      "severity": "wrong-result",
+      "timestamp": "2026-10-27T03:40:05Z",
+      "last_seen": "2026-10-27T03:40:05Z",
+      "occurrences": 1,
+      "error_message": "xceesd never reconnects after controller restart when its value is unchanged (debounce suppresses pushes)\n\n**Observed:**\nOld xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after\nthe controller was restarted:\n\n```\n(no socket() attempts; strace showed only collector forks + epoll_wait;\n /proc/net socket table had no trace of the controller connection)\n```\n\nLast log line was the boot-time `esd full re-push: 0/1 source(s)` — no\nfurther activity.\n\n**Expected:** within one poll cycle of the controller's restart the daemon\nmust re-establish the connection and re-push its value (the new primary has\nno record of the daemon or its value).\n\n**Impact:** after any controller restart/failover with a stable license\nvalue, the controller restarts with extlic read 0 (seed skipped for EXTERNAL\nby design) and no daemon re-push — every rusage[extlic] job PENDs forever\nwith NO_RESOURCE until the license value happens to change.\n\n**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).",
+      "file": "src/esd/src/esd.c, src/lib/client.c, src/lib/client_internal.h",
+      "root_cause": "Three mechanisms conspired:\n1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip\n   push) — with a stable value it pushes NOTHING.\n2. A reconnect inside the client library only fires while INSIDE a push's\n   failover call — no pushes, no reconnect.\n3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate;\n   with no push attempt it never runs.",
+      "fix": "`esd_run()`'s main loop now checks `xcecli_is_connection_broken()`\n(bug-1760's pure predicate — the transport's RDHUP handler sets\nXCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches.\nThe write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the\nfull re-push contract for the failover path.\n\nVerification:\n- Test: covered by the live contract (client-side predicate already unit\n  tested by bug-1760's suite); esd-side loop is integration-shaped (the\n  trigger is a transport callback), so live E2E is the proof.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 — controller killed and restarted; esd logged\n  `connection broken -- forcing a full re-push to reconnect`, rotated to the\n  new primary, re-pushed; `xcecli resources distribution list` shows extlic=4\n  and a fresh rusage[extlic=3] job RUNS.",
+      "tags": [
+        "esd",
+        "external-license",
+        "reconnect",
+        "E2"
+      ],
+      "related_bugs": [
+        "bug-1760",
+        "bug-1973",
+        "bug-1974"
+      ],
+      "fix_commits": [],
+      "record": ".wolf/bugs/bug-1975.md"
     }
   ]
 }