lei.chen 2fcacac2cb E2 Phase 1: external-license-resources — xceesd daemon, extres role, EXTERNAL admission

Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.

Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
  leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
  update_resval_list(c, NULL, ...) — host=="" entries route through
  update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
  non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
  — EEXIST-tolerant so definitions added by an activation register on a
  RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
  fields (valtype/ordtype/flags/interval) in place so a corrected config
  heals a stale registration (bug-1974); seed_sharedinst_from_config skips
  EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
  capacity while the daemon is down)

Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
  in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
  snapshot_load's host=="" precedent — previously WARN-skipped as unknown
  worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
  (F >= uncomit + wanted; F is already net of our checkouts)

xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
  fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
  nothing and reconnects only fire inside a push — after a controller
  restart the daemon sat on a dead connection forever; the main loop now
  forces a full re-push when xcecli_is_connection_broken() (bug-1760
  predicate) latches

Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
  xceadmin tests

Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.

Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.

Co-Authored-By: Claude Code <noreply@anthropic.com>

2026-10-27 03:45:28 +00:00

3.6 KiB

Raw Blame History

status

status_updated

status_note

component

severity

timestamp

last_seen

occurrences

files

bug-1975: xceesd idle on dead connection after controller restart (stable value: no re-push)

Description

Observed: Old xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after the controller was restarted:

(no socket() attempts; strace showed only collector forks + epoll_wait;
 /proc/net socket table had no trace of the controller connection)

Last log line was the boot-time esd full re-push: 0/1 source(s) — no further activity.

Expected: within one poll cycle of the controller's restart the daemon must re-establish the connection and re-push its value (the new primary has no record of the daemon or its value).

Impact: after any controller restart/failover with a stable license value, the controller restarts with extlic read 0 (seed skipped for EXTERNAL by design) and no daemon re-push — every rusage[extlic] job PENDs forever with NO_RESOURCE until the license value happens to change.

Found by: E2 Phase 1 live E2E on t1 (controller restart step).

Environment

OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
libc: glibc 2.42 (de-nix deployment closure)
Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
Build: debug (build.dbg)
Code: feature/license-resources @ ba42fdd8-dirty
Deployment: t1 single-node; xceesd extres daemon

Reproduction

xceesd running, pushing a value the collector keeps returning unchanged
Restart (or fail over) the controller
Observe the daemon: no reconnect attempts, pushes never resume

Frequency: deterministic whenever the value is stable across the restart
Evidence: strace captures /tmp/esd-strace*.out (t1, 2026-10-26); log silence quoted above

Root cause

Three mechanisms conspired:

esd_poll_round() debounces unchanged values (last_value == v → skip push) — with a stable value it pushes NOTHING.
A reconnect inside the client library only fires while INSIDE a push's failover call — no pushes, no reconnect.
esd_on_reconnect (the rejoin hook) only runs on a successful rotate; with no push attempt it never runs.

Fix

esd_run()'s main loop now checks xcecli_is_connection_broken() (bug-1760's pure predicate — the transport's RDHUP handler sets XCE_CTX_ERR asynchronously) and forces one esd_push_all() when it latches. The write attempt drives the rotate/reconnect; esd_on_reconnect keeps the full re-push contract for the failover path.

Verification

Test: covered by the live contract (client-side predicate already unit tested by bug-1760's suite); esd-side loop is integration-shaped (the trigger is a transport callback), so live E2E is the proof.
Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
Live: t1 — controller killed and restarted; esd logged connection broken -- forcing a full re-push to reconnect, rotated to the new primary, re-pushed; xcecli resources distribution list shows extlic=4 and a fresh rusage[extlic=3] job RUNS.

References

Related: bug-1760 (the pure broken-connection predicate this reuses), E2 spec design.md §3.5 E2E checklist (daemon kill/restart + controller failover items)

3.6 KiB Raw Blame History