lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.wolf/bugs/bug-1973.md
lei.chen 2fcacac2cb E2 Phase 1: external-license-resources — xceesd daemon, extres role, EXTERNAL admission 
Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.

Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
  leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
  update_resval_list(c, NULL, ...) — host=="" entries route through
  update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
  non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
  — EEXIST-tolerant so definitions added by an activation register on a
  RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
  fields (valtype/ordtype/flags/interval) in place so a corrected config
  heals a stale registration (bug-1974); seed_sharedinst_from_config skips
  EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
  capacity while the daemon is down)

Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
  in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
  snapshot_load's host=="" precedent — previously WARN-skipped as unknown
  worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
  (F >= uncomit + wanted; F is already net of our checkouts)

xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
  fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
  nothing and reconnects only fire inside a push — after a controller
  restart the daemon sat on a dead connection forever; the main loop now
  forces a full re-push when xcecli_is_connection_broken() (bug-1760
  predicate) latches

Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
  xceadmin tests

Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.

Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-27 03:45:28 +00:00

3.9 KiB
Raw Blame History

id status status_updated status_note component severity timestamp last_seen occurrences files tags related summary
bug-1973 fixed 2026-10-27 controller wrong-result 2026-10-27T03:27:47Z 2026-10-27T03:27:47Z 1
src/controller/cluster.c
src/controller/testsuite/test_config_resource_reconcile.c
config
reconcile
E2
external-license
bug-1854
cluster_build_resources aborted at first EEXIST: new resource defs never registered on a running controller

bug-1973: cluster_build_resources aborted whole loop on first EEXIST so new resource definitions never registered on a running controller

Description

Observed: Controller log on t1, after pushing a cluster.bin whose resources section ADDED the new extlic definition (external-license-resources E2 Phase 1):

[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory

The xceesd daemon joined and pushed extlic values, but the controller had no extlic registration at all (update_resval key_find → -ENOENT).

Expected: a config activation that adds a new resource definition must register it in the running controller's resmgr, so daemon pushes land.

Impact: every newly-added resource definition was silently inert until the controller process was restarted (boot-time registration path did run). For E2 this meant the whole extres-daemon chain was dead on a live controller.

Found by: E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).

Environment

  • OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
  • libc: glibc 2.42 (de-nix deployment closure)
  • Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
  • Build: debug (build.dbg), deployed with patchelf interpreter+RUNPATH
  • Code: feature/license-resources @ ba42fdd8-dirty
  • Deployment: t1 single-node (controller+scheduler+apiserver+worker), xceesd extres daemon

Reproduction

  1. Controller running with a cluster.bin containing resources {slots, ...}
  2. xceadmin config build && xceadmin config push with a bin that ADDS extlic
  3. Start xceesd; it joins and pushes extlic values
  4. Controller: update_resval 'extlic': No such file or directory (-ENOENT)
  • Frequency: deterministic
  • Evidence: controller log lines quoted above (t1, 2026-10-26 17:40)

Root cause

cluster_build_resources() looped over the decoded config's resource list calling makeResource() for each. makeResource_ (src/lib/resobj.c:244-247) returns NULL with errno=EEXIST when the name is already registered — which is the COMMON case at activation (boot already registered some definitions). The old loop treated that rc as fatal and aborted the whole loop, so every definition AFTER the first already-registered one was never registered.

Fix

Rewrite the loop on top of bug-1854's resourceApplyDef() (lookup+add under one write lock, EEXIST-tolerant): RES_APPLY_ADDED/SAME proceed normally; RES_APPLY_CHANGED updates the registered object's definition fields (valtype/ordtype/flags/interval) IN PLACE — none of them touches the registration slot (idx) or accumulated accounting; mirrors reconcile_config_queues' in-place refresh precedent (second live bug, see bug-1974 — kept here as one loop). Negative rc still propagates.

Verification

  • Test: test_reconcile_resources_registers_new_defs_past_eexist + test_reconcile_resources_hot_applies_changed_definition (src/controller/testsuite/test_config_resource_reconcile.c); RED before the fix (lic1 unregistered / flags stayed 0x00), GREEN after.
  • Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
  • Live: t1 E2E — config gen=15 push with extlic; xceesd push landed in sharedinst (extlic=4), rusage[extlic=3] job RUNNED; second job PENDed at free=1; release re-admitted it.

References

  • Related: bug-1854 (resourceApplyDef change-detection contract this loop is built on), E2 spec .kiro/specs/external-license-resources/design.md §3.2