lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.wolf/bugs/bug-1974.md
lei.chen 2fcacac2cb E2 Phase 1: external-license-resources — xceesd daemon, extres role, EXTERNAL admission 
Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.

Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
  leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
  update_resval_list(c, NULL, ...) — host=="" entries route through
  update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
  non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
  — EEXIST-tolerant so definitions added by an activation register on a
  RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
  fields (valtype/ordtype/flags/interval) in place so a corrected config
  heals a stale registration (bug-1974); seed_sharedinst_from_config skips
  EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
  capacity while the daemon is down)

Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
  in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
  snapshot_load's host=="" precedent — previously WARN-skipped as unknown
  worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
  (F >= uncomit + wanted; F is already net of our checkouts)

xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
  fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
  nothing and reconnects only fire inside a push — after a controller
  restart the daemon sat on a dead connection forever; the main loop now
  forces a full re-push when xcecli_is_connection_broken() (bug-1760
  predicate) latches

Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
  xceadmin tests

Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.

Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-27 03:45:28 +00:00

3.6 KiB
Raw Blame History

id status status_updated status_note component severity timestamp last_seen occurrences files tags related
bug-1974 fixed 2026-10-27 controller wrong-result 2026-10-27T03:35:12Z 2026-10-27T03:35:12Z 1
src/controller/cluster.c
src/controller/worker.c
src/xceadmin/src/xcecommon/mapping.py
config
flags
E2
external-license
bug-1854
bug-1973

bug-1974: config activation kept stale resource definition (flags=0): daemon pushes refused with -EPERM

Description

Observed: Controller log on t1 after re-pushing a corrected cluster.bin (extlic mapped with location: [external] → flags SHARED|GLOBAL|EXTERNAL):

[ERROR] [log_update_resval_failure] update_resval `extlic': Operation not permitted

Expected: update_val_res' SHARED branch routes a host=="" resval for a SHARED-flagged resource into c->sharedinst; with correct flags the push lands.

Impact: the extres daemon's pushes were permanently refused on a controller whose registration predated the flag fix — the stale shape was frozen for the process lifetime, so re-pushing a corrected config did not heal it.

Found by: E2 Phase 1 live E2E on t1.

Environment

  • OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
  • libc: glibc 2.42 (de-nix deployment closure)
  • Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
  • Build: debug (build.dbg)
  • Code: feature/license-resources @ ba42fdd8-dirty
  • Deployment: t1 single-node; xceesd extres daemon

Reproduction

  1. Register extlic with flags=0 (see contributing factor below)
  2. Push a corrected cluster.bin where extlic has SHARED|GLOBAL|EXTERNAL
  3. xceesd pushes extlic=4 (host=="")
  4. Controller: update_resval 'extlic': Operation not permitted (-EPERM)
  • Frequency: deterministic (persists until controller restart or the bug-1973 fix)
  • Evidence: controller log lines quoted above (t1, 2026-10-26 18:40/18:45)

Root cause

Two compounding factors:

  1. Producer: the DEPLOYED xceadmin venv (/opt/xce-denix/share/xce/ xceadmin-venv) carries a stale mapping.py with NO resourcemap handling (83-line diff vs tree) — cluster bins it builds carry flags=0 for every resource. Ops issue, fixed by building with the tree venv (src/build.dbg/xceadmin/.venv/bin/xceadmin); the deployed venv still needs redeployment.
  2. Controller: even with a corrected bin, cluster_build_resources() treated an already-registered name as "nothing to do" (bug-1854's RES_APPLY_CHANGED was reported but never applied), so the stale flags=0 shape survived every activation.

Fix

cluster_build_resources() now applies RES_APPLY_CHANGED by updating the registered object's definition fields (valtype/ordtype/flags/interval) in place under the resmgr write lock — none of them touches the registration slot (idx) or accumulated accounting; mirrors reconcile_config_queues' "updated by configuration activation" in-place refresh precedent. (Same loop as bug-1973's fix.) Producer side tracked separately as ops follow-up: redeploy the xceadmin venv.

Verification

  • Test: test_reconcile_resources_hot_applies_changed_definition (src/controller/testsuite/test_config_resource_reconcile.c); RED before (flags stayed 0x00), GREEN after (0x1C, idx preserved).
  • Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
  • Live: t1 — bin rebuilt with tree venv (gen=15) + fix deployed; xceesd push landed (extlic=4 in resources distribution list), rusage[extlic=3] job RUN.

References

  • Related: bug-1854 (resourceApplyDef change-detection), bug-1973 (same loop's EEXIST abort), worker.c update_val_res guard (the -EPERM surface)