lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.wolf/bugs/bug-1973.md
lei.chen 2fcacac2cb E2 Phase 1: external-license-resources — xceesd daemon, extres role, EXTERNAL admission 
Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.

Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
  leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
  update_resval_list(c, NULL, ...) — host=="" entries route through
  update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
  non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
  — EEXIST-tolerant so definitions added by an activation register on a
  RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
  fields (valtype/ordtype/flags/interval) in place so a corrected config
  heals a stale registration (bug-1974); seed_sharedinst_from_config skips
  EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
  capacity while the daemon is down)

Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
  in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
  snapshot_load's host=="" precedent — previously WARN-skipped as unknown
  worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
  (F >= uncomit + wanted; F is already net of our checkouts)

xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
  fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
  nothing and reconnects only fire inside a push — after a controller
  restart the daemon sat on a dead connection forever; the main loop now
  forces a full re-push when xcecli_is_connection_broken() (bug-1760
  predicate) latches

Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
  xceadmin tests

Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.

Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-10-27 03:45:28 +00:00

85 lines
3.9 KiB
Markdown
Raw Blame History

---
id: "bug-1973"
status: "fixed"
status_updated: "2026-10-27"
status_note: ""
component: "controller"
severity: "wrong-result"
timestamp: "2026-10-27T03:27:47Z"
last_seen: "2026-10-27T03:27:47Z"
occurrences: 1
files: ["src/controller/cluster.c", "src/controller/testsuite/test_config_resource_reconcile.c"]
tags: ["config", "reconcile", "E2", "external-license"]
related: ["bug-1854"]
summary: "cluster_build_resources aborted at first EEXIST: new resource defs never registered on a running controller"
---
# bug-1973: cluster_build_resources aborted whole loop on first EEXIST so new resource definitions never registered on a running controller
## Description
**Observed:**
Controller log on t1, after pushing a cluster.bin whose resources section ADDED
the new `extlic` definition (external-license-resources E2 Phase 1):
```
[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory
```
The xceesd daemon joined and pushed extlic values, but the controller had
no `extlic` registration at all (update_resval key_find → -ENOENT).
**Expected:** a config activation that adds a new resource definition must
register it in the running controller's resmgr, so daemon pushes land.
**Impact:** every newly-added resource definition was silently inert until
the controller process was restarted (boot-time registration path did run).
For E2 this meant the whole extres-daemon chain was dead on a live controller.
**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).
## Environment
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
- libc: glibc 2.42 (de-nix deployment closure)
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: debug (build.dbg), deployed with patchelf interpreter+RUNPATH
- Code: feature/license-resources @ ba42fdd8-dirty
- Deployment: t1 single-node (controller+scheduler+apiserver+worker), xceesd extres daemon
## Reproduction
1. Controller running with a cluster.bin containing resources {slots, ...}
2. `xceadmin config build && xceadmin config push` with a bin that ADDS `extlic`
3. Start xceesd; it joins and pushes extlic values
4. Controller: `update_resval 'extlic': No such file or directory` (-ENOENT)
- Frequency: deterministic
- Evidence: controller log lines quoted above (t1, 2026-10-26 17:40)
## Root cause
`cluster_build_resources()` looped over the decoded config's resource list
calling `makeResource()` for each. `makeResource_` (src/lib/resobj.c:244-247)
returns NULL with errno=EEXIST when the name is already registered — which is
the COMMON case at activation (boot already registered some definitions). The
old loop treated that rc as fatal and aborted the whole loop, so every
definition AFTER the first already-registered one was never registered.
## Fix
Rewrite the loop on top of bug-1854's `resourceApplyDef()` (lookup+add under
one write lock, EEXIST-tolerant): RES_APPLY_ADDED/SAME proceed normally;
RES_APPLY_CHANGED updates the registered object's definition fields
(valtype/ordtype/flags/interval) IN PLACE — none of them touches the
registration slot (idx) or accumulated accounting; mirrors
reconcile_config_queues' in-place refresh precedent (second live bug, see
bug-1974 — kept here as one loop). Negative rc still propagates.
## Verification
- Test: `test_reconcile_resources_registers_new_defs_past_eexist` +
`test_reconcile_resources_hot_applies_changed_definition`
(src/controller/testsuite/test_config_resource_reconcile.c); RED before the
fix (lic1 unregistered / flags stayed 0x00), GREEN after.
- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).
- Live: t1 E2E — config gen=15 push with extlic; xceesd push landed in
sharedinst (extlic=4), rusage[extlic=3] job RUNNED; second job PENDed at
free=1; release re-admitted it.
## References
- Related: bug-1854 (resourceApplyDef change-detection contract this loop is
built on), E2 spec .kiro/specs/external-license-resources/design.md §3.2
Reference in a new issue View git blame Copy permalink