The external-license-resources E2 fixture builder crr_build_config
registers five resources and four distributions, writing resources[4],
dists[3] and vals[4]. All four caller test functions declared only
resources[4]/dists[3]/vals[4], so every build_config call wrote past
the caller's stack arrays -- deterministic ASAN stack-buffer-overflow,
the whole controller suite (137 fixtures) red since 2fcacac2.
Resize the four caller arrays to match the builder's sizing
(resources[5]/dists[4]/vals[5]) with a bug-1977 comment on the sizing
contract. No production code touched.
Verified: src/tools/xce-build.sh controller (asan) 6/6 OK, controller
suite 91s green; buglog record .wolf/bugs/bug-1977.md.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Implements Phase 1 of .kiro/specs/external-license-resources/ (design.md):
externally-reported license pools pushed by a dedicated daemon through the
existing resval pipeline, with EXTERNAL-aware admission arithmetic.
Wire & controller:
- controller.capnp HostRole.extres (@5, appended); xce_join_req accepts it
- extres.c: HostRole_extres join table (endpoint+name, no workermgr slot),
leave cleanup; join.c/cluster.c dispatch
- opupd.c do_update_resvals: extres branch runs the resource hooks and
update_resval_list(c, NULL, ...) — host=="" entries route through
update_val_res' SHARED branch into c->sharedinst; per-entry -EPERM for
non-shared targets is logged, never fails the list
- worker.c update_val_res: NULL worker only writes SHARED resources
- cluster.c cluster_build_resources: rebuilt on resourceApplyDef (bug-1854)
— EEXIST-tolerant so definitions added by an activation register on a
RUNNING controller (bug-1973), and RES_APPLY_CHANGED updates definition
fields (valtype/ordtype/flags/interval) in place so a corrected config
heals a stale registration (bug-1974); seed_sharedinst_from_config skips
EXTERNAL pools (daemon-reported free counts only; seeding would fabricate
capacity while the daemon is down)
Scheduler:
- core_events.c RESVALS_DELTA: NULL/"" host marks a SHARED value and lands
in core->sharedInst via dup_val (bug-1585 ownership rule), mirroring
snapshot_load's host=="" precedent — previously WARN-skipped as unknown
worker
- resource.c unsatisfy_: EXTERNAL skips the inuse term
(F >= uncomit + wanted; F is already net of our checkouts)
xceesd (src/esd, new component):
- script collectors polled per source, debounce on unchanged values,
fail-open stale policy; pushes host=="" shared resvals over xcecli
- reconnect gap fix (bug-1975): with a stable value the debounce pushed
nothing and reconnects only fire inside a push — after a controller
restart the daemon sat on a dead connection forever; the main loop now
forces a full re-push when xcecli_is_connection_broken() (bug-1760
predicate) latches
Config/mapping:
- resources.yaml: extlic sample (location: [external])
- mapping.py: RESOURCE_FLAG_EXTERNAL (0x10) + location:["external"] mapping;
xceadmin tests
Tests (TDD, red->green per fix): test_extres_resvals.c (join+resval
pipeline), test_config_resource_reconcile.c +2 (register past EEXIST /
hot-apply changed definition), test_core_events.c +131 lines (shared DELTA
landing), mapping tests. Suites: controller 1358/0, scheduler 100/0,
esd 7/0, lib 1/0; check-no-fork covers src/esd.
Live t1 E2E: config gen=15 push -> xceesd push extlic=4 lands in
sharedinst -> rusage[extlic=3] job RUN; second job PENDs at free=1;
release re-admits it; controller restart -> forced re-push restores the
value and admission resumes. Full precommit job verification passed.
Co-Authored-By: Claude Code <noreply@anthropic.com>
rusage collection (merge cb0c7fe9 regression, root cause of 0 counters):
- collect_job_rusages builds the systemd unit name with
make_job_sd_unit_name (four-segment fmt_jobid_ex form) instead of a bare
%lu of the packed uint64, so the D-Bus property lookups hit the real
per-job unit again. E2E on t1: DONE job reports max_mem/avg_mem/cpu_time
non-zero (previously all 0).
XCE_JOBID written identity:
- exdata sets XCE_JOBID via fmt_jobid (four-segment
cluster-apiserver-timestamp-seq) instead of a bare packed decimal;
XCE_JOBINDEX handling unchanged.
- xrun parses XCE_JOBID with parse_jobid (accepts the four-segment form);
a bare decimal no longer round-trips through the written identity.
- E2E on t1: job sees XCE_JOBID=<submit id>; xrun remote-start round-trip
to another host matches the job.
select WARN (silent degradation):
- parse_resreq_primary WARNs when a select references an unregistered
resource ("...references an unregistered resource -- dropping the select
and skipping any rusage[] clause of this primary") instead of failing
silently; scheduler log shows the WARN on t1 and the job schedules
without the constraint.
CLI env handling (unit-tested; subEnvVars is not flag-bound so this is
not reachable from the production CLI):
- getEnvironmentVariables keeps B=b=c (split at first '=') and warns +
drops a bare key or an '=value' fragment instead of silently mis-parsing.
Docs:
- src/docs/xce.md: document the dead-clause semantics (select/rusage
primary drop behavior) matching the WARN.
bug-1972 (test-only UAF): test_job_live_gc asserted job destruction by
reading refcount from the freed job; rewrite the assertion to observe
destruction without touching freed memory. controller_unit_test is green
again (1354 tests).
Regenerate 3 repl_trace_baseline fixtures whose baselines shifted with the
select-WARN logging; replay tests green.
Spec naming: external-license-resources design.md/brief.md fix the
self-referential daemon-name note (old drafts said xceextresd/extresd).
Verified on t1: rusage_burn job (counters non-zero in DONE), echo_env job
(four-segment XCE_JOBID), select[nosuchres] job (WARN + schedules),
xrun t2 remote-start round-trip, CLI env unit tests, xcelib 762 /
controller 1354 / worker exdata 8+11 / sched repl_trace 4+14+4+10+17
tests green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
A failed test run's evidence (meson-logs/testlog.txt + the mkdtemp sweep's
FAILED-case /tmp/xce_* fixture dirs) was destroyed by the next build:
setup_build rm -rf's the build dir, and any subsequent meson test overwrites
testlog.* in place. t2 2026-10-22: a controller case failed at ~05:46, the
make clean + rerun at ~05:52 left 134/134 OK and zero trace of what failed.
Two chokes in xce-build.sh:
- finalize_failure_evidence (EXIT trap): on rc!=0 with RUN_TESTS=1, copy
meson-logs to testlogs.archive/<BUILD_DIR>/this-<stamp>-rc<N>/ and drop
the .meson_test_failed_rc marker; also releases the build lock (the old
lock-release trap folded in).
- archive_previous_failure (run start, pre-wipe): if the marker exists,
move the previous run's meson-logs to prev-<stamp>-rc<N>/ -- outside the
build dir, unreachable by the wipe; sweep aged (>5 min) /tmp/xce_*
FAILED-case fixture dirs into prev-<stamp>-fixtures/ so they sit with the
testlog that explains them; cap the archive at newest 10 per build dir.
Also -type d on BOTH /tmp/xce_* sweeps (this one and bug-924's
prune_stale_tmp_fixtures): mkdtemp fixtures are always directories, but
the bare name also matches regular files -- the untyped find live-moved
daemon debug logs (xce_scheduler.log, xce_strace*.log) into the archive,
and the bug-924 variant rm -rf's its matches.
Test: src/tools/test-failure-evidence-archive.sh, 6 scenarios / 11
assertions exercising the real function bodies (sourced with surgical
stubbing) -- all green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
