do_update_resvals() fired the AFTER_EVENT HOOK_DETAIL_RESOURCE_UPDATE with
the raw client xce_resval_list_t. update_resval_list() rejects unknown
resource names (-ENOENT) but only logged it, so both listeners -- the
scheduler push and the apiserver store (apiserver_resource_hook_fun ->
handler.go) -- received names the controller had rejected, and the
apiserver stored them.
- update_resval_list() takes an optional `applied` view (caller-sized,
borrowed pointers) and fills it with just the accepted items.
- do_update_resvals() hands that view to the AFTER_EVENT hook, and skips
it when nothing was applied. The request still owns every item.
New test_update_resval_list_reports_only_applied_items (RED before: both
items reported). Controller suite 1237/0. Live /xce-test-resources with
mem=5 totally-fake-resource-xyz=1: mem MATCH at controller, scheduler and
apiserver; the fake name is rejected at injection and appears 0 times in
scheduler.log / apiserver.log (verdict FAIL by design, for the rejected
name).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
update_val_res() memcpy'd the xce_val_t into w->host.resvals, so the
"type"/"model" string resources aliased the interned xstrings owned by
w->host.attrs. rejoin_worker() drops the old attrs strings, then
apply_join_state() -> resync_arm_round() resolves the previous join's
still-outstanding round, firing the worker hooks before
add_hostattrs_resvals() re-seeds resvals; the apiserver hook's
collect_resval() strdup()ed the freed hostmodel. Live: a newly promoted
primary died ~8s after election during an HA soak.
A string resval now stores a private strdup (taken before the old copy is
freed), and the owning arrays free their entries: host.resvals gets
kill_resval (replacing the FIXME NULL killer) and c->sharedinst's
resval_obj_killer (previously a no-op) frees the string and the value --
both also leaked every xce_val_t before. Also closes the same alias for
worker-reported string resvals and config values.
Regression test: test_rejoin_worker_string_resvals_survive_outstanding_round
(RED reproduced the live ASAN stack; GREEN after). meson test 109/109.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Closes bug-1755: worker_send_resync_query() hardcoded pktlimit=0 in
its xce_job_state_query_req() call, so the controller's outbound
OPCode_jobStateQuery always sent every non-terminal job ID in one
unbounded page regardless of count - the query direction never
actually exercised the pagination mechanism task 1.4/7.4 proved
correct, unlike the report direction's real, wired
XCECD_RESYNC_REPORT_PKTLIMIT bound.
Add XCE_RESYNC_QUERY_PKTLIMIT=100, mirroring that exact precedent
(same value - a JobStateQuery entry is a bare jobid, smaller than a
JobStateReport entry), and thread it into the one call site. Existing
small-job-count behavior is unchanged; a rejoin exceeding the cap now
genuinely produces multiple wire pages, proven against the real
production call site (not a bypass) with a 250-job/3-page test.
This is the last task in worker-rejoin-resync's implementation plan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Arm a reconciliation round in apply_join_state() the moment a worker
host becomes connected (first join, reconnect, or failover-reconnect -
all routed through the same shared call site, Req 1.1-1.3), then send
the query from on_worker_join() strictly after the join reply and
existing config push (Req 2.1-2.2), verified via a structural test
that reads real wire order off the endpoint queue.
Also closes Req 3.7's config-delivery seam end-to-end: worker_resync_
window_ms now rides the existing Conf4Worker push (clustername
precedent, no new opcode) and the worker applies it to its retention
window on receipt.
Fixes bug-1752: myreq_handler had no OPCode_push dispatch case at all,
so every config push - including the pre-existing clustername field -
was silently discarded. Confirmed harmless until now (clustername was
never read anywhere in src/worker/); closed narrowly for
worker_resync_window_ms only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Three bugs found while diagnosing a live controller log, all in the
worker resource-value path:
1. Every host relying on the documented hosts.yaml auto-detect default
(no max_slot, meaning "use CPU count") got 0 scheduling slots,
permanently blocking the scheduler from placing jobs on it. mxj is
UInt32 on the wire and can't carry the schema's -1 sentinel; mapping.py
worked around this by leaving it unset, which defaults to 0 -- and 0 is
never < 0, so apply_join_state()'s auto-detect fallback never fired.
Fixed by encoding the sentinel as 0xFFFFFFFF, which the controller's
`int mxj` field reinterprets back to -1 via its existing implicit
uint32_t->int conversion. No C-side or wire schema change needed for
this part; the fallback itself was already correct but had zero test
coverage for a negative mxj, now added.
2. update_resvals()/update_resval_list() logged the failure reason via
PERROR, which reads process-global errno -- but update_resval()'s
failure paths return raw negative codes directly and never touch
errno, so the controller log filled with misleading "update_resval:
Success" / "update_resval: Resource temporarily unavailable" lines
instead of the real reason. Fixed to decode the actual rc via
xce_strerror_r and name the failing resource.
3. maxmem is canonically stored/wired as raw bytes but had no unit
metadata, so resval/workerlist printed it as a bare number that read
as garbage (e.g. workerlist's fixed-width columns visually smeared a
multi-GB byte count across the MaxCPU/MaxMem headers). Added an
optional `unit` field to the Resource/ResourceVal capnp structs (and
xce_resource_t/xce_resval_t/resources.yaml) so resval and workerlist
can render a human-readable value (e.g. "30.90 GB") while the stored
value stays raw bytes -- display-only, every other resource unaffected.
Regenerated the xceadmin golden fixture (fixtures/full/expected.bin)
twice: once for the mxj sentinel encoding change, once for the new
Resource.unit field growing every resource's wire size by one pointer
slot. Both diffs verified byte-for-byte isolated to exactly the expected
change before regenerating.
Verified: 1095/1095 controller_unit_test (stable across repeated runs),
229/229 xceadmin pytest, clean rebuild of lib/controller/xceadmin/xce_test.
Two related findings logged to .wolf/buglog.json but deliberately not
fixed here (out of scope per discussion): a pre-existing flaky segfault
in the combined controller_unit_test binary, and maxswap/maxtmp being
hardcoded stub values on the worker side.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Only 21 of 86 test_*.c group functions called their own nested
UNITY_BEGIN()/UNITY_END(). Each nested call reset Unity's global
NumberOfTests/TestFailures counters, so controller_test.c's final
`return UNITY_END();` -- the value meson test actually gates on --
only ever reflected whichever small window of tests ran since the
LAST nested reset, silently discarding pass/fail evidence for
everything before it (measured: 16 of 972 total tests in one run).
The same nested-BEGIN calls also caused PASS/FAIL banners to show
the wrong source file for tests in files that don't reset it
themselves (Unity.TestFile just carries over from whichever group
last called UNITY_BEGIN()).
Fix: replace every nested UNITY_BEGIN() with UnitySetTestFile(__FILE__)
(sets file attribution only, no counter reset) and drop the matching
UNITY_END(); add the same UnitySetTestFile(__FILE__) as the first
statement to the other group functions that had neither. The 2
genuinely standalone binaries (test_job_array_sched_push.c,
test_job_array_apiserver_push.c -- each has its own int main(), not
registered via controller_test.c's TEST() macro) are untouched.
Verified: single consolidated summary now reads
"972 Tests 0 Failures 0 Ignored OK", exactly matching the
ground-truth RUN_TEST call-site count across all 83 registered
group files (976 total minus 4 belonging to the 2 standalone
binaries). Every PASS/FAIL banner now names its own true source
file. Full xce-build.sh all regression pass clean (Ok: 9, Fail: 0).
No production code touched -- test-harness plumbing only.
