fix(scheduler,controller): worker resource values always display as 0.00 (bug-1568)
xce_scheduler -i -s <replay>: "info worker <name>" showed total=0.00 available=0.00 for every resource, even though the live cluster (xce_test resval/workerlist) reported real non-zero values for the same worker. Two independent root causes, both live-traced end-to-end against a real deployed cluster (value-level TRACE at every hop: controller push, scheduler apply, snapshot save, snapshot load/decode): 1. src/scheduler/src/core_events.c: XCE_CORE_EVENT_WORKERS_PUSH called sched_add_worker() -- a plain key_add(), add-only -- unconditionally on every push. A worker connecting triggers multiple WORKERS_PUSH events in quick succession (an early push before the controller finishes writing real capacity values, carrying stale/zero data, followed moments later by a push carrying the real values). The FIRST push always won the race; every push after it for the same worker name silently failed -EEXIST and its resource values were discarded -- not just at join, for the rest of the process's lifetime (unlike jobs, there was no "update an existing worker" event path at all). Fixed via a new core_upsert_worker() (mirrors the existing core_upsert_queue() add-or-update precedent): looks up the worker by name first, updates status+resvals in place if found, else builds+adds a new one as before. 2. src/scheduler/src/repl_info.c: repl_info_worker_one()/ repl_info_resource_one() checked `xce_val_t.which == XCE_VAL_NUMERIC` -- but XCE_VAL_NUMERIC (xce/types.h, value 1) is a resource-DEFINITION value-type constant (xce_resource_t.valtype's own enum), a completely different enum from xce_val_t.which's real domain, Value_u_which (controller.capnp.h, Value_u_fvalue=5). Every producer in this codebase sets xce_val_t.which to Value_u_fvalue, never XCE_VAL_NUMERIC, so the comparison was always false -- every genuinely-numeric resval displayed as 0.00 unconditionally, independent of fix (1). Corrected all 4 occurrences to Value_u_fvalue. Both fixed via TDD (RED test before each fix, GREEN after): test_core_events.c's test_workers_push_twice_updates_existing_ worker_in_place (two pushes, same name, different values -- both must apply) and test_repl_breakpoints_inspection.c's test_info_worker_reports_resource_values (a real numeric resval must display, not fall back to 0.00). Live-validated by reproducing the user's exact original scenario end-to-end (deploy a real cluster, record, `xce_scheduler -i -s <replay>`, `info worker <name>`) -- now shows slots=22.00, maxmem=33182089216.00 (30.9GB), mem=17151664128.00 (16GB), matching the user's own live xce_test resval/workerlist numbers exactly. Full scheduler suite green (41/41, excluding pre-existing unrelated bug-1560); controller suite green (1102/1104, the same pre-existing bug-1561 pair, confirmed unrelated to this change). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
9f539583d3
commit
aabddf6025
9 changed files with 446 additions and 38 deletions
|
|
@ -264,6 +264,25 @@ static int xcesched_push_workers(cluster_t* c)
|
|||
workers.workers = (xce_worker_t**)wlist.data;
|
||||
workers.count = workers.n_total;
|
||||
|
||||
/* bug-1568 instrumentation: this is the FULL initial worker sync sent
|
||||
* once, at on_scheduler_join() -- distinct from sched_worker_change()'s
|
||||
* per-event incremental push (this file, above), which only fires
|
||||
* AFTER a scheduler is already connected. A scheduler that connects
|
||||
* before this controller's own resource_sync has completed its first
|
||||
* poll cycle would see 0 resvals here even though this push path
|
||||
* itself is working correctly. */
|
||||
if (workers.n_total > 0) {
|
||||
int total_resvals = 0;
|
||||
|
||||
for (i = 0; i < workers.n_total; i++) {
|
||||
xce_worker_t* w = workers.workers[i];
|
||||
|
||||
total_resvals += (w != NULL) ? w->nresvals_to_sched : 0;
|
||||
}
|
||||
TRACE("%s initial sync: %d worker(s), %d total resval(s)", __func__,
|
||||
workers.n_total, total_resvals);
|
||||
}
|
||||
|
||||
rc = xce_push_data_req(c->sched.ep, c->myhost, &data, check_controller, c,
|
||||
100);
|
||||
|
||||
|
|
@ -507,6 +526,14 @@ static int sched_worker_change(int type, int situation, void* data,
|
|||
if (c->sched.ep != NULL) {
|
||||
switch (situation) {
|
||||
case HOOK_AFTER_EVENT:
|
||||
/* bug-1568 instrumentation: the controller-side count actually
|
||||
* being pushed to the scheduler right now -- compare against
|
||||
* core_build_worker()'s "WORKERS_PUSH carries N resval(s)"
|
||||
* TRACE (scheduler/src/core_events.c) in the scheduler's own
|
||||
* log for the same worker/timestamp to confirm the wire push
|
||||
* itself is not where a resource value gets lost. */
|
||||
TRACE("%s worker %s pushing %d resval(s) to scheduler", __func__,
|
||||
w->host.name, resvalist.count);
|
||||
xlist_init(&wlist, 2);
|
||||
xlist_store(&wlist, w);
|
||||
rc = xcesched_push_workerlist(c, &wlist);
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue