myxce/.wolf/bugs/bug-1568.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	legacy
bug-1568	fixed	2026-09-24		scheduler	unknown	2026-08-17T09:00:00Z	2026-09-19T09:00:00Z	4	src/scheduler/src/repl_info.c src/scheduler/src/snapshot_wire.c src/scheduler/src/snapshot_load.c src/controller/sched.c	scheduler controller simulation-mode resource-values worker-resvals live-verified confirmed-bug sched_add_worker key_add-eexist tdd	bug-1567	true

bug-1568: xce_scheduler -i -s : 'info worker ' shows total=0.00 available=0.00 for every…

Description

xce_scheduler -i -s : 'info worker ' shows total=0.00 available=0.00 for every resource, even though the live cluster (xce_test resval/workerlist) reports real non-zero values for the same worker -- CONFIRMED ROOT CAUSE (superseding this entry's earlier 'architecture gap' theory, which was wrong -- corrected after the user pointed to worker.c's real mark_host_res()/update_val_res() write path)

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

CONFIRMED via live value-level TRACE instrumentation (not just counts this time): xce_core_push_events()'s XCE_CORE_EVENT_WORKERS_PUSH handler (src/scheduler/src/core_events.c) calls sched_add_worker() UNCONDITIONALLY on every push -- and sched_add_worker() (src/scheduler/src/core.c:270) is a plain key_add() (ADD-ONLY semantics: fails -EEXIST if the worker name is already registered, never updates in place). A worker connecting triggers MULTIPLE WORKERS_PUSH events in quick succession (confirmed live: sched.c's collect_resval()/xcesched_push_workers() fire more than once during join sequencing -- e.g. an early push before src/controller/worker.c's apply_join_state()->add_hostattrs_resvals()/set_host_slots() has finished writing real hostattrs-derived values into w->host.resvals, carrying stale/zero data, followed by a later push carrying the real values). The FIRST push's sched_add_worker() call succeeds and registers the worker (confirmed: 'sched_add_worker("pc13v") ok (idx=0)' at 10:02:34.848, immediately preceded by a TRACE showing it received maxmem=0.000000). The SECOND push's freshly-built xce_sched_worker_t -- this time carrying the REAL values (confirmed: 'resource maxmem received fvalue=33182089216.000000', matching xce_test's own real 30.9GB reading, at 10:02:36.849 -- silently fails sched_add_worker() with rc=-17 (-EEXIST) one line later in the SAME log, and its return value was unchecked until this investigation's own instrumentation added the WARN (core_events.c, added this session) that finally surfaced it. The correctly-valued worker object is discarded/leaked; core->workers['pc13v'] permanently keeps the FIRST, zero-valued object. This is not limited to the initial-join race -- because there is no 'update an existing worker in place' event type (unlike jobs, which have XCE_CORE_EVENT_JOB_STATE_CHANGE/JOB_UPDATE for in-place mutation of an already-added job), EVERY subsequent WORKERS_PUSH for an already-registered worker (sched_worker_change's ongoing per-event pushes throughout normal cluster operation, not just at join) hits the identical silent EEXIST-discard. The scheduler's view of a worker's resources freezes forever at whichever push happened to land first and win the race, and never updates again for the lifetime of the process. UPDATE: after fixing sched_add_worker's EEXIST discard, live re-testing showed 'info worker' STILL displayed 0.00 for every resource despite value-level TRACE evidence confirming the real number (e.g. slots=22, maxmem=33182089216) survived the FULL round trip intact -- controller push -> scheduler apply -> snapshot save (20/20 captured) -> snapshot load/decode (20/20 restored, correct fvalue) -- so the remaining gap had to be in the DISPLAY layer itself, unconnected to the push/save/load pipeline. Found: repl_info_worker_one()/repl_info_resource_one() (src/scheduler/src/repl_info.c) checked avail_v->which == XCE_VAL_NUMERIC to decide whether an xce_val_t held a real number. XCE_VAL_NUMERIC (xce/types.h, value 1) is a RESOURCE-DEFINITION value-type constant (xce_resource_t.valtype's own enum) -- a completely different enum from xce_val_t.which's real domain, Value_u_which (controller.capnp.h, Value_u_fvalue=5). Every producer in this codebase sets xce_val_t.which to Value_u_fvalue, never XCE_VAL_NUMERIC, so the comparison was ALWAYS false for a genuinely-numeric resval -- 'info worker'/'info resource' printed total=0.00/available=0.00 unconditionally, regardless of whether the underlying data was correct (which it was, once bug-1567's picker fix and the sched_add_worker EEXIST fix were both in place). This display bug is what made the user's originally-reported symptom persist even after both upstream fixes.

Fix

FIXED (TDD, RED->GREEN), TWO independent root causes, both confirmed and fixed: (1) src/scheduler/src/core_events.c: XCE_CORE_EVENT_WORKERS_PUSH now goes through a new core_upsert_worker() (mirrors the existing core_upsert_queue() add-or-update precedent) -- looks up the worker by name first (sched_get_worker()), updates status+resvals in place if found (via a new shared core_apply_worker_resvals() helper, safe without a wipe step since zarray_put() already frees any existing value at a given index), else builds+adds a new one via the original core_build_worker()/sched_add_worker() path. New regression test in test_core_events.c: two WORKERS_PUSH events for the SAME worker name with DIFFERENT resource values must both apply -- resvals end up at the SECOND push's values, not stuck at the first, no duplicate/orphaned worker object. (2) src/scheduler/src/repl_info.c: all 4 ->which == XCE_VAL_NUMERIC checks (2 in repl_info_worker_one(), 2 in repl_info_resource_one()) corrected to ->which == Value_u_fvalue (added #include "controller.capnp.h"). New regression test in test_repl_breakpoints_inspection.c: info worker on a worker with a real numeric resval (42.5) must display it, not fall back to 0.00. Both RED confirmed before the fix (unit tests), then GREEN after. Live-validated end-to-end against a real deployed cluster (controller + xcecd_virtual worker + scheduler), reproducing the user's EXACT original command sequence (deploy, record, xce_scheduler -i -s <replay>, info worker pc13v) -- now shows the real values matching the user's own report exactly: slots=22.00, maxmem=33182089216.00 (30.9GB), mem=17151664128.00 (16GB), etc. Full scheduler suite green (41/41 excluding pre-existing bug-1560); controller suite green (1102/1104, the same pre-existing bug-1561 pair, confirmed unrelated).

Verification

Not recorded (migrated from buglog.json).

References

• Related: bug-1567