xce_scheduler -i -s <replay>: "info worker <name>" showed
total=0.00 available=0.00 for every resource, even though the live
cluster (xce_test resval/workerlist) reported real non-zero values
for the same worker.
Two independent root causes, both live-traced end-to-end against a
real deployed cluster (value-level TRACE at every hop: controller
push, scheduler apply, snapshot save, snapshot load/decode):
1. src/scheduler/src/core_events.c: XCE_CORE_EVENT_WORKERS_PUSH
called sched_add_worker() -- a plain key_add(), add-only --
unconditionally on every push. A worker connecting triggers
multiple WORKERS_PUSH events in quick succession (an early push
before the controller finishes writing real capacity values,
carrying stale/zero data, followed moments later by a push
carrying the real values). The FIRST push always won the race;
every push after it for the same worker name silently failed
-EEXIST and its resource values were discarded -- not just at
join, for the rest of the process's lifetime (unlike jobs, there
was no "update an existing worker" event path at all). Fixed via
a new core_upsert_worker() (mirrors the existing
core_upsert_queue() add-or-update precedent): looks up the
worker by name first, updates status+resvals in place if found,
else builds+adds a new one as before.
2. src/scheduler/src/repl_info.c: repl_info_worker_one()/
repl_info_resource_one() checked `xce_val_t.which ==
XCE_VAL_NUMERIC` -- but XCE_VAL_NUMERIC (xce/types.h, value 1)
is a resource-DEFINITION value-type constant
(xce_resource_t.valtype's own enum), a completely different enum
from xce_val_t.which's real domain, Value_u_which
(controller.capnp.h, Value_u_fvalue=5). Every producer in this
codebase sets xce_val_t.which to Value_u_fvalue, never
XCE_VAL_NUMERIC, so the comparison was always false -- every
genuinely-numeric resval displayed as 0.00 unconditionally,
independent of fix (1). Corrected all 4 occurrences to
Value_u_fvalue.
Both fixed via TDD (RED test before each fix, GREEN after):
test_core_events.c's test_workers_push_twice_updates_existing_
worker_in_place (two pushes, same name, different values -- both
must apply) and test_repl_breakpoints_inspection.c's
test_info_worker_reports_resource_values (a real numeric resval
must display, not fall back to 0.00).
Live-validated by reproducing the user's exact original scenario
end-to-end (deploy a real cluster, record, `xce_scheduler -i -s
<replay>`, `info worker <name>`) -- now shows slots=22.00,
maxmem=33182089216.00 (30.9GB), mem=17151664128.00 (16GB), matching
the user's own live xce_test resval/workerlist numbers exactly.
Full scheduler suite green (41/41, excluding pre-existing unrelated
bug-1560); controller suite green (1102/1104, the same pre-existing
bug-1561 pair, confirmed unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Defines xce_core_event_type_t/xce_core_event_t (9 event kinds: config,
job, worker, resource, resval full/delta, job state change, job update,
queue update, job rusage) in core.h. Implements xce_core_push_events()
in a new core_events.c (split out of core.c to stay under the 800-line
cap): rejects -EBUSY mid-session, otherwise applies each event's core-
state mutation and fires the corresponding picker hook synchronously,
exactly once per event, before returning. CONFIG_PUSH decomposes the
pushed config's embedded queues/resources/resvals via shared helpers
(also reused by QUEUE_UPDATE's upsert path), mirroring the existing
sched_init_config()/sched_init_resources()/sched_init_sharedInst()/
sched_init_queues(). Narrows sched_resolve_job_picker() to take
xce_sched_core_t* (pure type change, zero behavior change). Purely
additive — not wired into the live scheduling path yet (task 5.x).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
