fix(scheduler,controller): worker resource values always display as 0.00 (bug-1568)

xce_scheduler -i -s <replay>: "info worker <name>" showed
total=0.00 available=0.00 for every resource, even though the live
cluster (xce_test resval/workerlist) reported real non-zero values
for the same worker.

Two independent root causes, both live-traced end-to-end against a
real deployed cluster (value-level TRACE at every hop: controller
push, scheduler apply, snapshot save, snapshot load/decode):

1. src/scheduler/src/core_events.c: XCE_CORE_EVENT_WORKERS_PUSH
   called sched_add_worker() -- a plain key_add(), add-only --
   unconditionally on every push. A worker connecting triggers
   multiple WORKERS_PUSH events in quick succession (an early push
   before the controller finishes writing real capacity values,
   carrying stale/zero data, followed moments later by a push
   carrying the real values). The FIRST push always won the race;
   every push after it for the same worker name silently failed
   -EEXIST and its resource values were discarded -- not just at
   join, for the rest of the process's lifetime (unlike jobs, there
   was no "update an existing worker" event path at all). Fixed via
   a new core_upsert_worker() (mirrors the existing
   core_upsert_queue() add-or-update precedent): looks up the
   worker by name first, updates status+resvals in place if found,
   else builds+adds a new one as before.

2. src/scheduler/src/repl_info.c: repl_info_worker_one()/
   repl_info_resource_one() checked `xce_val_t.which ==
   XCE_VAL_NUMERIC` -- but XCE_VAL_NUMERIC (xce/types.h, value 1)
   is a resource-DEFINITION value-type constant
   (xce_resource_t.valtype's own enum), a completely different enum
   from xce_val_t.which's real domain, Value_u_which
   (controller.capnp.h, Value_u_fvalue=5). Every producer in this
   codebase sets xce_val_t.which to Value_u_fvalue, never
   XCE_VAL_NUMERIC, so the comparison was always false -- every
   genuinely-numeric resval displayed as 0.00 unconditionally,
   independent of fix (1). Corrected all 4 occurrences to
   Value_u_fvalue.

Both fixed via TDD (RED test before each fix, GREEN after):
test_core_events.c's test_workers_push_twice_updates_existing_
worker_in_place (two pushes, same name, different values -- both
must apply) and test_repl_breakpoints_inspection.c's
test_info_worker_reports_resource_values (a real numeric resval
must display, not fall back to 0.00).

Live-validated by reproducing the user's exact original scenario
end-to-end (deploy a real cluster, record, `xce_scheduler -i -s
<replay>`, `info worker <name>`) -- now shows slots=22.00,
maxmem=33182089216.00 (30.9GB), mem=17151664128.00 (16GB), matching
the user's own live xce_test resval/workerlist numbers exactly.

Full scheduler suite green (41/41, excluding pre-existing unrelated
bug-1560); controller suite green (1102/1104, the same pre-existing
bug-1561 pair, confirmed unrelated to this change).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

src/controller/dataex.c

@@ -73,6 +73,21 @@ int collect_resval(var resmgr, zarray_t* array, xlist_t* l)
             memcpy(nv->val, src, sizeof(xce_val_t));
         }
 
+        /* bug-1568 instrumentation: the actual value read out of `array`
+         * (w->host.resvals for a per-worker call, c->sharedinst for a
+         * shared-resource call) right here -- the true "about to be
+         * wire-packaged" point shared by BOTH push paths
+         * (xcesched_push_workers -> collect_resvals_workers -> this
+         * function, and sched_worker_change -> this function directly).
+         * If this shows a real non-zero fvalue but the scheduler's own
+         * core_build_worker TRACE (core_events.c) shows 0 for the same
+         * resource, the loss is in the wire encode/decode or the
+         * scheduler-side resolution, not here or upstream. */
+        if (src->which == Value_u_fvalue) {
+            TRACE("%s resource '%s' idx %d -> fvalue=%f", __func__, r->name,
+                  r->idx, src->u.fvalue);
+        }
+
         xlist_store(l, nv);
     }
 

src/controller/sched.c

@@ -264,6 +264,25 @@ static int xcesched_push_workers(cluster_t* c)
     workers.workers = (xce_worker_t**)wlist.data;
     workers.count   = workers.n_total;
 
+    /* bug-1568 instrumentation: this is the FULL initial worker sync sent
+     * once, at on_scheduler_join() -- distinct from sched_worker_change()'s
+     * per-event incremental push (this file, above), which only fires
+     * AFTER a scheduler is already connected. A scheduler that connects
+     * before this controller's own resource_sync has completed its first
+     * poll cycle would see 0 resvals here even though this push path
+     * itself is working correctly. */
+    if (workers.n_total > 0) {
+        int total_resvals = 0;
+
+        for (i = 0; i < workers.n_total; i++) {
+            xce_worker_t* w = workers.workers[i];
+
+            total_resvals += (w != NULL) ? w->nresvals_to_sched : 0;
+        }
+        TRACE("%s initial sync: %d worker(s), %d total resval(s)", __func__,
+              workers.n_total, total_resvals);
+    }
+
     rc = xce_push_data_req(c->sched.ep, c->myhost, &data, check_controller, c,
                            100);
 
@@ -507,6 +526,14 @@ static int sched_worker_change(int type, int situation, void* data,
     if (c->sched.ep != NULL) {
         switch (situation) {
         case HOOK_AFTER_EVENT:
+            /* bug-1568 instrumentation: the controller-side count actually
+             * being pushed to the scheduler right now -- compare against
+             * core_build_worker()'s "WORKERS_PUSH carries N resval(s)"
+             * TRACE (scheduler/src/core_events.c) in the scheduler's own
+             * log for the same worker/timestamp to confirm the wire push
+             * itself is not where a resource value gets lost. */
+            TRACE("%s worker %s pushing %d resval(s) to scheduler", __func__,
+                  w->host.name, resvalist.count);
             xlist_init(&wlist, 2);
             xlist_store(&wlist, w);
             rc = xcesched_push_workerlist(c, &wlist);

src/controller/worker.c

@@ -97,6 +97,25 @@ static int update_val_res(cluster_t* c, worker_t* w, xce_resource_t* r,
 
     memcpy(v, val, sizeof(*v));
 
+    /* bug-1568 instrumentation: the actual VALUE being written into
+     * w->host.resvals[idx] (or c->sharedinst[idx] for a shared resource)
+     * -- earlier instrumentation (core_events.c/snapshot_wire.c/
+     * snapshot_load.c/sched.c) only ever counted entries flowing through
+     * the pipeline, never inspected their CONTENT; this is the true
+     * source write, so a 0.0 logged here for e.g. 'slots'/'maxmem' means
+     * the zero is genuine at the point of writing, not lost downstream. */
+    if (val->which == Value_u_fvalue) {
+        TRACE("%s resource '%s' worker '%s' idx %d -> fvalue=%f", __func__,
+              r->name, (r->flags & XCE_RESF_SHARED) ? "(shared)"
+                                                     : w->host.name,
+              idx, val->u.fvalue);
+    } else {
+        TRACE("%s resource '%s' worker '%s' idx %d -> which=%d (non-numeric)",
+              __func__, r->name,
+              (r->flags & XCE_RESF_SHARED) ? "(shared)" : w->host.name, idx,
+              val->which);
+    }
+
     return 0;
 }