admin-plane breakpoint 2: YAML resourcemap wiring + queue payload deep-copy

Breakpoint 2 of the admin-plane gaps (YAML flag/resourcemap wiring), plus
the queue payload ownership fix that the t1 controller segfault of
2026-10-23 12:41:13 traced to.

1. YAML resourcemap wiring (admin-plane-gaps 1.10):
   - schema: Resource.distributions @8 / hostBased @9 uncommented --
     resourcemap had no wire destination at all before this.
   - resobj.c: resobj_init's #if 0 stub replaced with a real deep copy
     (distributions + hbased_defined); resobj_destroy frees them. Deep
     copy is mandatory: the source is myconf->cluster_config, which
     keeps owning and later frees its tree.
   - controller: reconcile_config_resources + seed_sharedinst_from_config
     (cluster.c) seed c->sharedinst for SHARED|GLOBAL resources whose
     distribution carries a value (the license-pool shape), from both
     config_activate_committed paths, ahead of the conf4sched push.
   - xceadmin: mapping.py maps resourcemap -> distributions (only the
     location:[all]+value shape sets SHARED|GLOBAL), build.py renders
     shared=true/distributions; golden expected.bin regenerated.

2. xcesched_push_configs resval double free (t1 12:39:52 abort): the
   seeded push freed the resval entries AND the backing array via
   free_name_val_list, then xlist_fini freed the same array again.
   Detach xreslist before the FREEUP/xlist_fini block (same ownership
   class bug-1506 fixed in sched_worker_change). Latent since inception:
   every pre-1.10 push ran with nresvals==0 and never armed it.

3. queue payload deep-copy (t1 12:41:13 encode_Threshold_list segfault):
   queue_init (create) MOVED the definition's users/hosts/thresholds/
   window payloads out, and queue_refresh_def MOVED them out again --
   but reconcile_config_queues runs MORE THAN ONCE over the same
   cluster_config object (boot: pre-election activation AND phase2's
   build_queues), so the second pass refreshed live queues from the
   drained definition: thresholds became the poisoned (count>0,
   array=NULL) pair, the generated encode_Threshold_list has no NULL
   guard, and the next queue/worker push died on the NULL base.
   Both paths now deep-copy (queue_init and queue_refresh_def); the
   definition keeps its payloads, (NULL/0) genuinely means "no list in
   this config", and reconcile is idempotent. New dup_group_config
   helper in utils.c (the generated code has no GroupConfig dup).
   Mirrors resobj_init's distributions deep-copy and worker.c
   apply_join_state's dup pattern.

4. UT: test_reconcile_thresholds_survive_second_pass drives a triple
   reconcile of a threshold-carrying queue and asserts definition
   intactness plus a real encode_QueueConf -> wire -> decode round
   trip; new test_config_resource_reconcile.c covers seed/deep-copy/
   non-seed classes and idempotent re-seed.

Suites: controller_unit_test 1348/1348, sched_push 4, sched_worker_change 3,
apiserver_push 2, host_runusage 11, xcelib 759, xcealloc_oom 11,
99 scheduler binaries, all worker + harness suites, xceadmin pytest 375.
t1 E2E: boot double-reconcile clean, scheduler rejoin + full conf4sched
push applied, recovered crash-era PEND job reached DONE, 30-job saturation
flow (24 RUN + 6 PEND -> DONE), live gen-12 config activation with
re-push to the connected scheduler -- no crash, all four daemons healthy.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-23 14:29:29 +00:00
parent 563f8e6890
commit fc33a5da16
22 changed files with 1510 additions and 63 deletions

View file

@ -1552,6 +1552,140 @@ int cluster_build_resources(cluster_t* c)
return 0;
}
static void resval_obj_killer(void* arg);
/* admin-plane-gaps 1.10 (YAML resourcemap wiring): seed c->sharedinst[idx]
* for every config-declared STATIC SHARED|GLOBAL resource whose
* distributions carry a value (the YAML `location: [all]` + `value:` shape
* -- the license-pool scenario). The scheduler's two-phase shared-pool
* machinery (core_apply_config_resvals -> core->sharedInst, gated on
* XCE_RESF_GLOBAL in calc_wanted_global_resvals) consumes exactly these
* resvals from every conf4sched push, so seeding them here makes a
* YAML-declared shared pool schedulable WITHOUT any worker reporting it --
* jobs asking rusage[res=1] are admitted up to the declared capacity, and
* get PREASON_NO_RESOURCE once it is exhausted.
*
* Pattern mirrors update_val_res (worker.c): the array owns the xce_val_t
* (calloc'd here, string values privately strdup'd), and zarray_put's
* killer (resval_obj_killer) frees a replaced slot -- so re-running after
* a config activation that changed the value simply swaps in the new one.
* The distribution value is DEEP-COPIED out of myconf->cluster_config,
* which keeps owning its decoded tree.
*
* Per-host-value distributions (the other resourcemap shapes) are
* intentionally NOT seeded: host-subset static pooling needs the planned
* elastic-shared-resources scheduler work (admin-plane-gaps 1.10); today
* they stay definition-only, exactly as before this change. */
static void seed_sharedinst_from_config(cluster_t* c)
{
int i;
if ((c == NULL) || (c->myconf == NULL)
|| (c->myconf->cluster_config == NULL)) {
return;
}
for (i = 0; i < c->myconf->cluster_config->nres; i++) {
xce_resource_t* r = c->myconf->cluster_config->resources[i];
const res_distribution_t* d;
xce_resource_t* live;
xce_val_t* v;
if ((r == NULL) || (r->name == NULL)) {
continue;
}
if ((r->flags & XCE_RESF_SHARED) == 0
|| (r->flags & XCE_RESF_GLOBAL) == 0) {
continue;
}
if (r->ndistributions < 1 || r->distributions == NULL
|| r->distributions[0] == NULL || r->distributions[0]->value == NULL) {
continue;
}
d = r->distributions[0];
/* Seed at the RUNTIME idx: resmgr idx values are key_add-assigned
* (makeResource), not config-declared -- the decoded config's own
* idx field is unset (0) for YAML-sourced definitions. */
lock(c->resmgr, RTTI_LOCK_READ);
key_find(c->resmgr, (void*)r->name, (void**)&live);
unlock(c->resmgr);
if (live == NULL) {
WARN("seed_sharedinst: resource %s not found in resmgr -- "
"shared pool stays unseeded until the next config activation",
r->name);
continue;
}
v = (xce_val_t*)calloc(1, sizeof(*v));
if (v == NULL) {
WARN("seed_sharedinst: resource %s: out of memory -- shared "
"pool stays unseeded until the next config activation",
r->name);
continue;
}
if (d->value->which == Value_u_svalue) {
v->which = Value_u_svalue;
v->u.svalue = strdup(d->value->u.svalue != NULL
? d->value->u.svalue : "");
if (v->u.svalue == NULL) {
FREEUP(v);
WARN("seed_sharedinst: resource %s: out of memory -- shared "
"pool stays unseeded until the next config activation",
r->name);
continue;
}
} else {
memcpy(v, d->value, sizeof(*v));
}
if (zarray_put(&(c->sharedinst), live->idx, v) < 0) {
resval_obj_killer(v);
WARN("seed_sharedinst: resource %s: zarray_put idx %d failed -- "
"shared pool stays unseeded until the next config activation",
r->name, live->idx);
continue;
}
INFO("seeded shared resource %s (idx %d) from configuration",
r->name, live->idx);
}
}
/* admin-plane-gaps 1.10: materialize the active config's resource
* definitions into the runtime resmgr and seed static shared pools from
* them. Twin of reconcile_config_queues (bug-1469): makeResource tolerates
* EEXIST on re-run (definitions are registered at boot; a re-run after a
* config activation keeps the existing object -- resourceApplyDef's
* change-detection contract applies to the push path), and the sharedinst
* seeding is a value swap via zarray_put, so the whole pass is idempotent.
* Call at boot (xce_config_init) and on both config_activate_committed
* activation paths, BEFORE xcesched_push_configs so the scheduler's very
* next conf4sched push carries the seeded resvals. */
int reconcile_config_resources(cluster_t* c)
{
int rc;
if ((c == NULL) || (c->resmgr == NULL) || (c->myconf == NULL)) {
return 0;
}
rc = cluster_build_resources(c);
if (rc < 0 && rc != -EEXIST) {
/* EEXIST = a re-run (config activation after boot / a previous
* activation): definitions are already registered, keep them --
* the seeding below still applies. Any other error (ENOMEM) is
* reported so the caller logs it, but the seeding still runs for
* the definitions that DID register. */
WARN("%s: cluster_build_resources rc=%d -- keeping previous "
"definitions", __func__, rc);
return rc;
}
seed_sharedinst_from_config(c);
return 0;
}
int cluster_load_jobs(cluster_t* c)
{
load_jobs(c);
@ -1590,7 +1724,7 @@ static int xce_config_init(int type, int situation, void* data, va_list ap)
if (rc < 0) {
return rc;
}
rc = cluster_build_resources(c);
rc = reconcile_config_resources(c);
if (rc < 0) {
return rc;
}