admin-plane breakpoint 2: YAML resourcemap wiring + queue payload deep-copy
Breakpoint 2 of the admin-plane gaps (YAML flag/resourcemap wiring), plus
the queue payload ownership fix that the t1 controller segfault of
2026-10-23 12:41:13 traced to.
1. YAML resourcemap wiring (admin-plane-gaps 1.10):
- schema: Resource.distributions @8 / hostBased @9 uncommented --
resourcemap had no wire destination at all before this.
- resobj.c: resobj_init's #if 0 stub replaced with a real deep copy
(distributions + hbased_defined); resobj_destroy frees them. Deep
copy is mandatory: the source is myconf->cluster_config, which
keeps owning and later frees its tree.
- controller: reconcile_config_resources + seed_sharedinst_from_config
(cluster.c) seed c->sharedinst for SHARED|GLOBAL resources whose
distribution carries a value (the license-pool shape), from both
config_activate_committed paths, ahead of the conf4sched push.
- xceadmin: mapping.py maps resourcemap -> distributions (only the
location:[all]+value shape sets SHARED|GLOBAL), build.py renders
shared=true/distributions; golden expected.bin regenerated.
2. xcesched_push_configs resval double free (t1 12:39:52 abort): the
seeded push freed the resval entries AND the backing array via
free_name_val_list, then xlist_fini freed the same array again.
Detach xreslist before the FREEUP/xlist_fini block (same ownership
class bug-1506 fixed in sched_worker_change). Latent since inception:
every pre-1.10 push ran with nresvals==0 and never armed it.
3. queue payload deep-copy (t1 12:41:13 encode_Threshold_list segfault):
queue_init (create) MOVED the definition's users/hosts/thresholds/
window payloads out, and queue_refresh_def MOVED them out again --
but reconcile_config_queues runs MORE THAN ONCE over the same
cluster_config object (boot: pre-election activation AND phase2's
build_queues), so the second pass refreshed live queues from the
drained definition: thresholds became the poisoned (count>0,
array=NULL) pair, the generated encode_Threshold_list has no NULL
guard, and the next queue/worker push died on the NULL base.
Both paths now deep-copy (queue_init and queue_refresh_def); the
definition keeps its payloads, (NULL/0) genuinely means "no list in
this config", and reconcile is idempotent. New dup_group_config
helper in utils.c (the generated code has no GroupConfig dup).
Mirrors resobj_init's distributions deep-copy and worker.c
apply_join_state's dup pattern.
4. UT: test_reconcile_thresholds_survive_second_pass drives a triple
reconcile of a threshold-carrying queue and asserts definition
intactness plus a real encode_QueueConf -> wire -> decode round
trip; new test_config_resource_reconcile.c covers seed/deep-copy/
non-seed classes and idempotent re-seed.
Suites: controller_unit_test 1348/1348, sched_push 4, sched_worker_change 3,
apiserver_push 2, host_runusage 11, xcelib 759, xcealloc_oom 11,
99 scheduler binaries, all worker + harness suites, xceadmin pytest 375.
t1 E2E: boot double-reconcile clean, scheduler rejoin + full conf4sched
push applied, recovered crash-era PEND job reached DONE, 30-job saturation
flow (24 RUN + 6 PEND -> DONE), live gen-12 config activation with
re-push to the connected scheduler -- no crash, all four daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
parent
563f8e6890
commit
fc33a5da16
22 changed files with 1510 additions and 63 deletions
|
|
@ -1552,6 +1552,140 @@ int cluster_build_resources(cluster_t* c)
|
|||
return 0;
|
||||
}
|
||||
|
||||
static void resval_obj_killer(void* arg);
|
||||
|
||||
/* admin-plane-gaps 1.10 (YAML resourcemap wiring): seed c->sharedinst[idx]
|
||||
* for every config-declared STATIC SHARED|GLOBAL resource whose
|
||||
* distributions carry a value (the YAML `location: [all]` + `value:` shape
|
||||
* -- the license-pool scenario). The scheduler's two-phase shared-pool
|
||||
* machinery (core_apply_config_resvals -> core->sharedInst, gated on
|
||||
* XCE_RESF_GLOBAL in calc_wanted_global_resvals) consumes exactly these
|
||||
* resvals from every conf4sched push, so seeding them here makes a
|
||||
* YAML-declared shared pool schedulable WITHOUT any worker reporting it --
|
||||
* jobs asking rusage[res=1] are admitted up to the declared capacity, and
|
||||
* get PREASON_NO_RESOURCE once it is exhausted.
|
||||
*
|
||||
* Pattern mirrors update_val_res (worker.c): the array owns the xce_val_t
|
||||
* (calloc'd here, string values privately strdup'd), and zarray_put's
|
||||
* killer (resval_obj_killer) frees a replaced slot -- so re-running after
|
||||
* a config activation that changed the value simply swaps in the new one.
|
||||
* The distribution value is DEEP-COPIED out of myconf->cluster_config,
|
||||
* which keeps owning its decoded tree.
|
||||
*
|
||||
* Per-host-value distributions (the other resourcemap shapes) are
|
||||
* intentionally NOT seeded: host-subset static pooling needs the planned
|
||||
* elastic-shared-resources scheduler work (admin-plane-gaps 1.10); today
|
||||
* they stay definition-only, exactly as before this change. */
|
||||
static void seed_sharedinst_from_config(cluster_t* c)
|
||||
{
|
||||
int i;
|
||||
|
||||
if ((c == NULL) || (c->myconf == NULL)
|
||||
|| (c->myconf->cluster_config == NULL)) {
|
||||
return;
|
||||
}
|
||||
|
||||
for (i = 0; i < c->myconf->cluster_config->nres; i++) {
|
||||
xce_resource_t* r = c->myconf->cluster_config->resources[i];
|
||||
const res_distribution_t* d;
|
||||
xce_resource_t* live;
|
||||
xce_val_t* v;
|
||||
|
||||
if ((r == NULL) || (r->name == NULL)) {
|
||||
continue;
|
||||
}
|
||||
if ((r->flags & XCE_RESF_SHARED) == 0
|
||||
|| (r->flags & XCE_RESF_GLOBAL) == 0) {
|
||||
continue;
|
||||
}
|
||||
if (r->ndistributions < 1 || r->distributions == NULL
|
||||
|| r->distributions[0] == NULL || r->distributions[0]->value == NULL) {
|
||||
continue;
|
||||
}
|
||||
d = r->distributions[0];
|
||||
|
||||
/* Seed at the RUNTIME idx: resmgr idx values are key_add-assigned
|
||||
* (makeResource), not config-declared -- the decoded config's own
|
||||
* idx field is unset (0) for YAML-sourced definitions. */
|
||||
lock(c->resmgr, RTTI_LOCK_READ);
|
||||
key_find(c->resmgr, (void*)r->name, (void**)&live);
|
||||
unlock(c->resmgr);
|
||||
if (live == NULL) {
|
||||
WARN("seed_sharedinst: resource %s not found in resmgr -- "
|
||||
"shared pool stays unseeded until the next config activation",
|
||||
r->name);
|
||||
continue;
|
||||
}
|
||||
|
||||
v = (xce_val_t*)calloc(1, sizeof(*v));
|
||||
if (v == NULL) {
|
||||
WARN("seed_sharedinst: resource %s: out of memory -- shared "
|
||||
"pool stays unseeded until the next config activation",
|
||||
r->name);
|
||||
continue;
|
||||
}
|
||||
if (d->value->which == Value_u_svalue) {
|
||||
v->which = Value_u_svalue;
|
||||
v->u.svalue = strdup(d->value->u.svalue != NULL
|
||||
? d->value->u.svalue : "");
|
||||
if (v->u.svalue == NULL) {
|
||||
FREEUP(v);
|
||||
WARN("seed_sharedinst: resource %s: out of memory -- shared "
|
||||
"pool stays unseeded until the next config activation",
|
||||
r->name);
|
||||
continue;
|
||||
}
|
||||
} else {
|
||||
memcpy(v, d->value, sizeof(*v));
|
||||
}
|
||||
|
||||
if (zarray_put(&(c->sharedinst), live->idx, v) < 0) {
|
||||
resval_obj_killer(v);
|
||||
WARN("seed_sharedinst: resource %s: zarray_put idx %d failed -- "
|
||||
"shared pool stays unseeded until the next config activation",
|
||||
r->name, live->idx);
|
||||
continue;
|
||||
}
|
||||
INFO("seeded shared resource %s (idx %d) from configuration",
|
||||
r->name, live->idx);
|
||||
}
|
||||
}
|
||||
|
||||
/* admin-plane-gaps 1.10: materialize the active config's resource
|
||||
* definitions into the runtime resmgr and seed static shared pools from
|
||||
* them. Twin of reconcile_config_queues (bug-1469): makeResource tolerates
|
||||
* EEXIST on re-run (definitions are registered at boot; a re-run after a
|
||||
* config activation keeps the existing object -- resourceApplyDef's
|
||||
* change-detection contract applies to the push path), and the sharedinst
|
||||
* seeding is a value swap via zarray_put, so the whole pass is idempotent.
|
||||
* Call at boot (xce_config_init) and on both config_activate_committed
|
||||
* activation paths, BEFORE xcesched_push_configs so the scheduler's very
|
||||
* next conf4sched push carries the seeded resvals. */
|
||||
int reconcile_config_resources(cluster_t* c)
|
||||
{
|
||||
int rc;
|
||||
|
||||
if ((c == NULL) || (c->resmgr == NULL) || (c->myconf == NULL)) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
rc = cluster_build_resources(c);
|
||||
if (rc < 0 && rc != -EEXIST) {
|
||||
/* EEXIST = a re-run (config activation after boot / a previous
|
||||
* activation): definitions are already registered, keep them --
|
||||
* the seeding below still applies. Any other error (ENOMEM) is
|
||||
* reported so the caller logs it, but the seeding still runs for
|
||||
* the definitions that DID register. */
|
||||
WARN("%s: cluster_build_resources rc=%d -- keeping previous "
|
||||
"definitions", __func__, rc);
|
||||
return rc;
|
||||
}
|
||||
|
||||
seed_sharedinst_from_config(c);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int cluster_load_jobs(cluster_t* c)
|
||||
{
|
||||
load_jobs(c);
|
||||
|
|
@ -1590,7 +1724,7 @@ static int xce_config_init(int type, int situation, void* data, va_list ap)
|
|||
if (rc < 0) {
|
||||
return rc;
|
||||
}
|
||||
rc = cluster_build_resources(c);
|
||||
rc = reconcile_config_resources(c);
|
||||
if (rc < 0) {
|
||||
return rc;
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue