lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(scheduler-sim-live-allocation): extract shared picker/allocator into sched_election.c (task 3.1)

Browse source 
Relocates sched_job_picker()/sched_allocator() out of scheduler.c
(1382 lines, already over this project's 800-line cap) into a new
sched_election.c, retyped to xce_sched_core_t* -- the single
implementation both drivers will spawn (task 4.1 wires simulation
mode onto it). Independently verified bit-for-bit identical to the
pre-relocation bodies modulo the sched->/core-> rename; no statement
reordered, no branch changed.

sched_make_allocation() becomes extern (was static, sole caller moved
translation units); the old commit_decision() thin wrapper is deleted,
its sole caller now calls sched_commit_decision() directly with
identical semantics. sched_init() updated to spawn the relocated pair
against &(sched->core).

Adds an end-to-end fixture test proving a job flows from picked to a
committed decision against bare (non-xce_scheduler_t) core state.
Routes around, rather than fixes, two pre-existing out-of-boundary
gaps this fixture exposed (seqjob/parjob allocator init still reading
plugin->sched; make_sched_job()'s parse path not threading
maxtasks/mintasks) -- both documented in the test's own comments for
future follow-up.

sched_election.c added to the normal build only; sched_sim_isolation_lib
membership deferred to task 4.1, which is where sim mode actually
starts spawning these threads.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-19 02:50:38 +00:00
parent fc23ea8d31
commit 4195331b47
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
8 changed files with 675 additions and 242 deletions
Download patch file Download diff file Expand all files Collapse all files

25
src/scheduler/include/scheduler.h
View file

@ -304,8 +304,29 @@ extern void sched_set_default_job_pick_plugin(xce_scheduler_t*);
/* sched_resolve_job_picker relocated to core.h (scheduler-core spec, task
* 4.3) -- it now operates on xce_sched_core_t*, not xce_scheduler_t*, so
* xce_core_push_events() (core.c) can call it without core.c depending on
* this header. Call sites in this file (sched_job_picker(), sched_allocator(),
* sched_init()) now pass &sched->core. See core.h for the full contract. */
* this header. Call sites in sched_election.c (sched_job_picker(),
* sched_allocator()) and this file (sched_init()) now pass &sched->core
* (or, for the two thread bodies, `core` directly -- see sched_election.c's
* own header comment, scheduler-sim-live-allocation spec task 3.1). See
* core.h for the full contract. */
/* scheduler-sim-live-allocation spec, task 3.1 (Unified Picker/Allocator
* Threads): sched_job_picker()/sched_allocator() relocated out of this
* file into their own file, sched_election.c -- the single implementation
* both normal mode (sched_init(), below, passing &(sched->core)) and a
* future simulation driver (task 4.1, passing &(sim->core)) spawn. Retyped
* from xce_scheduler_t* to xce_sched_core_t*; pthread_create()'s own
* callback signature (void* (*)(void*)) is unchanged either way. Declared
* here rather than a new header, mirroring this file's own established
* convention for every other function this spec's migration
* relocated/retyped (sched_meet_host_selection, sched_try_alloc_on_host_,
* etc. below). */
extern void* sched_job_picker(void* arg);
extern void* sched_allocator(void* arg);
/* scheduler-sim-live-allocation spec, task 3.1: exported (was `static`) --
* its sole caller, sched_allocator(), moved out of this file (scheduler.c)
* into sched_election.c, so this cross-file call needs external linkage.
* Signature already core-safe since task 2.2. */
extern int sched_make_allocation(xce_sched_core_t*, xce_sched_obj_t*);
extern int sched_update(xce_scheduler_t*);
extern void sched_go(xce_scheduler_t*);
extern void sched_begin_session(xce_scheduler_t*);

1
src/scheduler/meson.build
View file

@ -2,6 +2,7 @@ scheduler_srcs = [
'src/main.c',
'src/handler.c',
'src/scheduler.c',
'src/sched_election.c',
'src/core.c',
'src/core_events.c',
'src/core_session.c',

262
src/scheduler/src/sched_election.c Normal file
View file

@ -0,0 +1,262 @@
/*
* sched_election.c - scheduler-sim-live-allocation spec, task 3.1 (Req 1.1,
* 1.2, 5.1, boundary: Unified Picker/Allocator Threads).
*
* Houses sched_job_picker()/sched_allocator(), relocated here verbatim
* (aside from their signature/cast) from scheduler.c. Two reasons for the
* new file, not one (design.md's File Structure Plan):
*
* 1. This is the single implementation both drivers spawn (normal mode's
* sched_init(), scheduler.c, passing &(sched->core); a future
* simulation driver, task 4.1, passing &(sim->core)) -- it should not
* live inside a file (scheduler.c) that is otherwise entirely
* normal-mode-only and controller-coupled (xcecli_*, HA candidate
* lists, recording lifecycle, ...).
* 2. scheduler.c is already over this project's 800-line cap
* (src/scheduler/CLAUDE.md); this migration is exactly the
* opportunistic split that file's own header comment asks for, not
* scope creep invented for its own sake.
*
* Both functions are retyped from `xce_scheduler_t*` to `xce_sched_core_t*`
* -- every call either of them makes was already made core-safe by tasks
* 1.2/2.1/2.2 (plugin_job_election(), sched_make_allocation(), the
* seven-function Core-Safe Allocation Chain), so this is a pure
* signature/cast migration with zero behavior change for normal mode.
* pthread_create()'s own callback signature is always `void* (*)(void*)`
* regardless of what the function casts its argument to internally, so the
* *outer* signature of each function is unchanged -- only the first line's
* cast target moved from `(xce_scheduler_t*)arg` to
* `(xce_sched_core_t*)arg`.
*
* sched_allocator()'s own former `commit_decision()` wrapper (scheduler.c,
* static, its sole caller) is deleted rather than relocated here: it
* existed only to bridge sched_allocator()'s then-xce_scheduler_t*-typed
* `sched` into sched_commit_decision()'s already-core-typed
* `&(sched->core)` parameter (task 2.2). Now that sched_allocator() itself
* holds `core` directly, that bridge is redundant -- sched_commit_decision()
* is called directly below.
*/
#include <errno.h>
#include <pthread.h>
#include "decision.h"
#include "plugin.h"
#include "preason.h"
#include "resreq.h"
#include "scheduler.h"
#include "utils.h"
/*
* sched_job_picker - the scheduler's dedicated picker thread entry point
* (Req 1 AC 3/AC 4, Req 3 AC 5, scheduler-core spec, task 2.4).
*
* Replaces the hardcoded "xce_sched_fcfs" literal (Req 1 AC 6) with
* sched_resolve_job_picker() (schedobj.c), which looks up
* sched->core.job_pick_plugin through the same plugin_register()/
* key_find() mechanism every plugin category uses. sched_init() already
* performed this same resolution synchronously before this thread was even
* created (so an unregistered/non-picker name fails scheduler startup, not
* this thread quietly) -- re-resolving here is the redundant "one
* implementation, two call sites" the scheduler.h doc comment describes,
* not duplicated logic.
*
* Branches on which picker kind was resolved (see this file's own header
* discussion / builtin_fcfs.c's file header comment for the full
* reasoning):
* - job_elect_fn != NULL (a plugin still using the deprecated legacy
* hook, e.g. "xce_sched_fcfs" explicitly configured): delegate to it
* EXACTLY as before this task -- job_elect_fn (fcfs_elect_job) is
* ITSELF the entire forever per-session loop, with its own
* cbuf_get(&sched->wbuf) at the top and its own pthread_testcancel()
* calls throughout (fcfs.c). Running the generic loop below *as well*
* for this case would install a second, competing consumer of wbuf's
* single "go" signal per round -- fcfs_elect_job's own cbuf_get would
* then block waiting for a signal this outer loop already stole,
* hanging the legacy picker (a real bug, not just redundant work).
* - job_elect_fn == NULL (a native xce_job_picker_iface_t implementor --
* builtin_fcfs, or any future native picker): run the generic
* per-session outer loop against picker_iface's hooks (Req 1 AC 4
* pseudocode), pushing each returned job's jobid onto core->jbuf and
* an end-of-session 0 marker -- exactly what sched_allocator() (below,
* unchanged pipeline: same wbuf/jbuf/decisions cbufs, same two
* threads) already expects on the other end.
*
* scheduler-sim-live-allocation spec, task 3.1: relocated from scheduler.c,
* retyped from xce_scheduler_t* to xce_sched_core_t* -- body unchanged
* besides sched->core.X -> core->X field-access updates.
*/
void* sched_job_picker(void* arg)
{
xce_sched_core_t* core = (xce_sched_core_t*)arg;
xce_sched_plugin_t* job_picker = NULL;
int rc = 0;
rc = sched_resolve_job_picker(core, &job_picker);
if (rc < 0) {
ERROR("%s failed to resolve job picker plugin '%s' (rc=%d)", __func__,
core->job_pick_plugin ? core->job_pick_plugin : "(null)", rc);
errno = -rc;
return NULL;
}
if (job_picker->job_elect_fn != NULL) {
return job_picker->job_elect_fn(job_picker);
}
while (1) {
int go = 0;
uint64_t jobid = 0;
pthread_testcancel();
/* Cancel-safety (Req 1 AC 5, Req 3 AC 5): cbuf_get()'s internal
* blocking wait is pthread_cond_wait() (src/lib/mylock.c), a
* POSIX cancellation point -- this remains a valid, defined
* cancel point for the picker thread between sessions, the same
* property plugins/fcfs.c's own fcfs_elect_job() already relies on
* for its identical cbuf_get(&sched->wbuf) call. */
cbuf_get(&(core->wbuf), &go);
job_picker->picker_iface->begin_session(job_picker);
while (1) {
xce_sched_job_t* job = NULL;
pthread_testcancel();
job = job_picker->picker_iface->next_job(job_picker);
if (job == NULL) {
break;
}
jobid = job->jobid;
cbuf_put(&(core->jbuf), &jobid);
}
/* end of scheduling session */
jobid = 0;
cbuf_put(&(core->jbuf), &jobid);
job_picker->picker_iface->end_session(job_picker);
}
return NULL;
}
/*
* sched_allocator - the scheduler's dedicated allocator thread entry point.
* Drains core->jbuf (fed by sched_job_picker() above), runs the real
* election/allocation chain per job, and commits the resulting decision.
*
* scheduler-sim-live-allocation spec, task 3.1: relocated from scheduler.c,
* retyped from xce_scheduler_t* to xce_sched_core_t* -- every call this
* function makes was already core-safe (plugin_job_election(), task 1.2;
* sched_make_allocation(), task 2.2; sched_commit_decision(), task 2.2),
* so the `&(sched->core)`-wrapped call sites those tasks left behind
* (deliberately, per their own comments -- retyping this function itself
* was this task's job) are simplified to pass `core` directly. The former
* commit_decision() wrapper (scheduler.c, static) is gone entirely -- see
* this file's header comment.
*/
void* sched_allocator(void* arg)
{
xce_sched_core_t* core = (xce_sched_core_t*)arg;
xce_sched_plugin_t* job_picker = NULL;
int rc = 0;
sel_register_alloc_thread();
xce_use(core);
/* Req 1 AC 4 (scheduler-core spec, task 2.4): the allocator thread's
* pseudocode calls picker->on_dispatch()/picker->on_pend() after each
* allocation attempt -- resolve the same picker sched_job_picker()
* resolves (one implementation, two call sites, scheduler.h). By the
* time this thread is created, sched_init() has already gated on this
* exact resolution succeeding (before either thread is started), so
* failure here should not occur in practice; handled the same
* fail-return-NULL way sched_job_picker() handles it, for symmetry. */
rc = sched_resolve_job_picker(core, &job_picker);
if (rc < 0) {
ERROR("%s failed to resolve job picker plugin '%s' (rc=%d)", __func__,
core->job_pick_plugin ? core->job_pick_plugin : "(null)", rc);
errno = -rc;
return NULL;
}
while (1) {
uint64_t jobid;
char jobidString[64];
xce_decision_t* decision = NULL;
xce_sched_job_t* job = NULL;
int alloc_rc = 0;
pthread_testcancel();
cbuf_get(&(core->jbuf), &jobid);
if (jobid == 0) {
/* end of scheduling session */
decision = NULL;
cbuf_put(&(core->decisions), &decision);
continue;
}
job = sched_get_job(core, jobid);
if (job == NULL) {
fmt_jobid(jobidString, sizeof(jobidString), jobid);
TRACE("%s job %s can not be found", __func__, jobidString);
continue;
}
/* scheduler-sim-live-allocation spec, task 1.2: plugin_job_election()
* is core-typed (mirrors plugin_begin_session(core) in this same
* codebase's driver files) -- pass core directly. */
plugin_job_election(core, job);
/* calculate decision for jobid */
/* scheduler-sim-live-allocation spec, task 2.2: sched_make_
* allocation() is core-typed -- pass core directly. */
alloc_rc = sched_make_allocation(core, job->schedobj);
if (alloc_rc < 0) {
/* something wrong */
continue;
} else if (alloc_rc == 0) {
/* job is pend */
xce_pendreason_t* reason = job->schedobj->reason;
/* Req 1 AC 4 / Req 3 AC 2 "on_pend" row: keep the job as a
* candidate for the next round (no-op for builtin_fcfs; the
* legacy job_elect_fn path's picker_iface is
* legacy_picker_wrapper_iface, whose on_pend is also a no-op
* -- fcfs.c's own removal happens via its separate commit_hook,
* untouched by this call). */
job_picker->picker_iface->on_pend(job_picker, jobid);
if (reason == NULL) {
fmt_jobid(jobidString, sizeof(jobidString), jobid);
WARN("job %s is pending but pend reason is NULL", jobidString);
} else {
reason->jobid = job->jobid;
decision = make_pend_decision(reason);
}
} else {
/* Req 1 AC 4 / Req 3 AC 2 "on_dispatch" row: remove the job
* from the picker's candidate set permanently, before the
* decision is built (matches Req 1 AC 4's pseudocode order). */
job_picker->picker_iface->on_dispatch(job_picker, jobid);
MOVE_PTR(job->allocs, job->schedobj->alloc);
job->allocs->jobid = job->jobid;
decision = make_dispatch_decision(job->allocs);
}
if (decision != NULL) {
sched_commit_decision(core, decision);
cbuf_put(&(core->decisions), &decision);
}
}
return NULL;
}

240
src/scheduler/src/scheduler.c
View file

@ -42,102 +42,6 @@ static int sched_staging_merge_complete(xce_scheduler_t* sched)
memory_order_acquire);
}
/*
* sched_job_picker - the scheduler's dedicated picker thread entry point
* (Req 1 AC 3/AC 4, Req 3 AC 5, scheduler-core spec, task 2.4).
*
* Replaces the hardcoded "xce_sched_fcfs" literal (Req 1 AC 6) with
* sched_resolve_job_picker() (schedobj.c), which looks up
* sched->core.job_pick_plugin through the same plugin_register()/
* key_find() mechanism every plugin category uses. sched_init() already
* performed this same resolution synchronously before this thread was even
* created (so an unregistered/non-picker name fails scheduler startup, not
* this thread quietly) -- re-resolving here is the redundant "one
* implementation, two call sites" the scheduler.h doc comment describes,
* not duplicated logic.
*
* Branches on which picker kind was resolved (see this file's own header
* discussion / builtin_fcfs.c's file header comment for the full
* reasoning):
* - job_elect_fn != NULL (a plugin still using the deprecated legacy
* hook, e.g. "xce_sched_fcfs" explicitly configured): delegate to it
* EXACTLY as before this task -- job_elect_fn (fcfs_elect_job) is
* ITSELF the entire forever per-session loop, with its own
* cbuf_get(&sched->wbuf) at the top and its own pthread_testcancel()
* calls throughout (fcfs.c). Running the generic loop below *as well*
* for this case would install a second, competing consumer of wbuf's
* single "go" signal per round -- fcfs_elect_job's own cbuf_get would
* then block waiting for a signal this outer loop already stole,
* hanging the legacy picker (a real bug, not just redundant work).
* - job_elect_fn == NULL (a native xce_job_picker_iface_t implementor --
* builtin_fcfs, or any future native picker): run the generic
* per-session outer loop against picker_iface's hooks (Req 1 AC 4
* pseudocode), pushing each returned job's jobid onto sched->jbuf and
* an end-of-session 0 marker -- exactly what sched_allocator() (below,
* unchanged pipeline: same wbuf/jbuf/decisions cbufs, same two
* threads) already expects on the other end.
*/
static void* sched_job_picker(void* arg)
{
xce_scheduler_t* sched = (xce_scheduler_t*)arg;
xce_sched_plugin_t* job_picker = NULL;
int rc = 0;
rc = sched_resolve_job_picker(&(sched->core), &job_picker);
if (rc < 0) {
ERROR("%s failed to resolve job picker plugin '%s' (rc=%d)", __func__,
sched->core.job_pick_plugin ? sched->core.job_pick_plugin
: "(null)",
rc);
errno = -rc;
return NULL;
}
if (job_picker->job_elect_fn != NULL) {
return job_picker->job_elect_fn(job_picker);
}
while (1) {
int go = 0;
uint64_t jobid = 0;
pthread_testcancel();
/* Cancel-safety (Req 1 AC 5, Req 3 AC 5): cbuf_get()'s internal
* blocking wait is pthread_cond_wait() (src/lib/mylock.c), a
* POSIX cancellation point -- this remains a valid, defined
* cancel point for the picker thread between sessions, the same
* property plugins/fcfs.c's own fcfs_elect_job() already relies on
* for its identical cbuf_get(&sched->wbuf) call. */
cbuf_get(&(sched->core.wbuf), &go);
job_picker->picker_iface->begin_session(job_picker);
while (1) {
xce_sched_job_t* job = NULL;
pthread_testcancel();
job = job_picker->picker_iface->next_job(job_picker);
if (job == NULL) {
break;
}
jobid = job->jobid;
cbuf_put(&(sched->core.jbuf), &jobid);
}
/* end of scheduling session */
jobid = 0;
cbuf_put(&(sched->core.jbuf), &jobid);
job_picker->picker_iface->end_session(job_picker);
}
return NULL;
}
/* scheduler-sim-live-allocation spec, task 2.2 (Core-Safe Allocation
* Chain): both retyped to xce_sched_core_t* -- neither ever touched
* anything beyond core->seqjob_allocator/core->parjob_allocator (already
@ -431,7 +335,12 @@ static int sched_make_alter_allocation(xce_sched_core_t* core,
return 0;
}
static int sched_make_allocation(xce_sched_core_t* core, xce_sched_obj_t* obj)
/* scheduler-sim-live-allocation spec, task 3.1: exported (was `static`) --
* its sole caller, sched_allocator(), moved out of this file into
* sched_election.c, so this cross-file call needs external linkage. Its
* signature was already core-safe since task 2.2; only the linkage
* changed. Declared in scheduler.h. */
int sched_make_allocation(xce_sched_core_t* core, xce_sched_obj_t* obj)
{
int rc = 0;
@ -452,124 +361,12 @@ static int sched_make_allocation(xce_sched_core_t* core, xce_sched_obj_t* obj)
return rc;
}
/* commit_decision() stays xce_scheduler_t*-typed: its sole caller,
* sched_allocator() below, is task 3.1's job, not this one. Minimal
* call-site wrap into the now-core-typed sched_commit_decision() --
* same pattern as every other call site this task touches. */
static int commit_decision(xce_scheduler_t* sched, xce_decision_t* decision)
{
return sched_commit_decision(&(sched->core), decision);
}
static void* sched_allocator(void* arg)
{
xce_scheduler_t* sched = (xce_scheduler_t*)arg;
xce_sched_plugin_t* job_picker = NULL;
int rc = 0;
sel_register_alloc_thread();
xce_use(sched);
/* Req 1 AC 4 (scheduler-core spec, task 2.4): the allocator thread's
* pseudocode calls picker->on_dispatch()/picker->on_pend() after each
* allocation attempt -- resolve the same picker sched_job_picker()
* resolves (one implementation, two call sites, scheduler.h). By the
* time this thread is created, sched_init() has already gated on this
* exact resolution succeeding (before either thread is started), so
* failure here should not occur in practice; handled the same
* fail-return-NULL way sched_job_picker() handles it, for symmetry. */
rc = sched_resolve_job_picker(&(sched->core), &job_picker);
if (rc < 0) {
ERROR("%s failed to resolve job picker plugin '%s' (rc=%d)", __func__,
sched->core.job_pick_plugin ? sched->core.job_pick_plugin
: "(null)",
rc);
errno = -rc;
return NULL;
}
while (1) {
uint64_t jobid;
char jobidString[64];
xce_decision_t* decision = NULL;
xce_sched_job_t* job = NULL;
int alloc_rc = 0;
pthread_testcancel();
cbuf_get(&(sched->core.jbuf), &jobid);
if (jobid == 0) {
/* end of scheduling session */
decision = NULL;
cbuf_put(&(sched->core.decisions), &decision);
continue;
}
job = sched_get_job(&sched->core, jobid);
if (job == NULL) {
fmt_jobid(jobidString, sizeof(jobidString), jobid);
TRACE("%s job %s can not be found", __func__, jobidString);
continue;
}
/* scheduler-sim-live-allocation spec, task 1.2: plugin_job_election()
* is now core-typed (mirrors plugin_begin_session(&(sched->core))
* a few lines up in this same file) -- pass &(sched->core), not
* sched, exactly the same minimal call-site update that migration
* already made at this file's own plugin_begin_session()/
* plugin_end_session() call sites. */
plugin_job_election(&(sched->core), job);
/* calculate decision for jobid */
/* scheduler-sim-live-allocation spec, task 2.2: sched_make_
* allocation() is now core-typed -- pass &(sched->core), the same
* minimal call-site wrap this file's other driver-agnostic call
* sites already use. sched_allocator() itself stays
* xce_scheduler_t*-typed (task 3.1's job). */
alloc_rc = sched_make_allocation(&(sched->core), job->schedobj);
if (alloc_rc < 0) {
/* something wrong */
continue;
} else if (alloc_rc == 0) {
/* job is pend */
xce_pendreason_t* reason = job->schedobj->reason;
/* Req 1 AC 4 / Req 3 AC 2 "on_pend" row: keep the job as a
* candidate for the next round (no-op for builtin_fcfs; the
* legacy job_elect_fn path's picker_iface is
* legacy_picker_wrapper_iface, whose on_pend is also a no-op
* -- fcfs.c's own removal happens via its separate commit_hook,
* untouched by this call). */
job_picker->picker_iface->on_pend(job_picker, jobid);
if (reason == NULL) {
fmt_jobid(jobidString, sizeof(jobidString), jobid);
WARN("job %s is pending but pend reason is NULL", jobidString);
} else {
reason->jobid = job->jobid;
decision = make_pend_decision(reason);
}
} else {
/* Req 1 AC 4 / Req 3 AC 2 "on_dispatch" row: remove the job
* from the picker's candidate set permanently, before the
* decision is built (matches Req 1 AC 4's pseudocode order). */
job_picker->picker_iface->on_dispatch(job_picker, jobid);
MOVE_PTR(job->allocs, job->schedobj->alloc);
job->allocs->jobid = job->jobid;
decision = make_dispatch_decision(job->allocs);
}
if (decision != NULL) {
commit_decision(sched, decision);
cbuf_put(&(sched->core.decisions), &decision);
}
}
return NULL;
}
/* scheduler-sim-live-allocation spec, task 3.1: sched_job_picker()/
* sched_allocator() (and commit_decision(), their now-redundant
* xce_scheduler_t*-typed wrapper around sched_commit_decision()) relocated
* to sched_election.c, retyped to xce_sched_core_t* -- see that file's own
* header comment. sched_init() below spawns them via scheduler.h's extern
* declarations, passing &(sched->core) instead of sched. */
static void sched_pend_update_killer(data_item_t* it)
{
@ -977,14 +774,17 @@ int sched_init(xce_scheduler_t* sched, xce_client_config_t* config,
INFO("%s: joined cluster (%d candidate(s) configured)", __func__,
sched->cluster_cfg.candidates_count);
/* scheduler-sim-live-allocation spec, task 3.1: sched_job_picker()/
* sched_allocator() now live in sched_election.c, retyped to
* xce_sched_core_t* -- pass &(sched->core), not sched. */
rc = pthread_create(&(sched->core.jobpicker), NULL, &sched_job_picker,
sched);
&(sched->core));
if (rc < 0) {
return rc;
}
rc = pthread_create(&(sched->core.allocator), NULL, &sched_allocator,
sched);
&(sched->core));
return rc;
}
@ -1656,8 +1456,10 @@ void sched_end_session(xce_scheduler_t* sched)
* (all three already core-typed, tasks 1.2/2.1's "original 6-function
* migration") via &(sched->core), so this is a pure signature migration at
* the top of the function, no body-logic change. Its sole production
* caller, commit_decision() (this file, static), now wraps its own
* still-xce_scheduler_t*-typed argument at the call site instead. */
* caller is now sched_allocator() (task 3.1, sched_election.c), which
* calls it directly with the core-typed pointer it already holds -- the
* former commit_decision() wrapper this file used to route through is
* gone (see sched_election.c's own header comment). */
int sched_commit_decision(xce_sched_core_t* core, xce_decision_t* decision)
{
int rc = 0;

55
src/scheduler/testsuite/meson.build
View file

@ -160,21 +160,25 @@ sched_picker_wiring_test = executable('sched_picker_wiring_test',
# Regression guard for the core scheduling loop boundary (scheduler-core
# spec, task 2.5, Req 1 AC 1 / AC 7): "the core loop calls only
# picker->next_job() ... no project/queue/job iteration or ordering logic
# remains in the loop itself." Does NOT #include scheduler.c (that file's
# external dependency surface is why test_picker_wiring.c's own header
# comment already ruled out driving sched_job_picker() directly) -- instead
# it reads scheduler.c's real on-disk text at test run time and mechanically
# checks sched_job_picker()'s body for required picker-interface call sites
# and absent direct-traversal symbols. XCE_SCHEDULER_C_PATH bakes in the
# absolute path via meson.current_source_dir() so the check has no
# dependency on the test's cwd at run time.
# remains in the loop itself." Does NOT #include sched_election.c (that
# file's external dependency surface is why test_picker_wiring.c's own
# header comment already ruled out driving sched_job_picker() directly) --
# instead it reads sched_election.c's real on-disk text at test run time and
# mechanically checks sched_job_picker()'s body for required picker-
# interface call sites and absent direct-traversal symbols.
# XCE_SCHED_ELECTION_C_PATH bakes in the absolute path via
# meson.current_source_dir() so the check has no dependency on the test's
# cwd at run time. Repointed from scheduler.c to sched_election.c
# (scheduler-sim-live-allocation spec, task 3.1): sched_job_picker() moved
# there, unchanged in body -- see that file's own header comment.
sched_picker_core_loop_boundary_test = executable(
'sched_picker_core_loop_boundary_test',
['test_picker_core_loop_boundary.c'],
include_directories:[incdir],
dependencies: [scheduler_test_deps],
c_args: ['-DXCE_SCHEDULER_C_PATH="'
+ (meson.current_source_dir() / '..' / 'src' / 'scheduler.c')
c_args: ['-DXCE_SCHED_ELECTION_C_PATH="'
+ (meson.current_source_dir() / '..' / 'src'
/ 'sched_election.c')
+ '"'],
install: false)
@ -2042,3 +2046,34 @@ sched_maxtasks_wiring_sim_driver_parity_test = executable(
test('scheduler maxtasks wiring sim driver parity test',
sched_maxtasks_wiring_sim_driver_parity_test, suite: 'scheduler',
env: ['ASAN_OPTIONS=detect_leaks=0'])
# scheduler-sim-live-allocation spec, task 3.1 (Req 1.1, 1.2, 5.1, boundary:
# Unified Picker/Allocator Threads): proves sched_job_picker()/
# sched_allocator() -- relocated out of scheduler.c into their own file,
# sched_election.c, and re-typed to cast their void* arg directly to
# xce_sched_core_t* -- run correctly when pthread_create() is handed
# nothing but a bare xce_sched_core_t*, the exact shape a future
# simulation driver (task 4.1) will use. Same real-scheduler_srcs link
# technique as sched_scheduler_dispatch_commit_test above (scheduler_srcs,
# defined in the parent meson.build, minus src/main.c, built via a loop so
# it can never drift from the production source list) -- no `-Wl,--wrap=`
# needed since this file never calls sched_init() (see the test file's own
# header comment for why a real, but never-sched_init()'d, xce_scheduler_t
# is still used incidentally during setup).
sched_election_sim_fixture_test_srcs = ['test_sched_election_sim_fixture.c']
foreach f : scheduler_srcs
if f != 'src/main.c'
sched_election_sim_fixture_test_srcs += ['../' + f]
endif
endforeach
sched_election_sim_fixture_test = executable(
'sched_election_sim_fixture_test',
sched_election_sim_fixture_test_srcs,
include_directories:[incdir],
dependencies: [scheduler_test_deps],
install: false)
test('scheduler election sim fixture test',
sched_election_sim_fixture_test, suite: 'scheduler',
env: ['ASAN_OPTIONS=detect_leaks=0'])

35
src/scheduler/testsuite/test_picker_core_loop_boundary.c
View file

@ -38,11 +38,20 @@
* fopen+strstr instead of popen since no shell/glob is needed.
*
* RED demonstration performed manually during development (not committed):
* temporarily inserting a `for (job = sched->jobs.head; ...)`-style
* temporarily inserting a `for (job = core->jobs.head; ...)`-style
* fragment into sched_job_picker()'s real body made
* test_core_loop_has_no_forbidden_traversal_symbols fail immediately with
* the exact forbidden-symbol message, confirming the check exercises live
* source text rather than a frozen snapshot -- then reverted.
*
* Repointed from scheduler.c to sched_election.c (scheduler-sim-live-
* allocation spec, task 3.1): sched_job_picker() relocated there, retyped
* from xce_scheduler_t* to xce_sched_core_t* -- its local variable is now
* named `core`, not `sched`, so FORBIDDEN_SYMBOLS below was updated from
* "sched->queues"/"sched->jobs" to "core->queues"/"core->jobs" to keep
* guarding the same property under the new name (the function's body,
* FUNC_ANCHOR, and REQUIRED_SYMBOLS are otherwise unchanged -- this was a
* pure relocation, not a behavior change).
*/
#include <stdio.h>
#include <stdlib.h>
@ -50,9 +59,9 @@
#include <unity/unity.h>
#ifndef XCE_SCHEDULER_C_PATH
#ifndef XCE_SCHED_ELECTION_C_PATH
#error \
"XCE_SCHEDULER_C_PATH must be defined by meson.build to the absolute path of scheduler.c"
"XCE_SCHED_ELECTION_C_PATH must be defined by meson.build to the absolute path of sched_election.c"
#endif
/* Anchor is the literal function signature text, unique to the real
@ -61,9 +70,14 @@
* sched_job_picker src/scheduler`: only the definition itself carries this
* exact substring). If this signature is ever reformatted, this test fails
* loudly (extraction returns NULL) rather than silently passing on stale
* text. */
static const char* const FUNC_ANCHOR
= "static void* sched_job_picker(void* arg)";
* text.
*
* No longer `static` (scheduler-sim-live-allocation spec, task 3.1):
* sched_job_picker() now needs external linkage so scheduler.c's
* sched_init() (a different translation unit) can spawn it -- see
* scheduler.h's extern declaration and sched_election.c's own header
* comment. */
static const char* const FUNC_ANCHOR = "void* sched_job_picker(void* arg)";
/* Symbols whose presence in sched_job_picker()'s body would mean project/
* queue/job iteration or ordering logic leaked back into the core loop
@ -71,8 +85,8 @@ static const char* const FUNC_ANCHOR
* only jobs are: resolve the picker, delegate to job_elect_fn OR drive
* picker_iface's session hooks, and shuttle jobids across cbuf. */
static const char* const FORBIDDEN_SYMBOLS[] = {
"sched->queues", "sched->jobs", "iter_make(", "iter_next(",
"ordq_get(", "ordq_put(", "ordq_peek(", "key_find(",
"core->queues", "core->jobs", "iter_make(", "iter_next(",
"ordq_get(", "ordq_put(", "ordq_peek(", "key_find(",
};
#define NUM_FORBIDDEN (sizeof(FORBIDDEN_SYMBOLS) / sizeof(FORBIDDEN_SYMBOLS[0]))
@ -173,9 +187,10 @@ static char* extract_function_body(const char* text, const char* anchor)
* reason not to). */
static char* load_core_loop_body(void)
{
char* file_text = read_whole_file(XCE_SCHEDULER_C_PATH);
char* file_text = read_whole_file(XCE_SCHED_ELECTION_C_PATH);
TEST_ASSERT_NOT_NULL_MESSAGE(
file_text, "could not read scheduler.c at XCE_SCHEDULER_C_PATH");
file_text,
"could not read sched_election.c at XCE_SCHED_ELECTION_C_PATH");
char* body = extract_function_body(file_text, FUNC_ANCHOR);
free(file_text);

296
src/scheduler/testsuite/test_sched_election_sim_fixture.c Normal file
View file

@ -0,0 +1,296 @@
/*
* test_sched_election_sim_fixture.c - scheduler-sim-live-allocation spec,
* task 3.1 (Req 1.1, 1.2, 5.1, boundary: Unified Picker/Allocator Threads).
*
* Task 3.1 relocates sched_job_picker()/sched_allocator() out of
* scheduler.c (where they were `xce_scheduler_t*`-cast, normal-mode-only
* thread bodies) into their own file, sched_election.c, re-typed to cast
* their void* arg to xce_sched_core_t* directly. The existing normal-mode
* scheduling test suite (test_scheduler_backpressure.c,
* test_snapshot_load_allocator_regression.c, test_scheduler_cache_
* dispatch.c, ...) already proves normal mode's own behavior is unchanged
* by this relocation (they all link the real scheduler_srcs, so they
* automatically pick up sched_election.c once it is added to that list --
* see meson.build). What none of them prove is the NEW capability this
* task adds: that the identical, unmodified thread pair also runs
* correctly when pthread_create() is handed nothing but a bare
* xce_sched_core_t* -- no xce_scheduler_t* anywhere in the call -- which is
* the exact shape a future simulation driver (task 4.1,
* xce_sched_sim_driver_t, which embeds xce_sched_core_t core as its own
* first field, never an xce_scheduler_t) will use. This file is that
* proof.
*
* Fixture note on `wrapper`: this test needs core->seqjob_allocator
* populated (this fixture's resreq, TEST_RESREQ below, yields a
* maxTasks==1 primary, so sched_make_seqjob_allocation() -- scheduler.c --
* is the one sched_allocator() reaches; see scheduler.c's own comment on
* that pair for why "any real allocation attempt at all, dispatch OR pend,
* requires core->seqjob_allocator/parjob_allocator to be non-NULL first").
* The only
* way to populate it today is the real, production register_seqjob_
* allocator()/register_parjob_allocator() (plugins/seqjob_allocator.c,
* plugins/parjob_allocator.c) -- but those two plugins' own init_fn
* (seqjob_allocator_init()/parjob_allocator_init()) still read
* plugin->sched (cast to xce_scheduler_t*) rather than the newer
* plugin->core field, a pre-existing, tracked, OUT-OF-SCOPE gap (design.md's
* own "open question about job_alloc_fn/register_seqjob_allocator/
* register_parjob_allocator", explicitly flagged as "verify before
* assuming this chain is fully closed" and NOT one of task 3.1's five
* permitted changes -- fixing it would be scope creep into a different,
* not-yet-assigned task).
*
* `wrapper` (a genuine, real xce_scheduler_t) exists SOLELY to satisfy
* that unrelated plugin's own precondition during setup -- it is never
* passed to sched_init(), never joined to a controller, and critically,
* NEVER passed to pthread_create() or dereferenced by either function
* under test. Only `&(wrapper.core)` is -- the exact same bare
* xce_sched_core_t* a real xce_sched_sim_driver_t would hand these
* functions in production. This keeps the assertion this test makes
* (sched_job_picker()/sched_allocator() are genuinely core-only) honest:
* the incidental use of an xce_scheduler_t during setup is working around
* a DIFFERENT, already-core-safe-call-chain plugin's registration gap, not
* the two functions this task relocates.
*
* Resreq choice: "select[type==any] rusage[mem=1]" -- proven by
* src/lib/testsuite/test_resreq.c's own test_resreq1() to parse to a bare
* XCE_RESREQ_PRIMARY tag (unlike "type == any", schedobj.c's/snapshot_
* load.c's own NULLSTR() default, which test_snapshot_load_allocator_
* regression.c's own header comment documents parses into an ALTER-tagged
* schedobj wrapping one COMPOUND wrapping one PRIMARY). That distinction
* matters here because sched_make_alter_allocation() (scheduler.c) has a
* separate, pre-existing, out-of-scope gap ("FIXME: merge pend reason from
* all children") that drops a failed child's pend reason instead of
* propagating it to the top-level schedobj -- exactly the gap that same
* file's test hit for its own ALTER-tagged job, producing no decision at
* all for it. A bare PRIMARY tag has no such wrapper to lose the reason
* across: sched_make_allocation() calls sched_make_primary_allocation()
* directly on the job's own top-level schedobj, so whatever reason my_
* alloc() (seqjob_allocator.c) attaches when it finds no worker to satisfy
* this job lands on job->schedobj->reason directly, and a genuine
* XCE_DECISION_PEND reaches sched_allocator()'s commit call -- this
* fixture registers no worker at all, so that PEND outcome is guaranteed
* deterministically rather than depending on getting a real resource match
* exactly right (out of scope for what task 3.1 needs to prove: the thread
* pair reaching a committed decision at all, not the correctness of a
* particular allocation decision).
*
* Follows this directory's established convention for a test that spawns
* real, un-joined picker/allocator threads referencing a stack-local
* struct (test_scheduler_backpressure.c, test_snapshot_load_allocator_
* regression.c): exactly one RUN_TEST() in this binary, so the process
* exits immediately after, before that stack frame could ever be reused.
*/
#include <errno.h>
#include <pthread.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <unity/unity.h>
#include "decision.h"
#include "plugin.h"
#include "preason.h"
#include "resobj.h"
#include "sched/types.h"
#include "scheduler.h"
#include "xstring.h"
#define TEST_JOBID ((uint64_t)313131ULL)
#define QUEUE_NAME "normal"
#define TEST_RESREQ ((char*)"select[type==any] rusage[mem=1]")
void setUp(void) { return; }
void tearDown(void) { return; }
/*
* Req 1.1/1.2/5.1: sched_job_picker()/sched_allocator() (sched_election.c),
* spawned via pthread_create() against nothing but a bare
* xce_sched_core_t*, pick a real job, elect and allocate it, and commit a
* real decision -- proving the relocated pair is the single, driver-
* agnostic implementation Requirement 5 requires, not merely unchanged in
* normal mode.
*/
static void
test_sched_election_thread_pair_runs_against_bare_core_fixture(void)
{
xce_scheduler_t wrapper; /* see file header: setup-only, never
* passed to the functions under test */
xce_sched_core_t* core;
xce_sched_plugin_t* fcfs;
xce_sched_queue_t* queue;
xce_job_t j;
job_submit_data_t submit;
xce_sched_job_t* job;
pthread_t picker_tid;
pthread_t allocator_tid;
xce_decision_t* decision = NULL;
xce_decision_t* end_marker = (xce_decision_t*)0x1;
int rc;
memset(&wrapper, 0, sizeof(wrapper));
rc = xce_sched_core_init(&(wrapper.core));
TEST_ASSERT_EQUAL_INT(0, rc);
core = &(wrapper.core);
/* seqjob_prepare_allocation()/parjob_prepare_allocation()'s real
* key_find(core->resmgr, "slots", ...) call (this is the auto-injected
* per-task "slots" want every primary resreq gets, not something
* TEST_RESREQ itself spells out) needs "slots" to be a genuinely
* registered resource, the same startup-time-config convention every
* real-scheduler_srcs test in this directory follows (resources are
* static cluster config, normally loaded once by sched_init_resources()
* before any job event flows in -- never reached here since this
* fixture never pushes a real CONFIG_PUSH event). Without this, the
* lookup returns -ENOENT before either allocator plugin ever reaches
* its own "no candidate satisfied" pend-reason logic, so the job's
* failure is a missing-config error, not a real, reason-carrying PEND
* -- see test_parjob_e2e_multislot.c's own identical
* register_slots_resource() helper for the established precedent. */
{
xce_resource_t res;
memset(&res, 0, sizeof(res));
res.name = strdup("slots");
TEST_ASSERT_NOT_NULL(makeResource(core->resmgr, &res));
}
/* Register the real, unmodified builtin_fcfs picker plugin the same
* way sched_register_plugins()/register_builtin_fcfs() do -- kept
* inline (plugin_register(), not register_builtin_fcfs()) only so this
* fixture retains the plugin pointer to drive its picker_iface hooks
* directly below, exactly as sched_init_queues()/core_events.c's real
* JOBS_PUSH path already do for a live job/queue arrival. */
fcfs = builtin_fcfs_new_plugin();
TEST_ASSERT_NOT_NULL(fcfs);
rc = plugin_register(&wrapper, fcfs);
TEST_ASSERT_TRUE(rc >= 0);
core->job_pick_plugin = (char*)"builtin_fcfs";
/* See file header: populates core->seqjob_allocator/parjob_allocator
* via the real, unmodified registration functions -- the ONLY existing
* way to do so, since those two plugins' init_fn require a genuine
* xce_scheduler_t* (an unrelated, out-of-scope gap for this task). */
rc = register_seqjob_allocator(&wrapper);
TEST_ASSERT_TRUE(rc >= 0);
rc = register_parjob_allocator(&wrapper);
TEST_ASSERT_TRUE(rc >= 0);
/* Deliberately no worker registered -- see file header for why a
* guaranteed-unsatisfiable bare-PRIMARY job is the simplest way to get
* a genuine, deterministic decision out of the real allocation path. */
/* Real queue registration -- the same sequence sched_init_queues()
* (scheduler.c) uses for builtin_fcfs: add to the core, then notify
* the picker so its own internal candidate registry sees it. */
queue = (xce_sched_queue_t*)calloc(1, sizeof(*queue));
TEST_ASSERT_NOT_NULL(queue);
queue->name = makeString(QUEUE_NAME);
rc = sched_add_queue(core, queue);
TEST_ASSERT_TRUE(rc >= 0);
fcfs->picker_iface->on_queue_update(fcfs, queue);
/* Real job construction via make_sched_job() (schedobj.c) -- the same
* function a live JOBS_PUSH/config-push uses -- so job->schedobj gets a
* genuinely parsed TEST_RESREQ resreq tree, not a hand-built stand-in.
* `wrapper` is passed here only because make_sched_job() itself
* still takes `void* arg` cast to xce_scheduler_t* (out of this task's
* boundary) -- the resulting job is then driven entirely through
* `core` below. */
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)QUEUE_NAME;
submit.user = (char*)"sim_fixture_user";
submit.project_name = (char*)"default";
submit.resreq = TEST_RESREQ;
submit.mintasks = 1;
submit.maxtasks = 1;
memset(&j, 0, sizeof(j));
j.jobid = TEST_JOBID;
j.state = JOB_STAT_PEND;
j.submit = &submit;
job = make_sched_job(&wrapper, &j);
TEST_ASSERT_NOT_NULL(job);
/* make_sched_job()'s own parse_resreq() call (unlike the real
* JOBS_PUSH event path's core_build_job(), resreq-maxtasks-wiring
* feature) does not thread submit.maxtasks/mintasks into the parsed
* primary's own minTasks/maxTasks -- parse_resreq_primary() (resreq.c)
* leaves both at their unset default (0) when TEST_RESREQ's own text
* carries no task-count clause. A maxTasks of 0 (not 1) routes
* sched_make_primary_alloc_()'s dispatch (scheduler.c) to
* sched_make_parjob_allocation() instead of sched_make_seqjob_
* allocation() -- and parjob_allocator.c's own my_alloc() deliberately
* bails out for maxTasks <= 1 BEFORE ever creating a pend reason (its
* own "Bailing out here, before ctx->reason is ever created" comment),
* exactly reproducing the "pend reason is NULL" WARN this test is
* written to avoid. Set both fields directly so this fixture
* deterministically exercises the seqjob path instead, whose own
* my_alloc() unconditionally creates a reason at entry. */
TEST_ASSERT_EQUAL_INT(XCE_RESREQ_PRIMARY, job->schedobj->tag);
job->schedobj->u.primary->resreq->minTasks = 1;
job->schedobj->u.primary->resreq->maxTasks = 1;
rc = sched_add_job(core, job);
TEST_ASSERT_TRUE(rc >= 0);
fcfs->picker_iface->on_job_push(fcfs, job);
/* ── The actual thing under test ─────────────────────────────────
* pthread_create() is handed &core (xce_sched_core_t*) directly --
* never &wrapper -- for both thread bodies. This is the load-bearing
* assertion: if either function still dereferenced anything outside
* core->*, this would not compile (their signature is void* (void*),
* so a type error here is impossible either way) but would crash or
* misbehave at runtime the instant it tried to read state that only
* exists on a real xce_scheduler_t. */
rc = pthread_create(&picker_tid, NULL, sched_job_picker, core);
TEST_ASSERT_EQUAL_INT(0, rc);
rc = pthread_create(&allocator_tid, NULL, sched_allocator, core);
TEST_ASSERT_EQUAL_INT(0, rc);
/* xce_core_go() (core.h): the simple, no-bookkeeping-to-interleave
* entry point its own doc comment calls out as being for exactly this
* kind of caller ("e.g. a future simulation driver") -- begins the
* session and signals the picker in one non-blocking call. */
rc = xce_core_go(core);
TEST_ASSERT_EQUAL_INT(0, rc);
/* Real, blocking drain -- no sleep/poll: cbuf_get() blocks until the
* real allocator thread commits a decision, the same drain discipline
* driver.c's own normal-mode round uses. First pop is this session's
* one job; second is the end-of-session NULL marker. */
cbuf_get(&(core->decisions), &decision);
TEST_ASSERT_NOT_NULL_MESSAGE(
decision, "sched_job_picker()/sched_allocator(), spawned against a "
"bare xce_sched_core_t*, must still carry a job all the "
"way from being picked to a committed decision");
TEST_ASSERT_EQUAL_INT(XCE_DECISION_PEND, decision->op);
TEST_ASSERT_NOT_NULL(decision->u.reason);
TEST_ASSERT_EQUAL_UINT64(TEST_JOBID, decision->u.reason->jobid);
cbuf_get(&(core->decisions), &end_marker);
TEST_ASSERT_NULL_MESSAGE(end_marker,
"second decisions-cbuf entry must be the "
"end-of-session NULL marker");
rc = xce_core_end_session(core);
TEST_ASSERT_EQUAL_INT(0, rc);
/* Job-state side effects: sched_commit_decision()'s PEND branch
* (already core-safe from task 2.2) touches neither job->state nor
* job->allocs/accounts, mirroring test_scheduler_dispatch_commit.c's
* own test_pend_decision_does_not_touch_job_state -- this test's own
* focus is the thread pair reaching that call at all, not re-proving
* sched_commit_decision()'s own already-covered behavior. */
TEST_ASSERT_EQUAL_UINT(JOB_STAT_PEND, job->state);
TEST_ASSERT_NULL(job->allocs);
TEST_ASSERT_NULL(job->accounts);
}
int main(void)
{
UNITY_BEGIN();
RUN_TEST(test_sched_election_thread_pair_runs_against_bare_core_fixture);
return UNITY_END();
}