Rongsong Shen
89297dc825
Adds a NULL-by-default lookup/populate callback seam on xce_sched_core_t, mirroring cache_clear_cb's proven shape, so the allocation dispatch chokepoint can consult the unsatisfiable-resreq cache without a normal-mode-only field access in shared code. Adapters live in alloc_cache.c and are registered only during normal-mode startup; simulation mode leaves both callbacks NULL and always computes fresh. Adds the scheduler-sim-live-allocation spec (requirements/design/ research/tasks) as the planning artifact this and subsequent tasks implement against. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
764 lines
40 KiB
C
764 lines
40 KiB
C
#if !defined(_CORE_H_)
|
|
|
|
#define _CORE_H_ 1
|
|
|
|
/*
|
|
* core.h - xce_sched_core_t: the scheduler's pure scheduling-state
|
|
* structure (scheduler-core spec, task 4.1, Req 5.1/5.2).
|
|
*
|
|
* Boundary (design.md "SchedulerCore" / "Existing Architecture Analysis"):
|
|
* `xce_scheduler_t` (scheduler.h) splits along the controller-I/O seam into
|
|
* this structure (no `apictx`/`cluster_cfg` -- zero controller I/O, network,
|
|
* or wall-clock dependency) and the driver-level fields that remain on
|
|
* `xce_scheduler_t` itself. Every field below is the SAME field, same name,
|
|
* same type, that lived directly on `xce_scheduler_t` before this task --
|
|
* this is a relocation, not a redesign; `xce_scheduler_t` now embeds this
|
|
* struct as its `core` member instead of declaring these fields itself.
|
|
*
|
|
* Allowed dependencies only (design.md "Allowed Dependencies", Req 5 AC 3):
|
|
* ordq, dataq, cbuf, xlist, zarray, kvmgr, rtti. This header and core.c
|
|
* must NEVER include client.h/netio.h/xceapi.h/delay.h/idgen.h/
|
|
* xce/cluster.h, and must never call a wall-clock function (time(),
|
|
* clock_gettime(), gettimeofday()) -- enforced by
|
|
* testsuite/test_core_isolation.c (source-text scan + an nm(1) undefined-
|
|
* symbol audit of core.c's compiled object, the build-time proof this
|
|
* task's Observable requires).
|
|
*
|
|
* One documented exception (task 4.5, Req 5 AC 3): the core's single
|
|
* sanctioned wall-clock call -- clock_gettime(CLOCK_MONOTONIC) inside
|
|
* xce_core_now() -- and its tlog format-function integration live in
|
|
* core_clock.c, a SEPARATE file from core.c, specifically so
|
|
* test_core_isolation.c's core.c-scoped "zero wall-clock calls" text scan
|
|
* keeps proving what it has always proven (core.c itself never reads real
|
|
* time) without needing to special-case the one legitimate call site AC 3
|
|
* mandates. tlog (tinylog) is likewise an allowed core dependency as of
|
|
* this task -- AC 3's own "Third-party library constraint" section
|
|
* discusses the core registering a tlog format function -- distinct from
|
|
* the forbidden client.h/netio.h/xceapi.h/delay.h/idgen.h/xce/cluster.h
|
|
* list above, none of which tlog is part of.
|
|
*/
|
|
|
|
#include <pthread.h>
|
|
#include <rtti.h>
|
|
#include <stdatomic.h>
|
|
#include <stdint.h>
|
|
|
|
#include "dataq.h"
|
|
#include "decision.h"
|
|
#include "mylock.h"
|
|
#include "plugin.h"
|
|
#include "sched/types.h"
|
|
#include "xce/types.h"
|
|
#include "zarray.h"
|
|
|
|
/* xce_sched_core_t is also forward-declared in plugin.h (picker lifecycle
|
|
* hook parameter type); this typedef and the full struct definition below
|
|
* are the canonical declaration -- same idiom as cluster_t across
|
|
* myconf.h/persist.h/ha.h/cluster_def.h (an identical repeated typedef is
|
|
* valid C11/gnu17, so include order between core.h and plugin.h doesn't
|
|
* matter). */
|
|
typedef struct xce_sched_core_s xce_sched_core_t;
|
|
|
|
/*
|
|
* xce_core_decision_cb_t - the driver-registered decision callback (task
|
|
* 4.4, Req 5 AC 6): `int (*on_decision)(void* driver_ctx, xce_decision_t*
|
|
* decision)`. Invoked once per decision, in allocator-output order, as the
|
|
* driver drains `core->decisions` between `xce_core_go()` and
|
|
* `xce_core_end_session()` -- see xce_core_go()/xce_core_end_session()'s
|
|
* own doc comments (core_session.c) for the full protocol.
|
|
*/
|
|
typedef int (*xce_core_decision_cb_t)(void*, xce_decision_t*);
|
|
|
|
/*
|
|
* xce_core_cache_clear_cb_t - driver-registered per-session cache-clear
|
|
* hook (scheduler-allocation spec, task 5.2 round 2, Req 7.2):
|
|
* `void (*)(xce_sched_core_t*)`. Mirrors xce_core_decision_cb_t's
|
|
* driver-registered-hook idiom just above, but generic on purpose -- no
|
|
* `void* driver_ctx` companion field, because the sole real
|
|
* implementation (alloc_cache.c's xce_alloc_cache_clear_cb() -- see that
|
|
* function's own doc comment for why it lives in alloc_cache.c rather
|
|
* than driver.c) recovers its owning xce_scheduler_t* via the
|
|
* pointer-to-first-member idiom this codebase already uses for this exact
|
|
* struct (scheduler.h: `xce_sched_core_t core` is xce_scheduler_t's first
|
|
* member), not via an opaque context pointer.
|
|
*
|
|
* WHY a hook instead of a direct call (task 5.2 round 1's CRITICAL defect):
|
|
* core_begin_session_internal() (core_session.c) is reached from BOTH
|
|
* driver.c's xce_sched_driver_run_normal_mode_round() (a real
|
|
* xce_scheduler_t*) and sim_loop.c's sched_sim_driver_run() (a
|
|
* structurally different xce_sched_sim_driver_t*, sim.h -- Req 5 AC 17
|
|
* "Simulation isolation" -- which has NO unsatisfiable_cache field at
|
|
* all). Round 1 called alloc_cache.c's real, strong
|
|
* cache_clear(xce_scheduler_t*) unconditionally for both callers (routed
|
|
* through a same-named weak fallback that the strong definition always
|
|
* overrides at link time in the production binary) -- since link-time
|
|
* weak/strong resolution picks ONE winner process-wide, not per call
|
|
* site, every simulation-mode session reinterpreted whatever field sits
|
|
* at unsatisfiable_cache's byte offset in xce_sched_sim_driver_t
|
|
* (alt_events_file, a live FILE*) as a kvtab_t* and clobbered it.
|
|
*
|
|
* A per-core-instance function pointer fixes this at the data-ownership
|
|
* level: each xce_sched_core_t only ever carries the hook ITS OWN driver
|
|
* installed via xce_core_set_cache_clear_cb() below. main.c installs
|
|
* alloc_cache.c's xce_alloc_cache_clear_cb() on the real xce_scheduler_t's
|
|
* core; sim_loop.c never calls the setter at all, so a simulation-mode core's
|
|
* hook stays NULL (xce_sched_core_init()'s memset() default) and
|
|
* core_begin_session_internal() treats NULL as "this driver has no cache
|
|
* to clear" and skips the call entirely -- a true no-op, not an
|
|
* accidental type-confused one.
|
|
*/
|
|
typedef void (*xce_core_cache_clear_cb_t)(xce_sched_core_t*);
|
|
|
|
/*
|
|
* xce_unsatisfiable_lookup_cb_t / xce_unsatisfiable_populate_cb_t -
|
|
* driver-registered, OPTIONAL per-session unsatisfiable-resreq cache
|
|
* consult/populate seam (scheduler-sim-live-allocation spec, task 1.1,
|
|
* Req 5.2). Mirrors xce_core_cache_clear_cb_t's own shape and rationale
|
|
* immediately above exactly: a bare function pointer taking
|
|
* xce_sched_core_t* directly, no wrapper struct, no opaque "impl"/
|
|
* user-data pointer -- single registrant (main.c's normal-mode branch),
|
|
* no foreseeable second implementation.
|
|
*
|
|
* Parameter shape mirrors the REAL cache's own existing entry points
|
|
* (alloc_cache.c's cache_lookup()/cache_insert(), design.md's
|
|
* "Unsatisfiable-Cache Callback Seam" component) rather than inventing a
|
|
* new bare-integer "fingerprint" type: xce_sched_obj_t* is the exact
|
|
* object cache_lookup()/cache_insert() already key on (its embedded
|
|
* xce_resreq_fingerprint_t plus user/queue/project identity, sched/
|
|
* alloc_cache.h) -- this seam adds no new type, only a NULL-safe
|
|
* indirection over the existing one. xce_sched_obj_t is already visible
|
|
* here as a forward-declared pointer type (sched/types.h, included
|
|
* above); only a pointer, never the full struct, is needed in this
|
|
* header.
|
|
*
|
|
* WHY a hook instead of a direct call (mirrors xce_core_cache_clear_cb_t's
|
|
* own documented incident above): the allocation dispatch chokepoint this
|
|
* seam wraps (sched_make_primary_alloc_(), scheduler.c) is reached
|
|
* identically by both drivers once a later task's Core-Safe Allocation
|
|
* Chain migration lands -- but the real cache
|
|
* (xce_scheduler_t.unsatisfiable_cache) exists ONLY on a normal-mode
|
|
* xce_scheduler_t, never on a simulation-mode xce_sched_sim_driver_t. A
|
|
* NULL-checked, driver-registered callback keeps that difference behind
|
|
* an explicit seam instead of a struct-prefix cast into mode-specific
|
|
* state from code both drivers share -- the same anti-pattern
|
|
* xce_core_cache_clear_cb_t's own doc comment documents a real, prior
|
|
* memory-corruption incident from.
|
|
*
|
|
* NULL (the xce_sched_core_init() memset() default, unless a driver
|
|
* registers one) means "always compute fresh" for BOTH callbacks -- never
|
|
* an error, never a degraded-correctness path, only a degraded-
|
|
* performance one. Wiring the actual consult/populate call sites into
|
|
* sched_make_primary_alloc_() is a LATER task's job (tasks.md 2.2,
|
|
* boundary "Core-Safe Allocation Chain, Unsatisfiable-Cache Callback
|
|
* Seam") -- this task adds only the seam (these two fields, their
|
|
* adapters in alloc_cache.c, and main.c's normal-mode-only registration),
|
|
* not that call site. Every simulation-mode core leaves both fields NULL
|
|
* forever (sim_loop.c never registers them, Req 5 AC 17 "Simulation
|
|
* isolation"), so a simulation-mode session never touches the real cache
|
|
* implementation at all.
|
|
*/
|
|
typedef int (*xce_unsatisfiable_lookup_cb_t)(xce_sched_core_t* core,
|
|
xce_sched_obj_t* obj,
|
|
int* out_reason_category);
|
|
typedef void (*xce_unsatisfiable_populate_cb_t)(xce_sched_core_t* core,
|
|
xce_sched_obj_t* obj,
|
|
int reason_category);
|
|
|
|
struct xce_sched_core_s {
|
|
mylock_t lock;
|
|
int stop;
|
|
uint64_t nextid;
|
|
xce_conf4sched_t* config;
|
|
uint64_t configver;
|
|
/* Req 2 AC 1 (scheduler-core spec, task 1.3): global picker plugin
|
|
* name. NULL until sched_set_default_job_pick_plugin() (schedobj.c)
|
|
* applies the "builtin_fcfs" default, or a future config-driven
|
|
* assignment sets it explicitly. */
|
|
char* job_pick_plugin;
|
|
var plugins;
|
|
void* seqjob_allocator;
|
|
void* parjob_allocator;
|
|
void* affinity_allocator;
|
|
void* gpu_allocator;
|
|
dataq_t* pend_updates[2];
|
|
var resmgr;
|
|
var ugmgr;
|
|
var qgmgr;
|
|
var pgmgr;
|
|
var hgmgr;
|
|
zarray_t sharedInst;
|
|
zarray_t sharedInUse;
|
|
var jobs;
|
|
var queues;
|
|
var workers;
|
|
var users;
|
|
var projects;
|
|
pthread_t jobpicker;
|
|
pthread_t allocator;
|
|
cbuf_t wbuf; /* action to trigger scheduling */
|
|
cbuf_t jbuf; /* job picked for allocation */
|
|
cbuf_t decisions; /* scheduler decisions */
|
|
/* Req 5 AC 6 (scheduler-core spec, task 4.4): driver-registered
|
|
* decision callback + opaque driver context, set via
|
|
* xce_core_set_decision_cb() (core_session.c). NULL until a driver
|
|
* registers one -- nothing in the still-live sched_go()/main.c path
|
|
* reads these fields today. */
|
|
void* driver_ctx;
|
|
xce_core_decision_cb_t on_decision;
|
|
/* Req 7.2 (scheduler-allocation spec, task 5.2 round 2): driver-
|
|
* registered per-session cache-clear hook -- see
|
|
* xce_core_cache_clear_cb_t's own doc comment above for the full
|
|
* rationale. NULL until a driver registers one (xce_core_set_
|
|
* cache_clear_cb()); core_begin_session_internal() (core_session.c)
|
|
* treats NULL as "this driver has no cache to clear" and skips the
|
|
* call -- true for every simulation-mode xce_sched_core_t, which never
|
|
* registers this hook. */
|
|
xce_core_cache_clear_cb_t cache_clear_cb;
|
|
/* Req 5.2 (scheduler-sim-live-allocation spec, task 1.1): driver-
|
|
* registered, OPTIONAL unsatisfiable-resreq cache consult/populate
|
|
* seam -- see xce_unsatisfiable_lookup_cb_t/xce_unsatisfiable_
|
|
* populate_cb_t's own doc comment above for the full rationale. NULL
|
|
* (xce_sched_core_init()'s memset() default) until main.c's
|
|
* normal-mode branch registers both, alongside cache_clear_cb just
|
|
* above. A simulation-mode core never registers either, so
|
|
* consulting/populating this seam is always a true no-op for sim
|
|
* mode ("always compute fresh"), never a fallback onto real cache
|
|
* state. */
|
|
xce_unsatisfiable_lookup_cb_t unsatisfiable_lookup_cb;
|
|
xce_unsatisfiable_populate_cb_t unsatisfiable_populate_cb;
|
|
_Atomic int session_active; /* 1 during scheduling session, 0 between */
|
|
_Atomic int eval_in_progress; /* 1 while eval_select loop runs */
|
|
_Atomic int staging_merge_complete; /* 1 after staging merge completes */
|
|
select_eval_ctx_t* sel_eval_ctx; /* session-level memoisation context; NULL
|
|
between sessions */
|
|
/*
|
|
* Clock virtualization (Req 5 AC 3, scheduler-core spec, task 4.5): the
|
|
* core's ONLY sanctioned time source is xce_core_now() (core_clock.c),
|
|
* which computes virtual time from these four fields via the
|
|
* two-anchor + speed formula:
|
|
*
|
|
* wall_ns = clock_gettime(CLOCK_MONOTONIC) // the one wall-clock
|
|
* // call in the whole
|
|
* // core -- lives in
|
|
* // core_clock.c, NOT
|
|
* // core.c (core.c's own
|
|
* // zero-wall-clock
|
|
* // guarantee, enforced
|
|
* // by
|
|
* // test_core_isolation.c,
|
|
* // stays intact)
|
|
* elapsed = wall_ns - real_clock_base
|
|
* xce_core_now() = virtual_clock_base + (elapsed * speed_num) / speed_den
|
|
*
|
|
* Atomically updated by the driver AS A GROUP before each event
|
|
* dispatch (design.md's State Management note) -- xce_core_now() reads
|
|
* them lock-free, no mutex, safe from both the picker and allocator
|
|
* threads. xce_sched_core_init() seeds these to a neutral
|
|
* (virtual_clock_base=0, real_clock_base=0, speed_num=1, speed_den=1)
|
|
* default -- NOT a real wall-clock anchor, despite this field's older
|
|
* doc comment claiming otherwise (bug-1584): with both anchors at 0,
|
|
* xce_core_now() degrades to raw CLOCK_MONOTONIC-since-boot
|
|
* nanoseconds, not epoch time. A real driver must seed a genuine
|
|
* wall-clock anchor itself -- the normal-mode driver does so via
|
|
* xce_core_set_wallclock_anchor() (scheduler.c's sched_init(), right
|
|
* after xce_sched_core_init()); the simulation driver does so via
|
|
* sim.c's sim_set_virtual_clock(). Only bare-core test fixtures that
|
|
* never call either keep the neutral (0,0,1,1) default.
|
|
*/
|
|
_Atomic uint64_t virtual_clock_base;
|
|
_Atomic uint64_t real_clock_base;
|
|
_Atomic uint64_t speed_num;
|
|
_Atomic uint64_t speed_den;
|
|
/*
|
|
* Session response-time diagnostics (Req 5 AC 12, task 4.5). Plain
|
|
* (non-atomic) fields -- touched only by the driver thread, inside
|
|
* xce_core_go()/xce_core_end_session() (core_session.c), the same
|
|
* single-writer discipline core->sel_eval_ctx above already follows.
|
|
* session_start is stamped at the top of xce_core_go(); the field name
|
|
* matches requirements.md's own pseudocode verbatim
|
|
* ("session_start = xce_core_now()"). session_duration_ns is computed
|
|
* at the top of xce_core_end_session() as xce_core_now() -
|
|
* session_start. Both are diagnostics only -- nothing in the
|
|
* scheduling/decision path reads them. The ReplaySession.summary
|
|
* SessionSummary wire struct these values eventually feed is explicitly
|
|
* OUT of this task's scope (Phase 3, recording/replay-format tasks
|
|
* 7.x) -- this struct only holds the plain numeric tracking.
|
|
*/
|
|
uint64_t session_start;
|
|
uint64_t session_duration_ns;
|
|
/*
|
|
* PRNG seed for deterministic tie-breaking (Req 5 AC 7 / task 8.1). This
|
|
* field did not exist before task 8.1 -- requirements.md's Req 5 AC 7
|
|
* explicitly names `core->prng_seed` ("Any required randomness ... shall
|
|
* use a PRNG seeded by the driver at init time") and design.md's
|
|
* eventual `xce_sched_core_init(core, decision_cb, driver_ctx,
|
|
* prng_seed)` signature (see this header's own doc comment above
|
|
* xce_sched_core_init()) both presuppose it, but no prior task added it
|
|
* or wired a PRNG consumer -- confirmed by grepping this entire spec's
|
|
* source tree for "prng"/"seed" before this task. Adding ONLY the field
|
|
* here, seeded to a deterministic constant (0) by
|
|
* xce_sched_core_init(), keeps this task's scope to "the field exists
|
|
* and round-trips through save_snapshot()/load_snapshot()"
|
|
* (Req 5 AC 21's "PRNG state" checklist row) -- NOT changing
|
|
* xce_sched_core_init()'s signature (still deliberately narrower per
|
|
* this header's doc comment below) and NOT wiring any actual PRNG
|
|
* consumer, since nothing in the codebase performs tie-breaking
|
|
* randomness yet (builtin_fcfs's ordering is fully deterministic via
|
|
* ordq comparators, no PRNG call site exists to seed). A future task
|
|
* that adds real PRNG-based tie-breaking is expected to add its own
|
|
* setter, the same pattern xce_core_set_decision_cb() already
|
|
* establishes for driver_ctx/on_decision.
|
|
*/
|
|
uint64_t prng_seed;
|
|
};
|
|
|
|
/*
|
|
* xce_sched_core_init/xce_sched_core_fini - allocate/initialize, and tear
|
|
* down, exactly the state above. Deliberately narrower than the design's
|
|
* eventual Service Interface (`xce_sched_core_init(core, decision_cb,
|
|
* driver_ctx, prng_seed)`, design.md "SchedulerCore" component) -- the
|
|
* decision callback, driver context, and PRNG seed belong to the event/
|
|
* decision protocol tasks 4.3-4.5 build; this task only extracts state
|
|
* init/fini, mirroring the non-I/O portion of today's sched_init()/
|
|
* sched_fini() (scheduler.c).
|
|
*/
|
|
extern int xce_sched_core_init(xce_sched_core_t*);
|
|
extern void xce_sched_core_fini(xce_sched_core_t*);
|
|
|
|
/* Relocated state-only accessors/mutators (task 4.1): moved out of
|
|
* scheduler.c/schedobj.c, now operating on xce_sched_core_t* -- zero
|
|
* controller I/O, exactly like the rest of this header. */
|
|
extern xce_sched_job_t* sched_get_job(xce_sched_core_t*, uint64_t);
|
|
extern int sched_add_job(xce_sched_core_t*, xce_sched_job_t*);
|
|
extern xce_sched_queue_t* sched_get_queue(xce_sched_core_t*, char*);
|
|
extern int sched_add_queue(xce_sched_core_t*, xce_sched_queue_t*);
|
|
extern xce_sched_worker_t* sched_get_worker(xce_sched_core_t*, char*);
|
|
extern int sched_add_worker(xce_sched_core_t*, xce_sched_worker_t*);
|
|
extern xce_sched_user_t* sched_get_user(xce_sched_core_t*, char*);
|
|
extern int sched_add_user(xce_sched_core_t*, xce_sched_user_t*);
|
|
extern xce_sched_project_t* sched_get_project(xce_sched_core_t*, char*);
|
|
extern int sched_add_project(xce_sched_core_t*, xce_sched_project_t*);
|
|
extern int sched_job_is_ready(xce_sched_job_t*);
|
|
extern int sched_queue_is_ready(xce_sched_queue_t*);
|
|
|
|
/*
|
|
* sched_resolve_job_picker - Req 1 AC 6 / Req 2 AC 5 (scheduler-core spec,
|
|
* task 2.4). Resolves core->job_pick_plugin against the plugin registry
|
|
* (core->plugins, the same plugin_register()/key_find() mechanism every
|
|
* other plugin category uses) and validates the result actually implements
|
|
* the picker MUST hooks (xce_job_picker_iface_t: next_job, begin_session,
|
|
* end_session, on_dispatch, on_pend). Returns 0 and sets *out on success; a
|
|
* negative errno on failure:
|
|
* -EINVAL bad arguments, or core->job_pick_plugin is unset (NULL), or
|
|
* the name IS registered but is not a usable picker
|
|
* -ENOENT the configured name is not registered at all
|
|
* Implemented in schedobj.c. Declared here (task 4.3, scheduler-core spec)
|
|
* -- moved from scheduler.h's xce_scheduler_t* signature to this
|
|
* xce_sched_core_t* signature so xce_core_push_events() (core.c, below) can
|
|
* call it directly without core.c depending on scheduler.h (forbidden --
|
|
* see this header's own boundary comment). Every existing call site already
|
|
* held an xce_scheduler_t* and now passes &sched->core instead; no
|
|
* resolution logic is duplicated.
|
|
*/
|
|
extern int sched_resolve_job_picker(xce_sched_core_t*, xce_sched_plugin_t**);
|
|
|
|
/*
|
|
* sched_account_jobs/sched_charge_account/sched_uncharge_account/
|
|
* sched_commit_job_alloc - resource-in-use accounting (scheduler-sim-
|
|
* decision-parity spec, task 1.1). Relocated from scheduler.c/resource.c's
|
|
* xce_scheduler_t* signature to this xce_sched_core_t* signature, following
|
|
* the exact same precedent sched_resolve_job_picker established above (task
|
|
* 4.3): pure signature relocation, body unchanged -- every reference in
|
|
* these functions' bodies already only touched sched->core.* before this
|
|
* move. Moved so the simulation driver (which has no xce_scheduler_t* of
|
|
* its own, only xce_sched_core_t) can call the same accounting service the
|
|
* normal-mode driver already uses, restoring the resource-in-use accounting
|
|
* parity `.kiro/steering/scheduler-driver-equivalence.md` documents.
|
|
*
|
|
* sched_account_jobs() - re-derives resInUse/sharedInUse from every
|
|
* currently-running job's known resource allocation (job->accounts != NULL)
|
|
* by re-charging each one via sched_charge_account(). Skips a RUN-state job
|
|
* whose allocation is not yet known -- not an error; see
|
|
* scheduler-sim-decision-parity design.md's Recorded-Decision Reconciliation
|
|
* component for why that is the expected state before this call runs.
|
|
* Callers (both drivers) must call this strictly after the session's resval
|
|
* reset and strictly before the picker/allocator are unblocked -- see
|
|
* xce_core_begin_session()/xce_core_signal_go()'s own doc comment below for
|
|
* the full ordering contract this depends on. Was exported (no longer
|
|
* `static`) as of scheduler-core spec task 5.2 for the identical reason;
|
|
* still defined in scheduler.c, only its declaration/signature move here.
|
|
*
|
|
* sched_charge_account()/sched_uncharge_account() - add/remove one job's
|
|
* xce_job_account_t (shared + per-host resource usage) to/from
|
|
* core->sharedInUse and each worker's resInUse.
|
|
*
|
|
* sched_commit_job_alloc() - charges a job's already-resolved allocation
|
|
* (job->accounts) via sched_charge_account(); a no-op (returns 0) if the
|
|
* job's allocation is not yet known. Called by sched_commit_decision()
|
|
* (scheduler.c) immediately after a fresh DISPATCH decision populates
|
|
* job->allocs/job->accounts.
|
|
*/
|
|
extern int sched_account_jobs(xce_sched_core_t*);
|
|
extern int sched_charge_account(xce_sched_core_t*, xce_job_account_t*);
|
|
extern int sched_uncharge_account(xce_sched_core_t*, xce_job_account_t*);
|
|
extern int sched_commit_job_alloc(xce_sched_core_t*, xce_sched_job_t*);
|
|
|
|
/*
|
|
* sched_reconcile_host_resource_usage()/sched_reconcile_job_resource_usage()
|
|
* - resource-accounting.md's reservation-vs-actual-usage policy for a REAL
|
|
* (gauge-style, e.g. "mem") resource: charged = max(reserved, actual),
|
|
* applied as a delta to one worker's resInUse via the existing
|
|
* used+reserved charge formula (sched_update_res_accounts() above), not a
|
|
* full uncharge/recharge of the whole account. See resource.c's own doc
|
|
* comments on each for the full contract (single-host-job scope, the
|
|
* worker-resname-vs-resource-name mapping, the no-rusage-requested no-op
|
|
* case).
|
|
*/
|
|
extern int sched_reconcile_host_resource_usage(xce_sched_core_t*,
|
|
xce_sched_worker_t*,
|
|
xce_res_account_t*, double);
|
|
extern int sched_reconcile_job_resource_usage(xce_sched_core_t*,
|
|
xce_job_account_t*,
|
|
xce_job_rusages_t*);
|
|
|
|
/*
|
|
* xce_core_event_type_t / xce_core_event_t - the core's typed input-event
|
|
* representation (scheduler-core spec, task 4.3, Req 5 AC 4). Exactly the 9
|
|
* event kinds task 4.3 scopes in: BEGIN_ROUND and END_SESSION belong to
|
|
* task 4.4's session-start signal, deliberately not added here.
|
|
*
|
|
* Each event carries a timestamp (`ts`, nanoseconds since epoch) and a
|
|
* type-specific payload (Req 5 AC 4's field tables). Six of the nine kinds
|
|
* additionally drive a picker state-change hook (xce_core_push_events()'s
|
|
* own doc comment, core.c, maps each type to its hook one-for-one);
|
|
* WORKERS_PUSH/RESOURCES_PUSH/RESVALS_FULL/RESVALS_DELTA are state-update
|
|
* only -- "picking is about job ordering, not resource availability"
|
|
* (plugin.h's own Req 1 AC 2 comment).
|
|
*/
|
|
typedef enum {
|
|
XCE_CORE_EVENT_CONFIG_PUSH = 1,
|
|
XCE_CORE_EVENT_JOBS_PUSH,
|
|
XCE_CORE_EVENT_WORKERS_PUSH,
|
|
XCE_CORE_EVENT_RESOURCES_PUSH,
|
|
XCE_CORE_EVENT_RESVALS_FULL,
|
|
XCE_CORE_EVENT_RESVALS_DELTA,
|
|
XCE_CORE_EVENT_JOB_STATE_CHANGE,
|
|
XCE_CORE_EVENT_JOB_UPDATE,
|
|
XCE_CORE_EVENT_QUEUE_UPDATE,
|
|
XCE_CORE_EVENT_JOB_RUSAGE,
|
|
} xce_core_event_type_t;
|
|
|
|
typedef struct {
|
|
xce_core_event_type_t type;
|
|
uint64_t ts; /* nanoseconds since epoch */
|
|
union {
|
|
struct {
|
|
xce_conf4sched_t* config;
|
|
} config_push;
|
|
struct {
|
|
int njobs;
|
|
xce_job_t** jobs;
|
|
} jobs_push;
|
|
struct {
|
|
int nworkers;
|
|
xce_worker_t** workers;
|
|
} workers_push;
|
|
struct {
|
|
int n;
|
|
xce_resource_t** resources;
|
|
} resources_push;
|
|
struct {
|
|
int n;
|
|
xce_resval_t** resvals;
|
|
} resvals_full;
|
|
struct {
|
|
xce_resval_list_t* deltas;
|
|
} resvals_delta;
|
|
struct {
|
|
uint64_t jobid;
|
|
uint32_t old_state;
|
|
uint32_t new_state;
|
|
int32_t exit_code;
|
|
} job_state_change;
|
|
struct {
|
|
uint64_t jobid;
|
|
uint32_t changed_fields;
|
|
xce_job_t* job;
|
|
} job_update;
|
|
struct {
|
|
xce_queue_t* queue;
|
|
} queue_update;
|
|
struct {
|
|
uint64_t jobid;
|
|
xce_job_rusages_t* rusages;
|
|
} job_rusage;
|
|
} u;
|
|
} xce_core_event_t;
|
|
|
|
/*
|
|
* xce_core_push_events - Req 5 AC 4 / AC 8 (scheduler-core spec, task 4.3;
|
|
* design.md "Driver -> Core Protocol", "Core Interface Contract"). Batch-
|
|
* pushes `nevents` events, applying each one's core-state mutation in
|
|
* order and -- for the six event kinds that have one -- synchronously
|
|
* calling the resolved picker's corresponding state-change hook exactly
|
|
* once per event, in the calling (driver) thread, before this call
|
|
* returns. See core.c for the full per-event-type contract and the
|
|
* type-to-hook mapping.
|
|
*
|
|
* Only valid between scheduling sessions (core->session_active == 0):
|
|
* this is what makes calling picker hooks synchronously, from the driver
|
|
* thread, safe -- the picker thread is blocked on cbuf_get(&wbuf) and the
|
|
* allocator thread on cbuf_get(&jbuf), so no session is concurrently
|
|
* reading the state this call mutates. Returns -EBUSY (no event applied)
|
|
* if a session is active; -EINVAL for bad arguments (core NULL, or events
|
|
* NULL with nevents > 0). Per-event failures (e.g. JOB_UPDATE naming an
|
|
* unknown jobid) are logged and do not abort the rest of the batch --
|
|
* same discipline scheduler.c's existing sched_update() dataq loop
|
|
* already follows.
|
|
*/
|
|
extern int xce_core_push_events(xce_sched_core_t*, xce_core_event_t*, int);
|
|
|
|
/*
|
|
* xce_core_set_decision_cb - registers the driver's decision callback +
|
|
* opaque driver context (task 4.4, Req 5 AC 6). May be called at any time
|
|
* (typically once, at driver startup, before the first xce_core_go()) --
|
|
* no session-active restriction, since it only stores two pointers and
|
|
* neither is read outside the decision-drain window between
|
|
* xce_core_go() and xce_core_end_session(). Returns -EINVAL if core is
|
|
* NULL.
|
|
*
|
|
* Deliberately a separate setter rather than an xce_sched_core_init()
|
|
* parameter (design.md's eventual `xce_sched_core_init(core, decision_cb,
|
|
* driver_ctx, prng_seed)` signature, not yet adopted): the sole existing
|
|
* xce_sched_core_init() call site (scheduler.c's sched_init(), still the
|
|
* live production path) has no real driver/callback to pass yet, and this
|
|
* task must not perturb that call site's behavior. A setter keeps
|
|
* xce_sched_core_init() untouched while still letting a future driver (or
|
|
* this task's own test) register a callback on an already-initialized
|
|
* core.
|
|
*/
|
|
extern int xce_core_set_decision_cb(xce_sched_core_t*, xce_core_decision_cb_t,
|
|
void*);
|
|
|
|
/*
|
|
* xce_core_set_cache_clear_cb - registers the driver's per-session
|
|
* cache-clear hook (scheduler-allocation spec, task 5.2 round 2, Req 7.2).
|
|
* Same calling convention as xce_core_set_decision_cb() above (may be
|
|
* called at any time, typically once at driver startup before the first
|
|
* xce_core_begin_session(); returns -EINVAL if core is NULL) -- no
|
|
* `driver_ctx` companion parameter, since xce_core_cache_clear_cb_t's own
|
|
* doc comment (above) explains why the real implementation needs none.
|
|
* Never called by sim_loop.c -- see that typedef's doc comment for why
|
|
* leaving this hook unset is the correct, safe default for simulation
|
|
* mode.
|
|
*/
|
|
extern int xce_core_set_cache_clear_cb(xce_sched_core_t*,
|
|
xce_core_cache_clear_cb_t);
|
|
|
|
/*
|
|
* xce_core_begin_session / xce_core_signal_go - Req 5 AC 5 (task 4.4) split
|
|
* into two exported halves (task 5.2, Req 1 AC 7 / Req 5 AC 9) so a driver
|
|
* can interleave ITS OWN begin-session bookkeeping strictly between the
|
|
* state reset and the picker signal -- specifically
|
|
* plugin_begin_session()/sched_account_jobs() (the OLD per-plugin hook
|
|
* category and per-running-job resource-in-use accounting; both real,
|
|
* load-bearing production behavior). Both now take xce_sched_core_t*
|
|
* directly (sched_account_jobs() since scheduler-sim-decision-parity task
|
|
* 1.1, plugin_begin_session()/plugin_end_session() since bug-1681) but stay
|
|
* outside this pure-core module's reach regardless -- same exclusion
|
|
* rationale core_session.c's own file header comment documents for why
|
|
* core_begin_session_internal() never called them itself.
|
|
*
|
|
* This split exists to preserve a safety property main.c's pre-task-5.2
|
|
* `sched_begin_session()` + `sched_go()` sequence already guaranteed: the
|
|
* picker thread must never observe a session whose resource-in-use
|
|
* accounting has been reset to zero (by the resval resets below) but not
|
|
* yet re-summed from currently running jobs (sched_account_jobs()) -- that
|
|
* would let the picker/allocator threads over-allocate resources already
|
|
* consumed by running jobs. A driver MUST call xce_core_signal_go() only
|
|
* AFTER any such bookkeeping following a successful xce_core_begin_session()
|
|
* completes; see driver.c's xce_sched_driver_run_normal_mode_round() for the
|
|
* real call site and testsuite/test_driver_session_ordering.c for the
|
|
* safety-property proof.
|
|
*
|
|
* xce_core_begin_session() - performs the begin-session state reset
|
|
* (select-eval-ctx, shared resval, per-worker resval/host-ctx). Does NOT
|
|
* signal the picker thread. Returns 0 on success; -EINVAL if core is NULL
|
|
* OR a session is already active (core->session_active != 0) -- the AC's
|
|
* required "distinct error" for the reentrancy-rejection case. On success,
|
|
* core->session_active is already 1 and core->session_start already
|
|
* stamped -- identical to xce_core_go()'s own post-condition at the
|
|
* equivalent point.
|
|
*
|
|
* xce_core_signal_go() - NON-BLOCKING: signals the picker thread via a
|
|
* single `cbuf_put(&core->wbuf, ...)` and returns immediately. Does NOT
|
|
* wait for the picker/allocator to produce any decisions -- draining
|
|
* `core->decisions` and invoking the registered `on_decision` callback is
|
|
* the driver's own responsibility, in the driver's own loop between this
|
|
* call and xce_core_end_session() (design.md's Normal-Mode Scheduling
|
|
* Round: "the drain loop in the driver thread interleaves
|
|
* cbuf_get/on_decision one at a time"). Only valid to call after a
|
|
* successful xce_core_begin_session() for the same session -- calling it
|
|
* without one first (or twice for one session) is a caller bug this
|
|
* function does not itself guard against (xce_core_go() below is the safe
|
|
* composition for callers with no bookkeeping to interleave). Returns
|
|
* -EINVAL if core is NULL.
|
|
*/
|
|
extern int xce_core_begin_session(xce_sched_core_t*);
|
|
extern int xce_core_signal_go(xce_sched_core_t*);
|
|
|
|
/*
|
|
* xce_core_go - Req 5 AC 5 (scheduler-core spec, task 4.4). Starts a
|
|
* scheduling session: NON-BLOCKING -- performs the begin-session state
|
|
* reset (select-eval-ctx, shared resval, per-worker resval/host-ctx) then
|
|
* signals the picker thread via a single `cbuf_put(&core->wbuf, ...)` and
|
|
* returns immediately. Does NOT wait for the picker/allocator to produce
|
|
* any decisions -- draining `core->decisions` and invoking the registered
|
|
* `on_decision` callback is the driver's own responsibility, in the
|
|
* driver's own loop between this call and xce_core_end_session()
|
|
* (design.md's Normal-Mode Scheduling Round: "the drain loop in the
|
|
* driver thread interleaves cbuf_get/on_decision one at a time").
|
|
*
|
|
* As of task 5.2, implemented as `xce_core_begin_session()` immediately
|
|
* followed by `xce_core_signal_go()` -- net behavior, return codes, and
|
|
* the existing test_core_session.c contract are all unchanged; this is a
|
|
* pure decomposition, kept as the simple, safe entry point for any caller
|
|
* (e.g. a future simulation driver) with no begin-session bookkeeping of
|
|
* its own to interleave.
|
|
*
|
|
* Returns 0 on success; -EINVAL if core is NULL OR a session is already
|
|
* active (core->session_active != 0) -- the AC's required "distinct
|
|
* error" for the reentrancy-rejection case.
|
|
*/
|
|
extern int xce_core_go(xce_sched_core_t*);
|
|
|
|
/*
|
|
* xce_core_end_session - Req 5 AC 5 (scheduler-core spec, task 4.4). Called
|
|
* by the driver after it has drained core->decisions down to the
|
|
* end-of-session NULL marker (the marker itself terminates the driver's
|
|
* drain loop -- it is never passed to on_decision; see core_session.c).
|
|
* Performs the end-session cleanup (select-eval-ctx teardown, select-node
|
|
* GC mark/sweep) then clears core->session_active to 0, unblocking the
|
|
* next xce_core_go() call. Returns -EINVAL if core is NULL.
|
|
*/
|
|
extern int xce_core_end_session(xce_sched_core_t*);
|
|
|
|
/*
|
|
* xce_core_now - Req 5 AC 3 (scheduler-core spec, task 4.5): the core's
|
|
* virtualized clock accessor, and the ONLY place inside the scheduler-core
|
|
* module family that calls clock_gettime() (implemented in core_clock.c,
|
|
* deliberately NOT core.c -- see this header's own top-of-file boundary
|
|
* comment and xce_sched_core_t's clock-field doc comment above for why).
|
|
* Computes virtual time via the two-anchor + speed formula from
|
|
* virtual_clock_base/real_clock_base/speed_num/speed_den, reading all four
|
|
* atomically (lock-free, safe from any thread). Returns 0 if core is NULL.
|
|
*/
|
|
extern uint64_t xce_core_now(xce_sched_core_t*);
|
|
|
|
/*
|
|
* xce_core_set_wallclock_anchor - bug-1584 fix. Seeds virtual_clock_base to
|
|
* real epoch time (CLOCK_REALTIME) and real_clock_base to the corresponding
|
|
* CLOCK_MONOTONIC reading, so a subsequent xce_core_now() tracks real
|
|
* wall-clock time from this instant -- the normal-mode driver's counterpart
|
|
* to sim.c's sim_set_virtual_clock() (which anchors virtual_clock_base to a
|
|
* `.replay` file's snapshot ts instead). xce_sched_core_init() alone does
|
|
* NOT do this -- it leaves both anchors at 0, under which xce_core_now()
|
|
* returns raw CLOCK_MONOTONIC-since-boot nanoseconds, not epoch time (see
|
|
* this file's own clock-field doc comment above, and core_clock.c's function
|
|
* header comment, for the full reasoning). Called once by scheduler.c's
|
|
* sched_init(), immediately after xce_sched_core_init() succeeds. No-op if
|
|
* core is NULL.
|
|
*/
|
|
extern void xce_core_set_wallclock_anchor(xce_sched_core_t*);
|
|
|
|
/*
|
|
* xce_core_set_active_for_logging / xce_core_register_tlog_format - Req 5
|
|
* AC 3's tlog integration (task 4.5). See core_clock.c's own file header
|
|
* comment for the full reasoning behind this exact mechanism, including why
|
|
* it deliberately does NOT match AC 3's literal
|
|
* `tlog_reg_format_func(fn, core_ptr)` pseudocode -- that call shape does
|
|
* not exist in the real vendored tlog.h (`tlog_reg_format_func()` takes
|
|
* only the callback, no userptr-binding parameter; userptr is a per-call-
|
|
* site argument that flows through tlog_ext()/tlog_vext(), not something
|
|
* bindable once at registration time).
|
|
*
|
|
* xce_core_set_active_for_logging(core) sets (or, passed NULL, clears) a
|
|
* module-level "active core for logging" pointer that core_clock.c's
|
|
* registered tlog_format_func reads via xce_core_now() instead of
|
|
* info->time. xce_core_register_tlog_format() registers that format
|
|
* function with tlog via the real tlog_reg_format_func(). Neither is
|
|
* called from mylog_init()/main.c by this task -- wiring the live
|
|
* scheduler process over to this mechanism is a later task's job,
|
|
* consistent with tasks 4.1-4.4's additive-only, unwired-until-a-driver-
|
|
* exists pattern.
|
|
*/
|
|
extern void xce_core_set_active_for_logging(xce_sched_core_t*);
|
|
extern int xce_core_register_tlog_format(void);
|
|
|
|
/*
|
|
* SCHED_SNAPSHOT_INTERNAL_VERSION - the current format tag for
|
|
* `sched_snapshot_data_t.internal_state`'s opaque byte layout (task 8.1).
|
|
* Declared here, rather than privately in snapshot.c, because BOTH
|
|
* xce_sched_core_save_snapshot() (snapshot.c, which writes it) and
|
|
* xce_sched_core_load_snapshot() (snapshot_load.c, which version-checks
|
|
* it) need the identical value -- a shared header is the single-source-of-
|
|
* truth home for a constant two separate translation units must agree on
|
|
* bit-for-bit; duplicating it as two independent #defines would risk
|
|
* exactly the kind of silent drift this task's own documentation
|
|
* (snapshot.c's file header, "internal_state contents") goes out of its
|
|
* way to avoid elsewhere. This is an internal implementation detail of the
|
|
* two functions below, not part of any wire/capnp contract -- callers of
|
|
* either function never need to read it directly.
|
|
*/
|
|
#define SCHED_SNAPSHOT_INTERNAL_VERSION (1u)
|
|
|
|
/*
|
|
* xce_sched_core_save_snapshot / xce_sched_core_load_snapshot - Req 5 AC 21
|
|
* / Req 6 AC 5 (scheduler-core spec, task 8.1; design.md "SchedulerCore"
|
|
* Service Interface). Serialize the full core state into (out of) a
|
|
* `sched_snapshot_data_t` (typemap.h, the C mirror of task 7.3's capnp
|
|
* `SnapshotData` struct) -- populate/consume it from `xce_sched_core_t`'s
|
|
* live in-memory state directly; NO capnp encode/decode happens here (that
|
|
* is a later, wire-format task's job). Implemented across three files --
|
|
* snapshot.c (the save direction + the full shared design-decision
|
|
* documentation), snapshot_wire.c (small helpers + save-direction wire
|
|
* builders), and snapshot_load.c (the load direction + picker-rebuild) --
|
|
* split for this project's 800-line-per-file convention; see snapshot.c's
|
|
* own file header comment, "Three-file split", for the full rationale.
|
|
*
|
|
* Both are only valid between scheduling sessions, matching every other
|
|
* core entry point's precondition (xce_core_push_events()'s own doc
|
|
* comment, above): return -EBUSY, with ZERO state touched, if
|
|
* `core->session_active != 0` -- checked BEFORE any mutation, so a rejected
|
|
* call is a true no-op (same pattern xce_core_begin_session()/
|
|
* xce_core_push_events() already establish). Both return -EINVAL for a
|
|
* NULL core or a NULL out/in pointer.
|
|
*
|
|
* xce_sched_core_save_snapshot() populates `*out` from `*core`'s current
|
|
* state (jobs/queues/workers/resources/resvals/users/projects/config/
|
|
* prng_seed/internal_state); see snapshot.c's own file header comment for
|
|
* the full field-by-field mapping and the documented scope decisions
|
|
* (job-level priority and in-flight allocations are NOT captured -- see
|
|
* that comment for why).
|
|
*
|
|
* xce_sched_core_load_snapshot() restores `*core` (which must be freshly
|
|
* initialized via xce_sched_core_init() -- loading into a core that
|
|
* already holds state is not this task's scope, and is not what the
|
|
* driver-side callers design.md documents -- record start, periodic
|
|
* checkpoint, simulation startup -- ever do) from `*in`, THEN rebuilds the
|
|
* resolved picker's private state by walking the just-restored core state
|
|
* and re-invoking the picker's state-change hooks in the same order
|
|
* xce_core_push_events() already establishes for state-change hooks:
|
|
* on_queue_update for every restored queue, THEN on_job_push for every
|
|
* restored job. Picker-private state is deliberately never stored in the
|
|
* snapshot itself (requirements.md Req 5 AC 21's own "Picker state is NOT
|
|
* stored in snapshot" rule) -- this rebuild is what makes post-load state
|
|
* indistinguishable from state built up via normal xce_core_push_events()
|
|
* calls, which is the whole point of the Observable this task proves.
|
|
*/
|
|
extern int xce_sched_core_save_snapshot(xce_sched_core_t*,
|
|
sched_snapshot_data_t*);
|
|
extern int xce_sched_core_load_snapshot(xce_sched_core_t*,
|
|
const sched_snapshot_data_t*);
|
|
|
|
#endif
|