Adds reservation-vs-actual-usage reconciliation for real (gauge-style)
resources, using mem as the concrete case -- documented in the new
resource-accounting.md steering doc.
Policy: charged = max(reserved, actual). A job's rusage[mem=...]
request is an estimate made before it has run, and the real usage the
worker later reports may land above or below it; charging a flat
reservation for the job's whole life either starves other jobs of
memory it never uses (when actual stays below the estimate) or lets
the host be driven into genuine oversubscription (when actual exceeds
it). Available memory for scheduling is then the host's OS-reported
available figure minus, per job, only the UNUSED SLACK of the
reservation (max(0, reserved - used)) -- never the job's actual usage
on top of that, since the OS's own figure already reflects it.
The existing xce_res_account_t{used, reserved, allocated} fields and
sched_update_res_accounts()'s used+reserved charge formula already
implement this exactly, once `used` is set to the OVERAGE beyond the
reservation (max(0, actual - reserved)) rather than left at its
permanent 0.0 default -- nothing previously computed it.
New (resource.c): sched_reconcile_host_resource_usage() sets that
overage on one xce_res_account_t and applies just the DELTA to the
affected worker's resInUse (not a full uncharge/recharge of the whole
job_account, which would needlessly re-touch sibling resources like
"slots" charged in the same account). sched_reconcile_job_resource_usage()
is the wrapper driven from a raw rusage report: maps the worker's real
wire resname ("memory_rss", same worker-resname-vs-scheduler-resource-
name split bug-1735 already established for the controller side) to
the scheduler's "mem" resource, and is a no-op when the job spans more
than one host (the wire report carries no per-host attribution today --
a separate, undocumented-until-now gap, out of this fix's scope) or
never requested rusage[mem=...] in the first place (reserved=0, nothing
to reconcile).
Wired into sched_apply_job_rusage() (schedobj.c), which already
receives every job rusage tick but previously only stored it for
display/plugin hooks with no resource-accounting effect at all.
Compatible with sched_account_jobs()'s existing per-round resInUse
reset+re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation
already reads from, not just a transient pool-level delta that a
round boundary would wipe.
Added link-satisfying stubs to the three white-box test files that
#include schedobj.c directly (test_job_update.c,
test_picker_config_fields.c, test_picker_wiring.c) -- none of them
exercise real job->accounts, so the new call is unreachable there, but
still needs to link. 10 new regression tests in
test_resource_accounting.c cover the core formula (below/at/above the
reservation, delta-only pool updates, sibling-resource isolation) and
the wrapper's scoping rules (resname mapping, multi-host skip,
no-request no-op).
Full scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
633 lines
32 KiB
C
633 lines
32 KiB
C
#if !defined(_CORE_H_)
|
|
|
|
#define _CORE_H_ 1
|
|
|
|
/*
|
|
* core.h - xce_sched_core_t: the scheduler's pure scheduling-state
|
|
* structure (scheduler-core spec, task 4.1, Req 5.1/5.2).
|
|
*
|
|
* Boundary (design.md "SchedulerCore" / "Existing Architecture Analysis"):
|
|
* `xce_scheduler_t` (scheduler.h) splits along the controller-I/O seam into
|
|
* this structure (no `apictx`/`cluster_cfg` -- zero controller I/O, network,
|
|
* or wall-clock dependency) and the driver-level fields that remain on
|
|
* `xce_scheduler_t` itself. Every field below is the SAME field, same name,
|
|
* same type, that lived directly on `xce_scheduler_t` before this task --
|
|
* this is a relocation, not a redesign; `xce_scheduler_t` now embeds this
|
|
* struct as its `core` member instead of declaring these fields itself.
|
|
*
|
|
* Allowed dependencies only (design.md "Allowed Dependencies", Req 5 AC 3):
|
|
* ordq, dataq, cbuf, xlist, zarray, kvmgr, rtti. This header and core.c
|
|
* must NEVER include client.h/netio.h/xceapi.h/delay.h/idgen.h/
|
|
* xce/cluster.h, and must never call a wall-clock function (time(),
|
|
* clock_gettime(), gettimeofday()) -- enforced by
|
|
* testsuite/test_core_isolation.c (source-text scan + an nm(1) undefined-
|
|
* symbol audit of core.c's compiled object, the build-time proof this
|
|
* task's Observable requires).
|
|
*
|
|
* One documented exception (task 4.5, Req 5 AC 3): the core's single
|
|
* sanctioned wall-clock call -- clock_gettime(CLOCK_MONOTONIC) inside
|
|
* xce_core_now() -- and its tlog format-function integration live in
|
|
* core_clock.c, a SEPARATE file from core.c, specifically so
|
|
* test_core_isolation.c's core.c-scoped "zero wall-clock calls" text scan
|
|
* keeps proving what it has always proven (core.c itself never reads real
|
|
* time) without needing to special-case the one legitimate call site AC 3
|
|
* mandates. tlog (tinylog) is likewise an allowed core dependency as of
|
|
* this task -- AC 3's own "Third-party library constraint" section
|
|
* discusses the core registering a tlog format function -- distinct from
|
|
* the forbidden client.h/netio.h/xceapi.h/delay.h/idgen.h/xce/cluster.h
|
|
* list above, none of which tlog is part of.
|
|
*/
|
|
|
|
#include <pthread.h>
|
|
#include <rtti.h>
|
|
#include <stdatomic.h>
|
|
#include <stdint.h>
|
|
|
|
#include "dataq.h"
|
|
#include "decision.h"
|
|
#include "mylock.h"
|
|
#include "plugin.h"
|
|
#include "sched/types.h"
|
|
#include "xce/types.h"
|
|
#include "zarray.h"
|
|
|
|
/* xce_sched_core_t is also forward-declared in plugin.h (picker lifecycle
|
|
* hook parameter type); this typedef and the full struct definition below
|
|
* are the canonical declaration -- same idiom as cluster_t across
|
|
* myconf.h/persist.h/ha.h/cluster_def.h (an identical repeated typedef is
|
|
* valid C11/gnu17, so include order between core.h and plugin.h doesn't
|
|
* matter). */
|
|
typedef struct xce_sched_core_s xce_sched_core_t;
|
|
|
|
/*
|
|
* xce_core_decision_cb_t - the driver-registered decision callback (task
|
|
* 4.4, Req 5 AC 6): `int (*on_decision)(void* driver_ctx, xce_decision_t*
|
|
* decision)`. Invoked once per decision, in allocator-output order, as the
|
|
* driver drains `core->decisions` between `xce_core_go()` and
|
|
* `xce_core_end_session()` -- see xce_core_go()/xce_core_end_session()'s
|
|
* own doc comments (core_session.c) for the full protocol.
|
|
*/
|
|
typedef int (*xce_core_decision_cb_t)(void*, xce_decision_t*);
|
|
|
|
struct xce_sched_core_s {
|
|
mylock_t lock;
|
|
int stop;
|
|
uint64_t nextid;
|
|
xce_conf4sched_t* config;
|
|
uint64_t configver;
|
|
/* Req 2 AC 1 (scheduler-core spec, task 1.3): global picker plugin
|
|
* name. NULL until sched_set_default_job_pick_plugin() (schedobj.c)
|
|
* applies the "builtin_fcfs" default, or a future config-driven
|
|
* assignment sets it explicitly. */
|
|
char* job_pick_plugin;
|
|
var plugins;
|
|
void* seqjob_allocator;
|
|
void* parjob_allocator;
|
|
void* affinity_allocator;
|
|
void* gpu_allocator;
|
|
dataq_t* pend_updates[2];
|
|
var resmgr;
|
|
var ugmgr;
|
|
var qgmgr;
|
|
var pgmgr;
|
|
var hgmgr;
|
|
zarray_t sharedInst;
|
|
zarray_t sharedInUse;
|
|
var jobs;
|
|
var queues;
|
|
var workers;
|
|
var users;
|
|
var projects;
|
|
pthread_t jobpicker;
|
|
pthread_t allocator;
|
|
cbuf_t wbuf; /* action to trigger scheduling */
|
|
cbuf_t jbuf; /* job picked for allocation */
|
|
cbuf_t decisions; /* scheduler decisions */
|
|
/* Req 5 AC 6 (scheduler-core spec, task 4.4): driver-registered
|
|
* decision callback + opaque driver context, set via
|
|
* xce_core_set_decision_cb() (core_session.c). NULL until a driver
|
|
* registers one -- nothing in the still-live sched_go()/main.c path
|
|
* reads these fields today. */
|
|
void* driver_ctx;
|
|
xce_core_decision_cb_t on_decision;
|
|
_Atomic int session_active; /* 1 during scheduling session, 0 between */
|
|
_Atomic int eval_in_progress; /* 1 while eval_select loop runs */
|
|
_Atomic int staging_merge_complete; /* 1 after staging merge completes */
|
|
select_eval_ctx_t* sel_eval_ctx; /* session-level memoisation context; NULL
|
|
between sessions */
|
|
/*
|
|
* Clock virtualization (Req 5 AC 3, scheduler-core spec, task 4.5): the
|
|
* core's ONLY sanctioned time source is xce_core_now() (core_clock.c),
|
|
* which computes virtual time from these four fields via the
|
|
* two-anchor + speed formula:
|
|
*
|
|
* wall_ns = clock_gettime(CLOCK_MONOTONIC) // the one wall-clock
|
|
* // call in the whole
|
|
* // core -- lives in
|
|
* // core_clock.c, NOT
|
|
* // core.c (core.c's own
|
|
* // zero-wall-clock
|
|
* // guarantee, enforced
|
|
* // by
|
|
* // test_core_isolation.c,
|
|
* // stays intact)
|
|
* elapsed = wall_ns - real_clock_base
|
|
* xce_core_now() = virtual_clock_base + (elapsed * speed_num) / speed_den
|
|
*
|
|
* Atomically updated by the driver AS A GROUP before each event
|
|
* dispatch (design.md's State Management note) -- xce_core_now() reads
|
|
* them lock-free, no mutex, safe from both the picker and allocator
|
|
* threads. xce_sched_core_init() seeds these to a neutral
|
|
* (virtual_clock_base=0, real_clock_base=0, speed_num=1, speed_den=1)
|
|
* default -- NOT a real wall-clock anchor, despite this field's older
|
|
* doc comment claiming otherwise (bug-1584): with both anchors at 0,
|
|
* xce_core_now() degrades to raw CLOCK_MONOTONIC-since-boot
|
|
* nanoseconds, not epoch time. A real driver must seed a genuine
|
|
* wall-clock anchor itself -- the normal-mode driver does so via
|
|
* xce_core_set_wallclock_anchor() (scheduler.c's sched_init(), right
|
|
* after xce_sched_core_init()); the simulation driver does so via
|
|
* sim.c's sim_set_virtual_clock(). Only bare-core test fixtures that
|
|
* never call either keep the neutral (0,0,1,1) default.
|
|
*/
|
|
_Atomic uint64_t virtual_clock_base;
|
|
_Atomic uint64_t real_clock_base;
|
|
_Atomic uint64_t speed_num;
|
|
_Atomic uint64_t speed_den;
|
|
/*
|
|
* Session response-time diagnostics (Req 5 AC 12, task 4.5). Plain
|
|
* (non-atomic) fields -- touched only by the driver thread, inside
|
|
* xce_core_go()/xce_core_end_session() (core_session.c), the same
|
|
* single-writer discipline core->sel_eval_ctx above already follows.
|
|
* session_start is stamped at the top of xce_core_go(); the field name
|
|
* matches requirements.md's own pseudocode verbatim
|
|
* ("session_start = xce_core_now()"). session_duration_ns is computed
|
|
* at the top of xce_core_end_session() as xce_core_now() -
|
|
* session_start. Both are diagnostics only -- nothing in the
|
|
* scheduling/decision path reads them. The ReplaySession.summary
|
|
* SessionSummary wire struct these values eventually feed is explicitly
|
|
* OUT of this task's scope (Phase 3, recording/replay-format tasks
|
|
* 7.x) -- this struct only holds the plain numeric tracking.
|
|
*/
|
|
uint64_t session_start;
|
|
uint64_t session_duration_ns;
|
|
/*
|
|
* PRNG seed for deterministic tie-breaking (Req 5 AC 7 / task 8.1). This
|
|
* field did not exist before task 8.1 -- requirements.md's Req 5 AC 7
|
|
* explicitly names `core->prng_seed` ("Any required randomness ... shall
|
|
* use a PRNG seeded by the driver at init time") and design.md's
|
|
* eventual `xce_sched_core_init(core, decision_cb, driver_ctx,
|
|
* prng_seed)` signature (see this header's own doc comment above
|
|
* xce_sched_core_init()) both presuppose it, but no prior task added it
|
|
* or wired a PRNG consumer -- confirmed by grepping this entire spec's
|
|
* source tree for "prng"/"seed" before this task. Adding ONLY the field
|
|
* here, seeded to a deterministic constant (0) by
|
|
* xce_sched_core_init(), keeps this task's scope to "the field exists
|
|
* and round-trips through save_snapshot()/load_snapshot()"
|
|
* (Req 5 AC 21's "PRNG state" checklist row) -- NOT changing
|
|
* xce_sched_core_init()'s signature (still deliberately narrower per
|
|
* this header's doc comment below) and NOT wiring any actual PRNG
|
|
* consumer, since nothing in the codebase performs tie-breaking
|
|
* randomness yet (builtin_fcfs's ordering is fully deterministic via
|
|
* ordq comparators, no PRNG call site exists to seed). A future task
|
|
* that adds real PRNG-based tie-breaking is expected to add its own
|
|
* setter, the same pattern xce_core_set_decision_cb() already
|
|
* establishes for driver_ctx/on_decision.
|
|
*/
|
|
uint64_t prng_seed;
|
|
};
|
|
|
|
/*
|
|
* xce_sched_core_init/xce_sched_core_fini - allocate/initialize, and tear
|
|
* down, exactly the state above. Deliberately narrower than the design's
|
|
* eventual Service Interface (`xce_sched_core_init(core, decision_cb,
|
|
* driver_ctx, prng_seed)`, design.md "SchedulerCore" component) -- the
|
|
* decision callback, driver context, and PRNG seed belong to the event/
|
|
* decision protocol tasks 4.3-4.5 build; this task only extracts state
|
|
* init/fini, mirroring the non-I/O portion of today's sched_init()/
|
|
* sched_fini() (scheduler.c).
|
|
*/
|
|
extern int xce_sched_core_init(xce_sched_core_t*);
|
|
extern void xce_sched_core_fini(xce_sched_core_t*);
|
|
|
|
/* Relocated state-only accessors/mutators (task 4.1): moved out of
|
|
* scheduler.c/schedobj.c, now operating on xce_sched_core_t* -- zero
|
|
* controller I/O, exactly like the rest of this header. */
|
|
extern xce_sched_job_t* sched_get_job(xce_sched_core_t*, uint64_t);
|
|
extern int sched_add_job(xce_sched_core_t*, xce_sched_job_t*);
|
|
extern xce_sched_queue_t* sched_get_queue(xce_sched_core_t*, char*);
|
|
extern int sched_add_queue(xce_sched_core_t*, xce_sched_queue_t*);
|
|
extern xce_sched_worker_t* sched_get_worker(xce_sched_core_t*, char*);
|
|
extern int sched_add_worker(xce_sched_core_t*, xce_sched_worker_t*);
|
|
extern xce_sched_user_t* sched_get_user(xce_sched_core_t*, char*);
|
|
extern int sched_add_user(xce_sched_core_t*, xce_sched_user_t*);
|
|
extern xce_sched_project_t* sched_get_project(xce_sched_core_t*, char*);
|
|
extern int sched_add_project(xce_sched_core_t*, xce_sched_project_t*);
|
|
extern int sched_job_is_ready(xce_sched_job_t*);
|
|
extern int sched_queue_is_ready(xce_sched_queue_t*);
|
|
|
|
/*
|
|
* sched_resolve_job_picker - Req 1 AC 6 / Req 2 AC 5 (scheduler-core spec,
|
|
* task 2.4). Resolves core->job_pick_plugin against the plugin registry
|
|
* (core->plugins, the same plugin_register()/key_find() mechanism every
|
|
* other plugin category uses) and validates the result actually implements
|
|
* the picker MUST hooks (xce_job_picker_iface_t: next_job, begin_session,
|
|
* end_session, on_dispatch, on_pend). Returns 0 and sets *out on success; a
|
|
* negative errno on failure:
|
|
* -EINVAL bad arguments, or core->job_pick_plugin is unset (NULL), or
|
|
* the name IS registered but is not a usable picker
|
|
* -ENOENT the configured name is not registered at all
|
|
* Implemented in schedobj.c. Declared here (task 4.3, scheduler-core spec)
|
|
* -- moved from scheduler.h's xce_scheduler_t* signature to this
|
|
* xce_sched_core_t* signature so xce_core_push_events() (core.c, below) can
|
|
* call it directly without core.c depending on scheduler.h (forbidden --
|
|
* see this header's own boundary comment). Every existing call site already
|
|
* held an xce_scheduler_t* and now passes &sched->core instead; no
|
|
* resolution logic is duplicated.
|
|
*/
|
|
extern int sched_resolve_job_picker(xce_sched_core_t*, xce_sched_plugin_t**);
|
|
|
|
/*
|
|
* sched_account_jobs/sched_charge_account/sched_uncharge_account/
|
|
* sched_commit_job_alloc - resource-in-use accounting (scheduler-sim-
|
|
* decision-parity spec, task 1.1). Relocated from scheduler.c/resource.c's
|
|
* xce_scheduler_t* signature to this xce_sched_core_t* signature, following
|
|
* the exact same precedent sched_resolve_job_picker established above (task
|
|
* 4.3): pure signature relocation, body unchanged -- every reference in
|
|
* these functions' bodies already only touched sched->core.* before this
|
|
* move. Moved so the simulation driver (which has no xce_scheduler_t* of
|
|
* its own, only xce_sched_core_t) can call the same accounting service the
|
|
* normal-mode driver already uses, restoring the resource-in-use accounting
|
|
* parity `.kiro/steering/scheduler-driver-equivalence.md` documents.
|
|
*
|
|
* sched_account_jobs() - re-derives resInUse/sharedInUse from every
|
|
* currently-running job's known resource allocation (job->accounts != NULL)
|
|
* by re-charging each one via sched_charge_account(). Skips a RUN-state job
|
|
* whose allocation is not yet known -- not an error; see
|
|
* scheduler-sim-decision-parity design.md's Recorded-Decision Reconciliation
|
|
* component for why that is the expected state before this call runs.
|
|
* Callers (both drivers) must call this strictly after the session's resval
|
|
* reset and strictly before the picker/allocator are unblocked -- see
|
|
* xce_core_begin_session()/xce_core_signal_go()'s own doc comment below for
|
|
* the full ordering contract this depends on. Was exported (no longer
|
|
* `static`) as of scheduler-core spec task 5.2 for the identical reason;
|
|
* still defined in scheduler.c, only its declaration/signature move here.
|
|
*
|
|
* sched_charge_account()/sched_uncharge_account() - add/remove one job's
|
|
* xce_job_account_t (shared + per-host resource usage) to/from
|
|
* core->sharedInUse and each worker's resInUse.
|
|
*
|
|
* sched_commit_job_alloc() - charges a job's already-resolved allocation
|
|
* (job->accounts) via sched_charge_account(); a no-op (returns 0) if the
|
|
* job's allocation is not yet known. Called by sched_commit_decision()
|
|
* (scheduler.c) immediately after a fresh DISPATCH decision populates
|
|
* job->allocs/job->accounts.
|
|
*/
|
|
extern int sched_account_jobs(xce_sched_core_t*);
|
|
extern int sched_charge_account(xce_sched_core_t*, xce_job_account_t*);
|
|
extern int sched_uncharge_account(xce_sched_core_t*, xce_job_account_t*);
|
|
extern int sched_commit_job_alloc(xce_sched_core_t*, xce_sched_job_t*);
|
|
|
|
/*
|
|
* sched_reconcile_host_resource_usage()/sched_reconcile_job_resource_usage()
|
|
* - resource-accounting.md's reservation-vs-actual-usage policy for a REAL
|
|
* (gauge-style, e.g. "mem") resource: charged = max(reserved, actual),
|
|
* applied as a delta to one worker's resInUse via the existing
|
|
* used+reserved charge formula (sched_update_res_accounts() above), not a
|
|
* full uncharge/recharge of the whole account. See resource.c's own doc
|
|
* comments on each for the full contract (single-host-job scope, the
|
|
* worker-resname-vs-resource-name mapping, the no-rusage-requested no-op
|
|
* case).
|
|
*/
|
|
extern int sched_reconcile_host_resource_usage(xce_sched_core_t*,
|
|
xce_sched_worker_t*,
|
|
xce_res_account_t*, double);
|
|
extern int sched_reconcile_job_resource_usage(xce_sched_core_t*,
|
|
xce_job_account_t*,
|
|
xce_job_rusages_t*);
|
|
|
|
/*
|
|
* xce_core_event_type_t / xce_core_event_t - the core's typed input-event
|
|
* representation (scheduler-core spec, task 4.3, Req 5 AC 4). Exactly the 9
|
|
* event kinds task 4.3 scopes in: BEGIN_ROUND and END_SESSION belong to
|
|
* task 4.4's session-start signal, deliberately not added here.
|
|
*
|
|
* Each event carries a timestamp (`ts`, nanoseconds since epoch) and a
|
|
* type-specific payload (Req 5 AC 4's field tables). Six of the nine kinds
|
|
* additionally drive a picker state-change hook (xce_core_push_events()'s
|
|
* own doc comment, core.c, maps each type to its hook one-for-one);
|
|
* WORKERS_PUSH/RESOURCES_PUSH/RESVALS_FULL/RESVALS_DELTA are state-update
|
|
* only -- "picking is about job ordering, not resource availability"
|
|
* (plugin.h's own Req 1 AC 2 comment).
|
|
*/
|
|
typedef enum {
|
|
XCE_CORE_EVENT_CONFIG_PUSH = 1,
|
|
XCE_CORE_EVENT_JOBS_PUSH,
|
|
XCE_CORE_EVENT_WORKERS_PUSH,
|
|
XCE_CORE_EVENT_RESOURCES_PUSH,
|
|
XCE_CORE_EVENT_RESVALS_FULL,
|
|
XCE_CORE_EVENT_RESVALS_DELTA,
|
|
XCE_CORE_EVENT_JOB_STATE_CHANGE,
|
|
XCE_CORE_EVENT_JOB_UPDATE,
|
|
XCE_CORE_EVENT_QUEUE_UPDATE,
|
|
XCE_CORE_EVENT_JOB_RUSAGE,
|
|
} xce_core_event_type_t;
|
|
|
|
typedef struct {
|
|
xce_core_event_type_t type;
|
|
uint64_t ts; /* nanoseconds since epoch */
|
|
union {
|
|
struct {
|
|
xce_conf4sched_t* config;
|
|
} config_push;
|
|
struct {
|
|
int njobs;
|
|
xce_job_t** jobs;
|
|
} jobs_push;
|
|
struct {
|
|
int nworkers;
|
|
xce_worker_t** workers;
|
|
} workers_push;
|
|
struct {
|
|
int n;
|
|
xce_resource_t** resources;
|
|
} resources_push;
|
|
struct {
|
|
int n;
|
|
xce_resval_t** resvals;
|
|
} resvals_full;
|
|
struct {
|
|
xce_resval_list_t* deltas;
|
|
} resvals_delta;
|
|
struct {
|
|
uint64_t jobid;
|
|
uint32_t old_state;
|
|
uint32_t new_state;
|
|
int32_t exit_code;
|
|
} job_state_change;
|
|
struct {
|
|
uint64_t jobid;
|
|
uint32_t changed_fields;
|
|
xce_job_t* job;
|
|
} job_update;
|
|
struct {
|
|
xce_queue_t* queue;
|
|
} queue_update;
|
|
struct {
|
|
uint64_t jobid;
|
|
xce_job_rusages_t* rusages;
|
|
} job_rusage;
|
|
} u;
|
|
} xce_core_event_t;
|
|
|
|
/*
|
|
* xce_core_push_events - Req 5 AC 4 / AC 8 (scheduler-core spec, task 4.3;
|
|
* design.md "Driver -> Core Protocol", "Core Interface Contract"). Batch-
|
|
* pushes `nevents` events, applying each one's core-state mutation in
|
|
* order and -- for the six event kinds that have one -- synchronously
|
|
* calling the resolved picker's corresponding state-change hook exactly
|
|
* once per event, in the calling (driver) thread, before this call
|
|
* returns. See core.c for the full per-event-type contract and the
|
|
* type-to-hook mapping.
|
|
*
|
|
* Only valid between scheduling sessions (core->session_active == 0):
|
|
* this is what makes calling picker hooks synchronously, from the driver
|
|
* thread, safe -- the picker thread is blocked on cbuf_get(&wbuf) and the
|
|
* allocator thread on cbuf_get(&jbuf), so no session is concurrently
|
|
* reading the state this call mutates. Returns -EBUSY (no event applied)
|
|
* if a session is active; -EINVAL for bad arguments (core NULL, or events
|
|
* NULL with nevents > 0). Per-event failures (e.g. JOB_UPDATE naming an
|
|
* unknown jobid) are logged and do not abort the rest of the batch --
|
|
* same discipline scheduler.c's existing sched_update() dataq loop
|
|
* already follows.
|
|
*/
|
|
extern int xce_core_push_events(xce_sched_core_t*, xce_core_event_t*, int);
|
|
|
|
/*
|
|
* xce_core_set_decision_cb - registers the driver's decision callback +
|
|
* opaque driver context (task 4.4, Req 5 AC 6). May be called at any time
|
|
* (typically once, at driver startup, before the first xce_core_go()) --
|
|
* no session-active restriction, since it only stores two pointers and
|
|
* neither is read outside the decision-drain window between
|
|
* xce_core_go() and xce_core_end_session(). Returns -EINVAL if core is
|
|
* NULL.
|
|
*
|
|
* Deliberately a separate setter rather than an xce_sched_core_init()
|
|
* parameter (design.md's eventual `xce_sched_core_init(core, decision_cb,
|
|
* driver_ctx, prng_seed)` signature, not yet adopted): the sole existing
|
|
* xce_sched_core_init() call site (scheduler.c's sched_init(), still the
|
|
* live production path) has no real driver/callback to pass yet, and this
|
|
* task must not perturb that call site's behavior. A setter keeps
|
|
* xce_sched_core_init() untouched while still letting a future driver (or
|
|
* this task's own test) register a callback on an already-initialized
|
|
* core.
|
|
*/
|
|
extern int xce_core_set_decision_cb(xce_sched_core_t*, xce_core_decision_cb_t,
|
|
void*);
|
|
|
|
/*
|
|
* xce_core_begin_session / xce_core_signal_go - Req 5 AC 5 (task 4.4) split
|
|
* into two exported halves (task 5.2, Req 1 AC 7 / Req 5 AC 9) so a driver
|
|
* can interleave ITS OWN begin-session bookkeeping strictly between the
|
|
* state reset and the picker signal -- specifically
|
|
* plugin_begin_session()/sched_account_jobs() (the OLD per-plugin hook
|
|
* category and per-running-job resource-in-use accounting; both real,
|
|
* load-bearing production behavior). Both now take xce_sched_core_t*
|
|
* directly (sched_account_jobs() since scheduler-sim-decision-parity task
|
|
* 1.1, plugin_begin_session()/plugin_end_session() since bug-1681) but stay
|
|
* outside this pure-core module's reach regardless -- same exclusion
|
|
* rationale core_session.c's own file header comment documents for why
|
|
* core_begin_session_internal() never called them itself.
|
|
*
|
|
* This split exists to preserve a safety property main.c's pre-task-5.2
|
|
* `sched_begin_session()` + `sched_go()` sequence already guaranteed: the
|
|
* picker thread must never observe a session whose resource-in-use
|
|
* accounting has been reset to zero (by the resval resets below) but not
|
|
* yet re-summed from currently running jobs (sched_account_jobs()) -- that
|
|
* would let the picker/allocator threads over-allocate resources already
|
|
* consumed by running jobs. A driver MUST call xce_core_signal_go() only
|
|
* AFTER any such bookkeeping following a successful xce_core_begin_session()
|
|
* completes; see driver.c's xce_sched_driver_run_normal_mode_round() for the
|
|
* real call site and testsuite/test_driver_session_ordering.c for the
|
|
* safety-property proof.
|
|
*
|
|
* xce_core_begin_session() - performs the begin-session state reset
|
|
* (select-eval-ctx, shared resval, per-worker resval/host-ctx). Does NOT
|
|
* signal the picker thread. Returns 0 on success; -EINVAL if core is NULL
|
|
* OR a session is already active (core->session_active != 0) -- the AC's
|
|
* required "distinct error" for the reentrancy-rejection case. On success,
|
|
* core->session_active is already 1 and core->session_start already
|
|
* stamped -- identical to xce_core_go()'s own post-condition at the
|
|
* equivalent point.
|
|
*
|
|
* xce_core_signal_go() - NON-BLOCKING: signals the picker thread via a
|
|
* single `cbuf_put(&core->wbuf, ...)` and returns immediately. Does NOT
|
|
* wait for the picker/allocator to produce any decisions -- draining
|
|
* `core->decisions` and invoking the registered `on_decision` callback is
|
|
* the driver's own responsibility, in the driver's own loop between this
|
|
* call and xce_core_end_session() (design.md's Normal-Mode Scheduling
|
|
* Round: "the drain loop in the driver thread interleaves
|
|
* cbuf_get/on_decision one at a time"). Only valid to call after a
|
|
* successful xce_core_begin_session() for the same session -- calling it
|
|
* without one first (or twice for one session) is a caller bug this
|
|
* function does not itself guard against (xce_core_go() below is the safe
|
|
* composition for callers with no bookkeeping to interleave). Returns
|
|
* -EINVAL if core is NULL.
|
|
*/
|
|
extern int xce_core_begin_session(xce_sched_core_t*);
|
|
extern int xce_core_signal_go(xce_sched_core_t*);
|
|
|
|
/*
|
|
* xce_core_go - Req 5 AC 5 (scheduler-core spec, task 4.4). Starts a
|
|
* scheduling session: NON-BLOCKING -- performs the begin-session state
|
|
* reset (select-eval-ctx, shared resval, per-worker resval/host-ctx) then
|
|
* signals the picker thread via a single `cbuf_put(&core->wbuf, ...)` and
|
|
* returns immediately. Does NOT wait for the picker/allocator to produce
|
|
* any decisions -- draining `core->decisions` and invoking the registered
|
|
* `on_decision` callback is the driver's own responsibility, in the
|
|
* driver's own loop between this call and xce_core_end_session()
|
|
* (design.md's Normal-Mode Scheduling Round: "the drain loop in the
|
|
* driver thread interleaves cbuf_get/on_decision one at a time").
|
|
*
|
|
* As of task 5.2, implemented as `xce_core_begin_session()` immediately
|
|
* followed by `xce_core_signal_go()` -- net behavior, return codes, and
|
|
* the existing test_core_session.c contract are all unchanged; this is a
|
|
* pure decomposition, kept as the simple, safe entry point for any caller
|
|
* (e.g. a future simulation driver) with no begin-session bookkeeping of
|
|
* its own to interleave.
|
|
*
|
|
* Returns 0 on success; -EINVAL if core is NULL OR a session is already
|
|
* active (core->session_active != 0) -- the AC's required "distinct
|
|
* error" for the reentrancy-rejection case.
|
|
*/
|
|
extern int xce_core_go(xce_sched_core_t*);
|
|
|
|
/*
|
|
* xce_core_end_session - Req 5 AC 5 (scheduler-core spec, task 4.4). Called
|
|
* by the driver after it has drained core->decisions down to the
|
|
* end-of-session NULL marker (the marker itself terminates the driver's
|
|
* drain loop -- it is never passed to on_decision; see core_session.c).
|
|
* Performs the end-session cleanup (select-eval-ctx teardown, select-node
|
|
* GC mark/sweep) then clears core->session_active to 0, unblocking the
|
|
* next xce_core_go() call. Returns -EINVAL if core is NULL.
|
|
*/
|
|
extern int xce_core_end_session(xce_sched_core_t*);
|
|
|
|
/*
|
|
* xce_core_now - Req 5 AC 3 (scheduler-core spec, task 4.5): the core's
|
|
* virtualized clock accessor, and the ONLY place inside the scheduler-core
|
|
* module family that calls clock_gettime() (implemented in core_clock.c,
|
|
* deliberately NOT core.c -- see this header's own top-of-file boundary
|
|
* comment and xce_sched_core_t's clock-field doc comment above for why).
|
|
* Computes virtual time via the two-anchor + speed formula from
|
|
* virtual_clock_base/real_clock_base/speed_num/speed_den, reading all four
|
|
* atomically (lock-free, safe from any thread). Returns 0 if core is NULL.
|
|
*/
|
|
extern uint64_t xce_core_now(xce_sched_core_t*);
|
|
|
|
/*
|
|
* xce_core_set_wallclock_anchor - bug-1584 fix. Seeds virtual_clock_base to
|
|
* real epoch time (CLOCK_REALTIME) and real_clock_base to the corresponding
|
|
* CLOCK_MONOTONIC reading, so a subsequent xce_core_now() tracks real
|
|
* wall-clock time from this instant -- the normal-mode driver's counterpart
|
|
* to sim.c's sim_set_virtual_clock() (which anchors virtual_clock_base to a
|
|
* `.replay` file's snapshot ts instead). xce_sched_core_init() alone does
|
|
* NOT do this -- it leaves both anchors at 0, under which xce_core_now()
|
|
* returns raw CLOCK_MONOTONIC-since-boot nanoseconds, not epoch time (see
|
|
* this file's own clock-field doc comment above, and core_clock.c's function
|
|
* header comment, for the full reasoning). Called once by scheduler.c's
|
|
* sched_init(), immediately after xce_sched_core_init() succeeds. No-op if
|
|
* core is NULL.
|
|
*/
|
|
extern void xce_core_set_wallclock_anchor(xce_sched_core_t*);
|
|
|
|
/*
|
|
* xce_core_set_active_for_logging / xce_core_register_tlog_format - Req 5
|
|
* AC 3's tlog integration (task 4.5). See core_clock.c's own file header
|
|
* comment for the full reasoning behind this exact mechanism, including why
|
|
* it deliberately does NOT match AC 3's literal
|
|
* `tlog_reg_format_func(fn, core_ptr)` pseudocode -- that call shape does
|
|
* not exist in the real vendored tlog.h (`tlog_reg_format_func()` takes
|
|
* only the callback, no userptr-binding parameter; userptr is a per-call-
|
|
* site argument that flows through tlog_ext()/tlog_vext(), not something
|
|
* bindable once at registration time).
|
|
*
|
|
* xce_core_set_active_for_logging(core) sets (or, passed NULL, clears) a
|
|
* module-level "active core for logging" pointer that core_clock.c's
|
|
* registered tlog_format_func reads via xce_core_now() instead of
|
|
* info->time. xce_core_register_tlog_format() registers that format
|
|
* function with tlog via the real tlog_reg_format_func(). Neither is
|
|
* called from mylog_init()/main.c by this task -- wiring the live
|
|
* scheduler process over to this mechanism is a later task's job,
|
|
* consistent with tasks 4.1-4.4's additive-only, unwired-until-a-driver-
|
|
* exists pattern.
|
|
*/
|
|
extern void xce_core_set_active_for_logging(xce_sched_core_t*);
|
|
extern int xce_core_register_tlog_format(void);
|
|
|
|
/*
|
|
* SCHED_SNAPSHOT_INTERNAL_VERSION - the current format tag for
|
|
* `sched_snapshot_data_t.internal_state`'s opaque byte layout (task 8.1).
|
|
* Declared here, rather than privately in snapshot.c, because BOTH
|
|
* xce_sched_core_save_snapshot() (snapshot.c, which writes it) and
|
|
* xce_sched_core_load_snapshot() (snapshot_load.c, which version-checks
|
|
* it) need the identical value -- a shared header is the single-source-of-
|
|
* truth home for a constant two separate translation units must agree on
|
|
* bit-for-bit; duplicating it as two independent #defines would risk
|
|
* exactly the kind of silent drift this task's own documentation
|
|
* (snapshot.c's file header, "internal_state contents") goes out of its
|
|
* way to avoid elsewhere. This is an internal implementation detail of the
|
|
* two functions below, not part of any wire/capnp contract -- callers of
|
|
* either function never need to read it directly.
|
|
*/
|
|
#define SCHED_SNAPSHOT_INTERNAL_VERSION (1u)
|
|
|
|
/*
|
|
* xce_sched_core_save_snapshot / xce_sched_core_load_snapshot - Req 5 AC 21
|
|
* / Req 6 AC 5 (scheduler-core spec, task 8.1; design.md "SchedulerCore"
|
|
* Service Interface). Serialize the full core state into (out of) a
|
|
* `sched_snapshot_data_t` (typemap.h, the C mirror of task 7.3's capnp
|
|
* `SnapshotData` struct) -- populate/consume it from `xce_sched_core_t`'s
|
|
* live in-memory state directly; NO capnp encode/decode happens here (that
|
|
* is a later, wire-format task's job). Implemented across three files --
|
|
* snapshot.c (the save direction + the full shared design-decision
|
|
* documentation), snapshot_wire.c (small helpers + save-direction wire
|
|
* builders), and snapshot_load.c (the load direction + picker-rebuild) --
|
|
* split for this project's 800-line-per-file convention; see snapshot.c's
|
|
* own file header comment, "Three-file split", for the full rationale.
|
|
*
|
|
* Both are only valid between scheduling sessions, matching every other
|
|
* core entry point's precondition (xce_core_push_events()'s own doc
|
|
* comment, above): return -EBUSY, with ZERO state touched, if
|
|
* `core->session_active != 0` -- checked BEFORE any mutation, so a rejected
|
|
* call is a true no-op (same pattern xce_core_begin_session()/
|
|
* xce_core_push_events() already establish). Both return -EINVAL for a
|
|
* NULL core or a NULL out/in pointer.
|
|
*
|
|
* xce_sched_core_save_snapshot() populates `*out` from `*core`'s current
|
|
* state (jobs/queues/workers/resources/resvals/users/projects/config/
|
|
* prng_seed/internal_state); see snapshot.c's own file header comment for
|
|
* the full field-by-field mapping and the documented scope decisions
|
|
* (job-level priority and in-flight allocations are NOT captured -- see
|
|
* that comment for why).
|
|
*
|
|
* xce_sched_core_load_snapshot() restores `*core` (which must be freshly
|
|
* initialized via xce_sched_core_init() -- loading into a core that
|
|
* already holds state is not this task's scope, and is not what the
|
|
* driver-side callers design.md documents -- record start, periodic
|
|
* checkpoint, simulation startup -- ever do) from `*in`, THEN rebuilds the
|
|
* resolved picker's private state by walking the just-restored core state
|
|
* and re-invoking the picker's state-change hooks in the same order
|
|
* xce_core_push_events() already establishes for state-change hooks:
|
|
* on_queue_update for every restored queue, THEN on_job_push for every
|
|
* restored job. Picker-private state is deliberately never stored in the
|
|
* snapshot itself (requirements.md Req 5 AC 21's own "Picker state is NOT
|
|
* stored in snapshot" rule) -- this rebuild is what makes post-load state
|
|
* indistinguishable from state built up via normal xce_core_push_events()
|
|
* calls, which is the whole point of the Observable this task proves.
|
|
*/
|
|
extern int xce_sched_core_save_snapshot(xce_sched_core_t*,
|
|
sched_snapshot_data_t*);
|
|
extern int xce_sched_core_load_snapshot(xce_sched_core_t*,
|
|
const sched_snapshot_data_t*);
|
|
|
|
#endif
|