Adds recording-state fields directly to xce_scheduler_t (no new
xce_sched_driver_t type, per task 9.1's precedent). sched_record_start()/
sched_record_stop() implement AC 10: validate via task 9.1's path
validator, open with a TOCTOU-hardened sequence (O_NOFOLLOW, no O_TRUNC,
fstat()+st_nlink==1 check before ftruncate()), snapshot the core, write
ReplayHeader+SnapshotEntry on start, ReplayEvent{endSession} on stop.
driver.c gains boolean-gated recording hooks in
xce_sched_driver_translate_pending_events() (per-event-batch ReplayEvent
writes, new recording_events.c) and xce_sched_driver_run_normal_mode_round()
(per-session ReplaySession writes, new recording_session.c) -- zero I/O/
allocation when inactive.
Two real bugs found and fixed during implementation: bug-1498 (O_TRUNC
would have truncated a hard-linked victim file before the TOCTOU check
could reject it) and bug-1499 (a transient edit dropped
xce_core_signal_go(), causing a 3-way thread deadlock).
test_record_lifecycle.c decodes the full start->events->session->stop
cycle with the real capnp decoders and asserts the exact message
sequence; TOCTOU regression tests use real symlink(2)/link(2) syscalls.
25/25 scheduler suite, clang-format clean.
Adds --record-dir CLI flag and sched_validate_record_filename() with
canonical-path containment (realpath()-based, not textual-prefix) plus
separator/".." rejection before any filesystem access.
Adversarial review found and this task fixed two CWE-59 inode-aliasing
bypasses: a direct hard link inside record_dir aliasing an outside file,
and a compound variant where a symlink pointed at an in-directory file
itself hard-linked outside record_dir. Both closed via st_nlink checks
on the direct entry and the fully-resolved symlink target respectively,
each verified with a live RED->GREEN reproduction. TOCTOU residual risk
documented and deferred to task 9.2 (O_NOFOLLOW + fstat() re-check).
18/18 recording subtests, 24/24 scheduler suite, clang-format clean.
Adds xce_sched_core_save_snapshot()/xce_sched_core_load_snapshot(),
split across snapshot.c/snapshot_wire.c/snapshot_load.c (800-line cap)
along real save/load/shared-helper boundaries. Serializes the full
core state (config, jobs, queues, workers, resources, resvals, users,
projects, prng_seed, internal_state) into/out of sched_snapshot_data_t
(task 7.3's committed C mirror of the capnp SnapshotData struct) as
plain in-memory structures -- no wire encoding here, that's a later
task. After load, rebuilds the picker's private state by re-invoking
its state-change hooks (queue updates, then job pushes) against the
restored core state, matching xce_core_push_events()'s own hook-firing
order; picker-private state itself is never stored in the snapshot.
Adds a new prng_seed field to xce_sched_core_t -- requirements.md/
design.md both presuppose it but no prior task added it. Seeded to 0,
no PRNG consumer wired yet (deliberately out of scope).
Cross-core behavioral-equivalence proven directly: two independent
cores, each with its own registered picker, save on one and load into
a freshly-initialized other, both driven through an identical real
session, decision sequences asserted identical in order.
Found and fixed two real bugs along the way: a missing prng_seed
restore in this task's own new code, and a genuine guard bug in
already-committed core.c (sched_get_user()/sched_get_project()
treating key_find()'s expected -ENOENT as fatal, silently breaking
get-or-create for any new name) that also affects the live production
path (make_sched_job(), JOBS_PUSH) -- a narrowly-scoped two-line fix,
independently traced to the primitive level and confirmed correct by
review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Rewires main.c's live scheduling loop onto the new core-driven
architecture: xce_sched_driver_translate_pending_events() (task 5.1)
-> xce_core_begin_session() -> plugin_begin_session()+sched_account_jobs()
-> xce_core_signal_go() -> decision-drain via the registered
xce_core_decision_cb_t -> plugin_end_session() -> xce_core_end_session(),
all in the new exported xce_sched_driver_run_normal_mode_round()
(driver.c), replacing the old inline sched_update()+sched_begin_session()
+sched_go()+switch+sched_end_session() sequence.
xce_core_go() (task 4.4) is split into xce_core_begin_session()/
xce_core_signal_go() -- a pure decomposition (xce_core_go() itself is
now defined in terms of both, task 4.4's approved test suite passes
unmodified) -- specifically so the driver can interleave
plugin_begin_session()/sched_account_jobs() in the exact safety-critical
window task 4.4 deliberately excluded from pure-core scope: after the
resval reset, strictly before the picker thread is signaled. This is
empirically proven by test_driver_session_ordering.c via real
cross-thread happens-before assertions, not just documented.
xce_sched_driver_on_decision() is the new registered callback:
DISPATCH/PEND forward to the existing sched_publish_dispatch()/
sched_publish_pend_reason(); PREEMPT fails loudly (-ENOSYS, logged)
since no xcecli_preempt() client API exists and nothing in the live
allocator produces one yet (scheduler-preemption's territory).
Adds a --sim CLI flag (parse-only, loud not-yet-implemented failure,
since SimDriver is Phase 3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds driver.h/driver.c: xce_sched_driver_translate_pending_events()
drains core->pend_updates via the existing dataq-swap pattern (same
mechanism sched_update() uses), translates all 10 xce_sched_update_
request_t op types into the matching xce_core_event_t, and pushes the
batch through xce_core_push_events(). Scoped to xce_sched_core_t*
alone -- pend_updates already lives on core since task 4.1, and this
task is translation-only (no decision publishing, no mode-selection,
no recording -- all later work). handler.c (the enqueue side) is
untouched. First task in Group 5 (Core: Normal-Mode Driver).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_now() (new core_clock.c, kept separate from core.c so
test_core_isolation.c's wall-clock scan still enforces zero direct
clock_gettime()/time() calls elsewhere in the core): the two-anchor
plus speed formula (virtual_clock_base + (elapsed * speed_num) /
speed_den), covering all four modes (normal, discrete/real-time/fast-
slow simulation) via four new atomic fields on xce_sched_core_t,
defaulted to normal-mode (1/1 speed) in xce_sched_core_init().
requirements.md's literal tlog_reg_format_func(fn, core_ptr) call
shape doesn't exist in the real vendored tlog library (registration
takes only the callback; userptr is per-call-site, not bindable at
registration) -- resolved with a module-level active-core pointer read
by a new core-scoped format function, not wired into mylog_init()/
main.c yet. xce_core_go()/xce_core_end_session() (task 4.4) now stamp
session_start/compute session_duration_ns via xce_core_now(), with
their existing session-active/reentrancy behavior untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_go()/xce_core_end_session() (new core_session.c):
xce_core_go() is non-blocking, performs begin-session state reset
(select-eval-ctx, resval resets), signals the picker thread via wbuf,
and returns -EINVAL on a reentrant call while a session is already
active (proven a true no-op, not just an early return). Draining
core->decisions and invoking the driver's on_decision callback is the
driver's own responsibility between core_go() and core_end_session()
(confirmed driver-owned, not core-owned, per the Driver <-> Core
protocol). xce_core_set_decision_cb() registers the callback + driver
context as a separate setter, keeping xce_sched_core_init()'s live
sched_init() call site untouched. Adds XCE_DECISION_PREEMPT to
decision.h (jobid/victims[]/reason) alongside the existing DISPATCH/
PEND types. Additive only — sched_go/sched_begin_session/
sched_end_session/sched_allocator/main.c remain untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Defines xce_core_event_type_t/xce_core_event_t (9 event kinds: config,
job, worker, resource, resval full/delta, job state change, job update,
queue update, job rusage) in core.h. Implements xce_core_push_events()
in a new core_events.c (split out of core.c to stay under the 800-line
cap): rejects -EBUSY mid-session, otherwise applies each event's core-
state mutation and fires the corresponding picker hook synchronously,
exactly once per event, before returning. CONFIG_PUSH decomposes the
pushed config's embedded queues/resources/resvals via shared helpers
(also reused by QUEUE_UPDATE's upsert path), mirroring the existing
sched_init_config()/sched_init_resources()/sched_init_sharedInst()/
sched_init_queues(). Narrows sched_resolve_job_picker() to take
xce_sched_core_t* (pure type change, zero behavior change). Purely
additive — not wired into the live scheduling path yet (task 5.x).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Splits xce_scheduler_t along the controller-I/O seam: xce_sched_core_t
(new core.h/core.c) now holds all scheduling state (jobs/queues/workers/
users/projects/resource managers/plugins/three-thread-cbuf pipeline),
while xce_scheduler_t shrinks to {core; apictx; cluster_cfg}. Proven via
an nm-based link-time isolation test (zero netio/xcecli/idgen/delaymgr
symbols reachable from core.c) plus a reproducible wall-clock-call guard.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Registers builtin_fcfs at scheduler startup and makes it the default
job_pick_plugin, replacing the hardcoded "xce_sched_fcfs" literal in
sched_job_picker() with a fail-loud lookup through sched_resolve_job_picker().
The picker thread now branches legacy job_elect_fn plugins (delegated
unchanged, preserving their self-contained wbuf-driven loop) from native
xce_job_picker_iface_t pickers (driven by a new per-session loop), and the
allocator now calls on_dispatch/on_pend for every picker as Req 1 AC 4
requires. Closes task 2.3's flagged picker/allocator concurrency race with a
mutex on fcfs_plugin_data_t, and adds cancel points to next_job().
plugin_register() now adapts an old-style job_elect_fn-only plugin to
xce_job_picker_iface_t via legacy_picker_wrapper_iface (next_job()
forwards to job_elect_fn; other hooks are safe no-ops covered elsewhere
in the existing lifecycle), logging a one-time deprecation warning. A
plugin that already sets picker_iface directly skips the wrapper and
the warning. Reuses the existing plugin_register()/key_find() mechanism
-- no parallel registration system.
Requirements: 1.6, 2.5
Add xce_job_picker_iface_t to plugin.h with all 13 hooks across
lifecycle/state-change/session/post-allocation/optional categories,
matching design.md's literal struct. Documents per-category thread
assignment, next_job() cancel-safety, and begin_session() session-reset
requirements. Forward-declares xce_sched_core_t (built in a later task).
Requirements: 1.2, 1.5, 2.6
bug-1453: remove unused client-side CA-certificate plumbing. C-side:
drop ca_cert_path from xce_cluster_config_t, do_connect()'s ca_cert
param, --ca-cert flags on worker/scheduler, and controller's cert
population (TLS is server-side only, SSL_VERIFY_NONE everywhere on
the client side already, verified zero SSL_CTX_load_verify_locations
calls in the C tree). Go-side: delete xcecli's self-signed cert
TOFU-pinning flow (cmd/auth/cert/*), CACert field, SetRootCAs/
loadRootCAs, and every call site.
bug-1454: fix xcecli_ctx_t leak in scheduler.c sched_init() — the
unconditional initial ctx_init was never freed before the HA
candidate loop re-initialized it on the first iteration.
bug-1455: sched_init()'s memset(sched, 0, ...) was silently wiping
sched->cluster_cfg that main.c had populated from --peers just before
calling sched_init(), so --peers was always ignored and the scheduler
fell back to the hardcoded localhost:9990 single-host default.
Restructured sched_init() to take cluster_cfg as an explicit parameter
(assigned after the memset) instead of relying on caller-side
ordering with a separate sched_set_cluster_cfg() setter, which is now
removed.
bug-1456: revert an unsafe stack-local xce_client_config_t introduced
while fixing bug-1455 — xcecli_ctx_init() stores that pointer as
ctx->conf and client_failover.c dereferences it for the ctx's entire
life (runtime reconnect), so a per-loop-iteration stack copy would
dangle the moment sched_init() returned. Back to mutating the
caller-owned, process-lifetime config in place.
Verified: full ASAN build (all components), meson test suite 9/9 OK,
manual scheduler testsuite 15/15, Go build/vet/test all pass.
5.5 (Req 6 / G-5): the JOB_RUSAGE notify path called the plugin but never
stored the usage data — xce_sched_job_t had no rusage field (UB-6.1 unmet).
- Add xce_job_rusages_t* rusages to xce_sched_job_t.
- New sched_apply_job_rusage() stores the report ON the job (ownership
transferred), replacing any prior via free_JobRusages_ptr; -ENOENT unknown,
-EINVAL NULL. Dequeue NULLs the request pointer on success; kill_sched_job
releases job->rusages.
5.8 (G-3): sched_pend_update_killer left five new-op payloads on empty
break cases — a per-message leak since ownership is transferred on enqueue.
- Free each with the exact generated deep-freer: free_QueueConf_ptr,
free_Resource_ptr[], free_ResourceVal_ptr[], free_ResValList_ptr,
free_JobRusages_ptr. Safe: handlers nullify on enqueue, makeResource copies
via makeInstance, dequeue copies values (no aliasing).
Tests: sched_job_update_test 7/7 (3 update + 4 rusage), regression
sched_test 15/15, ASAN clean (no bad-free/double-free).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The notify schedUpdateJob path shared XCE_SCHED_OP_UPDATE_JOB with the
dataset bulk-load, so the dequeue routed both to sched_update_jobs()
(create/replace) — a notify update to an existing job corrupted/duplicated
live job state and never returned -ENOENT (Req 3 UB-3.1/UB-3.2 unmet;
sched_update_existing_job was dead code copying only state).
- Add distinct op XCE_SCHED_OP_UPDATE_JOB_FIELDS (+ jobfieldsupdate union),
enqueued only by the notify handler, dequeued to sched_update_existing_job.
- Move sched_update_existing_job to schedobj.c; update all mutable fields
(state, priority, queue, resreq) in place; -ENOENT unknown, -EINVAL NULL.
- Dataset bulk-create path unchanged. Killer frees the new payload.
- New white-box test sched_job_update_test (3/3); regression sched_test 15/15.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire up 6 previously-stubbed message handlers in the scheduler:
resource dataset, resval dataset, job update notify, queue update
notify, resval update notify, and job rusage notify. Remove the
unreachable pushmsg branch (vtask_entry rewrites all pushmsg to
datamsg before the handler fires).
Changes by component:
- helper.h/helper.c: add xce_notify_event_name() for symbolic log names
- scheduler.h: add 5 new xce_sched_op_t values and matching union fields
- plugin.h/plugin.c: add plugin_job_rusage_hook_t and plugin_job_rusage_update()
- schedobj.c: add sched_update_existing_queue() with NULL/-ENOENT guards
- scheduler.c: wire 5 new dequeue cases; add sched_update_existing_job()
and static sched_op_name(); emit WARN log on handler failure
- handler.c: implement sched_handle_resources/resvals (dataset) and
sched_handle_sched_update_{job,queue,resvals}/job_rusage (notify);
remove dead sched_handle_jobpush/workerpush and pushmsg branch;
update TRACE logs to use symbolic name helpers
- test.c: 5 new unit tests covering notify event names, enum values,
and queue update NULL guard (15/15 pass)
Worker (xcecd):
- Add xce_cluster_config_t cluster_cfg field to controller_connector_ctx_t
(controller_connector.h includes xce/cluster.h)
- Populate cc_ctx->cluster_cfg from parsed cluster_cfg after xcecd_start()
sets cc_ctx (main.c, removes the P3 placeholder)
- Replace xcecd_join_cluster() single-shot connect with a candidate-iterating
loop: tries each candidate, calls xcecli_stop() before the next attempt,
falls back to the original single-host path when candidates_count == 0
Scheduler (xce_scheduler):
- Add xce_cluster_config_t cluster_cfg field to xce_scheduler_t
(scheduler.h includes xce/cluster.h)
- Add sched_set_cluster_cfg() setter
- Replace the hardcoded xcecli_start/xcecli_join pair in sched_init() with
the same candidate-iterating pattern; original single-host path preserved
- main.c calls sched_set_cluster_cfg() before sched_init() when --peers given
(removes the P3 placeholder comment)
checking_connection() already calls xcecd_join_cluster() on retry, giving
the worker free reconnect-after-failover behaviour for no additional code.
- Fix issue to xce_get() api cause client side crash
- add parse resreq section
- add rusage parser
- parse resreq with basic support alternative & compound, still miss select
