Chain the already-shipped recording (task 9.2) and sim-output (tasks
11.1-11.5) pipelines: record a synthetic multi-queue session through the
normal-mode driver, replay it through the simulation driver with
--sim-output, and assert the two .replay files' decisions match in full
structural content. This is the spec's central validation contract (Req 5
AC 7.1) -- if normal and simulation modes diverge, the core/driver
separation is broken.
Honestly scoped to PEND-only decisions: sim_loop.c's picker thread
(task 11.2) has no code path that can emit a live DISPATCH today, a
pre-existing gap that belongs to the not-yet-designed scheduler-allocation
spec, not something this task can close.
Add sim_record.c (scheduler_t-free .replay writer: ReplayHeader + loaded
SnapshotEntry once, then per-event pass-through and per-session
ReplaySession finalization) and wire --sim-alt-events event-source
selection into sim_loop.c's drain loop, coexisting with task 11.2's
stdout default and task 11.4's --sim-verify. Extract scheduler_t-free
halves out of recording_session.c's begin/end so sim-side code can reuse
the decision-encoding path without gaining an xce_scheduler_t* dependency,
preserving Req 5 AC 17 isolation. Per Req 5 AC 16/19.
Add sim_verify.c (sim_decision_eq/sim_log_mismatch: real structural
comparison of dispatch/pend decision payloads, not a jobid shortcut) and
wire it into sim_loop.c's ReplaySession drain loop via decode_ReplaySession_ptr()
(first real consumer of the generated capnp decode path). --sim-verify now
detects and reports genuine decision divergence with logical-time + detail,
per Req 5 AC 15/18.
Three-layer regression guard (test_sim_isolation.c) proving sim.h/sim.c/
sim_loop.c hold no controller-facing client/cluster-config/capnp-socket
state: source-text scan, struct-body field scan, and nm(1) link-time
undefined-symbol audit of a new isolated sched_sim_isolation_lib target.
Add sim_loop.c implementing sched_sim_driver_run(): decode .replay
messages in file order, jump the virtual clock to each ReplayEvent's
exact timestamp as it's queued, and run a real push-events/go/drain/
end-session cycle on each ReplaySession boundary (AC 14, AC 18).
Task 11.1 left no picker registered against the isolated core and no
thread backing its cbufs, which deadlocks the decision drain the moment
a real event loop exists. Add plugin_register_core() (registers a
picker against a bare xce_sched_core_t*, no xce_scheduler_t* required)
and extract builtin_fcfs_new_plugin() from register_builtin_fcfs() as a
behavior-preserving factory, plus a picker-driving thread for the sim
driver. Since real resource allocation is unreachable without a live
xce_scheduler_t* (scheduler-allocation spec's job), the sim picker
thread conservatively PENDs every candidate rather than dispatching.
28/28 scheduler suite, 32/32 whole-project tests green.
Add sim.h/sim.c implementing AC 13 steps 1-4: open+validate ReplayHeader,
locate the most recent SnapshotEntry at/before --sim-start-ts, load it via
xce_sched_core_load_snapshot(), and set the virtual clock anchors. Wire
all seven simulation CLI flags into main.c's task-5.2 mode-selection
branch. Event loop itself (step 5) deferred to task 11.2. Fixes two real
bugs found during TDD: a SnapshotEntry_get_ts() use-after-free and a
pend_updates NULL-deref in xce_sched_core_fini() for bare-core callers.
27/27 scheduler suite green, full project build clean.
Extends test_admin_request.c with 3 tests closing the gap between what
task 9.4's own suite already proved through scheduler_req_handler() and
what this task's text specifically asks for: a ".."-traversal rejection
and a real symlink(2)-escape rejection, both submitted as genuine
SchedAdminRequests (not internal calls), and a disk-full simulation
(fflush + setvbuf(_IONBF) + close(fd), matching task 9.3's technique)
proving sched_record_abort_on_error()'s cleanup is visible through a
subsequent real recordStatus request and that recording recovers cleanly
afterward.
Hard-link-aliasing coverage deliberately not re-added through the admin
path -- task 9.1/9.2 already exhaustively prove that rejection branch,
and the routing plumbing this task exists to prove is already retired by
the dotdot/symlink cases.
17/17 in sched_admin_request_test, 26/26 scheduler suite, clang-format
clean. Closes out Group 9 (Core: Recording Control).
Implements the scheduler's first request-reply opcode (OPCode_schedAdmin).
sched_handle_admin_request() (handler.c) decodes SchedAdminRequest and
dispatches by action: recordStatus answers synchronously (pure read);
recordStart/recordStop are deferred onto the existing pend_updates queue
(XCE_SCHED_OP_ADMIN_RECORD) since sched_record_start()'s snapshot call
requires the session_active==0 precondition only guaranteed on the driver
thread, not the async request-handling thread. driver.c's
xce_sched_driver_translate_pending_events() performs the deferred action
and sends the reply via a new xcecli_reply_schedadmin() (client.c/client.h),
correlated by seq copied before the original request is freed.
Adversarial review found a genuine cross-thread race: sched_record_status()
read driver-owned recording state (including a pointer the driver thread
frees) from the vtask thread with no lock -- the codebase's first such
race, missed by ASan (doesn't detect races). Fixed with a dedicated
record_lock scoped strictly to in-memory field access, never spanning file
I/O; sched_record_status() now strdup()s under the lock instead of
returning a borrowed live pointer. Verified with a real two-thread
lock-contention test (independently confirmed non-vacuous by removing the
lock and observing the test fail) plus a deterministic use-after-free
reproduction.
26/26 scheduler suite, clang-format clean.
Periodic checkpointing hooks into sched_record_write_events(): once per
call, when record_snapshot_interval > 0 and record_event_count crosses a
multiple of it, writes a fresh SnapshotEntry via the same
write_snapshot_entry() helper sched_record_start() already uses (made
non-static, zero duplication). Safe from the session_active precondition
because this call site is only ever reached from
xce_sched_driver_translate_pending_events(), which always runs strictly
between sessions.
Confirmed the write-failure error-handling half (AC 10) was already
complete from task 9.2's sched_record_abort_on_error() -- added the
missing end-to-end test using a genuine closed-fd write failure rather
than a mocked return code, proving the driver keeps processing sessions
afterward.
25/25 scheduler suite, clang-format clean. Two non-blocking follow-ups
logged in tasks.md: --record-snapshot-interval has no CLI wiring yet
(AC 11's own struct-field mechanism is fully tested, but nothing sets it
in production), and test_record_lifecycle.c has grown past this suite's
prior size precedent with a natural future split point identified.
Adds recording-state fields directly to xce_scheduler_t (no new
xce_sched_driver_t type, per task 9.1's precedent). sched_record_start()/
sched_record_stop() implement AC 10: validate via task 9.1's path
validator, open with a TOCTOU-hardened sequence (O_NOFOLLOW, no O_TRUNC,
fstat()+st_nlink==1 check before ftruncate()), snapshot the core, write
ReplayHeader+SnapshotEntry on start, ReplayEvent{endSession} on stop.
driver.c gains boolean-gated recording hooks in
xce_sched_driver_translate_pending_events() (per-event-batch ReplayEvent
writes, new recording_events.c) and xce_sched_driver_run_normal_mode_round()
(per-session ReplaySession writes, new recording_session.c) -- zero I/O/
allocation when inactive.
Two real bugs found and fixed during implementation: bug-1498 (O_TRUNC
would have truncated a hard-linked victim file before the TOCTOU check
could reject it) and bug-1499 (a transient edit dropped
xce_core_signal_go(), causing a 3-way thread deadlock).
test_record_lifecycle.c decodes the full start->events->session->stop
cycle with the real capnp decoders and asserts the exact message
sequence; TOCTOU regression tests use real symlink(2)/link(2) syscalls.
25/25 scheduler suite, clang-format clean.
Adds --record-dir CLI flag and sched_validate_record_filename() with
canonical-path containment (realpath()-based, not textual-prefix) plus
separator/".." rejection before any filesystem access.
Adversarial review found and this task fixed two CWE-59 inode-aliasing
bypasses: a direct hard link inside record_dir aliasing an outside file,
and a compound variant where a symlink pointed at an in-directory file
itself hard-linked outside record_dir. Both closed via st_nlink checks
on the direct entry and the fully-resolved symlink target respectively,
each verified with a live RED->GREEN reproduction. TOCTOU residual risk
documented and deferred to task 9.2 (O_NOFOLLOW + fstat() re-check).
18/18 recording subtests, 24/24 scheduler suite, clang-format clean.
Task 8.1's own -EBUSY guard test only proved rejection against a
synthetic single-thread session_active flag flip, with a comment
misattributing the technique to test_core_full_cycle.c's actually-
genuine-session precedent. Real gap, not busywork: xce_core_go()
publishes session_active via memory_order_release while the snapshot
guards read it via plain atomic_load() -- the synthetic same-thread
test never exercised any real cross-thread visibility of that flag.
Adds a genuine driver-driven session test (real picker thread + real
xce_core_go()) proving both save and load reject with -EBUSY during
the same active session, leaving state genuinely unchanged (sentinel
bytes for save; a distinguishable non-existent queue + unchanged
candidate count for load). Zero production changes -- task 8.1's
guards were already correct.
Closes out Group 8 (Core: Snapshot Serialization) entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_sched_core_save_snapshot()/xce_sched_core_load_snapshot(),
split across snapshot.c/snapshot_wire.c/snapshot_load.c (800-line cap)
along real save/load/shared-helper boundaries. Serializes the full
core state (config, jobs, queues, workers, resources, resvals, users,
projects, prng_seed, internal_state) into/out of sched_snapshot_data_t
(task 7.3's committed C mirror of the capnp SnapshotData struct) as
plain in-memory structures -- no wire encoding here, that's a later
task. After load, rebuilds the picker's private state by re-invoking
its state-change hooks (queue updates, then job pushes) against the
restored core state, matching xce_core_push_events()'s own hook-firing
order; picker-private state itself is never stored in the snapshot.
Adds a new prng_seed field to xce_sched_core_t -- requirements.md/
design.md both presuppose it but no prior task added it. Seeded to 0,
no PRNG consumer wired yet (deliberately out of scope).
Cross-core behavioral-equivalence proven directly: two independent
cores, each with its own registered picker, save on one and load into
a freshly-initialized other, both driven through an identical real
session, decision sequences asserted identical in order.
Found and fixed two real bugs along the way: a missing prng_seed
restore in this task's own new code, and a genuine guard bug in
already-committed core.c (sched_get_user()/sched_get_project()
treating key_find()'s expected -ENOENT as fatal, silently breaking
get-or-create for any new name) that also affects the live production
path (make_sched_job(), JOBS_PUSH) -- a narrowly-scoped two-line fix,
independently traced to the primitive level and confirmed correct by
review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Rewires main.c's live scheduling loop onto the new core-driven
architecture: xce_sched_driver_translate_pending_events() (task 5.1)
-> xce_core_begin_session() -> plugin_begin_session()+sched_account_jobs()
-> xce_core_signal_go() -> decision-drain via the registered
xce_core_decision_cb_t -> plugin_end_session() -> xce_core_end_session(),
all in the new exported xce_sched_driver_run_normal_mode_round()
(driver.c), replacing the old inline sched_update()+sched_begin_session()
+sched_go()+switch+sched_end_session() sequence.
xce_core_go() (task 4.4) is split into xce_core_begin_session()/
xce_core_signal_go() -- a pure decomposition (xce_core_go() itself is
now defined in terms of both, task 4.4's approved test suite passes
unmodified) -- specifically so the driver can interleave
plugin_begin_session()/sched_account_jobs() in the exact safety-critical
window task 4.4 deliberately excluded from pure-core scope: after the
resval reset, strictly before the picker thread is signaled. This is
empirically proven by test_driver_session_ordering.c via real
cross-thread happens-before assertions, not just documented.
xce_sched_driver_on_decision() is the new registered callback:
DISPATCH/PEND forward to the existing sched_publish_dispatch()/
sched_publish_pend_reason(); PREEMPT fails loudly (-ENOSYS, logged)
since no xcecli_preempt() client API exists and nothing in the live
allocator produces one yet (scheduler-preemption's territory).
Adds a --sim CLI flag (parse-only, loud not-yet-implemented failure,
since SimDriver is Phase 3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds driver.h/driver.c: xce_sched_driver_translate_pending_events()
drains core->pend_updates via the existing dataq-swap pattern (same
mechanism sched_update() uses), translates all 10 xce_sched_update_
request_t op types into the matching xce_core_event_t, and pushes the
batch through xce_core_push_events(). Scoped to xce_sched_core_t*
alone -- pend_updates already lives on core since task 4.1, and this
task is translation-only (no decision publishing, no mode-selection,
no recording -- all later work). handler.c (the enqueue side) is
untouched. First task in Group 5 (Core: Normal-Mode Driver).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds test_core_full_cycle.c: drives xce_core_push_events() -> xce_core_go()
-> decision drain -> xce_core_end_session() end-to-end against a real
picker pthread (running the real xce_job_picker_iface_t hooks on a
test-local stub picker) and a main-thread stand-in allocator, since
sched_allocator() is xce_scheduler_t*-coupled and out of pure-core
scope. Proves both preconditions (core_go reentrancy, push_events
mid-session) reject against a genuinely active session and recover
once it ends. Closes out group 4 (Foundation: Scheduler Core
Extraction) -- tasks 4.1-4.6 all complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_now() (new core_clock.c, kept separate from core.c so
test_core_isolation.c's wall-clock scan still enforces zero direct
clock_gettime()/time() calls elsewhere in the core): the two-anchor
plus speed formula (virtual_clock_base + (elapsed * speed_num) /
speed_den), covering all four modes (normal, discrete/real-time/fast-
slow simulation) via four new atomic fields on xce_sched_core_t,
defaulted to normal-mode (1/1 speed) in xce_sched_core_init().
requirements.md's literal tlog_reg_format_func(fn, core_ptr) call
shape doesn't exist in the real vendored tlog library (registration
takes only the callback; userptr is per-call-site, not bindable at
registration) -- resolved with a module-level active-core pointer read
by a new core-scoped format function, not wired into mylog_init()/
main.c yet. xce_core_go()/xce_core_end_session() (task 4.4) now stamp
session_start/compute session_duration_ns via xce_core_now(), with
their existing session-active/reentrancy behavior untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_go()/xce_core_end_session() (new core_session.c):
xce_core_go() is non-blocking, performs begin-session state reset
(select-eval-ctx, resval resets), signals the picker thread via wbuf,
and returns -EINVAL on a reentrant call while a session is already
active (proven a true no-op, not just an early return). Draining
core->decisions and invoking the driver's on_decision callback is the
driver's own responsibility between core_go() and core_end_session()
(confirmed driver-owned, not core-owned, per the Driver <-> Core
protocol). xce_core_set_decision_cb() registers the callback + driver
context as a separate setter, keeping xce_sched_core_init()'s live
sched_init() call site untouched. Adds XCE_DECISION_PREEMPT to
decision.h (jobid/victims[]/reason) alongside the existing DISPATCH/
PEND types. Additive only — sched_go/sched_begin_session/
sched_end_session/sched_allocator/main.c remain untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Defines xce_core_event_type_t/xce_core_event_t (9 event kinds: config,
job, worker, resource, resval full/delta, job state change, job update,
queue update, job rusage) in core.h. Implements xce_core_push_events()
in a new core_events.c (split out of core.c to stay under the 800-line
cap): rejects -EBUSY mid-session, otherwise applies each event's core-
state mutation and fires the corresponding picker hook synchronously,
exactly once per event, before returning. CONFIG_PUSH decomposes the
pushed config's embedded queues/resources/resvals via shared helpers
(also reused by QUEUE_UPDATE's upsert path), mirroring the existing
sched_init_config()/sched_init_resources()/sched_init_sharedInst()/
sched_init_queues(). Narrows sched_resolve_job_picker() to take
xce_sched_core_t* (pure type change, zero behavior change). Purely
additive — not wired into the live scheduling path yet (task 5.x).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Hardcodes jbuf's cbuf capacity to the literal 1 in xce_sched_core_init()
(no parameter path, no local variable to drift), backed by a regression
test. Audits every kvmgr iteration reachable from the scheduling loop
(jobs/queues/users/projects): only jobs is ever iterated, via xzarray's
deterministic ascending-slot walk with no process-random seeding —
findings recorded durably as a comment above xce_sched_core_init().
core->workers iteration-order stability is flagged as an open follow-up
(scheduler-allocation's territory), not this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Splits xce_scheduler_t along the controller-I/O seam: xce_sched_core_t
(new core.h/core.c) now holds all scheduling state (jobs/queues/workers/
users/projects/resource managers/plugins/three-thread-cbuf pipeline),
while xce_scheduler_t shrinks to {core; apictx; cluster_cfg}. Proven via
an nm-based link-time isolation test (zero netio/xcecli/idgen/delaymgr
symbols reachable from core.c) plus a reproducible wall-clock-call guard.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Closes out group 2 (Core: Builtin FCFS Picker). Confirms builtin_fcfs's
ordering matches legacy fcfs_elect_job (byte-identical comparators, same
sched_queue_is_ready/sched_job_is_ready gates, provably equivalent jobid
sequence), confirms tasks 1.1-2.5 left every pre-existing test file
unmodified, and confirms resreq immutability, pend-reason factory-only
allocation, and select_eval_ctx_t memoization all hold across every file
this spec has touched so far. Adds a durable regression lock for the
pend-reason factory invariant (no direct calloc(pendreason_t) outside
make_pendreason()).
Task 2.4 already made sched_job_picker() delegate entirely to the picker
interface with zero direct queue/job traversal, satisfying Req 1 AC 1/AC 7
as a byproduct. This adds a source-text regression test locking that
property in place: asserts sched_job_picker()'s body contains none of the
forbidden direct-traversal symbols and does contain the expected
picker-interface call sites.
Registers builtin_fcfs at scheduler startup and makes it the default
job_pick_plugin, replacing the hardcoded "xce_sched_fcfs" literal in
sched_job_picker() with a fail-loud lookup through sched_resolve_job_picker().
The picker thread now branches legacy job_elect_fn plugins (delegated
unchanged, preserving their self-contained wbuf-driven loop) from native
xce_job_picker_iface_t pickers (driven by a new per-session loop), and the
allocator now calls on_dispatch/on_pend for every picker as Req 1 AC 4
requires. Closes task 2.3's flagged picker/allocator concurrency race with a
mutex on fcfs_plugin_data_t, and adds cancel points to next_job().
Adds begin_session/next_job/end_session/on_dispatch/on_pend, driving a
dual-cursor walk (queue then job) over builtin_fcfs's private ordq_t
state. next_job() returns candidates in (queue priority desc, job
priority desc, queue_time asc) order and NULL once exhausted;
on_dispatch removes permanently, on_pend is a true no-op.
Review flagged an unsynchronized cross-thread access risk between
next_job() (picker thread) and on_dispatch/on_pend (allocator thread)
once wired live in task 2.4 — recorded in tasks.md Implementation Notes
for that task to close before registering builtin_fcfs as the default.
Requirements: 3.2, 3.4, 3.6
Adds on_queue_update, on_job_push, on_job_state_change, and on_job_update
to builtin_fcfs, driving the picker's private ordq_t candidate sets
directly (no core/sched queries), matching fcfs.c's existing ordering
semantics on the new finer-grained hook interface.
Requirements: 3.2
New src/scheduler/src/plugins/builtin_fcfs.c implements
builtin_fcfs_init()/fini() and the private fcfs_plugin_data_t (ordq_t
queues, per-queue ordq_t pendJobs) that all later builtin_fcfs hooks
build on. fcfs.c (today's live job_elect_fn-based picker) is untouched
-- still production, until task 2.4 flips the registration.
picker == xce_sched_plugin_t* calling convention (private state stored
in plugin->data) confirmed consistent with task 1.2's legacy wrapper.
Requirements: 3.1 (context), 3.3
plugin_register() now adapts an old-style job_elect_fn-only plugin to
xce_job_picker_iface_t via legacy_picker_wrapper_iface (next_job()
forwards to job_elect_fn; other hooks are safe no-ops covered elsewhere
in the existing lifecycle), logging a one-time deprecation warning. A
plugin that already sets picker_iface directly skips the wrapper and
the warning. Reuses the existing plugin_register()/key_find() mechanism
-- no parallel registration system.
Requirements: 1.6, 2.5
Add xce_job_picker_iface_t to plugin.h with all 13 hooks across
lifecycle/state-change/session/post-allocation/optional categories,
matching design.md's literal struct. Documents per-category thread
assignment, next_job() cancel-safety, and begin_session() session-reset
requirements. Forward-declares xce_sched_core_t (built in a later task).
Requirements: 1.2, 1.5, 2.6
controller.capnp declared `struct Data $C.mapname("xce_data_t")` (an
OPCode_get query-result tagged union) at file scope, which shadows
Cap'n Proto's builtin Data blob type for every later `: Data` field in
the same file. This silently miscompiled 7 fields across 6 structs as
a struct-pointer type instead of a raw byte blob: UpgradeConfigRequest
.sha256/.bytes, ReplConfigReply.sha256/.bytes, ReplConfigPushRequest
.sha256/.bytes, ReplEnvelope.payload, SnapshotTransfer's chunk
payload, WalCatchupEntry.payload/.batchPayloads, and CtrlHelloRequest
.key. Production C code had already routed around it for years via
hand-rolled raw capn_ptr manipulation (see request.c's own comments);
nothing before now needed the generated codec/accessors to actually
work for these fields.
Renamed the local struct to XceData (capnp-level name only; the
$C.mapname("xce_data_t") C typedef is unchanged) and explicitly
repointed getdata/pushmsg at XceData by name, since those two fields
correctly want the local tagged union. Updated every derived
generated-identifier reference across 19 hand-written files spanning
controller, lib, scheduler, and the Go apiserver/xcecli CGo bindings.
Fixed 5 call sites in lib/request.c that had been typed against the
old workaround alias but now legitimately receive the correct
capn_data type (a pure type-name swap -- Data_ptr/XceData_ptr and
capn_data share the identical {capn_ptr p} layout).
Verified via full rebuild+test: lib (1/1), controller (3/3, incl. the
job-array push regression tests), scheduler (15/15), Go apiserver +
xcecli (all packages green), and a direct pycapnp round-trip proving
UpgradeConfigRequest.sha256/.bytes now encode/decode correctly.
Bug: bug-1461 (.wolf/buglog.json). Unblocks admin-config-tool task 3.3.
bug-1453: remove unused client-side CA-certificate plumbing. C-side:
drop ca_cert_path from xce_cluster_config_t, do_connect()'s ca_cert
param, --ca-cert flags on worker/scheduler, and controller's cert
population (TLS is server-side only, SSL_VERIFY_NONE everywhere on
the client side already, verified zero SSL_CTX_load_verify_locations
calls in the C tree). Go-side: delete xcecli's self-signed cert
TOFU-pinning flow (cmd/auth/cert/*), CACert field, SetRootCAs/
loadRootCAs, and every call site.
bug-1454: fix xcecli_ctx_t leak in scheduler.c sched_init() — the
unconditional initial ctx_init was never freed before the HA
candidate loop re-initialized it on the first iteration.
bug-1455: sched_init()'s memset(sched, 0, ...) was silently wiping
sched->cluster_cfg that main.c had populated from --peers just before
calling sched_init(), so --peers was always ignored and the scheduler
fell back to the hardcoded localhost:9990 single-host default.
Restructured sched_init() to take cluster_cfg as an explicit parameter
(assigned after the memset) instead of relying on caller-side
ordering with a separate sched_set_cluster_cfg() setter, which is now
removed.
bug-1456: revert an unsafe stack-local xce_client_config_t introduced
while fixing bug-1455 — xcecli_ctx_init() stores that pointer as
ctx->conf and client_failover.c dereferences it for the ctx's entire
life (runtime reconnect), so a per-loop-iteration stack copy would
dangle the moment sched_init() returned. Back to mutating the
caller-owned, process-lifetime config in place.
Verified: full ASAN build (all components), meson test suite 9/9 OK,
manual scheduler testsuite 15/15, Go build/vet/test all pass.
sched_handle_dataset()'s Data_data_jobArray case never freed
dataset->rows (the decoded xce_job_submit_t*) after sched_on_job_array()
consumed it -- free_Dataset() is an empty no-op for every table type, so
nothing else owned this call. One bounded leak per delivered array push.
Free it via the existing free_JobSubmit_ptr() teardown on both the
success and error path, mirroring the sibling list-shaped cases'
ownership-transfer convention.
Rewrote test_job_array_expand.c's init_job_array() fixture to
heap-allocate + strdup its fields (matching the real wire-decode shape)
since freeing the old stack/string-literal fixture crashed. Redelivery
test now builds a fresh descriptor per delivery, matching how a real
redelivery is always a freshly-decoded wire message.
Closes task 7.3 (job-array-single-wal-record), the last of the three
post-GO residual gaps reopened 2026-07-16.
Cross-daemon integration test, test-only, spanning the scheduler (C)
and API-server (Go/CGO) receive paths built in tasks 5.1-5.4/5.3b.
Drives real xce_encode()/xce_decode() wire bytes through the actual
production dispatch chain on both daemons -- extract_data() ->
scheduler_req_handler() -> sched_handle_dataset() ->
sched_on_job_array() and go_push_handler() -> handleDataset() ->
cJobArrayToModels()/dispatchJobArray() -- skipping only vtask_entry()'s
~10 lines of static glue, mirrored verbatim in test code (the
if (callnext) treats-any-nonzero-as-proceed behavior that caused
5.3b's bug is reproduced exactly).
Redelivery is tested via genuinely independent fresh decodes of
separately-encoded bytes, not a decode-once-call-twice shortcut.
Bug-catching capability proven by temporarily reverting task 5.3b's
fix and confirming both the C and Go tests fail meaningfully, then
restoring to green -- confirmed independently by both the implementer
and reviewer.
Task 5.5 of job-array-single-wal-record (Requirements 6.1-6.5, 8.4).
This completes Phase 5 (Compact Scheduler/API-Server Push).
extract_data() (src/lib/request.c), the shared reassembly step every
inbound push passes through inside vtask_entry() before any receiver's
handler runs, had no case for Data_data_jobArray -- it hit
default: rc=-EINVAL. Because vtask_entry()'s callnext check treats any
nonzero return (including a negative errno) as "reassembly complete,
proceed," this never surfaced as an error: the real payload was freed
and the receiver silently handed an empty, zeroed dataset_t, with no
log anywhere in the chain. Tasks 5.1-5.3 were each individually
correct and tested in isolation, but the feature was dead on the real
wire end-to-end.
Adds case Data_data_jobArray: passing the single, unchunked
descriptor straight through (no pagination needed, unlike the
existing LIST-shaped cases) and nulls the source pointer after
transfer -- verified this is not cosmetic, since free_Request()'s
pushmsg case would otherwise double-free it via free_Data()'s live
Data_data_jobArray branch.
Proven with a real xce_encode()/xce_decode() round-trip through the
fixed extract_data() -> sched_handle_dataset() -> sched_on_job_array()
(task 5.3, unmodified) chain, not just a unit-level check on
extract_data() alone. Also confirms task 5.4's future API-server
receive path shares this exact fix, since both go through the same
xcecli_ctx_init() -> vtask_entry() -> extract_data() pipeline.
One self-flagged, non-blocking residual gap (pre-existing since task
5.3, out of this task's request.c-only boundary): dataset->rows for
the jobArray case is never deep-freed downstream, leaking one bounded
xce_job_submit_t per successfully-delivered array push commit.
Task 5.3b of job-array-single-wal-record (Requirements 6.1, 6.2, 6.4, 8.4).
Inserted mid-run after independent review confirmed the gap; user
confirmed sequencing (fix now, before task 5.4).
Adds sched_on_job_array() (handler.c), a sibling case to
Data_data_schedjobs decoding the new Data.jobArray variant and
expanding it locally into N ordinary schedulable tasks via
base_jobid + i, routed through the existing unmodified per-job
pipeline (sched_on_jobs()). Each element gets its own deep clone
(clone_job_submit_data/clone_job_startinfo) of the shared submit
spec, since per-element teardown (free_Job_ptr) frees it
independently and a shared pointer would double-free.
Dedup reuses sched_get_job() against the same kvtab sched_add_job()
guards, rather than a new mechanism. A confirmation log line fires
once per successful expansion (skipped on a fully-duplicate
redelivery).
Adds a new task (5.3b) discovered during this task's review: the
shared wire dispatcher extract_data() (src/lib/request.c) has no case
for Data_data_jobArray and silently drops every real array push
before any receiver's handler runs -- outside every current task's
boundary, confirmed to affect the future API-server receive path
(task 5.4) identically. design.md's File Structure Plan corrected to
record this. User confirmed sequencing: fix it now, before 5.4.
Task 5.3 of job-array-single-wal-record (Requirements 6.3, 6.4, 6.5, 8.4).
The Task 9 tests were marked [x] but were stub/absent. Convert the
correctness-critical ones to real white-box tests, registered as meson tests
under --suite scheduler (leaks-off per convention):
- test_handlers.c (new) — #include ../src/handler.c; covers 9.2/9.3: each
notify/dataset handler enqueues the correct op / skips NULL-empty / -EINVAL.
Asserts the 5.6 fix (update_job -> UPDATE_JOB_FIELDS, not bulk UPDATE_JOB).
- test_job_update.c — add 9.5 (real sched_update_existing_queue, replacing the
former local stub); already covers 9.4 (job) and 9.7 (rusage).
- meson.build — build + register all three scheduler test executables.
Suite GREEN: sched_test 15/15, sched_job_update_test 10/10,
sched_handlers_test 6/6; ASAN-clean.
Honest remaining: 9.6 dequeue worker-resvals APPLY is PARTIAL (enqueue tested;
apply is a dormant path, R-3); 9.8 is a documented compile-time absence check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5.5 (Req 6 / G-5): the JOB_RUSAGE notify path called the plugin but never
stored the usage data — xce_sched_job_t had no rusage field (UB-6.1 unmet).
- Add xce_job_rusages_t* rusages to xce_sched_job_t.
- New sched_apply_job_rusage() stores the report ON the job (ownership
transferred), replacing any prior via free_JobRusages_ptr; -ENOENT unknown,
-EINVAL NULL. Dequeue NULLs the request pointer on success; kill_sched_job
releases job->rusages.
5.8 (G-3): sched_pend_update_killer left five new-op payloads on empty
break cases — a per-message leak since ownership is transferred on enqueue.
- Free each with the exact generated deep-freer: free_QueueConf_ptr,
free_Resource_ptr[], free_ResourceVal_ptr[], free_ResValList_ptr,
free_JobRusages_ptr. Safe: handlers nullify on enqueue, makeResource copies
via makeInstance, dequeue copies values (no aliasing).
Tests: sched_job_update_test 7/7 (3 update + 4 rusage), regression
sched_test 15/15, ASAN clean (no bad-free/double-free).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The notify schedUpdateJob path shared XCE_SCHED_OP_UPDATE_JOB with the
dataset bulk-load, so the dequeue routed both to sched_update_jobs()
(create/replace) — a notify update to an existing job corrupted/duplicated
live job state and never returned -ENOENT (Req 3 UB-3.1/UB-3.2 unmet;
sched_update_existing_job was dead code copying only state).
- Add distinct op XCE_SCHED_OP_UPDATE_JOB_FIELDS (+ jobfieldsupdate union),
enqueued only by the notify handler, dequeued to sched_update_existing_job.
- Move sched_update_existing_job to schedobj.c; update all mutable fields
(state, priority, queue, resreq) in place; -ENOENT unknown, -EINVAL NULL.
- Dataset bulk-create path unchanged. Killer frees the new payload.
- New white-box test sched_job_update_test (3/3); regression sched_test 15/15.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire up 6 previously-stubbed message handlers in the scheduler:
resource dataset, resval dataset, job update notify, queue update
notify, resval update notify, and job rusage notify. Remove the
unreachable pushmsg branch (vtask_entry rewrites all pushmsg to
datamsg before the handler fires).
Changes by component:
- helper.h/helper.c: add xce_notify_event_name() for symbolic log names
- scheduler.h: add 5 new xce_sched_op_t values and matching union fields
- plugin.h/plugin.c: add plugin_job_rusage_hook_t and plugin_job_rusage_update()
- schedobj.c: add sched_update_existing_queue() with NULL/-ENOENT guards
- scheduler.c: wire 5 new dequeue cases; add sched_update_existing_job()
and static sched_op_name(); emit WARN log on handler failure
- handler.c: implement sched_handle_resources/resvals (dataset) and
sched_handle_sched_update_{job,queue,resvals}/job_rusage (notify);
remove dead sched_handle_jobpush/workerpush and pushmsg branch;
update TRACE logs to use symbolic name helpers
- test.c: 5 new unit tests covering notify event names, enum values,
and queue update NULL guard (15/15 pass)
