Implements the scheduler's first request-reply opcode (OPCode_schedAdmin).
sched_handle_admin_request() (handler.c) decodes SchedAdminRequest and
dispatches by action: recordStatus answers synchronously (pure read);
recordStart/recordStop are deferred onto the existing pend_updates queue
(XCE_SCHED_OP_ADMIN_RECORD) since sched_record_start()'s snapshot call
requires the session_active==0 precondition only guaranteed on the driver
thread, not the async request-handling thread. driver.c's
xce_sched_driver_translate_pending_events() performs the deferred action
and sends the reply via a new xcecli_reply_schedadmin() (client.c/client.h),
correlated by seq copied before the original request is freed.
Adversarial review found a genuine cross-thread race: sched_record_status()
read driver-owned recording state (including a pointer the driver thread
frees) from the vtask thread with no lock -- the codebase's first such
race, missed by ASan (doesn't detect races). Fixed with a dedicated
record_lock scoped strictly to in-memory field access, never spanning file
I/O; sched_record_status() now strdup()s under the lock instead of
returning a borrowed live pointer. Verified with a real two-thread
lock-contention test (independently confirmed non-vacuous by removing the
lock and observing the test fail) plus a deterministic use-after-free
reproduction.
26/26 scheduler suite, clang-format clean.
Periodic checkpointing hooks into sched_record_write_events(): once per
call, when record_snapshot_interval > 0 and record_event_count crosses a
multiple of it, writes a fresh SnapshotEntry via the same
write_snapshot_entry() helper sched_record_start() already uses (made
non-static, zero duplication). Safe from the session_active precondition
because this call site is only ever reached from
xce_sched_driver_translate_pending_events(), which always runs strictly
between sessions.
Confirmed the write-failure error-handling half (AC 10) was already
complete from task 9.2's sched_record_abort_on_error() -- added the
missing end-to-end test using a genuine closed-fd write failure rather
than a mocked return code, proving the driver keeps processing sessions
afterward.
25/25 scheduler suite, clang-format clean. Two non-blocking follow-ups
logged in tasks.md: --record-snapshot-interval has no CLI wiring yet
(AC 11's own struct-field mechanism is fully tested, but nothing sets it
in production), and test_record_lifecycle.c has grown past this suite's
prior size precedent with a natural future split point identified.
Test fixtures create WAL/snapshot scratch dirs via mkdtemp
(/tmp/xce_e2e_ckpt_*, /tmp/xce_ucaf61_*, /tmp/xce_ts_*) but their
teardown only frees the persist_t — the dir is never removed. persist_init
preallocates a 64MB WAL segment per init, so a full controller_unit_test
run leaves ~28G in /tmp in under a minute, filling the root filesystem.
Remove only xce_* DIRECTORIES at suite start (stale leftovers) and via
atexit (this run). Files are untouched: the user's /tmp/xce-mcp binary
and the suite's own /tmp/xce_test.log survive the -type d filter.
The apiserver's 30s heartbeat sends a fire-and-forget nop (xce_nop) that
never waits for the reply. The controller replies to every nop, and
xce_handle_reply's default case returned -ENOENT for OPCode_nop, so
vtask_entry called xce_wakeup and left the nop's status sitting in the
single-slot wbuf. The next blocking request (xce_fetchfile / xce_jobsub
/ xce_jobctrl) then consumed that stale status as its own reply:
the fetchfile waiter woke with status=0 but ctx->reply.u.fetchfilereply
still NULL -> 'nil reply' -> job peek 501.
Return 0 for OPCode_nop replies so they never touch wbuf. Control-op
replies (jobctrl/queuectrl/hostctrl) legitimately hit the default case
and still wake.
Regression test: test_handle_reply_nop_does_not_wake (red->green).
Adds recording-state fields directly to xce_scheduler_t (no new
xce_sched_driver_t type, per task 9.1's precedent). sched_record_start()/
sched_record_stop() implement AC 10: validate via task 9.1's path
validator, open with a TOCTOU-hardened sequence (O_NOFOLLOW, no O_TRUNC,
fstat()+st_nlink==1 check before ftruncate()), snapshot the core, write
ReplayHeader+SnapshotEntry on start, ReplayEvent{endSession} on stop.
driver.c gains boolean-gated recording hooks in
xce_sched_driver_translate_pending_events() (per-event-batch ReplayEvent
writes, new recording_events.c) and xce_sched_driver_run_normal_mode_round()
(per-session ReplaySession writes, new recording_session.c) -- zero I/O/
allocation when inactive.
Two real bugs found and fixed during implementation: bug-1498 (O_TRUNC
would have truncated a hard-linked victim file before the TOCTOU check
could reject it) and bug-1499 (a transient edit dropped
xce_core_signal_go(), causing a 3-way thread deadlock).
test_record_lifecycle.c decodes the full start->events->session->stop
cycle with the real capnp decoders and asserts the exact message
sequence; TOCTOU regression tests use real symlink(2)/link(2) syscalls.
25/25 scheduler suite, clang-format clean.
Adds --record-dir CLI flag and sched_validate_record_filename() with
canonical-path containment (realpath()-based, not textual-prefix) plus
separator/".." rejection before any filesystem access.
Adversarial review found and this task fixed two CWE-59 inode-aliasing
bypasses: a direct hard link inside record_dir aliasing an outside file,
and a compound variant where a symlink pointed at an in-directory file
itself hard-linked outside record_dir. Both closed via st_nlink checks
on the direct entry and the fully-resolved symlink target respectively,
each verified with a live RED->GREEN reproduction. TOCTOU residual risk
documented and deferred to task 9.2 (O_NOFOLLOW + fstat() re-check).
18/18 recording subtests, 24/24 scheduler suite, clang-format clean.
Task 8.1's own -EBUSY guard test only proved rejection against a
synthetic single-thread session_active flag flip, with a comment
misattributing the technique to test_core_full_cycle.c's actually-
genuine-session precedent. Real gap, not busywork: xce_core_go()
publishes session_active via memory_order_release while the snapshot
guards read it via plain atomic_load() -- the synthetic same-thread
test never exercised any real cross-thread visibility of that flag.
Adds a genuine driver-driven session test (real picker thread + real
xce_core_go()) proving both save and load reject with -EBUSY during
the same active session, leaving state genuinely unchanged (sentinel
bytes for save; a distinguishable non-existent queue + unchanged
candidate count for load). Zero production changes -- task 8.1's
guards were already correct.
Closes out Group 8 (Core: Snapshot Serialization) entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_sched_core_save_snapshot()/xce_sched_core_load_snapshot(),
split across snapshot.c/snapshot_wire.c/snapshot_load.c (800-line cap)
along real save/load/shared-helper boundaries. Serializes the full
core state (config, jobs, queues, workers, resources, resvals, users,
projects, prng_seed, internal_state) into/out of sched_snapshot_data_t
(task 7.3's committed C mirror of the capnp SnapshotData struct) as
plain in-memory structures -- no wire encoding here, that's a later
task. After load, rebuilds the picker's private state by re-invoking
its state-change hooks (queue updates, then job pushes) against the
restored core state, matching xce_core_push_events()'s own hook-firing
order; picker-private state itself is never stored in the snapshot.
Adds a new prng_seed field to xce_sched_core_t -- requirements.md/
design.md both presuppose it but no prior task added it. Seeded to 0,
no PRNG consumer wired yet (deliberately out of scope).
Cross-core behavioral-equivalence proven directly: two independent
cores, each with its own registered picker, save on one and load into
a freshly-initialized other, both driven through an identical real
session, decision sequences asserted identical in order.
Found and fixed two real bugs along the way: a missing prng_seed
restore in this task's own new code, and a genuine guard bug in
already-committed core.c (sched_get_user()/sched_get_project()
treating key_find()'s expected -ENOENT as fatal, silently breaking
get-or-create for any new name) that also affects the live production
path (make_sched_job(), JOBS_PUSH) -- a narrowly-scoped two-line fix,
independently traced to the primitive level and confirmed correct by
review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
bug-1506 (controller): sched_worker_change pointed w->host.resvals_to_sched
at resvalist.data, then xlist_fini freed it — a concurrent reader (the
WORKER_JOIN hook encoding the worker's resvals) hit an ASAN
heap-use-after-free that crashed the controller and failed every job. Take
ownership of the resval data via MOVE_PTR (mirrors collect_resvals_workers);
the prior array is intentionally leaked (freeing it races the same readers).
bug-1507 (worker): the apiserver's bug-1500 execuser fix sent the submit
user ("admin") as execuser, but that user does not exist in the worker
host's /etc/passwd. systemd failed the worker unit with EXIT_USER (217),
so every job died immediately after submit. job_start_local_task now
checks the execuser with getpwnam() and falls back to "nobody" when the
user does not exist.
Verified with a full job lifecycle (submit -> RUN as nobody -> DONE) on
the clean build: all 4 daemons healthy, zero ASAN.
Co-Authored-By: Claude <noreply@anthropic.com>
`xcecli jobs list -a` returned "no job found" even when jobs existed.
FilterRows routes every condition on a bitmask field (job_status /
host_status) to MatchesBitmaskCondition, which only implemented OpEQ/OpNE
bitmask semantics — the `-a` filter compiles to `job_status notnull`
(OpNOTNULL), which fell through to `return false` and filtered out every
job. (The default list's `notin(EXIT,DONE)` is compiled to OpEQ(mask), so
it was unaffected.)
Add OpISNULL/OpNOTNULL branches to MatchesBitmaskCondition that test the
raw field value. New query_test.go: 3 regression tests (notnull matches
all, null matches none, bitmask notin path unaffected), RED->GREEN.
Verified live: `xcecli jobs list -a` now lists all jobs including DONE.
Co-Authored-By: Claude <noreply@anthropic.com>
Job PEND root cause when the scheduler registered before the worker:
xcesched_push_workers only fires on scheduler join (sched.c on_scheduler_join),
so a later worker join's slots (set_host_slots from hconfig->mxj=8) is never
re-pushed to the scheduler -> allocation unsatisfied (slots total 0) -> PEND.
Operational fix (no code change): start controller -> worker -> scheduler so
the scheduler's join-time schedworkers push carries the worker's slots.
Co-Authored-By: Claude <noreply@anthropic.com>
Audit of all ~19 new types/enums from tasks 7.1-7.4 found one genuine
gap: ReplayEvent's own envelope union (11 members across 7.1+7.2) was
never actually constructed/encoded in any test, despite its individual
payload types each having their own round-trip test. Adds 3 tests
(jobStateChange/jobUpdate/endSession variants) proving the union
mechanism itself round-trips correctly.
No new schema/typemap changes -- audit-only. Decided against creating
design.md's originally-named test_replay_format.c: tasks 7.1-7.4
independently, consistently (4-for-4) chose the existing
test_e2e_capnp.c instead. Deliberately did not add Request.body/
RequestReply.data union-wiring tests for OPCode_schedAdmin, matching
universal codebase precedent (opcode-dispatch union members are always
tested via functional dispatch tests, never bare schema round-trips,
and the real dispatch logic doesn't exist until task 10.1).
Closes out Group 7 (Foundation: Replay Wire Format) entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds SchedAdminAction (recordStart/recordStop/recordStatus),
SchedAdminRequest (action/recordFile), SchedAdminReply (ok/error/
recording/file/eventCount/startedAt) to controller.capnp, matching
requirements.md Req 5 AC 10 verbatim. Adds schedAdmin@42 to OPCode,
schedadminmsg@31 to Request.body, schedadminreplymsg@19 to
RequestReply.data -- the exact next-available ordinals, verified live
before implementation and re-verified byte-identical against every
existing value/member.
Deliberately schema-only, per this task's own scope -- the controller
dispatch logic (cluster.c's case OPCode_schedAdmin:) is task 10.1's
job. cluster.c, src/frontend/, src/xceadmin/, src/scheduler/ all
untouched.
Round-trip tests cover all 3 SchedAdminAction values and both
ok=true/ok=false SchedAdminReply cases, including an asymmetric
ok=false/recording=true combination that rules out a Bool-field
bit-packing mixup at the wire level.
Closes out Group 7 (Foundation: Replay Wire Format) entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds DispatchDecision (wraps JobAlloc), PendDecision (wraps
JobPendReason), PreemptVictim, PreemptDecision to controller.capnp.
Replaces task 7.1's empty SnapshotData placeholder wholesale with the
real 11-field struct (config/jobs/queues/workers/resources/resvals/
users/projects/prngSeed/internalState/internalVersion), in both the
capnp schema and its typemap.h C mirror. Extends ReplayDecision's
ts-only placeholder with a purely additive body union (dispatch@1/
pend@2/preempt@3).
Deviates from design.md's literal SchedUser/SchedProject mapnames
(xce_sched_user_t/xce_sched_project_t): those names are already real
C structs in sched/types.h, which shares a translation unit with
typemap.h in core_events.c/schedobj.c -- reusing them would be a
genuine duplicate-typedef compile error, not a style choice. Used
fresh mapnames (sched_snapshot_user_t/sched_snapshot_project_t)
instead, field lists unchanged, documented in both files.
Found and fixed a real bug mid-implementation: a doc comment's
literal */ substring prematurely closed a C block comment, corrupting
sched_snapshot_data_t and cascading into ~20 unrelated-looking compile
errors (bug-1493).
Five deep round-trip tests added to test_e2e_capnp.c. Full project
build and controller_unit_test (1082/1082) verified clean, confirmed
independently three times.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds JobStateChangeEvent (jobid/oldState/newState/exitCode) and
JobUpdateEvent (jobid/changedFields/job: Job) to controller.capnp,
matching design.md's spec verbatim. JobUpdateEvent.job reuses the
existing full 12-field Job/xce_job_t record, verified distinct from
the narrower SchedJob struct sharing the same mapname.
Extends ReplayEvent.body's union (task 7.1) with jobStateChange@10/
jobUpdate@11 -- the only two valid next ordinals, since capnp requires
strictly sequential ordinals with no gaps across a struct and task
7.1 deliberately stopped at @9. Existing @0-@9 fields and every other
struct in the schema are byte-identical.
Deep round-trip tests in test_e2e_capnp.c exercise every field with
non-default values, including the 3-level nested Job substructs
(submitData/allocs/pidinfo/pendreason). JOB_F_* bitmask bit values
deliberately left undefined -- a later task's job, not this
schema-only one's.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
The RUSAGE_UPDATE job hook fires with (cluster_t*, xce_job_rusages_t*) as
its varargs — the apiserver and scheduler subscribers parse exactly that
contract — but persist_job_hook_fun's default case treated the second arg
as a job_t* and read j->job.state at offset 56. A xce_job_rusages_t is 56
bytes and shares only jobid@0 with xce_job_t (which is why the jobid read
coincidentally worked), so the state read landed one byte past the struct:
ASAN heap-buffer-overflow (READ of size 4) in thread T7 whenever a job runs
and the worker pushes rusage updates. This was the recurring controller
crash (term 3/4/5) blocking job-PENDING verification.
Add a dedicated HOOK_DETAIL_JOB_RUSAGE_UPDATE case that reads jobid from
the xce_job_rusages_t* and bakes the job's current state from a jobmgr
key_find on the vararg cluster_t* (race-free: the sole caller
update_job_rusages() holds that job's write lock while firing the
AFTER_EVENT hook; re-locking the job inside the hook would deadlock).
Regression test: test_job_hook_rusage_writes_wal_entry now passes a real
56-byte xce_job_rusages_t instead of a full job_t (which masked the
overflow) — ASAN stack-buffer-overflow under the old code, PASS under the
fix. Full controller unit suite: 458 tests, 0 failures.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds ReplayCluster/ReplayHeader/SnapshotEntry/ReplayEvent/
SessionSummary/ReplaySession to controller.capnp, additively, first
task of Phase 3 (Foundation: Replay Wire Format).
Requirements.md's AC 20 capnp sketch isn't self-contained -- ReplayEvent's
union references JobStateChangeEvent/JobUpdateEvent (task 7.2's job),
SnapshotEntry.data/ReplayDecision's union reference SnapshotData/
DispatchDecision/PendDecision/PreemptDecision (task 7.3's job), none
of which exist yet. Resolved by verifying capnp's real constraints
against the toolchain directly: SnapshotData is an empty {} placeholder
(task 7.3 replaces its mapname'd C struct wholesale when adding the
real ~11 fields), ReplayDecision holds only {ts} (task 7.3 appends a
body union at new ordinals), and ReplayEvent's union holds only the 9
members with pre-existing backing types (Conf4Sched/SchedJobData/
WorkerData/ResourceData/ResValList/QueueConf/SchedJobRusages/Void),
deferring jobStateChange/jobUpdate to task 7.2 as new ordinals --
capnp requires strictly sequential ordinals with no holes, so this
split is compiler-mandated, not stylistic.
Round-trip test added to test_e2e_capnp.c. Full project build (all
binaries sharing this schema) and 1075/1075 controller_unit_test
verified clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
tryConnect built the xce_client_config_t with C.CString(host/from/key), then a
defer freed all three. But xcecli_ctx_init stores the config POINTER
(ctx->conf = config), not a copy — client_failover.c dereferences
config->masters/from/key for the client's ENTIRE lifetime (e.g.
xce_failover_next_target strncpy's a master host on every write-fail/redirect).
Freeing the strings left config->masters[0] dangling → ASAN heap-use-after-free
(strncpy in the background C thread) after a few submits.
Fix: keep the three C strings for the process lifetime (tryConnect runs once at
startup, gCtx is a process-global); free them only on the init-failure path.
Verified: 6 consecutive job submits, apiserver stays up, 0 ASAN errors
(previously crashed after ~2-3 submits).
Co-Authored-By: Claude <noreply@anthropic.com>
Sequential HA soak (xce-test-ha skill), 15 rounds, 48 jobs, zero job
loss, all 3 nodes led as primary, committed-prefix walcompare
consistent every round. Confirms tasks 4.1-5.2's src/scheduler/
rewrite (core extraction + normal-mode driver wiring into main.c)
introduced no controller-side regression -- consistent with every
task's independent review confirming zero src/controller/ changes
throughout Group 4/5. Closes out Phase 2 entirely; gates Phase 2 ->
Phase 3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Rewires main.c's live scheduling loop onto the new core-driven
architecture: xce_sched_driver_translate_pending_events() (task 5.1)
-> xce_core_begin_session() -> plugin_begin_session()+sched_account_jobs()
-> xce_core_signal_go() -> decision-drain via the registered
xce_core_decision_cb_t -> plugin_end_session() -> xce_core_end_session(),
all in the new exported xce_sched_driver_run_normal_mode_round()
(driver.c), replacing the old inline sched_update()+sched_begin_session()
+sched_go()+switch+sched_end_session() sequence.
xce_core_go() (task 4.4) is split into xce_core_begin_session()/
xce_core_signal_go() -- a pure decomposition (xce_core_go() itself is
now defined in terms of both, task 4.4's approved test suite passes
unmodified) -- specifically so the driver can interleave
plugin_begin_session()/sched_account_jobs() in the exact safety-critical
window task 4.4 deliberately excluded from pure-core scope: after the
resval reset, strictly before the picker thread is signaled. This is
empirically proven by test_driver_session_ordering.c via real
cross-thread happens-before assertions, not just documented.
xce_sched_driver_on_decision() is the new registered callback:
DISPATCH/PEND forward to the existing sched_publish_dispatch()/
sched_publish_pend_reason(); PREEMPT fails loudly (-ENOSYS, logged)
since no xcecli_preempt() client API exists and nothing in the live
allocator produces one yet (scheduler-preemption's territory).
Adds a --sim CLI flag (parse-only, loud not-yet-implemented failure,
since SimDriver is Phase 3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
modelJobToSubmit built the C job_startinfo_t without setting execuser, so the
controller forwarded a NULL execuser in the startjob request and the worker
ran every job as "nobody". Now the submit carries the resolved exec user: an
explicit exec_user wins, otherwise the job runs as the submitting user.
- add resolveExecUser() (plain-Go, unit-testable) + TestResolveExecUser
- set si.execuser in modelJobToSubmit, free it in freeJobSubmit
Note: deploying this exposed an ASAN heap-use-after-free in the apiserver's
submit/notification path (strncpy reading freed memory); the fix is committed
but needs the memory-lifecycle investigation before it can ship safely.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds driver.h/driver.c: xce_sched_driver_translate_pending_events()
drains core->pend_updates via the existing dataq-swap pattern (same
mechanism sched_update() uses), translates all 10 xce_sched_update_
request_t op types into the matching xce_core_event_t, and pushes the
batch through xce_core_push_events(). Scoped to xce_sched_core_t*
alone -- pend_updates already lives on core since task 4.1, and this
task is translation-only (no decision publishing, no mode-selection,
no recording -- all later work). handler.c (the enqueue side) is
untouched. First task in Group 5 (Core: Normal-Mode Driver).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds test_core_full_cycle.c: drives xce_core_push_events() -> xce_core_go()
-> decision drain -> xce_core_end_session() end-to-end against a real
picker pthread (running the real xce_job_picker_iface_t hooks on a
test-local stub picker) and a main-thread stand-in allocator, since
sched_allocator() is xce_scheduler_t*-coupled and out of pure-core
scope. Proves both preconditions (core_go reentrancy, push_events
mid-session) reject against a genuinely active session and recover
once it ends. Closes out group 4 (Foundation: Scheduler Core
Extraction) -- tasks 4.1-4.6 all complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_now() (new core_clock.c, kept separate from core.c so
test_core_isolation.c's wall-clock scan still enforces zero direct
clock_gettime()/time() calls elsewhere in the core): the two-anchor
plus speed formula (virtual_clock_base + (elapsed * speed_num) /
speed_den), covering all four modes (normal, discrete/real-time/fast-
slow simulation) via four new atomic fields on xce_sched_core_t,
defaulted to normal-mode (1/1 speed) in xce_sched_core_init().
requirements.md's literal tlog_reg_format_func(fn, core_ptr) call
shape doesn't exist in the real vendored tlog library (registration
takes only the callback; userptr is per-call-site, not bindable at
registration) -- resolved with a module-level active-core pointer read
by a new core-scoped format function, not wired into mylog_init()/
main.c yet. xce_core_go()/xce_core_end_session() (task 4.4) now stamp
session_start/compute session_duration_ns via xce_core_now(), with
their existing session-active/reentrancy behavior untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_go()/xce_core_end_session() (new core_session.c):
xce_core_go() is non-blocking, performs begin-session state reset
(select-eval-ctx, resval resets), signals the picker thread via wbuf,
and returns -EINVAL on a reentrant call while a session is already
active (proven a true no-op, not just an early return). Draining
core->decisions and invoking the driver's on_decision callback is the
driver's own responsibility between core_go() and core_end_session()
(confirmed driver-owned, not core-owned, per the Driver <-> Core
protocol). xce_core_set_decision_cb() registers the callback + driver
context as a separate setter, keeping xce_sched_core_init()'s live
sched_init() call site untouched. Adds XCE_DECISION_PREEMPT to
decision.h (jobid/victims[]/reason) alongside the existing DISPATCH/
PEND types. Additive only — sched_go/sched_begin_session/
sched_end_session/sched_allocator/main.c remain untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
bug-1498 (re-dispatch): sched_job_is_ready() was a stub returning 1, so any
job lingering in the fcfs pendJobs queue was always elected. A dispatched job
was never marked non-PEND (the controller defers its schedStartJob notify to
the not-yet-implemented apply pump, controller/job.c), so fcfs_update_job
re-enqueued it on every job update and the election thread re-dispatched the
running job — the worker got conflicting signals and the job was SIGTERM'd.
- sched_job_is_ready: elect only PEND, non-suspended jobs
- sched_commit_decision (dispatch): mark the job JOB_STAT_RUN on dispatch so
it leaves the PEND set and is no longer re-electable
- fcfs_update_job: dedup via ordq_peek before enqueue
bug-1497 (double-free): sched_update's resval paths transferred ownership of
rv->val to sharedInst/w->resvals but left the request pointer set, so
sched_pend_update_killer double-freed it -> use-after-free in
build_host_eval_ctx. Clear rv->val = NULL on transfer.
Tests: new scheduler/testsuite/test_redispatch.c (8 cases): sched_job_is_ready
PEND gating + dispatch-commit marks the job RUN. Full scheduler meson suite
green. Live-verified: a `sleep 60` job is elected exactly once, runs the full
duration and exits 0 (`completed successfully`); before the fix it was
re-dispatched at ~11s and killed (SIGTERM 143).
Co-Authored-By: Claude <noreply@anthropic.com>
Defines xce_core_event_type_t/xce_core_event_t (9 event kinds: config,
job, worker, resource, resval full/delta, job state change, job update,
queue update, job rusage) in core.h. Implements xce_core_push_events()
in a new core_events.c (split out of core.c to stay under the 800-line
cap): rejects -EBUSY mid-session, otherwise applies each event's core-
state mutation and fires the corresponding picker hook synchronously,
exactly once per event, before returning. CONFIG_PUSH decomposes the
pushed config's embedded queues/resources/resvals via shared helpers
(also reused by QUEUE_UPDATE's upsert path), mirroring the existing
sched_init_config()/sched_init_resources()/sched_init_sharedInst()/
sched_init_queues(). Narrows sched_resolve_job_picker() to take
xce_sched_core_t* (pure type change, zero behavior change). Purely
additive — not wired into the live scheduling path yet (task 5.x).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Hardcodes jbuf's cbuf capacity to the literal 1 in xce_sched_core_init()
(no parameter path, no local variable to drift), backed by a regression
test. Audits every kvmgr iteration reachable from the scheduling loop
(jobs/queues/users/projects): only jobs is ever iterated, via xzarray's
deterministic ascending-slot walk with no process-random seeding —
findings recorded durably as a comment above xce_sched_core_init().
core->workers iteration-order stability is flagged as an open follow-up
(scheduler-allocation's territory), not this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Splits xce_scheduler_t along the controller-I/O seam: xce_sched_core_t
(new core.h/core.c) now holds all scheduling state (jobs/queues/workers/
users/projects/resource managers/plugins/three-thread-cbuf pipeline),
while xce_scheduler_t shrinks to {core; apictx; cluster_cfg}. Proven via
an nm-based link-time isolation test (zero netio/xcecli/idgen/delaymgr
symbols reachable from core.c) plus a reproducible wall-clock-call guard.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Closes out group 2 (Core: Builtin FCFS Picker). Confirms builtin_fcfs's
ordering matches legacy fcfs_elect_job (byte-identical comparators, same
sched_queue_is_ready/sched_job_is_ready gates, provably equivalent jobid
sequence), confirms tasks 1.1-2.5 left every pre-existing test file
unmodified, and confirms resreq immutability, pend-reason factory-only
allocation, and select_eval_ctx_t memoization all hold across every file
this spec has touched so far. Adds a durable regression lock for the
pend-reason factory invariant (no direct calloc(pendreason_t) outside
make_pendreason()).
Task 2.4 already made sched_job_picker() delegate entirely to the picker
interface with zero direct queue/job traversal, satisfying Req 1 AC 1/AC 7
as a byproduct. This adds a source-text regression test locking that
property in place: asserts sched_job_picker()'s body contains none of the
forbidden direct-traversal symbols and does contain the expected
picker-interface call sites.
Registers builtin_fcfs at scheduler startup and makes it the default
job_pick_plugin, replacing the hardcoded "xce_sched_fcfs" literal in
sched_job_picker() with a fail-loud lookup through sched_resolve_job_picker().
The picker thread now branches legacy job_elect_fn plugins (delegated
unchanged, preserving their self-contained wbuf-driven loop) from native
xce_job_picker_iface_t pickers (driven by a new per-session loop), and the
allocator now calls on_dispatch/on_pend for every picker as Req 1 AC 4
requires. Closes task 2.3's flagged picker/allocator concurrency race with a
mutex on fcfs_plugin_data_t, and adds cancel points to next_job().
Adds begin_session/next_job/end_session/on_dispatch/on_pend, driving a
dual-cursor walk (queue then job) over builtin_fcfs's private ordq_t
state. next_job() returns candidates in (queue priority desc, job
priority desc, queue_time asc) order and NULL once exhausted;
on_dispatch removes permanently, on_pend is a true no-op.
Review flagged an unsynchronized cross-thread access risk between
next_job() (picker thread) and on_dispatch/on_pend (allocator thread)
once wired live in task 2.4 — recorded in tasks.md Implementation Notes
for that task to close before registering builtin_fcfs as the default.
Requirements: 3.2, 3.4, 3.6
Adds on_queue_update, on_job_push, on_job_state_change, and on_job_update
to builtin_fcfs, driving the picker's private ordq_t candidate sets
directly (no core/sched queries), matching fcfs.c's existing ordering
semantics on the new finer-grained hook interface.
Requirements: 3.2
put_resvals (resource.c) used `idxs[i] <= 0` to skip "invalid" resource
indexes, but index 0 IS valid — it's the first resource in the table
(e.g. "slots"). For idx 0 the val was never allocated (xvals[i] stayed
NULL) and the write loop then SEGV'd. bug-1495.
Fix: skip only negative indexes (`idxs[i] < 0`), and NULL-guard the
write loop defensively.
Verified end-to-end: a locally-built scheduler now connects to the t1
controller and dispatches all 5 test jobs to node t1 (exec_hosts
populated, jobs run and exit). This was the final blocker for job
scheduling (bug-1494: all jobs PEND).
Also: add the t1 host definition to hosts.yaml (worker join was rejected
with -EINVAL because no matching hostconfig existed for hostname "t1").
Co-Authored-By: Claude <noreply@anthropic.com>
New src/scheduler/src/plugins/builtin_fcfs.c implements
builtin_fcfs_init()/fini() and the private fcfs_plugin_data_t (ordq_t
queues, per-queue ordq_t pendJobs) that all later builtin_fcfs hooks
build on. fcfs.c (today's live job_elect_fn-based picker) is untouched
-- still production, until task 2.4 flips the registration.
picker == xce_sched_plugin_t* calling convention (private state stored
in plugin->data) confirmed consistent with task 1.2's legacy wrapper.
Requirements: 3.1 (context), 3.3
plugin_register() now adapts an old-style job_elect_fn-only plugin to
xce_job_picker_iface_t via legacy_picker_wrapper_iface (next_job()
forwards to job_elect_fn; other hooks are safe no-ops covered elsewhere
in the existing lifecycle), logging a one-time deprecation warning. A
plugin that already sets picker_iface directly skips the wrapper and
the warning. Reuses the existing plugin_register()/key_find() mechanism
-- no parallel registration system.
Requirements: 1.6, 2.5
Add xce_job_picker_iface_t to plugin.h with all 13 hooks across
lifecycle/state-change/session/post-allocation/optional categories,
matching design.md's literal struct. Documents per-category thread
assignment, next_job() cancel-safety, and begin_session() session-reset
requirements. Forward-declares xce_sched_core_t (built in a later task).
Requirements: 1.2, 1.5, 2.6
worker/src/resource_sync.c init_builtin_load_indices registers 11 load
indices (r15s/r1m/r15m/ut/pg/ls/it/swp/mem/tmp/io) but resources.yaml only
defined r1m/r15s/mem/tmp. The 7 missing ones (r15m/ut/pg/ls/it/swp/io) caused
controller "update_resval: Success" ERRORs and scheduler "unknown resource in
per-worker resvals update" warnings, leaving the worker resource view
incomplete. Add the 7 definitions matching the existing r1m/r15s pattern.
Docs: log bug-1494 (all jobs PEND, scheduler never dispatches; after this
config fix jobs still PEND — remaining suspect is the scheduler→controller
join path after restart), update design doc + verification manual known-issue
tables and root CLAUDE.md cluster-verification status.
Co-Authored-By: Claude <noreply@anthropic.com>
The apiserver returns list rows in OrderMap columnar form —
{"Fields":["col",...],"Data":{"col":val,...}} — but the read clients
unmarshalled each row as a flat object, so every field came back empty
(counts were right, values were all zero/"").
Fix: decode rows as apiRow{Fields, Data map[string]json.RawMessage} and
pull named fields out of Data via rowStr/rowInt/rowInt64/rowFloat helpers.
ListNodes, ListQueues, and getJobs (ListJobs/GetJob) all use apiRow now.
TDD: 3 new ordermap_test.go tests built from live-captured apiserver
response shapes (red: fields empty → green: parsed). Live-verified against
t1: node name=t1 cores=8, 4 queues active, 3 jobs with real fields.
Co-Authored-By: Claude <noreply@anthropic.com>
buglog.json: corrected bug-1480/1481 (stale "not yet fixed" entries, code
landed same day the entries were written); bug-1483/1484 fixed; bug-1485
investigated, downgraded to low-priority/environment-flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
bug-1485 (flaky test_live_config_push_against_primary_reaches_success_path,
~24/25 fail rate under pytest, primary process going <defunct> after
promotion) could never be diagnosed because controller stderr was
DEVNULL'd -- no ASAN report or signal was ever captured from a real
failure.
Redirects each node's stderr to deploy_dir/logs/<name>.stderr.log and adds
an XCE_KEEP_DEPLOY_DIR=1 debug toggle to preserve the deploy dir across
fixture cleanup for inspection. Ran the live test 40 times in this
environment (25 idle + 15 under concurrent build contention): 40/40
passed, 0/120 captured stderr logs non-empty -- could not reproduce.
Downgraded to low-priority/environment-flake, left open; the capture
infrastructure stays in place for the next real failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
bug-1421 added two term-discovery fallback vectors so a stale-term primary
steps down instead of retrying AppendEntries forever: heartbeat-reply
state-report term adoption and WAL-NACK reply_term adoption, both inside
repl_ack_decode_and_dispatch. Neither had a test driving the real
production dispatch path and asserting raft_on_higher_term_observed
actually fires.
Adds two tests mirroring test_config_repl.c's existing config-push term
test, driving peer_ctrl_msg_handler -> repl_ack_decode_and_dispatch via
wire-shape capnp packets. Both pass immediately -- the underlying bug-1421
code was already correct, just unverified end-to-end -- closing
raft-adapter-conformance's Requirement 7.5 unverified claim.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
ha_arm_heartbeat_timeout_vtask published a calloc'd liveness_poll_ctx_t
into s_liveness_ctx unconditionally but only ever freed it via
liveness_poll_fn's cancelled-check path, which never runs if
delay_task_ms was never called (c==NULL, delaymgr not running) or itself
failed. Both no-schedule cases now free the ctx immediately, establishing
the invariant that s_liveness_ctx is non-NULL only when a delaymgr task
actually holds a live reference to it.
Regression test drives a real LOST PreVote round without stubbing
ha_prevote_rearm_liveness_hook (every other LOST-path test in the file
stubs it specifically to avoid this code path) and asserts the ctx isn't
left dangling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LXkenK8rqE8mdS1YKjsq78
New §5 in src/docs/chatbot-verification.md: how to connect Claude Code to
the scheduler-agent MCP server. Covers building the mcp-server binary,
`claude mcp add` CLI registration (verified against installed CLI
v2.1.220), the .mcp.json project config schema, the three scopes
(local/user/project), the first-run approval gate, verification commands,
and stdio/env gotchas. Original §5-§7 renumbered to §6-§8.
Co-Authored-By: Claude <noreply@anthropic.com>
New src/docs/chatbot-verification.md: reproducible steps to deploy the
scheduler-agent stack on the t1/t2/t3 test cluster (AppImage build, store
sync, controller/apiserver/scheduler/worker startup) and verify Phase 0-1
(login, mcp_operator API key, read ops, submit). Documents the vcpus quirk,
setsid launch pattern, conf/current pointer mechanism, and known bugs
(job-control HTTP 500 = bug-1490, HA election deadlock).
Also add the manual to the design doc's related-documents index.
Co-Authored-By: Claude <noreply@anthropic.com>
