Adds failing unit tests for repl_fmt_jobid_range() covering contiguous
ranges, non-contiguous sub-ranges, multiple array bases, mixed array/
non-array ids, and a too-small output buffer (Requirement 4, AC 4.1-4.5).
The function itself is not yet implemented (task 1.2) - build fails at
link time with undefined reference to repl_fmt_jobid_range, confirming
the RED phase.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Task 4.1: new test proving a snapshot-restored PENDING job (valid
resreq) reaches the real, live sched_allocator() thread's
sched_make_allocation() entry point without crashing -- the exact
bug-1569-class failure task 1.2 closed for the snapshot-load path.
New file (not an extension of test_snapshot_save_load.c) since driving
the real allocator needs scheduler.c's full dependency closure, too
heavy for that file's deliberately lean core-only TU. Follows the
already-established pattern from test_scheduler_connect.c /
test_scheduler_backpressure.c / test_scheduler_dispatch_commit.c: link
the real scheduler.c, wrap xcecli_start/xcecli_join, start a real
sched_init(), feed jobs directly into core.jbuf to drive the live
allocator thread. Wired into meson.build as a new executable/test.
Verified with two fault-injection experiments (reverted after):
disabling task 1.2's schedobj construction call makes this test fail
cleanly via its own assertion; going further and removing that
assertion reproduces a genuine ASAN-caught SEGV in the real allocator
path -- concrete proof this test would have caught bug-1569's failure
shape. Both polling loops (jbuf drain, decision wait) are bounded
(2s timeout, 5ms poll) -- no hang risk; confirmed non-flaky across 10
consecutive runs.
Requirements: 2.2, 2.4
User-reported live crash: xce_scheduler SIGSEGV in fcfs_cursor_enter_queue
(builtin_fcfs.c), called from builtin_fcfs_next_job <- sched_job_picker
(picker thread).
priv->cursor/priv->job_cursor (fcfs_plugin_data_t) are raw ordq_item_t*
pointers that persist across SEPARATE next_job() calls, each with its own
independent priv->lock acquisition/release. The existing mutual-exclusion
fix (task 2.4) correctly protects the cursor walk from corruption DURING
one locked call, but does nothing to stop a LATER call from dereferencing
a node a properly-locked, INTERVENING call already freed.
on_queue_update()'s re-sort (remove old ordq_item_t node, insert a fresh
one via ordq_put) is exactly such a free -- with only one queue in the
deployment, the picker's cursor is essentially always parked on it
between calls (sched_queue_is_ready/sched_job_is_ready are unconditional
true stubs today, so the cursor never advances away while jobs remain),
making a queue config update land in that window trivially reachable.
Fixed by invalidating (NULLing) whichever cursor points at a node right
before that node is freed, at all three sites that can free a node the
cursors might reference: on_queue_update's reuse-and-resort path
(priv->cursor + priv->job_cursor in lockstep), builtin_fcfs_on_job_state_change's
!JOB_IS_PEND removal, and fcfs_remove_job_everywhere's cross-queue removal
loop (both job_cursor only, via an ordq_peek-then-invalidate-then-delete
pattern). Always safe: next_job() treats a NULL cursor as "no more
candidates this round", and begin_session() resets both cursors fresh
every round regardless -- worst case is a bounded one-round delay, never
a crash or a permanently lost job.
New test_builtin_fcfs_cursor_uaf.c reproduces the bug fully single-threaded
(next_job() releases the lock between calls, so no real concurrency is
needed to hit the window). RED-phase independently confirmed a real ASAN
heap-use-after-free with the exact same call chain as the live crash
(freed by on_queue_update, read by next_job()); GREEN confirms 0 failures
and no job permanently lost across the invalidation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
User-reported live crash: xce_scheduler SIGABRT via glibc malloc_printerr
(heap corruption) inside kill_limit(), called from limit_update_config()
during a live config reload.
limit_update_config() called kill_limit(limit) in its error path after a
failed key_add(privdata->limits, ...). But privdata->limits is a kvmgr
created with .datakiller = kill_limit, so on a kvtab-level failure (e.g.
-EEXIST for a duplicate limit name -- exactly the rc=-17 captured in the
crash's own backtrace), xkvmgr_add() already rolls back via vclear() ->
xzarray_clear(), which fires kill_limit(limit) once automatically. The
caller's explicit second call was a double free.
Removed the redundant call; the kvmgr's own configured datakiller already
owns cleanup on this failure path (mirrors the ownership asymmetry
job.c's restore_job_array() already documents for the opposite case,
where its kvmgr has no datakiller configured and the caller must free
failed items itself). One accepted, documented residual gap: a vstore()-
level OOM failure (never reaching the kvtab step) would leak `limit`
instead -- vanishingly rare, and a leak is strictly better than the
guaranteed crash on the common duplicate-name case; xkvmgr_add's return
code doesn't distinguish which sub-case occurred, so there's no clean way
to free conditionally without a kvmgr API change (out of scope).
New test_limit_plugin.c (this plugin had zero test coverage before) --
RED-phase independently reproduced a genuine ASAN heap-use-after-free
matching the live crash's exact call chain; GREEN confirms 0 failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
sched_commit_decision()'s DISPATCH branch now sets job->state = JOB_STAT_RUN
in the same operation that already commits job->allocs/job->accounts, per
Requirement 1.1/1.2/1.3. sched_account_jobs()'s existing JOB_IS_RUN gate is
unmodified -- this closes bug-1624 by giving it the input it was missing: a
dispatched job now survives a round boundary in resource-in-use totals
regardless of how many rounds pass before worker confirmation arrives.
Regression test drives the real, unmodified sched_account_jobs() across a
simulated round boundary (resInUse reset, no confirmation event) and
independently reverting the one-line change reproduces bug-1624 exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The original motivating recording (scheduler-replay-20260820-070103.replay,
named in bug-1611) no longer exists anywhere on disk -- confirmed via
exhaustive filesystem search. Substituted a real, driven 24-round/12-job
resource-constrained recording using actual driver.c/builtin_fcfs/resource.c
code, replayed through the real, unmodified interactive REPL.
This also surfaced and honestly documents a real structural limit:
Recorded-Decision Reconciliation only succeeds when a job's recorded
DISPATCH decision and its JOB_STATE_CHANGE-to-RUN confirmation land in the
SAME replay round (reconciliation searches only that round's own decoded
data). Constructing that exact coincidence from organic, continuous
scheduling is structurally impossible -- a job can't be both "freshly PEND
for this round's picker" and "already RUN before this round's picker runs"
at once. One job (ANCHOR_ID) engineers this coincidence using entirely
real constructors (matching the same precedent tasks 3 and 4.1 already
established); the other 11 jobs arrive/contend/dispatch/finish with zero
special-casing across all 24 real rounds. Recorded in tasks.md's
Implementation Notes as a scope finding for follow-up: in a real deployment
with genuine dispatch-to-confirmation latency, this narrows the feature's
real-world coverage of bug-1611/1613's dominant manifestation.
RED phase: reverting task 3's reconciliation call site makes this test
fail with the exact bug-1611 symptom in its own assertion message, plus 3
sibling tests break too; restored and green. No production code touched
(final task in the feature -- all production work landed in task 3).
50/51 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Four checks in one run: (a) REPL-vs-batch decision equivalence unmodified
and still passing; (b) --sim-verify reports no false mismatch for a
recording with multiple already-resolved jobs across rounds (non-adjacent
memmove compaction, no cross-round bleed); (c) a replay starting from a
snapshot with a job already RUN-state and no in-window recorded decision
degrades gracefully (no crash, no fabricated value, a second ordinary job
unaffected) -- the first test to exercise reconciliation's "not found"
path through the real sched_sim_driver_run()/xce_sched_core_load_snapshot()
path rather than a hand-built core; (d) normal-mode behavior is byte-for-
byte unchanged (zero production diff, the strongest possible evidence).
Found and fixed a real test-authoring gap while building (b): builtin_fcfs
re-offers a still-PEND job as a fresh PEND every round (confirmed
pre-existing, unrelated, also visible in test_repl_equivalence.c's own
fixture) -- the first draft's round-2 fixture omitted this recurrence.
No production code touched (task 3 already completed the feature) --
git diff on sim_loop.c/resource.c/scheduler.c/driver.c/sim_verify.c is
empty. 49/50 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds an integration test comparing round-by-round resInUse and scheduling
outcomes between a real driven normal-mode recording and a simulation-mode
replay of the identical contention scenario (worker capacity 10.0, jobs
wanting 8.0/5.0/2.0, plus a 4th job proving cross-round persistence).
Confirmed (by direct read of sim_picker_thread_fn(), independently
re-verified in review) that sim's live picker is structurally PEND-only --
it has no capacity-check/dispatch path for a freshly-evaluated job. This
means task 1.2's single-round fixture cannot be literally replayed-and-
diffed for full parity: that comparison would be tautological for
freshly-arriving jobs (sim always PENDs them regardless of accounting
correctness) and can't prove multi-round persistence anyway. Built instead
as a real recording plus a hand-crafted sim fixture modeling the identical
facts, including an already-resolved-before-replay job to exercise
reconciliation -- the same documented-carve-out approach test_sim_equivalence.c
already established for PEND-only proofs, applied here to reconciled jobs too.
No production code touched (task 3 already completed the feature). RED
phase proven by temporarily reverting task 3's reconciliation call site:
new test failed with "Expected 10 Was 0", restored and green.
48/49 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This is the integration point that makes the whole feature real
end-to-end. Calls sim_reconcile_unaccounted_jobs() (tasks 2.2-2.4) at
exactly the position design.md's mermaid diagram requires: after
xce_core_begin_session()'s reset, strictly before sched_account_jobs()
(so the same round's re-charge picks up every newly-reconstructed job,
Req 1.3/1.4) and before xce_core_signal_go() unblocks the live picker (so
still-pending jobs' capacity checks this round see correct resInUse, Req
2.1/2.2). decoded_session and record_ctx are the same per-round locals
sched_sim_driver_run() already declares -- no new plumbing.
sim_verify.c needs zero changes: reconciliation consumes/compacts
decoded_session->decisions before the verify drain loop runs each round,
which is what keeps the existing positional comparison correct (Req 5.2)
-- confirmed by extending test_sim_verify.c with a no-false-mismatch test
against a recording containing already-resolved jobs.
New end-to-end test proves the ordering matters for real: a reconciled
job's resInUse is charged before a live sched_check_host_satisfy() call
for a different job in the same round correctly denies it (5.0 wanted vs
2.0 remaining) and grants an exact-fit request (2.0) -- using the real
scheduler.c/resource.c accounting and capacity-check code, not stubs.
Found and fixed a test-authoring bug (get_resvals() returns nres on
success, not 0) during GREEN verification -- logged as bug-1622, a test
bug, not production.
47/48 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds sim_reconcile_unaccounted_jobs(xce_sched_sim_driver_t*, replay_session_t*)
to sim_loop.c -- not yet wired into the round boundary (task 3). Once per
call, finds every RUN-state job with job->accounts == NULL (the signature of
a job already resolved before this replay's own live decision-making ever
considered it) and searches this round's decoded recorded-decision stream
for a matching DISPATCH entry by jobid.
This task implements only the trigger + lookup + not-found path: when no
match exists, the job's allocation is left unknown (not fabricated), no
error is raised, and the job is safe to re-check next round (Requirement
4.1, 4.2). Match-found reconstruction (ownership transfer, freeing/
compacting the consumed entry, reporting) is tasks 2.3/2.4, not implemented
here -- decoded_session's decisions are read but not yet mutated.
6 new tests cover: not-found leaves job unaccounted, cross-round safety,
already-accounted jobs are skipped, a same-jobid PEND entry is correctly
not mistaken for a DISPATCH match, non-RUN-state jobs are gated out, and
NULL-argument safety. RED phase: compile-error-before/link-succeeds-after,
the correct RED variant for genuinely new code with no prior behavior.
46/47 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces sim_loop.c's monolithic xce_core_go() with the same
xce_core_begin_session() -> sched_account_jobs() -> xce_core_signal_go()
sequence driver.c already uses for normal-mode rounds, opening the same
interleaving window and calling the now-core-level accounting service
(task 1.1) at the same relative position. Error-handling for a failed
session start is preserved byte-for-byte (same cleanup, same rc source).
Both interactive and batch simulation runs go through this one call site.
This is bug-1613's wiring half: sim_loop.c now calls sched_account_jobs()
at all (it never did before). resInUse is still zero end-to-end after this
task alone, because job->accounts is never populated by any live path in
sim mode yet -- that's Recorded-Decision Reconciliation (tasks 2.2-2.4),
not yet implemented; a comment marks where it slots in.
Adds a call-count/session-window regression test proving the wiring itself
(RED-phase reproduced live via git stash + rebuild, matching bug-1613's
symptom exactly). 8 sibling white-box test files needed a link-satisfying
sched_account_jobs() stub, discovered via actual linker errors.
45/46 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a dual-use .replay fixture (3 jobs competing for insufficient worker
capacity) that later tasks need to prove capacity-aware parity under
contention: a real recorded normal-mode session (built by driving the
genuine sched_check_host_satisfy()/sched_commit_job_alloc() capacity/
accounting chain, not mocked) that is also directly replayable through the
real simulation driver. Two tests: fixture shape (job B genuinely denied
while A+C hold the full 10.0-unit capacity, verified against the decoded
recorded decision stream) and sim-driver replayability.
RED-phase proven: raising WORKER_CAPACITY to eliminate contention makes the
shape assertion fail as expected before reverting to the genuine-contention
value. 44/45 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Relocates sched_account_jobs()/sched_charge_account()/sched_uncharge_account()/
sched_update_account()/sched_update_res_accounts()/sched_commit_job_alloc()
from xce_scheduler_t* to xce_sched_core_t*, mirroring the sched_resolve_job_picker()
precedent, so the simulation driver can call the same accounting service the
normal-mode driver already uses. Pure signature relocation -- every body
reference was already sched->core.*, no logic change. Updates the 3 real call
sites (driver.c's normal-mode round, scheduler.c's sched_begin_session and
sched_commit_decision) and 7 existing tests' link-satisfying stub signatures.
Adds test_resource_accounting.c proving Requirement 1.4: the relocated
accounting produces identical resInUse/sharedInUse output whether called
against a core embedded in xce_scheduler_t or xce_sched_sim_driver_t.
43/44 scheduler tests pass; the 1 failure is pre-existing bug-1560
(unrelated recording.c sandbox-permission ordering issue).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
xce_scheduler link-wraps printf (-Wl,--wrap=printf, src/meson.build's
wrap_link_args) to route every plain printf() call through tlog, stamping
the full "[timestamp] [pid] [LEVEL] [xce_vprintf]" prefix. sim_print_decision()
(sim_loop.c) used plain printf() for its DISPATCH/PEND/PREEMPT lines, so
those lines carried that prefix while every sibling "[sim] ..." line in
the same REPL/sim output (event echoes, breakpoint hits, banners, info
command output -- all fprintf(stdout, ...)) stayed prefix-free. Found via
a live deployment log. Switched to fprintf(stdout, ...), matching the
established convention everywhere else in this codebase.
Invisible to the entire prior test suite for a structural reason: only
the real xce_scheduler executable links with wrap_link_args, not the
white-box unit-test binaries, so printf() and fprintf(stdout, ...) were
indistinguishable inside any unit test. Added a mechanical regression
guard instead of a runtime test: test_no_bare_printf.c source-scans every
src/scheduler/src/*.c and src/scheduler/src/plugins/*.c file for a bare
printf( call (same source-text-scan genre as test_core_isolation.c).
test_core_isolation.c's build-time proof (nm(1) undefined-symbol scan of
sched_core_isolation_lib) previously covered only core.c -- core.c's own
800-line-cap siblings, holding scheduling logic just as boundary-sensitive
as core.c itself, were "isolation-clean by review" only, with nothing
mechanical catching a regression. Extends sched_core_isolation_lib to
compile all four files together, and the source-text scans (forbidden
headers/symbols, wall-clock calls) to also cover core_events.c/
core_session.c. core_clock.c joins only the nm(1) layer, not the
wall-clock text scan -- it's the one sanctioned exception carrying the
core family's single legitimate clock_gettime() call.
Bugs fixed, discovered across a day of live-operator testing against a
real deployed cluster:
- bug-1584: clock anchors never initialized to real wall-clock values,
timestamps showed system uptime instead of Unix epoch.
- bug-1585: heap-use-after-free in the normal-mode driver's periodic
resval update path -- core_events.c's RESVALS_FULL/_DELTA handlers do a
shallow zarray_put() of the wire-decoded xce_val_t*, but driver.c's
release_translated_payload() unconditionally freed the same decoded
array right after, so the next periodic report for the same
worker+resource triggered a UAF read via kill_sched_resval(). New
regression test: test_resval_ownership.c.
- bug-1587/1588/1591: raw-ns/raw-uint64 jobid/timestamp prints across the
REPL/SIM family converted to the same human-readable formatting already
used elsewhere (fmt_jobid(), timestamp formatting).
- bug-1590: repl_info.c's XCE_VAL_NUMERIC checks corrected to
Value_u_fvalue, fixing `info worker`/`info resource` falling back to
0.00 for real numeric resvals.
task 12.5 (Req 5.22 amendment) -- five REPL gaps surfaced by a live
operator session, each with its own bug id:
- bug-1594: `continue`/`step` reaching EOF with no breakpoint hit now
re-enters the REPL via xce_repl_on_replay_complete() instead of exiting
the process; step/continue issued after replay is already complete
re-prompt instead of silently exiting.
- bug-1595: `step`/`continue` now echo each input event (not just
decisions) via repl_print_event_summary(), in real event/decision
order.
- bug-1593: `info resource <name>|all` now reports resource definition
metadata restricted to XCE_RESF_GLOBAL/XCE_RESF_SHARED resources, read
from core->sharedInst, instead of aggregating host-based resources
across every worker.
- bug-1596: repl_split_tokens() strips one matching pair of surrounding
quote characters from each token.
See requirements.md's Req 5 AC 22 "Amendment (2026-08-19)" for the full
rationale on each of the five task 12.5 gaps.
Full scheduler suite green (41/42; the one remaining failure,
test_recordstart_symlink_escape_replies_error, is the pre-existing
unrelated bug-1560).
xce_scheduler -i -s <replay>: WARN "failed to resolve job picker
plugin (rc=-2)" on load, and every job already present in the
replay file's INITIAL snapshot stayed PEND forever -- never even
reached the picker, let alone got dispatched.
Root cause: sched_sim_driver_start() (sim.c) loads the initial
snapshot via xce_sched_core_load_snapshot() BEFORE any picker
plugin is registered (sim.h's own documented "Known scope
boundary" -- only the picker's NAME is threaded through from the
ReplayHeader). sched_resolve_job_picker() therefore always returns
-ENOENT there, and xce_sched_core_load_snapshot()'s own
picker-rebuild walk (on_queue_update/on_job_push per queue/job) is
silently skipped. The real builtin_fcfs picker only gets registered
moments later in sched_sim_driver_run() -- but nothing then replayed
that rebuild walk, so any job/queue from the INITIAL snapshot (as
opposed to one arriving later via a ReplayEvent, which correctly
reaches the picker through xce_core_push_events()'s own picker!=NULL
check) was permanently invisible to next_job().
Fixed by a new file, sim_picker.c: sim_bootstrap_picker() (replacing
sim_loop.c's sim_register_default_picker(), moved out because
sim_loop.c was already at its own documented 800-line "zero
line-budget headroom" cap) registers the picker AND then replays the
missed rebuild walk against it. Wired into scheduler_srcs,
sched_sim_isolation_lib, and all 9 white-box test TUs that #include
sim_loop.c directly.
New regression test (test_sim_event_loop.c): a job baked into a
fixture's INITIAL snapshot (not a ReplayEvent) now gets a live PEND
decision. RED confirmed by temporarily no-op'ing the rebuild walk
(test failed as expected), GREEN with the real fix. Full scheduler
suite green (41/41, excluding the pre-existing unrelated bug-1560).
Note: sim mode still never DISPATCHes (only PENDs) -- that is a
separate, intentionally out-of-scope gap (real resource-matching
allocation belongs to the not-yet-designed scheduler-allocation
spec), not addressed here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Proves task 3.2's retry-until-success-or-shutdown wrapper's guarantees
hold across multiple queued decisions drained through the real
xce_sched_driver_on_decision(), not just a single isolated call:
decisions queued behind a stuck one publish afterward in original
order (Req 7 AC3), and shutdown mid-retry costs at most one retry
cycle regardless of queue depth -- decisions behind the stuck one see
core.stop already true and are never attempted (Req 7 AC2). Also adds
a source-scan confirming this stays scoped to the scheduler's own
publish direction (Req 7 AC5) and a fresh-instance test confirming a
restart still loses in-flight queued decisions, unchanged by this work
(Req 7 AC6).
Pure test-coverage task -- zero production changes, matching the
review's independently-reproduced RED->GREEN evidence that the
existing task-3.2 implementation already satisfies these guarantees.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds a worst-case-occupancy test proving the scheduler's picker/allocator
pipeline buffers briefly past a stalled publish (decisions cbuf cap 5,
jbuf cap 1) then genuinely blocks until connectivity recovers, with zero
job loss (Req 3 AC2). Adds a source-scan regression guard confirming the
scheduler makes no periodic heartbeat call of its own (Req 3 AC3).
Also fixes a real gap found along the way: sched_init()'s
all-candidates-unreachable and join-failure log lines named only a
candidate count, never which candidates were attempted (same class of
bug worker's task 2.3 fixed). Added sched_format_candidate_list(),
covered by a real dup2()-based stdout capture around a live sched_init()
failure path (round-1 review caught this diagnostic shipping with zero
test coverage; fixed in round 2).
Multi-candidate join and kill-primary/reconnect were validated manually
against a live 3-node cluster rather than committed as an automated
test, following worker's task 2.3 precedent -- no automated live-deploy
E2E harness exists for any C daemon in this repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Wraps sched_publish_dispatch()/sched_publish_pend_reason() in a shared
retry loop that keeps retrying a -E_XCE_NOT_READY-exhausted publish
until it succeeds or the scheduler is shutting down (core.stop checked
before each attempt, not only after), composed with the pending-rejoin
follow-up established in task 3.1. Extended-retry success logs
distinctly from an ordinary first-attempt success (Req 7 AC4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replace scheduler's inline per-candidate loop and separate single-host
branch with one unified pipeline (mirrors worker's task 2.1 fix): build
exactly one validated candidate list, whether from --peers or synthesized
as a single candidate from the legacy default, then connect through the
shared xce_cluster_config_to_client_config() helper exactly once.
Adds sched_on_reconnect(), a rejoin callback that only sets a flag and
returns immediately (no I/O, no xcecli_* calls) to avoid the self-deadlock
worker hit in task 2.3 — registered via xcecli_set_reconnect_handler()
after xcecli_start(), since xcecli_start() resets any handler registered
earlier back to NULL.
Adds test_scheduler_connect.c: behavioral tests confirm no-peers and
single-candidate configs produce identical connect log sequences and that
unreachable candidates fail fast without retry loops; structural tests
source-scan scheduler.c to confirm a single xcecli_ctx_init() call site, a
single xcecli_join() call site, and a rejoin callback body free of I/O and
xcecli_* calls.
Live-validated against a 3-node HA controller cluster: a scheduler
configured with the standby listed first in --peers fails over via
xcecli's built-in retry-next mechanism to the primary and joins
successfully.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYSEeLmzjtD4RLDuUDhLoK
Adds the final task of "Core: Interactive REPL" (Req 5 AC 22): proves
the REPL's own documented invariant -- "step/continue to completion
produces byte-identical output to batch --sim" (design.md SimDriver
Invariants).
test_repl_equivalence.c runs the same sched_sim_driver_run() on the
same 3-session/9-decision fixture twice: once batch (interactive=
false), once REPL-driven via freopen()'d scripted "step 1\ncontinue\n"
stdin into the real repl_command_loop()/readline() path (same
mechanism test_repl_execution_control.c established) -- then
structurally decodes both runs' --sim-output and compares decision
content. Test-only; zero production code changed.
RED-phase probe (planted in xce_repl_on_event(), reverted, zero
residue) confirmed the comparison is real: corrupting one job's id
under interactive mode broke exactly this test plus the 3 other REPL
tests that share the same hook, none else.
Scheduler suite: 37/37 clean (was 36/36). Task group 12 (Interactive
REPL, tasks 12.1-12.4) is now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds `snapshot [FILE]` and `set speed|output|verify ...`/`info config`
(Req 5 AC 22 "Snapshots"/"Simulation parameters") to the interactive
simulation REPL.
`snapshot` reuses sim_record.c's existing sim_record_write_header_and_
snapshot() (task 11.5) rather than reimplementing capnp encoding, so
the produced file is genuinely loadable via a fresh --sim invocation
-- the task's own Observable. `set speed`/`set output`/`set verify`
mutate fields sim_loop.c and core_clock.c already read fresh on every
event/session, so no new hook points were needed and sim_loop.c stays
untouched at exactly 800/800 lines (this project's hard per-file cap).
New repl_params.c is a fourth sibling in the repl.c/repl_break.c/
repl_info.c split (task 12.2's precedent) -- neither existing file had
headroom left under the 800-line cap for this task's own logic.
Scheduler suite: 36/36 clean (was 35/35). Review independently
reproduced the RED-phase probe and additionally verified the
snapshot round-trip Observable is non-vacuous.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the interactive simulation REPL's breakpoint and state-inspection
commands (Req 5 AC 22): `break <event_type> [key=value ...]` / `info
break` / `delete <id>|all` on event and decision types, and `info
session/job/queue/worker/resource/summary`.
repl.c (370 lines after task 12.1) couldn't hold this task's full
surface under the project's 800-line cap, so it's split three ways
(matching the snapshot.c/snapshot_wire.c/snapshot_load.c precedent):
repl.c (execution control + dispatch), repl_break.c (matching/CRUD +
the three new driver-facing hooks), repl_info.c (info dumps), sharing
state via repl_internal.h. sim_loop.c gained 3 new bare hook calls
(repl.h inline wrappers, same pattern as task 12.1's two) at the
points design.md's pseudocode requires -- per decoded event before it
joins the batch, per session boundary, and per drained decision --
landing sim_loop.c at exactly 800/800 lines.
Two real bugs found and fixed during development (bug-1506, bug-1507):
an uninitialized xstr_table left breakpoint filter fields silently
NULL instead of failing loud, and repl_breakpoint_free() called a bare
free() on an RTTI-refcounted makeString() string, corrupting the heap.
Review found one gap: `info job` printed only a resreq presence flag
instead of the actual resreq text, when format_resreq() already
existed in this repo for exactly this purpose. Fixed to match.
Scheduler suite: 35/35 clean (was 34/34). RED-phase evidence (2 real
bugs plus 2 synthetic probes, all reverted with confirmed zero
residue) recorded in tasks.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the interactive simulation REPL's execution-control slice (Req 5
AC 22): a single-threaded input loop (readline-based) supporting
step [N], continue, quit, and help, pausing before the first replay
event and blocking the driver loop while active.
Review round 1 raised two findings, both resolved here:
- GNU readline (GPL) was linked directly into xce_scheduler, a shipped
production daemon. Switched the whole project to libedit (BSD) for
line editing -- scheduler and controller (xce_test) both now use
editline/readline.h, API-compatible (readline()/add_history()).
- Getting a clean regression run after the libedit swap required a
full clean rebuild, which exposed a genuine pre-existing SIGSEGV
(bug-1504): SnapshotData.internalState was a capnp Data field, and
this codebase's capnpc-c backend silently no-ops Data-field
encoding without zero-initializing the caller's stack struct,
leaving an uninitialized capn_ptr written onto the wire. Fixed by
switching internalState to List(UInt8) (with a new internalStateLen
wire field), matching the working FetchFileReply.content/bytesRead
precedent -- no hand-written accessor code needed.
A second, unrelated instance of the same bug class was found in the
controller's WAL-catchup path (WalCatchupEntry.payload) and
deliberately left unfixed as bug-1505, out of scheduler-core's
boundary; needs its own project-wide Data-field audit.
Scheduler suite: 34/34 clean. RED-phase reproduction recorded in
tasks.md (repl_should_pause() step-budget check disabled -> 2/4
tests fail as expected -> reverted -> 34/34 green).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Chain the already-shipped recording (task 9.2) and sim-output (tasks
11.1-11.5) pipelines: record a synthetic multi-queue session through the
normal-mode driver, replay it through the simulation driver with
--sim-output, and assert the two .replay files' decisions match in full
structural content. This is the spec's central validation contract (Req 5
AC 7.1) -- if normal and simulation modes diverge, the core/driver
separation is broken.
Honestly scoped to PEND-only decisions: sim_loop.c's picker thread
(task 11.2) has no code path that can emit a live DISPATCH today, a
pre-existing gap that belongs to the not-yet-designed scheduler-allocation
spec, not something this task can close.
Add sim_record.c (scheduler_t-free .replay writer: ReplayHeader + loaded
SnapshotEntry once, then per-event pass-through and per-session
ReplaySession finalization) and wire --sim-alt-events event-source
selection into sim_loop.c's drain loop, coexisting with task 11.2's
stdout default and task 11.4's --sim-verify. Extract scheduler_t-free
halves out of recording_session.c's begin/end so sim-side code can reuse
the decision-encoding path without gaining an xce_scheduler_t* dependency,
preserving Req 5 AC 17 isolation. Per Req 5 AC 16/19.
Add sim_verify.c (sim_decision_eq/sim_log_mismatch: real structural
comparison of dispatch/pend decision payloads, not a jobid shortcut) and
wire it into sim_loop.c's ReplaySession drain loop via decode_ReplaySession_ptr()
(first real consumer of the generated capnp decode path). --sim-verify now
detects and reports genuine decision divergence with logical-time + detail,
per Req 5 AC 15/18.
Three-layer regression guard (test_sim_isolation.c) proving sim.h/sim.c/
sim_loop.c hold no controller-facing client/cluster-config/capnp-socket
state: source-text scan, struct-body field scan, and nm(1) link-time
undefined-symbol audit of a new isolated sched_sim_isolation_lib target.
Add sim_loop.c implementing sched_sim_driver_run(): decode .replay
messages in file order, jump the virtual clock to each ReplayEvent's
exact timestamp as it's queued, and run a real push-events/go/drain/
end-session cycle on each ReplaySession boundary (AC 14, AC 18).
Task 11.1 left no picker registered against the isolated core and no
thread backing its cbufs, which deadlocks the decision drain the moment
a real event loop exists. Add plugin_register_core() (registers a
picker against a bare xce_sched_core_t*, no xce_scheduler_t* required)
and extract builtin_fcfs_new_plugin() from register_builtin_fcfs() as a
behavior-preserving factory, plus a picker-driving thread for the sim
driver. Since real resource allocation is unreachable without a live
xce_scheduler_t* (scheduler-allocation spec's job), the sim picker
thread conservatively PENDs every candidate rather than dispatching.
28/28 scheduler suite, 32/32 whole-project tests green.
Add sim.h/sim.c implementing AC 13 steps 1-4: open+validate ReplayHeader,
locate the most recent SnapshotEntry at/before --sim-start-ts, load it via
xce_sched_core_load_snapshot(), and set the virtual clock anchors. Wire
all seven simulation CLI flags into main.c's task-5.2 mode-selection
branch. Event loop itself (step 5) deferred to task 11.2. Fixes two real
bugs found during TDD: a SnapshotEntry_get_ts() use-after-free and a
pend_updates NULL-deref in xce_sched_core_fini() for bare-core callers.
27/27 scheduler suite green, full project build clean.
Implements the scheduler's first request-reply opcode (OPCode_schedAdmin).
sched_handle_admin_request() (handler.c) decodes SchedAdminRequest and
dispatches by action: recordStatus answers synchronously (pure read);
recordStart/recordStop are deferred onto the existing pend_updates queue
(XCE_SCHED_OP_ADMIN_RECORD) since sched_record_start()'s snapshot call
requires the session_active==0 precondition only guaranteed on the driver
thread, not the async request-handling thread. driver.c's
xce_sched_driver_translate_pending_events() performs the deferred action
and sends the reply via a new xcecli_reply_schedadmin() (client.c/client.h),
correlated by seq copied before the original request is freed.
Adversarial review found a genuine cross-thread race: sched_record_status()
read driver-owned recording state (including a pointer the driver thread
frees) from the vtask thread with no lock -- the codebase's first such
race, missed by ASan (doesn't detect races). Fixed with a dedicated
record_lock scoped strictly to in-memory field access, never spanning file
I/O; sched_record_status() now strdup()s under the lock instead of
returning a borrowed live pointer. Verified with a real two-thread
lock-contention test (independently confirmed non-vacuous by removing the
lock and observing the test fail) plus a deterministic use-after-free
reproduction.
26/26 scheduler suite, clang-format clean.
Adds recording-state fields directly to xce_scheduler_t (no new
xce_sched_driver_t type, per task 9.1's precedent). sched_record_start()/
sched_record_stop() implement AC 10: validate via task 9.1's path
validator, open with a TOCTOU-hardened sequence (O_NOFOLLOW, no O_TRUNC,
fstat()+st_nlink==1 check before ftruncate()), snapshot the core, write
ReplayHeader+SnapshotEntry on start, ReplayEvent{endSession} on stop.
driver.c gains boolean-gated recording hooks in
xce_sched_driver_translate_pending_events() (per-event-batch ReplayEvent
writes, new recording_events.c) and xce_sched_driver_run_normal_mode_round()
(per-session ReplaySession writes, new recording_session.c) -- zero I/O/
allocation when inactive.
Two real bugs found and fixed during implementation: bug-1498 (O_TRUNC
would have truncated a hard-linked victim file before the TOCTOU check
could reject it) and bug-1499 (a transient edit dropped
xce_core_signal_go(), causing a 3-way thread deadlock).
test_record_lifecycle.c decodes the full start->events->session->stop
cycle with the real capnp decoders and asserts the exact message
sequence; TOCTOU regression tests use real symlink(2)/link(2) syscalls.
25/25 scheduler suite, clang-format clean.
Adds --record-dir CLI flag and sched_validate_record_filename() with
canonical-path containment (realpath()-based, not textual-prefix) plus
separator/".." rejection before any filesystem access.
Adversarial review found and this task fixed two CWE-59 inode-aliasing
bypasses: a direct hard link inside record_dir aliasing an outside file,
and a compound variant where a symlink pointed at an in-directory file
itself hard-linked outside record_dir. Both closed via st_nlink checks
on the direct entry and the fully-resolved symlink target respectively,
each verified with a live RED->GREEN reproduction. TOCTOU residual risk
documented and deferred to task 9.2 (O_NOFOLLOW + fstat() re-check).
18/18 recording subtests, 24/24 scheduler suite, clang-format clean.
Adds xce_sched_core_save_snapshot()/xce_sched_core_load_snapshot(),
split across snapshot.c/snapshot_wire.c/snapshot_load.c (800-line cap)
along real save/load/shared-helper boundaries. Serializes the full
core state (config, jobs, queues, workers, resources, resvals, users,
projects, prng_seed, internal_state) into/out of sched_snapshot_data_t
(task 7.3's committed C mirror of the capnp SnapshotData struct) as
plain in-memory structures -- no wire encoding here, that's a later
task. After load, rebuilds the picker's private state by re-invoking
its state-change hooks (queue updates, then job pushes) against the
restored core state, matching xce_core_push_events()'s own hook-firing
order; picker-private state itself is never stored in the snapshot.
Adds a new prng_seed field to xce_sched_core_t -- requirements.md/
design.md both presuppose it but no prior task added it. Seeded to 0,
no PRNG consumer wired yet (deliberately out of scope).
Cross-core behavioral-equivalence proven directly: two independent
cores, each with its own registered picker, save on one and load into
a freshly-initialized other, both driven through an identical real
session, decision sequences asserted identical in order.
Found and fixed two real bugs along the way: a missing prng_seed
restore in this task's own new code, and a genuine guard bug in
already-committed core.c (sched_get_user()/sched_get_project()
treating key_find()'s expected -ENOENT as fatal, silently breaking
get-or-create for any new name) that also affects the live production
path (make_sched_job(), JOBS_PUSH) -- a narrowly-scoped two-line fix,
independently traced to the primitive level and confirmed correct by
review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Rewires main.c's live scheduling loop onto the new core-driven
architecture: xce_sched_driver_translate_pending_events() (task 5.1)
-> xce_core_begin_session() -> plugin_begin_session()+sched_account_jobs()
-> xce_core_signal_go() -> decision-drain via the registered
xce_core_decision_cb_t -> plugin_end_session() -> xce_core_end_session(),
all in the new exported xce_sched_driver_run_normal_mode_round()
(driver.c), replacing the old inline sched_update()+sched_begin_session()
+sched_go()+switch+sched_end_session() sequence.
xce_core_go() (task 4.4) is split into xce_core_begin_session()/
xce_core_signal_go() -- a pure decomposition (xce_core_go() itself is
now defined in terms of both, task 4.4's approved test suite passes
unmodified) -- specifically so the driver can interleave
plugin_begin_session()/sched_account_jobs() in the exact safety-critical
window task 4.4 deliberately excluded from pure-core scope: after the
resval reset, strictly before the picker thread is signaled. This is
empirically proven by test_driver_session_ordering.c via real
cross-thread happens-before assertions, not just documented.
xce_sched_driver_on_decision() is the new registered callback:
DISPATCH/PEND forward to the existing sched_publish_dispatch()/
sched_publish_pend_reason(); PREEMPT fails loudly (-ENOSYS, logged)
since no xcecli_preempt() client API exists and nothing in the live
allocator produces one yet (scheduler-preemption's territory).
Adds a --sim CLI flag (parse-only, loud not-yet-implemented failure,
since SimDriver is Phase 3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds driver.h/driver.c: xce_sched_driver_translate_pending_events()
drains core->pend_updates via the existing dataq-swap pattern (same
mechanism sched_update() uses), translates all 10 xce_sched_update_
request_t op types into the matching xce_core_event_t, and pushes the
batch through xce_core_push_events(). Scoped to xce_sched_core_t*
alone -- pend_updates already lives on core since task 4.1, and this
task is translation-only (no decision publishing, no mode-selection,
no recording -- all later work). handler.c (the enqueue side) is
untouched. First task in Group 5 (Core: Normal-Mode Driver).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds test_core_full_cycle.c: drives xce_core_push_events() -> xce_core_go()
-> decision drain -> xce_core_end_session() end-to-end against a real
picker pthread (running the real xce_job_picker_iface_t hooks on a
test-local stub picker) and a main-thread stand-in allocator, since
sched_allocator() is xce_scheduler_t*-coupled and out of pure-core
scope. Proves both preconditions (core_go reentrancy, push_events
mid-session) reject against a genuinely active session and recover
once it ends. Closes out group 4 (Foundation: Scheduler Core
Extraction) -- tasks 4.1-4.6 all complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_now() (new core_clock.c, kept separate from core.c so
test_core_isolation.c's wall-clock scan still enforces zero direct
clock_gettime()/time() calls elsewhere in the core): the two-anchor
plus speed formula (virtual_clock_base + (elapsed * speed_num) /
speed_den), covering all four modes (normal, discrete/real-time/fast-
slow simulation) via four new atomic fields on xce_sched_core_t,
defaulted to normal-mode (1/1 speed) in xce_sched_core_init().
requirements.md's literal tlog_reg_format_func(fn, core_ptr) call
shape doesn't exist in the real vendored tlog library (registration
takes only the callback; userptr is per-call-site, not bindable at
registration) -- resolved with a module-level active-core pointer read
by a new core-scoped format function, not wired into mylog_init()/
main.c yet. xce_core_go()/xce_core_end_session() (task 4.4) now stamp
session_start/compute session_duration_ns via xce_core_now(), with
their existing session-active/reentrancy behavior untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Adds xce_core_go()/xce_core_end_session() (new core_session.c):
xce_core_go() is non-blocking, performs begin-session state reset
(select-eval-ctx, resval resets), signals the picker thread via wbuf,
and returns -EINVAL on a reentrant call while a session is already
active (proven a true no-op, not just an early return). Draining
core->decisions and invoking the driver's on_decision callback is the
driver's own responsibility between core_go() and core_end_session()
(confirmed driver-owned, not core-owned, per the Driver <-> Core
protocol). xce_core_set_decision_cb() registers the callback + driver
context as a separate setter, keeping xce_sched_core_init()'s live
sched_init() call site untouched. Adds XCE_DECISION_PREEMPT to
decision.h (jobid/victims[]/reason) alongside the existing DISPATCH/
PEND types. Additive only — sched_go/sched_begin_session/
sched_end_session/sched_allocator/main.c remain untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Defines xce_core_event_type_t/xce_core_event_t (9 event kinds: config,
job, worker, resource, resval full/delta, job state change, job update,
queue update, job rusage) in core.h. Implements xce_core_push_events()
in a new core_events.c (split out of core.c to stay under the 800-line
cap): rejects -EBUSY mid-session, otherwise applies each event's core-
state mutation and fires the corresponding picker hook synchronously,
exactly once per event, before returning. CONFIG_PUSH decomposes the
pushed config's embedded queues/resources/resvals via shared helpers
(also reused by QUEUE_UPDATE's upsert path), mirroring the existing
sched_init_config()/sched_init_resources()/sched_init_sharedInst()/
sched_init_queues(). Narrows sched_resolve_job_picker() to take
xce_sched_core_t* (pure type change, zero behavior change). Purely
additive — not wired into the live scheduling path yet (task 5.x).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Hardcodes jbuf's cbuf capacity to the literal 1 in xce_sched_core_init()
(no parameter path, no local variable to drift), backed by a regression
test. Audits every kvmgr iteration reachable from the scheduling loop
(jobs/queues/users/projects): only jobs is ever iterated, via xzarray's
deterministic ascending-slot walk with no process-random seeding —
findings recorded durably as a comment above xce_sched_core_init().
core->workers iteration-order stability is flagged as an open follow-up
(scheduler-allocation's territory), not this task's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Splits xce_scheduler_t along the controller-I/O seam: xce_sched_core_t
(new core.h/core.c) now holds all scheduling state (jobs/queues/workers/
users/projects/resource managers/plugins/three-thread-cbuf pipeline),
while xce_scheduler_t shrinks to {core; apictx; cluster_cfg}. Proven via
an nm-based link-time isolation test (zero netio/xcecli/idgen/delaymgr
symbols reachable from core.c) plus a reproducible wall-clock-call guard.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dw5WnN3DXStaEaPYfACU1
Closes out group 2 (Core: Builtin FCFS Picker). Confirms builtin_fcfs's
ordering matches legacy fcfs_elect_job (byte-identical comparators, same
sched_queue_is_ready/sched_job_is_ready gates, provably equivalent jobid
sequence), confirms tasks 1.1-2.5 left every pre-existing test file
unmodified, and confirms resreq immutability, pend-reason factory-only
allocation, and select_eval_ctx_t memoization all hold across every file
this spec has touched so far. Adds a durable regression lock for the
pend-reason factory invariant (no direct calloc(pendreason_t) outside
make_pendreason()).
Task 2.4 already made sched_job_picker() delegate entirely to the picker
interface with zero direct queue/job traversal, satisfying Req 1 AC 1/AC 7
as a byproduct. This adds a source-text regression test locking that
property in place: asserts sched_job_picker()'s body contains none of the
forbidden direct-traversal symbols and does contain the expected
picker-interface call sites.
Registers builtin_fcfs at scheduler startup and makes it the default
job_pick_plugin, replacing the hardcoded "xce_sched_fcfs" literal in
sched_job_picker() with a fail-loud lookup through sched_resolve_job_picker().
The picker thread now branches legacy job_elect_fn plugins (delegated
unchanged, preserving their self-contained wbuf-driven loop) from native
xce_job_picker_iface_t pickers (driven by a new per-session loop), and the
allocator now calls on_dispatch/on_pend for every picker as Req 1 AC 4
requires. Closes task 2.3's flagged picker/allocator concurrency race with a
mutex on fcfs_plugin_data_t, and adds cancel points to next_job().
Adds begin_session/next_job/end_session/on_dispatch/on_pend, driving a
dual-cursor walk (queue then job) over builtin_fcfs's private ordq_t
state. next_job() returns candidates in (queue priority desc, job
priority desc, queue_time asc) order and NULL once exhausted;
on_dispatch removes permanently, on_pend is a true no-op.
Review flagged an unsynchronized cross-thread access risk between
next_job() (picker thread) and on_dispatch/on_pend (allocator thread)
once wired live in task 2.4 — recorded in tasks.md Implementation Notes
for that task to close before registering builtin_fcfs as the default.
Requirements: 3.2, 3.4, 3.6
New src/scheduler/src/plugins/builtin_fcfs.c implements
builtin_fcfs_init()/fini() and the private fcfs_plugin_data_t (ordq_t
queues, per-queue ordq_t pendJobs) that all later builtin_fcfs hooks
build on. fcfs.c (today's live job_elect_fn-based picker) is untouched
-- still production, until task 2.4 flips the registration.
picker == xce_sched_plugin_t* calling convention (private state stored
in plugin->data) confirmed consistent with task 1.2's legacy wrapper.
Requirements: 3.1 (context), 3.3
plugin_register() now adapts an old-style job_elect_fn-only plugin to
xce_job_picker_iface_t via legacy_picker_wrapper_iface (next_job()
forwards to job_elect_fn; other hooks are safe no-ops covered elsewhere
in the existing lifecycle), logging a one-time deprecation warning. A
plugin that already sets picker_iface directly skips the wrapper and
the warning. Reuses the existing plugin_register()/key_find() mechanism
-- no parallel registration system.
Requirements: 1.6, 2.5
