Adds an atomic degraded-until state (health.go: IsDegraded/MarkDegraded/
MarkHealthy) that lets request handlers fail fast with a clear
unavailable error instead of blocking on xcecli's own ~100s retry
budget when the backend is known-degraded.
Consolidates what tasks 4.3/4.4 left as duplicated inline
"call xcecli_*; checkAndPerformRejoin()" pairs across five wrapper
functions into one shared callBackend() helper: checks IsDegraded()
before every call (fail fast, skip the C call and the rejoin check
entirely on the degraded path -- checking rejoin there would itself
block up to ~100s, defeating the fail-fast purpose), marks degraded on
-E_XCE_NOT_READY, marks healthy at checkAndPerformRejoin()'s existing
success point (the single call site both the write path and the
idle-period heartbeat path already funnel through).
The circuit breaker's own TTL-based state is deliberately never read
by the rejoin-retry mechanism -- checkAndPerformRejoin() has no
IsDegraded() dependency, so a fast-fail window never suppresses
retry-until-actually-joined; heartbeat.go's unconditional per-tick
call keeps that cadence alive regardless of degraded state, and a
10s TTL (well under the ~100s retry budget) lets write traffic resume
contributing once it clears.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
xcecli_heartbeat() is XCECLI_FF_CALL-wrapped -- a deliberately
non-retrying fire-and-forget path that does its own best-effort
one-shot reconnect but never calls xcecli_invoke_rejoin() (that only
fires from the write-call retry path). Left as-is, a controller
failover with no in-flight write request would silently repair the
transport connection without ever re-joining, permanently starving
push notifications.
runHeartbeat() gains two callback hooks fired at the right points --
onFailureToSuccess on a genuine transition, onSuccessTick on every
successful tick thereafter -- wired directly to task 4.3's existing
setNeedsRejoin()/checkAndPerformRejoin() from StartHeartbeat(). No
second rejoin mechanism: checkAndPerformRejoin() is a cheap no-op when
nothing is pending, so calling it unconditionally on every successful
tick is safe and matches the requirement that retry isn't gated on the
transition edge alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Registers a rejoin flag-setter (go_backend_rejoin_handler) as the
client library's reconnect callback: zero I/O, sets an atomic flag,
returns immediately -- avoiding self-deadlock on the library's
per-ctx failover lock the same way worker's and scheduler's rejoin
callbacks already do. The C shim, its extern declaration, and the
xcecli_set_reconnect_handler() registration call all live together in
clib.go's xce_backend_init(), placed after xcecli_start()'s success
check (xcecli_start() wipes any earlier-registered handler as part of
its own init -- the same ordering bug found and fixed in worker's/
scheduler's task 2.3/3.1 work).
A shared checkAndPerformRejoin() helper wraps every gCtx.ctx-based
backend call (SubmitJob, ControlJob, ControlQueue, ControlHost,
FetchFile): after the call returns, if a rejoin is pending, issue
xcecli_join() as a separate step and clear the flag only on success --
on failure it stays set so the next backend call's return retries it,
no separate retry loop.
needsRejoin is a proper sync/atomic flag rather than a plain bool,
since apiserver (unlike the single-threaded C daemons) serves
concurrent HTTP requests that can all reach the shared helper at once.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replaces apiserver's two connect mechanisms -- a per-peer Go loop that
hardcoded nmasters=1 per attempt, and a separate direct-construction
path for the legacy single-host config -- with one pipeline:
buildClusterConfig() parses (via task 4.1's C parser) or synthesizes a
single-candidate config, validates it, then ConfigConverter
(xce_cluster_config_to_client_config) produces the client config for
one xce_backend_init() call. A 3-candidate config now actually
exercises the C library's own failover/standby-redirect handling on
initial connect, matching worker's and scheduler's task 2.1/3.1 work,
instead of a Go-level per-candidate retry loop that never built a real
multi-master config.
Along the way: found and fixed a genuine use-after-free. The deleted
tryConnect() freed its C.CString-allocated host address immediately
after its single connect call returned, but xcecli_ctx_init() stores a
raw pointer to the config forever (client.c:38) and every later
failover rotation re-dereferences it (client_failover.c) -- so any
reconnect after the first would read freed memory. Fixed by giving the
cluster/client config structs package-level, process-lifetime storage
(gClusterCfg/gClientCfg), mirroring worker's and scheduler's file-scope
static pattern, with host/from/key written into fixed-size C arrays
instead of freed heap pointers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replaces apiserver's hand-written Go parsePeer() with a CGO wrapper
around the shared C parser+validator pair (xce_parse_peers then
xce_cluster_config_validate, both src/lib/cluster.c) that worker and
scheduler already use from their own main.c's. Gives apiserver the same
[name=]host:port[,...] syntax (including the name= prefix the old Go
parser could not express) and identical malformed-candidate rejection
(empty host, port <= 0, non-numeric port) -- round-1 review caught the
first pass skipping the validate() half, silently accepting inputs
worker/scheduler both reject.
A malformed entry now rejects the whole candidate list rather than
being skipped with a warning (matches xce_parse_peers()'s own
all-or-nothing construction and worker/scheduler's precedent) --
surfaced loudly via RegisterToBackend's error, not silently.
Parsing only -- RegisterToBackend()'s connect loop and tryConnect()
are unchanged; collapsing them into a single connect call is task
4.2's job.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Proves task 3.2's retry-until-success-or-shutdown wrapper's guarantees
hold across multiple queued decisions drained through the real
xce_sched_driver_on_decision(), not just a single isolated call:
decisions queued behind a stuck one publish afterward in original
order (Req 7 AC3), and shutdown mid-retry costs at most one retry
cycle regardless of queue depth -- decisions behind the stuck one see
core.stop already true and are never attempted (Req 7 AC2). Also adds
a source-scan confirming this stays scoped to the scheduler's own
publish direction (Req 7 AC5) and a fresh-instance test confirming a
restart still loses in-flight queued decisions, unchanged by this work
(Req 7 AC6).
Pure test-coverage task -- zero production changes, matching the
review's independently-reproduced RED->GREEN evidence that the
existing task-3.2 implementation already satisfies these guarantees.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds a worst-case-occupancy test proving the scheduler's picker/allocator
pipeline buffers briefly past a stalled publish (decisions cbuf cap 5,
jbuf cap 1) then genuinely blocks until connectivity recovers, with zero
job loss (Req 3 AC2). Adds a source-scan regression guard confirming the
scheduler makes no periodic heartbeat call of its own (Req 3 AC3).
Also fixes a real gap found along the way: sched_init()'s
all-candidates-unreachable and join-failure log lines named only a
candidate count, never which candidates were attempted (same class of
bug worker's task 2.3 fixed). Added sched_format_candidate_list(),
covered by a real dup2()-based stdout capture around a live sched_init()
failure path (round-1 review caught this diagnostic shipping with zero
test coverage; fixed in round 2).
Multi-candidate join and kill-primary/reconnect were validated manually
against a live 3-node cluster rather than committed as an automated
test, following worker's task 2.3 precedent -- no automated live-deploy
E2E harness exists for any C daemon in this repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Wraps sched_publish_dispatch()/sched_publish_pend_reason() in a shared
retry loop that keeps retrying a -E_XCE_NOT_READY-exhausted publish
until it succeeds or the scheduler is shutting down (core.stop checked
before each attempt, not only after), composed with the pending-rejoin
follow-up established in task 3.1. Extended-retry success logs
distinctly from an ordinary first-attempt success (Req 7 AC4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Scheduler's unified connect pipeline is implemented, tested (RED→GREEN
verified), and live-validated against a 3-node HA controller cluster
(standby-first candidate list joins correctly via xcecli failover). See
a9f283d.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYSEeLmzjtD4RLDuUDhLoK
Replace scheduler's inline per-candidate loop and separate single-host
branch with one unified pipeline (mirrors worker's task 2.1 fix): build
exactly one validated candidate list, whether from --peers or synthesized
as a single candidate from the legacy default, then connect through the
shared xce_cluster_config_to_client_config() helper exactly once.
Adds sched_on_reconnect(), a rejoin callback that only sets a flag and
returns immediately (no I/O, no xcecli_* calls) to avoid the self-deadlock
worker hit in task 2.3 — registered via xcecli_set_reconnect_handler()
after xcecli_start(), since xcecli_start() resets any handler registered
earlier back to NULL.
Adds test_scheduler_connect.c: behavioral tests confirm no-peers and
single-candidate configs produce identical connect log sequences and that
unreachable candidates fail fast without retry loops; structural tests
source-scan scheduler.c to confirm a single xcecli_ctx_init() call site, a
single xcecli_join() call site, and a rejoin callback body free of I/O and
xcecli_* calls.
Live-validated against a 3-node HA controller cluster: a scheduler
configured with the standby listed first in --peers fails over via
xcecli's built-in retry-next mechanism to the primary and joins
successfully.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYSEeLmzjtD4RLDuUDhLoK
Confirm connector thread is isolated from worker_process_update() (main
thread) and resource_sync_thread() (own thread) via code read plus a real
pthread concurrency test. Add unit and live 3-node E2E coverage for
candidate/reconnect behavior, all-candidates-unreachable diagnostics, and
liveness cadence under concurrent reconnect.
The E2E run surfaced two production bugs the 2.1/2.2 unit tests missed:
xcecli_start() resets the reconnect callback to NULL after it runs, so
registering it before xcecli_start() silently no-ops the whole rejoin
mechanism; and a successful worker join returns its assigned slot index
as reply.status (any rc >= 0), not always exactly 0, so checking_connection()'s
rc == 0 success check never fired on a real non-zero-slot join. Both fixed,
plus per-candidate names added to the all-unreachable failure log.
Root-check bypass needed for the E2E run is gated at compile time via the
existing VIRTUAL_CLUSTER flag, not a runtime env var, so the real xcecd
binary's root requirement has zero bypass surface.
Thread cluster_cfg by value through xcecd_config_t -> xcecd_context_t ->
cc_ctx so it's populated before controller_connector_start() fires the
first join, instead of after xcecd_start() returns. checking_connection()
now retries a failed post-reconnect xcecli_join() on the next liveness
probe instead of leaving needs_rejoin cleared on mere attempt.
Replace worker's separate default/single-host and multi-candidate connect
branches with one path: synthesize-or-accept a candidate list, validate,
convert via xce_cluster_config_to_client_config(), connect once. Register
a flag-only rejoin callback (no I/O, no client-lib reentry) and remove the
liveness probe's bounded give-up so the library owns retry/recovery.
Task 1.3's full checklist (constant-equality assertion, candidate
order, one-candidate-no-branch, truncation-with-warning, NULL/empty
rejection) was already satisfied by tasks 1.1 and 1.2's test
coverage. Verified fresh against the current test suite; no
implementer/reviewer cycle needed since there is no new diff.
xce_cluster_config_to_client_config() converts a validated candidate
list into a fully-populated xce_client_config_t, copying candidates
in order with no special-casing for a single candidate. The
truncate-with-warning safety net stays in place as a defensive guard
now that N_MAX_MASTERS and MAX_HA_CANDIDATES match.
Also removes a WARN check in xce_cluster_config_validate() that
became unreachable dead code once task 1.1 reconciled the two
candidate-count constants (both checks tested the same field against
the same constant with no mutation between them).
Task 1.2 of component-failover-adoption (Requirement 1 AC1-AC7).
N_MAX_MASTERS (client.h) now derives from config.h's MAX_HA_CANDIDATES
instead of an independently-maintained literal (3 vs 4 previously).
Fixes worker's and scheduler's hardcoded `> 3` startup checks to
reference the constant symbolically, and updates the client-retry
test fixture so it doesn't exercise an uninitialized 4th slot.
Task 1.1 of component-failover-adoption (Requirement 1 AC7).
- Add OPCode_probeConfigTag dispatch in cluster.c
- Implement probe_config_tag handler returning config tag via
standard xce_reply() → RequestReply envelope
- Populate redirectHost/redirectPort for follow_redirects support
- Wire probeConfigTag reply through typemap
- Add probeConfigTagReply struct to controller.capnp schema
- probe_config() now wraps probe in Request envelope matching push_config()
- Controller OPCode_probeConfigTag handler replies through xce_reply()
with redirectHost/redirectPort populated
- follow_redirects works identically for both push and probe flows
- Change default record_dir from /var/log/xce/ to /tmp (always exists)
- Add mkdir(record_dir, 0755) at startup for custom --record-dir paths
- Split validation_error_text into 6 distinct per-errno messages
- Accept human-readable job IDs in 'info job' via parse_jobid()
- Add 'info jobs' command to list all jobs in sim REPL
- Format job IDs with fmt_jobid() for xce_test-consistent output
- Replace printf with fprintf(stdout, ...) in all REPL files
to bypass -Wl,--wrap=printf linker interception
The "Deploying the Scheduler" section predated scheduler-core and only
covered normal-mode startup. Adds session recording (--record-dir,
xceadmin scheduler record start|stop|status, filename validation),
xceadmin scheduler dump for offline .replay inspection, and simulation
mode (--sim, --sim-speed, --sim-verify, --sim-alt-events, --sim-output,
--interactive REPL) — all verified against the actual argp help output
and xceadmin cli.py source, not assumed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The file was untouched since before scheduler-core and still described
a plain plugin-selects-at-runtime engine with a generic meson build
command. Documents the actual core/driver split, the picker plugin
interface and its builtin_fcfs/fcfs.c split, the simulation driver and
replay/recording machinery, and the interactive REPL, plus the correct
/xce-build invocation and the project's 800-line-file-cap convention
this module's several *_events.c/*_session.c-style splits follow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
These edits (three-thread picker correction in brief.md, record_dir
path-validation and kvmgr-audit requirements in requirements.md,
OPCode_schedAdmin folded into the peer-authentication-hardening
backlog item in roadmap.md) were made during earlier requirements
review and design work but left uncommitted. Reviewed for accuracy
against the completed implementation — all still correct, persisting
as-is.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found by /kiro-validate-impl: tasks 2.1-2.6 (builtin_fcfs picker) were
implemented, reviewed, and committed on 2026-08-03 but never got a
_Verified: line, unlike every other task group. Backfills each from
git history (commits f7921b96, b7214ebe, fcb32a8b, 7ee9a16f, 9d91d470,
604a5649) and direct source inspection.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both files existed only on disk since the design phase (2026-08-03) and
were never committed despite ~40 implementation commits landing since.
Also corrects design.md drift found by /kiro-validate-impl: the File
Structure Plan and BuiltinFcfsPicker sections claimed plugins/fcfs.c
would be rewritten in place, but the actual implementation added a new
plugins/builtin_fcfs.c and deliberately kept fcfs.c untouched as the
job_elect_fn compat-wrapper's exercise target. Updates the directory
tree to the real file layout (core_events/session/clock.c, sim.c/
sim_loop.c/sim_record.c/sim_verify.c, snapshot/recording/repl splits,
test_e2e_capnp.c vs the planned test_replay_format.c, test_sched_admin_dispatch.c
vs test_sched_admin.c), and adds two Open Questions/Risks entries for
the AC 3 tlog-logging-integration gap and this fcfs.c/builtin_fcfs.c
documentation gap, both found during validation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
/kiro-validate-impl pass: GO. Two non-blocking follow-ups found and
documented — AC 3's tlog virtual-clock logging integration is built
but never activated (root-caused to sim.c's sched_sim_driver_start,
deferred pending a matching deactivation/fini design), and
plugins/fcfs.c vs builtin_fcfs.c is functionally correct but
undocumented against design.md and missing _Verified notes on 2.1-2.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Clean rebuild (asan, all components) then 15-round sequential and
15-round chaotic HA soak against the fully integrated feature. Both
PASS with zero job loss, zero committed-prefix divergence, zero
liveness stall. Closes scheduler-core: all 45 tasks complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
controller_unit_test crashed deterministically in write_WalCatchupEntry
(test_wal_catchup_append_term_fields_round_trip) -- the same bug class
as bug-1504 (SnapshotData.internalState): this codebase's capnpc-c
backend silently no-ops Data-field encoding in generated encode_XXX()
composite helpers without zero-initializing the caller's local struct
first, leaving an uninitialized capn_ptr written onto the wire.
Confirmed zero production exposure before fixing: xce_wal_batch_encode_
raw()/wal_entry_to_wire() (request.c) never call the generic
encode_WalCatchupEntry() at all -- they memset() their own struct and
call write_WalCatchupEntry() directly with every field hand-set, so
this was 100% test-only exposure (grepped every caller of the generic
encode/decode pair; only the crashing test uses it).
Fix mirrors bug-1504: WalCatchupEntry.payload (@2) and batchPayloads
(@10) switched from Data to List(UInt8) in controller.capnp (new
batchPayloadsLen @11 companion; payload reuses the existing
wirePayloadLen @9), typemap.h's wal_catchup_entry_t field types
updated, and request.c's extract_data_bytes() widened to take a raw
capn_ptr (it already only used .p internally) so its call sites work
regardless of which Data-shaped wrapper the generated accessor
returns now. batchPayloads' single-top-level-blob shape confirmed
this does not reintroduce bug-847's per-element cross-segment
far-pointer loss -- that issue is inherent to per-element data
regardless of Data vs List(UInt8) spelling.
Strengthened test_wal_catchup_append_term_fields_round_trip with real
payload/batchPayloads content assertions -- previously never
exercised, since the test's src fields were left NULL.
Verified: controller suite 3/3 (was 2/3, crashing), scheduler suite
unaffected at 37/37 (unrelated schema area).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the final task of "Core: Interactive REPL" (Req 5 AC 22): proves
the REPL's own documented invariant -- "step/continue to completion
produces byte-identical output to batch --sim" (design.md SimDriver
Invariants).
test_repl_equivalence.c runs the same sched_sim_driver_run() on the
same 3-session/9-decision fixture twice: once batch (interactive=
false), once REPL-driven via freopen()'d scripted "step 1\ncontinue\n"
stdin into the real repl_command_loop()/readline() path (same
mechanism test_repl_execution_control.c established) -- then
structurally decodes both runs' --sim-output and compares decision
content. Test-only; zero production code changed.
RED-phase probe (planted in xce_repl_on_event(), reverted, zero
residue) confirmed the comparison is real: corrupting one job's id
under interactive mode broke exactly this test plus the 3 other REPL
tests that share the same hook, none else.
Scheduler suite: 37/37 clean (was 36/36). Task group 12 (Interactive
REPL, tasks 12.1-12.4) is now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds `snapshot [FILE]` and `set speed|output|verify ...`/`info config`
(Req 5 AC 22 "Snapshots"/"Simulation parameters") to the interactive
simulation REPL.
`snapshot` reuses sim_record.c's existing sim_record_write_header_and_
snapshot() (task 11.5) rather than reimplementing capnp encoding, so
the produced file is genuinely loadable via a fresh --sim invocation
-- the task's own Observable. `set speed`/`set output`/`set verify`
mutate fields sim_loop.c and core_clock.c already read fresh on every
event/session, so no new hook points were needed and sim_loop.c stays
untouched at exactly 800/800 lines (this project's hard per-file cap).
New repl_params.c is a fourth sibling in the repl.c/repl_break.c/
repl_info.c split (task 12.2's precedent) -- neither existing file had
headroom left under the 800-line cap for this task's own logic.
Scheduler suite: 36/36 clean (was 35/35). Review independently
reproduced the RED-phase probe and additionally verified the
snapshot round-trip Observable is non-vacuous.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the interactive simulation REPL's breakpoint and state-inspection
commands (Req 5 AC 22): `break <event_type> [key=value ...]` / `info
break` / `delete <id>|all` on event and decision types, and `info
session/job/queue/worker/resource/summary`.
repl.c (370 lines after task 12.1) couldn't hold this task's full
surface under the project's 800-line cap, so it's split three ways
(matching the snapshot.c/snapshot_wire.c/snapshot_load.c precedent):
repl.c (execution control + dispatch), repl_break.c (matching/CRUD +
the three new driver-facing hooks), repl_info.c (info dumps), sharing
state via repl_internal.h. sim_loop.c gained 3 new bare hook calls
(repl.h inline wrappers, same pattern as task 12.1's two) at the
points design.md's pseudocode requires -- per decoded event before it
joins the batch, per session boundary, and per drained decision --
landing sim_loop.c at exactly 800/800 lines.
Two real bugs found and fixed during development (bug-1506, bug-1507):
an uninitialized xstr_table left breakpoint filter fields silently
NULL instead of failing loud, and repl_breakpoint_free() called a bare
free() on an RTTI-refcounted makeString() string, corrupting the heap.
Review found one gap: `info job` printed only a resreq presence flag
instead of the actual resreq text, when format_resreq() already
existed in this repo for exactly this purpose. Fixed to match.
Scheduler suite: 35/35 clean (was 34/34). RED-phase evidence (2 real
bugs plus 2 synthetic probes, all reverted with confirmed zero
residue) recorded in tasks.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the interactive simulation REPL's execution-control slice (Req 5
AC 22): a single-threaded input loop (readline-based) supporting
step [N], continue, quit, and help, pausing before the first replay
event and blocking the driver loop while active.
Review round 1 raised two findings, both resolved here:
- GNU readline (GPL) was linked directly into xce_scheduler, a shipped
production daemon. Switched the whole project to libedit (BSD) for
line editing -- scheduler and controller (xce_test) both now use
editline/readline.h, API-compatible (readline()/add_history()).
- Getting a clean regression run after the libedit swap required a
full clean rebuild, which exposed a genuine pre-existing SIGSEGV
(bug-1504): SnapshotData.internalState was a capnp Data field, and
this codebase's capnpc-c backend silently no-ops Data-field
encoding without zero-initializing the caller's stack struct,
leaving an uninitialized capn_ptr written onto the wire. Fixed by
switching internalState to List(UInt8) (with a new internalStateLen
wire field), matching the working FetchFileReply.content/bytesRead
precedent -- no hand-written accessor code needed.
A second, unrelated instance of the same bug class was found in the
controller's WAL-catchup path (WalCatchupEntry.payload) and
deliberately left unfixed as bug-1505, out of scheduler-core's
boundary; needs its own project-wide Data-field audit.
Scheduler suite: 34/34 clean. RED-phase reproduction recorded in
tasks.md (repl_should_pause() step-budget check disabled -> 2/4
tests fail as expected -> reverted -> 34/34 green).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Chain the already-shipped recording (task 9.2) and sim-output (tasks
11.1-11.5) pipelines: record a synthetic multi-queue session through the
normal-mode driver, replay it through the simulation driver with
--sim-output, and assert the two .replay files' decisions match in full
structural content. This is the spec's central validation contract (Req 5
AC 7.1) -- if normal and simulation modes diverge, the core/driver
separation is broken.
Honestly scoped to PEND-only decisions: sim_loop.c's picker thread
(task 11.2) has no code path that can emit a live DISPATCH today, a
pre-existing gap that belongs to the not-yet-designed scheduler-allocation
spec, not something this task can close.
Add sim_record.c (scheduler_t-free .replay writer: ReplayHeader + loaded
SnapshotEntry once, then per-event pass-through and per-session
ReplaySession finalization) and wire --sim-alt-events event-source
selection into sim_loop.c's drain loop, coexisting with task 11.2's
stdout default and task 11.4's --sim-verify. Extract scheduler_t-free
halves out of recording_session.c's begin/end so sim-side code can reuse
the decision-encoding path without gaining an xce_scheduler_t* dependency,
preserving Req 5 AC 17 isolation. Per Req 5 AC 16/19.
Add sim_verify.c (sim_decision_eq/sim_log_mismatch: real structural
comparison of dispatch/pend decision payloads, not a jobid shortcut) and
wire it into sim_loop.c's ReplaySession drain loop via decode_ReplaySession_ptr()
(first real consumer of the generated capnp decode path). --sim-verify now
detects and reports genuine decision divergence with logical-time + detail,
per Req 5 AC 15/18.
Three-layer regression guard (test_sim_isolation.c) proving sim.h/sim.c/
sim_loop.c hold no controller-facing client/cluster-config/capnp-socket
state: source-text scan, struct-body field scan, and nm(1) link-time
undefined-symbol audit of a new isolated sched_sim_isolation_lib target.
Add sim_loop.c implementing sched_sim_driver_run(): decode .replay
messages in file order, jump the virtual clock to each ReplayEvent's
exact timestamp as it's queued, and run a real push-events/go/drain/
end-session cycle on each ReplaySession boundary (AC 14, AC 18).
Task 11.1 left no picker registered against the isolated core and no
thread backing its cbufs, which deadlocks the decision drain the moment
a real event loop exists. Add plugin_register_core() (registers a
picker against a bare xce_sched_core_t*, no xce_scheduler_t* required)
and extract builtin_fcfs_new_plugin() from register_builtin_fcfs() as a
behavior-preserving factory, plus a picker-driving thread for the sim
driver. Since real resource allocation is unreachable without a live
xce_scheduler_t* (scheduler-allocation spec's job), the sim picker
thread conservatively PENDs every candidate rather than dispatching.
28/28 scheduler suite, 32/32 whole-project tests green.
Add sim.h/sim.c implementing AC 13 steps 1-4: open+validate ReplayHeader,
locate the most recent SnapshotEntry at/before --sim-start-ts, load it via
xce_sched_core_load_snapshot(), and set the virtual clock anchors. Wire
all seven simulation CLI flags into main.c's task-5.2 mode-selection
branch. Event loop itself (step 5) deferred to task 11.2. Fixes two real
bugs found during TDD: a SnapshotEntry_get_ts() use-after-free and a
pend_updates NULL-deref in xce_sched_core_fini() for bare-core callers.
27/27 scheduler suite green, full project build clean.
Add real CliRunner-through-real-mock-TLS-server coverage for scheduler
record start/stop/status (incl. standby->primary redirect handoff) and
dump, closing the gap task 10.5 left open (which only monkeypatched the
CLI's function bindings). 227/227 xceadmin tests green. Closes Group 10.
Add scheduler command group with nested record sub-group (start/stop/status)
and sibling dump command, delegating to the client functions from 10.3/10.4
with no business logic in the CLI layer. 220/220 xceadmin tests green.
Adds dump() to scheduler.py -- a pure local .replay file decoder, zero
network I/O. Key finding: pycapnp's from_bytes() does not validate a
message against the schema type you ask it to decode as (trial-decoding
SnapshotEntry bytes as ReplayEvent raises nothing, returns plausible
garbage), ruling out trial-decode-and-catch for message-type
discrimination. Instead reads each message's raw segment-table/
root-pointer framing to extract its (dataWordCount, pointerCount) shape,
computed live from the schema, and matches against the four replay
message types' distinct shapes before decoding. AC 20's stream-ordering
rules (msg 1 = ReplayHeader, msg 2 = SnapshotEntry) are enforced
structurally, not inferred.
Adversarial review reproduced a real (if forward-looking) gap: an actual
modified schema with a shape collision caused an uncaught raw capnp
exception to escape dump(). Closed with a pairwise-distinctness
assertion right after the four shapes are computed, raising a typed
ReplayFileError before any message is decoded if a future schema change
ever makes two shapes collide -- restoring dump()'s own docstring
guarantee unconditionally. New regression test simulates the exact
collision the review found.
201 passed, 3 skipped (up from 188 pre-task).
Adds record_start()/record_stop()/record_status() to new
src/xceadmin/src/xceadmin/scheduler.py, following push_config()'s exact
pipeline shape (load_schema -> follow_redirects -> decode_reply). Deliberately
diverges from push_config()'s "return failure as data" convention because
this task's own Observable text requires raising a typed error on an error
reply, not just connection/decode failure -- a new SchedAdminError
(xcecommon/errors.py) carries the full reply state (error/recording/file/
event_count/started_at) when SchedAdminReply.ok is false.
test_scheduler.py mirrors test_config_push_mock.py's real mock-TLS-server
technique, covering all three actions, redirect-following (standby-to-
primary), and the error-reply-raises-SchedAdminError path across the
error codes reference table.
188 passed, 3 skipped via the real xceadmin test runner (meson compile
-C xce.build xceadmin).
