Adds an atomic degraded-until state (health.go: IsDegraded/MarkDegraded/
MarkHealthy) that lets request handlers fail fast with a clear
unavailable error instead of blocking on xcecli's own ~100s retry
budget when the backend is known-degraded.
Consolidates what tasks 4.3/4.4 left as duplicated inline
"call xcecli_*; checkAndPerformRejoin()" pairs across five wrapper
functions into one shared callBackend() helper: checks IsDegraded()
before every call (fail fast, skip the C call and the rejoin check
entirely on the degraded path -- checking rejoin there would itself
block up to ~100s, defeating the fail-fast purpose), marks degraded on
-E_XCE_NOT_READY, marks healthy at checkAndPerformRejoin()'s existing
success point (the single call site both the write path and the
idle-period heartbeat path already funnel through).
The circuit breaker's own TTL-based state is deliberately never read
by the rejoin-retry mechanism -- checkAndPerformRejoin() has no
IsDegraded() dependency, so a fast-fail window never suppresses
retry-until-actually-joined; heartbeat.go's unconditional per-tick
call keeps that cadence alive regardless of degraded state, and a
10s TTL (well under the ~100s retry budget) lets write traffic resume
contributing once it clears.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
xcecli_heartbeat() is XCECLI_FF_CALL-wrapped -- a deliberately
non-retrying fire-and-forget path that does its own best-effort
one-shot reconnect but never calls xcecli_invoke_rejoin() (that only
fires from the write-call retry path). Left as-is, a controller
failover with no in-flight write request would silently repair the
transport connection without ever re-joining, permanently starving
push notifications.
runHeartbeat() gains two callback hooks fired at the right points --
onFailureToSuccess on a genuine transition, onSuccessTick on every
successful tick thereafter -- wired directly to task 4.3's existing
setNeedsRejoin()/checkAndPerformRejoin() from StartHeartbeat(). No
second rejoin mechanism: checkAndPerformRejoin() is a cheap no-op when
nothing is pending, so calling it unconditionally on every successful
tick is safe and matches the requirement that retry isn't gated on the
transition edge alone.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Registers a rejoin flag-setter (go_backend_rejoin_handler) as the
client library's reconnect callback: zero I/O, sets an atomic flag,
returns immediately -- avoiding self-deadlock on the library's
per-ctx failover lock the same way worker's and scheduler's rejoin
callbacks already do. The C shim, its extern declaration, and the
xcecli_set_reconnect_handler() registration call all live together in
clib.go's xce_backend_init(), placed after xcecli_start()'s success
check (xcecli_start() wipes any earlier-registered handler as part of
its own init -- the same ordering bug found and fixed in worker's/
scheduler's task 2.3/3.1 work).
A shared checkAndPerformRejoin() helper wraps every gCtx.ctx-based
backend call (SubmitJob, ControlJob, ControlQueue, ControlHost,
FetchFile): after the call returns, if a rejoin is pending, issue
xcecli_join() as a separate step and clear the flag only on success --
on failure it stays set so the next backend call's return retries it,
no separate retry loop.
needsRejoin is a proper sync/atomic flag rather than a plain bool,
since apiserver (unlike the single-threaded C daemons) serves
concurrent HTTP requests that can all reach the shared helper at once.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Reference the operator key via ${XCE_API_KEY} instead of a hardcoded value —
apiserver restarts clear in-memory keys, so a committed key goes stale
silently and is a credential risk. Export XCE_API_KEY before starting Claude
Code to use the scheduler-agent MCP server.
Co-Authored-By: Claude <noreply@anthropic.com>
Add four tools closing the submit→watch→analyze→suggest loop in the
request/response MCP model:
- scheduler_wait_job: poll a job to a terminal state or timeout.
- scheduler_analyze_failure: gather job record + output logs for failure
analysis (interpretation left to the LLM).
- scheduler_suggest_resources: conservative processor/node suggestions from
current cluster availability.
- scheduler_estimate_wait_time: rough wait estimate from pending count and
free cores (guidance, not a guarantee).
All verified live on t1 against the running apiserver.
Co-Authored-By: Claude <noreply@anthropic.com>
Add WatchJobs(ctx) to the Client interface and HTTPClient: opens the
apiserver's /xce/v1/events SSE stream and yields JobEvent values until the
context is cancelled or the stream ends. JobEvent carries the event type,
job id, status string, exit code, and a raw job snapshot.
Also fix ListJobs' filter params: the apiserver accepts filters via the
"filter" query param in field:op:value form (e.g. queue_name:eq:normal);
the individual queue_name/submit_user params it ignores with a 400, which
broke every filtered list. Multiple conditions are brace-wrapped {a},{b}.
Verified live on t1: WatchJobs captured job.updated PEND→RUN→DONE for a
submitted job.
Co-Authored-By: Claude <noreply@anthropic.com>
Add a Server-Sent Events endpoint /xce/v1/events streaming job lifecycle
events (job.created/updated/deleted/finished) to subscribers. The
DefaultHandler now publishes every job event to a process-wide EventBroker;
the SSE handler subscribes and streams JSON frames with a keep-alive comment
during idle.
- backend/broker.go: thread-safe fan-out Broker (drops for slow consumers so
the notification path never blocks), Event type with job_status string.
- backend/handler.go: OnJob* publish to DefaultBroker.
- controllers/events.go: SSE handler (gin, text/event-stream, keep-alive).
- router + whitelist + rbac: /xce/v1/events route, event:subscribe
permission granted to mcp_readonly/mcp_operator/cluster_admin.
- Tests: broker fan-out/slow-consumer/unsubscribe, handler publishes, SSE
end-to-end via httptest.
Verified live on t1: SSE stream delivered job.updated PEND→RUN events for a
submitted job.
Co-Authored-By: Claude <noreply@anthropic.com>
Full Agent chain verified on t1: mcp_operator API key → mcp-server
(scheduler_submit/get_job) → apiserver REST → controller → scheduler →
worker, job submit→run→DONE; cancel(kill)→EXIT. Note the apiserver
controller-restart rejoin gap (DataStore stays stale until apiserver
restart).
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces apiserver's two connect mechanisms -- a per-peer Go loop that
hardcoded nmasters=1 per attempt, and a separate direct-construction
path for the legacy single-host config -- with one pipeline:
buildClusterConfig() parses (via task 4.1's C parser) or synthesizes a
single-candidate config, validates it, then ConfigConverter
(xce_cluster_config_to_client_config) produces the client config for
one xce_backend_init() call. A 3-candidate config now actually
exercises the C library's own failover/standby-redirect handling on
initial connect, matching worker's and scheduler's task 2.1/3.1 work,
instead of a Go-level per-candidate retry loop that never built a real
multi-master config.
Along the way: found and fixed a genuine use-after-free. The deleted
tryConnect() freed its C.CString-allocated host address immediately
after its single connect call returned, but xcecli_ctx_init() stores a
raw pointer to the config forever (client.c:38) and every later
failover rotation re-dereferences it (client_failover.c) -- so any
reconnect after the first would read freed memory. Fixed by giving the
cluster/client config structs package-level, process-lifetime storage
(gClusterCfg/gClientCfg), mirroring worker's and scheduler's file-scope
static pattern, with host/from/key written into fixed-size C arrays
instead of freed heap pointers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Replaces apiserver's hand-written Go parsePeer() with a CGO wrapper
around the shared C parser+validator pair (xce_parse_peers then
xce_cluster_config_validate, both src/lib/cluster.c) that worker and
scheduler already use from their own main.c's. Gives apiserver the same
[name=]host:port[,...] syntax (including the name= prefix the old Go
parser could not express) and identical malformed-candidate rejection
(empty host, port <= 0, non-numeric port) -- round-1 review caught the
first pass skipping the validate() half, silently accepting inputs
worker/scheduler both reject.
A malformed entry now rejects the whole candidate list rather than
being skipped with a warning (matches xce_parse_peers()'s own
all-or-nothing construction and worker/scheduler's precedent) --
surfaced loudly via RegisterToBackend's error, not silently.
Parsing only -- RegisterToBackend()'s connect loop and tryConnect()
are unchanged; collapsing them into a single connect call is task
4.2's job.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Proves task 3.2's retry-until-success-or-shutdown wrapper's guarantees
hold across multiple queued decisions drained through the real
xce_sched_driver_on_decision(), not just a single isolated call:
decisions queued behind a stuck one publish afterward in original
order (Req 7 AC3), and shutdown mid-retry costs at most one retry
cycle regardless of queue depth -- decisions behind the stuck one see
core.stop already true and are never attempted (Req 7 AC2). Also adds
a source-scan confirming this stays scoped to the scheduler's own
publish direction (Req 7 AC5) and a fresh-instance test confirming a
restart still loses in-flight queued decisions, unchanged by this work
(Req 7 AC6).
Pure test-coverage task -- zero production changes, matching the
review's independently-reproduced RED->GREEN evidence that the
existing task-3.2 implementation already satisfies these guarantees.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Adds a worst-case-occupancy test proving the scheduler's picker/allocator
pipeline buffers briefly past a stalled publish (decisions cbuf cap 5,
jbuf cap 1) then genuinely blocks until connectivity recovers, with zero
job loss (Req 3 AC2). Adds a source-scan regression guard confirming the
scheduler makes no periodic heartbeat call of its own (Req 3 AC3).
Also fixes a real gap found along the way: sched_init()'s
all-candidates-unreachable and join-failure log lines named only a
candidate count, never which candidates were attempted (same class of
bug worker's task 2.3 fixed). Added sched_format_candidate_list(),
covered by a real dup2()-based stdout capture around a live sched_init()
failure path (round-1 review caught this diagnostic shipping with zero
test coverage; fixed in round 2).
Multi-candidate join and kill-primary/reconnect were validated manually
against a live 3-node cluster rather than committed as an automated
test, following worker's task 2.3 precedent -- no automated live-deploy
E2E harness exists for any C daemon in this repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Wraps sched_publish_dispatch()/sched_publish_pend_reason() in a shared
retry loop that keeps retrying a -E_XCE_NOT_READY-exhausted publish
until it succeeds or the scheduler is shutting down (core.stop checked
before each attempt, not only after), composed with the pending-rejoin
follow-up established in task 3.1. Extended-retry success logs
distinctly from an ordinary first-attempt success (Req 7 AC4).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNB4zGPCsHyDUCDJyMbiQN
Scheduler's unified connect pipeline is implemented, tested (RED→GREEN
verified), and live-validated against a 3-node HA controller cluster
(standby-first candidate list joins correctly via xcecli failover). See
a9f283d.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYSEeLmzjtD4RLDuUDhLoK
Replace scheduler's inline per-candidate loop and separate single-host
branch with one unified pipeline (mirrors worker's task 2.1 fix): build
exactly one validated candidate list, whether from --peers or synthesized
as a single candidate from the legacy default, then connect through the
shared xce_cluster_config_to_client_config() helper exactly once.
Adds sched_on_reconnect(), a rejoin callback that only sets a flag and
returns immediately (no I/O, no xcecli_* calls) to avoid the self-deadlock
worker hit in task 2.3 — registered via xcecli_set_reconnect_handler()
after xcecli_start(), since xcecli_start() resets any handler registered
earlier back to NULL.
Adds test_scheduler_connect.c: behavioral tests confirm no-peers and
single-candidate configs produce identical connect log sequences and that
unreachable candidates fail fast without retry loops; structural tests
source-scan scheduler.c to confirm a single xcecli_ctx_init() call site, a
single xcecli_join() call site, and a rejoin callback body free of I/O and
xcecli_* calls.
Live-validated against a 3-node HA controller cluster: a scheduler
configured with the standby listed first in --peers fails over via
xcecli's built-in retry-next mechanism to the primary and joins
successfully.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FYSEeLmzjtD4RLDuUDhLoK
Confirm connector thread is isolated from worker_process_update() (main
thread) and resource_sync_thread() (own thread) via code read plus a real
pthread concurrency test. Add unit and live 3-node E2E coverage for
candidate/reconnect behavior, all-candidates-unreachable diagnostics, and
liveness cadence under concurrent reconnect.
The E2E run surfaced two production bugs the 2.1/2.2 unit tests missed:
xcecli_start() resets the reconnect callback to NULL after it runs, so
registering it before xcecli_start() silently no-ops the whole rejoin
mechanism; and a successful worker join returns its assigned slot index
as reply.status (any rc >= 0), not always exactly 0, so checking_connection()'s
rc == 0 success check never fired on a real non-zero-slot join. Both fixed,
plus per-candidate names added to the all-unreachable failure log.
Root-check bypass needed for the E2E run is gated at compile time via the
existing VIRTUAL_CLUSTER flag, not a runtime env var, so the real xcecd
binary's root requirement has zero bypass surface.
Thread cluster_cfg by value through xcecd_config_t -> xcecd_context_t ->
cc_ctx so it's populated before controller_connector_start() fires the
first join, instead of after xcecd_start() returns. checking_connection()
now retries a failed post-reconnect xcecli_join() on the next liveness
probe instead of leaving needs_rejoin cleared on mere attempt.
bug-1510 (dispatch-fail re-PEND, 6928f74) + bug-1511 (sched_update_workers
in-place update on re-push, 1a2bd3f) together close bug-1494. Update
left_scheduler_work.md and buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
When a worker is re-pushed (e.g. it rejoined with fresh resvals after the
controller/worker had been down), sched_update_workers built a new worker and
sched_add_worker's key_add returned EEXIST — the fresh worker was dropped and
the scheduler kept the stale state (slots=0 from the dead/absent period).
PEND jobs never dispatched even after the worker returned with full capacity
(bug-1494, controller-restart self-heal).
sched_update_workers now updates an already-known worker in place: swap the
fresh resvals/resInUse/status into the existing worker (the stale arrays are
freed when the discarded object is torn down) and adopt its host_eval_ctx.
Also NULL-guard make_sched_worker (it can return NULL).
Verified on the t1 cluster: scheduler started while the worker was dead
(slots=0), worker rejoined (slots=8), previously-PEND jobs dispatched and
completed, no ASAN.
Co-Authored-By: Claude <noreply@anthropic.com>
Replace worker's separate default/single-host and multi-candidate connect
branches with one path: synthesize-or-accept a candidate list, validate,
convert via xce_cluster_config_to_client_config(), connect once. Register
a flag-only rejoin callback (no I/O, no client-lib reentry) and remove the
liveness probe's bounded give-up so the library owns retry/recovery.
Document that the scheduler dispatch-fail re-PEND fix (commit 6928f74)
resolves one root cause of the "all jobs PEND" symptom; the
controller-restart self-heal-reconnect aspect of bug-1494 remains open.
Log bug-1510 in buglog.
Co-Authored-By: Claude <noreply@anthropic.com>
Task 1.3's full checklist (constant-equality assertion, candidate
order, one-candidate-no-branch, truncation-with-warning, NULL/empty
rejection) was already satisfied by tasks 1.1 and 1.2's test
coverage. Verified fresh against the current test suite; no
implementer/reviewer cycle needed since there is no new diff.
xce_cluster_config_to_client_config() converts a validated candidate
list into a fully-populated xce_client_config_t, copying candidates
in order with no special-casing for a single candidate. The
truncate-with-warning safety net stays in place as a defensive guard
now that N_MAX_MASTERS and MAX_HA_CANDIDATES match.
Also removes a WARN check in xce_cluster_config_validate() that
became unreachable dead code once task 1.1 reconciled the two
candidate-count constants (both checks tested the same field against
the same constant with no mutation between them).
Task 1.2 of component-failover-adoption (Requirement 1 AC1-AC7).
N_MAX_MASTERS (client.h) now derives from config.h's MAX_HA_CANDIDATES
instead of an independently-maintained literal (3 vs 4 previously).
Fixes worker's and scheduler's hardcoded `> 3` startup checks to
reference the constant symbolically, and updates the client-retry
test fixture so it doesn't exercise an uninitialized 4th slot.
Task 1.1 of component-failover-adoption (Requirement 1 AC7).
The controller rejects a dispatch when the chosen worker is not in a
dispatchable state (UNAVAIL after a worker death, or still joining) and
replies with a jobDispatchFail notify. The scheduler had no handler for that
event, so a job it had just committed (state=RUN) was never returned to PEND
— it was never re-elected and stayed PEND cluster-wide even after the worker
recovered.
Handle NotifyData_data_jobDispatchFail: log the failure and call the new
sched_requeue_pend_job(), which frees the dispatch bookkeeping, resets the
job to PEND, and re-enqueues it in the fcfs pendJobs queue so the next
election session re-dispatches it. White-box test TUs that #include
handler.c get a sched_requeue_pend_job stub.
Co-Authored-By: Claude <noreply@anthropic.com>
Merge the scheduler-join NULL guard from the remote js branch: if the
controller's cluster config is not loaded yet (cold boot / config not
pushed yet), xcesched_push_configs would dereference a NULL cluster_config
and crash the controller (ASan SEGV on scheduler join). Skip the push; the
scheduler receives config on a later join once config is available.
Co-Authored-By: Claude <noreply@anthropic.com>
- Add OPCode_probeConfigTag dispatch in cluster.c
- Implement probe_config_tag handler returning config tag via
standard xce_reply() → RequestReply envelope
- Populate redirectHost/redirectPort for follow_redirects support
- Wire probeConfigTag reply through typemap
- Add probeConfigTagReply struct to controller.capnp schema
- probe_config() now wraps probe in Request envelope matching push_config()
- Controller OPCode_probeConfigTag handler replies through xce_reply()
with redirectHost/redirectPort populated
- follow_redirects works identically for both push and probe flows
- Change default record_dir from /var/log/xce/ to /tmp (always exists)
- Add mkdir(record_dir, 0755) at startup for custom --record-dir paths
- Split validation_error_text into 6 distinct per-errno messages
- Accept human-readable job IDs in 'info job' via parse_jobid()
- Add 'info jobs' command to list all jobs in sim REPL
- Format job IDs with fmt_jobid() for xce_test-consistent output
- Replace printf with fprintf(stdout, ...) in all REPL files
to bypass -Wl,--wrap=printf linker interception
The "Deploying the Scheduler" section predated scheduler-core and only
covered normal-mode startup. Adds session recording (--record-dir,
xceadmin scheduler record start|stop|status, filename validation),
xceadmin scheduler dump for offline .replay inspection, and simulation
mode (--sim, --sim-speed, --sim-verify, --sim-alt-events, --sim-output,
--interactive REPL) — all verified against the actual argp help output
and xceadmin cli.py source, not assumed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The file was untouched since before scheduler-core and still described
a plain plugin-selects-at-runtime engine with a generic meson build
command. Documents the actual core/driver split, the picker plugin
interface and its builtin_fcfs/fcfs.c split, the simulation driver and
replay/recording machinery, and the interactive REPL, plus the correct
/xce-build invocation and the project's 800-line-file-cap convention
this module's several *_events.c/*_session.c-style splits follow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
These edits (three-thread picker correction in brief.md, record_dir
path-validation and kvmgr-audit requirements in requirements.md,
OPCode_schedAdmin folded into the peer-authentication-hardening
backlog item in roadmap.md) were made during earlier requirements
review and design work but left uncommitted. Reviewed for accuracy
against the completed implementation — all still correct, persisting
as-is.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found by /kiro-validate-impl: tasks 2.1-2.6 (builtin_fcfs picker) were
implemented, reviewed, and committed on 2026-08-03 but never got a
_Verified: line, unlike every other task group. Backfills each from
git history (commits f7921b96, b7214ebe, fcb32a8b, 7ee9a16f, 9d91d470,
604a5649) and direct source inspection.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both files existed only on disk since the design phase (2026-08-03) and
were never committed despite ~40 implementation commits landing since.
Also corrects design.md drift found by /kiro-validate-impl: the File
Structure Plan and BuiltinFcfsPicker sections claimed plugins/fcfs.c
would be rewritten in place, but the actual implementation added a new
plugins/builtin_fcfs.c and deliberately kept fcfs.c untouched as the
job_elect_fn compat-wrapper's exercise target. Updates the directory
tree to the real file layout (core_events/session/clock.c, sim.c/
sim_loop.c/sim_record.c/sim_verify.c, snapshot/recording/repl splits,
test_e2e_capnp.c vs the planned test_replay_format.c, test_sched_admin_dispatch.c
vs test_sched_admin.c), and adds two Open Questions/Risks entries for
the AC 3 tlog-logging-integration gap and this fcfs.c/builtin_fcfs.c
documentation gap, both found during validation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
/kiro-validate-impl pass: GO. Two non-blocking follow-ups found and
documented — AC 3's tlog virtual-clock logging integration is built
but never activated (root-caused to sim.c's sched_sim_driver_start,
deferred pending a matching deactivation/fini design), and
plugins/fcfs.c vs builtin_fcfs.c is functionally correct but
undocumented against design.md and missing _Verified notes on 2.1-2.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Clean rebuild (asan, all components) then 15-round sequential and
15-round chaotic HA soak against the fully integrated feature. Both
PASS with zero job loss, zero committed-prefix divergence, zero
liveness stall. Closes scheduler-core: all 45 tasks complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
controller_unit_test crashed deterministically in write_WalCatchupEntry
(test_wal_catchup_append_term_fields_round_trip) -- the same bug class
as bug-1504 (SnapshotData.internalState): this codebase's capnpc-c
backend silently no-ops Data-field encoding in generated encode_XXX()
composite helpers without zero-initializing the caller's local struct
first, leaving an uninitialized capn_ptr written onto the wire.
Confirmed zero production exposure before fixing: xce_wal_batch_encode_
raw()/wal_entry_to_wire() (request.c) never call the generic
encode_WalCatchupEntry() at all -- they memset() their own struct and
call write_WalCatchupEntry() directly with every field hand-set, so
this was 100% test-only exposure (grepped every caller of the generic
encode/decode pair; only the crashing test uses it).
Fix mirrors bug-1504: WalCatchupEntry.payload (@2) and batchPayloads
(@10) switched from Data to List(UInt8) in controller.capnp (new
batchPayloadsLen @11 companion; payload reuses the existing
wirePayloadLen @9), typemap.h's wal_catchup_entry_t field types
updated, and request.c's extract_data_bytes() widened to take a raw
capn_ptr (it already only used .p internally) so its call sites work
regardless of which Data-shaped wrapper the generated accessor
returns now. batchPayloads' single-top-level-blob shape confirmed
this does not reintroduce bug-847's per-element cross-segment
far-pointer loss -- that issue is inherent to per-element data
regardless of Data vs List(UInt8) spelling.
Strengthened test_wal_catchup_append_term_fields_round_trip with real
payload/batchPayloads content assertions -- previously never
exercised, since the test's src fields were left NULL.
Verified: controller suite 3/3 (was 2/3, crashing), scheduler suite
unaffected at 37/37 (unrelated schema area).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the final task of "Core: Interactive REPL" (Req 5 AC 22): proves
the REPL's own documented invariant -- "step/continue to completion
produces byte-identical output to batch --sim" (design.md SimDriver
Invariants).
test_repl_equivalence.c runs the same sched_sim_driver_run() on the
same 3-session/9-decision fixture twice: once batch (interactive=
false), once REPL-driven via freopen()'d scripted "step 1\ncontinue\n"
stdin into the real repl_command_loop()/readline() path (same
mechanism test_repl_execution_control.c established) -- then
structurally decodes both runs' --sim-output and compares decision
content. Test-only; zero production code changed.
RED-phase probe (planted in xce_repl_on_event(), reverted, zero
residue) confirmed the comparison is real: corrupting one job's id
under interactive mode broke exactly this test plus the 3 other REPL
tests that share the same hook, none else.
Scheduler suite: 37/37 clean (was 36/36). Task group 12 (Interactive
REPL, tasks 12.1-12.4) is now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds `snapshot [FILE]` and `set speed|output|verify ...`/`info config`
(Req 5 AC 22 "Snapshots"/"Simulation parameters") to the interactive
simulation REPL.
`snapshot` reuses sim_record.c's existing sim_record_write_header_and_
snapshot() (task 11.5) rather than reimplementing capnp encoding, so
the produced file is genuinely loadable via a fresh --sim invocation
-- the task's own Observable. `set speed`/`set output`/`set verify`
mutate fields sim_loop.c and core_clock.c already read fresh on every
event/session, so no new hook points were needed and sim_loop.c stays
untouched at exactly 800/800 lines (this project's hard per-file cap).
New repl_params.c is a fourth sibling in the repl.c/repl_break.c/
repl_info.c split (task 12.2's precedent) -- neither existing file had
headroom left under the 800-line cap for this task's own logic.
Scheduler suite: 36/36 clean (was 35/35). Review independently
reproduced the RED-phase probe and additionally verified the
snapshot round-trip Observable is non-vacuous.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
Adds the interactive simulation REPL's breakpoint and state-inspection
commands (Req 5 AC 22): `break <event_type> [key=value ...]` / `info
break` / `delete <id>|all` on event and decision types, and `info
session/job/queue/worker/resource/summary`.
repl.c (370 lines after task 12.1) couldn't hold this task's full
surface under the project's 800-line cap, so it's split three ways
(matching the snapshot.c/snapshot_wire.c/snapshot_load.c precedent):
repl.c (execution control + dispatch), repl_break.c (matching/CRUD +
the three new driver-facing hooks), repl_info.c (info dumps), sharing
state via repl_internal.h. sim_loop.c gained 3 new bare hook calls
(repl.h inline wrappers, same pattern as task 12.1's two) at the
points design.md's pseudocode requires -- per decoded event before it
joins the batch, per session boundary, and per drained decision --
landing sim_loop.c at exactly 800/800 lines.
Two real bugs found and fixed during development (bug-1506, bug-1507):
an uninitialized xstr_table left breakpoint filter fields silently
NULL instead of failing loud, and repl_breakpoint_free() called a bare
free() on an RTTI-refcounted makeString() string, corrupting the heap.
Review found one gap: `info job` printed only a resreq presence flag
instead of the actual resreq text, when format_resreq() already
existed in this repo for exactly this purpose. Fixed to match.
Scheduler suite: 35/35 clean (was 34/34). RED-phase evidence (2 real
bugs plus 2 synthetic probes, all reverted with confirmed zero
residue) recorded in tasks.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPPXeKSUXUqcqvzNXY1rzJ
