diff --git a/.kiro/specs/build-time-versioning/spec.json b/.kiro/specs/build-time-versioning/spec.json index 51069ef0..22c30f87 100644 --- a/.kiro/specs/build-time-versioning/spec.json +++ b/.kiro/specs/build-time-versioning/spec.json @@ -1,9 +1,9 @@ { "feature_name": "build-time-versioning", "created_at": "2026-08-20T07:02:00Z", - "updated_at": "2026-08-20T09:20:00Z", + "updated_at": "2026-08-24T00:00:00Z", "language": "en", - "phase": "tasks-generated", + "phase": "implementation-complete", "approvals": { "requirements": { "generated": true, diff --git a/.kiro/specs/component-failover-adoption/tasks.md b/.kiro/specs/component-failover-adoption/tasks.md index 5d456f98..8e39b187 100644 --- a/.kiro/specs/component-failover-adoption/tasks.md +++ b/.kiro/specs/component-failover-adoption/tasks.md @@ -1,6 +1,6 @@ # Implementation Plan -- [ ] 1. Foundation: candidate-count constant reconciliation and shared conversion helper +- [x] 1. Foundation: candidate-count constant reconciliation and shared conversion helper - [x] 1.1 Reconcile the candidate-count limit constant across libxce - Derive `client.h`'s `N_MAX_MASTERS` from `config.h`'s `MAX_HA_CANDIDATES` instead of an independent literal, so the two constants can never silently diverge again - Fix the two hardcoded `> 3` candidate-count checks (worker's and scheduler's startup argument validation) to reference the constant symbolically instead of a magic number, so they track the raised value automatically @@ -23,7 +23,7 @@ - _Requirements: 1.5, 1.7_ - _Depends: 1.2_ -- [ ] 2. Worker: unified candidate-list connect and transparent reconnect +- [x] 2. Worker: unified candidate-list connect and transparent reconnect - [x] 2.1 (P) Replace worker's dual connect mechanisms with the single unified pipeline - Delete the standalone client-config construction used for worker's default connection and the separate single-host branch used when no candidate list is configured — both bypass the shared candidate-list validation and conversion helper entirely today - Build exactly one validated candidate list before connecting, whether it came from an operator-supplied peer list or was synthesized as a single candidate from the existing default when none was given, then run it through the shared helper and connect once @@ -50,7 +50,7 @@ - _Requirements: 1.6, 3.1, 3.3_ - _Depends: 2.1, 2.2_ -- [ ] 3. Scheduler: unified candidate-list connect, decision-publish redelivery, and runtime reconnect +- [x] 3. Scheduler: unified candidate-list connect, decision-publish redelivery, and runtime reconnect - [x] 3.1 (P) Replace scheduler's dual connect mechanisms with the single unified pipeline - Delete the scheduler's inline per-candidate loop and its separate single-host branch used when no candidate list is configured — both bypass the shared candidate-list validation and conversion helper entirely today - Build exactly one validated candidate list before connecting, whether from an operator-supplied peer list or synthesized as a single candidate from the existing default when none was given, then run it through the shared helper and connect once @@ -85,7 +85,7 @@ - _Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 2.5_ - _Depends: 3.2_ -- [ ] 4. apiserver: unified candidate-list connect via CGO, config parity, rejoin, and fail-fast +- [x] 4. apiserver: unified candidate-list connect via CGO, config parity, rejoin, and fail-fast - [x] 4.1 (P) Route apiserver's candidate-list parsing through the real C parser - Replace the hand-written Go candidate-list parser with the shared C parser used by worker and scheduler, retiring the Go parser and its call sites entirely - Map validation errors to clear Go error messages naming the malformed entry @@ -131,7 +131,7 @@ - _Requirements: 1.5, 4.1, 4.2, 6.1, 6.2_ - _Depends: 4.5_ -- [ ] 5. Integration and cross-component validation +- [x] 5. Integration and cross-component validation - [x] 5.1 E2E: full-cluster reconnect after primary failover - Extend the project's existing HA soak pattern to a scenario with worker, scheduler, and apiserver all configured with the full candidate list; kill the primary mid-session with a write request in flight - Assert: worker reconnects and resumes job-state reporting; scheduler reconnects and resumes dispatch; apiserver reconnects, resumes serving API requests, and resumes receiving push notifications — none of the three restarted diff --git a/.kiro/specs/dirty-content-fingerprint/spec.json b/.kiro/specs/dirty-content-fingerprint/spec.json index eb629504..50662e88 100644 --- a/.kiro/specs/dirty-content-fingerprint/spec.json +++ b/.kiro/specs/dirty-content-fingerprint/spec.json @@ -1,9 +1,9 @@ { "feature_name": "dirty-content-fingerprint", "created_at": "2026-08-21T02:44:43Z", - "updated_at": "2026-08-21T03:40:00Z", + "updated_at": "2026-08-24T00:00:00Z", "language": "en", - "phase": "tasks-generated", + "phase": "implementation-complete", "approvals": { "requirements": { "generated": true, diff --git a/.kiro/specs/resource-value-verification/tasks.md b/.kiro/specs/resource-value-verification/tasks.md index ee06278f..b7c46a9e 100644 --- a/.kiro/specs/resource-value-verification/tasks.md +++ b/.kiro/specs/resource-value-verification/tasks.md @@ -1,6 +1,6 @@ # Implementation Plan -- [ ] 1. Foundation: test scaffolding and skill directory setup +- [x] 1. Foundation: test scaffolding and skill directory setup - [x] 1.1 Extract resupdate's type-resolution logic and write failing unit tests for its fallback behavior - Move `set_resval()`'s resource-name → `Value_u_which` table lookup into a new, small, dependency-light `resval_typeresolve.c`/`.h` module with no networking/REPL/global-state dependency — `test.c` (4298 lines, its own `main()`, heavy `libedit`/network dependencies) cannot itself be white-box included into a unit test, so this extraction is a hard prerequisite for testing this logic in isolation at all - Create `test_resval_typeresolve.c` (Unity) and wire both new files into `src/controller/meson.build`: add `testsuite/resval_typeresolve.c` to `xce_test_src` (line 57) so `xce_test` still links, and add a new `test()` target for `test_resval_typeresolve.c` following the existing `controller_unit_test`/`sched_push_unit_test` pattern (lines 86, 215, 222) — these are hand-maintained explicit lists, not globs, so the module and test will not build without this step @@ -14,7 +14,7 @@ - _Requirements: 6.1_ - _Boundary: xce-test-resources skill_ -- [ ] 2. Core: resupdate resource-name coverage fix +- [x] 2. Core: resupdate resource-name coverage fix - [x] 2.1 Wire the extracted resolver into resupdate and add the type-override flag - Extend `resval_typeresolve.c`'s fallback path so any resource name not in the existing table defaults to numeric (`Value_u_fvalue`), while `type`/`model` and all currently-supported names keep their exact existing behavior - Make `test.c`'s `set_resval()` a thin wrapper delegating to `resval_typeresolve.c` (for type resolution) plus its own existing `.host` tagging (unchanged, still needs REPL state) @@ -31,7 +31,7 @@ - _Requirements: 1.2, 1.3_ - _Depends: 2.1_ -- [ ] 3. Core: apiserver local deployment capability +- [x] 3. Core: apiserver local deployment capability - [x] 3.1 (P) Extend `xce-deploy-ha`'s `deploy-ha.sh` with a conditional apiserver step and a combined `start-all.sh` - In `deploy-ha.sh` Step 3, add a new conditional branch in the same *opt-in-by-presence* spirit as the existing worker/scheduler branch: check for a pre-built apiserver binary at its own Go-build output path (e.g. `/tmp/apiserver`, `make build-server`'s default) — NOT alongside `$BINARY`, since apiserver is a separate build system, not the same meson build dir worker/scheduler share; `deploy-ha.sh` does not invoke `make build-server` itself, building it remains a caller prerequisite (document in SKILL.md's Prerequisites section, same pattern as the existing `/xce-build controller` note). If present, generate `$TOP/scripts/start-apiserver.sh` plus a minimal `logger.yaml` (modeled on `make test-integration`'s `XCE_BIN`/`SRV_URL` convention) — functional auth/RBAC is NOT required (verification is trace-log-based, task 6.4; no REST query is ever issued against this instance), but `logger.yaml` MUST set `logging.level: debug` or task 4.2's trace line is silently never written; absent the binary, generate nothing, so existing callers are unaffected - Also in Step 3, always generate `$TOP/scripts/start-all.sh`: runs `start.sh` for the controller node(s), waits on Step 5's existing health-check, then runs whichever of `start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually generated for this deploy (skipping any that weren't) — start the scheduler with `-vvv` specifically (verbosity 3, matching `tlog_debug3`/`TRACE`), overriding SKILL.md's existing `-v`-only example for this feature's needs, or task 4.1's trace line is silently never written @@ -41,7 +41,7 @@ - _Requirements: 5.3, 6.1_ - _Boundary: xce-deploy-ha (deploy-ha.sh, SKILL.md)_ -- [ ] 4. Core: scheduler and apiserver trace-log instrumentation +- [x] 4. Core: scheduler and apiserver trace-log instrumentation - [x] 4.1 (P) Add a trace log line to the scheduler's live resource-update handling - In `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case (`src/scheduler/src/core_events.c`), add one `TRACE`-level log line (this component's existing `tinylog` convention) immediately after the resolved worker's value is written, logging the resolved host, resource name, and applied value in a fixed, greppable format - Confirm the existing scheduler testsuite (`test_core_events.c` and friends) still passes unchanged — this is a pure log addition with no control-flow change, so no new unit test is required (consistent with this codebase's existing precedent: other `TRACE`/`WARN` calls in this area, e.g. `apiserver_worker_change`, have no dedicated unit test coverage either); the log line's actual content/format is confirmed live in task 8.1 @@ -54,7 +54,7 @@ - Observable completion: a live apiserver's log file contains one line per stored resource-location update, in the format task 6.4 will grep for; existing backend tests are unaffected - _Requirements: 5.1, 5.2, 5.4_ -- [ ] 5. Integration: full-stack deployment orchestration +- [x] 5. Integration: full-stack deployment orchestration - [x] 5.1 Wire controller, scheduler, and apiserver deployment into the skill via `start-all.sh` - Invoke `xce-deploy-ha`'s `deploy-ha.sh` (task 3.1) into a fresh scratch directory, then its generated `start-all.sh` — no manual per-component sequencing lives in this skill; `start-all.sh` already gates scheduler/apiserver startup on the controller health-check and handles the `-vvv`/debug-level logging needed for tasks 4.1/4.2's trace lines - Register a `trap`-based cleanup (or equivalent) before invoking `start-all.sh`, not just at the end of the script — this must cover a mid-deployment abort (e.g. apiserver fails to start after controller+scheduler are already up), not only a completed PASS/FAIL run; deliberately stronger than `xce-deploy-ha`'s own convention (a separate, manually-invoked `cleanup.sh`) because Requirement 6.5 requires cleanup-by-default on every path unless the operator passed the "keep running" flag @@ -62,7 +62,7 @@ - _Requirements: 5.3, 6.1, 6.5_ - _Depends: 1.2, 3.1, 4.1, 4.2_ -- [ ] 6. Core: resource injection and per-layer verification checks +- [x] 6. Core: resource injection and per-layer verification checks - [x] 6.1 Implement worker join and resource injection - Join a synthetic worker through the `xce_test` worker context (no real worker daemon process), then run `resupdate` for one or more resource names/values - Observable completion: after this step runs, `resupdate` has been accepted for every requested resource name with zero real worker daemon processes running @@ -92,7 +92,7 @@ - _Depends: 6.1, 6.3_ - _Boundary: xce-test-resources skill (apiserver check) — same file as 6.2/6.3, sequential_ -- [ ] 7. Integration: verdict aggregation, reporting, and cleanup +- [x] 7. Integration: verdict aggregation, reporting, and cleanup - [x] 7.1 Aggregate layer results, emit the verdict, and clean up - Combine the controller/scheduler/apiserver check results into a single per-resource, per-layer comparison, implemented as a self-contained comparison step that accepts a set of (resource, injected value, per-layer observed value) records — so it can be invoked directly with synthetic records for testing (task 8.2), not only reachable from a live run - Emit a machine-parseable VERDICT line (following `xce-test-ha`'s convention) plus the standard terminal PASS/FAIL oracle line, naming every layer that mismatched or was unreachable (timed out) @@ -102,7 +102,7 @@ - _Requirements: 6.2, 6.3, 6.4, 6.5_ - _Depends: 6.2, 6.3, 6.4_ -- [ ] 8. Validation: end-to-end live verification runs +- [x] 8. Validation: end-to-end live verification runs - [x] 8.1 Run the full flow live, including one deliberately-unregistered resource name - Execute the skill end-to-end against a freshly deployed controller/scheduler/apiserver, using the default resource set (task 2.1's previously-missing names) plus one resource name that does not exist in the cluster's registered set - Observable completion: every genuinely-registered resource individually reports matching values at all three layers (confirming tasks 4.1/4.2's trace-log lines are correctly formatted and greppable, not just present), AND the unregistered name is reported as a clear, specifically-identified rejection (not a silent omission) with correct per-name accept/reject accounting across the whole batch; per design.md's Error Handling (an unrecognized name is a run-level failure), the run's single overall VERDICT is FAIL, attributable specifically to the unregistered name — not a false overall PASS diff --git a/.kiro/specs/scheduler-core/tasks.md b/.kiro/specs/scheduler-core/tasks.md index cfe93b45..0e265118 100644 --- a/.kiro/specs/scheduler-core/tasks.md +++ b/.kiro/specs/scheduler-core/tasks.md @@ -1,6 +1,6 @@ # Implementation Plan -- [ ] 1. Foundation: Picker Interface & Plugin Registration +- [x] 1. Foundation: Picker Interface & Plugin Registration - [x] 1.1 Define the job picker plugin contract - Add the lifecycle hooks (init, fini), state-change hooks (on_queue_update, on_job_push, on_job_state_change, on_job_update), session hooks (begin_session, next_job, end_session), post-allocation hooks (on_dispatch, on_pend), and optional hooks (on_config, on_job_rusage MAY be NULL) to the picker plugin header @@ -23,7 +23,7 @@ - _Requirements: 2.1, 2.2, 2.3, 2.4_ - _Boundary: SchedulerCore config fields_ -- [ ] 2. Core: Builtin FCFS Picker +- [x] 2. Core: Builtin FCFS Picker - [x] 2.1 Implement builtin_fcfs lifecycle and private state - Implement `init`/`fini`: allocate/free the picker's private ordered-queue state (queue ordering by priority desc, and one job-ordering structure per queue) @@ -87,7 +87,7 @@ - _Depends: 2.6_ - _Verified: 15-round sequential soak, 2026-08-03 — PASS: 15 rounds, 48 jobs submitted, zero job loss; all 3 nodes led (ctrl-1,ctrl-2,ctrl-3)_ -- [ ] 4. Foundation: Scheduler Core Extraction +- [x] 4. Foundation: Scheduler Core Extraction - [x] 4.1 Extract the scheduling core from the monolithic scheduler struct - Split scheduling state (jobs, queues, workers, resources, resvals, users, projects, plugin instances) into its own core structure holding zero controller I/O state (no API client context, no cluster config) @@ -137,7 +137,7 @@ - _Depends: 4.4_ - _Verified: 2026-08-03 — new test_core_full_cycle.c drives push_events→go→drain→end_session against a real picker pthread (running the real xce_job_picker_iface_t hooks on a test-local stub picker) plus a main-thread stand-in allocator (sched_allocator() is xce_scheduler_t*-coupled, out of pure-core scope) calling real on_dispatch/on_pend/make_dispatch_decision/make_pend_decision. Both preconditions (-EINVAL reentrant core_go, -EBUSY mid-session push_events) proven against a genuinely active session, not a hand-set flag, and proven to un-latch after the session ends. Two live RED→GREEN reproductions (neutered each guard in turn), both independently reproduced by review. 19/19 scheduler suite passing, clang-format clean, 30/30 stress runs with zero flakiness, review APPROVED first pass. Zero production files touched — test-only, as scoped._ -- [ ] 5. Core: Normal-Mode Driver +- [x] 5. Core: Normal-Mode Driver - [x] 5.1 Build the normal-mode driver's event translation path - Translate controller capnp push/notify messages into core events, buffering them in a pending-event queue between scheduling rounds @@ -162,7 +162,7 @@ - _Depends: 5.2, 4.6_ - _Verified: 2026-08-04 — sequential HA soak (xce-test-ha skill), 15 rounds, 48 jobs submitted, zero job loss, all 3 nodes led as primary (ctrl-1/ctrl-2/ctrl-3), committed-prefix walcompare consistent every round. Confirms tasks 4.1-5.2's src/scheduler/ rewrite (core extraction + normal-mode driver wiring into main.c) introduced no controller-side regression — consistent with every task's independent review confirming zero src/controller/ changes throughout Group 4/5. Gates Phase 2 -> Phase 3._ -- [ ] 7. Foundation: Replay Wire Format +- [x] 7. Foundation: Replay Wire Format - [x] 7.1 Define the replay stream envelope schema - Add the replay header, snapshot entry, replay event, replay session, and replay decision wrapper message types to the shared wire schema, following the single-stream-of-consecutive-messages framing @@ -203,7 +203,7 @@ - _Depends: 7.2, 7.3, 7.4_ - _Verified: 2026-08-04 — audit of all ~19 new types/enums from tasks 7.1-7.4 found one genuine gap: ReplayEvent's own envelope union (spanning 11 members across 7.1+7.2) was never actually constructed/encoded in any test, despite its individual payload types each having their own round-trip test. Fixed with 3 new tests (jobStateChange/jobUpdate/endSession variants). All other 18 types independently re-verified genuinely field-for-field, non-shallow. Decided against creating design.md's originally-named test_replay_format.c: tasks 7.1-7.4 independently, consistently (4-for-4) chose the existing test_e2e_capnp.c instead, matching task 4.3's core_events.c-vs-core.c precedent for design.md file-naming being non-binding guidance. Deliberately did not add Request.body/RequestReply.data union-wiring tests for OPCode_schedAdmin, matching universal codebase precedent (opcode-dispatch union members are always tested via functional dispatch tests, never bare schema round-trips, and the real dispatch logic doesn't exist until task 10.1). Internal-state version-mismatch negative test re-verified real and correctly scoped (schema-tag level only, matching the Observable). Zero schema/typemap changes -- audit-only, as scoped. 1087/1087 controller_unit_test, review APPROVED first pass with every claim independently re-derived. Closes out Group 7 (Foundation: Replay Wire Format) entirely -- tasks 7.1-7.5 all complete._ -- [ ] 8. Core: Snapshot Serialization +- [x] 8. Core: Snapshot Serialization - [x] 8.1 Implement snapshot save and load - Serialize the full core state (as scoped in the snapshot schema) into the snapshot message; restore from a snapshot message back to that exact state @@ -220,7 +220,7 @@ - _Depends: 8.1_ - _Verified: 2026-08-04 — task 8.1's own -EBUSY guard test only proved rejection against a synthetic single-thread session_active flag flip, and its comment misattributed the technique to test_core_full_cycle.c's actually-genuine-session precedent. Real gap, not busywork: xce_core_go() publishes session_active via memory_order_release while the snapshot guards read it via plain atomic_load() — the synthetic same-thread test never exercised any real cross-thread visibility of that flag. Added a genuine driver-driven session test (real picker thread + real xce_core_go()) proving both save and load reject with -EBUSY during the same active session, leaving state genuinely unchanged (sentinel bytes for save; distinguishable non-existent queue + unchanged candidate count for load) -- zero production changes, task 8.1's guards were already correct. 23/23 scheduler suite, 10/10 standalone flakiness-clean, review APPROVED first pass with an even sharper independent justification than the implementer's own. Closes out Group 8 (Core: Snapshot Serialization) entirely -- tasks 8.1-8.2 both complete._ -- [ ] 9. Core: Recording Control +- [x] 9. Core: Recording Control - [x] 9.1 Implement recording directory provisioning and path validation - Add a scheduler startup CLI flag (parsed in the same argument-parsing pass as the rest of normal-mode's flags) for the recording directory — never settable from any request-path field — defaulting to a fixed system path @@ -262,7 +262,7 @@ - _Depends: 9.4_ - _Verified: 2026-08-05 — audited task 9.4's existing `test_admin_request.c` (already 14 tests driving everything through the real `scheduler_req_handler()`) against task 9.1's exhaustive path-validation suite and task 9.3's write-failure technique to find genuine gaps rather than duplicating coverage: only the bare path-separator rejection had been proven through the admin/handler path, and the disk-full scenario had never been proven with both the initiating `recordStart` and the verifying `recordStatus` driven through real admin requests. Added 3 tests (903→1163 lines): a `..`-traversal rejection (distinct `strstr("..")` branch) and a real-`symlink(2)`-escape rejection, both submitted via genuine `SchedAdminRequest`s through `scheduler_req_handler()` and asserted via the same reply-spy the rest of the file uses; and a write-failure test using task 9.3's exact `fflush()`+`setvbuf(_IONBF)`+`close(fd)` technique to force a genuine EIO, confirming `sched_record_abort_on_error()`'s cleanup is visible through a subsequent real `recordStatus` admin request and that a further `recordStart` afterward succeeds cleanly — the specific thing beyond task 9.3's internal-level proof. Hard-link-aliasing coverage through the admin path deliberately omitted as reasonable scope: task 9.1's 3-round adversarial hunt and task 9.2's direct-call TOCTOU test already exhaustively prove that specific rejection branch, and the routing/plumbing this task exists to prove is structurally identical regardless of which validation branch fires — already retired by the dotdot/symlink cases. Review independently verified every claim against source (confirmed distinct code branches, confirmed a real `rc=-5` EIO in test output, confirmed the write-failure test's single-threaded sequence never touches `record_lock` directly and cannot deadlock/race since all mutation flows through already-locked library calls) and confirmed the `record_start_ts`-unchanged-on-abort behavior matches `sched_record_stop()`'s pre-existing identical semantics and requirements.md's own reply-field table ("original start (historical)") — not a bug. One minor non-blocking note: the symlink-escape test verifies "untouched" via `stat()`+`size==0` rather than this suite's stronger write-content-then-readback pattern; logically sufficient, flagged for awareness only. 17/17 in `sched_admin_request_test`, 26/26 scheduler suite, clang-format clean — independently re-verified by me via a fresh `meson compile`/`meson test --suite scheduler` run before marking complete. Closes out Group 9 (Core: Recording Control) entirely — tasks 9.1-9.5 all complete._ -- [ ] 10. Integration: Controller Admin Opcode and xceadmin CLI +- [x] 10. Integration: Controller Admin Opcode and xceadmin CLI - [x] 10.1 Add the scheduler admin opcode dispatch to the controller - Add the new opcode's dispatch case to the controller's request handler: reject with a clear error when not primary, reject with a clear error when no scheduler is connected, forward the request to the scheduler and wait up to the timeout for a reply, and always send an explicit reply on every path (never a dropped request) @@ -309,7 +309,7 @@ - _Depends: 10.5_ - _Verified: 2026-08-06 — new `test_cli_scheduler_mock.py` (377 lines) closes the exact gap task 10.5 left open: real `click.testing.CliRunner` invocations of the actual `cli` entrypoint against real local TLS mock servers, zero monkeypatching of `scheduler.py`/`cli.py` bindings — distinct from `test_scheduler.py`'s function-level mock-server tests (10.3/10.4) and `test_cli.py`'s monkeypatched CLI tests (10.5). Covers all three record actions plus a genuine two-server standby→primary redirect handoff (`test_cli_scheduler_record_start_follows_redirect_to_primary`), a real `SchedAdminError` raise-site error path, and `dump` against a real on-disk `.replay` fixture built via pycapnp's write-side API. No production code changed (`cli.py`/`scheduler.py` untouched, confirmed via `git diff`). 227/227 passed, 3 skipped (baseline 220/3, +7 new, 0 failed, 0 new skips) via `meson compile -C xce.build xceadmin`. Review APPROVED first pass, zero findings. Closes out Group 10 (Integration: Controller Admin Opcode and xceadmin CLI) entirely — tasks 10.1-10.6 all complete._ -- [ ] 11. Core: Simulation Driver +- [x] 11. Core: Simulation Driver - [x] 11.1 Implement simulation startup - Add the simulation-mode CLI flags (replay file path, start timestamp, speed, verify, alt-events file, output file, interactive) that the process entry point's mode-selection branch (built in 5.2) routes into the simulation driver @@ -358,7 +358,7 @@ - _Depends: 11.4, 9.2_ - _Verified: 2026-08-06 — new `test_sim_equivalence.c` (740 lines, no production code changes — this task chains two already-shipped pipelines) records a synthetic session (2 queues at different priorities, 4 jobs) through a real normal-mode `xce_sched_driver_run_normal_mode_round()` with recording active (task 9.2's `sched_record_start()`) to produce `.replay` file R, then replays R through `sched_sim_driver_run()` with `--sim-output` (tasks 11.1-11.5) to produce `.replay` file S, decodes only each file's `ReplaySession.decisions` via `decode_ReplaySession_ptr()` (deliberately never decoding `ReplayHeader`/`SnapshotEntry`/events, so the test structurally cannot assert on the timestamp/wall-clock metadata AC 7.1 explicitly excludes), and asserts R and S's decisions match in count, type, and full structural content (reusing `sim_verify.c`'s `xce_job_alloc_t`/`xce_job_preason_t` field-by-field comparators, not a jobid-only shortcut). RED phase independently reproduced by review (corrupted a decoded jobid post-decode, confirmed FAIL, reverted, confirmed PASS). Honestly scoped to PEND-only decisions: independently confirmed (by review, reading `sim_loop.c`'s `sim_picker_thread_fn()` directly, committed under task 11.2) that the simulation picker thread has no code path capable of emitting a live DISPATCH decision at all — a genuine, pre-existing structural gap predating this task (real resource-matching allocation belongs to the not-yet-designed `scheduler-allocation` spec), not a shortcut taken here; documented prominently rather than silently avoided. 33/33 scheduler suite (was 32/32), whole-project build clean. Review APPROVED, zero blocking findings. -- [ ] 12. Core: Interactive REPL +- [x] 12. Core: Interactive REPL - [x] 12.1 Implement REPL execution control - Implement a single input-loop REPL (not a separate thread) that pauses before the first event and blocks the simulation loop while active, supporting step-N-sessions, continue-to-next-breakpoint-or-EOF, and quit