lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(scheduler-sim-repl-verbosity): add approved spec (requirements/design/tasks)

Browse source 
Full spec scaffold (requirements/design/research/tasks all approved,
ready_for_implementation=true) plus a roadmap backlog item recorded from
the dirty-content-fingerprint spec's task 4.3 findings (Makefile/meson
fingerprint-computation duplication risk).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-22 02:45:52 +00:00
parent 1c906e955c
commit 979cd24f6c
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
6 changed files with 519 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

291
.kiro/specs/scheduler-sim-repl-verbosity/design.md Normal file
View file

@ -0,0 +1,291 @@
# Design Document
## Overview
**Purpose**: This feature reduces and reorganizes the console output the `xce_scheduler --interactive` simulation REPL prints while executing `continue`, so an operator can watch a long replay's progress without wading through repetitive, low-information lines.
**Users**: XCE scheduler engineers and operators debugging scheduling behavior via `xce_scheduler -i -s <replay>`, a capability shipped by the `scheduler-core` spec.
**Impact**: Changes only the text `xce_repl_on_event()` (repl_break.c) and `sim_print_decision()` (sim_loop.c) print while `sim->repl_state->mode == XCE_REPL_MODE_CONTINUE`. No change to scheduling decisions, replay determinism, `step`-mode output, or any other REPL command.
### Goals
- Suppress routine per-event echo during `continue` execution, while still surfacing structural changes (Requirement 1).
- Replace unconditional per-decision echo with one summary line per round during `continue` (Requirement 2).
- Collapse consecutive idle or unchanged rounds into a single range notification (Requirement 3).
- Compress array-job identifier lists into index ranges everywhere the REPL displays multiple job identifiers together (Requirement 4).
- Leave `step` mode and every other existing REPL command byte-for-byte unchanged (Requirement 5).
### Non-Goals
- New REPL commands or newly inspectable data (limit usage, pend-reason detail) — tracked separately under the `scheduler-sim-diagnostics` brief.
- Any change to scheduling/allocation logic or to the deterministic outcome of a replay.
- Non-interactive (headless, no `-i`) scheduler output.
- Resolving bug-1611 (whether `DISPATCH` decisions are structurally invisible in replay mode) — this spec only guarantees Requirement 2.4 holds regardless of that investigation's outcome (see Design Decisions, `research.md`).
## Boundary Commitments
### This Spec Owns
- Console output formatting/verbosity logic inside `xce_repl_on_event()` (repl_break.c), the `continue`-mode gate on `sim_print_decision()`'s call site (sim_loop.c), and the new round-boundary rollup/collapse logic added at `xce_repl_post_session_check()` (repl.c).
- The new jobid-range-compression helper and every call site that adopts it.
- New `xce_sched_repl_state_s` fields required to track collapse state across rounds.
### Out of Boundary
- Any new `info`/`break`/`set` subcommand or newly inspectable scheduler data.
- Scheduling/allocation/decision logic itself (`scheduler.c`, `src/plugins/`).
- `sim_print_decision()`'s behavior when `sim->interactive` is false (headless mode) — unchanged.
- Root-causing bug-1611 — a separate investigation; this spec only ensures its own requirements hold regardless of the answer.
### Allowed Dependencies
- Existing `xce_sched_repl_state_s` fields (`mode`, `last_n_dispatch`/`last_n_pend`/`last_n_preempt`, `session_count`) — repl_internal.h.
- The existing `core->jobs` iteration idiom (`iter_make`/`iter_next`, `JOB_IS_PEND`) already used by `repl_info_jobs()`.
- Existing helpers `fmt_jobid_ex()` (lib/utils.c) and `repl_format_ts_human()` (repl.c).
- Must not add a new call site to `sim_loop.c`'s main loop beyond gating the two calls that already exist there (`sim_print_decision`, the `xce_repl_check_event` wrapper) — that file is documented at its line-budget ceiling (repl.h's own header comment).
### Revalidation Triggers
- If `scheduler-sim-diagnostics` adds pend-reason data, the round-rollup collapse fingerprint (Design Decisions, `research.md`) may need to incorporate it — revisit Requirement 3.2's implementation then.
- If bug-1611 is confirmed a real dispatch-visibility gap and separately fixed, the rollup's dispatch *count* source should switch from the decision tally to the same state-derived approach already used for the pend-jobid set.
- Any change to `xce_sched_repl_state_s`'s existing `mode`/tally fields by another spec should re-check this feature's rollup logic.
## Architecture
### Existing Architecture Analysis
- `repl.c`/`repl_break.c`/`repl_info.c`/`repl_params.c` form the REPL command family under `scheduler-core`. `repl.h` folds the `sim->interactive` gate into `static inline` wrapper functions so `sim_loop.c`'s own diff stays to single-line, brace-free call expressions — this feature's `sim_loop.c` change must follow that same convention rather than adding a bare `if` block to the loop body.
- `xce_sched_repl_state_s` (repl_internal.h) is the REPL's own session-scoped state struct, zero-initialized via `calloc` in `repl_state_ensure()` — new fields need no separate lifecycle.
- `xce_repl_post_session_check()` (repl.c) already runs exactly once per completed round, immediately after that round's decisions are drained, purely to check the `step`-budget pause condition today.
### Architecture Pattern & Boundary Map
**Correction (post-review)**: breakpoint-hit detection and structural-event classification both live in `repl_break.c` (`xce_repl_on_event`, `xce_repl_on_begin_round`, `xce_repl_on_decision` — each calls `repl_command_loop()` directly on a match), not in `repl.c`. Requirement 3.5 ("a structural event ends the active range") and Requirement 3.6 ("a breakpoint hit ends the active range") therefore cannot be satisfied by round-boundary logic in `repl.c` alone — `repl_break.c` needs a way to force-flush the Round Rollup's active range from its own call sites. The Round Rollup component now exposes an explicit flush entry point for this, called from both files.
```mermaid
flowchart TD
RoundEnd[Round decisions drained] --> Query[Query core jobs for PEND state]
Query --> Fingerprint[Build fingerprint: dispatch count, pend jobid set, preempt count]
Fingerprint --> Compare{Matches active collapsed range}
Compare -->|Yes| Extend[Extend active range silently]
Compare -->|No| FlushCall[Call round rollup flush]
FlushCall --> Begin[Begin new collapsed range with this round]
Extend --> NextRound[Advance to next round]
Begin --> NextRound
StructuralEvent[Structural event classified in repl_break.c] --> FlushCall
BreakpointHit[Breakpoint hit detected in repl_break.c] --> FlushCall
ReplayComplete[Replay completion in repl.c] --> FlushCall
FlushCall --> PrintRange[Print the flushed range, if one was open]
```
**Architecture Integration**:
- Selected pattern: a state accumulator attached to the existing round-boundary hook (`xce_repl_post_session_check`), exposing one shared flush entry point that `repl_break.c`'s event/breakpoint/decision hooks and `repl.c`'s replay-completion hook all call — no new hook call sites added to `sim_loop.c`.
- Domain/feature boundaries: the Round Rollup's state and flush/print logic live in `repl.c`; `repl_break.c` calls the flush entry point but does not own or mutate the collapse state directly; `sim_loop.c` only gains a mode gate on its two existing calls.
- Existing patterns preserved: `repl.h`'s `sim->interactive`-gated inline-wrapper convention; `repl.c`'s shared-helper convention (`repl_format_ts_human`, `repl_split_tokens`) extended with two new shared helpers (range compression, and the round-rollup flush entry point declared in `repl_internal.h` for `repl_break.c` to call).
- New components rationale: deciding "print now vs. extend silently" requires comparing against the *previous* round's fingerprint, which must persist across separate `xce_repl_post_session_check()` invocations — hence new persistent fields on `xce_sched_repl_state_s` rather than function-local state. The flush entry point exists because three independent call sites (round boundary, breakpoint hit, replay completion) across two files all need to end an active range the same way.
- Steering compliance: one domain per file preserved; no new production `.c` file needed; `snake_case`/`repl_*` naming conventions followed throughout.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|------------------|-------|
| Scheduler daemon (C, gnu17) | existing `src/scheduler/` toolchain, no new dependency | All logic lives in existing REPL/sim-driver translation units | Matches `tech.md`; no new library, no CGO/Go involvement |
## File Structure Plan
### Modified Files
- `src/scheduler/src/repl_internal.h` — extend `xce_sched_repl_state_s` with round-collapse state fields (see Data Models) and declare the new shared jobid-range-compression helper.
- `src/scheduler/src/repl_break.c` — `xce_repl_on_event()` gains routine-vs-structural event classification gated on `rs->mode` (Requirement 1); `repl_print_event_summary()`'s `JOB_PUSH`-style cases adopt the jobid-range-compression helper (Requirement 4); `xce_repl_on_event()`, `xce_repl_on_begin_round()`, and `xce_repl_on_decision()` each call the Round Rollup's flush entry point before printing a structural-event line or before re-entering `repl_command_loop()` on a breakpoint hit (Requirement 3.5, 3.6).
- `src/scheduler/src/repl.c` — `xce_repl_post_session_check()` gains the round-boundary fingerprint/collapse logic (Requirements 2, 3); implements the new shared jobid-range-compression helper (Requirement 4) and the Round Rollup's flush entry point (Requirement 3.5, 3.6), following the file's existing shared-helper convention; `xce_repl_on_replay_complete()` calls the same flush entry point before printing the existing replay-complete banner.
- `src/scheduler/src/sim_loop.c` — the existing call site invoking `sim_print_decision()` gains a `continue`-mode gate, mirroring `repl.h`'s inline-wrapper convention (Requirements 2, 5).
- `src/scheduler/include/repl.h` — no functional change expected; existing inline wrappers already cover the gating points this feature needs.
### New/Modified Test Files
- `src/scheduler/testsuite/test_repl_execution_control.c` — extended with child-process output-capture tests covering Requirements 1, 2, 3, and the Requirement 5 no-regression checks (this file already drives `sched_sim_driver_run()` end-to-end and captures real process output via `child_continue_then_inspect`-style helpers).
- `src/scheduler/testsuite/test_repl_output_compression.c` (new) — focused unit tests for the jobid-range-compression helper (Requirement 4): a pure function, tested without the process harness.
## System Flows
See the Architecture Pattern & Boundary Map diagram above — it is the only non-trivial flow this feature introduces (a per-round branching decision with two exit points: silent extension, or flush-and-restart). Per-event classification (Requirement 1) is a simple, non-branching gate and does not warrant a separate diagram.
## Requirements Traceability
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|-------------|-----------|-------|
| 1.1, 1.2, 1.3, 1.4 | Mode-aware event echo | Event Echo Gate | `xce_repl_on_event` | n/a (per-event, upstream of round boundary) |
| 2.1, 2.2, 2.3, 2.4 | Per-round decision summary | Round Rollup | `xce_repl_post_session_check`, state-derived pend query | Round-boundary flow |
| 3.1, 3.2, 3.3, 3.4 | Collapsing repetitive rounds (idle/unchanged detection, range content) | Round Rollup (collapse state) | fingerprint compare / flush | Round-boundary flow |
| 3.5, 3.6 | Ending an active collapsed range on outcome change, structural event, breakpoint hit, or replay completion | Round Rollup (flush entry point, and its own call from `xce_repl_on_replay_complete`), Event Echo Gate (calls it on structural events/breakpoint hits) | `repl_round_collapse_flush` | Round-boundary flow |
| 4.1, 4.2, 4.3, 4.4, 4.5 | Job identifier range compression | Jobid Range Compressor | `repl_fmt_jobid_range` | n/a (pure formatting) |
| 5.1, 5.2, 5.3, 5.4 | No regression to step mode / other commands | Event Echo Gate, `sim_print_decision` gate | mode checks | n/a |
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|---------------|--------|---------------|---------------------------|-----------|
| Event Echo Gate | REPL / repl_break.c | Classify each event as routine or structural and gate the per-event echo by mode | 1.1-1.4 | `rs->mode` (P0), Round Rollup flush entry point (P0) | State |
| Round Rollup | REPL / repl.c | Build a per-round fingerprint, print it, or extend/flush a collapsed range; expose a flush entry point callable from breakpoint/structural-event/replay-complete sites | 2.1-2.4, 3.1-3.6 | `core->jobs` query (P0), existing decision tally (P0) | State, Service |
| Jobid Range Compressor | REPL / repl.c (shared helper) | Compress array-job identifier lists into index ranges | 4.1-4.5 | `fmt_jobid_ex` decomposition (P0) | Service |
| `sim_print_decision` gate | Sim driver / sim_loop.c | Suppress the direct per-decision print when the Round Rollup will report it instead | 2.3, 5.1 | REPL mode state (P0) | State |
### REPL Output (repl.c / repl_break.c / sim_loop.c)
#### Event Echo Gate
| Field | Detail |
|-------|--------|
| Intent | Classify each decoded event and suppress its individual echo line under `continue`, while breakpoint matching still evaluates every event |
| Requirements | 1.1, 1.2, 1.3, 1.4 |
**Responsibilities & Constraints**
- Structural events (always echoed, any mode): `CONFIG_PUSH`, `JOBS_PUSH`, `WORKERS_PUSH`, `RESOURCES_PUSH`, `RESVALS_FULL` — population/config arrivals an operator needs to see even while `continue` is otherwise quiet. Before printing a structural event's line, this gate calls the Round Rollup's flush entry point so any active collapsed range is closed and reported first (Requirement 3.5) — a structural event and a stale collapsed range must never interleave out of order.
- Routine events (echoed only in `step` mode): `RESVALS_DELTA`, `JOB_STATE_CHANGE`, `JOB_UPDATE`, `QUEUE_UPDATE`, `JOB_RUSAGE`. `JOB_STATE_CHANGE` is classified routine deliberately — the Round Rollup (Requirement 2.4) already surfaces job status transitions at round granularity, so an individual `JOB_STATE_CHANGE` line under `continue` would duplicate that information.
- Breakpoint matching (`repl_find_event_breakpoint`, plus the equivalent checks in `xce_repl_on_begin_round`/`xce_repl_on_decision`) runs on every event regardless of echo suppression — Requirement 1.4 is unaffected by this gate. On a match, before re-entering `repl_command_loop()`, the matching hook calls the Round Rollup's flush entry point so the operator sees the collapsed range's final state before the prompt returns (Requirement 3.6).
**Dependencies**
- Inbound: `xce_repl_on_event` — calls this gate before its existing breakpoint check (P0)
- Outbound: `repl_print_event_summary` — invoked only when the gate allows it (P0); Round Rollup's flush entry point — called before a structural-event line or a breakpoint-triggered command-loop re-entry (P0)
**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [x]
##### State Management
- State model: reads `sim->repl_state->mode` (existing field); no new state owned by this component.
- Persistence & consistency: none — a per-call decision.
- Concurrency strategy: none required — single-threaded sim driver, same as every other REPL hook.
#### Round Rollup
| Field | Detail |
|-------|--------|
| Intent | Once per completed round in `continue` mode, compute a fingerprint and either extend an active collapsed range or flush it and begin reporting the new state |
| Requirements | 2.1, 2.2, 2.3, 2.4, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 |
**Responsibilities & Constraints**
- Active only while `sim->interactive && rs->mode == XCE_REPL_MODE_CONTINUE`; `step` mode keeps today's full per-decision path unchanged (Requirement 2.3, 5.1).
- Pend-jobid set is sourced by directly querying `core->jobs` for `JOB_IS_PEND` state (same iteration idiom as `repl_info_jobs()`) — not solely from the decision tally, so Requirement 2.4 holds independent of bug-1611 (see `research.md` Design Decisions).
- Dispatch/preempt counts are sourced from the existing `last_n_dispatch`/`last_n_preempt` tally.
- Must flush (print) any active collapsed range before: a breakpoint-triggered re-entry into `repl_command_loop()`, replay completion, or a fingerprint change from the active range (Requirement 3.5, 3.6). Because the breakpoint and structural-event triggers originate in `repl_break.c`, not `repl.c`, this component exposes a flush entry point other files call rather than owning every trigger site itself.
**Dependencies**
- Inbound: `xce_repl_post_session_check` — the round-boundary call site the fingerprint/collapse logic attaches to (P0); the Event Echo Gate (`xce_repl_on_event`) and the breakpoint checks in `xce_repl_on_begin_round`/`xce_repl_on_decision` (all repl_break.c) — call the flush entry point (P0); `xce_repl_on_replay_complete` (repl.c) — calls the flush entry point before its existing banner (P0)
- Outbound: `core->jobs` iterator (P0), `xce_repl_on_decision`'s existing tally fields (P0), Jobid Range Compressor for rendering the pend-jobid list (P0)
**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [x]
##### Service Interface
```c
/*
* Ends the active collapsed round range, if one is open, printing its
* summary line immediately. Safe to call when no range is active (a
* no-op). Called from the round boundary on a fingerprint change, and
* from repl_break.c before a structural-event line or a breakpoint-
* triggered command-loop re-entry, and from xce_repl_on_replay_complete
* before its existing banner.
*/
void repl_round_collapse_flush(xce_sched_sim_driver_t* sim);
```
- Preconditions: `sim`/`sim->repl_state` non-NULL (same "safe no-op on NULL" discipline every other REPL hook in this file family already follows).
- Postconditions: no collapsed range is active after the call; if one was active, its summary line has been printed exactly once.
- Invariants: idempotent — calling it again immediately afterward with no intervening round is a no-op, never a duplicate print.
##### State Management
- State model: new fields on `xce_sched_repl_state_s` hold the active collapsed range (see Data Models) — whether one is open, its round/timestamp bounds, and its cached fingerprint.
- Persistence & consistency: in-memory, scoped to one REPL session; zero-initialized for free by `repl_state_ensure()`'s existing `calloc`.
- Concurrency strategy: none required — single-threaded sim driver.
#### Jobid Range Compressor
| Field | Detail |
|-------|--------|
| Intent | Group a list of jobids by array base and render contiguous index runs as compressed ranges |
| Requirements | 4.1, 4.2, 4.3, 4.4, 4.5 |
**Responsibilities & Constraints**
- Pure, read-only formatting function — never mutates its input list or any scheduler/job state.
- Shared by `repl_print_event_summary()`'s job-arrival listings and the Round Rollup's pend-jobid display (Requirement 4.5).
**Dependencies**
- Inbound: `repl_print_event_summary` (P0), Round Rollup (P0)
- Outbound: `fmt_jobid_ex` (lib/utils.c) — reuses its existing `(cluster, apiserver, timestamp, job[, idx])` decomposition (P0)
**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]
##### Service Interface
```c
/*
* Formats `njobids` raw jobids into `buf` (bufsz-bounded): groups by
* array base (cluster, apiserver, timestamp, job) and compresses
* contiguous [idx] runs into [lo-hi] form, comma-joining non-contiguous
* runs within one base; non-array jobids are listed individually.
* Returns the number of bytes written (excluding the NUL terminator),
* or a negative value if buf was too small to hold the full listing.
*/
int repl_fmt_jobid_range(char* buf, size_t bufsz,
const uint64_t* jobids, int njobids);
```
- Preconditions: `jobids` non-NULL whenever `njobids > 0`; `buf`/`bufsz` describe a valid, writable buffer.
- Postconditions: `buf` holds a NUL-terminated, comma-separated listing; distinct array bases appear in input order, indices ascend within each base's compressed ranges.
- Invariants: never fabricates or silently drops a jobid — the compressed listing always represents exactly the same id set the uncompressed listing would show.
#### `sim_print_decision` Gate
| Field | Detail |
|-------|--------|
| Intent | Suppress `sim_print_decision()`'s direct per-decision stdout line when the Round Rollup will report the round instead |
| Requirements | 2.3, 5.1 |
**Responsibilities & Constraints**
- Skip the direct print only when `sim->interactive && sim->repl_state->mode == XCE_REPL_MODE_CONTINUE`; `step` mode and non-interactive (headless) runs are unaffected (Requirement 5.1, 5.4).
- Implemented as an inline-wrapper-style gate at the existing call site in `sched_sim_driver_run()`, consistent with `repl.h`'s established convention, rather than a bare `if` block added to the loop body.
**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [ ] / State [x]
##### State Management
- State model: reads `sim->interactive` and `sim->repl_state->mode` (both existing fields); owns no new state.
## Data Models
### Domain Model
This feature adds a "round collapse state" concept to the REPL's existing session-scoped state (`xce_sched_repl_state_s`, repl_internal.h) — not a new top-level entity. It follows the same pattern prior REPL task revisions used to add `output_path`/`replay_complete` to that struct.
### Logical Data Model
New fields on `xce_sched_repl_state_s` (illustrative — exact types finalized at implementation time):
| Field | Type | Purpose |
|-------|------|---------|
| `collapse_active` | `bool` | Whether a collapsed range is currently open |
| `collapse_start_round` / `collapse_end_round` | `uint64_t` | Round-number bounds of the active range |
| `collapse_start_ts` / `collapse_end_ts` | `uint64_t` | Sim-time bounds, for the collapsed range's time span (Requirement 3.4) |
| `collapse_dispatch_count` / `collapse_preempt_count` | `int` | Cached fingerprint counts |
| `collapse_pend_jobids` / `collapse_n_pend_jobids` | pointer + count | Cached fingerprint pend-jobid set, in whatever list form the Jobid Range Compressor consumes |
**Consistency & Integrity**: all fields reset together whenever a range flushes (print, then clear) or a new REPL session begins — `repl_state_ensure()`'s existing `calloc` zero-initializes new fields with no additional lifecycle code required.
No Physical Data Model, API Data Transfer, or Event Schema subsections apply: this feature has no persistent storage, no network wire format, and adds no new wire-level contract — it stays inside `scheduler-core`'s existing "simulation isolation" boundary (repl.h's own header comment: no `scheduler.h`, no `client.h`, no capnp types).
## Error Handling
### Error Strategy
Consistent with existing `repl.c`/`repl_break.c` conventions: allocation failures print an existing-style `"[sim] ..."` message and degrade to the safest available behavior rather than crashing or losing information, matching `repl_state_ensure()`'s own "never load-bearing, degrade to unattended" precedent.
### Error Categories and Responses
- **Allocation failure while building a round's fingerprint or growing the collapsed-range pend-jobid buffer**: fall back to printing that round's line directly (uncollapsed) rather than losing the round's information — preserves Requirement 2.1's obligation to report every round with decisions.
- **`repl_fmt_jobid_range()` buffer too small**: returns a negative value; the caller falls back to the existing uncompressed per-id listing for that call — never truncates silently, never crashes, preserves the "never drop a jobid" invariant even in a buffer-constrained edge case.
### Monitoring
No new logging framework is introduced — the existing `[sim] ...` stdout convention remains the only user-facing signal; this is an interactive CLI tool, not a daemon, so no zap/WARN-level logging applies here.
## Testing Strategy
### Unit Tests (new `test_repl_output_compression.c`)
- Contiguous indices from one array base compress to a single `[lo-hi]` range (Requirement 4.1).
- Non-contiguous indices compress to comma-joined sub-ranges within one base (Requirement 4.2).
- Multiple array bases in one listing compress independently (Requirement 4.3).
- A mixed array + non-array listing shows non-array ids individually and compresses only the array ids (Requirement 4.4).
- A buffer too small to hold the full listing returns a negative value and writes nothing past `bufsz` (Error Handling).
### Integration Tests (extend `test_repl_execution_control.c`, child-process output capture)
- `continue` across multiple idle rounds (zero decisions) produces one collapsed range line, not one line per round (Requirement 3.1).
- `continue` across rounds sharing an identical nonzero pend set produces one collapsed "unchanged" range line — the exact shape hand-validated against the real 108-round replay that motivated this spec (Requirement 3.2).
- A structural event mid-`continue` ends any active collapsed range and prints its own line before continuing (Requirement 1.3, 3.5).
- A breakpoint hit mid-collapse flushes the active range before entering the command loop (Requirement 3.6).
- Replay completion (EOF) flushes any active collapsed range before the existing "replay complete" banner (Requirement 3.6).
- `step`-mode output is unchanged versus this feature's pre-implementation baseline — regression check alongside the existing `test_should_pause_fires_on_exactly_the_nth_session`-style fixtures (Requirement 5.1, 5.3).
- A job's tracked status changing between rounds is reflected in the next round summary even under a fixture that never emits a synthetic `DISPATCH` decision — exercises the state-derived-pend-set design decision independent of bug-1611's resolution (Requirement 2.4).
### E2E
Not applicable beyond the integration tests above — this feature has no HTTP/API surface; the child-process harness already exercises the real `xce_scheduler` binary end-to-end at the process level.
## Optional Sections
Security Considerations, Performance & Scalability, and Migration Strategy are omitted: this is a local, single-operator interactive CLI output change with no new attack surface, no persistent or high-load data path (round counts are bounded by replay length, already tolerated by existing `step` output), and no schema or data migration.
## Supporting References
None beyond `research.md` — no external API, vendor library, or long schema requires separate documentation.

59
.kiro/specs/scheduler-sim-repl-verbosity/requirements.md Normal file
View file

@ -0,0 +1,59 @@
# Requirements Document
## Introduction
xce_scheduler's `--interactive` simulation REPL (shipped by the `scheduler-core` spec) lets an operator step or continue through a recorded replay while inspecting scheduler state. Under `continue`, the REPL currently prints the same volume of output as single-stepping every round — a real 108-round replay produced roughly 2,700 lines, most of it either routine per-event chatter that carries no new information under `continue`, or an unchanged pending-job decision repeated identically round after round. This feature reorganizes REPL output during `continue` execution so an operator can track a long replay's progress without wading through repetitive, low-information lines, while leaving `step` mode and every other REPL command unchanged.
## Boundary Context
- **In scope**: console output shown by the `--interactive` simulation REPL during `continue` execution — which events are echoed, how scheduling-decision outcomes are summarized per round, how repetitive rounds are collapsed, and how multi-job identifier lists are displayed.
- **Out of scope**: any new REPL command or newly inspectable data (for example limit usage or pend-reason detail — tracked separately under `scheduler-sim-diagnostics`); any change to scheduling/allocation behavior or to the deterministic outcome of a replay; any change to `step` mode's existing output; non-interactive (headless, no `-i`) scheduler output.
- **Adjacent expectations**: this feature extends the existing REPL shipped by `scheduler-core` (implementation-complete) without changing its command surface. `scheduler-sim-diagnostics` (currently a discovery brief, not yet a formal spec) is expected to separately add pend-reason and limit-usage inspection; this feature's round summaries should remain compatible with, and not duplicate, that future work.
## Requirements
### Requirement 1: Mode-Aware Event Echo During Continuous Execution
**Objective:** As an operator running a simulation replay, I want `continue` to suppress routine per-event chatter while still alerting me to major state changes, so that I can watch a long replay without being flooded by lines carrying no new information.
#### Acceptance Criteria
1. While the REPL is executing in continue mode, when a routine per-event notification (a job progress update, a job resource-usage update, or a resource value update) occurs, the REPL shall not print an individual line for that event.
2. While the REPL is executing in step mode, when any event occurs, the REPL shall print an individual line for that event.
3. When a structural change occurs (new jobs submitted, new workers added, new resources added, or configuration pushed), the REPL shall print a line describing that change regardless of execution mode.
4. When an event matches a defined breakpoint, the REPL shall interrupt continue-mode execution and enter the interactive command loop, regardless of whether the event is routine or structural.
### Requirement 2: Per-Round Decision Summary
**Objective:** As an operator watching simulation replay progress, I want one summary line per round showing what scheduling decisions were made, so that I can track scheduling outcomes without reading a separate line per affected job.
#### Acceptance Criteria
1. While the REPL is executing in continue mode, when a round completes with at least one scheduling decision (dispatch, pend, or preempt), the REPL shall print one summary line for that round showing the count of each decision type and the identifiers of the jobs affected.
2. When a round completes with zero scheduling decisions, the REPL shall not print an individual decision line for that round.
3. While the REPL is executing in step mode, the REPL shall print full per-decision detail for every decision.
4. When a job's tracked status changes from one round to the next, the REPL's round summary shall reflect that change.
### Requirement 3: Collapsing of Repetitive Round Output
**Objective:** As an operator, I want consecutive rounds that produce no new information to be reported as a single range instead of one line each, so that long stretches of idle or unchanged simulation state don't dominate the output.
#### Acceptance Criteria
1. When two or more consecutive rounds occur with zero scheduling decisions, the REPL shall report them as a single collapsed range instead of one line per round.
2. When two or more consecutive rounds occur with an identical decision outcome (same decision counts and the same affected job identifiers), the REPL shall report them as a single collapsed range instead of repeating the same summary line for each round.
3. The REPL shall include the first round number, the last round number, and the round count in every collapsed range notification.
4. Where round timestamps are displayed, the REPL shall include the covered time span in each collapsed range notification.
5. When a round's decision outcome differs from the immediately preceding reported round, or a structural event occurs, the REPL shall end the active collapsed range and report it before continuing.
6. When continue-mode execution stops for any reason (a breakpoint hit or replay completion), the REPL shall end and report any active collapsed range before the stop is reported to the operator.
### Requirement 4: Job Identifier Range Compression
**Objective:** As an operator reading REPL output that lists many job identifiers from the same job array submission, I want those identifiers compressed into an index range, so that I can read the list without scrolling past dozens of nearly-identical entries.
#### Acceptance Criteria
1. When the REPL displays two or more job identifiers that belong to the same job array submission and have contiguous array indices, the REPL shall display them as a single compressed index range.
2. When job identifiers from the same job array submission have non-contiguous indices, the REPL shall display each contiguous sub-range separately within the same listing.
3. When a listing contains job identifiers from more than one job array submission, the REPL shall group and compress each job array submission's identifiers independently.
4. When a listing mixes job-array and non-array job identifiers, the REPL shall display the non-array job identifiers individually while compressing the job-array identifiers as described above.
5. The REPL shall apply job identifier range compression consistently everywhere it displays multiple job identifiers together, including job-arrival notifications and round decision summaries.
### Requirement 5: No Change to Step-Mode Detail or Existing Commands
**Objective:** As an operator relying on today's `step` behavior and `info`/`break`/`delete`/`set`/`snapshot` commands, I want this feature to leave them unchanged, so that existing debugging workflows keep working exactly as before.
#### Acceptance Criteria
1. The REPL shall preserve existing `step` command output exactly as before this feature, with no reduction in per-event or per-decision detail.
2. The REPL shall preserve existing `info`, `break`, `delete`, `set`, and `snapshot` command behavior and output format unchanged.
3. The REPL shall introduce no new command as part of this feature.
4. The REPL shall not alter the scheduling decisions or final simulation state produced by a replay — only the console output describing them.

77
.kiro/specs/scheduler-sim-repl-verbosity/research.md Normal file
View file

@ -0,0 +1,77 @@
# Research & Design Decisions
## Summary
- **Feature**: `scheduler-sim-repl-verbosity`
- **Discovery Scope**: Extension (existing system) — light discovery
- **Key Findings**:
- Two independent, unconditional stdout print paths cause `continue`-mode volume to match `step`-mode volume: `xce_repl_on_event()` (repl_break.c:708-731, gated only by `sim->interactive`, not by `rs->mode`) and `sim_print_decision()` (sim_loop.c:388-411, called directly from `sched_sim_driver_run()`'s main loop with no `sim->interactive`/mode gate at all).
- `xce_repl_post_session_check()` (repl.c), called once per completed round from `sched_sim_driver_run()` right after each round's decisions are drained, is the correct integration point for round-boundary rollup/collapse logic — it already runs exactly once per round when interactive, and already exists purely for the `step`-budget pause check.
- `xce_repl_on_decision()` (repl_break.c:776-799) already tallies `last_n_dispatch`/`last_n_pend`/`last_n_preempt` per round, but that tally is decision-event-driven — real replay data shows it may never observe a fresh `DISPATCH` decision in replay mode even when jobs demonstrably transition to running (bug-1611). The pend-jobid set for the round rollup must therefore be derived by directly querying `core->jobs` for `JOB_IS_PEND` state (the same iteration idiom `repl_info_jobs()` already uses), not solely from the decision tally, to satisfy Requirement 2.4 regardless of bug-1611's outcome.
- Existing test file `test_repl_execution_control.c` already drives `sched_sim_driver_run()` end-to-end via a child-process harness (`child_continue_then_inspect`, `child_continue_twice_after_complete`) that captures real process output — this is the correct home for behavioral tests of Requirements 1-3 (mode gating, rollup, collapsing), since it observes actual printed text, not just internal state.
- No new external dependency is needed — all formatting is plain C string/stdio work consistent with the rest of `repl.c`/`repl_break.c`.
## Research Log
### Extension point analysis: where does continue-mode output actually originate?
- **Context**: Requirements 1-3 need continue mode to suppress/summarize output that today prints unconditionally.
- **Sources Consulted**: `repl_break.c` (`xce_repl_on_event`, `xce_repl_on_decision`), `sim_loop.c` (`sim_print_decision`), `repl.c` (`xce_repl_post_session_check`, `repl_should_pause`), `repl.h` (interactive-gated inline wrappers), `repl_internal.h` (`xce_sched_repl_state_s`), CodeGraph call-graph for `sched_sim_driver_run`.
- **Findings**:
- `xce_repl_check_event()` (repl.h, `static inline`) gates `xce_repl_on_event()` on `sim->interactive` only — inside `xce_repl_on_event()`, `repl_print_event_summary()` fires unconditionally before any breakpoint check, with no reference to `rs->mode`.
- `sim_print_decision()` has no interactive or mode gate at all; `sched_sim_driver_run()` calls it directly for every drained decision, the same call site that also drives `xce_repl_check_decision()` (which does the tally) and `sched_record_session_add_decision()` (recording).
- `xce_repl_post_session_check()` fires once per completed round, strictly after that round's decisions are drained and before the next round's events begin — confirmed via call-graph (`sched_sim_driver_run → xce_repl_post_session_pause`).
- **Implications**: Requirement 1's gate belongs inside `xce_repl_on_event()`/`repl_print_event_summary()`. Requirement 2/3's rollup and collapsing logic belongs at the `xce_repl_post_session_check()` boundary, reading state that `xce_repl_on_decision()`'s existing tally and a direct `core->jobs` query both feed. Requirement 2/3 also requires suppressing `sim_print_decision()`'s direct per-decision print when `sim->interactive && rs->mode == XCE_REPL_MODE_CONTINUE`, since that print path is the one bug-1610 identified as unconditional.
### Job identifier range compression: existing formatting conventions
- **Context**: Requirement 4 needs contiguous array-job indices displayed as ranges.
- **Sources Consulted**: `lib/utils.c` (`fmt_jobid`/`fmt_jobid_ex`), `repl_break.c`'s `repl_print_event_summary()` (current `JOB_PUSH` jobid list rendering).
- **Findings**: `fmt_jobid_ex()` already separates the packed 64-bit jobid into `(cluster, apiserver, timestamp, job[, idx])` components via `EXTRACT_*` macros; an array job's `idx` is already isolated during formatting. There is no existing range-compression helper anywhere in the REPL formatting code — every current jobid list (the `JOB_PUSH` line, and any future round-rollup pend list) is one `fmt_jobid()` call per id.
- **Implications**: A new helper is needed; the natural base for grouping is the already-decoded `(cluster, apiserver, timestamp, job)` tuple (identical for every element of one array submission), with only `idx` varying. No change to `fmt_jobid`/`fmt_jobid_ex` themselves — the new helper wraps a *list* of ids, not a single id.
## Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Gate at existing hook boundaries (chosen) | Add mode/collapse logic inside `xce_repl_on_event`, `sim_print_decision`'s call site, and `xce_repl_post_session_check` | Reuses every existing per-round/per-event integration point; no new hook call sites needed (sim_loop.c already at its documented line-budget ceiling per repl.h's own header comment) | Requires touching two files (`repl_break.c`, `sim_loop.c`) instead of one | Matches this codebase's own precedent (repl.h's inline-wrapper gating) |
| New dedicated "output policy" module | Introduce a separate `repl_output.c` owning all print decisions, called from a single new hook | Cleaner separation of concerns | Adds a 6th call site into `sim_loop.c`'s round loop, which repl.h's own header comment says there is no line budget for; larger diff for a formatting-only feature | Rejected — over-engineered for the boundary this spec owns |
## Design Decisions
### Decision: Round rollup's pend-jobid set is state-derived, not decision-tally-derived
- **Context**: Requirement 2.4 requires the round summary to reflect real job status changes even though bug-1611 shows the decision tally may never register a fresh `DISPATCH` in replay mode.
- **Alternatives Considered**:
1. Trust `xce_repl_on_decision()`'s existing `last_n_dispatch`/`last_n_pend`/`last_n_preempt` tally directly for the pend-jobid set — simplest, but inherits bug-1611's blind spot if it turns out to be a real gap rather than benign replay semantics.
2. Derive the pend-jobid set each round by directly iterating `core->jobs` for `JOB_IS_PEND` state (same idiom as `repl_info_jobs()`), independent of the decision tally.
- **Selected Approach**: Option 2 for the pend-jobid set; the existing tally is still used for the dispatch/preempt *counts* shown in the rollup header, since bug-1611 only questions dispatch visibility and preempt is unaffected.
- **Rationale**: Requirement 2.4 must hold regardless of how bug-1611 resolves. A state query is authoritative by construction; a decision-tally is only as complete as whatever decisions the replay happens to (re-)generate.
- **Trade-offs**: One extra `core->jobs` iteration per round in continue mode (bounded by job count, not a hot path — replay/step already tolerates equivalent iteration cost in `repl_info_jobs()`/`repl_count_jobs()`). If bug-1611 is later confirmed a real gap, the dispatch *count* shown in the rollup may still under-report until that separate bug is fixed — flagged as a known limitation, not silently masked.
- **Follow-up**: bug-1611 should be investigated (separately from this spec) before or during implementation; if it proves to be a real visibility gap rather than benign replay semantics, a follow-up task should switch the dispatch count to the same state-diff approach used for the pend set.
### Decision: Collapse fingerprint keys on (dispatch_count, pend_jobid_set, preempt_count), not raw decision objects
- **Context**: Requirement 3.2 needs to detect when consecutive rounds are "the same" to collapse them.
- **Alternatives Considered**:
1. Compare full decision-object lists round to round.
2. Compare a compact fingerprint: dispatch count + preempt count + the state-derived pend-jobid set (as an ordered/canonical list).
- **Selected Approach**: Option 2.
- **Rationale**: The real replay validated this exact fingerprint shape — 65 consecutive rounds shared an identical `(0, {8 jobids from base A + 6 jobids from base B}, 0)` fingerprint and collapsed correctly by hand-simulation. Comparing raw decision objects would be more expensive and no more precise, since the pend-jobid set already captures every observable difference the operator cares about.
- **Trade-offs**: A change in *why* a job is pending (a future pend-reason feature under `scheduler-sim-diagnostics`) would not by itself break the collapse, since the fingerprint doesn't include pend reason. Acceptable for this spec's boundary — pend-reason display is explicitly out of scope here.
- **Follow-up**: If `scheduler-sim-diagnostics` later adds pend-reason display, revisit whether the fingerprint needs to include it (a revalidation trigger, see design.md Boundary Commitments).
### Decision: Round Rollup exposes a shared flush entry point instead of owning every flush trigger itself
- **Context**: An independent task-graph sanity review (run before task generation) found that breakpoint-hit detection and structural-event classification both live in `repl_break.c` (`xce_repl_on_event`, `xce_repl_on_begin_round`, `xce_repl_on_decision`, each calling `repl_command_loop()` directly on a match) — not in `repl.c` where the initial design placed all Round Rollup logic. Requirements 3.5 and 3.6 cannot be satisfied by round-boundary logic alone.
- **Alternatives Considered**:
1. Duplicate collapse-flush logic inside `repl_break.c` as well as `repl.c` — rejected, violates single-ownership of the collapse state and risks divergent flush behavior.
2. Move all breakpoint/structural-event detection into `repl.c` so Round Rollup owns every trigger directly — rejected, a much larger and riskier change to existing, already-tested breakpoint-matching code with no benefit to this spec's boundary.
3. Round Rollup exposes one flush entry point (`repl_round_collapse_flush`); `repl_break.c`'s existing hooks call it before their existing structural-event print / breakpoint-triggered command-loop entry.
- **Selected Approach**: Option 3.
- **Rationale**: Keeps collapse-state ownership entirely inside `repl.c` (single writer, no duplicated logic) while letting the two other legitimate trigger sites (`repl_break.c`'s breakpoint/structural-event detection, and `repl.c`'s own `xce_repl_on_replay_complete`) call a small, idempotent, safe-on-NULL interface — consistent with this codebase's existing "safe no-op on NULL/not-yet-allocated state" convention already documented on every other REPL hook.
- **Trade-offs**: Event Echo Gate (repl_break.c) now has a real outbound dependency on Round Rollup's interface, removing the parallel-safety the initial task draft assumed between those two components' implementation work — tests for both can still be written independently, but implementation must be sequenced.
- **Follow-up**: none — resolved before task generation; `design.md` and the task graph both reflect this.
## Risks & Mitigations
- **bug-1611 (dispatch visibility in replay mode) is unresolved** — Mitigation: pend-set correctness (Requirement 2.4) does not depend on its resolution (see Design Decision above); dispatch-count accuracy is called out as a known, separately-tracked limitation rather than silently assumed correct.
- **`sim_print_decision()`'s exact call site inside `sched_sim_driver_run()` was confirmed via call-graph but its surrounding loop body was not fully read** — Mitigation: implementation task must re-confirm the precise insertion point and existing control flow around it before editing; this is a normal implementation-time verification, not a design ambiguity (the gating *contract* — suppress under `sim->interactive && continue mode` — is unambiguous regardless of exact line placement).
- **No existing test coverage on `xce_repl_on_event`/`sim_print_decision` text output** (per CodeGraph) — Mitigation: Requirement-driven tests added in `test_repl_execution_control.c` (child-process output capture) before implementation, per this repo's TDD mandate.
## References
- `.wolf/buglog.json` bug-1609, bug-1610, bug-1611 — the three findings that motivated this spec, each with file:line root causes already recorded.
- `.kiro/specs/scheduler-sim-diagnostics/brief.md` — sibling brief; Requirement 2.4's "why did it pend" is explicitly out of this spec's boundary and belongs to that brief's seed capability 2 if formalized later.

22
.kiro/specs/scheduler-sim-repl-verbosity/spec.json Normal file
View file

@ -0,0 +1,22 @@
{
"feature_name": "scheduler-sim-repl-verbosity",
"created_at": "2026-08-21T06:50:37Z",
"updated_at": "2026-08-21T07:10:00Z",
"language": "en",
"phase": "tasks-generated",
"approvals": {
"requirements": {
"generated": true,
"approved": true
},
"design": {
"generated": true,
"approved": true
},
"tasks": {
"generated": true,
"approved": true
}
},
"ready_for_implementation": true
}

69
.kiro/specs/scheduler-sim-repl-verbosity/tasks.md Normal file
View file

@ -0,0 +1,69 @@
# Implementation Plan
- [ ] 1. Foundation: job identifier range compression
- [ ] 1.1 Write failing unit tests for compressing job-array identifier listings into index ranges
- Cover a single contiguous range, non-contiguous indices within one array submission, multiple distinct array submissions in one listing, a mix of array and non-array identifiers, and an output buffer too small to hold the full listing
- Observable: the new test cases execute and fail against the current codebase, which has no compression behavior yet
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
- [ ] 1.2 Implement job identifier range compression so the new tests pass
- Group identifiers by their shared array submission and render contiguous index runs as a single compressed range, comma-joining non-contiguous runs
- Never drop or fabricate an identifier, and fail safely (without truncating or crashing) when the output buffer is too small
- Observable: all tests from 1.1 pass; the helper is ready for the event-echo and round-summary work to adopt
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5_
- [ ] 2. Core: per-round decision summary, repetitive-round collapsing, and the shared range-flush capability
- [ ] 2.1 Write failing tests for round-level decision summaries, collapsing, and the shared flush capability
- Cover: a round with at least one decision produces one summary line naming decision counts and affected job identifiers; a round with zero decisions produces no individual line; two or more consecutive idle rounds collapse into a single range notification; two or more consecutive rounds with an identical decision outcome collapse into a single range notification; a collapsed range notification states its round bounds, round count, and covered time span; a change in outcome ends an active collapsed range; a job's status change between rounds is reflected in the following round's summary even without a dedicated dispatch event to observe; the shared flush capability safely does nothing when no range is active and is safe to call more than once in a row
- Observable: the new test cases fail against today's unconditional per-decision echo, which has no summarizing, collapsing, or flush behavior
- _Requirements: 2.1, 2.2, 2.4, 3.1, 3.2, 3.3, 3.4_
- _Boundary: Round Rollup_
- _Depends: 1.2_
- [ ] 2.2 Implement the round-boundary decision summary, collapsing logic, and the shared range-flush capability
- Add the state needed to track an active collapsed range across rounds
- At each round boundary during continuous execution, determine the round's decision outcome directly from current job status (not solely from decision events) together with the existing per-round decision tally, then extend an active matching range or end it and begin reporting the new outcome
- Expose the range-ending behavior so other parts of the REPL can trigger it safely, including when nothing is active; wire it into the existing replay-completion notification so a range open at end-of-replay is closed and reported before that notification prints
- Observable: all tests from 2.1 pass; replaying the real 108-round capture that motivated this feature under continuous execution now produces the hand-validated collapsed shape (an idle range before the first job arrival, one line per job arrival, and one collapsed range per stable pending stretch, with any range still open at end-of-replay closed out before the completion notice) instead of thousands of repeated lines
- _Requirements: 2.1, 2.2, 2.4, 3.1, 3.2, 3.3, 3.4, 3.6_
- _Boundary: Round Rollup_
- [ ] 3. (P) Core: mode-aware event echo, including ending an active collapsed range on structural events or breakpoint hits — Depends: 2.2
- [ ] 3.1 (P) Write failing tests for suppressing routine event chatter and for ending an active collapsed range at the right moments
- Cover: a routine per-tick event (job progress, resource usage, or resource value update) produces no line under continuous execution; the same event still produces a line while single-stepping; a structural change (new jobs, workers, resources, or configuration arriving) still produces a line under continuous execution and ends any active collapsed range first; an event matching a breakpoint still interrupts continuous execution regardless of its classification, and ends any active collapsed range before the interruption is shown
- Observable: the new test cases fail against today's unconditional per-event echo, which has no classification or range-ending behavior
- _Requirements: 1.1, 1.2, 1.3, 1.4, 3.5, 3.6_
- _Boundary: Event Echo Gate_
- _Depends: 1.2, 2.2_
- [ ] 3.2 (P) Implement the mode-aware event echo, range compression adoption, and range-ending wiring
- Suppress routine per-event lines during continuous execution while preserving full detail while single-stepping; keep structural-change lines and breakpoint interruption behavior unaffected by the classification
- Render job-arrival listings using the compressed range format
- Before printing a structural-change line, and before a breakpoint interruption is shown, end any active collapsed range first so output never interleaves out of order
- Observable: all tests from 3.1 pass; a continuous-execution run over a replay with only routine activity produces no per-event lines, a job arrival still prints one compressed line and correctly closes out any prior collapsed range, and a breakpoint hit shows the collapsed range's final state before the interactive prompt returns
- _Requirements: 1.1, 1.2, 1.3, 1.4, 3.5, 3.6, 4.5_
- _Boundary: Event Echo Gate_
- [ ] 4. (P) Integration: retire the redundant per-decision echo during continuous execution — Depends: 2.2
- [ ] 4.1 (P) Write failing tests confirming the old per-decision echo no longer duplicates the new round summary
- Cover: during continuous execution, no bare per-decision line appears outside the round summary; while single-stepping, full per-decision detail still prints exactly as before; a non-interactive (headless) replay's decision output is unaffected
- Observable: the new test cases fail while the old unconditional per-decision print path still runs alongside the new round summary
- _Requirements: 2.3, 5.1, 5.4_
- _Boundary: sim_print_decision gate_
- _Depends: 2.2_
- [ ] 4.2 (P) Suppress the old per-decision echo during continuous execution so the round summary is the sole source of decision output
- Leave single-stepping and non-interactive output untouched
- Observable: all tests from 4.1 pass; a full continuous-execution run over a replay shows only round-summary lines, with no duplicate bare decision lines
- _Requirements: 2.3, 5.1, 5.4_
- _Boundary: sim_print_decision gate_
- [ ] 5. Validation: regression and real-data verification — Depends: 3.2, 4.2
- [ ] 5.1 Verify single-stepping and every other existing REPL command remain unchanged
- Confirm single-stepping output, and the inspection/breakpoint/parameter/snapshot commands' behavior and output, match their pre-feature baseline exactly; confirm no new command was introduced
- Observable: the existing REPL test suite's pre-existing assertions all still pass unmodified, and a step-mode transcript comparison shows no textual difference from the pre-feature baseline
- _Requirements: 5.1, 5.2, 5.3_
- [ ] 5.2 Run the full scheduler test suite and re-verify against the real replay capture that originally motivated this feature
- Observable: the full test suite passes with no regressions, and re-running the interactive REPL's continuous execution against the real capture reproduces the collapsed output shape validated during design, cutting a several-thousand-line transcript down to a handful of lines while preserving every state change
- _Requirements: 2.1, 2.2, 3.1, 3.2, 3.3, 3.4_

1
.kiro/steering/roadmap.md
View file

@ -79,6 +79,7 @@ The protocol **drive** is NOT fully event-driven. Root cause: `peer_ctrl_msg_han
Smaller items that likely don't warrant a full spec; pick up opportunistically or bundle into adjacent spec work.
- [ ] **eliminate src/frontend/Makefile, consolidate Go builds onto meson** — **recorded 2026-08-21, from `dirty-content-fingerprint` spec's task 4.3.** The Makefile's `build-cli`/`build-server` targets exist as a "no meson involved" fast dev-loop path for apiserver/xcecli, but they achieve this by INDEPENDENTLY REIMPLEMENTING the same `commit_id`/`build_time`/`dirty_fingerprint` computation `src/tools/xce-build.sh` already owns (two separate computation sites for the identical values, a pattern `build-time-versioning`'s own research.md already flagged as an accepted-but-real drift risk). That risk materialized concretely: task 4.3's Makefile-side fingerprint computation needed **three adversarial review rounds** to fix a shell multi-stage-pipe exit-status bug (`{ cmd1; cmd2; } >file`'s exit status only reflects the last command, silently swallowing an earlier failure and fabricating output from incomplete data) — a bug class that could not have existed if there were only one computation site. Since XCE's nix devShell already bundles the full C/Go/Python toolchain uniformly (no dependency-avoidance benefit to a separate meson-free path the way there might be in a project where Go contributors don't want the C toolchain installed), the Makefile's marginal value may just be "skip the initial `meson setup` step" — `meson compile -C <builddir> apiserver`/`xcecli` should already give an equivalent single-target incremental rebuild once a build directory exists. **Not started — investigation first**: measure actual dev-loop friction of `meson compile` vs. `make build-cli`/`build-server` (first-configure cost vs. per-rebuild cost) before committing to removing the Makefile; if removed, `build-deployment.md` and any CI/doc references to `make build-cli`/`build-server` need updating too. Promote via `/kiro-discovery` or direct investigation when ready.
- [ ] **job_ctrl direct-mutate path — migrate to commit-then-apply** — **recorded 2026-07-13, investigated 2026-08-19 (finding: real risk, wider scope than originally framed).** `raft-apply-decoupling` deliberately scoped `job_ctrl`'s `mark_job_update`/`mark_job_fail` (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes). The 2026-08-19 investigation found the original framing wrong in two ways: (1) **NOT the only remaining call site** — `on_job_stop`/`on_job_resume`/`on_job_ckpt` (`job.c:2067-2222`) share the identical unconverted mutate-then-notify-then-WAL-append pattern, fed by ordinary worker/scheduler traffic, not "rare, admin-only" as originally described; (2) **blast radius (a) is confirmed real, not theoretical** — the pre-commit mutation is synchronously pushed live to BOTH the scheduler (`sched_job_update_fun`) AND every connected apiserver/client (`apiserver_job_update_fun`) before the commit barrier resolves; on barrier timeout the admin gets a failure reply but `job->job.state` is never rolled back and no retraction is ever sent — matching the bug-795/828/851/1060 shape. Live-apply vs replay-apply (b) is narrower than feared for `job_ctrl` specifically (the `allocs != NULL` gate mutually excludes it from the converted deferred-apply fact types), but the investigation surfaced a SEPARATE bug underpinning that safety argument: `job_ctrl` (`job.c:999`) holds only a read lock while mutating job state — every other mutator in this file takes a write lock — so two concurrent admin commands on the same job aren't mutually excluded, undermining the "same lock, no staleness window" claim the replay-safety argument depends on (already flagged by a `raft-apply-decoupling` task 2.1 reviewer, never fixed). **Recommendation**: promote to a small follow-up spec covering all four call sites together (bounded, well-precedented fix — same shape as the fact-receipt conversion `raft-apply-decoupling` already built for job submit); the lock-type bug is independently fixable and low-risk regardless of when/whether the larger migration happens. Not yet promoted — user decision 2026-08-19 to correct scope/framing here first, promote later.
- [x] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — **IMPLEMENTED** as part of `raft-drive-async` task 3.2 (`raft-drive-async` is IMPLEMENTED, validate-impl GO 2026-07-18; see above). `trim_wal_segments` (`persist.c` ~2597-2644) now computes `floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s)` — folding in active-standby `acked_lsn` and disconnected-slot `last_disconnected_lsn` — and only unlinks segments below that floor; the code comment at `persist.c:2634` explicitly labels this the "(bug-1147 guard)". Far-behind/empty join (cursor=0) is additionally covered now that `entity-bearing snapshots` (strategy B) has landed as `ha-entity-snapshots` (see above), with a live decision path (`ha.c:4413`, "< oldest_wal_lsn → requesting snapshot transfer"). Related: bug-795, bug-828, bug-1060.
- [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_