Commit graph

1580 commits

Author SHA1 Message Date
56e007cb6b
feat(scheduler-sim-step-verbosity): task 1.2 - implement preamble deduplication
repl_print_event_summary() (repl_break.c) now compares the candidate
rendered timestamp and displayed session number against a new cache
on xce_sched_repl_state_s (repl_internal.h: last_echo_ts_human[32],
last_echo_session) before printing the "[sim] t=... session=..."
preamble. It prints and updates the cache only when either value has
changed, or no preamble has been printed yet this session (the cache's
zero-initialized default, which no real echoed event can produce);
otherwise the preamble is skipped and the cache is left untouched. The
event's own content line prints unconditionally either way
(Requirements 1.1-1.4, 2.2).

All 6 RED-phase tests (task 1.1) now pass, including the critical
raw-vs-rendered nuance test (Requirement 1.3). No regressions in the
Event Echo Gate suite or the sibling spec's breakpoints-inspection
suite (test_step_echoes_events_before_session_decisions specifically
confirmed still passing under the new dedup behavior).

repl_break.c grew from 949 to 980 lines (+31, 3.3% over the residual
already accepted in scheduler-sim-repl-verbosity task 9.2). Review
judged this small growth as accepted without a fresh empirical split
test: task 9.2's own split experiment (a larger +44-line growth)
already found extracting this file's lookup-table domain reaches only
838 lines (still over cap) while requiring 22 white-box test files'
#include lines to change - disproportionate churn this smaller
increment would only re-confirm.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-03 04:11:08 +00:00
6a2aa47f5e
test(scheduler-sim-step-verbosity): task 1.1 - RED tests for preamble dedup
Adds 6 failing tests for the Event Echo Preamble Deduplicator
(Requirements 1.1-1.4, 2.2): a second echoed event sharing its
predecessor's rendered timestamp+session omits its preamble; two
events across a session boundary both print their own preamble; the
critical raw-vs-rendered nuance (Requirement 1.3) - two events whose
raw nanosecond timestamps genuinely differ but whose rendered,
second-granularity timestamp and session are identical still dedupe,
with a fixture-premise self-check proving the two raw values really
do render identically; the first echoed event in a fresh session
always prints its preamble; a structural event under continue mode
after suppressed routine events follows the same rule; event content
and ordering are unaffected regardless of the preamble decision.

Lands in a new test_repl_step_verbosity.c (1010 lines, 6 tests) rather
than extending test_repl_event_echo.c (705 lines) - proactive split
per this file family's established practice, since appending would
have pushed that file well past the project's ~1000-line trigger.
Registered in testsuite/meson.build. No production code touched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-03 03:54:01 +00:00
1374540954
docs(scheduler-sim-step-verbosity): add approved spec (requirements/design/tasks)
Follow-on to scheduler-sim-repl-verbosity: collapses the REPL's
per-event "[sim] t=... session=..." preamble to print once per
distinct rendered (timestamp, session) run of consecutive echoed
events, instead of once per event, in both step mode and continue
mode's structural-event echo. Requirement 1.3 deliberately keys the
dedup on the rendered, second-granularity timestamp rather than the
raw nanosecond clock value, which ticks on every event and would
otherwise never repeat -- confirmed against a real user-supplied
transcript before writing requirements (see research.md). Requirement
2 preserves event selection, ordering, content, every other REPL
command, and all scheduling decisions/state exactly as
scheduler-sim-repl-verbosity's Requirement 5 already established.

Generated via kiro-spec-quick (single-pass, no approval gates) with
an inline requirements review gate, an inline design review gate, and
an independent task-graph sanity review (one NEEDS_FIXES round,
repaired: added file-size watch bullets to tasks 1.1/1.2 per
design.md's own housekeeping directives).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-03 03:53:24 +00:00
a51790dcbb
docs(scheduler-sim-repl-verbosity): fix two documentation gaps from final validation
Found by /kiro-validate-impl's cross-task requirements-coverage and
design-alignment checks, both documentation-only (no code changes):

1. AC 6.5 (whole-number/no-trailing-.00 formatting) is implemented as
   scoped to non-consumable (Load Metrics) values only - a consumable
   resource's value=/used= pair deliberately keeps its unconditional
   %.2f formatting to preserve bug-1601/1613/1727's already-pinned
   display shape. This carve-out existed only as an inline code
   comment (repl_info.c) and a test-file header comment, never
   sanctioned in requirements.md/design.md itself. Amended AC 6.5's
   text to state the scope explicitly and added a design.md Correction
   note cross-referencing it.

2. design.md's File Structure Plan never documented task 6.1's proactive
   split of the prompt-content tests into a new test_repl_prompt_context.c
   (every other analogous split in this spec got a Correction note;
   this one didn't), and task 9.1's later Correction note repeated the
   stale claim that test_repl_execution_control.c "already carries" the
   prompt-content tests - it doesn't, they moved to the new file at task
   6.1. Added the missing task 6.1 Correction note and fixed both stale
   references.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 21:55:50 +00:00
e87c0b3fc9
docs(scheduler-sim-repl-verbosity): task 10.2 - final regression and scale verification
Confirms the full scheduler test suite passes with no regressions
(62/63; sole failure is the pre-existing, unrelated bug-1758) and
that the feature holds at realistic scale. The literal named real
capture (scheduler-replay-20260820-070103.replay, bug-1611) no longer
exists in this environment, so verification used a synthetic 108-round
proxy fixture built at comparable shape (a 30-job batch, a 65-round
stable-PEND stretch matching design.md's own hand-analyzed fingerprint,
a RUN transition, and routine ticks) - disclosed explicitly as a
substitution, not silently worked around.

Continuous execution against the proxy collapsed 108 rounds to 15
output lines (~140x compression) versus 2,061 lines for the same
fixture in batch mode, with every genuine state change (job arrivals,
pend-set transitions, structural events) still surfacing individually
and the still-open range flushed before the replay-complete banner.
A step/info worker/help/break pass confirmed the new prompt content,
grouped resource table, reorganized help listing, and break's enriched
error, all live over a real pty. Decision-sequence extraction from a
batch run and a full step-mode interactive run of the identical
fixture matched byte-for-byte (2,060 lines each), confirming this
feature only changes display, never scheduling decisions.

This is the final task of the scheduler-sim-repl-verbosity spec; all
implementation tasks (1.1-10.2) are now complete.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 21:43:18 +00:00
5ab129cde1
docs(scheduler-sim-repl-verbosity): task 10.1 - validate no out-of-scope REPL regressions
Verified (via a fresh full regression build/test pass, a diff of this
spec's 19 commits against every pre-existing REPL test file, a direct
comparison of repl_dispatch()'s command set at the pre-feature baseline
vs HEAD, and a two-worktree step-mode transcript diff against a
synthetic multi-round replay) that: exactly the 4 already-documented
pre-existing test-assertion changes exist (task 4.2's 3, task 8.2's 1),
no top-level command was added/removed/reordered, and step mode's
per-event/per-decision output is byte-for-byte unchanged outside the
new prompt content (Requirement 5.5) - consistent with the Event Echo
Gate and Round Rollup being gated to continue mode only. The sole test
failure in the full suite (test_job_rusage_reconciles_mem_reservation_against_usage)
is confirmed pre-existing and unrelated (bug-1758, scheduler-dispatch-commit
spec). Read-only validation task; no production or test code changed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 21:03:07 +00:00
bd97397373
feat(scheduler-sim-repl-verbosity): task 9.2 - discoverable help and break event types
Adds repl_bp_kind_names(char* buf, size_t bufsz) (repl_break.c),
comma-joining BP_KIND_TABLE's tokens in table order into a caller
buffer. repl_print_help() (repl.c) is rewritten into four labeled
sections (Execution Control, Breakpoints, State Inspection, Snapshot &
Parameters) with a single shared "  %-38s%s\n" column format, info
config moved into State Inspection next to its siblings, set units
added to Snapshot & Parameters, and the break line's description now
built from repl_bp_kind_names() so it enumerates the complete event
type set. repl_cmd_break()'s unrecognized-event-type error calls the
same accessor, so help and the error message can never disagree with
each other or with BP_KIND_TABLE (Requirements 7.1-7.6). No command's
existence, argument parsing, or other output changed.

repl_break.c grew from 905 to 949 lines (18.6% over the 800-line cap,
up from task 3.2's already-accepted 13%-over residual). Review
empirically tested extracting the BP_KIND_TABLE lookup-table domain
into a new repl_bp_kind.c: it built and passed, but only reached 838
lines (still 4.75% over cap) while requiring 22 white-box test files'
#include lines to be updated - the same disproportionate-churn,
doesn't-achieve-compliance outcome already accepted at tasks 6.2/8.2.
Reverted; 949 lines accepted in place.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 20:30:45 +00:00
bd77eea0a8
test(scheduler-sim-repl-verbosity): task 9.1 - RED tests for help/break discoverability
Adds failing tests for Requirement 7 (Discoverable Help and Breakpoint
Event Types): help's four labeled sections in correct order, one
consistent column width across every listed command, info config's
placement alongside its info siblings, set units's placement alongside
set speed/output/verify, the break line's complete BP_KIND_TABLE-derived
event-type listing, and break's own unrecognized-event-type error
listing the same complete set. Each new test hardcodes BP_KIND_TABLE's
current 12 tokens rather than referencing the not-yet-implemented
repl_bp_kind_names() accessor, so RED comes from genuine assertion
failures against today's behavior rather than a link error.

New tests for Requirements 7.1-7.4 land in a new test_repl_help.c
rather than test_repl_execution_control.c (949 lines going in, within
reach of this spec's own ~1000-line split trigger already flagged at
task 2.1) - proactive split per this spec's established practice,
documented in design.md. break's Requirement 7.5 error-message test
lands in test_repl_breakpoints_inspection.c exactly as originally
planned, alongside the pre-existing test_break_unrecognized_event_type_rejected
(left unmodified).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 20:01:06 +00:00
9db99964f0
feat(scheduler-sim-repl-verbosity): task 8.2 - implement resource table rendering
Rewrites info worker's/info resource's resource-listing loops from a
flat, unconditional fprintf into a 3-pass, section-grouped table
(Consumable/Capacity, Load Metrics, Static Hardware Attributes),
units_mode-aware value rendering (raw/human/both, including swp's
MB->bytes conversion), whole-number formatting with no trailing .00
outside the consumable value=/used= pair, and jobid-range-compression
adoption for info worker's assigned-job listing (Requirements 6.4-6.6,
4.5). All 10 of task 8.1's tests pass. Preserves every named prior-bug
invariant (bug-1727's avail_v-as-total, bug-1613's consumable/
non-consumable split, bug-1593's GLOBAL/SHARED filter, bug-1688's
.caps-bound iteration).

Fixes one pre-existing test (bug-1688, predating this spec) whose
"value=5.00" assertion for a whole-number non-consumable resource
directly conflicts with Requirement 6.5's intentional no-trailing-.00
change - same class of fix as task 4.2's pre-existing-test updates.

repl_info.c grew from 985 to 1267 lines (58% over the 800-line cap,
worse than any other file this spec touched) - but unlike repl.c/
repl_break.c, it was already 23% over cap before this spec began.
Review empirically confirmed a narrow helper-only extraction doesn't
achieve compliance (still 41% over), and a compliance-achieving split
would mean restructuring ~690 lines of pre-existing display logic
this feature doesn't otherwise touch - a larger scheduler-core-level
decision, accepted as a residual rather than folded into this task.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 19:32:29 +00:00
e48e222bd1
feat(scheduler-sim-repl-verbosity): task 8.1 - RED-phase resource table tests
Adds 10 tests in a new test_repl_resource_table.c covering info
worker's/info resource's resource-listing table/grouping layout,
units_mode-aware rendering (raw/human/both, including swp's MB->bytes
conversion and a no-mutation check on the underlying stored value),
whole-number-vs-fractional formatting, and info worker's assigned-job
listing adopting the real repl_fmt_jobid_range() (Requirements
6.4-6.6, 4.5). All 10 fail as expected against today's flat,
unconditional per-resource fprintf loop and uncompressed listing.

Found and fixed a genuine ASAN stack-use-after-return in the test's
own fixture code (not production): WORKERS_PUSH's handler moves a raw
pointer into a job's resvals rather than deep-copying, so the fixture
must heap-allocate rather than stack-allocate the structures it feeds
through that path - independently reproduced by review.

New file rather than extending test_repl_breakpoints_inspection.c:
landing this content there first pushed it to 3215 lines, 3.2x this
file family's own ~1000-line split trigger already used to justify
splitting test_repl_round_rollup.c/test_repl_event_echo.c for
structurally identical reasons.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 18:47:14 +00:00
5ee1d6cb08
feat(scheduler-sim-repl-verbosity): task 7.2 - implement resource formatting helpers
Implements repl_is_bytes_valued_resource()/repl_format_bytes_human()
(mirroring xce_test's confirmed is_bytes_valued_resource()/
format_bytes_human() precedent exactly) and the units_mode setting
(explicitly defaulted to both, not left to zero-init) wired into
set units raw|human|both alongside the existing set speed/output/verify
branches (Requirements 6.1-6.3). All 8 of task 7.1's tests pass.

Split repl_format_ts_human(), the Jobid Range Compressor (task 1.2),
and these two new helpers out of repl.c into a new repl_fmt.c: adding
them in place pushed repl.c to 929 lines (16.1% over the 800-line
cap), and unlike task 6.2's single-function split attempt (which
didn't even achieve compliance), bundling these three pure-formatter
groups together does - repl.c settles at 587 lines, repl_fmt.c at 402,
both comfortably under cap. Required a matching #include line in the
20 test files that white-box #include repl.c, plus extending
test_sim_isolation.c's forbidden-include/symbol scan to cover the new
file, following the repl_rollup.c split precedent from task 2.2.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 17:45:06 +00:00
11fe6194ce
feat(scheduler-sim-repl-verbosity): task 7.1 - RED-phase resource formatting tests
Adds 5 pure-function tests in a new test_repl_resource_formatting.c
for repl_is_bytes_valued_resource()/repl_format_bytes_human()
(classification, adaptive B/KB/MB/GB/TB/PB scaling including exact
boundary values, swp's MB->bytes conversion), and 3 tests in
test_repl_snapshot_params.c for set units raw|human|both parsing
(Requirements 6.1-6.3). All fail as expected: link failure for the
not-yet-existing helpers, compile failure for the not-yet-existing
units_mode field/enum.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 16:45:59 +00:00
da6ba96eb4
feat(scheduler-sim-repl-verbosity): task 6.2 - implement dynamic REPL prompt
Adds repl_build_prompt(), building the readline() prompt fresh each
loop iteration: the plain "(sim) " before any session begins, else
"(sim session=<N> t=<ISO8601-ts>) " using the bare session count (not
+1, matching info session's convention) and the existing
repl_format_ts_human(). Both of task 6.1's tests pass.

repl.c grew 808->852 lines (6.5% over the 800-line cap). Review
empirically tried extracting repl_build_prompt() into a new
repl_prompt.c: it works, but requires updating 19 sibling white-box
test files' #include lists, and even then repl.c only drops to 814
lines - still over cap. Given the function is a small 15-line
formatter and the split doesn't even achieve compliance, accepted in
place rather than forcing disproportionate churn.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 16:14:52 +00:00
99fa45c41a
feat(scheduler-sim-repl-verbosity): task 6.1 - RED-phase REPL prompt tests
Adds 2 tests in a new test_repl_prompt_context.c covering the dynamic
prompt's plain-before-first-session baseline and its session/timestamp
content in both step and continue modes (Requirement 5.5).

Required genuinely new test infrastructure: libedit's readline() only
writes its prompt text when both stdin and stdout are real ptys, which
no existing REPL test in this codebase's freopen()-from-file harness
can observe. Built a PTY-based capture technique (openpty(), ECHO
disabled via termios) instead - independently reproduced via a
standalone probe and stress-tested 15x with zero flakiness during
review.

New file rather than extending test_repl_execution_control.c (already
at 948 lines, over the 800-line cap), per this spec's established
split precedent.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 15:49:41 +00:00
d9109b7c4b
feat(scheduler-sim-repl-verbosity): task 4.2 - retire redundant per-decision echo
Gates sim_print_decision() so the bare per-decision line no longer
prints under continue mode, making the Round Rollup's summary the
sole source of decision output (Requirement 2.3, closing bug-1610).
step mode and non-interactive/headless runs are completely
unaffected (Requirements 5.1, 5.4). Implemented as a file-local
inline wrapper in sim_loop.c rather than repl.h, since the gated
function is static to that file and repl.h can't see repl_state's
internals without leaking them to every other repl.h consumer.

Fixes 3 pre-existing tests (predating this spec, from scheduler-core)
that pinned the now-intentionally-changed bare-line behavior under
continue mode - each fixed differently based on its actual purpose
rather than uniformly: two kept continue mode with their anchor
switched to the Round Rollup's summary (since they specifically test
continue-mode breakpoint interruption, which step mode would
trivialize), one switched to step mode (since its actual invariant -
print-before-breakpoint-check call order - is mode-independent and
continue mode no longer has any observable anchor for it).

Review caught a second, unguarded sim_print_decision() call site in
sim_reconcile_unaccounted_jobs() (added by an earlier, separately-
merged spec) that design.md's boundary never accounted for - per the
review, this is actually the dominant real-world path for DISPATCH
decisions in replay mode, so leaving it unguarded would have defeated
Requirement 2.3's "sole source of decision output" claim for the
common case. Gated it through the same wrapper; added 2 new tests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 15:10:01 +00:00
3d08a43a35
feat(scheduler-sim-repl-verbosity): task 4.1 - RED-phase sim_print_decision Gate tests
Adds 3 tests in a new test_sim_print_decision_gate.c covering the old
unconditional per-decision echo (sim_loop.c's sim_print_decision())
no longer duplicating the Round Rollup's per-round summary under
`continue` mode (Requirement 2.3), while `step` mode and headless
(non-interactive) batch runs stay completely unaffected (Requirements
5.1, 5.4). This is exactly the duplication bug-1610 reported.

1 of 3 tests fails as expected RED evidence (continue-mode bare
"[sim] PEND jobid=..." line still appears alongside the Round Rollup's
own summary today); the other 2 (step-mode and headless regression
checks) correctly pass since they pin already-correct, unaffected
behavior.

New file rather than extending either sibling: test_repl_round_rollup.c
is already 1022 lines (222 over the 800-line cap), and
test_repl_event_echo.c (704 lines) deliberately keeps its fixtures
job-free (bug-1595 discipline), which this task's decision-producing
fixtures cannot follow.

No production code touched (sim_loop.c/repl.c/repl_rollup.c/repl_break.c
untouched) - tests only, per this task's RED-phase boundary. The actual
gate implementation is task 4.2.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 13:12:04 +00:00
cc78d09140
feat(scheduler-sim-repl-verbosity): task 3.2 - implement Event Echo Gate
Implements mode-aware event echo classification, job-arrival range
compression (Req 4.5), and round-collapse flush wiring at all 3
breakpoint-match/structural-event call sites (Requirements 1.1-1.4,
3.5, 3.6). All 4 of task 3.1's tests pass; also closes bug-1609
("continue is as verbose as step").

repl_break.c grew 827->905 lines (13% over the 800-line cap). Unlike
task 2.2's Round Rollup, this logic is wiring inside pre-existing
driver-facing hooks, not a relocatable subsystem - accepted in place
per design.md's established residual-tolerance precedent.

Review independently investigated a one-off ASAN stack-use-after-return
in an unrelated test (sched_sim_record_output_test) surfaced by this
change's stack-layout perturbation: traced to the same documented
pre-existing thread-lifecycle bug class as bug-1598/bug-1632 (an
unjoined picker thread reading a stack-local sim after test-function
return), confirmed via 11/11 clean reruns - logged as bug-1763, not a
regression from this task.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 13:00:09 +00:00
f587a441ac
feat(scheduler-sim-repl-verbosity): task 3.1 - RED-phase Event Echo Gate tests
Adds 4 tests in a new test_repl_event_echo.c covering mode-aware event
echo suppression and ending an active collapsed range on structural
events/breakpoint hits (Requirements 1.1-1.4, 3.5, 3.6). Job-free
fixtures (QUEUE_UPDATE/WORKERS_PUSH only) sidestep bug-1595's
jobid-reuse hazard entirely rather than merely avoiding it. 3 of 4
tests fail as expected RED evidence; the 4th (step-mode regression
check) correctly passes since it exercises unchanged behavior.

New file rather than extending test_repl_execution_control.c (already
over the 800-line cap at 937 lines) - applying the same file-size
lesson from tasks 2.1/2.2 proactively, decided before implementation
rather than after a review round.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 12:34:45 +00:00
921c72bb40
feat(scheduler-sim-repl-verbosity): task 2.2 - implement Round Rollup logic
Implements the round-boundary decision summary, collapsing of repetitive
rounds, and the shared repl_round_collapse_flush() entry point
(Requirements 2.1, 2.2, 2.4, 3.1-3.4, 3.6). Fingerprint composition
matches design.md exactly: dispatch/preempt counts from the existing
per-round tally, pend-jobid set state-derived from core->jobs (not
decision events), so Requirement 2.4 holds independent of bug-1611.
All 7 of task 2.1's RED-phase tests now pass.

Went through three review rounds:
- Round 1 approved the logic but rejected repl.c's size (1160 lines,
  45% over the 800-line cap). Remediated by splitting the Round Rollup
  functions into a new repl_rollup.c, following this codebase's own
  established repl.c/repl_break.c/repl_info.c/repl_params.c domain-split
  precedent.
- Round 2 approved the split but found the new file wasn't added to the
  "simulation isolation" regression guard (sched_sim_isolation_lib +
  test_sim_isolation.c's forbidden-include/symbol scans), plus two stale
  repl.c-location doc comments left over from the move. Both fixed.
- Round 3 found repl.c still 17 lines over cap post-split (808 after
  trimming, from 817) and rejected on that basis. Verified empirically
  that closing the remaining gap isn't achievable via comment trimming
  alone (even deleting both comments entirely only reaches 801) without
  another disproportionate file split for single-digit lines. Accepted
  808 lines (1% over) as the final state, consistent with round 1's own
  explicit judgment that a small residual is not a blocker once the
  original 45%-over violation is resolved - documented in design.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 12:08:31 +00:00
0c6ccb0324
feat(scheduler-sim-repl-verbosity): task 2.1 - RED-phase Round Rollup tests
Adds 7 tests in a new test_repl_round_rollup.c covering per-round
decision summaries, collapsing of repetitive rounds, and the shared
repl_round_collapse_flush() entry point (Requirements 2.1-2.4, 3.1-3.4,
3.6). repl_round_collapse_flush() itself is not yet implemented (task
2.2) - build fails at link time, confirming RED.

Split into its own file rather than the originally-planned
test_repl_execution_control.c: that file was already at the project's
800-line cap before this task, and landing these tests there would
have pushed it to 1563 lines. design.md's File Structure Plan is
corrected accordingly (mirrors the existing test_repl_snapshot_params.c
/test_repl_breakpoints_inspection.c split in this same file family).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 09:40:46 +00:00
e6c670194f
docs(scheduler-sim-repl-verbosity): expand spec with Requirements 5.5, 6, 7
Extends the original continue-mode-noise-reduction spec (Requirements
1-5, 4 tasks) with three more requirements surfaced during a cli-ux
review of the live REPL:

- Req 5.5: REPL prompt shows current session/timestamp (step+continue)
- Req 6: human-readable byte/megabyte resource value formatting, table
  layout for info worker/info resource, and a new `set units` option
- Req 7: help listing reorganized into labeled sections, and break's
  valid event types made discoverable in both help and its error text

design.md gains three matching components (REPL Prompt Context,
Resource Value Formatter, Help & Breakpoint Discoverability), each
grounded in direct source verification (node_information.c's mem/tmp/
swp unit docstrings, test.c's is_bytes_valued_resource()/
format_bytes_human() precedent, BP_KIND_TABLE's actual contents).
Went through a full self-review pass fixing 7 internal-consistency
issues (stale System Flows section, a factual error in the claimed
timestamp format, a missing Requirements Traceability row, an
enum-zero-init assumption, a session_count off-by-one trap, and
incomplete Domain Model prose) before design/tasks approval.

tasks.md's own expansion (task groups 6-10) is committed alongside
each task's implementation, not here.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 08:40:43 +00:00
140f0900b5
fix(scheduler): repl_cmd_delete() rejects non-numeric/invalid breakpoint ids
delete <id> used atoi(), which silently coerces a non-numeric or
trailing-garbage token to 0 instead of reporting an error, e.g.
"delete abc" printed "no breakpoint with id 0" rather than rejecting
the input. Switched to strtol()+endptr validation, mirroring the same
"fail loud" discipline repl_cmd_step() already established in this
file family. Added test_delete_non_numeric_id_rejected covering
non-numeric, zero, negative, and trailing-garbage ids.

Logged as bug-1761 (buglog.json entry already committed alongside an
unrelated task's changes earlier in this branch's history).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 08:40:22 +00:00
b639fae162
feat(scheduler-sim-repl-verbosity): task 1.2 - implement job identifier range compression
Implements repl_fmt_jobid_range() in repl.c (per design.md's "no new
production .c file" commitment), grouping job-array identifiers by base
and compressing contiguous index runs into [lo-hi] ranges. All 5 of task
1.1's RED-phase tests now pass.

Went through one review round: an initial GREEN implementation placed
the function in a new file (repl_jobid_range.c) to sidestep a linker
issue in the lean test target from task 1.1. Review found and verified
a cleaner fix: link the test against the full scheduler_srcs object list
(mirroring the existing sched_scheduler_connect_test precedent), which
resolves the same symbols without a new file. Remediated accordingly;
bug-1762 in buglog.json documents the investigation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 08:39:52 +00:00
f165e3a751
feat(scheduler-sim-repl-verbosity): task 1.1 - RED-phase tests for job identifier range compression
Adds failing unit tests for repl_fmt_jobid_range() covering contiguous
ranges, non-contiguous sub-ranges, multiple array bases, mixed array/
non-array ids, and a too-small output buffer (Requirement 4, AC 4.1-4.5).
The function itself is not yet implemented (task 1.2) - build fails at
link time with undefined reference to repl_fmt_jobid_range, confirming
the RED phase.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
2026-09-02 07:31:31 +00:00
00fc0a43e0
test(xce-test-ha): add worker-rejoin-resync coverage to sequential/chaotic soak modes
Starts a real worker (auto-detects xcecd_virtual vs. real xcecd, mirroring
xce-deploy-ha's own detection) + xce_scheduler alongside the controller
cluster, dispatches a genuine job, kills the primary while it's RUN, and
verifies the worker rejoins the new primary and the job recovers to DONE
via real reconciliation -- closing the "no real daemon-to-daemon smoke
test" gap flagged by worker-rejoin-resync's own feature validation.
Live-verified end-to-end multiple times (sequential and chaotic, 2 rounds
each) against a real 3-node cluster.

Also logs two bugs discovered along the way (not fixed, out of scope for
this test-infra change): bug-1759 (scheduler sim-mode dispatch/PEND
inconsistency) and bug-1760 (scheduler has no periodic post-failover
reconnect check, only a reactive one gated on its own next successful
publish -- a chicken-and-egg deadlock, live-confirmed via 60s+ of stuck
PEND with zero scheduler log activity). bug-1760 is worked around in this
harness only by restarting the scheduler process after each new-primary
election; the underlying gap is unfixed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019ypUeKgbAqLWUkaSJ79poN
2026-09-02 04:31:07 +00:00
ea00879af0
update README.md to the newest build guide 2026-09-02 11:06:19 +08:00
b6a2f5ec14
fix(worker-rejoin-resync): task 4.4 - gate periodic load-indices send on resync_pending
Extends task 4.3's post-join gating to resource_sync_thread()'s
independent periodic reporting pthread, closing the Requirement 3.8 gap
where load-indices sends could interleave with an in-flight job-state
reconciliation report. Also fixes two pre-existing latent bugs
(bug-1756, bug-1757) in resource_sync.c that blocked a truthful RED/GREEN
signal for the new tests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019ypUeKgbAqLWUkaSJ79poN
2026-09-02 03:01:56 +00:00
4140ecdbae
docs(worker-rejoin-resync): add task 4.4 - close feature-level Req 3.8 gap
kiro-validate-impl's requirements-coverage check found AC 3.8 only
partially satisfied: resource_sync.c's independent periodic thread
(load-indices RPCs, every 15s) is never gated by resync_pending, so
it can fire mid-reconciliation - task 4.3's gate only covered
checking_connection()'s rejoin_call(). NO-GO with this as the sole
concrete remediation item; adding task 4.4 to close it before
re-validating.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-02 01:42:30 +00:00
a1dc239623
docs(worker-rejoin-resync): check off completed group headers in tasks.md
Cosmetic only - tasks 4, 6, 7 are grouping headers whose sub-tasks
were all already complete; marking them [x] for consistency with
groups 1-3.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-02 01:24:19 +00:00
1c4cf8bd3b
fix(worker-rejoin-resync): task 7.5 - bound the outbound reconciliation query
Closes bug-1755: worker_send_resync_query() hardcoded pktlimit=0 in
its xce_job_state_query_req() call, so the controller's outbound
OPCode_jobStateQuery always sent every non-terminal job ID in one
unbounded page regardless of count - the query direction never
actually exercised the pagination mechanism task 1.4/7.4 proved
correct, unlike the report direction's real, wired
XCECD_RESYNC_REPORT_PKTLIMIT bound.

Add XCE_RESYNC_QUERY_PKTLIMIT=100, mirroring that exact precedent
(same value - a JobStateQuery entry is a bare jobid, smaller than a
JobStateReport entry), and thread it into the one call site. Existing
small-job-count behavior is unchanged; a rejoin exceeding the cap now
genuinely produces multiple wire pages, proven against the real
production call site (not a bypass) with a 250-job/3-page test.

This is the last task in worker-rejoin-resync's implementation plan.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-02 01:23:14 +00:00
613b80e991
test(worker-rejoin-resync): task 7.4 - large job-count pagination correctness
Add test_large_job_count_pagination_correctness, proving both
OPCode_jobStateQuery and OPCode_jobStateReport genuinely span many
wire pages (1042 jobs, 11 pages each direction, including a partial
final page) with zero jobs dropped/duplicated/truncated - exact-count
assertions throughout (page continuity, no-duplicate bitmap, exact WAL
delta, exact hook-fire count), extending task 6.2's 2-page proof by an
order of magnitude. Test-only, no production changes.

Logs bug-1755: the underlying query-pagination mechanism this test
proves correct is never exercised in production - worker_send_resync_
query() hardcodes pktlimit=0, so the controller's real query send
always produces one unbounded page regardless of job count, asymmetric
with the report direction's real XCECD_RESYNC_REPORT_PKTLIMIT bound.
No hard transport-layer size cap exists, so this isn't a demonstrated
failure mode, but it contradicts design.md's explicit "both paginated"
claim in this feature's own headline mechanism. Adds task 7.5 to close
it with a small, well-scoped fix before the feature-level gate, rather
than leaving a silently-false design claim in the completed feature.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-02 00:48:15 +00:00
165fc1586a
test(worker-rejoin-resync): task 7.3 - burst recovery unbounded by count
Add test_burst_recovery_unbounded_by_job_count, reproducing bug-1741's
exact original shape (7 jobs on one host, 6 lost finish reports) in
one reconciliation round and one wire page: 5 finished-success + 1
finished-failure recover correctly and distinctly, while an already-
DONE job re-reported as an unqueried extra (with a deliberately wrong
finish time) is proven structurally unchanged - exact +6 WAL LSN
delta, not >=, so a duplicate append is genuinely caught rather than
silently tolerated. Test-only, no production changes; deliberately
kept to a single page since page-boundary correctness is task 7.4's
separate concern.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-02 00:04:37 +00:00
bceca0c990
test(worker-rejoin-resync): task 7.2 - failover-during-job recovery
Add test_failover_during_job_recovers_lost_finish, a focused proof of
the feature's own original motivating scenario (bug-1741): a job
dispatched and RUN with no finish fact ever committed anywhere
(modeling a lost fire-and-forget finish notify), recovered to DONE
with the worker-reported finish time - not stuck in RUN - via the
real rejoin/query/report/apply chain. Test-only, no production
changes; reuses task 7.1's helper infrastructure rather than
duplicating its broader four-outcome coverage.

Updates the existing bug-924 (/tmp exhaustion / flaky teardown) entry
with a 5th occurrence hit during validation, unrelated to this task's
own change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 23:19:15 +00:00
8650cfd9ba
test(worker-rejoin-resync): task 7.1 - full rejoin round trip
Add test_full_rejoin_reconciliation_round_trip proving the whole
chain composes end-to-end in one round, for a mix of still-running,
finished-success, finished-failure, and unrecognized jobs: real
rejoin_worker() -> worker_send_resync_query() (real wire query) ->
hand-built worker answer through the real report codec ->
resync_handle_report_page() -> resync_apply_report(), asserting
correct final state and scheduler notification for each. Test-only,
no production changes.

Logs bug-1754: a worker-reported job failure's exit code is never
durably recorded anywhere in this codebase (distinct from bug-1747's
finish-time gap) - discovered honestly asserted by the new test,
out of scope for this validation-only task.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 21:47:22 +00:00
9f655c61c7
feat(worker-rejoin-resync): task 6.2 - route job-state report to reconciliation
Add OPCode_jobStateReport dispatch in myreq_handler, reassembling a
report across multiple incoming pages before handing it to task 3.2's
resync_apply_report() exactly once - proven via a test-only call
counter, not just end-state equivalence (a double-apply could pass an
end-state check if idempotent).

Standby-redirect investigation (Req 8.1/8.3): confirmed
ha_dispatch_gate already redirects every rejected opcode uniformly, so
no opcode-specific code was needed for that. OPCode_jobStateQuery has
no controller-side receive path at all (only sent, never received)
so needed no gate change either.

That same investigation found a different, more consequential gap:
fixes bug-1753, ha_dispatch_gate's quorum-loss fact-admission allowlist
(OPCode_notify/update/setpreason/clrpreason) was missing
OPCode_jobStateReport, so a report arriving during a transient
quorum-loss window - this feature's own core failover scenario - would
be rejected outright with no retry (worker sends fire-and-forget).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 21:06:42 +00:00
a3d12d37d9
feat(worker-rejoin-resync): task 6.1 - trigger reconciliation on rejoin
Arm a reconciliation round in apply_join_state() the moment a worker
host becomes connected (first join, reconnect, or failover-reconnect -
all routed through the same shared call site, Req 1.1-1.3), then send
the query from on_worker_join() strictly after the join reply and
existing config push (Req 2.1-2.2), verified via a structural test
that reads real wire order off the endpoint queue.

Also closes Req 3.7's config-delivery seam end-to-end: worker_resync_
window_ms now rides the existing Conf4Worker push (clustername
precedent, no new opcode) and the worker applies it to its retention
window on receipt.

Fixes bug-1752: myreq_handler had no OPCode_push dispatch case at all,
so every config push - including the pre-existing clustername field -
was silently discarded. Confirmed harmless until now (clustername was
never read anywhere in src/worker/); closed narrowly for
worker_resync_window_ms only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 19:43:18 +00:00
48c6cfd58a
docs: log JSON-reformat gotcha for buglog.json edits in cerebrum.md
Recorded as a Do-Not-Repeat: editing buglog.json via json.load/dump
without ensure_ascii=False silently reformats every existing entry's
unicode punctuation into \uXXXX escapes, ballooning the diff. Caught
and fixed forward twice already this session (once by me, once by a
dispatched task-5 implementer) - documenting so future task subagents
that touch buglog.json don't repeat it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 18:40:13 +00:00
f4c91babba
chore: restore literal unicode in buglog.json (undo escape reformat)
Task 5's own bug-1751 append re-serialized the whole file with
ensure_ascii=True, turning every existing em-dash/arrow into a \uXXXX
escape across 421 unrelated entries. No content changed - restoring
the literal characters to keep the file's history diffable.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 18:39:16 +00:00
80d5a06212
feat(worker-rejoin-resync): task 5 - exclude reconciling host from dispatch
Extend the IS_WORKER_UNAVAIL-style candidate-host filter to also skip
a host with XCE_WORKER_RECONCILING set (Req 5.1), naturally
re-including it once the bit clears with no separate mechanism (Req
5.2). Leaves the scheduler's decision-publish code and its
symbol-guard regression test untouched.

Fixes bug-1751: design.md's cited filter site (scheduler.c's
sched_reset_worker_(), via sched_begin_session()) is dead code on both
live drivers - driver.c/sim_loop.c actually call
xce_core_begin_session() -> core_session.c's core_reset_workers(), an
independently-maintained mirror not listed in design.md's File
Structure Plan. Applied the identical disjunct to both sites so the
task's own observable-completion criterion holds in production, not
just on the literally-cited dead path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 18:38:07 +00:00
5b69973677
feat(worker-rejoin-resync): task 4.3 - wire query receipt and post-join gate
Add OPCode_jobStateQuery dispatch to myreq_handler with multi-page query
reassembly, routing to task 4.2's xcecd_resync_answer_query() (Req
8.1, 8.3). Gate the worker's resumption of normal post-join operation
on a resync_pending flag until the full report has sent (Req 3.8).

Fixes bug-1749 (xcecd_resync.c was never added to the worker's
production source lists - task 4.2's build only ever exercised it via
the test binary's white-box include).

Round 1 review caught bug-1750: the gate sat as checking_connection()'s
entry guard, so a connection drop mid multi-page query left
resync_pending stuck forever, permanently disabling the worker's only
reconnect mechanism (Req 6.1 violation). Round 2 narrows the gate to
guard only the rejoin action, and resets it from xcecd_on_reconnect()
so a stalled query can never wedge it past one reconnect cycle.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 18:01:03 +00:00
cfe58c3dcf
feat(worker-rejoin-resync): task 4.2 - worker answers job-state query with extras
Classify each queried job ID as running/finished-ok/finished-fail/
unrecognized (Req 3.1-3.3, 3.6), report unqueried running/recently-
finished jobs as extras, and send both via task 1.4's paginated wire
helper (Req 8.2, 8.3). Composes correctly with the already-implemented
controller-side consumer (resync_apply_report).

Adds xcecd_finished_job_retention_snapshot() to xcecd_worker_utils, a
small additive companion needed to enumerate all retained jobs for the
extras set (task 4.1 only exposed single-jobid lookup).

Folds the still-open OPCode_jobStateQuery receive-dispatch wiring gap
into task 4.3's scope in tasks.md, since 4.3 already needs the real
receive path to implement its post-join gate.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 16:35:28 +00:00
e31012e8a7
feat(worker-rejoin-resync): task 4.1 - bounded finished-job retention on worker
Retain a job's exit status/code/true finish time for a configurable
window after update_worker_status() detects its exit, before
cleanup_worker_list() would otherwise discard all trace of it
(Requirements 3.4, 3.5, 3.7). Hooked into the single choke point both
production exit-detection paths (libuv IPC callback, dbus polling)
converge through, so retention happens-before any reap regardless of
which mechanism is active.

Also prunes on retain (not just on lookup), closing a review finding:
lookup currently has zero production callers until task 4.2 lands, so
without this the store would grow unboundedly for the daemon's uptime.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 16:03:19 +00:00
087b8c1804
docs(worker-rejoin-resync): fold worker config-delivery gap into task 6.1
Task 4.1 discovered and correctly self-scoped away from a real gap: the
worker daemon has no delivery path at all today for any ClusterConfig
field -- worker_push_config's existing wire push carries a generic
confvars key/value list plus a small number of directly-pulled fields
(clustername is the existing precedent). Confirmed by direct trace
(controller/worker.c's collect_confvars/conf4worker construction, and
grep of worker/src/controller_connector.c's opcode dispatch switch,
which has no clusterconf case). Added workerResyncWindowMs delivery to
task 6.1's scope, mirroring the clustername precedent exactly -- no new
opcode needed, reuses the existing OPCode_push/WorkerConfig struct that
task already touches (on_worker_join). Without this, task 4.1's
retention window (built correctly, as a settable seam with a documented
placeholder default) would never actually receive an operator-tuned
value in production.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 15:47:46 +00:00
eec2596c5d
feat(worker-rejoin-resync): apply reported outcomes idempotently (task 3.2)
Add resync_apply_report()/resync_apply_running_entry()/
resync_apply_finished_entry()/resync_apply_unrecognized_entry() to
resync.c: routes each entry in a worker's job-state report to the
correct existing idempotent path (still-running -> clear from outstanding
set, no fact; finished-success/failure -> record_job_finish_fact_at/
on_job_fail; unrecognized -> record_job_reconcile_pending_fact;
unqueried extra -> record_job_reconcile_assign_fact or no-op), discards
reports for superseded rounds (Req 6.1/6.3), and resolves the round once
its outstanding set empties.

Also fixes bug-1747 (Req 4.2): adds record_job_finish_fact_at(),
additive alongside the unmodified record_job_finish_fact, so a
finished-success reconciliation entry records the WORKER-reported
finish time instead of reconciliation-processing time -- proven via a
real WAL-write + persist_recover() restart-replay test. The FAIL-side
half of the same gap (WAL_JOB_FAIL carries no fail-time wire slot at
all, for any caller) is confirmed pre-existing and systemic, correctly
left out of this task's scope.

Three review rounds surfaced and fixed two real, distinct bugs in the
unqueried-extra-job routing guard (see tasks.md's Implementation Notes
for the full account and the general lesson about re-deriving a full
truth table after a partial boolean-guard fix): round 1 caught the
wrong axis (terminal-vs-non-terminal instead of assignment-status,
Req 4.8); round 2 caught that the first fix's short-circuit clobbered
an already-terminal job's DONE/EXIT back to RUN (Req 4.4), including
confirming the implementer's own test had certified the bug as correct;
round 3 independently re-derived the final three-way guard from scratch
and found no further defect. bug-1748 logged for the guard fix.

Full controller suite: 1239 tests, 0 failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 15:24:29 +00:00
442fcf4544
feat(worker-rejoin-resync): reconciliation round lifecycle (task 3.1)
New src/controller/resync.c/resync.h: resync_arm_round(cluster_t*,
worker_t*) enumerates every non-terminal job assigned to a host,
populates worker_t.resync (round_token, query job-ID set, deadline),
sets XCE_WORKER_RECONCILING, and schedules a non-blocking delaymgr
one-shot timeout. resync_resolve_round() clears the round; a fired
timeout resolves it too, via resync_round_is_current()'s by-value
round-token check -- delay.c/delay.h expose no true cancel/dequeue
primitive, so a stale timer from an already-superseded round becomes a
safe no-op instead of touching a newer round's live state (verified
safe against both misresolving the wrong round and use-after-free: the
timeout closure never holds a pointer into resync_state_t itself).

5 new tests: query-set correctness, fresh-token-per-arm, the
RECONCILING flag's set/clear lifecycle, and a direct (non-blocking)
timeout-fire proof via an XCE_TESTING-gated seam confirmed compiled out
of the production binary. Full suite: 1232 tests, 0 failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 12:53:09 +00:00
ea0658ecf4
feat(worker-rejoin-resync): fail-fast disconnected-worker dispatch (task 2.3)
Split dispatch_a_job_()'s combined "(w->ep == NULL) || (w->host.status
!= 0)" check into two branches, checked in order: w->ep == NULL now
resolves directly via record_job_reconcile_pending_fact() (task 2.1) --
no dispatch_fail_count increment, no mark_job_fail(), never reaching
xce_classify_dispatch_fail() at all -- before the code ever tests
w->host.status != 0, which keeps its existing retry-then-terminal-fail
behavior completely unchanged (byte-identical body, confirmed against
HEAD).

Completes Task 2 (all controller job-table reconciliation outcomes:
2.1 reconcile-to-pending, 2.2 reconcile-assign, 2.3 this split).

RED phase caught a genuine pre-existing test bug: test_worker_notready_
resolves_retryable was seeding a NULL-ep worker while claiming to test
the host.status!=0 retry path -- it never actually exercised that path.
Corrected to use a connected-but-busy worker, with self-asserting
TEST_ASSERT_NOT_NULL(w->ep)/TEST_ASSERT_TRUE(status!=0) guards so this
can't silently drift again.

2 new tests, including a full JOB_TRANSITION_APPLIED path proof (WAL
LSN advance, hook-fires-exactly-once per bug-1679, allocs released per
bug-1631) through this exact call site, both using correctly-timed
canary references on both the job and worker objects to catch any
over-release under ASAN. Full controller suite: 1227 tests, 0 failures,
zero regressions across every other dispatch_a_job_ failure site.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 12:13:26 +00:00
ecfb4c942f
feat(worker-rejoin-resync): reconcile unqueried job assignment (task 2.2)
Add record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*,
xce_job_state_entry_t*) to job.c: eagerly sets job->job.state to match
the worker's reported outcome (mirroring dispatch_a_job_'s inline
mutation, minus allocation fields -- there are none to set, job->job.allocs
stays NULL per Requirement 4.8's documented scope). Calls the existing
persist_submit_job_dispatch_fact(..., alloc=NULL) directly, then leaves
the post-unlock persist_notify_job_updated_already_persisted() call to
the caller, same convention as task 2.1.

Fixes bug-1746 along the way: capnp_encode_joballoc() (persist.c, part
of bug-1678's original fix, predates this feature) unconditionally
dereferenced a NULL alloc -- never triggered before since
dispatch_a_job_ (the only prior caller) always passes non-NULL. Fixed
with a minimal guard producing a true zero-length buffer, matching
apply_committed_entry's alloc_len > 0 decode gate exactly. This
directly contradicted design.md's "tested, backward-compatible on both
seams" claim (true only for decode, never actually exercised for
encode) -- corrected in 4 places.

2 tests: applies-running-no-allocs-and-notifies, and a real restart-
replay round trip (write to real disk, fresh persist_recover(), confirm
jobid/host/state/NULL-allocs survive) -- proving this task's own
"durably survives a restart replay" completion criterion literally, not
just the in-memory mutation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 11:40:47 +00:00
b7ed6de2a1
docs(worker-rejoin-resync): correct design.md's NULL-alloc encode claim
Found live during task 2.2 implementation (bug-1746): design.md claimed
persist_submit_job_dispatch_fact's NULL-alloc case was "explicit,
tested, backward-compatible" on both apply seams -- true for the DECODE
side (apply_committed_entry/apply_wal_entry, bug-1678's actual fix
target), but the ENCODE side (capnp_encode_joballoc) was never
exercised with NULL before this feature and had a latent unconditional
NULL dereference. Corrected all references to distinguish encode vs.
decode safety and note the bug-1746 fix task 2.2 required.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 10:47:46 +00:00
cfe66dde16
feat(worker-rejoin-resync): reconcile unrecognized job to pending (task 2.1)
Add record_job_reconcile_pending_fact(cluster_t*, job_t*) to job.c:
eagerly applies apply_job_update_transition(job, JOB_STAT_PEND) (real,
live mutation, not a deferred peek -- matching dispatch_a_job_'s
established pattern, not record_job_finish_fact's), releases
job->job.allocs on an applied (non-no-op) transition, then durably
appends via the generic persist_submit_job_fact(WAL_JOB_UPDATE). Caller
holds job's lock throughout and is responsible for calling
persist_notify_job_updated_already_persisted() after unlocking -- this
function deliberately does not call it itself.

Closes bug-1631 at this new call site: a RUN job with a real allocation
reconciled to PEND (Requirement 4.6, worker reports it unrecognized)
would otherwise carry a stale allocation into its next dispatch attempt,
exactly the gap bug-1631 documented as dormant until something re-admits
an already-dispatched job to the scheduler's PEND pool -- this feature
is that path. dispatch_a_job_ itself is unchanged (still correct for a
job's first-ever dispatch, the only case reachable there).

3 tests: allocs release + eager-mutation-before-unlock proof, no-op on
already-pending, and the Requirement 4.4 terminal-state-preservation
proof (rejects overwriting JOB_STAT_DONE).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 10:13:08 +00:00
eda2eb1bf0
docs(worker-rejoin-resync): mark Foundation phase complete (task 1)
All 5 sub-tasks (1.1-1.5) done: wire protocol schema, worker reconciling
status type, worker test infra (pre-existing), wire codec helpers, and
xceadmin config-build wiring.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
2026-09-01 09:47:21 +00:00