repl_print_event_summary() (repl_break.c) now compares the candidate
rendered timestamp and displayed session number against a new cache
on xce_sched_repl_state_s (repl_internal.h: last_echo_ts_human[32],
last_echo_session) before printing the "[sim] t=... session=..."
preamble. It prints and updates the cache only when either value has
changed, or no preamble has been printed yet this session (the cache's
zero-initialized default, which no real echoed event can produce);
otherwise the preamble is skipped and the cache is left untouched. The
event's own content line prints unconditionally either way
(Requirements 1.1-1.4, 2.2).
All 6 RED-phase tests (task 1.1) now pass, including the critical
raw-vs-rendered nuance test (Requirement 1.3). No regressions in the
Event Echo Gate suite or the sibling spec's breakpoints-inspection
suite (test_step_echoes_events_before_session_decisions specifically
confirmed still passing under the new dedup behavior).
repl_break.c grew from 949 to 980 lines (+31, 3.3% over the residual
already accepted in scheduler-sim-repl-verbosity task 9.2). Review
judged this small growth as accepted without a fresh empirical split
test: task 9.2's own split experiment (a larger +44-line growth)
already found extracting this file's lookup-table domain reaches only
838 lines (still over cap) while requiring 22 white-box test files'
#include lines to change - disproportionate churn this smaller
increment would only re-confirm.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 6 failing tests for the Event Echo Preamble Deduplicator
(Requirements 1.1-1.4, 2.2): a second echoed event sharing its
predecessor's rendered timestamp+session omits its preamble; two
events across a session boundary both print their own preamble; the
critical raw-vs-rendered nuance (Requirement 1.3) - two events whose
raw nanosecond timestamps genuinely differ but whose rendered,
second-granularity timestamp and session are identical still dedupe,
with a fixture-premise self-check proving the two raw values really
do render identically; the first echoed event in a fresh session
always prints its preamble; a structural event under continue mode
after suppressed routine events follows the same rule; event content
and ordering are unaffected regardless of the preamble decision.
Lands in a new test_repl_step_verbosity.c (1010 lines, 6 tests) rather
than extending test_repl_event_echo.c (705 lines) - proactive split
per this file family's established practice, since appending would
have pushed that file well past the project's ~1000-line trigger.
Registered in testsuite/meson.build. No production code touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds repl_bp_kind_names(char* buf, size_t bufsz) (repl_break.c),
comma-joining BP_KIND_TABLE's tokens in table order into a caller
buffer. repl_print_help() (repl.c) is rewritten into four labeled
sections (Execution Control, Breakpoints, State Inspection, Snapshot &
Parameters) with a single shared " %-38s%s\n" column format, info
config moved into State Inspection next to its siblings, set units
added to Snapshot & Parameters, and the break line's description now
built from repl_bp_kind_names() so it enumerates the complete event
type set. repl_cmd_break()'s unrecognized-event-type error calls the
same accessor, so help and the error message can never disagree with
each other or with BP_KIND_TABLE (Requirements 7.1-7.6). No command's
existence, argument parsing, or other output changed.
repl_break.c grew from 905 to 949 lines (18.6% over the 800-line cap,
up from task 3.2's already-accepted 13%-over residual). Review
empirically tested extracting the BP_KIND_TABLE lookup-table domain
into a new repl_bp_kind.c: it built and passed, but only reached 838
lines (still 4.75% over cap) while requiring 22 white-box test files'
#include lines to be updated - the same disproportionate-churn,
doesn't-achieve-compliance outcome already accepted at tasks 6.2/8.2.
Reverted; 949 lines accepted in place.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds failing tests for Requirement 7 (Discoverable Help and Breakpoint
Event Types): help's four labeled sections in correct order, one
consistent column width across every listed command, info config's
placement alongside its info siblings, set units's placement alongside
set speed/output/verify, the break line's complete BP_KIND_TABLE-derived
event-type listing, and break's own unrecognized-event-type error
listing the same complete set. Each new test hardcodes BP_KIND_TABLE's
current 12 tokens rather than referencing the not-yet-implemented
repl_bp_kind_names() accessor, so RED comes from genuine assertion
failures against today's behavior rather than a link error.
New tests for Requirements 7.1-7.4 land in a new test_repl_help.c
rather than test_repl_execution_control.c (949 lines going in, within
reach of this spec's own ~1000-line split trigger already flagged at
task 2.1) - proactive split per this spec's established practice,
documented in design.md. break's Requirement 7.5 error-message test
lands in test_repl_breakpoints_inspection.c exactly as originally
planned, alongside the pre-existing test_break_unrecognized_event_type_rejected
(left unmodified).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Rewrites info worker's/info resource's resource-listing loops from a
flat, unconditional fprintf into a 3-pass, section-grouped table
(Consumable/Capacity, Load Metrics, Static Hardware Attributes),
units_mode-aware value rendering (raw/human/both, including swp's
MB->bytes conversion), whole-number formatting with no trailing .00
outside the consumable value=/used= pair, and jobid-range-compression
adoption for info worker's assigned-job listing (Requirements 6.4-6.6,
4.5). All 10 of task 8.1's tests pass. Preserves every named prior-bug
invariant (bug-1727's avail_v-as-total, bug-1613's consumable/
non-consumable split, bug-1593's GLOBAL/SHARED filter, bug-1688's
.caps-bound iteration).
Fixes one pre-existing test (bug-1688, predating this spec) whose
"value=5.00" assertion for a whole-number non-consumable resource
directly conflicts with Requirement 6.5's intentional no-trailing-.00
change - same class of fix as task 4.2's pre-existing-test updates.
repl_info.c grew from 985 to 1267 lines (58% over the 800-line cap,
worse than any other file this spec touched) - but unlike repl.c/
repl_break.c, it was already 23% over cap before this spec began.
Review empirically confirmed a narrow helper-only extraction doesn't
achieve compliance (still 41% over), and a compliance-achieving split
would mean restructuring ~690 lines of pre-existing display logic
this feature doesn't otherwise touch - a larger scheduler-core-level
decision, accepted as a residual rather than folded into this task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 10 tests in a new test_repl_resource_table.c covering info
worker's/info resource's resource-listing table/grouping layout,
units_mode-aware rendering (raw/human/both, including swp's MB->bytes
conversion and a no-mutation check on the underlying stored value),
whole-number-vs-fractional formatting, and info worker's assigned-job
listing adopting the real repl_fmt_jobid_range() (Requirements
6.4-6.6, 4.5). All 10 fail as expected against today's flat,
unconditional per-resource fprintf loop and uncompressed listing.
Found and fixed a genuine ASAN stack-use-after-return in the test's
own fixture code (not production): WORKERS_PUSH's handler moves a raw
pointer into a job's resvals rather than deep-copying, so the fixture
must heap-allocate rather than stack-allocate the structures it feeds
through that path - independently reproduced by review.
New file rather than extending test_repl_breakpoints_inspection.c:
landing this content there first pushed it to 3215 lines, 3.2x this
file family's own ~1000-line split trigger already used to justify
splitting test_repl_round_rollup.c/test_repl_event_echo.c for
structurally identical reasons.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Implements repl_is_bytes_valued_resource()/repl_format_bytes_human()
(mirroring xce_test's confirmed is_bytes_valued_resource()/
format_bytes_human() precedent exactly) and the units_mode setting
(explicitly defaulted to both, not left to zero-init) wired into
set units raw|human|both alongside the existing set speed/output/verify
branches (Requirements 6.1-6.3). All 8 of task 7.1's tests pass.
Split repl_format_ts_human(), the Jobid Range Compressor (task 1.2),
and these two new helpers out of repl.c into a new repl_fmt.c: adding
them in place pushed repl.c to 929 lines (16.1% over the 800-line
cap), and unlike task 6.2's single-function split attempt (which
didn't even achieve compliance), bundling these three pure-formatter
groups together does - repl.c settles at 587 lines, repl_fmt.c at 402,
both comfortably under cap. Required a matching #include line in the
20 test files that white-box #include repl.c, plus extending
test_sim_isolation.c's forbidden-include/symbol scan to cover the new
file, following the repl_rollup.c split precedent from task 2.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 5 pure-function tests in a new test_repl_resource_formatting.c
for repl_is_bytes_valued_resource()/repl_format_bytes_human()
(classification, adaptive B/KB/MB/GB/TB/PB scaling including exact
boundary values, swp's MB->bytes conversion), and 3 tests in
test_repl_snapshot_params.c for set units raw|human|both parsing
(Requirements 6.1-6.3). All fail as expected: link failure for the
not-yet-existing helpers, compile failure for the not-yet-existing
units_mode field/enum.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds repl_build_prompt(), building the readline() prompt fresh each
loop iteration: the plain "(sim) " before any session begins, else
"(sim session=<N> t=<ISO8601-ts>) " using the bare session count (not
+1, matching info session's convention) and the existing
repl_format_ts_human(). Both of task 6.1's tests pass.
repl.c grew 808->852 lines (6.5% over the 800-line cap). Review
empirically tried extracting repl_build_prompt() into a new
repl_prompt.c: it works, but requires updating 19 sibling white-box
test files' #include lists, and even then repl.c only drops to 814
lines - still over cap. Given the function is a small 15-line
formatter and the split doesn't even achieve compliance, accepted in
place rather than forcing disproportionate churn.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 2 tests in a new test_repl_prompt_context.c covering the dynamic
prompt's plain-before-first-session baseline and its session/timestamp
content in both step and continue modes (Requirement 5.5).
Required genuinely new test infrastructure: libedit's readline() only
writes its prompt text when both stdin and stdout are real ptys, which
no existing REPL test in this codebase's freopen()-from-file harness
can observe. Built a PTY-based capture technique (openpty(), ECHO
disabled via termios) instead - independently reproduced via a
standalone probe and stress-tested 15x with zero flakiness during
review.
New file rather than extending test_repl_execution_control.c (already
at 948 lines, over the 800-line cap), per this spec's established
split precedent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Gates sim_print_decision() so the bare per-decision line no longer
prints under continue mode, making the Round Rollup's summary the
sole source of decision output (Requirement 2.3, closing bug-1610).
step mode and non-interactive/headless runs are completely
unaffected (Requirements 5.1, 5.4). Implemented as a file-local
inline wrapper in sim_loop.c rather than repl.h, since the gated
function is static to that file and repl.h can't see repl_state's
internals without leaking them to every other repl.h consumer.
Fixes 3 pre-existing tests (predating this spec, from scheduler-core)
that pinned the now-intentionally-changed bare-line behavior under
continue mode - each fixed differently based on its actual purpose
rather than uniformly: two kept continue mode with their anchor
switched to the Round Rollup's summary (since they specifically test
continue-mode breakpoint interruption, which step mode would
trivialize), one switched to step mode (since its actual invariant -
print-before-breakpoint-check call order - is mode-independent and
continue mode no longer has any observable anchor for it).
Review caught a second, unguarded sim_print_decision() call site in
sim_reconcile_unaccounted_jobs() (added by an earlier, separately-
merged spec) that design.md's boundary never accounted for - per the
review, this is actually the dominant real-world path for DISPATCH
decisions in replay mode, so leaving it unguarded would have defeated
Requirement 2.3's "sole source of decision output" claim for the
common case. Gated it through the same wrapper; added 2 new tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 3 tests in a new test_sim_print_decision_gate.c covering the old
unconditional per-decision echo (sim_loop.c's sim_print_decision())
no longer duplicating the Round Rollup's per-round summary under
`continue` mode (Requirement 2.3), while `step` mode and headless
(non-interactive) batch runs stay completely unaffected (Requirements
5.1, 5.4). This is exactly the duplication bug-1610 reported.
1 of 3 tests fails as expected RED evidence (continue-mode bare
"[sim] PEND jobid=..." line still appears alongside the Round Rollup's
own summary today); the other 2 (step-mode and headless regression
checks) correctly pass since they pin already-correct, unaffected
behavior.
New file rather than extending either sibling: test_repl_round_rollup.c
is already 1022 lines (222 over the 800-line cap), and
test_repl_event_echo.c (704 lines) deliberately keeps its fixtures
job-free (bug-1595 discipline), which this task's decision-producing
fixtures cannot follow.
No production code touched (sim_loop.c/repl.c/repl_rollup.c/repl_break.c
untouched) - tests only, per this task's RED-phase boundary. The actual
gate implementation is task 4.2.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Implements mode-aware event echo classification, job-arrival range
compression (Req 4.5), and round-collapse flush wiring at all 3
breakpoint-match/structural-event call sites (Requirements 1.1-1.4,
3.5, 3.6). All 4 of task 3.1's tests pass; also closes bug-1609
("continue is as verbose as step").
repl_break.c grew 827->905 lines (13% over the 800-line cap). Unlike
task 2.2's Round Rollup, this logic is wiring inside pre-existing
driver-facing hooks, not a relocatable subsystem - accepted in place
per design.md's established residual-tolerance precedent.
Review independently investigated a one-off ASAN stack-use-after-return
in an unrelated test (sched_sim_record_output_test) surfaced by this
change's stack-layout perturbation: traced to the same documented
pre-existing thread-lifecycle bug class as bug-1598/bug-1632 (an
unjoined picker thread reading a stack-local sim after test-function
return), confirmed via 11/11 clean reruns - logged as bug-1763, not a
regression from this task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 4 tests in a new test_repl_event_echo.c covering mode-aware event
echo suppression and ending an active collapsed range on structural
events/breakpoint hits (Requirements 1.1-1.4, 3.5, 3.6). Job-free
fixtures (QUEUE_UPDATE/WORKERS_PUSH only) sidestep bug-1595's
jobid-reuse hazard entirely rather than merely avoiding it. 3 of 4
tests fail as expected RED evidence; the 4th (step-mode regression
check) correctly passes since it exercises unchanged behavior.
New file rather than extending test_repl_execution_control.c (already
over the 800-line cap at 937 lines) - applying the same file-size
lesson from tasks 2.1/2.2 proactively, decided before implementation
rather than after a review round.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Implements the round-boundary decision summary, collapsing of repetitive
rounds, and the shared repl_round_collapse_flush() entry point
(Requirements 2.1, 2.2, 2.4, 3.1-3.4, 3.6). Fingerprint composition
matches design.md exactly: dispatch/preempt counts from the existing
per-round tally, pend-jobid set state-derived from core->jobs (not
decision events), so Requirement 2.4 holds independent of bug-1611.
All 7 of task 2.1's RED-phase tests now pass.
Went through three review rounds:
- Round 1 approved the logic but rejected repl.c's size (1160 lines,
45% over the 800-line cap). Remediated by splitting the Round Rollup
functions into a new repl_rollup.c, following this codebase's own
established repl.c/repl_break.c/repl_info.c/repl_params.c domain-split
precedent.
- Round 2 approved the split but found the new file wasn't added to the
"simulation isolation" regression guard (sched_sim_isolation_lib +
test_sim_isolation.c's forbidden-include/symbol scans), plus two stale
repl.c-location doc comments left over from the move. Both fixed.
- Round 3 found repl.c still 17 lines over cap post-split (808 after
trimming, from 817) and rejected on that basis. Verified empirically
that closing the remaining gap isn't achievable via comment trimming
alone (even deleting both comments entirely only reaches 801) without
another disproportionate file split for single-digit lines. Accepted
808 lines (1% over) as the final state, consistent with round 1's own
explicit judgment that a small residual is not a blocker once the
original 45%-over violation is resolved - documented in design.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds 7 tests in a new test_repl_round_rollup.c covering per-round
decision summaries, collapsing of repetitive rounds, and the shared
repl_round_collapse_flush() entry point (Requirements 2.1-2.4, 3.1-3.4,
3.6). repl_round_collapse_flush() itself is not yet implemented (task
2.2) - build fails at link time, confirming RED.
Split into its own file rather than the originally-planned
test_repl_execution_control.c: that file was already at the project's
800-line cap before this task, and landing these tests there would
have pushed it to 1563 lines. design.md's File Structure Plan is
corrected accordingly (mirrors the existing test_repl_snapshot_params.c
/test_repl_breakpoints_inspection.c split in this same file family).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
delete <id> used atoi(), which silently coerces a non-numeric or
trailing-garbage token to 0 instead of reporting an error, e.g.
"delete abc" printed "no breakpoint with id 0" rather than rejecting
the input. Switched to strtol()+endptr validation, mirroring the same
"fail loud" discipline repl_cmd_step() already established in this
file family. Added test_delete_non_numeric_id_rejected covering
non-numeric, zero, negative, and trailing-garbage ids.
Logged as bug-1761 (buglog.json entry already committed alongside an
unrelated task's changes earlier in this branch's history).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Implements repl_fmt_jobid_range() in repl.c (per design.md's "no new
production .c file" commitment), grouping job-array identifiers by base
and compressing contiguous index runs into [lo-hi] ranges. All 5 of task
1.1's RED-phase tests now pass.
Went through one review round: an initial GREEN implementation placed
the function in a new file (repl_jobid_range.c) to sidestep a linker
issue in the lean test target from task 1.1. Review found and verified
a cleaner fix: link the test against the full scheduler_srcs object list
(mirroring the existing sched_scheduler_connect_test precedent), which
resolves the same symbols without a new file. Remediated accordingly;
bug-1762 in buglog.json documents the investigation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Adds failing unit tests for repl_fmt_jobid_range() covering contiguous
ranges, non-contiguous sub-ranges, multiple array bases, mixed array/
non-array ids, and a too-small output buffer (Requirement 4, AC 4.1-4.5).
The function itself is not yet implemented (task 1.2) - build fails at
link time with undefined reference to repl_fmt_jobid_range, confirming
the RED phase.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAWMhfRJFnmmnG8h97yUDd
Extends task 4.3's post-join gating to resource_sync_thread()'s
independent periodic reporting pthread, closing the Requirement 3.8 gap
where load-indices sends could interleave with an in-flight job-state
reconciliation report. Also fixes two pre-existing latent bugs
(bug-1756, bug-1757) in resource_sync.c that blocked a truthful RED/GREEN
signal for the new tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019ypUeKgbAqLWUkaSJ79poN
Closes bug-1755: worker_send_resync_query() hardcoded pktlimit=0 in
its xce_job_state_query_req() call, so the controller's outbound
OPCode_jobStateQuery always sent every non-terminal job ID in one
unbounded page regardless of count - the query direction never
actually exercised the pagination mechanism task 1.4/7.4 proved
correct, unlike the report direction's real, wired
XCECD_RESYNC_REPORT_PKTLIMIT bound.
Add XCE_RESYNC_QUERY_PKTLIMIT=100, mirroring that exact precedent
(same value - a JobStateQuery entry is a bare jobid, smaller than a
JobStateReport entry), and thread it into the one call site. Existing
small-job-count behavior is unchanged; a rejoin exceeding the cap now
genuinely produces multiple wire pages, proven against the real
production call site (not a bypass) with a 250-job/3-page test.
This is the last task in worker-rejoin-resync's implementation plan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add test_large_job_count_pagination_correctness, proving both
OPCode_jobStateQuery and OPCode_jobStateReport genuinely span many
wire pages (1042 jobs, 11 pages each direction, including a partial
final page) with zero jobs dropped/duplicated/truncated - exact-count
assertions throughout (page continuity, no-duplicate bitmap, exact WAL
delta, exact hook-fire count), extending task 6.2's 2-page proof by an
order of magnitude. Test-only, no production changes.
Logs bug-1755: the underlying query-pagination mechanism this test
proves correct is never exercised in production - worker_send_resync_
query() hardcodes pktlimit=0, so the controller's real query send
always produces one unbounded page regardless of job count, asymmetric
with the report direction's real XCECD_RESYNC_REPORT_PKTLIMIT bound.
No hard transport-layer size cap exists, so this isn't a demonstrated
failure mode, but it contradicts design.md's explicit "both paginated"
claim in this feature's own headline mechanism. Adds task 7.5 to close
it with a small, well-scoped fix before the feature-level gate, rather
than leaving a silently-false design claim in the completed feature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add test_burst_recovery_unbounded_by_job_count, reproducing bug-1741's
exact original shape (7 jobs on one host, 6 lost finish reports) in
one reconciliation round and one wire page: 5 finished-success + 1
finished-failure recover correctly and distinctly, while an already-
DONE job re-reported as an unqueried extra (with a deliberately wrong
finish time) is proven structurally unchanged - exact +6 WAL LSN
delta, not >=, so a duplicate append is genuinely caught rather than
silently tolerated. Test-only, no production changes; deliberately
kept to a single page since page-boundary correctness is task 7.4's
separate concern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add test_failover_during_job_recovers_lost_finish, a focused proof of
the feature's own original motivating scenario (bug-1741): a job
dispatched and RUN with no finish fact ever committed anywhere
(modeling a lost fire-and-forget finish notify), recovered to DONE
with the worker-reported finish time - not stuck in RUN - via the
real rejoin/query/report/apply chain. Test-only, no production
changes; reuses task 7.1's helper infrastructure rather than
duplicating its broader four-outcome coverage.
Updates the existing bug-924 (/tmp exhaustion / flaky teardown) entry
with a 5th occurrence hit during validation, unrelated to this task's
own change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add test_full_rejoin_reconciliation_round_trip proving the whole
chain composes end-to-end in one round, for a mix of still-running,
finished-success, finished-failure, and unrecognized jobs: real
rejoin_worker() -> worker_send_resync_query() (real wire query) ->
hand-built worker answer through the real report codec ->
resync_handle_report_page() -> resync_apply_report(), asserting
correct final state and scheduler notification for each. Test-only,
no production changes.
Logs bug-1754: a worker-reported job failure's exit code is never
durably recorded anywhere in this codebase (distinct from bug-1747's
finish-time gap) - discovered honestly asserted by the new test,
out of scope for this validation-only task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add OPCode_jobStateReport dispatch in myreq_handler, reassembling a
report across multiple incoming pages before handing it to task 3.2's
resync_apply_report() exactly once - proven via a test-only call
counter, not just end-state equivalence (a double-apply could pass an
end-state check if idempotent).
Standby-redirect investigation (Req 8.1/8.3): confirmed
ha_dispatch_gate already redirects every rejected opcode uniformly, so
no opcode-specific code was needed for that. OPCode_jobStateQuery has
no controller-side receive path at all (only sent, never received)
so needed no gate change either.
That same investigation found a different, more consequential gap:
fixes bug-1753, ha_dispatch_gate's quorum-loss fact-admission allowlist
(OPCode_notify/update/setpreason/clrpreason) was missing
OPCode_jobStateReport, so a report arriving during a transient
quorum-loss window - this feature's own core failover scenario - would
be rejected outright with no retry (worker sends fire-and-forget).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Arm a reconciliation round in apply_join_state() the moment a worker
host becomes connected (first join, reconnect, or failover-reconnect -
all routed through the same shared call site, Req 1.1-1.3), then send
the query from on_worker_join() strictly after the join reply and
existing config push (Req 2.1-2.2), verified via a structural test
that reads real wire order off the endpoint queue.
Also closes Req 3.7's config-delivery seam end-to-end: worker_resync_
window_ms now rides the existing Conf4Worker push (clustername
precedent, no new opcode) and the worker applies it to its retention
window on receipt.
Fixes bug-1752: myreq_handler had no OPCode_push dispatch case at all,
so every config push - including the pre-existing clustername field -
was silently discarded. Confirmed harmless until now (clustername was
never read anywhere in src/worker/); closed narrowly for
worker_resync_window_ms only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Extend the IS_WORKER_UNAVAIL-style candidate-host filter to also skip
a host with XCE_WORKER_RECONCILING set (Req 5.1), naturally
re-including it once the bit clears with no separate mechanism (Req
5.2). Leaves the scheduler's decision-publish code and its
symbol-guard regression test untouched.
Fixes bug-1751: design.md's cited filter site (scheduler.c's
sched_reset_worker_(), via sched_begin_session()) is dead code on both
live drivers - driver.c/sim_loop.c actually call
xce_core_begin_session() -> core_session.c's core_reset_workers(), an
independently-maintained mirror not listed in design.md's File
Structure Plan. Applied the identical disjunct to both sites so the
task's own observable-completion criterion holds in production, not
just on the literally-cited dead path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add OPCode_jobStateQuery dispatch to myreq_handler with multi-page query
reassembly, routing to task 4.2's xcecd_resync_answer_query() (Req
8.1, 8.3). Gate the worker's resumption of normal post-join operation
on a resync_pending flag until the full report has sent (Req 3.8).
Fixes bug-1749 (xcecd_resync.c was never added to the worker's
production source lists - task 4.2's build only ever exercised it via
the test binary's white-box include).
Round 1 review caught bug-1750: the gate sat as checking_connection()'s
entry guard, so a connection drop mid multi-page query left
resync_pending stuck forever, permanently disabling the worker's only
reconnect mechanism (Req 6.1 violation). Round 2 narrows the gate to
guard only the rejoin action, and resets it from xcecd_on_reconnect()
so a stalled query can never wedge it past one reconnect cycle.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Classify each queried job ID as running/finished-ok/finished-fail/
unrecognized (Req 3.1-3.3, 3.6), report unqueried running/recently-
finished jobs as extras, and send both via task 1.4's paginated wire
helper (Req 8.2, 8.3). Composes correctly with the already-implemented
controller-side consumer (resync_apply_report).
Adds xcecd_finished_job_retention_snapshot() to xcecd_worker_utils, a
small additive companion needed to enumerate all retained jobs for the
extras set (task 4.1 only exposed single-jobid lookup).
Folds the still-open OPCode_jobStateQuery receive-dispatch wiring gap
into task 4.3's scope in tasks.md, since 4.3 already needs the real
receive path to implement its post-join gate.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Retain a job's exit status/code/true finish time for a configurable
window after update_worker_status() detects its exit, before
cleanup_worker_list() would otherwise discard all trace of it
(Requirements 3.4, 3.5, 3.7). Hooked into the single choke point both
production exit-detection paths (libuv IPC callback, dbus polling)
converge through, so retention happens-before any reap regardless of
which mechanism is active.
Also prunes on retain (not just on lookup), closing a review finding:
lookup currently has zero production callers until task 4.2 lands, so
without this the store would grow unboundedly for the daemon's uptime.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add resync_apply_report()/resync_apply_running_entry()/
resync_apply_finished_entry()/resync_apply_unrecognized_entry() to
resync.c: routes each entry in a worker's job-state report to the
correct existing idempotent path (still-running -> clear from outstanding
set, no fact; finished-success/failure -> record_job_finish_fact_at/
on_job_fail; unrecognized -> record_job_reconcile_pending_fact;
unqueried extra -> record_job_reconcile_assign_fact or no-op), discards
reports for superseded rounds (Req 6.1/6.3), and resolves the round once
its outstanding set empties.
Also fixes bug-1747 (Req 4.2): adds record_job_finish_fact_at(),
additive alongside the unmodified record_job_finish_fact, so a
finished-success reconciliation entry records the WORKER-reported
finish time instead of reconciliation-processing time -- proven via a
real WAL-write + persist_recover() restart-replay test. The FAIL-side
half of the same gap (WAL_JOB_FAIL carries no fail-time wire slot at
all, for any caller) is confirmed pre-existing and systemic, correctly
left out of this task's scope.
Three review rounds surfaced and fixed two real, distinct bugs in the
unqueried-extra-job routing guard (see tasks.md's Implementation Notes
for the full account and the general lesson about re-deriving a full
truth table after a partial boolean-guard fix): round 1 caught the
wrong axis (terminal-vs-non-terminal instead of assignment-status,
Req 4.8); round 2 caught that the first fix's short-circuit clobbered
an already-terminal job's DONE/EXIT back to RUN (Req 4.4), including
confirming the implementer's own test had certified the bug as correct;
round 3 independently re-derived the final three-way guard from scratch
and found no further defect. bug-1748 logged for the guard fix.
Full controller suite: 1239 tests, 0 failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
New src/controller/resync.c/resync.h: resync_arm_round(cluster_t*,
worker_t*) enumerates every non-terminal job assigned to a host,
populates worker_t.resync (round_token, query job-ID set, deadline),
sets XCE_WORKER_RECONCILING, and schedules a non-blocking delaymgr
one-shot timeout. resync_resolve_round() clears the round; a fired
timeout resolves it too, via resync_round_is_current()'s by-value
round-token check -- delay.c/delay.h expose no true cancel/dequeue
primitive, so a stale timer from an already-superseded round becomes a
safe no-op instead of touching a newer round's live state (verified
safe against both misresolving the wrong round and use-after-free: the
timeout closure never holds a pointer into resync_state_t itself).
5 new tests: query-set correctness, fresh-token-per-arm, the
RECONCILING flag's set/clear lifecycle, and a direct (non-blocking)
timeout-fire proof via an XCE_TESTING-gated seam confirmed compiled out
of the production binary. Full suite: 1232 tests, 0 failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Split dispatch_a_job_()'s combined "(w->ep == NULL) || (w->host.status
!= 0)" check into two branches, checked in order: w->ep == NULL now
resolves directly via record_job_reconcile_pending_fact() (task 2.1) --
no dispatch_fail_count increment, no mark_job_fail(), never reaching
xce_classify_dispatch_fail() at all -- before the code ever tests
w->host.status != 0, which keeps its existing retry-then-terminal-fail
behavior completely unchanged (byte-identical body, confirmed against
HEAD).
Completes Task 2 (all controller job-table reconciliation outcomes:
2.1 reconcile-to-pending, 2.2 reconcile-assign, 2.3 this split).
RED phase caught a genuine pre-existing test bug: test_worker_notready_
resolves_retryable was seeding a NULL-ep worker while claiming to test
the host.status!=0 retry path -- it never actually exercised that path.
Corrected to use a connected-but-busy worker, with self-asserting
TEST_ASSERT_NOT_NULL(w->ep)/TEST_ASSERT_TRUE(status!=0) guards so this
can't silently drift again.
2 new tests, including a full JOB_TRANSITION_APPLIED path proof (WAL
LSN advance, hook-fires-exactly-once per bug-1679, allocs released per
bug-1631) through this exact call site, both using correctly-timed
canary references on both the job and worker objects to catch any
over-release under ASAN. Full controller suite: 1227 tests, 0 failures,
zero regressions across every other dispatch_a_job_ failure site.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*,
xce_job_state_entry_t*) to job.c: eagerly sets job->job.state to match
the worker's reported outcome (mirroring dispatch_a_job_'s inline
mutation, minus allocation fields -- there are none to set, job->job.allocs
stays NULL per Requirement 4.8's documented scope). Calls the existing
persist_submit_job_dispatch_fact(..., alloc=NULL) directly, then leaves
the post-unlock persist_notify_job_updated_already_persisted() call to
the caller, same convention as task 2.1.
Fixes bug-1746 along the way: capnp_encode_joballoc() (persist.c, part
of bug-1678's original fix, predates this feature) unconditionally
dereferenced a NULL alloc -- never triggered before since
dispatch_a_job_ (the only prior caller) always passes non-NULL. Fixed
with a minimal guard producing a true zero-length buffer, matching
apply_committed_entry's alloc_len > 0 decode gate exactly. This
directly contradicted design.md's "tested, backward-compatible on both
seams" claim (true only for decode, never actually exercised for
encode) -- corrected in 4 places.
2 tests: applies-running-no-allocs-and-notifies, and a real restart-
replay round trip (write to real disk, fresh persist_recover(), confirm
jobid/host/state/NULL-allocs survive) -- proving this task's own
"durably survives a restart replay" completion criterion literally, not
just the in-memory mutation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add record_job_reconcile_pending_fact(cluster_t*, job_t*) to job.c:
eagerly applies apply_job_update_transition(job, JOB_STAT_PEND) (real,
live mutation, not a deferred peek -- matching dispatch_a_job_'s
established pattern, not record_job_finish_fact's), releases
job->job.allocs on an applied (non-no-op) transition, then durably
appends via the generic persist_submit_job_fact(WAL_JOB_UPDATE). Caller
holds job's lock throughout and is responsible for calling
persist_notify_job_updated_already_persisted() after unlocking -- this
function deliberately does not call it itself.
Closes bug-1631 at this new call site: a RUN job with a real allocation
reconciled to PEND (Requirement 4.6, worker reports it unrecognized)
would otherwise carry a stale allocation into its next dispatch attempt,
exactly the gap bug-1631 documented as dormant until something re-admits
an already-dispatched job to the scheduler's PEND pool -- this feature
is that path. dispatch_a_job_ itself is unchanged (still correct for a
job's first-ever dispatch, the only case reachable there).
3 tests: allocs release + eager-mutation-before-unlock proof, no-op on
already-pending, and the Requirement 4.4 terminal-state-preservation
proof (rejects overwriting JOB_STAT_DONE).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add workerResyncWindowMs/workerResyncTimeoutMs to xce.schema.json
(minimum:0, matching the optional-scalar-field precedent -- these
aren't `required` with a floor like heartbeatInterval, so minimum:1
would be wrong) and to mapping.py's YAML->capnp field table. Regenerate
both golden config-build fixtures (task 1.1 grew ClusterConfig's wire
layout by 2 fields; full/expected.bin and minimal/expected.bin each
grew by exactly 8 bytes). full's fixture YAML now sets real values
(900000ms/15000ms); minimal deliberately left untouched (no xce.yaml)
per its own locked-down "hosts+queues only" boundary test -- its
expected.bin still regenerates to reflect the new default-0 fields.
Two new tests assert specific decoded values, not just "build succeeds".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add xce_job_state_query_req()/xce_job_state_report_req() to request.c,
paginating job IDs / job-state entries across OPCode_jobStateQuery /
OPCode_jobStateReport messages via direct Request.body union arms
(mirroring xce_jobdispatch_req/xce_notify_req's direct-send shape, not
xce_push_data_req's XceData-table multiplexing -- these are dedicated
union arms, not multiplexed push/reply-get traffic). 8 round-trip tests
(normal/empty/multi-page for both query and report, covering all 4
JobStateStatus values and both queried states) all pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add XCE_WORKER_RECONCILING status bit (0x00800000, next in the existing
sequential bitmask) + IS_WORKER_RECONCILING macro mirroring
IS_WORKER_UNAVAIL's shape, and resync_state_t (round_token,
queried_jobids array+count, deadline) referenced via a nullable
worker_t.resync pointer -- NULL when no reconciliation round is
outstanding. Compiles and is reachable, unused by any caller yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Add OPCode_jobStateQuery/OPCode_jobStateReport, JobStateEntry/
JobStateQuery/JobStateReport structs (paginated, mirroring JobData/
WorkerData/ResourceVals' {ntotal,pos,count} shape), and
workerResyncWindowMs/workerResyncTimeoutMs ClusterConfig fields to
controller.capnp, plus the companion typemap.h/clusterconfig.h C type
declarations this codebase's capnp $C.mapname codegen requires to
compile. Both new opcodes are fire-and-forget (Request.body union only,
no RequestReply arm), matching design.md and the push/notify precedent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
sched_reconcile_job_resource_usage() was only reachable from
sched_apply_job_rusage() (scheduler.c's sched_update(), which has no live
caller), so real-usage reconciliation never actually ran outside tests.
Wire it into core_events.c's XCE_CORE_EVENT_JOB_RUSAGE handler instead,
the actual live path shared by both normal-mode (driver.c) and sim/replay
mode (sim_loop.c). Adds link-satisfying stubs to the testsuite files that
#include core_events.c directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
xcecli hosts list showed NUM_RUN/NUM_SSUSP/NUM_USUSP as always 0 --
the fields existed on HostInfo and were displayed, but nothing ever
assigned them. processHostData now derives per-host slot usage from
the live job store (exec_hosts + job_status), counting occupied slots
so the numbers are comparable to MAX_SLOT.
Partial fix for gitea#46 / bug-1743: the separate "dispatch stops at
8/16" question needs the reporter's live limits.yaml/pend-reason to
confirm a usergroup/queue slot limit, not addressed here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
xcecli jobs kill/stop/resume/checkpoint failed for every job with an
empty error body. models.JobInfo.JobStatus/.ArrayIdx are plain Go
`int`, but validateJobControl/foundJob asserted them as `int64` via a
bare type assertion, which always panicked; Gin's Recovery()
middleware swallowed the panic and returned an empty 500.
Fixes gitea#45 / bug-1742.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
A job with no rusage[mem=...] request (reserved=0) was being SKIPPED
entirely during reservation-vs-usage reconciliation, on the mistaken
assumption that "no reservation" meant "nothing needs charging". It
doesn't: this codebase's admission check has no live OS read at
decision time (sched_check_host_satisfy_(), resource.c) -- it compares
the worker's last-reported "mem" value against this scheduler's OWN
tracked resInUse tally, built entirely from summed xce_res_account_t
entries. Skipping an unreserved job left its real, live memory
consumption permanently invisible to that tally -- an actual
over-subscription risk (the scheduler could admit more onto a host
than it can support, believing memory was free that a live process was
genuinely using), not merely a display gap.
Fix: charged = max(reserved, actual) applies to EVERY job, reserved or
not. ensure_host_account_by_resname() (resource.c) now creates a fresh
reserved=0 account entry (resolving residx via key_find() against
core->resmgr, same pattern sched_update_res_accounts() already uses
for a fresh account) instead of skipping when a job has no existing
entry for a resource. Returns NULL (a genuine no-op) only when the
resource isn't registered in the cluster's config at all.
Rewrote resource-accounting.md to correct the mechanism it documented:
"available" is NOT a live OS read (mem vs the similarly-named but
static maxmem was worth naming explicitly too) -- it's worker's
last-reported mem minus this scheduler's own resInUse tally, and that
tally must include every job's real usage regardless of reservation.
Also documented a known, pre-existing property surfaced while
correcting this: since mem's own worker report already reflects real
consumption and the scheduler's charge is a separate layer that
doesn't get cleared on a fresh report, tracking can become
conservative over time (under-utilization risk, never over-
subscription) -- the same direction slots already accepts, and a
separate, larger question than this fix's scope.
Replaced test_reconcile_job_resource_usage_skips_when_no_mem_request
(asserted the now-wrong skip behavior) with two tests: one proving an
unreserved job's real usage is created and charged, one proving a
genuine no-op when "mem" isn't a registered resource at all. Full
scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
Adds reservation-vs-actual-usage reconciliation for real (gauge-style)
resources, using mem as the concrete case -- documented in the new
resource-accounting.md steering doc.
Policy: charged = max(reserved, actual). A job's rusage[mem=...]
request is an estimate made before it has run, and the real usage the
worker later reports may land above or below it; charging a flat
reservation for the job's whole life either starves other jobs of
memory it never uses (when actual stays below the estimate) or lets
the host be driven into genuine oversubscription (when actual exceeds
it). Available memory for scheduling is then the host's OS-reported
available figure minus, per job, only the UNUSED SLACK of the
reservation (max(0, reserved - used)) -- never the job's actual usage
on top of that, since the OS's own figure already reflects it.
The existing xce_res_account_t{used, reserved, allocated} fields and
sched_update_res_accounts()'s used+reserved charge formula already
implement this exactly, once `used` is set to the OVERAGE beyond the
reservation (max(0, actual - reserved)) rather than left at its
permanent 0.0 default -- nothing previously computed it.
New (resource.c): sched_reconcile_host_resource_usage() sets that
overage on one xce_res_account_t and applies just the DELTA to the
affected worker's resInUse (not a full uncharge/recharge of the whole
job_account, which would needlessly re-touch sibling resources like
"slots" charged in the same account). sched_reconcile_job_resource_usage()
is the wrapper driven from a raw rusage report: maps the worker's real
wire resname ("memory_rss", same worker-resname-vs-scheduler-resource-
name split bug-1735 already established for the controller side) to
the scheduler's "mem" resource, and is a no-op when the job spans more
than one host (the wire report carries no per-host attribution today --
a separate, undocumented-until-now gap, out of this fix's scope) or
never requested rusage[mem=...] in the first place (reserved=0, nothing
to reconcile).
Wired into sched_apply_job_rusage() (schedobj.c), which already
receives every job rusage tick but previously only stored it for
display/plugin hooks with no resource-accounting effect at all.
Compatible with sched_account_jobs()'s existing per-round resInUse
reset+re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation
already reads from, not just a transient pool-level delta that a
round boundary would wipe.
Added link-satisfying stubs to the three white-box test files that
#include schedobj.c directly (test_job_update.c,
test_picker_config_fields.c, test_picker_wiring.c) -- none of them
exercise real job->accounts, so the new call is unreachable there, but
still needs to link. 10 new regression tests in
test_resource_accounting.c cover the core formula (below/at/above the
reservation, delta-only pool updates, sibling-resource isolation) and
the wrapper's scoping rules (resname mapping, multi-host skip,
no-request no-op).
Full scheduler suite: 55/55 passing, zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
repl_info_worker_one()/repl_info_resource_one() (repl_info.c) printed
value=avail+used for a CONSUMABLE resource, on the assumption that
avail_v holds a live "remaining capacity" figure that shrinks as jobs
run. That assumption was false: w->resvals[idx] (and
core->sharedInst[idx] for shared resources) is the resource's STATIC
TOTAL capacity, self-reported by the worker at join/heartbeat time and
never decremented as jobs consume it -- confirmed against the actual
admission-control formula this same field feeds (resource.c's
unsatisfy_(): `total[i] < inuse[i] + uncomitted[i] + wanted[i]` treats
it as a total, not a pre-reduced remaining figure). Printing avail+used
therefore double-counted `used` on top of a value that already
included it.
Live-reproduced against a real 3-node cluster + worker: a fully-packed
22-slot worker showed "value=44.00 used=22.00" instead of the correct
"value=22.00 used=22.00" (22 never-decremented total + 22 used = 44,
matching this bug's mechanism exactly). This was previously
investigated as bug-1727 ("recorded for future investigation only") --
confirmed its hypothesis (a) and closed it here rather than opening a
new entry.
Fixed by printing avail_v as-is (no addition) at both call sites.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
bug-1738: a scheduler-side notify could clobber an already-terminal
job state. The controller's own terminal-transition guard
(job_state_is_terminal(), job.c) treats DONE/EXIT as permanent once
set -- apply_job_fail_transition()'s POST_EXEC/Q_POST_EXEC branch only
ever MERGES a JOB_STAT_PERR flag onto whatever DONE/EXIT already
holds, never re-deciding DONE vs EXIT after the fact. The scheduler
had no equivalent guard: HOOK_DETAIL_JOB_FAIL's schedFailJob notify
(sched_job_fail_fun, sched.c) unconditionally requests JOB_STAT_EXIT
with no regard for the job's current state, so a job that finished
successfully (DONE) and then hit an unrelated post-exec script failure
would have the scheduler's view forcibly clobbered back to EXIT.
Fixed with one shared helper, job_state_regresses_terminal(), applied
at both XCE_CORE_EVENT_JOB_STATE_CHANGE and the JOBS_PUSH duplicate-
jobid resync merge (a full resync can just as easily redeliver a stale
snapshot as a genuinely newer one) -- once a job is DONE/EXIT, neither
path can move it to a different terminal state.
bug-1739 (Symptom 1, "phantom RUN"): the bug-1734 fix's own JOBS_PUSH
merge adopted a redelivery's RUN state without transferring the real
allocation data core_build_job() had already attached to the
redelivered clone -- leaving existing->state==RUN with
existing->allocs==NULL, a job the scheduler's own resource/slot
bookkeeping has no record of. Live-reproduced: `info summary` showed
30 elements of an array marked RUN, but `info worker <name>` only
listed 22 real slot assignments for a 22-core worker. Fixed by
transferring job->accounts/job->allocs onto the existing job on a RUN
merge (freeing whatever it previously held), and refusing the merge
entirely (WARN-logged, existing state kept) if the redelivery carries
no allocation data -- treating an unbacked RUN claim as untrustworthy
rather than manufacturing a phantom.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
sched_job_exit_fun() (renamed sched_job_finish_fun, sched.c) subscribes
to HOOK_DETAIL_JOB_EXIT, which fires for EVERY job finish, both success
and failure -- record_job_finish_fact() (job.c) decides DONE vs EXIT
from finish->exit_code BEFORE firing this same hook detail either way
(mirrors persist.c's own HOOK_DETAIL_JOB_EXIT -> WAL_JOB_FINISH
mapping). This function discarded info->exit_code and sent only a bare
jobid, forcing sched_handle_notify() (scheduler/src/handler.c) to
unconditionally map EVERY finish -- including a clean, successful one
-- to JOB_STAT_EXIT. A subsequent HOOK_DETAIL_JOB_UPDATE notify usually
self-corrected this moments later via the full job state, but if that
second notify was dropped during a scheduler reconnect gap (the same
window bug-1731/bug-1734 already documented), the scheduler was left
permanently stuck showing EXIT for a job the controller correctly
recorded as DONE.
Live-reproduced on a real 3-node HA cluster + worker: job
0-8-1788143749-0[38] genuinely completed successfully (worker log:
"completed successfully", controller joblist: DONE) but the
scheduler's own replay showed EXIT.
Fix: carry {jobid, exit_code} across the wire via the existing
JobFinishInfo shape (the same struct apiserver_job_exit_fun already
reuses for this identical hook) instead of a bare UInt64, so
sched_handle_notify() can make the identical exit_code==0-vs-not
decision job.c itself makes. Renamed sched_job_exit_fun to
sched_job_finish_fun to reflect what it actually does.
Live-verified: a 51-job run (all exit 0) showed 100% DONE on the
scheduler side after the fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
The oldest unresolved bug in this project's tracker (first sighted
2026-06-20): controller_unit_test intermittently hung for the full
300s meson timeout with no crash/ASAN report, never previously caught
live under gdb.
Live-reproduced this session (first attempt): thread 1 stuck forever
in commit_engine_fini() -> pthread_cond_destroy() (test_commit_barrier_
rollout.c's jap_teardown), while two fsync_worker threads sat parked
in pthread_cond_timedwait on the same persist_t. Root cause: several
test teardown functions called commit_engine_fini() BEFORE
persist_stop() -- the exact opposite of commit_engine_fini()'s own
documented contract. This let the live background fsync_worker thread
call back into commit_on_durable_advance() while commit_engine_fini()
was mid-destroy of the engine's lock/condvar -- a genuine TOCTOU race
with no synchronization between the two threads.
This is the same race already root-caused once for a SEGV symptom in
bug-1452 (2026-07-19, test_persist_truncate.c) and partially re-fixed
in bug-1687 (2026-08-25, cbrp_teardown) -- but never swept across all
affected files. Bug-1452's own remediation note explicitly flagged
test_demotion_convergence.c as a known, unfixed instance; that flag
went unactioned until now.
Fix: insert persist_stop() immediately before commit_engine_fini() at
21 confirmed-vulnerable sites across 12 files (test_apply_pump.c,
test_apply_pump_integration.c, test_commit_barrier_rollout.c,
test_commit_durability.c, test_commit_engine.c,
test_convergence_verification.c, test_demotion_convergence.c,
test_egress_filter.c, test_ha_role_lifecycle.c, test_job_array_fence.c,
test_job_array_replay_fence.c, test_p2_integration.c) -- the same
pattern bug-1452/1687 already established. persist_stop() is
idempotent (guarded by an atomic "already running" check), so each
fixture's own pre-existing persist_stop() call becomes a safe no-op.
No production code changed.
Known remaining exposure, documented but not fixed this session:
test_config_repl.c (~23 sites) and test_term_safety_adapter.c
(~5 sites) have the same ordering bug in their own teardown helpers,
flagged as follow-up.
Verified: full controller suite run 4 times total (once as part of
the fix, three further independent sequential runs) -- Ok:6 Fail:0
every time, 38-42s wall clock each run, no hang, nowhere near the
300s timeout. 1220/1220 individual controller_unit_test cases pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
XCE_CORE_EVENT_JOBS_PUSH's duplicate-jobid handling (sched_add_job()
EEXIST) assumed a redelivery is always a harmless no-op -- true for
bug-1714's array-push TOCTOU race (both copies start identically
PEND), but not for a scheduler reconnect's full resync
(on_scheduler_join() -> xcesched_push_jobs(), sched.c), which
re-sends every currently-committed job through this same path
regardless of how far its state has moved on since this scheduler
last saw it.
Live-reproduced on a real 3-node HA cluster + worker: the scheduler's
outbound xcecli connection repeatedly failed over under a 50-element
array-job load; each reconnect's resync redelivered jobs that had
already finished (PEND/RUN -> DONE) while disconnected -- confirmed
via ctrl-1.log's existing bug-1731 "c->sched.ep is NULL, notify
SKIPPED" diagnostic firing for the same jobids during the gap, making
the resync that job's only remaining chance to reach the scheduler.
Discarding it unconditionally permanently froze the tracked job at
its stale state: controller shows DONE, scheduler stuck at RUN/PEND
forever.
Fix: on EEXIST, look up the existing tracked job and merge in the
redelivery's state if different (same terminal-transition
accounts/allocs cleanup XCE_CORE_EVENT_JOB_STATE_CHANGE/JOB_UPDATE
already do), instead of silently dropping it. Only state is merged;
broader field reconciliation is out of scope, same as the existing
allocs/accounts scope note this code path already carries. Bundled a
requested logging fix at the same site (raw jobid -> fmt_jobid()).
Verified live end to end: rebuilt scheduler, reran the same load
against the same cluster (reconnect storm still occurred, proving the
fix doesn't depend on suppressing it), both test arrays now show
consistent DONE state matching the controller exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
