Widens sim_reconcile_unaccounted_jobs()'s trigger gate in sim_loop.c: removes
!JOB_IS_RUN(job->state) so a job's recorded DISPATCH decision is captured as
soon as the replay reaches the round it occurred in, regardless of the job's
current state -- closing bug-1611/bug-1613's residual (a job whose RUN
confirmation lands in a later round than its dispatch decision, the realistic
case under any real dispatch-to-confirmation latency, was never reconciled).
sched_account_jobs()'s own unmodified JOB_IS_RUN charging gate continues to
defer actual resource-in-use accounting until the job genuinely transitions
to RUN -- no other line in the match/capture/report path changes.
Replaces test_reconcile_skips_non_run_state_job (which asserted the bug as
correct behavior) with test_reconcile_captures_non_run_state_job_when_dispatch_recorded,
proving the opposite for a PEND-state job. Adds the xce-test-rc-file-style
spec docs (requirements/design/research/tasks) for this feature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
_live_e2e_common.py's CONTROLLER_CANDIDATES/XCEADMIN_CANDIDATES only ever
resolved to the three default profile-dir names (xce.build, build.release,
build.dbg), with no way to point the live E2E suite at a binary built to a
custom/explicit build dir. Adds XCE_CONTROLLER_BIN/XCEADMIN_BIN env-var
overrides via a new resolve_binary_candidates() helper -- when set, the
override becomes the sole candidate; otherwise the existing three-name
fallback list is unchanged. Mirrors the ${XCE_CONTROLLER_BIN:-default}
convention already used by clean_failover_preservation.sh/run-ha-soak.sh.
Skip messages in both live E2E test files now mention the override.
TDD: 4 new unit tests in test_live_e2e_common.py (no override, empty
override, override-becomes-sole-candidate, and an end-to-end reload proving
the real module constants respect their env vars). Full xceadmin suite
(288 tests) confirmed unaffected: 287 pass, 1 pre-existing unrelated
failure (a stale redirect-gap assertion, unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Now that xce_test auto-loads a personal rc file at startup (this spec),
harden the three existing non-interactive xce_test callers so a future
developer's ~/.config/xce_test/rc can never silently affect a CI/soak run:
run-ha-soak.sh, run-resource-verify.sh, and clean_failover_preservation.sh
all now pass --norc to their xtest() invocation. No behavior change today
(no rc file exists in this environment yet, per xce-test-rc-file task 3.3's
own verification) -- this is forward-looking reproducibility hardening.
Independently re-verified end-to-end: run-ha-soak.sh (--sequential, 2
rounds, zero job loss) and run-resource-verify.sh (1 resource, 3 layers,
zero mismatches) both PASS unchanged with --norc added.
clean_failover_preservation.sh could not be run in this sandbox -- its
DATADIR precondition (<repo>/ha/data/cluster.bin + certs) was deleted by an
unrelated earlier commit (ee8473bd, admin-config-tool) and never restored;
confirmed via git history this is pre-existing and unrelated to this edit
(the precondition check runs before xtest() and fails identically on the
unmodified script). Logged as bug-1721, not fixed here -- out of this
spec's scope.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Extends test_rcfile_integration.sh with 2 scenarios proving Requirements
3.1-3.4: a failing rc-file command in piped/scripted mode aborts before
any of the piped script's own commands run, exiting nonzero; the same
failing command in an interactive session still reports the failure,
continues executing the rc file's remaining lines, and reaches the normal
prompt -- while still carrying the session's sticky nonzero exit-code
verdict (g_session_failed is never cleared, matching how any other
failing-command-then-quit interactive session already behaves).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds test_rcfile_integration.sh, mirroring clean_failover_preservation.sh's
shell-driven black-box pattern against the real built xce_test binary:
- a valid rc file's set masters override takes effect in both a piped run
and a genuine pty-driven interactive run, before the normal session begins
- --norc leaves the default masters value untouched with the same rc file
present
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds the integration call site in main(), between g_continue_on_error's
computation and the read loop: unless --norc is given, resolve the rc-file
path and load it through preprocess_and_run (the same command-execution
path used for typed/piped input). A NULL path (no XDG/HOME) or missing
file falls through silently to the read loop. A failing rc-file command
under scripted/piped abort-on-first-failure mode short-circuits straight
to the existing g_session_failed exit-code convention, never entering the
read loop -- so a piped script's own commands never run after an rc-file
failure. Verified end-to-end against the built binary, including a real
pty for the interactive path, in both the failure and success cases.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds struct arguments.norc, a --norc argp option, OPT_NORC (matching
main.c's OPT_PEERS/OPT_CONF_KEEP_N idiom), and its parse_opt case to
test.c. Parsing/recording only -- no wiring into main()'s control flow
or rcfile.c yet (task 2.2).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds rcfile_load(path, runner) to rcfile.c/rcfile.h: reads lines with
getline() (no fixed-size buffer, so a long line like a multi-peer
`set masters ...` can never be silently truncated or split), strips the
trailing newline, invokes the caller-supplied runner once per line, and
stops at the first nonzero runner return, propagating that value. A missing
rc file is a silent no-op; any other open failure prints one stderr warning
and returns 0.
Found during GREEN: fopen() succeeds on a directory path on this platform,
so the EISDIR failure only surfaces on the first getline() call (ferror(f)
set) -- added a post-loop ferror() check to catch it (bug-1716).
5 new unit tests (11 total with task 1.1): ENOENT-silent, per-line-in-order
with newline stripped, stop-on-nonzero with exact propagation, non-ENOENT
open-failure warning, and a >4KB single line proving no truncation/split.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Adds rcfile_resolve_path() (rcfile.c/rcfile.h): resolves $XDG_CONFIG_HOME/xce_test/rc
when set and non-empty, else $HOME/.config/xce_test/rc, else NULL. Standalone
libc-only module mirroring the configcompare.c pattern; wired into xce_test_src
and controller_unit_test_src, registered in controller_test.c's Unity runner.
6 unit tests in test_rcfile.c cover all XDG/HOME combinations plus a dedicated
env-var save/restore isolation test, addressing the cross-test leak risk found
during design review (controller_unit_test runs 80+ test files in one process).
Adds the xce-test-rc-file spec (requirements/design/tasks/research), including
design fixes from validation: getline()-based line reading (task 1.2, avoids
truncating long rc-file lines) and the env-isolation requirement above.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Merge follow-up fixes found during comprehensive unit-test + live-cluster
verification of the scheduler-framework merge:
- apiserver/backend/clib.go (ControlJob): Rongsong's clib.go had reverted
our bug-1490 fix — ctrl.action must be the capnp JobControl union
DISCRIMINANT (0=siginfo,1=ckptoption), not the models.JOB_* constant
(1/2/3/8), or every kill/stop/resume fails -EINVAL. Restored
ctrl.action = C.int(jobCtrlDiscriminant(action)); verified live (job
kill RUN->EXIT).
- controller/persist.c (fsync_worker): the merged bug-1563 no-progress
backoff used the PRE-FLUSH `pending` snapshot, so a no-op flush (another
path advanced durable_lsn to tip mid-flush) also slept wal_sync_ms and
raced commit_engine_fini in teardown, hanging the p2_integration suite
~60% (bug-1515). Re-evaluate the level AFTER the flush (pending_after)
so only a genuine sustained fdatasync failure backoffs. 20/20 clean p2
runs (was 3/5 hang). Also carries the bug-1509 B3 RUN->UNKWN marking in
persist_recover.
- scheduler/worker testsuite meson.build: add unity_dep parent includedir
(unity.pc Cflags only add -I${includedir}/unity, so <unity/unity.h> was
unresolvable) — same fix as src/lib/meson.build.
- test_handlers.c: re-point notify.data.dispatchfail after the first
sched_handle_notify NULLs it on success (code=0 branch was -EINVAL).
- test_controller_connector.c: give the rejoin-success fixtures a valid
worker_list pointer (my bug-1509 resend_job_status deref was NULL-
crashing).
- test_wal_job_dispatch_fact.c / test_dispatch_commit.c: Rongsong's
WAL_JOB_DISPATCH replay tests asserted JOB_STAT_RUN after persist_recover,
conflicting with bug-1509 B3 (RUN->UNKWN on restart). Assert JOB_IS_RUN
(base RUN recovered, incl. bug-1678 allocs) + UNKWN flag instead.
All suites green: controller 1145/0, scheduler 57/57, worker 15/15,
lib 303/0, apiserver CGO build ok.
Found via the mandatory live HA chaotic soak for this session's controller
audit campaign -- unrelated to any of the 9 audit fixes (bug-1684/1687/
1689/1690/1691/1692/1693/1694), all in ha.c/persist.c/job.c/cluster.c/
sched.c/worker.c; this is in myapp.c, untouched by any of them.
myapp_run()'s bootstrap "is a leader already known" check had a fallback
`c.ha.raft.term > 0` meant to detect genuine new raft activity processed
during the startup-stagger nap. But ha_restore_term() loads ha->term from
disk BEFORE this check runs, on every boot -- and Raft terms only ever
increase, never reset. So `raft.term > 0` is trivially true on every
restart for the rest of the cluster's on-disk lifetime, once it has ever
held one election, regardless of whether any leader is currently
reachable. Live-reproduced: every node, on every restart after the
cluster's first election, unconditionally skipped ha_start_election_async
forever -- no node could ever re-assert leadership through a real
election again. bug-1477's existing cluster_init_phase2 retry loop kept
firing correctly (confirmed in the reproduction logs) but could never
succeed, since the deeper cause -- no election ever runs to produce a
live primary to push config from -- was never addressed by that fix.
Fixed by snapshotting `term_at_boot` right after ha_restore_term()/
ha_init_raft() seed the term, strictly before the stagger nap, and
comparing `raft.term > term_at_boot` instead of `raft.term > 0` -- only a
term that has genuinely ADVANCED during this boot cycle (real vote/
heartbeat traffic processed) counts as evidence a leader is known. The
decision was extracted into a new, unit-testable myapp_leader_already_known()
(startup_validation.c, since controller_unit_test deliberately excludes
myapp.c/main.c). voted_for_name/vft==cur and primary_ctrl_fd>=0 checks
are unchanged.
7 new tests in test_ha_startup.c; RED/GREEN independently verified.
Rigorous A/B live-soak proof (identical 6-round --chaotic run, fixed vs
git-stashed-back-to-original): baseline showed term frozen for the whole
run and zero election activity on any non-primary node; fixed run showed
term genuinely progressing and ctrl-2 achieving a real, log-verified
CTRL_PRIMARY transition via a won PreVote round -- proof a node other
than the original primary can now take over, which was structurally
impossible before. Full controller suite: 1191 tests, 2 failures -- both
the pre-existing, unrelated bug-1561 overlayfs sandbox flake.
Residual, NOT independently verified yet (flagged honestly, not fixed
here): the fixed run's chaotic verdict showed committed_job_loss=6
(baseline 0) -- traced via waldump to one jobid missing from every node's
WAL, at the cold-boot window before round 1. Resembles the already-fixed
bug-1487 false-positive class (apply-pump-lag racing verify_classify's
expect-job check) but the reported mechanism doesn't perfectly match
(bug-1487's job was present-in-WAL/not-yet-visible; this one is described
as absent from every WAL entirely) -- needs independent re-verification
before being dismissed, tracked as a follow-up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Last of the 9 confirmed findings from the /xce-audit-invariants controller
pass. checkpoint_task_fn was registered directly as a delaymgr entry and
ran its full blocking body inline: checkpoint marker pwrite+fdatasync,
write_snapshot()'s genuinely unbounded per-entity walk (scales with
cluster state size) held under apply_pump_lock, WAL trim, and snapshot
rotation/unlink -- all on the same thread that drives heartbeats/election
timeouts. Unlike every other delaymgr operation (snapshot send,
config-push, reconnect, catch-up), checkpoint had no offload mechanism
at all -- the one gap left from the raft-drive-async hardening pass.
Design: full offload via the SlowLane (slowlane_submit), default ON --
NOT the vtask pool. persist_apply_pump_run() already holds apply_pump_lock
across its own per-entry walk while running ON the vtask pool
(VTASK_LIVE_BLOCK); putting checkpoint's unbounded, lock-held
write_snapshot() on that same pool risked starving it, exactly what
ha-async-io.md's own thread-role table warns against. The SlowLane is
the actual established pattern for this I/O class (push_config_transport_only,
the snapshot-send path, catchup_lane_fn, demotion_lane_fn all dispatch
there) -- and persist_checkpoint() already runs on the slow lane in
production TODAY via ha_snapshot_do_send() -> snapshot_lane_fn, so
apply_pump_lock's acquisition from a slow-lane thread is proven-safe,
exercised daily, not new surface. Unlike catchup/demotion (which stay
flag-gated OFF for an unrelated, still-not-root-caused hang, bug-1429),
checkpoint's new flag defaults ON given this reuses an already-proven
code path. Inline-fallback + WARN only on a genuine slowlane_submit
failure (not -EAGAIN/-EEXIST, both non-failures) -- never silently drops
a scheduled checkpoint.
New test_persist_checkpoint_offload.c (3 tests). RED-phase forced the
pre-fix always-inline path: the suite never printed a single result and
had to be killed, proving the test genuinely detects the violation, not
a tautology. GREEN: all 3 pass. Full suite: 1184 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake
(exact signature match). No live HA soak was run for this specific fix
in isolation (explicitly scoped out, not hidden) -- the mandatory
live-soak proof for this whole audit campaign covers it next.
This closes the last of the 9 confirmed findings from the controller
audit (bug-1684, bug-1687/1689/1690/1691/1692, bug-1693, bug-1694).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Violates this project's own recorded Decision Log entry: "No client
reply-wait timeout as a fix for a hung request -- fail loud at the real
transport/source instead of masking with a deadline backstop."
OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy
-- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy
pattern", confirming one shared design mistake, not two independent
bugs) both blocked on a multi-second pthread_cond_timedwait, replying
"timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the
actual disconnect handlers) only nulled the endpoint pointer -- neither
ever signaled the pending waiter, so a mid-wait disconnect was invisible
until the deadline lapsed. The timeout was the ONLY failure-detection
mechanism, exactly what the decision forbids.
Both sites already had a working signal function on their success path
(schedadmin_signal(), fetchfile_signal()) -- just never wired to the
disconnect handlers. Fixed by calling them from
handle_scheduler_leave()/handle_worker_leave() with a distinguishable
-ECONNRESET outcome, so the waiter wakes immediately on disconnect
instead of only on the deadline. Timeouts kept as defensive backstops
for a genuinely wedged peer that never disconnects cleanly either --
this fix is about not relying SOLELY on the timeout, not removing it.
schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave()
acquires c->sched.lock -- the dispatch wait holds that same lock for its
entire duration, so signaling after acquiring it would deadlock.
Found by an /xce-audit-invariants controller pass (Decision Log fidelity
dimension). New test_fetchfile_disconnect.c plus extensions to
test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN
independently verified against real elapsed-time assertions (pre-fix:
5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller
suite: 1181 tests, 2 failures -- both the pre-existing, unrelated
bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Last of the five field-parity gaps from the original audit (bug-1678,
bug-1687, bug-1689, bug-1690, bug-1691). set_preason()/clear_preason()
set job.pend_reasons (host/resource-based pending-job explanations), but
persist_job_hook_fun's default: arm only ever captured {jobid, state}.
Neither apply seam restored pend_reasons after a restart.
Reuses the ALREADY-EXISTING JobPendReason/HostBasedReason/ResourceBasedReason
capnp structs -- no schema change needed, unlike bug-1690's WAL_JOB_RUSAGE.
Safety-checked against bug-1690's lesson (RTTI/makeString()-backed fields
need merge-not-replace, not free-and-replace): pend_reasons is plain
malloc/strdup-backed throughout (erase_job_preason is FREEUP()-only,
add_host_preason/add_res_preason use strdup()/calloc()), so a
straightforward free-and-replace is safe here, matching bug-1678/1687's
pattern instead.
New wrinkle beyond the four prior fixes: pend_reasons has a genuine CLEAR
operation, and "trailing bytes present or absent" can't distinguish a
legitimate clear from an old-format payload -- added an explicit presence
byte (0=cleared/adopt NULL, 1=capnp bytes follow), verified by two
dedicated "clear survives restart" tests.
11 new tests in test_wal_job_preason_fact.c; RED/GREEN independently
verified (7 genuine pre-fix failures, one test's own wrong premise about
capnp's garbage-byte behavior caught and corrected during RED/GREEN
iteration). Full controller suite: 1178 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake. No ASAN findings.
This closes the field-complete apply parity invariant class the
xce-audit-invariants controller dimension 7 was added for.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
After merging origin/scheduler-framework (scheduler-core), re-apply and
reconcile our fixes that Rongsong's branch lacked or implemented differently:
- handler.c: keep the code<0 dispatch-fail check (bug-1512: a code-0
success ack must NOT re-PEND, or dispatched jobs spawn duplicates).
Use sched_job_state_change (Rongsong's enqueue mechanism) instead of
sched_requeue_pend_job so the merged test framework (pop_jstate_change)
stays valid. Drop Rongsong's duplicate unconditional re-PEND case.
- scheduler.c: port sched_requeue_pend_job (bug-1494 class) adapted to
sched_get_job(&sched->core, ...) signature.
- types.h: re-add xce_sched_queue_t.dispatchwindow (our time-window PEND
reason field) dropped when taking Rongsong's types.h.
- models/jobs.go: re-add JobSubmitResponse.JobIDHuman (human job id).
- worker controller_connector.c: our bug-1509 resend_job_status on rejoin
success was already ported; add test stub.
- tests: adapt dispatch-fail test (code=0 → no re-PEND); fix sched_get_job
signature in white-box stubs.
Build passes (nix build .#xce). Unit + live-cluster verification pending.
Co-Authored-By: Claude <noreply@anthropic.com>
Same class as bug-1678/1687/1689/1690: on_job_ckpt() sets job.ckptinfo
(a plain char*, xstring-backed via makeString()) but is purely hook-driven
-- persist_job_hook_fun's default: arm only ever captured {jobid, state}
for WAL_JOB_CKPT. Neither apply seam restored ckptinfo after a restart.
Simplest of the five field-parity fixes: ckptinfo is a bare string, so no
capnp struct is needed -- mirrors WAL_QUEUE_CTRL's existing NUL-terminated-
string-after-header wire shape instead. Adoption is unconditional (not
gated on a state transition), matching bug-1690's precedent: a checkpoint
report never changes job.state. restore_job_ckpt_state() releases the
prior value via dereference() (the correct RTTI verb for an xstring), not
free().
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension) -- the last of five sibling gaps found together
(bug-1678, bug-1687, bug-1689, bug-1690, this one). 7 new tests in
test_wal_job_ckpt_fact.c; RED/GREEN independently verified (5 genuine
pre-fix failures, 2 backward-compat tests correctly passed unchanged).
Full controller suite: 1167 tests, 2 failures -- both the pre-existing,
unrelated bug-1561 overlayfs sandbox flake. No ASAN findings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678/1687/1689: the WAL fact for WAL_JOB_RUSAGE only
carried {jobid, state}, so job.run_usage (client-reported per-host
CPU/mem usage) was never restored on either apply seam after a restart.
Unlike the prior three, xce_job_runusage_t/xce_host_runusage_t had no
existing Cap'n Proto mapping -- a genuine new schema addition
(HostRunusage/JobRunusage in controller.capnp, mirroring JobPendReason's
List(NestedStruct)+$C.maplistcount style), verified against the real
struct definitions in types.h, codegen confirmed via nm on the built
binary rather than assumed.
Two deliberate deviations from the established precedent, found by
reading the real code rather than copying the pattern blindly:
1. Adoption is unconditional, not gated on a state transition -- a
rusage report almost never changes job.state (repeat reports keep
state at RUN), so gating on JOB_TRANSITION_APPLIED like START/FINISH/
DISPATCH do would make the fix dead code for the realistic case.
2. Merge, not free-and-replace. update_host_runusage() builds run_usage
with host_name sourced from makeString() -- an RTTI-managed pointer,
not malloc() -- and the primary's own live apply pump re-applies every
fact it just wrote. A naive free_JobRunusage_ptr()-then-adopt (matching
the prior three fixes exactly) would free() a non-malloc'd pointer on
the very first live rusage report for a new host -- guaranteed heap
corruption. Fixed with merge_job_runusage(): existing hosts get
scalar-only in-place updates (host_name pointer never touched), only
genuinely new hosts adopt their freshly-decoded (safe, malloc'd)
host_name. Proved via a real makeString()-backed test asserting
pointer identity survives, not just value equality. job_destroy()
(src/lib/jobj.c) was found to have this identical latent mismatch,
pre-existing and independent of this fix -- flagged, not fixed here.
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension). 8 new tests in test_wal_job_rusage_fact.c; RED/GREEN
independently verified. Full controller suite: 1160 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake. No
ASAN/heap-corruption findings. Backward compatibility confirmed for
old-format bare-payload entries.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678/bug-1687: record_job_finish_fact() wrote only
{jobid, state} for WAL_JOB_FINISH. Both apply seams either fabricated
finish_time via time(NULL) or touched nothing beyond state.
Worse than "lost on restart": apply_job_finish_transition() -- the only
function that ever set job.exit_info -- is dead code on the real
production apply path. The raft-apply-decoupling redesign routes actual
mutation through the leaner apply_job_update_transition() instead, so
exit_info was never populated at all, restart or not, before this fix.
job_exitinfo_t ({int exitcode;}) has no existing capnp mapping, so this
uses a small native (non-capnp) trailing payload -- wal_jobstate_t header
+ uint64_t finish_time + int32_t exit_code -- mirroring wal_jobstate_t's
own native-struct style rather than inventing a capnp wrapper for one
field. run_time is never carried on the wire, always re-derived as
finish_time - start_time at apply time. Both apply seams updated;
old-format payloads (exactly sizeof(wal_jobstate_t)) decode unchanged
(fabricated finish_time, NULL exit_info) -- full backward compatibility.
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension). 6 new tests in test_wal_job_finish_fact.c; RED/GREEN
independently verified (4 genuine pre-fix failures with real
expected-vs-fabricated mismatches, 2 backward-compat tests correctly
passed both before and after). Full controller suite: 1152 tests, 2
failures -- both the pre-existing, unrelated bug-1561 overlayfs sandbox
flake (confirmed flaky across repeated re-runs of the same binary).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678: apply_job_start_transition() sets the real
job->job.pidinfo (worker-reported PIDs/PGIDs/exdata) and start_time, but
the WAL fact for WAL_JOB_START only carried the shared wal_jobstate_t
{jobid, state} header -- both apply seams (apply_committed_entry() for
primary boot replay, apply_wal_entry() for standby catch-up) restored
state only. pidinfo stayed permanently NULL after a restart; start_time
was fabricated with time(NULL) at replay instead of being carried,
drifting from the true value.
Mirrors bug-1678's exact fix shape: capnp_encode_pidinfo()/decode_pidinfo()
(persist.c, using the pre-existing JobPIDInfo capnp struct -- no schema
change needed) serialize job_pidinfo_t into the WAL payload, trailing a
new uint64_t start_time field appended after wal_jobstate_t.
persist_submit_job_start_fact() replaces the old state-only
persist_submit_job_fact() call for this type. Both apply seams updated;
a payload of exactly sizeof(wal_jobstate_t) (any entry written before
this fix) decodes with pidinfo left NULL and start_time fabricated via
time(NULL), exactly matching pre-fix behavior -- full backward
compatibility, verified by every pre-existing WAL_JOB_START test using
the old bare-struct shape.
Also fixed a genuine, unrelated deadlock this change's larger payload
exposed in test_commit_barrier_reply.c: cbrp_count_jobstate_entries_ex()
matched WAL entries by exact payload size (==), so once WAL_JOB_START
legitimately grew it silently undercounted to 0 and a downstream test
spun forever polling an uncounted target. Fixed the match to >= (jobid/
state always live at a fixed leading offset) -- bisected via /proc/*/wchan
futex inspection (no ptrace in this sandbox), confirmed as the actual
cause, not the teardown-ordering red herring first suspected.
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension, added after bug-1678). 6 new tests in
test_wal_job_start_fact.c; RED/GREEN independently verified (4 tests
genuinely failed pre-fix with exact expected-vs-fabricated value
mismatches, 2 backward-compat tests correctly passed both before and
after). Full controller suite: 1146 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
User-reported live crash: xce_scheduler SIGSEGV in fcfs_cursor_enter_queue
(builtin_fcfs.c), called from builtin_fcfs_next_job <- sched_job_picker
(picker thread).
priv->cursor/priv->job_cursor (fcfs_plugin_data_t) are raw ordq_item_t*
pointers that persist across SEPARATE next_job() calls, each with its own
independent priv->lock acquisition/release. The existing mutual-exclusion
fix (task 2.4) correctly protects the cursor walk from corruption DURING
one locked call, but does nothing to stop a LATER call from dereferencing
a node a properly-locked, INTERVENING call already freed.
on_queue_update()'s re-sort (remove old ordq_item_t node, insert a fresh
one via ordq_put) is exactly such a free -- with only one queue in the
deployment, the picker's cursor is essentially always parked on it
between calls (sched_queue_is_ready/sched_job_is_ready are unconditional
true stubs today, so the cursor never advances away while jobs remain),
making a queue config update land in that window trivially reachable.
Fixed by invalidating (NULLing) whichever cursor points at a node right
before that node is freed, at all three sites that can free a node the
cursors might reference: on_queue_update's reuse-and-resort path
(priv->cursor + priv->job_cursor in lockstep), builtin_fcfs_on_job_state_change's
!JOB_IS_PEND removal, and fcfs_remove_job_everywhere's cross-queue removal
loop (both job_cursor only, via an ordq_peek-then-invalidate-then-delete
pattern). Always safe: next_job() treats a NULL cursor as "no more
candidates this round", and begin_session() resets both cursors fresh
every round regardless -- worst case is a bounded one-round delay, never
a crash or a permanently lost job.
New test_builtin_fcfs_cursor_uaf.c reproduces the bug fully single-threaded
(next_job() releases the lock between calls, so no real concurrency is
needed to hit the window). RED-phase independently confirmed a real ASAN
heap-use-after-free with the exact same call chain as the live crash
(freed by on_queue_update, read by next_job()); GREEN confirms 0 failures
and no job permanently lost across the invalidation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
User-reported live crash: xce_scheduler SIGABRT via glibc malloc_printerr
(heap corruption) inside kill_limit(), called from limit_update_config()
during a live config reload.
limit_update_config() called kill_limit(limit) in its error path after a
failed key_add(privdata->limits, ...). But privdata->limits is a kvmgr
created with .datakiller = kill_limit, so on a kvtab-level failure (e.g.
-EEXIST for a duplicate limit name -- exactly the rc=-17 captured in the
crash's own backtrace), xkvmgr_add() already rolls back via vclear() ->
xzarray_clear(), which fires kill_limit(limit) once automatically. The
caller's explicit second call was a double free.
Removed the redundant call; the kvmgr's own configured datakiller already
owns cleanup on this failure path (mirrors the ownership asymmetry
job.c's restore_job_array() already documents for the opposite case,
where its kvmgr has no datakiller configured and the caller must free
failed items itself). One accepted, documented residual gap: a vstore()-
level OOM failure (never reaching the kvtab step) would leak `limit`
instead -- vanishingly rare, and a leak is strictly better than the
guaranteed crash on the common duplicate-name case; xkvmgr_add's return
code doesn't distinguish which sub-case occurred, so there's no clean way
to free conditionally without a kvmgr API change (out of scope).
New test_limit_plugin.c (this plugin had zero test coverage before) --
RED-phase independently reproduced a genuine ASAN heap-use-after-free
matching the live crash's exact call chain; GREEN confirms 0 failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
ha_persist_term() had a void return -- every open/write/fsync/rename
failure was silently swallowed ("best-effort; caller must not rely on
this for correctness"). Its sole vote-granting caller, raft_cb_persist(),
always returned 0 regardless, so the raft library always granted the
vote even when the term/voted_for record never reached disk -- a real
split-vote risk if the node then restarted and forgot it voted
(Election Safety requires durably-persisted votedFor).
ha_persist_term() now returns 0 on success (including the deliberate
NULL-workdir test-mode skip) or -errno on a genuine I/O failure
(fsync's previously-unchecked return is now checked too). raft_cb_persist()
propagates that failure to the raft library (denying the vote per its
own documented contract) and rolls back ha->voted_for_term/
ha->voted_for_name to their pre-call snapshot, so the adapter's
in-memory state stays consistent with the library's own "deny without
updating voted_for" behavior. ha->term is deliberately NOT rolled back:
the raft library adopts a higher observed term unconditionally at Step 2
of raft_on_vote_request, before and independent of the vote-grant/persist
outcome, so the adapter correctly mirrors that same semantics.
Also corrects a stale claim in controller/CLAUDE.md's Protocol
Verification Checklist (found by a sibling audit dimension the same
session): term+voted_for were never written "atomically under ha->lock"
-- term uses its own atomic store outside the lock.
Found by an /xce-audit-invariants controller pass (error-handling
dimension). 3 new tests in test_ha_durability.c drive the real
production path (ha_init_raft + raft_on_vote_request, the exact
function cluster.c's dispatch calls) against a forced ENOENT write
failure; RED/GREEN independently verified.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
sim_loop.c never called plugin_begin_session()/plugin_end_session() at
its round boundary, unlike driver.c's normal-mode round -- a driver-
equivalence gap found by an /xce-audit-invariants scheduler pass.
Currently inert (sim mode's only reachable picker, builtin_fcfs, leaves
both hooks NULL) but structurally identical to bug-1613's class of bug.
Relocated plugin_begin_session()/plugin_end_session() from an implicit
xce_scheduler_t* to xce_sched_core_t* directly (mirroring
sched_account_jobs()'s prior relocation) -- their bodies never touched
anything beyond core->plugins, so driver.c's comment claiming they
"needed the full xce_scheduler_t*" was itself stale. sim_loop.c now
calls both, same relative position as driver.c. A full-tree grep (done
after the first build attempt failed on an untouched file) turned up a
third live call site in scheduler.c's legacy sched_begin_session()/
sched_end_session(), plus four testsuite files with their own stub
definitions of the same functions -- all updated to match the new
signature.
Extended test_sim_session_ordering.c's existing bug-1613 regression
tests with new plugin_begin_session()/plugin_end_session() spies
(call-count + session_active-window proof, same technique as the
existing sched_account_jobs spy). RED/GREEN independently verified:
temporarily disabled the two new call sites, confirmed both new
assertions fail (Expected 2 Was 0 / Expected 1 Was 0), restored,
reconfirmed green. Full scheduler suite: 51/52 green, the one failure
is the pre-existing unrelated bug-1560 sandbox flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
FilterRows routed ANY condition on job_status/host_status through
MatchesBitmaskCondition based purely on the field name, regardless of
the condition's actual operator. MatchesBitmaskCondition only
implements OpEQ/OpNE (the only two shapes buildIntCondition's in/notin
bitmask branch ever produces) and silently returns false for anything
else -- so xcecli's `-a` flag, which builds `{job_status:notnull:}`
(Operator: OpNOTNULL), matched zero rows unconditionally. `xcecli jobs
list -a` showed "no job found" even when jobs genuinely existed and
were correctly queryable through every other view.
FilterRows now requires the operator to actually be OpEQ/OpNE before
taking the bitmask path; any other operator on these fields correctly
falls through to the generic MatchesCondition evaluator, which already
implements OpNOTNULL/OpISNULL correctly -- no changes needed to either
evaluator itself, only the dispatch condition.
New query_test.go (no test file existed for this package before):
TestFilterRows_JobStatusNotNull_MatchesRealRows reproduces the bug
(independently confirmed RED: 0 of 3 rows matched) and
TestFilterRows_JobStatusBitmaskEQ_StillWorks locks in that the actual
bitmask shape used by every other status flag is unaffected. Live-
verified: `xcecli jobs list -a` now correctly shows jobs across every
status, immediately, no restart needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
New test_dc_success_notifies_job_update_hook_exactly_once
(test_dispatch_commit.c): a spy hook counting HOOK_AFTER_EVENT/
HOOK_DETAIL_JOB_UPDATE fires specifically -- the exact hook
apiserver_job_update_fun subscribes to in production. Asserts exactly
one fire per successful live dispatch.
Independently reproduced RED->GREEN: disabled the new notify call
with the test kept, rebuilt, confirmed FAIL (Expected 1 Was 0);
restored, rebuilt, confirmed PASS.
Also independently confirmed no double-WAL-append regression via two
existing tests whose fixtures genuinely wire persist.cluster with
persist_job_hook_fun registered: test_jobdispatch_receipt_ack_no_wal_append
(asserts exactly one WAL entry for a successful dispatch) and
test_handle_jobdispatch_matches_dispatch_a_job_notify_count -- both
pass unchanged. Full controller_unit_test: 1137/1137 (2 pre-existing
test_config_repl.c failures, confirmed identical on stock code via
git-stash -- the known overlayfs fdatasync errno=5 flake, bug-1561).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
dispatch_a_job_() mutates job.state/job.allocs directly then persists,
firing no hooks at all. apply_committed_entry()'s later apply of that
same WAL_JOB_DISPATCH entry sees state already matches the value it
just set moments earlier (under the same lock) and reports
JOB_TRANSITION_NOOP_DUPLICATE, silently skipping its own would-be
notify. Result: the apiserver is never told a job reached RUN -- xcecli
jobs list shows it stuck at PEND forever, no restart involved.
New persist_notify_job_updated_already_persisted(cluster_t*, job_t*)
(persist.c/persist.h), called from dispatch_a_job_()'s success branch
right after unlock(j). Mirrors apply_committed_entry()'s own identical
Hazard-1-guarded hook fire exactly -- same t_apply_seam_notifying
guard, same hook (HOOK_DETAIL_JOB_UPDATE's AFTER_EVENT), same detail --
without which persist_job_hook_fun (also subscribed to this hook)
would redundantly re-append the transition as a second, duplicate
WAL_JOB_UPDATE entry. The flag stays file-private to persist.c; this
is job.c's only sanctioned way to reach it.
record_job_start_fact()/record_job_finish_fact() needed no change --
their durable-append-only pattern (defer mutation+notify to
apply_committed_entry entirely) is already correct; only
dispatch_a_job_()'s broken hybrid (direct mutation, no notify at all)
was ever the problem.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
test_dispatch_commit.c's test_dc_success_appends_wal_job_dispatch_durably
(already drove the real dispatch_a_job_() production path through a
real WAL-write + persist_recover() restart cycle, previously only
checking recovered state) now also asserts the recovered job.allocs
matches what was dispatched -- host, resource name, value. This is
the primary regression test for the user's exact acceptance
criterion: "after recover the job should get the same as it's state
before shutdown."
Independently reproduced RED->GREEN (not just narrated): reverted the
production fix with the test extension kept, rebuilt, confirmed FAIL
at exactly this assertion; restored, rebuilt, confirmed PASS.
test_wal_job_dispatch_fact.c: added 3 new tests exercising the richer
payload directly on both apply_committed_entry and apply_wal_entry,
plus a full restart-replay variant. All 3 pre-existing tests (bare
wal_jobstate_t payloads, no trailing bytes) pass completely unchanged,
proving backward compatibility with any WAL_JOB_DISPATCH entry written
before this fix.
Full controller_unit_test suite independently re-run: 1136/1136 pass,
zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
dispatch_a_job_() set j->job.allocs in memory at dispatch-commit time
but only ever persisted {jobid, state} via persist_submit_job_fact() --
the allocation itself was never written to the WAL. On a full restart,
job.state correctly recovered to RUN (both WAL_JOB_DISPATCH and the
later WAL_JOB_START fact carry state=RUN) but job.allocs stayed
permanently NULL, since apply_job_update_transition() only ever
touches state. EXEC_HOST displayed blank forever for a job that was
genuinely running before shutdown.
No capnp schema change needed -- JobAlloc (xce_job_alloc_t) was
already a full top-level struct (the existing Job.allocs field) with
generated encode/decode functions already in the build. New:
capnp_encode_joballoc()/decode_joballoc() (persist.c, mirroring the
existing capnp_encode_worker()/restore_job() patterns) and
persist_submit_job_dispatch_fact() -- WAL_JOB_DISPATCH's payload is
now a wal_jobstate_t header plus trailing capnp-encoded JobAlloc
bytes. dispatch_a_job_() now persists j->job.allocs (the canonical,
freshly-constructed struct, read under the same write lock) instead
of just the state.
Both apply seams updated: apply_committed_entry() (primary's own boot
replay) gets one added branch in its existing WAL_JOB_START/FINISH
stamp-chain; apply_wal_entry() (standby catch-up / replicated-apply --
also required, a promoted standby would hit the identical bug) has
WAL_JOB_DISPATCH split into its own case, calling new
restore_job_dispatch_state(). A payload of exactly sizeof(wal_jobstate_t)
(any entry written before this fix) decodes no allocation and behaves
exactly as before -- full backward compatibility.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
modelJobToSubmit/freeJobSubmit (submit direction) and
cJobToModel/cJobArrayToModels (receive direction, both single-job and
array-job paths) now carry from_host across the CGO boundary.
rawSubmitData/rawJobToModel (the pure-Go, CGO-free mapping layer) gain
the field too, completing the round-trip so JobInfo.FromHost reflects
whatever the controller actually has -- which, as of the prior two
commits, is now always the client-asserted hostname. backend/handler.go
needs no change: its existing blind overwrite in OnJobUpdated is
correct by construction once every controller push genuinely carries
the right value.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
- src/schema/controller.capnp: JobSubmitData gains
fromHost @10: Text $C.mapname("from_host") -- auto-regenerates
controller.capnp.c/h via the existing meson capnpc-c custom_target.
- src/include/xce/types.h: job_submit_data_t gains char* from_host.
- src/lib/jobj.c: jobsub_init/jobsub_destroy (the hand-written,
ref-counted, content-deduped job-submit interning object -- not
auto-generated) thread from_host through the deep-copy, the dedup
identity comparison, and the destroy path.
Closes the wire-format half of the round-trip: from_host now survives
from the apiserver's submission all the way through the controller and
back on every subsequent job-state push, and rides along for free in
the existing WAL_JOB_SUBMIT persistence across a controller restart.
Did not need to touch job.c's build_job_submit_payloads() (bare
pointer borrow of the already-decoded submit struct) or any WAL
apply-path code (capnp_encode_job/decode_Job already generically
handle the whole xce_job_t including its full .submit sub-struct).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
CreateJob now does FromHost: req.FromHost, a straight passthrough of
whatever xcecli (or any other client) sends in JobSubmitRequest.FromHost,
instead of deriving it server-side via ctx.ClientIP().
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
JobSubmitRequest gains a FromHost field, populated in BuildJobRequest()
via os.Hostname(). Matches how traditional batch schedulers report the
submitting host, and sidesteps having the server derive it from the
connection (which apiserver was doing via ctx.ClientIP() -- an IP not
a hostname, and spoofable via X-Forwarded-For since this apiserver
never configures gin's TrustedProxies).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
num_run was declared in the host model but never set anywhere — the
controller's worker push carries no running-job count, so hosts always
showed 0 regardless of running jobs. Compute it live in the hosts list
handler by counting RUN jobs per exec host from store.Jobs — the same
source the jobs list uses, so the two stay consistent.
Verified: t1 with 5 RUN jobs shows num_run 5, matching the jobs list.
Co-Authored-By: Claude <noreply@anthropic.com>
OnHostConfigUpdated only applied config-sourced fields (MaxSlot,
HostModel, HostType, DisptWindow) onto a host whose name exact-matched
an entry already in the store. hosts.yaml commonly defines only a
wildcard "default" entry, so a dynamically-joined worker never
received its own config push and MaxSlot stayed at Go's zero-value
forever -- decoupled from the C-side scheduling path, which already
resolves the real value via worker.c's wildcard fallback.
The "default" pseudo-host's config is now cached and retroactively
backfilled onto any registered host with no explicit config of its
own, in either arrival order (config-before-worker or
worker-before-config). The -1 HOSTCONFIG_MXJ_AUTO_DETECT sentinel
(bug-1556) is resolved to the host's real NumCPUs when applied via
fallback, mirroring worker.c's own resolution, so the display shows
usable capacity rather than a bare -1.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
GetJobsInfo/GetJobByIDs validated the -o/-l field list against
models.JobSubmitRequest{} -- the job-submission payload -- instead of
models.JobInfo{}, the query/response model (the already-correct
pattern used elsewhere in this file at line 1131). Response-only
fields like pending_reason don't exist on JobSubmitRequest, so any
such field made the whole listing fail outright with "unsupported
query field", not just omit that column.
core_limit/mem_limit exist on JobSubmitRequest but not on JobInfo --
switching wholesale would have newly broken those two previously-valid
fields. Added CoreLimit/MemLimit to JobAdditionalFields, the existing
merge-in bucket already used for pids/pgids/interactive_job_url.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
ExtractErrorMessage()'s connection-refused branch returned the bare
OS error string verbatim. A CLI call right after apiserver startup
(before its listener is up) surfaced a raw "get the request failed:
[connection refused]" with no indication it's a transient startup
race, not a real failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
on_apiserver_join pushed configs first, then workers. The API server's
OnHostConfigUpdated only applies config-sourced fields (max_slot,
host_type/host_model) to hosts already in its store — so on a fresh
connect every host was skipped and the API showed max_slot 0 and t1
host_type "unknown". Swap the order so the worker dataset populates the
host entries before the config push lands.
Verified on t1/t2/t3: all hosts now report max_slot 8 and
host_type x86_64-linux-gnu; jobs still submit → RUN → DONE.
Co-Authored-By: Claude <noreply@anthropic.com>
Fixes test_commit_barrier_reply.c's test_jobdispatch_receipt_ack_no_wal_append
(owned by ha-commit-barrier-rollout), the last remaining assertion this
feature's earlier tasks left stale:
- WAL-count delta corrected from "must be zero" to lsn_before + 1 -- the
batch's successful dispatch now appends one WAL_JOB_DISPATCH fact
(task 3.1); mark_job_fail() was traced and confirmed to append
nothing via any path.
- The batch's terminal-failure job was replaced with a genuinely
retryable one (WORKER_NULL site, unregistered host, new job id)
rather than degrading the notify-count assertion to zero, which
would also have silently killed the test's own reply-before-notify
ordering pin (nothing left to order against). This keeps the test
meaningfully exercising notify_dispatch()'s wiring through
handle_jobdispatch().
Full regression, independently re-verified by review:
- Scheduler suite at documented baseline (51/52, 1 pre-existing
unrelated failure). Investigated and logged bug-1632, a new sibling
of the already-known bug-1598 stack-lifetime pattern in a different
test file -- confirmed pre-existing, untouched by this feature, and
non-deterministic.
- Controller suite clean: 1133 tests, only the 2 pre-existing bug-1561
failures.
- Sequential HA soak: PASS (10 rounds, 36 jobs, zero loss, all 3 nodes
led).
- Chaotic HA soak: PASS (10 rounds, committed_divergence=0
committed_job_loss=0).
This completes the scheduler-dispatch-commit spec: bug-1624
(resource-in-use round-boundary loss) and bug-1625 (scheduler discards
dispatch-failure notify) are both fixed and regression-tested.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
Adds test_handle_jobdispatch_terminal_no_notify_no_leak to
test_jobdispatch_no_double_notify.c, closing a gap neither existing
test fully covered: test_dispatch_fail_terminal.c (task 3.2) bypasses
handle_jobdispatch() entirely via dispatch_a_job_test(), so it can't
exercise the double-notify race at all; test_jobdispatch_no_double_notify.c
(task 4.1) only proves an aggregate 3-job-batch notify count, not a
direct per-job zero-notify proof.
The new test drives a single terminal-classified job through the real
handle_jobdispatch() entry point and asserts, in one run: JOB_STAT_EXIT
with allocs left NULL, a direct (non-aggregate) zero-count read from
the notify-capture spy proving no jobDispatchFail was produced for this
job, and (via task 3.2's canary-reference technique, armed before the
call under test) no reference over-release.
Test-only; job.c/handler.c are untouched. RED phase verified via an
actual revert-rebuild-rerun of handle_jobdispatch() to its pre-task-4.1
shape. Independently reviewed and approved -- the reviewer notably
declined to trust a relayed build report and re-ran verification
first-hand before approving.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
Adds test_rpc_failure_bound_resets_after_intervening_success to
test_dispatch_fail_retry.c, proving Requirement 4.3's literal claim
end-to-end through the real dispatch_a_job_test() entry point: a fail,
then a real success, then another fail resets the consecutive-failure
counter so the second failure is evaluated as failure #1, not #2.
The first draft of this test also asserted job.state/job.allocs revert
to PEND/NULL on that post-success failure -- this genuinely fails
against current code (none of dispatch_a_job_()'s four retryable
branches revert state/allocs when failing from an already-RUN job,
since they were only ever designed for a job whose every attempt starts
from PEND). A debug investigation, independently re-verified even more
exhaustively during review (full trace of every pendJobs-reinsertion
and controller-side state-mutation site), confirmed this scenario is
unreachable via any real caller today -- the scheduler's picker
permanently removes a job from its PEND pool the instant it dispatches
it, and nothing already-implemented ever re-adds a RUN job to it. Only
the not-yet-implemented scheduler-preemption spec could someday create
that path. Logged as bug-1631, deliberately deferred rather than fixed
now (no reachable caller to regression-test against); the two
over-reaching assertions were trimmed, keeping only the counter proof
Requirement 4.3 actually asks for. Independently reviewed and approved.
Scenarios 1 (retryable -> PEND-eligible) and 2 (bound exceeded -> EXIT)
were already fully covered by tasks 2.2's and 3.3's existing tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
Extends task 2.1's test_scheduler_dispatch_commit.c with two tests
proving resource-in-use accounting survives more than a single round
boundary between dispatch and worker confirmation, per task 5.1's
"several rounds" / "matching a same-round confirmation" language:
- test_dispatch_commit_survives_several_rounds_before_confirmation:
loops 5 round-boundary cycles, asserting resInUse after EVERY round,
not just once.
- test_dispatch_commit_multi_round_gap_matches_same_round_confirmation:
compares resInUse after 1 round vs after 5 rounds and asserts
equality -- a same-round confirmation is modeled as nrounds==1 since
design.md documents re-applying JOB_STAT_RUN to an already-RUN job as
a confirmed no-op, so resInUse depends only on job->state, never on
confirmation timing.
Test-only; scheduler.c is untouched. RED phase independently verified
by both implementer and reviewer via an actual revert-rebuild-rerun
cycle (4/5 tests fail without the task 2.1 fix). Independently reviewed
and approved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
dispatch_a_job_() (tasks 3.1-3.3) now owns notification entirely for
every outcome -- notify_dispatch() on a retryable failure, mark_job_fail()'s
own hook chain on a terminal failure, nothing on success. handle_jobdispatch()'s
loop still unconditionally called notify_dispatch() itself for every job
regardless, producing two conflicting outcome reports for any failed
dispatch. Removed that caller-side call; the loop is now a thin
dispatch-and-free loop with no notification logic of its own.
New test_jobdispatch_no_double_notify.c drives a 3-job batch (success,
retryable, terminal) through handle_jobdispatch() and confirms exactly
one notification fires (from the retryable job only), not four.
Independently reviewed and approved. Confirmed handle_jobdispatch()'s
sole caller never depended on its notification behavior (always returns
0 unconditionally). Flagged that test_commit_barrier_reply.c's
test_jobdispatch_receipt_ack_no_wal_append is now stale for a second,
independent reason (notify count, not just WAL count) -- both left for
task 5.4 per Requirement 5.3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
The four retryable-classified failure sites in dispatch_a_job_() (worker
lookup, worker null, worker not ready, RPC failure) now resolve instead
of just returning an error code: lock(j, RTTI_LOCK_WRITE) -> genuinely
classify via xce_classify_dispatch_fail() -> increment dispatch_fail_count
-> within XCE_DISPATCH_FAIL_RETRY_BOUND (3) resolves retryable (nothing
further under the lock), past it resolves terminal via mark_job_fail()
exactly like task 3.2's sites -> unlock(j) -> dereference(j) (closing
bug-1627's 4-site reference leak) -> outside the lock, notify_dispatch()
fires only on the retryable outcome. Each site's natural error code is
captured into a stable local before any intervening call, since errno
isn't guaranteed stable once lock()/ERROR()/free_launchinfo() sit between
the original failure and the eventual return.
handle_jobdispatch() is intentionally untouched -- it still unconditionally
calls notify_dispatch() itself for every outcome, so the retryable path
now temporarily double-notifies until task 4.1 removes that caller-side
call. Expected, not a defect.
New test_dispatch_fail_retry.c (4 tests, including a bounded-escalation
test driving 4 consecutive RPC failures on the same job: 3rd retries,
4th resolves EXIT) plus a fixed stale assertion in test_dispatch_commit.c
and additive, read-only notify-capture accessors in
test_commit_barrier_reply.c (the binary's sole notify() stub host).
Independently reviewed and approved, including explicit scrutiny of the
cross-file test accessor addition and the stale-assertion fix -- both
confirmed legitimate, not scope creep or regression-hiding.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
The two terminal-classified failure sites in dispatch_a_job_() (invalid
allocation, queue-exec-info lookup failure) now actually resolve the job
instead of leaving it stuck at PEND forever with a dangling reference to
nowhere. Each site: lock(j, RTTI_LOCK_WRITE) -> genuinely calls
xce_classify_dispatch_fail(site) and branches on the result (not a
hardcoded assumption) -> mark_job_fail() under the lock (stage=0 routes
apply_job_fail_transition() to JOB_STAT_EXIT) -> unlock(j) -> the existing
dereference(j) (kept exactly where it was, not duplicated) -> the existing
return code, unchanged. No notify_dispatch() added -- mark_job_fail()'s
own hook chain is the sole notification for a terminal outcome per
design.md; handle_jobdispatch()'s still-unconditional notify_dispatch()
call is untouched (task 4.1's job).
New test_dispatch_fail_terminal.c proves both sites resolve to
JOB_STAT_EXIT with allocs left NULL, using a canary-reference technique
to prove no reference over-release (job_t's real refcount has no test
accessor). Independently reviewed and approved; applied the reviewer's
one non-blocking fast-follow (arm the canary before the call under test,
not after, so it's an independent proof rather than relying on other
assertions to crash first) and re-verified with a clean incremental build.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
dispatch_a_job_()'s previously-unconditional job.allocs commit is now
gated on xce_startjob_req() actually succeeding (rc >= 0). In that same
successful branch, under a newly-acquired lock(j, RTTI_LOCK_WRITE) (a
real pre-existing locking gap this call site had -- it took no lock on
the job object itself before this), the job transitions to JOB_STAT_RUN,
dispatch_fail_count resets to 0, and a WAL_JOB_DISPATCH fact is appended
for durability. The failure path is otherwise unchanged -- classification
and resolution of the six failure sites is tasks 3.2/3.3's job, not this
one's.
Adds an XCE_TESTING-only dispatch_a_job_test() wrapper (dispatch_a_job_()
is static) and test_dispatch_commit.c, proving: state->RUN, allocs
committed correctly, counter resets, and durability survives a real
persist_recover() replay with no WAL_JOB_START present.
Independently reviewed and approved. Two pre-existing, out-of-boundary
findings surfaced and logged for later tasks: test_commit_barrier_reply.c's
test_jobdispatch_receipt_ack_no_wal_append will need updating by task 5.4
(Req 5.3 anticipates this), and bug-1429's hang was reconfirmed a 3rd time
(unrelated to this change).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
Adds a case for NotifyData_data_jobDispatchFail to sched_handle_notify()'s
switch, fixing bug-1625 -- the notify was previously silently discarded
(falling to default: -EINVAL), leaving job->accounts/job->allocs stale
forever on any dispatch failure. No classification logic here: the
controller has already decided PEND is correct before sending the notify;
this reuses the same XCE_SCHED_OP_JSTATE_CHANGE enqueue path the adjacent
schedExitJob/schedFailJob cases already use, and relies on core_events.c's
existing free-on-non-RUN branch to release the stale accounts/allocs.
Independently reviewed and approved: full scheduler suite passes (51/52,
one pre-existing unrelated flake -- bug-1560).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
sched_commit_decision()'s DISPATCH branch now sets job->state = JOB_STAT_RUN
in the same operation that already commits job->allocs/job->accounts, per
Requirement 1.1/1.2/1.3. sched_account_jobs()'s existing JOB_IS_RUN gate is
unmodified -- this closes bug-1624 by giving it the input it was missing: a
dispatched job now survives a round boundary in resource-in-use totals
regardless of how many rounds pass before worker confirmation arrives.
Regression test drives the real, unmodified sched_account_jobs() across a
simulated round boundary (resInUse reset, no confirmation event) and
independently reverting the one-line change reproduces bug-1624 exactly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add uint32_t dispatch_fail_count to the job_t wrapper struct (xce/types.h,
alongside submit_lsn, same in-memory-only/reset-on-failover precedent),
per Requirement 4.3. Data model only -- xce_job_t (wire type) untouched,
job.c untouched; increment/reset wiring and the bounded-retry threshold
check are tasks 3.1/3.3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New wal_entry_type WAL_JOB_DISPATCH=12, reusing wal_jobstate_t verbatim
(_pad=0, absolute-value semantics), wired into both apply_wal_entry()'s
and apply_committed_entry()'s existing job-state fall-through groups per
Requirement 2.4. Restart-replay test writes only WAL_JOB_SUBMIT+
WAL_JOB_DISPATCH to a real on-disk WAL and confirms job.state==RUN after
a fresh persist_recover(), independent of WAL_JOB_START.
Review found apply_wal_entry() has zero production call sites today
(persist_recover() and standby catch-up both route through
apply_committed_entry() post the 2026-07-10 ha_apply_single_entry
refactor) -- design.md's stale characterization corrected; the dual
case-label registration itself is unaffected and correct.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
