- bug-1872 (worker): "Failed to find any disk statistics in
/proc/diskstats" ERROR every 15 s on an NVMe-only host. The whole-disk
filter guessed from the name and skipped anything ending in a digit or
containing 'p' -- every nvmeXnY, mmcblkN, even sdp -- so io was never
reported. diskstats.h now counts a device only when
/sys/block/<name>/device exists (real disks; not partitions, loop, dm,
md, zram), and a host with no disk warns once.
- bug-1873 (worker): "Failed to get worker pid" WARN on job start. MainPID
reads 0 until systemd has forked the new transient unit; xcecd.c already
retries, so the miss is DEBUG and the final failure names the unit.
- bug-1874 (scheduler): deprecation WARN for the legacy xce_sched_fcfs on
every start, though builtin_fcfs is the picker. Registration still wraps
a job_elect_fn plugin but no longer warns; selecting one warns once.
- bug-1875 (apiserver): "Not running as root -- PAM authentication will not
work" with the default provider config_file; now only for provider pam.
- bug-1876 (controller): standbys logged "cluster_init_phase2 ... failed
(rc=-22)" WARN every 3 s while waiting for the primary's config push,
a state the function itself calls normal. It returns -EAGAIN for that
wait; callers log it at INFO, other failures still WARN.
- bug-1877 (controller): "WAL/snapshot stored unencrypted" warned without
looking. persist_path_encrypted() checks fscrypt (statx) and dm-crypt
(sysfs dm/uuid CRYPT-, through LVM slaves); the warning stays when the
storage is not shown to be encrypted (e.g. a plain /tmp).
Tests: worker diskstats_test (2 of 4 RED on the old filter with pc13v's
/proc/diskstats), controller test_phase2_cold_boot and
test_persist_encryption, scheduler picker tests, apiserver
TestRootWarning. Full ASAN build: all 113 suites pass (controller 319/0,
harness 141/0).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Regression from 63fd3d2a (bug-1870), found by the --config-upgrade HA soak
(force-elect: "submit on {ctrl-2,ctrl-1} failed", ~640 standby NACKs;
bisected: 725301dd passes, 88c4310e fails).
load_snapshot() seeded current_lsn = checkpoint_lsn, but current_lsn is
the NEXT lsn to write and the snapshot covers 1..checkpoint_lsn. The
off-by-one was always there, hidden because every checkpoint also wrote a
WAL_CHECKPOINT marker ABOVE the checkpoint, so the WAL scan moved
current_lsn past it. With the marker gone, a standby whose WAL held
nothing above its config-marker checkpoint (LSN 11) recovered with
current_lsn=11: its raft layer counted entry 11 as held (ae_already_have,
not written), and persist_wal_write_replicated() then refused entry 12 as
a gap (ae_append_persist_fail) on every resend -- stuck at 11 forever.
Both of load_snapshot's commit points now set current_lsn =
checkpoint_lsn + 1, matching the apply-side catch-up path (lsn + 1).
New test_recover_from_snapshot_without_wal_resumes_after_it (RED before:
current_lsn 3, not 4). Two snapshot-entity tests asserted the old
current_lsn == checkpoint_lsn and now expect +1. Controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
build_queues() only ran at boot, so a queue a live config upgrade added
had no runtime object in c->qmgr (and the scheduler/apiserver never heard
of it) until the controller restarted. The checkpoint-ordering half of
bug-1469 was fixed in 959b0cda; this is the remaining half.
- reconcile_config_queues(c, notify): creates every queue in the active
config that has no runtime object, leaves existing ones untouched,
returns the number created.
- config_activate_committed() calls it (with notify) after every
successful activation -- verified-marker and no-marker paths alike.
- persist_notify_queue_updated_no_wal(): fires HOOK_DETAIL_QUEUE_UPDATE
under the t_apply_seam_notifying guard, so the scheduler/apiserver are
told while no WAL_QUEUE_CTRL is appended -- config queues are
config-owned, built on every member from the committed config.
- build_queues() now delegates to it without notify, so boot and
activation are idempotent in either order.
New test_config_queue_reconcile.c (RED before: the added queue was never
created): creates the added queue, leaves the existing object untouched,
notifies once, and a second reconcile/build_queues adds nothing.
Controller suite passes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
persist_checkpoint() and persist_checkpoint_marker_triggered() wrote a
local-only WAL_CHECKPOINT marker that consumed an LSN of the replicated log
(current_lsn++) but was never replicated. A checkpoint taken by one member
only -- the primary's periodic max-wal-records checkpoint, or the one before
a snapshot send -- left a hole no standby had: the leader's next entry
carried prev_log_index = the marker's LSN, logOk failed on every standby
forever, commits timed out and writes stalled ("could not submit 110 jobs"
in the --force-snapshot soak; reproduced on the pre-bug-1867 baseline). Even
the config-marker checkpoint, taken on every member, stamped the marker with
each member's own current_lsn at apply time, which need not agree.
Nothing reads the marker except to skip it, so checkpoints simply stop
writing it. The recovery, streamer and bug-1475 log_len skip paths stay, for
WALs written by older binaries; persist_test_write_legacy_checkpoint_marker()
(XCE_TESTING only) reproduces such a WAL for the tests that cover them.
- new test_bug1870_leader_only_checkpoint_leaves_no_log_hole (RED before):
a leader-only checkpoint takes no LSN and a standby that never
checkpointed catches up past it via the real streamer/receive path.
- test_checkpoint_writes_marker_to_wal inverted to ..._writes_no_marker_...
- bug-1475 x2 and the apply-correctness trailing-marker test now build
their legacy marker explicitly.
Controller suite 1234/0. HA soaks: --force-snapshot PASS (113 jobs, real
snapshot install -- also live-verifies bug-1867), --config-upgrade-wal-trim
PASS, --sequential 6 rounds PASS (SIMVERIFY 7 files, 0 mismatches).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The capnp Job struct -- the checkpoint/entity-snapshot form of xce_job_t --
had no exit-info field, so capnp_encode_job() never wrote job.exit_info
and a job restored from a snapshot (instead of WAL replay) came back with
exit_info NULL, for DONE/EXIT and failed jobs alike. bug-1689/bug-1754
made the WAL facts carry the code; anything older than the last
checkpoint's WAL trim still lost it.
- controller.capnp: struct JobExitInfo $C.mapname("job_exitinfo_t")
{ exitcode }, and Job.exitInfo @12 $C.mapname("exit_info"). The
generator now emits encode/decode/free for it inside encode_Job /
decode_Job / free_Job; restore_job()'s create_job() already takes
exit_info via MOVE_PTR. Null stays null (no fabricated exit 0). A new
ordinal, so older snapshots/recordings decode with exit_info NULL.
New test_snapent_checkpoint_exit_info_roundtrip (RED before): exit codes
0 and 137 and a NULL survive both the raw checkpoint frames and a real
persist_load_snapshot restore. Full /xce-build all (asan): 111 meson
suites, 0 failures; all Go packages pass.
Not live-verified: the --force-snapshot HA soak cannot reach its snapshot
step because of a pre-existing checkpoint-marker replication stall,
reproduced on the baseline without this change (bug-1870).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The FAIL half of bug-1747: apply_job_fail_transition() stamps
finish_time = time(NULL) at APPLY time, and WAL_JOB_FAIL carried no time,
so a failed job's finish_time changed on every restart replay / standby
catch-up, and a worker-reported failure found on rejoin could never keep
the worker's own time.
- WAL_JOB_FAIL's trailer grows to int32_t code + uint64_t finish_time
(seconds). Decoded by length: 16 B (pre-bug-1754), 20 B (code only,
bug-1754) and 28 B (code + time) all apply as before for what they carry.
- record_job_fail_fact_at(c, job, failinfo, finish_time_ms), the fail-side
sibling of record_job_finish_fact_at(): 0 = receipt time. on_job_fail()
uses receipt time; resync's finishedFail branch passes the worker-
reported finish_time_ms (Req 4.2 now holds for failures too).
- Both apply seams adopt the wire time after the transition.
4 new tests in test_wal_job_fail_fact.c (3 RED before the fix) and the
resync round-trip now asserts the worker-reported failure time (RED
before). Controller suite 1232/0. Factstream HA soak (2 rounds, 60 jobs)
PASS.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A worker reports every nonzero job exit as a FAIL (xcecd_job_finished ->
xcecd_notify_job_fail(jobid, stage, exit_code)), not a FINISH. The deferred
WAL_JOB_FAIL fact carried only {jobid, STAGE_FLAG|stage} and no apply seam
touched job.exit_info, so exit_info stayed NULL for every failed job, live
and after replay, and HOOK_DETAIL_JOB_FAIL subscribers saw code 0.
- persist_submit_job_fail_fact(): header + int32_t code trailer, used by
record_job_fail_fact(). Old entries (no trailer) decode as before.
- apply_job_fail_exit_code(): records the code in exit_info for pre-exit
stages only; a POST_EXEC code is the script's and leaves exit_info alone.
Kept out of apply_job_fail_transition() so mark_job_fail()'s internal
-errno failures (not replayed with a code) stay live/replay consistent.
- Both apply seams adopt the code. apply_wal_entry()'s WAL_JOB_FAIL case
also stops assigning the wire state 0 for STAGE_FLAG entries and runs the
real transition, as apply_committed_entry() already does.
New test_wal_job_fail_fact.c (5 tests, 3 RED before the fix); the two
test_worker_resync.c assertions that pinned exit_info == NULL now assert the
worker-reported code. Controller suite: 1228 tests, 0 failures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A failing TEST_ASSERT longjmps out and skips the fixture teardown, leaving
the commit engine's sweep thread and persist's fsync_worker asleep on a
static struct. The next test re-inits it underneath them, corrupting the
condvar, and a later teardown blocks in pthread_cond_destroy/join until
the 300s meson timeout. The failure line itself was lost because stdout
was fully buffered.
- controller_test: line-buffer stdout; tearDown() reaps what a failed test
leaked via new XCE_TESTING registries (persist_test_reap_unstopped, then
commit_test_reap_unfinalized_engines -- persist_stop before engine fini).
Clean run reaps 0; an injected failure now fails in 2s instead of hanging.
- persist: flusher_flush_once reports only its own durable advance, so
fsync_worker no longer re-fires the durable callback for an advance a
concurrent persist_flush made (flaky "must not re-fire" test). Regression
test test_flusher_pass_does_not_claim_concurrent_flush_advance.
- tests: jap observer is atomic and waits (bounded) for a continuation that
may resolve on the flusher thread; pump catch-up tests wait for trailing
replies instead of a single scan.
- config_repl: upgrade decode temp file via mkstemp instead of a
pthread_self()-derived /tmp name (same value across processes, opened
without O_EXCL -- collisions and a symlink target).
Verified: controller_unit_test 1222/1222; 24 parallel runs (6x4), no
hangs or ASAN errors.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Worker-rejoin-resync sets XCE_WORKER_RECONCILING and pushes it to the
scheduler, which correctly freezes that worker's resInUse mid-reconciliation
(bug-1799/1801). But resync_resolve_round() only cleared the bit on the
controller's own worker_t, never notifying the scheduler -- so once a
rejoin round resolved, the scheduler's copy stayed RECONCILING forever,
permanently freezing that worker's resource accounting even after every
job on it finished. Live-observed via a replay capture: a 22-slot worker
stuck at used=22.00 for the rest of the run with zero RUN jobs left.
Added persist_notify_worker_status_updated() (mirrors bug-1679's job-side
persist_notify_job_updated_already_persisted()) and call it from
resync_resolve_round() so HOOK_DETAIL_WORKER_UPDATE fires without a
spurious WAL append. TDD: new spy-hook regression test, RED confirmed
against the unfixed code, GREEN after the fix. Full controller_unit_test
suite: 1220 Tests, 0 Failures, 0 Ignored.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*,
xce_job_state_entry_t*) to job.c: eagerly sets job->job.state to match
the worker's reported outcome (mirroring dispatch_a_job_'s inline
mutation, minus allocation fields -- there are none to set, job->job.allocs
stays NULL per Requirement 4.8's documented scope). Calls the existing
persist_submit_job_dispatch_fact(..., alloc=NULL) directly, then leaves
the post-unlock persist_notify_job_updated_already_persisted() call to
the caller, same convention as task 2.1.
Fixes bug-1746 along the way: capnp_encode_joballoc() (persist.c, part
of bug-1678's original fix, predates this feature) unconditionally
dereferenced a NULL alloc -- never triggered before since
dispatch_a_job_ (the only prior caller) always passes non-NULL. Fixed
with a minimal guard producing a true zero-length buffer, matching
apply_committed_entry's alloc_len > 0 decode gate exactly. This
directly contradicted design.md's "tested, backward-compatible on both
seams" claim (true only for decode, never actually exercised for
encode) -- corrected in 4 places.
2 tests: applies-running-no-allocs-and-notifies, and a real restart-
replay round trip (write to real disk, fresh persist_recover(), confirm
jobid/host/state/NULL-allocs survive) -- proving this task's own
"durably survives a restart replay" completion criterion literally, not
just the in-memory mutation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Follow-up to ebe691c3 (HOOK_DETAIL_JOB_START/JOB_EXIT), same gap for job
failures: apply_committed_entry()'s WAL_JOB_FAIL case only ever fired
the generic HOOK_DETAIL_JOB_UPDATE, leaving sched_job_fail_fun (sched.c)
and apiserver_job_fail_fun (apiserver.c) dead -- the apiserver's
OnJobFinished was starved of data for job failures specifically (the
prior fix only closed this for clean finishes).
Fires HOOK_DETAIL_JOB_FAIL only for the WAL_JOB_FAIL_STAGE_FLAG branch,
which already constructs a real job_failinfo_t -- the legacy _pad==0
absolute-overwrite branch has no fail-specific data and must not
fabricate one, mirroring how the WAL_JOB_START fix only fires when
pidinfo was genuinely decoded. Unlike WAL_JOB_START, no gate-widening
was needed: apply_job_fail_transition() always returns
JOB_TRANSITION_APPLIED for a genuine new fail transition (confirmed by
reading it), so the existing result-gated early return already covers
this correctly.
TDD: RED confirmed (fix reverted, test failed "Expected 2 Was 1"),
GREEN after restoring it. Full regression: 1220 tests, only the known
bug-1561 flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
apply_committed_entry() (persist.c), the single seam that applies every
committed WAL entry, only ever fired the generic HOOK_DETAIL_JOB_UPDATE
for job-state changes -- HOOK_DETAIL_JOB_START/JOB_EXIT were
permanently dead code, since job.c only ever fires them at
HOOK_BEFORE_EVENT (for WAL-append purposes). This left sched_job_start_
fun/sched_job_exit_fun (sched.c) and apiserver_job_start_fun/
apiserver_job_exit_fun (apiserver.c) unreachable, and the Go apiserver's
already-wired OnJobFinished handler permanently starved of data -- a
finished job's state could stay stuck (e.g. showing RUN) in the
scheduler's own view and the apiserver never learned a job finished.
The wire format was already enriched by bug-1697/bug-1689 with the
pidinfo/exit_info data these hooks need, so this closes the gap rather
than requiring new wire work. WAL_JOB_FINISH now fires
HOOK_DETAIL_JOB_EXIT; WAL_JOB_START now fires HOOK_DETAIL_JOB_START,
gated on a call-scoped decode flag rather than the transition result
(WAL_JOB_START applies as a no-op duplicate for every real job in
production per bug-1697, so gating on the transition result would make
it dead code again). WAL_JOB_FAIL -> HOOK_DETAIL_JOB_FAIL has the same
gap and is left as a follow-up.
Also includes this investigation's supporting changes: a DEBUG3 log-
retention bump (main.c, both controller and scheduler) so race-condition
evidence survives a live burst without rotating out, and DEBUG3-gated
diagnostic trace logging at the two points a job-update notification
can fire or silently drop (job.c, sched.c).
Live-verified end to end on a real deployed cluster: scheduler-side
sched_job_state_change now fires for RUN and EXIT states, and the
apiserver's OnJobFinished -- previously never invoked in production --
now correctly updates job status on completion.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019QxaSzHRB7pmYFp4cVnwkA
dispatch_a_job_() synchronously sets state=JOB_STAT_RUN at dispatch time
(scheduler-dispatch-commit task 3.1's deliberate no-worker-ack-wait
semantic). By the time the worker's later WAL_JOB_START fact commits,
apply_job_update_transition() sees state already == RUN and returns
NOOP_DUPLICATE instead of APPLIED -- so bug-1687's start_time/pidinfo
decode/adopt block, gated on result==APPLIED, could never run for any
job. start_time (and consequently run_time) stayed 0 forever. Same bug
class already fixed for WAL_JOB_RUSAGE/CKPT/PREASON in this identical
apply function; WAL_JOB_START was missed because bug-1687 predated the
dispatch-time RUN semantic change.
Moves start_time to dispatch-owned: dispatch_a_job_() now stamps
job.start_time alongside state=RUN, under the same write lock.
WAL_JOB_DISPATCH gains a flag-gated (WAL_JOB_DISPATCH_HAS_START_TIME_FLAG)
start_time field so it survives replay/restart; both apply seams
(apply_committed_entry, apply_wal_entry) and restore_job_dispatch_state
decode it when present, falling through to the old alloc-only shape
otherwise. WAL_JOB_START's own now-permanently-dead start_time write is
removed; pidinfo adoption made unconditional to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Last of the 9 confirmed findings from the /xce-audit-invariants controller
pass. checkpoint_task_fn was registered directly as a delaymgr entry and
ran its full blocking body inline: checkpoint marker pwrite+fdatasync,
write_snapshot()'s genuinely unbounded per-entity walk (scales with
cluster state size) held under apply_pump_lock, WAL trim, and snapshot
rotation/unlink -- all on the same thread that drives heartbeats/election
timeouts. Unlike every other delaymgr operation (snapshot send,
config-push, reconnect, catch-up), checkpoint had no offload mechanism
at all -- the one gap left from the raft-drive-async hardening pass.
Design: full offload via the SlowLane (slowlane_submit), default ON --
NOT the vtask pool. persist_apply_pump_run() already holds apply_pump_lock
across its own per-entry walk while running ON the vtask pool
(VTASK_LIVE_BLOCK); putting checkpoint's unbounded, lock-held
write_snapshot() on that same pool risked starving it, exactly what
ha-async-io.md's own thread-role table warns against. The SlowLane is
the actual established pattern for this I/O class (push_config_transport_only,
the snapshot-send path, catchup_lane_fn, demotion_lane_fn all dispatch
there) -- and persist_checkpoint() already runs on the slow lane in
production TODAY via ha_snapshot_do_send() -> snapshot_lane_fn, so
apply_pump_lock's acquisition from a slow-lane thread is proven-safe,
exercised daily, not new surface. Unlike catchup/demotion (which stay
flag-gated OFF for an unrelated, still-not-root-caused hang, bug-1429),
checkpoint's new flag defaults ON given this reuses an already-proven
code path. Inline-fallback + WARN only on a genuine slowlane_submit
failure (not -EAGAIN/-EEXIST, both non-failures) -- never silently drops
a scheduled checkpoint.
New test_persist_checkpoint_offload.c (3 tests). RED-phase forced the
pre-fix always-inline path: the suite never printed a single result and
had to be killed, proving the test genuinely detects the violation, not
a tautology. GREEN: all 3 pass. Full suite: 1184 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake
(exact signature match). No live HA soak was run for this specific fix
in isolation (explicitly scoped out, not hidden) -- the mandatory
live-soak proof for this whole audit campaign covers it next.
This closes the last of the 9 confirmed findings from the controller
audit (bug-1684, bug-1687/1689/1690/1691/1692, bug-1693, bug-1694).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Last of the five field-parity gaps from the original audit (bug-1678,
bug-1687, bug-1689, bug-1690, bug-1691). set_preason()/clear_preason()
set job.pend_reasons (host/resource-based pending-job explanations), but
persist_job_hook_fun's default: arm only ever captured {jobid, state}.
Neither apply seam restored pend_reasons after a restart.
Reuses the ALREADY-EXISTING JobPendReason/HostBasedReason/ResourceBasedReason
capnp structs -- no schema change needed, unlike bug-1690's WAL_JOB_RUSAGE.
Safety-checked against bug-1690's lesson (RTTI/makeString()-backed fields
need merge-not-replace, not free-and-replace): pend_reasons is plain
malloc/strdup-backed throughout (erase_job_preason is FREEUP()-only,
add_host_preason/add_res_preason use strdup()/calloc()), so a
straightforward free-and-replace is safe here, matching bug-1678/1687's
pattern instead.
New wrinkle beyond the four prior fixes: pend_reasons has a genuine CLEAR
operation, and "trailing bytes present or absent" can't distinguish a
legitimate clear from an old-format payload -- added an explicit presence
byte (0=cleared/adopt NULL, 1=capnp bytes follow), verified by two
dedicated "clear survives restart" tests.
11 new tests in test_wal_job_preason_fact.c; RED/GREEN independently
verified (7 genuine pre-fix failures, one test's own wrong premise about
capnp's garbage-byte behavior caught and corrected during RED/GREEN
iteration). Full controller suite: 1178 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake. No ASAN findings.
This closes the field-complete apply parity invariant class the
xce-audit-invariants controller dimension 7 was added for.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678/1687/1689/1690: on_job_ckpt() sets job.ckptinfo
(a plain char*, xstring-backed via makeString()) but is purely hook-driven
-- persist_job_hook_fun's default: arm only ever captured {jobid, state}
for WAL_JOB_CKPT. Neither apply seam restored ckptinfo after a restart.
Simplest of the five field-parity fixes: ckptinfo is a bare string, so no
capnp struct is needed -- mirrors WAL_QUEUE_CTRL's existing NUL-terminated-
string-after-header wire shape instead. Adoption is unconditional (not
gated on a state transition), matching bug-1690's precedent: a checkpoint
report never changes job.state. restore_job_ckpt_state() releases the
prior value via dereference() (the correct RTTI verb for an xstring), not
free().
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension) -- the last of five sibling gaps found together
(bug-1678, bug-1687, bug-1689, bug-1690, this one). 7 new tests in
test_wal_job_ckpt_fact.c; RED/GREEN independently verified (5 genuine
pre-fix failures, 2 backward-compat tests correctly passed unchanged).
Full controller suite: 1167 tests, 2 failures -- both the pre-existing,
unrelated bug-1561 overlayfs sandbox flake. No ASAN findings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678/1687/1689: the WAL fact for WAL_JOB_RUSAGE only
carried {jobid, state}, so job.run_usage (client-reported per-host
CPU/mem usage) was never restored on either apply seam after a restart.
Unlike the prior three, xce_job_runusage_t/xce_host_runusage_t had no
existing Cap'n Proto mapping -- a genuine new schema addition
(HostRunusage/JobRunusage in controller.capnp, mirroring JobPendReason's
List(NestedStruct)+$C.maplistcount style), verified against the real
struct definitions in types.h, codegen confirmed via nm on the built
binary rather than assumed.
Two deliberate deviations from the established precedent, found by
reading the real code rather than copying the pattern blindly:
1. Adoption is unconditional, not gated on a state transition -- a
rusage report almost never changes job.state (repeat reports keep
state at RUN), so gating on JOB_TRANSITION_APPLIED like START/FINISH/
DISPATCH do would make the fix dead code for the realistic case.
2. Merge, not free-and-replace. update_host_runusage() builds run_usage
with host_name sourced from makeString() -- an RTTI-managed pointer,
not malloc() -- and the primary's own live apply pump re-applies every
fact it just wrote. A naive free_JobRunusage_ptr()-then-adopt (matching
the prior three fixes exactly) would free() a non-malloc'd pointer on
the very first live rusage report for a new host -- guaranteed heap
corruption. Fixed with merge_job_runusage(): existing hosts get
scalar-only in-place updates (host_name pointer never touched), only
genuinely new hosts adopt their freshly-decoded (safe, malloc'd)
host_name. Proved via a real makeString()-backed test asserting
pointer identity survives, not just value equality. job_destroy()
(src/lib/jobj.c) was found to have this identical latent mismatch,
pre-existing and independent of this fix -- flagged, not fixed here.
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension). 8 new tests in test_wal_job_rusage_fact.c; RED/GREEN
independently verified. Full controller suite: 1160 tests, 2 failures --
both the pre-existing, unrelated bug-1561 overlayfs sandbox flake. No
ASAN/heap-corruption findings. Backward compatibility confirmed for
old-format bare-payload entries.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678/bug-1687: record_job_finish_fact() wrote only
{jobid, state} for WAL_JOB_FINISH. Both apply seams either fabricated
finish_time via time(NULL) or touched nothing beyond state.
Worse than "lost on restart": apply_job_finish_transition() -- the only
function that ever set job.exit_info -- is dead code on the real
production apply path. The raft-apply-decoupling redesign routes actual
mutation through the leaner apply_job_update_transition() instead, so
exit_info was never populated at all, restart or not, before this fix.
job_exitinfo_t ({int exitcode;}) has no existing capnp mapping, so this
uses a small native (non-capnp) trailing payload -- wal_jobstate_t header
+ uint64_t finish_time + int32_t exit_code -- mirroring wal_jobstate_t's
own native-struct style rather than inventing a capnp wrapper for one
field. run_time is never carried on the wire, always re-derived as
finish_time - start_time at apply time. Both apply seams updated;
old-format payloads (exactly sizeof(wal_jobstate_t)) decode unchanged
(fabricated finish_time, NULL exit_info) -- full backward compatibility.
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension). 6 new tests in test_wal_job_finish_fact.c; RED/GREEN
independently verified (4 genuine pre-fix failures with real
expected-vs-fabricated mismatches, 2 backward-compat tests correctly
passed both before and after). Full controller suite: 1152 tests, 2
failures -- both the pre-existing, unrelated bug-1561 overlayfs sandbox
flake (confirmed flaky across repeated re-runs of the same binary).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
Same class as bug-1678: apply_job_start_transition() sets the real
job->job.pidinfo (worker-reported PIDs/PGIDs/exdata) and start_time, but
the WAL fact for WAL_JOB_START only carried the shared wal_jobstate_t
{jobid, state} header -- both apply seams (apply_committed_entry() for
primary boot replay, apply_wal_entry() for standby catch-up) restored
state only. pidinfo stayed permanently NULL after a restart; start_time
was fabricated with time(NULL) at replay instead of being carried,
drifting from the true value.
Mirrors bug-1678's exact fix shape: capnp_encode_pidinfo()/decode_pidinfo()
(persist.c, using the pre-existing JobPIDInfo capnp struct -- no schema
change needed) serialize job_pidinfo_t into the WAL payload, trailing a
new uint64_t start_time field appended after wal_jobstate_t.
persist_submit_job_start_fact() replaces the old state-only
persist_submit_job_fact() call for this type. Both apply seams updated;
a payload of exactly sizeof(wal_jobstate_t) (any entry written before
this fix) decodes with pidinfo left NULL and start_time fabricated via
time(NULL), exactly matching pre-fix behavior -- full backward
compatibility, verified by every pre-existing WAL_JOB_START test using
the old bare-struct shape.
Also fixed a genuine, unrelated deadlock this change's larger payload
exposed in test_commit_barrier_reply.c: cbrp_count_jobstate_entries_ex()
matched WAL entries by exact payload size (==), so once WAL_JOB_START
legitimately grew it silently undercounted to 0 and a downstream test
spun forever polling an uncounted target. Fixed the match to >= (jobid/
state always live at a fixed leading offset) -- bisected via /proc/*/wchan
futex inspection (no ptrace in this sandbox), confirmed as the actual
cause, not the teardown-ordering red herring first suspected.
Found by an /xce-audit-invariants controller pass (field-complete apply
parity dimension, added after bug-1678). 6 new tests in
test_wal_job_start_fact.c; RED/GREEN independently verified (4 tests
genuinely failed pre-fix with exact expected-vs-fabricated value
mismatches, 2 backward-compat tests correctly passed both before and
after). Full controller suite: 1146 tests, 2 failures -- both the
pre-existing, unrelated bug-1561 overlayfs sandbox flake.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
dispatch_a_job_() mutates job.state/job.allocs directly then persists,
firing no hooks at all. apply_committed_entry()'s later apply of that
same WAL_JOB_DISPATCH entry sees state already matches the value it
just set moments earlier (under the same lock) and reports
JOB_TRANSITION_NOOP_DUPLICATE, silently skipping its own would-be
notify. Result: the apiserver is never told a job reached RUN -- xcecli
jobs list shows it stuck at PEND forever, no restart involved.
New persist_notify_job_updated_already_persisted(cluster_t*, job_t*)
(persist.c/persist.h), called from dispatch_a_job_()'s success branch
right after unlock(j). Mirrors apply_committed_entry()'s own identical
Hazard-1-guarded hook fire exactly -- same t_apply_seam_notifying
guard, same hook (HOOK_DETAIL_JOB_UPDATE's AFTER_EVENT), same detail --
without which persist_job_hook_fun (also subscribed to this hook)
would redundantly re-append the transition as a second, duplicate
WAL_JOB_UPDATE entry. The flag stays file-private to persist.c; this
is job.c's only sanctioned way to reach it.
record_job_start_fact()/record_job_finish_fact() needed no change --
their durable-append-only pattern (defer mutation+notify to
apply_committed_entry entirely) is already correct; only
dispatch_a_job_()'s broken hybrid (direct mutation, no notify at all)
was ever the problem.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
dispatch_a_job_() set j->job.allocs in memory at dispatch-commit time
but only ever persisted {jobid, state} via persist_submit_job_fact() --
the allocation itself was never written to the WAL. On a full restart,
job.state correctly recovered to RUN (both WAL_JOB_DISPATCH and the
later WAL_JOB_START fact carry state=RUN) but job.allocs stayed
permanently NULL, since apply_job_update_transition() only ever
touches state. EXEC_HOST displayed blank forever for a job that was
genuinely running before shutdown.
No capnp schema change needed -- JobAlloc (xce_job_alloc_t) was
already a full top-level struct (the existing Job.allocs field) with
generated encode/decode functions already in the build. New:
capnp_encode_joballoc()/decode_joballoc() (persist.c, mirroring the
existing capnp_encode_worker()/restore_job() patterns) and
persist_submit_job_dispatch_fact() -- WAL_JOB_DISPATCH's payload is
now a wal_jobstate_t header plus trailing capnp-encoded JobAlloc
bytes. dispatch_a_job_() now persists j->job.allocs (the canonical,
freshly-constructed struct, read under the same write lock) instead
of just the state.
Both apply seams updated: apply_committed_entry() (primary's own boot
replay) gets one added branch in its existing WAL_JOB_START/FINISH
stamp-chain; apply_wal_entry() (standby catch-up / replicated-apply --
also required, a promoted standby would hit the identical bug) has
WAL_JOB_DISPATCH split into its own case, calling new
restore_job_dispatch_state(). A payload of exactly sizeof(wal_jobstate_t)
(any entry written before this fix) decodes no allocation and behaves
exactly as before -- full backward compatibility.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
New wal_entry_type WAL_JOB_DISPATCH=12, reusing wal_jobstate_t verbatim
(_pad=0, absolute-value semantics), wired into both apply_wal_entry()'s
and apply_committed_entry()'s existing job-state fall-through groups per
Requirement 2.4. Restart-replay test writes only WAL_JOB_SUBMIT+
WAL_JOB_DISPATCH to a real on-disk WAL and confirms job.state==RUN after
a fresh persist_recover(), independent of WAL_JOB_START.
Review found apply_wal_entry() has zero production call sites today
(persist_recover() and standby catch-up both route through
apply_committed_entry() post the 2026-07-10 ha_apply_single_entry
refactor) -- design.md's stale characterization corrected; the dual
case-label registration itself is unaffected and correct.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apply_committed_entry()'s shared WAL_JOB_START/FINISH/STOP/RESUME/RUSAGE/
CKPT/PREASON dispatch case only ever called the generic, state-only
apply_job_update_transition(j, state) -- start_time/finish_time/run_time
were never touched, so `xcecli joblist` always showed StartTime=0 for
RUN/DONE/EXIT jobs and FinishTime=0 for DONE jobs (EXIT jobs got a
finish_time only via the separate, richer WAL_JOB_FAIL dispatch path).
Fix is additive: stamp time(NULL) keyed on wal_type right after
apply_job_update_transition() succeeds, still under the same
lock(j, RTTI_LOCK_WRITE) -- WAL_JOB_START sets start_time, WAL_JOB_FINISH
sets finish_time and derives run_time. No change to
apply_job_update_transition()'s signature or the dispatch structure.
Add test_ace_job_start_sets_start_time and
test_ace_job_finish_sets_finish_time_and_run_time to
test_apply_committed_entry.c, following the existing tace_seed_job
fixture pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
persist_flush()'s synchronous fdatasync path advanced durable_lsn on
success but never called on_durable_advance() -- only the async
fsync_worker did. This gap was previously masked by bug-1563's own
busy-spin (a fast accidental repoll covered for the missing
notification); fixing that spin unmasked it, causing
test_durable_majority_commit_end_to_end to fail deterministically
(3/3 runs).
Fixed by firing on_durable_advance() from persist_flush() too,
whenever it genuinely advances durable_lsn. Fixed via TDD (RED test
before the fix, GREEN after) and re-validated with a fresh 40-round
sequential + chaotic HA soak (both pass cleanly, zero fdatasync
errors, zero divergence/loss).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bug-1562: ha_on_primary_ctrl_fd_rdhup's 5.10b debounce branch (ha.c) returned
without re-arming the liveness watchdog when a debounced RDHUP landed after
the in-flight PreVote round had already concluded (e.g. a lone cold-booting
node whose first round loses in milliseconds with no peers up yet). That was
the last timer ever scheduled -- the node froze permanently in CTRL_STANDBY,
so a 3-node cluster with a long inter-node start stagger could never elect a
primary. Root-caused via a live local reproduction matching the reporting
user's exact timing (byte-for-byte match of their hung ctrl-1.log), fixed by
re-arming the watchdog in that branch (mirroring the function's own 5.2
branch), and confirmed end-to-end: the same scenario now reaches CTRL_PRIMARY
and full 3-node config convergence.
bug-1563: fsync_worker's retry loop (persist.c) only backs off when no work
is pending, but flusher_work_pending_locked() can't distinguish "fresh work
just landed" from "still stuck from the last failed fdatasync" -- a
sustained sync failure left durable_lsn permanently behind tip, so the loop
skipped its wait every iteration and spun fdatasync as fast as the CPU/lock
allowed (a 40-round chaotic /xce-test-ha soak hit 300k+ failed attempts on
one node in under a minute). Fixed by waiting out the normal wal_sync_ms
cadence when a flush was pending but made no progress, leaving the
legitimate idle and just-succeeded fast paths untouched.
Both fixed via TDD (RED test before the fix, GREEN after) and live-validated
by re-running 40-round sequential and chaotic HA soaks end-to-end: both pass
cleanly, and the chaotic run's excluded fdatasync-error count dropped from
304,689 to 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bundles the fixes validated this session across ha.c/persist.c/worker.c:
- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
closed with a recovery-complete gate that closes at process-lifecycle
start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
check mirroring the existing job-count check (count_jobmgr_entries and
count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
applies mark_worker_unavail()'s mutations via a shared
apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
fixed for promoted vote endpoints via a shared votes_idx_for_peer()
index-scale converter.
Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A restarted standby could get permanently stuck unable to catch up,
with the primary re-streaming the identical doomed AppendEntries range
every heartbeat with zero progress -- discovered via a live 3-node
config-upgrade-snapshot soak. Liveness-only; no safety invariant
(Election Safety, Leader Completeness, State Machine Safety, Commit
Durability) was ever violated.
Three-layer root cause, all now closed:
1. ha_stream_wal_to_standby (config_repl.c) used to skip advancing its
prev_streamed_lsn/prev_streamed_term tracker across WAL_CHECKPOINT
entries (a local-only, never-replicated marker that still consumes
a real LSN slot). Now advances across them too, while still never
shipping a catchup entry for the checkpoint itself.
2. ha_apply_term_entry_via_library (ha.c) now JIT-resyncs
ha->raft.log_len from persist_durable_floor() before consuming an
AE, to correct for persist_write_checkpoint_marker advancing
current_lsn without going through raft_log_append/record_entry_term
(pre-existing gap, bug-1466, not itself fixed here).
3. The actual root cause, found via real instrumentation after layers
1-2 alone still left a regression test red: record_entry_term's
array-growth realloc zero-filled new slots, silently reclassifying
a checkpoint's deliberately-never-recorded gap slot from "absent"
to "present, term=0/WAL_LEGACY_TERM" the instant any later LSN got
recorded -- a time-dependent read where a permanent fact was
assumed. Fixed with a TERM_INDEX_UNSET (UINT64_MAX) sentinel that
growth now fills gap slots with instead of zero, so entry_term_locked
can distinguish "never explicitly written" from "explicitly
recorded as legacy term 0" regardless of later array growth.
The same determinism fix exposed a second, independent instance of the
identical wedge via the snapshot-install catch-up path instead of
WAL-streaming catch-up: persist_reconcile_checkpoint_term recorded a
wire checkpoint_term of 0 as a raw, explicit write (bypassing the
growth-fill sentinel entirely), so a snapshot-installed standby read
the checkpoint LSN back as WAL_LEGACY_TERM(1) forever while the leader
now correctly read its own checkpoint LSN as absent(0) -- the same
permanent logOk mismatch, a different code path. Fixed by substituting
TERM_INDEX_UNSET for a wire value of exactly 0 before recording; a
wire 0 is unambiguous (persist_entry_term's own contract never returns
raw 0 for a present entry, so 0 can only mean genuine absence).
Independently reviewed across three rounds: sentinel mechanism
verified via real RED->GREEN revert-and-rebuild; a documentation
claim that config_repl.c/ha.c were "comment-only" corrected; the
snapshot-path gap independently traced, confirmed real via its own
RED->GREEN cycle, and fixed; two stale doc comments (persist.h's
public contract, ha.c's call-site rationale) still describing the
disproven pre-fix legacy-0 convention corrected. Suite: 1028 -> 1035
tests, 0 failures.
Build the live 3-node chaotic-multi-kill config-upgrade soak scenario
(run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1,
6.4. This needed genuinely new client-side infrastructure: a hand-rolled
capnp send path for OP_upgradeConfig (the auto-generated encoder is a
no-op for its Data fields, worked around the same way
xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig
CLI commands.
Building and running this live surfaced two real production bugs, both
fixed here:
- bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's
primary/standby dispatch guard in cluster.c, making config upgrades
unreachable from any network client.
- bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called
config_activate_committed(), so the node that originates a config
upgrade never refreshed its own in-memory state as part of committing
it. Killing that node before some later, unrelated event happened to
refresh it left its checkpoint permanently stale -- the first live,
empirical reproduction of bug-857's originally hypothesized failure
mode. Fixed by activating before the marker-triggered checkpoint write,
since checkpoint content is built from live runtime tables that only
reflect the reload after activation runs.
A separate, narrower checkpoint divergence (bug-1471, not config-related)
remains open and still blocks a clean soak PASS -- task 8.6 stays
in-progress until it's resolved.
Wires task 6.1's upgrade_materialization_verify/upgrade_materialization_fenced
into ha_dispatch_gate's CTRL_PRIMARY branch (sequential block alongside,
not merged into, the existing config_serve_fenced check -- same
replication/read-only opcode allowlist, refuses state-changing ops
with -XCE_ERR_NOT_READY while fenced) and the promotion gate
(raft_cb_become_leader refuses to become CTRL_PRIMARY while fenced,
introduces no new WAL/log-completeness eligibility mechanism, never
self-terminates). Adds a 4-function self-triggering retry trampoline
(upgrade_materialization_retry_fn/_vtask_entry/_delaymgr_trampoline/
schedule_upgrade_materialization_retry) mirroring
config_activate_retry_fn's exact shape, hooked into
ha_cb_commit_advanced_stub alongside the existing config-byte retry.
Fixes a real bug caught on review: the fence had no production path
that could ever engage it from its zero-initialized default -- every
setter call site was gated on the fence already being set (closed
loop), making the new gate checks permanently dead code. Fixed with
two unconditional upgrade_materialization_verify() call sites:
persist.c's apply_committed_entry WAL_CONFIG_MARKER case (right after
persist_checkpoint_marker_triggered returns, regardless of its rc) and
myapp.c's boot sequence (right after config_activate_committed, for
Req 5.2 crash/restart resume). New test drives the fence through the
real production trigger (a forced -EBUSY checkpoint miss on a real
marker commit+apply), not a direct verify() call, proving the
end-to-end path actually works. Task 6.2 of 19.
Requirements: 3.1, 3.2, 3.3, 3.4, 4.3, 5.5
Adds an explicit case WAL_CONFIG_MARKER to apply_committed_entry,
replacing its no-op treatment, that forces a checkpoint at exactly the
marker's own (term, lsn) via two new helpers:
persist_checkpoint_write_locked(p, checkpoint_lsn) -- the checkpoint
core (pin lsn, scan config ref, write_snapshot, trim, rotate) taking
an explicit LSN and never acquiring apply_pump_lock itself
persist_checkpoint_marker_triggered(p, lsn) -- shares the existing
checkpoint_in_progress CAS guard, calls the above with the marker's
own lsn
This fixes two hazards design.md's "call persist_checkpoint() directly"
text didn't account for: (1) apply_committed_entry's live-pump caller
(persist_apply_pump_run, used by both primary and standby online-apply)
already holds apply_pump_lock across the dispatch call -- a direct
persist_checkpoint() call would self-deadlock on that mutex; (2)
last_applied_lsn/replay position only advance AFTER apply_committed_entry
returns at both call sites, so pinning checkpoint_lsn from
last_applied_lsn inside the marker case would label the checkpoint with
the previous entry's LSN, not the marker's own -- fixed by threading the
marker's own lsn parameter through explicitly. A -EBUSY outcome (a
concurrent periodic checkpoint mid-flight) is logged at TRACE and treated
as expected/self-resolving, never retried. The new case has no effect on
entity interpretation/re-homing (Req 1.8).
New test_upgrade_snapshot_apply.c drives WAL_CONFIG_MARKER through the
REAL production call paths (persist_apply_pump_run and boot replay via
persist_recover), not a bare apply_committed_entry() call, so the fix is
actually proven against the lock-holding context rather than assumed.
Also fixes a genuine regression in test_config_repl.c surfaced by
WAL_CONFIG_MARKER apply having a real side effect for the first time
(bug-1466): an fsync-worker-thread race (t42_wait_applied polling helper,
mirrors test_apply_pump.c's tap_wait_applied) and one test call site
trusting an unreliable raft-derived LSN. Surfaces (but does not fix, out
of this spec's boundary) a pre-existing ha-term-safety/raft-log-replication
gap: persist_write_checkpoint_marker bypasses ha_primary_log_append,
breaking its documented current_lsn==log_len+1 invariant once a
checkpoint interleaves with primary appends -- log_len feeds real
RequestVote freshness comparisons, not advisory-only as first assumed;
severity corrected in bug-1466. Task 5 of 19.
Requirements: 1.1, 1.5, 1.8, 4.4, 5.2, 5.3
load_snapshot_entities' queue-frame decode migrates from the old raw
{name_len,name,status} layout to {payload_len, capnp QueueAdminState
bytes}, still calling the unmodified restore_queue_state. Adds a new
host-frame section decode after queues, dispatching to
restore_host_state (worker.c, task 3) with the same bounds-checking
and restore-before-advance discipline as the existing job/queue loops.
Un-ignores test_snapent_load_snapshot_restores_entities_before_advancing_position
(blocked since task 2.2) -- now passes for real. Also fixes a genuine
regression in test_snapshot_recv_out_of_order_chunks_assemble_correctly
(test_config_repl.c): its synthetic checkpoint-body filler landed on
the new host_count field's offset, tripping the cluster==NULL guard --
pinned to a well-formed host_count=0, matching that test's existing
fmt/job_count/queue_count precedent. Task 4 of 19.
Requirements: 1.10
write_snapshot now appends a flat [config_bytes_len][raw bytes]
section after the host frames, reading the locally-held active config
straight off disk (read_active_config_bytes, persist.c) rather than
calling into config_repl.c -- mirrors scan_latest_committed_config_ref's
existing precedent of avoiding a persist.c -> config_repl.c
function-call dependency. Populated on every checkpoint, not only
marker-triggered ones (Req 2.4). No local config yet (early boot)
degrades to an explicit empty section rather than failing the
checkpoint; genuine corruption still fails loud. Task 2.3 of 19.
Requirements: 2.1, 2.3, 2.4, 5.1
collect_queue_frames now encodes each queue as a QueueAdminState capnp
frame (capnp_encode_queue_admin_state, mirroring task 2.1's host-frame
encoder) instead of the raw {name_len,name,status} layout;
write_snapshot's queue trailer writes [payload_len][capnp bytes] per
queue. restore_queue_state and load_snapshot's queue decode are
untouched (task 4's boundary).
Temporarily TEST_IGNORE_MESSAGE's
test_snapent_load_snapshot_restores_entities_before_advancing_position
(m_queues=2 real write+load round trip hits the still-raw loader) --
tracked in tasks.md Implementation Notes, must be re-enabled by task 4.
Matches existing codebase precedent (bug-847) for this pattern. Task
2.2 of 19.
Requirements: 1.9, 5.7
Adds collect_host_frames + capnp_encode_host_admin_state, walking
c->workermgr the same way collect_job_frames/collect_queue_frames do.
Every host currently in the table gets one HostAdminState frame
(status masked to XCE_WORKER_ADMIN_MASK only, no dynamic/liveness
fields), written into write_snapshot after the queue trailer. Task 2.1
of 19.
Requirements: 1.6, 1.7, 1.9
Adds independently-extensible Cap'n Proto frames for the upgrade
snapshot's host/queue admin-state sections (frame-only, not reusing
Worker/QueueConf which bundle dynamic state) and bumps
SNAPSHOT_FORMAT_VERSION 1->2 so a checkpoint written under the prior
raw queue-frame layout is fail-loud rejected rather than
misinterpreted. Task 1 of 19.
Requirements: 1.9, 5.7
replay_wal() now halts logical apply and engages persist_mark_apply_resource_fenced()
on a negative apply_committed_entry() return during boot/restart replay, mirroring
persist_apply_pump_run()'s existing live-path halt-and-fence behavior (task 4.1).
Physical scan bookkeeping (current_lsn, CRC chain) stays unconditional per entry,
consistent with persist_recover()'s own established current_lsn/last_applied_lsn
divergence precedent.
Went through one remediation round: round 1 exercised the wrong ha.c gate (periodic
re-alert timer); round 2 corrected the test to drive the real election-time gate
(raft_cb_become_leader_with_cluster_for_test()), with teeth independently confirmed
via a temporary fence-clear RED/GREEN cycle. Round-2 reviewer APPROVED.
Documents a real, NOT-fixed residual gap found during this task: a node's very
first-ever promotion evaluation in a fresh process runs before that node's own
first persist_recover()/replay_wal() call, so this fence is invisible to that one
promotion check (ha.c:1303-1408 gate runs before ha.c:1427-1439's first-boot
recovery). Every promotion evaluation after a process's first one is protected.
Out of this task's persist.c-only boundary; flagged for a dedicated ha.c follow-up.
restore_job_array()'s bulk-insert pass could fail partway through
(key_add() on realloc growth exhaustion), leaving an already-inserted
prefix permanently visible in c->jobmgr but excluded from
out_new_jobs -- so those elements existed and were individually
addressable but never reached HOOK_DETAIL_JOB_NEW, never scheduled or
pushed, not on this attempt or any later successful retry. A literal
violation of Requirements 2.3 and 9.4, tracked as bug-1443.
Adds kvmgr_reserve()/xzarray_reserve(), new purely-additive primitives
in shared kvmgr.c/zarray.c (following the existing kvmgr_capture_keys()
template -- bare extern function, no RTTI vtable slot, caller-holds-
write-lock contract), that pre-grow c->jobmgr's backing store before
the bulk-insert pass begins, closing the dominant N-scaling failure
trigger structurally.
Found a genuine off-by-one in the existing, unmodified
xzarray_ensure_items(): its strict less-than sufficiency check means
reserving exactly array_size and then performing array_size real
stores still triggers one more growth cycle on the last store.
Reserving array_size + 1 is the exact minimal correct compensation,
independently re-derived and confirmed by review with concrete
arithmetic.
For the now-much-narrower residual case (kvtab_add()'s own small
per-entry allocation, or a content collision -- both non-N-scaling,
honestly documented as still theoretically possible): restore_job_array()
now populates out_new_jobs with whatever prefix was genuinely inserted
before a failure, and apply_committed_entry() fires HOOK_DETAIL_JOB_NEW
for that non-empty partial prefix even when the overall return is
negative, while still propagating the negative rc so the LSN correctly
halts and the Live Apply Resource Fence still engages for the
remainder. A genuinely-inserted element is never left un-pushed again.
Task 7.1 of job-array-single-wal-record (Requirements 2.3, 9.4, 9.6).
replay_wal() (persist.c ~3902) tallied *new_job_submits_out only for
hdr.type == WAL_JOB_SUBMIT. WAL_JOB_ARRAY_SUBMIT entries were
dispatched and correctly materialized N real jobmgr entries via
apply_committed_entry()'s array case, but never counted --
persist_verify_apply_correctness() then saw real jobmgr growth exceed
the undercounted expectation and declared APPLY_DIVERGENT, engaging a
PERMANENT, non-self-healing fence (distinct from task 4.2's
self-healing Live Apply Resource Fence).
Live-reproduced during task 6.1's soak attempt on 2 of 3 nodes in a
single small-array (N=2) sub-round via two different trigger paths.
Deterministic, not size-dependent -- the first array submission to a
live cluster would fence any node that restarts or hits an
election-path recovery fallback before its next checkpoint, shrinking
the promotable pool. A pre-existing comment at persist.c ~3863-3874
(predating this spec) had already flagged this exact code region as
needing re-verification before tasks 4.1/6.1+ went live; this is that
re-verification, and it failed.
Fix adds a WAL_JOB_ARRAY_SUBMIT branch tallying apply_rc (the genuine
per-record element count apply_committed_entry()'s array case already
returns) when apply_rc > 0, mirroring the existing dedup-safe pattern
for WAL_JOB_SUBMIT. Traced the full return-value space of
restore_job_array()/apply_committed_entry()'s array case to confirm
every tallied value is provably equal to real jobmgr growth -- the fix
cannot overcount and so cannot mask a genuine divergence.
Verified via a from-scratch live 3-node HA soak re-run (--array-soak),
independently by both implementer and reviewer: zero APPLY_DIVERGENT
across all three original trigger paths, one-WAL-record/one-commit-
barrier confirmed for N=2 and N=10000, per-element post-failover
status/cancel/completion confirmed.
Task 6.0a of job-array-single-wal-record (Requirements 3.1, 3.2, 4.1,
9.4). Inserted mid-run after live soak testing surfaced the bug; user
confirmed sequencing (fix now, blocking task 6.1+).
Adds a new atomic LSN field on persist_t (distinct from the existing
boot-only unrecoverable-fence flag) plus three primitives:
persist_apply_resource_fenced(), persist_mark_apply_resource_fenced(),
persist_clear_apply_resource_fence(). Wired into
persist_apply_pump_run(): engages on task 4.1's halt condition, clears
unconditionally on the walk's next successful advance — provably
equivalent to "clears only when the same stuck LSN's retry succeeds"
since the walk is strictly sequential and never advances past a
failed position.
Known residual gap, out of this task's boundary (documented in
tasks.md Implementation Notes): replay_wal() during boot/restart
replay does not yet share this protection, so a node crashing while
fenced can silently re-pass the same failure on restart. Flagged for
a follow-up task, not fixed here.
Task 4.2 of job-array-single-wal-record (Requirements 9.4, 9.5).
persist_apply_pump_run() no longer discards apply_committed_entry()'s
return value. On a negative rc for any WAL type (not just the new
array type), the walk halts at that LSN — no last_applied_lsn advance,
no pending-apply hand-off — logs a loud error, and retries from the
same position on the next kick, mirroring the existing
apply_pump_read_entry failure branch's shape.
Investigated whether this newly turns the pre-existing job-fact
-ENOENT path into a permanent-wedge hazard: it does not, since
c->jobmgr never deletes entries and facts are only ever applied after
their causally-prior submit already succeeded, so a live -ENOENT there
can only mean genuine divergence — halting is the correct behavior.
Task 4.1 of job-array-single-wal-record (Requirement 9.4).
Adds case WAL_JOB_ARRAY_SUBMIT to apply_wal_entry() (standby
catch-up dispatch), calling restore_job_array() hook-free, matching
the existing WAL_JOB_SUBMIT case's shape exactly.
Review found apply_wal_entry() has zero production call sites -- a
2026-07-10 refactor (ha_standby_advance_committed_and_apply) moved
standby entity materialization onto the same apply_committed_entry()
pump the primary uses; the code comment claiming otherwise predated
that refactor and was never updated. Requirement 3.1/3.2/4.1 are
fully delivered by task 3.2 alone. This task's wiring is kept as
defensive parity with WAL_JOB_SUBMIT's existing, equally-dead case
in the same switch, not as an independent delivery path -- design.md
and tasks.md corrected accordingly.
Task 3.3 of job-array-single-wal-record.
Adds case WAL_JOB_ARRAY_SUBMIT to apply_committed_entry() (the live
apply pump / boot replay dispatch seam): calls restore_job_array()
(task 3.1) instead of restore_job(), and on success fires
HOOK_DETAIL_JOB_NEW exactly once carrying every newly created
element in restore_job_array()'s own out_new_jobs list -- never once
per element. No hook fires on an idempotent no-op (-EEXIST) or any
genuine construction failure, honestly propagating task 3.1's
documented bulk-insert-phase residual gap rather than papering over
it. apply_wal_entry() (standby catch-up) is untouched -- task 3.3.
Task 3.2 of job-array-single-wal-record.
Adds persist_submit_job_array_async(): encodes one JobSubmit-shaped
payload (base jobid, array size, one shared spec), appends exactly
one WAL_JOB_ARRAY_SUBMIT entry, and registers exactly one commit
continuation regardless of array size -- reuses the existing
job_commit_adapter_ctx_t/job_commit_resolve_adapter machinery, no
per-element loop. Not yet wired into handle_jobsubmit (task 2.2).
Also fixes a test-helper bug (WAL offset back-computed from the
post-append cursor, only valid for zero-payload entries) that
corrupted state across tests when exercising a nonzero-payload
array-submit entry.
Task 2.1 of job-array-single-wal-record.
Adds the array-shaped WAL entry type (distinct from WAL_JOB_SUBMIT,
value 11 in the unused WAL_JOB_* gap) and classifies it WAIT-mode via
wal_classify, matching the single-job type's commit-barrier semantics.
No dispatch/apply wiring yet -- foundation for task 2.x/3.x.
Task 1.1 of job-array-single-wal-record.
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).
bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.
Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
Found via extended chaotic /xce-test-ha soaking on top of the bug-1419 fix
(same session). Both bugs independently caused the 10-round chaotic soak to
fail; sequential and chaotic now pass cleanly and repeatably.
1. Term-desync liveness deadlock (new): a standby that loses a near-
simultaneous election race to a peer can legitimately end up at a HIGHER
term than the peer that actually won and became primary. The primary's
now-stale-term AppendEntries/heartbeats are correctly rejected by the raft
library, but every one of those rejected messages was ALSO resetting the
standby's own liveness deadline (ha_on_heartbeat_received), so its
liveness timeout never fired and it could never start a fresh,
higher-term election to let the cluster self-heal — a permanent stall.
Fix: ha_msg_term_is_stale() gates the liveness reset (and
ha_discover_primary step-down) on the incoming message's own term stamp,
skipping it for provably-stale messages, in cluster.c's wal_catchup_entry
and nop/heartbeat handlers.
2. bug-847 (root-caused 2026-06-30, never fixed): a per-sub-entry `payload`
Data blob in a multi-entry WAL catch-up batch loses its capnp pointer once
the message crosses a segment boundary — the receiver decodes
raw_payload_len=0 for entries at/after the boundary and silently persists
empty WAL entries (structurally intact: correct LSN/term/CRC chain, empty
content). Only surfaces when a node rejoins and catches up a real batch,
which is why it needed a live chaotic soak (not the 2-entry unit test) to
reproduce. Fix: carry every sub-entry's payload concatenated in ONE
top-level batchPayloads blob instead of one Data blob per entry — a
single top-level blob does not exhibit the cross-segment loss. Sliced
back apart on decode using each sub-entry's own wirePayloadLen, a plain
scalar that (like lsn/entryType/term) decodes via the generic generated
decode path and was never subject to the bug. Confirmed via targeted
diagnostic logging that the corrupted entry arrives already-empty over
the wire (not a receive-side race), and via a faithful reproduction at
live JOB_SUBMIT payload size (5×3560B) after the original 2-entry/
14×872B tests turned out not to cross a segment boundary in this
environment.
Incidental fixes found and fixed along the way:
- A real self-deadlock (persist_entry_term() re-acquiring p->lock from
inside a caller that already held it) introduced by an initial, ultimately
unnecessary WAL-read locking fix attempted before bug-847 was pinned down;
fixed via a new persist_entry_term_locked() lock-free counterpart. Left in
place as a genuine (if incidental) hardening of ha_stream_wal_to_standby's
read consistency.
- .claude/skills/xce-test-ha/SKILL.md's `context: fork` frontmatter caused
the skill's own forked sub-session teardown to SIGHUP the whole
xce_controller process group mid-soak, killing all 3 nodes silently with
no error logged, whenever the soak was launched via the Skill tool itself
(setsid/nohup inside run-ha-soak.sh only protects against a controlling
terminal's own SIGHUP, not a parent forked-session's teardown). Removed;
added explicit guidance to launch long soaks via direct Bash from the main
session instead.
Verified: full 78-group unit regression sweep clean (0 failures) after each
change; two consecutive clean 10-round chaotic soaks and one clean 10-round
sequential soak on the final binary (zero committed divergence, zero job
loss, all 3 nodes led in both modes).
Root cause (confirmed via targeted DIAG1419 logging on a live chaotic soak):
a client's commit-timeout retry during a genuine quorum-loss window appends
a second WAL_JOB_SUBMIT for the same jobid; the timed-out entry is never
retracted, so if it later achieves majority commit anyway both entries end
up committed. This is a legitimate, harmless duplicate — restore_job()'s own
-EEXIST correctly no-ops the second one — but replay_wal() tallied every raw
WAL_JOB_SUBMIT occurrence instead of genuine creations, so the duplicate
made new_job_submits disagree with jobmgr's real growth, firing a false
APPLY_DIVERGENT and deadlocking promotion.
Reverts the layer-3 claim/release machinery (job_wal_commit_claim/release,
claim_committed_job_submits, and their wiring): three attempts at preventing
the duplicate at submit/dispatch/commit time all failed to close the gap,
because the duplicate isn't a bug to prevent — a WAL entry can't be
retracted once appended, so any submit-time guard is structurally the wrong
layer.
Real fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now returns 1 for a
genuine creation vs 0 for an idempotent no-op (restore_job() -EEXIST), and
replay_wal() only tallies new_job_submits_out on rc==1. Updates the affected
apply-seam/reapply-idempotency tests for the new return contract, and
reverses two apply-correctness tests whose premise (duplicate-jobid replay
must diverge) is now understood to be the actual bug rather than the
expected behavior — they now force divergence directly via the same
production fencing seam instead of relying on that trigger.
Verified: two consecutive 10-round chaotic soaks, zero divergence, zero job
loss, no election stall (previously reproduced within ~5 rounds).
jobid_range_exists() (c->jobmgr) and the in-flight registry (job.c,
per-process, only populated by this node's own handle_jobsubmit calls)
both miss a real cross-node case: a client's XCECLI_FAILOVER_CALL retry
resends the same client-generated jobid after a connection drop, and the
resend lands on a newly elected primary that never processed the original
request. If the original was already committed before the old primary
died, the new primary's own local WAL already holds it (Raft Leader
Completeness) but jobmgr hasn't applied it yet — both existing guards see
nothing, so the resend appends a second, genuinely committed WAL_JOB_SUBMIT
entry for the same jobid. A live chaotic soak reproduced this exactly
(waldump: two committed entries, same term, matching apiserverId/timestamp/
sequence, one second apart) — apply's own idempotent dedup keeps jobmgr
correct, but replay's raw WAL_JOB_SUBMIT count no longer matches jobmgr's
growth, firing a false APPLY_DIVERGENT fence and deadlocking promotion.
Fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now claims the jobid in
job.c's in-flight registry (new job_wal_commit_claim/release, wrapping the
existing inflight_range_try_add/remove) for the duration of dispatch,
regardless of whether this node originated the submission or is applying a
replicated/replayed entry. A resend landing mid-dispatch on any node now
sees the same -EEXIST rejection self-originated in-flight submits already
get. Release only fires when the claim call added the entry, so a
self-originated submission's own claim/release lifecycle (unchanged) is
never double-released or stolen.
Also fixes 4 pre-existing tests that broke under the earlier checkpoint_lsn
fix (0eb1054) and were missed in that commit's regression sweep: cluster-
less fixtures where last_applied_lsn never advances now correctly get
checkpoint_lsn=0 unless the test simulates apply catch-up, matching the
new (correct) semantics. One assertion in
test_checkpoint_tail_with_no_snapshot_is_apply_clean compared Phase 1's
checkpoint_lsn against Phase 2's independently-reconstructed physical WAL
tip — that equality only held under the pre-fix current_lsn-tracking
behavior and is removed, not patched around.
Also hardens run-ha-soak.sh's start_node() with setsid+nohup: a forked/
background test-harness invocation whose session tears down mid-run was
silently killing every controller process via SIGHUP forwarding (observed
twice — deploy completes, ~1s of activity, then all 3 nodes die together
with no error logged).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
persist_checkpoint() stamped checkpoint_lsn from p->current_lsn (physical
WAL append tip) instead of p->last_applied_lsn (the true "jobmgr reflects
up through here" cursor), uncoordinated with the async apply pump
introduced by raft-apply-decoupling. write_snapshot() serializes live
jobmgr state, so a checkpoint taken while apply lags the WAL tip mislabels
the snapshot: on restart last_applied_lsn is re-seeded from checkpoint_lsn
and replay resumes strictly above it, silently skipping committed-but-not-
yet-applied entries jobmgr never received. This is the actual mechanism
behind kill-mid-apply producing a permanent APPLY_DIVERGENT fence —
apply_committed_entry() is deterministic/idempotent and write_snapshot()
is crash-atomic, so the bug was a mislabeled watermark, not a structural
divergence requiring a safety/liveness tradeoff.
Fix stamps checkpoint_lsn from last_applied_lsn and holds apply_pump_lock
across the label capture + write_snapshot() so the label and the jobmgr
state it describes are captured atomically w.r.t. the live apply pump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
