commit_submit_async() lets a WAIT-mode caller register a continuation
against an appended LSN instead of blocking on pthread_cond_timedwait,
reusing the existing admission-cap discipline (wait_admit_locked/
wait_release_locked) so the bounded-capacity guarantee carries over
unchanged. Purely additive: no existing call site (repl_wait_majority_ack)
is touched yet -- that switch lands in task 3.2 once commit_resolve()
(task 1.2) exists to drive it.
Full Kiro spec for the apply-after-commit refactor: canonical
append->commit->apply->notify ordering, a single apply_committed_entry
seam shared by live commit-advance and restart replay, and the
Facts-Apply invariant (ha-consistency-invariants criterion 2.9)
reconciling worker-reported facts onto the same seam.
buglog.json had two broken auto-detection hook signatures polluting it:
- 207 entries "Missing error handling in X" / fix "Added try/catch block"
(nonsensical on .md/.capnp files, and C has no try/catch — never a real
applied fix)
- 452 entries with a literally-empty template ("Significant refactor of "
with no subject ever filled in) / fix "Rewrote N->M lines"
Removed both (659 total). Also merged 24 legitimate duplicate bug IDs
(re-logged/updated entries sharing an ID) into single entries — kept the
most recent content as primary, unioned tags/related_bugs, summed
occurrences — and stripped noise tags ('auto-detected', 'refactor') that
had leaked onto real entries from a since-merged junk duplicate. Net:
1406 -> 721 entries, verified zero remaining duplicate IDs and zero
remaining junk signatures.
.gitignore: added patterns for the scratch soak-run logs and codegraph.html
this session generated at repo root (already deleted from the tree), and
fixed the ucai-formatter-cache.local.json pattern to match its nested
occurrences (.claude/skills/*/.claude/, src/*/.claude/) instead of only the
repo-root one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
All 4 sub-specs are complete: ha-commit-engine (18/18), ha-commit-barrier-
rollout (23/23), ha-single-leader-role (17/17), ha-convergence-recovery
(19/19). Reconciles ha-single-leader-role and ha-convergence-recovery's
spec.json phase to "implemented" (both mislabeled tasks-generated despite
full task completion — same staleness pattern fixed for raft-commit-safety/
raft-log-replication earlier).
Promotes the umbrella itself to "implemented": design.approved corrected
true (was false, contradicting its own note "Design approved as contract"),
phase implemented, with an explicit callout that criterion 2.9 (Facts-Apply
invariant, added 2026-06-25) remains a documented deviation owned by the
still-unstarted raft-apply-decoupling spec — not a gap in this umbrella or
its sub-specs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
bug-847 (catch-up WAL batch drops payloads across capnp segment boundaries):
its purpose-built RED reproducer, test_wal_batch_roundtrip_large_payloads
(14 entries x 872-byte payloads, 12.2KB total, genuinely spans multiple
4096-byte capnp segments per capn-malloc.c's create()), was still
TEST_IGNORE_MESSAGE'd. Un-ignoring it and running under ASAN with leak
detection: clean pass, byte-exact payloads, zero leaks, 2/2 runs. Traced the
vendored c-capnproto far-pointer machinery (write_ptr/capn_setp in capn.c)
to confirm the cross-segment struct->Data reference path is correct by
construction — whatever regressed this already got fixed elsewhere (library
bump or an incidental request.c change); the ignore was just never removed.
No production code changed.
bug-1307 (task-5.7 election-jitter data-safety regression: 7 divergences +
4 job losses over 6 chaotic rounds): re-ran the exact reproducing scenario
(6-round chaotic soak, real ext4, ASAN binaries) — 21 jobs, 0 divergence,
0 job loss, clean PASS. Confirms the raft-ha-core-safety Phase 5 work
resolved it.
Both bugs were the last 2 of the 7 flagged in the prior staleness sweep;
all 7 are now closed (6 were already fixed and undetected, 1 needed the
test un-ignored).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Verified against live code/tests instead of trusting the buglog text:
- bug-846 (EINVAL on dead ctx): FIXED — xcecli_retry_decide checks ctx.state
== XCE_CTX_ERR before classifying, bypassing the misclassification.
- bug-855 (6 wal_catchup tests failing): FIXED — ran controller_unit_test,
all pass.
- bug-1096 (stale-term vote-reply counted): FIXED — raft.c has the
reply_term != s->term drop.
- bug-1147 (WAL trim ignores standby cursor): FIXED — trim_wal_segments has
the cursor-bound floor guard, comment cites this bug ID directly.
- bug-1198 (snapshot-send liveness-pool hazard): FIXED — in-code CLOSED
banner, bug-1197's endpoint ref/unref bracket makes it unreachable.
- bug-1307 (election-jitter safety regression): left OPEN with a downgrade
note — circumstantial evidence (raft-ha-core-safety Phase 5 soak-clean)
but no direct code citation found; wants one more confirmation soak
before closing.
- bug-847 (catch-up WAL drops payloads at scale): RE-CONFIRMED genuinely
still open — its RED reproducer is still TEST_IGNORE'd.
Also noted (not acted on, out of scope for this pass): buglog.json has 26
duplicate bug IDs and ~239 "Missing error handling in unknown" / "Added
try/catch block" auto-detected junk entries — a separate cleanup task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Accidental npm-init artifacts from an earlier checkpoint commit (empty
package.json, auto-generated lockfile) in a pure C/Go project with no
Node.js build step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
ha-commit-durability: closes the "pending confirmation under simultaneous
total outage" caveat. Reproduced cold-restart-all (kill all 3 members +
restart) 5 times against a live 3-node cluster on genuine ext4 (not the
sandbox's overlayfs) — zero committed-job loss across 40 rolling-failover
rounds + 5 cold-restart-all cycles. Confirms bug-795 was an overlayfs
fdatasync artifact, not a real durability gap; no production fix needed.
Commit Durability is now proven for both rolling single-node failover and
simultaneous total-cluster outage (src/controller/CLAUDE.md updated).
Live-validated and marked complete against a real 3-node cluster:
- controller-status 6.4: off-primary committed fidelity + responder
timestamp, confirmed via a real failover
- ha-heartbeat-committed-propagation 4.1: standby committedLsn tracks the
primary via heartbeat within the 3s interval
- ha-wal-replication 5.4: standby WAL replication verified via committedLsn
match (job-facing ops are correctly primary-only by design; standbys
return E_XCE_STANDBY, not a bug)
- libxce-failover 6.2: live 3-node failover E2E, jobsub raced against a
primary kill resolves to exactly one job, no duplication
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Adds the Raft Apply-Ordering & Hook Decoupling roadmap phase: canonical
pre-hooks -> raft commit -> apply -> notify ordering, with the raft protocol
step lifted out of the notification hook fan-out (root of the bug-795/828/
851/1060 divergence family). Decision: apply-after-commit via a single
shared apply_committed_entry() seam reused by online apply and restart
replay.
- raft-drive-async: full spec docs (brief/requirements/research/design)
committed alongside its already-tracked tasks.md; async-drive prerequisite
for the apply-decoupling work, sequenced to start first (threatens
failover liveness today).
- raft-apply-decoupling: new spec (brief + facts-in-raft.md) covering the
apply seam and the facts (WAL_MODE_RETAIN) apply-timing question (open
decision D1).
- job-array-single-wal-record: promoted from backlog to a tracked spec,
sequenced after raft-apply-decoupling.
- ha-consistency-invariants: add criterion 2.9 (Facts-Apply invariant),
documenting the current on-receipt fact-reaction deviation this family
will reconcile.
- raft-commit-safety / raft-log-replication: reconcile spec.json phase to
"implemented" (both were mislabeled tasks-generated despite full 18/18
and 24/24 task completion + raft_sim GO).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
bug-1401: xce_free_ctrlstatus(s) was defined as free_CtrlStatus(s), which
only frees the fields inside xce_ctrlstatus_t. The real client reply path
(decode_CtrlStatus_ptr) calloc's the struct itself, so every ctrlstatus call
leaked sizeof(xce_ctrlstatus_t) (120 bytes). Existing unit tests never caught
it because they built/freed the struct via the raw stack-decode + bare
free_CtrlStatus() helpers, never exercising the actual client macro. Fixed
by switching the macro to free_CtrlStatus_ptr(&(s)) and added a regression
test that exercises the real heap-decode path; RED confirmed the exact
120-byte/1-alloc LeakSanitizer signature before the fix.
bug-1400: the originally reported SEGV in cmd_jobsub on malformed positional
input is stale — NULLSTR guards added 2025-11-05 already reject it cleanly.
The real residual defect was submit.start_info leaking on all 4 early-return
validation paths; fixed by routing them through a shared `goto cleanup`
(matches the existing goto-clean idiom in queue_ctrl).
bug-1396 (job-count anomaly after cold-restart-all, not yet root-caused):
instrumented test-commit-durability.sh's assert_job_count() to dump the live
joblist and ACKED_FILE ledger on any count mismatch instead of only the
count, per the bug's own recorded next step.
Verified: controller ASAN build + full controller_unit_test green; lib
xcelib_test 285/285 green, 0 ASAN leaks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
ha_send_snapshot_to_standby ignored the network send's return value in its
chunk loop, silently truncating a transfer on a mid-stream error while still
logging "complete" (bug-1386). Fixing the silent drop exposed a second,
independent bug in the same pipeline: ha_handle_snapshot_chunk appended
received chunks in arrival order instead of at their wire-carried
chunk_offset, so out-of-order delivery across the vtask pool corrupted the
reassembled file even though every chunk's bytes were individually correct
(bug-1389). Fixed by checking every fallible call in the send loop and
switching the receiver to pwrite-at-offset with a count-based completion
tracker.
Adds snapverify/snapdiff xce_test commands (offline checkpoint-file
structural validator and byte-level comparator) used to diagnose bug-1389 and
kept as standing diagnostic tools for future snapshot-transfer triage.
g_drive_async_configpush_offload flipped back ON after both fixes, validated
via 6-round chaotic + 3/3 force-snapshot soak (raft-drive-async task 3.3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Cross-references bug-1381 in tasks.md and the ha.c configpush flag comment.
No code behavior change -- push_config_transport_only's race (raw standby
endpoint pointers read/written by unprotected pthread_create workers,
independent of the offload flag) is a distinct, still-open blocker from
bug-1197/bug-1376 (both closed in the prior commit).
Root cause: peer_ctrl_msg_handler held ep->dispatch_inflight incremented
across the entire peer_ctrl_dispatch_to_pool call, including its inline
fallback (the common cold-start shape when ha->pool isn't wired yet). That
inline body can reach raft_cb_become_follower, which -- now that
cluster_init_phase2 ran synchronously earlier in the same call chain and
wired the pool -- tears down its own loser vote-ep (the very ep whose reply
triggered this call) via the deferred path (vote_ep_teardown_async), whose
endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that
SAME ep. That count is held only by this exact call, on this exact thread,
and can't reach 0 until the spin itself returns -- a permanent self-deadlock.
Confirmed live via a gdb thread backtrace: added a GDB_WRAP debug seam to
run-ha-soak.sh (launches a named node directly under `gdb --batch` so it
becomes ptrace-legal as gdb's own direct child under yama scope 1, bypassing
a live-attach restriction), then SIGINT to gdb once the log went stale for a
bounded interrupt + full backtrace. The frame dump showed pool=0x0 at
peer_ctrl_dispatch_to_pool, and the same ep pointer in the nested
vote_ep_teardown_async -> endpoint_mark_tearing_down frame, spinning on
sched_yield().
Fix: moved the dispatch_inflight release into peer_ctrl_dispatch_to_pool
itself, at the exact right point per exit path -- released BEFORE calling
the inline body on every fallback path (pool==NULL, OOM, enqueue failure),
since io_worker is single-threaded and the inline path never enqueues
anything (no pool-worker race left to protect against once inline execution
is committed to); released only AFTER the non-blocking enqueue completes on
the success path, preserving the original protection for the genuinely
concurrent pool-worker case.
New unit test test_vep_self_teardown_does_not_deadlock uses a bounded
pthread_timedjoin_np so a regression fails loud instead of hanging the
suite. RED confirmed via a temporary manual revert (ETIMEDOUT) before
restoring the fix.
Live-verified with both flags ON: --force-snapshot soak (the exact repro
shape) completes without hanging across repeated runs, confirms real
snapshot installs, zero divergence, zero job loss; a 6-round --chaotic soak
also passed clean; full controller suite green throughout.
g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled
now default ON in production, closing raft-drive-async tasks 3.1 and 3.2.
g_drive_async_configpush_offload stays OFF -- a distinct, still-open hazard
in push_config_transport_only's endpoint handling (task 3.3, unrelated to
this bug).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
Cross-references bug-1376 in tasks.md and the ha.c flag comment for the
unresolved hang found while investigating the raft-drive-async flag flip
(commit f3b6e6a). No code behavior change.
Both fixes are real and independently verified; the drive-async flags
(g_drive_async_snapshot_offload, g_drive_async_peerctrl_dispatch_enabled)
stay at their production-default OFF -- see the tasks.md session note for
why the flip itself doesn't land in this commit.
1. Snapshot-lane borrowed-endpoint hazard (task 3.2, bug-1197 family):
ha_snapshot_send_dispatch now takes endpoint_ref(ep) before submitting to
the slow lane, released on every non-lane terminal path and by
snapshot_lane_fn on completion. Reuses the general bug-1298 endpoint
refcount system (standby-ep teardown already routes through
endpoint_unref via cluster.c's endpoint_killer) -- a concurrent RDHUP
teardown can now only drop the owner's reference while a transfer is in
flight, never free the ep out from under it. New test:
test_snapshot_offload_holds_endpoint_ref_across_transfer (RED->GREEN).
2. Catch-up streamer WAL_CHECKPOINT bug (found while verifying #1 via a live
soak, pre-existing, unrelated to the drive-async flags): ha_stream_wal_
to_standby had no filter for WAL_CHECKPOINT markers -- local-only
bookkeeping that still consumes a real LSN slot via persist_write_
checkpoint_marker's own current_lsn++. It shipped one to a catching-up
standby, which correctly refused it (persist_wal_write_replicated
rejects type==WAL_CHECKPOINT) and got stuck retrying the same LSN
forever. Fixed by skipping WAL_CHECKPOINT entirely in the streamer loop,
mirroring the existing replay_wal/term-index precedent of treating
checkpoint-marker LSNs as invisible gaps. New test:
test_wal_stream_skips_checkpoint_marker (RED->GREEN).
Attempted flipping both drive-async flags ON after landing these fixes (the
"full fix, defer prod flip" decision's prerequisite -- 4.1+4.2 are green).
A 6-round --chaotic soak passed clean, but a --force-snapshot soak (a
genuinely empty node cold-start rejoin, the one scenario chaotic mode never
exercises) surfaced a THIRD, not-yet-root-caused issue: the freshly-wiped
node's own self-election appears to race cluster_init_phase2's vtask-pool
wiring, then the process hangs silently (no crash, no ASAN report) until
killed. Reverted both flags to OFF rather than ship an unproven state --
production behavior is unchanged by this commit. Full controller suite
green and both soaks (--force-snapshot, 6-round --chaotic) pass clean with
flags OFF, confirming no regression from the two logic fixes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
Chain of fixes needed to get --force-snapshot soak to a real live PASS
(raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1):
1. WAL segment rotation was gated on a hardcoded 64MB compile-time
constant; trim_wal_segments() never deletes the active (currently-open)
segment. At soak job counts a segment never rotated, so nothing was
ever physically trimmable and ha_evaluate_repl_action always chose
STREAM_WAL over SEND_SNAPSHOT for an empty-rejoining node. Fixed by
threading a wal_segment_bytes override through persist_config_t
(mirrors the --max-wal-records pattern: config.h default, persist.h
setter, persist.c 3 usage sites + minimum-floor enforcement, 6-point
CLI wiring through main.c/myconf.h/myapp.c/cluster.c) plus a new
--wal-segment-bytes CLI flag. run-ha-soak.sh's --force-snapshot mode
now sets WAL_SEGMENT_BYTES=4096 to force real rotation at soak scale.
2. This unmasked a genuine heap-use-after-free in INVOKE_MSG_HANDLER
(include/netio.h): the macro calls ep->msg_handler(ep) synchronously
(not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing
doesn't cover it) then writes ep->fsm_in.pkt = NULL. A stale vote-reply
handler chain (raft_on_vote_reply -> raft_cb_become_follower ->
endpoint_unref) could drop the endpoint's last ref and free it inside
the handler call, so the trailing write hit freed memory. This is the
first-ever live exercise of a node receiving a snapshot transfer while
mid-election. Fixed by holding a ref across the synchronous call, same
pattern task 5.8 established for the async path.
With both fixes: --force-snapshot soak now PASSES live (committed
divergence=0, job_loss=0, confirmed real "HA: snapshot" install on the
restarted node) — first genuine live proof of this scenario. 3-round
--chaotic regression clean. Full 217-test controller suite green
before/after every fix in the chain.
Builds on the delaymgr starvation fix (b9ad018) and walcompare
checkpoint-floor fix (f3c86a6) from earlier in this investigation.
Root-caused via raft-ha-core-safety 5.10a investigation: schedule_checkpoint()
(persist.c) fires correctly once wal_record_count crosses its threshold, but
its delay_task()-scheduled checkpoint_task_fn never executed in a live
multi-node deployment -- confirmed live via temporary diagnostic logging
(wal_record_count=100 max=100 logged, but the task itself never ran).
Root cause: delay_task_t carried TWO incomparable time representations --
`nextrun` (wall-clock time_t seconds, used by delay_task()) and `nextrun_ts`
(CLOCK_MONOTONIC timespec, used by delay_task_ms()). task_cmp()'s mixed-type
fallback compared them directly: an ms-task's nextrun is always 0, a
seconds-task's nextrun is a real epoch value (e.g. 1785600123) -- so 0
always sorted "less than" any seconds-task, regardless of actual due time.
ordq (src/lib/ordq.c) keeps its queue as an ascending sorted linked list by
this comparator, and delay_worker() only ever inspects the list HEAD
(ordq_peek with no matchfn returns q->head immediately, no scanning). Since
cluster_t.delaymgr is shared with ha.c's ~10 recurring delay_task_ms() call
sites (heartbeats, election timers, RDHUP polling), the head was
structurally almost-always occupied by an ms-task -- any delay_task()
(seconds) caller sharing that delaymgr, including schedule_checkpoint's
checkpoint_task_fn, could be starved indefinitely.
Fix: eliminate the dual representation entirely. Every task -- whether
scheduled via delay_task() (seconds) or delay_task_ms() (milliseconds) --
now stores one flat uint64_t CLOCK_MONOTONIC millisecond deadline
(nextrun_ms). delay_task() becomes a thin seconds->ms wrapper around
delay_task_ms(). task_cmp() and task_due() collapse to a single integer
comparison; delay_worker()'s reschedule is a flat integer add (no more
tv_sec/tv_nsec carry arithmetic, no more is_ms_task branching). Net -42
lines. Live-verified: after this fix, a forced checkpoint round produces a
real installed snapshot file (previously: schedule_checkpoint fired but the
task silently never ran).
controller_unit_test: full 58-group sweep, 0 Failures, 0 ASAN reports
(before and after this change) -- no regression, including every
delaymgr-touching group (test_demotion_convergence, test_ha_start_election_async,
test_ha_core_safety, test_ha_startup, etc).
Records this session's investigation: walcompare's per-member floor
fix (f3c86a6) and the --force-snapshot soak scaffolding (9f4292d) were
built specifically to close 5.10a with direct live evidence, but hit
a new, separate blocker -- schedule_checkpoint() fires correctly but
its delaymgr-scheduled checkpoint_task_fn never executes in a live
deployment, so no snapshot is ever installed to test against. Task
5.10a stays open pending a dedicated delaymgr investigation.
Adds --max-wal-records (mirrors the existing --wal-sync-ms pattern:
main.c argp option -> myconf_t -> persist_config_set_max_wal_records())
so a deployment can force the WAL-record-count checkpoint trigger to
fire on a job count a soak can actually reach, instead of the
production default of 10000.
Adds a new `--force-snapshot` soak mode (run-ha-soak.sh): submits
enough jobs to cross the configured threshold, kills a non-primary
node, wipes its data dir (a genuine empty rejoin), restarts it, and
asserts the cluster reconverges with zero walcompare divergence and
zero job loss. Fails loud (not a false PASS) if the restarted node's
log shows no actual snapshot install, so the round can't silently
report success without exercising the real scenario.
INVESTIGATION FINDING (not yet resolved, this commit does not close
5.10a or ha-entity-snapshots 7.1): live-tested with temporary
diagnostic logging (since reverted). schedule_checkpoint() correctly
fires once wal_record_count reaches the configured threshold
(confirmed: "wal_record_count=100 max=100" logged), but the delaymgr-
scheduled checkpoint_task_fn never executes afterward -- no checkpoint
ever completes, no snapshot file is ever written, in a live multi-node
deployment. This is a separate, deeper issue in the delaymgr scheduling
path (or its wiring for this specific call shape), not a bug in this
commit's CLI plumbing or soak scaffolding -- both are built, tested,
and confirmed working up to the point where the async checkpoint
should run. Needs its own dedicated investigation (likely src/lib
delaymgr, not persist.c) before --force-snapshot can produce a PASS.
This also has production implications beyond the test scenario: if
confirmed, WAL checkpoints may never fire automatically today, meaning
WAL segments are never trimmed via this path in a live deployment.
controller_unit_test: test_ha_startup 34/0, test_ha_core_safety 28/0,
test_persist_checkpoint 5/0, test_walcompare 17/0 (no regression).
Soak --selftest 17/17.
walcompare previously had no concept of a per-member compaction floor:
committed_lsn only capped the HIGH end (skip LSNs a member hasn't
caught up to yet), but the LOW end unconditionally expected every
member to hold raw WAL from LSN 1. A member that legitimately caught
up via a full-checkpoint snapshot holds no raw WAL at or below its own
checkpoint LSN by design (that range was entity-restored, not
WAL-replayed) -- so it was always flagged MISSING/divergent there,
regardless of whether the restore was correct. This is the root cause
ha-entity-snapshots task 7.1 diagnosed (committed_divergence=11 on an
otherwise-healthy soak) and raft-ha-core-safety task 5.10a's residual.
Adds member_snapshot_floor(): each member's own snap/ sibling
directory (matching the deployed <data>/wal + <data>/snap layout) is
scanned for the highest installed snap-<lsn>.{capnp,bin} file. Any LSN
at or below that member's own floor is excluded from its comparison
the same way an LSN above its own committed_lsn already is -- never
uniform/shared across members, since compaction is a per-member fact.
wc_status() gains a new "compacted" token distinct from "MISSING" so
the machine-readable report can tell the two apart.
Two new regression tests prove: (1) the floor correctly suppresses the
compacted range from comparison, and (2) it does NOT mask a genuine
divergence within the range a member still claims to hold. Caught and
fixed one bug myself before landing: the floor comparison was
off-by-one (< instead of <=), which the first test run surfaced
immediately (Expected 0 Was 1).
test_walcompare 17/17, test_ha_core_safety 28/0, test_config_repl
66/0, test_waldump 7/0 (all no-regression). Soak --selftest 17/17.
Confirms the live chaotic soak eliminates the divergence this spec's
Headline Finding hypothesized (LSN-9 committed-prefix divergence,
round-4 election stall) -- not by synthetic unit tests alone, but by
the task 7.2 acceptance soak (3 independent PASS runs, zero divergence,
zero job loss, 6/6 clean elections).
Documents the full resolution chain (Phase 2 payload loss -> Phase 5
election liveness, 5 layers -> Phase 3 snapshot boundary -> Phase 6
end-to-end + single-seam verification -> Phase 7.1 full regression
sweep) and reviews the final behavior against all four raft/raft.tla
safety invariants (Election Safety, Leader Completeness, State Machine
Safety, Commit Ordering/Durability). No residual divergence remains
open under this spec; two items explicitly routed to their owning
specs rather than closed here (dormant heartbeat-driver path, soak
harness's transient committedLsn=0 read).
This closes the last task in raft-ha-core-safety (18/18 sub-tasks).
The chaotic soak's walcheck()/walcheck_count() previously forced a
static `1 + 2*njobs` boundary onto walcompare, which doesn't account
for per-election NOOP WAL entries and overshoots the real tip --
producing spurious committed_divergence counts on an otherwise-clean
cluster (verified via waldump: byte-identical, gap-free WALs across
all 3 nodes despite a reported divergence=7).
Replaces the static estimate with the LIVE, authoritative committedLsn
(via the existing probe_primary client probe), applied as an explicit
--committed boundary only when the cluster is genuinely converged
(cluster_committed_converged) -- falling back to walcompare's own
auto-derive (min CRC-intact tip) when not, so a legitimately-lagging
just-restarted standby never gets a false "MISSING" verdict mid-chaos.
Pure auto-derive alone can't catch a member that permanently lost a
committed entry (its own low tip just becomes the shared boundary),
so this restores exact-boundary detection at every genuinely-settled
instant while keeping the auto-derive fallback for in-flight kills.
Also fixes a live-caught edge case in the same mechanism: a transient
ctrlstatus race can report committedLsn=0 for the primary right after
reconvergence; live_committed_boundary() now rejects a bare "0" as
untrustworthy and falls back to auto-derive, instead of handing
walcompare a vacuous --committed 0 boundary that would check nothing.
Selftest 17/17. Three independent fresh 6-round chaotic soaks (own
run + reviewer's reproduction + post-guard-fix re-run) all PASS:
committed_divergence=0, committed_job_loss=0, 6/6 elections, 21/21
jobs present throughout. No production C code touched.
A full 69-group sweep of controller_unit_test (this spec's harness had
never been run end-to-end before) found 2 deterministic failures,
both test-only staleness against already-shipped, already-approved
production behavior from earlier in this same spec:
- test_ha_vote_gate: test_pre_vote_catch_up_denies asserted the
pre-task-5.3 unconditional in_catch_up PreVote deny. Task 5.3
(bf88681) deliberately gated that deny on primary_ctrl_fd>=0 &&
ha_heartbeat_is_fresh to fix a real round-4 election-stall — this
test's exact scenario (stale heartbeat, primary_ctrl_fd=-1) is the
case 5.3 intentionally flipped to grant. Split into
test_pre_vote_catch_up_grants_when_primary_unreachable (matches the
current contract) and test_pre_vote_catch_up_denies_while_primary_live
(preserves the original protective intent, correctly scoped).
- test_egress_filter: test_live_submit_tags_submit_lsn called the
JOB_NEW persist_job_hook_fun AFTER_EVENT hook directly, but task 5.9
(15be5d9, bug-1306 fix) made that hook's WAL_MODE_WAIT branch
notify-only, moving the real WAL append + commit barrier into
persist_submit_wal_and_commit(). 5.9 relocated 4 equivalent tests in
test_persist_hooks.c but missed this one. Now drives
persist_submit_wal_and_commit() directly, mirroring that pattern.
No production code changed. Full 69-group suite: 0 Failures, 0 ASAN
reports (independently reproduced 3x: initial sweep, implementer,
reviewer).
The term-blind ha_apply_single_entry direct-persist branch (carried
forward from task 2.4) is unreachable with a real non-zero term in any
current production path: its only trigger, raft_send_append_entries,
has zero production call sites (library simulator only). Both real
senders (repl_broadcast, ha_stream_wal_to_standby) always emit a
non-zero term for a present LSN. Documents this in ha.c and design.md
rather than structurally retiring a dormant path (YAGNI), with an
explicit flag for future re-evaluation if a heartbeat-driver task ever
wires up raft_send_append_entries.
Adds two regression pins: restart preserves (term,lsn) identity
(replay_wal already reconstructs the term index via record_entry_term),
and real senders never produce the term-blind wire shape.
test_ha_core_safety 28/0, no regression, no ASAN reports.
Integration test proves the guarantee tasks 2.3/2.4 hardened in
isolation (synthetic fault injection on the streamer) also holds
end-to-end over two real persist_t/WAL instances through the real
production ingest seam (ha_apply_wal_catchup_entry). Covers live
in-term replication and cross-term catch-up: no term==0 sourced for a
live entry, no non-empty entry ever lands as PLEN=0, standby persists
the entry's stored term (not the AE header's leader-current-term,
the bug-1067 split), and walcompare reports a clean committed prefix.
test_ha_core_safety 26/0, no regression, no ASAN reports.
Adds checkpointTerm (@6, append-only) to the SnapshotTransfer wire
message. The leader stamps it via persist_entry_term(p, checkpoint_lsn)
when sending a snapshot; the standby reconciles it into its own
per-LSN term index (persist_reconcile_checkpoint_term) right after
persist_load_snapshot succeeds, before applied_lsn advances. This
authoritatively overwrites any stale local term a divergent
pre-snapshot entry left at the checkpoint LSN, closing the gap task
3.1's RED test proved (Expected 7 Was 3 -> now passes). Transport
stays persist_t-free; the reconcile lives in persist.c per the
existing dependency direction. Decode is auto-generated via capnp
mapname; only the hand-rolled encode path (request.c/typemap.h, needed
for the raw snapshot payload) required a matching manual update.
test_ha_core_safety 25/0, test_config_repl 66/0, no ASAN reports.
ha_handle_snapshot_chunk/persist_load_snapshot advance checkpoint_lsn,
current_lsn, and applied_lsn on snapshot install but never touch the
per-LSN term index, so a divergent pre-snapshot entry's stale local
term survives the install unreconciled. New test proves the gap
(Expected 7 Was 3); fix lands in task 3.2.
persist_recover's local snapshot load passed NULL for out_entities_restored,
so a rolling-upgrade restart against a pre-feature checkpoint had no
observability parity with the wire receive path's (task 6.1) INFO log.
Mirror the same conditional logging here; no behavior change, WAL replay
in Step 3 still applies unconditionally either way.
Live-verified in a 3-node cluster: planted a hand-crafted 16-byte legacy
sentinel as ctrl-1's own checkpoint file, restarted all 3 — no crash, INFO
log fired, ctrl-1 was elected primary serving the full 37-job list, and
walcompare confirmed all 3 members' committed prefixes (LSN 1..49) are
byte-identical.
10-round chaotic soak: committed_job_loss=0 (core goal confirmed fixed),
committed_divergence=11. Root-caused to walcompare having no per-member
compaction-floor concept (uniform --committed N for all members), which is
exactly raft-ha-core-safety task 3's unimplemented snapshot-boundary
reconciliation, not a gap in this spec's boundary. Also: this run never
exercised an actual snapshot install (start_node doesn't wipe data dirs) —
forcing a real empty-rejoin round is still outstanding for a true signal.
ha_handle_snapshot_chunk now logs whether a received checkpoint was
fully entity-restored or was a legacy position-only file (INFO level,
framed as an expected rollout state -- never an error, per design.md).
The rc-gates-applied_lsn control flow is byte-for-byte unchanged from
task 5.2 (confirmed by review) -- this task is purely additive
observability, matching design.md's own note that this call site
"needs no change" once persist_load_snapshot's gate is correct.
New test drives the legacy path end-to-end through the real production
handler (rc==0, file renamed, applied_lsn advances to the legacy LSN).
This completes the integration layer -- write path (jobs+queues),
restore-before-advance read path, and the production wiring are all in
place and independently reviewed. Only validation tasks remain (7.1
chaotic soak, 8.1 rollback compatibility).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Replaces the naive task-5.1 stopgap with the real restore contract:
detects legacy vs entity-bearing checkpoints by remaining file size;
restores every job then every queue status via restore_job/
restore_queue_state, in file order, with bounds-checked length-prefixed
reads; only after every entity restores successfully does
current_lsn/checkpoint_lsn commit to the checkpoint's position. Every
failure path (bounds check, decode, restore error, unrecognized
format_version) returns before that single commit point -- position is
provably untouched on failure, never partially advanced.
Also fixes the pre-existing restore_job decode leak (job.c): job_init
deep-copies submit (never nulled) and MOVE_PTRs allocs/pidinfo/
pend_reasons (nulled on the one path that runs) -- free_Job(&j) is safe
unconditionally on every reachable return path (EEXIST early-return,
create_job success, create_job failure), verified against the exact
generated free_*_ptr null-check guards.
Two pre-existing test_config_repl.c wire-protocol tests needed their
synthetic garbage snapshot bodies updated to a well-formed empty-entity
tail, since load_snapshot now genuinely parses past the legacy header
(previously ignored); their actual assertions (chunk-reassembly
correctness) are unchanged.
This is the exact function/contract that caused task 2.2's rejection
(position advancing before restore validated). Independent review
traced the memory-safety fix through the generated capnp code field-by-
field, confirmed the test_config_repl.c change doesn't weaken any
assertion, and confirmed restore_queue_state's ENOENT-aborts-the-whole-
checkpoint behavior is faithful to design.md's unconditional contract
(flagged as a watch-item for task 7.1's soak, not a defect).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Extends persist_load_snapshot's public signature with an
out_entities_restored bool*, per design.md's documented contract. The
static load_snapshot function -- the real restore logic -- is completely
untouched (still today's watermark-only implementation, byte-identical
to before this task); only a thin wrapper was added that naively maps
*out_entities_restored = (rc==0), deliberately scaffolding so the 3 new
RED tests fail for genuine reasons rather than accidentally passing.
ha_handle_snapshot_chunk's sole call site gets a mechanical
declare-and-pass update -- confirmed inert, never reads the new
out-param (task 6.1's job).
Three tests against persist_load_snapshot directly (unlike prior tasks'
tests, which deliberately stayed on the raw-byte-parsing side to avoid
this exact function): restore-before-advance (fails at the job-lookup
assertion -- nothing is ever restored today), corrupted-entity-leaves-
position-unchanged (fails at rc!=0 -- today's code ignores the corrupted
section entirely, since load_snapshot never reads past the legacy
header), legacy-file-reports-position-only (fails at entities_restored==
false -- the naive wrapper claims restoration for every rc==0 case,
including legacy files).
This is the exact function/call site that caused task 2.2's rejection
(a live position-advances-before-restore-validates hazard). Independent
review verified load_snapshot's body is byte-identical to commit
d52eb94, the wrapper does nothing beyond the naive mapping, and ha.c's
change never reads the new out-param anywhere -- no repeat of that
mistake here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
rotate_snapshots now parses each candidate file's LSN and skips deletion
via a new snapshot_lsn_in_flight() check against
p->cluster->ha.snapshot_inflight_lsn[] (task 4.1) before unlinking --
protected files are retried on the next rotation pass once the transfer
completes and the LSN clears.
Checks LSN directly (nonzero + matches) rather than gating on
snapshot_inflight[]==1 first, per task 4.1's own note: the LSN clears
before the flag in every completion/failure path, so a stale flag=1 can
momentarily coexist with an already-cleared LSN=0. A checkpoint LSN is
never 0, so this is race-free without needing the flag at all.
p->cluster==NULL (standalone lib-level tests) means no HA layer exists to
protect anything.
Task 4.2's RED rotation-guard test now passes; full regression sweep
(test_persist_checkpoint, test_persist_hooks, test_commit_barrier_reply,
test_commit_durability, test_e2e_hooks, test_hooks, test_config_repl,
test_ha_liveness) all clean. Both controller_unit_test and production
xce_controller build clean; nm confirms only the real production helper
(snapshot_lsn_in_flight) appears, no test-only symbols leaked in.
Executed directly (no subagent dispatch) per the hybrid execution-mode
decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
A checkpoint file whose position matches an in-flight peer transfer
(task 4.1's snapshot_inflight_lsn[]) must survive rotate_snapshots even
when snap_keep_n would otherwise delete it. Genuinely fails today (file
removed, access() returns -1) since rotate_snapshots has no awareness of
in-flight transfers yet.
Executed directly (no subagent dispatch) per the hybrid execution-mode
decision -- small, single-file, isolated test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Split remaining tasks by risk after a mid-session spend-limit hit: small/
isolated tasks (4.2, 4.3, 7.1, 8.1) execute directly with self-review;
load_snapshot restore (5.1/5.2, the exact function that caused task 2.2's
rejection) and the production wiring (6.1) keep the full implementer +
independent-reviewer subagent cycle, since that's where adversarial review
already proved its worth and where the remaining risk is concentrated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Adds snapshot_inflight_lsn[MAX_HA_CANDIDATES] to ha_state_t, alongside the
existing snapshot_inflight[] flag it mirrors exactly: set to the
checkpoint's LSN at ha_snapshot_send_dispatch before a transfer's first
byte goes out, cleared to 0 at every completion/failure path
(snapshot_lane_fn's normal completion, and all 3 early-return failure
sites: malloc failure, slowlane dedup-refuse, slowlane submit failure).
This is the foundation task 4.3 will consult from rotate_snapshots to
avoid deleting a checkpoint file a peer is still mid-transfer reading
from -- the one genuinely new retention guard this feature needs (the
existing write_snapshot-before-trim_wal_segments sequencing already
satisfies the WAL-trim side of Requirement 3).
Proven via a new gated unit test (test_ha_liveness.c) that observes the
tracked LSN mid-flight while the send body is parked at a test gate, not
just before/after -- confirms the lifecycle ordering, not just the
endpoints.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
write_snapshot replaces the queue_count=0 placeholder with a real
collect_queue_frames helper: enumerates every queue in the live table
under nested table+per-entity RTTI_LOCK_READ (no retention filter --
table presence already implies config-provisioned existence, unlike
jobs), writes each queue's (name, status) as a plain fixed record (no
capnp -- matches restore_queue_state's exact contract). Any failure
anywhere still discards the whole checkpoint, including any
already-collected job frames, no partial file at the final path.
Entirely contained to persist.c -- load_snapshot and persist.h remain
byte-identical to before this task, honoring the boundary lesson task 2.2
established (queue restoration stays task 5.2's scope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Queue's (name, status) round-trip through write_snapshot's raw on-disk
bytes -- a plain fixed record (no capnp; queue definitions are
config-owned, only runtime admin status is WAL/snapshot-tracked, per
design.md's corrected queue model). Fails today (Expected 4 Was 0) since
write_snapshot still hardcodes queue_count=0 from task 2.2's placeholder.
Does not call persist_load_snapshot, following task 2.2's established
raw-byte-parsing pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
write_snapshot now Cap'n-Proto-encodes every retained job under nested
table+per-entity RTTI_LOCK_READ (table lock alone does not exclude a
concurrent per-entity writer -- design.md's central correctness finding),
appends length-prefixed job frames after a new format_version marker, and
fixes a previously-unchecked fdatasync return. Any failure anywhere in the
path (encode, write, sync) discards the whole checkpoint -- no partial
file at the final path, proven by a real mid-encode-loop fault-injection
test (forces one specific job's encode to fail among N, asserts nothing
survives).
Also fixed two pre-existing bugs in task 1.1's test fixture: terminal-state
seeded jobs need a non-zero finish_time or job_should_retain's retention
window silently excludes them from any checkpoint; the fixture needs
xstring_table_init or every seeded job's submit/start_info silently
decodes as NULL.
Round 1 of this task also touched load_snapshot/persist.h to make the
round-trip tests pass end-to-end, but review caught a live hazard: it
advanced current_lsn/checkpoint_lsn BEFORE the entity-decode loop ran, so
a decode failure still left the position advanced -- reachable today
through the unmodified ha_handle_snapshot_chunk production path, not
hypothetical. Remediated by fully reverting load_snapshot/persist.h to
byte-identical pre-task-2.2 state and rewriting the round-trip tests to
parse write_snapshot's raw on-disk bytes directly (read_Job/decode_Job)
instead of round-tripping through persist_load_snapshot -- proves the
write side without depending on load_snapshot, which stays task 5.2's
scope entirely, not started here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Three tests against the current sentinel-only write_snapshot/load_snapshot:
job content round-trip and concurrent-mutation-not-torn genuinely FAIL
(RED) since no entity content is written today; write-failure-discard
already passes (write_snapshot's open()-failure path already discards
correctly) -- flagged transparently as a GREEN-from-day-one regression
guard rather than faked RED, per real fault injection (directory removal
-> genuine ENOENT).
Carries forward two notes for task 2.2: fdatasync's return value is
currently unchecked in write_snapshot (persist.c:1505), and a mid-encode-
loop fault-injection test should be added once the per-job encode loop
exists.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Adds the ha-entity-snapshots Kiro spec (requirements, design, research,
tasks) for task 5.10a / raft-ha-core-safety: making the HA checkpoint an
actually-restorable copy of committed job/queue state instead of a
position-only sentinel, closing the residual chaotic-soak divergence gap.
Task 1.1 (foundation): reusable test fixture snapent_make/snapent_teardown
in a new test_persist_snapshot_entities.c, seeding N jobs (varying state)
and M queues (varying status) into a real persist_t+cluster_t for later
write/load/rotation-guard tests to build on. Build clean; new suite 1/1;
regression spot-checks (test_persist_hooks 19/19, test_commit_barrier_reply
22/22) clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
A dead primary's control fd RDHUPs faster than an election settles. The 5.6
guard only drops an RDHUP while the role is instantaneously CTRL_ELECTING/
prevote; an RDHUP landing in the gap after a lost election (role briefly back
to CTRL_STANDBY) re-triggers a fresh ha_begin_failover -> PreVote -> real
election (term++). One bump per RDHUP -> unbounded escalation under a sustained
cadence (observed round 4: 48 begin_failovers, terms 6->24, round overran the
harness window).
Fix: ha_state_t.last_failover_ms (atomic) stamped at ha_begin_failover;
ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2*election_timeout_ms of the
last trigger (closes the dead fd, returns) so the election's own jittered retry
(5.7) runs without the external RDHUP cadence adding term bumps. First trigger
never debounced; genuine loss still fails over, at most once per window.
Validation: unit-proven — test_rdhup_within_cooldown_is_debounced (recent
last_failover -> stays CTRL_STANDBY) vs test_rdhup_single_node_reaches_primary
(last_failover=0 -> CTRL_PRIMARY) is a differential proof. Soak: 10/10 elect,
terms 4->14 bounded, begin_failover 14/14/12 (was 48), committed_job_loss=0, no
regression. The stochastic storm did not recur this run (debounce fired 0x), so
soak-reproduction of the storm is still pending; the mechanism rests on the
unit test. Election suites green (rdhup 8/0, prevote 12/0, core_safety 24/0,
election 28/0, failover_prevote 6/0).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
The round-4 term ladder is driven by repeated RDHUP->begin_failover->
PreVote-won->real-election(term++) cycles from the dead primary's socket, not
a behind-node vote loop. 5.6's instantaneous 'suppress while electing' guard is
insufficient under a sustained RDHUP cadence. Fix direction: per-failover
cooldown/debounce + hold the failover latch across the full prevote->election->
settle window. Needs a deterministic dead-primary-cadence repro.
Their WAL_MODE_WAIT hook (wal_hook_append_classified) does local-append only
and carries no durability verdict through the hook return, so there is nothing
for a later hook to clobber. JOB was unique in running the full commit barrier
inside the hook. Their commit barrier is the separate ha-commit-barrier-rollout
handler barrier — verify independently, do not mechanically extend the JOB fix.
freshness-deny escalation loop under a just-restarted-survivor overlap: a
behind-log candidate is freshness-denied every term (18->24 observed), times
out, re-campaigns at term+1 -> ~18 escalations before convergence. NOT RNG
(seed is /dev/urandom); 5.7 jitter bounds retry rate, not the escalation.
Fix direction: a repeatedly-freshness-denied candidate must recognize it is
behind and defer/catch-up instead of laddering the cluster term. Needs a
deterministic 2-survivor repro before a fix (5.5/5.6/5.7 family).
raft-apply-decoupling item #1 (down payment). xce_run_hooks returns only the
LAST-registered hook's rc, so a later JOB AFTER_EVENT subscriber (sched/
apiserver) returning 0 clobbered persist_job_hook_fun's commit -ETIMEDOUT.
handle_jobsubmit then replied 'jobsubmit ok' for an entry that never reached
majority, and a new-term leader legally discarded it (committed-job-loss,
Commit Durability violation).
- persist.c/.h: new persist_submit_wal_and_commit(p, jlist) — the explicit raft
step (WAL append + majority-commit barrier) returning the commit verdict
directly. persist_job_hook_fun's WAL_MODE_WAIT branch is now a no-op.
- job.c handle_jobsubmit: call the explicit step, read commit_rc DIRECTLY,
reject on <0 before firing the now notification-only AFTER hook
(canonical commit -> notify -> reply).
- tests: +test_job_hook_new_is_notify_only regression pin; 4 hook tests
relocated to drive the explicit step. persist_hooks 19/0, e2e_hooks 3/0,
commit_barrier_reply 22/0, commit_durability 38/0.
Proven: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code).
Residual soak FAILs (divergence catch-up lag + a round-4 election-escalation
tail) are NOT job loss — tracked as task 5.10.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
10-round soak waldump+log bisect: the chaotic committed_job_loss is NOT the
catch-up/entity-snapshot prefix gap. All 3 WALs gapless/CRC-intact/full
payloads; lost job only on ctrl-1@LSN30 term9 vs term-10 NOOP on the majority.
The durable-majority gate worked (commit_submit -> -ETIMEDOUT), but
xce_run_hooks (hook.c:134-157) returns only the LAST-registered hook's rc, so a
later JOB AFTER_EVENT hook (sched/apiserver) returning 0 clobbered persist's
-ETIMEDOUT -> handle_jobsubmit replied jobsubmit ok for an uncommitted entry ->
term-10 leader legally overwrote it. Fix per raft-apply-decoupling item #1:
move the WAL append + commit barrier out of the hook into an explicit
handle_jobsubmit-driven raft step whose rc is read directly (down payment on
the full apply-after-commit inversion). Entity-snapshot gap is a separate,
still-real defect (not this soak's mechanism).
