- xce-build.sh: `controller` component now always builds `controller/xce_test`
alongside `xce_controller`, in every profile — xce-test-ha needs both
binaries and previously had to build xce_test as a separate step.
- xce-build/SKILL.md: removed `context: fork` — reproducibly broken for this
skill (confirmed twice): whenever ARGUMENTS is non-empty, the forked
subagent never receives a usable task and silently does nothing. Documented
that this skill should instead always be dispatched via the plain Agent
tool with a fully-resolved, self-contained prompt (not `context: fork`,
which failed) — proven to work end-to-end.
- xce-test-ha/SKILL.md, xce-deploy-ha/SKILL.md: replaced the blanket "never
invoke via subagent" warning with the precise finding — `context: fork`
(Skill-tool isolation) is the confirmed-broken mechanism (its own teardown
SIGHUPs the whole controller process group mid-soak/mid-deploy), but
Agent-tool subagent dispatch is a different mechanism and is safe
(empirically verified: a `setsid nohup ... & disown` process launched from
inside an Agent subagent survives that subagent returning, reparented to
init). Documented both the foreground-blocking and fire-and-forget
subagent patterns for each skill; xce-deploy-ha additionally flagged that
its `start.sh &` should gain `setsid` to match the soak driver's stronger
detach pattern before relying on unattended subagent dispatch.
40-round sequential HA soak FAIL at round 36: verify_no_loss reported
"0/111 jobs confirmed present" right after a fresh election, even though the
new primary's own log confirmed it stayed CTRL_PRIMARY throughout and
successfully committed the new jobs. Not a real data-safety bug.
Root cause: the client's failover retry loop (client_failover.c) can exhaust
its 20-attempt budget during a rapid post-election retry storm without ever
reaching the new primary, and the budget-exhaustion path only overrides the
surfaced error for RETRY_BACKOFF exhaustion (-E_XCE_NOT_READY) — not for
RETRY_NOW exhaustion, so a stray E_XCE_STANDBY from a standby the retry loop
happened to land on gets surfaced as if it were a hard, permanent rejection.
verify_no_loss() did a single one-shot check with no retry at the driver
level, unlike the job-submit loop in the same file which already retries on
this exact "primary likely just promoted" condition.
Adds a bounded retry (3 attempts, 2s settle sleep) around the whole
verify_no_loss check, mirroring the existing submit-retry rationale. Left
client_failover.c untouched (battle-tested library, many other consumers) —
fixing at the test-driver level is safer and fully local.
Validated: 40-round sequential (123 jobs, zero loss, all 3 nodes led) and
40-round chaotic (123 jobs, zero committed divergence, zero committed job
loss) soaks both PASS.
A primary never learns of a follower's higher term via AppendEntries-reject
responses: ha_raft_on_repl_ack only routes AE replies through
raft_on_append_entries_reply for success=1 ACKs, and that function itself
drops any reply whose term != s->term (raft.c stale-term drop) before it
could adopt a higher term anyway. Net effect: once a follower outpaces a
stale primary's term, every AppendEntries the primary sends is rejected
forever and it can never regain write-quorum — a permanent election livelock
(surfaced as "no primary elected at term>8" in HA soak testing).
Adds two fallback term-discovery vectors that fire regardless of AE-reject
outcome:
- StateReport heartbeat-reply now carries the standby's current term
(controller.capnp, typemap.h) so the primary learns a higher term on every
convergence cycle.
- New raft_on_higher_term_observed() mirrors the UpdateTerm step (strictly
greater-than only; touches term/voted_for/role; persists before notifying
on_become_follower) and is wired into both the heartbeat state-report path
and the WAL-NACK path in repl_ack_decode_and_dispatch.
Validated: 265 controller + 287 lib unit tests pass; 10-round sequential and
chaotic HA soaks both PASS with zero job loss and full leadership coverage.
Found via extended chaotic /xce-test-ha soaking on top of the bug-1419 fix
(same session). Both bugs independently caused the 10-round chaotic soak to
fail; sequential and chaotic now pass cleanly and repeatably.
1. Term-desync liveness deadlock (new): a standby that loses a near-
simultaneous election race to a peer can legitimately end up at a HIGHER
term than the peer that actually won and became primary. The primary's
now-stale-term AppendEntries/heartbeats are correctly rejected by the raft
library, but every one of those rejected messages was ALSO resetting the
standby's own liveness deadline (ha_on_heartbeat_received), so its
liveness timeout never fired and it could never start a fresh,
higher-term election to let the cluster self-heal — a permanent stall.
Fix: ha_msg_term_is_stale() gates the liveness reset (and
ha_discover_primary step-down) on the incoming message's own term stamp,
skipping it for provably-stale messages, in cluster.c's wal_catchup_entry
and nop/heartbeat handlers.
2. bug-847 (root-caused 2026-06-30, never fixed): a per-sub-entry `payload`
Data blob in a multi-entry WAL catch-up batch loses its capnp pointer once
the message crosses a segment boundary — the receiver decodes
raw_payload_len=0 for entries at/after the boundary and silently persists
empty WAL entries (structurally intact: correct LSN/term/CRC chain, empty
content). Only surfaces when a node rejoins and catches up a real batch,
which is why it needed a live chaotic soak (not the 2-entry unit test) to
reproduce. Fix: carry every sub-entry's payload concatenated in ONE
top-level batchPayloads blob instead of one Data blob per entry — a
single top-level blob does not exhibit the cross-segment loss. Sliced
back apart on decode using each sub-entry's own wirePayloadLen, a plain
scalar that (like lsn/entryType/term) decodes via the generic generated
decode path and was never subject to the bug. Confirmed via targeted
diagnostic logging that the corrupted entry arrives already-empty over
the wire (not a receive-side race), and via a faithful reproduction at
live JOB_SUBMIT payload size (5×3560B) after the original 2-entry/
14×872B tests turned out not to cross a segment boundary in this
environment.
Incidental fixes found and fixed along the way:
- A real self-deadlock (persist_entry_term() re-acquiring p->lock from
inside a caller that already held it) introduced by an initial, ultimately
unnecessary WAL-read locking fix attempted before bug-847 was pinned down;
fixed via a new persist_entry_term_locked() lock-free counterpart. Left in
place as a genuine (if incidental) hardening of ha_stream_wal_to_standby's
read consistency.
- .claude/skills/xce-test-ha/SKILL.md's `context: fork` frontmatter caused
the skill's own forked sub-session teardown to SIGHUP the whole
xce_controller process group mid-soak, killing all 3 nodes silently with
no error logged, whenever the soak was launched via the Skill tool itself
(setsid/nohup inside run-ha-soak.sh only protects against a controlling
terminal's own SIGHUP, not a parent forked-session's teardown). Removed;
added explicit guidance to launch long soaks via direct Bash from the main
session instead.
Verified: full 78-group unit regression sweep clean (0 failures) after each
change; two consecutive clean 10-round chaotic soaks and one clean 10-round
sequential soak on the final binary (zero committed divergence, zero job
loss, all 3 nodes led in both modes).
Root cause (confirmed via targeted DIAG1419 logging on a live chaotic soak):
a client's commit-timeout retry during a genuine quorum-loss window appends
a second WAL_JOB_SUBMIT for the same jobid; the timed-out entry is never
retracted, so if it later achieves majority commit anyway both entries end
up committed. This is a legitimate, harmless duplicate — restore_job()'s own
-EEXIST correctly no-ops the second one — but replay_wal() tallied every raw
WAL_JOB_SUBMIT occurrence instead of genuine creations, so the duplicate
made new_job_submits disagree with jobmgr's real growth, firing a false
APPLY_DIVERGENT and deadlocking promotion.
Reverts the layer-3 claim/release machinery (job_wal_commit_claim/release,
claim_committed_job_submits, and their wiring): three attempts at preventing
the duplicate at submit/dispatch/commit time all failed to close the gap,
because the duplicate isn't a bug to prevent — a WAL entry can't be
retracted once appended, so any submit-time guard is structurally the wrong
layer.
Real fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now returns 1 for a
genuine creation vs 0 for an idempotent no-op (restore_job() -EEXIST), and
replay_wal() only tallies new_job_submits_out on rc==1. Updates the affected
apply-seam/reapply-idempotency tests for the new return contract, and
reverses two apply-correctness tests whose premise (duplicate-jobid replay
must diverge) is now understood to be the actual bug rather than the
expected behavior — they now force divergence directly via the same
production fencing seam instead of relying on that trigger.
Verified: two consecutive 10-round chaotic soaks, zero divergence, zero job
loss, no election stall (previously reproduced within ~5 rounds).
jobid_range_exists() (c->jobmgr) and the in-flight registry (job.c,
per-process, only populated by this node's own handle_jobsubmit calls)
both miss a real cross-node case: a client's XCECLI_FAILOVER_CALL retry
resends the same client-generated jobid after a connection drop, and the
resend lands on a newly elected primary that never processed the original
request. If the original was already committed before the old primary
died, the new primary's own local WAL already holds it (Raft Leader
Completeness) but jobmgr hasn't applied it yet — both existing guards see
nothing, so the resend appends a second, genuinely committed WAL_JOB_SUBMIT
entry for the same jobid. A live chaotic soak reproduced this exactly
(waldump: two committed entries, same term, matching apiserverId/timestamp/
sequence, one second apart) — apply's own idempotent dedup keeps jobmgr
correct, but replay's raw WAL_JOB_SUBMIT count no longer matches jobmgr's
growth, firing a false APPLY_DIVERGENT fence and deadlocking promotion.
Fix: apply_committed_entry()'s WAL_JOB_SUBMIT case now claims the jobid in
job.c's in-flight registry (new job_wal_commit_claim/release, wrapping the
existing inflight_range_try_add/remove) for the duration of dispatch,
regardless of whether this node originated the submission or is applying a
replicated/replayed entry. A resend landing mid-dispatch on any node now
sees the same -EEXIST rejection self-originated in-flight submits already
get. Release only fires when the claim call added the entry, so a
self-originated submission's own claim/release lifecycle (unchanged) is
never double-released or stolen.
Also fixes 4 pre-existing tests that broke under the earlier checkpoint_lsn
fix (0eb1054) and were missed in that commit's regression sweep: cluster-
less fixtures where last_applied_lsn never advances now correctly get
checkpoint_lsn=0 unless the test simulates apply catch-up, matching the
new (correct) semantics. One assertion in
test_checkpoint_tail_with_no_snapshot_is_apply_clean compared Phase 1's
checkpoint_lsn against Phase 2's independently-reconstructed physical WAL
tip — that equality only held under the pre-fix current_lsn-tracking
behavior and is removed, not patched around.
Also hardens run-ha-soak.sh's start_node() with setsid+nohup: a forked/
background test-harness invocation whose session tears down mid-run was
silently killing every controller process via SIGHUP forwarding (observed
twice — deploy completes, ~1s of activity, then all 3 nodes die together
with no error logged).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
xce_test's cmd_jsub hardcoded idgen(&idctx, 0, 0, ...) (clusterId=apiserverId=0),
and run-ha-soak.sh's submit_jobs() spawns a fresh xce_test process per retry
attempt, resetting idgen's static sequence counter each time. Two unrelated
job submissions from separate process launches landing in the same
wall-clock second collided on an identical jobid, surfacing as a spurious
APPLY_DIVERGENT fence in chaotic soak that was initially misattributed to
bug-1419's controller-side dedup logic.
Confirmed not a production risk: apiserver's GenerateJobID holds one
process-lifetime idgen_ctx_t via sync.Once, and each deployed instance has
its own distinct apiserverId, so config.h's jobid bit layout already
partitions the production namespace. Fixed by folding getpid() into the
harness's apiserverId so cross-process collisions become structurally
impossible there too.
Also adds test_jobsubmit_resubmit_after_full_resolution_rejected_no_second_job
to test_commit_barrier_reply.c, proving the post-resolution dedup window
(retry arriving after commit+apply+reply+release) is airtight, complementing
the existing before-resolution test -- both known bug-1419 dedup windows are
now provably race-free, confirming those fixes were never the cause of this
soak failure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
persist_checkpoint() stamped checkpoint_lsn from p->current_lsn (physical
WAL append tip) instead of p->last_applied_lsn (the true "jobmgr reflects
up through here" cursor), uncoordinated with the async apply pump
introduced by raft-apply-decoupling. write_snapshot() serializes live
jobmgr state, so a checkpoint taken while apply lags the WAL tip mislabels
the snapshot: on restart last_applied_lsn is re-seeded from checkpoint_lsn
and replay resumes strictly above it, silently skipping committed-but-not-
yet-applied entries jobmgr never received. This is the actual mechanism
behind kill-mid-apply producing a permanent APPLY_DIVERGENT fence —
apply_committed_entry() is deterministic/idempotent and write_snapshot()
is crash-atomic, so the bug was a mislabeled watermark, not a structural
divergence requiring a safety/liveness tradeoff.
Fix stamps checkpoint_lsn from last_applied_lsn and holds apply_pump_lock
across the label capture + write_snapshot() so the label and the jobmgr
state it describes are captured atomically w.r.t. the live apply pump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
jobid_range_exists() only checks c->jobmgr, but task 3.1 deliberately
defers job construction until after commit -- so a submission that is
durably WAL-appended and awaiting its commit outcome is invisible to
that check. A client-side retry (XCECLI_FAILOVER_CALL resending the
same jobid after a transport error or failover mid-flight) sails past
the guard and appends a second, non-identical WAL_JOB_SUBMIT entry for
the same jobid -- confirmed via waldump on a live chaotic-soak node
(duplicate submits, same jobid, differing subtime/CRC) -- which can
trip persist_recover()'s post-replay apply-correctness check into a
genuine APPLY_DIVERGENT verdict.
Adds a process-lifetime in-flight-jobid-range registry in job.c that
handle_jobsubmit atomically check-and-claims immediately after the
existing jobid_range_exists check, releasing from
jobsubmit_async_ctx_release (the single choke point already shared by
the commit-continuation, apply-complete, and OOM early-out paths). A
resubmit for a still-in-flight jobid now gets -EEXIST with no second
WAL append.
Adds a matching XCE_TESTING reset seam (declared in cluster.h) so a
test that deliberately leaves a submission's commit continuation
unresolved through teardown doesn't leak its jobid range into a later
test reusing the same range -- commit_engine_fini frees pending
continuations without invoking their callback.
Regression test added to test_commit_barrier_reply.c (RED->GREEN
confirmed). Full 78-group controller_unit_test sweep: 782 tests, 0
failures.
This fixes only the WAL duplicate-submit mechanism (layer 1 of
bug-1419). The deeper PreVote-freshness/apply-divergence liveness
deadlock (layer 2 -- no recovery path when the only fresh-enough node
is apply-divergent-fenced) remains open and needs its own design
discussion; buglog.json updated accordingly.
Chaotic mode's run_chaotic_rounds restarts "the previously-downed node"
every round unconditionally via start_node(\$DOWN), but a round that skips
(no primary to kill) never updates \$DOWN -- so a stale node name reaches
start_node again while its own earlier restart is still alive, stacking a
second xce_controller on the same --myname/port without ever killing the
first. Six stacked ctrl-2 processes were observed by the end of a 10-round
run.
Fixed inside start_node itself with a pgrep guard, rather than reworking
run_chaotic_rounds's DOWN bookkeeping, so every call site is covered:
refuse to launch a duplicate if "xce_controller --myname \$1" is already
running.
Verified: re-ran the identical 10-round --chaotic soak that surfaced this.
The guard fired on every skip-round; exactly 3 xce_controller processes
alive throughout the entire run.
Note: the same soak run still hits a genuine, separate election-liveness
stall (a surviving node fenced non-promotable by a real apply-correctness
divergence, traced to non-identical duplicate WAL submits for the same
jobid) -- filed as bug-1419, not fixed here.
vote_reply_dispatch passed the compressed, self-excluded local votes_idx
(vctx->peer_idx, set by raft_cb_send_vote_request's absolute->local remap)
straight through to raft_on_vote_reply, which indexes vote_replies[] in
ABSOLUTE cluster space -- the same space raft_start_election's self-mark
and per-peer send loop use.
For a 3-node cluster's middle node (self_idx=1), the peer at absolute
index 2 compresses to votes_idx=1, colliding with vote_replies[self_idx=1]
-- the slot raft_start_election already marked for the self-vote. The
library's per-peer dedup guard then silently drops the genuine reply as a
duplicate, so votes_granted never reaches quorum even though the peer
granted. Peer 0 is unaffected (compressed index equals absolute), which is
why sequential elections and prior soak runs never surfaced this -- only a
middle-priority node losing its higher-priority peer and depending on its
lower peer's vote hits the collision.
Fixed by routing through ha_repl_ack_abs_peer_idx (already existed,
already unit-tested, already used for the analogous heartbeat-ACK path)
before calling raft_on_vote_reply. ha_record_vote_reply immediately above
is unchanged -- it correctly wants the compressed index for ctx->votes[].
Confirmed pre_vote_reply_dispatch needs no equivalent fix: its send path
already stores the absolute index.
New regression test drives a real 3-node election end-to-end through the
actual decode/dispatch path (not the test-hook bypass every other
real-vote test in this file uses): Expected 2 Was 1 pre-fix, GREEN
post-fix. Full 79-group suite: zero new failures. 10-round --chaotic HA
soak: zero committed divergence, zero committed job loss.
Closes the two non-blocking follow-ups from the 2026-07-07 validate-impl
audit:
- design.md had 3 sections describing the original plan rather than what
landed, each already self-disclosed in tasks.md's Implementation Notes
but never reconciled back: repl.c's planned repl_submit_majority_commit()
was never built (task 3.2 calls commit_submit_async() directly instead);
the apply pump is wired via commit_resolve_pending() in commit.c, not
ha.c's on_commit_advanced; fact hooks fire one generic
HOOK_DETAIL_JOB_UPDATE, not per-type hooks (verified safe -- sched.c's
subscribers already consume the generic form).
- task 5.3 touched ha.c (two promotion/fence call sites) without a formal
_Boundary:_ exception the way task 1.4 got one for its own out-of-file
touch -- added retroactively, no behavior change.
controller-status, ha-commit-engine, ha-heartbeat-committed-propagation,
ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and
raft-ha-core-safety each had every numbered sub-task marked [x] but their
group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic
gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real
unchecked sub-tasks in any of the eight files. No task content changed.
raft-apply-decoupling: 32/32 tasks complete, validate-impl GO (79/79 test
groups, live smoke boot verified) -- update roadmap's In Flight entry with
outcome, residuals, and follow-ups.
raft-drive-async: all 17 sub-tasks were already [x] but the 4 phase-header
checkboxes were never ticked, leaving tasks.md looking incomplete. Confirmed
via source (g_drive_async_configpush_offload defaults to 1) and prior
session notes that all 4 phases genuinely landed and flipped ON in
production; checkboxes now match reality.
Measured sustained worker-fact throughput against the pre-feature baseline.
Commit throughput is unregressed (~2.5% delta, within noise) and no per-fact
synchronous commit barrier exists, confirming Req 11.4's core claim. But
apply/observable throughput -- the new hop this spec introduces -- regressed
56% with clearly superlinear scaling (2x volume -> 5.2x elapsed).
Root cause: apply_pump_read_entry has no memory across calls -- every LSN
lookup re-lists WAL segments and rescans its segment from byte 0 -- while its
only caller applies committed entries one at a time in strict ascending
order, turning a sequential access pattern into O(n^2) total work.
Fixed with a minimal segment-read cache on persist_t (fd + offset + expected
next LSN, guarded by the existing apply_pump_lock): a fast path reads the
next expected LSN directly at a remembered offset, falling back unchanged to
full rediscovery on any miss. Correctness never depends on the cache, only
performance does. Invalidated on init, fini, recovery, and WAL truncation --
the four points a stale cursor could otherwise misread from.
Two review rounds: round 1 caught a missing lock guard in the recovery
invalidation path and an inaccurate risk claim about the cache/trim
interaction (a real, bounded behavior change deferred to the existing
APPLY_DIVERGENT fence, not "no worse than before") -- both corrected and
re-verified in round 2. Post-fix apply-throughput scales sublinearly (1.46x
elapsed for 2x volume) and exceeds the pre-feature baseline. Zero new test
failures across the pump, correctness-verification, recovery, and truncation
groups.
An intermittent, pre-existing client hang in the synchronous jobsub path and
a trim-floor/apply-lag architectural gap were surfaced but are explicitly
out of scope for this fix -- flagged for separate follow-up.
Full 79-group unit/integration suite re-run found exactly 2 stale tests in
test_commit_barrier_reply.c asserting task 3.2's documented interim
contract ("no observable effect until task 6 completes it"), which the
already-committed ordered apply pump (tasks 6.1/6.2) closed by construction.
Updated both to assert the real, deterministic apply-after-commit settled
state instead of the no-longer-true no-effect-yet state, matching the idiom
already used elsewhere in the same file. No production code changed.
Full suite: 0 failures across 79 groups. HA soak (--sequential, 5 rounds):
zero committed-job loss, zero divergence, all 3 nodes served as primary.
Both independently re-verified by adversarial review.
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.
Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.
Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
Extend raft_sim.c's deterministic TLA+-conformance harness with two
fault-injectable checks: no reaction before commit, and idempotent
re-apply of an already-committed entry. Wired as dedicated tests only,
not into raft_sim_check_all's fixed four-invariant contract (owned by
a separate spec). RED/GREEN independently verified by adversarial
review; zero regression to existing scenario functions.
Extend test_apply_pump_integration.c with 2 tests, no production
changes. Test 3 interleaves commands and facts through a real
quorum-loss outage and proves strict original-log-order catch-up via
a sequence spy. Test 4 wires a real 4-vcpu vtaskmgr_t to drive 4
commands + 2 facts through one covering ACK, creating genuine N-way
concurrent races across task 6.2's self-healing fix rather than one
isolated forced pair -- stress-testing bug-1415's fix under a
realistic catch-up burst.
Closes out task-group 6 (ordered apply pump integration): the pump
is now live, hands off command replies correctly, and is proven
correct for isolated and mixed/bursty command+fact scenarios under
both normal and quorum-loss conditions.
Add test_apply_pump_integration.c (2 tests), no production changes.
Exercises the full live pump (real commit engine, real apply
dispatch, real hook subscribers) for both a job submission and a
fact: proves reply-after-visibility ordering for commands, and zero
fact-reaction side effects during a delayed-commit/quorum-loss
window (polled repeatedly across real time, not checked once).
Confirms Req 4.3 by construction, not assertion: the apply walk only
ever invokes apply_committed_entry() for LSNs already <= committed_lsn,
so a consumer acting on the post-apply notification never needs to
separately re-check a commit watermark.
Independently verifies apply_committed_entry()'s generic
HOOK_DETAIL_JOB_UPDATE routing (unchanged by task 6.1's _pad fix)
makes the type-specific sched.c/apiserver.c hook handlers
structurally unreachable for live-applied facts -- confirmed their
va_arg-typed signatures would type-confuse-crash if ever called with
the job_t* apply_committed_entry() actually passes, making the
single generic reaction spy a sound design, not a coverage gap.
Repo-wide audit confirms exactly 2 live call sites ever fire these
hook details (the guarded apply seam, and job_ctrl's legitimate
SIGCONT/SIGSTOP path) -- no orphaned third site.
Wire persist_apply_pump_run() to call persist_take_pending_apply()
immediately after each entry's apply_committed_entry() call,
invoking the registered continuation strictly after the mutation.
Adds jobsubmit_reply_success_once() (job.c) so a successful job
submission finally sends a client reply once every array element
applies -- closes the standing gap from task 3.2 where committed
submissions never got a response.
Fixes bug-1415: commit_resolve_pending() dispatches the commit
continuation and the apply-pump kick as two independent
VTASK_LIVE_BLOCK vtasks with no completion-order guarantee. On any
real multi-vcpu pool (the production default, n_vcpus=8), if the
apply pump won the race and advanced last_applied_lsn past an LSN
before its continuation registered, the registration was permanently
orphaned -- the job would be constructed and visible, but the client
would hang forever. Untested by every existing fixture in this spec
(all use pool=NULL, which trivially serializes and can't reproduce
the race).
Fixed with a self-healing check in persist_register_pending_apply():
if the LSN is already <= last_applied_lsn, invoke the callback
immediately instead of registering for later pickup. Verified by the
spec's first genuine multi-vcpu regression test (real vtaskmgr_t,
n_vcpus=4), independently RED->GREEN reproduced by two review rounds.
Build persist_apply_pump_run()/persist_apply_pump_kick() -- the
ordered walk from last_applied_lsn+1 through committed_lsn, applying
every entry via apply_committed_entry() regardless of commit mode.
This is the first live (non-replay) consumer of that dispatch
function. Kick is wired into commit_resolve_pending() (the single
choke point all 6 internal committed_lsn-advancing paths funnel
through, discovered in task 1.2), dispatched via
vtask_enq_classified(VTASK_LIVE_BLOCK) with a CAS-coalesced
fire-and-forget trigger.
Fixes the two hazards flagged since task 4.1 as required before
going live:
1. Double-append: a thread-local call-provenance guard
(t_apply_seam_notifying) prevents persist_job_hook_fun/
persist_queue_hook_fun/persist_worker_hook_fun from re-appending
entries the apply pump itself just applied, while preserving
job_ctrl's legitimate live command-path appends (a naive
detail-based filter would have broken SIGCONT/SIGSTOP). Also
closes an undocumented instance of the same hazard in the
queue/worker hook functions.
2. Merge-staleness: record_job_fail_fact/postdone's fact recording
previously baked a receipt-time-computed final job state into the
WAL entry, which goes stale under deferred apply -- a FINISH
followed shortly by an ordinary post-exec FAIL could silently
clobber a correct DONE completion with a stale RUN|EXIT|PERR, no
guard catching it. Fixed by repurposing wal_jobstate_t's unused
_pad field to carry just enough data (FAIL's stage) to recompute
against the job's actual current state at apply time via the real
apply_job_fail_transition()/new apply_job_postdone_transition()
guard cores, instead of overwriting with a stale absolute value.
Backward compatible: _pad==0 (all prior data) routes to the same
legacy overwrite behavior.
Also fixes bug-1409 (unbounded busy-loop in apply_pump_drain when
committed_lsn advances without a corresponding on-disk entry) and a
genuine test-flakiness hazard (a second, independent apply-pump
kicker from the fsync_worker thread that existing tests didn't
account for).
Two pre-existing tests in test_commit_barrier_reply.c now fail as
expected -- their own assertions state "no job may be constructed
until task 6's apply pump exists," now correctly invalidated by this
task landing; updating them is task 7.3's explicit job.
Add explicit test coverage for the DIVERGENT fence's stay-alive and
restart-only-clear guarantees (task 5.3), which surfaced a critical,
pre-existing safety gap: the sticky recovery_unrecoverable flag is
trigger-agnostic, and two pre-existing live call sites in ha.c
(promotion_fence_timeout_fn's periodic timer, and
raft_cb_become_leader's promotion-eligibility gate) clear it based
purely on structural conditions, never checking the apply verdict.
A node whose local apply dispatch diverges from the committed log,
but whose WAL bytes are otherwise structurally healthy, would have
its fence silently released by the live timer, then be eligible for
promotion to primary on a later election -- serving genuinely
divergent state.
Fix: gate both clear-call-sites on persist_apply_verdict(&c->persist)
!= APPLY_DIVERGENT in addition to their existing structural checks.
An apply-divergence-triggered fence can now only clear via a fresh
process restart's clean persist_recover() replay.
New tests: test_promotion_fence_apply_divergent_survives_structural_catchup
(test_commit_durability.c) covers the timer site; and
test_promotion_refused_when_apply_divergent_even_if_eligible
(test_convergence_verification.c) covers the promotion-gate site,
structurally isolated so it cannot pass for the wrong reason.
Closes out task-group 5 (restart apply-correctness verification).
Add bool apply_divergent to the existing recover_policy_inputs_t;
persist_recover_decision() unconditionally short-circuits to the
existing RECOVER_DECISION_FENCE_UNRECOVERABLE when true, reusing the
same sticky recovery_unrecoverable flag and
persist_recover_loud_fence_if_unrecoverable() -- no new fence
mechanism, no ha.c changes needed (the existing promotion gate
already reads the flag independently), no new self-termination path.
Fixes a real bug found during review: the fence's operator-facing
ERROR() log unconditionally asserted a "no healthy peer reachable...
durable_floor_lsn < hwm" clause that was only truthful for the
pre-existing structural-fence trigger. For the new apply_divergent
trigger this produced literally false claims in test output
("100 < 100", "2 < 0"). Fixed by branching the log call itself on
apply_divergent so it states only the verified reason.
Add apply_verdict_t (APPLY_CLEAN/APPLY_DIVERGENT) computed alongside
the existing structural (CRC) recover_verdict_t. Dual-check design:
an entity-count check (WAL_JOB_SUBMIT entries scanned vs live jobmgr
growth, genuinely independent of apply_committed_entry()'s return
code) and a position check against a new last_applied_lsn field
(forward-looking infrastructure for task 6's live pump).
Fixes a real bug found during review: the position check's
expectation was computed from the physical WAL-scan tip
(current_lsn - 1), which advances past checkpoint markers and
unknown entry types that never reach apply_committed_entry(). A
crash between persist_checkpoint()'s marker-fsync step and its
later snapshot-persist step leaves a trailing checkpoint marker on
next boot, which falsely reported APPLY_DIVERGENT on a perfectly
clean recovery -- reachable today, not gated behind a future task.
Fixed by reading last_applied_lsn directly, which only advances for
entries actually dispatched through the apply seam.
Known accepted gap: entity-count check covers jobs only, not
workers (which can genuinely be created via WAL_WORKER_CTRL) --
documented for future extension.
Add test_fact_state_unchanged_throughout_quorum_loss_outage to
test_commit_barrier_reply.c (30->31), no production changes. Drives
a real repl_remove_standby quorum-loss window through all 4
fact-receipt handlers, polls live state and committed_lsn at 5
time-spread points (not once), asserts a real hook-spy never sees
AFTER_EVENT fire, and confirms durable WAL presence via direct
on-disk read.
Documents an architectural asymmetry for task 6 planning: facts have
no commit-timeout concept (COMMIT_MODE_RETAIN registers no
continuation, "never honors the timeout") unlike commands'
commit_submit_async -- the apply pump must discover newly-committed
retained facts purely via committed_lsn advancement, not any
per-fact signal.
Closes out task-group 4 (fact apply-after-commit split).
Convert on_job_start/on_job_finish/on_job_fail/on_job_postdone to
validate and durably append only -- BEFORE_EVENT validation hook
unchanged, AFTER_EVENT (post-mutation) hook no longer fires at
receipt, live state mutation removed. Adds persist_submit_job_fact()
(persist.c) since the durable append previously lived inside the
very AFTER_EVENT hook this task silences; wire format is unchanged
and identical to what apply_committed_entry() already routes.
mark_job_start/mark_job_finish (single caller each) renamed in place
to record_job_start_fact/record_job_finish_fact. mark_job_fail/
mark_job_update (shared with job_ctrl's client-command path, out of
boundary) left untouched; new durable-append-only siblings
record_job_fail_fact/record_job_update_fact added instead.
Accepts a narrow, permanent regression: on_job_postdone's finish_time
anomaly backstop has no wire-format equivalent and is dropped
(affects only a rare no-prior-exit-status path).
Two critical hazards discovered by review, documented in tasks.md,
MUST be resolved before task 6 wires apply_committed_entry() into a
live path: (1) persist_job_hook_fun still fires on HOOK_DETAIL_JOB_UPDATE
and will double-append every live-applied fact; (2) record_job_fail_fact/
record_job_update_fact precompute a final state that can be silently
clobbered by a concurrent intervening fact once apply is deferred by
the commit round-trip.
Add 4 tests to test_commit_barrier_reply.c (25->29), no production
changes: explicit-FAILED via truncation (distinct from timeout),
repeated failure loops (5 cycles) for both timeout and truncation,
and an array_size>1 mixed-outcome test proving a committed sibling's
apply hand-off isn't blocked by failing siblings while producing
zero phantom jobs for the failures and exactly one batch-level
failure reply.
Closes out task-group 3 (command apply-after-commit ordering).
Replace handle_jobsubmit()'s blocking commit call (task 3.1) with a
real async registration against commit_submit_async(). On
FAILED/timeout, replies failure immediately with no job constructed.
On COMMITTED, hands off to a new pending-apply registration
(persist_register_pending_apply) for the ordered apply pump (task
6.1, not yet built) to consume -- does not apply or reply directly,
per design.
Deviates from design.md's stated integration site: appends via the
same ha_primary_log_append/persist_wal_async primitive the RETAIN-fact
path already uses and calls commit_submit_async directly on
p->repl.ce, bypassing repl_wait_majority_ack/repl_submit_majority_commit
in repl.c. Verified safe -- majority-ack detection is driven entirely
by (ce, lsn) events inside the commit engine, agnostic to which
wrapper registered the continuation -- but narrows the reusable
primitive design.md intended; documented in tasks.md.
Deletes (not preserves-dead) task 3.1's bug-1408 notification fix --
apply_committed_entry()'s WAL_JOB_SUBMIT case (task 2.2) already
fires the notification per-entry, immune to the batch-suppression
class bug-1408 fixed.
IMPORTANT: no live path applies a successfully-committed job
submission until task 6 lands (only restart replay does) -- every
successful submission commits durably but gets zero reply until
then. This commit range must not be soak-tested or deployed against
live traffic before task 6 completes.
Reorder handle_jobsubmit() so job objects are constructed via
create_job() only after the WAL commit succeeds, not before. Adds
jobid_range_exists() for a fast-fail pre-check with no WAL work, and
persist_submit_job_payloads_and_commit() (persist.c) operating on raw
xce_job_t* payloads instead of pre-existing job_t*. Commit remains
synchronous -- converting to the continuation-based commit API is
task 3.2.
Fixes bug-1408 (found during review): the post-commit array-
construction loop suppressed notification for an entire array batch
if any one sibling element failed to construct, stranding other
durably-committed and successfully-constructed jobs invisible until
the next restart's replay. Notification now fires unconditionally
for every successfully-constructed element, independent of sibling
failures; the client reply still separately reports the batch's
construction status.
Add test_apply_reapply_idempotency.c (3 tests, no production changes):
triple-reapplication of an identical job submit with hook-count and
pointer-identity assertions across every call, the NOOP_DUPLICATE
guard branch (untested by task 2.2), and the task's named scenario --
real SUBMIT then real FINISH then a stale out-of-order START, all
through the real apply_committed_entry() dispatch -- confirming the
job stays terminal with no spurious notification.
Closes out task-group 2 (shared apply seam for committed entries).
Rewire replay_wal()'s per-entry apply call from apply_wal_entry() to
apply_committed_entry() (task 2.2's shared dispatch seam), so restart
replay and the future live commit-advance pump (task 6.1) both
produce materialized state deterministically from the committed log.
apply_wal_entry() is not retired -- it remains ha.c's live standby
catch-up/replicated-apply dispatch, untouched.
Documents a real hazard: replay now fires AFTER_EVENT hooks for the
full post-checkpoint WAL range on every restart, including entries
already applied+notified in a prior process life. Traced and
confirmed inert today for both hook consumers (scheduler registers
post-boot, apiserver gates notify on a connected endpoint empty at
boot); flagged in tasks.md and in-line at the call site for
re-verification before tasks 4.1/6.1/6.2/6.3 make this path broadly
live.
Add apply_committed_entry(persist_t*, wal_type, lsn, payload, len) in
persist.c/persist.h as the single seam that turns a committed WAL
entry into an in-memory mutation plus its post-mutation notification
hook, covering job submit, job-state facts, queue control, and
worker control. Job-state facts route through 2.1's guard-only
apply_job_update_transition() core and fire a generic
HOOK_DETAIL_JOB_UPDATE, not per-type hooks — the wire format
(wal_jobstate_t {jobid, state}) cannot supply what the type-specific
cores/hooks require. This is a real, documented deviation from
design.md's per-type dispatch wording; tasks.md's Implementation
Notes flags it as unresolved and blocking for whoever wires this
into a live path (4.1/6.2/6.3).
Not yet wired into any live call site (replay wiring is 2.3).
Split mark_job_start/finish/fail/update in job.c into four hook-free
core functions (apply_job_*_transition) containing only the
monotonic-rank guard and mutation. Wrappers now peek the same guard
to decide hook-firing, then delegate to the core, preserving exact
pre-extraction hook-firing timing. Prerequisite for task 2.2's
apply_committed_entry() dispatch and the ordered apply pump (6.1).
Closes out task-group 1 (the commit-engine event-driven-resolve
keystone). Pure test-only: real pthreads racing the admission gate,
the sweep thread, and commit_on_ack, not sequential calls dressed as
concurrent. Adds the genuine two-real-thread ack-vs-sweep-thread race
both task-1.3 review rounds flagged as a nice-to-have (the existing
race test only proved the tie-break via single-thread white-box state
manipulation).
Also resolves the bug-1407 saga: task 1.4's "closed, 70/70" and this
task's initial "reopened, 10/10 failing" were both correct,
contradictory measurements of the identical source commit. Root cause
was a stale incremental-build artifact in the shared build/ directory
(ninja -C build rebuilt incrementally across 5 sequential tasks
without ever a fresh meson setup) -- not a source regression. Proven
by isolating each measurement in a throwaway git worktree with a
fresh build. bug-1407 is genuinely closed; the incremental-build
staleness gotcha is now documented in cerebrum.md for the remaining
task series.
commit_resolve_pending_above(ce, lsn_floor, rc) fails every registered
continuation for lsn > lsn_floor, reusing commit_resolve()'s existing
exactly-once/idempotent guarantee (no parallel bookkeeping path).
Truncation: a new on_truncate/truncate_cb_ctx callback on persist_t
(same inversion-of-control shape as the existing on_durable_advance
hook) fires from persist_truncate_wal_above after releasing p->lock,
only when entries were actually removed. This one wrapper covers all
three real truncation call sites: raft_cb_become_leader's
post-election truncate, the library's on_truncate_from
conflict-reconcile, and ha_demotion_convergence_fn's post-demotion
truncate.
Demotion: raft_cb_become_follower calls
commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY) --
lsn_floor=0 fails everything, since a demoted node can never
authoritatively confirm anything still pending (CP posture).
Without this, a demoted or truncated node would leave any pending
async command permanently unresolved -- the requester would simply
never get a reply, worse than today's bounded-timeout behavior.
Also closes bug-1407 (pre-existing test_config_repl flake, unrelated
to this feature) -- confirmed fixed as a side effect of prior tasks
in this series, 70/70 passing across 3 independent runs.
Converting COMMIT_MODE_WAIT away from a blocking wait removed the only
mechanism that ever detected a commit timing out (the caller's own
pthread_cond_timedwait deadline). Adds a dedicated per-commit-engine
sweep thread that arms an equivalent timeout for async-registered
continuations, resolving FAILED/-ETIMEDOUT within the same
DEFAULT_SYNC_TIMEOUT_MS window the old blocking path used, with no
thread parked to detect it.
Chose a private sweep thread over the shared delaymgr: wiring
delaymgr would touch cluster.c/ha.c (out of boundary) and its 200ms
poll granularity vs the 500ms default timeout would add jitter,
breaking the "identical failure timing" no-regression bar.
Round 1 review rejected a lost-wakeup race: the sweep loop computed
its next-deadline snapshot under lock, unlocked to dispatch, then
slept on the stale snapshot -- a registration landing mid-dispatch
had its broadcast lost (no thread parked on the condvar), so the
sweep thread could sleep on an absent/late deadline indefinitely.
Fixed by holding the lock continuously across scan-then-wait
decisions and always re-scanning immediately before ever waiting
(recheck-then-wait idiom). Round 2 review independently reproduced
the round-1 bug shape against the fix to confirm it actually closes
the race, not just narrows it.
commit_resolve() delivers a registered continuation's outcome exactly
once (idempotent re-resolve), dispatched off-thread via
vtask_enq_classified(VTASK_LIVE_BLOCK) so it never runs inline on the
replication-ACK or iomgr path. Wired into all 6 commit.c-internal
paths that can advance committed_lsn/writable_lsn.
Review caught a real task-graph gap, not a defect: converting
COMMIT_MODE_WAIT away from a blocking wait also removes the only
mechanism that ever detected a commit timing out (the caller's own
cond_timedwait deadline). Nothing yet arms an equivalent timeout for
async-registered continuations, and nothing wires demotion/log-truncation
detection to commit_resolve(FAILED, ...) -- left as-is, a pending async
command during quorum loss would hang forever instead of failing,
which is worse than today. Split into new tasks 1.3 (timeout arming)
and 1.4 (demotion/truncation wiring), renumbered the old concurrency
test to 1.5, and made 3.2 depend on both since its failure/timeout
observable is meaningless without them. design.md's File Structure
Plan updated to reflect that ha.c/persist.c need a narrow touch for
1.4's detection hooks.
commit_submit_async() lets a WAIT-mode caller register a continuation
against an appended LSN instead of blocking on pthread_cond_timedwait,
reusing the existing admission-cap discipline (wait_admit_locked/
wait_release_locked) so the bounded-capacity guarantee carries over
unchanged. Purely additive: no existing call site (repl_wait_majority_ack)
is touched yet -- that switch lands in task 3.2 once commit_resolve()
(task 1.2) exists to drive it.
Full Kiro spec for the apply-after-commit refactor: canonical
append->commit->apply->notify ordering, a single apply_committed_entry
seam shared by live commit-advance and restart replay, and the
Facts-Apply invariant (ha-consistency-invariants criterion 2.9)
reconciling worker-reported facts onto the same seam.
buglog.json had two broken auto-detection hook signatures polluting it:
- 207 entries "Missing error handling in X" / fix "Added try/catch block"
(nonsensical on .md/.capnp files, and C has no try/catch — never a real
applied fix)
- 452 entries with a literally-empty template ("Significant refactor of "
with no subject ever filled in) / fix "Rewrote N->M lines"
Removed both (659 total). Also merged 24 legitimate duplicate bug IDs
(re-logged/updated entries sharing an ID) into single entries — kept the
most recent content as primary, unioned tags/related_bugs, summed
occurrences — and stripped noise tags ('auto-detected', 'refactor') that
had leaked onto real entries from a since-merged junk duplicate. Net:
1406 -> 721 entries, verified zero remaining duplicate IDs and zero
remaining junk signatures.
.gitignore: added patterns for the scratch soak-run logs and codegraph.html
this session generated at repo root (already deleted from the tree), and
fixed the ucai-formatter-cache.local.json pattern to match its nested
occurrences (.claude/skills/*/.claude/, src/*/.claude/) instead of only the
repo-root one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
All 4 sub-specs are complete: ha-commit-engine (18/18), ha-commit-barrier-
rollout (23/23), ha-single-leader-role (17/17), ha-convergence-recovery
(19/19). Reconciles ha-single-leader-role and ha-convergence-recovery's
spec.json phase to "implemented" (both mislabeled tasks-generated despite
full task completion — same staleness pattern fixed for raft-commit-safety/
raft-log-replication earlier).
Promotes the umbrella itself to "implemented": design.approved corrected
true (was false, contradicting its own note "Design approved as contract"),
phase implemented, with an explicit callout that criterion 2.9 (Facts-Apply
invariant, added 2026-06-25) remains a documented deviation owned by the
still-unstarted raft-apply-decoupling spec — not a gap in this umbrella or
its sub-specs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
bug-847 (catch-up WAL batch drops payloads across capnp segment boundaries):
its purpose-built RED reproducer, test_wal_batch_roundtrip_large_payloads
(14 entries x 872-byte payloads, 12.2KB total, genuinely spans multiple
4096-byte capnp segments per capn-malloc.c's create()), was still
TEST_IGNORE_MESSAGE'd. Un-ignoring it and running under ASAN with leak
detection: clean pass, byte-exact payloads, zero leaks, 2/2 runs. Traced the
vendored c-capnproto far-pointer machinery (write_ptr/capn_setp in capn.c)
to confirm the cross-segment struct->Data reference path is correct by
construction — whatever regressed this already got fixed elsewhere (library
bump or an incidental request.c change); the ignore was just never removed.
No production code changed.
bug-1307 (task-5.7 election-jitter data-safety regression: 7 divergences +
4 job losses over 6 chaotic rounds): re-ran the exact reproducing scenario
(6-round chaotic soak, real ext4, ASAN binaries) — 21 jobs, 0 divergence,
0 job loss, clean PASS. Confirms the raft-ha-core-safety Phase 5 work
resolved it.
Both bugs were the last 2 of the 7 flagged in the prior staleness sweep;
all 7 are now closed (6 were already fixed and undetected, 1 needed the
test un-ignored).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Verified against live code/tests instead of trusting the buglog text:
- bug-846 (EINVAL on dead ctx): FIXED — xcecli_retry_decide checks ctx.state
== XCE_CTX_ERR before classifying, bypassing the misclassification.
- bug-855 (6 wal_catchup tests failing): FIXED — ran controller_unit_test,
all pass.
- bug-1096 (stale-term vote-reply counted): FIXED — raft.c has the
reply_term != s->term drop.
- bug-1147 (WAL trim ignores standby cursor): FIXED — trim_wal_segments has
the cursor-bound floor guard, comment cites this bug ID directly.
- bug-1198 (snapshot-send liveness-pool hazard): FIXED — in-code CLOSED
banner, bug-1197's endpoint ref/unref bracket makes it unreachable.
- bug-1307 (election-jitter safety regression): left OPEN with a downgrade
note — circumstantial evidence (raft-ha-core-safety Phase 5 soak-clean)
but no direct code citation found; wants one more confirmation soak
before closing.
- bug-847 (catch-up WAL drops payloads at scale): RE-CONFIRMED genuinely
still open — its RED reproducer is still TEST_IGNORE'd.
Also noted (not acted on, out of scope for this pass): buglog.json has 26
duplicate bug IDs and ~239 "Missing error handling in unknown" / "Added
try/catch block" auto-detected junk entries — a separate cleanup task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Accidental npm-init artifacts from an earlier checkpoint commit (empty
package.json, auto-generated lockfile) in a pure C/Go project with no
Node.js build step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
ha-commit-durability: closes the "pending confirmation under simultaneous
total outage" caveat. Reproduced cold-restart-all (kill all 3 members +
restart) 5 times against a live 3-node cluster on genuine ext4 (not the
sandbox's overlayfs) — zero committed-job loss across 40 rolling-failover
rounds + 5 cold-restart-all cycles. Confirms bug-795 was an overlayfs
fdatasync artifact, not a real durability gap; no production fix needed.
Commit Durability is now proven for both rolling single-node failover and
simultaneous total-cluster outage (src/controller/CLAUDE.md updated).
Live-validated and marked complete against a real 3-node cluster:
- controller-status 6.4: off-primary committed fidelity + responder
timestamp, confirmed via a real failover
- ha-heartbeat-committed-propagation 4.1: standby committedLsn tracks the
primary via heartbeat within the 3s interval
- ha-wal-replication 5.4: standby WAL replication verified via committedLsn
match (job-facing ops are correctly primary-only by design; standbys
return E_XCE_STANDBY, not a bug)
- libxce-failover 6.2: live 3-node failover E2E, jobsub raced against a
primary kill resolves to exactly one job, no duplication
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Adds the Raft Apply-Ordering & Hook Decoupling roadmap phase: canonical
pre-hooks -> raft commit -> apply -> notify ordering, with the raft protocol
step lifted out of the notification hook fan-out (root of the bug-795/828/
851/1060 divergence family). Decision: apply-after-commit via a single
shared apply_committed_entry() seam reused by online apply and restart
replay.
- raft-drive-async: full spec docs (brief/requirements/research/design)
committed alongside its already-tracked tasks.md; async-drive prerequisite
for the apply-decoupling work, sequenced to start first (threatens
failover liveness today).
- raft-apply-decoupling: new spec (brief + facts-in-raft.md) covering the
apply seam and the facts (WAL_MODE_RETAIN) apply-timing question (open
decision D1).
- job-array-single-wal-record: promoted from backlog to a tracked spec,
sequenced after raft-apply-decoupling.
- ha-consistency-invariants: add criterion 2.9 (Facts-Apply invariant),
documenting the current on-receipt fact-reaction deviation this family
will reconcile.
- raft-commit-safety / raft-log-replication: reconcile spec.json phase to
"implemented" (both were mislabeled tasks-generated despite full 18/18
and 24/24 task completion + raft_sim GO).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
bug-1401: xce_free_ctrlstatus(s) was defined as free_CtrlStatus(s), which
only frees the fields inside xce_ctrlstatus_t. The real client reply path
(decode_CtrlStatus_ptr) calloc's the struct itself, so every ctrlstatus call
leaked sizeof(xce_ctrlstatus_t) (120 bytes). Existing unit tests never caught
it because they built/freed the struct via the raw stack-decode + bare
free_CtrlStatus() helpers, never exercising the actual client macro. Fixed
by switching the macro to free_CtrlStatus_ptr(&(s)) and added a regression
test that exercises the real heap-decode path; RED confirmed the exact
120-byte/1-alloc LeakSanitizer signature before the fix.
bug-1400: the originally reported SEGV in cmd_jobsub on malformed positional
input is stale — NULLSTR guards added 2025-11-05 already reject it cleanly.
The real residual defect was submit.start_info leaking on all 4 early-return
validation paths; fixed by routing them through a shared `goto cleanup`
(matches the existing goto-clean idiom in queue_ctrl).
bug-1396 (job-count anomaly after cold-restart-all, not yet root-caused):
instrumented test-commit-durability.sh's assert_job_count() to dump the live
joblist and ACKED_FILE ledger on any count mismatch instead of only the
count, per the bug's own recorded next step.
Verified: controller ASAN build + full controller_unit_test green; lib
xcelib_test 285/285 green, 0 ASAN leaks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
