The commit-engine committed_lsn advance was TRACE-only; promoted to INFO (fires only
on a real advance, not noisy) so operators/tests can observe the authoritative primary
committed watermark. The clean-failover harness now polls this 'commit: committed_lsn
N -> M' line (not the lagging ha_process_standby_state heartbeat) with generous catch-up
headroom.
This made the live RED->GREEN visible: with Option A, committed_lsn reaches 5 on a
stable cluster ('jobs committed on a majority (committed_lsn=5 >= 5)') even via catch-up
delivery — the commit-stall is fixed.
Two coupled fixes for a live commit-stall (committed_lsn pinned at the own-term
NOOP while a 3-node cluster was durable to LSN 5 — jobs acked but never committed):
bug-991: the WAL catch-up applied_lsn was advanced to lsn+1 (next-to-apply) in
the three live/catch-up apply paths, but recovery/checkpoint/init and the leader's
catch-up streamer treat it as last-held. The over-advertise made the streamer skip
an LSN -> library logOk wedge -> no WalAck. Fixed to last-held (= lsn).
Option A (design reconsideration of REVIEW ISSUE 3): removed the
committed = min(durable-majority, raft_commit_index) ceiling. Under single-writer
the durable-majority watermark is already (term,lsn)-current-term-safe (term-carrying
replication + (term,lsn) reconcile/truncate-before-durable + own-term NOOP gate +
post-election truncate-to-safe), so the library commit_index was a redundant,
starvation-prone gate that pinned committed below the safe durable majority whenever
the library lagged. The durable-majority barrier + own-term writable gate are now
authoritative; raft_commit_index is inert-advisory.
Safety: reviewer-verified the durable-majority watermark is (term,lsn)-safe and
Figure-8 holds via the own-term gate (new RED->GREEN test); bug-851 cannot recur.
Live: commit trace 4->5 + walcompare committed-prefix identical across 3 nodes.
177 unit tests pass.
Ran clean_failover_preservation.sh on a real fs ($HOME/tmp/xce-ha-deploy, ext4).
PAYLOAD run CONFIRMED bug-990 fixed live: standbys persist the leader's real
term (2) for the committed prefix; walcompare OK.
Harness fixes: robust failover detection (any post-kill CTRL_PRIMARY promotion,
45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept not
commit, so the prior harness raced replication vs kill).
That commit-wait gate exposed bug-991: on a STABLE cluster, jobs replicate to
all members (durable LSN 5) but committed_lsn pins at 1. Catch-up-delivered
entries don't advance the library commit_index, so task-2.4's ceiling
min(durable-majority, raft_commit_index) starves commit below the durable
majority. A real regression from the commit-source switch (2.4+3.3), invisible
to the steady-state-only unit tests.
Closes bug-990: the live replicator repl_broadcast wire-encoded term=0
(entry.term was never set), so standbys persisted hdr.term=0 and took the
legacy direct-apply path instead of the library (term,lsn) intake. Now the
live broadcast and catch-up streamer carry the leader's real term + AE prev
tuple; ha_apply_wal_catchup_entry routes every term-carrying entry through
raft_on_append_entries_request -> on_persist_entry (leader term stamped),
retiring the term-0 direct-apply. Standbys now persist the leader's real term
and reconcile by (term,lsn); the live commit chain is end-to-end.
848/849/850 + persist-before-ACK + commit-barrier + bug-960 preserved.
Proven at integration level; the definitive live STRICT_TERM failover run is
env-blocked here (bug-924 overlayfs) and recommended on a real fs.
Task 5.2 harness deploys a real 3-node localhost cluster, kills the primary,
promotes a standby, restarts the old primary, and asserts convergence via
walcompare --committed + per-job expect-job. Runs GREEN in PAYLOAD mode
(guards 848/849/850 + no committed-job loss).
The term-identity half of R5.1 is BLOCKED (bug-990): live replication still
uses term-0 repl_broadcast; the library send loop (on_send_append_entries) has
no live caller, so standbys persist hdr.term=0. The live-send retirement was
deferred across tasks 3.2/3.3 into verification tasks 5.2/5.4 that cannot
contain the production cutover. Needs a dedicated implementation task.
Diffs member WALs by (lsn -> term, pcrc) over the committed prefix, exits non-zero
on any committed-prefix divergence with the diverging LSN + per-member entry
identities. Reuses the canonical waldump decoder via a new per-entry visitor seam
(no second WAL parser). Detects same-LSN/diff-term and same-LSN/diff-content
divergences (the bug-851 class) that an LSN-range check would mask.
Wires config_committed_marker to a production consumer (config_activate_committed):
verifies pushed bytes' sha256 against the committed marker before activation, fences
(loud, non-serving, stay-alive) on mismatch, and activates the latest committed marker
on recovery. Enforced via ha_dispatch_gate + promotion gate.
Fixes bug-981: a 2nd+ config upgrade could permanently fence a standby on a
push-before-commit ordering. Adds an off-iomgr delaymgr retry scheduled from the
commit-advance hook (atomic read + non-blocking enqueue only on the ACK thread) that
re-activates and clears the stale fence once the new marker commits and bytes verify.
Wires do_upgrade_config_with_bytes to append WAL_CONFIG_MARKER through
raft_log_append (live term-stamped consensus record), demotes committed_config_tag
to a non-authoritative cache, and derives the committed config from the latest
committed marker. Preserves the derive across checkpoint compaction via an optional,
backward-compatible config-ref tail in the snapshot sentinel.
Includes fix for bug-960: endpoint-type-aware peer-index resolution
(promoted-vote relative vs heartbeat-standby absolute) with fail-loud
bounds check, preventing an out-of-bounds match_index[] write / wrong-peer
false-quorum on the replication ACK path.
Replaces the stub: builds a WalCatchupEntry AppendEntries to standbys[peer_idx].ep
carrying prev_log_index/prev_log_term; for n_entries==1 sources the entry at
prev_log_index+1 from the local WAL (term via persist_entry_term, type+payload via
ha_read_wal_entry_at); n_entries==0 emits a zero-entry heartbeat. Async send
(xce_wal_batch_req_raw), payload freed on all paths. 2 tests via a capture seam.
Review-applied: corrected the misleading repl.c comment — leader_commit is accepted
but NOT yet wire-encoded (deferred to task 3.2: WalCatchupEntry leaderCommit field +
standby intake). Reviewer APPROVED; per-group 0 failures.
on_get_entry_term → persist_entry_term (recorded term, 0=absent, legacy-0→1; caveat
documented per 1.3 review). on_truncate_from → persist_truncate_wal_above(p, index-1)
(delete LSN>=index), returns rc for library/leader retry; non-promotable fence intact;
replaces the committed-boundary heuristic on the library-driven path. New
truncate_term_index_locked keeps the in-memory per-LSN term index consistent on both
truncate paths (under lock, cutoff-aligned) → no stale-term read after truncation.
3 tests, reviewer APPROVED, per-group 0 failures.
Replaces the iter-1 stub: persist_set_term(p, term) stamps the leader/library term
into hdr.term, then role-routes the write — primary via persist_wal_async, standby
via persist_wal_write_replicated(p, index, ...). Persist failure returns non-zero
(no false ACK; durable_lsn unchanged); bug-850 contiguity guard + persist-before-ACK
preserved beneath. In-flight raft_entry_blob_t {type,payload_len} envelope added to
wal_types.h (never enters the 40-byte header). 3 tests, reviewer APPROVED.
Deferred (recorded in tasks.md): task 3.1 must retire the hand-rolled primary append
so current_lsn tracks the library index; task 3.2 must not call on_persist_entry on
heartbeats. Per-group 0 failures (pre-existing test_commit_barrier_reply leak aside).
persist_set_term sets the live election term so appends stamp hdr.term (fixes the
G1 term-0 stamping); replay_wal builds an in-memory per-LSN term index (freed in
persist_fini, no on-disk change); persist_entry_term reads it back (0=absent).
Legacy-0 -> WAL_LEGACY_TERM=1 at read-back only (raw 0 retained so present-legacy-0
is distinguishable from absent). 40-byte wal_entry_hdr_t preserved (value fix, R1.7).
5 new tests (test_term_safety_adapter.c); reviewer APPROVED. Feeds task 2.2.
All controller test groups report 0 failures; the meson full-suite timeout is the
bug-924 overlayfs /tmp fdatasync artifact, not this task.
bug-918 resolved (c-capnproto flake repin); clean controller build passes full
suite (Ok:1 Fail:0). Earlier wal_catchup failures were a stale incremental-build
artifact, not fsync/regression. Validation gate reliable for remaining tasks.
1.1: WalCatchupEntry gains term@4/prevLogIndex@5/prevLogTerm@6 (append-only).
1.2: RequestVote/PreVote gain lastLogTerm@4/lastLogIndex@5; cluster.c vote callers
fixed to the extended library arity (RESTORES xce_controller + controller_unit_test
compile+link, broken by the raft-commit-safety signature extension); ha.c emits the
candidate's last-log tuple on send.
5 minimal stub log/commit callbacks wired into s_real_cbs so raft_init's new
non-NULL requirement is satisfied — tasks 2.x replace them with real bodies; the
logOk freshness gate is inert until the term seam (1.3) + on_get_entry_term (2.2).
Pre-existing test_repl_unit.c wal_catchup -EAGAIN failures + ASAN leaks are an
overlayfs /tmp fsync env artifact (bug-850), not from this task (untouched paths;
append-only schema cannot affect the @3 entries-list decode). Build/baseline
isolation is blocked by a c-capnproto flake-hash drift — see Implementation Notes.
Mermaid treats ';' as a statement separator, so 'Note over L: no callback;
barrier untouched' split mid-note and failed with a Parse error before 'end'.
Replaced the semicolon with a comma. Validated PARSE_OK via mermaid@11 parse().
The other ';' (config-bytes flowchart line) is inside a double-quoted node label,
which mermaid treats as a literal — valid, left unchanged.
Canonical clean build (tools/xce-build.sh lib: rm -rf + fresh meson ASAN setup) passes
from scratch: libxce test OK, Fail 0, ASAN-clean. All election + log-replication +
commit-safety (G4 Figure-8 commit advance) + freshness (G6) + simulator invariant tests
green. Closes the raft-commit-safety library half of bug-851 (G4/G6).
KNOWN: the controller (cluster.c) vote callers still use the pre-extension signatures and
will not compile until the ha-term-safety adapter wires last_log_term/index — the accepted
revalidation trigger, addressed next.
7.1 Figure-8 commit refusal (G4): prior-term entry at a genuine quorum HELD (only the
current-term gate keeps commit_index at 0), then a current-term entry advances over
both (indirect commit) with one fire. Closes the Req 6.3 two-phase coverage.
7.2 Leader Completeness: committed prefix on {0,1}; node 0 fails; the stale node 2 is
DENIED by the freshness gate so only the up-to-date node 1 can win — and it holds the
full committed prefix (term+bytes).
7.3 freshness lexicographic (vote + prevote): stale-term-higher-index DENIED,
fresher-term-lower-index GRANTED, tie decided by index; prevote side-effect-free.
7.4 on_commit_advanced trigger: fire-count tracks STRICT advances — held(0)/advance(1)/
no-op-redrive(1)/advance(2)/non-leader(2).
Test-only (sim_set_log_entry local helper); reviewed in-parent (transient API 529 on the
review subagent). ASAN green (243 tests).
logOk (raft.tla:285, closes G6): local_last_log_term = log_len>0 ? on_get_entry_term(log_len):0;
logOk = cand_last_log_term > local || (== && cand_last_log_index >= log_len). Lexicographic:
term dominates, tie decided by index. Shared raft_log_ok helper.
Vote (4.2): logOk deny inserted after stale-term/UpdateTerm/role-gate/one-vote-per-term and
BEFORE on_vote_eligible (denies even when the app gate would grant); persist-before-grant intact.
PreVote (5.2): identical logOk, fully side-effect-free (no term/vote/role/commit mutation, no
on_persist). Signatures extended (1.3) on raft_on_vote_request/pre_vote_request; all 39 lib call
sites + sim routing updated; sim captures the candidate's send-time last-log tuple (raft_sim_rpc_t).
Empty-log call sites pass (0,0) → logOk true → existing outcomes preserved. 9 freshness tests.
NOTE: controller cluster.c vote callers still use the old signatures and will NOT compile — an
accepted, design-documented revalidation trigger; the controller adapter plumbing is owned by the
downstream ha-term-safety spec (to be implemented next). Library suite ASAN green (239 tests).
AdvanceCommitIndex (raft.tla:259/270, closes G4): leader-only (-EPERM); Max(agreeIndexes)
= highest N with a quorum ((n_peers/2)+1) of match_index>=N, self counted as log_len;
scan log_len downward. CURRENT-TERM RULE: advance commit_index to N only when
on_get_entry_term(N)==s->term, else HOLD (Figure-8 safety — never commit a previous-term
entry on replication count alone; a later current-term commit indirectly commits it).
Monotonic; fires on_commit_advanced once per strict advance; commit_index<=log_len.
3.3: raft_on_append_entries_reply left untouched — verified it never advances commit
(separation, Req 2.6). 6.1: sim commit-driver helper raft_sim_advance_commit added.
Req 6.3 two-phase indirect-commit transition covered end-to-end by task 7.1 (sim).
ASAN green (230 tests).
1.1: raft_state_t.commit_index (monotonic, ==WAL LSN, <=log_len, advanced only via
raft_advance_commit, untouched by AE reply) + raft_config_t.initial_commit_index.
1.2: on_commit_advanced(ctx, commit_index) REQUIRED callback — fired only on strict
advance, a TRIGGER for the controller commit barrier (never does durability itself).
2.1: raft_init requires it (-EINVAL if NULL) + restores commit_index from
initial_commit_index. raft_advance_commit prototype declared (impl in 3.2).
Coupled callback wiring to keep the required-callback ripple green: sim wires
sim_on_commit_advanced (+per-node commit_index mirror & fire-count), all test cbs
builders set it, raft_sim_restart_node restores the durable commit position. Vote
signatures unchanged; no commit-advance logic yet. ASAN green (224 tests).
raft_sim_restart_node models a process restart: the durable in-memory log[]
(WAL) and persisted (term, voted_for) survive, while volatile consensus state
is cleared by re-running raft_init (returns the node to FOLLOWER). Callbacks,
ctx, idx, and partition flag are preserved.
test_sim_9_5_node_restart_resumes_replication restarts a follower mid-replication
(partial 2-of-3 log), asserts the log survived (load-bearing: log_count==2 would
fail if wiped), role/term restored, then replication resumes and converges with
Log Matching. Closes the 8.3 'node restart while replication proceeds' sub-clause
(8/8 requirement sections now fully covered). ASAN green (220 tests).
9.1 replication + partition catch-up: 3-entry quorum replication then Log Matching;
isolate a follower, heal, assert next_index walks back, catch-up, Log Matching.
9.2 divergent-suffix truncation (G3): follower with a same-index/different-term tail
converges via conflict-truncate; byte-for-byte prefix equality + Log Matching.
9.3 drop/dup/reorder convergence: dropped AE recovered; duplicate already-have AE is
a no-op; genuine out-of-order delivery (entry-3 AE before entry-2) rejected via the
logOk gap with no phantom gap; converges + Log Matching (Req 8.3).
9.4 chaotic-churn G5/bug-851 regression: stale-term node rejoins under a higher-term
leader, adopts the term via the AE/heartbeat UpdateTerm path (load-bearing: would
fail as stale-reject if UpdateTerm-on-AE regressed), voted_for cleared, no second
leader, and a post-converge election campaigns strictly above the old term (cannot
reuse). Log Matching holds throughout.
Review-applied: genuine reorder in 9.3; active re-election + voted_for assertions in
9.4. ASAN green (219 tests).
raft_sim_check_log_matching: for every node pair and every shared index i, if
both entries at i have equal term, assert the whole prefix 1..i is term-and-byte
identical (Req 8.4, raft.tla Log Matching). Read-only, framework-agnostic (returns
1/0), bounded. Self-tests: pass on a converged cluster, fail on a hand-divergent
pair sharing (i=2,term=7) but differing at the index-1 prefix. ASAN green.
raft_sim_deliver_append_entries drains the AE queue, routes each buffered AE to
the (non-partitioned) destination via raft_on_append_entries_request, and routes
the result back to the source via raft_on_append_entries_reply — mirroring the
vote-RPC routing; partitioned endpoints are skipped.
Review-applied (HIGH): raft_sim_ae_t now captures the leader's SEND-time term
(leader_term) at buffer time and delivery uses it instead of the live source
state.term — without this, reorder/churn delivery (9.3/9.4) would inject the
wrong term and defeat UpdateTerm/stale-drop (G5). Smoke test asserts replication
to a connected follower + isolation of a partitioned one. ASAN green.
HandleAppendEntriesResponse (raft.tla:393): stale-term drop first (applied only
when reply_term==s->term, raft.tla:415, Req 5.4 — both lower and higher dropped);
success -> next_index[peer]=match_index+1, match_index[peer]=match_index (Req
5.1-5.2); failure -> next_index[peer]=Max(next-1,1) floored, match_index unchanged
(Req 5.3); no commit advance. uint64 underflow blocked by the >1 floor guard.
3 tests incl. floor-at-1 and lower/higher stale. ASAN green.
Complete HandleAppendEntriesRequest (raft.tla:327) + UpdateTerm (raft.tla:406),
implemented as one function to avoid a misleading partial handler. Fixed step
order: UpdateTerm FIRST on every RPC incl. heartbeat (G5, strictly-greater,
clears voted_for, persists, leader step-down fires on_become_follower) ->
stale reject -> candidate step-down -> logOk (G2) -> reject if !logOk ->
accept (already-have / conflict-truncate-then-repersist (G3) / append).
Never advances commit, never does transport I/O.
Review-applied: logOk now includes the explicit prev_log_index <= log_len bound
(raft.tla:330 / Req 4.1) instead of relying solely on the 0-absent sentinel;
added a candidate-equal-term-logOk-fails step-ordering test. 15 tests, ASAN green.
Leader-only emit (raft.tla AppendEntries:204, Req 3.5 — no-op off leader);
prev_log_index=next_index[j]-1, prev_log_term via on_get_entry_term (0/unqueried
at index 0, Req 3.2-3.3); n_entries=1 when next_index[j]<=log_len else 0 heartbeat
(Req 3.4); leader_commit opaque passthrough. Trace fires only on successful send
(consistency with raft_log_append) and non-leader RAFT_OK asserted — both from
review. 4 spy-based tests. ASAN green.
Leader-only (-EPERM for follower/candidate, no mutation/callback); index=
log_len+1, term=currentTerm (raft.tla ClientRequest:246, Req 2.1-2.4);
persist-before-act: on_persist_entry called once BEFORE log_len advances,
non-zero return propagated with no phantom entry (Req 2.5); no commit-index
advance, no transport I/O. Spy-based tests for all four branches. ASAN green.
next_index[]/match_index[]/pre_vote_replies[] are fixed-size RAFT_MAX_PEERS
arrays indexed by peer; without an upper-bound check a caller passing
n_peers > 8 would OOB-write in the leader-progress and pre-vote loops.
Add the guard (returns -EINVAL, no mutation) + canary-checked test.
Latent since task 1.1; surfaced by task 2.2 review (first loop writes into
the new arrays). No current caller exceeds the bound (sim caps at 8, HA <= 4).
next_index[j]=log_len+1, match_index[j]=0 for every peer at both win paths
(single-node RAFT_ELECTED_IMMEDIATELY in raft_start_election; multi-node
RAFT_BECAME_LEADER in raft_on_vote_reply) via shared raft_init_leader_progress
helper (raft.tla BecomeLeader, Req 3.1). Suite green under ASAN.
2.1: raft_init returns -EINVAL when any of on_persist_entry/on_get_entry_term/
on_send_append_entries/on_truncate_from is NULL (Req 7.1-7.4), before any write
to *s; restores s->log_len = cfg->initial_log_len (Req 1.1).
8.1: raft_sim gains a per-node in-memory log (term+bytes per 1-based index) and
an AE buffer queue; wires all four callbacks (persist/get-term/truncate/send).
Pulled forward to land with 2.1 so the stricter init does not leave the
sim/proto suite red between tasks. AE delivery deferred to 8.2.
Suite green under ASAN (Ok:1 Fail:0).
Commit ca1d9ae raised the commit-barrier default 50->500ms in config.h
but left test_config.c asserting ==50, leaving test_persistence_constants
failing on HEAD. Sync the assertion to the committed constant.
Unrelated to raft-log-replication; fixed in isolation so the library
suite has a clean baseline for the spec's task-10.1 green gate.
WAL-byte decode proves REAL data loss (not commit-index): committed prefix
entries physically absent from live nodes (ctrl-2 WAL starts LSN 16, ctrl-3 LSN
13), present only on the down node. Mechanism: empty-rejoin catch-up streams
[watermark..committed] not [1..], no snapshot to backfill the prefix -> permanent
gap. Predates libxce-failover. Full findings + open forks + xce_test waldump
tooling plan in INVESTIGATION-failover-job-loss.md. Not yet fixed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
