ROOT CAUSE (confirmed via live instrumentation — NOT a durability violation):
ha_enter_real_election truncated the WAL to last_known_committed_lsn BEFORE the
vote. That watermark lags the true commit point (the primary committed LSN 2-5 on
a durable majority then died before a heartbeat carried committed=5 to the
standby), so the cutoff was 1; persist_truncate_wal_above -> clamp_durable_lsn
lowered the standby's durable_lsn 5->1, physically destroying durably-committed
entries before the election even computed the safe point. The candidate then
advertised floor=1, safe_committed_lsn=1, and the post-election truncate kept only
LSN 1. This directly violated the documented rule: 'Never use last_known_committed
_lsn for truncation — it may be stale.'
FIX: delete the premature truncate. The candidate keeps its full durable log; the
ONLY truncates are now the authoritative votes-derived post-election
persist_truncate_wal_above(safe_committed_lsn) (raft_cb_become_leader) + the
library on_truncate_from (term,lsn) reconcile — both before any commit under the
new leader.
SAFE (reviewer-verified): a divergent uncommitted tail can only exist at a
stale/lower term (single-writer); the library's term-primary (lastLogTerm,
lastLogIndex) freshness gate denies such a candidate — correct now that task 3.4
fixed the term axis (the prerequisite). No divergent entry can win-and-commit; no
bug-339 election-loop. 177 unit tests + a RED-proven regression test.
LIVE: clean failover now preserves committed jobs — walcompare committed-prefix
identical across all 3 members, both jobs present on disk, across multiple runs.
(Residual harness EXIT 1 is the SEPARATE pre-existing libxce client bug-854:
client can't reach the live primary when the dead node is first in the masters
list — a client-discovery defect, orthogonal to server-side data safety.)
Clean failover was truncating committed jobs because the post-election
safe-commit-from-votes computation was dead: vote_reply_dispatch decoded each
granting voter's advertised peerLastLogLsn but discarded it, so ha_compute_safe
_committed_lsn saw only the self-vote (the 'from 1 granted voters' fingerprint)
and truncated to self_lsn.
Part 1: vote_reply_dispatch now records the granting voter's (lsn,tag,grant) into
ctx->votes[] via a record-only ha_record_vote_reply seam (no double-promotion;
raft_on_vote_reply stays the sole production promotion driver) — so safe-commit
reflects the true voted majority.
Part 2: the candidate/voter advertise the real DURABLE (fsync'd) tip; persist_wal
_write_replicated advances durable_lsn after its fail-loud inline fdatasync.
Reviewer-verified SAFE (advertises only durable positions; safe-commit bounded by
the durable majority, preserving more committed entries, never marking a
non-majority-durable LSN safe). 177 unit tests; new test_ha_vote_gate + election
tests green.
PARTIAL: does NOT yet make the live clean-failover pass. A deeper slice of
bug-828/795 remains — a catch-up'd standby still under-advertises its durable
position in the live run (the documented architectural catch-up/durability gap),
so a job-less node can still win and truncate. Tracked for a dedicated effort.
The commit-engine committed_lsn advance was TRACE-only; promoted to INFO (fires only
on a real advance, not noisy) so operators/tests can observe the authoritative primary
committed watermark. The clean-failover harness now polls this 'commit: committed_lsn
N -> M' line (not the lagging ha_process_standby_state heartbeat) with generous catch-up
headroom.
This made the live RED->GREEN visible: with Option A, committed_lsn reaches 5 on a
stable cluster ('jobs committed on a majority (committed_lsn=5 >= 5)') even via catch-up
delivery — the commit-stall is fixed.
Two coupled fixes for a live commit-stall (committed_lsn pinned at the own-term
NOOP while a 3-node cluster was durable to LSN 5 — jobs acked but never committed):
bug-991: the WAL catch-up applied_lsn was advanced to lsn+1 (next-to-apply) in
the three live/catch-up apply paths, but recovery/checkpoint/init and the leader's
catch-up streamer treat it as last-held. The over-advertise made the streamer skip
an LSN -> library logOk wedge -> no WalAck. Fixed to last-held (= lsn).
Option A (design reconsideration of REVIEW ISSUE 3): removed the
committed = min(durable-majority, raft_commit_index) ceiling. Under single-writer
the durable-majority watermark is already (term,lsn)-current-term-safe (term-carrying
replication + (term,lsn) reconcile/truncate-before-durable + own-term NOOP gate +
post-election truncate-to-safe), so the library commit_index was a redundant,
starvation-prone gate that pinned committed below the safe durable majority whenever
the library lagged. The durable-majority barrier + own-term writable gate are now
authoritative; raft_commit_index is inert-advisory.
Safety: reviewer-verified the durable-majority watermark is (term,lsn)-safe and
Figure-8 holds via the own-term gate (new RED->GREEN test); bug-851 cannot recur.
Live: commit trace 4->5 + walcompare committed-prefix identical across 3 nodes.
177 unit tests pass.
Ran clean_failover_preservation.sh on a real fs ($HOME/tmp/xce-ha-deploy, ext4).
PAYLOAD run CONFIRMED bug-990 fixed live: standbys persist the leader's real
term (2) for the committed prefix; walcompare OK.
Harness fixes: robust failover detection (any post-kill CTRL_PRIMARY promotion,
45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept not
commit, so the prior harness raced replication vs kill).
That commit-wait gate exposed bug-991: on a STABLE cluster, jobs replicate to
all members (durable LSN 5) but committed_lsn pins at 1. Catch-up-delivered
entries don't advance the library commit_index, so task-2.4's ceiling
min(durable-majority, raft_commit_index) starves commit below the durable
majority. A real regression from the commit-source switch (2.4+3.3), invisible
to the steady-state-only unit tests.
Closes bug-990: the live replicator repl_broadcast wire-encoded term=0
(entry.term was never set), so standbys persisted hdr.term=0 and took the
legacy direct-apply path instead of the library (term,lsn) intake. Now the
live broadcast and catch-up streamer carry the leader's real term + AE prev
tuple; ha_apply_wal_catchup_entry routes every term-carrying entry through
raft_on_append_entries_request -> on_persist_entry (leader term stamped),
retiring the term-0 direct-apply. Standbys now persist the leader's real term
and reconcile by (term,lsn); the live commit chain is end-to-end.
848/849/850 + persist-before-ACK + commit-barrier + bug-960 preserved.
Proven at integration level; the definitive live STRICT_TERM failover run is
env-blocked here (bug-924 overlayfs) and recommended on a real fs.
Task 5.2 harness deploys a real 3-node localhost cluster, kills the primary,
promotes a standby, restarts the old primary, and asserts convergence via
walcompare --committed + per-job expect-job. Runs GREEN in PAYLOAD mode
(guards 848/849/850 + no committed-job loss).
The term-identity half of R5.1 is BLOCKED (bug-990): live replication still
uses term-0 repl_broadcast; the library send loop (on_send_append_entries) has
no live caller, so standbys persist hdr.term=0. The live-send retirement was
deferred across tasks 3.2/3.3 into verification tasks 5.2/5.4 that cannot
contain the production cutover. Needs a dedicated implementation task.
Diffs member WALs by (lsn -> term, pcrc) over the committed prefix, exits non-zero
on any committed-prefix divergence with the diverging LSN + per-member entry
identities. Reuses the canonical waldump decoder via a new per-entry visitor seam
(no second WAL parser). Detects same-LSN/diff-term and same-LSN/diff-content
divergences (the bug-851 class) that an LSN-range check would mask.
Wires config_committed_marker to a production consumer (config_activate_committed):
verifies pushed bytes' sha256 against the committed marker before activation, fences
(loud, non-serving, stay-alive) on mismatch, and activates the latest committed marker
on recovery. Enforced via ha_dispatch_gate + promotion gate.
Fixes bug-981: a 2nd+ config upgrade could permanently fence a standby on a
push-before-commit ordering. Adds an off-iomgr delaymgr retry scheduled from the
commit-advance hook (atomic read + non-blocking enqueue only on the ACK thread) that
re-activates and clears the stale fence once the new marker commits and bytes verify.
Wires do_upgrade_config_with_bytes to append WAL_CONFIG_MARKER through
raft_log_append (live term-stamped consensus record), demotes committed_config_tag
to a non-authoritative cache, and derives the committed config from the latest
committed marker. Preserves the derive across checkpoint compaction via an optional,
backward-compatible config-ref tail in the snapshot sentinel.
Includes fix for bug-960: endpoint-type-aware peer-index resolution
(promoted-vote relative vs heartbeat-standby absolute) with fail-loud
bounds check, preventing an out-of-bounds match_index[] write / wrong-peer
false-quorum on the replication ACK path.
Replaces the stub: builds a WalCatchupEntry AppendEntries to standbys[peer_idx].ep
carrying prev_log_index/prev_log_term; for n_entries==1 sources the entry at
prev_log_index+1 from the local WAL (term via persist_entry_term, type+payload via
ha_read_wal_entry_at); n_entries==0 emits a zero-entry heartbeat. Async send
(xce_wal_batch_req_raw), payload freed on all paths. 2 tests via a capture seam.
Review-applied: corrected the misleading repl.c comment — leader_commit is accepted
but NOT yet wire-encoded (deferred to task 3.2: WalCatchupEntry leaderCommit field +
standby intake). Reviewer APPROVED; per-group 0 failures.
on_get_entry_term → persist_entry_term (recorded term, 0=absent, legacy-0→1; caveat
documented per 1.3 review). on_truncate_from → persist_truncate_wal_above(p, index-1)
(delete LSN>=index), returns rc for library/leader retry; non-promotable fence intact;
replaces the committed-boundary heuristic on the library-driven path. New
truncate_term_index_locked keeps the in-memory per-LSN term index consistent on both
truncate paths (under lock, cutoff-aligned) → no stale-term read after truncation.
3 tests, reviewer APPROVED, per-group 0 failures.
Replaces the iter-1 stub: persist_set_term(p, term) stamps the leader/library term
into hdr.term, then role-routes the write — primary via persist_wal_async, standby
via persist_wal_write_replicated(p, index, ...). Persist failure returns non-zero
(no false ACK; durable_lsn unchanged); bug-850 contiguity guard + persist-before-ACK
preserved beneath. In-flight raft_entry_blob_t {type,payload_len} envelope added to
wal_types.h (never enters the 40-byte header). 3 tests, reviewer APPROVED.
Deferred (recorded in tasks.md): task 3.1 must retire the hand-rolled primary append
so current_lsn tracks the library index; task 3.2 must not call on_persist_entry on
heartbeats. Per-group 0 failures (pre-existing test_commit_barrier_reply leak aside).
persist_set_term sets the live election term so appends stamp hdr.term (fixes the
G1 term-0 stamping); replay_wal builds an in-memory per-LSN term index (freed in
persist_fini, no on-disk change); persist_entry_term reads it back (0=absent).
Legacy-0 -> WAL_LEGACY_TERM=1 at read-back only (raw 0 retained so present-legacy-0
is distinguishable from absent). 40-byte wal_entry_hdr_t preserved (value fix, R1.7).
5 new tests (test_term_safety_adapter.c); reviewer APPROVED. Feeds task 2.2.
All controller test groups report 0 failures; the meson full-suite timeout is the
bug-924 overlayfs /tmp fdatasync artifact, not this task.
1.1: WalCatchupEntry gains term@4/prevLogIndex@5/prevLogTerm@6 (append-only).
1.2: RequestVote/PreVote gain lastLogTerm@4/lastLogIndex@5; cluster.c vote callers
fixed to the extended library arity (RESTORES xce_controller + controller_unit_test
compile+link, broken by the raft-commit-safety signature extension); ha.c emits the
candidate's last-log tuple on send.
5 minimal stub log/commit callbacks wired into s_real_cbs so raft_init's new
non-NULL requirement is satisfied — tasks 2.x replace them with real bodies; the
logOk freshness gate is inert until the term seam (1.3) + on_get_entry_term (2.2).
Pre-existing test_repl_unit.c wal_catchup -EAGAIN failures + ASAN leaks are an
overlayfs /tmp fsync env artifact (bug-850), not from this task (untouched paths;
append-only schema cannot affect the @3 entries-list decode). Build/baseline
isolation is blocked by a c-capnproto flake-hash drift — see Implementation Notes.
Test-only: exercises the real ha_vote_self_freshness/handle_request_vote gate.
A heartbeat raise via ha_observe_heartbeat_committed lifts the floor to
max(durable=5, raised=9); candidate at 7 denied, at 9/12 granted; a lower
heartbeat never lowers it. Discriminating constants fail under either regression
(floor->durable-only or fold->min). No production/vote-path change (Req 3.2).
123/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
primary_heartbeat_fn now sends xce_ha_heartbeat_req with the primary's OWN
atomic ha->term / persist.committed_lsn / committed_config_tag (NOT last_known_*),
snapshotted once per tick (Req 1.1, 3.4). XCE_TESTING seam adds a capturing
ha_heartbeat_req_hook (precedence new->legacy->real) so existing ha_nop_req_hook
liveness tests stay green; send-failure WARN + last_push_time unchanged. New test
poisons last_known_* to prove own-committed. Direct binary 119/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append a committed-consume branch to the OP_nop handler (existing liveness/reply
byte-unchanged): ha_consume_heartbeat_committed applies the non-primary gate
(3.4) + stale-term guard (accept hb->term >= ha->term, 3.2), then delegates to
the monotonic ha_observe_heartbeat_committed (1.2). Absent body / zero / failed
gate are no-ops (4.2). No vote-floor edit — safety is the term-guard. 4 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Strict-monotonic updater for a standby's learned committed position: advance
last_known_committed_lsn only when strictly greater (zero/lower no-op), set the
config tag together (tag-before-lsn release ordering for reader safety). No
ha_rejoin_reconcile, no truncation, no other mutation (Req 2.1-2.3, 3.1, 3.3).
4 revert-detecting unit tests; controller suite 0 failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the sibling last_known_committed_lsn: all writes release, reads acquire.
Prereq for a heartbeat-receive writer + the ctrlstatus reader to interact
safely (Req 2.3). Pure access migration, no behavior change; controller suite
0 failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add ctrlstatus_fmt.h (ctrlstatus_format_time_ms): renders responder_time_ms as
local YYYY-MM-DD HH:MM:SS,mmm, byte-equivalent to the controller log prefix
(tlog/my_tlog_format) so a captured ctrlstatus lines up against ctrl-N.log.
cmd_ctrlstatus prints a responderTime line; TZ-pinned exact-string unit test.
Req 11.2, 11.3. ctrlstatus group passes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Role-branch the committed sourcing: PRIMARY reports its own authoritative
persist.committed_lsn / committed_config_tag; off-primary reports the LEARNED
ha->last_known_committed_lsn / last_known_committed_config_tag instead of the
primary-only raw markers that read 0 on a standby (fixes the applied>committed
inversion). Stamp responder_time_ms from CLOCK_REALTIME for log correlation.
Read-only (Req 10.1-10.4, 11.1). 4 new TDD tests; controller suite 110/0/0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The wait-primary probe loop EXPECTS repeated failure until a primary serves, so it
no longer spams the console: joblist_query gains a quiet flag (the poll passes
quiet=1) and the client log threshold is raised to warn for the loop duration,
restored to the user's level on every exit path. Interactive joblist + the
assertions keep their diagnostics (quiet=0).
On the standby replicated-apply path, ha_apply_single_entry wrote the entry via
persist_wal_write_replicated (which correctly set the self-computed CRC chain
link p->last_crc), then called apply_wal_entry with a SYNTHETIC header whose
crc=0. apply_wal_entry's `p->last_crc = hdr->crc` clobbered the chain link with
0, so every subsequent replicated write stamped prev_crc=0 on disk. The on-disk
CRC chain broke at the second replicated entry — invisible while live (standbys
serve in-memory state) but fatal on COLD RESTART: replay hit the break, returned
RECOVER_TRUNCATED, recovered only the config prefix, and lost all committed jobs
(joblist=0). The sticky truncated verdict then drove a promotion-fence
availability wedge (bug-777) when every survivor was truncated.
Fix — single ownership of the WAL header + CRC chain:
- apply_wal_entry is now LOGICAL entity-restore only: signature
(p, type, lsn, payload, payload_len); no header. It advances the current_lsn
high-water (monotonic) but NEVER touches p->last_crc.
- The physical layer is the sole owner of the CRC chain: persist_wal_write_-
replicated on the live path, replay_wal/persist_recover on recovery.
- ha_apply_single_entry drops the fabricated header and passes entry fields
directly. The persist_recover replay caller is updated to the new signature.
Verified: new RED->GREEN test test_replicated_apply_chain_survives_cold_restart
(drives the real ha_apply_wal_catchup_entry path, cold-replays -> RECOVER_CLEAN,
all N recovered); controller suite 0 failures; 3-node rolling-SIGKILL soak on a
real ext2/3 fs now passes 5/5 (was wedge+loss at round 3); full cold restart
recovers all 6 committed jobs (was 0) with CLEAN verdicts and a normal election.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
