- sim_tla_action_name: drop brittle per-case 'raft.tla:NNN' line-number comments
(some had drifted a few lines off; the return string IS the stable raft.tla
Next-disjunct name, and the function header already cites raft.tla:454).
- test_sim_5_1_partition_heal: the per-follower convergence loop was vacuous —
it bounded by commit_index, which stays 0 for followers under the leader-only
commit model, so the loop never ran. Bound by log_count instead (followers DO
advance the replicated log); Log Matching still guarantees the shared-prefix
terms agree. Now non-vacuous; still passes.
Test/comment-only; no behavior change. Lib suite Ok:1 Fail:0.
test_demotion_replay_yields_committed_only flaked (~16% on overlayfs, but
fs-independently too). Two root causes, both test-harness artifacts (NOT product
bugs; production self-restart is execv on a real fs and never overlaps persist
contexts or runs an auto-committer during demotion):
1. single-node commit race: the fixture's live commit engine on_durable_advance
auto-commits on n_peers=1, advancing committed_lsn past the test's boundary so
the 'uncommitted' suffix became committed. Fix: detach on_durable_advance +
re-pin committed_lsn to the test's own committed_boundary, so the suffix stays
genuinely uncommitted (the test's premise).
2. overlayfs fd-incoherence: a fresh persist_t recovering over the same files
reads a stale post-ftruncate size on overlayfs. Fix: hybrid statfs gate —
real fs keeps the full fresh boot-recovery assertion; overlayfs asserts the
same property via the coherent live-context current_lsn.
Test-only; production truncate (ftruncate+fdatasync) untouched. 0/25 runs fail
(was 4/25 with the prior persist_stop-only attempt).
The bug-850/851 forward-gap contiguity guard (persist.c:839, added in 20e59d0b)
correctly rejects a catch-up entry whose lsn is ahead of the WAL tail (-EAGAIN).
7 test_wal_catchup_* tests in test_config_repl.c predate the guard and applied
non-contiguous LSNs (42, 10-12, 1000+) to a fresh persist (current_lsn=1),
failing deterministically. Seed current_lsn to each test's first applied LSN so
the in-order stream is contiguous (mirroring the real catch-up streamer); the
apply->advance path and all asserted semantics are unchanged. Test-only; the
production contiguity guard is untouched. 179 controller tests, 0 failures (x3).
Adds two controller-level regression tests driving the production raft_on_vote_reply
via ha->raft (test_ha_election.c): a duplicated vote-grant from one peer must not
double-count toward quorum (bug-1095) and a stale-term reply must be dropped
(bug-1096) — both asserted on role/promotion (no two leaders per term). Formally
absorbs bug-1095/1096 into the ha-term-safety spec (Implementation Note) mapping to
its Requirement 1 / Objective; the raft.c fix (b5b7e01) is owned there with
protocol-suite regression at both the libxce (test_vote_reply_*) and controller
(test_ha_election) levels. No production source changed.
Adds raft_sim_scenario_figure8: a previous-term entry on a quorum under a
new-term leader is NOT committed until a current-term entry commits (library-
gated by raft_advance_commit; verified by a mutation test). Also restores the
raft_sim_scenario_drop_dup_reorder implementation that commit e445404 (task 5.3)
silently dropped from raft_sim.c (a stray git checkout reverted the uncommitted
impl), which had left the lib build broken with an undefined reference.
Build green: test_sim_5_3_drop_dup_reorder + test_sim_5_4_figure8 both PASS.
raft_on_vote_reply double-counted duplicated vote-responses (counter ignored
from_idx) and failed to drop stale-term replies, allowing two leaders at the
same term (bug-851 split-brain family). Mirror the existing pre-vote per-peer
dedup pattern (new vote_replies[] set) and add DropStaleResponse (term <
currentTerm). Surfaced by the raft-sim-conformance oracle (task 4.2).
47 raft / 264 overall lib tests pass.
Harness surfaced a real upstream raft.c split-brain bug (bug-1095/1096,
bug-851 family) in raft_on_vote_reply: counter-based vote tally double-counts
a duplicated vote-response and no stale-term drop -> two leaders same term.
Out of this spec's boundary (raft.c protocol); recorded as Implementation Note.
Mermaid parse errors in design.md's diagrams:
- Literal `|` in flowchart node labels is read as an edge-label delimiter:
system-flow `term | 0` -> `term | 0`; checker `|{i:...}|` ->
`|{i:...}|` (entity renders as a literal `|`, meaning preserved).
- `;` is a mermaid statement separator and truncated the text it appeared in:
sequence msgs `commit_index += ; apply()` -> `commit_index advanced, apply()`
and `(R5); halt` -> `(R5), then halt`; flowchart label `responses;<br/>` ->
`responses,<br/>`.
Valid edge-label pipes (-->|violation|) and all `;` in prose/tables outside the
mermaid fences are unchanged. All three diagrams now render.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
Ran the live 3-node E2E (the last open task). Result:
PASS: 8 rounds, 30 jobs submitted, zero job loss; all 3 nodes led
(ctrl-1,ctrl-2,ctrl-3) (exit 0)
Live-covered: kill-primary -> promoted standby owns all confirmed jobs
(zero loss across 8 failovers, jobs 3->30, expect-job ok each round);
restart ex-primary -> converged rejoin, 3-of-3 each round, no phantom jobs;
deterministic leadership coverage (ctrl-3 forced via RequestVote-freshness
fence). The soak also served as the live regression for bug-1077 (~40
failover client round-trips, no SEGV) after rebuilding the controller (the
deployed binaries predated that fix).
Not exercised live (remain unit-covered, tasks 2/4/5): withhold-ACK barrier
single-outcome (test_demotion_convergence) and isolate/heal quorum-WARN
(test_convergence_verification). All 8 tasks now [x]; spec closed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
Add an in-situ NOTICE at the config-upgrade restart site
(do_upgrade_config_with_bytes, step 9). Today an upgrade commits the
config-change marker, then each member restarts and re-materializes its
post-upgrade state by replaying its own pre-upgrade WAL under the NEW config;
replay is config-dependent (lost_and_found re-homes orphaned entities), so
members can diverge in materialized state (bug-857). The notice records the
intended fix — materialize a full entity-bearing snapshot at the marker's
(term,lsn) boundary, embed the new config, and install the SAME snapshot on
every member via snapshot_transfer_s — and its hard dependency on
entity-bearing-snapshots. Comment-only; anchors to
.kiro/specs/config-upgrade-snapshot/.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
ctx->reply.u is a single untagged union shared across opcodes (debugmsg /
get(dataset_t*) / jobid / clusterconf / ctrlstatus alias the same 8 bytes).
Commit 61e1391 added speculative free-guards
`if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(...)` (and the clusterconf
twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted
reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer)
because xce_get_with_condition/xce_resval only cleared the slot on the success
path, not the xce_get_req<0 / cbuf_get<0 early returns. The next ctrlstatus
reply (worker thread) freed that foreign pointer as a CtrlStatus -> ASan
SEGV / stack-use-after-return in free_CtrlStatus (controller.capnp.c:11638).
Fix (root cause, both layers):
- Revert the unsafe free-guards in xce_handle_reply: MOVE_PTR overwrite only,
never free/deref an untagged union slot by type.
- Clear ctx->reply.u.get = NULL on ALL exit paths of xce_get_with_condition
and xce_resval via a single `goto done` cleanup (forget-proof; the bug was
one forgotten failover path). Also drops the now-unused controller.capnp.h.
- Defense-in-depth: handle_get_reply drops a reply when the slot is NULL.
Regression: expose xce_handle_reply and add
test_handle_reply_ctrlstatus_does_not_free_foreign_slot, which poisons the
shared slot with a foreign heap value and proves the ctrlstatus reply handler
does not free it (RED reproduced the exact free_CtrlStatus_ptr double-free;
GREEN after fix). xcelib_test 244/0, controller_unit_test 166/0, all binaries
build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
Add an optional [dir] argument to the /xce-test-ha skill so the 3-node
soak cluster can be deployed to a caller-chosen path.
- run-ha-soak.sh: deploy dir resolves positional $1 > $XCE_HA_TOP > default;
relative paths absolutized in pure bash (this host's busybox realpath
lacks -m and would mangle nonexistent/relative paths).
- Fail-loud rm -rf guard: writes a .xce-ha-deploy sentinel at deploy time and
refuses to wipe a non-empty dir lacking it (protects a typo'd home/source
path); empty/nonexistent dirs stay safe to (re)deploy into.
- SKILL.md: document /xce-test-ha [rounds] [dir] and the safety semantics.
Verified: guard refuses an unmarked non-empty dir (dir survives, exit 1);
full 5-round soak into a custom dir PASS (18 jobs, zero job loss), cluster
landed in the named dir while the default dir was untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
The post-failover wait-primary/expect-job probes used the full masters list led
by the just-killed node, triggering libxce bug-854 (positional discovery can't
fail over from a dead-first master). Now they use a live-first list excluding the
killed primary.
NOTE: this exposes a deeper, separate post-failover AVAILABILITY issue (not data
loss): the new primary stays stuck in a catch-up loop with a lagging standby
(standby[0] lsn=5 < committed_lsn=6) and does not serve clients, so wait-primary
still times out. That is a catch-up/serving-readiness liveness concern orthogonal
to bug-828's committed-job DATA SAFETY, which is fixed and verified (walcompare
committed-prefix identical, jobs preserved on disk).
ROOT CAUSE (confirmed via live instrumentation — NOT a durability violation):
ha_enter_real_election truncated the WAL to last_known_committed_lsn BEFORE the
vote. That watermark lags the true commit point (the primary committed LSN 2-5 on
a durable majority then died before a heartbeat carried committed=5 to the
standby), so the cutoff was 1; persist_truncate_wal_above -> clamp_durable_lsn
lowered the standby's durable_lsn 5->1, physically destroying durably-committed
entries before the election even computed the safe point. The candidate then
advertised floor=1, safe_committed_lsn=1, and the post-election truncate kept only
LSN 1. This directly violated the documented rule: 'Never use last_known_committed
_lsn for truncation — it may be stale.'
FIX: delete the premature truncate. The candidate keeps its full durable log; the
ONLY truncates are now the authoritative votes-derived post-election
persist_truncate_wal_above(safe_committed_lsn) (raft_cb_become_leader) + the
library on_truncate_from (term,lsn) reconcile — both before any commit under the
new leader.
SAFE (reviewer-verified): a divergent uncommitted tail can only exist at a
stale/lower term (single-writer); the library's term-primary (lastLogTerm,
lastLogIndex) freshness gate denies such a candidate — correct now that task 3.4
fixed the term axis (the prerequisite). No divergent entry can win-and-commit; no
bug-339 election-loop. 177 unit tests + a RED-proven regression test.
LIVE: clean failover now preserves committed jobs — walcompare committed-prefix
identical across all 3 members, both jobs present on disk, across multiple runs.
(Residual harness EXIT 1 is the SEPARATE pre-existing libxce client bug-854:
client can't reach the live primary when the dead node is first in the masters
list — a client-discovery defect, orthogonal to server-side data safety.)
Clean failover was truncating committed jobs because the post-election
safe-commit-from-votes computation was dead: vote_reply_dispatch decoded each
granting voter's advertised peerLastLogLsn but discarded it, so ha_compute_safe
_committed_lsn saw only the self-vote (the 'from 1 granted voters' fingerprint)
and truncated to self_lsn.
Part 1: vote_reply_dispatch now records the granting voter's (lsn,tag,grant) into
ctx->votes[] via a record-only ha_record_vote_reply seam (no double-promotion;
raft_on_vote_reply stays the sole production promotion driver) — so safe-commit
reflects the true voted majority.
Part 2: the candidate/voter advertise the real DURABLE (fsync'd) tip; persist_wal
_write_replicated advances durable_lsn after its fail-loud inline fdatasync.
Reviewer-verified SAFE (advertises only durable positions; safe-commit bounded by
the durable majority, preserving more committed entries, never marking a
non-majority-durable LSN safe). 177 unit tests; new test_ha_vote_gate + election
tests green.
PARTIAL: does NOT yet make the live clean-failover pass. A deeper slice of
bug-828/795 remains — a catch-up'd standby still under-advertises its durable
position in the live run (the documented architectural catch-up/durability gap),
so a job-less node can still win and truncate. Tracked for a dedicated effort.
The commit-engine committed_lsn advance was TRACE-only; promoted to INFO (fires only
on a real advance, not noisy) so operators/tests can observe the authoritative primary
committed watermark. The clean-failover harness now polls this 'commit: committed_lsn
N -> M' line (not the lagging ha_process_standby_state heartbeat) with generous catch-up
headroom.
This made the live RED->GREEN visible: with Option A, committed_lsn reaches 5 on a
stable cluster ('jobs committed on a majority (committed_lsn=5 >= 5)') even via catch-up
delivery — the commit-stall is fixed.
Two coupled fixes for a live commit-stall (committed_lsn pinned at the own-term
NOOP while a 3-node cluster was durable to LSN 5 — jobs acked but never committed):
bug-991: the WAL catch-up applied_lsn was advanced to lsn+1 (next-to-apply) in
the three live/catch-up apply paths, but recovery/checkpoint/init and the leader's
catch-up streamer treat it as last-held. The over-advertise made the streamer skip
an LSN -> library logOk wedge -> no WalAck. Fixed to last-held (= lsn).
Option A (design reconsideration of REVIEW ISSUE 3): removed the
committed = min(durable-majority, raft_commit_index) ceiling. Under single-writer
the durable-majority watermark is already (term,lsn)-current-term-safe (term-carrying
replication + (term,lsn) reconcile/truncate-before-durable + own-term NOOP gate +
post-election truncate-to-safe), so the library commit_index was a redundant,
starvation-prone gate that pinned committed below the safe durable majority whenever
the library lagged. The durable-majority barrier + own-term writable gate are now
authoritative; raft_commit_index is inert-advisory.
Safety: reviewer-verified the durable-majority watermark is (term,lsn)-safe and
Figure-8 holds via the own-term gate (new RED->GREEN test); bug-851 cannot recur.
Live: commit trace 4->5 + walcompare committed-prefix identical across 3 nodes.
177 unit tests pass.
Ran clean_failover_preservation.sh on a real fs ($HOME/tmp/xce-ha-deploy, ext4).
PAYLOAD run CONFIRMED bug-990 fixed live: standbys persist the leader's real
term (2) for the committed prefix; walcompare OK.
Harness fixes: robust failover detection (any post-kill CTRL_PRIMARY promotion,
45s settle) + a wait-for-commit-before-kill gate (jobsub returns on accept not
commit, so the prior harness raced replication vs kill).
That commit-wait gate exposed bug-991: on a STABLE cluster, jobs replicate to
all members (durable LSN 5) but committed_lsn pins at 1. Catch-up-delivered
entries don't advance the library commit_index, so task-2.4's ceiling
min(durable-majority, raft_commit_index) starves commit below the durable
majority. A real regression from the commit-source switch (2.4+3.3), invisible
to the steady-state-only unit tests.
Closes bug-990: the live replicator repl_broadcast wire-encoded term=0
(entry.term was never set), so standbys persisted hdr.term=0 and took the
legacy direct-apply path instead of the library (term,lsn) intake. Now the
live broadcast and catch-up streamer carry the leader's real term + AE prev
tuple; ha_apply_wal_catchup_entry routes every term-carrying entry through
raft_on_append_entries_request -> on_persist_entry (leader term stamped),
retiring the term-0 direct-apply. Standbys now persist the leader's real term
and reconcile by (term,lsn); the live commit chain is end-to-end.
848/849/850 + persist-before-ACK + commit-barrier + bug-960 preserved.
Proven at integration level; the definitive live STRICT_TERM failover run is
env-blocked here (bug-924 overlayfs) and recommended on a real fs.
