Commit graph

209 commits

Author SHA1 Message Date
f139e1c08e docs(ha-commit-durability): flip bug-661 KNOWN GAP to GUARANTEED + re-verify protocol checklist (task 5.3) 2026-06-18 11:15:13 +00:00
d128cc96ef test(ha-commit-durability): verify zero acked-job loss + cluster availability across failover on no-self-exit build (task 5.2) 2026-06-18 11:12:15 +00:00
59d9df246c fix(ha-commit-durability)!: controller never self-terminates — loud non-promotable fence replaces recovery FATAL/abort
Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
2026-06-18 11:10:19 +00:00
addf53e3b4 feat(ha-commit-durability): bounded-window catch-up before FATAL — fence no longer FATALs synchronously (task 4.4) 2026-06-18 10:35:13 +00:00
e14bd62115 test(ha-commit-durability): end-to-end durable-majority commit + liveness integration tests (task 5.1) 2026-06-18 09:56:26 +00:00
e9e2a6d703 feat(ha-commit-durability): pre-promotion eligibility gate + catch-up fence/release (task 4.3) 2026-06-18 09:40:32 +00:00
3049b620f3 feat(ha-commit-durability): durable-backed vote/PreVote freshness floor + bare-durable candidate advertise (task 4.2) 2026-06-18 09:02:11 +00:00
4b7457144c docs(ha-commit-durability): pin durable-backed freshness-floor formula + election safety/liveness verification (task 4.1) 2026-06-18 08:44:42 +00:00
c1c35ab1b3 feat(ha-commit-durability): publish durable floor from recovered WAL tip (task 3.4) 2026-06-18 08:24:46 +00:00
e3fc42c11e feat(ha-commit-durability): loud bounded FATAL on unrecoverable recovery (task 3.3) 2026-06-18 08:15:22 +00:00
6483ee8afd docs(ha-commit-durability): pin boot-policy decision table + pending skeleton tests (task 3.2) 2026-06-18 08:00:52 +00:00
dd4d70863b feat(ha-commit-durability): WAL recovery integrity verdict (CLEAN/TRUNCATED) + recovery_truncated flag (task 3.1) 2026-06-18 07:43:01 +00:00
526bca9679 feat(ha-commit-durability): gate legacy sync-append commit store on durable_lsn + commit<=durable invariant (task 2.4) 2026-06-18 07:09:47 +00:00
e4c4968331 feat(ha-commit-durability): gate standby ACK on fdatasync success + persist-before-ACK regression test (task 2.3) 2026-06-18 06:45:21 +00:00
c41edc515e feat(ha-commit-durability): cap commit self-contribution at durable_lsn + durable-advance commit wake (task 2.2) 2026-06-18 06:29:47 +00:00
cc83caf5a0 feat(ha-commit-durability): demand-driven group-commit flusher + durable_lsn watermark (task 2.1) 2026-06-18 06:06:55 +00:00
7e6f09f804 test(ha-commit-durability): deterministic strace-based ack-before-durable gap repro (task 1.3) 2026-06-18 05:39:06 +00:00
f90395282c feat(ha-commit-durability): add XCE_TESTING durable-LSN-on-disk verification seam (task 1.2) 2026-06-18 04:56:50 +00:00
1ca2d640cb feat(ha-commit-durability): add --wal-sync-ms flush-cadence override (task 1.1) 2026-06-18 04:46:01 +00:00
223e1b8578 docs(libxce-failover): record live-E2E round-2 fixes (bug-609 EINVAL, bug-610 reconnect SEGV) 2026-06-17 06:54:28 +00:00
16f4ba67c7 docs(libxce-failover): log bug-609 (known-dead EINVAL) + record 6.2 live-E2E partial status 2026-06-17 04:46:45 +00:00
c616f41364 test(libxce-failover): consolidate unit suite green + jitter staggering coverage (task 6.1) 2026-06-17 04:26:00 +00:00
5d1d1e3f12 feat(libxce-failover): controller populates primary host/port on E_XCE_REDIRECT replies (task 5.3) 2026-06-17 04:16:26 +00:00
fdb8938059 feat(libxce-failover): best-effort single-shot reconnect for fire-and-forget ops (notify/heartbeat/nop) (task 5.2) 2026-06-17 04:08:58 +00:00
4345974b54 feat(libxce-failover): wrap 18 reply-bearing xcecli_* ops in failover loop under per-ctx lock; correct nop OpCategory (task 5.1) 2026-06-17 03:57:51 +00:00
8783ce5a9d feat(libxce-failover): reconnect/rejoin hook — public setter + post-reconnect invocation before resume (task 4.1) 2026-06-17 03:46:05 +00:00
12284bc3f5 feat(libxce-failover): reconnect recovery — ctx ERR->RUN reset + stale wbuf drain (fixes post-failover EINVAL) (task 3.2) 2026-06-17 03:36:24 +00:00
84c5e32c2c feat(libxce-failover): retry decision loop — pure decide (budget/backoff/jitter, D6 exhaustion) + thin next wrapper (task 3.1) 2026-06-17 03:27:38 +00:00
cec2e647dc feat(libxce-failover): capture redirect hint in handle_reply (before free, before status return); verify endpoint-error wake (task 2.3) 2026-06-17 03:11:18 +00:00
c7e11a4e9e feat(libxce-failover): endpoint rotator — hint-first target selection + master rotation; keep api_ctx_t decls private (task 2.2) 2026-06-17 02:56:48 +00:00
fa2ac05036 feat(libxce-failover): 4-action status classifier with attempt-aware DUP-replay rule (task 2.1) 2026-06-17 02:42:54 +00:00
d11c6e1d89 feat(libxce-failover): reusable failover test harness — fake masters, status taxonomy, concurrency runner (task 1.2) 2026-06-17 02:32:19 +00:00
387ccfa8dc feat(libxce-failover): foundation — per-ctx failover state, redirect-hint fields, remove orphaned retry primitive (task 1.1) 2026-06-17 00:05:41 +00:00
5503c7a176 chore(ha-prevote): resolve log-reason nit; track elect_ctx leak as follow-up spec
- Mark the PreVote denial log-reason nit RESOLVED in ha-prevote tasks notes (47f7338).
- File ha-elect-ctx-lifecycle spec scaffold (spec.json + brief.md) capturing the
  pre-existing, bounded, UAF-sensitive election-carrier leak and the refcount/
  teardown-ordered-free approach across both election and PreVote paths. Deferred
  by decision — not folded into ha-prevote to avoid a risky lifetime refactor.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 07:56:07 +00:00
c45f80ab2b commit missed 2026-06-16 15:42:43 +08:00
e6b234a6a4 test(ha-prevote): Raft safety-invariant property tests + RN-1 consistency argument (task 6.1)
Four property-style raft tests proving the PreVote API preserves the safety
invariants: term/voted_for/role never change and on_persist never fires across
mixed begin/request/reply sequences (WON and LOST); a WON round creates no leader
(stays FOLLOWER); and the real-vote higher-term-adopt + step-down + quorum-elect
path is unchanged (bug-339/340 intact). Records the TLA+/checklist consistency
argument (RN-1) in test_raft.c and research.md, citing the tests as evidence.
Raft suite 123/0/0; zero production change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 07:18:06 +00:00
22fa753fa6 test(ha-prevote): genuine-failover, re-attach, and liveness-rearm scenarios (task 5.3)
Three scenario tests (zero production change): a stale-heartbeat standby GRANTS
(grant<->deny flip vs 5.2) and the won round advances to CTRL_ELECTING (PreVote
does not block legitimate failover); a heartbeat mid-round cancels and re-attaches
as standby with term unchanged; a lost round re-arms liveness and a later genuine
failure still wins (reusable, not one-shot).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 07:09:01 +00:00
86ac7c4863 test(ha-prevote): frozen-then-resumed no-op scenario (task 5.2)
Two-angle scenario test (zero production change): a healthy PRIMARY and a
fresh-heartbeat STANDBY both DENY a higher-prospective-term preVoteRequest
without adopting the term or demoting; and a resumed standby whose round both
peers deny gets no majority -> stays CTRL_STANDBY/FOLLOWER, term byte-for-byte
unchanged, liveness re-armed. Regresses if the disruptive-server bug returns.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 06:54:27 +00:00
6d22f2eb1a feat(ha-prevote): route preVoteReply over real transport to the candidate tally (task 5.1)
peer_ctrl_msg_handler gains a CTRL_STANDBY + pre_vote_in_progress branch that
decodes an incoming preVoteReply (no term adoption) and feeds it to
pre_vote_reply_dispatch with the absolute peer index, paralleling the real-vote
reply path. Gives pre_vote_reply_dispatch its production caller (drops the used
attribute). STANDBY-no-round still drops; ELECTING/PRIMARY routing unchanged.
PreVote now drives rounds to a decision end-to-end over the wire.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 06:47:08 +00:00
b765fc4705 feat(ha-prevote): rewire failover triggers through PreVote, defer destructive step (task 4.2)
ha_on_primary_ctrl_fd_rdhup + liveness_poll_fn now begin a PreVote round via
ha_begin_failover -> ha_start_prevote_async instead of electing directly; WAL
truncate + CTRL_ELECTING deferred to PreVote WON (ha_enter_real_election).
Connection teardown + config-less voter guard preserved at the trigger.
ha_on_heartbeat_received cancels an in-progress round and re-attaches as standby
with no term change. Single-node immediate-WON synchronously enters the real
election. PreVote now live; full controller suite 563 pass / 0 fail; bug-339/340
preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 06:29:18 +00:00
7e19bf6110 feat(ha-prevote): controller candidate-side PreVote orchestration (task 4.1)
ha.c: raft_cb_send_pre_vote_request (emits preVoteRequest), ha_start_prevote_async
(side-effect-free round + bounded round timeout), pre_vote_reply_dispatch
(raft_on_pre_vote_reply + branch), ha_enter_real_election helper. WON -> truncate
+ CTRL_ELECTING + ha_start_election_async (unchanged real-election path); LOST/
timeout -> stay CTRL_STANDBY + re-arm liveness + log abandonment; PENDING -> no-op.
Round inert until task 4.2 wires triggers; absolute peer-index keyed tally. 6 new
test_ha_prevote tests via reply hook; daemon + suite link clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 05:45:06 +00:00
6f95e4cdfc feat(ha-prevote): controller voter-side PreVote dispatch (task 3.1)
Add OPCode_preVoteRequest case in cluster.c, mirroring the real-vote case:
freshness floor + reused eligibility/stickiness gate (raft_on_pre_vote_request),
reply over preVoteReply with the voter's UNCHANGED term (no adoption), denial
logging with candidate id + reason. Pure insertion; real-vote case untouched.
6 new vote-gate tests assert deny-while-leader-alive, grant-when-lapsed, and the
no-mutation/term-unchanged contract.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 05:21:12 +00:00
9fac681510 feat(ha-prevote): implement pure PreVote reply tally (task 2.3)
raft_on_pre_vote_reply: per-peer-deduped tally returning WON at quorum
((n_peers/2)+1), LOST when all non-self peers reply without quorum, else
PENDING; bounds-guards peer_idx. Never fires become-leader/follower, never
mutates term/voted_for/role, never persists. 8 RED-first tests (3- and 5-node,
dedup, out-of-range, purity). Raft PreVote pure core complete.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 04:45:10 +00:00
6db912b2a9 feat(ha-prevote): implement pure candidate-side PreVote start (task 2.2)
raft_begin_pre_vote: compute prospective term (term+1) locally without
persisting, arm the pre_vote_* tally with a self-grant, NULL-safely probe each
non-self peer, return RAFT_PRE_VOTE_WON for single-node else RAFT_OK. Stays
FOLLOWER, schedules no timeout, mutates no persistent state. 6 RED-first tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 04:33:53 +00:00
722fee5323 feat(ha-prevote): implement pure voter-side PreVote decision (task 2.1)
raft_on_pre_vote_request: deny on stale term, else reuse on_vote_eligible
(primary/heartbeat-stickiness/catch-up gates). Never mutates term/voted_for/
role and never persists or adopts a higher term. 6 RED-first raft unit tests
assert the purity invariant; libxce suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 04:24:23 +00:00
449b8e9e93 feat(ha-prevote): add raft-library PreVote types, state, callback seam (task 1.2)
Scaffolding only (no bodies): RAFT_PRE_VOTE_WON/LOST/PENDING outcome codes;
pre_vote_in_progress / pre_votes_granted / pre_vote_replies[] tally fields
appended to raft_state_t; optional on_send_pre_vote_request callback. Adds
RAFT_MAX_PEERS=8 to bound per-peer dedup (production cap MAX_HA_CANDIDATES=4).
Existing raft election state/layout unchanged; libxce test green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 04:08:23 +00:00
0d660264f5 feat(ha-prevote): add preVote request/reply wire opcodes (task 1.1)
Append preVoteRequest @38 / preVoteReply @39 to the OPCode enum, reusing the
existing RequestVoteRequest/RequestVoteReply payload structs and union arms
(no new structs, no isPreVote flag). Keeps the real-vote wire path unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 03:59:47 +00:00
513b965f98 chore(ha-convergence-recovery): full ASAN regression GREEN + boundary audit; task 8 E2E blocked-manual (task 7) 2026-06-14 07:28:30 +00:00
e399328339 test(ha-convergence-recovery): regression pins for fan-out, snapshot, degraded-gate, re-inclusion (task 6.1) 2026-06-14 06:56:57 +00:00
07de4581b0 feat(ha-convergence-recovery): quorum transition logging — CAS edge-detect + 4 call sites (task 5.2) 2026-06-14 06:31:25 +00:00