Commit graph

18 commits

Author SHA1 Message Date
d70bf6a402 update documents 2026-08-10 17:51:51 +08:00
7512dc83f0 docs(specs): kiro-validate-impl GO closure for 8 HA/raft specs
libxce-failover, raft-sim-conformance, ha-commit-durability,
ha-wal-replication, ha-heartbeat-committed-propagation, ha-prevote,
ha-elect-ctx-lifecycle, and ha-entity-snapshots all had 100% of tasks
checked but had never run a formal /kiro-validate-impl closure pass
(spec.json stuck at phase "tasks-generated", some with stale
ready_for_implementation flags). Each now has a fresh validate_impl
GO decision backed by an ASAN build, full standalone test-suite run,
and a live 10-round sequential HA soak with zero committed-job loss.
2026-07-18 13:27:36 +00:00
9f530f5742 docs: fix unchecked group-header checkboxes across 8 fully-implemented specs
controller-status, ha-commit-engine, ha-heartbeat-committed-propagation,
ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and
raft-ha-core-safety each had every numbered sub-task marked [x] but their
group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic
gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real
unchecked sub-tasks in any of the eight files. No task content changed.
2026-07-07 08:45:41 +00:00
20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00
5503c7a176 chore(ha-prevote): resolve log-reason nit; track elect_ctx leak as follow-up spec
- Mark the PreVote denial log-reason nit RESOLVED in ha-prevote tasks notes (47f7338).
- File ha-elect-ctx-lifecycle spec scaffold (spec.json + brief.md) capturing the
  pre-existing, bounded, UAF-sensitive election-carrier leak and the refcount/
  teardown-ordered-free approach across both election and PreVote paths. Deferred
  by decision — not folded into ha-prevote to avoid a risky lifetime refactor.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 07:56:07 +00:00
c45f80ab2b commit missed 2026-06-16 15:42:43 +08:00
e6b234a6a4 test(ha-prevote): Raft safety-invariant property tests + RN-1 consistency argument (task 6.1)
Four property-style raft tests proving the PreVote API preserves the safety
invariants: term/voted_for/role never change and on_persist never fires across
mixed begin/request/reply sequences (WON and LOST); a WON round creates no leader
(stays FOLLOWER); and the real-vote higher-term-adopt + step-down + quorum-elect
path is unchanged (bug-339/340 intact). Records the TLA+/checklist consistency
argument (RN-1) in test_raft.c and research.md, citing the tests as evidence.
Raft suite 123/0/0; zero production change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 07:18:06 +00:00
22fa753fa6 test(ha-prevote): genuine-failover, re-attach, and liveness-rearm scenarios (task 5.3)
Three scenario tests (zero production change): a stale-heartbeat standby GRANTS
(grant<->deny flip vs 5.2) and the won round advances to CTRL_ELECTING (PreVote
does not block legitimate failover); a heartbeat mid-round cancels and re-attaches
as standby with term unchanged; a lost round re-arms liveness and a later genuine
failure still wins (reusable, not one-shot).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 07:09:01 +00:00
86ac7c4863 test(ha-prevote): frozen-then-resumed no-op scenario (task 5.2)
Two-angle scenario test (zero production change): a healthy PRIMARY and a
fresh-heartbeat STANDBY both DENY a higher-prospective-term preVoteRequest
without adopting the term or demoting; and a resumed standby whose round both
peers deny gets no majority -> stays CTRL_STANDBY/FOLLOWER, term byte-for-byte
unchanged, liveness re-armed. Regresses if the disruptive-server bug returns.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 06:54:27 +00:00
6d22f2eb1a feat(ha-prevote): route preVoteReply over real transport to the candidate tally (task 5.1)
peer_ctrl_msg_handler gains a CTRL_STANDBY + pre_vote_in_progress branch that
decodes an incoming preVoteReply (no term adoption) and feeds it to
pre_vote_reply_dispatch with the absolute peer index, paralleling the real-vote
reply path. Gives pre_vote_reply_dispatch its production caller (drops the used
attribute). STANDBY-no-round still drops; ELECTING/PRIMARY routing unchanged.
PreVote now drives rounds to a decision end-to-end over the wire.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 06:47:08 +00:00
b765fc4705 feat(ha-prevote): rewire failover triggers through PreVote, defer destructive step (task 4.2)
ha_on_primary_ctrl_fd_rdhup + liveness_poll_fn now begin a PreVote round via
ha_begin_failover -> ha_start_prevote_async instead of electing directly; WAL
truncate + CTRL_ELECTING deferred to PreVote WON (ha_enter_real_election).
Connection teardown + config-less voter guard preserved at the trigger.
ha_on_heartbeat_received cancels an in-progress round and re-attaches as standby
with no term change. Single-node immediate-WON synchronously enters the real
election. PreVote now live; full controller suite 563 pass / 0 fail; bug-339/340
preserved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 06:29:18 +00:00
7e19bf6110 feat(ha-prevote): controller candidate-side PreVote orchestration (task 4.1)
ha.c: raft_cb_send_pre_vote_request (emits preVoteRequest), ha_start_prevote_async
(side-effect-free round + bounded round timeout), pre_vote_reply_dispatch
(raft_on_pre_vote_reply + branch), ha_enter_real_election helper. WON -> truncate
+ CTRL_ELECTING + ha_start_election_async (unchanged real-election path); LOST/
timeout -> stay CTRL_STANDBY + re-arm liveness + log abandonment; PENDING -> no-op.
Round inert until task 4.2 wires triggers; absolute peer-index keyed tally. 6 new
test_ha_prevote tests via reply hook; daemon + suite link clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 05:45:06 +00:00
6f95e4cdfc feat(ha-prevote): controller voter-side PreVote dispatch (task 3.1)
Add OPCode_preVoteRequest case in cluster.c, mirroring the real-vote case:
freshness floor + reused eligibility/stickiness gate (raft_on_pre_vote_request),
reply over preVoteReply with the voter's UNCHANGED term (no adoption), denial
logging with candidate id + reason. Pure insertion; real-vote case untouched.
6 new vote-gate tests assert deny-while-leader-alive, grant-when-lapsed, and the
no-mutation/term-unchanged contract.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 05:21:12 +00:00
9fac681510 feat(ha-prevote): implement pure PreVote reply tally (task 2.3)
raft_on_pre_vote_reply: per-peer-deduped tally returning WON at quorum
((n_peers/2)+1), LOST when all non-self peers reply without quorum, else
PENDING; bounds-guards peer_idx. Never fires become-leader/follower, never
mutates term/voted_for/role, never persists. 8 RED-first tests (3- and 5-node,
dedup, out-of-range, purity). Raft PreVote pure core complete.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 04:45:10 +00:00
6db912b2a9 feat(ha-prevote): implement pure candidate-side PreVote start (task 2.2)
raft_begin_pre_vote: compute prospective term (term+1) locally without
persisting, arm the pre_vote_* tally with a self-grant, NULL-safely probe each
non-self peer, return RAFT_PRE_VOTE_WON for single-node else RAFT_OK. Stays
FOLLOWER, schedules no timeout, mutates no persistent state. 6 RED-first tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 04:33:53 +00:00
722fee5323 feat(ha-prevote): implement pure voter-side PreVote decision (task 2.1)
raft_on_pre_vote_request: deny on stale term, else reuse on_vote_eligible
(primary/heartbeat-stickiness/catch-up gates). Never mutates term/voted_for/
role and never persists or adopts a higher term. 6 RED-first raft unit tests
assert the purity invariant; libxce suite green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 04:24:23 +00:00
449b8e9e93 feat(ha-prevote): add raft-library PreVote types, state, callback seam (task 1.2)
Scaffolding only (no bodies): RAFT_PRE_VOTE_WON/LOST/PENDING outcome codes;
pre_vote_in_progress / pre_votes_granted / pre_vote_replies[] tally fields
appended to raft_state_t; optional on_send_pre_vote_request callback. Adds
RAFT_MAX_PEERS=8 to bound per-peer dedup (production cap MAX_HA_CANDIDATES=4).
Existing raft election state/layout unchanged; libxce test green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 04:08:23 +00:00
0d660264f5 feat(ha-prevote): add preVote request/reply wire opcodes (task 1.1)
Append preVoteRequest @38 / preVoteReply @39 to the OPCode enum, reusing the
existing RequestVoteRequest/RequestVoteReply payload structs and union arms
(no new structs, no isPreVote flag). Keeps the real-vote wire path unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-16 03:59:47 +00:00