checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
parent
7d70820aac
commit
20e59d0b42
85 changed files with 10525 additions and 1993 deletions
|
|
@ -179,3 +179,118 @@ Overall remaining work: **S–M total, Low–Medium risk** — the library and i
|
|||
- **R3**: Confirm `raft_check_and_start_election` ignoring the `raft_start_election` return code
|
||||
(raft.c:287–289) is safe in all adapter paths (comment claims -EALREADY unreachable; verify when an
|
||||
inbound vote request flips role between RDHUP and the delayed check).
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
Post-implementation audit. Spec phase = implemented, tasks 24/24 [x]. This pass re-verifies every
|
||||
EARS requirement against the working tree and reconciles the prior audit's open items (C2, R1–R3).
|
||||
Assets are tagged: **[CLOSED]** verified met in code, **[CLOSED-AS-BUILT]** met but behavior differs
|
||||
from literal req text (improvement), **[RESIDUAL]** doc/hygiene gap, **[SCOPE-DRIFT]** code beyond spec.
|
||||
|
||||
### Requirement-to-Asset Map
|
||||
|
||||
| Req | Acceptance | Asset (file:line) | Status |
|
||||
|-----|-----------|-------------------|--------|
|
||||
| 1.1 | per-node state fields | `raft.h:28-50` raft_state_t (term/role/voted_for/voted_for_term/n_peers) | CLOSED |
|
||||
| 1.2 | init term=0 role=FOLLOWER voted_for=-1 | `raft.c:40-56` raft_init; defaults via cfg {0,-1} | CLOSED |
|
||||
| 1.3 | no transport types in header | `raft.h:4-6` includes only errno/stdint/string; no endpoint_t/SSL_CTX/cluster_t | CLOSED |
|
||||
| 2.1 | election: incr term + self-vote + send RPCs | `raft.c:68-71,98-102` | CLOSED |
|
||||
| 2.2 | single-node → immediate LEADER | `raft.c:87-92` returns RAFT_ELECTED_IMMEDIATELY | CLOSED |
|
||||
| 2.3 | already CANDIDATE/LEADER → error, no state change | `raft.c:64-65` -EALREADY | CLOSED |
|
||||
| 3.1 | term>cur → update,clear vote,FOLLOWER,evaluate | `raft.c:125-137` (also steps down leader via on_become_follower) | CLOSED-AS-BUILT |
|
||||
| 3.2 | term<cur → deny + return cur term | `raft.c:114-119` | CLOSED |
|
||||
| 3.3 | same term, diff voted_for → deny | `raft.c:161-168` (gated on voted_for_term==term) | CLOSED |
|
||||
| 3.4 | freshness gate pass → grant + persist | `raft.c:170-190`; freshness split: tag/lsn floor enforced caller-side (cluster.c:363), COMMITTING/liveness in on_vote_eligible (`ha.c:299` raft_cb_vote_eligible) | CLOSED-AS-BUILT |
|
||||
| 3.5 | LEADER/COMMITTING → deny all | `raft.c:148-152` LEADER deny; COMMITTING enum removed (ha-single-leader-role 93dfc02), now an on_vote_eligible policy | CLOSED-AS-BUILT |
|
||||
| 4.1 | quorum ceil((n+1)/2) → LEADER once | `raft.c:372-383`, quorum=(n_peers/2)+1 == ceil for odd n | CLOSED |
|
||||
| 4.2 | reply term>cur → FOLLOWER + on_become_follower | `raft.c:346-355` | CLOSED |
|
||||
| 4.3 | post-decision no-op | `raft.c:358-362` election_decided guard → RAFT_PENDING | CLOSED |
|
||||
| 4.4 | all replied, no quorum → FOLLOWER | `raft.c:386-393` replies_received==n_peers-1 | CLOSED |
|
||||
| 5.1 | leader-lost → clear vote + delay(priority*timeout) | `raft.c:417-425` | CLOSED |
|
||||
| 5.2 | voted non-self during delay → skip | `raft.c:429-433` RAFT_SKIP_ELECTION | CLOSED |
|
||||
| 5.3 | not voted / self → start election | `raft.c:435-438` RAFT_ELECTION_STARTED | CLOSED |
|
||||
| 6.1 | start election → on_schedule_timeout | `raft.c:105` | CLOSED |
|
||||
| 6.2 | timeout undecided → on_become_follower + cancel | `raft.c:409-414` | CLOSED |
|
||||
| 6.3 | timeout decided → on_cancel_timeout, no-op | `raft.c:401-407` | CLOSED |
|
||||
| 7.1 | raft_callbacks_t with required fn ptrs | `raft.h:79-117` (9 fields incl. on_send_pre_vote_request) | CLOSED |
|
||||
| 7.2 | void* ctx passed unchanged | `raft.h` ctx threaded; `raft.c` passes s->ctx everywhere | CLOSED |
|
||||
| 7.3 | NULL required callback → non-zero | `raft.c:23-38` 7 required-cb checks → -EINVAL | CLOSED |
|
||||
| 8.1 | deterministic N-node simulator | `raft_sim.c`, `raft.h:264-342` (XCE_TESTING) | CLOSED |
|
||||
| 8.2 | 4 invariant cases (a-d) | `test_raft.c`: A2 one-leader, B1/B4 deny-second, A1 single-node, E*/sim timeout | CLOSED |
|
||||
| 8.3 | compiles + tests pass under ASAN (leaks=0) | meson `raft/raft.c` (l.109) + `test_raft.c` (l.179); 209 RUN_TEST/test fns | CLOSED (re-run, see V1) |
|
||||
| 9.1 | adapter implements all callbacks, no dup state | `ha.c:1309-1320` s_real_cbs wires all 10; production state in raft_state_t | CLOSED |
|
||||
| 9.2 | ha delegates term/voted_for/quorum to library | `ha.c:4382-4522` ha_start_election_async→raft_start_election; `cluster.c:418` vote req; `ha.c:3237` vote reply | CLOSED-AS-BUILT |
|
||||
| 9.3 | XCE_TESTING hooks still functional | `ha_rdhup_election_hook` (ha.c:4528), ha_phase2 seams retained | CLOSED |
|
||||
|
||||
**Score: 24/24 acceptance-criteria groups CLOSED.** No requirement is unimplemented. 5 are CLOSED-AS-BUILT
|
||||
(real code is a strict superset/improvement over literal req text — see residual R-DOC).
|
||||
|
||||
### Reconciliation of Prior Audit Open Items
|
||||
|
||||
- **C2 (voted_for_term restore double-vote) — NOW FIXED.** `raft.c:47` sets
|
||||
`voted_for_term = (initial_voted_for != -1) ? initial_term : 0` at init, closing the post-restart
|
||||
second-vote window. Regression test `test_proto_H6_restored_vote_blocks_second_vote` (test_raft.c:2100,
|
||||
registered l.3978) guards it. Prior research's "only candidate correctness gap" is resolved (commit 9821ec4).
|
||||
- **R3 / Req 9.2 dual vote path — DOWNGRADED.** Legacy `handle_request_vote` (ha.c:2608) is now
|
||||
`#ifdef XCE_TESTING`-only — the production vote path is exclusively `cluster.c:418 → raft_on_vote_request`.
|
||||
The duplicated election-state logic no longer runs in production; it remains only as a test fixture for
|
||||
`test_ha_vote_gate.c` / `test_ha_election.c`. No production split-brain risk from the dual path. Residual:
|
||||
the test-only copy can drift from raft.c semantics over time (R-DUP below).
|
||||
- **R1 (even-n_peers policy) — STILL OPEN.** `raft_init` rejects even n_peers>1 with `-EINVAL`
|
||||
(raft.c:21-22), test `test_raft_init_even_peers_returns_einval`. Not stated in requirements; a 2-node HA
|
||||
deploy would silently fail raft_init. Needs requirements note + startup-time config validation.
|
||||
|
||||
### Residual Gaps (non-blocking)
|
||||
|
||||
- **R-DOC (Low):** requirements.md 3.4/3.5 still describe COMMITTING as an in-library role gate and the
|
||||
freshness gate as in-library; as-built, COMMITTING enum is removed and freshness is split caller-side +
|
||||
on_vote_eligible. Requirements text lags the (better) implementation. Update for traceability.
|
||||
- **R-DUP (Low):** test-only `handle_request_vote` duplicates vote-gate logic; risk of semantic drift from
|
||||
raft.c. Consider retargeting `test_ha_vote_gate.c` onto `raft_on_vote_request` directly.
|
||||
- **R1 (Medium):** even-n_peers rejection undocumented; surfaces as silent raft_init -EINVAL rather than a
|
||||
clear startup error. Add config-load validation OR a requirement clause.
|
||||
- **R-SCOPE / SCOPE-DRIFT (informational, not a gap):** PreVote (raft_begin_pre_vote,
|
||||
raft_on_pre_vote_request, raft_on_pre_vote_reply, on_send_pre_vote_request, RAFT_PRE_VOTE_* codes;
|
||||
raft.c:193-336, raft.h:160-259) lives in the raft-library files but belongs to the separate **ha-prevote**
|
||||
spec (commits 449b8e9–e6b234a). It is purely additive, mutates no persistent Raft state, and does not
|
||||
violate any raft-library requirement. Flag only so the raft-library boundary doc reflects the co-tenancy:
|
||||
raft.h/raft.c now host two specs' surfaces.
|
||||
|
||||
### Integration Risks
|
||||
|
||||
- **IR-1 (Low):** PreVote tally arrays are bounded by `RAFT_MAX_PEERS=8` (raft.h:18); `raft_on_pre_vote_reply`
|
||||
range-guards (raft.c:287). Real-vote path has no equivalent fixed array (uses scalar counters) so it is
|
||||
unbounded-safe. Mismatch is intentional but worth a one-line note in design.
|
||||
- **IR-2 (Low):** `raft_check_and_start_election` discards `raft_start_election` return (raft.c:436-437);
|
||||
comment claims -EALREADY unreachable because raft_on_leader_lost forces FOLLOWER. Valid for the current
|
||||
adapter, but an inbound vote/AppendEntries that flips role between RDHUP and the delayed fire could make it
|
||||
reachable. Low likelihood; assert instead of (void) for defense-in-depth.
|
||||
|
||||
### Open Work — Option Analysis
|
||||
|
||||
**Option A — Doc-only closure (recommended).** Update requirements.md 3.4/3.5/9.2 to as-built; add R1
|
||||
even-n_peers clause; note PreVote co-tenancy + IR-1/IR-2 in design.md. Then `/kiro-verify-completion`.
|
||||
- Effort: **S** (~1-2h, docs only). Risk: **Low**. No code change, no regression surface.
|
||||
|
||||
**Option B — Doc closure + hygiene.** Option A plus: (1) startup config validation for even n_peers with a
|
||||
loud error, (2) retarget test-only `handle_request_vote` onto raft_on_vote_request (kill R-DUP), (3) convert
|
||||
IR-2 `(void)rc` to an assert.
|
||||
- Effort: **M** (~half day). Risk: **Low-Medium** (touches ha.c test seam + config load; rerun full
|
||||
controller + lib test suites).
|
||||
|
||||
**Option C — Defer, accept as-is.** Ship implemented; track R-DOC/R-DUP/R1 as backlog. The only real
|
||||
correctness item (C2) is already fixed and tested, so functional risk of deferring is minimal.
|
||||
- Effort: **none**. Risk: **Low** functionally; **Medium** for spec/code traceability drift over time.
|
||||
|
||||
### Validation To Re-Run
|
||||
|
||||
- **V1:** `/xce-build lib` and `/xce-build controller` with `ASAN_OPTIONS=detect_leaks=0` — no build dir
|
||||
present in the worktree at audit time; the test_raft.c suite (209 cases) and controller tests should be
|
||||
re-run to confirm green on the current tree (which now includes the PreVote additions + C2 fix, post-dating
|
||||
the last recorded task-7 ASAN run).
|
||||
|
||||
**Bottom line:** raft-library is functionally complete — 24/24 met, the sole prior correctness gap (C2) is
|
||||
fixed and regression-tested, and the production dual-vote-path risk is eliminated (legacy path is test-only).
|
||||
Remaining work is documentation/hygiene (Option A, S effort, Low risk) plus a clean ASAN re-run (V1).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue