Spec reached GO with 15/15 tasks complete (see prior task-confirmation commits); these doc artifacts were generated during the design/research phases but never separately staged.
34 KiB
Gap Analysis: raft-adapter-conformance
Analysis date: 2026-07-29. Requirements phase: generated (not yet approved). Per skill fallback rule, proceeding — gap analysis findings can also inform requirement refinement before approval.
Method
Codebase investigation used direct Read/grep against src/controller/ha.c, repl.c, config_repl.c, cluster.c, src/lib/raft/raft.c/raft.h, src/schema/controller.capnp, and src/controller/testsuite/, plus the two Explore-agent passes already run during requirements generation (their findings are the primary input to Requirement-to-Asset mapping below; this pass adds the concrete integration-point evidence — schema fields, dispatcher code, struct layouts — needed to select an implementation approach). No external dependency research was needed; this is entirely an internal-codebase conformance question.
Requirement-to-Asset Map
Requirement 1 — Systematic conformance audit deliverable
| Asset | Status |
|---|---|
raft/raft.tla (canonical action list: Restart, Timeout, RequestVote, BecomeLeader, ClientRequest, AdvanceCommitIndex, AppendEntries, Receive→UpdateTerm/HandleRequestVoteRequest/HandleRequestVoteResponse/HandleAppendEntriesRequest/HandleAppendEntriesResponse) |
Present, unmodified upstream copy |
src/controller/CLAUDE.md → "TLA+ Action-to-Code Mapping" + "Protocol Verification Checklist" tables |
Present and already the natural home for this deliverable — but incomplete/partially stale (its own checklist entry admits a 2026-07-18 citation correction; it does not yet list the two already-fixed reply-integrity gaps or the two newly-found ones) |
ha-term-safety/spec.json known_limitations (3 entries) + tasks.md:195 |
Present — already contain two of the three documentation-drift notes this audit would otherwise re-derive (dormant on_send_append_entries/raft_send_append_entries send path; advisory-vs-authoritative commit index) |
raft-sim-conformance/spec.json known_limitations (3 entries) |
Present — contains the follower-commit-index documentation-drift note |
| A standalone, dedicated conformance-map document | Missing (no such file exists yet under .kiro/specs/raft-adapter-conformance/ or docs/) |
Gap tag: Constraint — CLAUDE.md's table is intentionally terse (a living quick-reference, not an evidentiary audit trail); the systematic pass's detailed findings (file:line citations, live-vs-dormant verdicts, reasoning) don't fit there without bloating it.
Requirement 2 — AppendEntries-equivalent rejection reply integrity
| Sub-area | Asset | Status |
|---|---|---|
| WAL catch-up reject reply | ha_send_wal_ack_reply_ex (bug-1479, ha.c ~6680+), WalAckReply schema (controller.capnp:876-880, already has term, lsn, success) |
Fixed and live. Gap tag: Unknown — need to confirm a regression test explicitly asserts a reject reply is sent (not just that the NACK routes once received); test_ha_core_safety.c has adjacent pin tests (test_2_3_termblind_route_payload_guarded, test_2_3_unrepairable_catchup_raises_nonpromotable_fence) but their scope wasn't fully re-verified this pass. |
| Config-push reject reply | handle_repl_config_push (config_repl.c:1261-1337); dispatch site cluster.c:652-655 (rc = handle_repl_config_push(...), no reply built on any rc); contrast cluster.c:767-772 default: case, which does send a generic error reply on unknown opcode |
Missing — confirmed live gap. Read in full this pass: 4 early returns (-EINVAL malformed body ~1266, -EINVAL missing bytes/sha256 ~1286, -EBADMSG sha mismatch ~1294, rc from conf_save_config failure ~1309) all return before constructing any repl_config_push_reply_t. Only the success tail (~1326-1335) builds and sends {accepted=1, version_tag}. The outer dispatcher never synthesizes a reply from a nonzero rc for this opcode — logged bug-1480. |
ReplConfigPushReply schema |
controller.capnp:805-808: struct ReplConfigPushReply { accepted @0: Bool; versionTag @1: UInt64; } |
Constraint, not itself missing for the reject-reply fix — adding accepted=0 needs no schema change (the field already exists); only Requirement 7's term field is a schema addition (see below). |
| Distinguishing reject from activation-deferred | config_activate_committed returning -EAGAIN (activation gated on committed marker, by design) does not early-return in handle_repl_config_push — the function still falls through to send accepted=1 in that case |
Important design nuance for the fix: accepted currently means "bytes saved out-of-band," not "activated." The reject-reply fix must not conflate the two — an -EAGAIN activation defer is expected/retriable and must keep sending accepted=1; only the four genuine save/validation failures above should send accepted=0. |
Requirement 3 — AppendEntries send retry/backoff, single authoritative mechanism
| Asset | Status |
|---|---|
raft_send_append_entries (raft.c:595) — protocol-conformant, nextIndex-driven |
Confirmed zero production callers (only raft_sim.c and an XCE_TESTING helper ha_cb_send_append_entries_for_test) — re-verified this pass via full-repo grep. |
ha_cb_send_append_entries_stub (ha.c:2533-2574) — the registered on_send_append_entries callback, delegates to repl_send_append_entries |
Registered (s_real_cbs) but only reachable via raft_send_append_entries, which nothing in the live controller calls — so this callback is wired but dormant, confirmed by its own call graph, not just the sibling-spec note. |
repl_broadcast (repl.c:295-380) + ha_stream_wal_to_standby (config_repl.c, catch-up streamer) |
This is the actual production send/retry mechanism — unconditional newest-entry broadcast plus heartbeat-driven catch-up range streaming, not nextIndex-indexed. |
A prior, already-recorded design decision — test_ha_core_safety.c:2474: "DECISION: DOCUMENT SAFE-BY-GUARD, do not structurally retire." Reasoning given: the term-blind branches this dead path would exercise are (a) unreachable with real data today, and (b) even if reached, already guarded elsewhere (persist_wal_write_replicated's (term,lsn,payload-crc) guard + sticky non-promotable fence). |
This materially changes the decision space for Requirement 3 AC3. A prior investigation already considered and rejected structural retirement, for reasons that still hold. This is not new information the design phase needs to re-derive — it's an existing, reasoned precedent. |
Gap tag: Constraint (design precedent already exists and argues against literal retirement) — see Options below.
Requirement 4 — RequestVote/PreVote reply integrity, all failure modes
| Asset | Status |
|---|---|
vote_ep_err_handler (ha.c:7564-7608, read in full this pass) |
Confirmed: election-phase branch calls handle_request_vote_reply(ha, vctx->ctx, vctx->peer_idx, vctx->my_term, 0, 0, 0) directly — the legacy path, bypassing vote_reply_dispatch/pre_vote_reply_dispatch. |
vote_ep_ctx_t (ha.c:46-61, full struct read this pass) |
Missing a discriminant. The struct has no field distinguishing a real-vote endpoint from a PreVote endpoint — peer_idx, my_term, my_tag, my_lsn, my_last_log_term/index, my_name, but nothing like is_prevote. vote_ep_err_handler is registered identically at both call sites (ha.c:688-689 real-vote, ha.c:827-828 PreVote) and currently has no way to route correctly even if it tried. |
vote_reply_dispatch (ha.c:6333-6394), pre_vote_reply_dispatch (ha.c:7982-8000) |
Both exist and are already the correct production tally paths for the connect-time/monitor-time failures bug-1423/1424 fixed. |
Gap tag: Missing (discriminant field + routing branch) — concrete, bounded fix; not an unknown.
Requirement 5 — ClientRequest / log append integrity
| Asset | Status |
|---|---|
raft_log_append (raft.c:562-593), ha_primary_log_append (ha.c:4777-4823), append_job_wal_entry_local/persist_submit_job_payloads_async (persist.c:6081-6199) |
Traced and confirmed conformant this pass (persist-before-act, no phantom log-length advance, failures propagated per-payload, first-failure rc propagated across a batch). |
Gap tag: Unknown (test-coverage only) — no code gap found; need to confirm which existing test IDs already assert "failed persist ⇒ no log-length advance ⇒ caller sees the failure" as a locked invariant, or whether this needs a new regression test. Not investigated further this pass (out of gap-analysis scope per the framework's "defer deep research to design phase").
Requirement 6 — AdvanceCommitIndex / commit-source-ownership
| Asset | Status |
|---|---|
raft_advance_commit (raft.c:769-821), ha_cb_commit_advanced_stub (ha.c:2616-2667) → commit_set_raft_commit_index (commit.c:1082-1095, advisory-only, monotonic) |
Confirmed conformant. |
majority_candidate_locked (commit.c:121-198) — the actual authoritative ceiling, capped at persist_durable_lsn |
Confirmed authoritative; commit.c:139-155 comment documents why a prior min(durable-majority, raft_commit_index) coupling was tried and removed (starvation). |
XCE_TESTING assert at commit.c:222 (committed_lsn never exceeds persist_durable_lsn) |
Existing regression guard already present — a strong asset; likely already catches any future accidental re-coupling. |
Gap tag: Low — mostly already regression-guarded by the existing XCE_TESTING assert; confirm during design whether an additional explicit test naming raft_commit_index as advisory-only is warranted, or whether the existing assert suffices.
Requirement 7 — Term discovery completeness
| Asset | Status |
|---|---|
Heartbeat-reply (ha.c:6636-6651) and WAL-NACK (ha.c:6746-6784) term-adoption vectors, both feeding raft_on_higher_term_observed (raft.c:855-874) |
Confirmed fixed and live (bug-1421). |
WalAckReply schema already carries term @0 |
Present — no schema work needed for the WAL-NACK vector. |
ReplConfigPushReply schema (controller.capnp:805-808) |
Missing a term field — confirmed by direct read this pass: only accepted/versionTag. Adding a reply-carried term here is an additive .capnp schema change (new field number, e.g. term @2: UInt64), not a breaking one. |
| Cap'n Proto codegen build step | Unknown this pass — need to confirm which Meson target regenerates the .capnp.c/.capnp.h from controller.capnp (likely capnpc invoked from the top-level meson.build), and whether the historical nixSupport/xce.nix capnpc-pin issue (bug-918, already fixed) has any bearing on a schema field addition (as opposed to a c-capnproto version bump) — almost certainly not, but worth a one-line confirmation in design rather than assuming. |
Gap tag: Missing (schema field) + Unknown (codegen build-step confirmation, low-risk).
Requirement 8 — Restart/recovery reconciliation
| Asset | Status |
|---|---|
ha_sync_applied_lsn_from_recovery/ha_sync_raft_log_from_recovery (ha.c:5296-5351), phase2 retry helper (ha.c:415-501), promotion call sites (ha.c:1867-1897, 2112-2131), JIT log_len resync (ha.c:3200-3204) |
Confirmed conformant and already fixed (bug-1042, bug-1477, bug-1475). |
Gap tag: Unknown (test-coverage only, same caveat as Requirement 5/6) — no code gap found.
Implementation Approach Options
Requirement 1 (audit deliverable)
- Option A — Extend
CLAUDE.md's existing tables. Update the "TLA+ Action-to-Code Mapping" and "Protocol Verification Checklist" in place; add rows/checklist items for the reply-integrity guarantees this feature establishes. ✅ single source of truth, no new doc to keep in sync. ❌ CLAUDE.md is meant to stay terse — doesn't have room for full per-finding evidence (file:line trails, live-vs-dormant reasoning) without bloating a document every session reads. - Option B — New standalone conformance-map document (e.g.
.kiro/specs/raft-adapter-conformance/design.md's own dedicated section, or adocs/raft-adapter-conformance.md). ✅ room for full evidentiary detail. ❌ a second place that can drift from CLAUDE.md's summary table, recreating exactly the "documentation drift" class of problem this feature exists to fix. - Option C — Hybrid (recommended direction for design phase). CLAUDE.md's table stays the terse, always-current pointer (updated as part of this feature's fixes); the full audit trail with citations and verdicts lives in this spec's own
design.md, referenced from CLAUDE.md by name so there is one canonical detailed source and one canonical terse pointer, not two competing summaries.
Requirement 2 (rejection reply integrity)
- Option A — Extend
handle_repl_config_pushin place, mirroring theha_send_wal_ack_reply_expattern already established for the WAL catch-up fix: build and send aReplConfigPushReply{accepted=0}at each of the four reject early-returns, taking care not to touch the-EAGAINactivation-defer path (which correctly staysaccepted=1). ✅ minimal, uses an established in-codebase pattern, touches one function. ❌ none significant — this is a narrow, well-bounded fix. - No Option B/C considered necessary — the fix is a direct, small extension of an existing function using a precedent already proven in production (bug-1479). Effort/risk assessment below reflects this.
Requirement 3 (single authoritative send/retry mechanism)
- Option A — Document as intentionally dormant (recommended, matches existing precedent). Formalize the
test_ha_core_safety.c:2474decision into the design doc and CLAUDE.md:raft_send_append_entries/ha_cb_send_append_entries_stubremain in the codebase as a library-conformant reference implementation exercised only byraft_sim/tests, explicitly marked as such;repl_broadcast+ catch-up streaming remains the sole production mechanism. No behavior change. ✅ lowest risk, respects a reasoned prior decision, zero regression surface. ❌ leaves two implementations in the codebase long-term (a documentation cost, not a correctness one). - Option B — Structurally retire the dead path (delete
raft_send_append_entries's production wiring, keep it library-internal/test-only via#ifdef XCE_TESTINGor move it fully intoraft_sim.c). ✅ removes the "two implementations" documentation burden entirely. ❌ contradicts the specific reasoning already recorded against this (loses a working, protocol-exact reference implementationraft_simand future specs may want); touches library-public API (raft.h:378) that other tests reference directly. - Option C — Wire in as the real production path, replacing
repl_broadcast/catch-up-streaming with nextIndex-driven sends. ✅ closest toraft.tlaliterally. ❌ highest risk by far — replaces a converging, live-verified (post bug-1479) core replication mechanism with an unexercised-in-production one; would need its own multi-round chaotic-soak verification before trust; effort XL, not justified by any live defect (the current substitute already converges correctly).
Requirement 4 (vote/PreVote failure routing)
- Option A — Add a discriminant field to
vote_ep_ctx_t(e.g.bool is_prevote), set at allocation (ha.c:629real-vote alloc →false,ha.c:784PreVote alloc →true), then branch insidevote_ep_err_handler's election-phase case to callvote_reply_dispatchorpre_vote_reply_dispatchaccordingly. ✅ single handler retained, minimal struct change, mirrors howmy_term/peer_idxare already threaded per-context. ❌ handler gains a branch, slightly more complex than today. - Option B — Split into two handlers (
vote_ep_err_handler/pre_vote_ep_err_handler), registered at their respective existing call sites (ha.c:688-689vs827-828, which are already separate registration points today). ✅ no runtime branch, each handler stays single-purpose, matches the existing pattern where connect/monitor failures are already handled at separate call sites for the two endpoint kinds. ❌ near-duplicate logic in two functions (theCTRL_PRIMARYbranch — standby-endpoint cleanup — is identical in both; would need a shared helper to avoid copy-paste). - Recommendation for design phase: Option B composed with a small shared helper for the
CTRL_PRIMARYcleanup branch is likely cleanest, but this is a genuine, low-stakes design choice — either is a bounded, low-risk change.
Requirements 5, 6, 8 (already-conformant areas)
- Option A only, no alternatives needed — these are regression-lock requirements. The task is to confirm or add test coverage that pins the already-correct behavior (existing
XCE_TESTINGassert atcommit.c:222for Requirement 6 is a strong existing asset; Requirements 5 and 8 need a design-phase check of which existing test IDs already exercise the traced code paths before deciding whether any new test is needed at all).
Requirement 7 (term discovery / config-push term field)
- Option A — Add
term @2: UInt64toReplConfigPushReply, populate it inhandle_repl_config_push's reply construction (both the existing success path and the new Requirement 2 reject paths), and route it through the sameraft_on_higher_term_observedadoption call already used for the heartbeat and WAL-NACK vectors. ✅ additive schema change (safe, no wire-format break), reuses the established adoption entry point, and — since Requirement 2 already touches every reply-construction site in this function for the reject-reply fix — can be implemented in the same edit pass with no extra integration surface. This pairing should be called out explicitly in the design/tasks phase so it isn't accidentally split into two separate change sets touching the same lines twice.
Effort & Risk
| Requirement | Effort | Risk | Justification |
|---|---|---|---|
| 1 — Audit deliverable | S (1-3 days) | Low | Documentation + citation verification; no code risk. |
| 2 — Rejection reply integrity | S (extends one existing function with a proven pattern) | Low | Bounded to handle_repl_config_push; must respect the -EAGAIN activation-defer nuance found this pass, otherwise trivial. |
| 3 — Single authoritative send mechanism | S if Option A (document only) / XL if Option C (wire in) | Low (Option A) to High (Option C) | A prior in-codebase decision already argues for Option A; Option C would require full chaotic-soak reverification of a currently-working guarantee for no confirmed live defect. |
| 4 — Vote/PreVote failure routing | S-M | Low-Medium | Small struct/handler change, but touches election-path code; needs a live/soak test that can actually inject a post-connect (not connect-time) endpoint failure — that injection capability's current existence is unconfirmed (see Research Needed). |
| 5 — ClientRequest integrity | S (test-only, likely) | Low | No code gap found; effort is test-coverage confirmation. |
| 6 — Commit source-of-truth | S (test-only, likely) | Low | Existing XCE_TESTING assert already guards this. |
| 7 — Term discovery completeness | S-M (schema change + wiring, shares an edit pass with Requirement 2) | Low-Medium | Additive schema field is low-risk; the only real unknown is confirming the capnp codegen build step, not the change itself. |
| 8 — Restart/recovery reconciliation | S (test-only, likely) | Low | No code gap found; already-fixed by bug-1042/1477/1475. |
Research Needed (carried forward to design phase)
- Test-coverage confirmation for Requirements 5, 6, 8 — identify the specific existing test IDs (in
test_ha_core_safety.c,test_apply_pump.c,test_commit_durability.c, or elsewhere) that already lock each already-conformant guarantee, versus needing a net-new regression test. Not investigated deeply this pass per the gap-analysis framework's "defer deep research to design." - Post-connect endpoint failure injection — confirm whether the existing test harness (
test_ha_vote_send.cor similar) can already simulate a failure arriving after a successful connect+monitor (as opposed to a connect-time failure, which is already tested per bug-1423/1424's regression coverage) — needed to properly test the Requirement 4 fix. - Cap'n Proto codegen build step — confirm the exact Meson target/invocation that regenerates
controller.capnp.c/.hfrom the schema, for the Requirement 7 field addition. Low risk expected (additive field, not ac-capnprotoversion change like bug-918), but unconfirmed this pass. handle_repl_config_push's WAL catch-up sibling regression test scope — re-confirm (rather than assume from prior-session notes) exactly whichtest_ha_core_safety.ctest IDs assert the bug-1479 WAL-ACK reject-reply fix, so the new config-push reject-reply test can be modeled on the same pattern.- Requirement 3 wording tension — Requirement 3 AC3 currently reads "...shall be either retired or made the production path." The asset found this pass (
test_ha_core_safety.c:2474's prior "document, do not structurally retire" decision) suggests a third real option — formally document as intentionally dormant — that isn't literally "retired." Flag for the design phase to either treat "retired" broadly enough to cover a documented-dormant-with-guard outcome, or revisit the requirement's wording before design finalizes.
Recommendations Summary
- Requirements 2 and 7 should be planned as a single combined change (same function, same schema, same edit pass) rather than sequenced separately.
- Requirement 3's design decision should start from the existing
test_ha_core_safety.c:2474precedent (Option A, document-only) rather than re-litigating retire-vs-wire-in from scratch; only revisit if the design phase surfaces a concrete reason the prior reasoning no longer holds. - Requirements 5, 6, and 8 are low-effort, test-coverage-confirmation items — worth batching into one design-phase task rather than three separate ones.
- Requirement 4's Option A vs B (discriminant field vs split handler) is a genuine, low-stakes design choice; either is acceptable, defer to design phase author preference.
Design-Phase Discovery & Decisions
This section extends the gap analysis above with the design-phase discovery pass and the decisions it resolved. Discovery type: Extension (existing controller subsystem, no new external dependency) — light discovery process applied.
Summary
- Feature:
raft-adapter-conformance - Discovery Scope: Extension
- Key Findings:
- The Cap'n Proto codegen path for the schema change (Req 2/7) is a pre-existing, unconditional Meson
custom_target(src/lib/meson.build:1-10,capnp compile -o c:lib) — an additive field needs no build-system change, closing Research Needed item 3 from the gap analysis with high confidence. ha_send_wal_ack_reply_ex(ha.c:3095-3116) already establishes the exact pattern to mirror for the config-push reject-reply fix: anXCE_TESTING-only send-hook (ha_wal_ack_send_hook) that lets a unit test observe the reply payload without a live endpoint. This resolves the "how do we test this" question for Req 2 by precedent rather than invention.- Existing regression coverage was confirmed strong for three of the four "already-conformant" requirements: Req 6 (
test_term_safety_adapter.c:646-780, explicit bidirectional advisory-vs-authoritative pins) and Req 8 (test_persist_recovery.c,test_config_repl.c) already lock the guarantees. Req 5 has partial coverage (test_commit_barrier_rollout.c,test_commit_barrier_reply.c) but no test was found asserting batch-wide failure propagation specifically — flagged as a small net-new test item, not a code gap. - No existing test exercises
vote_ep_err_handlerat all (grepacrosstest_ha_vote_send.cfor the handler name,RDHUP, oron_errreturned zero matches) — Req 4 needs net-new test infrastructure, not just a net-new test case. This is a real (small) infrastructure gap, not a research unknown anymore. - No existing test was found asserting that a WAL catch-up reject actually sends
WalAckReply{success=0}on the wire (as opposed to testing what happens downstream once a NACK is routed) — the bug-1479 fix itself is under-tested at the reply-construction boundary. Folded into this feature's Testing Strategy as a regression-lock item for Req 2.1.
- The Cap'n Proto codegen path for the schema change (Req 2/7) is a pre-existing, unconditional Meson
Research Log
Cap'n Proto schema regeneration path
- Context: Requirements 2 and 7 both need
ReplConfigPushReplyto gain atermfield; needed to confirm this doesn't require a Nix/build change given the historicalnixSupport/xce.nixcapnpc-pin issue (bug-918, memory-recorded from a prior session). - Sources Consulted:
src/lib/meson.build(read directly),src/schema/controller.capnp(read directly). - Findings:
src/lib/meson.build:1-10unconditionally regeneratescontroller.capnp.c/.hfromsrc/schema/controller.capnpon every build viacapnp compile -o c:libusing thecapnpc-cplugin and thec-capnprotodependency. Bug-918 was about the Nix derivation's own pin for thec-capnprototoolchain source (a mutable upstreammainref), not about schema content — an additive field is orthogonal to that concern. - Implications: No meson/Nix change needed for the schema addition; a normal
/xce-build controller(orall) picks it up. No new Technology Stack row needed beyond noting the existing Cap'n Proto toolchain already covers this.
Existing reply-hook test pattern
- Context: Needed a concrete, low-risk way to unit-test the new config-push reject-reply construction without standing up a live endpoint pair.
- Sources Consulted:
ha.c:3085-3131(ha_wal_ack_send_hook,ha_send_wal_ack_reply_ex,ha_send_wal_ack_reply), read directly. - Findings: The WAL-ACK path already has exactly this shape: an
XCE_TESTING-guarded global function pointer that, when set, receives the constructed reply payload instead of the function sending over the wire. This is bug-1479's own test-support mechanism. - Implications: The config-push reject-reply fix should add an analogous
XCE_TESTING-only hook (e.g.ha_config_push_reply_send_hook) rather than inventing a different test-injection mechanism — keeps the two reply-integrity fixes symmetric and reviewable by the same pattern.
Regression-coverage confirmation for Requirements 5, 6, 8
- Context: Gap analysis flagged these as "Unknown — test-coverage only"; needed to resolve before Testing Strategy could be written concretely.
- Sources Consulted:
grepacrosssrc/controller/testsuite/*.cfor the traced functions/concepts, read matched line ranges directly. - Findings: Requirement 6 has strong existing dedicated coverage (
test_term_safety_adapter.c, tests explicitly pin both directions: a lagging advisory index does not capcommitted_lsn, and a well-ahead advisory index does not raise it past the durable-majority floor). Requirement 8 has existing coverage intest_persist_recovery.candtest_config_repl.c. Requirement 5 has partial coverage; no test specifically named or scoped to "one failed payload in a batch propagates to all" was found. - Implications: Requirements 6 and 8 need no new tests — the design's Testing Strategy references the existing test files as the regression lock. Requirement 5 gets one small net-new test item (batch failure propagation) rather than a full new suite.
Vote endpoint failure-path test infrastructure
- Context: Needed to confirm whether
vote_ep_err_handler's failure path (the Requirement 4 fix target) has any existing test scaffolding. - Sources Consulted:
grep -n "vote_ep_err_handler\|RDHUP\|on_err" src/controller/testsuite/test_ha_vote_send.c— zero matches. - Findings: No existing test drives
vote_ep_err_handlerat all, for either the already-fixed connect-time paths or the not-yet-fixed post-connect path. This is a genuine test-infrastructure gap (need a way to invoke the handler directly with a syntheticvote_ep_ctx_t/endpoint, e.g. a smallXCE_TESTING-only constructor mirroringvote_ep_ctx_build_for_testatha.c:7337, which already exists for a different purpose and can likely be reused/extended). - Implications: Requirement 4's task will need a small test-support addition (reusing
vote_ep_ctx_build_for_test), not just a new test case in isolation.
Architecture Pattern Evaluation
| Option | Description | Strengths | Risks / Limitations | Notes |
|---|---|---|---|---|
| Extend in place (chosen for all code fixes) | Add the missing reply/routing logic directly inside the existing adapter functions (handle_repl_config_push, vote_ep_err_handler) |
Matches the bug-1479 precedent exactly; no new abstraction; minimal review surface | None material — these are narrow, well-bounded functions already | Consistent with ha_send_wal_ack_reply_ex's own extension of ha_send_wal_ack_reply |
| New shared "reply integrity" abstraction (rejected) | A generic cross-cutting helper/interface for "send an explicit reply on every reject path" spanning WAL-ACK, config-push, and vote replies | Would prevent the class of bug from recurring elsewhere by construction | The three reply shapes (WalAckReply, ReplConfigPushReply, RequestVoteReply) are distinct Cap'n Proto structs with different fields and different dispatch semantics (one is a direct reply, one is a tally-feed); forcing a shared interface over them adds indirection with only three call sites — fails the Simplification lens (no second concrete beneficiary today) |
Documented here so a future spec doesn't have to re-derive why this was rejected |
Design Decisions
Decision: Documentation-only resolution for the dead nextIndex-driven AppendEntries sender (Requirement 3)
- Context:
raft_send_append_entries/ha_cb_send_append_entries_stubis fully protocol-conformant but has zero production callers;repl_broadcast+ catch-up streaming is the actual production mechanism. - Alternatives Considered:
- Retire the dead path (remove production wiring, keep library-only/test-only).
- Wire it in as the real production path, replacing
repl_broadcast. - Formally document it as intentionally dormant, matching the reasoning already recorded at
test_ha_core_safety.c:2474.
- Selected Approach: Option 3. No source code changes. Update
src/controller/CLAUDE.md's TLA+ Action-to-Code Mapping to noteAppendEntries(i,j)production behavior isrepl_broadcast/ha_stream_wal_to_standby, withraft_send_append_entriesexplicitly marked as a dormant, protocol-exact reference path exercised only byraft_sim/tests. Add a one-line cross-reference note toha-term-safety/spec.json's existingknown_limitationsentries (index 30) pointing at this spec's resolution, so a future reader lands on the authoritative decision instead of re-discovering the drift. - Rationale: A prior investigation already weighed retire-vs-keep and rejected retirement because the term-blind branches this path would otherwise expose are unreachable with real data today and independently guarded by
persist_wal_write_replicated's conflict guard even if reached. Re-litigating that decision without new information would violate the Simplification lens in reverse (removing working, guarded code for documentation tidiness alone). Wiring it in (Option 2) is rejected as disproportionate risk (XL effort, full chaotic-soak reverification) for zero confirmed live defect in the current mechanism. - Trade-offs: Keeps two implementations of the same guarantee in the codebase long-term (a documentation cost); avoids all correctness/regression risk.
- Follow-up: Requirement 3's task should be scoped as a documentation task with zero
src/controller/*.cdiff; reviewers should treat any code-touching PR for this requirement as scope creep unless a new live defect is found.
Decision: Split vote_ep_err_handler into two handlers sharing a helper (Requirement 4)
- Context:
vote_ep_ctx_thas no field distinguishing a real-vote endpoint from a PreVote endpoint, and the shared handler's election-phase branch calls the legacyhandle_request_vote_replyinstead of the production tally path for post-connect failures. - Alternatives Considered:
- Add a boolean discriminant field to
vote_ep_ctx_tand branch inside the one handler. - Split into
vote_ep_err_handler/pre_vote_ep_err_handler, each registered at its existing distinct call site, sharing a small helper for the identicalCTRL_PRIMARYcleanup branch.
- Add a boolean discriminant field to
- Selected Approach: Option 2.
- Rationale: The two endpoint kinds are already registered at distinct call sites today (
ha.c:688-689vs827-828) even though they share thevote_ep_ctx_ttype — the split mirrors an architectural seam that already exists, rather than introducing a new one. It avoids adding a field that only exists to support a runtime branch in one function. - Trade-offs: Two small handler functions instead of one slightly larger one; a shared static helper for the
CTRL_PRIMARYbranch avoids duplicating that logic. - Follow-up: The task should reuse
vote_ep_ctx_build_for_test(ha.c:7337) as the test-construction seam for both new handlers' unit tests, per the test-infrastructure finding above.
Risks & Mitigations
- Adding
termtoReplConfigPushReplychanges a wire struct consumed during rolling upgrades — mitigated because Cap'n Proto field addition is backward/forward compatible by construction (old readers ignore the new field; an old sender's reply decodes withterm == 0on a new-code primary, which is naturally safe since0can never look like a higher term than the primary's own and so triggers no adoption — see Migration Strategy indesign.md). - Splitting
vote_ep_err_handlertouches live election-path code — mitigated by keeping theCTRL_PRIMARYbranch behavior byte-for-byte identical (extracted, not rewritten) and requiring the new dedicated unit tests (via the reused test-construction helper) before this task is considered done. - No test currently exists for the exact reply-construction boundary of the already-shipped bug-1479 fix — mitigated by adding it as an explicit Requirement 2.1 regression-lock test in this feature rather than assuming the existing catch-up/fence tests already cover it.
References
src/controller/testsuite/test_ha_core_safety.c:2440-2480— the prior "DOCUMENT SAFE-BY-GUARD, do not structurally retire" decision informing Requirement 3.src/lib/meson.build:1-10— Cap'n Proto codegen custom_target.src/controller/ha.c:3085-3131—ha_send_wal_ack_reply_ex/ha_wal_ack_send_hook, the precedent pattern for Requirement 2's test hook.