myxce/.kiro/specs/raft-adapter-conformance/research.md
Rongsong Shen d4b80010b1 docs(raft-adapter-conformance): commit final design.md/research.md and requirements/spec.json state
Spec reached GO with 15/15 tasks complete (see prior task-confirmation
commits); these doc artifacts were generated during the design/research
phases but never separately staged.
2026-07-31 09:29:35 +00:00

34 KiB

Gap Analysis: raft-adapter-conformance

Analysis date: 2026-07-29. Requirements phase: generated (not yet approved). Per skill fallback rule, proceeding — gap analysis findings can also inform requirement refinement before approval.

Method

Codebase investigation used direct Read/grep against src/controller/ha.c, repl.c, config_repl.c, cluster.c, src/lib/raft/raft.c/raft.h, src/schema/controller.capnp, and src/controller/testsuite/, plus the two Explore-agent passes already run during requirements generation (their findings are the primary input to Requirement-to-Asset mapping below; this pass adds the concrete integration-point evidence — schema fields, dispatcher code, struct layouts — needed to select an implementation approach). No external dependency research was needed; this is entirely an internal-codebase conformance question.

Requirement-to-Asset Map

Requirement 1 — Systematic conformance audit deliverable

Asset Status
raft/raft.tla (canonical action list: Restart, Timeout, RequestVote, BecomeLeader, ClientRequest, AdvanceCommitIndex, AppendEntries, Receive→UpdateTerm/HandleRequestVoteRequest/HandleRequestVoteResponse/HandleAppendEntriesRequest/HandleAppendEntriesResponse) Present, unmodified upstream copy
src/controller/CLAUDE.md → "TLA+ Action-to-Code Mapping" + "Protocol Verification Checklist" tables Present and already the natural home for this deliverable — but incomplete/partially stale (its own checklist entry admits a 2026-07-18 citation correction; it does not yet list the two already-fixed reply-integrity gaps or the two newly-found ones)
ha-term-safety/spec.json known_limitations (3 entries) + tasks.md:195 Present — already contain two of the three documentation-drift notes this audit would otherwise re-derive (dormant on_send_append_entries/raft_send_append_entries send path; advisory-vs-authoritative commit index)
raft-sim-conformance/spec.json known_limitations (3 entries) Present — contains the follower-commit-index documentation-drift note
A standalone, dedicated conformance-map document Missing (no such file exists yet under .kiro/specs/raft-adapter-conformance/ or docs/)

Gap tag: Constraint — CLAUDE.md's table is intentionally terse (a living quick-reference, not an evidentiary audit trail); the systematic pass's detailed findings (file:line citations, live-vs-dormant verdicts, reasoning) don't fit there without bloating it.

Requirement 2 — AppendEntries-equivalent rejection reply integrity

Sub-area Asset Status
WAL catch-up reject reply ha_send_wal_ack_reply_ex (bug-1479, ha.c ~6680+), WalAckReply schema (controller.capnp:876-880, already has term, lsn, success) Fixed and live. Gap tag: Unknown — need to confirm a regression test explicitly asserts a reject reply is sent (not just that the NACK routes once received); test_ha_core_safety.c has adjacent pin tests (test_2_3_termblind_route_payload_guarded, test_2_3_unrepairable_catchup_raises_nonpromotable_fence) but their scope wasn't fully re-verified this pass.
Config-push reject reply handle_repl_config_push (config_repl.c:1261-1337); dispatch site cluster.c:652-655 (rc = handle_repl_config_push(...), no reply built on any rc); contrast cluster.c:767-772 default: case, which does send a generic error reply on unknown opcode Missing — confirmed live gap. Read in full this pass: 4 early returns (-EINVAL malformed body ~1266, -EINVAL missing bytes/sha256 ~1286, -EBADMSG sha mismatch ~1294, rc from conf_save_config failure ~1309) all return before constructing any repl_config_push_reply_t. Only the success tail (~1326-1335) builds and sends {accepted=1, version_tag}. The outer dispatcher never synthesizes a reply from a nonzero rc for this opcode — logged bug-1480.
ReplConfigPushReply schema controller.capnp:805-808: struct ReplConfigPushReply { accepted @0: Bool; versionTag @1: UInt64; } Constraint, not itself missing for the reject-reply fix — adding accepted=0 needs no schema change (the field already exists); only Requirement 7's term field is a schema addition (see below).
Distinguishing reject from activation-deferred config_activate_committed returning -EAGAIN (activation gated on committed marker, by design) does not early-return in handle_repl_config_push — the function still falls through to send accepted=1 in that case Important design nuance for the fix: accepted currently means "bytes saved out-of-band," not "activated." The reject-reply fix must not conflate the two — an -EAGAIN activation defer is expected/retriable and must keep sending accepted=1; only the four genuine save/validation failures above should send accepted=0.

Requirement 3 — AppendEntries send retry/backoff, single authoritative mechanism

Asset Status
raft_send_append_entries (raft.c:595) — protocol-conformant, nextIndex-driven Confirmed zero production callers (only raft_sim.c and an XCE_TESTING helper ha_cb_send_append_entries_for_test) — re-verified this pass via full-repo grep.
ha_cb_send_append_entries_stub (ha.c:2533-2574) — the registered on_send_append_entries callback, delegates to repl_send_append_entries Registered (s_real_cbs) but only reachable via raft_send_append_entries, which nothing in the live controller calls — so this callback is wired but dormant, confirmed by its own call graph, not just the sibling-spec note.
repl_broadcast (repl.c:295-380) + ha_stream_wal_to_standby (config_repl.c, catch-up streamer) This is the actual production send/retry mechanism — unconditional newest-entry broadcast plus heartbeat-driven catch-up range streaming, not nextIndex-indexed.
A prior, already-recorded design decision — test_ha_core_safety.c:2474: "DECISION: DOCUMENT SAFE-BY-GUARD, do not structurally retire." Reasoning given: the term-blind branches this dead path would exercise are (a) unreachable with real data today, and (b) even if reached, already guarded elsewhere (persist_wal_write_replicated's (term,lsn,payload-crc) guard + sticky non-promotable fence). This materially changes the decision space for Requirement 3 AC3. A prior investigation already considered and rejected structural retirement, for reasons that still hold. This is not new information the design phase needs to re-derive — it's an existing, reasoned precedent.

Gap tag: Constraint (design precedent already exists and argues against literal retirement) — see Options below.

Requirement 4 — RequestVote/PreVote reply integrity, all failure modes

Asset Status
vote_ep_err_handler (ha.c:7564-7608, read in full this pass) Confirmed: election-phase branch calls handle_request_vote_reply(ha, vctx->ctx, vctx->peer_idx, vctx->my_term, 0, 0, 0) directly — the legacy path, bypassing vote_reply_dispatch/pre_vote_reply_dispatch.
vote_ep_ctx_t (ha.c:46-61, full struct read this pass) Missing a discriminant. The struct has no field distinguishing a real-vote endpoint from a PreVote endpoint — peer_idx, my_term, my_tag, my_lsn, my_last_log_term/index, my_name, but nothing like is_prevote. vote_ep_err_handler is registered identically at both call sites (ha.c:688-689 real-vote, ha.c:827-828 PreVote) and currently has no way to route correctly even if it tried.
vote_reply_dispatch (ha.c:6333-6394), pre_vote_reply_dispatch (ha.c:7982-8000) Both exist and are already the correct production tally paths for the connect-time/monitor-time failures bug-1423/1424 fixed.

Gap tag: Missing (discriminant field + routing branch) — concrete, bounded fix; not an unknown.

Requirement 5 — ClientRequest / log append integrity

Asset Status
raft_log_append (raft.c:562-593), ha_primary_log_append (ha.c:4777-4823), append_job_wal_entry_local/persist_submit_job_payloads_async (persist.c:6081-6199) Traced and confirmed conformant this pass (persist-before-act, no phantom log-length advance, failures propagated per-payload, first-failure rc propagated across a batch).

Gap tag: Unknown (test-coverage only) — no code gap found; need to confirm which existing test IDs already assert "failed persist ⇒ no log-length advance ⇒ caller sees the failure" as a locked invariant, or whether this needs a new regression test. Not investigated further this pass (out of gap-analysis scope per the framework's "defer deep research to design phase").

Requirement 6 — AdvanceCommitIndex / commit-source-ownership

Asset Status
raft_advance_commit (raft.c:769-821), ha_cb_commit_advanced_stub (ha.c:2616-2667) → commit_set_raft_commit_index (commit.c:1082-1095, advisory-only, monotonic) Confirmed conformant.
majority_candidate_locked (commit.c:121-198) — the actual authoritative ceiling, capped at persist_durable_lsn Confirmed authoritative; commit.c:139-155 comment documents why a prior min(durable-majority, raft_commit_index) coupling was tried and removed (starvation).
XCE_TESTING assert at commit.c:222 (committed_lsn never exceeds persist_durable_lsn) Existing regression guard already present — a strong asset; likely already catches any future accidental re-coupling.

Gap tag: Low — mostly already regression-guarded by the existing XCE_TESTING assert; confirm during design whether an additional explicit test naming raft_commit_index as advisory-only is warranted, or whether the existing assert suffices.

Requirement 7 — Term discovery completeness

Asset Status
Heartbeat-reply (ha.c:6636-6651) and WAL-NACK (ha.c:6746-6784) term-adoption vectors, both feeding raft_on_higher_term_observed (raft.c:855-874) Confirmed fixed and live (bug-1421).
WalAckReply schema already carries term @0 Present — no schema work needed for the WAL-NACK vector.
ReplConfigPushReply schema (controller.capnp:805-808) Missing a term field — confirmed by direct read this pass: only accepted/versionTag. Adding a reply-carried term here is an additive .capnp schema change (new field number, e.g. term @2: UInt64), not a breaking one.
Cap'n Proto codegen build step Unknown this pass — need to confirm which Meson target regenerates the .capnp.c/.capnp.h from controller.capnp (likely capnpc invoked from the top-level meson.build), and whether the historical nixSupport/xce.nix capnpc-pin issue (bug-918, already fixed) has any bearing on a schema field addition (as opposed to a c-capnproto version bump) — almost certainly not, but worth a one-line confirmation in design rather than assuming.

Gap tag: Missing (schema field) + Unknown (codegen build-step confirmation, low-risk).

Requirement 8 — Restart/recovery reconciliation

Asset Status
ha_sync_applied_lsn_from_recovery/ha_sync_raft_log_from_recovery (ha.c:5296-5351), phase2 retry helper (ha.c:415-501), promotion call sites (ha.c:1867-1897, 2112-2131), JIT log_len resync (ha.c:3200-3204) Confirmed conformant and already fixed (bug-1042, bug-1477, bug-1475).

Gap tag: Unknown (test-coverage only, same caveat as Requirement 5/6) — no code gap found.

Implementation Approach Options

Requirement 1 (audit deliverable)

  • Option A — Extend CLAUDE.md's existing tables. Update the "TLA+ Action-to-Code Mapping" and "Protocol Verification Checklist" in place; add rows/checklist items for the reply-integrity guarantees this feature establishes. ✅ single source of truth, no new doc to keep in sync. ❌ CLAUDE.md is meant to stay terse — doesn't have room for full per-finding evidence (file:line trails, live-vs-dormant reasoning) without bloating a document every session reads.
  • Option B — New standalone conformance-map document (e.g. .kiro/specs/raft-adapter-conformance/design.md's own dedicated section, or a docs/raft-adapter-conformance.md). ✅ room for full evidentiary detail. ❌ a second place that can drift from CLAUDE.md's summary table, recreating exactly the "documentation drift" class of problem this feature exists to fix.
  • Option C — Hybrid (recommended direction for design phase). CLAUDE.md's table stays the terse, always-current pointer (updated as part of this feature's fixes); the full audit trail with citations and verdicts lives in this spec's own design.md, referenced from CLAUDE.md by name so there is one canonical detailed source and one canonical terse pointer, not two competing summaries.

Requirement 2 (rejection reply integrity)

  • Option A — Extend handle_repl_config_push in place, mirroring the ha_send_wal_ack_reply_ex pattern already established for the WAL catch-up fix: build and send a ReplConfigPushReply{accepted=0} at each of the four reject early-returns, taking care not to touch the -EAGAIN activation-defer path (which correctly stays accepted=1). ✅ minimal, uses an established in-codebase pattern, touches one function. ❌ none significant — this is a narrow, well-bounded fix.
  • No Option B/C considered necessary — the fix is a direct, small extension of an existing function using a precedent already proven in production (bug-1479). Effort/risk assessment below reflects this.

Requirement 3 (single authoritative send/retry mechanism)

  • Option A — Document as intentionally dormant (recommended, matches existing precedent). Formalize the test_ha_core_safety.c:2474 decision into the design doc and CLAUDE.md: raft_send_append_entries/ha_cb_send_append_entries_stub remain in the codebase as a library-conformant reference implementation exercised only by raft_sim/tests, explicitly marked as such; repl_broadcast + catch-up streaming remains the sole production mechanism. No behavior change. ✅ lowest risk, respects a reasoned prior decision, zero regression surface. ❌ leaves two implementations in the codebase long-term (a documentation cost, not a correctness one).
  • Option B — Structurally retire the dead path (delete raft_send_append_entries's production wiring, keep it library-internal/test-only via #ifdef XCE_TESTING or move it fully into raft_sim.c). ✅ removes the "two implementations" documentation burden entirely. ❌ contradicts the specific reasoning already recorded against this (loses a working, protocol-exact reference implementation raft_sim and future specs may want); touches library-public API (raft.h:378) that other tests reference directly.
  • Option C — Wire in as the real production path, replacing repl_broadcast/catch-up-streaming with nextIndex-driven sends. ✅ closest to raft.tla literally. ❌ highest risk by far — replaces a converging, live-verified (post bug-1479) core replication mechanism with an unexercised-in-production one; would need its own multi-round chaotic-soak verification before trust; effort XL, not justified by any live defect (the current substitute already converges correctly).

Requirement 4 (vote/PreVote failure routing)

  • Option A — Add a discriminant field to vote_ep_ctx_t (e.g. bool is_prevote), set at allocation (ha.c:629 real-vote alloc → false, ha.c:784 PreVote alloc → true), then branch inside vote_ep_err_handler's election-phase case to call vote_reply_dispatch or pre_vote_reply_dispatch accordingly. ✅ single handler retained, minimal struct change, mirrors how my_term/peer_idx are already threaded per-context. ❌ handler gains a branch, slightly more complex than today.
  • Option B — Split into two handlers (vote_ep_err_handler / pre_vote_ep_err_handler), registered at their respective existing call sites (ha.c:688-689 vs 827-828, which are already separate registration points today). ✅ no runtime branch, each handler stays single-purpose, matches the existing pattern where connect/monitor failures are already handled at separate call sites for the two endpoint kinds. ❌ near-duplicate logic in two functions (the CTRL_PRIMARY branch — standby-endpoint cleanup — is identical in both; would need a shared helper to avoid copy-paste).
  • Recommendation for design phase: Option B composed with a small shared helper for the CTRL_PRIMARY cleanup branch is likely cleanest, but this is a genuine, low-stakes design choice — either is a bounded, low-risk change.

Requirements 5, 6, 8 (already-conformant areas)

  • Option A only, no alternatives needed — these are regression-lock requirements. The task is to confirm or add test coverage that pins the already-correct behavior (existing XCE_TESTING assert at commit.c:222 for Requirement 6 is a strong existing asset; Requirements 5 and 8 need a design-phase check of which existing test IDs already exercise the traced code paths before deciding whether any new test is needed at all).

Requirement 7 (term discovery / config-push term field)

  • Option A — Add term @2: UInt64 to ReplConfigPushReply, populate it in handle_repl_config_push's reply construction (both the existing success path and the new Requirement 2 reject paths), and route it through the same raft_on_higher_term_observed adoption call already used for the heartbeat and WAL-NACK vectors. ✅ additive schema change (safe, no wire-format break), reuses the established adoption entry point, and — since Requirement 2 already touches every reply-construction site in this function for the reject-reply fix — can be implemented in the same edit pass with no extra integration surface. This pairing should be called out explicitly in the design/tasks phase so it isn't accidentally split into two separate change sets touching the same lines twice.

Effort & Risk

Requirement Effort Risk Justification
1 — Audit deliverable S (1-3 days) Low Documentation + citation verification; no code risk.
2 — Rejection reply integrity S (extends one existing function with a proven pattern) Low Bounded to handle_repl_config_push; must respect the -EAGAIN activation-defer nuance found this pass, otherwise trivial.
3 — Single authoritative send mechanism S if Option A (document only) / XL if Option C (wire in) Low (Option A) to High (Option C) A prior in-codebase decision already argues for Option A; Option C would require full chaotic-soak reverification of a currently-working guarantee for no confirmed live defect.
4 — Vote/PreVote failure routing S-M Low-Medium Small struct/handler change, but touches election-path code; needs a live/soak test that can actually inject a post-connect (not connect-time) endpoint failure — that injection capability's current existence is unconfirmed (see Research Needed).
5 — ClientRequest integrity S (test-only, likely) Low No code gap found; effort is test-coverage confirmation.
6 — Commit source-of-truth S (test-only, likely) Low Existing XCE_TESTING assert already guards this.
7 — Term discovery completeness S-M (schema change + wiring, shares an edit pass with Requirement 2) Low-Medium Additive schema field is low-risk; the only real unknown is confirming the capnp codegen build step, not the change itself.
8 — Restart/recovery reconciliation S (test-only, likely) Low No code gap found; already-fixed by bug-1042/1477/1475.

Research Needed (carried forward to design phase)

  1. Test-coverage confirmation for Requirements 5, 6, 8 — identify the specific existing test IDs (in test_ha_core_safety.c, test_apply_pump.c, test_commit_durability.c, or elsewhere) that already lock each already-conformant guarantee, versus needing a net-new regression test. Not investigated deeply this pass per the gap-analysis framework's "defer deep research to design."
  2. Post-connect endpoint failure injection — confirm whether the existing test harness (test_ha_vote_send.c or similar) can already simulate a failure arriving after a successful connect+monitor (as opposed to a connect-time failure, which is already tested per bug-1423/1424's regression coverage) — needed to properly test the Requirement 4 fix.
  3. Cap'n Proto codegen build step — confirm the exact Meson target/invocation that regenerates controller.capnp.c/.h from the schema, for the Requirement 7 field addition. Low risk expected (additive field, not a c-capnproto version change like bug-918), but unconfirmed this pass.
  4. handle_repl_config_push's WAL catch-up sibling regression test scope — re-confirm (rather than assume from prior-session notes) exactly which test_ha_core_safety.c test IDs assert the bug-1479 WAL-ACK reject-reply fix, so the new config-push reject-reply test can be modeled on the same pattern.
  5. Requirement 3 wording tension — Requirement 3 AC3 currently reads "...shall be either retired or made the production path." The asset found this pass (test_ha_core_safety.c:2474's prior "document, do not structurally retire" decision) suggests a third real option — formally document as intentionally dormant — that isn't literally "retired." Flag for the design phase to either treat "retired" broadly enough to cover a documented-dormant-with-guard outcome, or revisit the requirement's wording before design finalizes.

Recommendations Summary

  • Requirements 2 and 7 should be planned as a single combined change (same function, same schema, same edit pass) rather than sequenced separately.
  • Requirement 3's design decision should start from the existing test_ha_core_safety.c:2474 precedent (Option A, document-only) rather than re-litigating retire-vs-wire-in from scratch; only revisit if the design phase surfaces a concrete reason the prior reasoning no longer holds.
  • Requirements 5, 6, and 8 are low-effort, test-coverage-confirmation items — worth batching into one design-phase task rather than three separate ones.
  • Requirement 4's Option A vs B (discriminant field vs split handler) is a genuine, low-stakes design choice; either is acceptable, defer to design phase author preference.

Design-Phase Discovery & Decisions

This section extends the gap analysis above with the design-phase discovery pass and the decisions it resolved. Discovery type: Extension (existing controller subsystem, no new external dependency) — light discovery process applied.

Summary

  • Feature: raft-adapter-conformance
  • Discovery Scope: Extension
  • Key Findings:
    • The Cap'n Proto codegen path for the schema change (Req 2/7) is a pre-existing, unconditional Meson custom_target (src/lib/meson.build:1-10, capnp compile -o c:lib) — an additive field needs no build-system change, closing Research Needed item 3 from the gap analysis with high confidence.
    • ha_send_wal_ack_reply_ex (ha.c:3095-3116) already establishes the exact pattern to mirror for the config-push reject-reply fix: an XCE_TESTING-only send-hook (ha_wal_ack_send_hook) that lets a unit test observe the reply payload without a live endpoint. This resolves the "how do we test this" question for Req 2 by precedent rather than invention.
    • Existing regression coverage was confirmed strong for three of the four "already-conformant" requirements: Req 6 (test_term_safety_adapter.c:646-780, explicit bidirectional advisory-vs-authoritative pins) and Req 8 (test_persist_recovery.c, test_config_repl.c) already lock the guarantees. Req 5 has partial coverage (test_commit_barrier_rollout.c, test_commit_barrier_reply.c) but no test was found asserting batch-wide failure propagation specifically — flagged as a small net-new test item, not a code gap.
    • No existing test exercises vote_ep_err_handler at all (grep across test_ha_vote_send.c for the handler name, RDHUP, or on_err returned zero matches) — Req 4 needs net-new test infrastructure, not just a net-new test case. This is a real (small) infrastructure gap, not a research unknown anymore.
    • No existing test was found asserting that a WAL catch-up reject actually sends WalAckReply{success=0} on the wire (as opposed to testing what happens downstream once a NACK is routed) — the bug-1479 fix itself is under-tested at the reply-construction boundary. Folded into this feature's Testing Strategy as a regression-lock item for Req 2.1.

Research Log

Cap'n Proto schema regeneration path

  • Context: Requirements 2 and 7 both need ReplConfigPushReply to gain a term field; needed to confirm this doesn't require a Nix/build change given the historical nixSupport/xce.nix capnpc-pin issue (bug-918, memory-recorded from a prior session).
  • Sources Consulted: src/lib/meson.build (read directly), src/schema/controller.capnp (read directly).
  • Findings: src/lib/meson.build:1-10 unconditionally regenerates controller.capnp.c/.h from src/schema/controller.capnp on every build via capnp compile -o c:lib using the capnpc-c plugin and the c-capnproto dependency. Bug-918 was about the Nix derivation's own pin for the c-capnproto toolchain source (a mutable upstream main ref), not about schema content — an additive field is orthogonal to that concern.
  • Implications: No meson/Nix change needed for the schema addition; a normal /xce-build controller (or all) picks it up. No new Technology Stack row needed beyond noting the existing Cap'n Proto toolchain already covers this.

Existing reply-hook test pattern

  • Context: Needed a concrete, low-risk way to unit-test the new config-push reject-reply construction without standing up a live endpoint pair.
  • Sources Consulted: ha.c:3085-3131 (ha_wal_ack_send_hook, ha_send_wal_ack_reply_ex, ha_send_wal_ack_reply), read directly.
  • Findings: The WAL-ACK path already has exactly this shape: an XCE_TESTING-guarded global function pointer that, when set, receives the constructed reply payload instead of the function sending over the wire. This is bug-1479's own test-support mechanism.
  • Implications: The config-push reject-reply fix should add an analogous XCE_TESTING-only hook (e.g. ha_config_push_reply_send_hook) rather than inventing a different test-injection mechanism — keeps the two reply-integrity fixes symmetric and reviewable by the same pattern.

Regression-coverage confirmation for Requirements 5, 6, 8

  • Context: Gap analysis flagged these as "Unknown — test-coverage only"; needed to resolve before Testing Strategy could be written concretely.
  • Sources Consulted: grep across src/controller/testsuite/*.c for the traced functions/concepts, read matched line ranges directly.
  • Findings: Requirement 6 has strong existing dedicated coverage (test_term_safety_adapter.c, tests explicitly pin both directions: a lagging advisory index does not cap committed_lsn, and a well-ahead advisory index does not raise it past the durable-majority floor). Requirement 8 has existing coverage in test_persist_recovery.c and test_config_repl.c. Requirement 5 has partial coverage; no test specifically named or scoped to "one failed payload in a batch propagates to all" was found.
  • Implications: Requirements 6 and 8 need no new tests — the design's Testing Strategy references the existing test files as the regression lock. Requirement 5 gets one small net-new test item (batch failure propagation) rather than a full new suite.

Vote endpoint failure-path test infrastructure

  • Context: Needed to confirm whether vote_ep_err_handler's failure path (the Requirement 4 fix target) has any existing test scaffolding.
  • Sources Consulted: grep -n "vote_ep_err_handler\|RDHUP\|on_err" src/controller/testsuite/test_ha_vote_send.c — zero matches.
  • Findings: No existing test drives vote_ep_err_handler at all, for either the already-fixed connect-time paths or the not-yet-fixed post-connect path. This is a genuine test-infrastructure gap (need a way to invoke the handler directly with a synthetic vote_ep_ctx_t/endpoint, e.g. a small XCE_TESTING-only constructor mirroring vote_ep_ctx_build_for_test at ha.c:7337, which already exists for a different purpose and can likely be reused/extended).
  • Implications: Requirement 4's task will need a small test-support addition (reusing vote_ep_ctx_build_for_test), not just a new test case in isolation.

Architecture Pattern Evaluation

Option Description Strengths Risks / Limitations Notes
Extend in place (chosen for all code fixes) Add the missing reply/routing logic directly inside the existing adapter functions (handle_repl_config_push, vote_ep_err_handler) Matches the bug-1479 precedent exactly; no new abstraction; minimal review surface None material — these are narrow, well-bounded functions already Consistent with ha_send_wal_ack_reply_ex's own extension of ha_send_wal_ack_reply
New shared "reply integrity" abstraction (rejected) A generic cross-cutting helper/interface for "send an explicit reply on every reject path" spanning WAL-ACK, config-push, and vote replies Would prevent the class of bug from recurring elsewhere by construction The three reply shapes (WalAckReply, ReplConfigPushReply, RequestVoteReply) are distinct Cap'n Proto structs with different fields and different dispatch semantics (one is a direct reply, one is a tally-feed); forcing a shared interface over them adds indirection with only three call sites — fails the Simplification lens (no second concrete beneficiary today) Documented here so a future spec doesn't have to re-derive why this was rejected

Design Decisions

Decision: Documentation-only resolution for the dead nextIndex-driven AppendEntries sender (Requirement 3)

  • Context: raft_send_append_entries/ha_cb_send_append_entries_stub is fully protocol-conformant but has zero production callers; repl_broadcast + catch-up streaming is the actual production mechanism.
  • Alternatives Considered:
    1. Retire the dead path (remove production wiring, keep library-only/test-only).
    2. Wire it in as the real production path, replacing repl_broadcast.
    3. Formally document it as intentionally dormant, matching the reasoning already recorded at test_ha_core_safety.c:2474.
  • Selected Approach: Option 3. No source code changes. Update src/controller/CLAUDE.md's TLA+ Action-to-Code Mapping to note AppendEntries(i,j) production behavior is repl_broadcast/ha_stream_wal_to_standby, with raft_send_append_entries explicitly marked as a dormant, protocol-exact reference path exercised only by raft_sim/tests. Add a one-line cross-reference note to ha-term-safety/spec.json's existing known_limitations entries (index 30) pointing at this spec's resolution, so a future reader lands on the authoritative decision instead of re-discovering the drift.
  • Rationale: A prior investigation already weighed retire-vs-keep and rejected retirement because the term-blind branches this path would otherwise expose are unreachable with real data today and independently guarded by persist_wal_write_replicated's conflict guard even if reached. Re-litigating that decision without new information would violate the Simplification lens in reverse (removing working, guarded code for documentation tidiness alone). Wiring it in (Option 2) is rejected as disproportionate risk (XL effort, full chaotic-soak reverification) for zero confirmed live defect in the current mechanism.
  • Trade-offs: Keeps two implementations of the same guarantee in the codebase long-term (a documentation cost); avoids all correctness/regression risk.
  • Follow-up: Requirement 3's task should be scoped as a documentation task with zero src/controller/*.c diff; reviewers should treat any code-touching PR for this requirement as scope creep unless a new live defect is found.

Decision: Split vote_ep_err_handler into two handlers sharing a helper (Requirement 4)

  • Context: vote_ep_ctx_t has no field distinguishing a real-vote endpoint from a PreVote endpoint, and the shared handler's election-phase branch calls the legacy handle_request_vote_reply instead of the production tally path for post-connect failures.
  • Alternatives Considered:
    1. Add a boolean discriminant field to vote_ep_ctx_t and branch inside the one handler.
    2. Split into vote_ep_err_handler / pre_vote_ep_err_handler, each registered at its existing distinct call site, sharing a small helper for the identical CTRL_PRIMARY cleanup branch.
  • Selected Approach: Option 2.
  • Rationale: The two endpoint kinds are already registered at distinct call sites today (ha.c:688-689 vs 827-828) even though they share the vote_ep_ctx_t type — the split mirrors an architectural seam that already exists, rather than introducing a new one. It avoids adding a field that only exists to support a runtime branch in one function.
  • Trade-offs: Two small handler functions instead of one slightly larger one; a shared static helper for the CTRL_PRIMARY branch avoids duplicating that logic.
  • Follow-up: The task should reuse vote_ep_ctx_build_for_test (ha.c:7337) as the test-construction seam for both new handlers' unit tests, per the test-infrastructure finding above.

Risks & Mitigations

  • Adding term to ReplConfigPushReply changes a wire struct consumed during rolling upgrades — mitigated because Cap'n Proto field addition is backward/forward compatible by construction (old readers ignore the new field; an old sender's reply decodes with term == 0 on a new-code primary, which is naturally safe since 0 can never look like a higher term than the primary's own and so triggers no adoption — see Migration Strategy in design.md).
  • Splitting vote_ep_err_handler touches live election-path code — mitigated by keeping the CTRL_PRIMARY branch behavior byte-for-byte identical (extracted, not rewritten) and requiring the new dedicated unit tests (via the reused test-construction helper) before this task is considered done.
  • No test currently exists for the exact reply-construction boundary of the already-shipped bug-1479 fix — mitigated by adding it as an explicit Requirement 2.1 regression-lock test in this feature rather than assuming the existing catch-up/fence tests already cover it.

References

  • src/controller/testsuite/test_ha_core_safety.c:2440-2480 — the prior "DOCUMENT SAFE-BY-GUARD, do not structurally retire" decision informing Requirement 3.
  • src/lib/meson.build:1-10 — Cap'n Proto codegen custom_target.
  • src/controller/ha.c:3085-3131 — ha_send_wal_ack_reply_ex/ha_wal_ack_send_hook, the precedent pattern for Requirement 2's test hook.