myxce/.kiro/specs/raft-adapter-conformance/design.md

Design Document: raft-adapter-conformance

Overview

This feature closes the wire-level reply-integrity gaps between the XCE controller's Raft adapter (src/controller/ha.c, repl.c, config_repl.c) and the protocol guarantees raft/raft.tla assumes always hold, and produces a durable, action-by-action conformance record so the next gap is found by inspection rather than by a live incident. It extends the work already landed for bug-1479 (the WAL catch-up reject-reply fix and the catch-up streaming-ceiling fix) to the two other places the same class of defect was found live during this feature's discovery pass: the configuration-push reply channel and the vote/PreVote endpoint failure-routing path.

Users: HA cluster operators (who experience the failure mode as a stalled or slow-to-converge cluster) and XCE maintainers (who consume the conformance record when reasoning about future HA changes).

Impact: Two adapter functions gain previously-missing reply/routing behavior (handle_repl_config_push, vote_ep_err_handler); one Cap'n Proto struct gains an additive field (ReplConfigPushReply.term); src/controller/CLAUDE.md's existing TLA+ mapping tables are extended and reconciled against production behavior; no other subsystem changes.

Goals

• Every reject of a replication-class message (WAL catch-up entry, configuration push) produces an explicit reply distinguishable from a lost message.
• Every vote/PreVote request outcome, including a failure arriving after connect, reaches the production tally.
• Every reply channel that can carry a peer's observed term can trigger term adoption.
• A single, explicit, written decision resolves which AppendEntries send mechanism is production-authoritative.
• A durable, precise mapping exists from each raft.tla action to the adapter code that realizes it, with a conformance verdict.
• The four already-conformant guarantee areas (ClientRequest, AdvanceCommitIndex/commit-source-ownership, restart/recovery, the two already-fixed term-adoption vectors) are protected by regression coverage so a future change cannot silently reopen them.

Non-Goals

• No change to src/lib/raft/raft.c's protocol logic (election, logOk, (term,index) truncation, UpdateTerm) — validated by raft-log-replication/raft-commit-safety/raft-sim-conformance.
• No configuration-upgrade materialized-state work (config-upgrade-snapshot's scope).
• No controller voting-set/membership change support (ha-config-bootstrap's scope).
• No change to committed_lsn advancement semantics beyond confirming the existing durable-majority-authoritative behavior is preserved.
• No wiring-in of raft_send_append_entries as the production send path (see Decision in Architecture below) — it remains dormant, now formally documented as such.

Boundary Commitments

This Spec Owns

• Reply-integrity behavior on the standby → primary direction for WAL catch-up rejects (regression-lock only, already shipped) and configuration-push rejects (new fix).
• Failure-routing behavior for vote/PreVote endpoints, across all failure timings (connect-time — regression-lock, already shipped; post-connect — new fix).
• The term field addition to ReplConfigPushReply and its use for term-adoption.
• The written conformance record (CLAUDE.md extension) mapping raft.tla actions to adapter code, including the explicit "documented dormant" resolution for the unused nextIndex-driven AppendEntries sender.
• Regression tests that lock in both the already-shipped guarantees and the guarantees this feature adds.

Out of Boundary

• The Raft protocol library itself (src/lib/raft/raft.c/raft.h) — this feature never modifies it, only verifies the adapter drives it faithfully.
• Configuration-upgrade materialized-state consistency (config-upgrade-snapshot).
• Controller membership/quorum-set changes (ha-config-bootstrap).
• Client-side primary-discovery (bug-854) and multi-segment catch-up payload handling (bug-847).
• Any change to committed_lsn advancement authority (ha-commit-engine/ha-term-safety own that decision; this feature only confirms it is unchanged).

Allowed Dependencies

• Raft library callback contract and entry points (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from, on_commit_advanced, raft_log_append, raft_on_append_entries_request/reply, raft_advance_commit, vote/pre-vote handling) — consumed as-is, never modified.
• The existing out-of-band configuration-push transport (ha-term-safety) — this feature adds reply integrity to it without changing what it carries.
• The existing Cap'n Proto codegen build step (src/lib/meson.build) — consumed as-is for the additive schema field.
• Constraint: no dependency direction change — the adapter continues to depend on the library, never the reverse.

Revalidation Triggers

• Any change to vote_ep_ctx_t's layout or to how vote/PreVote endpoints are registered.
• Any change to ReplConfigPushReply's wire shape (this feature is itself one such change; a future one must re-check config_activate_committed's -EAGAIN handling described below).
• Any change that reintroduces a hard coupling between raft_commit_index and committed_lsn advancement (would violate a guarantee this feature regression-locks).
• Any change that gives raft_send_append_entries a production caller (would invalidate the "documented dormant" resolution and require revisiting Requirement 3's decision).

Architecture

Existing Architecture Analysis

The controller is a thin adapter over the Raft library (ha-term-safety's framing, confirmed still accurate): the library owns election/commit/truncate decisions; the adapter's job is to persist what the library decides (on_persist_entry), source terms for it (on_get_entry_term), transmit its decisions on the wire, and turn wire replies back into library calls (raft_on_append_entries_reply, raft_on_vote_reply, raft_on_pre_vote_reply). This feature's defects are all instances of one pattern: an adapter code path receives or is meant to send a wire message, but a specific branch of that path (a reject, a late failure) returns without ever calling the corresponding "turn this into a library-visible outcome" step. The fix pattern established by bug-1479 (ha_send_wal_ack_reply_ex, plus its XCE_TESTING send-hook for testability) is reused verbatim for the two new fixes rather than inventing a new mechanism.

Integration points that must be maintained: the ha_dispatch_gate opcode allowlist (unaffected — no new opcode), the five-callback contract (unaffected — no callback signature changes), the two-tier replication model (unaffected — this feature only makes existing tiers report failure explicitly).

Technical debt addressed: three silent-drop-on-reject sites (WAL catch-up — done pre-feature; config-push — this feature) and one silent-drop-on-late-failure site (vote/PreVote post-connect). Technical debt formally reconciled rather than removed: the dormant nextIndex-driven AppendEntries sender (documented, not deleted — see Decision below).

Architecture Pattern & Boundary Map

graph TB
    subgraph Library
        RaftLib[Raft library raft.c]
    end
    subgraph Adapter
        HaC[ha.c callbacks and dispatch]
        ReplC[repl.c WAL replication]
        ConfigReplC[config_repl.c config push]
    end
    subgraph Wire
        WalAck[WalAckReply term lsn success]
        ConfigPushReply[ReplConfigPushReply accepted versionTag term]
        VoteReply[RequestVoteReply and PreVoteReply]
    end
    subgraph Peer
        Standby[Standby controller]
        Primary[Primary controller]
    end

    RaftLib --> HaC
    HaC --> ReplC
    HaC --> ConfigReplC
    ReplC --> WalAck
    ConfigReplC --> ConfigPushReply
    HaC --> VoteReply
    WalAck --> Primary
    ConfigPushReply --> Primary
    VoteReply --> Primary
    Standby --> WalAck
    Standby --> ConfigPushReply
    Standby --> VoteReply
    Primary --> RaftLib

Architecture Integration:

• Selected pattern: extend-in-place (see research.md Architecture Pattern Evaluation) — reply-integrity fixes land inside the existing functions that already own each reply, no new cross-cutting abstraction introduced.
• Domain boundaries: ha.c owns election/vote wiring and top-level dispatch; repl.c owns WAL replication send/ack; config_repl.c owns out-of-band configuration push/ack — unchanged by this feature.
• Existing patterns preserved: the XCE_TESTING send-hook pattern (ha_wal_ack_send_hook), the raft_on_higher_term_observed single term-adoption entry point, the vote_reply_dispatch/pre_vote_reply_dispatch production tally paths.
• New components: none introduce a new boundary; all four code changes (config-push reply, config-push term field, vote endpoint handler split, CLAUDE.md conformance record) extend existing ownership.
• Steering compliance: preserves the async-I/O thread-role contract (all changes are in already-non-blocking callback/handler code, no new blocking calls introduced) and the "controller never self-terminates" invariant (no new terminal paths added).

Decision: dormant AppendEntries sender resolution (Requirement 3)

No architecture change. raft_send_append_entries/ha_cb_send_append_entries_stub remains in the codebase, unused in production, and is now explicitly documented as such (see research.md Design Decisions for the full rationale). repl_broadcast + ha_stream_wal_to_standby remain the sole production AppendEntries-equivalent send/retry/backoff mechanism. Per Requirement 3.3 (amended after design review to permit this third resolution alongside retire/wire-in), this documentation-only resolution must additionally be backed by a non-regression guard: an XCE_TESTING-only call-counter on ha_cb_send_append_entries_stub (mirroring the ha_wal_ack_send_hook idiom) exercised by a test that drives the production heartbeat/replication path end-to-end and asserts the counter stays at zero while confirming repl_broadcast/ha_stream_wal_to_standby did fire — so a future change that accidentally starts calling this path is caught, not silently accepted. This adds one small ha.c/test diff beyond the pure documentation change described above (see File Structure Plan and Testing Strategy).

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Wire serialization	Cap'n Proto (existing c-capnproto)	ReplConfigPushReply gains an additive term @2: UInt64 field	Regenerated automatically by the existing capnp compile Meson custom_target (src/lib/meson.build:1-10); no build/Nix change needed (see research.md)
Controller adapter	C (gnu17), existing src/controller/	All behavior changes	No new dependency

No other stack layer is touched.

File Structure Plan

This is a small, surgical extension of an existing subsystem — files are listed individually rather than by directory pattern.

Modified Files

• src/schema/controller.capnp — add term @2: UInt64 $C.mapname("term") to struct ReplConfigPushReply (Requirement 7.4).
• src/controller/config_repl.c — handle_repl_config_push: construct and send ReplConfigPushReply{accepted=0, term} on each of its four reject early-returns (malformed request, missing bytes/sha256, sha256 mismatch, conf_save_config failure); populate term on the existing success reply too; preserve the existing -EAGAIN-from-config_activate_committed behavior unchanged (still sends accepted=1 — activation-deferred is not a rejection). Add an XCE_TESTING-only send-hook (e.g. ha_config_push_reply_send_hook), mirroring ha_wal_ack_send_hook (ha.c:3085-3086), so tests can observe the reply payload without a live endpoint. (Requirements 2.2, 2.3, 2.5, 7.4, 7.5)
• src/controller/cluster.c — the existing primary-side ReplConfigPushReply handler (myreq_handler, cluster.c:745-762) already branches on cack->accepted (ACKs via repl_on_config_ack on true, WARN-and-ignores on false — the false branch has simply never fired in production before this feature's fix). Add a term check alongside the existing accepted check: when cack->term exceeds the primary's current term, call raft_on_higher_term_observed, the same single adoption entry point the heartbeat-reply and WAL-NACK vectors already use. (Requirement 7.4, 7.5)
• src/controller/ha.c:
  • Split vote_ep_err_handler into vote_ep_err_handler (real-vote endpoints, registered at the existing ha.c:688-689 call site) and pre_vote_ep_err_handler (PreVote endpoints, registered at the existing ha.c:827-828 call site), each calling vote_reply_dispatch or pre_vote_reply_dispatch respectively for the election-phase branch instead of the legacy handle_request_vote_reply. Extract the identical CTRL_PRIMARY cleanup branch (standby-endpoint removal, repl_remove_standby, quorum-transition logging) into a shared static helper called by both. (Requirements 4.2, 4.3, 4.4)
  • Extend the on_send_append_entries doc-comment at ha.c:2520-2532 to state plainly that this callback is reachable only via raft_send_append_entries, which has no production caller, and point to the CLAUDE.md conformance record for the authoritative production mechanism. Add an XCE_TESTING-only call counter to ha_cb_send_append_entries_stub (e.g. ha_send_append_entries_stub_call_count) as the non-regression guard the amended Requirement 3.3 requires. (Requirement 3.3)
• src/controller/CLAUDE.md — extend the "TLA+ Action-to-Code Mapping" table with corrected/added rows for AppendEntries reply integrity (WAL catch-up and config-push), vote/PreVote failure routing, and the dormant-sender resolution; extend the "Protocol Verification Checklist" with a checklist item for "every reject path sends an explicit reply." (Requirement 1.1-1.6)
• .kiro/specs/ha-term-safety/spec.json — append a one-line pointer on the existing known_limitations entry (index 30, the dormant on_send_append_entries note) referencing this spec's resolution, so a future reader lands on the authoritative decision. (Requirement 1.2, 1.6)

New Test Additions (existing files extended, no new test files needed)

• src/controller/testsuite/test_config_repl.c — new tests: each of the four config-push reject paths sends accepted=0 with the current term; the -EAGAIN activation-defer path still sends accepted=1; a higher term in an inbound config-push reply triggers adoption. (Requirements 2.2, 2.3, 2.5, 7.4, 7.5)
• src/controller/testsuite/test_ha_vote_send.c — new tests: a post-connect endpoint failure on a real-vote endpoint reaches vote_reply_dispatch's tally; the same on a PreVote endpoint reaches pre_vote_reply_dispatch's tally; reuses/extends vote_ep_ctx_build_for_test (ha.c:7337) as the test-construction seam. (Requirements 4.2, 4.3, 4.5)
• src/controller/testsuite/test_ha_core_safety.c — new test asserting ha_send_wal_ack_reply_ex(success=0) is actually invoked (via ha_wal_ack_send_hook) on a WAL catch-up reject, closing the reply-construction-boundary coverage gap found during design discovery. (Requirement 2.1, regression-lock)
• src/controller/testsuite/test_commit_barrier_reply.c or test_commit_barrier_rollout.c — new test: one failed payload in a submitted batch propagates failure to every payload in that batch. (Requirement 5.3)

Unmodified, Regression-Locked Files

These files back the four already-conformant components (ClientLogAppendGuarantee, CommitSourceGuarantee, TermDiscoveryCompleteness, RestartRecoveryGuarantee). No source changes; cited here so each component in the table above resolves to a concrete file, and so a future diff touching them is recognized as revalidation-worthy rather than incidental.

• src/controller/persist.c — ha_primary_log_append, persist_submit_job_payloads_async (Requirement 5); confirmed conformant, existing tests in test_commit_barrier_rollout.c/test_commit_barrier_reply.c/test_term_safety_adapter.c cover it (plus the one new batch-propagation test above).
• src/controller/commit.c — majority_candidate_locked, commit_set_raft_commit_index (Requirement 6); confirmed conformant, test_term_safety_adapter.c:646-780 already pins both directions of the advisory-vs-authoritative split, including the XCE_TESTING assert at commit.c:222.
• src/controller/ha.c — heartbeat-reply and WAL-NACK term-adoption call sites (Requirement 7.1, 7.2), ha_sync_applied_lsn_from_recovery/ha_sync_raft_log_from_recovery/phase2 retry helper/JIT log-length resync (Requirement 8); confirmed conformant, existing coverage in test_persist_recovery.c/test_config_repl.c.

No files are created; no files outside src/controller/ (and its one schema dependency) are touched. src/lib/, src/frontend/, src/worker/, src/scheduler/ are untouched.

System Flows

Configuration-push reject reply (Requirement 2, 7)

sequenceDiagram
    participant Primary
    participant Standby

    Primary->>Standby: ReplConfigPushRequest bytes sha256 versionTag
    alt Malformed or missing fields
        Standby-->>Primary: ReplConfigPushReply accepted false term
    else SHA-256 mismatch
        Standby-->>Primary: ReplConfigPushReply accepted false term
    else conf_save_config fails
        Standby-->>Primary: ReplConfigPushReply accepted false term
    else Save succeeds, activation deferred EAGAIN
        Standby-->>Primary: ReplConfigPushReply accepted true term
        Note over Standby: Activation retried later on commit advance
    else Save and activation succeed
        Standby-->>Primary: ReplConfigPushReply accepted true term
    end
    Primary->>Primary: distinguish reject from timeout, adopt term if higher

Key decision not obvious from the diagram: accepted means "bytes saved out-of-band," not "activated" — the activation-deferred branch and the fully-succeeded branch both reply accepted=true by design, since activation is gated separately on the committed marker (ha-term-safety) and is expected to retry.

Vote/PreVote endpoint failure routing (Requirement 4)

sequenceDiagram
    participant Candidate
    participant Peer

    Candidate->>Peer: connect and monitor
    alt Connect or monitor fails
        Candidate->>Candidate: vote_reply_dispatch or pre_vote_reply_dispatch denied
    else Connect succeeds, later RDHUP or SSL error before reply
        Candidate->>Candidate: vote_ep_err_handler or pre_vote_ep_err_handler
        Candidate->>Candidate: vote_reply_dispatch or pre_vote_reply_dispatch denied
    else Peer replies before any failure
        Peer-->>Candidate: RequestVoteReply or PreVoteReply
        Candidate->>Candidate: vote_reply_dispatch or pre_vote_reply_dispatch
    end
    Candidate->>Candidate: fast quorum decision once every peer outcome is counted

Key decision not obvious from the diagram: today only the "connect or monitor fails" and "peer replies" branches reach the production tally; this feature adds the "later RDHUP" branch to the same tally instead of the legacy no-op-counting path.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1	Action-by-action mapping produced	ConformanceRecord	CLAUDE.md table	—
1.2	Live-vs-dormant path recorded on drift	ConformanceRecord	CLAUDE.md table, ha-term-safety spec.json pointer	Dormant sender decision
1.3	Gap detail sufficient to re-verify	ConformanceRecord	design.md + research.md	—
1.4	Fix vs documentation-drift classification	ConformanceRecord	design.md Decisions	—
1.5	Code-fix findings delivered under Req 2-8	ConformanceRecord	(cross-reference)	—
1.6	Documentation-drift findings recorded for reconciliation	ConformanceRecord	ha-term-safety spec.json pointer	Dormant sender decision
2.1	WAL catch-up reject reply (regression-lock)	WalCatchupReplyIntegrity (existing)	ha_send_wal_ack_reply_ex, ha_wal_ack_send_hook	—
2.2	Config-push reject reply (new)	ConfigPushReplyIntegrity	ReplConfigPushReply{accepted=0}	Config-push reject flow
2.3	Reject distinguishable from transport loss	ConfigPushReplyIntegrity	ReplConfigPushReply vs no reply/timeout	Config-push reject flow
2.4	Unknown outcome not inferred as success	ConfigPushReplyIntegrity, WalCatchupReplyIntegrity	timeout handling (existing)	—
2.5	No regression to success-reply behavior	ConfigPushReplyIntegrity, WalCatchupReplyIntegrity	existing accepted=1 / success=1 paths	Config-push reject flow
3.1	Retry on rejection	Existing (ha_raft_on_repl_nack, repl_start_catchup)	—	—
3.2	Multi-round convergence	Existing (catch-up subsystem)	—	—
3.3	Single authoritative send mechanism	ConformanceRecord (documentation decision)	CLAUDE.md table, ha.c:2520 doc-comment	Dormant sender decision
3.4	Transport-level send failure not treated as caught-up	Existing (repl_broadcast, heartbeat re-check)	—	—
4.1	Connect-time failure counted (regression-lock)	VoteEndpointFailureRouter (existing behavior)	vote_reply_dispatch, pre_vote_reply_dispatch	Vote failure routing
4.2	Post-connect failure counted (new)	VoteEndpointFailureRouter	vote_ep_err_handler, pre_vote_ep_err_handler	Vote failure routing
4.3	Same behavior for PreVote	VoteEndpointFailureRouter	pre_vote_ep_err_handler	Vote failure routing
4.4	No regression to connect/monitor routing	VoteEndpointFailureRouter	shared CTRL_PRIMARY cleanup helper	Vote failure routing
4.5	Fast quorum decision reachable	VoteEndpointFailureRouter	tally functions (existing)	Vote failure routing
5.1	Append persists or fails explicitly	ClientLogAppendGuarantee (existing)	raft_log_append, ha_primary_log_append	—
5.2	No log-length advance on failed persist	ClientLogAppendGuarantee	raft_log_append	—
5.3	Batch failure propagates to whole batch	ClientLogAppendGuarantee	persist_submit_job_payloads_async	—
5.4	No regression to failure propagation	ClientLogAppendGuarantee	existing tests	—
6.1	Durable-majority sole authority	CommitSourceGuarantee (existing)	majority_candidate_locked	—
6.2	Advisory index informational only	CommitSourceGuarantee	commit_set_raft_commit_index	—
6.3	No hard-ceiling reintroduction	CommitSourceGuarantee	commit.c:139-155 invariant	—
6.4	Advance proceeds despite lagging advisory index	CommitSourceGuarantee	existing XCE_TESTING assert	—
7.1	Heartbeat-reply term adoption (regression-lock)	TermDiscoveryCompleteness (existing)	raft_on_higher_term_observed	—
7.2	WAL-NACK term adoption (regression-lock)	TermDiscoveryCompleteness (existing)	raft_on_higher_term_observed	—
7.3	Vote/PreVote reply term adoption (regression-lock)	TermDiscoveryCompleteness (existing)	raft_on_vote_reply Step 1	—
7.4	Config-push reply carries term (new)	ConfigPushReplyIntegrity	ReplConfigPushReply.term	Config-push reject flow
7.5	No regression to existing term-adoption vectors	TermDiscoveryCompleteness, ConfigPushReplyIntegrity	—	Config-push reject flow
8.1	Recovery raises applied LSN/log length/term	RestartRecoveryGuarantee (existing)	ha_sync_applied_lsn_from_recovery, ha_sync_raft_log_from_recovery	—
8.2	Failed recovery step retries, no false-latch	RestartRecoveryGuarantee	phase2 retry helper (ha.c:415-501)	—
8.3	JIT log-length resync beyond boot	RestartRecoveryGuarantee	ha_apply_term_entry_via_library resync	—
8.4	No regression to clean-failover convergence	RestartRecoveryGuarantee	existing tests	—

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies (P0/P1)	Contracts
ConfigPushReplyIntegrity	Controller/Replication	Send an explicit accept/reject reply, carrying term, for every configuration push	2.2, 2.3, 2.5, 7.4, 7.5	ReplConfigPushReply schema (P0)	Event
WalCatchupReplyIntegrity	Controller/Replication	Existing bug-1479 fix; regression-locked, not modified	2.1, 2.4, 2.5	WalAckReply schema (P0, existing)	Event
VoteEndpointFailureRouter	Controller/Election	Route every vote/PreVote endpoint failure, at any timing, to the production tally	4.1-4.5	vote_reply_dispatch (P0), pre_vote_reply_dispatch (P0), vote_ep_ctx_build_for_test (P1, test-only)	Service
ConformanceRecord	Documentation	Durable action-by-action mapping and dormant-path resolution	1.1-1.6, 3.3	CLAUDE.md, ha-term-safety/spec.json	State
ClientLogAppendGuarantee	Controller/Persistence	Existing conformant append-or-explicit-failure behavior; regression-locked	5.1-5.4	persist.c (existing)	State
CommitSourceGuarantee	Controller/Commit	Existing conformant durable-majority-authoritative behavior; regression-locked	6.1-6.4	commit.c (existing)	State
TermDiscoveryCompleteness	Controller/Election	Existing conformant term-adoption vectors (heartbeat, WAL-NACK, vote reply); regression-locked	7.1-7.3, 7.5	raft_on_higher_term_observed (existing)	Event
RestartRecoveryGuarantee	Controller/Recovery	Existing conformant restart/recovery reconcile; regression-locked	8.1-8.4	ha.c recovery functions (existing)	State

Only the two components introducing new code (ConfigPushReplyIntegrity, VoteEndpointFailureRouter) warrant full detail blocks below; the regression-lock components need no new interface — their existing interfaces are cited in the table above and exercised by the tests in File Structure Plan.

Controller / Replication

ConfigPushReplyIntegrity

Field	Detail
Intent	Every configuration-push outcome (accept or reject) produces an explicit ReplConfigPushReply carrying the replying member's current term
Requirements	2.2, 2.3, 2.5, 7.4, 7.5

Responsibilities & Constraints

• Owns the reply-construction boundary inside handle_repl_config_push; does not own configuration activation (config_activate_committed remains the sole activation gate, unchanged).
• Must not conflate an activation-deferred outcome (-EAGAIN) with a rejection — that path continues to reply accepted=1.
• Must include the current term on every reply, success or failure.

Dependencies

• Outbound: ReplConfigPushReply schema field (P0) — the additive term field this component populates.
• Outbound: raft_on_higher_term_observed (P0) — invoked by the primary's inbound-reply handler when this component's reply carries a higher term (existing entry point, no change to it).
• External: none.

Contracts: Event [x]

Event Contract

• Published: ReplConfigPushReply {accepted: bool, versionTag: uint64, term: uint64} — one per ReplConfigPushRequest received, always sent (never silently dropped).
• Subscribed: none (this component only replies).
• Ordering/delivery: request/reply pair over the existing outbound control channel (ha-async-io.md connection ownership); no new ordering guarantee introduced.

Implementation Notes

• Integration: reuses the existing xce_reply/init_reply_by_req pattern already present in handle_repl_config_push's success path; the four reject branches gain the same call shape.
• Validation: XCE_TESTING-only send-hook (mirroring ha_wal_ack_send_hook) lets unit tests assert the reply payload without a live endpoint.
• Risks: none material — narrow, single-function change with a directly analogous shipped precedent (bug-1479).

Controller / Election

VoteEndpointFailureRouter

Field	Detail
Intent	Every vote/PreVote endpoint failure, regardless of when in the round it occurs, is counted through the same production tally as a delivered denial
Requirements	4.1-4.5

Responsibilities & Constraints

• Owns endpoint-failure handling for outgoing vote and PreVote connections (on_err callbacks); does not own the tally logic itself (vote_reply_dispatch/pre_vote_reply_dispatch are consumed, not modified).
• Must preserve the existing CTRL_PRIMARY cleanup behavior (standby-endpoint deregistration, quorum-transition logging) identically for both endpoint kinds.
• Must not introduce a new discriminant field on vote_ep_ctx_t; the split-handler registration already distinguishes endpoint kind by call site.

Dependencies

• Outbound: vote_reply_dispatch (P0) — real-vote tally, already correct for connect/monitor failures.
• Outbound: pre_vote_reply_dispatch (P0) — PreVote tally, already correct for connect/monitor failures.
• Outbound: vote_ep_ctx_build_for_test (P1, test-only) — reused/extended as the unit-test construction seam.
• Inbound: iomgr on_err callback invocation (existing, unchanged trigger).

Contracts: Service [x]

Service Interface

/* vote_ep_err_handler: on_err callback for outgoing real-vote endpoints.
 * Election-phase branch now counts a post-connect failure through
 * vote_reply_dispatch instead of handle_request_vote_reply. */
static int vote_ep_err_handler(void *ep_raw);

/* pre_vote_ep_err_handler: on_err callback for outgoing PreVote endpoints.
 * Election-phase branch counts a post-connect failure through
 * pre_vote_reply_dispatch. */
static int pre_vote_ep_err_handler(void *ep_raw);

/* shared helper: identical CTRL_PRIMARY cleanup extracted from both. */
static void vote_ep_handle_promoted_disconnect(ha_state_t *ha, endpoint_t *ep,
                                                vote_ep_ctx_t *vctx);

• Preconditions: ep->data is a populated vote_ep_ctx_t for the endpoint kind the handler is registered against.
• Postconditions: exactly one outcome (denied/counted) reaches the applicable tally function per invocation; CTRL_PRIMARY-role invocations perform standby cleanup identically to today.
• Invariants: a candidate's own term is never mistaken for an observed peer term through this path (the synthesized outcome carries the candidate's own term, granted=0 — unchanged from today's behavior).

Implementation Notes

• Integration: registration sites (ha.c:688-689, 827-828) each point at the correspondingly-named handler; no change to endpoint setup beyond the callback pointer.
• Validation: new unit tests per File Structure Plan, using the reused test-construction helper to simulate a post-connect failure without a live peer.
• Risks: touches live election-path code — mitigated by keeping the extracted CTRL_PRIMARY branch byte-for-byte identical and requiring the new tests before this task is considered done (see research.md Risks & Mitigations).

Data Models

Data Contracts & Integration

ReplConfigPushReply (Cap'n Proto, src/schema/controller.capnp)

Field	Ordinal	Type	Change
accepted	0	Bool	unchanged
versionTag	1	UInt64	unchanged
term	2	UInt64	new, additive

Backward/forward compatibility: Cap'n Proto additive fields are safe under mixed-version rolling upgrade — an old-code standby's reply (no term field) decodes as term=0 on a new-code primary. This is safe by construction: 0 can never present as a term higher than the primary's own current term (terms start at 1 and only increase), so no spurious adoption occurs; the primary simply doesn't get early term-discovery from an old-code peer's config-push replies, which is exactly today's (pre-feature) behavior for that peer. No explicit migration phasing is required beyond the existing rolling-upgrade tolerance the cluster already relies on.

WalAckReply is unchanged (already carries term, lsn, success); listed here only to confirm no modification is needed.

Error Handling

Error Strategy

This feature's domain is internal cluster wire-protocol reply integrity, not user-facing request handling — the applicable categories are protocol-rejection, transport-failure, and validation-failure, each with a distinct, already-established response pattern that this feature extends consistently.

Error Categories and Responses

• Validation/protocol rejection (malformed config-push request, SHA-256 mismatch, conf_save_config failure): explicit reject reply (accepted=0) carrying term, so the primary can distinguish "actively rejected" from "lost" and react (retry, log, or escalate) — this is the category this feature's fixes target.
• Transport failure (connection drop, send failure, RDHUP): counted as a failure outcome through the same production tally/reply path used for a delivered rejection wherever the protocol model requires an outcome (vote/PreVote tally); where no reply is architecturally expected (snapshot chunk transfer, by explicit design — see requirements.md Boundary Context), recovery remains via the next heartbeat cycle, unchanged by this feature.
• Activation-deferred, not an error (config_activate_committed returning -EAGAIN): explicitly not classified as a rejection; the existing loud-fence-and-retry pattern (steering ha-async-io.md, "controller never self-terminates") already governs it and is preserved unchanged.

Monitoring

No new metrics/logging channels introduced; existing WARN/INFO logging in handle_repl_config_push and vote_ep_err_handler is preserved and extended with the same log level conventions already used in each function.

Testing Strategy

• Unit Tests:
  1. handle_repl_config_push sends ReplConfigPushReply{accepted=0, term} for each of the four reject conditions (malformed request, missing bytes/sha256, sha256 mismatch, save failure) — via the new XCE_TESTING send-hook. (2.2, 2.3, 2.5)
  2. handle_repl_config_push still sends accepted=1 when config_activate_committed returns -EAGAIN (activation-deferred is not a rejection). (2.5, 7.5)
  3. ha_send_wal_ack_reply_ex(success=0) is actually invoked on a WAL catch-up reject (closing the reply-construction-boundary coverage gap identified during design discovery). (2.1)
  4. vote_ep_err_handler routes a synthetic post-connect failure through vote_reply_dispatch's tally, using the reused test-construction helper. (4.2, 4.4)
  5. pre_vote_ep_err_handler routes the same failure shape through pre_vote_reply_dispatch's tally. (4.3)
  6. A submitted batch with one failing payload propagates that failure to every payload in the batch. (5.3)
  7. Driving the production heartbeat/replication path end-to-end never increments ha_send_append_entries_stub_call_count, while repl_broadcast/ha_stream_wal_to_standby do fire for the same scenario — the Requirement 3.3 non-regression guard. (3.3)
• Integration Tests:
  1. A three-node cluster with one standby configured to reject a pushed configuration (bad hash) observes the primary correctly distinguishing the rejection from a timeout and not treating the push as silently lost.
  2. A three-node election round where one peer's vote endpoint fails after connect (simulated) still reaches a quorum decision without waiting for the full election timeout.
  3. Existing test_term_safety_adapter.c advisory-vs-authoritative commit tests re-run unmodified as the Requirement 6 regression lock (no new test needed, confirmed sufficient during design discovery).
  4. Existing test_persist_recovery.c/test_config_repl.c recovery tests re-run unmodified as the Requirement 8 regression lock.
• Live/Soak Tests (per project convention for HA changes, xce-test-ha skill):
  1. A rolling chaotic-failover soak where the test driver issues a configuration push whose payload deliberately does not match the sha256 it advertises (no special fault-injection hook needed — this exercises handle_repl_config_push's real SHA-256-mismatch reject branch), confirming the primary observes the explicit reject rather than degrading to the timeout path.
  2. A soak round exercising the post-connect vote/PreVote endpoint failure path end-to-end. Open question for the tasks phase: unlike item 1, this needs a way to sever an already-established outbound vote connection before a reply arrives (e.g. a firewall-rule toggle or proxy in the soak harness) — no existing xce-test-ha mechanism for this was confirmed during design. If no such mechanism exists cheaply, this item may be deferred; the unit tests (Requirement 4.2/4.3, via vote_ep_ctx_build_for_test) already give direct, deterministic coverage of the fix itself.

Supporting References

• research.md — full gap analysis (asset map, options, effort/risk), design-phase discovery log, and the two recorded design decisions (dormant sender resolution, vote-handler split) with full rationale.
• src/controller/CLAUDE.md — "TLA+ Action-to-Code Mapping" and "Protocol Verification Checklist," extended by this feature as the durable conformance record (Requirement 1).