myxce/.kiro/specs/raft-adapter-conformance/design.md

# Design Document: raft-adapter-conformance

## Overview

This feature closes the wire-level reply-integrity gaps between the XCE controller's Raft adapter (`src/controller/ha.c`, `repl.c`, `config_repl.c`) and the protocol guarantees `raft/raft.tla` assumes always hold, and produces a durable, action-by-action conformance record so the next gap is found by inspection rather than by a live incident. It extends the work already landed for bug-1479 (the WAL catch-up reject-reply fix and the catch-up streaming-ceiling fix) to the two other places the same class of defect was found live during this feature's discovery pass: the configuration-push reply channel and the vote/PreVote endpoint failure-routing path.

**Users**: HA cluster operators (who experience the failure mode as a stalled or slow-to-converge cluster) and XCE maintainers (who consume the conformance record when reasoning about future HA changes).

**Impact**: Two adapter functions gain previously-missing reply/routing behavior (`handle_repl_config_push`, `vote_ep_err_handler`); one Cap'n Proto struct gains an additive field (`ReplConfigPushReply.term`); `src/controller/CLAUDE.md`'s existing TLA+ mapping tables are extended and reconciled against production behavior; no other subsystem changes.

### Goals
- Every reject of a replication-class message (WAL catch-up entry, configuration push) produces an explicit reply distinguishable from a lost message.
- Every vote/PreVote request outcome, including a failure arriving after connect, reaches the production tally.
- Every reply channel that can carry a peer's observed term can trigger term adoption.
- A single, explicit, written decision resolves which `AppendEntries` send mechanism is production-authoritative.
- A durable, precise mapping exists from each `raft.tla` action to the adapter code that realizes it, with a conformance verdict.
- The four already-conformant guarantee areas (`ClientRequest`, `AdvanceCommitIndex`/commit-source-ownership, restart/recovery, the two already-fixed term-adoption vectors) are protected by regression coverage so a future change cannot silently reopen them.

### Non-Goals
- No change to `src/lib/raft/raft.c`'s protocol logic (election, `logOk`, `(term,index)` truncation, `UpdateTerm`) — validated by `raft-log-replication`/`raft-commit-safety`/`raft-sim-conformance`.
- No configuration-*upgrade* materialized-state work (`config-upgrade-snapshot`'s scope).
- No controller voting-set/membership change support (`ha-config-bootstrap`'s scope).
- No change to `committed_lsn` advancement semantics beyond confirming the existing durable-majority-authoritative behavior is preserved.
- No wiring-in of `raft_send_append_entries` as the production send path (see Decision in Architecture below) — it remains dormant, now formally documented as such.

## Boundary Commitments

### This Spec Owns
- Reply-integrity behavior on the standby → primary direction for WAL catch-up rejects (regression-lock only, already shipped) and configuration-push rejects (new fix).
- Failure-routing behavior for vote/PreVote endpoints, across all failure timings (connect-time — regression-lock, already shipped; post-connect — new fix).
- The `term` field addition to `ReplConfigPushReply` and its use for term-adoption.
- The written conformance record (`CLAUDE.md` extension) mapping `raft.tla` actions to adapter code, including the explicit "documented dormant" resolution for the unused nextIndex-driven `AppendEntries` sender.
- Regression tests that lock in both the already-shipped guarantees and the guarantees this feature adds.

### Out of Boundary
- The Raft protocol library itself (`src/lib/raft/raft.c`/`raft.h`) — this feature never modifies it, only verifies the adapter drives it faithfully.
- Configuration-upgrade materialized-state consistency (`config-upgrade-snapshot`).
- Controller membership/quorum-set changes (`ha-config-bootstrap`).
- Client-side primary-discovery (bug-854) and multi-segment catch-up payload handling (bug-847).
- Any change to `committed_lsn` advancement authority (`ha-commit-engine`/`ha-term-safety` own that decision; this feature only confirms it is unchanged).

### Allowed Dependencies
- Raft library callback contract and entry points (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`, `raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`, vote/pre-vote handling) — consumed as-is, never modified.
- The existing out-of-band configuration-push transport (`ha-term-safety`) — this feature adds reply integrity to it without changing what it carries.
- The existing Cap'n Proto codegen build step (`src/lib/meson.build`) — consumed as-is for the additive schema field.
- Constraint: no dependency direction change — the adapter continues to depend on the library, never the reverse.

### Revalidation Triggers
- Any change to `vote_ep_ctx_t`'s layout or to how vote/PreVote endpoints are registered.
- Any change to `ReplConfigPushReply`'s wire shape (this feature is itself one such change; a future one must re-check `config_activate_committed`'s `-EAGAIN` handling described below).
- Any change that reintroduces a hard coupling between `raft_commit_index` and `committed_lsn` advancement (would violate a guarantee this feature regression-locks).
- Any change that gives `raft_send_append_entries` a production caller (would invalidate the "documented dormant" resolution and require revisiting Requirement 3's decision).

## Architecture

### Existing Architecture Analysis

The controller is a thin adapter over the Raft library (`ha-term-safety`'s framing, confirmed still accurate): the library owns election/commit/truncate decisions; the adapter's job is to persist what the library decides (`on_persist_entry`), source terms for it (`on_get_entry_term`), transmit its decisions on the wire, and turn wire replies back into library calls (`raft_on_append_entries_reply`, `raft_on_vote_reply`, `raft_on_pre_vote_reply`). This feature's defects are all instances of one pattern: an adapter code path receives or is meant to send a wire message, but a specific branch of that path (a reject, a late failure) returns without ever calling the corresponding "turn this into a library-visible outcome" step. The fix pattern established by bug-1479 (`ha_send_wal_ack_reply_ex`, plus its `XCE_TESTING` send-hook for testability) is reused verbatim for the two new fixes rather than inventing a new mechanism.

Integration points that must be maintained: the `ha_dispatch_gate` opcode allowlist (unaffected — no new opcode), the five-callback contract (unaffected — no callback signature changes), the two-tier replication model (unaffected — this feature only makes existing tiers report failure explicitly).

Technical debt addressed: three silent-drop-on-reject sites (WAL catch-up — done pre-feature; config-push — this feature) and one silent-drop-on-late-failure site (vote/PreVote post-connect). Technical debt formally reconciled rather than removed: the dormant nextIndex-driven `AppendEntries` sender (documented, not deleted — see Decision below).

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    subgraph Library
        RaftLib[Raft library raft.c]
    end
    subgraph Adapter
        HaC[ha.c callbacks and dispatch]
        ReplC[repl.c WAL replication]
        ConfigReplC[config_repl.c config push]
    end
    subgraph Wire
        WalAck[WalAckReply term lsn success]
        ConfigPushReply[ReplConfigPushReply accepted versionTag term]
        VoteReply[RequestVoteReply and PreVoteReply]
    end
    subgraph Peer
        Standby[Standby controller]
        Primary[Primary controller]
    end

    RaftLib --> HaC
    HaC --> ReplC
    HaC --> ConfigReplC
    ReplC --> WalAck
    ConfigReplC --> ConfigPushReply
    HaC --> VoteReply
    WalAck --> Primary
    ConfigPushReply --> Primary
    VoteReply --> Primary
    Standby --> WalAck
    Standby --> ConfigPushReply
    Standby --> VoteReply
    Primary --> RaftLib
```

**Architecture Integration**:
- Selected pattern: extend-in-place (see `research.md` Architecture Pattern Evaluation) — reply-integrity fixes land inside the existing functions that already own each reply, no new cross-cutting abstraction introduced.
- Domain boundaries: `ha.c` owns election/vote wiring and top-level dispatch; `repl.c` owns WAL replication send/ack; `config_repl.c` owns out-of-band configuration push/ack — unchanged by this feature.
- Existing patterns preserved: the `XCE_TESTING` send-hook pattern (`ha_wal_ack_send_hook`), the `raft_on_higher_term_observed` single term-adoption entry point, the `vote_reply_dispatch`/`pre_vote_reply_dispatch` production tally paths.
- New components: none introduce a new boundary; all four code changes (config-push reply, config-push term field, vote endpoint handler split, CLAUDE.md conformance record) extend existing ownership.
- Steering compliance: preserves the async-I/O thread-role contract (all changes are in already-non-blocking callback/handler code, no new blocking calls introduced) and the "controller never self-terminates" invariant (no new terminal paths added).

### Decision: dormant `AppendEntries` sender resolution (Requirement 3)

No architecture change. `raft_send_append_entries`/`ha_cb_send_append_entries_stub` remains in the codebase, unused in production, and is now explicitly documented as such (see `research.md` Design Decisions for the full rationale). `repl_broadcast` + `ha_stream_wal_to_standby` remain the sole production `AppendEntries`-equivalent send/retry/backoff mechanism. Per Requirement 3.3 (amended after design review to permit this third resolution alongside retire/wire-in), this documentation-only resolution must additionally be backed by a non-regression guard: an `XCE_TESTING`-only call-counter on `ha_cb_send_append_entries_stub` (mirroring the `ha_wal_ack_send_hook` idiom) exercised by a test that drives the production heartbeat/replication path end-to-end and asserts the counter stays at zero while confirming `repl_broadcast`/`ha_stream_wal_to_standby` did fire — so a future change that accidentally starts calling this path is caught, not silently accepted. This adds one small `ha.c`/test diff beyond the pure documentation change described above (see File Structure Plan and Testing Strategy).

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Wire serialization | Cap'n Proto (existing `c-capnproto`) | `ReplConfigPushReply` gains an additive `term @2: UInt64` field | Regenerated automatically by the existing `capnp compile` Meson `custom_target` (`src/lib/meson.build:1-10`); no build/Nix change needed (see `research.md`) |
| Controller adapter | C (gnu17), existing `src/controller/` | All behavior changes | No new dependency |

No other stack layer is touched.

## File Structure Plan

This is a small, surgical extension of an existing subsystem — files are listed individually rather than by directory pattern.

### Modified Files
- `src/schema/controller.capnp` — add `term @2: UInt64 $C.mapname("term")` to `struct ReplConfigPushReply` (Requirement 7.4).
- `src/controller/config_repl.c` — `handle_repl_config_push`: construct and send `ReplConfigPushReply{accepted=0, term}` on each of its four reject early-returns (malformed request, missing bytes/sha256, sha256 mismatch, `conf_save_config` failure); populate `term` on the existing success reply too; preserve the existing `-EAGAIN`-from-`config_activate_committed` behavior unchanged (still sends `accepted=1` — activation-deferred is not a rejection). Add an `XCE_TESTING`-only send-hook (e.g. `ha_config_push_reply_send_hook`), mirroring `ha_wal_ack_send_hook` (`ha.c:3085-3086`), so tests can observe the reply payload without a live endpoint. (Requirements 2.2, 2.3, 2.5, 7.4, 7.5)
- `src/controller/cluster.c` — the existing primary-side `ReplConfigPushReply` handler (`myreq_handler`, `cluster.c:745-762`) already branches on `cack->accepted` (ACKs via `repl_on_config_ack` on true, WARN-and-ignores on false — the false branch has simply never fired in production before this feature's fix). Add a term check alongside the existing `accepted` check: when `cack->term` exceeds the primary's current term, call `raft_on_higher_term_observed`, the same single adoption entry point the heartbeat-reply and WAL-NACK vectors already use. (Requirement 7.4, 7.5)
- `src/controller/ha.c`:
  - Split `vote_ep_err_handler` into `vote_ep_err_handler` (real-vote endpoints, registered at the existing `ha.c:688-689` call site) and `pre_vote_ep_err_handler` (PreVote endpoints, registered at the existing `ha.c:827-828` call site), each calling `vote_reply_dispatch` or `pre_vote_reply_dispatch` respectively for the election-phase branch instead of the legacy `handle_request_vote_reply`. Extract the identical `CTRL_PRIMARY` cleanup branch (standby-endpoint removal, `repl_remove_standby`, quorum-transition logging) into a shared static helper called by both. (Requirements 4.2, 4.3, 4.4)
  - Extend the `on_send_append_entries` doc-comment at `ha.c:2520-2532` to state plainly that this callback is reachable only via `raft_send_append_entries`, which has no production caller, and point to the CLAUDE.md conformance record for the authoritative production mechanism. Add an `XCE_TESTING`-only call counter to `ha_cb_send_append_entries_stub` (e.g. `ha_send_append_entries_stub_call_count`) as the non-regression guard the amended Requirement 3.3 requires. (Requirement 3.3)
- `src/controller/CLAUDE.md` — extend the "TLA+ Action-to-Code Mapping" table with corrected/added rows for `AppendEntries` reply integrity (WAL catch-up and config-push), vote/PreVote failure routing, and the dormant-sender resolution; extend the "Protocol Verification Checklist" with a checklist item for "every reject path sends an explicit reply." (Requirement 1.1-1.6)
- `.kiro/specs/ha-term-safety/spec.json` — append a one-line pointer on the existing `known_limitations` entry (index 30, the dormant `on_send_append_entries` note) referencing this spec's resolution, so a future reader lands on the authoritative decision. (Requirement 1.2, 1.6)

### New Test Additions (existing files extended, no new test files needed)
- `src/controller/testsuite/test_config_repl.c` — new tests: each of the four config-push reject paths sends `accepted=0` with the current term; the `-EAGAIN` activation-defer path still sends `accepted=1`; a higher term in an inbound config-push reply triggers adoption. (Requirements 2.2, 2.3, 2.5, 7.4, 7.5)
- `src/controller/testsuite/test_ha_vote_send.c` — new tests: a post-connect endpoint failure on a real-vote endpoint reaches `vote_reply_dispatch`'s tally; the same on a PreVote endpoint reaches `pre_vote_reply_dispatch`'s tally; reuses/extends `vote_ep_ctx_build_for_test` (`ha.c:7337`) as the test-construction seam. (Requirements 4.2, 4.3, 4.5)
- `src/controller/testsuite/test_ha_core_safety.c` — new test asserting `ha_send_wal_ack_reply_ex(success=0)` is actually invoked (via `ha_wal_ack_send_hook`) on a WAL catch-up reject, closing the reply-construction-boundary coverage gap found during design discovery. (Requirement 2.1, regression-lock)
- `src/controller/testsuite/test_commit_barrier_reply.c` or `test_commit_barrier_rollout.c` — new test: one failed payload in a submitted batch propagates failure to every payload in that batch. (Requirement 5.3)

### Unmodified, Regression-Locked Files

These files back the four already-conformant components (`ClientLogAppendGuarantee`, `CommitSourceGuarantee`, `TermDiscoveryCompleteness`, `RestartRecoveryGuarantee`). No source changes; cited here so each component in the table above resolves to a concrete file, and so a future diff touching them is recognized as revalidation-worthy rather than incidental.
- `src/controller/persist.c` — `ha_primary_log_append`, `persist_submit_job_payloads_async` (Requirement 5); confirmed conformant, existing tests in `test_commit_barrier_rollout.c`/`test_commit_barrier_reply.c`/`test_term_safety_adapter.c` cover it (plus the one new batch-propagation test above).
- `src/controller/commit.c` — `majority_candidate_locked`, `commit_set_raft_commit_index` (Requirement 6); confirmed conformant, `test_term_safety_adapter.c:646-780` already pins both directions of the advisory-vs-authoritative split, including the `XCE_TESTING` assert at `commit.c:222`.
- `src/controller/ha.c` — heartbeat-reply and WAL-NACK term-adoption call sites (Requirement 7.1, 7.2), `ha_sync_applied_lsn_from_recovery`/`ha_sync_raft_log_from_recovery`/phase2 retry helper/JIT log-length resync (Requirement 8); confirmed conformant, existing coverage in `test_persist_recovery.c`/`test_config_repl.c`.

No files are created; no files outside `src/controller/` (and its one schema dependency) are touched. `src/lib/`, `src/frontend/`, `src/worker/`, `src/scheduler/` are untouched.

## System Flows

### Configuration-push reject reply (Requirement 2, 7)

```mermaid
sequenceDiagram
    participant Primary
    participant Standby

    Primary->>Standby: ReplConfigPushRequest bytes sha256 versionTag
    alt Malformed or missing fields
        Standby-->>Primary: ReplConfigPushReply accepted false term
    else SHA-256 mismatch
        Standby-->>Primary: ReplConfigPushReply accepted false term
    else conf_save_config fails
        Standby-->>Primary: ReplConfigPushReply accepted false term
    else Save succeeds, activation deferred EAGAIN
        Standby-->>Primary: ReplConfigPushReply accepted true term
        Note over Standby: Activation retried later on commit advance
    else Save and activation succeed
        Standby-->>Primary: ReplConfigPushReply accepted true term
    end
    Primary->>Primary: distinguish reject from timeout, adopt term if higher
```

Key decision not obvious from the diagram: `accepted` means "bytes saved out-of-band," not "activated" — the activation-deferred branch and the fully-succeeded branch both reply `accepted=true` by design, since activation is gated separately on the committed marker (`ha-term-safety`) and is expected to retry.

### Vote/PreVote endpoint failure routing (Requirement 4)

```mermaid
sequenceDiagram
    participant Candidate
    participant Peer

    Candidate->>Peer: connect and monitor
    alt Connect or monitor fails
        Candidate->>Candidate: vote_reply_dispatch or pre_vote_reply_dispatch denied
    else Connect succeeds, later RDHUP or SSL error before reply
        Candidate->>Candidate: vote_ep_err_handler or pre_vote_ep_err_handler
        Candidate->>Candidate: vote_reply_dispatch or pre_vote_reply_dispatch denied
    else Peer replies before any failure
        Peer-->>Candidate: RequestVoteReply or PreVoteReply
        Candidate->>Candidate: vote_reply_dispatch or pre_vote_reply_dispatch
    end
    Candidate->>Candidate: fast quorum decision once every peer outcome is counted
```

Key decision not obvious from the diagram: today only the "connect or monitor fails" and "peer replies" branches reach the production tally; this feature adds the "later RDHUP" branch to the same tally instead of the legacy no-op-counting path.

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|---|---|---|---|---|
| 1.1 | Action-by-action mapping produced | ConformanceRecord | CLAUDE.md table | — |
| 1.2 | Live-vs-dormant path recorded on drift | ConformanceRecord | CLAUDE.md table, ha-term-safety spec.json pointer | Dormant sender decision |
| 1.3 | Gap detail sufficient to re-verify | ConformanceRecord | design.md + research.md | — |
| 1.4 | Fix vs documentation-drift classification | ConformanceRecord | design.md Decisions | — |
| 1.5 | Code-fix findings delivered under Req 2-8 | ConformanceRecord | (cross-reference) | — |
| 1.6 | Documentation-drift findings recorded for reconciliation | ConformanceRecord | ha-term-safety spec.json pointer | Dormant sender decision |
| 2.1 | WAL catch-up reject reply (regression-lock) | WalCatchupReplyIntegrity (existing) | `ha_send_wal_ack_reply_ex`, `ha_wal_ack_send_hook` | — |
| 2.2 | Config-push reject reply (new) | ConfigPushReplyIntegrity | `ReplConfigPushReply{accepted=0}` | Config-push reject flow |
| 2.3 | Reject distinguishable from transport loss | ConfigPushReplyIntegrity | `ReplConfigPushReply` vs no reply/timeout | Config-push reject flow |
| 2.4 | Unknown outcome not inferred as success | ConfigPushReplyIntegrity, WalCatchupReplyIntegrity | timeout handling (existing) | — |
| 2.5 | No regression to success-reply behavior | ConfigPushReplyIntegrity, WalCatchupReplyIntegrity | existing `accepted=1` / `success=1` paths | Config-push reject flow |
| 3.1 | Retry on rejection | Existing (`ha_raft_on_repl_nack`, `repl_start_catchup`) | — | — |
| 3.2 | Multi-round convergence | Existing (catch-up subsystem) | — | — |
| 3.3 | Single authoritative send mechanism | ConformanceRecord (documentation decision) | CLAUDE.md table, `ha.c:2520` doc-comment | Dormant sender decision |
| 3.4 | Transport-level send failure not treated as caught-up | Existing (`repl_broadcast`, heartbeat re-check) | — | — |
| 4.1 | Connect-time failure counted (regression-lock) | VoteEndpointFailureRouter (existing behavior) | `vote_reply_dispatch`, `pre_vote_reply_dispatch` | Vote failure routing |
| 4.2 | Post-connect failure counted (new) | VoteEndpointFailureRouter | `vote_ep_err_handler`, `pre_vote_ep_err_handler` | Vote failure routing |
| 4.3 | Same behavior for PreVote | VoteEndpointFailureRouter | `pre_vote_ep_err_handler` | Vote failure routing |
| 4.4 | No regression to connect/monitor routing | VoteEndpointFailureRouter | shared `CTRL_PRIMARY` cleanup helper | Vote failure routing |
| 4.5 | Fast quorum decision reachable | VoteEndpointFailureRouter | tally functions (existing) | Vote failure routing |
| 5.1 | Append persists or fails explicitly | ClientLogAppendGuarantee (existing) | `raft_log_append`, `ha_primary_log_append` | — |
| 5.2 | No log-length advance on failed persist | ClientLogAppendGuarantee | `raft_log_append` | — |
| 5.3 | Batch failure propagates to whole batch | ClientLogAppendGuarantee | `persist_submit_job_payloads_async` | — |
| 5.4 | No regression to failure propagation | ClientLogAppendGuarantee | existing tests | — |
| 6.1 | Durable-majority sole authority | CommitSourceGuarantee (existing) | `majority_candidate_locked` | — |
| 6.2 | Advisory index informational only | CommitSourceGuarantee | `commit_set_raft_commit_index` | — |
| 6.3 | No hard-ceiling reintroduction | CommitSourceGuarantee | `commit.c:139-155` invariant | — |
| 6.4 | Advance proceeds despite lagging advisory index | CommitSourceGuarantee | existing `XCE_TESTING` assert | — |
| 7.1 | Heartbeat-reply term adoption (regression-lock) | TermDiscoveryCompleteness (existing) | `raft_on_higher_term_observed` | — |
| 7.2 | WAL-NACK term adoption (regression-lock) | TermDiscoveryCompleteness (existing) | `raft_on_higher_term_observed` | — |
| 7.3 | Vote/PreVote reply term adoption (regression-lock) | TermDiscoveryCompleteness (existing) | `raft_on_vote_reply` Step 1 | — |
| 7.4 | Config-push reply carries term (new) | ConfigPushReplyIntegrity | `ReplConfigPushReply.term` | Config-push reject flow |
| 7.5 | No regression to existing term-adoption vectors | TermDiscoveryCompleteness, ConfigPushReplyIntegrity | — | Config-push reject flow |
| 8.1 | Recovery raises applied LSN/log length/term | RestartRecoveryGuarantee (existing) | `ha_sync_applied_lsn_from_recovery`, `ha_sync_raft_log_from_recovery` | — |
| 8.2 | Failed recovery step retries, no false-latch | RestartRecoveryGuarantee | phase2 retry helper (`ha.c:415-501`) | — |
| 8.3 | JIT log-length resync beyond boot | RestartRecoveryGuarantee | `ha_apply_term_entry_via_library` resync | — |
| 8.4 | No regression to clean-failover convergence | RestartRecoveryGuarantee | existing tests | — |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|---|---|---|---|---|---|
| ConfigPushReplyIntegrity | Controller/Replication | Send an explicit accept/reject reply, carrying term, for every configuration push | 2.2, 2.3, 2.5, 7.4, 7.5 | `ReplConfigPushReply` schema (P0) | Event |
| WalCatchupReplyIntegrity | Controller/Replication | Existing bug-1479 fix; regression-locked, not modified | 2.1, 2.4, 2.5 | `WalAckReply` schema (P0, existing) | Event |
| VoteEndpointFailureRouter | Controller/Election | Route every vote/PreVote endpoint failure, at any timing, to the production tally | 4.1-4.5 | `vote_reply_dispatch` (P0), `pre_vote_reply_dispatch` (P0), `vote_ep_ctx_build_for_test` (P1, test-only) | Service |
| ConformanceRecord | Documentation | Durable action-by-action mapping and dormant-path resolution | 1.1-1.6, 3.3 | `CLAUDE.md`, `ha-term-safety/spec.json` | State |
| ClientLogAppendGuarantee | Controller/Persistence | Existing conformant append-or-explicit-failure behavior; regression-locked | 5.1-5.4 | `persist.c` (existing) | State |
| CommitSourceGuarantee | Controller/Commit | Existing conformant durable-majority-authoritative behavior; regression-locked | 6.1-6.4 | `commit.c` (existing) | State |
| TermDiscoveryCompleteness | Controller/Election | Existing conformant term-adoption vectors (heartbeat, WAL-NACK, vote reply); regression-locked | 7.1-7.3, 7.5 | `raft_on_higher_term_observed` (existing) | Event |
| RestartRecoveryGuarantee | Controller/Recovery | Existing conformant restart/recovery reconcile; regression-locked | 8.1-8.4 | `ha.c` recovery functions (existing) | State |

Only the two components introducing new code (`ConfigPushReplyIntegrity`, `VoteEndpointFailureRouter`) warrant full detail blocks below; the regression-lock components need no new interface — their existing interfaces are cited in the table above and exercised by the tests in File Structure Plan.

### Controller / Replication

#### ConfigPushReplyIntegrity

| Field | Detail |
|-------|--------|
| Intent | Every configuration-push outcome (accept or reject) produces an explicit `ReplConfigPushReply` carrying the replying member's current term |
| Requirements | 2.2, 2.3, 2.5, 7.4, 7.5 |

**Responsibilities & Constraints**
- Owns the reply-construction boundary inside `handle_repl_config_push`; does not own configuration activation (`config_activate_committed` remains the sole activation gate, unchanged).
- Must not conflate an activation-deferred outcome (`-EAGAIN`) with a rejection — that path continues to reply `accepted=1`.
- Must include the current term on every reply, success or failure.

**Dependencies**
- Outbound: `ReplConfigPushReply` schema field (P0) — the additive `term` field this component populates.
- Outbound: `raft_on_higher_term_observed` (P0) — invoked by the primary's inbound-reply handler when this component's reply carries a higher term (existing entry point, no change to it).
- External: none.

**Contracts**: Event [x]

##### Event Contract
- Published: `ReplConfigPushReply {accepted: bool, versionTag: uint64, term: uint64}` — one per `ReplConfigPushRequest` received, always sent (never silently dropped).
- Subscribed: none (this component only replies).
- Ordering/delivery: request/reply pair over the existing outbound control channel (`ha-async-io.md` connection ownership); no new ordering guarantee introduced.

**Implementation Notes**
- Integration: reuses the existing `xce_reply`/`init_reply_by_req` pattern already present in `handle_repl_config_push`'s success path; the four reject branches gain the same call shape.
- Validation: `XCE_TESTING`-only send-hook (mirroring `ha_wal_ack_send_hook`) lets unit tests assert the reply payload without a live endpoint.
- Risks: none material — narrow, single-function change with a directly analogous shipped precedent (bug-1479).

### Controller / Election

#### VoteEndpointFailureRouter

| Field | Detail |
|-------|--------|
| Intent | Every vote/PreVote endpoint failure, regardless of when in the round it occurs, is counted through the same production tally as a delivered denial |
| Requirements | 4.1-4.5 |

**Responsibilities & Constraints**
- Owns endpoint-failure handling for outgoing vote and PreVote connections (`on_err` callbacks); does not own the tally logic itself (`vote_reply_dispatch`/`pre_vote_reply_dispatch` are consumed, not modified).
- Must preserve the existing `CTRL_PRIMARY` cleanup behavior (standby-endpoint deregistration, quorum-transition logging) identically for both endpoint kinds.
- Must not introduce a new discriminant field on `vote_ep_ctx_t`; the split-handler registration already distinguishes endpoint kind by call site.

**Dependencies**
- Outbound: `vote_reply_dispatch` (P0) — real-vote tally, already correct for connect/monitor failures.
- Outbound: `pre_vote_reply_dispatch` (P0) — PreVote tally, already correct for connect/monitor failures.
- Outbound: `vote_ep_ctx_build_for_test` (P1, test-only) — reused/extended as the unit-test construction seam.
- Inbound: iomgr `on_err` callback invocation (existing, unchanged trigger).

**Contracts**: Service [x]

##### Service Interface
```c
/* vote_ep_err_handler: on_err callback for outgoing real-vote endpoints.
 * Election-phase branch now counts a post-connect failure through
 * vote_reply_dispatch instead of handle_request_vote_reply. */
static int vote_ep_err_handler(void *ep_raw);

/* pre_vote_ep_err_handler: on_err callback for outgoing PreVote endpoints.
 * Election-phase branch counts a post-connect failure through
 * pre_vote_reply_dispatch. */
static int pre_vote_ep_err_handler(void *ep_raw);

/* shared helper: identical CTRL_PRIMARY cleanup extracted from both. */
static void vote_ep_handle_promoted_disconnect(ha_state_t *ha, endpoint_t *ep,
                                                vote_ep_ctx_t *vctx);
```
- Preconditions: `ep->data` is a populated `vote_ep_ctx_t` for the endpoint kind the handler is registered against.
- Postconditions: exactly one outcome (denied/counted) reaches the applicable tally function per invocation; `CTRL_PRIMARY`-role invocations perform standby cleanup identically to today.
- Invariants: a candidate's own term is never mistaken for an observed peer term through this path (the synthesized outcome carries the candidate's own term, `granted=0` — unchanged from today's behavior).

**Implementation Notes**
- Integration: registration sites (`ha.c:688-689`, `827-828`) each point at the correspondingly-named handler; no change to endpoint setup beyond the callback pointer.
- Validation: new unit tests per File Structure Plan, using the reused test-construction helper to simulate a post-connect failure without a live peer.
- Risks: touches live election-path code — mitigated by keeping the extracted `CTRL_PRIMARY` branch byte-for-byte identical and requiring the new tests before this task is considered done (see `research.md` Risks & Mitigations).

## Data Models

### Data Contracts & Integration

**`ReplConfigPushReply` (Cap'n Proto, `src/schema/controller.capnp`)**

| Field | Ordinal | Type | Change |
|---|---|---|---|
| `accepted` | 0 | Bool | unchanged |
| `versionTag` | 1 | UInt64 | unchanged |
| `term` | 2 | UInt64 | **new, additive** |

Backward/forward compatibility: Cap'n Proto additive fields are safe under mixed-version rolling upgrade — an old-code standby's reply (no `term` field) decodes as `term=0` on a new-code primary. This is safe by construction: `0` can never present as a term higher than the primary's own current term (terms start at 1 and only increase), so no spurious adoption occurs; the primary simply doesn't get early term-discovery from an old-code peer's config-push replies, which is exactly today's (pre-feature) behavior for that peer. No explicit migration phasing is required beyond the existing rolling-upgrade tolerance the cluster already relies on.

`WalAckReply` is unchanged (already carries `term`, `lsn`, `success`); listed here only to confirm no modification is needed.

## Error Handling

### Error Strategy

This feature's domain is internal cluster wire-protocol reply integrity, not user-facing request handling — the applicable categories are protocol-rejection, transport-failure, and validation-failure, each with a distinct, already-established response pattern that this feature extends consistently.

### Error Categories and Responses
- **Validation/protocol rejection** (malformed config-push request, SHA-256 mismatch, `conf_save_config` failure): explicit reject reply (`accepted=0`) carrying term, so the primary can distinguish "actively rejected" from "lost" and react (retry, log, or escalate) — this is the category this feature's fixes target.
- **Transport failure** (connection drop, send failure, RDHUP): counted as a failure outcome through the same production tally/reply path used for a delivered rejection wherever the protocol model requires an outcome (vote/PreVote tally); where no reply is architecturally expected (snapshot chunk transfer, by explicit design — see requirements.md Boundary Context), recovery remains via the next heartbeat cycle, unchanged by this feature.
- **Activation-deferred, not an error** (`config_activate_committed` returning `-EAGAIN`): explicitly not classified as a rejection; the existing loud-fence-and-retry pattern (steering `ha-async-io.md`, "controller never self-terminates") already governs it and is preserved unchanged.

### Monitoring
No new metrics/logging channels introduced; existing `WARN`/`INFO` logging in `handle_repl_config_push` and `vote_ep_err_handler` is preserved and extended with the same log level conventions already used in each function.

## Testing Strategy

- **Unit Tests**:
  1. `handle_repl_config_push` sends `ReplConfigPushReply{accepted=0, term}` for each of the four reject conditions (malformed request, missing bytes/sha256, sha256 mismatch, save failure) — via the new `XCE_TESTING` send-hook. (2.2, 2.3, 2.5)
  2. `handle_repl_config_push` still sends `accepted=1` when `config_activate_committed` returns `-EAGAIN` (activation-deferred is not a rejection). (2.5, 7.5)
  3. `ha_send_wal_ack_reply_ex(success=0)` is actually invoked on a WAL catch-up reject (closing the reply-construction-boundary coverage gap identified during design discovery). (2.1)
  4. `vote_ep_err_handler` routes a synthetic post-connect failure through `vote_reply_dispatch`'s tally, using the reused test-construction helper. (4.2, 4.4)
  5. `pre_vote_ep_err_handler` routes the same failure shape through `pre_vote_reply_dispatch`'s tally. (4.3)
  6. A submitted batch with one failing payload propagates that failure to every payload in the batch. (5.3)
  7. Driving the production heartbeat/replication path end-to-end never increments `ha_send_append_entries_stub_call_count`, while `repl_broadcast`/`ha_stream_wal_to_standby` do fire for the same scenario — the Requirement 3.3 non-regression guard. (3.3)
- **Integration Tests**:
  1. A three-node cluster with one standby configured to reject a pushed configuration (bad hash) observes the primary correctly distinguishing the rejection from a timeout and not treating the push as silently lost.
  2. A three-node election round where one peer's vote endpoint fails after connect (simulated) still reaches a quorum decision without waiting for the full election timeout.
  3. Existing `test_term_safety_adapter.c` advisory-vs-authoritative commit tests re-run unmodified as the Requirement 6 regression lock (no new test needed, confirmed sufficient during design discovery).
  4. Existing `test_persist_recovery.c`/`test_config_repl.c` recovery tests re-run unmodified as the Requirement 8 regression lock.
- **Live/Soak Tests** (per project convention for HA changes, `xce-test-ha` skill):
  1. A rolling chaotic-failover soak where the test driver issues a configuration push whose payload deliberately does not match the sha256 it advertises (no special fault-injection hook needed — this exercises `handle_repl_config_push`'s real SHA-256-mismatch reject branch), confirming the primary observes the explicit reject rather than degrading to the timeout path.
  2. A soak round exercising the post-connect vote/PreVote endpoint failure path end-to-end. **Open question for the tasks phase**: unlike item 1, this needs a way to sever an already-established outbound vote connection before a reply arrives (e.g. a firewall-rule toggle or proxy in the soak harness) — no existing `xce-test-ha` mechanism for this was confirmed during design. If no such mechanism exists cheaply, this item may be deferred; the unit tests (Requirement 4.2/4.3, via `vote_ep_ctx_build_for_test`) already give direct, deterministic coverage of the fix itself.

## Supporting References
- `research.md` — full gap analysis (asset map, options, effort/risk), design-phase discovery log, and the two recorded design decisions (dormant sender resolution, vote-handler split) with full rationale.
- `src/controller/CLAUDE.md` — "TLA+ Action-to-Code Mapping" and "Protocol Verification Checklist," extended by this feature as the durable conformance record (Requirement 1).