myxce/.kiro/specs/raft-adapter-conformance/requirements.md

Requirements Document

Introduction

This feature performs a systematic, action-by-action conformance audit of the XCE controller's Raft replication adapter — the wire-level send/receive/reply code in src/controller/ha.c, repl.c, and config_repl.c — against the actions of the canonical Raft protocol model (raft/raft.tla: AppendEntries request/response, RequestVote/PreVote request/response, ClientRequest/log append, AdvanceCommitIndex, term/UpdateTerm handling, and restart/recovery reconcile). It is distinct from and complements raft-sim-conformance, whose boundary is the in-process, in-memory library harness (src/lib/raft/raft_sim.c) validating the library's own protocol semantics; this feature validates that the controller adapter faithfully carries the library's decisions onto the wire without silently dropping a response, retry, or backoff step the protocol model assumes always happens.

Motivating finding. A fresh live-soak re-verification of bug-1479's fix surfaced that the controller's only WalAckReply constructor hardcoded success, so no production code path ever sent a failing reply — a standby's WAL catch-up rejection was silently dropped rather than reaching the primary as a NACK, reproducing bug-1479's permanent-quorum-loss symptom by a different mechanism than the original defect. That gap, plus a second, independently-discovered adapter gap it exposed (a standby's catch-up streaming range was capped at the leader's committed point rather than its real log tail, creating a self-referential deadlock), are already fixed and live-verified as of this spec's seeding (see bug-1479 in .wolf/buglog.json and config-upgrade-snapshot task 8.6). This feature's job is the broader pass that finding was a symptom of: walk every Raft protocol action against the adapter code that is supposed to realize it, and treat each specific type of gap consistently rather than fixing instances one at a time as they surface via live incidents.

Audit results seeded into this requirements pass. A fresh action-by-action pass performed for this spec, cross-referenced against prior raft-log-replication/raft-commit-safety/ha-term-safety/raft-sim-conformance validation notes, confirmed:

• Two areas already conformant and worth protecting with regression coverage: ClientRequest/log append integrity, and the durable-majority-authoritative / advisory-raft-index split for AdvanceCommitIndex.
• Two live, unfixed gaps of the same class as the motivating finding: a standby's configuration-push rejection (malformed message, missing fields, checksum mismatch, or save failure) never sends an explicit reject reply, so the primary cannot distinguish an actively-rejected push from a lost packet; and a vote/PreVote endpoint failure arriving after connect (not at connect time) is counted through a legacy path instead of the production vote-tally path, reopening — for that one failure window — the same silent-drop defect class bug-1423 already fixed for connect-time failures.
• One architecture question requiring a design decision rather than a behavioral fix: a fully protocol-conformant, per-peer index-driven AppendEntries sender exists in the codebase but has zero production callers; the mechanism actually driving replication in production today is a different, coarser substitute (broadcast-plus-catch-up-marking) that converges correctly post-fix but is not the mechanism the design documentation and the protocol model depict.

Per this feature's scope decision, the confirmed live gaps found during this pass are fixed within this feature rather than deferred to separate specs, following the same precedent as the motivating finding.

Boundary Context

• In scope:
  • A documented, action-by-action mapping between each raft.tla protocol action and the controller adapter code path(s) that realize it, with an explicit conformance verdict per action and citations precise enough to be independently re-verified.
  • Reconciling documentation-vs-code drift the audit finds at this boundary (a design document describing a code path that is not the one actually exercised in production) by recording which path is live and which is dormant or superseded.
  • Fixing every live gap the audit confirms, where the adapter fails to produce a response, retry, or backoff step that the protocol model or an already-shipped sibling fix assumes exists — including the configuration-push rejection reply and the post-connect vote/PreVote failure routing named above, and any other such gap the systematic pass confirms.
  • Regression coverage that locks in both the gaps already fixed before this spec (WAL catch-up rejection replies, catch-up streaming ceiling, vote/PreVote connect-time failure routing, restart/recovery log resync) and the gaps this spec fixes, so a future change cannot silently reopen either.
  • A single, unambiguous decision (recorded as a requirement) about which AppendEntries send/retry/backoff mechanism is production-authoritative — the existing per-peer index-driven sender, or the broadcast-plus-catch-up substitute — so the codebase does not carry two divergent implementations of the same guarantee.
• Out of scope:
  • Any change to the Raft protocol logic itself (election, logOk freshness, (term,index) truncation, current-term commit, UpdateTerm) inside src/lib/raft/raft.c — that logic is validated by raft-log-replication, raft-commit-safety, and raft-sim-conformance; this feature validates only that the controller adapter carries the library's decisions onto the wire.
  • Configuration-upgrade materialized-state consistency (full entity-bearing state snapshot at a configuration change boundary) — remains the independent config-upgrade-snapshot spec's scope.
  • Changes to the controller voting set (adding/removing a controller peer) — remains ha-config-bootstrap's scope; this feature does not introduce quorum/membership transitions.
  • Client-side primary-discovery behavior (bug-854) and catch-up payload handling across multiple WAL segments (bug-847) — tracked separately, not part of the adapter's reply-integrity contract this feature audits.
  • Any change to when or how the primary decides to advance committed_lsn beyond preserving the existing durable-majority-authoritative behavior — that decision boundary belongs to ha-commit-engine/ha-term-safety.
• Adjacent expectations:
  • Relies on the raft library's callback contract (on_persist_entry, on_get_entry_term, on_send_append_entries, on_truncate_from, on_commit_advanced) and entry points (raft_log_append, raft_on_append_entries_request/reply, raft_advance_commit, vote/pre-vote handling) exactly as delivered by the Raft TLA+ Conformance series; this feature does not change that contract, only how faithfully the adapter drives it.
  • Relies on the out-of-band configuration-push transport (ha-term-safety) continuing to carry configuration bytes hash-verified against the committed (term, lsn) marker; this feature adds reply integrity to that transport's rejection path without changing what it carries.
  • Any gap this feature's audit finds that is genuinely new protocol-model territory (not covered by an existing sibling spec's boundary) is fixed here; any gap that falls within an existing sibling spec's stated scope is cross-referenced to that spec rather than duplicated.

Requirements

Requirement 1: Systematic Raft adapter conformance audit

Objective: As a maintainer, I want a documented, action-by-action mapping between the Raft protocol model's actions and the controller's adapter code, so that gaps between the model's assumptions and production behavior are found deliberately and once, instead of being rediscovered independently through separate live incidents.

Acceptance Criteria

1. The conformance audit shall produce a written mapping that lists, for each of AppendEntries request, AppendEntries response, RequestVote/PreVote request, RequestVote/PreVote response, ClientRequest/log append, AdvanceCommitIndex, term/UpdateTerm handling, and restart/recovery reconcile, the controller adapter code path(s) that realize it and a conformance verdict.
2. When the audit finds that a sibling spec's design documentation describes an adapter code path that differs from the code path actually exercised in production, the audit shall record which path is live and which is dormant or superseded, rather than leaving the discrepancy implicit.
3. When the audit finds a gap where the protocol model assumes a response, retry, or backoff step that the adapter does not perform, the audit shall record the affected code path, the triggering condition, and the observed effect in enough detail to be independently re-verified without re-deriving the finding from scratch.
4. The audit shall classify each finding as either requiring a code fix (a live path that violates the model's guarantee) or documentation-only drift (the guarantee holds in production, but design documentation describes a different mechanism than what runs).
5. Where a finding is classified as requiring a code fix, the fix shall be delivered under this feature's Requirements 2 through 8, not deferred to a separate spec.
6. Where a finding is classified as documentation-only drift, the audit shall record the correction needed so the affected design documentation can be reconciled without re-investigating the code.

Requirement 2: AppendEntries-equivalent rejection reply integrity

Objective: As an operator of an HA controller cluster, I want every replication message a standby rejects to produce an explicit reply back to the primary, so that the primary always learns of a rejection and can retry or re-route rather than the cluster stalling on a silently dropped reject.

Acceptance Criteria

1. When a standby rejects a WAL catch-up entry for any reason (log mismatch, forward gap, or apply failure), the Controller shall send an explicit reply indicating rejection back to the primary.
2. When a standby rejects a configuration push for any reason (malformed message, missing required fields, checksum mismatch, or a local save failure), the Controller shall send an explicit reply indicating rejection back to the primary, mirroring the acceptance reply already sent on a successful push.
3. The rejection reply for a configuration push shall be distinguishable by the primary from a lost or undelivered push, so that the primary's push-and-wait logic never treats an active rejection and a transport loss as the same outcome.
4. While no explicit rejection or acceptance reply has arrived within the applicable transport wait bound, the Controller shall continue to treat the outcome as unknown rather than inferring success.
5. The Controller shall not regress the acceptance-reply behavior for a successful WAL catch-up entry or a successful configuration push.

Requirement 3: AppendEntries send retry and backoff guarantee

Objective: As an operator, I want exactly one production-authoritative mechanism driving AppendEntries-equivalent replication retries and per-peer backoff, so that a rejected or lost replication attempt is corrected without relying on two divergent, only-one-of-which-is-live implementations of the same guarantee.

Acceptance Criteria

1. When the primary receives an explicit rejection reply for a replicated entry, the Controller shall retry replication to that standby from an earlier point rather than repeating the same rejected attempt.
2. While a standby remains behind after a rejection, the Controller shall continue advancing that standby's catch-up point across successive rounds until it converges, rather than requiring convergence in a single round.
3. The Controller shall have exactly one send/retry/backoff mechanism that is authoritative in production for AppendEntries-equivalent replication; where more than one fully-implemented mechanism exists in the codebase, the non-authoritative one shall be resolved by one of: retiring it, making it the production path, or formally documenting it as an intentionally-dormant reference path guarded against silent regression (e.g. an explicit non-production marker plus a test asserting it stays uncalled in production) — so the codebase never carries two divergent, unreconciled implementations of the same guarantee.
4. If a replication send fails at the transport level (not a rejection), the Controller shall not treat the standby as caught up, and shall re-attempt replication to it within a bounded number of subsequent replication cycles.

Requirement 4: RequestVote/PreVote reply integrity across all failure modes

Objective: As an operator, I want every vote or pre-vote request outcome — granted, denied, or a failure that prevents a reply — to be counted through the same production tally, regardless of when in the round the failure occurs, so that an election or pre-vote round can reach its fast "all replied" decision instead of always falling back to the full round timeout.

Acceptance Criteria

1. When a vote request cannot be delivered because the connection could not be established, the Controller shall count that outcome through the same production vote-tally path used for a delivered denial.
2. When a vote request connection is established but then fails before a reply arrives (a post-connect failure), the Controller shall count that outcome through the same production vote-tally path used for a connect-time failure and for a delivered denial, rather than through a separate legacy path that does not update the tally.
3. The Controller shall apply the same failure-counting behavior to a pre-vote request as to a real vote request.
4. The Controller shall not regress the existing connect-time and monitor-time failure routing already verified for vote and pre-vote requests.
5. When every peer in an election or pre-vote round has replied or had its outcome counted, the Controller shall be able to reach a quorum decision without waiting for the round's full timeout.

Requirement 5: ClientRequest / log append integrity

Objective: As a maintainer, I want a client write to always resolve to either an acknowledged append or an explicit, propagated failure, so that a write is never silently lost between submission and its outcome.

Acceptance Criteria

1. When the primary appends a client-submitted entry, the Controller shall either persist the entry and make it eligible for replication, or return an explicit failure to the submitter.
2. The Controller shall not advance the log length for an entry whose persistence failed.
3. When a batch of submitted entries includes one that fails to persist, the Controller shall propagate that failure to every entry in the same batch rather than silently completing a subset.
4. The Controller shall not regress the existing behavior where a persistence failure is returned to the caller rather than swallowed.

Requirement 6: Commit advancement source-of-truth preserved

Objective: As an operator, I want the durable-majority barrier to remain the sole authority for advancing the committed point, so that the advisory Raft commit index can never gate, delay, or override a safe commit advance.

Acceptance Criteria

1. The Controller shall advance the committed point using the durable-majority barrier as the sole authoritative source.
2. The Controller shall treat the Raft library's advisory commit index as informational only; it shall neither block nor be required for the durable-majority barrier to advance the committed point.
3. The Controller shall not reintroduce a hard ceiling that requires both the durable-majority barrier and the advisory Raft commit index to agree before advancing the committed point.
4. When the advisory Raft commit index lags the durable-majority watermark, the Controller shall still advance the committed point according to the durable-majority barrier.

Requirement 7: Term discovery completeness

Objective: As an operator, I want every reply channel capable of carrying a peer's observed election term to trigger term adoption when that peer's term is higher, so that a stale-term node cannot remain stuck retrying against a peer that has already moved to a higher term.

Acceptance Criteria

1. When a heartbeat reply carries a term higher than the primary's current term, the Controller shall adopt the higher term and step down from any leadership held at the lower term.
2. When a WAL rejection reply carries a term higher than the sender's current term, the Controller shall adopt the higher term.
3. When a vote or pre-vote reply carries a term higher than the candidate's current term, the Controller shall adopt the higher term.
4. When a configuration-push reply is sent, the Controller shall include the replying member's current term, so a higher term can be discovered through that reply channel as it already is through heartbeat and WAL-rejection replies.
5. The Controller shall not regress the existing heartbeat-reply and WAL-rejection term-adoption behavior already verified for this cluster.

Requirement 8: Restart / recovery reconciliation conformance

Objective: As an operator, I want a restarted or rejoining controller to reconcile its applied state, log length, and term from its durable floor every time recovery runs, so that a boot-ordering or partial-failure hazard cannot leave a node permanently out of sync with what it has actually persisted.

Acceptance Criteria

1. When a controller completes startup recovery, the Controller shall raise its applied LSN, log length, and term from the recovered durable floor, never leaving them below what recovery found.
2. If a recovery step fails partway through cluster initialization, the Controller shall retry that step rather than permanently latching a completed state without having finished it.
3. While a term-safe append-entries receive occurs after startup, the Controller shall re-check and, if needed, re-raise its log length from the durable floor, not only at boot time.
4. The Controller shall not regress the existing byte-identical convergence behavior for a single clean failover and restart.