fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479)

Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:

1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
   library's nextIndex backoff and immediately repl_start_catchup's the
   standby at the corrected position, instead of the 16-strike silent
   eviction with no catch-up driver after it. Correct from the start, but
   live soak still failed identically: nothing ever produced the NACK this
   code was waiting for.

2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
   path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
   silently dropping the reject. ha_send_wal_ack_reply had hardcoded
   success=1 unconditionally -- the only WalAckReply sender in the
   controller -- making (1)'s NACK-routing correct but unreachable dead
   code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
   the NACK without a live endpoint. Live-confirmed the NACK now reaches
   the primary -- but the scenario still failed for a third reason.

3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
   catch-up range was hard-bounded at committed_lsn, which can never advance
   in this exact scenario because advancing it requires an ACK from the very
   standby that's stuck catching up (2-of-3 alive, this standby is the only
   path to a new majority) -- a self-referential deadlock. Widens the
   STREAM_WAL ceiling (and the convergence check gating it) to the leader's
   own log tail, matching raft.tla's real AppendEntries semantics
   (leaderCommit is advisory, never a send-cap), while reusing bug-1478's
   exact guard so this doesn't reopen that trap (never widen onto a trailing
   local-only checkpoint's own unstreamable LSN).

Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

src/controller/ha.h

@@ -596,6 +596,32 @@ int ha_primary_log_append(cluster_t *c, uint32_t type,
 int ha_raft_on_repl_ack(cluster_t *c, int abs_peer_idx, uint64_t acked_lsn,
                         uint64_t reply_term);
 
+/*
+ * ha_raft_on_repl_nack (bug-1479): drive the library on a REJECTED
+ * AppendEntries (WalAckReply.success == 0, i.e. the standby's log conflicts
+ * at the sent prevLogIndex/prevLogTerm). Routes the rejection through
+ * raft_on_append_entries_reply(success=0), which backs off
+ * next_index[abs_peer_idx] by one (raft.tla's AppendEntries failure branch,
+ * floored at 1) -- mirrors ha_raft_on_repl_ack's bounds-checked peer-index
+ * resolution exactly, but does NOT call raft_advance_commit (a rejection
+ * can never advance commit).
+ *
+ * Before this fix, a NACK was only ever used for stale-term discovery
+ * (raft_on_higher_term_observed) and otherwise silently dropped -- the
+ * library's own, already-correct nextIndex backoff mechanism was never
+ * invoked, so a standby stuck in a genuine log conflict (an ordinary
+ * post-election occurrence) never had its resend position corrected: the
+ * primary kept broadcasting new entries assuming agreement at an index the
+ * standby had already rejected, forever.
+ *
+ * Returns the corrected resume LSN (next_index[abs_peer_idx] - 1, i.e. the
+ * position the standby is now believed caught up through) via
+ * *corrected_lsn_out on success; returns -EINVAL (leaving *corrected_lsn_out
+ * untouched) on a no-cluster context or a rejected peer index, matching
+ * ha_raft_on_repl_ack's defense-in-depth bounds check. */
+int ha_raft_on_repl_nack(cluster_t *c, int abs_peer_idx, uint64_t reply_term,
+                         uint64_t *corrected_lsn_out);
+
 /*
  * ha_repl_ack_abs_peer_idx (ha-term-safety task 3.3 remediation): resolve the
  * ABSOLUTE library peer index from an ACK endpoint's stored peer_idx, branching
@@ -917,15 +943,36 @@ int ha_schedule_phase2_retry(cluster_t *c);
  * On success (rc == 0): applied_lsn advances to entry->lsn+1 and a
  *   WalAckReply {term, lsn, success=1} is sent back to the primary via
  *   xce_reply(ep, c->myhost, &reply) (Task 3.1, Req 3.1).
- * On error (rc < 0):   logs ERROR, returns error; no ACK sent (Task 3.2).
+ * On a genuine apply/persist ERROR (rc < 0): logs ERROR, returns error; no ACK
+ *   sent at all (Task 3.2) -- there is nothing coherent to report back yet.
+ * On a term-carrying entry's AppendEntries REJECT (logOk fails -- forward gap
+ *   or a genuine (term,lsn) conflict; rc == 0, entry not accepted): bug-1479 --
+ *   a WalAckReply {term, lsn, success=0} (a NACK) IS now sent, so the primary's
+ *   ha_raft_on_repl_nack can back off nextIndex and immediately mark this
+ *   standby for catch-up. Before this fix the reject was dropped with no reply
+ *   at all, so the primary's NACK-handling could never engage no matter how
+ *   correct it was -- the standby would loop rejecting the identical
+ *   re-broadcast entry forever.
  * applied_lsn is in-memory only — not persisted to disk.
- * ep and c may be NULL in unit-test builds (XCE_TESTING); ACK is skipped.
+ * ep and c may be NULL in unit-test builds (XCE_TESTING); ACK/NACK is skipped
+ * (or captured via ha_wal_ack_send_hook when set).
  * Requirements: 3.1, 3.2, 3.3, 3.4, 3.5
  */
 int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
                                 wal_catchup_entry_t *entry,
                                 struct endpoint_t *ep, cluster_t *c);
 
+#ifdef XCE_TESTING
+/*
+ * ha_wal_ack_send_hook: test-only seam for the outbound WalAckReply built by
+ * ha_send_wal_ack_reply_ex (bug-1479). When non-NULL, called with
+ * (term, lsn, success) INSTEAD OF building/sending the real reply over ep --
+ * lets a unit test assert that an AppendEntries reject now sends a NACK
+ * (success=0), without needing a live endpoint. NULL in production.
+ */
+extern void (*ha_wal_ack_send_hook)(uint64_t term, uint64_t lsn, int success);
+#endif
+
 /* ha_nop_reply_state_report — populate a heartbeat reply with stateReport.
  * Fills *sr from ha->applied_lsn (atomic read) and myconf->config_version_tag,
  * then sets reply_out->data_which = stateReportMsg and
@@ -1195,6 +1242,36 @@ void ha_process_standby_state(ha_state_t *ha, int peer_idx,
 uint64_t ha_compute_repl_target_lsn(uint64_t committed_lsn,
                                      uint64_t last_written);
 
+/*
+ * ha_compute_repl_stream_ceiling_lsn (bug-1479, gap 3): the upper bound of
+ * what a leader may actually STREAM to a standby via ha_stream_wal_to_standby
+ * — distinct from ha_compute_repl_target_lsn's CONVERGENCE target above.
+ *
+ * Bounding the stream range at committed_lsn is a permanent deadlock whenever
+ * the very standby being streamed to must ACK before committed_lsn can
+ * advance at all (e.g. a 3-node cluster with one member down and this
+ * standby log-conflicted after a chaotic election): it can never receive the
+ * entries it needs to ack to unblock the majority. raft.tla's real
+ * AppendEntries replicates the leader's FULL log — leaderCommit is advisory
+ * to the follower, never a cap on what gets sent. Streaming past commit is
+ * always safe: a later leadership change's own (term,lsn) conflict-truncate
+ * (ha_consume_append_entries) correctly unwinds anything that turns out not
+ * to have been committed.
+ *
+ * bug-1478 guard: never widen the ceiling to exactly the trailing local-only
+ * checkpoint's own consumed LSN slot — no standby's applied_lsn can ever
+ * legitimately reach that value (WAL_CHECKPOINT is never streamed or applied
+ * by any standby, by design), so doing so would reintroduce bug-1478's
+ * permanent-non-convergence trap. Falling back to committed_lsn for just
+ * that transient instant is safe — the very next real entry appended after
+ * the checkpoint widens the ceiling normally again.
+ *
+ * Pure, side-effect-free — safe to unit test directly.
+ */
+uint64_t ha_compute_repl_stream_ceiling_lsn(uint64_t committed_lsn,
+                                             uint64_t last_written,
+                                             uint64_t checkpoint_lsn);
+
 /* ── Task 3.5c: Standby snapshot receive handler ──────────────────────────── */
 
 /*
@@ -1864,8 +1941,15 @@ uint32_t ha_jittered_election_delay(ha_state_t *ha, uint32_t base);
  *   - HA_REPL_PUSH_CONFIG: In production builds, calls push_config_to_standbys(c, …).
  *     In XCE_TESTING builds, calls ha_repl_push_config_for_standby_hook(c, peer_idx).
  *   - HA_REPL_STREAM_WAL: calls ha_stream_wal_to_standby(ep, wal_path, standby_lsn,
- *     committed_lsn) to batch-stream WAL entries via xce_request() (non-blocking).
- *     In XCE_TESTING builds, calls ha_repl_stream_wal_hook if non-NULL.
+ *     stream_ceiling_lsn) to batch-stream WAL entries via xce_request()
+ *     (non-blocking). stream_ceiling_lsn (bug-1479 gap 3) is
+ *     ha_compute_repl_stream_ceiling_lsn(committed_lsn, last_written,
+ *     checkpoint_lsn) when c is non-NULL — the leader's own log tail, NOT
+ *     just committed_lsn, so a standby whose own ack is required to advance
+ *     committed_lsn can still receive the entries it needs to ack. Falls
+ *     back to committed_lsn when c is NULL (test fixtures with no real
+ *     persist_t). In XCE_TESTING builds, calls ha_repl_stream_wal_hook if
+ *     non-NULL.
  *   - HA_REPL_SEND_SNAPSHOT: calls ha_send_snapshot_to_standby(ep, snap_path, lsn).
  *     In XCE_TESTING builds, calls ha_repl_send_snapshot_hook(peer_idx) if non-NULL.
  *   - HA_REPL_CONVERGED: no-op.