--- id: "bug-1067" status: "fixed" status_updated: "2026-09-24" status_note: "" component: "controller" severity: "unknown" timestamp: "2026-06-24T00:00:00.000Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 1 files: ["src/controller/config_repl.c", "src/controller/ha.c"] tags: ["catch-up", "replication", "raft", "ae_reject_stale", "ae_reject_logmismatch", "walCatchupEntry", "cross-term", "FIXED"] related: ["bug-1028", "bug-991"] legacy: true --- # bug-1067: Cross-term WAL catch-up permanently stalls: ctrl-2 stuck at lsn=8, all walCatchupEntry rejected… ## Description Cross-term WAL catch-up permanently stalls: ctrl-2 stuck at lsn=8, all walCatchupEntry rejected (ae_reject_stale + ae_reject_logmismatch) ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction _Not recorded (migrated from buglog.json)._ ## Root cause AE term conflation: entry->term is the WAL entry's stored term (e.g. 4 for entries written by ctrl-1 at term=4), but ha_consume_append_entries uses it as the AE header ae_term. For live appends this is correct (leader appends at current term), but for catch-up streaming of historical entries from prior terms, ae_term = entry_term < leader.current_term. The Raft AE stale-term check (if term < s->term → ae_reject_stale) correctly rejects, but it should never fire — the AE header term must always be the leader's current term. There is no separate leader_term field in wal_catchup_entry_t. Consequence: historical entries at entry_term < raft.term are permanently rejected; standby_reported_lsn never advances; streamer retransmits forever. ## Fix IMPLEMENTED (2026-06-24): separated the AppendEntries HEADER term (leader's CURRENT term) from the per-entry STORED term. Added wire field WalCatchupEntry.leaderCurrentTerm @8 (controller.capnp) + C field wal_catchup_entry_t.leader_current_term (typemap.h, hand-defined struct — capnp generates encode/decode against it). Encoder (request.c wal_entry_to_wire + top container) writes it; decode auto-populates (generated, scalar mapname field). The catch-up streamer ha_stream_wal_to_standby gained a leader_term param (callers pass atomic_load(&c->ha.term)) and stamps entry->leader_current_term=leader_term on every streamed entry. Receiver ha_consume_append_entries now uses ae_term = e->leader_current_term ? e->leader_current_term : entry_term (single-entry branch) — the leader term rides on the sub-entry e by reference through ha_apply_term_entry_via_library, so no extra threading needed. entry->term (@4) remains the per-entry stored term for logOk/on_persist_entry. 0 = unset (legacy/live sender) → falls back to stored term (live appends have entry_term==leader_term, so unchanged). repl.c live paths UNTOUCHED. Tests (test_term_safety_adapter.c): test_intake_crossterm_catchup_uses_leader_current_term (historical entry@term4 streamed by leader@term5 to follower@term5 is ACCEPTED, not stale-rejected) + test_intake_stale_leader_term_still_rejected (leader_current_term=3 < follower term 5 still REJECTED — safety preserved). GREEN: 185 PASS term-safety+config-repl, 6 PASS wal_wire wire round-trip, 80 PASS failover/repl/durability, 0 failures. ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-1028, bug-991