---
id: "bug-850"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "unknown"
timestamp: "2026-06-20T15:14:36.135Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 2
files: ["src/controller/repl.c repl_add_standby (line ~119) + repl_broadcast in_catch_up gate"]
tags: ["wrong-value", "c", "ha", "raft", "replication", "catchup", "ordering", "race", "in_catch_up", "data-loss", "critical"]
related: ["bug-849", "bug-848", "bug-847", "bug-828", "bug-795"]
legacy: true
---

# bug-850: A rejoining standby receives LIVE replication of new entries before its historical catch-up…

## Description
A rejoining standby receives LIVE replication of new entries before its historical catch-up, advancing current_lsn past the gap so the idempotency guard skips the catch-up of the prefix. Explains ctrl-1 ending with [15..16] only (gap [1..14]). Distinct from bug-849; persists even after recovery.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
repl_add_standby initializes a freshly-registered standby with in_catch_up=0, so it is IMMEDIATELY eligible for repl_broadcast (which correctly skips in_catch_up standbys at repl.c:27). Mechanism (confirmed from ctrl-1.log 11:07:21): ctrl-1 reconnects -> registered in_catch_up=0 -> the new job [15..16] is live-broadcast to it (not gated) -> persist_wal_write_replicated writes them, current_lsn jumps to 17 -> THEN the catch-up stream of [1..14] arrives but persist_wal_write_replicated's idempotency guard (lsn+1 <= current_lsn) skips ALL of them (1+1<=17 ... 14+1<=17) -> permanent prefix gap [1..14]. Independent of bug-849: even with recovery (current_lsn=8), a live [15..16] arriving first jumps current_lsn to 17 and the catch-up of [8..14] is skipped (gap [8..14]). Root architectural cause: catch-up streaming and live broadcast are SEPARATE unordered paths (unlike Raft AppendEntries which is one ordered per-follower stream).

## Fix
FIXED 2026-06-21 via a CONTIGUITY GUARD at the persist layer (Raft log-matching), NOT in_catch_up. First tried defaulting in_catch_up=1 in repl_add_standby — REVERTED: it broke the whole controller suite (commit-WAIT tests TIMEOUT 300s because a fresh standby no longer counts toward write quorum until convergence, and the unit tests don't run the heartbeat dispatch that would clear it). Correct fix: persist_wal_write_replicated now REJECTS a forward gap (lsn > current_lsn) with -EAGAIN instead of writing it — so a live broadcast that arrives ahead of the in-order catch-up is DROPPED (no write, no ack, no current_lsn jump), and the ACK-driven catch-up continuation re-streams from current_lsn so the log fills in order. ha_apply_single_entry + ha_apply_wal_catchup_entry treat -EAGAIN as a quiet skip (TRACE, no ack, no ERROR spam). Combined with bug-848 (advertise durable_floor == current_lsn-1) the catch-up always starts AT current_lsn so it is never spuriously rejected; only genuine forward gaps are. Files: src/controller/persist.c (guard), src/controller/ha.c (quiet -EAGAIN). TDD: test_repl_write_rejects_forward_gap (test_repl_apply.c) — contiguous write ok, forward-gap lsn 5 @ current_lsn 2 -> -EAGAIN + current_lsn unchanged, fill 2..4 then 5 ok, recovery clean. FULL controller suite green (129/0/0, no timeout); daemon builds. LIVE VALIDATION PENDING with bug-848+849+850 together.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-849, bug-848, bug-847, bug-828, bug-795