Rongsong Shen 06135954ff chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family

Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x),
heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup.
New: ha-consistency-invariants umbrella (requirements/design/research) and
its four implementation sub-specs (specs generated via batch, cross-spec
review applied), steering roadmap + product.md Raft+CP decision.

2026-06-11 13:21:50 +00:00

2.9 KiB

Raw Blame History

Brief: ha-convergence-recovery

Sub-spec 4 of 4 implementing the ha-consistency-invariants umbrella contract. Authoritative requirements/design/research: .kiro/specs/ha-consistency-invariants/ (design.md §Demotion Convergence + §Egress Filter + §ACK-Driven Catch-up + §Quorum Observability; research.md §C five convergence gaps). This brief scopes the slice; do not re-derive decisions.

Problem

Five verified convergence gaps: a demoted ex-primary never truncates its divergent WAL suffix; phantom provisional in-memory state is never evicted; restart replays the uncommitted suffix (committed_lsn is volatile, correctly per Raft, but nothing reconciles at rejoin); standby apply has no conflict detection; and uncommitted jobs leak to the scheduler via dataex during the commit window (orphan-dispatch risk).

Desired Outcome

Cluster-wide single-outcome convergence and commit-gated egress: demotion truncates above committed_lsn and self-restarts through the existing boot recovery (provisional state discarded); a rejoining standby whose WAL tip exceeds the primary's committed boundary truncates and restarts once; the scheduler export path never sees a job whose submit entry is uncommitted; catch-up streaming continues on walAck without waiting for heartbeats; quorum-loss transitions are operator-visible in logs.

Scope

In: demotion path in ha.c (raft_cb_become_follower from PRIMARY: disable hooks, truncate above committed_lsn, WARN-log, self-restart; XCE_TESTING restart-hook seam); rejoin reconciliation (generalize RDHUP truncation to the rejoin/hello path); job_t.submit_lsn + scheduler-export filter in dataex.c (status/API queries intentionally unfiltered — documented deviation); ACK-driven catch-up continuation in config_repl.c (heartbeat path remains fallback + snapshot initiator); quorum enter/exit WARN logging; verification tests for unchanged behaviors (full fan-out 6.1, snapshot fallback 6.3, re-inclusion 6.4, degraded status service 7.3); 3-node E2E (extends ha-wal-replication task 5.4: confirmed jobs survive failover, unconfirmed jobs converge to absent).
Out: entity-bearing snapshots (roadmap backlog — would later upgrade eviction to option iii); read-consistency filtering of status queries; primary step-down on quorum loss; worker rejoin job-state resync.

Requirements (umbrella IDs)

2.4, 2.5, 2.7, 6.1, 6.2, 6.3, 6.4, 7.2, 7.3.

Upstream / Downstream

Upstream: ha-commit-engine (committed_lsn semantics). Parallel-safe with specs 2–3 except: full E2E requires ha-single-leader-role landed.
Downstream: entity-bearing snapshots (eviction upgrade path); worker-rejoin-resync (recovers facts across the restart this spec introduces).

Constraints

Self-restart must FATAL rather than continue if exec fails (a divergent ex-primary must not keep serving); restart seam mockable under XCE_TESTING; truncation always precedes restart so boot replay yields committed-only state.

2.9 KiB Raw Blame History Unescape Escape