myxce/.kiro/specs/ha-convergence-recovery/design.md

Design: ha-convergence-recovery

Sub-spec 4 of 4 of the ha-consistency-invariants umbrella. This design slices the umbrella design's Demotion Convergence, Egress Filter, ACK-Driven Catch-up, and Quorum Observability components plus the role-lifecycle flow (.kiro/specs/ha-consistency-invariants/design.md) — umbrella Migration Strategy phase 4. Discovery is complete via the umbrella's research.md (deep-dive §C five convergence gaps; locked decisions) plus this spec's anchor-verification pass (research.md, 2026-06-11). Requirement IDs below are THIS spec's IDs (see requirements.md for the umbrella mapping).

Overview

Purpose: This feature closes the five verified convergence gaps so the cluster converges to a single outcome for every log entry and the scheduler never acts on uncommitted state: a demoted ex-primary truncates its uncommitted WAL suffix and self-restarts through boot recovery; a rejoining node whose log runs ahead of the cluster's committed boundary reconciles by truncation and one restart; scheduler-bound job exports are gated on the commit watermark; catch-up streaming continues on WAL ACK instead of waiting for heartbeats; and quorum-loss/restore transitions are operator-visible.

Users: Cluster operators gain single-outcome failover semantics (no phantom state on demoted/rejoined nodes) and observable degraded periods; the scheduler is protected from orphan dispatch; lagging standbys catch up at ACK speed.

Impact: Extends raft_cb_become_follower (ha.c:2104-2247 — line drift only, was ha.c:627-690 at design time; currently neither truncates nor disables hooks, verified gap) with the demotion-convergence sequence; generalizes the RDHUP truncation (ha.c:3416-3420 at design time) to the rejoin path (Note: superseded by ha-term-safety — the rejoin/RDHUP watermark-truncate heuristic this spec built was retired and replaced by the raft library's (term,lsn)-driven on_truncate_from; see the Rejoin Reconciliation component below and that spec's design.md); adds submit_lsn to the job_t wrapper and a watermark predicate on the scheduler export seams; adds an ACK-driven continuation to catch-up streaming (ha_stream_wal_to_standby, now config_repl.c:2351 — today re-driven only by the next heartbeat reply); adds quorum-transition WARN logging. No commit semantics change: this spec consumes committed_lsn (ha-commit-engine) and never modifies commit.c.

Goals

• Demotion = disable hooks → truncate above committed_lsn → WARN-log range → process self-restart via existing boot recovery; exec failure is FATAL; XCE_TESTING restart seam (1.1–1.7)
• Rejoin reconciliation: local tip beyond the primary's committed boundary → clamped truncation + one restart (2.1–2.3); single-outcome convergence E2E-verified (2.4)
• Commit-watermark egress filter on scheduler-bound exports only; status/API reads untouched (3.1–3.4)
• ACK-driven catch-up continuation with single-flight guard; heartbeat path stays fallback + snapshot initiator; parity triggers re-inclusion (5.1–5.3, 7.1)
• Quorum-loss/restore transitions WARN-logged once per change (8.1–8.3)
• Regression tests pinning existing full fan-out, snapshot fallback, re-inclusion, degraded-mode service (4.1, 6.1, 7.1, 9.1–9.2)

Non-Goals

• Commit-index advancement, majority evaluation, admission cap (ha-commit-engine — upstream, complete)
• WAL-type classification, reply ordering, config barrier (ha-commit-barrier-rollout)
• Promotion no-op, own-term gate consumers, CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role; this spec's E2E depends on it having landed)
• Entity-bearing snapshots (roadmap — would upgrade eviction to in-place rebuild); read-consistency filtering of status queries; primary step-down on quorum loss; worker rejoin job-state resync (roadmap backstop)
• Standby-apply conflict detection beyond the rejoin truncation rule (umbrella gap 5 is subsumed by reconciliation: divergent suffixes are truncated at role/lifecycle events, never patched in place)

Boundary Commitments

This Spec Owns

• The demotion path of the role lifecycle: everything raft_cb_become_follower does when leaving PRIMARY (hook disable, truncation, logging, self-restart) and the self-restart primitive (saved argv, exec, FATAL fallback, test seam)
• Rejoin reconciliation at the standby's CtrlHelloReply intake (truncate-above-boundary decision, clamp, once-guard) (Note: superseded by ha-term-safety — ha_rejoin_reconcile is retained only as the intake hook that records the learned boundary; the truncate-above-boundary decision, clamp, and once-guard it describes were retired in favor of the raft library's term-aware truncation, see Rejoin Reconciliation component below)
• job_t.submit_lsn tagging (live append + replay) and the commit-watermark predicate on scheduler-bound export seams
• The ACK-driven catch-up continuation trigger (walAck branch → scheduled streaming) including the single-flight guard and the parity call to repl_complete_catchup
• Quorum-loss/restore transition logging and its quorum_lost state flag
• Regression tests for umbrella 6.1/6.3/6.4/7.3 and the 3-node convergence E2E (extends ha-wal-replication task 5.4)

Out of Boundary

• commit.c / include/commit.h — read-only consumer of committed_lsn; never modified here
• repl_wait_majority_ack, write-path classification, reply ordering (ha-commit-barrier-rollout); promotion path of the lifecycle (raft_cb_become_leader) and COMMITTING removal (ha-single-leader-role)
• ha_stream_wal_to_standby internals and the WAL batch wire format (ha-wal-replication — the continuation calls it, unchanged)
• persist_truncate_wal_above, persist_recover, replay_wal internals — reused as-is
• Status/API read paths (apiserver.c:83, opget.c:173) — intentionally unfiltered (documented deviation)
• Worker/scheduler reconnection behavior after the restart this spec introduces (worker-join-refactor / worker-rejoin-resync)

Allowed Dependencies

• persist_t.committed_lsn atomic (lock-free reads) — the watermark and truncation boundary; ha-commit-engine owns advancement
• persist_truncate_wal_above (persist.c:1306), persist_recover boot path — truncation + rebuild primitives
• ha_stream_wal_to_standby (config_repl.c:1450) and repl_start_catchup/repl_complete_catchup (repl.c:507/521) — existing catch-up machinery; repl_complete_catchup already feeds commit_on_membership_change via ha-commit-engine wiring
• iomgr/vtaskmgr/delaymgr threading rules per ha-async-io.md: the walAck branch and raft_cb_become_follower may run on iomgr — file I/O (truncation, WAL streaming) and exec are always deferred to delaymgr/vtask tasks
• xce_disable_hook / hook registration guard (hooks_registered) — existing hook lifecycle
• Established XCE_TESTING function-pointer seam pattern (e.g., ha_repl_stream_wal_hook, xce_wal_stream_request_hook)

Revalidation Triggers

• ha-commit-engine changing committed_lsn ownership or making it persisted → re-verify rejoin reconciliation and the egress watermark
• ha-commit-barrier-rollout landing (fact allowlist) → re-verify the degraded-mode gate pin: the no-quorum rejected set becomes request-shaped opcodes only; fact-shaped opcodes (OPCode_notify/update/setpreason/clrpreason) are admitted
• ha-single-leader-role landing → re-anchor the quorum-transition log call site at promotion standby registration (ha.c:531), which lives inside raft_cb_become_leader (ha.c:445-617) — code that spec owns and rewrites wholesale. (Resolved as-built: re-anchored to ha.c:2042 inside raft_cb_become_leader, now ha.c:1615-2103; the demotion insertion point re-verified at ha.c:2104-2247. Both current per the fact-check pass below.)
• ha-single-leader-role changing the become-follower/become-leader callback contract → re-verify the demotion sequence insertion point
• Entity-bearing snapshots landing → re-evaluate restart-based eviction vs snapshot-based rebuild (umbrella revalidation trigger)
• Any new scheduler-bound export path → must adopt the watermark predicate (same rule as new WAL types must be classified)
• worker-rejoin-resync landing → re-verify restart interaction (facts lost across the restart are its recovery target)

Architecture

Existing Architecture Analysis

Note: this subsection is a research-time (2026-06-11) snapshot of the PRE-implementation baseline — the "verified gap" facts are what the code lacked before this spec ran. Where a later spec has since changed the mechanism, a note is added; otherwise only stale file:line citations are corrected in place.

• raft_cb_become_follower (now ha.c:2104-2247, was ha.c:627-690 at design time) cancels timers, closes vote endpoints, sets CTRL_STANDBY — verified: no truncation, no hook disable (the disable logic exists only in the uncalled ha_demote_to_standby, now ha.c:5081-5094, was ha.c:1712-1726 at design time)
• The RDHUP failover path already truncates above the locally known committed boundary (ha.c:3416-3420 at design time) — the pattern this spec generalizes to demotion and rejoin. (Note: superseded by ha-term-safety — this ad hoc watermark-truncate pattern, and the rejoin generalization built on it, was itself replaced by the raft library's (term,lsn)-driven on_truncate_from / ha_cb_truncate_from_stub, ha.c:2638-2655, whose own doc comment states it "REPLACES the committed-boundary truncate heuristic on the library-driven path"; see that spec's design.md)
• The standby learns the primary's committed_lsn at the CtrlHelloReply intake (now cluster.c:411-419, was cluster.c:283-298 at design time, stored to ha->last_known_committed_lsn) — the rejoin-reconciliation evaluation point
• Catch-up is heartbeat-polled: the stateReport branch (now ha.c:6683, was ha.c:2734) drives ha_dispatch_repl_action_for_standby (now ha.c:4331-4466, was ha.c:1373-1478), one action per heartbeat cycle; the walAck branch (now ha.c:6764-6812, was ha.c:2737-2754) only records the ACK. Replication target is already derived as max(committed_lsn, last_written) (now via ha_compute_repl_target_lsn, called at ha.c:6759, was inline at ha.c:2727-2732)
• repl_complete_catchup (now repl.c:769, was repl.c:521 at design time) has no production caller (discovered gap — in_catch_up is set by broadcast-overflow eviction, repl.c:240, and never cleared); the ACK-driven continuation supplies the missing parity trigger
• Scheduler-bound egress has two seams: collect-based full push (xcesched_push_jobs now sched.c:214, was sched.c:169, on scheduler join) and the incremental JOB_NEW push (sched_job_new_fun now sched.c:286, was sched.c:236-248 → xcesched_push_joblist); both must apply the watermark. Read paths (apiserver.c:83, opget.c:173 — both confirmed current, unchanged) share collect_jobs (dataex.c:294 — confirmed current, unchanged) and stay unfiltered
• No saved argv / exec machinery exists anywhere in the controller — self-restart is net-new (main.c:112 at design time; landed at main.c:291-301 as-built)

Architecture Pattern & Boundary Map

graph TB
    subgraph role_lifecycle[role lifecycle - ha.c]
        BF[raft_cb_become_follower<br/>from PRIMARY]
        DC[demotion convergence task<br/>delaymgr]
        SR[ha_self_restart<br/>exec or test seam]
    end
    subgraph rejoin[rejoin - cluster.c + ha.c]
        CHR[CtrlHelloReply intake]
        RR[rejoin reconciliation]
    end
    subgraph persist[persist.c - reused]
        TR[persist_truncate_wal_above]
        REC[persist_recover boot replay]
    end
    subgraph egress[egress - dataex.c, sched.c, job.c]
        TAG[submit_lsn tagging]
        PRED[job_submit_committed predicate]
        PUSH[scheduler push paths]
    end
    subgraph catchup[catch-up - ha.c + config_repl.c]
        ACK[walAck branch iomgr]
        CONT[continuation task delaymgr]
        STREAM[ha_stream_wal_to_standby]
        RCC[repl_complete_catchup]
    end
    QO[quorum observability - ha.c]
    CL[(committed_lsn<br/>ha-commit-engine owned)]

    BF --> DC --> TR
    DC --> SR
    CHR --> RR --> TR
    RR --> SR
    SR -. process restart .-> REC
    TAG --> PRED
    CL --> PRED --> PUSH
    ACK -->|behind target| CONT --> STREAM
    ACK -->|at parity| RCC
    CL --> ACK
    QO -. WARN once per transition .- CL

Architecture Integration (sliced from umbrella):

• Selected pattern: convergence by recovery, not by transaction — discard events coincide with role/lifecycle events (demotion, rejoin), handled by truncate + boot-recovery rebuild; the only steady-state mechanism is the one-comparison watermark filter (umbrella-locked)
• Existing patterns preserved: RDHUP truncation pattern, heartbeat decision loop (kept as fallback), enqueue-only streaming, XCE_TESTING seams, errnox early-return style
• New code rationale: each gap closes with the smallest insertion at a verified seam; no new module — all changes live in the files that own the respective seams
• Steering compliance: no file I/O or blocking on iomgr (convergence + continuation run on delaymgr tasks); TDD with testsuite/ tests first

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Backend / Services	C (gnu17), pthreads, stdatomic	demotion/rejoin/egress/catch-up logic	no new dependencies, no .capnp changes
Data / Storage	existing WAL + snapshot sentinel (persist.c)	truncation boundary + boot rebuild	submit_lsn is in-memory only (recomputed from entry LSN at replay)
Process	execv re-exec of saved argv	demotion/rejoin eviction	FATAL on failure; seam under XCE_TESTING
Build / Test	meson, testsuite/ pattern	new test files in controller_unit_test_src	E2E via /xce-deploy-ha

File Structure Plan

Directory Structure

src/controller/
├── ha.c              # MODIFIED — demotion branch inline in raft_cb_become_follower, walAck
│                     #            continuation trigger, quorum transition call sites
├── ha_convergence.c  # NEW (as-built, not in original plan) — ha_self_restart,
│                     #            ha_rejoin_reconcile, cluster_apply_ctrlhello_reply,
│                     #            ha_log_quorum_transition, ha_demotion_convergence_do_run
│                     #            all landed here instead of ha.c
├── ha.h              # MODIFIED — quorum_lost flag, catchup_inflight tracking, restart-hook
│                     #            seam declaration, new function prototypes
├── main.c            # MODIFIED — save argv for re-exec (controller_saved_argv accessor)
├── cluster.c         # MODIFIED — CtrlHelloReply intake calls rejoin reconciliation
├── config_repl.c     # MODIFIED — catch-up continuation task fn (wraps ha_stream_wal_to_standby)
├── egress.c          # NEW (as-built, not in original plan) — job_submit_committed predicate +
│                     #            collect_jobs_committed, factored out of dataex.c for
│                     #            test-binary linkage (research D3 refinement; declared in
│                     #            include/cluster.h, not dataex.h)
├── sched.c           # MODIFIED — scheduler push paths consume the predicate (full + JOB_NEW push)
├── job.c             # MODIFIED — restore_job tags submit_lsn from the replayed entry
├── persist.c         # MODIFIED — persist_job_hook_fun tags submit_lsn at live append;
│                     #            apply_wal_entry passes entry LSN to restore_job
└── testsuite/
    ├── test_demotion_convergence.c   # NEW — demotion + rejoin reconciliation + restart seam
    ├── test_egress_filter.c          # NEW — watermark filter on scheduler export seams
    └── test_convergence_verification.c # NEW — ACK-driven catch-up, quorum logging, and
                                        #       regression pins for 6.1/6.3/6.4/7.3 (umbrella)
src/include/xce/types.h               # MODIFIED — job_t wrapper gains uint64_t submit_lsn
src/controller/meson.build            # MODIFIED — register new test files

Modified Files (what and why)

• ha.c / ha.h — demotion branch in raft_cb_become_follower (ha.c:2104-2247; detect prior role PRIMARY → disable hooks inline → schedule convergence task); walAck-branch (ha.c:6764-6812) continuation scheduling with per-standby single-flight guard and parity repl_complete_catchup call; ha_log_quorum_transition() called from the standby_endpoints mutation sites (now ha.c:2042, 7651, 9350) and the gate rejection backstop (ha.c:4617). As-built, ha_self_restart() and ha_rejoin_reconcile() landed in a new ha_convergence.c (see File Structure Plan) rather than ha.c directly.
• main.c — store argc/argv before parsing (main.c:291-301); expose controller_saved_argv() for re-exec
• cluster.c — CtrlHelloReply intake (now cluster.c:411-419) additionally invokes ha_rejoin_reconcile(c, chrep->committed_lsn) via cluster_apply_ctrlhello_reply. (Note: superseded by ha-term-safety — ha_rejoin_reconcile now only records the boundary; see Rejoin Reconciliation component.)
• config_repl.c — ha_catchup_continuation_fn (now config_repl.c:2839, delaymgr task body): re-derive target, call ha_stream_wal_to_standby (config_repl.c:2351), clear the in-flight flag
• egress.c (as-built; design originally assigned this to dataex.c, see File Structure Plan) — job_submit_committed(cluster_t*, job_t*) predicate; collect_jobs_committed() wrapper used only by scheduler-bound callers (read paths keep plain collect_jobs, dataex.c:294)
• sched.c — xcesched_push_jobs (sched.c:214) collects via the committed-only wrapper; xcesched_push_joblist (sched.c:70, JOB_NEW incremental push) filters its list with the same predicate
• job.c / persist.c — tagging: live path sets submit_lsn = written_lsn in persist_job_hook_fun (now persist.c:6509); replay path threads hdr->lsn from apply_wal_entry (now persist.c:3281) into restore_job (now job.c:274)
• src/include/xce/types.h — job_t wrapper (types.h:483-490) gains uint64_t submit_lsn (0 = unknown/pre-feature = exportable); wire type xce_job_t untouched

System Flows

Demotion convergence (role lifecycle, after this spec — umbrella flow sliced)

sequenceDiagram
    participant RL as raft library
    participant BF as raft_cb_become_follower
    participant DM as delaymgr task
    participant P as persist.c
    participant OS as process
    RL->>BF: higher term seen while PRIMARY
    BF->>BF: disable persistence hooks (inline, first)
    BF->>BF: set CTRL_STANDBY, cancel timers (existing)
    BF->>DM: schedule demotion convergence (one-shot)
    DM->>P: persist_truncate_wal_above(committed_lsn)
    DM->>DM: WARN log: truncated (committed_lsn, prior tip]
    DM->>OS: ha_self_restart: execv(saved argv)
    alt exec fails
        OS->>OS: FATAL (abort) — never continue divergent
    else XCE_TESTING
        DM->>DM: ha_restart_hook seam invoked instead
    end
    OS-->>P: boot: persist_recover replays committed prefix only

Key decisions: hooks are disabled synchronously in the callback (no provisional entry may be produced after the higher term is observed — 1.1); truncation and exec are deferred off the callback thread (iomgr-safe — D1); truncation always precedes restart so boot replay yields committed-only state (1.2, 1.3, 1.7); crash before/during the task is safe — the rejoin reconciliation flow covers the untruncated-suffix case (Note: as of ha-term-safety, that coverage is via the raft library's term-aware truncation, not this spec's rejoin-reconcile truncate path — see Rejoin Reconciliation component). The "exec fails → FATAL (abort)" branch in the diagram above is also superseded: see the Self-Restart Primitive component and Error Handling section for the current fence-and-retry behavior.

Rejoin reconciliation (generalizes RDHUP truncation, ha.c:3416-3420 at design time)

(Note: superseded by ha-term-safety, closed 2026-07-17.) The flow and "Key decisions" below describe this spec's original clamped-truncate-and-restart mechanism, which is no longer live. ha_rejoin_reconcile (ha_convergence.c:467-493) is now a documented no-op: the truncate-above-boundary decision, clamp, and once-guard were retired in favor of the raft library's own (term,lsn) log-matching on incoming AppendEntries, which drives on_truncate_from to delete exactly the divergent suffix without a self-restart. The function is retained only as the CtrlHelloReply hook that records the learned committed boundary. See ha-term-safety's design.md for the current mechanism.

sequenceDiagram
    participant PRI as primary
    participant CL as cluster.c intake
    participant RR as ha_rejoin_reconcile
    participant P as persist.c
    PRI->>CL: CtrlHelloReply (committed_lsn)
    CL->>CL: store last_known_committed_lsn (existing)
    CL->>RR: evaluate(local tip vs boundary)
    alt tip > boundary and not already reconciled
        RR->>P: truncate above max(boundary, local committed_lsn)
        RR->>RR: WARN log range, set once-guard
        RR->>RR: ha_self_restart (same primitive)
    else tip <= boundary
        RR->>CL: no action — join proceeds (2.3)
    end

Key decisions: the clamp max(primary_reported, local committed_lsn) never discards the locally known committed prefix (2.2); "once" is structural — after restart the tip equals the boundary, so the condition cannot re-fire (plus a defensive in-process flag) (2.1); together with demotion convergence and the never-retroactive-success rule this yields single-outcome convergence, asserted end-to-end (2.4).

Commit-gated scheduler egress

Live submit: persist_job_hook_fun appends the submit entry and tags j->submit_lsn = written_lsn. Export time (scheduler-bound only — full push on scheduler join, JOB_NEW incremental push): job_submit_committed admits the job iff submit_lsn == 0 || submit_lsn <= committed_lsn (one atomic load, lock-free — 3.1, 3.2). Replay/standby apply tags from the entry LSN; on a promoted primary, replayed entries sit at or below the promotion truncation point, so restored jobs are exportable by construction (3.3). Status/API readers bypass the predicate entirely (3.4). Accepted edge (documented): a timeout-failed submit that later commits reaches the scheduler at the next scheduler (re)join full push — the requester keeps its failure answer (2.4 alignment).

ACK-driven catch-up continuation

sequenceDiagram
    participant S as standby
    participant IO as iomgr (walAck branch)
    participant DM as delaymgr continuation
    participant CR as ha_stream_wal_to_standby
    S->>IO: WalAckReply(acked_lsn)
    IO->>IO: repl_on_ack (existing; feeds commit engine)
    alt acked_lsn < target = max(committed, last_written)
        IO->>DM: schedule continuation if not already in flight
        DM->>CR: stream [acked_lsn+1, target] (enqueue-only)
        DM->>DM: clear in-flight flag
        CR-->>S: next batches
    else acked_lsn >= target
        IO->>IO: repl_complete_catchup → re-inclusion (7.1)
    end

Key decisions: continuation is scheduled from the walAck branch (ha.c:2737-2754) but executes on delaymgr (streaming reads WAL files — never on iomgr); the per-standby single-flight flag bounds concurrency (5.2); target re-derivation each round naturally extends the stream to entries appended meanwhile (5.1); the heartbeat decision loop is untouched — it remains the fallback initiator and the only snapshot-decision owner (5.3, 6.1); parity wires the existing-but-untriggered repl_complete_catchup (7.1).

Requirements Traceability

Requirement	Summary	Components	Flows
1.1	hooks disabled on demotion before anything else	Demotion Convergence	demotion
1.2	truncate above committed_lsn on demotion	Demotion Convergence	demotion
1.3	self-restart rebuilds via boot recovery	Demotion Convergence, Self-Restart Primitive	demotion
1.4	exec failure → FATAL	Self-Restart Primitive	demotion
1.5	WARN log with truncation range	Demotion Convergence	demotion
1.6	XCE_TESTING restart seam	Self-Restart Primitive	demotion
1.7	state reconstructible from committed log	Demotion Convergence (boot recovery reuse)	demotion
2.1	rejoin: tip > boundary → truncate + restart once	Rejoin Reconciliation	rejoin
2.2	never truncate own committed prefix	Rejoin Reconciliation (clamp)	rejoin
2.3	tip <= boundary → no action	Rejoin Reconciliation	rejoin
2.4	single outcome, never retroactive success	Demotion Convergence + Rejoin Reconciliation (E2E)	demotion, rejoin
3.1	uncommitted submit excluded from scheduler export	Egress Filter	egress
3.2	included once committed	Egress Filter	egress
3.3	submit_lsn tagging live + replay	Egress Filter (tagging)	egress
3.4	status/API reads unfiltered	Egress Filter (boundary)	egress
4.1	full fan-out to all reachable standbys	Convergence Verification Suite (existing repl_broadcast)	—
5.1	continuation on walAck, no heartbeat wait	ACK-Driven Catch-up	catch-up
5.2	single-flight per standby	ACK-Driven Catch-up	catch-up
5.3	heartbeat loop stays fallback + snapshot initiator	ACK-Driven Catch-up (boundary)	catch-up
6.1	snapshot fallback for far-behind standby	Convergence Verification Suite (existing path)	—
7.1	re-inclusion at parity	ACK-Driven Catch-up (parity trigger), Convergence Verification Suite	catch-up
8.1	quorum-loss logged with counts	Quorum Observability	—
8.2	quorum-restore logged	Quorum Observability	—
8.3	once per transition, no storms	Quorum Observability	—
9.1	status served while quorum lost	Convergence Verification Suite (existing gate allowlist)	—
9.2	replication traffic served while quorum lost	Convergence Verification Suite (existing gate allowlist)	—

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies	Contracts
Demotion Convergence	controller/ha.c	PRIMARY→follower: disable hooks, truncate, log, restart	1.1–1.3, 1.5, 1.7, 2.4	persist truncation + recovery (P0), delaymgr (P0), Self-Restart (P0)	State
Self-Restart Primitive	controller/ha.c, main.c	re-exec saved argv; FATAL on failure; test seam	1.3, 1.4, 1.6	saved argv (P0)	Service
Rejoin Reconciliation	controller/ha.c, cluster.c	clamped truncation + one restart at rejoin	2.1–2.4	CtrlHelloReply intake (P0), persist truncation (P0), Self-Restart (P0)	State
Egress Filter	controller/dataex.c, sched.c, job.c, persist.c, types.h	watermark gate on scheduler-bound exports	3.1–3.4	committed_lsn (P0, read-only)	Service
ACK-Driven Catch-up	controller/ha.c, config_repl.c	continuation on walAck; parity re-inclusion	5.1–5.3, 7.1	walAck branch (P0), ha_stream_wal_to_standby (P0), repl_complete_catchup (P0), delaymgr (P0)	Event
Quorum Observability	controller/ha.c, ha.h	WARN once per quorum transition	8.1–8.3	ha_has_write_quorum (P0)	Service
Convergence Verification Suite	testsuite/, E2E	pin existing 6.1/6.3/6.4/7.3 behaviors + 3-node convergence	2.4, 4.1, 6.1, 7.1, 9.1, 9.2	/xce-deploy-ha (E2E only; needs ha-single-leader-role)	—

Demotion Convergence + Self-Restart Primitive (ha.c, main.c)

Field	Detail
Intent	Discard the demoted ex-primary's provisional state by truncation + boot-recovery restart
Requirements	1.1–1.7, 2.4

Responsibilities & Constraints

• raft_cb_become_follower gains a was-PRIMARY branch (prior role read before the existing role store): synchronously disable the three persistence hooks (reuse the ha_demote_to_standby disable sequence + hooks_registered reset, ha.c:1712-1726 — currently uncalled from this path), then schedule the one-shot convergence task; all existing behavior (timer cancel, vote-ep close, STANDBY store, watchdog arm) is preserved for the non-PRIMARY cases
• Convergence task (delaymgr): persist_truncate_wal_above(&c->persist, committed_lsn) where committed_lsn is read from persist_t at task time; WARN with both LSNs (boundary and prior tip); then ha_self_restart(c)
• Deliberate scoping: a demotion from CTRL_PRIMARY_COMMITTING (a state still extant if this spec lands before ha-single-leader-role removes it) takes the legacy become-follower path with no truncation/restart — acceptable because a COMMITTING node has no registered persistence hooks and has appended no new entries, so its exposure is identical to today's
• ha_self_restart: flush logs; execv of controller_saved_argv(); if exec returns → ERROR + abort() (FATAL — 1.4). Under XCE_TESTING, invoke ha_restart_hook(c) instead and return (1.6); a settable fatal seam lets tests assert the FATAL path without dying
• fd hygiene: before exec, verify/ensure listener and peer fds do not leak into the new image (CLOEXEC audit; boot path re-creates all endpoints)
• Crash safety: if the node dies before the task runs, its untruncated suffix is handled by Rejoin Reconciliation at the next join — the two components together guarantee 2.4

Service Interface

/* ha.h — controller-internal */
/* Saved-argv accessor (main.c populates before argp_parse). NULL-terminated. */
char **controller_saved_argv(void);

/* Truncate-and-restart primitive. Never returns in production (exec or abort).
 * Under XCE_TESTING returns 0 after invoking ha_restart_hook. */
int ha_self_restart(cluster_t *c);

#ifdef XCE_TESTING
extern int (*ha_restart_hook)(cluster_t *c);   /* invoked instead of execv */
extern void (*ha_fatal_hook)(const char *msg); /* invoked instead of abort */
#endif

• Preconditions: persistence hooks already disabled; truncation completed before ha_self_restart is called
• Postconditions (production): process image replaced; next boot replays only the surviving prefix
• Invariants: no new WAL entry is appended by this node between hook-disable and exec

Rejoin Reconciliation (ha.c + cluster.c intake)

Field	Detail
Intent	Converge a rejoining node whose log runs ahead of the cluster's committed boundary
Requirements	2.1–2.4

(Note: superseded by ha-term-safety, closed 2026-07-17 — the mechanism below describes this spec's original design/as-shipped behavior; see the current-state summary immediately after.)

Original design/as-shipped: new int ha_rejoin_reconcile(cluster_t *c, uint64_t primary_committed_lsn) called from the CtrlHelloReply intake (cluster.c:283-298 at design time, now cluster.c:411-419) after the existing last_known_committed_lsn store. Decision: local_tip = persist.current_lsn - 1; if local_tip > primary_committed_lsn and the once-guard is clear → cutoff = max(primary_committed_lsn, local committed_lsn) (clamp, 2.2) → persist_truncate_wal_above(cutoff) → WARN with range → set once-guard → ha_self_restart. Otherwise return 0 with no side effects (2.3). The truncation/restart work is deferred to a delaymgr task if the intake runs on iomgr (same rule as demotion). Generalizes the RDHUP truncation (ha.c:3416-3420 at design time), which the design assumed remains in place for the failover path. Once-guard is structural (post-restart tip equals the boundary) plus a defensive _Atomic int rejoin_reconciled for the process lifetime (2.1).

Current state (ha_convergence.c:467-493): ha-term-safety task 3.2 retired this entire clamped-truncate-and-restart body. ha_rejoin_reconcile now ignores primary_committed_lsn and unconditionally returns 0 — divergent-suffix reconciliation is owned by the raft library instead: an incoming AppendEntries is routed through raft_on_append_entries_request, whose (term,lsn) log-matching drives on_truncate_from (ha_cb_truncate_from_stub, ha.c:2638-2655) to delete exactly the divergent suffix, never a committed-prefix entry, and never by an LSN watermark. A node whose tip runs ahead keeps its entries until the leader's AE proves the conflict by term; until reconciled the existing promotion-time fence keeps it non-promotable without self-restarting. The function is retained solely as the CtrlHelloReply hook (now wired via cluster_apply_ctrlhello_reply, ha_convergence.c:512-524) that still records the learned committed boundary. The Testing Strategy section's rejoin-reconcile test descriptions below are similarly stale; current tests (test_rejoin_no_watermark_truncate_no_restart, test_rejoin_repeated_calls_remain_no_op) pin the no-op contract instead.

Egress Filter (dataex.c, sched.c, job.c, persist.c, types.h)

Field	Detail
Intent	One-watermark commit gate on scheduler-bound job exports
Requirements	3.1–3.4

Responsibilities & Constraints

• job_t wrapper (types.h:483-485) gains uint64_t submit_lsn; 0 = unknown/pre-feature = exportable (legacy WALs and pre-tagging jobs never strand); wire type xce_job_t unchanged
• Tagging: live — persist_job_hook_fun (persist.c:1107-1125) sets submit_lsn = written_lsn on each submitted job after a successful append; replay/standby apply — apply_wal_entry (persist.c:729-731) passes hdr->lsn to restore_job (job.c:71), which tags the restored job (3.3)
• Predicate (dataex.c): bool job_submit_committed(cluster_t *c, job_t *j) → j->submit_lsn == 0 || j->submit_lsn <= atomic_load(committed_lsn); lock-free, one comparison
• Consumers (scheduler-bound only): collect_jobs_committed() wrapper in dataex.c used by xcesched_push_jobs (sched.c:183); the JOB_NEW incremental push (xcesched_push_joblist, fed by sched_job_new_fun sched.c:236-248) filters its list with the same predicate. Both seams verified in the anchor pass — research D3
• Non-consumers (3.4): apiserver.c:83 and opget.c:173 keep plain collect_jobs — status/API reads intentionally unfiltered (documented deviation; read consistency out of scope)

Service Interface

/* dataex.h additions — controller-internal */
bool job_submit_committed(cluster_t *c, job_t *j);          /* watermark predicate */
void collect_jobs_committed(cluster_t *c, collect_check_fn_t fn, xlist_t *l);
/* identical contract to collect_jobs, additionally excluding jobs whose
 * submit entry is uncommitted; for scheduler-bound callers ONLY */

• Invariant: a job with submit_lsn > committed_lsn is never present in a scheduler-bound export payload; the same job remains visible to status/API reads

ACK-Driven Catch-up (ha.c walAck branch + config_repl.c continuation)

Field	Detail
Intent	Continue catch-up streaming at ACK speed; re-include at parity
Requirements	5.1–5.3, 7.1

Responsibilities & Constraints

• Trigger (iomgr, ha.c:2737-2754): after the existing repl_on_ack, compare acked_lsn to target = max(committed_lsn, last_written) (same derivation as ha.c:2727-2732). Behind target → schedule the continuation task unless the standby's catchup_inflight flag is set (5.2); at/above target → repl_complete_catchup(&c->persist.repl, ep) (7.1 — wires the existing, currently untriggered re-inclusion; ha-commit-engine's wiring propagates commit_on_membership_change)
• Continuation task (delaymgr, config_repl.c): re-derive target, call ha_stream_wal_to_standby(ep, wal_path, acked_lsn, target) (unchanged internals, enqueue-only), clear catchup_inflight. Endpoint validity is re-checked against the standby registry at task time (the ep may have RDHUPed between schedule and run)
• Per-standby state: _Atomic int catchup_inflight keyed by the standby slot (placed alongside the existing per-peer arrays in ha_state_t or standby_conn_t — slot identity must survive between schedule and completion)
• The heartbeat decision loop (ha_dispatch_repl_action_for_standby, ha.c:1373-1478) is byte-identical: fallback initiator for standbys that never ACK, and sole owner of the snapshot decision (5.3; snapshot fallback 6.1 of this spec verified, not modified)
• Never blocks iomgr: scheduling is O(1); all file I/O is inside the delaymgr task

Quorum Observability (ha.c, ha.h)

Field	Detail
Intent	Operator-visible quorum-loss/restore transitions
Requirements	8.1–8.3

Summary: ha_state_t gains _Atomic int quorum_lost (0 at init). New ha_log_quorum_transition(ha_state_t *ha): evaluate ha_has_write_quorum; on edge, atomically flip the flag and WARN once — loss: "HA quorum lost: connected=%d required=%d"; restore: "HA quorum restored: connected=%d required=%d" (8.1, 8.2). Call sites: the three standby_endpoints mutation points (promotion registration ha.c:531, RDHUP removal ha.c:3160-3166, on_standby_connected ha.c:3764+) plus the ha_dispatch_gate quorum-rejection branch as a backstop (flag check makes repeated rejections silent — 8.3). CAS on the flag guarantees once-per-transition under concurrency. Note: the promotion-registration anchor (ha.c:531) lives inside raft_cb_become_leader (ha.c:445-617), leader-role promotion code that ha-single-leader-role owns and rewrites wholesale — this call site must be re-anchored after ha-single-leader-role lands (or coordinated with that spec if this one is implemented first); see Revalidation Triggers.

Convergence Verification Suite (testsuite/, E2E)

Summary: regression pins for the four existing behaviors the umbrella marked "Done (verify)": full fan-out (repl_broadcast reaches every active standby — 4.1), snapshot-fallback decision (standby_lsn < oldest_wal_lsn → HA_REPL_SEND_SNAPSHOT — 6.1), re-inclusion mechanism (repl_complete_catchup clears in_catch_up, standby re-enters broadcast and quorum — 7.1), degraded-mode allowlist (gate admits status + replication opcodes without quorum — 9.1, 9.2). Plus the 3-node E2E extending ha-wal-replication task 5.4: confirmed jobs survive failover; a timeout-failed submit converges to a single outcome on all members; the demoted ex-primary restarts once and rejoins converged; status served during isolation (2.4 + live 9.1). E2E requires ha-single-leader-role landed and a live /xce-deploy-ha cluster.

Data Models

Domain Model (changes only)

• job_t (types.h wrapper): + uint64_t submit_lsn — entry LSN of the job's submit record; 0 = unknown/exportable; in-memory only (recomputed at replay from the applied entry; never serialized — wire type unchanged)
• ha_state_t: + _Atomic int quorum_lost (transition edge detector); + per-standby _Atomic int catchup_inflight[MAX_HA_CANDIDATES] (single-flight guard); + _Atomic int rejoin_reconciled (defensive once-guard)
• process: saved argv (main.c static, NULL-terminated) — the restart vehicle
• Invariants: scheduler-bound exports contain only jobs with submit_lsn <= committed_lsn (or 0); at most one continuation in flight per standby; at most one rejoin reconciliation per process lifetime; truncation always precedes restart

Error Handling

Error Strategy

• Self-restart exec failure → ERROR log + abort() (FATAL) — a divergent ex-primary must never continue serving (locked constraint); under XCE_TESTING the fatal seam records instead of aborting
• Truncation failure during demotion/rejoin → ERROR, then restart anyway: boot replay of an untruncated log is reconciled at the next rejoin (the reconciliation loop converges); never proceed as a serving node past a failed convergence step
• Continuation-task endpoint gone at run time → clear in-flight flag and return (the heartbeat fallback re-evaluates) — no error surfaced
• Egress predicate is infallible (atomic load + compare); tagging failures cannot occur on the success path (tag set only after a successful append)
• Quorum logging is observability-only: WARN once per transition, never an error path, never gates the operation outcome (the gate's existing -XCE_ERR_NOT_READY behavior is unchanged)

Monitoring

• WARN logs: demotion truncation range; rejoin reconciliation range; quorum lost/restored with counts. INFO: continuation scheduling at TRACE/INFO consistent with existing streaming logs. Metrics export out of scope (umbrella).

Testing Strategy

Unit Tests (testsuite/, XCE_TESTING)

• test_demotion_convergence.c: become-follower from PRIMARY → hooks disabled before role store, truncation called with committed_lsn cutoff, WARN logged, ha_restart_hook invoked (1.1–1.3, 1.5, 1.6); exec-failure path invokes the fatal seam (1.4); become-follower from non-PRIMARY roles → no truncation/restart (regression); rejoin reconcile: tip > boundary → clamped cutoff max(boundary, local committed) + restart-once (2.1, 2.2); tip <= boundary → no action (2.3); second invocation in one lifetime → no second restart (2.1)
• test_egress_filter.c: live submit tags submit_lsn = written_lsn; replayed WAL_JOB_SUBMIT tags from entry LSN (3.3); job with submit_lsn > committed_lsn absent from collect_jobs_committed and from the JOB_NEW push list, present after committed_lsn advances (3.1, 3.2); submit_lsn == 0 exportable; plain collect_jobs (status path) returns the uncommitted job (3.4)
• test_convergence_verification.c: walAck behind target schedules exactly one continuation (hook-counted via ha_repl_stream_wal_hook/xce_wal_stream_request_hook), duplicate ACKs while in flight schedule none (5.1, 5.2); ACK at parity calls repl_complete_catchup and clears in_catch_up (7.1); heartbeat decision path byte-equivalent for snapshot decisions (5.3, 6.1 pin); repl_broadcast reaches all active standbys, skips only in_catch_up/empty (4.1 pin); quorum transition logged once across repeated gate rejections, restore logged on reconnection (8.1–8.3); dispatch gate admits status (OPCode_get) and replication opcodes without quorum, rejects request-shaped opcodes only (OPCode_jobsub/jobcontrol/queuecontrol/hostcontrol/jobdispatch/join) and admits fact-shaped opcodes (OPCode_notify/update/setpreason/clrpreason) per ha-commit-barrier-rollout's fact allowlist (9.1, 9.2 pin)

Integration Tests (testsuite/)

• Demotion seam end-to-end: simulated PRIMARY with appended uncommitted suffix sees higher term → truncation + restart hook; a fresh replay over the surviving log yields committed-only state (1.7, 2.4 in-process half)
• Rejoin reconciliation through the cluster.c intake: deliver a CtrlHelloReply with a boundary below the local tip → reconcile fires; boundary at/above tip → join proceeds untouched

E2E (live cluster, extends ha-wal-replication task 5.4 — REQUIRES ha-single-leader-role landed)

• 3-node via /xce-deploy-ha: submit confirmed jobs → kill primary → promoted standby owns all confirmed jobs; a submit forced to time out at the barrier converges to a single outcome on all members and is never retroactively reported successful; the demoted ex-primary restarts once and rejoins converged (2.4, 1.3, 2.1)
• 3-node: isolate primary → submits fail, status still served, quorum-loss WARN present; heal → quorum-restored WARN, submits succeed (8.1, 8.2, 9.1 live)

Performance & Scalability

• Egress predicate: one atomic load + compare per job per export — negligible against existing per-job serialization
• Continuation: at most one in-flight stream per standby; scheduling O(1) on iomgr; streaming cost unchanged (same ha_stream_wal_to_standby), just driven at ACK cadence instead of 3s heartbeats — catch-up latency improves by up to one heartbeat period per round
• Demotion/rejoin restart: process restart cost accepted by design (rare role/lifecycle events; connections migrate to the new primary anyway — umbrella-locked rationale)

Migration Strategy

This spec is phase 4 (final) of the umbrella's hard-ordered landing. It depends on ha-commit-engine (committed_lsn semantics) being live; it is parallel-safe with ha-commit-barrier-rollout and ha-single-leader-role except that the 3-node E2E requires ha-single-leader-role landed (direct promotion — the demotion path asserted here is the post-COMMITTING-removal one). Despite that parallel-safety, implementation in roadmap order is recommended regardless: all four sub-specs touch ha_dispatch_gate and raft_cb_become_follower in disjoint but adjacent regions, and in-order landing avoids anchor churn and merge coordination (per cross-spec review 2026-06-11). Landing order inside the spec: egress filter (pure additive) → demotion convergence + self-restart → rejoin reconciliation (reuses the primitive) → ACK-driven catch-up → quorum observability → verification suite → E2E. Rollback trigger per umbrella: controller testsuite regression or live-cluster E2E failure; each step leaves the system no less consistent than before it (filter defaults open via submit_lsn == 0; demotion/rejoin only ever discard above the committed boundary).