myxce/.kiro/specs/ha-convergence-recovery/research.md

Research & Design Decisions: ha-convergence-recovery

Date: 2026-06-11. Discovery type: integration-focused (extension of existing HA subsystem); the umbrella's research.md (deep-dive §C — five convergence gaps) is the primary discovery source. This log records the slice-level anchor verification pass and the design decisions local to this sub-spec. Umbrella-locked decisions are restated, not re-derived.

Umbrella-Locked Decisions (inputs, not open questions)

• Demotion eviction = truncate above committed_lsn + process SELF-RESTART via the existing boot recovery (persist_recover + replay_wal) — option (ii) of umbrella research §C, user-decided 2026-06-11. Restart-hook seam under XCE_TESTING; exec failure = FATAL (a divergent ex-primary must not keep serving). Truncation always precedes restart.
• commitIndex is volatile per Raft — rejoin reconciliation, NOT persisted committed_lsn.
• Provisional-state model with watermark egress filter: job_t.submit_lsn, scheduler export only; status/API queries intentionally unfiltered (documented deviation). No undo log, no MVCC, no committed-view machinery.
• ACK-driven catch-up continuation; heartbeat evaluation remains the fallback initiator and the snapshot-decision owner.
• Umbrella 6.1/6.3/6.4/7.3 are existing behaviors — verified by regression test, no new production code.

1. Anchor Verification Pass (2026-06-11, this worktree)

All file:line anchors from the umbrella audit re-verified by grep before being written into design/tasks. Drift from the umbrella's 2026-06-10 numbers was negligible.

Anchor	Verified location	Finding
Demotion path	raft_cb_become_follower ha.c:627-690	Confirmed gap: cancels timers, closes vote eps, sets CTRL_STANDBY — never truncates, never disables hooks (comment at ha.c:904 confirms hook disable lives only in ha_demote_to_standby)
Hook disable (unused on this path)	ha_demote_to_standby ha.c:1712-1726	Disables the three persistence hooks + resets hooks_registered; NOT called from raft_cb_become_follower
Hook registration	ha_promote_to_primary ha.c:1686-1707	Idempotency guard hooks_registered; symmetric disable exists
Truncation primitive	persist_truncate_wal_above persist.c:1306-1444	Reused as-is; stores committed_lsn = cutoff at persist.c:1438
RDHUP truncation pattern (to generalize)	ha.c:3416-3420 in ha_on_primary_ctrl_fd_rdhup	Truncates above local ha->last_known_committed_lsn
Rejoin signal point	cluster.c:283-298	Standby stores the primary-reported committed_lsn from CtrlHelloReply into ha->last_known_committed_lsn — the natural rejoin-reconciliation hook
handle_ctrlhello	config_repl.c:1368	Connection bookkeeping only; decisions live in the heartbeat loop
Boot recovery (reused by restart)	persist_recover persist.c:957, replay_wal persist.c:781	Replays everything above checkpoint_lsn — no committed gate (umbrella gap 4; rejoin reconciliation is the fix, by design)
Heartbeat-driven repl decision loop	ha_dispatch_repl_action_for_standby ha.c:1373-1478; invoked from stateReport branch at ha.c:2734	One action per heartbeat cycle per standby — confirms 6.2 gap
WAL ACK ingress (continuation hook point)	repl_ack_decode_and_dispatch ha.c:2632, walAck branch ha.c:2737-2754 → repl_on_ack repl.c:176	Runs on iomgr (peer_ctrl_msg_handler ha.c:2777); replication target already computed as max(committed_lsn, last_written) at ha.c:2727-2732
Catch-up streaming	ha_stream_wal_to_standby config_repl.c:1450-1583	Streams [standby_lsn+1, target] in one call, enqueue-only; xce_wal_stream_request_hook test seam exists
Catch-up state	repl_start_catchup repl.c:507, repl_complete_catchup repl.c:521	Discovered gap: repl_complete_catchup has NO production caller (in_catch_up set by eviction inline repl.c:240; documented protocol step 5 in config_repl.h:173 never wired). The ACK-driven continuation supplies the missing parity trigger (Req 7.1)
Quorum gate	ha_has_write_quorum ha.c:1507-1525, ha_dispatch_gate ha.c:1527+	No transition logging anywhere; rejection is silent -XCE_ERR_NOT_READY. Allowlist for degraded mode at ha.c:1538-1558 (Req 9 regression target)
standby_endpoints mutations (quorum-change points)	ha.c:531 (promotion), ha.c:3160-3166 (RDHUP removal), on_standby_connected ha.c:3764+	The three production sites where the quorum count changes
Scheduler export seams	collect_jobs dataex.c:294; scheduler full push xcesched_push_jobs sched.c:169/183 (on join, sched.c:609); incremental push sched_job_new_fun sched.c:236-248 → xcesched_push_joblist sched.c:14	Two scheduler-bound egress seams, not one — see decision D3
Unfiltered read paths (stay unfiltered)	apiserver.c:83, opget.c:173 (OPCode_get)	Both use collect_jobs; must NOT gain the filter (Req 3.4)
job_t	src/include/xce/types.h:483-485 — typedef struct { xce_job_t job; } job_t;	Controller wrapper around the wire type; submit_lsn goes on the wrapper, wire format untouched
Replay restore	apply_wal_entry persist.c:691 → restore_job job.c:71 (WAL_JOB_SUBMIT at persist.c:729-731)	restore_job does not currently receive the entry LSN — tagging at replay needs it passed through
Submit-path append	persist_job_hook_fun persist.c:1107-1125, persist_wal_sync_append persist.c:1003+ (written_lsn at 1018-1020)	Tagging point for live submits
Controller entry	main main.c:112	No saved argv, no execv anywhere in src/controller or src/lib — self-restart machinery is net-new
Hook chain semantics	xce_run_hooks hook.c:101-141	AFTER hooks all run in registration order; rc = last hook's rc; sched hooks registered at scheduler join (sched.c:616), persist hooks at promotion (ha.c:1692) — relative order is timing-dependent (input to D3)
E2E baseline	ha-wal-replication tasks.md task 5.4 (line 111)	Live 3-node test, /xce-deploy-ha, still unimplemented (blocked-manual); this spec's E2E extends it
Test seam pattern	ha_repl_stream_wal_hook ha.c:2104, ha_phase2_hook, xce_wal_stream_request_hook	Established XCE_TESTING function-pointer seam style for the restart hook

2. Slice-Level Design Decisions

D1 — Demotion convergence runs off the callback thread. raft_cb_become_follower can fire from vote handling (vtask) or peer message handling (iomgr). Truncation is file I/O and restart is process-fatal; per ha-async-io.md neither may run on iomgr. Decision: the become-follower callback detects PRIMARY→follower, disables persistence hooks inline (atomic, cheap — must happen before any further write), then schedules a one-shot demotion-convergence task (delaymgr, existing pattern) that truncates, WARN-logs the range, and self-restarts. The restart (exec) ends participation; any entry applied in the tiny window between truncation and exec is re-replayed at boot — harmless.

D2 — Self-restart mechanism. main() saves argv (new controller_saved_argv accessor in main.c) before argp_parse. New ha_self_restart() in ha.c: flush logs, execv(saved_argv[0], saved_argv); on exec failure log ERROR and abort() (FATAL — locked constraint). Under XCE_TESTING, ha_restart_hook function-pointer seam is invoked instead (and FATAL is asserted via a settable fatal hook rather than aborting the test binary). Implementation note: audit listener/connection fds for FD_CLOEXEC before exec (task bullet) so the re-exec'd image doesn't inherit live sockets.

D3 — Egress filter seams (refinement, not contradiction). The umbrella names collect_jobs (dataex.c) as the scheduler export path. Anchor verification found TWO scheduler-bound egress seams: (a) the collect-based full push (xcesched_push_jobs, on scheduler join) and (b) the incremental JOB_NEW push (sched_job_new_fun → xcesched_push_joblist), whose ordering relative to the persist hook (and hence the commit wait) is registration-timing-dependent. Both are "exports that drive scheduling decisions" (umbrella 2.4), so the same watermark rule applies at both. Decision: one shared predicate job_submit_committed(cluster_t*, job_t*) (submit_lsn == 0 || submit_lsn <= committed_lsn; exported from dataex.c next to the collect machinery), consumed by (a) a scheduler-bound collect wrapper in dataex.c and (b) a filter pass in the JOB_NEW push list. Status/API readers (apiserver.c, opget.c) keep calling unfiltered collect_jsons/collect_jobs — Req 3.4. Known accepted edge (documented deviation, matches umbrella 2.5 semantics): a submit that times out at the barrier but commits later is visible to the scheduler at the next scheduler (re)join full push — the requester was already told "failed"; never retroactively reported successful.

D4 — submit_lsn semantics. 0 = pre-feature/unknown → treated as committed (export allowed) so replayed legacy WALs and standby-restored jobs do not strand. Live submit: persist_job_hook_fun sets j->submit_lsn = written_lsn after append. Replay/standby apply: apply_wal_entry passes hdr->lsn through to restore_job, which tags the restored job. On the primary, replayed entries are ≤ the promotion truncation point (safe_lsn → committed_lsn), so restored jobs pass the filter by construction — matching the umbrella's "replay-restored jobs carry their entry LSN (≤ committed by definition)".

D5 — Rejoin reconciliation trigger and clamp. The reconciliation evaluates at the standby's CtrlHelloReply intake (cluster.c:283-298) — the moment the rejoining node learns the primary's committed boundary. Condition: local WAL tip (persist.current_lsn - 1) > primary-reported committed_lsn. Action: truncate above max(primary_reported_committed, local committed_lsn) (clamp — never truncate the locally known committed prefix; Req 2.2), WARN-log, self-restart once (reuses D2). "Once" is structural: after restart the tip equals the truncation boundary, so the condition cannot re-fire for the same divergence; a defensive in-process flag prevents re-entry within one process lifetime.

D6 — ACK-driven continuation with single-flight guard. Hook point: walAck branch of repl_ack_decode_and_dispatch (ha.c:2737-2754, iomgr). After repl_on_ack, if acked_lsn < target (target = max(committed_lsn, last_written), same derivation as ha.c:2727-2732), schedule a continuation task (delaymgr — streaming reads WAL files, must leave iomgr) that calls the existing ha_stream_wal_to_standby(ep, wal_path, acked_lsn, target). Per-standby catchup_inflight flag (set when scheduled, cleared when the task completes) prevents duplicate continuations (Req 5.2). At parity (acked_lsn >= target): call repl_complete_catchup — wiring the discovered missing production trigger for re-inclusion (Req 7.1); ha-commit-engine's wiring makes that call feed commit_on_membership_change. Heartbeat loop untouched as fallback + snapshot initiator (Req 5.3, 6.1).

D7 — Quorum transition logging. New helper ha_log_quorum_transition(ha) compares ha_has_write_quorum against an _Atomic int quorum_lost flag on ha_state_t and WARN-logs only on change ("HA quorum lost: connected=X required=Y" / "HA quorum restored: connected=X required=Y"). Called from the three standby_endpoints mutation sites (promotion registration, RDHUP removal, on_standby_connected) plus the dispatch-gate rejection path as a backstop. No per-rejection logging (Req 8.3).

3. Verification-of-Existing Findings (Reqs 4, 6, 7, 9)

• 6.1 full fan-out: repl_broadcast (repl.c:210+) iterates all active standbys, skipping only empty slots and in_catch_up — behavior present; regression test pins it.
• 6.3 snapshot fallback: HA_REPL_SEND_SNAPSHOT decision + ha_send_snapshot_to_standby (ha.c:1420-1443) present, including fresh-checkpoint-then-send; regression test pins the decision boundary (standby_lsn < oldest_wal_lsn).
• 6.4 re-inclusion: mechanism (repl_complete_catchup) present but trigger missing in production (see anchor table) — the regression test pins the mechanism; D6 supplies the trigger. This is the one "verification" item that needed a production wiring change, delivered inside the Req 5 component rather than as new scope.
• 7.3 degraded service: dispatch-gate quorum-loss allowlist (ha.c:1538-1558) admits heartbeat/replication opcodes + OPCode_get/probe; regression test pins the allowlist behavior for status + replication ops.

4. Risks

Risk	Mitigation
Self-restart loops (restart → rejoin → truncate → restart …)	Structural once-guard (D5); E2E asserts a single restart per divergence event
exec inherits live fds	CLOEXEC audit task; listeners re-created by boot path
Demotion convergence racing standby apply before restart	Hooks disabled first; truncate+exec in one task; post-truncation appends replay at boot (D1)
Egress filter starving the scheduler after primary boot	submit_lsn ≤ promotion truncation point for replayed jobs (D4); filter passes by construction once promoted
Continuation storms under bursty ACKs	Single-flight per-standby guard (D6); streaming remains enqueue-only

5. Design Synthesis Outcomes

• Generalization: demotion truncation and rejoin reconciliation are one capability ("converge local log to the cluster's committed boundary, then rebuild via boot recovery") with two triggers; both reuse persist_truncate_wal_above + one new ha_self_restart primitive.
• Build vs adopt: zero new libraries; all five workstreams reuse existing primitives (truncation, boot recovery, streaming, dispatch-gate allowlist, hook seam pattern). Net-new machinery is limited to saved-argv re-exec, the watermark predicate, the continuation scheduler, and the transition-logging helper.
• Simplification (locked): no in-place memory teardown (restart instead), no persisted commit index, no per-entry undo, no filtering of read paths.
• Design review gate: all 26 requirement IDs traced; no orphan components; boundary ↔ file-structure alignment verified.

---

Gap Re-Audit (2026-06-18)

Framing: POST-IMPLEMENTATION GAP AUDIT. Tasks 1-7 (18 sub-tasks) are complete and code-verified; one task remains open ([ ] Task 8 — live 3-node E2E). This pass re-verifies every EARS requirement against the on-disk code (codegraph + grep, 2026-06-18 worktree) and characterizes the single open task.

Verdict: PARTIAL — all 9 requirements (26 acceptance criteria) are IMPLEMENTED and unit/integration-tested in code. The only residual gap is the environment-dependent end-to-end verification of the cross-node behaviors (umbrella 2.4/8.1/8.2/9.1/9.2 over a real cluster), which is Task 8.

Requirement-to-Asset Map (re-verified 2026-06-18)

Req (criteria)	Asset (file:line)	Status	Notes
1.1 hooks disabled on demotion before anything else	raft_cb_become_follower ha.c:1202-1232 (prior_role == CTRL_PRIMARY && c != NULL branch reads role before store, disables hooks inline)	MET	guard at ha.c:1205; demotion scheduled at ha.c:1232
1.2 truncate above committed_lsn on demotion	ha_demotion_convergence_fn ha_convergence.c:144-181 → persist_truncate_wal_above(&c->persist, committed) ha_convergence.c:163	MET	committed read via atomic acquire ha_convergence.c:153
1.3 / 1.7 self-restart rebuilds via boot recovery	ha_self_restart ha_convergence.c:82-120 (execv(argv[0], argv) :111); boot path persist_recover/replay_wal reused	MET	truncation precedes restart (1.2 before :180)
1.4 exec failure → FATAL	ha_convergence.c:115-117 ERROR + abort(); no-saved-argv → abort ha_convergence.c:101-105	MET	controller never self-terminates on operational condition — this is crash-equivalent fatal, per locked constraint
1.5 WARN log with truncation range	ha_demotion_convergence_fn WARN with committed_lsn + prior_tip, ha_convergence.c:170-173	MET	prior_tip read under persist lock :157-161
1.6 XCE_TESTING restart seam	ha_restart_hook / ha_fatal_hook ha_convergence.c:84-96	MET	tests drive via these seams
2.1 rejoin tip>boundary → truncate+restart once	ha_rejoin_reconcile ha_convergence.c:271+ → ha_rejoin_reconcile_fn :240-269; _Atomic rejoin_reconciled once-guard	MET	test_rejoin_clamp_…_restarts_once, test_rejoin_second_call_no_second_restart
2.2 never truncate own committed prefix (clamp)	cutoff = max(primary_committed, local committed_lsn) (test asserts cutoff=6 when boundary=4, local committed=6)	MET	test_demotion_convergence.c:599-635
2.3 tip<=boundary → no action	ha_rejoin_reconcile early-return path, returns 0 no side effects	MET	test_rejoin_tip_at_or_below_boundary_no_action :642-665
2.4 single outcome, never retroactive success	demotion + rejoin convergence (in-process halves tested: append committed prefix + uncommitted suffix → demote seam → replay → committed-only). Cross-node end-to-end = Task 8	PARTIAL (in-process MET; live E2E = Task 8)	the only criterion split across unit + E2E
3.1 / 3.2 uncommitted submit excluded from scheduler export / included once committed	job_submit_committed egress.c:47-56 (submit_lsn==0 || submit_lsn<=committed_lsn); collect_jobs_committed egress.c:76-109	MET	sched.c:218 (full push) + sched.c:38 (JOB_NEW push) consume it
3.3 submit_lsn tagging live + replay	live: persist_job_hook_fun persist.c:2202-2220 (j->submit_lsn = submit_lsn); replay: restore_job(entry_lsn) from apply_wal_entry (hdr->lsn)	MET	tag set on local-write success incl. -ETIMEDOUT/-NOT_READY
3.4 status/API reads unfiltered	apiserver.c:83 + opget.c:173 use plain collect_jobs	MET	grep-confirmed: no read path calls the committed-only wrapper
4.1 full fan-out to all reachable standbys	repl_broadcast regression pin in test_convergence_verification.c (existing behavior)	MET	verification-only, no new production code
5.1 continuation on walAck, no heartbeat wait	walAck branch → ha_schedule_catchup_continuation config_repl.c:1784; body ha_catchup_continuation_fn :1739 streams [acked+1, target] on delaymgr	MET	re-derives target each round (extends to entries appended meanwhile)
5.2 single-flight per standby	_Atomic catchup_inflight[] CAS expected=0→1 config_repl.c:1839; cleared on every exit path :1772/:1791/:1811	MET	duplicate ACK while in flight schedules nothing
5.3 heartbeat loop stays fallback + snapshot initiator	ha_dispatch_repl_action_for_standby byte-unchanged (heartbeat path)	MET	snapshot decision ownership preserved
6.1 snapshot fallback for far-behind standby	repl_needs_snapshot repl.c:630 + HA_REPL_SEND_SNAPSHOT decision pin	MET	verification-only
7.1 re-inclusion at parity	repl_complete_catchup repl.c:602-628 (clears in_catch_up, calls commit_on_membership_change) — production caller now at parity branch config_repl.c:1831	MET	closed the "no production caller" gap from §1
8.1 / 8.2 / 8.3 quorum loss/restore logged once per transition with counts	ha_log_quorum_transition ha_convergence.c:412-452 (CAS edge-detect on quorum_lost, WARN once); 5 call sites ha.c:1085, 2076, 3969, 5056, gate backstop	MET	mirrors ha_has_write_quorum body (connected=repl_active_standby_count, required=n_peers/2)
9.1 / 9.2 status + replication served while quorum lost	ha_dispatch_gate degraded-mode allowlist regression pin (admits OPCode_get/heartbeat/repl/fact opcodes; rejects request-shaped)	MET	verification-only

Boundary audits re-confirmed (per Task 7 completion notes): commit.c/commit.h UNTOUCHED by the spec; status/API read paths UNFILTERED; non-PRIMARY raft_cb_become_follower behavior byte-equivalent (guarded by prior_role==CTRL_PRIMARY && c!=NULL). New TUs egress.c and ha_convergence.c registered in both controller_src and controller_unit_test_src (meson.build:6-8, 137-138); three new test files registered (meson.build:130-132).

The One Open Task — Task 8 (3-node E2E convergence test)

Nature: ENVIRONMENT-DEPENDENT INTEGRATION TEST — not code work. No production code or unit-test gap remains; every in-process half of Task 8's assertions is already covered by tasks 1-6 (single-outcome demotion+replay in test_demotion_convergence; rejoin clamp/once-guard; commit-gated egress in test_egress_filter; quorum-loss/restore WARN-once in test_convergence_verification). Task 8 exercises the cross-node, cross-process behaviors that a single-process unit binary structurally cannot:

1. Kill primary → promoted standby owns all confirmed jobs.
2. Force a submit to time out at the barrier (withhold standby ACKs) → entry committed on all members OR absent on all members; requester's failure answer never retroactively changed; scheduler never dispatched the unconfirmed job (2.4 end-to-end).
3. Restart the killed ex-primary → exactly one self-restart/reconciliation cycle, converged rejoin, no phantom jobs (1.3 + 2.1 over a real re-exec).
4. Isolate primary → submits fail, joblist/status still served, quorum-loss WARN; heal → quorum-restored WARN, submits succeed (8.1/8.2/9.1/9.2 live).

Prerequisites:

• A running 3-node cluster via /xce-deploy-ha (e.g. localhost:9991 9992 9993).
• ha-single-leader-role landed (direct promotion, CTRL_PRIMARY_COMMITTING removed) — confirmed satisfied in this worktree: raft_cb_become_leader ha.c:1026-1036 enters CTRL_PRIMARY directly ("ha-single-leader-role (Req 1.1): enter CTRL_PRIMARY directly"). So the dependency is already met; only the live infra is missing.
• Operator-observed assertions (log inspection for WARN transitions; joblist/status probes during isolation).

Effort: LOW-MEDIUM. The scenario script (deploy → kill/isolate/heal → assert) is mechanical; the cost is standing up and observing a live cluster and authoring the convergence assertions. Estimated a few hours of hands-on cluster time, no design or code uncertainty. Shares the exact blocked-manual status as ha-wal-replication task 5.4.

Risk: LOW for the spec's correctness claims (all logic is unit-verified). Residual risk is integration-surface only — re-exec inheriting an unexpected fd, scheduler/worker reconnection after the restart (explicitly out of scope here; worker-rejoin-resync roadmap backstop), or a timing artifact in the live barrier-timeout path. These are discoverable ONLY on real infra, which is precisely why Task 8 exists.

Options for closing Task 8

• Option A — Run the live E2E now (recommended when infra is available): execute /xce-deploy-ha 3-node, run the four scenarios, record converged-outcome + WARN-log assertions in completion notes. Closes 2.4/8.1/8.2/9.1/9.2 end-to-end and marks the spec fully complete. Effort LOW-MEDIUM, Risk LOW. Gated only on cluster availability + ha-single-leader-role (already landed).
• Option B — Accept as blocked-manual and ship the spec on unit coverage (current de-facto state): document Task 8 as deferred-to-infra, identical to ha-wal-replication 5.4. All production code is complete and regression-pinned; the umbrella's single-outcome guarantee is established in-process. Effort ZERO, Risk LOW-MEDIUM (integration-surface bugs would surface later, in a deploy/soak rather than in CI).
• Option C — Invest in automated multi-process E2E harness (out of current scope): build a scripted kill/isolate/heal driver around /xce-deploy-ha so this and ha-wal-replication 5.4 run in CI. Effort HIGH (new test infrastructure), Risk LOW. Pays off across the HA spec family but is a separate workstream, not part of this spec's task list.

Recommendation: Option A if a cluster can be stood up this session; otherwise Option B with an explicit deferral note (consistent with how the sibling ha-wal-replication 5.4 is tracked). Option C is the right long-term move for the HA family but belongs in a dedicated test-infra effort.

---

Live finding (2026-06-20): cluster "won't hold a primary" — candidate manifestation for the deferred Task 8 E2E

Surfaced during controller-status ctrlstatus live validation on a 3-node /xce-deploy-ha cluster. Logged here (not as a new spec) because the convergence-recovery production code is already complete (per the Options above) — so the symptom is a runtime instability to diagnose against this spec's domain, not missing implementation.

Observation

• wait-primary named ctrl-2 as primary in 42 ms, but seconds later ctrlstatus answered by ctrl-3 (standby) reported Primary: unknown, term: 3, then client failover exhausted ("cluster not ready after 21 attempts", E_XCE_NOT_READY). The cluster elected a primary then lost it / could not keep one serving.
• Matches the original /kiro-discovery logs (the motivating logs for the controller-status spec): repeated PreVote rounds, term bumps, "PreVote round timed out before a decision — treating as LOSS", "no primary served within 30000 ms" — recurring election churn, not a single clean failover.
• The controller unit suite (this session) shows the convergence machinery active and looping in-process: repeated HA quorum lost: connected=0 required=1, PRIMARY demoted (term=7) — scheduling demotion convergence (truncate above committed_lsn + self-restart), and HA rejoin reconcile: local tip exceeded committed boundary; truncated ... restarting once. The demotion path's self-restart is exactly the action that, if re-triggered in a loop, produces primary churn.

Differential (root cause NOT yet confirmed — needs live diagnosis)

1. Demotion self-restart churn (this spec's domain): quorum-loss → demote → truncate-above-committed → self-restart → rejoin → quorum-loss → … If a transient quorum flap re-enters demotion convergence repeatedly, the cluster never stabilizes a primary. Owned here.
2. PreVote / election thrash (ha-prevote / ha-single-leader-role): PreVote rounds timing out as LOSS and re-arming, term bumping without a stable leader. Owned upstream, not here.
3. Environment (fsync): on an overlayfs sandbox, fdatasync errno=5 blocks the durability/commit path (bug-795), so a primary may never advance committed and never stabilize. May be a sandbox artifact — the user's real-fs /tmp/xce-ha-deploy could behave differently. MUST be ruled out first by reproducing on a real fs.

Recommended next step

Run this spec's deferred Task 8 (Option A) on a real-fs cluster with instrumentation capturing, per node and per term: quorum enter/exit transitions, demotion-convergence entries, PreVote round outcomes, and committed-LSN advance. That evidence tells us whether the churn is convergence self-restart (close it here), election thrash (route to ha-prevote / ha-single-leader-role), or an fsync/env artifact (no code owner). Until that diagnosis runs, do NOT assume this is a convergence-recovery code defect — the code is present and unit-green; this is an integration/runtime observation.

Related

• controller-status ctrlstatus is the diagnostic tool here (answers in every role incl. fenced/no-quorum; term/commit/apply visible; responder timestamp for log correlation) — use it to watch the churn live.
• The failover-exhaustion error rendering crashed (xce_strerror_r global-buffer-overflow, bug-827, fixed in libxce utils.c) — rebuild before re-observing so the client prints a clean not-ready instead of aborting.

---

Gap Analysis (re-validation) — 2026-06-24

Framing: Third re-validation pass, verified against the on-disk code in this worktree (raft_tla). Primary question: is bug-1028 (post-failover availability — new primary stuck in catch-up loop, not serving) addressed in code or still open, and has the spec's status changed since the 2026-06-18/06-20 audits. Spec phase = tasks-generated, all three approval gates green, ready_for_implementation=true; roadmap.md:30 marks the spec [x] done.

Verdict: STILL PARTIAL on the spec's own task list (Task 8 live E2E remains the lone open item), but the bug-1028 availability concern is RESOLVED in code — fixed 2026-06-23 in the term-safety domain, NOT in this spec's self-restart machinery. Two of the three differentials raised in the 2026-06-20 live finding have since been closed by sibling specs. The convergence-recovery production code is intact and unchanged in substance; one anchor evolved materially (rejoin self-restart retired by ha-term-safety).

bug-1028 disposition (the headline question)

bug-1028 = "after a clean failover the new primary never becomes available to serve clients (wait-primary times out); standby[0] never converges." This is FIXED (2026-06-23) and the fix is live:

• Root cause (per buglog bug-1028): ha_stream_wal_to_standby (config_repl.c) stamped each streamed entry's term and the AppendEntries prev_log_term from the raw on-disk hdr.term, while the standby's logOk (raft_on_append_entries_request) and the library send path source terms from the authoritative per-LSN index (persist_entry_term). The term mismatch made logOk reject the catch-up stream forever → standby stuck in catch-up → new primary never reaches commit quorum → never serves. This is a term-safety / catch-up-correctness defect, distinct from bug-828 (data safety) and bug-854 (client discovery).
• Fix verified in code: ha_stream_wal_to_standby now takes const persist_t *p (config_repl.h:394-395, config_repl.c:2032-2033); all three former hdr.term source sites use p ? persist_entry_term(p, hdr.lsn) : hdr.term (config_repl.c:2125, 2143, 2197). Both production callers pass &c->persist: the heartbeat-driven streamer (ha.c:2734-2739, explicit "bug-1028" comment) and the ACK-driven continuation (config_repl.c:2283, this spec's Req 5 component). bug-1067 layered on top (leader-current-term AE header @8) for the cross-term catch-up case.
• Bearing on this spec: bug-1028's "stuck not serving" is the live manifestation that the 2026-06-20 finding flagged as possibly owned here (differential #1: demotion self-restart churn). It was NOT a convergence self-restart defect — it was the streamer's term source. This spec's Req 5 continuation consumed the same fix (its config_repl.c streamer call passes &c->persist), so the catch-up component is correct, but ownership of the bug sits with ha-term-safety / raft-log-replication.

Material drift since the 2026-06-18 audit (Constraint)

The 2026-06-18 map listed Req 2.1-2.3 rejoin reconciliation as MET via ha_rejoin_reconcile_fn (a delaymgr truncate+self-restart task). ha-term-safety task 3.2 RETIRED that (ha_convergence.c:254-280): ha_rejoin_reconcile is now a no-op that only retains the CtrlHelloReply hook for recording the learned boundary. Divergent-suffix reconciliation is now owned by the raft library's term-aware on_truncate_from (driven by raft_on_append_entries_request's (term,lsn) log-matching), which truncates exactly the divergent suffix and never a committed-prefix entry — replacing the old term-blind LSN-watermark truncate+restart. Until reconciled, the existing promotion-time fence keeps the member non-promotable WITHOUT self-restarting. This removes the rejoin self-restart loop (differential #1 / #2 of the 2026-06-20 live finding) as a churn source.

The demotion convergence half (Req 1.x) is UNCHANGED and still active: raft_cb_become_follower reads prior role (ha.c:1274), and on PRIMARY→follower disables hooks inline + WARN-logs + calls ha_schedule_demotion_convergence (ha.c:1277-1304) → ha_demotion_convergence_fn truncates above committed then ha_self_restart execv's (ha_convergence.c:144-180, 82-120). So a demoted ex-primary still self-restarts; a rejoining standby no longer does.

Requirement-to-Asset Map (delta only — full map in the 2026-06-18 section)

Req (criteria)	Status now	Note / gap tag
1.1-1.7 demotion truncate+self-restart	MET (unchanged)	ha_convergence.c:82-211; scheduled ha.c:1277-1304
2.1-2.3 rejoin reconcile	MET-by-different-mechanism	Constraint: self-restart path retired by ha-term-safety 3.2; now library on_truncate_from. Acceptance intent (converge a log running ahead, never lose committed prefix) still satisfied, by a stronger term-aware primitive
2.4 single outcome, never retroactive	PARTIAL	in-process MET; cross-node = Task 8
3.1-3.4 egress watermark filter	MET (unchanged)	egress.c:47-109; sched.c consumers
5.1-5.3 ACK-driven continuation	MET; also carries the bug-1028 fix	config_repl.c:2032+ streamer now term-correct via persist_entry_term; single-flight catchup_inflight[] ha.h:204
7.1 re-inclusion at parity	MET (unchanged)	repl_complete_catchup
8.1-8.3 quorum transition logging	MET (unchanged)	ha_log_quorum_transition ha_convergence.c:340+
9.1-9.2 degraded service	MET (unchanged)	dispatch-gate allowlist
Task 8 live 3-node E2E	OPEN (Missing — env-dependent)	only residual gap

Options A/B/C (for closing the residual gap)

• Option A — Run Task 8 live E2E now (recommended). Both former E2E blockers are now satisfied: ha-single-leader-role landed (direct CTRL_PRIMARY) AND bug-1028 (the very symptom Task 8 would have hit) is fixed, so a clean failover should now serve. Execute /xce-deploy-ha 3-node, run the four scenarios, record converged-outcome + WARN-log assertions. Closes 2.4/8.1/8.2/9.1/9.2 end-to-end. Effort LOW-MEDIUM, Risk LOW. Pairs naturally with /xce-test-ha soak which already exists in this worktree.
• Option B — Ship on unit coverage, defer Task 8 to infra (current de-facto state). All production code complete + regression-pinned; bug-1028 closed independently. Effort ZERO, Risk LOW-MEDIUM (integration-surface bugs surface in deploy/soak not CI). Consistent with sibling ha-wal-replication 5.4.
• Option C — Automated multi-process E2E harness in CI. Out of this spec's scope; the /xce-test-ha soak skill is the seed. Effort HIGH, Risk LOW; pays off across the HA family.

Effort / Risk

• Closing the spec: Effort S (Task 8 is mechanical scenario scripting, no code/design uncertainty), Risk Low (all logic unit-green; bug-1028 — the integration-surface risk that mattered — is already fixed).
• No new production work is indicated by this re-validation. The one "change" since 2026-06-18 (rejoin self-restart retirement) was delivered by ha-term-safety, not owed here, and strictly de-risks the availability story.

Recommendations for design phase

• Preferred: Option A. With bug-1028 fixed and ha-single-leader-role landed, run the deferred Task 8 against a real-fs /xce-deploy-ha cluster (or piggyback on /xce-test-ha) to convert 2.4/8.x/9.x from in-process MET to end-to-end MET, then mark the spec fully complete and the roadmap [x] is fully earned.
• Update spec text (low priority docs): annotate Req 2.1-2.3 in requirements/design to reflect that rejoin reconciliation is now the library's term-aware on_truncate_from (ha-term-safety 3.2), not a self-restart — so a future reader does not re-derive the retired delaymgr path. The acceptance intent is unchanged; only the mechanism moved.
• Research-Needed (carry forward): when Task 8 runs, capture per-node/per-term quorum transitions, demotion-convergence entries, and committed-LSN advance to confirm (a) no demotion self-restart churn under transient quorum flaps and (b) the rule-out of the overlayfs fdatasync errno=5 artifact (bug-795) on real fs. These were the open differentials from the 2026-06-20 live finding; bug-1028's fix retires the catch-up-loop differential, leaving only demotion-churn and fsync-env to confirm empirically.