myxce/.kiro/specs/ha-single-leader-role/research.md

Research & Discovery Log: ha-single-leader-role

Date: 2026-06-11. Discovery type: light/integration-focused — the umbrella spec (ha-consistency-invariants) completed full discovery (research.md, Option C phased hybrid, locked decisions); this log records the slice-specific verification pass: file:line anchors re-verified by grep against the working tree, the COMMITTING reference census that drives the sweep, and the upstream interface alignment (ha-commit-engine, ha-commit-barrier-rollout designs).

1. Locked Umbrella Decisions Consumed (not re-derived)

• Own-term commit vehicle: explicit WAL_NOOP entry appended at promotion (umbrella research item 4, DECIDED 2026-06-11) — bounded post-election write-unavailability independent of traffic; an idle cluster's new primary becomes writable as soon as the no-op commits.
• No wall-clock leadership/confirmation path: committing_timeout_fn force-promotion deleted without replacement (umbrella 1.5/3.3).
• HARD ORDERING: this spec lands only after ha-commit-barrier-rollout — the universal barrier must be in force before the transitional state is removed (umbrella migration phase 3; barrier-rollout design: "ha-single-leader-role must not land until the sweep is complete and audited").
• Commit state is volatile (Raft-conformant): writable_lsn lives in the commit engine, never persisted; a restart re-runs promotion or rejoins as standby.

2. Upstream Interface Alignment (verified against upstream designs)

ha-commit-engine (sub-spec 1) — consumed as-is

• commit_set_writable_lsn(commit_engine_t *ce, uint64_t noop_lsn) ships in include/commit.h with no caller in that spec ("API ships here; no caller in this spec — unset = no gate"). This spec is the designated caller.
• Engine invariant: "no waiter completes successfully while committed_lsn < its lsn or (gate set) committed_lsn < writable_lsn"; commit_submit returns -XCE_ERR_NOT_READY when "admission cap exceeded or own-term gate unmet at entry" — so umbrella 3.2's "barrier or NOT_READY" disjunction is engine-implemented; this spec only sets the gate.
• n_peers <= 1 self-quorum: self-ACK constitutes majority — a standalone node's promotion no-op commits immediately and the gate opens with no peer traffic (standalone behavior preserved).
• Engine instance is cluster_t.commit (c->commit), wired at cluster init; RETAIN registration mode exists for non-blocking entries.

ha-commit-barrier-rollout (sub-spec 2) — must not be disturbed

• Classification: wal_classify(wal_type, detail) totality is enforced by a fail-fast guard and a totality unit test over every wal_entry_type value. Their design pre-declares the row this spec adds: "WAL_NOOP (future) | promotion | internal | RETAIN | added downstream; guard forces classification". Adding the enum value without the classification row breaks their totality test — the row lands in the same task as the enum value.
• Dispatch gate: barrier-rollout adds a fact allowlist to the CTRL_PRIMARY no-write-quorum branch (OPCode_notify, OPCode_update, OPCode_setpreason, OPCode_clrpreason). Their design: "The CTRL_PRIMARY_COMMITTING branch is intentionally untouched (deleted wholesale by ha-single-leader-role — known residual until then)." This spec deletes ONLY the COMMITTING branch; the PRIMARY branch including the fact allowlist is preserved byte-for-byte.
• Reply ordering: every request-shaped handler already commit-first-then-reply; this spec adds no reply sites.

3. Verified Code Anchors (grep pass 2026-06-11 against working tree)

Audit numbers from the brief mostly held; drift noted inline. Re-verify with grep before editing — barrier-rollout landing first WILL shift ha.c line numbers (it inserts into the PRIMARY gate branch above the COMMITTING branch).

ha.h

Anchor	Verified	Notes
CTRL_PRIMARY_COMMITTING = 4 enum value	ha.h:64 (enum block 59-65)	brief said 59-65 ✓
HA_DEFAULT_COMMITTING_TIMEOUT_MS 60000	ha.h:131	delete
committing_timeout_ms config field	ha.h:141	delete
repl_ack_count field	ha.h:180	delete
committing_timeout_ctx field	ha.h:187	delete
ha_check_majority_convergence decl + comment	ha.h:396-412	delete
ha_repl_ack_received decl + comment	ha.h:626-630	delete
Stale comments mentioning COMMITTING	ha.h:214-230, 244-262, 373, 503-519, 607-614, 673-700, 791	comment sweep
KEEP: standby_reported_tag/lsn, standby_converged arrays	ha.h:191-193	still feed ha_process_standby_state catch-up decision loop + observability

ha.c (23 CTRL_PRIMARY_COMMITTING refs + machinery)

Anchor	Verified	Action
committing_timeout_ctx_t type + fwd decl	ha.c:64-69, 83	delete
raft_cb_vote_eligible role gate PRIMARY || COMMITTING	ha.c:210-211	collapse to PRIMARY
raft_cb_become_leader	ha.c:445-617 (brief said ~445-505; body extends through watchdog arm at 593-616)	rewrite (see design)
— sets COMMITTING	ha.c:505-508	→ direct CTRL_PRIMARY
— repl_ack_count reset	ha.c:511	delete
— convergence-array memsets	ha.c:517-519	KEEP (per-term reset still feeds catch-up loop)
— heartbeat start + watchdog arm	ha.c:590, 593-616	heartbeat stays; watchdog block deleted
raft_cb_become_follower committing-timeout cancel block	ha.c:640-648	delete block
ha_init repl_ack_count init + COMMITTING comment	ha.c:835, 843	delete/update
ha_do_promote (term increment + PRIMARY store)	ha.c:887-897	unchanged; production rewrite does NOT call it (raft owns term — avoid double increment; existing behavior already syncs term at become_leader top, ha.c:451)
ha_is_primary	ha.c:917-920 ✓	no code change — honest by construction once the state is gone; comment notes
heartbeat-reply convergence tracking comment	ha.c:1257, 1279-1282	ha_process_standby_state KEPT; comment sweep
ha_check_majority_convergence	ha.c:1295-1353 (brief said 1313-1353; comment starts 1295)	delete (incl. ha_promote_to_primary call at 1347, ha_do_promote test branch at 1351)
dispatch-gate COMMITTING branch	ha.c:1594-1612 ✓ (pre-barrier-rollout numbering)	delete branch only; PRIMARY branch + fact allowlist untouched
ha_should_redirect	ha.c:1655-1664 ✓	no code change (PRIMARY→0 already); fixed by state removal; comment
ha_promote_to_primary hook registration	ha.c:1686-1706 ✓	registration extracted to shared helper; fn retained for legacy/test callers (cluster.c:1121, test_ha_cluster.c, test_p3_manual_failover.c)
handle_request_vote (XCE_TESTING) role gates	ha.c:1984, 2004 (comments 1940, 1971)	collapse to PRIMARY
stale type/comment block	ha.c:2057-2060	comment sweep
ha_elect_evaluate_and_signal legacy COMMITTING block	ha.c:2247-2300 (brief said ~2248 ✓)	symmetric rewrite: direct PRIMARY; delete watchdog arm + repl_ack_count reset
repl_ack_decode_and_dispatch replCatchUpDone → ha_repl_ack_received	ha.c:2625-2627 (comment), 2659-2661 (call)	branch becomes INFO no-op (opcode still gate-allowed; standbys still send it)
heartbeat-reply ha_check_majority_convergence call	ha.c:2710-2715	delete call; rest of decision loop unchanged
peer_ctrl_msg_handler COMMITTING case label	ha.c:2773, 2789	drop label (falls to PRIMARY case)
committing_timeout_fn_for_test	ha.c:2950-2990 (region 2945-2995)	delete
vote_ep_err_handler role check	ha.c:3159	collapse to PRIMARY
committing_timeout_fn	ha.c:3080-3121 (brief said 3082-3125 ≈)	delete
ha_repl_ack_received	ha.c:3123-3141 (brief said 3126-3139 ≈)	delete
test-helper comment	ha.c:3044-3045	sweep
primary_heartbeat_fn re-arm/run role check	ha.c:3956	drop COMMITTING term

Other code

Anchor	Verified	Action
cluster.c config init .committing_timeout_ms = HA_DEFAULT_COMMITTING_TIMEOUT_MS	cluster.c:899	delete line
cluster.c ctrlhello comment "COMMITTING path"	cluster.c:1402-1410	comment update
raft.c comment "HA FSM stays in CTRL_PRIMARY_COMMITTING"	src/lib/raft/raft.c:123	comment update (raft-library code untouched otherwise)
wal_types.h enum	src/include/wal_types.h:44-58	add WAL_NOOP = 40 (gap 31-99 free; < WAL_ENTRY_TYPE_MAX=100, config.h:80)
apply_wal_entry type switch	persist.c:691-770 (switch at 728; default:break at 765)	explicit case WAL_NOOP: break; — LSN advance happens in common path (persist.c:719-721) so standbys ACK it
persist_wal_async (append + repl_broadcast at persist.c:358)	persist.c:295	reused as-is for the promotion no-op (rejects only WAL_CHECKPOINT)
wal_classify table (barrier-rollout)	persist.c (landed by sub-spec 2)	add WAL_NOOP → RETAIN row
meson.build controller_unit_test_src	src/controller/meson.build:74-131	remove deleted test files; add test_ha_role_lifecycle.c

COMMITTING reference census (sweep population — grep CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence)

Test fixtures (counts = matching lines at audit time):

File	Refs	Disposition
testsuite/test_ha_repl_ack.c	20 COMMITTING + 34 deleted-symbol	DELETE — subject is ha_repl_ack_received/repl_ack_count counting; valid concerns re-covered by test_ha_role_lifecycle.c
testsuite/test_ha_evaluate_committing.c	24 + 13 deleted-symbol	DELETE — subject is the COMMITTING evaluation path + committing_timeout_fn; direct-PRIMARY evaluation asserted in test_ha_role_lifecycle.c
testsuite/test_ha_role.c	37 (+16 deleted-symbol)	UPDATE — role-enum/transition expectations retargeted COMMITTING→PRIMARY
testsuite/test_ha_rdhup_integration.c	20 (+8)	UPDATE
testsuite/test_ha_repl_push.c	14	UPDATE
testsuite/test_ha_startup.c	12	UPDATE
testsuite/test_ha_connection_reuse.c	11 (+12)	UPDATE
testsuite/test_ha_start_election_async.c	7	UPDATE
testsuite/test_ha_vote_gate.c	6	UPDATE (vote-deny: PRIMARY-only deny expectations)
testsuite/test_config_repl.c	11 × cfg.committing_timeout_ms = 10000;	UPDATE — delete the field initializers (compiler-forced)

Documentation:

File	Site	Disposition
.kiro/steering/ha-async-io.md	:51 (heartbeat re-arm), :57 (two-tier/COMMITTING promotion), :68 (dispatch-gate table row), :79 (Raft-to-code mapping)	steering sync task
src/controller/CLAUDE.md	:40 (role mapping table)	sweep
.claude/skills/xce-deploy-ha/SKILL.md	:98 (expected log line "→ CTRL_PRIMARY_COMMITTING then → CTRL_PRIMARY")	sweep
src/docs/cluster-config-split-analysis.md	:34 (committing_timeout_ms in config listing)	sweep (historical doc — annotate or update listing)

4. Slice-Specific Findings

1. ha_is_primary and ha_should_redirect need zero code change. Both already test role == CTRL_PRIMARY only (ha.c:917-920, 1655-1664). The brief's "lies during the window"/"redirect loop" defects are properties of the COMMITTING state existing (role ≠ PRIMARY while leading); deleting the state fixes both by construction. The "simplification" is the collapse of every PRIMARY || COMMITTING compound check (4 production sites: ha.c:210, 1984, 2004, 3159, 3956, 2789) plus comment honesty.
2. Term double-increment hazard. ha_do_promote (ha.c:887) increments ha->term; production raft_cb_become_leader already syncs the raft-owned term at entry (ha.c:451). The old flow reached ha_do_promote later via convergence/watchdog → ha_promote_to_primary, producing a second increment beyond the raft term — a pre-existing wart that dies with the deleted callers. The rewrite stores CTRL_PRIMARY directly (no ha_do_promote) in the raft path; the legacy XCE_TESTING path keeps ha_do_promote semantics (it owns its term there).
3. OPCode_replCatchUpDone survives the deletion. Standbys still send it on catch-up completion and the dispatch gate still admits it; its only consumer on the promoted-vote-endpoint path was ha_repl_ack_received ack counting (ha.c:2661). The handler branch becomes a logged no-op; repl_complete_catchup (repl.c:521, replicator-registry path) is unrelated and unchanged.
4. Convergence tracking arrays stay. ha_process_standby_state (heartbeat stateReport) feeds the per-standby catch-up decision loop (ha_dispatch_repl_action_for_standby), which is unchanged by this spec; only its promotion consumer (ha_check_majority_convergence) is deleted. The per-term memsets in become_leader stay.
5. WAL_NOOP value/compatibility: WAL_NOOP = 40 sits below WAL_ENTRY_TYPE_MAX (100, config.h:80) so current standbys advance LSN and ACK it via the common path in apply_wal_entry; replay is a no-op via an explicit case. persist_wal_async appends AND broadcasts (repl_broadcast, persist.c:358) — one call does both at promotion. Standbys not yet registered in the replicator at promotion receive the entry via catch-up streaming of the uncommitted suffix (existing mechanism).
6. Engine registration of the no-op: after append, the promotion path registers the entry with commit_submit(ce, noop_lsn, COMMIT_MODE_RETAIN, 0) (non-blocking, observability) then calls commit_set_writable_lsn(ce, noop_lsn). Standalone (n_peers <= 1): engine self-quorum commits it immediately — gate opens with no peer traffic (umbrella 7.5 preserved).
7. meson test list (src/controller/meson.build:74-131) registers each test file individually — deleting test_ha_repl_ack.c / test_ha_evaluate_committing.c requires list edits, and test_ha_role_lifecycle.c must be added.

5. Risks

Risk	Mitigation
Line-number drift after barrier-rollout lands	every task instructs grep-before-edit; anchors here are 2026-06-11 pre-rollout
Missed COMMITTING reference (silent semantic residue)	dedicated grep-verification task gating completion (zero matches across repo excluding .git and spec history)
Deleting the gate branch disturbs barrier-rollout's fact allowlist	deletion task scoped to the case CTRL_PRIMARY_COMMITTING: block only; gate tests from barrier-rollout must stay green
Legacy XCE_TESTING path divergence from production	symmetric-update task with its own lifecycle assertions; both paths share the promotion helper
Election-win window with unset gate (crash between role store and commit_set_writable_lsn)	ordering inside the rewrite: gate set before the role store is observable to request traffic is NOT required — the engine rejects/holds via writable_lsn only after set; design orders noop-append + gate-set immediately at promotion, and until set the barrier still requires per-entry majority (entries of the new term); documented in design §Promotion Ordering

6. Design Synthesis Outcomes

• Generalization: one shared promotion sequence (role store → hook registration → heartbeat → noop append/broadcast → engine registration → writable gate) used by both the raft callback and the legacy evaluation path — no second promotion flavor survives.
• Build vs adopt: nothing new is built beyond WAL_NOOP; the gate, barrier, classification guard, broadcast, and catch-up are all consumed from upstream specs/existing code. Deletions dominate (net-negative LOC in ha.c).
• Simplification: ha_promote_to_primary survives only as the legacy/test-path wrapper around the shared helper; no compatibility shim for COMMITTING is kept anywhere (tests asserting it are rewritten or deleted, not stubbed).
• Review gate: all 8 requirement IDs traced to components; no orphan components; boundary↔file alignment verified against the census table.

---

Gap Re-Audit (2026-06-18)

Re-audit context: roadmap.md lists this spec as DONE, but spec.json.phase = tasks-generated and tasks.md shows 6 unchecked [ ] against 11 checked [x]. This pass determines whether the 6 open items are genuine remaining work or non-work artifacts, and re-verifies every EARS requirement against the live working tree (not the 2026-06-11 pre-implementation anchors in §3).

A. Verification method

• Repo-wide grep for the deleted-symbol set (CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence), excluding .git/.kiro/.wolf/.codegraph/builddir.
• Docs/steering grep over the task-6.1 target set.
• codegraph_explore of the promotion path, dispatch gate, role predicates, legacy path; direct grep of ha.c, ha.h, wal_types.h, persist.c, meson.build.
• git history of .kiro/specs/ha-single-leader-role/ and the controller tree.

B. Requirement-to-Asset map (verified against live code, 2026-06-18)

Req	EARS criterion	Asset (file:line)	Verdict
1.1	Election winner → PRIMARY directly, no transitional role	raft_cb_become_leader direct store ha.c:1034 (→ CTRL_PRIMARY log ha.c:1036); legacy ha_elect_evaluate_and_signal direct store ha.c:~2919 (→ CTRL_PRIMARY log)	MET
1.2	Single leader role; zero residual refs (code+tests+docs)	ctrl_role_t = 4 values ha.h:60-64 (no CTRL_PRIMARY_COMMITTING); repo-wide deleted-symbol grep = 0; docs/steering grep = 0	MET
1.3	Live primary processes via barrier, never redirects	ha_dispatch_gate PRIMARY branch ha.c:2067 (no COMMITTING case); ha_should_redirect PRIMARY→0 unchanged; barrier-rollout fact allowlist (notify/update/setpreason/clrpreason) preserved ha.c:~2099-2105	MET
1.4	Primary self-reports as primary pre-own-term-commit	ha_is_primary ha.c:1466-1470 tests PRIMARY only; promotion stores PRIMARY before/without waiting on noop commit	MET
1.5	No wall-clock leadership change	committing_timeout_fn/_for_test/committing_timeout_ctx_t deleted (grep 0); no watchdog armed in either promotion path; only ha_arm_own_term_gate (no timeout) on the tail	MET
2.1	No confirmation before an own-term entry commits; entry created at promotion	WAL_NOOP=40 wal_types.h:57; ha_arm_own_term_gate ha.c:2259-2286 appends WAL_NOOP → commit_submit(RETAIN) → commit_set_writable_lsn(noop_lsn); apply no-op persist.c:1381; replay no-op persist.c:2037	MET
2.2	Pre-own-term request: barrier or NOT_READY only, no third path	Gate COMMITTING-reject branch gone; PRIMARY branch returns 0 (→ barrier) or -XCE_ERR_NOT_READY (ha.c:~2105); no promotion-side completion path exists	MET
2.3	No timeout bypassing the own-term gate	watchdog/ack/convergence machinery deleted without replacement (grep 0); gate is committed_lsn >= writable_lsn only, engine-owned	MET

Requirements met: 8 / 8.

Supporting structural confirmations:

• ha_register_persist_hooks (ha.c:2223) + ha_arm_own_term_gate (ha.c:2259) are the shared promotion sequence; both the raft path (ha.c:1122,1134) and ha_promote_to_primary (ha.c:2305,2312) and the legacy path (ha.c:~2920) call them — single promotion flavor, no divergence.
• Raft path does NOT call ha_do_promote (term synced at entry); ha_promote_to_primary keeps ha_do_promote (owns term) — double-increment wart resolved per research §4 finding 2.
• Test disposition exact-match to plan: test_ha_role_lifecycle.c present + registered (meson.build:94); test_ha_repl_ack.c and test_ha_evaluate_committing.c deleted (absent on disk, absent from meson).
• barrier-rollout gate tests intact and asserting the preserved PRIMARY fact allowlist (test_commit_barrier_rollout.c fact_gate_* cases).

C. Open-task classification (the 6 unchecked [ ])

tasks.md uses a two-level checkbox scheme: integer rows (- [ ] 1.…6.) are phase group headers; the actual work lives in decimal sub-tasks (- [x] 1.1, 1.2, 2.1, …). All 11 decimal sub-tasks are [x]. The 6 unchecked items are EXACTLY and ONLY the 6 group headers.

[ ] item	Type	Evidence
1. WAL_NOOP entry type	GROUP-HEADER-ARTIFACT	sub-tasks 1.1 (6df1283), 1.2 (e6abd7c) both [x] + committed; WAL_NOOP live
2. Promotion rewrite	GROUP-HEADER-ARTIFACT	2.1 (b7a69af), 2.2 (c089fb6) [x] + committed; direct-PRIMARY + gate live
3. Legacy path symmetry	GROUP-HEADER-ARTIFACT	3.1 (b7b951b) [x] + committed; legacy path symmetric, live
4. Machinery deletions	GROUP-HEADER-ARTIFACT	4.1 (a5bf861), 4.2 (71ce41f) [x] + committed; grep 0
5. Fixture sweep + symbol removal	GROUP-HEADER-ARTIFACT	5.1 (7f974e7), 5.2 (93dfc02) [x] + committed; enum collapsed
6. Docs sync + grep verification	GROUP-HEADER-ARTIFACT	6.1 (dbc6316), 6.2 (e558a62) [x] + committed; docs grep 0, ASAN-green evidence in commit

Genuine gaps: 0. Deferred: 0. Group-header artifacts: 6/6.

The headers are unchecked purely because the kiro task generator does not auto-check parent rows when all children complete; they carry no independent deliverable. Task 6.2's commit (e558a62) explicitly records the final zero-match grep gate + ASAN regression green — the spec's own completion gate, which this re-audit independently reproduces (grep = 0 across code and docs).

D. Options for the open items (no genuine engineering work exists)

• Option A (recommended) — Administrative close-out. Tick the 6 group-header checkboxes and flip spec.json.phase to implemented/completed. Effort: trivial (~2 min, doc-only). Risk: none. Rationale: every child task is committed and independently verified; roadmap already reflects DONE.
• Option B — Leave as-is. Accept the cosmetic header/spec.json lag. Effort: zero. Risk: low-but-real — future /kiro-spec-status will keep reporting 6 "open" tasks and a tasks-generated phase, inviting repeat audits like this one (false-signal cost).
• Option C — Re-run a clean regression as fresh evidence. /xce-build controller (debug + ASAN + suite) to re-confirm green on the current tree, then close out as Option A. Effort: one build cycle (note research §Implementation-Notes 5.1: meson 300s cap is borderline — prefer the direct binary run; a Timeout: 1 is a wrapper artifact, not a failure). Risk: none; pure belt-and-suspenders. Recommended only if a green build artifact is required for sign-off beyond the existing task-6.2 commit evidence.

Effort/Risk for genuine work: N/A — there is no genuine remaining work. Closing recommendation: Option A.