myxce/.kiro/specs/ha-single-leader-role/design.md

Design: ha-single-leader-role

Sub-spec 3 of 4 of the ha-consistency-invariants umbrella. This design slices the umbrella design's HA Role Lifecycle component and role-lifecycle state diagram (.kiro/specs/ha-consistency-invariants/design.md), implementing umbrella Migration Strategy phase 3 ("Promotion no-op + own-term gate; then delete COMMITTING (+ watchdog, convergence checker); redirect fix"). Discovery is complete via the umbrella's research.md plus this spec's slice-verification pass (research.md here — verified anchors and the COMMITTING census). The commit engine interface (include/commit.h) is consumed exactly as shipped by ha-commit-engine — this spec is the designated first caller of commit_set_writable_lsn and never modifies engine internals. The classification table and dispatch-gate fact allowlist shipped by ha-commit-barrier-rollout are preserved untouched except for the additions/deletions they themselves pre-declared for this spec. Requirement IDs below are THIS spec's IDs (see requirements.md for the umbrella mapping). HARD ORDERING: implementation must not begin until ha-commit-barrier-rollout is implemented and green — removing the transitional state before the universal barrier is in force opens a window where a fresh primary accepts unreplicated writes.

Overview

Purpose: This feature gives cluster operators Raft's three-state leadership model: an election winner becomes CTRL_PRIMARY directly, post-election write safety comes from committing a WAL_NOOP entry of the current term through the commit engine's writable_lsn gate (Raft §5.4.2), and no wall-clock path can ever change leadership or confirm operations.

Users: Cluster operators stop seeing redirect loops to a primary that does not exist yet, dishonest "not primary" status from the node that won the election, and 60-second force-promotions that violate the commit rule with zero converged standbys. Downstream, ha-convergence-recovery gains the simplified lifecycle its E2E validation assumes.

Impact: Rewrites raft_cb_become_leader (direct PRIMARY, hook registration moved here, no-op append+broadcast, commit_set_writable_lsn); deletes CTRL_PRIMARY_COMMITTING and everything that existed only to serve it — the dispatch-gate COMMITTING branch, committing_timeout_fn + watchdog, ha_check_majority_convergence, repl_ack_count/ha_repl_ack_received; collapses every PRIMARY || COMMITTING compound check; updates the legacy XCE_TESTING election path symmetrically; sweeps every COMMITTING reference across ha.c, ha.h, cluster.c, raft.c comments, ten test fixtures, and four documentation files to grep-verified zero.

Goals

• Election winner assumes CTRL_PRIMARY directly; the role enum has exactly one leader value (1.1, 1.2)
• Promotion appends a WAL_NOOP of the current term and arms the engine's own-term gate; confirmation of any operation waits behind the no-op's commit (2.1, 2.2)
• Zero wall-clock paths that change leadership or confirm operations — watchdog and ack/convergence counters deleted without replacement (1.5, 2.3)
• A live primary never redirects and always self-reports as primary, even before its own-term entry commits (1.3, 1.4)
• Grep-verified zero references to the transitional state across code, tests, and docs (1.2)

Non-Goals

• Election/vote correctness, term management, freshness gates (raft-library — untouched except one stale comment)
• Commit-engine internals: gate enforcement mechanics, per-LSN evaluation, admission cap (ha-commit-engine — consumed)
• Operation classification, reply ordering, fact allowlist contents (ha-commit-barrier-rollout — preserved)
• Demotion truncate+restart, rejoin reconciliation, egress filter, ACK-driven catch-up, quorum logging (ha-convergence-recovery)
• Catch-up decision loop changes — ha_process_standby_state and ha_dispatch_repl_action_for_standby are unchanged

Boundary Commitments

This Spec Owns

• The promotion sequence on election win (both the raft callback and the legacy XCE_TESTING evaluation path) and the single shared promotion helper realizing it
• The WAL_NOOP entry type: enum value, append/broadcast at promotion, replay no-op, classification row
• The lifetime of CTRL_PRIMARY_COMMITTING and all machinery existing solely for it (watchdog, convergence checker, ack counters, gate branch, config knob) — i.e., their complete removal
• The role-predicate truth contract: ha_is_primary/ha_should_redirect/vote-deny/heartbeat re-arm semantics under the three-state model
• testsuite/test_ha_role_lifecycle.c and the disposition of every test fixture referencing the deleted state
• Synchronization of ha-async-io.md (dispatch-gate table, replication-consistency section, heartbeat re-arm, Raft mapping) and the other three documentation references

Out of Boundary

• commit.c / include/commit.h — consumed, never modified (the writable_lsn gate semantics are engine-owned)
• wal_classify dispatch mechanics and the fact allowlist in the gate's PRIMARY branch (ha-commit-barrier-rollout) — this spec adds the pre-declared WAL_NOOP row and deletes the pre-declared COMMITTING branch, nothing else in either
• raft_cb_become_follower demotion semantics beyond deleting its committing-timeout cancel block (truncate+self-restart is ha-convergence-recovery)
• Standby-side apply, ACK wire format, catch-up streaming (ha-wal-replication)
• ha_process_standby_state convergence bookkeeping and the per-standby replication decision loop (kept as-is; only its promotion consumer dies)

Allowed Dependencies

• commit_submit(ce, lsn, COMMIT_MODE_RETAIN, 0) and commit_set_writable_lsn(ce, noop_lsn) on cluster_t.commit (ha-commit-engine, P0)
• persist_wal_async append+broadcast primitive (persist.c:295; broadcasts via repl_broadcast at persist.c:358) and apply_wal_entry/replay_wal (P0)
• wal_classify classification seam + totality guard (ha-commit-barrier-rollout, P0)
• Existing promotion-adjacent machinery reused unchanged: safe-LSN truncation at become-leader, cluster_init_phase2, vote-endpoint promotion, config push, ctrlhello, ha_start_primary_heartbeat
• Threading rules per ha-async-io.md: raft_cb_become_leader runs where the raft FSM fires today (no new blocking; persist_wal_async and commit_submit(RETAIN) are non-blocking)
• Dependency direction: ha.c → persist.c append → commit engine; the engine calls back into nothing

Revalidation Triggers

• Any change to commit_set_writable_lsn semantics or the engine's NOT_READY contract (upstream) → re-verify promotion ordering and 2.2 behavior
• ha-convergence-recovery's demotion rewrite of raft_cb_become_follower → must not resurrect any transitional state; lifecycle test asserts the role set is closed
• Any new role added to ctrl_role_t → dispatch gate, vote gates, heartbeat re-arm, and the lifecycle test must be revisited
• New WAL entry types → classified by the barrier-rollout guard (unchanged trigger; WAL_NOOP row is now part of the table)

Architecture

Existing Architecture Analysis

• Production promotion today: raft_cb_become_leader (ha.c:445) truncates to safe LSN, runs phase2, stores CTRL_PRIMARY_COMMITTING, resets repl_ack_count, promotes vote endpoints, pushes config + ctrlhello, starts heartbeat, arms committing_timeout_fn (60s force-promote). Full PRIMARY is reached later by one of three paths: ha_check_majority_convergence (heartbeat stateReports), ha_repl_ack_received (replCatchUpDone counting), or the watchdog — all three call ha_promote_to_primary (term++ via ha_do_promote, hook registration, heartbeat re-arm)
• Defects removed (verified, brief + umbrella audit): redirect loop while COMMITTING (ha_should_redirect ≠ PRIMARY → redirect to a primary that doesn't exist yet); ha_is_primary false on the elected node; watchdog promotion with zero converged standbys violating the commit rule; double term increment (raft-synced term at become-leader + ha_do_promote's term++ at convergence)
• The replacement guarantee already exists upstream: every confirmation path goes through the commit engine (barrier-rollout), and the engine withholds completion until committed_lsn >= writable_lsn once the gate is set — the transitional state's only legitimate purpose (don't confirm until converged) is provided correctly by Raft's own-term rule
• Legacy XCE_TESTING election path (ha_elect_evaluate_and_signal, ha.c:2141) mirrors the COMMITTING block (ha.c:2247-2300) and must be updated symmetrically — tests drive lifecycle through it

Architecture Pattern & Boundary Map

graph TB
    subgraph raft_library
        RAFT[raft FSM<br/>on_become_leader]
    end
    subgraph ha_role_lifecycle[ha.c role lifecycle - this spec]
        BL[raft_cb_become_leader<br/>rewritten]
        HELPER[ha promotion sequence<br/>shared helper]
        LEGACY[ha_elect_evaluate_and_signal<br/>XCE_TESTING path]
        GATE[ha_dispatch_gate<br/>COMMITTING branch deleted]
        PRED[role predicates<br/>is_primary, should_redirect, vote gates]
    end
    subgraph persist_repl[persist.c / repl.c - consumed]
        NOOP[persist_wal_async WAL_NOOP<br/>append + broadcast]
        CLS[wal_classify<br/>WAL_NOOP → RETAIN row]
    end
    subgraph commit_engine[commit engine - consumed]
        REG[commit_submit RETAIN]
        WL[commit_set_writable_lsn]
    end
    STBY[standbys ACK]

    RAFT --> BL
    BL --> HELPER
    LEGACY --> HELPER
    HELPER -->|CTRL_PRIMARY + hooks + heartbeat| PRED
    HELPER --> NOOP
    NOOP --> CLS
    HELPER --> REG
    HELPER --> WL
    NOOP --> STBY
    STBY -->|walAck ≥ noop_lsn| commit_engine

Architecture Integration (sliced from umbrella):

• Selected pattern: one shared promotion sequence with two entry points (raft callback, legacy evaluation) — no second promotion flavor survives; write-safety is delegated entirely to the engine gate rather than a role sub-state
• Existing patterns preserved: safe-LSN truncation, phase2 ordering, endpoint promotion, config push + ctrlhello, heartbeat arming, catch-up decision loop, errnox early-return style, barrier-rollout's gate allowlist
• Removal rationale (umbrella): the lifecycle diagram has no COMMITTING node; "no other component may observe COMMITTING after this spec"
• Steering compliance: no blocking added on iomgr or in the raft callback; TDD with testsuite/ tests first; ha-async-io.md synchronized as part of this spec

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Backend / Services	C (gnu17), stdatomic	promotion rewrite, deletions	no new dependencies, no .capnp changes
Data / Storage	existing WAL (persist.c)	WAL_NOOP = 40 entry, empty payload	on-disk format unchanged (new type value only, below WAL_ENTRY_TYPE_MAX)
Build / Test	meson, testsuite/ pattern	test_ha_role_lifecycle.c; two test files deleted	XCE_TESTING seams

File Structure Plan

Directory Structure

src/controller/
├── ha.c                                # MODIFIED — promotion rewrite, all deletions, predicate collapse
├── ha.h                                # MODIFIED — enum value, fields, defaults, decls removed; comments
├── cluster.c                           # MODIFIED — committing_timeout_ms init removed; comment sweep
├── persist.c                           # MODIFIED — apply_wal_entry WAL_NOOP case; wal_classify row
├── meson.build                         # MODIFIED — test list: -test_ha_repl_ack.c, -test_ha_evaluate_committing.c,
│                                       #            +test_ha_role_lifecycle.c
└── testsuite/
    ├── test_ha_role_lifecycle.c        # NEW — direct promotion, noop+gate, predicate honesty, no watchdog
    ├── test_ha_repl_ack.c              # DELETED — subject machinery removed
    ├── test_ha_evaluate_committing.c   # DELETED — subject machinery removed
    └── (8 fixtures)                    # MODIFIED — COMMITTING expectations retargeted (census in research.md)
src/include/wal_types.h                 # MODIFIED — WAL_NOOP = 40
src/lib/raft/raft.c                     # MODIFIED — stale comment only (line 123)
.kiro/steering/ha-async-io.md           # MODIFIED — steering sync (4 sites)
src/controller/CLAUDE.md                # MODIFIED — role mapping table (line 40)
.claude/skills/xce-deploy-ha/SKILL.md   # MODIFIED — expected log line (line 98)
src/docs/cluster-config-split-analysis.md # MODIFIED — config listing (line 34)

Modified Files (what and why)

• src/controller/ha.c — the promotion rewrite and every deletion (verified anchor table in research.md §3); net-negative LOC
• src/controller/ha.h — remove CTRL_PRIMARY_COMMITTING, HA_DEFAULT_COMMITTING_TIMEOUT_MS, committing_timeout_ms, repl_ack_count, committing_timeout_ctx, decls of the two deleted functions; comment sweep; standby_reported_*/standby_converged arrays stay
• src/controller/cluster.c — drop .committing_timeout_ms initializer (cluster.c:899); update the ctrlhello comment (cluster.c:1402-1410)
• src/controller/persist.c — apply_wal_entry gains case WAL_NOOP: break; (LSN-advance/ACK via the existing common path); wal_classify gains the WAL_NOOP → WAL_MODE_RETAIN row pre-declared by barrier-rollout
• src/include/wal_types.h — WAL_NOOP = 40 with comment (internal, empty payload, replay no-op, promotion-only)
• src/controller/testsuite/test_config_repl.c — delete the 11 cfg.committing_timeout_ms initializers (compiler-forced once the field is gone)

System Flows

Promotion (after this spec) — raft_cb_become_leader

sequenceDiagram
    participant RF as raft FSM
    participant BL as raft_cb_become_leader
    participant P as persist
    participant CE as commit engine
    participant S as standbys
    RF->>BL: on_become_leader(term)
    BL->>BL: sync term, truncate to safe_lsn, phase2 (existing)
    BL->>BL: role = CTRL_PRIMARY (direct), register persist hooks
    BL->>BL: promote vote eps, push config, ctrlhello, start heartbeat (existing)
    BL->>P: persist_wal_async(WAL_NOOP, len=0) → lsn=N, broadcast
    BL->>CE: commit_submit(N, RETAIN, 0)
    BL->>CE: commit_set_writable_lsn(N)
    S-->>CE: walAck acked_lsn >= N (live or via catch-up)
    CE->>CE: committed_lsn >= N → own-term gate open
    Note over CE: request waiters complete only when<br/>committed_lsn >= max(their lsn, N)

Promotion ordering decisions: hooks are registered BEFORE the no-op append so the persistence seam is live for any racing client write (their entries land after the no-op in LSN order and wait behind it — 2.1). The gate is set immediately after the append; in the sub-microsecond window before commit_set_writable_lsn returns, request-shaped entries still cannot be falsely confirmed — their own commits require per-entry majority of new-term entries, which transitively covers the no-op (single-writer contiguous advancement). Standalone (n_peers <= 1): the engine's self-quorum commits the no-op at registration and the gate opens immediately — standalone promotion latency is unchanged (umbrella 7.5). No watchdog of any kind is armed (1.5, 2.3).

Role lifecycle (umbrella diagram, the contract this spec realizes)

stateDiagram-v2
    [*] --> INIT
    INIT --> STANDBY: heartbeat seen
    INIT --> ELECTING: timeout
    ELECTING --> PRIMARY: quorum won, truncate to safe_lsn, append noop
    ELECTING --> STANDBY: lost or higher term
    PRIMARY --> RESTART: higher term seen (ha-convergence-recovery)
    STANDBY --> ELECTING: primary lost
    RESTART --> INIT: process self restart via boot recovery

No COMMITTING node exists; the PRIMARY → RESTART demotion edge is owned by ha-convergence-recovery (this spec only deletes the committing-timeout cancel block inside raft_cb_become_follower, leaving its current demote-to-STANDBY behavior otherwise as-is).

Pre-own-term request handling (2.2)

A state-changing request arriving after the role store but before the no-op commits takes exactly one of two paths, both engine-owned: (a) through the barrier — the handler's commit_submit(WAIT)/persist_commit_barrier blocks until committed_lsn covers both the entry and writable_lsn, confirming only on real majority; or (b) -XCE_ERR_NOT_READY fast rejection (admission cap, or gate-unmet engine policy), surfaced to the requester via the existing errnox reply path. No third path exists because the dispatch gate's COMMITTING rejection branch is gone and handlers have no other completion mechanism (barrier-rollout sweep). Status queries and replication traffic are unaffected: the PRIMARY gate branch (including the fact allowlist) admits them as today (1.3, 1.4).

Requirements Traceability

Requirement	Summary	Components	Flows
1.1	election winner → PRIMARY directly	Promotion Rewrite, Legacy Path Symmetry	promotion
1.2	single leader role; zero residual references	Transitional-State Deletions, Reference Sweep	—
1.3	primary processes via barrier, never redirects	Role Predicate Simplification (ha_should_redirect fixed by state removal), Transitional-State Deletions (gate branch)	pre-own-term handling
1.4	status reports primary pre-own-term-commit	Role Predicate Simplification (ha_is_primary honest by construction)	promotion
1.5	no wall-clock leadership change	Transitional-State Deletions (watchdog + counters)	—
2.1	no confirmation before own-term entry commits	WAL_NOOP Entry, Promotion Rewrite (commit_set_writable_lsn)	promotion
2.2	pre-own-term requests: barrier or NOT_READY only	Promotion Rewrite (gate armed), engine contract (consumed)	pre-own-term handling
2.3	no timeout bypassing the own-term gate	Transitional-State Deletions (watchdog deleted without replacement)	—

Components and Interfaces

Component	Layer	Intent	Req Coverage	Key Dependencies	Contracts
WAL_NOOP Entry	wal_types.h, persist.c	own-term commit vehicle: empty entry, replay no-op	2.1	persist append/broadcast (P0), wal_classify (P0)	State
Promotion Rewrite	controller/ha.c	direct PRIMARY + hooks + noop + gate, one shared sequence	1.1, 1.4, 2.1, 2.2	raft callback (P0), commit engine (P0), WAL_NOOP (P0)	State
Transitional-State Deletions	controller/ha.c, ha.h, cluster.c	remove COMMITTING and all machinery serving it	1.2, 1.3, 1.5, 2.3	—	State
Role Predicate Simplification	controller/ha.c	collapse compound role checks; honest predicates	1.3, 1.4	—	Service
Legacy Path Symmetry	controller/ha.c (XCE_TESTING)	ha_elect_evaluate_and_signal mirrors production promotion	1.1, 1.5	shared promotion sequence (P0)	State
Reference Sweep	tests, docs, steering	grep-verified zero COMMITTING references	1.2	census (research.md §3)	—
Role Lifecycle Tests	testsuite/test_ha_role_lifecycle.c	pin the new lifecycle contract	all	XCE_TESTING seams	—

WAL_NOOP Entry (src/include/wal_types.h, src/controller/persist.c)

Field	Detail
Intent	A current-term log entry with no state-machine effect, existing purely to satisfy Raft §5.4.2
Requirements	2.1

Responsibilities & Constraints

• WAL_NOOP = 40 in wal_entry_type (wal_types.h:44-58): below WAL_ENTRY_TYPE_MAX (100) so existing standby apply advances current_lsn and ACKs it through the common path (persist.c:719-721); the value gap 31–99 is otherwise free
• payload_len = 0; appended only by the promotion sequence via persist_wal_async(p, WAL_NOOP, NULL, 0) — one call appends, fsync-schedules, and broadcasts (repl_broadcast, persist.c:358)
• Replay/apply: explicit case WAL_NOOP: break; in apply_wal_entry's type switch (persist.c:728-765) — no entity restore, no hooks; boot replay and standby apply are byte-identical no-ops
• Classification: wal_classify(WAL_NOOP, *) → WAL_MODE_RETAIN — the row barrier-rollout pre-declared ("guard forces classification"); the enum value and the row land in the same change so the totality test never breaks
• One entry per election win; no compaction concern (checkpointing treats it like any committed entry)

Contracts: State [x] — the entry's LSN is the term's writable_lsn; committed_lsn >= noop_lsn ⇔ own-term condition met

Promotion Rewrite (ha.c — raft_cb_become_leader + shared sequence)

Field	Detail
Intent	One promotion sequence: ELECTING → PRIMARY directly with hooks, heartbeat, no-op, and gate
Requirements	1.1, 1.4, 2.1, 2.2

Responsibilities & Constraints

• raft_cb_become_leader (ha.c:445-617) keeps unchanged: term sync from raft, ha_compute_safe_committed_lsn + persist_truncate_wal_above, last_known_committed_lsn/committed_config_tag stores, first-boot cluster_init_phase2, per-term convergence-array memsets (still feed the catch-up loop), vote-endpoint promotion, config push, ctrlhello, ha_start_primary_heartbeat
• Replaces the COMMITTING store (ha.c:505-508) with a direct CTRL_PRIMARY store + INFO("HA FSM (raft): → CTRL_PRIMARY ..."); deletes the repl_ack_count reset (ha.c:511) and the entire watchdog-arm block (ha.c:593-616)
• Hook registration moves here (from ha_promote_to_primary, ha.c:1691-1698): extracted into a static helper (ha_register_persist_hooks(cluster_t *c) with the existing hooks_registered idempotency guard) called by both the rewrite and the surviving ha_promote_to_primary; registration precedes the no-op append so the persistence seam is live before any client write can race
• Appends the no-op and arms the gate (production, c != NULL): persist_wal_async(WAL_NOOP) → derive noop_lsn = current_lsn - 1 under p->lock (same pattern as persist.c:1019-1021) → commit_submit(&c->commit, noop_lsn, COMMIT_MODE_RETAIN, 0) → commit_set_writable_lsn(&c->commit, noop_lsn). Under XCE_TESTING with no cluster/persist context, the append and gate are skipped (gate unset = no gate, engine default) — lifecycle tests assert role/predicate behavior; gate behavior is asserted with a real engine instance
• No ha_do_promote call in this path: the raft-owned term was already synced at entry (ha.c:451); the legacy double-increment wart dies with the deleted convergence callers
• ha_promote_to_primary (ha.c:1686-1706) survives for its non-raft callers (cluster.c:1121 init-time, legacy path, tests): body becomes ha_do_promote + ha_register_persist_hooks + ha_start_primary_heartbeat + the same noop+gate tail (guarded on persist/engine availability) so every road to PRIMARY arms the own-term gate

Dependencies

• Inbound: raft FSM on_become_leader (P0); legacy evaluation path (P0, XCE_TESTING)
• Outbound: persist append/broadcast (P0); commit engine registration + gate (P0)

Contracts: State [x]

• Postconditions: role == CTRL_PRIMARY; persistence hooks registered; heartbeat armed; (production) a WAL_NOOP of the current term is appended+broadcast and writable_lsn == noop_lsn
• Invariants: no path sets any role value other than {PRIMARY, STANDBY, INIT, ELECTING}; promotion never blocks on the no-op committing (1.4 — status honesty does not wait)

Implementation Notes

• Integration: cluster_init_phase2 call point unchanged; the noop append happens after endpoint promotion so the broadcast reaches the freshly promoted standby endpoints; standbys not yet in the replicator registry receive the entry via the existing uncommitted-suffix catch-up streaming
• Validation: lifecycle test asserts no observable role transition through any intermediate value (atomic store — single transition ELECTING→PRIMARY)
• Risks: none new at runtime — the window "role==PRIMARY, gate not yet set" cannot falsely confirm (see §Promotion ordering decisions)

Transitional-State Deletions (ha.c, ha.h, cluster.c)

Field	Detail
Intent	Remove the state and every mechanism that existed only to serve it — no replacements, no shims
Requirements	1.2, 1.3, 1.5, 2.3

Deletion list (verified anchors in research.md §3; grep before editing):

• CTRL_PRIMARY_COMMITTING enum value (ha.h:64) — final removal step, compiler-enforced sweep
• Dispatch-gate COMMITTING branch (ha.c:1594-1612) — the case CTRL_PRIMARY_COMMITTING: block ONLY; the PRIMARY branch, including barrier-rollout's fact allowlist (OPCode_notify/update/setpreason/clrpreason under no-quorum), is preserved byte-for-byte
• committing_timeout_fn (ha.c:3080-3121), committing_timeout_fn_for_test (ha.c:2950-2990), committing_timeout_ctx_t (ha.c:64-69, fwd decl :83), watchdog arming in become-leader (ha.c:593-616) and in the legacy path (ha.c:2275-2299), the cancel block in raft_cb_become_follower (ha.c:640-648), committing_timeout_ctx field (ha.h:187), committing_timeout_ms knob (ha.h:131, :141; cluster.c:899; test_config_repl.c ×11)
• ha_check_majority_convergence (ha.c:1295-1353; decl ha.h:396-412) and its call site in the heartbeat-reply path (ha.c:2710-2715)
• repl_ack_count (ha.h:180; resets ha.c:511, :835, :2255) and ha_repl_ack_received (ha.c:3123-3141; decl ha.h:626-630); its OPCode_replCatchUpDone call site (ha.c:2659-2661) becomes a logged no-op — the opcode remains gate-allowed (standbys still send it; repl_complete_catchup on the replicator path is unrelated and unchanged)
• KEPT deliberately: standby_reported_tag/lsn, standby_converged, ha_process_standby_state (catch-up decision loop + observability); the per-term memsets in become-leader

Contracts: State [x] — after this spec, ctrl_role_t has exactly four values and no component, test, or document observes a fifth

Role Predicate Simplification (ha.c)

Summary-only: collapse every role == CTRL_PRIMARY || role == CTRL_PRIMARY_COMMITTING compound to the PRIMARY check — raft_cb_vote_eligible (ha.c:210-211), handle_request_vote XCE_TESTING gates (ha.c:1984, 2004), vote_ep_err_handler (ha.c:3159), peer_ctrl_msg_handler case labels (ha.c:2789), and the heartbeat re-arm/run condition in primary_heartbeat_fn (ha.c:3956 — drops the COMMITTING term, satisfying the umbrella's "heartbeat re-arm condition drops COMMITTING"). ha_is_primary (ha.c:917-920) and ha_should_redirect (ha.c:1655-1664) need zero code change — both already test PRIMARY only; deleting the state makes them honest by construction (1.4: the elected node's role IS primary; 1.3: a live primary can never satisfy the redirect condition). Vote denial semantics are preserved exactly: a node that won the election denies same-term challengers as PRIMARY (it previously denied as COMMITTING). Comments updated at each site.

Legacy Path Symmetry (ha.c — ha_elect_evaluate_and_signal, XCE_TESTING)

Summary-only: the legacy election evaluation's CTRL_PRIMARY branch (ha.c:2247-2300) is updated symmetrically — direct CTRL_PRIMARY store (replacing the COMMITTING store at 2248-2252), repl_ack_count reset deleted (2255), config-push/ctrlhello test hooks kept, watchdog-arm block deleted (2275-2299). Promotion side effects route through the same shared sequence (ha_promote_to_primary / ha_register_persist_hooks) so test-driven and raft-driven promotion cannot diverge; the noop+gate tail is guarded on persist/engine availability as in production. Existing tests that drove COMMITTING through this path are retargeted in the sweep (1.1, 1.5).

Reference Sweep (tests + documentation)

Summary-only: the census in research.md §3 is the authoritative population. Test fixtures: delete test_ha_repl_ack.c and test_ha_evaluate_committing.c (their subjects are the deleted machinery; surviving concerns — promotion on quorum, no-promotion-without-majority — are re-pinned in test_ha_role_lifecycle.c against the new mechanism); update test_ha_role.c, test_ha_startup.c, test_ha_rdhup_integration.c, test_ha_repl_push.c, test_ha_connection_reuse.c, test_ha_start_election_async.c, test_ha_vote_gate.c (COMMITTING expectations → PRIMARY), and test_config_repl.c (drop the deleted config field); update meson.build. Documentation: ha-async-io.md (heartbeat re-arm :51, two-tier/promotion :57, dispatch-gate table :68, Raft mapping :79 — rewritten to the three-state model + own-term gate), src/controller/CLAUDE.md:40, .claude/skills/xce-deploy-ha/SKILL.md:98 (expected log becomes → CTRL_PRIMARY once), src/docs/cluster-config-split-analysis.md:34. Completion is grep-verified: zero matches for CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence outside .git/ and .kiro/specs/ history (1.2).

Data Models

Domain Model (changes only)

• ctrl_role_t: 5 values → 4 (CTRL_PRIMARY, CTRL_STANDBY, CTRL_INIT, CTRL_ELECTING); enum numbering of survivors unchanged (no persisted role values exist — role is volatile)
• WAL entry: new type WAL_NOOP = 40, header-only (payload_len 0); wire/disk format otherwise unchanged
• ha_state_t: − repl_ack_count, − committing_timeout_ctx; ha_config_t − committing_timeout_ms
• writable_lsn (engine-owned, set by this spec): volatile, never persisted (Raft-conformant; a restarted node re-promotes or rejoins)
• Invariants: committed_lsn >= writable_lsn ⇔ the current term has a committed entry; exactly one leader role value exists

Error Handling

Error Strategy

• Promotion no-op append failure (persist_wal_async < 0, e.g. disk error): ERROR log; no LSN exists, so the gate stays unset for this term and promotion completes. This is degraded-but-safe: with the gate unset the engine applies no own-term hold, but every confirmation still requires per-entry majority of a new-term entry (universal barrier) — the first state-changing entry of this term becomes the de-facto own-term vehicle (2.2's "process through the barrier" arm). A WARN documents the condition; a node that cannot write its WAL at all fails operations through the existing persist error paths
• Pre-own-term request rejection: -XCE_ERR_NOT_READY via the engine (distinct log message, retriable) — unchanged engine contract, no new codes
• No new timeout semantics anywhere: the only deadline that can fail an operation is the commit timeout owned by the barrier (upstream)
• Deletion regressions are compile-time by design: removing the enum value last forces the compiler to surface every missed reference

Monitoring

• Promotion INFO logs: → CTRL_PRIMARY (term=N) plus own-term noop appended lsn=N and gate-set confirmation; commit latency of the promotion no-op is observable from existing engine counters (umbrella monitoring item)

Testing Strategy

Unit Tests (testsuite/test_ha_role_lifecycle.c, XCE_TESTING)

• Direct promotion: driving the become-leader path (raft callback seam) yields role == CTRL_PRIMARY with no intermediate observable state; ha_is_primary true immediately; INFO transition logged once (1.1, 1.4)
• No redirect from a live primary: ha_should_redirect == 0 the instant promotion returns, before any ACK arrives (1.3)
• Own-term gate: with a real engine instance (n_peers=3 stub registry), promotion appends WAL_NOOP (entry observable in the WAL with payload_len 0) and sets writable_lsn to its LSN; a request-shaped commit_submit(WAIT) for a later entry does not complete while ACKs cover the entry but not the no-op — completes only when committed_lsn >= noop_lsn (2.1)
• Pre-own-term behavior: with the gate unmet, WAIT submission resolves only via barrier success or NOT_READY/timeout — never via any promotion-side path; no delaymgr task is armed by promotion (assert via delaymgr task count seam) (2.2, 2.3, 1.5)
• Standalone: n_peers=1 promotion → gate opens immediately (noop self-committed), WAIT submissions complete as before
• Replay: boot replay over a WAL containing WAL_NOOP applies cleanly with no entity side effects (entry count advances, jobmgr untouched)
• Vote honesty: elected node denies same-term votes as PRIMARY (gate parity with the old COMMITTING denial) (1.1)

Integration Tests (testsuite/)

• Legacy path symmetry: ha_elect_evaluate_and_signal quorum-win drives the same sequence — role PRIMARY directly, hooks registered, no watchdog armed (1.1, 1.5)
• Gate-branch deletion safety: barrier-rollout's gate tests stay green — PRIMARY fact allowlist intact under no-quorum; COMMITTING-specific gate cases removed with the branch (1.3)
• Updated fixtures (sweep): all 8 surviving HA test files green with PRIMARY-retargeted expectations; deleted files absent from meson

Regression

• Full controller testsuite + ASAN (/xce-build controller) green; grep verification: zero matches for the deleted-symbol pattern set across the repo (excluding .git/, .kiro/specs/)

E2E (deferred to ha-convergence-recovery's 3-node validation, which assumes this lifecycle)

• Kill-primary failover with the promoted standby confirming writes only after its no-op commits — owned downstream; this spec's testsuite seams (promotion hook, engine stub) are designed so that test can drive them

Migration Strategy

This spec is phase 3 of the umbrella's hard-ordered landing ("barrier before state removal") and is itself ordered so the tree compiles and tests stay green at every step:

1. WAL_NOOP entry type + replay/classification (additive, no behavior change)
2. Promotion rewrite + lifecycle tests (COMMITTING no longer entered in production; symbols still exist)
3. Legacy-path symmetry (COMMITTING no longer entered anywhere)
4. Machinery deletions (watchdog, convergence checker, ack counters, gate branch, predicates) with their dedicated test files deleted/retargeted in the same step
5. Fixture sweep, then enum-value removal last (compiler enforces completeness), then docs/steering sync and grep verification Rollback trigger per step: controller testsuite regression or failover E2E failure — reverting restores the prior (transitional-state) lifecycle, which the umbrella tolerates as "no less consistent than before". After step 5 there is no partial state: the grep gate guarantees all-or-nothing completion.