myxce/.kiro/specs/ha-single-leader-role/tasks.md

# Implementation Plan: ha-single-leader-role

> TDD ordering per project steering (`testing.md`): every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first; deletion tasks are driven by retargeted test expectations plus compiler enforcement (the enum value is removed LAST so the compiler surfaces every missed reference). New tests live in `testsuite/test_ha_role_lifecycle.c`; two existing test files are deleted (their subject is the deleted machinery) and eight are updated — this shared-ha.c reality is why no `(P)` markers are applied: the plan is sequential.
> **Hard upstream prerequisite**: ha-commit-engine AND ha-commit-barrier-rollout must be implemented and green first — the promotion path calls `commit_set_writable_lsn` on the engine in `cluster_t.commit`, and the universal barrier + dispatch-gate fact allowlist must be in force before the transitional state is removed (umbrella hard ordering).
> Verify exact line numbers with grep before editing — research.md §3 has the 2026-06-11 verified anchor table, but barrier-rollout landing first WILL shift ha.c line numbers.

- [x] 1. WAL_NOOP entry type (additive foundation)
- [x] 1.1 Write RED tests for the no-op entry
  - Create `testsuite/test_ha_role_lifecycle.c` (XCE_TESTING pattern, AAA structure) and register it in `controller_unit_test_src` (src/controller/meson.build:74-131); first cases cover the entry type only
  - Tests: `persist_wal_async(p, WAL_NOOP, NULL, 0)` appends a header-only entry (payload_len 0) and broadcasts it (stub replicator observes the entry); `apply_wal_entry` on a WAL_NOOP advances `current_lsn` past the entry's LSN and touches no entity state (jobmgr/queue/worker untouched); boot `replay_wal` over a segment containing WAL_NOOP replays cleanly with the same no-effect guarantee; `wal_classify(WAL_NOOP, 0) == WAL_MODE_RETAIN` (keeps barrier-rollout's totality test green)
  - Observable completion: build succeeds with the new test file; all new cases FAIL (red) — `WAL_NOOP` does not exist yet
  - _Requirements: 2.1_

- [x] 1.2 Implement WAL_NOOP to green
  - Add `WAL_NOOP = 40` to `wal_entry_type` (src/include/wal_types.h:44-58) with comment: internal, empty payload, replay no-op, appended only at promotion; value stays below `WAL_ENTRY_TYPE_MAX` (100, src/include/config.h:80) so standbys LSN-advance and ACK it via the existing common path
  - Add explicit `case WAL_NOOP: break;` to `apply_wal_entry`'s type switch (persist.c:728-765 anchor); add the `WAL_NOOP → WAL_MODE_RETAIN` row to `wal_classify` (the row barrier-rollout pre-declared)
  - Observable completion: all task 1.1 tests PASS; barrier-rollout's classification totality test still PASSES
  - _Requirements: 2.1_

- [x] 2. Promotion rewrite: direct PRIMARY + own-term gate
- [x] 2.1 Write RED lifecycle tests for direct promotion and the own-term gate
  - Tests in `test_ha_role_lifecycle.c`: driving the become-leader path yields `ha->role == CTRL_PRIMARY` directly (no intermediate value observable; `ha_is_primary()` true immediately); `ha_should_redirect()` returns 0 the instant promotion returns, before any standby ACK; persistence hooks are registered by promotion (hooksys observes the three persist hook fns; `hooks_registered == 1`); promotion arms NO delaymgr watchdog (delaymgr task-count seam unchanged across promotion); with a real engine (n_peers=3, stub registry): promotion appends WAL_NOOP at lsn=N, registers it RETAIN, and `commit_set_writable_lsn(N)` is in effect — a `commit_submit(WAIT)` for entry N+1 does NOT complete while ACKs cover N+1 but not N, and completes once `committed_lsn >= N`; with the gate unmet a WAIT submission resolves only as barrier success, `-ETIMEDOUT`, or `-XCE_ERR_NOT_READY` — never via any promotion-side path; n_peers=1: gate opens immediately at promotion (noop self-committed), WAIT returns 0 as before
  - Observable completion: new cases FAIL (red) against the current COMMITTING-entering `raft_cb_become_leader`
  - _Requirements: 1.1, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3_

- [x] 2.2 Rewrite raft_cb_become_leader and share the promotion sequence
  - In `raft_cb_become_leader` (ha.c:445-617 anchor): keep term sync, safe-LSN truncation, `last_known_committed_lsn`/`committed_config_tag` stores, phase2, convergence-array memsets, vote-endpoint promotion, config push, ctrlhello, `ha_start_primary_heartbeat`; replace the `CTRL_PRIMARY_COMMITTING` store (ha.c:505-508) with a direct `CTRL_PRIMARY` store + `INFO("HA FSM (raft): → CTRL_PRIMARY (term=...)")`; delete the `repl_ack_count` reset (ha.c:511) and the watchdog-arm block (ha.c:593-616)
  - Extract hook registration from `ha_promote_to_primary` (ha.c:1691-1698) into static `ha_register_persist_hooks(cluster_t *c)` (keeps the `hooks_registered` idempotency guard); call it in the rewrite BEFORE the no-op append; `ha_promote_to_primary` body becomes `ha_do_promote` + `ha_register_persist_hooks` + `ha_start_primary_heartbeat` + the same noop+gate tail — every road to PRIMARY arms the gate; the rewrite does NOT call `ha_do_promote` (raft owns the term; no double increment)
  - Noop+gate tail (production, guarded on `c != NULL` and engine availability): `persist_wal_async(WAL_NOOP)` after endpoint promotion → `noop_lsn = current_lsn - 1` under `p->lock` → `commit_submit(&c->commit, noop_lsn, COMMIT_MODE_RETAIN, 0)` → `commit_set_writable_lsn(&c->commit, noop_lsn)`; append failure → ERROR + WARN, gate left unset (degraded-but-safe per design Error Strategy), promotion still completes
  - Observable completion: all task 2.1 tests PASS; promotion never blocks (returns without waiting for the noop to commit)
  - _Requirements: 1.1, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3_

- [x] 3. Legacy XCE_TESTING election path symmetry
- [x] 3.1 Update ha_elect_evaluate_and_signal and its direct tests (RED → GREEN in one pass)
  - Retarget existing legacy-path expectations first (test_ha_start_election_async.c, the evaluation cases in test_ha_role.c): quorum win → role `CTRL_PRIMARY` directly, no watchdog armed — run to confirm RED against current code
  - In the CTRL_PRIMARY branch of `ha_elect_evaluate_and_signal` (ha.c:2247-2300 anchor): replace the COMMITTING store (2248-2252) with the direct `CTRL_PRIMARY` store, delete the `repl_ack_count` reset (2255) and the watchdog-arm block (2275-2299); keep the config-push/ctrlhello XCE_TESTING hooks; route promotion side effects through the same shared sequence (noop+gate tail guarded on persist/engine availability, as in production)
  - Observable completion: retargeted legacy-path tests PASS; lifecycle test for symmetry PASSES (both entry points produce identical role/hook/heartbeat/no-watchdog state)
  - _Requirements: 1.1, 1.5_

- [x] 4. Machinery deletions (watchdog, convergence, ack counters, gate branch, predicates)
- [x] 4.1 Delete the force-promotion watchdog and ack/convergence machinery
  - Delete from ha.c: `committing_timeout_fn` (3080-3121), `committing_timeout_fn_for_test` (2950-2990), `committing_timeout_ctx_t` type + fwd decl (64-69, 83), the cancel block in `raft_cb_become_follower` (640-648); `ha_check_majority_convergence` (1295-1353) and its heartbeat-reply call site (2710-2715); `ha_repl_ack_received` (3123-3141) — its `OPCode_replCatchUpDone` branch in `repl_ack_decode_and_dispatch` (2659-2661) becomes an INFO-logged no-op (opcode stays gate-allowed; `repl_complete_catchup` untouched); `repl_ack_count` init at ha.c:835
  - Delete declarations from ha.h: `ha_check_majority_convergence` (396-412), `ha_repl_ack_received` (626-630)
  - Delete test files whose subject is this machinery: remove `testsuite/test_ha_repl_ack.c` and `testsuite/test_ha_evaluate_committing.c` and their entries in `controller_unit_test_src` (meson.build:116, 118 anchors); concerns still valid (promotion only on real quorum, no promotion without majority) are already pinned by task 2.1's gate tests
  - Observable completion: controller builds; `grep -rn "committing_timeout_fn\|ha_check_majority_convergence\|ha_repl_ack_received" src/controller/*.c src/controller/*.h` returns zero matches; test suite green
  - _Requirements: 1.5, 2.3_

- [x] 4.2 Delete the dispatch-gate COMMITTING branch and collapse role predicates
  - Delete ONLY the `case CTRL_PRIMARY_COMMITTING:` block from `ha_dispatch_gate` (ha.c:1594-1612 pre-rollout anchor — re-grep; the block follows the ELECTING case); the CTRL_PRIMARY branch — including barrier-rollout's no-quorum fact allowlist (`OPCode_notify/update/setpreason/clrpreason`) — is preserved byte-for-byte
  - Collapse compound role checks to PRIMARY-only: `raft_cb_vote_eligible` (ha.c:210-211), `handle_request_vote` gates (ha.c:1984, 2004), `vote_ep_err_handler` (ha.c:3159), `peer_ctrl_msg_handler` case labels (ha.c:2789), `primary_heartbeat_fn` re-arm/run condition (ha.c:3956 — heartbeat re-arm condition drops COMMITTING); update the stale comments at each site plus ha.c:1940, 1971, 2057-2060, 2625-2627, 3044-3045 and the `ha_is_primary`/`ha_should_redirect` doc comments (no code change in those two functions)
  - Update gate tests: COMMITTING-specific cases in `test_ha_vote_gate.c` and gate fixtures retargeted (elected node denies same-term votes as PRIMARY — denial parity pinned)
  - Observable completion: barrier-rollout's gate tests (PRIMARY fact allowlist under no-quorum) still PASS; vote-gate tests PASS with PRIMARY-only denial; suite green
  - _Requirements: 1.2, 1.3, 1.4_

- [x] 5. Test-fixture sweep and final symbol removal
- [x] 5.1 Retarget the remaining COMMITTING-referencing fixtures
  - Update expectations COMMITTING→PRIMARY (census: research.md §3): `test_ha_role.c` (37 refs), `test_ha_rdhup_integration.c` (20), `test_ha_repl_push.c` (14), `test_ha_startup.c` (12), `test_ha_connection_reuse.c` (11), `test_ha_start_election_async.c` (7), `test_ha_vote_gate.c` (6, residual from 4.2); delete the 11 `cfg.committing_timeout_ms = 10000;` initializers in `test_config_repl.c`
  - Each updated test asserts the new contract (direct PRIMARY, no transitional observations), not stubbed compatibility
  - Observable completion: full controller suite green; `grep -rln "COMMITTING" src/controller/testsuite/` returns zero files
  - _Requirements: 1.1, 1.2_

- [x] 5.2 Remove the enum value and remaining declarations (compiler-enforced completeness)
  - Delete `CTRL_PRIMARY_COMMITTING = 4` from `ctrl_role_t` (ha.h:64); delete `HA_DEFAULT_COMMITTING_TIMEOUT_MS` (ha.h:131), `committing_timeout_ms` (ha.h:141), `repl_ack_count` (ha.h:180), `committing_timeout_ctx` (ha.h:187); delete the `.committing_timeout_ms` initializer (cluster.c:899); sweep remaining ha.h comments (214-230, 244-262, 373, 503-519, 607-614, 673-700, 791), the cluster.c ctrlhello comment (1402-1410), and the raft.c comment (src/lib/raft/raft.c:123)
  - Fix everything the compiler then surfaces — this is the completeness gate for code references
  - Observable completion: clean build with zero warnings about the removed symbols; full controller testsuite + ASAN green via `/xce-build controller`
  - _Requirements: 1.2_

- [x] 6. Documentation sync and grep verification
- [x] 6.1 Synchronize steering and contributor docs to the three-state model
  - `.kiro/steering/ha-async-io.md`: heartbeat re-arm sentence (:51 — "PRIMARY only"); Replication Consistency section (:57 — replace the COMMITTING→PRIMARY convergence-promotion sentence with the own-term no-op gate: promotion appends WAL_NOOP, confirmation waits behind `committed_lsn >= writable_lsn`); dispatch-gate table (:68 — remove the CTRL_PRIMARY_COMMITTING row; note the PRIMARY no-quorum fact allowlist); Raft-to-code mapping (:79 — role set without PRIMARY_COMMITTING)
  - `src/controller/CLAUDE.md:40` role-mapping table; `.claude/skills/xce-deploy-ha/SKILL.md:98` expected log line (now a single `→ CTRL_PRIMARY`); `src/docs/cluster-config-split-analysis.md:34` config listing (drop the deleted knob)
  - Observable completion: `grep -rn "COMMITTING" .kiro/steering/ src/controller/CLAUDE.md .claude/skills/xce-deploy-ha/ src/docs/` returns zero matches
  - _Requirements: 1.2_

- [x] 6.2 Final grep verification and full regression
  - Run the completion gate: `grep -rEn "CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence" --exclude-dir=.git --exclude-dir=specs --exclude-dir=.wolf --exclude-dir=.codegraph --exclude-dir=builddir .` — zero matches; the only permitted residue is git history and `.kiro/specs/` documents (this spec's own census and the umbrella audit). Note `.kiro/steering/` is deliberately INSIDE the gate — a missed steering reference fails it
  - Run `/xce-build controller` (debug + ASAN + full testsuite) — green; record the grep output and build result as task evidence
  - Observable completion: zero-match grep output committed as evidence; controller suite + ASAN green
  - _Requirements: 1.2, 1.5, 2.3_

## Implementation Notes

- **(2.1/2.2) Two promotion seams in the lifecycle tests**: cases 1-2 (role/redirect) drive `raft_cb_become_leader_for_test` which forces `c==NULL` (the noop+gate tail is `c!=NULL`-guarded and skipped); cases 3-7 (hooks/gate) drive the real-cluster shared seam `ha_promote_to_primary(&g_lc)`. Hook registration cannot be pinned on the `c==NULL` path (the helper needs `&c->hooksys`) — case 3 is a green contract pin via `ha_promote_to_primary`.
- **(2.2) Expected category-(a) failures open from task 2.2 until retargeted**: after the raft path enters `CTRL_PRIMARY` directly, three single-node tests still assert `CTRL_PRIMARY_COMMITTING (4)` and FAIL (`Expected 4 Was 0`): `test_single_node_shortcircuit_evaluates_immediately` + `test_single_node_not_primary_directly` (test_ha_start_election_async.c) → retargeted by **task 3.1**; `test_nonblocking_single_node_election_enters_committing` (test_ha_startup.c) → retargeted by **task 5.1**. Single-node election reaches `raft_cb_become_leader` via the `raft_start_election` shortcircuit firing `on_become_leader`.
- **(2.2) ha.c shared sequence**: raft path stores `CTRL_PRIMARY` directly with NO `ha_do_promote` (term synced at entry — avoids double-increment); `ha_promote_to_primary` keeps `ha_do_promote` (owns its term). Both call static `ha_register_persist_hooks(c)` (BEFORE noop append) and static `ha_arm_own_term_gate(c)` (`persist_wal_async(WAL_NOOP)` → `noop_lsn = current_lsn-1` under p->lock → `commit_submit(RETAIN)` → `commit_set_writable_lsn`); append-failure degrades safe (gate unset, promotion completes); non-blocking. `committing_timeout_fn` + enum retained until tasks 4.x/5.2.
- **Unity runner gotcha**: the aggregated controller runner prints the PREVIOUS passing test's `file:line` as a stale prefix on a FAIL line — attribute failures by test NAME, not the printed file:line prefix.
- **(3.1) Legacy path routes through BARE helpers, not `ha_promote_to_primary`**: `ha_elect_evaluate_and_signal`'s PRIMARY branch calls `ha_register_persist_hooks` + `ha_start_primary_heartbeat` + `ha_arm_own_term_gate` directly (matching the raft path), NOT `ha_promote_to_primary` — because the latter calls `ha_do_promote` (term increment) and the legacy branch historically has NO term store/increment (research §4 finding 2). This is a deliberate deviation from design §283's loose wording ("route through ha_promote_to_primary / ha_register_persist_hooks"); the bare-helper route preserves term semantics and is the correct reading.
- **(4.1) No shims — DELETE, don't stub** (design §Transitional-State Deletions / §Design Synthesis: "rewritten or DELETED, not stubbed"). Machinery-subject test cases (convergence checker, committing watchdog, ack-driven promotion, ack counting) were DELETED, not shimmed: test_ha_role.c (5 majority_convergence + 2 committing_timeout_promotes_degraded), test_ha_rdhup_integration.c (rdhup_committing_to_primary_on_ack, rdhup_3node_one_ack_promotes), test_ha_connection_reuse.c (dispatcher_committing_handles_repl_ack, promoted_ep_repl_ack_promotes_to_primary). Kept state-entry cases had their deleted-fn CALLS excised (not shimmed). `test_ha_connection_reuse.c` was overlooked by the spec's §3 census disposition for machinery cases — confirmed in census (line 97) but its ack cases needed DELETE, not UPDATE.
- **(4.2) Residual stale COMMITTING comments in ha.c** (~lines 443, 826, 1240): documentary prose on the legacy raft_cb_become_leader/convergence path — NOT live branches. Scrub in 5.2/6.x (they fail the task-6.2 repo-wide grep). After 4.2, ZERO live code branches on `CTRL_PRIMARY_COMMITTING` in ha.c; the enum value persists in ha.h:64 until 5.2.
- **(5.2) Code is COMMITTING-free; only DOCS remain for task 6.1**: after 5.2 the grep `CTRL_PRIMARY_COMMITTING|HA_DEFAULT_COMMITTING_TIMEOUT_MS|committing_timeout_ms|repl_ack_count|committing_timeout_ctx` is ZERO across `src/controller/` + `src/lib/`. Remaining tree references (task 6.1 docs-sync targets, then 6.2 verifies): `src/docs/cluster-config-split-analysis.md:34` (committing_timeout_ms) AND `src/controller/CLAUDE.md` (TLA+/role-mapping table still names PRIMARY_COMMITTING — 6.1 must scrub ALL its COMMITTING refs, not just the :40 line) plus the steering/skill docs already listed in 6.1. ctrl_role_t now has 4 values: CTRL_PRIMARY=0, CTRL_STANDBY=1, CTRL_INIT=2, CTRL_ELECTING=3.
- **(5.1) Meson 300s timeout is now routinely hit** — barrier tests' commit-deadline waits push controller-suite wall-time over the 300s meson cap under load (`Fail: 0, Timeout: 1`). The AUTHORITATIVE green check is the DIRECT binary run (`src/xce.build/controller/controller_unit_test`, EXIT=0, 167 Tests/0 Failures aggregated). For task 6.2's "suite + ASAN green" gate: run the direct binary OR re-run meson until under-cap; a `Timeout: 1` is a wrapper artifact, NOT a test failure. `test_committing_timeout_default_is_60s` was DELETED in 5.1 (not 5.2) because its body used the uppercase production macro `HA_DEFAULT_COMMITTING_TIMEOUT_MS` (caught by the grep gate); the knob it tested dies in 5.2 regardless.
- **(4.2) Test-leak / suite-timeout facts**: the controller test target sets `ASAN_OPTIONS=detect_leaks=0` (controller/meson.build:163), so a pre-existing suite-wide LSan leak (~153KB, present at baseline 678b7f8, this spec leaks slightly LESS) is suppressed under the meson gate — LSan exit=1 only on a RAW direct-binary run, not a regression. The controller suite runs ~268s against a 300s meson timeout — borderline; the final task-6.2 full regression may TIMEOUT under machine load — re-run if so (it is not a failure).
- **(4.1) Residual for task 6.2 final grep**: 3 DOCUMENTARY PROSE comments in `test_ha_role_lifecycle.c` (~lines 760/815/825) still mention `ha_repl_ack_received`/`committing_timeout_fn` by name (no live code). The 4.1 gate (`src/controller/*.c|*.h`) excludes testsuite so this passed; the task-6.2 repo-wide grep WILL catch them — scrub these comments in 5.x/6.x. Also a non-tracked editor backup `src/controller/.ha.c.~undo-tree~` holds stale refs (irrelevant; not a build input — do not commit).
- **(3.1) Expected-failure set after task 3.1 (8 category-(a), all owned downstream)**: task 4.1 DELETES `test_ha_evaluate_committing.c` (3 fails: test_winning_quorum_enters_committing, test_committing_resets_repl_ack_count, test_election_timeout_fn_fires_evaluate); task 5.1 retargets `test_nonblocking_single_node_election_enters_committing` (test_ha_startup.c), `test_3node_win_calls_both_pushes` (test_ha_repl_push.c), and 3 RDHUP cases (test_ha_rdhup_integration.c: rdhup_full_path_reaches_committing, rdhup_committing_to_primary_on_ack, rdhup_3node_one_ack_promotes). `committing_timeout_fn` now has a benign `-Wunused-function` warning (no arming site left) until task 4.1 deletes it.
- **Lifecycle fixture (test_ha_role_lifecycle.c)**: `lifecycle_cluster_setup` installs `repl_broadcast_send_hook = lc_drop_send_hook` (drop-only, reset to NULL in teardown) — required once promotion broadcasts the WAL_NOOP, else fake `int` endpoints cause an ASAN overflow. Gate cases prime `current_lsn = noop_lsn` so the post-increment append lands the noop at `noop_lsn` (production derives `noop_lsn = current_lsn-1`).