myxce/.kiro/specs/ha-single-leader-role/tasks.md

Implementation Plan: ha-single-leader-role

TDD ordering per project steering (testing.md): every implementation sub-task is preceded by a testsuite sub-task that lands failing (RED) tests first; deletion tasks are driven by retargeted test expectations plus compiler enforcement (the enum value is removed LAST so the compiler surfaces every missed reference). New tests live in testsuite/test_ha_role_lifecycle.c; two existing test files are deleted (their subject is the deleted machinery) and eight are updated — this shared-ha.c reality is why no (P) markers are applied: the plan is sequential. Hard upstream prerequisite: ha-commit-engine AND ha-commit-barrier-rollout must be implemented and green first — the promotion path calls commit_set_writable_lsn on the engine in cluster_t.commit, and the universal barrier + dispatch-gate fact allowlist must be in force before the transitional state is removed (umbrella hard ordering). Verify exact line numbers with grep before editing — research.md §3 has the 2026-06-11 verified anchor table, but barrier-rollout landing first WILL shift ha.c line numbers.

• 1. WAL_NOOP entry type (additive foundation)

• 1.1 Write RED tests for the no-op entry

  • Create testsuite/test_ha_role_lifecycle.c (XCE_TESTING pattern, AAA structure) and register it in controller_unit_test_src (src/controller/meson.build:74-131); first cases cover the entry type only
  • Tests: persist_wal_async(p, WAL_NOOP, NULL, 0) appends a header-only entry (payload_len 0) and broadcasts it (stub replicator observes the entry); apply_wal_entry on a WAL_NOOP advances current_lsn past the entry's LSN and touches no entity state (jobmgr/queue/worker untouched); boot replay_wal over a segment containing WAL_NOOP replays cleanly with the same no-effect guarantee; wal_classify(WAL_NOOP, 0) == WAL_MODE_RETAIN (keeps barrier-rollout's totality test green)
  • Observable completion: build succeeds with the new test file; all new cases FAIL (red) — WAL_NOOP does not exist yet
  • Requirements: 2.1

• 1.2 Implement WAL_NOOP to green

  • Add WAL_NOOP = 40 to wal_entry_type (src/include/wal_types.h:44-58) with comment: internal, empty payload, replay no-op, appended only at promotion; value stays below WAL_ENTRY_TYPE_MAX (100, src/include/config.h:80) so standbys LSN-advance and ACK it via the existing common path
  • Add explicit case WAL_NOOP: break; to apply_wal_entry's type switch (persist.c:728-765 anchor); add the WAL_NOOP → WAL_MODE_RETAIN row to wal_classify (the row barrier-rollout pre-declared)
  • Observable completion: all task 1.1 tests PASS; barrier-rollout's classification totality test still PASSES
  • Requirements: 2.1

• 2. Promotion rewrite: direct PRIMARY + own-term gate

• 2.1 Write RED lifecycle tests for direct promotion and the own-term gate

  • Tests in test_ha_role_lifecycle.c: driving the become-leader path yields ha->role == CTRL_PRIMARY directly (no intermediate value observable; ha_is_primary() true immediately); ha_should_redirect() returns 0 the instant promotion returns, before any standby ACK; persistence hooks are registered by promotion (hooksys observes the three persist hook fns; hooks_registered == 1); promotion arms NO delaymgr watchdog (delaymgr task-count seam unchanged across promotion); with a real engine (n_peers=3, stub registry): promotion appends WAL_NOOP at lsn=N, registers it RETAIN, and commit_set_writable_lsn(N) is in effect — a commit_submit(WAIT) for entry N+1 does NOT complete while ACKs cover N+1 but not N, and completes once committed_lsn >= N; with the gate unmet a WAIT submission resolves only as barrier success, -ETIMEDOUT, or -XCE_ERR_NOT_READY — never via any promotion-side path; n_peers=1: gate opens immediately at promotion (noop self-committed), WAIT returns 0 as before
  • Observable completion: new cases FAIL (red) against the current COMMITTING-entering raft_cb_become_leader
  • Requirements: 1.1, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3

• 2.2 Rewrite raft_cb_become_leader and share the promotion sequence

  • In raft_cb_become_leader (ha.c:445-617 anchor): keep term sync, safe-LSN truncation, last_known_committed_lsn/committed_config_tag stores, phase2, convergence-array memsets, vote-endpoint promotion, config push, ctrlhello, ha_start_primary_heartbeat; replace the CTRL_PRIMARY_COMMITTING store (ha.c:505-508) with a direct CTRL_PRIMARY store + INFO("HA FSM (raft): → CTRL_PRIMARY (term=...)"); delete the repl_ack_count reset (ha.c:511) and the watchdog-arm block (ha.c:593-616)
  • Extract hook registration from ha_promote_to_primary (ha.c:1691-1698) into static ha_register_persist_hooks(cluster_t *c) (keeps the hooks_registered idempotency guard); call it in the rewrite BEFORE the no-op append; ha_promote_to_primary body becomes ha_do_promote + ha_register_persist_hooks + ha_start_primary_heartbeat + the same noop+gate tail — every road to PRIMARY arms the gate; the rewrite does NOT call ha_do_promote (raft owns the term; no double increment)
  • Noop+gate tail (production, guarded on c != NULL and engine availability): persist_wal_async(WAL_NOOP) after endpoint promotion → noop_lsn = current_lsn - 1 under p->lock → commit_submit(&c->commit, noop_lsn, COMMIT_MODE_RETAIN, 0) → commit_set_writable_lsn(&c->commit, noop_lsn); append failure → ERROR + WARN, gate left unset (degraded-but-safe per design Error Strategy), promotion still completes
  • Observable completion: all task 2.1 tests PASS; promotion never blocks (returns without waiting for the noop to commit)
  • Requirements: 1.1, 1.3, 1.4, 1.5, 2.1, 2.2, 2.3

• 3. Legacy XCE_TESTING election path symmetry

• 3.1 Update ha_elect_evaluate_and_signal and its direct tests (RED → GREEN in one pass)

  • Retarget existing legacy-path expectations first (test_ha_start_election_async.c, the evaluation cases in test_ha_role.c): quorum win → role CTRL_PRIMARY directly, no watchdog armed — run to confirm RED against current code
  • In the CTRL_PRIMARY branch of ha_elect_evaluate_and_signal (ha.c:2247-2300 anchor): replace the COMMITTING store (2248-2252) with the direct CTRL_PRIMARY store, delete the repl_ack_count reset (2255) and the watchdog-arm block (2275-2299); keep the config-push/ctrlhello XCE_TESTING hooks; route promotion side effects through the same shared sequence (noop+gate tail guarded on persist/engine availability, as in production)
  • Observable completion: retargeted legacy-path tests PASS; lifecycle test for symmetry PASSES (both entry points produce identical role/hook/heartbeat/no-watchdog state)
  • Requirements: 1.1, 1.5

• 4. Machinery deletions (watchdog, convergence, ack counters, gate branch, predicates)

• 4.1 Delete the force-promotion watchdog and ack/convergence machinery

  • Delete from ha.c: committing_timeout_fn (3080-3121), committing_timeout_fn_for_test (2950-2990), committing_timeout_ctx_t type + fwd decl (64-69, 83), the cancel block in raft_cb_become_follower (640-648); ha_check_majority_convergence (1295-1353) and its heartbeat-reply call site (2710-2715); ha_repl_ack_received (3123-3141) — its OPCode_replCatchUpDone branch in repl_ack_decode_and_dispatch (2659-2661) becomes an INFO-logged no-op (opcode stays gate-allowed; repl_complete_catchup untouched); repl_ack_count init at ha.c:835
  • Delete declarations from ha.h: ha_check_majority_convergence (396-412), ha_repl_ack_received (626-630)
  • Delete test files whose subject is this machinery: remove testsuite/test_ha_repl_ack.c and testsuite/test_ha_evaluate_committing.c and their entries in controller_unit_test_src (meson.build:116, 118 anchors); concerns still valid (promotion only on real quorum, no promotion without majority) are already pinned by task 2.1's gate tests
  • Observable completion: controller builds; grep -rn "committing_timeout_fn\|ha_check_majority_convergence\|ha_repl_ack_received" src/controller/*.c src/controller/*.h returns zero matches; test suite green
  • Requirements: 1.5, 2.3

• 4.2 Delete the dispatch-gate COMMITTING branch and collapse role predicates

  • Delete ONLY the case CTRL_PRIMARY_COMMITTING: block from ha_dispatch_gate (ha.c:1594-1612 pre-rollout anchor — re-grep; the block follows the ELECTING case); the CTRL_PRIMARY branch — including barrier-rollout's no-quorum fact allowlist (OPCode_notify/update/setpreason/clrpreason) — is preserved byte-for-byte
  • Collapse compound role checks to PRIMARY-only: raft_cb_vote_eligible (ha.c:210-211), handle_request_vote gates (ha.c:1984, 2004), vote_ep_err_handler (ha.c:3159), peer_ctrl_msg_handler case labels (ha.c:2789), primary_heartbeat_fn re-arm/run condition (ha.c:3956 — heartbeat re-arm condition drops COMMITTING); update the stale comments at each site plus ha.c:1940, 1971, 2057-2060, 2625-2627, 3044-3045 and the ha_is_primary/ha_should_redirect doc comments (no code change in those two functions)
  • Update gate tests: COMMITTING-specific cases in test_ha_vote_gate.c and gate fixtures retargeted (elected node denies same-term votes as PRIMARY — denial parity pinned)
  • Observable completion: barrier-rollout's gate tests (PRIMARY fact allowlist under no-quorum) still PASS; vote-gate tests PASS with PRIMARY-only denial; suite green
  • Requirements: 1.2, 1.3, 1.4

• 5. Test-fixture sweep and final symbol removal

• 5.1 Retarget the remaining COMMITTING-referencing fixtures

  • Update expectations COMMITTING→PRIMARY (census: research.md §3): test_ha_role.c (37 refs), test_ha_rdhup_integration.c (20), test_ha_repl_push.c (14), test_ha_startup.c (12), test_ha_connection_reuse.c (11), test_ha_start_election_async.c (7), test_ha_vote_gate.c (6, residual from 4.2); delete the 11 cfg.committing_timeout_ms = 10000; initializers in test_config_repl.c
  • Each updated test asserts the new contract (direct PRIMARY, no transitional observations), not stubbed compatibility
  • Observable completion: full controller suite green; grep -rln "COMMITTING" src/controller/testsuite/ returns zero files
  • Requirements: 1.1, 1.2

• 5.2 Remove the enum value and remaining declarations (compiler-enforced completeness)

  • Delete CTRL_PRIMARY_COMMITTING = 4 from ctrl_role_t (ha.h:64); delete HA_DEFAULT_COMMITTING_TIMEOUT_MS (ha.h:131), committing_timeout_ms (ha.h:141), repl_ack_count (ha.h:180), committing_timeout_ctx (ha.h:187); delete the .committing_timeout_ms initializer (cluster.c:899); sweep remaining ha.h comments (214-230, 244-262, 373, 503-519, 607-614, 673-700, 791), the cluster.c ctrlhello comment (1402-1410), and the raft.c comment (src/lib/raft/raft.c:123)
  • Fix everything the compiler then surfaces — this is the completeness gate for code references
  • Observable completion: clean build with zero warnings about the removed symbols; full controller testsuite + ASAN green via /xce-build controller
  • Requirements: 1.2

• 6. Documentation sync and grep verification

• 6.1 Synchronize steering and contributor docs to the three-state model

  • .kiro/steering/ha-async-io.md: heartbeat re-arm sentence (:51 — "PRIMARY only"); Replication Consistency section (:57 — replace the COMMITTING→PRIMARY convergence-promotion sentence with the own-term no-op gate: promotion appends WAL_NOOP, confirmation waits behind committed_lsn >= writable_lsn); dispatch-gate table (:68 — remove the CTRL_PRIMARY_COMMITTING row; note the PRIMARY no-quorum fact allowlist); Raft-to-code mapping (:79 — role set without PRIMARY_COMMITTING)
  • src/controller/CLAUDE.md:40 role-mapping table; .claude/skills/xce-deploy-ha/SKILL.md:98 expected log line (now a single → CTRL_PRIMARY); src/docs/cluster-config-split-analysis.md:34 config listing (drop the deleted knob)
  • Observable completion: grep -rn "COMMITTING" .kiro/steering/ src/controller/CLAUDE.md .claude/skills/xce-deploy-ha/ src/docs/ returns zero matches
  • Requirements: 1.2

• 6.2 Final grep verification and full regression

  • Run the completion gate: grep -rEn "CTRL_PRIMARY_COMMITTING|PRIMARY_COMMITTING|committing_timeout|repl_ack_count|ha_repl_ack_received|ha_check_majority_convergence" --exclude-dir=.git --exclude-dir=specs --exclude-dir=.wolf --exclude-dir=.codegraph --exclude-dir=builddir . — zero matches; the only permitted residue is git history and .kiro/specs/ documents (this spec's own census and the umbrella audit). Note .kiro/steering/ is deliberately INSIDE the gate — a missed steering reference fails it
  • Run /xce-build controller (debug + ASAN + full testsuite) — green; record the grep output and build result as task evidence
  • Observable completion: zero-match grep output committed as evidence; controller suite + ASAN green
  • Requirements: 1.2, 1.5, 2.3

Implementation Notes

• (2.1/2.2) Two promotion seams in the lifecycle tests: cases 1-2 (role/redirect) drive raft_cb_become_leader_for_test which forces c==NULL (the noop+gate tail is c!=NULL-guarded and skipped); cases 3-7 (hooks/gate) drive the real-cluster shared seam ha_promote_to_primary(&g_lc). Hook registration cannot be pinned on the c==NULL path (the helper needs &c->hooksys) — case 3 is a green contract pin via ha_promote_to_primary.
• (2.2) Expected category-(a) failures open from task 2.2 until retargeted: after the raft path enters CTRL_PRIMARY directly, three single-node tests still assert CTRL_PRIMARY_COMMITTING (4) and FAIL (Expected 4 Was 0): test_single_node_shortcircuit_evaluates_immediately + test_single_node_not_primary_directly (test_ha_start_election_async.c) → retargeted by task 3.1; test_nonblocking_single_node_election_enters_committing (test_ha_startup.c) → retargeted by task 5.1. Single-node election reaches raft_cb_become_leader via the raft_start_election shortcircuit firing on_become_leader.
• (2.2) ha.c shared sequence: raft path stores CTRL_PRIMARY directly with NO ha_do_promote (term synced at entry — avoids double-increment); ha_promote_to_primary keeps ha_do_promote (owns its term). Both call static ha_register_persist_hooks(c) (BEFORE noop append) and static ha_arm_own_term_gate(c) (persist_wal_async(WAL_NOOP) → noop_lsn = current_lsn-1 under p->lock → commit_submit(RETAIN) → commit_set_writable_lsn); append-failure degrades safe (gate unset, promotion completes); non-blocking. committing_timeout_fn + enum retained until tasks 4.x/5.2.
• Unity runner gotcha: the aggregated controller runner prints the PREVIOUS passing test's file:line as a stale prefix on a FAIL line — attribute failures by test NAME, not the printed file:line prefix.
• (3.1) Legacy path routes through BARE helpers, not ha_promote_to_primary: ha_elect_evaluate_and_signal's PRIMARY branch calls ha_register_persist_hooks + ha_start_primary_heartbeat + ha_arm_own_term_gate directly (matching the raft path), NOT ha_promote_to_primary — because the latter calls ha_do_promote (term increment) and the legacy branch historically has NO term store/increment (research §4 finding 2). This is a deliberate deviation from design §283's loose wording ("route through ha_promote_to_primary / ha_register_persist_hooks"); the bare-helper route preserves term semantics and is the correct reading.
• (4.1) No shims — DELETE, don't stub (design §Transitional-State Deletions / §Design Synthesis: "rewritten or DELETED, not stubbed"). Machinery-subject test cases (convergence checker, committing watchdog, ack-driven promotion, ack counting) were DELETED, not shimmed: test_ha_role.c (5 majority_convergence + 2 committing_timeout_promotes_degraded), test_ha_rdhup_integration.c (rdhup_committing_to_primary_on_ack, rdhup_3node_one_ack_promotes), test_ha_connection_reuse.c (dispatcher_committing_handles_repl_ack, promoted_ep_repl_ack_promotes_to_primary). Kept state-entry cases had their deleted-fn CALLS excised (not shimmed). test_ha_connection_reuse.c was overlooked by the spec's §3 census disposition for machinery cases — confirmed in census (line 97) but its ack cases needed DELETE, not UPDATE.
• (4.2) Residual stale COMMITTING comments in ha.c (~lines 443, 826, 1240): documentary prose on the legacy raft_cb_become_leader/convergence path — NOT live branches. Scrub in 5.2/6.x (they fail the task-6.2 repo-wide grep). After 4.2, ZERO live code branches on CTRL_PRIMARY_COMMITTING in ha.c; the enum value persists in ha.h:64 until 5.2.
• (5.2) Code is COMMITTING-free; only DOCS remain for task 6.1: after 5.2 the grep CTRL_PRIMARY_COMMITTING|HA_DEFAULT_COMMITTING_TIMEOUT_MS|committing_timeout_ms|repl_ack_count|committing_timeout_ctx is ZERO across src/controller/ + src/lib/. Remaining tree references (task 6.1 docs-sync targets, then 6.2 verifies): src/docs/cluster-config-split-analysis.md:34 (committing_timeout_ms) AND src/controller/CLAUDE.md (TLA+/role-mapping table still names PRIMARY_COMMITTING — 6.1 must scrub ALL its COMMITTING refs, not just the :40 line) plus the steering/skill docs already listed in 6.1. ctrl_role_t now has 4 values: CTRL_PRIMARY=0, CTRL_STANDBY=1, CTRL_INIT=2, CTRL_ELECTING=3.
• (5.1) Meson 300s timeout is now routinely hit — barrier tests' commit-deadline waits push controller-suite wall-time over the 300s meson cap under load (Fail: 0, Timeout: 1). The AUTHORITATIVE green check is the DIRECT binary run (src/xce.build/controller/controller_unit_test, EXIT=0, 167 Tests/0 Failures aggregated). For task 6.2's "suite + ASAN green" gate: run the direct binary OR re-run meson until under-cap; a Timeout: 1 is a wrapper artifact, NOT a test failure. test_committing_timeout_default_is_60s was DELETED in 5.1 (not 5.2) because its body used the uppercase production macro HA_DEFAULT_COMMITTING_TIMEOUT_MS (caught by the grep gate); the knob it tested dies in 5.2 regardless.
• (4.2) Test-leak / suite-timeout facts: the controller test target sets ASAN_OPTIONS=detect_leaks=0 (controller/meson.build:163), so a pre-existing suite-wide LSan leak (~153KB, present at baseline 678b7f8, this spec leaks slightly LESS) is suppressed under the meson gate — LSan exit=1 only on a RAW direct-binary run, not a regression. The controller suite runs ~268s against a 300s meson timeout — borderline; the final task-6.2 full regression may TIMEOUT under machine load — re-run if so (it is not a failure).
• (4.1) Residual for task 6.2 final grep: 3 DOCUMENTARY PROSE comments in test_ha_role_lifecycle.c (~lines 760/815/825) still mention ha_repl_ack_received/committing_timeout_fn by name (no live code). The 4.1 gate (src/controller/*.c|*.h) excludes testsuite so this passed; the task-6.2 repo-wide grep WILL catch them — scrub these comments in 5.x/6.x. Also a non-tracked editor backup src/controller/.ha.c.~undo-tree~ holds stale refs (irrelevant; not a build input — do not commit).
• (3.1) Expected-failure set after task 3.1 (8 category-(a), all owned downstream): task 4.1 DELETES test_ha_evaluate_committing.c (3 fails: test_winning_quorum_enters_committing, test_committing_resets_repl_ack_count, test_election_timeout_fn_fires_evaluate); task 5.1 retargets test_nonblocking_single_node_election_enters_committing (test_ha_startup.c), test_3node_win_calls_both_pushes (test_ha_repl_push.c), and 3 RDHUP cases (test_ha_rdhup_integration.c: rdhup_full_path_reaches_committing, rdhup_committing_to_primary_on_ack, rdhup_3node_one_ack_promotes). committing_timeout_fn now has a benign -Wunused-function warning (no arming site left) until task 4.1 deletes it.
• Lifecycle fixture (test_ha_role_lifecycle.c): lifecycle_cluster_setup installs repl_broadcast_send_hook = lc_drop_send_hook (drop-only, reset to NULL in teardown) — required once promotion broadcasts the WAL_NOOP, else fake int endpoints cause an ASAN overflow. Gate cases prime current_lsn = noop_lsn so the post-increment append lands the noop at noop_lsn (production derives noop_lsn = current_lsn-1).