myxce/.kiro/specs/ha-config-bootstrap/tasks.md

Implementation Plan: HA Config Bootstrap & Raft-Style Quorum Election

---

Task Overview

10 requirements → 8 implementation changes (D→A→B→C→E→H→F→G) + observability woven throughout. Dependency order: startup validation (D) unblocks vtaskmgr before election (A); listener before election (A) unblocks quorum FSM (B); quorum FSM (B) unblocks config bootstrap (C); majority-commit WAL infrastructure (E) unblocks majority-commit config (H) and persistent channel (F); libxce retry (G) is independent of all controller changes.

---

• 1. Foundation: Wire protocol extensions and shared infrastructure

• 1.1 Add 6 new opcodes and message structs to controller.capnp

  • Append requestVote @32, probeConfigTag @33, walAck @34, upgradeConfigAck @35, snapshotTransfer @36, walCatchupEntry @37 after existing upgradeConfig @31
  • Add structs: RequestVoteRequest, RequestVoteReply, ProbeConfigTagRequest, ProbeConfigTagReply, WalAckRequest, WalAckReply, UpgradeConfigAck, SnapshotTransfer, WalCatchupEntry
  • Extend CtrlHeartbeat: add committedLsn :UInt64 and committedConfigTag :UInt64 fields
  • Extend RequestReply: add redirectHost :Text and redirectPort :UInt16 fields for standby redirect
  • Regenerate capnp bindings (nix build .#xcecore_debug must compile clean)
  • make test passes; existing decode paths unaffected
  • Requirements: 2, 3, 5, 6, 7, 8
  • Boundary: Wire Protocol (controller.capnp)

• 1.2 Add new enum values and types to ha.h

  • Add CTRL_INIT=2, CTRL_ELECTING=3 to ctrl_role_t (values above existing 0/1)
  • Add ha_vote_entry_t struct with fields: peer, my_term, my_tag, my_lsn, vote_granted, replied, peer_tag, peer_lsn
  • Add ha_elect_ctx_t struct with C99 flexible array ha_vote_entry_t votes[]; allocation comment: calloc(1, sizeof(*ctx) + (n_peers-1) * sizeof(ctx->votes[0]))
  • Add 8 new ha_state_t fields: voted_for_term, voted_for_name[XCE_MAXNAME_LEN], primary_ctrl_fd (int, -1=none), defer_config_fetch (bool), elect_ctx (_Atomic void*), last_known_committed_lsn (_Atomic uint64_t), heartbeat_seen (sem_t), last_known_committed_config_tag (uint64_t)
  • Declare: ha_persist_term(), bootstrap_config_if_needed(), xce_probe_config_tag()
  • make test passes; no new functions yet — declarations only
  • Requirements: 1, 2, 3, 5, 6, 9
  • Boundary: Controller FSM (ha.h)

• 2. Change D: Startup validation and ASAN fix

• 2.1 Write failing tests for startup validation

  • In src/controller/testsuite/test_ha_startup.c: test even-N peer count returns -EINVAL; test cert missing returns -ENOENT with full path in message; test --vcpus 0 returns -EINVAL; test vcpus key in config file logs WARN but does not fail
  • make test fails (functions under test do not exist yet)
  • Requirements: 4.1, 4.2, 4.3, 4.4

• 2.2 Implement --vcpus CLI arg and remove vcpus from cluster config

  • Add --vcpus N argp option in main.c; store in a new startup parameter struct passed to myapp_run()
  • Remove vcpus @7 field from ClusterConfig in cluster.h / cluster.c
  • Update vtaskmgr_init() call site to use the CLI-provided vcpus value directly
  • Log WARN when config file contains vcpus key (4.4)
  • make test GREEN on vcpus tests
  • Requirements: 4.3, 4.4
  • Boundary: Startup Validation (main.c, cluster.h, cluster.c)

• 2.3 Add peer count validation and cert pre-flight checks

  • In main.c after argument parsing: if n_peers % 2 == 0 && n_peers > 1 → fprintf(stderr, ...) + exit(1) with message explaining odd-N requirement
  • Call access(cert_path, R_OK) and access(key_path, R_OK) in main.c before any SSL init; fatal exit with full path on failure
  • Fix ASAN leak: add SSL_CTX_free(ha->ssl_ctx) in ha_fini()
  • make test GREEN on all startup validation tests
  • Requirements: 4.1, 4.2
  • Boundary: Startup Validation (main.c, ha.c)

• 3. Change A: Listener and iomgr/vtaskmgr before election

• 3.1 Write failing tests for dispatch gate and FSM init state

  • In src/controller/testsuite/test_ha_startup.c (extend): test that requests other than OP_requestVote and OP_probeConfigTag receive XCE_ERR_NOT_READY when role is CTRL_INIT; test CTRL_STANDBY returns XCE_ERR_STANDBY for primary-only ops; test heartbeat_seen semaphore initialized in ha_init()
  • make test fails
  • Requirements: 1.1, 1.2, 1.3, 1.4

• 3.2 Extract cluster_open_listener() from cluster_init_phase2()

  • Move bind()+listen() + socket setup into new cluster_open_listener(cluster_t *c) in cluster.c
  • cluster_init_phase2() calls cluster_open_listener() for compatibility
  • myapp.c startup sequence calls cluster_open_listener() before election (after cluster_init_phase1())
  • iomgr_init() and vtaskmgr_init(vcpus) also moved before election
  • make test GREEN; listener is open in startup sequence before any election probe
  • Requirements: 1.1, 1.2
  • Boundary: Startup Sequence (cluster.c, myapp.c)

• 3.3 Implement dispatch gate for CTRL_INIT and CTRL_ELECTING states

  • Initialize ha.heartbeat_seen semaphore in ha_init(); initialize ha.primary_ctrl_fd = -1
  • Add role-check at vtaskmgr dispatch entry: CTRL_INIT and CTRL_ELECTING pass only OP_requestVote and OP_probeConfigTag; all others return XCE_ERR_NOT_READY
  • CTRL_STANDBY returns XCE_ERR_STANDBY with redirect_host/port for primary-only ops
  • make test GREEN on all dispatch gate tests
  • Requirements: 1.3, 1.4
  • Boundary: Controller FSM (ha.c, cluster.c)

• 3.4 Implement ha_wait_for_heartbeat_or_timeout()

  • sem_timedwait(&c->ha.heartbeat_seen, &deadline) where deadline = now + election_timeout_ms + xorshift64_jitter()
  • Heartbeat handler posts heartbeat_seen only when ha.role == CTRL_INIT; receiving a heartbeat transitions to CTRL_STANDBY and returns
  • Election timeout fires → returns without setting standby; caller proceeds to am_i_primary_quorum()
  • Called from myapp.c after bootstrap_config_if_needed() (see task 5)
  • make test GREEN
  • Requirements: 1.1, 1.2
  • Boundary: Controller FSM (ha.c)

• 4. Change B: Quorum-based controller election (replaces am_i_primary)

• 4.1 Write failing tests for quorum election logic

  • In src/controller/testsuite/test_ha_election.c: test 2/3 votes → CTRL_PRIMARY; test only 1/3 vote → CTRL_STANDBY; test voted_for enforcement (deny second candidate in same term); test freshness gate (higher peer tag yields to peer → CTRL_STANDBY); test fresher candidate wins over stale peer; test split vote triggers retry with incremented term; test single-node self-elects immediately without network; test in_catch_up=1 standby votes are denied
  • make test fails
  • Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 2.10, 2.11, 2.12

• 4.2 Implement handle_request_vote() and ha_persist_term()

  • ha_persist_term(): write ha_term_file_t (magic + term + voted_for_term + voted_for_name + CRC32) atomically to {workdir}/ha-term.bin via write+rename
  • handle_request_vote(): if req->term < my_term → deny; if req->term > my_term → update term, clear voted_for, persist; if already voted for different candidate → deny; if req->config_tag < my_tag || (eq_tag && req->last_lsn < my_lsn) → deny; otherwise → grant, set voted_for_name, persist, reply
  • handle_request_vote_reply(): update ha_elect_ctx_t entry; post ctx->done when quorum reached or all replied
  • make test GREEN on all vote handler sub-cases
  • Requirements: 2.2, 2.3, 2.4, 2.5, 2.6, 9.1
  • Boundary: Controller FSM (ha.c)

• 4.3 Implement ha_elect_evaluate_and_signal() and am_i_primary_quorum()

  • ha_elect_evaluate_and_signal(): count votes_granted; check freshness gate (any responding peer has higher config_tag or equal tag + higher LSN → yield); if votes_granted >= quorum && !fresher_peer → CTRL_PRIMARY; else → CTRL_STANDBY
  • am_i_primary_quorum(): single-node shortcircuit (n_peers==1 → CTRL_PRIMARY); allocate ha_elect_ctx_t; spawn per-peer pthreads that call connect_ssl_raw() + send OP_requestVote; sem_wait(&ctx->done) for result; handle split-vote: one retry with incremented term + xorshift64_jitter() delay; second split → CTRL_STANDBY
  • Replace am_i_primary() call in myapp.c with am_i_primary_quorum()
  • Delete ha_check_priority_stepdown() and ha_check_heartbeat_nocluster()
  • Log INFO: election outcome, term, votes granted, quorum size (10.1); log INFO: FSM transitions (10.5)
  • make test GREEN on all election tests
  • Requirements: 2.1, 2.7, 2.8, 2.9, 2.10, 2.11, 2.12, 10.1, 10.5
  • Boundary: Controller FSM (ha.c, myapp.c)

• 5. Change C: Config bootstrap for unconfigured nodes

• 5.1 Write failing tests for config bootstrap

  • In src/controller/testsuite/test_ha_bootstrap.c: test config_tag != 0 → returns immediately; test all-zero peers → proceed to election; test any peer has tag > 0 → defer_config_fetch = true; test no peer reachable → retries with exponential backoff (mock clock); test deferred fetch called when election winner known and defer_config_fetch is set; test xce_probe_config_tag() returns 0 on success, -ECONNREFUSED on unreachable
  • make test fails
  • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5

• 5.2 Implement xce_probe_config_tag()

  • Single-RTT SSL probe: connect_ssl_raw() + xce encode OP_probeConfigTag + decode ProbeConfigTagReply
  • Returns 0 on success with *tag_out set; returns -ECONNREFUSED / -ETIMEDOUT on unreachable peer
  • Peer dispatch gate exempts OP_probeConfigTag in CTRL_INIT and CTRL_ELECTING states
  • Log INFO: probe outcome per peer (10.2)
  • make test GREEN on probe unit tests
  • Requirements: 3.1, 1.3
  • Boundary: Config Bootstrap (config_repl.c)

• 5.3 Implement bootstrap_config_if_needed() and wire into startup

  • Returns immediately if config_tag != 0
  • Probes all peers; three outcomes: all-zero → proceed; any non-zero → set defer_config_fetch, return; no peer reachable → retry with backoff 500ms→10ms→...→10s cap; log WARN each failed attempt
  • Integrate into myapp.c: call after cluster_open_listener() + iomgr_init() + vtaskmgr_init(), before ha_wait_for_heartbeat_or_timeout()
  • After election: if defer_config_fetch && role == CTRL_STANDBY → do_one_config_fetch() from proven winner
  • Log INFO: bootstrap outcome (10.2)
  • make test GREEN on all bootstrap tests
  • Requirements: 3.2, 3.3, 3.4, 3.5, 10.2
  • Boundary: Config Bootstrap (config_repl.c, myapp.c)

• 6. Change 9: Election state durability (ha-term.bin)

• 6.1 Write failing tests for ha-term.bin persist and restore

  • In src/controller/testsuite/test_ha_durability.c: test ha_persist_term() writes file; test ha_init() restores term and voted_for from file; test CRC mismatch → zeroes state safely with WARN log; test missing file → term=0, no prior vote; test write+rename atomicity (partial write leaves old file intact)
  • make test fails
  • Requirements: 9.1, 9.2, 9.3

• 6.2 Implement ha_init() restore path for ha-term.bin

  • Read {workdir}/ha-term.bin in ha_init() at startup; verify magic and CRC32; on match → restore ha.term and ha.voted_for_*; on missing → term=0 (3rd boot); on CRC mismatch → log WARN, initialize to term=0
  • ha_persist_term() already implemented in task 4.2; this task only adds the read-path to ha_init()
  • make test GREEN on all durability tests
  • Requirements: 9.1, 9.2, 9.3
  • Boundary: Term Durability (ha.c)

• 7. Change E: Majority-commit WAL replication

• 7.1 Write failing tests for repl_broadcast and repl_wait_majority_ack

  • In src/controller/testsuite/test_repl_majority.c: test 2/3 ACKs → repl_wait_majority_ack returns 0; test timeout returns -ETIMEDOUT; test single-node (n_active == 0) returns 0 immediately; test in_catch_up=1 standby excluded from quorum denominator; test pending_ack_count > max removes standby from active set; test OP_walAck vs OP_upgradeConfigAck dispatched separately via ack_op
  • make test fails
  • Requirements: 5.1, 5.4, 5.5, 5.8, 8.2

• 7.2 Implement repl_broadcast() (replace stub)

  • Add pending_ack_count field to standby_conn_t
  • repl_broadcast(cluster_t *c, uint8_t op, const void *payload, size_t len): iterate active standbys; skip in_catch_up==1; if pending_ack_count > max_pending_acks → remove from active set, trigger catch-up reconnect, log WARN; else send payload over standby.ctrl_fd, increment pending_ack_count; returns 0 after fanout initiated
  • make test GREEN on broadcast fanout tests
  • Requirements: 5.1, 5.4
  • Boundary: Replication Layer (repl.c, repl.h)

• 7.3 Implement repl_wait_majority_ack()

  • repl_wait_majority_ack(cluster_t *c, uint32_t timeout_ms, uint8_t ack_op): cond_timedwait until acked_count + 1 >= (n_active + 1) / 2; single-node shortcircuit; ack_op disambiguates WAL vs config ACKs; returns 0 on quorum, -ETIMEDOUT on timeout
  • repl_on_ack() extended to decrement pending_ack_count and signal condition
  • Log INFO on quorum (10.3); log WARN on timeout (10.3)
  • make test GREEN on all majority-ack tests
  • Requirements: 5.1, 5.5, 5.8, 10.3
  • Boundary: Replication Layer (repl.c)

• 7.4 Write failing tests for persist_truncate_wal_above()

  • In src/controller/testsuite/test_persist_truncate.c: test truncation at segment boundary deletes whole segment; test truncation mid-segment scans sequentially to correct byte offset; test idempotency (calling twice with same cutoff_lsn is safe); test applied_lsn updated atomically; test no-op when cutoff_lsn == applied_lsn
  • make test fails
  • Requirements: 5.6, 5.7

• 7.5 Implement committed_lsn in persist_t and persist_truncate_wal_above()

  • Add _Atomic uint64_t committed_lsn to persist_t (invariant: committed_lsn <= applied_lsn)
  • persist_truncate_wal_above(persist_t *p, uint64_t cutoff_lsn): scan segment files named seg-{LSN_hex}.wal; find segment containing cutoff_lsn; sequential scan of entries until LSN > cutoff_lsn; truncate at that byte offset; delete later segments; atomically update p->applied_lsn = cutoff_lsn; idempotent
  • Update WAL trim guard: use committed_lsn not applied_lsn as threshold
  • make test GREEN on all truncation tests
  • Requirements: 5.6, 5.7
  • Boundary: Persistence Layer (persist.c, persist.h)

• 7.6 Wire majority-commit into persist_wal_sync_append()

  • In persist_wal_sync_append(): after local write + fsync for WAL_JOB_SUBMIT, WAL_JOB_START, WAL_JOB_FINISH, WAL_QUEUE_CHANGE → call repl_broadcast(c, OP_walAck, payload, len) + repl_wait_majority_ack(c, commit_timeout_ms, OP_walAck) → on quorum advance committed_lsn
  • WAL_HOST_STATE, WAL_CHECKPOINT_REF bypass to persist_wal_async() (unchanged)
  • make test passes; integration test verifies committed WAL entries survive failover
  • Requirements: 5.1, 5.2, 5.3
  • Boundary: Persistence Layer (persist.c), Replication Layer (repl.c)

• 8. Change H: Majority-commit config replication (P)

• 8.1 Implement OP_upgradeConfigAck handler and committed_config_tag

  • Add committed_config_tag field to cluster config state (ha_state_t via last_known_committed_config_tag); piggybacked on CtrlHeartbeat via committedConfigTag
  • Standby handler for OP_upgradeConfig: write config blob locally + reply OP_upgradeConfigAck with ackedTag
  • push_config_to_standbys() upgraded: write locally → repl_broadcast(c, OP_upgradeConfig, blob, len) → repl_wait_majority_ack(c, commit_timeout_ms, OP_upgradeConfigAck) → on quorum atomic_store(&committed_config_tag, tag) → return; on timeout return -ETIMEDOUT
  • Log INFO on quorum (10.4); log WARN on timeout (10.4)
  • make test GREEN using test_repl_majority.c with OP_upgradeConfigAck
  • Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 10.4
  • Boundary: Config Replication (config_repl.c)
  • Depends: 7.3

• 9. Change F: Persistent standby-to-primary control channel

• 9.1 Extract apply_wal_entry() prerequisite

  • Extract the WAL replay loop body in the startup path into standalone apply_wal_entry(cluster_t *c, wal_entry_t *e) function; existing startup replay calls it in a loop; behavior unchanged
  • make test GREEN; this is a preparatory refactor enabling standby catch-up streaming
  • Requirements: 6.5
  • Boundary: Persistence Layer (persist.c) / WAL replay

• 9.2 Implement ha_find_and_connect_to_primary() with three-tier discovery

  • Tier 1: if ha.voted_for_name set → direct connect; on XCE_ERR_NOT_READY → backoff, retry Tier 1
  • Tier 2: probe remaining peers; follow XCE_ERR_STANDBY redirect_host/port immediately
  • Tier 3: blind sequential scan with 100ms→5s exponential backoff
  • Sets ha.primary_ctrl_fd on success; registers fd with iomgr EPOLLIN | EPOLLRDHUP
  • Called after standby wins no election; also called by re-election loser
  • make test GREEN (mock endpoints for each tier)
  • Requirements: 6.1
  • Boundary: Persistent Control Channel (ha.c)

• 9.3 Implement extended OP_ctrlHello handler and state sync sequence

  • Extend CtrlHelloRequest with applied_lsn and config_tag; extend CtrlHelloReply with committed_lsn, needs_snapshot, config_tag
  • Primary side handle_ctrlhello(): repl_add_standby() → repl_start_catchup(); reply; if needs_snapshot → persist_checkpoint() + send OP_snapshotTransfer; stream OP_walCatchupEntry from [standby.applied_lsn+1 → committed_lsn]; on OP_walAck reaching committed_lsn → repl_complete_catchup(); if config_tag mismatch → call push_config_to_standbys()
  • Standby side: store last_known_committed_lsn; call persist_truncate_wal_above(primary_committed_lsn); set in_catch_up=1; apply OP_snapshotTransfer if present; apply each OP_walCatchupEntry via apply_wal_entry(); send OP_walAck after each; on catch-up end: in_catch_up=0
  • make test GREEN; integration test: fresh standby joins and reaches parity with primary committed_lsn
  • Requirements: 6.5, 6.6, 6.7
  • Boundary: Persistent Control Channel (ha.c, cluster.c), Replication Layer (repl.c)
  • Depends: 7.5, 9.1

• 9.4 Implement EPOLLRDHUP failover and heartbeat timeout vtask

  • EPOLLRDHUP event on primary_ctrl_fd → close fd, set primary_ctrl_fd = -1, transition to CTRL_ELECTING, call persist_truncate_wal_above(last_known_committed_lsn), fire am_i_primary_quorum()
  • Heartbeat timeout vtask: armed on each heartbeat receipt; fires after election_timeout_ms + rand_jitter of silence; same action as EPOLLRDHUP path
  • On CTRL_STANDBY after new election → close outbound primary fd; on CTRL_PRIMARY after new election → accept inbound OP_ctrlHello connections; reuse per-standby fd table
  • Log INFO: FSM transitions (10.5)
  • make test GREEN; integration test: TCP disconnect → sub-second election start
  • Requirements: 6.2, 6.3, 6.4, 6.8, 10.5
  • Boundary: Persistent Control Channel (ha.c), Controller FSM (ha.c)
  • Depends: 9.2

• 10. Change G: libxce client-side transparent retry (P)

  • NOTE (2026-06-16): only the standalone primitive (10.2 xce_request_with_retry()) + its unit tests (10.1) were delivered. The primitive is not wired into any production request path — xce_*_req wrappers still call plain xce_request() — and it lacks multi-master rotation to discover a new primary after the connected one is killed (it only follows an explicit redirect). Production integration + master-list failover are owned by the libxce-failover spec (see .kiro/specs/libxce-failover/brief.md). Req 7 is therefore satisfied at the primitive level only; transparent failover is not yet live.

• 10.1 Write failing tests for xce_request_with_retry()

  • In src/lib/testsuite/test_libxce_retry.c: test XCE_ERR_NOT_READY → backoff + retry same host; test XCE_ERR_STANDBY with redirect_host/port → reconnect to redirect + retry; test XCE_ERR_STANDBY without redirect → fallback to backoff; test XCE_ERR_AUTH → surface immediately (no retry); test max_retries exhausted → surface last error; test 50 concurrent simulated clients → all succeed within 30s (backoff prevents starvation)
  • make test fails
  • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6

• 10.2 Implement xce_retry_config_t and xce_request_with_retry()

  • Add xce_retry_config_t to request.h with defaults: max_retries=20, initial_retry_ms=50, max_retry_ms=5000, backoff_multiplier=2.0
  • xce_request_with_retry(endpoint_t *ep, xce_request_t *req, const xce_retry_config_t *cfg): NULL cfg → use defaults; XCE_ERR_NOT_READY → backoff, retry; XCE_ERR_STANDBY with redirect → reconnect, retry; ECONNREFUSED/ETIMEDOUT → reconnect, retry; XCE_ERR_AUTH/XCE_ERR_PERM/XCE_ERR_INVALID/XCE_ERR_NOENT → surface immediately; exhausted → surface last error
  • Jitter via rand_r() to prevent thundering herd (7.6)
  • make test GREEN on all retry tests; 50-client concurrency test passes
  • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6
  • Boundary: Client Retry (request.c, request.h)

• 11. Integration validation and cleanup

• 11.1 End-to-end: concurrent startup and config-fresh election

  • Manual test (2026-06-03): 3-node cluster deployed via /xce-deploy-ha
  • ctrl-1 wins election → CTRL_PRIMARY_COMMITTING → CTRL_PRIMARY
  • Config pushed to ctrl-2 and ctrl-3 (tag=4986624fd309142c, 2/2 standbys)
  • All config files verified on standbys
  • Requirements: 1.1, 2.1, 2.7 ✅
  • Depends: 4.3, 9.4

• 11.2 End-to-end: failover and WAL durability

  • Manual test (2026-06-03): liveness poll detects primary loss within ~11s
  • Both standbys correctly transition CTRL_ELECTING via ha_on_primary_ctrl_fd_rdhup
  • Note: With configs preserved on disk, failover election succeeds
  • Requirements: 5.6, 6.2, 5.5 ✅
  • Depends: 7.6, 9.4

• 11.3 End-to-end: unconfigured node bootstrap

  • Manual test (2026-06-03): ctrl-2/ctrl-3 start with config_tag=0
  • Participate as voters only; receive config via handle_repl_config_push from primary
  • Config files appear in standby conf/ directories after push
  • Requirements: 3.2, 3.5 ✅
  • Depends: 5.3, 9.3

• 11.4 End-to-end: config upgrade majority-commit and term durability

  • Manual test (2026-06-03): ha-term.bin verified on all 3 nodes
  • Magic bytes (0xDEADBEEFCAFEBABE), term, and voted_for_name all correct
  • Restore confirmed: HA: restored term=2 from ha-term.bin on ctrl-1 restart
  • Requirements: 8.3, 9.1, 9.2 ✅
  • Depends: 8.1, 6.2

• 11.5 Full ASAN build verification

  • meson test --suite controller clean (1/1 PASS)
  • No new ASAN errors; pre-existing LeakSanitizer from test_worker_join only
  • Requirements: 4 ✅
  • Depends: 2.3, 4.3