lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/ha-config-bootstrap/tasks.md
Rongsong Shen 403dd17f8b test(ha-config-bootstrap): RED phase tests for quorum election logic (task 4.1) 
- 21 test functions covering all 12 acceptance criteria (req 2.1-2.12)
- Tests: 2/3 quorum wins, 1/3 loses, voted_for enforcement, freshness gate,
  single-node self-elects, split-vote retry, in_catch_up denial, jitter
- Build fails with undefined reference to handle_request_vote,
  ha_elect_evaluate_and_signal, am_i_primary_quorum (expected RED proof)
2026-05-28 07:10:02 +00:00

22 KiB
Raw Blame History

Implementation Plan: HA Config Bootstrap & Raft-Style Quorum Election

Task Overview

10 requirements → 8 implementation changes (D→A→B→C→E→H→F→G) + observability woven throughout. Dependency order: startup validation (D) unblocks vtaskmgr before election (A); listener before election (A) unblocks quorum FSM (B); quorum FSM (B) unblocks config bootstrap (C); majority-commit WAL infrastructure (E) unblocks majority-commit config (H) and persistent channel (F); libxce retry (G) is independent of all controller changes.

  • 1. Foundation: Wire protocol extensions and shared infrastructure

  • 1.1 Add 6 new opcodes and message structs to controller.capnp

    • Append requestVote @32, probeConfigTag @33, walAck @34, upgradeConfigAck @35, snapshotTransfer @36, walCatchupEntry @37 after existing upgradeConfig @31
    • Add structs: RequestVoteRequest, RequestVoteReply, ProbeConfigTagRequest, ProbeConfigTagReply, WalAckRequest, WalAckReply, UpgradeConfigAck, SnapshotTransfer, WalCatchupEntry
    • Extend CtrlHeartbeat: add committedLsn :UInt64 and committedConfigTag :UInt64 fields
    • Extend RequestReply: add redirectHost :Text and redirectPort :UInt16 fields for standby redirect
    • Regenerate capnp bindings (nix build .#xcecore_debug must compile clean)
    • make test passes; existing decode paths unaffected
    • Requirements: 2, 3, 5, 6, 7, 8
    • Boundary: Wire Protocol (controller.capnp)

  • 1.2 Add new enum values and types to ha.h

    • Add CTRL_INIT=2, CTRL_ELECTING=3 to ctrl_role_t (values above existing 0/1)
    • Add ha_vote_entry_t struct with fields: peer, my_term, my_tag, my_lsn, vote_granted, replied, peer_tag, peer_lsn
    • Add ha_elect_ctx_t struct with C99 flexible array ha_vote_entry_t votes[]; allocation comment: calloc(1, sizeof(*ctx) + (n_peers-1) * sizeof(ctx->votes[0]))
    • Add 8 new ha_state_t fields: voted_for_term, voted_for_name[XCE_MAXNAME_LEN], primary_ctrl_fd (int, -1=none), defer_config_fetch (bool), elect_ctx (_Atomic void*), last_known_committed_lsn (_Atomic uint64_t), heartbeat_seen (sem_t), last_known_committed_config_tag (uint64_t)
    • Declare: ha_persist_term(), bootstrap_config_if_needed(), xce_probe_config_tag()
    • make test passes; no new functions yet — declarations only
    • Requirements: 1, 2, 3, 5, 6, 9
    • Boundary: Controller FSM (ha.h)

  • 2. Change D: Startup validation and ASAN fix

  • 2.1 Write failing tests for startup validation

    • In src/controller/testsuite/test_ha_startup.c: test even-N peer count returns -EINVAL; test cert missing returns -ENOENT with full path in message; test --vcpus 0 returns -EINVAL; test vcpus key in config file logs WARN but does not fail
    • make test fails (functions under test do not exist yet)
    • Requirements: 4.1, 4.2, 4.3, 4.4

  • 2.2 Implement --vcpus CLI arg and remove vcpus from cluster config

    • Add --vcpus N argp option in main.c; store in a new startup parameter struct passed to myapp_run()
    • Remove vcpus @7 field from ClusterConfig in cluster.h / cluster.c
    • Update vtaskmgr_init() call site to use the CLI-provided vcpus value directly
    • Log WARN when config file contains vcpus key (4.4)
    • make test GREEN on vcpus tests
    • Requirements: 4.3, 4.4
    • Boundary: Startup Validation (main.c, cluster.h, cluster.c)

  • 2.3 Add peer count validation and cert pre-flight checks

    • In main.c after argument parsing: if n_peers % 2 == 0 && n_peers > 1 → fprintf(stderr, ...) + exit(1) with message explaining odd-N requirement
    • Call access(cert_path, R_OK) and access(key_path, R_OK) in main.c before any SSL init; fatal exit with full path on failure
    • Fix ASAN leak: add SSL_CTX_free(ha->ssl_ctx) in ha_fini()
    • make test GREEN on all startup validation tests
    • Requirements: 4.1, 4.2
    • Boundary: Startup Validation (main.c, ha.c)

  • 3. Change A: Listener and iomgr/vtaskmgr before election

  • 3.1 Write failing tests for dispatch gate and FSM init state

    • In src/controller/testsuite/test_ha_startup.c (extend): test that requests other than OP_requestVote and OP_probeConfigTag receive XCE_ERR_NOT_READY when role is CTRL_INIT; test CTRL_STANDBY returns XCE_ERR_STANDBY for primary-only ops; test heartbeat_seen semaphore initialized in ha_init()
    • make test fails
    • Requirements: 1.1, 1.2, 1.3, 1.4

  • 3.2 Extract cluster_open_listener() from cluster_init_phase2()

    • Move bind()+listen() + socket setup into new cluster_open_listener(cluster_t *c) in cluster.c
    • cluster_init_phase2() calls cluster_open_listener() for compatibility
    • myapp.c startup sequence calls cluster_open_listener() before election (after cluster_init_phase1())
    • iomgr_init() and vtaskmgr_init(vcpus) also moved before election
    • make test GREEN; listener is open in startup sequence before any election probe
    • Requirements: 1.1, 1.2
    • Boundary: Startup Sequence (cluster.c, myapp.c)

  • 3.3 Implement dispatch gate for CTRL_INIT and CTRL_ELECTING states

    • Initialize ha.heartbeat_seen semaphore in ha_init(); initialize ha.primary_ctrl_fd = -1
    • Add role-check at vtaskmgr dispatch entry: CTRL_INIT and CTRL_ELECTING pass only OP_requestVote and OP_probeConfigTag; all others return XCE_ERR_NOT_READY
    • CTRL_STANDBY returns XCE_ERR_STANDBY with redirect_host/port for primary-only ops
    • make test GREEN on all dispatch gate tests
    • Requirements: 1.3, 1.4
    • Boundary: Controller FSM (ha.c, cluster.c)

  • 3.4 Implement ha_wait_for_heartbeat_or_timeout()

    • sem_timedwait(&c->ha.heartbeat_seen, &deadline) where deadline = now + election_timeout_ms + xorshift64_jitter()
    • Heartbeat handler posts heartbeat_seen only when ha.role == CTRL_INIT; receiving a heartbeat transitions to CTRL_STANDBY and returns
    • Election timeout fires → returns without setting standby; caller proceeds to am_i_primary_quorum()
    • Called from myapp.c after bootstrap_config_if_needed() (see task 5)
    • make test GREEN
    • Requirements: 1.1, 1.2
    • Boundary: Controller FSM (ha.c)

  • 4. Change B: Quorum-based controller election (replaces am_i_primary)

  • 4.1 Write failing tests for quorum election logic

    • In src/controller/testsuite/test_ha_election.c: test 2/3 votes → CTRL_PRIMARY; test only 1/3 vote → CTRL_STANDBY; test voted_for enforcement (deny second candidate in same term); test freshness gate (higher peer tag yields to peer → CTRL_STANDBY); test fresher candidate wins over stale peer; test split vote triggers retry with incremented term; test single-node self-elects immediately without network; test in_catch_up=1 standby votes are denied
    • make test fails
    • Requirements: 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 2.10, 2.11, 2.12

  • 4.2 Implement handle_request_vote() and ha_persist_term()

    • ha_persist_term(): write ha_term_file_t (magic + term + voted_for_term + voted_for_name + CRC32) atomically to {workdir}/ha-term.bin via write+rename
    • handle_request_vote(): if req->term < my_term → deny; if req->term > my_term → update term, clear voted_for, persist; if already voted for different candidate → deny; if req->config_tag < my_tag || (eq_tag && req->last_lsn < my_lsn) → deny; otherwise → grant, set voted_for_name, persist, reply
    • handle_request_vote_reply(): update ha_elect_ctx_t entry; post ctx->done when quorum reached or all replied
    • make test GREEN on all vote handler sub-cases
    • Requirements: 2.2, 2.3, 2.4, 2.5, 2.6, 9.1
    • Boundary: Controller FSM (ha.c)

  • 4.3 Implement ha_elect_evaluate_and_signal() and am_i_primary_quorum()

    • ha_elect_evaluate_and_signal(): count votes_granted; check freshness gate (any responding peer has higher config_tag or equal tag + higher LSN → yield); if votes_granted >= quorum && !fresher_peer → CTRL_PRIMARY; else → CTRL_STANDBY
    • am_i_primary_quorum(): single-node shortcircuit (n_peers==1 → CTRL_PRIMARY); allocate ha_elect_ctx_t; spawn per-peer pthreads that call connect_ssl_raw() + send OP_requestVote; sem_wait(&ctx->done) for result; handle split-vote: one retry with incremented term + xorshift64_jitter() delay; second split → CTRL_STANDBY
    • Replace am_i_primary() call in myapp.c with am_i_primary_quorum()
    • Delete ha_check_priority_stepdown() and ha_check_heartbeat_nocluster()
    • Log INFO: election outcome, term, votes granted, quorum size (10.1); log INFO: FSM transitions (10.5)
    • make test GREEN on all election tests
    • Requirements: 2.1, 2.7, 2.8, 2.9, 2.10, 2.11, 2.12, 10.1, 10.5
    • Boundary: Controller FSM (ha.c, myapp.c)

  • 5. Change C: Config bootstrap for unconfigured nodes

  • 5.1 Write failing tests for config bootstrap

    • In src/controller/testsuite/test_ha_bootstrap.c: test config_tag != 0 → returns immediately; test all-zero peers → proceed to election; test any peer has tag > 0 → defer_config_fetch = true; test no peer reachable → retries with exponential backoff (mock clock); test deferred fetch called when election winner known and defer_config_fetch is set; test xce_probe_config_tag() returns 0 on success, -ECONNREFUSED on unreachable
    • make test fails
    • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5

  • 5.2 Implement xce_probe_config_tag()

    • Single-RTT SSL probe: connect_ssl_raw() + xce encode OP_probeConfigTag + decode ProbeConfigTagReply
    • Returns 0 on success with *tag_out set; returns -ECONNREFUSED / -ETIMEDOUT on unreachable peer
    • Peer dispatch gate exempts OP_probeConfigTag in CTRL_INIT and CTRL_ELECTING states
    • Log INFO: probe outcome per peer (10.2)
    • make test GREEN on probe unit tests
    • Requirements: 3.1, 1.3
    • Boundary: Config Bootstrap (config_repl.c)

  • 5.3 Implement bootstrap_config_if_needed() and wire into startup

    • Returns immediately if config_tag != 0
    • Probes all peers; three outcomes: all-zero → proceed; any non-zero → set defer_config_fetch, return; no peer reachable → retry with backoff 500ms→10ms→...→10s cap; log WARN each failed attempt
    • Integrate into myapp.c: call after cluster_open_listener() + iomgr_init() + vtaskmgr_init(), before ha_wait_for_heartbeat_or_timeout()
    • After election: if defer_config_fetch && role == CTRL_STANDBY → do_one_config_fetch() from proven winner
    • Log INFO: bootstrap outcome (10.2)
    • make test GREEN on all bootstrap tests
    • Requirements: 3.2, 3.3, 3.4, 3.5, 10.2
    • Boundary: Config Bootstrap (config_repl.c, myapp.c)

  • 6. Change 9: Election state durability (ha-term.bin)

  • 6.1 Write failing tests for ha-term.bin persist and restore

    • In src/controller/testsuite/test_ha_durability.c: test ha_persist_term() writes file; test ha_init() restores term and voted_for from file; test CRC mismatch → zeroes state safely with WARN log; test missing file → term=0, no prior vote; test write+rename atomicity (partial write leaves old file intact)
    • make test fails
    • Requirements: 9.1, 9.2, 9.3

  • 6.2 Implement ha_init() restore path for ha-term.bin

    • Read {workdir}/ha-term.bin in ha_init() at startup; verify magic and CRC32; on match → restore ha.term and ha.voted_for_*; on missing → term=0 (3rd boot); on CRC mismatch → log WARN, initialize to term=0
    • ha_persist_term() already implemented in task 4.2; this task only adds the read-path to ha_init()
    • make test GREEN on all durability tests
    • Requirements: 9.1, 9.2, 9.3
    • Boundary: Term Durability (ha.c)

  • 7. Change E: Majority-commit WAL replication

  • 7.1 Write failing tests for repl_broadcast and repl_wait_majority_ack

    • In src/controller/testsuite/test_repl_majority.c: test 2/3 ACKs → repl_wait_majority_ack returns 0; test timeout returns -ETIMEDOUT; test single-node (n_active == 0) returns 0 immediately; test in_catch_up=1 standby excluded from quorum denominator; test pending_ack_count > max removes standby from active set; test OP_walAck vs OP_upgradeConfigAck dispatched separately via ack_op
    • make test fails
    • Requirements: 5.1, 5.4, 5.5, 5.8, 8.2

  • 7.2 Implement repl_broadcast() (replace stub)

    • Add pending_ack_count field to standby_conn_t
    • repl_broadcast(cluster_t *c, uint8_t op, const void *payload, size_t len): iterate active standbys; skip in_catch_up==1; if pending_ack_count > max_pending_acks → remove from active set, trigger catch-up reconnect, log WARN; else send payload over standby.ctrl_fd, increment pending_ack_count; returns 0 after fanout initiated
    • make test GREEN on broadcast fanout tests
    • Requirements: 5.1, 5.4
    • Boundary: Replication Layer (repl.c, repl.h)

  • 7.3 Implement repl_wait_majority_ack()

    • repl_wait_majority_ack(cluster_t *c, uint32_t timeout_ms, uint8_t ack_op): cond_timedwait until acked_count + 1 >= (n_active + 1) / 2; single-node shortcircuit; ack_op disambiguates WAL vs config ACKs; returns 0 on quorum, -ETIMEDOUT on timeout
    • repl_on_ack() extended to decrement pending_ack_count and signal condition
    • Log INFO on quorum (10.3); log WARN on timeout (10.3)
    • make test GREEN on all majority-ack tests
    • Requirements: 5.1, 5.5, 5.8, 10.3
    • Boundary: Replication Layer (repl.c)

  • 7.4 Write failing tests for persist_truncate_wal_above()

    • In src/controller/testsuite/test_persist_truncate.c: test truncation at segment boundary deletes whole segment; test truncation mid-segment scans sequentially to correct byte offset; test idempotency (calling twice with same cutoff_lsn is safe); test applied_lsn updated atomically; test no-op when cutoff_lsn == applied_lsn
    • make test fails
    • Requirements: 5.6, 5.7

  • 7.5 Implement committed_lsn in persist_t and persist_truncate_wal_above()

    • Add _Atomic uint64_t committed_lsn to persist_t (invariant: committed_lsn <= applied_lsn)
    • persist_truncate_wal_above(persist_t *p, uint64_t cutoff_lsn): scan segment files named seg-{LSN_hex}.wal; find segment containing cutoff_lsn; sequential scan of entries until LSN > cutoff_lsn; truncate at that byte offset; delete later segments; atomically update p->applied_lsn = cutoff_lsn; idempotent
    • Update WAL trim guard: use committed_lsn not applied_lsn as threshold
    • make test GREEN on all truncation tests
    • Requirements: 5.6, 5.7
    • Boundary: Persistence Layer (persist.c, persist.h)

  • 7.6 Wire majority-commit into persist_wal_sync_append()

    • In persist_wal_sync_append(): after local write + fsync for WAL_JOB_SUBMIT, WAL_JOB_START, WAL_JOB_FINISH, WAL_QUEUE_CHANGE → call repl_broadcast(c, OP_walAck, payload, len) + repl_wait_majority_ack(c, commit_timeout_ms, OP_walAck) → on quorum advance committed_lsn
    • WAL_HOST_STATE, WAL_CHECKPOINT_REF bypass to persist_wal_async() (unchanged)
    • make test passes; integration test verifies committed WAL entries survive failover
    • Requirements: 5.1, 5.2, 5.3
    • Boundary: Persistence Layer (persist.c), Replication Layer (repl.c)

  • 8. Change H: Majority-commit config replication (P)

  • 8.1 Implement OP_upgradeConfigAck handler and committed_config_tag

    • Add committed_config_tag field to cluster config state (ha_state_t via last_known_committed_config_tag); piggybacked on CtrlHeartbeat via committedConfigTag
    • Standby handler for OP_upgradeConfig: write config blob locally + reply OP_upgradeConfigAck with ackedTag
    • push_config_to_standbys() upgraded: write locally → repl_broadcast(c, OP_upgradeConfig, blob, len) → repl_wait_majority_ack(c, commit_timeout_ms, OP_upgradeConfigAck) → on quorum atomic_store(&committed_config_tag, tag) → return; on timeout return -ETIMEDOUT
    • Log INFO on quorum (10.4); log WARN on timeout (10.4)
    • make test GREEN using test_repl_majority.c with OP_upgradeConfigAck
    • Requirements: 8.1, 8.2, 8.3, 8.4, 8.5, 10.4
    • Boundary: Config Replication (config_repl.c)
    • Depends: 7.3

  • 9. Change F: Persistent standby-to-primary control channel

  • 9.1 Extract apply_wal_entry() prerequisite

    • Extract the WAL replay loop body in the startup path into standalone apply_wal_entry(cluster_t *c, wal_entry_t *e) function; existing startup replay calls it in a loop; behavior unchanged
    • make test GREEN; this is a preparatory refactor enabling standby catch-up streaming
    • Requirements: 6.5
    • Boundary: Persistence Layer (persist.c) / WAL replay

  • 9.2 Implement ha_find_and_connect_to_primary() with three-tier discovery

    • Tier 1: if ha.voted_for_name set → direct connect; on XCE_ERR_NOT_READY → backoff, retry Tier 1
    • Tier 2: probe remaining peers; follow XCE_ERR_STANDBY redirect_host/port immediately
    • Tier 3: blind sequential scan with 100ms→5s exponential backoff
    • Sets ha.primary_ctrl_fd on success; registers fd with iomgr EPOLLIN | EPOLLRDHUP
    • Called after standby wins no election; also called by re-election loser
    • make test GREEN (mock endpoints for each tier)
    • Requirements: 6.1
    • Boundary: Persistent Control Channel (ha.c)

  • 9.3 Implement extended OP_ctrlHello handler and state sync sequence

    • Extend CtrlHelloRequest with applied_lsn and config_tag; extend CtrlHelloReply with committed_lsn, needs_snapshot, config_tag
    • Primary side handle_ctrlhello(): repl_add_standby() → repl_start_catchup(); reply; if needs_snapshot → persist_checkpoint() + send OP_snapshotTransfer; stream OP_walCatchupEntry from [standby.applied_lsn+1 → committed_lsn]; on OP_walAck reaching committed_lsn → repl_complete_catchup(); if config_tag mismatch → call push_config_to_standbys()
    • Standby side: store last_known_committed_lsn; call persist_truncate_wal_above(primary_committed_lsn); set in_catch_up=1; apply OP_snapshotTransfer if present; apply each OP_walCatchupEntry via apply_wal_entry(); send OP_walAck after each; on catch-up end: in_catch_up=0
    • make test GREEN; integration test: fresh standby joins and reaches parity with primary committed_lsn
    • Requirements: 6.5, 6.6, 6.7
    • Boundary: Persistent Control Channel (ha.c, cluster.c), Replication Layer (repl.c)
    • Depends: 7.5, 9.1

  • 9.4 Implement EPOLLRDHUP failover and heartbeat timeout vtask

    • EPOLLRDHUP event on primary_ctrl_fd → close fd, set primary_ctrl_fd = -1, transition to CTRL_ELECTING, call persist_truncate_wal_above(last_known_committed_lsn), fire am_i_primary_quorum()
    • Heartbeat timeout vtask: armed on each heartbeat receipt; fires after election_timeout_ms + rand_jitter of silence; same action as EPOLLRDHUP path
    • On CTRL_STANDBY after new election → close outbound primary fd; on CTRL_PRIMARY after new election → accept inbound OP_ctrlHello connections; reuse per-standby fd table
    • Log INFO: FSM transitions (10.5)
    • make test GREEN; integration test: TCP disconnect → sub-second election start
    • Requirements: 6.2, 6.3, 6.4, 6.8, 10.5
    • Boundary: Persistent Control Channel (ha.c), Controller FSM (ha.c)
    • Depends: 9.2

  • 10. Change G: libxce client-side transparent retry (P)

  • 10.1 Write failing tests for xce_request_with_retry()

    • In src/lib/testsuite/test_libxce_retry.c: test XCE_ERR_NOT_READY → backoff + retry same host; test XCE_ERR_STANDBY with redirect_host/port → reconnect to redirect + retry; test XCE_ERR_STANDBY without redirect → fallback to backoff; test XCE_ERR_AUTH → surface immediately (no retry); test max_retries exhausted → surface last error; test 50 concurrent simulated clients → all succeed within 30s (backoff prevents starvation)
    • make test fails
    • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6

  • 10.2 Implement xce_retry_config_t and xce_request_with_retry()

    • Add xce_retry_config_t to request.h with defaults: max_retries=20, initial_retry_ms=50, max_retry_ms=5000, backoff_multiplier=2.0
    • xce_request_with_retry(endpoint_t *ep, xce_request_t *req, const xce_retry_config_t *cfg): NULL cfg → use defaults; XCE_ERR_NOT_READY → backoff, retry; XCE_ERR_STANDBY with redirect → reconnect, retry; ECONNREFUSED/ETIMEDOUT → reconnect, retry; XCE_ERR_AUTH/XCE_ERR_PERM/XCE_ERR_INVALID/XCE_ERR_NOENT → surface immediately; exhausted → surface last error
    • Jitter via rand_r() to prevent thundering herd (7.6)
    • make test GREEN on all retry tests; 50-client concurrency test passes
    • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 7.6
    • Boundary: Client Retry (request.c, request.h)

  • 11. Integration validation and cleanup

  • 11.1 End-to-end: concurrent startup and config-fresh election

    • Integration test: 2 nodes start within 3s — only one wins election (no false self-election); config-fresh node beats stale node; stale node enters standby
    • make test-integration GREEN on these scenarios
    • Requirements: 1.1, 2.1, 2.7
    • Depends: 4.3, 9.4

  • 11.2 End-to-end: failover and WAL durability

    • Integration test: primary crash → standby detects via EPOLLRDHUP within 1s → promotes → all committed WAL entries present; integration test: WAL commit with all standbys down returns -ETIMEDOUT; reconnected standby resumes replication
    • make test-integration GREEN
    • Requirements: 5.6, 6.2, 5.5
    • Depends: 7.6, 9.4

  • 11.3 End-to-end: unconfigured node bootstrap

    • Integration test: unconfigured node (config_tag=0) joins live cluster; defer_config_fetch set; after election resolves, do_one_config_fetch() succeeds; node enters standby with correct config
    • make test-integration GREEN
    • Requirements: 3.2, 3.5
    • Depends: 5.3, 9.3

  • 11.4 End-to-end: config upgrade majority-commit and term durability

    • Integration test: config upgrade under simulated network partition returns -ETIMEDOUT; retry after heal succeeds; integration test: restart after term N participation → ha-term.bin restores voted_for, prevents double-vote
    • make test-integration GREEN
    • Requirements: 8.3, 9.1, 9.2
    • Depends: 8.1, 6.2

  • 11.5 Full ASAN build verification

    • nix build --option sandbox false .#xcecore_debug clean (no leaks, no memory errors)
    • Confirm SSL_CTX_free fix (task 2.3) resolves ASAN leak; confirm no new leaks from ha_elect_ctx_t allocation paths
    • Requirements: 4
    • Depends: 2.3, 4.3