Commit graph

409 commits

Author SHA1 Message Date
45f40c51c1 fix(ha-bootstrap): restore vcpus@7 field as deprecated UInt32 for config blob backward compatibility 2026-05-29 13:55:23 +00:00
ad7ea12563 fix(ha-bootstrap): pre-load config before election so node enters with correct config_tag 2026-05-29 13:42:15 +00:00
b8e6404fe2 docs(ha-bootstrap): fix config tag length: 12 hex chars (6 bytes), not 16 2026-05-29 13:27:01 +00:00
c7badd3f96 fix(ha-bootstrap): skip pre-election config probe; unconfigured nodes defer to post-election fetch 2026-05-29 13:15:46 +00:00
4e9924e95c fix(ha-config-bootstrap): start iomgr and vtaskmgr before election so vote RPCs are accepted 2026-05-29 12:11:12 +00:00
96df7285da docs(ha-config-bootstrap): add path verification warning — config must be in workdir/myname/conf/ 2026-05-29 11:11:29 +00:00
25c71adffb docs(ha-config-bootstrap): Test 11.1 now covers config replication verification 2026-05-29 11:02:45 +00:00
bc2aaff089 docs(ha-config-bootstrap): only primary needs initial config — standbys get it via replication 2026-05-29 10:59:51 +00:00
f465af0c0c docs(ha-config-bootstrap): add initial cluster config installation steps 2026-05-29 10:57:09 +00:00
bb4e23663a docs(ha-config-bootstrap): update manual tests to 3-node cluster (odd peer count design requirement) 2026-05-29 10:41:11 +00:00
abdb195c8e docs(ha-config-bootstrap): add TLS cert setup instructions to manual test guide 2026-05-29 10:34:41 +00:00
e88c7bebe9 docs(ha-config-bootstrap): add manual integration test guide with correct worktree paths 2026-05-29 10:20:03 +00:00
6b821eb4d5 fix(frontend): remove conf.vcpus from cClusterConfToClusterInfo — field moved to myconf_t 2026-05-29 10:18:01 +00:00
b6c8aca458 fix(ha-config-bootstrap): wire ha_restore_term in cluster_init_phase1 and store last_known_committed_lsn from CtrlHelloReply (validation fixes) 2026-05-29 08:43:36 +00:00
0d5f2cb1bb feat(ha-config-bootstrap): implement EPOLLRDHUP failover and heartbeat timeout vtask (task 9.4) 2026-05-29 07:17:55 +00:00
9bbff7b016 claude skills 2026-05-29 15:00:06 +08:00
94bf5d94b8 feat(ha-config-bootstrap): implement OP_ctrlHello state sync handler (task 9.3) 2026-05-29 05:22:17 +00:00
0ec0ffa7e3 feat(ha-config-bootstrap): implement xce_retry_config_t and xce_request_with_retry (tasks 10.1, 10.2) 2026-05-29 03:00:34 +00:00
bb8e531a12 test(ha-config-bootstrap): add RED-phase tests for xce_request_with_retry (task 10.1) 2026-05-29 02:45:24 +00:00
97a76617e6 feat(ha-config-bootstrap): implement ha_find_and_connect_to_primary three-tier discovery (task 9.2) 2026-05-28 15:09:08 +00:00
3252c89e7a docs(ha-config-bootstrap): mark task 9.1 complete — apply_wal_entry already extracted 2026-05-28 14:53:52 +00:00
c0cfc1cf83 feat(ha-config-bootstrap): implement committed_config_tag and config quorum commit (task 8.1) 2026-05-28 14:52:29 +00:00
684d3b6608 feat(ha-config-bootstrap): wire repl_wait_majority_ack into persist_wal_sync_append (task 7.6) 2026-05-28 14:28:38 +00:00
85c8565957 feat(ha-config-bootstrap): implement persist_truncate_wal_above and committed_lsn (task 7.5) 2026-05-28 14:13:29 +00:00
e7da075315 test(ha-config-bootstrap): add RED-phase tests for persist_truncate_wal_above (task 7.4) 2026-05-28 14:00:54 +00:00
30bcb74a67 feat(ha-config-bootstrap): implement repl_wait_majority_ack with quorum formula and ack_op discrimination (task 7.3) 2026-05-28 13:57:24 +00:00
3969064e11 feat(ha-config-bootstrap): implement repl_broadcast eviction and pending_ack_count (task 7.2) 2026-05-28 13:44:11 +00:00
f6030749de test(ha-config-bootstrap): add RED-phase tests for repl_wait_majority_ack and pending_ack_count (task 7.1) 2026-05-28 13:27:27 +00:00
7c27ff1954 feat(ha-config-bootstrap): implement ha_restore_term() for ha-term.bin read path (task 6.2) 2026-05-28 13:21:36 +00:00
722d7824c4 test(ha-config-bootstrap): RED-phase tests for ha-term.bin durability (task 6.1)
5 Unity tests covering persist (req 9.1), restore (req 9.2 RED),
missing-file zero-init (req 9.3), CRC-mismatch safe-zero, and
write+rename atomicity. Build fails with 3 undefined-reference linker
errors on ha_restore_term() — expected RED phase proof.
2026-05-28 10:11:56 +00:00
ab174c8b06 feat(ha-config-bootstrap): implement bootstrap_config_if_needed() (task 5.3)
- Probe all peers: any tag>0 -> defer_config_fetch=true; all-zero -> proceed;
  none reachable -> exponential backoff 500ms->10s cap with WARN per attempt
- Wire into myapp.c: before ha_wait_for_heartbeat_or_timeout()
- Post-election: defer_config_fetch && CTRL_STANDBY -> fetch_config_from_primary()
- INFO log for all three bootstrap outcomes (Req 10.2)
- XCE_TESTING hook for probe mock; test_ha_bootstrap_support.c installs mock
- 9/9 bootstrap tests GREEN, 141/141 full suite GREEN
2026-05-28 09:48:24 +00:00
0c4a0760d1 feat(ha-config-bootstrap): implement xce_probe_config_tag() (task 5.2)
- Non-blocking TCP connect + poll(500ms) + capnp ProbeConfigTagRequest/Reply
- Returns 0 with *tag_out on success; -ECONNREFUSED/-ETIMEDOUT on failure
- INFO log per probe outcome (Req 10.2)
- bootstrap_config_if_needed() minimal linkage stub (task 5.3 will complete)
- Dispatch gate already exempts OP_probeConfigTag in CTRL_INIT/CTRL_ELECTING
- Probe test GREEN; 1 expected task-5.3 failure remains
2026-05-28 09:03:09 +00:00
3cf82721d6 test(ha-config-bootstrap): RED phase tests for config bootstrap (task 5.1)
- 9 Unity tests covering Req 3.1-3.5: immediate return on nonzero tag,
  all-zero peers no defer, peer tag>0 sets defer_config_fetch, probe
  error codes, deferred fetch preconditions, ha_init zero invariant
- Linker fails with undefined refs to bootstrap_config_if_needed
  and xce_probe_config_tag (expected RED proof)
2026-05-28 08:37:32 +00:00
0c9c720b03 feat(ha-config-bootstrap): complete quorum election with mutex safety (task 4.3)
- am_i_primary_quorum(): per-peer pthreads, connect_ssl_raw + OP_requestVote
- pthread_mutex_t lock on ha_elect_ctx_t guards votes_granted/n_replied/max_resp
- done_posted guard in ha_elect_evaluate_and_signal prevents double sem_post
- Replace am_i_primary() with am_i_primary_quorum() in myapp.c
- Delete ha_check_priority_stepdown and ha_check_heartbeat_nocluster
- INFO logs: election outcome (Req 10.1), FSM transitions (Req 10.5)
- 24 election tests + 24 startup tests, 0 failures
2026-05-28 08:28:35 +00:00
0f88d5d32a feat(ha-config-bootstrap): implement vote handlers and term persistence (task 4.2)
- ha_persist_term(): atomic write+rename of ha_term_file_t with CRC32
- handle_request_vote(): term/voted_for/freshness/catch-up gate chain
- handle_request_vote_reply(): vote counting, quorum+all-replied signaling
- ha_elect_evaluate_and_signal(): quorum+yield+zero-config gates (4.3 early)
- am_i_primary_quorum(): single-node shortcircuit + 2-round retry (4.3 partial)
- 24 Tests 0 Failures in test_ha_election suite
2026-05-28 07:32:48 +00:00
403dd17f8b test(ha-config-bootstrap): RED phase tests for quorum election logic (task 4.1)
- 21 test functions covering all 12 acceptance criteria (req 2.1-2.12)
- Tests: 2/3 quorum wins, 1/3 loses, voted_for enforcement, freshness gate,
  single-node self-elects, split-vote retry, in_catch_up denial, jitter
- Build fails with undefined reference to handle_request_vote,
  ha_elect_evaluate_and_signal, am_i_primary_quorum (expected RED proof)
2026-05-28 07:10:02 +00:00
f0029bf000 feat(ha-config-bootstrap): implement ha_wait_for_heartbeat_or_timeout (task 3.4)
- sem_timedwait on heartbeat_seen semaphore with CLOCK_REALTIME deadline
- deadline = now + election_timeout_ms + xorshift64 jitter
- Returns CTRL_STANDBY on heartbeat arrival (calls ha_do_demote)
- Returns CTRL_ELECTING on timeout (caller drives election)
- Allows OPCode_heartbeat through dispatch gate in CTRL_INIT state
- 3 new unit tests: standby-on-heartbeat, electing-on-timeout, gate-allows-heartbeat
- 24 Tests 0 Failures in test_ha_startup suite
2026-05-28 06:58:59 +00:00
72776f18ff feat(ha-config-bootstrap): implement dispatch gate (task 3.3)
- errnox.h: add E_XCE_NOT_READY and E_XCE_STANDBY error codes with
  extern int XCE_ERR_NOT_READY / XCE_ERR_STANDBY runtime variables
- ha.h: declare ha_dispatch_gate(ha_state_t *, int opcode)
- ha.c: define XCE_ERR_* variables; initialize heartbeat_seen semaphore
  and primary_ctrl_fd = -1 in ha_init(); sem_destroy in ha_fini();
  implement ha_dispatch_gate() with allowlist for CTRL_INIT/CTRL_ELECTING
  (only OP_requestVote and OP_probeConfigTag pass) and CTRL_STANDBY
  (replication, election, and housekeeping ops pass; primary-only ops
  return -XCE_ERR_STANDBY)
- cluster.c: wire ha_dispatch_gate() in myreq_handler() before opcode
  switch; gate failure sends error reply and returns early

All 393 unit tests pass. Satisfies requirements 1.3 and 1.4.
2026-05-28 05:50:42 +00:00
6272fc8eb2 feat(ha-config-bootstrap): extract cluster_open_listener, reorder startup before election (task 3.2) 2026-05-28 05:03:50 +00:00
0a881804a0 test(ha-config-bootstrap): add failing dispatch gate tests (task 3.1)
RED-phase tests for ha_dispatch_gate() and FSM init state:
- CTRL_INIT + non-election ops -> XCE_ERR_NOT_READY
- CTRL_INIT/ELECTING + OP_requestVote/OP_probeConfigTag -> 0 (allowed)
- CTRL_STANDBY + primary-only ops -> XCE_ERR_STANDBY
- CTRL_PRIMARY + any op -> 0
- heartbeat_seen semaphore usable after ha_init()

ha_dispatch_gate, XCE_ERR_NOT_READY, XCE_ERR_STANDBY are all undefined
-> linker errors confirm RED phase
2026-05-28 04:42:30 +00:00
488fd1bf49 feat(ha-config-bootstrap): add peer count and cert pre-flight validation (task 2.3)
- Implement validate_peer_count() in startup_validation.c: -EINVAL for even N>1
- Implement validate_cert_files() with err_buf output: access() check, writes
  missing path into err_buf, returns -ENOENT
- Wire validate_peer_count() in main.c after peer list parse; fatal exit on fail
- Wire validate_cert_files() in main.c before SSL init; fatal exit on fail
- Fix ASAN leak: SSL_CTX_free in ha_fini() with NULL guard
- Fix double-free: null c->ha.ssl_ctx in cluster_fini() before ha_fini(),
  so iomgr_fini() retains sole SSL_CTX ownership (design per ha.h comment)
- All 96 tests pass; all 12 ha_startup tests GREEN
2026-05-28 04:29:31 +00:00
3ab2e6e4bf feat(ha-config-bootstrap): implement --vcpus CLI, wire validation, remove vcpus from config (task 2.2)
- Add --vcpus N argp option in main.c; store in myconf.vcpus
- Replace inline argp guard with validate_vcpus() call (returns -EINVAL for <=0)
- Add validate_startup_config() in startup_validation.c: returns 0; writes
  deprecation warning into caller-supplied warn_buf when vcpus key present
- Wire validate_startup_config() into myapp_run() after conffile resolved;
  emit WARN() if warn_buf non-empty (Req 4.4)
- Remove vcpus field from cluster_config_t (clusterconfig.h)
- Preserve capnp ordinal: vcpus @7 -> vcpusDeprecated @7: Void in controller.capnp
- vtaskmgr_init uses only myconf->vcpus with hardcoded default 8
- Add stub C types in typemap.h for capnp schema extensions from task 1.1
- 96 tests pass
2026-05-28 04:17:06 +00:00
bfa41eb135 test(ha-config-bootstrap): add failing startup validation tests (task 2.1)
RED-phase tests for 4 startup validation functions — all fail at link time
because the functions under test do not exist yet:
- validate_peer_count: even N>1 returns -EINVAL, odd N returns 0
- validate_cert_files: missing cert/key returns -ENOENT with full path
  in err_buf output param (asserted via strstr)
- validate_vcpus: vcpus<=0 returns -EINVAL, positive returns 0
- validate_startup_config: vcpus key in config returns 0 (not fatal)
  and writes deprecation text into warn_buf (asserted via strstr)
2026-05-28 03:31:30 +00:00
560d8fa1c8 feat(ha-config-bootstrap): add election types and new fields to ha.h (task 1.2)
- Add CTRL_INIT=2, CTRL_ELECTING=3 to ctrl_role_t
- Add ha_vote_entry_t per-peer vote tracking struct
- Add ha_elect_ctx_t with C99 flexible array votes[]
- Add 8 new ha_state_t fields: voted_for_term/name, primary_ctrl_fd,
  defer_config_fetch, elect_ctx (void * _Atomic), last_known_committed_lsn,
  heartbeat_seen (sem_t), last_known_committed_config_tag
- Declare ha_persist_term(), bootstrap_config_if_needed(), xce_probe_config_tag()
- Fix: use void * _Atomic (postfix) not _Atomic void * (prefix) for atomic
  pointer-to-void — C11 §6.7.2.4 prohibits _Atomic on incomplete type void
2026-05-28 03:15:29 +00:00
6adb7e806f feat(ha-config-bootstrap): add 6 new opcodes and HA message structs to controller.capnp 2026-05-28 03:02:40 +00:00
7a922cf184 docs(config-repl): update impl-status — Phase 1 wire protocol complete
- Mark Phase 1 as WIRE PROTOCOL COMPLETE (was: ACTIVE with stubs)
- Add commits 2ed7e34-5053c8d to component table
- Remove resolved limitations (do_one_config_fetch stub, push no-op)
- Update test count: 18 → 24 tests, total 84 passing
- Update Phase 2 table: handle_upgrade_config wired, push timeout enforced
- Update files changed to include request.c shims and new test additions
2026-05-22 09:20:29 +00:00
5053c8d42f feat(config-repl): wire handle_upgrade_config() via raw packet interception
Pre-decode opcode peek in myreq_handler() captures raw packet bytes for
OP_upgradeConfig before xce_decode() frees the capnp segment. Passes
raw bytes to xce_upgrade_config_req_extract_raw() to extract sha256 and
config blob, then delegates to do_upgrade_config_with_bytes().

Removes Task 11 dependency. Phase 2 still gated by CONF_ENABLE_UPGRADE_CONFIG.
2026-05-21 13:51:52 +00:00
2aae45f046 feat(config-repl): wire do_one_config_fetch() — full SSL roundtrip
Replaces -ENOSYS stub with:
- connect_ssl_raw() helper (non-blocking connect + poll + SSL handshake,
  checks ha_is_primary() between iterations for early promotion exit)
- xce_replconfig_req_ssl() to send OP_replConfig request
- xce_replconfig_reply_ssl_decode() to receive and decode reply
- SHA-256 integrity verify (retry on mismatch with -EBADMSG)
- conf_save_config() atomic disk write to workdir/name/conf/
- load_primary_config() to reload cluster_config from newly saved file

Standby can now fetch ClusterConfig from primary on startup.
2026-05-21 13:38:21 +00:00
4ef2089fd4 fix(config-repl): wire handle_config_request() to send real bytes
Replace NULL sha256/bytes stub with xce_replconfig_reply_raw() call.
Primary now sends actual config blob and SHA-256 in OP_replConfig reply.
2026-05-21 13:16:38 +00:00
26393e919c feat(config-repl): add raw capnp shims for OP_replConfig wire protocol
xce_replconfig_reply_raw() - primary sends reply with real bytes via endpoint
xce_replconfig_req_ssl() - standby sends request over raw SSL socket
xce_replconfig_reply_ssl_decode() - standby decodes reply from raw SSL
xce_upgrade_config_req_extract() - documents architectural limitation; returns -ENOTSUP
xce_upgrade_config_req_extract_raw() - extract bytes from raw packet for OP_upgradeConfig

Same bypass pattern as existing xce_replconfigpush_req_raw(): build the
capnp message directly via capn_new_list8 + capn_setv8, sidestepping the
encode_Data_ptr path that routes raw bytes through xce_data_t (a
discriminated cluster-data union that cannot carry raw byte arrays).

Unblocks handle_config_request() and do_one_config_fetch() wiring.

Task A research finding: xce_upgrade_config_req_extract(xce_request_t*)
is unimplementable because xce_decode() calls capn_free() before any
handler runs, and decode_Data_ptr converts the wire-format List(UInt8)
into a zero-filled xce_data_t. Task D must capture the raw io_packet_t
bytes before xce_decode tears them down, then call the _raw variant.
2026-05-21 13:02:49 +00:00