Commit graph

847 commits

Author SHA1 Message Date
27ea14cd9a feat(election-iomgr-refactor): remove old blocking election code (am_i_primary_quorum, vote_request_thread, bootstrap_config_if_needed) — task 5.3 2026-05-30 13:04:19 +00:00
2e5681387a feat(election-iomgr-refactor): restructure myapp_run to fully non-blocking async startup — task 5.1 2026-05-30 12:26:56 +00:00
f4a6726f1b feat(election-iomgr-refactor): COMMITTING path in ha_elect_evaluate_and_signal + committing/election timeout watchdogs — tasks 4.1, 4.3, 4.4 2026-05-30 11:37:12 +00:00
e6135ea828 feat(election-iomgr-refactor): extend ha_dispatch_gate for CTRL_PRIMARY_COMMITTING — task 3.1 2026-05-30 10:28:36 +00:00
b8e6404fe2 docs(ha-bootstrap): fix config tag length: 12 hex chars (6 bytes), not 16 2026-05-29 13:27:01 +00:00
96df7285da docs(ha-config-bootstrap): add path verification warning — config must be in workdir/myname/conf/ 2026-05-29 11:11:29 +00:00
25c71adffb docs(ha-config-bootstrap): Test 11.1 now covers config replication verification 2026-05-29 11:02:45 +00:00
bc2aaff089 docs(ha-config-bootstrap): only primary needs initial config — standbys get it via replication 2026-05-29 10:59:51 +00:00
f465af0c0c docs(ha-config-bootstrap): add initial cluster config installation steps 2026-05-29 10:57:09 +00:00
bb4e23663a docs(ha-config-bootstrap): update manual tests to 3-node cluster (odd peer count design requirement) 2026-05-29 10:41:11 +00:00
abdb195c8e docs(ha-config-bootstrap): add TLS cert setup instructions to manual test guide 2026-05-29 10:34:41 +00:00
e88c7bebe9 docs(ha-config-bootstrap): add manual integration test guide with correct worktree paths 2026-05-29 10:20:03 +00:00
0d5f2cb1bb feat(ha-config-bootstrap): implement EPOLLRDHUP failover and heartbeat timeout vtask (task 9.4) 2026-05-29 07:17:55 +00:00
9bbff7b016 claude skills 2026-05-29 15:00:06 +08:00
94bf5d94b8 feat(ha-config-bootstrap): implement OP_ctrlHello state sync handler (task 9.3) 2026-05-29 05:22:17 +00:00
0ec0ffa7e3 feat(ha-config-bootstrap): implement xce_retry_config_t and xce_request_with_retry (tasks 10.1, 10.2) 2026-05-29 03:00:34 +00:00
bb8e531a12 test(ha-config-bootstrap): add RED-phase tests for xce_request_with_retry (task 10.1) 2026-05-29 02:45:24 +00:00
97a76617e6 feat(ha-config-bootstrap): implement ha_find_and_connect_to_primary three-tier discovery (task 9.2) 2026-05-28 15:09:08 +00:00
3252c89e7a docs(ha-config-bootstrap): mark task 9.1 complete — apply_wal_entry already extracted 2026-05-28 14:53:52 +00:00
c0cfc1cf83 feat(ha-config-bootstrap): implement committed_config_tag and config quorum commit (task 8.1) 2026-05-28 14:52:29 +00:00
684d3b6608 feat(ha-config-bootstrap): wire repl_wait_majority_ack into persist_wal_sync_append (task 7.6) 2026-05-28 14:28:38 +00:00
85c8565957 feat(ha-config-bootstrap): implement persist_truncate_wal_above and committed_lsn (task 7.5) 2026-05-28 14:13:29 +00:00
e7da075315 test(ha-config-bootstrap): add RED-phase tests for persist_truncate_wal_above (task 7.4) 2026-05-28 14:00:54 +00:00
30bcb74a67 feat(ha-config-bootstrap): implement repl_wait_majority_ack with quorum formula and ack_op discrimination (task 7.3) 2026-05-28 13:57:24 +00:00
3969064e11 feat(ha-config-bootstrap): implement repl_broadcast eviction and pending_ack_count (task 7.2) 2026-05-28 13:44:11 +00:00
f6030749de test(ha-config-bootstrap): add RED-phase tests for repl_wait_majority_ack and pending_ack_count (task 7.1) 2026-05-28 13:27:27 +00:00
7c27ff1954 feat(ha-config-bootstrap): implement ha_restore_term() for ha-term.bin read path (task 6.2) 2026-05-28 13:21:36 +00:00
722d7824c4 test(ha-config-bootstrap): RED-phase tests for ha-term.bin durability (task 6.1)
5 Unity tests covering persist (req 9.1), restore (req 9.2 RED),
missing-file zero-init (req 9.3), CRC-mismatch safe-zero, and
write+rename atomicity. Build fails with 3 undefined-reference linker
errors on ha_restore_term() — expected RED phase proof.
2026-05-28 10:11:56 +00:00
ab174c8b06 feat(ha-config-bootstrap): implement bootstrap_config_if_needed() (task 5.3)
- Probe all peers: any tag>0 -> defer_config_fetch=true; all-zero -> proceed;
  none reachable -> exponential backoff 500ms->10s cap with WARN per attempt
- Wire into myapp.c: before ha_wait_for_heartbeat_or_timeout()
- Post-election: defer_config_fetch && CTRL_STANDBY -> fetch_config_from_primary()
- INFO log for all three bootstrap outcomes (Req 10.2)
- XCE_TESTING hook for probe mock; test_ha_bootstrap_support.c installs mock
- 9/9 bootstrap tests GREEN, 141/141 full suite GREEN
2026-05-28 09:48:24 +00:00
0c4a0760d1 feat(ha-config-bootstrap): implement xce_probe_config_tag() (task 5.2)
- Non-blocking TCP connect + poll(500ms) + capnp ProbeConfigTagRequest/Reply
- Returns 0 with *tag_out on success; -ECONNREFUSED/-ETIMEDOUT on failure
- INFO log per probe outcome (Req 10.2)
- bootstrap_config_if_needed() minimal linkage stub (task 5.3 will complete)
- Dispatch gate already exempts OP_probeConfigTag in CTRL_INIT/CTRL_ELECTING
- Probe test GREEN; 1 expected task-5.3 failure remains
2026-05-28 09:03:09 +00:00
3cf82721d6 test(ha-config-bootstrap): RED phase tests for config bootstrap (task 5.1)
- 9 Unity tests covering Req 3.1-3.5: immediate return on nonzero tag,
  all-zero peers no defer, peer tag>0 sets defer_config_fetch, probe
  error codes, deferred fetch preconditions, ha_init zero invariant
- Linker fails with undefined refs to bootstrap_config_if_needed
  and xce_probe_config_tag (expected RED proof)
2026-05-28 08:37:32 +00:00
0c9c720b03 feat(ha-config-bootstrap): complete quorum election with mutex safety (task 4.3)
- am_i_primary_quorum(): per-peer pthreads, connect_ssl_raw + OP_requestVote
- pthread_mutex_t lock on ha_elect_ctx_t guards votes_granted/n_replied/max_resp
- done_posted guard in ha_elect_evaluate_and_signal prevents double sem_post
- Replace am_i_primary() with am_i_primary_quorum() in myapp.c
- Delete ha_check_priority_stepdown and ha_check_heartbeat_nocluster
- INFO logs: election outcome (Req 10.1), FSM transitions (Req 10.5)
- 24 election tests + 24 startup tests, 0 failures
2026-05-28 08:28:35 +00:00
0f88d5d32a feat(ha-config-bootstrap): implement vote handlers and term persistence (task 4.2)
- ha_persist_term(): atomic write+rename of ha_term_file_t with CRC32
- handle_request_vote(): term/voted_for/freshness/catch-up gate chain
- handle_request_vote_reply(): vote counting, quorum+all-replied signaling
- ha_elect_evaluate_and_signal(): quorum+yield+zero-config gates (4.3 early)
- am_i_primary_quorum(): single-node shortcircuit + 2-round retry (4.3 partial)
- 24 Tests 0 Failures in test_ha_election suite
2026-05-28 07:32:48 +00:00
403dd17f8b test(ha-config-bootstrap): RED phase tests for quorum election logic (task 4.1)
- 21 test functions covering all 12 acceptance criteria (req 2.1-2.12)
- Tests: 2/3 quorum wins, 1/3 loses, voted_for enforcement, freshness gate,
  single-node self-elects, split-vote retry, in_catch_up denial, jitter
- Build fails with undefined reference to handle_request_vote,
  ha_elect_evaluate_and_signal, am_i_primary_quorum (expected RED proof)
2026-05-28 07:10:02 +00:00
f0029bf000 feat(ha-config-bootstrap): implement ha_wait_for_heartbeat_or_timeout (task 3.4)
- sem_timedwait on heartbeat_seen semaphore with CLOCK_REALTIME deadline
- deadline = now + election_timeout_ms + xorshift64 jitter
- Returns CTRL_STANDBY on heartbeat arrival (calls ha_do_demote)
- Returns CTRL_ELECTING on timeout (caller drives election)
- Allows OPCode_heartbeat through dispatch gate in CTRL_INIT state
- 3 new unit tests: standby-on-heartbeat, electing-on-timeout, gate-allows-heartbeat
- 24 Tests 0 Failures in test_ha_startup suite
2026-05-28 06:58:59 +00:00
72776f18ff feat(ha-config-bootstrap): implement dispatch gate (task 3.3)
- errnox.h: add E_XCE_NOT_READY and E_XCE_STANDBY error codes with
  extern int XCE_ERR_NOT_READY / XCE_ERR_STANDBY runtime variables
- ha.h: declare ha_dispatch_gate(ha_state_t *, int opcode)
- ha.c: define XCE_ERR_* variables; initialize heartbeat_seen semaphore
  and primary_ctrl_fd = -1 in ha_init(); sem_destroy in ha_fini();
  implement ha_dispatch_gate() with allowlist for CTRL_INIT/CTRL_ELECTING
  (only OP_requestVote and OP_probeConfigTag pass) and CTRL_STANDBY
  (replication, election, and housekeeping ops pass; primary-only ops
  return -XCE_ERR_STANDBY)
- cluster.c: wire ha_dispatch_gate() in myreq_handler() before opcode
  switch; gate failure sends error reply and returns early

All 393 unit tests pass. Satisfies requirements 1.3 and 1.4.
2026-05-28 05:50:42 +00:00
6272fc8eb2 feat(ha-config-bootstrap): extract cluster_open_listener, reorder startup before election (task 3.2) 2026-05-28 05:03:50 +00:00
0a881804a0 test(ha-config-bootstrap): add failing dispatch gate tests (task 3.1)
RED-phase tests for ha_dispatch_gate() and FSM init state:
- CTRL_INIT + non-election ops -> XCE_ERR_NOT_READY
- CTRL_INIT/ELECTING + OP_requestVote/OP_probeConfigTag -> 0 (allowed)
- CTRL_STANDBY + primary-only ops -> XCE_ERR_STANDBY
- CTRL_PRIMARY + any op -> 0
- heartbeat_seen semaphore usable after ha_init()

ha_dispatch_gate, XCE_ERR_NOT_READY, XCE_ERR_STANDBY are all undefined
-> linker errors confirm RED phase
2026-05-28 04:42:30 +00:00
488fd1bf49 feat(ha-config-bootstrap): add peer count and cert pre-flight validation (task 2.3)
- Implement validate_peer_count() in startup_validation.c: -EINVAL for even N>1
- Implement validate_cert_files() with err_buf output: access() check, writes
  missing path into err_buf, returns -ENOENT
- Wire validate_peer_count() in main.c after peer list parse; fatal exit on fail
- Wire validate_cert_files() in main.c before SSL init; fatal exit on fail
- Fix ASAN leak: SSL_CTX_free in ha_fini() with NULL guard
- Fix double-free: null c->ha.ssl_ctx in cluster_fini() before ha_fini(),
  so iomgr_fini() retains sole SSL_CTX ownership (design per ha.h comment)
- All 96 tests pass; all 12 ha_startup tests GREEN
2026-05-28 04:29:31 +00:00
3ab2e6e4bf feat(ha-config-bootstrap): implement --vcpus CLI, wire validation, remove vcpus from config (task 2.2)
- Add --vcpus N argp option in main.c; store in myconf.vcpus
- Replace inline argp guard with validate_vcpus() call (returns -EINVAL for <=0)
- Add validate_startup_config() in startup_validation.c: returns 0; writes
  deprecation warning into caller-supplied warn_buf when vcpus key present
- Wire validate_startup_config() into myapp_run() after conffile resolved;
  emit WARN() if warn_buf non-empty (Req 4.4)
- Remove vcpus field from cluster_config_t (clusterconfig.h)
- Preserve capnp ordinal: vcpus @7 -> vcpusDeprecated @7: Void in controller.capnp
- vtaskmgr_init uses only myconf->vcpus with hardcoded default 8
- Add stub C types in typemap.h for capnp schema extensions from task 1.1
- 96 tests pass
2026-05-28 04:17:06 +00:00
bfa41eb135 test(ha-config-bootstrap): add failing startup validation tests (task 2.1)
RED-phase tests for 4 startup validation functions — all fail at link time
because the functions under test do not exist yet:
- validate_peer_count: even N>1 returns -EINVAL, odd N returns 0
- validate_cert_files: missing cert/key returns -ENOENT with full path
  in err_buf output param (asserted via strstr)
- validate_vcpus: vcpus<=0 returns -EINVAL, positive returns 0
- validate_startup_config: vcpus key in config returns 0 (not fatal)
  and writes deprecation text into warn_buf (asserted via strstr)
2026-05-28 03:31:30 +00:00
560d8fa1c8 feat(ha-config-bootstrap): add election types and new fields to ha.h (task 1.2)
- Add CTRL_INIT=2, CTRL_ELECTING=3 to ctrl_role_t
- Add ha_vote_entry_t per-peer vote tracking struct
- Add ha_elect_ctx_t with C99 flexible array votes[]
- Add 8 new ha_state_t fields: voted_for_term/name, primary_ctrl_fd,
  defer_config_fetch, elect_ctx (void * _Atomic), last_known_committed_lsn,
  heartbeat_seen (sem_t), last_known_committed_config_tag
- Declare ha_persist_term(), bootstrap_config_if_needed(), xce_probe_config_tag()
- Fix: use void * _Atomic (postfix) not _Atomic void * (prefix) for atomic
  pointer-to-void — C11 §6.7.2.4 prohibits _Atomic on incomplete type void
2026-05-28 03:15:29 +00:00
6adb7e806f feat(ha-config-bootstrap): add 6 new opcodes and HA message structs to controller.capnp 2026-05-28 03:02:40 +00:00
86fe35e671 add kiro skills 2026-05-21 03:41:57 +00:00
93ed282ef7 feat(worker-join-refactor): implement apply_join_state, init_worker, rejoin_worker
Replace monolithic add_worker with three focused functions:
- apply_join_state (static under production, exported under XCE_TESTING):
  shared status/resource update called by both create and rejoin paths;
  clears XCE_WORKER_UNAVAIL, applies hconfig overrides, updates resources
- init_worker: first-time join path via makeWorker + apply_join_state
- rejoin_worker: reconnect path with explicit admin-bit preservation
  (XCE_WORKER_ADMIN_MASK = LOCKEDU|CLOSED|HIDDEN|CLOSED_BY_ADMIN)

Also makes find_matched_hostconfig non-static so join.c can call it
via the extern declaration already in worker.h.

Removes add_worker (no longer declared in worker.h, no callers).

All 7 tests in test_worker_join pass:
  apply_join_state (clears UNAVAIL, hosttype override, resource values),
  init_worker (creates entry, returns correct idx),
  rejoin_worker (updates attrs/ep, preserves admin bits).
2026-05-20 08:24:26 +00:00
5dbe404015 save kiro settings 2026-05-13 11:00:32 +00:00
033f34c07f add kiro-sdd documents 2026-05-13 10:58:51 +00:00