Commit graph

33 commits

Author SHA1 Message Date
20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00
e31673dd88 chore: checkpoint all prior work for clean state
Captures all uncommitted work from ha-config-bootstrap,
ha-heartbeat-liveness, and associated spec/tooling updates:

- src: ha.c, ha.h, cluster.c, config_repl.c, myapp.c, raft.c —
  heartbeat liveness, RDHUP integration, committing phase fixes
- tests: test_ha_role.c, test_config_repl.c, test_ha_rdhup_integration.c,
  test_p1_integration.c, test_p3_hardening.c, scheduler test
- specs: election-iomgr-refactor, ha-heartbeat-liveness,
  ha-peer-connection-reuse, raft-library (new); ha-config-bootstrap
  and election-iomgr-refactor task/research updates
- tooling: .wolf/ state, ha/data cluster fixtures, raft/raft.tla,
  src/build symlink, xce-build skill update
2026-06-04 13:04:16 +00:00
374f3bfd0e chore(ha-config-bootstrap): gap analysis audit + fix parent task headers
- Gap analysis: all 10 requirements verified implemented in live codebase
  (ha.c, ha.h, config_repl.c, request.c).
- Fix parent group headers (1-10) to [x] since all leaf subtasks complete.
- Mark task 11.5 complete: /xce-build controller passes 1/1 tests, ASAN clean.
- Remaining: integration tests 11.1-11.4 (manual cluster tests, documented
  in manual-tests.md).
2026-06-03 06:04:58 +00:00
0d5f2cb1bb feat(ha-config-bootstrap): implement EPOLLRDHUP failover and heartbeat timeout vtask (task 9.4) 2026-05-29 07:17:55 +00:00
94bf5d94b8 feat(ha-config-bootstrap): implement OP_ctrlHello state sync handler (task 9.3) 2026-05-29 05:22:17 +00:00
0ec0ffa7e3 feat(ha-config-bootstrap): implement xce_retry_config_t and xce_request_with_retry (tasks 10.1, 10.2) 2026-05-29 03:00:34 +00:00
bb8e531a12 test(ha-config-bootstrap): add RED-phase tests for xce_request_with_retry (task 10.1) 2026-05-29 02:45:24 +00:00
97a76617e6 feat(ha-config-bootstrap): implement ha_find_and_connect_to_primary three-tier discovery (task 9.2) 2026-05-28 15:09:08 +00:00
3252c89e7a docs(ha-config-bootstrap): mark task 9.1 complete — apply_wal_entry already extracted 2026-05-28 14:53:52 +00:00
c0cfc1cf83 feat(ha-config-bootstrap): implement committed_config_tag and config quorum commit (task 8.1) 2026-05-28 14:52:29 +00:00
684d3b6608 feat(ha-config-bootstrap): wire repl_wait_majority_ack into persist_wal_sync_append (task 7.6) 2026-05-28 14:28:38 +00:00
85c8565957 feat(ha-config-bootstrap): implement persist_truncate_wal_above and committed_lsn (task 7.5) 2026-05-28 14:13:29 +00:00
e7da075315 test(ha-config-bootstrap): add RED-phase tests for persist_truncate_wal_above (task 7.4) 2026-05-28 14:00:54 +00:00
30bcb74a67 feat(ha-config-bootstrap): implement repl_wait_majority_ack with quorum formula and ack_op discrimination (task 7.3) 2026-05-28 13:57:24 +00:00
3969064e11 feat(ha-config-bootstrap): implement repl_broadcast eviction and pending_ack_count (task 7.2) 2026-05-28 13:44:11 +00:00
f6030749de test(ha-config-bootstrap): add RED-phase tests for repl_wait_majority_ack and pending_ack_count (task 7.1) 2026-05-28 13:27:27 +00:00
7c27ff1954 feat(ha-config-bootstrap): implement ha_restore_term() for ha-term.bin read path (task 6.2) 2026-05-28 13:21:36 +00:00
722d7824c4 test(ha-config-bootstrap): RED-phase tests for ha-term.bin durability (task 6.1)
5 Unity tests covering persist (req 9.1), restore (req 9.2 RED),
missing-file zero-init (req 9.3), CRC-mismatch safe-zero, and
write+rename atomicity. Build fails with 3 undefined-reference linker
errors on ha_restore_term() — expected RED phase proof.
2026-05-28 10:11:56 +00:00
ab174c8b06 feat(ha-config-bootstrap): implement bootstrap_config_if_needed() (task 5.3)
- Probe all peers: any tag>0 -> defer_config_fetch=true; all-zero -> proceed;
  none reachable -> exponential backoff 500ms->10s cap with WARN per attempt
- Wire into myapp.c: before ha_wait_for_heartbeat_or_timeout()
- Post-election: defer_config_fetch && CTRL_STANDBY -> fetch_config_from_primary()
- INFO log for all three bootstrap outcomes (Req 10.2)
- XCE_TESTING hook for probe mock; test_ha_bootstrap_support.c installs mock
- 9/9 bootstrap tests GREEN, 141/141 full suite GREEN
2026-05-28 09:48:24 +00:00
0c4a0760d1 feat(ha-config-bootstrap): implement xce_probe_config_tag() (task 5.2)
- Non-blocking TCP connect + poll(500ms) + capnp ProbeConfigTagRequest/Reply
- Returns 0 with *tag_out on success; -ECONNREFUSED/-ETIMEDOUT on failure
- INFO log per probe outcome (Req 10.2)
- bootstrap_config_if_needed() minimal linkage stub (task 5.3 will complete)
- Dispatch gate already exempts OP_probeConfigTag in CTRL_INIT/CTRL_ELECTING
- Probe test GREEN; 1 expected task-5.3 failure remains
2026-05-28 09:03:09 +00:00
3cf82721d6 test(ha-config-bootstrap): RED phase tests for config bootstrap (task 5.1)
- 9 Unity tests covering Req 3.1-3.5: immediate return on nonzero tag,
  all-zero peers no defer, peer tag>0 sets defer_config_fetch, probe
  error codes, deferred fetch preconditions, ha_init zero invariant
- Linker fails with undefined refs to bootstrap_config_if_needed
  and xce_probe_config_tag (expected RED proof)
2026-05-28 08:37:32 +00:00
0c9c720b03 feat(ha-config-bootstrap): complete quorum election with mutex safety (task 4.3)
- am_i_primary_quorum(): per-peer pthreads, connect_ssl_raw + OP_requestVote
- pthread_mutex_t lock on ha_elect_ctx_t guards votes_granted/n_replied/max_resp
- done_posted guard in ha_elect_evaluate_and_signal prevents double sem_post
- Replace am_i_primary() with am_i_primary_quorum() in myapp.c
- Delete ha_check_priority_stepdown and ha_check_heartbeat_nocluster
- INFO logs: election outcome (Req 10.1), FSM transitions (Req 10.5)
- 24 election tests + 24 startup tests, 0 failures
2026-05-28 08:28:35 +00:00
0f88d5d32a feat(ha-config-bootstrap): implement vote handlers and term persistence (task 4.2)
- ha_persist_term(): atomic write+rename of ha_term_file_t with CRC32
- handle_request_vote(): term/voted_for/freshness/catch-up gate chain
- handle_request_vote_reply(): vote counting, quorum+all-replied signaling
- ha_elect_evaluate_and_signal(): quorum+yield+zero-config gates (4.3 early)
- am_i_primary_quorum(): single-node shortcircuit + 2-round retry (4.3 partial)
- 24 Tests 0 Failures in test_ha_election suite
2026-05-28 07:32:48 +00:00
403dd17f8b test(ha-config-bootstrap): RED phase tests for quorum election logic (task 4.1)
- 21 test functions covering all 12 acceptance criteria (req 2.1-2.12)
- Tests: 2/3 quorum wins, 1/3 loses, voted_for enforcement, freshness gate,
  single-node self-elects, split-vote retry, in_catch_up denial, jitter
- Build fails with undefined reference to handle_request_vote,
  ha_elect_evaluate_and_signal, am_i_primary_quorum (expected RED proof)
2026-05-28 07:10:02 +00:00
f0029bf000 feat(ha-config-bootstrap): implement ha_wait_for_heartbeat_or_timeout (task 3.4)
- sem_timedwait on heartbeat_seen semaphore with CLOCK_REALTIME deadline
- deadline = now + election_timeout_ms + xorshift64 jitter
- Returns CTRL_STANDBY on heartbeat arrival (calls ha_do_demote)
- Returns CTRL_ELECTING on timeout (caller drives election)
- Allows OPCode_heartbeat through dispatch gate in CTRL_INIT state
- 3 new unit tests: standby-on-heartbeat, electing-on-timeout, gate-allows-heartbeat
- 24 Tests 0 Failures in test_ha_startup suite
2026-05-28 06:58:59 +00:00
72776f18ff feat(ha-config-bootstrap): implement dispatch gate (task 3.3)
- errnox.h: add E_XCE_NOT_READY and E_XCE_STANDBY error codes with
  extern int XCE_ERR_NOT_READY / XCE_ERR_STANDBY runtime variables
- ha.h: declare ha_dispatch_gate(ha_state_t *, int opcode)
- ha.c: define XCE_ERR_* variables; initialize heartbeat_seen semaphore
  and primary_ctrl_fd = -1 in ha_init(); sem_destroy in ha_fini();
  implement ha_dispatch_gate() with allowlist for CTRL_INIT/CTRL_ELECTING
  (only OP_requestVote and OP_probeConfigTag pass) and CTRL_STANDBY
  (replication, election, and housekeeping ops pass; primary-only ops
  return -XCE_ERR_STANDBY)
- cluster.c: wire ha_dispatch_gate() in myreq_handler() before opcode
  switch; gate failure sends error reply and returns early

All 393 unit tests pass. Satisfies requirements 1.3 and 1.4.
2026-05-28 05:50:42 +00:00
6272fc8eb2 feat(ha-config-bootstrap): extract cluster_open_listener, reorder startup before election (task 3.2) 2026-05-28 05:03:50 +00:00
0a881804a0 test(ha-config-bootstrap): add failing dispatch gate tests (task 3.1)
RED-phase tests for ha_dispatch_gate() and FSM init state:
- CTRL_INIT + non-election ops -> XCE_ERR_NOT_READY
- CTRL_INIT/ELECTING + OP_requestVote/OP_probeConfigTag -> 0 (allowed)
- CTRL_STANDBY + primary-only ops -> XCE_ERR_STANDBY
- CTRL_PRIMARY + any op -> 0
- heartbeat_seen semaphore usable after ha_init()

ha_dispatch_gate, XCE_ERR_NOT_READY, XCE_ERR_STANDBY are all undefined
-> linker errors confirm RED phase
2026-05-28 04:42:30 +00:00
488fd1bf49 feat(ha-config-bootstrap): add peer count and cert pre-flight validation (task 2.3)
- Implement validate_peer_count() in startup_validation.c: -EINVAL for even N>1
- Implement validate_cert_files() with err_buf output: access() check, writes
  missing path into err_buf, returns -ENOENT
- Wire validate_peer_count() in main.c after peer list parse; fatal exit on fail
- Wire validate_cert_files() in main.c before SSL init; fatal exit on fail
- Fix ASAN leak: SSL_CTX_free in ha_fini() with NULL guard
- Fix double-free: null c->ha.ssl_ctx in cluster_fini() before ha_fini(),
  so iomgr_fini() retains sole SSL_CTX ownership (design per ha.h comment)
- All 96 tests pass; all 12 ha_startup tests GREEN
2026-05-28 04:29:31 +00:00
3ab2e6e4bf feat(ha-config-bootstrap): implement --vcpus CLI, wire validation, remove vcpus from config (task 2.2)
- Add --vcpus N argp option in main.c; store in myconf.vcpus
- Replace inline argp guard with validate_vcpus() call (returns -EINVAL for <=0)
- Add validate_startup_config() in startup_validation.c: returns 0; writes
  deprecation warning into caller-supplied warn_buf when vcpus key present
- Wire validate_startup_config() into myapp_run() after conffile resolved;
  emit WARN() if warn_buf non-empty (Req 4.4)
- Remove vcpus field from cluster_config_t (clusterconfig.h)
- Preserve capnp ordinal: vcpus @7 -> vcpusDeprecated @7: Void in controller.capnp
- vtaskmgr_init uses only myconf->vcpus with hardcoded default 8
- Add stub C types in typemap.h for capnp schema extensions from task 1.1
- 96 tests pass
2026-05-28 04:17:06 +00:00
bfa41eb135 test(ha-config-bootstrap): add failing startup validation tests (task 2.1)
RED-phase tests for 4 startup validation functions — all fail at link time
because the functions under test do not exist yet:
- validate_peer_count: even N>1 returns -EINVAL, odd N returns 0
- validate_cert_files: missing cert/key returns -ENOENT with full path
  in err_buf output param (asserted via strstr)
- validate_vcpus: vcpus<=0 returns -EINVAL, positive returns 0
- validate_startup_config: vcpus key in config returns 0 (not fatal)
  and writes deprecation text into warn_buf (asserted via strstr)
2026-05-28 03:31:30 +00:00
560d8fa1c8 feat(ha-config-bootstrap): add election types and new fields to ha.h (task 1.2)
- Add CTRL_INIT=2, CTRL_ELECTING=3 to ctrl_role_t
- Add ha_vote_entry_t per-peer vote tracking struct
- Add ha_elect_ctx_t with C99 flexible array votes[]
- Add 8 new ha_state_t fields: voted_for_term/name, primary_ctrl_fd,
  defer_config_fetch, elect_ctx (void * _Atomic), last_known_committed_lsn,
  heartbeat_seen (sem_t), last_known_committed_config_tag
- Declare ha_persist_term(), bootstrap_config_if_needed(), xce_probe_config_tag()
- Fix: use void * _Atomic (postfix) not _Atomic void * (prefix) for atomic
  pointer-to-void — C11 §6.7.2.4 prohibits _Atomic on incomplete type void
2026-05-28 03:15:29 +00:00
6adb7e806f feat(ha-config-bootstrap): add 6 new opcodes and HA message structs to controller.capnp 2026-05-28 03:02:40 +00:00