Commit graph

16 commits

Author SHA1 Message Date
d70bf6a402 update documents 2026-08-10 17:51:51 +08:00
9f530f5742 docs: fix unchecked group-header checkboxes across 8 fully-implemented specs
controller-status, ha-commit-engine, ha-heartbeat-committed-propagation,
ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and
raft-ha-core-safety each had every numbered sub-task marked [x] but their
group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic
gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real
unchecked sub-tasks in any of the eight files. No task content changed.
2026-07-07 08:45:41 +00:00
20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00
8fe2f1f96c chore(ha-commit-engine): mark spec implemented — validate-impl GO 2026-06-12 03:41:21 +00:00
95a10cedf3 test(ha-commit-engine): full regression green + legacy-path audit (task 6.1)
Fixed the 3 bug-411 baseline failures honestly: stale timeout
expectation updated to the reject-on-timeout contract (engine-wired
fixture, killing power retained); two healthy-primary gate fixtures
populated with standby_endpoints (gate logic untouched). Census: 46
groups / 467 tests / 0 failures standalone under default ASAN.
Audit (reviewer-confirmed, recorded in tasks.md): no production walAck
path reaches the deprecated acked_lsn>0 counting or n_active==0
short-circuit except the documented LSN-0 first entry. Leak inventory
for 6 groups documented as follow-up.
2026-06-12 03:32:19 +00:00
f0d6074d4e feat(ha-commit-engine): engine instantiation + sync-append path switch (task 5.3)
cluster_t embeds commit_engine_t; init in cluster_init_phase2 after the
vtaskmgr guard (idempotence-guarded — phase2 can run twice), wiring
repl_set_commit_engine + ha.repl back-pointer before any standby can
register; fini after iomgr/vtask shutdown. persist_wal_sync_append
passes written_lsn; engine-wired path drops -ENODEV and the
committed_lsn store (engine owns advancement); LSN-0 first-entry
routes to legacy per the accepted interim. p2 -ENODEV pin updated to
the Req 6.3 contract. 34/34 + 6/6 green through honest delegation;
census exactly at the 3 validated baselines; production binary links.
2026-06-12 00:16:22 +00:00
6600fd5d1f feat(ha-commit-engine): repl-side wiring — ACK feed, membership hooks, adapter (task 5.2)
replicator_t gains ce pointer (set-once-before-traffic contract).
repl_on_ack feeds commit_on_ack after releasing r->lock (iomgr-safe);
four membership mutators feed commit_on_membership_change post-unlock,
gated on actual mutation. walAck branch delegates to commit_submit(WAIT)
when ce wired AND entry_lsn != 0 (LSN-0 interim guard until 5.3);
legacy counting kept for ce==NULL, marked DEPRECATED. commit.c linked
into production target (inert until 5.3 wiring). Tests 30-31 + all
repl_majority walAck cases green; 33/34 remain RED for 5.3; baseline
unchanged.
2026-06-11 23:32:59 +00:00
d1353360cc test(ha-commit-engine): RED integration tests for ACK feed and sync-append (task 5.1)
Tests 30-34: wired repl_on_ack feeds engine, membership mutators
trigger re-evaluation, per-entry sync-append contract (stale-ACK
rejection, committed_lsn unmoved on timeout, n_peers=1 replaces
-ENODEV). test_repl_majority walAck cases rewired to per-LSN
semantics (configAck byte-identical). Sanctioned scaffolding:
entry_lsn param on repl_wait_majority_ack (legacy body byte-identical,
callers pass 0), inert repl_set_commit_engine stub. 9 RED failures;
baseline signatures unchanged. LSN-0 edge documented for 5.2/5.3.
2026-06-11 23:09:05 +00:00
c9154bd5cc feat(ha-commit-engine): honest write-admission quorum accounting (task 4.2)
repl_active_standby_count: registry truth (ep != NULL && !in_catch_up)
under r->lock. ha_state_t gains repl back-pointer (wired at init by
task 5.3); ha_has_write_quorum uses registry truth when wired, exact
legacy fallback when NULL; formula unchanged. Test 29 (wired gate,
catch-up-only standby → NOT_READY, admits after clear) captured RED
pre-fix. 29/29 group tests; suite at the 3-failure baseline.
2026-06-11 22:28:44 +00:00
b2598823c0 test(ha-commit-engine): RED tests for write-admission quorum counting (task 4.1)
Tests 21-28: repl_active_standby_count contract (5 RED vs inert -1
stub) + 3 gate-level pins (n_peers<=1 always-true, zero-connected
reject, stateless re-accept) that 4.2 must preserve. Declaration in
repl.h; ha.c/ha.h untouched (4.2 boundary). 28 tests / 5 RED; suite
at the 3-failure baseline with unchanged signatures.
2026-06-11 16:15:21 +00:00
631a08939b test(ha-commit-engine): membership re-evaluation regression tests (tasks 3.1+3.2 joint)
Tests 17-20: catch-up ACK exclusion vs pending waiter, membership-change
wake of covered waiter, removal-below-majority fails at own deadline
(no forced early-fail), non-blocking bound. Landed GREEN: task 1.2's
commit_on_membership_change already implements the full 3.2 contract,
so RED was unachievable; reviewer confirmed mutation-killing power and
zero production changes. 20/20 group tests; suite at baseline.
2026-06-11 15:48:42 +00:00
3bc6fd14de feat(ha-commit-engine): WAIT mode, admission cap, writable_lsn gate (task 2.2)
commit_submit(WAIT): n_vcpus-relative admission cap with fast retriable
NOT_READY (admission-cap-only per corrected contract), CLOCK_REALTIME
timedwait loop on the per-LSN + gate predicate, post-timeout predicate
re-check, n_peers<=1 self-quorum through the single advancement site.
Inflight counter accurate under engine lock for the full blocking
interval with XCE_TESTING asserts. 16/16 group tests green unmodified;
full suite at the 3-failure baseline.
2026-06-11 15:24:30 +00:00
d91f446e9e test(ha-commit-engine): RED tests for WAIT mode, admission cap, own-term gate (task 2.1)
9 new cases (8-16): block-until-majority, deadline timeout, standalone
fast path, zero-ACK regression, n_vcpus-relative cap with floor, ACK
progress at saturation, gate as blocking predicate (NOT_READY is
admission-cap-only per corrected contract). Main-thread-only assertions
after joins; 16 tests / 9 RED failures; baseline unchanged.
2026-06-11 14:56:31 +00:00
eae4767d7d feat(ha-commit-engine): per-LSN majority commit engine core (task 1.2)
commit_on_ack: active-standby (non-catch-up, connected) watermark scan,
descending sort, candidate arr[n_peers/2-1], clamped to appended log,
monotonic single-site committed_lsn store, condvar broadcast, never
blocks beyond engine mutex. RETAIN returns immediately. Self-quorum for
n_peers<=1; zero-fill makes zero-standby commit impossible. 7/7 task-1.1
tests green unmodified; full suite at known 3-failure baseline.
2026-06-11 14:37:01 +00:00
d5a0f9c64d test(ha-commit-engine): RED tests for per-LSN majority evaluation (task 1.1)
commit.h umbrella-verbatim interface + inert stub commit.c (test target
only); 7 Unity tests covering per-LSN vs acked_lsn>0 counting, majority
median, monotonicity, append clamp, lagging standby, RETAIN immediacy.
All 7 fail RED as intended. Documented 3 pre-existing masked suite
failures (bug-411) and the per-group exit-code masking root cause.
2026-06-11 14:17:09 +00:00
06135954ff chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x),
heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup.
New: ha-consistency-invariants umbrella (requirements/design/research) and
its four implementation sub-specs (specs generated via batch, cross-spec
review applied), steering roadmap + product.md Raft+CP decision.
2026-06-11 13:21:50 +00:00