checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions

View file

@ -236,6 +236,13 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
- _Depends: 9.2_
- [x] 10. Change G: libxce client-side transparent retry (P)
- **NOTE (2026-06-16):** only the standalone primitive (10.2 `xce_request_with_retry()`) +
its unit tests (10.1) were delivered. The primitive is **not wired into any production
request path** — `xce_*_req` wrappers still call plain `xce_request()` — and it lacks
multi-master rotation to discover a new primary after the connected one is killed (it only
follows an explicit redirect). Production integration + master-list failover are owned by
the **`libxce-failover`** spec (see `.kiro/specs/libxce-failover/brief.md`). Req 7 is
therefore satisfied at the primitive level only; transparent failover is not yet live.
- [x] 10.1 Write failing tests for xce_request_with_retry()
- In `src/lib/testsuite/test_libxce_retry.c`: test `XCE_ERR_NOT_READY` → backoff + retry same host; test `XCE_ERR_STANDBY` with `redirect_host/port` → reconnect to redirect + retry; test `XCE_ERR_STANDBY` without redirect → fallback to backoff; test `XCE_ERR_AUTH` → surface immediately (no retry); test `max_retries` exhausted → surface last error; test 50 concurrent simulated clients → all succeed within 30s (backoff prevents starvation)
- `make test` fails