wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

60
.kiro/specs/ha-commit-engine/research.md
View file

@ -36,3 +36,63 @@ Date: 2026-06-11. Discovery status: **complete via umbrella** — this sub-spec
- **Standalone behavior change (intentional)**: with the engine wired, `n_peers <= 1` sync appends now advance `committed_lsn` (previously the -ENODEV path left it unmoved). Raft-correct (self-quorum), satisfies spec req 6.3; flagged in design.
- **Membership-shrink semantics**: re-evaluation = wake + re-check; no forced early-fail (umbrella 4.3 requires fail-not-confirm, satisfied by timeout; reconnection within the deadline may still legitimately commit).
- Review gate: design verified against the umbrella contract — interface verbatim, no contradiction with locked decisions, all 20 slice requirement IDs traced.
---
## Gap Re-Audit (2026-06-18)
**Type:** Post-implementation gap audit (not a pre-impl gap analysis). Roadmap declares `ha-commit-engine` IMPLEMENTED 12/12 (2026-06-12, validate-impl GO, 467 tests green). `tasks.md` shows 6 `[ ]` checkboxes; all 6 are **section/group headers** ("1.", "2.", "3.", "4.", "5.", "6."), not leaf tasks — every leaf sub-task (1.1, 1.2, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1) is `[x]`. The "12/12" count is the leaf count. The unchecked group headers are an artifact of the checklist format, documented as such in §Implementation Notes — they are **not** deferred work and **not** core gaps.
### Verdict
**STATUS: CLOSED.** All 7 requirements (20 EARS acceptance criteria) are implemented and wired in production. Verified against current on-disk source (codegraph verbatim reads + grep, 2026-06-18). The engine has since been extended by `ha-commit-durability` (durable self-cap) — that extension strengthens, and does not regress, the ha-commit-engine contract.
### Requirement-to-Asset Map (verified, file:line)
| Req | Criterion | Asset (verified) | Status |
|-----|-----------|------------------|--------|
| 1.1 | confirm only after per-entry majority | `commit_submit(WAIT)` predicate `wait_done_locked` commit.c:184-189; sync path persist.c:1897-1900 → `repl_wait_majority_ack` walAck delegation repl.c:447 → `commit_submit` | MET |
| 1.2 | count ACK toward N only if acked_lsn >= N | `majority_candidate_locked` reads `s->acked_lsn`, sorts desc, candidate = `scratch[n_peers/2-1]` commit.c:107-134 | MET |
| 1.3 | connectivity never substitutes for ACKs | ACK-time eval only; watermarks read from registry at `commit_on_ack` commit.c:366-381; gate (ha_has_write_quorum) is fast-reject only, engine re-verifies | MET |
| 2.1 | no commit advance without majority | `commit_advance_locked` stores only `majority_candidate_locked` result commit.c:145-168 | MET |
| 2.2 | monotonic committed_lsn | `if (cand <= cur) return;` + `atomic_store(max)` commit.c:150-165; XCE_TESTING monotonic assert | MET |
| 2.3 | contiguous, bounded by appended log | candidate clamped to `cap` (durable, ≤ `highest_appended_lsn`) commit.c:136-137, 52-55; cumulative watermarks → contiguous | MET (strengthened: cap is now durable_lsn per ha-commit-durability) |
| 3.1/3.2/3.3 | admission quorum counts only members able to receive; catch-up + disconnected excluded | `repl_active_standby_count` (ep != NULL && !in_catch_up) repl.c:668-686; consumed by `ha_has_write_quorum` ha.c:2049-2058 | MET |
| 4.1 | post-acceptance shrink → fail not confirm | WAIT times out (`-ETIMEDOUT`) when majority never reached; no false-success path; `commit_on_membership_change` re-evals monotonically (shrink never regresses) commit.c:397-409 | MET |
| 4.2 | membership change re-evaluates pending waits | `commit_on_membership_change` wired at 4 mutators: repl.c:130 (add), :162 (remove), :598 (start_catchup), :621 (complete_catchup) | MET |
| 4.3/4.4 | zero peer ACKs never majority (n_peers>1); no zero-active short-circuit | engine derives from `n_peers` not n_active; `k = n_peers/2-1`, absent peers zero-filled commit.c:117-134; legacy `n_active==0→success` short-circuit removed (sync path persist.c:1897 engine branch; configAck has explicit "no n_active shortcut" repl.c:506-510) | MET |
| 5.1 | lag never blocks majority | majority is the only threshold; far-lagging standby sorts low, does not affect `scratch[n_peers/2-1]` commit.c:129-134 | MET |
| 6.1 | reject without quorum | `ha_has_write_quorum` false → `ha_dispatch_gate` -XCE_ERR_NOT_READY ha.c:2031-2059 | MET |
| 6.2 | auto-resume on quorum restore | gate is stateless per request (recomputes `repl_active_standby_count` each call); no reset needed | MET |
| 6.3 | standalone self-quorum | `n_peers <= 1` → `commit_advance_locked` immediate, returns cap commit.c:100-102, 330-332; sync path -ENODEV special case removed on engine branch | MET |
| 7.1/7.2 | n_vcpus-relative admission cap, NOT_READY beyond | `wait_cap = max(1, n_vcpus - COMMIT_WAIT_RESERVED)` commit.c:175-180; `inflight_waits >= cap → -XCE_ERR_NOT_READY` commit.c:313-320 | MET |
| 7.3 | ACK processing unaffected at cap | `commit_on_ack` takes only ce->lock, never the inflight gate; advances + broadcasts regardless of cap commit.c:366-381 | MET |
### The 6 Open `[ ]` Tasks — Classified
All 6 are group-header checkboxes whose every child is `[x]`. None is deferred work; none is a core gap. They are a **checklist-rendering artifact** (parent headers left unchecked while leaves complete).
| Header | Children | Classification | Effort/Risk |
|--------|----------|----------------|-------------|
| 1. Commit engine core | 1.1 [x], 1.2 [x] | DONE (artifact) | none |
| 2. WAIT/self-quorum/zero-peer/cap/gate | 2.1 [x], 2.2 [x] | DONE (artifact) | none |
| 3. Membership re-evaluation | 3.1 [x], 3.2 [x] | DONE (artifact; jointly completed — RED unachievable, §Notes) | none |
| 4. Quorum-accounting fix | 4.1 [x], 4.2 [x] | DONE (artifact) | none |
| 5. Integration (wire engine behind job submit) | 5.1 [x], 5.2 [x], 5.3 [x] | DONE (artifact) | none |
| 6. Validation / dishonest-path audit | 6.1 [x] | DONE (artifact; 6.1 audit record in §Notes) | none |
**Bottom line: 0 deferred follow-ups remain as core gaps. The "deferred follow-ups in §Implementation Notes" referenced by the roadmap are advisory/out-of-scope items, not open tasks** — see the genuine residuals below; all are explicitly out-of-spec-boundary and owned elsewhere.
### Genuine Residuals (advisory, all out-of-boundary or already resolved downstream)
These are the §Implementation Notes "deferred follow-ups". None blocks CLOSED status; each is owned by a later spec or is cosmetic.
1. **LSN-0 first-entry legacy edge** (Notes 5.1, 6.1 audit). The first-ever WAL entry (LSN 0) routes to the legacy branch (persist.c:1903-1929) because `commit_submit` rejects `lsn==0`. — **RESOLVED downstream:** ha-single-leader-role's promotion no-op entry makes LSN 0 never a client write in practice; repl.c:438-440 now confirms LSN 0 immediately. Once-per-cluster-lifetime, no consistency loss. *Effort: 0 (closed). Risk: none.*
2. **`ce->n_peers` not re-derived on dynamic membership change** (Notes 6.1(b)). If `ha->n_peers` is reassigned post-init (e.g. ha.c RDHUP re-election), the engine's `n_peers` is stale. — **Out of boundary** (dynamic membership = future work); current cluster topology is fixed at init. *Option A: assert n_peers stability at re-election callback; Option B: re-derive `ce->n_peers` in a `commit_set_n_peers` setter called from the election path; Option C: leave as documented latent gap. Effort: A ~0.5d, B ~1d (+revalidate barrier-rollout/durability). Risk: LOW today (no dynamic membership in production); MEDIUM if dynamic reconfiguration is ever added without addressing it.*
3. **Pre-existing ASAN leaks in 6 test groups** (Notes 6.1 LEAK FOLLOW-UP: worker_join 33KB, rdhup_integration, failover, start_election_async, liveness, connection_reuse). — **Out of scope** (pre-existing, production-side allocations; suite runs with `detect_leaks=0`). *Option: dedicated leak-cleanup pass. Effort: ~1-2d. Risk: LOW (test-harness hygiene, not a correctness gap).*
4. **Stale comment persist.c:1151-1154 / pre-engine contract text** (Notes 6.1). One-line doc follow-up. *Effort: trivial. Risk: none.*
5. **Untracked `src/lib/testsuite/test_wal_wire.c`** referenced by committed meson.build (Notes 5.1). Repo hygiene. *Effort: trivial (git add). Risk: none.*
### Drift Note (informational)
The engine source has been extended by **ha-commit-durability** (implemented 2026-06-18) AFTER ha-commit-engine: `majority_candidate_locked` now caps the self-contribution at `persist_durable_lsn` (commit.c:93) rather than the appended tip, and a `commit_on_durable_advance` flusher wake was added (commit.c:218). This is additive — it tightens Req 2.3 (committed never beyond durable, not merely never beyond appended) and closes bug-661. The ha-commit-engine contract is unchanged and remains fully satisfied. Original research-table anchors (e.g. repl.c:432, persist.c:1035, ha.c:1507) have drifted; verified-current anchors are in the Requirement-to-Asset Map above.