checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
parent
7d70820aac
commit
20e59d0b42
85 changed files with 10525 additions and 1993 deletions
|
|
@ -710,3 +710,85 @@ Aggregate remaining: **~M total for Phase 1 (all S items), ~L for Phase 2** (G2+
|
|||
- **RN3**: Restart-path divergence (G5) — construct the failure: standby with uncommitted WAL tail restarts, rejoins, receives catch-up; verify whether `apply_wal_entry` overwrite semantics mask or corrupt the divergent tail.
|
||||
- **RN4**: All-zero cold-start convergence time and liveness (G6) — 3-node cluster, no configs anywhere; confirm watchdog-driven election converges and measure time-to-primary.
|
||||
- **RN5**: Serve-time enforcement for 8.4 — can a promoted standby serve a config tag above `committed_config_tag` in any window between promotion and first quorum push?
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
> Trigger: post-implementation gap audit. Tasks 46/46 `[x]`, `phase=implemented`.
|
||||
> Method: re-verified the 2026-06-09 gap list (G1–G10) against the current working
|
||||
> tree on `persistent_ha`. Since the prior pass, two successor specs landed and
|
||||
> closed several of the gaps this spec recorded: **`libxce-failover`** (client-side
|
||||
> transparent failover/redirect) and **`ha-commit-durability`** (bug-661, landed
|
||||
> 2026-06-18). The earlier majority-commit model was also superseded by a
|
||||
> `wal_classify()` WAIT/RETAIN scheme from `ha-commit-barrier-rollout`. Line numbers
|
||||
> below reflect the current tree (ha.c is now 5,338 lines; persist.c 2,587).
|
||||
|
||||
### Headline change vs 2026-06-09
|
||||
|
||||
The 2026-06-09 pass found ~40/50 ACs fully implemented with 10 open gaps (G1–G10).
|
||||
**Five of those gaps are now CLOSED** by successor-spec work; the residual set is
|
||||
smaller and almost entirely *spec-text reconciliation debt* rather than missing code.
|
||||
|
||||
### Requirement-to-Asset Map — Delta Since 2026-06-09
|
||||
|
||||
| Gap (2026-06-09) | Req | New Status (2026-06-18) | Evidence |
|
||||
|---|---|---|---|
|
||||
| **G1** Redirect payload never populated | R1.4, R7.2 | **CLOSED** | `fill_redirect_reply()` populates `reply->redirect_host/redirect_port` from `ha->primary_host/port` (`join.c:28-32`); called on the `-E_XCE_REDIRECT` path (`join.c:282`); covered by `test_join_redirect.c`. Comment cites `libxce-failover Req 8.1`. |
|
||||
| **G7** libxce retry has zero adopters | R7 obj. | **CLOSED (re-homed)** | Orphaned no-op `xce_request_with_retry()` + `xce_retry_send_hook` were **removed** (`request.c:1923-1936`); superseded by the `xcecli_*` failover loop in `src/lib/client.c` + `src/lib/client_failover.c`. `XCE_RETRY_DEFAULTS` retained as the shared tunable. Production redirect-follow is now live end-to-end (G1 + client_failover). |
|
||||
| **G10** `ha_find_and_connect_to_primary` dead code | cleanup | **CLOSED** | No production `.c` reference remains (grep over `src/controller/*.c` returns nothing). Dead-code deletion done. |
|
||||
| **G2** WAL majority-commit covers only job submit | R5.1 | **SUPERSEDED (model changed)** | Replaced by `wal_classify()` WAIT/RETAIN scheme (`persist.c:2004`). `WAL_JOB_SUBMIT` and `WAL_QUEUE_CTRL` are `WAL_MODE_WAIT` (commit-gated before requester reply); `WAL_WORKER_CTRL` JOIN/UPDATE are WAIT; job state transitions (`WAL_JOB_START/FINISH/FAIL/...`) are `WAL_MODE_RETAIN` — replicated and retained-until-committed but not blocking the worker-fact report. The request-shaped side of dual-origin events is realized by the handler barrier in `do_jobcontrol_op`. The original "everything sync-blocks" framing was deliberately not adopted (latency); coverage is now classification-driven, owned by `ha-commit-barrier-rollout` / `ha-commit-durability`. |
|
||||
| **G3** Commit timeout degraded-mode vs CP | R5.5 | **SUPERSEDED (bug-661 / ha-commit-durability)** | Durability invariant now GUARANTEED: acknowledged ⇒ `fdatasync`'d on a fault-tolerant majority. `majority_candidate_locked` caps the primary self-contribution at `persist_durable_lsn`; `fsync_worker` group-commit flusher advances `durable_lsn` only on `fdatasync` success (fail-loud rc-gate); standby `fdatasync`s before WalAck. Verified by `test_commit_durability.c`, the ack-before-fdatasync E2E, and the rolling-failover soak (zero acked-job loss). Controller CLAUDE.md checklist item flipped to GUARANTEED 2026-06-18. The old "silent local-only degrade" path is gone. |
|
||||
|
||||
### Residual Gaps (still open after 2026-06-18)
|
||||
|
||||
| ID | Req | Status | Detail |
|
||||
|---|---|---|---|
|
||||
| **R1 (was G6)** Config probe path unwired | R3.1, R3.3, R3.4 | **Missing-as-specified (stable)** | `xce_probe_config_tag()` is fully implemented + unit-tested (`config_repl.c:1219`) but **still not called in production** — `myapp.c` has no probe call (grep confirms). The tag-0 voter-only gate (`ha.c:4495`) uses static `my_config_tag`/`n_peers`, not probe results. The comment at `ha.c:4493` ("xce_probe_config_tag path already called in myapp_run") is **stale/incorrect** — no such call exists. All-zero cold-start converges only via watchdog-driven re-election (~heartbeat_timeout), and that path lacks an automated convergence test (prior RN4). |
|
||||
| **R2 (was G9)** Spec/code reconciliation debt | 2.8, 2.10–2.12, 3.1–3.5, 5.5, 6.1 | **Open (docs only)** | requirements.md EARS text still describes: fatal-exit for tag-0 candidate (code: voter-only standby), exactly-once split-vote retry (code: unbounded watchdog retry loop), exponential-backoff probe (code: passive watchdog), standby-dials-primary (code: primary dials out — inverted by design per controller CLAUDE.md), CP-timeout-error (code: now GUARANTEED durability, but worded as the old model). spec.json `note` still lists the now-closed G1/G3 framings. The TLA+ mapping + controller CLAUDE.md are authoritative and correct; the spec is the lagging artifact. |
|
||||
| **R3 (was part of G4)** Single-node `committed_lsn` runtime advance | R5.8 | **Likely closed — verify** | `test_single_node_committed_lsn_advances` (`test_p2_integration.c:251`) and `test_legacy_sync_append_holds_committed_at_durable` (`test_commit_durability.c:660`) now exist, suggesting ha-commit-durability addressed the prior `-ENODEV` short-circuit concern. Recommend a quick read of those tests to confirm RN1 is retired before closing. |
|
||||
| **R4 (was G5)** Standby restart-path divergent-tail truncation | R5.7 | **Re-scoped to recovery** | The RDHUP/watchdog failover path truncates to `last_known_committed_lsn`. The *restart* path is now governed by `ha-commit-durability` recovery (`recover_verdict_t` clean-vs-corruption, durable-floor re-derivation, pre-promotion fence). Whether a restarted standby with an uncommitted tail is correctly fenced/caught-up before rejoin is owned by that spec's recovery tests, not this one. Verify via `ha-commit-durability` rather than re-opening here. |
|
||||
| **R5 (was G8)** Consolidated election-outcome log | R10.1 | **Open (minor)** | FSM-transition INFO logs carry term, and majority-ack logs carry acks+quorum, but there is still no single election-complete record emitting *votes-granted AND quorum-size* together. Data is available in `ha_elect_ctx_t`/raft state at the decision point. Low value, low effort. |
|
||||
|
||||
### Residual Tally
|
||||
|
||||
- **Closed since 2026-06-09:** G1, G7, G10 (outright), G2, G3 (superseded by a better model in successor specs).
|
||||
- **Still open:** R1 (probe unwired + stale comment), R2 (spec-text reconciliation), R5 (consolidated election log). R3 likely-closed-pending-verify, R4 re-homed to ha-commit-durability recovery.
|
||||
- **Net AC posture:** the durability and redirect deltas mean the *functional* CP promise of this spec is now met or exceeded; the open items are documentation honesty (R2), one unwired-but-replaced bootstrap path (R1), and one cosmetic log (R5).
|
||||
|
||||
### Implementation Approach Options (for the residual work)
|
||||
|
||||
#### Option A — Close-and-reconcile in place (recommended)
|
||||
Phase-1-only, all S-size, no successor spec needed:
|
||||
1. **R2** rewrite requirements.md AC 2.8 / 2.10–2.12 / 3.1–3.5 / 5.5 / 6.1 to describe the as-built push/watchdog/outbound-primary/guaranteed-durability architecture; refresh the spec.json `note` (G1/G3 are no longer "known deviations"). Docs only.
|
||||
2. **R1** delete the stale `ha.c:4493` comment; either (a) delete the unused `xce_probe_config_tag` production path and document the static-gate + watchdog-convergence model, or (b) add a cold-start convergence integration test (3-node, all configs absent) so the watchdog fallback is covered. Recommend (a)+test.
|
||||
3. **R5** add a single INFO "election complete: term=T votes=G/Q role=…" record at the become-leader / become-follower decision point.
|
||||
4. **R3** read the two new single-node durability tests; if they cover runtime advance, retire RN1 and mark R5.8 closed.
|
||||
- ✅ Closes the spec honestly with no policy decisions left implicit; smallest surface.
|
||||
- ✅ Every item has an existing seam + testsuite file.
|
||||
- ❌ Touches requirements.md (aging artifact) — but that is the point.
|
||||
- **Effort: S total. Risk: Low** (docs + dead-code delete + one log line + one test).
|
||||
|
||||
#### Option B — Defer reconciliation, close only code defects
|
||||
Fix R1 comment + R5 log now; leave requirements.md as-is with an expanded spec.json deviation note pointing at controller CLAUDE.md as the source of truth.
|
||||
- ✅ Even smaller; no requirements.md churn.
|
||||
- ❌ Leaves EARS text contradicting shipped behavior — the recurring complaint across every prior pass. Reconciliation debt compounds.
|
||||
- **Effort: XS. Risk: Low (but accrues doc debt).**
|
||||
|
||||
#### Option C — Fold residuals into successor specs and close this one verbatim
|
||||
Mark ha-config-bootstrap closed; track R1/R2/R5 under `ha-commit-durability` (R3/R4 already there) or a small `ha-spec-reconcile` doc spec.
|
||||
- ✅ This spec stops being touched; clean lineage.
|
||||
- ❌ R2 spans this spec's own EARS text — odd to reconcile it from another spec. R5 is trivially this spec's.
|
||||
- **Effort: S (mostly bookkeeping). Risk: Low; tracking-loss risk medium.**
|
||||
|
||||
### Effort & Risk — Residual
|
||||
|
||||
| Item | Effort | Risk | Notes |
|
||||
|---|---|---|---|
|
||||
| R1 probe path + stale comment + cold-start test | S | Low | Behavior already converges via watchdog; delete dead path or add test |
|
||||
| R2 requirements.md / spec.json reconciliation | S | Low | Docs only; 6 EARS deltas; CLAUDE.md is the correct source |
|
||||
| R3 single-node commit verify | XS | Low | Read 2 existing tests; likely already closed |
|
||||
| R4 restart-path truncation | — | — | Re-homed to ha-commit-durability recovery; verify there, not here |
|
||||
| R5 consolidated election log | S | Low | One INFO line; data already in elect_ctx |
|
||||
|
||||
**Recommendation:** Option A. The heavy lifting (durability, redirect, retry adoption, dead-code) is done by successor specs; the only thing standing between this spec and an honest `implemented`/closed state is requirements-text reconciliation (R2), one unwired-but-superseded path (R1, delete + document), and a cosmetic log (R5). All S/XS, all Low risk.
|
||||
|
|
|
|||
|
|
@ -236,6 +236,13 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
|
|||
- _Depends: 9.2_
|
||||
|
||||
- [x] 10. Change G: libxce client-side transparent retry (P)
|
||||
- **NOTE (2026-06-16):** only the standalone primitive (10.2 `xce_request_with_retry()`) +
|
||||
its unit tests (10.1) were delivered. The primitive is **not wired into any production
|
||||
request path** — `xce_*_req` wrappers still call plain `xce_request()` — and it lacks
|
||||
multi-master rotation to discover a new primary after the connected one is killed (it only
|
||||
follows an explicit redirect). Production integration + master-list failover are owned by
|
||||
the **`libxce-failover`** spec (see `.kiro/specs/libxce-failover/brief.md`). Req 7 is
|
||||
therefore satisfied at the primitive level only; transparent failover is not yet live.
|
||||
- [x] 10.1 Write failing tests for xce_request_with_retry()
|
||||
- In `src/lib/testsuite/test_libxce_retry.c`: test `XCE_ERR_NOT_READY` → backoff + retry same host; test `XCE_ERR_STANDBY` with `redirect_host/port` → reconnect to redirect + retry; test `XCE_ERR_STANDBY` without redirect → fallback to backoff; test `XCE_ERR_AUTH` → surface immediately (no retry); test `max_retries` exhausted → surface last error; test 50 concurrent simulated clients → all succeed within 30s (backoff prevents starvation)
|
||||
- `make test` fails
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue