lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

checkpoint: bug-851 root cause + raft TLA+ conformance spec family

Browse source 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
Rongsong Shen 2026-06-21 14:02:15 +00:00
parent 7d70820aac
commit 20e59d0b42
85 changed files with 10525 additions and 1993 deletions
Download patch file Download diff file Expand all files Collapse all files

62
.kiro/specs/ha-commit-barrier-rollout/research.md
View file

@ -106,3 +106,65 @@ Upstream interface consumed (ha-commit-engine design, generated 2026-06-11): `co
- `.kiro/steering/roadmap.md` — dependency ordering; superseded item "Quorum-ACK gate before config restart" residual
- `.kiro/steering/ha-async-io.md`, `testing.md`, `error-handling.md` — threading, TDD, errnox conventions
- Raft paper §5.3 (commit rule) — via umbrella; this spec adds no new consensus logic
---
## Gap Re-Audit (2026-06-18)
> **Mode**: POST-IMPLEMENTATION GAP AUDIT. All tasks 1.1–7.2 are `[x]` (phase=implemented). This pass re-verifies every EARS acceptance criterion against the current on-disk controller source (codegraph + grep, `persistent_ha` working tree) rather than against the design intent, and records residual/deferred items the implementation explicitly left open.
> **Verdict up front**: **CLOSED** — 19/19 acceptance criteria MET in code. The only open items are pre-declared, out-of-boundary residuals (bug-448 ref leak, bug-451 RUSAGE OOB, the CTRL_PRIMARY_COMMITTING fact-rejection deferred to ha-single-leader-role, and the WORKER_UPDATE-via-heartbeat WAIT-classification refinement). None block this spec's contract.
### Requirement-to-Asset Map (verified file:line)
| Req | Criterion (short) | Asset (verified) | Verdict |
|-----|-------------------|------------------|---------|
| 1.1 | success reply only after majority commit | `persist_commit_barrier` (persist.c:2069) called post-`job_ctrl` in `do_jobcontrol_op` (job.c:1023-1027), post-`queue_ctrl` in `do_queuecontrol_op` (queue.c:144), post-`worker_ctrl` in `do_hostcontrol_op` (worker.c:712-716), post-AFTER_EVENT in `handle_worker_join` (join.c:130) — reply (`xce_reply`) strictly after the barrier in every case | **MET** |
| 1.2 | commit timeout → error reply | barrier returns `-ETIMEDOUT` (persist.c:2105 → `commit_submit(...,COMMIT_MODE_WAIT,timeout)`), `reply.status = rc` in all four handlers (job.c:1029, queue.c, worker.c:718); `timeout_ms==0` borrows `p->config.sync_timeout_ms` (persist.c:2104-2107) | **MET** |
| 1.3 | not-ready rejection → error reply | engine `-XCE_ERR_NOT_READY` passes through barrier unmodified (persist.c:2105 returns engine code); handler maps to `reply.status` | **MET** |
| 1.4 | no success-interpretable reply pre-commit | handlers send exactly one `xce_reply` after the barrier; integration test `test_commit_barrier_rollout.c` pins "no success reply while entries exceed committed_lsn" (task 3.1/7.2) | **MET** |
| 2.1 | every WAL type classified WAIT/RETAIN | `wal_classify` (persist.c:2004-2054) total switch over all 13 `wal_entry_type` values incl. WAL_NOOP row (persist.c:2037, pre-declared); consulted in all 3 hook fns (`persist_job_hook_fun`/`persist_queue_hook_fun`/`persist_worker_hook_fun`, codegraph callers) | **MET** |
| 2.2 | unclassified type fails fast / fact-fallback | default arm: `ERROR(...)` log + `assert(0)` under `XCE_TESTING` + `return WAL_MODE_RETAIN` production fallback (persist.c:2045-2053) | **MET** |
| 2.3 | dispatch ACK is receipt-only | `handle_jobdispatch`/`dispatch_a_job_` produce no WAL (research §3); `do_set_preason`/`do_clr_preason` reply-first then RETAIN-append (job.c:1150/1178) — receipt verdict pinned by task 3.3 tests | **MET** |
| 2.4 | exhaustive reply-site verification | task 7.1 audit (tasks.md Impl Note 7.1): ZERO request-shaped stragglers; 8 category-A barrier-gated, 6 category-B receipt-ACK, ~29 category-C sites enumerated and committed | **MET** |
| 3.1 | facts appended, retried, no timeout | RETAIN types → `persist_wal_async` + `commit_submit(...,COMMIT_MODE_RETAIN,0)` (returns immediately, no timeout); hook callers get local-append rc only | **MET** |
| 3.2 | facts accepted during quorum loss | `ha_dispatch_gate` CTRL_PRIMARY no-quorum allowlist adds `OPCode_notify/update/setpreason/clrpreason` (ha.c:2099-2102) before the `-XCE_ERR_NOT_READY` default (ha.c:2105) | **MET** |
| 3.3 | uncommitted facts never discarded | log-is-the-queue: pending purely by `lsn > committed_lsn`; no drop path in the RETAIN seam | **MET** |
| 3.4 | facts commit after quorum restore, no redelivery | `committed_lsn` advances over retained suffix via `commit_on_ack`; task 4.3 integration test asserts no re-delivery, no timeout applied | **MET** |
| 4.1 | config confirmed only after majority | `do_upgrade_config_with_bytes` (config_repl.c:1115) calls `push_config_to_standbys` (config_repl.c:941) which returns barrier rc; success ACK (config_repl.c:1133-1142) only when `rc == 0` | **MET** |
| 4.2 | config timeout → failed to requester | `rc != 0` → `send_upgrade_error(ep,req,c,rc,...)` then `return rc` (config_repl.c:1118-1126), forwarding `-ETIMEDOUT` | **MET** |
| 4.3 | unacked config never committed | `committed_config_tag` advance is inside `push_config_to_standbys`'s success branch only (config_repl.c:963-969, atomic_store gated on majority) | **MET** |
| 4.4 | failed upgrade does not restart | failure path returns before `restart_requested` store + `cbuf_put(&c->wbuf)` (config_repl.c:1159-1161); restart unreachable on failure | **MET** |
| 5.1 | duplicate transition is logged no-op | `mark_job_finish` duplicate-on-terminal → `INFO(...)` no-op (job.c:702); `mark_job_update` no-op update → `INFO` skip (job.c:810) — no AFTER hook, no WAL | **MET** |
| 5.2 | terminal→non-terminal rejected with WARN | shared `job_state_is_backward` predicate (job.c:39); `mark_job_start` rejects backward START (job.c:632-633 WARN); `mark_job_update` rejects backward (job.c:803-804 WARN) | **MET** |
| 5.3 | replay-once equivalence | guards make RETAIN-retry / `restore_job_state` replay converge to single-application state; double-apply test (task 6.1) green | **MET** |
**Score: 19/19 MET.**
### Supporting-mechanism confirmations
- **Legacy walAck retirement (design goal, Req 1.x)**: `repl_wait_majority_ack` (repl.c:422) is now configAck-only; walAck delegates unconditionally to `commit_submit` (repl.c:657-659 comment + `repl_set_commit_engine` repl.c:660). configAck branch hardened: `acked_config_lsn >= push_tag` (repl.c:500-501), majority `1 + covered > n_peers/2` over `n_peers` (repl.c:511), `n_peers <= 1` immediate (repl.c:470-472), zero-cover multi-member → timeout (repl.c:529). **MET / no residual.**
- **Config-ACK production ingestion (remediation r1 / former CRITICAL F1)**: `repl_on_config_ack` now has live producers — `repl_ack_decode_and_dispatch` decodes `replconfigpushreplymsg` on the promoted-vote ep (ha.c:3403-3549) and a defensive `cluster.c:630-640` OPCode_reply ingestion. Closes the "bricked multi-member config upgrade" hole. **MET.**
- **Barrier-off-iomgr (remediation r1 / former CRITICAL F2)**: `push_config_transport_only` (config_repl.c:879, non-blocking) serves all best-effort ha.c/join.c streaming sites; `push_config_to_standbys` (transport+barrier) has exactly one caller in vtask context (`do_upgrade_config_with_bytes`). **MET.**
### Residual / Deferred Gaps (all pre-declared out-of-boundary — none block CLOSED)
| Item | Tag | Where | Status & Rationale |
|------|-----|-------|--------------------|
| bug-448: `job_ctrl` reference leak | **Constraint** | `reference(j)` at job.c:348 on the not-started branch; normal exit never `dereference`s | NOT FIXED. Pre-existing refcount leak surfaced by task 3.2 adversarial review; explicitly out of this spec's boundary (no behavior in the commit contract). Future fix. |
| bug-451: RUSAGE va_list OOB read | **Constraint** | `update_job_rusages` fires `HOOK_DETAIL_JOB_RUSAGE_UPDATE` with `xce_job_rusages_t*` (job.c:1046/1093-1094); `persist_job_hook_fun` default arm reads vararg 2 as `job_t*` → 4-byte OOB read at offset 56, garbage `state` in every WAL_JOB_RUSAGE payload | NOT FIXED. Reviewer-verified byte-precise; declared a boundary item at task 3.3. Production fix candidate = RUSAGE-specific va_list signature in the persist arm. Does not affect any acceptance criterion (jobid lines up; RUSAGE is a fact, retained either way). |
| CTRL_PRIMARY_COMMITTING still rejects facts | **Missing (deferred)** | `ha_dispatch_gate` COMMITTING branch intentionally untouched (design Non-Goals; ha.c) | DEFERRED to ha-single-leader-role, which deletes the transitional state wholesale. During the brief COMMITTING window facts can still be rejected — known residual, hard-ordered downstream. |
| WORKER_UPDATE WAIT-classification too broad | **Constraint** | `mark_worker_back` (worker.c:589) via heartbeat sweep `check_worker_timeout_` (cluster.c:1288) produces a WAIT-classified WAL_WORKER_CTRL/UPDATE with NO `xce_reply` | NOT a 7.1 violation (no reply ⇒ cannot confirm-before-commit; un-barriered WAIT behaves as RETAIN — commits via ACKs). Future-refinement candidate: narrow WORKER_UPDATE WAIT to the request-bearing detail. Carry to ha-single-leader-role / hardening pass. |
| Half-pushed config on disk after failed upgrade | **Constraint** | `do_upgrade_config_with_bytes` saves to disk + bumps generation before the barrier; failure leaves the new config on the primary's disk but cluster-uncommitted | Accepted (design Decision "Failed config upgrade does not restart"). Reconciled by existing OP_replConfig anti-entropy (out of boundary). Operator sees explicit error; tag stays uncommitted. |
### Integration Risks (forward-looking, not gaps in this spec)
- **ha-single-leader-role hard ordering**: must land AFTER this spec (it removes CTRL_PRIMARY_COMMITTING and adds WAL_NOOP). The classification guard (persist.c:2049) already forces WAL_NOOP to be classified — pre-declared row at persist.c:2037 (RETAIN) means the totality sweep will not trip when the type is wired. Low risk.
- **worker-rejoin-resync**: shares the `mark_job_*` idempotency guards (job.c:39/632/803). Revalidation trigger: re-verify the backward predicate against resync replay ordering when that spec lands.
- **New error codes to clients** (`-ETIMEDOUT`/`-XCE_ERR_NOT_READY` on control ops): deliberate CP behavior; flows through existing reply envelopes; caller retry semantics unchanged. Confirmed by design + reply.status wiring.
### Open-Work Options (only for the deferrable refinements — the spec itself needs none)
bug-451 (RUSAGE OOB) is the only residual with a correctness flavor inside this spec's files, so it carries options:
- **Option A — Fix now (RUSAGE-specific va_list arm in `persist_job_hook_fun`)**: read vararg 2 as `xce_job_rusages_t*` for `HOOK_DETAIL_JOB_RUSAGE_UPDATE`. *Effort: S (≈1 file, 1 arm, 1 ASAN test). Risk: Low — isolated to the WAL_JOB_RUSAGE payload encode; no contract change.*
- **Option B — Defer to a dedicated hardening pass / worker-rejoin-resync**: keep the boundary call as recorded. *Effort: 0 now. Risk: Low-Medium — OOB read persists in production (4-byte over-read, no crash observed; garbage state field in a fact payload that is retained regardless).*
- **Option C — Fold into ha-single-leader-role sweep** (which already touches WORKER_UPDATE classification). *Effort: S, batched. Risk: Low, but couples an unrelated correctness fix to a larger spec's timeline.*
**Recommendation**: Option A for bug-451 (cheap, isolated ASAN-clean fix); leave bug-448 and the COMMITTING/WORKER_UPDATE items as recorded boundary residuals. The spec's own 19/19 contract is CLOSED — no spec-internal open work.