fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool, which hardcodes VTASK_LIVENESS for every inbound peer-control packet including vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in persist_promotion_advance_committed) was reachable from that never-blocks task class -- same bug family as bug-1198. Removed the force-flush; round-3 (durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL: persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path and does bounded pread() -- lower risk (page-cache-hot) but strictly still blocking I/O; not safe to remove outright (reopens bug-1043-family job loss), needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask). bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop, lock-free) and routed both call sites through it. Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
This commit is contained in:
parent
ea0ac81c92
commit
40f510fb14
25 changed files with 1617 additions and 226 deletions
|
|
@ -11,18 +11,22 @@ All builds use `/xce-build` (skill at `.claude/skills/xce-build/`):
|
|||
```bash
|
||||
/xce-build # all components + all tests
|
||||
/xce-build lib # libxce + lib tests
|
||||
/xce-build controller # lib + controller + controller tests
|
||||
/xce-build controller # lib + controller + controller tests + xce_test driver
|
||||
/xce-build scheduler # lib + scheduler + scheduler tests
|
||||
/xce-build worker # lib + worker (daemon only, no unit tests)
|
||||
/xce-build apiserver # lib + apiserver + xcecli + Go unit tests
|
||||
/xce-build xcecli # xcecli + Go tests
|
||||
```
|
||||
|
||||
`controller` always builds `xce_test` alongside `xce_controller` — the driver `/xce-test-ha` needs, in every profile below. There is no separate build step for it.
|
||||
|
||||
Each invocation:
|
||||
1. `nix develop` into the hermetic shell
|
||||
2. `meson setup build --buildtype=debug -Db_sanitize=address` (fresh clean build)
|
||||
3. `ninja -C build` (compile all + test executables)
|
||||
4. Run registered meson tests
|
||||
2. `meson setup <dir> --buildtype=<type> -Db_sanitize=<mode>` (fresh clean build)
|
||||
3. `ninja -C <dir>` (compile all + test executables)
|
||||
4. Run registered meson tests (skipped for non-`asan` profiles — see below)
|
||||
|
||||
**Three profiles, three build dirs** (`/xce-build [component] [profile]`, second arg optional): `asan` (default, debug+ASAN, `xce.build/`, runs the meson test suite) for everyday development and CI-style verification; `debug` (`-O0 -g3` no sanitizer, `build.dbg/`, build-only) for gdb/core-dump root-causing of heisenbugs that ASAN's allocator perturbs away; `release` (`-O3` no sanitizer, `build.release/`, build-only) for soak/perf runs against production-shaped binaries. Each profile's own dir means the three never clobber each other — a `debug` rebuild doesn't invalidate an `asan` binary already in use.
|
||||
|
||||
## ASAN Integration
|
||||
|
||||
|
|
|
|||
|
|
@ -42,12 +42,14 @@ Outgoing sends use `xce_request(ep, &req)` which enqueues to the endpoint's send
|
|||
|
||||
The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed verbatim** (election, `logOk` freshness, `(term,index)` truncation, current-term commit, `UpdateTerm`). The controller (`ha.c`/`cluster.c`) is a **thin adapter** over it via five callbacks (`on_persist_entry`, `on_get_entry_term`, `on_send_append_entries`, `on_truncate_from`, `on_commit_advanced`) and the entry points (`raft_log_append`, `raft_on_append_entries_request`/`reply`, `raft_advance_commit`). The controller does NOT re-implement Raft protocol decisions — landed by spec `ha-term-safety` (closes bug-851: silent committed-prefix divergence from term-blind `hdr.term==0` logs). When wiring HA code, drive these entry points; never hand-roll a parallel election/commit/truncate path.
|
||||
|
||||
**Vote-request send failure MUST still reach the reply tally (bug-1423/1424).** `raft_cb_send_vote_request` / `raft_cb_send_pre_vote_request` send RequestVote/PreVote to every OTHER configured peer (dead or alive) and feed replies back to the library via `vote_reply_dispatch`/`pre_vote_reply_dispatch` (→ `raft_on_vote_reply`/`raft_on_pre_vote_reply`). A `endpoint_connect`/`iomgr_monitor` failure (the normal outcome when a peer is down) is itself an outcome that must be dispatched as a "not granted" reply through that SAME production path — never silently dropped, and never routed through the legacy test-only `handle_request_vote_reply`. Without this, the library's `replies_received == n_peers - 1` fast path ("everyone answered, decide now") is structurally unreachable whenever any peer is down, and recovery falls back entirely on the raw per-campaign timeout — real-election observed 2+ minutes permanently stuck in `CTRL_ELECTING` in a 3-node cluster with one node down before this fix.
|
||||
|
||||
- **RequestVote + RequestVoteReply** over Cap'n Proto
|
||||
- **Freshness gate** (`(term,lsn)`, term-primary): a voter grants iff `cand_term > local_term ∨ (cand_term == local_term ∧ cand_index >= local_len)`. The term axis is authoritative — a longer-but-stale-term divergent tail can NOT out-rank a node holding the committed prefix. This only works because the term axis is now correct (per-entry terms, below).
|
||||
- **Votes persisted atomically** with term increment (raft_cb_persist)
|
||||
- **Majority wins**: quorum = ceil(n_peers / 2)
|
||||
- **safe_committed_lsn** derived from voter RequestVoteReply messages — no round-trip needed after winning. The granting voter's advertised `(tag, durable-floor LSN)` MUST be recorded into the election context so the safe-commit reflects the true voted majority (a discarded vote LSN degrades safe-commit to `self_lsn` and truncates committed entries — bug-828).
|
||||
- **WAL truncation rule (load-bearing — bug-828)**: the **ONLY** truncates are (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, and (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile — both run *before* the new leader commits anything. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`) — it lags the true commit point, so a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello".
|
||||
- **WAL truncation rule (load-bearing — bug-828)**: every `persist_truncate_wal_above` call site must cut at a value that is provably safe — never at the lagging learned-committed watermark (`last_known_committed_lsn`), since it lags the true commit point and a pre-election/heuristic truncate on it physically destroys committed entries (clamps `durable_lsn` down). Four production call sites, all audited safe (2026-07-09): (a) the votes-derived post-election `persist_truncate_wal_above(safe_committed_lsn)` in `raft_cb_become_leader`, (b) the library `on_truncate_from` `(term,lsn)` conflict reconcile, both running *before* the new leader commits anything; (c) `ha_demotion_convergence_fn` (`ha_convergence.c`), which truncates to the node's own local `committed_lsn` on demotion-triggered restart — a safe, never-stale local watermark, not the cross-node ctrlhello field this rule warns about; (d) `persist_wal_write_replicated`'s same-LSN payload-conflict repair (`persist.c`, `truncate_wal_above_locked(p, lsn-1)`), which removes exactly one divergent tip entry before re-appending the authoritative one at the same LSN and restores `committed_lsn` verbatim afterward — never regresses the committed boundary. **NEVER truncate on the lagging learned-committed watermark** (`last_known_committed_lsn`). This generalizes `persistence.md`'s "never truncate on a stale ctrlhello".
|
||||
|
||||
**Key safety invariants**:
|
||||
- `currentTerm` is monotonic (never decreases)
|
||||
|
|
@ -59,7 +61,8 @@ The Raft protocol logic lives in the **library `src/lib/raft/raft.c`, consumed v
|
|||
Bidirectional heartbeat with Raft-style AppendEntries semantics:
|
||||
|
||||
- **Primary → standby**: `OPCode_nop` every `heartbeat_interval_ms` (3s) via self-perpetuating delaymgr one-shot (`ha_start_primary_heartbeat`). Re-arms only while role is PRIMARY.
|
||||
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
|
||||
- **Standby → primary**: Each heartbeat reply carries `(applied_lsn, config_tag, term)` via `RequestReply.stateReport`, giving the primary continuous visibility into standby convergence.
|
||||
- **Heartbeat-reply term-discovery fallback (bug-1421).** A stale-term primary does NOT reliably learn a higher term from an AE reject: `ha_raft_on_repl_ack` only routes AE replies through `raft_on_append_entries_reply` on `success=1`, and that function itself drops any reply whose term doesn't match (`raft.c` stale-term drop) — so an AE reject can never adopt a higher term through the normal path. `stateReport.term` (above) and a WAL-NACK's carried term are the two fallback vectors: both call `raft_on_higher_term_observed(s, observed_term)` (strictly-greater-than only; touches `term`/`voted_for`/`role`; `on_persist` before `on_become_follower`), so a stale primary steps down on the next heartbeat cycle instead of retrying AppendEntries against a follower that rejects forever.
|
||||
- **Standby liveness watchdog**: `ha_arm_heartbeat_timeout_vtask` armed on entering CTRL_STANDBY. If no message arrives from the primary within `heartbeat_timeout_ms` (10s), triggers re-election via `ha_on_primary_ctrl_fd_rdhup`.
|
||||
|
||||
## Replication Consistency (Two-Tier)
|
||||
|
|
|
|||
|
|
@ -58,6 +58,12 @@ Result: a truncated/behind node cannot win an election or pass the promotion gat
|
|||
- **Never write `committed_lsn` to disk.** Raft keeps the commit index volatile; durability is about the **log**, not the commit index. The post-election safe-commit point is re-derived from voter replies (`ha_compute_safe_committed_lsn`), never from a stale ctrlhello.
|
||||
- **LSN 0 is a reserved sentinel** — never a real entry; the first append is LSN 1. The commit engine rejects `lsn == 0`.
|
||||
|
||||
## Apply Correctness: `APPLY_DIVERGENT` (materialized state vs. WAL, not durability)
|
||||
|
||||
A **separate** verdict from `recover_verdict_t` (above): `persist_apply_verdict(p)` (`APPLY_CLEAN`/`APPLY_DIVERGENT`) answers "does the live materialized state (job/queue tables) actually match what a clean replay of the durable WAL would produce" — durability says the *bytes* are safe; this says the *applied result* is trustworthy. Computed once, at boot, in `persist_recover()`: an entity-count check (WAL_JOB_SUBMIT entries scanned vs. live table growth) plus a position check against `p->last_applied_lsn` (the LOGICAL apply cursor — never the physical WAL-scan tip, which also advances on checkpoint-marker/unknown-type entries and so falsely triggers DIVERGENT on an ordinary crash-during-checkpoint, bug-1419).
|
||||
|
||||
**Unconditional, restart-only fence (deliberate — landed by `raft-apply-decoupling` task 5.1-5.3, closes bug-1418's promotion-gate liveness-deadlock family):** once `APPLY_DIVERGENT`, the promotion gate (`ha.c` `raft_cb_become_leader` and `promotion_fence_timeout_fn`) refuses `CTRL_PRIMARY` **unconditionally** — no peer/HWM/durable-floor condition may clear it live, since a divergent local materialization can't be trusted for a peer/catch-up comparison either. Only a fresh, clean `persist_recover()` replay clears it (`apply_verdict` has exactly two production write sites: `persist_init()`'s reset and `persist_recover()`'s boot-time computation — no live path resets it). **Do not "fix" an apply-divergent liveness stall by relaxing this gate** — the actual fix is eliminating the false-positive DIVERGENT verdicts at the source (a boot-time TOCTOU where a returning node's verify pass raced the live apply pump / live-replication apply path, bug-1416, closed via `persist_t.recovery_complete` + `persist_mark_recovery_pending` called before any network traffic is processed).
|
||||
|
||||
## Fail-Loud, Never Self-Exit
|
||||
|
||||
When a member cannot reach a committed-consistent state (recovery-truncated/behind, no healthy peer), it enters a **loud permanent fence**: log a loud operator-visible ERROR (truncation boundary vs cluster HWM), set the sticky `recovery_unrecoverable` flag, hold non-promotable, **stay alive, and keep retrying** — it self-heals when a peer returns. It MUST NEVER `exit`/`abort` as a designed response. The decision is the pure `persist_recover_decision` → `RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}`; the action is `persist_recover_loud_fence_if_unrecoverable` (no terminal). Authoritative statement: `error-handling.md` + `src/controller/CLAUDE.md` "Process Lifecycle".
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue