forked from lei.chen/myxce
docs(ha-commit-durability): pin boot-policy decision table + pending skeleton tests (task 3.2)
This commit is contained in:
parent
dd4d70863b
commit
6483ee8afd
3 changed files with 511 additions and 1 deletions
370
.kiro/specs/ha-commit-durability/design.md
Normal file
370
.kiro/specs/ha-commit-durability/design.md
Normal file
|
|
@ -0,0 +1,370 @@
|
|||
# Design Document — ha-commit-durability
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose:** Make acknowledgment imply durability in the XCE HA controller. Today the controller advances its commit position on a majority of replication **acknowledgments**, but the primary counts its own contribution from a page-cache write (`commit.c:majority_candidate_locked` uses `highest_appended_lsn`), while background `fdatasync` runs on a timer (`fsync_worker`). A "committed" operation can therefore be durable on as few as one member — bug-661, observed as permanent job loss across failover.
|
||||
|
||||
**Users:** Cluster operators and every client whose acknowledged job/queue/worker operations must survive controller failover.
|
||||
|
||||
**Impact:** Strengthens the meaning of "committed" from "majority-ACKed (volatile)" to "persisted to stable storage on a fault-tolerant majority." The client reply path is already gated on the commit barrier, so once `committed_lsn` implies a durable majority, the invariant holds with no change to the reply path. Durability becomes **group-commit**: `fsync_worker` is repurposed from a fire-and-forget timer into a demand-driven flusher that, on append, batches `fdatasync` across all pending entries, advances `durable_lsn`, and wakes the commit evaluation. The commit majority counts `durable_lsn` (not appended). This amortizes one `fdatasync` across many entries — important because the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage), which are not client-latency-bound but must be durable before `committed_lsn` passes them. A second concern — a member with a corrupt or truncated log becoming authoritative — is closed by a recovery-integrity verdict and a promotion fence.
|
||||
|
||||
### Goals
|
||||
- **Durable commit:** when the controller replies *ok*, the operation's log entry is `fdatasync`'d on the acknowledging member plus ⌊N/2⌋ others (Req 1).
|
||||
- **Recovery integrity & promotion fence:** a member that cannot reconstruct its committed state, or that lags the cluster's committed high-water mark, is barred from leadership until caught up — fail-loud when it cannot recover safely (Req 2).
|
||||
- **Deterministic verification:** durability timing is controllable in a test mode so the guarantee is provable without racing the clock (Req 3).
|
||||
- Preserve cluster liveness: durability work never stalls heartbeat/election processing.
|
||||
|
||||
### Non-Goals
|
||||
- Single-record job-array persistence (`job-array-single-wal-record`, roadmap) — complements this spec by bounding per-array fsync count; independent of the group-commit flusher introduced here.
|
||||
- Per-entry synchronous fsync as the steady-state mechanism — superseded by group-commit (decision: the high-volume `WAL_MODE_RETAIN` path needs batching; see Components/DurableCommit).
|
||||
- Replication/ACK wire format (`ha-wal-replication`), per-entry majority arithmetic (`ha-commit-engine`), demotion truncate / rejoin reconciliation of an intact log (`ha-convergence-recovery`), promotion no-op / own-term gate (`ha-single-leader-role`), raw vote-grant arithmetic (`raft-library`).
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
- The **durability precondition** on the commit majority: a member contributes to the commit majority for an LSN only once that LSN is `fdatasync`'d locally. Introduces and is authoritative for `persist_t.durable_lsn`.
|
||||
- **Group-commit durability**: `fsync_worker` repurposed from a timer into a demand-driven flusher that batches `fdatasync`, advances `durable_lsn`, and wakes commit evaluation; `persist_wal_async` signals the flusher instead of deferring to a timer. Benefits all members on the high-volume `WAL_MODE_RETAIN` path.
|
||||
- **Recovery integrity verdict**: `replay_wal`/`persist_recover` distinguish a clean log end from a corruption-truncated end, and re-derive the member's durable floor from the recovered WAL.
|
||||
- **Promotion eligibility fence**: a recovery-truncated member, or one whose durable position is below the cluster committed high-water mark, is ineligible to become primary until caught up; re-eligible on catch-up.
|
||||
- **Durable-backed freshness advertisement**: a candidate's advertised log position and a voter's freshness floor derive from durable WAL state, not from volatile in-memory positions that survive a truncating restart.
|
||||
- **Durability verification seam**: a configurable durability cadence / background-flush control plus an `XCE_TESTING` hook.
|
||||
|
||||
### Out of Boundary
|
||||
- Counting the majority or advancing `committed_lsn` arithmetic (`ha-commit-engine` — consumed; this spec changes only the *durability precondition* of its inputs).
|
||||
- The reply ordering / classification sweep (`ha-commit-barrier-rollout` — reused unchanged).
|
||||
- Demotion truncate + self-restart and rejoin reconciliation of an **intact** log (`ha-convergence-recovery`). This spec fences **promotion**; convergence-recovery handles **demotion** and **rejoin**.
|
||||
- Vote-grant arithmetic and term handling (`raft-library`); this spec adds only the durability/high-water **precondition** to the existing freshness gate.
|
||||
- WAL group-commit batching; job-array single-record persistence.
|
||||
|
||||
### Allowed Dependencies
|
||||
- `ha-commit-engine`: `commit_engine_t`, `commit_submit`, `majority_candidate_locked` (modify self-contribution input only).
|
||||
- `ha-wal-replication`: standby ACK path (`persist_wal_write_replicated`) and catch-up streaming (consumed; `in_catch_up` reused).
|
||||
- `ha-commit-barrier-rollout`: `persist_commit_barrier` reply gate (consumed unchanged).
|
||||
- `persist_t` internals (`current_lsn`, `committed_lsn`, `last_crc`, segment I/O), `ha_state_t` (`applied_lsn`, `last_known_committed_lsn`, `in_catch_up`, `term`).
|
||||
- Constraint: must not write/persist `committed_lsn` to disk (Raft keeps the commit index volatile); durability is about the **log**, not the commit index.
|
||||
|
||||
### Revalidation Triggers
|
||||
- Change to the `durable_lsn` semantics or to `majority_candidate_locked`'s self-contribution → re-check `ha-commit-engine` and `ha-convergence-recovery` (which assume durable committed state).
|
||||
- Change to the promotion-eligibility gate or freshness-floor source → re-check `ha-single-leader-role`, `ha-prevote`, `raft-library`, and the `CLAUDE.md` TLA+ checklist.
|
||||
- Removal of `fsync_worker` → re-check any caller that assumed deferred fsync (audit `wal_classify`).
|
||||
- New `wal_sync` override surface → re-check `main.c` flag conventions and deployment scripts.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
- **Commit gate already off the iomgr thread:** the AFTER_EVENT hook "writes WAL and waits for majority replication" before the reply (`job.c:568-569`); `persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT)` blocks on a condvar. Adding `fdatasync` to this path is liveness-safe.
|
||||
- **Standby already persists-before-ACK:** `persist_wal_write_replicated` `fdatasync`s before returning (`persist.c:515`). The primary is the asymmetric gap.
|
||||
- **Recovery is silent-truncating:** `replay_wal` `break`s on CRC chain break / mismatch / short read (`persist.c:1021/1050/1042`) and returns the replayed count as success; `persist_recover` never sets `committed_lsn` (stays 0).
|
||||
- **Promotion has no integrity gate:** `raft_cb_become_leader` computes `safe_lsn` from votes then `persist_truncate_wal_above(safe_lsn)` (`ha.c:572/577`) with no check that the local log is intact or ≥ `safe_lsn`.
|
||||
- **Freshness floor is volatile:** `ha_vote_self_freshness = max(applied_lsn, last_known_committed_lsn)` (`ha.c:2127`), in-memory atomics not re-derived from durable WAL.
|
||||
- **Test-seam convention:** `#ifdef XCE_TESTING` hook shims (`ha.c:2270-2335`); `wal_sync_ms` is a hard constant (`config.h:86`) with no override.
|
||||
|
||||
### Architecture Pattern & Boundary Map
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph RequestPath
|
||||
Hook[AFTER_EVENT hook off iomgr]
|
||||
Barrier[persist_commit_barrier]
|
||||
end
|
||||
subgraph Persistence
|
||||
Append[persist_wal_async sync fsync]
|
||||
DurLsn[durable_lsn]
|
||||
Recover[persist_recover integrity verdict]
|
||||
ReplApply[persist_wal_write_replicated standby fsync]
|
||||
end
|
||||
subgraph Commit
|
||||
Majority[majority_candidate_locked durable self]
|
||||
Committed[committed_lsn]
|
||||
end
|
||||
subgraph Election
|
||||
Fresh[freshness floor from durable WAL]
|
||||
Promote[raft_cb_become_leader promotion fence]
|
||||
end
|
||||
Hook --> Append
|
||||
Append --> DurLsn
|
||||
Append --> ReplApply
|
||||
DurLsn --> Majority
|
||||
ReplApply --> Majority
|
||||
Majority --> Committed
|
||||
Committed --> Barrier
|
||||
Barrier --> Reply[reply ok to client]
|
||||
Recover --> Fresh
|
||||
Recover --> Promote
|
||||
Fresh --> Promote
|
||||
DurLsn --> Promote
|
||||
```
|
||||
|
||||
**Architecture Integration**
|
||||
- **Selected pattern:** extend-in-place for the durability precondition (Req 1, 3) plus a small testable fence helper for promotion eligibility (Req 2). No new module — `durable_lsn` is one atomic on `persist_t`.
|
||||
- **Boundaries:** persistence owns `durable_lsn` and the recovery verdict; the commit engine consumes `durable_lsn`; election consumes the recovery verdict + durable-backed floor. No shared ownership of the commit index.
|
||||
- **Patterns preserved:** commit-barrier reply gate, `in_catch_up` fence concept, `XCE_TESTING` hooks, the standby persist-before-ACK already in place.
|
||||
- **Steering compliance:** never block the iomgr thread (fsync lands on the off-iomgr commit path); re-verify against `raft/raft.tla` (commit rule, leader completeness) per `CLAUDE.md`.
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice / Version | Role in Feature | Notes |
|
||||
|-------|------------------|-----------------|-------|
|
||||
| Backend / Services | C (gnu17), controller daemon | Durable commit, recovery integrity, promotion fence | Extends `persist.c`, `commit.c`, `ha.c`, `cluster.c` |
|
||||
| Data / Storage | Local WAL segments + `fdatasync` | Synchronous durability; integrity-verified recovery | `posix_fallocate`'d segments; `fdatasync` (data + pre-allocated metadata) |
|
||||
| Infrastructure / Runtime | `--wal-sync-ms` (new) + `XCE_TESTING` seam | Deterministic durability verification | Follows `--sync-timeout-ms` (`OPT_SYNC_TIMEOUT`) precedent |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Modified Files
|
||||
- `src/controller/persist.c` — `persist_wal_async`: append then **signal the flusher** (no inline per-entry fsync); convert `fsync_worker` from a timer wait into a **demand-driven group-commit flusher** (wake on append-signal, `fdatasync`, advance `durable_lsn` to the highest synced LSN, broadcast the commit-eval condvar); keep its thread/mutex/cond lifecycle in `persist_start`/`persist_stop`/`persist_fini`. `replay_wal`/`persist_recover`: return an integrity verdict; re-derive durable floor from the recovered WAL tip. `persist_wal_sync_append` (dormant, no production caller): delete or re-route through the durable-majority evaluation — it must not advance the commit position on majority ACK without the durable precondition (task 2.4).
|
||||
- `src/include/persist.h` — add `_Atomic uint64_t durable_lsn` and a recovery-verdict type to `persist_t`/the recover signature; keep `fsync_*` fields (now driving the group-commit flusher); add an override field to `persist_config_t` (or its setter).
|
||||
- `src/controller/commit.c` — `majority_candidate_locked`: use `durable_lsn` (not `highest_appended_lsn`) for the self contribution `cap`.
|
||||
- `src/controller/ha.c` — `raft_cb_become_leader`: insert the pre-promotion eligibility gate; `ha_vote_self_freshness`: derive floor from durable WAL state; fence-release on catch-up completion. New `XCE_TESTING` durability-verification hook in the seam region (`ha.c:2270-2335`).
|
||||
- `src/controller/cluster.c` — vote / PreVote freshness comparison (`cluster.c:380-396`, `467-474`) consume the durable-backed floor; promotion path honors the fence.
|
||||
- `src/include/config.h` — keep `DEFAULT_WAL_SYNC_MS`; reconcile the stale `50` in `persistence-and-ha-design.md`.
|
||||
- `src/controller/main.c` — parse `--wal-sync-ms` (and optionally `--wal-sync-disable`) into `persist_config_t`.
|
||||
- `src/controller/myapp.c` / `cluster.c` (`persist_config_defaults` callers) — apply the override before `persist_init`.
|
||||
|
||||
### New Files
|
||||
- `src/controller/testsuite/test_commit_durability.c` — unit/white-box tests for `durable_lsn` advancement, self-counts-durable, recovery verdict, promotion fence (Req 1, 2, 3).
|
||||
- `src/tools/test-commit-durability.sh` — **exists** (soak E2E); extend to drive the new `--wal-sync-*` seam for a deterministic mode.
|
||||
|
||||
> Each modified function keeps one responsibility: persistence owns durability + recovery verdict; commit owns majority math (consuming `durable_lsn`); election owns eligibility (consuming the verdict + floor).
|
||||
|
||||
## System Flows
|
||||
|
||||
### Durable commit (reply ok ⇒ persisted on a majority)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant Primary
|
||||
participant Standby
|
||||
participant Flusher
|
||||
Client->>Primary: submit op
|
||||
Primary->>Primary: persist_wal_async pwrite then signal flusher
|
||||
Primary->>Standby: replicate entry
|
||||
Flusher->>Flusher: batched fdatasync then durable_lsn advance then wake commit
|
||||
Standby->>Standby: persist_wal_write_replicated fdatasync
|
||||
Standby-->>Primary: WalAck acked_lsn
|
||||
Primary->>Primary: majority_candidate_locked self equals durable_lsn plus standby acks
|
||||
Primary->>Primary: committed_lsn advance then commit barrier releases
|
||||
Primary-->>Client: reply ok
|
||||
```
|
||||
|
||||
Key decisions: the reply is released only after `committed_lsn` covers the entry, and `committed_lsn` now requires a **durable** self plus a **durable** standby majority. The `fdatasync` runs on the dedicated flusher thread (the repurposed `fsync_worker`), batched across all pending appends; the AFTER_EVENT caller blocks on the commit barrier (off iomgr) until `durable_lsn` + standby acks cover its entry, so heartbeats are unaffected (Req 1.6) and one `fdatasync` amortizes across the high-volume RETAIN-fact stream.
|
||||
|
||||
### Recovery integrity & promotion fence
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
Start[startup persist_recover] --> Replay[replay_wal]
|
||||
Replay --> Verdict{integrity verdict}
|
||||
Verdict -->|clean eof| Floor[set durable floor from WAL tip]
|
||||
Verdict -->|corruption truncated| Truncated[mark recovery_truncated]
|
||||
Floor --> Eligible[eligible to lead]
|
||||
Truncated --> CanCatchUp{healthy peer reachable}
|
||||
CanCatchUp -->|yes| Fence[fence from promotion until caught up]
|
||||
CanCatchUp -->|no and needed for quorum| Fatal[FATAL fail loud]
|
||||
Fence --> Caught{caught up past last committed boundary}
|
||||
Caught -->|yes| Eligible
|
||||
```
|
||||
|
||||
Key decisions: corruption is never silently accepted (Req 2.1). A truncated member is fenced from promotion and catches up from a healthy primary (Req 2.2, 2.6); if it cannot reach a consistent committed state and the cluster needs it, the controller fails loudly (Req 2.4), honoring the project's fail-loud preference. A candidate never advertises or leads beyond its durable WAL tip (Req 2.3), so a hollow node cannot win even under degraded quorum (Req 2.5).
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Components | Interfaces | Flows |
|
||||
|-------------|---------|------------|------------|-------|
|
||||
| 1.1 | reply ok ⇒ durable on majority | DurableCommit | `commit barrier`, `durable_lsn` | Durable commit |
|
||||
| 1.2 | self counts only when durable | DurableCommit | `majority_candidate_locked` | Durable commit |
|
||||
| 1.3 | standby persists before ACK | DurableCommit (pin) | `persist_wal_write_replicated` | Durable commit |
|
||||
| 1.4 | no premature ok | DurableCommit | `persist_commit_barrier` | Durable commit |
|
||||
| 1.5 | survives minority crash+restart | DurableCommit | durable-majority property | Durable commit / soak |
|
||||
| 1.6 | durability work preserves liveness | DurableCommit | off-iomgr placement | Durable commit |
|
||||
| 2.1 | no silent truncate-accept | RecoveryIntegrity | `replay_wal` verdict | Recovery fence |
|
||||
| 2.2 | fence until caught up | PromotionFence | promotion gate, `in_catch_up` | Recovery fence |
|
||||
| 2.3 | never lead below high-water | PromotionFence + FreshnessFloor | `raft_cb_become_leader`, `ha_vote_self_freshness` | Recovery fence |
|
||||
| 2.4 | fail-loud when unrecoverable | RecoveryIntegrity | FATAL policy | Recovery fence |
|
||||
| 2.5 | degraded quorum preserves invariant | DurableCommit + PromotionFence | durable-majority + fence | Both |
|
||||
| 2.6 | re-eligibility on catch-up | PromotionFence | fence-release hook | Recovery fence |
|
||||
| 3.1 | configurable durability cadence | DurabilityConfig | `--wal-sync-ms` | — |
|
||||
| 3.2 | recoverable after ungraceful kill in verify mode | DurabilityTestSeam | `XCE_TESTING` hook | soak deterministic mode |
|
||||
| 3.3 | identical external semantics in/out of mode | DurabilityTestSeam | seam affects timing only | — |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|
||||
|-----------|-------|--------|--------------|--------------------------|-----------|
|
||||
| DurableCommit | Persistence + Commit | Synchronous persist + self-counts-durable | 1.1–1.6, 2.5 | persist_t (P0), commit_engine (P0), repl ACK (P1) | State |
|
||||
| RecoveryIntegrity | Persistence | Verdict on recovery; durable floor; FATAL policy | 2.1, 2.4 | replay_wal (P0) | State |
|
||||
| PromotionFence | Election | Eligibility gate + fence-release | 2.2, 2.3, 2.6, 2.5 | RecoveryIntegrity (P0), in_catch_up (P1) | State |
|
||||
| FreshnessFloor | Election | Durable-backed vote floor & advertisement | 2.3 | durable_lsn (P0), ha_state (P1) | State |
|
||||
| DurabilityConfig + TestSeam | Runtime / Test | Override cadence; verification hook | 3.1–3.3 | persist_config (P0), XCE_TESTING (P1) | State |
|
||||
|
||||
### Persistence + Commit
|
||||
|
||||
#### DurableCommit
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Make the primary persist before it contributes to the commit majority |
|
||||
| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 2.5 |
|
||||
|
||||
- `persist_wal_async` appends and signals the flusher; it does **not** fsync inline per entry.
|
||||
- The **group-commit flusher** (repurposed `fsync_worker`) wakes on the append-signal, performs one `fdatasync`, advances `durable_lsn` to the highest synced LSN, and broadcasts the commit-eval condvar — amortizing one sync across all entries pending at wake time (the high-volume RETAIN stream benefits most).
|
||||
- `majority_candidate_locked` caps the self contribution at `durable_lsn`, not `highest_appended_lsn`.
|
||||
- Standby persist-before-ACK is preserved and regression-pinned (`persist_wal_write_replicated:515`); the standby path may later adopt the same flusher (out of scope here).
|
||||
- Checkpoint markers retain their existing inline `fdatasync` (`persist.c:573`).
|
||||
- `durable_lsn` is monotonic, never exceeds the live segment tip; on `persist_truncate_wal_above`, clamp `durable_lsn` to the new tip.
|
||||
|
||||
**Dependencies**
|
||||
- Inbound: AFTER_EVENT hooks (`job.c`, `queue.c`, `worker.c`, `join.c`), `ha.c` WAL_NOOP — purpose: append path (P0).
|
||||
- Outbound: `commit_engine_t` — purpose: self contribution input (P0).
|
||||
|
||||
**Contracts**: State [x]
|
||||
|
||||
##### State Management
|
||||
- **State model:** `_Atomic uint64_t durable_lsn` on `persist_t`; invariant `durable_lsn <= current_lsn - 1`, monotonic non-decreasing except clamp-on-truncate.
|
||||
- **Persistence & consistency:** `durable_lsn` advances only after a successful `fdatasync` in the flusher; the commit engine reads it for self-coverage so `committed_lsn` ⇒ durable-majority.
|
||||
- **Concurrency strategy:** the flusher takes `p->lock` to read the appended tip and `fdatasync`, advances `durable_lsn`, releases, then signals commit-eval; `majority_candidate_locked` reads `durable_lsn` via atomic load. Define the append-signal → flush → commit-wake ordering precisely so a WAIT caller cannot miss a wake (lost-wakeup guard).
|
||||
|
||||
**Implementation Notes**
|
||||
- Integration: the WAIT-mode caller blocks on the commit barrier off the iomgr thread; only the flusher thread blocks on `fdatasync`. Heartbeats unaffected.
|
||||
- Validation: assert `committed_lsn <= durable_lsn` on the local member after every commit advance (debug build); verify the flusher batches (one fsync advances `durable_lsn` past multiple LSNs under load).
|
||||
- Risks: **lost-wakeup / batching-window correctness** between `persist_wal_async` signal and the flusher (an entry must never be considered durable before its `fdatasync`); design the signal as level-triggered on the appended-vs-synced gap, not edge-triggered. Bounded flush latency vs throughput is a tunable (`--wal-sync-ms` becomes the max batching delay, not a durability gap).
|
||||
|
||||
### Persistence — RecoveryIntegrity
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Distinguish clean log end from corruption; re-derive durable floor; fail loud when unrecoverable |
|
||||
| Requirements | 2.1, 2.4 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `replay_wal`/`persist_recover` return a verdict: `RECOVER_CLEAN` (reached a clean entry boundary) vs `RECOVER_TRUNCATED` (stopped at CRC break / mismatch / short read).
|
||||
- On `RECOVER_TRUNCATED`, the member is marked recovery-truncated and is NOT treated as a clean recovery; it must not advertise or lead beyond the last intact LSN.
|
||||
- The recovered durable floor (highest intact replayed LSN) is published so the freshness advertisement is durable-backed.
|
||||
- FATAL policy: if a recovery-truncated member cannot reach a healthy peer to catch up and the cluster requires it, the controller terminates loudly rather than serve divergent state.
|
||||
|
||||
**Contracts**: State [x]
|
||||
|
||||
##### State Management
|
||||
- **State model:** verdict enum + `recovery_truncated` flag (on `ha_state_t` or `persist_t`); recovered durable floor = highest intact LSN.
|
||||
- **Persistence & consistency:** does not persist `committed_lsn`; only re-derives the durable log tip from on-disk segments.
|
||||
- **Concurrency strategy:** runs single-threaded during `cluster_init_phase2` before the node participates.
|
||||
|
||||
**Implementation Notes**
|
||||
- Integration: composes with `ha-convergence-recovery` — that spec truncates an **intact** log on demotion/rejoin; this spec refuses to accept a **corrupt** log silently.
|
||||
- Validation: unit test injects a CRC break at LSN 1 and asserts verdict `RECOVER_TRUNCATED` + non-promotability.
|
||||
- Risks: must not FATAL on a legitimately empty first-boot WAL (no entries ≠ corruption).
|
||||
|
||||
### Boot Policy Decision Table (task 3.2)
|
||||
|
||||
> **Status:** DECISION recorded (task 3.2). No production behavior change is made by 3.2 — this table is the contract that task **3.3** (the FATAL branch) and task **4.3** (the promotion fence + release) implement. The skeleton tests in `test_commit_durability.c` (one per row, `TEST_IGNORE`-skipped pending 3.3/4.3) encode each row's expected outcome.
|
||||
|
||||
**Decision (Req 2.2, 2.4):** A member that boots with a structurally clean log but below the cluster's committed high-water mark, or with a recovery-**truncated** log, is **fenced from leadership** at boot (never promoted while behind/truncated). It then **attempts catch-up** from a healthy peer. It becomes eligible **only** once catch-up brings its durable position to a committed-consistent state at/above the cluster boundary. It terminates **loudly (FATAL)** only on a **bounded** failure to reach that state — i.e. it cannot recover locally **and** cannot catch up within the bounded window (no healthy peer reachable / cannot reach a committed-consistent state).
|
||||
|
||||
**Invariant (authoritative):** the fence is **never a silent indefinite wait**. A truncated/behind member always resolves to exactly one of two terminal outcomes: it **catches up → ELIGIBLE**, or it **FATALs (fail-loud)** on a bounded failure. There is no third "fenced forever, still running, silently divergent" state. Snapshot-sentinel recovery is treated as CLEAN-but-behind (catch-up), never FATAL on a legitimate sentinel.
|
||||
|
||||
**Inputs (evaluated at boot / promotion-eligibility):**
|
||||
- **verdict** — recovery integrity verdict from task 3.1: `CLEAN` vs `TRUNCATED`.
|
||||
- **durable-vs-HWM** — local durable position relative to the cluster committed high-water mark: `at/above` vs `below`.
|
||||
- **peer** — a healthy peer reachable for catch-up: `yes` vs `no`.
|
||||
- **catch-up** — catch-up reaches a committed-consistent state within the bounded window: `succeeds` vs `fails` (n/a when no catch-up is attempted).
|
||||
|
||||
**Outcomes:**
|
||||
- **ELIGIBLE** — may promote (no fence).
|
||||
- **FENCE+CATCH-UP → ELIGIBLE** — ineligible to promote; catch up from a healthy peer; regain eligibility on success (Req 2.6).
|
||||
- **FATAL** — fail loud; terminate with the truncation/boundary logged rather than serve divergent state (Req 2.4).
|
||||
|
||||
| # | verdict | durable-vs-HWM | healthy peer | catch-up result | Outcome | Rationale / Req |
|
||||
|---|---------|----------------|--------------|-----------------|---------|-----------------|
|
||||
| 1 | CLEAN | at/above | n/a | n/a | **ELIGIBLE** | Log intact and not behind — normal promotable member. (Req 2.3) |
|
||||
| 2 | CLEAN | below | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Intact but behind the HWM; must not lead below the boundary; catch up then promote. (Req 2.2, 2.3, 2.6) |
|
||||
| 3 | TRUNCATED | (below — a truncated log is by definition not provably at/above) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Structurally truncated but a healthy peer exists; fence, catch up, regain eligibility. Never FATAL when a peer is reachable. (Req 2.2, 2.6) |
|
||||
| 4 | TRUNCATED | below | no | fails (bounded window elapses / no committed-consistent state reachable) | **FATAL** | Cannot recover locally AND cannot catch up — bounded failure ⇒ fail loud, never silent indefinite fence. (Req 2.4) |
|
||||
| 4b | CLEAN | below | no | fails (bounded window elapses, no peer to advance from) | **FATAL** | Behind the HWM and no healthy peer to catch up from within the bounded window — same bounded-failure terminal as row 4; never a silent indefinite fence. (Req 2.4, 2.2) |
|
||||
| 5 | CLEAN (snapshot-sentinel: committed at sentinel LSN, no sub-sentinel entries) | below (no entries below the sentinel) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption; treat as must-catch-up, **never FATAL** on a legitimate sentinel. (Req 2.1, 2.2) |
|
||||
| 6 | CLEAN | at/above (empty first-boot WAL, fresh cold start: durable=0, HWM=0) | n/a | n/a | **ELIGIBLE** | An empty first-boot WAL is not behind anything — cold-start members are promotable. (Req 2.1, 2.3) |
|
||||
|
||||
**Notes on the dimensions:**
|
||||
- A `TRUNCATED` verdict is never simultaneously "provably at/above the HWM": truncation means the log stops below its last intact boundary, so a truncated member is treated as behind for fence purposes regardless of the volatile in-memory position (this is why 4.2/3.4 derive the freshness floor from the **durable** WAL tip, not a volatile value).
|
||||
- "bounded window" is the catch-up deadline pinned by task 3.3; the policy requires it to be **finite** so the fence cannot become a silent indefinite wait.
|
||||
- The `healthy peer = no` + `at/above` combination does not need catch-up and is covered by row 1 (ELIGIBLE) — a member already at/above the HWM is promotable without a peer.
|
||||
|
||||
### Election — PromotionFence + FreshnessFloor
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Bar a truncated/behind member from leadership until caught up; advertise/vote on durable WAL state |
|
||||
| Requirements | 2.2, 2.3, 2.5, 2.6 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- Pre-promotion gate in `raft_cb_become_leader`: if recovery-truncated, or local durable position < cluster committed high-water mark, refuse promotion (set/extend `in_catch_up`-style fence) and instead catch up from a healthy primary.
|
||||
- `ha_vote_self_freshness`: floor derives from the **durable** WAL tip (re-derived on recovery), so a truncated member cannot over-advertise a volatile `last_known_committed_lsn`. Honors `CLAUDE.md`: the post-election safe-commit point still comes from voter replies, never from a stale ctrlhello.
|
||||
- Fence-release: when catch-up brings the member past its last-known committed boundary, restore eligibility (hook the existing catch-up completion / re-inclusion path).
|
||||
- Degraded quorum: because a candidate cannot lead beyond its durable tip, a hollow node cannot win even at minimal quorum — preserving Req 1's invariant across leadership changes.
|
||||
|
||||
**Contracts**: State [x]
|
||||
|
||||
**Implementation Notes**
|
||||
- Integration: reuses `in_catch_up` (ha.c:954, cluster.c:529) and the PreVote/vote freshness comparison (cluster.c:380-396, 467-474); extends the fence from "streaming standby" to "recovery-truncated / behind-high-water promoter."
|
||||
- Validation: simulate a node restarting truncated; assert it is denied promotion until catch-up, then re-enabled (Req 2.2/2.6).
|
||||
- Risks: interaction with `ha-single-leader-role` own-term gate and `ha-prevote` — re-verify the combined eligibility predicate against `raft.tla`.
|
||||
|
||||
### Runtime / Test — DurabilityConfig + TestSeam
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Deterministic, configurable durability timing for verification |
|
||||
| Requirements | 3.1, 3.2, 3.3 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `--wal-sync-ms` plumbed `main.c → persist_config_t` before `persist_init`; default unchanged (`DEFAULT_WAL_SYNC_MS`). Under group-commit this bounds the **maximum batching delay** (a flush fires on the append-signal or when this elapses), never a durability gap — an acked op is always durable regardless of the value.
|
||||
- The seam's verification role: an `XCE_TESTING` hook (and/or a large `--wal-sync-ms` to widen the batching window) lets a test assert an acknowledged op is recoverable after an immediate ungraceful termination — proving durability comes from the commit gate, not the flush cadence.
|
||||
- External acknowledgment semantics are identical with or without the mode (timing controllability only).
|
||||
|
||||
**Contracts**: State [x]
|
||||
|
||||
**Implementation Notes**
|
||||
- Integration: follows `--sync-timeout-ms` (`OPT_SYNC_TIMEOUT`) and the `ha.c:2270-2335` hook convention.
|
||||
- Validation: deterministic E2E mode in `test-commit-durability.sh` drives the seam so the soak becomes a deterministic RED on the unfixed build and GREEN after.
|
||||
- Risks: ensure the seam is compiled out of production (`#ifdef XCE_TESTING`).
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Error Strategy
|
||||
- **Durability failure** (`fdatasync` error in the flusher): `durable_lsn` does not advance, so the entry is NOT counted durable, the commit does not advance, and the client does not receive *ok* (Req 1.4). The flusher surfaces the I/O error loudly (fail-loud); a persistent local sync failure is treated as a member fault, not silently retried into a false-durable state.
|
||||
- **Recovery corruption** (`RECOVER_TRUNCATED`): never silently accepted (Req 2.1). Fence from promotion + catch up; FATAL when catch-up is impossible and the node is required (Req 2.4) — fail-loud.
|
||||
- **Promotion while behind**: refused at the gate (Req 2.3); logged with the local durable tip vs the cluster high-water mark.
|
||||
|
||||
### Monitoring
|
||||
- Log every `durable_lsn` regression-clamp on truncate, every recovery verdict (with the truncation boundary), every promotion refusal (durable tip vs high-water), and every fence set/release — operator-visible, once per transition (consistent with the quorum-transition logging already in the HA stack).
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- `durable_lsn` advances only after `fdatasync`; `majority_candidate_locked` self-cap equals `durable_lsn` not `highest_appended_lsn` (1.2).
|
||||
- `replay_wal` returns `RECOVER_TRUNCATED` on an injected CRC break at LSN 1 and `RECOVER_CLEAN` on an intact log (2.1).
|
||||
- Promotion gate denies a recovery-truncated / behind-high-water member and re-enables it after catch-up (2.2, 2.6).
|
||||
- `ha_vote_self_freshness` floor reflects the durable WAL tip, not a stale in-memory `last_known_committed_lsn` (2.3).
|
||||
- Standby `persist_wal_write_replicated` `fdatasync`s before ACK — regression pin (1.3).
|
||||
|
||||
### Integration Tests
|
||||
- Commit advances `committed_lsn` only when self is durable + a standby ACK is durable; reply released only after (1.1, 1.4).
|
||||
- Kill the sole durable contributor mid-commit; acknowledged op survives on the majority (1.5).
|
||||
- Durability work does not delay heartbeat handling beyond tolerance (1.6).
|
||||
|
||||
### E2E / Failover
|
||||
- `src/tools/test-commit-durability.sh` soak: rolling SIGKILL+restart under load asserts every acknowledged job survives (1.1, 1.5, 2.x). With the `--wal-sync-*` seam, run a **deterministic** variant: widen/disable background timing, kill a durable contributor in-window, assert survival — RED on the current build, GREEN after the fix (3.1, 3.2).
|
||||
|
||||
### Performance
|
||||
- Measure submit latency delta from synchronous `fdatasync` vs the existing replication round-trip; confirm it is within the round-trip already paid before deciding group-commit is needed (research item).
|
||||
|
||||
## Performance & Scalability
|
||||
- **Group-commit** amortizes one `fdatasync` across all entries pending at flush time, so the per-entry sync cost falls as load rises — critical for the high-volume `WAL_MODE_RETAIN` fact stream (job start/finish/fail/rusage) which is not client-latency-bound. `--wal-sync-ms` bounds the maximum batching delay (a latency/throughput tunable), never a durability gap. The `job-array-single-wal-record` roadmap item independently cuts per-array entry count. Target: zero acknowledged-job loss across failover (correctness) takes precedence over per-submit latency; group-commit keeps the throughput cost sub-linear in write rate.
|
||||
|
||||
## Open Questions / Gating Items (resolve in `/kiro-spec-tasks`)
|
||||
- **Freshness-floor formula (gating, election-safety):** pin the exact reconciliation of the durable-WAL-derived floor vs the existing `max(applied_lsn, last_known_committed_lsn)`. Too low ⇒ a truncated node still wins (the bug); too high ⇒ election livelock. Make a `raft/raft.tla` Leader-Completeness **and liveness** re-check a blocking sub-task before the freshness change merges (Req 2.3).
|
||||
- **FATAL-vs-fence decision point (gating):** a booting truncated member cannot know it is "required for quorum" at `cluster_init`. Define the concrete policy: fence at boot, attempt catch-up from a healthy primary, and FATAL only on a bounded failure to reach a committed-consistent state — never a silent indefinite fence (Req 2.2, 2.4). **RESOLVED (task 3.2):** see the **Boot Policy Decision Table (task 3.2)** above for the recorded input-state → outcome table; tasks 3.3 (FATAL branch) and 4.3 (fence + release) implement it.
|
||||
- **Flusher lost-wakeup proof:** the append-signal → flush → commit-wake path must be shown free of lost wakeups (level-triggered on the appended-vs-synced gap) before relying on it for the durability invariant (Req 1.1, 1.2).
|
||||
- **Snapshot-recovery edge:** current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but **no log entries below it**. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never FATAL on a legitimate sentinel snapshot). Interacts with the roadmap `entity-bearing snapshots` item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2).
|
||||
- **Dormant synchronous-append path:** `persist_wal_sync_append` advances the commit position directly on majority ACK and assumes inline fsync — a premise group-commit removes (task 2.4). It has no production caller today but must be deleted or re-routed so it cannot reintroduce ACK-without-durability commit (Req 1.1, 1.2).
|
||||
|
|
@ -57,7 +57,7 @@
|
|||
- _Requirements: 2.1_
|
||||
- _Boundary: RecoveryIntegrity_
|
||||
|
||||
- [ ] 3.2 Pin the fail-loud vs fence boot policy as a decision with a verification table
|
||||
- [x] 3.2 Pin the fail-loud vs fence boot policy as a decision with a verification table
|
||||
- Pin the concrete policy (gating item, decision only — no behavior change yet): a truncated member fences from leadership at boot, attempts catch-up from a healthy peer, and terminates loudly only on a bounded failure to reach a committed-consistent state — never a silent indefinite fence. Encode the decision as a table of input states → expected outcome.
|
||||
- Observable: the policy and its decision table are recorded, and skeleton tests assert each row's expected outcome (initially failing), establishing the contract 3.3 and 4.3 implement.
|
||||
- _Requirements: 2.2, 2.4_
|
||||
|
|
@ -128,3 +128,4 @@
|
|||
- 2.3: Standby persist-before-ACK pinned + Req-1.3 hole closed. `persist_wal_write_replicated` (persist.c:741) now rc-gates the inline `fdatasync`: on failure (or `flusher_fsync_forced_fail`) it logs ERROR + `return -EIO` (symmetric with the 2.1 flusher gate) — the standby will NOT ACK a non-durable entry. Caller `ha_apply_single_entry` (ha.c:1135-1166) ALREADY routes a negative return to no-apply/no-ACK (`ha_send_wal_ack_reply` reached only on rc==0), so NO ha.c change was needed (boundary stayed persistence-only). Ordering pin = `#ifdef XCE_TESTING atomic_uint test_replicated_fdatasync_count` incremented at the sync site; `test_replicated_persist_syncs_before_return` asserts it increments inside the call (page-cache means a disk re-read can't prove fsync — use the counter). `test_replicated_sync_failure_no_ack` REDs without the gate. Seam production-excluded (nm-verified). HOUSEKEEPING: controller tests `mkdtemp` into `/tmp/xce_*` and don't clean up — thousands accumulate and eventually make `mkdtemp` fail → misleading suite-wide "Expected Non-NULL" cascade. Run `rm -rf /tmp/xce_*` if that appears (pre-existing test pattern, not a regression).
|
||||
- 2.4: SPEC INACCURACY corrected — `persist_wal_sync_append` is NOT dormant; it is the LIVE job-submit path (`persist_job_hook_fun` calls it at persist.c:1911). Took the "route through durable-majority eval" option (delete infeasible). Production path (ce!=NULL, lsn!=0) was already durable-safe via 2.2's `majority_candidate_locked` self-cap. Closed the hazard in the LEGACY branch: the former unconditional `atomic_store(committed_lsn, written_lsn)` on majority ACK is now `sync_append_legacy_commit_store()` which advances ONLY when `written_lsn <= persist_durable_lsn(p)` (LSN-0 edge: store of 0 valid). Added the design's debug invariant `committed_lsn <= durable_lsn` (`#ifdef XCE_TESTING` assert at commit.c:162 + inside the legacy store) as the future-proof guard. Fixed the stale "after local fsync via persist_wal_async" comment (2.1 deferred the fsync). CODE-SEARCH (authoritative): the ONLY `committed_lsn` stores are — engine advance (commit.c, durable-capped by 2.2), legacy sync_append (now gated), and the two `persist_truncate_wal_above` cutoff stores (NOT ACK-driven; each immediately `clamp_durable_lsn_locked`). ha.c/ha_convergence.c stores are `last_known_committed_lsn` (DIFFERENT field). So no path advances commit on ACK without the durable precondition. Seam `persist_test_sync_append_legacy_commit_store` is production-excluded.
|
||||
- 3.1: `recover_verdict_t {RECOVER_CLEAN, RECOVER_TRUNCATED}` (persist.h) + `persist_t.recovery_verdict` field + accessors `persist_recovery_verdict()`/`persist_recovery_truncated()` (NOT XCE_TESTING-gated — 4.3 consumes them in production). `replay_wal` classifies: CRC chain break / CRC mismatch / short header/payload read → TRUNCATED; clean entry boundary / pre-allocated-zero tail / empty WAL / snapshot-sentinel+clean-replay → CLEAN. Verdict is monotonic (init CLEAN once; only ever set TRUNCATED; a later clean segment can't un-set it). Verdict = STRUCTURAL integrity, NOT "behind the cluster" (the behind-high-water case is 4.2/4.3). 3.1 ONLY sets state + WARNs — does NOT FATAL (3.3) or fence (4.3). DRY: did NOT extract a shared scanner (replay_wal mutates persist_t; the 1.2 seam is read-only) — no third copy added. ENV FINDING (see cerebrum 2026-06-18): this sandbox's /tmp can be an overlay that EIOs on fdatasync; because committed_lsn is now durable-gated (2.x), an fdatasync EIO correctly HOLDS committed_lsn, which makes `test_fact_hook_append_is_retained_and_nonblocking` (commit_barrier_rollout) FAIL on overlay /tmp — confirmed pre-existing (fails identically on the clean pre-spec baseline), NOT a regression. Final validation (5.x) must run on a real fsync-able fs.
|
||||
- 3.2: DECISION-ONLY (no production change). Boot Policy Decision Table recorded in design.md §"Boot Policy Decision Table (task 3.2)" — 7 rows over {verdict CLEAN/TRUNCATED × at-above/below HWM × peer reachable × bounded-catch-up succeeds/fails} + snapshot-sentinel + empty-first-boot. Outcomes: ELIGIBLE / FENCE+CATCH-UP→ELIGIBLE / bounded FATAL. AUTHORITATIVE INVARIANT: never a silent indefinite fence — every truncated/behind member resolves to exactly one terminal (catch-up→ELIGIBLE, or BOUNDED FATAL). Snapshot-sentinel is NEVER FATAL on a legitimate sentinel. 7 skeleton tests in test_commit_durability.c (row1,2,3,4,4b,5,6) are `TEST_IGNORE_MESSAGE`-skipped (PENDING 3.3/4.3) so the suite stays GREEN — they are the RED contract 3.3 (FATAL rows) and 4.3 (fence/eligible rows) turn green. NOTE: pre-existing ASAN leaks in unrelated tests (test_demotion_convergence, ha_start_election_async, ha_rdhup_integration) make the STANDALONE controller_unit_test exit=1, but meson reports Fail:0 — don't be fooled by standalone exit code.
|
||||
|
|
|
|||
|
|
@ -990,6 +990,135 @@ static void test_recover_verdict_snapshot_sentinel_clean(void)
|
|||
printf("test_recover_verdict_snapshot_sentinel_clean: PASS\n");
|
||||
}
|
||||
|
||||
/* ================================================================== *
|
||||
* task 3.2 — Boot Policy Decision Table (DECISION ONLY) *
|
||||
* *
|
||||
* Task 3.2 pins the fail-loud-vs-fence boot policy as a DECISION and *
|
||||
* encodes it as a table of input states -> expected outcome. It makes *
|
||||
* NO production behavior change: the FATAL branch is implemented by *
|
||||
* task 3.3 and the promotion fence + release by task 4.3. *
|
||||
* *
|
||||
* The decision table lives in design.md ("Boot Policy Decision Table *
|
||||
* (task 3.2)"). The skeletons below mirror it ONE TEST PER ROW. Each *
|
||||
* skeleton documents the row's inputs + the expected outcome it WILL *
|
||||
* assert once 3.3/4.3 land, then TEST_IGNORE_MESSAGE()-skips so the *
|
||||
* row is REGISTERED and reported as IGNORED (not failed) — the suite *
|
||||
* stays GREEN. These become the RED tests 3.3/4.3 turn GREEN. *
|
||||
* *
|
||||
* Inputs (boot / promotion-eligibility evaluation): *
|
||||
* verdict : CLEAN | TRUNCATED (from task 3.1) *
|
||||
* durable-vs-HWM : at/above | below (durable position) *
|
||||
* healthy peer : yes | no (catch-up source) *
|
||||
* catch-up : succeeds | fails (bounded window) *
|
||||
* Outcomes: *
|
||||
* ELIGIBLE | FENCE+CATCH-UP -> ELIGIBLE | FATAL *
|
||||
* INVARIANT: never a silent indefinite fence — a truncated/behind *
|
||||
* member either catches up (-> eligible) or FATALs on a BOUNDED *
|
||||
* failure. (Req 2.2, 2.4) *
|
||||
* ------------------------------------------------------------------ */
|
||||
|
||||
/* Row 1: CLEAN + durable at/above HWM -> ELIGIBLE.
|
||||
* A structurally intact log that is not behind the cluster boundary is
|
||||
* a normal promotable member: the promotion gate (4.3) must NOT fence
|
||||
* it. WILL ASSERT (4.3): the eligibility predicate returns "eligible"
|
||||
* for a CLEAN, at/above member; promotion proceeds. (Req 2.3) */
|
||||
static void test_boot_policy_row1_clean_at_or_above_eligible(void)
|
||||
{
|
||||
TEST_IGNORE_MESSAGE(
|
||||
"row 1 (CLEAN + at/above HWM -> ELIGIBLE): pending task 4.3 "
|
||||
"(promotion-eligibility gate). Will assert the gate admits a clean, "
|
||||
"not-behind member.");
|
||||
}
|
||||
|
||||
/* Row 2: CLEAN + durable below HWM + healthy peer + catch-up succeeds
|
||||
* -> FENCE+CATCH-UP -> ELIGIBLE.
|
||||
* Intact but behind: must NOT lead below the boundary; fence, catch up
|
||||
* from a healthy peer, regain eligibility. WILL ASSERT (4.3): a
|
||||
* below-HWM member is denied promotion until catch-up passes the last
|
||||
* committed boundary, then regains eligibility. (Req 2.2, 2.3, 2.6) */
|
||||
static void test_boot_policy_row2_clean_below_peer_catchup_eligible(void)
|
||||
{
|
||||
TEST_IGNORE_MESSAGE(
|
||||
"row 2 (CLEAN + below HWM + peer + catch-up ok -> FENCE+CATCH-UP -> "
|
||||
"ELIGIBLE): pending task 4.3 (fence + release). Will assert denied "
|
||||
"until caught up, then re-eligible.");
|
||||
}
|
||||
|
||||
/* Row 3: TRUNCATED + healthy peer + catch-up succeeds
|
||||
* -> FENCE+CATCH-UP -> ELIGIBLE.
|
||||
* Structurally truncated (task-3.1 verdict) but a healthy peer exists:
|
||||
* fence, catch up, regain eligibility — NEVER FATAL when a peer is
|
||||
* reachable. WILL ASSERT (4.3): a recovery_truncated member is denied
|
||||
* promotion, catches up from the peer, and is re-enabled; FATAL is NOT
|
||||
* triggered. (Req 2.2, 2.6) */
|
||||
static void test_boot_policy_row3_truncated_peer_catchup_eligible(void)
|
||||
{
|
||||
TEST_IGNORE_MESSAGE(
|
||||
"row 3 (TRUNCATED + peer + catch-up ok -> FENCE+CATCH-UP -> "
|
||||
"ELIGIBLE): pending tasks 3.3/4.3. Will assert truncated member "
|
||||
"fences, catches up, re-eligible; no FATAL when a peer is reachable.");
|
||||
}
|
||||
|
||||
/* Row 4: TRUNCATED + below HWM + NO healthy peer + catch-up fails
|
||||
* (bounded window elapses) -> FATAL.
|
||||
* Cannot recover locally AND cannot catch up within the bounded window:
|
||||
* fail loud rather than serve divergent state. WILL ASSERT (3.3): the
|
||||
* member terminates loudly (FATAL) with the truncation boundary logged
|
||||
* when the bounded catch-up window elapses without a committed-consistent
|
||||
* state. (Req 2.4) */
|
||||
static void test_boot_policy_row4_truncated_no_peer_bounded_fatal(void)
|
||||
{
|
||||
TEST_IGNORE_MESSAGE(
|
||||
"row 4 (TRUNCATED + below + no peer + bounded catch-up fails -> "
|
||||
"FATAL): pending task 3.3 (FATAL branch). Will assert loud "
|
||||
"termination with the truncation boundary logged on bounded failure.");
|
||||
}
|
||||
|
||||
/* Row 4b: CLEAN + below HWM + NO healthy peer + catch-up fails (bounded
|
||||
* window elapses) -> FATAL.
|
||||
* Behind the HWM with no peer to advance from within the bounded window:
|
||||
* same bounded-failure terminal as row 4 — never a silent indefinite
|
||||
* fence. WILL ASSERT (3.3): a behind member with no reachable peer FATALs
|
||||
* once the bounded window elapses. (Req 2.4, 2.2) */
|
||||
static void test_boot_policy_row4b_clean_below_no_peer_bounded_fatal(void)
|
||||
{
|
||||
TEST_IGNORE_MESSAGE(
|
||||
"row 4b (CLEAN + below + no peer + bounded catch-up fails -> FATAL): "
|
||||
"pending task 3.3. Will assert the behind-no-peer member FATALs on "
|
||||
"bounded failure — never a silent indefinite fence.");
|
||||
}
|
||||
|
||||
/* Row 5: CLEAN snapshot-sentinel (committed at sentinel LSN, NO
|
||||
* sub-sentinel entries) + below + healthy peer + catch-up succeeds
|
||||
* -> FENCE+CATCH-UP -> ELIGIBLE.
|
||||
* A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption:
|
||||
* treat as must-catch-up, NEVER FATAL on a legitimate sentinel. WILL
|
||||
* ASSERT (4.3): a snapshot-sentinel member is fenced+caught-up like any
|
||||
* behind member and is NEVER routed to the FATAL branch. (Req 2.1, 2.2)
|
||||
* (3.1 already verifies the verdict itself is CLEAN — see
|
||||
* test_recover_verdict_snapshot_sentinel_clean; this row pins the
|
||||
* downstream promotion-fence outcome.) */
|
||||
static void test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal(void)
|
||||
{
|
||||
TEST_IGNORE_MESSAGE(
|
||||
"row 5 (snapshot-sentinel CLEAN + below + peer -> FENCE+CATCH-UP -> "
|
||||
"ELIGIBLE, NEVER FATAL): pending task 4.3. Will assert a legitimate "
|
||||
"sentinel snapshot is treated as behind-must-catch-up, not FATAL.");
|
||||
}
|
||||
|
||||
/* Row 6: CLEAN empty first-boot WAL (durable=0, HWM=0, fresh cold start)
|
||||
* -> ELIGIBLE.
|
||||
* An empty first-boot WAL is not behind anything — cold-start members
|
||||
* are promotable. WILL ASSERT (4.3): the eligibility gate admits a
|
||||
* fresh cold-start member (no fence, no FATAL). (Req 2.1, 2.3) */
|
||||
static void test_boot_policy_row6_empty_first_boot_eligible(void)
|
||||
{
|
||||
TEST_IGNORE_MESSAGE(
|
||||
"row 6 (empty first-boot WAL CLEAN, durable=0/HWM=0 -> ELIGIBLE): "
|
||||
"pending task 4.3. Will assert a fresh cold-start member is "
|
||||
"promotable (no fence, no FATAL).");
|
||||
}
|
||||
|
||||
#endif /* XCE_TESTING */
|
||||
|
||||
/* ------------------------------------------------------------------ */
|
||||
|
|
@ -1016,6 +1145,16 @@ void test_commit_durability(void)
|
|||
RUN_TEST(test_recover_verdict_empty_first_boot_clean);
|
||||
RUN_TEST(test_recover_verdict_crc_break_first_entry_truncated);
|
||||
RUN_TEST(test_recover_verdict_snapshot_sentinel_clean);
|
||||
/* task 3.2 — Boot Policy Decision Table skeletons (IGNORED, pending
|
||||
* 3.3/4.3). Registered so each row is reported as IGNORED, not failed;
|
||||
* the suite stays GREEN. See design.md "Boot Policy Decision Table". */
|
||||
RUN_TEST(test_boot_policy_row1_clean_at_or_above_eligible);
|
||||
RUN_TEST(test_boot_policy_row2_clean_below_peer_catchup_eligible);
|
||||
RUN_TEST(test_boot_policy_row3_truncated_peer_catchup_eligible);
|
||||
RUN_TEST(test_boot_policy_row4_truncated_no_peer_bounded_fatal);
|
||||
RUN_TEST(test_boot_policy_row4b_clean_below_no_peer_bounded_fatal);
|
||||
RUN_TEST(test_boot_policy_row5_snapshot_sentinel_catchup_never_fatal);
|
||||
RUN_TEST(test_boot_policy_row6_empty_first_boot_eligible);
|
||||
#else
|
||||
printf("test_commit_durability: skipped (no XCE_TESTING)\n");
|
||||
#endif
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue