diff --git a/.claude/skills/xce-audit-invariants/SKILL.md b/.claude/skills/xce-audit-invariants/SKILL.md index cb87c389..1d20de65 100644 --- a/.claude/skills/xce-audit-invariants/SKILL.md +++ b/.claude/skills/xce-audit-invariants/SKILL.md @@ -76,6 +76,18 @@ have historically found real violations are: ref-whole-batch-before-dispatch fix). Check other batch-processing loops (vtaskmgr's drain, any peer-list iteration that can trigger eviction) for the same shape, not just `io_worker`. +7. **Field-complete apply parity** (new 2026-08-24, bug-1678) — for every WAL + fact type, is every field of the entity it materializes actually (a) + encoded into the WAL payload at write time, not just held in the + in-memory struct, and (b) decoded and adopted on BOTH apply seams + (`apply_committed_entry()` boot replay and `apply_wal_entry()` standby + catch-up)? `persist_apply_verdict()`'s `APPLY_CLEAN` only checks entity + count + cursor position — see `persistence.md`'s "`APPLY_CLEAN` is + entity-count + cursor-position only" note — so a field silently dropped + between dispatch-time mutation and WAL encode is invisible to it. Check + each `WAL_*` fact type's payload struct against the full entity fields a + client can observe post-restart (`xcecli`/apiserver-visible fields + especially — that's how bug-1678's dropped `allocs`/EXEC_HOST surfaced). Each subagent must report file:line evidence for every claimed violation — no speculative findings. Use the Agent tool directly (general-purpose or diff --git a/.kiro/steering/persistence.md b/.kiro/steering/persistence.md index 14a84fbf..8508a7dd 100644 --- a/.kiro/steering/persistence.md +++ b/.kiro/steering/persistence.md @@ -65,6 +65,8 @@ A **separate** verdict from `recover_verdict_t` (above): `persist_apply_verdict( **Unconditional, restart-only fence (deliberate — landed by `raft-apply-decoupling` task 5.1-5.3, closes bug-1418's promotion-gate liveness-deadlock family):** once `APPLY_DIVERGENT`, the promotion gate (`ha.c` `raft_cb_become_leader` and `promotion_fence_timeout_fn`) refuses `CTRL_PRIMARY` **unconditionally** — no peer/HWM/durable-floor condition may clear it live, since a divergent local materialization can't be trusted for a peer/catch-up comparison either. Only a fresh, clean `persist_recover()` replay clears it (`apply_verdict` has exactly two production write sites: `persist_init()`'s reset and `persist_recover()`'s boot-time computation — no live path resets it). **Do not "fix" an apply-divergent liveness stall by relaxing this gate** — the actual fix is eliminating the false-positive DIVERGENT verdicts at the source (a boot-time TOCTOU where a returning node's verify pass raced the live apply pump / live-replication apply path, bug-1416, closed via `persist_t.recovery_complete` + `persist_mark_recovery_pending` called before any network traffic is processed). +**`APPLY_CLEAN` is entity-count + cursor-position only — it does NOT prove per-field completeness (bug-1678).** `persist_apply_verdict()`'s two checks (WAL_JOB_SUBMIT-count-vs-table-growth, `last_applied_lsn` position) both passed while a job's `allocs` (the resource allocation the scheduler assigned) silently replayed as `NULL` after a full restart — `job.state` was correctly `RUN`, the row existed, the cursor was in the right place, and `APPLY_CLEAN` reported healthy the entire time. Root cause: `WAL_JOB_DISPATCH`'s payload (`wal_jobstate_t`) never carried the allocation at all, so no apply-side check could have caught the gap even in principle — the field was never in the log to compare against. **The real invariant `APPLY_CLEAN` should imply — but today only partially enforces — is: online-apply and replay-apply of the same committed log MUST produce byte-identical materialized state for every persisted field of every entity, not just matching row counts and an advancing cursor.** When adding a new WAL fact type or a new field to an existing one, verify explicitly that (a) the field is actually encoded into the WAL payload (not just the in-memory struct at dispatch time), and (b) both apply seams — `apply_committed_entry()` (primary boot replay) and `apply_wal_entry()` (standby catch-up / replicated-apply) — decode and adopt it. Tracked as unimplemented future work at the systemic level in `roadmap.md`'s raft-apply-decoupling item #4 / raft-sim-conformance follow-up (a generic per-field snapshot-diff regression harness); until that lands, treat this as a standing audit dimension (see `xce-audit-invariants` skill) rather than a one-time fix. + ## Fail-Loud, Never Self-Exit When a member cannot reach a committed-consistent state (recovery-truncated/behind, no healthy peer), it enters a **loud permanent fence**: log a loud operator-visible ERROR (truncation boundary vs cluster HWM), set the sticky `recovery_unrecoverable` flag, hold non-promotable, **stay alive, and keep retrying** — it self-heals when a peer returns. It MUST NEVER `exit`/`abort` as a designed response. The decision is the pure `persist_recover_decision` → `RECOVER_DECISION_{ELIGIBLE,FENCE,FENCE_UNRECOVERABLE}`; the action is `persist_recover_loud_fence_if_unrecoverable` (no terminal). Authoritative statement: `error-handling.md` + `src/controller/CLAUDE.md` "Process Lifecycle".