lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(raft-apply-decoupling): route DIVERGENT verdict through existing fence (task 5.2)

Browse source 
Add bool apply_divergent to the existing recover_policy_inputs_t;
persist_recover_decision() unconditionally short-circuits to the
existing RECOVER_DECISION_FENCE_UNRECOVERABLE when true, reusing the
same sticky recovery_unrecoverable flag and
persist_recover_loud_fence_if_unrecoverable() -- no new fence
mechanism, no ha.c changes needed (the existing promotion gate
already reads the flag independently), no new self-termination path.

Fixes a real bug found during review: the fence's operator-facing
ERROR() log unconditionally asserted a "no healthy peer reachable...
durable_floor_lsn < hwm" clause that was only truthful for the
pre-existing structural-fence trigger. For the new apply_divergent
trigger this produced literally false claims in test output
("100 < 100", "2 < 0"). Fixed by branching the log call itself on
apply_divergent so it states only the verified reason.
This commit is contained in:
Rongsong Shen 2026-07-04 12:22:09 +00:00
parent c005d40741
commit 099ee1a216
4 changed files with 417 additions and 24 deletions
Download patch file Download diff file Expand all files Collapse all files

100
src/controller/persist.c
View file

@ -3311,7 +3311,7 @@ int persist_recover(persist_t* p)
};
p->apply_verdict = persist_verify_apply_correctness(
p, &apply_exp, jobmgr_count_before_replay);
if (p->apply_verdict == APPLY_DIVERGENT)
if (p->apply_verdict == APPLY_DIVERGENT) {
ERROR("persist_recover: APPLY_DIVERGENT — materialized state does "
"not match the committed log tip (expected last_applied_lsn=%"
PRIu64 " new_job_count=%" PRIu64 "); a hidden apply defect is "
@ -3320,7 +3320,32 @@ int persist_recover(persist_t* p)
apply_exp.expected_new_job_count,
(verdict == RECOVER_TRUNCATED) ? "RECOVER_TRUNCATED"
: "RECOVER_CLEAN");
else
/* raft-apply-decoupling task 5.2 (Req 8.3/8.4): route the DIVERGENT
* verdict through the EXISTING non-promotable loud-fence mechanism
* (persist_recover_loud_fence_if_unrecoverable + the sticky
* recovery_unrecoverable flag) instead of a new fence path.
* apply_divergent=true fences UNCONDITIONALLY (see
* persist_recover_decision), so the peer/HWM fields below are
* inert placeholders here -- they are not yet known at this
* boot-time replay call site (unlike the structural verdict's own
* fence check, which stays deferred to promotion time in ha.c,
* where live peer/HWM context exists). verdict/recovered_tip are
* passed through only for accurate, consistent logging. The member
* stays ALIVE, non-promotable, and keeps retrying catch-up -- it
* never exits (controller-never-self-terminate invariant,
* CLAUDE.md). */
recover_policy_inputs_t apply_fence_in = {
.verdict = verdict,
.durable_below_hwm = false,
.healthy_peer_reachable = false,
.catch_up_succeeded = false,
.durable_floor_lsn = recovered_tip,
.cluster_hwm_lsn = 0,
.apply_divergent = true,
};
persist_recover_loud_fence_if_unrecoverable(p, &apply_fence_in);
} else
INFO("persist_recover: APPLY_CLEAN — materialized state matches the "
"committed log tip (last_applied_lsn=%" PRIu64 ")",
apply_cursor);
@ -3366,6 +3391,20 @@ recover_decision_t persist_recover_decision(const recover_policy_inputs_t* in)
if (in == NULL)
return RECOVER_DECISION_ELIGIBLE; /* defensive: nothing to fence */
/* raft-apply-decoupling task 5.2 (Req 8.3/8.4): an APPLY_DIVERGENT
* post-replay verdict is a NEW INPUT to this same decision table, not a
* new fence mechanism (design.md "Restart Replay & Verification"). It
* fences UNCONDITIONALLY -- every other field below is irrelevant once
* this is true. Unlike a structurally truncated/behind log (rows 2/3/5),
* where a healthy peer's missing entries can repair the gap and legitimately
* downgrade the outcome to a plain FENCE-then-catch-up, an apply-time
* defect means the LOCAL apply path itself produced the wrong
* materialized state from bytes it already had -- streaming more entries
* from a peer does not prove that path is now trustworthy, so this is
* never downgraded by peer reachability or catch-up success. */
if (in->apply_divergent)
return RECOVER_DECISION_FENCE_UNRECOVERABLE;
bool truncated = (in->verdict == RECOVER_TRUNCATED);
/* Rows 1 & 6: a structurally CLEAN log that is NOT behind the cluster
@ -3430,22 +3469,47 @@ persist_recover_loud_fence_if_unrecoverable(persist_t* p,
if (decision != RECOVER_DECISION_FENCE_UNRECOVERABLE)
return decision;
/* Loud, operator-visible alert: log the truncation boundary
* (durable_floor_lsn) and the cluster high-water mark so an operator sees
* exactly where the local log stops short of the cluster's committed state.
* The member STAYS ALIVE, stays non-promotable, and keeps retrying catch-up
* (Req 2.4) — it NEVER self-exits. */
const char* why = (in->verdict == RECOVER_TRUNCATED)
? "recovery-truncated log (structural corruption)"
: "log behind the cluster committed high-water mark";
ERROR("persist_recover: UNRECOVERABLE — %s and no healthy peer reachable to "
"catch up; truncation boundary durable_floor_lsn=%" PRIu64
" < cluster committed high-water hwm=%" PRIu64 ". Staying ALIVE and "
"PERMANENTLY non-promotable (refusing to serve divergent state) and "
"RETRYING catch-up indefinitely — will self-heal when a healthy peer "
"returns; operator attention may be needed "
"(ha-commit-durability Req 2.4)",
why, in->durable_floor_lsn, in->cluster_hwm_lsn);
/* Loud, operator-visible alert. The two triggers reaching this terminal
* are NOT logged with the same clause (round-2 fix, raft-apply-
* decoupling task 5.2 remediation):
*
* - apply_divergent: fences UNCONDITIONALLY (persist_recover_decision),
* evaluated at boot-replay time BEFORE peer/HWM context exists (see
* the call site in persist_recover()). `healthy_peer_reachable` and
* `cluster_hwm_lsn` are inert placeholders there, not real facts —
* asserting "no healthy peer reachable" or a durable_floor_lsn < hwm
* comparison for this trigger would report unevaluated-or-false
* information to the operator (round-1 review: the placeholders can
* make the comparison numerically impossible, e.g. "2 < 0"). This
* branch names the real reason (apply-correctness divergence) only.
* - structural (truncated / behind-HWM): reaches this terminal only
* via the pre-existing "no healthy peer reachable" branch in
* persist_recover_decision, so the peer-reachability and
* durable_floor_lsn/hwm comparison ARE guaranteed true here — that
* clause is unchanged from before this task.
*/
if (in->apply_divergent) {
ERROR("persist_recover: UNRECOVERABLE — post-replay apply-correctness "
"verification is DIVERGENT (materialized state does not match "
"the committed log tip, raft-apply-decoupling Req 8.3); a local "
"apply defect cannot be proven fixed by peer reachability or "
"catch-up alone. Staying ALIVE and PERMANENTLY non-promotable "
"(refusing to serve divergent state) and RETRYING catch-up "
"indefinitely — operator attention is needed to diagnose the "
"apply path (raft-apply-decoupling Req 8.3/8.4)");
} else {
const char* why = (in->verdict == RECOVER_TRUNCATED)
? "recovery-truncated log (structural corruption)"
: "log behind the cluster committed high-water mark";
ERROR("persist_recover: UNRECOVERABLE — %s and no healthy peer reachable to "
"catch up; truncation boundary durable_floor_lsn=%" PRIu64
" < cluster committed high-water hwm=%" PRIu64 ". Staying ALIVE and "
"PERMANENTLY non-promotable (refusing to serve divergent state) and "
"RETRYING catch-up indefinitely — will self-heal when a healthy peer "
"returns; operator attention may be needed "
"(ha-commit-durability Req 2.4)",
why, in->durable_floor_lsn, in->cluster_hwm_lsn);
}
/* Set the sticky non-promotable flag so the promotion gate refuses
* leadership as belt-and-suspenders until catch-up clears it (Req 2.4/2.6).