fix(raft-apply-decoupling): prevent apply-divergence fence from being live-cleared (task 5.3)

Add explicit test coverage for the DIVERGENT fence's stay-alive and
restart-only-clear guarantees (task 5.3), which surfaced a critical,
pre-existing safety gap: the sticky recovery_unrecoverable flag is
trigger-agnostic, and two pre-existing live call sites in ha.c
(promotion_fence_timeout_fn's periodic timer, and
raft_cb_become_leader's promotion-eligibility gate) clear it based
purely on structural conditions, never checking the apply verdict.

A node whose local apply dispatch diverges from the committed log,
but whose WAL bytes are otherwise structurally healthy, would have
its fence silently released by the live timer, then be eligible for
promotion to primary on a later election -- serving genuinely
divergent state.

Fix: gate both clear-call-sites on persist_apply_verdict(&c->persist)
!= APPLY_DIVERGENT in addition to their existing structural checks.
An apply-divergence-triggered fence can now only clear via a fresh
process restart's clean persist_recover() replay.

New tests: test_promotion_fence_apply_divergent_survives_structural_catchup
(test_commit_durability.c) covers the timer site; and
test_promotion_refused_when_apply_divergent_even_if_eligible
(test_convergence_verification.c) covers the promotion-gate site,
structurally isolated so it cannot pass for the wrong reason.

Closes out task-group 5 (restart apply-correctness verification).

src/controller/testsuite/test_commit_durability.c

@@ -2315,6 +2315,85 @@ static void test_promotion_fence_cancel_stops_realert(void)
     printf("test_promotion_fence_cancel_stops_realert: PASS\n");
 }
 
+/* ── raft-apply-decoupling task 5.3 (round-2 remediation, Req 8.4) ───────────
+ * CROSS-CONTAMINATION REGRESSION: an APPLY_DIVERGENT-triggered fence must
+ * NEVER be live-cleared by this purely-structural re-check timer, even when
+ * the structural conditions it actually inspects (truncated / durable_floor
+ * vs HWM) become healthy.  Round-1 review found BOTH ha.c live-clear call
+ * sites (this timer, ha.c ~867, and the promotion-eligibility gate in
+ * raft_cb_become_leader, ha.c ~1115) blind to WHY the sticky flag was set —
+ * they only ever inspected structural getters.  A node whose local apply
+ * dispatch is defective (materialized state diverges from the committed log
+ * — task 5.1's APPLY_DIVERGENT verdict) can have a perfectly healthy WAL
+ * (not truncated, durable_floor >= HWM) by construction, since the two
+ * properties are orthogonal.  Before the fix, this test drives exactly that
+ * cross-contamination and asserts the fence does NOT clear.
+ *
+ * This mirrors test_promotion_fence_released_on_catchup's harness and
+ * catch-up simulation EXACTLY, differing only in also marking the fence as
+ * apply_divergent-triggered up front — proving the fix discriminates on the
+ * apply verdict, not merely on the structural fields. */
+static void test_promotion_fence_apply_divergent_survives_structural_catchup(void)
+{
+    char tmp[256];
+    strcpy(tmp, "/tmp/xce_fence_apply_div_XXXXXX");
+    TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
+
+    cluster_t* c = dur_make_fence_cluster(tmp, /*truncated=*/true,
+                                          /*durable_floor=*/4);
+    TEST_ASSERT_NOT_NULL(c);
+
+    /* Mark the fence as APPLY_DIVERGENT-triggered, exactly as persist_recover()
+     * does at boot (task 5.1/5.2) — the WAL-structural fields above
+     * (truncated/durable_floor) were already set independently above, proving
+     * the two conditions are orthogonal by construction. */
+    c->persist.apply_verdict = APPLY_DIVERGENT;
+    atomic_store_explicit(&c->persist.recovery_unrecoverable, true,
+                          memory_order_release);
+
+    ha_set_promotion_fence_window_ms(120); /* generous so we can mutate first */
+
+    extern void ha_arm_promotion_fence_timer_for_test(cluster_t*, bool, uint64_t,
+                                                      uint64_t);
+    ha_arm_promotion_fence_timer_for_test(c, /*truncated=*/true,
+                                          /*durable_floor=*/4, /*hwm=*/10);
+    TEST_ASSERT_EQUAL_INT(
+        1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire));
+
+    /* Simulate the EXACT SAME structural catch-up as
+     * test_promotion_fence_released_on_catchup: log now intact and durable
+     * past the HWM.  The apply_verdict / apply-divergent fence trigger is
+     * UNTOUCHED by this — it can only ever be recomputed by a fresh
+     * persist_recover() boot-time replay (task 5.1), never by a live
+     * structural mutation like this one. */
+    c->persist.recovery_verdict = RECOVER_CLEAN;
+    atomic_store_explicit(&c->persist.durable_lsn, 10, memory_order_release);
+
+    /* Wait long enough for the window to fire. */
+    usleep(300 * 1000);
+
+    /* THE REGRESSION ASSERTION (Req 8.4): even though the structural
+     * conditions the timer inspects are now fully healthy, an
+     * APPLY_DIVERGENT-triggered fence must NOT be released — a local apply
+     * defect cannot be proven fixed by more WAL bytes arriving.  Before the
+     * fix this assertion FAILS (the timer clears the flag purely on the
+     * structural check, oblivious to apply_verdict). */
+    TEST_ASSERT_TRUE_MESSAGE(
+        persist_recovery_unrecoverable(&c->persist),
+        "5.3 round-2: an APPLY_DIVERGENT-triggered fence must survive a "
+        "purely-structural catch-up re-check — only a fresh, clean "
+        "persist_recover() restart may clear it (Req 8.4)");
+    TEST_ASSERT_EQUAL_INT_MESSAGE(
+        1, atomic_load_explicit(&c->ha.in_catch_up, memory_order_acquire),
+        "5.3 round-2: the member must remain fenced (non-promotable) while "
+        "apply_divergent, regardless of structural catch-up");
+
+    ha_set_promotion_fence_window_ms(0);
+    dur_destroy_fence_cluster(c);
+    printf("test_promotion_fence_apply_divergent_survives_structural_catchup: "
+           "PASS\n");
+}
+
 #endif /* XCE_TESTING */
 
 /* ------------------------------------------------------------------ */
@@ -2375,6 +2454,10 @@ void test_commit_durability(void)
     RUN_TEST(test_promotion_fence_unrecoverable_stays_alive_loud);
     RUN_TEST(test_promotion_fence_released_on_catchup);
     RUN_TEST(test_promotion_fence_cancel_stops_realert);
+    /* task 5.3 round-2 remediation (Req 8.4): cross-contamination regression —
+     * an APPLY_DIVERGENT-triggered fence must not clear on structural catch-up
+     * alone. */
+    RUN_TEST(test_promotion_fence_apply_divergent_survives_structural_catchup);
 #else
     printf("test_commit_durability: skipped (no XCE_TESTING)\n");
 #endif