fix(controller): checkpoints no longer punch a hole in the replicated log (bug-1870)

persist_checkpoint() and persist_checkpoint_marker_triggered() wrote a
local-only WAL_CHECKPOINT marker that consumed an LSN of the replicated log
(current_lsn++) but was never replicated. A checkpoint taken by one member
only -- the primary's periodic max-wal-records checkpoint, or the one before
a snapshot send -- left a hole no standby had: the leader's next entry
carried prev_log_index = the marker's LSN, logOk failed on every standby
forever, commits timed out and writes stalled ("could not submit 110 jobs"
in the --force-snapshot soak; reproduced on the pre-bug-1867 baseline). Even
the config-marker checkpoint, taken on every member, stamped the marker with
each member's own current_lsn at apply time, which need not agree.

Nothing reads the marker except to skip it, so checkpoints simply stop
writing it. The recovery, streamer and bug-1475 log_len skip paths stay, for
WALs written by older binaries; persist_test_write_legacy_checkpoint_marker()
(XCE_TESTING only) reproduces such a WAL for the tests that cover them.

- new test_bug1870_leader_only_checkpoint_leaves_no_log_hole (RED before):
  a leader-only checkpoint takes no LSN and a standby that never
  checkpointed catches up past it via the real streamer/receive path.
- test_checkpoint_writes_marker_to_wal inverted to ..._writes_no_marker_...
- bug-1475 x2 and the apply-correctness trailing-marker test now build
  their legacy marker explicitly.

Controller suite 1234/0. HA soaks: --force-snapshot PASS (113 jobs, real
snapshot install -- also live-verifies bug-1867), --config-upgrade-wal-trim
PASS, --sequential 6 rounds PASS (SIMVERIFY 7 files, 0 mismatches).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

src/controller/persist.c

@@ -2275,9 +2275,24 @@ uint64_t persist_test_durable_lsn_on_disk(persist_t* p)
 /* Checkpoint internals                                                 */
 /* ------------------------------------------------------------------ */
 
-/* Write a WAL_CHECKPOINT marker to the current segment.
- * Never calls repl_broadcast() — checkpoint markers are local only. */
-static int persist_write_checkpoint_marker(persist_t* p)
+/* bug-1870: checkpoints no longer write a WAL_CHECKPOINT marker. The marker
+ * was local-only (never replicated) yet consumed an LSN of the replicated log
+ * (current_lsn++), so a checkpoint taken by one member only -- the periodic
+ * max-wal-records one, or the one before a snapshot send -- left a hole the
+ * other members never had: the leader's next entry carried prev_log_index =
+ * the marker's LSN, logOk failed on every standby and writes stalled. Even a
+ * config-marker-triggered checkpoint, taken on every member, stamped the
+ * marker with each member's own current_lsn at apply time, which need not
+ * agree. Nothing reads the marker back except to skip it (recovery,
+ * catch-up streaming), so it is simply not written any more; those skip
+ * paths stay for markers already present in existing WALs. */
+
+#ifdef XCE_TESTING
+/* bug-1870: the WAL_CHECKPOINT marker writer checkpoints used before the fix,
+ * kept for tests only, to produce the on-disk state of a WAL written by an
+ * older binary (a local marker that consumed an LSN) -- recovery and catch-up
+ * streaming must still handle such WALs. Never called in production. */
+int persist_test_write_legacy_checkpoint_marker(persist_t* p)
 {
     pthread_mutex_lock(&p->lock);
 
@@ -2320,6 +2335,7 @@ static int persist_write_checkpoint_marker(persist_t* p)
     pthread_mutex_unlock(&p->lock);
     return 0;
 }
+#endif /* XCE_TESTING */
 
 /* Entity-bearing checkpoint payload (ha-entity-snapshots task 2.2/3.2, Req
  * 1.1-1.4). Physical layout appended after the existing legacy header
@@ -3581,10 +3597,8 @@ int persist_checkpoint(persist_t* p)
 
     int rc = 0;
 
-    /* 1. Write WAL_CHECKPOINT marker (local only — never replicated) */
-    rc = persist_write_checkpoint_marker(p);
-    if (rc != 0)
-        goto done;
+    /* 1. (bug-1870) no WAL_CHECKPOINT marker any more -- see the bug-1870
+     *    note above the checkpoint payload layout comment. */
 
     /* 2. Record checkpoint_lsn = the APPLIED cursor (last_applied_lsn), not
      *    the physical WAL append tip (current_lsn). write_snapshot() below
@@ -3666,9 +3680,8 @@ static int persist_checkpoint_marker_triggered(persist_t* p,
     if (atomic_exchange(&p->checkpoint_in_progress, 1) != 0)
         return -EBUSY;
 
-    int rc = persist_write_checkpoint_marker(p);
-    if (rc == 0)
-        rc = persist_checkpoint_write_locked(p, marker_lsn);
+    /* bug-1870: no WAL_CHECKPOINT marker (see persist_checkpoint). */
+    int rc = persist_checkpoint_write_locked(p, marker_lsn);
 
     if (rc == 0) {
         trim_wal_segments(p);