fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU

Bundles the fixes validated this session across ha.c/persist.c/worker.c:

- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
  fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
  closed with a recovery-complete gate that closes at process-lifecycle
  start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
  instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
  apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
  check mirroring the existing job-count check (count_jobmgr_entries and
  count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
  close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
  applies mark_worker_unavail()'s mutations via a shared
  apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
  fixed for promoted vote endpoints via a shared votes_idx_for_peer()
  index-scale converter.

Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

src/controller/testsuite/test_commit_barrier_reply.c

@@ -2136,17 +2136,22 @@ static void test_clr_preason_receipt_reply_fact_retained(void)
 
 /* ── do_update_op rusage path fixture ────────────────────────────────── */
 
-/* KNOWN PRE-EXISTING MISMATCH (discovered while pinning, reported in the
- * task evidence — NOT fixed here, production is out of boundary):
- * update_job_rusages fires the AFTER_EVENT HOOK_DETAIL_JOB_RUSAGE_UPDATE
- * hook with (cluster_t*, xce_job_rusages_t*) [job.c:1015-1016], but
- * persist_job_hook_fun's default arm reads the second vararg as a job_t*
- * [persist.c:1354-1359] and dereferences j->job.state at offset 56 —
- * exactly sizeof(xce_job_rusages_t) — an out-of-bounds read on an
- * exactly-sized allocation (the WAL_JOB_RUSAGE payload's state field is
- * garbage). The padded box below keeps THIS pin's read in-bounds under
- * ASAN without masking the report; the jobid field (offset 0 in both
- * types) lines up by accident. */
+/* bug-451, FIXED: update_job_rusages fires the AFTER_EVENT
+ * HOOK_DETAIL_JOB_RUSAGE_UPDATE hook with (cluster_t*, xce_job_rusages_t*)
+ * — a different va_list shape than every other job hook detail — because
+ * sched_job_rusage_update_fun/apiserver_job_rusage_update_fun (this hook's
+ * other two listeners) genuinely need the full rusages payload
+ * (pids/pgids/usages), not just a job_t*. persist_job_hook_fun previously
+ * had no dedicated arm for this detail and fell through to the default
+ * arm, which misread the xce_job_rusages_t* as a job_t* and dereferenced
+ * j->job.state at offset 56 — exactly sizeof(xce_job_rusages_t) — an
+ * out-of-bounds read on an exactly-sized allocation. Now fixed with a
+ * dedicated arm that reads the real xce_job_rusages_t* and looks the job
+ * up by jobid via the cluster_t* the hook already carries. The padded box
+ * below predates the fix (kept so this box's own read stays in-bounds
+ * under ASAN regardless); the jobid field (offset 0 in both types) lined
+ * up by accident even under the old bug — see the new assertion below for
+ * proof the state field (which did NOT line up) is now correct too. */
 static union {
     xce_job_rusages_t r;
     unsigned char     pad[sizeof(xce_job_rusages_t) + 64];
@@ -2198,12 +2203,35 @@ static void cbrp_arrange_update_rusage(uint64_t jobid)
  * WAL_JOB_UPDATE, HOOK_DETAIL_JOB_RUSAGE_UPDATE → WAL_JOB_RUSAGE, both
  * RETAIN per wal_classify) stay pending until ACK-covered. Umbrella
  * Reply-Ordering Contract: receipt-only ACK for fact-shaped delivery. */
+/* Forward-declared: defined further below, alongside its _ex sibling and
+ * every other caller (this test is the only one that needs it this early
+ * in the file). */
+static int cbrp_count_jobstate_entries(persist_t *p, uint32_t wal_type,
+                                       uint64_t jobid, uint32_t *state_out);
+
 static void test_update_op_rusage_receipt_reply_facts_retained(void)
 {
+    uint32_t rusage_state = 0xdeadbeef;
+    int      rusage_count;
+
     cbrp_setup(3000);
-    cbrp_arrange_jobctrl(72, 1); /* job 72 exists; jobctrl req unused */
+    cbrp_arrange_jobctrl(72, 1); /* job 72 exists, state = JOB_STAT_PEND */
     cbrp_arrange_update_rusage(72);
     cbrp_drive_receipt_fact(do_update_op, &g_update_req, 2);
+
+    /* bug-451 regression pin: the WAL_JOB_RUSAGE fact's jobid AND state
+     * must both be correct — under the pre-fix bug, jobid happened to
+     * line up by accident (offset 0 in both misread types) but state
+     * (offset 56, past xce_job_rusages_t's end) read back as 0 from the
+     * zeroed g_rusages_box padding, not the job's real JOB_STAT_PEND. */
+    rusage_count = cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_RUSAGE,
+                                               72, &rusage_state);
+    TEST_ASSERT_EQUAL_INT_MESSAGE(1, rusage_count,
+        "WAL_JOB_RUSAGE fact for job 72 must be found by jobid");
+    TEST_ASSERT_EQUAL_UINT32_MESSAGE((uint32_t)JOB_STAT_PEND, rusage_state,
+        "bug-451: WAL_JOB_RUSAGE's state must be the job's real state, not "
+        "a type-confused OOB read past xce_job_rusages_t's end");
+
     g_stub_ep_worker = NULL;
     cbrp_teardown();
     printf("test_update_op_rusage_receipt_reply_facts_retained: PASS\n");