fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU

Bundles the fixes validated this session across ha.c/persist.c/worker.c:

- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
  fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
  closed with a recovery-complete gate that closes at process-lifecycle
  start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
  instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
  apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
  check mirroring the existing job-count check (count_jobmgr_entries and
  count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
  close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
  applies mark_worker_unavail()'s mutations via a shared
  apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
  fixed for promoted vote endpoints via a shared votes_idx_for_peer()
  index-scale converter.

Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

src/controller/worker.c

@@ -676,20 +676,32 @@ int get_resval(cluster_t* c, char* resname, char* host, xce_val_t* val)
     return 1;
 }
 
-int mark_worker_unavail(cluster_t* c, worker_t* w)
+/* Local mutation body of "mark a worker dead" -- status flag, dead_since
+ * (only if not already set), and the derived hostattrs/resvals reset. No
+ * hook side effects and no locking of its own: callers hold w's lock (or,
+ * for mark_worker_unavail, are about to fire the surrounding hooks) and
+ * decide where dead_since comes from (wall clock live, or the WAL payload's
+ * recorded timestamp on replay -- bug-1472-remainder). */
+static void apply_worker_dead_mutation(cluster_t* c, worker_t* w,
+                                       time_t dead_since)
 {
-    xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_WORKER, HOOK_BEFORE_EVENT,
-                  HOOK_DETAIL_WORKER_DEAD, c, w);
-
     w->host.status |= XCE_WORKER_UNAVAIL;
     if (w->host.dead_since == 0)
-        w->host.dead_since = time(NULL);
+        w->host.dead_since = dead_since;
 
     if (w->host.attrs != NULL) {
         reset_hostattrs_unknown(w->host.attrs);
         add_hostattrs_resvals(c, w);
         set_host_slots(c, w, 0);
     }
+}
+
+int mark_worker_unavail(cluster_t* c, worker_t* w)
+{
+    xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_WORKER, HOOK_BEFORE_EVENT,
+                  HOOK_DETAIL_WORKER_DEAD, c, w);
+
+    apply_worker_dead_mutation(c, w, time(NULL));
 
     xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_WORKER, HOOK_AFTER_EVENT,
                   HOOK_DETAIL_WORKER_DEAD, c, w);
@@ -855,15 +867,26 @@ int do_hostcontrol_op(cluster_t* c, endpoint_t* ep, xce_request_t* request)
  *     return 0 so the caller (persist.c's WAL_WORKER_CTRL apply case)
  *     falls through to fire the HOOK_DETAIL_WORKER_UPDATE notification
  *     instead of silently discarding the change.
- *   - HOOK_DETAIL_WORKER_DEAD: left as -EEXIST (unchanged) -- see
- *     buglog.json bug-1472 for why DEAD's replication needs its own,
- *     separately-tracked fix: its mutation isn't confined to
- *     XCE_WORKER_ADMIN_MASK (XCE_WORKER_UNAVAIL + dead_since + hostattrs
- *     reset), and the live per-host resource-value array
- *     (xce_worker_t.resvals, a zarray_t) that mark_worker_unavail()
- *     zeroes via set_host_slots() never round-trips through the Worker
- *     capnp schema at all (only resvals_to_sched does) -- a naive
- *     status-replace here would be an incomplete fix. */
+ *   - HOOK_DETAIL_WORKER_DEAD (bug-1472-remainder): replays
+ *     mark_worker_unavail()'s own local mutations on the existing worker --
+ *     XCE_WORKER_UNAVAIL, dead_since (from the decoded payload's deadSince
+ *     field, the PRIMARY's own death timestamp, not this member's clock),
+ *     hostattrs reset to "unknown" sentinels, and the derived resvals
+ *     re-computed from that reset state (add_hostattrs_resvals + slots=0
+ *     via set_host_slots). This does NOT need a new wire field despite
+ *     the original investigation's design-decision framing: the live
+ *     xce_worker_t.resvals zarray_t the earlier analysis worried about
+ *     never needs to travel on the wire at all, because
+ *     reset-to-"unknown"-and-zero is a fixed, data-independent operation
+ *     -- add_hostattrs_resvals/set_host_slots derive their values FROM
+ *     w->host.attrs's (already-reset) fields, not from anything the
+ *     primary computed. capnp_encode_worker (persist_worker_hook_fun)
+ *     already serializes the FULL xce_worker_t.host on every WAL_WORKER_
+ *     CTRL entry including DEAD, and mark_worker_unavail() calls
+ *     reset_hostattrs_unknown() BEFORE firing the hook that triggers that
+ *     encode -- so the Worker schema's existing `attrs` field was never
+ *     actually the gap; this function's own DEAD branch simply never
+ *     existed to apply anything. */
 int worker_from_capnp(cluster_t* c, const void* capnp_payload, size_t len)
 {
     if (!capnp_payload || len <= sizeof(uint32_t))
@@ -906,10 +929,29 @@ int worker_from_capnp(cluster_t* c, const void* capnp_payload, size_t len)
             return 0;
         }
 
+        if (hook_detail == HOOK_DETAIL_WORKER_DEAD) {
+            lock(existing, RTTI_LOCK_WRITE);
+            apply_worker_dead_mutation(c, existing, host.dead_since);
+            unlock(existing);
+
+            free_Worker(&host);
+            return 0;
+        }
+
         return -EEXIST;
     }
 
-    return restore_worker(c, host);
+    /* bug-1427: distinguish "genuinely created a new worker" from a
+     * negative errno, discarding restore_worker()'s raw idx (its own
+     * return value, w->idx, is NOT a creation flag -- it can legitimately
+     * be 0 for the first worker, which would collide with the
+     * update-applied return above). Neither of this function's two
+     * callers (persist.c) inspects the exact non-negative value, only
+     * sign and -EEXIST, so this is a safe, backward-compatible
+     * repurposing -- see apply_committed_entry()'s WAL_WORKER_CTRL case
+     * and persist_verify_apply_correctness()'s new worker-count check. */
+    int rc = restore_worker(c, host);
+    return (rc < 0) ? rc : 1;
 }
 
 /* worker_should_retain lives in persist.c —