fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU

Bundles the fixes validated this session across ha.c/persist.c/worker.c:

- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
  fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
  closed with a recovery-complete gate that closes at process-lifecycle
  start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
  instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
  apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
  check mirroring the existing job-count check (count_jobmgr_entries and
  count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
  close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
  applies mark_worker_unavail()'s mutations via a shared
  apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
  fixed for promoted vote endpoints via a shared votes_idx_for_peer()
  index-scale converter.

Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-16 10:20:02 +00:00
parent 3afe3d5549
commit 45400c3a28
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
18 changed files with 1499 additions and 195 deletions

View file

@ -151,10 +151,18 @@ typedef enum {
* the loaded snapshot's checkpoint_lsn (entries at/below it are skipped
* before ever reaching dispatch — see replay_wal), so each one names a
* job that must now exist in the live job table that did NOT exist
* before this replay pass began. */
* before this replay pass began.
* expected_new_worker_count — bug-1427: the worker analog of
* expected_new_job_count. The number of WAL_WORKER_CTRL/
* HOOK_DETAIL_WORKER_JOIN entries replay dispatched in this pass that
* genuinely created a new worker (worker_from_capnp() returns 1 for
* that case, 0 for an UPDATE/DEAD applied to an already-existing
* worker or -EEXIST for a duplicate JOIN — only the 1 case is tallied,
* mirroring WAL_JOB_SUBMIT's apply_rc == 1 convention). */
typedef struct {
uint64_t expected_last_applied_lsn;
uint64_t expected_new_job_count;
uint64_t expected_new_worker_count;
} apply_expectation_t;
/* ============================================================
@ -915,24 +923,28 @@ bool persist_recovery_complete(persist_t* p);
void persist_mark_recovery_pending(persist_t* p);
/* persist_verify_apply_correctness (raft-apply-decoupling task 5.1, Req
* 8.1/8.2). Pure comparison — mirrors persist_recover_decision's separation
* of decision from action: no logging, no fence side effects, no mutation of
* `p` beyond the read of last_applied_lsn. Compares `exp` (derived by
* scanning the committed log during replay — see apply_expectation_t) against
* the actual materialized state: p's replay-derived apply cursor, and — when
* p->cluster is non-NULL — the live job table's growth since
* `jobmgr_count_before_replay` (the job count captured just before replay
* began). Returns APPLY_CLEAN on a match. Returns APPLY_DIVERGENT when either
* the apply cursor or the live job count disagrees with `exp`.
* 8.1/8.2; worker half added bug-1427). Pure comparison — mirrors
* persist_recover_decision's separation of decision from action: no
* logging, no fence side effects, no mutation of `p` beyond the read of
* last_applied_lsn. Compares `exp` (derived by scanning the committed log
* during replay — see apply_expectation_t) against the actual materialized
* state: p's replay-derived apply cursor, and — when p->cluster is
* non-NULL — the live job table's growth since `jobmgr_count_before_replay`
* and the live worker table's growth since `workermgr_count_before_replay`
* (both counts captured just before replay began). Returns APPLY_CLEAN on a
* match. Returns APPLY_DIVERGENT when the apply cursor, the live job count,
* or the live worker count disagrees with `exp`.
*
* Safe to call with p->cluster == NULL (a bare unit-test persist_t, or any
* caller with nothing to dispatch into): the job-count half of the check is
* skipped — only the position check applies — mirroring
* apply_committed_entry()'s own "nothing to dispatch into" convention. Safe
* on NULL p or NULL exp (returns APPLY_CLEAN — nothing to verify). */
* caller with nothing to dispatch into): the job-count and worker-count
* halves of the check are both skipped — only the position check applies —
* mirroring apply_committed_entry()'s own "nothing to dispatch into"
* convention. Safe on NULL p or NULL exp (returns APPLY_CLEAN — nothing to
* verify). */
apply_verdict_t persist_verify_apply_correctness(
persist_t* p, const apply_expectation_t* exp,
uint64_t jobmgr_count_before_replay);
uint64_t jobmgr_count_before_replay,
uint64_t workermgr_count_before_replay);
/* Accessor for the sticky verdict computed by the last persist_recover()
* (task 5.1). APPLY_CLEAN before recovery has run (fresh store default,