fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU

Bundles the fixes validated this session across ha.c/persist.c/worker.c:

- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
  fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
  closed with a recovery-complete gate that closes at process-lifecycle
  start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
  instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
  apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
  check mirroring the existing job-count check (count_jobmgr_entries and
  count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
  close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
  applies mark_worker_unavail()'s mutations via a shared
  apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
  fixed for promoted vote endpoints via a shared votes_idx_for_peer()
  index-scale converter.

Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-16 10:20:02 +00:00
parent 3afe3d5549
commit 45400c3a28
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
18 changed files with 1499 additions and 195 deletions

View file

@ -11,9 +11,9 @@ resources:
- name: mem
type: numeric
interval: 60
consumable: false
consumable: true
order: decrease
description: "available memory in MB (built-in dynamic load index)"
description: "available memory in bytes (built-in dynamic load index; CLI display converts to a human-readable unit)"
# built-in: worker load indices registered by
# worker/src/resource_sync.c init_builtin_load_indices()
@ -21,22 +21,71 @@ resources:
type: numeric
interval: 60
consumable: false
order: decrease
order: increase
description: "1-minute load average (built-in dynamic load index)"
- name: r15m
type: numeric
interval: 60
consumable: false
order: increase
description: "15-minute load average (built-in dynamic load index)"
- name: r15s
type: numeric
interval: 60
consumable: false
order: decrease
order: increase
description: "15-second load average (built-in dynamic load index)"
- name: tmp
- name: ut
type: numeric
interval: 60
consumable: false
order: increase
description: "cpu utilization"
- name: pg
type: numeric
interval: 60
consumable: false
order: increase
description: "page ratio"
- name: ls
type: numeric
interval: 60
consumable: false
order: increase
description: "login count"
- name: it
type: numeric
interval: 60
consumable: false
order: increase
description: "idle time"
- name: swp
type: numeric
interval: 60
consumable: true
order: decrease
description: "available tmp disk space in MB (built-in dynamic load index)"
description: "swap size"
- name: io
type: numeric
interval: 60
consumable: false
order: increase
description: "io rate"
- name: tmp
type: numeric
interval: 60
consumable: true
order: decrease
description: "available tmp disk space in bytes (built-in dynamic load index; CLI display converts to a human-readable unit)"
# built-in host attributes, registered per-worker by
# controller/worker.c add_hostattrs_resvals()
@ -62,11 +111,11 @@ resources:
- name: maxswap
type: numeric
description: "max swap in MB (built-in host attribute)"
description: "max swap in bytes (built-in host attribute; CLI display converts to a human-readable unit)"
- name: maxtmp
type: numeric
description: "max tmp disk space in MB (built-in host attribute)"
description: "max tmp disk space in bytes (built-in host attribute; CLI display converts to a human-readable unit)"
- name: ndisks
type: numeric