fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU

Bundles the fixes validated this session across ha.c/persist.c/worker.c:

- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
  fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
  closed with a recovery-complete gate that closes at process-lifecycle
  start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
  instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
  apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
  check mirroring the existing job-count check (count_jobmgr_entries and
  count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
  close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
  applies mark_worker_unavail()'s mutations via a shared
  apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
  fixed for promoted vote endpoints via a shared votes_idx_for_peer()
  index-scale converter.

Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-16 10:20:02 +00:00
parent 3afe3d5549
commit 45400c3a28
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
18 changed files with 1499 additions and 195 deletions

View file

@ -9,6 +9,21 @@ XCE_BIN ?= /tmp/xce_static
SRV_URL ?= http://localhost:8080
GOFLAGS := CGO_ENABLED=0
# bug-1558: test-backend's CGO_CFLAGS/CGO_LDFLAGS, self-derived via
# pkg-config so `make test-backend` works standalone instead of only inside
# meson's custom_target env (apiserver/meson.build is the source of truth
# this mirrors -- librtti/c-capnproto/openssl/tinylog resolve via
# pkg-config there; math/rt are cc.find_library() results with no
# pkg-config package, added directly as -lm/-lrt to match).
# BUILD_DIR defaults to the tree tools/xce-build.sh produces (profile asan);
# override for a non-ASAN build, e.g.
# make test-backend BUILD_DIR=$(CURDIR)/../build.dbg FRONTEND_CGO_SANITIZE=
BUILD_DIR ?= $(CURDIR)/../xce.build
FRONTEND_CGO_PKGS := librtti c-capnproto openssl tinylog
FRONTEND_CGO_SANITIZE ?= -fsanitize=address -g
FRONTEND_CGO_CFLAGS := $(FRONTEND_CGO_SANITIZE) $(shell pkg-config --cflags $(FRONTEND_CGO_PKGS)) -I$(CURDIR)/../include -I$(BUILD_DIR)/lib
FRONTEND_CGO_LDFLAGS := $(FRONTEND_CGO_SANITIZE) $(shell pkg-config --libs $(FRONTEND_CGO_PKGS)) -lm -lrt -L$(BUILD_DIR)/lib -lxce
.PHONY: build-cli build-server test test-backend test-integration test-all
build-cli:
@ -25,12 +40,18 @@ test:
@echo "=== Unit tests: xcecli ==="
cd xcecli && $(GOFLAGS) go test ./... -count=1
## Run backend CGO conversion tests.
## Requires CGO_ENABLED=1 and CGO_CFLAGS/CGO_LDFLAGS pointing at the build tree.
## Run from within the nix develop shell after a full meson build.
## Run backend CGO conversion tests. Standalone-runnable: CGO_CFLAGS/
## CGO_LDFLAGS are self-derived above via pkg-config (bug-1558) -- no manual
## env setup needed. Still requires the nix develop shell (for pkg-config's
## .pc files) and a prior `xce-build.sh lib` (or any component that builds
## libxce.a) so BUILD_DIR/lib/libxce.a and BUILD_DIR/lib's generated headers
## exist.
test-backend:
@echo "=== Backend CGO tests: apiserver/backend ==="
cd apiserver && CGO_ENABLED=1 go test ./backend/... -count=1
cd apiserver && CGO_ENABLED=1 \
CGO_CFLAGS="$(FRONTEND_CGO_CFLAGS)" \
CGO_LDFLAGS="$(FRONTEND_CGO_LDFLAGS)" \
go test ./backend/... -count=1
## Run regression suite against a live server (requires XCE_BIN and SRV_URL)
test-integration: build-cli