fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU
Bundles the fixes validated this session across ha.c/persist.c/worker.c: - raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression fixed via a segment-read cache. - bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness() closed with a recovery-complete gate that closes at process-lifecycle start, not function entry. - bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm instead of silently falling through the default job-state case. - bug-1426: trim_wal_segments() also caps its trim floor at the local apply pump's own progress, not just remote standby-acked positions. - bug-1427: persist_verify_apply_correctness() extended with a worker-count check mirroring the existing job-count check (count_jobmgr_entries and count_workermgr_entries now share one count_manager_entries(var) helper). - bug-1428: persist_register_pending_apply's critical section widened to close the actual race instead of patching a symptom. - bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now applies mark_worker_unavail()'s mutations via a shared apply_worker_dead_mutation() helper instead of duplicating them. - bug-1474 (angle-a): on_standby_connected's stale-duplicate detection fixed for promoted vote endpoints via a shared votes_idx_for_peer() index-scale converter. Each fix independently verified via TDD (RED before fix, GREEN after). Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live single-node HA deployment validated Task 7.4 and bug-1416 across 5 restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT fences). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
3afe3d5549
commit
45400c3a28
18 changed files with 1499 additions and 195 deletions
|
|
@ -151,10 +151,18 @@ typedef enum {
|
|||
* the loaded snapshot's checkpoint_lsn (entries at/below it are skipped
|
||||
* before ever reaching dispatch — see replay_wal), so each one names a
|
||||
* job that must now exist in the live job table that did NOT exist
|
||||
* before this replay pass began. */
|
||||
* before this replay pass began.
|
||||
* expected_new_worker_count — bug-1427: the worker analog of
|
||||
* expected_new_job_count. The number of WAL_WORKER_CTRL/
|
||||
* HOOK_DETAIL_WORKER_JOIN entries replay dispatched in this pass that
|
||||
* genuinely created a new worker (worker_from_capnp() returns 1 for
|
||||
* that case, 0 for an UPDATE/DEAD applied to an already-existing
|
||||
* worker or -EEXIST for a duplicate JOIN — only the 1 case is tallied,
|
||||
* mirroring WAL_JOB_SUBMIT's apply_rc == 1 convention). */
|
||||
typedef struct {
|
||||
uint64_t expected_last_applied_lsn;
|
||||
uint64_t expected_new_job_count;
|
||||
uint64_t expected_new_worker_count;
|
||||
} apply_expectation_t;
|
||||
|
||||
/* ============================================================
|
||||
|
|
@ -915,24 +923,28 @@ bool persist_recovery_complete(persist_t* p);
|
|||
void persist_mark_recovery_pending(persist_t* p);
|
||||
|
||||
/* persist_verify_apply_correctness (raft-apply-decoupling task 5.1, Req
|
||||
* 8.1/8.2). Pure comparison — mirrors persist_recover_decision's separation
|
||||
* of decision from action: no logging, no fence side effects, no mutation of
|
||||
* `p` beyond the read of last_applied_lsn. Compares `exp` (derived by
|
||||
* scanning the committed log during replay — see apply_expectation_t) against
|
||||
* the actual materialized state: p's replay-derived apply cursor, and — when
|
||||
* p->cluster is non-NULL — the live job table's growth since
|
||||
* `jobmgr_count_before_replay` (the job count captured just before replay
|
||||
* began). Returns APPLY_CLEAN on a match. Returns APPLY_DIVERGENT when either
|
||||
* the apply cursor or the live job count disagrees with `exp`.
|
||||
* 8.1/8.2; worker half added bug-1427). Pure comparison — mirrors
|
||||
* persist_recover_decision's separation of decision from action: no
|
||||
* logging, no fence side effects, no mutation of `p` beyond the read of
|
||||
* last_applied_lsn. Compares `exp` (derived by scanning the committed log
|
||||
* during replay — see apply_expectation_t) against the actual materialized
|
||||
* state: p's replay-derived apply cursor, and — when p->cluster is
|
||||
* non-NULL — the live job table's growth since `jobmgr_count_before_replay`
|
||||
* and the live worker table's growth since `workermgr_count_before_replay`
|
||||
* (both counts captured just before replay began). Returns APPLY_CLEAN on a
|
||||
* match. Returns APPLY_DIVERGENT when the apply cursor, the live job count,
|
||||
* or the live worker count disagrees with `exp`.
|
||||
*
|
||||
* Safe to call with p->cluster == NULL (a bare unit-test persist_t, or any
|
||||
* caller with nothing to dispatch into): the job-count half of the check is
|
||||
* skipped — only the position check applies — mirroring
|
||||
* apply_committed_entry()'s own "nothing to dispatch into" convention. Safe
|
||||
* on NULL p or NULL exp (returns APPLY_CLEAN — nothing to verify). */
|
||||
* caller with nothing to dispatch into): the job-count and worker-count
|
||||
* halves of the check are both skipped — only the position check applies —
|
||||
* mirroring apply_committed_entry()'s own "nothing to dispatch into"
|
||||
* convention. Safe on NULL p or NULL exp (returns APPLY_CLEAN — nothing to
|
||||
* verify). */
|
||||
apply_verdict_t persist_verify_apply_correctness(
|
||||
persist_t* p, const apply_expectation_t* exp,
|
||||
uint64_t jobmgr_count_before_replay);
|
||||
uint64_t jobmgr_count_before_replay,
|
||||
uint64_t workermgr_count_before_replay);
|
||||
|
||||
/* Accessor for the sticky verdict computed by the last persist_recover()
|
||||
* (task 5.1). APPLY_CLEAN before recovery has run (fresh store default,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue