---
id: "bug-1428"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "controller"
severity: "security"
timestamp: "2026-07-09T21:15:00.000Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/controller/persist.c (persist_register_pending_apply)"]
tags: ["ha", "persistence", "apply-pump", "toctou", "unconfirmed", "raft-apply-decoupling-followup"]
related: ["bug-1415"]
legacy: true
---

# bug-1428: Narrow TOCTOU residual in persist_register_pending_apply() (task 6.2, part of bug-1415's fix)…

## Description
Narrow TOCTOU residual in persist_register_pending_apply() (task 6.2, part of bug-1415's fix) — not reliably exercised by any current test.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
Flagged during raft-apply-decoupling task 6.2 as a non-blocking fast-follow while closing bug-1415. Specific race window not fully documented in tasks.md's own notes; needs a fresh look at the function to characterize exactly what window remains and whether it's reachable in production timing.

## Fix
FIXED (characterized then TDD RED->GREEN, per the entry's own 3-step plan). Characterized the exact window: persist_register_pending_apply()'s self-heal check (an UNLOCKED read of last_applied_lsn) ran BEFORE acquiring pending_applies_lock, separate from the locked insert that followed. Window: this thread reads last_applied_lsn (sees 'not yet applied') -> BEFORE it locks and inserts, the apply pump's thread advances last_applied_lsn past this lsn AND completes its own persist_take_pending_apply(lsn) call (finding nothing, since this thread hasn't inserted yet) -> this thread then inserts a node the pump will never revisit -- permanently orphaned, identical failure mode to bug-1415 (a hang, on_applied never fires) via a different, narrower trigger window. Production reachability: yes -- persist_register_pending_apply() is called from a different thread/vtask than the apply pump's walk (commit.c's continuation dispatch vs persist_apply_pump_run, per the function's own pre-existing doc comment), with no synchronization between them before this fix beyond the already-locked insert. Fix: moved the self-heal check inside pending_applies_lock, so the whole check-then-(insert-or-self-heal) sequence is one atomic critical section against persist_take_pending_apply's own locked scan-and-remove -- either this thread's check now runs before the pump's take (sees 'not yet applied', inserts, and the pump's later take, now forced to wait on the same lock, finds the freshly-inserted node) or after (sees 'already applied', self-heals instead of inserting, and the pump's take correctly finds nothing) -- never the interleaved gap that orphaned it. Regression test (test_apply_pump.c's test_register_pending_apply_race_against_concurrent_apply_self_heals): deterministic reproduction via controlled state (a main thread holding pending_applies_lock forces the register call to observe last_applied_lsn only after the lock's own update, removing dependence on real-time race timing) -- RED under the pre-fix code (on_applied never fires, node orphaned on pending_applies), GREEN under the fix (self-heals correctly, no orphan). Full controller_unit_test suite: 1098 tests, 2 pre-existing unrelated failures (bug-1561), no regressions.

[Original entry]
NOT FIXED — explicitly deferred as a fast-follow, not a blocker for task 6.2 or the spec's closure. Needs: (1) characterize the exact TOCTOU window, (2) determine production reachability, (3) add a regression test that exercises it before fixing.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1415