myxce/.kiro/specs/raft-apply-decoupling/tasks.md

Implementation Plan

• 1. Commit engine event-driven resolve (keystone prerequisite)

• 1.1 (P) Add per-LSN continuation registration to the commit engine for the synchronous (WAIT) commit mode, replacing the blocking wait

  • Register a continuation (callback + context) against an appended LSN instead of parking the calling thread
  • Preserve the existing per-LSN bookkeeping and admission-cap discipline already used by the commit engine
  • Observable: a unit test can register a continuation for an LSN and the calling thread returns immediately, before the LSN commits
  • Requirements: 1.3, 1.4, 9.1, 9.2
  • Boundary: Commit Engine Resolve — commit.c per-LSN registry only

• 1.2 Build the generic resolve primitive and wire the majority-ack (committed) outcome

  • Implement commit_resolve(): invoke a registered continuation with a committed or failed outcome, exactly once, idempotent on re-resolve
  • On majority ack, invoke the continuation with a committed outcome
  • Dispatch every continuation invocation off the calling thread using the existing non-blocking task-submission primitive, never inline on the replication-ACK or iomgr path
  • Observable: forcing majority-ack in a test resolves the registered continuation exactly once with a committed outcome
  • Requirements: 1.3, 9.3, 9.4
  • Note: real timeout/demotion/truncation detection (as opposed to the generic resolve primitive) is 1.3/1.4 — see Implementation Notes

• 1.3 Arm a timeout for registered async continuations

  • commit_submit_async() no longer parks a thread that would otherwise notice its own pthread_cond_timedwait deadline expire — converting away from the blocking wait removes the ONLY mechanism that ever detected a commit timing out. Add an explicit timeout (e.g., a delaymgr-driven check, or a deadline recorded at registration and swept periodically) that resolves a still-pending registered continuation as FAILED with -ETIMEDOUT if majority is not reached within the caller-supplied timeout
  • Preserve the existing default timeout value and error code so callers observe identical failure timing/vocabulary to today's blocking path — this is a "no regression" requirement (11.1), not just a new capability
  • Observable: an async-registered continuation for an LSN that never reaches majority is resolved as FAILED with -ETIMEDOUT within the configured timeout window, with no thread blocked to detect it, verified by a test that lets real time (or a mocked clock) elapse past the deadline without ever calling commit_resolve() manually
  • Requirements: 1.3, 1.4, 9.3, 9.4
  • Boundary: Commit Engine Resolve — commit.c only

• 1.4 Wire demotion and log-truncation detection to resolve pending continuations as failed

  • When this node is demoted (loses leadership) or truncates its log tail (conflict reconcile), any still-pending registered continuation for an LSN affected by that transition must resolve as FAILED — otherwise a demoted/truncated node leaves requesters permanently unresolved
  • Find the existing demotion/role-transition point and the existing log-truncation point and add a call into the commit engine's resolve path for every pending LSN they invalidate
  • Observable: forcing a demotion or a truncation while a continuation is registered for an affected LSN resolves it as FAILED exactly once, with the existing error-code vocabulary
  • Requirements: 1.3, 1.4, 9.3, 9.4
  • Boundary: Commit Engine Resolve integration points — this task is a narrow, explicit exception to the "commit.c only" boundary used by 1.1-1.3: it may touch the demotion/role-transition site and the log-truncation site (wherever they currently live) in addition to commit.c, but must not touch job.c or any command-handler code
  • Depends: 1.3

• 1.5 Unit test the commit engine's resolve behavior under concurrent in-flight commits

  • Cover exactly-once resolution across majority ack, timeout, demotion, and truncation, exercised through the real detection paths built in 1.3/1.4 (not by calling commit_resolve() directly, which 1.2's own tests already cover)
  • Cover that a burst of concurrently pending commits does not exhaust the bounded request-handling capacity
  • Observable: a test suite run demonstrates bounded worker usage and exactly-once resolution under a concurrent-commit burst
  • Requirements: 9.1, 9.2, 9.3, 9.4
  • Depends: 1.4

• 2. Shared apply seam for committed entries

• 2.1 (P) Extract the guarded state-transition logic used by fact handling into a form usable outside the receipt-time call site

  • Keep the existing monotonic-rank checks (reject backward transitions, no-op duplicate terminal transitions) intact and unchanged in behavior
  • Decouple the transition logic from firing its own hook inline, so hook-firing can be relocated by a later task without touching this logic again
  • Observable: the extracted transition logic can be invoked directly by a test without going through the worker-report receipt path, and produces the same guard behavior as before extraction
  • Requirements: 3.3, 7.1, 7.2
  • Boundary: Apply Seam — persist.c / job.c apply-dispatch and replay call sites only, no dependency on the commit-engine change in task 1

• 2.2 Build the single dispatch function that applies one committed entry to in-memory state, covering every classified entry type

  • Route job submission through the existing idempotent, jobid-addressable construction primitive instead of a raw uncoordinated insert
  • Route job state transitions through the extracted guarded logic from 2.1
  • Route queue control and worker control entries through their existing restore primitives
  • Fire the entry type's post-mutation notification hook from inside this function, after the mutation completes
  • Observable: calling the dispatch function directly with a sample entry of each classified type produces the correct in-memory mutation and fires exactly one post-mutation notification
  • Requirements: 2.2, 3.1, 3.2, 4.1, 4.2, 5.2
  • Depends: 2.1

• 2.3 Point restart replay's per-entry application at the new dispatch function

  • Replace replay's direct calls to the individual restore primitives with calls to the dispatch function from 2.2
  • Observable: restart replay of a log containing every classified entry type produces identical materialized state to before this change, verified by an existing or updated replay test
  • Requirements: 3.1, 3.2
  • Depends: 2.2

• 2.4 Unit test idempotent and ordered re-application of the dispatch function

  • Cover applying the same committed entry twice is a no-op with respect to observable state
  • Cover that a duplicate or out-of-order fact entry does not regress a job's lifecycle state
  • Observable: a test applies a FINISH entry followed by a stale, previously-unapplied START entry for the same job and confirms the job remains in its terminal state
  • Requirements: 3.4, 7.1, 7.2, 7.3
  • Depends: 2.2

• 3. Command apply-after-commit ordering

• 3.1 (P) Stop constructing the in-memory job object before the commit barrier

  • Change job submission to carry forward only the data needed to build the WAL entry payload, without inserting anything into the in-memory job table until the (still-synchronous, at this point) commit call returns success
  • Remove the now-unreachable code path that previously ran the durability step from inside the notification hook for job submission
  • Observable: a job submission whose commit fails or times out leaves no job object in the in-memory job table; a successful submission still replies synchronously exactly as before this task
  • Requirements: 1.2, 2.1, 2.3
  • Boundary: Raft Step (Command) — job.c handle_jobsubmit and its call chain only, no dependency on task 1

• 3.2 Convert job submission to register its commit outcome through the new continuation mechanism instead of blocking

  • Replace the synchronous commit call from 3.1 with a registration against the task 1 continuation API
  • On a failed or timed-out outcome, reply failure directly from the continuation — no apply is needed on this path
  • On a committed outcome, hand the entry off to the ordered apply pump (task 6) rather than applying or replying from this continuation directly, so this entry's apply and reply happen in strict commit order relative to every other entry
  • Observable: the calling thread returns before the commit resolves; a forced failure/timeout outcome still replies failure immediately with no job ever appearing (the committed-outcome path has no independently observable behavior until task 6 completes it)
  • Requirements: 1.1, 1.4, 2.1, 2.3
  • Depends: 1.1, 1.2, 1.3, 1.4

• 3.3 Unit test the command failure path leaves no observable trace

  • Force repeated commit failures and timeouts for job submission and confirm no job object is ever created
  • Observable: the test suite demonstrates zero phantom job objects across repeated forced commit failures
  • Requirements: 2.1, 2.3
  • Depends: 3.2

• 4. Fact apply-after-commit split

• 4.1 (P) Change worker-fact receipt handling to validate and durably append only

  • Remove the state mutation and the post-mutation notification hook firing from the receipt-time handlers for job start, finish, fail, and update reports
  • Keep the existing pre-mutation validation hook firing at receipt time, unchanged
  • Observable: after a worker reports a job event, the job's in-memory state is unchanged immediately after the receipt handler returns
  • Requirements: 5.1
  • Depends: 2.1
  • Boundary: Fact Ingestion — job.c on_job_start/on_job_finish/on_job_fail/on_job_update receipt handlers only, disjoint from handle_jobsubmit

• 4.2 Unit test durable retention of facts submitted while majority is unreachable

  • Submit facts while majority is unreachable and confirm they are durably appended and that job state remains unchanged for the duration of the outage
  • Confirm no local-only apply code path exists by inspection of the receipt handler from 4.1 (no runtime check needed — the guarantee is structural)
  • Observable: the test demonstrates zero state mutation during a simulated quorum-loss window
  • Requirements: 6.1, 6.2
  • Depends: 4.1

• 5. Restart apply-correctness verification

• 5.1 (P) Compute a post-replay expectation from the committed log tip and compare it against materialized state

  • Derive a lightweight expectation (e.g., last-applied position and a count or checksum of materialized entities) from the recovered log
  • Compare the expectation against the actual state produced by replay through the shared apply dispatch function
  • Observable: a clean replay produces a CLEAN verdict alongside the existing structural (CRC) verdict
  • Requirements: 8.1, 8.2
  • Depends: 2.3
  • Boundary: Restart Replay & Verification — persist.c replay/recovery path only

• 5.2 Route a verification mismatch through the existing non-promotable fence mechanism

  • Reuse the existing loud-fence function and sticky fence flag rather than introducing a new terminal or fencing path
  • Observable: an injected materialization mismatch produces a DIVERGENT verdict, a loud operator-visible log entry, and a non-promotable fenced state
  • Requirements: 8.3, 8.4
  • Depends: 5.1

• 5.3 Unit test the DIVERGENT verdict path stays alive and keeps retrying

  • Force a DIVERGENT verdict and confirm the process does not exit or abort
  • Confirm the fence clears once a subsequent recovery reaches a consistent state
  • Observable: the test process remains running and retrying after a forced DIVERGENT verdict, never terminating
  • Requirements: 8.4
  • Depends: 5.2
  • Boundary: Restart Replay & Verification — retroactive exception (added 2026-07-07 validate-impl audit, mirroring task 1.4's precedent): review found the sticky recovery_unrecoverable flag has two pre-existing, trigger-agnostic live-clear call sites in ha.c (promotion_fence_timeout_fn, raft_cb_become_leader's promotion gate) that could silently release an apply-divergence-triggered fence. The fix is confined to those two existing call sites only — an apply_divergent guard ANDed into each — never job.c or any command-handler code.

• 6. Ordered apply pump integration

• 6.1 Build the strict-order walk that applies every newly committed entry

  • Walk entries from the last-applied position up to the current committed position, in ascending order, with no skipping or reordering
  • Invoke the shared apply dispatch function for every entry in that range, regardless of its commit mode
  • Observable: a burst of interleaved command and fact entries committing out of submission order still applies in commit order
  • Requirements: 3.3, 7.3
  • Depends: 1.2, 2.2

• 6.2 Fire a command's registered continuation immediately after its entry applies

  • After the walk in 6.1 applies an entry that has a registered commit continuation (from task 3.2), invoke that continuation now, after the mutation, not before
  • Leave fact entries with no further action beyond what the apply dispatch function itself performs
  • Observable: a successful job submission's client reply carries the job id only after the job is visible in the in-memory job table; the reply always occurs strictly after the mutation
  • Requirements: 1.1, 2.1, 2.2, 2.3
  • Depends: 6.1, 3.2

• 6.3 Integration test command and fact apply/notify timing through the full pump

  • Cover a successful job submission becoming visible and notified only after commit, exercised through the full pump (superseding the earlier synchronous-path check from task 3.1)
  • Cover that worker slot release, scheduler dispatch triggering, and client/apiserver notification for a fact do not occur until that fact's entry is committed
  • Observable: the test suite shows zero fact-reaction side effects during a delayed-commit window, and a submitted job's reply always follows its visibility in the job table
  • Confirm a consumer acting on the post-apply notification never needs to separately re-check a commit watermark, since the notification's occurrence already implies the entry is committed and applied
  • Requirements: 2.2, 4.3, 5.2, 5.3
  • Depends: 6.2, 4.1

• 6.4 Integration test quorum-loss-then-restore catch-up ordering through the full pump

  • Submit a mix of facts and commands while majority is unreachable, then restore majority and confirm every retained entry commits and applies in its original log order with no local-apply escape hatch exercised
  • Observable: the test demonstrates full, correctly-ordered catch-up after quorum returns, with zero out-of-order application
  • Requirements: 6.3, 7.3, 9.4
  • Depends: 6.1, 4.2

• 7. Conformance, regression, and existing-behavior preservation

• 7.1 Extend the deterministic conformance harness with apply-ordering assertions

  • Add an assertion that no entry's state mutation or reactions occur before that entry commits
  • Add an assertion that re-applying an already-committed entry is idempotent
  • Observable: the conformance harness run reports both new assertions passing alongside its existing safety checks
  • Requirements: 10.1, 10.3
  • Depends: 6.3, 6.4

• 7.2 Add a live-cluster regression scenario that kills the primary mid-fact-stream

  • Drive a continuous worker-fact stream, kill the primary partway through, and let a new primary take over
  • Assert no phantom slot-frees, no committed-state divergence, and no notification for a fact that never committed
  • Observable: the scenario completes with zero divergence and zero phantom reactions across repeated runs
  • Requirements: 10.2, 10.4
  • Depends: 6.3, 6.4

• 7.3 Re-run the full existing automated test suite and update only the tests tied to the old ordering

  • Run the existing unit, integration, and HA soak suites
  • Update only the specific tests that asserted the old apply-before-commit ordering to assert the new ordering instead
  • Observable: the full suite passes with zero committed-job loss and zero state divergence across the existing failover soak
  • Requirements: 11.1, 11.2, 11.3
  • Depends: 7.1, 7.2

• 7.4 Measure sustained fact-stream throughput against the pre-feature baseline

  • Record a throughput baseline on the pre-feature code, then drive the same sustained high-volume worker-fact stream against the completed feature
  • Confirm no per-fact synchronous commit barrier was introduced and throughput does not regress
  • Observable: a recorded throughput comparison shows no regression relative to the pre-feature baseline
  • Requirements: 11.4
  • Depends: 7.3

Implementation Notes

• 7.1: correctly identified src/lib/raft/raft_sim.c (pre-existing #ifdef XCE_TESTING TLA+-conformance simulator, driven by src/lib/testsuite/test_raft.c in the xcelib_test Unity binary — a different binary/library boundary than every other task in this spec, which use controller_unit_test) as the intended "deterministic conformance harness," confirmed verbatim against design.md's Testing Strategy line 442 ("Extend raft_sim.c's conformance harness with 'no reaction before commit' and 'idempotent re-apply' assertions layered on its existing apply-after-on_commit_advanced loop"). Added two fault-injectable checker functions (raft_sim_check_apply_after_commit, raft_sim_check_idempotent_reapply) plus raft_sim_replay_commit, gated by two new flags (RAFT_SIM_BREAK_APPLY_BEFORE_COMMIT, RAFT_SIM_BREAK_REAPPLY_GUARD) and a new apply_reaction_count[] field on raft_sim_node_t. Wired into sim_persist_entry/sim_conf_on_commit_advanced, both no-ops (byte-identical to prior behavior, confirmed by reviewer) when the flags are unset — zero regression risk to the 5 pre-existing scenario functions. RED phase independently reproduced by reviewer for both checks (Expected 39338 Was 0, Expected 2 Was 1); GREEN 287/0/0 confirmed twice (implementer + reviewer), ASan-clean. Scoping decision, independently upheld by reviewer: the two new checks run only via their own 2 dedicated tests, NOT wired into raft_sim_check_all/sim_scen_check (the umbrella the 5 pre-existing scenario functions use). Reviewer traced this to raft-sim-conformance spec's Requirement 3 Criterion 5 (fixed four-invariant, fixed-order, halt-on-first-failure contract) — an already-shipped contract owned by a different spec that 7.1 has no charter to modify. Also: these checks operate on the harness's synthetic applied[]/apply_reaction_count[] model, not real production apply_committed_entry()/persist_apply_pump_run() — real production coverage of this ordering already exists via tasks 6.1-6.4's test_apply_pump_integration.c. Confirmed legitimate scoping, not a corner-cut. Residual, non-blocking documentation nitpick: the applied[k-1] != 0 "has-been-applied" sentinel is not structurally guaranteed non-zero the way log-entry term 0 is (raft.tla guarantees terms start at 1; arbitrary client payloads have no such guarantee) — the header comment's "same zero-means-absent convention as log-entry terms" analogy slightly overclaims precedent (the pre-existing raft_sim_check_state_machine_safety bounds by commit_index directly, doesn't use a zero-sentinel at all). Currently harmless because the only two call sites use hardcoded non-zero literals (0x99AA, 0x77BB). Fast-follow flag for whoever later widens this harness to PRNG-drawn payload values: apply_reaction_count[] cannot substitute as the "ever-applied" indicator for check_apply_after_commit, since the BREAK_APPLY_BEFORE_COMMIT fault only touches applied[] — a real generalization needs a third parallel "ever-written" flag array, not a reuse of either existing array.
• 1.1: commit_submit_async() reuses the exact wait_admit_locked/wait_release_locked/inflight_waits admission-cap discipline as COMMIT_MODE_WAIT — task 1.2's commit_resolve() MUST call wait_release_locked() when resolving an async-registered entry, or the shared admission cap leaks. Continuation registry is calloc(n_vcpus, ...), sized so n_continuations >= wait_cap(ce) always holds; lsn == 0 is the free-slot sentinel.
• 1.1 review: bare controller_unit_test (no --run filter) segfaults deterministically in this environment (SSL_CTX_free/iomgr_fini crash, test_ha_vep_teardown.c) — reproduced identically on the pre-task base commit, confirmed pre-existing and unrelated. Logged as bug-1406. Also test_commit_advance_retry_clears_push_before_commit_fence (test_config_repl.c) fails independently of this feature, also reproduced on base — logged as bug-1407, flagged because it's adjacent to commit_advance_locked, which task 1.2 modifies. Always validate with a filtered --run <group> invocation or the tightly-coupled commit-engine cluster, not the bare unfiltered binary, until 1406/1407 are separately fixed.
• 1.2: implemented commit_resolve() + wired it into all 6 commit.c-internal paths that can advance committed_lsn/writable_lsn (the COMMITTED/majority-ack outcome). Chose VTASK_LIVE_BLOCK for dispatch (never admission-gated, unlike VTASK_REQ_BLOCK which can fast-reject — unacceptable for "exactly-once, never unresolved"). Reviewer caught a real task-graph gap, not a defect in 1.2 itself: converting COMMIT_MODE_WAIT away from a blocking wait also removes the only mechanism that ever detected a commit timing out (the caller's own pthread_cond_timedwait deadline) — nothing yet arms an equivalent timeout for async-registered continuations, and nothing yet wires demotion/log-truncation detection to call commit_resolve(FAILED, ...). Without this, a pending async command during quorum loss would hang forever instead of failing — worse than today's behavior. Split out as new tasks 1.3 (timeout arming) and 1.4 (demotion/truncation wiring), inserted before the renumbered 1.5 (was 1.3, concurrency test — now exercises the real detection paths from 1.3/1.4 instead of calling commit_resolve() directly). Task 3.2 now also depends on 1.3/1.4, since its own "failure/timeout outcome replies" observable is meaningless without them.
• 1.3: added a dedicated per-commit-engine sweep thread (not the shared delaymgr — would violate boundary + delaymgr's 200ms poll granularity vs. DEFAULT_SYNC_TIMEOUT_MS=500ms would add timing jitter). Race rule: majority-satisfied always wins over an already-expired deadline when both are true under the same lock hold. Round 1 REJECTED: commit_sweep_once() computed the next-deadline snapshot under ce->lock, unlocked to dispatch, then slept on the stale snapshot — a commit_submit_async() registration/broadcast landing during that unlocked dispatch window was lost (no thread parked on the condvar to receive it), so the sweep thread could sleep on an absent/late deadline indefinitely (worst case: unbounded delay in a low-traffic quorum-loss scenario, violating the "identical failure timing" no-regression bar). Fixed by holding ce->lock continuously across scan→wait decisions (releasing only for the unlocked dispatch step) and always re-scanning under the lock immediately before ever calling cond_wait/cond_timedwait — the standard recheck-then-wait idiom. Round 2 reviewer independently reproduced the round-1 bug shape to confirm the fix actually closes it, not just narrows it. Precedent for 1.4: any future code that needs to interrupt the sweep thread's wait (e.g., a demotion/truncation handler resolving a pending LSN out-of-band) must go through the same lock+broadcast discipline commit_submit_async() uses, or it risks the identical lost-wakeup class of bug.
• 1.4: added commit_resolve_pending_above(ce, lsn_floor, rc) (snapshot-then-commit_resolve()-per-entry, no parallel bookkeeping). Truncation: a single new on_truncate/truncate_cb_ctx callback slot on persist_t (same inversion-of-control shape as the existing on_durable_advance hook) fires from persist_truncate_wal_above after releasing p->lock, only when a truncation actually removed entries — this ONE wrapper covers all three real truncation call sites (raft_cb_become_leader's post-election truncate, the library's on_truncate_from conflict-reconcile, ha_demotion_convergence_fn's post-demotion truncate), verified by the reviewer reading all three. Demotion: raft_cb_become_follower calls commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY) (lsn_floor=0 → fails everything — a demoted node can never authoritatively confirm anything pending; verified against the CP posture). Known residual, non-blocking: a fourth internal truncation path (persist.c inside persist_wal_write_replicated's tip-conflict-repair branch, STANDBY replicated-apply only) calls truncate_wal_above_locked() directly, bypassing the new wrapper — safe today only because demotion resolves ALL pending continuations before role flips to STANDBY (so nothing can still be registered by the time that path runs), but this is an implicit ordering invariant, not an asserted one. Flag for whoever next touches the STANDBY replicated-apply path: add a defensive comment/assertion there, or route it through the wrapper too, before relying on it further. Also: bug-1407 was believed closed at this point (70/70 across 3 runs) — see 1.5's note below, this was a false reading, not a real fix.
• 7.4: closes out task-group 7 and the spec. Measured sustained jfinish fact-stream throughput at the pre-feature baseline commit vs. the feature-complete commit on a single-node HA cluster, distinguishing two signals since facts are fire-and-forget: commit throughput (durable-append + majority-registration only) and apply/observable throughput (commit + the ordered apply pump's mutation, the new hop this spec introduces). Commit throughput: unregressed (~3111/s baseline vs ~3033/s feature, ~2.5% delta, within noise) — Req 11.4's "no per-fact synchronous commit barrier" claim independently confirmed both by this measurement and by code inspection (commit_submit(COMMIT_MODE_RETAIN, ...) in commit.c returns immediately, byte-identical mechanism to pre-feature code). Apply/observable throughput: initially found regressed 56% (2831/s baseline vs 1237/s feature) with clearly superlinear scaling (2x fact volume → 5.2x elapsed time) — a genuine, in-scope defect this task's own benchmark surfaced, not a false alarm, correctly reported BLOCKED rather than silently accepted or patched outside boundary. Root-caused (fresh debug investigation) to apply_pump_read_entry (persist.c, introduced this spec's task 6.1): the function has zero memory across calls — every single LSN lookup re-runs a full opendir/readdir/segment-sort and rescans its segment linearly from byte offset 0 — and its only caller, persist_apply_pump_run, calls it once per LSN in strict ascending order with no batching, so applying N contiguous entries does O(N²) total work (entry k's read rescans ~k prior headers). Fixed with a minimal segment-read cache on persist_t (apply_pump_cache_fd/_seg_lsn/_off/_next_lsn, guarded entirely by the existing apply_pump_lock): a fast path preads the next expected LSN directly at a remembered offset on an already-open fd (O(1) hit, same CRC verification as the slow path), falling back unchanged to the original full-rediscovery logic on ANY miss (gap, first call, post-truncation, post-recovery) — correctness never depends on the cache, only performance does. Invalidated at all four lifecycle points a stale cursor could otherwise leak or misread from: persist_init (starts invalid), persist_fini (closes if open), persist_recover (resets before any pump kick can trust a pre-recovery cursor), and persist_truncate_wal_above (closes inside the existing real-truncation gate, since a truncation can ftruncate/unlink segment bytes a stale cached fd/offset might otherwise still successfully, and wrongly, read). Round-1 review REJECTED on two findings, both remediated in round 2: (1) persist_recover's cache invalidation initially mutated the new fields without holding apply_pump_lock, contradicting the fields' own documented invariant (fixed: wrapped in lock/unlock matching persist_truncate_wal_above's existing pattern; independently re-verified deadlock-free by tracing every call site of both persist_recover and apply_pump_lock itself — the lock is touched in exactly 3 places, all private to persist.c, no re-entrant path exists); (2) the initial fix's "no worse than before" characterization of the cache/trim interaction understated a real, bounded risk — the OLD code failed loudly and immediately (-ENOENT) if a needed segment was already trimmed, while the NEW cache can (POSIX unlink-keeps-fd-alive) silently advance last_applied_lsn past already-deleted data if the pump is lagging when trim_wal_segments fires, a reachable scenario since apply-lag-behind-commit is this feature's entire premise — bounded by the pre-existing APPLY_DIVERGENT boot fence (tasks 5.1/5.2) catching it at next restart (deferred loud fencing, not silent corruption, but a real behavior change, not literally risk-free) — corrected via an accurate doc comment, no redesign of trim_wal_segments attempted or needed. Verified end-to-end (implementer + two independent review rounds): post-fix apply-throughput scales sublinearly (1.46x elapsed for 2x volume, vs. the pre-fix 7.4x reproduced independently on a stashed pre-fix binary in the same environment) and exceeds both the regressed 1237/s and the original 2831/s pre-feature baseline; test_apply_pump 5/5, test_apply_pump_integration 4/4, test_apply_correctness_verification 11/11, test_persist_recovery 7/7, test_persist_truncate 8/8, plus a broader sweep (test_demotion_convergence, test_ha_core_safety, test_commit_resolve, test_repl_apply, test_commit_barrier_reply) all pass with zero new failures, independently re-run by both reviewers, not just trusted from the implementer's report. Two items explicitly flagged as out-of-scope, not fixed here: (1) an intermittent (~2 of 9 attempts) client hang in the synchronous jobsub (COMMIT_MODE_WAIT) path shortly after a fresh single-node election, reproduced identically on both pre-fix and post-fix binaries (confirmed unrelated to this fix), resembling but not confirmed identical to the closed bug-1416 boot-race class — worked around via retry for benchmarking only, needs separate root-cause investigation; (2) trim_wal_segments's retention floor has no awareness of the local apply pump's own lag (a distinct, pre-existing architectural gap, same family as an already-tracked standby-cursor variant) — out of scope for this targeted perf fix, flagged for a future spec/task rather than redesigned here.
• 7.3: closes out task-group 7's regression gate. Full re-run confirmed exactly 2 of 33 tests in test_commit_barrier_reply.c failing — test_jobsubmit_returns_before_commit_and_registers_pending_apply and test_jobsubmit_array_mixed_outcome_no_phantom_job_for_failed_elements — both asserting task 3.2's own explicitly-documented interim contract ("no independently observable behavior until task 6 completes it") that tasks 6.1/6.2 (already committed, ordered apply pump) closed by construction. Updated both to assert the new deterministic settled state (poll for reply, then assert success/job-exists), mirroring the idiom already used by the file's other, already-updated passing tests — no production code touched. One judgment call flagged and independently confirmed by reviewer: in the mixed-outcome array test, a committed sibling now applies (job visible) independently of a differently-failing sibling, while the batch's single client-visible reply still reflects the failure — this is the documented, intended interim behavior (collapsing array submission into one atomic WAL record is explicitly out-of-scope, owned by the separate job-array-single-wal-record spec), not a regression or scope violation. Verified (implementer + independent reviewer, both fresh runs): test_commit_barrier_reply 33/33 pass (reviewer ran 3x to rule out flakiness); full 79-group/926-test suite 0 failures including both bare-binary runs (reviewer ran the full unfiltered controller_unit_test binary directly and it passed clean, even through the two tests flagged elsewhere in this file's notes as environment-flaky, bug-1406/1407 — no regression signal, strengthens confidence); /xce-test-ha --sequential 5 HA soak PASS twice independently (implementer's own run + reviewer's fresh re-run in a separate scratch dir), zero job loss, zero divergence, all 3 nodes led both times.
• 7.2: added --factstream mode to run-ha-soak.sh (kill-mid-fact-stream against a live 3-node cluster, real jstart/jfinish/jfail wire traffic via xce_test's worker REPL context, extended in this task since libxce already had the calls but the REPL didn't expose them) and a phantom-fact checker that reads waldump's decoded state_raw bits (extended in this task to decode every worker-fact WAL type, not just WAL_JOB_SUBMIT) rather than inferring "did START apply" from the pretty-printed job state string — job_state_str()'s RUN-checked-before-EXIT priority ordering hides co-set EXIT/PERR bits on a fail-without-finish job, a genuine phantom-check false-positive caught and fixed here (pre-existing, minor, display-only quirk in job_state_str itself is out of boundary, not touched). Round 2 found a real production bug this task's own scenario was needed to surface: bug-1416, a boot-time TOCTOU in persist_verify_apply_correctness() — a returning node's persist_recover() replay+verify pass can race the live apply pump and the standby live-replication apply path (both write the same last_applied_lsn/job-table state persist_verify_apply_correctness re-reads live), producing a false APPLY_DIVERGENT that permanently, incorrectly fences a healthy node (task 5.3's sticky fence never self-clears on a false positive) — reproduced 10/10 across ASAN/release builds, 8-20 job counts, only on the RESTART/catch-up path (a clean 3-node cold boot with no kill never hit it). Fixed with a new _Atomic bool persist_t.recovery_complete gate (persist.h/persist.c): persist_apply_pump_kick() no-ops while closed; ha_apply_single_entry/ha_apply_term_entry_via_library (ha.c) skip only the in-memory apply_wal_entry() call while closed — the durability write, replication ACK, and applied_lsn advance are never gated (mirrors the bug-1043 durability/materialization-decoupling precedent, confirmed in .wolf/cerebrum.md since bug-1043 predates the current buglog retention window). persist_recover() closes the gate at entry, reopens it plus issues one deliberate catch-up kick right before returning, unconditionally of verdict — a genuine divergence still fences exactly as before, this flag only prevents the comparison itself from racing. A second, harder-to-find gap surfaced only via live verification, not code review: a node stuck retrying its own losing/timing-out election as a candidate can receive and durably apply live-replicated entries through the ungated (persist_init default=true) path for tens of seconds before cluster_init_phase2() ever calls persist_recover() for the first time — reproducing the identical false-DIVERGENT symptom through an entry point persist_recover()'s own bracket can't see. Closed with persist_mark_recovery_pending(&c.persist) in myapp_run() (myapp.c), called immediately after persist_init()/persist_start() succeed and before ha_start_election_async()/cluster_run() can process any network traffic. persist_init()'s default stays true so none of this spec's ~20 direct-persist_t-construction unit fixtures are affected — persist_mark_recovery_pending is called only from myapp.c. Round-2 review independently verified, not just re-reported: ran test_apply_correctness_verification itself (11/11, including all 3 real-divergence fence tests, confirming detection wasn't accidentally suppressed), confirmed the ACK/durability paths are genuinely ungated by reading the ha.c call sites directly, confirmed the myapp.c fix is load-bearing (not noise) by reverting only that hunk and reproducing the original failure live (2 of 3 nodes fenced), confirmed bug-1043 precedent against .wolf/cerebrum.md (not in .wolf/buglog.json — predates current retention). Verified: live 3-node --factstream soak (2 rounds, kill+restart both) PASS, zero APPLY_DIVERGENT across 5 persist_recover() calls; broader controller_unit_test sweep (23 groups) zero new failures (only the 2 pre-existing task-6.1/6.2-documented test_commit_barrier_reply.c failures reserved for task 7.3 remain). Do-not-repeat lesson: when gating a live path behind a "recovery in progress" flag, the flag's own bracket around the recovery function itself is insufficient if OTHER code paths (election retry, live replication) can mutate the same state before that function is ever first called — the gate must close at the earliest point in the real process lifecycle traffic could possibly begin, not just around the specific function whose output it protects.
• 6.4: closes out task-group 6. Approved first round, zero production changes — group ends 2-for-4 on real bugs found (6.1, 6.2), 2-for-4 clean confirmations (6.3, 6.4). Extended test_apply_pump_integration.c (task 6.3) with 2 tests covering the MIXED facts+commands scenario 6.3 explicitly deferred here. Test 3: genuinely interleaved COMMAND/FACT/COMMAND/FACT/COMMAND sequence through a real repl_remove_standby outage, polls for zero commit/apply/reply throughout, restores quorum, and asserts STRICT original-log-order catch-up via a real append-only sequence spy (not a set-membership check) — an adversarial probe (corrupting the expected slot-0 jobid) was independently reproduced by the reviewer to confirm the assertion genuinely discriminates reordering, not just typos. Test 4 — the highest-value test in this task: wires a REAL 4-vcpu vtaskmgr_t (mirroring task 6.2's bug-1415 regression harness) to drive 4 commands + 2 facts through ONE covering ACK, independently traced by the reviewer to confirm this genuinely creates 5 vtasks (4 continuation dispatches + 1 apply-pump walk) competing for 4 real worker threads — true N-way concurrent racing of task 6.2's self-healing fix, not just one isolated forced pair like 6.2's own test. The "exactly once" check is per-jobid (not an aggregate total), so a skewed double-fire-on-one/hang-on-another distribution would be caught even though the aggregate count would look fine. Adversarial probe (corrupting expected reply count) independently reproduced. Two honestly-scoped, explicitly non-blocking caveats, not gaps requiring fixes: (1) task 6.2's own reviewer flagged a narrow, nanosecond-scale TOCTOU residual (unlocked self-heal check vs. locked list-link in persist_register_pending_apply()) as a non-blocking fast-follow — Test 4 does NOT deliberately stress this specific window (no injected synchronization primitive forces it, unlike 6.2's own barrier-based test); 50 clean runs across two independent builds are consistent with "didn't hit it by chance," not proof it's closed — stated explicitly here so it isn't assumed covered by this task's "several concurrent races" framing. (2) Test 4 doesn't directly re-assert per-command reply-after-apply ordering the way Test 1 (task 6.3) does via reply_already_enqueued — this property holds BY CONSTRUCTION (both the walk's take-call and the self-heal's lsn <= last_applied_lsn check only ever fire after last_applied_lsn already reflects that entry, verified in the actual code) and is independently covered by task 6.2's own dedicated test, so this is non-blocking, not a real gap. Task-group 6 final summary: the ordered apply pump is now live, activated (6.1), correctly hands off command replies (6.2), proven correct for both isolated (6.3) and mixed/bursty (6.4) command+fact scenarios under both normal and quorum-loss conditions — with 2 real, independently-verified, production-severity bugs found and fixed along the way (6.1's job-state corruption from stale receipt-time computation; 6.2's client-hang race from unordered async dispatch), both closed with dedicated regression coverage, the first ever in this spec to wire a real multi-worker vtaskmgr_t. Optional fast-follow noted, not blocking: a dedicated stress test that deliberately widens the persist_register_pending_apply() check-to-lock window (e.g. via a test-only injection point) to reliably exercise 6.2's disclosed residual TOCTOU gap, since neither 6.2's nor 6.4's tests reliably reach it today.
• 6.3: pure integration-test close-out, approved first round, zero production changes — breaks task-group 6's 2-for-2 real-bug streak (6.1, 6.2 each found genuine production-severity bugs; 6.3 confirmed the pump's existing behavior is correct rather than finding a new one). New test_apply_pump_integration.c (2 tests) exercises the FULL live pump (real commit engine, real apply dispatch, real hook subscribers — pool=NULL like every other fixture, does not overclaim bug-1415/6.2's dispatch-race coverage, which is that test's own job) for both a job submission and a fact. Key design decision, independently verified true by the reviewer, not just asserted: apply_committed_entry() fires ALL job-state facts via the generic HOOK_DETAIL_JOB_UPDATE only (task 2.2's documented deviation, confirmed UNCHANGED by 6.1's _pad fix — that fix changed which state-computation core runs, not which hook detail fires) — so sched_job_start_fun/_exit_fun/_fail_fun/apiserver.c's equivalents are structurally unreachable for any live-applied fact, meaning ONE reaction spy wired to HOOK_DETAIL_JOB_NEW+HOOK_DETAIL_JOB_UPDATE legitimately covers all three named reactions (worker slot release, scheduler dispatch trigger, client/apiserver notify) since they're all downstream of the identical single firing. Strong independent confirmation this is load-bearing, not incidental: the type-specific handlers are unconditionally va_arg-typed for job_pidinfo_t*/xce_jobfinish_t*/job_failinfo_t* — types apply_committed_entry() never passes (always job_t*) — so if a type-specific hook ever DID fire from the live pump, it would be an immediate type-confusion crash, not just a coverage gap. Repo-wide re-audit (not just job.c/persist.c) confirms exactly 2 live call sites ever fire these hook details: apply_committed_entry() (guarded) and job_ctrl's mark_job_update (legitimate SIGCONT/SIGSTOP) — no orphaned third site. Both tests prove REAL ordering, not coincidental timing: reply-after-visibility is sampled synchronously inside the reaction-spy callback (not two separately-timed checks); Req 4.3's "no separate watermark re-check needed" is proven by construction (the walk only ever invokes apply_committed_entry() for LSNs already <= committed_lsn, so committed_lsn_at_fire >= entry_lsn holds structurally, not by scheduling luck) rather than merely asserted in prose. Delayed-commit and genuine quorum-loss windows are both polled repeatedly across real time (5×20ms), not checked once. RED phase independently reproduced by both implementer and reviewer via re-injecting the pre-4.1 eager-mutation bug class. Task-group 6 running tally: 3 tasks in, 2 real production-severity bugs found and fixed (6.1 data corruption, 6.2 client-hang race), 1 clean confirmation (6.3) — the pump's core mechanics and hook-routing design are now thoroughly adversarially tested from multiple independent angles. Remaining: 6.4 (quorum-loss catch-up ordering, mixing facts AND commands — 6.3's fact-only quorum-loss scenario is explicitly non-duplicative defense-in-depth per the reviewer, not a preemption of 6.4's charter).
• 6.2: closes the standing operational gap flagged since task 3.2 — successful job submissions finally get a client reply again. Wired persist_apply_pump_run() (task 6.1) to call persist_take_pending_apply(p, lsn, &out_fn, &out_ctx) immediately after each entry's apply_committed_entry() call, invoking the continuation when found — strictly after the mutation, per the task's named observable. Facts need no special-casing: persist_take_pending_apply() naturally returns 0 for them since the fact-receipt path (task 4.1) never calls persist_register_pending_apply(). Added jobsubmit_reply_success_once() (job.c, sibling of the pre-existing jobsubmit_reply_failure_once()) — jobsubmit_apply_complete() now calls it once every array element resolves, sending RequestReply_data_jobid to the client. Took 2 review rounds. Round 1 REJECTED — a second real, production-severity bug found this task-group, on top of task 6.1's data-corruption bug: self-disclosed as bug-1415, confirmed by the reviewer via direct code trace: commit_resolve_pending() (commit.c) enqueues the WAIT-mode continuation's dispatch and persist_apply_pump_kick()'s walk as TWO INDEPENDENT VTASK_LIVE_BLOCK vtasks with NO completion-order guarantee — vdispatcher_entry() binds each ready task to a different idle vcpu and does not wait for a prior dispatch to complete. Confirmed reachable on EVERY default production deployment (cluster_init_phase2() always wires a real vtaskmgr_t, default n_vcpus=8, never NULL) — and confirmed completely untested by every fixture in this entire spec, all of which use pool=NULL (which trivially serializes the two dispatches in enqueue order by falling back to synchronous inline execution, and so can never exercise the race). Consequence: if the apply-pump task wins, it applies the entry and advances last_applied_lsn past the LSN before the continuation task ever calls persist_register_pending_apply() — when that registration finally arrives for an already-passed LSN, nothing will ever come back to consume it. The job is correctly constructed and visible, but the client hangs forever, plus a leaked jobsubmit_async_ctx_t/endpoint ref — a coin-flip-or-worse failure mode on the feature's own primary success path, for every default multi-vcpu production controller. Round 2 fixed it with a self-healing check at the top of persist_register_pending_apply(): if (lsn <= persist_last_applied_lsn(p)) { on_applied(ctx, lsn); return; } — using the existing persist_last_applied_lsn() accessor, no signature change. Verified race-safe in both interleavings: non-racing case is a no-op (confirmed unchanged behavior for every existing test, all of which use pool=NULL and so never trigger the check); racing case fires the callback exactly once, immediately, on the registering thread — and can never double-fire because a self-healed registration is never linked onto pending_applies, so the walk's own persist_take_pending_apply() call for that LSN (which already ran and found nothing, which is precisely how we know the registration is late) has nothing left to find. First and only test in this entire spec to wire a REAL multi-vcpu vtaskmgr_t (n_vcpus=4, via commit_set_pool()) — every other fixture in the spec deliberately uses pool=NULL and structurally cannot reproduce this class of bug; the new test's poll-barrier ties directly to the real race predicate (persist_last_applied_lsn(p) < lsn), not a timing guess, and self-checks its own precondition so it fails loudly rather than silently degrading if the forced interleaving doesn't occur. RED→GREEN independently reproduced by both rounds' reviewers. One narrow, non-blocking residual finding for a fast-follow, not required to close bug-1415: the self-heal check (an unlocked atomic read) and the subsequent list-link (under pending_applies_lock) are not one atomic unit — a sufficiently long preemption between the check and the lock could theoretically still reproduce a vastly narrower version of the original race. Recommended fix: perform the self-heal check while already holding pending_applies_lock (the same lock persist_take_pending_apply() uses), making check-and-link atomic with respect to the walk's take. Not blocking since the reported, reproduced, production-relevant (dispatch-level) race is genuinely and completely closed. Two pre-existing tests in test_commit_barrier_reply.c still fail as EXPECTED (unchanged from task 6.1 — same root cause, task 7.3's job). Task-group-6 running tally: 2 of 2 tasks landed so far have each required a full REJECTED→fixed round for a genuine, independently-verified production-severity bug (6.1: job-state corruption from stale receipt-time fact computation; 6.2: a client-hang race from unordered async dispatch) — this task group is where deferred-apply's real concurrency implications are surfacing, exactly as expected for the task that finally activates the live path. Remaining: 6.3/6.4 (integration tests through the full pump) should be reviewed with the same level of scrutiny for further concurrency surprises.
• 6.1: the most consequential task in the entire spec — this is what finally makes apply_committed_entry() live (previously only restart replay, task 2.3, ever called it). Built persist_apply_pump_run() (walks [last_applied_lsn+1..committed_lsn] strictly ascending, calls apply_committed_entry() for every entry regardless of commit mode) and persist_apply_pump_kick() (off-thread trigger via vtask_enq_classified(VTASK_LIVE_BLOCK), inline fallback for pool-less fixtures, CAS-coalesced so at most one caller runs the walk at a time — the loser hands off asynchronously, by design, to whichever thread wins). Design deviation, verified correct and a strict superset: wired the trigger into commit_resolve_pending() (commit.c) rather than only ha.c's on_commit_advanced — task 1.2 already discovered commit_resolve_pending is the single choke point all 6 internal committed_lsn-advancing paths funnel through (verified: exactly 6 call sites, commit.c:353/558/1042/1060/1099/1116, all firing after ce->lock releases), while on_commit_advanced covers only 1 of them; wiring only there would silently starve the pump for single-node/self-quorum clusters. design.md should be updated to reflect this as the actual integration point. Fixed Hazard 1 (double-append, flagged since task 4.1): a naive detail==HOOK_DETAIL_JOB_UPDATE filter would have broken job_ctrl's legitimate live SIGCONT/SIGSTOP append (same hook detail, still-legitimate command path) — fixed instead with a _Thread_local bool t_apply_seam_notifying call-provenance guard set only around apply_committed_entry()'s own hook-firing calls; persist_job_hook_fun/persist_queue_hook_fun/persist_worker_hook_fun early-return when set. Also caught and fixed an UNDOCUMENTED instance of the identical hazard class in the queue/worker hook functions (neither had ANY mode-based early-return before this, unlike job's pre-existing dead WAL_MODE_WAIT branch) — would have double-appended+re-triggered a nested majority-commit wait for every live-applied WAL_QUEUE_CTRL/WAL_WORKER_CTRL entry. Took 2 review rounds. Round 1 REJECTED — the most severe finding of the entire session, a real data-corruption bug, not a doc/log issue: Hazard 2 (merge-staleness, also flagged since 4.1) was investigated and ESCALATED as requiring wire-format changes beyond scope, while still landing the pump that activates it. Round-1 review proved with field-level precision this was wrong to ship: job state DONE, correctly baked by a FINISH fact — followed shortly after (an entirely ordinary HPC lifecycle event, not adversarial) by a POST_EXEC-stage FAIL fact whose receipt-time snapshot is STILL stale (RUN, since FINISH hadn't applied yet, apply now being deferred) — bakes RUN|EXIT|PERR into its WAL entry. Both apply in correct LSN order (FINISH then FAIL), but FAIL's stale absolute overwrite wins over the now-correct DONE, because the terminal/non-terminal guard doesn't catch it (both values happen to carry a terminal-ish bit). Final result: a job's correct DONE completion silently replaced by RUN|EXIT|PERR — a Req 11.3 regression, and a real, operator-and-billing-visible corruption newly introduced by deferring apply (this feature's whole purpose), reachable under completely ordinary traffic. The reviewer also found a scoped fix contradicting the "needs wire-format enrichment" escalation: wal_jobstate_t's existing unused _pad field had room. Round 2 fixed it properly: _pad now carries a discriminator (WAL_JOB_FAIL_STAGE_FLAG/WAL_JOB_UPDATE_POSTDONE_MARK) plus the minimal data needed (FAIL's stage) to RECOMPUTE against the job's ACTUAL CURRENT state AT APPLY TIME via the real apply_job_fail_transition()/new apply_job_postdone_transition() guard cores — no more baking a stale absolute value at receipt time. WAL_JOB_START/WAL_JOB_FINISH are unchanged (confirmed safe on their own — deterministic, state-independent payloads that were never the problem, only the thing getting clobbered). Backward-compatible: _pad==0 (all pre-existing/legacy data) deterministically routes to the same absolute-overwrite behavior that was universal before this fix — no crash, no new corruption, correctly out of scope for further hardening since this feature has never been deployed. Bonus finding during round-2 review — a real test-flakiness hazard, root-caused and fixed on the spot, not just reported: a rare (initially estimated ~2%, actually verified ~7.5%) flake in the new regression test was root-caused to a genuine SECOND, independent kicker of the apply pump (commit_on_durable_advance, fired from the real fsync_worker background thread on every durable_lsn advance, entirely separate from repl_on_ack) that the test's own header comment didn't account for — this is by-design eventually-consistent behavior (task 6.2's continuation-firing will correctly rely on it), not a production bug, but the TEST needed a proper poll-for-applied-state helper instead of asserting immediately. Fixed (tap_wait_applied() polling helper added to 2 tests) and a SECOND, related flake found during the investigation in a third test, also fixed — 120/120 clean runs afterward. Also fixed a stale orphaned forward-declaration left over from the round-2 rename. Also fixed a genuine, unrelated bug found via regression sweep, logged as bug-1409: an early version of apply_pump_drain's retry loop didn't check persist_apply_pump_run's return code, causing an unbounded busy-loop (18.5M+ log lines) in test_demotion_convergence whenever committed_lsn advances without a corresponding real on-disk WAL entry (a legitimate scenario for commit-engine-focused unit fixtures) — fixed by returning immediately on a negative rc. Hand-off built for task 6.2: int persist_take_pending_apply(persist_t* p, uint64_t lsn, pending_apply_notify_fn* out_fn, void** out_ctx) — call AFTER apply_committed_entry() applies lsn; returns 1+populates once (atomically unlinking) when a registration exists (the command path, task 3.2), 0 otherwise (the common case, facts never register); a second call for the same LSN is guaranteed to return 0; safe to call unconditionally for every applied LSN. Task 6.2 should call this right after each apply inside/after the walk and invoke the returned continuation — deliberately NOT wired into the walk itself, per boundary. Two pre-existing tests in test_commit_barrier_reply.c now fail as EXPECTED (not a regression): their own assertion text literally says "no job may be constructed until task 6's apply pump exists" — now correctly invalidated by this task landing; explicitly task 7.3's job to update, per task 3.2's own Implementation Notes anticipating exactly this. Minor, non-blocking coverage gap for a fast-follow: no test explicitly replays a WAL_JOB_FAIL/WAL_JOB_UPDATE entry with a real _pad marker through the REPLAY path specifically (only through live apply) — low risk given apply_committed_entry() is the literal shared function both paths call, but untested for that specific combination.
• 5.3: closes task-group 5. Took 3 review rounds — the heaviest task in the entire series, and the only one that surfaced a genuine CRITICAL production safety bug (not a doc/log issue like 5.1/5.2's rounds). Round 1 (test-only) REJECTED: while writing tests for "the fence stays alive" and "the fence clears on a subsequent consistent recovery," review discovered the sticky recovery_unrecoverable flag is trigger-agnostic — it has ZERO provenance about WHY it was set. Task 5.2 correctly reused this flag for the new apply_divergent trigger, but TWO PRE-EXISTING, live, running-process call sites in ha.c already clear this SAME flag based purely on STRUCTURAL conditions (never checking the apply verdict): promotion_fence_timeout_fn (a self-re-arming periodic timer) and the promotion-eligibility gate in raft_cb_become_leader. Concrete, reachable attack path: a node's local apply dispatch has a defect (materialized state diverges from the committed log) while its WAL bytes are otherwise fine (structurally healthy — orthogonal to the apply defect by construction). The apply-divergent fence sets correctly at boot. But WHILE THE PROCESS KEEPS RUNNING, the periodic timer sees the structural conditions were always fine and silently calls persist_clear_recovery_unrecoverable() — releasing a fence that was never actually resolved. If this node then wins a later election, the promotion gate now sees unrecoverable==false and promotes it to primary — a genuinely divergent node serving authoritative state, a direct Req 8.4 / CLAUDE.md "never serve divergent state" violation. Round 2 fixed the code: added apply_divergent = persist_apply_verdict(&c->persist) == APPLY_DIVERGENT as an additional guard ANDed into BOTH clear-call-site conditions — promotion_fence_timeout_fn (line ~876) and raft_cb_become_leader's gate (line ~1130) — so an apply-divergence-triggered fence can now ONLY be cleared by a fresh process restart's clean persist_recover() (since apply_verdict has exactly two production write sites: persist_init()'s reset-to-APPLY_CLEAN and persist_recover()'s boot-time computation — no live path resets it). Verified via full trace + RED→GREEN for site 1. Round 2 REJECTED anyway: the fix was logically sound, but review PROVED site 2's guard had ZERO test coverage — by deleting only that guard and re-running the ENTIRE 298-test regression sweep across 13 groups, every single test still passed. An unverified defense-in-depth guard on promotion/election-safety code doesn't meet the bar. Round 3 (test-only) closed it: added a test that structurally isolates site 2 from site 1 — proven not to be exercising the already-covered sticky-flag path, since ha_promotion_gate_decision() never populates apply_divergent in its own inputs, so ONLY the round-2-added standalone guard at the call site can catch this scenario. RED→GREEN independently reproduced by two separate reviewers reverting the guard and confirming the exact predicted test failure. Final closed picture, verified end-to-end across all 3 rounds: (a) apply-divergence sets the fence correctly (5.2); (b) survives the structural timer's periodic re-check (round 2); (c) survives the promotion gate's re-check even in the hypothetical where the sticky flag itself somehow reads false (round 3); (d) only clears via a fresh process restart replaying cleanly — confirmed by round 1's own two-separate-persist_t-instances test. Forward note for task 6, already flagged in-repo (test_apply_correctness_verification.c comments), not a defect today: this entire safety picture rests on apply_verdict only ever being recomputed once, at boot. If task 6's live apply pump ever adds a live re-verification of apply-correctness (not currently planned, but worth checking), the "restart-only" assumption these 3 rounds of guards depend on would need re-examination at that time. Lesson for the whole series: reusing an existing mechanism correctly at the logic/decision level (5.2's apply_divergent short-circuit) does not automatically make every OTHER consumer of that same mechanism's output (ha.c's two independent, pre-existing live-clear call sites, written before this feature existed and with no reason to have considered a future trigger) still behave safely — a shared, trigger-agnostic sticky flag is exactly the kind of implicit coupling that needs an explicit audit of ALL its readers/writers whenever a new trigger is added, not just the one you're actively wiring.
• 5.2: routed APPLY_DIVERGENT (task 5.1) into the EXISTING loud-fence mechanism as a new INPUT, not a new fencing path — added bool apply_divergent to the existing recover_policy_inputs_t; persist_recover_decision() now unconditionally short-circuits to the pre-existing RECOVER_DECISION_FENCE_UNRECOVERABLE enum value when it's true (correct and design-mandated: a divergent local materialization can't be trusted for a peer/catch-up comparison either, so it must always fence regardless of peer state); reuses the same sticky recovery_unrecoverable flag and the same persist_recover_loud_fence_if_unrecoverable() function — confirmed by both review rounds to be the identical pre-existing symbols, zero new fence mechanism. Confirmed the existing promotion-gate consultation in ha.c (~line 1078, persist_recovery_unrecoverable(&c->persist)) already independently reads this flag — zero ha.c changes needed. Zero new self-termination paths (verified by grep both rounds) — stays alive, keeps retrying, per the CLAUDE.md process-lifecycle invariant. Took 2 review rounds. Round 1 REJECTED — real, test-reproduced defect, not documentation: the operator-facing ERROR() log's fixed boilerplate clause ("no healthy peer reachable to catch up; durable_floor_lsn=X < cluster committed high-water hwm=Y") fired UNCONDITIONALLY for every fence decision, including the new apply_divergent trigger — but for that trigger the fields are either hardcoded placeholders or explicitly-set values that made the clause LITERALLY FALSE in test output ("100 < 100", "2 < 0" — numerically impossible claims presented as fact to an operator during a real incident). Before this task, RECOVER_DECISION_FENCE_UNRECOVERABLE was only reachable via the structural "no healthy peer" branch, so the clause was truthful by construction; the new unconditional short-circuit broke that invariant. This directly undermines Req 8.3's "loud, operator-visible" requirement — a log an operator can't trust during an incident is worse than no log. Round 2 fixed by branching the ERROR() call itself (not just the inner reason string) on apply_divergent — that branch now states only the real, verified reason with no peer/HWM claim; the structural-fence branch is byte-identical to before. New regression test uses a genuine (non-mocked) stdout-capture mechanism relying on controller_unit_test never calling tlog_init() (verified true across all ~85 files in the test binary — confirmed the vendored tinylog fallback path used is a real raw write(STDOUT_FILENO,...), not a stub) — both review rounds independently reverted the fix and reproduced the exact false-clause failure before re-confirming GREEN. Lesson, consistent with task 2.2's earlier pattern: reusing an existing decision path correctly at the LOGIC level doesn't automatically make every SIDE EFFECT of that path (like a log message assuming context from the old, narrower set of triggers) still valid for a newly-added trigger — check accompanying log/error text for assumptions that were true only because of the old, narrower reachability, not just the returned enum/decision value itself.
• 5.1: added a new apply_verdict_t (APPLY_CLEAN/APPLY_DIVERGENT) computed alongside the EXISTING, untouched structural recover_verdict_t (CRC) verdict — a dual-check design: (1) an entity-count check comparing WAL_JOB_SUBMIT entries scanned during replay against the live c->jobmgr table's actual growth (count_jobmgr_entries()), genuinely independent since it queries live state rather than trusting apply_committed_entry()'s return code — verified real by both review rounds via a forced job-count-mismatch test; (2) a position check comparing an "expected" LSN against p->last_applied_lsn (new _Atomic uint64_t field, per design.md, forward-looking infrastructure for task 6's live pump). Took 2 review rounds. Round 1 REJECTED — real, live-today bug, not documentation: the position check's "expected" value was computed as recovered_tip = current_lsn - 1 (the PHYSICAL WAL-scan tip, which replay_wal() advances for every entry including WAL_CHECKPOINT markers and unknown types) instead of the LOGICAL apply cursor. persist_checkpoint() durably fsyncs the checkpoint marker in one step, then persists the advanced checkpoint_lsn to the on-disk snapshot in a LATER step — a crash between those two steps (a real, unavoidable crash-safety window this codebase already engineers hard for, cf. bug-661/ha-commit-durability) leaves a trailing checkpoint marker as the highest-LSN entry on next boot, causing recovered_tip to exceed the true last_applied_lsn and falsely report APPLY_DIVERGENT on a perfectly clean recovery — a direct Req 8.2 violation, reachable TODAY on any node that crashes at exactly the wrong moment during a checkpoint. Round 2 fixed by reading p->last_applied_lsn directly (the value replay_wal() already correctly advances only for entries actually dispatched through apply_committed_entry(), skipping checkpoint-marker/unknown-type entries) instead of recovered_tip; recovered_tip itself is untouched and still correctly feeds durable_lsn, which does need the physical tip. New regression test genuinely reproduces the trigger condition (real persist_checkpoint(), then unlink()s the exact snapshot file it wrote, landing on-disk state in the identical place a real crash would) — both review rounds independently reverted the fix and reproduced the false-DIVERGENT failure before re-confirming GREEN. Known, disclosed, accepted scope gap (not a defect, flagged for whoever extends this check): the entity-count check covers jobs only. Queues are correctly excluded (config-derived, WAL_QUEUE_CTRL never creates a new one, verified by both review rounds). Workers are NOT covered but CAN diverge: worker_from_capnp()/restore_worker() genuinely creates a new worker on a WAL_WORKER_CTRL/HOOK_DETAIL_WORKER_JOIN entry when absent from workermgr — a worker-creation divergence is currently invisible to this verification. Extend persist_verify_apply_correctness() with a worker-count check (mirroring count_jobmgr_entries()) if/when this needs to be closed. Self-disclosed, accepted limitation: because replay_wal()'s single-threaded caller in persist_recover() derives both the "expected" and "actual" position values from the same atomic within moments of each other on the same thread, the position check is diagnostically tautological FOR THIS SPECIFIC CALL SITE today (though the entity-count half remains genuinely independent) — this will change once task 6's live apply pump gives expectation and actual state a real chance to diverge across threads/time; task 5.1's boundary does not require solving that now.
• 4.2: pure test-only close-out of task-group 4 — one new test test_fact_state_unchanged_throughout_quorum_loss_outage in test_commit_barrier_reply.c (30→31), approved first round, zero production changes. Drives a genuine repl_remove_standby-based quorum-loss window through all 4 real fact-receipt handlers, polls live job state + committed_lsn at 5 time-spread points (real 20ms sleeps, ~100ms total) rather than once, asserts a real hook-spy never sees AFTER_EVENT fire, and confirms durable WAL presence via direct on-disk segment read (cbrp_count_jobstate_entries) — combining and strengthening two pre-existing tests that each covered only half (4.1's own single-point-in-time check under normal conditions; an older pre-4.1 test that proved WAL durability but never inspected live state). Static/inspection half of Req 6.2 confirmed independently by the reviewer, re-reading 4.1's landed diff: none of the four record_*_fact functions call apply_committed_entry/restore_job/mutate the real job object — record_job_fail_fact specifically operates on a throwaway job_t scratch = *job shallow copy. Architecturally significant finding for task 6 planning: facts have NO commit-timeout/failure concept at the receipt layer, unlike commands (task 3.2's commit_submit_async). persist_submit_job_fact() → wal_hook_append_classified(WAL_MODE_RETAIN) → commit_submit(..., COMMIT_MODE_RETAIN, 0), and COMMIT_MODE_RETAIN's branch in commit.c is an immediate, unconditional return with NO continuation registered (explicit comment: "RETAIN never honors the timeout") — commit_resolve() never fires for a retained fact. A fact's outage is therefore open-ended by construction; its eventual commit is discovered only when commit_advance_locked()/commit_on_ack notices committed_lsn can advance past it via the ACK-registry watermarks, purely. Task 6's apply pump must rely entirely on committed_lsn advancement to discover newly-eligible retained facts — there is no per-fact signal to wait on. Confirmed neither of 4.1's two flagged hazards (double-append, merge-staleness) is triggered by this test, since it never drives apply. Task-group 4 (fact apply-after-commit split) is now complete: 4.1 (receipt converted to durable-append-only, two critical hazards discovered and documented for task 6) → 4.2 (retention proven under sustained quorum loss, fact-vs-command architectural asymmetry documented). Both task-groups 1-4 are now done; task 6 (ordered apply pump) is the critical next milestone that must resolve, before going live: the double-append hazard, the merge-staleness hazard, AND design around facts' lack of a per-entry commit signal (commit_advance/committed_lsn-driven discovery only). Groups 5 (restart verification) and 6 (apply pump) remain; 5.1 depends only on 2.3 (already done) so could run in parallel with 6, per tasks.md's dependency graph — continuing in numeric order (5 next) unless reprioritized.
• 4.1: converted the four worker-fact receipt handlers (on_job_start, on_job_finish, on_job_fail, on_job_postdone — NOT literally named on_job_update, task text's naming was approximate) to validate-and-durably-append-only: BEFORE_EVENT validation hook unchanged, AFTER_EVENT (post-mutation) hook no longer fires at receipt, live state mutation removed. Key discovery: the durable WAL append was NOT a separate call — it lived INSIDE the very persist_job_hook_fun AFTER_EVENT subscriber this task needed to silence, so a new persist_submit_job_fact() (persist.c) was added that calls the same unmodified wal_hook_append_classified(WAL_MODE_RETAIN, wal_jobstate_t{jobid,state}, ...) directly — verified wire-format-identical to before, apply_committed_entry()'s existing switch (task 2.2) routes it unchanged. mark_job_start/mark_job_finish (single caller each, verified via diff against 41d3620) renamed in place to record_job_start_fact/record_job_finish_fact; mark_job_fail/mark_job_update (shared with job_ctrl's OUT-OF-BOUNDARY client-command path, 2 and 3 additional call sites) left untouched, with new durable-append-only siblings record_job_fail_fact/record_job_update_fact added instead — job_ctrl/do_jobcontrol_op confirmed byte-identical to pre-task by the reviewer. Approved first round. Real, permanent regression accepted, not fixed (Req 5.1 required the removal): on_job_postdone's finish_time = time(NULL) anomaly backstop is gone with no wire-format-level equivalent — concretely, a job hitting this rare no-prior-exit-status path now has finish_time stuck at 0 forever, which job_should_retain() (persist.c) evaluates as immediately GC-eligible instead of retained for the configured window. Narrow, rare-path, but real. TWO CRITICAL HAZARDS DISCOVERED BY REVIEW, MUST BE RESOLVED BEFORE TASK 6 WIRES apply_committed_entry() INTO A LIVE PATH — currently inert only because replay runs before persist_start() registers hooks (cluster.c: persist_recover() at line ~1451 vs persist_start() at ~1694):
  1. Latent double-append: persist_job_hook_fun is STILL registered on (XCTL_HOOK_JOB, HOOK_AFTER_EVENT) with no detail filter for HOOK_DETAIL_JOB_UPDATE, and does NOT early-return for that detail (it only early-returns for the WAIT-mode submit path). The moment task 6 wires apply_committed_entry() into live commit-advance, EVERY fact apply_committed_entry() applies will fire HOOK_DETAIL_JOB_UPDATE, which persist_job_hook_fun will catch and append AGAIN — a duplicate, mis-typed WAL entry per live-applied fact. Task 6.1/6.2 must retire or detail-filter persist_job_hook_fun's fact-handling branch before going live, since no live receipt path fires AFTER_EVENT for facts anymore (this task removed the only other source).
  2. Merge-staleness on deferred apply: record_job_fail_fact/record_job_update_fact precompute a FINAL target state at receipt time from a snapshot of current job state and bake it verbatim into the WAL entry (their own doc comments claim this "can never drift from what apply time will eventually compute" — that claim was TRUE only when receipt and apply were the same event; it becomes FALSE the moment they're separated by a commit round-trip, which is this entire feature's purpose). apply_committed_entry() routes all job-state facts through the generic apply_job_update_transition, which does a blind job->job.state = state overwrite — not the type-specific |= merge apply_job_fail_transition uses. Once two facts for the same job are in flight simultaneously (now newly possible, since apply is deferred), the second one to apply can silently clobber bits the first one set. Root cause is task 2.2's wire-format/generic-routing reality, not this task's diff — but no task in the current plan (4.2, 5.x, 6.x) tests or fixes this. Task 6 planning must resolve this before wiring live apply — likely requires either wire-format enrichment (baking the merge operation itself into the WAL entry, not just the precomputed result) or an apply-time re-derivation strategy.
• 3.3: pure test-only close-out of task-group 3 — 4 new tests added to test_commit_barrier_reply.c (25→29), approved first round, zero production changes. Closes two gaps 3.2 flagged: (1) an explicit-FAILED test via persist_truncate_wal_above/commit_on_truncate/commit_resolve_pending_above(-ENOENT), distinct from the existing timeout (-ETIMEDOUT) path; (2) the array_size > 1 mixed-outcome gap — one element ACKed to real majority while two time out, proving the committed sibling's persist_register_pending_apply hand-off isn't blocked by failing siblings, zero phantom jobs for the failures, and exactly one batch-level failure reply (via jobsubmit_reply_failure_once's ref-counted first-failure-wins gate). Also added two repeated-failure loop tests (5 independent cycles each, fresh jobid/LSN per iteration) proving the no-phantom-job guarantee isn't a one-shot fluke. Confirmed correct, non-obvious harness detail: this file's fixture wires job-submit continuations through a standalone commit_engine_t g_hce, NOT cluster_t's embedded c->commit field that demotion (raft_cb_become_follower) resolves against — so truncation, not demotion, is the only deterministic way to force an explicit FAILED (non-timeout) outcome in this specific test file. Demotion-forces-FAILED already has its own dedicated coverage in test_demotion_convergence.c (tasks 1.4/1.5) via a different fixture — not a gap, a file-boundary distinction. Non-blocking observation carried forward, not fixed here (pre-existing since 3.2, correctly out of scope for a test-only task): the single batch-level failure reply carries no per-element granularity — a client can't tell "1 of 3 committed" from "all 3 failed" from the reply alone. Task-group 3 (command apply-after-commit ordering) is now complete: 3.1 (job construction deferred past commit, bug-1408 caught+fixed) → 3.2 (converted to real async continuation-based commit, majority-ack safety independently verified intact, several documented design deviations) → 3.3 (failure-path guarantee proven under repetition and array mixed-outcomes). Standing operational blocker carried from 3.2, still true: no live path applies a successfully-committed job submission until task 6 exists — this branch must not be soak-tested or deployed against live traffic yet.
• 3.2: converted handle_jobsubmit() from 3.1's still-synchronous commit to a real async registration against task 1's continuation API. New persist_submit_job_payloads_async() (persist.c) appends the WAL entries via the same ha_primary_log_append/persist_wal_async primitive the pre-existing RETAIN-fact path (wal_hook_append_classified) already uses, then calls commit_submit_async() directly on p->repl.ce — bypassing repl_wait_majority_ack/repl_submit_majority_commit in repl.c, which is what design.md's text actually specifies as the integration site. Independently verified safe (reviewer traced the full ACK→commit_resolve() path in commit.c): majority-ack detection is driven entirely by (ce, lsn) events inside the commit engine, agnostic to which wrapper registered the continuation — no false-commit risk. But it's a real, permanent design deviation, not just an implementation detail: this creates a THIRD parallel "append+track a commit" implementation (alongside the old sync one, now alive only in 3 test files, and the RETAIN-fact one), narrowing the reusable primitive design.md wanted — a future command handler (e.g. a config-change command) gets no shared plumbing from this work and would need its own bespoke path. On FAILED/timeout, replies failure immediately via jobsubmit_reply_failure_once(), no job ever constructed. On COMMITTED, calls new persist_register_pending_apply(p, lsn, on_applied, ctx) — a hand-off linked list on persist_t for task 6.1 to consume — and does NOT apply or reply, per the task's explicit instruction. Deleted (not preserved-dead) task 3.1's bug-1408 create_job-loop/notify block: verified correct — apply_committed_entry()'s existing WAL_JOB_SUBMIT case (task 2.2) already fires HOOK_DETAIL_JOB_NEW per-entry independently, structurally immune to bug-1408's batch-suppression class; resurrecting the old loop in task 6 would double-construct/double-notify. Original code fully recoverable from git show 6f1fa97:src/controller/job.c if ever needed. CRITICAL OPERATIONAL CALLOUT — this branch must NOT be soak-tested or deployed against live traffic until task 6 lands: there is currently NO live code path that applies a successfully-committed job submission (only restart replay, task 2.3, does) — every successful job submission produces a durably-committed WAL entry but ZERO reply to the client, forever, until task 6.1/6.2 exist to consume the pending-apply hand-off and complete the reply. This is exactly what the task text sanctions ("no independently observable behavior until task 6"), not a bug, but it means this commit range is only safe in isolated unit-test form right now. Hand-off mechanism for task 6.1's briefing: persist_register_pending_apply()/persist_has_pending_apply() exist; there is NO "take and remove" accessor yet — task 6.1 must build the consumer side itself. The per-batch jobsubmit_async_ctx_t (ref-counted across array elements, frees only once every element has either failed-and-replied or been applied via jobsubmit_apply_complete) will not free for any batch containing a committed element until task 6 exists — an accepted, bounded-by-test-scope leak today, but would be unbounded growth under any live/soak run before task 6 lands. Outcome encoding uses a plain int rc (not commit_outcome_t+rc) to avoid a persist.h→commit.h circular dependency — verified unambiguous today (every real FAILED call site in commit.c passes non-zero rc) but not compiler-enforced; fragile by convention, not by construction. New regression found by review, not yet fixed, non-blocking: array_size == 0 now hangs the client forever and leaks one jobsubmit_async_ctx_t per request (3.1's version replied synchronously even for this input; 3.2's async loop never runs so the ctx never releases) — confirmed NOT reachable via any current legitimate client (Go apiserver always sets arraySize >= 1; xce_test's cmd_jobsub clamps arsize <= 0 to 1), so flagged as a defense-in-depth hardening gap for a future task rather than fixed here. Also no array_size > 1 mixed-outcome test exists yet (per-element independent resolution verified correct by code inspection, not by test) — fast-follow recommended.
• 3.1: reordered handle_jobsubmit() (job.c) so job objects are only constructed via create_job() AFTER the (still-synchronous) WAL commit succeeds — extracted jobid_range_exists() (fast-fail pre-check, no WAL work), build_job_submit_payloads() (WAL-payload construction with no jobmgr insertion), and a new persist.c sibling persist_submit_job_payloads_and_commit() (operates on raw xce_job_t* instead of pre-existing job_t*, single production call site in job.c — legitimately within handle_jobsubmit's call chain, not a boundary spillover; no task-1 continuation API dependency, genuinely still synchronous). Took 3 review rounds. Round 1 REJECTED — real defect, not documentation: the post-commit array-construction loop suppressed the AFTER_EVENT/HOOK_DETAIL_JOB_NEW notification for an ENTIRE array batch if even one sibling element failed to construct (e.g. a jobid collision slipping past jobid_range_exists()'s inherent TOCTOU window, or a plain allocation failure — no race needed) — stranding OTHER elements that were durably committed AND successfully constructed, invisible to the purely event-driven scheduler/apiserver until the next restart's replay. Logged as bug-1408. Round 2 fixed the defect (moved the construct_rc < 0 early-return to AFTER the jlist-build + hook-fire block, so every successfully-constructed sibling is always notified regardless of a sibling's failure; client reply still separately reports the batch's construct_rc) but was REJECTED again on a narrower, documentation-only ground: bug-1408's RED-phase writeup claimed a clean assertion FAIL, but under the standard full-group --run test_commit_barrier_reply invocation the failing assertion actually longjmps past the test's own cbrp_teardown() call (Unity TEST_ABORT semantics) and HANGS the next test — the same longjmp-past-teardown pathology already known from task 1.3's era. Round 3 (parent-handled directly, no code regression risk): corrected bug-1408's buglog.json description to accurately describe the hang mode, and hardened the new test itself — every assertion-relevant observable is now captured into a local, cbrp_teardown() is called unconditionally, and assertions run only after teardown, so a future regression in this exact test can no longer skip cleanup and deadlock sibling tests. No production code changed in round 3; job.c's fix from round 2 was independently verified correct by the round-2 reviewer (traced jlist-build/hook-fire at job.c:717-766 running unconditionally before the construct_rc check at line 775) and needed no further changes. Known low-severity residual, not blocking, flagged for a future task: apiserver_push_joblist (apiserver.c) lacks the nfiltered > 0 guard sched.c's equivalent has, so a total-batch construction failure (every array element fails, jlist ends up empty) now sends an empty/no-op jobs push to every connected apiserver — informationless, can't corrupt state, but a minor behavioral quirk introduced by firing the hook unconditionally.
• 2.4: pure test-only close-out of task-group 2 — new test_apply_reapply_idempotency.c (3 tests), no production changes, approved first round. Covers three distinct cases beyond 2.2's existing coverage: (1) triple-reapplication of an identical WAL_JOB_SUBMIT through apply_committed_entry() with hook-fire-count AND pointer-identity assertions across all 3 calls (not just final state); (2) the JOB_TRANSITION_NOOP_DUPLICATE guard branch (state == current state) — a genuinely untested path, distinct from 2.2's forward-once and backward-rejected cases; (3) the task's named scenario, driven entirely through the real dispatch seam (not seeded/shortcut): real SUBMIT → real FINISH → stale out-of-order START (lower LSN, arriving after), asserting the job stays in its FINISH/terminal state and no third hook fires for the rejected START. Test assertions key on hook-fire COUNT and job STATE, never a specific HOOK_DETAIL_* value — so these tests stay valid even if 2.2's documented generic-hook-routing deviation is later corrected. Task-group 2 (shared apply seam) is now complete: 2.1 (guard cores extracted) → 2.2 (dispatch function built, generic job-state-family routing is a confirmed permanent deviation from design.md, tracked) → 2.3 (replay wired to the seam, duplicate-notification-on-restart hazard confirmed inert today but must be re-verified before 4.1/6.1/6.2/6.3) → 2.4 (idempotency proven through the real dispatch path). Next: task-group 3 (command apply-after-commit ordering), which is the first consumer of both this seam and task-group 1's commit-engine continuation API.
• 2.3: rewired replay_wal() (persist.c) to call apply_committed_entry() (task 2.2's seam) instead of apply_wal_entry() for restart replay's per-entry apply. apply_wal_entry/apply_committed_entry are siblings, not wrapper/wrappee — apply_wal_entry is NOT retired, it remains ha.c's live standby catch-up/replicated-apply dispatch (untouched, different call sites, out of this task's boundary). Confirmed hazard, investigated to resolution by the reviewer (not left open): replay_wal(p, p->checkpoint_lsn, ...) replays the FULL WAL range since the last snapshot on every restart (last_applied_lsn is deliberately not persisted — recomputed at replay time per design.md), not just a narrow crash-gap tail. Since apply_committed_entry fires AFTER_EVENT hooks (unlike the old apply_wal_entry, which never did), this means every restart re-fires notification hooks for entries already applied+notified in a prior process life — a duplicate-notification hazard in principle. Independently traced and confirmed inert today for both existing hook consumers: scheduler hooks aren't registered until on_scheduler_join() (sched.c), which happens post-boot via network JOIN handshake, strictly after persist_recover() runs; apiserver hooks ARE registered before persist_recover() (apiserver_init() in cluster_init_phase2()), but every apiserver hook body gates its actual network notify on apiservers[i].ep != NULL, and the apiserver-join table starts empty each process life — a no-op except via a narrow reconnect race. MUST be re-verified before tasks 4.1/6.1/6.2/6.3 make this path unambiguously live and broad (e.g. if a future task registers apiserver connections earlier, or scheduler hooks get registered pre-replay) — at that point this either needs a replay-mode flag suppressing hook-firing, or an explicit accepted-idempotent-duplicate-notification decision recorded in design.md. A loud in-line comment cross-referencing this note now sits at the apply_committed_entry() call site in replay_wal() (persist.c). Also confirmed: job-state facts now replay through 2.1's guarded apply_job_update_transition() instead of the previously-unguarded restore_job_state() — intentional (Req 3/7 online-replay divergence fix), verified no existing test depended on the old unguarded semantics. apply_committed_entry's missing oversized-payload guard (vs. apply_wal_entry's MAX_WAL_PAYLOAD_BYTES/-EMSGSIZE check) confirmed inert — persist_wal_write_replicated() already rejects oversized payloads at write time, before anything reaches disk.
• 2.2: implemented apply_committed_entry(persist_t*, uint32_t wal_type, uint64_t lsn, const void* payload, uint32_t len) in persist.c/persist.h, matching design.md's signature. Routes WAL_JOB_SUBMIT through restore_job() (idempotent on duplicate jobid — no-op, Req 7.1), WAL_QUEUE_CTRL/WAL_WORKER_CTRL through their existing restore primitives, and fires each type's post-mutation hook exactly once, after mutation. Confirmed permanent, non-cosmetic deviation from design.md's Apply Seam wording: ALL nine job-state fact WAL types (WAL_JOB_{START,FINISH,FAIL,STOP,RESUME,UPDATE,RUSAGE,CKPT,PREASON}) route through the single guard-only apply_job_update_transition() core from 2.1 — NOT the type-specific apply_job_{start,finish,fail}_transition cores — and fire one generic HOOK_DETAIL_JOB_UPDATE hook, not the type-specific HOOK_DETAIL_JOB_START/_EXIT/_FAIL. Root cause, independently verified by two review rounds: the on-disk wire format for every one of these types (wal_jobstate_t {jobid, state}) genuinely only carries the resulting state, not the richer job_pidinfo_t/xce_jobfinish_t/job_failinfo_t payload the type-specific cores/hooks require — this is a pre-existing wire-format reality (apply_wal_entry already collapsed these the same way before this task), not a shortcut invented here. design.md's Apply Seam section and Fact Ingestion → Event Contract section currently describe per-type hook dispatch that this wire format cannot support as written — this is a real, standing design/reality conflict, not yet reconciled in design.md itself. No task in the current plan (2.3, 4.1, 4.2, 6.1, 6.2, 6.3) enriches the wire format to close this gap — confirmed by reading all of them; task 4.1 is exclusively about removing receipt-time mutation/hook-firing, not payload enrichment. Explicit blocking-style call-out for whoever implements 4.1/6.2/6.3: wiring this dispatch function into a live fact-apply path will permanently change every worker-reported job start/finish/fail notification to a generic UPDATE detail instead of the type-specific one, unless the wire format is enriched first or design.md is corrected to formally accept this substitution — verify this is still the accepted behavior before wiring it live, don't assume it'll be fixed by then. Took 3 review rounds to land: round 1 rejected two doc-comment defects (a citation blaming the wrong crash site, and a dangling pointer to task 4.1 as the fix location); round 2's fix corrected the crash site (real one is sched.c's sched_job_start_fun/_exit_fun/_fail_fun, which unconditionally dereference info->jobid before any NULL check) but fabricated a nonexistent function name (encode_JobPIDInfo_ptr) as supporting evidence; round 3 replaced it with a narrower, fully source-verified claim (apiserver.c's equivalent functions store the raw pointer into a notify union without dereferencing it at that call site — real, checked line-by-line). Lesson: when a reviewer catches a citation error, the safest fix is the narrowest claim you can personally verify by reading the actual code, not a broader-sounding replacement — round 2's attempt to explain "why the alternative is fine" reached for an unverified downstream mechanism and got caught fabricating it.
• 2.1: split mark_job_start/mark_job_finish/mark_job_fail/mark_job_update (job.c) into four hook-free cores (apply_job_{start,finish,fail,update}_transition, new job_transition_result_t enum in cluster.h) containing only the monotonic-rank guard + mutation, verified an unmodified lift line-by-line against pre-extraction code. Wrappers peek the same guard predicate to decide hook-firing, then delegate to the core — no unlock/relock gap between peek and delegated call, so no new race from the double-evaluation. mark_job_fail's pre-existing lack of a guard is preserved, not added or removed. New test_apply_seam.c (8 tests) calls the cores directly, bypassing on_job_* receipt handlers entirely — this is the form task 2.2 (dispatch function) and task 6.1 (apply pump) will call into. Non-blocking follow-up flagged by reviewer: the new doc-comments on apply_job_fail_transition/apply_job_update_transition overclaim a universal "caller holds RTTI_LOCK_WRITE" precondition — job_ctrl (job.c:396, pre-existing, untouched) calls mark_job_update/mark_job_fail under only RTTI_LOCK_READ. No new race (extraction added no unlock/relock), but whoever writes task 2.2/6.1 should not trust that comment's blanket claim — verify the actual lock held at each call site instead.
• 1.5: pure test-only close-out of task-group 1 — 3 new genuine-multi-thread tests added across test_commit_resolve.c/test_demotion_convergence.c/test_persist_truncate.c (real pthreads, not sequential calls dressed as concurrent): an admission-cap burst with a concurrent monitor thread, a mixed majority-ack/timeout-sweep burst proving bounded dispatch-thread usage, and a genuine two-real-thread ack-vs-sweep-thread race (closing the "single-thread white-box only" gap both task-1.3 review rounds had flagged as a nice-to-have). No production code touched. bug-1407 saga, definitively resolved: it was NEVER a source bug. 1.4's "closed, 70/70" and 1.5's initial "reopened, 10/10 failing" were BOTH real, correct, contradictory measurements of the exact same source commit (9b4b805) — the difference was a stale/corrupted incremental-build artifact in the shared build/ directory, which had been ninja -C build-rebuilt incrementally across all 5 tasks without ever a fresh meson setup. Proven by isolating each measurement in its own throwaway git worktree add + fresh meson setup/ninja: pre-feature commit and the exact failing commit both pass cleanly (0 failures) on a fresh build; only the main worktree's stale incremental build/ produced the failure. Fixed with rm -rf build && meson setup build --buildtype=debug -Db_sanitize=address && ninja -C build; bug-1407 is now genuinely CLOSED with high confidence. Lesson for every remaining task in this series: distrust a surprising/contradictory ninja -C build incremental result before concluding a source regression — rebuild clean (or isolate via a throwaway worktree) first. Never use git checkout <sha> -- . to inspect an old commit in place — it silently reverts working-tree files against current HEAD; use git worktree add <path> <sha> instead. Both lessons are now in .wolf/cerebrum.md Do-Not-Repeat and .wolf/buglog.json bug-1407.