persist_checkpoint() stamped checkpoint_lsn from p->current_lsn (physical
WAL append tip) instead of p->last_applied_lsn (the true "jobmgr reflects
up through here" cursor), uncoordinated with the async apply pump
introduced by raft-apply-decoupling. write_snapshot() serializes live
jobmgr state, so a checkpoint taken while apply lags the WAL tip mislabels
the snapshot: on restart last_applied_lsn is re-seeded from checkpoint_lsn
and replay resumes strictly above it, silently skipping committed-but-not-
yet-applied entries jobmgr never received. This is the actual mechanism
behind kill-mid-apply producing a permanent APPLY_DIVERGENT fence —
apply_committed_entry() is deterministic/idempotent and write_snapshot()
is crash-atomic, so the bug was a mislabeled watermark, not a structural
divergence requiring a safety/liveness tradeoff.
Fix stamps checkpoint_lsn from last_applied_lsn and holds apply_pump_lock
across the label capture + write_snapshot() so the label and the jobmgr
state it describes are captured atomically w.r.t. the live apply pump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VgT3n4vTv4ZjCuCTmzed1F
jobid_range_exists() only checks c->jobmgr, but task 3.1 deliberately
defers job construction until after commit -- so a submission that is
durably WAL-appended and awaiting its commit outcome is invisible to
that check. A client-side retry (XCECLI_FAILOVER_CALL resending the
same jobid after a transport error or failover mid-flight) sails past
the guard and appends a second, non-identical WAL_JOB_SUBMIT entry for
the same jobid -- confirmed via waldump on a live chaotic-soak node
(duplicate submits, same jobid, differing subtime/CRC) -- which can
trip persist_recover()'s post-replay apply-correctness check into a
genuine APPLY_DIVERGENT verdict.
Adds a process-lifetime in-flight-jobid-range registry in job.c that
handle_jobsubmit atomically check-and-claims immediately after the
existing jobid_range_exists check, releasing from
jobsubmit_async_ctx_release (the single choke point already shared by
the commit-continuation, apply-complete, and OOM early-out paths). A
resubmit for a still-in-flight jobid now gets -EEXIST with no second
WAL append.
Adds a matching XCE_TESTING reset seam (declared in cluster.h) so a
test that deliberately leaves a submission's commit continuation
unresolved through teardown doesn't leak its jobid range into a later
test reusing the same range -- commit_engine_fini frees pending
continuations without invoking their callback.
Regression test added to test_commit_barrier_reply.c (RED->GREEN
confirmed). Full 78-group controller_unit_test sweep: 782 tests, 0
failures.
This fixes only the WAL duplicate-submit mechanism (layer 1 of
bug-1419). The deeper PreVote-freshness/apply-divergence liveness
deadlock (layer 2 -- no recovery path when the only fresh-enough node
is apply-divergent-fenced) remains open and needs its own design
discussion; buglog.json updated accordingly.
Chaotic mode's run_chaotic_rounds restarts "the previously-downed node"
every round unconditionally via start_node(\$DOWN), but a round that skips
(no primary to kill) never updates \$DOWN -- so a stale node name reaches
start_node again while its own earlier restart is still alive, stacking a
second xce_controller on the same --myname/port without ever killing the
first. Six stacked ctrl-2 processes were observed by the end of a 10-round
run.
Fixed inside start_node itself with a pgrep guard, rather than reworking
run_chaotic_rounds's DOWN bookkeeping, so every call site is covered:
refuse to launch a duplicate if "xce_controller --myname \$1" is already
running.
Verified: re-ran the identical 10-round --chaotic soak that surfaced this.
The guard fired on every skip-round; exactly 3 xce_controller processes
alive throughout the entire run.
Note: the same soak run still hits a genuine, separate election-liveness
stall (a surviving node fenced non-promotable by a real apply-correctness
divergence, traced to non-identical duplicate WAL submits for the same
jobid) -- filed as bug-1419, not fixed here.
vote_reply_dispatch passed the compressed, self-excluded local votes_idx
(vctx->peer_idx, set by raft_cb_send_vote_request's absolute->local remap)
straight through to raft_on_vote_reply, which indexes vote_replies[] in
ABSOLUTE cluster space -- the same space raft_start_election's self-mark
and per-peer send loop use.
For a 3-node cluster's middle node (self_idx=1), the peer at absolute
index 2 compresses to votes_idx=1, colliding with vote_replies[self_idx=1]
-- the slot raft_start_election already marked for the self-vote. The
library's per-peer dedup guard then silently drops the genuine reply as a
duplicate, so votes_granted never reaches quorum even though the peer
granted. Peer 0 is unaffected (compressed index equals absolute), which is
why sequential elections and prior soak runs never surfaced this -- only a
middle-priority node losing its higher-priority peer and depending on its
lower peer's vote hits the collision.
Fixed by routing through ha_repl_ack_abs_peer_idx (already existed,
already unit-tested, already used for the analogous heartbeat-ACK path)
before calling raft_on_vote_reply. ha_record_vote_reply immediately above
is unchanged -- it correctly wants the compressed index for ctx->votes[].
Confirmed pre_vote_reply_dispatch needs no equivalent fix: its send path
already stores the absolute index.
New regression test drives a real 3-node election end-to-end through the
actual decode/dispatch path (not the test-hook bypass every other
real-vote test in this file uses): Expected 2 Was 1 pre-fix, GREEN
post-fix. Full 79-group suite: zero new failures. 10-round --chaotic HA
soak: zero committed divergence, zero committed job loss.
Closes the two non-blocking follow-ups from the 2026-07-07 validate-impl
audit:
- design.md had 3 sections describing the original plan rather than what
landed, each already self-disclosed in tasks.md's Implementation Notes
but never reconciled back: repl.c's planned repl_submit_majority_commit()
was never built (task 3.2 calls commit_submit_async() directly instead);
the apply pump is wired via commit_resolve_pending() in commit.c, not
ha.c's on_commit_advanced; fact hooks fire one generic
HOOK_DETAIL_JOB_UPDATE, not per-type hooks (verified safe -- sched.c's
subscribers already consume the generic form).
- task 5.3 touched ha.c (two promotion/fence call sites) without a formal
_Boundary:_ exception the way task 1.4 got one for its own out-of-file
touch -- added retroactively, no behavior change.
controller-status, ha-commit-engine, ha-heartbeat-committed-propagation,
ha-prevote, ha-single-leader-role, ha-wal-replication, libxce-failover, and
raft-ha-core-safety each had every numbered sub-task marked [x] but their
group-header lines (e.g. "1. Foundation: ...") left unchecked -- a cosmetic
gap, confirmed by roadmap.md's own IMPLEMENTED/VERIFIED notes and zero real
unchecked sub-tasks in any of the eight files. No task content changed.
raft-apply-decoupling: 32/32 tasks complete, validate-impl GO (79/79 test
groups, live smoke boot verified) -- update roadmap's In Flight entry with
outcome, residuals, and follow-ups.
raft-drive-async: all 17 sub-tasks were already [x] but the 4 phase-header
checkboxes were never ticked, leaving tasks.md looking incomplete. Confirmed
via source (g_drive_async_configpush_offload defaults to 1) and prior
session notes that all 4 phases genuinely landed and flipped ON in
production; checkboxes now match reality.
Measured sustained worker-fact throughput against the pre-feature baseline.
Commit throughput is unregressed (~2.5% delta, within noise) and no per-fact
synchronous commit barrier exists, confirming Req 11.4's core claim. But
apply/observable throughput -- the new hop this spec introduces -- regressed
56% with clearly superlinear scaling (2x volume -> 5.2x elapsed).
Root cause: apply_pump_read_entry has no memory across calls -- every LSN
lookup re-lists WAL segments and rescans its segment from byte 0 -- while its
only caller applies committed entries one at a time in strict ascending
order, turning a sequential access pattern into O(n^2) total work.
Fixed with a minimal segment-read cache on persist_t (fd + offset + expected
next LSN, guarded by the existing apply_pump_lock): a fast path reads the
next expected LSN directly at a remembered offset, falling back unchanged to
full rediscovery on any miss. Correctness never depends on the cache, only
performance does. Invalidated on init, fini, recovery, and WAL truncation --
the four points a stale cursor could otherwise misread from.
Two review rounds: round 1 caught a missing lock guard in the recovery
invalidation path and an inaccurate risk claim about the cache/trim
interaction (a real, bounded behavior change deferred to the existing
APPLY_DIVERGENT fence, not "no worse than before") -- both corrected and
re-verified in round 2. Post-fix apply-throughput scales sublinearly (1.46x
elapsed for 2x volume) and exceeds the pre-feature baseline. Zero new test
failures across the pump, correctness-verification, recovery, and truncation
groups.
An intermittent, pre-existing client hang in the synchronous jobsub path and
a trim-floor/apply-lag architectural gap were surfaced but are explicitly
out of scope for this fix -- flagged for separate follow-up.
Full 79-group unit/integration suite re-run found exactly 2 stale tests in
test_commit_barrier_reply.c asserting task 3.2's documented interim
contract ("no observable effect until task 6 completes it"), which the
already-committed ordered apply pump (tasks 6.1/6.2) closed by construction.
Updated both to assert the real, deterministic apply-after-commit settled
state instead of the no-longer-true no-effect-yet state, matching the idiom
already used elsewhere in the same file. No production code changed.
Full suite: 0 failures across 79 groups. HA soak (--sequential, 5 rounds):
zero committed-job loss, zero divergence, all 3 nodes served as primary.
Both independently re-verified by adversarial review.
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.
Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.
Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
Extend raft_sim.c's deterministic TLA+-conformance harness with two
fault-injectable checks: no reaction before commit, and idempotent
re-apply of an already-committed entry. Wired as dedicated tests only,
not into raft_sim_check_all's fixed four-invariant contract (owned by
a separate spec). RED/GREEN independently verified by adversarial
review; zero regression to existing scenario functions.
Extend test_apply_pump_integration.c with 2 tests, no production
changes. Test 3 interleaves commands and facts through a real
quorum-loss outage and proves strict original-log-order catch-up via
a sequence spy. Test 4 wires a real 4-vcpu vtaskmgr_t to drive 4
commands + 2 facts through one covering ACK, creating genuine N-way
concurrent races across task 6.2's self-healing fix rather than one
isolated forced pair -- stress-testing bug-1415's fix under a
realistic catch-up burst.
Closes out task-group 6 (ordered apply pump integration): the pump
is now live, hands off command replies correctly, and is proven
correct for isolated and mixed/bursty command+fact scenarios under
both normal and quorum-loss conditions.
Add test_apply_pump_integration.c (2 tests), no production changes.
Exercises the full live pump (real commit engine, real apply
dispatch, real hook subscribers) for both a job submission and a
fact: proves reply-after-visibility ordering for commands, and zero
fact-reaction side effects during a delayed-commit/quorum-loss
window (polled repeatedly across real time, not checked once).
Confirms Req 4.3 by construction, not assertion: the apply walk only
ever invokes apply_committed_entry() for LSNs already <= committed_lsn,
so a consumer acting on the post-apply notification never needs to
separately re-check a commit watermark.
Independently verifies apply_committed_entry()'s generic
HOOK_DETAIL_JOB_UPDATE routing (unchanged by task 6.1's _pad fix)
makes the type-specific sched.c/apiserver.c hook handlers
structurally unreachable for live-applied facts -- confirmed their
va_arg-typed signatures would type-confuse-crash if ever called with
the job_t* apply_committed_entry() actually passes, making the
single generic reaction spy a sound design, not a coverage gap.
Repo-wide audit confirms exactly 2 live call sites ever fire these
hook details (the guarded apply seam, and job_ctrl's legitimate
SIGCONT/SIGSTOP path) -- no orphaned third site.
Wire persist_apply_pump_run() to call persist_take_pending_apply()
immediately after each entry's apply_committed_entry() call,
invoking the registered continuation strictly after the mutation.
Adds jobsubmit_reply_success_once() (job.c) so a successful job
submission finally sends a client reply once every array element
applies -- closes the standing gap from task 3.2 where committed
submissions never got a response.
Fixes bug-1415: commit_resolve_pending() dispatches the commit
continuation and the apply-pump kick as two independent
VTASK_LIVE_BLOCK vtasks with no completion-order guarantee. On any
real multi-vcpu pool (the production default, n_vcpus=8), if the
apply pump won the race and advanced last_applied_lsn past an LSN
before its continuation registered, the registration was permanently
orphaned -- the job would be constructed and visible, but the client
would hang forever. Untested by every existing fixture in this spec
(all use pool=NULL, which trivially serializes and can't reproduce
the race).
Fixed with a self-healing check in persist_register_pending_apply():
if the LSN is already <= last_applied_lsn, invoke the callback
immediately instead of registering for later pickup. Verified by the
spec's first genuine multi-vcpu regression test (real vtaskmgr_t,
n_vcpus=4), independently RED->GREEN reproduced by two review rounds.
Build persist_apply_pump_run()/persist_apply_pump_kick() -- the
ordered walk from last_applied_lsn+1 through committed_lsn, applying
every entry via apply_committed_entry() regardless of commit mode.
This is the first live (non-replay) consumer of that dispatch
function. Kick is wired into commit_resolve_pending() (the single
choke point all 6 internal committed_lsn-advancing paths funnel
through, discovered in task 1.2), dispatched via
vtask_enq_classified(VTASK_LIVE_BLOCK) with a CAS-coalesced
fire-and-forget trigger.
Fixes the two hazards flagged since task 4.1 as required before
going live:
1. Double-append: a thread-local call-provenance guard
(t_apply_seam_notifying) prevents persist_job_hook_fun/
persist_queue_hook_fun/persist_worker_hook_fun from re-appending
entries the apply pump itself just applied, while preserving
job_ctrl's legitimate live command-path appends (a naive
detail-based filter would have broken SIGCONT/SIGSTOP). Also
closes an undocumented instance of the same hazard in the
queue/worker hook functions.
2. Merge-staleness: record_job_fail_fact/postdone's fact recording
previously baked a receipt-time-computed final job state into the
WAL entry, which goes stale under deferred apply -- a FINISH
followed shortly by an ordinary post-exec FAIL could silently
clobber a correct DONE completion with a stale RUN|EXIT|PERR, no
guard catching it. Fixed by repurposing wal_jobstate_t's unused
_pad field to carry just enough data (FAIL's stage) to recompute
against the job's actual current state at apply time via the real
apply_job_fail_transition()/new apply_job_postdone_transition()
guard cores, instead of overwriting with a stale absolute value.
Backward compatible: _pad==0 (all prior data) routes to the same
legacy overwrite behavior.
Also fixes bug-1409 (unbounded busy-loop in apply_pump_drain when
committed_lsn advances without a corresponding on-disk entry) and a
genuine test-flakiness hazard (a second, independent apply-pump
kicker from the fsync_worker thread that existing tests didn't
account for).
Two pre-existing tests in test_commit_barrier_reply.c now fail as
expected -- their own assertions state "no job may be constructed
until task 6's apply pump exists," now correctly invalidated by this
task landing; updating them is task 7.3's explicit job.
Add explicit test coverage for the DIVERGENT fence's stay-alive and
restart-only-clear guarantees (task 5.3), which surfaced a critical,
pre-existing safety gap: the sticky recovery_unrecoverable flag is
trigger-agnostic, and two pre-existing live call sites in ha.c
(promotion_fence_timeout_fn's periodic timer, and
raft_cb_become_leader's promotion-eligibility gate) clear it based
purely on structural conditions, never checking the apply verdict.
A node whose local apply dispatch diverges from the committed log,
but whose WAL bytes are otherwise structurally healthy, would have
its fence silently released by the live timer, then be eligible for
promotion to primary on a later election -- serving genuinely
divergent state.
Fix: gate both clear-call-sites on persist_apply_verdict(&c->persist)
!= APPLY_DIVERGENT in addition to their existing structural checks.
An apply-divergence-triggered fence can now only clear via a fresh
process restart's clean persist_recover() replay.
New tests: test_promotion_fence_apply_divergent_survives_structural_catchup
(test_commit_durability.c) covers the timer site; and
test_promotion_refused_when_apply_divergent_even_if_eligible
(test_convergence_verification.c) covers the promotion-gate site,
structurally isolated so it cannot pass for the wrong reason.
Closes out task-group 5 (restart apply-correctness verification).
Add bool apply_divergent to the existing recover_policy_inputs_t;
persist_recover_decision() unconditionally short-circuits to the
existing RECOVER_DECISION_FENCE_UNRECOVERABLE when true, reusing the
same sticky recovery_unrecoverable flag and
persist_recover_loud_fence_if_unrecoverable() -- no new fence
mechanism, no ha.c changes needed (the existing promotion gate
already reads the flag independently), no new self-termination path.
Fixes a real bug found during review: the fence's operator-facing
ERROR() log unconditionally asserted a "no healthy peer reachable...
durable_floor_lsn < hwm" clause that was only truthful for the
pre-existing structural-fence trigger. For the new apply_divergent
trigger this produced literally false claims in test output
("100 < 100", "2 < 0"). Fixed by branching the log call itself on
apply_divergent so it states only the verified reason.
Add apply_verdict_t (APPLY_CLEAN/APPLY_DIVERGENT) computed alongside
the existing structural (CRC) recover_verdict_t. Dual-check design:
an entity-count check (WAL_JOB_SUBMIT entries scanned vs live jobmgr
growth, genuinely independent of apply_committed_entry()'s return
code) and a position check against a new last_applied_lsn field
(forward-looking infrastructure for task 6's live pump).
Fixes a real bug found during review: the position check's
expectation was computed from the physical WAL-scan tip
(current_lsn - 1), which advances past checkpoint markers and
unknown entry types that never reach apply_committed_entry(). A
crash between persist_checkpoint()'s marker-fsync step and its
later snapshot-persist step leaves a trailing checkpoint marker on
next boot, which falsely reported APPLY_DIVERGENT on a perfectly
clean recovery -- reachable today, not gated behind a future task.
Fixed by reading last_applied_lsn directly, which only advances for
entries actually dispatched through the apply seam.
Known accepted gap: entity-count check covers jobs only, not
workers (which can genuinely be created via WAL_WORKER_CTRL) --
documented for future extension.
Add test_fact_state_unchanged_throughout_quorum_loss_outage to
test_commit_barrier_reply.c (30->31), no production changes. Drives
a real repl_remove_standby quorum-loss window through all 4
fact-receipt handlers, polls live state and committed_lsn at 5
time-spread points (not once), asserts a real hook-spy never sees
AFTER_EVENT fire, and confirms durable WAL presence via direct
on-disk read.
Documents an architectural asymmetry for task 6 planning: facts have
no commit-timeout concept (COMMIT_MODE_RETAIN registers no
continuation, "never honors the timeout") unlike commands'
commit_submit_async -- the apply pump must discover newly-committed
retained facts purely via committed_lsn advancement, not any
per-fact signal.
Closes out task-group 4 (fact apply-after-commit split).
Convert on_job_start/on_job_finish/on_job_fail/on_job_postdone to
validate and durably append only -- BEFORE_EVENT validation hook
unchanged, AFTER_EVENT (post-mutation) hook no longer fires at
receipt, live state mutation removed. Adds persist_submit_job_fact()
(persist.c) since the durable append previously lived inside the
very AFTER_EVENT hook this task silences; wire format is unchanged
and identical to what apply_committed_entry() already routes.
mark_job_start/mark_job_finish (single caller each) renamed in place
to record_job_start_fact/record_job_finish_fact. mark_job_fail/
mark_job_update (shared with job_ctrl's client-command path, out of
boundary) left untouched; new durable-append-only siblings
record_job_fail_fact/record_job_update_fact added instead.
Accepts a narrow, permanent regression: on_job_postdone's finish_time
anomaly backstop has no wire-format equivalent and is dropped
(affects only a rare no-prior-exit-status path).
Two critical hazards discovered by review, documented in tasks.md,
MUST be resolved before task 6 wires apply_committed_entry() into a
live path: (1) persist_job_hook_fun still fires on HOOK_DETAIL_JOB_UPDATE
and will double-append every live-applied fact; (2) record_job_fail_fact/
record_job_update_fact precompute a final state that can be silently
clobbered by a concurrent intervening fact once apply is deferred by
the commit round-trip.
Add 4 tests to test_commit_barrier_reply.c (25->29), no production
changes: explicit-FAILED via truncation (distinct from timeout),
repeated failure loops (5 cycles) for both timeout and truncation,
and an array_size>1 mixed-outcome test proving a committed sibling's
apply hand-off isn't blocked by failing siblings while producing
zero phantom jobs for the failures and exactly one batch-level
failure reply.
Closes out task-group 3 (command apply-after-commit ordering).
Replace handle_jobsubmit()'s blocking commit call (task 3.1) with a
real async registration against commit_submit_async(). On
FAILED/timeout, replies failure immediately with no job constructed.
On COMMITTED, hands off to a new pending-apply registration
(persist_register_pending_apply) for the ordered apply pump (task
6.1, not yet built) to consume -- does not apply or reply directly,
per design.
Deviates from design.md's stated integration site: appends via the
same ha_primary_log_append/persist_wal_async primitive the RETAIN-fact
path already uses and calls commit_submit_async directly on
p->repl.ce, bypassing repl_wait_majority_ack/repl_submit_majority_commit
in repl.c. Verified safe -- majority-ack detection is driven entirely
by (ce, lsn) events inside the commit engine, agnostic to which
wrapper registered the continuation -- but narrows the reusable
primitive design.md intended; documented in tasks.md.
Deletes (not preserves-dead) task 3.1's bug-1408 notification fix --
apply_committed_entry()'s WAL_JOB_SUBMIT case (task 2.2) already
fires the notification per-entry, immune to the batch-suppression
class bug-1408 fixed.
IMPORTANT: no live path applies a successfully-committed job
submission until task 6 lands (only restart replay does) -- every
successful submission commits durably but gets zero reply until
then. This commit range must not be soak-tested or deployed against
live traffic before task 6 completes.
Reorder handle_jobsubmit() so job objects are constructed via
create_job() only after the WAL commit succeeds, not before. Adds
jobid_range_exists() for a fast-fail pre-check with no WAL work, and
persist_submit_job_payloads_and_commit() (persist.c) operating on raw
xce_job_t* payloads instead of pre-existing job_t*. Commit remains
synchronous -- converting to the continuation-based commit API is
task 3.2.
Fixes bug-1408 (found during review): the post-commit array-
construction loop suppressed notification for an entire array batch
if any one sibling element failed to construct, stranding other
durably-committed and successfully-constructed jobs invisible until
the next restart's replay. Notification now fires unconditionally
for every successfully-constructed element, independent of sibling
failures; the client reply still separately reports the batch's
construction status.
Add test_apply_reapply_idempotency.c (3 tests, no production changes):
triple-reapplication of an identical job submit with hook-count and
pointer-identity assertions across every call, the NOOP_DUPLICATE
guard branch (untested by task 2.2), and the task's named scenario --
real SUBMIT then real FINISH then a stale out-of-order START, all
through the real apply_committed_entry() dispatch -- confirming the
job stays terminal with no spurious notification.
Closes out task-group 2 (shared apply seam for committed entries).
Rewire replay_wal()'s per-entry apply call from apply_wal_entry() to
apply_committed_entry() (task 2.2's shared dispatch seam), so restart
replay and the future live commit-advance pump (task 6.1) both
produce materialized state deterministically from the committed log.
apply_wal_entry() is not retired -- it remains ha.c's live standby
catch-up/replicated-apply dispatch, untouched.
Documents a real hazard: replay now fires AFTER_EVENT hooks for the
full post-checkpoint WAL range on every restart, including entries
already applied+notified in a prior process life. Traced and
confirmed inert today for both hook consumers (scheduler registers
post-boot, apiserver gates notify on a connected endpoint empty at
boot); flagged in tasks.md and in-line at the call site for
re-verification before tasks 4.1/6.1/6.2/6.3 make this path broadly
live.
Add apply_committed_entry(persist_t*, wal_type, lsn, payload, len) in
persist.c/persist.h as the single seam that turns a committed WAL
entry into an in-memory mutation plus its post-mutation notification
hook, covering job submit, job-state facts, queue control, and
worker control. Job-state facts route through 2.1's guard-only
apply_job_update_transition() core and fire a generic
HOOK_DETAIL_JOB_UPDATE, not per-type hooks — the wire format
(wal_jobstate_t {jobid, state}) cannot supply what the type-specific
cores/hooks require. This is a real, documented deviation from
design.md's per-type dispatch wording; tasks.md's Implementation
Notes flags it as unresolved and blocking for whoever wires this
into a live path (4.1/6.2/6.3).
Not yet wired into any live call site (replay wiring is 2.3).
Split mark_job_start/finish/fail/update in job.c into four hook-free
core functions (apply_job_*_transition) containing only the
monotonic-rank guard and mutation. Wrappers now peek the same guard
to decide hook-firing, then delegate to the core, preserving exact
pre-extraction hook-firing timing. Prerequisite for task 2.2's
apply_committed_entry() dispatch and the ordered apply pump (6.1).
Closes out task-group 1 (the commit-engine event-driven-resolve
keystone). Pure test-only: real pthreads racing the admission gate,
the sweep thread, and commit_on_ack, not sequential calls dressed as
concurrent. Adds the genuine two-real-thread ack-vs-sweep-thread race
both task-1.3 review rounds flagged as a nice-to-have (the existing
race test only proved the tie-break via single-thread white-box state
manipulation).
Also resolves the bug-1407 saga: task 1.4's "closed, 70/70" and this
task's initial "reopened, 10/10 failing" were both correct,
contradictory measurements of the identical source commit. Root cause
was a stale incremental-build artifact in the shared build/ directory
(ninja -C build rebuilt incrementally across 5 sequential tasks
without ever a fresh meson setup) -- not a source regression. Proven
by isolating each measurement in a throwaway git worktree with a
fresh build. bug-1407 is genuinely closed; the incremental-build
staleness gotcha is now documented in cerebrum.md for the remaining
task series.
commit_resolve_pending_above(ce, lsn_floor, rc) fails every registered
continuation for lsn > lsn_floor, reusing commit_resolve()'s existing
exactly-once/idempotent guarantee (no parallel bookkeeping path).
Truncation: a new on_truncate/truncate_cb_ctx callback on persist_t
(same inversion-of-control shape as the existing on_durable_advance
hook) fires from persist_truncate_wal_above after releasing p->lock,
only when entries were actually removed. This one wrapper covers all
three real truncation call sites: raft_cb_become_leader's
post-election truncate, the library's on_truncate_from
conflict-reconcile, and ha_demotion_convergence_fn's post-demotion
truncate.
Demotion: raft_cb_become_follower calls
commit_resolve_pending_above(&c->commit, 0, -XCE_ERR_STANDBY) --
lsn_floor=0 fails everything, since a demoted node can never
authoritatively confirm anything still pending (CP posture).
Without this, a demoted or truncated node would leave any pending
async command permanently unresolved -- the requester would simply
never get a reply, worse than today's bounded-timeout behavior.
Also closes bug-1407 (pre-existing test_config_repl flake, unrelated
to this feature) -- confirmed fixed as a side effect of prior tasks
in this series, 70/70 passing across 3 independent runs.
Converting COMMIT_MODE_WAIT away from a blocking wait removed the only
mechanism that ever detected a commit timing out (the caller's own
pthread_cond_timedwait deadline). Adds a dedicated per-commit-engine
sweep thread that arms an equivalent timeout for async-registered
continuations, resolving FAILED/-ETIMEDOUT within the same
DEFAULT_SYNC_TIMEOUT_MS window the old blocking path used, with no
thread parked to detect it.
Chose a private sweep thread over the shared delaymgr: wiring
delaymgr would touch cluster.c/ha.c (out of boundary) and its 200ms
poll granularity vs the 500ms default timeout would add jitter,
breaking the "identical failure timing" no-regression bar.
Round 1 review rejected a lost-wakeup race: the sweep loop computed
its next-deadline snapshot under lock, unlocked to dispatch, then
slept on the stale snapshot -- a registration landing mid-dispatch
had its broadcast lost (no thread parked on the condvar), so the
sweep thread could sleep on an absent/late deadline indefinitely.
Fixed by holding the lock continuously across scan-then-wait
decisions and always re-scanning immediately before ever waiting
(recheck-then-wait idiom). Round 2 review independently reproduced
the round-1 bug shape against the fix to confirm it actually closes
the race, not just narrows it.
commit_resolve() delivers a registered continuation's outcome exactly
once (idempotent re-resolve), dispatched off-thread via
vtask_enq_classified(VTASK_LIVE_BLOCK) so it never runs inline on the
replication-ACK or iomgr path. Wired into all 6 commit.c-internal
paths that can advance committed_lsn/writable_lsn.
Review caught a real task-graph gap, not a defect: converting
COMMIT_MODE_WAIT away from a blocking wait also removes the only
mechanism that ever detected a commit timing out (the caller's own
cond_timedwait deadline). Nothing yet arms an equivalent timeout for
async-registered continuations, and nothing wires demotion/log-truncation
detection to commit_resolve(FAILED, ...) -- left as-is, a pending async
command during quorum loss would hang forever instead of failing,
which is worse than today. Split into new tasks 1.3 (timeout arming)
and 1.4 (demotion/truncation wiring), renumbered the old concurrency
test to 1.5, and made 3.2 depend on both since its failure/timeout
observable is meaningless without them. design.md's File Structure
Plan updated to reflect that ha.c/persist.c need a narrow touch for
1.4's detection hooks.
commit_submit_async() lets a WAIT-mode caller register a continuation
against an appended LSN instead of blocking on pthread_cond_timedwait,
reusing the existing admission-cap discipline (wait_admit_locked/
wait_release_locked) so the bounded-capacity guarantee carries over
unchanged. Purely additive: no existing call site (repl_wait_majority_ack)
is touched yet -- that switch lands in task 3.2 once commit_resolve()
(task 1.2) exists to drive it.
Full Kiro spec for the apply-after-commit refactor: canonical
append->commit->apply->notify ordering, a single apply_committed_entry
seam shared by live commit-advance and restart replay, and the
Facts-Apply invariant (ha-consistency-invariants criterion 2.9)
reconciling worker-reported facts onto the same seam.
buglog.json had two broken auto-detection hook signatures polluting it:
- 207 entries "Missing error handling in X" / fix "Added try/catch block"
(nonsensical on .md/.capnp files, and C has no try/catch — never a real
applied fix)
- 452 entries with a literally-empty template ("Significant refactor of "
with no subject ever filled in) / fix "Rewrote N->M lines"
Removed both (659 total). Also merged 24 legitimate duplicate bug IDs
(re-logged/updated entries sharing an ID) into single entries — kept the
most recent content as primary, unioned tags/related_bugs, summed
occurrences — and stripped noise tags ('auto-detected', 'refactor') that
had leaked onto real entries from a since-merged junk duplicate. Net:
1406 -> 721 entries, verified zero remaining duplicate IDs and zero
remaining junk signatures.
.gitignore: added patterns for the scratch soak-run logs and codegraph.html
this session generated at repo root (already deleted from the tree), and
fixed the ucai-formatter-cache.local.json pattern to match its nested
occurrences (.claude/skills/*/.claude/, src/*/.claude/) instead of only the
repo-root one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
All 4 sub-specs are complete: ha-commit-engine (18/18), ha-commit-barrier-
rollout (23/23), ha-single-leader-role (17/17), ha-convergence-recovery
(19/19). Reconciles ha-single-leader-role and ha-convergence-recovery's
spec.json phase to "implemented" (both mislabeled tasks-generated despite
full task completion — same staleness pattern fixed for raft-commit-safety/
raft-log-replication earlier).
Promotes the umbrella itself to "implemented": design.approved corrected
true (was false, contradicting its own note "Design approved as contract"),
phase implemented, with an explicit callout that criterion 2.9 (Facts-Apply
invariant, added 2026-06-25) remains a documented deviation owned by the
still-unstarted raft-apply-decoupling spec — not a gap in this umbrella or
its sub-specs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
bug-847 (catch-up WAL batch drops payloads across capnp segment boundaries):
its purpose-built RED reproducer, test_wal_batch_roundtrip_large_payloads
(14 entries x 872-byte payloads, 12.2KB total, genuinely spans multiple
4096-byte capnp segments per capn-malloc.c's create()), was still
TEST_IGNORE_MESSAGE'd. Un-ignoring it and running under ASAN with leak
detection: clean pass, byte-exact payloads, zero leaks, 2/2 runs. Traced the
vendored c-capnproto far-pointer machinery (write_ptr/capn_setp in capn.c)
to confirm the cross-segment struct->Data reference path is correct by
construction — whatever regressed this already got fixed elsewhere (library
bump or an incidental request.c change); the ignore was just never removed.
No production code changed.
bug-1307 (task-5.7 election-jitter data-safety regression: 7 divergences +
4 job losses over 6 chaotic rounds): re-ran the exact reproducing scenario
(6-round chaotic soak, real ext4, ASAN binaries) — 21 jobs, 0 divergence,
0 job loss, clean PASS. Confirms the raft-ha-core-safety Phase 5 work
resolved it.
Both bugs were the last 2 of the 7 flagged in the prior staleness sweep;
all 7 are now closed (6 were already fixed and undetected, 1 needed the
test un-ignored).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Verified against live code/tests instead of trusting the buglog text:
- bug-846 (EINVAL on dead ctx): FIXED — xcecli_retry_decide checks ctx.state
== XCE_CTX_ERR before classifying, bypassing the misclassification.
- bug-855 (6 wal_catchup tests failing): FIXED — ran controller_unit_test,
all pass.
- bug-1096 (stale-term vote-reply counted): FIXED — raft.c has the
reply_term != s->term drop.
- bug-1147 (WAL trim ignores standby cursor): FIXED — trim_wal_segments has
the cursor-bound floor guard, comment cites this bug ID directly.
- bug-1198 (snapshot-send liveness-pool hazard): FIXED — in-code CLOSED
banner, bug-1197's endpoint ref/unref bracket makes it unreachable.
- bug-1307 (election-jitter safety regression): left OPEN with a downgrade
note — circumstantial evidence (raft-ha-core-safety Phase 5 soak-clean)
but no direct code citation found; wants one more confirmation soak
before closing.
- bug-847 (catch-up WAL drops payloads at scale): RE-CONFIRMED genuinely
still open — its RED reproducer is still TEST_IGNORE'd.
Also noted (not acted on, out of scope for this pass): buglog.json has 26
duplicate bug IDs and ~239 "Missing error handling in unknown" / "Added
try/catch block" auto-detected junk entries — a separate cleanup task.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Accidental npm-init artifacts from an earlier checkpoint commit (empty
package.json, auto-generated lockfile) in a pure C/Go project with no
Node.js build step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
ha-commit-durability: closes the "pending confirmation under simultaneous
total outage" caveat. Reproduced cold-restart-all (kill all 3 members +
restart) 5 times against a live 3-node cluster on genuine ext4 (not the
sandbox's overlayfs) — zero committed-job loss across 40 rolling-failover
rounds + 5 cold-restart-all cycles. Confirms bug-795 was an overlayfs
fdatasync artifact, not a real durability gap; no production fix needed.
Commit Durability is now proven for both rolling single-node failover and
simultaneous total-cluster outage (src/controller/CLAUDE.md updated).
Live-validated and marked complete against a real 3-node cluster:
- controller-status 6.4: off-primary committed fidelity + responder
timestamp, confirmed via a real failover
- ha-heartbeat-committed-propagation 4.1: standby committedLsn tracks the
primary via heartbeat within the 3s interval
- ha-wal-replication 5.4: standby WAL replication verified via committedLsn
match (job-facing ops are correctly primary-only by design; standbys
return E_XCE_STANDBY, not a bug)
- libxce-failover 6.2: live 3-node failover E2E, jobsub raced against a
primary kill resolves to exactly one job, no duplication
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Adds the Raft Apply-Ordering & Hook Decoupling roadmap phase: canonical
pre-hooks -> raft commit -> apply -> notify ordering, with the raft protocol
step lifted out of the notification hook fan-out (root of the bug-795/828/
851/1060 divergence family). Decision: apply-after-commit via a single
shared apply_committed_entry() seam reused by online apply and restart
replay.
- raft-drive-async: full spec docs (brief/requirements/research/design)
committed alongside its already-tracked tasks.md; async-drive prerequisite
for the apply-decoupling work, sequenced to start first (threatens
failover liveness today).
- raft-apply-decoupling: new spec (brief + facts-in-raft.md) covering the
apply seam and the facts (WAL_MODE_RETAIN) apply-timing question (open
decision D1).
- job-array-single-wal-record: promoted from backlog to a tracked spec,
sequenced after raft-apply-decoupling.
- ha-consistency-invariants: add criterion 2.9 (Facts-Apply invariant),
documenting the current on-receipt fact-reaction deviation this family
will reconcile.
- raft-commit-safety / raft-log-replication: reconcile spec.json phase to
"implemented" (both were mislabeled tasks-generated despite full 18/18
and 24/24 task completion + raft_sim GO).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
bug-1401: xce_free_ctrlstatus(s) was defined as free_CtrlStatus(s), which
only frees the fields inside xce_ctrlstatus_t. The real client reply path
(decode_CtrlStatus_ptr) calloc's the struct itself, so every ctrlstatus call
leaked sizeof(xce_ctrlstatus_t) (120 bytes). Existing unit tests never caught
it because they built/freed the struct via the raw stack-decode + bare
free_CtrlStatus() helpers, never exercising the actual client macro. Fixed
by switching the macro to free_CtrlStatus_ptr(&(s)) and added a regression
test that exercises the real heap-decode path; RED confirmed the exact
120-byte/1-alloc LeakSanitizer signature before the fix.
bug-1400: the originally reported SEGV in cmd_jobsub on malformed positional
input is stale — NULLSTR guards added 2025-11-05 already reject it cleanly.
The real residual defect was submit.start_info leaking on all 4 early-return
validation paths; fixed by routing them through a shared `goto cleanup`
(matches the existing goto-clean idiom in queue_ctrl).
bug-1396 (job-count anomaly after cold-restart-all, not yet root-caused):
instrumented test-commit-durability.sh's assert_job_count() to dump the live
joblist and ACKED_FILE ledger on any count mismatch instead of only the
count, per the bug's own recorded next step.
Verified: controller ASAN build + full controller_unit_test green; lib
xcelib_test 285/285 green, 0 ASAN leaks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
ha_send_snapshot_to_standby ignored the network send's return value in its
chunk loop, silently truncating a transfer on a mid-stream error while still
logging "complete" (bug-1386). Fixing the silent drop exposed a second,
independent bug in the same pipeline: ha_handle_snapshot_chunk appended
received chunks in arrival order instead of at their wire-carried
chunk_offset, so out-of-order delivery across the vtask pool corrupted the
reassembled file even though every chunk's bytes were individually correct
(bug-1389). Fixed by checking every fallible call in the send loop and
switching the receiver to pwrite-at-offset with a count-based completion
tracker.
Adds snapverify/snapdiff xce_test commands (offline checkpoint-file
structural validator and byte-level comparator) used to diagnose bug-1389 and
kept as standing diagnostic tools for future snapshot-transfer triage.
g_drive_async_configpush_offload flipped back ON after both fixes, validated
via 6-round chaotic + 3/3 force-snapshot soak (raft-drive-async task 3.3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Cross-references bug-1381 in tasks.md and the ha.c configpush flag comment.
No code behavior change -- push_config_transport_only's race (raw standby
endpoint pointers read/written by unprotected pthread_create workers,
independent of the offload flag) is a distinct, still-open blocker from
bug-1197/bug-1376 (both closed in the prior commit).
Root cause: peer_ctrl_msg_handler held ep->dispatch_inflight incremented
across the entire peer_ctrl_dispatch_to_pool call, including its inline
fallback (the common cold-start shape when ha->pool isn't wired yet). That
inline body can reach raft_cb_become_follower, which -- now that
cluster_init_phase2 ran synchronously earlier in the same call chain and
wired the pool -- tears down its own loser vote-ep (the very ep whose reply
triggered this call) via the deferred path (vote_ep_teardown_async), whose
endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that
SAME ep. That count is held only by this exact call, on this exact thread,
and can't reach 0 until the spin itself returns -- a permanent self-deadlock.
Confirmed live via a gdb thread backtrace: added a GDB_WRAP debug seam to
run-ha-soak.sh (launches a named node directly under `gdb --batch` so it
becomes ptrace-legal as gdb's own direct child under yama scope 1, bypassing
a live-attach restriction), then SIGINT to gdb once the log went stale for a
bounded interrupt + full backtrace. The frame dump showed pool=0x0 at
peer_ctrl_dispatch_to_pool, and the same ep pointer in the nested
vote_ep_teardown_async -> endpoint_mark_tearing_down frame, spinning on
sched_yield().
Fix: moved the dispatch_inflight release into peer_ctrl_dispatch_to_pool
itself, at the exact right point per exit path -- released BEFORE calling
the inline body on every fallback path (pool==NULL, OOM, enqueue failure),
since io_worker is single-threaded and the inline path never enqueues
anything (no pool-worker race left to protect against once inline execution
is committed to); released only AFTER the non-blocking enqueue completes on
the success path, preserving the original protection for the genuinely
concurrent pool-worker case.
New unit test test_vep_self_teardown_does_not_deadlock uses a bounded
pthread_timedjoin_np so a regression fails loud instead of hanging the
suite. RED confirmed via a temporary manual revert (ETIMEDOUT) before
restoring the fix.
Live-verified with both flags ON: --force-snapshot soak (the exact repro
shape) completes without hanging across repeated runs, confirms real
snapshot installs, zero divergence, zero job loss; a 6-round --chaotic soak
also passed clean; full controller suite green throughout.
g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled
now default ON in production, closing raft-drive-async tasks 3.1 and 3.2.
g_drive_async_configpush_offload stays OFF -- a distinct, still-open hazard
in push_config_transport_only's endpoint handling (task 3.3, unrelated to
this bug).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
Cross-references bug-1376 in tasks.md and the ha.c flag comment for the
unresolved hang found while investigating the raft-drive-async flag flip
(commit f3b6e6a). No code behavior change.
Both fixes are real and independently verified; the drive-async flags
(g_drive_async_snapshot_offload, g_drive_async_peerctrl_dispatch_enabled)
stay at their production-default OFF -- see the tasks.md session note for
why the flip itself doesn't land in this commit.
1. Snapshot-lane borrowed-endpoint hazard (task 3.2, bug-1197 family):
ha_snapshot_send_dispatch now takes endpoint_ref(ep) before submitting to
the slow lane, released on every non-lane terminal path and by
snapshot_lane_fn on completion. Reuses the general bug-1298 endpoint
refcount system (standby-ep teardown already routes through
endpoint_unref via cluster.c's endpoint_killer) -- a concurrent RDHUP
teardown can now only drop the owner's reference while a transfer is in
flight, never free the ep out from under it. New test:
test_snapshot_offload_holds_endpoint_ref_across_transfer (RED->GREEN).
2. Catch-up streamer WAL_CHECKPOINT bug (found while verifying #1 via a live
soak, pre-existing, unrelated to the drive-async flags): ha_stream_wal_
to_standby had no filter for WAL_CHECKPOINT markers -- local-only
bookkeeping that still consumes a real LSN slot via persist_write_
checkpoint_marker's own current_lsn++. It shipped one to a catching-up
standby, which correctly refused it (persist_wal_write_replicated
rejects type==WAL_CHECKPOINT) and got stuck retrying the same LSN
forever. Fixed by skipping WAL_CHECKPOINT entirely in the streamer loop,
mirroring the existing replay_wal/term-index precedent of treating
checkpoint-marker LSNs as invisible gaps. New test:
test_wal_stream_skips_checkpoint_marker (RED->GREEN).
Attempted flipping both drive-async flags ON after landing these fixes (the
"full fix, defer prod flip" decision's prerequisite -- 4.1+4.2 are green).
A 6-round --chaotic soak passed clean, but a --force-snapshot soak (a
genuinely empty node cold-start rejoin, the one scenario chaotic mode never
exercises) surfaced a THIRD, not-yet-root-caused issue: the freshly-wiped
node's own self-election appears to race cluster_init_phase2's vtask-pool
wiring, then the process hangs silently (no crash, no ASAN report) until
killed. Reverted both flags to OFF rather than ship an unproven state --
production behavior is unchanged by this commit. Full controller suite
green and both soaks (--force-snapshot, 6-round --chaotic) pass clean with
flags OFF, confirming no regression from the two logic fixes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
Chain of fixes needed to get --force-snapshot soak to a real live PASS
(raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1):
1. WAL segment rotation was gated on a hardcoded 64MB compile-time
constant; trim_wal_segments() never deletes the active (currently-open)
segment. At soak job counts a segment never rotated, so nothing was
ever physically trimmable and ha_evaluate_repl_action always chose
STREAM_WAL over SEND_SNAPSHOT for an empty-rejoining node. Fixed by
threading a wal_segment_bytes override through persist_config_t
(mirrors the --max-wal-records pattern: config.h default, persist.h
setter, persist.c 3 usage sites + minimum-floor enforcement, 6-point
CLI wiring through main.c/myconf.h/myapp.c/cluster.c) plus a new
--wal-segment-bytes CLI flag. run-ha-soak.sh's --force-snapshot mode
now sets WAL_SEGMENT_BYTES=4096 to force real rotation at soak scale.
2. This unmasked a genuine heap-use-after-free in INVOKE_MSG_HANDLER
(include/netio.h): the macro calls ep->msg_handler(ep) synchronously
(not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing
doesn't cover it) then writes ep->fsm_in.pkt = NULL. A stale vote-reply
handler chain (raft_on_vote_reply -> raft_cb_become_follower ->
endpoint_unref) could drop the endpoint's last ref and free it inside
the handler call, so the trailing write hit freed memory. This is the
first-ever live exercise of a node receiving a snapshot transfer while
mid-election. Fixed by holding a ref across the synchronous call, same
pattern task 5.8 established for the async path.
With both fixes: --force-snapshot soak now PASSES live (committed
divergence=0, job_loss=0, confirmed real "HA: snapshot" install on the
restarted node) — first genuine live proof of this scenario. 3-round
--chaotic regression clean. Full 217-test controller suite green
before/after every fix in the chain.
Builds on the delaymgr starvation fix (b9ad018) and walcompare
checkpoint-floor fix (f3c86a6) from earlier in this investigation.
Root-caused via raft-ha-core-safety 5.10a investigation: schedule_checkpoint()
(persist.c) fires correctly once wal_record_count crosses its threshold, but
its delay_task()-scheduled checkpoint_task_fn never executed in a live
multi-node deployment -- confirmed live via temporary diagnostic logging
(wal_record_count=100 max=100 logged, but the task itself never ran).
Root cause: delay_task_t carried TWO incomparable time representations --
`nextrun` (wall-clock time_t seconds, used by delay_task()) and `nextrun_ts`
(CLOCK_MONOTONIC timespec, used by delay_task_ms()). task_cmp()'s mixed-type
fallback compared them directly: an ms-task's nextrun is always 0, a
seconds-task's nextrun is a real epoch value (e.g. 1785600123) -- so 0
always sorted "less than" any seconds-task, regardless of actual due time.
ordq (src/lib/ordq.c) keeps its queue as an ascending sorted linked list by
this comparator, and delay_worker() only ever inspects the list HEAD
(ordq_peek with no matchfn returns q->head immediately, no scanning). Since
cluster_t.delaymgr is shared with ha.c's ~10 recurring delay_task_ms() call
sites (heartbeats, election timers, RDHUP polling), the head was
structurally almost-always occupied by an ms-task -- any delay_task()
(seconds) caller sharing that delaymgr, including schedule_checkpoint's
checkpoint_task_fn, could be starved indefinitely.
Fix: eliminate the dual representation entirely. Every task -- whether
scheduled via delay_task() (seconds) or delay_task_ms() (milliseconds) --
now stores one flat uint64_t CLOCK_MONOTONIC millisecond deadline
(nextrun_ms). delay_task() becomes a thin seconds->ms wrapper around
delay_task_ms(). task_cmp() and task_due() collapse to a single integer
comparison; delay_worker()'s reschedule is a flat integer add (no more
tv_sec/tv_nsec carry arithmetic, no more is_ms_task branching). Net -42
lines. Live-verified: after this fix, a forced checkpoint round produces a
real installed snapshot file (previously: schedule_checkpoint fired but the
task silently never ran).
controller_unit_test: full 58-group sweep, 0 Failures, 0 ASAN reports
(before and after this change) -- no regression, including every
delaymgr-touching group (test_demotion_convergence, test_ha_start_election_async,
test_ha_core_safety, test_ha_startup, etc).
