ha-commit-durability: closes the "pending confirmation under simultaneous
total outage" caveat. Reproduced cold-restart-all (kill all 3 members +
restart) 5 times against a live 3-node cluster on genuine ext4 (not the
sandbox's overlayfs) — zero committed-job loss across 40 rolling-failover
rounds + 5 cold-restart-all cycles. Confirms bug-795 was an overlayfs
fdatasync artifact, not a real durability gap; no production fix needed.
Commit Durability is now proven for both rolling single-node failover and
simultaneous total-cluster outage (src/controller/CLAUDE.md updated).
Live-validated and marked complete against a real 3-node cluster:
- controller-status 6.4: off-primary committed fidelity + responder
timestamp, confirmed via a real failover
- ha-heartbeat-committed-propagation 4.1: standby committedLsn tracks the
primary via heartbeat within the 3s interval
- ha-wal-replication 5.4: standby WAL replication verified via committedLsn
match (job-facing ops are correctly primary-only by design; standbys
return E_XCE_STANDBY, not a bug)
- libxce-failover 6.2: live 3-node failover E2E, jobsub raced against a
primary kill resolves to exactly one job, no duplication
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
bug-1401: xce_free_ctrlstatus(s) was defined as free_CtrlStatus(s), which
only frees the fields inside xce_ctrlstatus_t. The real client reply path
(decode_CtrlStatus_ptr) calloc's the struct itself, so every ctrlstatus call
leaked sizeof(xce_ctrlstatus_t) (120 bytes). Existing unit tests never caught
it because they built/freed the struct via the raw stack-decode + bare
free_CtrlStatus() helpers, never exercising the actual client macro. Fixed
by switching the macro to free_CtrlStatus_ptr(&(s)) and added a regression
test that exercises the real heap-decode path; RED confirmed the exact
120-byte/1-alloc LeakSanitizer signature before the fix.
bug-1400: the originally reported SEGV in cmd_jobsub on malformed positional
input is stale — NULLSTR guards added 2025-11-05 already reject it cleanly.
The real residual defect was submit.start_info leaking on all 4 early-return
validation paths; fixed by routing them through a shared `goto cleanup`
(matches the existing goto-clean idiom in queue_ctrl).
bug-1396 (job-count anomaly after cold-restart-all, not yet root-caused):
instrumented test-commit-durability.sh's assert_job_count() to dump the live
joblist and ACKED_FILE ledger on any count mismatch instead of only the
count, per the bug's own recorded next step.
Verified: controller ASAN build + full controller_unit_test green; lib
xcelib_test 285/285 green, 0 ASAN leaks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
ha_send_snapshot_to_standby ignored the network send's return value in its
chunk loop, silently truncating a transfer on a mid-stream error while still
logging "complete" (bug-1386). Fixing the silent drop exposed a second,
independent bug in the same pipeline: ha_handle_snapshot_chunk appended
received chunks in arrival order instead of at their wire-carried
chunk_offset, so out-of-order delivery across the vtask pool corrupted the
reassembled file even though every chunk's bytes were individually correct
(bug-1389). Fixed by checking every fallible call in the send loop and
switching the receiver to pwrite-at-offset with a count-based completion
tracker.
Adds snapverify/snapdiff xce_test commands (offline checkpoint-file
structural validator and byte-level comparator) used to diagnose bug-1389 and
kept as standing diagnostic tools for future snapshot-transfer triage.
g_drive_async_configpush_offload flipped back ON after both fixes, validated
via 6-round chaotic + 3/3 force-snapshot soak (raft-drive-async task 3.3).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Cross-references bug-1381 in tasks.md and the ha.c configpush flag comment.
No code behavior change -- push_config_transport_only's race (raw standby
endpoint pointers read/written by unprotected pthread_create workers,
independent of the offload flag) is a distinct, still-open blocker from
bug-1197/bug-1376 (both closed in the prior commit).
Root cause: peer_ctrl_msg_handler held ep->dispatch_inflight incremented
across the entire peer_ctrl_dispatch_to_pool call, including its inline
fallback (the common cold-start shape when ha->pool isn't wired yet). That
inline body can reach raft_cb_become_follower, which -- now that
cluster_init_phase2 ran synchronously earlier in the same call chain and
wired the pool -- tears down its own loser vote-ep (the very ep whose reply
triggered this call) via the deferred path (vote_ep_teardown_async), whose
endpoint_mark_tearing_down spins waiting for dispatch_inflight==0 on that
SAME ep. That count is held only by this exact call, on this exact thread,
and can't reach 0 until the spin itself returns -- a permanent self-deadlock.
Confirmed live via a gdb thread backtrace: added a GDB_WRAP debug seam to
run-ha-soak.sh (launches a named node directly under `gdb --batch` so it
becomes ptrace-legal as gdb's own direct child under yama scope 1, bypassing
a live-attach restriction), then SIGINT to gdb once the log went stale for a
bounded interrupt + full backtrace. The frame dump showed pool=0x0 at
peer_ctrl_dispatch_to_pool, and the same ep pointer in the nested
vote_ep_teardown_async -> endpoint_mark_tearing_down frame, spinning on
sched_yield().
Fix: moved the dispatch_inflight release into peer_ctrl_dispatch_to_pool
itself, at the exact right point per exit path -- released BEFORE calling
the inline body on every fallback path (pool==NULL, OOM, enqueue failure),
since io_worker is single-threaded and the inline path never enqueues
anything (no pool-worker race left to protect against once inline execution
is committed to); released only AFTER the non-blocking enqueue completes on
the success path, preserving the original protection for the genuinely
concurrent pool-worker case.
New unit test test_vep_self_teardown_does_not_deadlock uses a bounded
pthread_timedjoin_np so a regression fails loud instead of hanging the
suite. RED confirmed via a temporary manual revert (ETIMEDOUT) before
restoring the fix.
Live-verified with both flags ON: --force-snapshot soak (the exact repro
shape) completes without hanging across repeated runs, confirms real
snapshot installs, zero divergence, zero job loss; a 6-round --chaotic soak
also passed clean; full controller suite green throughout.
g_drive_async_snapshot_offload and g_drive_async_peerctrl_dispatch_enabled
now default ON in production, closing raft-drive-async tasks 3.1 and 3.2.
g_drive_async_configpush_offload stays OFF -- a distinct, still-open hazard
in push_config_transport_only's endpoint handling (task 3.3, unrelated to
this bug).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
Cross-references bug-1376 in tasks.md and the ha.c flag comment for the
unresolved hang found while investigating the raft-drive-async flag flip
(commit f3b6e6a). No code behavior change.
Both fixes are real and independently verified; the drive-async flags
(g_drive_async_snapshot_offload, g_drive_async_peerctrl_dispatch_enabled)
stay at their production-default OFF -- see the tasks.md session note for
why the flip itself doesn't land in this commit.
1. Snapshot-lane borrowed-endpoint hazard (task 3.2, bug-1197 family):
ha_snapshot_send_dispatch now takes endpoint_ref(ep) before submitting to
the slow lane, released on every non-lane terminal path and by
snapshot_lane_fn on completion. Reuses the general bug-1298 endpoint
refcount system (standby-ep teardown already routes through
endpoint_unref via cluster.c's endpoint_killer) -- a concurrent RDHUP
teardown can now only drop the owner's reference while a transfer is in
flight, never free the ep out from under it. New test:
test_snapshot_offload_holds_endpoint_ref_across_transfer (RED->GREEN).
2. Catch-up streamer WAL_CHECKPOINT bug (found while verifying #1 via a live
soak, pre-existing, unrelated to the drive-async flags): ha_stream_wal_
to_standby had no filter for WAL_CHECKPOINT markers -- local-only
bookkeeping that still consumes a real LSN slot via persist_write_
checkpoint_marker's own current_lsn++. It shipped one to a catching-up
standby, which correctly refused it (persist_wal_write_replicated
rejects type==WAL_CHECKPOINT) and got stuck retrying the same LSN
forever. Fixed by skipping WAL_CHECKPOINT entirely in the streamer loop,
mirroring the existing replay_wal/term-index precedent of treating
checkpoint-marker LSNs as invisible gaps. New test:
test_wal_stream_skips_checkpoint_marker (RED->GREEN).
Attempted flipping both drive-async flags ON after landing these fixes (the
"full fix, defer prod flip" decision's prerequisite -- 4.1+4.2 are green).
A 6-round --chaotic soak passed clean, but a --force-snapshot soak (a
genuinely empty node cold-start rejoin, the one scenario chaotic mode never
exercises) surfaced a THIRD, not-yet-root-caused issue: the freshly-wiped
node's own self-election appears to race cluster_init_phase2's vtask-pool
wiring, then the process hangs silently (no crash, no ASAN report) until
killed. Reverted both flags to OFF rather than ship an unproven state --
production behavior is unchanged by this commit. Full controller suite
green and both soaks (--force-snapshot, 6-round --chaotic) pass clean with
flags OFF, confirming no regression from the two logic fixes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014dZVSx3CdBjys9BB6hqpxP
Chain of fixes needed to get --force-snapshot soak to a real live PASS
(raft-ha-core-safety task 5.10a / ha-entity-snapshots task 7.1):
1. WAL segment rotation was gated on a hardcoded 64MB compile-time
constant; trim_wal_segments() never deletes the active (currently-open)
segment. At soak job counts a segment never rotated, so nothing was
ever physically trimmable and ha_evaluate_repl_action always chose
STREAM_WAL over SEND_SNAPSHOT for an empty-rejoining node. Fixed by
threading a wal_segment_bytes override through persist_config_t
(mirrors the --max-wal-records pattern: config.h default, persist.h
setter, persist.c 3 usage sites + minimum-floor enforcement, 6-point
CLI wiring through main.c/myconf.h/myapp.c/cluster.c) plus a new
--wal-segment-bytes CLI flag. run-ha-soak.sh's --force-snapshot mode
now sets WAL_SEGMENT_BYTES=4096 to force real rotation at soak scale.
2. This unmasked a genuine heap-use-after-free in INVOKE_MSG_HANDLER
(include/netio.h): the macro calls ep->msg_handler(ep) synchronously
(not via the vtask dispatcher, so the bug-1298 vtask ref/unref pairing
doesn't cover it) then writes ep->fsm_in.pkt = NULL. A stale vote-reply
handler chain (raft_on_vote_reply -> raft_cb_become_follower ->
endpoint_unref) could drop the endpoint's last ref and free it inside
the handler call, so the trailing write hit freed memory. This is the
first-ever live exercise of a node receiving a snapshot transfer while
mid-election. Fixed by holding a ref across the synchronous call, same
pattern task 5.8 established for the async path.
With both fixes: --force-snapshot soak now PASSES live (committed
divergence=0, job_loss=0, confirmed real "HA: snapshot" install on the
restarted node) — first genuine live proof of this scenario. 3-round
--chaotic regression clean. Full 217-test controller suite green
before/after every fix in the chain.
Builds on the delaymgr starvation fix (b9ad018) and walcompare
checkpoint-floor fix (f3c86a6) from earlier in this investigation.
Root-caused via raft-ha-core-safety 5.10a investigation: schedule_checkpoint()
(persist.c) fires correctly once wal_record_count crosses its threshold, but
its delay_task()-scheduled checkpoint_task_fn never executed in a live
multi-node deployment -- confirmed live via temporary diagnostic logging
(wal_record_count=100 max=100 logged, but the task itself never ran).
Root cause: delay_task_t carried TWO incomparable time representations --
`nextrun` (wall-clock time_t seconds, used by delay_task()) and `nextrun_ts`
(CLOCK_MONOTONIC timespec, used by delay_task_ms()). task_cmp()'s mixed-type
fallback compared them directly: an ms-task's nextrun is always 0, a
seconds-task's nextrun is a real epoch value (e.g. 1785600123) -- so 0
always sorted "less than" any seconds-task, regardless of actual due time.
ordq (src/lib/ordq.c) keeps its queue as an ascending sorted linked list by
this comparator, and delay_worker() only ever inspects the list HEAD
(ordq_peek with no matchfn returns q->head immediately, no scanning). Since
cluster_t.delaymgr is shared with ha.c's ~10 recurring delay_task_ms() call
sites (heartbeats, election timers, RDHUP polling), the head was
structurally almost-always occupied by an ms-task -- any delay_task()
(seconds) caller sharing that delaymgr, including schedule_checkpoint's
checkpoint_task_fn, could be starved indefinitely.
Fix: eliminate the dual representation entirely. Every task -- whether
scheduled via delay_task() (seconds) or delay_task_ms() (milliseconds) --
now stores one flat uint64_t CLOCK_MONOTONIC millisecond deadline
(nextrun_ms). delay_task() becomes a thin seconds->ms wrapper around
delay_task_ms(). task_cmp() and task_due() collapse to a single integer
comparison; delay_worker()'s reschedule is a flat integer add (no more
tv_sec/tv_nsec carry arithmetic, no more is_ms_task branching). Net -42
lines. Live-verified: after this fix, a forced checkpoint round produces a
real installed snapshot file (previously: schedule_checkpoint fired but the
task silently never ran).
controller_unit_test: full 58-group sweep, 0 Failures, 0 ASAN reports
(before and after this change) -- no regression, including every
delaymgr-touching group (test_demotion_convergence, test_ha_start_election_async,
test_ha_core_safety, test_ha_startup, etc).
Adds --max-wal-records (mirrors the existing --wal-sync-ms pattern:
main.c argp option -> myconf_t -> persist_config_set_max_wal_records())
so a deployment can force the WAL-record-count checkpoint trigger to
fire on a job count a soak can actually reach, instead of the
production default of 10000.
Adds a new `--force-snapshot` soak mode (run-ha-soak.sh): submits
enough jobs to cross the configured threshold, kills a non-primary
node, wipes its data dir (a genuine empty rejoin), restarts it, and
asserts the cluster reconverges with zero walcompare divergence and
zero job loss. Fails loud (not a false PASS) if the restarted node's
log shows no actual snapshot install, so the round can't silently
report success without exercising the real scenario.
INVESTIGATION FINDING (not yet resolved, this commit does not close
5.10a or ha-entity-snapshots 7.1): live-tested with temporary
diagnostic logging (since reverted). schedule_checkpoint() correctly
fires once wal_record_count reaches the configured threshold
(confirmed: "wal_record_count=100 max=100" logged), but the delaymgr-
scheduled checkpoint_task_fn never executes afterward -- no checkpoint
ever completes, no snapshot file is ever written, in a live multi-node
deployment. This is a separate, deeper issue in the delaymgr scheduling
path (or its wiring for this specific call shape), not a bug in this
commit's CLI plumbing or soak scaffolding -- both are built, tested,
and confirmed working up to the point where the async checkpoint
should run. Needs its own dedicated investigation (likely src/lib
delaymgr, not persist.c) before --force-snapshot can produce a PASS.
This also has production implications beyond the test scenario: if
confirmed, WAL checkpoints may never fire automatically today, meaning
WAL segments are never trimmed via this path in a live deployment.
controller_unit_test: test_ha_startup 34/0, test_ha_core_safety 28/0,
test_persist_checkpoint 5/0, test_walcompare 17/0 (no regression).
Soak --selftest 17/17.
walcompare previously had no concept of a per-member compaction floor:
committed_lsn only capped the HIGH end (skip LSNs a member hasn't
caught up to yet), but the LOW end unconditionally expected every
member to hold raw WAL from LSN 1. A member that legitimately caught
up via a full-checkpoint snapshot holds no raw WAL at or below its own
checkpoint LSN by design (that range was entity-restored, not
WAL-replayed) -- so it was always flagged MISSING/divergent there,
regardless of whether the restore was correct. This is the root cause
ha-entity-snapshots task 7.1 diagnosed (committed_divergence=11 on an
otherwise-healthy soak) and raft-ha-core-safety task 5.10a's residual.
Adds member_snapshot_floor(): each member's own snap/ sibling
directory (matching the deployed <data>/wal + <data>/snap layout) is
scanned for the highest installed snap-<lsn>.{capnp,bin} file. Any LSN
at or below that member's own floor is excluded from its comparison
the same way an LSN above its own committed_lsn already is -- never
uniform/shared across members, since compaction is a per-member fact.
wc_status() gains a new "compacted" token distinct from "MISSING" so
the machine-readable report can tell the two apart.
Two new regression tests prove: (1) the floor correctly suppresses the
compacted range from comparison, and (2) it does NOT mask a genuine
divergence within the range a member still claims to hold. Caught and
fixed one bug myself before landing: the floor comparison was
off-by-one (< instead of <=), which the first test run surfaced
immediately (Expected 0 Was 1).
test_walcompare 17/17, test_ha_core_safety 28/0, test_config_repl
66/0, test_waldump 7/0 (all no-regression). Soak --selftest 17/17.
A full 69-group sweep of controller_unit_test (this spec's harness had
never been run end-to-end before) found 2 deterministic failures,
both test-only staleness against already-shipped, already-approved
production behavior from earlier in this same spec:
- test_ha_vote_gate: test_pre_vote_catch_up_denies asserted the
pre-task-5.3 unconditional in_catch_up PreVote deny. Task 5.3
(bf88681) deliberately gated that deny on primary_ctrl_fd>=0 &&
ha_heartbeat_is_fresh to fix a real round-4 election-stall — this
test's exact scenario (stale heartbeat, primary_ctrl_fd=-1) is the
case 5.3 intentionally flipped to grant. Split into
test_pre_vote_catch_up_grants_when_primary_unreachable (matches the
current contract) and test_pre_vote_catch_up_denies_while_primary_live
(preserves the original protective intent, correctly scoped).
- test_egress_filter: test_live_submit_tags_submit_lsn called the
JOB_NEW persist_job_hook_fun AFTER_EVENT hook directly, but task 5.9
(15be5d9, bug-1306 fix) made that hook's WAL_MODE_WAIT branch
notify-only, moving the real WAL append + commit barrier into
persist_submit_wal_and_commit(). 5.9 relocated 4 equivalent tests in
test_persist_hooks.c but missed this one. Now drives
persist_submit_wal_and_commit() directly, mirroring that pattern.
No production code changed. Full 69-group suite: 0 Failures, 0 ASAN
reports (independently reproduced 3x: initial sweep, implementer,
reviewer).
The term-blind ha_apply_single_entry direct-persist branch (carried
forward from task 2.4) is unreachable with a real non-zero term in any
current production path: its only trigger, raft_send_append_entries,
has zero production call sites (library simulator only). Both real
senders (repl_broadcast, ha_stream_wal_to_standby) always emit a
non-zero term for a present LSN. Documents this in ha.c and design.md
rather than structurally retiring a dormant path (YAGNI), with an
explicit flag for future re-evaluation if a heartbeat-driver task ever
wires up raft_send_append_entries.
Adds two regression pins: restart preserves (term,lsn) identity
(replay_wal already reconstructs the term index via record_entry_term),
and real senders never produce the term-blind wire shape.
test_ha_core_safety 28/0, no regression, no ASAN reports.
Integration test proves the guarantee tasks 2.3/2.4 hardened in
isolation (synthetic fault injection on the streamer) also holds
end-to-end over two real persist_t/WAL instances through the real
production ingest seam (ha_apply_wal_catchup_entry). Covers live
in-term replication and cross-term catch-up: no term==0 sourced for a
live entry, no non-empty entry ever lands as PLEN=0, standby persists
the entry's stored term (not the AE header's leader-current-term,
the bug-1067 split), and walcompare reports a clean committed prefix.
test_ha_core_safety 26/0, no regression, no ASAN reports.
Adds checkpointTerm (@6, append-only) to the SnapshotTransfer wire
message. The leader stamps it via persist_entry_term(p, checkpoint_lsn)
when sending a snapshot; the standby reconciles it into its own
per-LSN term index (persist_reconcile_checkpoint_term) right after
persist_load_snapshot succeeds, before applied_lsn advances. This
authoritatively overwrites any stale local term a divergent
pre-snapshot entry left at the checkpoint LSN, closing the gap task
3.1's RED test proved (Expected 7 Was 3 -> now passes). Transport
stays persist_t-free; the reconcile lives in persist.c per the
existing dependency direction. Decode is auto-generated via capnp
mapname; only the hand-rolled encode path (request.c/typemap.h, needed
for the raw snapshot payload) required a matching manual update.
test_ha_core_safety 25/0, test_config_repl 66/0, no ASAN reports.
ha_handle_snapshot_chunk/persist_load_snapshot advance checkpoint_lsn,
current_lsn, and applied_lsn on snapshot install but never touch the
per-LSN term index, so a divergent pre-snapshot entry's stale local
term survives the install unreconciled. New test proves the gap
(Expected 7 Was 3); fix lands in task 3.2.
persist_recover's local snapshot load passed NULL for out_entities_restored,
so a rolling-upgrade restart against a pre-feature checkpoint had no
observability parity with the wire receive path's (task 6.1) INFO log.
Mirror the same conditional logging here; no behavior change, WAL replay
in Step 3 still applies unconditionally either way.
Live-verified in a 3-node cluster: planted a hand-crafted 16-byte legacy
sentinel as ctrl-1's own checkpoint file, restarted all 3 — no crash, INFO
log fired, ctrl-1 was elected primary serving the full 37-job list, and
walcompare confirmed all 3 members' committed prefixes (LSN 1..49) are
byte-identical.
ha_handle_snapshot_chunk now logs whether a received checkpoint was
fully entity-restored or was a legacy position-only file (INFO level,
framed as an expected rollout state -- never an error, per design.md).
The rc-gates-applied_lsn control flow is byte-for-byte unchanged from
task 5.2 (confirmed by review) -- this task is purely additive
observability, matching design.md's own note that this call site
"needs no change" once persist_load_snapshot's gate is correct.
New test drives the legacy path end-to-end through the real production
handler (rc==0, file renamed, applied_lsn advances to the legacy LSN).
This completes the integration layer -- write path (jobs+queues),
restore-before-advance read path, and the production wiring are all in
place and independently reviewed. Only validation tasks remain (7.1
chaotic soak, 8.1 rollback compatibility).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Replaces the naive task-5.1 stopgap with the real restore contract:
detects legacy vs entity-bearing checkpoints by remaining file size;
restores every job then every queue status via restore_job/
restore_queue_state, in file order, with bounds-checked length-prefixed
reads; only after every entity restores successfully does
current_lsn/checkpoint_lsn commit to the checkpoint's position. Every
failure path (bounds check, decode, restore error, unrecognized
format_version) returns before that single commit point -- position is
provably untouched on failure, never partially advanced.
Also fixes the pre-existing restore_job decode leak (job.c): job_init
deep-copies submit (never nulled) and MOVE_PTRs allocs/pidinfo/
pend_reasons (nulled on the one path that runs) -- free_Job(&j) is safe
unconditionally on every reachable return path (EEXIST early-return,
create_job success, create_job failure), verified against the exact
generated free_*_ptr null-check guards.
Two pre-existing test_config_repl.c wire-protocol tests needed their
synthetic garbage snapshot bodies updated to a well-formed empty-entity
tail, since load_snapshot now genuinely parses past the legacy header
(previously ignored); their actual assertions (chunk-reassembly
correctness) are unchanged.
This is the exact function/contract that caused task 2.2's rejection
(position advancing before restore validated). Independent review
traced the memory-safety fix through the generated capnp code field-by-
field, confirmed the test_config_repl.c change doesn't weaken any
assertion, and confirmed restore_queue_state's ENOENT-aborts-the-whole-
checkpoint behavior is faithful to design.md's unconditional contract
(flagged as a watch-item for task 7.1's soak, not a defect).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Extends persist_load_snapshot's public signature with an
out_entities_restored bool*, per design.md's documented contract. The
static load_snapshot function -- the real restore logic -- is completely
untouched (still today's watermark-only implementation, byte-identical
to before this task); only a thin wrapper was added that naively maps
*out_entities_restored = (rc==0), deliberately scaffolding so the 3 new
RED tests fail for genuine reasons rather than accidentally passing.
ha_handle_snapshot_chunk's sole call site gets a mechanical
declare-and-pass update -- confirmed inert, never reads the new
out-param (task 6.1's job).
Three tests against persist_load_snapshot directly (unlike prior tasks'
tests, which deliberately stayed on the raw-byte-parsing side to avoid
this exact function): restore-before-advance (fails at the job-lookup
assertion -- nothing is ever restored today), corrupted-entity-leaves-
position-unchanged (fails at rc!=0 -- today's code ignores the corrupted
section entirely, since load_snapshot never reads past the legacy
header), legacy-file-reports-position-only (fails at entities_restored==
false -- the naive wrapper claims restoration for every rc==0 case,
including legacy files).
This is the exact function/call site that caused task 2.2's rejection
(a live position-advances-before-restore-validates hazard). Independent
review verified load_snapshot's body is byte-identical to commit
d52eb94, the wrapper does nothing beyond the naive mapping, and ha.c's
change never reads the new out-param anywhere -- no repeat of that
mistake here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
rotate_snapshots now parses each candidate file's LSN and skips deletion
via a new snapshot_lsn_in_flight() check against
p->cluster->ha.snapshot_inflight_lsn[] (task 4.1) before unlinking --
protected files are retried on the next rotation pass once the transfer
completes and the LSN clears.
Checks LSN directly (nonzero + matches) rather than gating on
snapshot_inflight[]==1 first, per task 4.1's own note: the LSN clears
before the flag in every completion/failure path, so a stale flag=1 can
momentarily coexist with an already-cleared LSN=0. A checkpoint LSN is
never 0, so this is race-free without needing the flag at all.
p->cluster==NULL (standalone lib-level tests) means no HA layer exists to
protect anything.
Task 4.2's RED rotation-guard test now passes; full regression sweep
(test_persist_checkpoint, test_persist_hooks, test_commit_barrier_reply,
test_commit_durability, test_e2e_hooks, test_hooks, test_config_repl,
test_ha_liveness) all clean. Both controller_unit_test and production
xce_controller build clean; nm confirms only the real production helper
(snapshot_lsn_in_flight) appears, no test-only symbols leaked in.
Executed directly (no subagent dispatch) per the hybrid execution-mode
decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
A checkpoint file whose position matches an in-flight peer transfer
(task 4.1's snapshot_inflight_lsn[]) must survive rotate_snapshots even
when snap_keep_n would otherwise delete it. Genuinely fails today (file
removed, access() returns -1) since rotate_snapshots has no awareness of
in-flight transfers yet.
Executed directly (no subagent dispatch) per the hybrid execution-mode
decision -- small, single-file, isolated test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Adds snapshot_inflight_lsn[MAX_HA_CANDIDATES] to ha_state_t, alongside the
existing snapshot_inflight[] flag it mirrors exactly: set to the
checkpoint's LSN at ha_snapshot_send_dispatch before a transfer's first
byte goes out, cleared to 0 at every completion/failure path
(snapshot_lane_fn's normal completion, and all 3 early-return failure
sites: malloc failure, slowlane dedup-refuse, slowlane submit failure).
This is the foundation task 4.3 will consult from rotate_snapshots to
avoid deleting a checkpoint file a peer is still mid-transfer reading
from -- the one genuinely new retention guard this feature needs (the
existing write_snapshot-before-trim_wal_segments sequencing already
satisfies the WAL-trim side of Requirement 3).
Proven via a new gated unit test (test_ha_liveness.c) that observes the
tracked LSN mid-flight while the send body is parked at a test gate, not
just before/after -- confirms the lifecycle ordering, not just the
endpoints.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
write_snapshot replaces the queue_count=0 placeholder with a real
collect_queue_frames helper: enumerates every queue in the live table
under nested table+per-entity RTTI_LOCK_READ (no retention filter --
table presence already implies config-provisioned existence, unlike
jobs), writes each queue's (name, status) as a plain fixed record (no
capnp -- matches restore_queue_state's exact contract). Any failure
anywhere still discards the whole checkpoint, including any
already-collected job frames, no partial file at the final path.
Entirely contained to persist.c -- load_snapshot and persist.h remain
byte-identical to before this task, honoring the boundary lesson task 2.2
established (queue restoration stays task 5.2's scope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Queue's (name, status) round-trip through write_snapshot's raw on-disk
bytes -- a plain fixed record (no capnp; queue definitions are
config-owned, only runtime admin status is WAL/snapshot-tracked, per
design.md's corrected queue model). Fails today (Expected 4 Was 0) since
write_snapshot still hardcodes queue_count=0 from task 2.2's placeholder.
Does not call persist_load_snapshot, following task 2.2's established
raw-byte-parsing pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
write_snapshot now Cap'n-Proto-encodes every retained job under nested
table+per-entity RTTI_LOCK_READ (table lock alone does not exclude a
concurrent per-entity writer -- design.md's central correctness finding),
appends length-prefixed job frames after a new format_version marker, and
fixes a previously-unchecked fdatasync return. Any failure anywhere in the
path (encode, write, sync) discards the whole checkpoint -- no partial
file at the final path, proven by a real mid-encode-loop fault-injection
test (forces one specific job's encode to fail among N, asserts nothing
survives).
Also fixed two pre-existing bugs in task 1.1's test fixture: terminal-state
seeded jobs need a non-zero finish_time or job_should_retain's retention
window silently excludes them from any checkpoint; the fixture needs
xstring_table_init or every seeded job's submit/start_info silently
decodes as NULL.
Round 1 of this task also touched load_snapshot/persist.h to make the
round-trip tests pass end-to-end, but review caught a live hazard: it
advanced current_lsn/checkpoint_lsn BEFORE the entity-decode loop ran, so
a decode failure still left the position advanced -- reachable today
through the unmodified ha_handle_snapshot_chunk production path, not
hypothetical. Remediated by fully reverting load_snapshot/persist.h to
byte-identical pre-task-2.2 state and rewriting the round-trip tests to
parse write_snapshot's raw on-disk bytes directly (read_Job/decode_Job)
instead of round-tripping through persist_load_snapshot -- proves the
write side without depending on load_snapshot, which stays task 5.2's
scope entirely, not started here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Three tests against the current sentinel-only write_snapshot/load_snapshot:
job content round-trip and concurrent-mutation-not-torn genuinely FAIL
(RED) since no entity content is written today; write-failure-discard
already passes (write_snapshot's open()-failure path already discards
correctly) -- flagged transparently as a GREEN-from-day-one regression
guard rather than faked RED, per real fault injection (directory removal
-> genuine ENOENT).
Carries forward two notes for task 2.2: fdatasync's return value is
currently unchecked in write_snapshot (persist.c:1505), and a mid-encode-
loop fault-injection test should be added once the per-job encode loop
exists.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Adds the ha-entity-snapshots Kiro spec (requirements, design, research,
tasks) for task 5.10a / raft-ha-core-safety: making the HA checkpoint an
actually-restorable copy of committed job/queue state instead of a
position-only sentinel, closing the residual chaotic-soak divergence gap.
Task 1.1 (foundation): reusable test fixture snapent_make/snapent_teardown
in a new test_persist_snapshot_entities.c, seeding N jobs (varying state)
and M queues (varying status) into a real persist_t+cluster_t for later
write/load/rotation-guard tests to build on. Build clean; new suite 1/1;
regression spot-checks (test_persist_hooks 19/19, test_commit_barrier_reply
22/22) clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
A dead primary's control fd RDHUPs faster than an election settles. The 5.6
guard only drops an RDHUP while the role is instantaneously CTRL_ELECTING/
prevote; an RDHUP landing in the gap after a lost election (role briefly back
to CTRL_STANDBY) re-triggers a fresh ha_begin_failover -> PreVote -> real
election (term++). One bump per RDHUP -> unbounded escalation under a sustained
cadence (observed round 4: 48 begin_failovers, terms 6->24, round overran the
harness window).
Fix: ha_state_t.last_failover_ms (atomic) stamped at ha_begin_failover;
ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2*election_timeout_ms of the
last trigger (closes the dead fd, returns) so the election's own jittered retry
(5.7) runs without the external RDHUP cadence adding term bumps. First trigger
never debounced; genuine loss still fails over, at most once per window.
Validation: unit-proven — test_rdhup_within_cooldown_is_debounced (recent
last_failover -> stays CTRL_STANDBY) vs test_rdhup_single_node_reaches_primary
(last_failover=0 -> CTRL_PRIMARY) is a differential proof. Soak: 10/10 elect,
terms 4->14 bounded, begin_failover 14/14/12 (was 48), committed_job_loss=0, no
regression. The stochastic storm did not recur this run (debounce fired 0x), so
soak-reproduction of the storm is still pending; the mechanism rests on the
unit test. Election suites green (rdhup 8/0, prevote 12/0, core_safety 24/0,
election 28/0, failover_prevote 6/0).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
raft-apply-decoupling item #1 (down payment). xce_run_hooks returns only the
LAST-registered hook's rc, so a later JOB AFTER_EVENT subscriber (sched/
apiserver) returning 0 clobbered persist_job_hook_fun's commit -ETIMEDOUT.
handle_jobsubmit then replied 'jobsubmit ok' for an entry that never reached
majority, and a new-term leader legally discarded it (committed-job-loss,
Commit Durability violation).
- persist.c/.h: new persist_submit_wal_and_commit(p, jlist) — the explicit raft
step (WAL append + majority-commit barrier) returning the commit verdict
directly. persist_job_hook_fun's WAL_MODE_WAIT branch is now a no-op.
- job.c handle_jobsubmit: call the explicit step, read commit_rc DIRECTLY,
reject on <0 before firing the now notification-only AFTER hook
(canonical commit -> notify -> reply).
- tests: +test_job_hook_new_is_notify_only regression pin; 4 hook tests
relocated to drive the explicit step. persist_hooks 19/0, e2e_hooks 3/0,
commit_barrier_reply 22/0, commit_durability 38/0.
Proven: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code).
Residual soak FAILs (divergence catch-up lag + a round-4 election-escalation
tail) are NOT job loss — tracked as task 5.10.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
The io_worker freed an endpoint (endpoint_killer, cluster.c) on connection
teardown while the vtask dispatcher concurrently read ep->busy
(endpoint_is_task_blocked, netio.c) on a queued vtask — a heap-use-after-free
that ABORTed a controller mid-chaotic-soak (round 4), dropping the cluster
below quorum so no primary could be elected.
Replace the accumulation of ad-hoc ep-lifetime patches (bug-1282 lockless busy,
bug-1197 dispatch_inflight drain + deferred teardown vtask) with one principled
invariant: an endpoint lives until its reference count reaches 0.
- endpoint_t.refcnt (_Atomic) + heap_alloc flag; endpoint_get starts a heap ep
at refcnt=1 (the owner ref).
- endpoint_ref/endpoint_unref: the last unref fini's + frees a heap ep.
Embedded eps (client ctx->ep, heap_alloc=0) are never freed here — their
container owns them — so a vtask-balanced ref/unref pair is a safe no-op.
- vtask_enq takes a per-vtask ref on the bound ep BEFORE publishing the task to
the dispatcher; run_task / taskq_item_killer release it. So the dispatcher can
read ep->busy on any queued vtask without the ep being freed underneath it.
- every HEAP-ep teardown site (endpoint_killer, handle_endpoint_error, the
vote/prevote connect-fail paths, the vote-ep loser/teardown paths, the standby
reconnect/eviction paths) now drops its owner ref via endpoint_unref instead of
raw endpoint_fini+free. Several of those also leaked the ep struct before
(endpoint_fini-only); unref fixes the leak too.
Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown
updated — its OFF-path test asserted the old struct leak, now fixed). 6-round
ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 no longer crashes at
round 4, all 6 rounds elect at bounded terms. (The pre-existing iomgr_fini ->
SSL_CTX_free abort under openssl 3.6 in the aggregate test binary is unrelated.)
Note: with the crash gone, the soak now reaches rounds 4-6 for the first time and
surfaces a SEPARATE, pre-existing committed-job-loss/divergence (the bug-795/828
failover catch-up gap; ha-commit-durability domain) — not a refcount regression
(ASAN clean; rounds 1-3 are loss-free).
The per-campaign election watchdog was armed with a FIXED election_timeout_ms
(raft_start_election -> on_schedule_timeout -> raft_cb_schedule_timeout). Under
cold-connection churn a just-restarted survivor opens a fresh TCP+SSL endpoint
per vote round, so its single needed grant can land a hair AFTER a constant
deadline on EVERY campaign -- a resonance: the campaign times out, the term
re-escalates, and the now-stale grant is dropped (DropStaleResponse). Forever.
Terms climb unbounded; the freshest survivor never tallies quorum. This is the
final round-4 chaotic-soak election-liveness stall.
Fix (Ongaro thesis 3.4): re-randomize the watchdog per campaign to the canonical
[T, 2T] interval in the HA adapter, so some campaign's deadline falls AFTER its
grant RTT and the grant lands in-window. Done in raft_cb_schedule_timeout so the
pure (sim-tested) raft library stays deterministic. Independent of
election_jitter_max_ms (which tunes only the sub-second startup/stagger jitter)
so it spreads even when jmax is small/0; never below the configured floor so
genuine-loss failover is not sped up past election_timeout_ms.
Preserves Election Safety (no change to term/vote/DropStaleResponse logic).
new ha_jittered_election_delay() helper + 2 unit tests (RED verified: stubbing
jitter to 0 fails test_5_7_campaign_timeout_is_randomized with the resonance
message; PIN guards floor + modulo-by-zero). Suite: 24 Tests / 0 Failures.
After 5.5 let the freshest survivor win PreVote and ENTER the real election,
round 4 still stalled: the dead primary's control fd keeps RDHUP'ing on a
deterministic ~15s cadence, and ha_on_primary_ctrl_fd_rdhup's 5.2 suppress only
catches CTRL_STANDBY — so an ELECTING node fell through to ha_begin_failover and
restarted a fresh PreVote, preempting its own in-progress real election. Every
cycle bumped the term (21->25->28->29...) and kept two survivors dueling in
lockstep, defeating the jittered election-timeout tiebreak.
Guard the RDHUP entry: if role==CTRL_ELECTING or a PreVote round is in progress,
close the dead control fd and return WITHOUT restarting failover — let the
in-progress round finish or time out (its election timeout is jittered, which
desynchronizes the duel so one survivor wins). A node not currently electing
still takes the normal 5.2-suppress / genuine-loss failover path.
RED+PIN in test_ha_core_safety.c (22 Tests / 0 Failures): CORE RED — CTRL_ELECTING
+ RDHUP must not start a new PreVote (Expected 0 Was 1 on baseline); PIN — genuine
loss (STANDBY, not electing) still fails over. Preserves Election Safety, 5.2
suppression, and genuine-loss liveness. Re-running 7.2.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
The election cross-deny that stalled round 4 (after bug-1282+bug-1286 fixed): the
freshest survivor (e.g. ctrl-2 tip 15 @ term 5) could not win because it
advertised lastLogTerm = c->persist.term (the persist layer's RUNNING term,
stale/0 after a restart) instead of the term STAMPED ON its last WAL entry. So a
voter (ctrl-3, last entry term 4) saw the candidate's lastLogTerm <= its own and
raft_log_ok rejected the genuinely-fresher candidate -> every round timed out ->
permanent stall. This was a documented stopgap (ha.c "task 1.3" NOTE) and is the
bug-851 term-identity family.
Fix both advertise sites (real-vote raft_cb_send_vote_request + PreVote
raft_cb_send_pre_vote_request): lastLogTerm = persist_entry_term(&c->persist,
ha->raft.log_len) when log_len>0 — the SAME accessor voters use to compute their
own last-log term (on_get_entry_term -> persist_entry_term), so logOk is now
symmetric and the freshest survivor wins. Strengthens Election Safety/Leader
Completeness (only genuinely-fresher candidates are granted); the index source
(ha->raft.log_len, seeded from the recovered WAL tip) is unchanged.
Root-caused from a frozen round-4 soak: ctrl-2 (15@t5) vs ctrl-3 (11@t4); ctrl-3
denied ctrl-2 on logOk (mislabeled "eligibility"). Re-running 7.2 to confirm.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Root-caused via the non-ASAN debug build + gdb: on a chaotic restart an inbound
endpoint dispatches a request (e.g. walCatchupEntry op 37) while ep->owner
(cluster_t) is NULL — not yet wired at accept, or cleared during teardown. The
handler did `c = ep->owner` then `ha_dispatch_gate(&c->ha, ...)`; with c==NULL,
&c->ha = NULL+offsetof(cluster,ha) = 0x3ff0, a garbage pointer ha_dispatch_gate
dereferenced (atomic_load &ha->role) -> SIGSEGV, dropping the node below quorum
and stalling the election.
Guard c==NULL after decode (so the decoded request is freed, no leak): log a
loud WARN and drop the request — there is no cluster context to dispatch through
and no c->myhost to address a reply. The controller stays alive (never
self-terminate on an operational condition); the peer retries once the endpoint
is owned. Backtrace: ha_dispatch_gate(ha=0x3ff0) <- myreq_handler cluster.c <-
vtask_entry <- run_task.
Second of the two round-4 blockers (bug-1282 deadlock already fixed). Re-running
the 7.2 chaotic soak with both fixes.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
/xce-build now takes a build PROFILE: asan (default, debug+ASAN -> xce.build,
backward-compatible), debug (non-ASAN -O0 -g3 -fno-inline -fno-omit-frame-pointer
-> build.dbg, for gdb/cores), release (-O3 -> build.release). Each profile uses
its own default build dir so variants don't clobber each other; arg2 may still be
a legacy build_dir. asan runs the unit suites; debug/release are build-only (the
suites are ASAN-oriented and the meson test harness is overlay-fs fragile, bug-924)
and print how to run a suite directly.
Why: bug-1282/bug-1286 are heisenbugs ASAN perturbs away (no SIGSEGV) and that
gdb-on-release turns into a hang/timeout. The non-ASAN -O0 -g3 build reproduces
them AND yields a symbolized backtrace under gdb.
It immediately paid off: the debug profile ROOT-CAUSED bug-1286 (the round-4
segfault) — myreq_handler (cluster.c:251) does `c = ep->owner` then
`ha_dispatch_gate(&c->ha,…)`; on chaotic restart a walCatchupEntry (op 37) vtask
runs with ep->owner==NULL, so &c->ha = 0x3ff0 -> deref crash at ha.c:3194. Fix is
a NULL-guard in myreq_handler (+ owner-lifecycle). 7.2 now has one remaining
blocker (bug-1286, root-caused); bug-1282 deadlock is fixed.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Root-caused via gdb (all-thread backtrace on two independently-wedged controllers
during chaotic failover): an ABBA lock-order deadlock in the async dispatch path.
io_worker (netio.c): holds ep->lock, blocks on q->lock (vtask_enq -> dataq_enq)
vdispatcher: dataq_deq holds q->lock, blocks on ep->lock via the
vtask_eligible -> endpoint_is_task_blocked match callback
endpoint_is_task_blocked took ep->lock while the caller (dataq_deq) held the dataq
q->lock, inverting io_worker's ep->lock -> q->lock order. Under churn the two
orders collide and the iomgr wedges (no more accepts / vote replies).
Fix (mirrors the dispatch_inflight precedent, bug-1197): promote endpoint_t.busy
from a 1-bit bitfield to _Atomic int and make endpoint_block_tasks /
endpoint_unblock_tasks / endpoint_is_task_blocked lockless (release/acquire). The
ep->lock that wrapped each access protected nothing else (busy only gates the
vtask queue), so removing it is safe; the inherent set-then-check TOCTOU is
unchanged (the lock never prevented it — the decision is advisory).
Verified: post-fix gdb soak shows ZERO threads in __lll_lock_wait and
endpoint_is_task_blocked no longer appears in any lock path (deadlock gone);
controller_unit_test all green (no regression). NOTE: a SEPARATE release-only
restart segfault remains and still blocks the raft-ha-core-safety 7.2 gate
(tracked separately) — deadlocks hang, they do not SIGSEGV, so it is a distinct
memory bug.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
The round-4 election stall surfaced by the 7.2 soak (bug-1272): after several
overlapping kills the only fully-up-to-date node is killed and the two survivors
are both restarted-and-lagging (in_catch_up=1) with no live primary left to catch
up FROM — so the flag never clears. Each holds the full committed prefix and is a
SAFE candidate, yet raft_cb_vote_eligible denied ALL PreVotes while in_catch_up
UNCONDITIONALLY, so they mutually denied and the election deadlocked forever.
Gate the in_catch_up deny on a LIVE primary (primary_ctrl_fd>=0 &&
ha_heartbeat_is_fresh), mirroring the heartbeat-fresh deny one block up. The
deny's purpose is to avoid disrupting a primary that is actively replicating to
us; once the primary is unreachable, raft_log_ok (already applied in Step 2 of
raft_on_pre_vote_request, before this callback) guarantees the candidate's log
>= ours, so granting a fresher candidate is safe (Leader Completeness preserved).
Adds 3 cases to test_ha_core_safety.c driving the real raft_on_pre_vote_request:
CORE RED (catching-up survivor + primary gone must GRANT a fresh candidate),
PIN (still DENY while a live primary feeds it), PIN (STALE candidate still DENIED
by logOk). test_ha_core_safety 20 Tests / 0 Failures; 5.1/5.2 unaffected.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
A recovered standby whose control channel flaps (transient RDHUP) while a
current-term primary is alive treated each flap as "primary died" and launched
an unwinnable PreVote failover (peers see the live primary and deny, the round
times out, the channel re-establishes, repeat ~15s) — so the node prevote-spun
forever instead of rejoining, wedging the chaotic soak (research.md §"Phase 4/5").
Add a freshness guard at the RDHUP failover-initiation path
(ha_on_primary_ctrl_fd_rdhup): when CTRL_STANDBY and ha_heartbeat_is_fresh,
stay CTRL_STANDBY, re-arm the liveness watchdog (production), and await the
primary's reconnect instead of campaigning — the same guard
ha_failover_stagger_fn already applies on the deferred PreVote path. Genuine
loss (stale heartbeat) falls through to failover unchanged; no term bump / no
role change, so Election Safety is untouched.
Keeping the standby attached lets catch-up complete, which clears in_catch_up
and lets votes flow again — so this one guard also resolves the downstream
catching-up-node vote-deny deadlock (obs 1240).
Flips task 5.1's CORE RED to GREEN: test_ha_core_safety 17 Tests / 0 Failures
(spurious launches=0/5; genuine-loss final_term=8; Election Safety A=3/B=0 →
1 primary). Zero failing suites binary-wide; LSan unchanged vs baseline.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Add three Phase-5 cases to test_ha_core_safety.c pinning the spurious-failover
root cause (research.md §"Phase 4/5"):
- CORE RED test_5_1_spurious_failover_suppressed_while_primary_alive: a
control-channel flap (transient RDHUP) while a current-term primary is alive
(fresh heartbeat) must NOT launch a PreVote failover. FAILs on baseline
(Expected 0 Was 5) — the RDHUP-initiation path (ha_on_primary_ctrl_fd_rdhup
-> ha_begin_failover -> ha_start_prevote_async synchronous branch) has no
ha_heartbeat_is_fresh guard, unlike the deferred ha_failover_stagger_fn path.
GREEN when task 5.2 adds that guard.
- LIVENESS test_5_1_genuine_loss_elects_and_advances_term: genuine loss (stale
heartbeat) still launches a real failover; a failed round re-arms; rapid
rounds advance the term. Guards 5.2 against over-suppression (Req 6.1/6.2/6.3).
- SAFETY test_5_1_election_safety_one_primary_per_term: two candidates contest
the same term; the real handle_request_vote grants at most one vote per term
(A=3/B=0), so at most one committing primary per term. Falsifiable Election
Safety check mapped to raft/raft.tla (Req 4.1/4.3).
Deterministic (no sleeps; clock reads only), c==NULL unit-test mode, clean
ha_fini teardown. test_ha_core_safety suite: 17 Tests / 1 Failure (the
intended RED); all 2.x/4.x/seam cases still pass; no new ASAN leaks.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Root-cause source fix: ha_stream_wal_to_standby reads the entry body via
stream_read_entry_body (bounded retry max 4, no sleep); on persistent short
pread/malloc-fail it fails loud + ABORTS the stream (flush valid prefix,
drop unreadable entry, -EIO) instead of shipping PLEN=0. Incomplete stream
re-driven by ACK-driven catch-up. With the 2.2 guard this closes the live
cross-term payload-loss divergence. 2.3 CORE RED -> GREEN (fail-loud: no
empty persisted; + retry-recover test ships full 64). Term-blind structural
retirement deferred to 6.2 (empty-over-full safety enforced by 2.2 guard +
pinned). 12/12 core suite; siblings green; no regression introduced.
Phase 2 complete.
Minimal XCE_TESTING fault-injection seam in ha_stream_wal_to_standby
(NULL/production-inert) forces a short body read on a cross-term entry.
CORE RED proves the streamer ships PLEN=0 for a full (source_len=64)
cross-term entry: instrumentation source_len=64 SEND_len=0 RECV_len=0 ->
loss is SEND-side (answers task 1.3's open send-vs-transitive question).
2 honest pins (term-blind route guarded by 2.2; below-tip fence). 11
tests/1 RED; production xce_controller links clean without the seam.
Case (a) empty-then-full = genuine RED (payload-blind guard at persist.c:822
no-ops the corrective write, Expected 64 Was 0). Cases (b) full-then-empty
and (c) identical-rewrite = baseline-passing pins that constrain the 2.2
fix asymmetry (empty must never overwrite stored full; identical re-stream
stays idempotent). 7 tests/1 RED/no ASAN.
New test_ha_core_safety.c with deterministic gate/probe seams (no sleeps,
drain-before-fini per bug-1224): payload-controllable WAL writer, on-disk
(term,pcrc,len) readback probe, walcompare observer, apply-gap + kill/
election injection for phases 2.3/4.1/5.1. Smoke-RED (Expected 64 Was 0)
proves the payload-blind (term,lsn)-only guard at persist.c:822 drops a
corrective full-payload write at a committed LSN. 5 tests/1 RED/4 seam
PASS, no ASAN errors.
Per-member (lsn,term,pcrc,present) report + committed-vs-uncommitted
divergence classification via walcompare_report()/--report. Soak signal
(OK/exit-code) preserved. 4 new unit tests, all GREEN.
With the dispatch flags flipped ON in-test, prove the three receiver-side
liveness properties (production defaults stay OFF):
- P1: while a snapshot (and config-push) transfer is parked in flight on the
slow lane, another endpoint's peer-control message is still processed
promptly on the vtask pool (different thread), with the transfer still
parked — the offloaded transfer does not block the drive path.
- P2: a peer-control ack (classified VTASK_LIVENESS) runs promptly even while
the pool is saturated to the LivenessAdmission reservation boundary with
REQ_BLOCK tasks (one more REQ_BLOCK fast-rejects with -E_XCE_NOT_READY,
proving real saturation). Narrowed to the liveness-dispatch-under-saturation
assertion; commit-advance itself is covered by the commit-engine tests.
- P3: while one endpoint's peer-control body is parked (holding a worker + that
ep busy), another endpoint's body runs concurrently on a different worker —
per-endpoint FIFO does not serialize across endpoints.
All deterministic (cond-var gates, bounded polls). Fixed a flaky teardown-race
hang (bug-1224): the tests released a parked vtask body then immediately called
the cancel-based vtaskmgr_fini while the just-released body was still unwinding
(P2's REQ_BLOCK bodies racing blk_lock teardown the worst). Fix adds end-of-body
completion counters polled to the full expected count before fini, so every
pool body has fully returned (worker idle) before the cancel. slowlane_fini is
graceful (join) so the lane side was already safe.
Verified: 5/5 deterministic isolated runs; full controller suite EXIT=0;
ASan-clean. Completes task 3.4 (3.4a UAF fix + 3.4b tests); the production
flag-flip remains deferred to after the 4.1 partition soak + 4.2 regression
sweep.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
When the (still-default-OFF) peer-control dispatch flag is ON,
raft_cb_become_follower runs as a vtask pool worker and frees every peer
vote-ep while other pool workers may still hold those eps' data — a UAF
(bug-1197). This makes that path quiescence-safe so the flag can later be
flipped ON (deferred to the 4.x prod-flip gate; production stays OFF here).
Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO +
busy serialization):
- Deferred teardown-as-vtask: the loser loop, when dispatch is ON and the pool
is wired, does per vote-ep endpoint_mark_tearing_down -> iomgr_unmonitor ->
enqueue a VTASK_IO/VTASK_LIVENESS teardown vtask bound to that ep. Per-ep busy
FIFO makes it run strictly after any in-flight/queued same-ep peer-control
vtask; the body NULLs iod->ep before endpoint_fini+free(ep) to disarm
run_task's post-body endpoint_unblock_tasks (which would touch the freed
ep->lock). OFF / pool-unwired path is byte-for-byte the legacy endpoint_fini.
- RCU-like drain closes the dispatch/teardown TOCTOU window: new _Atomic int
endpoint_t.dispatch_inflight (+ tearing_down:1 bit). The ON dispatch brackets
its flag-check+enqueue with fetch_add/fetch_sub (seq_cst);
endpoint_mark_tearing_down sets the flag under ep->lock, releases, then spins
until dispatch_inflight drains to 0. So every dispatch that read the flag
false finishes its enqueue before mark returns (FIFO-before the teardown
vtask); any later dispatch reads the flag set and drops the packet. ep->lock
is deliberately NOT held across the enqueue (it inverts with the taskq lock in
vtask_eligible); the counter is the lock-free handshake. Drain is bounded: the
held window is just a calloc + non-blocking enqueue on the single io_worker.
Still open before the production flag flip (4.x): the borrowed-ep race in
snapshot_lane_fn/configpush_lane_fn is distinct (slowlane thread vs io_worker
RDHUP teardown, not the vtask-pool FIFO) and needs its own per-standby-ep
discipline.
Tests: test_ha_vep_teardown.c — 5 ASan tests (ordering, late-event drop, run_task
unblock disarm, OFF-path unchanged, deterministic mark-drains-inflight-dispatch).
xcelib_test 282/0; controller_unit_test EXIT=0; ASan-clean. Adversarial
cpp-review: no CRITICAL/HIGH; handshake formally sound, flag-ON path race-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
