Adds --max-wal-records (mirrors the existing --wal-sync-ms pattern:
main.c argp option -> myconf_t -> persist_config_set_max_wal_records())
so a deployment can force the WAL-record-count checkpoint trigger to
fire on a job count a soak can actually reach, instead of the
production default of 10000.
Adds a new `--force-snapshot` soak mode (run-ha-soak.sh): submits
enough jobs to cross the configured threshold, kills a non-primary
node, wipes its data dir (a genuine empty rejoin), restarts it, and
asserts the cluster reconverges with zero walcompare divergence and
zero job loss. Fails loud (not a false PASS) if the restarted node's
log shows no actual snapshot install, so the round can't silently
report success without exercising the real scenario.
INVESTIGATION FINDING (not yet resolved, this commit does not close
5.10a or ha-entity-snapshots 7.1): live-tested with temporary
diagnostic logging (since reverted). schedule_checkpoint() correctly
fires once wal_record_count reaches the configured threshold
(confirmed: "wal_record_count=100 max=100" logged), but the delaymgr-
scheduled checkpoint_task_fn never executes afterward -- no checkpoint
ever completes, no snapshot file is ever written, in a live multi-node
deployment. This is a separate, deeper issue in the delaymgr scheduling
path (or its wiring for this specific call shape), not a bug in this
commit's CLI plumbing or soak scaffolding -- both are built, tested,
and confirmed working up to the point where the async checkpoint
should run. Needs its own dedicated investigation (likely src/lib
delaymgr, not persist.c) before --force-snapshot can produce a PASS.
This also has production implications beyond the test scenario: if
confirmed, WAL checkpoints may never fire automatically today, meaning
WAL segments are never trimmed via this path in a live deployment.
controller_unit_test: test_ha_startup 34/0, test_ha_core_safety 28/0,
test_persist_checkpoint 5/0, test_walcompare 17/0 (no regression).
Soak --selftest 17/17.
walcompare previously had no concept of a per-member compaction floor:
committed_lsn only capped the HIGH end (skip LSNs a member hasn't
caught up to yet), but the LOW end unconditionally expected every
member to hold raw WAL from LSN 1. A member that legitimately caught
up via a full-checkpoint snapshot holds no raw WAL at or below its own
checkpoint LSN by design (that range was entity-restored, not
WAL-replayed) -- so it was always flagged MISSING/divergent there,
regardless of whether the restore was correct. This is the root cause
ha-entity-snapshots task 7.1 diagnosed (committed_divergence=11 on an
otherwise-healthy soak) and raft-ha-core-safety task 5.10a's residual.
Adds member_snapshot_floor(): each member's own snap/ sibling
directory (matching the deployed <data>/wal + <data>/snap layout) is
scanned for the highest installed snap-<lsn>.{capnp,bin} file. Any LSN
at or below that member's own floor is excluded from its comparison
the same way an LSN above its own committed_lsn already is -- never
uniform/shared across members, since compaction is a per-member fact.
wc_status() gains a new "compacted" token distinct from "MISSING" so
the machine-readable report can tell the two apart.
Two new regression tests prove: (1) the floor correctly suppresses the
compacted range from comparison, and (2) it does NOT mask a genuine
divergence within the range a member still claims to hold. Caught and
fixed one bug myself before landing: the floor comparison was
off-by-one (< instead of <=), which the first test run surfaced
immediately (Expected 0 Was 1).
test_walcompare 17/17, test_ha_core_safety 28/0, test_config_repl
66/0, test_waldump 7/0 (all no-regression). Soak --selftest 17/17.
A full 69-group sweep of controller_unit_test (this spec's harness had
never been run end-to-end before) found 2 deterministic failures,
both test-only staleness against already-shipped, already-approved
production behavior from earlier in this same spec:
- test_ha_vote_gate: test_pre_vote_catch_up_denies asserted the
pre-task-5.3 unconditional in_catch_up PreVote deny. Task 5.3
(bf88681) deliberately gated that deny on primary_ctrl_fd>=0 &&
ha_heartbeat_is_fresh to fix a real round-4 election-stall — this
test's exact scenario (stale heartbeat, primary_ctrl_fd=-1) is the
case 5.3 intentionally flipped to grant. Split into
test_pre_vote_catch_up_grants_when_primary_unreachable (matches the
current contract) and test_pre_vote_catch_up_denies_while_primary_live
(preserves the original protective intent, correctly scoped).
- test_egress_filter: test_live_submit_tags_submit_lsn called the
JOB_NEW persist_job_hook_fun AFTER_EVENT hook directly, but task 5.9
(15be5d9, bug-1306 fix) made that hook's WAL_MODE_WAIT branch
notify-only, moving the real WAL append + commit barrier into
persist_submit_wal_and_commit(). 5.9 relocated 4 equivalent tests in
test_persist_hooks.c but missed this one. Now drives
persist_submit_wal_and_commit() directly, mirroring that pattern.
No production code changed. Full 69-group suite: 0 Failures, 0 ASAN
reports (independently reproduced 3x: initial sweep, implementer,
reviewer).
The term-blind ha_apply_single_entry direct-persist branch (carried
forward from task 2.4) is unreachable with a real non-zero term in any
current production path: its only trigger, raft_send_append_entries,
has zero production call sites (library simulator only). Both real
senders (repl_broadcast, ha_stream_wal_to_standby) always emit a
non-zero term for a present LSN. Documents this in ha.c and design.md
rather than structurally retiring a dormant path (YAGNI), with an
explicit flag for future re-evaluation if a heartbeat-driver task ever
wires up raft_send_append_entries.
Adds two regression pins: restart preserves (term,lsn) identity
(replay_wal already reconstructs the term index via record_entry_term),
and real senders never produce the term-blind wire shape.
test_ha_core_safety 28/0, no regression, no ASAN reports.
Integration test proves the guarantee tasks 2.3/2.4 hardened in
isolation (synthetic fault injection on the streamer) also holds
end-to-end over two real persist_t/WAL instances through the real
production ingest seam (ha_apply_wal_catchup_entry). Covers live
in-term replication and cross-term catch-up: no term==0 sourced for a
live entry, no non-empty entry ever lands as PLEN=0, standby persists
the entry's stored term (not the AE header's leader-current-term,
the bug-1067 split), and walcompare reports a clean committed prefix.
test_ha_core_safety 26/0, no regression, no ASAN reports.
Adds checkpointTerm (@6, append-only) to the SnapshotTransfer wire
message. The leader stamps it via persist_entry_term(p, checkpoint_lsn)
when sending a snapshot; the standby reconciles it into its own
per-LSN term index (persist_reconcile_checkpoint_term) right after
persist_load_snapshot succeeds, before applied_lsn advances. This
authoritatively overwrites any stale local term a divergent
pre-snapshot entry left at the checkpoint LSN, closing the gap task
3.1's RED test proved (Expected 7 Was 3 -> now passes). Transport
stays persist_t-free; the reconcile lives in persist.c per the
existing dependency direction. Decode is auto-generated via capnp
mapname; only the hand-rolled encode path (request.c/typemap.h, needed
for the raw snapshot payload) required a matching manual update.
test_ha_core_safety 25/0, test_config_repl 66/0, no ASAN reports.
ha_handle_snapshot_chunk/persist_load_snapshot advance checkpoint_lsn,
current_lsn, and applied_lsn on snapshot install but never touch the
per-LSN term index, so a divergent pre-snapshot entry's stale local
term survives the install unreconciled. New test proves the gap
(Expected 7 Was 3); fix lands in task 3.2.
persist_recover's local snapshot load passed NULL for out_entities_restored,
so a rolling-upgrade restart against a pre-feature checkpoint had no
observability parity with the wire receive path's (task 6.1) INFO log.
Mirror the same conditional logging here; no behavior change, WAL replay
in Step 3 still applies unconditionally either way.
Live-verified in a 3-node cluster: planted a hand-crafted 16-byte legacy
sentinel as ctrl-1's own checkpoint file, restarted all 3 — no crash, INFO
log fired, ctrl-1 was elected primary serving the full 37-job list, and
walcompare confirmed all 3 members' committed prefixes (LSN 1..49) are
byte-identical.
ha_handle_snapshot_chunk now logs whether a received checkpoint was
fully entity-restored or was a legacy position-only file (INFO level,
framed as an expected rollout state -- never an error, per design.md).
The rc-gates-applied_lsn control flow is byte-for-byte unchanged from
task 5.2 (confirmed by review) -- this task is purely additive
observability, matching design.md's own note that this call site
"needs no change" once persist_load_snapshot's gate is correct.
New test drives the legacy path end-to-end through the real production
handler (rc==0, file renamed, applied_lsn advances to the legacy LSN).
This completes the integration layer -- write path (jobs+queues),
restore-before-advance read path, and the production wiring are all in
place and independently reviewed. Only validation tasks remain (7.1
chaotic soak, 8.1 rollback compatibility).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Replaces the naive task-5.1 stopgap with the real restore contract:
detects legacy vs entity-bearing checkpoints by remaining file size;
restores every job then every queue status via restore_job/
restore_queue_state, in file order, with bounds-checked length-prefixed
reads; only after every entity restores successfully does
current_lsn/checkpoint_lsn commit to the checkpoint's position. Every
failure path (bounds check, decode, restore error, unrecognized
format_version) returns before that single commit point -- position is
provably untouched on failure, never partially advanced.
Also fixes the pre-existing restore_job decode leak (job.c): job_init
deep-copies submit (never nulled) and MOVE_PTRs allocs/pidinfo/
pend_reasons (nulled on the one path that runs) -- free_Job(&j) is safe
unconditionally on every reachable return path (EEXIST early-return,
create_job success, create_job failure), verified against the exact
generated free_*_ptr null-check guards.
Two pre-existing test_config_repl.c wire-protocol tests needed their
synthetic garbage snapshot bodies updated to a well-formed empty-entity
tail, since load_snapshot now genuinely parses past the legacy header
(previously ignored); their actual assertions (chunk-reassembly
correctness) are unchanged.
This is the exact function/contract that caused task 2.2's rejection
(position advancing before restore validated). Independent review
traced the memory-safety fix through the generated capnp code field-by-
field, confirmed the test_config_repl.c change doesn't weaken any
assertion, and confirmed restore_queue_state's ENOENT-aborts-the-whole-
checkpoint behavior is faithful to design.md's unconditional contract
(flagged as a watch-item for task 7.1's soak, not a defect).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Extends persist_load_snapshot's public signature with an
out_entities_restored bool*, per design.md's documented contract. The
static load_snapshot function -- the real restore logic -- is completely
untouched (still today's watermark-only implementation, byte-identical
to before this task); only a thin wrapper was added that naively maps
*out_entities_restored = (rc==0), deliberately scaffolding so the 3 new
RED tests fail for genuine reasons rather than accidentally passing.
ha_handle_snapshot_chunk's sole call site gets a mechanical
declare-and-pass update -- confirmed inert, never reads the new
out-param (task 6.1's job).
Three tests against persist_load_snapshot directly (unlike prior tasks'
tests, which deliberately stayed on the raw-byte-parsing side to avoid
this exact function): restore-before-advance (fails at the job-lookup
assertion -- nothing is ever restored today), corrupted-entity-leaves-
position-unchanged (fails at rc!=0 -- today's code ignores the corrupted
section entirely, since load_snapshot never reads past the legacy
header), legacy-file-reports-position-only (fails at entities_restored==
false -- the naive wrapper claims restoration for every rc==0 case,
including legacy files).
This is the exact function/call site that caused task 2.2's rejection
(a live position-advances-before-restore-validates hazard). Independent
review verified load_snapshot's body is byte-identical to commit
d52eb94, the wrapper does nothing beyond the naive mapping, and ha.c's
change never reads the new out-param anywhere -- no repeat of that
mistake here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
rotate_snapshots now parses each candidate file's LSN and skips deletion
via a new snapshot_lsn_in_flight() check against
p->cluster->ha.snapshot_inflight_lsn[] (task 4.1) before unlinking --
protected files are retried on the next rotation pass once the transfer
completes and the LSN clears.
Checks LSN directly (nonzero + matches) rather than gating on
snapshot_inflight[]==1 first, per task 4.1's own note: the LSN clears
before the flag in every completion/failure path, so a stale flag=1 can
momentarily coexist with an already-cleared LSN=0. A checkpoint LSN is
never 0, so this is race-free without needing the flag at all.
p->cluster==NULL (standalone lib-level tests) means no HA layer exists to
protect anything.
Task 4.2's RED rotation-guard test now passes; full regression sweep
(test_persist_checkpoint, test_persist_hooks, test_commit_barrier_reply,
test_commit_durability, test_e2e_hooks, test_hooks, test_config_repl,
test_ha_liveness) all clean. Both controller_unit_test and production
xce_controller build clean; nm confirms only the real production helper
(snapshot_lsn_in_flight) appears, no test-only symbols leaked in.
Executed directly (no subagent dispatch) per the hybrid execution-mode
decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
A checkpoint file whose position matches an in-flight peer transfer
(task 4.1's snapshot_inflight_lsn[]) must survive rotate_snapshots even
when snap_keep_n would otherwise delete it. Genuinely fails today (file
removed, access() returns -1) since rotate_snapshots has no awareness of
in-flight transfers yet.
Executed directly (no subagent dispatch) per the hybrid execution-mode
decision -- small, single-file, isolated test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Adds snapshot_inflight_lsn[MAX_HA_CANDIDATES] to ha_state_t, alongside the
existing snapshot_inflight[] flag it mirrors exactly: set to the
checkpoint's LSN at ha_snapshot_send_dispatch before a transfer's first
byte goes out, cleared to 0 at every completion/failure path
(snapshot_lane_fn's normal completion, and all 3 early-return failure
sites: malloc failure, slowlane dedup-refuse, slowlane submit failure).
This is the foundation task 4.3 will consult from rotate_snapshots to
avoid deleting a checkpoint file a peer is still mid-transfer reading
from -- the one genuinely new retention guard this feature needs (the
existing write_snapshot-before-trim_wal_segments sequencing already
satisfies the WAL-trim side of Requirement 3).
Proven via a new gated unit test (test_ha_liveness.c) that observes the
tracked LSN mid-flight while the send body is parked at a test gate, not
just before/after -- confirms the lifecycle ordering, not just the
endpoints.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
write_snapshot replaces the queue_count=0 placeholder with a real
collect_queue_frames helper: enumerates every queue in the live table
under nested table+per-entity RTTI_LOCK_READ (no retention filter --
table presence already implies config-provisioned existence, unlike
jobs), writes each queue's (name, status) as a plain fixed record (no
capnp -- matches restore_queue_state's exact contract). Any failure
anywhere still discards the whole checkpoint, including any
already-collected job frames, no partial file at the final path.
Entirely contained to persist.c -- load_snapshot and persist.h remain
byte-identical to before this task, honoring the boundary lesson task 2.2
established (queue restoration stays task 5.2's scope).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Queue's (name, status) round-trip through write_snapshot's raw on-disk
bytes -- a plain fixed record (no capnp; queue definitions are
config-owned, only runtime admin status is WAL/snapshot-tracked, per
design.md's corrected queue model). Fails today (Expected 4 Was 0) since
write_snapshot still hardcodes queue_count=0 from task 2.2's placeholder.
Does not call persist_load_snapshot, following task 2.2's established
raw-byte-parsing pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
write_snapshot now Cap'n-Proto-encodes every retained job under nested
table+per-entity RTTI_LOCK_READ (table lock alone does not exclude a
concurrent per-entity writer -- design.md's central correctness finding),
appends length-prefixed job frames after a new format_version marker, and
fixes a previously-unchecked fdatasync return. Any failure anywhere in the
path (encode, write, sync) discards the whole checkpoint -- no partial
file at the final path, proven by a real mid-encode-loop fault-injection
test (forces one specific job's encode to fail among N, asserts nothing
survives).
Also fixed two pre-existing bugs in task 1.1's test fixture: terminal-state
seeded jobs need a non-zero finish_time or job_should_retain's retention
window silently excludes them from any checkpoint; the fixture needs
xstring_table_init or every seeded job's submit/start_info silently
decodes as NULL.
Round 1 of this task also touched load_snapshot/persist.h to make the
round-trip tests pass end-to-end, but review caught a live hazard: it
advanced current_lsn/checkpoint_lsn BEFORE the entity-decode loop ran, so
a decode failure still left the position advanced -- reachable today
through the unmodified ha_handle_snapshot_chunk production path, not
hypothetical. Remediated by fully reverting load_snapshot/persist.h to
byte-identical pre-task-2.2 state and rewriting the round-trip tests to
parse write_snapshot's raw on-disk bytes directly (read_Job/decode_Job)
instead of round-tripping through persist_load_snapshot -- proves the
write side without depending on load_snapshot, which stays task 5.2's
scope entirely, not started here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Three tests against the current sentinel-only write_snapshot/load_snapshot:
job content round-trip and concurrent-mutation-not-torn genuinely FAIL
(RED) since no entity content is written today; write-failure-discard
already passes (write_snapshot's open()-failure path already discards
correctly) -- flagged transparently as a GREEN-from-day-one regression
guard rather than faked RED, per real fault injection (directory removal
-> genuine ENOENT).
Carries forward two notes for task 2.2: fdatasync's return value is
currently unchecked in write_snapshot (persist.c:1505), and a mid-encode-
loop fault-injection test should be added once the per-job encode loop
exists.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
Adds the ha-entity-snapshots Kiro spec (requirements, design, research,
tasks) for task 5.10a / raft-ha-core-safety: making the HA checkpoint an
actually-restorable copy of committed job/queue state instead of a
position-only sentinel, closing the residual chaotic-soak divergence gap.
Task 1.1 (foundation): reusable test fixture snapent_make/snapent_teardown
in a new test_persist_snapshot_entities.c, seeding N jobs (varying state)
and M queues (varying status) into a real persist_t+cluster_t for later
write/load/rotation-guard tests to build on. Build clean; new suite 1/1;
regression spot-checks (test_persist_hooks 19/19, test_commit_barrier_reply
22/22) clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
A dead primary's control fd RDHUPs faster than an election settles. The 5.6
guard only drops an RDHUP while the role is instantaneously CTRL_ELECTING/
prevote; an RDHUP landing in the gap after a lost election (role briefly back
to CTRL_STANDBY) re-triggers a fresh ha_begin_failover -> PreVote -> real
election (term++). One bump per RDHUP -> unbounded escalation under a sustained
cadence (observed round 4: 48 begin_failovers, terms 6->24, round overran the
harness window).
Fix: ha_state_t.last_failover_ms (atomic) stamped at ha_begin_failover;
ha_on_primary_ctrl_fd_rdhup drops an RDHUP within 2*election_timeout_ms of the
last trigger (closes the dead fd, returns) so the election's own jittered retry
(5.7) runs without the external RDHUP cadence adding term bumps. First trigger
never debounced; genuine loss still fails over, at most once per window.
Validation: unit-proven — test_rdhup_within_cooldown_is_debounced (recent
last_failover -> stays CTRL_STANDBY) vs test_rdhup_single_node_reaches_primary
(last_failover=0 -> CTRL_PRIMARY) is a differential proof. Soak: 10/10 elect,
terms 4->14 bounded, begin_failover 14/14/12 (was 48), committed_job_loss=0, no
regression. The stochastic storm did not recur this run (debounce fired 0x), so
soak-reproduction of the storm is still pending; the mechanism rests on the
unit test. Election suites green (rdhup 8/0, prevote 12/0, core_safety 24/0,
election 28/0, failover_prevote 6/0).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
raft-apply-decoupling item #1 (down payment). xce_run_hooks returns only the
LAST-registered hook's rc, so a later JOB AFTER_EVENT subscriber (sched/
apiserver) returning 0 clobbered persist_job_hook_fun's commit -ETIMEDOUT.
handle_jobsubmit then replied 'jobsubmit ok' for an entry that never reached
majority, and a new-term leader legally discarded it (committed-job-loss,
Commit Durability violation).
- persist.c/.h: new persist_submit_wal_and_commit(p, jlist) — the explicit raft
step (WAL append + majority-commit barrier) returning the commit verdict
directly. persist_job_hook_fun's WAL_MODE_WAIT branch is now a no-op.
- job.c handle_jobsubmit: call the explicit step, read commit_rc DIRECTLY,
reject on <0 before firing the now notification-only AFTER hook
(canonical commit -> notify -> reply).
- tests: +test_job_hook_new_is_notify_only regression pin; 4 hook tests
relocated to drive the explicit step. persist_hooks 19/0, e2e_hooks 3/0,
commit_barrier_reply 22/0, commit_durability 38/0.
Proven: 10-round chaotic soak committed_job_loss 5 -> 0 (same election code).
Residual soak FAILs (divergence catch-up lag + a round-4 election-escalation
tail) are NOT job loss — tracked as task 5.10.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDYFbrgyDHbvNYcxApKhrX
The io_worker freed an endpoint (endpoint_killer, cluster.c) on connection
teardown while the vtask dispatcher concurrently read ep->busy
(endpoint_is_task_blocked, netio.c) on a queued vtask — a heap-use-after-free
that ABORTed a controller mid-chaotic-soak (round 4), dropping the cluster
below quorum so no primary could be elected.
Replace the accumulation of ad-hoc ep-lifetime patches (bug-1282 lockless busy,
bug-1197 dispatch_inflight drain + deferred teardown vtask) with one principled
invariant: an endpoint lives until its reference count reaches 0.
- endpoint_t.refcnt (_Atomic) + heap_alloc flag; endpoint_get starts a heap ep
at refcnt=1 (the owner ref).
- endpoint_ref/endpoint_unref: the last unref fini's + frees a heap ep.
Embedded eps (client ctx->ep, heap_alloc=0) are never freed here — their
container owns them — so a vtask-balanced ref/unref pair is a safe no-op.
- vtask_enq takes a per-vtask ref on the bound ep BEFORE publishing the task to
the dispatcher; run_task / taskq_item_killer release it. So the dispatcher can
read ep->busy on any queued vtask without the ep being freed underneath it.
- every HEAP-ep teardown site (endpoint_killer, handle_endpoint_error, the
vote/prevote connect-fail paths, the vote-ep loser/teardown paths, the standby
reconnect/eviction paths) now drops its owner ref via endpoint_unref instead of
raw endpoint_fini+free. Several of those also leaked the ep struct before
(endpoint_fini-only); unref fixes the leak too.
Validation: controller_unit_test 182 tests / 0 failures (test_ha_vep_teardown
updated — its OFF-path test asserted the old struct leak, now fixed). 6-round
ASAN chaotic soak: ZERO AddressSanitizer reports, ctrl-3 no longer crashes at
round 4, all 6 rounds elect at bounded terms. (The pre-existing iomgr_fini ->
SSL_CTX_free abort under openssl 3.6 in the aggregate test binary is unrelated.)
Note: with the crash gone, the soak now reaches rounds 4-6 for the first time and
surfaces a SEPARATE, pre-existing committed-job-loss/divergence (the bug-795/828
failover catch-up gap; ha-commit-durability domain) — not a refcount regression
(ASAN clean; rounds 1-3 are loss-free).
The per-campaign election watchdog was armed with a FIXED election_timeout_ms
(raft_start_election -> on_schedule_timeout -> raft_cb_schedule_timeout). Under
cold-connection churn a just-restarted survivor opens a fresh TCP+SSL endpoint
per vote round, so its single needed grant can land a hair AFTER a constant
deadline on EVERY campaign -- a resonance: the campaign times out, the term
re-escalates, and the now-stale grant is dropped (DropStaleResponse). Forever.
Terms climb unbounded; the freshest survivor never tallies quorum. This is the
final round-4 chaotic-soak election-liveness stall.
Fix (Ongaro thesis 3.4): re-randomize the watchdog per campaign to the canonical
[T, 2T] interval in the HA adapter, so some campaign's deadline falls AFTER its
grant RTT and the grant lands in-window. Done in raft_cb_schedule_timeout so the
pure (sim-tested) raft library stays deterministic. Independent of
election_jitter_max_ms (which tunes only the sub-second startup/stagger jitter)
so it spreads even when jmax is small/0; never below the configured floor so
genuine-loss failover is not sped up past election_timeout_ms.
Preserves Election Safety (no change to term/vote/DropStaleResponse logic).
new ha_jittered_election_delay() helper + 2 unit tests (RED verified: stubbing
jitter to 0 fails test_5_7_campaign_timeout_is_randomized with the resonance
message; PIN guards floor + modulo-by-zero). Suite: 24 Tests / 0 Failures.
After 5.5 let the freshest survivor win PreVote and ENTER the real election,
round 4 still stalled: the dead primary's control fd keeps RDHUP'ing on a
deterministic ~15s cadence, and ha_on_primary_ctrl_fd_rdhup's 5.2 suppress only
catches CTRL_STANDBY — so an ELECTING node fell through to ha_begin_failover and
restarted a fresh PreVote, preempting its own in-progress real election. Every
cycle bumped the term (21->25->28->29...) and kept two survivors dueling in
lockstep, defeating the jittered election-timeout tiebreak.
Guard the RDHUP entry: if role==CTRL_ELECTING or a PreVote round is in progress,
close the dead control fd and return WITHOUT restarting failover — let the
in-progress round finish or time out (its election timeout is jittered, which
desynchronizes the duel so one survivor wins). A node not currently electing
still takes the normal 5.2-suppress / genuine-loss failover path.
RED+PIN in test_ha_core_safety.c (22 Tests / 0 Failures): CORE RED — CTRL_ELECTING
+ RDHUP must not start a new PreVote (Expected 0 Was 1 on baseline); PIN — genuine
loss (STANDBY, not electing) still fails over. Preserves Election Safety, 5.2
suppression, and genuine-loss liveness. Re-running 7.2.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
The election cross-deny that stalled round 4 (after bug-1282+bug-1286 fixed): the
freshest survivor (e.g. ctrl-2 tip 15 @ term 5) could not win because it
advertised lastLogTerm = c->persist.term (the persist layer's RUNNING term,
stale/0 after a restart) instead of the term STAMPED ON its last WAL entry. So a
voter (ctrl-3, last entry term 4) saw the candidate's lastLogTerm <= its own and
raft_log_ok rejected the genuinely-fresher candidate -> every round timed out ->
permanent stall. This was a documented stopgap (ha.c "task 1.3" NOTE) and is the
bug-851 term-identity family.
Fix both advertise sites (real-vote raft_cb_send_vote_request + PreVote
raft_cb_send_pre_vote_request): lastLogTerm = persist_entry_term(&c->persist,
ha->raft.log_len) when log_len>0 — the SAME accessor voters use to compute their
own last-log term (on_get_entry_term -> persist_entry_term), so logOk is now
symmetric and the freshest survivor wins. Strengthens Election Safety/Leader
Completeness (only genuinely-fresher candidates are granted); the index source
(ha->raft.log_len, seeded from the recovered WAL tip) is unchanged.
Root-caused from a frozen round-4 soak: ctrl-2 (15@t5) vs ctrl-3 (11@t4); ctrl-3
denied ctrl-2 on logOk (mislabeled "eligibility"). Re-running 7.2 to confirm.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Root-caused via the non-ASAN debug build + gdb: on a chaotic restart an inbound
endpoint dispatches a request (e.g. walCatchupEntry op 37) while ep->owner
(cluster_t) is NULL — not yet wired at accept, or cleared during teardown. The
handler did `c = ep->owner` then `ha_dispatch_gate(&c->ha, ...)`; with c==NULL,
&c->ha = NULL+offsetof(cluster,ha) = 0x3ff0, a garbage pointer ha_dispatch_gate
dereferenced (atomic_load &ha->role) -> SIGSEGV, dropping the node below quorum
and stalling the election.
Guard c==NULL after decode (so the decoded request is freed, no leak): log a
loud WARN and drop the request — there is no cluster context to dispatch through
and no c->myhost to address a reply. The controller stays alive (never
self-terminate on an operational condition); the peer retries once the endpoint
is owned. Backtrace: ha_dispatch_gate(ha=0x3ff0) <- myreq_handler cluster.c <-
vtask_entry <- run_task.
Second of the two round-4 blockers (bug-1282 deadlock already fixed). Re-running
the 7.2 chaotic soak with both fixes.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
/xce-build now takes a build PROFILE: asan (default, debug+ASAN -> xce.build,
backward-compatible), debug (non-ASAN -O0 -g3 -fno-inline -fno-omit-frame-pointer
-> build.dbg, for gdb/cores), release (-O3 -> build.release). Each profile uses
its own default build dir so variants don't clobber each other; arg2 may still be
a legacy build_dir. asan runs the unit suites; debug/release are build-only (the
suites are ASAN-oriented and the meson test harness is overlay-fs fragile, bug-924)
and print how to run a suite directly.
Why: bug-1282/bug-1286 are heisenbugs ASAN perturbs away (no SIGSEGV) and that
gdb-on-release turns into a hang/timeout. The non-ASAN -O0 -g3 build reproduces
them AND yields a symbolized backtrace under gdb.
It immediately paid off: the debug profile ROOT-CAUSED bug-1286 (the round-4
segfault) — myreq_handler (cluster.c:251) does `c = ep->owner` then
`ha_dispatch_gate(&c->ha,…)`; on chaotic restart a walCatchupEntry (op 37) vtask
runs with ep->owner==NULL, so &c->ha = 0x3ff0 -> deref crash at ha.c:3194. Fix is
a NULL-guard in myreq_handler (+ owner-lifecycle). 7.2 now has one remaining
blocker (bug-1286, root-caused); bug-1282 deadlock is fixed.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Root-caused via gdb (all-thread backtrace on two independently-wedged controllers
during chaotic failover): an ABBA lock-order deadlock in the async dispatch path.
io_worker (netio.c): holds ep->lock, blocks on q->lock (vtask_enq -> dataq_enq)
vdispatcher: dataq_deq holds q->lock, blocks on ep->lock via the
vtask_eligible -> endpoint_is_task_blocked match callback
endpoint_is_task_blocked took ep->lock while the caller (dataq_deq) held the dataq
q->lock, inverting io_worker's ep->lock -> q->lock order. Under churn the two
orders collide and the iomgr wedges (no more accepts / vote replies).
Fix (mirrors the dispatch_inflight precedent, bug-1197): promote endpoint_t.busy
from a 1-bit bitfield to _Atomic int and make endpoint_block_tasks /
endpoint_unblock_tasks / endpoint_is_task_blocked lockless (release/acquire). The
ep->lock that wrapped each access protected nothing else (busy only gates the
vtask queue), so removing it is safe; the inherent set-then-check TOCTOU is
unchanged (the lock never prevented it — the decision is advisory).
Verified: post-fix gdb soak shows ZERO threads in __lll_lock_wait and
endpoint_is_task_blocked no longer appears in any lock path (deadlock gone);
controller_unit_test all green (no regression). NOTE: a SEPARATE release-only
restart segfault remains and still blocks the raft-ha-core-safety 7.2 gate
(tracked separately) — deadlocks hang, they do not SIGSEGV, so it is a distinct
memory bug.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
The round-4 election stall surfaced by the 7.2 soak (bug-1272): after several
overlapping kills the only fully-up-to-date node is killed and the two survivors
are both restarted-and-lagging (in_catch_up=1) with no live primary left to catch
up FROM — so the flag never clears. Each holds the full committed prefix and is a
SAFE candidate, yet raft_cb_vote_eligible denied ALL PreVotes while in_catch_up
UNCONDITIONALLY, so they mutually denied and the election deadlocked forever.
Gate the in_catch_up deny on a LIVE primary (primary_ctrl_fd>=0 &&
ha_heartbeat_is_fresh), mirroring the heartbeat-fresh deny one block up. The
deny's purpose is to avoid disrupting a primary that is actively replicating to
us; once the primary is unreachable, raft_log_ok (already applied in Step 2 of
raft_on_pre_vote_request, before this callback) guarantees the candidate's log
>= ours, so granting a fresher candidate is safe (Leader Completeness preserved).
Adds 3 cases to test_ha_core_safety.c driving the real raft_on_pre_vote_request:
CORE RED (catching-up survivor + primary gone must GRANT a fresh candidate),
PIN (still DENY while a live primary feeds it), PIN (STALE candidate still DENIED
by logOk). test_ha_core_safety 20 Tests / 0 Failures; 5.1/5.2 unaffected.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
A recovered standby whose control channel flaps (transient RDHUP) while a
current-term primary is alive treated each flap as "primary died" and launched
an unwinnable PreVote failover (peers see the live primary and deny, the round
times out, the channel re-establishes, repeat ~15s) — so the node prevote-spun
forever instead of rejoining, wedging the chaotic soak (research.md §"Phase 4/5").
Add a freshness guard at the RDHUP failover-initiation path
(ha_on_primary_ctrl_fd_rdhup): when CTRL_STANDBY and ha_heartbeat_is_fresh,
stay CTRL_STANDBY, re-arm the liveness watchdog (production), and await the
primary's reconnect instead of campaigning — the same guard
ha_failover_stagger_fn already applies on the deferred PreVote path. Genuine
loss (stale heartbeat) falls through to failover unchanged; no term bump / no
role change, so Election Safety is untouched.
Keeping the standby attached lets catch-up complete, which clears in_catch_up
and lets votes flow again — so this one guard also resolves the downstream
catching-up-node vote-deny deadlock (obs 1240).
Flips task 5.1's CORE RED to GREEN: test_ha_core_safety 17 Tests / 0 Failures
(spurious launches=0/5; genuine-loss final_term=8; Election Safety A=3/B=0 →
1 primary). Zero failing suites binary-wide; LSan unchanged vs baseline.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Add three Phase-5 cases to test_ha_core_safety.c pinning the spurious-failover
root cause (research.md §"Phase 4/5"):
- CORE RED test_5_1_spurious_failover_suppressed_while_primary_alive: a
control-channel flap (transient RDHUP) while a current-term primary is alive
(fresh heartbeat) must NOT launch a PreVote failover. FAILs on baseline
(Expected 0 Was 5) — the RDHUP-initiation path (ha_on_primary_ctrl_fd_rdhup
-> ha_begin_failover -> ha_start_prevote_async synchronous branch) has no
ha_heartbeat_is_fresh guard, unlike the deferred ha_failover_stagger_fn path.
GREEN when task 5.2 adds that guard.
- LIVENESS test_5_1_genuine_loss_elects_and_advances_term: genuine loss (stale
heartbeat) still launches a real failover; a failed round re-arms; rapid
rounds advance the term. Guards 5.2 against over-suppression (Req 6.1/6.2/6.3).
- SAFETY test_5_1_election_safety_one_primary_per_term: two candidates contest
the same term; the real handle_request_vote grants at most one vote per term
(A=3/B=0), so at most one committing primary per term. Falsifiable Election
Safety check mapped to raft/raft.tla (Req 4.1/4.3).
Deterministic (no sleeps; clock reads only), c==NULL unit-test mode, clean
ha_fini teardown. test_ha_core_safety suite: 17 Tests / 1 Failure (the
intended RED); all 2.x/4.x/seam cases still pass; no new ASAN leaks.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019dbTHR9CsEYK9RsqmQpQpC
Root-cause source fix: ha_stream_wal_to_standby reads the entry body via
stream_read_entry_body (bounded retry max 4, no sleep); on persistent short
pread/malloc-fail it fails loud + ABORTS the stream (flush valid prefix,
drop unreadable entry, -EIO) instead of shipping PLEN=0. Incomplete stream
re-driven by ACK-driven catch-up. With the 2.2 guard this closes the live
cross-term payload-loss divergence. 2.3 CORE RED -> GREEN (fail-loud: no
empty persisted; + retry-recover test ships full 64). Term-blind structural
retirement deferred to 6.2 (empty-over-full safety enforced by 2.2 guard +
pinned). 12/12 core suite; siblings green; no regression introduced.
Phase 2 complete.
Minimal XCE_TESTING fault-injection seam in ha_stream_wal_to_standby
(NULL/production-inert) forces a short body read on a cross-term entry.
CORE RED proves the streamer ships PLEN=0 for a full (source_len=64)
cross-term entry: instrumentation source_len=64 SEND_len=0 RECV_len=0 ->
loss is SEND-side (answers task 1.3's open send-vs-transitive question).
2 honest pins (term-blind route guarded by 2.2; below-tip fence). 11
tests/1 RED; production xce_controller links clean without the seam.
Case (a) empty-then-full = genuine RED (payload-blind guard at persist.c:822
no-ops the corrective write, Expected 64 Was 0). Cases (b) full-then-empty
and (c) identical-rewrite = baseline-passing pins that constrain the 2.2
fix asymmetry (empty must never overwrite stored full; identical re-stream
stays idempotent). 7 tests/1 RED/no ASAN.
New test_ha_core_safety.c with deterministic gate/probe seams (no sleeps,
drain-before-fini per bug-1224): payload-controllable WAL writer, on-disk
(term,pcrc,len) readback probe, walcompare observer, apply-gap + kill/
election injection for phases 2.3/4.1/5.1. Smoke-RED (Expected 64 Was 0)
proves the payload-blind (term,lsn)-only guard at persist.c:822 drops a
corrective full-payload write at a committed LSN. 5 tests/1 RED/4 seam
PASS, no ASAN errors.
Per-member (lsn,term,pcrc,present) report + committed-vs-uncommitted
divergence classification via walcompare_report()/--report. Soak signal
(OK/exit-code) preserved. 4 new unit tests, all GREEN.
With the dispatch flags flipped ON in-test, prove the three receiver-side
liveness properties (production defaults stay OFF):
- P1: while a snapshot (and config-push) transfer is parked in flight on the
slow lane, another endpoint's peer-control message is still processed
promptly on the vtask pool (different thread), with the transfer still
parked — the offloaded transfer does not block the drive path.
- P2: a peer-control ack (classified VTASK_LIVENESS) runs promptly even while
the pool is saturated to the LivenessAdmission reservation boundary with
REQ_BLOCK tasks (one more REQ_BLOCK fast-rejects with -E_XCE_NOT_READY,
proving real saturation). Narrowed to the liveness-dispatch-under-saturation
assertion; commit-advance itself is covered by the commit-engine tests.
- P3: while one endpoint's peer-control body is parked (holding a worker + that
ep busy), another endpoint's body runs concurrently on a different worker —
per-endpoint FIFO does not serialize across endpoints.
All deterministic (cond-var gates, bounded polls). Fixed a flaky teardown-race
hang (bug-1224): the tests released a parked vtask body then immediately called
the cancel-based vtaskmgr_fini while the just-released body was still unwinding
(P2's REQ_BLOCK bodies racing blk_lock teardown the worst). Fix adds end-of-body
completion counters polled to the full expected count before fini, so every
pool body has fully returned (worker idle) before the cancel. slowlane_fini is
graceful (join) so the lane side was already safe.
Verified: 5/5 deterministic isolated runs; full controller suite EXIT=0;
ASan-clean. Completes task 3.4 (3.4a UAF fix + 3.4b tests); the production
flag-flip remains deferred to after the 4.1 partition soak + 4.2 regression
sweep.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
When the (still-default-OFF) peer-control dispatch flag is ON,
raft_cb_become_follower runs as a vtask pool worker and frees every peer
vote-ep while other pool workers may still hold those eps' data — a UAF
(bug-1197). This makes that path quiescence-safe so the flag can later be
flipped ON (deferred to the 4.x prod-flip gate; production stays OFF here).
Mechanism (no libxce-wide endpoint refcount; reuses the existing per-ep FIFO +
busy serialization):
- Deferred teardown-as-vtask: the loser loop, when dispatch is ON and the pool
is wired, does per vote-ep endpoint_mark_tearing_down -> iomgr_unmonitor ->
enqueue a VTASK_IO/VTASK_LIVENESS teardown vtask bound to that ep. Per-ep busy
FIFO makes it run strictly after any in-flight/queued same-ep peer-control
vtask; the body NULLs iod->ep before endpoint_fini+free(ep) to disarm
run_task's post-body endpoint_unblock_tasks (which would touch the freed
ep->lock). OFF / pool-unwired path is byte-for-byte the legacy endpoint_fini.
- RCU-like drain closes the dispatch/teardown TOCTOU window: new _Atomic int
endpoint_t.dispatch_inflight (+ tearing_down:1 bit). The ON dispatch brackets
its flag-check+enqueue with fetch_add/fetch_sub (seq_cst);
endpoint_mark_tearing_down sets the flag under ep->lock, releases, then spins
until dispatch_inflight drains to 0. So every dispatch that read the flag
false finishes its enqueue before mark returns (FIFO-before the teardown
vtask); any later dispatch reads the flag set and drops the packet. ep->lock
is deliberately NOT held across the enqueue (it inverts with the taskq lock in
vtask_eligible); the counter is the lock-free handshake. Drain is bounded: the
held window is just a calloc + non-blocking enqueue on the single io_worker.
Still open before the production flag flip (4.x): the borrowed-ep race in
snapshot_lane_fn/configpush_lane_fn is distinct (slowlane thread vs io_worker
RDHUP teardown, not the vtask-pool FIFO) and needs its own per-standby-ep
discipline.
Tests: test_ha_vep_teardown.c — 5 ASan tests (ordering, late-event drop, run_task
unblock disarm, OFF-path unchanged, deterministic mark-drains-inflight-dispatch).
xcelib_test 282/0; controller_unit_test EXIT=0; ASan-clean. Adversarial
cpp-review: no CRITICAL/HIGH; handshake formally sound, flag-ON path race-free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
push_config_transport_only spawns one thread per standby then joins them with a
CONF_PUSH_TIMEOUT_MS bounded wait — blocking the caller. On the requester-less
drive-thread paths (promotion + heartbeat convergence) that wait stalls the
io_worker. Route those through ha_config_push_offloadable, gated by
g_drive_async_configpush_offload (default OFF → inline, byte-for-byte legacy;
ON → slow lane, non-blocking).
Flag ON deep-copies the config bytes + sha into a lane ctx (caller keeps owning
and freeing its own buffer); a single cluster-wide single-flight guard
(ha_state_t.configpush_inflight scalar CAS + lane key=1) bounds it to one
config-push fan-out in flight; the in-flight flag is cleared on every terminal
path (two alloc failures, -EEXIST, hard submit error, lane completion). Send
body factored into ha_config_push_do_send.
Three requester-less drive-thread call sites converted: raft_cb_become_leader
(promotion), the HA_REPL_PUSH_CONFIG heartbeat case, and the direct-CTRL_PRIMARY
FSM promotion branch. The admin-upgrade barrier (push_config_to_standbys, has a
waiting requester) and join.c are untouched.
Lands DORMANT like tasks 3.1/3.2: the lane reads c->ha.standby_endpoints and
writes those borrowed standby endpoints, racing io_worker RDHUP teardown if run
off-thread (bug-1197 family). Flag OFF keeps push+teardown serialized on the
io_worker. Task 3.4 flips this ON together with the snapshot and peerctrl flags
after establishing endpoint-lifetime discipline.
Tests (controller suite EXIT=0): test_configpush_offload_runs_on_lane,
test_configpush_inline_when_flag_off, test_configpush_offload_dedups.
Adversarial cpp-review: no CRITICAL/HIGH/MEDIUM findings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Part B (LIVE, default-ON): trim_wal_segments now bounds WAL-segment deletion
by repl_trim_lsn(&p->repl, active_seg_lsn) = min(active segment LSN, every
active standby acked_lsn, every valid disconnected-slot last_disconnected_lsn),
after repl_expire_disconnect_slots ages out stale slot guards. floor <=
active_seg_lsn structurally, so the guard can only preserve more segments,
never delete one a live standby still needs — closes the bug-795/828
trim-vs-cursor race for connected/recently-disconnected standbys (bug-1147
stopgap; empty/cursor=0 join still needs entity-bearing snapshots).
Part A (DORMANT, g_drive_async_snapshot_offload default OFF): ha_snapshot_send_
dispatch routes HA_REPL_SEND_SNAPSHOT inline (flag OFF — byte-for-byte legacy,
on io_worker) or to slowlane_submit+snapshot_lane_fn (flag ON). Send body
factored into ha_snapshot_do_send. Per-peer single-flight via
ha_state_t.snapshot_inflight[] CAS + slowlane key; in-flight flag cleared on
all four terminal paths, mirroring the reconnect offload (task 2.1).
Lands dormant like task 3.1: snapshot_lane_fn borrows a standby endpoint it
does not own, so flag-ON would race io_worker RDHUP teardown (cross-thread UAF,
bug-1197 family). Flag OFF keeps send+teardown serialized on the io_worker.
Task 3.4 is the single gate that flips both this and the peerctrl flag ON after
establishing endpoint-lifetime discipline; bug-1198 stays open by design until
then. The offload flag was initially authored default-ON (would ship the UAF)
and corrected to OFF in ha.c and the ha.h accessor doc (bug-1210).
Tests (controller 196/0, 0 new leaks): test_snapshot_offload_runs_on_lane,
test_snapshot_inline_when_flag_off, test_snapshot_offload_dedups_per_peer,
test_trim_guard_preserves_standby_segments,
test_trim_without_standbys_proceeds_normally. Adversarial cpp-review: GO.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S6fDCtXkkEgupu7rUah1y2
Lands the PeerCtrlDispatch machinery behind a DEFAULT-OFF flag
(g_drive_async_peerctrl_dispatch_enabled); production behavior is
UNCHANGED. peer_ctrl_msg_handler becomes a thin shim that takes
ownership of ep->fsm_in.pkt and either runs the routing body inline
(flag OFF, byte-for-byte the prior path) or dispatches it to the vtask
pool via vtask_enq_classified (flag ON). The 3 routers now take pkt as
an explicit param (no cross-thread ep->fsm_in.pkt re-read). Per-endpoint
arrival order is enforced by bind_vtask's endpoint_block_tasks.
Packet ownership: shim clears ep->fsm_in.pkt at dispatch (sole-owner
transfer); peer_ctrl_vtask_entry NULLs iod->pkt before the body frees
it once. New vtask_t.owns_io_pkt (default 0) lets taskq_item_killer free
the carried packet for an un-executed queued task at vtaskmgr_fini WITHOUT
double-freeing the libxce endpoint-owned path (bug-1199).
Adversarial review found two FLAG-ON hazards, documented with loud
comments (flag is OFF so production is unaffected), to be resolved before
task 3.4 flips the flag ON:
- bug-1197 (CRITICAL): raft_cb_become_follower endpoint_fini()s peer
vote-eps from a pool worker, racing the io_worker's read of ep->data;
needs iomgr_unmonitor before endpoint_fini in every pool-reachable
teardown.
- bug-1198 (HIGH): peer_ctrl_classify tags all work VTASK_LIVENESS, but
the CTRL_PRIMARY repl-ack path can run a blocking snapshot checkpoint
inline; resolved by task 3.2 (offload snapshot to SlowLane).
Per spec, 3.1 is not 'done' until the task 3.4 deadlock gate is green.
Tests: 3 new peerctrl tests; controller 196/196, lib 282/0; 0 net leaks.
Escalates the single-tick unit tests (15-17) to sustained liveness: with
one peer unreachable (reconnect parked in-flight on the lane for the whole
window), drives 3 consecutive heartbeat ticks against a 4-node cluster
(two survivors) and asserts every survivor keeps receiving heartbeats on
every tick (g_nop_calls == 2*3), each tick returns promptly (delaymgr
never stalled), the reconnect stays in-flight throughout, and is deduped
across all ticks (connect hook submitted exactly once — Req 2.5).
RED arm (offload flag OFF) proves the inline connect blocks the first
tick, so survivors stop getting heartbeats (g_nop_calls < 6). The flag
toggle is the RED<->GREEN switch; production is unchanged (test-only task).
test_ha_liveness suite 23 -> 25 tests; full controller suite 196/196 pass.
When a reconnect registers a new outbound control channel for peer X,
on_standby_connected now evicts any pre-existing standby_endpoints entry
for the same peer_idx (a promoted vote-ep or a stale prior channel),
keeping exactly one live outbound channel per peer (Req 2.6/4.4). The
stale ep is torn down OUTSIDE ha->lock via standby_ep_err_handler_primary
(which re-takes the lock) and reclaimed; resolution is strictly per-peer.
Also fixes endpoint leaks on the endpoint_get() calloc paths: every
endpoint_fini() error path now free(ep) (fini reclaims inner members
only), closing the task-2.1 GREEN-test leak masked by detect_leaks=0.
Tests 18-19 in test_ha_liveness.c assert single-survivor resolution and
per-peer isolation. 196/196 controller unit tests pass.
Heartbeat timer detects a disconnected peer, CAS-acquires a per-peer
single-flight flag, slowlane_submits the bounded connect, and returns — the
delaymgr never blocks on connect. Lane worker (reconnect_lane_fn) does the
bounded endpoint_connect + lock-guarded iomgr_monitor; flag cleared on all
terminal paths (incl. -EEXIST race fix). Default-ON kill-switch retains the
inline path for TDD-RED/fallback. Never self-terminates on failure. 2.2 adds
the inbound/outbound race + failure hardening (documented seam). 196/0 green.
vtaskmgr gains vtask_class_t (LIVENESS/LIVE_BLOCK/REQ_BLOCK) + a pool-level
blocking reservation: REQ_BLOCK fast-rejects -XCE_ERR_NOT_READY past
n_vcpus-R; LIVENESS never gated (deadlock-freedom). commit_submit folds its
inflight_waits onto the pool via commit_set_pool; production wired in
cluster.c phase2 (vtaskmgr_set_liveness_reserved(R) + commit_set_pool).
Lock order p->lock->ce->lock->blk_lock. Dormant until 3.1. Commit suites
38/30/22/38 green; lib 282/0, controller 196/0, ASAN clean. Phase 1 done.
socket_connect sets O_NONBLOCK before connect and bounds the attempt with
poll(timeout); new connect_timeout_ms param (0=DEFAULT_CONNECT_TIMEOUT_MS);
all 5 callers updated (ha.c x3, client.c, xcectx.c). Revalidation Trigger:
shared netio helper signature changed. Broad build links all components;
xcelib_test 278/0, controller_unit_test 194/0. 1.4/1.5 unit tests satisfied
by the TDD output of 1.2/1.3 (test_slowlane.c, test_netio_bounded_connect.c).
Dedicated-pthread pool isolated from the vtask pool; non-blocking slowlane_submit
(0 enqueued / -EEXIST deduped / -EAGAIN cap-full-deferred), admission cap on
running tasks, per-key in-flight dedup, run-exactly-once, drain+join fini. Wired
into controller lifecycle (dormant; consumers land in 2.x/3.x). 7 unit tests,
mutation-verified cap+dedup; full suite 194/0 green, 15x ASAN.
