30-round chaotic soak reported committed_job_loss=1 at round 25, but the
flagged job was confirmed present on every later verify and the final
sweep -- a real loss never comes back. Root-caused from ctrl-1/ctrl-2/ctrl-3
logs, no controller change needed: a newly-elected primary cannot fold a
prior-term entry into its own committed_lsn merely because it holds it
durably (Raft Figure-8 safety via ha_arm_own_term_gate) -- it needs a fresh
majority ack for its own-term WAL_NOOP first. That normally closes in
<200ms, but chaotic mode's overlapping-restart design let it stretch to
6.232s this run (the peer whose ack was needed was itself mid-catch-up from
its own recent restart). verify_classify ran its expect-job check
immediately on primary detection with no settle/retry, unlike verify_no_loss
(bug-1422) or wait_for_full_convergence, so it landed inside that window.
Added a bounded retry (5 attempts, 2s settle) around the LOSS classification
only, mirroring verify_no_loss's precedent -- a job still missing after that
still counts as a genuine loss. Verified with a fresh 30-round chaotic
re-soak: committed_divergence=0 committed_job_loss=0 result=PASS.
Adds test_5_1_wal_catchup_reject_sends_nack_reply, driving a genuine
(term,lsn) conflict against a real persisted WAL entry through
ha_apply_term_entry_via_library and asserting
ha_send_wal_ack_reply_ex(success=0) fires via ha_wal_ack_send_hook.
Note: test_config_repl.c's test_bug1479_ae_reject_sends_nack_not_silence
already covers the same call site via a different (forward-gap)
stimulus, added in the original bug-1479 commit. design.md's own File
Structure Plan explicitly calls for this new test in this file
regardless, so this is disclosed, design-sanctioned defense-in-depth,
not accidental duplication -- both tests independently confirmed to
fail together when the reply call is removed.
1065 -> 1066 tests, 0 failures.
bug-1480/1481: config-push silent-drop reply gap and vote_ep_err_handler
legacy-routing gap, found during raft-adapter-conformance requirements
discovery (not yet fixed at time of logging).
bug-1482: myreq_handler_for_test's doc comment wrongly claimed the
caller owns/frees the io_packet_t after the call; xce_decode actually
consumes and frees it unconditionally. Found + corrected during task
2.2 code review remediation.
Chaotic election churn (config-upgrade-snapshot task 8.6's force_elect_node
rotation) can leave a standby log-conflicted after a rejoin. Fixing this
needed three compounding layers, each unit-green but still failing the
identical live scenario until the next layer landed:
1. ha_raft_on_repl_nack (ha.c/ha.h): routes a WAL-ACK NACK into the raft
library's nextIndex backoff and immediately repl_start_catchup's the
standby at the corrected position, instead of the 16-strike silent
eviction with no catch-up driver after it. Correct from the start, but
live soak still failed identically: nothing ever produced the NACK this
code was waiting for.
2. Standby-side NACK sending: ha_apply_term_entry_via_library's logOk-reject
path now calls a new ha_send_wal_ack_reply_ex(..., success=0) instead of
silently dropping the reject. ha_send_wal_ack_reply had hardcoded
success=1 unconditionally -- the only WalAckReply sender in the
controller -- making (1)'s NACK-routing correct but unreachable dead
code. New ha_wal_ack_send_hook (XCE_TESTING) lets a unit test observe
the NACK without a live endpoint. Live-confirmed the NACK now reaches
the primary -- but the scenario still failed for a third reason.
3. ha_compute_repl_stream_ceiling_lsn (ha.c/ha.h): ha_stream_wal_to_standby's
catch-up range was hard-bounded at committed_lsn, which can never advance
in this exact scenario because advancing it requires an ACK from the very
standby that's stuck catching up (2-of-3 alive, this standby is the only
path to a new majority) -- a self-referential deadlock. Widens the
STREAM_WAL ceiling (and the convergence check gating it) to the leader's
own log tail, matching raft.tla's real AppendEntries semantics
(leaderCommit is advisory, never a send-cap), while reusing bug-1478's
exact guard so this doesn't reopen that trap (never widen onto a trailing
local-only checkpoint's own unstreamable LSN).
Verification: controller_unit_test 1053/1053 PASS, clean twice, including
new regression tests for all three layers (test_config_repl.c,
test_term_safety_adapter.c, test_ha_role.c), each RED-confirmed against its
pre-fix behavior before the real fix was restored. Three successive live
run-ha-soak.sh --config-upgrade re-runs tracked each layer landing; the
final run passed end-to-end -- all 3 force_elect_node rounds succeeded
(ctrl-2: 3/3 jobs acked term=6; ctrl-3: 3/3 jobs acked term=7, never
previously reached), zero materialized-state divergence, zero job loss.
Unblocks config-upgrade-snapshot task 8.6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A node that takes myapp_run's skip_election passive path (bug-849 --
"leader already known after the startup stagger") makes exactly ONE
cluster_init_phase2 attempt. All 5 call sites (myapp.c's skip_election
block, raft_cb_become_leader, raft_cb_become_follower,
ha_elect_evaluate_and_signal's two branches) latched ha->phase2_done = 1
unconditionally after calling cluster_init_phase2, discarding its return
value. If that one attempt raced config availability and failed, the node
was permanently stranded: commit_engine_init never ran, so
apply_committed_entry never applied any committed WAL entry for the rest
of the process's life -- and for a skip_election node specifically,
nothing else would ever retry it, since the other 4 call sites only fire
via that node's own raft-library state transitions, which never happen
for a passive standby whose primary keeps heartbeating normally.
Extracted a shared, idempotent retry (ha_phase2_retry_fn, mirrors
upgrade_materialization_retry_fn's shape) plus a delaymgr-scheduled,
self-rescheduling trigger (ha_schedule_phase2_retry) -- self-rescheduling
because, unlike config_activate_retry/upgrade_materialization_retry which
are re-triggered externally by recurring commit-advance events, nothing
else recurs for a node stuck on the passive path. Wired into all 5 call
sites; raft_cb_become_leader additionally refuses promotion on failure
(mirroring the sibling upgrade_materialization_fenced refusal already in
that function -- a promoted primary with an unwired commit engine is far
worse than a delayed promotion).
TDD: 4 new tests, RED confirmed by temporarily reverting the rc-gating.
GREEN: full suite 1044 tests, 0 failures. Live-verified via
run-ha-soak.sh --config-upgrade (the scenario that found this bug): the
retry fires for real (4 failed attempts, then success) and the soak's
checkpoint-identity/config-identity/host-admin-status assertions -- the
things bug-1477 blocked -- all pass.
Two independent call sites duplicated target = max(committed_lsn,
last_written): ha.c's heartbeat/stateReport dispatch (repl_ack_decode_and_
dispatch) and config_repl.c's catchup_derive_target (used by the ACK-driven
catch-up continuation path). Once config-upgrade-snapshot's marker-triggered
checkpoint advances last_written one LSN past an already-committed
committed_lsn (the checkpoint's own consumed, never-replicated LSN slot),
the unconditional max() permanently wedges every standby's catch-up target
at an unreachable value -- the primary re-streams an empty range every
heartbeat forever.
Extracted the shared, pure ha_compute_repl_target_lsn(committed_lsn,
last_written) (ha.h/ha.c), which only falls back to last_written on genuine
committed_lsn==0 bootstrap, and pointed both call sites at it so they can
never disagree again.
Fixing the second call site exposed three pre-existing tests in
test_convergence_verification.c that hardcoded the old buggy formula's
output as their expected target; updated their fixture values (committed_lsn
4->9, matching appended_lsn) to decouple them from the now-fixed edge case,
which already has dedicated pure-function coverage in test_ha_role.c.
GREEN: full controller_unit_test suite = 1040 tests, 0 failures, clean
across multiple runs. Live-verified via run-ha-soak.sh
--config-upgrade-wal-trim (18 jobs, zero job loss, ctrl-1 reconverged 3-of-3
using the preserved floor segment) -- unblocks config-upgrade-snapshot
task 8.9.
bug-1472 (the prior blocker, worker-control replication) is fixed and
verified (7547a8f1). Rebuilding and re-running --config-upgrade on top of
that fix surfaced a new, unrelated defect: a node that takes the
skip_election fast path in myapp_run can permanently lose commit_engine_init
if its one cluster_init_phase2() attempt races config availability, since
none of ha.c's four retry call sites re-attempt for a node that never
contests an election. That member then never fires the marker-triggered
checkpoint (task 5), leaving it permanently unable to catch up past that
LSN (WAL_CHECKPOINT is local-only, non-replicated by design) -- a distinct,
permanent wedge from bug-1475's already-fixed transient race.
Root-caused via a dedicated debug pass (log/grep/git-blame evidence, logged
as bug-1477). Out of config-upgrade-snapshot's boundary -- same pattern as
bug-1471/1472/1474 -- so task 8.6 stays blocked until bug-1477 is fixed
upstream in HA bootstrap/election.
myconf->lock (mylock_init/fini wiring) had landed but load_primary_config()
itself was left in its RED-phase reproduction state (unguarded, decoding
directly into the shared myconf->cluster_config field), so the fix's own
regression test failed on resume: thread B raced through while thread A was
parked mid-call.
Wraps the whole free-decode-publish sequence in myconf->lock (acquired first,
released on every return path) and decodes into a local cluster_config_t*
first, publishing into myconf->cluster_config only after full decode and
validation succeed -- closing the UAF/double-free window a live 3-node ASan
soak caught (two concurrent handle_repl_config_push deliveries for the same
standby racing two load_primary_config calls).
Verified 4/4 clean full-suite runs (1036 tests, 0 failures) including the
specific race test.
Records bug-1461/1462 fixes, admin-config-tool completion, and
config-upgrade-snapshot spec work across anatomy.md, buglog.json,
cerebrum.md, memory.md, and the session/token-ledger state files.
buglog.json: bug-1452 marked FIXED with root cause and fix writeup.
cerebrum.md: Do-Not-Repeat entry for the commit_engine_fini/persist_stop
ordering contract, plus the flag-noted latent duplicate in
test_demotion_convergence.c.
memory.md/anatomy.md/session state: routine session bookkeeping.
Records the controller_unit_test nested UNITY_BEGIN/UNITY_END fix
(commit a81f2a2) in cerebrum.md for future sessions, plus routine
anatomy/memory/token-ledger session bookkeeping.
run-ha-soak.sh: scope every pgrep/pkill to this run's --workdir, not just
binary name + --myname. An unscoped pkill -9 -x xce_controller (old
cleanup()) killed every xce_controller on the machine, including a
different concurrently-running soak's live primary — root cause of the
first bug-1436 misdiagnosis (a killed-out-from-under-it primary produced
a genuine, correctly-reported ECONNRESET that looked like a libxce bug).
.wolf/*: anatomy/cerebrum/buglog updated for bug-1435/1436, session and
token-ledger trail.
bug-1433: raft_cb_become_leader runs synchronously off peer_ctrl_dispatch_to_pool,
which hardcodes VTASK_LIVENESS for every inbound peer-control packet including
vote replies. The bug-1431 round-2 fix (flusher_flush_once force-flush in
persist_promotion_advance_committed) was reachable from that never-blocks task
class -- same bug family as bug-1198. Removed the force-flush; round-3
(durable-capped only) plus the already-shipped bug-1432 immediate-first-heartbeat
fix close the visibility-latency gap non-blockingly. TRACKED RESIDUAL:
persist_apply_pump_run_sync is still reachable on the same VTASK_LIVENESS path
and does bounded pread() -- lower risk (page-cache-hot) but strictly still
blocking I/O; not safe to remove outright (reopens bug-1043-family job loss),
needs its own follow-up (reclassify peer_ctrl_classify's winning-vote-reply case
or dispatch promotion tail onto a fresh VTASK_LIVE_BLOCK vtask).
bug-1434: persist_promotion_advance_committed and ha_standby_advance_committed_and_apply
raced on persist_t.committed_lsn via an unlocked load-then-conditional-store during
the promotion window (before ha->role flips to CTRL_PRIMARY) -- a losing racer could
regress committed_lsn. Added persist_committed_lsn_advance_atomic (CAS retry loop,
lock-free) and routed both call sites through it.
Validated: 19/19 unit test suites, 6-round chaotic soak (0 loss/divergence) + 6-round
sequential+leadership-coverage soak, both PASS. Also folds in raft-apply-decoupling
and raft-ha-core-safety validate-impl phase bumps to "implemented" and clears
config-upgrade-snapshot's now-satisfied entity-bearing-snapshots block.
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.
Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.
Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
Closes out task-group 1 (the commit-engine event-driven-resolve
keystone). Pure test-only: real pthreads racing the admission gate,
the sweep thread, and commit_on_ack, not sequential calls dressed as
concurrent. Adds the genuine two-real-thread ack-vs-sweep-thread race
both task-1.3 review rounds flagged as a nice-to-have (the existing
race test only proved the tie-break via single-thread white-box state
manipulation).
Also resolves the bug-1407 saga: task 1.4's "closed, 70/70" and this
task's initial "reopened, 10/10 failing" were both correct,
contradictory measurements of the identical source commit. Root cause
was a stale incremental-build artifact in the shared build/ directory
(ninja -C build rebuilt incrementally across 5 sequential tasks
without ever a fresh meson setup) -- not a source regression. Proven
by isolating each measurement in a throwaway git worktree with a
fresh build. bug-1407 is genuinely closed; the incremental-build
staleness gotcha is now documented in cerebrum.md for the remaining
task series.
bug-847 (catch-up WAL batch drops payloads across capnp segment boundaries):
its purpose-built RED reproducer, test_wal_batch_roundtrip_large_payloads
(14 entries x 872-byte payloads, 12.2KB total, genuinely spans multiple
4096-byte capnp segments per capn-malloc.c's create()), was still
TEST_IGNORE_MESSAGE'd. Un-ignoring it and running under ASAN with leak
detection: clean pass, byte-exact payloads, zero leaks, 2/2 runs. Traced the
vendored c-capnproto far-pointer machinery (write_ptr/capn_setp in capn.c)
to confirm the cross-segment struct->Data reference path is correct by
construction — whatever regressed this already got fixed elsewhere (library
bump or an incidental request.c change); the ignore was just never removed.
No production code changed.
bug-1307 (task-5.7 election-jitter data-safety regression: 7 divergences +
4 job losses over 6 chaotic rounds): re-ran the exact reproducing scenario
(6-round chaotic soak, real ext4, ASAN binaries) — 21 jobs, 0 divergence,
0 job loss, clean PASS. Confirms the raft-ha-core-safety Phase 5 work
resolved it.
Both bugs were the last 2 of the 7 flagged in the prior staleness sweep;
all 7 are now closed (6 were already fixed and undetected, 1 needed the
test un-ignored).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
Their WAL_MODE_WAIT hook (wal_hook_append_classified) does local-append only
and carries no durability verdict through the hook return, so there is nothing
for a later hook to clobber. JOB was unique in running the full commit barrier
inside the hook. Their commit barrier is the separate ha-commit-barrier-rollout
handler barrier — verify independently, do not mechanically extend the JOB fix.
Ran the live 3-node E2E (the last open task). Result:
PASS: 8 rounds, 30 jobs submitted, zero job loss; all 3 nodes led
(ctrl-1,ctrl-2,ctrl-3) (exit 0)
Live-covered: kill-primary -> promoted standby owns all confirmed jobs
(zero loss across 8 failovers, jobs 3->30, expect-job ok each round);
restart ex-primary -> converged rejoin, 3-of-3 each round, no phantom jobs;
deterministic leadership coverage (ctrl-3 forced via RequestVote-freshness
fence). The soak also served as the live regression for bug-1077 (~40
failover client round-trips, no SEGV) after rebuilding the controller (the
deployed binaries predated that fix).
Not exercised live (remain unit-covered, tasks 2/4/5): withhold-ACK barrier
single-outcome (test_demotion_convergence) and isolate/heal quorum-WARN
(test_convergence_verification). All 8 tasks now [x]; spec closed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
ctx->reply.u is a single untagged union shared across opcodes (debugmsg /
get(dataset_t*) / jobid / clusterconf / ctrlstatus alias the same 8 bytes).
Commit 61e1391 added speculative free-guards
`if (ctx->reply.u.ctrlstatus) free_CtrlStatus_ptr(...)` (and the clusterconf
twin). A non-NULL slot does NOT imply a CtrlStatus*: a failover-aborted
reslist/joblist leaves ctx->reply.u.get == &getreply (a dead STACK pointer)
because xce_get_with_condition/xce_resval only cleared the slot on the success
path, not the xce_get_req<0 / cbuf_get<0 early returns. The next ctrlstatus
reply (worker thread) freed that foreign pointer as a CtrlStatus -> ASan
SEGV / stack-use-after-return in free_CtrlStatus (controller.capnp.c:11638).
Fix (root cause, both layers):
- Revert the unsafe free-guards in xce_handle_reply: MOVE_PTR overwrite only,
never free/deref an untagged union slot by type.
- Clear ctx->reply.u.get = NULL on ALL exit paths of xce_get_with_condition
and xce_resval via a single `goto done` cleanup (forget-proof; the bug was
one forgotten failover path). Also drops the now-unused controller.capnp.h.
- Defense-in-depth: handle_get_reply drops a reply when the slot is NULL.
Regression: expose xce_handle_reply and add
test_handle_reply_ctrlstatus_does_not_free_foreign_slot, which poisons the
shared slot with a foreign heap value and proves the ctrlstatus reply handler
does not free it (RED reproduced the exact free_CtrlStatus_ptr double-free;
GREEN after fix). xcelib_test 244/0, controller_unit_test 166/0, all binaries
build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MSxwb2Xt5F84ATgiAZnH8V
- Mark the PreVote denial log-reason nit RESOLVED in ha-prevote tasks notes (47f7338).
- File ha-elect-ctx-lifecycle spec scaffold (spec.json + brief.md) capturing the
pre-existing, bounded, UAF-sensitive election-carrier leak and the refcount/
teardown-ordered-free approach across both election and PreVote paths. Deferred
by decision — not folded into ha-prevote to avoid a risky lifetime refactor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
3 RED carriers (duplicate FINISH on DONE appends a 2nd WAL entry +
overwrites state; START on terminal clobbers it; replay non-equivalent)
+ 2 preservation pins (forward postdone PDONE-merge, fail-after-finish
PERR-merge) that 6.2 must not break. Reviewer-confirmed: the
UPDATE-clears-terminal arm is production-unreachable (pinned, not
driven). 22 tests / 3 RED; census 508 distinct triples, only those 3
fail.
configAck branch: per-push majority (acked_config_lsn >= push_tag,
1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate,
ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead.
push_config_to_standbys returns the barrier result; upgrade failure ->
send_upgrade_error (rc forwarded), no success ACK, no restart,
committed_config_tag success-only.
Remediation r1 (two CRITICAL review findings):
- F1: wire production config-ACK ingestion — replconfigpushreplymsg
decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack
on the promoted-vote ep; + defensive cluster.c OPCode_reply case.
Closes the unlanded half of ha-config-bootstrap 8.1; without this the
barrier would brick every multi-member config upgrade. No schema change.
- F2: keep the barrier off the iomgr thread — export
push_config_transport_only (non-blocking) for the 5 ha.c + join.c
streaming sites; barrier path now vtask-only (do_upgrade_config).
Wire-path test drives real capnp ACK bytes through the production decode
branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0
FAIL; pre-existing fixture leak only (less than baseline).
Green integration proof: 4 facts driven during total quorum loss append
RETAIN with rc 0 in <1000ms (vs 5000ms timeout — wrongly-applied WAIT
tripwire), pending with retained-depth=4 logged, present exactly once
on disk (real WAL header scan); reconnect + one covering ACK commits
the suffix with current_lsn unchanged (no redelivery). Completes the
bug-396 fix evidence (umbrella Req 2.8). Census 495/495.
7 cases: two covered short-circuits, timeout with committed_lsn
unmoved, covering-ACK release, sync_timeout_ms fallback (bounded both
directions), n_peers=1 standalone, NOT_READY passthrough via cap
saturation. Watermark resolved to current_lsn - 1 (design's literal
phrasing unsatisfiable per persist.h:71 invariant) with off-by-one
killing power both ways. 15 tests / 7 RED; suite baseline zero.
Verified table implemented (STOP/RESUME/UPDATE/FAIL RETAIN per
supersession; WORKER detail switch; CHECKPOINT LOCAL); unknown type =
ERROR + XCE_TESTING abort + production RETAIN fallback. All three
hooks route through wal_hook_append_classified (zero bare async
appends); RETAIN registers with the engine lock-free; facts propagate
local append rc only. 8/8 group tests green; full suite 465/465.
replicator_t gains ce pointer (set-once-before-traffic contract).
repl_on_ack feeds commit_on_ack after releasing r->lock (iomgr-safe);
four membership mutators feed commit_on_membership_change post-unlock,
gated on actual mutation. walAck branch delegates to commit_submit(WAIT)
when ce wired AND entry_lsn != 0 (LSN-0 interim guard until 5.3);
legacy counting kept for ce==NULL, marked DEPRECATED. commit.c linked
into production target (inert until 5.3 wiring). Tests 30-31 + all
repl_majority walAck cases green; 33/34 remain RED for 5.3; baseline
unchanged.
Tests 21-28: repl_active_standby_count contract (5 RED vs inert -1
stub) + 3 gate-level pins (n_peers<=1 always-true, zero-connected
reject, stateless re-accept) that 4.2 must preserve. Declaration in
repl.h; ha.c/ha.h untouched (4.2 boundary). 28 tests / 5 RED; suite
at the 3-failure baseline with unchanged signatures.
Tests 17-20: catch-up ACK exclusion vs pending waiter, membership-change
wake of covered waiter, removal-below-majority fails at own deadline
(no forced early-fail), non-blocking bound. Landed GREEN: task 1.2's
commit_on_membership_change already implements the full 3.2 contract,
so RED was unachievable; reviewer confirmed mutation-killing power and
zero production changes. 20/20 group tests; suite at baseline.
