fix: close bug-847 (un-ignore fixed reproducer), confirm bug-1307 fixed via soak
bug-847 (catch-up WAL batch drops payloads across capnp segment boundaries): its purpose-built RED reproducer, test_wal_batch_roundtrip_large_payloads (14 entries x 872-byte payloads, 12.2KB total, genuinely spans multiple 4096-byte capnp segments per capn-malloc.c's create()), was still TEST_IGNORE_MESSAGE'd. Un-ignoring it and running under ASAN with leak detection: clean pass, byte-exact payloads, zero leaks, 2/2 runs. Traced the vendored c-capnproto far-pointer machinery (write_ptr/capn_setp in capn.c) to confirm the cross-segment struct->Data reference path is correct by construction — whatever regressed this already got fixed elsewhere (library bump or an incidental request.c change); the ignore was just never removed. No production code changed. bug-1307 (task-5.7 election-jitter data-safety regression: 7 divergences + 4 job losses over 6 chaotic rounds): re-ran the exact reproducing scenario (6-round chaotic soak, real ext4, ASAN binaries) — 21 jobs, 0 divergence, 0 job loss, clean PASS. Confirms the raft-ha-core-safety Phase 5 work resolved it. Both bugs were the last 2 of the 7 flagged in the prior staleness sweep; all 7 are now closed (6 were already fixed and undetected, 1 needed the test un-ignored). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
This commit is contained in:
parent
acddc3204a
commit
78e8ec817e
3 changed files with 28 additions and 22 deletions
|
|
@ -13997,18 +13997,21 @@
|
|||
"error_message": "Catch-up WAL streaming silently DROPS entry payloads at scale: a rejoining controller (ctrl-1 restarted) received [1..14] with correct LSN/type/CRC but PLEN=0 / empty job blobs (waldump PCRC=00000000, jobid=?). Structurally-intact but content-empty WAL = silent data corruption. THE committed-job-loss mechanism.",
|
||||
"file": "src/lib/request.c (xce_wal_batch_encode_raw / wal_entry_to_wire / xce_decode extract loop) \u2014 capnp multi-segment Data pointers",
|
||||
"root_cause": "Multi-segment capnp bug. xce_wal_batch_encode_raw builds a WalCatchupEntry struct-list where each element's payload is a capnp Data (List(UInt8)) allocated via capn_new_list8(cs=root_seg,...). When the message outgrows its first segment (large batch), the per-element payload Data pointer is lost across the segment boundary \u2014 on decode extract_data_bytes(we.payload) reads CAPN_NULL and returns len 0. write_WalCatchupEntry uses capn_setp (far-pointer-capable) so the fault is in the overflow/segment handling as used here. REPRODUCED deterministically: src/lib/testsuite/test_wal_wire.c test_wal_batch_roundtrip_large_payloads (14 entries x 872B) FAILS 'Expected 872 Was 0'; the 2-entry test_wal_batch_roundtrip_payload_bytes PASSES. Threshold is size, not count. Live replication (repl_broadcast) carries payloads fine \u2014 ONLY the catch-up/rejoin path corrupts, which is why loss only appears after a node restarts and catches up.",
|
||||
"fix": "STILL OPEN as of 2026-07-02 \u2014 confirmed via re-verification, not a stale report. src/lib/testsuite/test_wal_wire.c:107 test_wal_batch_roundtrip_large_payloads is still TEST_IGNORE_MESSAGE'd citing this bug ID; the RED reproducer has not been un-ignored. Genuine remaining gap \u2014 needs an owner.",
|
||||
"fix": "FIXED \u2014 CONFIRMED via un-ignoring its own purpose-built RED reproducer (strongest possible evidence). Removed the TEST_IGNORE_MESSAGE guard from test_wal_batch_roundtrip_large_payloads (src/lib/testsuite/test_wal_wire.c); rebuilt and ran it 2x under ASAN with leak detection ON \u2014 clean PASS both times, all 14 entries' 872-byte payloads byte-exact, zero leaks (285/285 lib tests, 0 failures, 0 ignored). Root-cause investigation traced the encode path (xce_wal_batch_encode_raw / wal_entry_to_wire in src/lib/request.c) and the underlying c-capnproto far-pointer machinery (write_ptr/capn_setp in capn.c) \u2014 the cross-segment struct->Data far-pointer path is correct by construction (new_object relocates and re-tags .seg on segment overflow; write_ptr emits a proper far/double-far pointer when segments differ). capn-malloc.c's create() confirms genuine multi-segment allocation occurs at this test's scale (4096-byte segments, 12.2KB of payload), so the test IS exercising the real segment-boundary path, not a false negative from staying single-segment. No production code changed \u2014 whatever regressed this (library version bump via the documented nix flake repin, or an incidental request.c change) already fixed it; the TEST_IGNORE was simply never removed. This is the 6th bug in this sweep found to be already-fixed-but-untracked (846/855/1096/1147/1198/847) \u2014 buglog.json has significant staleness debt from fixes landing without their originating bug entries being closed.",
|
||||
"tags": [
|
||||
"replication",
|
||||
"critical",
|
||||
"data-loss",
|
||||
"reproducer-unignored",
|
||||
"corruption",
|
||||
"ha",
|
||||
"wal",
|
||||
"waldump",
|
||||
"stale-report",
|
||||
"reverified-open",
|
||||
"capnp",
|
||||
"catchup",
|
||||
"waldump"
|
||||
"data-loss",
|
||||
"replication",
|
||||
"capnp",
|
||||
"critical",
|
||||
"wal",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-828",
|
||||
|
|
@ -14016,7 +14019,7 @@
|
|||
"bug-661"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-02T17:30:00.000Z"
|
||||
"last_seen": "2026-07-03T03:15:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-855",
|
||||
|
|
@ -21704,17 +21707,18 @@
|
|||
"error_message": "Task 5.7 election-resonance jitter (74784b2) introduced data-safety REGRESSION: 6-round chaotic soak shows 7 committed divergences (LSNs 8,12,16,19) + 4 committed job losses (job 0-0-1782831451-1). Previous run (pre-5.7, commit 81cb09a) had ZERO divergences and ZERO job losses through 6 rounds. LSN 19 divergence is present=3 (ALL 3 members have different content at same LSN \u2014 silent divergence, textbook bug-851). Election liveness improved (all 6 rounds complete vs stalled at round 7), but the jitter defeats the election-safety balance: faster-starting node wins before catching up on committed prefix \u2192 Leader Completeness violation.",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "Task 5.7 jittered watchdog ([T,2T] randomized) lets a less-complete node start and win an election before catching up on the full committed prefix from the dead primary. With the original fixed watchdog, both survivors fired RDHUP simultaneously \u2192 cross-denied \u2192 the library's priority stagger resolved the tie safely (the fresher node won after deterministic desynchronization). With jitter, one node can fire much earlier (near T) and the other much later (near 2T) \u2192 the early node campaigns with an incomplete log \u2192 voters grant based on stale view \u2192 winner truncates committed entries it never held. The 5.3 in_catch_up release (which drops the vote-deny when the primary is dead) opens a window where a still-catching-up candidate can win \u2014 with the new jitter timings this window is hit frequently.",
|
||||
"fix": "LIKELY FIXED but not code-confirmed (checked 2026-07-02): no direct 'bug-1307' code citation found. Circumstantial evidence it's resolved: raft-ha-core-safety Phase 5 ('Election liveness and safety under churn') is fully complete (all leaf tasks checked) and soak-validated at 0 committed-prefix divergences / 0 job losses across 6-round chaotic soaks per session memory \u2014 the exact scenario this bug's regression was found in. Re-run the 6-round chaotic soak once more before fully closing this entry; downgrading from 'not yet fixed' to 'probably fixed, wants one more soak confirmation' rather than asserting closed outright.",
|
||||
"fix": "CONFIRMED FIXED via live re-soak (2026-07-03): ran the exact reproducing scenario \u2014 6-round chaotic soak (overlapping kill-the-primary, real ext4 at ~/tmp, ASAN controller binary) \u2014 that originally showed 7 committed divergences + 4 job losses. This run: 21 jobs submitted, 0 committed_divergence, 0 committed_job_loss, PASS across all 6 rounds (terms 3->9, all 3 nodes rotated through primary). Matches the earlier circumstantial evidence (raft-ha-core-safety Phase 5 fully complete + prior soak-clean sessions) with a fresh, direct confirmation run \u2014 no residual doubt.",
|
||||
"tags": [
|
||||
"committed-divergence",
|
||||
"bug-1296-followup",
|
||||
"task-5.7",
|
||||
"needs-confirmation-soak",
|
||||
"election-liveness",
|
||||
"leader-completeness-violation",
|
||||
"chaotic-soak",
|
||||
"data-safety-regression",
|
||||
"committed-job-loss"
|
||||
"task-5.7",
|
||||
"chaotic-soak",
|
||||
"leader-completeness-violation",
|
||||
"confirmed-via-soak",
|
||||
"committed-job-loss",
|
||||
"election-liveness",
|
||||
"bug-1296-followup",
|
||||
"committed-divergence",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1296",
|
||||
|
|
@ -21724,7 +21728,7 @@
|
|||
"bug-851"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-02T17:30:00.000Z"
|
||||
"last_seen": "2026-07-03T03:15:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1308",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue