fix: close bug-847 (un-ignore fixed reproducer), confirm bug-1307 fixed via soak
bug-847 (catch-up WAL batch drops payloads across capnp segment boundaries): its purpose-built RED reproducer, test_wal_batch_roundtrip_large_payloads (14 entries x 872-byte payloads, 12.2KB total, genuinely spans multiple 4096-byte capnp segments per capn-malloc.c's create()), was still TEST_IGNORE_MESSAGE'd. Un-ignoring it and running under ASAN with leak detection: clean pass, byte-exact payloads, zero leaks, 2/2 runs. Traced the vendored c-capnproto far-pointer machinery (write_ptr/capn_setp in capn.c) to confirm the cross-segment struct->Data reference path is correct by construction — whatever regressed this already got fixed elsewhere (library bump or an incidental request.c change); the ignore was just never removed. No production code changed. bug-1307 (task-5.7 election-jitter data-safety regression: 7 divergences + 4 job losses over 6 chaotic rounds): re-ran the exact reproducing scenario (6-round chaotic soak, real ext4, ASAN binaries) — 21 jobs, 0 divergence, 0 job loss, clean PASS. Confirms the raft-ha-core-safety Phase 5 work resolved it. Both bugs were the last 2 of the 7 flagged in the prior staleness sweep; all 7 are now closed (6 were already fixed and undetected, 1 needed the test un-ignored). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
This commit is contained in:
parent
acddc3204a
commit
78e8ec817e
3 changed files with 28 additions and 22 deletions
|
|
@ -13997,18 +13997,21 @@
|
|||
"error_message": "Catch-up WAL streaming silently DROPS entry payloads at scale: a rejoining controller (ctrl-1 restarted) received [1..14] with correct LSN/type/CRC but PLEN=0 / empty job blobs (waldump PCRC=00000000, jobid=?). Structurally-intact but content-empty WAL = silent data corruption. THE committed-job-loss mechanism.",
|
||||
"file": "src/lib/request.c (xce_wal_batch_encode_raw / wal_entry_to_wire / xce_decode extract loop) \u2014 capnp multi-segment Data pointers",
|
||||
"root_cause": "Multi-segment capnp bug. xce_wal_batch_encode_raw builds a WalCatchupEntry struct-list where each element's payload is a capnp Data (List(UInt8)) allocated via capn_new_list8(cs=root_seg,...). When the message outgrows its first segment (large batch), the per-element payload Data pointer is lost across the segment boundary \u2014 on decode extract_data_bytes(we.payload) reads CAPN_NULL and returns len 0. write_WalCatchupEntry uses capn_setp (far-pointer-capable) so the fault is in the overflow/segment handling as used here. REPRODUCED deterministically: src/lib/testsuite/test_wal_wire.c test_wal_batch_roundtrip_large_payloads (14 entries x 872B) FAILS 'Expected 872 Was 0'; the 2-entry test_wal_batch_roundtrip_payload_bytes PASSES. Threshold is size, not count. Live replication (repl_broadcast) carries payloads fine \u2014 ONLY the catch-up/rejoin path corrupts, which is why loss only appears after a node restarts and catches up.",
|
||||
"fix": "STILL OPEN as of 2026-07-02 \u2014 confirmed via re-verification, not a stale report. src/lib/testsuite/test_wal_wire.c:107 test_wal_batch_roundtrip_large_payloads is still TEST_IGNORE_MESSAGE'd citing this bug ID; the RED reproducer has not been un-ignored. Genuine remaining gap \u2014 needs an owner.",
|
||||
"fix": "FIXED \u2014 CONFIRMED via un-ignoring its own purpose-built RED reproducer (strongest possible evidence). Removed the TEST_IGNORE_MESSAGE guard from test_wal_batch_roundtrip_large_payloads (src/lib/testsuite/test_wal_wire.c); rebuilt and ran it 2x under ASAN with leak detection ON \u2014 clean PASS both times, all 14 entries' 872-byte payloads byte-exact, zero leaks (285/285 lib tests, 0 failures, 0 ignored). Root-cause investigation traced the encode path (xce_wal_batch_encode_raw / wal_entry_to_wire in src/lib/request.c) and the underlying c-capnproto far-pointer machinery (write_ptr/capn_setp in capn.c) \u2014 the cross-segment struct->Data far-pointer path is correct by construction (new_object relocates and re-tags .seg on segment overflow; write_ptr emits a proper far/double-far pointer when segments differ). capn-malloc.c's create() confirms genuine multi-segment allocation occurs at this test's scale (4096-byte segments, 12.2KB of payload), so the test IS exercising the real segment-boundary path, not a false negative from staying single-segment. No production code changed \u2014 whatever regressed this (library version bump via the documented nix flake repin, or an incidental request.c change) already fixed it; the TEST_IGNORE was simply never removed. This is the 6th bug in this sweep found to be already-fixed-but-untracked (846/855/1096/1147/1198/847) \u2014 buglog.json has significant staleness debt from fixes landing without their originating bug entries being closed.",
|
||||
"tags": [
|
||||
"replication",
|
||||
"critical",
|
||||
"data-loss",
|
||||
"reproducer-unignored",
|
||||
"corruption",
|
||||
"ha",
|
||||
"wal",
|
||||
"waldump",
|
||||
"stale-report",
|
||||
"reverified-open",
|
||||
"capnp",
|
||||
"catchup",
|
||||
"waldump"
|
||||
"data-loss",
|
||||
"replication",
|
||||
"capnp",
|
||||
"critical",
|
||||
"wal",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-828",
|
||||
|
|
@ -14016,7 +14019,7 @@
|
|||
"bug-661"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-02T17:30:00.000Z"
|
||||
"last_seen": "2026-07-03T03:15:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-855",
|
||||
|
|
@ -21704,17 +21707,18 @@
|
|||
"error_message": "Task 5.7 election-resonance jitter (74784b2) introduced data-safety REGRESSION: 6-round chaotic soak shows 7 committed divergences (LSNs 8,12,16,19) + 4 committed job losses (job 0-0-1782831451-1). Previous run (pre-5.7, commit 81cb09a) had ZERO divergences and ZERO job losses through 6 rounds. LSN 19 divergence is present=3 (ALL 3 members have different content at same LSN \u2014 silent divergence, textbook bug-851). Election liveness improved (all 6 rounds complete vs stalled at round 7), but the jitter defeats the election-safety balance: faster-starting node wins before catching up on committed prefix \u2192 Leader Completeness violation.",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "Task 5.7 jittered watchdog ([T,2T] randomized) lets a less-complete node start and win an election before catching up on the full committed prefix from the dead primary. With the original fixed watchdog, both survivors fired RDHUP simultaneously \u2192 cross-denied \u2192 the library's priority stagger resolved the tie safely (the fresher node won after deterministic desynchronization). With jitter, one node can fire much earlier (near T) and the other much later (near 2T) \u2192 the early node campaigns with an incomplete log \u2192 voters grant based on stale view \u2192 winner truncates committed entries it never held. The 5.3 in_catch_up release (which drops the vote-deny when the primary is dead) opens a window where a still-catching-up candidate can win \u2014 with the new jitter timings this window is hit frequently.",
|
||||
"fix": "LIKELY FIXED but not code-confirmed (checked 2026-07-02): no direct 'bug-1307' code citation found. Circumstantial evidence it's resolved: raft-ha-core-safety Phase 5 ('Election liveness and safety under churn') is fully complete (all leaf tasks checked) and soak-validated at 0 committed-prefix divergences / 0 job losses across 6-round chaotic soaks per session memory \u2014 the exact scenario this bug's regression was found in. Re-run the 6-round chaotic soak once more before fully closing this entry; downgrading from 'not yet fixed' to 'probably fixed, wants one more soak confirmation' rather than asserting closed outright.",
|
||||
"fix": "CONFIRMED FIXED via live re-soak (2026-07-03): ran the exact reproducing scenario \u2014 6-round chaotic soak (overlapping kill-the-primary, real ext4 at ~/tmp, ASAN controller binary) \u2014 that originally showed 7 committed divergences + 4 job losses. This run: 21 jobs submitted, 0 committed_divergence, 0 committed_job_loss, PASS across all 6 rounds (terms 3->9, all 3 nodes rotated through primary). Matches the earlier circumstantial evidence (raft-ha-core-safety Phase 5 fully complete + prior soak-clean sessions) with a fresh, direct confirmation run \u2014 no residual doubt.",
|
||||
"tags": [
|
||||
"committed-divergence",
|
||||
"bug-1296-followup",
|
||||
"task-5.7",
|
||||
"needs-confirmation-soak",
|
||||
"election-liveness",
|
||||
"leader-completeness-violation",
|
||||
"chaotic-soak",
|
||||
"data-safety-regression",
|
||||
"committed-job-loss"
|
||||
"task-5.7",
|
||||
"chaotic-soak",
|
||||
"leader-completeness-violation",
|
||||
"confirmed-via-soak",
|
||||
"committed-job-loss",
|
||||
"election-liveness",
|
||||
"bug-1296-followup",
|
||||
"committed-divergence",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1296",
|
||||
|
|
@ -21724,7 +21728,7 @@
|
|||
"bug-851"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-02T17:30:00.000Z"
|
||||
"last_seen": "2026-07-03T03:15:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1308",
|
||||
|
|
|
|||
|
|
@ -354,3 +354,9 @@ Attempted to validate ha-wal-replication 5.4 ("job appears in joblist on primary
|
|||
|
||||
## Key Learning (2026-07-02, bug-1401 FIXED) — a generated codec's `free_X(struct*)` vs `free_X_ptr(struct**)` split is easy to get wrong in a hand-written client wrapper
|
||||
`decode_X_ptr()` (capnp-generated) `calloc`s the top-level struct THEN fills its fields; the matching free is `free_X_ptr(&ptr)` (frees fields + the struct + nulls the caller's pointer), NOT `free_X(ptr)` (fields only — the codegen intentionally splits these because `free_X` is also reused on stack-allocated structs, e.g. this project's own `test_ctrlstatus_roundtrip_all_fields` test calls `free_CtrlStatus(&out)` on a stack var, which is correct there but wrong for a heap ptr). `src/include/client.h`'s `xce_free_ctrlstatus(s)` macro used the fields-only variant, leaking `sizeof(xce_ctrlstatus_t)` (120 bytes) on every real `ctrlstatus` call. **Root cause of it going undetected**: every existing ctrlstatus unit test built its `xce_ctrlstatus_t` via the stack-decode helper (`decode_CtrlStatus`) and freed it with the correct raw `free_CtrlStatus(&out)` call directly — none of them ever exercised the actual client-facing `xce_free_ctrlstatus()` macro that real code paths use. **Rule going forward**: whenever a generated codec exposes both a bare `free_X` and a `free_X_ptr`, any hand-written `xce_free_*`/`xcecli_free_*` wrapper macro over it MUST use `free_X_ptr(&(s))` if the value being freed came from a `decode_X_ptr()` heap allocation (check what the corresponding `xce_X(...)` API call's `MOVE_PTR` source was decoded with, not just what looks intuitively "free-shaped"). And: a roundtrip test that exercises only the raw encode/decode/free primitives is NOT a substitute for a test that goes through the actual client macro/wrapper — add at least one regression per free-wrapper that uses the real heap-allocating decode path + the real macro. RED confirmed the exact 120-byte/1-alloc LeakSanitizer signature from the bug report before applying the fix.
|
||||
|
||||
## Key Learning (2026-07-03) — buglog.json's biggest lie is silence: 6 of 7 swept "open" bugs were already fixed and just never marked
|
||||
Swept 7 old "not yet fixed" buglog entries (846, 847, 855, 1096, 1147, 1198, 1307) by checking the ACTUAL current code/tests instead of trusting the buglog text — 6 were already fixed (some for over a week), only bug-847 needed real action (and even that action was "un-ignore the regression test," not a production code change — its underlying condition was already gone). Common thread: every one of these had a purpose-built regression test or an explicit in-code citation that would have revealed the fix immediately, but nobody closed the loop after landing the fix under a DIFFERENT bug number or spec task. **Rule going forward**: before spending real investigation budget on an "open" bug, first check (a) does it have a named regression test — run it directly; (b) does its bug ID appear in a code comment — grep for it; (c) does its described symptom show up in the relevant spec's tasks.md as a since-closed task. All three are cheap (single tool call) and would have caught 6/7 of these immediately. **Second rule**: when you land a fix, grep buglog.json for the bug ID(s) it plausibly closes (including "family" bugs like bug-851's many sub-numbers) BEFORE moving to the next task — don't rely on remembering to circle back later, because "later" doesn't reliably happen and staleness compounds silently. [[project_ha_bootstrap_progress]]
|
||||
|
||||
## Key Learning (2026-07-03, bug-847 investigation) — trace the actual library mechanism before trusting a plausible-sounding root-cause hypothesis
|
||||
A prior investigation (and a fresh subagent re-investigation) both converged on "the encoder captures a capnp segment pointer once outside a loop and reuses it stale across iterations" as bug-847's root cause — plausible-sounding, cites real code (`cs = capn_root(&c).seg;` before a `for` loop in `xce_wal_batch_encode_raw`, src/lib/request.c). Tracing the ACTUAL c-capnproto library (`new_object`/`new_data`/`write_ptr`/`capn_setp` in the vendored `capn.c`) showed this is a non-issue by design: `new_object` always re-resolves and re-tags `.seg` on segment overflow regardless of what stale segment pointer was passed in, and `write_ptr`/`capn_setp` correctly emit far/double-far pointers whenever the source and target objects land in different segments. The theory was never tested against the library source, only against the observed symptom — and the observed symptom (RED test failure) turned out to no longer reproduce at all once actually run. Lesson: a hypothesis that "sounds mechanistically right" and cites real line numbers is not the same as a verified root cause — for a data-loss bug specifically, always run the existing/write a NEW regression test before accepting a hypothesis, even one two independent investigations agreed on.
|
||||
|
|
|
|||
|
|
@ -104,10 +104,6 @@ static void test_wal_batch_roundtrip_large_payloads(void)
|
|||
* this TEST_IGNORE line when bug-847 is fixed and assert the payloads
|
||||
* survive. Decisive evidence: this fails "Expected 872 Was 0" while the
|
||||
* 2-entry round-trip above passes. */
|
||||
TEST_IGNORE_MESSAGE(
|
||||
"bug-847: catch-up batch drops payloads across capnp segments — "
|
||||
"un-ignore when fixed");
|
||||
|
||||
enum { N = 14, PSZ = 872 };
|
||||
wal_catchup_entry_t e[N];
|
||||
wal_catchup_entry_t *entries[N];
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue