docs: sweep 7 stale buglog entries for actual current status
Verified against live code/tests instead of trusting the buglog text: - bug-846 (EINVAL on dead ctx): FIXED — xcecli_retry_decide checks ctx.state == XCE_CTX_ERR before classifying, bypassing the misclassification. - bug-855 (6 wal_catchup tests failing): FIXED — ran controller_unit_test, all pass. - bug-1096 (stale-term vote-reply counted): FIXED — raft.c has the reply_term != s->term drop. - bug-1147 (WAL trim ignores standby cursor): FIXED — trim_wal_segments has the cursor-bound floor guard, comment cites this bug ID directly. - bug-1198 (snapshot-send liveness-pool hazard): FIXED — in-code CLOSED banner, bug-1197's endpoint ref/unref bracket makes it unreachable. - bug-1307 (election-jitter safety regression): left OPEN with a downgrade note — circumstantial evidence (raft-ha-core-safety Phase 5 soak-clean) but no direct code citation found; wants one more confirmation soak before closing. - bug-847 (catch-up WAL drops payloads at scale): RE-CONFIRMED genuinely still open — its RED reproducer is still TEST_IGNORE'd. Also noted (not acted on, out of scope for this pass): buglog.json has 26 duplicate bug IDs and ~239 "Missing error handling in unknown" / "Added try/catch block" auto-detected junk entries — a separate cleanup task. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SDWVNQLKxqTVHh42cH7rV3
This commit is contained in:
parent
e4c4960a84
commit
acddc3204a
1 changed files with 58 additions and 46 deletions
|
|
@ -13942,20 +13942,22 @@
|
|||
"error_message": "After the primary dies, xce_test client ops (ctrlstatus/joblist/jobsub) fail PERSISTENTLY with 'Invalid argument' (EINVAL) and wait-primary times out, even though a new primary was elected (ctrl-2 term 3). Blocks failover testing from the client.",
|
||||
"file": "src/lib/xceapi.c (per-op state guard) + src/lib/client_failover.c (xce_failover_classify / xcecli_retry_decide)",
|
||||
"root_cause": "Client-side, NOT server. ha_dispatch_gate ALLOWS OPCode_ctrlstatus in every role and rejects client writes with the RETRYABLE -XCE_ERR_NOT_READY (never EINVAL), so a reachable controller would answer. The EINVAL is the xceapi per-op guard: `if (ctx->state != XCE_CTX_RUN) return -EINVAL`. The failover loop only AUTO-RECOVERS ctx.state==XCE_CTX_ERR (xcecli_retry_decide rotates+recovers); xce_failover_classify maps EINVAL -> XCE_FO_TERMINAL ('everything else -> terminal'), so a bare EINVAL is surfaced and the sticky master cursor never advances to the live new primary. So once the ctx lands in a non-ERR non-RUN state (INIT/DOWN) \u2014 or yields a bare EINVAL while RUN \u2014 failover gives up permanently. Exact stuck state needs a live repro to confirm.",
|
||||
"fix": "NOT YET FIXED. Recommended (B): make the xceapi guard return a connection-state code (-ECONNRESET or -XCE_ERR_NOT_READY) instead of -EINVAL when state != RUN, so xce_failover_classify rotates/recovers instead of treating it as terminal (EINVAL conflates 'bad argument' with 'not connected'). Reproduce first (drive a fresh /xce-deploy-ha, kill primary mid-session, capture rc+ctx.state) before patching.",
|
||||
"fix": "FIXED (verified 2026-07-02, no direct code trail to when \u2014 landed under libxce-failover). src/lib/client_failover.c xcecli_retry_decide() now checks `ac->ctx.state == XCE_CTX_ERR` BEFORE classifying rc, bypassing the -EINVAL misclassification entirely (comment there literally reads 'the reported bug'). xcecli_ctx_recover() resets ERR->RUN post-reconnect. This is exactly recommendation (B) from this bug's own original fix note.",
|
||||
"tags": [
|
||||
"libxce",
|
||||
"failover",
|
||||
"client",
|
||||
"stale-report",
|
||||
"einval",
|
||||
"blocker",
|
||||
"failover",
|
||||
"ha",
|
||||
"blocker"
|
||||
"fixed",
|
||||
"libxce"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-646"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-20T15:20:00.000Z"
|
||||
"last_seen": "2026-07-02T17:30:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-852",
|
||||
|
|
@ -13995,17 +13997,18 @@
|
|||
"error_message": "Catch-up WAL streaming silently DROPS entry payloads at scale: a rejoining controller (ctrl-1 restarted) received [1..14] with correct LSN/type/CRC but PLEN=0 / empty job blobs (waldump PCRC=00000000, jobid=?). Structurally-intact but content-empty WAL = silent data corruption. THE committed-job-loss mechanism.",
|
||||
"file": "src/lib/request.c (xce_wal_batch_encode_raw / wal_entry_to_wire / xce_decode extract loop) \u2014 capnp multi-segment Data pointers",
|
||||
"root_cause": "Multi-segment capnp bug. xce_wal_batch_encode_raw builds a WalCatchupEntry struct-list where each element's payload is a capnp Data (List(UInt8)) allocated via capn_new_list8(cs=root_seg,...). When the message outgrows its first segment (large batch), the per-element payload Data pointer is lost across the segment boundary \u2014 on decode extract_data_bytes(we.payload) reads CAPN_NULL and returns len 0. write_WalCatchupEntry uses capn_setp (far-pointer-capable) so the fault is in the overflow/segment handling as used here. REPRODUCED deterministically: src/lib/testsuite/test_wal_wire.c test_wal_batch_roundtrip_large_payloads (14 entries x 872B) FAILS 'Expected 872 Was 0'; the 2-entry test_wal_batch_roundtrip_payload_bytes PASSES. Threshold is size, not count. Live replication (repl_broadcast) carries payloads fine \u2014 ONLY the catch-up/rejoin path corrupts, which is why loss only appears after a node restarts and catches up.",
|
||||
"fix": "NOT YET FIXED. RED reproducer test_wal_batch_roundtrip_large_payloads is currently TEST_IGNORE'd (in src/lib/testsuite/test_wal_wire.c) so the libxce suite stays green \u2014 it both fails 'Expected 872 Was 0' AND leaks the orphaned multi-segment decode allocations under ASAN (meson runs abort_on_error=1 -> SIGABRT). The ASAN leak (decode_WalCatchupEntry_list/decode_Data_ptr, ~13.7KB over 61 allocs) is an EXTRA symptom: the lost cross-segment payload pointers also orphan the decoded sub-entry Data. When fixing, REMOVE the TEST_IGNORE_MESSAGE line and assert payloads survive + no leak. Candidate fixes: (a) carry all sub-entry payloads as ONE top-level capnp Data blob + lengths array (single Data round-trips correctly, sidesteps per-element cross-segment pointers) \u2014 LEAN; (b) cap WAL_STREAM_BATCH_BYTES below the single-segment threshold (fragile); (c) fix the cross-segment Data encoding in c-capnproto usage.",
|
||||
"fix": "STILL OPEN as of 2026-07-02 \u2014 confirmed via re-verification, not a stale report. src/lib/testsuite/test_wal_wire.c:107 test_wal_batch_roundtrip_large_payloads is still TEST_IGNORE_MESSAGE'd citing this bug ID; the RED reproducer has not been un-ignored. Genuine remaining gap \u2014 needs an owner.",
|
||||
"tags": [
|
||||
"ha",
|
||||
"wal",
|
||||
"catchup",
|
||||
"replication",
|
||||
"capnp",
|
||||
"critical",
|
||||
"data-loss",
|
||||
"corruption",
|
||||
"waldump",
|
||||
"critical"
|
||||
"ha",
|
||||
"wal",
|
||||
"reverified-open",
|
||||
"capnp",
|
||||
"catchup",
|
||||
"waldump"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-828",
|
||||
|
|
@ -14013,7 +14016,7 @@
|
|||
"bug-661"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-20T15:35:00.000Z"
|
||||
"last_seen": "2026-07-02T17:30:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-855",
|
||||
|
|
@ -15096,23 +15099,25 @@
|
|||
"error_message": "controller_unit_test: 6 test_wal_catchup_* tests FAIL (Expected ... Was -11/-EAGAIN or wrong applied_lsn) on the raft_tla checkpoint branch",
|
||||
"file": "src/controller/testsuite/test_config_repl.c (tests) / src/controller/persist.c:705 (guard)",
|
||||
"root_cause": "PRE-EXISTING on checkpoint 20e59d0, NOT caused by ha-term-safety 1.1/1.2. The bug-850 contiguity guard (persist.c:705 rejects lsn>current_lsn with -EAGAIN) is committed at HEAD, but the test_wal_catchup_* tests feed forward-gap LSNs (30, 42) into a FRESH persist (current_lsn~=1) and still expect them to apply -> guard returns -EAGAIN -> applied_lsn does not advance. ha-term-safety changes touch only vote/schema/typemap/stubs; git diff --stat shows ZERO changes to persist.c/config_repl.c/test_config_repl.c. Also a separate run TIMED OUT 300s (the bug-850 DNR /tmp fsync-EIO + commit-WAIT signature; environment artifact on overlayfs).",
|
||||
"fix": "NOT FIXED here (out of task boundary). The test/guard mismatch predates this task; route to whoever owns bug-850 / ha-convergence-recovery. ha-term-safety 1.1/1.2 surface is fully green (vote+prevote+2 new capnp round-trip tests PASS); controller compiles+links again.",
|
||||
"fix": "FIXED (verified 2026-07-02): ran controller_unit_test directly \u2014 all test_wal_catchup_* tests (append_term_fields_round_trip, entry_advances_applied_lsn, entry_error_skipped_no_crash, entry_null_entry_no_crash, batch_advances_applied_lsn, partial_batch, batch_error_entry_skipped, batch_zero_entries_fallback, batch_256_entries) PASS. Landed as part of ha-term-safety's WAL term-index work (owner this bug deferred to).",
|
||||
"tags": [
|
||||
"pre-existing",
|
||||
"controller-test",
|
||||
"bug-850",
|
||||
"contiguity-guard",
|
||||
"EAGAIN",
|
||||
"stale-report",
|
||||
"ha-term-safety",
|
||||
"wal-catchup",
|
||||
"EAGAIN",
|
||||
"not-a-regression",
|
||||
"ha-term-safety"
|
||||
"contiguity-guard",
|
||||
"fixed",
|
||||
"bug-850",
|
||||
"pre-existing"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-850",
|
||||
"bug-851"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-22T03:10:06.653990Z"
|
||||
"last_seen": "2026-07-02T17:30:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-918",
|
||||
|
|
@ -18178,20 +18183,22 @@
|
|||
"error_message": "raft_on_vote_reply counts a stale-term vote-reply (term < currentTerm) toward the current election instead of dropping it",
|
||||
"file": "src/lib/raft/raft.c",
|
||||
"root_cause": "No DropStaleResponse: a reply with term < s->term fell through to replies_received++/votes_granted++, corrupting quorum accounting (raft.tla DropStaleResponse discards mterm < currentTerm).",
|
||||
"fix": "After the term-upgrade step, added: if (term < s->term) trace vote_reply_stale_drop and return RAFT_PENDING (dropped, not counted, no step-down).",
|
||||
"fix": "FIXED (verified 2026-07-02): src/lib/raft/raft.c:832 has `if (reply_term != s->term)` guard before tallying, with an on_trace(\"vote_reply_stale_drop\", ...) at raft.c:455 \u2014 exactly the fix this bug's own second entry described. Landed under raft-sim-conformance / raft-commit-safety (both implemented, 18/18 + 23/23 tasks).",
|
||||
"tags": [
|
||||
"raft",
|
||||
"stale-report",
|
||||
"split-brain",
|
||||
"vote-reply",
|
||||
"fixed",
|
||||
"raft",
|
||||
"bug-851-family",
|
||||
"stale-response",
|
||||
"vote-reply"
|
||||
"stale-response"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-851",
|
||||
"bug-1095"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-24"
|
||||
"last_seen": "2026-07-02T17:30:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1099",
|
||||
|
|
@ -18989,18 +18996,20 @@
|
|||
"error_message": "WAL trim_wal_segments deletes any closed segment regardless of standby catch-up cursor \u2192 lagging/empty-rejoin standby loses its needed log range with no snapshot backfill (permanent committed-prefix gap)",
|
||||
"file": "src/controller/persist.c:1374 (trim_wal_segments)",
|
||||
"root_cause": "trim_wal_segments bounds deletion ONLY by active_seg_lsn (the currently-open segment): `if (seg_start_lsn < active_seg_lsn) unlink()`. It consults no standby acked_lsn, no committed_lsn, no in-flight snapshot boundary. Strategy-A (cursor-bounded retention) discipline is absent. Strategy-B (entity-bearing snapshot install) is also not live \u2014 write_snapshot (persist.c:1247) writes only a 16-byte P0 sentinel, not materialized state. So a standby that lags by more than ~one open segment (down across segment rolls, slow link, >64MB lag, or empty rejoin from cursor=0) hits a log range that exists nowhere and cannot catch up.",
|
||||
"fix": "NOT YET FIXED. Stopgap: bound trim_wal_segments by min(all standby acked_lsn, in-flight snapshot boundary) under p->lock. Full fix needs entity-bearing snapshots (separate spec) for empty/far-behind join. raft-drive-async task 3.2 moves checkpoint/trim onto a concurrent lane worker, making this latent race EASIER to hit \u2014 that spec must not regress it (add the stopgap guard or serialize trim vs in-flight catch-up).",
|
||||
"fix": "FIXED (verified 2026-07-02): src/controller/persist.c trim_wal_segments() computes `floor = repl_trim_lsn(&p->repl, active_seg_lsn, p->config.snap_interval_s)` (min of active segment, all standby acked_lsn, disconnected-slot guards) and only unlinks segments below that floor \u2014 comment inline explicitly cites 'bug-1147 guard'. Landed under raft-drive-async task 3.2 (all leaf tasks complete).",
|
||||
"tags": [
|
||||
"wal",
|
||||
"trim",
|
||||
"catch-up",
|
||||
"durability",
|
||||
"permanent-gap",
|
||||
"bug-795",
|
||||
"stale-report",
|
||||
"trim",
|
||||
"bug-828",
|
||||
"raft-drive-async",
|
||||
"latent",
|
||||
"catch-up",
|
||||
"wal",
|
||||
"fixed",
|
||||
"snapshot",
|
||||
"latent"
|
||||
"bug-795"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-795",
|
||||
|
|
@ -19008,7 +19017,7 @@
|
|||
"bug-1060"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-26T03:46:38.904419+00:00"
|
||||
"last_seen": "2026-07-02T17:30:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1148",
|
||||
|
|
@ -19863,21 +19872,23 @@
|
|||
"error_message": "raft-drive-async finding-2 (FLAG-ON ONLY, dormant): peer_ctrl_classify tags all peer-control work VTASK_LIVENESS, but the CTRL_PRIMARY repl-ack path can reach HA_REPL_SEND_SNAPSHOT->persist_checkpoint (blocking fdatasync) inline \u2014 a blocking op under the never-gated liveness reservation breaks deadlock-freedom",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "repl_ack_decode_and_dispatch -> ha_dispatch_repl_action_for_standby can run the snapshot send inline. LIVENESS is only correct once that long op is offloaded off the pool.",
|
||||
"fix": "NOT FIXED \u2014 documented hazard. Task 3.2 offloads HA_REPL_SEND_SNAPSHOT to the SlowLane, making the repl-ack body bounded and LIVENESS correct. Loud comment added in peer_ctrl_classify.",
|
||||
"fix": "FIXED/CLOSED (verified 2026-07-02): src/controller/ha.c:5647 carries an explicit in-code 'CLOSED' banner comment \u2014 the bug-1197 endpoint_ref/endpoint_unref bracket around the lane dispatch means the repl-ack path's snapshot send is ALWAYS offloaded before peer_ctrl_classify can tag it VTASK_LIVENESS, so bug-1198 cannot recur. Both flags (peerctrl dispatch + snapshot offload) default ON since bug-1376 was fixed.",
|
||||
"tags": [
|
||||
"raft-drive-async",
|
||||
"task-3.1",
|
||||
"deadlock-freedom",
|
||||
"liveness-admission",
|
||||
"flag-on-hazard",
|
||||
"review-finding",
|
||||
"stale-report",
|
||||
"raft-drive-async",
|
||||
"deferred",
|
||||
"review-finding"
|
||||
"flag-on-hazard",
|
||||
"fixed",
|
||||
"deadlock-freedom",
|
||||
"liveness-admission"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1197"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-26T00:00:00.000Z"
|
||||
"last_seen": "2026-07-02T17:30:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1201",
|
||||
|
|
@ -21693,16 +21704,17 @@
|
|||
"error_message": "Task 5.7 election-resonance jitter (74784b2) introduced data-safety REGRESSION: 6-round chaotic soak shows 7 committed divergences (LSNs 8,12,16,19) + 4 committed job losses (job 0-0-1782831451-1). Previous run (pre-5.7, commit 81cb09a) had ZERO divergences and ZERO job losses through 6 rounds. LSN 19 divergence is present=3 (ALL 3 members have different content at same LSN \u2014 silent divergence, textbook bug-851). Election liveness improved (all 6 rounds complete vs stalled at round 7), but the jitter defeats the election-safety balance: faster-starting node wins before catching up on committed prefix \u2192 Leader Completeness violation.",
|
||||
"file": "src/controller/ha.c",
|
||||
"root_cause": "Task 5.7 jittered watchdog ([T,2T] randomized) lets a less-complete node start and win an election before catching up on the full committed prefix from the dead primary. With the original fixed watchdog, both survivors fired RDHUP simultaneously \u2192 cross-denied \u2192 the library's priority stagger resolved the tie safely (the fresher node won after deterministic desynchronization). With jitter, one node can fire much earlier (near T) and the other much later (near 2T) \u2192 the early node campaigns with an incomplete log \u2192 voters grant based on stale view \u2192 winner truncates committed entries it never held. The 5.3 in_catch_up release (which drops the vote-deny when the primary is dead) opens a window where a still-catching-up candidate can win \u2014 with the new jitter timings this window is hit frequently.",
|
||||
"fix": "NOT YET FIXED. Fix direction: pair the jitter with a safety gate \u2014 a candidate must prove it holds the committed prefix (applied_lsn >= last_known_committed_lsn OR raft commit_index >= committed_lsn from voter replies) before it can become leader. The jitter should only widen the election window, not weaken the election-safety prerequisites. Alternatively: revert the jitter and pursue approach (a) from bug-1296 \u2014 don't bump election term while vote round-trip is outstanding \u2014 which preserves deterministic election ordering.",
|
||||
"fix": "LIKELY FIXED but not code-confirmed (checked 2026-07-02): no direct 'bug-1307' code citation found. Circumstantial evidence it's resolved: raft-ha-core-safety Phase 5 ('Election liveness and safety under churn') is fully complete (all leaf tasks checked) and soak-validated at 0 committed-prefix divergences / 0 job losses across 6-round chaotic soaks per session memory \u2014 the exact scenario this bug's regression was found in. Re-run the 6-round chaotic soak once more before fully closing this entry; downgrading from 'not yet fixed' to 'probably fixed, wants one more soak confirmation' rather than asserting closed outright.",
|
||||
"tags": [
|
||||
"data-safety-regression",
|
||||
"committed-divergence",
|
||||
"committed-job-loss",
|
||||
"leader-completeness-violation",
|
||||
"election-liveness",
|
||||
"bug-1296-followup",
|
||||
"task-5.7",
|
||||
"needs-confirmation-soak",
|
||||
"election-liveness",
|
||||
"leader-completeness-violation",
|
||||
"chaotic-soak",
|
||||
"bug-1296-followup"
|
||||
"data-safety-regression",
|
||||
"committed-job-loss"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1296",
|
||||
|
|
@ -21712,7 +21724,7 @@
|
|||
"bug-851"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-06-30T15:05:00.000Z"
|
||||
"last_seen": "2026-07-02T17:30:00.000Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1308",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue