chore(wolf): restore buglog.json, log bug-1421/bug-1422, memory trail

buglog.json had been reset to a small auto-detected-noise-only file at some
point this session, losing the real 738-entry history (still present in git
HEAD). Restored the original list verbatim and appended bug-1421 (reusing
the ID a prior session had already reserved for this exact election
regression before the file got wiped) and bug-1422, both logged with full
root-cause/fix detail in this branch's other commits.

memory.md carries the OpenWolf session action log (auto-appended by hooks
plus this session's manual entries) for both fixes.
This commit is contained in:
Rongsong Shen 2026-07-09 11:57:19 +00:00
parent 5e0579dede
commit c68935127c
2 changed files with 332 additions and 0 deletions

View file

@ -12706,5 +12706,46 @@
"related_bugs": ["bug-1418", "bug-1419", "bug-1417"],
"occurrences": 4,
"last_seen": "2026-07-08T03:20:00.000Z"
},
{
"id": "bug-1421",
"timestamp": "2026-07-08T06:55:15.000Z",
"error_message": "Sequential HA soak regression: no primary elected at term>8 by round 6 (initially suspected to stem from the bug-1417 fix); root-caused 2026-07-09 as the primary never discovering a follower's higher term via AppendEntries-reject responses",
"file": "src/controller/ha.c",
"root_cause": "ha_raft_on_repl_ack only routes AE-reply term-adoption for success=1 ACKs; AE rejects (success=0) never reach raft_on_append_entries_reply. That function additionally drops any reply with reply_term != s->term (raft.c:832-833, stale-term drop) before it can adopt a higher term. Net effect: once a follower is ahead in term (e.g. it steps down from PreVote to CTRL_STANDBY at a higher term while an existing primary keeps campaigning/replicating at its old term), every AppendEntries the stale primary sends is rejected with ae_reject_stale, the primary never learns the higher term, and it can never regain write-quorum (needs 2/3, only self-vote counts) — a permanent livelock. First surfaced as an election regression (sequential soak: no primary elected at term>8 by round 6, chaotic stalled at round 5) suspected to be caused by the bug-1417 vote-index-mapping fix; re-reproduced 2026-07-09 as a round-4 sequential stall (primary ctrl-2 term=6, follower ctrl-3 term=7, 490 ae_reject_stale events, primary writable_lsn stuck at 17), which pinned the exact mechanism above.",
"fix": "Added two fallback term-discovery vectors that fire regardless of AE-reject outcome. (1) StateReport heartbeat-reply now carries the standby's current term (controller.capnp +term field @2, typemap.h state_report_t +term, ha_nop_reply_state_report populates it from ha->term) so the primary learns a higher term on every convergence cycle. (2) New library entry point raft_on_higher_term_observed(s, observed_term) mirrors UpdateTerm (strictly-greater-than only; touches term/voted_for/role; on_persist before on_become_follower for durability ordering) and is called from repl_ack_decode_and_dispatch on both the heartbeat state-report branch and the WAL-NACK branch (ha.c). Primary now steps down instead of retrying AppendEntries against a follower that rejects forever. Validated: 265 controller + 287 lib unit tests pass (zero regressions); 10-round sequential soak PASS (33 jobs, zero loss, all 3 nodes led ctrl-1/ctrl-2/ctrl-3); 10-round chaotic soak PASS (33 jobs, zero committed divergence, zero committed job loss).",
"tags": [
"ha",
"raft",
"term-desync",
"liveness",
"election-regression",
"ae-reject",
"replication",
"fixed"
],
"related_bugs": ["bug-1417", "bug-1418", "bug-851"],
"occurrences": 2,
"last_seen": "2026-07-09T08:50:00.000Z"
},
{
"id": "bug-1422",
"timestamp": "2026-07-09T10:14:14.000Z",
"error_message": "40-round sequential HA soak FAIL at round 36: verify_no_loss reported '0/111 jobs confirmed present' right after a fresh election (new primary ctrl-1, term=40) — xce_test's expect-job got E_XCE_STANDBY (10014) from a fresh client session whose own wait-primary had JUST confirmed ctrl-1 as serving 45ms earlier",
"file": ".claude/skills/xce-test-ha/run-ha-soak.sh",
"root_cause": "False-positive test-harness failure, NOT real committed-job loss: ctrl-1's own controller log confirms it stayed CTRL_PRIMARY throughout the window and successfully committed the 3 new post-failover jobs (log_append/ae_reply_success/commit_advanced at 10:14:22-23). ha_dispatch_gate's CTRL_PRIMARY branch (ha.c:3451-3531) unconditionally permits OPCode_get (joblist) even while fenced, so ctrl-1 itself could not have returned E_XCE_STANDBY for this query — the reply must have come from a standby (ctrl-2, just killed, or ctrl-3) that the client's failover retry loop landed on. Root cause traced to client_failover.c's xcecli_retry_next/xcecli_retry_decide (client_failover.c:305-319): E_XCE_STANDBY classifies as XCE_FO_RETRY_NOW (client_failover.c:59-60, prompt rotate, no backoff), and the 20-attempt retry budget (3 masters) should reliably reach the real primary — but when the WHOLE budget is exhausted while every recent classification was RETRY_NOW (as opposed to RETRY_BACKOFF), xcecli_retry_decide's budget-exhaustion branch (line 312-318) does NOT override *rc the way it does for RETRY_BACKOFF exhaustion (which maps to the distinct -E_XCE_NOT_READY, line 316-317) — it just returns false with *rc left at whatever the last attempt's raw status was (here, a stray E_XCE_STANDBY from a standby hit during the rapid post-election retry storm). verify_no_loss() (run-ha-soak.sh) then did a single one-shot check with zero retry at the driver level, unlike the job-submit loop just above it in the same file which already retries on this exact 'primary likely just promoted, regaining write-quorum' condition (run-ha-soak.sh:483-486). Only surfaced at round 36 of a 40-round run — narrow enough timing that shorter 6-10 round soaks across many prior sessions never hit it.",
"fix": "Added a bounded retry (3 total attempts, 2s settle sleep between) around the WHOLE verify_no_loss() check in run-ha-soak.sh, mirroring the existing submit-retry loop's rationale: a job still missing after a real settle window + 2 retries is a genuine loss, not this client-side retry-storm race. Did not touch client_failover.c (the battle-tested libxce-failover retry/rotation library with many other production consumers) — fixing at the test-driver level is safer and fully local. Re-ran both 40-round soaks after the fix: sequential PASS (123 jobs, zero loss, all 3 nodes led) and chaotic PASS (123 jobs, zero committed divergence, zero committed job loss).",
"tags": [
"xce-test-ha",
"false-positive",
"client-failover",
"retry-budget-exhaustion",
"E_XCE_STANDBY",
"test-harness",
"fixed"
],
"related_bugs": ["bug-854", "bug-1421"],
"occurrences": 1,
"last_seen": "2026-07-09T10:15:00.000Z"
}
]