forked from lei.chen/myxce
docs(job-array-single-wal-record): backfill bug-1444 for task 6.0a's replay_wal fix
A stale, long-orphaned background agent (dispatched early in this session for task 6.1, interrupted by an API error, and only resuming ~8 hours later after task 6.0a/6.1/6.2/6.3 and the validate-impl GO decision were already committed) independently re-discovered and filed the exact bug commit16747aaalready fixed, before noticing the fix was already on the branch. Its filed entry (bug-1444) accurately describes the error/root-cause but was left marked NOT FIXED. Corrected the fix field to reflect the actual resolution (commit16747aa, task 6.0a) and cross-referenced it. Kept as its own entry rather than deleting it -- task 6.0a's fix was documented in tasks.md's Implementation Notes but had never been logged to buglog.json, so this backfills that gap. Distinct from bug-1443 (the still-open bulk-insert partial-visibility gap in the same feature). No code changes; the stale agent's independently-derived run-ha-soak.sh/soak-verdict.sh/waldump.c changes were confirmed byte-identical to HEAD and made no git writes.
This commit is contained in:
parent
2b0a37acc7
commit
2a769407f5
1 changed files with 28 additions and 0 deletions
|
|
@ -13273,5 +13273,33 @@
|
|||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-16T02:05:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1444",
|
||||
"timestamp": "2026-07-16T02:48:18Z",
|
||||
"error_message": "persist_recover: APPLY_DIVERGENT -- materialized state does not match the committed log tip (expected last_applied_lsn=5 new_job_count=3); a hidden apply defect is present (ha-commit-durability recover_verdict=RECOVER_CLEAN) / persist_recover: UNRECOVERABLE -- post-replay apply-correctness verification is DIVERGENT ... Staying ALIVE and PERMANENTLY non-promotable (refusing to serve divergent state) and RETRYING catch-up indefinitely -- reproduced live during job-array-single-wal-record task 6.1's HA soak scenario the FIRST time a node is killed and restarted after ANY WAL_JOB_ARRAY_SUBMIT record has ever been committed to its local WAL (e.g. after a 2-element array submission).",
|
||||
"file": "src/controller/persist.c (replay_wal(), ~line 3902-3904; consumed by persist_recover()/persist_verify_apply_correctness(), ~line 3960-3991)",
|
||||
"root_cause": "replay_wal()'s post-replay apply-correctness expectation counter only tallies WAL_JOB_SUBMIT entries toward new_job_submits_out: `if (hdr.type == WAL_JOB_SUBMIT && new_job_submits_out && apply_rc == 1) (*new_job_submits_out)++;` (persist.c ~3902-3904). It has NO corresponding branch for WAL_JOB_ARRAY_SUBMIT (the new record type job-array-single-wal-record/task 1.1 introduced), even though apply_committed_entry()'s WAL_JOB_ARRAY_SUBMIT case (persist.c ~2809-2851, restore_job_array()) constructs N new job_t entries per record, not 1. So expected_new_job_count silently undercounts by N for every array record replayed during boot -- while persist_verify_apply_correctness() (persist.c ~3982-3988) measures actual_new_jobs via a REAL jobmgr entry-count delta (count_jobmgr_entries before/after), which correctly reflects the N elements. The mismatch (actual_new_jobs != expected_new_job_count) trips APPLY_DIVERGENT on every single restart of every node whose local WAL contains a WAL_JOB_ARRAY_SUBMIT entry above its checkpoint -- deterministic and 100% reproducible, not a race or resource-exhaustion condition. persist_recover_loud_fence_if_unrecoverable then classifies this as UNRECOVERABLE, permanently non-promotable-fencing that node for the rest of its process life (own boot-time verification re-runs identically on every subsequent restart too, since the miscounting bug reproduces every time). This is a NEW, separate gap from this spec's three self-flagged residual items (task 3.1 bulk-insert partial-visibility / bug-1443, task 4.2 boot-replay-doesn't-respect-apply-failures gap, task 5.3b's small per-commit leak) -- it lives in a DIFFERENT, pre-existing mechanism (raft-apply-decoupling task 5.1's post-replay apply-correctness verification / ha-commit-durability's Req 8.1-8.4), which this spec's own task list never touched to account for the new WAL_JOB_ARRAY_SUBMIT type. Practical severity: in any cluster where a job array has EVER been submitted, every ordinary node restart (routine maintenance, crash-restart, rolling upgrade) permanently non-promotable-fences that node; over enough restarts across a 3-node cluster this can leave ZERO nodes eligible for promotion -- a serious HA-availability regression, not a cosmetic log line. Discovered live via job-array-single-wal-record task 6.1's HA soak harness (.claude/skills/xce-test-ha/run-ha-soak.sh, new --array-soak mode) -- specifically NOT reproducible by this spec's existing white-box unit tests (task 2.3/3.4), which never drive a real persist_recover()/replay_wal() boot cycle after committing an array record.",
|
||||
"fix": "FIXED and VERIFIED (2026-07-15/16, commit 16747aa, task 6.0a of job-array-single-wal-record) -- independently root-caused by a separate implementation pass on the same branch, inserted as a blocking task ahead of 6.1/6.2/6.3 per explicit user sign-off after the exact same bug was live-reproduced during this task's own --array-soak run. Fix matches the shape predicted below almost exactly: replay_wal()'s tally gained a `WAL_JOB_ARRAY_SUBMIT` branch that adds `apply_rc` (the genuine per-record new-element count apply_committed_entry()'s array case already returns, not the flat/undercounted +1 the WAL_JOB_SUBMIT branch uses) to *new_job_submits_out when apply_rc > 0, mirroring the existing WAL_JOB_SUBMIT pattern. Verified via: (a) new unit tests in test_apply_correctness_verification.c with an independently-reproduced RED-before-fix/GREEN-after-fix cycle; (b) two from-scratch live 3-node HA soak re-runs (--array-soak), one by the fixing pass's implementer and one by its independent reviewer, both showing zero APPLY_DIVERGENT across all three of this bug's trigger paths (restart-mid-small-array, standby election-path recovery fallback, restart-mid-large-array) with one-WAL-record/one-commit-barrier and per-element status/cancel/completion all confirmed post-failover. Full detail in .kiro/specs/job-array-single-wal-record/tasks.md's Implementation Notes, task 6.0a entry. This buglog entry was originally filed (independently, before its author discovered the fix already existed on the branch) with fix=NOT FIXED; corrected here once the duplication was noticed, since the underlying finding is accurate and well-characterized and deserves its own tracked entry (task 6.0a's fix was documented in tasks.md but never separately logged here) -- kept as a distinct entry from bug-1443 (a different, still-open gap in the same feature) rather than merged or deleted.",
|
||||
"tags": [
|
||||
"job-array-single-wal-record",
|
||||
"persist_recover",
|
||||
"replay_wal",
|
||||
"apply_verify_correctness",
|
||||
"APPLY_DIVERGENT",
|
||||
"boot-replay",
|
||||
"ha-availability",
|
||||
"non-promotable-fence",
|
||||
"xce-test-ha",
|
||||
"array-soak",
|
||||
"task-6.1",
|
||||
"raft-apply-decoupling",
|
||||
"ha-commit-durability"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1443"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-07-16T02:48:18Z"
|
||||
}
|
||||
]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue