From 2a769407f5ac819777901f242e47c438fde0c792 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Thu, 16 Jul 2026 10:41:13 +0000 Subject: [PATCH] docs(job-array-single-wal-record): backfill bug-1444 for task 6.0a's replay_wal fix A stale, long-orphaned background agent (dispatched early in this session for task 6.1, interrupted by an API error, and only resuming ~8 hours later after task 6.0a/6.1/6.2/6.3 and the validate-impl GO decision were already committed) independently re-discovered and filed the exact bug commit 16747aa already fixed, before noticing the fix was already on the branch. Its filed entry (bug-1444) accurately describes the error/root-cause but was left marked NOT FIXED. Corrected the fix field to reflect the actual resolution (commit 16747aa, task 6.0a) and cross-referenced it. Kept as its own entry rather than deleting it -- task 6.0a's fix was documented in tasks.md's Implementation Notes but had never been logged to buglog.json, so this backfills that gap. Distinct from bug-1443 (the still-open bulk-insert partial-visibility gap in the same feature). No code changes; the stale agent's independently-derived run-ha-soak.sh/soak-verdict.sh/waldump.c changes were confirmed byte-identical to HEAD and made no git writes. --- .wolf/buglog.json | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index e697b76..2caa69f 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -13273,5 +13273,33 @@ "related_bugs": [], "occurrences": 1, "last_seen": "2026-07-16T02:05:00Z" + }, + { + "id": "bug-1444", + "timestamp": "2026-07-16T02:48:18Z", + "error_message": "persist_recover: APPLY_DIVERGENT -- materialized state does not match the committed log tip (expected last_applied_lsn=5 new_job_count=3); a hidden apply defect is present (ha-commit-durability recover_verdict=RECOVER_CLEAN) / persist_recover: UNRECOVERABLE -- post-replay apply-correctness verification is DIVERGENT ... Staying ALIVE and PERMANENTLY non-promotable (refusing to serve divergent state) and RETRYING catch-up indefinitely -- reproduced live during job-array-single-wal-record task 6.1's HA soak scenario the FIRST time a node is killed and restarted after ANY WAL_JOB_ARRAY_SUBMIT record has ever been committed to its local WAL (e.g. after a 2-element array submission).", + "file": "src/controller/persist.c (replay_wal(), ~line 3902-3904; consumed by persist_recover()/persist_verify_apply_correctness(), ~line 3960-3991)", + "root_cause": "replay_wal()'s post-replay apply-correctness expectation counter only tallies WAL_JOB_SUBMIT entries toward new_job_submits_out: `if (hdr.type == WAL_JOB_SUBMIT && new_job_submits_out && apply_rc == 1) (*new_job_submits_out)++;` (persist.c ~3902-3904). It has NO corresponding branch for WAL_JOB_ARRAY_SUBMIT (the new record type job-array-single-wal-record/task 1.1 introduced), even though apply_committed_entry()'s WAL_JOB_ARRAY_SUBMIT case (persist.c ~2809-2851, restore_job_array()) constructs N new job_t entries per record, not 1. So expected_new_job_count silently undercounts by N for every array record replayed during boot -- while persist_verify_apply_correctness() (persist.c ~3982-3988) measures actual_new_jobs via a REAL jobmgr entry-count delta (count_jobmgr_entries before/after), which correctly reflects the N elements. The mismatch (actual_new_jobs != expected_new_job_count) trips APPLY_DIVERGENT on every single restart of every node whose local WAL contains a WAL_JOB_ARRAY_SUBMIT entry above its checkpoint -- deterministic and 100% reproducible, not a race or resource-exhaustion condition. persist_recover_loud_fence_if_unrecoverable then classifies this as UNRECOVERABLE, permanently non-promotable-fencing that node for the rest of its process life (own boot-time verification re-runs identically on every subsequent restart too, since the miscounting bug reproduces every time). This is a NEW, separate gap from this spec's three self-flagged residual items (task 3.1 bulk-insert partial-visibility / bug-1443, task 4.2 boot-replay-doesn't-respect-apply-failures gap, task 5.3b's small per-commit leak) -- it lives in a DIFFERENT, pre-existing mechanism (raft-apply-decoupling task 5.1's post-replay apply-correctness verification / ha-commit-durability's Req 8.1-8.4), which this spec's own task list never touched to account for the new WAL_JOB_ARRAY_SUBMIT type. Practical severity: in any cluster where a job array has EVER been submitted, every ordinary node restart (routine maintenance, crash-restart, rolling upgrade) permanently non-promotable-fences that node; over enough restarts across a 3-node cluster this can leave ZERO nodes eligible for promotion -- a serious HA-availability regression, not a cosmetic log line. Discovered live via job-array-single-wal-record task 6.1's HA soak harness (.claude/skills/xce-test-ha/run-ha-soak.sh, new --array-soak mode) -- specifically NOT reproducible by this spec's existing white-box unit tests (task 2.3/3.4), which never drive a real persist_recover()/replay_wal() boot cycle after committing an array record.", + "fix": "FIXED and VERIFIED (2026-07-15/16, commit 16747aa, task 6.0a of job-array-single-wal-record) -- independently root-caused by a separate implementation pass on the same branch, inserted as a blocking task ahead of 6.1/6.2/6.3 per explicit user sign-off after the exact same bug was live-reproduced during this task's own --array-soak run. Fix matches the shape predicted below almost exactly: replay_wal()'s tally gained a `WAL_JOB_ARRAY_SUBMIT` branch that adds `apply_rc` (the genuine per-record new-element count apply_committed_entry()'s array case already returns, not the flat/undercounted +1 the WAL_JOB_SUBMIT branch uses) to *new_job_submits_out when apply_rc > 0, mirroring the existing WAL_JOB_SUBMIT pattern. Verified via: (a) new unit tests in test_apply_correctness_verification.c with an independently-reproduced RED-before-fix/GREEN-after-fix cycle; (b) two from-scratch live 3-node HA soak re-runs (--array-soak), one by the fixing pass's implementer and one by its independent reviewer, both showing zero APPLY_DIVERGENT across all three of this bug's trigger paths (restart-mid-small-array, standby election-path recovery fallback, restart-mid-large-array) with one-WAL-record/one-commit-barrier and per-element status/cancel/completion all confirmed post-failover. Full detail in .kiro/specs/job-array-single-wal-record/tasks.md's Implementation Notes, task 6.0a entry. This buglog entry was originally filed (independently, before its author discovered the fix already existed on the branch) with fix=NOT FIXED; corrected here once the duplication was noticed, since the underlying finding is accurate and well-characterized and deserves its own tracked entry (task 6.0a's fix was documented in tasks.md but never separately logged here) -- kept as a distinct entry from bug-1443 (a different, still-open gap in the same feature) rather than merged or deleted.", + "tags": [ + "job-array-single-wal-record", + "persist_recover", + "replay_wal", + "apply_verify_correctness", + "APPLY_DIVERGENT", + "boot-replay", + "ha-availability", + "non-promotable-fence", + "xce-test-ha", + "array-soak", + "task-6.1", + "raft-apply-decoupling", + "ha-commit-durability" + ], + "related_bugs": [ + "bug-1443" + ], + "occurrences": 1, + "last_seen": "2026-07-16T02:48:18Z" } ]