--- id: "bug-1396" status: "cannot-reproduce" status_updated: "2026-09-24" status_note: "extra committed job (20 vs 19) never reproduced after 9318f997 harness hardening" component: "build" severity: "unknown" timestamp: "2026-07-02T14:00:00.000Z" last_seen: "2026-09-19T09:00:00Z" occurrences: 3 files: ["src/tools/test-commit-durability.sh (harness) or controller job-recovery path (unconfirmed which)"] tags: ["ha-commit-durability", "low-priority", "cold-restart-all", "job-count-mismatch", "harness-instrumented", "unconfirmed", "related-fix-applied"] related: ["bug-661", "bug-1422"] repro_attempts: "2 fresh repro attempts x3 rounds (2026-07-09), 0 reproduced" legacy: true --- # bug-1396: ha-commit-durability task 6.1 (cold-restart-all real-fs reproduction… ## Description ha-commit-durability task 6.1 (cold-restart-all real-fs reproduction, src/tools/test-commit-durability.sh on a genuine ext4 deploy dir, ~/tmp not overlay): run 5 of 5 reported `expect-jobs: expected 19, actual 20` after a full cold-stop-all + cold-start-all cycle — the cluster had ONE MORE committed job than the harness's own ACKED_FILE (deduplicated via `sort -u`) tracked. All 19 tracked job IDs individually verified present (`expect-job` loop all `ok`); the 20th job's identity was not captured (cluster had already been torn down by the script's EXIT trap by the time this was investigated). This is the OPPOSITE of bug-661/795's symptom (loss) — an unexplained EXTRA job, not a missing one. 4 of the other 5 runs (and every prior round within run 5 itself, rounds 1-8) showed correct exact-count durability with zero anomalies. ## Environment _Not recorded (migrated from buglog.json)._ ## Reproduction - Attempts: 2 fresh repro attempts x3 rounds (2026-07-09), 0 reproduced ## Root cause NOT INVESTIGATED — deprioritized because it doesn't match the loss scenario task 6.1 was probing for, and reproducing it requires re-running with a 'joblist' capture at the exact expect-jobs failure point (the script doesn't dump the full list on a count mismatch, only the count). Two live hypotheses, unconfirmed: (a) harness-side race — ACKED_FILE's write (submit_n) and run_cold_restart_recovery's read (`committed=$(sort -u ACKED_FILE | grep -c .)`) are not synchronized against a submission whose ack arrives asynchronously after the harness moves on, so a genuinely-committed job simply never got recorded in ACKED_FILE — a test-harness bookkeeping gap, not a controller bug; (b) a genuine job-duplication bug on the controller's recovery/replay path creating an extra job entry with a fresh ID. Also observed in this run's ctrl-1.log: repeated unrelated `handle_jobsubmit fail to lock job 63-15--1-1015808[32751]` errors and one `job not durable (rc=-110) — rejecting submission` for a DIFFERENTLY-FORMATTED job ID (not this harness's `0-0--0` convention) — almost certainly a stray unrelated process hitting the same cluster ports in the shared sandbox, not this test's own traffic, but flagged here in case it's connected. ## Fix STILL NOT ROOT-CAUSED for the original exact symptom (successful query, count=20 vs expected=19) — 2 fresh repro attempts (2026-07-09, 3 rounds each, real ext4 fs under ~/tmp) did not reproduce an extra job; both runs showed the tracked count exactly matched after the fix below. HOWEVER, both attempts DID hit a DIFFERENT, related bug in the exact same assert_job_count() check: right after the cold-restart-all election, the aggregate expect-jobs/joblist query itself FAILED outright with 'xce_joblist: Unknown error 10014' / 'could not determine job count' (a query FAILURE, not a wrong-but-successful count) — this is the identical client-side race as bug-1422 (libxce's failover retry loop, client_failover.c, can exhaust its 20-attempt budget in a rapid post-election retry storm and surface a stale E_XCE_STANDBY). Confirmed as a false alarm both times: the very next per-id assert_no_loss check (individual expect-job calls) found every job present. Fixed assert_job_count() with the same bounded-retry pattern used for bug-1422 (3 attempts, 2s settle) — re-run after the fix: 7 passed, 0 failed, clean. This closes the QUERY-FAILURE false-negative class in this check, and makes the harness reliable enough to actually catch the ORIGINAL extra-job symptom if it recurs (previously a query failure and a genuine extra-job count were indistinguishable at a glance; now the retry-exhausted case self-heals and no longer masks a real mismatch). The original 'extra job' phenomenon itself remains open/unconfirmed — needs a longer run (original was 1-of-5 full 8-round soaks) to have a realistic chance of firing again. ## Verification _Not recorded (migrated from buglog.json)._ ## References - Related: bug-661, bug-1422