fix(raft-apply-decoupling): bug-1419 close resubmit-dedup gap left by apply-after-commit deferral

jobid_range_exists() only checks c->jobmgr, but task 3.1 deliberately
defers job construction until after commit -- so a submission that is
durably WAL-appended and awaiting its commit outcome is invisible to
that check. A client-side retry (XCECLI_FAILOVER_CALL resending the
same jobid after a transport error or failover mid-flight) sails past
the guard and appends a second, non-identical WAL_JOB_SUBMIT entry for
the same jobid -- confirmed via waldump on a live chaotic-soak node
(duplicate submits, same jobid, differing subtime/CRC) -- which can
trip persist_recover()'s post-replay apply-correctness check into a
genuine APPLY_DIVERGENT verdict.

Adds a process-lifetime in-flight-jobid-range registry in job.c that
handle_jobsubmit atomically check-and-claims immediately after the
existing jobid_range_exists check, releasing from
jobsubmit_async_ctx_release (the single choke point already shared by
the commit-continuation, apply-complete, and OOM early-out paths). A
resubmit for a still-in-flight jobid now gets -EEXIST with no second
WAL append.

Adds a matching XCE_TESTING reset seam (declared in cluster.h) so a
test that deliberately leaves a submission's commit continuation
unresolved through teardown doesn't leak its jobid range into a later
test reusing the same range -- commit_engine_fini frees pending
continuations without invoking their callback.

Regression test added to test_commit_barrier_reply.c (RED->GREEN
confirmed). Full 78-group controller_unit_test sweep: 782 tests, 0
failures.

This fixes only the WAL duplicate-submit mechanism (layer 1 of
bug-1419). The deeper PreVote-freshness/apply-divergence liveness
deadlock (layer 2 -- no recovery path when the only fresh-enough node
is apply-divergent-fenced) remains open and needs its own design
discussion; buglog.json updated accordingly.

.wolf/buglog.json

@@ -12640,7 +12640,7 @@
     "error_message": "[xce-test-ha --chaotic] genuine, reproducible election-liveness stall (not the bug-1418 harness process-stacking artifact, which was a separate, now-fixed issue): after 2+ overlapping kills, the surviving 2-node set can permanently deadlock -- neither node can ever become primary again for the rest of the run.",
     "file": "src/controller/persist.c (apply-correctness verification) + src/controller/ha.c (PreVote freshness gate) + WAL duplicate-submit mechanism (unlocated)",
     "root_cause": "Direct log/WAL analysis (waldump on the affected node's data dir) during a live repro superseded two prior, less-precise theories filed as bug-1418 (a shell-script process-stacking artifact, unrelated and separately fixed; and an unverified ha->term/ha->raft.term \"term desync\" claim that does not match the code -- raft_begin_pre_vote/ha_start_prevote_async both read s->term == ha->raft.term directly, there is no separate stale-field read on the PreVote path).\n\nACTUAL mechanism observed: waldump on ctrl-3's WAL (the node with the longest/freshest log after the chaotic kill sequence) showed \"duplicate submits: 3 (byte-identical: 0)\" -- the SAME jobid appended to the WAL twice with DIFFERING payloads (different subtime, different CRC), e.g. lsn=13 and lsn=14 both jobid=0-0-1783418222-0 but distinct subtime/PCRC. This is NOT bug-845's mechanism (that fix, landed 2026-06-23, was for byte-IDENTICAL double-hook-registration dupes and is confirmed still working -- these are non-identical). count_jobmgr_entries() legitimately sees these as 2 distinct job-table mutations (or the entity count otherwise fails to match the WAL-replay-derived expectation), so persist_recover()'s post-replay apply-correctness check (task 5.1, Req 8.1/8.2) correctly fires APPLY_DIVERGENT -> the loud, sticky, non-promotable fence (raft-apply-decoupling Req 8.3/8.4) -- this fencing behavior is WORKING AS DESIGNED, not itself a bug.\n\nThe resulting LIVENESS bug: with the fenced node holding the only sufficiently-fresh (longest) log, the other surviving node can never win the PreVote freshness gate against it (correctly, per Raft/PreVote semantics -- it really is behind), while the fenced node itself can win Raft elections (quorum_reached/became_leader observed at terms 27/30/32 in logs) but its own raft_cb_become_leader promotion gate refuses CTRL_PRIMARY every time (apply_divergent=1). With the third node dead (killed by the chaotic driver) and never restarted long enough to catch up, NO node can ever become primary again -- a genuine, reproducible liveness deadlock, distinct from (and downstream of) the WAL duplicate-submit root cause. Confirmed via direct log grep across ctrl-1.log/ctrl-3.log at the exact stall timestamps (09:57:16 divergence detected, repeating \"PreVote DENIED ... reason=freshness\" and \"promotion REFUSED ... apply_divergent=1\" through 10:05+); NOT root-caused via speculation.",
-    "fix": "NOT FIXED. Two separate, layered problems, neither resolved here: (1) the WAL duplicate-non-identical-submit mechanism itself is unlocated -- needs its own investigation (likely a distinct submit-path race from bug-845's already-fixed double-hook-registration, since these pairs are NOT byte-identical; check for a retry/resubmit path that re-stamps subtime on the same client-generated jobid, or a race in the commit-continuation/apply-pump interaction from raft-apply-decoupling that could double-materialize a submit under a specific kill-timing window). (2) Even with (1) fixed, the raft-apply-decoupling promotion-refusal design has a known, currently-unaddressed liveness gap when the ONLY sufficiently-fresh candidate is apply-divergent and the other members cannot catch up (no reconciliation/truncation-then-promote recovery path exists today) -- this needs its own design discussion, not a quick patch, since forcing promotion of a divergent node or truncating its uncommitted-but-locally-applied entries both have real correctness implications.",
+    "fix": "PARTIALLY FIXED (2026-07-07). Root cause of layer (1), the WAL duplicate-non-identical-submit mechanism, is now located and fixed: raft-apply-decoupling task 3.1 deferred job/entity construction in c->jobmgr until AFTER commit (apply_committed_entry, post-commit apply pump), but jobid_range_exists() -- the sole dup guard in handle_jobsubmit -- only checks c->jobmgr. A submission that is durably WAL-appended and awaiting its commit outcome does not yet exist in c->jobmgr, so a client-side retry for the SAME jobid (XCECLI_FAILOVER_CALL/xce_failover_classify's idempotent-replay contract resends on a transport error or failover-mid-flight) sailed straight past the guard and appended a second, non-identical WAL_JOB_SUBMIT entry (different subtime/CRC) for the same jobid -- exactly the 'duplicate submits: N (byte-identical: 0)' signature waldump showed. Fix: src/controller/job.c adds a process-lifetime in-flight-jobid-range registry (s_inflight_submits, guarded by s_inflight_submits_lock) that handle_jobsubmit atomically check-and-claims (inflight_range_try_add) immediately after the existing jobid_range_exists check, and releases (inflight_range_remove) from jobsubmit_async_ctx_release -- the same release point already shared by the commit-continuation and apply-complete paths, so the range stays claimed for exactly the window a duplicate could land in. A resubmit for a still-in-flight jobid now gets -EEXIST immediately, with no second WAL append. Regression test test_jobsubmit_resubmit_before_resolution_rejected_no_dup_wal_entry added to test_commit_barrier_reply.c (RED confirmed -EEXIST not returned pre-fix; GREEN after). Verified via the full 78-group controller_unit_test sweep (782 tests, 0 failures) -- zero regressions, including the 6 apply-pump/commit-barrier groups most adjacent to this change.\n\nLayer (2), the PreVote-freshness/apply-divergence liveness deadlock itself (when the ONLY sufficiently-fresh candidate is apply-divergent-fenced and no reconciliation path exists), remains UNFIXED and is now the sole open part of this bug -- it needs its own design discussion (forcing promotion of a divergent node, or truncating its uncommitted-but-locally-applied entries, both have real correctness implications), not a quick patch. With layer (1) fixed, this deadlock should no longer be reachable via the duplicate-submit path specifically, but the underlying design gap in raft-apply-decoupling's promotion-refusal logic is independent and could still be triggered by any other cause of a genuine, permanent APPLY_DIVERGENT fence on the only fresh-enough node.",
     "tags": [
       "ha",
       "election",
@@ -12650,7 +12650,8 @@
       "chaotic-failover",
       "liveness",
       "wal-duplicate-submit",
-      "unresolved"
+      "partially-fixed",
+      "liveness-gap-open"
     ],
     "related_bugs": [
       "bug-1417",
@@ -12658,7 +12659,7 @@
       "bug-845",
       "bug-1418"
     ],
-    "occurrences": 1,
-    "last_seen": "2026-07-07T14:00:00.000Z"
+    "occurrences": 2,
+    "last_seen": "2026-07-07T14:30:00.000Z"
   }
-]
+]