feat(scheduler-dispatch-commit): controller resolves retryable dispatch failures with bounded escalation (task 3.3)

The four retryable-classified failure sites in dispatch_a_job_() (worker
lookup, worker null, worker not ready, RPC failure) now resolve instead
of just returning an error code: lock(j, RTTI_LOCK_WRITE) -> genuinely
classify via xce_classify_dispatch_fail() -> increment dispatch_fail_count
-> within XCE_DISPATCH_FAIL_RETRY_BOUND (3) resolves retryable (nothing
further under the lock), past it resolves terminal via mark_job_fail()
exactly like task 3.2's sites -> unlock(j) -> dereference(j) (closing
bug-1627's 4-site reference leak) -> outside the lock, notify_dispatch()
fires only on the retryable outcome. Each site's natural error code is
captured into a stable local before any intervening call, since errno
isn't guaranteed stable once lock()/ERROR()/free_launchinfo() sit between
the original failure and the eventual return.

handle_jobdispatch() is intentionally untouched -- it still unconditionally
calls notify_dispatch() itself for every outcome, so the retryable path
now temporarily double-notifies until task 4.1 removes that caller-side
call. Expected, not a defect.

New test_dispatch_fail_retry.c (4 tests, including a bounded-escalation
test driving 4 consecutive RPC failures on the same job: 3rd retries,
4th resolves EXIT) plus a fixed stale assertion in test_dispatch_commit.c
and additive, read-only notify-capture accessors in
test_commit_barrier_reply.c (the binary's sole notify() stub host).

Independently reviewed and approved, including explicit scrutiny of the
cross-file test accessor addition and the stale-assertion fix -- both
confirmed legitimate, not scope creep or regression-hiding.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN

src/controller/testsuite/test_dispatch_commit.c

@@ -277,12 +277,24 @@ static void test_dc_success_commits_alloc_state_and_resets_counter(void)
 #define DC_JOBID_NOTFOUND 91002ULL
 
 /* Regression safety net (untouched failure branch): a dispatch whose
- * target worker cannot be found must leave the job exactly as it was
- * before the attempt -- still PEND, allocs still uncommitted, the
- * dispatch_fail_count untouched (that counter's own increment is task
- * 3.3's job, not 3.1's) -- and must append no WAL entry. Confirms this
- * task's restructuring did not disturb the five failure branches ahead of
- * xce_startjob_req(). */
+ * target worker cannot be found must leave the job's ALLOCATION exactly
+ * as it was before the attempt -- still PEND, allocs still uncommitted --
+ * and must append no WAL entry. Confirms this task's restructuring did
+ * not disturb the five failure branches ahead of xce_startjob_req().
+ *
+ * scheduler-dispatch-commit task 3.3 update: dispatch_fail_count is no
+ * longer asserted unchanged here -- task 3.3 wired this exact failure
+ * branch (whichever of XCE_DISPATCH_FAIL_SITE_WORKER_LOOKUP/WORKER_NULL
+ * the runtime takes for an unregistered worker key) to increment the
+ * counter by exactly 1 as part of its retryable resolution (Req 4.3),
+ * which this test's own original comment explicitly anticipated
+ * ("incrementing it is task 3.3's job, not 3.1's"). The counter's own
+ * increment-by-1/bound behavior is covered in depth by
+ * test_dispatch_fail_retry.c; this test only re-confirms the seeded
+ * starting value (2) plus this task's own increment (+1) lands at 3,
+ * still within the bound (3), so the job stays PEND -- not a regression
+ * in this test's own subject (allocation-gating), just an update to an
+ * assertion that depended on timing task 3.3 had not landed yet. */
 static void test_dc_worker_not_found_leaves_job_pend_and_alloc_uncommitted(void)
 {
     job_t*           j;
@@ -346,9 +358,10 @@ static void test_dc_worker_not_found_leaves_job_pend_and_alloc_uncommitted(void)
         "commit, it does not change any failure branch's behavior");
     TEST_ASSERT_NULL_MESSAGE(j->job.allocs,
         "job.allocs must remain uncommitted on this failure path");
-    TEST_ASSERT_EQUAL_UINT32_MESSAGE(2, j->dispatch_fail_count,
-        "this task does not touch dispatch_fail_count on any failure path "
-        "(incrementing it is task 3.3's job) -- it must be unchanged");
+    TEST_ASSERT_EQUAL_UINT32_MESSAGE(3, j->dispatch_fail_count,
+        "task 3.3: this retryable failure path increments the counter by "
+        "exactly 1 from its seeded starting value (2 -> 3), still within "
+        "the bound (3)");
     TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn_before, lsn_after,
         "a failed dispatch must append no WAL entry");