fix: the controller is the ground truth for job state on scheduler rejoin (bug-1860)

A job the controller requeued to PEND while the scheduler was disconnected
stayed RUN (with its old allocation) in the scheduler forever: the
bug-1812 guard in core_events.c's JOBS_PUSH merge refused any non-RUN,
non-terminal resync claim over a job holding a real allocation, because
it could not tell a genuine requeue from a push built before the
scheduler's own fresh dispatch.

Design (per review): the scheduler persists nothing, so on rejoin the
controller's state wins, and the race the guard protected against is
closed where the truth lives:

- scheduler JOBS_PUSH: the bug-1812 refusal is removed; the controller's
  state replaces the scheduler's and its allocation is released. The
  terminal-is-permanent and phantom-RUN (RUN without allocation data)
  guards stay.
- controller dispatch_a_job_(): refuses a job that is not PEND (RUN or
  finished) before any worker is touched -- previously nothing checked,
  so a stale second dispatch would start a running job again (or, through
  a retryable failure, requeue it). The job is left as is, no WAL fact,
  and its true state is re-sent (HOOK_DETAIL_JOB_UPDATE). A PEND job with
  a leftover allocation stays dispatchable.
- scheduler JOB_UPDATE: a RUN report carrying the controller's committed
  allocation is adopted, so a corrected job is never RUN with nothing
  charged.

Tests (all RED before): test_jobs_push_duplicate_jobid_controller_pend_
wins_over_local_run (replaces the bug-1812 test that pinned the
refusal), test_job_update_run_adopts_controller_allocation,
test_dispatch_of_non_pending_job_is_refused. Two tests that seeded a RUN
job to reach retry branches (a precondition their own headers called
unreachable) now use the real paths: the counter-reset test requeues via
record_job_reconcile_pending_fact() first; the disconnected-worker RUN
test asserts the refusal contract.

Controller suite 1238/0, scheduler 87/87. HA soaks: chaotic 6 rounds PASS
(0 divergence/loss, 6 worker-resync jobs), sequential 6 rounds PASS
(SIMVERIFY 7 files, 0 mismatches). The race did not occur in either run
(0 refusals logged), so the new paths are proven by the unit tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

src/controller/job.c

@@ -1313,6 +1313,36 @@ static int dispatch_a_job_(cluster_t* c, endpoint_t* ep, xce_job_alloc_t* alloc)
         return -errno;
     }
 
+    /* bug-1860: only a PEND job may be dispatched. Nothing checked this, so
+     * a second dispatch for a job already RUN -- e.g. from a scheduler
+     * acting on a rejoin push older than the job's own dispatch -- would
+     * start it on a worker again (or, via a retryable failure, requeue a
+     * running job). Refuse it before any worker is touched, leave the job
+     * exactly as it is, and re-send its true state to the scheduler, which
+     * adopts it (the controller is the ground truth). */
+    {
+        int  state;
+        bool dispatchable;
+
+        lock(j, RTTI_LOCK_READ);
+        state        = j->job.state;
+        /* State only: a PEND job still carrying a leftover allocation
+         * stays dispatchable (dispatch replaces it), never refused
+         * forever. */
+        dispatchable = JOB_IS_PEND(state) && !JOB_IS_RUN(state)
+                       && !JOB_IS_FINISH(state);
+        unlock(j);
+
+        if (!dispatchable) {
+            WARN("dispatch_a_job_: job %s is not pending (state 0x%x) -- "
+                 "refusing a stale dispatch and re-sending its state",
+                 jobidStr, state);
+            persist_notify_job_updated_already_persisted(c, j);
+            dereference(j);
+            return -EALREADY;
+        }
+    }
+
     nc = alloc->nhosts;
     if (nc <= 0) {
         /* scheduler-dispatch-commit task 3.2 (Req 4.2, 4.5): resolves this