feat(worker-rejoin-resync): apply reported outcomes idempotently (task 3.2)

Add resync_apply_report()/resync_apply_running_entry()/
resync_apply_finished_entry()/resync_apply_unrecognized_entry() to
resync.c: routes each entry in a worker's job-state report to the
correct existing idempotent path (still-running -> clear from outstanding
set, no fact; finished-success/failure -> record_job_finish_fact_at/
on_job_fail; unrecognized -> record_job_reconcile_pending_fact;
unqueried extra -> record_job_reconcile_assign_fact or no-op), discards
reports for superseded rounds (Req 6.1/6.3), and resolves the round once
its outstanding set empties.

Also fixes bug-1747 (Req 4.2): adds record_job_finish_fact_at(),
additive alongside the unmodified record_job_finish_fact, so a
finished-success reconciliation entry records the WORKER-reported
finish time instead of reconciliation-processing time -- proven via a
real WAL-write + persist_recover() restart-replay test. The FAIL-side
half of the same gap (WAL_JOB_FAIL carries no fail-time wire slot at
all, for any caller) is confirmed pre-existing and systemic, correctly
left out of this task's scope.

Three review rounds surfaced and fixed two real, distinct bugs in the
unqueried-extra-job routing guard (see tasks.md's Implementation Notes
for the full account and the general lesson about re-deriving a full
truth table after a partial boolean-guard fix): round 1 caught the
wrong axis (terminal-vs-non-terminal instead of assignment-status,
Req 4.8); round 2 caught that the first fix's short-circuit clobbered
an already-terminal job's DONE/EXIT back to RUN (Req 4.4), including
confirming the implementer's own test had certified the bug as correct;
round 3 independently re-derived the final three-way guard from scratch
and found no further defect. bug-1748 logged for the guard fix.

Full controller suite: 1239 tests, 0 failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku

src/controller/include/cluster.h

@@ -241,6 +241,28 @@ extern int record_job_reconcile_pending_fact(cluster_t*, job_t*);
 extern int record_job_reconcile_assign_fact(cluster_t*, job_t*, worker_t*,
                                             xce_job_state_entry_t*);
 
+/* worker-rejoin-resync task 3.2 (bug-1747 fix, Req 4.2): record_job_finish_
+ * fact()'s worker-reported-finish-time sibling, for the reconciliation
+ * finished-success path only. Same body/guard/hook-fire/deferred-append
+ * pattern as record_job_finish_fact() (job.c, unmodified) -- differs only
+ * in passing `finish_time_ms` (converted ms -> s) to persist_submit_job_
+ * finish_fact() instead of time(NULL). See job.c's own doc comment on this
+ * function for the full rationale, including the persist_submit_job_
+ * finish_fact() unit confirmation.
+ *
+ * Precondition: caller holds `job` under RTTI_LOCK_WRITE on entry and keeps
+ * holding it until this call returns -- this function does not lock/unlock
+ * `job` itself, matching record_job_finish_fact's own convention.
+ *
+ * DEFERRED pattern (unlike record_job_reconcile_pending_fact/assign_fact
+ * above): this function only durably appends the fact; it does NOT mutate
+ * job's in-memory state and the caller must NOT call persist_notify_job_
+ * updated_already_persisted() afterward -- exactly like record_job_finish_
+ * fact(), the mutation and notify happen later, via apply_committed_entry's
+ * replay of the committed WAL_JOB_FINISH entry. */
+extern int record_job_finish_fact_at(cluster_t*, job_t*, xce_jobfinish_t*,
+                                     uint64_t);
+
 extern int       on_job_start(cluster_t*, job_pidinfo_t*);
 extern int       on_job_finish(cluster_t*, xce_jobfinish_t*);
 extern int       on_job_fail(cluster_t*, job_failinfo_t*);

src/controller/include/resync.h

@@ -5,6 +5,8 @@
 
 #include <stdint.h>
 
+#include "typemap.h" /* xce_job_state_entry_t/xce_job_state_report_t --
+                       * worker-rejoin-resync task 3.2 */
 #include "xce/cluster_def.h"
 #include "xce/types.h"
 
@@ -75,6 +77,103 @@ extern void resync_resolve_round(cluster_t* c, worker_t* w);
  * Precondition: caller holds at least `w`'s read lock. */
 extern int resync_round_is_current(worker_t* w, uint64_t token);
 
+/* worker-rejoin-resync task 3.2 (Req 4.1-4.5, 6.1-6.3): applies a worker's
+ * already-decoded job-state report to `c`'s job table -- the glue that maps
+ * each `xce_job_state_entry_t` outcome in `report` to the correct existing
+ * (record_job_fail_fact via on_job_fail, job.c) or new (record_job_finish_
+ * fact_at, task 3.2/bug-1747; record_job_reconcile_pending_fact/record_job_
+ * reconcile_assign_fact, task 2.1/2.2, job.c) idempotent fact-recording
+ * call, per design.md's "System Flows" sequence diagram (`loop each
+ * reported entry`).
+ *
+ * Routing per `xce_job_state_entry_t.status`:
+ *   - JobStateStatus_running, entry->queried true: cleared from the round's
+ *     outstanding set; no fact recorded.
+ *   - JobStateStatus_running, entry->queried false (Req 3.6 extra): a
+ *     genuine THREE-WAY split (review round 2 fix -- a two-way boolean
+ *     guard is WRONG and introduces a Req 4.4 violation, see resync.c's
+ *     own doc comment on resync_apply_running_entry for the full incident):
+ *     (a) TERMINAL -> ALWAYS a no-op, regardless of host (Req 4.4 -- a
+ *     terminal job is never assign-eligible; record_job_reconcile_assign_
+ *     fact() has no backward-transition guard of its own and would
+ *     unconditionally clobber DONE/EXIT back to RUN if called here).
+ *     (b) non-terminal AND already has SOME live non-terminal host
+ *     assignment (`job->job.allocs != NULL` -- to THIS reporting worker's
+ *     own host, or to a DIFFERENT one, it does not matter which) -> a
+ *     no-op (already validly assigned; calling assign here would still
+ *     wrongly reset start_time and re-append a WAL_JOB_DISPATCH fact for a
+ *     job that was never actually lost).
+ *     (c) non-terminal AND genuinely UNASSIGNED (`allocs == NULL`) -> Req
+ *     4.8's real trigger ("no existing non-terminal record... assigned
+ *     elsewhere" -- there IS a non-terminal record, but it carries no
+ *     assignment at all, e.g. still PEND because its WAL_JOB_DISPATCH fact
+ *     never reached durable majority commit before a controller failover,
+ *     even though the worker really is running it) -> reconciles via
+ *     record_job_reconcile_assign_fact() (Req 4.8), confirmed against that
+ *     function's own doc comment/assert (job.c), which documents itself as
+ *     the running-only case. A jobid with NO record at all in the job
+ *     table cannot be reconciled this way (that function mutates an
+ *     existing job_t, it does not construct one) and is logged and skipped
+ *     -- out of this feature's scope; every job a worker can legitimately
+ *     run was submitted through this controller at some point, so this is
+ *     expected to be unreachable in practice.
+ *   - JobStateStatus_finishedOk: routed through record_job_finish_fact_at()
+ *     (task 3.2/bug-1747 fix, job.c) -- record_job_finish_fact()'s own
+ *     worker-reported-finish-time sibling, so the recorded finish_time is
+ *     entry->finish_time_ms (Req 3.5), not reconciliation-processing time
+ *     (Req 4.2). record_job_finish_fact() itself is unmodified; its own
+ *     caller (on_job_finish, an ordinary worker report) still uses
+ *     receipt-time, correctly, for that near-instantaneous case.
+ *   - JobStateStatus_finishedFail: routed through on_job_fail() (job.c),
+ *     unchanged -- Requirement 4.2 remains unsatisfiable for this outcome
+ *     via any existing path in this codebase (WAL_JOB_FAIL carries no
+ *     finish-time slot at any layer, for any caller -- a pre-existing,
+ *     systemic gap that predates this feature; see buglog.json bug-1747).
+ *   - Both finished outcomes apply identically whether entry->queried is
+ *     true or false (Req 4.7: an unqueried finished extra reconciles
+ *     through the same path as a queried one; there is no "assign" case for
+ *     a finished outcome). Cleared from the outstanding set when
+ *     entry->queried is true.
+ *   - JobStateStatus_unrecognized: routed through record_job_reconcile_
+ *     pending_fact() (Req 4.6, task 2.1); cleared from the outstanding set.
+ *
+ * Req 6.1/6.3: if `round_token` is not `w`'s CURRENTLY outstanding round
+ * (resync_round_is_current), the entire report is discarded before
+ * anything else is touched -- no fact is applied, no outstanding-set
+ * mutation occurs. This also makes a duplicate re-application of an
+ * already-fully-resolved report (Req 6.2) a pure no-op by the same
+ * mechanism: once every queried job has a reconciled outcome,
+ * resync_resolve_round() clears w->resync, so `round_token` is no longer
+ * current on a later identical call -- no facts are re-applied, no
+ * additional WAL entries appended (a duplicate applied WHILE the round is
+ * still open instead re-applies each fact through job.c's own idempotent
+ * guards, which are themselves safe no-ops on a repeat).
+ *
+ * Req 5.1/5.2: once the round's outstanding set becomes empty (every
+ * queried jobid has been cleared), the round is resolved
+ * (resync_resolve_round()), clearing XCE_WORKER_RECONCILING so `w` becomes
+ * eligible for dispatch again.
+ *
+ * Locking: unlike resync_arm_round/resync_resolve_round/resync_round_is_
+ * current above, this function manages its OWN `w` write-locking
+ * internally and narrowly (never nested with a job's own write lock) --
+ * mirroring dispatch_a_job_'s established discipline (job.c: every branch
+ * releases `w`'s lock BEFORE acquiring a job's) rather than this file's
+ * other three functions' "caller holds w's lock for the whole call"
+ * convention, which fits their simple, single-object, single-call-site
+ * mutations but not this function's fan-out into per-job job.c calls, each
+ * requiring its own job-level lock. The caller must NOT pre-hold `w`'s
+ * lock when calling this function.
+ *
+ * `report` is treated as a single, already-fully-assembled list of
+ * `report->count` entries (`report->entries[0..count)`) -- multi-page wire
+ * reassembly is a later task's concern (6.2), out of this function's scope.
+ *
+ * Returns 0 on success (whether the report was applied or discarded as
+ * stale), or a negative errno for invalid arguments (NULL c/w/report). */
+extern int resync_apply_report(cluster_t* c, worker_t* w, uint64_t round_token,
+                               xce_job_state_report_t* report);
+
 #ifdef XCE_TESTING
 /* worker-rejoin-resync task 3.1 test seam: directly invokes the SAME
  * internal timeout-fire logic resync_arm_round's own delaymgr callback runs