feat(worker-rejoin-resync): reconciliation round lifecycle (task 3.1)

New src/controller/resync.c/resync.h: resync_arm_round(cluster_t*,
worker_t*) enumerates every non-terminal job assigned to a host,
populates worker_t.resync (round_token, query job-ID set, deadline),
sets XCE_WORKER_RECONCILING, and schedules a non-blocking delaymgr
one-shot timeout. resync_resolve_round() clears the round; a fired
timeout resolves it too, via resync_round_is_current()'s by-value
round-token check -- delay.c/delay.h expose no true cancel/dequeue
primitive, so a stale timer from an already-superseded round becomes a
safe no-op instead of touching a newer round's live state (verified
safe against both misresolving the wrong round and use-after-free: the
timeout closure never holds a pointer into resync_state_t itself).

5 new tests: query-set correctness, fresh-token-per-arm, the
RECONCILING flag's set/clear lifecycle, and a direct (non-blocking)
timeout-fire proof via an XCE_TESTING-gated seam confirmed compiled out
of the production binary. Full suite: 1232 tests, 0 failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku

src/controller/include/resync.h

@@ -0,0 +1,97 @@
+#if !defined(_RESYNC_H_)
+#define _RESYNC_H_ 1
+
+#include "config.h"
+
+#include <stdint.h>
+
+#include "xce/cluster_def.h"
+#include "xce/types.h"
+
+/* worker-rejoin-resync task 3.1 (Req 1.1-1.3, 2.1-2.2, 5.3, 6.1, 6.3):
+ * per-worker job-state reconciliation round lifecycle -- arm on trigger,
+ * track queried job IDs + a round token, and resolve on either the
+ * Requirement 5.3 delaymgr timeout or (later, task 3.2) normal report
+ * completion. `resync_state_t` itself lives in xce/types.h (task 1.2),
+ * attached via `worker_t.resync`; this file owns building/populating it
+ * and its lifecycle, not its shape. See design.md "Controller: resync.c".
+ *
+ * Every function below follows the SAME locking convention job.c's
+ * record_job_reconcile_pending_fact/record_job_reconcile_assign_fact
+ * already established (design.md Components): the caller holds the
+ * relevant per-worker write lock (matching rejoin_worker's/apply_join_
+ * state's own convention, worker.c) BEFORE calling and keeps holding it
+ * until the call returns -- these functions never lock/unlock `w`
+ * themselves.
+ */
+
+/* Arms a fresh reconciliation round for `w`: enumerates every non-terminal
+ * job in `c`'s job table currently assigned to w's host (head/primary host,
+ * matching job.c's own existing "job's assigned host" representation --
+ * job->job.allocs->allocs[0]->host, e.g. job.c's exechost lookup and
+ * job-control routing), builds the round's query-job-ID set, assigns a
+ * fresh monotonic round token (Req 6.1/6.3), sets deadline = now +
+ * ClusterConfig.workerResyncTimeoutMs (Req 5.3), attaches the resulting
+ * resync_state_t to w->resync, sets XCE_WORKER_RECONCILING on
+ * w->host.status, and schedules a non-blocking delaymgr one-shot timeout
+ * that resolves the round if nothing else does first.
+ *
+ * A worker with zero matching non-terminal jobs is a documented no-op (Req
+ * 1.2): w->resync stays NULL/untouched, XCE_WORKER_RECONCILING is not set,
+ * and no timeout is scheduled.
+ *
+ * If `w` already carries an outstanding round on entry (w->resync != NULL
+ * -- e.g. this rejoin interrupts one that never resolved), that prior round
+ * is discarded first (Req 6.1, via resync_resolve_round) before the fresh
+ * one is armed, so the new round's token always supersedes the old one and
+ * a later report carrying the old token is provably stale (Req 6.3).
+ *
+ * Precondition: caller already holds `w`'s write lock.
+ *
+ * Returns the number of jobs armed into the round (>=0; 0 for the
+ * documented Req 1.2 no-op case) on success, or a negative errno (no round
+ * is armed/left partially armed on failure). */
+extern int resync_arm_round(cluster_t* c, worker_t* w);
+
+/* Resolves (clears) whatever reconciliation round is currently outstanding
+ * for `w`, if any: clears XCE_WORKER_RECONCILING on w->host.status, frees
+ * and detaches w->resync. Called both by the Req 5.3 timeout firing
+ * (internally, via resync_arm_round's own delaymgr callback) and, later
+ * (task 3.2), once every queried job in the round has a reconciled
+ * outcome. Idempotent: a no-op when w->resync is already NULL -- safe to
+ * call speculatively.
+ *
+ * Precondition: caller holds `w`'s write lock. */
+extern void resync_resolve_round(cluster_t* c, worker_t* w);
+
+/* Req 6.3: true (nonzero) iff `token` is still the round token CURRENTLY
+ * outstanding for `w` -- i.e. w->resync != NULL && w->resync->round_token
+ * == token. False for a stale/superseded round (a newer resync_arm_round
+ * call replaced it) or an already-resolved one (w->resync == NULL). This is
+ * the discard-on-stale-report mechanism Req 6.1/6.3 require: a later
+ * report-apply caller (task 3.2) checks a report's carried round token
+ * against this before applying anything from it.
+ *
+ * Precondition: caller holds at least `w`'s read lock. */
+extern int resync_round_is_current(worker_t* w, uint64_t token);
+
+#ifdef XCE_TESTING
+/* worker-rejoin-resync task 3.1 test seam: directly invokes the SAME
+ * internal timeout-fire logic resync_arm_round's own delaymgr callback runs
+ * (resync_on_timeout, resync.c -- file-static, not otherwise reachable from
+ * a test) for the given `token`, without needing a real delaymgr timer
+ * thread or any real wall-clock wait -- treats "the timer fired" as a
+ * directly-testable event, exactly like any other function call. Exercises
+ * the round-token staleness check exactly as a genuine late fire would: a
+ * no-op if `token` is no longer `w`'s current round. Unlike resync_arm_
+ * round/resync_resolve_round above, this ACQUIRES `w`'s own write lock
+ * itself (mirroring the real delaymgr callback, which runs with no lock
+ * already held by any caller) -- so `w` must be a real RTTI-managed
+ * worker_t (e.g. via makeWorker), not a bare stack struct, or the lock call
+ * is undefined behavior. Compiled out of production (mirrors cluster.h's
+ * job_array_apply_test_* seams). */
+extern void resync_test_fire_timeout(cluster_t* c, worker_t* w,
+                                     uint64_t token);
+#endif
+
+#endif