worker: re-report job state on transparent failover reconnect

Covers the case where the worker process stays alive and only the TCP link
to the controller drops (or the controller fails over): libxce's failover
loop reconnects transparently and xcecli_hello() returns 0, so
checking_connection() sees ret==0 and never takes its
xcecd_join_cluster()/resend_job_status() path — job state changes that
occurred while disconnected are never re-reported, leaving stale RUN/UNKWN
jobs on the controller.

- worker/controller_connector.c: new xcecd_reconnect_cb registered via
  xcecli_set_reconnect_handler after a successful join, resending the actual
  job state (bug-1509 semantics) on every transparent reconnect.
- lib/client.c (bug-1522): xcecli_start no longer resets rejoin_cb/rejoin_arg
  to NULL — it is initialized by xcecli_ctx_init's calloc, and a pre-start
  registration (scheduler sched_reconnect_cb, worker xcecd_reconnect_cb) must
  survive first connect; the failover loop invokes it after a transparent
  reconnect (xce_ctx_reconnect), which never re-runs xcecli_start. Previously
  every pre-start registration was dead code.
- scheduler.c: move xcecli_set_reconnect_handler to AFTER a successful join —
  xcecli_join also runs the failover loop, and a candidate-iteration rotate
  (standby redirect) must not fire the rejoin hook before the session exists
  (was firing sched_reconnect_cb during initial join → rejoin on
  uninitialized state → invalid-opcode crash).
- test_libxce_retry.c: regression test for bug-1522
  (test_reconnect_handler_survives_start_state_reset). Suite 296/296.

Functional verification on the test cluster: worker joins (host OK), runs a
job, and after a controller-link break re-joins ("Rejoined cluster
successfully after retry N"). Scheduler stays up (no invalid-opcode crash).

Co-Authored-By: Claude <noreply@anthropic.com>

src/worker/src/controller_connector.c

@@ -160,6 +160,44 @@ static void init_client_config(xce_client_config_t* config)
               config->key);
 }
 
+/**
+ * @brief Reconnect callback — invoked by libxce's transparent failover after
+ *        a connection is re-established (and the ctx recovered), BEFORE the
+ *        interrupted op resumes.
+ *
+ * This covers the case where the worker process stays alive and only the
+ * TCP link to the controller dropped (or the controller failed over): the
+ * failover retry loop in xcecli_hello() reconnects transparently and returns
+ * success, so checking_connection() sees ret==0 and never takes its
+ * xcecd_join_cluster()/resend_job_status() path. Without this hook, job
+ * state changes that occurred while disconnected are never re-reported and
+ * the controller is left with stale RUN/UNKWN jobs.
+ *
+ * @return 0 to let the interrupted op resume; <0 would be treated by the
+ *         failover loop as "new primary rejected rejoin" and it would rotate.
+ */
+static int xcecd_reconnect_cb(xcecli_ctx_t c, void* arg)
+{
+    (void)c;
+    (void)arg;
+
+    if (cc_ctx == NULL || cc_ctx->worker_list == NULL
+        || *cc_ctx->worker_list == NULL) {
+        return 0;
+    }
+
+    XCECD_LOG(XCE_LOG_INFO,
+              "Controller connection restored; resending job status");
+
+    /* Same semantics as checking_connection()'s rejoin resend (bug-1509
+     * B2/B3): report the ACTUAL job state so the controller reconciles its
+     * view (clears UNKWN on a live claim; converges to a terminal state for
+     * processes that are gone). */
+    (void)resend_job_status(*cc_ctx->worker_list);
+
+    return 0;
+}
+
 /**
  * @brief Handles I/O events for the endpoint, focusing on error conditions
  *        and connection liveness checks using EPOLLIN/EPOLLOUT.
@@ -267,6 +305,17 @@ int xcecd_join_cluster(hostattrs_t* attrs)
                 XCECD_LOG(XCE_LOG_INFO,
                           "xcecd_join_cluster: joined primary at %s:%d",
                           host, port);
+
+                /* Re-report running job state when libxce transparently
+                 * reconnects (failover) — checking_connection() only covers
+                 * the explicit xcecd_join_cluster path, not the auto-reconnect
+                 * path. Registered AFTER a successful join: xcecli_join also
+                 * runs the failover loop, and a candidate-iteration rotate
+                 * (standby redirect) must NOT fire the rejoin hook before the
+                 * session exists. */
+                xcecli_set_reconnect_handler(xcecli_ctx, xcecd_reconnect_cb,
+                                             NULL);
+
                 return 0;
             }
 
@@ -299,6 +348,10 @@ int xcecd_join_cluster(hostattrs_t* attrs)
         return rc;
     }
 
+    /* Same reconnect hook as the HA path above — after a successful join so a
+     * candidate-iteration rotate never fires it pre-session. */
+    xcecli_set_reconnect_handler(xcecli_ctx, xcecd_reconnect_cb, NULL);
+
     return 0;
 }