worker: re-report job state on transparent failover reconnect

Covers the case where the worker process stays alive and only the TCP link
to the controller drops (or the controller fails over): libxce's failover
loop reconnects transparently and xcecli_hello() returns 0, so
checking_connection() sees ret==0 and never takes its
xcecd_join_cluster()/resend_job_status() path — job state changes that
occurred while disconnected are never re-reported, leaving stale RUN/UNKWN
jobs on the controller.

- worker/controller_connector.c: new xcecd_reconnect_cb registered via
  xcecli_set_reconnect_handler after a successful join, resending the actual
  job state (bug-1509 semantics) on every transparent reconnect.
- lib/client.c (bug-1522): xcecli_start no longer resets rejoin_cb/rejoin_arg
  to NULL — it is initialized by xcecli_ctx_init's calloc, and a pre-start
  registration (scheduler sched_reconnect_cb, worker xcecd_reconnect_cb) must
  survive first connect; the failover loop invokes it after a transparent
  reconnect (xce_ctx_reconnect), which never re-runs xcecli_start. Previously
  every pre-start registration was dead code.
- scheduler.c: move xcecli_set_reconnect_handler to AFTER a successful join —
  xcecli_join also runs the failover loop, and a candidate-iteration rotate
  (standby redirect) must not fire the rejoin hook before the session exists
  (was firing sched_reconnect_cb during initial join → rejoin on
  uninitialized state → invalid-opcode crash).
- test_libxce_retry.c: regression test for bug-1522
  (test_reconnect_handler_survives_start_state_reset). Suite 296/296.

Functional verification on the test cluster: worker joins (host OK), runs a
job, and after a controller-link break re-joins ("Rejoined cluster
successfully after retry N"). Scheduler stays up (no invalid-opcode crash).

Co-Authored-By: Claude <noreply@anthropic.com>

src/scheduler/CLAUDE.md

@@ -17,6 +17,29 @@ meson build && ninja -C build
 
 Links against `libxce` from `src/lib/`.
 
+## Current scheduling policies (2026-08-15)
+
+There is exactly **one** job-selection policy today, plus one constraint hook and two
+allocators. The plugin registry (`scheduler.c:356-359`) registers all four; the job
+picker is selected by a **hardcoded** name in `sched_job_picker()` (`scheduler.c:44`,
+`job_pick_plugin = "xce_sched_fcfs"`) — there is no runtime way to swap it.
+
+| Plugin | Role | Notes |
+|---|---|---|
+| `fcfs.c` | **Job picker / policy** | Only election plugin. Orders queues by priority, jobs by (priority desc, queue_time asc). Runs in a dedicated `jobpicker` thread; emits picked jobid via `jbuf` cbuf. |
+| `limit.c` | **Constraint hook, NOT a policy** | Hooks into `job_election_hook` + `prepare_allocation_hook`: per-queue/user limits/quota. Blocks over-limit jobs and attaches a PEND reason. Runs alongside the picker, does not choose jobs itself. |
+| `seqjob_allocator.c` | **Allocator** | `maxTasks == 1` (serial) resource placement. |
+| `parjob_allocator.c` | **Allocator (stub)** | `my_alloc()` is effectively empty — parallel/multi-slot jobs get no allocation. |
+
+Election (pick) and allocation are two decoupled steps: fcfs picks *which* job, the
+allocators decide *where* it runs. No fairshare / priority-weighted / backfill policy
+exists yet.
+
+**Future work** — see `.kiro/specs/scheduler-framework/brief.md` ("Current State" +
+"What's missing"): configurable job picker (deferred), parallel allocation, iterator
+generation, preemption, project-level scheduling, simulation mode. This doc is the
+authoritative *current-state* note; the brief is the roadmap.
+
 ## Conventions
 - C standard: gnu17
 - Format with `clang-format`
@@ -26,3 +49,4 @@ Links against `libxce` from `src/lib/`.
 ## See Also
 - Shared library: `src/lib/CLAUDE.md`
 - Root guidelines: `CLAUDE.md` (repo root)
+- Framework roadmap: `.kiro/specs/scheduler-framework/brief.md`

src/scheduler/src/scheduler.c

@@ -506,7 +506,6 @@ int sched_init(xce_scheduler_t* sched, xce_client_config_t* config,
     }
 
     xcecli_set_handler(sched->apictx, handler, sched);
-    xcecli_set_reconnect_handler(sched->apictx, sched_reconnect_cb, sched);
 
     /* HA mode: iterate candidates until one accepts as primary */
     if (sched->cluster_cfg.candidates_count > 0) {
@@ -536,8 +535,6 @@ int sched_init(xce_scheduler_t* sched, xce_client_config_t* config,
                 continue;
             }
             xcecli_set_handler(sched->apictx, handler, sched);
-            xcecli_set_reconnect_handler(sched->apictx, sched_reconnect_cb,
-                                         sched);
 
             rc = xcecli_start(sched->apictx);
             if (rc < 0) {
@@ -548,6 +545,12 @@ int sched_init(xce_scheduler_t* sched, xce_client_config_t* config,
             rc = xcecli_join(sched->apictx, HostRole_scheduler, NULL);
             if (rc == 0) {
                 joined = 1;
+                /* Register AFTER a successful join: xcecli_join also runs the
+                 * failover loop, and a candidate-iteration rotate (standby
+                 * redirect) must NOT fire the rejoin hook before the session
+                 * exists. Also after start(), which resets rejoin_cb. */
+                xcecli_set_reconnect_handler(sched->apictx,
+                                             sched_reconnect_cb, sched);
                 break;
             }
 
@@ -570,6 +573,9 @@ int sched_init(xce_scheduler_t* sched, xce_client_config_t* config,
         if (rc < 0) {
             return rc;
         }
+        /* After successful join (and after start, which resets rejoin_cb) —
+         * same reasoning as the HA path above. */
+        xcecli_set_reconnect_handler(sched->apictx, sched_reconnect_cb, sched);
     }
 
     rc = pthread_create(&(sched->jobpicker), NULL, &sched_job_picker, sched);