myxce/src/controller
lei.chen 9d7628ef3f worker: re-report job state on transparent failover reconnect
Covers the case where the worker process stays alive and only the TCP link
to the controller drops (or the controller fails over): libxce's failover
loop reconnects transparently and xcecli_hello() returns 0, so
checking_connection() sees ret==0 and never takes its
xcecd_join_cluster()/resend_job_status() path — job state changes that
occurred while disconnected are never re-reported, leaving stale RUN/UNKWN
jobs on the controller.

- worker/controller_connector.c: new xcecd_reconnect_cb registered via
  xcecli_set_reconnect_handler after a successful join, resending the actual
  job state (bug-1509 semantics) on every transparent reconnect.
- lib/client.c (bug-1522): xcecli_start no longer resets rejoin_cb/rejoin_arg
  to NULL — it is initialized by xcecli_ctx_init's calloc, and a pre-start
  registration (scheduler sched_reconnect_cb, worker xcecd_reconnect_cb) must
  survive first connect; the failover loop invokes it after a transparent
  reconnect (xce_ctx_reconnect), which never re-runs xcecli_start. Previously
  every pre-start registration was dead code.
- scheduler.c: move xcecli_set_reconnect_handler to AFTER a successful join —
  xcecli_join also runs the failover loop, and a candidate-iteration rotate
  (standby redirect) must not fire the rejoin hook before the session exists
  (was firing sched_reconnect_cb during initial join → rejoin on
  uninitialized state → invalid-opcode crash).
- test_libxce_retry.c: regression test for bug-1522
  (test_reconnect_handler_survives_start_state_reset). Suite 296/296.

Functional verification on the test cluster: worker joins (host OK), runs a
job, and after a controller-link break re-joins ("Rejoined cluster
successfully after retry N"). Scheduler stays up (no invalid-opcode crash).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-15 08:40:20 +00:00
..
include fix(controller,worker): dual-state worker loss + resend actual job state (bug-1509 P3) 2026-08-06 06:49:24 +00:00
testsuite fix(controller,worker): dual-state worker loss + resend actual job state (bug-1509 P3) 2026-08-06 06:49:24 +00:00
apiserver.c feat(apiserver): expose PEND reason end-to-end (scheduler → API) 2026-08-13 09:16:13 +00:00
CLAUDE.md worker: re-report job state on transparent failover reconnect 2026-08-15 08:40:20 +00:00
cluster.c fix(controller): mark RUN jobs unknown on worker loss and restart (bug-1509 P2) 2026-08-06 05:57:59 +00:00
commit.c feat(raft-apply-decoupling): ordered live apply pump activates apply_committed_entry (task 6.1) 2026-07-04 19:00:43 +00:00
config_repl.c fix(controller): trim config versions by ns mtime; unique test tmpdir 2026-08-01 03:34:55 +00:00
config_repl.h feat(raft-adapter-conformance): send explicit reply on every config-push outcome 2026-07-29 09:43:35 +00:00
ctrl_metrics.c feat: P3-7 ctrl_metrics — 7 atomic counters/gauges + descriptor table + persist wiring 2026-05-08 00:04:06 +00:00
ctrlstatus.c feat(ha-heartbeat-committed-propagation): migrate last_known_committed_config_tag to _Atomic (task 1.1) 2026-06-20 11:27:46 +00:00
dataex.c fix(persist): zero nresvals_to_sched after free_name_val_list 2026-05-20 09:39:07 +00:00
egress.c feat(ha-convergence-recovery): commit-gated scheduler egress — submit_lsn tagging + watermark gate (task 1.2) 2026-06-14 02:33:36 +00:00
ha.c docs(raft-adapter-conformance): document dormant AppendEntries sender + non-regression guard 2026-07-29 15:04:07 +00:00
ha.h docs(raft-adapter-conformance): document dormant AppendEntries sender + non-regression guard 2026-07-29 15:04:07 +00:00
ha_convergence.c fix(controller): bug-1450/1451 offload delaymgr-inline blocking work to the slow lane 2026-07-19 08:14:00 +00:00
job.c fix(controller,worker): dual-state worker loss + resend actual job state (bug-1509 P3) 2026-08-06 06:49:24 +00:00
join.c fix(controller): bug-1433 async-thread fdatasync block + bug-1434 committed_lsn CAS race 2026-07-11 12:16:28 +00:00
main.c fix(config): serialize load_primary_config against concurrent invocations (bug-1476) 2026-07-28 08:13:54 +00:00
meson.build feat(raft-adapter-conformance): wire term-adoption into primary-side config-push reply handler 2026-07-29 10:56:48 +00:00
misc.c save works 2026-03-05 16:03:37 +08:00
myapp.c fix(ha): retry cluster_init_phase2 on failure instead of latching phase2_done unconditionally (bug-1477) 2026-07-28 13:25:48 +00:00
myconf.c fix(config): serialize load_primary_config against concurrent invocations (bug-1476) 2026-07-28 08:13:54 +00:00
opget.c fix(schema): rename struct Data -> XceData to unshadow capnp builtin blob type 2026-07-22 09:22:04 +00:00
opupd.c save works 2026-03-03 16:49:26 +08:00
persist.c fix(controller): mark RUN jobs unknown on worker loss and restart (bug-1509 P2) 2026-08-06 05:57:59 +00:00
queue.c feat(ha-commit-barrier-rollout): commit-first-then-reply in all four handlers (task 3.2) 2026-06-12 10:28:25 +00:00
repl.c fix(ha): close permanent quorum-loss deadlock on log-conflicted standby (bug-1479) 2026-07-29 04:11:32 +00:00
reply.c fix(persist): zero nresvals_to_sched after free_name_val_list 2026-05-20 09:39:07 +00:00
resources.c update controller code 2025-11-05 16:58:57 +08:00
sched.c fix(controller): guard config push against NULL cluster_config on cold boot 2026-08-11 03:19:02 +00:00
slowlane.c feat(raft-drive-async): SlowLane bounded executor for long blocking I/O (task 1.2) 2026-06-26 05:24:00 +00:00
startup_validation.c feat(ha-heartbeat-liveness): add validate_heartbeat_config startup check (task 1.2) 2026-06-01 12:26:29 +00:00
task.c save works 2026-03-04 18:18:53 +08:00
worker.c fix(controller,worker): dual-state worker loss + resend actual job state (bug-1509 P3) 2026-08-06 06:49:24 +00:00