worker: re-report job state on transparent failover reconnect

Covers the case where the worker process stays alive and only the TCP link
to the controller drops (or the controller fails over): libxce's failover
loop reconnects transparently and xcecli_hello() returns 0, so
checking_connection() sees ret==0 and never takes its
xcecd_join_cluster()/resend_job_status() path — job state changes that
occurred while disconnected are never re-reported, leaving stale RUN/UNKWN
jobs on the controller.

- worker/controller_connector.c: new xcecd_reconnect_cb registered via
  xcecli_set_reconnect_handler after a successful join, resending the actual
  job state (bug-1509 semantics) on every transparent reconnect.
- lib/client.c (bug-1522): xcecli_start no longer resets rejoin_cb/rejoin_arg
  to NULL — it is initialized by xcecli_ctx_init's calloc, and a pre-start
  registration (scheduler sched_reconnect_cb, worker xcecd_reconnect_cb) must
  survive first connect; the failover loop invokes it after a transparent
  reconnect (xce_ctx_reconnect), which never re-runs xcecli_start. Previously
  every pre-start registration was dead code.
- scheduler.c: move xcecli_set_reconnect_handler to AFTER a successful join —
  xcecli_join also runs the failover loop, and a candidate-iteration rotate
  (standby redirect) must not fire the rejoin hook before the session exists
  (was firing sched_reconnect_cb during initial join → rejoin on
  uninitialized state → invalid-opcode crash).
- test_libxce_retry.c: regression test for bug-1522
  (test_reconnect_handler_survives_start_state_reset). Suite 296/296.

Functional verification on the test cluster: worker joins (host OK), runs a
job, and after a controller-link break re-joins ("Rejoined cluster
successfully after retry N"). Scheduler stays up (no invalid-opcode crash).

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-08-15 08:40:20 +00:00
parent ee791791bd
commit 9d7628ef3f
6 changed files with 160 additions and 5 deletions

View file

@ -123,6 +123,33 @@ Covered in `.kiro/steering/ha-async-io.md` (connection ownership, async I/O, hea
## Key Files
## Current Functional Status (2026-08-15)
### Completed (HA / reliability layer is mature)
- **Raft/HA family fully landed**: term-safety, commit-durability (ack ⇒ durable on majority), commit-barrier-rollout, commit-engine, wal-replication, apply-decoupling (canonical pre-hooks → commit → apply → notify), single-leader-role, convergence-recovery, config-bootstrap, libxce-failover, controller-status (18/18 tasks).
- **Job-array single WAL record** (job-array-single-wal-record).
- **Scheduler message-gap** G-4/G-5/G-6/G-8 largely implemented (`sched_update_existing_job`, rusage storage, distinct UPDATE_JOB_FIELDS op).
- **job_ctrl** (suspend/resume/hold/release/kill) and **queue_ctrl** (open/close/inactivate) exist in `job.c`/`queue.c`.
### Gaps / missing functionality
These are the main functional shortfalls (see `.kiro/steering/roadmap.md` for full backlog):
| Area | Gap | Location / note |
|---|---|---|
| **Job control residual** | `job_ctrl` direct-mutate path (admin suspend/resume) still mutates state then fires hook — not commit-then-apply (same family as bug-795/828/851/1060, narrowly scoped to admin suspend/resume) | roadmap Hardening: "investigate: job_ctrl direct-mutate path" |
| **Worker rejoin resync** | On rejoin/failover, worker does NOT re-report running/finished job states (`xce_join_req` carries only `hostattrs_t`, no job state) → facts sent during disconnect/failover window are lost | roadmap backlog: worker-rejoin-resync |
| **Component failover adoption** | Daemons still pass single endpoint (`conf->masters[0]`) to libxce; bounded-retry liveness not yet verified per component | roadmap backlog: component-failover-adoption |
| **Config-upgrade snapshot** | `do_upgrade_config_with_bytes` does config-push+restart+per-node replay — can leave divergent materialized state (bug-857); needs entity-bearing snapshots (HARD dep, still 16-byte LSN sentinel) | roadmap backlog: config-upgrade-snapshot, entity-bearing-snapshots |
| **Job-array bulk query/cancel** | Single client op to query/cancel whole array not implemented (per-element only) | roadmap backlog: bulk job-array query/cancel |
| **Preemption** | No mechanism to preempt lower-priority running jobs (high-priority waits for free resources) | roadmap backlog / scheduler-preemption spec |
| **Group building** | `cluster_build_groups()` is an empty TODO stub | `cluster.c:997` |
| **API-server fact delivery** | FIXMEs: job pid / exit code / fail reason not sent to apiserver | `apiserver.c:248,283,318` |
| **Peer auth** | `validate_host()` is a stub returning success for all peers; `OP_upgradeConfig`/`OP_replConfig` unauthenticated (trusted-LAN model) | roadmap Hardening: validate_host, peer-authentication-hardening (security-critical) |
| **Admin login** | xceadmin uses zero-config TLS, no login/token/session/audit | roadmap Hardening: peer-authentication-hardening (d) |
### Note
Reliability (HA/durability/consistency) is the mature layer; the gaps above are mostly **business/operational features** (bulk ops, preemption, auth, rejoin resync) plus a few correctness residuals (job_ctrl mutate-before-commit). See also `src/scheduler/CLAUDE.md` for scheduler-side status.
## See Also
- Shared library: `src/lib/CLAUDE.md`
- Root guidelines: `CLAUDE.md` (repo root)