lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

worker: handle pre-exec failure without dying; persist: skip snapshot queues not in config

Browse source 
Two independent fixes:

1. Worker pre-exec failure (TODO fix). queue_pre_exec()/job_pre_exec()
   previously called exit(1) on failure, killing the one-shot worker
   before any status report reached the master -- the job then hung with
   no user-visible reason and the worker process vanished. Now the
   failure path reports the job FAILED via ipc_worker_send_status with
   stage JOB_EXEC_STAGE_{Q_PRE,PRE}_EXEC and the command's real exit
   code, skips the job body (no post-exec -- the body never ran,
   mirroring LSF pre-exec semantics), and takes the same tail as the
   VNC/X11 start failures. main.c now carries job_status->stage through
   to xcecd_job_finished() so the stage (not the MAIN default) reaches
   the controller's fail fact, deciding both the exit-status merge
   (apply_job_fail_transition) and whether the exit code is recorded
   (apply_job_fail_exit_code). Also fixes the error print: uv_proc_pre_exec
   returns a WEXITSTATUS, not a uv errno, so uv_strerror mangled a plain
   exit status into "Unknown system error N".

   E2E verified on t1: job-level pre-exec `exit 7` -> job EXIT,
   worker logs "Job pre-exec command failed with exit code 7", marker
   file shows only the pre-exec ran (no post-exec, no job body), and
   the exit code reaches the apiserver's OnJobFinished (exit_code=7)
   and the committed WAL_JOB_FAIL fact.

2. Snapshot queue no longer in config wedged election forever.
   Checkpoint snapshots record every queue in the live qmgr; if a queue
   is deleted from queues.yaml between the checkpoint and a restart,
   restore_queue_state returns -ENOENT for it ("queue not yet built
   from config -- skip" per its own contract), but
   load_snapshot_entities treated any non-zero as fatal ->
   persist_recover failed -> cluster_init_phase2 retried forever
   (bug-1477 never self-terminates) and no primary was ever elected.
   The snapshot queue loop now treats -ENOENT/-EINVAL as a benign skip
   with a WARN, matching the same documented policy as
   apply_committed_entry's and apply_wal_entry's WAL_QUEUE_CTRL cases;
   every other non-zero return stays a genuine failure. Doc comments
   updated accordingly.

   Live-reproduced and verified on t1 (2026-10-22): deleting a test
   queue + restarting the controller previously wedged election
   permanently; with the fix the controller boots past the stale frame
   with a WARN and reaches CTRL_PRIMARY.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-22 07:17:01 +00:00
parent 5b9d727086
commit e8cb6d5743
3 changed files with 115 additions and 36 deletions
Download patch file Download diff file Expand all files Collapse all files

48
src/controller/persist.c
View file

@ -3875,12 +3875,15 @@ static int persist_checkpoint_marker_triggered(persist_t* p,
* own documented idempotent-replay guard (job already present), needed so a
* retried load — after an earlier partial attempt already created some jobs
* before failing on a later one — can continue past already-restored entries
* rather than aborting on them again. restore_queue_state and
* restore_host_state have no such idempotent-duplicate case of their own to
* special-case here (re-applying the same status is already a harmless
* overwrite/OR, and restore_host_state's create-if-missing placeholder path
* is independently idempotent under retry — Requirement 5.8, worker.c), so
* any non-zero return from either is a genuine failure. */
* rather than aborting on them again. restore_host_state has no
* idempotent-duplicate case of its own to special-case here (re-applying the
* same status is already a harmless overwrite/OR, and its create-if-missing
* placeholder path is independently idempotent under retry — Requirement 5.8,
* worker.c), so any non-zero return from it is a genuine failure.
* restore_queue_state's -ENOENT/-EINVAL are NOT failures (see the queue loop
* below): a snapshot can name a queue the current boot's config no longer
* provisions, and that frame is skipped with a WARN — every other non-zero
* return from it is a genuine failure. */
static int load_snapshot_entities(int fd, off_t file_size, off_t consumed,
persist_t* p, uint64_t stored_lsn)
{
@ -3967,9 +3970,28 @@ static int load_snapshot_entities(int fd, off_t file_size, off_t consumed,
free(buf);
int qrc = restore_queue_state(p->cluster, decoded.name, decoded.status);
free_QueueAdminState(&decoded);
if (qrc != 0)
if (qrc == -ENOENT || qrc == -EINVAL) {
/* The snapshot can legitimately name a queue this boot's config
* no longer provisions (queue deleted from queues.yaml between
* the checkpoint and this restart): restore_queue_state's
* "queue not yet built from config" contract means there is
* nothing to apply the admin state to. Treat it as a benign
* skip — the SAME policy apply_committed_entry's and
* apply_wal_entry's WAL_QUEUE_CTRL cases already document
* ("restore_queue_state's -ENOENT ... benign skip") — rather
* than a fatal restore error. Failing loud here wedges the
* member permanently: cluster_init_phase2 retries forever
* (bug-1477 never self-terminates) and the cluster never
* elects a primary, live-reproduced 2026-10-22 on t1 after
* removing a test queue + restarting the controller. */
WARN("persist_recover: snapshot queue '%s' not in this boot's "
"config (rc=%d) — skipping its admin state",
decoded.name ? decoded.name : "(null)", qrc);
} else if (qrc != 0) {
free_QueueAdminState(&decoded);
return qrc;
}
free_QueueAdminState(&decoded);
}
/* Host section (config-upgrade-snapshot task 4, Req 1.10): new in
@ -3981,7 +4003,9 @@ static int load_snapshot_entities(int fd, off_t file_size, off_t consumed,
* — Design Invariant 1) and idempotent under retry on its own
* (Requirement 5.8), so this loop applies the exact same "any failure
* returns immediately" discipline as the job/queue loops above without
* needing any extra idempotency handling here. */
* needing any extra idempotency handling here (the queue loop's one
* exception — a benign -ENOENT/-EINVAL skip for a queue this boot's
* config no longer provisions — is documented there). */
uint32_t host_count = 0;
if (read(fd, &host_count, sizeof(host_count)) !=
(ssize_t)sizeof(host_count))
@ -4062,8 +4086,10 @@ static int load_snapshot_entities(int fd, off_t file_size, off_t consumed,
*
* Returns 0 on success, -EINVAL on bad magic / unrecognized entity format,
* -EIO on a short/corrupt read, -ENOMEM on allocation failure, -errno on
* open/fstat failure, or a restore_job/restore_queue_state/restore_host_state
* error code. */
* open/fstat failure, or a genuine restore_job/restore_queue_state/
* restore_host_state error code (restore_queue_state's -ENOENT/-EINVAL for a
* queue no longer in this boot's config is a documented benign skip, not a
* failure). */
static int load_snapshot(persist_t* p, const char* path, uint64_t lsn,
bool* out_entities_restored)
{