fix(controller): apply worker finish arriving before job materializes (bug-1509)

A worker-reported finish may arrive before the job exists: job
construction is deferred to apply_committed_entry (the submit's WAL
entry), leaving a commit->apply window. on_job_finish dropped such
finishes ('can not find job'), so a fast job that finished in that
window stayed RUN forever (e.g. noexec-test/newcfg-test/verify-fix).

Add an in-memory unknown-finish registry: on_job_finish registers the
finish when the job isn't materialized, and apply_committed_entry's
WAL_JOB_SUBMIT case drains it right after restore_job — appending the
durable WAL_JOB_FINISH and mutating the job to DONE/EXIT immediately so
the scheduler never dispatches a job that already finished.

Design: docs/Job-Finish-Robustness-设计文档.md (Part A of A+B; Part B
worker-liveness + reconnect reconciliation deferred to later phases).

Unit tests (red->green): registry register/take, apply-once-job-
materializes (DONE), exit-code maps to EXIT, drain no-op without
registration.
This commit is contained in:
lei.chen 2026-08-06 05:40:56 +00:00
parent 8b81ba7336
commit a793015f74
5 changed files with 520 additions and 0 deletions

View file

@ -45,6 +45,16 @@ extern int update_resvals(cluster_t*, worker_t*, int, xce_resval_t*);
extern int load_jobs(void*);
extern int restore_job(cluster_t*, uint64_t, const void*, size_t, uint64_t);
/* bug-1509: drain a worker finish that arrived before the job was
* materialized (registered by on_job_finish's unknown-finish path). Called
* from apply_committed_entry's WAL_JOB_SUBMIT case after restore_job. */
extern int apply_registered_finish_if_any(cluster_t*, job_t*);
#ifdef XCE_TESTING
/* Direct probes of the bug-1509 unknown-finish registry (test binary only). */
extern int test_unknown_finish_register(const xce_jobfinish_t*);
extern int test_unknown_finish_take(uint64_t, xce_jobfinish_t*);
#endif
/* job-array-single-wal-record task 3.1 (design.md Array Apply Expansion):
* shared N-way expansion primitive for a committed WAL_JOB_ARRAY_SUBMIT
* payload -- see the doc comment on the definition (job.c) for the full