forked from lei.chen/myxce
fix(controller): apply worker finish arriving before job materializes (bug-1509)
A worker-reported finish may arrive before the job exists: job
construction is deferred to apply_committed_entry (the submit's WAL
entry), leaving a commit->apply window. on_job_finish dropped such
finishes ('can not find job'), so a fast job that finished in that
window stayed RUN forever (e.g. noexec-test/newcfg-test/verify-fix).
Add an in-memory unknown-finish registry: on_job_finish registers the
finish when the job isn't materialized, and apply_committed_entry's
WAL_JOB_SUBMIT case drains it right after restore_job — appending the
durable WAL_JOB_FINISH and mutating the job to DONE/EXIT immediately so
the scheduler never dispatches a job that already finished.
Design: docs/Job-Finish-Robustness-设计文档.md (Part A of A+B; Part B
worker-liveness + reconnect reconciliation deferred to later phases).
Unit tests (red->green): registry register/take, apply-once-job-
materializes (DONE), exit-code maps to EXIT, drain no-op without
registration.
This commit is contained in:
parent
8b81ba7336
commit
a793015f74
5 changed files with 520 additions and 0 deletions
|
|
@ -45,6 +45,16 @@ extern int update_resvals(cluster_t*, worker_t*, int, xce_resval_t*);
|
|||
|
||||
extern int load_jobs(void*);
|
||||
extern int restore_job(cluster_t*, uint64_t, const void*, size_t, uint64_t);
|
||||
/* bug-1509: drain a worker finish that arrived before the job was
|
||||
* materialized (registered by on_job_finish's unknown-finish path). Called
|
||||
* from apply_committed_entry's WAL_JOB_SUBMIT case after restore_job. */
|
||||
extern int apply_registered_finish_if_any(cluster_t*, job_t*);
|
||||
|
||||
#ifdef XCE_TESTING
|
||||
/* Direct probes of the bug-1509 unknown-finish registry (test binary only). */
|
||||
extern int test_unknown_finish_register(const xce_jobfinish_t*);
|
||||
extern int test_unknown_finish_take(uint64_t, xce_jobfinish_t*);
|
||||
#endif
|
||||
/* job-array-single-wal-record task 3.1 (design.md Array Apply Expansion):
|
||||
* shared N-way expansion primitive for a committed WAL_JOB_ARRAY_SUBMIT
|
||||
* payload -- see the doc comment on the definition (job.c) for the full
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue