lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

merge: sync Rongsong's scheduler-framework (122 commits, 070c7869..f01ae353)

Browse source 
Introduces WAL replay field-completeness fixes (bug-1687..1697, 1731), scheduler snapshot job-allocation restore, heap UAF fixes (vtask_enq, xcealloc, builtin_fcfs, limit), HA fixes (bug-1694/1695/1696), xceadmin retry, worker signal-exit severity, and their test suites.

Conflicts resolved keeping both sides (drop-audit passed):
- worker.c: mark_worker_jobs_exit (bug-1509 B1) + fetchfile_signal_worker_disconnect (bug-1693)
- handler.go: DefaultBroker.Publish (EventJobFinished) + bug-1731 logging
- config_repl.c / test_config_repl.c: our ns-resolution comment / unique-tmpdir
- test_dispatch_commit.c: both B3-UNKWN and bug-1697 start_time assertions

Excluded Rongsong's bookkeeping per policy: .wolf/, .kiro/, CHANGES.md, reports/;
.claude/ + CLAUDE.md kept ours (his settings.json has machine-absolute paths).
This commit is contained in:
lei.chen 2026-08-31 14:24:17 +00:00
commit 3b6f53bb65
168 changed files with 20744 additions and 568495 deletions
Download patch file Download diff file Expand all files Collapse all files

89
src/include/persist.h
View file

@ -1058,11 +1058,41 @@ int persist_test_force_new_segment(persist_t* p);
* comment above). Pass a negative index to disable. Lock-free atomic
* write; compiled out of production. */
void persist_test_set_force_job_encode_fail_at(persist_t* p, int index);
/* Test seam: directly drive the delaymgr-triggered checkpoint task
* (checkpoint_task_fn, persist.c) without waiting out delay_task's real 1s
* delay. Returns the task fn's own return value (always 0). Compiled out of
* production — the only real call site is via delay_task. */
int persist_checkpoint_task_fn_for_test(persist_t* p);
/* Test-injectable substitute for the real (blocking) checkpoint work; when
* non-NULL, checkpoint_lane_fn (persist.c) calls this INSTEAD OF
* persist_checkpoint() — lets a test park/gate the slow-lane worker in
* flight. NULL in every fixture that never installs it. Symbol always
* compiled (like ha.c's ha_repl_send_snapshot_hook) so test TUs can extern
* it; unused in a non-XCE_TESTING build. */
extern void (*persist_checkpoint_test_hook)(persist_t*);
#endif /* XCE_TESTING */
/* Checkpoint */
int persist_checkpoint(persist_t* p);
/*
* persist_set_checkpoint_offload_for_test / persist_checkpoint_offload_enabled:
* Feature-flag accessors (xce-audit-invariants controller pass, dimension 1
* async-I/O thread-role fix, bug-1694). The offload flag gates whether
* checkpoint_task_fn (persist.c) submits the real checkpoint work to the
* cluster's slow lane (ON, the production default — the delaymgr thread
* never blocks on checkpoint I/O) or runs the legacy inline blocking
* checkpoint on the delaymgr thread (OFF, retained only so the TDD RED test
* can demonstrate the pre-fix stall). See checkpoint_task_fn's own doc
* comment (persist.c) for the full design rationale, including why the
* destination is the slow lane and not the vtaskmgr pool. These accessors
* are compiled in ALL builds; the setter is used by tests to flip the flag.
*/
void persist_set_checkpoint_offload_for_test(int on);
int persist_checkpoint_offload_enabled(void);
/* WAL truncation — remove all entries with LSN > cutoff_lsn.
* Idempotent; returns 0 on success, negative errno on I/O error. */
int persist_truncate_wal_above(persist_t* p, uint64_t cutoff_lsn);
@ -1417,10 +1447,67 @@ int persist_submit_job_fact_ex(persist_t* p, uint32_t wal_type,
* itself was never sufficient for this one fact type. `alloc` must be
* non-NULL (the real call site, dispatch_a_job_(), always has one by
* construction). Returns the local-append rc, same contract as
* persist_submit_job_fact(). */
* persist_submit_job_fact().
*
* bug-1697: additionally takes `start_time` (the dispatch-time clock,
* captured once by the caller under the job's write lock -- the same
* snapshot stamped into job.start_time itself, mirroring how
* persist_submit_job_start_fact() below takes its own receipt-time
* start_time). Encoded between the wal_jobstate_t header and the trailing
* JobAlloc bytes, discriminated by the new WAL_JOB_DISPATCH_HAS_START_TIME_
* FLAG bit in wal_jobstate_t._pad (wal_types.h) -- NOT a payload-length
* heuristic, since JobAlloc's own variable length makes old-vs-new length
* ambiguous. job.start_time is now DISPATCH-owned: dispatch_a_job_() sets
* job.state = JOB_STAT_RUN synchronously at DISPATCH time, so any LATER
* WAL_JOB_START fact for the same job always finds state already == RUN by
* the time it applies and can never be the genuine first RUN transition --
* see wal_types.h's WAL_JOB_START doc comment for the full rationale. */
int persist_submit_job_dispatch_fact(persist_t* p, uint64_t jobid, int state,
uint64_t start_time,
const xce_job_alloc_t* alloc);
/* bug-1687: WAL_JOB_START's own sibling of persist_submit_job_fact(),
* mirroring persist_submit_job_dispatch_fact() (bug-1678) exactly but for
* the worker-confirmed-start fact -- additionally encoding the
* worker-reported job_pidinfo_t (real pids/pgids/exdata) AND a receipt-time
* start_time into the fact's payload (header {jobid, state} + a fixed
* uint64_t start_time + trailing Cap'n Proto JobPIDInfo bytes) so a
* restart/catch-up replay can reconstruct job.pidinfo. `info` must be
* non-NULL (the real call site, record_job_start_fact(), always has one by
* construction -- it is the worker's START report itself). Returns the
* local-append rc, same contract as persist_submit_job_fact().
*
* bug-1697: `start_time` is still written here (wire layout unchanged, for
* compatibility with any reader of the on-disk format) but is NO LONGER
* adopted by either apply seam -- job.start_time is now exclusively
* DISPATCH-owned (see persist_submit_job_dispatch_fact() above and
* wal_types.h's WAL_JOB_START doc comment for the full rationale). Kept as
* a caller parameter and wire field purely for layout/diagnostic
* continuity, not because it is still consumed. */
int persist_submit_job_start_fact(persist_t* p, uint64_t jobid, int state,
uint64_t start_time,
const job_pidinfo_t* info);
/* bug-1689: WAL_JOB_FINISH's own sibling of persist_submit_job_fact(),
* mirroring persist_submit_job_start_fact() (bug-1687) exactly but for the
* worker-confirmed-finish fact -- additionally encoding a receipt-time
* finish_time AND the worker-reported exit_code into the fact's payload
* (header {jobid, state} + a fixed uint64_t finish_time + a fixed int32_t
* exit_code) so a restart/catch-up replay can reconstruct job.exit_info and
* the true job.finish_time instead of leaving exit_info permanently NULL and
* finish_time fabricated fresh (time(NULL)) at every apply. job_exitinfo_t
* is a single int with no existing Cap'n Proto mapping (no JobExitInfo
* struct in controller.capnp) -- this is a small NATIVE trailing payload,
* mirroring wal_jobstate_t's own native-struct style, not a capnp wrapper
* (unlike persist_submit_job_start_fact()'s variable-length JobPIDInfo,
* which genuinely needed one). run_time is NOT carried on the wire -- it is
* a pure derivation (finish_time - start_time), re-derived at each apply
* seam once both are known, mirroring apply_job_finish_transition()'s own
* formula. Returns the local-append rc, same contract as
* persist_submit_job_fact(). */
int persist_submit_job_finish_fact(persist_t* p, uint64_t jobid, int state,
uint64_t finish_time, int exit_code);
/* bug-1679: fire XCE_CONTROLLER_HOOK_JOB's HOOK_DETAIL_JOB_UPDATE AFTER_EVENT
* for a job whose durable WAL fact the caller has ALREADY appended itself
* (e.g. dispatch_a_job_()'s own persist_submit_job_dispatch_fact() call,

192
src/include/wal_types.h
View file

@ -54,6 +54,32 @@ typedef struct {
* (conditional EXIT backstop + OR JOB_STAT_PDONE) fresh against current
* state. Any other nonzero value is reserved/unused today. */
/* wal_jobstate_t._pad discriminator for WAL_JOB_DISPATCH (bug-1697).
* dispatch_a_job_() (job.c, scheduler-dispatch-commit task 3.1) sets
* job.state = JOB_STAT_RUN synchronously at DISPATCH time, before the
* worker ever reports back — so by the time the LATER WAL_JOB_START fact
* commits and applies, apply_job_update_transition() always finds state
* already == RUN and reports JOB_TRANSITION_NOOP_DUPLICATE, permanently
* disabling any adoption gated on a genuine forward transition for that
* fact type. start_time is therefore moved to be DISPATCH-owned instead:
* WAL_JOB_DISPATCH's payload can now carry a fixed uint64_t start_time
* (the dispatch-time clock, stamped once under the job's write lock)
* between the wal_jobstate_t header and the trailing Cap'n Proto JobAlloc
* bytes (bug-1678). Because JobAlloc's own encoding is variable-length and
* always present in every post-bug-1678 WAL_JOB_DISPATCH entry, a
* length-based "has start_time or not" test would be ambiguous (an
* old-format entry with a large alloc can look identical in length to a
* new-format entry with a small one) — this explicit _pad bit is the
* discriminator instead, mirroring WAL_JOB_FAIL_STAGE_FLAG/WAL_JOB_UPDATE_
* POSTDONE_MARK's own precedent above. _pad is unconditionally 0 for every
* WAL_JOB_DISPATCH entry written before this fix, so introducing this bit
* is safe: a payload with the flag CLEAR decodes exactly the pre-fix shape
* (header immediately followed by JobAlloc bytes, start_time left
* untouched — the accepted one-time migration-boundary gap for any entry
* written by an older binary); a payload with the flag SET decodes
* [header][uint64_t start_time][JobAlloc bytes]. */
#define WAL_JOB_DISPATCH_HAS_START_TIME_FLAG 0x1u
/* WAL_QUEUE_CTRL — queue name is a NUL-terminated string immediately after
* this header. Total payload = sizeof(wal_queuehdr_t) + strlen(name)+1. */
typedef struct {
@ -101,15 +127,143 @@ _Static_assert(sizeof(wal_entry_hdr_t) == 40,
* ============================================================ */
typedef enum {
WAL_JOB_SUBMIT = 1, /* SYNC replication */
WAL_JOB_START = 2,
WAL_JOB_FINISH = 3,
WAL_JOB_START = 2, /* worker-confirmed job start. Payload is
* wal_jobstate_t{jobid, state=RUN, _pad=0} for any
* entry written before bug-1687; as of bug-1687 the
* real producer (persist_submit_job_start_fact)
* appends a fixed uint64_t start_time (receipt-time
* clock) plus trailing Cap'n Proto JobPIDInfo bytes
* (the worker-reported pids/pgids/exdata) -- both
* apply seams treat a payload of exactly
* sizeof(wal_jobstate_t) as the legacy state-only
* shape for full backward compatibility.
* bug-1697: the wire-carried start_time field is
* still WRITTEN (wire layout unchanged, so an
* older reader keeps working) but is no longer
* READ/adopted by either apply seam -- dispatch_a_
* job_() setting job.state = JOB_STAT_RUN
* synchronously at DISPATCH time means this fact
* always finds state already == RUN by the time
* it applies, so it can never be the genuine
* first RUN transition; WAL_JOB_DISPATCH (below)
* is now the sole owner/source of job.start_time.
* pidinfo adoption is unaffected and remains
* unconditional (not gated on a forward
* transition), since it is now known this fact's
* transition is a permanent, expected no-op, not
* an edge case. */
WAL_JOB_FINISH = 3, /* worker-confirmed job finish. Payload is
* wal_jobstate_t{jobid, state=DONE|EXIT, _pad=0}
* for any entry written before bug-1689; as of
* bug-1689 the real producer
* (persist_submit_job_finish_fact) appends a fixed
* uint64_t finish_time (receipt-time clock, not
* fabricated at apply time) plus a fixed int32_t
* exit_code -- both apply seams treat a payload of
* exactly sizeof(wal_jobstate_t) as the legacy
* state-only shape for full backward
* compatibility. job_exitinfo_t is a single int
* with no existing Cap'n Proto mapping, so this is
* a small NATIVE trailing payload (mirroring
* wal_jobstate_t's own native-struct style), not a
* capnp wrapper. run_time is NOT carried on the
* wire -- it is a pure derivation (finish_time -
* start_time), re-derived at each apply seam once
* both are known. */
WAL_JOB_FAIL = 4,
WAL_JOB_STOP = 5,
WAL_JOB_RESUME = 6,
WAL_JOB_UPDATE = 7,
WAL_JOB_RUSAGE = 8,
WAL_JOB_CKPT = 9,
WAL_JOB_PREASON = 10,
WAL_JOB_RUSAGE = 8, /* client-reported per-host CPU/mem usage update.
* Payload is wal_jobstate_t{jobid, state, _pad=0}
* for any entry written before bug-1690, or one
* fired while job.run_usage was still NULL at
* write time (e.g. a test double stubbing out
* update_host_runusage); as of bug-1690
* persist_job_hook_fun's HOOK_DETAIL_JOB_RUSAGE_
* UPDATE case appends trailing Cap'n Proto
* JobRunusage bytes (the job's accumulated
* xce_job_runusage_t) whenever job.run_usage is
* non-NULL -- both apply seams treat a payload of
* exactly sizeof(wal_jobstate_t) as the legacy
* state-only shape for full backward
* compatibility. Unlike WAL_JOB_START/FINISH/
* DISPATCH, run_usage adoption on the
* apply_committed_entry() seam is NOT gated on a
* forward state transition -- a rusage report
* almost never changes job.state, so gating it
* the same way would make it dead code for the
* realistic, repeated-report case. Also unlike
* those three, adoption is a MERGE
* (merge_job_runusage(), job.c), never a
* free-and-replace: job.run_usage can be the
* job's LIVE, in-place-mutated structure (an
* xstring-backed host_name from makeString(),
* not a plain malloc()) when the primary's own
* live apply pump re-applies a fact it just
* wrote -- freeing it via the capnp-generated
* free_JobRunusage_ptr() would corrupt the
* heap. */
WAL_JOB_CKPT = 9, /* worker/scheduler-reported checkpoint info update.
* Payload is wal_jobstate_t{jobid, state, _pad=0}
* for any entry written before bug-1691, or one
* fired while job.ckptinfo was still NULL at write
* time; as of bug-1691 persist_job_hook_fun's
* HOOK_DETAIL_JOB_CKPT case appends a trailing
* NUL-terminated ckptinfo string whenever
* job.ckptinfo is non-NULL -- both apply seams
* treat a payload of exactly sizeof(wal_jobstate_t)
* as the legacy state-only shape for full backward
* compatibility. job.ckptinfo (xce/types.h) is a
* plain char* -- NOT the richer xce_ckptinfo_t/
* CheckpointInfo capnp struct used elsewhere for a
* different purpose -- and is xstring-backed
* (makeString(), xstring.c), so this mirrors
* WAL_QUEUE_CTRL's own [header][name
* NUL-terminated] wire shape rather than inventing
* a capnp wrapper for a bare string. Like
* WAL_JOB_RUSAGE, adoption on the
* apply_committed_entry() seam is NOT gated on a
* forward state transition -- a checkpoint report
* never changes job.state, so gating it the same
* way would make it dead code. */
WAL_JOB_PREASON = 10, /* scheduler-reported host/resource-based pending
* reasons (job.pend_reasons, set by set_preason()/
* cleared by do_clr_preason(), job.c). Payload is
* wal_jobstate_t{jobid, state, _pad=0} for any
* entry written before bug-1692. As of bug-1692,
* persist_job_hook_fun's HOOK_DETAIL_SETPREASON/
* HOOK_DETAIL_CLRPREASON arm appends an explicit
* presence byte after the header: 0 = pend_reasons
* was genuinely cleared (adopt NULL on replay), 1
* = trailing Cap'n Proto JobPendReason bytes
* follow (adopt the decoded value). Unlike
* WAL_JOB_CKPT/WAL_JOB_RUSAGE, "trailing bytes
* present or absent" alone cannot distinguish
* "old-format payload" from "field is legitimately
* NULL", because pend_reasons -- uniquely among
* this fact-type family -- has a real CLEAR
* operation (do_clr_preason) that must survive
* replay durably, not just its SET side; the
* presence byte resolves that ambiguity. A payload
* of exactly sizeof(wal_jobstate_t) (no presence
* byte at all) leaves pend_reasons untouched on
* both apply seams, matching the pre-fix
* state-only behavior exactly. Like
* WAL_JOB_RUSAGE/WAL_JOB_CKPT, adoption on the
* apply_committed_entry() seam is NOT gated on a
* forward state transition -- set_preason()/
* do_clr_preason() never change job.state, so
* gating it the same way would make it dead code.
* Adoption is a plain free-and-replace (via the
* capnp-generated free_JobPendReason_ptr()), NOT a
* merge like WAL_JOB_RUSAGE -- job.pend_reasons
* and its nested host/resource reason arrays are
* plain calloc()/strdup() throughout (verified via
* erase_job_preason()'s FREEUP-only release, NOT
* xstring/RTTI-backed like ckptinfo/rusage's
* host_name), so no merge-not-replace hazard
* exists here. */
WAL_JOB_ARRAY_SUBMIT = 11, /* job-array-single-wal-record: one record for
* the whole array (base spec + index range),
* distinct from WAL_JOB_SUBMIT's single-job
@ -119,10 +273,30 @@ typedef enum {
* running, distinct from (and independent of)
* WAL_JOB_START's later worker-confirmed-start
* fact. Payload is wal_jobstate_t{jobid, state,
* _pad=0} -- always the legacy absolute-value
* form (no racing same-job fact to defer against,
* since this is the first transition after a job
* leaves PEND). */
* _pad=0} immediately followed by trailing Cap'n
* Proto JobAlloc bytes for any entry written
* before bug-1697 (state is always the legacy
* absolute-value form -- no racing same-job fact
* to defer against, since this is the first
* transition after a job leaves PEND). As of
* bug-1697, dispatch_a_job_() sets job.start_time
* synchronously alongside job.state at DISPATCH
* time (see WAL_JOB_START's own updated comment
* above for why start_time moved here): when
* _pad has WAL_JOB_DISPATCH_HAS_START_TIME_FLAG
* set, the payload is instead wal_jobstate_t
* header + a fixed uint64_t start_time (the
* dispatch-time clock) + the trailing JobAlloc
* bytes. The flag, not payload length, is the
* discriminator -- see
* WAL_JOB_DISPATCH_HAS_START_TIME_FLAG's own doc
* comment above for why a length heuristic would
* be ambiguous against JobAlloc's own variable
* length. Any WAL_JOB_DISPATCH entry written
* before this fix has _pad==0 (flag clear) and
* falls back to the pre-fix state+allocs-only
* behavior, leaving start_time untouched -- an
* accepted one-time migration-boundary gap. */
WAL_QUEUE_CTRL = 20,
WAL_WORKER_CTRL = 30,
WAL_NOOP = 40, /* Internal: empty payload, replay no-op; appended only at promotion */

2
src/include/xce/types.h
View file

@ -416,6 +416,8 @@ typedef struct {
char* host_name;
int cpu_time;
double mem_usage;
double mem_swap;
int tasks_current;
} xce_host_runusage_t;
typedef struct {

1
src/include/xcealloc.h
View file

@ -13,5 +13,6 @@ extern xce_job_alloc_t* make_job_alloc(uint64_t, int, xce_host_alloc_t**);
extern xce_alloc_t** make_allocs(int, ...);
extern xce_host_alloc_t** make_host_allocs(int, ...);
extern xce_job_account_t* make_job_account(xce_job_alloc_t*);
extern xce_job_alloc_t* dup_job_alloc(const xce_job_alloc_t*);
#endif