merge: sync Rongsong's scheduler-framework (c5b172c7..3442a57d, 781 commits)

Merge origin/scheduler-framework into merge-sf-20260903 (97 commits on
ours: bug-1490/1494/1498/1509/1513/1521/1522 fixes, PEND-reason P1,
scheduler-agent Phase 2, bug-924 owner-pid shim).

Bookkeeping excluded per policy: .kiro/, .wolf/, CHANGES.md, reports/
(92 files dropped); .claude/ + CLAUDE.md keep ours.

Key theirs-side changes absorbed:
- bug-1938/1960: QUEUE_STAT_CLOSED no longer gates dispatch (closed
  queue refuses new submissions only; pending jobs still run) --
  supersedes our QUEUE_CLOSED preason on the dispatch path;
  sched_queue_status_reason drops that branch
- bug-1846: config push now stashes and applies between scheduling
  rounds (sched_stash_config/sched_apply_pending_config), applied at
  the same main-thread round boundary that preserves bug-1513
  serialization
- bug-1760: sched_check_pending_rejoin exposed +
  xcecli_is_connection_broken public predicate
- bug-1936: execuser/exechome/execcwd/exdata carried on submission
  (superset of our bug-1500 execuser fallback; our job-typed
  resolveExecUser removed as dead code)
- bug-1945~1962: apiserver auth hardening (HS256 pinning, default-
  secret rejection, logout revocation, bcrypt, RBAC) -- our API-key
  branch folded into controllers.ParseTokenClaims so both coexist
- bug-1736: rawRunUsage job run-usage fields in the Go model
- xce_test relocated src/controller/testsuite/test.c ->
  src/harness/test.c; controller suite split into
  controller_test.c + fixtures; their clitest fixtures spawn xce_test
  mid-suite under `meson test -j`

Ours preserved across the merge:
- PEND-reason P1 (codes 1-9, preason.h, job_preason machinery) --
  gate re-ported into their relocated allocator (sched_election.c)
  after plugin_job_election, with attach on the allocator thread
  only (bug-1521 race discipline)
- PEND-reason P0 main_reason semantics RECONCILED to theirs: the
  merge initially re-carried our first-detail-add-promotes-main_reason
  fallback in add_job_preason (4e2ac86), but their whole trace/verify
  stack treats "detail-only => main_reason 0" as a contract --
  sched_trace_capture counts main_reason as its own occurrence
  ("Expected 2 Was 3" in test_trace_capture), the outcome renderer
  prints "main reason: none" for detail-only PENDs (test_repl_trace_e2e),
  and the committed .replay/.sim_verify baselines encode main_reason=0
  (test_repl_trace_baseline byte compare, test_trace_equivalence
  off-vs-traced). The fallback is dropped from the shared lib;
  the user-visible "overall reason" survives as an API-layer view:
  rawJobToModel (convert_pure.go) promotes the first detail row to
  main_reason_name when main_reason is 0 (main_reason VALUE marshals
  unchanged). add_job_preason keeps only the NULL-reasons guard
- bug-1498: sched_job_is_ready rejects non-PEND/suspended jobs
- bug-1490/1781: xce_jobctrl_t.action as capnp union discriminant
  (their jobCtrlUnionDiscriminant, same semantics)
- xcecli typed PendingReason field (53c00834), bug-1522 reconnect
  resend, xcecli_force_reconnect + their is_connection_broken both
  exported in client.h/libclient
- t1/t2/t3 test-node hostconfig in src/configs/hosts.yaml (bug-1494),
  /xce/v1/events + /xce/v1/apikeys whitelist routes (agent Phase 2)
- lib ASAN_OPTIONS=detect_leaks=0 test env + their oom-test block
- drop-audit: our bug-1504 fix covered by their bug-1680 (no action)

bug-924 follow-up (replaces the 67f3f6ad scratch-dir wipe): the
take-theirs audit's one "drop" was re-evaluated and REVERTED instead of
re-applied -- a global `find /tmp -name 'xce_*' -exec rm -rf` at suite
start/exit races the LIVE /tmp/xce_* fixtures of sibling suites under
`meson test -j` (harness_unit_test's clitest fixtures spawn xce_test as
a child mid-suite; the wipe destroyed the parent's fixtures ->
test_test_cli failures). Coverage instead:
- mkdtemp_clean shim gains mkdtemp_clean_mark/sweep_since
  (testsuite_support/mkdtemp_clean.[ch])
- new case_scratch_sweep.[ch]: aggregate runners (controller_test.c,
  harness_test.c -- hundreds of cases per process) call
  case_scratch_sweep_begin/_end_ok from setUp/tearDown; a PASSED case's
  dirs are swept immediately in tearDown (after engine reaping), FAILED
  cases keep their dirs for post-mortem; XCE_MKDTEMP_KEEP_ALL=1
  disables. One full controller_unit_test run was observed to leak 12G+
  to ENOSPC without this, cascading into unrelated "Expected Non-NULL"
  failures
- harness_unit_test_deps links the shim library as well

Test reconciliations on adopt:
- core-ready-gates: sched_queue_is_ready now gates only
  INACT_ADMIN|INACT_WINDOW (bug-1938/1960); CLOSED queues elect --
  covered end-to-end by test_queue_admin_control.c
- test_worker_resync: the 4 restart-replay RUN assertions predate the
  bug-1509 B3 marking (8858afed) landing on the same lineage; replayed
  RUN jobs now assert RUN|UNKWN|TRANSIENT (TWR_ASSERT_REPLAYED_RUN);
  terminal DONE/EXIT assertions unchanged
- convert_pure.go: our job-typed resolveExecUser deleted (superseded
  by their bug-1780/1936 two-arg version)
- test_preason: the 3 P0-promotion assertions reverted to the
  detail-keeps-main-0 contract; new tests cover limit-code detail
  rows and explicit-main-wins-over-later-details; apiserver
  convert_test.go gains the detail-only name-promotion cases

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-14 05:17:03 +00:00
commit 0c8607345e
663 changed files with 139848 additions and 13614 deletions

View file

@ -3,12 +3,22 @@
* job-array push receive path (job-array-single-wal-record task 5.3, Req
* 6.3, 6.4, 6.5, 8.4).
*
* REVISED (connection-loss-under-burst fix, see XCE_SCHED_OP_UPDATE_JOB_
* ARRAY's own comment, scheduler.h): sched_on_job_array() no longer expands
* the compact descriptor into N tasks itself -- it just takes ownership and
* enqueues it as a single XCE_SCHED_OP_UPDATE_JOB_ARRAY request. The actual
* per-element expansion (and its dedup check) now runs in driver.c's
* translate_one_pending_update(), covered by test_driver_translate.c's own
* UPDATE_JOB_ARRAY tests. This file now covers only: (a) sched_on_job_array()
* /sched_handle_dataset()'s own input validation and ownership-transfer
* bookkeeping, and (b) that the wire-decode -> handler -> queue chain
* delivers the descriptor's fields correctly, unmodified.
*
* Strategy: pull the real (static) sched_handle_dataset / sched_on_job_array
* into this TU via #include "../src/handler.c" (same white-box pattern as
* test_handlers.c), driven against a REAL sched->jobs kvmgr so the
* per-element dedup check (sched_get_job) is exercised for real rather than
* faked. sched_add_job/sched_get_job below are the SAME kvmgr key_add/
* key_find path production uses (mirrors test_job_update.c's precedent).
* test_handlers.c). sched_add_job/sched_get_job stubs below are unused by
* anything this file still exercises (kept, harmless, for parity with other
* handler.c white-box test files that DO need them).
*/
#define UNIT_TEST 1
#include <errno.h>
@ -27,6 +37,12 @@
* elsewhere is sched_init_config, stubbed below (same as test_handlers.c). */
#include "../src/handler.c"
/* job-array-one-based-index (Req 5.9): array first IDs are tagged
* TAG_JOB_ARRAY, block-aligned, numbered from 1. */
#define ARR_FIRST(blk) \
MAKE_JOBID(1, 2, 1000, TAG_JOB_ARRAY, \
(((uint64_t)(blk)) << N_ARRAY_IDX_BITS) | 1)
int sched_init_config(xce_scheduler_t* s, xce_conf4sched_t* c)
{
(void)s;
@ -34,6 +50,16 @@ int sched_init_config(xce_scheduler_t* s, xce_conf4sched_t* c)
return 0;
}
/* bug-1846: handler.c's config-push path now stashes via sched_stash_config()
* (scheduler.c, not linked here) instead of calling sched_init_config() --
* link-satisfying stub, same rationale as the one above. */
int sched_stash_config(xce_scheduler_t* s, xce_conf4sched_t* c)
{
(void)s;
(void)c;
return 0;
}
/* sched_record_status (recording.c, scheduler-core spec, task 9.4) --
* handler.c's OPCode_schedAdmin/recordStatus path references it; this file
* never exercises OPCode_schedAdmin, so a link-satisfying stub is enough,
@ -106,13 +132,6 @@ static void fixture_init(xce_scheduler_t* sched)
sched->core.jobs = makeInstance(&xkvmgr_type, &jobs_params);
}
static void seed_existing_job(xce_scheduler_t* sched, uint64_t jobid)
{
xce_sched_job_t* job = (xce_sched_job_t*)calloc(1, sizeof(*job));
job->jobid = jobid;
sched_add_job(&(sched->core), job);
}
/* Builds a heap-allocated, heap-owned-strings (jobArray, submit) pair -- the
* same shape decode_JobSubmit_ptr() produces on the real wire-decode path
* (controller.capnp.c), not a stack/string-literal stand-in. Task 7.3 made
@ -140,114 +159,63 @@ static xce_job_submit_t* init_job_array(uint64_t base_jobid,
return ja;
}
/* Pop one pending update request, asserting it is an UPDATE_JOB op; hands
* back njobs/jobs (caller now owns them -- free via free_Job_ptr + free() to
* mirror sched_pend_update_killer). Returns -1 if the queue is empty. */
static int pop_job_update(xce_scheduler_t* sched, xce_job_t*** jobs_out)
/* Pop one pending update request, asserting it is an UPDATE_JOB_ARRAY op;
* hands back the raw job_array descriptor (caller now owns it -- free via
* free_JobSubmit_ptr to mirror release_translated_payload()'s own cleanup on
* the failure/never-translated path). Returns NULL if the queue is empty. */
static xce_job_submit_t* pop_job_array_update(xce_scheduler_t* sched)
{
data_item_t* it = dataq_deq(sched->core.pend_updates[0], NULL);
xce_sched_update_request_t* req;
int njobs;
xce_job_submit_t* job_array;
if (it == NULL) {
return -1;
return NULL;
}
req = ITPTR_TO_UPDATE_REQ(it);
TEST_ASSERT_EQUAL_INT(XCE_SCHED_OP_UPDATE_JOB, req->op);
TEST_ASSERT_EQUAL_INT(XCE_SCHED_OP_UPDATE_JOB_ARRAY, req->op);
njobs = req->u.jobupdate.njobs;
if (jobs_out != NULL) {
*jobs_out = req->u.jobupdate.jobs;
}
job_array = req->u.jobarrayupdate.job_array;
TEST_ASSERT_EQUAL_INT(0, req->u.jobarrayupdate.expanded_njobs);
TEST_ASSERT_NULL(req->u.jobarrayupdate.expanded_jobs);
free(req);
return njobs;
return job_array;
}
/* ── Req 6.3/6.4: a compact Data.jobArray push expands locally into N
* ordinary schedulable tasks via base_jobid + i arithmetic ── */
/* ── Connection-loss-under-burst fix: a compact Data.jobArray push is now
* queued UNEXPANDED -- ownership of the raw descriptor transfers to the
* XCE_SCHED_OP_UPDATE_JOB_ARRAY request as-is, no per-element work happens
* on this (request-handling) thread. Expansion into N tasks is
* test_driver_translate.c's concern now (translate_one_pending_update()). ── */
static void test_job_array_push_expands_into_n_tasks(void)
static void test_job_array_push_enqueues_raw_descriptor_unexpanded(void)
{
xce_scheduler_t sched;
xce_job_submit_t* ja;
dataset_t dataset;
xce_job_t** jobs = NULL;
int njobs;
int i;
xce_job_submit_t* popped;
fixture_init(&sched);
ja = init_job_array(5000, 3);
ja = init_job_array(ARR_FIRST(1), 3);
memset(&dataset, 0, sizeof(dataset));
dataset.table = XceData_data_jobArray;
dataset.rows = (void*)ja;
/* sched_handle_dataset() owns and frees dataset.rows on return (task
* 7.3) -- do not touch ja/dataset.rows again after this call. */
TEST_ASSERT_TRUE(sched_handle_dataset(&sched, &dataset) >= 0);
njobs = pop_job_update(&sched, &jobs);
TEST_ASSERT_EQUAL_INT(3, njobs);
TEST_ASSERT_EQUAL_UINT64(5000, jobs[0]->jobid);
TEST_ASSERT_EQUAL_UINT64(5001, jobs[1]->jobid);
TEST_ASSERT_EQUAL_UINT64(5002, jobs[2]->jobid);
TEST_ASSERT_EQUAL_STRING("normal", jobs[0]->submit->queue);
TEST_ASSERT_EQUAL_STRING("normal", jobs[2]->submit->queue);
/* every element gets its OWN independent clone, not a shared pointer --
* required so each is independently free_Job_ptr()-able downstream
* (sched_pend_update_killer) without a double-free. */
TEST_ASSERT_TRUE(jobs[0]->submit != jobs[1]->submit);
TEST_ASSERT_TRUE(jobs[1]->submit != jobs[2]->submit);
/* Ownership transferred to the queued request -- sched_handle_dataset()
* must not have freed it and must have nulled dataset.rows out. */
TEST_ASSERT_NULL(dataset.rows);
for (i = 0; i < njobs; i++) {
free_Job_ptr(&jobs[i]);
}
free(jobs);
}
popped = pop_job_array_update(&sched);
TEST_ASSERT_EQUAL_PTR(ja, popped); /* same descriptor, not a clone */
TEST_ASSERT_EQUAL_UINT64(ARR_FIRST(1), popped->jobid);
TEST_ASSERT_EQUAL_UINT64(3, popped->array_size);
TEST_ASSERT_EQUAL_STRING("normal", popped->submit->queue);
/* ── Req 6.5/8.4: a redelivered push (e.g. after a scheduler reconnect)
* must not create duplicate tasks -- reuse the jobid-keyed exists-check
* (sched_get_job) the ordinary single-job path relies on ── */
static void test_job_array_redelivery_does_not_duplicate(void)
{
xce_scheduler_t sched;
dataset_t dataset;
xce_job_t** jobs = NULL;
int njobs;
int i;
fixture_init(&sched);
memset(&dataset, 0, sizeof(dataset));
dataset.table = XceData_data_jobArray;
dataset.rows = (void*)init_job_array(6000, 2);
/* first delivery: 2 new tasks enqueued. sched_handle_dataset() owns and
* frees dataset.rows on return (task 7.3). */
TEST_ASSERT_TRUE(sched_handle_dataset(&sched, &dataset) >= 0);
njobs = pop_job_update(&sched, &jobs);
TEST_ASSERT_EQUAL_INT(2, njobs);
for (i = 0; i < njobs; i++) {
/* simulate the async consumer (scheduler.c's sched_update_jobs ->
* sched_add_job, not linked into this white-box TU) materializing
* the element into sched->jobs, so the redelivery's dedup check
* sees them as already existing -- exactly what a real reconnect
* scenario looks like from the handler's point of view. */
seed_existing_job(&sched, jobs[i]->jobid);
free_Job_ptr(&jobs[i]);
}
free(jobs);
/* redelivery of the IDENTICAL push: a genuinely new decode of the
* retransmitted wire message (same jobid/array_size/submit content, but
* its own freshly-allocated descriptor -- exactly what a real reconnect
* redelivers; the first descriptor is already freed by the call above).
* Every index already exists. */
dataset.rows = (void*)init_job_array(6000, 2);
TEST_ASSERT_EQUAL_INT(0, sched_handle_dataset(&sched, &dataset));
TEST_ASSERT_EQUAL_INT(-1, pop_job_update(&sched, NULL)); /* nothing new */
free_JobSubmit_ptr(&popped);
}
/* ── Guard-clause coverage on the decode step itself ── */
@ -260,7 +228,7 @@ static void test_job_array_null_submit_returns_einval(void)
fixture_init(&sched);
memset(&ja, 0, sizeof(ja));
ja.jobid = 1;
ja.jobid = ARR_FIRST(2);
ja.array_size = 3;
ja.submit = NULL;
@ -291,7 +259,8 @@ static void test_job_array_null_descriptor_returns_einval(void)
TEST_ASSERT_EQUAL_INT(-EINVAL, sched_on_job_array(&sched, NULL));
}
/* ── job-array-single-wal-record task 5.3b: full wire-dispatch chain ──────
/* ── job-array-single-wal-record task 5.3b: full wire-dispatch chain
* (REVISED for the connection-loss-under-burst fix) ──────────────────────
* Every test above feeds a stack-built xce_job_submit_t straight into
* sched_handle_dataset(), bypassing the transport layer entirely -- the same
* white-box convention every sibling handler.c test uses (task 5.3's own
@ -302,10 +271,11 @@ static void test_job_array_null_descriptor_returns_einval(void)
* sends -- OPCode_push / Request_body_pushmsg carrying one Data.jobArray
* descriptor -- then hands extract_data()'s reassembled dataset_t to
* sched_handle_dataset(), proving the sender-payload -> wire-dispatch ->
* receiver-expand chain is intact end to end, not just sched_on_job_array()
* in isolation. */
* receiver-enqueue chain delivers the descriptor's fields correctly, intact
* end to end -- expansion itself is no longer this chain's job (driver.c's
* concern now), so this only proves the queued descriptor. */
static void test_wire_dispatch_chain_delivers_n_tasks(void)
static void test_wire_dispatch_chain_delivers_job_array_descriptor(void)
{
xce_scheduler_t sched;
job_submit_data_t submit;
@ -315,9 +285,7 @@ static void test_wire_dispatch_chain_delivers_n_tasks(void)
xce_request_t decoded;
io_packet_t* pkt;
dataset_t dataset;
xce_job_t** jobs = NULL;
int njobs;
int i;
xce_job_submit_t* popped;
fixture_init(&sched);
@ -330,7 +298,7 @@ static void test_wire_dispatch_chain_delivers_n_tasks(void)
submit.resreq = "type == local";
memset(&job_array, 0, sizeof(job_array));
job_array.jobid = 8000;
job_array.jobid = ARR_FIRST(3);
job_array.array_size = 4;
job_array.submit = &submit;
@ -362,27 +330,22 @@ static void test_wire_dispatch_chain_delivers_n_tasks(void)
* handler (task 5.3, unmodified) -- proving the two layers compose. */
TEST_ASSERT_TRUE(sched_handle_dataset(&sched, &dataset) >= 0);
/* Assert: N schedulable tasks, correctly addressed via base_jobid + i. */
njobs = pop_job_update(&sched, &jobs);
TEST_ASSERT_EQUAL_INT(4, njobs);
TEST_ASSERT_EQUAL_UINT64(8000, jobs[0]->jobid);
TEST_ASSERT_EQUAL_UINT64(8001, jobs[1]->jobid);
TEST_ASSERT_EQUAL_UINT64(8002, jobs[2]->jobid);
TEST_ASSERT_EQUAL_UINT64(8003, jobs[3]->jobid);
TEST_ASSERT_EQUAL_STRING("normal", jobs[0]->submit->queue);
TEST_ASSERT_EQUAL_STRING("normal", jobs[3]->submit->queue);
for (i = 0; i < njobs; i++) {
free_Job_ptr(&jobs[i]);
}
free(jobs);
/* Assert: the decoded descriptor was queued unexpanded, fields intact,
* correctly addressed via base_jobid + array_size. */
popped = pop_job_array_update(&sched);
TEST_ASSERT_NOT_NULL(popped);
TEST_ASSERT_EQUAL_UINT64(ARR_FIRST(3), popped->jobid);
TEST_ASSERT_EQUAL_UINT64(4, popped->array_size);
TEST_ASSERT_EQUAL_STRING("normal", popped->submit->queue);
free_JobSubmit_ptr(&popped);
/* Cleanup: extract_data() transferred ownership of the decoded
* xce_job_submit_t into dataset.rows (nulling the source pointer inside
* decoded.body.pushmsg); sched_handle_dataset()'s jobArray case now
* frees dataset.rows itself and nulls it out (task 7.3), so there is
* nothing left for this test to free there. decoded.body.pushmsg is a
* hollow container, safe to free via the normal request teardown. */
* decoded.body.pushmsg); sched_handle_dataset()'s jobArray case
* transfers that ownership again, into the queued request, and nulls
* dataset.rows out -- nothing left for this test to free there.
* decoded.body.pushmsg is a hollow container, safe to free via the
* normal request teardown. */
TEST_ASSERT_NULL(dataset.rows);
xce_free_request(&decoded);
}
@ -392,11 +355,10 @@ int main(int argc, char* argv[])
(void)argc;
(void)argv;
UNITY_BEGIN();
RUN_TEST(test_job_array_push_expands_into_n_tasks, 0);
RUN_TEST(test_job_array_redelivery_does_not_duplicate, 0);
RUN_TEST(test_job_array_push_enqueues_raw_descriptor_unexpanded, 0);
RUN_TEST(test_job_array_null_submit_returns_einval, 0);
RUN_TEST(test_job_array_zero_size_returns_einval, 0);
RUN_TEST(test_job_array_null_descriptor_returns_einval, 0);
RUN_TEST(test_wire_dispatch_chain_delivers_n_tasks, 0);
RUN_TEST(test_wire_dispatch_chain_delivers_job_array_descriptor, 0);
return UNITY_END();
}