lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(scheduler): defer job-array expansion off the vtask thread pool

Browse source 
sched_on_job_array() expanded a compact array-job push (up to
hundreds of elements: a per-element dedup lookup plus a full deep
clone) inline, synchronously, on whatever vtaskmgr thread handled the
incoming request -- the same shared pool the scheduler's own
controller connection traffic depends on, violating ha-async-io.md's
"bounded handler work only" contract for that pool (a project-wide
design invariant). A prior finding already showed a burst of
6x50-element array pushes (300 job events landing within ~2ms of each
other) causing the scheduler to lose its controller connection within
245ms and enter a reconnect/failover loop -- directly consistent with
this mechanism, and the same reconnect storm whose resync races
motivated the companion JOBS_PUSH fix in this branch.

The request handler now does O(1) work only: validate the descriptor
and enqueue it unexpanded (XCE_SCHED_OP_UPDATE_JOB_ARRAY, mirroring
the existing XCE_SCHED_OP_ADMIN_RECORD deferred-dispatch pattern).
The actual per-element expansion and dedup check moved to
translate_one_pending_update(), which runs on the round-processing
thread (main.c's xce_sched_driver_run_normal_mode_round(), before
xce_core_begin_session() each round) -- a thread with no connection-
liveness responsibility. As a side effect this also closes the
dedup check's own pre-existing cross-thread TOCTOU race (previously
documented as bug-1714 follow-up "Bug B"): only one thread ever runs
it now.

test_job_array_expand.c and test_job_array_wire_e2e.c, which asserted
the old inline-expansion behavior, are updated to assert the new
deferred-enqueue behavior instead; expansion, dedup, and full
drain-push-release ownership coverage moved to new tests in
test_driver_translate.c.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
This commit is contained in:
Rongsong Shen 2026-09-15 09:01:47 +00:00
parent 6c201bb3e8
commit d5ec1eb3ae
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
10 changed files with 838 additions and 517 deletions
Download patch file Download diff file Expand all files Collapse all files

221
src/scheduler/testsuite/test_job_array_wire_e2e.c
View file

@ -16,12 +16,23 @@
* -> sched_handle_dataset()
* -> sched_on_job_array() (task 5.3)
*
* ...delivers exactly N correctly-addressed scheduler tasks and fires task
* 5.3's confirmation log line exactly once (Req 6.3, 6.4, 8.4); and that an
* identical byte-for-byte SECOND delivery (independently re-decoded from the
* same encoded bytes -- the wire-level equivalent of "the same push arrives
* twice", e.g. after a reconnect) creates zero new tasks and logs zero
* additional confirmation lines (Req 6.5, 8.4).
* ...delivers the compact descriptor into the pend-updates queue intact and
* unexpanded (Req 6.3, 6.4), correctly addressed by base jobid/array_size.
*
* REVISED (connection-loss-under-burst fix, see XCE_SCHED_OP_UPDATE_JOB_
* ARRAY's own comment, scheduler.h): sched_on_job_array() no longer expands
* the descriptor into N tasks, deduplicates a redelivery, or logs a
* confirmation line itself -- all three now happen in driver.c's
* translate_one_pending_update(), on the round-processing thread, covered by
* test_driver_translate.c's own UPDATE_JOB_ARRAY tests
* (test_translate_job_array_expands_into_n_events,
* test_translate_job_array_skips_already_existing_elements,
* test_translate_job_array_all_duplicates_returns_zero_noop). This file's
* own two tests were revised accordingly: a redelivery of byte-identical
* wire bytes through THIS layer alone is correctly and safely a second,
* independent, unexpanded descriptor -- the wire/handler layer has no
* dedup check of its own to lose; the actual dedup coverage lives in
* test_driver_translate.c now.
*
* Existing coverage this test does NOT duplicate:
* - test_job_array_sched_push.c (task 5.1) already proves the CONTROLLER
@ -29,14 +40,13 @@
* - test_request.c's test_extract_data_job_array_passes_through_intact
* (task 5.3b) already proves extract_data() itself decodes the payload
* correctly via a real encode/decode round trip.
* - test_job_array_expand.c's test_wire_dispatch_chain_delivers_n_tasks
* (task 5.3b) already proves extract_data() -> sched_handle_dataset()
* compose, but stops one layer below the real registered handler and
* never exercises redelivery through the wire path or the log line.
* - test_job_array_expand.c's test_wire_dispatch_chain_delivers_job_array_
* descriptor (task 5.3b) already proves extract_data() ->
* sched_handle_dataset() compose, but stops one layer below the real
* registered handler.
* This test's unique contribution: the REAL top-level dispatcher
* (scheduler_req_handler(), not sched_handle_dataset() directly), the
* redelivery-under-real-wire-bytes case, and the confirmation log line's
* actual emission count.
* (scheduler_req_handler(), not sched_handle_dataset() directly) and the
* redelivery-under-real-wire-bytes case.
*
* vtask_entry() (xcectx.c) is `static` and not exported for testing -- it is
* also the trigger normally driven by a live socket + iomgr epoll loop, which
@ -51,13 +61,11 @@
* Bug-catching demonstration (this task's "RED phase"): temporarily
* reverting extract_data()'s `case XceData_data_jobArray:` (task 5.3b's fix)
* back to falling through to `default: rc = -EINVAL` makes this test's first
* assertion (`callnext != 0` while dataset->rows stays NULL) fail loudly,
* and the subsequent job-count assertions fail too (zero tasks created) --
* assertion (`callnext != 0` while dataset->rows stays NULL) fail loudly --
* confirming this test has teeth against the exact class of bug 5.3b fixed.
*/
#define UNIT_TEST 1
#include <errno.h>
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@ -139,96 +147,28 @@ static void fixture_init(xce_scheduler_t* sched)
sched->core.jobs = makeInstance(&xkvmgr_type, &jobs_params);
}
/* Pop one pending update request, asserting it is an UPDATE_JOB op; hands
* back njobs/jobs (caller now owns them -- free via free_Job_ptr + free() to
* mirror sched_pend_update_killer). Returns -1 if the queue is empty. Mirrors
/* Pop one pending update request, asserting it is an UPDATE_JOB_ARRAY op;
* hands back the raw job_array descriptor (caller now owns it -- free via
* free_JobSubmit_ptr). Returns NULL if the queue is empty. Mirrors
* test_job_array_expand.c's identical helper. */
static int pop_job_update(xce_scheduler_t* sched, xce_job_t*** jobs_out)
static xce_job_submit_t* pop_job_array_update(xce_scheduler_t* sched)
{
data_item_t* it = dataq_deq(sched->core.pend_updates[0], NULL);
xce_sched_update_request_t* req;
int njobs;
xce_sched_update_request_t* req;
xce_job_submit_t* job_array;
if (it == NULL) {
return -1;
return NULL;
}
req = ITPTR_TO_UPDATE_REQ(it);
TEST_ASSERT_EQUAL_INT(XCE_SCHED_OP_UPDATE_JOB, req->op);
TEST_ASSERT_EQUAL_INT(XCE_SCHED_OP_UPDATE_JOB_ARRAY, req->op);
njobs = req->u.jobupdate.njobs;
if (jobs_out != NULL) {
*jobs_out = req->u.jobupdate.jobs;
}
job_array = req->u.jobarrayupdate.job_array;
free(req);
return njobs;
return job_array;
}
/* Materializes a popped job array into sched->jobs (mirroring what the
* async consumer, scheduler.c's sched_update_jobs -> sched_add_job, not
* linked into this white-box TU, would do) -- so a SECOND, redelivered
* push's dedup check sees the first delivery's elements as already
* existing. Frees each popped xce_job_t and the array itself, same
* simulated-reconnect technique test_job_array_expand.c's
* test_job_array_redelivery_does_not_duplicate uses (there via
* seed_existing_job, here folded into one step since njobs is already
* known). */
static void materialize_popped_jobs(xce_scheduler_t* sched, xce_job_t** jobs,
int njobs)
{
int i;
for (i = 0; i < njobs; i++) {
xce_sched_job_t* job = (xce_sched_job_t*)calloc(1, sizeof(*job));
job->jobid = jobs[i]->jobid;
sched_add_job(&(sched->core), job);
free_Job_ptr(&jobs[i]);
}
free(jobs);
}
/* ---------------------------------------------------------------------- */
/* Early-print log capture: sched_on_job_array()'s confirmation line uses */
/* INFO() (tlog_info -> TLOG_INFO). No test binary in this codebase calls */
/* tlog_init(), so every INFO() call routes through tlog's pre-init */
/* "early print" fallback (tlog.c's _tlog_early_print(), confirmed via the */
/* vendored tlog source). tlog_reg_early_printf_callback() lets a test */
/* intercept that fallback synchronously, on the calling thread -- no */
/* tlog_init()/tlog_exit() lifecycle, so it cannot disturb any other test */
/* sharing this process. va_copy() is used so the caller's own va_list is */
/* never consumed, in case tlog's own early-print path also reads it. */
/* ---------------------------------------------------------------------- */
static int g_confirm_log_count = 0;
static void wire_e2e_early_print_cb(struct tlog_loginfo* loginfo,
const char* format, va_list ap)
{
char buf[512];
va_list ap_copy;
if (loginfo == NULL || loginfo->level != TLOG_INFO) {
return;
}
va_copy(ap_copy, ap);
vsnprintf(buf, sizeof(buf), format, ap_copy);
va_end(ap_copy);
if (strstr(buf, "job array push received and expanded") != NULL) {
g_confirm_log_count++;
}
}
static void start_log_capture(void)
{
g_confirm_log_count = 0;
tlog_set_early_printf(1, 1, 0);
tlog_reg_early_printf_callback(wire_e2e_early_print_cb);
}
static void stop_log_capture(void) { tlog_reg_early_printf_callback(NULL); }
/* ---------------------------------------------------------------------- */
/* Real wire construction: the exact message xcesched_push_jobarray() */
/* (sched.c) sends -- OPCode_push / Request_body_pushmsg carrying one */
@ -328,52 +268,48 @@ static int deliver_via_real_dispatch_chain(io_packet_t* pkt,
return rc;
}
/* ── Test 1: one real wire push -> N correctly-addressed tasks, log fires */
/* exactly once (Req 6.3, 6.4, 8.4) */
/* ── Test 1: one real wire push -> the compact descriptor is queued intact */
/* and unexpanded (Req 6.3, 6.4) -- expansion itself is driver.c's concern */
/* now, covered by test_driver_translate.c. */
static void test_one_push_produces_n_tasks_and_logs_once(void)
static void test_one_push_enqueues_job_array_descriptor(void)
{
xce_scheduler_t sched;
job_submit_data_t submit_storage;
io_packet_t* pkt = NULL;
dataset_t* ds = NULL;
xce_job_t** jobs = NULL;
int njobs;
int i;
xce_job_submit_t* popped;
fixture_init(&sched);
start_log_capture();
build_encoded_push(&pkt, 42000, 6, &submit_storage);
TEST_ASSERT_TRUE(
deliver_via_real_dispatch_chain(pkt, &sched, &ds) >= 0);
njobs = pop_job_update(&sched, &jobs);
TEST_ASSERT_EQUAL_INT(6, njobs);
for (i = 0; i < njobs; i++) {
TEST_ASSERT_EQUAL_UINT64(42000 + (uint64_t)i, jobs[i]->jobid);
TEST_ASSERT_EQUAL_STRING("normal", jobs[i]->submit->queue);
free_Job_ptr(&jobs[i]);
}
free(jobs);
popped = pop_job_array_update(&sched);
TEST_ASSERT_NOT_NULL(popped);
TEST_ASSERT_EQUAL_UINT64(42000, popped->jobid);
TEST_ASSERT_EQUAL_UINT64(6, popped->array_size);
TEST_ASSERT_EQUAL_STRING("normal", popped->submit->queue);
free_JobSubmit_ptr(&popped);
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, g_confirm_log_count,
"task 5.3's confirmation log line must fire exactly once for one "
"successful array expansion");
stop_log_capture();
free_JobSubmit_ptr((xce_job_submit_t**)&ds->rows);
free(ds);
printf("test_one_push_produces_n_tasks_and_logs_once: PASS\n");
printf("test_one_push_enqueues_job_array_descriptor: PASS\n");
}
/* ── Test 2: redelivery of the IDENTICAL encoded bytes (independently */
/* re-decoded, simulating a reconnect replay) creates zero new tasks and */
/* zero additional log lines (Req 6.5, 8.4) */
/* re-decoded, simulating a reconnect replay) -- REVISED (connection-loss- */
/* under-burst fix): the wire/handler layer no longer deduplicates at all */
/* (see XCE_SCHED_OP_UPDATE_JOB_ARRAY's own comment, scheduler.h), so a */
/* redelivery through THIS layer alone correctly and safely produces a */
/* second, independent, unexpanded descriptor -- dedup coverage now lives */
/* in test_driver_translate.c's own */
/* test_translate_job_array_skips_already_existing_elements / */
/* test_translate_job_array_all_duplicates_returns_zero_noop (Req 6.5). */
static void test_redelivery_creates_no_new_tasks_and_logs_nothing(void)
static void test_redelivery_enqueues_independent_second_descriptor(void)
{
xce_scheduler_t sched;
job_submit_data_t submit_storage;
@ -381,8 +317,8 @@ static void test_redelivery_creates_no_new_tasks_and_logs_nothing(void)
io_packet_t* pkt2 = NULL;
dataset_t* ds1 = NULL;
dataset_t* ds2 = NULL;
xce_job_t** jobs = NULL;
int njobs;
xce_job_submit_t* popped1;
xce_job_submit_t* popped2;
fixture_init(&sched);
@ -392,34 +328,31 @@ static void test_redelivery_creates_no_new_tasks_and_logs_nothing(void)
build_encoded_push(&pkt1, 43000, 4, &submit_storage);
build_encoded_push(&pkt2, 43000, 4, &submit_storage);
start_log_capture();
/* First delivery: 4 new tasks. */
/* First delivery. */
TEST_ASSERT_TRUE(deliver_via_real_dispatch_chain(pkt1, &sched, &ds1) >= 0);
njobs = pop_job_update(&sched, &jobs);
TEST_ASSERT_EQUAL_INT(4, njobs);
materialize_popped_jobs(&sched, jobs, njobs);
TEST_ASSERT_EQUAL_INT(1, g_confirm_log_count);
popped1 = pop_job_array_update(&sched);
TEST_ASSERT_NOT_NULL(popped1);
TEST_ASSERT_EQUAL_UINT64(43000, popped1->jobid);
TEST_ASSERT_EQUAL_UINT64(4, popped1->array_size);
free_JobSubmit_ptr(&popped1);
free_JobSubmit_ptr((xce_job_submit_t**)&ds1->rows);
free(ds1);
/* Second, byte-identical delivery (simulated redelivery/reconnect):
* every index (43000..43003) already exists in sched->jobs -- must be a
* pure no-op, and must NOT log a second confirmation line. */
TEST_ASSERT_EQUAL_INT(
0, deliver_via_real_dispatch_chain(pkt2, &sched, &ds2));
TEST_ASSERT_EQUAL_INT_MESSAGE(
-1, pop_job_update(&sched, &jobs),
"redelivery must enqueue nothing new");
TEST_ASSERT_EQUAL_INT_MESSAGE(
1, g_confirm_log_count,
"redelivery must not add a second confirmation log line -- still "
"exactly 1 total, from the first delivery only");
/* Second, byte-identical delivery (simulated redelivery/reconnect): a
* second, independent descriptor -- correct and safe by design (the
* downstream dedup check, now in driver.c, runs against core->jobs,
* which this white-box TU never populates from a queued-but-not-yet-
* translated request). */
TEST_ASSERT_TRUE(deliver_via_real_dispatch_chain(pkt2, &sched, &ds2) >= 0);
popped2 = pop_job_array_update(&sched);
TEST_ASSERT_NOT_NULL(popped2);
TEST_ASSERT_EQUAL_UINT64(43000, popped2->jobid);
TEST_ASSERT_EQUAL_UINT64(4, popped2->array_size);
free_JobSubmit_ptr(&popped2);
stop_log_capture();
free_JobSubmit_ptr((xce_job_submit_t**)&ds2->rows);
free(ds2);
printf("test_redelivery_creates_no_new_tasks_and_logs_nothing: PASS\n");
printf("test_redelivery_enqueues_independent_second_descriptor: PASS\n");
}
int main(int argc, char* argv[])
@ -427,7 +360,7 @@ int main(int argc, char* argv[])
(void)argc;
(void)argv;
UNITY_BEGIN();
RUN_TEST(test_one_push_produces_n_tasks_and_logs_once, 0);
RUN_TEST(test_redelivery_creates_no_new_tasks_and_logs_nothing, 0);
RUN_TEST(test_one_push_enqueues_job_array_descriptor, 0);
RUN_TEST(test_redelivery_enqueues_independent_second_descriptor, 0);
return UNITY_END();
}