fix(worker-rejoin-resync): task 7.5 - bound the outbound reconciliation query
Closes bug-1755: worker_send_resync_query() hardcoded pktlimit=0 in its xce_job_state_query_req() call, so the controller's outbound OPCode_jobStateQuery always sent every non-terminal job ID in one unbounded page regardless of count - the query direction never actually exercised the pagination mechanism task 1.4/7.4 proved correct, unlike the report direction's real, wired XCECD_RESYNC_REPORT_PKTLIMIT bound. Add XCE_RESYNC_QUERY_PKTLIMIT=100, mirroring that exact precedent (same value - a JobStateQuery entry is a bare jobid, smaller than a JobStateReport entry), and thread it into the one call site. Existing small-job-count behavior is unchanged; a rejoin exceeding the cap now genuinely produces multiple wire pages, proven against the real production call site (not a bypass) with a 250-job/3-page test. This is the last task in worker-rejoin-resync's implementation plan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
parent
613b80e991
commit
1c4cf8bd3b
4 changed files with 194 additions and 5 deletions
|
|
@ -153,7 +153,7 @@
|
|||
- _Requirements: 8.1, 8.2, 8.3_
|
||||
- _Depends: 7.1_
|
||||
|
||||
- [ ] 7.5 Wire a real page-size bound into the controller's outbound reconciliation query
|
||||
- [x] 7.5 Wire a real page-size bound into the controller's outbound reconciliation query
|
||||
- Close bug-1755, found and correctly left unfixed by task 7.4's own test-only boundary: `worker_send_resync_query()` (`src/controller/worker.c`) hardcodes `pktlimit=0` in its `xce_job_state_query_req()` call, which task 7.4 independently confirmed means "send every non-terminal job ID in one unbounded page" — so the query direction never actually paginates in production today, even though task 1.4's underlying pagination mechanism (proven correct by task 7.4's own test) fully supports it, and design.md's Boundary Commitments explicitly states both opcodes are paginated
|
||||
- Give the query send site a real, bounded page size, mirroring the worker's own already-wired `XCECD_RESYNC_REPORT_PKTLIMIT` precedent (`src/worker/src/controller_connector.c`) — same constant-driven approach, not a new configuration surface
|
||||
- Observable completion: a worker rejoin with a non-terminal job count exceeding the chosen page size produces a query genuinely sent across multiple wire pages (observably, not just theoretically supported), with the same job-count-independent completeness task 7.4 already proved at the mechanism level; a rejoin with a job count under one page still produces exactly one page, unchanged from today's observable behavior
|
||||
|
|
|
|||
|
|
@ -20589,7 +20589,7 @@
|
|||
"error_message": "Discovered while building worker-rejoin-resync task 7.4's large job-count pagination test (test_large_job_count_pagination_correctness, test_worker_resync.c): the controller's real production OPCode_jobStateQuery send site, worker_send_resync_query() (src/controller/worker.c:730), calls xce_job_state_query_req(ep, c->myhost, &query, NULL, NULL, 0) with a hardcoded pktlimit of 0. xce_job_state_query_req()'s own pktlimit<=0 convention (request.c:1822-1824, \"if (pktlimit <= 0) { pktlimit = nrows; }\") means this ALWAYS sends the entire query as exactly ONE wire page, however many non-terminal jobs the controller queries a worker about -- there is no bound at all. This is asymmetric with the worker's own real report-send site (controller_connector.c's handle_jobstatequery -> XCECD_RESYNC_REPORT_PKTLIMIT == 100), which DOES genuinely cap and paginate OPCode_jobStateReport in production. The query-direction pagination mechanism (xce_job_state_query_req()'s own do/while page loop, task 1.4) is fully implemented and correct -- task 7.4's own test calls it directly with an explicit page size and proves it genuinely spans many wire pages with no drops/duplicates -- but it is effectively dead code in production today because its only call site never supplies a bound.",
|
||||
"file": "src/controller/worker.c",
|
||||
"root_cause": "worker_send_resync_query() was written to pass pktlimit=0 to xce_job_state_query_req(), which the callee treats as \"unbounded / exactly one page\" rather than \"use a real per-page cap.\" No ClusterConfig field or fixed constant analogous to XCECD_RESYNC_REPORT_PKTLIMIT was ever wired in for the query direction, unlike the report direction. Requirement 8.1 ('regardless of how many non-terminal jobs... without... failing solely because of the number of jobs involved') is still technically satisfied -- an unbounded single message doesn't fail merely due to job count, and no hard message-size cap was found elsewhere in the transport/capnp layer that would make a very large single-page query actually fail or truncate. But design.md's own stated intent (both opcodes described as 'paginated'; the Testing Strategy's Performance/Load line about the exchange genuinely spanning multiple pages) is not actually exercised by any real caller for the query direction -- only the report direction is genuinely bounded/paginated in production as shipped.",
|
||||
"fix": "NOT FIXED -- out of scope for worker-rejoin-resync task 7.4 (a test-only pagination-correctness validation task, correctly declined to patch a production defect/gap it incidentally surfaced). Recommend a dedicated future task: either wire a real config-driven or fixed page cap into worker_send_resync_query()'s xce_job_state_query_req() call (mirroring XCECD_RESYNC_REPORT_PKTLIMIT, or reusing one of the two existing ClusterConfig fields this feature already added -- workerResyncWindowMs/workerResyncTimeoutMs are not a natural fit, so a new field would likely be needed), or explicitly document that the query direction is deliberately left unbounded by design (e.g. because a worker-side job count large enough to matter is not expected in practice). Until then, a worker rejoin with a very large non-terminal job count will always receive its OPCode_jobStateQuery as one very large single message, not multiple bounded pages, even though the receiving/decoding machinery on both sides already correctly supports true multi-page queries.",
|
||||
"fix": "FIXED (2026-09-01, worker-rejoin-resync task 7.5). worker_send_resync_query() (src/controller/worker.c:730, now ~745) no longer hardcodes pktlimit=0 in its xce_job_state_query_req() call -- it now passes a new, named page-size cap, XCE_RESYNC_QUERY_PKTLIMIT (worker.c, value 100), mirroring the worker's own already-wired XCECD_RESYNC_REPORT_PKTLIMIT (controller_connector.c, also 100) for the report direction. Same value used for both directions: a JobStateQuery entry (bare jobid) is smaller on the wire than a JobStateReport entry (jobid + status + exitCode + finishTimeMs + queried), so the query direction safely supports at least the same page size, with no reason found to invent a smaller one. This closes the query-direction gap task 7.4 found and correctly declined to fix (a test-only task) -- the query direction now genuinely, observably paginates in production, not just at the xce_job_state_query_req() mechanism level task 7.4's own test proved. Verified with a new RED->GREEN test exercising the REAL production call site directly (not a xce_job_state_query_req() bypass): test_worker_send_resync_query_paginates_across_multiple_wire_pages (test_worker_join.c) arms a reconciliation round over 250 non-terminal jobs (> the 100-job page cap, not a multiple of it) and asserts worker_send_resync_query() itself splits the query into exactly 3 wire pages (100+100+50), each with correct pos/count/n_total, every jobid appearing exactly once, no gaps or overlaps. Confirmed genuinely RED beforehand by reverting the fix and rerunning: failed with 'no single page may exceed the configured page cap' (a single collapsed page, matching the pre-fix hardcoded-0 behavior). Confirmed the fix does not regress small-job-count behavior: both existing task 6.1 tests (test_worker_send_resync_query_sends_armed_round, 2 jobs; test_worker_send_resync_query_sends_nothing_when_no_round, 0 jobs) and task 7.4's own test_large_job_count_pagination_correctness (which exercises xce_job_state_query_req() directly with an explicit page-size override, still bypassing the wrapper on purpose -- kept unmodified, it targets the mechanism, not the wrapper) all still pass unchanged. Full controller test suite (meson test -C src/xce.build 'controller test'): 6/6 OK, zero regressions.",
|
||||
"tags": [
|
||||
"worker-rejoin-resync",
|
||||
"task-7.4",
|
||||
|
|
@ -20600,8 +20600,9 @@
|
|||
"requirement-8.3",
|
||||
"pagination",
|
||||
"design-reality-mismatch",
|
||||
"discovered-not-fixed",
|
||||
"asymmetric-with-report-direction"
|
||||
"asymmetric-with-report-direction",
|
||||
"task-7.5",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1746",
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@
|
|||
#include <rtti.h>
|
||||
|
||||
#include <errno.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
|
@ -1176,6 +1177,175 @@ static void test_worker_send_resync_query_sends_nothing_when_no_round(void)
|
|||
"PASS\n");
|
||||
}
|
||||
|
||||
/* ── Task 7.5: worker_send_resync_query genuinely paginates (bug-1755) ──
|
||||
*
|
||||
* Prior to task 7.5, worker_send_resync_query() hardcoded pktlimit=0 in its
|
||||
* xce_job_state_query_req() call -- xce_job_state_query_req()'s own
|
||||
* pktlimit<=0 convention (request.c) means "pktlimit = nrows", i.e. always
|
||||
* exactly ONE wire page, however many jobs are queried. Task 7.4's own test
|
||||
* (test_worker_resync.c, test_large_job_count_pagination_correctness) only
|
||||
* proved the underlying xce_job_state_query_req() pagination MECHANISM by
|
||||
* calling it directly with an explicit page-size override, bypassing this
|
||||
* production call site entirely (documented live as bug-1755: "effectively
|
||||
* dead code in production today because its only call site never supplies
|
||||
* a bound").
|
||||
*
|
||||
* This test drives the REAL production call site -- worker_send_resync_
|
||||
* query() itself, not a direct xce_job_state_query_req() call -- with a
|
||||
* non-terminal job count on the host that genuinely exceeds the new
|
||||
* XCE_RESYNC_QUERY_PKTLIMIT page cap (worker.c), and asserts the query is
|
||||
* observably split across multiple wire pages: more than one packet is
|
||||
* enqueued, each page's `pos` picks up exactly where the previous page's
|
||||
* `count` left off, every jobid appears in exactly one page, and every
|
||||
* page (including the final, deliberately partial one) carries the same
|
||||
* full n_total. Req 8.1-8.3.
|
||||
*/
|
||||
#define TWJ_RESYNC_QUERY_PKTLIMIT 100 /* mirrors production
|
||||
* XCE_RESYNC_QUERY_PKTLIMIT
|
||||
* (worker.c) -- not a real
|
||||
* production symbol reference (that
|
||||
* #define has file-local linkage),
|
||||
* kept in sync by this test's own
|
||||
* page-count assertion below, which
|
||||
* would fail loudly if the two ever
|
||||
* drifted apart */
|
||||
#define TWJ_RESYNC_QUERY_N_JOBS 250 /* > TWJ_RESYNC_QUERY_PKTLIMIT, and not
|
||||
* a multiple of it, so the final wire
|
||||
* page is genuinely partial (50
|
||||
* entries), not just N full pages */
|
||||
#define TWJ_RESYNC_QUERY_JOBID_BASE 970000ULL
|
||||
#define TWJ_RESYNC_QUERY_N_PAGES \
|
||||
((TWJ_RESYNC_QUERY_N_JOBS + TWJ_RESYNC_QUERY_PKTLIMIT - 1) \
|
||||
/ TWJ_RESYNC_QUERY_PKTLIMIT)
|
||||
|
||||
static void
|
||||
test_worker_send_resync_query_paginates_across_multiple_wire_pages(void)
|
||||
{
|
||||
setup_test_cluster();
|
||||
|
||||
hostattrs_t attrs;
|
||||
memset(&attrs, 0, sizeof(attrs));
|
||||
attrs.hosttype = makeString("LINUX");
|
||||
attrs.hostmodel = makeString("x86");
|
||||
attrs.cpufactor = 1.0f;
|
||||
attrs.maxcpus = 4;
|
||||
attrs.maxmem = 16;
|
||||
attrs.maxswap = 32;
|
||||
attrs.maxtmp = 64;
|
||||
attrs.ndisks = 1;
|
||||
|
||||
worker_t* w = make_test_worker("query-host-paginate", &attrs);
|
||||
TEST_ASSERT_NOT_NULL(w);
|
||||
|
||||
for (int i = 0; i < TWJ_RESYNC_QUERY_N_JOBS; i++) {
|
||||
uint64_t jobid = TWJ_RESYNC_QUERY_JOBID_BASE + (uint64_t)i;
|
||||
job_t* j = seed_test_job(jobid, JOB_STAT_RUN);
|
||||
TEST_ASSERT_NOT_NULL(j);
|
||||
j->job.allocs = make_test_job_alloc(jobid, "query-host-paginate");
|
||||
TEST_ASSERT_NOT_NULL(j->job.allocs);
|
||||
}
|
||||
|
||||
hostattrs_t new_attrs = attrs;
|
||||
new_attrs.hosttype = makeString("LINUX");
|
||||
new_attrs.hostmodel = makeString("x86");
|
||||
|
||||
xce_hostconfig_t hconfig;
|
||||
memset(&hconfig, 0, sizeof(hconfig));
|
||||
hconfig.hosttype = (char*)"!";
|
||||
hconfig.hostmodel = (char*)"!";
|
||||
hconfig.mxj = 4;
|
||||
|
||||
endpoint_t stack_ep;
|
||||
memset(&stack_ep, 0, sizeof(stack_ep));
|
||||
rejoin_worker(&g_tc.c, w, &stack_ep, &new_attrs, &hconfig);
|
||||
TEST_ASSERT_NOT_NULL(w->resync);
|
||||
TEST_ASSERT_EQUAL_INT(TWJ_RESYNC_QUERY_N_JOBS, w->resync->n_queried_jobids);
|
||||
|
||||
endpoint_t* ep = new_test_endpoint();
|
||||
|
||||
/* The REAL production call site -- no xce_job_state_query_req()
|
||||
* bypass. */
|
||||
TEST_ASSERT_EQUAL_INT(0, worker_send_resync_query(&g_tc.c, w, ep));
|
||||
|
||||
{
|
||||
bool seen[TWJ_RESYNC_QUERY_N_JOBS];
|
||||
int running_total = 0;
|
||||
int pages = 0;
|
||||
|
||||
memset(seen, 0, sizeof(seen));
|
||||
|
||||
while (running_total < TWJ_RESYNC_QUERY_N_JOBS) {
|
||||
xce_request_t decoded;
|
||||
int j;
|
||||
|
||||
pop_and_decode(ep, &decoded);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(OPCode_jobStateQuery, decoded.opcode);
|
||||
TEST_ASSERT_EQUAL_INT(Request_body_jobstatequerymsg,
|
||||
decoded.body_which);
|
||||
TEST_ASSERT_NOT_NULL(decoded.body.jobstatequerymsg);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(TWJ_RESYNC_QUERY_N_JOBS,
|
||||
decoded.body.jobstatequerymsg->n_total,
|
||||
"every page must carry the same full n_total, not a "
|
||||
"per-page total (Req 2.1)");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(running_total,
|
||||
decoded.body.jobstatequerymsg->pos,
|
||||
"each page's pos must pick up exactly where the previous "
|
||||
"page's count left off -- no gap, no overlap (Req 8.3)");
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
decoded.body.jobstatequerymsg->count
|
||||
<= TWJ_RESYNC_QUERY_PKTLIMIT,
|
||||
"no single page may exceed the configured page cap "
|
||||
"(Req 8.3)");
|
||||
|
||||
for (j = 0; j < decoded.body.jobstatequerymsg->count; j++) {
|
||||
uint64_t jobid = decoded.body.jobstatequerymsg->jobids[j];
|
||||
uint64_t idx = jobid - TWJ_RESYNC_QUERY_JOBID_BASE;
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
idx < (uint64_t)TWJ_RESYNC_QUERY_N_JOBS,
|
||||
"a jobid appearing on the wire must be one of the "
|
||||
"armed round's own jobids");
|
||||
TEST_ASSERT_FALSE_MESSAGE(seen[idx],
|
||||
"no jobid may be sent twice across pages (Req 8.3)");
|
||||
seen[idx] = true;
|
||||
}
|
||||
|
||||
running_total += decoded.body.jobstatequerymsg->count;
|
||||
pages++;
|
||||
xce_free_request(&decoded);
|
||||
}
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(TWJ_RESYNC_QUERY_N_PAGES, pages,
|
||||
"Req 8.1-8.3: a job count exceeding the real production page "
|
||||
"cap must genuinely span multiple wire pages -- observably, "
|
||||
"via the REAL worker_send_resync_query() call site, not just "
|
||||
"theoretically supported by xce_job_state_query_req()'s own "
|
||||
"pagination loop (bug-1755, closed by task 7.5)");
|
||||
TEST_ASSERT_GREATER_THAN_INT_MESSAGE(1, pages,
|
||||
"a single collapsed page here would mean the hardcoded-0 "
|
||||
"regression (bug-1755) is back");
|
||||
|
||||
for (int i = 0; i < TWJ_RESYNC_QUERY_N_JOBS; i++) {
|
||||
TEST_ASSERT_TRUE_MESSAGE(seen[i],
|
||||
"every armed jobid must appear exactly once across all "
|
||||
"pages -- none silently dropped past a page boundary "
|
||||
"(Req 8.3)");
|
||||
}
|
||||
}
|
||||
|
||||
assert_queue_empty(ep);
|
||||
|
||||
dereference(new_attrs.hosttype);
|
||||
dereference(new_attrs.hostmodel);
|
||||
endpoint_fini(ep);
|
||||
free(ep);
|
||||
|
||||
teardown_test_cluster();
|
||||
printf("test_worker_send_resync_query_paginates_across_multiple_wire_"
|
||||
"pages: PASS\n");
|
||||
}
|
||||
|
||||
/* ── Task 6.1: config push, then query, in that fixed order ──────────── */
|
||||
|
||||
/*
|
||||
|
|
@ -1359,6 +1529,8 @@ void test_worker_join(void)
|
|||
RUN_TEST(test_init_worker_arms_reconciliation_round_when_jobs_exist);
|
||||
RUN_TEST(test_worker_send_resync_query_sends_armed_round);
|
||||
RUN_TEST(test_worker_send_resync_query_sends_nothing_when_no_round);
|
||||
RUN_TEST(
|
||||
test_worker_send_resync_query_paginates_across_multiple_wire_pages);
|
||||
RUN_TEST(test_config_push_then_resync_query_wire_order);
|
||||
RUN_TEST(test_worker_push_config_carries_configured_resync_window);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -663,6 +663,21 @@ static int worker_push_config(cluster_t* c, endpoint_t* ep)
|
|||
return rc;
|
||||
}
|
||||
|
||||
/*
|
||||
* worker-rejoin-resync task 7.5 (bug-1755, Req 8.1-8.3): page-size cap for
|
||||
* this controller's own OPCode_jobStateQuery sends, mirroring the worker's
|
||||
* own already-wired XCECD_RESYNC_REPORT_PKTLIMIT (controller_connector.c)
|
||||
* for the report direction -- same constant-driven approach, applied here
|
||||
* to the query direction so xce_job_state_query_req()'s already-implemented
|
||||
* pagination loop (task 1.4) is genuinely exercised in production instead
|
||||
* of always collapsing to a single unbounded page (bug-1755's finding). A
|
||||
* JobStateQuery entry (bare jobid) is smaller on the wire than a
|
||||
* JobStateReport entry (jobid + status + exitCode + finishTimeMs +
|
||||
* queried), so the query direction can safely use the same page size the
|
||||
* report direction already proves safe, without inventing a smaller one.
|
||||
*/
|
||||
#define XCE_RESYNC_QUERY_PKTLIMIT 100
|
||||
|
||||
/*
|
||||
* worker_send_resync_query — worker-rejoin-resync task 6.1 (Req 1.1-1.3,
|
||||
* 2.1, 2.2): sends the round's job-state query to a worker for which
|
||||
|
|
@ -727,7 +742,8 @@ static int worker_send_resync_query(cluster_t* c, worker_t* w, endpoint_t* ep)
|
|||
query.n_total = n;
|
||||
query.jobids = ids;
|
||||
|
||||
xce_job_state_query_req(ep, c->myhost, &query, NULL, NULL, 0);
|
||||
xce_job_state_query_req(ep, c->myhost, &query, NULL, NULL,
|
||||
XCE_RESYNC_QUERY_PKTLIMIT);
|
||||
|
||||
free(ids);
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue