fix(worker-rejoin-resync): task 7.5 - bound the outbound reconciliation query
Closes bug-1755: worker_send_resync_query() hardcoded pktlimit=0 in its xce_job_state_query_req() call, so the controller's outbound OPCode_jobStateQuery always sent every non-terminal job ID in one unbounded page regardless of count - the query direction never actually exercised the pagination mechanism task 1.4/7.4 proved correct, unlike the report direction's real, wired XCECD_RESYNC_REPORT_PKTLIMIT bound. Add XCE_RESYNC_QUERY_PKTLIMIT=100, mirroring that exact precedent (same value - a JobStateQuery entry is a bare jobid, smaller than a JobStateReport entry), and thread it into the one call site. Existing small-job-count behavior is unchanged; a rejoin exceeding the cap now genuinely produces multiple wire pages, proven against the real production call site (not a bypass) with a 250-job/3-page test. This is the last task in worker-rejoin-resync's implementation plan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
parent
613b80e991
commit
1c4cf8bd3b
4 changed files with 194 additions and 5 deletions
|
|
@ -20589,7 +20589,7 @@
|
|||
"error_message": "Discovered while building worker-rejoin-resync task 7.4's large job-count pagination test (test_large_job_count_pagination_correctness, test_worker_resync.c): the controller's real production OPCode_jobStateQuery send site, worker_send_resync_query() (src/controller/worker.c:730), calls xce_job_state_query_req(ep, c->myhost, &query, NULL, NULL, 0) with a hardcoded pktlimit of 0. xce_job_state_query_req()'s own pktlimit<=0 convention (request.c:1822-1824, \"if (pktlimit <= 0) { pktlimit = nrows; }\") means this ALWAYS sends the entire query as exactly ONE wire page, however many non-terminal jobs the controller queries a worker about -- there is no bound at all. This is asymmetric with the worker's own real report-send site (controller_connector.c's handle_jobstatequery -> XCECD_RESYNC_REPORT_PKTLIMIT == 100), which DOES genuinely cap and paginate OPCode_jobStateReport in production. The query-direction pagination mechanism (xce_job_state_query_req()'s own do/while page loop, task 1.4) is fully implemented and correct -- task 7.4's own test calls it directly with an explicit page size and proves it genuinely spans many wire pages with no drops/duplicates -- but it is effectively dead code in production today because its only call site never supplies a bound.",
|
||||
"file": "src/controller/worker.c",
|
||||
"root_cause": "worker_send_resync_query() was written to pass pktlimit=0 to xce_job_state_query_req(), which the callee treats as \"unbounded / exactly one page\" rather than \"use a real per-page cap.\" No ClusterConfig field or fixed constant analogous to XCECD_RESYNC_REPORT_PKTLIMIT was ever wired in for the query direction, unlike the report direction. Requirement 8.1 ('regardless of how many non-terminal jobs... without... failing solely because of the number of jobs involved') is still technically satisfied -- an unbounded single message doesn't fail merely due to job count, and no hard message-size cap was found elsewhere in the transport/capnp layer that would make a very large single-page query actually fail or truncate. But design.md's own stated intent (both opcodes described as 'paginated'; the Testing Strategy's Performance/Load line about the exchange genuinely spanning multiple pages) is not actually exercised by any real caller for the query direction -- only the report direction is genuinely bounded/paginated in production as shipped.",
|
||||
"fix": "NOT FIXED -- out of scope for worker-rejoin-resync task 7.4 (a test-only pagination-correctness validation task, correctly declined to patch a production defect/gap it incidentally surfaced). Recommend a dedicated future task: either wire a real config-driven or fixed page cap into worker_send_resync_query()'s xce_job_state_query_req() call (mirroring XCECD_RESYNC_REPORT_PKTLIMIT, or reusing one of the two existing ClusterConfig fields this feature already added -- workerResyncWindowMs/workerResyncTimeoutMs are not a natural fit, so a new field would likely be needed), or explicitly document that the query direction is deliberately left unbounded by design (e.g. because a worker-side job count large enough to matter is not expected in practice). Until then, a worker rejoin with a very large non-terminal job count will always receive its OPCode_jobStateQuery as one very large single message, not multiple bounded pages, even though the receiving/decoding machinery on both sides already correctly supports true multi-page queries.",
|
||||
"fix": "FIXED (2026-09-01, worker-rejoin-resync task 7.5). worker_send_resync_query() (src/controller/worker.c:730, now ~745) no longer hardcodes pktlimit=0 in its xce_job_state_query_req() call -- it now passes a new, named page-size cap, XCE_RESYNC_QUERY_PKTLIMIT (worker.c, value 100), mirroring the worker's own already-wired XCECD_RESYNC_REPORT_PKTLIMIT (controller_connector.c, also 100) for the report direction. Same value used for both directions: a JobStateQuery entry (bare jobid) is smaller on the wire than a JobStateReport entry (jobid + status + exitCode + finishTimeMs + queried), so the query direction safely supports at least the same page size, with no reason found to invent a smaller one. This closes the query-direction gap task 7.4 found and correctly declined to fix (a test-only task) -- the query direction now genuinely, observably paginates in production, not just at the xce_job_state_query_req() mechanism level task 7.4's own test proved. Verified with a new RED->GREEN test exercising the REAL production call site directly (not a xce_job_state_query_req() bypass): test_worker_send_resync_query_paginates_across_multiple_wire_pages (test_worker_join.c) arms a reconciliation round over 250 non-terminal jobs (> the 100-job page cap, not a multiple of it) and asserts worker_send_resync_query() itself splits the query into exactly 3 wire pages (100+100+50), each with correct pos/count/n_total, every jobid appearing exactly once, no gaps or overlaps. Confirmed genuinely RED beforehand by reverting the fix and rerunning: failed with 'no single page may exceed the configured page cap' (a single collapsed page, matching the pre-fix hardcoded-0 behavior). Confirmed the fix does not regress small-job-count behavior: both existing task 6.1 tests (test_worker_send_resync_query_sends_armed_round, 2 jobs; test_worker_send_resync_query_sends_nothing_when_no_round, 0 jobs) and task 7.4's own test_large_job_count_pagination_correctness (which exercises xce_job_state_query_req() directly with an explicit page-size override, still bypassing the wrapper on purpose -- kept unmodified, it targets the mechanism, not the wrapper) all still pass unchanged. Full controller test suite (meson test -C src/xce.build 'controller test'): 6/6 OK, zero regressions.",
|
||||
"tags": [
|
||||
"worker-rejoin-resync",
|
||||
"task-7.4",
|
||||
|
|
@ -20600,8 +20600,9 @@
|
|||
"requirement-8.3",
|
||||
"pagination",
|
||||
"design-reality-mismatch",
|
||||
"discovered-not-fixed",
|
||||
"asymmetric-with-report-direction"
|
||||
"asymmetric-with-report-direction",
|
||||
"task-7.5",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1746",
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue