test(worker-rejoin-resync): task 7.4 - large job-count pagination correctness
Add test_large_job_count_pagination_correctness, proving both OPCode_jobStateQuery and OPCode_jobStateReport genuinely span many wire pages (1042 jobs, 11 pages each direction, including a partial final page) with zero jobs dropped/duplicated/truncated - exact-count assertions throughout (page continuity, no-duplicate bitmap, exact WAL delta, exact hook-fire count), extending task 6.2's 2-page proof by an order of magnitude. Test-only, no production changes. Logs bug-1755: the underlying query-pagination mechanism this test proves correct is never exercised in production - worker_send_resync_ query() hardcodes pktlimit=0, so the controller's real query send always produces one unbounded page regardless of job count, asymmetric with the report direction's real XCECD_RESYNC_REPORT_PKTLIMIT bound. No hard transport-layer size cap exists, so this isn't a demonstrated failure mode, but it contradicts design.md's explicit "both paginated" claim in this feature's own headline mechanism. Adds task 7.5 to close it with a small, well-scoped fix before the feature-level gate, rather than leaving a silently-false design claim in the completed feature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
parent
165fc1586a
commit
613b80e991
3 changed files with 715 additions and 1 deletions
|
|
@ -147,12 +147,19 @@
|
|||
- _Requirements: 8.1, 8.2, 8.3_
|
||||
- _Depends: 7.1_
|
||||
|
||||
- [ ] 7.4 Large job-count pagination correctness
|
||||
- [x] 7.4 Large job-count pagination correctness
|
||||
- Drive the reconciliation exchange for a worker running enough jobs to require multiple pages of both the query and the report, confirming the exchange completes correctly rather than truncating or silently dropping jobs past a single page
|
||||
- Observable completion: a job count an order of magnitude larger than a single page holds is fully queried and fully reported, with every job accounted for
|
||||
- _Requirements: 8.1, 8.2, 8.3_
|
||||
- _Depends: 7.1_
|
||||
|
||||
- [ ] 7.5 Wire a real page-size bound into the controller's outbound reconciliation query
|
||||
- Close bug-1755, found and correctly left unfixed by task 7.4's own test-only boundary: `worker_send_resync_query()` (`src/controller/worker.c`) hardcodes `pktlimit=0` in its `xce_job_state_query_req()` call, which task 7.4 independently confirmed means "send every non-terminal job ID in one unbounded page" — so the query direction never actually paginates in production today, even though task 1.4's underlying pagination mechanism (proven correct by task 7.4's own test) fully supports it, and design.md's Boundary Commitments explicitly states both opcodes are paginated
|
||||
- Give the query send site a real, bounded page size, mirroring the worker's own already-wired `XCECD_RESYNC_REPORT_PKTLIMIT` precedent (`src/worker/src/controller_connector.c`) — same constant-driven approach, not a new configuration surface
|
||||
- Observable completion: a worker rejoin with a non-terminal job count exceeding the chosen page size produces a query genuinely sent across multiple wire pages (observably, not just theoretically supported), with the same job-count-independent completeness task 7.4 already proved at the mechanism level; a rejoin with a job count under one page still produces exactly one page, unchanged from today's observable behavior
|
||||
- _Requirements: 8.1, 8.2, 8.3_
|
||||
- _Depends: 6.1, 7.4_
|
||||
|
||||
## Implementation Notes
|
||||
- Task 1.3: this task's premise (`src/worker` has no `testsuite/`) is FALSE as of implementation time — real, tracked worker test infrastructure already exists (`src/worker/testsuite/{meson.build,test_*.c}`, 5 passing tests, `suite: 'worker'` registered in meson, `/xce-build worker` already runs it), built by `component-failover-adoption` (commits `ce9f734a`/`331fa024`/`619b119e`, 2026-08-11) weeks before this spec was written. `.kiro/steering/structure.md`, `testing.md:55`, and `build-deployment.md:16` are all stale on this point — worth a `/kiro-steering` sync, out of this feature's boundary to fix. Confirmed directly (fresh build + `meson test -C src/xce.build --suite worker`: 5/5 OK) rather than trusting the finding secondhand. No files changed for this task — closed as satisfied by pre-existing infrastructure.
|
||||
- **Task 4.2 must follow the EXISTING worker testsuite convention**, not the controller aggregator-runner pattern the original task 1.3 text pointed at: each test file is its own standalone Unity executable (own `UNITY_BEGIN()`/`RUN_TEST`/`UNITY_END()`, white-box `#include`s the production `.c` file under test — see `test_resource_sync.c`'s shape) with its own explicit `executable()` + `test(..., suite: 'worker', ...)` block added to `src/worker/testsuite/meson.build`, mirroring `test_controller_connector.c`'s registration exactly. Do not introduce a second, competing test-runner shape.
|
||||
|
|
@ -172,3 +179,4 @@
|
|||
- **Task 7.1 (test-only, zero production changes) discovered and honestly asserted a genuine, pre-existing, systemic gap distinct from bug-1747**: a worker-reported (or any) job failure's exit code is never durably recorded anywhere in this codebase — `record_job_fail_fact()`'s `WAL_JOB_FAIL` payload has no exit-code slot at all, and `apply_job_fail_transition()` never touches `job.job.exit_info` for the FAIL path, so it stays `NULL` unconditionally. Bug-1747 only covers the FAIL finish-*time* mistiming; this is the exit-*code* gap, logged separately as bug-1754 (out of scope for this test-only task, correctly not fixed here — flagged for a dedicated future task). The round-trip test asserts `exit_info == NULL` honestly with an inline comment rather than encoding a wrong expectation the codebase can't actually produce.
|
||||
- Task 7.2 (test-only): added a focused, single-job proof of the feature's own original motivating scenario (bug-1741) — a job dispatched and RUN with NO finish fact ever committed (modeling the lost fire-and-forget notify), recovered to DONE with the worker-reported finish time via the real rejoin/query/report chain. Deliberately scoped smaller than task 7.1's four-outcome round trip (reuses its helper infrastructure, adds no new scaffolding) rather than duplicating it. tasks.md's own `_Requirements: 4.1, 4.2, 4.6_` tag includes 4.6 (unrecognized→PEND), which is genuinely unrelated to this finished-success scenario (that's task 7.1/2.1's territory) — `REQUIREMENTS_CHECKED` correctly narrowed to 4.1/4.2 instead of force-fitting an unrelated assertion, same accepted-imprecise-tag pattern as task 6.2's own citation nit.
|
||||
- Task 7.3 (test-only): reproduced bug-1741's exact original shape (7 jobs, 6 lost) in one round/one wire page — 5 finished-success + 1 finished-failure recovered, plus one already-DONE job re-reported as an unqueried extra with a deliberately wrong finish time, proving reconciliation neither leaves any lost job stuck nor duplicates/clobbers an already-terminal one (exact `+6` WAL LSN delta, not `>=`, so a duplicate append is genuinely caught — confirmed via RED-phase reproduction against `record_job_finish_fact_at`'s terminal-state guard). Deliberately kept to a single wire page (page-boundary correctness is task 7.4's separate concern). Two reviewer/implementer subagents in this session's task 6.2/7.2 review rounds hit the "background command with no external wakeup" trap (ended their turn on a backgrounded build, never resumed) — both implementer and reviewer prompts for 7.3 onward now carry an explicit foreground-only warning up front; worth keeping in every future subagent dispatch in this codebase, not just implementer roles.
|
||||
- **Task 7.4 (test-only) found and correctly declined to fix bug-1755, but the coordinator escalated it to a new task 7.5 rather than leaving it passively logged.** Proved 1042 jobs correctly query and report across 11 wire pages each direction (exact-count assertions throughout — page continuity, no-duplicate bitmap, exact WAL delta, exact hook-fire count), extending task 6.2's 2-page proof by an order of magnitude, per the task's own "no truncation past a single page" mandate. While building it, discovered the underlying `xce_job_state_query_req()` pagination mechanism it was proving correct is never actually exercised in production: `worker_send_resync_query()` (task 6.1) hardcodes `pktlimit=0`, meaning the controller's real query send ALWAYS produces exactly one unbounded page regardless of job count — asymmetric with the report direction's real, wired `XCECD_RESYNC_REPORT_PKTLIMIT=100` bound. No hard transport-layer message-size cap was found (dynamic malloc, `uint64_t` length prefix), so this isn't a demonstrated crash/hang/data-loss risk and Requirement 8.1's literal "without... failing" text is technically satisfied — but design.md's Boundary Commitments explicitly states both opcodes are "paginated," which is not true of the shipped code, in this feature's own headline mechanism, discovered in this feature's own final validation task. Per the review's explicit recommendation, added task 7.5 (small, well-scoped: mirror the report direction's constant-driven precedent) rather than accepting a silently-false design claim into the completed feature — same "found a real gap, fold it into an explicit task rather than a passive note" pattern this spec has followed since tasks 1.5/6.1's config-delivery gaps.
|
||||
|
|
|
|||
|
|
@ -20582,6 +20582,33 @@
|
|||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-01T13:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1755",
|
||||
"timestamp": "2026-09-02T00:30:00Z",
|
||||
"error_message": "Discovered while building worker-rejoin-resync task 7.4's large job-count pagination test (test_large_job_count_pagination_correctness, test_worker_resync.c): the controller's real production OPCode_jobStateQuery send site, worker_send_resync_query() (src/controller/worker.c:730), calls xce_job_state_query_req(ep, c->myhost, &query, NULL, NULL, 0) with a hardcoded pktlimit of 0. xce_job_state_query_req()'s own pktlimit<=0 convention (request.c:1822-1824, \"if (pktlimit <= 0) { pktlimit = nrows; }\") means this ALWAYS sends the entire query as exactly ONE wire page, however many non-terminal jobs the controller queries a worker about -- there is no bound at all. This is asymmetric with the worker's own real report-send site (controller_connector.c's handle_jobstatequery -> XCECD_RESYNC_REPORT_PKTLIMIT == 100), which DOES genuinely cap and paginate OPCode_jobStateReport in production. The query-direction pagination mechanism (xce_job_state_query_req()'s own do/while page loop, task 1.4) is fully implemented and correct -- task 7.4's own test calls it directly with an explicit page size and proves it genuinely spans many wire pages with no drops/duplicates -- but it is effectively dead code in production today because its only call site never supplies a bound.",
|
||||
"file": "src/controller/worker.c",
|
||||
"root_cause": "worker_send_resync_query() was written to pass pktlimit=0 to xce_job_state_query_req(), which the callee treats as \"unbounded / exactly one page\" rather than \"use a real per-page cap.\" No ClusterConfig field or fixed constant analogous to XCECD_RESYNC_REPORT_PKTLIMIT was ever wired in for the query direction, unlike the report direction. Requirement 8.1 ('regardless of how many non-terminal jobs... without... failing solely because of the number of jobs involved') is still technically satisfied -- an unbounded single message doesn't fail merely due to job count, and no hard message-size cap was found elsewhere in the transport/capnp layer that would make a very large single-page query actually fail or truncate. But design.md's own stated intent (both opcodes described as 'paginated'; the Testing Strategy's Performance/Load line about the exchange genuinely spanning multiple pages) is not actually exercised by any real caller for the query direction -- only the report direction is genuinely bounded/paginated in production as shipped.",
|
||||
"fix": "NOT FIXED -- out of scope for worker-rejoin-resync task 7.4 (a test-only pagination-correctness validation task, correctly declined to patch a production defect/gap it incidentally surfaced). Recommend a dedicated future task: either wire a real config-driven or fixed page cap into worker_send_resync_query()'s xce_job_state_query_req() call (mirroring XCECD_RESYNC_REPORT_PKTLIMIT, or reusing one of the two existing ClusterConfig fields this feature already added -- workerResyncWindowMs/workerResyncTimeoutMs are not a natural fit, so a new field would likely be needed), or explicitly document that the query direction is deliberately left unbounded by design (e.g. because a worker-side job count large enough to matter is not expected in practice). Until then, a worker rejoin with a very large non-terminal job count will always receive its OPCode_jobStateQuery as one very large single message, not multiple bounded pages, even though the receiving/decoding machinery on both sides already correctly supports true multi-page queries.",
|
||||
"tags": [
|
||||
"worker-rejoin-resync",
|
||||
"task-7.4",
|
||||
"worker-c",
|
||||
"worker_send_resync_query",
|
||||
"xce_job_state_query_req",
|
||||
"requirement-8.1",
|
||||
"requirement-8.3",
|
||||
"pagination",
|
||||
"design-reality-mismatch",
|
||||
"discovered-not-fixed",
|
||||
"asymmetric-with-report-direction"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1746",
|
||||
"bug-1747"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-02T00:30:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -4067,6 +4067,684 @@ static void test_burst_recovery_unbounded_by_job_count(void)
|
|||
printf("test_burst_recovery_unbounded_by_job_count: PASS\n");
|
||||
}
|
||||
|
||||
/* ── Task 7.4: Large job-count pagination correctness (Req 8.1-8.3) ──────
|
||||
*
|
||||
* design.md's own "Performance/Load" testing-strategy line: "Reconciliation
|
||||
* exchange for a worker running an order of magnitude more jobs than a
|
||||
* single message can carry completes correctly across multiple pages, not
|
||||
* truncated (Requirement 8.1-8.3)." Tasks 7.1-7.3 all deliberately stayed
|
||||
* within ONE wire page each (task 7.3's own Implementation Notes:
|
||||
* "Deliberately kept to a single wire page (page-boundary correctness is
|
||||
* task 7.4's separate concern)") -- THIS is that separate concern: proving
|
||||
* BOTH directions of the exchange (controller->worker OPCode_jobStateQuery,
|
||||
* worker->controller OPCode_jobStateReport) genuinely span MULTIPLE wire
|
||||
* pages for a large job count, and that resync_handle_report_page()'s
|
||||
* accumulator (task 6.2) reassembles all of them into exactly one applied
|
||||
* report -- not merely re-proving task 7.1's four-outcome/task 7.3's burst
|
||||
* recovery logic at a bigger N.
|
||||
*
|
||||
* Real production page-size constants (found by reading the actual call
|
||||
* sites, not assumed):
|
||||
* - REPORT direction (worker -> controller): xcecd's real production call
|
||||
* site (controller_connector.c's handle_jobstatequery ->
|
||||
* xcecd_resync_handle_query_page -> xcecd_resync_answer_query) passes
|
||||
* XCECD_RESYNC_REPORT_PKTLIMIT == 100 (controller_connector.c) into
|
||||
* xce_job_state_report_req() (task 1.4, request.c) -- a REAL, wired,
|
||||
* production page cap.
|
||||
* - QUERY direction (controller -> worker): the controller's own real
|
||||
* production call site, worker_send_resync_query() (worker.c:730),
|
||||
* hardcodes pktlimit=0 in its xce_job_state_query_req() call --
|
||||
* xce_job_state_query_req()'s own pktlimit<=0 convention means
|
||||
* "pktlimit = nrows", i.e. ALWAYS exactly ONE wire page, however many
|
||||
* jobs are queried. There is no real production constant to cite for
|
||||
* this direction -- see this task's own CONCERNS (bug-1755). To
|
||||
* genuinely exercise the QUERY side's already-implemented pagination
|
||||
* loop (task 1.4) at scale, this test calls xce_job_state_query_req()
|
||||
* directly (bypassing worker_send_resync_query()'s hardcoded 0),
|
||||
* supplying the SAME page size as the worker's own real report cap for
|
||||
* symmetry -- the exact same production function, exercised with a
|
||||
* real bound instead of an effectively-disabled one.
|
||||
*
|
||||
* TWR_LARGE_N_TOTAL (below) is an order of magnitude larger than
|
||||
* TWR_LARGE_PKTLIMIT (100), yielding a double-digit page count in EACH
|
||||
* direction, with a genuinely PARTIAL final page (the total, and each of
|
||||
* the four outcome-group sizes, are deliberately NOT multiples of the page
|
||||
* limit) -- so several wire pages straddle two different outcome groups, a
|
||||
* genuine page-boundary-falls-mid-group case, and the very last page of
|
||||
* each direction is short. Covers all four outcome kinds (running,
|
||||
* finished-success, finished-failure, unrecognized) so this is not a
|
||||
* degenerate single-outcome test.
|
||||
*/
|
||||
|
||||
#define TWR_JOBID_LARGE_BASE 900000ULL
|
||||
#define TWR_LARGE_N_RUNNING 260
|
||||
#define TWR_LARGE_N_FINOK 245
|
||||
#define TWR_LARGE_N_FINFAIL 235
|
||||
#define TWR_LARGE_N_UNREC 302
|
||||
#define TWR_LARGE_N_TOTAL 1042 /* 260+245+235+302 -- not a multiple of
|
||||
* TWR_LARGE_PKTLIMIT, so the final wire
|
||||
* page in each direction is genuinely
|
||||
* partial (42 entries), not just N full
|
||||
* pages */
|
||||
|
||||
/* Shared page-size cap for both directions in this test -- == the REAL
|
||||
* production XCECD_RESYNC_REPORT_PKTLIMIT for the report direction; the
|
||||
* query direction has no real production constant (see header comment
|
||||
* above / CONCERNS, bug-1755), so this test uses the same value for
|
||||
* symmetry while directly exercising xce_job_state_query_req()'s already-
|
||||
* implemented pagination loop. */
|
||||
#define TWR_LARGE_PKTLIMIT 100
|
||||
#define TWR_LARGE_N_PAGES \
|
||||
((TWR_LARGE_N_TOTAL + TWR_LARGE_PKTLIMIT - 1) / TWR_LARGE_PKTLIMIT)
|
||||
|
||||
static uint64_t twr_large_jobid(int i)
|
||||
{
|
||||
return TWR_JOBID_LARGE_BASE + (uint64_t)i;
|
||||
}
|
||||
|
||||
/* 0 = running, 1 = finished-success, 2 = finished-failure, 3 =
|
||||
* unrecognized -- see this test's own header comment on why the group
|
||||
* boundaries are deliberately not page-aligned. */
|
||||
static int twr_large_group(int i)
|
||||
{
|
||||
if (i < TWR_LARGE_N_RUNNING)
|
||||
return 0;
|
||||
if (i < TWR_LARGE_N_RUNNING + TWR_LARGE_N_FINOK)
|
||||
return 1;
|
||||
if (i < TWR_LARGE_N_RUNNING + TWR_LARGE_N_FINOK + TWR_LARGE_N_FINFAIL)
|
||||
return 2;
|
||||
return 3;
|
||||
}
|
||||
|
||||
static void test_large_job_count_pagination_correctness(void)
|
||||
{
|
||||
char tmp[256];
|
||||
uint64_t finok_anchor_s;
|
||||
|
||||
strcpy(tmp, "/tmp/xce_twr74_large_XXXXXX");
|
||||
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
|
||||
|
||||
/* Deliberately far from "now" -- same defensive idiom as every other
|
||||
* finish-time test in this file. */
|
||||
finok_anchor_s = (uint64_t)time(NULL) - 3600;
|
||||
|
||||
/* Phase 1: real cluster (workermgr + resmgr + jobmgr + hooksys) + real
|
||||
* disk persist -- drives the REAL rejoin/query-construction chain over
|
||||
* TWR_LARGE_N_TOTAL jobs, then a REAL, explicitly-bounded wire query
|
||||
* send and a REAL, real-production-bounded wire report send, both
|
||||
* genuinely multi-page. Own local cluster_t (not the shared g_twr
|
||||
* fixture), mirroring every other restart-replay test in this file. */
|
||||
{
|
||||
cluster_t c;
|
||||
kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING,
|
||||
.nslots = 64, .delta = 16 };
|
||||
kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
|
||||
.nslots = 1024, .delta = 256 };
|
||||
persist_config_t cfg;
|
||||
worker_t* w;
|
||||
job_t** jobs;
|
||||
hostattrs_t attrs, new_attrs;
|
||||
xce_hostconfig_t hconfig;
|
||||
endpoint_t stack_ep;
|
||||
endpoint_t* ep;
|
||||
xce_job_state_entry_t* storage;
|
||||
xce_job_state_entry_t** entries;
|
||||
xce_job_state_query_t query;
|
||||
xce_job_state_report_t report;
|
||||
int i, rc;
|
||||
uint64_t lsn_before, lsn_after;
|
||||
int calls_before, calls_after;
|
||||
|
||||
static const char* res_names[TWR_RT_N_RES] = {
|
||||
"type", "model", "cpuf", "maxcpus", "maxmem",
|
||||
"maxswap", "maxtmp", "ndisks", "slots"
|
||||
};
|
||||
static const int res_valtypes[TWR_RT_N_RES] = {
|
||||
XCE_VAL_STRING, XCE_VAL_STRING,
|
||||
XCE_VAL_NUMERIC, XCE_VAL_NUMERIC, XCE_VAL_NUMERIC,
|
||||
XCE_VAL_NUMERIC, XCE_VAL_NUMERIC, XCE_VAL_NUMERIC,
|
||||
XCE_VAL_NUMERIC
|
||||
};
|
||||
|
||||
if (makeString((char*)"twr74-probe") == NULL)
|
||||
xstring_table_init(64);
|
||||
if (!g_twr_jobobj_ready) {
|
||||
TEST_ASSERT_EQUAL_INT(0, job_obj_init(32, 32));
|
||||
g_twr_jobobj_ready = 1;
|
||||
}
|
||||
|
||||
memset(&c, 0, sizeof(c));
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
xce_hook_init(&c.hooksys, XCE_CONTROLLER_HOOK_MAX));
|
||||
|
||||
c.workermgr = makeInstance(&xkvmgr_type, &str_params);
|
||||
TEST_ASSERT_NOT_NULL(c.workermgr);
|
||||
c.resmgr = makeInstance(&xkvmgr_type, &str_params);
|
||||
TEST_ASSERT_NOT_NULL(c.resmgr);
|
||||
c.jobmgr = makeInstance(&xkvmgr_type, &int_params);
|
||||
TEST_ASSERT_NOT_NULL(c.jobmgr);
|
||||
|
||||
for (i = 0; i < TWR_RT_N_RES; i++) {
|
||||
xce_resource_t rdef;
|
||||
memset(&rdef, 0, sizeof(rdef));
|
||||
rdef.name = (char*)res_names[i];
|
||||
rdef.valtype = res_valtypes[i];
|
||||
rdef.flags = 0;
|
||||
rdef.idx = -1;
|
||||
TEST_ASSERT_NOT_NULL(makeResource(c.resmgr, &rdef));
|
||||
}
|
||||
|
||||
c.myhost_info.name = (char*)"twr74-controller";
|
||||
c.myhost_info.key = (char*)"twr74-key";
|
||||
c.myhost = &c.myhost_info;
|
||||
|
||||
cfg = persist_config_defaults();
|
||||
cfg.wal_sync_ms = 5000;
|
||||
persist_set_workdir(&c.persist, tmp, "ctrl-test");
|
||||
TEST_ASSERT_EQUAL_INT(0, persist_init(&c.persist, &cfg, NULL));
|
||||
TEST_ASSERT_EQUAL_INT(0, persist_start(&c.persist));
|
||||
|
||||
jobs = (job_t**)malloc(sizeof(job_t*) * TWR_LARGE_N_TOTAL);
|
||||
TEST_ASSERT_NOT_NULL(jobs);
|
||||
|
||||
/* Seed a durable WAL_JOB_SUBMIT fact + a matching in-memory job_t
|
||||
* for all TWR_LARGE_N_TOTAL jobs -- all RUN, all assigned to the
|
||||
* SAME worker host, exactly what the controller believes right
|
||||
* before this rejoin (bypassing dispatch/replay, mirroring every
|
||||
* sibling restart-replay test in this file). */
|
||||
for (i = 0; i < TWR_LARGE_N_TOTAL; i++) {
|
||||
uint64_t jobid = twr_large_jobid(i);
|
||||
job_submit_data_t wal_submit;
|
||||
xce_job_t wal_jdef;
|
||||
uint8_t* buf = NULL;
|
||||
size_t sz = 0;
|
||||
job_submit_data_t submit;
|
||||
xce_job_t jdef;
|
||||
|
||||
memset(&wal_submit, 0, sizeof(wal_submit));
|
||||
wal_submit.queue = (char*)"q";
|
||||
wal_submit.user = (char*)"tester";
|
||||
memset(&wal_jdef, 0, sizeof(wal_jdef));
|
||||
wal_jdef.jobid = jobid;
|
||||
wal_jdef.state = JOB_STAT_RUN;
|
||||
wal_jdef.submit = &wal_submit;
|
||||
TEST_ASSERT_EQUAL_INT(0, twr_encode_job(&wal_jdef, &buf, &sz));
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
persist_wal_async(&c.persist, WAL_JOB_SUBMIT, buf,
|
||||
(uint32_t)sz));
|
||||
free(buf);
|
||||
|
||||
memset(&submit, 0, sizeof(submit));
|
||||
submit.queue = (char*)"q";
|
||||
submit.user = (char*)"tester";
|
||||
memset(&jdef, 0, sizeof(jdef));
|
||||
jdef.jobid = jobid;
|
||||
jdef.state = JOB_STAT_RUN;
|
||||
jdef.submit = &submit;
|
||||
jobs[i] = makeJob(c.jobmgr, &jdef);
|
||||
TEST_ASSERT_NOT_NULL(jobs[i]);
|
||||
jobs[i]->job.allocs
|
||||
= twr_make_test_alloc_for_host(jobid, "resync-large-host");
|
||||
TEST_ASSERT_NOT_NULL(jobs[i]->job.allocs);
|
||||
}
|
||||
|
||||
/* ── Join: rejoin_worker (the REAL join entry point) arms the round
|
||||
* over all TWR_LARGE_N_TOTAL non-terminal jobs -- Req 1.1, 2.1, and
|
||||
* the "no artificial cap" proof (Req 8.1) that resync_collect_
|
||||
* query_jobids' realloc-doubling growth (resync.c) genuinely has no
|
||||
* fixed-size ceiling. */
|
||||
memset(&attrs, 0, sizeof(attrs));
|
||||
attrs.hosttype = makeString("LINUX");
|
||||
attrs.hostmodel = makeString("x86");
|
||||
attrs.cpufactor = 1.0f;
|
||||
attrs.maxcpus = 4;
|
||||
attrs.maxmem = 16;
|
||||
attrs.maxswap = 32;
|
||||
attrs.maxtmp = 64;
|
||||
attrs.ndisks = 1;
|
||||
|
||||
{
|
||||
xce_worker_t host;
|
||||
memset(&host, 0, sizeof(host));
|
||||
host.name = (char*)"resync-large-host";
|
||||
host.attrs = &attrs;
|
||||
w = makeWorker(c.workermgr, &host);
|
||||
}
|
||||
TEST_ASSERT_NOT_NULL(w);
|
||||
|
||||
memset(&new_attrs, 0, sizeof(new_attrs));
|
||||
new_attrs.hosttype = makeString("LINUX");
|
||||
new_attrs.hostmodel = makeString("x86");
|
||||
new_attrs.cpufactor = 1.0f;
|
||||
new_attrs.maxcpus = 4;
|
||||
new_attrs.maxmem = 16;
|
||||
new_attrs.maxswap = 32;
|
||||
new_attrs.maxtmp = 64;
|
||||
new_attrs.ndisks = 1;
|
||||
|
||||
memset(&hconfig, 0, sizeof(hconfig));
|
||||
hconfig.hosttype = (char*)"!";
|
||||
hconfig.hostmodel = (char*)"!";
|
||||
hconfig.mxj = 4;
|
||||
|
||||
memset(&stack_ep, 0, sizeof(stack_ep));
|
||||
rc = rejoin_worker(&c, w, &stack_ep, &new_attrs, &hconfig);
|
||||
TEST_ASSERT_GREATER_OR_EQUAL(0, rc);
|
||||
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(w->resync,
|
||||
"a rejoin with TWR_LARGE_N_TOTAL known non-terminal jobs on "
|
||||
"this host must arm a reconciliation round (Req 1.1)");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(TWR_LARGE_N_TOTAL,
|
||||
w->resync->n_queried_jobids,
|
||||
"the round's query set must match exactly the host's non-"
|
||||
"terminal jobs, with no artificial cap on how many resync_"
|
||||
"arm_round will collect (Req 1.1, 2.1, 8.1)");
|
||||
TEST_ASSERT_TRUE(IS_WORKER_RECONCILING(w->host.status));
|
||||
|
||||
ep = twr_rt_new_test_endpoint();
|
||||
|
||||
/* ── QUERY direction: a REAL wire-encoded xce_job_state_query_req()
|
||||
* send, called directly (not via worker_send_resync_query(), which
|
||||
* hardcodes pktlimit=0 -- see this test's own header comment and
|
||||
* CONCERNS) with an explicit page size, genuinely producing
|
||||
* multiple wire pages for this job count. */
|
||||
query = (xce_job_state_query_t){
|
||||
.n_total = w->resync->n_queried_jobids,
|
||||
.pos = 0,
|
||||
.count = w->resync->n_queried_jobids,
|
||||
.jobids = w->resync->queried_jobids
|
||||
};
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
xce_job_state_query_req(ep, c.myhost, &query, NULL, NULL,
|
||||
TWR_LARGE_PKTLIMIT));
|
||||
|
||||
{
|
||||
bool seen_query[TWR_LARGE_N_TOTAL];
|
||||
int running_total = 0;
|
||||
int query_pages = 0;
|
||||
|
||||
memset(seen_query, 0, sizeof(seen_query));
|
||||
|
||||
while (running_total < TWR_LARGE_N_TOTAL) {
|
||||
xce_request_t decoded_query;
|
||||
int j;
|
||||
|
||||
twr_rt_pop_and_decode(ep, &decoded_query);
|
||||
TEST_ASSERT_EQUAL_INT(OPCode_jobStateQuery,
|
||||
decoded_query.opcode);
|
||||
TEST_ASSERT_EQUAL_INT(Request_body_jobstatequerymsg,
|
||||
decoded_query.body_which);
|
||||
TEST_ASSERT_NOT_NULL(decoded_query.body.jobstatequerymsg);
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(TWR_LARGE_N_TOTAL,
|
||||
decoded_query.body.jobstatequerymsg->n_total,
|
||||
"Req 2.1: every query page must carry the SAME full "
|
||||
"n_total, not a per-page total");
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(running_total,
|
||||
decoded_query.body.jobstatequerymsg->pos,
|
||||
"Req 8.3: each page's pos must pick up exactly where "
|
||||
"the previous page's count left off -- no gap, no "
|
||||
"overlap");
|
||||
|
||||
for (j = 0; j < decoded_query.body.jobstatequerymsg->count;
|
||||
j++) {
|
||||
uint64_t jobid
|
||||
= decoded_query.body.jobstatequerymsg->jobids[j];
|
||||
uint64_t idx = jobid - TWR_JOBID_LARGE_BASE;
|
||||
|
||||
TEST_ASSERT_TRUE_MESSAGE(
|
||||
idx < (uint64_t)TWR_LARGE_N_TOTAL,
|
||||
"a jobid appearing on the wire query must be one "
|
||||
"of the armed round's own jobids");
|
||||
TEST_ASSERT_FALSE_MESSAGE(seen_query[idx],
|
||||
"Req 8.3: no jobid may be sent twice across query "
|
||||
"pages");
|
||||
seen_query[idx] = true;
|
||||
}
|
||||
|
||||
running_total
|
||||
+= decoded_query.body.jobstatequerymsg->count;
|
||||
query_pages++;
|
||||
xce_free_request(&decoded_query);
|
||||
}
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(TWR_LARGE_N_PAGES, query_pages,
|
||||
"Req 8.1-8.3: an order-of-magnitude-larger-than-one-page "
|
||||
"query must genuinely span many wire pages, including a "
|
||||
"partial final page, not silently collapse to one (the "
|
||||
"real production worker_send_resync_query() call site "
|
||||
"would collapse to one -- see CONCERNS, bug-1755 -- this "
|
||||
"assertion is against xce_job_state_query_req()'s own "
|
||||
"pagination mechanism, exercised directly)");
|
||||
|
||||
for (i = 0; i < TWR_LARGE_N_TOTAL; i++) {
|
||||
TEST_ASSERT_TRUE_MESSAGE(seen_query[i],
|
||||
"Req 8.3: every armed jobid must appear exactly once "
|
||||
"across all query pages -- none silently dropped past "
|
||||
"a page boundary");
|
||||
}
|
||||
}
|
||||
|
||||
/* ── Worker's answer: a REAL wire-encoded-then-decoded OPCode_
|
||||
* jobStateReport, sent at the REAL production report page size
|
||||
* (XCECD_RESYNC_REPORT_PKTLIMIT, controller_connector.c) --
|
||||
* genuinely many wire pages -- covering all four outcome kinds,
|
||||
* fed page-by-page into resync_handle_report_page() (task 6.2's
|
||||
* real reassembly entry point), never resync_apply_report()
|
||||
* directly. */
|
||||
storage = (xce_job_state_entry_t*)malloc(
|
||||
sizeof(xce_job_state_entry_t) * TWR_LARGE_N_TOTAL);
|
||||
entries = (xce_job_state_entry_t**)malloc(
|
||||
sizeof(xce_job_state_entry_t*) * TWR_LARGE_N_TOTAL);
|
||||
TEST_ASSERT_NOT_NULL(storage);
|
||||
TEST_ASSERT_NOT_NULL(entries);
|
||||
|
||||
for (i = 0; i < TWR_LARGE_N_TOTAL; i++) {
|
||||
xce_job_state_entry_t* e = &storage[i];
|
||||
|
||||
memset(e, 0, sizeof(*e));
|
||||
e->jobid = twr_large_jobid(i);
|
||||
e->queried = 1;
|
||||
|
||||
switch (twr_large_group(i)) {
|
||||
case 0:
|
||||
e->status = JobStateStatus_running;
|
||||
break;
|
||||
case 1: {
|
||||
int local = i - TWR_LARGE_N_RUNNING;
|
||||
uint64_t fin_s = finok_anchor_s - (uint64_t)(local * 3);
|
||||
e->status = JobStateStatus_finishedOk;
|
||||
e->finish_time_ms = fin_s * 1000ULL;
|
||||
break;
|
||||
}
|
||||
case 2: {
|
||||
int local = i - TWR_LARGE_N_RUNNING - TWR_LARGE_N_FINOK;
|
||||
e->status = JobStateStatus_finishedFail;
|
||||
e->exit_code = 200 + local;
|
||||
break;
|
||||
}
|
||||
default:
|
||||
e->status = JobStateStatus_unrecognized;
|
||||
break;
|
||||
}
|
||||
|
||||
entries[i] = e;
|
||||
}
|
||||
|
||||
report = (xce_job_state_report_t){
|
||||
.n_total = TWR_LARGE_N_TOTAL,
|
||||
.pos = 0,
|
||||
.count = TWR_LARGE_N_TOTAL,
|
||||
.entries = entries
|
||||
};
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
xce_job_state_report_req(ep, c.myhost, &report, NULL, NULL,
|
||||
TWR_LARGE_PKTLIMIT));
|
||||
|
||||
g_twr_update_hook_fires = 0;
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
xce_add_hook(&c.hooksys, XCE_CONTROLLER_HOOK_JOB,
|
||||
twr_job_update_spy_hook, NULL));
|
||||
|
||||
pthread_mutex_lock(&c.persist.lock);
|
||||
lsn_before = c.persist.current_lsn;
|
||||
pthread_mutex_unlock(&c.persist.lock);
|
||||
|
||||
resync_apply_report_call_count_reset_for_test();
|
||||
calls_before = resync_apply_report_call_count_for_test();
|
||||
|
||||
{
|
||||
int running_total = 0;
|
||||
int report_pages = 0;
|
||||
|
||||
while (running_total < TWR_LARGE_N_TOTAL) {
|
||||
xce_request_t decoded_report;
|
||||
int rc_page;
|
||||
|
||||
twr_rt_pop_and_decode(ep, &decoded_report);
|
||||
TEST_ASSERT_EQUAL_INT(OPCode_jobStateReport,
|
||||
decoded_report.opcode);
|
||||
TEST_ASSERT_EQUAL_INT(Request_body_jobstatereportmsg,
|
||||
decoded_report.body_which);
|
||||
TEST_ASSERT_NOT_NULL(decoded_report.body.jobstatereportmsg);
|
||||
|
||||
running_total
|
||||
+= decoded_report.body.jobstatereportmsg->count;
|
||||
report_pages++;
|
||||
|
||||
rc_page = resync_handle_report_page(
|
||||
&c, w, decoded_report.body.jobstatereportmsg);
|
||||
|
||||
if (running_total < TWR_LARGE_N_TOTAL) {
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc_page,
|
||||
"a non-final report page must not complete the "
|
||||
"report");
|
||||
} else {
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, rc_page,
|
||||
"the final (partial) report page must complete "
|
||||
"the report");
|
||||
}
|
||||
|
||||
xce_free_request(&decoded_report);
|
||||
}
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(TWR_LARGE_N_PAGES, report_pages,
|
||||
"Req 8.1-8.3: at the real production per-page limit, this "
|
||||
"job count must genuinely span many wire pages, including "
|
||||
"a partial final page");
|
||||
}
|
||||
|
||||
calls_after = resync_apply_report_call_count_for_test();
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(1, calls_after - calls_before,
|
||||
"resync_apply_report must be invoked EXACTLY ONCE for the "
|
||||
"whole large, many-page report -- not once per page (task "
|
||||
"6.2's own structural proof, generalized from 2 pages to "
|
||||
"double digits)");
|
||||
|
||||
pthread_mutex_lock(&c.persist.lock);
|
||||
lsn_after = c.persist.current_lsn;
|
||||
pthread_mutex_unlock(&c.persist.lock);
|
||||
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(
|
||||
lsn_before
|
||||
+ (uint64_t)(TWR_LARGE_N_FINOK + TWR_LARGE_N_FINFAIL
|
||||
+ TWR_LARGE_N_UNREC),
|
||||
lsn_after,
|
||||
"EXACTLY one durable fact per non-running job (finished-"
|
||||
"success x WAL_JOB_FINISH + finished-failure x WAL_JOB_FAIL + "
|
||||
"unrecognized x WAL_JOB_UPDATE), and NONE for the still-"
|
||||
"running jobs -- an exact count (not merely a lower bound) is "
|
||||
"the genuine 'none dropped, none duplicated across page "
|
||||
"boundaries' proof");
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(TWR_LARGE_N_UNREC,
|
||||
g_twr_update_hook_fires,
|
||||
"Req 7.1-7.3: the EAGER unrecognized-to-pending transition "
|
||||
"must reach the scheduler's/apiserver's notify path "
|
||||
"synchronously, exactly once per unrecognized job, regardless "
|
||||
"of which page it happened to land on");
|
||||
|
||||
TEST_ASSERT_NULL_MESSAGE(w->resync,
|
||||
"once every queried job across every report page has a "
|
||||
"reconciled outcome, the round must resolve (Req 5.1, 5.2)");
|
||||
TEST_ASSERT_FALSE_MESSAGE(IS_WORKER_RECONCILING(w->host.status),
|
||||
"XCE_WORKER_RECONCILING must clear once the round resolves");
|
||||
|
||||
/* Phase 1 end-to-end per-job assertions (Req 8.1-8.3: EVERY job
|
||||
* accounted for, not a sample) -- everything observable before any
|
||||
* commit round-trip. */
|
||||
for (i = 0; i < TWR_LARGE_N_TOTAL; i++) {
|
||||
switch (twr_large_group(i)) {
|
||||
case 0:
|
||||
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_RUN,
|
||||
jobs[i]->job.state,
|
||||
"still-running (queried) must be left exactly as-is");
|
||||
TEST_ASSERT_NOT_NULL(jobs[i]->job.allocs);
|
||||
break;
|
||||
case 1:
|
||||
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_RUN,
|
||||
jobs[i]->job.state,
|
||||
"finished-success stays DEFERRED (record_job_finish_"
|
||||
"fact_at) -- mutates later via apply_committed_entry, "
|
||||
"not synchronously here");
|
||||
break;
|
||||
case 2:
|
||||
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_RUN,
|
||||
jobs[i]->job.state,
|
||||
"finished-failure stays DEFERRED (on_job_fail/record_"
|
||||
"job_fail_fact), same as finished-success");
|
||||
break;
|
||||
default:
|
||||
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_PEND,
|
||||
jobs[i]->job.state,
|
||||
"unrecognized reconciles EAGERLY -- PEND "
|
||||
"synchronously, before any commit round-trip");
|
||||
TEST_ASSERT_NULL_MESSAGE(jobs[i]->job.allocs,
|
||||
"bug-1631: reconciled-to-pending must release the "
|
||||
"allocation synchronously, at scale, across page "
|
||||
"boundaries");
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
xce_disable_hook(&c.hooksys, XCE_CONTROLLER_HOOK_JOB,
|
||||
twr_job_update_spy_hook);
|
||||
|
||||
TEST_ASSERT_EQUAL_INT(0, persist_flush(&c.persist));
|
||||
|
||||
persist_stop(&c.persist);
|
||||
persist_fini(&c.persist);
|
||||
if (c.workermgr) dereference(c.workermgr);
|
||||
if (c.resmgr) dereference(c.resmgr);
|
||||
if (c.jobmgr) dereference(c.jobmgr);
|
||||
xce_hook_fini(&c.hooksys);
|
||||
|
||||
dereference(new_attrs.hosttype);
|
||||
dereference(new_attrs.hostmodel);
|
||||
endpoint_fini(ep);
|
||||
free(ep);
|
||||
|
||||
free(jobs);
|
||||
free(storage);
|
||||
free(entries);
|
||||
}
|
||||
|
||||
/* Phase 2: fresh persist_t + fresh cluster_t (jobmgr/hooksys only),
|
||||
* recover from disk -- simulates a controller restart, settling the two
|
||||
* DEFERRED outcomes (finished-success/failure) across every page they
|
||||
* happened to land on, and re-confirming the EAGER unrecognized->
|
||||
* pending outcome also round-trips through a real WAL replay, not just
|
||||
* the live eager-mutation path -- for EVERY one of the TWR_LARGE_N_
|
||||
* TOTAL jobs, individually, not a sample. */
|
||||
{
|
||||
persist_t* p2 = twr_make_fresh_persist(tmp);
|
||||
cluster_t c2;
|
||||
job_t** jobs2;
|
||||
int rc, i;
|
||||
kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
|
||||
.nslots = 1024, .delta = 256 };
|
||||
|
||||
TEST_ASSERT_NOT_NULL(p2);
|
||||
|
||||
memset(&c2, 0, sizeof(c2));
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
xce_hook_init(&c2.hooksys, XCE_CONTROLLER_HOOK_MAX));
|
||||
c2.jobmgr = makeInstance(&xkvmgr_type, &int_params);
|
||||
TEST_ASSERT_NOT_NULL(c2.jobmgr);
|
||||
|
||||
g_twr_update_hook_fires = 0;
|
||||
TEST_ASSERT_EQUAL_INT(0,
|
||||
xce_add_hook(&c2.hooksys, XCE_CONTROLLER_HOOK_JOB,
|
||||
twr_job_update_spy_hook, NULL));
|
||||
|
||||
p2->cluster = &c2;
|
||||
rc = persist_recover(p2);
|
||||
TEST_ASSERT_EQUAL_INT(0, rc);
|
||||
|
||||
jobs2 = (job_t**)malloc(sizeof(job_t*) * TWR_LARGE_N_TOTAL);
|
||||
TEST_ASSERT_NOT_NULL(jobs2);
|
||||
|
||||
lock(c2.jobmgr, RTTI_LOCK_READ);
|
||||
for (i = 0; i < TWR_LARGE_N_TOTAL; i++) {
|
||||
jobs2[i] = NULL;
|
||||
key_find(c2.jobmgr, (void*)twr_large_jobid(i),
|
||||
(void**)&jobs2[i]);
|
||||
}
|
||||
unlock(c2.jobmgr);
|
||||
|
||||
for (i = 0; i < TWR_LARGE_N_TOTAL; i++) {
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(jobs2[i],
|
||||
"restart replay must reconstruct every job via its own "
|
||||
"WAL_JOB_SUBMIT fact");
|
||||
|
||||
switch (twr_large_group(i)) {
|
||||
case 0:
|
||||
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_RUN,
|
||||
jobs2[i]->job.state,
|
||||
"still-running: no fact was ever recorded, so replay "
|
||||
"leaves it exactly as WAL_JOB_SUBMIT alone "
|
||||
"reconstructs it");
|
||||
break;
|
||||
case 1: {
|
||||
int local = i - TWR_LARGE_N_RUNNING;
|
||||
uint64_t fin_s = finok_anchor_s - (uint64_t)(local * 3);
|
||||
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_DONE,
|
||||
jobs2[i]->job.state,
|
||||
"finished-success: the DEFERRED record_job_finish_"
|
||||
"fact_at fact must apply to DONE on replay");
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(fin_s,
|
||||
(uint64_t)jobs2[i]->job.finish_time,
|
||||
"each finished-success job must recover ITS OWN "
|
||||
"worker-reported finish_time -- no cross-job mixing "
|
||||
"across a page boundary");
|
||||
break;
|
||||
}
|
||||
case 2:
|
||||
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_EXIT,
|
||||
jobs2[i]->job.state,
|
||||
"finished-failure: the DEFERRED on_job_fail/record_"
|
||||
"job_fail_fact fact must apply to EXIT on replay");
|
||||
break;
|
||||
default:
|
||||
TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_PEND,
|
||||
jobs2[i]->job.state,
|
||||
"unrecognized: the EAGER record_job_reconcile_"
|
||||
"pending_fact's durable WAL_JOB_UPDATE fact must ALSO "
|
||||
"reconstruct correctly on replay");
|
||||
TEST_ASSERT_NULL(jobs2[i]->job.allocs);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(
|
||||
TWR_LARGE_N_FINOK + TWR_LARGE_N_FINFAIL + TWR_LARGE_N_UNREC,
|
||||
g_twr_update_hook_fires,
|
||||
"Req 7.1-7.3: replaying the durable facts this reconciliation "
|
||||
"round produced must reach the scheduler's/apiserver's notify "
|
||||
"path for EVERY actual state change -- finished-success, "
|
||||
"finished-failure, and unrecognized-to-pending -- exactly "
|
||||
"once each, across every job (still-running recorded no fact, "
|
||||
"and WAL_JOB_SUBMIT fires HOOK_DETAIL_JOB_NEW, not JOB_UPDATE, "
|
||||
"so neither adds to this count)");
|
||||
|
||||
xce_disable_hook(&c2.hooksys, XCE_CONTROLLER_HOOK_JOB,
|
||||
twr_job_update_spy_hook);
|
||||
xce_hook_fini(&c2.hooksys);
|
||||
if (c2.jobmgr) dereference(c2.jobmgr);
|
||||
twr_teardown_persist(p2);
|
||||
free(jobs2);
|
||||
}
|
||||
|
||||
{
|
||||
char cmd[512];
|
||||
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmp);
|
||||
if (system(cmd) != 0)
|
||||
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", tmp);
|
||||
}
|
||||
|
||||
printf("test_large_job_count_pagination_correctness: PASS\n");
|
||||
}
|
||||
|
||||
/* ── Test group entry point ───────────────────────────────────────────── */
|
||||
|
||||
void test_worker_resync(void)
|
||||
|
|
@ -4098,4 +4776,5 @@ void test_worker_resync(void)
|
|||
RUN_TEST(test_full_rejoin_reconciliation_round_trip);
|
||||
RUN_TEST(test_failover_during_job_recovers_lost_finish);
|
||||
RUN_TEST(test_burst_recovery_unbounded_by_job_count);
|
||||
RUN_TEST(test_large_job_count_pagination_correctness);
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue