lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

test(scheduler-allocator-runner): characterize the seqjob allocator (task 2.1)

Browse source 
12 tests drive the real seqjob plugin through job_alloc_fn and record
today's results, reasons and debits, with switch-stable OOM injection
(sched_make_allocs stub, --wrap of make_host_alloc/make_job_alloc).
Assertions the runner changes on purpose sit in *_today tests.

The characterization showed bug-1893's real effect: vget() returns 0 on
success, so the shared-reason loop (rc <= 0 skip) never records a shared
NO_RESOURCE reason. Bug record and spec wording corrected; the runner
skips only on rc < 0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-30 08:11:26 +00:00
parent 31ad79d325
commit 91a8580948
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
8 changed files with 835 additions and 16 deletions
Download patch file Download diff file Expand all files Collapse all files

16
.kiro/specs/scheduler-allocator-runner/brief.md
View file

@ -36,7 +36,7 @@ Driver: `issues/new_requirements.md` item 4.
test and sim-parity fixture. Intentional differences, all outside the
decision and main pend reason: pending seqjob jobs may gain per-host
`NO_RESOURCE` reasons (Constraints item 1); spurious shared reasons are
dropped (bug-1893, item 6); seqjob gets the `-EINVAL` guard and OOM
recorded for the first time (bug-1893, item 6); seqjob gets the `-EINVAL` guard and OOM
rollback (Req 5 AC 1/5).
- `new_requirements.md` item 4 closed.
@ -173,12 +173,14 @@ goal that plugins no longer export `my_alloc()`).
testsuite files reference the allocators).
5. Proposal-sketch defects to avoid: `sched_ctx_primary == NULL` typo;
`claims[plan->nbuckets]` VLA (heap-allocate or bound by `maxTasks`).
6. **Decided 2026-09-29 (requirements review): the shared pend-reason loop
iterates the unmet list (`ctx->unmeet.count`), fixing bug-1893.** Both
current `my_alloc()`s loop to `ctx->alloc_req.shared.count` over
`ctx->unmeet`, whose tail is zero-filled, and add reasons naming
resource index 0. Moving the loop into the runner is the natural point
to fix it; main reason and decisions are unaffected.
6. **Decided 2026-09-29 (requirements review), corrected 2026-09-30 (task
2.1): the shared pend-reason loop iterates the unmet list
(`ctx->unmeet.count`) and skips only on `vget() < 0`, fixing bug-1893.**
Both current `my_alloc()`s skip on `vget() <= 0`, but `vget` returns 0
on success, so today no shared `NO_RESOURCE` reason is ever recorded
(the loop also runs to `alloc_req.shared.count`). After the fix a job
pending on shared resources names each short one; main reason and
decisions are unaffected.
- Claim `host_req` snapshots are owned by the framework: fini on every exit
path (success, bucket failure, OOM).
- Verify with: `test_seqjob_*`, `test_parjob_*`, `test_seqjob_rewrap_equivalence`,

2
.kiro/specs/scheduler-allocator-runner/design.md
View file

@ -269,7 +269,7 @@ const xce_sched_allocator_iface_t* allocator_iface;
**Responsibilities & Constraints**
- Order: (1) input guard: `plugin`, `ctx`, `plugin->core`, `plugin->sched`, `ctx->primary`, `plugin->allocator_iface` and its `plan_*` members non-NULL, else `-EINVAL` before touching `ctx->reason` (5.1); (2) `accepts` if non-NULL, reject → 0 (1.3, 1.4); (3) `ensure_reason`: create `make_pendreason(0)` if `ctx->reason == NULL`, `-ENOMEM` on failure (2.1, 2.4); (4) `prepare_shared` (below); (5) `plan_init(iter, ctx->primary, plugin->core)`, non-zero → 0 without `plan_fini` (3.4, as today); (6) plan loop; (7) cleanup.
- `prepare_shared`: `sched_ctx_get_uncomit_shared`; `xvec_idval_fini(&ctx->unmeet)`; `sched_check_shared_satisfy(core, uncomit, &ctx->alloc_req.shared, &ctx->unmeet)`; `< 0` → return it; `== 0` → for `i < ctx->unmeet.count`: `vget(resmgr, unmeet.idxs[i])`, skip on `rc <= 0`, `add_job_preason(reason, PREASON_NO_RESOURCE, NULL, r->name)`; return 0 (2.2); `> 0` → `xvec_idval(uncomit, &ctx->alloc_req.shared, XVEC_ADD_FLOAT)` once, error → return it (2.3, 2.4). Shared amounts are never subtracted by the runner (2.3).
- `prepare_shared`: `sched_ctx_get_uncomit_shared`; `xvec_idval_fini(&ctx->unmeet)`; `sched_check_shared_satisfy(core, uncomit, &ctx->alloc_req.shared, &ctx->unmeet)`; `< 0` → return it; `== 0` → for `i < ctx->unmeet.count`: `vget(resmgr, unmeet.idxs[i])`, skip only on `rc < 0` (`vget` returns 0 on success; today's `rc <= 0` skips every hit, bug-1893), `add_job_preason(reason, PREASON_NO_RESOURCE, NULL, r->name)`; return 0 (2.2); `> 0` → `xvec_idval(uncomit, &ctx->alloc_req.shared, XVEC_ADD_FLOAT)` once, error → return it (2.3, 2.4). Shared amounts are never subtracted by the runner (2.3).
- Claims buffer: `sched_alloc_claim_t* claims` (heap), capacity grown with `realloc` to `plan->nbuckets` before a plan is tried; growth failure → `plan_fini`, free, `-ENOMEM` (3.8). No claim is outstanding between plans.
- Uses only `ctx->primary` and `ctx->alloc_req` for sizing (5.8).
- Precondition `ctx->alloc == NULL` on entry (the dispatch and COMPOUND/ALTER callers move or free any previous result before the next attempt); the runner never frees or overwrites a non-NULL `ctx->alloc` except with its own result on success.

6
.kiro/specs/scheduler-allocator-runner/requirements.md
View file

@ -5,12 +5,12 @@
**Current situation:** `seqjob_allocator.c` (`my_alloc`, ~215 lines) and `parjob_allocator.c` (`my_alloc`, ~220 lines) each implement the whole allocation algorithm, although only plan generation is plugin-specific. Over 100 lines are duplicated almost verbatim: lazy pend-reason creation, getting the uncommitted shared resources, the shared-resource satisfaction check with its `PREASON_NO_RESOURCE` fill, adding the uncommitted shared resources, the plan-iterator loop skeleton, and the commit tail (`make_job_alloc` + `sched_make_allocs` + the `-ENOMEM` cleanup chain). Every allocation bug fix (bug-1882, bug-1767, bug-1768) has to land twice, and a new allocator plugin would have to copy the whole `my_alloc()`. Plan generation is already abstracted behind `xce_alloc_plan_iter_t`, whose `init` takes an unused `xce_scheduler_t*` even though sim mode has no real scheduler. Tracked as `issues/new_requirements.md` item 4.
**What should change:** the scheduler framework owns allocation. A new `xce_sched_allocator_iface_t` (an optional `accepts` guard plus `plan_init`/`plan_next`/`plan_fini`), referenced from `xce_sched_plugin_t` in the same way as `picker_iface`, lets one framework function, `sched_allocator_run()`, carry out shared-resource preparation, the bucket → host claim loop with rollback, and the commit tail. Both plugins register `.job_alloc_fn = sched_allocator_run`, supply only their plan iterator and `prepare_allocation`, and no longer define `my_alloc()`. The third parameter of `xce_alloc_plan_iter_t.init` is retyped to `xce_sched_core_t*`. Allocation decisions, pend reasons (except seqjob's per-host reasons, which become complete, Requirement 3 AC 6-7, and the spurious shared reasons of bug-1893, Requirement 2 AC 2), and error and `-ENOMEM` behaviour (except the seqjob hardening of Requirement 5 AC 1 and AC 5) stay identical for every existing test and sim-parity fixture, including bug-1882, bug-1767 and bug-1768. There is no per-plugin `commit` hook and no `sched_make_allocs()` retype (both left out of scope). The design must settle explicitly the seqjob/parjob divergences listed in `brief.md` → Constraints (the per-bucket `alloc_req.host` reset, decided 2026-09-29 as an unconditional reset for both allocators; seqjob's new OOM snapshot path, where `nslots`/`nhosts` are consumed, seqjob `accepts` for direct callers, and no VLA). Full context: `.kiro/specs/scheduler-allocator-runner/brief.md`.
**What should change:** the scheduler framework owns allocation. A new `xce_sched_allocator_iface_t` (an optional `accepts` guard plus `plan_init`/`plan_next`/`plan_fini`), referenced from `xce_sched_plugin_t` in the same way as `picker_iface`, lets one framework function, `sched_allocator_run()`, carry out shared-resource preparation, the bucket → host claim loop with rollback, and the commit tail. Both plugins register `.job_alloc_fn = sched_allocator_run`, supply only their plan iterator and `prepare_allocation`, and no longer define `my_alloc()`. The third parameter of `xce_alloc_plan_iter_t.init` is retyped to `xce_sched_core_t*`. Allocation decisions, pend reasons (except seqjob's per-host reasons, which become complete, Requirement 3 AC 6-7, and the shared-resource reasons bug-1893 never recorded, Requirement 2 AC 2), and error and `-ENOMEM` behaviour (except the seqjob hardening of Requirement 5 AC 1 and AC 5) stay identical for every existing test and sim-parity fixture, including bug-1882, bug-1767 and bug-1768. There is no per-plugin `commit` hook and no `sched_make_allocs()` retype (both left out of scope). The design must settle explicitly the seqjob/parjob divergences listed in `brief.md` → Constraints (the per-bucket `alloc_req.host` reset, decided 2026-09-29 as an unconditional reset for both allocators; seqjob's new OOM snapshot path, where `nslots`/`nhosts` are consumed, seqjob `accepts` for direct callers, and no VLA). Full context: `.kiro/specs/scheduler-allocator-runner/brief.md`.
## Introduction
The scheduler's two built-in allocator plugins, the sequential-job allocator (seqjob, jobs with exactly one task) and the parallel-job allocator (parjob, jobs with more than one task), each carry a private copy of the whole allocation algorithm. This feature moves that algorithm into the scheduler plugin framework. It becomes one framework-owned allocation routine, and an allocator plugin contributes only what is genuinely its own: the candidate-plan generator, an optional "is this job mine" guard, and its existing per-allocation preparation hook.
This is a behaviour-preserving refactor. For every job, the scheduler must reach the same allocation decision, choose the same hosts, debit the same resources, record the same main pend reason, and return the same result codes as it does today, in normal (live) mode and in simulation mode. The only intentional behaviour changes are the two hardening points in Requirement 5 (AC 1 and AC 5), which make formerly undefined sequential-allocator behaviour defined, the sequential-allocator host pend reasons of Requirement 3 AC 6 and AC 7, which become complete instead of depending on stale state, and the shared-resource pend reasons of Requirement 2 AC 2, which lose the spurious entries today's loop reads past the unmet list (bug-1893). For every input that reaches the allocators today, host choice, main pend reason, and result codes stay the same.
This is a behaviour-preserving refactor. For every job, the scheduler must reach the same allocation decision, choose the same hosts, debit the same resources, record the same main pend reason, and return the same result codes as it does today, in normal (live) mode and in simulation mode. The only intentional behaviour changes are the two hardening points in Requirement 5 (AC 1 and AC 5), which make formerly undefined sequential-allocator behaviour defined, the sequential-allocator host pend reasons of Requirement 3 AC 6 and AC 7, which become complete instead of depending on stale state, and the shared-resource pend reasons of Requirement 2 AC 2, which today are never recorded (bug-1893). For every input that reaches the allocators today, host choice, main pend reason, and result codes stay the same.
Terms:
- **Allocation attempt**: one call from the scheduler's allocation dispatch to an allocator for one primary resource request.
@ -41,7 +41,7 @@ Terms:
#### Acceptance Criteria
1. When an allocation attempt has passed the missing-input check and the acceptance guard, and the attempt has no pend-reason record yet, the Scheduler Plugin Framework shall create one before checking shared resources.
2. When the job's shared-resource request cannot be met, the Scheduler Plugin Framework shall add one "no resource" pend reason, with no host and naming the resource, for each entry of the unmet-resource list the shared check returns, and no other shared-resource reason. It shall then return the pending signal without trying any host. (Intentional change, bug-1893: today both allocators loop over the number of requested shared resources instead of unmet ones and add reasons naming resource index 0.)
2. When the job's shared-resource request cannot be met, the Scheduler Plugin Framework shall add one "no resource" pend reason, with no host and naming the resource, for each entry of the unmet-resource list the shared check returns, and no other shared-resource reason. It shall then return the pending signal without trying any host. (Intentional change, bug-1893: today both allocators skip every successful resource lookup, because the lookup returns 0 on success and the loop skips on a result of 0 or less, so no shared-resource reason is ever recorded; the loop also runs over the number of requested shared resources instead of the unmet ones.)
3. When the job's shared-resource request can be met, the Scheduler Plugin Framework shall count the requested shared amounts as uncommitted for the attempt before trying any plan. It shall count them once per attempt, whatever the number of plans or buckets tried, and shall not release them when the attempt then pends or fails, as today (the attempt context, which holds them, is discarded by its caller, or shared with the next primary of the same COMPOUND job or of the same ALTER alternative; each ALTER alternative gets a fresh context).
4. If creating the pend-reason record, reading the uncommitted shared state, checking shared satisfaction, or counting shared amounts as uncommitted fails, the Scheduler Plugin Framework shall return that failure's negative error code, with no allocation on the attempt.

9
.kiro/specs/scheduler-allocator-runner/tasks.md
View file

@ -14,7 +14,7 @@
- _Requirements: 1.2, 1.6_
- [ ] 2. Characterization tests against the current allocators
- [ ] 2.1 Characterize the sequential-job allocator through its registered entry point
- [x] 2.1 Characterize the sequential-job allocator through its registered entry point
- New test target that drives the real seqjob plugin through its job-allocation entry point, using the existing prepare-allocation stub convention and an allocation-record stub with out-of-memory injection
- Set up the out-of-memory injection that 2.2 reuses, using only injection points that stay the same across the switch: the existing allocation-record stub failing on its N-th call (per-host records in claim order, then shared records) and link-time wrapping (the testsuite's existing wrap precedent) of the libxce host-allocation and job-allocation constructors with fail-on-call-N counters; the host-array allocation is not injected here because its origin changes at the switch (seqjob's libxce helper and parjob's in-file allocation both become a runner allocation) and is covered by 3.3
- Fixtures: success; pending on a shared resource where the job requests exactly the shared resources that are short (so bug-1893's extra reasons cannot appear); pending on hosts with the pre-filter state arranged so no host is skipped on stale state (assert host reasons only where 3.7 does not change them, and the main reason); out of memory at the host allocation, the job allocation and the shared records (today's result code and allocation state; for the host-allocation case the per-host debit is asserted separately so 4.2 can change it); task counts 0 and 2 reaching the plugin directly; the second primary of a COMPOUND job with non-overlapping resources (bug-1892 not exercised)
@ -58,7 +58,7 @@
- [ ] 4. Move both allocators onto the framework routine
- [ ] 4.1 Switch the parallel-job allocator
- First add the parjob bug-1893 fixture (two shared resources, one short: only the short one is named) to the parjob characterization tests and see it fail against the old code
- First add the parjob bug-1893 fixture (two shared resources, one short: exactly one host-less reason, naming the short one; today none is recorded) to the parjob characterization tests and see it fail against the old code
- Give parjob its allocator contract with a task-count acceptance guard (more than one task), register the framework routine as its allocation entry point, and remove its private allocation routine and claim helpers
- Build the runner into the parjob characterization target (2.2) so it keeps reaching the plugin through its entry point
- Update the parjob combinatorial tests, the only permitted kind of change: build the runner into the white-box test, give their hand-built plugin structs the parjob allocator contract, and call the framework routine and the framework's claim search and rollback instead of the removed helpers
@ -67,7 +67,7 @@
- _Requirements: 1.1, 1.2, 1.5, 1.6, 2.2, 5.2, 7.1_
- [ ] 4.2 Switch the sequential-job allocator
- First add the seqjob intentional-change fixtures to the seqjob characterization tests and see them fail against the old code: a candidate host that the stale per-host request left by the job-level pass used to skip is now checked; complete per-host "no resource" reasons for every selection-matching short host with the main reason unchanged; bug-1893; release of the claimed host's debit when result building runs out of memory (this rewrites the 2.1 host-allocation out-of-memory fixture's per-host debit assertion to the new expected value)
- First add the seqjob intentional-change fixtures to the seqjob characterization tests and see them fail against the old code: a candidate host that the stale per-host request left by the job-level pass used to skip is now checked; complete per-host "no resource" reasons for every selection-matching short host with the main reason unchanged; bug-1893 (the 2.1 fixture that records no shared reason today now expects one host-less reason naming the short resource); release of the claimed host's debit when result building runs out of memory (this rewrites the 2.1 host-allocation out-of-memory fixture's per-host debit assertion to the new expected value)
- Give seqjob its allocator contract with no acceptance guard, register the framework routine as its allocation entry point, and remove its private allocation routine
- After the switch, add seqjob missing-input fixtures (invalid-argument, no pend-reason record)
- Build the runner into the seqjob characterization target (2.1)
@ -94,3 +94,6 @@
## Implementation Notes
- Build/test only in the dedicated dir `src/build.alloc` (`bash src/tools/xce-build.sh scheduler asan build.alloc --jobs N --incremental`; baseline 90/90). Another session uses `src/xce.build`.
- Several white-box scheduler test targets run with `detect_leaks=0` because their fixtures are not torn down (`testsuite/meson.build:1756`); a leak summary from running such a binary directly is pre-existing, not a regression. New tests that assert "no leak" (3.3) need leak detection on for their own target.
- bug-1893 (found in 2.1): `vget()` returns 0 on success, so today's shared-reason loop (`rc <= 0` skip) records NO shared reason at all. The runner must skip only on `rc < 0`. Characterization fixtures record "no shared reason today" in a separate `_today` test that 4.1/4.2 rewrite.
- `xce_sched_core_fini()` does not free the string table `xce_sched_core_init()` creates; leak-checked test targets call `xstring_table_fini()` in teardown (see `test_alloc_equivalence_seqjob.c`).
- `test_alloc_equivalence_seqjob.c` is at ~794/800 lines: 4.2's new fixtures go in a second file or after trimming.

2
.wolf/buglog.json
View file

@ -16558,7 +16558,7 @@
"timestamp": "2026-09-30T03:58:52Z",
"last_seen": "2026-09-30T03:58:52Z",
"occurrences": 1,
"error_message": "xce_scheduler allocators add spurious resource-0 shared pend reasons (loop reads past unmet list)\n\n**Observed:** found by code reading, so there is no failing run yet. When the shared-resource check\nfails, both allocators add the pend reasons with\n```c\nfor (i = 0; i < ctx->alloc_req.shared.count; i++) {\n rc = vget(core->resmgr, ((int*)ctx->unmeet.idxs.data)[i], (void**)&r);\n if (rc <= 0) continue;\n add_job_preason(reason->reasons, PREASON_NO_RESOURCE, NULL, r->name);\n}\n```\n(`seqjob_allocator.c` `my_alloc`, `parjob_allocator.c` `my_alloc`). The loop\nbound is the number of *requested* shared resources, but `ctx->unmeet` holds\nonly the *unmet* ones, packed at `[0, unmeet.count)` by `unsatisfy_()`\n(`resource.c:152-165`). Its arrays were `calloc`ed with capacity\n`shared.count` (`resource.c:280`), so entries `[unmeet.count, shared.count)`\nare 0.\n\n**Expected:** iterate `i < ctx->unmeet.count` (as the host path does,\n`resource.c:566`), giving one `NO_RESOURCE` reason per unmet shared resource.\n\n**Impact:** wrong pend reason, noise level. When a job requests N shared\nresources and only M < N are unmet, the loop adds N - M extra\n`PREASON_NO_RESOURCE` reasons naming whatever resource has index 0 (if one\nexists), which that job may never have requested. Main reason and allocation\ndecision are unaffected. No out-of-bounds memory access (the tail is inside the\nallocation).\n\n**Found by:** review of the `scheduler-allocator-runner` requirements\n(2026-09-29); the loop moves into the framework runner in that spec.",
"error_message": "xce_scheduler allocators never record shared-resource pend reasons (vget rc check inverted)\n\n**Observed:** found by code reading, so there is no failing run yet. When the shared-resource check\nfails, both allocators add the pend reasons with\n```c\nfor (i = 0; i < ctx->alloc_req.shared.count; i++) {\n rc = vget(core->resmgr, ((int*)ctx->unmeet.idxs.data)[i], (void**)&r);\n if (rc <= 0) continue;\n add_job_preason(reason->reasons, PREASON_NO_RESOURCE, NULL, r->name);\n}\n```\n(`seqjob_allocator.c` `my_alloc`, `parjob_allocator.c` `my_alloc`). The loop\nbound is the number of *requested* shared resources, but `ctx->unmeet` holds\nonly the *unmet* ones, packed at `[0, unmeet.count)` by `unsatisfy_()`\n(`resource.c:152-165`). Its arrays were `calloc`ed with capacity\n`shared.count` (`resource.c:280`), so entries `[unmeet.count, shared.count)`\nare 0.\n\n**Expected:** iterate `i < ctx->unmeet.count` (as the host path does,\n`resource.c:566`), giving one `NO_RESOURCE` reason per unmet shared resource.\n\n**Impact:** wrong pend reason, noise level. When a job requests N shared\nresources and only M < N are unmet, the loop adds N - M extra\n`PREASON_NO_RESOURCE` reasons naming whatever resource has index 0 (if one\nexists), which that job may never have requested. Main reason and allocation\ndecision are unaffected. No out-of-bounds memory access (the tail is inside the\nallocation).\n\n**Found by:** review of the `scheduler-allocator-runner` requirements\n(2026-09-29); the loop moves into the framework runner in that spec.\n\nUPDATE (2026-09-30): Found in scheduler-allocator-runner task 2.1 (characterization test test_seqjob_pending_on_short_shared_no_reason_today): the observable effect is not an extra resource-0 reason. vget() (xzarray_get_vec, src/lib/zarray.c) returns 0 on success, and the loop skips on rc <= 0, so it skips every successful lookup and NO shared NO_RESOURCE reason is ever recorded; the over-read tail entries (idx 0) are skipped the same way. Real impact: a job pending on a short shared resource carries no resource reason naming it (main_reason stays 0). The host path (resource.c) correctly skips only rc < 0.",
"file": "src/scheduler/src/plugins/seqjob_allocator.c, src/scheduler/src/plugins/parjob_allocator.c",
"root_cause": "TODO: required for fixed/partial",
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",

4
.wolf/bugs/bug-1893.md
View file

@ -13,7 +13,7 @@ tags: ["scheduler", "allocation", "preason", "code-review"]
related: ["bug-1892"]
---
# bug-1893: xce_scheduler allocators add spurious resource-0 shared pend reasons (loop reads past unmet list)
# bug-1893: xce_scheduler allocators never record shared-resource pend reasons (vget rc check inverted)
## Description
**Observed:** found by code reading, so there is no failing run yet. When the shared-resource check
@ -45,6 +45,8 @@ allocation).
**Found by:** review of the `scheduler-allocator-runner` requirements
(2026-09-29); the loop moves into the framework runner in that spec.
UPDATE (2026-09-30): Found in scheduler-allocator-runner task 2.1 (characterization test test_seqjob_pending_on_short_shared_no_reason_today): the observable effect is not an extra resource-0 reason. vget() (xzarray_get_vec, src/lib/zarray.c) returns 0 on success, and the loop skips on rc <= 0, so it skips every successful lookup and NO shared NO_RESOURCE reason is ever recorded; the over-read tail entries (idx 0) are skipped the same way. Real impact: a job pending on a short shared resource carries no resource reason naming it (main_reason stays 0). The host path (resource.c) correctly skips only rc < 0.
## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)

18
src/scheduler/testsuite/meson.build
View file

@ -1714,6 +1714,21 @@ sched_seqjob_allocator_test = executable(
dependencies: [scheduler_test_deps],
install: false)
# scheduler-allocator-runner task 2.1 (Req 4.5, 5.4, 5.8, 7.2): seqjob
# characterization through the plugin's registered job_alloc_fn, against
# today's allocator and (from task 4.2) the framework runner. Same white-box
# stack as sched_seqjob_allocator_test. `--wrap` on libxce's
# make_host_alloc/make_job_alloc gives the test fail-on-call-N out-of-memory
# injection points that stay the same across the switch.
sched_alloc_equivalence_seqjob_test = executable(
'sched_alloc_equivalence_seqjob_test',
['test_alloc_equivalence_seqjob.c'],
include_directories:[incdir],
dependencies: [scheduler_test_deps],
link_args: ['-Wl,--wrap=make_host_alloc',
'-Wl,--wrap=make_job_alloc'],
install: false)
# Task 3.2 (scheduler-allocation spec, Req 9.1): the dedicated pre/post-
# rewrap host-selection EQUIVALENCE proof for seqjob_allocator.c's
# my_alloc() -- fixture states task 3.1's own smoke test does not cover
@ -1980,6 +1995,9 @@ test('scheduler sim cache clear isolation test',
test('scheduler seqjob allocator test',
sched_seqjob_allocator_test, suite: 'scheduler',
env: ['ASAN_OPTIONS=detect_leaks=0'])
# Leak detection stays on: every fixture tears down its ctx, core and plugin.
test('scheduler alloc equivalence seqjob test',
sched_alloc_equivalence_seqjob_test, suite: 'scheduler')
test('scheduler seqjob rewrap equivalence test',
sched_seqjob_rewrap_equivalence_test, suite: 'scheduler',
env: ['ASAN_OPTIONS=detect_leaks=0'])

794
src/scheduler/testsuite/test_alloc_equivalence_seqjob.c Normal file
View file

@ -0,0 +1,794 @@
/*
* test_alloc_equivalence_seqjob.c - scheduler-allocator-runner task 2.1:
* characterization of the sequential-job allocator (Req 4.5, 5.4, 5.8,
* 7.2), driven only through the registered plugin->job_alloc_fn of
* seqjob_allocator_new_plugin(), never the static routine by name. The
* expectations must still hold once task 4.2 replaces that routine with the
* framework runner (one more white-box include below). Each fixture asserts
* result code, hosts, per-host records, shared records, per-host and shared
* uncommitted amounts and pend reasons.
*
* Out-of-memory injection (task 2.2 copies it), only at points that stay
* the same across the switch: the sched_make_allocs() stub failing on call
* N (1 = claimed host's records, 2 = shared records) and -Wl,--wrap of
* libxce make_host_alloc/make_job_alloc (meson.build) failing on call N.
* The host-array allocation is not injected (its origin changes; task 3.3).
*
* Kept out on purpose: multi-host pending (Req 3.7), more shared requests
* than short ones (bug-1893), COMPOUND layouts where the positional
* uncommitted read (bug-1892) matters. Values the switch changes on purpose
* (Req 5.5 host-stage debit, Req 2.2 shared reasons) sit in their own
* `_today` test functions.
*/
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <unity/unity.h>
#include "preason.h"
#include "resobj.h"
#include "resreq.h"
#include "sched/alloc_iter.h"
#include "scheduler.h"
#include "xcealloc.h"
#include "xstring.h"
/* White-box sources under test. Task 4.2 adds the framework runner
* ("../src/alloc_runner.c") as one more line of this list. */
#include "../src/core.c"
#include "../src/resource.c"
#include "../src/schedobj.c"
#include "../src/host_iter.c"
#include "../src/plugins/seqjob_allocator.c"
/* ---------------------------------------------------------------------- */
/* Link-satisfying stubs (same convention as test_seqjob_allocator.c) */
/* ---------------------------------------------------------------------- */
int plugin_register(void* sched, void* plugin)
{
(void)sched;
(void)plugin;
return 0;
}
void plugin_unregister(void* arg) { (void)arg; }
/* Stands in for plugin.c's fan-out with exactly one plugin (seqjob's own
* prepare hook): reset the request vectors exactly as plugin.c's
* plugin_prepare_allocation() does (job-level pass: .shared and .host,
* bug-1883; per-host pass: .host), then call the hook. */
int plugin_prepare_allocation(xce_sched_core_t* sched, xce_sched_ctx_t* ctx)
{
xce_sched_plugin_t fake_plugin;
if (ctx->worker == NULL) {
xvec_idval_fini(&(ctx->alloc_req.shared));
xvec_idval_init(&(ctx->alloc_req.shared), 8);
xvec_idval_fini(&(ctx->alloc_req.host));
xvec_idval_init(&(ctx->alloc_req.host), 8);
} else {
xvec_idval_fini(&(ctx->alloc_req.host));
xvec_idval_init(&(ctx->alloc_req.host), 8);
}
memset(&fake_plugin, 0, sizeof(fake_plugin));
fake_plugin.sched = sched;
fake_plugin.core = sched;
return seqjob_prepare_allocation(&fake_plugin, ctx);
}
int plugin_job_update(void* s, xce_sched_job_t* j)
{
return (void)s, (void)j, 0;
}
int plugin_queue_update(void* s, xce_sched_queue_t* q)
{
return (void)s, (void)q, 0;
}
int plugin_job_rusage_update(void* s, xce_sched_job_t* j, xce_job_rusages_t* r)
{
return (void)s, (void)j, (void)r, 0;
}
/* ---------------------------------------------------------------------- */
/* Out-of-memory injection */
/* ---------------------------------------------------------------------- */
/* Each counter counts every call; a fail_on value N > 0 makes call N fail
* (NULL, errno ENOMEM). 0 disarms. oom_injection_reset() clears all. */
static int g_make_allocs_calls = 0;
static int g_make_allocs_fail_on = 0;
static int g_host_alloc_calls = 0;
static int g_host_alloc_fail_on = 0;
static int g_job_alloc_calls = 0;
static int g_job_alloc_fail_on = 0;
static void oom_injection_reset(void)
{
g_make_allocs_calls = 0;
g_make_allocs_fail_on = 0;
g_host_alloc_calls = 0;
g_host_alloc_fail_on = 0;
g_job_alloc_calls = 0;
g_job_alloc_fail_on = 0;
}
/* Stub for scheduler.c's sched_make_allocs(): builds real records with
* make_alloc() (resource name from the resmgr, the requested value, empty
* reservation name, duration -1 as for a NULL duration array), and
* returns NULL for nres <= 0 as the real one does. instid is 0. */
xce_alloc_t** sched_make_allocs(xce_scheduler_t* sched, char* reserve,
int nres, int* idxs, float* vals,
int* duration)
{
xce_alloc_t** allocs;
int i;
g_make_allocs_calls++;
if ((g_make_allocs_fail_on > 0)
&& (g_make_allocs_calls == g_make_allocs_fail_on)) {
errno = ENOMEM;
return NULL;
}
if ((sched == NULL) || (nres <= 0) || (idxs == NULL) || (vals == NULL)) {
errno = EINVAL;
return NULL;
}
allocs = (xce_alloc_t**)calloc(nres, sizeof(*allocs));
if (allocs == NULL) {
return NULL;
}
for (i = 0; i < nres; i++) {
xce_resource_t* r = NULL;
vget(sched->core.resmgr, idxs[i], (void**)&r);
if (r == NULL) {
continue;
}
allocs[i] = make_alloc(r->name, reserve, 0, vals[i],
(duration != NULL) ? duration[i] : -1);
if (allocs[i] == NULL) {
free_ResAlloc_list(nres, allocs);
errno = ENOMEM;
return NULL;
}
}
return allocs;
}
xce_host_alloc_t* __real_make_host_alloc(char*, int, xce_alloc_t**);
xce_job_alloc_t* __real_make_job_alloc(uint64_t, int, xce_host_alloc_t**);
xce_host_alloc_t* __wrap_make_host_alloc(char* host, int nalloc,
xce_alloc_t** allocs)
{
g_host_alloc_calls++;
if ((g_host_alloc_fail_on > 0)
&& (g_host_alloc_calls == g_host_alloc_fail_on)) {
errno = ENOMEM;
return NULL;
}
return __real_make_host_alloc(host, nalloc, allocs);
}
xce_job_alloc_t* __wrap_make_job_alloc(uint64_t jobid, int nhost,
xce_host_alloc_t** hallocs)
{
g_job_alloc_calls++;
if ((g_job_alloc_fail_on > 0)
&& (g_job_alloc_calls == g_job_alloc_fail_on)) {
errno = ENOMEM;
return NULL;
}
return __real_make_job_alloc(jobid, nhost, hallocs);
}
void setUp(void) { oom_injection_reset(); }
void tearDown(void) { oom_injection_reset(); }
/* ---------------------------------------------------------------------- */
/* Fixture */
/* ---------------------------------------------------------------------- */
#define HOST_A "eq-host-a"
#define HOST_B "eq-host-b"
typedef struct {
xce_scheduler_t sched;
xce_resource_t* slots;
xce_resource_t* mem;
xce_resource_t* rack;
xce_resource_t* lic; /* shared (XCE_RESF_GLOBAL) */
xce_sched_plugin_t* plugin;
xce_sched_obj_t obj;
xce_primary_schedobj_t primary_obj;
xce_sched_ctx_t ctx;
int ctx_ready;
} eq_fixture_t;
/* One primary request: `mem` per task, `lic` per job (0 = not requested),
* optional `rack == N` select. */
typedef struct {
xce_primary_resreq_t req;
rusage_value_t mem_rusage;
rusage_value_t lic_rusage;
xce_select_t select;
xce_select_node_t sel_res, sel_num, sel_eq;
} eq_primary_t;
static xce_resource_t* eq_register_resource(var resmgr, const char* name,
int flags)
{
xce_resource_t res;
memset(&res, 0, sizeof(res));
res.name = (char*)name;
res.flags = flags;
return makeResource(resmgr, &res);
}
/* Shared pool "licenses" has `lic_total` capacity, none in use. */
static void eq_fixture_init(eq_fixture_t* f, float lic_total)
{
int idx[1];
float val[1];
memset(f, 0, sizeof(*f));
TEST_ASSERT_EQUAL_INT(0, xce_sched_core_init(&(f->sched.core)));
/* xce_sched_core_fini() finalizes these; both drivers calloc them */
for (idx[0] = 0; idx[0] < 2; idx[0]++) {
f->sched.core.pend_updates[idx[0]]
= (dataq_t*)calloc(1, sizeof(dataq_t));
TEST_ASSERT_NOT_NULL(f->sched.core.pend_updates[idx[0]]);
dataq_init(f->sched.core.pend_updates[idx[0]], NULL);
}
f->slots = eq_register_resource(f->sched.core.resmgr, "slots", 0);
f->mem = eq_register_resource(f->sched.core.resmgr, "mem", 0);
f->rack = eq_register_resource(f->sched.core.resmgr, "rack", 0);
f->lic = eq_register_resource(f->sched.core.resmgr, "licenses",
XCE_RESF_GLOBAL);
TEST_ASSERT_NOT_NULL(f->slots);
TEST_ASSERT_NOT_NULL(f->mem);
TEST_ASSERT_NOT_NULL(f->rack);
TEST_ASSERT_NOT_NULL(f->lic);
idx[0] = f->lic->idx;
val[0] = lic_total;
put_resvals(&(f->sched.core.sharedInst), 1, idx, val);
f->plugin = seqjob_allocator_new_plugin();
TEST_ASSERT_NOT_NULL(f->plugin);
TEST_ASSERT_NOT_NULL(f->plugin->job_alloc_fn);
f->plugin->sched = &(f->sched);
f->plugin->core = &(f->sched.core);
}
static xce_sched_worker_t* eq_add_host(eq_fixture_t* f, const char* name,
float slots, float mem, float rack)
{
xce_sched_worker_t* w;
int idx[3] = { f->slots->idx, f->mem->idx, f->rack->idx };
float total[3] = { slots, mem, rack };
float inuse[2] = { 0.0f, 0.0f };
w = (xce_sched_worker_t*)calloc(1, sizeof(*w));
TEST_ASSERT_NOT_NULL(w);
w->name = makeString((char*)name);
TEST_ASSERT_NOT_NULL(w->name);
zarray_init(&(w->resvals), 8, (zarray_item_killer_t)kill_sched_resval);
zarray_init(&(w->resInUse), 8, (zarray_item_killer_t)kill_sched_resval);
put_resvals(&(w->resvals), 3, idx, total);
put_resvals(&(w->resInUse), 2, idx, inuse);
TEST_ASSERT_TRUE(sched_add_worker(&(f->sched.core), w) >= 0);
return w;
}
static void eq_primary_init(eq_primary_t* p, int max_tasks, float mem,
float lic, const eq_fixture_t* f, int rack)
{
memset(p, 0, sizeof(*p));
p->req.minTasks = (max_tasks > 0) ? 1 : 0;
p->req.maxTasks = max_tasks;
xlist_init(&(p->req.rusages), 4);
if (mem > 0.0f) {
p->mem_rusage.res = (char*)"mem";
p->mem_rusage.value = mem;
p->mem_rusage.per_flag = PER_TASK;
xlist_store(&(p->req.rusages), &(p->mem_rusage));
}
if (lic > 0.0f) {
p->lic_rusage.res = (char*)"licenses";
p->lic_rusage.value = lic;
p->lic_rusage.per_flag = PER_JOB;
xlist_store(&(p->req.rusages), &(p->lic_rusage));
}
if (rack > 0) {
p->sel_res.type = SEL_RESNAME;
p->sel_res.u.resname.res_idx = f->rack->idx;
p->sel_res.u.resname.res_type = RES_TYPE_NUMERIC;
p->sel_num.type = SEL_NUM;
p->sel_num.u.numval = (double)rack;
p->sel_eq.type = SEL_EQ;
p->sel_eq.u.binop.left = &(p->sel_res);
p->sel_eq.u.binop.right = &(p->sel_num);
p->select.root = &(p->sel_eq);
p->req.select = &(p->select);
}
}
/* ctx over a PRIMARY obj for `p`, job-level prepare pass included
* (sched_ctx_set_primary() -> plugin_prepare_allocation(), ctx->worker
* NULL), exactly as sched_make_primary_allocation() drives it. */
static void eq_ctx_begin_primary(eq_fixture_t* f, eq_primary_t* p)
{
f->primary_obj.resreq = &(p->req);
f->obj.tag = XCE_RESREQ_PRIMARY;
f->obj.u.primary = &(f->primary_obj);
TEST_ASSERT_EQUAL_INT(0, sched_ctx_init(&(f->ctx), &(f->sched.core),
&(f->obj)));
f->ctx_ready = 1;
sched_ctx_set_primary(&(f->ctx), &(p->req));
}
static int eq_alloc(eq_fixture_t* f)
{
return f->plugin->job_alloc_fn(f->plugin, &(f->ctx));
}
static void eq_fixture_fini(eq_fixture_t* f)
{
if (f->ctx_ready) {
free_JobAlloc_ptr(&(f->ctx.alloc));
free_PendReason_ptr(&(f->ctx.reason));
sched_ctx_fini(&(f->ctx));
}
f->plugin->fini_fn(f->plugin);
xce_sched_core_fini(&(f->sched.core));
/* xce_sched_core_init() creates the process-wide string table on every
* call and xce_sched_core_fini() leaves it; release it per fixture */
xstring_table_fini();
}
/* ---------------------------------------------------------------------- */
/* Assertion helpers */
/* ---------------------------------------------------------------------- */
static float eq_host_uncomit(eq_fixture_t* f, xce_sched_worker_t* w,
xce_resource_t* r)
{
xvec_t* v = (xvec_t*)zarray_get(&(f->ctx.uncomitted.hosts), w->idx);
return (v == NULL) ? 0.0f : ((float*)v->data)[r->idx];
}
static float eq_shared_uncomit(eq_fixture_t* f)
{
return ((float*)f->ctx.uncomitted.shared->data)[f->lic->idx];
}
static void eq_assert_record(xce_alloc_t* a, const char* resname, float value)
{
TEST_ASSERT_NOT_NULL(a);
TEST_ASSERT_EQUAL_STRING(resname, a->resname);
TEST_ASSERT_EQUAL_STRING("", a->reserve);
TEST_ASSERT_EQUAL_FLOAT(value, a->value);
TEST_ASSERT_EQUAL_INT(-1, a->duration);
}
/* One host allocation carrying mem then slots (seqjob's per-host prepare
* computes the rusage wants first, then stores slots). */
static void eq_assert_host_mem_slots(xce_host_alloc_t* ha, const char* host,
float mem)
{
TEST_ASSERT_NOT_NULL(ha);
TEST_ASSERT_EQUAL_STRING(host, ha->host);
TEST_ASSERT_EQUAL_INT(2, ha->nallocs);
TEST_ASSERT_NOT_NULL(ha->allocs);
eq_assert_record(ha->allocs[0], "mem", mem);
eq_assert_record(ha->allocs[1], "slots", 1.0f);
}
static void eq_assert_reason_counts(eq_fixture_t* f, int main_reason,
int nhost, int nres)
{
TEST_ASSERT_NOT_NULL(f->ctx.reason);
TEST_ASSERT_NOT_NULL(f->ctx.reason->reasons);
TEST_ASSERT_EQUAL_INT(main_reason, f->ctx.reason->reasons->main_reason);
TEST_ASSERT_EQUAL_INT(nhost, f->ctx.reason->reasons->nhost_reasons);
TEST_ASSERT_EQUAL_INT(nres, f->ctx.reason->reasons->nres_reasons);
}
static void eq_assert_res_reason(eq_fixture_t* f, int i, const char* host,
const char* res)
{
xce_res_reason_t* rr = f->ctx.reason->reasons->res_based_reasons[i];
TEST_ASSERT_NOT_NULL(rr);
TEST_ASSERT_EQUAL_STRING(host, rr->host);
TEST_ASSERT_EQUAL_STRING(res, rr->resource);
TEST_ASSERT_EQUAL_INT(PREASON_NO_RESOURCE, rr->reason);
}
static void eq_primary_fini(eq_primary_t* p) { xlist_fini(&(p->req.rusages)); }
static void eq_assert_calls(int make_allocs, int host_alloc, int job_alloc)
{
TEST_ASSERT_EQUAL_INT(make_allocs, g_make_allocs_calls);
TEST_ASSERT_EQUAL_INT(host_alloc, g_host_alloc_calls);
TEST_ASSERT_EQUAL_INT(job_alloc, g_job_alloc_calls);
}
/* The standard satisfiable job: one host (4 slots, 8 mem, rack 1), 10
* shared licenses; the job wants 2 mem per task and 3 licenses per job.
* Returns the job_alloc_fn result; the fixture is left for the caller. */
static int eq_run_standard(eq_fixture_t* f, eq_primary_t* p,
xce_sched_worker_t** host)
{
eq_fixture_init(f, 10.0f);
*host = eq_add_host(f, HOST_A, 4.0f, 8.0f, 1.0f);
eq_primary_init(p, 1, 2.0f, 3.0f, f, 0);
eq_ctx_begin_primary(f, p);
return eq_alloc(f);
}
static void eq_finish(eq_fixture_t* f, eq_primary_t* p)
{
eq_fixture_fini(f);
eq_primary_fini(p);
}
/* ---------------------------------------------------------------------- */
/* Fixtures */
/* ---------------------------------------------------------------------- */
/* 1. Success: one host allocation (mem, slots) and one shared record. */
static void test_seqjob_success_one_host_with_shared(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* a;
TEST_ASSERT_EQUAL_INT(1, eq_run_standard(&f, &p, &a));
TEST_ASSERT_NOT_NULL(f.ctx.alloc);
TEST_ASSERT_EQUAL_UINT64(0, f.ctx.alloc->jobid);
TEST_ASSERT_EQUAL_INT(1, f.ctx.alloc->nhosts);
eq_assert_host_mem_slots(f.ctx.alloc->allocs[0], HOST_A, 2.0f);
TEST_ASSERT_EQUAL_INT(1, f.ctx.alloc->nres_shared);
eq_assert_record(f.ctx.alloc->allocs_shared[0], "licenses", 3.0f);
TEST_ASSERT_EQUAL_FLOAT(2.0f, eq_host_uncomit(&f, a, f.mem));
TEST_ASSERT_EQUAL_FLOAT(1.0f, eq_host_uncomit(&f, a, f.slots));
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
eq_assert_reason_counts(&f, 0, 0, 0);
eq_assert_calls(2, 1, 1);
eq_finish(&f, &p);
}
/* 2. Pending on the one shared resource the job requests, which is short
* (2 licenses, 3 wanted): no host is tried and nothing is counted. */
static int eq_run_short_shared(eq_fixture_t* f, eq_primary_t* p,
xce_sched_worker_t** host)
{
eq_fixture_init(f, 2.0f);
*host = eq_add_host(f, HOST_A, 4.0f, 8.0f, 1.0f);
eq_primary_init(p, 1, 2.0f, 3.0f, f, 0);
eq_ctx_begin_primary(f, p);
return eq_alloc(f);
}
static void test_seqjob_pending_on_short_shared_resource(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* a;
TEST_ASSERT_EQUAL_INT(0, eq_run_short_shared(&f, &p, &a));
TEST_ASSERT_NULL(f.ctx.alloc);
TEST_ASSERT_NOT_NULL(f.ctx.reason);
TEST_ASSERT_EQUAL_INT(0, f.ctx.reason->reasons->main_reason);
TEST_ASSERT_EQUAL_INT(0, f.ctx.reason->reasons->nhost_reasons);
TEST_ASSERT_NULL(zarray_get(&(f.ctx.uncomitted.hosts), a->idx));
TEST_ASSERT_EQUAL_FLOAT(0.0f, eq_shared_uncomit(&f));
eq_assert_calls(0, 0, 0);
eq_finish(&f, &p);
}
/* 2, resource reasons, TODAY's behaviour: none. The reason loop skips
* every unmet entry because vget() returns 0 on success and the loop
* tests `rc <= 0`. Requirement 2.2 changes this in task 4.2 (one host-less
* NO_RESOURCE reason naming "licenses"). */
static void test_seqjob_pending_on_short_shared_no_reason_today(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* a;
TEST_ASSERT_EQUAL_INT(0, eq_run_short_shared(&f, &p, &a));
eq_assert_reason_counts(&f, 0, 0, 0);
eq_finish(&f, &p);
}
/* 3. Pending on hosts: the only candidate has a free slot but too little
* mem. The pre-filter state (slots only) lets it through, and the per-
* host check records NO_RESOURCE for (host, mem). The met shared request
* stays counted as uncommitted. */
static void test_seqjob_pending_single_host_short_of_mem(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* a;
eq_fixture_init(&f, 10.0f);
a = eq_add_host(&f, HOST_A, 4.0f, 1.0f, 1.0f);
eq_primary_init(&p, 1, 4.0f, 3.0f, &f, 0);
eq_ctx_begin_primary(&f, &p);
TEST_ASSERT_EQUAL_INT(0, eq_alloc(&f));
TEST_ASSERT_NULL(f.ctx.alloc);
eq_assert_reason_counts(&f, 0, 0, 1);
eq_assert_res_reason(&f, 0, HOST_A, "mem");
TEST_ASSERT_EQUAL_FLOAT(0.0f, eq_host_uncomit(&f, a, f.mem));
TEST_ASSERT_EQUAL_FLOAT(0.0f, eq_host_uncomit(&f, a, f.slots));
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
eq_assert_calls(0, 0, 0);
eq_finish(&f, &p);
}
/* 4a. Out of memory building the claimed host's allocation records
* (sched_make_allocs() call 1). Per-host debit: see the _debit_today test. */
static void test_seqjob_oom_host_records_returns_enomem(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* a;
g_make_allocs_fail_on = 1;
TEST_ASSERT_EQUAL_INT(-ENOMEM, eq_run_standard(&f, &p, &a));
TEST_ASSERT_NULL(f.ctx.alloc);
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
eq_assert_reason_counts(&f, 0, 0, 0);
eq_assert_calls(1, 0, 0);
eq_finish(&f, &p);
}
/* 4b. Out of memory in make_host_alloc(). Per-host debit: see the
* _debit_today test. */
static void test_seqjob_oom_host_alloc_returns_enomem(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* a;
g_host_alloc_fail_on = 1;
TEST_ASSERT_EQUAL_INT(-ENOMEM, eq_run_standard(&f, &p, &a));
TEST_ASSERT_NULL(f.ctx.alloc);
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
eq_assert_reason_counts(&f, 0, 0, 0);
eq_assert_calls(1, 1, 0);
eq_finish(&f, &p);
}
/* 4a/4b per-host debit, TODAY's behaviour: the claimed host's debit is
* left in the attempt context. Requirement 5.5 changes this in task 4.2
* (the debit of a claim not turned into a host allocation is released). */
static void test_seqjob_oom_host_stage_leaves_debit_today(void)
{
int stage;
for (stage = 0; stage < 2; stage++) {
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* a;
oom_injection_reset();
if (stage == 0) {
g_make_allocs_fail_on = 1; /* host allocation records */
} else {
g_host_alloc_fail_on = 1; /* make_host_alloc() */
}
TEST_ASSERT_EQUAL_INT(-ENOMEM, eq_run_standard(&f, &p, &a));
TEST_ASSERT_EQUAL_FLOAT(2.0f, eq_host_uncomit(&f, a, f.mem));
TEST_ASSERT_EQUAL_FLOAT(1.0f, eq_host_uncomit(&f, a, f.slots));
eq_finish(&f, &p);
}
}
/* 4c. Out of memory in make_job_alloc(); the debit stays (also after the
* switch: the host allocation was built). */
static void test_seqjob_oom_job_alloc_returns_enomem_debit_stays(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* a;
g_job_alloc_fail_on = 1;
TEST_ASSERT_EQUAL_INT(-ENOMEM, eq_run_standard(&f, &p, &a));
TEST_ASSERT_NULL(f.ctx.alloc);
TEST_ASSERT_EQUAL_FLOAT(2.0f, eq_host_uncomit(&f, a, f.mem));
TEST_ASSERT_EQUAL_FLOAT(1.0f, eq_host_uncomit(&f, a, f.slots));
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
eq_assert_reason_counts(&f, 0, 0, 0);
eq_assert_calls(1, 1, 1);
eq_finish(&f, &p);
}
/* 4d. Out of memory building the shared records (sched_make_allocs()
* call 2); the debit stays. */
static void test_seqjob_oom_shared_records_returns_enomem_debit_stays(void)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* a;
g_make_allocs_fail_on = 2;
TEST_ASSERT_EQUAL_INT(-ENOMEM, eq_run_standard(&f, &p, &a));
TEST_ASSERT_NULL(f.ctx.alloc);
TEST_ASSERT_EQUAL_FLOAT(2.0f, eq_host_uncomit(&f, a, f.mem));
TEST_ASSERT_EQUAL_FLOAT(1.0f, eq_host_uncomit(&f, a, f.slots));
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
eq_assert_reason_counts(&f, 0, 0, 0);
eq_assert_calls(2, 1, 1);
eq_finish(&f, &p);
}
/* 5. A task count other than 1 reaching seqjob directly: its prepare hook
* is a no-op for maxTasks != 1, so both request vectors stay empty and the
* job is placed on the first host with no resource entries at all. */
static void eq_check_task_count(int max_tasks)
{
eq_fixture_t f;
eq_primary_t p;
xce_sched_worker_t* a;
eq_fixture_init(&f, 10.0f);
a = eq_add_host(&f, HOST_A, 4.0f, 8.0f, 1.0f);
eq_primary_init(&p, max_tasks, 2.0f, 3.0f, &f, 0);
eq_ctx_begin_primary(&f, &p);
TEST_ASSERT_EQUAL_INT(1, eq_alloc(&f));
TEST_ASSERT_NOT_NULL(f.ctx.alloc);
TEST_ASSERT_EQUAL_INT(1, f.ctx.alloc->nhosts);
TEST_ASSERT_NOT_NULL(f.ctx.alloc->allocs[0]);
TEST_ASSERT_EQUAL_STRING(HOST_A, f.ctx.alloc->allocs[0]->host);
TEST_ASSERT_EQUAL_INT(0, f.ctx.alloc->allocs[0]->nallocs);
TEST_ASSERT_NULL(f.ctx.alloc->allocs[0]->allocs);
TEST_ASSERT_EQUAL_INT(0, f.ctx.alloc->nres_shared);
TEST_ASSERT_NULL(f.ctx.alloc->allocs_shared);
TEST_ASSERT_EQUAL_FLOAT(0.0f, eq_host_uncomit(&f, a, f.mem));
TEST_ASSERT_EQUAL_FLOAT(0.0f, eq_host_uncomit(&f, a, f.slots));
TEST_ASSERT_EQUAL_FLOAT(0.0f, eq_shared_uncomit(&f));
eq_assert_reason_counts(&f, 0, 0, 0);
eq_assert_calls(1, 1, 1);
eq_finish(&f, &p);
}
static void test_seqjob_task_count_zero_reaches_plugin(void)
{
eq_check_task_count(0);
}
static void test_seqjob_task_count_two_reaches_plugin(void)
{
eq_check_task_count(2);
}
/* 6. COMPOUND: one ctx, two primaries allocated in turn as
* sched_make_compound_alloc_() does. The selects (rack 1 / rack 2) keep
* the second primary off the host the first claimed, so the positional
* uncommitted read (bug-1892) cannot touch the expected values. Each
* primary is sized from its own request (Req 5.8, bug-1768). */
static void test_seqjob_compound_second_primary_sized_from_own_request(void)
{
eq_fixture_t f;
eq_primary_t p1;
eq_primary_t p2;
xce_compound_resreq_t cres;
xce_compound_schedobj_t cobj;
xce_job_alloc_t* first = NULL;
xce_sched_worker_t* a;
xce_sched_worker_t* b;
eq_fixture_init(&f, 10.0f);
a = eq_add_host(&f, HOST_A, 1.0f, 8.0f, 1.0f);
b = eq_add_host(&f, HOST_B, 1.0f, 8.0f, 2.0f);
eq_primary_init(&p1, 1, 2.0f, 3.0f, &f, 1);
eq_primary_init(&p2, 1, 3.0f, 0.0f, &f, 2);
xlist_init(&(cres.primaries), 2);
xlist_store(&(cres.primaries), &(p1.req));
xlist_store(&(cres.primaries), &(p2.req));
memset(&cobj, 0, sizeof(cobj));
cobj.owner = &(f.obj);
cobj.resreq = &cres;
f.obj.tag = XCE_RESREQ_COMPOUND;
f.obj.u.compound = &cobj;
TEST_ASSERT_EQUAL_INT(0, sched_ctx_init(&(f.ctx), &(f.sched.core),
&(f.obj)));
f.ctx_ready = 1;
sched_ctx_set_primary(&(f.ctx), &(p1.req));
TEST_ASSERT_EQUAL_INT(1, eq_alloc(&f));
MOVE_PTR(first, f.ctx.alloc);
sched_ctx_set_primary(&(f.ctx), &(p2.req));
TEST_ASSERT_EQUAL_INT(1, eq_alloc(&f));
TEST_ASSERT_NOT_NULL(first);
TEST_ASSERT_EQUAL_INT(1, first->nhosts);
eq_assert_host_mem_slots(first->allocs[0], HOST_A, 2.0f);
TEST_ASSERT_EQUAL_INT(1, first->nres_shared);
eq_assert_record(first->allocs_shared[0], "licenses", 3.0f);
TEST_ASSERT_NOT_NULL(f.ctx.alloc);
TEST_ASSERT_EQUAL_INT(1, f.ctx.alloc->nhosts);
eq_assert_host_mem_slots(f.ctx.alloc->allocs[0], HOST_B, 3.0f);
TEST_ASSERT_EQUAL_INT(0, f.ctx.alloc->nres_shared);
TEST_ASSERT_NULL(f.ctx.alloc->allocs_shared);
TEST_ASSERT_EQUAL_FLOAT(2.0f, eq_host_uncomit(&f, a, f.mem));
TEST_ASSERT_EQUAL_FLOAT(1.0f, eq_host_uncomit(&f, a, f.slots));
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_host_uncomit(&f, b, f.mem));
TEST_ASSERT_EQUAL_FLOAT(1.0f, eq_host_uncomit(&f, b, f.slots));
TEST_ASSERT_EQUAL_FLOAT(3.0f, eq_shared_uncomit(&f));
eq_assert_reason_counts(&f, 0, 0, 0);
eq_assert_calls(3, 2, 2);
free_JobAlloc_ptr(&first);
xlist_fini(&(cres.primaries));
eq_fixture_fini(&f);
eq_primary_fini(&p1);
eq_primary_fini(&p2);
}
int main(void)
{
UNITY_BEGIN();
RUN_TEST(test_seqjob_success_one_host_with_shared);
RUN_TEST(test_seqjob_pending_on_short_shared_resource);
RUN_TEST(test_seqjob_pending_on_short_shared_no_reason_today);
RUN_TEST(test_seqjob_pending_single_host_short_of_mem);
RUN_TEST(test_seqjob_oom_host_records_returns_enomem);
RUN_TEST(test_seqjob_oom_host_alloc_returns_enomem);
RUN_TEST(test_seqjob_oom_host_stage_leaves_debit_today);
RUN_TEST(test_seqjob_oom_job_alloc_returns_enomem_debit_stays);
RUN_TEST(test_seqjob_oom_shared_records_returns_enomem_debit_stays);
RUN_TEST(test_seqjob_task_count_zero_reaches_plugin);
RUN_TEST(test_seqjob_task_count_two_reaches_plugin);
RUN_TEST(test_seqjob_compound_second_primary_sized_from_own_request);
return UNITY_END();
}