83 lines
16 KiB
Markdown
83 lines
16 KiB
Markdown
# Requirements Document
|
|
|
|
## Introduction
|
|
|
|
Engineers maintaining the XCE scheduler who need to determine whether two scheduler allocation or pending-reason objects represent the same decision currently have no shared way to do so, and when two objects differ nothing tells them *where*. The structural equality comparators for the shared data structures `xce_alloc_t`, `xce_host_alloc_t`, and `xce_job_alloc_t` (all defined in libxce's `src/include/xce/types.h`) live as `static` functions inside the scheduler's simulation-verification module `src/scheduler/src/sim_verify.c`, written for `--sim-verify` replay checking, beside a pending-reason comparator for `xce_job_preason_t` that deliberately compares only the main reason code (bug-1845). They return only yes/no, so a `--sim-verify` mismatch log cannot say which field differs (bug-1881: 43 mismatch lines in one replay, none showing the difference). Three scheduler test files (`test_sim_equivalence.c`, `test_repl_equivalence.c`, `test_repl_limit_table.c`) reuse two of these comparators only because they white-box `#include "../src/sim_verify.c"` and can therefore call the `static` functions directly.
|
|
|
|
This should change so that:
|
|
- the allocation equality comparators become public libxce API — `alloc_eq`/`host_alloc_eq`/`job_alloc_eq` in `xcealloc.{c,h}` (the module that already constructs and deep-copies them);
|
|
- libxce gains a new, full-structural-match pending-reason comparator `job_preason_eq` in `preason.{c,h}` (new behavior, not a relocation: bug-1845 narrowed the simulator's comparator to main-reason-only);
|
|
- every comparator can, on request, return a newly allocated description of the first difference it found, and `--sim-verify` puts that description in its mismatch log;
|
|
- an allocation's reservation start time `from` becomes part of allocation equality. Today `make_alloc()` stamps it with the wall clock (`time(NULL)`), so a replay can never reproduce it and the simulator had to ignore it (bug-1838-followup). Instead the scheduler stamps `from` with the start time of the scheduling session that made the allocation, a value a replay reproduces exactly.
|
|
|
|
The decision-level dispatcher (`sim_decision_eq`), its main-reason-only pending-decision check, and mismatch reporting (`sim_log_mismatch`) are simulator-specific and stay in `sim_verify.c`. The controller also carries these structs (`persist.c`, `resync.c`, `job.c`) and is a likely future consumer, but adopting the API there is not part of this feature.
|
|
|
|
## Boundary Context
|
|
|
|
- **In scope**: moving the three allocation comparators from the scheduler simulator into public libxce API with a documented "same allocation decision" contract that compares `from` and excludes only `instid`; adding a new full-structural-match pending-reason comparator to libxce; an optional out-parameter on every comparator that returns a newly allocated description of the first difference, owned by the caller; stamping `from` with the scheduling session's start time, taken from the one timestamp that is also recorded as the session's `beginRound`; using the difference description in `--sim-verify` mismatch logs (resolves bug-1881); migrating the simulator and the three scheduler test files that call the static comparators to the public API; libxce unit tests for the new functions; updating comments that name the removed `sim_verify_*_eq` functions.
|
|
- **Out of scope**: moving the decision-level comparison dispatcher (`sim_decision_eq`) or mismatch reporting (`sim_log_mismatch` and its describe helpers) out of the scheduler; changing the simulator's main-reason-only pending-decision check (bug-1845); `merge_job_alloc()` and its `same_alloc_slot()` helper in `xcealloc.c` (bug-1863), which answer a different question — whether two records are the same reservation slot to merge, where `instid` *is* significant — and are neither replaced by nor reimplemented on top of `alloc_eq` (their NULL-dereference defect, bug-1882, is already fixed); removing the three test files' white-box `#include "../src/sim_verify.c"`, which they still need because they also white-box-include `sim_loop.c`, which calls `sim_decision_eq`/`sim_log_mismatch`; any new comparison call site beyond the existing consumers, including the controller; any change to how the controller or worker interpret `from`; any change to the `.replay` wire format; removing `--sim-speed` and making the simulator clock move only at recorded event and round timestamps, which is the separate spec `scheduler-sim-discrete-clock`.
|
|
- **Adjacent expectations**: this feature depends on spec `scheduler-sim-discrete-clock` for Requirement 4.3 (a replayed session starts at exactly the recorded `beginRound`); the `.replay` wire format and the `replay_decision_t`/`xce_decision_t` decision types are unchanged; the `xce/types.h` struct layouts are unchanged, and `from` keeps its unit (seconds since the Unix epoch); libxce's deep-copy and merge helpers (`dup_job_alloc`, `merge_job_alloc`) keep copying `from` unchanged; the controller's Cap'n Proto persistence round-trips for these structures are unchanged. Recordings made before this change carry wall-clock `from` values; replaying them with `--sim-verify` may report a DISPATCH mismatch when an allocation's original `time(NULL)` fell in a later second than its session's start. That is an accepted limitation for old recordings, and the new difference description names `from` as the cause.
|
|
|
|
## Requirements
|
|
|
|
### Requirement 1: Shared equality API for allocation structures
|
|
**Objective:** As a scheduler engineer who needs to compare two allocation objects, I want a public libxce equality function for each allocation structure, so that I can compare them correctly by linking libxce rather than reimplementing the comparison or calling another module's private helpers.
|
|
|
|
#### Acceptance Criteria
|
|
1. The library shall provide a public equality function for `xce_alloc_t` that reports equal if and only if the two records have the same resource name, reserve, value, reservation start time (`from`), and duration, where `value` is compared exactly (bit-for-bit `==`), not within a floating-point tolerance.
|
|
2. The library shall provide a public equality function for `xce_host_alloc_t` that reports equal if and only if the two records have the same host name and, in order, the same per-resource `xce_alloc_t` records.
|
|
3. The library shall provide a public equality function for `xce_job_alloc_t` that reports equal if and only if the two records have the same job id, the same per-host `xce_host_alloc_t` records in order, and the same shared-resource `xce_alloc_t` records in order.
|
|
4. The allocation equality functions shall answer "do these two records describe the same allocation decision" and shall not consider a record's allocation-instance id (`instid`), because it is taken from a per-process counter (`sched_get_id()`) whose value depends on how many allocations that process built before, so a recording and its replay give the same allocation different `instid` values (bug-1844). This contract is distinct from `merge_job_alloc()`'s slot matching, where `instid` is significant; the library's documentation of `alloc_eq` shall state that difference.
|
|
5. When the two arguments to an allocation equality function are the identical pointer (including two NULL pointers), the function shall report equal; when exactly one of the two arguments is NULL at the top level, or exactly one of two corresponding list entries or string fields is NULL at any nested level, the function shall report not-equal without crashing.
|
|
6. A record whose entry count is non-zero while its entry array pointer is NULL is malformed; the allocation equality functions are not required to handle it, and the library's documentation of these functions shall state that precondition.
|
|
7. The allocation equality functions shall not modify either argument.
|
|
|
|
### Requirement 2: Shared equality API for the pending-reason structure
|
|
**Objective:** As a scheduler engineer who needs to compare two pending-reason objects, I want a public libxce equality function that reports a full structural match, so that I can determine whether two pending reasons are identical by linking libxce rather than reimplementing the comparison.
|
|
|
|
#### Acceptance Criteria
|
|
1. The library shall provide a public equality function for `xce_job_preason_t` that reports equal if and only if the two records have the same main reason code, the same host-based reason entries in order, and the same resource-based reason entries in order. This is new comparison behavior; it is intentionally stricter than the simulator's existing main-reason-only check.
|
|
2. The pending-reason equality function shall compare each host-based reason entry by its host name and reason code, and each resource-based reason entry by its resource name, host name, and reason code.
|
|
3. The pending-reason equality function shall handle NULL arguments and malformed records the same way as the allocation equality functions (Requirement 1, criteria 5 and 6) and shall not modify either argument.
|
|
|
|
### Requirement 3: Description of the first difference
|
|
**Objective:** As an engineer debugging a mismatch, I want an equality function to tell me which field differs and the two values, so that I can diagnose the mismatch without re-running it under a debugger.
|
|
|
|
#### Acceptance Criteria
|
|
1. Each public equality function (Requirements 1 and 2) shall take an optional out-parameter `char** why`. When `why` is NULL, the function shall behave exactly as without it: no description is produced and nothing is allocated.
|
|
2. When `why` is non-NULL and the two arguments are not equal, the function shall set `*why` to a newly allocated, NUL-terminated, human-readable description of the first difference found, in comparison order, naming: the path to the differing field (for example the host-list index and host name, the resource-list index and resource name), the field name, and the two differing values (or which side is NULL, or the two list lengths). The function sizes the string itself; the description is never truncated.
|
|
3. When `why` is non-NULL and the two arguments are equal, the function shall set `*why` to NULL.
|
|
4. Ownership of a returned description passes to the caller, who releases it with `free()`. The function shall never read or free the value `*why` held on entry.
|
|
5. If allocating the description fails, the function shall set `*why` to NULL and still return the correct equal/not-equal result; the comparison result never depends on whether a description was produced.
|
|
6. Producing the description shall not log.
|
|
|
|
### Requirement 4: Reproducible reservation start time
|
|
**Objective:** As an operator relying on `--sim-verify`, I want an allocation's `from` to be a value a replay reproduces exactly, so that `from` can be part of allocation equality without causing false mismatches.
|
|
|
|
#### Acceptance Criteria
|
|
1. The scheduler shall set each allocation's `from` to the start time of the scheduling session that produced the allocation, in whole seconds since the Unix epoch (the session start in nanoseconds, divided by 10^9 and rounded down), instead of the wall-clock time at which the record was built.
|
|
2. In normal mode, the session start time used for `from` shall be the same single clock reading that the session records as its `beginRound` when recording is enabled; the scheduler shall not read the clock separately for the two.
|
|
3. In simulation mode, the session start time used for `from` shall be exactly the recorded session's `beginRound`, so that a replayed allocation's `from` equals the recorded one. This is provided by spec `scheduler-sim-discrete-clock` (the simulator clock does not advance between recorded timestamps); this feature shall not add its own simulator clock mechanism.
|
|
4. All allocations produced in one session, including every primary of a COMPOUND request merged by `merge_job_alloc()`, shall carry the same `from`.
|
|
5. Deep copies (`dup_job_alloc()`) and merges (`merge_job_alloc()`) shall keep copying `from` from their source, as they do today.
|
|
|
|
### Requirement 5: Simulator verification behavior
|
|
**Objective:** As an operator relying on `--sim-verify` replay checking, I want the simulator's decision verification to keep catching real divergence without new false mismatches, and to tell me what differs, so that a mismatch is both trustworthy and diagnosable.
|
|
|
|
#### Acceptance Criteria
|
|
1. The simulator shall compare a recorded DISPATCH decision against a live one with the libxce allocation comparison (Requirement 1), which now also compares `from`, rather than defining its own.
|
|
2. The simulator shall keep its pending-decision comparison as a simulator-local main-reason-only check, and shall not use the full-match `job_preason_eq`, because a recorded cache-hit pending reason carries no host/resource detail while a live replayed one always does (bug-1845). This asymmetry is a simulator concern, not a property of pending-reason equality.
|
|
3. When the simulator reports a verification mismatch, the log line shall keep its existing logical-time and decision context and shall add the description of the first difference (Requirement 3), or, for a PEND mismatch, the two jobids or the two main reason codes, or, when one side has no decision, which side is missing. Only if allocating the description fails (Requirement 3.5) may the line be logged without it. This resolves bug-1881.
|
|
4. Replaying a recording made after this change against its own previously-recorded output with verification enabled shall report zero mismatches.
|
|
|
|
### Requirement 6: Single shared implementation across all consumers
|
|
**Objective:** As a maintainer, I want the allocation comparisons and the full pending-reason comparison to exist exactly once, in libxce, so that future changes to their semantics are made in one place and every consumer picks them up.
|
|
|
|
#### Acceptance Criteria
|
|
1. The simulator and the three scheduler test files that currently call the static comparators (`test_sim_equivalence.c`, `test_repl_equivalence.c`, `test_repl_limit_table.c`) shall obtain the allocation comparison from the libxce public API, and no scheduler source or test shall retain a copy of it. The simulator's main-reason-only pending-decision check (Requirement 5, criterion 2) is the single documented simulator-local pending-reason comparison.
|
|
2. The three test files shall compare pending reasons with the full-match `job_preason_eq`. This makes their pending-decision checks stricter than today's main-reason-only comparison. The change is valid because both sides of each of their comparisons are produced without the scheduler's unsatisfiable-allocation cache (none of these tests registers `unsatisfiable_lookup_cb`), so the detail lists are built by the same full per-host computation on both sides and are expected to match; each test file shall document this reasoning where it calls `job_preason_eq`. Their allocation comparisons also become stricter (`from` is compared). Hand-built test fixtures that stand in for a recording shall carry the recorded session's start time as `from` (Requirement 4.1), like a real recording. If a test fails after these changes, the failure shall be investigated as a real divergence, not resolved by loosening the comparison.
|
|
3. A consumer shall be able to call the public comparators by including `xcealloc.h`/`preason.h` and linking libxce, without white-box-including any scheduler source. The three test files may keep their existing white-box `#include "../src/sim_verify.c"`, since they need it for the decision-level functions that `sim_loop.c` calls, but shall no longer call any comparator that is private to `sim_verify.c`.
|
|
4. The library shall provide unit test coverage for each new equality function, covering: a match; a mismatch on each compared field, with the expected difference description; that an `instid` difference does not affect allocation equality; that a `from` difference does; that list order matters (the same entries in a different order compare not-equal); a list-length mismatch; the NULL-handling cases at the top level and at nested entries and string fields; a NULL `why`; `*why` set to NULL on equal; and a failed description allocation (`*why` NULL, same result).
|
|
5. The scheduler shall have tests showing that allocations made in one session carry that session's start time as `from`, and that the recorded `beginRound` equals that start time.
|
|
6. Adding these functions shall not alter the behavior of any existing libxce function other than the `from` stamping required by Requirement 4, and shall not alter `merge_job_alloc()`'s slot matching.
|
|
7. Comments in scheduler sources and tests that name the removed `sim_verify_alloc_eq`, `sim_verify_host_alloc_eq`, `sim_verify_job_alloc_eq`, or `sim_verify_job_preason_eq` functions, or that describe `from` as excluded from comparison, shall be updated.
|