myxce/.kiro/specs/libxce-alloc-preason-equality/requirements.md

Requirements Document

Introduction

Engineers maintaining the XCE scheduler who need to determine whether two scheduler allocation or pending-reason objects represent the same decision currently have no shared way to do so, and when two objects differ nothing tells them where. The structural equality comparators for the shared data structures xce_alloc_t, xce_host_alloc_t, and xce_job_alloc_t (all defined in libxce's src/include/xce/types.h) live as static functions inside the scheduler's simulation-verification module src/scheduler/src/sim_verify.c, written for --sim-verify replay checking, beside a pending-reason comparator for xce_job_preason_t that deliberately compares only the main reason code (bug-1845). They return only yes/no, so a --sim-verify mismatch log cannot say which field differs (bug-1881: 43 mismatch lines in one replay, none showing the difference). Three scheduler test files (test_sim_equivalence.c, test_repl_equivalence.c, test_repl_limit_table.c) reuse two of these comparators only because they white-box #include "../src/sim_verify.c" and can therefore call the static functions directly.

This should change so that:

• the allocation equality comparators become public libxce API — alloc_eq/host_alloc_eq/job_alloc_eq in xcealloc.{c,h} (the module that already constructs and deep-copies them);
• libxce gains a new, full-structural-match pending-reason comparator job_preason_eq in preason.{c,h} (new behavior, not a relocation: bug-1845 narrowed the simulator's comparator to main-reason-only);
• every comparator can, on request, return a newly allocated description of the first difference it found, and --sim-verify puts that description in its mismatch log;
• an allocation's reservation start time from becomes part of allocation equality. Today make_alloc() stamps it with the wall clock (time(NULL)), so a replay can never reproduce it and the simulator had to ignore it (bug-1838-followup). Instead the scheduler stamps from with the start time of the scheduling session that made the allocation, a value a replay reproduces exactly.

The decision-level dispatcher (sim_decision_eq), its main-reason-only pending-decision check, and mismatch reporting (sim_log_mismatch) are simulator-specific and stay in sim_verify.c. The controller also carries these structs (persist.c, resync.c, job.c) and is a likely future consumer, but adopting the API there is not part of this feature.

Boundary Context

• In scope: moving the three allocation comparators from the scheduler simulator into public libxce API with a documented "same allocation decision" contract that compares from and excludes only instid; adding a new full-structural-match pending-reason comparator to libxce; an optional out-parameter on every comparator that returns a newly allocated description of the first difference, owned by the caller; stamping from with the scheduling session's start time, taken from the one timestamp that is also recorded as the session's beginRound; using the difference description in --sim-verify mismatch logs (resolves bug-1881); migrating the simulator and the three scheduler test files that call the static comparators to the public API; libxce unit tests for the new functions; updating comments that name the removed sim_verify_*_eq functions.
• Out of scope: moving the decision-level comparison dispatcher (sim_decision_eq) or mismatch reporting (sim_log_mismatch and its describe helpers) out of the scheduler; changing the simulator's main-reason-only pending-decision check (bug-1845); merge_job_alloc() and its same_alloc_slot() helper in xcealloc.c (bug-1863), which answer a different question — whether two records are the same reservation slot to merge, where instid is significant — and are neither replaced by nor reimplemented on top of alloc_eq (their NULL-dereference defect, bug-1882, is already fixed); removing the three test files' white-box #include "../src/sim_verify.c", which they still need because they also white-box-include sim_loop.c, which calls sim_decision_eq/sim_log_mismatch; any new comparison call site beyond the existing consumers, including the controller; any change to how the controller or worker interpret from; any change to the .replay wire format; removing --sim-speed and making the simulator clock move only at recorded event and round timestamps, which is the separate spec scheduler-sim-discrete-clock.
• Adjacent expectations: this feature depends on spec scheduler-sim-discrete-clock for Requirement 4.3 (a replayed session starts at exactly the recorded beginRound); the .replay wire format and the replay_decision_t/xce_decision_t decision types are unchanged; the xce/types.h struct layouts are unchanged, and from keeps its unit (seconds since the Unix epoch); libxce's deep-copy and merge helpers (dup_job_alloc, merge_job_alloc) keep copying from unchanged; the controller's Cap'n Proto persistence round-trips for these structures are unchanged. Recordings made before this change carry wall-clock from values; replaying them with --sim-verify may report a DISPATCH mismatch when an allocation's original time(NULL) fell in a later second than its session's start. That is an accepted limitation for old recordings, and the new difference description names from as the cause.

Requirements

Requirement 1: Shared equality API for allocation structures

Objective: As a scheduler engineer who needs to compare two allocation objects, I want a public libxce equality function for each allocation structure, so that I can compare them correctly by linking libxce rather than reimplementing the comparison or calling another module's private helpers.

Acceptance Criteria

1. The library shall provide a public equality function for xce_alloc_t that reports equal if and only if the two records have the same resource name, reserve, value, reservation start time (from), and duration, where value is compared exactly (bit-for-bit ==), not within a floating-point tolerance.
2. The library shall provide a public equality function for xce_host_alloc_t that reports equal if and only if the two records have the same host name and, in order, the same per-resource xce_alloc_t records.
3. The library shall provide a public equality function for xce_job_alloc_t that reports equal if and only if the two records have the same job id, the same per-host xce_host_alloc_t records in order, and the same shared-resource xce_alloc_t records in order.
4. The allocation equality functions shall answer "do these two records describe the same allocation decision" and shall not consider a record's allocation-instance id (instid), because it is taken from a per-process counter (sched_get_id()) whose value depends on how many allocations that process built before, so a recording and its replay give the same allocation different instid values (bug-1844). This contract is distinct from merge_job_alloc()'s slot matching, where instid is significant; the library's documentation of alloc_eq shall state that difference.
5. When the two arguments to an allocation equality function are the identical pointer (including two NULL pointers), the function shall report equal; when exactly one of the two arguments is NULL at the top level, or exactly one of two corresponding list entries or string fields is NULL at any nested level, the function shall report not-equal without crashing.
6. A record whose entry count is non-zero while its entry array pointer is NULL is malformed; the allocation equality functions are not required to handle it, and the library's documentation of these functions shall state that precondition.
7. The allocation equality functions shall not modify either argument.

Requirement 2: Shared equality API for the pending-reason structure

Objective: As a scheduler engineer who needs to compare two pending-reason objects, I want a public libxce equality function that reports a full structural match, so that I can determine whether two pending reasons are identical by linking libxce rather than reimplementing the comparison.

Acceptance Criteria

1. The library shall provide a public equality function for xce_job_preason_t that reports equal if and only if the two records have the same main reason code, the same host-based reason entries in order, and the same resource-based reason entries in order. This is new comparison behavior; it is intentionally stricter than the simulator's existing main-reason-only check.
2. The pending-reason equality function shall compare each host-based reason entry by its host name and reason code, and each resource-based reason entry by its resource name, host name, and reason code.
3. The pending-reason equality function shall handle NULL arguments and malformed records the same way as the allocation equality functions (Requirement 1, criteria 5 and 6) and shall not modify either argument.

Requirement 3: Description of the first difference

Objective: As an engineer debugging a mismatch, I want an equality function to tell me which field differs and the two values, so that I can diagnose the mismatch without re-running it under a debugger.

Acceptance Criteria

1. Each public equality function (Requirements 1 and 2) shall take an optional out-parameter char** why. When why is NULL, the function shall behave exactly as without it: no description is produced and nothing is allocated.
2. When why is non-NULL and the two arguments are not equal, the function shall set *why to a newly allocated, NUL-terminated, human-readable description of the first difference found, in comparison order, naming: the path to the differing field (for example the host-list index and host name, the resource-list index and resource name), the field name, and the two differing values (or which side is NULL, or the two list lengths). The function sizes the string itself; the description is never truncated.
3. When why is non-NULL and the two arguments are equal, the function shall set *why to NULL.
4. Ownership of a returned description passes to the caller, who releases it with free(). The function shall never read or free the value *why held on entry.
5. If allocating the description fails, the function shall set *why to NULL and still return the correct equal/not-equal result; the comparison result never depends on whether a description was produced.
6. Producing the description shall not log.

Requirement 4: Reproducible reservation start time

Objective: As an operator relying on --sim-verify, I want an allocation's from to be a value a replay reproduces exactly, so that from can be part of allocation equality without causing false mismatches.

Acceptance Criteria

1. The scheduler shall set each allocation's from to the start time of the scheduling session that produced the allocation, in whole seconds since the Unix epoch (the session start in nanoseconds, divided by 10^9 and rounded down), instead of the wall-clock time at which the record was built.
2. In normal mode, the session start time used for from shall be the same single clock reading that the session records as its beginRound when recording is enabled; the scheduler shall not read the clock separately for the two.
3. In simulation mode, the session start time used for from shall be exactly the recorded session's beginRound, so that a replayed allocation's from equals the recorded one. This is provided by spec scheduler-sim-discrete-clock (the simulator clock does not advance between recorded timestamps); this feature shall not add its own simulator clock mechanism.
4. All allocations produced in one session, including every primary of a COMPOUND request merged by merge_job_alloc(), shall carry the same from.
5. Deep copies (dup_job_alloc()) and merges (merge_job_alloc()) shall keep copying from from their source, as they do today.

Requirement 5: Simulator verification behavior

Objective: As an operator relying on --sim-verify replay checking, I want the simulator's decision verification to keep catching real divergence without new false mismatches, and to tell me what differs, so that a mismatch is both trustworthy and diagnosable.

Acceptance Criteria

1. The simulator shall compare a recorded DISPATCH decision against a live one with the libxce allocation comparison (Requirement 1), which now also compares from, rather than defining its own.
2. The simulator shall keep its pending-decision comparison as a simulator-local main-reason-only check, and shall not use the full-match job_preason_eq, because a recorded cache-hit pending reason carries no host/resource detail while a live replayed one always does (bug-1845). This asymmetry is a simulator concern, not a property of pending-reason equality.
3. When the simulator reports a verification mismatch, the log line shall keep its existing logical-time and decision context and shall add the description of the first difference (Requirement 3), or, for a PEND mismatch, the two jobids or the two main reason codes, or, when one side has no decision, which side is missing. Only if allocating the description fails (Requirement 3.5) may the line be logged without it. This resolves bug-1881.
4. Replaying a recording made after this change against its own previously-recorded output with verification enabled shall report zero mismatches.

Requirement 6: Single shared implementation across all consumers

Objective: As a maintainer, I want the allocation comparisons and the full pending-reason comparison to exist exactly once, in libxce, so that future changes to their semantics are made in one place and every consumer picks them up.

Acceptance Criteria

1. The simulator and the three scheduler test files that currently call the static comparators (test_sim_equivalence.c, test_repl_equivalence.c, test_repl_limit_table.c) shall obtain the allocation comparison from the libxce public API, and no scheduler source or test shall retain a copy of it. The simulator's main-reason-only pending-decision check (Requirement 5, criterion 2) is the single documented simulator-local pending-reason comparison.
2. The three test files shall compare pending reasons with the full-match job_preason_eq. This makes their pending-decision checks stricter than today's main-reason-only comparison. The change is valid because both sides of each of their comparisons are produced without the scheduler's unsatisfiable-allocation cache (none of these tests registers unsatisfiable_lookup_cb), so the detail lists are built by the same full per-host computation on both sides and are expected to match; each test file shall document this reasoning where it calls job_preason_eq. Their allocation comparisons also become stricter (from is compared). Hand-built test fixtures that stand in for a recording shall carry the recorded session's start time as from (Requirement 4.1), like a real recording. If a test fails after these changes, the failure shall be investigated as a real divergence, not resolved by loosening the comparison.
3. A consumer shall be able to call the public comparators by including xcealloc.h/preason.h and linking libxce, without white-box-including any scheduler source. The three test files may keep their existing white-box #include "../src/sim_verify.c", since they need it for the decision-level functions that sim_loop.c calls, but shall no longer call any comparator that is private to sim_verify.c.
4. The library shall provide unit test coverage for each new equality function, covering: a match; a mismatch on each compared field, with the expected difference description; that an instid difference does not affect allocation equality; that a from difference does; that list order matters (the same entries in a different order compare not-equal); a list-length mismatch; the NULL-handling cases at the top level and at nested entries and string fields; a NULL why; *why set to NULL on equal; and a failed description allocation (*why NULL, same result).
5. The scheduler shall have tests showing that allocations made in one session carry that session's start time as from, and that the recorded beginRound equals that start time.
6. Adding these functions shall not alter the behavior of any existing libxce function other than the from stamping required by Requirement 4, and shall not alter merge_job_alloc()'s slot matching.
7. Comments in scheduler sources and tests that name the removed sim_verify_alloc_eq, sim_verify_host_alloc_eq, sim_verify_job_alloc_eq, or sim_verify_job_preason_eq functions, or that describe from as excluded from comparison, shall be updated.