docs(spec): libxce-alloc-preason-equality review fixes; new scheduler-sim-discrete-clock spec

libxce-alloc-preason-equality (requirements, design, tasks; not yet approved):
- alloc_eq compares `from` and excludes only instid; the scheduler stamps
  `from` with the deciding session's start, the same clock reading that
  is recorded as beginRound, so a replay reproduces it.
- every comparator takes an optional char** why: a malloc'd description
  of the first difference, owned and freed by the caller; --sim-verify
  mismatch lines print it (resolves bug-1881).
- job_preason_eq is new full-match behavior, not a relocation (bug-1845
  keeps the simulator main-reason-only); the three white-box test files
  switch to it and keep their sim_verify.c include.
- alloc_eq stays separate from merge_job_alloc()'s same_alloc_slot()
  (bug-1863), which does match on instid.

scheduler-sim-discrete-clock (initialized): remove --sim-speed so the
simulator clock moves only at recorded timestamps and a replayed
session_start equals the recorded beginRound exactly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-28 08:50:07 +00:00
parent 8f51a4997d
commit dcf54387ae
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
6 changed files with 335 additions and 157 deletions

View file

@ -2,123 +2,169 @@
## Overview
**Purpose**: This feature delivers a single, shared implementation of structural equality for the scheduler's allocation and pending-reason data structures, published as public libxce API.
**Purpose**: This feature delivers a single, shared implementation of structural equality for the scheduler's allocation and pending-reason data structures, published as public libxce API, that can explain *why* two objects differ, together with a reproducible allocation start time (`from`) so that `from` can take part in equality.
**Users**: Scheduler engineers (today: `--sim-verify` replay checking and three scheduler test files) and, in future, controller engineers (`persist.c`, `resync.c`, `job.c` already carry these structs) who need to ask "do these two objects represent the same thing".
**Users**: Scheduler engineers (today: `--sim-verify` replay checking and three scheduler test files) and, in future, controller engineers (`persist.c`, `resync.c`, `job.c` already carry these structs) who need to ask "do these two objects represent the same thing, and if not, where do they differ".
**Impact**: Changes the current state by extracting four `static` comparison functions out of `src/scheduler/src/sim_verify.c` and into `src/lib/xcealloc.c`/`src/lib/preason.c`, and by switching the simulator and three white-box test consumers to the public API. The allocation comparators keep their documented `from`/`instid` exclusion; the pending-reason comparator becomes a full structural match (all fields, detail lists included). The decision-level dispatcher (`sim_decision_eq`) and mismatch reporting (`sim_log_mismatch`) stay in the scheduler, and `sim_decision_eq`'s PEND branch keeps a simulator-local main-reason-only comparison (bug-1845). No wire format or struct layout changes.
**Impact**:
- Moves the three `static` allocation comparators out of `src/scheduler/src/sim_verify.c` into `src/lib/xcealloc.c`. `alloc_eq` now compares `from` and excludes only `instid`.
- Adds a **new** full-structural-match pending-reason comparator to `src/lib/preason.c`. No full-match comparator exists today; bug-1845 narrowed the simulator's to main-reason-only.
- Gives every comparator an optional `char** why` out-parameter that returns a newly allocated description of the first difference, owned by the caller. `sim_log_mismatch` prints it (resolves bug-1881).
- Changes the scheduler to stamp `from` with the scheduling session's start time, a single clock reading shared with the recorded `beginRound`, instead of `time(NULL)`. That a replayed session starts at exactly the recorded `beginRound` comes from the separate spec `scheduler-sim-discrete-clock` (removes `--sim-speed`; the simulator clock moves only at recorded timestamps).
- Switches the simulator and three white-box test consumers to the public API. The three test files' checks become stricter: full-match pending reasons (Decision 7) and `from`-inclusive allocations.
- `sim_decision_eq` and `sim_log_mismatch` stay in the scheduler, and `sim_decision_eq`'s PEND branch keeps a simulator-local main-reason-only comparison (bug-1845). No wire format or struct layout changes.
### Goals
- Publish `alloc_eq`, `host_alloc_eq`, `job_alloc_eq` in `xcealloc.h` and `job_preason_eq` in `preason.h`.
- Publish `alloc_eq`, `host_alloc_eq`, `job_alloc_eq` in `xcealloc.h` and `job_preason_eq` in `preason.h`, each with an optional first-difference description.
- Make `job_preason_eq` a full structural match (main reason + host-based reasons + resource-based reasons, in order).
- Make the existing consumers use the public API, removing the white-box `#include` dependency on `sim_verify.c` for the type-level comparisons.
- Preserve `--sim-verify` behavior bit-for-bit (no new false/missed mismatches), which means the simulator's PEND check remains main-reason-only.
- Add libxce unit tests that pin the allocation-exclusion, full-match preason, and NULL-handling semantics.
- Make `from` reproducible under replay, then compare it.
- Make `--sim-verify` mismatch logs name the differing field and values.
- Make the existing consumers call the public API instead of `sim_verify.c`'s private `static` comparators. (The three test files keep their white-box `#include "../src/sim_verify.c"`: they also white-box-include `sim_loop.c`, which calls `sim_decision_eq`/`sim_log_mismatch`.)
- Keep `--sim-verify` free of false mismatches for recordings made after this change.
- Add libxce unit tests that pin the `instid` exclusion, `from` inclusion, full-match preason, description, and NULL-handling semantics.
### Non-Goals
- Do not move `sim_decision_eq`, `sim_log_mismatch`, or the `describe_*` helpers (simulator-specific; depend on capnp/replay types and `ERROR()`).
- Do not change the simulator's PEND outcome check from main-reason-only to full-match — that would reintroduce bug-1845's false mismatches on cache-hit PENDs (see Decision 3).
- Do not introduce any new comparison call site beyond the existing four consumers.
- Do not touch the controller's persistence/resync code paths.
- Do not touch the controller's persistence/resync code paths, or how the controller/worker interpret `from`.
- Do not change `make_alloc()`'s signature or its own `time(NULL)` default (controller and many tests call it); the scheduler overwrites `from` after construction (Decision 8).
- Do not touch `merge_job_alloc()`/`same_alloc_slot()` (bug-1863); `alloc_eq` neither replaces nor reuses `same_alloc_slot` (see Decision 6). Their NULL-dereference defect is bug-1882.
- Do not remove the test files' white-box `#include "../src/sim_verify.c"`.
- Do not make recordings made before this change verify cleanly; their wall-clock `from` values are an accepted limitation.
- Do not change the simulator's clock or `--sim-speed`; that is spec `scheduler-sim-discrete-clock`, which this spec depends on.
## Boundary Commitments
### This Spec Owns
- The public equality functions `alloc_eq`, `host_alloc_eq`, `job_alloc_eq` (`xcealloc.{c,h}`) and `job_preason_eq` (`preason.{c,h}`), and their documented equality contracts.
- The public equality functions `alloc_eq`, `host_alloc_eq`, `job_alloc_eq` (`xcealloc.{c,h}`) and `job_preason_eq` (`preason.{c,h}`), their equality contracts and their difference-description format.
- The shared NULL-safe string-equality helper both modules need (see Decision 5).
- The migration of `sim_verify.c` and the three white-box test files onto those functions.
- libxce unit tests for the new functions (`test_xcealloc.c`, `test_preason.c`).
- The scheduler's stamping of `from` (scheduler.c) and the single normal-mode session-start reading shared by `core->session_start`, the recorded `beginRound`, and `from` (driver.c).
- The migration of `sim_verify.c` and the three white-box test files onto those functions, and the difference text in `sim_log_mismatch`.
- libxce unit tests for the new functions (`test_xcealloc.c`, `test_preason.c`) and scheduler tests for `from` stamping.
### Out of Boundary
- The decision-level comparison and mismatch reporting that remain in `sim_verify.c`, including the main-reason-only PEND outcome check (simulator-local).
- The decision-level comparison and mismatch reporting structure that remain in `sim_verify.c`, including the main-reason-only PEND outcome check (simulator-local).
- The `.replay` wire format and the `replay_decision_t`/`xce_decision_t` types.
- Any change to the controller's consumption of these structs.
- `merge_job_alloc()` and its `same_alloc_slot()` slot-matching helper in `xcealloc.c`.
- The simulator clock and `--sim-speed` (spec `scheduler-sim-discrete-clock`).
### Allowed Dependencies
- `src/include/xce/types.h` (the struct definitions — unchanged).
- `xcealloc.c`/`xcealloc.h`, `preason.c`/`preason.h`, and `utils.{c,h}` (the modules that already own construction of these structs, plus the shared string helper).
- `string.h` (`strcmp`) for the string-field comparisons.
- `string.h` (`strcmp`) and `stdio.h` (`vasprintf`) for comparisons and descriptions.
- Scheduler: `driver.c`, `scheduler.c`, `sim_verify.c`, `sim_loop.c` (the `sim_decision_eq` call only).
- Spec `scheduler-sim-discrete-clock`: after it, a replayed session's `core->session_start` equals the recorded `beginRound` exactly.
### Revalidation Triggers
- A change to the `xce_alloc_t`/`xce_host_alloc_t`/`xce_job_alloc_t`/`xce_job_preason_t` struct layouts must re-check the field lists in the equality functions and the exclusion documentation.
- A change to the `xce_alloc_t`/`xce_host_alloc_t`/`xce_job_alloc_t`/`xce_job_preason_t` struct layouts must re-check the field lists, the exclusion documentation, and the description text.
- A change to `same_alloc_slot()`'s matching rule, or a proposal to merge it with `alloc_eq`, must re-check Decision 6 (the two answer different questions and disagree on `instid`, `from`, and `value`).
- A change to the pending-reason construction paths (cache-hit vs. full-computation) that alters whether detail lists are populated must re-check the simulator's main-reason-only PEND rationale (bug-1845) and the full-match `job_preason_eq` contract.
- Any new path that builds an `xce_alloc_t` in the scheduler outside a session, a second clock read between session start and recording, or a change to `scheduler-sim-discrete-clock`'s clock guarantee, must re-check Decision 8.
## Architecture
### Existing Architecture Analysis
`src/scheduler/src/sim_verify.c` (474 lines) owns two layers:
1. **Type-level comparators** — five `static` functions (`sim_verify_str_eq`, `sim_verify_alloc_eq`, `sim_verify_host_alloc_eq`, `sim_verify_job_alloc_eq`, `sim_verify_job_preason_eq`) that walk two libxce structs field-by-field. They have zero dependency on the scheduler, capnp, or the replay layer. Four of them are cleanly movable; `sim_verify_job_preason_eq` is movable but its current contract (main-reason-only) is wrong for a *public* equality function — it encodes a simulator replay asymmetry, not an object-identity property.
2. **Decision-level dispatcher + reporting** — `sim_decision_eq` (dispatches on `xce_decision_t.op` vs `ReplayDecision_body_which`), `sim_log_mismatch`, and two `describe_*` helpers. These depend on `replay_decision_t`, capnp enums, and `ERROR()`.
1. **Type-level comparators** — five `static` functions (`sim_verify_str_eq`, `sim_verify_alloc_eq`, `sim_verify_host_alloc_eq`, `sim_verify_job_alloc_eq`, `sim_verify_job_preason_eq`) that walk two libxce structs field-by-field and return only 0/1. They have zero dependency on the scheduler, capnp, or the replay layer. `sim_verify_alloc_eq` excludes `from` (bug-1838-followup) and `instid` (bug-1844). `sim_verify_job_preason_eq` compares only `main_reason` (bug-1845), a simulator replay asymmetry rather than an object-identity property.
2. **Decision-level dispatcher + reporting** — `sim_decision_eq` (dispatches on `xce_decision_t.op` vs `ReplayDecision_body_which`), `sim_log_mismatch`, and two `describe_*` helpers. These depend on `replay_decision_t`, capnp enums, and `ERROR()`. `sim_log_mismatch` prints only jobid/nhosts for each side, so a mismatch below that level is invisible (bug-1881).
The four allocation/pending-reason structs are defined in `src/include/xce/types.h` and already constructed/deep-copied by `src/lib/xcealloc.c` (`make_alloc`, `make_host_alloc`, `make_job_alloc`, `dup_job_alloc`) and `src/lib/preason.c` (`add_job_preason`, `make_pendreason`). The controller consumes these same structs in `persist.c`, `resync.c`, and `job.c` with no equality API available.
Since this design was first drafted, bug-1863 (`aea34738`) added `merge_job_alloc()` to `xcealloc.c` with a private `same_alloc_slot()` helper that matches two `xce_alloc_t` records by `resname`, `instid`, and `reserve`, to decide whether they are the same reservation slot to merge. It is a near neighbour of `alloc_eq` but has a different contract (see Decision 6).
**How `from` is set today.** `make_alloc()` (`xcealloc.c:30`) stamps `from = time(NULL)`. The scheduler's only production constructor call is `scheduler.c:1893`, inside a scheduling session. `from` is on the wire (`controller.capnp` `from @4: UInt64 # alloc start time`) and `dup_alloc()` copies it.
**How session time is read today.** Two separate clock reads per session:
- `xce_core_begin_session()` sets `core->session_start = xce_core_now(core)` (`core_session.c:429`).
- The recording then reads the clock again for `beginRound`: `driver.c:1030` after `plugin_begin_session`/`sched_account_jobs`, and `sim_loop.c` before `xce_core_begin_session`.
In simulation, `sim_advance_virtual_clock(sim, begin_round)` re-anchors the virtual clock to the recorded `beginRound`. At `--sim-speed` ≥ 1 the clock keeps advancing from there, so `session_start` lands slightly after `begin_round`. At `--sim-speed 0` it is frozen and exact. The simulator never sleeps, so `--sim-speed` paces nothing; it only adds drift. Spec `scheduler-sim-discrete-clock` removes it, which makes `session_start == begin_round` in every replay.
Three scheduler test files already reuse two of the leaf comparators — but via white-box `#include "../src/sim_verify.c"` + direct calls to the `static` functions: `test_sim_equivalence.c` (lines 894, 902), `test_repl_equivalence.c` (lines 734, 742), and `test_repl_limit_table.c` (lines 2662, 2670).
### Architecture Pattern & Boundary Map
```mermaid
flowchart LR
subgraph scheduler["src/scheduler"]
subgraph scheduler
SV["sim_verify.c\n(sim_decision_eq, sim_log_mismatch, describe_*,\nmain-reason-only PEND outcome check)"]
DR["driver.c\nbeginRound = core.session_start"]
SC["scheduler.c\nalloc->from = session_start / 1e9"]
CS["core_session.c\nsession_start"]
DC["spec scheduler-sim-discrete-clock\nreplayed session_start == recorded beginRound"]
TE["test_sim_equivalence.c / test_repl_equivalence.c / test_repl_limit_table.c"]
TSV["test_sim_verify.c"]
end
subgraph libxce["src/lib + src/include"]
XA["xcealloc.{c,h}\nalloc_eq / host_alloc_eq / job_alloc_eq"]
PR["preason.{c,h}\njob_preason_eq (full match)"]
subgraph libxce
XA["xcealloc.{c,h}\nalloc_eq / host_alloc_eq / job_alloc_eq (+why)"]
PR["preason.{c,h}\njob_preason_eq (full match, +why)"]
UT["utils.{c,h}\nstr_eq (NULL-safe)"]
TY["xce/types.h\n(struct defs, unchanged)"]
end
SV -->|"DISPATCH: job_alloc_eq (public API)"| XA
TE -.->|"calls (replaces white-box static calls)"| XA
TE -.->|"calls"| PR
TSV -->|"exercises via sim_decision_eq"| SV
SV --> XA
TE --> XA
TE --> PR
XA --> UT
PR --> UT
XA --> TY
PR --> TY
DR --> CS
SC --> CS
DC -.-> CS
```
**Architecture Integration**:
- Selected pattern: co-locate equality with the module that already owns each struct's construction and deep-copy — `alloc` family in `xcealloc.c` (mirrors `dup_job_alloc`), `job_preason_eq` in `preason.c` (mirrors `add_job_preason`). No new module is introduced.
- Domain boundary: the library owns *struct-level* equality (full structural match); the simulator owns *decision-level* comparison and reporting, including its one deliberate loosening (main-reason-only PEND). This is the clean seam: everything below it is libxce-typed and identity-honest, everything above it touches `replay_decision_t`/capnp and may narrow for replay reasons.
- Existing patterns preserved: public headers in `src/include/`, implementations in `src/lib/`, `const`-qualified read-only parameters, lowercase unprefixed function names (`make_*`/`dup_*`/`erase_*`).
- Steering compliance: no changes to `types.h` layouts, no new persistence surface, no HA/RBAC/persistence semantics touched.
- Domain boundary: the library owns *struct-level* equality and its explanation (full structural match); the simulator owns *decision-level* comparison and reporting, including its one deliberate loosening (main-reason-only PEND). The scheduler core owns the session-start value that makes `from` reproducible.
- Steering compliance: no changes to `types.h` layouts or the wire schema; `from` keeps its unit and meaning ("alloc start time"), now defined as the start of the deciding session. Persistence: `from` already round-trips through the snapshot (`dup_job_alloc`) and the controller's capnp encoding, so no new save/restore field is introduced.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Language | C (gnu17) | equality functions + tests | unchanged from libxce |
| Build | Meson via `xce-build` | compile libxce + run `src/lib/testsuite` | unchanged |
| Test | Unity | `test_xcealloc.c`, `test_preason.c` | existing libxce test harness |
| Language | C (gnu17) | equality functions, `from` stamping, tests | unchanged |
| Build | Meson via `xce-build` | compile libxce + scheduler, run `src/lib/testsuite` and `src/scheduler/testsuite` | unchanged |
| Test | Unity | `test_xcealloc.c`, `test_preason.c`, scheduler tests | existing harnesses |
## File Structure Plan
### Modified Files
- `src/lib/xcealloc.c` — add `alloc_eq`, `host_alloc_eq`, `job_alloc_eq` (using the shared `str_eq` helper).
- `src/include/xcealloc.h` — declare the three allocation equality functions.
- `src/lib/preason.c` — add `job_preason_eq` (full match, using the shared `str_eq` helper).
- `src/include/preason.h` — declare `job_preason_eq`.
- `src/lib/utils.c` / `src/include/utils.h` — add a NULL-safe `str_eq(const char*, const char*)` helper (see Decision 5).
- `src/scheduler/src/sim_verify.c` — delete the five `static` comparators; have `sim_decision_eq` call `job_alloc_eq` for DISPATCH and keep a simulator-local `static` main-reason-only helper for PEND. Keep `sim_log_mismatch` + describe helpers.
- `src/lib/testsuite/test_xcealloc.c` — add allocation equality unit tests.
- `src/lib/testsuite/test_preason.c` — add `job_preason_eq` full-match unit tests.
- `src/scheduler/testsuite/test_sim_equivalence.c`, `test_repl_equivalence.c`, `test_repl_limit_table.c` — replace direct calls to `sim_verify_job_alloc_eq`/`sim_verify_job_preason_eq` with `job_alloc_eq`/`job_preason_eq`; add `#include "xcealloc.h"`/`#include "preason.h"`.
- `src/scheduler/testsuite/test_sim_verify.c` — remove the DISPATCH alloc-level unit tests that now belong to libxce; keep the PEND/decision-level tests (they exercise `sim_decision_eq`'s simulator-local main-reason-only behavior) and the integration-level tests.
- `src/lib/xcealloc.c` / `src/include/xcealloc.h` — add `alloc_eq`, `host_alloc_eq`, `job_alloc_eq` with the optional `char** why` description.
- `src/lib/preason.c` / `src/include/preason.h` — add `job_preason_eq` (full match) with the optional description.
- `src/scheduler/src/driver.c` — record `beginRound` from `sched->core.session_start` instead of a second `xce_core_now()` read.
- `src/scheduler/src/scheduler.c` — after `make_alloc()` at `:1893`, set `allocs[i]->from = sched->core.session_start / 1000000000ULL`.
- `src/scheduler/src/sim_verify.c` — delete the five `static` comparators; `sim_decision_eq` gains an optional `char** why` and calls `job_alloc_eq` for DISPATCH; keep a simulator-local `static` main-reason-only helper for PEND; `sim_log_mismatch` asks `sim_decision_eq` for the description, appends it, and frees it. Update the header comments that describe `from` as excluded.
- `src/scheduler/src/sim_loop.c` — pass the description through (its `extern` declaration of `sim_decision_eq` gets the new parameters).
- `src/lib/testsuite/test_xcealloc.c`, `src/lib/testsuite/test_preason.c` — unit tests.
- `src/scheduler/testsuite/test_sim_equivalence.c`, `test_repl_equivalence.c`, `test_repl_limit_table.c` — replace direct calls with `job_alloc_eq(..., NULL)`/`job_preason_eq(..., NULL)` (or take the description, print it on failure, and free it); add `#include "xcealloc.h"`/`#include "preason.h"`; comment at each `job_preason_eq` call why full match is valid (Decision 7); keep the white-box `#include "../src/sim_verify.c"`.
- `src/scheduler/testsuite/test_sim_verify.c` — drop the DISPATCH alloc-level tests now owned by libxce; replace the "`from` is ignored" test with "`from` from the same session matches"; add a test that the mismatch log names the differing field; keep PEND/decision-level and integration tests.
- A scheduler test (new or in an existing session/allocation test file) — allocations made in one session carry `session_start / 1e9` as `from`, and the recorded `beginRound` equals `session_start`.
- `src/scheduler/testsuite/test_sim_live_allocation.c`, `test_sim_reconcile_unaccounted.c`, and the header comments of `test_sim_equivalence.c`/`test_repl_equivalence.c` — update comments that name `sim_verify_*_eq` or describe `from` as excluded.
## System Flows
The comparison flow is unchanged from today — this feature only relocates the leaf logic. `sim_decision_eq` remains the entry point:
```mermaid
flowchart TD
A["sim_decision_eq(expected, actual)"] --> B{"actual->op"}
B -->|DISPATCH| C["job_alloc_eq(expected->body.dispatch->alloc, actual->u.alloc)"]
B -->|PEND| D["jobid equal AND main_reason equal (sim-local, NOT job_preason_eq)"]
A["sim_decision_eq(expected, actual, why)"] --> B{"actual->op"}
B -->|DISPATCH| C["job_alloc_eq(expected alloc, actual alloc, why)"]
B -->|PEND| D["jobid equal AND main_reason equal (sim-local, NOT job_preason_eq); why names which"]
B -->|PREEMPT| E["log + report mismatch (unchanged)"]
C --> F["host_alloc_eq per host"] --> G["alloc_eq per resource (excl. from/instid)"]
C --> F["host_alloc_eq per host"] --> G["alloc_eq per resource (incl. from, excl. instid)"]
D --> H["main-reason only, detail ignored (bug-1845)"]
C --> M["sim_log_mismatch appends *why, then free()"]
D --> M
```
`from` stamping per session:
```mermaid
sequenceDiagram
participant Drv as driver.c
participant Core as core_session.c
participant Sch as scheduler.c
participant Rec as recording
Drv->>Core: xce_core_begin_session()
Core-->>Core: session_start = xce_core_now()<br/>(replay: == recorded beginRound, via scheduler-sim-discrete-clock)
Drv->>Rec: beginRound = session_start
Sch->>Sch: make_alloc(...); from = session_start / 1e9
```
## Requirements Traceability
@ -126,14 +172,19 @@ flowchart TD
| Requirement | Summary | Components | Interfaces |
|-------------|---------|------------|------------|
| 1.1–1.3 | allocation equality for `xce_alloc_t`/`xce_host_alloc_t`/`xce_job_alloc_t` | `xcealloc.c` | `alloc_eq`, `host_alloc_eq`, `job_alloc_eq` |
| 1.4 | exclude `from`/`instid` | `xcealloc.c` | `alloc_eq` field list |
| 1.5–1.6 | NULL handling + read-only | all four functions | const params |
| 1.4 | exclude `instid` only; document difference from `merge_job_alloc` | `xcealloc.c/.h` | `alloc_eq` field list + header comment |
| 1.5–1.7 | NULL handling, malformed-record precondition, read-only | all four functions | const params |
| 2.1–2.3 | pending-reason full structural equality | `preason.c` | `job_preason_eq` |
| 3.1–3.2, 3.5 | simulator behavior preserved | `sim_verify.c` | `sim_decision_eq` |
| 3.3 | simulator uses libxce allocation comparison | `sim_verify.c` | `job_alloc_eq` |
| 3.4 | simulator-local main-reason-only PEND check | `sim_verify.c` | local static helper |
| 4.1–4.2 | single shared implementation | `sim_verify.c` + 3 test files | public API |
| 4.3–4.4 | libxce unit tests + no regressions | `test_xcealloc.c`, `test_preason.c` | — |
| 3.1–3.6 | first-difference description, caller-owned | all four functions | `char** why` |
| 4.1–4.2, 4.4–4.5 | reproducible `from` (normal mode) | `driver.c`, `scheduler.c` | `core->session_start` |
| 4.3 | replayed session start == recorded `beginRound` | spec `scheduler-sim-discrete-clock` | — |
| 5.1 | simulator uses libxce allocation comparison | `sim_verify.c` | `job_alloc_eq` |
| 5.2 | simulator-local main-reason-only PEND check | `sim_verify.c` | local static helper |
| 5.3 | mismatch log names the difference (bug-1881) | `sim_verify.c`, `sim_loop.c` | `sim_decision_eq(..., why)`, `sim_log_mismatch` |
| 5.4 | zero-mismatch self-replay | integration tests | — |
| 6.1–6.3 | single shared implementation; test files use full-match preason; include stays | `sim_verify.c` + 3 test files | public API |
| 6.4–6.6 | libxce + scheduler tests, no other behavior change | test files | — |
| 6.7 | stale comment references updated | scheduler sources/tests | — |
## Components and Interfaces
@ -143,33 +194,37 @@ flowchart TD
| Field | Detail |
|-------|--------|
| Intent | Public structural equality for the allocation structs |
| Requirements | 1.1–1.6 |
| Intent | Public structural equality for the allocation structs, with an optional first-difference description |
| Requirements | 1.1–1.7, 3.1–3.6 |
```c
/* src/include/xcealloc.h */
extern int alloc_eq(const xce_alloc_t* a, const xce_alloc_t* b);
extern int host_alloc_eq(const xce_host_alloc_t* a, const xce_host_alloc_t* b);
extern int job_alloc_eq(const xce_job_alloc_t* a, const xce_job_alloc_t* b);
extern int alloc_eq(const xce_alloc_t* a, const xce_alloc_t* b, char** why);
extern int host_alloc_eq(const xce_host_alloc_t* a, const xce_host_alloc_t* b,
char** why);
extern int job_alloc_eq(const xce_job_alloc_t* a, const xce_job_alloc_t* b,
char** why);
```
- **Preconditions**: none (NULL-safe).
- **Postconditions**: returns non-zero iff the two arguments are equal per the documented contract; does not mutate either argument.
- **Invariants**: `alloc_eq(a, b)` ignores `from` and `instid`; `host_alloc_eq`/`job_alloc_eq` are order-sensitive (both sides come from the same deterministic construction, so array order is expected to match, not just multiset).
- **Preconditions**: a non-zero entry count implies a non-NULL entry array (a malformed record is not handled). NULL top-level arguments, NULL list entries, and NULL string fields are handled. `why` may be NULL; `*why` on entry is ignored (never read or freed).
- **Postconditions**: returns non-zero iff the two arguments are equal per the documented contract; does not mutate either argument. If `why != NULL`: on equal, `*why = NULL`; on not-equal, `*why` is a `malloc`'d, NUL-terminated description of the first difference (see Data Models), sized by the function and never truncated; the caller owns it and releases it with `free()`. If that allocation fails, `*why = NULL` and the return value is unaffected.
- **Invariants**: `alloc_eq(a, b)` answers "same allocation decision", compares `from`, and ignores `instid`; the header comment states that this differs from `merge_job_alloc()`'s slot matching, where `instid` is significant; `host_alloc_eq`/`job_alloc_eq` are order-sensitive (both sides come from the same deterministic construction, so array order is expected to match, not just multiset). The result never depends on `why` or on whether the description could be allocated.
#### Pending-reason equality (`preason.c`)
| Field | Detail |
|-------|--------|
| Intent | Public full structural equality for `xce_job_preason_t` |
| Requirements | 2.1–2.3 |
| Intent | Public full structural equality for `xce_job_preason_t`, with an optional first-difference description |
| Requirements | 2.1–2.3, 3.1–3.6 |
```c
/* src/include/preason.h */
extern int job_preason_eq(const xce_job_preason_t* a, const xce_job_preason_t* b);
extern int job_preason_eq(const xce_job_preason_t* a,
const xce_job_preason_t* b,
char** why);
```
- **Postconditions**: returns non-zero iff `main_reason` matches, the host-based reason list matches in order (each entry by host name + reason), and the resource-based reason list matches in order (each entry by resource name + host name + reason); NULL-safe per the shared convention.
- **Postconditions**: returns non-zero iff `main_reason` matches, the host-based reason list matches in order (each entry by host name + reason), and the resource-based reason list matches in order (each entry by resource name + host name + reason); NULL-safe per the shared convention; `why` as for the allocation functions.
- **Invariants**: this is a full match — it is the honest "are these two pending reasons identical" predicate, not the simulator's replay loosening (see Decision 3).
#### Shared string helper (`utils.c`)
@ -181,25 +236,45 @@ extern int str_eq(const char* a, const char* b);
- **Postconditions**: returns non-zero iff `a` and `b` are both NULL, or both non-NULL and `strcmp(a, b) == 0`; zero otherwise.
### Scheduler
#### Recording (`driver.c`)
- Normal mode: `sched_record_session_begin(sched, sched->core.session_start)` replaces the separate `xce_core_now()` read, so the recorded `beginRound` is the session start.
- Simulation: no change here. Spec `scheduler-sim-discrete-clock` guarantees `core->session_start == begin_round` for a replayed session (clock frozen between recorded timestamps).
#### `from` stamping (`scheduler.c`)
- After `make_alloc()` at `:1893`: `allocs[i]->from = sched->core.session_start / 1000000000ULL` (seconds, floor).
#### Simulator verification (`sim_verify.c`)
- `int sim_decision_eq(const replay_decision_t* expected, const xce_decision_t* actual, char** why)`: DISPATCH calls `job_alloc_eq(..., why)`; PEND sets `*why` to `jobid A != B` or `main_reason A != B`; kind mismatch sets `kind DISPATCH != PEND`. Same ownership contract as the libxce functions.
- `sim_log_mismatch` calls `sim_decision_eq(expected, actual, &why)`, appends ` -- diff: <why>` to its existing line (or ` -- diff: <unavailable>` if `why` is NULL), then `free(why)`.
## Data Models
No struct is added or modified. The equality contract itself is the only new "model", captured in the field lists below.
No struct is added or modified. The equality contract and the description format are the only new "models".
- `alloc_eq` compares: `value` (exact `==`, documented bit-exact-safe because both sides are exact copies of the same `double`), `duration`, `resname`, `reserve`. **Excluded**: `from` (wall-clock `time(NULL)` stamp), `instid` (per-process `nextid++` counter) — both runtime bookkeeping, not identity.
- `host_alloc_eq` compares: `host`, then `allocs[i]` in order for `i < nallocs`.
- `job_alloc_eq` compares: `jobid`, then `allocs[i]` in order for `i < nhosts`, then `allocs_shared[i]` in order for `i < nres_shared`.
- `job_preason_eq` compares: `main_reason`; then `host_based_reasons[i]` in order for `i < nhost_reasons`, each entry by `host` and `reason`; then `res_based_reasons[i]` in order for `i < nres_reasons`, each entry by `resource`, `host`, and `reason`. **Nothing is excluded** — this is a full structural match.
- `alloc_eq` compares, in order: `resname`, `reserve`, `value` (exact `==`, documented bit-exact-safe because both sides are exact copies of the same `double`), `duration`, `from`. **Excluded**: `instid` (per-process `nextid++` counter).
- `host_alloc_eq` compares: `host`, then `nallocs`, then `allocs[i]` in order.
- `job_alloc_eq` compares: `jobid`, `nhosts`, `nres_shared`, then `allocs[i]` in order, then `allocs_shared[i]` in order.
- `job_preason_eq` compares: `main_reason`; then `nhost_reasons` and `host_based_reasons[i]` in order, each entry by `host` and `reason`; then `nres_reasons` and `res_based_reasons[i]` in order, each entry by `resource`, `host`, and `reason`. **Nothing is excluded.**
- **Description format** (first difference only, path from the outermost struct compared):
- scalar: `allocs[1](host=h1).allocs[0](resname=mem).value: 2 != 3`
- string: `allocs[0](host=h0).host: "h0" != "h2"`
- NULL: `allocs_shared[0]: NULL != non-NULL`
- length: `nhosts: 1 != 2`
- `jobid` formatted with libxce's own `fmt_jobid()` (`utils.h`), so it reads like the rest of the scheduler log; `from` as integer seconds; `value` with `%.17g` so an exact-compare difference is always visible.
## Error Handling
These are pure read-only functions with no allocation, no I/O, and no failure path. The only "error" case is NULL input, which is handled by the shared convention (identical pointers including both-NULL → equal; exactly one NULL → not-equal) rather than an error return. No logging is added.
These are read-only functions with no I/O and no failure path for the comparison itself. The only allocation is the optional description; if it fails, `*why` is NULL and the result is unchanged. The only "error" case is NULL input, which is handled by the shared convention (identical pointers including both-NULL → equal; exactly one NULL → not-equal) rather than an error return. A malformed record (non-zero count, NULL array) is a documented precondition violation, not a handled case. No logging is added; the description is built with `vasprintf` (glibc, already used under `_GNU_SOURCE`), and returned to the caller.
## Testing Strategy
- **Unit (libxce)**: `test_xcealloc.c` — match/mismatch on each compared field, the `from`/`instid` exclusion, ordered-list sensitivity, and NULL handling at top and nested levels; `test_preason.c` — full-match: main-reason match/mismatch, host-based reason list match/mismatch (per field + order), resource-based reason list match/mismatch (per field + order), and NULL handling.
- **Unit (scheduler)**: `test_sim_verify.c` keeps `sim_decision_eq`-level tests (DISPATCH/PEND match and mismatch, kind mismatch, PREEMPT always-false, NULL defense) and drops the DISPATCH alloc-level tests that now belong to libxce. The PEND tests stay because they pin the simulator-local main-reason-only behavior.
- **Integration (scheduler)**: `test_sim_verify.c`'s `--sim-verify` round-trips (zero-mismatch self-replay, exact-mismatch altered replay, reconcile cases) continue to assert unchanged behavior, proving the relocation is behavior-preserving.
- **Regression**: `test_sim_equivalence.c`, `test_repl_equivalence.c`, `test_repl_limit_table.c` must stay green after migrating to the public API.
- **Unit (libxce)**: `test_xcealloc.c` — match/mismatch on each compared field with its expected description, `instid` ignored, `from` compared, ordered-list sensitivity, list-length mismatch, NULL handling at top level, nested entries, and string fields, NULL `why`, `*why == NULL` on equal, a failed description allocation (`--wrap` on `vasprintf` or `malloc` in a dedicated test binary: `*why` NULL, same result); every test frees the returned description, so ASAN/LeakSanitizer proves ownership; an existing `merge_job_alloc` test still passes. `test_preason.c` — full-match: main-reason, host-based and resource-based lists per field + order + length, NULL handling, description cases; plus `str_eq` NULL/NULL and NULL/non-NULL.
- **Unit (scheduler)**: `test_sim_verify.c` keeps `sim_decision_eq`-level tests (DISPATCH/PEND match and mismatch, kind mismatch, PREEMPT always-false, NULL defense), asserts the mismatch description, and drops the alloc-level tests now owned by libxce. A session-level test asserts every allocation of one session (including a COMPOUND merge) carries `session_start / 1e9`, and that the recorded `beginRound` equals `session_start`.
- **Integration (scheduler)**: `test_sim_verify.c`'s `--sim-verify` round-trips (zero-mismatch self-replay, exact-mismatch altered replay, reconcile cases); the self-replay reports zero mismatches. Until `scheduler-sim-discrete-clock` lands, run them at `--sim-speed 0`.
- **Regression**: `test_sim_equivalence.c`, `test_repl_equivalence.c`, `test_repl_limit_table.c` stay green under the stricter checks (full-match preason, `from` compared). A failure there is investigated as a real divergence.
- **Live**: record a session on a local cluster, replay it with `--sim-verify` at the default speed: zero mismatches.
## Design Decisions
@ -220,19 +295,46 @@ These are pure read-only functions with no allocation, no I/O, and no failure pa
### Decision 3: `job_preason_eq` is a full structural match; the simulator keeps its own main-reason-only PEND check
- **Context**: the current `sim_verify_job_preason_eq` compares only `main_reason`, deliberately (bug-1845): the simulator has no alloc cache, so its full-detail PEND can never match a recorded cache-hit's main-reason-only shape, and requiring detail to match produced systematic false mismatches. That narrowing is a property of *recorded-vs-live replay comparison*, not of "are these two pending-reason objects identical".
- **Alternatives**: (a) publish a full-deep `job_preason_eq` and keep the main-reason-only comparison as a simulator-local static helper (selected); (b) publish main-reason-only `job_preason_eq` (rejected — leaks a replay artifact into a general-purpose equality API and contradicts the identity semantics a controller engineer would expect).
- **Selected**: `job_preason_eq(a, b)` returns non-zero iff `main_reason` matches and both detail lists match in order, field-for-field. The simulator's `sim_decision_eq` PEND branch keeps a small `static` main-reason-only helper (documented with the bug-1845 rationale) instead of calling `job_preason_eq`.
- **Selected**: `job_preason_eq(a, b, why)` returns non-zero iff `main_reason` matches and both detail lists match in order, field-for-field. The simulator's `sim_decision_eq` PEND branch keeps a small `static` main-reason-only helper (documented with the bug-1845 rationale) instead of calling `job_preason_eq`.
- **Rationale**: a public "same pending reason" predicate must be identity-honest — a controller comparing two pending reasons will want the detail to matter. The replay asymmetry is a scheduler-only concern and stays where it belongs.
- **Trade-offs**: two PEND-comparison shapes now exist (full-match in libxce, main-reason-only in sim_verify.c). This is intentional: they answer different questions ("identical objects" vs. "same replay outcome"). The sim-local helper is `static` and small, so the duplication is negligible and clearly scoped.
- **Follow-up**: none expected; the sim-local helper is not exported.
### Decision 4: Preserve NULL and order semantics exactly
### Decision 4: Preserve NULL and order semantics
- **Context**: NULL handling and list-order sensitivity.
- **Selected**: keep today's behavior — identical pointer (including both NULL) → equal; exactly one NULL → not-equal; ordered (not multiset) list comparison.
- **Rationale**: the point of the relocation is zero behavioral change; both sides of every current comparison come from the same deterministic construction, so order-sensitivity is correct. The full-match `job_preason_eq` uses the same convention.
- **Selected**: keep today's behavior — identical pointer (including both NULL) → equal; exactly one NULL → not-equal; ordered (not multiset) list comparison. A malformed record (non-zero count, NULL array) is not handled — it is a documented precondition, matching today's comparators.
- **Rationale**: both sides of every current comparison come from the same deterministic construction, so order-sensitivity is correct. The new full-match `job_preason_eq` uses the same convention.
### Decision 5: Promote the NULL-safe string comparison to a shared `str_eq` in `utils.{c,h}`
- **Context**: with `job_preason_eq` now a full match, both `xcealloc.c` (host/resname/reserve strings) and `preason.c` (host/resource strings) need NULL-safe string comparison. Keeping it `static` in one file forces the other to duplicate it.
- **Alternatives**: (a) shared `str_eq` in `utils.{c,h}` (selected); (b) duplicate a `static` helper in each of the two files.
- **Selected**: add `int str_eq(const char* a, const char* b)` to `utils.{c,h}`, beside the existing `float_equal` helper.
- **Rationale**: one consumer each would have justified a `static`; two consumers and a third on the horizon (controller) justify a shared helper. `utils.h` already centralizes small generic C helpers (`fmt_jobid`, `float_equal`).
- **Trade-offs**: a new public symbol in `utils.h`; negligible surface, and it will be reused by the controller when it adopts these comparators.
- **Rationale**: two consumers and a third on the horizon (controller) justify a shared helper. `utils.h` already centralizes small generic C helpers (`fmt_jobid`, `float_equal`). It could also make `same_alloc_slot()` NULL-safe when bug-1882 is fixed.
- **Trade-offs**: a new public symbol in `utils.h`; negligible surface.
### Decision 6: `alloc_eq` and `same_alloc_slot` stay separate
- **Context**: bug-1863 added a private `same_alloc_slot(a, b)` to `xcealloc.c` (`resname` + `instid` + `reserve`), used by `merge_job_alloc()` to decide whether a source allocation should be added into an existing destination entry. `alloc_eq` will live in the same file and also compares `xce_alloc_t` records.
- **Alternatives**: (a) keep them separate and document the difference (selected); (b) implement `same_alloc_slot` on top of `alloc_eq` or vice versa.
- **Selected**: keep them separate. `alloc_eq` answers "do these two records describe the same allocation decision" (resname, reserve, value, duration, from; `instid` excluded). `same_alloc_slot` answers "are these the same reservation slot to accumulate into" (resname, instid, reserve; `value` excluded because merging adds values). The `alloc_eq` header comment names the difference.
- **Rationale**: the two predicates disagree on `instid`, `value`, `duration`, and `from`, so neither can be expressed through the other without changing behavior. `merge_job_alloc()` is out of scope.
- **Follow-up**: `same_alloc_slot` calls `strcmp(a->resname, …)` without a NULL check, and `resname` is NULL when `make_alloc()`'s unchecked `strdup()` fails; tracked as bug-1882, not part of this spec.
### Decision 7: The three test files adopt the full-match `job_preason_eq`
- **Context**: `test_sim_equivalence.c`, `test_repl_equivalence.c`, and `test_repl_limit_table.c` compare two decoded `replay_decision_t` entries and today call the main-reason-only `sim_verify_job_preason_eq`.
- **Alternatives**: (a) switch them to the full-match `job_preason_eq` (selected); (b) give them their own main-reason-only helper.
- **Selected**: (a). None of these tests registers `unsatisfiable_lookup_cb` (the alloc cache), so both sides of every pending-reason comparison are produced by the same full per-host computation and their detail lists are expected to match. The bug-1845 asymmetry (recorded cache-hit vs. live full detail) does not apply.
- **Rationale**: the stricter check catches detail divergence the old one missed, at no false-positive cost for these fixtures. Each call site gets a short comment stating why full match is valid there.
- **Trade-offs**: if a future test records with the alloc cache active, it must use a main-reason-only comparison instead; the call-site comment and the Revalidation Trigger on pending-reason construction paths cover this.
### Decision 8: `from` is the deciding session's start, from one clock reading
- **Context**: `alloc_eq` must compare `from`, but `make_alloc()` stamps `time(NULL)`, which a replay cannot reproduce (bug-1838-followup). The scheduler already has a virtualized session start (`core->session_start`), but it is a different clock read from the recorded `beginRound`, and in simulation at `--sim-speed` ≥ 1 it drifts past the recorded `beginRound`.
- **Alternatives**: (a) `from = session_start / 1e9`, with the recorded `beginRound` taken from `session_start`; the simulator clock stops drifting (spec `scheduler-sim-discrete-clock`) so a replayed `session_start` is exactly the recorded `beginRound` (selected); (a′) as (a), but with an explicit `xce_core_begin_session_at(core, begin_round)` in the simulator instead of removing the drift (superseded: the user chose to remove `--sim-speed`, which fixes the drift for every clock reader, not just `from`); (b) keep excluding `from` from equality (rejected by the user: `from` is part of what was decided); (c) `from = xce_core_now()` at allocation time (rejected: at `--sim-speed` ≥ 1 it drifts with replay pace, and a second-boundary crossing produces a false mismatch); (d) change `make_alloc()`'s signature to take `from` (rejected: churns the controller and ~20 test callers for no benefit; the scheduler is the only producer that needs a different value).
- **Selected**: (a). The recording's `beginRound` is taken from `core->session_start`; `scheduler.c` overwrites `from` after `make_alloc()`; the simulator side comes from `scheduler-sim-discrete-clock`.
- **Rationale**: every allocation of one session gets the same `from`, which a replay reproduces bit-for-bit. `from` changes meaning only slightly — from "second the record was built" to "second the deciding session started", normally the same second and at most one session's duration earlier.
- **Trade-offs**: recorded `beginRound` moves a few milliseconds earlier (to `xce_core_begin_session`, before `plugin_begin_session`/`sched_account_jobs`). Recordings made before this change carry wall-clock `from` values and may show a false DISPATCH mismatch when an allocation's `time(NULL)` fell in a later second than its session start; the difference description names `from`, so these are recognisable. `make_alloc()` keeps `time(NULL)` for its other callers.
### Decision 9: First-difference description returned through `char** why`, owned by the caller
- **Context**: a yes/no comparator leaves a mismatch undiagnosable (bug-1881).
- **Alternatives**: (a) optional `char** why`: the function allocates a string sized to fit and hands ownership to the caller (selected, per the user); (b) caller buffer `char* why, size_t whylen` (superseded: the caller must guess a size, and a long path through many hosts gets truncated); (c) a separate `*_diff()` function family; (d) a list of every difference; (e) a callback per difference.
- **Selected**: (a). One code path computes both the result and the explanation, so they cannot disagree; callers that do not care pass `NULL`. On not-equal the function sets `*why` to a `vasprintf`'d string; on equal, or if the allocation fails, `*why = NULL`. The caller frees with `free()`.
- **Rationale**: the function knows the length; the caller only decides whether to ask. Allocation happens only on the not-equal path with `why` requested, i.e. only when something is about to be reported, so the equal fast path stays allocation-free. The first difference is the one that matters for a deterministic replay; later ones usually follow from it.
- **Trade-offs**: every caller that asks must free (ASAN/LeakSanitizer in the test suites catches a missed `free`); only the first difference is reported.