myxce/.kiro/specs/libxce-alloc-preason-equality/design.md

# Design Document

## Overview

**Purpose**: This feature delivers a single, shared implementation of structural equality for the scheduler's allocation and pending-reason data structures, published as public libxce API, that can explain *why* two objects differ, together with a reproducible allocation start time (`from`) so that `from` can take part in equality.

**Users**: Scheduler engineers (today: `--sim-verify` replay checking and three scheduler test files) and, in future, controller engineers (`persist.c`, `resync.c`, `job.c` already carry these structs) who need to ask "do these two objects represent the same thing, and if not, where do they differ".

**Impact**:
- Moves the three `static` allocation comparators out of `src/scheduler/src/sim_verify.c` into `src/lib/xcealloc.c`. `alloc_eq` now compares `from` and excludes only `instid`.
- Adds a **new** full-structural-match pending-reason comparator to `src/lib/preason.c`. No full-match comparator exists today; bug-1845 narrowed the simulator's to main-reason-only.
- Gives every comparator an optional `char** why` out-parameter that returns a newly allocated description of the first difference, owned by the caller. `sim_log_mismatch` prints it (resolves bug-1881).
- Changes the scheduler to stamp `from` with the scheduling session's start time, a single clock reading shared with the recorded `beginRound`, instead of `time(NULL)`. That a replayed session starts at exactly the recorded `beginRound` comes from the separate spec `scheduler-sim-discrete-clock` (removes `--sim-speed`; the simulator clock moves only at recorded timestamps).
- Switches the simulator and three white-box test consumers to the public API. The three test files' checks become stricter: full-match pending reasons (Decision 7) and `from`-inclusive allocations.
- `sim_decision_eq` and `sim_log_mismatch` stay in the scheduler, and `sim_decision_eq`'s PEND branch keeps a simulator-local main-reason-only comparison (bug-1845). No wire format or struct layout changes.

### Goals
- Publish `alloc_eq`, `host_alloc_eq`, `job_alloc_eq` in `xcealloc.h` and `job_preason_eq` in `preason.h`, each with an optional first-difference description.
- Make `job_preason_eq` a full structural match (main reason + host-based reasons + resource-based reasons, in order).
- Make `from` reproducible under replay, then compare it.
- Make `--sim-verify` mismatch logs name the differing field and values.
- Make the existing consumers call the public API instead of `sim_verify.c`'s private `static` comparators. (The three test files keep their white-box `#include "../src/sim_verify.c"`: they also white-box-include `sim_loop.c`, which calls `sim_decision_eq`/`sim_log_mismatch`.)
- Keep `--sim-verify` free of false mismatches for recordings made after this change.
- Add libxce unit tests that pin the `instid` exclusion, `from` inclusion, full-match preason, description, and NULL-handling semantics.

### Non-Goals
- Do not move `sim_decision_eq`, `sim_log_mismatch`, or the `describe_*` helpers (simulator-specific; depend on capnp/replay types and `ERROR()`).
- Do not change the simulator's PEND outcome check from main-reason-only to full-match — that would reintroduce bug-1845's false mismatches on cache-hit PENDs (see Decision 3).
- Do not introduce any new comparison call site beyond the existing four consumers.
- Do not touch the controller's persistence/resync code paths, or how the controller/worker interpret `from`.
- Do not change `make_alloc()`'s signature or its own `time(NULL)` default (controller and many tests call it); the scheduler overwrites `from` after construction (Decision 8).
- Do not touch `merge_job_alloc()`/`same_alloc_slot()` (bug-1863); `alloc_eq` neither replaces nor reuses `same_alloc_slot` (see Decision 6). (Their NULL-dereference defect, bug-1882, is already fixed in f9146864.)
- Do not remove the test files' white-box `#include "../src/sim_verify.c"`.
- Do not make recordings made before this change verify cleanly; their wall-clock `from` values are an accepted limitation.
- Do not change the simulator's clock or `--sim-speed`; that is spec `scheduler-sim-discrete-clock`, which this spec depends on.

## Boundary Commitments

### This Spec Owns
- The public equality functions `alloc_eq`, `host_alloc_eq`, `job_alloc_eq` (`xcealloc.{c,h}`) and `job_preason_eq` (`preason.{c,h}`), their equality contracts and their difference-description format.
- The shared NULL-safe string-equality helper both modules need (see Decision 5).
- The scheduler's stamping of `from` (`sched_make_primary_alloc_()` in scheduler.c, from the session core) and the single normal-mode session-start reading shared by `core->session_start`, the recorded `beginRound`, and `from` (driver.c).
- The migration of `sim_verify.c` and the three white-box test files onto those functions, and the difference text in `sim_log_mismatch`.
- libxce unit tests for the new functions (`test_xcealloc.c`, `test_preason.c`) and scheduler tests for `from` stamping.

### Out of Boundary
- The decision-level comparison and mismatch reporting structure that remain in `sim_verify.c`, including the main-reason-only PEND outcome check (simulator-local).
- The `.replay` wire format and the `replay_decision_t`/`xce_decision_t` types.
- Any change to the controller's consumption of these structs.
- `merge_job_alloc()` and its `same_alloc_slot()` slot-matching helper in `xcealloc.c`.
- The simulator clock and `--sim-speed` (spec `scheduler-sim-discrete-clock`).

### Allowed Dependencies
- `src/include/xce/types.h` (the struct definitions — unchanged).
- `xcealloc.c`/`xcealloc.h`, `preason.c`/`preason.h`, and `utils.{c,h}` (the modules that already own construction of these structs, plus the shared string helper).
- `string.h` (`strcmp`) and `stdio.h` (`vasprintf`, a GNU extension: `xcealloc.c`/`preason.c` define `_GNU_SOURCE` before their first include, as `client_failover.c:14` does) for comparisons and descriptions.
- Scheduler: `driver.c`, `scheduler.c`, `sim_verify.c`, `sim_loop.c` (the `sim_decision_eq`/`sim_log_mismatch` call sites only).
- Spec `scheduler-sim-discrete-clock`: after it, a replayed session's `core->session_start` equals the recorded `beginRound` exactly.

### Revalidation Triggers
- A change to the `xce_alloc_t`/`xce_host_alloc_t`/`xce_job_alloc_t`/`xce_job_preason_t` struct layouts must re-check the field lists, the exclusion documentation, and the description text.
- A change to `same_alloc_slot()`'s matching rule, or a proposal to merge it with `alloc_eq`, must re-check Decision 6 (the two answer different questions and disagree on `instid`, `from`, and `value`).
- A change to the pending-reason construction paths (cache-hit vs. full-computation) that alters whether detail lists are populated must re-check the simulator's main-reason-only PEND rationale (bug-1845) and the full-match `job_preason_eq` contract.
- Any new allocation path that does not go through `sched_make_primary_alloc_()`, a second clock read between session start and recording, or a change to `scheduler-sim-discrete-clock`'s clock guarantee, must re-check Decision 8.

## Architecture

### Existing Architecture Analysis
`src/scheduler/src/sim_verify.c` (474 lines) owns two layers:
1. **Type-level comparators** — five `static` functions (`sim_verify_str_eq`, `sim_verify_alloc_eq`, `sim_verify_host_alloc_eq`, `sim_verify_job_alloc_eq`, `sim_verify_job_preason_eq`) that walk two libxce structs field-by-field and return only 0/1. They have zero dependency on the scheduler, capnp, or the replay layer. `sim_verify_alloc_eq` excludes `from` (bug-1838-followup) and `instid` (bug-1844). `sim_verify_job_preason_eq` compares only `main_reason` (bug-1845), a simulator replay asymmetry rather than an object-identity property.
2. **Decision-level dispatcher + reporting** — `sim_decision_eq` (dispatches on `xce_decision_t.op` vs `ReplayDecision_body_which`), `sim_log_mismatch`, and two `describe_*` helpers. These depend on `replay_decision_t`, capnp enums, and `ERROR()`. `sim_log_mismatch` prints only jobid/nhosts for each side, so a mismatch below that level is invisible (bug-1881).

The four allocation/pending-reason structs are defined in `src/include/xce/types.h` and already constructed/deep-copied by `src/lib/xcealloc.c` (`make_alloc`, `make_host_alloc`, `make_job_alloc`, `dup_job_alloc`) and `src/lib/preason.c` (`add_job_preason`, `make_pendreason`). The controller consumes these same structs in `persist.c`, `resync.c`, and `job.c` with no equality API available.

Since this design was first drafted, bug-1863 (`aea34738`) added `merge_job_alloc()` to `xcealloc.c` with a private `same_alloc_slot()` helper that matches two `xce_alloc_t` records by `resname`, `instid`, and `reserve`, to decide whether they are the same reservation slot to merge. It is a near neighbour of `alloc_eq` but has a different contract (see Decision 6).

**How `from` is set today.** `make_alloc()` (`xcealloc.c`) stamps `from = time(NULL)`. The scheduler's only production constructor call is in `sched_make_allocs()` (`scheduler.c`), called by the seqjob/parjob allocators' `my_alloc()` inside a scheduling session. `from` is on the wire (`controller.capnp` `from @4: UInt64 # alloc start time`) and `dup_alloc()` copies it. Neither the controller nor the worker reads an allocation's `from` today.

**Which core `sched_make_allocs()` sees.** Its first argument is `plugin->sched`. In normal mode that is the real scheduler, whose `core` runs the sessions. In simulation it is **not**: `sim_alloc.c` (`sim_bootstrap_seqjob_allocator()`/`..._parjob_...`) hands the allocators a throwaway `xce_scheduler_t` wrapper with its own `xce_sched_core_init()`'d core, aliasing only `resmgr`; that core never begins a session, so its `session_start` is 0. The honest session core is the `core` argument threaded through `sched_make_allocation()` → `sched_make_primary_alloc_(core, ctx)` (`sim->core` in simulation, `sched->core` in normal mode). Every allocation — PRIMARY, each primary of a COMPOUND, each alternative of an ALTER — is produced by one `sched_make_primary_alloc_()` call.

**How session time is read today.** Two separate clock reads per session:
- `xce_core_begin_session()` sets `core->session_start = xce_core_now(core)` (`core_session.c:429`).
- The recording then reads the clock again for `beginRound`: `driver.c:1030` after `plugin_begin_session`/`sched_account_jobs`, and `sim_loop.c` before `xce_core_begin_session`.

In simulation, `sim_advance_virtual_clock(sim, begin_round)` re-anchors the virtual clock to the recorded `beginRound`. At `--sim-speed` ≥ 1 the clock keeps advancing from there, so `session_start` lands slightly after `begin_round`. At `--sim-speed 0` it is frozen and exact. The simulator never sleeps, so `--sim-speed` paces nothing; it only adds drift. Spec `scheduler-sim-discrete-clock` removes it, which makes `session_start == begin_round` in every replay.

Three scheduler test files already reuse two of the leaf comparators — but via white-box `#include "../src/sim_verify.c"` + direct calls to the `static` functions: `test_sim_equivalence.c` (lines 894, 902), `test_repl_equivalence.c` (lines 734, 742), and `test_repl_limit_table.c` (lines 2662, 2670).

### Architecture Pattern & Boundary Map

```mermaid
flowchart LR
    subgraph scheduler
        SV["sim_verify.c\n(sim_decision_eq, sim_log_mismatch, describe_*,\nmain-reason-only PEND outcome check)"]
        DR["driver.c\nbeginRound = core.session_start"]
        SC["scheduler.c\nalloc->from = session_start / 1e9"]
        CS["core_session.c\nsession_start"]
        DC["spec scheduler-sim-discrete-clock\nreplayed session_start == recorded beginRound"]
        TE["test_sim_equivalence.c / test_repl_equivalence.c / test_repl_limit_table.c"]
        TSV["test_sim_verify.c"]
    end
    subgraph libxce
        XA["xcealloc.{c,h}\nalloc_eq / host_alloc_eq / job_alloc_eq (+why)"]
        PR["preason.{c,h}\njob_preason_eq (full match, +why)"]
        UT["utils.{c,h}\nstr_eq (NULL-safe)"]
    end
    SV --> XA
    TE --> XA
    TE --> PR
    XA --> UT
    PR --> UT
    DR --> CS
    SC --> CS
    DC -.-> CS
```

- Domain boundary: the library owns *struct-level* equality and its explanation (full structural match); the simulator owns *decision-level* comparison and reporting, including its one deliberate loosening (main-reason-only PEND). The scheduler core owns the session-start value that makes `from` reproducible.
- Steering compliance: no changes to `types.h` layouts or the wire schema; `from` keeps its unit and meaning ("alloc start time"), now defined as the start of the deciding session. Persistence: `from` already round-trips through the snapshot (`dup_job_alloc`) and the controller's capnp encoding, so no new save/restore field is introduced.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| Language | C (gnu17) | equality functions, `from` stamping, tests | unchanged |
| Build | Meson via `xce-build` | compile libxce + scheduler, run `src/lib/testsuite` and `src/scheduler/testsuite` | unchanged |
| Test | Unity | `test_xcealloc.c`, `test_preason.c`, scheduler tests | existing harnesses |

## File Structure Plan

### Modified Files
- `src/lib/utils.c` / `src/include/utils.h` — add a NULL-safe `str_eq(const char*, const char*)` helper (see Decision 5).
- `src/lib/xcealloc.c` / `src/include/xcealloc.h` — add `alloc_eq`, `host_alloc_eq`, `job_alloc_eq` with the optional `char** why` description.
- `src/lib/preason.c` / `src/include/preason.h` — add `job_preason_eq` (full match) with the optional description.
- `src/scheduler/src/driver.c` — record `beginRound` from `sched->core.session_start` instead of a second `xce_core_now()` read.
- `src/scheduler/src/scheduler.c` — in `sched_make_primary_alloc_(core, ctx)`, when the allocator returns `rc > 0` with `ctx->alloc != NULL`, set `from = core->session_start / 1000000000ULL` on every `xce_alloc_t` in `ctx->alloc` (per-host and shared). Not in `sched_make_allocs()`: in simulation its `sched` is `sim_alloc.c`'s wrapper, whose core has no session.
- `src/scheduler/src/sim_verify.c` — delete the five `static` comparators; `sim_decision_eq` gains an optional `char** why` and calls `job_alloc_eq` for DISPATCH; keep a simulator-local `static` main-reason-only helper for PEND; `sim_log_mismatch` gains a `const char* why` parameter and appends it. Update the header comments that describe `from` as excluded.
- `src/scheduler/src/sim_loop.c` — update both `extern` declarations; in the per-decision loop call `sim_decision_eq(expected, d, &why)`, pass `why` to `sim_log_mismatch`, then `free(why)`; the leftover-recorded loop (recorded decisions with no live counterpart) passes the string literal `"no live decision"`, which it does not free.
- `src/lib/testsuite/test_xcealloc.c`, `src/lib/testsuite/test_preason.c` — unit tests.
- `src/scheduler/testsuite/test_sim_equivalence.c`, `test_repl_equivalence.c`, `test_repl_limit_table.c` — replace direct calls with `job_alloc_eq(..., NULL)`/`job_preason_eq(..., NULL)` (or take the description, print it on failure, and free it); add `#include "xcealloc.h"`/`#include "preason.h"`; comment at each `job_preason_eq` call why full match is valid (Decision 7); keep the white-box `#include "../src/sim_verify.c"`.
- `src/scheduler/testsuite/test_sim_verify.c` — add the `why` argument (`NULL`) to its 16 direct `sim_decision_eq()` calls; drop the DISPATCH alloc-level tests now owned by libxce; invert `test_dispatch_decisions_differing_from_timestamp_still_match` (a `from` difference is now a mismatch, and `why` names `from`); add a test that the mismatch description names the differing field; keep PEND/decision-level and integration tests.
- Hand-built recorded DISPATCH fixtures that `--sim-verify` or an equivalence test compares against a live replay — they build the recorded allocation with `make_alloc()` (`from = time(NULL)`) or a literal `from`, which no longer matches the replay's `session_start / 1e9`: set the recorded `from` to the fixture session's `begin_round / 1000000000`. Audit every test with a hand-built `ReplayDecision_body_dispatch`: `test_sim_live_allocation.c` (the `make_alloc()` fixture near line 726, `begin_round = 2000` → `from = 0`), `test_sim_contention_parity.c`, `test_sim_resource_contention.c`, `test_sim_round_boundary_reconciliation.c`, `test_sim_reconcile_unaccounted.c`, `test_sim_verify.c`; fixtures that are never compared against a live allocation need no change.
- A scheduler test (new or in an existing session/allocation test file) — allocations made in one session carry `session_start / 1e9` as `from`, including in simulation (the `sim_alloc.c` wrapper path), and the recorded `beginRound` equals `session_start`.
- `src/scheduler/testsuite/test_sim_live_allocation.c`, `test_sim_reconcile_unaccounted.c`, and the header comments of `test_sim_equivalence.c`/`test_repl_equivalence.c` — update comments that name `sim_verify_*_eq` or describe `from` as excluded.

## System Flows

```mermaid
flowchart TD
    A["sim_decision_eq(expected, actual, why)"] --> B{"actual->op"}
    B -->|DISPATCH| C["job_alloc_eq(expected alloc, actual alloc, why)"]
    B -->|PEND| D["jobid equal AND main_reason equal (sim-local, NOT job_preason_eq); why names which"]
    B -->|PREEMPT| E["log + report mismatch (unchanged)"]
    C --> F["host_alloc_eq per host"] --> G["alloc_eq per resource (incl. from, excl. instid)"]
    D --> H["main-reason only, detail ignored (bug-1845)"]
    C --> M["sim_loop: sim_log_mismatch(..., why); free(why)"]
    D --> M
```

`from` stamping per session:

```mermaid
sequenceDiagram
    participant Drv as driver.c
    participant Core as core_session.c
    participant Sch as scheduler.c sched_make_primary_alloc_(core, ctx)
    participant Rec as recording
    Drv->>Core: xce_core_begin_session()
    Core-->>Core: session_start = xce_core_now()<br/>(replay: == recorded beginRound, via scheduler-sim-discrete-clock)
    Drv->>Rec: beginRound = session_start
    Sch->>Sch: allocator builds ctx->alloc (make_alloc: from = time(NULL))
    Sch->>Sch: every alloc in ctx->alloc: from = core->session_start / 1e9
```

## Requirements Traceability

| Requirement | Summary | Components | Interfaces |
|-------------|---------|------------|------------|
| 1.1–1.3 | allocation equality for `xce_alloc_t`/`xce_host_alloc_t`/`xce_job_alloc_t` | `xcealloc.c` | `alloc_eq`, `host_alloc_eq`, `job_alloc_eq` |
| 1.4 | exclude `instid` only; document difference from `merge_job_alloc` | `xcealloc.c/.h` | `alloc_eq` field list + header comment |
| 1.5–1.7 | NULL handling, malformed-record precondition, read-only | all four functions | const params |
| 2.1–2.3 | pending-reason full structural equality | `preason.c` | `job_preason_eq` |
| 3.1–3.6 | first-difference description, caller-owned | all four functions | `char** why` |
| 4.1–4.2, 4.4–4.5 | reproducible `from` | `driver.c`, `scheduler.c` `sched_make_primary_alloc_()` | `core->session_start` |
| 4.3 | replayed session start == recorded `beginRound` | spec `scheduler-sim-discrete-clock` | — |
| 5.1 | simulator uses libxce allocation comparison | `sim_verify.c` | `job_alloc_eq` |
| 5.2 | simulator-local main-reason-only PEND check | `sim_verify.c` | local static helper |
| 5.3 | mismatch log names the difference (bug-1881) | `sim_verify.c`, `sim_loop.c` | `sim_decision_eq(..., why)`, `sim_log_mismatch(..., why)` |
| 5.4 | zero-mismatch self-replay | integration tests | — |
| 6.1–6.3 | single shared implementation; test files use full-match preason; include stays | `sim_verify.c` + 3 test files | public API |
| 6.4–6.6 | libxce + scheduler tests, no other behavior change | test files | — |
| 6.7 | stale comment references updated | scheduler sources/tests | — |

## Components and Interfaces

### libxce

#### Allocation equality (`xcealloc.c`)

| Field | Detail |
|-------|--------|
| Intent | Public structural equality for the allocation structs, with an optional first-difference description |
| Requirements | 1.1–1.7, 3.1–3.6 |

```c
/* src/include/xcealloc.h */
extern int alloc_eq(const xce_alloc_t* a, const xce_alloc_t* b, char** why);
extern int host_alloc_eq(const xce_host_alloc_t* a, const xce_host_alloc_t* b,
                         char** why);
extern int job_alloc_eq(const xce_job_alloc_t* a, const xce_job_alloc_t* b,
                        char** why);
```

- **Preconditions**: a non-zero entry count implies a non-NULL entry array (a malformed record is not handled). NULL top-level arguments, NULL list entries, and NULL string fields are handled. `why` may be NULL; `*why` on entry is ignored (never read or freed).
- **Postconditions**: returns non-zero iff the two arguments are equal per the documented contract; does not mutate either argument. If `why != NULL`: on equal, `*why = NULL`; on not-equal, `*why` is a `malloc`'d, NUL-terminated description of the first difference (see Data Models), sized by the function and never truncated; the caller owns it and releases it with `free()`. If that allocation fails, `*why = NULL` and the return value is unaffected.
- **Invariants**: `alloc_eq(a, b)` answers "same allocation decision", compares `from`, and ignores `instid`; the header comment states that this differs from `merge_job_alloc()`'s slot matching, where `instid` is significant; `host_alloc_eq`/`job_alloc_eq` are order-sensitive (both sides come from the same deterministic construction, so array order is expected to match, not just multiset). The result never depends on `why` or on whether the description could be allocated.

#### Pending-reason equality (`preason.c`)

| Field | Detail |
|-------|--------|
| Intent | Public full structural equality for `xce_job_preason_t`, with an optional first-difference description |
| Requirements | 2.1–2.3, 3.1–3.6 |

```c
/* src/include/preason.h */
extern int job_preason_eq(const xce_job_preason_t* a,
                          const xce_job_preason_t* b,
                          char** why);
```

- **Postconditions**: returns non-zero iff `main_reason` matches, the host-based reason list matches in order (each entry by host name + reason), and the resource-based reason list matches in order (each entry by resource name + host name + reason); NULL-safe per the shared convention; `why` as for the allocation functions.
- **Invariants**: this is a full match — it is the honest "are these two pending reasons identical" predicate, not the simulator's replay loosening (see Decision 3).

#### Shared string helper (`utils.c`)

```c
/* src/include/utils.h */
extern int str_eq(const char* a, const char* b);
```

- **Postconditions**: returns non-zero iff `a` and `b` are both NULL, or both non-NULL and `strcmp(a, b) == 0`; zero otherwise.

### Scheduler

#### Recording (`driver.c`)
- Normal mode: `sched_record_session_begin(sched, sched->core.session_start)` replaces the separate `xce_core_now()` read, so the recorded `beginRound` is the session start.
- Simulation: no change here. Spec `scheduler-sim-discrete-clock` guarantees `core->session_start == begin_round` for a replayed session (clock frozen between recorded timestamps).

#### `from` stamping (`scheduler.c`)
- In `sched_make_primary_alloc_(core, ctx)`, after the allocator returns `rc > 0`: a small `static` helper walks `ctx->alloc` (every `allocs[i]->allocs[j]` and every `allocs_shared[k]`, skipping NULL entries) and sets `from = core->session_start / 1000000000ULL` (seconds, floor). A COMPOUND's primaries are each stamped before `merge_job_alloc()` folds them, and all carry the same session value, so the merged allocation is consistent.
- Why here and not in `sched_make_allocs()`: `core` is the session core in both modes; `sched_make_allocs()`'s `sched` is `sim_alloc.c`'s session-less wrapper in simulation.

#### Simulator verification (`sim_verify.c`)
- `int sim_decision_eq(const replay_decision_t* expected, const xce_decision_t* actual, char** why)`: DISPATCH calls `job_alloc_eq(..., why)`; PEND sets `*why` to `jobid A != B` or `main_reason A != B`; kind mismatch sets `kind DISPATCH != PEND`; a NULL side sets `no recorded decision at this index` / `no live decision`; PREEMPT sets `PREEMPT comparison not supported`. Same ownership contract as the libxce functions.
- `void sim_log_mismatch(uint64_t begin_round, int decision_index, const replay_decision_t* expected, const xce_decision_t* actual, const char* why)`: appends ` -- diff: <why>` to its existing line when `why != NULL`, otherwise the line is unchanged. It does not compare again and does not free `why`; `sim_loop.c`, which already calls `sim_decision_eq()` right before it, owns and frees the string.

## Data Models

No struct is added or modified. The equality contract and the description format are the only new "models".

- `alloc_eq` compares, in order: `resname`, `reserve`, `value` (exact `==`, documented bit-exact-safe because both sides are exact copies of the same `double`), `duration`, `from`. **Excluded**: `instid` (per-process `nextid++` counter).
- `host_alloc_eq` compares: `host`, then `nallocs`, then `allocs[i]` in order.
- `job_alloc_eq` compares: `jobid`, `nhosts`, `nres_shared`, then `allocs[i]` in order, then `allocs_shared[i]` in order.
- `job_preason_eq` compares: `main_reason`; then `nhost_reasons` and `host_based_reasons[i]` in order, each entry by `host` and `reason`; then `nres_reasons` and `res_based_reasons[i]` in order, each entry by `resource`, `host`, and `reason`. **Nothing is excluded.**
- **Description format** (first difference only, path from the outermost struct compared):
  - scalar: `allocs[1](host=h1).allocs[0](resname=mem).value: 2 != 3`
  - string: `allocs[0](host=h0).host: "h0" != "h2"`
  - NULL: `allocs_shared[0]: NULL != non-NULL`
  - length: `nhosts: 1 != 2`
  - path labels (`host=`, `resname=`) are taken from the first argument (`a`), or from `b` when `a`'s entry or string is NULL, or `?` when both are.
  - `jobid` formatted with libxce's own `fmt_jobid()` (`utils.h`), so it reads like the rest of the scheduler log; `from` as integer seconds; `value` with `%.17g` so an exact-compare difference is always visible.

## Error Handling

These are read-only functions with no I/O and no failure path for the comparison itself. The only allocation is the optional description; if it fails, `*why` is NULL and the result is unchanged (`vasprintf` leaves its output undefined on failure, so the implementation sets `*why = NULL` explicitly). A nested call's description is re-wrapped with the parent's path segment and the inner string freed. The only "error" case is NULL input, which is handled by the shared convention (identical pointers including both-NULL → equal; exactly one NULL → not-equal) rather than an error return. A malformed record (non-zero count, NULL array) is a documented precondition violation, not a handled case. No logging is added; the description is built with `vasprintf` (glibc; `_GNU_SOURCE` defined at the top of `xcealloc.c`/`preason.c`) and returned to the caller.

## Testing Strategy

- **Unit (libxce)**: `test_xcealloc.c` — match/mismatch on each compared field with its expected description, `instid` ignored, `from` compared, ordered-list sensitivity, list-length mismatch, NULL handling at top level, nested entries, and string fields, NULL `why`, `*why == NULL` on equal, a failed description allocation (`--wrap` on `vasprintf` or `malloc` in a dedicated test binary: `*why` NULL, same result); every test frees the returned description, so ASAN/LeakSanitizer proves ownership; an existing `merge_job_alloc` test still passes. `test_preason.c` — full-match: main-reason, host-based and resource-based lists per field + order + length, NULL handling, description cases; plus `str_eq` NULL/NULL and NULL/non-NULL.
- **Unit (scheduler)**: `test_sim_verify.c` keeps `sim_decision_eq`-level tests (DISPATCH/PEND match and mismatch, kind mismatch, PREEMPT always-false, NULL defense), asserts the description for each, and drops the alloc-level tests now owned by libxce. A session-level test asserts every allocation of one session (including a COMPOUND merge and the simulation wrapper path) carries `session_start / 1e9`, and that the recorded `beginRound` equals `session_start`.
- **Integration (scheduler)**: `test_sim_verify.c`'s `--sim-verify` round-trips (zero-mismatch self-replay, exact-mismatch altered replay, reconcile cases); the self-replay reports zero mismatches. These tests, and `test_sim_equivalence.c`/`test_repl_equivalence.c`/`test_repl_limit_table.c`/`test_sim_live_allocation.c`, already run the simulator at `sim_speed = 0` (clock frozen), so `session_start == begin_round` holds for them before `scheduler-sim-discrete-clock` lands; only the live `--sim-verify` check at the default speed waits for it.
- **Regression**: `test_sim_equivalence.c`, `test_repl_equivalence.c`, `test_repl_limit_table.c` stay green under the stricter checks (full-match preason, `from` compared); the hand-built recorded DISPATCH fixtures listed in the File Structure Plan are updated to carry the session's `from`. A failure after that is investigated as a real divergence.
- **Live**: record a session on a local cluster, replay it with `--sim-verify` at the default speed: zero mismatches.

## Design Decisions

### Decision 1: Co-locate equality with construction
- **Context**: where to put the four functions.
- **Alternatives**: a single new `xce_equality.{c,h}` module; or one function per struct's existing owner module.
- **Selected**: co-locate — `alloc` family in `xcealloc.c`, `job_preason_eq` in `preason.c`.
- **Rationale**: matches the existing precedent (`dup_job_alloc` lives beside `make_*` in `xcealloc.c`; `add_job_preason`/`make_pendreason` in `preason.c`); avoids a new module for four small functions.
- **Trade-offs**: none material.

### Decision 2: Naming (`alloc_eq`, not `xce_alloc_eq`)
- **Context**: public API names.
- **Alternatives**: `xce_`-prefixed names; `_equals` suffix.
- **Selected**: `alloc_eq`/`host_alloc_eq`/`job_alloc_eq`/`job_preason_eq`.
- **Rationale**: follows the existing unprefixed lowercase convention (`make_alloc`, `dup_job_alloc`, `erase_job_preason`), with the `_eq` suffix as the standard C equality idiom. The `job_alloc`/`job_preason` stems match the struct names.
- **Trade-offs**: `alloc_eq` is slightly generic, but it is unambiguous in a header (`xcealloc.h`) that already owns the unprefixed `alloc` family.

### Decision 3: `job_preason_eq` is a full structural match; the simulator keeps its own main-reason-only PEND check
- **Context**: the current `sim_verify_job_preason_eq` compares only `main_reason`, deliberately (bug-1845): the simulator has no alloc cache, so its full-detail PEND can never match a recorded cache-hit's main-reason-only shape, and requiring detail to match produced systematic false mismatches. That narrowing is a property of *recorded-vs-live replay comparison*, not of "are these two pending-reason objects identical".
- **Alternatives**: (a) publish a full-deep `job_preason_eq` and keep the main-reason-only comparison as a simulator-local static helper (selected); (b) publish main-reason-only `job_preason_eq` (rejected — leaks a replay artifact into a general-purpose equality API and contradicts the identity semantics a controller engineer would expect).
- **Selected**: `job_preason_eq(a, b, why)` returns non-zero iff `main_reason` matches and both detail lists match in order, field-for-field. The simulator's `sim_decision_eq` PEND branch keeps a small `static` main-reason-only helper (documented with the bug-1845 rationale) instead of calling `job_preason_eq`.
- **Rationale**: a public "same pending reason" predicate must be identity-honest — a controller comparing two pending reasons will want the detail to matter. The replay asymmetry is a scheduler-only concern and stays where it belongs.
- **Trade-offs**: two PEND-comparison shapes now exist (full-match in libxce, main-reason-only in sim_verify.c). This is intentional: they answer different questions ("identical objects" vs. "same replay outcome"). The sim-local helper is `static` and small, so the duplication is negligible and clearly scoped.

### Decision 4: Preserve NULL and order semantics
- **Context**: NULL handling and list-order sensitivity.
- **Selected**: keep today's behavior — identical pointer (including both NULL) → equal; exactly one NULL → not-equal; ordered (not multiset) list comparison. A malformed record (non-zero count, NULL array) is not handled — it is a documented precondition, matching today's comparators.
- **Rationale**: both sides of every current comparison come from the same deterministic construction, so order-sensitivity is correct. The new full-match `job_preason_eq` uses the same convention.

### Decision 5: Promote the NULL-safe string comparison to a shared `str_eq` in `utils.{c,h}`
- **Context**: with `job_preason_eq` now a full match, both `xcealloc.c` (host/resname/reserve strings) and `preason.c` (host/resource strings) need NULL-safe string comparison. Keeping it `static` in one file forces the other to duplicate it.
- **Alternatives**: (a) shared `str_eq` in `utils.{c,h}` (selected); (b) duplicate a `static` helper in each of the two files.
- **Selected**: add `int str_eq(const char* a, const char* b)` to `utils.{c,h}`, beside the existing `float_equal` helper.
- **Rationale**: two consumers and a third on the horizon (controller) justify a shared helper. `utils.h` already centralizes small generic C helpers (`fmt_jobid`, `float_equal`).
- **Trade-offs**: a new public symbol in `utils.h`; negligible surface.

### Decision 6: `alloc_eq` and `same_alloc_slot` stay separate
- **Context**: bug-1863 added a private `same_alloc_slot(a, b)` to `xcealloc.c` (`resname` + `instid` + `reserve`), used by `merge_job_alloc()` to decide whether a source allocation should be added into an existing destination entry. `alloc_eq` will live in the same file and also compares `xce_alloc_t` records.
- **Alternatives**: (a) keep them separate and document the difference (selected); (b) implement `same_alloc_slot` on top of `alloc_eq` or vice versa.
- **Selected**: keep them separate. `alloc_eq` answers "do these two records describe the same allocation decision" (resname, reserve, value, duration, from; `instid` excluded). `same_alloc_slot` answers "are these the same reservation slot to accumulate into" (resname, instid, reserve; `value` excluded because merging adds values). The `alloc_eq` header comment names the difference.
- **Rationale**: the two predicates disagree on `instid`, `value`, `duration`, and `from`, so neither can be expressed through the other without changing behavior. `merge_job_alloc()` is out of scope.
- **Follow-up**: none. bug-1882 (a NULL `resname` reaching `same_alloc_slot()`'s `strcmp()` after a failed `strdup()`) is fixed in f9146864: `make_alloc()`/`make_host_alloc()` no longer return records with NULL strings.

### Decision 7: The three test files adopt the full-match `job_preason_eq`
- **Context**: `test_sim_equivalence.c`, `test_repl_equivalence.c`, and `test_repl_limit_table.c` compare two decoded `replay_decision_t` entries and today call the main-reason-only `sim_verify_job_preason_eq`.
- **Alternatives**: (a) switch them to the full-match `job_preason_eq` (selected); (b) give them their own main-reason-only helper.
- **Selected**: (a). None of these tests registers `unsatisfiable_lookup_cb` (the alloc cache), so both sides of every pending-reason comparison are produced by the same full per-host computation and their detail lists are expected to match. The bug-1845 asymmetry (recorded cache-hit vs. live full detail) does not apply.
- **Rationale**: the stricter check catches detail divergence the old one missed, at no false-positive cost for these fixtures. Each call site gets a short comment stating why full match is valid there.
- **Trade-offs**: if a future test records with the alloc cache active, it must use a main-reason-only comparison instead; the call-site comment and the Revalidation Trigger on pending-reason construction paths cover this.

### Decision 8: `from` is the deciding session's start, from one clock reading
- **Context**: `alloc_eq` must compare `from`, but `make_alloc()` stamps `time(NULL)`, which a replay cannot reproduce (bug-1838-followup). The scheduler already has a virtualized session start (`core->session_start`), but it is a different clock read from the recorded `beginRound`, and in simulation at `--sim-speed` ≥ 1 it drifts past the recorded `beginRound`.
- **Alternatives**: (a) `from = session_start / 1e9`, with the recorded `beginRound` taken from `session_start`; the simulator clock stops drifting (spec `scheduler-sim-discrete-clock`) so a replayed `session_start` is exactly the recorded `beginRound` (selected); (a′) as (a), but with an explicit `xce_core_begin_session_at(core, begin_round)` in the simulator instead of removing the drift (superseded: the user chose to remove `--sim-speed`, which fixes the drift for every clock reader, not just `from`); (b) keep excluding `from` from equality (rejected by the user: `from` is part of what was decided); (c) `from = xce_core_now()` at allocation time (rejected: at `--sim-speed` ≥ 1 it drifts with replay pace, and a second-boundary crossing produces a false mismatch); (d) change `make_alloc()`'s signature to take `from` (rejected: churns the controller and ~20 test callers for no benefit; the scheduler is the only producer that needs a different value).
- **Selected**: (a). The recording's `beginRound` is taken from `core->session_start`; `sched_make_primary_alloc_(core, ctx)` overwrites `from` on the allocator's result using the session core (not `sched_make_allocs()`, whose `sched` is a session-less wrapper in simulation); the simulator side comes from `scheduler-sim-discrete-clock`.
- **Rationale**: every allocation of one session gets the same `from`, which a replay reproduces bit-for-bit. `from` changes meaning only slightly — from "second the record was built" to "second the deciding session started", normally the same second and at most one session's duration earlier.
- **Trade-offs**: hand-built test fixtures that stand in for a recording must now carry the session's `from`. Recorded `beginRound` moves a few milliseconds earlier (to `xce_core_begin_session`, before `plugin_begin_session`/`sched_account_jobs`). Recordings made before this change carry wall-clock `from` values and may show a false DISPATCH mismatch when an allocation's `time(NULL)` fell in a later second than its session start; the difference description names `from`, so these are recognisable. `make_alloc()` keeps `time(NULL)` for its other callers.

### Decision 9: First-difference description returned through `char** why`, owned by the caller
- **Context**: a yes/no comparator leaves a mismatch undiagnosable (bug-1881).
- **Alternatives**: (a) optional `char** why`: the function allocates a string sized to fit and hands ownership to the caller (selected, per the user); (b) caller buffer `char* why, size_t whylen` (superseded: the caller must guess a size, and a long path through many hosts gets truncated); (c) a separate `*_diff()` function family; (d) a list of every difference; (e) a callback per difference.
- **Selected**: (a). One code path computes both the result and the explanation, so they cannot disagree; callers that do not care pass `NULL`. On not-equal the function sets `*why` to a `vasprintf`'d string; on equal, or if the allocation fails, `*why = NULL`. The caller frees with `free()`.
- **Rationale**: the function knows the length; the caller only decides whether to ask. Allocation happens only on the not-equal path with `why` requested, i.e. only when something is about to be reported, so the equal fast path stays allocation-free. The first difference is the one that matters for a deterministic replay; later ones usually follow from it.
- **Trade-offs**: every caller that asks must free (ASAN/LeakSanitizer in the test suites catches a missed `free`); only the first difference is reported.