myxce/.kiro/specs/libxce-alloc-preason-equality/design.md

Design Document

Overview

Purpose: This feature delivers a single, shared implementation of structural equality for the scheduler's allocation and pending-reason data structures, published as public libxce API, that can explain why two objects differ, together with a reproducible allocation start time (from) so that from can take part in equality.

Users: Scheduler engineers (today: --sim-verify replay checking and three scheduler test files) and, in future, controller engineers (persist.c, resync.c, job.c already carry these structs) who need to ask "do these two objects represent the same thing, and if not, where do they differ".

Impact:

• Moves the three static allocation comparators out of src/scheduler/src/sim_verify.c into src/lib/xcealloc.c. alloc_eq now compares from and excludes only instid.
• Adds a new full-structural-match pending-reason comparator to src/lib/preason.c. No full-match comparator exists today; bug-1845 narrowed the simulator's to main-reason-only.
• Gives every comparator an optional char** why out-parameter that returns a newly allocated description of the first difference, owned by the caller. sim_log_mismatch prints it (resolves bug-1881).
• Changes the scheduler to stamp from with the scheduling session's start time, a single clock reading shared with the recorded beginRound, instead of time(NULL). That a replayed session starts at exactly the recorded beginRound comes from the separate spec scheduler-sim-discrete-clock (removes --sim-speed; the simulator clock moves only at recorded timestamps).
• Switches the simulator and three white-box test consumers to the public API. The three test files' checks become stricter: full-match pending reasons (Decision 7) and from-inclusive allocations.
• sim_decision_eq and sim_log_mismatch stay in the scheduler, and sim_decision_eq's PEND branch keeps a simulator-local main-reason-only comparison (bug-1845). No wire format or struct layout changes.

Goals

• Publish alloc_eq, host_alloc_eq, job_alloc_eq in xcealloc.h and job_preason_eq in preason.h, each with an optional first-difference description.
• Make job_preason_eq a full structural match (main reason + host-based reasons + resource-based reasons, in order).
• Make from reproducible under replay, then compare it.
• Make --sim-verify mismatch logs name the differing field and values.
• Make the existing consumers call the public API instead of sim_verify.c's private static comparators. (The three test files keep their white-box #include "../src/sim_verify.c": they also white-box-include sim_loop.c, which calls sim_decision_eq/sim_log_mismatch.)
• Keep --sim-verify free of false mismatches for recordings made after this change.
• Add libxce unit tests that pin the instid exclusion, from inclusion, full-match preason, description, and NULL-handling semantics.

Non-Goals

• Do not move sim_decision_eq, sim_log_mismatch, or the describe_* helpers (simulator-specific; depend on capnp/replay types and ERROR()).
• Do not change the simulator's PEND outcome check from main-reason-only to full-match — that would reintroduce bug-1845's false mismatches on cache-hit PENDs (see Decision 3).
• Do not introduce any new comparison call site beyond the existing four consumers.
• Do not touch the controller's persistence/resync code paths, or how the controller/worker interpret from.
• Do not change make_alloc()'s signature or its own time(NULL) default (controller and many tests call it); the scheduler overwrites from after construction (Decision 8).
• Do not touch merge_job_alloc()/same_alloc_slot() (bug-1863); alloc_eq neither replaces nor reuses same_alloc_slot (see Decision 6). (Their NULL-dereference defect, bug-1882, is already fixed in f9146864.)
• Do not remove the test files' white-box #include "../src/sim_verify.c".
• Do not make recordings made before this change verify cleanly; their wall-clock from values are an accepted limitation.
• Do not change the simulator's clock or --sim-speed; that is spec scheduler-sim-discrete-clock, which this spec depends on.

Boundary Commitments

This Spec Owns

• The public equality functions alloc_eq, host_alloc_eq, job_alloc_eq (xcealloc.{c,h}) and job_preason_eq (preason.{c,h}), their equality contracts and their difference-description format.
• The shared NULL-safe string-equality helper both modules need (see Decision 5).
• The scheduler's stamping of from (sched_make_primary_alloc_() in scheduler.c, from the session core) and the single normal-mode session-start reading shared by core->session_start, the recorded beginRound, and from (driver.c).
• The migration of sim_verify.c and the three white-box test files onto those functions, and the difference text in sim_log_mismatch.
• libxce unit tests for the new functions (test_xcealloc.c, test_preason.c) and scheduler tests for from stamping.

Out of Boundary

• The decision-level comparison and mismatch reporting structure that remain in sim_verify.c, including the main-reason-only PEND outcome check (simulator-local).
• The .replay wire format and the replay_decision_t/xce_decision_t types.
• Any change to the controller's consumption of these structs.
• merge_job_alloc() and its same_alloc_slot() slot-matching helper in xcealloc.c.
• The simulator clock and --sim-speed (spec scheduler-sim-discrete-clock).

Allowed Dependencies

• src/include/xce/types.h (the struct definitions — unchanged).
• xcealloc.c/xcealloc.h, preason.c/preason.h, and utils.{c,h} (the modules that already own construction of these structs, plus the shared string helper).
• string.h (strcmp) and stdio.h (vasprintf, a GNU extension: xcealloc.c/preason.c define _GNU_SOURCE before their first include, as client_failover.c:14 does) for comparisons and descriptions.
• Scheduler: driver.c, scheduler.c, sim_verify.c, sim_loop.c (the sim_decision_eq/sim_log_mismatch call sites only).
• Spec scheduler-sim-discrete-clock: after it, a replayed session's core->session_start equals the recorded beginRound exactly.

Revalidation Triggers

• A change to the xce_alloc_t/xce_host_alloc_t/xce_job_alloc_t/xce_job_preason_t struct layouts must re-check the field lists, the exclusion documentation, and the description text.
• A change to same_alloc_slot()'s matching rule, or a proposal to merge it with alloc_eq, must re-check Decision 6 (the two answer different questions and disagree on instid, from, and value).
• A change to the pending-reason construction paths (cache-hit vs. full-computation) that alters whether detail lists are populated must re-check the simulator's main-reason-only PEND rationale (bug-1845) and the full-match job_preason_eq contract.
• Any new allocation path that does not go through sched_make_primary_alloc_(), a second clock read between session start and recording, or a change to scheduler-sim-discrete-clock's clock guarantee, must re-check Decision 8.

Architecture

Existing Architecture Analysis

src/scheduler/src/sim_verify.c (474 lines) owns two layers:

1. Type-level comparators — five static functions (sim_verify_str_eq, sim_verify_alloc_eq, sim_verify_host_alloc_eq, sim_verify_job_alloc_eq, sim_verify_job_preason_eq) that walk two libxce structs field-by-field and return only 0/1. They have zero dependency on the scheduler, capnp, or the replay layer. sim_verify_alloc_eq excludes from (bug-1838-followup) and instid (bug-1844). sim_verify_job_preason_eq compares only main_reason (bug-1845), a simulator replay asymmetry rather than an object-identity property.
2. Decision-level dispatcher + reporting — sim_decision_eq (dispatches on xce_decision_t.op vs ReplayDecision_body_which), sim_log_mismatch, and two describe_* helpers. These depend on replay_decision_t, capnp enums, and ERROR(). sim_log_mismatch prints only jobid/nhosts for each side, so a mismatch below that level is invisible (bug-1881).

The four allocation/pending-reason structs are defined in src/include/xce/types.h and already constructed/deep-copied by src/lib/xcealloc.c (make_alloc, make_host_alloc, make_job_alloc, dup_job_alloc) and src/lib/preason.c (add_job_preason, make_pendreason). The controller consumes these same structs in persist.c, resync.c, and job.c with no equality API available.

Since this design was first drafted, bug-1863 (aea34738) added merge_job_alloc() to xcealloc.c with a private same_alloc_slot() helper that matches two xce_alloc_t records by resname, instid, and reserve, to decide whether they are the same reservation slot to merge. It is a near neighbour of alloc_eq but has a different contract (see Decision 6).

How from is set today. make_alloc() (xcealloc.c) stamps from = time(NULL). The scheduler's only production constructor call is in sched_make_allocs() (scheduler.c), called by the seqjob/parjob allocators' my_alloc() inside a scheduling session. from is on the wire (controller.capnp from @4: UInt64 # alloc start time) and dup_alloc() copies it. Neither the controller nor the worker reads an allocation's from today.

Which core sched_make_allocs() sees. Its first argument is plugin->sched. In normal mode that is the real scheduler, whose core runs the sessions. In simulation it is not: sim_alloc.c (sim_bootstrap_seqjob_allocator()/..._parjob_...) hands the allocators a throwaway xce_scheduler_t wrapper with its own xce_sched_core_init()'d core, aliasing only resmgr; that core never begins a session, so its session_start is 0. The honest session core is the core argument threaded through sched_make_allocation() → sched_make_primary_alloc_(core, ctx) (sim->core in simulation, sched->core in normal mode). Every allocation — PRIMARY, each primary of a COMPOUND, each alternative of an ALTER — is produced by one sched_make_primary_alloc_() call.

How session time is read today. Two separate clock reads per session:

• xce_core_begin_session() sets core->session_start = xce_core_now(core) (core_session.c:429).
• The recording then reads the clock again for beginRound: driver.c:1030 after plugin_begin_session/sched_account_jobs, and sim_loop.c before xce_core_begin_session.

In simulation, sim_advance_virtual_clock(sim, begin_round) re-anchors the virtual clock to the recorded beginRound. At --sim-speed ≥ 1 the clock keeps advancing from there, so session_start lands slightly after begin_round. At --sim-speed 0 it is frozen and exact. The simulator never sleeps, so --sim-speed paces nothing; it only adds drift. Spec scheduler-sim-discrete-clock removes it, which makes session_start == begin_round in every replay.

Three scheduler test files already reuse two of the leaf comparators — but via white-box #include "../src/sim_verify.c" + direct calls to the static functions: test_sim_equivalence.c (lines 894, 902), test_repl_equivalence.c (lines 734, 742), and test_repl_limit_table.c (lines 2662, 2670).

Architecture Pattern & Boundary Map

flowchart LR
    subgraph scheduler
        SV["sim_verify.c\n(sim_decision_eq, sim_log_mismatch, describe_*,\nmain-reason-only PEND outcome check)"]
        DR["driver.c\nbeginRound = core.session_start"]
        SC["scheduler.c\nalloc->from = session_start / 1e9"]
        CS["core_session.c\nsession_start"]
        DC["spec scheduler-sim-discrete-clock\nreplayed session_start == recorded beginRound"]
        TE["test_sim_equivalence.c / test_repl_equivalence.c / test_repl_limit_table.c"]
        TSV["test_sim_verify.c"]
    end
    subgraph libxce
        XA["xcealloc.{c,h}\nalloc_eq / host_alloc_eq / job_alloc_eq (+why)"]
        PR["preason.{c,h}\njob_preason_eq (full match, +why)"]
        UT["utils.{c,h}\nstr_eq (NULL-safe)"]
    end
    SV --> XA
    TE --> XA
    TE --> PR
    XA --> UT
    PR --> UT
    DR --> CS
    SC --> CS
    DC -.-> CS

• Domain boundary: the library owns struct-level equality and its explanation (full structural match); the simulator owns decision-level comparison and reporting, including its one deliberate loosening (main-reason-only PEND). The scheduler core owns the session-start value that makes from reproducible.
• Steering compliance: no changes to types.h layouts or the wire schema; from keeps its unit and meaning ("alloc start time"), now defined as the start of the deciding session. Persistence: from already round-trips through the snapshot (dup_job_alloc) and the controller's capnp encoding, so no new save/restore field is introduced.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Language	C (gnu17)	equality functions, from stamping, tests	unchanged
Build	Meson via xce-build	compile libxce + scheduler, run src/lib/testsuite and src/scheduler/testsuite	unchanged
Test	Unity	test_xcealloc.c, test_preason.c, scheduler tests	existing harnesses

File Structure Plan

Modified Files

• src/lib/utils.c / src/include/utils.h — add a NULL-safe str_eq(const char*, const char*) helper (see Decision 5).
• src/lib/xcealloc.c / src/include/xcealloc.h — add alloc_eq, host_alloc_eq, job_alloc_eq with the optional char** why description.
• src/lib/preason.c / src/include/preason.h — add job_preason_eq (full match) with the optional description.
• src/scheduler/src/driver.c — record beginRound from sched->core.session_start instead of a second xce_core_now() read.
• src/scheduler/src/scheduler.c — in sched_make_primary_alloc_(core, ctx), when the allocator returns rc > 0 with ctx->alloc != NULL, set from = core->session_start / 1000000000ULL on every xce_alloc_t in ctx->alloc (per-host and shared). Not in sched_make_allocs(): in simulation its sched is sim_alloc.c's wrapper, whose core has no session.
• src/scheduler/src/sim_verify.c — delete the five static comparators; sim_decision_eq gains an optional char** why and calls job_alloc_eq for DISPATCH; keep a simulator-local static main-reason-only helper for PEND; sim_log_mismatch gains a const char* why parameter and appends it. Update the header comments that describe from as excluded.
• src/scheduler/src/sim_loop.c — update both extern declarations; in the per-decision loop call sim_decision_eq(expected, d, &why), pass why to sim_log_mismatch, then free(why); the leftover-recorded loop (recorded decisions with no live counterpart) passes the string literal "no live decision", which it does not free.
• src/lib/testsuite/test_xcealloc.c, src/lib/testsuite/test_preason.c — unit tests.
• src/scheduler/testsuite/test_sim_equivalence.c, test_repl_equivalence.c, test_repl_limit_table.c — replace direct calls with job_alloc_eq(..., NULL)/job_preason_eq(..., NULL) (or take the description, print it on failure, and free it); add #include "xcealloc.h"/#include "preason.h"; comment at each job_preason_eq call why full match is valid (Decision 7); keep the white-box #include "../src/sim_verify.c".
• src/scheduler/testsuite/test_sim_verify.c — add the why argument (NULL) to its 16 direct sim_decision_eq() calls; drop the DISPATCH alloc-level tests now owned by libxce; invert test_dispatch_decisions_differing_from_timestamp_still_match (a from difference is now a mismatch, and why names from); add a test that the mismatch description names the differing field; keep PEND/decision-level and integration tests.
• Hand-built recorded DISPATCH fixtures that --sim-verify or an equivalence test compares against a live replay — they build the recorded allocation with make_alloc() (from = time(NULL)) or a literal from, which no longer matches the replay's session_start / 1e9: set the recorded from to the fixture session's begin_round / 1000000000. Audit every test with a hand-built ReplayDecision_body_dispatch: test_sim_live_allocation.c (the make_alloc() fixture near line 726, begin_round = 2000 → from = 0), test_sim_contention_parity.c, test_sim_resource_contention.c, test_sim_round_boundary_reconciliation.c, test_sim_reconcile_unaccounted.c, test_sim_verify.c; fixtures that are never compared against a live allocation need no change.
• A scheduler test (new or in an existing session/allocation test file) — allocations made in one session carry session_start / 1e9 as from, including in simulation (the sim_alloc.c wrapper path), and the recorded beginRound equals session_start.
• src/scheduler/testsuite/test_sim_live_allocation.c, test_sim_reconcile_unaccounted.c, and the header comments of test_sim_equivalence.c/test_repl_equivalence.c — update comments that name sim_verify_*_eq or describe from as excluded.

System Flows

flowchart TD
    A["sim_decision_eq(expected, actual, why)"] --> B{"actual->op"}
    B -->|DISPATCH| C["job_alloc_eq(expected alloc, actual alloc, why)"]
    B -->|PEND| D["jobid equal AND main_reason equal (sim-local, NOT job_preason_eq); why names which"]
    B -->|PREEMPT| E["log + report mismatch (unchanged)"]
    C --> F["host_alloc_eq per host"] --> G["alloc_eq per resource (incl. from, excl. instid)"]
    D --> H["main-reason only, detail ignored (bug-1845)"]
    C --> M["sim_loop: sim_log_mismatch(..., why); free(why)"]
    D --> M

from stamping per session:

sequenceDiagram
    participant Drv as driver.c
    participant Core as core_session.c
    participant Sch as scheduler.c sched_make_primary_alloc_(core, ctx)
    participant Rec as recording
    Drv->>Core: xce_core_begin_session()
    Core-->>Core: session_start = xce_core_now()<br/>(replay: == recorded beginRound, via scheduler-sim-discrete-clock)
    Drv->>Rec: beginRound = session_start
    Sch->>Sch: allocator builds ctx->alloc (make_alloc: from = time(NULL))
    Sch->>Sch: every alloc in ctx->alloc: from = core->session_start / 1e9

Requirements Traceability

Requirement	Summary	Components	Interfaces
1.1–1.3	allocation equality for xce_alloc_t/xce_host_alloc_t/xce_job_alloc_t	xcealloc.c	alloc_eq, host_alloc_eq, job_alloc_eq
1.4	exclude instid only; document difference from merge_job_alloc	xcealloc.c/.h	alloc_eq field list + header comment
1.5–1.7	NULL handling, malformed-record precondition, read-only	all four functions	const params
2.1–2.3	pending-reason full structural equality	preason.c	job_preason_eq
3.1–3.6	first-difference description, caller-owned	all four functions	char** why
4.1–4.2, 4.4–4.5	reproducible from	driver.c, scheduler.c sched_make_primary_alloc_()	core->session_start
4.3	replayed session start == recorded beginRound	spec scheduler-sim-discrete-clock	—
5.1	simulator uses libxce allocation comparison	sim_verify.c	job_alloc_eq
5.2	simulator-local main-reason-only PEND check	sim_verify.c	local static helper
5.3	mismatch log names the difference (bug-1881)	sim_verify.c, sim_loop.c	sim_decision_eq(..., why), sim_log_mismatch(..., why)
5.4	zero-mismatch self-replay	integration tests	—
6.1–6.3	single shared implementation; test files use full-match preason; include stays	sim_verify.c + 3 test files	public API
6.4–6.6	libxce + scheduler tests, no other behavior change	test files	—
6.7	stale comment references updated	scheduler sources/tests	—

Components and Interfaces

libxce

Allocation equality (xcealloc.c)

Field	Detail
Intent	Public structural equality for the allocation structs, with an optional first-difference description
Requirements	1.1–1.7, 3.1–3.6

/* src/include/xcealloc.h */
extern int alloc_eq(const xce_alloc_t* a, const xce_alloc_t* b, char** why);
extern int host_alloc_eq(const xce_host_alloc_t* a, const xce_host_alloc_t* b,
                         char** why);
extern int job_alloc_eq(const xce_job_alloc_t* a, const xce_job_alloc_t* b,
                        char** why);

• Preconditions: a non-zero entry count implies a non-NULL entry array (a malformed record is not handled). NULL top-level arguments, NULL list entries, and NULL string fields are handled. why may be NULL; *why on entry is ignored (never read or freed).
• Postconditions: returns non-zero iff the two arguments are equal per the documented contract; does not mutate either argument. If why != NULL: on equal, *why = NULL; on not-equal, *why is a malloc'd, NUL-terminated description of the first difference (see Data Models), sized by the function and never truncated; the caller owns it and releases it with free(). If that allocation fails, *why = NULL and the return value is unaffected.
• Invariants: alloc_eq(a, b) answers "same allocation decision", compares from, and ignores instid; the header comment states that this differs from merge_job_alloc()'s slot matching, where instid is significant; host_alloc_eq/job_alloc_eq are order-sensitive (both sides come from the same deterministic construction, so array order is expected to match, not just multiset). The result never depends on why or on whether the description could be allocated.

Pending-reason equality (preason.c)

Field	Detail
Intent	Public full structural equality for xce_job_preason_t, with an optional first-difference description
Requirements	2.1–2.3, 3.1–3.6

/* src/include/preason.h */
extern int job_preason_eq(const xce_job_preason_t* a,
                          const xce_job_preason_t* b,
                          char** why);

• Postconditions: returns non-zero iff main_reason matches, the host-based reason list matches in order (each entry by host name + reason), and the resource-based reason list matches in order (each entry by resource name + host name + reason); NULL-safe per the shared convention; why as for the allocation functions.
• Invariants: this is a full match — it is the honest "are these two pending reasons identical" predicate, not the simulator's replay loosening (see Decision 3).

Shared string helper (utils.c)

/* src/include/utils.h */
extern int str_eq(const char* a, const char* b);

• Postconditions: returns non-zero iff a and b are both NULL, or both non-NULL and strcmp(a, b) == 0; zero otherwise.

Scheduler

Recording (driver.c)

• Normal mode: sched_record_session_begin(sched, sched->core.session_start) replaces the separate xce_core_now() read, so the recorded beginRound is the session start.
• Simulation: no change here. Spec scheduler-sim-discrete-clock guarantees core->session_start == begin_round for a replayed session (clock frozen between recorded timestamps).

from stamping (scheduler.c)

• In sched_make_primary_alloc_(core, ctx), after the allocator returns rc > 0: a small static helper walks ctx->alloc (every allocs[i]->allocs[j] and every allocs_shared[k], skipping NULL entries) and sets from = core->session_start / 1000000000ULL (seconds, floor). A COMPOUND's primaries are each stamped before merge_job_alloc() folds them, and all carry the same session value, so the merged allocation is consistent.
• Why here and not in sched_make_allocs(): core is the session core in both modes; sched_make_allocs()'s sched is sim_alloc.c's session-less wrapper in simulation.

Simulator verification (sim_verify.c)

• int sim_decision_eq(const replay_decision_t* expected, const xce_decision_t* actual, char** why): DISPATCH calls job_alloc_eq(..., why); PEND sets *why to jobid A != B or main_reason A != B; kind mismatch sets kind DISPATCH != PEND; a NULL side sets no recorded decision at this index / no live decision; PREEMPT sets PREEMPT comparison not supported. Same ownership contract as the libxce functions.
• void sim_log_mismatch(uint64_t begin_round, int decision_index, const replay_decision_t* expected, const xce_decision_t* actual, const char* why): appends -- diff: <why> to its existing line when why != NULL, otherwise the line is unchanged. It does not compare again and does not free why; sim_loop.c, which already calls sim_decision_eq() right before it, owns and frees the string.

Data Models

No struct is added or modified. The equality contract and the description format are the only new "models".

• alloc_eq compares, in order: resname, reserve, value (exact ==, documented bit-exact-safe because both sides are exact copies of the same double), duration, from. Excluded: instid (per-process nextid++ counter).
• host_alloc_eq compares: host, then nallocs, then allocs[i] in order.
• job_alloc_eq compares: jobid, nhosts, nres_shared, then allocs[i] in order, then allocs_shared[i] in order.
• job_preason_eq compares: main_reason; then nhost_reasons and host_based_reasons[i] in order, each entry by host and reason; then nres_reasons and res_based_reasons[i] in order, each entry by resource, host, and reason. Nothing is excluded.
• Description format (first difference only, path from the outermost struct compared):
  • scalar: allocs[1](host=h1).allocs[0](resname=mem).value: 2 != 3
  • string: allocs[0](host=h0).host: "h0" != "h2"
  • NULL: allocs_shared[0]: NULL != non-NULL
  • length: nhosts: 1 != 2
  • path labels (host=, resname=) are taken from the first argument (a), or from b when a's entry or string is NULL, or ? when both are.
  • jobid formatted with libxce's own fmt_jobid() (utils.h), so it reads like the rest of the scheduler log; from as integer seconds; value with %.17g so an exact-compare difference is always visible.

Error Handling

These are read-only functions with no I/O and no failure path for the comparison itself. The only allocation is the optional description; if it fails, *why is NULL and the result is unchanged (vasprintf leaves its output undefined on failure, so the implementation sets *why = NULL explicitly). A nested call's description is re-wrapped with the parent's path segment and the inner string freed. The only "error" case is NULL input, which is handled by the shared convention (identical pointers including both-NULL → equal; exactly one NULL → not-equal) rather than an error return. A malformed record (non-zero count, NULL array) is a documented precondition violation, not a handled case. No logging is added; the description is built with vasprintf (glibc; _GNU_SOURCE defined at the top of xcealloc.c/preason.c) and returned to the caller.

Testing Strategy

• Unit (libxce): test_xcealloc.c — match/mismatch on each compared field with its expected description, instid ignored, from compared, ordered-list sensitivity, list-length mismatch, NULL handling at top level, nested entries, and string fields, NULL why, *why == NULL on equal, a failed description allocation (--wrap on vasprintf or malloc in a dedicated test binary: *why NULL, same result); every test frees the returned description, so ASAN/LeakSanitizer proves ownership; an existing merge_job_alloc test still passes. test_preason.c — full-match: main-reason, host-based and resource-based lists per field + order + length, NULL handling, description cases; plus str_eq NULL/NULL and NULL/non-NULL.
• Unit (scheduler): test_sim_verify.c keeps sim_decision_eq-level tests (DISPATCH/PEND match and mismatch, kind mismatch, PREEMPT always-false, NULL defense), asserts the description for each, and drops the alloc-level tests now owned by libxce. A session-level test asserts every allocation of one session (including a COMPOUND merge and the simulation wrapper path) carries session_start / 1e9, and that the recorded beginRound equals session_start.
• Integration (scheduler): test_sim_verify.c's --sim-verify round-trips (zero-mismatch self-replay, exact-mismatch altered replay, reconcile cases); the self-replay reports zero mismatches. These tests, and test_sim_equivalence.c/test_repl_equivalence.c/test_repl_limit_table.c/test_sim_live_allocation.c, already run the simulator at sim_speed = 0 (clock frozen), so session_start == begin_round holds for them before scheduler-sim-discrete-clock lands; only the live --sim-verify check at the default speed waits for it.
• Regression: test_sim_equivalence.c, test_repl_equivalence.c, test_repl_limit_table.c stay green under the stricter checks (full-match preason, from compared); the hand-built recorded DISPATCH fixtures listed in the File Structure Plan are updated to carry the session's from. A failure after that is investigated as a real divergence.
• Live: record a session on a local cluster, replay it with --sim-verify at the default speed: zero mismatches.

Design Decisions

Decision 1: Co-locate equality with construction

• Context: where to put the four functions.
• Alternatives: a single new xce_equality.{c,h} module; or one function per struct's existing owner module.
• Selected: co-locate — alloc family in xcealloc.c, job_preason_eq in preason.c.
• Rationale: matches the existing precedent (dup_job_alloc lives beside make_* in xcealloc.c; add_job_preason/make_pendreason in preason.c); avoids a new module for four small functions.
• Trade-offs: none material.

Decision 2: Naming (alloc_eq, not xce_alloc_eq)

• Context: public API names.
• Alternatives: xce_-prefixed names; _equals suffix.
• Selected: alloc_eq/host_alloc_eq/job_alloc_eq/job_preason_eq.
• Rationale: follows the existing unprefixed lowercase convention (make_alloc, dup_job_alloc, erase_job_preason), with the _eq suffix as the standard C equality idiom. The job_alloc/job_preason stems match the struct names.
• Trade-offs: alloc_eq is slightly generic, but it is unambiguous in a header (xcealloc.h) that already owns the unprefixed alloc family.

Decision 3: job_preason_eq is a full structural match; the simulator keeps its own main-reason-only PEND check

• Context: the current sim_verify_job_preason_eq compares only main_reason, deliberately (bug-1845): the simulator has no alloc cache, so its full-detail PEND can never match a recorded cache-hit's main-reason-only shape, and requiring detail to match produced systematic false mismatches. That narrowing is a property of recorded-vs-live replay comparison, not of "are these two pending-reason objects identical".
• Alternatives: (a) publish a full-deep job_preason_eq and keep the main-reason-only comparison as a simulator-local static helper (selected); (b) publish main-reason-only job_preason_eq (rejected — leaks a replay artifact into a general-purpose equality API and contradicts the identity semantics a controller engineer would expect).
• Selected: job_preason_eq(a, b, why) returns non-zero iff main_reason matches and both detail lists match in order, field-for-field. The simulator's sim_decision_eq PEND branch keeps a small static main-reason-only helper (documented with the bug-1845 rationale) instead of calling job_preason_eq.
• Rationale: a public "same pending reason" predicate must be identity-honest — a controller comparing two pending reasons will want the detail to matter. The replay asymmetry is a scheduler-only concern and stays where it belongs.
• Trade-offs: two PEND-comparison shapes now exist (full-match in libxce, main-reason-only in sim_verify.c). This is intentional: they answer different questions ("identical objects" vs. "same replay outcome"). The sim-local helper is static and small, so the duplication is negligible and clearly scoped.

Decision 4: Preserve NULL and order semantics

• Context: NULL handling and list-order sensitivity.
• Selected: keep today's behavior — identical pointer (including both NULL) → equal; exactly one NULL → not-equal; ordered (not multiset) list comparison. A malformed record (non-zero count, NULL array) is not handled — it is a documented precondition, matching today's comparators.
• Rationale: both sides of every current comparison come from the same deterministic construction, so order-sensitivity is correct. The new full-match job_preason_eq uses the same convention.

Decision 5: Promote the NULL-safe string comparison to a shared str_eq in utils.{c,h}

• Context: with job_preason_eq now a full match, both xcealloc.c (host/resname/reserve strings) and preason.c (host/resource strings) need NULL-safe string comparison. Keeping it static in one file forces the other to duplicate it.
• Alternatives: (a) shared str_eq in utils.{c,h} (selected); (b) duplicate a static helper in each of the two files.
• Selected: add int str_eq(const char* a, const char* b) to utils.{c,h}, beside the existing float_equal helper.
• Rationale: two consumers and a third on the horizon (controller) justify a shared helper. utils.h already centralizes small generic C helpers (fmt_jobid, float_equal).
• Trade-offs: a new public symbol in utils.h; negligible surface.

Decision 6: alloc_eq and same_alloc_slot stay separate

• Context: bug-1863 added a private same_alloc_slot(a, b) to xcealloc.c (resname + instid + reserve), used by merge_job_alloc() to decide whether a source allocation should be added into an existing destination entry. alloc_eq will live in the same file and also compares xce_alloc_t records.
• Alternatives: (a) keep them separate and document the difference (selected); (b) implement same_alloc_slot on top of alloc_eq or vice versa.
• Selected: keep them separate. alloc_eq answers "do these two records describe the same allocation decision" (resname, reserve, value, duration, from; instid excluded). same_alloc_slot answers "are these the same reservation slot to accumulate into" (resname, instid, reserve; value excluded because merging adds values). The alloc_eq header comment names the difference.
• Rationale: the two predicates disagree on instid, value, duration, and from, so neither can be expressed through the other without changing behavior. merge_job_alloc() is out of scope.
• Follow-up: none. bug-1882 (a NULL resname reaching same_alloc_slot()'s strcmp() after a failed strdup()) is fixed in f9146864: make_alloc()/make_host_alloc() no longer return records with NULL strings.

Decision 7: The three test files adopt the full-match job_preason_eq

• Context: test_sim_equivalence.c, test_repl_equivalence.c, and test_repl_limit_table.c compare two decoded replay_decision_t entries and today call the main-reason-only sim_verify_job_preason_eq.
• Alternatives: (a) switch them to the full-match job_preason_eq (selected); (b) give them their own main-reason-only helper.
• Selected: (a). None of these tests registers unsatisfiable_lookup_cb (the alloc cache), so both sides of every pending-reason comparison are produced by the same full per-host computation and their detail lists are expected to match. The bug-1845 asymmetry (recorded cache-hit vs. live full detail) does not apply.
• Rationale: the stricter check catches detail divergence the old one missed, at no false-positive cost for these fixtures. Each call site gets a short comment stating why full match is valid there.
• Trade-offs: if a future test records with the alloc cache active, it must use a main-reason-only comparison instead; the call-site comment and the Revalidation Trigger on pending-reason construction paths cover this.

Decision 8: from is the deciding session's start, from one clock reading

• Context: alloc_eq must compare from, but make_alloc() stamps time(NULL), which a replay cannot reproduce (bug-1838-followup). The scheduler already has a virtualized session start (core->session_start), but it is a different clock read from the recorded beginRound, and in simulation at --sim-speed ≥ 1 it drifts past the recorded beginRound.
• Alternatives: (a) from = session_start / 1e9, with the recorded beginRound taken from session_start; the simulator clock stops drifting (spec scheduler-sim-discrete-clock) so a replayed session_start is exactly the recorded beginRound (selected); (a′) as (a), but with an explicit xce_core_begin_session_at(core, begin_round) in the simulator instead of removing the drift (superseded: the user chose to remove --sim-speed, which fixes the drift for every clock reader, not just from); (b) keep excluding from from equality (rejected by the user: from is part of what was decided); (c) from = xce_core_now() at allocation time (rejected: at --sim-speed ≥ 1 it drifts with replay pace, and a second-boundary crossing produces a false mismatch); (d) change make_alloc()'s signature to take from (rejected: churns the controller and ~20 test callers for no benefit; the scheduler is the only producer that needs a different value).
• Selected: (a). The recording's beginRound is taken from core->session_start; sched_make_primary_alloc_(core, ctx) overwrites from on the allocator's result using the session core (not sched_make_allocs(), whose sched is a session-less wrapper in simulation); the simulator side comes from scheduler-sim-discrete-clock.
• Rationale: every allocation of one session gets the same from, which a replay reproduces bit-for-bit. from changes meaning only slightly — from "second the record was built" to "second the deciding session started", normally the same second and at most one session's duration earlier.
• Trade-offs: hand-built test fixtures that stand in for a recording must now carry the session's from. Recorded beginRound moves a few milliseconds earlier (to xce_core_begin_session, before plugin_begin_session/sched_account_jobs). Recordings made before this change carry wall-clock from values and may show a false DISPATCH mismatch when an allocation's time(NULL) fell in a later second than its session start; the difference description names from, so these are recognisable. make_alloc() keeps time(NULL) for its other callers.

Decision 9: First-difference description returned through char** why, owned by the caller

• Context: a yes/no comparator leaves a mismatch undiagnosable (bug-1881).
• Alternatives: (a) optional char** why: the function allocates a string sized to fit and hands ownership to the caller (selected, per the user); (b) caller buffer char* why, size_t whylen (superseded: the caller must guess a size, and a long path through many hosts gets truncated); (c) a separate *_diff() function family; (d) a list of every difference; (e) a callback per difference.
• Selected: (a). One code path computes both the result and the explanation, so they cannot disagree; callers that do not care pass NULL. On not-equal the function sets *why to a vasprintf'd string; on equal, or if the allocation fails, *why = NULL. The caller frees with free().
• Rationale: the function knows the length; the caller only decides whether to ask. Allocation happens only on the not-equal path with why requested, i.e. only when something is about to be reported, so the equal fast path stays allocation-free. The first difference is the one that matters for a deterministic replay; later ones usually follow from it.
• Trade-offs: every caller that asks must free (ASAN/LeakSanitizer in the test suites catches a missed free); only the first difference is reported.