lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/scheduler-sim-decision-trace/tasks.md
Rongsong Shen 1b78fa9638
test(scheduler-sim-decision-trace): bounds under large traces; update the scheduler guide 
Task 6.4: a 1000-host, 64-task parallel job traced at full level stores
exactly the line bound with exact omitted counts and counters; 10,000
traced decisions keep the trace store within 64 MiB without evicting the
newest. The scheduler guide lists the REPL trace files and test rules
and carries measured file sizes. Full all-components ASAN build passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 17:11:52 +00:00

45 KiB
Raw Blame History

Implementation Plan

Notes for every task below:

  • Every new test file is registered in the scheduler test build with the source set it needs, and runs in the scheduler suite. Registration only appends entries, so parallel tasks that each register a test do not conflict.
  • Every new trace source file is added to the scheduler's main source list by the task that creates it.
  • "Under ASAN" means the standard debug + ASAN scheduler build.

  • 1. Foundation: baselines, shared formatting, trace model and transport

  • 1.1 Capture the trace-off REPL transcript baseline before any feature change

    • Build a scripted REPL session over a resource-contention replay fixture. The fixture has several jobs competing for a shared license, a cluster-wide limit and a host-scoped limit, plus a parallel job and an alternative-requirement job.
    • The session exercises step, continue with collapsed ranges, a decision breakpoint, a round-begin breakpoint, info subcommands, set units and set output.
    • Store the captured console transcript, the --sim-output recording and the --sim-verify result as committed test fixtures. Add a test that replays the script and compares byte for byte, ignoring only wall-clock metadata fields.
    • Produce the baseline on the tree before any trace code exists. Later tasks use it to prove that output is unchanged with tracing off.
    • Done when the new baseline test passes on the unmodified tree under ASAN and the fixtures are committed.
    • Requirements: 10.4, 10.5

  • 1.2 Move the units and value formatting helpers into the shared REPL formatting module

    • Move these helpers out of the info module so a new renderer can reuse them, with no change in behaviour:
      • units-mode translation;
      • numeric-resource and consumable value formatting;
      • the current-units lookup.
    • Done when every existing REPL test and the 1.1 baseline test pass unchanged under ASAN, and the helpers can be called from other REPL files.
    • Requirements: 7.7

  • 1.3 Build the trace record model foundation, test first

    • Define these types: record, settings snapshot, verdict kinds, record kinds, path, no-plan and position-end. Define these constants: line bound (default 1000, range 50 to 100000) and outcome hosts (50).
    • The per-record arena copies every string and number, so a record never points into scheduler memory. The record reports its own byte size, and freeing a NULL record is a no-op.
    • Job selection matching covers all jobs, a plain job, an array element, and every element of an array (including elements created later).
    • Finishing a record as a no-decision kind records the kind and the error.
    • Create the capture source next to the model source, as an empty module for now, and register both in the scheduler build.
    • Add both to the core isolation test's source scan, with a new check that rejects stdio and file I/O in those two files. They are not added to the isolated core state-machine library.
    • Done when model unit tests pass under ASAN for selection matching, arena copying and size, and record finishing, and the isolation test passes with the new files scanned.
    • Requirements: 1.2, 1.3, 1.8, 2.4, 9.1, 11.2

  • 1.4 Build the trace entry sequence and the line budget, test first

    • Path push and pop, with balanced requirement begin and end entries. Each entry stores its full nesting path: single, compound container, alternative i/n, or member j/m.
    • Entry kinds, each rendering as exactly one line:
      • requirement begin and end;
      • a skipped alternative, the alternative result, and a member merge failure;
      • attempt reasons;
      • cluster-wide check end;
      • no plan;
      • plan begin, discarded, not tried, and end;
      • position begin, end, and scan error;
      • host verdict;
      • commit;
      • finalize failure.
    • The line budget stops storing entries once the bound is reached and counts the omitted lines exactly.
    • Done when model tests pass under ASAN for nesting and balance, every entry kind, and budget edges (bound-1, bound and bound+1) with exact omitted counts.
    • Requirements: 3.3, 3.4, 3.6, 4.3, 4.4, 4.5, 4.7, 4.8, 5.10, 9.1, 9.2

  • 1.5 Add plan and verdict counters, search-end statements, summary tallies and rollback merging, test first

    • The model counts generated, tried, discarded and not-tried plans for each requirement scope.
    • The search-end entry's statement follows the design's conditions: none generated, all discarded, or none placed. The decision-wide counters use the same conditions.
    • Per-verdict host evaluations and requirement outcomes stay exact after truncation.
    • At summary level, host entries are not stored. Instead, a position-summary entry holds the accepted host, a count for each verdict kind (a host with several kinds counts once under each), and a tally of hits per limit.
    • Consecutive released-host calls merge into one rollback entry.
    • Done when model tests pass under ASAN for:
      • every search-end condition, including a plan that is not tried after a discarded one;
      • counters after truncation;
      • summary tallies for a host with both a shortfall and a limit hit;
      • rollback merging.
    • Requirements: 4.6, 4.9, 5.8, 5.9, 5.10, 9.3

  • 1.6 Make every white-box test that includes a hooked scheduler source also compile the trace model and capture sources

    • Use the design's include-pattern search to find the 28 tests that include the election, scheduler, allocator-runner, host-iterator, resource or live-driver sources. Give each one the trace model and capture sources, so the live-driver change in 1.7 and the hook tasks after it still link.
    • Done when the full scheduler test suite builds and passes under ASAN with the new includes and no hooks in place yet.
    • Requirements: 11.4

  • 1.7 Add the trace attachment and trace-only op to scheduling decisions

    • Decisions gain a trace attachment, which every existing constructor leaves empty.
    • A new trace-only op, with its own constructor, is appended after the existing ops without renumbering them.
    • The live driver's drain loop skips the trace-only op and frees any attached trace defensively. Live never produces either.
    • Done when a decision test shows every existing constructor leaves the attachment empty, existing op values are unchanged and the trace-only constructor works, and the existing live-driver and decision tests pass unchanged under ASAN.
    • Requirements: 2.4, 10.1, 11.4

  • 1.8 Add the core trace gate and the hook macro, test first

    • The scheduling core gains a session-settings pointer and a record-under-construction pointer. Both stay empty unless the simulator REPL sets them.
    • The hook macro evaluates its call only when a record is being built.
    • Done when a gate test passes under ASAN showing that:
      • a hook argument with a side effect is not evaluated when no record is being built;
      • a full allocation with tracing never enabled creates no record.
    • Requirements: 11.1, 11.2, 11.3

  • 2. Allocator-side capture of scheduler state

  • 2.1 Capture job selection facts, cluster-wide limits and attempt failure reasons, test first

    • Selection capture: policy name, the job's queue and queue priority, the job's priority and its queue time.
    • Cluster-wide limit capture: each matched limit's name and scope instance, read from the job's matched cluster-wide limits.
    • Attempt failure reasons: summarized per pend-reason code, with the hosts and resources they name, captured before the scheduler replaces the job's stored reason.
    • Done when capture tests over a fixture core and job pass under ASAN, showing exact copied values that do not change when the source structures change later.
    • Requirements: 2.2, 2.3, 3.5

  • 2.2 Capture shortfall and limit rows from the satisfaction check, test first

    • Each checked row carries the resource name, flags, value type, total, in-use, provisionally reserved, requested and met.
    • A row whose resource is flagged as a limit is a limit row. It is resolved to its limit through the scope's matched limits and carries the limit name, the scope instance (plus the host for host scope) and the limited resource's name.
    • The cluster-wide scope keeps every row. A host scope keeps only unmet rows, and these set the host's shortfall and/or limit verdict.
    • Done when capture tests pass under ASAN showing that limit rows are never reported as shortfalls, scope strings are correct, and each scope keeps the right rows.
    • Requirements: 4.1, 4.2, 5.4, 5.5

  • 2.3 Capture decision outcomes and reconstructed dispatches, test first

    • Dispatch: per-host allocation rows and shared rows, with limit charges listed separately.
      • A limit charge is identified by the limit flag, or by its name prefix when the resource is unknown.
      • It is resolved to its limit by name through the job's matched limits. An unresolved charge falls back to a raw number.
      • The first 50 hosts are kept in detail. The rest are aggregated into per-resource totals (excluding limits), per-limit totals and an omitted-host count.
    • Pend: a copy of the decision's pend-reason set.
      • The main reason and host-less entries are always kept.
      • The first 50 hosts are kept with all their entries; the rest are counted per pend-reason code.
    • Reconstructed dispatch: a record holding the reconstruction notice and the dispatch outcome.
    • Done when capture tests pass under ASAN for:
      • a dispatch with shared and host limit charges;
      • 51-host dispatch and pend aggregates;
      • a pend copy equal to its source under the existing pend-reason equality, for 50 or fewer hosts;
      • the raw-number fallback.
    • The full scheduler test suite, including the 28 tests updated in 1.6, must still build and pass under ASAN with the completed capture source. Missing scheduler sources are added to those tests, or stubbed.
    • Requirements: 2.5, 6.1, 6.2, 6.3, 6.4, 9.4

  • 2.4 Split the trace sources under the 800-line file cap, with no behaviour change

    • Added during implementation (2026-10-08, user decision). After 2.3, sched_trace.c is 1130 lines and sched_trace_capture.c is 1065, both over the scheduler's 800-line cap (src/scheduler/CLAUDE.md).
    • Move outcome capture (dispatch, pend, reconstructed record) into a new src/sched_trace_outcome.c. Split sched_trace.c along a clear seam, e.g. entries and line budget versus counters, summaries and rollback, so every trace source is under 800 lines. Shared static helpers go behind an internal header (e.g. src/sched_trace_internal.h); the public include/sched/trace.h interface is unchanged.
    • Register every new source in the scheduler's main source list, in the core isolation test's scan and no-stdio check, and in each white-box test that includes the trace sources (the 27 from 1.6, plus the trace model, capture and gate tests).
    • Update design.md's File Structure Plan and TraceModel/TraceCapture file references to the new files.
    • Done when every trace source is under 800 lines, the full scheduler suite passes unchanged under ASAN, and the core isolation test scans the new files.
    • Requirements: 11.2, 11.4
    • Depends: 2.3

  • 3. Allocator-thread hooks at the decision seams

  • 3.1 Hook the allocator loop: position, record lifecycle, selection and no-decision paths (P)

    • Position: when session settings are present, every real job pop increments it.
    • Record start: a wanted job starts a record with its selection facts; its cluster-wide limits are added after election.
    • No-decision paths:
      • not found: no selection;
      • no scheduling object: selection without limits;
      • allocation error, pend without a reason, or decision-build failure: the record is finished as that kind and pushed as a trace-only decision.
    • Normal decisions: a built decision gets its outcome captured and the record attached.
    • Pointer clearing: every path that hands a record away clears the record-under-construction pointer; the session-end marker also clears it.
    • Done when allocator-loop tests using the real allocator thread pass under ASAN for each no-decision kind, the position count, outcome attachment, and the pointer being clear after each job.
    • Requirements: 2.1, 2.2, 2.3, 2.4, 6.1, 6.4, 11.2
    • Boundary: TraceHooks (allocator loop)
    • Depends: 2.1, 2.3

  • 3.2 Hook resource-requirement branching (P)

    • Single, top-level compound, each alternative and each member push their path level and record exactly one begin and one end.
    • The begin carries task counts and requested resources for single requirements and members, and the member count for containers.
    • A member that fails or errors records the number of members not tried. A merge failure after a member is a separate entry.
    • A skipped alternative is recorded in its position. Either the chosen alternative is recorded, with the alternatives after it as not tried, or "all failed" is recorded.
    • Attempt failure reasons are captured before the job's stored reason is replaced.
    • Done when branching tests pass under ASAN for:
      • a single requirement;
      • a compound with a failing member and a merge failure;
      • an alternative job with a skipped, a failed and a chosen alternative;
      • per-alternative reasons; with every scheduler result unchanged.
    • Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6
    • Boundary: TraceHooks (resource-requirement branch)
    • Depends: 2.1

  • 3.3 Hook the host pre-filter, the host scan and the satisfaction check (P)

    • The pre-filter records a selection mismatch and a pre-filter internal error as host verdicts.
    • A host scan that stops on an error marks the current position as a scan error.
    • The satisfaction check hands every row to the current record.
    • Done when host-iterator and resource tests pass under ASAN for the selection-mismatch verdict, the pre-filter internal error, scan-error marking, and row capture in both scopes, with iterator and check results identical to an untraced run.
    • Requirements: 4.1, 4.2, 4.4, 5.2, 5.3, 5.4, 5.5, 5.7
    • Boundary: TraceHooks (host iterator, satisfaction check)
    • Depends: 2.2

  • 3.4 Hook the allocator runner: cluster-wide check, plans, positions, candidates and finalize

    • The runner records:
      • an allocator decline and a generator start failure, as no-plan reasons;
      • the cluster-wide check scope and its end, with "no plans tried" on failure;
      • each generated plan as tried, discarded or not tried;
      • each plan's end, with the failed position or the internal error;
      • each position's begin and end (filled, unfilled, scan error, or internal error after acceptance; the unfilled end is shown as a scan error when 3.3 marked the position);
      • each claimed-host skip;
      • each candidate's per-host result: accepted; rejected, with the verdict taken from the rows 3.3 records, or a selection mismatch when no rows were recorded; or internal error;
      • the released hosts, when a failed plan had claims;
      • commit or finalize failure;
      • search end.
    • Done when runner tests pass under ASAN covering every recorded outcome above, with runner return values and pend reasons identical to an untraced run.
    • Requirements: 4.1, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9, 5.1, 5.2, 5.6, 5.7, 5.8
    • Depends: 2.2, 3.3

  • 4. Simulator REPL trace surface

  • 4.1 Render trace records to text, test first (P)

    • Hierarchy: header, selection, requirements, cluster-wide checks and plans, positions and hosts, then the outcome. Every budgeted entry renders as exactly one line, in the order shown in the design's example.
    • Summary level: renders position summaries with per-limit names.
    • Truncated records: render a marker with the omitted line count, followed by the fixed-size totals block.
    • Outcome: at most 50 hosts, then aggregates and an omitted-host marker. Limit charges are listed separately, with limit amounts formatted by the limited resource.
    • No-decision records: render the "unavailable" lines. Internal-error and pend-without-reason records also render the captured hierarchy.
    • Reconstructed records: render the reconstruction notice and the outcome.
    • Formatting: amounts use the units mode passed in, and job identifiers use the existing job-identifier formatting. Output goes to one or two sinks, with identical bytes in each.
    • Add the renderer source to the scheduler's main source list.
    • The renderer test includes only the model, capture-free fixtures and the shared formatting module.
    • Done when golden renderer tests pass under ASAN for full and summary levels, every units mode, truncation, the outcome cap, each no-decision kind and reconstructed records.
    • Requirements: 2.1, 2.4, 2.5, 3.1, 3.2, 4.3, 5.1, 5.9, 6.1, 6.2, 6.3, 6.4, 7.6, 7.7, 9.2, 9.3, 9.4
    • Boundary: TraceRenderer

  • 4.2 Store the latest trace per job and serve info trace, test first

    • Storing: a new trace for a job replaces its earlier one.
    • Memory bound: accounted from each record's size.
      • At 64 MiB, the oldest traces are evicted, never the newest.
      • Evicted jobs are marked as discarded; a newer capture clears the mark.
    • info trace for a job or element: renders the stored trace at the level it was captured, using the current units mode. With no stored trace, it reports one of: "discarded", "no trace" with the job's state, or "unknown job".
    • info trace for an array: renders each element's trace in index order and marks discarded elements. When no element has a stored trace, it lists the elements' states.
    • Add the store source to the scheduler's main source list.
    • The store test includes the model, the renderer and the shared formatting module, with a fixture job table.
    • Done when store tests pass under ASAN for replacement, eviction that never drops the newest trace, the three no-trace replies, and an array mixing stored and discarded elements.
    • Requirements: 1.7, 7.8, 7.9, 7.10, 7.11, 9.5, 9.6
    • Depends: 4.1

  • 4.3 Parse the trace command and manage trace settings, test first

    • trace re-parses the raw input line, so it is not limited by the normal tokenizer. It accepts:
      • on all, or on job followed by any number of job selectors;
      • an optional level (default summary) and an optional line bound (50 to 100000, default 1000);
      • an optional trailing file path of up to 4095 bytes, taken from the rest of the line, with optional quotes;
      • off and status.
    • An invalid argument is named in an error, and the settings stay unchanged.
    • This task only parses and stores the file path. Opening the file, the same-file checks and leaving settings unchanged when the open fails are added in 4.5.
    • Add the trace control source to the scheduler's main source list.
    • Settings objects are reference-counted and held as "next session" and "current session". Commands change only "next".
    • trace status shows:
      • the settings in effect, or the next settings when no session is running;
      • any replacement waiting for the next session;
      • stored-trace memory use against its bound.
    • Done when command tests pass under ASAN for every grammar form, every invalid-argument class, a path longer than 63 characters, 20 job selectors, and status output before, during and after a session.
    • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.7, 1.8, 1.9, 1.10, 1.12
    • Depends: 4.2

  • 4.4 Pass the scheduler log path to the simulation driver (P)

    • The scheduler's log path goes from the command line through the simulation arguments into the simulation driver, where the trace file checks can read it. Only the simulation driver and the command-line entry point change.
    • Done when a simulation start test under ASAN shows the driver holds the configured log path, or the default path when none is given.
    • Requirements: 8.3
    • Boundary: SimIntegration (driver arguments)

  • 4.5 Manage the trace file copy and output-file conflicts, test first

    • When a command names a file, it is opened for appending at once. An open failure names the file and the cause, and the settings stay unchanged.
    • A path that is the same file (device and inode) as the current simulation output file or the scheduler log is rejected.
    • Expose a conflict check, so set output can reject any trace file that is still open.
    • A replaced or disabled file stays open until the session in progress ends; with no session in progress it is closed immediately.
    • Shutdown flushes and closes every open trace file. Trace code never writes to the scheduler log.
    • Done when file tests pass under ASAN for:
      • an open failure;
      • each same-file rejection, including through a hard link;
      • a replaced file kept open until the session ends;
      • an immediate close;
      • flushing at shutdown.
    • Requirements: 8.2, 8.3, 8.4, 8.5, 8.6, 8.7
    • Depends: 4.3, 4.4

  • 4.6 Emit traces inline and apply settings at session boundaries, test first

    • At session begin:
      • reset the per-round traced and flushed flags;
      • make the next settings current, stamping the session number and resetting the position;
      • publish the settings to the scheduling core, or publish nothing when tracing is off.
    • At session end:
      • withdraw the published settings;
      • release the current settings, closing their file if it was replaced;
      • clear the in-progress state.
    • This task owns making the simulator's decision echo printer non-static and declaring it in the simulator header, with its format unchanged, so continue-mode emission prints exactly the echo line that was suppressed. No other simulation-loop change belongs here.
    • Emitting a traced decision or trace-only record:
      • marks the round as traced, in any mode;
      • in continue mode, flushes any active collapsed range on the round's first traced item, then prints the suppressed echo line for traced decisions;
      • renders the trace to the console and to the current file with identical bytes, flushes both, and hands the record to the store.
    • The emission test includes the REPL and simulation sources it calls: the collapsed-range flush and the echo printer.
    • Done when emission tests pass under ASAN showing:
      • trace placement in step mode;
      • flush, then echo, then trace in continue mode;
      • the flags reset after a round that ended in step mode;
      • the file copy byte-equal to the console output.
    • Requirements: 1.6, 7.1, 7.2, 7.3, 8.1, 8.5

  • 5. Wire the trace into the REPL and the simulation driver

  • 5.1 Make every white-box test that includes REPL or simulation sources compile the trace sources, and register the trace REPL sources with the simulation isolation checks

    • Add the trace control, store and renderer sources to the simulation isolation library and to the simulation isolation test's scan list.
    • Find the tests with the design's full include-pattern search. To the 26 tests that include the REPL or simulation-loop sources, add the trace REPL sources.
    • Of those 26, 19 include no hooked source, so they also get the trace model and capture sources. Where the capture source needs a scheduler source those tests do not already include, add it or provide a stub, and list it in the task notes.
    • Done when all of the following pass under ASAN: a full scheduler build including the scheduler binary, the full scheduler test suite, and the simulation isolation test.
    • Requirements: 10.5, 11.3

  • 5.2 Wire the trace commands into the REPL

    • Route trace with the raw line, and route info trace.
    • Help gains a trailing Trace section, and the info subcommand list and info usage line include info trace.
    • set output rejects an open trace file. Quit and end of input shut tracing down.
    • The continue-mode rollup emits a traced round on its own, after its pend list is built, using the same idle rule as range flushes, and never starts a collapsed range from it.
    • Add the trace entry points and their interactive-only gates to the REPL header.
    • Update the 1.1 baseline fixture for exactly the changed help and info-usage lines and nothing else. The test's diff against the old fixture shows only those lines.
    • Done when REPL tests pass under ASAN for help sections and alignment, command routing, the set output rejection and the traced-round rollup, and the updated baseline test passes.
    • Requirements: 1.11, 7.3, 7.4, 8.4, 8.6, 10.4, 10.5

  • 5.3 Wire tracing into the simulation driver's session and decision flow

    • Session hooks: session begin runs after a successful begin-session and before reconciliation. Session end runs after the decision stream ends.
    • Drain loop:
      • trace-only decisions are handled first, with no verification, recording, echo or breakpoint check;
      • traced decisions go to the trace emitter after the echo gate and before the breakpoint check;
      • unclaimed traces are freed before each decision is freed.
    • Reconstructed dispatches: wanted jobs get a reconstructed record, handled in the same order.
    • The echo-printer export belongs to 4.6.
    • Done when driver tests pass under ASAN showing:
      • trace-only decisions never reach verification or recording;
      • a decision breakpoint stops after that decision's trace is printed and stored, with no later trace stored;
      • reconstructed traces print before the round's live decisions;
      • a non-interactive simulation never publishes settings.
    • Requirements: 1.6, 2.5, 7.1, 7.2, 7.3, 7.5, 10.1, 11.3

  • 6. Validation

  • 6.1 End-to-end trace scenarios over the contention fixture

    • Drive the real simulator REPL with tracing on for selected jobs and for all jobs, at both levels.
    • Assert that every section of Requirements 2 to 6 appears with the expected content:
      • alternative and compound labels;
      • cluster-wide rows and limit rejections;
      • plans, positions and host verdicts;
      • rollback;
      • dispatch and pend outcomes.
    • Assert ordering in step mode, continue mode and at breakpoints.
    • Assert info trace at a breakpoint, after the job finishes, for an array, and after changing units mode.
    • The trace file must equal the inline traces byte for byte.
    • Done when the end-to-end test passes under ASAN.
    • Requirements: 1.1, 1.2, 1.6, 2.1, 2.2, 2.3, 3.1, 3.4, 4.1, 4.2, 4.3, 4.5, 5.1, 5.3, 5.4, 5.5, 5.6, 5.8, 5.9, 6.1, 6.2, 6.4, 7.1, 7.3, 7.4, 7.5, 7.8, 7.9, 8.1

  • 6.2 Error-path trace scenarios with the real allocator

    • Include the scheduler sources directly, so source-level and link-level fault injection apply.
    • Inject each fault and assert its traced outcome:
      • a plan-claims growth failure, through the heap-allocator macro;
      • a snapshot failure, through the vector-copy wrap;
      • finalize failures, through the allocation-constructor wraps;
      • a failing host scan;
      • a failing per-host uncommitted lookup;
      • an allocation function that returns an error;
      • an unknown job identifier placed directly on the job hand-off buffer.
    • Done when the error-path test passes under ASAN and each injected fault produces its specified trace lines.
    • Requirements: 2.4, 3.2, 4.4, 4.5, 5.7

  • 6.3 Prove tracing never changes scheduling

    • Run the contention replay four ways: tracing off, all jobs at summary level, all jobs at full level, and a job subset.
    • Every run must produce identical decision sequences, pend-reason sets, final accounting state, --sim-verify results and --sim-output recordings (apart from wall-clock fields).
    • With tracing off, the baseline transcript (as updated in 5.2) must stay unchanged.
    • The existing equivalence, verification and live-driver suites must pass unchanged.
    • Done when the equivalence test and the full scheduler suite pass under ASAN.
    • Requirements: 10.1, 10.2, 10.3, 10.4, 10.5, 11.4

  • 6.4 Bounds under large traces and the final full build

    • Trace a 1000-host parallel job with up to 64 tasks, at full level with the default bound. The stored entries must stay within the bound, with exact counters.
    • Run 10,000 traced decisions. Stored-trace memory must never exceed 64 MiB.
    • Correct the stale file sizes in the scheduler component guide and list the new files.
    • Done when the bounds test passes and a full ASAN build of all components passes with no test failures.
    • Requirements: 9.1, 9.3, 9.5, 11.2

Implementation Notes

  • 1.1: build dir is src/xce.build. Baseline test sched_repl_trace_baseline_test runs the built xce_scheduler (depends: scheduler); regenerate fixtures with XCE_REPL_BASELINE_REGEN=1 (script.txt is hand-written). Fixture sends the license value as a RESVALS_FULL event because shared resvals in the snapshot are dropped on load (bug-1923). sim_output.replay has no session decisions (set output switches files before session 1 is finalized); decisions are covered by set_output.replay and sim_verify.txt.
  • 1.3: sched_trace_ctl_wants whole-array selectors match elements only, not the bare key. A bare id in trace on job <id> can't be told apart from an array base for a job that doesn't exist yet, so 4.3's parser must emit both {key,false} and {key,true}, or the matcher must also accept jobid == key.
  • 1.3: design's record header lists a "file tag", but xce_sched_trace_ctl_t has no such field; the 4.2/4.6 store/emit tasks must set it or drop it explicitly.
  • 1.3: trace sources (sched_trace.c, sched_trace_capture.c) may not include <stdio.h> (core isolation test); capture copies raw strings/numbers and leaves formatting to the renderer.
  • 1.4: sched_trace_host_verdict(rec, 0, err) means "derive from rows" (rec->host_row_verdicts shortfall/limit, else SELECT, phase per-host); an explicit SELECT is the pre-filter phase. 2.2's row capture must set host_row_verdicts and 3.3/3.4 hooks must follow this. sched_trace_pos_scan_error stores no entry; it turns the next UNFILLED position end into SCAN_ERROR. Plan numbers restart per requirement. trace.h forward-declares struct xce_alloc_plan_s (tag added in alloc_iter.h) to avoid the core.h include loop. resreq_begin may store a NULL req name for a NULL rusage; 4.1's renderer must tolerate it.
  • 1.5: the per-limit summary tally goes through sched_trace_host_limit_hit(rec, limit, scope) (added; not in the design's interface); 2.2 calls it for each unmet limit row in a host scope and keeps setting host_row_verdicts |= SHORTFALL directly. At summary level pos_end stores a POS_SUMMARY in place of POS_END, keeping the end state, error and host; 4.1 must render a scan error in that line. The search-end statement is written only when rc==0.
  • 1.5 OPEN for 3.4/4.1: a requirement the allocator declines (alloc_runner.c ~474) or whose plan generator fails to start (~427, returns 0) never reaches sched_trace_search_end when that call sits only at the last return, so reqs_none_generated misses it (Req 4.7, 9.3); calling search_end on that path as well double-reports "no plan generated". Settle it in 3.4, e.g. sched_trace_no_plan bumps the counter and search_end skips its statement when a no-plan entry is open. test_search_end_nonzero_rc_has_no_statement models a generator failure as rc -12, but the runner returns 0 there.
  • 1.6: 27 tests include a hooked source and now #include sched_trace.c and sched_trace_capture.c. test_snapshot_load_allocator_regression.c matches the grep only in a comment and links scheduler_srcs, which already has both. Still open for the hook tasks: sched_sim_isolation_lib (src/scheduler/meson.build ~165-173) compiles the hooked sched_election.c without the trace sources (1.8/3.1 must add them or a stub). The 19 repl/sim tests matched via repl.c/sim_loop.c belong to 5.1.
  • 1.7 OPEN for 5.3: the sim decision-free sites at sim_loop.c ~1327 (reconstructed dispatch) and ~1709 (drain loop) don't yet call sched_trace_rec_free(d->trace), and the TRACE op still falls to the sim's switch defaults. 5.3 must add the early TRACE branch before verify/record/breakpoint, or sim_verify reports false mismatches and breakpoints count it as "NONE". decision.h forward-declares struct xce_sched_trace_rec_s rather than including sched/trace.h.
  • 1.8: core->trace_ctl and core->trace_rec are plain forward-declared pointers (core.h does not include sched/trace.h), and xce_sched_core_fini() clears them without freeing because core.c can't link the trace sources. test_trace_hooks_gate.c counts sched_trace_rec_new with a link-time wrap. Its "full allocation creates no record" check only becomes meaningful once 3.1 adds the per-job hook. The 11.3 check (non-interactive sim never publishes trace_ctl) is in 5.3.
  • 2.1: capture reads struct fields only and calls no scheduler functions, so the white-box tests need no extra sources. Policy name = ctl policy_name, else core->job_pick_plugin; the queue picker override is ignored because election always uses the core picker. Cluster-wide limit scope is stored as "u=..,q=..,p=.." ("" when unscoped); 2.2 should reuse split_vres_name/copy_scope and append "h=". 4.1 must render selection_set/glimits_set false as "unavailable" and an ATTEMPT_REASONS entry with ncodes==0.
  • 2.2: rows go through sched_trace_row(), a model addition not named in the design, as CHECK_ROW (cluster-wide check, all rows) or HOST_ROW (host scope, unmet rows only). Host rows are held until the host verdict is stored and then stored one level deeper, each counting against the budget; they are dropped at summary level, where the verdict bits and limit tally are still set. Rows arriving outside an open check (rec->check_open) or a host scope are dropped, so 3.3/3.4 must open a host scope before host_iter's pre-filter calls sched_check_host_satisfy and must call sched_trace_check_end on every exit path of the cluster-wide check. 4.1 must render a NULL row name by its index, limited_res == NULL as a raw amount, and HOST_ROW at host depth + 1. Capture now calls librtti vget(), which every test already links.
  • 2.2 OPEN for 6.2: if limit_row() runs out of memory, sched_trace_check_rows returns early, so that row's LIMIT bit and later rows' SHORTFALL bits are lost. The verdict derived from rows then becomes SELECT and the Req 9.3 counts are off. This needs fault injection to test, so fix it in 6.2: keep setting the verdict bits and continue.
  • 2.3: the outcome capture functions are sched_trace_capture_dispatch, sched_trace_capture_pend and sched_trace_rec_reconstructed, declared in trace.h; 3.x and 5.3 call them. Limit charges are classified by the limit flag or a limit= prefix and resolved via glimits (shared) or the worker's hlimits (host, worker looked up by host name). An unresolved charge has limited_res == NULL, which is the raw-number fallback. Shared pend entries carry host "" and are treated as host-less. For 4.1: limit totals over omitted hosts have a scope without the host, and a NULL host name yields the scope "h=". Follow-up (low): capture's key_find/vget on resmgr/workers run without the RTTI read lock that sched_get_worker takes. That is safe while the sim runs in lock-step; the 2.4 split may wrap them.
  • 2.4: the trace sources are now sched_trace.c (record, arena, entries, budget, requirement/plan/check), sched_trace_scan.c (positions, host verdicts/rows, tallies, rollback, commit, search end), sched_trace_capture.c, sched_trace_outcome.c (dispatch, pend, reconstructed) and sched_trace_internal.h (12 shared sti_ helpers). Every target that compiles one trace .c file needs all four. Later tasks that extend the trace must keep each file under 800 lines and add new files to the core isolation scan.
  • 3.1: allocator-loop hooks live in sched_allocator() (sched_election.c), with helpers sched_trace_push_no_decision() and sched_trace_attach(). Selection is captured at job lookup, before election, so NO_SCHEDOBJ still has its selection; cluster-wide limits are captured after election. The position counter (ctl->position) is not SCHED_TRACE-gated (it creates the record) and counts only real pops. sched_sim_isolation_lib stays without the trace sources (unresolved references in a static archive are harmless); 5.1 decides whether to add them. core->decisions holds 5 items, so trace-only decisions take FIFO slots and cannot reorder real ones.
  • 3.2: requirement-branch hooks are in scheduler.c (sched_make_allocation, sched_make_alter_allocation, sched_make_compound_alloc_). The alternatives after the chosen one are not stored as entries; 4.1 renders "alternatives i+1..n not tried" from the alt_result (i,n). A merge failure stores an ATTEMPT_REASONS entry with ncodes==0. A failing single requirement records no attempt reasons; its pend outcome carries them. A single job with a NULL resreq stores its begin as a container with 0 members before returning -EINVAL, and 4.1 must tolerate that.
  • 3.3: the pre-filter (sched_host_iter_next) opens a host scope (sched_trace_host_begin) for every candidate and closes it with a verdict on each rejection: SELECT (pre-filter), INTERNAL(rc) for an uncommitted-fetch or check error, from-rows for a resource rejection (unreachable from the runner). A host the iterator returns is left in an open scope with no entry, so 3.4 must end or replace it for the accepted, already-claimed, snapshot-failure and per-host-error paths, or later rows attach to a stale host. A scan stopped with rc<0 calls pos_scan_error; exhaustion does not. unsatisfy_ hands every row to sched_trace_check_rows. The sim thread's reconciliation reaches that hook but sees trace_rec NULL.
  • 3.4: runner hooks are in alloc_runner.c. The 1.5 OPEN item is settled: sched_trace_no_plan counts reqs_none_generated once and sets scope_no_plan, after which search_end makes no statement and resreq_begin/search_end reset the flag. So a declined or generator-failed requirement has a NO_PLAN entry and no SEARCH_END line, and 4.1 must render the NO_PLAN line as "no plan generated: ". The 3.3 open scope is closed: every returned candidate gets a fresh host scope and one verdict, and check_begin drops any stale host rows. When the iterator can't start, the position ends as SCAN_ERROR with err 0, so 4.1 must render a scan error with no code. trace.h is at 790 lines; split it before adding to it.
  • 4.1: the renderer is repl_trace_render(rec, units, out1, out2), declared in repl_internal.h, in repl_trace_render.c plus repl_trace_render_outcome.c, with a private repl_trace_render.h. It renders once per sink, so the bytes are identical. Amounts use repl_format_numeric_resource_value ("8.00 GB" in human mode) and times use repl_format_ts_human. Wording where the design is silent is fixed by the golden tests in test_trace_render.c, so 4.2/4.6/6.1 must reuse the renderer, never re-format. Pend-reason arrays may contain NULL entries (capture keeps them); every walk must skip them. The renderer is not yet in sched_sim_isolation_lib or the sim isolation scan list (5.1).
  • 4.2: the store API is repl_trace_store_put, repl_trace_store_bytes, repl_cmd_info_trace and repl_trace_store_fini (an addition; nothing calls it at shutdown yet, so 4.5/4.6/5.2 must). xce_repl_trace_state_t (repl_internal.h) holds only the store fields; 4.3 adds next/cur/session flags to it. The 1.3 "file tag" is dropped: info trace is console-only (Req 8.1) and emission writes through the session's cur settings. info trace is not yet routed from repl_cmd_info (5.2). Open (6.2): an id_list_add OOM can silently drop array elements from a listing.
  • 4.3: trace is parsed from the raw line by repl_trace.c. The settings are a refcounted REPL-side wrapper around xce_sched_trace_ctl_t (trace.h untouched), held as next/cur in xce_repl_trace_state_t together with session_in_progress, round_traced and round_flushed. The 1.3 bare-id item is settled: an array-aligned bare id adds both an exact and a whole-array selector, so later elements match; status shows that as <base>[*]. A range expands to one exact selector per element. The wrapper frees ctl.policy_name on its last release, so 4.6 must heap-allocate it. Nothing frees next/cur at REPL shutdown yet, so 4.5/4.6 must add the shutdown path; routing trace from repl.c is 5.2.
  • 4.4: the driver holds its own copy of the scheduler log path in sim->log_path (from sched_sim_args_t.log_path, else SCHED_DEFAULT_LOG_PATH in sim.h, "/tmp/xce_scheduler.log"). The log is always open in sim mode, so 4.5 can stat(sim->log_path) for its dev/ino same-file check.
  • 4.5: the trace FILE* and its dev/ino belong to the refcounted settings, and the last unref flushes and closes it. New functions in repl_internal.h: repl_trace_settings_file(), repl_trace_file_conflicts(sim, path) (5.2 wires it into set output) and repl_trace_shutdown(sim) (5.2 calls it on quit/EOF). Shutdown clears core.trace_ctl, releases cur/next, then calls store_fini, which memsets the trace state, so keep that order. 4.6 must emit only through cur's file while it still holds cur, and flush after every record, because two append streams may share one file.
  • 4.6: the session hooks and emitter are in repl_trace_emit.c: repl_trace_session_begin/end(sim) and repl_trace_on_decision/on_trace_only(sim, d), declared in repl_internal.h. sim_print_decision is now exported from sim.h. Session begin stamps session_no = rs->session_count, which sim_loop.c increments in xce_repl_check_begin_round before xce_core_begin_session. 5.3 must call session_begin after a successful begin-session and before signal_go/reconciliation, and session_end after the decision stream ends. Emission moves the record to the store (d->trace=NULL); write errors are reported once per session. policy_name is not stamped; capture falls back to the core picker. repl_trace_emit.c is not yet in sched_sim_isolation_lib or its scan (5.1).
  • 5.1: all 26 tests that include repl.c/sim_loop.c now #include the five REPL trace sources (repl_trace_render.c, repl_trace_render_outcome.c, repl_trace_store.c, repl_trace.c, repl_trace_emit.c) and the four core trace sources, exactly once. sched_sim_isolation_lib carries all nine trace sources. test_sim_isolation.c's test_sim_trace_repl_sources_are_isolated scans the REPL trace files plus repl_trace_render.h; stdio is allowed there, controller symbols are not. A new REPL trace file must be added to the 26 tests, the lib and the scan. test_sim_startup.c includes only sim.c, so if sim.c starts calling trace code it needs the sources too.
  • 5.2: repl.h now has the interactive-only gates xce_repl_trace_session_begin/end(sim) and xce_repl_check_trace/xce_repl_check_trace_only(sim, d); the two check gates also no-op when d->trace is NULL. 5.3 calls these from sim_loop.c. trace is routed with the raw line, info trace is in usage and help, set output checks repl_trace_file_conflicts first, and quit/EOF call repl_trace_shutdown. The continue-mode rollup prints a traced round on its own via rs->trace.round_traced. The baseline transcript changed only in the help and info-usage lines; regenerating also rewrites the .replay fixtures' wall-clock fields, so restore those. Pre-existing: repl_count_jobs (repl.c) never destroys its iterator, leaking 16 B per continue round; the rollup tests run with detect_leaks=0.
  • 5.3: sim_loop.c calls the repl.h gates. Session begin runs after a successful begin-session and before reconcile/signal_go; session end runs after the drain and verify tail, before plugin_end_session. The drain loop handles TRACE first (skipping verify, record, echo and breakpoint), then echo gate, check_trace, check_decision, rec_free, free. The reconstructed block uses the new gate xce_repl_trace_reconstructed (repl_trace_on_reconstructed in repl_trace_emit.c). Settles the 1.7 OPEN item (both free sites). A reviewer's trace-on vs trace-off run of the contention fixture gave byte-identical --sim-output, identical verify and identical decisions.
  • 6.1: the e2e test (test_repl_trace_e2e.c, its _fixture.c and .h) drives the built xce_scheduler: step/full/all with the file copy, continue/breakpoint/summary/info trace, and selected jobs over a run-time extra fixture with compounds, an array and a selection mismatch. XCE_TRACE_E2E_DUMP=1 prints the transcript. Findings: a compound dispatch shows one entry per member on a shared host, which is faithful (merge_job_alloc keys on instid); break DECISION jobid= matches only raw decimal ids, a pre-existing repl_break.c bug logged separately.
  • 6.2: test_trace_error_paths.c injects seven faults into the real allocator thread. Hooked sources are #included for the macro renames; the rest are linked so the wraps see their callers. XCE_TRACE_ERR_DUMP=1 prints the rendered traces. The 2.2 OPEN item is fixed: a limit_row OOM now marks the record incomplete, sets LIMIT via sched_trace_host_limit_hit(rec, NULL, NULL) and continues. The 4.2 OPEN item is fixed: an array listing OOM prints "[sim] info trace: array : cannot list its elements: ". The capture and store tests now #include their sources.
  • 6.3: test_trace_equivalence.c (with _cmp.c and .h) runs four variants (off, all/summary, all/full+file, subset) in step and continue mode over the contention and 6.1 extra replays. It compares decoded decisions with full pend reasons, recording and snapshot bytes, verify output, and per-step worker/resource/limit dumps; XCE_TRACE_EQUIV_DUMP=1 prints the transcripts. No difference was found. job->accounts is compared only indirectly, through snapshot job->allocs and the used totals.
  • 6.4: test_trace_bounds.c (real allocator, 1000 workers, maxTasks=64) stores exactly 1000 entries and counts 187,930 omitted, with counters equal to the unbounded run. Over 10,000 traced puts the store peaks at 67,108,800 bytes and never evicts the newest. XCE_TRACE_BOUNDS_DUMP=1 prints the numbers; the meson timeout is 300 s. The full all-components ASAN build passed: 148 C tests, xceadmin 373, Go ok. The large pend outcome showed ~63 duplicate NO_RESOURCE reasons per host, a pre-existing libxce/scheduler behaviour logged separately.