myxce/.kiro/specs/scheduler-sim-decision-trace/requirements.md

Requirements Document

Project Description (Input)

Who has the problem: People investigating scheduler behavior in the xce_scheduler simulator REPL (--sim -i): scheduler developers and operators debugging why jobs were placed or pended.

Current situation: The REPL shows only what was decided (dispatch/pend, jobid, at best a coarse xce_pendreason_t). It never shows why. The reasoning the scheduler computes is discarded:

• Hosts rejected in host_iter.c leave no record.
• Per-resource shortfalls (unsatisfy_) are only TRACE-logged.
• ALTER keeps only the last alternative's reason.
• A limit hit is indistinguishable from NO_RESOURCE.
• Plan and bucket attempts are never recorded.

The only workaround is printf debugging or -vvv tlog, which is written to a log file rather than the REPL and cannot be filtered to one job.

What should change: Add a simulator-only decision trace, enabled at runtime from the REPL for every job or for a chosen set of jobids. For each traced job, the REPL console prints a readable, nested explanation of the full decision:

1. Why the job was selected (picker, queue, sort keys, position) and the election result.
2. Each resreq tried (primary / COMPOUND member / ALTER alternative), in order, with its outcome.
3. Each plan the planner generated and tried (host count, buckets, slots) and which one was committed.
4. For each bucket, the hosts considered in order and why each could or could not meet the resreq: select mismatch, per-resource shortfall with numbers, shared-resource shortfall, the named limit hit, host already claimed, or rolled back.
5. The final allocation details (per-host/per-resource amounts, shared allocs, limits applied) or the final pend reason.

info trace <jobid> re-prints a job's latest trace, and an optional file mirror is available. Records are collected in a per-decision buffer on the allocator thread, attached to the decision, and rendered by the sim driver, so the core does no I/O.

Constraints:

• The live scheduler never enables the trace. Because tracing has a performance cost, when it is off each hook costs at most one branch.
• Enabling the trace must not change any decision, allocation, pend reason, accounting state, or --sim-verify result.
• With the trace off, REPL output is unchanged.
• No new cross-thread synchronization is added.
• The trace is capped per decision.
• Trace code goes in new files.
• Sequenced after scheduler-allocator-runner, so the plan/bucket hooks go into sched_allocator_run().

Full context: brief.md in this directory.

Introduction

The xce_scheduler simulator (--sim with the interactive REPL, -i) replays recorded scheduling sessions so developers and operators can investigate how the scheduler behaves. Today the REPL reports only the outcome of each scheduling decision (dispatch or pend, plus at best a coarse pend reason). The reasoning behind that outcome is not shown:

• why the job was selected when it was
• which resource requirement was tried
• which placement plans were attempted
• which hosts were considered and why each could or could not satisfy the job
• which resources and limits were finally allocated

In this document a round (the REPL's term) and a scheduling session are the same unit: one pass of the scheduler over the pending jobs. A scheduling session is in progress from the moment the scheduler begins it, before its first reconstructed dispatch (Requirement 2.5) or job selection, until the simulator has reported the session's last decision or no-decision selection; it has ended after that point. A breakpoint that stops before the session begins (for example a round-begin breakpoint) is not within that session. A decision or no-decision selection is reported when the simulator processes it in the order the scheduler produced it, which is the point at which step mode echoes a decision.

A placement plan divides a resource requirement's tasks across a number of hosts. Each plan position of a plan has a fixed number of tasks and is filled by exactly one host, and no host fills two positions of the same plan. (In the code a plan position is called a bucket.)

This feature adds a decision trace to the simulator REPL. The operator turns it on at runtime for all jobs or for chosen jobs, at a chosen level of detail. The explanation is printed inline as the replay proceeds and can be recalled per job afterwards. The trace is purely an investigation aid: it exists only in the simulator, so the production scheduler pays no cost for it, and enabling it never changes what the scheduler decides.

Boundary Context

• In scope:
  • REPL commands to enable, disable, scope and inspect decision tracing.
  • The content of a decision trace for each traced job's dispatch or pend decision: job selection, resource-requirement selection, placement plans, per-host verdicts, final allocation or pend reason.
  • Inline display of traces during step and continue; recall of a job's latest trace.
  • An optional file copy of trace output.
  • Bounds on per-decision trace size and on stored-trace memory.
  • The guarantee that tracing does not alter scheduling results.
• Out of scope:
  • Tracing in the live (controller-connected) scheduler, by any means (CLI flag, configuration, admin command).
  • Tracing in a non-interactive simulation (--sim without -i).
  • Preemption decisions.
  • Any change to scheduling, job-picking, or allocation behavior.
  • Any change to the pend reasons published to the controller, written to recordings, or compared by --sim-verify.
  • Fixing scheduler defects the trace makes visible (for example, an alternative-requirement job reporting only its last alternative's pend reasons); those are tracked as separate bugs.
  • Limit configuration/usage inspection (info limit, owned by scheduler-sim-diagnostics).
• Adjacent expectations:
  • This feature explains decisions made by the existing job picker and allocators (scheduler-allocation, as restructured by scheduler-allocator-runner) and does not redefine them.
  • It extends the REPL command set and output conventions shipped by scheduler-core, scheduler-sim-repl-verbosity and scheduler-sim-step-verbosity, including the set units display mode, job-identifier range compression, and the sectioned help layout.
  • It amends scheduler-sim-repl-verbosity Requirements 2.1 and 3, only while tracing is enabled: a continue-mode round that contains a traced decision also prints that decision's echo line and trace (Requirement 7.3 below), and never joins a collapsed range (Requirement 7.4 below). With tracing off, all REPL behavior defined by those specs is unchanged.
  • It relies on scheduler-sim-discrete-clock: the simulation clock must not advance with the wall-clock time spent collecting traces, so that the time recorded for each decision is the same with tracing on or off.
  • It takes over the decision-reasoning trace that scheduler-sim-diagnostics had deferred.

Requirements

Requirement 1: Trace Control

Objective: As an operator investigating a replay, I want to turn decision tracing on and off at runtime and choose which jobs are traced and at what detail, so that I get explanations for exactly the jobs I care about without restarting the simulation.

Acceptance Criteria

1. When the operator issues a trace-enable command naming "all jobs", the Simulator REPL shall trace every dispatch and pend decision made in the scheduling sessions that start after the command, until tracing is disabled.
2. When the operator issues a trace-enable command naming one or more job identifiers, the Simulator REPL shall trace only decisions for those jobs, where a job-array identifier selects all of its elements and a job-array element identifier selects only that element.
3. When a trace-enable command names a job identifier that does not yet exist in the replay, the Simulator REPL shall accept it and trace that job's decisions once the job appears.
4. When the operator issues a trace-enable command while tracing is already enabled, the Simulator REPL shall replace the previous job selection, detail level, size bound and file setting with the new command's settings.
5. When the operator issues the trace-disable command, the Simulator REPL shall stop tracing decisions in the scheduling sessions that start after the command and shall keep already-captured traces available for recall.
6. If a trace-enable or trace-disable command is issued while a scheduling session is in progress (for example at a decision breakpoint), the Simulator REPL shall apply the new settings from the next scheduling session, leaving the in-progress session traced according to the settings it started with.
7. When the operator issues the trace-status command, the Simulator REPL shall display the settings in effect for the scheduling session in progress, or, when none is in progress, the settings the next scheduling session will use, and separately any settings that will replace them at the next scheduling session, each showing whether tracing is enabled, the traced job selection, the detail level, the per-decision size bound, the file copy destination if any, the stored-trace memory bound and current stored-trace memory use.
8. The Simulator REPL shall support two detail levels, summary and full, selectable in the trace-enable command, and shall use summary when no level is given; the two levels differ only in how host verdicts are shown (Requirement 5).
9. If a trace command has an invalid argument (unknown option, malformed job identifier, invalid level, a size bound that is not a whole number from 50 to 100000, or a file path longer than 4095 bytes), the Simulator REPL shall print an error naming the invalid argument and leave the current trace settings unchanged.
10. While tracing is enabled, the Simulator REPL shall keep the trace settings in effect across scheduling sessions until the operator changes or disables them.
11. The Simulator REPL shall list the trace commands and info trace in its help output, with the trace commands in their own labeled section and info trace among the other info subcommands.
12. The Simulator REPL shall accept in a trace-enable command any number of job identifiers and a file path of up to 4095 bytes, even where these exceed the argument count or argument length accepted by other REPL commands.

Requirement 2: Job Selection Explanation

Objective: As an operator, I want each traced decision to explain why the job was considered when it was, so that I can understand job ordering and why a job was or was not reached earlier.

Acceptance Criteria

1. When a traced job is selected for allocation in a scheduling session, the Simulator REPL shall show in its trace the session number (numbered as in the REPL's event summary for that session) and the job's 1-based position among the jobs selected for allocation in that session.
2. When a traced job is selected for allocation, the Simulator REPL shall show the job-ordering policy in use and the job's ordering keys under that policy with the values in effect when the job was selected (for the default first-come-first-served policy: the job's queue, the queue's priority, the job's priority and the job's queue time).
3. When a traced job is selected for allocation, the Simulator REPL shall show the cluster-wide limits matched to the job before placement, and shall state that host-scoped limits are reported with each host's verdict.
4. If a traced job is selected for allocation but no scheduling decision is produced for it, the Simulator REPL shall show a trace stating which of these occurred: the job could not be found by the allocator; the job has no scheduling object (its resource requirement could not be parsed); the allocation attempt ended in an internal error, including a failure to build the decision; or the job was pended without any pend reason. When the job could not be found, the trace shall show the session, position and job identifier and state that the ordering keys and limits are unavailable; when the job has no scheduling object, the trace shall show the selection section without limits and state that the limits are unavailable.
5. When a traced job is dispatched by the simulator's reconstruction of a recorded allocation, rather than by the scheduler's own placement, the Simulator REPL shall state this in place of the selection, requirement, plan and host sections of the trace.

Requirement 3: Resource Requirement Selection Explanation

Objective: As an operator, I want to see each resource requirement the scheduler tried for a job and the result of each, so that I can understand which requirement was satisfied and why others were not.

Acceptance Criteria

1. When the scheduler evaluates a traced job, the Simulator REPL shall show in the trace every resource requirement tried, in evaluation order, each labelled as a single requirement, the i-th member of n of a compound requirement, or the i-th of n alternatives, with the full nesting path shown when requirements are nested.
2. The Simulator REPL shall show for each tried single requirement or compound member its minimum and maximum task counts, its requested resources and whether it was satisfied, not satisfied, or ended in an internal error, and for each tried compound requirement or alternative whether it was satisfied, not satisfied, or ended in an internal error.
3. When a member of a compound requirement cannot be satisfied or ends in an internal error, the Simulator REPL shall show that member as the point of failure with that outcome and list the remaining members as not tried.
4. When an alternative requirement is satisfied, the Simulator REPL shall mark it as the chosen alternative and list the remaining alternatives as not tried.
5. When every alternative requirement fails, the Simulator REPL shall show the failure reasons of each alternative, including an internal error, not only those of the last one.
6. When the scheduler skips an alternative without evaluating it because the alternative has no requirement content, the Simulator REPL shall show it as skipped, in its position in the evaluation order.

Requirement 4: Shared Resource and Placement Plan Explanation

Objective: As an operator, I want to see the cluster-wide checks and the placement plans tried for each resource requirement, so that I understand how the scheduler decided how many hosts and how many tasks per host to use.

Acceptance Criteria

1. When the scheduler checks cluster-wide shared resources for a traced job's resource requirement, the Simulator REPL shall show each shared resource checked, other than cluster-wide limits (which are shown as limit checks with the fields of criterion 2 and whether they passed), with its total, the amount in use, the amount provisionally reserved by earlier parts of this job's evaluation, the amount requested, and whether the check passed.
2. When the cluster-wide check fails on a cluster-wide limit, the Simulator REPL shall show the failure as a limit rejection naming the limit, the scope instance it applies to (for example the user, queue or project), the limit value, the current usage, the amount provisionally reserved by earlier parts of this job's evaluation and the amount requested, and shall not also show it as a resource shortfall.
3. When the planner tries a placement plan for a traced job, the Simulator REPL shall show the plan in the order tried, with the number of hosts and the number of tasks assigned to each plan position.
4. The Simulator REPL shall show for each tried plan whether it succeeded or, if it failed, which plan position could not be filled or that the attempt ended in an internal error, in which case the Simulator REPL shall state that no further plans were tried; if a generated plan cannot be tried because of an internal error, the Simulator REPL shall show that plan as not tried with the error and state that no further plans were tried. If a plan position could not be filled because the host scan could not start or stopped on an internal error, the Simulator REPL shall show that position as not filled because of an internal error in the host scan, with the hosts considered before the error shown as Requirement 5 specifies for the current detail level, and shall show the plans tried after that plan.
5. When a plan succeeds and the allocation built from it is completed, the Simulator REPL shall mark it as the committed plan and show that no further plans were tried for that resource requirement; if every position of a plan is filled but building the allocation from it ends in an internal error, the Simulator REPL shall show that plan as ended in an internal error, not as committed, and state that no further plans were tried.
6. If every tried plan for a resource requirement fails, the Simulator REPL shall state that no plan could be placed for that requirement.
7. If no placement plan could be generated for a resource requirement, including when the allocator declines the requirement before generating plans, the Simulator REPL shall state that no plan was generated and the reason.
8. When the cluster-wide check for a resource requirement fails, the Simulator REPL shall state that no placement plans were tried for that requirement because of that failure.
9. When the planner generates a plan that the scheduler discards without trying it, the Simulator REPL shall show that plan as discarded, in the order generated; if every generated plan is discarded, the Simulator REPL shall state that no plan was tried because all generated plans were discarded.

Requirement 5: Host Verdict Explanation

Objective: As an operator, I want to know for each host considered why it could or could not meet the job's resource requirement, so that I can pinpoint the resource, selection expression, or limit that blocked a placement.

Acceptance Criteria

1. While the detail level is full, when a host is considered for a position of a traced job's plan, the Simulator REPL shall show that host, in the order considered, with its verdict: accepted, or rejected with every reason found by the checks the scheduler performed for that host; checks the scheduler did not perform for that host shall be shown as not performed, which is not a verdict kind (Requirement 5.10), and the Simulator REPL shall not perform additional checks to find further reasons.
2. While the detail level is full, when a host is rejected with a selection-mismatch, resource-shortfall or limit-exceeded verdict, the Simulator REPL shall indicate whether the host was rejected by the host-selection pre-filter or by the per-host resource and limit check.
3. While the detail level is full, when a host does not match the job's host selection expression, the Simulator REPL shall show a selection-mismatch verdict for that host.
4. While the detail level is full, when a host is short of one or more resources, the Simulator REPL shall show each resource short on that host with the host's total, the amount already in use, the amount provisionally reserved on that host by earlier parts of this job's evaluation, and the requested amount the check actually compared against.
5. While the detail level is full, when a host-scoped limit would be exceeded on a host, the Simulator REPL shall show a limit rejection naming the limit, the scope instance it applies to, the limit value, the current usage, the amount provisionally reserved by earlier parts of this job's evaluation and the requested amount the check actually compared against, and shall not also show that limit as a resource shortfall.
6. While the detail level is full, when a host is skipped because an earlier position of the same plan already claimed it, the Simulator REPL shall show the host as already claimed.
7. While the detail level is full, when a host could not be evaluated because of an internal error, the Simulator REPL shall show the host as not evaluated with the error.
8. When a plan fails after some of its positions were filled, the Simulator REPL shall show that the hosts claimed for the earlier positions were released.
9. While the detail level is summary, the Simulator REPL shall show, for each plan position, the host accepted (if any) and the number of hosts rejected for each verdict kind, counting a host with several rejection reasons once under each verdict kind it matched as in Requirement 9.3, and, for the limit-exceeded count, the name and scope instance of each limit hit with its count, instead of one line per host.
10. The Simulator REPL shall use these host verdict kinds wherever host verdicts are listed or counted: accepted, selection mismatch, resource shortfall, limit exceeded, already claimed, and not evaluated because of an internal error.

Requirement 6: Decision Outcome Explanation

Objective: As an operator, I want each trace to end with the decision's concrete result, so that I can see exactly what was allocated or why the job stays pending.

Acceptance Criteria

1. When a traced job is dispatched, the Simulator REPL shall show in the trace, for each allocated host, each resource allocated with its amount, and, separately and labelled as limit charges, each host-scoped limit charged on that host with the limit name, its scope instance and the amount charged, subject to the host limit of Requirement 9.4.
2. When a traced job is dispatched, the Simulator REPL shall show once for the job the shared resources allocated with their amounts and, separately and labelled as limit charges, each cluster-wide limit charged with the limit name, its scope instance and the amount charged.
3. The Simulator REPL shall show each limit charge only as a limit charge, never among a host's allocated resources or the job's shared resources.
4. When a traced job is pended, the Simulator REPL shall show the pend reason set of the decision, with the same content as the pend decision the simulator records (--sim-output) and compares (--sim-verify), subject to the host limit of Requirement 9.4.

Requirement 7: Trace Display and Recall

Objective: As an operator, I want traces printed inline as the replay runs and recallable afterwards, so that I can read explanations as decisions happen and revisit a job later.

Acceptance Criteria

1. While the REPL is in step mode, when a traced decision is echoed, the Simulator REPL shall print that decision's trace immediately after the decision's echo line.
2. While the REPL is in step mode, when a traced job's selection produces no decision (Requirement 2.4), the Simulator REPL shall print its trace at the point where the job's decision would have been echoed.
3. While the REPL is in continue mode, when a traced decision is reported or a traced job's selection produces no decision, the Simulator REPL shall at that point end and report any active collapsed range and print the traced decision's echo line followed by its trace (or the no-decision trace), so that a round's traces appear in the order its decisions are reported, with reconstructed dispatches (Requirement 2.5) first, followed by the round's summary line when the round completes.
4. While the REPL is in continue mode, the Simulator REPL shall not include a round that contains a traced decision or traced no-decision selection in a collapsed range.
5. When execution stops at a breakpoint, the Simulator REPL shall have printed the traces of the round's traced decisions and no-decision selections reported up to and including the one at which execution stopped, and no later trace from the current round shall be available to info trace at the breakpoint; those printed traces shall be available to info trace unless discarded under the memory bound (Requirement 9.5), in which case Requirement 9.6 applies.
6. The Simulator REPL shall present each trace as an indented hierarchy: job selection, then resource requirements, then shared-resource checks and plans, then plan positions and hosts, then the outcome.
7. The Simulator REPL shall show resource amounts in traces using the operator's current set units display mode and job identifiers using the REPL's existing job-identifier formatting.
8. When the operator issues info trace <jobid>, the Simulator REPL shall print the most recent stored trace for that job, at the detail level at which it was captured and with resource amounts shown in the set units display mode in effect when info trace is issued, including for a job that has since finished or been removed from the replay.
9. When the operator issues info trace with a job-array identifier, the Simulator REPL shall print the most recent stored trace of each element of that array that has one, mark each element whose trace was discarded under the memory bound, and, if no element has a stored trace, report that and list the elements' current states.
10. When a new trace is captured for a job that already has a stored trace, the Simulator REPL shall replace the stored trace with the new one.
11. If the operator issues info trace <jobid> for a job that has no stored trace and none discarded under Requirement 9.5, the Simulator REPL shall report that no trace is available and show the job's current state, or state that the job is unknown.

Requirement 8: Trace File Copy

Objective: As an operator, I want an optional copy of trace output written to a file, so that I can search or share long traces after the session.

Acceptance Criteria

1. Where the trace-enable command names a file, the Simulator REPL shall append to that file every trace printed inline under Requirement 7 (including no-decision traces and reconstructed-dispatch notices), with the same content as shown on the console, and shall not append info trace output.
2. If the named file cannot be opened for writing, the Simulator REPL shall print an error naming the file and the cause, and shall leave the current trace settings unchanged.
3. If the named path refers to the same file (same device and inode) as the simulation output file currently in effect (from --sim-output or set output) or the scheduler's log file if one is in use, the Simulator REPL shall reject the command with an error and leave the current trace settings unchanged.
4. If the operator issues set output naming the same file (same device and inode) as any trace file still open for writing, including one kept open under criterion 5 until its scheduling session ends, the Simulator REPL shall reject the command with an error and leave the simulation output file unchanged.
5. When tracing is disabled or the trace settings are replaced, the Simulator REPL shall keep writing traces of decisions captured under the previous settings to the previous file, and shall close that file once the scheduling session in progress when the command was issued has ended, or immediately if no scheduling session was in progress.
6. When the operator quits the REPL, REPL input ends, or the replay completes, the Simulator REPL shall finish writing every trace already printed inline to the trace file and close it before the simulator exits.
7. The Simulator REPL shall not write trace content to the scheduler's regular log as a consequence of tracing.

Requirement 9: Trace Size and Memory Bounds

Objective: As an operator tracing large jobs on large clusters or all jobs of a long replay, I want trace size and memory bounded, so that one decision cannot flood the console and tracing cannot exhaust simulator memory.

Acceptance Criteria

1. The Simulator REPL shall limit the requirement, cluster-wide check, plan and host sections of each decision's trace to a maximum number of lines, which the operator can set in the trace-enable command to a whole number from 50 to 100000, with a default of 1000 lines when none is given.
2. When a decision's requirement, cluster-wide check, plan and host sections would exceed the size bound, the Simulator REPL shall stop capturing lines for them and show a truncation marker stating how many lines were omitted.
3. When a trace is truncated, the Simulator REPL shall still show the job selection section and, in a number of lines that does not depend on the number of requirements, plans or hosts, the exact decision-wide numbers of plans tried and plans discarded and the exact decision-wide number of host evaluations for each verdict kind, where a host evaluation is one host considered for one position of one plan and an evaluation with several rejection reasons is counted once under each verdict kind it matched (Requirement 5.10).
4. The Simulator REPL shall always show the decision outcome, listing at most 50 hosts, each with all of its entries, and always listing the entries not tied to a host; when more hosts exist, it shall follow the list with exact counts of the omitted hosts' pend-reason entries per pend-reason code (for a pend) or the total amount of each resource allocated across the omitted hosts, excluding limit charges, which are totalled per limit (for a dispatch), and a marker stating how many hosts were omitted.
5. The Simulator REPL shall bound the total memory used by stored traces to 64 MiB and, when the bound is reached, shall discard the oldest stored traces first, never discarding the most recently captured trace.
6. If the operator issues info trace <jobid> for a job whose stored trace was discarded under the memory bound, the Simulator REPL shall report that the trace was discarded.

Requirement 10: No Interference with Scheduling

Objective: As a developer relying on simulation for faithful reproduction, I want tracing to be purely observational, so that enabling it never changes the behavior I am investigating.

Acceptance Criteria

1. The xce_scheduler simulator shall produce the same sequence of scheduling decisions, allocations and pend reasons for a replay whether tracing is enabled or disabled, at either detail level and for any job selection.
2. The xce_scheduler simulator shall produce the same --sim-verify results, and the same --sim-output recording and REPL snapshot content apart from wall-clock metadata fields, whether tracing is enabled or disabled.
3. The xce_scheduler simulator shall produce the same final accounting and job state for a replay whether tracing is enabled or disabled.
4. While tracing is disabled and no scheduling session that started with tracing enabled is still in progress, the Simulator REPL shall produce output identical to the output it produces without this feature, except for the help listing, the output of the new trace commands and info trace themselves, and collapsed-range boundaries at the first round after the last traced round.
5. The Simulator REPL shall not change the behavior or output of any existing REPL command, except that help lists the new commands, info accepts the trace subcommand (Requirements 7.8-7.11), set output rejects any trace file still open for writing (Requirement 8.4), and, while tracing is enabled or a scheduling session that started with tracing enabled is in progress, step and continue output includes traces and the collapsed-range changes defined in Requirement 7, with the collapsed-range boundary effect allowed by criterion 4.

Requirement 11: Simulator-Only Availability

Objective: As an operator of a production cluster, I want decision tracing to be impossible to enable in the live scheduler, so that production scheduling never pays the cost of trace collection.

Acceptance Criteria

1. The xce_scheduler shall provide no command-line option, configuration setting or administrative request that enables decision tracing in the live (controller-connected) scheduler.
2. While running as the live scheduler, the xce_scheduler shall not collect, store or output decision-trace information, and shall do no trace-related work beyond checking that tracing is disabled.
3. While running a simulation without the interactive REPL, the xce_scheduler shall not collect or output decision-trace information.
4. The xce_scheduler shall make the same scheduling decisions in the live scheduler as it did before this feature.