Rongsong Shen
5ca48206b2
Feature validation round 1: tests for trace off keeping recall (1.5), no logging from REPL trace sources (8.7), reconstructed records in the trace file (8.1) and first in continue mode (7.3), output after trace off matching an untraced run (10.4), and tracing enabled only from the REPL (11.1). design.md now reflects the REPL-side file split, the added API, the dropped file tag and the test files; duplicate trace declarations removed from repl_internal.h. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
884 lines
78 KiB
Markdown
884 lines
78 KiB
Markdown
# Design Document: scheduler-sim-decision-trace
|
||
|
||
## Overview
|
||
|
||
**Purpose**: This feature adds a decision trace to the xce_scheduler simulator REPL. For each traced job it explains why the job was selected, which resource requirements were tried, which cluster-wide checks and placement plans were attempted, why each host was accepted or rejected, and what was finally allocated or why the job pends.
|
||
|
||
**Users**: Scheduler developers and operators who replay recorded sessions with `xce_scheduler --sim <file> -i`. They enable tracing at runtime for all jobs or for selected jobs, read traces inline during `step`/`continue`, recall them with `info trace`, and optionally copy them to a file.
|
||
|
||
**Impact**:
|
||
- The allocator thread gains NULL-gated capture hooks at the decision seams. These seams are election, resource-requirement branching, the allocator runner, the host iterator and the shortfall check.
|
||
- `xce_decision_t` gains a trace attachment and a trace-only op.
|
||
- The simulation driver and REPL gain a trace control, store, renderer and file copy.
|
||
- The live scheduler is unchanged except for one never-taken branch per hook.
|
||
|
||
### Goals
|
||
- Explain every traced dispatch, pend and no-decision selection with the hierarchy and content defined in Requirements 2–6.
|
||
- Keep scheduling bit-for-bit identical with tracing on or off (Requirement 10) and keep the live scheduler free of trace work (Requirement 11).
|
||
- Bound trace size per decision and stored-trace memory (Requirement 9).
|
||
|
||
### Non-Goals
|
||
- Live-mode, headless-sim or recording-file tracing.
|
||
- Preemption decisions (`XCE_DECISION_PREEMPT`) are never traced.
|
||
- Changing pend-reason content, the `--sim-verify` comparison or the `ReplayDecision` format.
|
||
- Fixing defects the trace exposes. Examples are ALTER keeping only the last alternative's `obj->reason`, a stale `obj->reason` after an `accepts()` decline, pend reasons duplicated across parjob plans, and an unreachable `PREASON_MATCH_FAIL`. They are logged as separate bugs.
|
||
- `info limit` and limit-usage display (`scheduler-sim-diagnostics`).
|
||
|
||
## Boundary Commitments
|
||
|
||
### This Spec Owns
|
||
- The trace record model, its arena storage, entry budget and counters (`sched_trace.c`, `sched_trace_scan.c`), and the allocator-side capture helpers that read scheduler state (`sched_trace_capture.c`, `sched_trace_outcome.c`), with their shared internal header `sched_trace_internal.h`.
|
||
- Two gate fields on `xce_sched_core_t`: `trace_ctl` (session settings snapshot) and `trace_rec` (record under construction). It also owns the `SCHED_TRACE` hook macro.
|
||
- The `trace` field on `xce_decision_t`, the `XCE_DECISION_TRACE` op and `make_trace_decision()`.
|
||
- The hook call sites in `sched_election.c`, `scheduler.c`, `alloc_runner.c`, `host_iter.c` and `resource.c`. Each is a single gated statement that never changes control flow or return values.
|
||
- The REPL `trace` command, trace settings lifecycle, trace file copy, the `set output` trace-file conflict check, the trace store, `info trace`, trace rendering, the help "Trace" section, and the continue-mode traced-round rule.
|
||
- Plumbing the scheduler log path into the simulation driver (`sched_sim_args_t.log_path`, `xce_sched_sim_driver_t.log_path`).
|
||
|
||
### Out of Boundary
|
||
- Scheduling, picking, allocation and accounting behaviour, and the content of `xce_pendreason_t` / `job->allocs`.
|
||
- The `jbuf` hand-off (capacity 1, `uint64_t` items) and the picker thread. No hook runs on the picker thread.
|
||
- `--sim-verify` comparison logic (`sim_verify.c`), the recording format (`recording*.c`, `sim_record.c`) and snapshot content.
|
||
- The unsatisfiable-resreq cache. It is live-only; sim never consults it, so a trace never explains a cache hit.
|
||
- Rollup fingerprint semantics for untraced rounds (`scheduler-sim-repl-verbosity`).
|
||
|
||
### Allowed Dependencies
|
||
- Core side (`sched_trace*.c`): libxce containers and types (`xvec`, `xlist`, `zarray`, `xce/types.h`), `include/sched/alloc_iter.h` (for `xce_alloc_plan_t`), `src/include/utils.h` (only `array_jobid`/`array_idx` for selector matching), `include/sched/types.h`, `include/core.h`, and `include/preason.h`. These files perform no stdio, file or network I/O and do not include REPL or sim headers.
|
||
- REPL side (`repl_trace*.c`): `include/sched/trace.h` (read-only use of records), `repl_internal.h`, `repl_fmt.c` helpers, libxce `htab`, and `parse_jobsel`/`array_jobid`/`fmt_jobid` (`utils.h`). It must not call scheduler mutation APIs.
|
||
- Dependency direction: `trace.h` types → `sched_trace.c`/`sched_trace_scan.c` (model) → `sched_trace_capture.c`/`sched_trace_outcome.c` (allocator-side readers) → hook sites. Separately, `trace.h` → `repl_trace_render.c` → `repl_trace_store.c` → `repl_trace.c` → `repl.c`/`repl_info.c`/`repl_params.c`/`repl_rollup.c`/`sim_loop.c` call sites. Core-side files never include REPL-side headers.
|
||
|
||
### Revalidation Triggers
|
||
- Any change to `xce_decision_t` or `xce_decision_op_t` (shared with `scheduler-preemption`).
|
||
- Changes to the allocator runner seams (`alloc_runner.c` plan, position and host loop), `host_iter.c` filters, or `unsatisfy_`'s signature.
|
||
- Changes to the sim drain-loop order (verify → record → print gate → breakpoint) or to the rollup call site.
|
||
- Changes to the limit plugin's vres naming or to `obj->glimits`/`obj->hlimits` ownership.
|
||
- Any change that lets the host iterator's resource-availability pre-filter (`host_iter.c:150-166`) see a non-empty request. Today it is a no-op because the runner empties `alloc_req.host` before every `next()` (`alloc_runner.c:216, 253`), and `sched_check_host_satisfy` returns 1 on an empty request (`wanted->count == 0`, `resource.c:418-420`) before it reaches the `unsatisfy_` hook. The design hooks only the selection and internal-error pre-filter paths; a live resource pre-filter would need its own `host_begin` plus pre-filter SHORTFALL/LIMIT verdicts.
|
||
|
||
## Architecture
|
||
|
||
### Existing Architecture Analysis
|
||
- The core (`core*.c`) is a passive state machine driven by two drivers. The picker and allocator threads communicate through `jbuf` (capacity 1). Decisions flow through the `core->decisions` cbuf to the driver's drain loop. Live (`driver.c`) and sim (`sim_loop.c`) run the identical election, allocation and accounting chain (`scheduler-driver-equivalence.md`).
|
||
- The precedent for driver-specific behaviour in the core is NULL-gated core fields set by one driver only (`unsatisfiable_lookup_cb`, `cache_clear_cb`). This design follows it.
|
||
- Files already over the 800-line cap are `scheduler.c` (2002), `sim_loop.c` (1819), `repl_info.c` (1893), `repl_break.c` (980) and `driver.c` (1104). They receive only call sites. All logic lives in new files.
|
||
|
||
### Architecture Pattern & Boundary Map
|
||
|
||
```mermaid
|
||
graph TB
|
||
subgraph AllocatorThread
|
||
Election[sched_election allocator loop]
|
||
Branch[scheduler resreq branch]
|
||
Runner[alloc_runner plans and positions]
|
||
HostIter[host_iter prefilter]
|
||
Satisfy[resource unsatisfy]
|
||
Capture[sched_trace_capture]
|
||
Model[sched_trace record model]
|
||
end
|
||
subgraph SimThread
|
||
Drain[sim_loop drain loop]
|
||
ReplTrace[repl_trace control and file]
|
||
Store[repl_trace_store and info trace]
|
||
Render[repl_trace_render]
|
||
Rollup[repl_rollup]
|
||
end
|
||
Election --> Capture
|
||
Branch --> Capture
|
||
Runner --> Capture
|
||
HostIter --> Capture
|
||
Satisfy --> Capture
|
||
Capture --> Model
|
||
Election -->|decision with trace| Decisions[core decisions cbuf]
|
||
Decisions --> Drain
|
||
Drain --> ReplTrace
|
||
ReplTrace --> Render
|
||
ReplTrace --> Store
|
||
Store --> Render
|
||
ReplTrace --> Rollup
|
||
ReplTrace -->|session settings| CoreCtl[core trace_ctl]
|
||
CoreCtl --> Election
|
||
```
|
||
|
||
**Architecture Integration**:
|
||
- **Selected pattern**: per-decision structured record built on the allocator thread, transferred with the decision, then rendered and stored by the sim REPL.
|
||
- **Boundaries**:
|
||
- The core side builds records and never renders.
|
||
- The REPL side renders, stores and writes. Inline emission reads only the record, never live scheduler state. The single exception is `info trace` (7.9, 7.11), which reads `job->state` at the prompt. That is the same read-only access, with the same timing, as the existing `info job` at a decision breakpoint; it adds no new race class.
|
||
- **Existing patterns preserved**:
|
||
- The NULL-gated core field, the `sim->interactive` inline gates in `repl.h`, and drain-loop ownership of decisions.
|
||
- `fprintf(stdout, ...)` for all REPL output (`test_no_bare_printf.c`).
|
||
- `jbuf` is unchanged.
|
||
- **New components rationale**: the record model, capture, control, store and renderer each have a distinct owner and thread (see Components).
|
||
- **Steering compliance**:
|
||
- The core does no I/O.
|
||
- Driver equivalence: the trace only reads state, and live never enables it.
|
||
- TDD (`testing.md`) and the 800-line cap.
|
||
|
||
### Technology Stack
|
||
|
||
| Layer | Choice / Version | Role in Feature | Notes |
|
||
|-------|------------------|-----------------|-------|
|
||
| Scheduler core | C (gnu17), existing libxce | Record model, capture hooks | No new libraries |
|
||
| Simulator REPL | C, libedit `readline()` (existing) | `trace` / `info trace` commands, rendering, file copy | Raw-line re-parse for long arguments |
|
||
| Build / test | Meson, existing Unity-style testsuite | New sources added to `scheduler_src`, isolation libs and test include lists | — |
|
||
|
||
## File Structure Plan
|
||
|
||
### Directory Structure
|
||
```
|
||
src/scheduler/
|
||
├── include/sched/
|
||
│ └── trace.h # Record, entry, settings types; SCHED_TRACE macro; model + capture API
|
||
├── src/
|
||
│ ├── sched_trace.c # Record model: arena, entry append, budget, requirement/plan/check
|
||
│ │ # entries, jobsel predicate, record free/size. No I/O, no
|
||
│ │ # scheduler-state reads.
|
||
│ ├── sched_trace_scan.c # Record model, part 2: positions, host verdicts and rows, counters,
|
||
│ │ # verdict tallies, rollback, commit, search end. No I/O.
|
||
│ ├── sched_trace_capture.c # Allocator-side capture helpers that read scheduler structures
|
||
│ │ # (job/queue keys, glimits, resource flags, vres lookup, check
|
||
│ │ # rows, attempt reasons). No I/O.
|
||
│ ├── sched_trace_outcome.c # Capture, part 2: alloc/pend outcome copy with 50-host cap,
|
||
│ │ # reconstructed-dispatch record. No I/O.
|
||
│ ├── sched_trace_internal.h # Helpers shared by the sched_trace*.c files (task 2.4 split to
|
||
│ │ # stay under the 800-line cap); not installed.
|
||
│ ├── repl_trace.c # `trace on|off|status` parsing (raw line), refcounted settings
|
||
│ │ # (next/cur references), file open/close, dev/inode conflict
|
||
│ │ # checks, shutdown
|
||
│ ├── repl_trace_emit.c # Session begin/end hooks, inline emission (live, trace-only and
|
||
│ │ # reconstructed records), file write-error reporting
|
||
│ ├── repl_trace_store.c # jobid -> latest record index, capture-order eviction at 64 MiB,
|
||
│ │ # discarded marks, `info trace` (job, array element, array)
|
||
│ ├── repl_trace_render.c # Record -> lines on one or two FILE* sinks; units mode; jobid fmt;
|
||
│ │ # hierarchy/indent; summary vs full host section
|
||
│ ├── repl_trace_render_outcome.c # Renderer, part 2: outcome section (50-host cap), pend reasons,
|
||
│ │ # no-decision and reconstructed endings (800-line split)
|
||
│ └── repl_trace_render.h # Helpers shared by the two renderer files; not installed
|
||
└── testsuite/
|
||
├── test_trace_model.c # sched_trace.c + sched_trace_scan.c unit tests
|
||
├── test_trace_capture.c # capture helpers against fixture core/schedobj/workers
|
||
├── test_decision_trace.c # XCE_DECISION_TRACE, make_trace_decision, decision->trace transport
|
||
├── test_trace_hooks_gate.c # SCHED_TRACE no-evaluation proof; live/non-interactive never trace
|
||
├── test_trace_hooks_alloc_loop.c # sched_allocator hooks: selection, limits, no-decision records
|
||
├── test_trace_hooks_resreq.c # requirement-branch hooks (single, compound, alternatives)
|
||
├── test_trace_host_hooks.c # host pre-filter, host scan and satisfaction-check row hooks
|
||
├── test_trace_hooks_runner.c # alloc_runner hooks: plans, positions, hosts, rollback, commit
|
||
├── test_trace_render.c # renderer output per section, units modes, truncation, outcome cap
|
||
├── test_trace_store.c # store replace/evict/discarded marks, info trace variants
|
||
├── test_repl_trace_baseline.c # pre-feature REPL transcript baseline (tracing off, 10.4)
|
||
├── test_repl_trace_cmd.c # trace command grammar, errors, status, lifecycle
|
||
├── test_repl_trace_file.c # trace file rules: open errors, conflicts, replace, shutdown
|
||
├── test_repl_trace_emit.c # session hooks, emission order per mode, file copy, write errors
|
||
├── test_repl_trace_wiring.c # repl.c/repl_info.c/repl_params.c routing, interactive-only gates
|
||
├── test_sim_trace_driver.c # sim driver wiring: trace-only path, breakpoints, reconstructed
|
||
│ # records, trace off retention and output (1.5, 10.4), 11.3
|
||
├── test_repl_trace_e2e.c # step/continue/breakpoint integration over replay fixtures
|
||
│ # (+ test_repl_trace_e2e_fixture.c, test_repl_trace_e2e.h)
|
||
├── test_trace_error_paths.c # fault injection into the real allocator thread
|
||
├── test_trace_bounds.c # line bound and 64 MiB store bound under large traces
|
||
└── test_trace_equivalence.c # trace on vs off: decisions, --sim-verify, --sim-output, accounting
|
||
# (+ test_trace_equivalence_cmp.c, test_trace_equivalence.h)
|
||
```
|
||
|
||
### Modified Files
|
||
- `src/scheduler/include/core.h`: add `xce_sched_trace_ctl_t* trace_ctl` and `xce_sched_trace_rec_t* trace_rec` to `struct xce_sched_core_s`, each documented as allocator-thread-read and sim-window-written, and NULL in live.
|
||
- `src/scheduler/include/decision.h`, `src/decision.c`:
|
||
- append `XCE_DECISION_TRACE` after `XCE_DECISION_PREEMPT`;
|
||
- add `struct xce_sched_trace_rec_s* trace;` after the union;
|
||
- add `make_trace_decision(rec)`.
|
||
- `src/scheduler/src/sched_election.c`: hooks in `sched_allocator`:
|
||
- per-job prologue: position counter, traced predicate, record begin with selection facts;
|
||
- after `plugin_job_election`: cluster-wide limits;
|
||
- each no-decision path: no-decision record pushed as a trace decision;
|
||
- after the decision is built: outcome capture and attach;
|
||
- session-end marker: reset `trace_rec`.
|
||
- `src/scheduler/src/scheduler.c`: hooks in `sched_make_allocation` (PRIMARY and top-level COMPOUND containers), `sched_make_compound_alloc_` (member push/begin/end/pop, merge failure, attempt reason snapshot) and the ALTER loop (alternative push/begin/end/pop, skipped, result).
|
||
- `src/scheduler/src/alloc_runner.c`: hooks for:
|
||
- the accepts decline;
|
||
- cluster-wide check begin/end;
|
||
- `plan_init` failure;
|
||
- each generated plan (discarded, not tried on reserve failure, tried);
|
||
- each position (begin, end: filled, unfilled or scan error);
|
||
- each candidate (claimed skip, per-host result `>0/0/<0`);
|
||
- rollback;
|
||
- finalize failure;
|
||
- commit.
|
||
- `src/scheduler/src/host_iter.c`: hooks for selection mismatch, pre-filter internal error (`sched_ctx_get_uncomit_host` failure) and scan stop on `iter_next < 0`.
|
||
- `src/scheduler/src/resource.c`: one hook at the top of `unsatisfy_` passing all rows.
|
||
- `src/scheduler/src/sim_loop.c`:
|
||
- `xce_repl_trace_session_begin(sim)` right after `xce_core_begin_session`;
|
||
- reconstructed-dispatch trace in `sim_reconcile_unaccounted_jobs`' report block;
|
||
- drain loop: `XCE_DECISION_TRACE` handled first, then `xce_repl_check_trace(sim, d)` after `sim_print_decision_gate`;
|
||
- `xce_repl_trace_session_end(sim)` after the drain loop.
|
||
- `src/scheduler/include/repl.h`: `extern` declarations of the forwarded entry points (`repl_trace_session_begin/end`, `repl_trace_on_decision`, `repl_trace_on_trace_only`, `repl_trace_on_reconstructed`), following the existing `xce_repl_on_*` pattern so `repl.h` stays self-contained; this is their only declaration site. It also adds inline gates `xce_repl_trace_session_begin/end`, `xce_repl_check_trace`, `xce_repl_check_trace_only` and `xce_repl_trace_reconstructed`, all forwarding only when `sim->interactive`.
|
||
- `src/scheduler/src/repl_internal.h`: `xce_repl_trace_state_t trace` member in `xce_sched_repl_state_t`; declarations for the other `repl_trace*` entry points (command, store, renderer, settings file, conflicts, shutdown) and moved formatting helpers.
|
||
- `src/scheduler/src/repl.c`: dispatch `trace` (passing the raw line); help gains a trailing "Trace" section and an `info trace` line; `repl_cmd_quit` and EOF call `repl_trace_shutdown`.
|
||
- `src/scheduler/src/repl_info.c`: `info trace` route (one branch); usage string updated; units helpers moved out.
|
||
- `src/scheduler/src/repl_fmt.c`: receives `translate_units_mode`, `repl_format_numeric_resource_value`, `repl_format_consumable_value_and_used` and `repl_current_units_mode` (behaviour unchanged, now non-static).
|
||
- `src/scheduler/src/repl_params.c`: `set output` calls `repl_trace_file_conflicts(sim, path)` before switching.
|
||
- `src/scheduler/src/repl_rollup.c`: the traced-round rule in `repl_round_rollup_process`, placed after `repl_query_pend_jobids` succeeds (see ReplIntegration).
|
||
- `src/scheduler/src/driver.c`: the drain loop frees and skips `XCE_DECISION_TRACE` defensively; it is never produced in live.
|
||
- `src/scheduler/include/sim.h`, `src/sim.c`, `src/main.c`: `log_path` plumbing. `sim.h` also declares `sim_print_decision`, which `sim_loop.c` makes non-static (format unchanged) so continue-mode trace emission reuses the exact echo line.
|
||
- `src/scheduler/meson.build`:
|
||
- add the new sources to the scheduler sources;
|
||
- add the trace sources (`sched_trace.c`, `sched_trace_scan.c`, `sched_trace_capture.c`, `sched_trace_outcome.c` and `sched_trace_internal.h`) to `test_core_isolation.c`'s source-text scan list (forbidden headers and symbols, plus a new no-stdio check for these files only). They are not added to `sched_core_isolation_lib`, whose `nm` check covers the four core state-machine files; the capture file references scheduler symbols outside that archive's scope;
|
||
- add `repl_trace*.c` to `sched_sim_isolation_lib`.
|
||
- `src/scheduler/testsuite/meson.build`, `test_core_isolation.c`, `test_sim_isolation.c`, plus every white-box test that `#include`s a hooked or calling source: register the new tests, scan the new files, and add `#include` lines for `sched_trace*.c` / `repl_trace*.c`. `SCHED_TRACE` bodies are compiled even when they never run, so a test that includes a hooked file must also include (or link) `sched_trace.c`, `sched_trace_scan.c`, `sched_trace_capture.c` and `sched_trace_outcome.c`.
|
||
- **Hooked sources**: `resource.c`, `host_iter.c`, `alloc_runner.c`, `scheduler.c`, `sched_election.c` and `driver.c`. These are included by `test_alloc_runner*.c`, `test_host_iter.c`, `test_alloc_equivalence_*.c`, `test_parjob_combinatorial.c`, `test_core_events.c`, `test.c` and others (about 20).
|
||
- **Calling sources**: `repl.c` and `sim_loop.c`, included by about 26 tests.
|
||
- The exact list is produced by `grep -l '#include "../src/\(resource\|host_iter\|alloc_runner\|scheduler\|sched_election\|driver\|repl\|sim_loop\).c"' testsuite/*.c` and updated in one dedicated task right after the new files exist.
|
||
- `src/scheduler/CLAUDE.md`: correct the stale line counts and list the new files.
|
||
|
||
## System Flows
|
||
|
||
### Session lifecycle and capture
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant Op as Operator
|
||
participant Repl as repl_trace
|
||
participant Sim as sim_loop
|
||
participant Core as core
|
||
participant Alloc as allocator thread
|
||
Op->>Repl: trace on job 42 level full
|
||
Repl->>Repl: set next settings
|
||
Sim->>Core: begin session
|
||
Sim->>Repl: session begin hook
|
||
Repl->>Core: publish trace_ctl and reset position
|
||
Sim->>Sim: reconcile and report reconstructed dispatches
|
||
Sim->>Core: signal go
|
||
Alloc->>Core: pop jobid and count position
|
||
Alloc->>Alloc: traced so set trace_rec and capture selection
|
||
Alloc->>Alloc: hooks append resreq check plan host entries
|
||
Alloc->>Alloc: capture outcome and attach record to decision
|
||
Alloc->>Core: push decision and clear trace_rec
|
||
Sim->>Core: drain decision
|
||
Sim->>Repl: check trace
|
||
Repl->>Op: echo line and rendered trace and file copy
|
||
Repl->>Repl: store record
|
||
Sim->>Repl: session end hook
|
||
Repl->>Core: clear trace_ctl and release replaced settings
|
||
```
|
||
|
||
Key decisions:
|
||
- Settings take effect only in the session-begin hook (Requirement 1.6). Commands issued during a session change only `next`.
|
||
- `trace_rec` is non-NULL only between the per-job prologue and the decision push of a traced job. The allocator sets and clears it; nothing else touches it.
|
||
- A no-decision selection pushes a decision with op `XCE_DECISION_TRACE` that carries only the record. The drain loop handles it before verify, record, print gate and breakpoint check.
|
||
|
||
### Drain-loop emission order (per decision)
|
||
|
||
```mermaid
|
||
graph TB
|
||
Get[cbuf get decision] --> IsTrace{op is TRACE}
|
||
IsTrace -->|yes| TraceOnly[check trace only then free and continue]
|
||
IsTrace -->|no| Verify[verify]
|
||
Verify --> Record[record add]
|
||
Record --> Gate[print gate]
|
||
Gate --> CheckTrace[check trace]
|
||
CheckTrace --> Break[breakpoint check]
|
||
Break --> Free[free decision]
|
||
```
|
||
|
||
- `xce_repl_check_trace` runs before the breakpoint check. A breakpoint therefore stops after the stopping decision's trace is printed and stored, and before any later trace (Requirement 7.5).
|
||
- In continue mode, the first traced item of a round flushes the active collapsed range, prints the suppressed echo line, then prints the trace. The round is then marked traced (Requirement 7.3).
|
||
|
||
### Trace settings lifecycle
|
||
|
||
```mermaid
|
||
stateDiagram-v2
|
||
[*] --> NextOnly: trace on
|
||
NextOnly --> InSession: session begin
|
||
InSession --> NextOnly: session end and still referenced as next
|
||
InSession --> Released: session end and replaced or disabled
|
||
NextOnly --> Released: replaced or disabled with no session in progress
|
||
Released --> [*]
|
||
```
|
||
|
||
A settings object (selection, level, bound, open file) is referenced by `rs->trace.next` (the settings the next session will use) and, while a session runs, by `rs->trace.cur` (the settings the session started with). Commands only replace `next`. The session-begin hook sets `cur = next`, and the session-end hook clears `cur`. An object is released, and its file closed, when neither reference holds it. This gives 8.5 directly: a replaced object stays open until the in-progress session ends, or is closed immediately when no session is in progress.
|
||
|
||
## Requirements Traceability
|
||
|
||
| Requirement | Summary | Components | Interfaces | Flows |
|
||
|-------------|---------|------------|------------|-------|
|
||
| 1.1, 1.2, 1.3 | Enable all or by jobs (arrays, future ids) | ReplTraceControl, TraceModel | `repl_trace_cmd`, `sched_trace_ctl_wants` | Lifecycle |
|
||
| 1.4, 1.5, 1.6, 1.10 | Replace, disable, next-session application, persistence | ReplTraceControl | `next`/`cur` settings, session hooks | Lifecycle, Session |
|
||
| 1.7 | Status: in effect, next, memory | ReplTraceControl, TraceStore | `repl_trace_cmd` status, `repl_trace_store_bytes` | — |
|
||
| 1.8 | Levels summary/full (default summary) | ReplTraceControl, TraceModel, TraceRenderer | `xce_sched_trace_level_t` | — |
|
||
| 1.9, 1.12 | Argument validation; unbounded jobs, 4095-byte path | ReplTraceControl | raw-line parser | — |
|
||
| 1.11 | Help section and `info trace` listing | ReplIntegration | `repl_print_help` | — |
|
||
| 2.1 | Session number and position | TraceHooks (election), TraceModel | `sched_trace_ctl_t.position`, `.session_no` | Session |
|
||
| 2.2 | Policy and FCFS keys | TraceCapture | `sched_trace_capture_selection` | Session |
|
||
| 2.3 | Cluster-wide limits matched | TraceCapture | `sched_trace_capture_glimits` | Session |
|
||
| 2.4 | No-decision selections | TraceHooks (election), DecisionTransport | `sched_trace_rec_finish`, `XCE_DECISION_TRACE` | Drain order |
|
||
| 2.5 | Reconstructed dispatch notice | SimIntegration, TraceCapture | `sched_trace_rec_reconstructed` | Session |
|
||
| 3.1, 3.2 | Resreq labels, path, tasks, resources, outcome | TraceHooks (scheduler.c), TraceCapture | `sched_trace_resreq_begin/end` | — |
|
||
| 3.3, 3.4, 3.6 | Failure point, chosen alt, not tried, skipped | TraceModel, TraceHooks (scheduler.c) | `resreq_end(not_tried)`, `sched_trace_member_failed`, `sched_trace_alt_result`, `sched_trace_alt_skipped` | — |
|
||
| 3.5 | Per-alternative failure reasons | TraceCapture | attempt reason summary entry | — |
|
||
| 4.1, 4.2 | Cluster-wide rows; limit rejection | TraceHooks (unsatisfy_, runner), TraceCapture | `sched_trace_check_rows` | — |
|
||
| 4.3, 4.4, 4.5, 4.6 | Plans: shape, outcome, failed position, errors, commit | TraceHooks (runner), TraceModel | `sched_trace_plan_*`, `sched_trace_pos_*`, `sched_trace_pos_scan_error`, `sched_trace_commit`, `sched_trace_finalize_failed`, `sched_trace_search_end` | — |
|
||
| 4.7, 4.8, 4.9 | No plan generated or declined; no plans after check fail; discarded | TraceHooks (runner) | `sched_trace_no_plan`, `sched_trace_check_end`, `sched_trace_plan_discarded`, `sched_trace_search_end` | — |
|
||
| 5.1–5.7, 5.10 | Host verdicts, phases, kinds, not performed | TraceHooks (host_iter, runner, unsatisfy_), TraceModel | `sched_trace_host_*`, verdict bitmask | — |
|
||
| 5.8 | Released hosts on rollback | TraceHooks (runner) | `sched_trace_rollback_host` | — |
|
||
| 5.9 | Summary per-position counts and limit names | TraceModel, TraceRenderer | `POS_SUMMARY` entry | — |
|
||
| 6.1, 6.2, 6.3 | Dispatch outcome with separate limit charges | TraceCapture, TraceRenderer | `sched_trace_capture_dispatch` | — |
|
||
| 6.4 | Pend reason set, same content | TraceCapture, TraceRenderer | `sched_trace_capture_pend` | — |
|
||
| 7.1, 7.2 | Step-mode inline placement | SimIntegration, ReplTraceControl | `xce_repl_check_trace`, `..._trace_only` | Drain order |
|
||
| 7.3, 7.4 | Continue-mode echo and no collapse | ReplTraceControl, RollupIntegration | `rs->trace.round_traced` | Drain order |
|
||
| 7.5 | Breakpoint ordering | SimIntegration | hook before `xce_repl_check_decision` | Drain order |
|
||
| 7.6, 7.7 | Hierarchy, units, jobid format | TraceRenderer | `repl_trace_render` | — |
|
||
| 7.8, 7.9, 7.10, 7.11 | `info trace` job, array, replace, none | TraceStore | `repl_cmd_info_trace` | — |
|
||
| 8.1, 8.2 | File copy content; open errors | ReplTraceControl, TraceRenderer | two-sink render | — |
|
||
| 8.3, 8.4 | dev/inode conflicts both directions | ReplTraceControl, ReplIntegration (`set output`) | `repl_trace_file_conflicts` | — |
|
||
| 8.5, 8.6 | Replaced file kept until session end; shutdown flush | ReplTraceControl | `cur`/`next` references, `repl_trace_shutdown` | Lifecycle |
|
||
| 8.7 | Nothing to scheduler log | ReplTraceControl, TraceModel | no tlog in trace code | — |
|
||
| 9.1, 9.2, 9.3 | Line bound, marker, exact counts | TraceModel, TraceRenderer | entry budget, counters | — |
|
||
| 9.4 | Outcome 50-host cap and aggregates | TraceCapture, TraceRenderer | capped outcome copy | — |
|
||
| 9.5, 9.6 | 64 MiB store, eviction, discarded report | TraceStore | `repl_trace_store_put`, discarded marks | — |
|
||
| 10.1, 10.2, 10.3 | No scheduling change | All hooks (read-only), DecisionTransport | equivalence tests | — |
|
||
| 10.4, 10.5 | REPL output unchanged when off | ReplTraceControl, RollupIntegration, SimIntegration | gates on `rs->trace` | — |
|
||
| 11.1–11.4 | Sim-only; single branch in live | TraceHooks, DecisionTransport | `trace_ctl`/`trace_rec` NULL in live | — |
|
||
|
||
## Components and Interfaces
|
||
|
||
| Component | Domain / Thread | Intent | Requirements | Key Dependencies | Contracts |
|
||
|-----------|-----------------|--------|--------------|------------------|-----------|
|
||
| TraceModel | Core / allocator (build), sim (read) | Record types, arena, entry budget, counters, jobsel predicate | 1.2, 1.3, 1.8, 3.3, 3.4, 3.6, 4.x, 5.9, 5.10, 9.1–9.3 | libxce containers (P0) | Service, State |
|
||
| TraceCapture | Core / allocator (and sim pre-go window) | Read scheduler state into record entries | 2.2, 2.3, 2.5, 3.1, 3.2, 3.5, 4.1, 4.2, 5.4, 5.5, 6.x, 9.4 | core resmgr, schedobj limits, job allocs (P0) | Service |
|
||
| TraceHooks | Core / allocator | Gated call sites at the decision seams | 2.1, 2.4, 3.x, 4.x, 5.x, 11.2 | TraceModel, TraceCapture (P0) | — |
|
||
| DecisionTransport | Core | Carry records on decisions; trace-only op | 2.4, 7.2, 10.1, 10.2 | `decision.h` consumers (P0) | State |
|
||
| SimIntegration | Sim driver | Session hooks, reconstructed notices, drain-loop hand-off | 1.6, 2.5, 7.1–7.5 | repl.h gates (P0) | — |
|
||
| ReplTraceControl | REPL / sim thread | Command, settings lifecycle, inline emission, file copy | 1.x, 7.1–7.4, 8.x, 10.4, 10.5 | TraceRenderer, TraceStore (P0) | Service, State |
|
||
| TraceStore | REPL / sim thread | Latest record per job; eviction; `info trace` | 1.7, 7.8–7.11, 9.5, 9.6 | libxce htab (P1) | Service, State |
|
||
| TraceRenderer | REPL / sim thread | Record to text lines | 6.x, 7.6, 7.7, 8.1, 9.2–9.4 | repl_fmt helpers (P1) | Service |
|
||
| ReplIntegration | REPL | help, dispatch, `info` route, `set output`, rollup rule | 1.11, 7.3, 7.4, 8.4, 10.5 | existing REPL files (P1) | — |
|
||
|
||
### Core Side
|
||
|
||
#### TraceModel (`include/sched/trace.h`, `src/sched_trace.c`, `src/sched_trace_scan.c`)
|
||
|
||
**Responsibilities & Constraints**
|
||
- Owns `xce_sched_trace_rec_t`: an append-only sequence of typed entries in a per-record arena, plus fixed-size header, selection, counters and outcome blocks. A record is built by exactly one thread at a time and is immutable once attached to a decision.
|
||
- Enforces the entry budget (`line_bound`) over the requirement, cluster-wide check, plan and host sections. When the budget is exhausted it stops appending and counts omitted entries, while counters keep updating (9.2, 9.3).
|
||
- Owns verdict classification, the per-position summary tallies (5.9) and the decision-wide host-evaluation counts per verdict kind (9.3).
|
||
- No I/O and no tlog. It reads only the arguments passed to it (copying names, task counts, requested resources and plan shapes into the arena). It keeps the scope's `obj`/`w` pointers only until the matching scope ends, for TraceCapture's limit lookup.
|
||
|
||
**Contracts**: Service [x] / State [x]
|
||
|
||
##### Service Interface
|
||
```c
|
||
typedef enum { XCE_TRACE_LEVEL_SUMMARY = 0, XCE_TRACE_LEVEL_FULL } xce_sched_trace_level_t;
|
||
|
||
typedef enum { /* Requirement 5.10 verdict kinds, bit flags */
|
||
XCE_TRACE_V_ACCEPTED = 1u << 0,
|
||
XCE_TRACE_V_SELECT = 1u << 1, /* selection mismatch (pre-filter) */
|
||
XCE_TRACE_V_SHORTFALL = 1u << 2, /* resource shortfall (per-host check) */
|
||
XCE_TRACE_V_LIMIT = 1u << 3, /* limit exceeded (per-host check) */
|
||
XCE_TRACE_V_CLAIMED = 1u << 4, /* already claimed by earlier position */
|
||
XCE_TRACE_V_INTERNAL = 1u << 5 /* not evaluated: internal error */
|
||
} xce_sched_trace_verdict_t;
|
||
|
||
typedef enum { /* Requirement 2.4 / 2.5 record kinds */
|
||
XCE_TRACE_REC_DECISION = 0,
|
||
XCE_TRACE_REC_NOT_FOUND,
|
||
XCE_TRACE_REC_NO_SCHEDOBJ,
|
||
XCE_TRACE_REC_INTERNAL_ERROR, /* alloc_rc < 0 or decision build failure */
|
||
XCE_TRACE_REC_PEND_NO_REASON,
|
||
XCE_TRACE_REC_RECONSTRUCTED
|
||
} xce_sched_trace_rec_kind_t;
|
||
|
||
typedef struct xce_sched_trace_sel_s { /* one selector from `trace on job ...` */
|
||
uint64_t key; /* plain jobid, array base id, or element id */
|
||
bool whole_array;/* true: key is an array base, matches every element */
|
||
} xce_sched_trace_sel_t;
|
||
|
||
typedef struct xce_sched_trace_ctl_s { /* immutable during a session except position */
|
||
bool all;
|
||
int nsel;
|
||
xce_sched_trace_sel_t* sel;
|
||
xce_sched_trace_level_t level;
|
||
int line_bound; /* 50..100000 */
|
||
uint64_t session_no; /* rs->session_count of this session */
|
||
const char* policy_name; /* picker name, owned by the ctl */
|
||
int position; /* allocator-owned, reset at session begin */
|
||
} xce_sched_trace_ctl_t;
|
||
|
||
typedef struct xce_sched_trace_rec_s xce_sched_trace_rec_t; /* opaque outside sched_trace*.c / renderer */
|
||
|
||
#define XCE_TRACE_LINE_BOUND_DEFAULT 1000 /* Requirement 9.1 */
|
||
#define XCE_TRACE_LINE_BOUND_MIN 50
|
||
#define XCE_TRACE_LINE_BOUND_MAX 100000
|
||
#define XCE_TRACE_OUTCOME_HOSTS 50 /* Requirement 9.4 */
|
||
|
||
typedef enum {
|
||
XCE_TRACE_PATH_SINGLE, /* PRIMARY job: the single requirement */
|
||
XCE_TRACE_PATH_COMPOUND, /* top-level COMPOUND container */
|
||
XCE_TRACE_PATH_ALT, /* alternative i/n of an ALTER (a compound) */
|
||
XCE_TRACE_PATH_MEMBER /* member j/m of the enclosing compound */
|
||
} xce_sched_trace_path_kind_t;
|
||
typedef struct { /* kept inside the record, not passed by hooks */
|
||
int depth; /* 0..2: container (COMPOUND|ALT) > MEMBER, or SINGLE */
|
||
struct { xce_sched_trace_path_kind_t kind; int i; int n; } lvl[2];
|
||
} sched_trace_path_t;
|
||
|
||
typedef enum { /* Requirement 4.7 reasons */
|
||
XCE_TRACE_NOPLAN_DECLINED, /* allocator accepts() declined */
|
||
XCE_TRACE_NOPLAN_GENERATOR_FAILED, /* plan_init failed */
|
||
XCE_TRACE_NOPLAN_NONE_GENERATED /* generator produced no plan */
|
||
} xce_sched_trace_noplan_t;
|
||
|
||
typedef enum {
|
||
XCE_TRACE_POS_FILLED,
|
||
XCE_TRACE_POS_UNFILLED,
|
||
XCE_TRACE_POS_SCAN_ERROR, /* host scan could not start or stopped (4.4) */
|
||
XCE_TRACE_POS_INTERNAL /* host accepted but claim snapshot failed */
|
||
} xce_sched_trace_pos_end_t;
|
||
|
||
/* gate: evaluates `call` only when a record is being built */
|
||
#define SCHED_TRACE(core, call) \
|
||
do { if ((core)->trace_rec != NULL) { call; } } while (0)
|
||
|
||
bool sched_trace_ctl_wants(const xce_sched_trace_ctl_t* ctl, uint64_t jobid);
|
||
xce_sched_trace_rec_t* sched_trace_rec_new(const xce_sched_trace_ctl_t* ctl, uint64_t jobid,
|
||
xce_sched_trace_rec_kind_t kind);
|
||
void sched_trace_rec_free(xce_sched_trace_rec_t* rec); /* NULL-safe */
|
||
/* finish a record as a no-decision kind (NOT_FOUND, NO_SCHEDOBJ, INTERNAL_ERROR,
|
||
* PEND_NO_REASON) before it is pushed via make_trace_decision (2.4) */
|
||
void sched_trace_rec_finish(xce_sched_trace_rec_t* rec, xce_sched_trace_rec_kind_t kind, int err);
|
||
size_t sched_trace_rec_bytes(const xce_sched_trace_rec_t* rec); /* arena + blocks */
|
||
|
||
/* structural entries (each = one rendered line in the budgeted sections) */
|
||
void sched_trace_path_push(xce_sched_trace_rec_t*, xce_sched_trace_path_kind_t kind, int i, int n);
|
||
void sched_trace_path_pop(xce_sched_trace_rec_t*);
|
||
/* begin/end of the requirement named by the current path; exactly one end per begin */
|
||
void sched_trace_resreq_begin(xce_sched_trace_rec_t*,
|
||
const xce_primary_resreq_t* primary /* NULL for a container */,
|
||
int nmembers /* container: resreq->primaries.count; else 0 */);
|
||
void sched_trace_resreq_end(xce_sched_trace_rec_t*, int rc, int not_tried_after);
|
||
/* non-scope entries */
|
||
void sched_trace_member_failed(xce_sched_trace_rec_t*, int j, int m, int err); /* merge failure after member j of m */
|
||
void sched_trace_alt_result(xce_sched_trace_rec_t*, int chosen_i /* 0 = none */, int n);
|
||
void sched_trace_alt_skipped(xce_sched_trace_rec_t*, int i, int n);
|
||
void sched_trace_check_begin(xce_sched_trace_rec_t*, const xce_sched_obj_t* obj); /* cluster-wide scope */
|
||
void sched_trace_check_end(xce_sched_trace_rec_t*, int rc, bool plans_follow);
|
||
void sched_trace_no_plan(xce_sched_trace_rec_t*, xce_sched_trace_noplan_t why);
|
||
void sched_trace_plan_begin(xce_sched_trace_rec_t*, const xce_alloc_plan_t* plan);
|
||
void sched_trace_plan_discarded(xce_sched_trace_rec_t*, const xce_alloc_plan_t* plan);
|
||
void sched_trace_plan_not_tried(xce_sched_trace_rec_t*, const xce_alloc_plan_t* plan, int err);
|
||
void sched_trace_plan_end(xce_sched_trace_rec_t*, int rc, int failed_pos);
|
||
void sched_trace_pos_begin(xce_sched_trace_rec_t*, int pos, int slots);
|
||
void sched_trace_pos_end(xce_sched_trace_rec_t*, xce_sched_trace_pos_end_t how, const char* host);
|
||
void sched_trace_host_begin(xce_sched_trace_rec_t*, const xce_sched_obj_t* obj,
|
||
const xce_sched_worker_t* w); /* per-host scope */
|
||
void sched_trace_pos_scan_error(xce_sched_trace_rec_t*, int err); /* host scan stopped */
|
||
void sched_trace_host_verdict(xce_sched_trace_rec_t*, unsigned verdicts, int err);
|
||
/* added in implementation (Implementation Notes 1.5, 2.2): */
|
||
void sched_trace_row(xce_sched_trace_rec_t*, const sched_trace_row_t* row); /* CHECK_ROW or held HOST_ROW */
|
||
void sched_trace_host_limit_hit(xce_sched_trace_rec_t*, const char* limit, const char* scope); /* summary tally */
|
||
void sched_trace_rollback_host(xce_sched_trace_rec_t*, const char* host); /* one call per released claim */
|
||
void sched_trace_commit(xce_sched_trace_rec_t*); /* committed plan = the last plan_begin */
|
||
/* end of sched_alloc_search_: one budgeted line; also updates decision-wide counters */
|
||
void sched_trace_search_end(xce_sched_trace_rec_t*, int rc); /* counts come from the model (see below) */
|
||
void sched_trace_finalize_failed(xce_sched_trace_rec_t*, int err);
|
||
```
|
||
- **Preconditions**: `rec != NULL` for every call (guaranteed by `SCHED_TRACE`). Calls nest in the order the allocator executes them. Every `path_push` has exactly one `path_pop`, and every `*_begin` has exactly one `*_end`, on every return path at the call site. Outcome corrections that occur after an `*_end` (a merge failure, an alternative's selection) are separate non-scope entries, never a second `*_end`.
|
||
- **Postconditions**:
|
||
- Entries beyond `line_bound` are not stored; `omitted` is incremented by the number of lines they would have produced.
|
||
- In `XCE_TRACE_LEVEL_SUMMARY`, host entries are not stored. Instead `pos_end` stores one position-summary entry: accepted host, count per verdict kind, and (limit name, scope, count) tallies.
|
||
- Counters (`plans_tried`, `plans_discarded`, `host_evals[kind]`) are exact regardless of truncation.
|
||
- **Invariants**: a record never references scheduler-owned memory. All strings and numbers are copied into its arena.
|
||
|
||
##### State Management
|
||
- **Record states**: building (allocator) → attached (on decision) → delivered (REPL) → stored or freed.
|
||
- **Concurrency**: no locks. Ownership moves with the decision through the `core->decisions` cbuf.
|
||
|
||
**Implementation Notes**
|
||
- Integration: the record keeps the current `sched_trace_path_t`. Callers push the container level (SINGLE, COMPOUND 1/1, or ALT i/n) before calling into the allocation chain, and `sched_make_compound_alloc_` pushes MEMBER j/m per member. Each `resreq_begin` stores the full current path, so the renderer prints labels such as `alternative 2/3 > member 1/2` (3.1) without the compound function needing to know whether it serves an ALTER.
|
||
- Validation: `test_trace_model.c` covers budget edges (bound-1, bound, bound+1), summary tallies with multi-kind hosts, counters under truncation, and `ctl_wants` for plain, element and array selectors.
|
||
- Risk: arena growth uses `malloc` on the allocator thread. This happens only for traced jobs, never in live.
|
||
|
||
#### TraceCapture (`src/sched_trace_capture.c`, `src/sched_trace_outcome.c`)
|
||
|
||
**Responsibilities & Constraints**
|
||
- Reads scheduler state on the thread that owns it at the moment of capture, and writes entries through TraceModel:
|
||
- `sched_trace_capture_selection(rec, core, job)`: policy name, queue name and priority, job priority, `queue_time` (2.2).
|
||
- `sched_trace_capture_glimits(rec, obj)`: each `xce_vres_t` in `obj->glimits` as (limit name, scope instance) (2.3).
|
||
- `sched_trace_check_rows(rec, core, nres, idxs, total, inuse, uncomitted, wanted)`: called from `unsatisfy_`.
|
||
- For each row it resolves `xce_resource_t` through `vget(core->resmgr, idx)`.
|
||
- A row whose flags include `XCE_RESF_LIMIT` becomes a limit row. Its limit name comes from the matching `xce_vres_t` in the current scope's limit list: `obj->glimits` for the cluster-wide scope, `obj->hlimits[w->idx]` for a host scope. The scope instance is the vres name with its `limit=<name>` prefix removed, rendered as `u=..,q=..,p=..`. In a host scope it is followed by `h=<w->name>`, because host-limit vres names do not encode the host. When no scope pointer exists (reconstructed records, Requirement 2.5, captured on the sim thread), the limit name and scope are parsed from the `XCE_RESF_LIMIT` resource's own name (`limit=<name>/...`). The value, usage, reserved and requested amounts are `total`, `inuse`, `uncomitted` and `wanted`.
|
||
- Every other row is a resource row.
|
||
- In the cluster-wide scope every row is kept (4.1). In a host scope only unmet rows are kept, and they set `SHORTFALL` and/or `LIMIT` on the host's verdict (5.4, 5.5).
|
||
- `sched_trace_capture_attempt_reasons(rec, reason)`: per-code counts and the hosts/resources named, from a compound attempt's `ctx.reason`. It is captured before `scheduler.c` replaces `obj->reason` and renders as the failure-reasons line of a failed member or alternative (3.5).
|
||
- `sched_trace_capture_dispatch(rec, core, obj, alloc)`: per-host allocation rows and shared rows, with limit charges separated by `XCE_RESF_LIMIT` (6.1–6.3).
|
||
- Allocation rows carry only a resource name (`xce_alloc_t.resname`). Each row is classified as a limit charge when its resource, looked up by name in `core->resmgr`, has `XCE_RESF_LIMIT`; when the name is not registered in this replay, the `limit=` name prefix decides.
|
||
- A limit charge is resolved to its `xce_vres_t` by matching `resname` against `vres->vres->name`. Shared charges are searched in `obj->glimits`. Host charges are searched in `obj->hlimits[w->idx]`, where `w` is found by the allocation's host name.
|
||
- The vres supplies the limit name, scope and limited resource name. The first 50 hosts are stored in detail. The rest are aggregated into per-resource totals (excluding limits) and per-limit totals, plus an omitted-host count (9.4).
|
||
- `sched_trace_capture_pend(rec, reason)`: a deep copy of the decision's `xce_pendreason_t`. `main_reason` and host-less entries are always kept. Host entries of the first 50 distinct hosts are kept, and the rest are counted per pend-reason code with an omitted-host count (6.4, 9.4).
|
||
- `sched_trace_rec_reconstructed(ctl, core, jobid, obj, allocs)`: builds a `RECONSTRUCTED` record with a dispatch outcome (2.5), resolving limit charges the same way.
|
||
- A job never evaluated by this replay has empty `glimits`/`hlimits`, or a NULL `obj`. A limit charge with no matching vres is still listed as a limit charge (by its `XCE_RESF_LIMIT` flag), with its name and scope parsed from the vres resource name.
|
||
- Its amount is then shown as a raw number, because the limited resource is unknown. This is the only case where a limit amount is not unit-formatted.
|
||
- **Constraint**: inputs are read-only. No function here calls a scheduler API that mutates state (Requirement 10).
|
||
|
||
**Contracts**: Service [x]
|
||
|
||
```c
|
||
/* declared in include/sched/trace.h; implemented in sched_trace_capture.c
|
||
* (selection, cluster-wide limits, check rows, attempt reasons) and
|
||
* sched_trace_outcome.c (dispatch, pend, reconstructed record) */
|
||
void sched_trace_capture_selection(xce_sched_trace_rec_t*, const xce_sched_core_t* core,
|
||
const xce_sched_job_t* job);
|
||
void sched_trace_capture_glimits(xce_sched_trace_rec_t*, const xce_sched_obj_t* obj);
|
||
void sched_trace_check_rows(xce_sched_trace_rec_t*, xce_sched_core_t* core, int nres,
|
||
const int* idxs, const float* total, const float* inuse,
|
||
const xvec_t* uncomitted, const float* wanted);
|
||
void sched_trace_capture_attempt_reasons(xce_sched_trace_rec_t*, const xce_pendreason_t* reason);
|
||
void sched_trace_capture_dispatch(xce_sched_trace_rec_t*, xce_sched_core_t* core,
|
||
const xce_sched_obj_t* obj, const xce_job_alloc_t* alloc);
|
||
void sched_trace_capture_pend(xce_sched_trace_rec_t*, const xce_pendreason_t* reason);
|
||
xce_sched_trace_rec_t* sched_trace_rec_reconstructed(const xce_sched_trace_ctl_t* ctl,
|
||
xce_sched_core_t* core, uint64_t jobid,
|
||
const xce_sched_obj_t* obj /* job->schedobj, nullable */,
|
||
const xce_job_alloc_t* allocs);
|
||
```
|
||
`core` is non-const only because `vget(core->resmgr, ...)` takes a non-const container; the functions never write to it.
|
||
|
||
**Implementation Notes**
|
||
- Integration: the current scope's schedobj and worker are remembered by the record when `check_begin` / `host_begin` run (the runner and the host iterator pass `ctx->schedobj` and the candidate worker). `unsatisfy_` therefore needs only `core`.
|
||
- Per-host verdict when `sched_try_alloc_on_host_` returns 0: it is SHORTFALL and/or LIMIT, from the unmet rows recorded during the call. If no row was recorded, the 0 came from the function's own selection branch (`PREASON_MATCH_FAIL`), so the verdict is SELECT in the per-host phase.
|
||
- Validation: `test_trace_capture.c` uses a fixture core with a limit vres on a shared and a host resource to check:
|
||
- limit rows are never reported as shortfalls;
|
||
- scope strings are correct;
|
||
- dispatch limit charges are listed separately;
|
||
- the 51-host aggregation;
|
||
- the pend copy equals the source under `job_preason_eq` when there are 50 or fewer hosts.
|
||
- Risk: the vres lookup is linear in the number of matched limits per job. That number is small (configured limits), and it is paid only for traced jobs.
|
||
|
||
#### TraceHooks (call sites)
|
||
|
||
Each hook is one `SCHED_TRACE(core, ...)` statement placed after the value it reports has been computed. It does not change control flow, return values or arguments to existing calls.
|
||
|
||
| Seam | File:anchor | Hook(s) | Requirements |
|
||
|------|-------------|---------|--------------|
|
||
| Allocator per-job prologue | `sched_election.c` after `cbuf_get` (jobid ≠ 0, not RECONCILE_WAKE) | `if (core->trace_ctl) { ctl->position++; if (wants) core->trace_rec = rec_new(...) }` (the only non-macro hook; one branch in live) | 2.1, 11.2 |
|
||
| Job not found / no schedobj | `sched_election.c:226-241` | `NOT_FOUND`: no selection capture (`job == NULL`). `NO_SCHEDOBJ`: `capture_selection` without limits. Then `rec_finish`, push the trace decision and clear `trace_rec` | 2.4 |
|
||
| After election | after `plugin_job_election` | `capture_selection`, `capture_glimits` | 2.2, 2.3 |
|
||
| Alloc error / pend without reason / decision build failure | `sched_election.c:250-254, 265-269, 286` | `rec_finish(INTERNAL_ERROR / PEND_NO_REASON, err)`, push the trace decision, and clear `trace_rec` before the `continue`. Every path that hands a record away clears `trace_rec`, so no later job in the session can append to a record the REPL owns. | 2.4 |
|
||
| Decision built | before `sched_commit_decision` | `capture_dispatch(core, job->schedobj, alloc)` / `capture_pend`; `d->trace = rec`; clear `trace_rec` | 6.x, 9.4 |
|
||
| Session-end marker | jobid 0 | `core->trace_rec = NULL` (defensive) | — |
|
||
| PRIMARY job | `scheduler.c` `sched_make_allocation`, PRIMARY case | `path_push(SINGLE,1,1)`; `resreq_begin(primary, 0)`; call; `resreq_end(rc, 0)`; `path_pop` | 3.1, 3.2 |
|
||
| Top-level COMPOUND | `sched_make_allocation`, COMPOUND case, around the `sched_make_compound_allocation` call | `path_push(COMPOUND,1,1)`; `resreq_begin(NULL, resreq->primaries.count)`; call; `resreq_end(rc, 0)`; `path_pop` | 3.1, 3.2 |
|
||
| ALTER loop | `scheduler.c:409-436`, per alternative i of n | NULL resreq → `alt_skipped(i,n)`. Otherwise `path_push(ALT,i,n)`; `resreq_begin(NULL, alt->resreq->primaries.count)`; call `sched_make_compound_alloc_`; `resreq_end(rc, 0)`; `path_pop`. On the first success, `alt_result(i,n)` renders "alternative i/n chosen; alternatives i+1..n not tried" (3.4). If the loop ends with none chosen, `alt_result(0,n)` renders "all alternatives failed" (3.5). | 3.4–3.6 |
|
||
| Compound members | `sched_make_compound_alloc_` per member j of m | `path_push(MEMBER,j,m)`; `resreq_begin(primary, 0)`; call; `resreq_end(rc, rc<=0 ? m-j : 0)`; `path_pop`. Here `m-j` is the number of members after j, with j 1-based. If `merge_job_alloc` then fails (`scheduler.c:339-401`), `member_failed(j, m, mrc)` renders "member j/m: merge failed (internal error); members j+1..m not tried" (3.3). At the end of a failed attempt, `capture_attempt_reasons(ctx.reason)` records the failure reasons before `obj->reason` is replaced (3.5). | 3.1–3.3, 3.5 |
|
||
| Runner entry | `alloc_runner.c` accepts decline | `no_plan(DECLINED)` | 4.7 |
|
||
| Cluster-wide check | around `sched_alloc_prepare_shared_` | `check_begin` / `check_end(rc, plans_follow=rc>0)` → on failure renders "no plans tried" | 4.1, 4.2, 4.8 |
|
||
| Generator start failure | `plan_init != 0` | `no_plan(GENERATOR_FAILED)` | 4.7 |
|
||
| Search end | end of `sched_alloc_search_` | `search_end(rc)`. The model counts generated, tried and discarded plans per requirement scope from the `plan_begin` / `plan_discarded` / `plan_not_tried` calls it already receives, so the runner keeps no extra bookkeeping outside the gate (11.2). It always renders as one line, `plans: g generated, t tried, d discarded, x not tried`, extended on the same line by at most one statement:
|
||
- `; no plan generated` (`NONE_GENERATED`, 4.7) when g = 0;
|
||
- `; no plan tried: all generated plans discarded` (4.9) when g > 0 and d = g;
|
||
- `; no plan could be placed` (4.6) when rc = 0, t > 0 and no plan ended in an internal error.
|
||
|
||
When rc > 0 or rc < 0 there is no statement; the commit or internal-error entries already say what happened. A declined requirement and a generator that cannot start end the runner before the search end: their `no_plan` entry is the requirement's "no plan generated" line and bumps that counter itself, once; a search end after a `no_plan` entry in the same requirement makes no statement and counts nothing. Like every other entry it counts against the line budget. It also bumps the matching decision-wide counter (requirements with no plan generated / all discarded / none placed), under the same conditions, which appear in the fixed-size totals block after truncation (9.3). | 4.6, 4.7, 4.9 |
|
||
| Plan loop | malformed / reserve failure / try / result | `plan_discarded`; `plan_not_tried(err)`; `plan_begin`; `plan_end(rc, failed_pos)` | 4.3, 4.4, 4.6, 4.9 |
|
||
| Position | `sched_claim_bucket_` | `pos_begin`; iterator init failure → `pos_end(SCAN_ERROR)`; snapshot failure after acceptance (`alloc_runner.c:237-243`) → `pos_end(INTERNAL)`; else `pos_end(FILLED/UNFILLED/SCAN_ERROR, host)` | 4.4, 5.9 |
|
||
| Candidate | claimed skip / per-host result | `host_begin` + `host_verdict(CLAIMED)`; `host_begin`, call, `host_verdict(rc>0 ? ACCEPTED : rc==0 ? from rows : INTERNAL, err)` | 5.1–5.7 |
|
||
| Pre-filter | `host_iter.c:136-156` | `host_begin` + `host_verdict(SELECT)`; uncomit error → `host_verdict(INTERNAL)` | 5.2, 5.3, 5.7 |
|
||
| Scan stop | `host_iter.c:124-127` (`rc < 0`) | `pos_scan_error(rc)`; the runner's following `pos_end(UNFILLED)` is recorded as SCAN_ERROR | 4.4 |
|
||
| Rollback | in `sched_try_plan_` after a failed position, only when `n > 0` (not inside `sched_rollback_claims_`, which is also called with `n == 0` and from finalize's partial rollbacks) | `SCHED_TRACE(core, for (k = 0; k < n; k++) sched_trace_rollback_host(core->trace_rec, claims[k].w->name))`, a single gated statement. The model merges consecutive calls into one `ROLLBACK` entry, "released hosts: n01, n02" (one line). | 5.8 |
|
||
| Finalize | `sched_finalize_job_alloc_` result | `commit` or `finalize_failed(err)` | 4.5 |
|
||
| Shortfall rows | `resource.c` top of `unsatisfy_` | `check_rows(...)` | 4.1, 4.2, 5.4, 5.5 |
|
||
|
||
- The "pre-filter" phase covers verdicts produced inside `host_iter.c`. The "per-host" phase covers verdicts produced by `sched_try_alloc_on_host_` (5.2).
|
||
- The "not performed" rendering (5.1) follows from phase order: a pre-filter rejection implies the per-host check was not performed.
|
||
- `ALREADY CLAIMED` and `INTERNAL` carry no phase label (5.2 applies only to the selection, shortfall and limit kinds).
|
||
|
||
#### DecisionTransport (`include/decision.h`, `src/decision.c`)
|
||
|
||
**Contracts**: State [x]
|
||
- `xce_decision_op_t` gains `XCE_DECISION_TRACE` after `XCE_DECISION_PREEMPT`. Existing values are unchanged.
|
||
- `xce_decision_t` gains `struct xce_sched_trace_rec_s* trace;`. It is NULL for every existing constructor (`calloc`).
|
||
- `xce_decision_t* make_trace_decision(xce_sched_trace_rec_t* rec)` returns a decision with `op = XCE_DECISION_TRACE`, a zeroed union and `trace = rec`.
|
||
- **Ownership**:
|
||
- The drain loop that receives a decision owns `d->trace` and hands it to the REPL (`xce_repl_check_trace*`), which stores it or frees it.
|
||
- The REPL takes ownership by setting `d->trace = NULL`. Every sim call site (drain loop, trace-only branch, reconstructed block) then calls `sched_trace_rec_free(d->trace)` right before `free(d)`. This frees the record only if the REPL did not take it, for example when `!sim->interactive`, which cannot happen because `trace_ctl` is only set when interactive.
|
||
- `driver.c` frees any non-NULL `d->trace` and skips `XCE_DECISION_TRACE`. Both are unreachable in live; this is defensive.
|
||
- **Coordination**: `scheduler-preemption` also extends `decision.h`. Both specs append only, and neither reorders.
|
||
|
||
### Simulator Side
|
||
|
||
#### SimIntegration (`sim_loop.c`, `repl.h` gates)
|
||
|
||
- `xce_repl_trace_session_begin(sim)` runs after `xce_core_begin_session` has succeeded, which means after its `rc < 0` check (`sim_loop.c:1620-1631`), whose failure branch `continue`s without draining. It runs before reconciliation. A failed session begin therefore publishes nothing and holds no settings reference. It resets `rs->trace.round_traced` and `rs->trace.round_flushed` to false in every mode, so a round that ended in step mode (for example continue → breakpoint → `step`) cannot leak the flag into the next continue-mode round. It then sets `cur = next`, then sets `ctl->session_no = rs->session_count` and `ctl->position = 0`, and publishes `sim->core.trace_ctl = cur ? &cur->ctl : NULL`.
|
||
- In the reconstructed-dispatch report block (`sim_loop.c:1309-1329`), when `trace_ctl` wants the job, the block builds a `RECONSTRUCTED` record and sets `d->trace = rec` right after `make_dispatch_decision`, through the gate `xce_repl_trace_reconstructed(sim, d, job)` (`repl_trace_on_reconstructed` in `repl_trace_emit.c`). The order is then: record add → `sim_print_decision_gate` → `xce_repl_check_trace(sim, d)` → `xce_repl_check_decision(sim, d)` → `sched_trace_rec_free(d->trace)` → `free(d)`. This is the same order as the drain loop, so 7.5 holds for reconstructed dispatches too (2.5). This runs before `signal_go`, so these traces precede the round's live decisions (7.3).
|
||
- Drain loop:
|
||
- `if (d->op == XCE_DECISION_TRACE) { xce_repl_check_trace_only(sim, d); sched_trace_rec_free(d->trace); free(d); continue; }` comes first.
|
||
- `xce_repl_check_trace(sim, d)` sits between `sim_print_decision_gate` and `xce_repl_check_decision`. It is a no-op when `d->trace == NULL`. `sched_trace_rec_free(d->trace)` is called just before the existing `free(d)`.
|
||
- `xce_repl_trace_session_end(sim)` runs after the drain loop, before `plugin_end_session`. It sets `sim->core.trace_ctl = NULL`, drops `cur` (closing its file if it is no longer `next`), and clears the in-progress flag.
|
||
|
||
#### ReplTraceControl (`src/repl_trace.c`, `src/repl_trace_emit.c`)
|
||
|
||
`repl_trace.c` holds the command, status, settings, file and shutdown code; `repl_trace_emit.c` holds the session hooks and inline emission.
|
||
|
||
**Responsibilities & Constraints**
|
||
- Parses the raw input line for `trace` (1.9, 1.12):
|
||
```
|
||
trace on all [level summary|full] [lines N] [file PATH]
|
||
trace on job JOBSEL [JOBSEL ...] [level summary|full] [lines N] [file PATH]
|
||
trace off
|
||
trace status
|
||
```
|
||
- `JOBSEL` is `parse_jobsel` syntax (job, `job[idx]`, array base).
|
||
- `file` must be the last option. `PATH` is the rest of the line, trimmed, with one optional pair of surrounding quotes removed, and must be at most 4095 bytes.
|
||
- On any invalid argument, it prints `[sim] trace: invalid <what>: <arg>` and changes nothing.
|
||
- Keeps settings in `rs->trace` as reference-counted settings objects (see Trace settings lifecycle):
|
||
- `next`: the settings the next session will use, or NULL when tracing is off.
|
||
- `cur`: the settings of the session in progress, or NULL.
|
||
- `session_in_progress` and `round_traced`.
|
||
- File rules:
|
||
- The file is opened at command time in append mode, so open errors are reported immediately (8.2).
|
||
- Before opening, the path is compared by `stat()` dev/inode against `fstat(fileno(sim->record_file))` when an output file is set, and against `stat(sim->log_path)` (8.3).
|
||
- Replacing or disabling `next` drops its reference. If `cur` still holds that object, its file stays open until the session-end hook drops `cur`; otherwise the file is closed immediately (8.5).
|
||
- `repl_trace_file_conflicts(sim, path)` returns true when `path` matches, by dev/inode, the file of `next` or `cur`. These are every trace file still open. `set output` rejects such a path (8.4).
|
||
- Inline emission (`xce_repl_check_trace`, `xce_repl_check_trace_only`) runs on the sim thread:
|
||
1. In any mode it sets `round_traced = true`, so a round traced in step mode and finished with `continue` is never collapsed (7.4). In continue mode only: if this is the round's first traced item, it calls `repl_round_collapse_flush(sim)`; then, for every traced decision (not trace-only), it prints the echo line that the print gate suppressed, by calling `sim_print_decision(d)`. That function is made non-static and declared in `sim.h`, with its format unchanged (7.3). A per-round `round_flushed` bit, also reset at session begin, records whether the flush has been done.
|
||
2. Renders the record to stdout and, if `cur` (the settings the record was captured under) has a file, to that file, then `fflush`es both (7.1, 7.2, 8.1, 8.6).
|
||
3. Hands the record to the TraceStore (7.10).
|
||
- `repl_trace_shutdown(sim)` is called from `repl_cmd_quit`, the EOF path and process exit. It flushes and closes the files of `cur` and `next` (8.6).
|
||
- Writes nothing through tlog (8.7).
|
||
|
||
**Contracts**: Service [x] / State [x]
|
||
```c
|
||
/* repl_trace.c (declared in repl_internal.h) */
|
||
bool repl_trace_cmd(xce_sched_sim_driver_t* sim, const char* raw_line); /* returns false: re-prompt */
|
||
FILE* repl_trace_settings_file(const xce_repl_trace_settings_t* s); /* NULL: no file */
|
||
bool repl_trace_file_conflicts(const xce_sched_sim_driver_t* sim, const char* path);
|
||
void repl_trace_shutdown(xce_sched_sim_driver_t* sim); /* clears trace_ctl, releases cur/next, store_fini */
|
||
/* repl_trace_emit.c (declared in repl.h, called through its interactive-only gates) */
|
||
void repl_trace_session_begin(xce_sched_sim_driver_t* sim);
|
||
void repl_trace_session_end(xce_sched_sim_driver_t* sim);
|
||
void repl_trace_on_decision(xce_sched_sim_driver_t* sim, xce_decision_t* d); /* takes d->trace */
|
||
void repl_trace_on_trace_only(xce_sched_sim_driver_t* sim, xce_decision_t* d);
|
||
void repl_trace_on_reconstructed(xce_sched_sim_driver_t* sim, xce_decision_t* d,
|
||
const xce_sched_job_t* job); /* attaches a RECONSTRUCTED record (2.5) */
|
||
```
|
||
|
||
`trace status` output (1.7) shows a block for `cur` ("in effect", when a session is in progress) or `next` ("next session", otherwise), plus a second "replaces at next session" block when `next` differs from `cur`. Each block has enabled, selection (rendered with `repl_fmt_jobid_range`), level, line bound and file. A final line gives stored-trace memory `used / 64 MiB`.
|
||
|
||
#### TraceStore (`src/repl_trace_store.c`)
|
||
|
||
**Responsibilities & Constraints**
|
||
- An `htab` keyed by jobid maps to `{rec, bytes, seq}`. A capture-order FIFO supports eviction, and a set of discarded jobids (with their seq) supports 9.6.
|
||
- `put(rec)` replaces any existing entry for the jobid and frees the old record (7.10). It then evicts the oldest entries until `total_bytes <= 64 MiB`, never evicting the newest; evicted jobids are marked discarded (9.5). A newer capture clears the discarded mark.
|
||
- `info trace <jobsel>`:
|
||
- **Element or plain job**: renders the stored record at its captured level with the current units mode (7.8). If the trace was discarded it reports `trace discarded (memory bound)` (9.6). Otherwise it reports `no trace available` with the job's current state, or `unknown job` (7.11).
|
||
- **Array base**: iterates the stored and discarded entries whose `array_jobid` matches, in index order. If there are none, it lists the elements' current states from `core->jobs` (7.9).
|
||
- The store is only read and written on the sim thread. It never reads live scheduler state except job state for 7.9 and 7.11, at the REPL prompt.
|
||
|
||
**Contracts**: Service [x] / State [x]
|
||
```c
|
||
void repl_trace_store_put(xce_repl_trace_state_t* ts, xce_sched_trace_rec_t* rec);
|
||
size_t repl_trace_store_bytes(const xce_repl_trace_state_t* ts);
|
||
void repl_trace_store_fini(xce_repl_trace_state_t* ts); /* frees all records; called by repl_trace_shutdown */
|
||
bool repl_cmd_info_trace(xce_sched_sim_driver_t* sim, char tokens[][XCE_REPL_TOKEN_LEN], int ntok);
|
||
|
||
/* repl_internal.h: the state block embedded in xce_sched_repl_state_t */
|
||
typedef struct xce_repl_trace_settings_s xce_repl_trace_settings_t; /* refcounted; holds ctl + file */
|
||
typedef struct {
|
||
xce_repl_trace_settings_t* next; /* settings the next session will use, NULL = off */
|
||
xce_repl_trace_settings_t* cur; /* settings of the session in progress, NULL = none */
|
||
bool session_in_progress;
|
||
bool round_traced;
|
||
bool round_flushed;
|
||
htab_t by_job; /* jobid -> stored entry */
|
||
htab_t discarded; /* jobid -> eviction seq */
|
||
/* capture-order FIFO head/tail, total_bytes, seq counter */
|
||
} xce_repl_trace_state_t;
|
||
```
|
||
|
||
#### TraceRenderer (`src/repl_trace_render.c`, `src/repl_trace_render_outcome.c`, private `src/repl_trace_render.h`)
|
||
|
||
**Responsibilities & Constraints**
|
||
- `repl_trace_render(const xce_sched_trace_rec_t* rec, xce_repl_units_mode_t units, FILE* out1, FILE* out2 /* nullable */)`.
|
||
- **Line contract**: every stored entry in the budgeted sections produces exactly one line. Header, selection and outcome lines are outside the budget.
|
||
- **Hierarchy** (7.6), two-space indent per nesting level. The example below shows every stored entry: each line from the first `requirement` line to the last `alternative` line is exactly one entry and counts against the line budget. The header, selection and outcome lines are outside the budget.
|
||
```
|
||
[trace] job 1-0-0-42 session 7 position 3 level full
|
||
selection: policy builtin_fcfs queue normal (priority 50) job priority 0 queued 2026-10-07 10:00:01
|
||
cluster-wide limits: lim_user (u=alice); host-scoped limits are reported with each host's verdict
|
||
requirement alternative 1/2 (compound, 1 member)
|
||
requirement alternative 1/2 > member 1/1 tasks 4..4 requests mem=8 GiB slots=1 license=2
|
||
cluster-wide: license total 10 in-use 9 reserved 0 requested 2 failed
|
||
cluster-wide check failed; no placement plans tried
|
||
member 1/1 not satisfied
|
||
failure reasons: NO_RESOURCE x1 (resources: license)
|
||
alternative 1/2 not satisfied
|
||
requirement alternative 2/2 (compound, 1 member)
|
||
requirement alternative 2/2 > member 1/1 tasks 4..4 requests mem=8 GiB slots=1 license=1
|
||
cluster-wide: license total 10 in-use 9 reserved 0 requested 1 passed
|
||
cluster-wide limit lim_user (u=alice) on slots: value 100 usage 96 reserved 0 requested 4 passed
|
||
cluster-wide check passed
|
||
plan 1 hosts 1 tasks per position [4]
|
||
position 1 (4 tasks)
|
||
host n01 rejected (per-host check): resource shortfall
|
||
mem total 64 GiB in-use 60 GiB reserved 0 requested 8 GiB
|
||
host n02 rejected (pre-filter): selection mismatch; per-host check not performed
|
||
host n03 accepted
|
||
position 1 filled by n03
|
||
plan 1 succeeded
|
||
plan 1 committed; no further plans tried
|
||
plans: 1 generated, 1 tried, 0 discarded, 0 not tried
|
||
member 1/1 satisfied
|
||
alternative 2/2 satisfied
|
||
alternative 2/2 chosen; no alternatives after it
|
||
outcome: DISPATCH
|
||
host n03: mem=8 GiB slots=4 limit charges: lim_host (h=n03) slots 4
|
||
shared: license=1 limit charges: lim_user (u=alice) slots 4
|
||
```
|
||
- **Summary level**: the position line is followed by `accepted n03; rejected: selection mismatch 1, resource shortfall 1, limit exceeded 0 (...)` and no per-host lines (5.9).
|
||
- **Truncation** (9.2, 9.3): after the last stored entry, the marker `... <k> trace lines omitted (line bound N)`, then a fixed-size totals block: plans tried, plans discarded, requirements with no plan generated / all plans discarded / no plan placed, and host evaluations per verdict kind.
|
||
- **Outcome** (6.x, 9.4): up to 50 hosts with all their entries, then aggregates and `... <k> hosts omitted`. A pend lists `main_reason`, host-less entries, then host entries.
|
||
- **Values** use the moved `repl_fmt` helpers with the units mode passed in (7.7). Jobids use `fmt_jobid` and selection lists use `repl_fmt_jobid_range`.
|
||
- **No-decision records** (2.4): a header, then:
|
||
- `NOT_FOUND`: `selection: ordering keys and limits unavailable (job not found by the allocator)`.
|
||
- `NO_SCHEDOBJ`: the selection line, then `cluster-wide limits: unavailable (no scheduling object)`.
|
||
- `INTERNAL_ERROR` and `PEND_NO_REASON`: the selection and limit lines, then the full entry hierarchy the hooks captured. Allocator internal errors that end the attempt (a plan not tried, a snapshot failure, a finalize failure, an error in the cluster-wide check, a member merge failure) surface as `alloc_rc < 0` for PRIMARY and top-level COMPOUND jobs. Their entries therefore carry the 3.2 / 4.4 / 4.5 outcomes. Per-host errors (5.7) and host-scan errors (4.4) do not end the attempt; they appear as host and position entries in any record kind.
|
||
- Every kind ends with one line in place of the outcome, naming the kind and the error.
|
||
- **Reconstructed records** (2.5): a header, one line stating the dispatch was reconstructed from the recorded allocation, and the outcome.
|
||
|
||
**Contracts**: Service [x]
|
||
|
||
#### ReplIntegration (existing REPL files)
|
||
|
||
Summary only; no new boundaries.
|
||
- **`repl.c`**:
|
||
- `repl_dispatch` routes `trace` to `repl_trace_cmd(sim, line)`.
|
||
- `repl_print_help` adds a trailing "Trace" section (`trace on all|job ...`, `trace off`, `trace status`) and an `info trace <jobid>` line in State Inspection (1.11).
|
||
- `repl_cmd_quit` calls `repl_trace_shutdown`.
|
||
- **`repl_info.c`**: `repl_cmd_info` routes `tokens[1] == "trace"` to `repl_cmd_info_trace`.
|
||
- **`repl_params.c`**: `set output` returns an error `[sim] set output: <path> is an open trace file` when `repl_trace_file_conflicts` is true (8.4, 10.5).
|
||
- **`repl_rollup.c`**: in `repl_round_rollup_process`, the traced-round rule runs right after `repl_query_pend_jobids` has built this round's pend list. When `rs->trace.round_traced` is set:
|
||
1. call `repl_round_collapse_flush(sim)` (idempotent, and covers rounds that were traced only in step mode);
|
||
2. emit this round alone via `repl_round_emit(round_no, round_no, round_ts, round_ts, dispatch_count, preempt_count, pend_jobids, n_pend_jobids, idle)` with `idle = (dispatch_count == 0) && (preempt_count == 0) && (n_pend_jobids == 0)`. This `idle` matches `repl_round_collapse_flush` (`repl_rollup.c:251-253`), not the allocation-failure fallback, so a pend-only traced round is not reported as idle;
|
||
3. free the pend list and return without starting a range (7.3, 7.4).
|
||
|
||
If the pend query itself fails, the existing fallback already emits the round alone. `round_traced` is reset at the next session begin.
|
||
|
||
## Data Models
|
||
|
||
### Domain Model
|
||
- **TraceSettings** (REPL-owned, reference-counted): `all`, selectors, level, line bound, file (path + `FILE*` + dev/inode), and the `xce_sched_trace_ctl_t` it publishes. It is referenced by `next` and/or `cur` (see Trace settings lifecycle).
|
||
- **TraceControlBlock** (`xce_sched_trace_ctl_t`, REPL-owned, published to the core for one session): an immutable copy of the selection, level and bound, plus `session_no`, `policy_name` and the allocator-owned `position`.
|
||
- **TraceRecord** (aggregate root, one per traced selection):
|
||
- **Header**: jobid, kind, level, session_no, position, line_bound. (The planned "file tag of the settings it was captured under" was dropped in implementation, Implementation Note 4.2: `info trace` is console-only (8.1) and emission writes through the session's `cur` settings, so a record needs no file reference.)
|
||
- **Selection block**: policy, queue name and priority, job priority, `queue_time`, cluster-wide limits list.
|
||
- **Entries**: an ordered list. Each entry has a type, a depth, and a payload of copied strings and floats with resource flags.
|
||
- **Counters**: `plans_tried`, `plans_discarded`, `reqs_none_generated`, `reqs_all_discarded`, `reqs_none_placed`, `host_evals[6]`, `omitted_lines`.
|
||
- **Outcome**: a dispatch block (≤ 50 hosts in detail plus aggregates) or a pend block (≤ 50 hosts plus per-code counts), or none.
|
||
- **Invariants**:
|
||
- A record is self-contained (no pointers into scheduler memory).
|
||
- `stored entries ≤ line_bound`.
|
||
- Counters are exact.
|
||
|
||
### Logical Data Model
|
||
- **Entry types** (each renders as exactly one line):
|
||
- `RESREQ_BEGIN`, `RESREQ_END`, `ALT_SKIPPED`, `ALT_RESULT`, `MEMBER_FAILED`, `ATTEMPT_REASONS`
|
||
- `CHECK_ROW` (resource or limit), `CHECK_END` (passed / failed, with "no placement plans tried" on failure), `NO_PLAN`
|
||
- `PLAN_BEGIN`, `PLAN_DISCARDED`, `PLAN_NOT_TRIED`, `PLAN_END`, `COMMIT`, `FINALIZE_FAILED`, `SEARCH_END`
|
||
- `POS_BEGIN`, `POS_SUMMARY` (summary level), `POS_END` (filled / unfilled / scan error / internal; the scan error is set by `pos_scan_error`)
|
||
- `HOST` (verdict bitmask, phase, error), `HOST_ROW` (shortfall or limit row under a host)
|
||
- `ROLLBACK`
|
||
- **Row payload**: resource name, flags, valtype, total, in-use, reserved, requested, met. Limit rows and limit charges add the limit name, the scope instance and the limited resource's name (`vres->res->name`). When no vres is in scope, the limited resource is unknown: the name and scope are parsed from the `limit=` resource name and the amount is a raw number (TraceCapture's fallback). The renderer formats limit amounts by that limited resource's name, matching the existing `info` limit display (`repl_info.c:1359-1360`), so a memory limit renders in bytes units (7.7).
|
||
- **Store**: `htab<jobid → entry>`, a FIFO by capture sequence, and `htab<jobid → discarded seq>`. Memory accounting is the sum of `sched_trace_rec_bytes`. The discarded-mark table is not counted; it stays small because it holds one mark per evicted jobid.
|
||
|
||
## Error Handling
|
||
|
||
### Error Strategy
|
||
- **Capture-side allocation failure** (arena growth on the allocator thread): the record sets an `incomplete` flag and stops appending. The renderer prints `... trace incomplete (out of memory)`. Scheduling is unaffected. If allocating the record itself fails, the job is not traced.
|
||
- **Trace decision allocation failure** (`make_trace_decision` returns NULL): the record is freed and that no-decision trace is lost. This is the only best-effort gap, and it happens only after an earlier allocation failure.
|
||
- **User errors** (1.9, 8.2, 8.3, 8.4): one `[sim] trace: ...` or `[sim] set output: ...` line naming the argument or file and the cause (`strerror(errno)`). Settings are left unchanged.
|
||
- **File write errors** during emission: reported once per session as `[sim] trace: write to <path> failed: <cause>`. The file stays attached until the settings change, and console output continues.
|
||
|
||
### Monitoring
|
||
None beyond REPL output. Trace code never writes to tlog (8.7).
|
||
|
||
## Testing Strategy
|
||
|
||
TDD per `testing.md`: each test below is written before the code it covers.
|
||
|
||
### Unit Tests
|
||
1. `test_trace_model.c`:
|
||
- entry budget at bound−1, bound and bound+1, with the exact omitted count;
|
||
- counters stay exact after truncation;
|
||
- summary tallies count a host with SHORTFALL+LIMIT once per kind (5.9, 9.3);
|
||
- `sched_trace_ctl_wants` for plain, element, whole-array and unknown-yet-future ids (1.2, 1.3).
|
||
2. `test_trace_capture.c`:
|
||
- limit rows vs resource rows in both cluster-wide and host scope;
|
||
- limit never reported as a shortfall (4.2, 5.5);
|
||
- dispatch limit charges separated (6.1–6.3);
|
||
- 51-host dispatch and pend aggregates (9.4);
|
||
- the pend copy equals the source under `job_preason_eq` (6.4).
|
||
3. `test_trace_hooks_gate.c`:
|
||
- `SCHED_TRACE` does not evaluate a side-effecting argument when `trace_rec == NULL`;
|
||
- a full live-style allocation with `trace_ctl == NULL` allocates no record (11.2);
|
||
- a non-interactive sim run never publishes `trace_ctl` (11.3).
|
||
4. `test_trace_render.c`:
|
||
- golden output per section for full and summary levels;
|
||
- units modes (7.7);
|
||
- truncation marker and totals block (9.2, 9.3);
|
||
- outcome cap rendering (9.4);
|
||
- no-decision and reconstructed records (2.4, 2.5);
|
||
- one line per budgeted entry.
|
||
5. `test_trace_store.c`:
|
||
- replace (7.10);
|
||
- eviction at 64 MiB never drops the newest (9.5);
|
||
- discarded vs no-trace vs unknown job (7.11, 9.6);
|
||
- array listing with mixed stored and discarded entries (7.9).
|
||
|
||
### Integration Tests (sim driver with real picker and allocator threads)
|
||
1. `test_repl_trace_cmd.c`:
|
||
- grammar and every invalid-argument class (1.9);
|
||
- a path longer than 63 characters and 20 job ids (1.12);
|
||
- `next` vs `cur` across a decision breakpoint and a round-begin breakpoint (1.6);
|
||
- `trace status` blocks (1.7);
|
||
- file open failure (8.2);
|
||
- dev/inode rejection against `--sim-output`, `set output` and the log path, including a hard link (8.3, 8.4);
|
||
- a replaced file is kept open until session end, or closed immediately between sessions (8.5);
|
||
- quit flushes (8.6).
|
||
2. `test_repl_trace_e2e.c`, over a contention fixture with an ALTER job, a parjob, a shared license, a cluster-wide limit and a host-scoped limit:
|
||
- every section of Requirements 2–6 present;
|
||
- step-mode placement (7.1, 7.2);
|
||
- continue-mode flush and no collapse (7.3, 7.4);
|
||
- breakpoint ordering with `info trace` at the stop (7.5);
|
||
- the file copy byte-equals the console trace output (8.1);
|
||
- `info trace` after the job finishes, with the units mode changed (7.8).
|
||
3. Error-path seams use the **real** `sched_allocator`, never a stub `sched_allocator()`, because a stub would bypass the hooks under test.
|
||
- The test `#include`s `sched_election.c`, `scheduler.c`, `alloc_runner.c`, `host_iter.c`, `resource.c`, `sched_trace*.c` and the other needed sources directly, instead of linking `scheduler_srcs`. This lets source-level macros apply.
|
||
- Faults are injected as follows:
|
||
- **Plan reserve failure** (4.4): `#define realloc test_realloc` around the `alloc_runner.c` include, as in `test_alloc_runner_search.c:30-33` (macro) and `:87-95` (`test_realloc`).
|
||
- **Snapshot and finalize failures** (4.4, 4.5): the link-level wraps of two existing tests, all three listed in the new test's `link_args`:
|
||
- `test_alloc_runner_search.c`: `--wrap=xvec_dup` (`testsuite/meson.build:1780`, `__wrap_xvec_dup` at `test_alloc_runner_search.c:102-114`);
|
||
- `test_alloc_runner_result.c`: `--wrap=make_host_alloc` and `--wrap=make_job_alloc` (`meson.build:1793-1794`).
|
||
|
||
With every scheduler source included, `xvec_dup` is called from more places, so the existing pattern of arming the failure on a specific call number (`test_alloc_runner_search.c:98-114`) is kept. `test_alloc_runner_result.c`'s `sched_make_allocs` stub is not used, because it would collide with the real `scheduler.c`.
|
||
- **Host-scan error** (4.4): a worker iterator whose `iter_next` returns `< 0`.
|
||
- **Per-host internal error** (5.7): a failing per-host uncommitted-vector fetch.
|
||
- **`alloc_rc < 0` no-decision trace** (2.4): a stub `job_alloc_fn` returning `-EIO`.
|
||
- **Job-not-found** (2.4): a direct `cbuf_put` of an unknown jobid on `jbuf`.
|
||
|
||
### Equivalence Tests (Requirement 10)
|
||
1. `test_trace_equivalence.c`: the same contention replay is run four ways: trace off, `all`/`summary`, `all`/`full`, and a job subset. Each run must match on:
|
||
- the decision sequence and pend-reason sets (`job_preason_eq`);
|
||
- final `resInUse`/`sharedInUse`/`job->accounts`;
|
||
- `--sim-verify` mismatch count;
|
||
- `--sim-output` bytes, excluding wall-clock metadata (10.1–10.3).
|
||
2. With trace off, the REPL transcript of a scripted session is byte-identical to a baseline captured before the feature, except for the `help` text (10.4, 10.5).
|
||
3. The existing `test_repl_equivalence.c`, `test_sim_verify*` and live driver tests stay green unchanged (11.4).
|
||
|
||
### Performance
|
||
- Live and trace-off cost: `test_trace_hooks_gate.c` asserts no record allocation. Code review checks that each hook is a single `SCHED_TRACE` statement (11.2).
|
||
- Large trace: a 1000-host, maxTasks = 64 parjob traced at `full` with the default bound completes. The record stays within the line bound, and the store stays within 64 MiB after 10,000 traced decisions (9.1, 9.5).
|
||
|
||
## Security Considerations
|
||
- The trace file path is operator-supplied in an interactive local tool. It is opened in append mode with the operator's permissions. The dev/inode checks stop the trace from overwriting the replay output or the scheduler log. There are no new network or privileged surfaces.
|
||
|
||
## Invariant Cross-Check (scheduler audit dimensions)
|
||
1. **Driver state-equivalence**: hooks only read state. The live driver never sets `trace_ctl`/`trace_rec`. The equivalence tests use a contention fixture, as `scheduler-driver-equivalence.md` requires. **Holds.**
|
||
2. **Core I/O isolation**: `sched_trace*.c` do no I/O and are added to `test_core_isolation.c`'s source-text scan, with an additional no-stdio check (not to `sched_core_isolation_lib`; see Modified Files). `core.h` gains two opaque pointers only. **Holds.**
|
||
3. **jbuf capacity-1 rendezvous**: unchanged. The position is counted by the allocator from its own pops, and no picker-side hook exists. **Holds.**
|
||
4. **Snapshot field-completeness**: not applicable. Trace state is not part of the scheduler snapshot, recording or WAL; `snapshot` content must be unchanged (10.2). No save/restore mechanism is added. Stored traces are deliberately not persisted.
|