diff --git a/.kiro/steering/resource-questions.md b/.kiro/steering/resource-questions.md index 718ac598..75811514 100644 --- a/.kiro/steering/resource-questions.md +++ b/.kiro/steering/resource-questions.md @@ -255,6 +255,28 @@ silently. - **Would become** a spec, e.g. `task-relationship-spec` (the canonical jobspec), after N31. - **Open:** see N41–N44. +### PR-15: relationships and compound resreqs (status: discussing) +- **User (2026-10-12):** consider the feature interaction with compound resreqs. +- **Repo facts (2026-10-12):** + - Grammar: `alter := {compound} || ...`, `compound := N*{primary} + ...`. Each part has its own select/rusage/span/same/affinity and a fixed count (`minTasks = maxTasks = N`, `src/lib/xce_resreq.peg:83-99`). Parts are unnamed. + - The scheduler allocates a compound part by part, greedily, on one context (`sched_ctx_set_primary`, `src/scheduler/src/schedobj.c:547`; bug-1768, bug-1931). An alternative resreq tries one alternative at a time (bug-1769). There is no backtracking between parts. + - `xce_alloc_t` (`instid`, `from`, `duration`) has no part index, so allocations lose which part a host serves. +- **Interactions and proposed rules:** + 1. **Roles are compound parts.** Add named parts (N31); unnamed parts get implicit names `part0`, `part1`, … for LSF compatibility. The jobspec's `roles:` is the compound, and relationships refer to part names. + 2. **Cross-part relationships vs greedy allocation:** part 1 may take a domain where part 2 cannot fit, so the job pends although a placement exists. Use relationship-aware allocation instead: order parts by constraint tightness, enumerate candidate domains for a cross-part `colocate scope=D`, and place all related parts inside one domain (a bounded search, shared with PR-10). + 3. **Unsatisfiable-resreq cache:** the key must include a hash of the relationships. Never cache results that depend on another job's placement (relationships between jobs). + 4. **Alternatives:** relationships are declared once per job over role names. Every alternative must define all roles named by a *required* relationship (checked at submit); a *preferred* relationship naming a missing role is skipped. Pend reasons are per alternative. + 5. **Per-part `span`/`same`/`affinity`** compile into self-relationships (`span[hosts=1]` → `colocate scope=host`; `same[...]` → same-attribute constraint). Contradictions are rejected at submit. + 6. **Sidecar roles:** by default attached to the parent's tasks, sharing their allocation and hosts, with no separate part. Optionally a separate part with a derived count (N48). + 7. **Task ranges** (`-n min,max`, `parjob-placement` range search): ranges per role, or only on primary jobs (N49). + 8. **Allocation record:** each host allocation records its role (part index). WAL and snapshot stay field-complete. Dispatch gives each task its role, rank and hostfile (N31); `info job` shows allocations per role. + 9. **Configuration layering:** the profile owns the compound template and relationships; the job overrides only counts or ranges marked overridable. + 10. **Resize** (N33): per-role resize re-checks the role's relationships. + 11. **Pend reasons and trace:** per part and per relationship (e.g. "decode: no switch with 3 free 8-GPU hosts near prefill"); trace capture per part; deterministic for sim/replay. + 12. **Accounting:** actual-usage reconciliation (`max(reserved, actual)`) is single-host only (`resource-accounting.md`); multi-role multi-host jobs inherit that gap. Record it explicitly (N50). +- **Recommendation:** the compound resreq is the resource half of the canonical jobspec (PR-14), and relationships are the other half. The essential engineering is items 1, 2 and 8. Items 3 and 4 are correctness rules required in v1. +- **Affects:** resreq grammar and `resreq.h`, `schedobj.c` and the allocator plugins (`parjob_allocator.c`), `alloc_cache.c`, `xce_alloc_t` and the WAL/snapshot allocation encoding, worker dispatch and task environment, `parjob-placement`, `job-resize`, `scheduler-alter-pend-reasons`, the decision trace. + ## Gang use cases (2026-10-11, for N24) - **Two kinds of gang:** - **Inside one job:** all tasks of a parallel job start together. XCE already does this, and compound resreqs mix role shapes in one job (like Slurm heterogeneous jobs). @@ -385,9 +407,16 @@ Source: the CSI specification, from general knowledge; to be checked against the | N42 | PR-14: relationship kinds in v1? | open | Suggested v1: `colocate`/`spread`, `exchange`, `with`/`after`, `sidecar-of`/`init-of`, `fate`. Later: `isolate`/`share` QoS, bandwidth, `consumes`/`produces`. | | N43 | PR-14: where relationships live in config layering: profile templates, which are overridable, per-job additions? | open | | | N44 | PR-14: does the same vocabulary cover relationships between jobs, replacing a separate placement-group mechanism (PR-11)? | open | Leaning yes. | +| N45 | PR-15: implicit names (`part0`…) for unnamed compound parts; is the name syntax inside the brace (`N*{name=prefill: ...}`) or in the jobspec only? | open | | +| N46 | PR-15: allocation search for cross-part relationships: domain-first enumeration, bounded backtracking, or both? What bound keeps the session cost acceptable? | open | | +| N47 | PR-15: relationships vs alternatives: one relationship set per job over role names (validated per alternative), or separate relationship sets per alternative? | open | Leaning one set per job. | +| N48 | PR-15: sidecar roles: attached to the parent's tasks (shared allocation), or a separate compound part with a derived count? | open | Leaning attached by default. | +| N49 | PR-15: task ranges in compound jobs: per-role ranges searched by the planner, or ranges only for primary (single-part) jobs? | open | | +| N50 | PR-15: actual-usage reconciliation for multi-role, multi-host jobs (today single-host only): extend it, or record as an accepted gap? | open | | | C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). | ## History +- 2026-10-12: user asked about the interaction with compound resreqs → PR-15 (12 interactions; named parts, relationship-aware cross-part search, role on every allocation, cache-key and alternative rules); N45–N50 added. - 2026-10-12: user asked for a uniform approach to describe task relationships → PR-14 (canonical jobspec with typed relationship records; shorthands compile into it); N41–N44 added. - 2026-10-12: user raised task-level cooperation (contention, data exchange, piped dependencies, more) → PR-13 (task relationship model with typed edges between roles; allocation plus job-step runtime); N35–N40 added. - 2026-10-12: user asked whether disaggregated serving can be one job → PR-12 (one compound job per unit, replicated by `service-replica-set`); N31–N34 added; `gang-groups` scope narrowed.