From a14595deb057c48a652e0a2065bf83e91e0387b7 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Mon, 12 Oct 2026 07:58:01 +0000 Subject: [PATCH] docs(steering): resource log: uniform task relationship description, canonical jobspec (PR-14, N41-N44) Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/steering/resource-questions.md | 37 ++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/.kiro/steering/resource-questions.md b/.kiro/steering/resource-questions.md index 0ef8d7c6..718ac598 100644 --- a/.kiro/steering/resource-questions.md +++ b/.kiro/steering/resource-questions.md @@ -223,6 +223,38 @@ silently. 5. Job-step runtime: staged and streaming pipelines inside an allocation, with data handoff. - **Affects:** resreq syntax (roles and edges), `scheduler-framework` (constraints and scoring), worker (`xcecd_cgroups.c`, isolation, namespaces), `service-supervision` (lifecycle), `job-dependency` (task-level DAG), `parjob-placement`, `resource-instances`. +### PR-14: one uniform way to describe task relationships, a canonical jobspec (status: discussing) +- **User (2026-10-12):** should we add a uniform approach to describe the relationships of tasks? +- **Proposal: yes.** Without one, each relationship kind grows its own syntax, as in Kubernetes (node and pod affinity, topology spread, init containers, sidecars, dependencies). Precedent: Flux's jobspec, a canonical description of resources and tasks. + - **One record shape:** `{kind, from, to, scope, strength, params}`. + - **Subjects and objects:** `role:` (N31), `job:`, `group:`, `data:`, `claim:`. + - **A fixed set of kinds**, each with its enforcing layer: + - `colocate`/`spread`: scope numa…zone, strength required or preferred, weight. Scheduler. + - `exchange`: channel shm, localdisk, sharedfs or network, plus bandwidth. Scheduler (the channel implies a scope) and worker (plumbing). + - `with`/`after`/`ready-after`: co-start, after completion, after readiness. Controller or job-step runtime. + - `sidecar-of`/`init-of`: worker and controller. + - `fate`: fail-together, restart-member, tolerate. Controller. + - `isolate`/`share`: exclusivity or QoS class, deliberate sharing (MPS, cache). Worker and scheduler. + - `consumes`/`produces`: data dependency and locality (PR-8). Scheduler and controller. + - **One vocabulary at three levels:** inside a job (role to role), between jobs (`job:`, `group:`; PR-11 placement groups become relations toward a `group:`), and to data (`data:`, `claim:`). +- **Canonical form plus shorthands:** + - **Canonical:** a structured job spec (YAML/JSON over REST), persisted on the job (WAL submit fact, field-complete in snapshots) and shown by `info job`. The resreq stays the per-role resource request inside it. + - **Shorthands that compile into it:** xcecli flags (`--placement-group`, `--after`, `--data`); application-profile relationship templates (e.g. "1P3D serving unit", "training with a data-loader sidecar"); and existing `span[]`/`same[]`/`affinity[]`, translated into `colocate`, `spread` and `isolate`. +- **Example** (a PR-12 unit): roles `router` (1), `prefill` (1), `decode` (3), `metrics` (per host). Relations: + - `colocate prefill↔decode scope=switch required` + - `exchange prefill→decode channel=network rdma` + - `ready-after router→{prefill,decode}` + - `sidecar-of metrics→{prefill,decode}` + - `fate {prefill,decode,router} fail-together` +- **Rules:** + - The scheduler sees only placement relations, compiled into filters and scores; the rest passes through. The hot path stays cheap and deterministic. + - Validation at submit: contradictions, ordering cycles, unknown roles, impossible scopes. + - A pending job names the relation that blocks it (`scheduler-alter-pend-reasons`). + - Configuration layering: profile relations are profile-owned, and a job overrides only overridable ones (`job-config-layering.md`). + - The schema is versioned, and new kinds are additive. +- **Would become** a spec, e.g. `task-relationship-spec` (the canonical jobspec), after N31. +- **Open:** see N41–N44. + ## Gang use cases (2026-10-11, for N24) - **Two kinds of gang:** - **Inside one job:** all tasks of a parallel job start together. XCE already does this, and compound resreqs mix role shapes in one job (like Slurm heterogeneous jobs). @@ -349,9 +381,14 @@ Source: the CSI specification, from general knowledge; to be checked against the | N38 | PR-13: network, memory and I/O bandwidth as reserved counters (L2), or best-effort QoS? | open | | | N39 | PR-13: QoS classes for co-locating latency-critical serving with batch work on the same node? | open | | | N40 | PR-13: defaults for sidecar and init tasks and fate sharing: MPI-style fail-together, or tolerate? | open | | +| N41 | PR-14: a canonical structured jobspec plus resreq and CLI shorthands, or extend the resreq string only? | open | Leaning canonical plus shorthands. | +| N42 | PR-14: relationship kinds in v1? | open | Suggested v1: `colocate`/`spread`, `exchange`, `with`/`after`, `sidecar-of`/`init-of`, `fate`. Later: `isolate`/`share` QoS, bandwidth, `consumes`/`produces`. | +| N43 | PR-14: where relationships live in config layering: profile templates, which are overridable, per-job additions? | open | | +| N44 | PR-14: does the same vocabulary cover relationships between jobs, replacing a separate placement-group mechanism (PR-11)? | open | Leaning yes. | | C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). | ## History +- 2026-10-12: user asked for a uniform approach to describe task relationships → PR-14 (canonical jobspec with typed relationship records; shorthands compile into it); N41–N44 added. - 2026-10-12: user raised task-level cooperation (contention, data exchange, piped dependencies, more) → PR-13 (task relationship model with typed edges between roles; allocation plus job-step runtime); N35–N40 added. - 2026-10-12: user asked whether disaggregated serving can be one job → PR-12 (one compound job per unit, replicated by `service-replica-set`); N31–N34 added; `gang-groups` scope narrowed. - 2026-10-12: N24 decided (user): `gang-groups` demoted to the backlog and re-scoped as a co-start group for RL post-training, disaggregated serving and streaming pipelines.