diff --git a/.kiro/steering/resource-questions.md b/.kiro/steering/resource-questions.md index 69130cb9..0ef8d7c6 100644 --- a/.kiro/steering/resource-questions.md +++ b/.kiro/steering/resource-questions.md @@ -187,6 +187,42 @@ silently. - **Effect on N24:** disaggregated serving mostly moves to single-job units. `gang-groups` remains for independent pools behind a global router, RL post-training and streaming pipelines. - **Affects:** the resreq syntax (named parts), worker task environment and hostfiles (N31, also needed by PR-10 training), `service-replica-set`, `service-supervision` (N32), `job-resize` (N33), `service-endpoint` (N34), `gang-groups` (scope note). +### PR-13: task relationship model, typed edges between roles (status: discussing) +- **User (2026-10-12):** consider how different work cooperates at the task level. Tasks race for resources (CPU, memory, network bandwidth, cache), exchange data by memory, disk or network, and depend on each other in piped processing, plus other relationships. We need to manage complex relationships between tasks. +- **Relationship types:** + 1. **Contention:** CPU cores, memory capacity and bandwidth, last-level cache, NUMA, PCIe/NVLink, NIC bandwidth, disk I/O, GPU SMs (MPS). + - *Isolate* (worker): cgroups cpuset/memory/io; NUMA binding via `affinity[]` (parsed, unused, `lib/resreq.c:127`); Intel RDT cache and memory-bandwidth allocation; NIC traffic classes; MIG. + - *Place around interference* (scheduler): keep noisy tasks apart, pair complementary ones. + - Measurable bandwidth can become L2 counters. + 2. **Data exchange:** + - shared memory or local disk → same host or NUMA scope; + - network (RDMA, NCCL, TCP) → switch or rail proximity, plus bandwidth; + - plumbing: shared IPC namespace or shm mount, a shared scratch claim, endpoint and rank discovery (N31). + 3. **Pipelines and dependencies:** + - streaming (concurrent): co-start, a channel, lifecycle coupling; + - staged (B after A): a DAG inside the job, i.e. PR-8 at task granularity; + - flow control and backpressure stay in the application. + 4. **Others:** + - **lifecycle coupling:** fate policy (fail together, restart one, tolerate); sidecar tasks (log shipper, data loader, proxy); init tasks (download weights, warm cache); readiness gating between roles; + - **co-location and spread:** with a scope from NUMA to block, hard or soft; + - **communication pattern:** ring, all-to-all (MoE), pipeline neighbours → rank-to-topology mapping (PR-10); + - **QoS between co-located tasks:** latency-critical serving next to batch; + - **isolation between tenants** sharing a node (PR-9; exclusive hosts); + - **shared budgets:** power and thermal caps as pool counters. +- **Making it manageable:** + - **Edges between roles** (N31 named parts), expanded per task by the scheduler. + - **One vocabulary** for task-to-task, job-to-job (PR-11 placement groups) and job-to-data (PR-5 locality). + - **Layering:** the scheduler turns edges into placement constraints and scores from its cached snapshot; the worker enforces contention and exchange (cgroups, RDT, namespaces, mounts) and runs local sidecar and init ordering; the controller runs cross-host lifecycle (fate sharing, restarts, readiness gates); the application keeps flow control. + - **Allocation plus steps** (N36): the scheduler allocates once, topology-aware; a job-step runtime inside the allocation launches tasks and steps by the graph (init, then main plus sidecars, then staged pipeline steps), like Slurm job steps or Flux nested scheduling. Sequencing stays out of the scheduler's hot path. +- **Prior art:** Kubernetes pods (shared network, IPC and volumes), sidecar and init containers, pod affinity, Topology/CPU Manager, QoS classes; Volcano task-topology plugin (affinity between task roles); Ray placement-group bundles; Slurm job steps, heterogeneous jobs and CPU binding; LSF `affinity[]`, `blaunch`, compound resreqs; Flux hierarchical scheduling; Intel RDT. +- **Suggested phasing:** + 1. N31 role identity. + 2. Placement edges between roles (PR-11 vocabulary). + 3. Node isolation: cpuset/NUMA via `affinity[]`, memory/io cgroups; RDT, NIC QoS and bandwidth counters later. + 4. Lifecycle edges: init and sidecar tasks, fate policy, per-task restart (N32). + 5. Job-step runtime: staged and streaming pipelines inside an allocation, with data handoff. +- **Affects:** resreq syntax (roles and edges), `scheduler-framework` (constraints and scoring), worker (`xcecd_cgroups.c`, isolation, namespaces), `service-supervision` (lifecycle), `job-dependency` (task-level DAG), `parjob-placement`, `resource-instances`. + ## Gang use cases (2026-10-11, for N24) - **Two kinds of gang:** - **Inside one job:** all tasks of a parallel job start together. XCE already does this, and compound resreqs mix role shapes in one job (like Slurm heterogeneous jobs). @@ -307,9 +343,16 @@ Source: the CSI specification, from general knowledge; to be checked against the | N32 | PR-12: long-running service jobs: restart policy and per-task replacement inside a running job, instead of failing the whole unit? | open | Extends `service-supervision`. | | N33 | PR-12: per-part resize of a compound job (`job-resize`), or scaling by whole units only? | open | Leaning units first. | | N34 | PR-12: a router inside each unit plus an outer load balancer, or one global router across units? | open | A global router pulls toward independent pools (`gang-groups`). | +| N35 | PR-13: relationship edges between roles only, or per-task edges too? | open | Leaning roles only (compact for jobs with thousands of tasks). | +| N36 | PR-13: who sequences tasks and steps: the scheduler, or a job-step runtime inside the allocation (Slurm steps, Flux)? | open | Leaning the job-step runtime. | +| N37 | PR-13: contention in v1: isolation only, or also interference-aware placement (needs telemetry and a model)? | open | Leaning isolation only in v1. | +| N38 | PR-13: network, memory and I/O bandwidth as reserved counters (L2), or best-effort QoS? | open | | +| N39 | PR-13: QoS classes for co-locating latency-critical serving with batch work on the same node? | open | | +| N40 | PR-13: defaults for sidecar and init tasks and fate sharing: MPI-style fail-together, or tolerate? | open | | | C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). | ## History +- 2026-10-12: user raised task-level cooperation (contention, data exchange, piped dependencies, more) → PR-13 (task relationship model with typed edges between roles; allocation plus job-step runtime); N35–N40 added. - 2026-10-12: user asked whether disaggregated serving can be one job → PR-12 (one compound job per unit, replicated by `service-replica-set`); N31–N34 added; `gang-groups` scope narrowed. - 2026-10-12: N24 decided (user): `gang-groups` demoted to the backlog and re-scoped as a co-start group for RL post-training, disaggregated serving and streaming pipelines. - 2026-10-11: user asked about data cache, data and communication topology between jobs → PR-11 (one topology model; data locality with delay scheduling; placement groups with pack or spread); N26–N30 added.