From da46c68d15821161e0e64c29c68b51ce4738969b Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Mon, 12 Oct 2026 07:27:13 +0000 Subject: [PATCH] docs(steering): resource log: disaggregated serving as one job per unit (PR-12, N31-N34) Co-Authored-By: Claude Opus 5.5 (1M context) --- .kiro/specs/gang-groups/brief.md | 9 ++++++--- .kiro/steering/resource-questions.md | 21 +++++++++++++++++++++ 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/.kiro/specs/gang-groups/brief.md b/.kiro/specs/gang-groups/brief.md index 08b70bf9..522b5d8f 100644 --- a/.kiro/specs/gang-groups/brief.md +++ b/.kiro/specs/gang-groups/brief.md @@ -18,9 +18,10 @@ when parts wait on each other. - **RL post-training** (RLHF, GRPO): a training job, rollout engines (e.g. vLLM), and reward and reference models. Each is a different app; rollouts restart and scale on their own. -- **Disaggregated LLM serving:** prefill pool, decode pool and router. They - come up together and are placed close for KV-cache transfer, then scale - independently as long-running services. +- **Disaggregated LLM serving with independent pools:** prefill and decode + pools that scale independently behind a global router. Fixed xPyD units + are one compound job each, replicated by `service-replica-set` (PR-12), + and need no gang. - **Streaming pipelines:** a live producer feeding a consumer (preprocessing into training, online evaluation) with different profiles or owners. @@ -30,6 +31,8 @@ when parts wait on each other. requirements (PR-10, `resource-instances`, `parjob-placement`); - a multi-host replica of a served model: one parallel job per replica in `service-replica-set`; +- a disaggregated serving unit (router, prefill and decode in a fixed + ratio): one compound job per unit (PR-12); - jobs that only need to be placed near each other: placement groups (PR-11), which give co-placement without a co-start. diff --git a/.kiro/steering/resource-questions.md b/.kiro/steering/resource-questions.md index 9c9970fc..69130cb9 100644 --- a/.kiro/steering/resource-questions.md +++ b/.kiro/steering/resource-questions.md @@ -171,6 +171,22 @@ silently. - **Affects:** `scheduler-framework` (scoring and filters), `parjob-placement` (shared topology labels), `service-replica-set` (spread), PR-4 (labels on host entries), PR-5/PR-8 (copy locations), `custom-resource-reporting` (topology discovery), N22/N25. - **Open:** see N26–N30. +### PR-12: disaggregated LLM serving as one job per unit, replicated (status: discussing) +- **User (2026-10-12):** can disaggregated LLM serving be put into a single job? +- **Proposal: yes, one job per unit.** + - **A unit is one compound job:** a router, P prefill and D decode hosts, e.g. 1P3D: `1*{router: select[!gpu]} + 1*{prefill: rusage[gpu=8] span[ptile=1]} + 3*{decode: rusage[gpu=8] span[ptile=1]}`. + - **What one job gives for free:** all-or-nothing start with no gang; topology packing inside the job, so prefill and decode share a switch or NVLink domain for KV transfer (PR-10); one lifecycle and one accounting record. + - **Scaling:** `service-replica-set` scales whole units (each replica is one unit job); `service-endpoint` balances across units' routers. + - **Prior art:** fixed xPyD units, replicated as Kubernetes LeaderWorkerSet groups or NVIDIA Dynamo/llm-d graphs. +- **Limits and remedies:** + - **Prefill:decode ratio** is fixed per unit: offer several unit shapes (1P3D, 2P2D) as profiles; later, per-part resize (N33). + - **Scaling one pool alone** is impossible within a unit: scale units. Independent pools behind a global router remain a `gang-groups` co-start case (N34). + - **A host failure** fails the unit: acceptable while other replicas serve; later, per-task replacement (N32). + - **Rolling updates** happen per unit through `service-replica-set`. +- **Repo facts (2026-10-12):** compound parts are an unnamed list (`xce_compound_resreq_t.primaries`, `src/include/resreq.h:255-257`). Tasks see only `XCE_JOBID`, `XCE_JOBINDEX` and internal variables; there is no role, rank or hostfile (N31). +- **Effect on N24:** disaggregated serving mostly moves to single-job units. `gang-groups` remains for independent pools behind a global router, RL post-training and streaming pipelines. +- **Affects:** the resreq syntax (named parts), worker task environment and hostfiles (N31, also needed by PR-10 training), `service-replica-set`, `service-supervision` (N32), `job-resize` (N33), `service-endpoint` (N34), `gang-groups` (scope note). + ## Gang use cases (2026-10-11, for N24) - **Two kinds of gang:** - **Inside one job:** all tasks of a parallel job start together. XCE already does this, and compound resreqs mix role shapes in one job (like Slurm heterogeneous jobs). @@ -287,9 +303,14 @@ Source: the CSI specification, from general knowledge; to be checked against the | N28 | PR-11: delay scheduling for data locality: wait per job, queue or profile; default? | open | Same mechanism as N22. | | N29 | PR-11: where topology comes from: admin config only in v1, discovery (`ibnetdiscover`, LLDP) later via a reporter? | open | | | N30 | PR-11: data distance: simple locality tiers, or a size-weighted cost deciding between staging a copy and placing far? | open | Leaning tiers in v1. | +| N31 | PR-12/PR-10: role identity in compound jobs: named parts in the resreq (`name=prefill`), plus per-task part name, part index, rank and per-part hostfile or environment (`XCE_ROLE`, `XCE_ROLE_INDEX`, `XCE_HOSTS_`)? None exists today. | open | Needed for disaggregated serving units and for training (coordinator plus workers). | +| N32 | PR-12: long-running service jobs: restart policy and per-task replacement inside a running job, instead of failing the whole unit? | open | Extends `service-supervision`. | +| N33 | PR-12: per-part resize of a compound job (`job-resize`), or scaling by whole units only? | open | Leaning units first. | +| N34 | PR-12: a router inside each unit plus an outer load balancer, or one global router across units? | open | A global router pulls toward independent pools (`gang-groups`). | | C3 | CSI compatibility: nice-to-have or a must? If a must, which side: the controller plugin through the proxy, the node plugin on the worker, or both? | discussing | Background and recommendation in "CSI notes" below (2026-10-11). | ## History +- 2026-10-12: user asked whether disaggregated serving can be one job → PR-12 (one compound job per unit, replicated by `service-replica-set`); N31–N34 added; `gang-groups` scope narrowed. - 2026-10-12: N24 decided (user): `gang-groups` demoted to the backlog and re-scoped as a co-start group for RL post-training, disaggregated serving and streaming pipelines. - 2026-10-11: user asked about data cache, data and communication topology between jobs → PR-11 (one topology model; data locality with delay scheduling; placement groups with pack or spread); N26–N30 added. - 2026-10-11: user asked for the use cases of gangs → "Gang use cases" section; N24 leaning (demote `gang-groups`, re-scope later as a co-start group for RL, disaggregated serving and streaming).