spec(scheduler-framework): finalize 4-level job-picker design
Redesign the plugin picker hierarchy to project→queue→job+preempt (was 3-level). Adds LSF-style cross-queue preemption (SIGSTP) and a unified v1/v2 dispatch loop. Requirements grow 6→9 to cover the new preemption and cross-queue levels.
This commit is contained in:
parent
6529dcf08f
commit
2b15c09646
3 changed files with 491 additions and 55 deletions
|
|
@ -1,12 +1,14 @@
|
|||
# Brief: scheduler-framework
|
||||
|
||||
## Problem
|
||||
The XCE scheduler has a plugin architecture but three critical gaps prevent it from being a real framework:
|
||||
1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:48` uses the literal string `"xce_sched_fcfs"`. No way to swap policies without recompilation.
|
||||
The XCE scheduler has a plugin architecture but critical gaps prevent it from being a real framework:
|
||||
1. **Job selection policy is hardcoded** — `sched_job_picker()` in `scheduler.c:39` uses the literal string `"xce_sched_fcfs"`. No way to swap policies without recompilation. The picker is monolithic — it both chooses which queue to serve and which job within the queue.
|
||||
2. **Parallel job allocation is a stub** — `parjob_allocator.c:my_alloc()` returns 0 with no logic. Jobs requiring multiple slots (`maxTasks > 1`) produce no allocation.
|
||||
3. **Allocation is one-shot** — `sched_make_allocation()` tries the first matching host and stops. No mechanism to try alternative allocation plans (different host×slot combinations) when the first fails.
|
||||
4. **No preemption** — when a high-priority job cannot be scheduled due to insufficient free resources, there is no mechanism to preempt lower-priority running jobs to free resources.
|
||||
5. **No project-level scheduling** — `xce_sched_project_t` is just `{ char* name }`. Projects have no resource ownership, quota, or scheduling policy. Queue and project roles are conflated.
|
||||
|
||||
Operators need configurable scheduling policies and parallel job support. The scheduler must generate multiple candidate allocation plans and try them in preference order.
|
||||
Operators need configurable scheduling policies at multiple levels, parallel job support, and preemption. The scheduler must generate multiple candidate allocation plans and try them in preference order.
|
||||
|
||||
## Current State
|
||||
|
||||
|
|
@ -21,18 +23,46 @@ Operators need configurable scheduling policies and parallel job support. The sc
|
|||
- **Scheduling loop** (`scheduler.c:sched_allocator()`): pick job → `sched_make_allocation()` → pend or dispatch → next.
|
||||
|
||||
### What's missing
|
||||
- Configurable job picker: the plugin name is a hardcoded string literal
|
||||
- Configurable job picker: the plugin name is a hardcoded string literal; a monolithic picker conflates inter-queue and intra-queue selection
|
||||
- Parallel job allocation: no combinatorial plan generation, no multi-slot host matching
|
||||
- Iterator/lazy generation: allocation is eager one-shot, no retry with alternative plans
|
||||
- Preemption: no mechanism to preempt lower-priority running jobs when a high-priority job cannot be scheduled
|
||||
- Project-level scheduling: `xce_sched_project_t` is just `{ char* name }` — no resource ownership, quota, or scheduling policy
|
||||
- Cross-job resreq optimization: no mechanism to skip already-failed resreq patterns within a session
|
||||
- Picker composition: no way to assign different pickers to different queues or projects
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
1. **Configurable job selection**: operator can set `job_pick_plugin` in scheduler config to swap policies (FCFS, priority, custom) without recompilation. Plugin interface contract is documented (cancel-safe, session-aware).
|
||||
2. **Lazy allocation plan iterator**: `xce_alloc_plan_iter_t` — generates candidate (nhosts, slots_per_host) plans one at a time, sorted by preference. Consumer tries each plan until one succeeds or all fail.
|
||||
3. **Lazy host candidate iterator**: for a given allocation plan, iterate candidate hosts filtered by select expression + resource availability. Composes with plan iterator.
|
||||
4. **Parallel job combinatorial allocation**: `parjob_allocator` implements the plan iterator. For a job needing N slots: generate plans [1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1]. Each plan: collect hosts with enough slots, try each until success.
|
||||
5. **Session-level unsatisfiable resreq cache**: if a primary resreq cannot be satisfied in the current scheduling session (all plans exhausted), fingerprint it. Skip subsequent jobs with the same fingerprint for the rest of the session.
|
||||
### Four-Level Scheduling Architecture
|
||||
|
||||
The scheduling loop is always `project → queue → job`, composing pickers at each level:
|
||||
|
||||
| Level | Picker | Plugin slot | Default (v1) |
|
||||
|-------|--------|-------------|---------------|
|
||||
| 0 | Project picker | `sched->project_pick_plugin` | `builtin_flat` — single "default" project |
|
||||
| 1 | Inter-queue picker | `sched->queue_pick_plugin` | `builtin_strict_priority` |
|
||||
| 2 | Intra-queue picker | `queue→project→sched` chain | `builtin_fcfs` |
|
||||
| 3 | Preemption policy | `queue->preempt_policy` | `lowest_priority_first` |
|
||||
|
||||
1. **Unified picker loop**: A single scheduling loop that always follows `project → queue → job`. In v1 (flat project space), the project picker `builtin_flat` yields one "default" project containing all queues — identical to today's behavior. In v2 (future), the project picker becomes quota-aware and hierarchical, without changing the loop code.
|
||||
2. **Configurable job selection at every level**: each picker slot is a plugin. Level 1 (inter-queue) picks which queue to serve next. Level 2 (intra-queue) picks which job within that queue, with per-queue customization via the `queue → project → global` precedence chain.
|
||||
3. **Lazy allocation plan iterator**: `xce_alloc_plan_iter_t` — generates candidate (nhosts, slots_per_host) plans one at a time, sorted by preference. Consumer tries each plan until one succeeds or all fail.
|
||||
4. **Lazy host candidate iterator**: for a given allocation plan, iterate candidate hosts filtered by select expression + resource availability. Composes with plan iterator.
|
||||
5. **Parallel job combinatorial allocation**: `parjob_allocator` implements the plan iterator. For a job needing N slots: generate plans [1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1]. Each plan: collect hosts with enough slots, try each until success.
|
||||
6. **Preemption**: when allocation fails for a high-priority job, select victim jobs from lower-priority queues (LSF-style per-queue `preemptible_queues` list), send SIGSTP via controller→worker pipeline, and retry allocation next round.
|
||||
7. **Session-level unsatisfiable resreq cache**: if a primary resreq cannot be satisfied in the current scheduling session (all plans + preemption exhausted), fingerprint it. Skip subsequent jobs with the same fingerprint for the rest of the session.
|
||||
|
||||
### Queue vs Project — Roles
|
||||
|
||||
| | Queue | Project |
|
||||
|---|---|---|
|
||||
| **Role** | Scheduling mechanism | Resource ownership |
|
||||
| **Question** | HOW are jobs ordered and constrained? | WHO owns resources and gets charged? |
|
||||
| **Scope** | One hardware/scheduling profile | Cross-queue, cross-cluster |
|
||||
| **Key attributes** | Priority, limits, time windows, picker, preemption config | Quota, usage, hierarchy, members (future) |
|
||||
| **Cardinality** | 1 job → 1 queue, many-to-many | 1 job → 1 project, many-to-many |
|
||||
|
||||
One project submits to many queues (different hardware profiles). One queue receives from many projects (shared hardware).
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
|
|
@ -73,94 +103,202 @@ Operators need configurable scheduling policies and parallel job support. The sc
|
|||
|
||||
### Decision rationale
|
||||
|
||||
Approach B chosen over A because the iterator abstraction is a one-time design cost that pays off immediately (parjob combinatorial allocation needs it) and compounds (every future allocator reuses it). Approach C's additional layers (policy chains, resreq registry, state machine) are deferred until real use cases emerge — the iterator interfaces are designed to compose upward when needed.
|
||||
Approach B chosen over A because the iterator abstraction is a one-time design cost that pays off immediately (parjob combinatorial allocation needs it) and compounds (every future allocator reuses it). The four-level picker architecture (project → queue → job + preemption) extends Approach B with design elements from Approach C (policy composition, formalized level separation) that are needed NOW (per-queue pickers, preemption), while deferring the remaining Approach C layers (policy chains with scoring, resreq registry, formal state machine) until real use cases emerge. The iterator interfaces and picker slots are designed to compose upward when needed.
|
||||
|
||||
**Updated decision (2026-07-10):** The original Approach B treated the job picker as a single global plugin (`job_pick_plugin` on `xce_scheduler_t`). Design discussion identified that different queues/projects need different pickers, and that preemption is an essential missing mechanism. The architecture now specifies four composable levels (project, queue, job, preemption) with per-queue configuration, LSF-style cross-queue preemption, and a unified loop that works identically for v1 (flat projects) and v2 (hierarchical projects). The project picker level is deferred to a future spec but the architecture slot and interface contract are defined now.
|
||||
|
||||
## Approach
|
||||
|
||||
**Iterator-Driven Allocation Architecture** (Approach B):
|
||||
**Iterator-Driven Allocation Architecture with Four-Level Picker Composition** (extended from original Approach B):
|
||||
|
||||
### Core abstractions
|
||||
|
||||
```
|
||||
xce_alloc_plan_iter_t xce_host_iter_t
|
||||
┌─────────────────────┐ ┌──────────────────────┐
|
||||
│ init(resreq, sched) │ │ init(plan, workers) │
|
||||
│ next() → plan │───────▶│ next() → worker │
|
||||
│ fini() │ │ fini() │
|
||||
└─────────────────────┘ └──────────────────────┘
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Level 0: Project Picker [sched->project_pick_plugin] │
|
||||
│ │
|
||||
│ v1 (builtin_flat): yields one "default" project │
|
||||
│ v2 (future): quota-aware project selection │
|
||||
└──────────────────────┬───────────────────────────────────┘
|
||||
│ project scope
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Level 1: Inter-Queue Picker [sched->queue_pick_plugin] │
|
||||
│ │
|
||||
│ Iterates queues scoped to the selected project │
|
||||
│ Default: strict_priority (descending queue->priority) │
|
||||
└──────────────────────┬───────────────────────────────────┘
|
||||
│ queue
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Level 2: Intra-Queue Picker [queue→project→sched] │
|
||||
│ │
|
||||
│ Picks one job from the selected queue │
|
||||
│ Default: builtin_fcfs (priority desc, queue_time asc) │
|
||||
└──────────────────────┬───────────────────────────────────┘
|
||||
│ job
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Allocation │
|
||||
│ │
|
||||
│ xce_alloc_plan_iter_t xce_host_iter_t │
|
||||
│ ┌─────────────────────┐ ┌──────────────────────┐ │
|
||||
│ │ init(resreq, sched) │ │ init(plan, workers) │ │
|
||||
│ │ next() → plan │───────▶│ next() → worker │ │
|
||||
│ │ fini() │ │ fini() │ │
|
||||
│ └─────────────────────┘ └──────────────────────┘ │
|
||||
│ │
|
||||
│ success → dispatch │
|
||||
│ fail → Level 3 (preemption) │
|
||||
└──────────────────────┬──────────────────────────────────┘
|
||||
│ fail
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Level 3: Preemption │
|
||||
│ │
|
||||
│ if queue->preemptible_queues: │
|
||||
│ victims = preempt_policy.select_victims(job) │
|
||||
│ if victims: commit XCE_DECISION_PREEMPT │
|
||||
│ → controller → worker → SIGSTP victims │
|
||||
│ → victims → JOB_SUSPENDED, release resources │
|
||||
│ → pend job for next round │
|
||||
│ else: pend(job, "Resources") │
|
||||
│ else: pend(job, "Resources") │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
- **Plan iterator**: owned by allocator plugin (`parjob_allocator` or `seqjob_allocator`). Generates `(nhosts, slots_per_host, host_filter)` plans lazily.
|
||||
- **Host iterator**: per-plan. Wraps `sched_collect_workers()` + select expression filtering into a lazy interface. Each `next()` returns the next candidate host or NULL.
|
||||
### Picker plugin interface
|
||||
|
||||
### Scheduling loop (modified)
|
||||
|
||||
```
|
||||
for each elected job:
|
||||
plan_iter = job.allocator.make_plan_iter(resreq)
|
||||
while (plan = plan_iter.next()):
|
||||
host_iter = make_host_iter(plan, all_workers)
|
||||
while (host = host_iter.next()):
|
||||
if try_alloc(host, plan) succeeds:
|
||||
dispatch(job, host, plan)
|
||||
goto next_job
|
||||
host_iter.advance()
|
||||
host_iter.fini()
|
||||
plan_iter.fini()
|
||||
pend(job, reason)
|
||||
Each picker level follows the same contract:
|
||||
```c
|
||||
typedef struct {
|
||||
void* (*init)(xce_scheduler_t* sched, void* parent_scope);
|
||||
void* (*next)(void* iter); // returns scope or NULL
|
||||
void (*fini)(void* iter);
|
||||
} picker_iface_t;
|
||||
```
|
||||
|
||||
### Configurable job picker
|
||||
### Unified scheduling loop
|
||||
|
||||
- Add `char* job_pick_plugin` to `xce_scheduler_t` (default `"xce_sched_fcfs"`)
|
||||
- `sched_job_picker()` reads this field instead of hardcoded string
|
||||
- Set via scheduler config or command-line flag
|
||||
- Plugin interface contract: `job_elect_fn` must be cancel-safe (respond to `pthread_cancel`), session-aware (reset state on `end_session`), and communicate via `jbuf` cbuf
|
||||
```c
|
||||
void* sched_allocator(void* arg) {
|
||||
while (1) {
|
||||
// Level 0: project scope
|
||||
scope = project_picker.next(sched);
|
||||
if (!scope) { sleep(1); continue; }
|
||||
|
||||
### Unsatisfiable resreq cache
|
||||
// Level 1: queue (scoped to project)
|
||||
queue = queue_picker.next(sched, scope);
|
||||
if (!queue) { project_picker.advance(); continue; }
|
||||
|
||||
- Per scheduling session (`xce_scheduler_t`), cleared on `begin_session`
|
||||
- Key: hash of `{select_string, rusage_string, user, queue, project}`
|
||||
- Before full allocation: quick lookup. If present → skip to next job immediately
|
||||
- Populated when a job's plan iterator exhausts all plans without success
|
||||
// Level 2: job (within queue)
|
||||
// Precedence: queue->job_pick_plugin → project->job_pick_plugin → sched->job_pick_plugin
|
||||
picker = resolve_job_picker(sched, queue);
|
||||
job = picker->elect_one(queue);
|
||||
|
||||
if (!job) { queue_picker.advance(); continue; }
|
||||
|
||||
// Allocation + preemption
|
||||
if (try_allocate(sched, job)) {
|
||||
dispatch(sched, job);
|
||||
} else if (can_preempt(sched, job)) {
|
||||
preempt_and_pend(sched, job);
|
||||
} else {
|
||||
pend(sched, job, "Resources");
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This loop is identical for v1 and v2. v1 uses `builtin_flat` for Level 0. v2 replaces it with a quota-aware project picker.
|
||||
|
||||
### Preemption flow
|
||||
|
||||
```
|
||||
scheduler controller worker
|
||||
─────────── ────────── ──────
|
||||
preempt_policy.select_victims(J)
|
||||
→ [victim1, victim2]
|
||||
|
||||
XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim
|
||||
.victim_jobids[] deliver to each job → JOB_SUSPENDED
|
||||
.signal = SIGSTP victim's worker release resources
|
||||
.preempting_jobid = J
|
||||
|
||||
J pended → next round → retry alloc
|
||||
```
|
||||
|
||||
V1 uses SIGSTP (terminal stop — process groups paused, job stays in queue). Future escalation to SIGKILL kill-and-requeue is deferred.
|
||||
|
||||
### What stays unchanged
|
||||
- Existing plugin hooks (config, queue, job, worker, state_change, commit, election, prepare_allocation, rusage) — all preserved
|
||||
- `seqjob_allocator` — wrapped with iterator interface, internally still 1host×1slot
|
||||
- `fcfs.c` — continues as default job picker, no changes needed
|
||||
- Select expression system — no changes
|
||||
- COMPOUND and ALTER resreq types — unchanged (PRIMARY focus)
|
||||
- Controller communication (handler.c opcodes, decision cbuf) — unchanged
|
||||
- Controller communication (handler.c opcodes, decision cbuf) — extended, not replaced
|
||||
- `HOOK_DETAIL_JOB_STOP` — reused for preemption stop notifications
|
||||
|
||||
### What's new
|
||||
- `XCE_DECISION_PREEMPT` decision type
|
||||
- `xce_preempt_decision_t` payload
|
||||
- Three builtin plugins: `builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`
|
||||
- `project_name` on queue structs
|
||||
- `job_pick_plugin` on queue + project structs
|
||||
- `preemptible_queues`, `preemptible`, `preempt_policy` on queue struct
|
||||
- `project_pick_plugin`, `queue_pick_plugin`, `job_pick_plugin` on scheduler struct
|
||||
|
||||
## Scope
|
||||
|
||||
### In
|
||||
- Unified picker loop: `project → queue → job` with plugin slots at each level
|
||||
- Three builtin plugins: `builtin_flat` (Level 0), `builtin_strict_priority` (Level 1), `builtin_fcfs` (Level 2)
|
||||
- Picker plugin interface: `init/next/fini` — cancel-safe, session-aware
|
||||
- Picker precedence chain: `queue → project → global`
|
||||
- `xce_alloc_plan_iter_t` interface (header + implementation)
|
||||
- `xce_host_iter_t` interface (header + implementation)
|
||||
- `parjob_allocator` combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhhosts×1)
|
||||
- Configurable job picker: `job_pick_plugin` field + config wiring
|
||||
- Preemption: `XCE_DECISION_PREEMPT` decision type, `xce_preempt_decision_t` payload, LSF-style cross-queue victim selection, SIGSTP mechanism via controller→worker pipeline
|
||||
- Per-queue preemption config: `preemptible_queues`, `preemptible`, `preempt_policy`
|
||||
- Session-level unsatisfiable resreq fingerprint cache
|
||||
- Data structure changes to `xce_scheduler_t`, `xce_sched_queue_t`, `xce_sched_project_t`, `decision.h`
|
||||
- Plugin interface contract documentation (cancel-safe, session-aware)
|
||||
- Unit tests for plan generation, host iteration, cache hit/miss
|
||||
- Integration test: multi-slot parallel job allocation end-to-end
|
||||
- Unit tests for plan generation, host iteration, cache hit/miss, picker resolution, preemption victim selection
|
||||
- Integration test: multi-slot parallel job allocation and preemption flow end-to-end
|
||||
|
||||
### Out
|
||||
- New job selection policies (priority-only, fair-share, preemption, backfill) — framework enables them but they ship separately
|
||||
- New job selection policy implementations beyond the three builtins (priority, fair-share, deadline, preemption, backfill, etc.) — the configurable slots enable them, but none ship in this spec
|
||||
- Hierarchical project resource quotas (Level 0 full) — deferred to `project-resource-ownership` spec
|
||||
- Resreq template registry (named/shared resreq objects) — future enhancement
|
||||
- Select expression language changes
|
||||
- Controller-side scheduling state replication (separate concern: commit barrier wiring, see roadmap hardening item)
|
||||
- Compound/ALTER resreq allocation changes
|
||||
- GPU/affinity-aware allocation (pluggable later via allocator plugins)
|
||||
- Kill-and-requeue preemption escalation (v1 is SIGSTP only, suspend-and-retry)
|
||||
- Deadline field on `xce_sched_job_t` (needed for deadline-based pickers — deferred)
|
||||
|
||||
## Future: Project Resource Ownership
|
||||
|
||||
Deferred to a separate `project-resource-ownership` spec. When implemented:
|
||||
|
||||
- Projects form a tree with resource quota at each node
|
||||
- Level 0 project picker selects which project gets resources (fair-share, priority, proportional)
|
||||
- Level 1 queue picker is scoped to the selected project's queues
|
||||
- Level 2 job picker sees only jobs from the selected project within each queue
|
||||
- Project quota enforcement: projects at or over quota are skipped by Level 0
|
||||
- The unified picker loop (above) requires zero code changes — only a new project picker plugin replaces `builtin_flat`
|
||||
|
||||
The architecture is designed so that v1 (flat project space) and v2 (hierarchical projects) share the identical scheduling loop.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
The work naturally splits into these implementation phases (single spec, sequential):
|
||||
|
||||
1. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction
|
||||
2. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator
|
||||
3. **Configurable job picker** — add `job_pick_plugin` config, document plugin contract
|
||||
4. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop
|
||||
5. **Integration + testing** — end-to-end multi-slot job allocation, cache behavior, config swapping
|
||||
1. **Unified picker loop + builtin plugins** — implement the `project → queue → job` loop with `builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`; retire `sched_job_picker()`; add plugin slots and precedence chain to structs
|
||||
2. **Iterator interfaces** — define `xce_alloc_plan_iter_t` and `xce_host_iter_t`, implement for seqjob path first (trivial wrapper), validate the abstraction
|
||||
3. **Parallel job allocator** — implement combinatorial plan generation in `parjob_allocator`, wire through host iterator
|
||||
4. **Preemption** — implement `XCE_DECISION_PREEMPT` decision type, cross-queue victim selection policy, SIGSTP pipeline (scheduler→controller→worker), queue preemption config fields
|
||||
5. **Unsatisfiable resreq cache** — session-level fingerprint cache, wire into scheduling loop (skip before allocation + preemption attempt)
|
||||
6. **Integration + testing** — end-to-end multi-slot parallel job allocation, preemption flow, picker resolution, cache behavior, HA soak regression
|
||||
|
||||
## Out of Boundary
|
||||
- New job selection policy implementations (priority, fair-share, etc.) — these are new plugins that USE the configurable picker, not part of this spec
|
||||
|
|
|
|||
275
.kiro/specs/scheduler-framework/requirements.md
Normal file
275
.kiro/specs/scheduler-framework/requirements.md
Normal file
|
|
@ -0,0 +1,275 @@
|
|||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
The XCE scheduler has a plugin architecture (`plugin.h`/`plugin.c`, `xce_sched_plugin_t`) but three gaps keep it from being a real framework. First, job selection policy is hardcoded: `sched_job_picker()` (`scheduler.c:39`) sets `job_pick_plugin = "xce_sched_fcfs"` as a literal, so swapping policies requires recompilation. Second, parallel job allocation is a stub: `parjob_allocator.c:73`'s `my_alloc()` takes `(plugin, ctx)` and immediately `return 0` — any job with `maxTasks > 1` (routed there by `sched_make_primary_alloc_()` at `scheduler.c:93-103`) produces no allocation at all. Third, allocation is eager and one-shot: the sibling `seqjob_allocator.c:107` `my_alloc()` (used when `maxTasks == 1`) calls `sched_collect_workers()` then `xlist_find()` for the first host that satisfies the request and stops — there is no mechanism to generate alternative host/slot combinations and retry when the first plan fails.
|
||||
|
||||
This feature closes all three gaps with an iterator-driven allocation architecture: a lazy `xce_alloc_plan_iter_t` (candidate `(nhosts, slots_per_host)` plans, most-preferred first) composed with a lazy `xce_host_iter_t` (candidate hosts for a given plan, filtered by the existing select-expression evaluator), a `parjob_allocator` implementation that walks the plan iterator to do real combinatorial multi-slot allocation, and a session-scoped unsatisfiable-resreq fingerprint cache so a job whose resreq no plan can satisfy doesn't re-pay full allocation cost for every later job with the same fingerprint in the same scheduling session.
|
||||
|
||||
Beyond the three gaps, this feature introduces a **four-level scheduling architecture** that separates concerns cleanly and composes upward:
|
||||
|
||||
| Level | Picker | Question it answers | Plugin slot | Default (v1) |
|
||||
|-------|--------|---------------------|-------------|---------------|
|
||||
| 0 | Project picker | WHO gets resources this round? | `sched->project_pick_plugin` | `"builtin_flat"` — single "default" project |
|
||||
| 1 | Inter-queue picker | WHICH queue within the project? | `sched->queue_pick_plugin` | `"builtin_strict_priority"` |
|
||||
| 2 | Intra-queue picker | WHICH job from this queue? | `queue→project→sched` precedence chain | `"builtin_fcfs"` |
|
||||
| 3 | Preemption policy | WHAT to preempt when allocation fails? | `queue->preempt_policy` | `"lowest_priority_first"` |
|
||||
|
||||
The scheduling loop is always `project → queue → job`, with Level 0 collapsing to a single implicit "default" project in v1 and expanding to hierarchical resource ownership in a future spec (`project-resource-ownership`). Level 3 (preemption) fires only when allocation fails for a preemptible job.
|
||||
|
||||
> Full discovery context (problem, current state, alternatives considered, approach, boundary, upstream/downstream) is in `.kiro/specs/scheduler-framework/brief.md`.
|
||||
|
||||
## Queue vs Project — Roles
|
||||
|
||||
These are fundamentally different concerns:
|
||||
|
||||
| | Queue | Project |
|
||||
|---|---|---|
|
||||
| **Role** | Scheduling mechanism | Resource ownership |
|
||||
| **Question** | HOW are jobs ordered and constrained? | WHO owns resources and gets charged? |
|
||||
| **Scope** | One hardware/scheduling profile | Cross-queue, cross-cluster |
|
||||
| **Key attributes** | Priority, limits, time windows, picker, preemption config | Quota, usage, hierarchy, members (future) |
|
||||
| **Cardinality to job** | 1 job → 1 queue | 1 job → 1 project |
|
||||
| **Cardinality to each other** | Many-to-many | Many-to-many |
|
||||
|
||||
- **Queue** = submission channel with scheduling policy. Owns resource thresholds (RUNLIMIT, MAX_RUNNING), time windows, inter-queue priority, and the intra-queue job picker.
|
||||
- **Project** = resource ownership boundary. Owns quota, consumption tracking, hierarchical nesting, and accounting. One project submits to many queues; one queue receives from many projects.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: Unified picker loop (`project → queue → job`) with plugin slots at each level; `xce_alloc_plan_iter_t` and `xce_host_iter_t` interfaces (lazy, cancel-safe, composable); `seqjob_allocator` rewrapped onto the iterator interface as a trivial 1-plan/1-host-loop case (behavior unchanged, `maxTasks == 1` only); `parjob_allocator`'s `my_alloc()` implementing real combinatorial plan generation (1host×N, 2hosts×⌈N/2⌉, …, Nhosts×1) for `maxTasks > 1`; three builtin plugins (`builtin_flat`, `builtin_strict_priority`, `builtin_fcfs`) replicating today's behavior through the new architecture; configurable picker slots with precedence chain (`queue → project → global`); preemption decision type (`XCE_DECISION_PREEMPT`) flowing scheduler→controller→worker with SIGSTP mechanism; cross-queue victim selection (LSF-style per-queue `preemptible_queues` list); session-level unsatisfiable-resreq fingerprint cache keyed on `{select, rusage, user, queue, project}`; documentation of plugin interface contracts (cancel-safe, session-aware); unit tests for plan generation, host iteration, cache hit/miss, picker resolution, preemption victim selection; integration test for end-to-end multi-slot parallel job allocation and preemption flow.
|
||||
|
||||
- **Out of scope**: New job selection policies beyond the three builtins (priority, fair-share, deadline, preemption, backfill) — the configurable slots enable them, but none ship in this spec beyond the builtins; a resreq template registry (named/shared resreq objects); changes to the select-expression language or evaluator; controller-side scheduling-state replication / commit-barrier wiring for scheduling decisions (separate roadmap hardening item, tracked adjacent to `ha-consistency-invariants`); `COMPOUND`/`ALTER` resreq allocation changes beyond what composing over `PRIMARY` already provides; GPU/topology/affinity-aware allocation (left pluggable via future allocator plugins); hierarchical project resource quotas (Level 0 full — deferred to `project-resource-ownership` spec); kill-and-requeue preemption escalation (deferred — v1 is SIGSTP only); deadline field on `xce_sched_job_t` (deferred — needed for deadline-based pickers).
|
||||
|
||||
- **Adjacent expectations**: depends on the existing plugin framework (`plugin.h`/`plugin.c`), the existing select-expression evaluator (`resreq.h`, memoized via `select_eval_ctx_t`), and existing shared-resource tracking (`xce_scheduler_t.sharedInst`/`sharedInUse`) — none of these change. `scheduler-controller-msg-gap` (implemented) governs controller↔scheduler message handling this feature reads state through but does not modify. `ha-consistency-invariants`' commit barrier is a future consumer of scheduling decisions; this feature does not wire it. Controller hook `HOOK_DETAIL_JOB_STOP` (enum value 65 in `cluster_def.h`) already exists and is reused for preemption stop actions.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Unified picker loop (project → queue → job)
|
||||
|
||||
**Objective:** As a maintainer of the scheduler, I want a single scheduling loop that composes project, queue, and job pickers in a fixed order, so that v1 (flat project space) and v2 (hierarchical projects with resource quotas) share the same code path.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The Scheduler shall implement a scheduling loop that always follows the order: Level 0 (project picker) → Level 1 (inter-queue picker) → Level 2 (intra-queue picker) → allocation.
|
||||
2. Each picker level shall conform to a common interface contract: `init(sched, parent_scope)`, `next(iter) → scope_or_NULL`, `fini(iter)`.
|
||||
3. When a picker at any level returns NULL (exhausted), the loop shall advance the parent picker to its next scope and restart the child picker from that new scope.
|
||||
4. In v1 (no project hierarchy), the default project picker `builtin_flat` shall yield exactly one scope: a "default" project containing all configured queues, so the loop behavior is identical to today's flat queue iteration.
|
||||
5. Plugin resolution for Level 2 shall follow the precedence chain: `queue->job_pick_plugin` → `project->job_pick_plugin` → `sched->job_pick_plugin` → hardcoded default `"builtin_fcfs"`.
|
||||
6. `sched_job_picker()` shall be retired and replaced by the unified loop; the hardcoded `"xce_sched_fcfs"` literal at `scheduler.c:43` shall be removed.
|
||||
|
||||
### Requirement 2: Lazy allocation plan iterator
|
||||
|
||||
**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy plan-generation abstraction, so that every allocator (current and future) can propose alternative host/slot combinations without materializing the full combinatorial space up front.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The Scheduler shall define an `xce_alloc_plan_iter_t` interface with `init(resreq, sched)`, `next() → plan`, and `fini()` operations.
|
||||
2. Each call to `next()` shall generate exactly one candidate `(nhosts, slots_per_host, host_filter)` plan without generating or materializing any other plan in the same call.
|
||||
3. The Scheduler shall order plans returned by successive `next()` calls from most-preferred to least-preferred, per the owning allocator's preference policy.
|
||||
4. `next()` shall return an end-of-iteration signal (not a plan) once every candidate plan has been produced, and this signal shall be stable across repeated calls after exhaustion.
|
||||
5. Calling `fini()` shall release all resources held by the iterator, and the interface shall define no other required cleanup step.
|
||||
|
||||
### Requirement 3: Lazy host candidate iterator
|
||||
|
||||
**Objective:** As a maintainer of the scheduler's allocator plugins, I want a reusable lazy host-candidate abstraction per allocation plan, so that host filtering reuses the existing select-expression evaluator instead of each allocator reimplementing worker collection and filtering.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The Scheduler shall define an `xce_host_iter_t` interface with `init(plan, workers)`, `next() → worker`, and `fini()` operations.
|
||||
2. Each call to `next()` shall return the next candidate host satisfying the given plan's host filter (select expression plus resource availability), evaluated against the existing select-expression evaluator without changes to that evaluator.
|
||||
3. `next()` shall return NULL once no further candidate host exists for the current plan, and the caller shall not need to call `next()` again after receiving NULL for that plan.
|
||||
4. The host iterator shall compose with the plan iterator: a new `xce_host_iter_t` shall be constructible from any plan produced by Requirement 2's `xce_alloc_plan_iter_t.next()`.
|
||||
5. Calling `fini()` shall release all resources held by the host iterator, independent of the plan iterator's lifecycle.
|
||||
|
||||
### Requirement 4: Parallel job combinatorial allocation
|
||||
|
||||
**Objective:** As a cluster operator submitting a job that needs multiple task slots, I want `parjob_allocator` to actually allocate across hosts, so that jobs with `maxTasks > 1` are no longer silently unallocatable.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. When a job's `xce_primary_resreq_t.maxTasks > 1` reaches `parjob_allocator`'s `job_alloc_fn` (`my_alloc()` in `parjob_allocator.c`), the Scheduler shall generate candidate plans covering at minimum the sequence 1 host × N slots, 2 hosts × ⌈N/2⌉ slots, …, N hosts × 1 slot, using Requirement 2's plan iterator.
|
||||
2. For each candidate plan, the Scheduler shall use Requirement 3's host iterator to collect hosts with enough available slots to satisfy that plan, and shall attempt allocation before advancing to the next plan.
|
||||
3. When a plan's host collection satisfies the plan's slot requirement, the Scheduler shall commit that allocation and shall not evaluate any remaining plan for that job.
|
||||
4. When no plan among those generated in Criterion 1 can be satisfied, `my_alloc()` shall return the same not-satisfied signal (`0`) that the current stub returns today, so pend-reason handling in `sched_make_primary_alloc_()` is unaffected.
|
||||
5. `parjob_allocator`'s `job_alloc_fn` shall no longer unconditionally `return 0` regardless of `ctx` contents — the current stub behavior — for any job with `maxTasks > 1`.
|
||||
|
||||
### Requirement 5: Configurable picker plugin slots
|
||||
|
||||
**Objective:** As a cluster operator, I want to configure job-picking policies at the scheduler, queue, and project levels without recompiling the scheduler, so that FCFS, priority, or custom pickers can be swapped operationally and composed.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The Scheduler shall add a `project_pick_plugin` field to `xce_scheduler_t`, defaulting to `"builtin_flat"` when unset.
|
||||
2. The Scheduler shall add a `queue_pick_plugin` field to `xce_scheduler_t`, defaulting to `"builtin_strict_priority"` when unset.
|
||||
3. The Scheduler shall add a `job_pick_plugin` field to `xce_scheduler_t`, defaulting to `"builtin_fcfs"` when unset — serving as the global fallback.
|
||||
4. The Scheduler shall add a `job_pick_plugin` field to `xce_sched_queue_t`, defaulting to NULL (inherit from project or global).
|
||||
5. The Scheduler shall add a `job_pick_plugin` field to `xce_sched_project_t`, defaulting to NULL (inherit from global).
|
||||
6. The Scheduler shall add a `project_name` field to `xce_sched_queue_t`, set from queue configuration, establishing the queue→project relationship for Level 1 scoping.
|
||||
7. The Scheduler shall accept all picker plugin names from scheduler configuration (or equivalent operator-facing settings), applied at scheduler startup without requiring a rebuild.
|
||||
8. When a plugin slot names a plugin not registered via the existing `plugin_register`/`key_find` mechanism, the Scheduler shall fail with a clear configuration error rather than silently falling back to the builtin default.
|
||||
9. The Scheduler shall document the picker plugin interface contract: each picker must be cancel-safe (responds to `pthread_cancel` at defined points) and session-aware (resets its state on `end_session`).
|
||||
|
||||
### Requirement 6: Preemption policy and execution
|
||||
|
||||
**Objective:** As a cluster operator, I want high-priority jobs to trigger preemption of lower-priority running jobs when resources are insufficient, so that critical work can proceed without manual intervention.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The Scheduler shall add a `XCE_DECISION_PREEMPT` opcode to `xce_decision_op_t` in `decision.h`, with a payload type `xce_preempt_decision_t` carrying `victim_jobids[]`, `n_victims`, `signal` (SIGSTP in v1), and `preempting_jobid`.
|
||||
2. When allocation fails for a job J whose queue has a non-NULL `preemptible_queues` list, the Scheduler shall invoke the queue's `preempt_policy` plugin to select victim jobs from the target queues.
|
||||
3. The default preemption policy `"lowest_priority_first"` shall: collect running jobs from target queues, filter to those with `priority < J.priority` and `queue->preemptible == true`, rank by priority ascending, and select the minimum subset whose freed resources satisfy J's resource request.
|
||||
4. When a sufficient victim set exists, the Scheduler shall commit the preemption decision via `sched_commit_decision(XCE_DECISION_PREEMPT)`, publish it through the controller→worker pipeline, and pend job J for the next scheduling round (async preemption — the job is picked up after resources are freed).
|
||||
5. When no sufficient victim set exists, the Scheduler shall pend job J with a resource-shortage reason without invoking preemption.
|
||||
6. The worker receiving a preemption action shall send the specified signal (SIGSTP) to the victim job's process groups, and the controller shall transition the victim job's state to `JOB_SUSPENDED`.
|
||||
7. The Scheduler shall add `preemptible_queues` (list of queue names this queue CAN preempt from), `preemptible` (bool — CAN this queue's jobs BE preempted by others), and `preempt_policy` (victim-selection plugin name, NULL = default) fields to `xce_sched_queue_t`.
|
||||
8. The existing `HOOK_DETAIL_JOB_STOP` hook detail enum shall be reused for preemption stop notifications.
|
||||
9. Preemption victim selection shall never select a job whose priority is greater than or equal to the preempting job's priority, nor a job whose queue has `preemptible == false`.
|
||||
|
||||
### Requirement 7: Session-level unsatisfiable resreq cache
|
||||
|
||||
**Objective:** As a maintainer of scheduling-loop performance, I want jobs with a resreq the current session already proved unsatisfiable to skip allocation immediately, so that a batch of similarly-shaped unsatisfiable jobs doesn't each re-pay full plan/host iteration cost.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. The Scheduler shall maintain a per-session unsatisfiable-resreq fingerprint cache on `xce_scheduler_t`, keyed by a fingerprint of `{select_string, rusage_string, user, queue, project}`.
|
||||
2. The Scheduler shall clear the unsatisfiable-resreq cache at the start of every scheduling session (`begin_session`).
|
||||
3. When a job's primary resreq's plan iterator (Requirement 2) is exhausted without a successful allocation AND preemption (Requirement 6) also fails or is inapplicable, the Scheduler shall populate the cache with that resreq's fingerprint before pending the job.
|
||||
4. Before starting full allocation for a job, the Scheduler shall look up the job's resreq fingerprint in the cache; on a hit, it shall pend the job immediately without invoking the plan or host iterators or preemption.
|
||||
5. A cache hit shall attribute the same pend reason category as the original unsatisfiable determination, so operators are not misled about why a later job was skipped.
|
||||
|
||||
### Requirement 8: Builtin default plugins
|
||||
|
||||
**Objective:** As a maintainer, I want the new architecture to ship with builtin plugins that replicate today's exact scheduling behavior, so that operators who don't configure custom pickers see no behavioral change.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. **`builtin_flat`** (Level 0, project picker): shall yield exactly one "default" project scope per scheduling round. All queues not explicitly assigned to a project belong to this default project. Always eligible; never exhausted during a session.
|
||||
2. **`builtin_strict_priority`** (Level 1, inter-queue picker): shall iterate queues in descending priority order (`queue->priority`). Shall skip queues with no pending jobs (`sched_queue_is_ready` returns false). When scoped to a project, shall only iterate queues belonging to that project. Shall return NULL when all eligible queues are exhausted.
|
||||
3. **`builtin_fcfs`** (Level 2, intra-queue picker): shall replicate today's FCFS behavior — iterate pending jobs within the queue by `priority desc, queue_time asc`. Shall skip jobs where `sched_job_is_ready` returns false. Shall communicate the picked jobid via the existing `jbuf` cbuf mechanism. Shall be cancel-safe and session-aware.
|
||||
4. All three builtin plugins shall be registered automatically at scheduler startup without requiring explicit configuration.
|
||||
|
||||
### Requirement 9: No regression to existing allocators, job picker, or HA soak
|
||||
|
||||
**Objective:** As a maintainer, I want the iterator-driven rework and picker architecture scoped so that existing single-slot allocation and FCFS behavior are unaffected, so that this framework change is safe to ship independently of any new policy or allocator.
|
||||
|
||||
#### Acceptance Criteria
|
||||
1. `seqjob_allocator`'s externally observable allocation behavior (which host a `maxTasks == 1` job lands on, given the same worker/resource state) shall be unchanged after it is rewrapped onto the `xce_host_iter_t` interface.
|
||||
2. `builtin_fcfs` shall produce the same job-picking order as today's `fcfs_elect_job`, given the same queue and job state.
|
||||
3. `builtin_strict_priority` shall produce the same inter-queue order as today's `fcfs_elect_job` queue iteration loop.
|
||||
4. `builtin_flat` shall produce the same effective behavior as today's lack of project scoping — all queues in one flat namespace.
|
||||
5. The `COMPOUND` and `ALTER` resreq allocation paths (`sched_make_compound_allocation`, `sched_make_alter_allocation` in `scheduler.c`) shall remain unchanged, continuing to call into the per-primary allocation path (now iterator-driven) exactly as they do today.
|
||||
6. Existing scheduler unit tests (`scheduler` meson test suite) shall continue to pass unmodified in assertions, adjusted only where they directly exercised the removed hardcoded-string or eager-one-shot behavior this feature replaces.
|
||||
7. A full HA soak (sequential and chaotic `/xce-test-ha` regression) shall show no new job loss, divergence, or liveness stall attributable to this feature, confirming the scheduler-side change does not perturb controller-side commit/replication behavior it reads state from.
|
||||
|
||||
## Data Structure Changes Summary
|
||||
|
||||
### `xce_scheduler_t` additions
|
||||
```c
|
||||
char* project_pick_plugin; // default: "builtin_flat"
|
||||
char* queue_pick_plugin; // default: "builtin_strict_priority"
|
||||
char* job_pick_plugin; // default: "builtin_fcfs" (global fallback)
|
||||
// unsatisfiable resreq cache (Req 7)
|
||||
```
|
||||
|
||||
### `xce_sched_queue_t` additions
|
||||
```c
|
||||
char* project_name; // owning project (for Level 1 scoping)
|
||||
char* job_pick_plugin; // intra-queue picker (NULL = inherit)
|
||||
char** preemptible_queues; // queues this queue CAN preempt from
|
||||
int n_preemptible_queues;
|
||||
bool preemptible; // CAN this queue's jobs BE preempted?
|
||||
char* preempt_policy; // victim selection plugin (NULL = default)
|
||||
```
|
||||
|
||||
### `xce_sched_project_t` additions
|
||||
```c
|
||||
char* job_pick_plugin; // per-project picker (NULL = inherit, v1 field, v2 logic)
|
||||
int32_t priority; // inter-project ordering (future spec)
|
||||
// future: resource_quota, resource_used, parent_project (project-resource-ownership spec)
|
||||
```
|
||||
|
||||
### `decision.h` additions
|
||||
```c
|
||||
typedef enum {
|
||||
XCE_DECISION_NONE = 0,
|
||||
XCE_DECISION_DISPATCH,
|
||||
XCE_DECISION_PEND,
|
||||
XCE_DECISION_PREEMPT, // NEW
|
||||
XCE_DECISION_NEW_SESSION
|
||||
} xce_decision_op_t;
|
||||
|
||||
typedef struct {
|
||||
uint64_t* victim_jobids; // jobs to preempt
|
||||
int n_victims;
|
||||
int signal; // SIGSTP (v1), SIGKILL (future)
|
||||
uint64_t preempting_jobid; // job that triggered preemption
|
||||
} xce_preempt_decision_t;
|
||||
```
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Level 0: Project Picker [sched->project_pick_plugin] │
|
||||
│ │
|
||||
│ v1 (builtin_flat): yields one "default" project │
|
||||
│ v2 (future): quota-aware project selection │
|
||||
└──────────────────────┬───────────────────────────────────┘
|
||||
│ project scope
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Level 1: Inter-Queue Picker [sched->queue_pick_plugin] │
|
||||
│ │
|
||||
│ Iterates queues scoped to the selected project │
|
||||
│ Default: strict_priority (descending queue->priority) │
|
||||
└──────────────────────┬───────────────────────────────────┘
|
||||
│ queue
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Level 2: Intra-Queue Picker [queue→project→sched] │
|
||||
│ │
|
||||
│ Picks one job from the selected queue │
|
||||
│ Default: builtin_fcfs (priority desc, queue_time asc) │
|
||||
└──────────────────────┬───────────────────────────────────┘
|
||||
│ job
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Allocation + Preemption │
|
||||
│ │
|
||||
│ plan_iter → host_iter → try_alloc │
|
||||
│ success → dispatch │
|
||||
│ fail → Level 3: preempt? → select victims → SIGSTP │
|
||||
│ → pend (no preemption possible) │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Preemption Flow
|
||||
|
||||
```
|
||||
scheduler controller worker
|
||||
─────────── ────────── ──────
|
||||
alloc fails for high-prio job J
|
||||
|
||||
preempt_policy.select_victims(J):
|
||||
collect running jobs from
|
||||
Q_aggressor.preemptible_queues
|
||||
filter: victim.prio < J.prio
|
||||
filter: victim.queue.preemptible
|
||||
rank: lowest priority first
|
||||
select minimum set satisfying J.resreq
|
||||
|
||||
→ [victim1, victim2]
|
||||
|
||||
XCE_DECISION_PREEMPT ────────────▶ handle_job_preempt() ──────▶ SIGSTP victim1
|
||||
.victim_jobids = [v1, v2] deliver to each job → JOB_SUSPENDED
|
||||
.signal = SIGSTP victim's worker release resources
|
||||
.preempting_jobid = J
|
||||
|
||||
J pended → next scheduling round → allocation retries on freed resources
|
||||
```
|
||||
|
||||
## Future: Project Resource Ownership (Level 0 full)
|
||||
|
||||
Deferred to a separate `project-resource-ownership` spec. When implemented:
|
||||
|
||||
- Projects form a tree with resource quota at each node
|
||||
- Level 0 project picker selects which project gets resources (fair-share, priority, proportional)
|
||||
- Level 1 queue picker is scoped to the selected project's queues
|
||||
- Level 2 job picker sees only jobs from the selected project within each queue
|
||||
- Project quota enforcement: projects over quota are skipped by Level 0
|
||||
- The unified picker loop (Requirement 1) requires zero code changes — only a new project picker plugin replaces `builtin_flat`
|
||||
|
||||
The architecture is designed so that v1 (flat) and v2 (hierarchical) share the identical scheduling loop. The only difference is which project picker plugin is configured.
|
||||
23
.kiro/specs/scheduler-framework/spec.json
Normal file
23
.kiro/specs/scheduler-framework/spec.json
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"feature_name": "scheduler-framework",
|
||||
"created_at": "2026-07-09T21:20:00Z",
|
||||
"updated_at": "2026-07-10T10:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "requirements-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"blocked_on": []
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue