lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

merge: sync Rongsong's scheduler-framework (88 commits, f01ae353..c5b172c7)

Browse source 
Two-sided divergence from f01ae353 (2026-08-30): theirs +88 (worker
rejoin-resync reconciliation, terminal-state bug-1737/1738/1739/1764,
scheduler-sim verbosity, hermetic offline build, go mod vendor of Go deps
in apiserver -- 3375 pure-adds), ours +89 (job-finish robustness bug-1509
B1/B3, PEND-reason, worker reconnect resend, apiserver job control).

Conflict resolutions (kept both sides' fixes):
- controller_connector.c checking_connection(): combine theirs
  xcecd_resync_is_pending() rejoin deferral gate with ours
  resend_job_status() actual-state re-report on rejoin success.
- job.c: keep both unknown-finish registry (ours, bug-1509) and
  record_job_finish_fact_at (theirs, bug-1747); keep both
  mark_jobs_unknown_core family (ours) and record_job_reconcile_*
  (theirs). No double-notify: on_job_finish unchanged, theirs is
  resync-path-only.
- scheduler handler.c/test_handlers.c/meson: theirs schedExitJob
  exit_code branch (bug-1737) + ours jobDispatchFail success-ack no-op
  (bug-1512) + configpush enq_pend_update (bug-1513) both present; both
  test suites and REPL/sim targets wired.
- apiserver jobs.go: keep ours normalizeJobInt64/jobArrayIdx superset +
  JobIDHuman; drop theirs strict IntOrInt64 early-reject (superseded --
  ours tolerates float64/json.Number, no panic path).
- jobs_test.go: keep both regression suites (ours bug-1490, theirs
  bug-1742).
- .kiro/specs scheduler-sim-*/worker-rejoin-resync + .wolf/*: keep our
  deletions (established convention -- drop Rongsong bookkeeping on merge).

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-09-03 14:11:43 +00:00
commit 5e2985542f
3469 changed files with 1466599 additions and 382 deletions
Download patch file Download diff file Expand all files Collapse all files

230
.kiro/steering/resource-accounting.md Normal file
View file

@ -0,0 +1,230 @@
# Resource Accounting: Reservation vs. Actual Usage for Real Resources
## The distinction this document is about
XCE's schedulable resources split into two kinds, and they must be accounted
differently:
- **Virtual resources** (e.g. `slots`) are pure allocation counts. A slot is
either allocated to a job or it isn't — there is no OS ground truth for
"how much of a slot is actually being used." Reservation and usage are the
same number, always.
- **Real resources** (e.g. `mem`) have an OS-measurable actual value that
moves independently of what a job asked for. A job's `rusage[mem=4M]`
request is an *estimate* of the memory it will need over its lifetime,
supplied at submit time, before the job has run — it is not a hard
guarantee, and the real usage the worker later reports may come in above
or below it.
This document covers the accounting policy for the second kind, using `mem`
as the running example. The same policy applies to any resource that is
both `XCE_RESF_DYNAMIC` and `XCE_RESF_CONSUMABLE` (`types.h`) with a
worker-reported instantaneous (gauge-style) value — not to `slots`, and not
to a cumulative counter like `cpu_time` (a usage metric reported for
accounting/billing, not a capacity resource the allocator schedules
against).
**`mem` is not `maxmem`.** These are two distinct, similarly-named
per-host resources, and it is easy to reach for the wrong one:
- `maxmem` is a **static** value, self-reported once by the worker at
join time and never refreshed — the host's total installed physical
memory.
- `mem` is a **dynamic load index** (`resource_sync.c`'s
`resource_sync_register_load_index("mem", ...)`, refreshed on the
worker's regular load-report cadence, same family as `r1m`/`r15m`/`swp`).
Its value is `node_info_get_mem()` -> `read_memory_info()`'s `mem_free`
— the OS's own currently-available-physical-memory reading, in bytes.
This whole document is about `mem`, never `maxmem`. Admission control cares
about what's free *right now*, not the host's total capacity.
## How admission control actually reads "available" (read this before the formula)
The scheduler's admission check does **not** query the OS live at
decision time — deliberately, not as an oversight. All candidate-host
resource information is injected into the scheduler once per session
(pushed by the worker, cached in `w->resvals`/`w->resInUse`); a single
scheduling session then evaluates many pending jobs against many
candidate hosts using that one cached snapshot, without re-querying
between decisions. Querying live on every candidate-host check a session
performs would multiply that already-high decision volume by a live I/O
round-trip each time — the cost this design avoids. `sched_check_host_
satisfy_()` (`resource.c`) reads two already-in-memory arrays and hands
them to `unsatisfy_()`:
```c
total = get_resvals(&(w->resvals), ...); // the worker's last-reported "mem" value
inuse = get_resvals(&(w->resInUse), ...); // THIS scheduler's own running charge tally
...
if (total[i] < inuse[i] + uncomitted[i] + wanted[i]) { /* can't admit */ }
```
`total` for `mem` is whatever the worker last pushed (`mem_free` as of that
report — already a live OS reading, just not read synchronously at
decision time). `inuse` is built *entirely* from this scheduler's own
bookkeeping: `sched_account_jobs()` re-derives it every scheduling round by
summing `used + reserved` (`sched_update_res_accounts()`) across every
`xce_res_account_t` belonging to every currently-`RUN` job, for the job's
**entire lifetime** — not just until the worker's next report catches up.
This is the same mechanism `slots` has always used; `mem` uses it too, it
just also happens to have a `total` that independently drifts on its own
schedule.
**Available memory for scheduling, precisely:**
```
available = w->resvals["mem"] (worker's last-reported mem_free)
- w->resInUse["mem"] (this scheduler's own running charge tally)
```
where `w->resInUse["mem"]` is the sum, over every job on that host, of
`max(reserved_j, used_j)` (see the policy below for how each job's term is
computed).
**Known, pre-existing property worth naming, not a bug this document is
introducing:** because `mem`'s own worker-reported figure already reflects
real OS-level consumption from every process on the host (dispatched by
this scheduler or not), and the scheduler's reservation charge is a
*separate, additive* layer that isn't cleared when a fresh `mem` report
arrives (`core_apply_worker_resvals()`/`RESVALS_DELTA`, `core_events.c`,
overwrites `w->resvals` wholesale but never touches `w->resInUse`), a
running job's footprint can end up reflected twice — once because the
worker's next `mem` report is already lower, and again because this
scheduler's own charge for that job hasn't gone anywhere. This makes
admission control *more conservative* over time (under-utilization risk),
never less (it cannot cause over-subscription) — the same direction this
codebase already accepts for `slots`, which has no live ground truth to
reconcile against at all. Untangling that fully (e.g. reconciling
`resInUse` against a fresh worker push instead of letting both layers
accumulate independently) is a separate, larger architectural question,
out of scope here.
## The policy
1. **Request.** A job may submit `rusage[mem=4M]`. If it doesn't, treat the
reservation as 0 — the job made no promise about memory, but see step 4:
a reservation of 0 does **not** mean nothing gets charged.
2. **Admission.** Before dispatching to a candidate host, the scheduler
checks the host has at least 4MB of *available* `mem` per the formula
above.
3. **Reservation charge.** Once dispatched, 4MB is charged against the
host's in-use memory — this is the reservation, and it holds regardless
of what the job actually ends up using, until real usage data says
otherwise.
4. **Reconciliation against real usage.** Once the worker starts reporting
the job's actual memory usage (rusage ticks), for **every** job on the
host — including one that requested no `rusage[mem=...]` at all, whose
reservation is 0:
- **actual < reserved:** keep charging the full reservation (0 for an
unreserved job means: charge nothing extra beyond its actual usage,
which is exactly what the max() below produces). The estimate might
have been conservative and the job may still grow into it later — the
unused slack must not be handed to another job on the strength of a
snapshot that could be stale a moment later.
- **actual >= reserved:** charge the real usage instead. For a job that
never reserved anything, this is *always* this branch (reserved = 0,
actual >= 0 always) — its full real consumption must be charged,
because nothing else in this codebase's admission path (see above —
there is no live OS read at decision time) will ever account for it
otherwise. Skipping an unreserved job's real usage would leave the
scheduler's own `inuse` tally blind to memory a live process is
genuinely sitting on — an actual over-subscription risk, not merely a
display gap.
In short: **charged = max(reserved, actual)** — for every job, reserved
or not.
## Existing infrastructure this policy builds on
The account-level data model already had almost exactly the right shape
for this — it just wasn't wired up to real usage data, and didn't create
an account entry at all for a job that made no reservation:
- `xce_res_account_t` (`types.h`) carries three fields per job-resource
pairing: `reserved`, `allocated` (both set to the job's request at
dispatch time, `make_res_account()`, `xcealloc.c`), and `used`.
- `sched_update_res_accounts()` (`resource.c`) sums
`accounts[i]->used + accounts[i]->reserved` as the amount charged into
`w->resInUse`/`core->sharedInUse`.
Setting `used = max(0, actual - reserved)` (the *overage* beyond the
reservation, not the raw actual value) makes that existing sum come out
exactly right, with no change needed to the charge formula itself:
```
used + reserved = reserved + max(0, actual - reserved) = max(reserved, actual)
```
For a job with no reservation at all (`reserved = 0`), there is no existing
account entry to update in the first place — `make_res_account()` is only
ever called for resources the job's own `resreq` specified. The
implementation below creates one on first use rather than skipping.
## Implementation (`resource.c`, `schedobj.c`)
Implemented, not just designed:
- **`ensure_host_account_by_resname()`** — finds a job's existing
`xce_res_account_t` for a resname, or creates a fresh `reserved = 0`
entry (resolving `residx` via `key_find()` against `core->resmgr`, same
pattern `sched_update_res_accounts()` already uses for a fresh account)
if the job never requested that resource. Returns `NULL` only if the
resource isn't registered in the cluster's config at all — nothing to
charge, not an error.
- **`sched_reconcile_host_resource_usage()`** — the core formula: sets one
`xce_res_account_t.used` to `max(0, actual - reserved)` and applies just
the resulting **delta** to the worker's `resInUse` (never a full
uncharge/recharge of the whole job account, which would needlessly
re-touch sibling resources like `slots` charged in the same account).
- **`sched_reconcile_job_resource_usage()`** — the wrapper driven from a
raw rusage report (`xce_job_rusages_t`, the worker's own wire shape,
forwarded verbatim to the scheduler by `sched_job_rusage_update_fun()`,
`controller/sched.c`). Maps the worker's real wire resname
(`"memory_rss"` — same worker-resname-vs-scheduler-resource-name split
bug-1735 already established for the controller side) to the scheduler's
`"mem"` resource. Scoped to **single-host jobs only**: the wire report
carries no host field at all (`update_job_rusages()`, `controller/job.c`,
derives the reporting host locally from the endpoint the request arrived
on and never attaches it to the forwarded struct), so there is no way to
attribute a report to one specific host of a multi-host job today. That
is a separate, undocumented-until-now wire-protocol gap — a multi-host
job's memory stays charged at its flat per-host reservation, exactly as
it always has; this feature does not make that worse, and does not fix
it either.
- Wired into **`sched_apply_job_rusage()`** (`schedobj.c`), which already
received every job rusage tick but previously only stored it for
display/plugin hooks with zero resource-accounting effect.
Compatible with `sched_account_jobs()`'s existing per-round `resInUse`
reset-and-re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation reads
from directly — not a transient pool-level delta a round boundary would
wipe.
## Why estimates in either direction are expected, not exceptional
`rusage[mem=...]` is supplied before the job has run — it is a best-effort
prediction of peak lifetime memory, not a measurement. Both directions of
error are routine, not edge cases to special-case away:
- **Over-estimated** (the common conservative-user case): the job never
grows into the memory it asked for. The reservation still protects the
host from another job being packed into that never-realized headroom —
a job that legitimately reserved memory it turns out not to need for
most of its run can still burst into it later without being starved by a
neighbor that was admitted on the strength of a low-water-mark reading.
- **Under-estimated**: the job genuinely needs more than it asked for (or
never asked for anything at all). Reconciliation keeps admission control
honest about what the host is really carrying once that happens, rather
than continuing to trust a promise the job has already broken — or, for
an unreserved job, rather than trusting no promise at all as if that
meant no consumption.
Neither case is a failure mode to reject or warn about at the accounting
layer — the reservation is advisory input to admission control, not a hard
cap enforced against the job itself (cgroup-level enforcement, if desired,
is a separate concern from scheduling admission).