myxce/.kiro/steering/resource-accounting.md

# Resource Accounting: Reservation vs. Actual Usage for Real Resources

## The distinction this document is about

XCE's schedulable resources split into two kinds, and they must be accounted
differently:

- **Virtual resources** (e.g. `slots`) are pure allocation counts. A slot is
  either allocated to a job or it isn't — there is no OS ground truth for
  "how much of a slot is actually being used." Reservation and usage are the
  same number, always.
- **Real resources** (e.g. `mem`) have an OS-measurable actual value that
  moves independently of what a job asked for. A job's `rusage[mem=4M]`
  request is an *estimate* of the memory it will need over its lifetime,
  supplied at submit time, before the job has run — it is not a hard
  guarantee, and the real usage the worker later reports may come in above
  or below it.

This document covers the accounting policy for the second kind, using `mem`
as the running example. The same policy applies to any resource that is
both `XCE_RESF_DYNAMIC` and `XCE_RESF_CONSUMABLE` (`types.h`) with a
worker-reported instantaneous (gauge-style) value — not to `slots`, and not
to a cumulative counter like `cpu_time` (a usage metric reported for
accounting/billing, not a capacity resource the allocator schedules
against).

**`mem` is not `maxmem`.** These are two distinct, similarly-named
per-host resources, and it is easy to reach for the wrong one:

- `maxmem` is a **static** value, self-reported once by the worker at
  join time and never refreshed — the host's total installed physical
  memory.
- `mem` is a **dynamic load index** (`resource_sync.c`'s
  `resource_sync_register_load_index("mem", ...)`, refreshed on the
  worker's regular load-report cadence, same family as `r1m`/`r15m`/`swp`).
  Its value is `node_info_get_mem()` -> `read_memory_info()`'s `mem_free`
  — the OS's own currently-available-physical-memory reading, in bytes.

This whole document is about `mem`, never `maxmem`. Admission control cares
about what's free *right now*, not the host's total capacity.

## How admission control actually reads "available" (read this before the formula)

The scheduler's admission check does **not** query the OS live at
decision time — deliberately, not as an oversight. All candidate-host
resource information is injected into the scheduler once per session
(pushed by the worker, cached in `w->resvals`/`w->resInUse`); a single
scheduling session then evaluates many pending jobs against many
candidate hosts using that one cached snapshot, without re-querying
between decisions. Querying live on every candidate-host check a session
performs would multiply that already-high decision volume by a live I/O
round-trip each time — the cost this design avoids. `sched_check_host_
satisfy_()` (`resource.c`) reads two already-in-memory arrays and hands
them to `unsatisfy_()`:

```c
total = get_resvals(&(w->resvals), ...);    // the worker's last-reported "mem" value
inuse = get_resvals(&(w->resInUse), ...);   // THIS scheduler's own running charge tally
...
if (total[i] < inuse[i] + uncomitted[i] + wanted[i]) { /* can't admit */ }
```

`total` for `mem` is whatever the worker last pushed (`mem_free` as of that
report — already a live OS reading, just not read synchronously at
decision time). `inuse` is built *entirely* from this scheduler's own
bookkeeping: `sched_account_jobs()` re-derives it every scheduling round by
summing `used + reserved` (`sched_update_res_accounts()`) across every
`xce_res_account_t` belonging to every currently-`RUN` job, for the job's
**entire lifetime** — not just until the worker's next report catches up.
This is the same mechanism `slots` has always used; `mem` uses it too, it
just also happens to have a `total` that independently drifts on its own
schedule.

**Available memory for scheduling, precisely:**

```
available = w->resvals["mem"]      (worker's last-reported mem_free)
          - w->resInUse["mem"]     (this scheduler's own running charge tally)
```

where `w->resInUse["mem"]` is the sum, over every job on that host, of
`max(reserved_j, used_j)` (see the policy below for how each job's term is
computed).

**Known, pre-existing property worth naming, not a bug this document is
introducing:** because `mem`'s own worker-reported figure already reflects
real OS-level consumption from every process on the host (dispatched by
this scheduler or not), and the scheduler's reservation charge is a
*separate, additive* layer that isn't cleared when a fresh `mem` report
arrives (`core_apply_worker_resvals()`/`RESVALS_DELTA`, `core_events.c`,
overwrites `w->resvals` wholesale but never touches `w->resInUse`), a
running job's footprint can end up reflected twice — once because the
worker's next `mem` report is already lower, and again because this
scheduler's own charge for that job hasn't gone anywhere. This makes
admission control *more conservative* over time (under-utilization risk),
never less (it cannot cause over-subscription) — the same direction this
codebase already accepts for `slots`, which has no live ground truth to
reconcile against at all. Untangling that fully (e.g. reconciling
`resInUse` against a fresh worker push instead of letting both layers
accumulate independently) is a separate, larger architectural question,
out of scope here.

## The policy

1. **Request.** A job may submit `rusage[mem=4M]`. If it doesn't, treat the
   reservation as 0 — the job made no promise about memory, but see step 4:
   a reservation of 0 does **not** mean nothing gets charged.

2. **Admission.** Before dispatching to a candidate host, the scheduler
   checks the host has at least 4MB of *available* `mem` per the formula
   above.

3. **Reservation charge.** Once dispatched, 4MB is charged against the
   host's in-use memory — this is the reservation, and it holds regardless
   of what the job actually ends up using, until real usage data says
   otherwise.

4. **Reconciliation against real usage.** Once the worker starts reporting
   the job's actual memory usage (rusage ticks), for **every** job on the
   host — including one that requested no `rusage[mem=...]` at all, whose
   reservation is 0:
   - **actual < reserved:** keep charging the full reservation (0 for an
     unreserved job means: charge nothing extra beyond its actual usage,
     which is exactly what the max() below produces). The estimate might
     have been conservative and the job may still grow into it later — the
     unused slack must not be handed to another job on the strength of a
     snapshot that could be stale a moment later.
   - **actual >= reserved:** charge the real usage instead. For a job that
     never reserved anything, this is *always* this branch (reserved = 0,
     actual >= 0 always) — its full real consumption must be charged,
     because nothing else in this codebase's admission path (see above —
     there is no live OS read at decision time) will ever account for it
     otherwise. Skipping an unreserved job's real usage would leave the
     scheduler's own `inuse` tally blind to memory a live process is
     genuinely sitting on — an actual over-subscription risk, not merely a
     display gap.

   In short: **charged = max(reserved, actual)** — for every job, reserved
   or not.

## Existing infrastructure this policy builds on

The account-level data model already had almost exactly the right shape
for this — it just wasn't wired up to real usage data, and didn't create
an account entry at all for a job that made no reservation:

- `xce_res_account_t` (`types.h`) carries three fields per job-resource
  pairing: `reserved`, `allocated` (both set to the job's request at
  dispatch time, `make_res_account()`, `xcealloc.c`), and `used`.
- `sched_update_res_accounts()` (`resource.c`) sums
  `accounts[i]->used + accounts[i]->reserved` as the amount charged into
  `w->resInUse`/`core->sharedInUse`.

Setting `used = max(0, actual - reserved)` (the *overage* beyond the
reservation, not the raw actual value) makes that existing sum come out
exactly right, with no change needed to the charge formula itself:

```
used + reserved = reserved + max(0, actual - reserved) = max(reserved, actual)
```

For a job with no reservation at all (`reserved = 0`), there is no existing
account entry to update in the first place — `make_res_account()` is only
ever called for resources the job's own `resreq` specified. The
implementation below creates one on first use rather than skipping.

## Implementation (`resource.c`, `schedobj.c`)

Implemented, not just designed:

- **`ensure_host_account_by_resname()`** — finds a job's existing
  `xce_res_account_t` for a resname, or creates a fresh `reserved = 0`
  entry (resolving `residx` via `key_find()` against `core->resmgr`, same
  pattern `sched_update_res_accounts()` already uses for a fresh account)
  if the job never requested that resource. Returns `NULL` only if the
  resource isn't registered in the cluster's config at all — nothing to
  charge, not an error.
- **`sched_reconcile_host_resource_usage()`** — the core formula: sets one
  `xce_res_account_t.used` to `max(0, actual - reserved)` and applies just
  the resulting **delta** to the worker's `resInUse` (never a full
  uncharge/recharge of the whole job account, which would needlessly
  re-touch sibling resources like `slots` charged in the same account).
- **`sched_reconcile_job_resource_usage()`** — the wrapper driven from a
  raw rusage report (`xce_job_rusages_t`, the worker's own wire shape,
  forwarded verbatim to the scheduler by `sched_job_rusage_update_fun()`,
  `controller/sched.c`). Maps the worker's real wire resname
  (`"memory_rss"` — same worker-resname-vs-scheduler-resource-name split
  bug-1735 already established for the controller side) to the scheduler's
  `"mem"` resource. Scoped to **single-host jobs only**: the wire report
  carries no host field at all (`update_job_rusages()`, `controller/job.c`,
  derives the reporting host locally from the endpoint the request arrived
  on and never attaches it to the forwarded struct), so there is no way to
  attribute a report to one specific host of a multi-host job today. That
  is a separate, undocumented-until-now wire-protocol gap — a multi-host
  job's memory stays charged at its flat per-host reservation, exactly as
  it always has; this feature does not make that worse, and does not fix
  it either.
- Wired into **`sched_apply_job_rusage()`** (`schedobj.c`), which already
  received every job rusage tick but previously only stored it for
  display/plugin hooks with zero resource-accounting effect.

Compatible with `sched_account_jobs()`'s existing per-round `resInUse`
reset-and-re-derive cycle (bug-1624): the reconciled `used` value is
persisted on the job's own account record, which that re-derivation reads
from directly — not a transient pool-level delta a round boundary would
wipe.

## Why estimates in either direction are expected, not exceptional

`rusage[mem=...]` is supplied before the job has run — it is a best-effort
prediction of peak lifetime memory, not a measurement. Both directions of
error are routine, not edge cases to special-case away:

- **Over-estimated** (the common conservative-user case): the job never
  grows into the memory it asked for. The reservation still protects the
  host from another job being packed into that never-realized headroom —
  a job that legitimately reserved memory it turns out not to need for
  most of its run can still burst into it later without being starved by a
  neighbor that was admitted on the strength of a low-water-mark reading.
- **Under-estimated**: the job genuinely needs more than it asked for (or
  never asked for anything at all). Reconciliation keeps admission control
  honest about what the host is really carrying once that happens, rather
  than continuing to trust a promise the job has already broken — or, for
  an unreserved job, rather than trusting no promise at all as if that
  meant no consumption.

Neither case is a failure mode to reject or warn about at the accounting
layer — the reservation is advisory input to admission control, not a hard
cap enforced against the job itself (cgroup-level enforcement, if desired,
is a separate concern from scheduling admission).