myxce/.kiro/steering/resource-accounting.md

Resource Accounting: Reservation vs. Actual Usage for Real Resources

The distinction this document is about

XCE's schedulable resources split into two kinds, and they must be accounted differently:

• Virtual resources (e.g. slots) are pure allocation counts. A slot is either allocated to a job or it isn't — there is no OS ground truth for "how much of a slot is actually being used." Reservation and usage are the same number, always.
• Real resources (e.g. mem) have an OS-measurable actual value that moves independently of what a job asked for. A job's rusage[mem=4M] request is an estimate of the memory it will need over its lifetime, supplied at submit time, before the job has run — it is not a hard guarantee, and the real usage the worker later reports may come in above or below it.

This document covers the accounting policy for the second kind, using mem as the running example. The same policy applies to any resource that is both XCE_RESF_DYNAMIC and XCE_RESF_CONSUMABLE (types.h) with a worker-reported instantaneous (gauge-style) value — not to slots, and not to a cumulative counter like cpu_time (a usage metric reported for accounting/billing, not a capacity resource the allocator schedules against).

mem is not maxmem. These are two distinct, similarly-named per-host resources, and it is easy to reach for the wrong one:

• maxmem is a static value, self-reported once by the worker at join time and never refreshed — the host's total installed physical memory.
• mem is a dynamic load index (resource_sync.c's resource_sync_register_load_index("mem", ...), refreshed on the worker's regular load-report cadence, same family as r1m/r15m/swp). Its value is node_info_get_mem() -> read_memory_info()'s mem_free — the OS's own currently-available-physical-memory reading, in bytes.

This whole document is about mem, never maxmem. Admission control cares about what's free right now, not the host's total capacity.

How admission control actually reads "available" (read this before the formula)

The scheduler's admission check does not query the OS live at decision time — deliberately, not as an oversight. All candidate-host resource information is injected into the scheduler once per session (pushed by the worker, cached in w->resvals/w->resInUse); a single scheduling session then evaluates many pending jobs against many candidate hosts using that one cached snapshot, without re-querying between decisions. Querying live on every candidate-host check a session performs would multiply that already-high decision volume by a live I/O round-trip each time — the cost this design avoids. sched_check_host_ satisfy_() (resource.c) reads two already-in-memory arrays and hands them to unsatisfy_():

total = get_resvals(&(w->resvals), ...);    // the worker's last-reported "mem" value
inuse = get_resvals(&(w->resInUse), ...);   // THIS scheduler's own running charge tally
...
if (total[i] < inuse[i] + uncomitted[i] + wanted[i]) { /* can't admit */ }

total for mem is whatever the worker last pushed (mem_free as of that report — already a live OS reading, just not read synchronously at decision time). inuse is built entirely from this scheduler's own bookkeeping: sched_account_jobs() re-derives it every scheduling round by summing used + reserved (sched_update_res_accounts()) across every xce_res_account_t belonging to every currently-RUN job, for the job's entire lifetime — not just until the worker's next report catches up. This is the same mechanism slots has always used; mem uses it too, it just also happens to have a total that independently drifts on its own schedule.

Available memory for scheduling, precisely:

available = w->resvals["mem"]      (worker's last-reported mem_free)
          - w->resInUse["mem"]     (this scheduler's own running charge tally)

where w->resInUse["mem"] is the sum, over every job on that host, of max(reserved_j, used_j) (see the policy below for how each job's term is computed).

Known, pre-existing property worth naming, not a bug this document is introducing: because mem's own worker-reported figure already reflects real OS-level consumption from every process on the host (dispatched by this scheduler or not), and the scheduler's reservation charge is a separate, additive layer that isn't cleared when a fresh mem report arrives (core_apply_worker_resvals()/RESVALS_DELTA, core_events.c, overwrites w->resvals wholesale but never touches w->resInUse), a running job's footprint can end up reflected twice — once because the worker's next mem report is already lower, and again because this scheduler's own charge for that job hasn't gone anywhere. This makes admission control more conservative over time (under-utilization risk), never less (it cannot cause over-subscription) — the same direction this codebase already accepts for slots, which has no live ground truth to reconcile against at all. Untangling that fully (e.g. reconciling resInUse against a fresh worker push instead of letting both layers accumulate independently) is a separate, larger architectural question, out of scope here.

The policy

1. Request. A job may submit rusage[mem=4M]. If it doesn't, treat the reservation as 0 — the job made no promise about memory, but see step 4: a reservation of 0 does not mean nothing gets charged.

2. Admission. Before dispatching to a candidate host, the scheduler checks the host has at least 4MB of available mem per the formula above.

3. Reservation charge. Once dispatched, 4MB is charged against the host's in-use memory — this is the reservation, and it holds regardless of what the job actually ends up using, until real usage data says otherwise.

4. Reconciliation against real usage. Once the worker starts reporting the job's actual memory usage (rusage ticks), for every job on the host — including one that requested no rusage[mem=...] at all, whose reservation is 0:

   • actual < reserved: keep charging the full reservation (0 for an unreserved job means: charge nothing extra beyond its actual usage, which is exactly what the max() below produces). The estimate might have been conservative and the job may still grow into it later — the unused slack must not be handed to another job on the strength of a snapshot that could be stale a moment later.
   • actual >= reserved: charge the real usage instead. For a job that never reserved anything, this is always this branch (reserved = 0, actual >= 0 always) — its full real consumption must be charged, because nothing else in this codebase's admission path (see above — there is no live OS read at decision time) will ever account for it otherwise. Skipping an unreserved job's real usage would leave the scheduler's own inuse tally blind to memory a live process is genuinely sitting on — an actual over-subscription risk, not merely a display gap.

   In short: charged = max(reserved, actual) — for every job, reserved or not.

Existing infrastructure this policy builds on

The account-level data model already had almost exactly the right shape for this — it just wasn't wired up to real usage data, and didn't create an account entry at all for a job that made no reservation:

• xce_res_account_t (types.h) carries three fields per job-resource pairing: reserved, allocated (both set to the job's request at dispatch time, make_res_account(), xcealloc.c), and used.
• sched_update_res_accounts() (resource.c) sums accounts[i]->used + accounts[i]->reserved as the amount charged into w->resInUse/core->sharedInUse.

Setting used = max(0, actual - reserved) (the overage beyond the reservation, not the raw actual value) makes that existing sum come out exactly right, with no change needed to the charge formula itself:

used + reserved = reserved + max(0, actual - reserved) = max(reserved, actual)

For a job with no reservation at all (reserved = 0), there is no existing account entry to update in the first place — make_res_account() is only ever called for resources the job's own resreq specified. The implementation below creates one on first use rather than skipping.

Implementation (resource.c, schedobj.c)

Implemented, not just designed:

• ensure_host_account_by_resname() — finds a job's existing xce_res_account_t for a resname, or creates a fresh reserved = 0 entry (resolving residx via key_find() against core->resmgr, same pattern sched_update_res_accounts() already uses for a fresh account) if the job never requested that resource. Returns NULL only if the resource isn't registered in the cluster's config at all — nothing to charge, not an error.
• sched_reconcile_host_resource_usage() — the core formula: sets one xce_res_account_t.used to max(0, actual - reserved) and applies just the resulting delta to the worker's resInUse (never a full uncharge/recharge of the whole job account, which would needlessly re-touch sibling resources like slots charged in the same account).
• sched_reconcile_job_resource_usage() — the wrapper driven from a raw rusage report (xce_job_rusages_t, the worker's own wire shape, forwarded verbatim to the scheduler by sched_job_rusage_update_fun(), controller/sched.c). Maps the worker's real wire resname ("memory_rss" — same worker-resname-vs-scheduler-resource-name split bug-1735 already established for the controller side) to the scheduler's "mem" resource. Scoped to single-host jobs only: the wire report carries no host field at all (update_job_rusages(), controller/job.c, derives the reporting host locally from the endpoint the request arrived on and never attaches it to the forwarded struct), so there is no way to attribute a report to one specific host of a multi-host job today. That is a separate, undocumented-until-now wire-protocol gap — a multi-host job's memory stays charged at its flat per-host reservation, exactly as it always has; this feature does not make that worse, and does not fix it either.
• Wired into sched_apply_job_rusage() (schedobj.c), which already received every job rusage tick but previously only stored it for display/plugin hooks with zero resource-accounting effect.

Compatible with sched_account_jobs()'s existing per-round resInUse reset-and-re-derive cycle (bug-1624): the reconciled used value is persisted on the job's own account record, which that re-derivation reads from directly — not a transient pool-level delta a round boundary would wipe.

Why estimates in either direction are expected, not exceptional

rusage[mem=...] is supplied before the job has run — it is a best-effort prediction of peak lifetime memory, not a measurement. Both directions of error are routine, not edge cases to special-case away:

• Over-estimated (the common conservative-user case): the job never grows into the memory it asked for. The reservation still protects the host from another job being packed into that never-realized headroom — a job that legitimately reserved memory it turns out not to need for most of its run can still burst into it later without being starved by a neighbor that was admitted on the strength of a low-water-mark reading.
• Under-estimated: the job genuinely needs more than it asked for (or never asked for anything at all). Reconciliation keeps admission control honest about what the host is really carrying once that happens, rather than continuing to trust a promise the job has already broken — or, for an unreserved job, rather than trusting no promise at all as if that meant no consumption.

Neither case is a failure mode to reject or warn about at the accounting layer — the reservation is advisory input to admission control, not a hard cap enforced against the job itself (cgroup-level enforcement, if desired, is a separate concern from scheduling admission).