90 lines
4 KiB
Markdown
90 lines
4 KiB
Markdown
---
|
|
id: "bug-1976"
|
|
status: "fixed"
|
|
status_updated: "2026-10-28"
|
|
status_note: ""
|
|
component: "controller"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-10-28T05:37:09Z"
|
|
last_seen: "2026-10-28T05:26:00Z"
|
|
occurrences: 1
|
|
files: ["src/controller/job.c", "src/controller/testsuite/test_dispatch_commit.c"]
|
|
tags: ["external-license-resources", "limits", "shared-resources", "dispatch", "e2-phase3"]
|
|
related: ["bug-1860"]
|
|
fix_commits: ["76ddd75f"]
|
|
summary: "controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced"
|
|
---
|
|
|
|
# bug-1976: controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced
|
|
|
|
## Description
|
|
**Observed:**
|
|
E2 Phase 3 t1 E2E: 4 projA jobs with `-R "rusage[extlic=1]"` against a
|
|
per-project limit `extlic_share` (scope projA, cap 3.0) — ALL FOUR ran
|
|
instead of the 4th PENDing with reason limit_global. Live gdb on the
|
|
scheduler showed `sharedInUse[36/37/38]` (the vres instances) = 0.0 with
|
|
3 projA jobs RUN: the vres charge never landed.
|
|
|
|
**Expected:** the 4th projA job PENDs (limit_global); killing a RUN job
|
|
re-admits it; projB unaffected (own per-project instance).
|
|
|
|
**Impact:** every scheduler-side global limit (the limit vres mechanism)
|
|
was advisory only — no quota was ever enforced, for ANY shared resource
|
|
carried via allocs_shared, since dispatch-time adoption overwrote the
|
|
scheduler's own charged accounts with a shared-stripped copy.
|
|
|
|
**Found by:** spec task (external-license-resources E2 Phase 3 E2E).
|
|
|
|
## Environment
|
|
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
|
|
- libc: glibc 2.39
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: debug (build.dbg)
|
|
- Code: feature/license-resources @ 76ddd75f (bug present since dispatch
|
|
commit was introduced, f4b1ba4c-era)
|
|
- Deployment: t1 single-node controller + t2/t3 workers (de-nix binaries
|
|
in /opt/xce-denix), xceesd reporting extlic via fake_lic.sh
|
|
|
|
## Reproduction
|
|
1. Push a config with a per-project limit (limits.yaml `extlic_share`,
|
|
cap 3.0) and an EXTERNAL resource extlic with ≥4 free.
|
|
2. Submit 4 jobs `-P projA -R "rusage[extlic=1]"`; observe all 4 RUN.
|
|
3. gdb scheduler: sharedInUse[vres idx] = 0.0 despite 3+ RUN jobs.
|
|
- Frequency: deterministic.
|
|
- Evidence: this record + t1 E2E transcript in session (2026-10-28).
|
|
|
|
## Root cause
|
|
The controller's dispatch-time commit (`dispatch_a_job_()`, job.c)
|
|
rebuilt `j->job.allocs` from the scheduler's dispatch payload but
|
|
migrated ONLY the per-host records (`MOVE_PTR(j->job.allocs->allocs,
|
|
allocs)`): `nres_shared` stayed 0 and the payload's `allocs_shared`
|
|
was neither moved nor freed (a leak on top). That rebuilt record is
|
|
what `sched_job_update_fun()` re-sends to the scheduler as ground
|
|
truth, and the scheduler's JOB_UPDATE adoption (bug-1860 branch,
|
|
core_events.c) rebuilds `job->accounts` from exactly that wire copy —
|
|
so `accounts_shared == NULL`, `sched_account_jobs` charges nothing to
|
|
the vres, and `unsatisfy_` sees cap ≥ 0 + want forever. The scheduler's
|
|
own commit (`sched_commit_decision`, scheduler.c:1806-1812) built the
|
|
accounts correctly — the controller's "ground truth" clobbered it.
|
|
|
|
## Fix
|
|
job.c dispatch-time commit: migrate the shared records alongside the
|
|
host records — `MOVE_PTR(j->job.allocs->allocs_shared,
|
|
alloc->allocs_shared); j->job.allocs->nres_shared =
|
|
alloc->nres_shared; alloc->nres_shared = 0;` (MOVE, not copy — the
|
|
caller frees the payload; also closes the leak).
|
|
|
|
## Verification
|
|
- Test: `test_dc_success_migrates_shared_allocs`
|
|
(src/controller/testsuite/test_dispatch_commit.c). RED before the fix
|
|
("Expected 2 Was 0" on nres_shared), GREEN after.
|
|
- Suites: full meson test 136/136 OK (build.dbg), including controller
|
|
unit suite 1361/0.
|
|
- Live: t1 E2E — 3 projA RUN / 4th PEND, projB RUN (own instance), kill
|
|
one RUN job → pending projA re-admitted, external free pool
|
|
(fake_lic) enforced independently; all daemons healthy throughout.
|
|
|
|
## References
|
|
- Related: bug-1860 (the JOB_UPDATE ground-truth adoption that made the
|
|
stripped copy authoritative), external-license-resources design.md
|
|
§3.5/§3.6 (per-app→per-project terminology).
|