--- id: "bug-1976" status: "fixed" status_updated: "2026-10-28" status_note: "" component: "controller" severity: "wrong-result" timestamp: "2026-10-28T05:37:09Z" last_seen: "2026-10-28T05:26:00Z" occurrences: 1 files: ["src/controller/job.c", "src/controller/testsuite/test_dispatch_commit.c"] tags: ["external-license-resources", "limits", "shared-resources", "dispatch", "e2-phase3"] related: ["bug-1860"] fix_commits: ["76ddd75f"] summary: "controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced" --- # bug-1976: controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced ## Description **Observed:** E2 Phase 3 t1 E2E: 4 projA jobs with `-R "rusage[extlic=1]"` against a per-project limit `extlic_share` (scope projA, cap 3.0) — ALL FOUR ran instead of the 4th PENDing with reason limit_global. Live gdb on the scheduler showed `sharedInUse[36/37/38]` (the vres instances) = 0.0 with 3 projA jobs RUN: the vres charge never landed. **Expected:** the 4th projA job PENDs (limit_global); killing a RUN job re-admits it; projB unaffected (own per-project instance). **Impact:** every scheduler-side global limit (the limit vres mechanism) was advisory only — no quota was ever enforced, for ANY shared resource carried via allocs_shared, since dispatch-time adoption overwrote the scheduler's own charged accounts with a shared-stripped copy. **Found by:** spec task (external-license-resources E2 Phase 3 E2E). ## Environment - OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64 - libc: glibc 2.39 - Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 - Build: debug (build.dbg) - Code: feature/license-resources @ 76ddd75f (bug present since dispatch commit was introduced, f4b1ba4c-era) - Deployment: t1 single-node controller + t2/t3 workers (de-nix binaries in /opt/xce-denix), xceesd reporting extlic via fake_lic.sh ## Reproduction 1. Push a config with a per-project limit (limits.yaml `extlic_share`, cap 3.0) and an EXTERNAL resource extlic with ≥4 free. 2. Submit 4 jobs `-P projA -R "rusage[extlic=1]"`; observe all 4 RUN. 3. gdb scheduler: sharedInUse[vres idx] = 0.0 despite 3+ RUN jobs. - Frequency: deterministic. - Evidence: this record + t1 E2E transcript in session (2026-10-28). ## Root cause The controller's dispatch-time commit (`dispatch_a_job_()`, job.c) rebuilt `j->job.allocs` from the scheduler's dispatch payload but migrated ONLY the per-host records (`MOVE_PTR(j->job.allocs->allocs, allocs)`): `nres_shared` stayed 0 and the payload's `allocs_shared` was neither moved nor freed (a leak on top). That rebuilt record is what `sched_job_update_fun()` re-sends to the scheduler as ground truth, and the scheduler's JOB_UPDATE adoption (bug-1860 branch, core_events.c) rebuilds `job->accounts` from exactly that wire copy — so `accounts_shared == NULL`, `sched_account_jobs` charges nothing to the vres, and `unsatisfy_` sees cap ≥ 0 + want forever. The scheduler's own commit (`sched_commit_decision`, scheduler.c:1806-1812) built the accounts correctly — the controller's "ground truth" clobbered it. ## Fix job.c dispatch-time commit: migrate the shared records alongside the host records — `MOVE_PTR(j->job.allocs->allocs_shared, alloc->allocs_shared); j->job.allocs->nres_shared = alloc->nres_shared; alloc->nres_shared = 0;` (MOVE, not copy — the caller frees the payload; also closes the leak). ## Verification - Test: `test_dc_success_migrates_shared_allocs` (src/controller/testsuite/test_dispatch_commit.c). RED before the fix ("Expected 2 Was 0" on nres_shared), GREEN after. - Suites: full meson test 136/136 OK (build.dbg), including controller unit suite 1361/0. - Live: t1 E2E — 3 projA RUN / 4th PEND, projB RUN (own instance), kill one RUN job → pending projA re-admitted, external free pool (fake_lic) enforced independently; all daemons healthy throughout. ## References - Related: bug-1860 (the JOB_UPDATE ground-truth adoption that made the stripped copy authoritative), external-license-resources design.md §3.5/§3.6 (per-app→per-project terminology).