myxce/.wolf/bugs/bug-1976.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	summary
bug-1976	fixed	2026-10-28		controller	wrong-result	2026-10-28T05:37:09Z	2026-10-28T05:26:00Z	1	src/controller/job.c src/controller/testsuite/test_dispatch_commit.c	external-license-resources limits shared-resources dispatch e2-phase3	bug-1860	76ddd75f	controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced

bug-1976: controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced

Description

Observed: E2 Phase 3 t1 E2E: 4 projA jobs with -R "rusage[extlic=1]" against a per-project limit extlic_share (scope projA, cap 3.0) — ALL FOUR ran instead of the 4th PENDing with reason limit_global. Live gdb on the scheduler showed sharedInUse[36/37/38] (the vres instances) = 0.0 with 3 projA jobs RUN: the vres charge never landed.

Expected: the 4th projA job PENDs (limit_global); killing a RUN job re-admits it; projB unaffected (own per-project instance).

Impact: every scheduler-side global limit (the limit vres mechanism) was advisory only — no quota was ever enforced, for ANY shared resource carried via allocs_shared, since dispatch-time adoption overwrote the scheduler's own charged accounts with a shared-stripped copy.

Found by: spec task (external-license-resources E2 Phase 3 E2E).

Environment

• OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
• libc: glibc 2.39
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: debug (build.dbg)
• Code: feature/license-resources @ 76ddd75f (bug present since dispatch commit was introduced, f4b1ba4c-era)
• Deployment: t1 single-node controller + t2/t3 workers (de-nix binaries in /opt/xce-denix), xceesd reporting extlic via fake_lic.sh

Reproduction

1. Push a config with a per-project limit (limits.yaml extlic_share, cap 3.0) and an EXTERNAL resource extlic with ≥4 free.
2. Submit 4 jobs -P projA -R "rusage[extlic=1]"; observe all 4 RUN.
3. gdb scheduler: sharedInUse[vres idx] = 0.0 despite 3+ RUN jobs.

• Frequency: deterministic.
• Evidence: this record + t1 E2E transcript in session (2026-10-28).

Root cause

The controller's dispatch-time commit (dispatch_a_job_(), job.c) rebuilt j->job.allocs from the scheduler's dispatch payload but migrated ONLY the per-host records (MOVE_PTR(j->job.allocs->allocs, allocs)): nres_shared stayed 0 and the payload's allocs_shared was neither moved nor freed (a leak on top). That rebuilt record is what sched_job_update_fun() re-sends to the scheduler as ground truth, and the scheduler's JOB_UPDATE adoption (bug-1860 branch, core_events.c) rebuilds job->accounts from exactly that wire copy — so accounts_shared == NULL, sched_account_jobs charges nothing to the vres, and unsatisfy_ sees cap ≥ 0 + want forever. The scheduler's own commit (sched_commit_decision, scheduler.c:1806-1812) built the accounts correctly — the controller's "ground truth" clobbered it.

Fix

job.c dispatch-time commit: migrate the shared records alongside the host records — MOVE_PTR(j->job.allocs->allocs_shared, alloc->allocs_shared); j->job.allocs->nres_shared = alloc->nres_shared; alloc->nres_shared = 0; (MOVE, not copy — the caller frees the payload; also closes the leak).

Verification

• Test: test_dc_success_migrates_shared_allocs (src/controller/testsuite/test_dispatch_commit.c). RED before the fix ("Expected 2 Was 0" on nres_shared), GREEN after.
• Suites: full meson test 136/136 OK (build.dbg), including controller unit suite 1361/0.
• Live: t1 E2E — 3 projA RUN / 4th PEND, projB RUN (own instance), kill one RUN job → pending projA re-admitted, external free pool (fake_lic) enforced independently; all daemons healthy throughout.

References

• Related: bug-1860 (the JOB_UPDATE ground-truth adoption that made the stripped copy authoritative), external-license-resources design.md §3.5/§3.6 (per-app→per-project terminology).