From 13a28a387a72e33f03a48020d85523cfac96e71e Mon Sep 17 00:00:00 2001 From: "lei.chen" Date: Wed, 28 Oct 2026 05:40:03 +0000 Subject: [PATCH] wolf: log bug-1976 fixed (controller dispatch commit dropped allocs_shared) Co-Authored-By: Claude Code --- .wolf/buglog.json | 29 ++++++++++++++ .wolf/bugs/bug-1976.md | 90 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 119 insertions(+) create mode 100644 .wolf/bugs/bug-1976.md diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 1694cb2d..aafdf660 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -139,6 +139,35 @@ "2fcacac2" ], "record": ".wolf/bugs/bug-1975.md" + }, + { + "id": "bug-1976", + "status": "fixed", + "status_updated": "2026-10-28", + "status_note": "", + "component": "controller", + "severity": "wrong-result", + "timestamp": "2026-10-28T05:37:09Z", + "last_seen": "2026-10-28T05:26:00Z", + "occurrences": 1, + "error_message": "controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced\n\n**Observed:**\nE2 Phase 3 t1 E2E: 4 projA jobs with `-R \"rusage[extlic=1]\"` against a\nper-project limit `extlic_share` (scope projA, cap 3.0) — ALL FOUR ran\ninstead of the 4th PENDing with reason limit_global. Live gdb on the\nscheduler showed `sharedInUse[36/37/38]` (the vres instances) = 0.0 with\n3 projA jobs RUN: the vres charge never landed.\n\n**Expected:** the 4th projA job PENDs (limit_global); killing a RUN job\nre-admits it; projB unaffected (own per-project instance).\n\n**Impact:** every scheduler-side global limit (the limit vres mechanism)\nwas advisory only — no quota was ever enforced, for ANY shared resource\ncarried via allocs_shared, since dispatch-time adoption overwrote the\nscheduler's own charged accounts with a shared-stripped copy.\n\n**Found by:** spec task (external-license-resources E2 Phase 3 E2E).", + "file": "src/controller/job.c, src/controller/testsuite/test_dispatch_commit.c", + "root_cause": "The controller's dispatch-time commit (`dispatch_a_job_()`, job.c)\nrebuilt `j->job.allocs` from the scheduler's dispatch payload but\nmigrated ONLY the per-host records (`MOVE_PTR(j->job.allocs->allocs,\nallocs)`): `nres_shared` stayed 0 and the payload's `allocs_shared`\nwas neither moved nor freed (a leak on top). That rebuilt record is\nwhat `sched_job_update_fun()` re-sends to the scheduler as ground\ntruth, and the scheduler's JOB_UPDATE adoption (bug-1860 branch,\ncore_events.c) rebuilds `job->accounts` from exactly that wire copy —\nso `accounts_shared == NULL`, `sched_account_jobs` charges nothing to\nthe vres, and `unsatisfy_` sees cap ≥ 0 + want forever. The scheduler's\nown commit (`sched_commit_decision`, scheduler.c:1806-1812) built the\naccounts correctly — the controller's \"ground truth\" clobbered it.", + "fix": "job.c dispatch-time commit: migrate the shared records alongside the\nhost records — `MOVE_PTR(j->job.allocs->allocs_shared,\nalloc->allocs_shared); j->job.allocs->nres_shared =\nalloc->nres_shared; alloc->nres_shared = 0;` (MOVE, not copy — the\ncaller frees the payload; also closes the leak).\n\nVerification:\n- Test: `test_dc_success_migrates_shared_allocs`\n (src/controller/testsuite/test_dispatch_commit.c). RED before the fix\n (\"Expected 2 Was 0\" on nres_shared), GREEN after.\n- Suites: full meson test 136/136 OK (build.dbg), including controller\n unit suite 1361/0.\n- Live: t1 E2E — 3 projA RUN / 4th PEND, projB RUN (own instance), kill\n one RUN job → pending projA re-admitted, external free pool\n (fake_lic) enforced independently; all daemons healthy throughout.", + "tags": [ + "external-license-resources", + "limits", + "shared-resources", + "dispatch", + "e2-phase3" + ], + "related_bugs": [ + "bug-1860" + ], + "fix_commits": [ + "76ddd75f" + ], + "record": ".wolf/bugs/bug-1976.md" } ] } diff --git a/.wolf/bugs/bug-1976.md b/.wolf/bugs/bug-1976.md new file mode 100644 index 00000000..9a1e7b66 --- /dev/null +++ b/.wolf/bugs/bug-1976.md @@ -0,0 +1,90 @@ +--- +id: "bug-1976" +status: "fixed" +status_updated: "2026-10-28" +status_note: "" +component: "controller" +severity: "wrong-result" +timestamp: "2026-10-28T05:37:09Z" +last_seen: "2026-10-28T05:26:00Z" +occurrences: 1 +files: ["src/controller/job.c", "src/controller/testsuite/test_dispatch_commit.c"] +tags: ["external-license-resources", "limits", "shared-resources", "dispatch", "e2-phase3"] +related: ["bug-1860"] +fix_commits: ["76ddd75f"] +summary: "controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced" +--- + +# bug-1976: controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced + +## Description +**Observed:** +E2 Phase 3 t1 E2E: 4 projA jobs with `-R "rusage[extlic=1]"` against a +per-project limit `extlic_share` (scope projA, cap 3.0) — ALL FOUR ran +instead of the 4th PENDing with reason limit_global. Live gdb on the +scheduler showed `sharedInUse[36/37/38]` (the vres instances) = 0.0 with +3 projA jobs RUN: the vres charge never landed. + +**Expected:** the 4th projA job PENDs (limit_global); killing a RUN job +re-admits it; projB unaffected (own per-project instance). + +**Impact:** every scheduler-side global limit (the limit vres mechanism) +was advisory only — no quota was ever enforced, for ANY shared resource +carried via allocs_shared, since dispatch-time adoption overwrote the +scheduler's own charged accounts with a shared-stripped copy. + +**Found by:** spec task (external-license-resources E2 Phase 3 E2E). + +## Environment +- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64 +- libc: glibc 2.39 +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: debug (build.dbg) +- Code: feature/license-resources @ 76ddd75f (bug present since dispatch + commit was introduced, f4b1ba4c-era) +- Deployment: t1 single-node controller + t2/t3 workers (de-nix binaries + in /opt/xce-denix), xceesd reporting extlic via fake_lic.sh + +## Reproduction +1. Push a config with a per-project limit (limits.yaml `extlic_share`, + cap 3.0) and an EXTERNAL resource extlic with ≥4 free. +2. Submit 4 jobs `-P projA -R "rusage[extlic=1]"`; observe all 4 RUN. +3. gdb scheduler: sharedInUse[vres idx] = 0.0 despite 3+ RUN jobs. +- Frequency: deterministic. +- Evidence: this record + t1 E2E transcript in session (2026-10-28). + +## Root cause +The controller's dispatch-time commit (`dispatch_a_job_()`, job.c) +rebuilt `j->job.allocs` from the scheduler's dispatch payload but +migrated ONLY the per-host records (`MOVE_PTR(j->job.allocs->allocs, +allocs)`): `nres_shared` stayed 0 and the payload's `allocs_shared` +was neither moved nor freed (a leak on top). That rebuilt record is +what `sched_job_update_fun()` re-sends to the scheduler as ground +truth, and the scheduler's JOB_UPDATE adoption (bug-1860 branch, +core_events.c) rebuilds `job->accounts` from exactly that wire copy — +so `accounts_shared == NULL`, `sched_account_jobs` charges nothing to +the vres, and `unsatisfy_` sees cap ≥ 0 + want forever. The scheduler's +own commit (`sched_commit_decision`, scheduler.c:1806-1812) built the +accounts correctly — the controller's "ground truth" clobbered it. + +## Fix +job.c dispatch-time commit: migrate the shared records alongside the +host records — `MOVE_PTR(j->job.allocs->allocs_shared, +alloc->allocs_shared); j->job.allocs->nres_shared = +alloc->nres_shared; alloc->nres_shared = 0;` (MOVE, not copy — the +caller frees the payload; also closes the leak). + +## Verification +- Test: `test_dc_success_migrates_shared_allocs` + (src/controller/testsuite/test_dispatch_commit.c). RED before the fix + ("Expected 2 Was 0" on nres_shared), GREEN after. +- Suites: full meson test 136/136 OK (build.dbg), including controller + unit suite 1361/0. +- Live: t1 E2E — 3 projA RUN / 4th PEND, projB RUN (own instance), kill + one RUN job → pending projA re-admitted, external free pool + (fake_lic) enforced independently; all daemons healthy throughout. + +## References +- Related: bug-1860 (the JOB_UPDATE ground-truth adoption that made the + stripped copy authoritative), external-license-resources design.md + §3.5/§3.6 (per-app→per-project terminology).