wolf: log bug-1976 fixed (controller dispatch commit dropped allocs_shared)

Co-Authored-By: Claude Code <noreply@anthropic.com>

.wolf/buglog.json

@@ -139,6 +139,35 @@
         "2fcacac2"
       ],
       "record": ".wolf/bugs/bug-1975.md"
+    },
+    {
+      "id": "bug-1976",
+      "status": "fixed",
+      "status_updated": "2026-10-28",
+      "status_note": "",
+      "component": "controller",
+      "severity": "wrong-result",
+      "timestamp": "2026-10-28T05:37:09Z",
+      "last_seen": "2026-10-28T05:26:00Z",
+      "occurrences": 1,
+      "error_message": "controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced\n\n**Observed:**\nE2 Phase 3 t1 E2E: 4 projA jobs with `-R \"rusage[extlic=1]\"` against a\nper-project limit `extlic_share` (scope projA, cap 3.0) — ALL FOUR ran\ninstead of the 4th PENDing with reason limit_global. Live gdb on the\nscheduler showed `sharedInUse[36/37/38]` (the vres instances) = 0.0 with\n3 projA jobs RUN: the vres charge never landed.\n\n**Expected:** the 4th projA job PENDs (limit_global); killing a RUN job\nre-admits it; projB unaffected (own per-project instance).\n\n**Impact:** every scheduler-side global limit (the limit vres mechanism)\nwas advisory only — no quota was ever enforced, for ANY shared resource\ncarried via allocs_shared, since dispatch-time adoption overwrote the\nscheduler's own charged accounts with a shared-stripped copy.\n\n**Found by:** spec task (external-license-resources E2 Phase 3 E2E).",
+      "file": "src/controller/job.c, src/controller/testsuite/test_dispatch_commit.c",
+      "root_cause": "The controller's dispatch-time commit (`dispatch_a_job_()`, job.c)\nrebuilt `j->job.allocs` from the scheduler's dispatch payload but\nmigrated ONLY the per-host records (`MOVE_PTR(j->job.allocs->allocs,\nallocs)`): `nres_shared` stayed 0 and the payload's `allocs_shared`\nwas neither moved nor freed (a leak on top). That rebuilt record is\nwhat `sched_job_update_fun()` re-sends to the scheduler as ground\ntruth, and the scheduler's JOB_UPDATE adoption (bug-1860 branch,\ncore_events.c) rebuilds `job->accounts` from exactly that wire copy —\nso `accounts_shared == NULL`, `sched_account_jobs` charges nothing to\nthe vres, and `unsatisfy_` sees cap ≥ 0 + want forever. The scheduler's\nown commit (`sched_commit_decision`, scheduler.c:1806-1812) built the\naccounts correctly — the controller's \"ground truth\" clobbered it.",
+      "fix": "job.c dispatch-time commit: migrate the shared records alongside the\nhost records — `MOVE_PTR(j->job.allocs->allocs_shared,\nalloc->allocs_shared); j->job.allocs->nres_shared =\nalloc->nres_shared; alloc->nres_shared = 0;` (MOVE, not copy — the\ncaller frees the payload; also closes the leak).\n\nVerification:\n- Test: `test_dc_success_migrates_shared_allocs`\n  (src/controller/testsuite/test_dispatch_commit.c). RED before the fix\n  (\"Expected 2 Was 0\" on nres_shared), GREEN after.\n- Suites: full meson test 136/136 OK (build.dbg), including controller\n  unit suite 1361/0.\n- Live: t1 E2E — 3 projA RUN / 4th PEND, projB RUN (own instance), kill\n  one RUN job → pending projA re-admitted, external free pool\n  (fake_lic) enforced independently; all daemons healthy throughout.",
+      "tags": [
+        "external-license-resources",
+        "limits",
+        "shared-resources",
+        "dispatch",
+        "e2-phase3"
+      ],
+      "related_bugs": [
+        "bug-1860"
+      ],
+      "fix_commits": [
+        "76ddd75f"
+      ],
+      "record": ".wolf/bugs/bug-1976.md"
     }
   ]
 }

.wolf/bugs/bug-1976.md

@@ -0,0 +1,90 @@
+---
+id: "bug-1976"
+status: "fixed"
+status_updated: "2026-10-28"
+status_note: ""
+component: "controller"
+severity: "wrong-result"
+timestamp: "2026-10-28T05:37:09Z"
+last_seen: "2026-10-28T05:26:00Z"
+occurrences: 1
+files: ["src/controller/job.c", "src/controller/testsuite/test_dispatch_commit.c"]
+tags: ["external-license-resources", "limits", "shared-resources", "dispatch", "e2-phase3"]
+related: ["bug-1860"]
+fix_commits: ["76ddd75f"]
+summary: "controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced"
+---
+
+# bug-1976: controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced
+
+## Description
+**Observed:**
+E2 Phase 3 t1 E2E: 4 projA jobs with `-R "rusage[extlic=1]"` against a
+per-project limit `extlic_share` (scope projA, cap 3.0) — ALL FOUR ran
+instead of the 4th PENDing with reason limit_global. Live gdb on the
+scheduler showed `sharedInUse[36/37/38]` (the vres instances) = 0.0 with
+3 projA jobs RUN: the vres charge never landed.
+
+**Expected:** the 4th projA job PENDs (limit_global); killing a RUN job
+re-admits it; projB unaffected (own per-project instance).
+
+**Impact:** every scheduler-side global limit (the limit vres mechanism)
+was advisory only — no quota was ever enforced, for ANY shared resource
+carried via allocs_shared, since dispatch-time adoption overwrote the
+scheduler's own charged accounts with a shared-stripped copy.
+
+**Found by:** spec task (external-license-resources E2 Phase 3 E2E).
+
+## Environment
+- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
+- libc: glibc 2.39
+- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
+- Build: debug (build.dbg)
+- Code: feature/license-resources @ 76ddd75f (bug present since dispatch
+  commit was introduced, f4b1ba4c-era)
+- Deployment: t1 single-node controller + t2/t3 workers (de-nix binaries
+  in /opt/xce-denix), xceesd reporting extlic via fake_lic.sh
+
+## Reproduction
+1. Push a config with a per-project limit (limits.yaml `extlic_share`,
+   cap 3.0) and an EXTERNAL resource extlic with ≥4 free.
+2. Submit 4 jobs `-P projA -R "rusage[extlic=1]"`; observe all 4 RUN.
+3. gdb scheduler: sharedInUse[vres idx] = 0.0 despite 3+ RUN jobs.
+- Frequency: deterministic.
+- Evidence: this record + t1 E2E transcript in session (2026-10-28).
+
+## Root cause
+The controller's dispatch-time commit (`dispatch_a_job_()`, job.c)
+rebuilt `j->job.allocs` from the scheduler's dispatch payload but
+migrated ONLY the per-host records (`MOVE_PTR(j->job.allocs->allocs,
+allocs)`): `nres_shared` stayed 0 and the payload's `allocs_shared`
+was neither moved nor freed (a leak on top). That rebuilt record is
+what `sched_job_update_fun()` re-sends to the scheduler as ground
+truth, and the scheduler's JOB_UPDATE adoption (bug-1860 branch,
+core_events.c) rebuilds `job->accounts` from exactly that wire copy —
+so `accounts_shared == NULL`, `sched_account_jobs` charges nothing to
+the vres, and `unsatisfy_` sees cap ≥ 0 + want forever. The scheduler's
+own commit (`sched_commit_decision`, scheduler.c:1806-1812) built the
+accounts correctly — the controller's "ground truth" clobbered it.
+
+## Fix
+job.c dispatch-time commit: migrate the shared records alongside the
+host records — `MOVE_PTR(j->job.allocs->allocs_shared,
+alloc->allocs_shared); j->job.allocs->nres_shared =
+alloc->nres_shared; alloc->nres_shared = 0;` (MOVE, not copy — the
+caller frees the payload; also closes the leak).
+
+## Verification
+- Test: `test_dc_success_migrates_shared_allocs`
+  (src/controller/testsuite/test_dispatch_commit.c). RED before the fix
+  ("Expected 2 Was 0" on nres_shared), GREEN after.
+- Suites: full meson test 136/136 OK (build.dbg), including controller
+  unit suite 1361/0.
+- Live: t1 E2E — 3 projA RUN / 4th PEND, projB RUN (own instance), kill
+  one RUN job → pending projA re-admitted, external free pool
+  (fake_lic) enforced independently; all daemons healthy throughout.
+
+## References
+- Related: bug-1860 (the JOB_UPDATE ground-truth adoption that made the
+  stripped copy authoritative), external-license-resources design.md
+  §3.5/§3.6 (per-app→per-project terminology).