myxce/.wolf/bugs/bug-1967.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1967	fixed	2026-10-10		scheduler	wrong-result	2026-10-10T10:47:20Z	2026-10-10T10:47:20Z	1	src/scheduler/src/plugins/limit.c	scheduler limit config	bug-1964 bug-1685	8faf5961	398	fixed	c6b64b7827ab

bug-1967: xce_scheduler ignores limit changes in a re-pushed config: seeding stops at the first existing name

Description

Observed: (source reading, 2026-10-10; reported by the bug-1964 fix) limit_seed_config_impl() (src/scheduler/src/plugins/limit.c) builds each configured limit and adds it with key_add(privdata->limits, limit->name, ...), returning at the first failure:

rc = key_add(privdata->limits, (void*)limit->name, (void*)limit);
...
if (rc < 0) {
    ...            /* bug-1685: the kvmgr already freed `limit` */
    return rc;
}

On a later config push (live config upgrade, or a mid-replay CONFIG_PUSH in the simulator since bug-1964) every limit name already exists, so the first key_add fails with -EEXIST: no limit is updated, a limit whose value or scope changed keeps its old definition, and limits listed after an existing one are never added.

Expected: a config push replaces the configured limits: changed limits take their new definition, new ones are added, removed ones disappear (their usage instances handled consistently). Impact: wrong result: limit changes made by a live config upgrade do not take effect until the scheduler restarts; live and simulator agree (both wrong). Found by: fix of bug-1964 (sim limit config push), 2026-10-10.

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ f3898457-dirty
• Deployment: any scheduler (live config upgrade) and xce_scheduler --sim replays with a config push

Reproduction

1. Configure limit L (value 4) and start the scheduler.
2. Push a config where L has value 2 and a new limit M follows it.
3. The scheduler still enforces L = 4 and has no M (REPL info limit all).

• Frequency: deterministic (code path)
• Evidence: the code quoted in Description; no logs kept.

Root cause

limit_seed_config_impl() (src/scheduler/src/plugins/limit.c) added each pushed limit to the existing privdata->limits table and returned at the first key_add() failure; kvmgr has no delete, so on any re-push the first already-present name failed with -EEXIST: old definitions stayed, later new limits were never added and removed ones never dropped. Live config upgrade (limit_update_config) and the sim's CONFIG_PUSH (plugin_config_update_core, bug-1964) share this path. Pre-existing since limits were seeded from config.

Fix

New src/scheduler/src/plugins/limit_config.c limit_replace_config() (private data type moved to limit_internal.h; limit.c was at its line cap):

• builds the whole new limit set in a fresh table, all-or-nothing: a build failure or duplicate name discards it and keeps the previous limits and instances (bug-1685 ownership kept: a key_add-rejected limit is not freed twice; a build_limit failure now frees the half-built limit);
• carries usage instances across: a limit kept under the same name with the same per-user/queue/project/host scoping, resource and percent flag keeps its instances, rebound to the new definition (usage lives in the resource manager's vres, untouched); instances of removed or re-scoped limits are dropped and re-created on the next match;
• swaps the tables, then frees old instances, then old limits. Jobs' glimits/hlimits may briefly hold stale instance pointers; they are rebuilt by limit_job_election() before any read (pushes apply between rounds and bump configver), documented in the file header. Limits/instances are not in the scheduler snapshot (it reseeds from config), so no save/restore change. Behaviour change: a config containing one bad limit now yields no new limits (the previous set stays) instead of keeping the limits listed before the bad one.

Verification

• Test: test_repush_changed_value_takes_effect, test_repush_adds_new_limit_after_existing, test_repush_removes_dropped_limit, test_repush_rebinds_kept_and_drops_removed_instances, test_repush_scope_change_drops_instance, test_repush_failure_keeps_previous_set (src/scheduler/testsuite/test_limit_plugin.c): RED test_limit_plugin.c:595:test_repush_changed_value_takes_effect:FAIL: Expected 0 Was -17 (same at :614, :636, :669, :702), GREEN after. test_repl_limit_table.c now pushes limitA+limitB in one config (it relied on the bug).
• Suites: xce-build.sh all asan xce.build --incremental on the merged branch (scheduler-framework @ edad8c73), 2026-10-10: all C suites Ok 158 / Fail 0, frontend Go unit tests (apiserver + xcecli) ok; apiserver CGO_ENABLED=0 go test ./... (all packages) ok
• Live: n/a: unit-level only; no live config-upgrade run (follow-up: push a changed limit to a running scheduler and check REPL info limit all)

References

• Fix commit: 8faf5961 (worktree original 9bd927d8)
• Related: bug-1964, bug-1685