80 lines
5.3 KiB
Markdown
80 lines
5.3 KiB
Markdown
---
|
|
id: "bug-1967"
|
|
status: "fixed"
|
|
status_updated: "2026-10-10"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-10-10T10:47:20Z"
|
|
last_seen: "2026-10-10T10:47:20Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/plugins/limit.c"]
|
|
tags: ["scheduler", "limit", "config"]
|
|
related: ["bug-1964", "bug-1685"]
|
|
fix_commits: ["8faf5961"]
|
|
gitea_issue: 398
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "c6b64b7827ab"
|
|
---
|
|
|
|
# bug-1967: xce_scheduler ignores limit changes in a re-pushed config: seeding stops at the first existing name
|
|
|
|
## Description
|
|
**Observed:** (source reading, 2026-10-10; reported by the bug-1964 fix)
|
|
`limit_seed_config_impl()` (`src/scheduler/src/plugins/limit.c`) builds each
|
|
configured limit and adds it with `key_add(privdata->limits, limit->name, ...)`,
|
|
returning at the first failure:
|
|
```c
|
|
rc = key_add(privdata->limits, (void*)limit->name, (void*)limit);
|
|
...
|
|
if (rc < 0) {
|
|
... /* bug-1685: the kvmgr already freed `limit` */
|
|
return rc;
|
|
}
|
|
```
|
|
On a later config push (live config upgrade, or a mid-replay CONFIG_PUSH in
|
|
the simulator since bug-1964) every limit name already exists, so the first
|
|
`key_add` fails with `-EEXIST`: no limit is updated, a limit whose value or
|
|
scope changed keeps its old definition, and limits listed after an existing
|
|
one are never added.
|
|
|
|
**Expected:** a config push replaces the configured limits: changed limits
|
|
take their new definition, new ones are added, removed ones disappear
|
|
(their usage instances handled consistently).
|
|
**Impact:** wrong result: limit changes made by a live config upgrade do not
|
|
take effect until the scheduler restarts; live and simulator agree (both
|
|
wrong).
|
|
**Found by:** fix of bug-1964 (sim limit config push), 2026-10-10.
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ f3898457-dirty
|
|
- Deployment: any scheduler (live config upgrade) and `xce_scheduler --sim` replays with a config push
|
|
|
|
## Reproduction
|
|
1. Configure limit `L` (value 4) and start the scheduler.
|
|
2. Push a config where `L` has value 2 and a new limit `M` follows it.
|
|
3. The scheduler still enforces `L` = 4 and has no `M` (REPL `info limit all`).
|
|
- Frequency: deterministic (code path)
|
|
- Evidence: the code quoted in Description; no logs kept.
|
|
## Root cause
|
|
`limit_seed_config_impl()` (`src/scheduler/src/plugins/limit.c`) added each pushed limit to the existing `privdata->limits` table and returned at the first `key_add()` failure; kvmgr has no delete, so on any re-push the first already-present name failed with `-EEXIST`: old definitions stayed, later new limits were never added and removed ones never dropped. Live config upgrade (`limit_update_config`) and the sim's CONFIG_PUSH (`plugin_config_update_core`, bug-1964) share this path. Pre-existing since limits were seeded from config.
|
|
|
|
## Fix
|
|
New `src/scheduler/src/plugins/limit_config.c` `limit_replace_config()` (private data type moved to `limit_internal.h`; `limit.c` was at its line cap):
|
|
- builds the whole new limit set in a fresh table, all-or-nothing: a build failure or duplicate name discards it and keeps the previous limits and instances (bug-1685 ownership kept: a `key_add`-rejected limit is not freed twice; a `build_limit` failure now frees the half-built limit);
|
|
- carries usage instances across: a limit kept under the same name with the same per-user/queue/project/host scoping, resource and percent flag keeps its instances, rebound to the new definition (usage lives in the resource manager's vres, untouched); instances of removed or re-scoped limits are dropped and re-created on the next match;
|
|
- swaps the tables, then frees old instances, then old limits.
|
|
Jobs' `glimits`/`hlimits` may briefly hold stale instance pointers; they are rebuilt by `limit_job_election()` before any read (pushes apply between rounds and bump `configver`), documented in the file header. Limits/instances are not in the scheduler snapshot (it reseeds from config), so no save/restore change. Behaviour change: a config containing one bad limit now yields no new limits (the previous set stays) instead of keeping the limits listed before the bad one.
|
|
|
|
## Verification
|
|
- Test: `test_repush_changed_value_takes_effect`, `test_repush_adds_new_limit_after_existing`, `test_repush_removes_dropped_limit`, `test_repush_rebinds_kept_and_drops_removed_instances`, `test_repush_scope_change_drops_instance`, `test_repush_failure_keeps_previous_set` (`src/scheduler/testsuite/test_limit_plugin.c`): RED `test_limit_plugin.c:595:test_repush_changed_value_takes_effect:FAIL: Expected 0 Was -17` (same at :614, :636, :669, :702), GREEN after. `test_repl_limit_table.c` now pushes limitA+limitB in one config (it relied on the bug).
|
|
- Suites: `xce-build.sh all asan xce.build --incremental` on the merged branch (scheduler-framework @ edad8c73), 2026-10-10: all C suites Ok 158 / Fail 0, frontend Go unit tests (apiserver + xcecli) ok; apiserver `CGO_ENABLED=0 go test ./...` (all packages) ok
|
|
- Live: n/a: unit-level only; no live config-upgrade run (follow-up: push a changed limit to a running scheduler and check REPL `info limit all`)
|
|
|
|
## References
|
|
- Fix commit: `8faf5961` (worktree original `9bd927d8`)
|
|
- Related: bug-1964, bug-1685
|
|
|