Specs: scheduler-allocator-runner (#67: 317-319), scheduler-sim-decision-trace (#68: 320, 324, 325), job-status-detail-display (#69: 321-323), scheduler-sim-discrete-clock (#70: 326-328). Bugs: #329-#333. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
77 lines
8.4 KiB
Markdown
77 lines
8.4 KiB
Markdown
---
|
|
id: "bug-1882"
|
|
status: "fixed"
|
|
status_updated: "2026-09-28"
|
|
status_note: ""
|
|
component: "lib"
|
|
severity: "crash"
|
|
timestamp: "2026-09-28T08:08:11Z"
|
|
last_seen: "2026-09-28T08:08:11Z"
|
|
occurrences: 1
|
|
files: ["src/lib/xcealloc.c"]
|
|
tags: ["xcealloc", "oom", "null-deref", "merge_job_alloc", "bug-1863-followup"]
|
|
related: ["bug-1863"]
|
|
fix_commits: ["f9146864"]
|
|
gitea_issue: 333
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "719dc3dbbc28"
|
|
---
|
|
|
|
# bug-1882: libxce merge_job_alloc segfaults on an allocation whose strdup'd resname/host is NULL after OOM
|
|
|
|
## Description
|
|
**Observed:** with the first `strdup()` inside `make_alloc()` forced to fail (as it does under memory exhaustion), `make_alloc()` still returns a non-NULL `xce_alloc_t` whose `resname` is NULL, and a later `merge_job_alloc()` over that record crashes:
|
|
```
|
|
==3836272==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7ffff78e3d4d bp 0x7fffffff7020 sp 0x7fffffff67b0 T0)
|
|
==3836272==The signal is caused by a READ memory access.
|
|
#0 0x7ffff78e3d4d in strcmp.part.0 (.../gcc-15.2.0-lib/lib/libasan.so.8+0xe3d4d)
|
|
#1 0x55555556305d in same_alloc_slot ../lib/xcealloc.c:337
|
|
#2 0x5555555632be in merge_allocs ../lib/xcealloc.c:354
|
|
#3 0x555555563a5c in merge_job_alloc ../lib/xcealloc.c:401
|
|
#4 0x555555561290 in main /home/claude/tmp/bug-1882/repro.c:31
|
|
SUMMARY: AddressSanitizer: SEGV ../lib/xcealloc.c:337 in same_alloc_slot
|
|
```
|
|
**Expected:** `make_alloc()`/`make_host_alloc()` report the allocation failure (return NULL, freeing what they built) so no record with a NULL `resname`/`reserve`/`host` exists; `merge_job_alloc()` then never sees one and keeps its documented `-ENOMEM` convention.
|
|
**Impact:** crash (NULL dereference) of `xce_scheduler`, only under memory exhaustion. The only production caller of `merge_job_alloc()` is `sched_make_compound_alloc_()` (`src/scheduler/src/scheduler.c:307`, bug-1863), which folds each COMPOUND primary's allocation into the job's merged allocation; every record there comes from `make_alloc()` at `scheduler.c:1893` with a non-NULL `r->name`, so a NULL `resname` needs a failed `strdup()`. Low likelihood, but the scheduler would segfault instead of failing the allocation with `-ENOMEM`.
|
|
**Found by:** review of spec `libxce-alloc-preason-equality` (2026-09-28): the new `same_alloc_slot()` helper added by bug-1863 calls `strcmp()` on `resname` without a NULL check; the reproducer above confirmed the path.
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (`/nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2`, the reproducer's interpreter); host ldd reports glibc 2.40
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (`src/xce.build`, `libxce.a` built 2026-09-28 04:31 UTC); reproducer compiled with `-fsanitize=address`, run with `ASAN_OPTIONS=detect_leaks=0`
|
|
- Code: scheduler-framework @ c02d8e5c-dirty (dirty files are `.wolf/`, `.claude/`, `.kiro/` only; `src/` clean; `src/lib/xcealloc.c` last changed in aea34738)
|
|
- Deployment: none; standalone reproducer linked against `libxce.a`
|
|
|
|
## Reproduction
|
|
1. Build libxce with ASAN: `/xce-build lib` (or `nix develop ./src --command bash src/tools/xce-build.sh lib asan`).
|
|
2. The reproducer `/home/claude/tmp/bug-1882/repro.c` wraps `strdup()` (`-Wl,--wrap=strdup`) so the next call returns NULL, calls `make_alloc("mem", "", 1, 1.0f, 0)` (its `resname` strdup fails), builds two one-host job allocations with the same host, and calls `merge_job_alloc(dst, src)`.
|
|
3. Build and run it:
|
|
```
|
|
nix develop ./src --command bash -c 'cd src/xce.build && gcc -g -fsanitize=address -I../include -o /home/claude/tmp/bug-1882/repro /home/claude/tmp/bug-1882/repro.c lib/libxce.a -Wl,--wrap=strdup && ASAN_OPTIONS=detect_leaks=0 /home/claude/tmp/bug-1882/repro'
|
|
```
|
|
- Frequency: deterministic (1/1 runs with the forced failure, 2026-09-28).
|
|
- Evidence: `/home/claude/tmp/bug-1882/repro.c` (reproducer), `/home/claude/tmp/bug-1882/asan.log` (full ASAN report).
|
|
|
|
## Root cause
|
|
- `make_alloc()` (`src/lib/xcealloc.c:21-25`) assigns `strdup(resname)` and `strdup(reserve)` without checking for NULL, and `make_host_alloc()` (`xcealloc.c:92`) does the same for `strdup(host)`; `dup_alloc()` goes through `make_alloc()`, so `dup_job_alloc()`/`merge_job_alloc()` copies inherit it. Under OOM these return a record with a NULL string field instead of NULL.
|
|
- `same_alloc_slot()` (`xcealloc.c:337`, added by bug-1863 in aea34738) passes `a->resname`/`b->resname` straight to `strcmp()`; `reserve` is guarded (`reserve ? reserve : ""`) but `resname` is not. `merge_job_alloc()` (`xcealloc.c:396`) likewise calls `strcmp(dst->allocs[j]->host, sh->host)` unguarded. A NULL field reaching either comparison dereferences NULL.
|
|
- Since: the unchecked `strdup()`s are pre-existing (origin unknown); the crash sites were introduced by aea34738 (bug-1863).
|
|
- Scope: `make_alloc()` also dereferenced a NULL `resname` argument (`strdup(NULL)`). The scheduler's allocator plugins (`seqjob_allocator.c` `my_alloc()`, `parjob_allocator.c` `my_alloc()`) and `sched_make_allocs()` (`scheduler.c`) checked no constructor result at all, and `sched_make_primary_allocation()` moves `ctx.alloc` onto the job whatever the rc, so making `make_alloc()` return NULL alone would only have turned the crash into an allocation silently missing a resource. Not fixed here (same unchecked-`strdup()` pattern, accounting only, not compared): `make_res_account()` (`xcealloc.c:46`) and `make_host_account()` (`:119`). Not checked: decoders (capnp `decode_*`) that may produce a NULL `resname`/`host`.
|
|
|
|
## Fix
|
|
- `src/lib/xcealloc.c`: `make_alloc()` rejects a NULL `resname` (NULL, errno `EINVAL`) and checks both `strdup()`s, freeing what it built and returning NULL (errno `ENOMEM`) on failure. `make_host_alloc()` does the same for `host`; on failure the caller keeps ownership of `allocs`. `dup_host_alloc()` frees its partial copy when a `dup_alloc()` or `make_host_alloc()` fails (it used to leak it). So no constructor returns a record with a NULL string, and `merge_job_alloc()` (which already maps a NULL from `dup_host_alloc()`/`dup_alloc()` to `-ENOMEM`) reports OOM instead of crashing.
|
|
- `src/scheduler/src/scheduler.c` `sched_make_allocs()`: if `make_alloc()` fails for a resource, free the array and return NULL (errno `ENOMEM`) instead of leaving a NULL entry.
|
|
- `src/scheduler/src/plugins/seqjob_allocator.c`, `parjob_allocator.c` `my_alloc()`: check `sched_make_allocs()`, `make_host_alloc()`, `make_host_allocs()`, `make_job_alloc()` and the shared `sched_make_allocs()`; on failure free what was built (parjob also rolls back the claims not yet turned into host records and finishes the plan iterator) and return `-ENOMEM` with `ctx->alloc` NULL, the existing infrastructure-error contract (`rc < 0`: never cached as unsatisfiable, never committed).
|
|
- Why this removes the cause: the NULL string can no longer be created, and every caller up to the allocator turns the failure into the error return the scheduler already handles.
|
|
|
|
## Verification
|
|
- Test: `src/lib/testsuite/test_xcealloc_oom.c` (new binary `xcealloc_oom_test`, `-Wl,--wrap=strdup`), 8 tests: RED before the fix (first test SEGV in `make_alloc` via `strdup(NULL)`; with the two NULL-argument tests skipped, 5 of the other 6 FAIL, incl. `merge_job_alloc ... Expected -12 Was 0`), GREEN after (8/8, with LeakSanitizer on).
|
|
- Test: `test_sched_make_allocs_returns_null_when_a_record_fails` in `src/scheduler/testsuite/test_sched_make_allocs_oom.c` (new binary `sched_make_allocs_oom_test`, real `scheduler_srcs` + `--wrap=strdup`): RED (non-NULL array with a NULL entry), GREEN after.
|
|
- Test: `test_my_alloc_reports_enomem_when_allocs_cannot_be_built` in `test_seqjob_allocator.c` and in `test_parjob_combinatorial.c` (stubbed `sched_make_allocs()` armed to fail; parjob fails on host 0 and on host 1): RED `Expected -12 Was 1` in both, GREEN after.
|
|
- Suites: `xce-build.sh all asan` (2026-09-28): 115/115 meson test suites OK, Go unit tests OK. No new compiler warnings in the changed files. RED/GREEN logs: `/home/claude/tmp/bug-1882/red.log`, `green.log`, `build-all.log`.
|
|
- Live: n/a. The failure needs a `strdup()` to fail under memory exhaustion, which a deployment cannot trigger deterministically; the unit tests force exactly that path.
|
|
|
|
## References
|
|
- Related: bug-1863 (introduced the crashing `same_alloc_slot()`/`merge_job_alloc()` comparisons)
|
|
- Docs: `.kiro/specs/libxce-alloc-preason-equality/design.md` (Decision 6)
|