A-batch admin-plane fixes: rusage unit name, XCE_JOBID identity, select WARN, CLI env, docs
rusage collection (merge cb0c7fe9 regression, root cause of 0 counters):
- collect_job_rusages builds the systemd unit name with
make_job_sd_unit_name (four-segment fmt_jobid_ex form) instead of a bare
%lu of the packed uint64, so the D-Bus property lookups hit the real
per-job unit again. E2E on t1: DONE job reports max_mem/avg_mem/cpu_time
non-zero (previously all 0).
XCE_JOBID written identity:
- exdata sets XCE_JOBID via fmt_jobid (four-segment
cluster-apiserver-timestamp-seq) instead of a bare packed decimal;
XCE_JOBINDEX handling unchanged.
- xrun parses XCE_JOBID with parse_jobid (accepts the four-segment form);
a bare decimal no longer round-trips through the written identity.
- E2E on t1: job sees XCE_JOBID=<submit id>; xrun remote-start round-trip
to another host matches the job.
select WARN (silent degradation):
- parse_resreq_primary WARNs when a select references an unregistered
resource ("...references an unregistered resource -- dropping the select
and skipping any rusage[] clause of this primary") instead of failing
silently; scheduler log shows the WARN on t1 and the job schedules
without the constraint.
CLI env handling (unit-tested; subEnvVars is not flag-bound so this is
not reachable from the production CLI):
- getEnvironmentVariables keeps B=b=c (split at first '=') and warns +
drops a bare key or an '=value' fragment instead of silently mis-parsing.
Docs:
- src/docs/xce.md: document the dead-clause semantics (select/rusage
primary drop behavior) matching the WARN.
bug-1972 (test-only UAF): test_job_live_gc asserted job destruction by
reading refcount from the freed job; rewrite the assertion to observe
destruction without touching freed memory. controller_unit_test is green
again (1354 tests).
Regenerate 3 repl_trace_baseline fixtures whose baselines shifted with the
select-WARN logging; replay tests green.
Spec naming: external-license-resources design.md/brief.md fix the
self-referential daemon-name note (old drafts said xceextresd/extresd).
Verified on t1: rusage_burn job (counters non-zero in DONE), echo_env job
(four-segment XCE_JOBID), select[nosuchres] job (WARN + schedules),
xrun t2 remote-start round-trip, CLI env unit tests, xcelib 762 /
controller 1354 / worker exdata 8+11 / sched repl_trace 4+14+4+10+17
tests green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen
parent
68eabd8a4a
commit
f780a1660a
17 changed files with 329 additions and 44 deletions
|
|
@ -6,7 +6,7 @@
|
|||
"id": "bug-1971",
|
||||
"status": "fixed",
|
||||
"status_updated": "2026-10-22",
|
||||
"status_note": "",
|
||||
"status_note": "two-choke archive in xce-build.sh; test harness green",
|
||||
"component": "build",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-22T07:50:00Z",
|
||||
|
|
@ -27,6 +27,29 @@
|
|||
"68c3ccd7"
|
||||
],
|
||||
"record": ".wolf/bugs/bug-1971.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1972",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-26",
|
||||
"status_note": "fix in working tree, commit pending with A-batch",
|
||||
"component": "controller",
|
||||
"severity": "crash",
|
||||
"timestamp": "2026-10-26T09:46:58Z",
|
||||
"last_seen": "2026-10-26T09:46:58Z",
|
||||
"occurrences": 4,
|
||||
"error_message": "test_job_live_gc asserted destruction by reading freed memory\n\n**Observed:**\n`controller_unit_test` aborts under ASAN at `test_sweep_purges_expired_done_job`\n(test_job_live_gc.c:169):\n\n```\n==27834==ERROR: AddressSanitizer: heap-use-after-free on address 0x5110000da290\nREAD of size 8 at 0x5110000da290 thread T0\n #0 job_get_refcount ../lib/jobj.c:715\n #1 test_sweep_purges_expired_done_job test_job_live_gc.c:169\nfreed by:\n #3 job_dereference ../lib/jobj.c:705\n #5 job_live_gc_sweep ../controller/job.c:4777\n```\n\n**Expected:** a correct sweep dereferences the purged job down to refcount 0\nand destroys it; the test must verify destruction without touching the freed\nobject.\n\n**Impact:** controller suite aborts (killed by SIGABRT); the whole\n`controller test` meson entry fails. Test-only — the production sweep\n(job.c:4765-4777, double dereference: collection ref + table's makeJob ref)\nis correct.\n\n**Found by:** ASAN, first full-suite run after d028c8c3 (the archives' earlier\nruns all predate that commit).",
|
||||
"file": "src/controller/testsuite/test_job_live_gc.c",
|
||||
"root_cause": "The assertion `TEST_ASSERT_EQUAL_UINT32(0, job_get_refcount(expired))` runs\nAFTER `job_live_gc_sweep` has already dereferenced the job to 0 and freed the\nentire RTTI object (header + private + data in one calloc block). Reading the\nrefcount through the dangling `job_t*` is itself the use-after-free: the\nstronger the sweep does its job, the more surely the test crashes. The same\npattern is absent from test_apply_pump.c's refcount test because there the\njob is never destroyed.",
|
||||
"fix": "Move the refcount probe BEFORE the sweep and invert its meaning: assert\nrefcount == 1 (table's seed reference, nothing else) before `job_live_gc_sweep`,\nthen verify destruction by absence — `jlg_find_job(81001) == NULL` plus the\nidempotent second sweep (both already present). `expired` is never dereferenced\nafter the sweep; a comment marks the last safe point.\n\nVerification:\n- Test: test_sweep_purges_expired_done_job (same file) — RED before (deterministic\n ASAN abort 4/4), GREEN after (rc=0, 1354/1354 controller unity tests, zero\n ASAN reports).\n- Suites: controller_unit_test 1354 tests 0 failures (ASAN, detect_leaks=0);\n sched_repl_trace_baseline_test 4/4 with regenerated fixtures (separate issue,\n bug-free fixture staleness from fc33a5da's schema change, handled in the same\n working tree).\n- Live: n/a (test-only defect; production sweep unchanged).",
|
||||
"tags": [
|
||||
"test-only",
|
||||
"uaf",
|
||||
"gc"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1972.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
86
.wolf/bugs/bug-1972.md
Normal file
86
.wolf/bugs/bug-1972.md
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
---
|
||||
id: "bug-1972"
|
||||
status: "open"
|
||||
status_updated: "2026-10-26"
|
||||
status_note: "fix in working tree, commit pending with A-batch"
|
||||
component: "controller"
|
||||
severity: "crash"
|
||||
timestamp: "2026-10-26T09:46:58Z"
|
||||
last_seen: "2026-10-26T09:46:58Z"
|
||||
occurrences: 4
|
||||
files: ["src/controller/testsuite/test_job_live_gc.c"]
|
||||
tags: ["test-only", "uaf", "gc"]
|
||||
related: []
|
||||
fix_commits: []
|
||||
---
|
||||
|
||||
# bug-1972: test_job_live_gc asserted destruction by reading freed memory
|
||||
|
||||
## Description
|
||||
**Observed:**
|
||||
`controller_unit_test` aborts under ASAN at `test_sweep_purges_expired_done_job`
|
||||
(test_job_live_gc.c:169):
|
||||
|
||||
```
|
||||
==27834==ERROR: AddressSanitizer: heap-use-after-free on address 0x5110000da290
|
||||
READ of size 8 at 0x5110000da290 thread T0
|
||||
#0 job_get_refcount ../lib/jobj.c:715
|
||||
#1 test_sweep_purges_expired_done_job test_job_live_gc.c:169
|
||||
freed by:
|
||||
#3 job_dereference ../lib/jobj.c:705
|
||||
#5 job_live_gc_sweep ../controller/job.c:4777
|
||||
```
|
||||
|
||||
**Expected:** a correct sweep dereferences the purged job down to refcount 0
|
||||
and destroys it; the test must verify destruction without touching the freed
|
||||
object.
|
||||
|
||||
**Impact:** controller suite aborts (killed by SIGABRT); the whole
|
||||
`controller test` meson entry fails. Test-only — the production sweep
|
||||
(job.c:4765-4777, double dereference: collection ref + table's makeJob ref)
|
||||
is correct.
|
||||
|
||||
**Found by:** ASAN, first full-suite run after d028c8c3 (the archives' earlier
|
||||
runs all predate that commit).
|
||||
|
||||
## Environment
|
||||
- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64
|
||||
- libc: glibc 2.39 (host ldd)
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: asan (xce.build)
|
||||
- Code: feature/license-resources @ 68eabd8a-dirty (introduced in d028c8c3)
|
||||
- Deployment: n/a (unit test)
|
||||
|
||||
## Reproduction
|
||||
1. `cd src/xce.build && ASAN_OPTIONS=detect_leaks=0 ./controller/controller_unit_test`
|
||||
- Frequency: deterministic (4/4 runs, 2026-10-26)
|
||||
- Evidence: /home/lchen/code/xce/src/testlogs.archive/xce.build/this-20261026-082318-rc1
|
||||
|
||||
## Root cause
|
||||
The assertion `TEST_ASSERT_EQUAL_UINT32(0, job_get_refcount(expired))` runs
|
||||
AFTER `job_live_gc_sweep` has already dereferenced the job to 0 and freed the
|
||||
entire RTTI object (header + private + data in one calloc block). Reading the
|
||||
refcount through the dangling `job_t*` is itself the use-after-free: the
|
||||
stronger the sweep does its job, the more surely the test crashes. The same
|
||||
pattern is absent from test_apply_pump.c's refcount test because there the
|
||||
job is never destroyed.
|
||||
|
||||
## Fix
|
||||
Move the refcount probe BEFORE the sweep and invert its meaning: assert
|
||||
refcount == 1 (table's seed reference, nothing else) before `job_live_gc_sweep`,
|
||||
then verify destruction by absence — `jlg_find_job(81001) == NULL` plus the
|
||||
idempotent second sweep (both already present). `expired` is never dereferenced
|
||||
after the sweep; a comment marks the last safe point.
|
||||
|
||||
## Verification
|
||||
- Test: test_sweep_purges_expired_done_job (same file) — RED before (deterministic
|
||||
ASAN abort 4/4), GREEN after (rc=0, 1354/1354 controller unity tests, zero
|
||||
ASAN reports).
|
||||
- Suites: controller_unit_test 1354 tests 0 failures (ASAN, detect_leaks=0);
|
||||
sched_repl_trace_baseline_test 4/4 with regenerated fixtures (separate issue,
|
||||
bug-free fixture staleness from fc33a5da's schema change, handled in the same
|
||||
working tree).
|
||||
- Live: n/a (test-only defect; production sweep unchanged).
|
||||
|
||||
## References
|
||||
- Related: none
|
||||
Loading…
Add table
Add a link
Reference in a new issue