f780a1660a
rusage collection (merge cb0c7fe9 regression, root cause of 0 counters):
- collect_job_rusages builds the systemd unit name with
make_job_sd_unit_name (four-segment fmt_jobid_ex form) instead of a bare
%lu of the packed uint64, so the D-Bus property lookups hit the real
per-job unit again. E2E on t1: DONE job reports max_mem/avg_mem/cpu_time
non-zero (previously all 0).
XCE_JOBID written identity:
- exdata sets XCE_JOBID via fmt_jobid (four-segment
cluster-apiserver-timestamp-seq) instead of a bare packed decimal;
XCE_JOBINDEX handling unchanged.
- xrun parses XCE_JOBID with parse_jobid (accepts the four-segment form);
a bare decimal no longer round-trips through the written identity.
- E2E on t1: job sees XCE_JOBID=<submit id>; xrun remote-start round-trip
to another host matches the job.
select WARN (silent degradation):
- parse_resreq_primary WARNs when a select references an unregistered
resource ("...references an unregistered resource -- dropping the select
and skipping any rusage[] clause of this primary") instead of failing
silently; scheduler log shows the WARN on t1 and the job schedules
without the constraint.
CLI env handling (unit-tested; subEnvVars is not flag-bound so this is
not reachable from the production CLI):
- getEnvironmentVariables keeps B=b=c (split at first '=') and warns +
drops a bare key or an '=value' fragment instead of silently mis-parsing.
Docs:
- src/docs/xce.md: document the dead-clause semantics (select/rusage
primary drop behavior) matching the WARN.
bug-1972 (test-only UAF): test_job_live_gc asserted job destruction by
reading refcount from the freed job; rewrite the assertion to observe
destruction without touching freed memory. controller_unit_test is green
again (1354 tests).
Regenerate 3 repl_trace_baseline fixtures whose baselines shifted with the
select-WARN logging; replay tests green.
Spec naming: external-license-resources design.md/brief.md fix the
self-referential daemon-name note (old drafts said xceextresd/extresd).
Verified on t1: rusage_burn job (counters non-zero in DONE), echo_env job
(four-segment XCE_JOBID), select[nosuchres] job (WARN + schedules),
xrun t2 remote-start round-trip, CLI env unit tests, xcelib 762 /
controller 1354 / worker exdata 8+11 / sched repl_trace 4+14+4+10+17
tests green.
Co-Authored-By: Claude Code <noreply@anthropic.com>
55 lines
8.7 KiB
JSON
55 lines
8.7 KiB
JSON
{
|
|
"version": 1,
|
|
"generated": "by scripts/buglog.py from .wolf/bugs/*.md -- do not edit",
|
|
"bugs": [
|
|
{
|
|
"id": "bug-1971",
|
|
"status": "fixed",
|
|
"status_updated": "2026-10-22",
|
|
"status_note": "two-choke archive in xce-build.sh; test harness green",
|
|
"component": "build",
|
|
"severity": "wrong-result",
|
|
"timestamp": "2026-10-22T07:50:00Z",
|
|
"last_seen": "2026-10-22T07:50:00Z",
|
|
"occurrences": 2,
|
|
"error_message": "next xce-build.sh run destroys the prior testlog — flaky-failure evidence unrecoverable\n\n**Observed:** (t2, 2026-10-22) a full `make build` (asan profile) failed on a controller test case; a following `make clean` + `make build` passed 134/134. The failing run's `meson-logs/testlog.{txt,json,junit.xml}` had been destroyed twice over: `setup_build`'s unconditional `rm -rf \"$BUILD_DIR\"` deletes the whole build dir (meson-logs included) before the retry, and meson itself overwrites `testlog.*` on every `meson test` invocation regardless. Nothing in the surviving tree identifies which case failed: the worker suite's daemon logs (`/tmp/test_xcecd_resync.log` and rotations) showed no failure signature, and no `sched_test`/Go test output is retained anywhere.\n\nReconstruction from filesystem timestamps alone (`/tmp/test_*.log` rotations, `meson-log.txt` build options line, `/usr/local/go` ctime, bash history) established the two-run timeline but could not name the failed case.\n\nThe same evidence-loss shape hit t1 earlier (2026-10-21, the two controller flakes fixed in 5b9d7270) — there the testlog happened to still exist because no clean rebuild ran in between; that was luck, not design.\n\n**Expected:** when a build/test run fails, its meson-logs (testlog.txt/json/junit.xml) and any per-case fixtures it kept (the mkdtemp sweep contract deliberately preserves FAILED cases' `/tmp/xce_*` dirs) survive at least until the next run starts — so a flaky failure can be diagnosed from disk afterwards without re-running and hoping to reproduce.\n\n**Impact:** intermittent/flaky test failures (the exact class this project keeps hitting: 5b9d7270's two flakes, bug-1521's random heap corruption, the t2 2026-10-22 case) leave zero durable evidence. Each recurrence costs a full rebuild cycle and often fails to reproduce. Also buries the mkdtemp \"failed cases keep their dirs for post-mortem\" contract, since the next `make build` never looks for those dirs and a clean build gives the appearance of none existing.\n\n**Found by:** user report (t2 build failure, 2026-10-22) + post-mortem on the t2 filesystem.",
|
|
"file": "src/tools/xce-build.sh",
|
|
"root_cause": "`setup_build()` in `src/tools/xce-build.sh` unconditionally `rm -rf \"$BUILD_DIR\"` on every non-incremental invocation, and `meson test` overwrites `$BUILD_DIR/meson-logs/testlog.*` on every run. xce-build.sh never copies failure state anywhere durable, and nothing else in the tree does either.",
|
|
"fix": "Two chokes in `src/tools/xce-build.sh` (this commit):\n\n1. `archive_previous_failure()` — runs at script start, before `setup_build`'s\n wipe. If the marker `.meson_test_failed_rc` (written by the previous run's\n EXIT trap) exists, the previous run's `meson-logs` are MOVED to\n `$SRC_DIR/testlogs.archive/<BUILD_DIR>/prev-<stamp>-rc<N>/` — outside the\n build dir, so the wipe can't touch them. It also sweeps aged (>5 min, i.e.\n not a concurrent session's) `/tmp/xce_*` FAILED-case fixture DIRS into\n `prev-<stamp>-fixtures/`, and caps the archive at the newest 10 entries\n per build dir.\n2. `finalize_failure_evidence()` — the EXIT trap (replaces the old\n lock-release trap, which becomes its last statement). On rc≠0 with\n RUN_TESTS=1, COPIES `meson-logs` to\n `testlogs.archive/<BUILD_DIR>/this-<stamp>-rc<N>/` and writes the marker.\n Copy-not-move: meson-logs still belongs to the live build dir for this\n run's own inspection; the next run moves the copy out at its start.\n Configure/compile failures (RUN_TESTS unset) archive nothing — those are\n deterministic and self-explanatory on rerun; it's the testlog-bearing\n flaky failures that need preservation.\n\nAdditionally `-type d` added to BOTH `/tmp/xce_*` sweeps (this one and\nbug-924's `prune_stale_tmp_fixtures`): mkdtemp fixtures are always\ndirectories (`/tmp/xce_<label>_XXXXXX`), but the bare name also matches\nregular files — the untyped find live-moved daemon debug logs\n(`/tmp/xce_scheduler.log`, `/tmp/xce_strace*.log`) into the archive on\n2026-10-22, and the bug-924 variant `rm -rf`s its matches.\n\nVerification:\n- Test: `src/tools/test-failure-evidence-archive.sh` — 6 scenarios /\n 11 assertions exercising the real function bodies (sourced with surgical\n stubbing): failed run archives as this-*-rc3 + marker; success archives\n nothing; next run moves prior failure to prev-*-rc3 + consumes marker;\n aged fixture dirs move / fresh stay / regular files never touched;\n retention caps at newest 10; RUN_TESTS=0 archives nothing. All green\n (11 passed, 0 failed, bash, 2026-10-22).\n- Suites: n/a — this bug lives in the build script, not in C code; no meson\n suite exercises it. `bash -n` syntax check green.\n- Real wipe path not exercised by the harness test (it stubs meson);\n validated by inspection: the archive root is `$SRC_DIR/testlogs.archive/`,\n which `rm -rf \"$BUILD_DIR\"` cannot reach.\n- Live: first t2/sandbox failing run after this commit leaves\n `testlogs.archive/xce.build/this-*-rc*` with the testlog — that is the\n evidence this bug asks for.",
|
|
"tags": [
|
|
"test",
|
|
"flaky",
|
|
"observability",
|
|
"build"
|
|
],
|
|
"related_bugs": [],
|
|
"fix_commits": [
|
|
"68c3ccd7"
|
|
],
|
|
"record": ".wolf/bugs/bug-1971.md"
|
|
},
|
|
{
|
|
"id": "bug-1972",
|
|
"status": "open",
|
|
"status_updated": "2026-10-26",
|
|
"status_note": "fix in working tree, commit pending with A-batch",
|
|
"component": "controller",
|
|
"severity": "crash",
|
|
"timestamp": "2026-10-26T09:46:58Z",
|
|
"last_seen": "2026-10-26T09:46:58Z",
|
|
"occurrences": 4,
|
|
"error_message": "test_job_live_gc asserted destruction by reading freed memory\n\n**Observed:**\n`controller_unit_test` aborts under ASAN at `test_sweep_purges_expired_done_job`\n(test_job_live_gc.c:169):\n\n```\n==27834==ERROR: AddressSanitizer: heap-use-after-free on address 0x5110000da290\nREAD of size 8 at 0x5110000da290 thread T0\n #0 job_get_refcount ../lib/jobj.c:715\n #1 test_sweep_purges_expired_done_job test_job_live_gc.c:169\nfreed by:\n #3 job_dereference ../lib/jobj.c:705\n #5 job_live_gc_sweep ../controller/job.c:4777\n```\n\n**Expected:** a correct sweep dereferences the purged job down to refcount 0\nand destroys it; the test must verify destruction without touching the freed\nobject.\n\n**Impact:** controller suite aborts (killed by SIGABRT); the whole\n`controller test` meson entry fails. Test-only — the production sweep\n(job.c:4765-4777, double dereference: collection ref + table's makeJob ref)\nis correct.\n\n**Found by:** ASAN, first full-suite run after d028c8c3 (the archives' earlier\nruns all predate that commit).",
|
|
"file": "src/controller/testsuite/test_job_live_gc.c",
|
|
"root_cause": "The assertion `TEST_ASSERT_EQUAL_UINT32(0, job_get_refcount(expired))` runs\nAFTER `job_live_gc_sweep` has already dereferenced the job to 0 and freed the\nentire RTTI object (header + private + data in one calloc block). Reading the\nrefcount through the dangling `job_t*` is itself the use-after-free: the\nstronger the sweep does its job, the more surely the test crashes. The same\npattern is absent from test_apply_pump.c's refcount test because there the\njob is never destroyed.",
|
|
"fix": "Move the refcount probe BEFORE the sweep and invert its meaning: assert\nrefcount == 1 (table's seed reference, nothing else) before `job_live_gc_sweep`,\nthen verify destruction by absence — `jlg_find_job(81001) == NULL` plus the\nidempotent second sweep (both already present). `expired` is never dereferenced\nafter the sweep; a comment marks the last safe point.\n\nVerification:\n- Test: test_sweep_purges_expired_done_job (same file) — RED before (deterministic\n ASAN abort 4/4), GREEN after (rc=0, 1354/1354 controller unity tests, zero\n ASAN reports).\n- Suites: controller_unit_test 1354 tests 0 failures (ASAN, detect_leaks=0);\n sched_repl_trace_baseline_test 4/4 with regenerated fixtures (separate issue,\n bug-free fixture staleness from fc33a5da's schema change, handled in the same\n working tree).\n- Live: n/a (test-only defect; production sweep unchanged).",
|
|
"tags": [
|
|
"test-only",
|
|
"uaf",
|
|
"gc"
|
|
],
|
|
"related_bugs": [],
|
|
"fix_commits": [],
|
|
"record": ".wolf/bugs/bug-1972.md"
|
|
}
|
|
]
|
|
}
|