fix(controller): post-purge job facts must be benign skips, not fence-engaging failures (bug-1970)
A committed job-state fact (WAL_JOB_UPDATE/FAIL/START/...) whose jobid no
longer exists in the jobmgr used to return -ENOENT from
apply_committed_entry(). Both of its callers (replay_wal during boot,
persist_apply_pump_run live) halt-and-fence on rc < 0, engaging the Live
Apply Resource Fence. A finished job is purged from the live table once
finished_retain_s expires, but its committed WAL entries remain; a boot
replay walks the full log and re-hits a late post-finish UPDATE for the
purged jobid on EVERY restart. No in-memory state change can ever make a
purged job re-appear, so the fence can never self-heal: the controller
logs "promotion FENCED -- durable_floor < cluster HWM=0" every 30s
forever, rejects all component joins ("standby has no primary yet"), and
cannot serve. Observed live on the t1 rollout (2026-10-21): the old
primary had logged the same entry as "duplicate UPDATE to state 0x4 --
no-op" while the job was still in memory; the first restart after the
job's purge tripped the fence. Pre-existing since the July 2026
apply_committed_entry dispatch table (2fa955b9/387ea35c) -- exposed by
any controller restart, not a de-nix regression.
Fix: all three case groups (shared START/FINISH/STOP/RESUME/RUSAGE/CKPT/
PREASON/DISPATCH group, WAL_JOB_FAIL, WAL_JOB_UPDATE) now return 0 for a
missing job, mirroring apply_wal_entry()'s existing behavior on the same
seam (restore_job_state()'s rc discarded; the FAIL branch guards on
j != NULL) and this function's own WAL_QUEUE_CTRL benign-skip precedent.
Genuine construction/ENOMEM failures keep their fence semantics.
Also:
- test_apply_committed_entry: three new tests covering the skip for all
three groups (incl. the POSTDONE-mark variant), asserting rc == 0 and
no hook fires.
- lib/meson.build: ASAN_OPTIONS=detect_leaks=0 on the capnp generation
command. capnpc-c deliberately never frees its compile-time structures
(one-shot tool); under the de-nix asan profile the wrap-built generator
carries -fsanitize=address and LSAN's end-of-process check failed the
generation step. The legacy nix build never propagated -fsanitize to
the subproject. Same exemption convention as every other one-shot/
never-torn-down boundary in this repo.
- xce-package.sh: ALWAYS-bundle name list (libcyaml/libyaml/libtirpc/
libcgroup/libhwloc/libedit) alongside the ldd-location rule -- libcyaml
and libcgroup proved missing on the t2/t3 Ubuntu 24.04 runtime hosts
even though the build host resolves them from /usr/lib.
Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
parent
50cf424146
commit
6bf26cbab3
4 changed files with 178 additions and 10 deletions
|
|
@ -15,7 +15,17 @@ gen_srcs = custom_target(
|
|||
'controllercapnp.[ch]',
|
||||
output:['controller.capnp.c', 'controller.capnp.h'],
|
||||
input: ['../schema/controller.capnp', capnpc_c],
|
||||
# ASAN_OPTIONS=detect_leaks=0: capnpc-c (the code generator) deliberately
|
||||
# never frees its compile-time structures -- a one-shot tool whose process
|
||||
# exits immediately. Under the asan profile the wrap-subproject binary is
|
||||
# built with -fsanitize=address, and LSAN's default end-of-process leak
|
||||
# check then fails the generation step (exit 1, ~2MB reported across
|
||||
# capnpc-c.c allocations) -- de-nix build path only; the legacy nix build
|
||||
# never propagated -fsanitize to the subproject. Same exemption
|
||||
# convention as every other "one-shot, never-torn-down" boundary in this
|
||||
# repo (test('libxce test', ...) above uses the identical env var).
|
||||
command: [env_program, 'PATH=' + fs.parent(capnpc_c.full_path()) + ':/usr/bin:/bin',
|
||||
'ASAN_OPTIONS=detect_leaks=0',
|
||||
capnp, 'compile','-o' ,'c:lib', '-I', schema_dir, '--src-prefix', schema_dir,
|
||||
'@INPUT0@']
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue