fix(controller): post-purge job facts must be benign skips, not fence-engaging failures (bug-1970)

A committed job-state fact (WAL_JOB_UPDATE/FAIL/START/...) whose jobid no
longer exists in the jobmgr used to return -ENOENT from
apply_committed_entry(). Both of its callers (replay_wal during boot,
persist_apply_pump_run live) halt-and-fence on rc < 0, engaging the Live
Apply Resource Fence. A finished job is purged from the live table once
finished_retain_s expires, but its committed WAL entries remain; a boot
replay walks the full log and re-hits a late post-finish UPDATE for the
purged jobid on EVERY restart. No in-memory state change can ever make a
purged job re-appear, so the fence can never self-heal: the controller
logs "promotion FENCED -- durable_floor < cluster HWM=0" every 30s
forever, rejects all component joins ("standby has no primary yet"), and
cannot serve. Observed live on the t1 rollout (2026-10-21): the old
primary had logged the same entry as "duplicate UPDATE to state 0x4 --
no-op" while the job was still in memory; the first restart after the
job's purge tripped the fence. Pre-existing since the July 2026
apply_committed_entry dispatch table (2fa955b9/387ea35c) -- exposed by
any controller restart, not a de-nix regression.

Fix: all three case groups (shared START/FINISH/STOP/RESUME/RUSAGE/CKPT/
PREASON/DISPATCH group, WAL_JOB_FAIL, WAL_JOB_UPDATE) now return 0 for a
missing job, mirroring apply_wal_entry()'s existing behavior on the same
seam (restore_job_state()'s rc discarded; the FAIL branch guards on
j != NULL) and this function's own WAL_QUEUE_CTRL benign-skip precedent.
Genuine construction/ENOMEM failures keep their fence semantics.

Also:
- test_apply_committed_entry: three new tests covering the skip for all
  three groups (incl. the POSTDONE-mark variant), asserting rc == 0 and
  no hook fires.
- lib/meson.build: ASAN_OPTIONS=detect_leaks=0 on the capnp generation
  command. capnpc-c deliberately never frees its compile-time structures
  (one-shot tool); under the de-nix asan profile the wrap-built generator
  carries -fsanitize=address and LSAN's end-of-process check failed the
  generation step. The legacy nix build never propagated -fsanitize to
  the subproject. Same exemption convention as every other one-shot/
  never-torn-down boundary in this repo.
- xce-package.sh: ALWAYS-bundle name list (libcyaml/libyaml/libtirpc/
  libcgroup/libhwloc/libedit) alongside the ldd-location rule -- libcyaml
  and libcgroup proved missing on the t2/t3 Ubuntu 24.04 runtime hosts
  even though the build host resolves them from /usr/lib.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-21 02:21:22 +00:00
parent 50cf424146
commit 6bf26cbab3
4 changed files with 178 additions and 10 deletions

View file

@ -15,7 +15,17 @@ gen_srcs = custom_target(
'controllercapnp.[ch]',
output:['controller.capnp.c', 'controller.capnp.h'],
input: ['../schema/controller.capnp', capnpc_c],
# ASAN_OPTIONS=detect_leaks=0: capnpc-c (the code generator) deliberately
# never frees its compile-time structures -- a one-shot tool whose process
# exits immediately. Under the asan profile the wrap-subproject binary is
# built with -fsanitize=address, and LSAN's default end-of-process leak
# check then fails the generation step (exit 1, ~2MB reported across
# capnpc-c.c allocations) -- de-nix build path only; the legacy nix build
# never propagated -fsanitize to the subproject. Same exemption
# convention as every other "one-shot, never-torn-down" boundary in this
# repo (test('libxce test', ...) above uses the identical env var).
command: [env_program, 'PATH=' + fs.parent(capnpc_c.full_path()) + ':/usr/bin:/bin',
'ASAN_OPTIONS=detect_leaks=0',
capnp, 'compile','-o' ,'c:lib', '-I', schema_dir, '--src-prefix', schema_dir,
'@INPUT0@']
)