Terminal jobs (DONE/EXIT/PERR) were only filtered from checkpoints and
WAL replay (job_should_retain, finished_retain_s=3600) — the live jobmgr
never dropped them, so joblist and the scheduler's core->jobs view grew
without bound across a controller's uptime (N4).
Three pieces:
1. kvmgr delete path (src/lib/kvmgr.c): xkvmgr_type had .del_key=NULL
and .vclear=NULL, so key_del()/vclear() on a kvmgr only returned
-EINVAL. New xkvmgr_del_key (key_find to locate the slot idx ->
key_del — kvtab's val_killer is NULL so the idx scalar is untouched ->
vclear(data, idx)) and xkvmgr_vclear forwarder, wired into the
typeinfo. A missing key propagates -ENOENT (same contract as
xkvmgr_find).
2. Controller periodic sweep (job.c): job_live_gc_sweep — primary-only
(standbys rebuild their live table from WAL replay; deleting there
diverges), collects !job_should_retain jobs under the jobmgr READ
lock with reference() (check_workers_timeout collect pattern), then
per-job WRITE-locked key_del. TWO dereferences per purge: the
collection reference and the table's own makeJob reference — the
jobmgr's datakiller (job_obj_killer) is a deliberate no-op, so
key_del transfers the table's ownership duty to the caller; one
dereference would leak at refcount 1 forever (the very accumulation
this sweep drains). Registered by cluster_start_job_live_gc with
ha_start_primary_heartbeat's idempotent static-arm pattern +
delaymgr.running guard (60s, runcount=-1).
3. Registration on both promotion tails (ha.c): raft road after
ha_start_primary_heartbeat and the non-raft ha_promote_to_primary.
NOT cluster_start — it is orphaned on the HA myapp_run path.
Safety: apply_committed_entry's jobstate group and WAL_JOB_UPDATE
already treat a key_find miss as a benign skip (bug-1970, which
explicitly anticipates post-purge facts); persist_verify_apply_correctness
counts only within the boot replay window. The TRANSIENT TTL now takes
live effect too (previously checkpoint-exclusion only).
UTs: kvmgr del INT+STRING (miss propagation, sibling intact, freed-slot
reuse, killer invocation), controller test_job_live_gc.c (expired DONE
purged + refcount 0, fresh/RUN retained, idempotent re-run, EXIT/PERR,
TRANSIENT TTL + transient_since==0 guard, standby no-op, NULL-safe,
arming guard). xcelib 761/761, controller 1354/1354.
E2E (t1): promote arms the sweep; the pre-restart DONE array jobs were
purged across five sweeps (33 jobs) as their 3600s windows elapsed;
submit→RUN→END flows and all daemons healthy throughout.
Co-Authored-By: Claude Code <noreply@anthropic.com>
