admin-plane breakpoint 3: done-job live GC

Terminal jobs (DONE/EXIT/PERR) were only filtered from checkpoints and
WAL replay (job_should_retain, finished_retain_s=3600) — the live jobmgr
never dropped them, so joblist and the scheduler's core->jobs view grew
without bound across a controller's uptime (N4).

Three pieces:

1. kvmgr delete path (src/lib/kvmgr.c): xkvmgr_type had .del_key=NULL
   and .vclear=NULL, so key_del()/vclear() on a kvmgr only returned
   -EINVAL. New xkvmgr_del_key (key_find to locate the slot idx ->
   key_del — kvtab's val_killer is NULL so the idx scalar is untouched ->
   vclear(data, idx)) and xkvmgr_vclear forwarder, wired into the
   typeinfo. A missing key propagates -ENOENT (same contract as
   xkvmgr_find).

2. Controller periodic sweep (job.c): job_live_gc_sweep — primary-only
   (standbys rebuild their live table from WAL replay; deleting there
   diverges), collects !job_should_retain jobs under the jobmgr READ
   lock with reference() (check_workers_timeout collect pattern), then
   per-job WRITE-locked key_del. TWO dereferences per purge: the
   collection reference and the table's own makeJob reference — the
   jobmgr's datakiller (job_obj_killer) is a deliberate no-op, so
   key_del transfers the table's ownership duty to the caller; one
   dereference would leak at refcount 1 forever (the very accumulation
   this sweep drains). Registered by cluster_start_job_live_gc with
   ha_start_primary_heartbeat's idempotent static-arm pattern +
   delaymgr.running guard (60s, runcount=-1).

3. Registration on both promotion tails (ha.c): raft road after
   ha_start_primary_heartbeat and the non-raft ha_promote_to_primary.
   NOT cluster_start — it is orphaned on the HA myapp_run path.

Safety: apply_committed_entry's jobstate group and WAL_JOB_UPDATE
already treat a key_find miss as a benign skip (bug-1970, which
explicitly anticipates post-purge facts); persist_verify_apply_correctness
counts only within the boot replay window. The TRANSIENT TTL now takes
live effect too (previously checkpoint-exclusion only).

UTs: kvmgr del INT+STRING (miss propagation, sibling intact, freed-slot
reuse, killer invocation), controller test_job_live_gc.c (expired DONE
purged + refcount 0, fresh/RUN retained, idempotent re-run, EXIT/PERR,
TRANSIENT TTL + transient_since==0 guard, standby no-op, NULL-safe,
arming guard). xcelib 761/761, controller 1354/1354.

E2E (t1): promote arms the sweep; the pre-restart DONE array jobs were
purged across five sweeps (33 jobs) as their 3600s windows elapsed;
submit→RUN→END flows and all daemons healthy throughout.

Co-Authored-By: Claude Code <noreply@anthropic.com>

src/controller/job.c

@@ -4676,3 +4676,144 @@ int on_fetchfile(cluster_t* c, endpoint_t* caller_ep, xce_request_t* request)
 /* job_is_terminal and job_should_retain live in persist.c —
  * they only depend on xce_job_t fields and persist_config_t,
  * and must be accessible from the unit test binary which links persist.c. */
+
+/* ============================================================
+ * admin-plane-gaps 断点③: done-job live GC
+ *
+ * Terminal jobs (DONE/EXIT/PERR) were purged from checkpoints/WAL replay
+ * by job_should_retain, but never from the live jobmgr — every finished
+ * job stayed resident in c->jobmgr until restart, so joblist and the
+ * scheduler's core->jobs table grew without bound across a controller's
+ * uptime. This sweep applies the same retention predicate to the live
+ * table: once finished_retain_s has elapsed past finish_time, the job is
+ * unlinked (key_del) and its reference dropped.
+ *
+ * Ownership: the jobmgr's datakiller (job_obj_killer, cluster.c) is a
+ * deliberate no-op — job_t is refcounted RTTI and its teardown runs via
+ * dereference() only. So each purge is: key_del (unlink + clear the
+ * varray slot, which does NOT free the value) followed by dereference.
+ *
+ * Lock discipline (B5, mirrors mark_jobs_unknown_core's comment): the
+ * caller must not hold any jobmgr lock. Collect expired jobs under the
+ * jobmgr READ lock (referencing each), then re-lock per job in WRITE
+ * mode to unlink — a jobmgr WRITE lock cannot nest inside the READ hold
+ * we need for iteration. A job finishing between the collect and the
+ * unlink is harmless: it was already terminal when collected.
+ * ============================================================ */
+
+/* delaymgr registration interval for the live-GC sweep. The sweep itself
+ * is idempotent (expired jobs are simply absent), so a fixed 60s cadence
+ * adds at most a minute of retention slack on top of finished_retain_s. */
+#define JOB_LIVE_GC_INTERVAL_S 60
+
+void job_live_gc_sweep(cluster_t* c)
+{
+    iterator_t it;
+    xlist_t    doomed;
+    job_t*     j;
+    int        purged = 0;
+
+    if (c == NULL || c->jobmgr == NULL) {
+        return;
+    }
+
+    /* Standbys rebuild their live table from WAL replay; deleting there
+     * would diverge from the replay stream (apply_committed_entry's
+     * jobstate group re-materializes on demand). Primary-only, matching
+     * the retention predicate's checkpoint-side role. */
+    if (!ha_is_primary(&c->ha)) {
+        return;
+    }
+
+    xlist_init(&doomed, 64);
+
+    /* Phase 1 — collect under READ, referencing each match so the
+     * pointer stays valid after the lock drops. Mirrors
+     * check_workers_timeout's collect pattern. */
+    lock(c->jobmgr, RTTI_LOCK_READ);
+    if (iter_make(&it, c->jobmgr) < 0) {
+        unlock(c->jobmgr);
+        xlist_fini(&doomed);
+        return;
+    }
+    for (;;) {
+        int rc = iter_next(&it, (void**)&j);
+        if (rc < 0 || rc == RTTI_ITER_NO_DATA) {
+            break;
+        }
+        if (j == NULL) {
+            continue;
+        }
+        if (!job_should_retain(j, &c->persist.config)) {
+            reference(j);
+            xlist_store(&doomed, j);
+        }
+    }
+    iter_destroy(&it);
+    unlock(c->jobmgr);
+
+    /* Phase 2 — per-job WRITE-locked unlink. key_del needs the jobmgr
+     * WRITE lock, which cannot nest inside the phase-1 READ hold.
+     *
+     * Two dereferences per purge: (1) the collection reference taken in
+     * phase 1, and (2) the TABLE's own reference. makeJob gave the table
+     * one reference (refcount 1) and the jobmgr's datakiller
+     * (job_obj_killer, cluster.c) is a deliberate no-op, so unlinking via
+     * key_del transfers the table's ownership duty to this caller —
+     * without the second dereference the job_t would leak at refcount 1
+     * forever (the very accumulation this sweep exists to drain). */
+    for (int i = 0; i < doomed.count; i++) {
+        j = xlist_get(&doomed, i);
+        if (j == NULL) {
+            continue;
+        }
+
+        lock(c->jobmgr, RTTI_LOCK_WRITE);
+        key_del(c->jobmgr, (void*)j->job.jobid);
+        unlock(c->jobmgr);
+
+        purged++;
+        dereference(j); /* phase-1 collection reference */
+        dereference(j); /* the table's makeJob reference */
+    }
+
+    xlist_fini(&doomed);
+
+    if (purged > 0) {
+        INFO("job live GC: purged %d finished job(s) from the live table",
+             purged);
+    }
+}
+
+/* Idempotent periodic arming for job_live_gc_sweep — same
+ * once-per-process pattern as ha_start_primary_heartbeat (with
+ * runcount=-1 the delaymgr re-enqueues the task itself, so duplicate
+ * calls on re-election must not stack tasks). Safe on a zeroed
+ * cluster_t (unit tests): the delaymgr.running guard skips arming. */
+void cluster_start_job_live_gc(cluster_t* c)
+{
+    static _Atomic int s_armed = 0;
+    int                expected = 0;
+
+    if (c == NULL) {
+        return;
+    }
+
+    if (!atomic_compare_exchange_strong(&s_armed, &expected, 1)) {
+        return; /* already armed — task is still running */
+    }
+
+    if (!c->delaymgr.running) {
+        INFO("job live GC skipped (delaymgr not running)");
+        return;
+    }
+
+    int rc = delay_task(&c->delaymgr, JOB_LIVE_GC_INTERVAL_S, -1,
+                        (delay_task_entry_fn_t)job_live_gc_sweep, c);
+    if (rc < 0) {
+        WARN("cluster_start_job_live_gc: delay_task failed: %d", rc);
+    } else {
+        INFO("job live GC armed (interval=%ds, rc=%d)",
+             JOB_LIVE_GC_INTERVAL_S, rc);
+    }
+}