myxce/.kiro/specs/controller-job-cleanup/brief.md

# Brief: controller-job-cleanup

## Problem
Operators of long-running XCE clusters have no way to get rid of finished jobs.
- The controller keeps every job it has ever accepted in memory, forever. So do
  the scheduler (`core->jobs`) and the apiserver store.
- Memory use, full dataset pushes (scheduler join, apiserver connect), and
  `xcecli jobs list` output all grow without bound.
- A hidden, hard-coded one-hour rule drops finished jobs only from **snapshots**,
  not from memory. Nodes therefore disagree about which jobs exist:
  - a restarted node holds a different job set than one that never restarted;
  - a standby that installs a snapshot keeps jobs the snapshot omits.

## Current State
Codebase exploration, 2026-09-29.

**Job table**
- Jobs live in `c->jobmgr`, an `xkvmgr` table with integer keys
  (`controller/cluster.c:1668-1685`). It has **no delete operation**: `.del_key =
  NULL` (`src/lib/kvmgr.c:289`, bug-1443).
- The underlying pieces exist: `kvtab_del` (`kv.c:288`) and `vclear` (the rollback
  path at `kvmgr.c:107-128`).
- Jobs are reference-counted RTTI objects, and `job_obj_killer`
  (`cluster.c:1591`) does nothing.

**WAL**
- There is no "job removed" entry type (`include/wal_types.h:132-310`).
  Numbers 13-19 are free.
- A new type has to be handled in:
  - both apply paths, `apply_wal_entry` (`persist.c:4195`) and
    `apply_committed_entry` (`persist.c:4607`), with identical decoding
    (`persistence.md:68`);
  - `wal_classify()` (`persist.c:7395`, where an unclassified type asserts under
    XCE_TESTING);
  - the replay verdict. `persist_apply_verdict` (`persist.c:6660-6690`) treats a
    shrinking job table as DIVERGENT, and `apply_expectation_t` (`persist.h:162`)
    has to count removals.
- Fact append pattern to copy: `persist_submit_job_fact_ex` (`persist.c:8105`) →
  `wal_hook_append_classified(..., WAL_MODE_RETAIN, ...)`.

**The existing snapshot-only retention rule**
- `job_should_retain()` (`persist.c:8695-8714`) keeps a terminal (EXIT|DONE|PERR)
  job until `finish_time + finished_retain_s` has passed.
- `finished_retain_s` defaults to `DEFAULT_FINISHED_RETAIN_S=3600`
  (`include/config.h:91`) and cannot be configured.
- Its only caller is `collect_job_frames` (`persist.c:2513`), which runs at
  snapshot time, per node, using that node's `time(NULL)`.
- The same rule also expires TRANSIENT jobs after `transient_job_timeout_s`.
- `load_snapshot_entities` (`persist.c:3827`) only adds jobs and never clears the
  table.

**Terminal states**
- `JOB_IS_FINISH` = EXIT || DONE (`include/xce/types.h:425-429`).
- Two definitions of "terminal" disagree: `job.c:45` says DONE|EXIT, and
  `persist.c:8690` says EXIT|DONE|PERR.
- `finish_time` is set live (`job.c:2756`), carried in the WAL_JOB_FINISH payload,
  and restored on apply and resync.

**Periodic, primary-only work**
- Pattern: `ha_start_primary_heartbeat` (`ha.c:10142`) uses `delay_task_ms(...,
  -1, fn, c)`, and the callback does nothing unless `role == CTRL_PRIMARY`.
- Offload pattern: `checkpoint_task_fn` (`persist.c:3545-3592`) runs on the
  delaymgr thread and calls `slowlane_submit(...)` (`slowlane.c:185`).
- delaymgr is trigger-only (`ha-async-io.md`).

**Cluster config**
- Schema `ClusterConfig` (`schema/controller.capnp:850-869`); the next free ordinal
  is `@15`.
- C struct: `cluster_config_t` (`include/clusterconfig.h`).
- YAML and validation: `configs/xce.yaml` and `configs/schema/xce.schema.json`
  (`additionalProperties:false`).
- xceadmin: `_XCE_SCALAR_FIELDS` (`xceadmin/src/xcecommon/mapping.py:176`).
- Defaults: `myconf.c:65-76`.
- Live config upgrade swaps `myconf->cluster_config` (`config_repl.c:747`), so
  read the value on every sweep.
- Model to follow: `worker_resync_timeout_ms`.

**Downstream**
- Scheduler: no job-removed hook detail exists (`include/xce/cluster_def.h:43-67`),
  and `core->jobs` never deletes.
- Apiserver: `OnJobDeleted` (`backend/notification.go:11`, `handler.go:83`) exists
  but nothing calls it. Reconnect never prunes the store.
- libxce: querying a missing jobid returns 0 jobs with status 0
  (`opget.c:159-176`, `xceapi.c:787`). Control operations return `-ENOENT`.
- Worker resync ignores unknown jobids (`resync.c:638-647`), which is safe.
- Array jobs: there is no parent object. Each element is an independent job with
  its own jobid. Elements are numbered 1..N per `job-array-one-based-index`
  (2026-10-01): element k of array `<array-id>` is `<array-id>[k]`, k = 1..N
  (arrays submitted before that change may still have elements 0..N-1). The
  earlier note here, jobid in `[base, base+array_size)`, described numbering
  from 0.
- There is no accounting or history file, so a removed job leaves no record.

**Job dependencies**
- None exist (no field in `job_submit_data_t`). The user asked for a stub.

## Desired Outcome
- Finished jobs (DONE or EXIT) are removed permanently once they have been
  finished longer than a configured retention period and no other job depends on
  them.
- Every controller node removes the same jobs at the same point in the log.
  Removal survives replay, restart and failover, and snapshots match memory.
- The scheduler and apiserver drop removed jobs too, so a removed job disappears
  from `xcecli` and from REST.
- Both periods can be configured in the cluster configuration, and cleanup can be
  turned off.

## Approach
**Chosen: approach A, a replicated removal record** (user decision, 2026-09-29).

1. **Trigger (primary only).** A delaymgr timer fires every
   `job_cleanup_period`. The callback returns unless this node is PRIMARY, and
   otherwise hands the sweep to the slow lane. It never blocks.
2. **Sweep (slow lane).** Walk `jobmgr` under a read lock and pick jobs that meet
   all of these:
   - state is DONE or EXIT, not TRANSIENT;
   - `finish_time + job_retention < now`;
   - the dependency stub (`job_has_dependents(job)`, which always returns false
     for now) lets it go.

   Append the chosen jobids as new WAL job-removal facts (RETAIN), batched and
   bounded per sweep.
3. **Apply (every node, both apply paths).** Remove the job from `jobmgr` through
   a new `xkvmgr` delete operation that drops the table's reference; in-flight
   holders keep the job alive until they release it. Update the replay-verdict
   expectations so that removals are counted.
4. **Propagate.** Tell the scheduler to drop the job from `core->jobs` (a new
   hook detail or event), and tell the apiserver (wire up the existing
   `OnJobDeleted`).
5. **Retire the snapshot-only filter** for finished jobs, so a snapshot contains
   exactly the jobs in memory. The requirements decide what happens to the
   TRANSIENT-timeout part of `job_should_retain` (keep it separate or fold it in).

Rejected alternatives:
- **B, each node cleans up locally without replication.** Clock skew makes nodes
  disagree, and the replay verdict flags a shrinking table as DIVERGENT.
- **C, extend the snapshot filter.** Still decided per node, and makes the
  existing drift permanent.

## Scope
- **In**:
  - Two cluster configuration parameters, both configurable (names are
    provisional):
    - `job_retention`: how long a finished job is kept. Default 3600 s, matching
      today's snapshot rule. 0 turns cleanup off, and jobs are kept forever.
    - `job_cleanup_period`: the sweep interval. Default 300 s.
    - Both need schema, YAML, JSON-schema validation, xceadmin mapping, defaults
      and live-upgrade pickup.
  - The primary-only periodic sweep.
  - The dependency-check stub.
  - The new WAL removal fact, handled on both apply paths and in classification,
    with replay-verdict accounting.
  - The `xkvmgr` delete operation, safe with reference counts.
  - Removal on every node.
  - Scheduler and apiserver propagation.
  - Retiring the snapshot-only filter for finished jobs, so snapshot and memory
    agree.
  - Tests: unit, persistence round-trip, and live HA.
- **Out**:
  - A real job-dependency feature. Only the stub is in scope.
  - Accounting or job-history archiving before removal.
  - Manual per-job purge commands.
  - Cleaning up PEND, RUN or other non-terminal jobs.
  - The lost-and-found drain (`roadmap.md`).

## Boundary Candidates
- libxce `xkvmgr` delete primitive (reference-count lifetime).
- Controller cleanup policy: config, sweep, dependency stub.
- Persistence: removal fact, both apply paths, classification, replay verdict,
  snapshot alignment.
- Propagation: scheduler removal event, apiserver `OnJobDeleted`.

## Out of Boundary
- The job-dependency model and its submit option.
- Job history or accounting storage.
- How jobs are displayed (`job-status-detail-display`).

## Upstream / Downstream
- **Upstream**:
  - The WAL and replication machinery (`ha-wal-replication`,
    `raft-apply-decoupling`'s `apply_committed_entry` seam).
  - `ha-entity-snapshots` (where `job_should_retain` lives).
  - The config pipeline (`clusterconfig-schema-split`, `config-upgrade-snapshot`,
    `admin-config-tool`).
- **Downstream**:
  - `job-dependency` (2026-09-29) replaces the stub `job_has_dependents()` with a
    real check against the controller's in-memory reverse index. Keep the stub's
    signature stable so that swap is drop-in. A job is never removed while any
    **pending** job depends on it.
  - A future accounting or history feature, which archives before removal.
  - `job-status-detail-display` (a removed job simply stops appearing).

## Existing Spec Touchpoints
- **Extends**: `ha-entity-snapshots`. Its finished-job retention filter is retired
  or replaced; note this in that spec's history.
- **Adjacent**:
  - `worker-rejoin-resync` (unknown jobids on resync)
  - `job-status-detail-display`
  - `scheduler-core` (its job table)

## Constraints
- HA/persistence spec: use the full 3-phase workflow, not `/kiro-spec-quick`.
- Only the primary decides a removal, and every removal goes through the
  replicated WAL. No node removes a job on its own.
- The delaymgr callback is trigger-only. The sweep runs on the slow lane.
- Save/restore field-completeness (`persistence.md`):
  - the removal fact decodes identically on both apply paths;
  - snapshot and memory agree after removal;
  - run `/xce-audit-invariants controller` (dimension 7).
- Wire and schema changes are additive Cap'n Proto only. Old WAL and snapshots
  must still load. The requirements must also cover what happens when a node
  without this feature meets the new fact (`WAL_ENTRY_TYPE_MAX` skip-with-WARN).
- Removal must not free a job that another thread still references.
- Sweeps must be bounded, so a large backlog does not create one giant WAL entry
  or stall the slow lane.
- Known edge case: a very late resubmit retry of a removed job's jobid is
  accepted as a new job, because dedup checks `jobmgr` presence. Record it as
  accepted or mitigated.