myxce/.kiro/specs/controller-job-cleanup/brief.md

Brief: controller-job-cleanup

Problem

Operators of long-running XCE clusters have no way to get rid of finished jobs.

• The controller keeps every job it has ever accepted in memory, forever. So do the scheduler (core->jobs) and the apiserver store.
• Memory use, full dataset pushes (scheduler join, apiserver connect), and xcecli jobs list output all grow without bound.
• A hidden, hard-coded one-hour rule drops finished jobs only from snapshots, not from memory. Nodes therefore disagree about which jobs exist:
  • a restarted node holds a different job set than one that never restarted;
  • a standby that installs a snapshot keeps jobs the snapshot omits.

Current State

Codebase exploration, 2026-09-29.

Job table

• Jobs live in c->jobmgr, an xkvmgr table with integer keys (controller/cluster.c:1668-1685). It has no delete operation: .del_key = NULL (src/lib/kvmgr.c:289, bug-1443).
• The underlying pieces exist: kvtab_del (kv.c:288) and vclear (the rollback path at kvmgr.c:107-128).
• Jobs are reference-counted RTTI objects, and job_obj_killer (cluster.c:1591) does nothing.

WAL

• There is no "job removed" entry type (include/wal_types.h:132-310). Numbers 13-19 are free.
• A new type has to be handled in:
  • both apply paths, apply_wal_entry (persist.c:4195) and apply_committed_entry (persist.c:4607), with identical decoding (persistence.md:68);
  • wal_classify() (persist.c:7395, where an unclassified type asserts under XCE_TESTING);
  • the replay verdict. persist_apply_verdict (persist.c:6660-6690) treats a shrinking job table as DIVERGENT, and apply_expectation_t (persist.h:162) has to count removals.
• Fact append pattern to copy: persist_submit_job_fact_ex (persist.c:8105) → wal_hook_append_classified(..., WAL_MODE_RETAIN, ...).

The existing snapshot-only retention rule

• job_should_retain() (persist.c:8695-8714) keeps a terminal (EXIT|DONE|PERR) job until finish_time + finished_retain_s has passed.
• finished_retain_s defaults to DEFAULT_FINISHED_RETAIN_S=3600 (include/config.h:91) and cannot be configured.
• Its only caller is collect_job_frames (persist.c:2513), which runs at snapshot time, per node, using that node's time(NULL).
• The same rule also expires TRANSIENT jobs after transient_job_timeout_s.
• load_snapshot_entities (persist.c:3827) only adds jobs and never clears the table.

Terminal states

• JOB_IS_FINISH = EXIT || DONE (include/xce/types.h:425-429).
• Two definitions of "terminal" disagree: job.c:45 says DONE|EXIT, and persist.c:8690 says EXIT|DONE|PERR.
• finish_time is set live (job.c:2756), carried in the WAL_JOB_FINISH payload, and restored on apply and resync.

Periodic, primary-only work

• Pattern: ha_start_primary_heartbeat (ha.c:10142) uses delay_task_ms(..., -1, fn, c), and the callback does nothing unless role == CTRL_PRIMARY.
• Offload pattern: checkpoint_task_fn (persist.c:3545-3592) runs on the delaymgr thread and calls slowlane_submit(...) (slowlane.c:185).
• delaymgr is trigger-only (ha-async-io.md).

Cluster config

• Schema ClusterConfig (schema/controller.capnp:850-869); the next free ordinal is @15.
• C struct: cluster_config_t (include/clusterconfig.h).
• YAML and validation: configs/xce.yaml and configs/schema/xce.schema.json (additionalProperties:false).
• xceadmin: _XCE_SCALAR_FIELDS (xceadmin/src/xcecommon/mapping.py:176).
• Defaults: myconf.c:65-76.
• Live config upgrade swaps myconf->cluster_config (config_repl.c:747), so read the value on every sweep.
• Model to follow: worker_resync_timeout_ms.

Downstream

• Scheduler: no job-removed hook detail exists (include/xce/cluster_def.h:43-67), and core->jobs never deletes.
• Apiserver: OnJobDeleted (backend/notification.go:11, handler.go:83) exists but nothing calls it. Reconnect never prunes the store.
• libxce: querying a missing jobid returns 0 jobs with status 0 (opget.c:159-176, xceapi.c:787). Control operations return -ENOENT.
• Worker resync ignores unknown jobids (resync.c:638-647), which is safe.
• Array jobs: there is no parent object. Each element is an independent job with its own jobid. Elements are numbered 1..N per job-array-one-based-index (2026-10-01): element k of array <array-id> is <array-id>[k], k = 1..N (arrays submitted before that change may still have elements 0..N-1). The earlier note here, jobid in [base, base+array_size), described numbering from 0.
• There is no accounting or history file, so a removed job leaves no record.

Job dependencies

• None exist (no field in job_submit_data_t). The user asked for a stub.

Desired Outcome

• Finished jobs (DONE or EXIT) are removed permanently once they have been finished longer than a configured retention period and no other job depends on them.
• Every controller node removes the same jobs at the same point in the log. Removal survives replay, restart and failover, and snapshots match memory.
• The scheduler and apiserver drop removed jobs too, so a removed job disappears from xcecli and from REST.
• Both periods can be configured in the cluster configuration, and cleanup can be turned off.

Approach

Chosen: approach A, a replicated removal record (user decision, 2026-09-29).

1. Trigger (primary only). A delaymgr timer fires every job_cleanup_period. The callback returns unless this node is PRIMARY, and otherwise hands the sweep to the slow lane. It never blocks.

2. Sweep (slow lane). Walk jobmgr under a read lock and pick jobs that meet all of these:

   • state is DONE or EXIT, not TRANSIENT;
   • finish_time + job_retention < now;
   • the dependency stub (job_has_dependents(job), which always returns false for now) lets it go.

   Append the chosen jobids as new WAL job-removal facts (RETAIN), batched and bounded per sweep.

3. Apply (every node, both apply paths). Remove the job from jobmgr through a new xkvmgr delete operation that drops the table's reference; in-flight holders keep the job alive until they release it. Update the replay-verdict expectations so that removals are counted.

4. Propagate. Tell the scheduler to drop the job from core->jobs (a new hook detail or event), and tell the apiserver (wire up the existing OnJobDeleted).

5. Retire the snapshot-only filter for finished jobs, so a snapshot contains exactly the jobs in memory. The requirements decide what happens to the TRANSIENT-timeout part of job_should_retain (keep it separate or fold it in).

Rejected alternatives:

• B, each node cleans up locally without replication. Clock skew makes nodes disagree, and the replay verdict flags a shrinking table as DIVERGENT.
• C, extend the snapshot filter. Still decided per node, and makes the existing drift permanent.

Scope

• In:
  • Two cluster configuration parameters, both configurable (names are provisional):
    • job_retention: how long a finished job is kept. Default 3600 s, matching today's snapshot rule. 0 turns cleanup off, and jobs are kept forever.
    • job_cleanup_period: the sweep interval. Default 300 s.
    • Both need schema, YAML, JSON-schema validation, xceadmin mapping, defaults and live-upgrade pickup.
  • The primary-only periodic sweep.
  • The dependency-check stub.
  • The new WAL removal fact, handled on both apply paths and in classification, with replay-verdict accounting.
  • The xkvmgr delete operation, safe with reference counts.
  • Removal on every node.
  • Scheduler and apiserver propagation.
  • Retiring the snapshot-only filter for finished jobs, so snapshot and memory agree.
  • Tests: unit, persistence round-trip, and live HA.
• Out:
  • A real job-dependency feature. Only the stub is in scope.
  • Accounting or job-history archiving before removal.
  • Manual per-job purge commands.
  • Cleaning up PEND, RUN or other non-terminal jobs.
  • The lost-and-found drain (roadmap.md).

Boundary Candidates

• libxce xkvmgr delete primitive (reference-count lifetime).
• Controller cleanup policy: config, sweep, dependency stub.
• Persistence: removal fact, both apply paths, classification, replay verdict, snapshot alignment.
• Propagation: scheduler removal event, apiserver OnJobDeleted.

Out of Boundary

• The job-dependency model and its submit option.
• Job history or accounting storage.
• How jobs are displayed (job-status-detail-display).

Upstream / Downstream

• Upstream:
  • The WAL and replication machinery (ha-wal-replication, raft-apply-decoupling's apply_committed_entry seam).
  • ha-entity-snapshots (where job_should_retain lives).
  • The config pipeline (clusterconfig-schema-split, config-upgrade-snapshot, admin-config-tool).
• Downstream:
  • job-dependency (2026-09-29) replaces the stub job_has_dependents() with a real check against the controller's in-memory reverse index. Keep the stub's signature stable so that swap is drop-in. A job is never removed while any pending job depends on it.
  • A future accounting or history feature, which archives before removal.
  • job-status-detail-display (a removed job simply stops appearing).

Existing Spec Touchpoints

• Extends: ha-entity-snapshots. Its finished-job retention filter is retired or replaced; note this in that spec's history.
• Adjacent:
  • worker-rejoin-resync (unknown jobids on resync)
  • job-status-detail-display
  • scheduler-core (its job table)

Constraints

• HA/persistence spec: use the full 3-phase workflow, not /kiro-spec-quick.
• Only the primary decides a removal, and every removal goes through the replicated WAL. No node removes a job on its own.
• The delaymgr callback is trigger-only. The sweep runs on the slow lane.
• Save/restore field-completeness (persistence.md):
  • the removal fact decodes identically on both apply paths;
  • snapshot and memory agree after removal;
  • run /xce-audit-invariants controller (dimension 7).
• Wire and schema changes are additive Cap'n Proto only. Old WAL and snapshots must still load. The requirements must also cover what happens when a node without this feature meets the new fact (WAL_ENTRY_TYPE_MAX skip-with-WARN).
• Removal must not free a job that another thread still references.
• Sweeps must be bounded, so a large backlog does not create one giant WAL entry or stall the slow lane.
• Known edge case: a very late resubmit retry of a removed job's jobid is accepted as a new job, because dedup checks jobmgr presence. Record it as accepted or mitigated.