EARS requirements for a controller status query (XCE lsid analog): primary identity
(unknown when unelected), answered-by node+role (distinct from cluster view),
configured peers+count, committed config version + content checksum (catch same-version
divergence), term/commit-apply/fenced/cluster-name, failover-aware delivery, test-client
command, and wait-primary switching its probe to it. C tier only.
The wait-primary probe loop EXPECTS repeated failure until a primary serves, so it
no longer spams the console: joblist_query gains a quiet flag (the poll passes
quiet=1) and the client log threshold is raised to warn for the loop duration,
restored to the user's level on every exit path. Interactive joblist + the
assertions keep their diagnostics (quiet=0).
Conflation fix: gate every kill + durability assertion on cluster health (3 nodes
up + serving + rejoin settle); assert_no_loss probes serveability first and reports
INCONCLUSIVE (not DATA LOSS) when unserveable; confirm-settle re-check distinguishes
catch-up lag from persistent loss; post-kill election SLA configurable (60s default).
DATA LOSS now reported ONLY for a job persistently absent from a SERVING cluster.
On the standby replicated-apply path, ha_apply_single_entry wrote the entry via
persist_wal_write_replicated (which correctly set the self-computed CRC chain
link p->last_crc), then called apply_wal_entry with a SYNTHETIC header whose
crc=0. apply_wal_entry's `p->last_crc = hdr->crc` clobbered the chain link with
0, so every subsequent replicated write stamped prev_crc=0 on disk. The on-disk
CRC chain broke at the second replicated entry — invisible while live (standbys
serve in-memory state) but fatal on COLD RESTART: replay hit the break, returned
RECOVER_TRUNCATED, recovered only the config prefix, and lost all committed jobs
(joblist=0). The sticky truncated verdict then drove a promotion-fence
availability wedge (bug-777) when every survivor was truncated.
Fix — single ownership of the WAL header + CRC chain:
- apply_wal_entry is now LOGICAL entity-restore only: signature
(p, type, lsn, payload, payload_len); no header. It advances the current_lsn
high-water (monotonic) but NEVER touches p->last_crc.
- The physical layer is the sole owner of the CRC chain: persist_wal_write_-
replicated on the live path, replay_wal/persist_recover on recovery.
- ha_apply_single_entry drops the fabricated header and passes entry fields
directly. The persist_recover replay caller is updated to the new signature.
Verified: new RED->GREEN test test_replicated_apply_chain_survives_cold_restart
(drives the real ha_apply_wal_catchup_entry path, cold-replays -> RECOVER_CLEAN,
all N recovered); controller suite 0 failures; 3-node rolling-SIGKILL soak on a
real ext2/3 fs now passes 5/5 (was wedge+loss at round 3); full cold restart
recovers all 6 committed jobs (was 0) with CLEAN verdicts and a normal election.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Task 9 tests were marked [x] but were stub/absent. Convert the
correctness-critical ones to real white-box tests, registered as meson tests
under --suite scheduler (leaks-off per convention):
- test_handlers.c (new) — #include ../src/handler.c; covers 9.2/9.3: each
notify/dataset handler enqueues the correct op / skips NULL-empty / -EINVAL.
Asserts the 5.6 fix (update_job -> UPDATE_JOB_FIELDS, not bulk UPDATE_JOB).
- test_job_update.c — add 9.5 (real sched_update_existing_queue, replacing the
former local stub); already covers 9.4 (job) and 9.7 (rusage).
- meson.build — build + register all three scheduler test executables.
Suite GREEN: sched_test 15/15, sched_job_update_test 10/10,
sched_handlers_test 6/6; ASAN-clean.
Honest remaining: 9.6 dequeue worker-resvals APPLY is PARTIAL (enqueue tested;
apply is a dormant path, R-3); 9.8 is a documented compile-time absence check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spec/design placed the scheduler-op name lookup as xce_sched_op_name in
lib/helper, but xce_sched_op_t is a scheduler-only type and src/lib must not
depend on scheduler headers — so it is correctly a static sched_op_name() in
scheduler.c. Code kept as-is (correct); docs amended to match:
- Task 1 intro + 1.1/1.3 + 5.7 + 9.1: scheduler-layer placement + layering note.
- design.md Component 8/9, file table, risks: same reconciliation.
Only xce_notify_event_name (int-typed, lib-level) lives in lib/helper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5.5 (Req 6 / G-5): the JOB_RUSAGE notify path called the plugin but never
stored the usage data — xce_sched_job_t had no rusage field (UB-6.1 unmet).
- Add xce_job_rusages_t* rusages to xce_sched_job_t.
- New sched_apply_job_rusage() stores the report ON the job (ownership
transferred), replacing any prior via free_JobRusages_ptr; -ENOENT unknown,
-EINVAL NULL. Dequeue NULLs the request pointer on success; kill_sched_job
releases job->rusages.
5.8 (G-3): sched_pend_update_killer left five new-op payloads on empty
break cases — a per-message leak since ownership is transferred on enqueue.
- Free each with the exact generated deep-freer: free_QueueConf_ptr,
free_Resource_ptr[], free_ResourceVal_ptr[], free_ResValList_ptr,
free_JobRusages_ptr. Safe: handlers nullify on enqueue, makeResource copies
via makeInstance, dequeue copies values (no aliasing).
Tests: sched_job_update_test 7/7 (3 update + 4 rusage), regression
sched_test 15/15, ASAN clean (no bad-free/double-free).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The notify schedUpdateJob path shared XCE_SCHED_OP_UPDATE_JOB with the
dataset bulk-load, so the dequeue routed both to sched_update_jobs()
(create/replace) — a notify update to an existing job corrupted/duplicated
live job state and never returned -ENOENT (Req 3 UB-3.1/UB-3.2 unmet;
sched_update_existing_job was dead code copying only state).
- Add distinct op XCE_SCHED_OP_UPDATE_JOB_FIELDS (+ jobfieldsupdate union),
enqueued only by the notify handler, dequeued to sched_update_existing_job.
- Move sched_update_existing_job to schedobj.c; update all mutable fields
(state, priority, queue, resreq) in place; -ENOENT unknown, -EINVAL NULL.
- Dataset bulk-create path unchanged. Killer frees the new payload.
- New white-box test sched_job_update_test (3/3); regression sched_test 15/15.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per design invariant (a controller exits only on crash or admin kill), replace the
self-abort recovery terminal (tasks 3.3/4.3/4.4) with a loud, permanent, non-promotable
fence that stays alive and keeps retrying catch-up (self-heals on peer return; no manual
admin recovery). Records the never-self-terminate invariant as project-wide guidance in
src/controller/CLAUDE.md + .kiro/steering/ha-async-io.md.
