myxce/.kiro/specs/ha-commit-barrier-rollout/research.md

# Research & Design Decisions: ha-commit-barrier-rollout

## Summary
- **Feature**: `ha-commit-barrier-rollout` (sub-spec 2 of 4, ha-consistency-invariants umbrella)
- **Discovery Scope**: Extension (light discovery — umbrella research.md is the primary discovery artifact; this pass verified its anchors against the current tree, 2026-06-11)
- **Key Findings**:
  - Every reply-before-commit call site named by the umbrella still exists; line anchors re-verified (table below)
  - Call-site verification of the umbrella classification table found two deviations the umbrella anticipated ("verified against call sites during implementation; table is the contract"): WAL_JOB_STOP/RESUME are fact-origin only, and WAL_JOB_UPDATE (absent from the umbrella table) is dual-origin
  - The configAck branch of `repl_wait_majority_ack` has the same three counting defects the commit engine fixed for walAck (`> 0` counting, `n_active == 0 → success`, `n_active` denominator)
  - The dispatch gate rejects `OPCode_notify` (and update/preason) under quorum loss — the bug-396 silent fact-loss hole is gate-level, not just replication-level

## Verified Code Anchors (2026-06-11, persistent_ha working tree)

| Concern | Anchor (verified by grep/read) | Status vs umbrella audit |
|---|---|---|
| do_jobcontrol_op replies after mutation, async WAL only | job.c:921-934 (reply at 932) | unchanged |
| handle_jobdispatch early reply | job.c:459-491 (reply at 466-471) | umbrella cited 459-471 — still accurate |
| handle_jobsubmit reference implementation | job.c:495-581 (hook rc checked 563-571, reply 573-578) | umbrella cited 563-578 — still accurate |
| do_queuecontrol_op | queue.c:119-132 (reply at 130); queue_ctrl fires AFTER hook under queue WRITE lock (queue.c:76-112, hook at 107) | new precision: lock-scope drives WAIT placement |
| do_hostcontrol_op | worker.c:689-702 (reply at 700) | new anchor |
| handle_worker_join | join.c:33-119 (AFTER hook 104-105, success reply 107-109) | new anchor |
| do_set_preason / do_clr_preason early replies | job.c:1040-1066 (reply at 1050) / job.c:1068+ (reply at 1078) | scheduler-origin facts — receipt-ACK verdict |
| do_update_op family | opupd.c:48/64/88 reply sites | worker-origin facts — receipt-ACK verdict |
| do_notify (no reply, fire-and-forget) | cluster.c:118-157 | unchanged |
| Hook → WAL type map | persist.c:1068-1088 (`detail_to_wal_type`) | classification dispatch insertion point |
| Job hook: submit sync path / fact async path | persist.c:1093-1169 (sync 1107-1129, async 1169) | unchanged |
| Queue / worker hooks (async) | persist.c:1172-1202 (append 1199) / persist.c:1204-1244 (append 1241) | unchanged |
| persist_wal_sync_append + timeout | persist.c:1003-1049; timeout `p->config.sync_timeout_ms` (DEFAULT_SYNC_TIMEOUT_MS=50, config.h:93) | committed_lsn store at 1035-1036 is removed by ha-commit-engine (upstream) |
| Dispatch gate PRIMARY no-quorum allowlist | ha.c:1527 (`ha_dispatch_gate`), allowlist 1537-1556; notify falls to `default → -XCE_ERR_NOT_READY` | fact admission insertion point |
| push_config_to_standbys ACK result ignored | config_repl.c:866-930 (wait at 916-917, rc logged-only 918-927, returns acked at 929) | umbrella cited 916-928 — still accurate |
| upgradeConfig caller ignores result | config_repl.c:1056-1094 (push at 1057, unconditional success ACK 1062-1073, restart 1091-1093) | umbrella cited 1057 — still accurate |
| configAck counting defects | repl.c:386-462 (`acked_config_lsn > 0` at 422-424, `n_active==0 → success` at 432-435, denominator `n_active` at 437) | mirrors the walAck defects fixed upstream |
| repl_on_config_ack stores reported tag/lsn | repl.c:351-371; `standby_conn_t.acked_config_lsn` (repl.h:25) | enables per-push `>= push_tag` comparison |
| mark_job_start/finish/fail/update unguarded | job.c:583/638/696/726 | unchanged |
| job_ctrl branches | job.c:306-438 — running job forwards to worker (357, no local entry); not-started job mutates via mark_job_update (391/401) / mark_job_fail (415) | classification evidence (deviation §2) |
| dispatch_a_job_ produces no WAL | job.c:180-304 — sends xce_startjob_req, mutates j->job.allocs, fires no hooks | handle_jobdispatch receipt-ACK evidence |
| WAL type enum | src/include/wal_types.h:44-58 (no WAL_NOOP yet) | WAL_NOOP arrives with ha-single-leader-role |
| Control-path xce_reply population | job.c:471,524,569,578,932,1050,1078; queue.c:130; worker.c:700; join.c:28,54,66,109,139,152,198,225,248,257,283; opupd.c:48,64,88; cluster.c:255-490; config_repl.c:619-1129; ha.c:1035; opget.c:222; reply.c:61; task.c:97 | sweep-task input; re-grep before editing |
| Opcode → handler dispatch | cluster.c:325-363 (jobsub/jobdispatch/notify/update/jobcontrol/queuecontrol/hostcontrol/setpreason/clrpreason/join) | gate allowlist + sweep scoping |

Upstream interface consumed (ha-commit-engine design, generated 2026-06-11): `commit_submit(ce, lsn, COMMIT_MODE_WAIT|RETAIN, timeout_ms)` → 0 / -ETIMEDOUT / -XCE_ERR_NOT_READY; RETAIN returns immediately; n_peers<=1 self-quorum; admission cap n_vcpus-relative; engine embedded in `cluster_t`, reachable from `persist_t` via `p->repl.ce`. `commit.c`/`commit.h` do not exist in the tree yet — ha-commit-engine must be implemented first (hard upstream dependency; tasks assume its interface as specified).

## Research Log

### §1 Reply-ordering reality and the WAIT placement question
- **Context**: The umbrella Reply-Ordering Contract fixes the handler sequence (validate → mutate → append → commit_submit(WAIT) → reply). Where should the WAIT physically run — inside the persistence hook (like job submit) or in the handler?
- **Findings**: queue/worker/job-state hooks fire while the entity WRITE lock is held (queue.c:107 inside lock taken at 76; mark_job_* called under job WRITE lock, e.g. job.c:629-631). Job submit is the exception: its AFTER hook runs against a job list after per-job locks are released (job.c:553), which is why its in-hook sync wait is safe.
- **Implications**: In-hook WAIT for queue/worker/job-control would block entity readers for up to the commit timeout. Handler-level barrier (after locks released, before reply) keeps the same completion semantics and matches the umbrella's handler-sequence wording. Decision below.

### §2 Classification table call-site verification (umbrella-mandated)
- **Context**: Umbrella table marks WAL_JOB_STOP/RESUME as "request (client job control) → WAIT" and omits WAL_JOB_UPDATE.
- **Findings**: WAL_JOB_STOP/RESUME entries are appended only by `on_job_stop`/`on_job_resume`, reachable solely from `do_notify` (worker-reported stoppedJob/resumedJob facts — cluster.c:131-136). Client job control (`do_jobcontrol_op` → `job_ctrl`) never produces them: for a running job it forwards the signal to the worker (job.c:357) and the resulting transition comes back later as a worker fact; for a not-yet-started job it mutates locally through `mark_job_update` (WAL_JOB_UPDATE) or `mark_job_fail` (WAL_JOB_FAIL). WAL_JOB_UPDATE/WAL_JOB_FAIL are therefore dual-origin (request via job_ctrl, fact via postdone/notify/rusage).
- **Implications**: Classify STOP/RESUME/UPDATE/FAIL as fact/RETAIN at the seam; realize the umbrella's request-shaped guarantee ("error to client") for client job control at the handler barrier in `do_jobcontrol_op`. This satisfies the umbrella's intent through its own prescribed handler sequence — the barrier is a property of the handler, not of the entry type. Not a contradiction: the umbrella explicitly delegated call-site verification, and the failure-column behavior is preserved.

### §3 handle_jobdispatch classification (umbrella left it open)
- **Findings**: `dispatch_a_job_` (job.c:180-304) sends `xce_startjob_req` to the worker and updates `j->job.allocs` in memory; it fires no hooks and appends no WAL entries. Per-job outcomes return to the scheduler via `notify_dispatch` (job.c:440-457, called at 479-486). The job's externally meaningful state change (RUN) arrives later as a worker fact (WAL_JOB_START).
- **Implications**: Per the umbrella's own criterion ("if its outcomes are fact-shaped … the early reply is an ACK-of-receipt and may stand"), the verdict is **receipt-ACK stands** (this spec's Req 2.3). The orphan-dispatch-of-uncommitted-submit race is closed by the egress watermark filter (ha-convergence-recovery), not here. The classification guard plus a no-WAL-from-dispatch test pin the verdict.

### §4 Fact loss is gate-level too (bug-396 family)
- **Findings**: `ha_dispatch_gate`'s CTRL_PRIMARY branch rejects `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` with `-XCE_ERR_NOT_READY` whenever write quorum is absent (ha.c:1537-1556 default arm). The worker never reads notify replies (umbrella research §A), so each rejection is a permanently lost fact.
- **Implications**: Umbrella 2.8 ("retain … including across quorum-loss periods") requires admitting these opcodes without write quorum; durability is then owned by RETAIN. Allowlist change is confined to the PRIMARY branch; the COMMITTING branch is left for ha-single-leader-role's wholesale deletion.

### §5 Config track hardening scope
- **Findings**: configAck counting (repl.c:418-438) shares all three walAck defects; `repl_on_config_ack` already stores the standby's acknowledged tag (repl.c:351-371), so per-push comparison (`>= push_tag`) needs only a parameter, not new state. `committed_config_tag` advance is already success-gated (config_repl.c:918-922).
- **Implications**: Hardening lives in the configAck branch + `push_config_to_standbys` return contract + caller failure path. No engine involvement (umbrella-locked: config is a parallel, simpler track sharing the majority arithmetic rule — one in-flight change at a time).

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Verdict |
|--------|-------------|-----------|---------------------|---------|
| In-hook WAIT for all request types | extend the job-submit sync-append pattern to queue/worker/job-control hooks | one mechanism, no handler edits | blocks commit waits under entity WRITE locks (readers stall up to timeout); job-control hooks fire per-transition, not per-operation | rejected |
| Handler-level barrier (chosen) | hooks append (RETAIN-registered); request handlers call `persist_commit_barrier()` after locks, before reply | matches umbrella handler sequence verbatim; no lock-hold waits; uniform mechanical sweep pattern; covers dual-origin types from request contexts | barrier watermark may over-wait on unrelated concurrent entries (bounded by timeout) | selected |
| Per-entry LSN plumbing (hooks return entry LSNs to handlers) | exact-entry waits | no over-wait | invasive signature changes through the hook system for negligible benefit (over-wait is bounded and rare) | rejected (YAGNI) |

## Design Decisions

### Decision: Handler-level commit barrier on the appended watermark
- **Context**: §1 lock-scope finding vs the umbrella WAIT mode for queue/worker/job-control.
- **Selected Approach**: `persist_commit_barrier(p, timeout_ms)` = snapshot `current_lsn`, `commit_submit(WAIT)` on it; called by each request handler between mutation success and reply.
- **Rationale**: Single-writer appends + contiguous commit advancement (engine postcondition) make the watermark cover everything the operation appended; the umbrella's Reply-Ordering Contract literally specifies the handler sequence.
- **Trade-offs**: Possible over-wait on concurrent unrelated uncommitted entries (bounded by `sync_timeout_ms`; those entries must commit anyway). A no-op control operation may wait behind unrelated traffic — accepted (request-shaped ops are gate-rejected during quorum loss, so the wait resolves quickly in healthy clusters).
- **Follow-up**: Integration test pins reply-after-commit ordering per opcode.

### Decision: Verified classification deviations (STOP/RESUME fact-only; UPDATE dual-origin)
- See Research Log §2. Umbrella conformance preserved through the handler barrier; the verified table in design.md is the implementation contract. Inline umbrella tags kept (2.1/2.8).

### Decision: Production fallback for unclassified WAL types = RETAIN
- **Context**: Req 2.2 requires fail-fast in testing; production must not crash the primary on a programmer omission.
- **Selected Approach**: XCE_TESTING assert; production ERROR log + RETAIN treatment.
- **Rationale**: Fail-safe direction is retention (never dropped, never falsely confirmed) — a misrouted request-shaped type degrades to a missing barrier caught by the sweep audit/tests, not to data loss.

### Decision: Failed config upgrade does not restart
- **Context**: Roadmap residual ("don't restart the primary before the commit barrier passes"); umbrella 5.2.
- **Selected Approach**: `do_upgrade_config_with_bytes` gates the success ACK and `restart_requested` on the barrier result; disk save/generation bump remain pre-barrier (existing sequence), reconciled by the existing OP_replConfig anti-entropy (out of boundary).
- **Trade-offs**: A failed upgrade leaves the new config on the primary's disk but uncommitted cluster-wide; explicit failure to the operator makes the state visible. Full save-after-commit resequencing deferred (would touch checkpoint/restart machinery beyond this slice).

## Risks & Mitigations
- Missed reply site silently violates umbrella 2.1 — exhaustive grep audit task over all `xce_reply` sites + behavioral integration test (no success reply before commit)
- Upstream ha-commit-engine not yet implemented — tasks consume only its published interface; if signatures drift, revalidation trigger fires (boundary section)
- New timeout errors surfacing to scheduler/worker/CLI callers — deliberate CP semantics (umbrella); error codes flow through existing reply envelopes; callers' retry behavior unchanged
- Guard rejections changing `mark_job_*` return codes — sweep checks all callers treat duplicate-as-success-no-op

## References
- `.kiro/specs/ha-consistency-invariants/{requirements,design,research}.md` — authoritative umbrella contract (esp. design §Write-Path Classification, §Config Barrier, §Idempotency Guards; research locked decisions 1, 2, 4)
- `.kiro/specs/ha-commit-engine/design.md` — consumed engine interface (commit.h verbatim), migration note "retired by ha-commit-barrier-rollout"
- `.kiro/steering/roadmap.md` — dependency ordering; superseded item "Quorum-ACK gate before config restart" residual
- `.kiro/steering/ha-async-io.md`, `testing.md`, `error-handling.md` — threading, TDD, errnox conventions
- Raft paper §5.3 (commit rule) — via umbrella; this spec adds no new consensus logic

---

## Gap Re-Audit (2026-06-18)

> **Mode**: POST-IMPLEMENTATION GAP AUDIT. All tasks 1.1–7.2 are `[x]` (phase=implemented). This pass re-verifies every EARS acceptance criterion against the current on-disk controller source (codegraph + grep, `persistent_ha` working tree) rather than against the design intent, and records residual/deferred items the implementation explicitly left open.
> **Verdict up front**: **CLOSED** — 19/19 acceptance criteria MET in code. The only open items are pre-declared, out-of-boundary residuals (bug-448 ref leak, bug-451 RUSAGE OOB, the CTRL_PRIMARY_COMMITTING fact-rejection deferred to ha-single-leader-role, and the WORKER_UPDATE-via-heartbeat WAIT-classification refinement). None block this spec's contract.

### Requirement-to-Asset Map (verified file:line)

| Req | Criterion (short) | Asset (verified) | Verdict |
|-----|-------------------|------------------|---------|
| 1.1 | success reply only after majority commit | `persist_commit_barrier` (persist.c:2069) called post-`job_ctrl` in `do_jobcontrol_op` (job.c:1023-1027), post-`queue_ctrl` in `do_queuecontrol_op` (queue.c:144), post-`worker_ctrl` in `do_hostcontrol_op` (worker.c:712-716), post-AFTER_EVENT in `handle_worker_join` (join.c:130) — reply (`xce_reply`) strictly after the barrier in every case | **MET** |
| 1.2 | commit timeout → error reply | barrier returns `-ETIMEDOUT` (persist.c:2105 → `commit_submit(...,COMMIT_MODE_WAIT,timeout)`), `reply.status = rc` in all four handlers (job.c:1029, queue.c, worker.c:718); `timeout_ms==0` borrows `p->config.sync_timeout_ms` (persist.c:2104-2107) | **MET** |
| 1.3 | not-ready rejection → error reply | engine `-XCE_ERR_NOT_READY` passes through barrier unmodified (persist.c:2105 returns engine code); handler maps to `reply.status` | **MET** |
| 1.4 | no success-interpretable reply pre-commit | handlers send exactly one `xce_reply` after the barrier; integration test `test_commit_barrier_rollout.c` pins "no success reply while entries exceed committed_lsn" (task 3.1/7.2) | **MET** |
| 2.1 | every WAL type classified WAIT/RETAIN | `wal_classify` (persist.c:2004-2054) total switch over all 13 `wal_entry_type` values incl. WAL_NOOP row (persist.c:2037, pre-declared); consulted in all 3 hook fns (`persist_job_hook_fun`/`persist_queue_hook_fun`/`persist_worker_hook_fun`, codegraph callers) | **MET** |
| 2.2 | unclassified type fails fast / fact-fallback | default arm: `ERROR(...)` log + `assert(0)` under `XCE_TESTING` + `return WAL_MODE_RETAIN` production fallback (persist.c:2045-2053) | **MET** |
| 2.3 | dispatch ACK is receipt-only | `handle_jobdispatch`/`dispatch_a_job_` produce no WAL (research §3); `do_set_preason`/`do_clr_preason` reply-first then RETAIN-append (job.c:1150/1178) — receipt verdict pinned by task 3.3 tests | **MET** |
| 2.4 | exhaustive reply-site verification | task 7.1 audit (tasks.md Impl Note 7.1): ZERO request-shaped stragglers; 8 category-A barrier-gated, 6 category-B receipt-ACK, ~29 category-C sites enumerated and committed | **MET** |
| 3.1 | facts appended, retried, no timeout | RETAIN types → `persist_wal_async` + `commit_submit(...,COMMIT_MODE_RETAIN,0)` (returns immediately, no timeout); hook callers get local-append rc only | **MET** |
| 3.2 | facts accepted during quorum loss | `ha_dispatch_gate` CTRL_PRIMARY no-quorum allowlist adds `OPCode_notify/update/setpreason/clrpreason` (ha.c:2099-2102) before the `-XCE_ERR_NOT_READY` default (ha.c:2105) | **MET** |
| 3.3 | uncommitted facts never discarded | log-is-the-queue: pending purely by `lsn > committed_lsn`; no drop path in the RETAIN seam | **MET** |
| 3.4 | facts commit after quorum restore, no redelivery | `committed_lsn` advances over retained suffix via `commit_on_ack`; task 4.3 integration test asserts no re-delivery, no timeout applied | **MET** |
| 4.1 | config confirmed only after majority | `do_upgrade_config_with_bytes` (config_repl.c:1115) calls `push_config_to_standbys` (config_repl.c:941) which returns barrier rc; success ACK (config_repl.c:1133-1142) only when `rc == 0` | **MET** |
| 4.2 | config timeout → failed to requester | `rc != 0` → `send_upgrade_error(ep,req,c,rc,...)` then `return rc` (config_repl.c:1118-1126), forwarding `-ETIMEDOUT` | **MET** |
| 4.3 | unacked config never committed | `committed_config_tag` advance is inside `push_config_to_standbys`'s success branch only (config_repl.c:963-969, atomic_store gated on majority) | **MET** |
| 4.4 | failed upgrade does not restart | failure path returns before `restart_requested` store + `cbuf_put(&c->wbuf)` (config_repl.c:1159-1161); restart unreachable on failure | **MET** |
| 5.1 | duplicate transition is logged no-op | `mark_job_finish` duplicate-on-terminal → `INFO(...)` no-op (job.c:702); `mark_job_update` no-op update → `INFO` skip (job.c:810) — no AFTER hook, no WAL | **MET** |
| 5.2 | terminal→non-terminal rejected with WARN | shared `job_state_is_backward` predicate (job.c:39); `mark_job_start` rejects backward START (job.c:632-633 WARN); `mark_job_update` rejects backward (job.c:803-804 WARN) | **MET** |
| 5.3 | replay-once equivalence | guards make RETAIN-retry / `restore_job_state` replay converge to single-application state; double-apply test (task 6.1) green | **MET** |

**Score: 19/19 MET.**

### Supporting-mechanism confirmations
- **Legacy walAck retirement (design goal, Req 1.x)**: `repl_wait_majority_ack` (repl.c:422) is now configAck-only; walAck delegates unconditionally to `commit_submit` (repl.c:657-659 comment + `repl_set_commit_engine` repl.c:660). configAck branch hardened: `acked_config_lsn >= push_tag` (repl.c:500-501), majority `1 + covered > n_peers/2` over `n_peers` (repl.c:511), `n_peers <= 1` immediate (repl.c:470-472), zero-cover multi-member → timeout (repl.c:529). **MET / no residual.**
- **Config-ACK production ingestion (remediation r1 / former CRITICAL F1)**: `repl_on_config_ack` now has live producers — `repl_ack_decode_and_dispatch` decodes `replconfigpushreplymsg` on the promoted-vote ep (ha.c:3403-3549) and a defensive `cluster.c:630-640` OPCode_reply ingestion. Closes the "bricked multi-member config upgrade" hole. **MET.**
- **Barrier-off-iomgr (remediation r1 / former CRITICAL F2)**: `push_config_transport_only` (config_repl.c:879, non-blocking) serves all best-effort ha.c/join.c streaming sites; `push_config_to_standbys` (transport+barrier) has exactly one caller in vtask context (`do_upgrade_config_with_bytes`). **MET.**

### Residual / Deferred Gaps (all pre-declared out-of-boundary — none block CLOSED)

| Item | Tag | Where | Status & Rationale |
|------|-----|-------|--------------------|
| bug-448: `job_ctrl` reference leak | **Constraint** | `reference(j)` at job.c:348 on the not-started branch; normal exit never `dereference`s | NOT FIXED. Pre-existing refcount leak surfaced by task 3.2 adversarial review; explicitly out of this spec's boundary (no behavior in the commit contract). Future fix. |
| bug-451: RUSAGE va_list OOB read | **Constraint** | `update_job_rusages` fires `HOOK_DETAIL_JOB_RUSAGE_UPDATE` with `xce_job_rusages_t*` (job.c:1046/1093-1094); `persist_job_hook_fun` default arm reads vararg 2 as `job_t*` → 4-byte OOB read at offset 56, garbage `state` in every WAL_JOB_RUSAGE payload | NOT FIXED. Reviewer-verified byte-precise; declared a boundary item at task 3.3. Production fix candidate = RUSAGE-specific va_list signature in the persist arm. Does not affect any acceptance criterion (jobid lines up; RUSAGE is a fact, retained either way). |
| CTRL_PRIMARY_COMMITTING still rejects facts | **Missing (deferred)** | `ha_dispatch_gate` COMMITTING branch intentionally untouched (design Non-Goals; ha.c) | DEFERRED to ha-single-leader-role, which deletes the transitional state wholesale. During the brief COMMITTING window facts can still be rejected — known residual, hard-ordered downstream. |
| WORKER_UPDATE WAIT-classification too broad | **Constraint** | `mark_worker_back` (worker.c:589) via heartbeat sweep `check_worker_timeout_` (cluster.c:1288) produces a WAIT-classified WAL_WORKER_CTRL/UPDATE with NO `xce_reply` | NOT a 7.1 violation (no reply ⇒ cannot confirm-before-commit; un-barriered WAIT behaves as RETAIN — commits via ACKs). Future-refinement candidate: narrow WORKER_UPDATE WAIT to the request-bearing detail. Carry to ha-single-leader-role / hardening pass. |
| Half-pushed config on disk after failed upgrade | **Constraint** | `do_upgrade_config_with_bytes` saves to disk + bumps generation before the barrier; failure leaves the new config on the primary's disk but cluster-uncommitted | Accepted (design Decision "Failed config upgrade does not restart"). Reconciled by existing OP_replConfig anti-entropy (out of boundary). Operator sees explicit error; tag stays uncommitted. |

### Integration Risks (forward-looking, not gaps in this spec)
- **ha-single-leader-role hard ordering**: must land AFTER this spec (it removes CTRL_PRIMARY_COMMITTING and adds WAL_NOOP). The classification guard (persist.c:2049) already forces WAL_NOOP to be classified — pre-declared row at persist.c:2037 (RETAIN) means the totality sweep will not trip when the type is wired. Low risk.
- **worker-rejoin-resync**: shares the `mark_job_*` idempotency guards (job.c:39/632/803). Revalidation trigger: re-verify the backward predicate against resync replay ordering when that spec lands.
- **New error codes to clients** (`-ETIMEDOUT`/`-XCE_ERR_NOT_READY` on control ops): deliberate CP behavior; flows through existing reply envelopes; caller retry semantics unchanged. Confirmed by design + reply.status wiring.

### Open-Work Options (only for the deferrable refinements — the spec itself needs none)

bug-451 (RUSAGE OOB) is the only residual with a correctness flavor inside this spec's files, so it carries options:
- **Option A — Fix now (RUSAGE-specific va_list arm in `persist_job_hook_fun`)**: read vararg 2 as `xce_job_rusages_t*` for `HOOK_DETAIL_JOB_RUSAGE_UPDATE`. *Effort: S (≈1 file, 1 arm, 1 ASAN test). Risk: Low — isolated to the WAL_JOB_RUSAGE payload encode; no contract change.*
- **Option B — Defer to a dedicated hardening pass / worker-rejoin-resync**: keep the boundary call as recorded. *Effort: 0 now. Risk: Low-Medium — OOB read persists in production (4-byte over-read, no crash observed; garbage state field in a fact payload that is retained regardless).*
- **Option C — Fold into ha-single-leader-role sweep** (which already touches WORKER_UPDATE classification). *Effort: S, batched. Risk: Low, but couples an unrelated correctness fix to a larger spec's timeline.*

**Recommendation**: Option A for bug-451 (cheap, isolated ASAN-clean fix); leave bug-448 and the COMMITTING/WORKER_UPDATE items as recorded boundary residuals. The spec's own 19/19 contract is CLOSED — no spec-internal open work.