update documents

.kiro/specs/ha-commit-barrier-rollout/design.md

@@ -8,10 +8,10 @@
 
 **Users**: Cluster operators gain the guarantee that any confirmed control operation survives failover and that worker-reported reality is never silently dropped; downstream sub-specs (ha-single-leader-role, ha-convergence-recovery) gain a write path that is barrier-complete before the transitional leader state is removed.
 
-**Impact**: Adds a classification dispatch at the persistence hook seam (persist.c) with a fail-fast guard for unclassified WAL types; restructures `do_jobcontrol_op` (job.c:921-934), `do_queuecontrol_op` (queue.c:119-132), `do_hostcontrol_op` (worker.c:689-702), and `handle_worker_join` (join.c:33-119) to commit-first-then-reply; admits fact-bearing opcodes through the dispatch gate during quorum loss (ha.c:1527); propagates the config majority-ACK result from `push_config_to_standbys` (config_repl.c:866-930) to the `upgradeConfig` requester (config_repl.c:1057-1094); adds state-machine guards to `mark_job_*` (job.c:583/638/696/726); and retires the legacy walAck counting fallback in `repl_wait_majority_ack` (repl.c:386-462) while hardening its configAck branch.
+**Impact**: Adds a classification dispatch at the persistence hook seam (persist.c) with a fail-fast guard for unclassified WAL types; restructures `do_jobcontrol_op` (job.c:2198-2224), `do_queuecontrol_op` (queue.c:128-154), `do_hostcontrol_op` (worker.c:796-822), and `handle_worker_join` (join.c:48-149) to commit-first-then-reply [line numbers re-verified against current source — task-evidence line numbers had drifted]; admits fact-bearing opcodes through the dispatch gate during quorum loss (`ha_dispatch_gate`, ha.c:4525, allowlist at ha.c:4644-4648); propagates the config majority-ACK result from `push_config_to_standbys` (config_repl.c:1609-1654) to the `upgradeConfig` requester (`do_upgrade_config_with_bytes`, config_repl.c:1696-1855 — Note: as of ha-term-safety, the requester's commit gate is `config_commit_change`'s WAL_CONFIG_MARKER barrier, which internally calls `push_config_to_standbys`; see Config Barrier section below); adds state-machine guards to `mark_job_*` (job.c: `mark_job_fail` 1830, `mark_job_update` 1929 — Note: `mark_job_start`/`mark_job_finish` were renamed and restructured by raft-apply-decoupling, see Idempotency Guards section below); and retires the legacy walAck counting fallback in `repl_wait_majority_ack` (repl.c:589-700) while hardening its configAck branch.
 
 ### Goals
-- Commit-first-then-reply for every request-shaped control handler; `handle_jobsubmit` (job.c:495-581) is the reference implementation and stays unchanged
+- Commit-first-then-reply for every request-shaped control handler; `handle_jobsubmit` (job.c:1412 onward) is the reference implementation (Note: superseded by raft-apply-decoupling — `handle_jobsubmit` no longer "stays unchanged"; it was rearchitected to append via an async commit continuation (`jobsubmit_commit_continuation`/`jobsubmit_apply_complete`) rather than a synchronous append-then-reply, and job-array-single-wal-record added a sibling array-submit path. The commit-first-then-reply *outcome* for the requester is preserved, but the reference shape this design points to has changed; see those specs' design.md)
 - Fact WAL types routed RETAIN (log-is-the-queue): appended + broadcast, pending purely by `lsn > committed_lsn`, never dropped while running+leading, accepted even during quorum loss
 - An unclassified state-changing WAL type cannot silently bypass the barrier (debug assert; production fact-fallback)
 - Config upgrade fails to the requester on missed majority and does not restart into the failed config
@@ -22,7 +22,7 @@
 - Promotion no-op / own-term gate consumers / CTRL_PRIMARY_COMMITTING removal (ha-single-leader-role; the gate's COMMITTING branch keeps rejecting facts until that spec deletes the state — known residual)
 - Demotion truncate+restart, rejoin reconciliation, egress watermark filter, ACK-driven catch-up, quorum-transition logging (ha-convergence-recovery)
 - Worker protocol changes; recovery of facts that never reached the controller or died with it (worker-rejoin-resync backlog — umbrella-locked: crash-loss of uncommitted facts accepted until then)
-- WAL_NOOP introduction (ha-single-leader-role adds the type; the classification guard here forces it to be classified when it lands)
+- WAL_NOOP introduction (ha-single-leader-role adds the type; the classification guard here forces it to be classified when it lands — confirmed landed: `wal_classify` (persist.c:5796) now classifies WAL_NOOP as RETAIN exactly as pre-declared below)
 
 ## Boundary Commitments
 
@@ -61,12 +61,12 @@
 ## Architecture
 
 ### Existing Architecture Analysis
-- All state-changing writes flow through one seam: entity mutation fires `HOOK_AFTER_EVENT`, and `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:1093/1172/1204) map hook details to WAL types via `detail_to_wal_type` (persist.c:1068-1088) and append. Only WAL_JOB_SUBMIT uses the sync barrier (persist.c:1107-1129); everything else is `persist_wal_async` fire-and-forget
-- Request handlers reply immediately after mutation: `do_jobcontrol_op` (job.c:932), `do_queuecontrol_op` (queue.c:130), `do_hostcontrol_op` (worker.c:700), `handle_worker_join` (join.c:109) — all violate umbrella 2.1 today. `handle_jobsubmit` (job.c:563-578) is the proven correct shape
-- Queue/worker/job-state hooks fire while the entity WRITE lock is held (queue.c:76-112, job.c:629-631/687-689); job-submit's hook runs after the job locks are released (job.c:553) — this asymmetry drives WAIT placement (below)
-- Facts arrive via `do_notify` (cluster.c:118-157, no reply) and `do_update_op`/`do_set_preason`/`do_clr_preason` (receipt-style replies); the dispatch gate (ha.c:1527) rejects `OPCode_notify` with `-XCE_ERR_NOT_READY` under quorum loss → permanent silent fact loss (bug-396 family, verified pre-existing)
-- Call-site verification of the umbrella classification table (mandated by the umbrella): WAL_JOB_STOP/RESUME are appended only from `on_job_stop`/`on_job_resume`, which are invoked exclusively from `do_notify` (worker-reported stop/resume facts). Client job control (`do_jobcontrol_op` → `job_ctrl`, job.c:306-438) either forwards to the worker (running job — no local entry) or mutates locally via `mark_job_update`/`mark_job_fail` (not-started job → WAL_JOB_UPDATE/WAL_JOB_FAIL). WAL_JOB_UPDATE is absent from the umbrella table and is dual-origin (request: job_ctrl; fact: on_job_postdone, rusage update). Resolution: STOP/RESUME/UPDATE/FAIL classify as fact/RETAIN at the seam, and the umbrella's intent for client job control — "error to client" — is realized by the handler-level commit barrier in `do_jobcontrol_op` (the Reply-Ordering Contract's handler sequence is the umbrella-authoritative mechanism). Recorded in research.md
-- `push_config_to_standbys` (config_repl.c:866-930) calls `repl_wait_majority_ack(configAck)` at 916-917 but only logs the result and returns the push count; the caller (config_repl.c:1057) ACKs the admin tool with status=0 and restarts regardless. The configAck counting (repl.c:418-438) has the same defects the engine fixed for walAck: `acked_config_lsn > 0` (any previous ack counts), `n_active == 0 → success`, denominator `n_active` instead of `n_peers`
+- All state-changing writes flow through one seam: entity mutation fires `HOOK_AFTER_EVENT`, and `persist_job_hook_fun` / `persist_queue_hook_fun` / `persist_worker_hook_fun` (persist.c:6509/6589/6638 — re-verified, was 1093/1172/1204) map hook details to WAL types via `detail_to_wal_type` (persist.c:5769-5789, was 1068-1088) and append. WAL_JOB_SUBMIT is now committed via an explicit call from `handle_jobsubmit` (`persist_submit_job_payloads_and_commit`) strictly before any job object is constructed, never from this hook — the hook's WAL_MODE_WAIT branch is now a pure post-commit no-op (Note: superseded by raft-apply-decoupling task 3.1; the "in-hook sync barrier" description below and elsewhere in this doc no longer matches — see that spec's design.md). Everything else still routes through `wal_hook_append_classified` (append + RETAIN registration), functionally equivalent to the `persist_wal_async` fire-and-forget this doc describes
+- Request handlers reply immediately after mutation: `do_jobcontrol_op` (job.c:2198), `do_queuecontrol_op` (queue.c:128), `do_hostcontrol_op` (worker.c:796), `handle_worker_join` (join.c:48) — all violate umbrella 2.1 today [historical framing — this spec fixes it; see Reply-Ordering Sweep]. `handle_jobsubmit` (job.c:1412) is the proven correct shape at the time this spec closed (Note: superseded by raft-apply-decoupling — see Goals section note above)
+- Queue/worker/job-state hooks fire while the entity WRITE lock is held; job-submit's hook now fires post-commit outside any entity lock as a pure notification (raft-apply-decoupling task 3.1) — this asymmetry still drives WAIT placement (below), though the job-submit half of the asymmetry is now "no hook work at all" rather than "hook runs lock-free"
+- Facts arrive via `do_notify` (cluster.c, no reply) and `do_update_op`/`do_set_preason`/`do_clr_preason` (receipt-style replies); the dispatch gate (`ha_dispatch_gate`, ha.c:4525) rejects `OPCode_notify` with `-XCE_ERR_NOT_READY` under quorum loss → permanent silent fact loss (bug-396 family, verified pre-existing; fixed by this spec, see Fact Gate Admission)
+- Call-site verification of the umbrella classification table (mandated by the umbrella): WAL_JOB_STOP/RESUME are appended only from `on_job_stop`/`on_job_resume`, which are invoked exclusively from `do_notify` (worker-reported stop/resume facts). Client job control (`do_jobcontrol_op` → `job_ctrl`, job.c:957) either forwards to the worker (running job — no local entry) or mutates locally via `mark_job_update`/`mark_job_fail` (not-started job → WAL_JOB_UPDATE/WAL_JOB_FAIL). WAL_JOB_UPDATE is absent from the umbrella table and is dual-origin (request: job_ctrl; fact: on_job_postdone, rusage update). Resolution: STOP/RESUME/UPDATE/FAIL classify as fact/RETAIN at the seam, and the umbrella's intent for client job control — "error to client" — is realized by the handler-level commit barrier in `do_jobcontrol_op` (the Reply-Ordering Contract's handler sequence is the umbrella-authoritative mechanism). Recorded in research.md
+- `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) calls `repl_wait_majority_ack(configAck)` but only logs the result and returns the push count; the caller ACKed the admin tool with status=0 and restarted regardless [historical framing — this spec fixes it]. The configAck counting (repl.c, was 418-438) has the same defects the engine fixed for walAck: `acked_config_lsn > 0` (any previous ack counts), `n_active == 0 → success`, denominator `n_active` instead of `n_peers` [historical — fixed, see Config Barrier]
 
 ### Architecture Pattern & Boundary Map
 
@@ -143,16 +143,17 @@ src/include/repl.h                     # MODIFIED — configAck evaluation reuse
 ```
 
 ### Modified Files (what and why)
+[Line citations below re-verified against current source (was accurate as of this spec's 2026-06-13 close; substantial drift since from file growth — see per-line notes)]
 - `src/include/persist.h` — declares `wal_commit_mode_t { WAL_MODE_WAIT, WAL_MODE_RETAIN, WAL_MODE_LOCAL }`, `wal_commit_mode_t wal_classify(uint32_t wal_type, int detail)`, `int persist_commit_barrier(persist_t *p, uint32_t timeout_ms)`
-- `src/controller/persist.c` — `wal_classify` table; all three hook functions consult it: WAIT → existing sync path (submit) or async append with handler-owed barrier; RETAIN → `persist_wal_async` + `commit_submit(RETAIN)` registration; LOCAL → unchanged (WAL_CHECKPOINT); unclassified → `assert(0)` under XCE_TESTING, ERROR + RETAIN fallback in production. `persist_commit_barrier` implemented here (snapshot `current_lsn`, `commit_submit(WAIT)`)
-- `src/controller/job.c` — `do_jobcontrol_op` (job.c:921-934): barrier between `job_ctrl` success and the reply. `mark_job_start/finish/fail/update` (job.c:583/638/696/726): transition guards before mutation
-- `src/controller/queue.c` — `do_queuecontrol_op` (queue.c:119-132): barrier between `queue_ctrl` success and the reply
-- `src/controller/worker.c` — `do_hostcontrol_op` (worker.c:689-702): barrier between `worker_ctrl` success and the reply
-- `src/controller/join.c` — `handle_worker_join` (join.c:33-119): barrier between the AFTER_EVENT hook and the reply at join.c:107-109; join failure (idx < 0) keeps replying immediately
-- `src/controller/ha.c` — `ha_dispatch_gate` PRIMARY no-quorum allowlist (ha.c:1537-1556): add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason`
-- `src/controller/config_repl.c` — `push_config_to_standbys` returns the barrier result (0 / `-ETIMEDOUT`), push count via out-param; passes the in-flight tag for per-push evaluation; caller `do_upgrade_config_with_bytes` (config_repl.c:1057): on failure send error reply, skip restart
-- `src/controller/repl.c` — walAck branch of `repl_wait_majority_ack`: legacy counting removed (engine is mandatory on this path; `ce == NULL` becomes a programming error); configAck branch: per-push comparison (`acked_config_lsn >= push_tag`), majority over `n_peers` counting self, `n_peers > 1` with zero standby acks never succeeds
-- `src/controller/meson.build` — register `testsuite/test_commit_barrier_rollout.c`
+- `src/controller/persist.c` — `wal_classify` (persist.c:5796) table; all three hook functions consult it: WAIT → existing sync path (submit) or async append with handler-owed barrier; RETAIN → `persist_wal_async` + `commit_submit(RETAIN)` registration; LOCAL → unchanged (WAL_CHECKPOINT); unclassified → `assert(0)` under XCE_TESTING, ERROR + RETAIN fallback in production. `persist_commit_barrier` implemented here (persist.c:5875 — snapshot `current_lsn`, `commit_submit(WAIT)`). Note: the table has since grown two rows this design never described — `WAL_JOB_ARRAY_SUBMIT` (WAIT, added by job-array-single-wal-record) and `WAL_CONFIG_MARKER` (RETAIN, added by ha-term-safety task 4.2) — see both specs' design.md
+- `src/controller/job.c` — `do_jobcontrol_op` (job.c:2198-2224, was 921-934): barrier between `job_ctrl` success and the reply. `mark_job_fail`/`mark_job_update` (job.c:1830/1929, was 696/726): transition guards before mutation. `mark_job_start`/`mark_job_finish` (was job.c:583/638) no longer exist under those names — renamed `record_job_start_fact`/`record_job_finish_fact` (job.c:1625/1734) with the guard+mutation split across receipt time and apply time (Note: superseded by raft-apply-decoupling, see Idempotency Guards below and that spec's design.md)
+- `src/controller/queue.c` — `do_queuecontrol_op` (queue.c:128-154, was 119-132): barrier between `queue_ctrl` success and the reply
+- `src/controller/worker.c` — `do_hostcontrol_op` (worker.c:796-822, was 689-702): barrier between `worker_ctrl` success and the reply
+- `src/controller/join.c` — `handle_worker_join` (join.c:48-149, was 33-119): barrier between the AFTER_EVENT hook and the reply at join.c:128-135 (was 107-109); join failure (idx < 0) keeps replying immediately
+- `src/controller/ha.c` — `ha_dispatch_gate` (ha.c:4525, was implicitly ~1527) PRIMARY no-quorum allowlist (ha.c:4644-4648, was 1537-1556): adds `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` — confirmed present in current source. The gate has since grown two additional fence layers checked before this allowlist (`config_serve_fenced`, `upgrade_materialization_fenced` — ha-term-safety / config-upgrade-snapshot work); those fences do NOT include the fact opcodes in their own allowlists, so a config-fenced primary rejects facts too — a narrower, config-specific case this design did not anticipate and does not contradict (quorum-loss admission, this spec's concern, is unaffected)
+- `src/controller/config_repl.c` — `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) returns the barrier result (0 / `-ETIMEDOUT`), push count via out-param; passes the in-flight tag for per-push evaluation; caller `do_upgrade_config_with_bytes` (config_repl.c:1696-1855, was 1057): on failure send error reply, skip restart. Note: superseded by ha-term-safety task 4.2 — the direct caller of `push_config_to_standbys` is now `config_commit_change` (config_repl.c:535), which `do_upgrade_config_with_bytes` calls instead; the requester's success/failure now gates on a WAL_CONFIG_MARKER consensus record reaching the committed prefix, with `push_config_to_standbys`'s majority-ACK wait as the out-of-band byte-delivery quorum underneath it. See Config Barrier section and ha-term-safety's design.md
+- `src/controller/repl.c` — walAck branch of `repl_wait_majority_ack` (repl.c:589-700, was 386-462): legacy counting removed (engine is mandatory on this path; `ce == NULL` becomes a programming error); configAck branch: per-push comparison (`acked_config_lsn >= push_tag`), majority over `n_peers` counting self, `n_peers > 1` with zero standby acks never succeeds — confirmed unchanged in current source
+- `src/controller/meson.build` — register `testsuite/test_commit_barrier_rollout.c` (confirmed present)
 
 ## System Flows
 
@@ -190,6 +191,8 @@ Gate admits the opcode even without write quorum (3.2) → handler applies the t
 
 ### Configuration upgrade
 
+> **Note: the diagram and "request restart" step below are superseded by ha-term-safety (task 4.2), see that spec's design.md.** Current source inserts `config_commit_change` (config_repl.c:535, appending a WAL_CONFIG_MARKER) between `do_upgrade_config_with_bytes` and `push_config_to_standbys`, and no restart is requested on success — materialization now happens via `apply_committed_entry`'s WAL_CONFIG_MARKER case on every member's own apply of the committed marker. The majority-ACK shape below (push → parallel threads → per-standby ack → majority-of-n_peers) is otherwise still accurate as the mechanism `push_config_to_standbys` uses underneath the marker.
+
 ```mermaid
 sequenceDiagram
     participant A as admin tool
@@ -214,7 +217,7 @@ sequenceDiagram
     end
 ```
 
-Flow decisions: `committed_config_tag` advances only inside the majority branch (4.3 — already correct, preserved); the failure path uses the existing `send_upgrade_error` shape so the admin tool sees a definite failure (4.2); restart is gated on success (4.4 — closes the roadmap residual "don't restart before the commit barrier passes"); config remains a parallel, simpler track sharing only the majority arithmetic rule with the engine (umbrella-locked: one in-flight config change at a time).
+Flow decisions: `committed_config_tag` advances only inside the majority branch (4.3 — already correct, preserved, now framed by ha-term-safety as a non-authoritative cache of the marker-derived committed config); the failure path uses the existing `send_upgrade_error` shape so the admin tool sees a definite failure (4.2); restart was originally gated on success (4.4 — closed the roadmap residual "don't restart before the commit barrier passes") but ha-term-safety later removed the restart step entirely in favor of marker-driven materialization on every member — see the supersession note above; config remains a parallel, simpler track sharing only the majority arithmetic rule with the engine (umbrella-locked: one in-flight config change at a time).
 
 ## Requirements Traceability
 
@@ -267,7 +270,9 @@ Flow decisions: `committed_config_tag` advances only inside the majority branch
 | WAL_WORKER_CTRL, detail JOIN/UPDATE | handle_worker_join, do_hostcontrol_op | request | WAIT (handler barrier) | error reply to worker/client |
 | WAL_WORKER_CTRL, detail DEAD | controller-detected (worker.c:565/578) | fact | RETAIN | none — retained until committed |
 | WAL_JOB_START / FINISH / FAIL / STOP / RESUME / UPDATE / RUSAGE / CKPT / PREASON | worker notify, scheduler preason, postdone, rusage; FAIL/UPDATE also from job_ctrl (request origin → covered by do_jobcontrol_op's handler barrier) | fact | RETAIN | none — retained until committed |
-| WAL_NOOP (future) | promotion (ha-single-leader-role) | internal | RETAIN | added downstream; guard forces classification |
+| WAL_NOOP | promotion (ha-single-leader-role) | internal | RETAIN | landed and confirmed classified exactly as pre-declared (persist.c:5835-5839) — no longer future |
+| WAL_JOB_ARRAY_SUBMIT | handle_jobsubmit_array (client, job-array-single-wal-record) | request | WAIT (handler barrier, single barrier for the whole array) | error to client — row added by job-array-single-wal-record, not in this design's original table (Note: superseded/extended, see that spec's design.md) |
+| WAL_CONFIG_MARKER | config_commit_change (ha-term-safety task 4.2) | request-adjacent consensus record | RETAIN | commits through the same barrier as job-state facts; the upgrade requester's own barrier is owned by `do_upgrade_config_with_bytes`, not this row — row added by ha-term-safety, not in this design's original table (Note: superseded/extended, see that spec's design.md and the Config Barrier section below) |
 | WAL_CHECKPOINT | local checkpoint marker | local-only | LOCAL | unchanged (never replicated, by design) |
 
 **Responsibilities & Constraints**
@@ -338,13 +343,13 @@ int persist_commit_barrier(persist_t *p, uint32_t timeout_ms);
 | Requirements | 1.1, 1.2, 1.3, 1.4, 2.3, 2.4 |
 
 **Responsibilities & Constraints**
-- Restructure pattern (uniform): `rc = <ctrl fn>(...); if (rc == 0) rc = persist_commit_barrier(&c->persist, 0); reply.status = rc; xce_reply(...)`. Applies to:
-  - `do_jobcontrol_op` (job.c:921-934) — covers the not-started branch's WAL_JOB_UPDATE/WAL_JOB_FAIL entries; the running-job branch (forward to worker, no local entry) waits on an already-covered watermark and returns immediately
-  - `do_queuecontrol_op` (queue.c:119-132)
-  - `do_hostcontrol_op` (worker.c:689-702)
-  - `handle_worker_join` (join.c:33-119) — barrier inserted between the AFTER_EVENT hook (join.c:104-105) and the success reply (join.c:107-109); error replies (validate_host, hostconfig, idx < 0) stay immediate; `on_worker_join` dataex push stays after the reply
-- Receipt-ACK verdicts (verified, kept, classification-guard-enforced): `handle_jobdispatch` (job.c:459-491) — early reply at job.c:466-471 is ACK-of-receipt; `dispatch_a_job_` produces no WAL entries (allocation results return via `notify_dispatch`, and resulting job-state changes arrive later as worker facts) → per the umbrella contract the early reply stands (2.3). `do_set_preason`/`do_clr_preason` (job.c:1040/1068) — scheduler-origin facts (WAL_JOB_PREASON, RETAIN); early reply is receipt-only. `do_update_op` paths (opupd.c) — worker-origin facts (rusage → WAL_JOB_RUSAGE/WAL_JOB_UPDATE RETAIN; resvals produce no WAL); replies are receipt-only
-- Exhaustive audit (2.4): every `xce_reply` call site under src/controller/ (excluding testsuite) is enumerated by grep and assigned one of {post-commit confirmation, receipt-only ACK, read-only/replication/error reply}; the audit table is committed with the task evidence. Known control-path population at design time: job.c:471,524,569,578,932,1050,1078; queue.c:130; worker.c:700; join.c:28,54,66,109,139,152,198,225,248,257,283; opupd.c:48,64,88; cluster.c (gate/shutdown/hello/notify-path replies); config_repl.c:619-1129 (config/replication track); ha.c:1035; opget.c:222 (read-only); reply.c:61; task.c:97 — line numbers re-verified by grep during implementation
+- Restructure pattern (uniform): `rc = <ctrl fn>(...); if (rc == 0) rc = persist_commit_barrier(&c->persist, 0); reply.status = rc; xce_reply(...)`. Confirmed still in place at (line numbers re-verified against current source, drifted from file growth since this spec's 2026-06-13 close — see File Structure Plan §Modified Files for the specific old→new lines):
+  - `do_jobcontrol_op` (job.c:2198-2224) — covers the not-started branch's WAL_JOB_UPDATE/WAL_JOB_FAIL entries; the running-job branch (forward to worker, no local entry) waits on an already-covered watermark and returns immediately
+  - `do_queuecontrol_op` (queue.c:128-154)
+  - `do_hostcontrol_op` (worker.c:796-822)
+  - `handle_worker_join` (join.c:48-149) — barrier inserted between the AFTER_EVENT hook (join.c:118-120) and the success reply (join.c:137-139); error replies (validate_host, hostconfig, idx < 0) stay immediate; `on_worker_join` dataex push stays after the reply
+- Receipt-ACK verdicts (verified, kept, classification-guard-enforced): `handle_jobdispatch` (job.c:1119, was 459-491) — early reply is ACK-of-receipt; `dispatch_a_job_` produces no WAL entries (allocation results return via `notify_dispatch`, and resulting job-state changes arrive later as worker facts) → per the umbrella contract the early reply stands (2.3). `do_set_preason`/`do_clr_preason` (job.c:2330/2358, was 1040/1068) — scheduler-origin facts (WAL_JOB_PREASON, RETAIN); early reply is receipt-only. `do_update_op` paths (opupd.c:48/64/88) — worker-origin facts (rusage → WAL_JOB_RUSAGE/WAL_JOB_UPDATE RETAIN; resvals produce no WAL); replies are receipt-only
+- Exhaustive audit (2.4): every `xce_reply` call site under src/controller/ (excluding testsuite) is enumerated by grep and assigned one of {post-commit confirmation, receipt-only ACK, read-only/replication/error reply}; the audit table was committed with the task evidence. The design-time line-number snapshot originally recorded here (job.c/queue.c/worker.c/join.c/opupd.c/cluster.c/config_repl.c/ha.c/opget.c/reply.c/task.c line lists) has drifted wholesale with subsequent file growth across a dozen-plus later specs and is no longer independently meaningful as a citation list; the audit's *coverage claim* (every reply site classified, zero unclassified) is not re-verifiable line-by-line within this pass's scope — re-run the grep sweep against current source before relying on the exhaustiveness claim
 - A missed handler silently violates umbrella 2.1 — the integration test asserting no-success-reply-before-commit is the behavioral backstop (1.4)
 
 **Contracts**: Service [x] — external reply semantics: success now implies majority durability; new error codes `-ETIMEDOUT`/`-XCE_ERR_NOT_READY` surface to requesters through existing reply envelopes (Go tier passes them through, per umbrella out-of-boundary)
@@ -365,9 +370,12 @@ Summary-only: add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode
 | Requirements | 4.1, 4.2, 4.3, 4.4 |
 
 **Responsibilities & Constraints**
-- `push_config_to_standbys` (config_repl.c:866-930) signature: `int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen, const uint8_t sha256[32], uint64_t new_tag, int *acked_out)` — returns the barrier result (0 / `-ETIMEDOUT`), push count via `acked_out`. The majority wait (config_repl.c:916-917) passes `new_tag` down for per-push evaluation; `committed_config_tag` advance stays inside the success branch only (config_repl.c:918-922 — 4.3 preserved)
-- configAck branch of `repl_wait_majority_ack` (repl.c:418-438) hardened to mirror the engine's rules on the config track (umbrella: "same majority arithmetic, parallel simpler track"): count standbys with `acked_config_lsn >= push_tag` (not `> 0` — a previous upgrade's ack no longer counts), majority = `1 (self) + count > n_peers / 2` over configured members (not `n_active`), `n_peers <= 1` → immediate success, `n_peers > 1` with zero covering acks → `-ETIMEDOUT` at deadline (the `n_active == 0 → success` short-circuit is removed from this branch). The configAck branch reuses the `entry_lsn` parameter already added to `repl_wait_majority_ack` by ha-commit-engine as the config push tag — no second parameter is added (`repl.h` comment updated accordingly); the function is now configAck-only (walAck callers are on the engine)
-- Caller `do_upgrade_config_with_bytes` (config_repl.c:1056-1094): on non-zero return — `send_upgrade_error(ep, req, c, ...)` to the admin tool (4.2), skip the success ACK, skip `restart_requested` (4.4). Disk save and generation bump have already happened (pre-existing sequence); the reply makes the failure explicit and the cluster-wide tag stays uncommitted — convergence/rollback of a half-pushed config remains the existing OP_replConfig anti-entropy behavior (out of boundary)
+
+> **Note: partially superseded by ha-term-safety (task 4.2), see that spec's design.md.** As of this spec's 2026-06-13 close, `do_upgrade_config_with_bytes` called `push_config_to_standbys` directly and its return code alone gated the requester's success/failure reply (described below, still architecturally accurate for `push_config_to_standbys` itself). ha-term-safety inserted a layer: `do_upgrade_config_with_bytes` now calls `config_commit_change` (config_repl.c:535), which appends a `WAL_CONFIG_MARKER` as the authoritative (term,lsn) consensus record and *internally* calls `push_config_to_standbys` as the out-of-band byte-delivery quorum underneath that marker. The requester's success/failure now gates on the marker reaching the committed prefix, not directly on `push_config_to_standbys`'s return. The majority-ACK arithmetic itself (below) is unchanged and still lives in `push_config_to_standbys`/`repl_wait_majority_ack`.
+
+- `push_config_to_standbys` (config_repl.c:1609-1654, was 866-930) signature: `int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen, const uint8_t sha256[32], uint64_t new_tag, int *acked_out)` — confirmed unchanged — returns the barrier result (0 / `-ETIMEDOUT`), push count via `acked_out`. The majority wait passes `new_tag` down for per-push evaluation; `committed_config_tag` advance stays inside the success branch only (4.3 preserved, now additionally described by ha-term-safety as a "non-authoritative cache" of the marker-derived committed config)
+- configAck branch of `repl_wait_majority_ack` (repl.c:616-699, was 418-438) hardened to mirror the engine's rules on the config track (umbrella: "same majority arithmetic, parallel simpler track") — confirmed unchanged: count standbys with `acked_config_lsn >= push_tag` (not `> 0` — a previous upgrade's ack no longer counts), majority = `1 (self) + count > n_peers / 2` over configured members (not `n_active`), `n_peers <= 1` → immediate success, `n_peers > 1` with zero covering acks → `-ETIMEDOUT` at deadline (the `n_active == 0 → success` short-circuit is removed from this branch). The configAck branch reuses the `entry_lsn` parameter already added to `repl_wait_majority_ack` by ha-commit-engine as the config push tag — no second parameter is added (`repl.h` comment updated accordingly); the function is now configAck-only (walAck callers are on the engine)
+- Caller `do_upgrade_config_with_bytes` (config_repl.c:1696-1855, was 1056-1094): on non-zero return from `config_commit_change` (was directly from `push_config_to_standbys`, see supersession note above) — `send_upgrade_error(ep, req, c, ...)` to the admin tool (4.2), skip the success ACK, skip restart. Disk save and generation bump have already happened (pre-existing sequence); the reply makes the failure explicit and the cluster-wide tag stays uncommitted — convergence/rollback of a half-pushed config remains the existing OP_replConfig anti-entropy behavior (out of boundary). Materialization on success now happens via `apply_committed_entry`'s WAL_CONFIG_MARKER case on every member's own apply of the marker — no restart is requested or required (a further ha-term-safety change beyond this spec's original "restart is gated on success" framing; see the Flow decisions paragraph below, now stale on this point)
 - One in-flight config change at a time (umbrella-locked) — no per-entry registry needed
 
 **Contracts**: Service [x] — `upgradeConfig` reply contract: success ⇔ majority-acked config; failure carries an error string and implies no restart
@@ -385,7 +393,10 @@ Summary-only: add `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode
 | Requirements | 5.1, 5.2, 5.3 |
 
 **Responsibilities & Constraints**
-- Guard placement: at the top of `mark_job_start` (job.c:583), `mark_job_finish` (job.c:638), `mark_job_fail` (job.c:696), `mark_job_update` (job.c:726), under the caller-held job WRITE lock, before the BEFORE_EVENT hook fires (a guarded-out transition runs no hooks → no WAL entry, no double-fire)
+
+> **Note: partially superseded by raft-apply-decoupling, see that spec's design.md.** `mark_job_fail`/`mark_job_update` (job.c:1830/1929) still exist essentially as described below — guard-then-mutate synchronously under the caller-held WRITE lock, used by the client-request path (`job_ctrl`). `mark_job_start`/`mark_job_finish` no longer exist under those names: they were renamed `record_job_start_fact`/`record_job_finish_fact` (job.c:1625/1734, worker-fact receipt path only) and split into two phases — a receipt-time "peek" of the identical guard (log-only, for the rejected-transition WARN; no mutation) plus a durable-append-only WAL write with no immediate mutation and no AFTER_EVENT hook, followed by the *actual* guarded mutation + hook firing later, deferred to `apply_committed_entry()` (via the extracted `apply_job_start_transition`/`apply_job_finish_transition` cores) once the entry reaches the committed prefix and the apply pump dispatches it. The guard *rules* (duplicate → no-op, backward → rejected+WARN) and their *outcome* are unchanged from what this design specifies below; the *timing* — guard-before-append vs. guard-at-apply-time — is not.
+
+- Guard placement (as designed; see supersession note above for the current split for START/FINISH): at the top of `mark_job_start` (job.c:583), `mark_job_finish` (job.c:638), `mark_job_fail` (job.c:696), `mark_job_update` (job.c:726), under the caller-held job WRITE lock, before the BEFORE_EVENT hook fires (a guarded-out transition runs no hooks → no WAL entry, no double-fire)
 - Rules (terminal = any of DONE | EXIT, with or without PERR/PDONE modifiers):
   - duplicate: requested resulting state == current state → INFO log, return 0, no mutation, no hooks (5.1)
   - backward: current state terminal AND requested state non-terminal (START on finished job, UPDATE clearing terminal bits) → WARN log, return `-EALREADY`-style rejection, state preserved (5.2)