myxce/.kiro/specs/ha-commit-barrier-rollout/research.md

# Research & Design Decisions: ha-commit-barrier-rollout

## Summary
- **Feature**: `ha-commit-barrier-rollout` (sub-spec 2 of 4, ha-consistency-invariants umbrella)
- **Discovery Scope**: Extension (light discovery — umbrella research.md is the primary discovery artifact; this pass verified its anchors against the current tree, 2026-06-11)
- **Key Findings**:
  - Every reply-before-commit call site named by the umbrella still exists; line anchors re-verified (table below)
  - Call-site verification of the umbrella classification table found two deviations the umbrella anticipated ("verified against call sites during implementation; table is the contract"): WAL_JOB_STOP/RESUME are fact-origin only, and WAL_JOB_UPDATE (absent from the umbrella table) is dual-origin
  - The configAck branch of `repl_wait_majority_ack` has the same three counting defects the commit engine fixed for walAck (`> 0` counting, `n_active == 0 → success`, `n_active` denominator)
  - The dispatch gate rejects `OPCode_notify` (and update/preason) under quorum loss — the bug-396 silent fact-loss hole is gate-level, not just replication-level

## Verified Code Anchors (2026-06-11, persistent_ha working tree)

| Concern | Anchor (verified by grep/read) | Status vs umbrella audit |
|---|---|---|
| do_jobcontrol_op replies after mutation, async WAL only | job.c:921-934 (reply at 932) | unchanged |
| handle_jobdispatch early reply | job.c:459-491 (reply at 466-471) | umbrella cited 459-471 — still accurate |
| handle_jobsubmit reference implementation | job.c:495-581 (hook rc checked 563-571, reply 573-578) | umbrella cited 563-578 — still accurate |
| do_queuecontrol_op | queue.c:119-132 (reply at 130); queue_ctrl fires AFTER hook under queue WRITE lock (queue.c:76-112, hook at 107) | new precision: lock-scope drives WAIT placement |
| do_hostcontrol_op | worker.c:689-702 (reply at 700) | new anchor |
| handle_worker_join | join.c:33-119 (AFTER hook 104-105, success reply 107-109) | new anchor |
| do_set_preason / do_clr_preason early replies | job.c:1040-1066 (reply at 1050) / job.c:1068+ (reply at 1078) | scheduler-origin facts — receipt-ACK verdict |
| do_update_op family | opupd.c:48/64/88 reply sites | worker-origin facts — receipt-ACK verdict |
| do_notify (no reply, fire-and-forget) | cluster.c:118-157 | unchanged |
| Hook → WAL type map | persist.c:1068-1088 (`detail_to_wal_type`) | classification dispatch insertion point |
| Job hook: submit sync path / fact async path | persist.c:1093-1169 (sync 1107-1129, async 1169) | unchanged |
| Queue / worker hooks (async) | persist.c:1172-1202 (append 1199) / persist.c:1204-1244 (append 1241) | unchanged |
| persist_wal_sync_append + timeout | persist.c:1003-1049; timeout `p->config.sync_timeout_ms` (DEFAULT_SYNC_TIMEOUT_MS=50, config.h:93) | committed_lsn store at 1035-1036 is removed by ha-commit-engine (upstream) |
| Dispatch gate PRIMARY no-quorum allowlist | ha.c:1527 (`ha_dispatch_gate`), allowlist 1537-1556; notify falls to `default → -XCE_ERR_NOT_READY` | fact admission insertion point |
| push_config_to_standbys ACK result ignored | config_repl.c:866-930 (wait at 916-917, rc logged-only 918-927, returns acked at 929) | umbrella cited 916-928 — still accurate |
| upgradeConfig caller ignores result | config_repl.c:1056-1094 (push at 1057, unconditional success ACK 1062-1073, restart 1091-1093) | umbrella cited 1057 — still accurate |
| configAck counting defects | repl.c:386-462 (`acked_config_lsn > 0` at 422-424, `n_active==0 → success` at 432-435, denominator `n_active` at 437) | mirrors the walAck defects fixed upstream |
| repl_on_config_ack stores reported tag/lsn | repl.c:351-371; `standby_conn_t.acked_config_lsn` (repl.h:25) | enables per-push `>= push_tag` comparison |
| mark_job_start/finish/fail/update unguarded | job.c:583/638/696/726 | unchanged |
| job_ctrl branches | job.c:306-438 — running job forwards to worker (357, no local entry); not-started job mutates via mark_job_update (391/401) / mark_job_fail (415) | classification evidence (deviation §2) |
| dispatch_a_job_ produces no WAL | job.c:180-304 — sends xce_startjob_req, mutates j->job.allocs, fires no hooks | handle_jobdispatch receipt-ACK evidence |
| WAL type enum | src/include/wal_types.h:44-58 (no WAL_NOOP yet) | WAL_NOOP arrives with ha-single-leader-role |
| Control-path xce_reply population | job.c:471,524,569,578,932,1050,1078; queue.c:130; worker.c:700; join.c:28,54,66,109,139,152,198,225,248,257,283; opupd.c:48,64,88; cluster.c:255-490; config_repl.c:619-1129; ha.c:1035; opget.c:222; reply.c:61; task.c:97 | sweep-task input; re-grep before editing |
| Opcode → handler dispatch | cluster.c:325-363 (jobsub/jobdispatch/notify/update/jobcontrol/queuecontrol/hostcontrol/setpreason/clrpreason/join) | gate allowlist + sweep scoping |

Upstream interface consumed (ha-commit-engine design, generated 2026-06-11): `commit_submit(ce, lsn, COMMIT_MODE_WAIT|RETAIN, timeout_ms)` → 0 / -ETIMEDOUT / -XCE_ERR_NOT_READY; RETAIN returns immediately; n_peers<=1 self-quorum; admission cap n_vcpus-relative; engine embedded in `cluster_t`, reachable from `persist_t` via `p->repl.ce`. `commit.c`/`commit.h` do not exist in the tree yet — ha-commit-engine must be implemented first (hard upstream dependency; tasks assume its interface as specified).

## Research Log

### §1 Reply-ordering reality and the WAIT placement question
- **Context**: The umbrella Reply-Ordering Contract fixes the handler sequence (validate → mutate → append → commit_submit(WAIT) → reply). Where should the WAIT physically run — inside the persistence hook (like job submit) or in the handler?
- **Findings**: queue/worker/job-state hooks fire while the entity WRITE lock is held (queue.c:107 inside lock taken at 76; mark_job_* called under job WRITE lock, e.g. job.c:629-631). Job submit is the exception: its AFTER hook runs against a job list after per-job locks are released (job.c:553), which is why its in-hook sync wait is safe.
- **Implications**: In-hook WAIT for queue/worker/job-control would block entity readers for up to the commit timeout. Handler-level barrier (after locks released, before reply) keeps the same completion semantics and matches the umbrella's handler-sequence wording. Decision below.

### §2 Classification table call-site verification (umbrella-mandated)
- **Context**: Umbrella table marks WAL_JOB_STOP/RESUME as "request (client job control) → WAIT" and omits WAL_JOB_UPDATE.
- **Findings**: WAL_JOB_STOP/RESUME entries are appended only by `on_job_stop`/`on_job_resume`, reachable solely from `do_notify` (worker-reported stoppedJob/resumedJob facts — cluster.c:131-136). Client job control (`do_jobcontrol_op` → `job_ctrl`) never produces them: for a running job it forwards the signal to the worker (job.c:357) and the resulting transition comes back later as a worker fact; for a not-yet-started job it mutates locally through `mark_job_update` (WAL_JOB_UPDATE) or `mark_job_fail` (WAL_JOB_FAIL). WAL_JOB_UPDATE/WAL_JOB_FAIL are therefore dual-origin (request via job_ctrl, fact via postdone/notify/rusage).
- **Implications**: Classify STOP/RESUME/UPDATE/FAIL as fact/RETAIN at the seam; realize the umbrella's request-shaped guarantee ("error to client") for client job control at the handler barrier in `do_jobcontrol_op`. This satisfies the umbrella's intent through its own prescribed handler sequence — the barrier is a property of the handler, not of the entry type. Not a contradiction: the umbrella explicitly delegated call-site verification, and the failure-column behavior is preserved.

### §3 handle_jobdispatch classification (umbrella left it open)
- **Findings**: `dispatch_a_job_` (job.c:180-304) sends `xce_startjob_req` to the worker and updates `j->job.allocs` in memory; it fires no hooks and appends no WAL entries. Per-job outcomes return to the scheduler via `notify_dispatch` (job.c:440-457, called at 479-486). The job's externally meaningful state change (RUN) arrives later as a worker fact (WAL_JOB_START).
- **Implications**: Per the umbrella's own criterion ("if its outcomes are fact-shaped … the early reply is an ACK-of-receipt and may stand"), the verdict is **receipt-ACK stands** (this spec's Req 2.3). The orphan-dispatch-of-uncommitted-submit race is closed by the egress watermark filter (ha-convergence-recovery), not here. The classification guard plus a no-WAL-from-dispatch test pin the verdict.

### §4 Fact loss is gate-level too (bug-396 family)
- **Findings**: `ha_dispatch_gate`'s CTRL_PRIMARY branch rejects `OPCode_notify`, `OPCode_update`, `OPCode_setpreason`, `OPCode_clrpreason` with `-XCE_ERR_NOT_READY` whenever write quorum is absent (ha.c:1537-1556 default arm). The worker never reads notify replies (umbrella research §A), so each rejection is a permanently lost fact.
- **Implications**: Umbrella 2.8 ("retain … including across quorum-loss periods") requires admitting these opcodes without write quorum; durability is then owned by RETAIN. Allowlist change is confined to the PRIMARY branch; the COMMITTING branch is left for ha-single-leader-role's wholesale deletion.

### §5 Config track hardening scope
- **Findings**: configAck counting (repl.c:418-438) shares all three walAck defects; `repl_on_config_ack` already stores the standby's acknowledged tag (repl.c:351-371), so per-push comparison (`>= push_tag`) needs only a parameter, not new state. `committed_config_tag` advance is already success-gated (config_repl.c:918-922).
- **Implications**: Hardening lives in the configAck branch + `push_config_to_standbys` return contract + caller failure path. No engine involvement (umbrella-locked: config is a parallel, simpler track sharing the majority arithmetic rule — one in-flight change at a time).

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Verdict |
|--------|-------------|-----------|---------------------|---------|
| In-hook WAIT for all request types | extend the job-submit sync-append pattern to queue/worker/job-control hooks | one mechanism, no handler edits | blocks commit waits under entity WRITE locks (readers stall up to timeout); job-control hooks fire per-transition, not per-operation | rejected |
| Handler-level barrier (chosen) | hooks append (RETAIN-registered); request handlers call `persist_commit_barrier()` after locks, before reply | matches umbrella handler sequence verbatim; no lock-hold waits; uniform mechanical sweep pattern; covers dual-origin types from request contexts | barrier watermark may over-wait on unrelated concurrent entries (bounded by timeout) | selected |
| Per-entry LSN plumbing (hooks return entry LSNs to handlers) | exact-entry waits | no over-wait | invasive signature changes through the hook system for negligible benefit (over-wait is bounded and rare) | rejected (YAGNI) |

## Design Decisions

### Decision: Handler-level commit barrier on the appended watermark
- **Context**: §1 lock-scope finding vs the umbrella WAIT mode for queue/worker/job-control.
- **Selected Approach**: `persist_commit_barrier(p, timeout_ms)` = snapshot `current_lsn`, `commit_submit(WAIT)` on it; called by each request handler between mutation success and reply.
- **Rationale**: Single-writer appends + contiguous commit advancement (engine postcondition) make the watermark cover everything the operation appended; the umbrella's Reply-Ordering Contract literally specifies the handler sequence.
- **Trade-offs**: Possible over-wait on concurrent unrelated uncommitted entries (bounded by `sync_timeout_ms`; those entries must commit anyway). A no-op control operation may wait behind unrelated traffic — accepted (request-shaped ops are gate-rejected during quorum loss, so the wait resolves quickly in healthy clusters).
- **Follow-up**: Integration test pins reply-after-commit ordering per opcode.

### Decision: Verified classification deviations (STOP/RESUME fact-only; UPDATE dual-origin)
- See Research Log §2. Umbrella conformance preserved through the handler barrier; the verified table in design.md is the implementation contract. Inline umbrella tags kept (2.1/2.8).

### Decision: Production fallback for unclassified WAL types = RETAIN
- **Context**: Req 2.2 requires fail-fast in testing; production must not crash the primary on a programmer omission.
- **Selected Approach**: XCE_TESTING assert; production ERROR log + RETAIN treatment.
- **Rationale**: Fail-safe direction is retention (never dropped, never falsely confirmed) — a misrouted request-shaped type degrades to a missing barrier caught by the sweep audit/tests, not to data loss.

### Decision: Failed config upgrade does not restart
- **Context**: Roadmap residual ("don't restart the primary before the commit barrier passes"); umbrella 5.2.
- **Selected Approach**: `do_upgrade_config_with_bytes` gates the success ACK and `restart_requested` on the barrier result; disk save/generation bump remain pre-barrier (existing sequence), reconciled by the existing OP_replConfig anti-entropy (out of boundary).
- **Trade-offs**: A failed upgrade leaves the new config on the primary's disk but uncommitted cluster-wide; explicit failure to the operator makes the state visible. Full save-after-commit resequencing deferred (would touch checkpoint/restart machinery beyond this slice).

## Risks & Mitigations
- Missed reply site silently violates umbrella 2.1 — exhaustive grep audit task over all `xce_reply` sites + behavioral integration test (no success reply before commit)
- Upstream ha-commit-engine not yet implemented — tasks consume only its published interface; if signatures drift, revalidation trigger fires (boundary section)
- New timeout errors surfacing to scheduler/worker/CLI callers — deliberate CP semantics (umbrella); error codes flow through existing reply envelopes; callers' retry behavior unchanged
- Guard rejections changing `mark_job_*` return codes — sweep checks all callers treat duplicate-as-success-no-op

## References
- `.kiro/specs/ha-consistency-invariants/{requirements,design,research}.md` — authoritative umbrella contract (esp. design §Write-Path Classification, §Config Barrier, §Idempotency Guards; research locked decisions 1, 2, 4)
- `.kiro/specs/ha-commit-engine/design.md` — consumed engine interface (commit.h verbatim), migration note "retired by ha-commit-barrier-rollout"
- `.kiro/steering/roadmap.md` — dependency ordering; superseded item "Quorum-ACK gate before config restart" residual
- `.kiro/steering/ha-async-io.md`, `testing.md`, `error-handling.md` — threading, TDD, errnox conventions
- Raft paper §5.3 (commit rule) — via umbrella; this spec adds no new consensus logic