myxce/.kiro/specs/ha-commit-barrier-rollout/research.md

Research & Design Decisions: ha-commit-barrier-rollout

Summary

• Feature: ha-commit-barrier-rollout (sub-spec 2 of 4, ha-consistency-invariants umbrella)
• Discovery Scope: Extension (light discovery — umbrella research.md is the primary discovery artifact; this pass verified its anchors against the current tree, 2026-06-11)
• Key Findings:
  • Every reply-before-commit call site named by the umbrella still exists; line anchors re-verified (table below)
  • Call-site verification of the umbrella classification table found two deviations the umbrella anticipated ("verified against call sites during implementation; table is the contract"): WAL_JOB_STOP/RESUME are fact-origin only, and WAL_JOB_UPDATE (absent from the umbrella table) is dual-origin
  • The configAck branch of repl_wait_majority_ack has the same three counting defects the commit engine fixed for walAck (> 0 counting, n_active == 0 → success, n_active denominator)
  • The dispatch gate rejects OPCode_notify (and update/preason) under quorum loss — the bug-396 silent fact-loss hole is gate-level, not just replication-level

Verified Code Anchors (2026-06-11, persistent_ha working tree)

Concern	Anchor (verified by grep/read)	Status vs umbrella audit
do_jobcontrol_op replies after mutation, async WAL only	job.c:921-934 (reply at 932)	unchanged
handle_jobdispatch early reply	job.c:459-491 (reply at 466-471)	umbrella cited 459-471 — still accurate
handle_jobsubmit reference implementation	job.c:495-581 (hook rc checked 563-571, reply 573-578)	umbrella cited 563-578 — still accurate
do_queuecontrol_op	queue.c:119-132 (reply at 130); queue_ctrl fires AFTER hook under queue WRITE lock (queue.c:76-112, hook at 107)	new precision: lock-scope drives WAIT placement
do_hostcontrol_op	worker.c:689-702 (reply at 700)	new anchor
handle_worker_join	join.c:33-119 (AFTER hook 104-105, success reply 107-109)	new anchor
do_set_preason / do_clr_preason early replies	job.c:1040-1066 (reply at 1050) / job.c:1068+ (reply at 1078)	scheduler-origin facts — receipt-ACK verdict
do_update_op family	opupd.c:48/64/88 reply sites	worker-origin facts — receipt-ACK verdict
do_notify (no reply, fire-and-forget)	cluster.c:118-157	unchanged
Hook → WAL type map	persist.c:1068-1088 (detail_to_wal_type)	classification dispatch insertion point
Job hook: submit sync path / fact async path	persist.c:1093-1169 (sync 1107-1129, async 1169)	unchanged
Queue / worker hooks (async)	persist.c:1172-1202 (append 1199) / persist.c:1204-1244 (append 1241)	unchanged
persist_wal_sync_append + timeout	persist.c:1003-1049; timeout p->config.sync_timeout_ms (DEFAULT_SYNC_TIMEOUT_MS=50, config.h:93)	committed_lsn store at 1035-1036 is removed by ha-commit-engine (upstream)
Dispatch gate PRIMARY no-quorum allowlist	ha.c:1527 (ha_dispatch_gate), allowlist 1537-1556; notify falls to default → -XCE_ERR_NOT_READY	fact admission insertion point
push_config_to_standbys ACK result ignored	config_repl.c:866-930 (wait at 916-917, rc logged-only 918-927, returns acked at 929)	umbrella cited 916-928 — still accurate
upgradeConfig caller ignores result	config_repl.c:1056-1094 (push at 1057, unconditional success ACK 1062-1073, restart 1091-1093)	umbrella cited 1057 — still accurate
configAck counting defects	repl.c:386-462 (acked_config_lsn > 0 at 422-424, n_active==0 → success at 432-435, denominator n_active at 437)	mirrors the walAck defects fixed upstream
repl_on_config_ack stores reported tag/lsn	repl.c:351-371; standby_conn_t.acked_config_lsn (repl.h:25)	enables per-push >= push_tag comparison
mark_job_start/finish/fail/update unguarded	job.c:583/638/696/726	unchanged
job_ctrl branches	job.c:306-438 — running job forwards to worker (357, no local entry); not-started job mutates via mark_job_update (391/401) / mark_job_fail (415)	classification evidence (deviation §2)
dispatch_a_job_ produces no WAL	job.c:180-304 — sends xce_startjob_req, mutates j->job.allocs, fires no hooks	handle_jobdispatch receipt-ACK evidence
WAL type enum	src/include/wal_types.h:44-58 (no WAL_NOOP yet)	WAL_NOOP arrives with ha-single-leader-role
Control-path xce_reply population	job.c:471,524,569,578,932,1050,1078; queue.c:130; worker.c:700; join.c:28,54,66,109,139,152,198,225,248,257,283; opupd.c:48,64,88; cluster.c:255-490; config_repl.c:619-1129; ha.c:1035; opget.c:222; reply.c:61; task.c:97	sweep-task input; re-grep before editing
Opcode → handler dispatch	cluster.c:325-363 (jobsub/jobdispatch/notify/update/jobcontrol/queuecontrol/hostcontrol/setpreason/clrpreason/join)	gate allowlist + sweep scoping

Upstream interface consumed (ha-commit-engine design, generated 2026-06-11): commit_submit(ce, lsn, COMMIT_MODE_WAIT|RETAIN, timeout_ms) → 0 / -ETIMEDOUT / -XCE_ERR_NOT_READY; RETAIN returns immediately; n_peers<=1 self-quorum; admission cap n_vcpus-relative; engine embedded in cluster_t, reachable from persist_t via p->repl.ce. commit.c/commit.h do not exist in the tree yet — ha-commit-engine must be implemented first (hard upstream dependency; tasks assume its interface as specified).

Research Log

§1 Reply-ordering reality and the WAIT placement question

• Context: The umbrella Reply-Ordering Contract fixes the handler sequence (validate → mutate → append → commit_submit(WAIT) → reply). Where should the WAIT physically run — inside the persistence hook (like job submit) or in the handler?
• Findings: queue/worker/job-state hooks fire while the entity WRITE lock is held (queue.c:107 inside lock taken at 76; mark_job_* called under job WRITE lock, e.g. job.c:629-631). Job submit is the exception: its AFTER hook runs against a job list after per-job locks are released (job.c:553), which is why its in-hook sync wait is safe.
• Implications: In-hook WAIT for queue/worker/job-control would block entity readers for up to the commit timeout. Handler-level barrier (after locks released, before reply) keeps the same completion semantics and matches the umbrella's handler-sequence wording. Decision below.

§2 Classification table call-site verification (umbrella-mandated)

• Context: Umbrella table marks WAL_JOB_STOP/RESUME as "request (client job control) → WAIT" and omits WAL_JOB_UPDATE.
• Findings: WAL_JOB_STOP/RESUME entries are appended only by on_job_stop/on_job_resume, reachable solely from do_notify (worker-reported stoppedJob/resumedJob facts — cluster.c:131-136). Client job control (do_jobcontrol_op → job_ctrl) never produces them: for a running job it forwards the signal to the worker (job.c:357) and the resulting transition comes back later as a worker fact; for a not-yet-started job it mutates locally through mark_job_update (WAL_JOB_UPDATE) or mark_job_fail (WAL_JOB_FAIL). WAL_JOB_UPDATE/WAL_JOB_FAIL are therefore dual-origin (request via job_ctrl, fact via postdone/notify/rusage).
• Implications: Classify STOP/RESUME/UPDATE/FAIL as fact/RETAIN at the seam; realize the umbrella's request-shaped guarantee ("error to client") for client job control at the handler barrier in do_jobcontrol_op. This satisfies the umbrella's intent through its own prescribed handler sequence — the barrier is a property of the handler, not of the entry type. Not a contradiction: the umbrella explicitly delegated call-site verification, and the failure-column behavior is preserved.

§3 handle_jobdispatch classification (umbrella left it open)

• Findings: dispatch_a_job_ (job.c:180-304) sends xce_startjob_req to the worker and updates j->job.allocs in memory; it fires no hooks and appends no WAL entries. Per-job outcomes return to the scheduler via notify_dispatch (job.c:440-457, called at 479-486). The job's externally meaningful state change (RUN) arrives later as a worker fact (WAL_JOB_START).
• Implications: Per the umbrella's own criterion ("if its outcomes are fact-shaped … the early reply is an ACK-of-receipt and may stand"), the verdict is receipt-ACK stands (this spec's Req 2.3). The orphan-dispatch-of-uncommitted-submit race is closed by the egress watermark filter (ha-convergence-recovery), not here. The classification guard plus a no-WAL-from-dispatch test pin the verdict.

§4 Fact loss is gate-level too (bug-396 family)

• Findings: ha_dispatch_gate's CTRL_PRIMARY branch rejects OPCode_notify, OPCode_update, OPCode_setpreason, OPCode_clrpreason with -XCE_ERR_NOT_READY whenever write quorum is absent (ha.c:1537-1556 default arm). The worker never reads notify replies (umbrella research §A), so each rejection is a permanently lost fact.
• Implications: Umbrella 2.8 ("retain … including across quorum-loss periods") requires admitting these opcodes without write quorum; durability is then owned by RETAIN. Allowlist change is confined to the PRIMARY branch; the COMMITTING branch is left for ha-single-leader-role's wholesale deletion.

§5 Config track hardening scope

• Findings: configAck counting (repl.c:418-438) shares all three walAck defects; repl_on_config_ack already stores the standby's acknowledged tag (repl.c:351-371), so per-push comparison (>= push_tag) needs only a parameter, not new state. committed_config_tag advance is already success-gated (config_repl.c:918-922).
• Implications: Hardening lives in the configAck branch + push_config_to_standbys return contract + caller failure path. No engine involvement (umbrella-locked: config is a parallel, simpler track sharing the majority arithmetic rule — one in-flight change at a time).

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Verdict
In-hook WAIT for all request types	extend the job-submit sync-append pattern to queue/worker/job-control hooks	one mechanism, no handler edits	blocks commit waits under entity WRITE locks (readers stall up to timeout); job-control hooks fire per-transition, not per-operation	rejected
Handler-level barrier (chosen)	hooks append (RETAIN-registered); request handlers call persist_commit_barrier() after locks, before reply	matches umbrella handler sequence verbatim; no lock-hold waits; uniform mechanical sweep pattern; covers dual-origin types from request contexts	barrier watermark may over-wait on unrelated concurrent entries (bounded by timeout)	selected
Per-entry LSN plumbing (hooks return entry LSNs to handlers)	exact-entry waits	no over-wait	invasive signature changes through the hook system for negligible benefit (over-wait is bounded and rare)	rejected (YAGNI)

Design Decisions

Decision: Handler-level commit barrier on the appended watermark

• Context: §1 lock-scope finding vs the umbrella WAIT mode for queue/worker/job-control.
• Selected Approach: persist_commit_barrier(p, timeout_ms) = snapshot current_lsn, commit_submit(WAIT) on it; called by each request handler between mutation success and reply.
• Rationale: Single-writer appends + contiguous commit advancement (engine postcondition) make the watermark cover everything the operation appended; the umbrella's Reply-Ordering Contract literally specifies the handler sequence.
• Trade-offs: Possible over-wait on concurrent unrelated uncommitted entries (bounded by sync_timeout_ms; those entries must commit anyway). A no-op control operation may wait behind unrelated traffic — accepted (request-shaped ops are gate-rejected during quorum loss, so the wait resolves quickly in healthy clusters).
• Follow-up: Integration test pins reply-after-commit ordering per opcode.

Decision: Verified classification deviations (STOP/RESUME fact-only; UPDATE dual-origin)

• See Research Log §2. Umbrella conformance preserved through the handler barrier; the verified table in design.md is the implementation contract. Inline umbrella tags kept (2.1/2.8).

Decision: Production fallback for unclassified WAL types = RETAIN

• Context: Req 2.2 requires fail-fast in testing; production must not crash the primary on a programmer omission.
• Selected Approach: XCE_TESTING assert; production ERROR log + RETAIN treatment.
• Rationale: Fail-safe direction is retention (never dropped, never falsely confirmed) — a misrouted request-shaped type degrades to a missing barrier caught by the sweep audit/tests, not to data loss.

Decision: Failed config upgrade does not restart

• Context: Roadmap residual ("don't restart the primary before the commit barrier passes"); umbrella 5.2.
• Selected Approach: do_upgrade_config_with_bytes gates the success ACK and restart_requested on the barrier result; disk save/generation bump remain pre-barrier (existing sequence), reconciled by the existing OP_replConfig anti-entropy (out of boundary).
• Trade-offs: A failed upgrade leaves the new config on the primary's disk but uncommitted cluster-wide; explicit failure to the operator makes the state visible. Full save-after-commit resequencing deferred (would touch checkpoint/restart machinery beyond this slice).

Risks & Mitigations

• Missed reply site silently violates umbrella 2.1 — exhaustive grep audit task over all xce_reply sites + behavioral integration test (no success reply before commit)
• Upstream ha-commit-engine not yet implemented — tasks consume only its published interface; if signatures drift, revalidation trigger fires (boundary section)
• New timeout errors surfacing to scheduler/worker/CLI callers — deliberate CP semantics (umbrella); error codes flow through existing reply envelopes; callers' retry behavior unchanged
• Guard rejections changing mark_job_* return codes — sweep checks all callers treat duplicate-as-success-no-op

References

• .kiro/specs/ha-consistency-invariants/{requirements,design,research}.md — authoritative umbrella contract (esp. design §Write-Path Classification, §Config Barrier, §Idempotency Guards; research locked decisions 1, 2, 4)
• .kiro/specs/ha-commit-engine/design.md — consumed engine interface (commit.h verbatim), migration note "retired by ha-commit-barrier-rollout"
• .kiro/steering/roadmap.md — dependency ordering; superseded item "Quorum-ACK gate before config restart" residual
• .kiro/steering/ha-async-io.md, testing.md, error-handling.md — threading, TDD, errnox conventions
• Raft paper §5.3 (commit rule) — via umbrella; this spec adds no new consensus logic