myxce/.kiro/specs/ha-commit-barrier-rollout/research.md

Research & Design Decisions: ha-commit-barrier-rollout

Summary

• Feature: ha-commit-barrier-rollout (sub-spec 2 of 4, ha-consistency-invariants umbrella)
• Discovery Scope: Extension (light discovery — umbrella research.md is the primary discovery artifact; this pass verified its anchors against the current tree, 2026-06-11)
• Key Findings:
  • Every reply-before-commit call site named by the umbrella still exists; line anchors re-verified (table below)
  • Call-site verification of the umbrella classification table found two deviations the umbrella anticipated ("verified against call sites during implementation; table is the contract"): WAL_JOB_STOP/RESUME are fact-origin only, and WAL_JOB_UPDATE (absent from the umbrella table) is dual-origin
  • The configAck branch of repl_wait_majority_ack has the same three counting defects the commit engine fixed for walAck (> 0 counting, n_active == 0 → success, n_active denominator)
  • The dispatch gate rejects OPCode_notify (and update/preason) under quorum loss — the bug-396 silent fact-loss hole is gate-level, not just replication-level

Verified Code Anchors (2026-06-11, persistent_ha working tree)

Concern	Anchor (verified by grep/read)	Status vs umbrella audit
do_jobcontrol_op replies after mutation, async WAL only	job.c:921-934 (reply at 932)	unchanged
handle_jobdispatch early reply	job.c:459-491 (reply at 466-471)	umbrella cited 459-471 — still accurate
handle_jobsubmit reference implementation	job.c:495-581 (hook rc checked 563-571, reply 573-578)	umbrella cited 563-578 — still accurate
do_queuecontrol_op	queue.c:119-132 (reply at 130); queue_ctrl fires AFTER hook under queue WRITE lock (queue.c:76-112, hook at 107)	new precision: lock-scope drives WAIT placement
do_hostcontrol_op	worker.c:689-702 (reply at 700)	new anchor
handle_worker_join	join.c:33-119 (AFTER hook 104-105, success reply 107-109)	new anchor
do_set_preason / do_clr_preason early replies	job.c:1040-1066 (reply at 1050) / job.c:1068+ (reply at 1078)	scheduler-origin facts — receipt-ACK verdict
do_update_op family	opupd.c:48/64/88 reply sites	worker-origin facts — receipt-ACK verdict
do_notify (no reply, fire-and-forget)	cluster.c:118-157	unchanged
Hook → WAL type map	persist.c:1068-1088 (detail_to_wal_type)	classification dispatch insertion point
Job hook: submit sync path / fact async path	persist.c:1093-1169 (sync 1107-1129, async 1169)	unchanged
Queue / worker hooks (async)	persist.c:1172-1202 (append 1199) / persist.c:1204-1244 (append 1241)	unchanged
persist_wal_sync_append + timeout	persist.c:1003-1049; timeout p->config.sync_timeout_ms (DEFAULT_SYNC_TIMEOUT_MS=50, config.h:93)	committed_lsn store at 1035-1036 is removed by ha-commit-engine (upstream)
Dispatch gate PRIMARY no-quorum allowlist	ha.c:1527 (ha_dispatch_gate), allowlist 1537-1556; notify falls to default → -XCE_ERR_NOT_READY	fact admission insertion point
push_config_to_standbys ACK result ignored	config_repl.c:866-930 (wait at 916-917, rc logged-only 918-927, returns acked at 929)	umbrella cited 916-928 — still accurate
upgradeConfig caller ignores result	config_repl.c:1056-1094 (push at 1057, unconditional success ACK 1062-1073, restart 1091-1093)	umbrella cited 1057 — still accurate
configAck counting defects	repl.c:386-462 (acked_config_lsn > 0 at 422-424, n_active==0 → success at 432-435, denominator n_active at 437)	mirrors the walAck defects fixed upstream
repl_on_config_ack stores reported tag/lsn	repl.c:351-371; standby_conn_t.acked_config_lsn (repl.h:25)	enables per-push >= push_tag comparison
mark_job_start/finish/fail/update unguarded	job.c:583/638/696/726	unchanged
job_ctrl branches	job.c:306-438 — running job forwards to worker (357, no local entry); not-started job mutates via mark_job_update (391/401) / mark_job_fail (415)	classification evidence (deviation §2)
dispatch_a_job_ produces no WAL	job.c:180-304 — sends xce_startjob_req, mutates j->job.allocs, fires no hooks	handle_jobdispatch receipt-ACK evidence
WAL type enum	src/include/wal_types.h:44-58 (no WAL_NOOP yet)	WAL_NOOP arrives with ha-single-leader-role
Control-path xce_reply population	job.c:471,524,569,578,932,1050,1078; queue.c:130; worker.c:700; join.c:28,54,66,109,139,152,198,225,248,257,283; opupd.c:48,64,88; cluster.c:255-490; config_repl.c:619-1129; ha.c:1035; opget.c:222; reply.c:61; task.c:97	sweep-task input; re-grep before editing
Opcode → handler dispatch	cluster.c:325-363 (jobsub/jobdispatch/notify/update/jobcontrol/queuecontrol/hostcontrol/setpreason/clrpreason/join)	gate allowlist + sweep scoping

Upstream interface consumed (ha-commit-engine design, generated 2026-06-11): commit_submit(ce, lsn, COMMIT_MODE_WAIT|RETAIN, timeout_ms) → 0 / -ETIMEDOUT / -XCE_ERR_NOT_READY; RETAIN returns immediately; n_peers<=1 self-quorum; admission cap n_vcpus-relative; engine embedded in cluster_t, reachable from persist_t via p->repl.ce. commit.c/commit.h do not exist in the tree yet — ha-commit-engine must be implemented first (hard upstream dependency; tasks assume its interface as specified).

Research Log

§1 Reply-ordering reality and the WAIT placement question

• Context: The umbrella Reply-Ordering Contract fixes the handler sequence (validate → mutate → append → commit_submit(WAIT) → reply). Where should the WAIT physically run — inside the persistence hook (like job submit) or in the handler?
• Findings: queue/worker/job-state hooks fire while the entity WRITE lock is held (queue.c:107 inside lock taken at 76; mark_job_* called under job WRITE lock, e.g. job.c:629-631). Job submit is the exception: its AFTER hook runs against a job list after per-job locks are released (job.c:553), which is why its in-hook sync wait is safe.
• Implications: In-hook WAIT for queue/worker/job-control would block entity readers for up to the commit timeout. Handler-level barrier (after locks released, before reply) keeps the same completion semantics and matches the umbrella's handler-sequence wording. Decision below.

§2 Classification table call-site verification (umbrella-mandated)

• Context: Umbrella table marks WAL_JOB_STOP/RESUME as "request (client job control) → WAIT" and omits WAL_JOB_UPDATE.
• Findings: WAL_JOB_STOP/RESUME entries are appended only by on_job_stop/on_job_resume, reachable solely from do_notify (worker-reported stoppedJob/resumedJob facts — cluster.c:131-136). Client job control (do_jobcontrol_op → job_ctrl) never produces them: for a running job it forwards the signal to the worker (job.c:357) and the resulting transition comes back later as a worker fact; for a not-yet-started job it mutates locally through mark_job_update (WAL_JOB_UPDATE) or mark_job_fail (WAL_JOB_FAIL). WAL_JOB_UPDATE/WAL_JOB_FAIL are therefore dual-origin (request via job_ctrl, fact via postdone/notify/rusage).
• Implications: Classify STOP/RESUME/UPDATE/FAIL as fact/RETAIN at the seam; realize the umbrella's request-shaped guarantee ("error to client") for client job control at the handler barrier in do_jobcontrol_op. This satisfies the umbrella's intent through its own prescribed handler sequence — the barrier is a property of the handler, not of the entry type. Not a contradiction: the umbrella explicitly delegated call-site verification, and the failure-column behavior is preserved.

§3 handle_jobdispatch classification (umbrella left it open)

• Findings: dispatch_a_job_ (job.c:180-304) sends xce_startjob_req to the worker and updates j->job.allocs in memory; it fires no hooks and appends no WAL entries. Per-job outcomes return to the scheduler via notify_dispatch (job.c:440-457, called at 479-486). The job's externally meaningful state change (RUN) arrives later as a worker fact (WAL_JOB_START).
• Implications: Per the umbrella's own criterion ("if its outcomes are fact-shaped … the early reply is an ACK-of-receipt and may stand"), the verdict is receipt-ACK stands (this spec's Req 2.3). The orphan-dispatch-of-uncommitted-submit race is closed by the egress watermark filter (ha-convergence-recovery), not here. The classification guard plus a no-WAL-from-dispatch test pin the verdict.

§4 Fact loss is gate-level too (bug-396 family)

• Findings: ha_dispatch_gate's CTRL_PRIMARY branch rejects OPCode_notify, OPCode_update, OPCode_setpreason, OPCode_clrpreason with -XCE_ERR_NOT_READY whenever write quorum is absent (ha.c:1537-1556 default arm). The worker never reads notify replies (umbrella research §A), so each rejection is a permanently lost fact.
• Implications: Umbrella 2.8 ("retain … including across quorum-loss periods") requires admitting these opcodes without write quorum; durability is then owned by RETAIN. Allowlist change is confined to the PRIMARY branch; the COMMITTING branch is left for ha-single-leader-role's wholesale deletion.

§5 Config track hardening scope

• Findings: configAck counting (repl.c:418-438) shares all three walAck defects; repl_on_config_ack already stores the standby's acknowledged tag (repl.c:351-371), so per-push comparison (>= push_tag) needs only a parameter, not new state. committed_config_tag advance is already success-gated (config_repl.c:918-922).
• Implications: Hardening lives in the configAck branch + push_config_to_standbys return contract + caller failure path. No engine involvement (umbrella-locked: config is a parallel, simpler track sharing the majority arithmetic rule — one in-flight change at a time).

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Verdict
In-hook WAIT for all request types	extend the job-submit sync-append pattern to queue/worker/job-control hooks	one mechanism, no handler edits	blocks commit waits under entity WRITE locks (readers stall up to timeout); job-control hooks fire per-transition, not per-operation	rejected
Handler-level barrier (chosen)	hooks append (RETAIN-registered); request handlers call persist_commit_barrier() after locks, before reply	matches umbrella handler sequence verbatim; no lock-hold waits; uniform mechanical sweep pattern; covers dual-origin types from request contexts	barrier watermark may over-wait on unrelated concurrent entries (bounded by timeout)	selected
Per-entry LSN plumbing (hooks return entry LSNs to handlers)	exact-entry waits	no over-wait	invasive signature changes through the hook system for negligible benefit (over-wait is bounded and rare)	rejected (YAGNI)

Design Decisions

Decision: Handler-level commit barrier on the appended watermark

• Context: §1 lock-scope finding vs the umbrella WAIT mode for queue/worker/job-control.
• Selected Approach: persist_commit_barrier(p, timeout_ms) = snapshot current_lsn, commit_submit(WAIT) on it; called by each request handler between mutation success and reply.
• Rationale: Single-writer appends + contiguous commit advancement (engine postcondition) make the watermark cover everything the operation appended; the umbrella's Reply-Ordering Contract literally specifies the handler sequence.
• Trade-offs: Possible over-wait on concurrent unrelated uncommitted entries (bounded by sync_timeout_ms; those entries must commit anyway). A no-op control operation may wait behind unrelated traffic — accepted (request-shaped ops are gate-rejected during quorum loss, so the wait resolves quickly in healthy clusters).
• Follow-up: Integration test pins reply-after-commit ordering per opcode.

Decision: Verified classification deviations (STOP/RESUME fact-only; UPDATE dual-origin)

• See Research Log §2. Umbrella conformance preserved through the handler barrier; the verified table in design.md is the implementation contract. Inline umbrella tags kept (2.1/2.8).

Decision: Production fallback for unclassified WAL types = RETAIN

• Context: Req 2.2 requires fail-fast in testing; production must not crash the primary on a programmer omission.
• Selected Approach: XCE_TESTING assert; production ERROR log + RETAIN treatment.
• Rationale: Fail-safe direction is retention (never dropped, never falsely confirmed) — a misrouted request-shaped type degrades to a missing barrier caught by the sweep audit/tests, not to data loss.

Decision: Failed config upgrade does not restart

• Context: Roadmap residual ("don't restart the primary before the commit barrier passes"); umbrella 5.2.
• Selected Approach: do_upgrade_config_with_bytes gates the success ACK and restart_requested on the barrier result; disk save/generation bump remain pre-barrier (existing sequence), reconciled by the existing OP_replConfig anti-entropy (out of boundary).
• Trade-offs: A failed upgrade leaves the new config on the primary's disk but uncommitted cluster-wide; explicit failure to the operator makes the state visible. Full save-after-commit resequencing deferred (would touch checkpoint/restart machinery beyond this slice).

Risks & Mitigations

• Missed reply site silently violates umbrella 2.1 — exhaustive grep audit task over all xce_reply sites + behavioral integration test (no success reply before commit)
• Upstream ha-commit-engine not yet implemented — tasks consume only its published interface; if signatures drift, revalidation trigger fires (boundary section)
• New timeout errors surfacing to scheduler/worker/CLI callers — deliberate CP semantics (umbrella); error codes flow through existing reply envelopes; callers' retry behavior unchanged
• Guard rejections changing mark_job_* return codes — sweep checks all callers treat duplicate-as-success-no-op

References

• .kiro/specs/ha-consistency-invariants/{requirements,design,research}.md — authoritative umbrella contract (esp. design §Write-Path Classification, §Config Barrier, §Idempotency Guards; research locked decisions 1, 2, 4)
• .kiro/specs/ha-commit-engine/design.md — consumed engine interface (commit.h verbatim), migration note "retired by ha-commit-barrier-rollout"
• .kiro/steering/roadmap.md — dependency ordering; superseded item "Quorum-ACK gate before config restart" residual
• .kiro/steering/ha-async-io.md, testing.md, error-handling.md — threading, TDD, errnox conventions
• Raft paper §5.3 (commit rule) — via umbrella; this spec adds no new consensus logic

---

Gap Re-Audit (2026-06-18)

Mode: POST-IMPLEMENTATION GAP AUDIT. All tasks 1.1–7.2 are [x] (phase=implemented). This pass re-verifies every EARS acceptance criterion against the current on-disk controller source (codegraph + grep, persistent_ha working tree) rather than against the design intent, and records residual/deferred items the implementation explicitly left open. Verdict up front: CLOSED — 19/19 acceptance criteria MET in code. The only open items are pre-declared, out-of-boundary residuals (bug-448 ref leak, bug-451 RUSAGE OOB, the CTRL_PRIMARY_COMMITTING fact-rejection deferred to ha-single-leader-role, and the WORKER_UPDATE-via-heartbeat WAIT-classification refinement). None block this spec's contract.

Requirement-to-Asset Map (verified file:line)

Req	Criterion (short)	Asset (verified)	Verdict
1.1	success reply only after majority commit	persist_commit_barrier (persist.c:2069) called post-job_ctrl in do_jobcontrol_op (job.c:1023-1027), post-queue_ctrl in do_queuecontrol_op (queue.c:144), post-worker_ctrl in do_hostcontrol_op (worker.c:712-716), post-AFTER_EVENT in handle_worker_join (join.c:130) — reply (xce_reply) strictly after the barrier in every case	MET
1.2	commit timeout → error reply	barrier returns -ETIMEDOUT (persist.c:2105 → commit_submit(...,COMMIT_MODE_WAIT,timeout)), reply.status = rc in all four handlers (job.c:1029, queue.c, worker.c:718); timeout_ms==0 borrows p->config.sync_timeout_ms (persist.c:2104-2107)	MET
1.3	not-ready rejection → error reply	engine -XCE_ERR_NOT_READY passes through barrier unmodified (persist.c:2105 returns engine code); handler maps to reply.status	MET
1.4	no success-interpretable reply pre-commit	handlers send exactly one xce_reply after the barrier; integration test test_commit_barrier_rollout.c pins "no success reply while entries exceed committed_lsn" (task 3.1/7.2)	MET
2.1	every WAL type classified WAIT/RETAIN	wal_classify (persist.c:2004-2054) total switch over all 13 wal_entry_type values incl. WAL_NOOP row (persist.c:2037, pre-declared); consulted in all 3 hook fns (persist_job_hook_fun/persist_queue_hook_fun/persist_worker_hook_fun, codegraph callers)	MET
2.2	unclassified type fails fast / fact-fallback	default arm: ERROR(...) log + assert(0) under XCE_TESTING + return WAL_MODE_RETAIN production fallback (persist.c:2045-2053)	MET
2.3	dispatch ACK is receipt-only	handle_jobdispatch/dispatch_a_job_ produce no WAL (research §3); do_set_preason/do_clr_preason reply-first then RETAIN-append (job.c:1150/1178) — receipt verdict pinned by task 3.3 tests	MET
2.4	exhaustive reply-site verification	task 7.1 audit (tasks.md Impl Note 7.1): ZERO request-shaped stragglers; 8 category-A barrier-gated, 6 category-B receipt-ACK, ~29 category-C sites enumerated and committed	MET
3.1	facts appended, retried, no timeout	RETAIN types → persist_wal_async + commit_submit(...,COMMIT_MODE_RETAIN,0) (returns immediately, no timeout); hook callers get local-append rc only	MET
3.2	facts accepted during quorum loss	ha_dispatch_gate CTRL_PRIMARY no-quorum allowlist adds OPCode_notify/update/setpreason/clrpreason (ha.c:2099-2102) before the -XCE_ERR_NOT_READY default (ha.c:2105)	MET
3.3	uncommitted facts never discarded	log-is-the-queue: pending purely by lsn > committed_lsn; no drop path in the RETAIN seam	MET
3.4	facts commit after quorum restore, no redelivery	committed_lsn advances over retained suffix via commit_on_ack; task 4.3 integration test asserts no re-delivery, no timeout applied	MET
4.1	config confirmed only after majority	do_upgrade_config_with_bytes (config_repl.c:1115) calls push_config_to_standbys (config_repl.c:941) which returns barrier rc; success ACK (config_repl.c:1133-1142) only when rc == 0	MET
4.2	config timeout → failed to requester	rc != 0 → send_upgrade_error(ep,req,c,rc,...) then return rc (config_repl.c:1118-1126), forwarding -ETIMEDOUT	MET
4.3	unacked config never committed	committed_config_tag advance is inside push_config_to_standbys's success branch only (config_repl.c:963-969, atomic_store gated on majority)	MET
4.4	failed upgrade does not restart	failure path returns before restart_requested store + cbuf_put(&c->wbuf) (config_repl.c:1159-1161); restart unreachable on failure	MET
5.1	duplicate transition is logged no-op	mark_job_finish duplicate-on-terminal → INFO(...) no-op (job.c:702); mark_job_update no-op update → INFO skip (job.c:810) — no AFTER hook, no WAL	MET
5.2	terminal→non-terminal rejected with WARN	shared job_state_is_backward predicate (job.c:39); mark_job_start rejects backward START (job.c:632-633 WARN); mark_job_update rejects backward (job.c:803-804 WARN)	MET
5.3	replay-once equivalence	guards make RETAIN-retry / restore_job_state replay converge to single-application state; double-apply test (task 6.1) green	MET

Score: 19/19 MET.

Supporting-mechanism confirmations

• Legacy walAck retirement (design goal, Req 1.x): repl_wait_majority_ack (repl.c:422) is now configAck-only; walAck delegates unconditionally to commit_submit (repl.c:657-659 comment + repl_set_commit_engine repl.c:660). configAck branch hardened: acked_config_lsn >= push_tag (repl.c:500-501), majority 1 + covered > n_peers/2 over n_peers (repl.c:511), n_peers <= 1 immediate (repl.c:470-472), zero-cover multi-member → timeout (repl.c:529). MET / no residual.
• Config-ACK production ingestion (remediation r1 / former CRITICAL F1): repl_on_config_ack now has live producers — repl_ack_decode_and_dispatch decodes replconfigpushreplymsg on the promoted-vote ep (ha.c:3403-3549) and a defensive cluster.c:630-640 OPCode_reply ingestion. Closes the "bricked multi-member config upgrade" hole. MET.
• Barrier-off-iomgr (remediation r1 / former CRITICAL F2): push_config_transport_only (config_repl.c:879, non-blocking) serves all best-effort ha.c/join.c streaming sites; push_config_to_standbys (transport+barrier) has exactly one caller in vtask context (do_upgrade_config_with_bytes). MET.

Residual / Deferred Gaps (all pre-declared out-of-boundary — none block CLOSED)

Item	Tag	Where	Status & Rationale
bug-448: job_ctrl reference leak	Constraint	reference(j) at job.c:348 on the not-started branch; normal exit never dereferences	NOT FIXED. Pre-existing refcount leak surfaced by task 3.2 adversarial review; explicitly out of this spec's boundary (no behavior in the commit contract). Future fix.
bug-451: RUSAGE va_list OOB read	Constraint	update_job_rusages fires HOOK_DETAIL_JOB_RUSAGE_UPDATE with xce_job_rusages_t* (job.c:1046/1093-1094); persist_job_hook_fun default arm reads vararg 2 as job_t* → 4-byte OOB read at offset 56, garbage state in every WAL_JOB_RUSAGE payload	NOT FIXED. Reviewer-verified byte-precise; declared a boundary item at task 3.3. Production fix candidate = RUSAGE-specific va_list signature in the persist arm. Does not affect any acceptance criterion (jobid lines up; RUSAGE is a fact, retained either way).
CTRL_PRIMARY_COMMITTING still rejects facts	Missing (deferred)	ha_dispatch_gate COMMITTING branch intentionally untouched (design Non-Goals; ha.c)	DEFERRED to ha-single-leader-role, which deletes the transitional state wholesale. During the brief COMMITTING window facts can still be rejected — known residual, hard-ordered downstream.
WORKER_UPDATE WAIT-classification too broad	Constraint	mark_worker_back (worker.c:589) via heartbeat sweep check_worker_timeout_ (cluster.c:1288) produces a WAIT-classified WAL_WORKER_CTRL/UPDATE with NO xce_reply	NOT a 7.1 violation (no reply ⇒ cannot confirm-before-commit; un-barriered WAIT behaves as RETAIN — commits via ACKs). Future-refinement candidate: narrow WORKER_UPDATE WAIT to the request-bearing detail. Carry to ha-single-leader-role / hardening pass.
Half-pushed config on disk after failed upgrade	Constraint	do_upgrade_config_with_bytes saves to disk + bumps generation before the barrier; failure leaves the new config on the primary's disk but cluster-uncommitted	Accepted (design Decision "Failed config upgrade does not restart"). Reconciled by existing OP_replConfig anti-entropy (out of boundary). Operator sees explicit error; tag stays uncommitted.

Integration Risks (forward-looking, not gaps in this spec)

• ha-single-leader-role hard ordering: must land AFTER this spec (it removes CTRL_PRIMARY_COMMITTING and adds WAL_NOOP). The classification guard (persist.c:2049) already forces WAL_NOOP to be classified — pre-declared row at persist.c:2037 (RETAIN) means the totality sweep will not trip when the type is wired. Low risk.
• worker-rejoin-resync: shares the mark_job_* idempotency guards (job.c:39/632/803). Revalidation trigger: re-verify the backward predicate against resync replay ordering when that spec lands.
• New error codes to clients (-ETIMEDOUT/-XCE_ERR_NOT_READY on control ops): deliberate CP behavior; flows through existing reply envelopes; caller retry semantics unchanged. Confirmed by design + reply.status wiring.

Open-Work Options (only for the deferrable refinements — the spec itself needs none)

bug-451 (RUSAGE OOB) is the only residual with a correctness flavor inside this spec's files, so it carries options:

• Option A — Fix now (RUSAGE-specific va_list arm in persist_job_hook_fun): read vararg 2 as xce_job_rusages_t* for HOOK_DETAIL_JOB_RUSAGE_UPDATE. Effort: S (≈1 file, 1 arm, 1 ASAN test). Risk: Low — isolated to the WAL_JOB_RUSAGE payload encode; no contract change.
• Option B — Defer to a dedicated hardening pass / worker-rejoin-resync: keep the boundary call as recorded. Effort: 0 now. Risk: Low-Medium — OOB read persists in production (4-byte over-read, no crash observed; garbage state field in a fact payload that is retained regardless).
• Option C — Fold into ha-single-leader-role sweep (which already touches WORKER_UPDATE classification). Effort: S, batched. Risk: Low, but couples an unrelated correctness fix to a larger spec's timeline.

Recommendation: Option A for bug-451 (cheap, isolated ASAN-clean fix); leave bug-448 and the COMMITTING/WORKER_UPDATE items as recorded boundary residuals. The spec's own 19/19 contract is CLOSED — no spec-internal open work.