feat(worker-rejoin-resync): task 6.2 - route job-state report to reconciliation

Add OPCode_jobStateReport dispatch in myreq_handler, reassembling a
report across multiple incoming pages before handing it to task 3.2's
resync_apply_report() exactly once - proven via a test-only call
counter, not just end-state equivalence (a double-apply could pass an
end-state check if idempotent).

Standby-redirect investigation (Req 8.1/8.3): confirmed
ha_dispatch_gate already redirects every rejected opcode uniformly, so
no opcode-specific code was needed for that. OPCode_jobStateQuery has
no controller-side receive path at all (only sent, never received)
so needed no gate change either.

That same investigation found a different, more consequential gap:
fixes bug-1753, ha_dispatch_gate's quorum-loss fact-admission allowlist
(OPCode_notify/update/setpreason/clrpreason) was missing
OPCode_jobStateReport, so a report arriving during a transient
quorum-loss window - this feature's own core failover scenario - would
be rejected outright with no retry (worker sends fire-and-forget).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
Rongsong Shen 2026-09-01 21:06:42 +00:00
parent a3d12d37d9
commit 9f655c61c7
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
12 changed files with 1178 additions and 12 deletions

View file

@ -20522,6 +20522,38 @@
],
"occurrences": 1,
"last_seen": "2026-09-01T10:00:00Z"
},
{
"id": "bug-1753",
"timestamp": "2026-09-01T20:30:00Z",
"error_message": "worker-rejoin-resync task 6.2 (controller-side OPCode_jobStateReport dispatch): design.md's Error Handling section only anticipated the STANDBY-role case for this opcode pair (\"verify... whether OPCode_jobStateQuery/OPCode_jobStateReport need an explicit redirect-reply path... Expected answer: no... but this must be confirmed, not assumed, given bug-1700's documented history of this exact class of gap recurring across two other opcodes\"). Auditing ha_dispatch_gate() (ha.c) in full per that instruction found the STANDBY case was indeed already correctly handled automatically (OPCode_jobStateReport, left absent from the CTRL_STANDBY allow-list, falls to the generic default: return -XCE_ERR_STANDBY branch, and cluster.c's existing bug-1700 post-gate block already fills the redirect hint for any such rejection with zero opcode-specific code needed) -- but the same audit surfaced a DIFFERENT, unanticipated gap in the SAME function: a CTRL_PRIMARY controller that has temporarily lost write quorum rejects OPCode_jobStateReport outright with -XCE_ERR_NOT_READY, because the opcode was absent from ha_dispatch_gate's quorum-loss \"fact-bearing ops\" allowlist (OPCode_notify/OPCode_update/OPCode_setpreason/OPCode_clrpreason, ha.c ~line 4686-4696) that already exists specifically to prevent this exact class of loss (bug-396) for other worker/scheduler fact reports.",
"root_cause": "OPCode_jobStateReport is sent fire-and-forget (xce_job_state_report_req(), request.c/xce_request()) with no redelivery on rejection -- structurally identical to OPCode_notify, the opcode bug-396's fix (ha-commit-barrier-rollout) originally added this allowlist to protect. Since worker-rejoin-resync's two new opcodes were added to controller.capnp's OPCode enum without also being added to ha_dispatch_gate's quorum-loss allowlist, a job-state reconciliation report arriving at a primary during a transient write-quorum-loss window (e.g. mid-failover, the exact scenario this whole feature exists to recover from) would be rejected outright and never retried by the worker -- reproducing bug-396's own loss mode for precisely the facts worker-rejoin-resync (bug-1741) exists to recover.",
"file": "src/controller/ha.c (ha_dispatch_gate, CTRL_PRIMARY quorum-loss branch, ~line 4686-4699)",
"fix": "FIXED (2026-09-01, task 6.2, found live during the task's own mandated standby-redirect investigation -- not assumed safe, confirmed by reading the full gate function as instructed). Added `case OPCode_jobStateReport:` alongside the existing OPCode_notify/OPCode_update/OPCode_setpreason/OPCode_clrpreason cases in ha_dispatch_gate's quorum-loss fact-bearing-ops allowlist, so a report arriving while the primary temporarily lacks write quorum is admitted (not rejected) and can flow into the same RETAIN/pending-facts WAL-append mechanism those sibling opcodes already rely on. OPCode_jobStateQuery needed no corresponding change: the controller only ever SENDS that opcode (controller -> worker), never receives it, so it never reaches ha_dispatch_gate at all. New regression test test_gate_no_quorum_admits_jobstatereport_fact (test_commit_barrier_rollout.c), mirroring test_gate_no_quorum_admits_notify_fact's exact fixture/assertion shape; also added to test_gate_with_quorum_admits_all_opcodes's opcode list for parity. The CTRL_STANDBY case needed and received no code change at all -- proven correct as-is by three new tests in the new src/controller/testsuite/test_jobstatereport_dispatch.c (rejects-when-not-primary, standby-with-known-primary-redirects, standby-no-known-primary-no-redirect), mirroring test_sched_admin_dispatch.c's identical bug-1700 regression tests for OPCode_schedAdmin, the 'real existing primary-only opcode' ground truth this task's brief called for.",
"tags": [
"worker-rejoin-resync",
"task-6.2",
"ha-c",
"ha_dispatch_gate",
"requirement-8.1",
"requirement-8.3",
"bug-396-class",
"quorum-loss",
"design-reality-mismatch",
"found-live-during-implementation",
"fixed"
],
"related_bugs": [
"bug-396",
"bug-1700",
"bug-1746",
"bug-1747",
"bug-1748",
"bug-1751",
"bug-1752"
],
"occurrences": 1,
"last_seen": "2026-09-01T20:30:00Z"
}
]
}