lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(worker-rejoin-resync): task 6.2 - route job-state report to reconciliation

Browse source 
Add OPCode_jobStateReport dispatch in myreq_handler, reassembling a
report across multiple incoming pages before handing it to task 3.2's
resync_apply_report() exactly once - proven via a test-only call
counter, not just end-state equivalence (a double-apply could pass an
end-state check if idempotent).

Standby-redirect investigation (Req 8.1/8.3): confirmed
ha_dispatch_gate already redirects every rejected opcode uniformly, so
no opcode-specific code was needed for that. OPCode_jobStateQuery has
no controller-side receive path at all (only sent, never received)
so needed no gate change either.

That same investigation found a different, more consequential gap:
fixes bug-1753, ha_dispatch_gate's quorum-loss fact-admission allowlist
(OPCode_notify/update/setpreason/clrpreason) was missing
OPCode_jobStateReport, so a report arriving during a transient
quorum-loss window - this feature's own core failover scenario - would
be rejected outright with no retry (worker sends fire-and-forget).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
Rongsong Shen 2026-09-01 21:06:42 +00:00
parent a3d12d37d9
commit 9f655c61c7
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
12 changed files with 1178 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

76
src/controller/cluster.c
View file

@ -25,6 +25,7 @@
#include "jobj.h"
#include "resobj.h"
#include "resync.h" /* worker-rejoin-resync task 6.2: resync_handle_report_page */
#include "utils.h"
#include "worker.h"
#include "xce/errnox.h"
@ -237,6 +238,58 @@ static int handle_snapshot_transfer(cluster_t *c, xce_request_t *req)
return 0;
}
/* handle_jobstatereport: dispatch handler for OPCode_jobStateReport
* (worker-rejoin-resync task 6.2, Req 8.1, 8.3) -- the worker's answer to a
* job-state reconciliation query the controller sent it (worker.c's
* worker_send_resync_query(), task 6.1). Resolves the reporting worker from
* the inbound connection via get_ep_worker() (misc.c) -- the same ep-
* >worker_t resolution on_worker_join() already uses to SEND that query
* (worker.c) -- then delegates wire-page reassembly + apply to resync_
* handle_report_page() (resync.c, task 6.2), which reassembles a report
* that arrives across multiple pages and hands the complete result to
* resync_apply_report() (task 3.2) exactly once, never once per page.
*
* Fire-and-forget from the worker's side (xce_job_state_report_req(),
* request.c -- the same fire-and-forget shape xce_job_state_query_req()
* already uses to SEND the query) -- no reply is expected or sent back,
* matching OPCode_walCatchupEntry/OPCode_snapshotTransfer's existing
* no-reply convention immediately above for this class of one-way,
* already-happened-fact daemon-to-daemon message.
*
* A page from a connection that does not resolve to a known worker
* (get_ep_worker() returns NULL -- e.g. a stale/torn-down connection racing
* teardown) is logged and dropped: there is no worker_t to reassemble or
* apply this report against, and nothing to reply to either way. */
static int handle_jobstatereport(cluster_t* c, endpoint_t* ep,
xce_request_t* req)
{
xce_job_state_report_t* page = req->body.jobstatereportmsg;
worker_t* w;
int rc;
if (page == NULL)
return 0;
w = get_ep_worker(c, ep);
if (w == NULL) {
WARN("handle_jobstatereport: endpoint %p does not resolve to a "
"known worker -- dropping report page (pos=%d count=%d "
"ntotal=%d)", (void*)ep, page->pos, page->count, page->n_total);
return -ENOENT;
}
rc = resync_handle_report_page(c, w, page);
if (rc < 0)
ERROR("handle_jobstatereport: worker %s: failed to reassemble/apply "
"report page (pos=%d count=%d ntotal=%d) rc=%d",
(w->host.name != NULL) ? w->host.name : "?", page->pos,
page->count, page->n_total, rc);
dereference(w);
return (rc < 0) ? rc : 0;
}
/* ---- OPCode_schedAdmin: forward-and-block-for-reply support (task 10.1) --
*
* The controller is a stateless pass-through for OPCode_schedAdmin: it
@ -522,6 +575,29 @@ static int myreq_handler(vtask_t* task)
case OPCode_jobdispatch:
rc = handle_jobdispatch(c, ep, &request);
break;
case OPCode_jobStateReport:
/* worker-rejoin-resync task 6.2 (Req 8.1, 8.3): the controller only
* ever RECEIVES OPCode_jobStateReport (worker -> controller) -- the
* sibling opcode, OPCode_jobStateQuery, flows controller -> worker
* only (sent by worker_send_resync_query(), worker.c) and is never
* inbound to this switch; the worker's own myreq_handler
* (controller_connector.c, task 4.3) is what dispatches THAT one.
* No standby-redirect handling is added here: ha_dispatch_gate()
* above already rejects this opcode on every non-primary role with
* -XCE_ERR_STANDBY (it is deliberately absent from the CTRL_STANDBY
* allow-list, ha.c) BEFORE this switch is ever reached, and the
* generic post-gate rejection block a few lines up already
* populates the redirect hint (bug-1700) for exactly that status --
* so a standby-received report is redirected the same way every
* other primary-only opcode already is, with zero opcode-specific
* code needed here. See ha.c's CTRL_PRIMARY-without-quorum branch
* for the ADJACENT, genuinely-needed fix this task found: admitting
* this opcode as a fact-bearing op there (alongside OPCode_notify)
* so a report is never rejected outright during a transient quorum
* loss (bug-396's exact failure mode, which this whole feature
* exists to correct). */
rc = handle_jobstatereport(c, ep, &request);
break;
case OPCode_jobsub:
rc = handle_jobsubmit(c, ep, &request);
break;