merge: sync Rongsong's scheduler-framework (88 commits, f01ae353..c5b172c7)

Two-sided divergence from f01ae353 (2026-08-30): theirs +88 (worker
rejoin-resync reconciliation, terminal-state bug-1737/1738/1739/1764,
scheduler-sim verbosity, hermetic offline build, go mod vendor of Go deps
in apiserver -- 3375 pure-adds), ours +89 (job-finish robustness bug-1509
B1/B3, PEND-reason, worker reconnect resend, apiserver job control).

Conflict resolutions (kept both sides' fixes):
- controller_connector.c checking_connection(): combine theirs
  xcecd_resync_is_pending() rejoin deferral gate with ours
  resend_job_status() actual-state re-report on rejoin success.
- job.c: keep both unknown-finish registry (ours, bug-1509) and
  record_job_finish_fact_at (theirs, bug-1747); keep both
  mark_jobs_unknown_core family (ours) and record_job_reconcile_*
  (theirs). No double-notify: on_job_finish unchanged, theirs is
  resync-path-only.
- scheduler handler.c/test_handlers.c/meson: theirs schedExitJob
  exit_code branch (bug-1737) + ours jobDispatchFail success-ack no-op
  (bug-1512) + configpush enq_pend_update (bug-1513) both present; both
  test suites and REPL/sim targets wired.
- apiserver jobs.go: keep ours normalizeJobInt64/jobArrayIdx superset +
  JobIDHuman; drop theirs strict IntOrInt64 early-reject (superseded --
  ours tolerates float64/json.Number, no panic path).
- jobs_test.go: keep both regression suites (ours bug-1490, theirs
  bug-1742).
- .kiro/specs scheduler-sim-*/worker-rejoin-resync + .wolf/*: keep our
  deletions (established convention -- drop Rongsong bookkeeping on merge).

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-09-03 14:11:43 +00:00
commit 5e2985542f
3469 changed files with 1466599 additions and 382 deletions

View file

@ -25,6 +25,7 @@
#include "jobj.h"
#include "resobj.h"
#include "resync.h" /* worker-rejoin-resync task 6.2: resync_handle_report_page */
#include "utils.h"
#include "worker.h"
#include "xce/errnox.h"
@ -237,6 +238,58 @@ static int handle_snapshot_transfer(cluster_t *c, xce_request_t *req)
return 0;
}
/* handle_jobstatereport: dispatch handler for OPCode_jobStateReport
* (worker-rejoin-resync task 6.2, Req 8.1, 8.3) -- the worker's answer to a
* job-state reconciliation query the controller sent it (worker.c's
* worker_send_resync_query(), task 6.1). Resolves the reporting worker from
* the inbound connection via get_ep_worker() (misc.c) -- the same ep-
* >worker_t resolution on_worker_join() already uses to SEND that query
* (worker.c) -- then delegates wire-page reassembly + apply to resync_
* handle_report_page() (resync.c, task 6.2), which reassembles a report
* that arrives across multiple pages and hands the complete result to
* resync_apply_report() (task 3.2) exactly once, never once per page.
*
* Fire-and-forget from the worker's side (xce_job_state_report_req(),
* request.c -- the same fire-and-forget shape xce_job_state_query_req()
* already uses to SEND the query) -- no reply is expected or sent back,
* matching OPCode_walCatchupEntry/OPCode_snapshotTransfer's existing
* no-reply convention immediately above for this class of one-way,
* already-happened-fact daemon-to-daemon message.
*
* A page from a connection that does not resolve to a known worker
* (get_ep_worker() returns NULL -- e.g. a stale/torn-down connection racing
* teardown) is logged and dropped: there is no worker_t to reassemble or
* apply this report against, and nothing to reply to either way. */
static int handle_jobstatereport(cluster_t* c, endpoint_t* ep,
xce_request_t* req)
{
xce_job_state_report_t* page = req->body.jobstatereportmsg;
worker_t* w;
int rc;
if (page == NULL)
return 0;
w = get_ep_worker(c, ep);
if (w == NULL) {
WARN("handle_jobstatereport: endpoint %p does not resolve to a "
"known worker -- dropping report page (pos=%d count=%d "
"ntotal=%d)", (void*)ep, page->pos, page->count, page->n_total);
return -ENOENT;
}
rc = resync_handle_report_page(c, w, page);
if (rc < 0)
ERROR("handle_jobstatereport: worker %s: failed to reassemble/apply "
"report page (pos=%d count=%d ntotal=%d) rc=%d",
(w->host.name != NULL) ? w->host.name : "?", page->pos,
page->count, page->n_total, rc);
dereference(w);
return (rc < 0) ? rc : 0;
}
/* ---- OPCode_schedAdmin: forward-and-block-for-reply support (task 10.1) --
*
* The controller is a stateless pass-through for OPCode_schedAdmin: it
@ -522,6 +575,29 @@ static int myreq_handler(vtask_t* task)
case OPCode_jobdispatch:
rc = handle_jobdispatch(c, ep, &request);
break;
case OPCode_jobStateReport:
/* worker-rejoin-resync task 6.2 (Req 8.1, 8.3): the controller only
* ever RECEIVES OPCode_jobStateReport (worker -> controller) -- the
* sibling opcode, OPCode_jobStateQuery, flows controller -> worker
* only (sent by worker_send_resync_query(), worker.c) and is never
* inbound to this switch; the worker's own myreq_handler
* (controller_connector.c, task 4.3) is what dispatches THAT one.
* No standby-redirect handling is added here: ha_dispatch_gate()
* above already rejects this opcode on every non-primary role with
* -XCE_ERR_STANDBY (it is deliberately absent from the CTRL_STANDBY
* allow-list, ha.c) BEFORE this switch is ever reached, and the
* generic post-gate rejection block a few lines up already
* populates the redirect hint (bug-1700) for exactly that status --
* so a standby-received report is redirected the same way every
* other primary-only opcode already is, with zero opcode-specific
* code needed here. See ha.c's CTRL_PRIMARY-without-quorum branch
* for the ADJACENT, genuinely-needed fix this task found: admitting
* this opcode as a fact-bearing op there (alongside OPCode_notify)
* so a report is never rejected outright during a transient quorum
* loss (bug-396's exact failure mode, which this whole feature
* exists to correct). */
rc = handle_jobstatereport(c, ep, &request);
break;
case OPCode_jobsub:
rc = handle_jobsubmit(c, ep, &request);
break;