lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(worker-rejoin-resync): task 6.2 - route job-state report to reconciliation

Browse source 
Add OPCode_jobStateReport dispatch in myreq_handler, reassembling a
report across multiple incoming pages before handing it to task 3.2's
resync_apply_report() exactly once - proven via a test-only call
counter, not just end-state equivalence (a double-apply could pass an
end-state check if idempotent).

Standby-redirect investigation (Req 8.1/8.3): confirmed
ha_dispatch_gate already redirects every rejected opcode uniformly, so
no opcode-specific code was needed for that. OPCode_jobStateQuery has
no controller-side receive path at all (only sent, never received)
so needed no gate change either.

That same investigation found a different, more consequential gap:
fixes bug-1753, ha_dispatch_gate's quorum-loss fact-admission allowlist
(OPCode_notify/update/setpreason/clrpreason) was missing
OPCode_jobStateReport, so a report arriving during a transient
quorum-loss window - this feature's own core failover scenario - would
be rejected outright with no retry (worker sends fire-and-forget).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
Rongsong Shen 2026-09-01 21:06:42 +00:00
parent a3d12d37d9
commit 9f655c61c7
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
12 changed files with 1178 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

87
src/controller/include/resync.h
View file

@ -174,6 +174,93 @@ extern int resync_round_is_current(worker_t* w, uint64_t token);
extern int resync_apply_report(cluster_t* c, worker_t* w, uint64_t round_token,
xce_job_state_report_t* report);
/* worker-rejoin-resync task 6.2 (Req 8.1, 8.3): the real type behind
* worker_t.report_accum (xce/types.h -- stored there as an opaque void* to
* avoid a types.h/typemap.h include cycle, since xce_job_state_entry_t is a
* typemap.h type). Reassembles an OPCode_jobStateReport that arrives split
* across multiple wire pages, before resync_apply_report() ever sees it --
* mirrors xcecd_resync.c's worker-side query_accum/query_accum_ntotal/
* query_accum_received statics (task 4.3), moved from file-static (a
* worker daemon has only one controller connection) to a per-worker_t
* field (a controller serves many workers concurrently, so the
* accumulator must be keyed per reporting worker). Lives separately from
* resync_state_t (w->resync) rather than as a field on it: resync_apply_
* report() may resolve (free) w->resync partway through settling a fully-
* assembled report's entries, and this accumulator's own memory must not
* be tied to that lifetime -- resync_handle_report_page() below always
* detaches (NULLs w->report_accum) and takes sole ownership of this struct
* BEFORE calling resync_apply_report(), so there is nothing for that call
* to free out from under it either way. */
typedef struct {
/* Captured from w->resync->round_token the instant this report's first
* page (pos == 0) arrives (0, the reserved "no token" sentinel, if no
* round is outstanding yet) -- NOT re-read at apply time. This is what
* lets a report that started under one round but finishes assembling
* after that round was superseded/discarded (Req 6.1, 6.3) still
* discard correctly via resync_apply_report()'s own resync_round_is_
* current() check, rather than being wrongly matched against whatever
* round happens to be current once the LAST page arrives. */
uint64_t round_token;
int n_total;
int n_received;
/* Deep copies (plain struct-value copies -- xce_job_state_entry_t has
* no nested heap pointers), one malloc'd xce_job_state_entry_t per
* entry, indexed positionally by wire pos, mirroring xce_job_state_
* report_t.entries' own array-of-pointers shape (typemap.h) so the
* assembled report can be hand off to resync_apply_report() directly,
* with no reshaping. A deep copy is required, not merely a borrowed
* pointer into the incoming page: each page's xce_job_state_entry_t*
* values are owned by that page's own xce_request_t, freed by xce_
* free_request() the moment the dispatch handler that decoded it
* returns -- long before a later page (let alone the final one) has
* even arrived. */
xce_job_state_entry_t** entries;
} resync_report_accum_t;
/* Handles one incoming OPCode_jobStateReport wire page for worker `w`:
* accumulates it into w->report_accum (allocating/resetting on pos == 0,
* exactly like xcecd_resync_handle_query_page()'s query_accum reset-on-
* pos-0 convention -- a stale/incomplete accumulation left over from an
* aborted prior report, e.g. this worker disconnected mid-report on a
* prior round, is always discarded by the next report's own first page
* rather than corrupting it). Once every entry has arrived (n_received ==
* n_total), hands the fully-assembled report to resync_apply_report()
* EXACTLY ONCE -- never once per page -- using the round_token captured at
* this report's first page, then frees the accumulator itself.
*
* Precondition: caller does NOT hold w's lock -- this function acquires it
* itself, narrowly, for each page's accumulator mutation, and always
* releases it again BEFORE calling resync_apply_report() (which manages
* its own w locking internally -- resync.h's own doc comment on that
* function). Mirrors resync_apply_report's own "never nested" locking
* discipline.
*
* Returns 0 if this page was accepted but the report is not yet complete
* (more pages still expected), 1 if this page completed the report and it
* was handed to resync_apply_report() (regardless of whether resync_apply_
* report applied it or discarded it as stale -- see above), or a negative
* errno for a malformed page (NULL c/w/page, or a page whose pos/count
* fall outside [0, n_total) -- the in-progress accumulation, if any, is
* discarded in that case too, so a later page cannot be mismatched against
* a corrupt partial buffer). */
extern int resync_handle_report_page(cluster_t* c, worker_t* w,
const xce_job_state_report_t* page);
#ifdef XCE_TESTING
/* worker-rejoin-resync task 6.2 test seam: counts how many times resync_
* handle_report_page() has actually invoked resync_apply_report() (process-
* wide, across every worker) -- a direct, structural proof that a report
* spanning N wire pages results in exactly ONE resync_apply_report() call,
* never N, distinct from (and stronger than) merely asserting the resulting
* job-table end state looks correct (job.c's own idempotent fact guards
* would make even a buggy once-per-page implementation LOOK correct on a
* happy-path report with no interleaved round changes -- see this function's
* own doc comment above and testing.md's "structural, not just behavioral"
* guidance for placement/count-sensitive logic). */
extern int resync_apply_report_call_count_for_test(void);
extern void resync_apply_report_call_count_reset_for_test(void);
#endif
#ifdef XCE_TESTING
/* worker-rejoin-resync task 3.1 test seam: directly invokes the SAME
* internal timeout-fire logic resync_arm_round's own delaymgr callback runs