feat(worker-rejoin-resync): task 4.3 - wire query receipt and post-join gate

Add OPCode_jobStateQuery dispatch to myreq_handler with multi-page query
reassembly, routing to task 4.2's xcecd_resync_answer_query() (Req
8.1, 8.3). Gate the worker's resumption of normal post-join operation
on a resync_pending flag until the full report has sent (Req 3.8).

Fixes bug-1749 (xcecd_resync.c was never added to the worker's
production source lists - task 4.2's build only ever exercised it via
the test binary's white-box include).

Round 1 review caught bug-1750: the gate sat as checking_connection()'s
entry guard, so a connection drop mid multi-page query left
resync_pending stuck forever, permanently disabling the worker's only
reconnect mechanism (Req 6.1 violation). Round 2 narrows the gate to
guard only the rejoin action, and resets it from xcecd_on_reconnect()
so a stalled query can never wedge it past one reconnect cycle.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
This commit is contained in:
Rongsong Shen 2026-09-01 18:01:03 +00:00
parent cfe58c3dcf
commit 5b69973677
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
8 changed files with 1161 additions and 34 deletions

View file

@ -20433,6 +20433,34 @@
],
"occurrences": 1,
"last_seen": "2026-09-01T02:00:00Z"
},
{
"id": "bug-1750",
"timestamp": "2026-09-01T09:00:00Z",
"error_message": "worker-rejoin-resync task 4.3, round-1 review: the resync_pending gate was checked as checking_connection()'s very first statement (an entry guard), returning early before xcecli_hello() or the needs_rejoin-triggered rejoin_call() could run. xcecd_resync_handle_query_page() sets resync_pending = true on receiving a job-state query's first page and only clears it when called again with a later page. If the worker's connection to the controller drops after a non-final page of a multi-page OPCode_jobStateQuery arrives (a query spanning more non-terminal jobs than fit in one page -- explicitly in scope per Requirement 8.1/8.3), no further page ever arrives, so xcecd_resync_handle_query_page() is never invoked again and resync_pending is stuck true forever -- permanently disabling checking_connection()'s entire body, including the worker's ONLY connection-recovery mechanism (xcecli_hello() / needs_rejoin-triggered rejoin). Directly violated Requirement 6.1 (the worker must be able to disconnect-and-rejoin during an incomplete reconciliation round) and regressed the intent of the pre-existing invariant test test_checking_connection_never_permanently_disables_worker, though none of round 1's own new tests drove the multi-page-then-disconnect sequence so it passed the full suite undetected.",
"file": "src/worker/src/controller_connector.c",
"root_cause": "design.md directed mirroring the existing needs_rejoin boolean pattern for the new resync_pending gate, but needs_rejoin is self-healing by construction -- it gates one step INSIDE checking_connection() (the rejoin action itself), so the function's body including xcecli_hello() still runs every tick regardless of its value, giving the worker a fresh chance to detect and recover every cycle. Round 1 placed resync_pending's check as the function's entry guard instead, which is not actually the same pattern in the load-bearing respect that matters: it makes the gate's only exit condition (a later query page arriving) depend on the very connection health the gate is supposed to be resilient to, with no independent path back to a working state.",
"fix": "FIXED (2026-09-01, round-2 review, task 4.3). Two changes, defense-in-depth: (a) narrowed the gate so xcecd_resync_is_pending() is checked only inside the needs_rejoin branch, guarding solely rejoin_call() -- xcecli_hello() and the needs_rejoin check itself now run unconditionally on every checking_connection() tick, restoring the self-healing property. (b) added xcecd_resync_reset(), called from xcecd_on_reconnect() (the same signal that sets needs_rejoin, confirmed to fire only after xce_failover_rotate()/xcecli_invoke_rejoin() has already repaired a real, previously-detected transport failure -- never spuriously on a healthy connection), which clears resync_pending and discards any stalled partial multi-page accumulation (frees and zeroes the accumulator, not just the boolean), bounding the worst case to at most one reconnect cycle instead of forever. Verified: round-2 reviewer independently reintroduced the round-1 entry-guard bug, rebuilt, and confirmed a new structural regression test (test_checking_connection_hello_and_needs_rejoin_precede_resync_gate_in_source, asserting strict source-order) fails while the companion behavioral test (test_checking_connection_survives_permanently_stuck_resync_pending) does NOT catch the reintroduced bug alone -- establishing that placement-sensitive gates need a structural/ordering assertion, not just a behavioral one. Full worker suite green after the fix (xcecd_resync_test 28/28, controller_connector_test 22/22 including the pre-existing test_checking_connection_never_permanently_disables_worker invariant). One advisory, non-blocking follow-up noted but not fixed: resync_pending/query_accum are unsynchronized statics touched from two different OS threads (connector thread vs. a vtaskmgr pool thread) -- a narrow theoretical race on xcecd_resync_reset()'s free() vs. a concurrent read, mirroring needs_rejoin's own accepted unsynchronized-static risk profile but riskier since it frees a heap pointer.",
"tags": [
"worker-rejoin-resync",
"task-4.3",
"xcecd_resync-c",
"controller_connector-c",
"requirement-6.1",
"requirement-3.8",
"deadlock",
"gate-placement",
"checking_connection",
"needs_rejoin",
"review-caught",
"fixed"
],
"related_bugs": [
"bug-1749",
"bug-1748"
],
"occurrences": 1,
"last_seen": "2026-09-01T09:00:00Z"
}
]
}