myxce/.kiro/specs/worker-rejoin-resync/research.md

Research & Design Decisions: worker-rejoin-resync

Summary

• Feature: worker-rejoin-resync
• Discovery Scope: Extension (existing system) — integration-focused, cross-cutting across controller, worker, and scheduler around an already-implemented join/rejoin path and an already-implemented commit-then-apply seam.
• Key Findings:
  • rejoin_worker/apply_join_state (worker.c, from worker-join-refactor) is the exact moment a host becomes "connected" again — the natural trigger point for Requirement 1, and a new worker_t status bit here gates new dispatch "for free" through the existing dispatch_a_job_ connected-state check with zero changes to that function.
  • The job table's idempotent commit-then-apply split (record_job_*_fact / apply_job_*_transition, with job_state_is_terminal/job_state_is_backward guards) is the exact dedup mechanism the requirements' Boundary Context says this feature must reuse, not duplicate. Three of four reconciliation outcomes (finished-success, finished-failure, still-running) already have a fully-idempotent home; only "unrecognized → return to pending" and "unqueried job → assign" are genuinely new fact shapes.
  • The controller→scheduler notification path (Requirement 7) is a hook → notify → wire → dequeue pipeline (XCE_CONTROLLER_HOOK_JOB → sched.c hook callbacks → notify() → scheduler's handler.c inbound switch → sched_job_state_change()), and it is strictly separate from the scheduler's own outbound decision-publish path (scheduler.c/driver.c), which a standing text-scan test (test_no_worker_resync_symbols_in_scheduler_publish_path) guards against ever learning about "resync" concepts. This fixes the correct integration seam: handler.c's inbound switch only.
  • The worker has no existing retention of finished-job state — cleanup_worker_list() reaps and deletes an exited job's record (memory + sqlite row) on the very next cleanup pass, with no finish_time field anywhere in the worker's job structs today. New short-lived retention is required, not a rewiring of an existing cache.
  • xce_reply_get()/xce_push_data_req() ({pos, count, n_total} chunking, used today for joblist/workerlist/reslist replies) is the established bounded/paginated multi-message idiom in this codebase and is the direct template for Requirement 8, not the heavier WAL catch-up/replication protocol and not XCECLI_FF_CALL (whose whole point is "loss is fine" — the opposite of what this feature needs).
  • config-design.md's rule set classifies the new "recently finished" window (Requirement 3.7) the same way heartbeatInterval/inactiveHeartbeatCount were classified: passes the independence rule (no cross-node correctness dependency) but is kept in shared config anyway via the documented operational-uniformity override, for the same "predictable cluster-wide behavior" reason.

Research Log

Controller-side join/rejoin flow and job state machine

• Context: Where does reconciliation trigger, and how does it reuse the existing idempotent fact/apply path?
• Sources Consulted: src/controller/worker.c (rejoin_worker:403, apply_join_state:321, on_worker_join:623), src/controller/join.c (handle_worker_join:48), src/controller/job.c (record_job_finish_fact:2626, record_job_fail_fact:2781, record_job_postdone_fact:2921, apply_job_update_transition:2808, dispatch_a_job_:1278), src/controller/persist.c (apply_committed_entry:4333), src/controller/commit.c (commit_submit:485), src/include/xce/types.h (worker_t:284, status bits 170-184), .kiro/specs/worker-join-refactor/design.md.
• Findings:
  • handle_worker_join durably commits the join (persist_commit_barrier) before on_worker_join runs — a second, later plug-in point after the join is durable.
  • worker_t.host.status is a bitmask; dispatch_a_job_ (job.c:1511-1512) already refuses dispatch when w->ep == NULL || w->host.status != 0. A new XCE_WORKER_RECONCILING-style bit therefore gates Requirement 5.1 with no change to dispatch_a_job_ itself.
  • record_job_finish_fact/apply_job_finish_transition already sets finish_time at receipt time (bug-1689), matching Requirement 3.5/4.2 exactly — the finished-job reconciliation case is a straight reuse.
  • JOB_STAT_PEND is otherwise only ever set at job creation — there is no existing "return a RUN job to PEND" transition. Since RUN→PEND is non-terminal→non-terminal, the existing job_state_is_backward guard does not block it; a thin new fact wrapper routed through the same WAL_JOB_UPDATE/apply_job_update_transition seam is the correct reuse (not a new state-machine core).
  • No existing "assign a job to a worker the controller didn't previously have a record for" fact exists; the closest analog is dispatch_a_job_'s own assignment logic (job.c:1686-1734).
  • The pending-facts queue is not a separate structure — it is the commit engine (commit_submit, COMMIT_MODE_RETAIN) plus apply_committed_entry's guards. Reconciliation must submit through this exact seam, never a parallel one (this is also stated explicitly in the requirements' own Boundary Context).
• Implications: Reconciliation needs exactly two new fact-recording functions in job.c (reconcile-to-pending, reconcile-assign) plus straight reuse of the three existing ones (finish, fail, no-op-for-running).

Worker-side job bookkeeping and wire protocol

• Context: What can the worker actually report, and how should the query/report exchange ride the wire?
• Sources Consulted: src/worker/include/xcecd_worker_utils.h, src/worker/src/xcecd_worker_utils.c (cleanup_worker_list:385-404), src/worker/include/xcecd_utils.h, src/lib/request.c (xce_join_req:1398-1429, xce_reply_get:1627-1784, xce_push_data_req:1491-1610), src/schema/controller.capnp (JoinData:155-161, HostAttrs:126-135), src/lib/client_internal.h (XCECLI_FF_CALL:290-302), src/lib/client.c.
• Findings:
  • xcecd_worker_t (PID-keyed) and xcecd_job_t carry no timestamp field at all; cleanup_worker_list() deletes both the in-memory entry and its sqlite row essentially immediately after exit — there is no retention window today.
  • JoinData's capnp union has exactly two arms (nodata, attrs) with no job-state content, confirming the brief's own claim. Cramming a paginated exchange inline into JoinData would force it through xce_join()'s single-shot synchronous cbuf_timedget call shape, which cannot paginate — a new follow-up opcode/message pair after the join reply is the right shape, not a JoinData union extension.
  • xce_reply_get/xce_push_data_req's {pos, count, n_total} chunking (already used for joblist/workerlist/reslist) is the established, directly-reusable bounded-exchange template for Requirement 8.
  • XCECLI_FF_CALL's own header comment explicitly names "worker-rejoin-resync" as the intended fix for the loss it tolerates — confirming this feature's problem statement directly in code, not just in the bug report.
• Implications: New capnp opcode pair (controller→worker query, worker→controller report), both using the existing pagination shape; new worker-side finished-job retention structure with a timestamp field, bounded by the Requirement 3.7 window.

Scheduler notification path and scope-fence test

• Context: Requirement 7 requires reconciliation-driven job-state changes to reach the scheduler through "the same path" as an ordinary change — what is that path, concretely, and are there any standing tests that constrain where new code may live?
• Sources Consulted: src/controller/job.c (hook call sites HOOK_DETAIL_JOB_EXIT:2641, HOOK_DETAIL_JOB_FAIL:2745), src/controller/sched.c (sched_job_hook_fun:547, sched_job_finish_fun:392, sched_job_fail_fun:438, sched_job_update_fun:468), src/controller/dataex.c (notify:399), src/controller/job.c (notify_dispatch:1971), src/scheduler/src/handler.c (sched_handle_notify:525, sched_job_state_change:407), src/scheduler/testsuite/test_scheduler_decision_redelivery.c:424-456.
• Findings:
  • Job-state-change code fires XCE_CONTROLLER_HOOK_JOB (not HOOK_WORKER — that category is for worker join/dead events); sched.c's registered hook callback dispatches by HOOK_DETAIL_* to one of three functions, each of which calls the shared notify() with a distinct NotifyData_data_sched* event.
  • notify_dispatch()/NotifyData_data_jobDispatchFail is an existing, direct precedent for "tell the scheduler this job is back to pending" — its handler in handler.c already calls sched_job_state_change(sched, jobid, JOB_STAT_PEND).
  • test_no_worker_resync_symbols_in_scheduler_publish_path is a static text-scan (strstr for "resync") over scheduler.c and driver.c — the scheduler's own outbound decision-publish direction. It is a deliberate scope-fence: that direction must never learn about worker-resync concepts. It does not cover handler.c (the inbound Controller→Scheduler direction), which is unambiguously the correct, and only permitted, integration seam.
  • The async-I/O steering doc's thread-role contract applies identically to the scheduler; a bounded reconciliation wait must follow the vtask + admission-reservation pattern (commit.c's COMMIT_WAIT_RESERVED, wait_cap/wait_admit_locked) rather than any raw blocking wait on iomgr/delaymgr, with the timeout itself fired by a delaymgr trigger.
• Implications: Reconciliation's finish/fail cases need zero new scheduler-facing code (they already drive schedExitJob/schedFailJob). The reconcile-to-pending case should reuse the existing jobDispatchFail-style sched_job_state_change(..., JOB_STAT_PEND) call shape (either by literally reusing that notify event or adding a same-shaped one) — never by adding anything to scheduler.c/driver.c.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Inline extension of JoinData union	Add job-state content as a third JoinData union arm	Minimal new opcode surface	Forces the exchange through xce_join()'s single-shot synchronous reply — cannot paginate (Requirement 8)	Rejected
New opcode pair, ad hoc payload shape	New OPCode_jobStateQuery/OPCode_jobStateReport, custom framing	Full control over shape	Reinvents pagination the codebase already solved	Rejected
New opcode pair reusing {pos,count,n_total} pagination	New opcode pair, payload shape mirrors JobData/WorkerData/ResourceVals	Directly reuses an established, tested idiom (xce_reply_get/xce_push_data_req); minimal new protocol concepts	None significant	Selected
Reuse WAL catch-up/replication streaming protocol	Model the exchange on replEntry/walCatchupEntry	Already resumable/ack-per-entry	Heavier machinery designed for log replication, wrong semantic fit for a bounded one-time query/report	Rejected

Design Decisions

Decision: New opcode pair for the job-state query/report exchange, following the xce_reply_get/xce_push_data_req pagination template

• Context: Requirements 2, 3, and 8 need a controller-initiated, bounded/paginated, bidirectional exchange that today's join message cannot carry.
• Alternatives Considered: inline JoinData extension; ad hoc new framing; WAL-catch-up-style streaming (see table above).
• Selected Approach: Two new opcodes — OPCode_jobStateQuery (controller→worker) and OPCode_jobStateReport (worker→controller) — both paginated with the existing {pos, count, n_total} shape, issued as a follow-up exchange after the join reply (not inline in JoinData).
• Rationale: Reuses a proven, already-tested idiom instead of inventing a new one; keeps the join message itself unchanged (no regression risk to worker-join-refactor's existing behavior, per its own Boundary Commitments).
• Trade-offs: One more opcode pair to add to ha_dispatch_gate's role tables and myreq_handler's switch, versus a cleaner, template-following protocol.
• Follow-up: Confirm at implementation time whether OPCode_jobStateReport needs an explicit redirect-reply path if ever received by a non-primary (per api-standards.md's "Adding a New Opcode" step 6 / bug-1700 precedent) — expected answer is no, since workers only ever connect to the current primary, but this must be verified, not assumed.

Decision: Reconciliation-outstanding state lives on worker_t as a status bit + a small tracking record, not a new global registry

• Context: Requirement 5 needs to gate new dispatch per-worker while reconciliation is outstanding, and Requirement 6 needs "discard on disconnect, re-trigger on next rejoin" semantics.
• Alternatives Considered: a separate global map keyed by worker; per-worker state (selected).
• Selected Approach: A new XCE_WORKER_RECONCILING status bit (reusing the existing w->host.status != 0 dispatch check "for free") plus a small struct on worker_t holding the queried job-ID set, a round/generation token (for Requirement 6.3's late-report discard), and the timeout deadline.
• Rationale: worker_t is already the natural per-host session object (confirmed by worker-join-refactor's own design); a new global structure would duplicate lifecycle management the existing object already handles (creation, teardown, locking).
• Trade-offs: None significant — this is the minimal-surface option.
• Follow-up: None.

Decision: Reconcile-to-pending and reconcile-assign are new, narrow fact-recording functions in job.c, not a new state-machine core

• Context: Requirement 4.6 (unrecognized → pending) and Requirement 4.8 (unqueried job → assign) have no existing idempotent fact path.
• Alternatives Considered: a new, separate reconciliation-specific mutation path; thin wrappers over the existing WAL_JOB_UPDATE/dispatch-assignment seams (selected).
• Selected Approach: record_job_reconcile_pending_fact() submits through the existing WAL_JOB_UPDATE/apply_job_update_transition seam with target state PEND (same shape as any other job-state-update fact, so it drives the existing HOOK_DETAIL_JOB_UPDATE → scheduler-notify path for free). record_job_reconcile_assign_fact() mirrors dispatch_a_job_'s own assignment logic for host/state but does not attempt to reconstruct the job's original resource allocation (see Risks & Mitigations).
• Rationale: The requirements' Boundary Context and Constraints explicitly forbid a second, competing idempotency mechanism; both new functions are additive call sites into the existing guarded seam, not new guards.
• Trade-offs: The reconcile-assign case inherits a real, pre-existing limitation (below) rather than solving it.
• Follow-up: None — the limitation is out of this spec's boundary by design.

Decision: The "recently finished" reporting window (Requirement 3.7) is shared cluster config, via the operational-uniformity override

• Context: Requirement 3.7 asks for a bounded, cluster-configurable window; config-design.md governs where any such field belongs.
• Alternatives Considered: per-node local config (passes the independence rule cleanly — no cross-node correctness dependency); shared config via override (selected).
• Selected Approach: Add the window (and, for the same reason, the Requirement 5.3 reconciliation timeout) as new fields on the existing ClusterConfig capnp blob, alongside heartbeatInterval/inactiveHeartbeatCount.
• Rationale: config-design.md's own worked example classifies the structurally identical heartbeatInterval/inactiveHeartbeatCount fields (controller→worker liveness, not HA/Raft election heartbeat) as shared-via-override for exactly this reason: no correctness invariant demands cluster-wide agreement, but operators want uniform, predictable behavior rather than per-node drift. This feature's window is the same shape of parameter in the same domain (worker liveness/reconnection).
• Trade-offs: A config push is required to change it (heavier than a per-node flag), accepted for the same reason the precedent field accepted it.
• Follow-up: None.

Decision: Config push and the job-state query are sent in a fixed order (config first) with no new ack — ordering is guaranteed by the single connection, not invented

• Context: on_worker_join (worker.c:623) must trigger both the existing config push and the new job-state query for a rejoining worker. Does the worker need to fully apply config before it can correctly answer the query, and if so, how is that ordering enforced?
• Alternatives Considered: (1) have the controller wait for an explicit worker-sent config-applied ack before sending the query (initially adopted, then retracted); (2) rely on the connection's existing ordering guarantees (selected).
• Selected Approach: on_worker_join sends config push (unchanged worker_push_config call), then immediately sends the new job-state query — both fire-and-forget, no new ack/reply added.
• Rationale: Initially assumed the worker might dispatch incoming controller requests concurrently across its thread_pool_t, which would make arrival order insufficient to guarantee handling order — this assumption was wrong. Reading controller_connector.c:1563's myreq_handler directly confirmed it is a plain synchronous switch, invoked one message at a time off the worker's single long-lived client connection (xcecli_ctx); the worker's thread_pool_t is scoped to job-forking/job-control work only, not request dispatch. On one connection, TCP ordering + this synchronous single-message dispatch together guarantee config push is handled to completion before the job-state-query handler even begins — not just sent first. No config content is actually needed to answer the query correctly either way (job IDs are globally unique, independent of host config), so this was a belt-and-suspenders concern, not a load-bearing correctness dependency — but the ordering guarantee turned out to be free, so it's stated explicitly rather than left implicit.
• Trade-offs: None — this is strictly simpler than the retracted ack-based alternative, with no loss of guarantee.
• Follow-up: If the worker's request-dispatch model is ever changed to fan out across a thread pool (unlike today), this ordering guarantee must be re-derived — it is not inherent to the wire protocol, only to the current single-threaded-dispatch implementation.

Risks & Mitigations

• Risk: A reconciliation-assigned job (Requirement 4.8) has no recoverable original resource allocation (job->allocs) — the controller never held a dispatch record for it, so there is nothing to read the allocation back from. — Mitigation: Document as a deliberate, bounded scope limitation (see design.md's Data Models persistence-completeness note): the job's state and host are corrected; its resource reservation accounting starts from zero/unknown, converging via the existing resource-accounting.md "charged = max(reserved, actual)" policy on the job's next rusage tick — identical to how any legitimately-unreserved job is already handled today, not a new class of gap.
• Risk: A new WAL fact type (reconcile-assign) must be decoded/adopted identically by both apply seams (apply_committed_entry primary-boot-replay and the standby catch-up apply path) or it reproduces the exact bug-1678 field-completeness failure mode. — Mitigation: design.md calls this out explicitly as a persistence-completeness checklist item, per design-principles.md's Persistence Review requirement.
• Risk: A blocking wait for the worker's report, implemented carelessly, could stall a liveness-critical thread (iomgr/delaymgr). — Mitigation: Reuse commit.c's COMMIT_WAIT_RESERVED admission-reservation pattern for any bounded wait; drive the timeout via a delaymgr-scheduled trigger, never a blocking sleep.
• Risk: The requirements' Boundary Context explicitly disclaims coupling the Requirement 3.7 window to commit-batch-window's (separately parked) commit-urgency tuning. — Mitigation: design.md does not reference or depend on commit-batch-window in any component contract.

References

• .kiro/specs/worker-rejoin-resync/requirements.md — approved-for-design requirements (8 requirement groups, 30 acceptance criteria)
• .kiro/specs/worker-rejoin-resync/brief.md — original discovery brief with file:line evidence
• .kiro/specs/worker-join-refactor/design.md — rejoin_worker/apply_join_state contract this feature extends
• .kiro/steering/config-design.md — shared-vs-local config classification rule, applied to Requirement 3.7
• .kiro/steering/resource-accounting.md — reserved/used accounting model the reconcile-assign limitation converges into
• .kiro/steering/persistence.md — APPLY_DIVERGENT/bug-1678 per-field completeness precedent
• .kiro/steering/api-standards.md — "Adding a New Opcode" checklist (bug-1700 redirect-reply precedent)
• .kiro/steering/ha-async-io.md — thread-role contract for the bounded reconciliation wait