44 lines
3.6 KiB
Markdown
44 lines
3.6 KiB
Markdown
---
|
|
id: "bug-1859"
|
|
status: "fixed"
|
|
status_updated: "2026-09-25"
|
|
status_note: "7c98c056"
|
|
component: "scheduler"
|
|
severity: "unknown"
|
|
timestamp: "2026-09-24T13:45:31Z"
|
|
last_seen: "2026-09-24T13:45:31Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/core_events.c", "src/controller/job.c"]
|
|
tags: ["ha", "chaotic-soak", "resync", "scheduler", "controller", "intermittent"]
|
|
related: []
|
|
fix_commits: ["7c98c056"]
|
|
legacy: true
|
|
gitea_issue: 278
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "e33402a89415"
|
|
---
|
|
|
|
# bug-1859: xce-test-ha --chaotic FAIL: chaotic: 1/4 worker-resync job(s) never reconciled after the loop…
|
|
|
|
## Description
|
|
xce-test-ha --chaotic FAIL: chaotic: 1/4 worker-resync job(s) never reconciled after the loop settled: 0-2-1790256202-546579(last=PEND). scheduler.log: 'JOBS_PUSH: refusing resync state 0x1 for jobid ... -- existing job is RUN with a real allocation on record (existing state 0x2 kept)'
|
|
|
|
## Environment
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Reproduction
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## Root cause
|
|
Not yet confirmed. Timeline (2026-09-24 chaotic soak, 6 rounds): job dispatched on primary ctrl-1 (term 6) at 13:23:23.74, commit_advance 80->81 at 13:23:23.79; worker started it and finished it at 13:23:43 (retained outcome exit_status=1); ctrl-1 killed in round 5, overlapping kills follow; after settle the cluster reports the job PEND while the scheduler keeps RUN+alloc and refuses the PEND resync (core_events.c:984), so neither side converges. Also: round 6 'scheduler restart did not confirm join within 20s (bug-1760 workaround)'. Candidates: dispatch/RUN not in the surviving committed prefix while the scheduler already applied it, or worker's retained finish never re-delivered to the new primary (cf. bug-1741 fire-and-forget finish notify). Same chaotic soak passed earlier the same day (21 jobs, 6 resync jobs reconciled), so intermittent.
|
|
|
|
## Fix
|
|
UPDATE (2026-09-24): Frequency 2026-09-24: 1 failure in 3 chaotic 6-round soaks (pass 12:5x, FAIL 13:2x, pass 13:5x on 9e9f2798). Failing run's logs kept at scratchpad ha-chaos-bug1859-run1 (session-local).
|
|
|
|
UPDATE (2026-09-24): ROOT CAUSE (from preserved logs of the failing run): job 546579 dispatched+committed (LSN 81) on ctrl-1 term 6; worker finished it 13:23:43 and sent the fire-and-forget finish notify (xcecli_notify = single-shot send, rc ignored: 'FIXME: add error handling when msg send failure') into a dead connection during overlapping failovers; worker could not reach any primary for 84s; its retained finished outcome expired at finish_time + 60s (default window; xce.yaml sets no worker_resync_window_ms, controller pushes 0 -> default) DURING the outage; at rejoin (13:25:07) ctrl-2's resync query (op 43) got 'unrecognized' -> record_job_reconcile_pending_fact requeued it PEND while the scheduler was disconnected; scheduler rejoined 13:25:11 and its JOBS_PUSH bug-1812 guard refused the PEND (RUN+alloc on record) -> permanent divergence. FIX (uncommitted): worker retention window no longer counts outage time -- xcecd_on_reconnect() records the reconnect time and expiry is measured from max(finish_time, last_reconnect); nothing expires while checking_connection()'s hello probe reports unhealthy. Reported finish_time_ms unchanged. Tests: test_finished_job_retention.c test_reconnect_restarts_window_for_outcome_finished_before_outage (84s outage) + test_unhealthy_connection_suspends_expiry, both red first; worker suite 10/10. The scheduler-side divergence (controller requeue while scheduler disconnected is refused by bug-1812 guard) logged separately.
|
|
|
|
## Verification
|
|
_Not recorded (migrated from buglog.json)._
|
|
|
|
## References
|
|
- Gitea: #278
|