3.6 KiB
| id | status | status_updated | status_note | component | severity | timestamp | last_seen | occurrences | files | tags | related | fix_commits | legacy | gitea_issue | gitea_synced_status | gitea_synced_hash | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bug-1859 | fixed | 2026-09-25 | 7c98c056 |
scheduler | unknown | 2026-09-24T13:45:31Z | 2026-09-24T13:45:31Z | 1 |
|
|
|
true | 278 | fixed | e33402a89415 |
bug-1859: xce-test-ha --chaotic FAIL: chaotic: 1/4 worker-resync job(s) never reconciled after the loop…
Description
xce-test-ha --chaotic FAIL: chaotic: 1/4 worker-resync job(s) never reconciled after the loop settled: 0-2-1790256202-546579(last=PEND). scheduler.log: 'JOBS_PUSH: refusing resync state 0x1 for jobid ... -- existing job is RUN with a real allocation on record (existing state 0x2 kept)'
Environment
Not recorded (migrated from buglog.json).
Reproduction
Not recorded (migrated from buglog.json).
Root cause
Not yet confirmed. Timeline (2026-09-24 chaotic soak, 6 rounds): job dispatched on primary ctrl-1 (term 6) at 13:23:23.74, commit_advance 80->81 at 13:23:23.79; worker started it and finished it at 13:23:43 (retained outcome exit_status=1); ctrl-1 killed in round 5, overlapping kills follow; after settle the cluster reports the job PEND while the scheduler keeps RUN+alloc and refuses the PEND resync (core_events.c:984), so neither side converges. Also: round 6 'scheduler restart did not confirm join within 20s (bug-1760 workaround)'. Candidates: dispatch/RUN not in the surviving committed prefix while the scheduler already applied it, or worker's retained finish never re-delivered to the new primary (cf. bug-1741 fire-and-forget finish notify). Same chaotic soak passed earlier the same day (21 jobs, 6 resync jobs reconciled), so intermittent.
Fix
UPDATE (2026-09-24): Frequency 2026-09-24: 1 failure in 3 chaotic 6-round soaks (pass 12:5x, FAIL 13:2x, pass 13:5x on 9e9f2798). Failing run's logs kept at scratchpad ha-chaos-bug1859-run1 (session-local).
UPDATE (2026-09-24): ROOT CAUSE (from preserved logs of the failing run): job 546579 dispatched+committed (LSN 81) on ctrl-1 term 6; worker finished it 13:23:43 and sent the fire-and-forget finish notify (xcecli_notify = single-shot send, rc ignored: 'FIXME: add error handling when msg send failure') into a dead connection during overlapping failovers; worker could not reach any primary for 84s; its retained finished outcome expired at finish_time + 60s (default window; xce.yaml sets no worker_resync_window_ms, controller pushes 0 -> default) DURING the outage; at rejoin (13:25:07) ctrl-2's resync query (op 43) got 'unrecognized' -> record_job_reconcile_pending_fact requeued it PEND while the scheduler was disconnected; scheduler rejoined 13:25:11 and its JOBS_PUSH bug-1812 guard refused the PEND (RUN+alloc on record) -> permanent divergence. FIX (uncommitted): worker retention window no longer counts outage time -- xcecd_on_reconnect() records the reconnect time and expiry is measured from max(finish_time, last_reconnect); nothing expires while checking_connection()'s hello probe reports unhealthy. Reported finish_time_ms unchanged. Tests: test_finished_job_retention.c test_reconnect_restarts_window_for_outcome_finished_before_outage (84s outage) + test_unhealthy_connection_suspends_expiry, both red first; worker suite 10/10. The scheduler-side divergence (controller requeue while scheduler disconnected is refused by bug-1812 guard) logged separately.
Verification
Not recorded (migrated from buglog.json).
References
- Gitea: #278