docs: record bug-1512 multi-worker over-dispatch fix + wolf bookkeeping
- left_scheduler_work.md: bug-1494 子问题三(bug-1512)已修复,多 worker 8/8/8 均匀分布验证 - buglog.json: bug-1512 entry(controller code=0 成功 ack 被误判为失败) - cerebrum.md: Do-Not-Repeat — notify 带 outcome code 字段时需按 controller 侧实际值做 gate - memory.md: session log Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
e4aaef4b3b
commit
7b7294dd1e
4 changed files with 30 additions and 1 deletions
|
|
@ -14994,5 +14994,29 @@
|
|||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-11"
|
||||
},
|
||||
{
|
||||
"id": "bug-1512",
|
||||
"timestamp": "2026-08-13",
|
||||
"error_message": "多 worker 无法正常使用:t1/t2/t3 三 worker 下提交 24 个作业跑出 38 个实例,且堆积在第一个 worker(t1),分布 44:8:0 / 13:3:8 等严重不均",
|
||||
"file": "src/scheduler/src/handler.c (sched_handle_notify jobDispatchFail case)",
|
||||
"root_cause": "controller handle_jobdispatch (controller/job.c) 对每个派发的作业都发 NotifyData_data_jobDispatchFail:失败分支 code<0,但成功分支 else 也调 notify_dispatch(jobid, 0)——code=0 是成功 ack。scheduler sched_handle_notify 之前(bug-1510 修复 6928f74)无 code 判断,把成功 ack 也当失败 sched_requeue_pend_job → 每个成功派发的作业立即 re-PEND → 下次 session 重新选举 → 再派发,原实例继续跑 → 重复实例/过派发。是 bug-1510 修复引入的回归。",
|
||||
"fix": "sched_handle_notify 的 jobDispatchFail case 加 code 判断:fail->code >= 0 是成功 ack,当 no-op 消费(作业已在 commit 时标记 RUN);只有 code < 0 才 sched_requeue_pend_job。加白盒单测 test_handle_dispatch_fail_requeues_only_on_negative_code(code=0 保持 RUN、code<0 re-PEND),原 15 测试全过。集群验证(新 scheduler 部署 t1):24 个 sleep-300 作业均匀 8/8/8 分布、零重复、全部 DONE;dispatchFail 日志从 38 降到 0。commit e4aaef4。",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"over-dispatch",
|
||||
"duplicate",
|
||||
"multi-worker",
|
||||
"dispatchfail",
|
||||
"code-0-ack",
|
||||
"regression",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1494",
|
||||
"bug-1510"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-13"
|
||||
}
|
||||
]
|
||||
|
|
@ -609,3 +609,6 @@ As a subagent (spawned via the `Agent` tool, running in the background from the
|
|||
|
||||
## Key Learning (2026-07-31, bug-1487) — a "committed job loss" in `--chaotic` xce-test-ha soaks can be a false positive caused by the Raft own-term commit barrier, not real data loss — always check `verify_classify`'s timing before touching controller code
|
||||
Root-caused a `committed_job_loss=1` chaotic-soak FAIL entirely from logs/timestamps, no code change to `src/controller/`. Mechanism: a newly-elected primary cannot fold a PRIOR-term entry into its own `committed_lsn` watermark just because it holds it durably (Raft Figure-8 safety; `ha_arm_own_term_gate` in `ha.c` — every promotion appends an own-term `WAL_NOOP` and only advances `committed_lsn` past the inherited prefix once THAT noop itself gets a fresh majority ack under the new term). `expect-job` reads `c->jobmgr`, populated only by the apply pump once entries cross `committed_lsn` — so a genuinely majority-committed job can be transiently invisible on the brand-new primary until that fresh ack lands. In steady state this closes in <200ms, but `--chaotic` mode's own "restart the previous round's downed node with no convergence wait" design means the peer whose ack is needed can itself be mid-catch-up, stretching the window to several seconds (measured live: 6.232s). `run-ha-soak.sh`'s chaotic-mode `verify_classify()` ran its `expect-job` check immediately on primary detection with zero settle/retry (unlike `verify_no_loss`, which already had a bug-1422 3-attempt/2s retry, and unlike `wait_for_full_convergence`, which chaotic mode deliberately skips between kills). Fix was a bounded retry in `verify_classify` alone (5×2s), never touching `src/controller/`. **Do-not-repeat**: before treating any xce-test-ha `committed_job_loss` count as a real controller bug, first check (a) whether the SAME job shows up as present in every LATER verify/final-sweep in the same run (a real loss never comes back; this class of false-positive always does), and (b) whether `verify_classify`/whichever check flagged it has an equivalent settle/retry to its sibling `verify_no_loss` — chaotic and sequential modes have historically drifted out of sync on this (bug-1422 fixed sequential only; bug-1487 caught chaotic's matching gap 3 weeks later).
|
||||
|
||||
## Do-Not-Repeat (2026-08-13, bug-1512) — a controller ACK that shares the same event enum as the real failure is not the failure; a re-PEND-on-any-notify fix silently over-dispatches every job
|
||||
bug-1494's re-PEND fix (bug-1510, commit 6928f74) added a `jobDispatchFail` case to `sched_handle_notify` that requeued the job to PEND on ANY such notification. But the controller sends `NotifyData_data_jobDispatchFail` for EVERY dispatched job — the else branch of `handle_jobdispatch` calls `notify_dispatch(jobid, 0)`, so **code 0 is the SUCCESS ack**, and only `code < 0` is a real rejection. The unguarded handler therefore re-PENDed every successfully-dispatched job, the next election re-dispatched it while the original instance kept running, and a 3-worker cluster ran 38 instances for 24 submitted jobs, piling onto the first worker. **Do-not-repeat**: before writing a handler for a controller notify that carries an outcome `code`/`status` field, check the CONTROLLER side to see what values it actually sends on both the success and failure paths (here: the same event, code 0 vs negative) — a "handle the failure" fix that doesn't gate on the code field turns every success into a failure and amplifies, not fixes, the original symptom. Verify live by comparing running-instance count to submitted-job count, not just by checking jobs reach RUN.
|
||||
|
|
|
|||
|
|
@ -7397,3 +7397,4 @@
|
|||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
| 02:1x | Added human-readable jobid (job_id_human/human_id) to apiserver responses (backend.FormatJobID reuse); renamed SSE perm event:subscribe->event:view; updated docs (left_scheduler_work, design doc v1.3, cerebrum) | src/frontend/apiserver, src/frontend/schedulerclient, docs | verified live (submit + get_job return 1-0-... form); all tests pass; pushed | ~90k tok |
|
||||
| 06:21 | bug-1512 多 worker 过派发修复:handler.c code<0 才 re-PEND;24 作业 8/8/8 全 DONE;单测+提交 e4aaef4 | handler.c test_handlers.c | fixed+verified | ~2k |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue