forked from lei.chen/myxce
Rongsong Shen
ca1d9ae42d
Live symptom: jobsub intermittently failed with 'Connection timed out' even though the cluster had quorum. Root cause: the synchronous commit barrier waits only DEFAULT_SYNC_TIMEOUT_MS=50ms for durable majority replication, and that was not configurable. 50ms is too tight for durable sync majority replication — especially since standbys now fdatasync each replicated entry before ACK — so a multi-entry jobsub's later WAL entry intermittently misses the window: commit: WAIT deadline expired without commit (lsn=3 committed_lsn=2 timeout_ms=50) -> persist hook returns -ETIMEDOUT -> handle_jobsubmit rejects -> client ETIMEDOUT. Fix: - Raise DEFAULT_SYNC_TIMEOUT_MS 50 -> 500 (config.h) — realistic for durable sync majority replication, still bounded for failure detection. - Add a --sync-timeout-ms controller flag threaded arguments -> myconf -> persist pcfg (override applied before persist_init); 0 = use default, positive overrides — so operators can tune per environment. TDD: literal-pinned persist_config_defaults().sync_timeout_ms == 500. All timeout-sensitive tests already pass an explicit timeout_ms (none rely on the default), so none are slowed; full controller suite green under ASAN. No commit-engine/barrier/replication/wire change beyond the default value + plumbing. bug-561. |
||
|---|---|---|
| .. | ||
| autoscaler | ||
| controller | ||
| docs | ||
| frontend | ||
| include | ||
| lib | ||
| nixSupport | ||
| scheduler | ||
| schema | ||
| tools | ||
| worker | ||
| build | ||
| flake.lock | ||
| flake.nix | ||
| meson.build | ||
| meson.options | ||