fix(ha-commit-engine): raise commit-barrier default 50->500ms + add --sync-timeout-ms

Live symptom: jobsub intermittently failed with 'Connection timed out' even
though the cluster had quorum. Root cause: the synchronous commit barrier waits
only DEFAULT_SYNC_TIMEOUT_MS=50ms for durable majority replication, and that was
not configurable. 50ms is too tight for durable sync majority replication —
especially since standbys now fdatasync each replicated entry before ACK — so a
multi-entry jobsub's later WAL entry intermittently misses the window:
  commit: WAIT deadline expired without commit (lsn=3 committed_lsn=2 timeout_ms=50)
-> persist hook returns -ETIMEDOUT -> handle_jobsubmit rejects -> client ETIMEDOUT.

Fix:
- Raise DEFAULT_SYNC_TIMEOUT_MS 50 -> 500 (config.h) — realistic for durable sync
  majority replication, still bounded for failure detection.
- Add a --sync-timeout-ms controller flag threaded arguments -> myconf ->
  persist pcfg (override applied before persist_init); 0 = use default, positive
  overrides — so operators can tune per environment.

TDD: literal-pinned persist_config_defaults().sync_timeout_ms == 500. All
timeout-sensitive tests already pass an explicit timeout_ms (none rely on the
default), so none are slowed; full controller suite green under ASAN. No
commit-engine/barrier/replication/wire change beyond the default value + plumbing.
bug-561.
This commit is contained in:
Rongsong Shen 2026-06-14 20:19:05 +00:00
parent c4fd3e1c8f
commit ca1d9ae42d
5 changed files with 24 additions and 1 deletions

View file

@ -1139,6 +1139,10 @@ int cluster_init_phase2(cluster_t* c)
: "controller";
persist_config_t pcfg = persist_config_defaults();
/* --sync-timeout-ms override (0 = keep the persist default). */
if (c->myconf->sync_timeout_ms > 0)
pcfg.sync_timeout_ms = c->myconf->sync_timeout_ms;
persist_set_workdir(&c->persist, workdir, name);
rc = persist_init(&c->persist, &pcfg, c);
if (rc != 0) {