fix(ha-commit-engine): raise commit-barrier default 50->500ms + add --sync-timeout-ms

Live symptom: jobsub intermittently failed with 'Connection timed out' even
though the cluster had quorum. Root cause: the synchronous commit barrier waits
only DEFAULT_SYNC_TIMEOUT_MS=50ms for durable majority replication, and that was
not configurable. 50ms is too tight for durable sync majority replication —
especially since standbys now fdatasync each replicated entry before ACK — so a
multi-entry jobsub's later WAL entry intermittently misses the window:
  commit: WAIT deadline expired without commit (lsn=3 committed_lsn=2 timeout_ms=50)
-> persist hook returns -ETIMEDOUT -> handle_jobsubmit rejects -> client ETIMEDOUT.

Fix:
- Raise DEFAULT_SYNC_TIMEOUT_MS 50 -> 500 (config.h) — realistic for durable sync
  majority replication, still bounded for failure detection.
- Add a --sync-timeout-ms controller flag threaded arguments -> myconf ->
  persist pcfg (override applied before persist_init); 0 = use default, positive
  overrides — so operators can tune per environment.

TDD: literal-pinned persist_config_defaults().sync_timeout_ms == 500. All
timeout-sensitive tests already pass an explicit timeout_ms (none rely on the
default), so none are slowed; full controller suite green under ASAN. No
commit-engine/barrier/replication/wire change beyond the default value + plumbing.
bug-561.
This commit is contained in:
Rongsong Shen 2026-06-14 20:19:05 +00:00
parent c4fd3e1c8f
commit ca1d9ae42d
5 changed files with 24 additions and 1 deletions

View file

@ -90,7 +90,7 @@ typedef uint64_t jobid_t;
#define DEFAULT_TRANSIENT_JOB_TIMEOUT_S 300
#define DEFAULT_FINISHED_RETAIN_S 3600
#define DEFAULT_WORKER_RETAIN_S 86400
#define DEFAULT_SYNC_TIMEOUT_MS 50
#define DEFAULT_SYNC_TIMEOUT_MS 500
#define DEFAULT_HB_INTERVAL_MS 3000
#define DEFAULT_MAX_WAL_RECORDS 10000

View file

@ -35,6 +35,7 @@ typedef struct {
time_t conf_stale_timeout; /* seconds before saved config flagged stale (default 86400) */
int conf_stale; /* 1 when saved config older than conf_stale_timeout */
int vcpus; /* Req 4.3: --vcpus CLI value; 0 = use config default */
uint32_t sync_timeout_ms; /* --sync-timeout-ms; 0 = use persist default */
} myconf_t;
extern cluster_config_t* get_default_conf(void*);