fix(ha-commit-engine): raise commit-barrier default 50->500ms + add --sync-timeout-ms
Live symptom: jobsub intermittently failed with 'Connection timed out' even though the cluster had quorum. Root cause: the synchronous commit barrier waits only DEFAULT_SYNC_TIMEOUT_MS=50ms for durable majority replication, and that was not configurable. 50ms is too tight for durable sync majority replication — especially since standbys now fdatasync each replicated entry before ACK — so a multi-entry jobsub's later WAL entry intermittently misses the window: commit: WAIT deadline expired without commit (lsn=3 committed_lsn=2 timeout_ms=50) -> persist hook returns -ETIMEDOUT -> handle_jobsubmit rejects -> client ETIMEDOUT. Fix: - Raise DEFAULT_SYNC_TIMEOUT_MS 50 -> 500 (config.h) — realistic for durable sync majority replication, still bounded for failure detection. - Add a --sync-timeout-ms controller flag threaded arguments -> myconf -> persist pcfg (override applied before persist_init); 0 = use default, positive overrides — so operators can tune per environment. TDD: literal-pinned persist_config_defaults().sync_timeout_ms == 500. All timeout-sensitive tests already pass an explicit timeout_ms (none rely on the default), so none are slowed; full controller suite green under ASAN. No commit-engine/barrier/replication/wire change beyond the default value + plumbing. bug-561.
This commit is contained in:
parent
c4fd3e1c8f
commit
ca1d9ae42d
5 changed files with 24 additions and 1 deletions
|
|
@ -60,6 +60,11 @@ static void test_persist_config_defaults_finished_retain_s(void)
|
|||
static void test_persist_config_defaults_sync_timeout_ms(void)
|
||||
{
|
||||
persist_config_t cfg = persist_config_defaults();
|
||||
/* Default raised to 500 ms: realistic for durable sync majority
|
||||
* replication (each standby fdatasyncs before ACK), still bounded for
|
||||
* failure detection. Literal-pinned so a regression to the old 50 ms
|
||||
* tight window fails here, not just on a live cluster. */
|
||||
TEST_ASSERT_EQUAL_UINT32(500, cfg.sync_timeout_ms);
|
||||
TEST_ASSERT_EQUAL_UINT32(DEFAULT_SYNC_TIMEOUT_MS, cfg.sync_timeout_ms);
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue