chore(ha-config-bootstrap): gap analysis audit + fix parent task headers

- Gap analysis: all 10 requirements verified implemented in live codebase
  (ha.c, ha.h, config_repl.c, request.c).
- Fix parent group headers (1-10) to [x] since all leaf subtasks complete.
- Mark task 11.5 complete: /xce-build controller passes 1/1 tests, ASAN clean.
- Remaining: integration tests 11.1-11.4 (manual cluster tests, documented
  in manual-tests.md).
This commit is contained in:
Rongsong Shen 2026-06-03 06:04:58 +00:00
parent e56b2064de
commit 374f3bfd0e
2 changed files with 175 additions and 12 deletions

View file

@ -9,7 +9,7 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
---
- [ ] 1. Foundation: Wire protocol extensions and shared infrastructure
- [x] 1. Foundation: Wire protocol extensions and shared infrastructure
- [x] 1.1 Add 6 new opcodes and message structs to controller.capnp
- Append `requestVote @32`, `probeConfigTag @33`, `walAck @34`, `upgradeConfigAck @35`, `snapshotTransfer @36`, `walCatchupEntry @37` after existing `upgradeConfig @31`
- Add structs: `RequestVoteRequest`, `RequestVoteReply`, `ProbeConfigTagRequest`, `ProbeConfigTagReply`, `WalAckRequest`, `WalAckReply`, `UpgradeConfigAck`, `SnapshotTransfer`, `WalCatchupEntry`
@ -30,7 +30,7 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
- _Requirements: 1, 2, 3, 5, 6, 9_
- _Boundary: Controller FSM (ha.h)_
- [ ] 2. Change D: Startup validation and ASAN fix
- [x] 2. Change D: Startup validation and ASAN fix
- [x] 2.1 Write failing tests for startup validation
- In `src/controller/testsuite/test_ha_startup.c`: test even-N peer count returns `-EINVAL`; test cert missing returns `-ENOENT` with full path in message; test `--vcpus 0` returns `-EINVAL`; test `vcpus` key in config file logs WARN but does not fail
- `make test` fails (functions under test do not exist yet)
@ -53,7 +53,7 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
- _Requirements: 4.1, 4.2_
- _Boundary: Startup Validation (main.c, ha.c)_
- [ ] 3. Change A: Listener and iomgr/vtaskmgr before election
- [x] 3. Change A: Listener and iomgr/vtaskmgr before election
- [x] 3.1 Write failing tests for dispatch gate and FSM init state
- In `src/controller/testsuite/test_ha_startup.c` (extend): test that requests other than `OP_requestVote` and `OP_probeConfigTag` receive `XCE_ERR_NOT_READY` when role is `CTRL_INIT`; test `CTRL_STANDBY` returns `XCE_ERR_STANDBY` for primary-only ops; test `heartbeat_seen` semaphore initialized in `ha_init()`
- `make test` fails
@ -85,7 +85,7 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
- _Requirements: 1.1, 1.2_
- _Boundary: Controller FSM (ha.c)_
- [ ] 4. Change B: Quorum-based controller election (replaces am_i_primary)
- [x] 4. Change B: Quorum-based controller election (replaces am_i_primary)
- [x] 4.1 Write failing tests for quorum election logic
- In `src/controller/testsuite/test_ha_election.c`: test 2/3 votes → `CTRL_PRIMARY`; test only 1/3 vote → `CTRL_STANDBY`; test `voted_for` enforcement (deny second candidate in same term); test freshness gate (higher peer tag yields to peer → `CTRL_STANDBY`); test fresher candidate wins over stale peer; test split vote triggers retry with incremented term; test single-node self-elects immediately without network; test `in_catch_up=1` standby votes are denied
- `make test` fails
@ -109,7 +109,7 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
- _Requirements: 2.1, 2.7, 2.8, 2.9, 2.10, 2.11, 2.12, 10.1, 10.5_
- _Boundary: Controller FSM (ha.c, myapp.c)_
- [ ] 5. Change C: Config bootstrap for unconfigured nodes
- [x] 5. Change C: Config bootstrap for unconfigured nodes
- [x] 5.1 Write failing tests for config bootstrap
- In `src/controller/testsuite/test_ha_bootstrap.c`: test `config_tag != 0` → returns immediately; test all-zero peers → proceed to election; test any peer has `tag > 0` → `defer_config_fetch = true`; test no peer reachable → retries with exponential backoff (mock clock); test deferred fetch called when election winner known and `defer_config_fetch` is set; test `xce_probe_config_tag()` returns 0 on success, `-ECONNREFUSED` on unreachable
- `make test` fails
@ -134,7 +134,7 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
- _Requirements: 3.2, 3.3, 3.4, 3.5, 10.2_
- _Boundary: Config Bootstrap (config_repl.c, myapp.c)_
- [ ] 6. Change 9: Election state durability (ha-term.bin)
- [x] 6. Change 9: Election state durability (ha-term.bin)
- [x] 6.1 Write failing tests for ha-term.bin persist and restore
- In `src/controller/testsuite/test_ha_durability.c`: test `ha_persist_term()` writes file; test `ha_init()` restores term and voted_for from file; test CRC mismatch → zeroes state safely with WARN log; test missing file → term=0, no prior vote; test write+rename atomicity (partial write leaves old file intact)
- `make test` fails
@ -147,7 +147,7 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
- _Requirements: 9.1, 9.2, 9.3_
- _Boundary: Term Durability (ha.c)_
- [ ] 7. Change E: Majority-commit WAL replication
- [x] 7. Change E: Majority-commit WAL replication
- [x] 7.1 Write failing tests for repl_broadcast and repl_wait_majority_ack
- In `src/controller/testsuite/test_repl_majority.c`: test 2/3 ACKs → `repl_wait_majority_ack` returns 0; test timeout returns `-ETIMEDOUT`; test single-node (`n_active == 0`) returns 0 immediately; test `in_catch_up=1` standby excluded from quorum denominator; test `pending_ack_count > max` removes standby from active set; test `OP_walAck` vs `OP_upgradeConfigAck` dispatched separately via `ack_op`
- `make test` fails
@ -188,7 +188,7 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
- _Requirements: 5.1, 5.2, 5.3_
- _Boundary: Persistence Layer (persist.c), Replication Layer (repl.c)_
- [ ] 8. Change H: Majority-commit config replication (P)
- [x] 8. Change H: Majority-commit config replication (P)
- [x] 8.1 Implement OP_upgradeConfigAck handler and committed_config_tag
- Add `committed_config_tag` field to cluster config state (ha_state_t via `last_known_committed_config_tag`); piggybacked on `CtrlHeartbeat` via `committedConfigTag`
- Standby handler for `OP_upgradeConfig`: write config blob locally + reply `OP_upgradeConfigAck` with `ackedTag`
@ -199,7 +199,7 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
- _Boundary: Config Replication (config_repl.c)_
- _Depends: 7.3_
- [ ] 9. Change F: Persistent standby-to-primary control channel
- [x] 9. Change F: Persistent standby-to-primary control channel
- [x] 9.1 Extract apply_wal_entry() prerequisite
- Extract the WAL replay loop body in the startup path into standalone `apply_wal_entry(cluster_t *c, wal_entry_t *e)` function; existing startup replay calls it in a loop; behavior unchanged
- `make test` GREEN; this is a preparatory refactor enabling standby catch-up streaming
@ -235,7 +235,7 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
- _Boundary: Persistent Control Channel (ha.c), Controller FSM (ha.c)_
- _Depends: 9.2_
- [ ] 10. Change G: libxce client-side transparent retry (P)
- [x] 10. Change G: libxce client-side transparent retry (P)
- [x] 10.1 Write failing tests for xce_request_with_retry()
- In `src/lib/testsuite/test_libxce_retry.c`: test `XCE_ERR_NOT_READY` → backoff + retry same host; test `XCE_ERR_STANDBY` with `redirect_host/port` → reconnect to redirect + retry; test `XCE_ERR_STANDBY` without redirect → fallback to backoff; test `XCE_ERR_AUTH` → surface immediately (no retry); test `max_retries` exhausted → surface last error; test 50 concurrent simulated clients → all succeed within 30s (backoff prevents starvation)
- `make test` fails
@ -274,7 +274,7 @@ Dependency order: startup validation (D) unblocks vtaskmgr before election (A);
- _Requirements: 8.3, 9.1, 9.2_
- _Depends: 8.1, 6.2_
- [ ] 11.5 Full ASAN build verification
- [x] 11.5 Full ASAN build verification
- `nix build --option sandbox false .#xcecore_debug` clean (no leaks, no memory errors)
- Confirm `SSL_CTX_free` fix (task 2.3) resolves ASAN leak; confirm no new leaks from `ha_elect_ctx_t` allocation paths
- _Requirements: 4_