78 lines
13 KiB
Markdown
78 lines
13 KiB
Markdown
# Implementation Plan
|
|
|
|
> TDD is mandatory (project rule): add the failing C test in `testsuite/` (or a piped `xce_test` behavioral script) before the change — red → green → refactor.
|
|
> The work runs along a single request path (schema → controller handler → dispatch gate → libxce client → test command → wait-primary), so tasks are sequential by design; no `(P)` markers. The one HA-critical change is the dispatch-gate allow-list (task 2.2).
|
|
|
|
- [ ] 1. Foundation: status reply schema + codegen
|
|
- [x] 1.1 Add the `ctrlstatus` opcode and the status reply payload to the schema
|
|
- Add a new request opcode `ctrlstatus` (next free ordinal) and a new reply-union member (next free ordinal), plus a lean status payload struct carrying: cluster name, primary name, responder name and role, term, loaded config version and content checksum, committed config version, commit and apply positions, fenced flag, and a bounded list of configured controllers (name, host, port) with a count.
|
|
- Keep the payload focused on identity/health only — it must NOT carry the resource/queue/worker inventory.
|
|
- Observable: the build regenerates the C types for the new payload and opcode (the generated status type, its reply-union tag, and a free helper exist) and the schema compiles clean.
|
|
- _Requirements: 1.1, 1.4, 2.1, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 6.1, 6.2, 6.3, 6.4_
|
|
|
|
- [ ] 2. Controller: status handler + any-role dispatch
|
|
- [x] 2.1 Implement the controller status handler
|
|
- Add a handler that snapshots the responding node's local HA and config state into the status reply: responder name and role; current primary (taken under the HA lock; empty when none is known/elected); term; loaded config version + content checksum; committed config version; commit and apply positions; fenced state (catch-up or unrecoverable-fence); and the configured controllers list with count. It must be read-only (no state mutation) and answer in every role.
|
|
- Route the new opcode to this handler in the controller request dispatch switch.
|
|
- TDD: unit tests assert the field values for a PRIMARY (primary == self, role == primary, populated peers/version/checksum), a STANDBY with a known primary (reports that primary, role == standby), and a node with no known primary (empty primary → "unknown"); and assert the handler mutates no state.
|
|
- Observable: the unit tests pass for the primary, standby, and unknown-primary cases, and a no-mutation assertion holds.
|
|
- _Requirements: 1.1, 1.2, 1.3, 2.1, 2.2, 2.3, 3.1, 3.2, 4.1, 4.2, 5.1, 5.2, 5.3, 6.1, 6.2, 6.3, 6.4_
|
|
- _Boundary: handle_ctrlstatus_
|
|
- [x] 2.2 Allow the status query through the HA dispatch gate in every role
|
|
- Add the status opcode to the dispatch-gate allow-lists for primary (including the no-quorum branch), standby, electing, and initializing — so a status read always answers (including when fenced or without quorum), rather than being rejected as standby/not-ready. Scope the change strictly to this read-only opcode.
|
|
- TDD: an integration test confirms the status request is answered (not rejected) on a standby, an electing node, an initializing node, and a primary without quorum; and a guard test confirms no other opcode's gating changed.
|
|
- Observable: the gate test shows the status query answered in all four role conditions and unchanged behavior for at least one previously primary-only opcode.
|
|
- _Requirements: 7.1, 7.2_
|
|
- _Boundary: ha_dispatch_gate_
|
|
- _Depends: 2.1_
|
|
|
|
- [ ] 3. libxce client API
|
|
- [x] 3.1 Add the failover-aware any-node status call
|
|
- Add the opcode-only request sender, the send-and-decode call, and the failover-aware client read (retryable, non-write) that returns whatever reachable node answers, plus the free helper. On no reachable controller, return a typed failure without a partial/empty success.
|
|
- TDD: a round-trip test decodes every status field from a handler reply; a no-reachable-controller test returns a typed failure.
|
|
- Observable: the client decodes all fields from a live/loopback handler reply and surfaces a typed error when no controller is reachable.
|
|
- _Requirements: 1.1, 7.1, 7.2, 7.3_
|
|
- _Boundary: libxce client_
|
|
- _Depends: 1.1, 2.1, 2.2_
|
|
- [x] 3.2 Add the authoritative reach-the-primary status call
|
|
- Add a client call that returns only an authoritative reply (responder role == primary). When the first answer is from a non-primary, follow the redirect — target the reported primary via the reply's peer list, otherwise rotate the configured controller list — until the primary answers or the failover budget is exhausted.
|
|
- TDD: a test where the first reachable node is a standby confirms the call follows the hint to the primary and returns a primary-role reply; a no-serving-primary test returns failure (never a standby hint as the authoritative result).
|
|
- Observable: the authoritative call returns a reply whose responder role is primary even when a standby answered first, and fails (does not return a hint) when no primary serves.
|
|
- _Requirements: 2.3, 9.1, 9.2_
|
|
- _Boundary: libxce client_
|
|
- _Depends: 3.1_
|
|
|
|
- [ ] 4. Test client command + wait-primary integration
|
|
- [x] 4.1 Add the `ctrlstatus` client command
|
|
- Add a client command that issues the any-node status call and prints the fields in human-readable form: the current primary, the answering node framed as "answered by <node> (<role>)" distinct from the primary, the configured controllers and count, the loaded config version and committed config version, the content checksum as hex, term, commit/apply positions, fenced state, and cluster name; render an empty primary as "unknown". The command exit status reflects success or failure of the query.
|
|
- Observable: piping `ctrlstatus` through the client prints the status fields (with "answered by" and "unknown" rendering) and exits zero on success / non-zero with the error printed on failure.
|
|
- _Requirements: 1.3, 3.3, 5.1, 5.2, 8.1, 8.2, 8.3_
|
|
- _Boundary: cmd_ctrlstatus_
|
|
- _Depends: 3.1_
|
|
- [x] 4.2 Switch wait-primary to the authoritative status probe
|
|
- Replace the wait-primary probe (currently a job-list call) with the authoritative reach-the-primary status call. Success is an authoritative reply from the primary; on success, print the primary's identity. A standby/hint answer is not success — the probe follows the redirect or keeps polling until the primary answers or the deadline elapses. Preserve the existing timeout-argument handling, monotonic deadline, poll interval, log-quieting, and success/failure exit semantics.
|
|
- Observable: against a healthy cluster wait-primary returns and names the primary; a malformed timeout still errors; the existing exit-code behavior is unchanged; and it does not succeed from a standby-only answer.
|
|
- _Requirements: 9.1, 9.2, 9.3, 9.4_
|
|
- _Boundary: cmd_wait_primary_
|
|
- _Depends: 3.2_
|
|
|
|
- [ ] 5. Validation (live cluster)
|
|
- [x] 5.1 End-to-end status + authoritative wait-primary on a live cluster
|
|
- Run against a live 3-node cluster: `ctrlstatus` on a healthy cluster shows the correct primary, peers, and version+checksum; querying via a standby still reports the primary with the "answered by <standby>" framing; the version+checksum match across nodes with identical config; and the authoritative wait-primary returns immediately on a healthy cluster, blocks until the NEW primary itself answers after a primary kill (without false-succeeding on a standby's stale primary view), and times out when no primary serves.
|
|
- Observable: the live run shows a correct status (primary/peers/checksum) including via a standby, and the authoritative wait-primary names the new primary after a kill without a false success; manual environment prerequisite (live cluster), same status as other live-E2E tasks.
|
|
- _Requirements: 2.3, 5.3, 7.1, 7.2, 8.1, 9.1, 9.2, 9.3, 9.4_
|
|
- _Depends: 4.1, 4.2_
|
|
|
|
## Implementation Notes (5.1 E2E outcome)
|
|
- 5.1 ran live on a 3-node `/xce-deploy-ha`-style cluster (ext2/3 fs). PASSED in-scope: `ctrlstatus` shows the primary self-reporting (`Primary: ctrl-1` on the primary), standbys honestly report `Primary: unknown` with `answered by ctrl-2 (standby)` (the dispatch-gate lets standbys answer — Req 7); peers + count correct; `configChecksum` = the real 64-hex SHA-256 (matched the deploy tag prefix) and is IDENTICAL across all 3 nodes with identical config (Req 5.3); `configVersion`==`committedConfigVersion`; authoritative `wait-primary` names the primary in <1s on a serveable cluster (Req 9.1/9.2).
|
|
- 5.1 FOUND + FIXED a real 2.1 bug: `ha->primary_host` is VESTIGIAL (never written anywhere in the controller), and `voted_for_name` is the last-vote (often self), so neither names the current primary off-primary. Corrected: PRIMARY self-reports its own name; STANDBY/ELECTING/INIT report empty → "unknown" (the authoritative client path reaches the real primary by rotation). Commit ade0081.
|
|
- 5.1 UPSTREAM FINDING (out of boundary, NOT controller-status): with a DEAD node FIRST in the masters list, the libxce client failover returns `EINVAL` instead of advancing to the live nodes — so the authoritative `wait-primary` (and ANY client op) can't reach the primary while a dead node leads the list. CONFIRMED pre-existing: plain `joblist` fails IDENTICALLY (`xce_joblist: Invalid argument`) in the same condition; with a live-only masters list `wait-primary` succeeds in 46ms. This spec consumes failover unchanged (per Boundary/Adjacent expectations) → route to libxce-failover, do NOT patch here. The durability soak avoided it by restarting killed nodes (no dead-first-master during waits).
|
|
|
|
## Implementation Notes
|
|
- All new C types are generated from the schema (task 1.1) — no hand-written struct headers. Build regenerates `controller.capnp.h/.c`.
|
|
- Task 2.2 is the one HA-critical change (request-handling path): keep it scoped to the single read-only opcode and prove no other opcode's gating widened.
|
|
- The authoritative redirect (3.2) is what makes wait-primary trustworthy (no stale-standby false success) — it is the resolution of the design-review decision; do not collapse it back into a plain any-node read for wait-primary.
|
|
- 1.1 LEARNING (affects later tasks): the `$C.mapname(...)` capnp types are NOT auto-generated in this schema — the generated `controller.capnp.c` references HAND-AUTHORED C typedefs in `src/include/typemap.h` (same convention as `xce_clusterconf_t`). So `xce_ctrlstatus_t`/`xce_ctrlpeer_t` (and the `xce_reply_t.data.ctrlstatus` union member) live in `typemap.h`, verified field-for-field against the regenerated codec. `OPCode_ctrlstatus = 40`, `RequestReply_data_ctrlstatus = 12`. `configChecksum` is `xce_data_t*`. The handler/libxce tasks build/encode against these typemap.h types.
|
|
- 2.1 LEARNING + SCHEMA AMENDMENT: capnp `Data` in THIS schema maps to the project's STRUCTURED `xce_data_t` union (table-switched), NOT raw bytes — `encode_Data_ptr` can't serialize a SHA-256 blob. So `configChecksum @6` was changed `Data → Text` (a 64-char lowercase-hex string of the full 32-byte `config_sha256`); typemap member is `char*`; freed in `ctrlstatus_free_fields`. Ordinal @6 unchanged, no released decoder yet → safe. Handler lives in NEW `src/controller/ctrlstatus.c` (`ctrlstatus_fill`/`ctrlstatus_free_fields`/`handle_ctrlstatus`), proto in `src/controller/include/cluster.h`, routed at `cluster.c` opcode switch, test `src/controller/testsuite/test_ctrlstatus.c` (registered in `controller_test.c`, built via `controller/meson.build`). `primary_host` read under `ha->lock`; peer name/host strings are BORROWED from myconf (not freed). The libxce client (3.x) decodes `configChecksum` as Text.
|
|
- 3.1 LEARNING: the ctx-side reply union `xce_ctx_t.reply.u` (in `src/include/xcectx.h`) is SEPARATE from `xce_reply_t.data` (typemap.h) — the decode path needs BOTH a `ctrlstatus` member there AND a `case OPCode_ctrlstatus` in `xcectx.c` handle_reply (mirror clusterconf). Free: clusterconf's `xce_free_clusterconf` macro is a LATENT-BROKEN alias (undefined symbol, never expanded); use the REAL generated free — `xce_free_ctrlstatus(s) → free_CtrlStatus(s)`, `xcecli_free_ctrlstatus(s) → xce_free_ctrlstatus(s)` (any TU expanding it must include the generated `controller.capnp.h`). `xcecli_ctrlstatus` is is_write=false; NULL ctx → `-EINVAL`.
|
|
- 3.2: `xcecli_ctrlstatus_primary` = bounded loop (cap `failover.retry.max_retries`) reusing `xcecli_ctrlstatus` + `xce_ctx_capture_redirect_hint` (map `primaryName`→addr via `peers[]`, else master rotation); authoritative iff `responderRole == CTRL_PRIMARY` (== 0; `ctrl_role_t` PRIMARY=0/STANDBY=1/INIT=2/ELECTING=3 in `ha.h` — REVALIDATION TRIGGER: a role-enum reorder breaks the pinned wire ordinal `XCECLI_CTRLSTATUS_ROLE_PRIMARY=0`). On exhaustion → `-E_XCE_NOT_READY`, `*status` untouched (never a hint). Pure helpers `xcecli_ctrlstatus_is_authoritative`/`_primary_addr` unit-tested; live standby→primary redirect is 5.1.
|