docs(scheduler-core): commit accumulated brief/requirements/roadmap edits

These edits (three-thread picker correction in brief.md, record_dir
path-validation and kvmgr-audit requirements in requirements.md,
OPCode_schedAdmin folded into the peer-authentication-hardening
backlog item in roadmap.md) were made during earlier requirements
review and design work but left uncommitted. Reviewed for accuracy
against the completed implementation — all still correct, persisting
as-is.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-08 03:19:14 +00:00
parent d7b4969095
commit dfd8f5b342
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
3 changed files with 32 additions and 10 deletions

View file

@ -79,8 +79,8 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o
- [ ] **investigate: job_ctrl direct-mutate path (admin suspend/resume) — migrate to commit-then-apply?** — **recorded 2026-07-13, surfaced during job-array-single-wal-record's `persist_job_hook_fun` investigation.** `raft-apply-decoupling` deliberately scoped `job_ctrl`'s `mark_job_update`/`mark_job_fail` (admin SIGCONT/SIGSTOP) OUT-OF-BOUNDARY (task 4.1/6.1 Implementation Notes) — confirmed the ONLY remaining live call site that still mutates state directly, THEN fires the `AFTER_EVENT` hook, with `persist_job_hook_fun` doing the real WAL append + async commit from inside the hook (not through `apply_committed_entry()`). This is the same mutate-before-commit shape as the bug-795/828/851/1060 divergence family raft-apply-decoupling was built to eliminate for job submit + worker facts, narrowly scoped to admin suspend/resume. Not yet known to have caused a live incident — rare, synchronous, admin-only traffic. **Investigate before committing to a fix**: (a) actual blast radius if a commit fails/times out post-mutation (does it ever get observed/notified before commit — violates criterion 2.4-style provisional-state-export concerns?); (b) live-apply vs replay-apply asymmetry for this one WAL entry type — does replay actually diverge from live in any reachable scenario, or is it benign in practice; (c) cost of migrating vs. accepted-residual status quo. Promote to a spec (likely a small `raft-apply-decoupling` follow-up, not standalone) only if the investigation finds real risk, not proactively.
- [ ] **WAL-trim cursor-bound guard (bug-1147; surfaced 2026-06-25 by raft-drive-async design review)** — `trim_wal_segments` (`persist.c:1374`) bounds deletion ONLY by the open segment (`if (seg_start_lsn < active_seg_lsn) unlink()`) — it consults **no** standby `acked_lsn`, **no** `committed_lsn`, **no** in-flight snapshot boundary. With no working snapshot backfill (`write_snapshot` emits a 16-byte sentinel only), a standby lagging by more than ~one open segment hits a log range that exists nowhere → permanent committed-prefix gap (the bug-795/828 family). This is **strategy-A retention discipline that should exist regardless**. Stopgap fix: compute `floor = min(all standby acked_lsn, in-flight snapshot boundaries)` under `p->lock` and refuse to delete at/above `floor`. **Folded into `raft-drive-async` task 3.2** (which moves checkpoint/trim onto a concurrent lane, worsening the race) — but stands alone if that slips. Does NOT fix empty/far-behind join (cursor=0); that needs `entity-bearing snapshots` (strategy B, backlog). Related: bug-795, bug-828, bug-1060.
- [ ] OP_heartbeat peers cross-validation — include a hash of the local `--peers` string in heartbeat requests; log ERROR on mismatch between candidates. Detects misconfigured nodes at election time instead of silently risking dual-primary split-brain. _(FUTURE_WORK.md §3.)_
- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists, admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`.
- [ ] validate_host() real authorization — currently a stub returning success for all peers; MUST be implemented before `CONF_ENABLE_UPGRADE_CONFIG=1`. Minimum: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.**
- [ ] **peer-authentication-hardening** — **NEW 2026-07-06 from admin-config-tool discovery** (`/kiro-discovery`). XCE currently operates under a **trusted-LAN model** — any CA-signed SSL peer can connect to the controller and perform ANY operation. Three gaps identified: **(a) Admin→Controller**: `validate_host()` is a stub returning success for all peers (roadmap § below). `OP_upgradeConfig` has no real authorization gate today. **Same gap confirmed 2026-08-02 for `OPCode_schedAdmin`** (`scheduler-core` design, `cluster.c`'s `case OPCode_upgradeConfig` was checked as the closest precedent and has only a primary/standby routing gate, no RBAC/auth check) — `scheduler-core` ships `OPCode_schedAdmin` at parity with that existing gap (project owner decision: not a `scheduler-core` blocker), so this item now covers both opcodes; when `validate_host()` is implemented, wire it to both `OP_upgradeConfig` and `OPCode_schedAdmin` in the same pass. **(b) Worker/Scheduler→Controller**: workers and schedulers connect via `controller_connector.c` using the same TLS context as controller peers, with no role differentiation — a compromised worker could send `OP_upgradeConfig` or read full config via `OP_replConfig`. **(c) Cross-component trust**: no distinction between controller peers, admin tools, workers, and schedulers — they share the same X.509 identity pool. **(d) xceadmin login (future)**: xceadmin currently uses zero-config TLS (same as xce_test). Production deployments likely need an admin login requirement — token-based session (`xceadmin login → token → config push`) or password-per-command. X.509 CN alone proves identity but not intent; a login step provides audit trail, session expiry, and operator accountability. **Needs design**: role-based X.509 CN validation (controller/admin/worker/scheduler roles), per-role OP code allowlists (now including `OPCode_schedAdmin`), admin login/session model, audit logging for admin operations. Separate from the existing `validate_host()` hardening (below) which is the minimal gate for `OP_upgradeConfig`/`OPCode_schedAdmin` specifically. Discuss + design before implementing; likely promotes to a spec via `/kiro-discovery`.
- [ ] validate_host() real authorization — currently a stub returning success for all peers; MUST be implemented before `CONF_ENABLE_UPGRADE_CONFIG=1`. Minimum: verify peer X.509 CN against a configured allowlist (`--admin-cn` flag or `admin_cn_list` config), reject non-allowlisted `OP_upgradeConfig` **and `OPCode_schedAdmin`** (added 2026-08-02, `scheduler-core`) with clear error, WARN-log rejected attempts with CN+IP. Future: IP allowlist, role-based access, admin-specific CA. _(FUTURE_WORK.md §6; security-critical.)_ **Part of the broader `peer-authentication-hardening` item above.**
- [ ] OP_replConfig access control hardening — currently serves the full config blob to any CA-signed SSL peer (trusted-LAN model). Consider: verify requester is a known `--peers` controller, rate-limit per peer, `minimum_role` check (controllers only). _(FUTURE_WORK.md §7.)_ **Part of the broader `peer-authentication-hardening` item above.**
- [ ] lost-and-found drain command — `xcecli lost-and-found drain`: reassign or cancel sidelined jobs, reset `lost_and_found_job_count`. Automates operator cleanup after config upgrade. _(FUTURE_WORK.md §4.)_
- [ ] OP_upgradeConfig standby redirect verification — test that a standby returns the primary's address to an admin tool and the tool retries against the primary. Note: redirect semantics change under ha-consistency-invariants Req 1 (no transitional-state redirect); verify against the new behavior. _(FUTURE_WORK.md §5.)_