Rongsong Shen
20e59d0b42
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
94 lines
4 KiB
Markdown
94 lines
4 KiB
Markdown
# Error Handling Standards
|
|
|
|
## Philosophy
|
|
|
|
Fail fast with a consistent, machine-readable response. Handle errors at the controller boundary; let business logic return typed errors. Never silently swallow errors.
|
|
|
|
## Go API Response Pattern
|
|
|
|
All HTTP responses go through two helpers in `controllers/response.go`:
|
|
|
|
```go
|
|
// Success (always HTTP 200)
|
|
ReturnSuccess(ctx, common.SUCCESS_CODE, common.SUCCESS_MSG, data)
|
|
|
|
// Error (HTTP status == code parameter)
|
|
ReturnError(ctx, common.ERROR_AUTH_CODE, common.ERROR_ACCESS_FORBIDDEN_MSG, nil)
|
|
ctx.Abort() // always Abort() after ReturnError in middleware
|
|
return // always return immediately after ReturnError in handlers
|
|
```
|
|
|
|
**The early-return pattern is non-negotiable.** Every error path ends with `ReturnError(...); return`.
|
|
|
|
## HTTP Status Code Mapping
|
|
|
|
| Code | Constant | When to use |
|
|
|------|----------|-------------|
|
|
| 200 | `SUCCESS_CODE` | Successful operation |
|
|
| 201 | `CREATED` | Resource created |
|
|
| 400 | `INVALID_PARAMS_CODE` | Malformed or invalid input |
|
|
| 401 | `ERROR_AUTH_CHECK_TOKEN_CODE` | Missing or invalid JWT |
|
|
| 403 | `ERROR_AUTH_CODE` | Valid JWT but insufficient permission |
|
|
| 404 | `ERROR_NOT_FOUND_CODE` | Resource not found |
|
|
| 429 | `ERROR_TOO_MANY_REQUESTS_CODE` | Rate limit exceeded |
|
|
| 500 | `ERROR_CODE` | Unexpected server error |
|
|
|
|
All code constants are defined in `common/common.go`. Add new codes there — do not inline numeric literals in handlers.
|
|
|
|
## Structured Logging with Zap
|
|
|
|
Log at the call site before returning an error. Use typed field helpers — never `fmt.Sprintf` in zap calls.
|
|
|
|
```go
|
|
// Preferred pattern
|
|
logger.Error("rate limit exceeded",
|
|
zap.String("path", path),
|
|
zap.String("client_ip", ctx.ClientIP()),
|
|
zap.Int("errorcode", code),
|
|
)
|
|
|
|
// For error values
|
|
logger.Error("validate token error", zap.Error(err))
|
|
|
|
// For complex types
|
|
logger.Debug("permissions resolved", zap.Any("permsWithType", permsWithType))
|
|
```
|
|
|
|
**Log levels**:
|
|
- `Error` — operation failed, client or server fault
|
|
- `Warn` — recoverable edge case (e.g., retried operation succeeded)
|
|
- `Info` — key lifecycle events (server start, config load)
|
|
- `Debug` — diagnostic detail, disabled in production
|
|
|
|
**Never log**: JWT tokens, passwords, secret keys, or full request bodies containing sensitive data.
|
|
|
|
## C Error Propagation
|
|
|
|
Return negative errno-style codes for C daemon errors. Use the `errnox.h` definitions from `src/include/xce/`. Check and propagate immediately — do not continue on error:
|
|
|
|
```c
|
|
int rc = xce_send_request(&req);
|
|
if (rc < 0) {
|
|
log_error("send_request failed: %d", rc);
|
|
return rc;
|
|
}
|
|
```
|
|
|
|
**Controller daemons MUST NEVER self-terminate on an operational condition.** A controller exits only on a genuine crash or an admin/supervisor kill — never via `exit()`/`_exit()`/`abort()`/`tlog_exit()` as a designed response to an operational failure (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" here means: log a loud operator-visible ERROR (with the actionable boundary), enter a permanent non-promotable/fenced state, **stay alive, and keep retrying** so the node self-heals when conditions recover — never terminate (a self-exited controller needs manual recovery and degrades quorum meanwhile). `assert()`/abort is reserved for genuine programming-invariant violations only. Authoritative statement: `ha-async-io.md` (process-lifecycle invariant) and `src/controller/CLAUDE.md` ("Process Lifecycle"); precedent: `ha-commit-durability`'s `persist_recover_loud_fence_if_unrecoverable` + `recovery_unrecoverable` sticky fence.
|
|
|
|
## Input Validation Pattern
|
|
|
|
Validate at the controller boundary using a `preCheck{Resource}Request()` helper before touching services:
|
|
|
|
```go
|
|
func (c JobController) SubmitJob(ctx *gin.Context) {
|
|
req, err := preCheckSubmitJobRequest(ctx)
|
|
if err != nil {
|
|
ReturnError(ctx, common.INVALID_PARAMS_CODE, err.Error(), nil)
|
|
return
|
|
}
|
|
// ... service call
|
|
}
|
|
```
|
|
|
|
Keep `preCheck*` functions in the same file as the handler. They bind, validate, and return the typed request struct or an error.
|