Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
4 KiB
Error Handling Standards
Philosophy
Fail fast with a consistent, machine-readable response. Handle errors at the controller boundary; let business logic return typed errors. Never silently swallow errors.
Go API Response Pattern
All HTTP responses go through two helpers in controllers/response.go:
// Success (always HTTP 200)
ReturnSuccess(ctx, common.SUCCESS_CODE, common.SUCCESS_MSG, data)
// Error (HTTP status == code parameter)
ReturnError(ctx, common.ERROR_AUTH_CODE, common.ERROR_ACCESS_FORBIDDEN_MSG, nil)
ctx.Abort() // always Abort() after ReturnError in middleware
return // always return immediately after ReturnError in handlers
The early-return pattern is non-negotiable. Every error path ends with ReturnError(...); return.
HTTP Status Code Mapping
| Code | Constant | When to use |
|---|---|---|
| 200 | SUCCESS_CODE |
Successful operation |
| 201 | CREATED |
Resource created |
| 400 | INVALID_PARAMS_CODE |
Malformed or invalid input |
| 401 | ERROR_AUTH_CHECK_TOKEN_CODE |
Missing or invalid JWT |
| 403 | ERROR_AUTH_CODE |
Valid JWT but insufficient permission |
| 404 | ERROR_NOT_FOUND_CODE |
Resource not found |
| 429 | ERROR_TOO_MANY_REQUESTS_CODE |
Rate limit exceeded |
| 500 | ERROR_CODE |
Unexpected server error |
All code constants are defined in common/common.go. Add new codes there — do not inline numeric literals in handlers.
Structured Logging with Zap
Log at the call site before returning an error. Use typed field helpers — never fmt.Sprintf in zap calls.
// Preferred pattern
logger.Error("rate limit exceeded",
zap.String("path", path),
zap.String("client_ip", ctx.ClientIP()),
zap.Int("errorcode", code),
)
// For error values
logger.Error("validate token error", zap.Error(err))
// For complex types
logger.Debug("permissions resolved", zap.Any("permsWithType", permsWithType))
Log levels:
Error— operation failed, client or server faultWarn— recoverable edge case (e.g., retried operation succeeded)Info— key lifecycle events (server start, config load)Debug— diagnostic detail, disabled in production
Never log: JWT tokens, passwords, secret keys, or full request bodies containing sensitive data.
C Error Propagation
Return negative errno-style codes for C daemon errors. Use the errnox.h definitions from src/include/xce/. Check and propagate immediately — do not continue on error:
int rc = xce_send_request(&req);
if (rc < 0) {
log_error("send_request failed: %d", rc);
return rc;
}
Controller daemons MUST NEVER self-terminate on an operational condition. A controller exits only on a genuine crash or an admin/supervisor kill — never via exit()/_exit()/abort()/tlog_exit() as a designed response to an operational failure (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" here means: log a loud operator-visible ERROR (with the actionable boundary), enter a permanent non-promotable/fenced state, stay alive, and keep retrying so the node self-heals when conditions recover — never terminate (a self-exited controller needs manual recovery and degrades quorum meanwhile). assert()/abort is reserved for genuine programming-invariant violations only. Authoritative statement: ha-async-io.md (process-lifecycle invariant) and src/controller/CLAUDE.md ("Process Lifecycle"); precedent: ha-commit-durability's persist_recover_loud_fence_if_unrecoverable + recovery_unrecoverable sticky fence.
Input Validation Pattern
Validate at the controller boundary using a preCheck{Resource}Request() helper before touching services:
func (c JobController) SubmitJob(ctx *gin.Context) {
req, err := preCheckSubmitJobRequest(ctx)
if err != nil {
ReturnError(ctx, common.INVALID_PARAMS_CODE, err.Error(), nil)
return
}
// ... service call
}
Keep preCheck* functions in the same file as the handler. They bind, validate, and return the typed request struct or an error.