myxce/.kiro/steering/error-handling.md
Rongsong Shen 20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00

4 KiB

Error Handling Standards

Philosophy

Fail fast with a consistent, machine-readable response. Handle errors at the controller boundary; let business logic return typed errors. Never silently swallow errors.

Go API Response Pattern

All HTTP responses go through two helpers in controllers/response.go:

// Success (always HTTP 200)
ReturnSuccess(ctx, common.SUCCESS_CODE, common.SUCCESS_MSG, data)

// Error (HTTP status == code parameter)
ReturnError(ctx, common.ERROR_AUTH_CODE, common.ERROR_ACCESS_FORBIDDEN_MSG, nil)
ctx.Abort()   // always Abort() after ReturnError in middleware
return        // always return immediately after ReturnError in handlers

The early-return pattern is non-negotiable. Every error path ends with ReturnError(...); return.

HTTP Status Code Mapping

Code Constant When to use
200 SUCCESS_CODE Successful operation
201 CREATED Resource created
400 INVALID_PARAMS_CODE Malformed or invalid input
401 ERROR_AUTH_CHECK_TOKEN_CODE Missing or invalid JWT
403 ERROR_AUTH_CODE Valid JWT but insufficient permission
404 ERROR_NOT_FOUND_CODE Resource not found
429 ERROR_TOO_MANY_REQUESTS_CODE Rate limit exceeded
500 ERROR_CODE Unexpected server error

All code constants are defined in common/common.go. Add new codes there — do not inline numeric literals in handlers.

Structured Logging with Zap

Log at the call site before returning an error. Use typed field helpers — never fmt.Sprintf in zap calls.

// Preferred pattern
logger.Error("rate limit exceeded",
    zap.String("path", path),
    zap.String("client_ip", ctx.ClientIP()),
    zap.Int("errorcode", code),
)

// For error values
logger.Error("validate token error", zap.Error(err))

// For complex types
logger.Debug("permissions resolved", zap.Any("permsWithType", permsWithType))

Log levels:

  • Error — operation failed, client or server fault
  • Warn — recoverable edge case (e.g., retried operation succeeded)
  • Info — key lifecycle events (server start, config load)
  • Debug — diagnostic detail, disabled in production

Never log: JWT tokens, passwords, secret keys, or full request bodies containing sensitive data.

C Error Propagation

Return negative errno-style codes for C daemon errors. Use the errnox.h definitions from src/include/xce/. Check and propagate immediately — do not continue on error:

int rc = xce_send_request(&req);
if (rc < 0) {
    log_error("send_request failed: %d", rc);
    return rc;
}

Controller daemons MUST NEVER self-terminate on an operational condition. A controller exits only on a genuine crash or an admin/supervisor kill — never via exit()/_exit()/abort()/tlog_exit() as a designed response to an operational failure (unrecoverable/corrupt WAL, lost quorum, unreachable peer, failed catch-up). "Fail loud" here means: log a loud operator-visible ERROR (with the actionable boundary), enter a permanent non-promotable/fenced state, stay alive, and keep retrying so the node self-heals when conditions recover — never terminate (a self-exited controller needs manual recovery and degrades quorum meanwhile). assert()/abort is reserved for genuine programming-invariant violations only. Authoritative statement: ha-async-io.md (process-lifecycle invariant) and src/controller/CLAUDE.md ("Process Lifecycle"); precedent: ha-commit-durability's persist_recover_loud_fence_if_unrecoverable + recovery_unrecoverable sticky fence.

Input Validation Pattern

Validate at the controller boundary using a preCheck{Resource}Request() helper before touching services:

func (c JobController) SubmitJob(ctx *gin.Context) {
    req, err := preCheckSubmitJobRequest(ctx)
    if err != nil {
        ReturnError(ctx, common.INVALID_PARAMS_CODE, err.Error(), nil)
        return
    }
    // ... service call
}

Keep preCheck* functions in the same file as the handler. They bind, validate, and return the typed request struct or an error.