Designing Resilient APIs: Error Handling, Rate Limiting & Graceful Degradation
The enterprise guide to building fault-tolerant backend services that never crash under unexpected payloads or third-party downtime.
1. Introduction: The Fallacy of Network Reliability
In the 1990s, engineers at Sun Microsystems wrote down the assumptions that get distributed systems into trouble. The list became the Fallacies of Distributed Computing: the network is reliable, latency is zero, bandwidth is infinite, the topology doesn't change. Thirty years later, most production incidents are still one of those assumptions failing at 2 a.m.
Your API sits in the middle of a chain of things that fail on their own schedule. Packets get dropped. A payment gateway starts answering in nine seconds instead of nine hundred milliseconds. A CRM vendor returns intermittent 500s for four minutes. A client retries aggressively and triples the traffic you were already struggling with. None of this is exceptional. It's the normal operating environment.
Defensive API engineering starts from that premise: failure is a normal state, and the system's job is to keep behaving sensibly while it happens. The design goals for this guide are:
- Contain the blast radius. A failure in one dependency degrades one feature, not the whole service.
- Prevent cascades. Slow or failing dependencies must not consume the resources everything else needs.
- Fail loudly and predictably. Clients get consistent, machine-readable errors and clear retry guidance.
- Crash correctly. When the process truly can't continue, it exits in an orderly way and gets replaced, instead of limping along in a corrupted state.
A note on the Node.js failure model, because it's often misunderstood. A slow downstream service does not block the event loop, since I/O is asynchronous. What it does is more insidious: every in-flight request keeps a socket, a closure, buffers and a timer alive. Under a stalled dependency, pending work accumulates, memory climbs, connection limits are hit, latency for unrelated endpoints rises, and eventually the process is killed by the OOM killer or the orchestrator. The patterns below all target that mechanism: cap it, shed it, and isolate it.
The examples assume Node 20+, TypeScript 5, Express 5 (which forwards rejected promises from async handlers to error middleware), ioredis 5, and Zod for validation.
2. Pattern 1: Standardizing Error Responses with RFC 7807
The chaos of ad hoc errors
Look at the error responses in a mature API and you'll often find every style at once: a bare string here, { "error": "..." } there, { "message": ..., "code": ... } from another team, and an HTML stack trace from the framework when something slips through. Clients end up with a pile of special cases, and support can't correlate a customer complaint with a log line.
RFC 7807 Problem Details defines a common shape, served as application/problem+json. One detail worth knowing: RFC 9457 (2023) obsoletes RFC 7807 with the same core structure, plus clarifications and a registry for problem types. Everything below is compatible with both.
Transient vs. permanent errors
The most valuable thing an error can tell a client is whether retrying can help:
The server side: a typed problem catalog and global handler
// src/errors/problem.ts
export interface InvalidParam {
name: string;
reason: string;
}
export interface ProblemDetails {
type: string;
title: string;
status: number;
detail?: string;
instance?: string;
retryable: boolean;
invalid_params?: InvalidParam[];
[extension: string]: unknown;
}
export const PROBLEM_BASE = "https://api.example.com/problems";
export const PROBLEMS = {
validation_failed: { status: 422, title: "Request validation failed", retryable: false },
malformed_body: { status: 400, title: "Malformed request body", retryable: false },
payload_too_large: { status: 413, title: "Payload too large", retryable: false },
unauthorized: { status: 401, title: "Authentication required", retryable: false },
not_found: { status: 404, title: "Resource not found", retryable: false },
rate_limited: { status: 429, title: "Rate limit exceeded", retryable: true },
dependency_unavailable: { status: 503, title: "A dependency is temporarily unavailable", retryable: true },
internal_error: { status: 500, title: "Internal server error", retryable: false },
} as const satisfies Record<string, { status: number; title: string; retryable: boolean }>;
export type ProblemCode = keyof typeof PROBLEMS;
export interface AppErrorOptions {
detail?: string;
retryAfterSec?: number;
invalidParams?: InvalidParam[];
extensions?: Record<string, unknown>;
cause?: unknown;
}
export class AppError extends Error {
readonly code: ProblemCode;
readonly options: AppErrorOptions;
constructor(code: ProblemCode, options: AppErrorOptions = {}) {
super(options.detail ?? PROBLEMS[code].title, { cause: options.cause });
this.name = "AppError";
this.code = code;
this.options = options;
}
}// src/errors/problem-handler.ts
import { randomUUID } from "node:crypto";
import type { ErrorRequestHandler } from "express";
import { ZodError } from "zod";
import { logger } from "../logger.js";
import { BulkheadFullError, CircuitOpenError } from "../resilience/circuit-breaker.js";
import {
AppError, PROBLEMS, PROBLEM_BASE,
type AppErrorOptions, type ProblemCode, type ProblemDetails,
} from "./problem.js";
// body-parser errors carry a `type` such as "entity.parse.failed" / "entity.too.large"
interface BodyParserError extends Error { type: string; status: number }
function isBodyParserError(err: unknown): err is BodyParserError {
return err instanceof Error
&& typeof (err as Partial<BodyParserError>).type === "string"
&& (err as BodyParserError).type.startsWith("entity.");
}
function classify(err: unknown): { code: ProblemCode; options: AppErrorOptions } {
if (err instanceof AppError) return { code: err.code, options: err.options };
if (err instanceof ZodError) {
return {
code: "validation_failed",
options: {
detail: "One or more parameters failed validation.",
invalidParams: err.issues.map((i) => ({ name: i.path.join("."), reason: i.message })),
},
};
}
if (isBodyParserError(err)) {
return {
code: err.type === "entity.too.large" ? "payload_too_large" : "malformed_body",
options: {},
};
}
if (err instanceof CircuitOpenError) {
return {
code: "dependency_unavailable",
options: { retryAfterSec: Math.max(1, Math.ceil(err.retryAfterMs / 1000)) },
};
}
if (err instanceof BulkheadFullError) {
return { code: "dependency_unavailable", options: { retryAfterSec: 1 } };
}
return { code: "internal_error", options: {} };
}
export const problemHandler: ErrorRequestHandler = (err, req, res, next) => {
// If the response has started streaming, Express's default handler must close the socket
if (res.headersSent) return next(err);
const { code, options } = classify(err);
const spec = PROBLEMS[code];
const requestId = (res.locals.requestId as string | undefined) ?? randomUUID();
const log = spec.status >= 500 ? logger.error : logger.warn;
log.call(logger, { err, requestId, method: req.method, path: req.path, code }, "request failed");
const problem: ProblemDetails = {
type: `${PROBLEM_BASE}/${code}`,
title: spec.title,
status: spec.status,
// Never echo err.message on unknown errors: it can leak hostnames, SQL or secrets.
detail: code === "internal_error"
? "An unexpected error occurred. Quote the instance ID when contacting support."
: options.detail,
instance: `urn:uuid:${requestId}`,
retryable: spec.retryable,
...(options.invalidParams && { invalid_params: options.invalidParams }),
...options.extensions,
};
if (options.retryAfterSec !== undefined) res.set("Retry-After", String(options.retryAfterSec));
res
.status(spec.status)
.set("Cache-Control", "no-store")
.type("application/problem+json")
.send(JSON.stringify(problem));
};Stack traces never reach the client in any environment, because the client gets an instance ID and the stack lives in your logs, joined by that ID.
The client side: jittered exponential backoff
Retries without jitter cause synchronized retry storms, where every client that failed at the same moment retries at the same moment. Full jitter picks a random delay between zero and an exponentially growing ceiling:
// src/resilience/retry.ts
export class HttpStatusError extends Error {
constructor(readonly status: number, readonly retryAfterMs?: number) {
super(`HTTP ${status}`);
this.name = "HttpStatusError";
}
}
const RETRYABLE_STATUS = new Set([408, 429, 502, 503, 504]);
export function isTransient(err: unknown): boolean {
if (err instanceof HttpStatusError) return RETRYABLE_STATUS.has(err.status);
return err instanceof TypeError // fetch network failure
|| (err instanceof DOMException && err.name === "TimeoutError");
}
function sleep(ms: number, signal?: AbortSignal): Promise<void> {
return new Promise((resolve, reject) => {
if (signal?.aborted) return reject(signal.reason);
const onAbort = () => { clearTimeout(timer); reject(signal!.reason); };
const timer = setTimeout(() => {
signal?.removeEventListener("abort", onAbort);
resolve();
}, ms);
signal?.addEventListener("abort", onAbort, { once: true });
});
}
export interface RetryOptions {
retries: number;
baseMs: number;
capMs: number;
signal?: AbortSignal;
shouldRetry?: (err: unknown) => boolean;
}
export async function retry<T>(fn: (attempt: number) => Promise<T>, opts: RetryOptions): Promise<T> {
const shouldRetry = opts.shouldRetry ?? isTransient;
for (let attempt = 0; ; attempt++) {
try {
return await fn(attempt);
} catch (err) {
if (attempt >= opts.retries || !shouldRetry(err)) throw err;
const ceiling = Math.min(opts.capMs, opts.baseMs * 2 ** attempt);
const jittered = Math.random() * ceiling; // full jitter
const serverHint = err instanceof HttpStatusError ? err.retryAfterMs ?? 0 : 0;
await sleep(Math.max(jittered, serverHint), opts.signal); // never retry sooner than asked
}
}
}3. Pattern 2: Distributed Rate Limiting with Redis
Why fixed windows fail
A fixed-window limiter (INCR key, EXPIRE key 60) is simple and cheap and has a well-known flaw at the window boundary. With a limit of 100 per minute, a client can send 100 requests at 00:59 and another 100 at 01:00. That's 200 requests in about two seconds, twice the intended limit, and it's exactly the burst pattern that hurts downstream systems.
Sliding window log vs. sliding window counter
Sliding window counter (approximation)
previous window current window
|----------------|xxxxxxxx|-------------|
t-60s t-window now (elapsed = 25s of 60s)
estimated = prev_count × (60−25)/60 + curr_count
= prev_count × 0.583 + curr_count
The log is exact but stores one entry per request, so a 10,000/minute limit keeps up to 10,000 members per key. The counter trades a small, bounded error for constant memory and cost.
Atomicity: why Lua
Read-check-write across separate Redis calls has a race: two API instances both read 99, both decide "allowed," and both write, so you exceed the limit. A Lua script runs atomically on the Redis server, and it also lets us use Redis's clock instead of every app server's (avoiding clock skew).
// src/rate-limit/redis-limiter.ts
import { randomUUID } from "node:crypto";
import type { Request, RequestHandler } from "express";
import { Redis } from "ioredis";
import { AppError } from "../errors/problem.js";
import { logger } from "../logger.js";
// Sliding window LOG: exact, ZSET of request timestamps.
const SLIDING_LOG_LUA = `
local t = redis.call('TIME')
local now = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)
local window = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now - window)
local count = redis.call('ZCARD', KEYS[1])
if count < limit then
redis.call('ZADD', KEYS[1], now, ARGV[3])
redis.call('PEXPIRE', KEYS[1], window)
return {1, limit - count - 1, 0}
end
local oldest = redis.call('ZRANGE', KEYS[1], 0, 0, 'WITHSCORES')
local retryAfter = window - (now - tonumber(oldest[2]))
return {0, 0, retryAfter}
`;
// Sliding window COUNTER: O(1) memory, weighted previous window.
const SLIDING_COUNTER_LUA = `
local window = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local elapsed = now % window
local prev = tonumber(redis.call('GET', KEYS[2]) or '0')
local curr = tonumber(redis.call('GET', KEYS[1]) or '0')
local weight = (window - elapsed) / window
if (prev * weight + curr) >= limit then
return {0, 0, window - elapsed}
end
curr = redis.call('INCR', KEYS[1])
if curr == 1 then redis.call('PEXPIRE', KEYS[1], window * 2) end
return {1, math.max(0, math.floor(limit - (prev * weight + curr))), 0}
`;
declare module "ioredis" {
interface RedisCommander<Context> {
slidingWindowLog(
key: string, windowMs: number, limit: number, member: string
): Promise<[allowed: number, remaining: number, retryAfterMs: number]>;
slidingWindowCounter(
currKey: string, prevKey: string, windowMs: number, limit: number, nowMs: number
): Promise<[allowed: number, remaining: number, retryAfterMs: number]>;
}
}
export const redis = new Redis(process.env.REDIS_URL!, {
enableOfflineQueue: false, // fail fast when disconnected instead of queueing commands in memory
commandTimeout: 50, // ms; tune to your Redis p99
maxRetriesPerRequest: 1,
});
redis.defineCommand("slidingWindowLog", { numberOfKeys: 1, lua: SLIDING_LOG_LUA });
redis.defineCommand("slidingWindowCounter", { numberOfKeys: 2, lua: SLIDING_COUNTER_LUA });
export interface RateLimitOptions {
name: string;
limit: number;
windowMs: number;
algorithm?: "log" | "counter";
keyFor?: (req: Request) => string;
failOpen?: boolean;
}
interface Decision { allowed: boolean; remaining: number; retryAfterMs: number }
async function decide(opts: RateLimitOptions, id: string): Promise<Decision> {
const now = Date.now();
if (opts.algorithm === "log") {
const [allowed, remaining, retryAfterMs] = await redis.slidingWindowLog(
`rl:log:${opts.name}:${id}`, opts.windowMs, opts.limit, `${now}-${randomUUID()}`
);
return { allowed: allowed === 1, remaining, retryAfterMs };
}
const idx = Math.floor(now / opts.windowMs);
const tag = `rl:{${opts.name}:${id}}`; // hash tag: both keys land in one Redis Cluster slot
const [allowed, remaining, retryAfterMs] = await redis.slidingWindowCounter(
`${tag}:${idx}`, `${tag}:${idx - 1}`, opts.windowMs, opts.limit, now
);
return { allowed: allowed === 1, remaining, retryAfterMs };
}
export function rateLimit(opts: RateLimitOptions): RequestHandler {
const failOpen = opts.failOpen ?? true;
const keyFor = opts.keyFor ?? ((req: Request) => req.ip ?? "unknown"); // set `trust proxy` correctly!
return async (req, res, next) => {
let decision: Decision;
try {
decision = await decide(opts, keyFor(req));
} catch (err) {
// The limiter is itself a dependency: decide deliberately what happens when it is down.
logger.error({ err, limiter: opts.name }, "rate limiter backend unavailable");
if (failOpen) return next();
return next(new AppError("dependency_unavailable", { retryAfterSec: 1, cause: err }));
}
const windowSec = Math.ceil(opts.windowMs / 1000);
res.set({
"RateLimit-Limit": String(opts.limit),
"RateLimit-Remaining": String(decision.remaining),
"RateLimit-Reset": String(decision.allowed ? windowSec : Math.ceil(decision.retryAfterMs / 1000)),
});
if (decision.allowed) return next();
const retryAfterSec = Math.max(1, Math.ceil(decision.retryAfterMs / 1000));
return next(new AppError("rate_limited", {
retryAfterSec,
detail: `Limit of ${opts.limit} requests per ${windowSec}s exceeded. Retry in ${retryAfterSec}s.`,
}));
};
}Usage: app.use("/v1/search", rateLimit({ name: "search", limit: 600, windowMs: 60_000, algorithm: "counter" })), and algorithm: "log" for something like /v1/auth/login where exactness matters.
Headers
Return 429 Too Many Requests with Retry-After (seconds), and the RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset trio so well-behaved clients can pace themselves. The header format comes from an IETF draft that has been evolving toward consolidated RateLimit/RateLimit-Policy fields, but the three-header form is what most clients and libraries understand today.
4. Pattern 3: Circuit Breaker Pattern and Fail-Fast Mechanics
How one slow dependency takes down everything
Imagine a payment gateway that starts answering in 30 seconds instead of 300 ms. Each request to your API that touches it now waits 30 seconds. At 200 requests per second, you accumulate 6,000 in-flight requests within 30 seconds: 6,000 sockets, 6,000 pending handlers, 6,000 sets of buffers. The gateway didn't crash, but your service is now slow, memory-hungry and hitting connection limits, and requests that never touch payments are suffering too. By default, the global fetch dispatcher doesn't cap connections per origin, so nothing in the runtime pushes back.
A circuit breaker stops the accumulation. Once a dependency is clearly unhealthy, calls to it fail immediately, which frees resources, protects the dependency from a retry stampede while it recovers, and lets the caller apply a fallback.
The state machine
failure rate ≥ threshold
┌───────────────────────────────────────┐
│ ▼
┌─────────┐ ┌─────────┐
│ CLOSED │ │ OPEN │ all calls rejected instantly
│ (normal)│ │ │ (CircuitOpenError)
└─────────┘ └────┬────┘
▲ │ openDuration elapsed
│ probes succeed ▼
│ ┌───────────┐
└──────────────────────────────────│ HALF_OPEN │ limited probe calls allowed
probe fails ───────▶│ │──▶ back to OPEN
└───────────┘Three details that matter in practice:
- Minimum volume. Don't trip on 1 failure out of 2 calls. Require a minimum sample size before evaluating the failure rate.
- Limited probes in HALF_OPEN. Let only a few calls through, or the recovering service gets flooded.
- Timeouts are part of the breaker. Without a hard timeout, a hung call is never recorded as a failure.
A production-grade implementation
This one also includes a bulkhead, a cap on concurrent in-flight calls, which is what actually protects your sockets and memory.
// src/resilience/circuit-breaker.ts
export type CircuitState = "CLOSED" | "OPEN" | "HALF_OPEN";
export class CircuitOpenError extends Error {
constructor(readonly circuit: string, readonly retryAfterMs: number) {
super(`Circuit "${circuit}" is open`);
this.name = "CircuitOpenError";
}
}
export class BulkheadFullError extends Error {
constructor(readonly circuit: string) {
super(`Bulkhead for "${circuit}" is full`);
this.name = "BulkheadFullError";
}
}
export class UpstreamError extends Error {
constructor(readonly status: number) {
super(`Upstream responded ${status}`);
this.name = "UpstreamError";
}
}
/** What counts against the breaker: network errors, timeouts and 5xx. 4xx are the caller's fault. */
export function isDependencyFailure(err: unknown): boolean {
return err instanceof UpstreamError
|| err instanceof TypeError
|| (err instanceof DOMException && err.name === "TimeoutError");
}
export interface CircuitBreakerOptions {
name: string;
failureRateThreshold: number; // 0..1
minimumCalls: number; // samples required before the rate is evaluated
windowSize: number; // rolling window of the last N outcomes
openDurationMs: number;
halfOpenMaxProbes: number;
maxConcurrent: number; // bulkhead
callTimeoutMs: number;
onStateChange?: (name: string, from: CircuitState, to: CircuitState) => void;
}
export class CircuitBreaker {
private state: CircuitState = "CLOSED";
private outcomes: boolean[] = []; // true = success
private openedAt = 0;
private inFlight = 0;
private probesStarted = 0;
private probeSuccesses = 0;
constructor(private readonly opts: CircuitBreakerOptions) {}
get currentState(): CircuitState { return this.state; }
async exec<T>(fn: (signal: AbortSignal) => Promise<T>, parent?: AbortSignal): Promise<T> {
this.admit();
this.inFlight++;
const timeout = AbortSignal.timeout(this.opts.callTimeoutMs);
const signal = parent ? AbortSignal.any([parent, timeout]) : timeout;
try {
const result = await fn(signal);
this.record(true);
return result;
} catch (err) {
// A caller abort says nothing about the dependency's health
if (!(parent?.aborted)) this.record(!isDependencyFailure(err));
throw err;
} finally {
this.inFlight--;
}
}
private admit(): void {
const now = Date.now();
if (this.state === "OPEN") {
const remaining = this.openedAt + this.opts.openDurationMs - now;
if (remaining > 0) throw new CircuitOpenError(this.opts.name, remaining);
this.transition("HALF_OPEN");
}
if (this.inFlight >= this.opts.maxConcurrent) throw new BulkheadFullError(this.opts.name);
if (this.state === "HALF_OPEN") {
if (this.probesStarted >= this.opts.halfOpenMaxProbes) {
throw new CircuitOpenError(this.opts.name, 1_000);
}
this.probesStarted++;
}
}
private record(success: boolean): void {
if (this.state === "HALF_OPEN") {
if (!success) return this.trip();
if (++this.probeSuccesses >= this.opts.halfOpenMaxProbes) this.transition("CLOSED");
return;
}
if (this.state !== "CLOSED") return; // late results from calls started before the trip
this.outcomes.push(success);
if (this.outcomes.length > this.opts.windowSize) this.outcomes.shift();
if (this.outcomes.length < this.opts.minimumCalls) return;
const failures = this.outcomes.filter((ok) => !ok).length;
if (failures / this.outcomes.length >= this.opts.failureRateThreshold) this.trip();
}
private trip(): void {
this.openedAt = Date.now();
this.transition("OPEN");
}
private transition(to: CircuitState): void {
const from = this.state;
if (from === to) return;
this.state = to;
this.outcomes = [];
this.probesStarted = 0;
this.probeSuccesses = 0;
this.opts.onStateChange?.(this.opts.name, from, to);
}
}Fallbacks: serving stale data safely
A tripped breaker doesn't have to mean an error page. For read-only, non-critical data (catalog details, exchange rates, recommendations), serve the last good value and mark it degraded. The cache needs a size bound, because unbounded caches are their own memory-pressure bug:
// src/resilience/resilient-client.ts
import {
BulkheadFullError, CircuitBreaker, CircuitOpenError, UpstreamError, isDependencyFailure,
} from "./circuit-breaker.js";
class StaleCache<V> {
private readonly map = new Map<string, { value: V; storedAt: number }>();
constructor(private readonly maxEntries: number, private readonly maxStaleMs: number) {}
set(key: string, value: V): void {
this.map.delete(key); // refresh insertion order (LRU)
this.map.set(key, { value, storedAt: Date.now() });
if (this.map.size > this.maxEntries) {
const oldest = this.map.keys().next().value;
if (oldest !== undefined) this.map.delete(oldest);
}
}
get(key: string): { value: V; ageMs: number } | undefined {
const entry = this.map.get(key);
if (!entry) return undefined;
const ageMs = Date.now() - entry.storedAt;
if (ageMs > this.maxStaleMs) { this.map.delete(key); return undefined; }
return { value: entry.value, ageMs };
}
}
export interface Fetched<T> { data: T; degraded: boolean; ageMs: number }
export class ResilientHttpClient {
private readonly cache: StaleCache<unknown>;
constructor(
private readonly breaker: CircuitBreaker,
cacheOpts: { maxEntries: number; maxStaleMs: number },
) {
this.cache = new StaleCache(cacheOpts.maxEntries, cacheOpts.maxStaleMs);
}
async getJson<T>(url: string, opts: { freshMs: number; signal?: AbortSignal }): Promise<Fetched<T>> {
const cached = this.cache.get(url) as { value: T; ageMs: number } | undefined;
if (cached && cached.ageMs < opts.freshMs) {
return { data: cached.value, degraded: false, ageMs: cached.ageMs };
}
try {
const data = await this.breaker.exec(async (signal) => {
const res = await fetch(url, { signal, headers: { accept: "application/json" } });
if (res.status >= 500) throw new UpstreamError(res.status);
if (!res.ok) throw new Error(`Upstream rejected request: ${res.status}`); // 4xx: not a health signal
return (await res.json()) as T;
}, opts.signal);
this.cache.set(url, data);
return { data, degraded: false, ageMs: 0 };
} catch (err) {
const dependencyProblem = isDependencyFailure(err)
|| err instanceof CircuitOpenError
|| err instanceof BulkheadFullError;
if (dependencyProblem && cached) {
return { data: cached.value, degraded: true, ageMs: cached.ageMs };
}
throw err;
}
}
}export const catalogBreaker = new CircuitBreaker({
name: "catalog",
failureRateThreshold: 0.5,
minimumCalls: 20,
windowSize: 50,
openDurationMs: 15_000,
halfOpenMaxProbes: 3,
maxConcurrent: 100,
callTimeoutMs: 2_500,
onStateChange: (name, from, to) => logger.warn({ circuit: name, from, to }, "circuit state change"),
});5. Pattern 4: Graceful Degradation and Partial Responses
Separate critical paths from enrichment
Every endpoint has a critical core (the order and its line items) and optional enrichment (recommendations, review counts, loyalty points). If enrichment shares a failure domain with the core, a broken recommendation service can take down checkout. Decouple them explicitly:
Resilient GraphQL through nullability
GraphQL is well suited to this because its execution model has partial failure built in: a field error is recorded in errors with its path, the field becomes null, and the nulling propagates up to the nearest nullable parent. So the schema's nullability is your degradation policy:
type Order {
id: ID!
status: OrderStatus!
total: Money!
lines: [OrderLine!]!
# Optional enrichment: nullable on purpose. A failure here can't null the order.
recommendations: [Product!]
reviewSummary: ReviewSummary
}
type Query {
order(id: ID!): Order # nullable: if a critical field fails, only this branch is lost
}If total: Money! fails, the error propagates to Query.order, which is nullable, so a critical failure correctly nulls the order. If recommendations fails, it's nullable and the rest of the order survives. The resolvers add timeouts and record degradation:
// src/graphql/server.ts
import { ApolloServer, type ApolloServerPlugin } from "@apollo/server";
import { logger } from "../logger.js";
interface Ctx {
signal: AbortSignal;
degraded: Set<string>;
services: {
orders: { get(id: string, signal: AbortSignal): Promise<OrderModel> };
recommendations: { forOrder(id: string, signal: AbortSignal): Promise<ProductModel[]> };
reviews: { summary(id: string, signal: AbortSignal): Promise<ReviewSummaryModel> };
};
}
async function degradable<T>(
ctx: Ctx,
dependency: string,
budgetMs: number,
fn: (signal: AbortSignal) => Promise<T>,
): Promise<T | null> {
try {
return await fn(AbortSignal.any([ctx.signal, AbortSignal.timeout(budgetMs)]));
} catch (err) {
ctx.degraded.add(dependency);
logger.warn({ err, dependency }, "optional dependency failed; returning null");
return null;
}
}
const resolvers = {
Query: {
order: (_: unknown, args: { id: string }, ctx: Ctx) => ctx.services.orders.get(args.id, ctx.signal),
},
Order: {
recommendations: (o: OrderModel, _: unknown, ctx: Ctx) =>
degradable(ctx, "recommendations", 250, (s) => ctx.services.recommendations.forOrder(o.id, s)),
reviewSummary: (o: OrderModel, _: unknown, ctx: Ctx) =>
degradable(ctx, "reviews", 250, (s) => ctx.services.reviews.summary(o.id, s)),
},
};
// Surface degradation to clients and monitors without failing the query
const degradationPlugin: ApolloServerPlugin<Ctx> = {
async requestDidStart() {
return {
async willSendResponse({ contextValue, response }) {
if (contextValue.degraded.size > 0 && response.body.kind === "single") {
response.body.singleResult.extensions = {
...response.body.singleResult.extensions,
degraded: [...contextValue.degraded],
};
}
},
};
},
};Returning null silently versus throwing a GraphQLError is a real design choice. Throwing gives clients an explicit entry in errors with a path, which suits cases where they must react. Returning null plus extensions.degraded keeps clients simple for purely decorative data. Either way, remember that partial success is usually an HTTP 200, so monitor errors and extensions, not just status codes, and configure error formatting so internals don't leak into messages.
The same principle for REST
function requestSignal(res: Response): AbortSignal {
const ctrl = new AbortController();
res.on("close", () => { if (!res.writableFinished) ctrl.abort(new Error("client disconnected")); });
return ctrl.signal;
}
const within = <T>(ms: number, parent: AbortSignal, fn: (s: AbortSignal) => Promise<T>) =>
fn(AbortSignal.any([parent, AbortSignal.timeout(ms)]));
app.get("/orders/:id", async (req, res) => {
const signal = requestSignal(res);
const order = await orders.get(req.params.id, signal); // critical: errors go to problemHandler
const [recs, reviews] = await Promise.allSettled([
within(250, signal, (s) => recommendations.forOrder(order.id, s)),
within(250, signal, (s) => reviewService.summary(order.id, s)),
]);
const degraded: string[] = [];
if (recs.status === "rejected") degraded.push("recommendations");
if (reviews.status === "rejected") degraded.push("reviews");
res.json({
...order,
recommendations: recs.status === "fulfilled" ? recs.value : null,
reviewSummary: reviews.status === "fulfilled" ? reviews.value : null,
meta: { degraded },
});
});Stale-while-revalidate at the edge
For cacheable reads, let the CDN or gateway absorb dependency failures too. Cache-Control: public, max-age=30, stale-while-revalidate=120, stale-if-error=86400 (RFC 5861) tells caches to serve the stale copy while refreshing in the background, and to keep serving it if the origin returns errors. Support varies by CDN, so verify the behavior on yours before relying on it.
6. Pattern 5: Node.js Process Hygiene and Graceful Shutdowns
Why containers change the rules
In Docker and Kubernetes, deploys, scale-downs and node drains all end in SIGTERM. Three traps:
- PID 1 doesn't get default signal handling. If your Node process is PID 1 with no handler, SIGTERM can be ignored and the container eventually gets SIGKILL, dropping in-flight requests. Install handlers, and use an init like tini (docker run --init).
- npm start doesn't forward signals reliably. Use the exec form: CMD ["node", "dist/server.js"].
- Load balancers lag. Kubernetes removes a pod from endpoints concurrently with sending SIGTERM, so for a few seconds you can still receive new traffic. Close the listener immediately and you'll refuse requests that were routed to you legitimately.
The correct shutdown sequence
SIGTERM ─▶ readiness=503 ─▶ wait for LB to notice ─▶ stop accepting ─▶ drain in-flight
(t=0) (propagation delay) server.close() (bounded)
│
exit(0) ◀─ close DB/Redis ◀─ flush BullMQ workers ◀────────────────────┘
hard deadline: force exit before SIGKILL (terminationGracePeriodSeconds)// src/lifecycle.ts
import type { Server } from "node:http";
import { setTimeout as sleep } from "node:timers/promises";
import type { Logger } from "pino";
export const lifecycle = { ready: true }; // /readyz returns 503 when false
export interface ShutdownStep {
name: string;
run: () => Promise<unknown>;
timeoutMs: number;
}
export interface ShutdownOptions {
server: Server;
log: Logger;
steps: ShutdownStep[]; // executed in order
propagationDelayMs: number; // let load balancers stop routing to us
drainTimeoutMs: number; // then forcibly close remaining connections
forceExitAfterMs: number; // must be < terminationGracePeriodSeconds
}
export function installGracefulShutdown(opts: ShutdownOptions): void {
const { server, log } = opts;
let shuttingDown = false;
const shutdown = async (reason: string, exitCode: number): Promise<void> => {
if (shuttingDown) return;
shuttingDown = true;
log.warn({ reason }, "shutdown started");
const hardStop = setTimeout(() => {
log.error("shutdown deadline exceeded; forcing exit");
process.exit(1);
}, opts.forceExitAfterMs);
hardStop.unref();
lifecycle.ready = false; // 1. fail readiness immediately
await sleep(opts.propagationDelayMs); // 2. wait for endpoints/LB propagation
await new Promise<void>((resolve) => { // 3. stop accepting, drain in-flight
const forceClose = setTimeout(() => server.closeAllConnections(), opts.drainTimeoutMs);
forceClose.unref();
server.close(() => { clearTimeout(forceClose); resolve(); });
server.closeIdleConnections(); // idle keep-alives would otherwise hold close() open
});
for (const step of opts.steps) { // 4. workers → pools → cache, in dependency order
try {
await Promise.race([
step.run(),
sleep(step.timeoutMs).then(() => { throw new Error(`timed out after ${step.timeoutMs}ms`); }),
]);
log.info({ step: step.name }, "shutdown step complete");
} catch (err) {
log.error({ err, step: step.name }, "shutdown step failed");
}
}
log.info("shutdown complete");
process.exit(exitCode);
};
process.once("SIGTERM", () => void shutdown("SIGTERM", 0));
process.once("SIGINT", () => void shutdown("SIGINT", 0));
// Crash-only: after these, in-memory state can't be trusted. Log, drain briefly, exit, get replaced.
process.on("unhandledRejection", (reason) => {
log.fatal({ err: reason }, "unhandled promise rejection");
void shutdown("unhandledRejection", 1);
});
process.on("uncaughtException", (err) => {
log.fatal({ err }, "uncaught exception");
void shutdown("uncaughtException", 1);
});
}Wiring it up:
app.use((_req, res, next) => {
if (!lifecycle.ready) res.set("Connection", "close"); // tell keep-alive clients to reconnect elsewhere
next();
});
app.get("/healthz", (_req, res) => res.sendStatus(200)); // liveness: process is up
app.get("/readyz", (_req, res) => res.sendStatus(lifecycle.ready ? 200 : 503)); // readiness: send me traffic?
const server = app.listen(PORT);
installGracefulShutdown({
server, log: logger,
propagationDelayMs: 3_000,
drainTimeoutMs: 8_000,
forceExitAfterMs: 25_000, // budget: 3 + 8 + steps (13) = 24s, inside Kubernetes' default 30s
steps: [
{ name: "bullmq-worker", run: () => worker.close(), timeoutMs: 8_000 }, // waits for active jobs
{ name: "bullmq-queue", run: () => queue.close(), timeoutMs: 1_000 },
{ name: "postgres-pool", run: () => pool.end(), timeoutMs: 3_000 },
{ name: "redis", run: () => redis.quit(), timeoutMs: 1_000 },
],
});About "zero crashes"
It's tempting to catch uncaughtException and carry on. Don't. After an uncaught exception or unhandled rejection, the process may hold half-updated state, leaked locks or corrupted caches, and Node's default for unhandled rejections is to crash for exactly this reason. The resilient design is two-layered: request-level containment (the patterns above) makes such events rare, and process-level crash-only behavior makes them safe when they happen, with an orderly exit and a fresh replacement from the orchestrator. Zero unhandled crashes means every crash is handled, not that crashes never occur.
7. Architecture Summary and Production Checklist
Six checks for an audit-ready service:
- One error contract. Every failure is an RFC 9457/7807 application/problem+json response with a stable type, an instance ID correlated to logs, a retryable flag, and no stack traces or internal messages.
- Disciplined retries. Full-jitter exponential backoff, a small retry budget, Retry-After honored, and retries limited to idempotent operations.
- Shared, atomic rate limits. Redis-backed Lua scripts (sliding window counter for throughput, log for strictness), standard headers, and an explicit fail-open or fail-closed decision.
- Per-dependency breakers, timeouts and bulkheads. No outbound call without a hard timeout, a concurrency cap and a failure policy.
- Explicit critical/optional split. Nullable enrichment fields, per-call time budgets, degraded flags in responses, and stale fallbacks only where staleness is safe.
- Crash-only processes with graceful shutdown. Readiness that flips first, bounded draining, ordered resource teardown, and a hard exit deadline inside your platform's grace period.
Resilience isn't a library you install. It's a set of decisions made per endpoint and per dependency, in advance: what do we do when this is slow, when it's down, when it's abused, when we're being killed? Teams that write those answers into code and rehearse them (with chaos tests, load tests and deliberately killed dependencies) are the ones whose incident reviews get boring. That's the goal.
Written by Nazmul Hawlader
Top RatedSenior Full-Stack Engineer & Official Shopify App Store developer. Founder of Stockly and Kilo (kilo.nazmulcodes.org), specializing in high-performance Shopify apps, client-side media compression, and sub-second web performance.