Every timeout must be shorter than the one in front

The claim

Most cascading outages are not caused by a component failing. They are caused by a component being slow while every timeout in the stack is longer than the one upstream of it, so nothing gives up in the right order and every layer holds resources waiting for a layer that has already been abandoned. Timeouts must decrease as you move inward. If they do not, your system has no way to shed load.

What goes wrong with the defaults

Consider a common stack running entirely on defaults:

A slow query now occupies an application worker for as long as the database takes. The CDN already told the customer it failed at 30 seconds, and the customer already refreshed — creating a second request that also waits. Your workers fill with requests nobody is listening to any more, and the site stops responding to everything, including the pages that have nothing to do with the slow query.

The ordering rule

Each layer's timeout must be shorter than the layer in front of it, with enough headroom to return a useful error:

CDN / load balancer   30 s
  Nginx proxy_read    25 s
    App request       20 s
      DB statement     8 s
      HTTP client      5 s  (with 1 retry = 10 s worst case)

The database and outbound HTTP budgets must fit inside the application budget including retries. A 5-second client timeout with three retries is 15 seconds, which blows a 20-second request budget as soon as anything else in the handler is slow. Count the worst case, not the happy path.

Setting them

# Postgres, per connection or per role
SET statement_timeout = '8s';
SET idle_in_transaction_session_timeout = '30s';

# Nginx
proxy_connect_timeout 3s;
proxy_send_timeout   25s;
proxy_read_timeout   25s;

idle_in_transaction_session_timeout is the one nearly everyone omits, and it prevents a distinct and nastier failure: an application that opens a transaction, calls a slow external API inside it, and holds locks and prevents vacuum for the duration. Set it and that class of incident disappears.

Connect timeouts should be short — three seconds is generous for a TCP handshake within a region. A long connect timeout gains nothing: if the connection is not established quickly, the host is not there.

Pool size is a timeout problem in disguise

Timeouts only shed load if there is a queue limit behind them. A connection pool of 20 with 8 workers means 8 requests can be in flight and the rest wait — for how long? Set that explicitly:

pool_acquire_timeout = 2s

Failing fast with a 503 when the pool is exhausted is the correct behaviour. It sheds the excess, keeps the queue bounded, and lets the requests that can be served complete quickly. Queueing everything until the database recovers means the recovery arrives to find a backlog it cannot clear.

Retries make the problem worse before they make it better

A retry is an extra request sent to a system that is already struggling, which is why naive retry logic converts a brief slowdown into a sustained outage. Three rules keep them useful. Retry only idempotent operations, or ones carrying an idempotency key. Add jitter — a random spread of a few hundred milliseconds — so that a thousand clients do not retry in unison and reproduce the original spike exactly one second later. And stop retrying entirely once the failure rate to a dependency crosses a threshold, letting requests fail fast until it recovers.

Test it deliberately

Introduce the slowness on purpose in a non-production environment and watch what happens:

psql -c "SELECT pg_sleep(30)"   # in one session
ab -n 200 -c 20 https://staging.example.ca/    # observe status codes

The correct outcome is fast 503s for the affected route and normal service everywhere else. If instead the entire site becomes unresponsive, your timeouts are ordered wrongly and you have just found it in a controlled setting rather than during a supplier's outage.

Write them down

Put the full ladder in one file in your repository, with the value for every layer and a line explaining why. Timeouts are configured in five different places by five different people over three years, and without a single source the ordering silently inverts the next time someone raises one value to fix an unrelated complaint about a slow report.