Request hedging is not a retry: how to tame P99 latency

Request hedging is not a retry: how to tame P99 latency

Your P50 latency is fine. Your P99 is embarrassing. The slowest one percent of requests takes 40 times longer than the median, and a retry will not fix it, because the requests that need retrying are the slow ones, not the failed ones. The pattern that actually helps is request hedging: fire a second identical request after a short delay, take whichever response arrives first, and cancel the loser. It has been in Google's toolbox since the BigTable paper, and it is becoming a production norm for a new reason: LLM inference.

Hedging is not retrying

Retries and hedges both send the same request more than once, but they solve different problems. A retry is pessimistic: something failed, so try again, usually with backoff. A hedge is optimistic: nothing has failed yet, but the request might be slow, so start a backup before the deadline. A retry responds to an error. A hedge responds to time.

That distinction drives most of the rules for when hedging is safe. A failed call has already done nothing, or you can tell it did nothing. A hedged call is still in flight and may already have done its work. So hedging demands idempotency in a stricter sense than retries do: the second copy must be safe to execute even if the first copy is halfway done. Think about charging a card. Retrying after a timeout is risky because the first call may have succeeded. Hedging, which fires a second charge 500 ms later, is worse. The rule of thumb: hedge reads, and hedge writes only when you already wrap them in an idempotency key.

The mechanics

When to fire the hedge is the central tuning knob. Fire too early and you pay double cost for most requests. Fire too late and the duplicate arrives after the deadline anyway. The BigTable paper's answer: pick the delay so that only the tail of requests triggers a hedge. In their benchmark, hedging after a 10 ms delay cut the 99.9th percentile of a 1,000-key read from 1,800 ms to 74 ms, with a 2% increase in total backend load. That is the whole trade in one sentence: a small steady cost in exchange for a collapsed tail.

In practice you need three numbers: the hedge delay (how long to wait before firing the duplicate), the deadline (the absolute budget, after which you give up), and a max hedge count (usually one or two, never unbounded). A naive implementation in Node:

async function hedgedCall(fn, { delay = 100, deadline = 1000 }) {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), deadline);
  try {
    const primary = fn(controller.signal);
    const hedge = new Promise((resolve, reject) => {
      setTimeout(() => {
        fn(controller.signal).then(resolve, reject);
      }, delay);
    });
    return await Promise.race([primary, hedge]);
  } finally {
    clearTimeout(timer);
  }
}

Two things this snippet gets wrong on purpose. The loser is never cancelled: the hedged request keeps running on the server unless you propagate a cancellation token. And it races two promises that share one abort signal, so the deadline kills both copies at once. Real implementations, whether gRPC's hedging support, an Envoy hedging policy, or a resilience library, handle cancellation by sending a cancel to the losing request, and they key hedging on a per-call budget rather than a global timer.

When hedging is the wrong tool

Hedging is a load generator with a deadline. It helps only when the extra load lands somewhere useful. Skip it when:

• The backend is a single instance. A hedge against one overloaded node just doubles that node's work. Hedging pays off when you have replicas or multiple backends, so the second copy has a chance of landing somewhere healthier.
• The call is not idempotent. Creating an order, charging a card, sending an email. Hedge any of those and you are paying twice.
• The tail comes from your own queue. If P99 is slow because your thread pool is saturated, hedging makes the saturation worse. Fix the queue first.
• The cost is not symmetric. Hedging an expensive operation, like a batch job or a large model inference, can cost more than the latency it saves. Multiply your hedge rate by the cost per call and compare it to what the latency reduction is worth.

The 2026 twist: hedging LLM inference

This is where the pattern became a production norm. LLM inference is the worst-case tail latency workload. A single request can hit a cold replica, sit behind a long prefill queue, or land on a GPU mid-eviction, and P99 can be several times the P50. Providers now hedge their own inference internally, and teams are hedging across providers.

The pattern in practice: send the same prompt to two providers, or two endpoints on the same provider, with a short stagger. Accept the first response that meets your quality bar. Cancel the rest. For latency-sensitive features like voice assistants and agentic tool calls, where a two-second tail means a dropped conversation, the math works even at token prices.

The caveats are new but familiar. Token costs, not request counts, are the currency: a hedged call that gets cancelled mid-generation has already spent tokens on the prefix, and providers differ on whether partial output is billed. Quality is not uniform: the first response to arrive is not always the best one, and hedging across different models introduces a non-determinism your evals may not cover. And cancellation is often fake: many inference APIs have no cancel endpoint, so the loser runs to completion and you pay for it.

Hedging is a small pattern with a strict contract: idempotent calls, multiple backends, a bounded hedge count, and a real cancellation story. Get those four things right and the tail collapses. Get them wrong and you have built a load generator with a deadline. The LLM wave has made the pattern more valuable and the cost math harder. Do the math first, then pick your delay from your own P99 curve, not from someone else's benchmark.