Your Cache Proxy Layer Runs 4x Slower and Costs 3x More

Your Cache Proxy Layer Runs 4x Slower and Costs 3x More

An AI prediction service gets about 100 milliseconds end to end. One request needs hundreds of features, a number DoorDash put in its own writeup of the ML platform behind its delivery app, so the natural move is fan-out: fire off a hundred parallel fetches against a cache and merge the results. Fan-out has a catch. Your request latency becomes the latency of the slowest fetch in the batch. With a hundred parallel calls, a p99 of 10ms shows up in essentially every request. That is why feature-store teams care about microsecond data tiers, and why most of them live with millisecond ones. Dumanshu Goyal, a senior principal engineer at AWS who previously ran online data at Airbnb, gave a QCon San Francisco talk (published on InfoQ) about closing that gap. The answer was a layer most teams have in their architecture and have never measured.

Why the proxy tier crept in

Redis, and its open-source fork Valkey, began as a single process on a single box. Growing past one box means sharding, and the classic way to make independent nodes look like one cluster is a proxy tier (Envoy, Redis Enterprise Proxy, the older twemproxy crowd). The proxy accepts client connections, hashes keys to pick a shard, multiplexes onto fewer backend connections, and hides topology from the client. It is stateless, so you scale it by adding instances. For a legacy client that cannot speak the cluster protocol, that is genuinely useful. What nobody measures is the tax. Goyal's talk takes three measurements of it: latency, dollars, and blast radius.

What the benchmark showed

The setup is deliberately boring, which is what makes it believable. memtier_benchmark driving load on EC2, proxy and Valkey nodes on identical 8-core machine types, same availability zone, a read-heavy 90/10 workload like most real caches. The proxied stack served roughly 500,000 queries per second with p99 latency around 2.5 milliseconds. Then the CPU picture: Envoy at 90%, Valkey at 60%. The database had spare capacity; the proxy was the chokepoint.

The latency breakdown is more revealing. At a 1ms p50, about 600 microseconds is pure network: 300 microseconds client to proxy, 300 more proxy to server, two hops that did not exist before. Four hundred microseconds remain to cover proxy routing, server processing, and the client's own measurement work. A profile of the proxy splits its CPU roughly 45% user mode, 45% kernel mode, with the kernel time dominated by the TCP stack. Half the box exists to move bytes on someone else's behalf. The obvious move for a stateless tier that maxes out is to add instances, and the talk did exactly that, load-balancing across two proxies. More boxes, and the tail barely budged.

The failure test that settles it

The failure test is the part to steal. Goyal injected slowness into one of three shards using a server-side function that stalls the main thread for five seconds (a benchmark trick, not for Friday afternoon). Traffic was evenly distributed, so shard one carried a third of the load while shards two and three stayed healthy. Expected client-side availability: about 66%. Measured availability in the proxied architecture: zero, for the entire five seconds.

Root cause: the proxy multiplexes clients onto shared backend connections, so the slow shard occupies the pool and requests headed for healthy shards queue behind it. One slow shard takes down the whole cluster. That was three nodes; run the math with a hundred shards and one sick node. The talk cites Werner Vogel's frugal-architecture laws as the frame: what does one bad node cost you?

Direct access: the smart client with bulkheads

The alternative is direct access. A cluster-aware client learns the shard topology from the cluster itself and keeps its own connection pool per shard. Same fault injection, same traffic mix: availability hovers between 60 and 80 percent, right around the two thirds the healthy shards should serve. When shard one slows down, it saturates its own pool and its requests get throttled, while the other two keep serving. Per-pool throttling is the bulkhead pattern: ship compartments so one flooded hold does not sink the vessel.

The other two axes moved too:

One third the cost for the same million QPS, four times lower tail latency, and a fault model where one bad shard costs you one third of capacity instead of everything. Worth noting the $700 was itself optimistic: no high availability, no replicas, no cross-zone hops. Real deployments pay more on both designs; the ratio is the point.

When the proxy still earns its keep

These are directional numbers from one benchmark run, same-AZ, no HA on either side. Treat them as a probe to replicate on your own workload, not a verdict. And there are real reasons a proxy layer stays. Your clients may not be cluster-aware: legacy apps, odd language runtimes, managed services that only speak plain RESP to a single endpoint. Connection counts explode when every app instance holds a pool to every shard, and a few thousand pods times a dozen shards is tens of thousands of sockets the database never planned for. Some shops want one entry point for firewalls, TLS termination in one place, or per-tenant throttling at a shared edge. The claim worth keeping is narrower than "never proxy": measure the tier before you scale the tier.

What to copy on Monday

• Profile your proxy layer before sizing it. If most of its CPU is kernel-mode TCP, you are paying to route, not to compute, and a second instance changes nothing.
• Run the slow-shard test. Stall one backend for five seconds in staging and watch client-side availability. Zero means shared pools; partial means bulkheads.
• Count the two network hops your p50 carries. If the cache round trip is 1ms and 600 microseconds is the proxy detour, a client-library upgrade is cheaper than infrastructure.
• Check whether your client is already cluster-aware. Most mainstream Valkey and Redis clients are; the "smart client" may be a config flag, not a project.
• Keep the proxy where it pays: legacy clients, connection ceilings, shared-edge governance. Delete it where it is only inertia.

The general shape is older than Valkey. Abstraction layers get added because they remove a problem you can see, and nobody invoices you for the problems they add. A proxy tier charges three bills: two extra network hops per request, a second fleet to run, and a shared failure domain that turns one slow node into an outage. Goyal's contribution is asking all three questions in one benchmark harness. The fault-injection number is the one to remember. A million QPS is easy to put on a slide; availability of zero while two thirds of your shards sit idle is the number for your design review.