Losing a Zone on Four Clouds: Same 6-Second Freeze, Very Different Bills
The textbook version of high availability fits on a sticky note: three availability zones, a cluster spread across them, a replicated database. Nobody tells you what the cluster actually does when one zone goes silent, or what that costs on providers that are not AWS. A developer building an open-source deployment platform called Flui ran the experiment with a stopwatch: the same K3s cluster on OVHcloud, Scaleway, Hetzner, and AWS, isolating a zone three times per provider and timing every write. The scripts are public in the dawit-io/ha-eu-bench repo, and they measure failure instead of promising it.
The setup: one cluster, four providers, nftables as the wrecking ball
The test rig was deliberately plain. Three machines per cluster, one per zone, each running K3s as both control plane and worker. No EKS, no managed databases, no tuned flags. In Milan: AWS c7i.large, OVH c3-4, and Scaleway BASIC3-X2C-4G, all at 2 vCPUs and 4 GB, the cheapest size every provider sells in all three zones. Hetzner got separate treatment: it has no availability zones inside one region, so its three "zones" are three cities, Falkenstein and Nuremberg in Germany plus Helsinki in Finland.
To kill a zone, the author did not shut anything down. He isolated it with nftables so the nodes stayed powered and reachable by nobody. That choice matters: an isolated zone is harsher than a dead one, because the partitioned nodes are alive and still trying to participate. To the cluster, both cases look identical. A probe wrote to the Kubernetes API continuously from a healthy zone, and a three-replica test app (one replica pinned per zone, deliberately) served requests while the failure ran for seven minutes.
Result one: the freeze is identical everywhere, and etcd owns the clock
All four providers survived every test, three runs each. When the isolated zone held the etcd leader, writes on Kubernetes blocked for 6 to 10 seconds, then resumed after a single leader election. When the leader was elsewhere, the pause was 0.03 to 0.07 seconds. The numbers were the same on AWS, OVH, Scaleway, and Hetzner, and the author's extra tests confirmed why: etcd decides the timing, not the machine. K3s sets the election timeout conservatively so a slow network does not trigger spurious elections, and etcd jitters it per election. Dedicated vCPUs changed nothing; a crash and a network partition changed nothing.
If you take one planning number from this, take the 6-to-10-second write freeze. Your workloads keep serving during it, but nothing gets scheduled, updated, or rescaled. Any automation that assumes the API server is always writable is a small time bomb with a 10-second fuse.
Result two: the routing black hole is the part users feel
What users see differs from cluster survival. After isolation began, Kubernetes kept sending traffic to the dead replica for 43 to 78 seconds, until the node was marked unhealthy. During that window, between a quarter and a half of requests failed, each after waiting out a 2-second timeout. The app never went down; one in three requests just hung and died for about a minute.
Then came the second clock: at 344 to 379 seconds, Kubernetes gave up on the node and deleted the lost replica. With one replica pinned per zone, capacity stayed down until the zone returned. Two delays to design around: roughly a minute of traffic black-holing, and roughly six minutes before the control plane admits the node is gone. The cheapest mitigation is boring and client-side: short timeouts plus one retry, which usually lands on a healthy replica.
Result three: the single-copy components are the real single point of failure
K3s installs CoreDNS, Traefik, and metrics-server as single replicas by default. Spread your app across three zones however you like: if the zone that dies happens to host Traefik, your three-replica app is unreachable from outside for the full 352 to 376 seconds until the pod is evicted and rescheduled. Metrics-server took 395 to 409 seconds, leaving kubectl top and autoscaling blind. CoreDNS was worse before the fix: names stopped resolving cluster-wide, and a replicated database that should have restarted in a minute stayed down until the zone came back. Three replicas are useless if one invisible component lives in exactly one zone and nobody decided that was acceptable. The fix costs one line each: three replicas for every system component, including the ones you did not install.
Result four: three cities are not three zones, and the leader pays for it
On Hetzner, the cluster behaved identically during failures, same 6-to-10-second elections, same eviction clocks. The tax shows up on every ordinary write. A write is confirmed when a majority of the three etcd copies have it. With the leader in Germany, the second German node answers in about 3 milliseconds. With the leader in Helsinki, the peer being quizzed is 24 to 36 milliseconds away, and every write takes 63 ms, six to twelve times slower, on both cheap and dedicated machines. After any failover or reboot, the leader can land in Helsinki, and nothing moves it back on its own. Three clusters built the same day showed Falkenstein-to-Nuremberg bandwidth of 4.9, 0.26, and 1.6 Gbit/s: each server lands on different hardware. A single benchmark run there is folklore, not data.
The bills: about 109 euros versus about 259
The real cost of three zones is the machines. On-demand list prices from October 2026, disk and IPv4 included: Scaleway about 109 euros per month, OVH about 122, AWS about 259, and AWS also charges roughly one cent per GB for cross-zone traffic in each direction while OVH's private network is free. The dedicated-versus-shared split matters too: Hetzner's "half the price" reputation holds only for shared vCPUs, and on Scaleway, steal ranged from 1 to 13 percent, pushing slow writes from 0.35 percent of the total to 9 percent when the receiving node was CPU-starved.
What the AWS premium bought in this test was the most consistent network disk (3 ms fsync, p99 at 3.4 ms) and non-shared cores, plus managed services the experiment never touched. The European providers gave up mostly noise: one 97 ms p99 outlier on Scaleway, ephemeral local disks on OVH that suit etcd perfectly and would wreck a lone Postgres.
What to copy, and what not to trust
Isolate a zone with firewall rules on your own cluster and watch what your dashboards do, because every gap listed above was invisible until someone timed it. Set system components to three replicas before your first incident, keep client timeouts short, and if you pick a budget provider with distant "zones," pin your etcd leader or accept paying the speed of light on every write. The caveats are the author's own: three runs per provider is a demonstration, not statistics, and the failed zone's disks, databases, load balancers, and public DNS were outside the test. Still, the cluster on a 109-euro bill held the line as well as the one at 259.
Comments