01 article

Spotify doubled monthly changes to 17,000. The bottleneck is verification, not AI

Spotify's own incident data found no direct AI-authored failure signature, even as merged changes doubled to 17,000 a month. The real risk is verification controls frozen at last year's volume, and the metric to copy is rework rate, not churn.

Spotify doubled monthly changes to 17,000. The bottleneck is verification, not AI

Spotify doubled monthly changes to 17,000. The bottleneck is verification, not AI

Every engineer I know has a pet theory about what AI slop will do to software quality. Spotify just published a year of its own data, and the answer is more boring than any of those theories: nothing broke because of the models. Things broke because the company started shipping twice as much of everything, and the machinery that checks what ships didn't get the memo.

The headline number from the Spotify Engineering post is that monthly merged changes more than doubled year over year in August 2026, from roughly 8,100 to about 17,000. This is a platform running nearly 3,000 production services for 777 million monthly users, handling 11 to 12 million backend requests per second. When they asked the quality question directly, reviewing every major incident monthly and asking whether AI-authored code contributed, the answer across the incidents reviewed so far was no. Not "we think no." The review found no material direct contribution. What they did find is the second question in their incident template: whether the increased volume of change put pressure on review, testing, rollout, or observability. That one kept lighting up.

What actually broke instead

The post walks through four areas that tested them, and none of them are what the doom threads predicted.

The content processing pipeline handles more than 500,000 new songs, videos, podcasts, and audiobooks daily. On June 24, three ordinary things combined: a scheduled batch job competing with new episode uploads, a quality improvement that had quietly increased the compute each episode needed, and a scheduling bug that cut throughput about 10%. Episodes that normally publish in minutes sat for hours. The root causes were, in Spotify's own words, garden variety: failure alerting gaps, capacity planning misses, small workload control holes. Some of the fixes shipped faster with agents involved. None of the problems were caused by them.

Fleet automation is where the new failure modes actually live. Spotify's Fleet Management framework has merged more than 2.5 million automated maintenance pull requests, most auto-merged after safety checks, and a background coding agent called Honk (Claude on the Agent SDK, running in Kubernetes pods) now handles messier migrations. A Java migration across backend services took three days. Then an automated dependency upgrade passed every existing check and still failed in production, affecting end users. That's the shape of the real risk: not that AI writes bad code, but that a system merging millions of changes has verification tuned for yesterday's volume and yesterday's definition of safe.

The other two areas are stranger. Compute shortages: the industry-wide AI demand spike shrank Spotify's spare capacity, so regional failovers that used to be routine, invisible events started degrading user-visible quality. Their response was doubling reserved edge capacity and reworking service tiering so lower-priority services accept having no home during a failover. And mobile app quality, which has always moved in cycles: ship features fast, accumulate small regressions, shift capacity toward recovery, broaden the guardrails, repeat. AI didn't introduce the cycle. It turned the frequency knob up, so gaps surface faster than the guardrails were designed to catch them, and individual releases can look healthy while smaller regressions pile up underneath, hitting particular phone models or sitting outside the signals anyone is watching. Spotify added longer-term trend views to its release checkpoints to catch that drift.

The metrics nobody else publishes

Here's the part I wish more companies would copy. Spotify classifies every merged PR by work type: features, quality and optimization, maintenance, documentation. After the AI ramp, quality and optimization work rose from 27% of the mix to 31%, while maintenance and configuration fell from 31% to 25%. Because the denominator doubled, absolute work on code quality more than doubled too. That's the opposite of the quality-for-velocity trade everyone assumes, at least at Spotify.

They also rebuilt their rework rate metric, and this distinction deserves attention. Code churn measures how much code gets removed relative to what gets added. Rework rate weighs the age of the code being removed, which is a much better proxy for whether a change actually held up. The FAROS 2026 report found a sharp industry-wide rise in churn, the famous "AI code gets thrown away" narrative. Spotify sees no corresponding rise in rework. New code isn't being deleted at an unusual rate. Churn without rework-weighting, they argue, mostly reflects refactoring and growth, not rot.

Two warning signals are creeping up, and I respect what they did with them. PR size and code complexity are both rising. Pre-AI, unambiguous red flags. Now a bigger PR might just mean a human and an agent reasoned through a bigger unit of work together, and complexity thresholds calibrated for one person's working memory may not fit anymore. Spotify says plainly that they don't have conviction in either interpretation, so they are deliberately not rewriting the thresholds to make themselves feel better. They're watching the trend instead. That's the most honest sentence in the whole post.

What a normal team can actually copy

You run maybe 20 services, not 3,000, so which of this transfers?

Treat verification as the constraint. If agent output is doubling your merge rate and your CI, staging environment, and alerting didn't change, the bottleneck moved and nobody told your infrastructure. This matches Google's 2025 DORA finding: AI adoption correlates with higher throughput and product performance, and with lower delivery stability. The stability drop isn't bad code. It's controls standing still while change accelerates.

Measure rework, not churn. Churn is easy to compute and nearly meaningless on its own. Reweight deleted code by its age and check whether recent work is the stuff getting deleted. If not, you don't have a quality crisis, you have a volume crisis, and the two need different fixes.

Schedule robot work during human hours. Spotify now runs automated changes while the owning team is awake. Obvious in hindsight, and almost nobody does it. An agent merging at 3am into a service whose on-call engineer is asleep is just an incident with extra steps.

Keep the threshold and watch the trend. When a metric you've trusted for years starts drifting in an AI-accelerated workflow, resist the urge to recalibrate it immediately. The drift might be signal, and re-tuning the dial to make the dashboard green is exactly how quality regressions hide.

The part I'd push on: "across incidents reviewed so far" is doing a lot of quiet work in that sentence, and every number here is self-reported by the company with the most to lose if the finding is wrong. A year from now, when the 17,000-changes-per-month cadence has fully aged into the codebase and the rework data gets re-run, we'll know more. Until then, the most useful thing in the post isn't Spotify's conclusion. It's the question they added to their incident template, which any team can copy this week: did the volume of change outpace our ability to verify it? Most teams that adopted agents in the last eighteen months don't know the answer, and the honest guess is that they'd say yes.

Comments