A developer sent me a message: were we having problems with the clusters? I opened production and looked. Everything was green. So I replied: "No, everything looks fine. If we had a problem, my Teams would be lighting up like a Christmas tree."
Two minutes later, my Teams lit up like a Christmas tree.
Our managed Kubernetes clusters in production had stopped working. Not because of anything we deployed. Our provider had rolled out an update to its platform, and one component in it partly broke the network. The nodes could no longer be managed, and the load balancer stopped accepting traffic. From the outside, production was simply down.
Green is a snapshot, not a promise
The first lesson is the humbling one. My answer was honest – it was what I saw. But a green dashboard only tells you that the things you measure looked fine the last time they were measured. In this case, a developer noticed before my alerts did.
I don't think the answer is more dashboards. It is checking what users actually experience: synthetic requests that go through the real entry point, from somewhere outside the platform you are watching. When the platform's own network is the problem, monitoring that lives inside it can be the last to know. And when someone asks "is something wrong?", that question is a signal in itself. Treat it as one.
The layer you don't see is the one that bites
Kubernetes as a service sells a clean boundary: the provider runs the platform, you run the workloads. But a cluster is never just Kubernetes. It depends on the cloud underneath – the network, the load balancers, and the cloud APIs used to create, replace and manage nodes. When that layer changes, your cluster changes with it, whether you touched anything or not.
Most of the good habits in our field – infrastructure as code, GitOps, tested upgrades, disruption budgets – protect you against your own changes. None of them protect you against a change in a layer you don't control. That is a different kind of risk, and it deserves its own plan.
Redundancy that shares a fate isn't redundancy
Several nodes, several replicas, even several clusters. If they all sit on the same network and the same load-balancing service from the same provider, one update can take them down together.
The useful question isn't "how many copies do we have?" It is "what do all our copies depend on?" Write that list down: networking, load balancing, DNS, identity, the container registry, the secrets store. Each shared dependency is a place where everything can fail at once.
What I take from it
Understand the provider's change process before you need it. How are platform updates rolled out – all at once, or zone by zone and region by region? How much notice do customers get? What is the rollback plan if an update breaks customer workloads? These are good questions in a procurement, and awkward ones in the middle of an outage.
Make rebuilding somewhere else a tested procedure. If clusters, add-ons and workloads are all code, standing them up in another region or with another provider is possible. Possible isn't the same as practised. And the data is the hard part: backups that live outside the provider are what make a rebuild meaningful.
Keep the traffic switch in your own hands. DNS with sensible TTLs, or an entry point you can repoint, decides whether moving traffic takes minutes or depends on the very platform that is down.
Agree on what happens afterwards. A root-cause analysis, a timeline, and what the provider will change. That belongs in the contract, not in a goodwill email.
The trade-off
A second site with another provider costs money and adds complexity. For many workloads, accepting a few hours of downtime is the right business decision. The mistake is not choosing that – it is never choosing at all, and finding out during the outage what your availability target really was. Decide per workload, and write it down.
This doesn't change my view on European providers
I still believe in running on Swedish and European infrastructure. But sovereignty isn't only about which jurisdiction your data sits in. It is also about being able to keep running – or move – when your provider has a bad day. A sovereign platform without an exit and a fallback is only half sovereign.
And the next time someone asks if something is wrong, I'll check from the outside before I answer.