Kubernetes Kubelet Fails to Reconcile Partial Cgroup Configuration Failures
Is it fixed? Not yet.
Ongoing since Nov 12, 05:06 UTC. First reported Nov 12, 05:06 UTC.
What happened
Kubelet creates the pod cgroup directory and configures it in two separate steps, but failures during cgroup configuration are only logged. Subsequent SyncPod iterations only check for the directory's existence and never reapply the configuration, leaving partial failures unmitigated.
What caused it?
Kubelet gates pcm.EnsureExists on !pcm.Exists(pod) during SyncPod. If cgroup configuration fails after the directory is created, subsequent syncs only check if the cgroup exists and skip reapplying the configuration.
No verified workaround yet
None of the linked reports describe a mitigation. This page updates when one appears.
Official fixTimeline
- Investigating(current)
- Workaround available(upcoming)
- Resolved(upcoming)
- First report on GitHub issue
Kubelet doesn't reconcile partial failures configuring pod cgroups
Source
Evidence
1 report · 1 GitHubShow
- [1]GitHub First reportKubelet doesn't reconcile partial failures configuring pod cgroups(opens GitHub issue in a new tab)
/kind bug Problem Kubelet creates the pod cgroup directory and configures it in 2 separate steps. Failures during cgroup configuration are only logged, and subsequent SyncPod iterations only check for the existence of the cgroup, so Kubelet never reapplies the configuration. <details><summary>Full Explanation (AI summary)</summary> <p> When the kubelet sets up a pod's cgroup in SyncPod, it gates pcm.EnsureExists on !pcm.Exists(pod), and podContainerManagerImpl.EnsureExists likewise returns…
by tallclair 1upvotes 5comments
FAQ
What happened in the Kubernetes outage?
Kubelet creates the pod cgroup directory and configures it in two separate steps, but failures during cgroup configuration are only logged. Subsequent SyncPod iterations only check for the directory's existence and never reapply the configuration, leaving partial failures unmitigated.
What is affected by the Kubernetes outage?
Kubernetes.
When did the Kubernetes outage start and end?
It started Nov 12, 05:06 UTC and is still ongoing (6 d 23 h so far). This page updates as it develops.
How do I get alerts for Kubernetes?
Subscribe to the Kubernetes RSS feed for every new Kubernetes incident, or connect an AI assistant to WhatBrokeToday over MCP and ask whether Kubernetes is down. Kubernetes feed
Where does this information come from?
WhatBrokeToday collects public status pages, developer community reports and security advisories, groups the ones about the same problem, and links every report under Evidence on this page. This one has 1 report from 1 independent source.