Kubernetes Kubelet Re-Runs Terminated Containers After Runtime Container Removal
Is it fixed? Not yet.
Ongoing since Oct 31, 13:49 UTC. First reported Oct 31, 13:49 UTC.
What happened
Kubelet incorrectly re-runs containers that terminated successfully with exit code 0 under restartPolicy 'Never' or 'OnFailure' approximately one second after their exited runtime container is removed via crictl rm. This causes container commands and side effects to execute a second time on live pods where another container is still running.
What caused it?
Under active investigation. No report has identified a root cause yet.
No verified workaround yet
None of the linked reports describe a mitigation. This page updates when one appears.
Timeline
- Investigating(current)
- Workaround available(upcoming)
- Resolved(upcoming)
- First report on GitHub issue
kubelet re-runs terminated regular containers (Never / OnFailure exit 0) after their runtime container is removed
Source
Evidence
1 report · 1 GitHubShow
- [1]GitHub First reportkubelet re-runs terminated regular containers (Never / OnFailure exit 0) after their runtime container is removed(opens GitHub issue in a new tab)
/kind bug /sig node /area kubelet What happened? On a live pod (another container still running), a container that had terminated successfully (exit code 0) under restartPolicy: Never — and, in a second test case, under restartPolicy: OnFailure — was re-started by kubelet ~1 second after its exited runtime container was removed (crictl rm <id>). The container's command executed a second time (side effects re-ran), visible via a run-counter on a shared emptyDir volume and the new container's…
by zl86790 0upvotes 10comments
FAQ
What happened in the Kubernetes issue?
Kubelet incorrectly re-runs containers that terminated successfully with exit code 0 under restartPolicy 'Never' or 'OnFailure' approximately one second after their exited runtime container is removed via crictl rm. This causes container commands and side effects to execute a second time on live pods where another container is still running.
What is affected by the Kubernetes issue?
Kubernetes.
When did the Kubernetes issue start and end?
It started Oct 31, 13:49 UTC and is still ongoing (18 d 14 h so far). This page updates as it develops.
How do I get alerts for Kubernetes?
Subscribe to the Kubernetes RSS feed for every new Kubernetes incident, or connect an AI assistant to WhatBrokeToday over MCP and ask whether Kubernetes is down. Kubernetes feed
Where does this information come from?
WhatBrokeToday collects public status pages, developer community reports and security advisories, groups the ones about the same problem, and links every report under Evidence on this page. This one has 1 report from 1 independent source.