WhatBrokeToday
GTA 6FAQ
Kubernetes

Kubernetes Kubelet Restart Kills Running Pods Using Device Plugin Resources

high

Is it fixed? Not yet.

Ongoing since Nov 16, 18:44 UTC. First reported Nov 16, 18:44 UTC.

What happened

When the kubelet restarts, running pods configured with a RuntimeClass that sets overhead.podFixed to an extended resource advertised by a device plugin are incorrectly rejected at re-admission and terminated.

What caused it?

Pods utilizing device plugin resources through pod overhead fail re-admission checks when the kubelet restarts.

No verified workaround yet

None of the linked reports describe a mitigation. This page updates when one appears.

Official fix

Timeline

  1. Investigating(current)
  2. Workaround available(upcoming)
  3. Resolved(upcoming)
  1. First report on GitHub issue

    kubelet: restart kills running pods that consume a device plugin resource through pod overhead

    Source

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    kubelet: restart kills running pods that consume a device plugin resource through pod overhead(opens GitHub issue in a new tab)

    What happened? A RuntimeClass can set overhead.podFixed to an extended resource that a device plugin advertises. When kubelet restarts, every running pod of that RuntimeClass is rejected at re-admission and killed. What did you expect to happen? Pods that are running when kubelet restarts should be re-admitted, as they are when they request the same resource in a container. New pods should still wait until the plugin re-registers. How can we reproduce it (as minimally and precisely as…

    by tzneal 0upvotes 2comments

FAQ

What happened in the Kubernetes issue?

When the kubelet restarts, running pods configured with a RuntimeClass that sets overhead.podFixed to an extended resource advertised by a device plugin are incorrectly rejected at re-admission and terminated.

What is affected by the Kubernetes issue?

Kubernetes.

When did the Kubernetes issue start and end?

It started Nov 16, 18:44 UTC and is still ongoing (2 d 9 h so far). This page updates as it develops.

How do I get alerts for Kubernetes?

Subscribe to the Kubernetes RSS feed for every new Kubernetes incident, or connect an AI assistant to WhatBrokeToday over MCP and ask whether Kubernetes is down. Kubernetes feed

Where does this information come from?

WhatBrokeToday collects public status pages, developer community reports and security advisories, groups the ones about the same problem, and links every report under Evidence on this page. This one has 1 report from 1 independent source.