WhatBrokeToday
FAQ
Today's Trends
mediumInvestigatingKubernetesv1.36

Kubernetes Pod with Ephemeral Local Volume Preempted During PreBind Stays Pinned

  1. Investigating(current)
  2. Trending(upcoming)
  3. Workaround available(upcoming)
  4. Resolved(upcoming)

Executive summary

Since version v1.36, when the scheduler preempts a victim in PreBind in memory, it cancels the binding cycle and requeues the pod without deleting it. For a pod with an unbound WaitForFirstConsumer PVC, VolumeBinding PreBind leaves the selected-node annotation on the PVC which is not reverted by Unreserve, causing the pod to stay pinned to the node.

No verified workaround yet

None of the linked reports describe a mitigation. Follow this incident to hear when one appears.

Official fix
Technical root cause

The scheduler preempts a victim in PreBind in memory and cancels the binding cycle without deleting the pod, while the VolumeBinding PreBind API writes (such as the selected-node annotation on the PVC) are not reverted because Unreserve only reverts the assume cache.

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    Pod with an ephemeral local volume preempted during PreBind stays pinned to the node(opens GitHub issue in a new tab)

    What happened? Since #135502 (v1.36), the scheduler preempts a victim in PreBind in memory. It cancels the binding cycle and puts the pod back in the queue. It does not delete the pod. For a pod with an unbound WaitForFirstConsumer PVC, VolumeBinding PreBind already made API writes before the cancel. It set the volume.kubernetes.io/selected-node annotation on the PVC. The cancel does not revert this write, because Unreserve only reverts the assume cache. The external provisioner completes, and…

    by barkbay 0upvotes 2comments

Timeline

  1. First report on GitHub issue

    Pod with an ephemeral local volume preempted during PreBind stays pinned to the node

    Source