WhatBrokeToday
FAQ
Today's Trends
mediumInvestigatingKubernetesv1.37+

Kubernetes Pods Rejected by Missing CSI Driver Not Requeued Promptly on Registration

  1. Investigating(current)
  2. Trending(upcoming)
  3. Workaround available(upcoming)
  4. Resolved(upcoming)

Executive summary

Pods rejected with "CSI driver is not installed on the node" due to preventPodSchedulingIfMissing: true are not automatically retried when the driver subsequently registers without a volume limit. Affected pods remain in a Pending state for up to several minutes until the scheduler flushes unschedulable pods.

No verified workaround yet

None of the linked reports describe a mitigation. Follow this incident to hear when one appears.

Official fix
Technical root cause

When preventPodSchedulingIfMissing is enabled (VolumeLimitScaling, enabled by default in 1.37), NodeVolumeLimits rejects pods referencing an unregistered CSI driver, and subsequent driver registration fails to trigger a pod retry, leaving them stuck until the unschedulable pods duration timeout expires.

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    Pods rejected with "CSI driver is not installed on the node" are not requeued when the driver registers without a volume limit(opens GitHub issue in a new tab)

    What happened? With preventPodSchedulingIfMissing: true on a CSIDriver (VolumeLimitScaling, on by default in 1.37), NodeVolumeLimits rejects a Pod that uses a volume of that driver with <driver> CSI driver is not installed on the node while the driver isn't registered on the node. When the driver then registers, the Pod isn't retried. It stays Pending until the scheduler flushes unschedulable Pods after podMaxInUnschedulablePodsDuration (5 minutes). We saw Pods scheduled 4m16s to 4m56s after…

    by yerzhan7 0upvotes 2comments

Timeline

  1. First report on GitHub issue

    Pods rejected with "CSI driver is not installed on the node" are not requeued when the driver registers without a volume limit

    Source