WhatBrokeToday
FAQ
Today's Trends
highInvestigatingKubernetesv1.36, 1.37

Kubernetes 1.36 and 1.37 PodResources API Breaks Device Allocation at Sandbox Creation

  1. Investigating(current)
  2. Trending(upcoming)
  3. Workaround available(upcoming)
  4. Resolved(upcoming)

Executive summary

In Kubernetes 1.36 and 1.37, legacy device-plugin allocation is deferred until container creation time rather than sandbox creation. This change causes issues for PodResources API consumers that require devices at sandbox creation, such as Kata GPU cold-plug workflows.

No verified workaround yet

None of the linked reports describe a mitigation. Follow this incident to hear when one appears.

Official fix
Technical root cause

Device Manager does not support pod-level resource allocation in these versions, making AllocatePod a no-op so that the device plugin's Allocate RPC only fires when the container requesting the device is created.

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    Device allocation deferred to container start breaks PodResources API consumers that need devices at sandbox creation (e.g. Kata GPU cold-plug)(opens GitHub issue in a new tab)

    What happened? In Kubernetes 1.36 and 1.37, legacy device-plugin allocation (including nvidia.com/pgpu / nvidia.com/gpu) is deferred until container creation time. In pkg/kubelet/cm/devicemanager/manager.go (v1.37.0), AllocatePod is a no-op: func (m *ManagerImpl) AllocatePod(logger klog.Logger, pod v1.Pod, lifecycle.Operation) error { // Device Manager does not support pod level resource allocation. // ... } The device plugin’s Allocate RPC only fires when the container that requests the…

    by akowalsk 0upvotes 5comments

Timeline

  1. First report on GitHub issue

    Device allocation deferred to container start breaks PodResources API consumers that need devices at sandbox creation (e.g. Kata GPU cold-plug)

    Source