WhatBrokeToday
FAQ
Today's Trends
mediumInvestigatingKubernetes

Kubernetes Device Plugin NUMA Hints Ignore Pending Device Requests

  1. Investigating(current)
  2. Trending(upcoming)
  3. Workaround available(upcoming)
  4. Resolved(upcoming)

Executive summary

Kubernetes device plugin NUMA hints ignore pending device requests from admitted pods when using pod-level resources, pod-scoped Topology Manager, and Pods using pod-level CPU/memory resources. This leads to incorrect NUMA accounting and resource placement issues during consecutive pod admissions.

No verified workaround yet

None of the linked reports describe a mitigation. Follow this incident to hear when one appears.

Official fix
Technical root cause

Device plugin NUMA hints fail to account for pending device requests from admitted pods when PodLevelResourceManagers is enabled with pod-scoped Topology Manager.

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    Device Plugin NUMA hints ignore pending device requests from admitted pods with pod-level resources(opens GitHub issue in a new tab)

    Device Plugin NUMA hints ignore pending device requests from admitted pods with pod-level resources What happened? KevinTMtz pointed out this NUMA accounting problem during review of my PR #143494. I subsequently reproduced and verified the behavior; I did not originally identify it. KevinTMtz's original report My follow-up reproduction and A/B test results With PodLevelResourceManagers=true, pod-scoped Topology Manager, and Pods using pod-level CPU/memory resources, consecutive Pod admissions…

    by zl86790 0upvotes 3comments

Timeline

  1. First report on GitHub issue

    Device Plugin NUMA hints ignore pending device requests from admitted pods with pod-level resources

    Source