WhatBrokeToday
GTA 6FAQ
Kubernetes

Kubelet Incorrect Memory Accounting With Systemd 259 cgroup v2

high

Is it fixed? Not yet.

Ongoing since Nov 13, 16:59 UTC. First reported Nov 13, 16:59 UTC.

What happened

Systemd 259+ enables the memory_hugetlb_accounting cgroup v2 mount option without an opt-out switch, causing hugepages and memory to account against the same pool. Because kubelet tracks memory and hugepages limits separately, it performs incorrect accounting on systems using systemd >= 259.

What caused it?

Systemd 259+ enables memory_hugetlb_accounting combining hugepages and memory into the same accounting pool, which conflicts with kubelet tracking them separately.

No verified workaround yet

None of the linked reports describe a mitigation. This page updates when one appears.

Official fix

Timeline

  1. Investigating(current)
  2. Workaround available(upcoming)
  3. Resolved(upcoming)
  1. First report on GitHub issue

    kubelet support for cgroup v2 `memory_hugetlb_accounting`

    Source

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    kubelet support for cgroup v2 `memory_hugetlb_accounting`(opens GitHub issue in a new tab)

    What happened? systemd 259+ is enabling the memory_hugetlb_accounting cgroup v2 mount option. https://github.com/systemd/systemd/releases/tag/v259 This means the hugepages and memory are now accounting against the same pool. To the best of my knowledge, there's no opt-out switch. When the cloud images/server OSes will pull systemd >= 259, this will affect us. Currently, kubelet tracks memory and hugepages limit separately, therefore it does the accounting wrong onsystems consuming systemd >=…

    by ffromani 0upvotes 12comments

FAQ

What happened in the Kubernetes issue?

Systemd 259+ enables the memory_hugetlb_accounting cgroup v2 mount option without an opt-out switch, causing hugepages and memory to account against the same pool. Because kubelet tracks memory and hugepages limits separately, it performs incorrect accounting on systems using systemd >= 259.

What is affected by the Kubernetes issue?

Kubernetes.

When did the Kubernetes issue start and end?

It started Nov 13, 16:59 UTC and is still ongoing (5 d 11 h so far). This page updates as it develops.

How do I get alerts for Kubernetes?

Subscribe to the Kubernetes RSS feed for every new Kubernetes incident, or connect an AI assistant to WhatBrokeToday over MCP and ask whether Kubernetes is down. Kubernetes feed

Where does this information come from?

WhatBrokeToday collects public status pages, developer community reports and security advisories, groups the ones about the same problem, and links every report under Evidence on this page. This one has 1 report from 1 independent source.