WhatBrokeToday
GTA 6FAQ
Kubernetes

Kubernetes StatefulSet Controller Memory Leak from Undeleted Metrics

medium

Is it fixed? Not yet.

Ongoing since Nov 18, 23:37 UTC. First reported Nov 18, 23:37 UTC.

What happened

The Kubernetes StatefulSet controller fails to delete unavailable metric series when a StatefulSet is deleted, causing the Kubernetes Controller Manager (KCM) memory to leak. Affected operators experience memory growth in KCM when StatefulSets are frequently created and destroyed.

What caused it?

Under active investigation. No report has identified a root cause yet.

No verified workaround yet

None of the linked reports describe a mitigation. This page updates when one appears.

Timeline

  1. Investigating(current)
  2. Workaround available(upcoming)
  3. Resolved(upcoming)
  1. First report on GitHub issue

    StatefulSet controller doesn't deletes unavailable metric series causing the KCM memory to leak

    Source

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    StatefulSet controller doesn't deletes unavailable metric series causing the KCM memory to leak(opens GitHub issue in a new tab)

    What happened? The StatefulSet controller reports two gauges with one series per StatefulSet: statefulset_controller_statefulset_max_unavailable{statefulset_namespace, statefulset_name, pod_management_policy} statefulset_controller_statefulset_unavailable_replicas{statefulset_namespace, statefulset_name, pod_management_policy} These series are never deleted. When a StatefulSet is deleted, sync finds it missing from the lister, logs "StatefulSet has been deleted" and returns, leaving both…

    by kmala 0upvotes 2comments

FAQ

What happened in the Kubernetes issue?

The Kubernetes StatefulSet controller fails to delete unavailable metric series when a StatefulSet is deleted, causing the Kubernetes Controller Manager (KCM) memory to leak. Affected operators experience memory growth in KCM when StatefulSets are frequently created and destroyed.

What is affected by the Kubernetes issue?

Kubernetes.

When did the Kubernetes issue start and end?

It started Nov 18, 23:37 UTC and is still ongoing (5 h 7 min so far). This page updates as it develops.

How do I get alerts for Kubernetes?

Subscribe to the Kubernetes RSS feed for every new Kubernetes incident, or connect an AI assistant to WhatBrokeToday over MCP and ask whether Kubernetes is down. Kubernetes feed

Where does this information come from?

WhatBrokeToday collects public status pages, developer community reports and security advisories, groups the ones about the same problem, and links every report under Evidence on this page. This one has 1 report from 1 independent source.