WhatBrokeToday
FAQ
Today's Trends
mediumInvestigatingKubernetes

Kubernetes Uncached OpenAPI v3 Schemas Cause Excessive Heap Allocations

  1. Investigating(current)
  2. Trending(upcoming)
  3. Workaround available(upcoming)
  4. Resolved(upcoming)

Executive summary

Applying manifest updates using kubectl apply causes uncached OpenAPI v3 schemas to unmarshal dozens of times during a single run. This results in hundreds of megabytes of cumulative heap allocations and significant CPU overhead for standard resource applies.

No verified workaround yet

None of the linked reports describe a mitigation. Follow this incident to hear when one appears.

Official fix
Technical root cause

Uncached OpenAPI v3 JSON schemas are repeatedly unmarshaled multiple times per resource during patch and verification operations.

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    Uncached OpenAPI v3 schemas cause 41+ unmarshals on a 20-resource apply(opens GitHub issue in a new tab)

    What happened? Applying updates to an existing 20-resource core/v1 manifest (kubectl apply -f) unmarshals the 1.53 MB api/v1 OpenAPI v3 JSON 41 to 47 times, causing 351–406 MB of cumulative heap allocations (~8.5 MB per unmarshal, ~800–900 ms CPU): 20 ConfigMaps (1 Kind): 41 unmarshals (1 in queryParamVerifierV3 + 40 in Patcher) = 351.6 MB TotalAlloc (785 ms). 20 core/v1 resources across 7 Kinds: 47 unmarshals (7 in queryParamVerifierV3 + 40 in Patcher) = 406.2 MB TotalAlloc (905 ms). 20…

    by seans3 0upvotes 1comments

Timeline

  1. First report on GitHub issue

    Uncached OpenAPI v3 schemas cause 41+ unmarshals on a 20-resource apply

    Source