WhatBrokeToday
Today's Trends
mediumInvestigatingKubernetes

Kubernetes ResourceClaim Updates Fail Due to Non-Canonical JSON Opaque Parameters

  1. Investigating(current)
  2. Trending(upcoming)
  3. Workaround available(upcoming)
  4. Resolved(upcoming)

Executive summary

ResourceClaim updates fail with a 'spec: field is immutable' error when opaque parameters are not in canonical JSON format, such as when created via the Python client. The issue occurs because the apiserver re-encodes JSON fields, causing a byte mismatch during the immutable spec check.

No verified workaround yet

None of the linked reports describe a mitigation. Follow this incident to hear when one appears.

Official fix
Technical root cause

The client sends non-canonical JSON bytes that are stored by etcd, but the apiserver re-encodes them upon return or patch application. Because the immutable spec check compares raw bytes, the mismatch causes the update to be rejected.

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    ResourceClaim updates fail with "spec: field is immutable" when opaque parameters are not canonical JSON(opens GitHub issue in a new tab)

    What happened? A ResourceClaim created with the Python client (kubernetes 37.0.2) cannot be labeled afterwards: $ kubectl label resourceclaim claim-py x=y The ResourceClaim "claim-py" is invalid: spec: Invalid value: null: field is immutable The client sends "parameters": {"a": "b", "z": 1}, and etcd stores those bytes. When the apiserver returns JSON or applies a patch, it re-encodes them as {"a":"b","z":1}. The immutable check on spec compares bytes, so the update is rejected. kubectl…

    by thc1006 0upvotes 5comments

Timeline

  1. First report on GitHub issue

    ResourceClaim updates fail with "spec: field is immutable" when opaque parameters are not canonical JSON

    Source