WhatBrokeToday
FAQ
Today's Trends
minorInvestigatingKubernetes

Kubernetes cluster/log-dump Fails to Report GCE Node Failures Due to Deprecated Logs

  1. Investigating(current)
  2. Trending(upcoming)
  3. Workaround available(upcoming)
  4. Resolved(upcoming)

Executive summary

The detect_node_failures script in Kubernetes cluster/log-dump queries deprecated GCE activity logs, causing host errors and automatic restarts to go unreported. The issue stems from relying on the legacy Compute Engine activity log format.

No verified workaround yet

None of the linked reports describe a mitigation. Follow this incident to hear when one appears.

Official fix
Technical root cause

detect_node_failures in cluster/log-dump/log-dump.sh queries the legacy Compute Engine activity log (compute.googleapis.com%2Factivity_log), which is deprecated and results in host errors never being reported.

Evidence

1 report · 1 GitHubShow
  1. [1]
    GitHub First report
    cluster/log-dump: detect_node_failures queries deprecated GCE activity logs, so host errors are never reported(opens GitHub issue in a new tab)

    What happened? detect_node_failures in cluster/log-dump/log-dump.sh looks for host errors and automatic restarts in the legacy Compute Engine activity log: gcloud logging read --order=asc \ --format='table(timestamp,jsonPayload.resource.name,jsonPayload.event_subtype)' \ --project "${PROJECT}" \ "resource.type=\"gce_instance\" logName=\"projects/${PROJECT}/logs/compute.googleapis.com%2Factivity_log\" (jsonPayload.event_subtype=\"compute.instances.hostError\" OR…

    by anshulchikhale30-p 0upvotes 3comments

Timeline

  1. First report on GitHub issue

    cluster/log-dump: detect_node_failures queries deprecated GCE activity logs, so host errors are never reported

    Source