Skip to content

Host health monitor discards exceeded counter names, making "Host is unhealthy" difficult to diagnose #12010

Description

@satvu

Problem

When the periodic host health check trips, the host logs Host is unhealthy. Initiating a restart. but never says which counter exceeded.

IsHostHealthy collects the names into exceededCounters, formats them, and then drops them at line 863 — the Host thresholds exceeded: [...] message only exists in the throwWhenUnhealthy branch. OnHostHealthCheckTimer is the only caller using the default false, so on the periodic path that message is unreachable. No log level or configuration setting can surface it.

Why it matters

The counters (Connections, Sections, Threads, etc.) come from the WEBSITE_COUNTERS_APP environment variable and are not published to Azure Monitor or Application Insights — if the host does not log them, they are unavailable from every direction.

Recent customer case: a Windows Consumption app restarting every 7–11 minutes. Weeks were spent correctly eliminating memory, threads, timeouts, SNAT and sampling, plus raising the log level as guidance suggests. The cause was one counter sitting slightly over the 80% threshold. One log line would have made it a five-minute diagnosis.

Proposed fix

Add a HostThresholdsExceeded message to ScriptHostServiceLoggerExtension at LogLevel.Warning (next free EventId is 532), and call it before return false:

_logger.HostThresholdsExceeded(formattedCounters);

Checks run every 10s; if log volume is a concern, log only when the counter set changes.

Also worth fixing

  • HttpThrottleMiddleware logs different wording for the same condition, so a search for one misses the other.
  • Document the 80% threshold — customers compare against the published limit with no way to know the effective ceiling is 80% of it.

Environment

Observed on host 4.1053.200, Windows Consumption. The code path is long-standing and not specific to that build; it applies to any SKU where ShouldMonitorHostHealth is true.


Written by Azure SRE Agent following a customer incident investigation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions