The Monitoring Model That No Longer Works
Traditional infrastructure monitoring was designed for a world that enterprise Kubernetes has made obsolete. In that world, hosts were durable and inventoried. Applications ran on known servers at known addresses. When something went wrong, the signal was usually a host metric: CPU saturation, memory pressure, disk exhaustion. The monitoring system tracked the host, and the host tracked the application.
Kubernetes changes each of these assumptions. Pods are ephemeral. The host that ran a specific pod ten minutes ago may be running a different pod now, and the pod that served a specific request may have been evicted and replaced. Workloads scale horizontally, so application behaviour emerges from the collective performance of dozens or hundreds of pod instances rather than from a single identifiable server. Service dependencies create failure cascades where an upstream degradation manifests as a downstream error in a way that no host metric can capture.
Monitoring tools built for the previous world apply metrics, alerting, and dashboards designed for durable infrastructure to ephemeral workload patterns. The result is an observability programme that tells you a host is healthy while the application it is running is failing requests, or that tells you an application is up while its dependencies are degraded in ways that make it functionally unavailable for specific user journeys.
This is not a tooling problem that a new monitoring agent solves. It requires a different observability architecture.
The Three Instrumentation Layers Kubernetes Observability Requires
Genuine observability for Kubernetes workloads requires instrumentation at three distinct layers, each providing information that the others cannot.
The cluster infrastructure layer covers the health of the Kubernetes infrastructure itself: node availability, resource utilisation and headroom, control plane performance, and the scheduling and eviction activity that reflects how well the cluster is managing workload placement. These metrics tell you whether the cluster can run the workloads it is asked to run, and whether it is approaching the capacity limits that would affect scheduling decisions. They are necessary but not sufficient. A healthy cluster can be running unhealthy applications.
The service layer covers the health of services as observed through their network behaviour: request rates, error rates, and latency distributions for traffic flowing between services. In a microservices architecture, this layer is where the consequences of dependency failures become visible. A database that is responding slowly creates elevated response latency in every service that queries it, and that latency propagates through service call chains to produce user-visible degradation that no single service’s internal metrics fully explain. The service layer is where these propagation patterns become observable.
The application layer covers the health of the application as understood by the application itself: business transaction success rates, operation durations, error categories and frequencies, and the business-specific health signals that infrastructure and network metrics cannot capture. A payment processing service that is successfully returning HTTP 200 responses but is failing all payment authorisations is healthy at the network layer and catastrophically unhealthy at the application layer. Instrumentation at this layer, typically through application performance monitoring or custom business metrics, is what makes the distinction visible.
The observability programme that covers only the first layer knows whether the infrastructure is running. The one that covers all three knows whether the application is delivering its intended value.
Correlation as the Core Capability
Collecting metrics, logs, and traces from three instrumentation layers produces a large volume of observability data. The capability that converts this data into operational insight is correlation: the ability to connect signals from different layers and different services into a coherent picture of a specific incident or degradation.
When a user experiences a failed transaction, the observable evidence is distributed across multiple systems. The application log shows a timeout exception. The service metric shows elevated latency to a downstream dependency. The distributed trace shows the specific call chain that timed out and the specific service in that chain where time was spent. The infrastructure metric shows the node where the relevant pod was scheduled was approaching memory pressure at the time of the request. No single signal is sufficient to explain the failure. The correlation of signals across layers and services makes the explanation visible.
Distributed tracing is the instrumentation capability that makes this correlation tractable. A trace that follows a request across multiple services, capturing timing and context at each hop, provides the backbone against which infrastructure and application signals can be correlated. The engineering investment in trace instrumentation, which requires adding trace context propagation to application code and service communication, is the investment that makes the correlation layer functional.
Without distributed tracing, incident diagnosis in a microservices environment devolves into the manual correlation of signals from separate tools by engineers who know the system architecture well enough to construct the likely failure sequence from partial evidence. That approach works when the systems are small and the engineers are experienced. It does not scale.
Alerting Philosophy for Distributed Systems
The alerting model that monitoring vendors ship by default was designed for infrastructure monitoring. It alerts on metric threshold crossings: CPU above eighty percent, memory above ninety percent, disk above ninety-five percent. For distributed systems at enterprise scale, this alerting model generates enormous noise while missing the failures that matter.
The alternative is symptom-based alerting: alert on the observable symptoms that indicate user impact rather than on the infrastructure conditions that may or may not be causing that impact. The canonical symptom-based alerting framework for service reliability is the Four Golden Signals: latency, traffic, errors, and saturation. Alerts on latency and error rate provide early warning of user-visible degradation. Alerts on saturation approaching limits provide advance warning of resource constraints before they cause symptoms. Traffic anomalies provide signals that something unexpected is happening even before symptoms are visible.
What this approach eliminates is the large volume of infrastructure alerts that reflect normal variability in a distributed system without indicating user impact. A pod restart is a normal event in Kubernetes. An alert on a pod restart tells on-call engineers that something happened without telling them whether that something matters. An alert on elevated error rate in the service that pod was part of tells them something the user is experiencing. The first alert trains people to ignore alerts. The second alert demands attention.
On-Call Practices That Match the Architecture
The on-call practices that work for monolithic applications break for distributed systems because they rely on institutional knowledge that does not scale. The engineer who responded to the last fifty incidents knows from experience what the alert pattern means and where to look. As the system grows in complexity and the team grows in headcount, this knowledge becomes a bottleneck and a single point of failure.
Runbooks encoded in muscle memory need to become runbooks encoded in documentation and tooling. Diagnostic steps that an experienced engineer performs from memory need to be captured as structured investigation guides that a less experienced on-call engineer can follow without previous incident context. Alert annotations that explain what the alert means, what it is likely to be caused by, and what the first diagnostic steps are reduce the time-to-insight in an incident without requiring the on-call engineer to remember what they learned the last time this alert fired.
The architecture of the on-call practice should match the architecture of the system it is supporting. A distributed system requires a distributed observability practice: clear ownership of services and their alerts, escalation paths that match the service dependency structure, and communication norms that allow parallel investigation of a cascading failure without stepping on each other.
What the Investment Justifies
The observability infrastructure described above requires investment: in instrumentation libraries, in distributed tracing platforms, in correlation tooling, in the engineering time to add trace context propagation and business metric emission to applications. This investment is frequently undervalued in architecture and platform budget conversations because its returns are measured in time saved during incidents rather than in capabilities added to the product.
The calculation that makes the case is straightforward. The mean time to resolution for a production incident in a system without the correlation capabilities above is measured in hours. With them, it is measured in minutes. The fully loaded cost of an engineer-hour for the senior engineers typically involved in production incidents is significant. The revenue impact of customer-facing degradation during a multi-hour incident is larger still. The observability investment that reduces mean time to resolution from four hours to thirty minutes, across the incident frequency typical of a large cloud-native estate, pays back within the first year in reduced incident cost alone. The value of preventing incidents that the observability investment makes visible before they become user-facing is additional.
The question is not whether the investment is justified. It is whether the organisation making the investment calculation has made the full cost of the alternative visible.