A pod of mine stopped serving and I found out when someone else told me. Nothing in the cluster said a word, because nothing was watching it. So I put Prometheus and Grafana on a kind cluster, wired a single alert, and killed the pod on purpose to see what actually happens between a pod dying and a person hearing about it. The alert did come, but late enough that a user could still have been first. Then I changed three settings, the same failure was caught almost at once, and I started paying for it in a different way. This post is about that gap, the three settings that make it, and the trade-off you take on when you close it.
What is observability?
Observability is being able to answer "what is happening inside this system, and since when" from the outside, without adding new print statements and redeploying. It usually rests on three kinds of signal:
- Metrics: values sampled over time (request rate, error count, memory,
upordown). Cheap, and what alerts are built on. - Logs: discrete events with detail, for reading after something fires.
- Traces: one request's path across services.
This post is about the metrics half, because that is where alerting lives: Prometheus collects the values, Alertmanager decides when one of them has been bad for long enough to wake someone, and Grafana draws them. On kind, a local cluster, the whole thing runs on a laptop.
The problem, and why it matters
Before this, a pod could stop serving and the first signal was a user. There was no record of when it went down, because nothing was recording "down" at all. You cannot alert on a signal you never collect, and you cannot explain an outage you have no timeline for. The question an on-call person actually needs, at 3am, is "is it down, since when, and did it page me." Without metrics you are guessing at all three.
The second cost is quieter. Even once you collect metrics, the delay between a failure and the alert is not zero, and it is not obvious. It is the sum of three waits most people never look at: how often you scrape, how often you evaluate the rule, and how long you make the rule wait before it is sure. This post walks through each of them, and through what you give up when you shorten them.
What Prometheus and Grafana give you
| In practice | What changed in the lab | |
|---|---|---|
| Automatic target discovery | A ServiceMonitor tells Prometheus what to scrape; new pods are picked up without editing config |
Prometheus found the cluster's own components and my app on its own; I never listed a target by hand |
| An alert that fires on its own | A PrometheusRule watches a metric and routes to Alertmanager |
the dead pod reached Alertmanager without anyone looking at a dashboard; before, a person was the alert |
| Dashboards without wiring | kube-prometheus-stack ships Grafana with cluster dashboards already loaded | Grafana showed live node and pod views the moment the stack came up; I built no panel |
| One install, not five | The operator runs Prometheus, Alertmanager, Grafana, node-exporter and kube-state-metrics together | one helm install, one values file, and the five pieces already knew about each other |
How it works: where the delay comes from
Prometheus does not get pushed events. It pulls: it scrapes each target's /metrics on a fixed interval and stores what it reads. An alert is a query that runs on its own interval, and it only fires after it has been true for a set duration. So a failure has to clear three waits before anyone hears about it, and each wait is a setting you choose.
The lab
The app is a one-pod HTTP service that exposes Prometheus metrics. A ServiceMonitor tells Prometheus to scrape it, and a PrometheusRule says "alert if this target is not up." Then I scale the deployment to zero and watch what the stack does, first with the chart's defaults, then with the three waits shortened.
- The gap is real, and it is three waits. With the defaults, killing the pod played out in slow motion. First Prometheus noticed the target was gone, at its next scrape. Then, at the next rule evaluation, the alert went pending. Then it sat there, pending, while the
fortimer ran, and only after that did it go firing and reach Alertmanager. Nothing was broken. That is simply what three timers in a row feel like from the outside: long enough to refresh a page, frown, and refresh again. A user who was on the app at that moment would have known before the pager did. - Shortening the three waits changed who finds out first. I scraped more often, evaluated the rule more often, and shortened
for. Same kill, same app: the alert reached Alertmanager before I had finished reading the terminal. The gap did not disappear, it cannot, because the pull model always waits for the next scrape. But it moved from "a user could be first" to "the pager is first." The only change was three values in two YAML files, and the bill for it is in the trade-off section below. - Grafana was live with no panels built. kube-prometheus-stack installs Grafana with cluster dashboards already loaded, so the node and pod views showed real data the moment the stack came up. Before the install there was no dashboard at all; after one command there was a live one, and the dead pod showed up on it as a gap in the graph.
Where it broke
1. up == 0 never fired, because the target was gone, not down
My first alert was up == 0. It never fired. When I scaled the deployment to zero, the pod's endpoint disappeared, so Prometheus dropped the target entirely and the up series for that job stopped existing. up == 0 matches a target that is present and failing, not one that is absent. A query over a series that has no samples returns nothing, and nothing is not zero, so the rule stayed quiet. The fix was up == 0 or absent(up{job="sample-app"}): absent() returns a value exactly when the series is missing, which is the case a vanished pod actually produces.
2. Faster is not free
The tuned config is not strictly better than the default one. A shorter for means the alert trusts a shorter blip, so a single missed scrape during a rollout or a one-off network hiccup can page someone for a service that is fine. Shorter scrape and evaluation intervals also mean more queries and more writes, on every target, all day. The right values depend on how bad a false page is for that service, not on making everything as small as it goes.
3. The rule change took a reload, not effect, immediately
Editing the PrometheusRule did not change behaviour the instant I applied it. The operator has to regenerate Prometheus's rule files and Prometheus has to reload them, and that took long enough that my first run after the change was still using the old rule. If you test right after an apply, you are testing the previous config.
Tuning: the three settings, and what each costs
| Setting | What it controls | Shorter means | The cost |
|---|---|---|---|
Scrape interval (ServiceMonitor) |
how often a target is read | failures seen sooner | more scrapes, more stored samples |
| Evaluation interval (rule) | how often the alert query runs | pending sooner | more query load |
for duration (rule) |
how long it must stay true | firing sooner | more false pages on brief blips |
Rule of thumb: pick for from how bad a false page is, not from how fast you want to look. A user-facing API can take a short for; a batch job that flaps is better left with a longer one.
The trade-off, side by side
| Alert fast | Alert slow | |
|---|---|---|
| Plus | the pager is first, not a user; short outages leave a timeline you can explain | a quiet pager; rollouts and blips do not wake anyone; less load on Prometheus |
| Minus | false pages on blips and rollouts; more scrape and query load; on-call learns to ignore the pager | a user can be first; short outages leave no trace; "since when" has a coarse answer |
| Fits | user-facing APIs, anything with a person waiting on it | batch jobs, internal tools, anything that flaps by design |
Neither column is the right one. I run the user-facing app on the left column and the batch job on the right, in the same cluster, and I expect to move things between them as the pager teaches me.
Prometheus vs CloudWatch, briefly
If you came from AWS, the mental model is different. CloudWatch is push and per-metric billing: you pay for custom metrics and for higher resolution, and the standard resolution is fixed and coarse. Prometheus is pull and self-hosted: the resolution is the scrape interval you set, and the cost is the CPU and disk it runs on, not a line item per metric. On kind it is free and local, which is the point of learning it this way before paying for it in a cloud.
When to use which
- Reach for kube-prometheus-stack when you want Prometheus, Alertmanager and Grafana wired together in one install rather than three.
- Define scrape targets with a
ServiceMonitor, not hand-edited config, so new services are picked up on their own. - Write the "is it down" alert as
up == 0 or absent(up{...}), so a vanished pod fires too. - Set
forfrom the cost of a false page, and scrape and evaluation intervals from how fast you truly need to know.
Your call
I picked a short for for the user-facing app and left the batch job slow. That is the right split for my cluster this week; it is not a rule for yours. Two things I would like to read in the comments, because they are the part of this I cannot learn on a laptop:
- Who noticed your last outage first, the pager or a person?
- If you ever tuned alerting faster, what did it cost you: false pages, load, or nothing at all?
I am not after who is right. I want the pluses and minuses of both columns above to be visible to everyone who reads this, in words from real clusters, and the comments are where that happens. If you would rather vote than write, the same question is a poll on my LinkedIn post for this article: https://www.linkedin.com/in/mirac-can-yilmaz
Summary
- Observability is answering "what is happening, and since when" from the outside. Metrics are the half that alerts run on.
- The delay from failure to alert is three waits: scrape interval, evaluation interval, and
for. With defaults a user can be first; shortened, the pager is first. up == 0does not fire for a pod that was removed; useabsent()for the target-gone case.- Faster alerting trades against false pages and load. Pick
forfrom how bad a false page is, and expect to revisit it.
Next in the series
This is post 6 of Core Skills. The previous one, Docker Best Practices: CMD vs ENTRYPOINT and ADD vs COPY, looked at what one word in a Dockerfile does to a container's shutdown; this one looked at how a cluster tells you a pod stopped, and who hears it first. Next: reading the three signals together, logs and traces alongside these metrics, when a single dashboard is not enough.
The code for this lab (the sample app, the ServiceMonitor and PrometheusRule, the kube-prometheus-stack values and the kind manifests) is shared with Lab Access members on Buy Me a Coffee; the repo invite comes by email after you join: https://buymeacoffee.com/miraccanyilmaz/membership
