Metrics and monitoring - cardinality is the thing that kills you

Ask these before drawing anything

• How many machines, processes and containers, and how fast does that number change? A fleet that redeploys twenty times a day creates a new pod id every time, and if pod id is a label, every deploy is a cardinality event.
• Who is allowed to define a metric, and does anyone review it? If any engineer can add a label in a pull request nobody from this team reads, the limiter is not a safeguard, it is the review process.
• What is the alerting latency anybody actually needs? The floor is the scrape interval plus the evaluation interval. Asking for five seconds means scraping every five seconds, which is four times everything.
• How far back do people query, and at what resolution? "A year" and "a year at 15-second resolution" differ by a factor of two hundred and forty in storage.
• Is this multi-tenant? If yes, one tenant can take out every other tenant, and per-tenant limits stop being nice-to-have.
• Are these metrics or are they events? If somebody wants to know which user hit the error, they want logs or traces, not metrics, and giving them a label is how the system dies.

Functional

• Scrape every instance every 15 seconds and store every sample with its timestamp and its full label set.
• Answer range queries over any time window with aggregation, rate and quantile functions.
• Evaluate alerting rules every 30 seconds and notify when a condition has held for a stated duration.
• Downsample automatically into a 5-minute and a 1-hour tier and expire the raw samples on schedule.
• Enforce a per-tenant active series limit, reject over it, and make the rejection visible to the tenant rather than only to the platform team.
• Let anyone see the cardinality of their own metrics, by metric and by label, without asking the platform team.

Non-functional

• The monitoring system must be more available than the systems it monitors - 99.9 percent - and must not share their dependencies. A metrics stack that runs on the cluster it watches is blind during exactly the outage that matters.
• Query p95 under 1 second for a 6 hour window, under 10 seconds for a 30 day window.
• Ingest must not be affected by query load. One expensive query must not delay a write, which is why the read and write paths are separate fleets rather than one process.
• An alert fires within 60 seconds of its condition becoming true, and that number is the scrape interval plus the evaluation interval, not a target anyone can tune.
• No single tenant may consume more than 10 percent of cluster memory, enforced rather than requested.

Out of scope

• Logs and traces. They answer "which request" and metrics answer "how many", and the most expensive mistake in this whole category is using one for the other.
• Long-term analytics over metrics. Two years of hourly rollups is for capacity planning, not for a business report.
• Anomaly detection. Worth having, worth building second, and it changes none of the arithmetic here.