Log aggregation - the index is bigger than the data

Ask these before drawing anything

• What question are the logs for? Debugging one request, auditing who did what, and counting how often something happens are three different systems. The third one should be a metric and using logs for it is the most expensive mistake in this category.
• How long must a line be searchable, and how long must it merely EXIST? These are different numbers with a thirteen-to-one cost ratio between them, and treating them as one number is how a log bill becomes the largest line item in an infrastructure budget.
• Is any of this subject to audit or legal hold? If yes, deletion stops being a cost optimisation and becomes a compliance operation, with an approval and a record.
• Do logs contain personal data or secrets? They do. Everyone says no and everyone is wrong - a request body logged on error, a token in a URL, an email in a stack trace. Whether redaction happens at the agent or never happens is decided here.
• How much does volume rise during an incident? Nobody measures this until the first time the log system falls over during an outage. Five to twenty times is the usual answer.
• Who pays? If it is a central platform budget, every team logs everything. If it is charged back per team, volume drops by half in a month without a single engineering change.

Functional

• Collect from every process on every node without the application knowing how, and without a slow collector ever blocking the application.
• Enrich every line with its service, pod, region, deploy version and trace id at collection time - not at query time, when the pod that produced it no longer exists.
• Make a line searchable within 10 seconds of it being written, at p95.
• Full-text search plus structured field filters over any time range within the hot window.
• Keep an unindexed, compressed copy of EVERY line - including sampled-out ones where the policy allows - for one year, retrievable by time and source.
• Apply sampling and drop rules centrally, with a record of what was dropped and by which rule.
• Evaluate saved queries as alerts every 60 seconds.

Non-functional

• Ingest must never block a producing application. The agent writes to local disk and a full disk drops lines rather than applying backpressure into the process - a logging system that can stall a production service is worse than no logging system.
• Search p95 under 3 seconds over a 24 hour window; under 30 seconds over the full hot window.
• Survive a 20x volume spike for 30 minutes without losing a line. This is the requirement that sizes the transport queue and the node buffer, and it is the one that gets cut first and regretted first.
• Tenant isolation on every query: one team must not read another team's logs, and the filter is applied by the search API, never by the UI.
• Cost per GB ingested is a published number that teams can see against their own volume.

Out of scope

• Metrics. Counting things with a log query works until it costs a hundred times what a counter would and is a hundred times slower.
• Distributed tracing. The trace id is carried here so the two systems can be joined, but spans belong somewhere built for them.
• Long-term analytics over logs. A year of archive is for answering a specific question about a specific past moment, not for a quarterly report.