Web crawler - a frontier, and the trap of crawling forever

Ask these before drawing anything

• How many pages, and how fresh? A billion pages refreshed monthly and a hundred million refreshed hourly are different systems, not the same system with a different number in it.
• Whose web is it - the open web, one customer site, or a set of known domains? A crawl of known domains can negotiate rate limits and skip almost everything on this board.
• Is JavaScript rendering required? A headless browser is roughly fifty times the CPU of an HTTP fetch, and that single answer moves the fleet from twenty machines to a thousand.
• What counts as a duplicate: the same URL, the same bytes, or the same article on two domains? Each needs a different mechanism and they are often all three.
• What is the politeness policy, and who owns the relationship when a site complains? There will be complaints.
• Is the crawl allowed to be stopped and restarted, or must it be continuous? That decides whether the frontier has to be durable.
• What happens to content that disappears - is a 404 a deletion from the corpus or an outage to retry?

Functional

• Crawl one billion pages and revisit each within thirty days.
• Obey robots.txt, including crawl-delay, and cache it per host.
• Never fetch the same URL twice within a refresh cycle, and never store the same content twice regardless of how many URLs served it.
• Discover new URLs from fetched pages and feed them back into the frontier.
• Prioritise: a news homepage is worth more, more often, than a ten-year-old forum thread.
• Survive a restart without losing the frontier or the seen set.

Non-functional

• Politeness is a hard constraint, not a target: at most one in-flight request per host, and honour crawl-delay when it is longer than the default one second.
• Throughput of 400 pages a second sustained, 800 at peak.
• The crawler may be down for an hour without anybody caring. It is a batch system - it degrades by crawling more slowly, which is a completely different availability story from anything serving user requests.
• The frontier and the seen set are the only components whose loss is expensive, and they must be durable.
• Identify honestly in the User-Agent, with a URL explaining the crawler and a contact address.

Out of scope

• Ranking and the index itself. This system fills a queue; somebody else reads it.
• Paywalled or authenticated content.
• Rendering JavaScript, in version one. The number is in the capacity page so that the decision is costed rather than assumed.