Point the crawler at a domain
Index a site
Give it an eTLD+1 and it starts at the homepage, follows the internal graph rather than reading a sitemap, mints a property for every subdomain and language it finds, and converts what it fetches to clean markdown. Sitemaps still get read — they are one cheap request and the only way to find an orphan — but a URL only ranks once the graph confirms it.
The second field is the page budget — the hard cap on what this crawl may fetch.
Plain fetch clears most blogs. A bot wall is an escalation through five more rungs, not a failure — and page_runs records which one landed it.
Every fetch goes through one host throttle across the whole fleet. Ten containers on one host without it is a small DDoS with our name on it.
Coverage is a view with no write path, so it cannot drift. Every known URL lands in exactly one bucket with a reason on it, and unaccounted must be zero.