# stealpage

AI web scraping API for agents, with search across everything it scrapes.

A web scraping API built for agents. Point it at a domain and it crawls every page — blog, docs, pricing, changelog — converts them to clean markdown, scores what each one actually is, and refreshes on a schedule. Then search across all of it at once, by meaning or by exact words, over MCP or HTTP.

Currently 146 pages across 9 companies in 89 language(s).

Sibling to stealshot.com, which does the same for launch films frame by frame.

## Why query this instead of fetching the page yourself

- The markdown is already extracted and cleaned. No boilerplate, no navigation.
- Every page is scored on axes you can filter by: technicality, depth,
  originality, and how much of it is a sales pitch. "Technical content" is not
  a text query — the word "technical" appears in almost no technical article.
- We keep every crawl, so you can ask what a page said last month.
- Coverage is stated honestly. If we have 300 of a site's 1,100 pages the
  response says so, and says which bucket the other 800 are in.

## MCP

  claude mcp add --transport http stealpage https://stealpage.com/api/mcp

Tools:
- search_content: Hybrid search across the whole corpus, or one company.
- list_content: Faceted list — technicality, format, audience, funnel stage.
- get_page: One page's markdown plus everything we scored about it.
- get_domain: A company: coverage, health, languages, sitemaps, recent changes.
- index_domain: Crawl a domain we do not have yet.
- job_status: Poll a crawl.

## HTTP

GET /api/content
  Faceted list. Every filter is a column scored at index time, not a text match.
  curl "https://stealpage.com/api/content?technicality_min=4&promotional_max=2&sort=cited"

GET /api/content/:id
  One page: markdown, facets, assets with alt text, screenshots, crawl history, and who cites it.
  curl https://stealpage.com/api/content/pg_abc123

GET /api/search
  Hybrid vector + full text over page chunks, fused on rank. Filters apply inside both engines.
  curl "https://stealpage.com/api/search?q=retrying+failed+webhooks&technicality_min=4"

GET /api/domains/:root
  Coverage, health, properties per language, their sitemaps, what changed, who they link to.
  curl https://stealpage.com/api/domains/stripe.com

POST /api/crawl
  Index a domain. Returns a job id immediately — a crawl is minutes to hours, so nothing is held open.
  curl -X POST https://stealpage.com/api/crawl -H 'content-type: application/json' -d '{"domain":"stripe.com","budget":500}'

GET /api/jobs/:id
  Live status: indexed, pending, discovered, elapsed, pages/min, eta, and the coverage buckets.
  curl https://stealpage.com/api/jobs/job_abc123

## Reading a response

- coverage.unaccounted is always 0. Every known URL is in exactly one bucket.
- inlinks_body counts body citations only; template links are excluded. Raw
  inlinks ranks /login top on every site on the web.
- estimate is a range with a confidence, never a bare number.
- state 'skipped' with skip_reason 'excluded-type' means we found the page and
  deliberately did not fetch it (login, legal, contact). Not a failure.