What it does
- Fetches a page when an API user searches for something or asks for that page, and returns it to them as Markdown.
- Adds public pages it has seen in search results to our search index in the background.
- Crawls a site on request (crawl and map endpoints), at most 50 pages per request.
robots.txt
Background indexing, crawls and site maps obey robots.txt, including Crawl-delay, and wait at least one second between requests to the same host. A single page fetched because a user asked for it is treated like a browser visit on their behalf.
To keep your site out of our index and crawls:
User-agent: ContextFetchBot
Disallow: /Contact
To report a problem or have pages removed from the index, email hello@contextfetch.dev.