Skip to content

New: deep research, balance and auto-reload tools in the keyless MCP server. Install in one line

Fix url_not_allowed (400) and fetch_failed (502) errors

HTTP 400 url_not_allowed means ContextFetch will not fetch that address at all: it is not a public http or https URL, or its hostname does not resolve to a public address. HTTP 502 fetch_failed means the address was fine but the site answered with an error such as 404 or 500. /v1/contents reports failed pages per URL in statuses instead of failing the whole call, and failed fetches are not billed.

Verified Python/v1/contents/r/{url}

1. 400 url_not_allowed: the address is not public

ContextFetch only fetches public http and https URLs. It rejects private, loopback and link-local addresses such as 127.0.0.1, 10.0.0.5, 192.168.1.1 and localhost, and it checks again after every redirect, so a public URL that redirects to a private one is rejected too. A hostname that does not resolve at all also gets this error.

Retrying never helps, because the answer will not change. Drop the URL, or if it is an internal page you need, fetch it from inside your own network and pass the text to your agent yourself.

2. 502 fetch_failed: the site returned an error

This comes from the /r/ reader and other single-URL calls when the site itself fails. The message names the cause, for example FetchError: HTTP 404 or FetchError: HTTP 500. Other causes are the fetch limits: at most 5 redirects, a 5 MB body and a 15 second timeout.

  • 404 and 410: the page is gone. Do not retry; search for the current page instead.
  • 500, 502, 503 and timeouts: the site is having a bad moment. Retry once after a short wait.
  • 403: the site refuses automated clients. Find the same information on another source.

3. In /v1/contents, read statuses

/v1/contents takes up to 10 URLs, so one bad URL does not fail the call. The response is HTTP 200, the pages that loaded are in results, and every URL has an entry in statuses: success, or error with an error.message such as HTTP 404. Only pages that load are billed, so a call where every URL failed costs nothing.

4. Handle both in code

Treat 400 as permanent and remove the URL from your queue. Treat a 404 or 410 as permanent too. Retry 5xx errors and timeouts once, and then move on. The program below triggers each case against the live API and prints exactly what comes back, so you can match your error handling to the real responses.

The complete code

Python, runnable as is with CONTEXTFETCH_API_KEY set. We ran it against the live API; the output below is what it printed.

import jsonimport osimport urllib.errorimport urllib.request API = "https://api.contextfetch.dev"KEY = os.environ["CONTEXTFETCH_API_KEY"]HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}  def call(path, body=None):    data = json.dumps(body).encode() if body is not None else None    req = urllib.request.Request(API + path, data=data, headers=HEADERS)    try:        with urllib.request.urlopen(req, timeout=60) as resp:            return resp.status, resp.read().decode()    except urllib.error.HTTPError as err:        return err.code, err.read().decode()  print("1. A private address")print(call("/v1/contents", {"urls": ["http://127.0.0.1/admin"], "text": True})) print("\n2. A page that does not exist, through /v1/contents")status, body = call("/v1/contents", {"urls": ["https://www.python.org/this-page-does-not-exist"], "text": True})res = json.loads(body)print(status, res["statuses"], "cost:", res["costDollars"]["total"]) print("\n3. The same page through the /r/ reader")print(call("/r/https://www.python.org/this-page-does-not-exist"))
OutputRecorded 
1. A private address
(400, '{"error":"url_not_allowed"}')

2. A page that does not exist, through /v1/contents
200 [{'id': 'https://www.python.org/this-page-does-not-exist', 'status': 'error', 'error': {'tag': 'CRAWL_NOT_FOUND', 'message': 'HTTP 404'}}] cost: 0.0

3. The same page through the /r/ reader
(502, '{"error":"fetch_failed","message":"FetchError: HTTP 404"}')

The same call with curl

The raw HTTP request, and the response it returned (long text fields and lists shortened).

curl -s -w "\nHTTP %{http_code}\n" "https://api.contextfetch.dev/r/https://www.python.org/this-page-does-not-exist" \  -H "Authorization: Bearer $CONTEXTFETCH_API_KEY"
ResponseRecorded 
{"error":"fetch_failed","message":"FetchError: HTTP 404"}
HTTP 502

Questions

Am I charged for failed fetches?

No. In /v1/contents only pages that load are billed, and a paid request that fails with a fetch error or a server error is refunded. A 400 url_not_allowed is never billed.

Why does a URL that works in my browser fail?

Some sites answer automated clients differently from browsers, for example with a 403 or a login page, and some addresses are only reachable from inside a private network. ContextFetch fetches from its own servers, so it sees what an outside client sees.

What are the fetch limits?

http and https only, at most 5 redirects, a 5 MB response body and a 15 second timeout per fetch. Private, loopback and link-local addresses are rejected, including after redirects.

Is 503 recall_unavailable the same kind of error?

No. 503 recall_unavailable comes from search, not from fetching a page: the web search backends are briefly unavailable. Retry the search shortly.

Run it with your own key

A free key has 2,500 searches and needs no signup. Agents can claim one themselves with the POST below.

curl -X POST https://api.contextfetch.dev/v1/keys/free
Get a free key

Related guides