API reference

One key, plain JSON, one fast endpoint. Turn any URL into clean Markdown, search the web, or pull typed JSON out of a page. Every call is authenticated with your API key and returns in one round trip.

Base URL https://gate.crawl4ai.com  ·  Auth header Authorization: Bearer sk_live_...  ·  get a key

POST/scrape

Fetch a page and return clean Markdown (and/or HTML). Handles JS-heavy pages and bot walls automatically — you don't pick an engine.

curl https://gate.crawl4ai.com/scrape \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "format": "both",
    "proxy": "residential",
    "country": "us",
    "parse": { "links": true, "media": true, "metadata": true, "tables": true }
  }'

Body

GET/search

Web search, browser-free, results ranked and cleaned. Add rich=1 to also get the on-page extras — follow-up questions, related queries, an entity card and more (see below). For a direct answer, use /answer.

curl "https://gate.crawl4ai.com/search?q=rust+web+crawler" \
  -H "Authorization: Bearer sk_live_..."

Query

Rich response rich=1

Every key below is optional — each appears only when that block is on the page. For a direct answer to a question, use /answer.

{
  "results": [ … ranked links, same as always … ],
  "rich": {
    "follow_up_questions": [ "Why is the sky blue at sunset?" ],
    "related_queries":    [ "why is the ocean blue" ],
    "entity":  { "title": "Apple Inc", "subtitle": "NASDAQ: AAPL" },
    "videos": [ { "title": "…", "url": "https://…" } ],
    "news":   [ { "title": "…", "url": "https://…" } ],
    "discussions":  [ … ],
    "did_you_mean": "corrected spelling"
  }
}

GET/answer experimental

Ask a question, get a direct answer. Some questions won't have one yet — then answered is false (use /search for links). Add deep=0 for a direct answer only when one is readily available (no page reading). Experimental: the shape may change as we improve it.

curl "https://gate.crawl4ai.com/answer?q=why+is+the+sky+blue" \
  -H "Authorization: Bearer sk_live_..."

{
  "answered": true,
  "answer": {
    "kind": "generated",
    "text": "The sky is blue because Earth's atmosphere scatters sunlight…",
    "sources": [ { "title": "NASA", "url": "https://…" } ]
  },
  "experimental": true
}

POST/extract

Pull structured, typed data out of a page with an instruction and/or a JSON schema. Give a URL (we fetch it) or your own content.

curl https://gate.crawl4ai.com/extract \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com",
    "instruction": "the top stories on the front page",
    "schema": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "title":  { "type": "string"  },
          "points": { "type": "integer" },
          "url":    { "type": "string"  }
        },
        "required": ["title", "points"]
      }
    },
    "example": [
      { "title": "Show HN: My project", "points": 128, "url": "https://..." }
    ]
  }'

Body

POST/scrape/batch

Scrape many URLs in one call — up to 50 — and stream a result per line as each finishes.

# -N streams each result line as it lands
curl -N https://gate.crawl4ai.com/scrape/batch \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://a.com", "https://b.com"], "format": "md"}'

import json, requests

with requests.post(
    "https://gate.crawl4ai.com/scrape/batch",
    headers={"Authorization": "Bearer sk_live_..."},
    json={"urls": ["https://a.com", "https://b.com"], "format": "md"},
    stream=True,
) as r:
    for line in r.iter_lines():
        if line:
            row = json.loads(line)
            print(row["url"], "ok" if row["ok"] else row["error"])

const res = await fetch("https://gate.crawl4ai.com/scrape/batch", {
  method: "POST",
  headers: { "Authorization": "Bearer sk_live_...", "Content-Type": "application/json" },
  body: JSON.stringify({ urls: ["https://a.com", "https://b.com"], format: "md" }),
});

// read the NDJSON stream line by line
const reader = res.body.getReader(), dec = new TextDecoder();
let buf = "";
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  buf += dec.decode(value, { stream: true });
  let nl;
  while ((nl = buf.indexOf("\n")) >= 0) {
    const line = buf.slice(0, nl); buf = buf.slice(nl + 1);
    if (line.trim()) {
      const row = JSON.parse(line);
      console.log(row.url, row.ok ? "ok" : row.error);
    }
  }
}

Body

POST/scrape/jobs

For big lists — up to 10,000 URLs. Submit once, get a job id, then poll for results while it drains in the background.

# 1. submit -> job id
JOB=$(curl -s https://gate.crawl4ai.com/scrape/jobs \
  -H "Authorization: Bearer sk_live_..." -H "Content-Type: application/json" \
  -d '{"urls": ["https://a.com", "https://b.com"]}' | jq -r .job_id)

# 2. poll until done
until [ "$(curl -s https://gate.crawl4ai.com/scrape/jobs/$JOB \
       -H "Authorization: Bearer sk_live_..." | jq -r .status)" = "done" ]; \
  do sleep 2; done

# 3. fetch results (NDJSON, one line per URL)
curl -s https://gate.crawl4ai.com/scrape/jobs/$JOB/results \
  -H "Authorization: Bearer sk_live_..."

import time, requests

H = {"Authorization": "Bearer sk_live_..."}
base = "https://gate.crawl4ai.com"

# 1. submit
job = requests.post(f"{base}/scrape/jobs", headers=H,
                    json={"urls": ["https://a.com", "https://b.com"]}).json()["job_id"]

# 2. poll until done
while requests.get(f"{base}/scrape/jobs/{job}", headers=H).json()["status"] != "done":
    time.sleep(2)

# 3. page through the results
after = 0
while True:
    lines = requests.get(f"{base}/scrape/jobs/{job}/results?after={after}",
                         headers=H).text.splitlines()
    if not lines: break
    for line in lines: print(json.loads(line)["url"])
    after += len(lines)

const H = { "Authorization": "Bearer sk_live_...", "Content-Type": "application/json" };
const base = "https://gate.crawl4ai.com";

// 1. submit
const { job_id } = await (await fetch(`${base}/scrape/jobs`, {
  method: "POST", headers: H,
  body: JSON.stringify({ urls: ["https://a.com", "https://b.com"] }),
})).json();

// 2. poll until done
let status;
do {
  await new Promise(r => setTimeout(r, 2000));
  ({ status } = await (await fetch(`${base}/scrape/jobs/${job_id}`, { headers: H })).json());
} while (status !== "done");

// 3. fetch results (NDJSON)
const out = await (await fetch(`${base}/scrape/jobs/${job_id}/results`, { headers: H })).text();
console.log(out);

Endpoints

POST/recipes/{name}

Ready-made scrapers. Pick a recipe from the catalog, send its inputs, get rows back. Browse and try them at /recipes/.

curl https://gate.crawl4ai.com/recipes/hn-hiring \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{ "thread": 49522897, "keyword": "Rust" }'

Body

Response

{
  "recipe": "hn-hiring", "version": 1,
  "rows": [ { "company": "Senzing", "author": "samlk", "text": "Senzing | Platform Engineer | Remote (USA) ..." } ],
  "usage": {
    "units": 1, "boxes": ["crawl"], "cache": "miss", "engine": "recipe",
    "lines": [ { "call": "scrape", "url": "https://news.ycombinator.com/item?id=49522897", "units": 1, "cache": "archive", "engine": "archive" } ]
  },
  "meta": { "pages_fetched": 1, "from_cache": false, "elapsed_ms": 812, "llm_calls": 0, "warnings": [] }
}

The result cache

Every recipe declares fresh_for_s. A second call with the same inputs inside that window is served from the cache: usage.cache is "hit", usage.units is 0, meta.from_cache is true. Send "bypass_cache": true to run again. Recipes with a secret input are never cached.

Your session

Some recipes read a site as you: they take a session input, your own login cookies for that site as one Cookie header line. The Crawl4AI Session extension copies it in one click. Crawl4AI uses it for this run only and does not store it: no archive, no cache, no log. Your account and the site's terms are your responsibility. The catalog says per input whether it is a secret and whether the session is required or optional; without an optional session the recipe reads what a logged-out visitor sees. A wrong or expired session costs the run's unit: rows is empty, meta.warnings carries bad_session, and meta.blocked names the site.

Staged recipes and health

Every recipe in the catalog carries a stage: published, or staged for a new recipe that runs but is not yet promoted. Each region checks every recipe every six hours and the catalog carries the result as health (ok, fail or unknown, with the last check's rows, time and error). GET /recipes/health returns the same, without a key.

Errors

The catalog GET /recipes

curl https://gate.crawl4ai.com/recipes -H "Authorization: Bearer sk_live_..."

Free (no unit). Returns {"recipes": [...]}: for each recipe its name, version, title, stage, health, kind (json, module or script), inputs (type, default, required), output fields, fresh_for_s, tags, a cost hint, run (sync or job) and a description.

MCP/mcp

Use Crawl4AI as native tools inside Claude, Cursor, or any MCP client — no install, just a URL and your key.

claude mcp add --transport http crawl4ai \
  https://gate.crawl4ai.com/mcp \
  --header "Authorization: Bearer sk_live_..."