Scrape SDK

How it works

Routing, retries, timeouts, and capabilities.

View Markdown
  1. Validate the URL (http / https only).
  2. Select providers that advertise the required capability (scrape, search, crawl, extract, js).
  3. Order them by strategy: priority (config order) or cost (cheaper first).
  4. Execute with an AbortSignal timeout. Retryable errors (429, 5xx, timeout) retry on the same provider, then fail over. Auth errors (401/403) do not retry.
  5. Return a unified result, or AllProvidersFailedError / CapabilityError.

Local Cheerio is the only adapter that sanitizes HTML itself. Cloud adapters return the vendor's markdown.

There is no fake Browserbase page content and no crawl that pretends POST /crawl already contains pages.