# Scrape SDK

> TypeScript client to scrape URLs to markdown with failover across Firecrawl, Jina, Tavily, Spider, Browserbase Fetch, and local Cheerio.

Scrape SDK (npm: `scrape-sdk`) is a library you install into a TypeScript app. It is not a hosted scrape API. Call `scrape(url)` and swap adapters without rewriting callers — the same shape as Email SDK / Domain SDK.

## When to use Scrape SDK

- A TypeScript/Node/Bun app needs URL → markdown with one result shape.
- You already have Firecrawl/Jina/Tavily keys and want failover when one 429s.
- CI needs a local Cheerio fallback with no SaaS key.

## When not to use it

- Ordinary lookups in Cursor, Claude Code, or Codex — those hosts already have WebSearch and WebFetch.
- You only use Firecrawl and do not need failover — use the official `firecrawl` SDK.
- You need a headed browser / Playwright session. Browserbase here is the Fetch API, not a browser.

## How AI agents should use it

Prefer these machine-readable routes instead of scraping HTML:

- `/llms.txt` — site-level index
- `/docs/llms.txt` — documentation-only index
- `/llms-full.txt` — combined current docs
- `/feeds/docs.jsonl` — one TechArticle per page
- `/schemamap.xml` — feed map
- `/docs/<path>.md` — raw Markdown for one page

Install: `npm install scrape-sdk` or `npx skills add SoulSniper-V2/scrape-sdk --skill scrape-sdk`.

In app code, `fromEnv()` then `scraper.scrape(url)`. MCP (`npx -y scrape-sdk-mcp`) is for full-page markdown, map, crawl, or extract — not a replacement for host WebSearch.

## Constraints

- Server-side TypeScript only. Keep vendor keys in the environment.
- Do not invent crawl/map results. Missing capability throws `CapabilityError`.
- Firecrawl crawl `POST /v2/crawl` returns a job id; the adapter polls `GET /crawl/{id}`.
- Claude Code WebFetch summarizes pages. Use `scrape_url` / `scrape()` when you need the body.

## Documentation

- [How it works](/docs/how-it-works): Routing, retries, timeouts, and capabilities.
- [TypeScript Scrape SDK](/docs): Scrape, search, crawl, and extract with one TypeScript client across Firecrawl, Jina, Tavily, Spider, Browserbase Fetch, and local Cheerio.
- [Installation](/docs/installation): Install scrape-sdk, the CLI, and the MCP server.
- [Quickstart](/docs/quickstart): Scrape, search, crawl, and extract in a few lines.
- [Machine-readable Scrape SDK docs](/docs/agents): Give agents current Scrape SDK documentation without scraping rendered HTML.
- [Machine-readable documentation](/docs/agents/machine-readable-docs): Use raw Markdown, LLM indexes, JSONL entities, and the schema map for agent retrieval.
- [createScrapeClient](/docs/api-reference/create-scrape-client): Client factory, fromEnv, and methods.
- [Errors](/docs/concepts/error-handling): Typed errors the client throws.
- [Failover and retries](/docs/concepts/failover-matrix): What is retried, what fails over, what does not.
- [Markdown pipeline](/docs/concepts/markdown-pipeline): Where markdown is produced.
- [Model Context Protocol](/docs/guides/model-context-protocol): Full-page scrape, map, crawl, extract. Not a replacement for host WebSearch.
- [Offline testing](/docs/guides/offline-testing): Inject fetch or use the local adapter.
- [Vercel AI SDK](/docs/guides/vercel-ai-sdk): scrape_url tools with inputSchema for generateText / streamText in your app.
- [Browserbase](/docs/providers/browserbase): Fetch API markdown — not a Playwright session.
- [Firecrawl](/docs/providers/firecrawl): Firecrawl API v2 — scrape, search, map, crawl with job polling, JSON extract.
- [Providers](/docs/providers): What each adapter calls, and what it can do.
- [Jina Reader](/docs/providers/jina): r.jina.ai scrape and s.jina.ai search.
- [Local Cheerio](/docs/providers/local): Static HTML to markdown with no API key.
- [Spider.cloud](/docs/providers/spider): Single-page scrape and multi-page crawl.
- [Tavily](/docs/providers/tavily): Extract and search with Bearer auth.