Scrape SDK | Web scraping for agents, handled.

1. Create a client

import { createScrapeClient, fromEnv } from "scrape-sdk";
import { firecrawl } from "scrape-sdk/firecrawl";
import { jina } from "scrape-sdk/jina";
import { local } from "scrape-sdk/local";

const scraper = createScrapeClient({
  providers: [
    firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY! }),
    jina(),
    local(),
  ],
  strategy: "priority",
  cache: { ttlMs: 60_000 },
});

// Equivalent if you export keys in the environment:
// const scraper = fromEnv();

provider + fallback still works. Prefer providers for more than two backends.

2. Scrape a URL

const page = await scraper.scrape("https://news.ycombinator.com", {
  format: "markdown",
  onlyMainContent: true,
});
console.log(page.markdown, page.provider, page.latencyMs);

Pass maxChars when the caller is an LLM. Agent tools default to 20_000.

3. Search when you do not have a URL

const found = await scraper.search("firecrawl vs jina reader", { limit: 5 });

4. Map a site (URLs only)

Cheaper than crawl. Needs a provider with map (Firecrawl).

const urls = await scraper.map("https://docs.firecrawl.dev", { limit: 80 });

5. Crawl a site

Firecrawl starts a job and polls GET /v2/crawl/{id} until it completes. Spider returns pages in the crawl response.

const site = await scraper.crawl("https://docs.firecrawl.dev", { limit: 8, maxDepth: 2 });

6. Extract JSON

const data = await scraper.extract("https://firecrawl.dev", {
  schema: {
    type: "object",
    properties: { mission: { type: "string" } },
    required: ["mission"],
  },
});

Quickstart