Scrape SDK - Unified web scraping for TypeScript

1. Scrape a URL

import { scrape } from "scrape-sdk";

const page = await scrape("https://news.ycombinator.com");
console.log(page.markdown, page.provider, page.latencyMs);

No keys required. Jina + local are always in the chain. Site and /docs roots try /llms.txt before HTML.

Pass maxChars when the caller is an LLM. Agent tools default to 20_000.

2. Create a client (optional)

import { createScrapeClient, fromEnv } from "scrape-sdk";
import { firecrawl } from "scrape-sdk/firecrawl";
import { jina } from "scrape-sdk/jina";
import { local } from "scrape-sdk/local";

const scraper = createScrapeClient({
  providers: [
    firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY! }),
    jina(),
    local(),
  ],
  strategy: "priority",
  cache: { ttlMs: 60_000 },
});

// Equivalent if you export keys in the environment:
// const scraper = fromEnv();

provider + fallback still works. Prefer providers for more than two backends.

Set FIRECRAWL_KEYLESS=1 or call fromEnv({ firecrawlKeyless: true }) to opt into Firecrawl's no-key free surface. Set TINYFISH_API_KEY to add TinyFish Fetch/Search. TinyFish's metered goal-based Agent is opt-in with fromEnv({ tinyfishAgent: true }).

3. Search when you do not have a URL

const found = await scraper.search("firecrawl vs jina reader", { limit: 5 });

4. Map a site (URLs only)

Cheaper than crawl. Needs a provider with map (Firecrawl).

const urls = await scraper.map("https://docs.firecrawl.dev", { limit: 80 });

5. Crawl a site

Firecrawl starts a job and polls GET /v2/crawl/{id} until it completes. Spider returns pages in the crawl response.

const site = await scraper.crawl("https://docs.firecrawl.dev", { limit: 8, maxDepth: 2 });

6. Extract JSON

const data = await scraper.extract("https://firecrawl.dev", {
  schema: {
    type: "object",
    properties: { mission: { type: "string" } },
    required: ["mission"],
  },
});

7. Run an interactive web agent

Only clients configured with an agent-capable provider expose this operation. It may consume paid provider usage:

const agentResult = await scraper.agent("https://example.com", {
  goal: "Find the pricing page and return the current plan names.",
});

Quickstart