createScrapeClient

createScrapeClient({
  providers?: ScrapeProvider[];
  provider?: ScrapeProvider;          // alias for a single primary
  fallback?: ScrapeProvider | ScrapeProvider[];
  timeoutMs?: number;                 // default 30000, abortable
  retries?: number;                   // extra attempts per provider on retryable errors; default 1
  strategy?: "priority" | "cost";
  cache?: boolean | { ttlMs?: number; maxEntries?: number };
  onFailover?: (error: Error, from: string, to: string) => void;
})

fromEnv({ firecrawlKeyless?, tinyfishAgent? }) // builds providers from process.env keys, always ends with jina + local
scrape(url) // same chain, no client. Also search/map/crawl/extract/agent/scrapeMany

Methods

• scrape(url, options?) — maxChars truncates markdown and sets truncated / charCount. Site/docs roots try /llms.txt first (preferLlmsTxt, default true). failedOverFrom lists providers that died before the winner.
• Scrape options include format: "markdown" | "html" | "text" | "json", signal, waitForMs, link/image inclusion, and target-page headers when the selected adapter supports them. Requests with target-page headers bypass the in-memory cache; unsupported options fail over or raise UnsupportedOptionError.
• maxChars is a hard maximum for returned markdown/text, including the [truncated] marker.
• scrapeMany(urls, { concurrency?, ...scrapeOptions })
• search(query, { limit?, includeAnswer?, includeRawContent? })
• map(url, { limit?, search? }) — URL list without page bodies (Firecrawl)
• crawl(url, { limit?, maxDepth?, pollIntervalMs?, maxChars?, ... })
• extract(url, { schema, prompt? })
• agent(url, { goal, schema?, maxSteps?, maxDurationSeconds?, browserProfile? }) — opt-in, potentially metered goal-based web automation.
• supports(capability) — "scrape" | "search" | "crawl" | "extract" | "js" | "map" | "agent"
• listProviders()

0.3.0 behavior notes

• The client timeout is cumulative across the /llms.txt probe, retries, and fallback providers.
• maxChars: 0 is a hard zero-character limit; non-empty output is marked truncated.
• Unsupported adapter options now fail explicitly instead of silently degrading. In particular, target-page headers and format: "html" can raise UnsupportedOptionError on adapters that cannot honor them.