Guides
Vercel AI SDK
scrape_url tools with inputSchema for generateText / streamText in your app.
Use these tools inside your product agent. Cursor, Claude Code, and Codex already ship WebSearch/WebFetch — do not wire this as a duplicate of those.
import { generateText, stepCountIs } from "ai";
import { fromEnv } from "scrape-sdk";
import { createTools } from "scrape-sdk/ai";
const scraper = fromEnv();
const { text } = await generateText({
model: "openai/gpt-5.4",
tools: createTools(scraper),
stopWhen: stepCountIs(6),
prompt: "Summarize the latest posts on https://news.ycombinator.com",
});| Tool | When |
|---|---|
scrape_url | You have a URL. Full markdown, default maxChars 20000. Returns truncated and charCount. |
search_web | App agent with no host search. Then scrape the best hits. |
map_site | List URLs on a site (Firecrawl). |
crawl_site | Many page bodies. Expensive; prefer map + scrape. |
extract_json | Structured fields via JSON Schema. |
Each tool uses Zod inputSchema (AI SDK 5/6). Search/map/crawl/extract are omitted when the client cannot do them.