Crawl Website

$0.01

Scrape & Extract powered by Firecrawlasync

Crawl an entire website and extract content from all pages

Parameters

url required

limit optional

maxDiscoveryDepth optional

excludePaths optional

includePaths optional

allowSubdomains optional

delay optional

Example

clawcard agent pay crawl-site '{"url": "https://docs.stripe.com", "limit": 10}' --json

Async — Requires Polling

This capability returns a job/task ID instead of immediate results. Poll using crawl-status:

clawcard agent pay crawl-status '{"taskId":"<id from response>"}' --json

Each poll costs money. Wait 10-15 seconds between polls.

API Endpoint

POST https://www.clawcard.sh/api/catalog/crawl-site

Requires API key in Authorization header. Balance deducted per call.