Skip to main content
POST
Web services API
The Website Crawling service allows you to systematically explore and extract content from websites.

Required Parameters

string
required
The base URL to start crawling from

Optional Parameters

integer
default:"2"
Maximum depth to crawl relative to the base URL
integer
default:"10000"
Maximum number of pages to crawl
Enable navigation to previously linked pages
Allow following links to external websites
array
Regex patterns for URLs to include (e.g., ["blog/*"])
array
Regex patterns for URLs to exclude (e.g., ["admin/*"])
boolean
default:"false"
Ignore the website’s sitemap.xml
string
Webhook URL for crawl events:
  • crawl.started
  • crawl.page
  • crawl.completed
  • crawl.failed
object
Configure how each page is scraped:

Example Request

Example Response

Status Codes

Crawl job started successfully
Insufficient credits or requires payment
Rate limit exceeded
Internal server error

Authorizations

Authorization
string
header
required

Enter your API key prefixed with 'Bearer '

Body

application/json
model
enum<string>
required

Type of web service to use

Available options:
scrape,
crawl,
map,
taam-ai-search,
crawl-status
params
object

Parameters specific to the selected model

Response

200 - application/json

Successful response

id
string

Unique identifier for the request

object
string

Type of completion (e.g., scrape.completion)

created
integer

Unix timestamp of when the request was created

model
string

Model used for the request

data
object

Model-specific response data

usage
object
system_fingerprint
string