CRAWL API

Map a site without managing a crawler fleet.

Start at a root URL, set discovery boundaries, and track one durable crawl job while Basecrawl handles canonicalization, rendering, and extraction.

ENDPOINT REFERENCE

Create a crawl job

A crawl begins with one site root. Basecrawl follows eligible links, deduplicates canonical URLs, and emits completed pages with the output formats you requested.

ENDPOINT

POST/v1/crawl

Discover, render, and extract multiple pages.

Parameters

Configure the request with only the controls your workflow needs.

Parameter: url
Type: string; REQUIRED
Description: Root URL where Basecrawl begins discovery.

Parameter: maxDepth
Type: number
DEFAULT: 2
Description: Maximum link depth from the root URL, starting at zero.

Parameter: limit
Type: number
DEFAULT: 100
Description: Hard cap on pages fetched before the crawl completes.

Parameter: includePaths
Type: string[]
Description: Glob-like path prefixes that are eligible for discovery.

Parameter: formats
Type: string[]
DEFAULT: [markdown]
Description: Output fields attached to each completed page.

PARAMETER	TYPE	DEFAULT	DESCRIPTION
urlREQUIRED	string	None	Root URL where Basecrawl begins discovery.
maxDepth	number	2	Maximum link depth from the root URL, starting at zero.
limit	number	100	Hard cap on pages fetched before the crawl completes.
includePaths	string[]	None	Glob-like path prefixes that are eligible for discovery.
formats	string[]	[markdown]	Output fields attached to each completed page.

Example request

POST /v1/crawl

curl https://api.basecrawl.dev/v1/crawl \
  -H "Authorization: Bearer bc_live_your_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "maxDepth": 3,
    "limit": 250,
    "includePaths": ["/guide/*", "/reference/*"],
    "formats": ["markdown", "json"]
  }'

Track the job lifecycle

Crawl creation returns a durable job identifier. Poll the job endpoint or use an SDK helper to wait for completion. Jobs move through queued, crawling, processing, and done or failed states.

202 Accepted

{
  "id": "cr_7f2ea81c",
  "status": "queued",
  "url": "https://docs.example.com",
  "maxDepth": 3,
  "limit": 250
}

KEEP EXPLORING

OverviewChoose a focused path into the Basecrawl platform.QuickstartMake your first request in under two minutes.API referenceRequest conventions, authentication, and response shapes.ScrapeExtract clean content from a single page.SDKsTyped clients for JavaScript and Python.