P95 1.2s
Web Scraper
One URL to clean markdown in a single request.
POST /v1/scrape
CRAWLING INFRASTRUCTURE, REBUILT
Turn the open web into your data layer.
The reliable crawling API for products that need clean, structured web data. Ship /scrape in minutes, or map an entire site with /crawl.
500 pages free every monthNo credit card required
basecrawl-cli
LIVEcrawl_7f29d8
25%$ POST /crawl · api.basecrawl.dev/v1 -d '{ "url": "docs.lumina.dev", "depth": 2, "format": "markdown" }'
→ resolving docs.lumina.dev
INFRASTRUCTURE, NOT BUSYWORK
Web data that arrives ready to use.
Basecrawl handles the unreliable parts of the open web, so your team can focus on the product that comes after the request.
Crawl every reachable page
Discover sitemaps, follow links, and control crawl depth without building a queueing system.
Clean markdown by default
Remove navigation, cookie banners, and boilerplate so every response is ready for retrieval.
Choose your output
Return markdown, JSON, or HTML from the same endpoint, with source metadata included.
Built for LLM pipelines
Get stable document chunks and predictable content for indexing, agents, and evaluations.
Render JavaScript sites
Wait for client-side routes, hydrated content, and lazy-loaded data before extraction starts.
One API, any scale
Start with a single /scrape request, then move to resilient multi-page /crawl jobs when ready.
PRODUCT SYSTEM
From open-web URL to data your product can ship.
See how Basecrawl turns a single scrape or full-site crawl into predictable markdown, JSON, and HTML for agents and pipelines.
UI product frame
playground · /scrape
request200 OK · 812ms
URLhttps://docs.northstack.dev/pricing
markdownjsonhtml
response.markdown4.1 kB · clean
# Product docs ## Pricing Growth ships 20k pages/mo with stable markdown and metadata. - Clean signal for RAG - Source URL + status - No nav chrome leftover
source
200
depth
0
format
md
Flow diagram
architecture · crawl system
Job stream
queued → crawling → done
Live status, retries, and page discovery without custom queues.
Output contracts
markdown | json | html
Same endpoint surface for agents, RAG, and analytics ingest.
Request path
- 01
URL
doc or domain
- 02
/scrape · /crawl
render + extract
- 03
Clean
strip chrome
- 04
LLM-ready
md · json · html
Capability bento
Product surfaces that ship as one API family, not a toolkit of one-off scrapers.
depth 0–5
Site Crawler
Map sitemaps, follow depth, queue resiliently.
POST /v1/crawl
links graph
Map
Discover routes before you spend crawl budget.
POST /v1/map
structured
Extract
Schema-shaped JSON from messy public pages.
POST /v1/extract
A SHORT PATH TO SIGNAL
From URL to useful data in three steps.
01
Send a URL
Call /scrape for one page or /crawl for a whole domain. Pick a format and set the crawl depth.
02
We navigate the hard parts
Basecrawl resolves links, renders JavaScript, manages retries, and filters the content you do not need.
03
Use your data
Receive normalized markdown, JSON, or HTML with metadata, ready for search, agents, or analytics.
TRUSTED BY TEAMS WHO BUILD ON THE OPEN WEB
NORTHSTACKHARBORLINEPEREGRINECIRRUS LABSTERRAFORMEDNEXUS FIELD
BUILT WITH PRACTITIONERS
Useful data should not be the hard part.
“We replaced a week of brittle browser scripts with one crawl job. The markdown is clean enough to ship directly into our retrieval pipeline.”
DR
Devika Rao
Staff Engineer, Harborline AI
“The difference is operational. We can see what was discovered, retry a failed route, and hand our analysts data they actually trust.”
MC
Malik Chen
Head of Data, Peregrine Research
“Basecrawl gave our product team the web coverage we wanted without turning our roadmap into a crawler maintenance project.”
EV
Elena Voss
Co-founder, Northstack
YOUR NEXT DATA SOURCE IS ALREADY ONLINE
Build on the web without building the crawler.
Make your first request in minutes, then grow from a single page to the whole web.
Get started for free- 500 pages each month
- Production-ready API