01
The web should not be an integration tax
Most teams do not set out to build a crawler. They set out to build a research workflow, a monitoring product, a support assistant, or a search experience that needs current information. Yet the moment a product needs one reliable document from the public web, the roadmap fills with work that has nothing to do with the customer problem. Someone has to choose a browser runtime, manage retries, normalize URLs, remove navigation, and decide what happens when a page returns a challenge screen instead of the article the user asked for.
We watched this pattern repeat with teams of every size. A prototype begins with a simple fetch request. Then a site ships client rendering, a cookie banner moves into the main content, duplicate pages appear behind tracking parameters, and a scheduled job retries the same failing origin all night. The data layer becomes a collection of exceptions that only one person understands. It works just well enough to make a rewrite feel risky, but not well enough to become a dependable building block.
Basecrawl exists to make web data feel like an API primitive rather than an ongoing infrastructure project. We want a developer to describe the page or site they need, choose markdown, JSON, or HTML, and receive a durable document model. That sounds modest, but it changes where teams spend their attention. Instead of operating browser pools and writing selectors, they can improve the product that uses the resulting knowledge.
02
A clean document is more useful than a successful request
An HTTP 200 does not mean a page is useful. A product page can return a response full of menus, consent prompts, newsletter forms, unrelated recommendations, and invisible analytics markup. A help center article can look complete in a browser while its initial HTML contains only a shell. Treating every successful response as a document pushes cleanup downstream, where each team writes its own rules and no one can tell whether a result is comparable to the one from yesterday.
We designed Basecrawl around a canonical document because output quality is the real contract. The crawler resolves redirects, renders when the target needs a browser, follows a bounded discovery plan, and produces content that is ready for downstream use. The same request can include markdown for retrieval, JSON for structured product logic, and HTML when a team needs its own transform. The goal is not to hide the web. It is to make the messy parts predictable enough that they no longer leak into every feature.
That document-first approach also lets us expose useful observability. A crawl has a stable identifier, a known lifecycle, discovered URLs, and logs that explain whether an origin was rate limited, rendered slowly, or rejected a request. When data is late, developers should be able to answer why without stitching together logs from a queue, a headless browser, and three serverless functions.
const document = await basecrawl.scrape({
url: "https://docs.example.com/guide",
formats: ["markdown", "json"],
onlyMainContent: true,
});
await index.upsert({
id: document.url,
content: document.markdown,
metadata: document.metadata,
});03
Built for both experiments and durable systems
We care about the first successful request because it tells a developer whether a tool belongs in their stack. But we care just as much about the hundred-thousandth request, when the target site changes a frontend framework or a scheduled crawl needs to stay inside a strict page budget. Those are not separate products. The same simple API should let a developer start small and then add depth limits, path boundaries, output formats, and job tracking as the use case becomes more serious.
That is why Basecrawl has two focused primitives. Use /scrape when you know the page and need one clean document. Use /crawl when the task begins with a root URL and discovery matters. Both return the same output language, which means a team can move from one-off extraction to an indexed site without teaching every downstream system a new response shape. The surface area stays small, while the operational work remains behind the API.
We have also made choices that favor boring reliability over impressive demos. Crawl jobs are explicit instead of being hidden behind a long request. Limits are part of the request rather than an afterthought. Output formats are declared, not inferred from a model. These details make workflows easier to test, easier to budget, and much easier to explain when they run in production.
04
The kind of platform we want to operate
We are building Basecrawl in the open with a strong bias toward legible behavior. If a request fails, the response should say what failed and what can be retried. If a crawl finds fewer pages than expected, its discovery boundaries should make that unsurprising. If an origin is dynamic, the render policy should be observable rather than magical. Developers trust platforms that give them useful control without making them assemble the implementation themselves.
There is an important boundary here. Basecrawl is not a shortcut around a publisher's rules, and it should never encourage people to treat the web as an unowned resource. Responsible crawling means respecting documented constraints, setting sensible rate limits, honoring scope, and collecting only what a product genuinely needs. A reliable data layer is also one that can be operated responsibly by the people who use it.
Our first release is the beginning of that platform. We are focused on clean output, clear crawl controls, and a workflow that feels native to developers building data-rich applications. The web changes constantly. The systems that depend on it should not have to be fragile by default. That is the problem we built Basecrawl to solve.
Building a crawl pipeline of your own? Start with a clean document, then keep the rest of your stack focused on the product problem.
Explore all articles