01
The page you fetch is not always the page a person reads
A modern web page can have several meaningful states. The server may return complete HTML, a partial document with data embedded in a script tag, or an application shell that becomes useful only after a browser executes JavaScript and waits for a network request. A crawler that treats all three states the same will either waste expensive browser capacity on simple pages or extract an empty shell from a site that looked healthy in a manual check.
The tempting response is to render every URL in a headless browser. It feels comprehensive, but it creates a different set of problems. Browser startup adds latency, client applications can keep network activity alive indefinitely, and pages with decorative animation can look busy long after the useful content has settled. Rendering everything also makes a crawl more sensitive to transient third-party scripts that have no bearing on the document a user actually needs.
A better strategy begins with classification. Start with a fast fetch, inspect the response for meaningful text and structural signals, then promote only the pages that need it to a browser rendering step. This does not require perfect prediction. It requires a conservative policy that recognizes when the initial response is probably not the final document and records why a render was chosen.
02
Wait for content, not for silence
Network idle is a convenient browser concept, but it is a poor universal definition of readiness. Analytics beacons, chat widgets, long polls, and service workers can keep a page active long after the main article has appeared. A crawler that waits for total silence will be slow and unpredictable. One that uses a fixed delay will be fast on some sites and incomplete on others.
Basecrawl uses content-oriented completion signals. We look for a stable document body, meaningful text growth, and the presence of the structural regions a target page normally uses. The crawl has a bounded wait budget, so a problematic page cannot consume an unlimited amount of time. If the page reaches the budget without producing useful content, the job records that condition instead of quietly treating a shell as a successful extraction.
The key is to make the policy observable. Engineers should be able to see whether a URL was fetched or rendered, how long rendering took, and what completion signal ended the wait. That information turns an intermittent problem into something a team can tune. It also helps distinguish a slow but valid page from an origin that has changed its client architecture.
const crawl = await basecrawl.crawl({
url: "https://app.example.com/docs",
maxDepth: 2,
limit: 120,
render: {
mode: "auto",
waitFor: "main article",
timeoutMs: 8_000,
},
formats: ["markdown", "json"],
});03
Discovery needs rules before it needs speed
JavaScript-heavy sites often expose more routes than a human expects to see. Client routers can generate search pages, account screens, locale variants, infinite calendar states, and tracking URLs from a single navigation event. If a crawler blindly follows every anchor, it can spend a page budget on duplicate or low-value routes before it reaches the documentation or catalog that motivated the crawl.
Set the scope at the request boundary. Include paths that matter, exclude paths that cannot produce useful public content, canonicalize known tracking parameters, and cap depth and total pages. These controls are not merely safeguards for cost. They make results repeatable. A crawl that begins with clear boundaries can be compared over time because a small frontend change is less likely to redirect it into an unbounded part of the site.
The rules should be deliberate without turning every job into a custom script. A good default follows same-origin links, respects canonical URLs, and ignores obvious non-document assets. From there, path filters and a maximum depth give application teams enough control to express an intent like, index product guides but not account flows. The crawler remains general, while the work stays understandable.
04
Extraction is where rendering becomes product data
Rendering only solves half the problem. A fully rendered page can still contain a header, a footer, a floating chat prompt, a cookie banner, and three recommendations around the paragraph that matters. The extraction stage needs to preserve the relationships in the primary document while removing recurring chrome that confuses search, retrieval, and downstream language models.
We favor structural extraction over a single brittle selector. Headings, paragraphs, lists, tables, code samples, and links are recognized as parts of a document before they are transformed into markdown or JSON. Repeated navigation regions become less likely to survive when they appear across many pages in the same crawl. The output is not a screenshot of the browser. It is a representation designed for the systems that need to read and reason over it.
When an extraction is surprising, sample the raw HTML and the normalized output together. The difference often reveals the problem immediately: the content loaded behind an interaction, the article was inside an unexpected container, or the origin returned a challenge response. That debugging path is much more useful than a generic report that the browser completed successfully.
05
Design for graceful failure
No crawling system can guarantee that every public site will render forever. Origins rate limit, deploy broken bundles, require authentication, and sometimes intentionally block automation. The engineering goal is not to erase those facts. It is to make them explicit, bounded, and recoverable. A single bad URL should not poison a full-site crawl, and a transient failure should not force an entire pipeline to start over.
Treat each page as an accountable unit of work. Record status, duration, response class, rendering choice, and a compact error message. Retry only errors that are plausibly transient, with backoff that does not amplify load on the origin. Keep the crawl job alive when its failure policy allows partial results, then expose both the completed documents and the skipped pages so a product can decide what to do next.
That combination of selective rendering, scoped discovery, content-first extraction, and legible failure handling is how we approach JavaScript-heavy sites. The web will keep changing frameworks. A crawler should respond with durable policies rather than a growing folder of one-off fixes.
Building a crawl pipeline of your own? Start with a clean document, then keep the rest of your stack focused on the product problem.
Explore all articles