PRODUCT UPDATES

A steadier path to the web.

What changed in Basecrawl, from the earliest document model to the tools that keep production crawls legible.

RELEASE NOTES

Every release, in context.

V1.4

Source-aware document freshness

  • NewAdded content hashes and canonical source metadata to normalized documents.
  • ImprovedRecurring crawls now skip unchanged extracted content before indexing.

V1.3

More legible crawl diagnostics

  • NewJob timelines now record fetch, render, extract, and retry decisions per URL.
  • FixedResolved duplicate status events when a rendered page completed after a retry.

V1.2

Smarter dynamic rendering

  • ImprovedAuto rendering now promotes application shells without opening a browser for complete HTML pages.
  • FixedBounded render waits no longer stall on long-lived analytics connections.

V1.1

Crawl boundaries you can reason about

  • NewIntroduced includePaths and excludePaths controls for scoped site discovery.
  • ImprovedCanonical URL handling now removes common tracking parameters before scheduling.

V1.0

Basecrawl is generally available

  • NewLaunched the stable /scrape and /crawl APIs with markdown, JSON, and HTML output.
  • NewReleased durable crawl jobs with page-level results and structured logs.

V0.9

Document extraction preview

  • ImprovedPrimary-content extraction better preserves headings, tables, and code examples.
  • FixedRemoved repeated navigation fragments from markdown output on multi-page crawls.

V0.8

Crawl control plane beta

  • NewAdded crawl depth and page-limit controls to make discovery budgets predictable.
  • ImprovedStatus updates now distinguish queued, crawling, processing, done, and failed jobs.

V0.7

Private beta foundations

  • NewIntroduced API key authentication and the first normalized markdown document model.
  • FixedCorrected redirect handling for origins that canonicalize between www and apex domains.