Designing LLM-ready data pipelines from the web

A useful answer should be traceable to a useful source. Preserve the canonical URL, crawl timestamp, content hash, title, and extraction metadata alongside every normalized document. Those fields let a product show citations, identify stale content, and avoid re-embedding a document whose meaningful body has not changed. They also make an incident easier to investigate when a user asks why an assistant cited a page that no longer says what it once did.

Provenance becomes more important as a corpus grows. Two pages can have nearly identical text but different ownership, dates, or audiences. A product update can invalidate a support article without changing the rest of the site. Storing document-level facts means a retrieval layer can filter by freshness, source domain, language, or content type before similarity search. The model receives context that matches the task, not merely the nearest string in a large index.

A content hash is particularly helpful for recurring crawls. Hash the normalized body after extraction, not the raw response, because third-party scripts and insignificant markup changes should not trigger a costly indexing run. When a hash does change, create a new version and retain the previous source reference long enough for evaluation and auditing. Freshness should be a controlled transition, not a surprise.

Character windows are simple to implement, but they frequently separate a heading from its explanation, split a code block in the middle, or attach a table row to unrelated prose. A better chunking policy starts from the document structure that extraction preserved. Keep a heading with the paragraphs it introduces, retain list items together when they describe one procedure, and carry a small amount of parent context into each child chunk.

There is no universal chunk size. The right boundary depends on the questions your product expects, the embedding model, and the shape of the source. Documentation often benefits from smaller sections with clear headings. Policy pages may need a broader context window. Code examples should usually remain whole so a retrieval result contains a runnable unit rather than an incomplete fragment. The important part is that chunking decisions are visible and testable, not buried inside a library default.

Store both the parent document identifier and a stable chunk identifier. That gives your retrieval layer room to rerank neighboring chunks, reconstruct a fuller section for generation, or deduplicate results from the same page. It also means an evaluation can identify whether an answer failed because the right document was absent, the right chunk was poorly formed, or the retriever simply ranked it too low.

Web data decays at different rates. Product documentation may change every week, an investor relations page may update quarterly, and a legal policy may need immediate reprocessing after a single publish event. A fixed daily crawl is easy to explain, but it can be wasteful for stable sources and too slow for volatile ones. Pipelines should express a freshness policy that matches the value and risk of each source.

Start with a crawl cadence, then use lightweight checks and content hashes to decide whether extraction and indexing are necessary. Schedule high-value paths more frequently, keep a strict page limit, and alert when a known source suddenly produces much less content than usual. Those signals catch changes that a basic success metric misses, such as a help center migrating to a new route layout or an origin returning a generic interstitial.

A good freshness system also keeps old versions available long enough to compare behavior. When an answer changes, operators should be able to tell whether the source changed, the chunking policy changed, or the model retrieved a different result. That history turns maintenance into an engineering practice rather than a guessing game.

It is tempting to evaluate only the final generated answer, but a reliable LLM experience has several independent stages. Did the crawler find the expected page? Did extraction preserve the relevant section? Did chunking keep it intact? Did retrieval rank it highly enough? Did the prompt retain the citation and ask the model to abstain when context was insufficient? A single score at the end cannot explain which of those stages deserves work.

Build a small, representative evaluation set from real user questions and known source passages. Track document recall, chunk recall, citation correctness, and answer quality separately. When a regression appears, inspect the normalized document before changing the model or prompt. In many cases, the highest-leverage fix is to remove duplicated chrome, preserve a heading, or refresh a stale page rather than introducing another layer of generation logic.

LLM-ready web data is not a one-time export. It is a living pipeline with explicit boundaries, source-aware freshness, normalized documents, and evidence for every answer. Basecrawl provides the crawling and document layer so teams can spend their time on the product decisions that follow: what to index, how to evaluate it, and how to help users trust the answers they receive.