# Context Retrieval: guides and posts

> Context retrieval for RAG over web pages: what to keep after the fetch, how to chunk Markdown, retrieve and cite it, and when to refresh the index.

# Web content ingestion for RAG: what to keep after the fetch

Source: https://contextretrieval.net/guides/web-content-ingestion-for-rag

> What to keep from every web page before you chunk it: a reviewed source list, clean Markdown, a metadata schema, stable IDs, deduplication and failed pages.

Ingestion is the step between "we fetched the page" and "we chunked it". It is easy to rush, because nothing visibly breaks when it is wrong. The damage shows up later: a retriever that cannot tell two versions of a page apart, a citation that points at a redirect, an index that quietly lost a fifth of its sources. This guide covers what to keep from each page so the later steps have something to work with.

## Why ingestion decides retrieval quality

Everything downstream reads from what ingestion stored. The chunker can only keep a heading path if the headings survived. The retriever can only filter by source or date if those fields exist. The answer can only cite a page correctly if you stored the URL that the page actually lives at. And a refresh job can only re-embed what changed if you kept something to compare against.

Web pages make this easier than most document types, because they arrive with structure: a title, headings, a URL, often a date. The mistake is to throw that structure away at the start and pay a model to reconstruct it later.

## Start from a source list, not a crawl

Keep the pages you index in a reviewed list, under version control, grouped by source type. Three reasons:

- **Focus.** Every page in the list is one you would be willing to quote. A crawl indexes whatever links lead to, including tag pages, archives and duplicates.
- **Cost.** Fetch and embedding costs grow with the number of pages and how often you refresh them. A list makes both numbers explicit.
- **Accountability.** When someone asks why the assistant said something, you can show where the page came from and why it was included.

Stay with public pages. Pages behind a login, a paywall or other access control are out of scope. Read each site's terms of use and its robots.txt, which is specified in RFC 9309.[^1] Neither replaces the other, and none of this is legal advice: if your use case raises copyright or data-protection questions, ask counsel.

## Clean Markdown, and what main content leaves out

Fetch each page as Markdown with the main content extracted, so navigation, footers and cookie banners are gone before you embed anything. Markdown is the right intermediate format for retrieval because it keeps what matters for chunking: headings, lists, tables and code blocks.

Main-content extraction is a heuristic, so check its output once per source type:

- **Residue.** "Related articles", "Was this page helpful?" and share prompts sometimes survive. Strip them with a short per-source rule list.
- **Over-trimming.** Some layouts put real content in sidebars, such as version notes or parameter tables. If you need it, fetch that source with the full body and clean it yourself.
- **Empty pages.** A page that comes back with a title and almost no text was probably assembled in the browser by JavaScript. Record it as a failure and find another source for the same information, such as a static docs page or a published PDF.

## A metadata schema you can copy

Store these fields on every page and copy the page-level ones onto every chunk:

| Field | Why you need it |
| --- | --- |
| `url` | The final URL after redirects. Cite this and use it as the document ID |
| `requested_url` | The URL in your list. Different from `url` when the page redirected |
| `title` | Shown in citations and prepended to chunks for retrieval |
| `published_date` | Only when the page states it. Nullable, never guessed |
| `retrieved_at` | When your pipeline fetched it. Separate from the published date |
| `fetch_source` | Whether the copy was fetched live or came from a cache |
| `content_hash` | Hash of the cleaned Markdown, for change detection |
| `source_group` | Docs, news, reference and so on. Drives refresh intervals |
| `heading_path` | Per chunk: the headings above it, set by the chunker |

Two details are easy to get wrong. First, `published_date` and `retrieved_at` answer different questions ("how old is the claim?" and "how old is our copy?"), so never fill one from the other. Second, if your fetcher serves from a cache, `retrieved_at` is when you received the copy, not when the page was last read from the origin. Store the cache status next to it so you can tell the two apart.

## IDs, canonical URLs and duplicates

Use the final URL as the document ID and derive chunk IDs from it, for example `https://example.com/docs/a#3`. Normalise it once: lowercase the host, drop tracking parameters such as `utm_*`, drop the fragment, and keep the query string only when it changes the content.

Sites often expose the same content under several URLs: with and without a trailing slash, print views, old paths that redirect. Google's canonicalization guide describes how sites signal the preferred version.[^2] For your index, the practical rule is simpler: collapse URLs that redirect to the same final URL, and catch the rest by comparing content hashes. Two pages with the same hash are one document with two addresses; keep one and record the other as an alias.

Near-duplicates, such as the same article with a different date line, need a fuzzier check. Shingling or MinHash over the cleaned text works well enough for a few thousand pages. Deduplicate before embedding, because duplicates crowd out other results in the top-k.

Frameworks can track some of this for you. LlamaIndex's ingestion pipeline keeps a map of document IDs to content hashes in a docstore, skips unchanged documents and upserts changed ones.[^3] LangChain's indexing API does similar bookkeeping with a record manager.[^4] Both work better when the ID you give them is stable, which is another reason to use the final URL.

## Failed and empty pages: record them, do not skip them

A batch job that logs "98 of 120 pages indexed" and moves on has lost information you will need. Keep a failure log with the URL, the error, the time and the run, and decide per error what to do:

| Outcome | Action |
| --- | --- |
| Timeout or temporary refusal | Keep the previous version, retry next run |
| 404 or 410 | Delete the page's chunks (see [index freshness](/guides/rag-index-freshness)) |
| Not HTML or text | Remove from the list or handle with a different loader |
| Empty or no usable content | Find another source for the same information |

Report three numbers per run: pages fetched, pages failed and pages changed. A jump in failures for one source usually means the site changed its structure or moved its content.

## Where the fetch step fits

Everything above works with any fetcher. If you use Live Web Data API for the fetch, the response already carries several of these fields: `results[].url` is the final URL after redirects, `title`, `author` and `publishedDate` are returned when the page exposes them, `statuses[].fetchedAt` says when the page was read (every request is live; there is no cache), and each requested URL gets its own status with an error tag such as `CRAWL_NOT_FOUND` or `CRAWL_NOT_RENDERED`. Failed URLs cost 0 credits, so recording a failure is free. The API fetches one or more URLs per request; it does not crawl, chunk, embed or store. Markdown is truncated at `text.maxCharacters`, 100,000 characters by default: raise it for any page that hits the limit before chunking or hashing, and record every page with `results[].textTruncated: true` as truncated, so a cut-off copy never passes as complete. The automation docs cover [batch fetching and retries](/go/docs-automation?from=post:web-content-ingestion-for-rag), [the SDKs](/go/docs-sdks?from=post:web-content-ingestion-for-rag) wrap the request, and the [API reference](/go/api-reference?from=post:web-content-ingestion-for-rag) lists the per-request URL limit and every per-URL error tag.

## Where to go next

With clean Markdown and a metadata schema in place, the next step is splitting pages into chunks that keep their context: read [chunking Markdown for RAG](/guides/chunking-markdown-for-rag). To decide how often to re-fetch each source group, see [RAG index freshness](/guides/rag-index-freshness).

[Request API access](/go/signup?from=post:web-content-ingestion-for-rag). Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day.

[^1]: RFC 9309, "Robots Exclusion Protocol" (2022).
[^2]: Google Search Central, "What is URL canonicalization".
[^3]: LlamaIndex documentation, "Ingestion pipeline", document management with a docstore.
[^4]: LangChain reference, `langchain_core.indexing` API.

---

# Chunking Markdown for RAG: structure first, size second

Source: https://contextretrieval.net/guides/chunking-markdown-for-rag

> How to split web Markdown for retrieval: headings first, size caps as starting points, heading paths on every chunk, and whole tables and code blocks.

Chunking decides what a retriever can return. A chunk that mixes two topics matches neither well; a chunk that lost its heading matches nothing at all. Web pages fetched as Markdown make this easier than PDFs or raw HTML, because the author has already divided the content into sections. This guide shows how to use that structure, where size limits still matter, and what to do with the parts that do not split cleanly.

## Split on headings first

Use the page's own outline as the first cut. Each `##` or `###` section becomes one chunk. Sections are written to cover one idea, which is exactly what you want a retrieved passage to do.

LangChain's `MarkdownHeaderTextSplitter` takes a list of heading levels and records the headings above each section as metadata.[^1] LlamaIndex's `MarkdownNodeParser` does the same for nodes.[^2] Either is enough for most docs sites and articles.

```python
from langchain_text_splitters import MarkdownHeaderTextSplitter

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")],
    strip_headers=False,  # keep the heading inside the chunk text too
)
sections = splitter.split_text(markdown)
# sections[i].metadata -> {"h1": "Configure the client", "h2": "Timeouts"}
```

## Size caps and overlap as starting points

Headings alone do not bound size. A reference page can have a 4,000-word section; a changelog can have hundreds of three-line ones. Add two rules:

- **Cap long sections.** Split them on paragraph boundaries, then on sentences only if a paragraph is still too long. A cap of roughly 800 to 1,200 tokens is a reasonable place to start for documentation and articles.
- **Merge tiny sections.** Join consecutive short sections under the same parent heading until they reach a few hundred tokens, so a page of one-line FAQ answers does not become forty near-empty chunks.

Treat these numbers as defaults to test, not findings. We have not benchmarked them across corpora, and the right values depend on your embedding model, your questions and how much text you pass to the model per answer. The [retrieval guide](/guides/retrieval-for-web-sourced-rag) shows how to build a small question set to tune them on, and the note on [chunk size for RAG](/blog/chunk-size-for-rag) walks through one tuning pass.

Overlap matters less with heading-based splits, because chunks start at natural boundaries. Keep it for paragraph-split sections only, and keep it small: a sentence or two. Large overlaps mainly produce near-duplicate results.

Note that many splitters count characters, not tokens, by default. `RecursiveCharacterTextSplitter` measures length with Python's `len` unless you pass a tokenizer-based length function.

## Heading path and page title on every chunk

A section called "Limits" means nothing on its own. "Billing API › Rate limits › Limits" does. Store the heading path as metadata, and also prepend a short header to the text you embed:

```text
Page: Billing API reference (example.com/docs/billing)
Section: Rate limits > Limits

Each key allows 10 requests per second...
```

This is the cheapest form of the idea behind contextual chunk headers and Anthropic's contextual retrieval: give each chunk the context it lost when it was cut out of its page. For web pages most of that context already exists in the title and headings, so you get it without an extra model call per chunk. Keep the header short; a long one dilutes the embedding.

## Tables, lists and code blocks

Some blocks lose their meaning when they are split:

- **Tables.** A row without its header row is a list of unlabelled values. Keep a table whole when it fits under your cap. When it does not, split by rows and repeat the header row in every piece.
- **Code blocks.** Never split inside a fenced block. A half function is worse than no function. If a block is longer than the cap, keep it whole as an oversized chunk or split it at blank lines between functions and repeat the fence language.
- **Lists.** Keep a list with the sentence that introduces it. "The following settings are required:" followed by nothing is a common failure.

Check these by reading chunks, not by reading code. Pick 20 chunks per source type and look for orphaned rows, broken fences and lead-in sentences without their lists.

## When the Markdown is bad

Main-content extraction is good but not perfect. Watch for three problems before chunking:

- **Navigation residue.** Lines such as "Previous page" or "Edit on GitHub" repeated across every page end up in every chunk. Strip them with a per-source rule list; they are easy to spot because they repeat.
- **Flat pages.** Some pages have no headings at all. Fall back to paragraph splitting with the cap, and use the page title as the only path.
- **Near-empty pages.** A page with a title and a cookie notice was probably rendered in the browser. Do not chunk it; log it as a failure and find another source.

## Parent documents and small-to-big retrieval

Small chunks match queries precisely; large chunks give the model enough to answer. You can have both. Index small units (paragraphs or short sections) for matching, and when one matches, pass its parent section to the model instead. LangChain's `ParentDocumentRetriever` and LlamaIndex's hierarchical node parser with auto-merging retrieval implement this pattern.

The Dense X Retrieval paper takes the small side further and indexes propositions: single self-contained factual statements. It reports better retrieval than passage- or sentence-level units on the open-domain QA sets it tested, at the cost of a model call to extract the propositions.[^3] Late chunking goes the other way: it embeds the whole document with a long-context model, then pools token embeddings per chunk, so each chunk vector carries context from the rest of the page.[^4] Both are worth testing once structure-first chunking works and your question set shows what is still missing.

## Tooling

You do not need a framework for any of this; a regular expression that splits before `##` lines gets you most of the way. If you use one, LangChain's text splitters and LlamaIndex's node parsers both support Markdown headings, recursive splitting and metadata out of the box. Whatever you use, store a chunker version on every chunk, so a change to the rules can be re-run on all pages and compared.

## Where to go next

Clean input makes all of this simpler; see [web content ingestion for RAG](/guides/web-content-ingestion-for-rag) for the metadata to keep before you split. Then measure the result with the question set in [retrieval for web-sourced RAG](/guides/retrieval-for-web-sourced-rag).

[^1]: LangChain documentation, "Split Markdown by headers" (`MarkdownHeaderTextSplitter`).
[^2]: LlamaIndex documentation, "Node parsers" (`MarkdownNodeParser`, hierarchical node parser).
[^3]: Chen et al., "Dense X Retrieval: What Retrieval Granularity Should We Use?", arXiv 2312.06648 (2023).
[^4]: Günther et al., "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models", arXiv 2409.04701 (2024).

---

# RAG retrieval for web pages: evaluate, hybrid search, rerank

Source: https://contextretrieval.net/guides/retrieval-for-web-sourced-rag

> RAG retrieval evaluation with a small golden question set, then tuning for web pages: chunk headers, hybrid BM25 and vector search, reranking and top-k.

Retrieval is where most RAG answers are won or lost. If the passage with the answer is not in what the model sees, no prompt will fix it. The tempting move is to add techniques: a reranker, query rewriting, a graph. The better move is to measure first, so you know which problem you have. This guide covers a small evaluation set, then the techniques in the order they usually pay off for web-sourced indexes.

## Measure first: a golden question set

Before tuning anything, write down 30 to 50 questions that real users ask or will ask. For each one, record the URL and section that contains the answer. This takes an afternoon and is the most useful asset in the whole pipeline.

Then measure two things:

- **Recall at k.** For what share of questions does a correct chunk appear in the top k results? This tells you whether retrieval can work at all.
- **Rank of the first correct chunk.** Mean reciprocal rank (MRR) summarises it. It tells you whether the right chunk is near the top or buried.

Keep the set small enough that you read every miss. A miss is either a retrieval miss (the right chunk was not returned) or an [ingestion miss](/guides/web-content-ingestion-for-rag) (the right chunk does not exist, for example because a table was split or a page failed to load). The fixes are different, and only reading the misses tells you which you have. Frameworks such as Ragas automate scoring later; start by hand.

## Contextual retrieval and chunk headers

A chunk cut out of a page loses the context that made it meaningful: which product, which version, which section. Anthropic's contextual retrieval addresses this by asking a model to write a short note, usually 50 to 100 tokens, that situates each chunk in its document, and indexing the note with the chunk. On its own benchmarks, Anthropic reported that this cut failed top-20 retrievals by 35% with embeddings alone, by 49% when combined with BM25, and by 67% with a reranker added.[^1]

For web pages, start cheaper. The page title and heading path already state most of what a generated note would say. Prepend them to each chunk before embedding (see [chunking Markdown for RAG](/guides/chunking-markdown-for-rag)), measure, and only add generated context if your misses show chunks that are still ambiguous with their headers attached. Those numbers are Anthropic's, on its datasets; we have not reproduced them, and your gains will depend on how much context your headers already carry.

## Hybrid search: when keywords beat embeddings

Embeddings match meaning; they are weaker on exact strings. Documentation and reference pages are full of exact strings: error codes, function names, version numbers, product names. A user who pastes `ERR_CERT_DATE_INVALID` wants the page that contains that string, not one about certificates in general.

Hybrid search runs a keyword search (usually BM25) and a vector search, then fuses the two rankings. Two common fusion methods:

- **Reciprocal rank fusion (RRF).** Each document scores the sum of `1 / (k + rank)` over the lists it appears in, with `k = 60` in the original paper.[^2] It ignores raw scores, so it needs no calibration.
- **Weighted score fusion.** Normalise both score lists and mix them with a weight. Weaviate exposes this as an `alpha` parameter between keyword-only and vector-only.[^3]

Many vector databases support hybrid queries directly. If yours does not, run both searches and fuse with RRF in a few lines of code. Check the effect on your question set; questions with exact terms are where you should see it.

## Reranking and top-k

First-stage retrieval compares a query vector with chunk vectors that were computed separately. A cross-encoder reranker reads the query and each candidate together and scores the pair, which is more precise and much slower. The usual pattern, described in the Sentence Transformers retrieve-and-rerank docs, is to retrieve a larger candidate set cheaply and rerank only that.[^4]

A reasonable starting configuration: retrieve 50 to 150 candidates with hybrid search, rerank them, and pass the top 10 to 20 to the model. Anthropic's tests used 150 candidates reranked to 20, and found 20 chunks worked better than 5 or 10 in their setup. Your chunk sizes and context budget may point to a different number, so tune k on your question set.

Add a reranker when recall at 50 is high but recall at 5 is low. If recall at 50 is already low, a reranker cannot help; the problem is in chunking or ingestion.

## Query rewriting, carefully

Users ask short, vague or multi-part questions. Rewriting can help: expand acronyms, split a two-part question into two queries, or add the product name from the conversation. Keep the user's original wording as one of the queries, because rewrites can drop the exact term that BM25 needed. Generating a hypothetical answer and searching with it (the HyDE approach) is another option; test it against your misses rather than adding it by default.

## Packing retrieved context with its sources

How you place chunks in the prompt matters as much as which ones you pick:

- **Label each chunk** with a short ID, the page title and the URL, so the model can cite it. See [RAG citations from web sources](/guides/rag-citations-from-web-sources).
- **Remove near-duplicates** before packing. Two copies of the same paragraph waste budget and over-weight one source.
- **Group by page and keep document order** within a page, so adjacent sections read as one passage.
- **Put the strongest evidence at the edges.** The "Lost in the Middle" study found that models used information at the beginning and end of a long context more reliably than information in the middle.[^5]

## Reading failures

When an answer is wrong, check retrieval first: was the right chunk in what the model saw? If not, it is a retrieval or ingestion miss. If it was, it is a generation problem, and the fix is in the prompt, the model or the citation check. Log the retrieved chunk IDs with every answer so this check takes seconds, not an investigation.

## Where to go next

If many misses trace back to pages that changed or disappeared after indexing, the fix is in [RAG index freshness](/guides/rag-index-freshness), not in the retriever.

[^1]: Anthropic, "Introducing Contextual Retrieval" (September 2024). Figures for top-20 retrieval on Anthropic's own datasets.
[^2]: Cormack, Clarke and Büttcher, "Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods", SIGIR 2009.
[^3]: Weaviate blog, "Hybrid Search Explained".
[^4]: Sentence Transformers documentation, "Retrieve & Re-Rank".
[^5]: Liu et al., "Lost in the Middle: How Language Models Use Long Contexts", arXiv 2307.03172 (2023).

---

# RAG citations from web sources: cite the passage, URL and date

Source: https://contextretrieval.net/guides/rag-citations-from-web-sources

> How to make RAG answers cite web pages correctly: final URLs, quote-level spans, verification after generation, link rot, and citation formats readers trust.

A citation is a promise: "this page, at this address, says this". Readers check citations exactly when the stakes are high, so a wrong one costs more trust than no citation at all. With web sources there are extra ways to break the promise: redirects, edits, removed pages. This guide covers what to cite, how to carry the evidence through the pipeline, and how to check it before and after the answer ships.

## Bibliography or attribution: know which one you ship

There are two different features that both get called "citations":

- **A source list.** "These pages were used to answer." Cheap, and honest if the list is exactly the retrieved chunks. It does not tell the reader which claim came from where.
- **Attribution.** Each sentence or claim points to the passage that supports it. Harder, and the only kind a reviewer can check claim by claim.

Decide which one your product needs. Support and research tools usually need attribution. A source list is fine for casual Q&A, but then do not format it as if every sentence were attributed.

## Cite the final URL, the title and the date

Three fields make a web citation checkable:

- **The final URL after redirects.** The URL in your source list may redirect to a new path, add a locale or drop a trailing slash. Cite where the content actually lives. Store it at ingestion (see [web content ingestion for RAG](/guides/web-content-ingestion-for-rag)).
- **The page title and the section heading.** "Rate limits, in Billing API reference" lets a reader find the passage without scrolling.
- **The date your pipeline read the page.** Pages change. "Retrieved 27 Oct 2026" tells the reader which version the answer is based on. Keep it separate from any published date the page shows.

## Quote-level citations

To support claim-level attribution, carry the evidence, not just the source:

1. Give each retrieved chunk a short ID in the prompt, such as `[S3]`, with its title and URL.
2. Ask the model to put the ID after each sentence that uses that chunk, and to quote the supporting phrase when it is short.
3. In code, map IDs back to chunk records. Never let the model write URLs; it will invent plausible ones.
4. Store the quoted span with a little text before and after it.

That last step follows the W3C Web Annotation model, whose `TextQuoteSelector` identifies a passage by its exact text plus a prefix and suffix.[^3] It survives small edits better than character offsets. For links that jump to the passage, browsers that support URL text fragments (`#:~:text=`) scroll to and highlight the quoted text; others simply open the page.[^4]

## Verifying citations after generation

Models cite incorrectly more often than their confident formatting suggests. The ALCE benchmark (Gao et al., 2023) measured citation recall and precision on three datasets and found that even the best models tested lacked complete citation support about half the time on ELI5.[^1] CiteFix (2025) proposes correcting citations after generation by matching each claim against the retrieved documents, using keyword, semantic and model-based matching; the authors report a 15.46% relative improvement in overall accuracy on their data.[^2]

A practical check that needs no research code:

- For each cited sentence, confirm the cited chunk was actually in the context.
- Confirm the quoted span, if any, appears in that chunk word for word.
- Score whether the chunk supports the sentence, with an entailment model or a small LLM judge.
- If a check fails, re-attribute to another retrieved chunk that passes, or drop the citation and flag the sentence.

Log failures by source. A page that often fails verification usually has chunking problems, such as a table split away from its header row.

## Link rot and pages that changed

Between indexing and the moment a reader clicks, the page can change or vanish. Two habits keep citations true:

- **Refresh cited sources first.** Pages that are cited often deserve a shorter refresh interval than the rest of the index. See [RAG index freshness](/guides/rag-index-freshness).
- **Re-check before high-stakes answers.** For legal, policy or pricing questions, re-fetch the cited URL live and confirm the quoted span still appears. If it does not, re-answer from the current version or tell the user the source has changed.

A page that now returns 404 or 410 should be removed from the index, not just hidden from citations, or it will keep shaping answers.

If you use Live Web Data API for the re-check, send the cited URLs (every request is a live fetch; there is no cache), compare the returned Markdown with the stored span, and treat a `CRAWL_NOT_FOUND` error as a removed page. The API cuts content at `text.maxCharacters`, 100,000 characters by default, so check `results[].textTruncated` and raise the limit for any page that hits it; otherwise a quote near the end of a long page looks as if it is gone. A URL that comes back as an error costs 0 credits, so re-checking citations to pages that have vanished costs nothing. The [automation docs](/go/docs-automation?from=post:rag-citations-from-web-sources) show batching and retries, and [the SDKs](/go/docs-sdks?from=post:rag-citations-from-web-sources) wrap the request. The API only fetches; matching spans and judging support is your code.

## Showing citations readers trust

Format matters for whether people check:

- Put numbered markers inline and a source list underneath, with title, site name and the date read.
- Show the quoted passage on hover or expand, so the reader can check without leaving.
- Link to the passage with a text fragment where possible.
- Mark sources that were re-checked recently, and say plainly when a source could not be re-checked.

## Where to go next

Citations are only as good as the chunks behind them. If verification often fails, revisit [chunking Markdown for RAG](/guides/chunking-markdown-for-rag); if the right chunk is often missing, revisit [retrieval for web-sourced RAG](/guides/retrieval-for-web-sourced-rag).

[Request API access](/go/signup?from=post:rag-citations-from-web-sources). Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day.

[^1]: Gao et al., "Enabling Large Language Models to Generate Text with Citations" (ALCE), arXiv 2305.14627 (2023).
[^2]: "CiteFix: Enhancing RAG Accuracy Through Post-Processing Citation Correction", arXiv 2504.15629 (2025).
[^3]: W3C, "Web Annotation Data Model", section on `TextQuoteSelector`.
[^4]: WICG, "URL Fragment Text Directives".

---

# RAG index freshness: re-fetch, re-embed, delete what is gone

Source: https://contextretrieval.net/guides/rag-index-freshness

> Keep a web-sourced RAG index true: refresh cadence per source type, content-hash change detection, incremental upserts, deleting removed pages, stale citations.

A web-sourced index starts going stale the day you build it. Pages get edited, moved and retired, and nothing in the vector store notices. The result is the write-once index: confident answers from a version of the web that no longer exists. This guide covers how to keep an index true: how often to refresh, how to detect real changes, how to update without mixing versions, and when to delete.

## How an index goes stale

There are three different kinds of staleness, and each needs its own fix:

- **Edited pages.** The URL is the same, the content is not. Answers quote text that is no longer there.
- **Removed pages.** The URL now returns 404 or 410, but its chunks are still retrieved. Retired features and withdrawn policies keep being explained.
- **Moved pages.** The old URL redirects to a new one. Without a check, you end up with two copies, or with citations that point at a redirect.

A refresh job that only re-fetches and re-embeds handles the first, partially. Handling all three needs IDs, hashes and an explicit deletion step.

## Cadence per source type

Refreshing everything daily wastes fetches and embeddings on pages that change twice a year. Refreshing everything monthly leaves fast-moving pages wrong for weeks. Group sources by how they change and give each group its own interval:

| Source group | Starting interval |
| --- | --- |
| Changelogs, release notes, status and news pages | Daily |
| Product documentation | Weekly, plus a run after each release |
| Help center and policy pages | Weekly |
| Standards, specifications, reference articles | Monthly |

These are starting points, not measurements. After a few runs, look at how many pages per group actually changed. If a weekly group shows almost no changes, slow it down; if a monthly group changes every run, speed it up. Pages that are cited often deserve a shorter interval than their group.

## Change detection with content hashes

Compare a hash of the **cleaned Markdown**, not of the raw HTML. Raw HTML changes on every request on many sites, because of rotating ads, session tokens or build IDs. Hashing the extracted main content means only real text changes trigger re-embedding.

Normalise before hashing: collapse whitespace, and strip lines you already treat as noise, such as "Last updated" stamps that change without the content changing. Store the hash on the page record and on every chunk.

LlamaIndex's ingestion pipeline does this bookkeeping when you attach a docstore: it maps each document ID to a hash, skips unchanged documents and upserts changed ones.[^1] LangChain's indexing API does the same with a record manager and offers cleanup modes that remove old versions of changed documents.[^2] Both depend on a stable document ID; use the final URL after redirects.

## Incremental upserts and deletions

When a page changes, replace all of its chunks at once:

1. Re-chunk the new version with the current chunker.
2. Embed the new chunks.
3. Write the new chunks and delete every old chunk with that page ID in the same transaction, or behind an index alias you switch atomically.

Updating chunk by chunk leaves windows where old and new sections are retrieved together, which produces answers that mix two versions of a page.

Deletions need more care than updates, because a failed fetch is not the same as a removed page:

| Outcome on refresh | Action |
| --- | --- |
| 404 or 410 | Delete after it repeats on the next run (410 means gone on purpose) |
| Redirect to a new URL | Re-key to the new final URL, delete the old ID |
| Timeout or temporary refusal | Keep the previous version, retry next run |
| Empty or no usable content | Keep the previous version, flag for review |

RFC 9110 defines 410 Gone as an intentional, likely permanent removal, so it is safe to act on immediately.[^3] A 404 can be a temporary mistake on the site's side, so confirming it on a second run is a reasonable default.

## Fetch age in refresh jobs

If your fetcher serves cached copies, a refresh run can quietly re-embed an old version. Know which kind of fetch you get. If it caches, set the maximum cache age to match the group's interval; if it reads live, keep the last fetch time per URL in your own store and skip URLs whose copy is still younger than the group's interval, so two runs in the same window do not fetch the same page twice.

Live Web Data API reads every page live and keeps no cache, so each run sees the current version. Store `statuses[].fetchedAt` with every page and compare it with the group's interval before the next run. A `CRAWL_NOT_FOUND` error marks a page that returned 404 or 410 when it was read live, so it can feed the deletion rule above directly; still confirm a 404 on the next run. The API cuts content at `text.maxCharacters`, 100,000 characters by default, and sets `results[].textTruncated` when it did; raise the limit for any page that hits it before hashing, or an edit past the cut never changes the hash. The [automation docs](/go/docs-automation?from=post:rag-index-freshness) cover batching several URLs per request and retries, and [the SDKs](/go/docs-sdks?from=post:rag-index-freshness) wrap the request. The API fetches; hashing, upserts and deletes stay in your pipeline.

## Stale citations: re-check before you answer

Even with a good schedule, a page can change the day after a refresh. For answers that people act on, re-fetch the cited pages live before showing the answer, or at least show the date each source was read. [RAG citations from web sources](/guides/rag-citations-from-web-sources) covers the re-check in detail.

## What to log

Every run should report, per source group: pages fetched, pages changed, pages failed (by error), pages deleted and the oldest `retrieved_at` still in the index. The last number is the one to alert on. It is the honest answer to "how out of date can this assistant be?"

## Where to go next

Freshness depends on decisions made at ingestion: stable IDs, hashes and a failure log. If those are missing, start with [web content ingestion for RAG](/guides/web-content-ingestion-for-rag).

[Request API access](/go/signup?from=post:rag-index-freshness). Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day.

[^1]: LlamaIndex documentation, "Ingestion pipeline", document management with a docstore.
[^2]: LangChain reference, `langchain_core.indexing` API (record manager and cleanup modes).
[^3]: RFC 9110, "HTTP Semantics", sections 15.5.5 (404 Not Found) and 15.5.11 (410 Gone).

---

# Delete stale vectors without deleting pages that only failed once

Source: https://contextretrieval.net/blog/delete-stale-vectors

> Removed pages keep answering until their vectors go. Delete on a confirmed 404 or 410, never on a timeout, and watch sync modes that delete what is missing.

A page that was removed from the web keeps answering questions until its vectors are deleted. The fix sounds simple: delete what is gone. The trap is that "gone" and "failed to load this time" look the same to a careless refresh job.

## Gone is a status code, not a missing result

RFC 9110 defines 404 Not Found as "no current representation" and 410 Gone as a removal that is likely permanent and intentional.[^3] A 410 is safe to act on at once. A 404 can be a temporary mistake on the site, so delete after it repeats on the next run.

Everything else, such as timeouts, server errors, rate limits and empty responses, is a failure to read, not evidence of removal. Keep the previous version and retry.

## Watch out for sync modes

Frameworks make deletion easy, sometimes too easy. LangChain's indexing API has cleanup modes: `incremental` removes old versions of documents you re-index, while `full` also deletes every document that is not in the batch you pass in.[^1] LlamaIndex's ingestion pipeline with a docstore offers an upserts-and-delete strategy that removes documents missing from the input.[^2]

Both behave correctly for what they are given. The risk is upstream: if your job drops pages that failed to fetch from the batch, a full sync deletes them. One bad network minute can empty part of an index.

Two safe patterns:

- **Pass failures through.** For pages that failed this run, pass the previous version to the sync step, so they count as unchanged.
- **Delete explicitly.** Use an incremental mode for updates, and delete by page ID only for pages your own rule marks as gone.

## Delete the whole page at once

Give every chunk the page ID, the final URL after redirects, so one delete call removes all of a page's chunks. Deleting chunk by chunk leaves fragments that keep matching queries. The [index freshness guide](/guides/rag-index-freshness) covers the related case of replacing a changed page in one step.

## With Live Web Data API

Each requested URL gets its own status. A `CRAWL_NOT_FOUND` error marks a page that returned 404 or 410, while timeouts and other errors carry their own tags, so the rule above maps directly onto the response. Every request is a live fetch with no cache, so the tag reflects the page as it is now; still confirm a removal on a second run before deleting, because a site can return a 404 briefly during a deploy. The [API reference](/go/api-reference?from=post:delete-stale-vectors) lists every per-URL error tag. The API fetches; deciding what to delete and deleting it stay in your pipeline.

[Request API access](/go/signup?from=post:delete-stale-vectors). Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day.

[^1]: LangChain reference, `langchain_core.indexing` API, cleanup modes.
[^2]: LlamaIndex documentation, "Ingestion pipeline", document management.
[^3]: RFC 9110, "HTTP Semantics", sections 15.5.5 and 15.5.11.

---

# BM25 vs embeddings on documentation pages: which queries each wins

Source: https://contextretrieval.net/blog/bm25-vs-embeddings

> BM25 finds exact strings such as error codes and versions; embeddings find paraphrases. On docs pages you usually need both, fused into one ranking.

Keyword search and vector search fail on different questions. On documentation pages, which are full of identifiers, that difference decides whether a user who pastes an error message gets the right page.

## What each one matches

BM25 scores a chunk by how often the query's terms appear in it, weighted by how rare each term is across the index. A rare string such as `ERR_CERT_DATE_INVALID` or `max_retries` gets a high weight, so the chunks that contain it rise to the top.

An embedding model maps the query and each chunk to vectors and compares them. It matches meaning, so "how do I stop retrying failed calls" can find a section titled "Retry policy" that shares no words with the question. The same property makes it weaker on identifiers: an error code carries little meaning for the model to match.

## What the benchmarks say

The BEIR benchmark compared retrieval models zero-shot across 18 datasets from different domains. BM25 proved a strong baseline, and several dense retrievers trained on one dataset fell below it on others.[^1] Embedding models have improved since, but the lesson holds for specialised corpora: a model has not necessarily seen your product's vocabulary.

Anthropic's contextual retrieval write-up reports that combining embeddings with BM25 reduced failed retrievals compared with embeddings alone on its own datasets.[^2] Those are its numbers on its data; we have not reproduced them.

## Use both, then check

Hybrid search runs both and fuses the rankings, either with reciprocal rank fusion or with a weighted score that many databases expose as a single parameter, such as Weaviate's `alpha`.[^3] The [retrieval guide](/guides/retrieval-for-web-sourced-rag) covers both fusion methods.

To check whether you need it, tag each question in your evaluation set by type: identifier, paraphrase or concept. Measure recall at k per type with keyword-only, vector-only and hybrid retrieval. If identifier questions miss with vectors alone, hybrid search is the fix; if paraphrases miss with keywords alone, keep the embeddings.

## Two details that change the result

- **Tokenisation.** Many BM25 analysers split on punctuation, so `max_retries` may become two tokens. Index identifiers in a field that keeps them whole, or test how your analyser handles them.
- **Headers in the text.** If you prepend the page title and heading path to each chunk, as the [chunking guide](/guides/chunking-markdown-for-rag) suggests, BM25 can match product and section names too.

[^1]: Thakur et al., "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models", arXiv 2104.08663 (2021).
[^2]: Anthropic, "Introducing Contextual Retrieval" (September 2024).
[^3]: Weaviate blog, "Hybrid Search Explained".

---

# Chunk size for RAG: a starting point and one tuning pass

Source: https://contextretrieval.net/blog/chunk-size-for-rag

> There is no best chunk size for RAG. Start from headings with a cap, count in tokens, and change one setting at a time against real questions.

"What chunk size should I use?" is the most common RAG question, and the honest answer is that no single number wins. What you can have is a sensible starting point and a short, repeatable way to move from it.

## Fix the boundaries before the size

On web pages fetched as Markdown, most of the chunking work is already done by the author: sections under `##` and `###` headings each cover one idea. Split on those first, with a heading-aware splitter such as LangChain's `MarkdownHeaderTextSplitter`[^1] or LlamaIndex's `MarkdownNodeParser`.[^2] Size only matters for what is left over: very long sections and very short ones.

If you start from fixed-size windows instead, every size you test also moves the boundaries, so you cannot tell whether a better score came from the size or from luckier cuts.

## A starting point

For documentation and articles, a reasonable first configuration is:

- split on headings;
- cap sections at roughly 800 to 1,200 tokens, splitting longer ones on paragraph boundaries;
- merge consecutive sections shorter than a few hundred tokens under the same parent heading;
- little or no overlap, because heading-based chunks already start at natural boundaries.

These are defaults to test, not results. We have not benchmarked them across corpora. The [chunking guide](/guides/chunking-markdown-for-rag) explains each rule, including tables and code blocks that should never be split.

## Count in the right unit

Embedding models truncate by tokens, not characters. Many splitters measure length with a character count unless you give them a tokenizer-based length function, so a "1,000" cap can mean 1,000 characters, roughly 250 tokens of English. Check which one your splitter uses before you compare numbers with anyone else's.

## One tuning pass

Chroma's technical report on chunking evaluation measures retrieval at the token level and compares several chunkers and sizes on the same questions; its results differ by chunker, by size and by corpus.[^3] The practical lesson is the method, not a number:

1. Write about 30 real questions and record the page and section that answers each.
2. Index with the starting configuration and measure recall at k: how often the right section appears in the top results.
3. Read every miss. If the answer was split across two chunks, raise the cap. If the right chunk was found but ranked below longer, vaguer ones, lower it.
4. Change one setting, re-index and measure again.

Two or three passes are usually enough to find a range where results stop changing much. Store the chunker settings on every chunk, so you can re-run a page with new settings and compare.

## When size is not the problem

If misses persist at every size, the cause is usually elsewhere: the right page was never fetched, the Markdown kept navigation residue, or the chunk lost its page title and heading path. The [retrieval guide](/guides/retrieval-for-web-sourced-rag) shows how to tell these apart.

[^1]: LangChain documentation, "Split Markdown by headers".
[^2]: LlamaIndex documentation, "Node parsers".
[^3]: Chroma, "Evaluating Chunking Strategies for Retrieval" (technical report, 2024).

---

# Live Web Data API — full documentation

# Overview

Live Web Data API turns any public URL into clean Markdown, text or HTML — fetched live through our fetch network.

## What it does

You send up to 10 URLs (and optionally a country). We fetch each page through a residential or mobile connection, extract the main content and return it in the format your agent or pipeline needs. Successful pages cost 1 credit; failures cost nothing.

| | Value |
| --- | --- |
| Base URL | `https://livewebdata.com/api` |
| Endpoints | `POST /v1/contents`, `GET /v1/account` |
| Formats | markdown and text by default; html with `html: true` |
| OpenAPI | [`/openapi.json`](https://contextretrieval.net/openapi.json) · [`/openapi.yaml`](https://contextretrieval.net/openapi.yaml) |
| Fetch strategy (v1) | static fetch (`fast`), a retry through another connection (`retry`), or a real browser (`browser`) with `render` or `renderFallback` |

## Authentication

Create a key in the dashboard under [API keys](https://contextretrieval.net/dashboard/keys) and send it as a bearer token. Keys start with `1337_live_` (test keys with `1337_test_`) and are shown only once. Keep them in an environment variable such as `LIVE_WEB_DATA_API_KEY` — never in client-side code or repositories.

```bash title="Headers"
Authorization: Bearer 1337_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
```

## Quickstart

Fetch a page as Markdown:

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/article"]}'
```

The response contains one entry in `results` per successful page and one entry in `statuses` per requested URL. The page HTML is included only when you send `"html": true`:

```json title="Response (abridged)"
{
  "requestId": "req_01J…",
  "results": [{
    "id": "https://example.com/article",
    "url": "https://example.com/article",
    "title": "Article",
    "markdown": "# Article\n\n…",
    "text": "Article\n\n…"
  }],
  "statuses": [{
    "id": "https://example.com/article",
    "status": "success",
    "source": "livecrawled",
    "fetchedAt": "2026-10-23T09:15:02.118Z",
    "httpStatusCode": 200,
    "strategy": "fast",
    "credits": 1,
    "error": null
  }],
  "searchTime": 1240
}
```

When `text` or `markdown` was cut at `text.maxCharacters`, `results[].textTruncated` is `true`. `results[].warnings` (absent when empty) flags pages that could only be partly retrieved (`PAGE_TRUNCATED`), probably need `render: true` (`THIN_CONTENT`), were converted as plain text (`EXTRACTION_FALLBACK`) or lost their `html` (and maybe part of their text) to the response's 24 MiB size limit (`RESPONSE_SIZE_LIMIT`).

See the [API reference](https://contextretrieval.net/docs/api-reference) for every parameter, or try it without code in the dashboard [playground](https://contextretrieval.net/dashboard/playground).

## SDKs

Thin, typed clients for JavaScript/TypeScript and Python with built-in retries and typed errors. See [SDKs](https://contextretrieval.net/docs/sdks).

```javascript title="JS SDK"
import { Teneo } from "@teneo/contents";

const client = new Teneo({
  apiKey: process.env.LIVE_WEB_DATA_API_KEY,
  baseUrl: "https://livewebdata.com/api",
});

const res = await client.contents({
  urls: ["https://example.com/article"]
});
for (const s of res.statuses) console.log(s.id, s.status, s.error?.tag ?? "");
console.log(res.results[0]?.markdown);
```

```python title="Python SDK"
import os
from teneo import Teneo

client = Teneo(
    api_key=os.environ["LIVE_WEB_DATA_API_KEY"],
    base_url="https://livewebdata.com/api",
)

res = client.contents(
    urls=["https://example.com/article"],
)
for s in res["statuses"]:
    print(s["id"], s["status"], (s["error"] or {}).get("tag", ""))
print(res["results"][0]["markdown"] if res["results"] else "")
```

## Live fetching

Every request is fetched live; there is no cache.

## Geotargeting

Set `countryCode` to an ISO-3166-1 alpha-2 code (for example `de`, `jp`, `br`) to fetch the page as seen from that country. The default is `us`.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/pricing"
  ],
  "countryCode": "de"
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/pricing"],
        "countryCode": "de"
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/pricing"
    ],
    "countryCode": "de"
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Multiple URLs

Send up to 10 URLs in one request. They are processed in parallel; `statuses` has one entry per URL (matched by `id`) and credits are charged per successful URL. A restricted URL gets a per-URL `RESTRICTED_TARGET` status instead of failing the whole request. More than 10 URLs return `400 TOO_MANY_URLS`. Rate limits count URLs, not HTTP requests.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/a",
    "https://example.com/b",
    "https://example.com/blocked"
  ]
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"]
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/a",
      "https://example.com/b",
      "https://example.com/blocked"
    ]
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Restricted targets

Some domain categories are blocked for legal and regulatory reasons. Requests are rejected with `451` and cost no credits. Repeated attempts lead to automatic key suspension. Report abuse to abuse@livewebdata.com.

## MCP for agents

Agents can use the API as a tool over MCP with your existing key — two tools, `get_page_contents` and `get_account_status`. Setup for Claude Code, Cursor, VS Code and Claude Desktop: [MCP for agents](https://contextretrieval.net/docs/mcp).

## For AI coding agents

Every docs page is also available as plain Markdown by appending `.md` to its URL (e.g. [`/docs/api-reference.md`](https://contextretrieval.net/docs/api-reference.md)). An index for LLMs lives at [`/llms.txt`](https://contextretrieval.net/llms.txt) and the full docs in one file at [`/llms-full.txt`](https://contextretrieval.net/llms-full.txt).


---

# API reference

Two endpoints. HTTP 200 is returned even when individual pages failed — check statuses[] for the per-URL outcome.

## Endpoint

```bash title="Request"
POST https://livewebdata.com/api/v1/contents
Authorization: Bearer <api_key>
Content-Type: application/json
```

Machine-readable specs:

- OpenAPI 3.1 (JSON): https://contextretrieval.net/openapi.json
- OpenAPI 3.1 (YAML): https://contextretrieval.net/openapi.yaml
- Postman collection: https://contextretrieval.net/live-web-data-api.postman_collection.json
- Interactive reference: https://contextretrieval.net/docs/api-reference/interactive

## Request parameters

| Parameter | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `urls` | `string[]` | yes | — | The pages to fetch, 1–10 http or https URLs of at most 8,192 characters each. Processed in parallel; one entry per URL in results/statuses. Duplicates (same URL, ignoring #fragment) are fetched once; repeated positions cost 0 credits. |
| `countryCode` | `string` | no | "us" | ISO-3166-1 alpha-2 code. The page is fetched through a connection in this country. |
| `text` | `boolean \| object` | no | true | Content options. Pass an object to set maxCharacters and verbosity. |
| `text.maxCharacters` | `integer` | no | 100000 | Truncates text and markdown to this many characters (max 1,000,000); `results[].textTruncated` tells you when it did. The MCP tool defaults to 8,000 (`max_characters`). |
| `text.verbosity` | `"compact" \| "full"` | no | "compact" | compact = main content only. full = whole body incl. navigation and footer. |
| `html` | `boolean` | no | false | Include the page HTML as served in results[].html. A response carries at most 24 MiB of JSON: past that, the remaining results come without html (warning RESPONSE_SIZE_LIMIT). |
| `livecrawlTimeout` | `integer (ms)` | no | 30000 (60000 with render) | Maximum time for a live fetch, 1–90000 ms. A lower value is raised to 30000, the time the browser fallback needs, unless you send renderFallback: false (then any value, default 10000). With render or renderFallback: true at least 30000. |
| `render` | `boolean` | no | false | Render every URL in a real browser, as a normal visitor sees it: 3 credits, 5 when the page needed a retry through another connection. Cannot be combined with renderFallback. |
| `renderFallback` | `boolean` | no | on | Fetch normally first and render in a real browser only when the first attempt did not return the page, shows no or very little visible text, or was blocked. On by default: omit it. Send false for a plain fetch only (never a browser). A normal success costs its normal 1 or 2 credits, a rendered one only the render credits; statuses[].fallbackUsed tells you when it rendered. Cannot be combined with render: true. |

```json title="Example body"
{
  "urls": ["https://example.com/article"],
  "countryCode": "de",
  "text": { "maxCharacters": 50000, "verbosity": "compact" },
  "livecrawlTimeout": 45000
}
```

Unknown fields are rejected with `400 INVALID_REQUEST`; the message names the field. So is a URL longer than 8,192 characters (`urls.0: Each URL may be at most 8,192 characters.`).

## Response

| Field | Type | Description |
| --- | --- | --- |
| `requestId` | `string` | Id of this request, the same for every repeat with the same Idempotency-Key. Also returned as X-Request-Id. |
| `results[]` | `object[]` | One entry per successfully fetched URL. |
| `results[].url` | `string` | Final URL after redirects. |
| `results[].title` | `string \| null` | Page title. |
| `results[].author / publishedDate / image` | `string \| null` | Metadata when the page exposes it. |
| `results[].favicon` | `string \| null` | Favicon URL. |
| `results[].markdown` | `string` | Main content as Markdown. |
| `results[].text` | `string` | Plain text, tags removed, whitespace normalized. |
| `results[].html` | `string` | Page HTML as served. Only with html: true, and only while the response stays within its 24 MiB size limit (else warning RESPONSE_SIZE_LIMIT). |
| `results[].textTruncated` | `boolean` | true when text or markdown was cut at text.maxCharacters, or at the response size limit (warning RESPONSE_SIZE_LIMIT). Present whenever text is enabled. |
| `results[].warnings` | `{ code, message }[]` | Absent when empty. "PAGE_TRUNCATED" = only part of the page could be retrieved; content after that point is missing. "THIN_CONTENT" = very little text, the page probably needs render: true. "EXTRACTION_FALLBACK" = the page was too complex to convert in time; text is plain and markdown has no structure. "RESPONSE_SIZE_LIMIT" = the response reached its 24 MiB size limit: this result has no html, or no html and cut text/markdown. |
| `statuses[]` | `object[]` | One entry per requested URL, success or error. |
| `statuses[].status` | `"success" \| "error"` | Outcome for this URL. |
| `statuses[].source` | `"livecrawled" \| null` | Always "livecrawled" on success: every request is fetched live. null on error. |
| `statuses[].fetchedAt` | `string \| null` | When the page was fetched (ISO 8601). null on error. |
| `statuses[].strategy` | `"fast" \| "retry" \| "browser"` | Fetch strategy. "fast" = standard page (1 credit); "retry" = the first attempt did not return the page, a retry through another connection did (2 credits); "browser" = rendered in a real browser, as a normal visitor sees it (3 credits, 5 when it needed a retry through another connection). |
| `statuses[].credits` | `integer` | Credits charged for this URL: 1 standard, 2 retried, 3 rendered, 5 rendered + retried, 0 on error and for a short page (thin). |
| `statuses[].fallbackUsed` | `boolean` | Present (true) only when the browser fallback ran for this URL. With strategy "browser" the content is that render (fast → render); otherwise rendering was tried and did not help. |
| `statuses[].thin` | `boolean` | Present (true) only for a short page: fewer than 500 characters of visible text even after rendering. Delivered as a success at 0 credits, with only that visible text. |
| `statuses[].error` | `{ tag, message } \| null` | Per-URL error, see crawl errors. |
| `searchTime` | `integer (ms)` | Total processing time. |

Browser fallback is on by default: a page the fast fetch cannot read (no or very little visible text, blocked, 401/403, a timeout) is rendered in a real browser and charged only the render credits. `statuses[].fallbackUsed` is `true` when that happened (`strategy: "browser"` = the content is the render). Send `renderFallback: false` for a fast fetch only. A page that still shows fewer than 500 characters of visible text after rendering is delivered as a success with `statuses[].thin: true` at 0 credits, holding only that visible text — it is not an error.

`text.maxCharacters` (default 100,000, max 1,000,000) cuts `text` and `markdown`; `results[].textTruncated` is `true` when it did. A response carries at most 24 MiB of JSON: from the result that would pass it on, results come without `html`, and if that is not enough, `text` and `markdown` are cut (`textTruncated: true`), each such result flagged `RESPONSE_SIZE_LIMIT`. The request never fails because of it. `title` and `author` are at most 2,000 characters each; an `image` or `favicon` URL longer than 4,096 characters is left out (`favicon` then falls back to `/favicon.ico`). The MCP tool `get_page_contents` has its own, smaller default of 8,000 (`max_characters`), because its result goes into an agent's context. `results[].warnings` is present only when there is something to know about a page:

| Code | Meaning | What to do |
| --- | --- | --- |
| `PAGE_TRUNCATED` | Only part of the page could be retrieved (size limit or an interrupted transfer); content after that point is missing. | Nothing to retry; the result is the part that arrived. |
| `THIN_CONTENT` | Very little text next to scripts or an empty app root: the page probably loads its content with JavaScript. The URL is still charged as a success. | Retry with `render: true`. The default browser fallback renders a page with fewer than 500 characters of visible text by itself, not one with more. |
| `EXTRACTION_FALLBACK` | The page structure was too complex to convert in time; `text` is plain and `markdown` carries no structure. | Use `text`. |
| `RESPONSE_SIZE_LIMIT` | The response reached its 24 MiB size limit. This result and every later one come without `html`; when that was not enough, `text` and `markdown` were cut too (`textTruncated: true`). The URL is still a success and is charged. | Request fewer URLs at once, or without `html`. |

## Response headers

| Header | Description |
| --- | --- |
| `X-Request-Id` | Same as requestId in the body. |
| `X-Credits-Remaining` | Credit balance after this request. |
| `X-RateLimit-Limit` | URLs you can send at once (bucket size, = rateLimit.burst in GET /v1/account). The bucket refills at rateLimit.requestsPerSecond URLs per second. |
| `X-RateLimit-Remaining` | URLs you can send right now. |
| `Retry-After` | Seconds to wait: before retrying a 429, or before repeating a 202 with the same Idempotency-Key. |
| `Idempotency-Key` | The request's Idempotency-Key: yours, or one the API created when you sent none. On every answer (200, 202, errors) except a 400 for a malformed key and a 413 for a body over 10 MiB (refused unread). Repeat a 202 with it. |

## Errors

### Per-URL errors

Returned inside `statuses[].error` with HTTP 200. These never cost credits.

| Tag | Meaning |
| --- | --- |
| `CRAWL_NOT_FOUND` | The page returned 404/410. |
| `SOURCE_NOT_AVAILABLE` | The target refused access (e.g. 403 or 429) after retries. |
| `CRAWL_LIVECRAWL_TIMEOUT` | The live fetch did not finish within livecrawlTimeout. |
| `CRAWL_NOT_RENDERED` | No usable content, e.g. the page needs JavaScript rendering (retry without renderFallback: false, or with render: true), or a rendered page shows no visible text. 0 credits. |
| `UNSUPPORTED_CONTENT_TYPE` | The URL does not serve HTML or text. |
| `CRAWL_UNKNOWN_ERROR` | Something else went wrong. Safe to retry. |
| `SOURCE_BLOCKED` | Access denied by the site, or it answered with an interstitial page (HTTP 202) instead of the content. 0 credits. |
| `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable; no credits were charged. |
| `RATE_LIMITED` | Rendering limit for your plan reached; no credits were charged. |
| `RESTRICTED_TARGET` | This URL is restricted. Other URLs in the same request are still processed. 0 credits. |

### Request errors

Returned with a non-200 status and no results.

| HTTP | Tag | Meaning |
| --- | --- | --- |
| 400 | `INVALID_REQUEST` | Invalid parameters. |
| 400 | `TOO_MANY_URLS` | More URLs than allowed per request (10). |
| 401 | `UNAUTHORIZED` | Missing or invalid API key. |
| 402 | `INSUFFICIENT_CREDITS` | Not enough credits. Top up or upgrade. |
| 403 | `KEY_SUSPENDED` | The key or account is suspended or not yet approved. |
| 429 | `RATE_LIMITED` | Rate limit exceeded. Retry after the Retry-After header. |
| 409 | `IDEMPOTENCY_CONFLICT` | The Idempotency-Key was already used with a different request, or its result is no longer available (results are kept 15 minutes). Use a new key. |
| 413 | `PAYLOAD_TOO_LARGE` | The request body is larger than 10 MiB. Nothing was read or charged. |
| 451 | `RESTRICTED_TARGET` | All requested URLs are restricted. No credits charged. |
| 503 | `SERVICE_UNAVAILABLE` | Temporarily unavailable. Retry shortly. |

```json title="Error body"
{
  "error": {
    "code": 402,
    "tag": "INSUFFICIENT_CREDITS",
    "message": "Not enough credits. Top up or upgrade your plan."
  },
  "requestId": "req_01J…"
}
```

## Long requests and Idempotency-Key

Most requests answer within seconds. A request still fetching after about 100 seconds (slow sites, rendering, a long `livecrawlTimeout`) answers `202` instead of making you wait on an open connection:

```json title="202 Accepted"
{
  "status": "pending",
  "requestId": "req_01J…",
  "idempotencyKey": "lwda_3f1c2b7e-0d4a-4c1e-9a55-2b8f0e6c7d10",
  "retryAfter": 1,
  "message": "Still fetching. Send the same request again with the same Idempotency-Key header to get the result; it is not charged again."
}
```

Send the **same request body** again with the header `Idempotency-Key: <idempotencyKey>` after `Retry-After` seconds. The repeat waits for the fetch and answers `200` with the result, or `202` again if it is still running. A repeat is never charged again. The SDKs and the MCP server do this for you.

**Always send your own `Idempotency-Key`** (8–128 characters of `A–Z a–z 0–9 _ - : .`, for example a UUID) on the first request, and keep it until you have the answer. A connection can break after the API received the request: a network error, a timeout on your side, a deploy. Send the same request again with the same key, and you get that fetch (its result, or `202` while it still runs) instead of a new one, so you pay once. Without your own key, a cut connection is a lost answer: the retry has no key, so it is a new request and charged again.

Every answer of `POST /v1/contents` (`200`, `202` and errors) carries the request's key in the `Idempotency-Key` response header: yours, or one the API created when you sent none. The only exceptions are the `400` for a malformed key and the `413` for a request body over 10 MiB, which is refused before it is read (nothing was charged, there is nothing to repeat). The SDKs create a key for each `contents()` call when you pass none, send it on every retry and put it on the error they raise (`idempotencyKey` / `idempotency_key`).

- Results are kept for **15 minutes**. A result nobody collects stays charged; after 15 minutes a repeat answers `409 IDEMPOTENCY_CONFLICT`, and a new key fetches the pages again.
- A key is bound to its request body for 24 hours: reusing it with a different body answers `409 IDEMPOTENCY_CONFLICT`. Use a new key for every new request.

## Rate limits

Limits are per account and count **URLs per second** (a request with 3 URLs uses 3). When exceeded, the API returns `429` with a `Retry-After` header (seconds). Limits work as a bucket: `X-RateLimit-Limit` is how many URLs you can send at once (`rateLimit.burst` in `GET /v1/account`), `X-RateLimit-Remaining` how many right now, and the bucket refills at your plan's URLs per second (`rateLimit.requestsPerSecond`).

| Plan | Price / month | Rate limit (URLs/s) |
| --- | --- | --- |
| Lite | $10 | 1 |
| Starter | $100 | 2 |
| Developer | $500 | 5 |
| Business | $900 | 10 |
| Enterprise | $2,250 | 20 |
| Custom | Custom | Custom |

## Credits

| Request | Credits |
| --- | --- |
| Standard page (static / server-rendered) | 1 |
| Retried page (succeeded on a retry through another connection) | 2 |
| JS-rendered page | 3 |
| Rendered + retried page | 5 |
| Short page (under 500 characters of visible text; delivered as that text, a success) | 0 |
| Failed request (incl. access denied by the site) | 0 |

- Plan credits reset monthly.
- Top-up credits never expire.

When your balance is too low for the URLs in a request, the API returns `402 INSUFFICIENT_CREDITS`. The current balance is returned in `X-Credits-Remaining` on every response.

## Account

`GET https://livewebdata.com/api/v1/account` returns your plan, remaining credits and rate limit. It is free — call it before large jobs.

```json title="GET /v1/account"
{
  "plan": "starter",
  "status": "active",
  "credits": { "plan": 81234, "topup": 10000, "total": 91234 },
  "rateLimit": { "requestsPerSecond": 2, "burst": 4 },
  "periodEndsAt": "2026-11-01T00:00:00Z"
}
```


---

# Examples

Copy-paste examples in cURL, Python and JavaScript. Replace YOUR_API_KEY with a key from the dashboard.

## Basic fetch

The smallest possible request — one URL, defaults for everything else.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/article"
  ]
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/article"]
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/article"
    ]
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Geotargeting

Fetch a page as seen from Japan.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/pricing"
  ],
  "countryCode": "jp"
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/pricing"],
        "countryCode": "jp"
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/pricing"
    ],
    "countryCode": "jp"
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Text limits

Return at most 5,000 characters of the full page body, including navigation and footer.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/docs/getting-started"
  ],
  "text": {
    "maxCharacters": 5000,
    "verbosity": "full"
  }
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/docs/getting-started"],
        "text": {
            "maxCharacters": 5000,
            "verbosity": "full"
        }
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/docs/getting-started"
    ],
    "text": {
      "maxCharacters": 5000,
      "verbosity": "full"
    }
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Several URLs

Fetch up to 10 pages in one request. Each URL gets its own status; only successful pages cost credits.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/a",
    "https://example.com/b",
    "https://example.com/blocked"
  ]
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"]
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/a",
      "https://example.com/b",
      "https://example.com/blocked"
    ]
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Whole jobs

For URL lists, retries and scheduling see [Automate requests](https://contextretrieval.net/docs/automation) — including ready-made Python and Node scripts.


---

# Automate requests

Move from clicking Run in the playground to fetching URL lists from your own code — with rate limits, retries and scheduling handled.

## From playground to code

Every run in the [playground](https://contextretrieval.net/dashboard/playground) shows a **Use it in code** panel with cURL, Python, JavaScript and SDK snippets for exactly the request you just made. Start there, then move to the scripts below.

Download all example scripts: https://contextretrieval.net/live-web-data-api-examples.zip

| File | What it does |
| --- | --- |
| `examples/python/quickstart.py` | One request, prints title + first 500 chars of markdown |
| `examples/python/batch_from_csv.py` | Reads `urls.csv` (`url`, optional `country`), fetches with concurrency = rate limit, retries with backoff, writes `results.jsonl` and prints a summary |
| `examples/node/quickstart.mjs` | Same as the Python quickstart, native `fetch`, no dependencies |
| `examples/node/batch_from_csv.mjs` | Same as the Python batch script, no dependencies |
| `examples/urls.csv` | 10 sample URLs incl. a restricted, a missing and a timing-out URL to show error handling |

```bash title="Run the batch script"
export LIVE_WEB_DATA_API_KEY=1337_live_…   # never hard-code keys
python examples/python/batch_from_csv.py examples/urls.csv
node examples/node/batch_from_csv.mjs examples/urls.csv
```

## Process a list of URLs

The simplest loop: one request per URL, one after the other.

```python title="Python"
import os, requests

API = os.environ.get("LIVE_WEB_DATA_API_BASE_URL", "https://livewebdata.com/api")
headers = {"Authorization": f"Bearer {os.environ['LIVE_WEB_DATA_API_KEY']}"}

for url in ["https://example.com/a", "https://example.com/b"]:
    r = requests.post(f"{API}/v1/contents", json={"urls": [url]}, headers=headers, timeout=60)
    status = r.json()["statuses"][0]
    print(url, status["status"], status["error"] and status["error"]["tag"])
```

```javascript title="Node"
const API = process.env.LIVE_WEB_DATA_API_BASE_URL ?? "https://livewebdata.com/api";
const headers = { Authorization: `Bearer ${process.env.LIVE_WEB_DATA_API_KEY}`, "Content-Type": "application/json" };

for (const url of ["https://example.com/a", "https://example.com/b"]) {
  const res = await fetch(`${API}/v1/contents`, { method: "POST", headers, body: JSON.stringify({ urls: [url] }) });
  const { statuses } = await res.json();
  console.log(url, statuses[0].status, statuses[0].error?.tag ?? "");
}
```

You can also send up to 10 URLs per request — see [Several URLs](https://contextretrieval.net/docs/examples#several-urls).

## Respect rate limits

- Keep a steady pace of **your plan's URLs per second** (from `GET /v1/account` → `rateLimit.requestsPerSecond`). Bursts up to `rateLimit.burst` URLs (the bucket size, also sent as `X-RateLimit-Limit`) are fine; the bucket refills at that rate.
- Watch `X-RateLimit-Remaining` on each response: it is how many URLs you can send right now.
- On `429`, wait `Retry-After` seconds and retry the same request.

## Retries

Retry `429`, `5xx` and per-URL `CRAWL_LIVECRAWL_TIMEOUT` up to **3 times** with exponential backoff and jitter (e.g. `0.5s · 2^n + random(0–0.25s)`). Do **not** retry:

| Outcome | Why |
| --- | --- |
| `451 RESTRICTED_TARGET` | The domain is blocked; repeated attempts can suspend your key |
| `400 INVALID_REQUEST` / `TOO_MANY_URLS` | Fix the request first |
| `401` / `403` | Key invalid, revoked or suspended |
| `402 INSUFFICIENT_CREDITS` | Stop the job cleanly, top up, resume |
| `CRAWL_NOT_FOUND` | The page doesn't exist |

**Send your own `Idempotency-Key`** with every `POST /v1/contents` (one per logical request, for example per URL of your list) and send the same key on every retry of that request. A connection cut after the API received the request (a network error, a timeout, a deploy) is then retried without a second charge: the repeat returns that fetch's result. A `202` (still fetching) is repeated the same way, after `Retry-After`. Use a **new** key for a new attempt at a URL that already has an answer, such as a per-URL `CRAWL_LIVECRAWL_TIMEOUT`: the same key returns the same answer. See [Long requests and Idempotency-Key](https://contextretrieval.net/docs/api-reference#long-requests-and-idempotency-key).

The [SDKs](https://contextretrieval.net/docs/sdks) implement this policy for you.

## Check your balance first

Call `GET /v1/account` before a large job and compare `credits.total` with the number of URLs. Stop cleanly when a request returns `402`.

```bash title="cURL"
curl https://livewebdata.com/api/v1/account -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"
```

## Scheduling

Any scheduler works — the scripts are plain processes that exit when done.

```bash title="cron (daily at 06:00)"
0 6 * * * cd /opt/jobs && LIVE_WEB_DATA_API_KEY=… /usr/bin/python3 batch_from_csv.py urls.csv --out results.jsonl
```

```yaml title=".github/workflows/fetch.yml"
name: Fetch pages
on:
  schedule: [{ cron: "0 6 * * *" }]
  workflow_dispatch:
jobs:
  fetch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: node examples/node/batch_from_csv.mjs examples/urls.csv --out results.jsonl
        env:
          LIVE_WEB_DATA_API_KEY: ${{ secrets.LIVE_WEB_DATA_API_KEY }}
```

Other schedulers (Cloud Scheduler, Kubernetes CronJob, n8n, Airflow) run the same command.

## Keep your key safe

- Store it in an environment variable (`LIVE_WEB_DATA_API_KEY`) or your platform's secret manager.
- Never put it in client-side code, mobile apps, browser extensions or repositories.
- Use one key per environment (e.g. production, staging, n8n) — you can label keys in the dashboard and see the label in the usage log.
- Rotate a key immediately if it leaks: create a new one, then revoke the old one.


---

# SDKs

Thin, typed clients for JavaScript/TypeScript and Python — built-in retries, typed errors, one-line page fetches.

## Install

The SDKs live in the repository under `packages/` and are not published yet.

```bash title="JavaScript / TypeScript"
cd packages/sdk-js && npm install && npm run build
npm install ./packages/sdk-js        # in your project
```

```bash title="Python"
pip install ./packages/sdk-python    # requires httpx
```

## JavaScript / TypeScript

```javascript title="JS SDK"
import { Teneo } from "@teneo/contents";

const client = new Teneo({ apiKey: process.env.LIVE_WEB_DATA_API_KEY, baseUrl: "https://livewebdata.com/api" });

const page = await client.getPage("https://example.com/article", { country: "de", format: "markdown" });
console.log(page.title, page.credits);

const res = await client.contents({ urls: ["https://example.com/a", "https://example.com/b"] });
const account = await client.account();
```

## Python

```python title="Python SDK"
import os
from teneo import Teneo

client = Teneo(api_key=os.environ["LIVE_WEB_DATA_API_KEY"], base_url="https://livewebdata.com/api")

page = client.get_page("https://example.com/article", country="de", format="markdown")
print(page.title, page.credits)

res = client.contents(urls=["https://example.com/a", "https://example.com/b"])
account = client.account()
```

Both clients read the key from `LIVE_WEB_DATA_API_KEY` and the base URL from `LIVE_WEB_DATA_API_BASE_URL` when you don't pass them.

## Methods

| JS | Python | Description |
| --- | --- | --- |
| `contents(request, { idempotencyKey })` | `contents(urls=…, country_code=…, text=…, html=…, livecrawl_timeout=…, render=…, render_fallback=…, idempotency_key=…)` | Raw `POST /v1/contents` (1–10 URLs). `render_fallback=False` / `renderFallback: false` turns the default browser fallback off |
| `getPage(url, opts)` | `get_page(url, country=…, format=…)` | One URL → `{ title, url, content, credits, source }`; raises if that URL failed. `format: "html"` requests the HTML (`html: true`) |
| `account()` | `account()` | `GET /v1/account` |

## Errors

| Error | When |
| --- | --- |
| `TeneoAuthError` | 401 / 403 — key invalid, revoked or suspended |
| `TeneoInsufficientCreditsError` | 402 |
| `TeneoRateLimitError` | 429 after all retries (`retryAfter` / `retry_after`) |
| `TeneoIdempotencyError` | 409 — the idempotency key was used with a different request, or its result expired (15 minutes) |
| `TeneoRestrictedTargetError` | 451 |
| `TeneoConnectionError` | Network error or timeout after all retries, or a `2xx` answer cut while it was read. The API may have received the request: repeat it with the error's `idempotencyKey` / `idempotency_key` |
| `TeneoPendingError` | Still `202` when `maxWaitMs` / `max_wait` ran out; `idempotencyKey` / `idempotency_key` collects the result later |
| `TeneoPageError` | `getPage` / `get_page`: the URL failed (`tag`, e.g. `CRAWL_NOT_FOUND`) |
| `TeneoAPIError` | Base class (status, tag, requestId, and `idempotencyKey` / `idempotency_key`: the key of a `contents()` call, else `null` / `None`) |

## Retries

Both SDKs retry `429` (respecting `Retry-After`), `5xx`, network errors, a `2xx` answer cut while it is read and — for `getPage` — `CRAWL_LIVECRAWL_TIMEOUT`, up to 3 times with exponential backoff and jitter. `400`, `401`, `402`, `403`, `409` and `451` are never retried. Configure with `maxRetries` / `max_retries`.

Every attempt of one `contents()` call sends the same `Idempotency-Key`, so a retry picks up the first fetch and is never charged twice (render requests included). A fetch that runs longer than about 100 seconds answers `202`; the SDKs repeat it after `Retry-After` until the result arrives, for up to 5 minutes (`maxWaitMs` / `max_wait`). These repeats do not count as retries. See [Long requests](https://contextretrieval.net/docs/api-reference#long-requests-and-idempotency-key).

Without your own key, the SDKs create one per `contents()` call. When the retries run out, the error carries that key: call `contents()` again with the same request and `{ idempotencyKey: e.idempotencyKey }` / `idempotency_key=e.idempotency_key` within 15 minutes, and a fetch the API already received returns its result instead of being charged again. **Better: always pass your own key** (one per logical request, for example per row of your input), so a request cut by a network error, a timeout or a deploy can be retried without a second charge even after your process restarts.


---

# MCP for agents

Use the Live Web Data API as a tool in Claude, Cursor, VS Code and any other MCP client — with one URL and your existing API key.

## What is MCP

The Model Context Protocol (MCP) is an open standard that lets AI agents discover and call external tools. Our MCP server exposes the Contents API as two tools, so an agent can read any public web page on its own.

## Server & auth

| | Value |
| --- | --- |
| Server URL | `https://contextretrieval.net/mcp` |
| Transport | Streamable HTTP (stateless): JSON-RPC over `POST`; `GET` and `DELETE` answer `405` |
| Auth | `Authorization: Bearer <api_key>` |
| Keys | Same keys as the REST API — create one under [API keys](https://contextretrieval.net/dashboard/keys) |

Missing or invalid keys are rejected with HTTP 401 (suspended accounts: 403, "API access for this account is suspended. Contact support.") before any tool runs.

## Client setup

Replace `YOUR_KEY` with your API key. In the dashboard, the [Integrations](https://contextretrieval.net/dashboard/integrations) page fills in your key for you.

### Claude Code

Run this in your terminal. The server is then available in every Claude Code session for this project.

```bash
claude mcp add --transport http live-web-data https://contextretrieval.net/mcp \
  --header "Authorization: Bearer YOUR_KEY"
```

Add `--scope user` to make it available in all projects. Check with `claude mcp list`.

### Cursor

Add this to your Cursor MCP config, globally or per project.

File: `~/.cursor/mcp.json  or  .cursor/mcp.json`

```json
{
  "mcpServers": {
    "live-web-data": {
      "url": "https://contextretrieval.net/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_KEY"
      }
    }
  }
}
```

Prefer an env var: `"Bearer ${env:LIVE_WEB_DATA_API_KEY}"`.

### VS Code

Add this to your workspace. VS Code asks for the key the first time the server starts and stores it securely.

File: `.vscode/mcp.json`

```json
{
  "inputs": [
    {
      "type": "promptString",
      "id": "live-web-data-key",
      "description": "Live Web Data API key",
      "password": true
    }
  ],
  "servers": {
    "live-web-data": {
      "type": "http",
      "url": "https://contextretrieval.net/mcp",
      "headers": {
        "Authorization": "Bearer ${input:live-web-data-key}"
      }
    }
  }
}
```

Uses the top-level key `servers` (not `mcpServers`). Paste `YOUR_KEY` when prompted.

### Claude Desktop

Claude Desktop connects to remote servers through the mcp-remote bridge. Add this to your Claude Desktop config and restart the app.

File: `claude_desktop_config.json`

```json
{
  "mcpServers": {
    "live-web-data": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://contextretrieval.net/mcp",
        "--header",
        "Authorization:${AUTH_HEADER}",
        "--transport",
        "http-only"
      ],
      "env": {
        "AUTH_HEADER": "Bearer YOUR_KEY"
      }
    }
  }
}
```

Requires Node.js. Alternative on paid plans: Settings → Connectors → Add custom connector with request header `authorization` = `Bearer …` (where available; the server must be publicly reachable).

### Other MCP clients

Any MCP client that supports Streamable HTTP can connect directly with the URL and the Authorization header. For stdio-only clients, use the mcp-remote bridge:

```bash
npx -y mcp-remote https://contextretrieval.net/mcp \
  --header "Authorization: Bearer YOUR_KEY" --transport http-only
```

Server URL: `https://contextretrieval.net/mcp` · Transport: Streamable HTTP · Auth header: `Authorization: Bearer <key>`

### REST API

Prefer plain HTTP? The same API key works with the REST endpoint.

```bash
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/article"]}'
```

## get_page_contents

Fetch a public web page live and return its main content as clean Markdown, plain text or HTML. Read-only, idempotent. Single URL per call.

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `url` | string (http/https, at most 8,192 characters) | required | The public URL to fetch. |
| `country` | ISO-3166 alpha-2 | `"us"` | Fetch the page as seen from this country, e.g. 'de', 'gb', 'us'. |
| `format` | `"markdown"` \| `"text"` \| `"html"` | `"markdown"` | Output format. Prefer markdown. The page HTML is fetched and returned only for `"html"`. |
| `max_characters` | integer 500–100000 | `8000` | Maximum characters of content to return. Smaller than the REST API's default (`text.maxCharacters`, 100,000) because the result goes into the agent's context; raise it when the content is cut. |
| `verbosity` | `"compact"` \| `"full"` | `"compact"` | compact = main content only, full = whole page body. |

The result is one text item with a short header, followed by the content. The full REST-style JSON is also returned as `structuredContent` (with `results[].html` only for `format: "html"`).

```markdown title="Tool result"
# Example article
Source: https://example.com/article · country: us · live · credits used: 1 · credits left: 8,411

# Example article

Main content as clean Markdown…
```

## get_account_status

Check the remaining credits, plan and rate limit for the connected account. Agents should call it before large batches. No parameters, free. Returns the same numbers as `GET /v1/account`.

```text title="Tool result"
Plan: Starter · Credits left: 8,412 · Rate limit: 2 URL/s (burst 4) · Status: active
```

## Errors

Errors are returned with `isError: true` and one actionable sentence the agent can follow. When nothing was charged, the message ends with “No credits were charged.”

| Case | Message to the agent |
| --- | --- |
| 451 restricted target | This website is restricted and cannot be retrieved. Do not retry; use another source. |
| `CRAWL_NOT_RENDERED` | The page has no usable text: it needs JavaScript rendering, or shows no text even when rendered. Try another source or a different URL for the same information. |
| `CRAWL_NOT_FOUND` | Page not found (404/410). |
| `SOURCE_NOT_AVAILABLE` | The website refused this request. Retry once with a different country, otherwise use another source. |
| `SOURCE_BLOCKED` | Access denied by the website. Retry later or with a different country, otherwise use another source. |
| `CRAWL_LIVECRAWL_TIMEOUT` | The page timed out. Retry once with a longer timeout, otherwise use another source. |
| `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable. Retry in a few minutes, otherwise use another source. |
| `RATE_LIMITED` | The rendering limit for the user's plan is reached. Do not retry rendering now; use another source. |
| 402 insufficient credits | The account is out of credits. Tell the user to top up at …/dashboard/billing. |
| 429 rate limited | Rate limit reached. Wait N seconds before retrying. |
| 401 / 403 | The API key is invalid or suspended. Tell the user to check their key at …/dashboard/keys. |
| Still fetching after 5 minutes | The page did not finish loading within 5 minutes, and the fetch may still be charged. Use another source. |

## Credits

MCP calls are billed exactly like REST calls: 1 credit per successful page, failed requests (including access denied by the site) are free. They appear in Usage with channel `mcp`. See [Credits](https://contextretrieval.net/docs/api-reference#credits).


---

# No-code recipes

Call the API from n8n, Make or Zapier with each tool's generic HTTP step. Native integrations are on the roadmap.

## Before you start

You need an API key from [API keys](https://contextretrieval.net/dashboard/keys). Every recipe sends the same request:

| Setting | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Header | `Authorization: Bearer YOUR_KEY` |
| Body (JSON) | `{"urls": ["https://example.com/article"], "countryCode": "us"}` |

The response is JSON: page content in `results[0].markdown`, the outcome in `statuses[0].status` and `statuses[0].error.tag`.

## n8n

> **Native integration coming soon.** Use the built-in **HTTP Request** node for now.

_Screenshot coming soon: n8n HTTP Request node_

| Field (HTTP Request node) | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Authentication | Generic Credential Type → **Bearer Auth** (token = your key) — or **Header Auth** with Name `Authorization`, Value `Bearer YOUR_KEY` |
| Send Body | on |
| Body Content Type | `JSON` |
| Specify Body | `Using JSON` → `{"urls": ["{{ $json.url }}"]}` |
| Options → Response → Response Format | `JSON` (default: autodetect) |

**URL lists:** feed one item per URL into the node. Use the node's **Options → Batching** (*Items per Batch* = your rate limit, *Batch Interval (ms)* = 1000) — or a **Loop Over Items** node plus a **Wait** node (1 second) to stay within your rate limit.

## Make

> **Native integration coming soon.** Use the **HTTP** app's **Make a request** module for now.

_Screenshot coming soon: Make HTTP "Make a request" module_

| Field (HTTP → Make a request) | Value |
| --- | --- |
| Authentication type | `API key` (stores the key in a keychain) — or `No authentication` plus the header below |
| URL | `https://livewebdata.com/api/v1/contents` |
| Method | `POST` |
| Headers | `Authorization` = `Bearer YOUR_KEY` |
| Body content type | `application/JSON` |
| Body input method | `JSON string` |
| Body content | `{"urls": ["https://example.com/article"]}` |
| Parse response | `Yes` |

**URL lists:** use an **Iterator** over your URLs before the HTTP module and set the scenario's scheduling so the number of operations per second stays within your rate limit.

## Zapier

> **Native integration coming soon.** Use **Webhooks by Zapier** → **Custom Request** for now (not available on the Free plan).

_Screenshot coming soon: Zapier Webhooks by Zapier – Custom Request_

| Field (Custom Request) | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Data Pass-Through? | `False` |
| Data | `{"urls": ["https://example.com/article"]}` |
| Headers | `Authorization` = `Bearer YOUR_KEY`, `Content-Type` = `application/json` |

The JSON response is available as fields in later steps.

## Tips

- Failed pages return HTTP 200 with `statuses[0].status = "error"` — add a filter/router on that field.
- Keep the key in the tool's credential store, not in plain step fields, wherever possible.


---