# Context Retrieval

> Context retrieval for RAG over web pages: what to keep after the fetch, how to chunk Markdown, retrieve and cite it, and when to refresh the index.

What happens after the fetch: chunking, retrieval and citations for web RAG. A Teneo Labs project · Built on Live Web Data API.

## Guides

- [Web content ingestion for RAG: what to keep after the fetch](https://contextretrieval.net/guides/web-content-ingestion-for-rag): What to keep from every web page before you chunk it: a reviewed source list, clean Markdown, a metadata schema, stable IDs, deduplication and failed pages.
- [Chunking Markdown for RAG: structure first, size second](https://contextretrieval.net/guides/chunking-markdown-for-rag): How to split web Markdown for retrieval: headings first, size caps as starting points, heading paths on every chunk, and whole tables and code blocks.
- [RAG retrieval for web pages: evaluate, hybrid search, rerank](https://contextretrieval.net/guides/retrieval-for-web-sourced-rag): RAG retrieval evaluation with a small golden question set, then tuning for web pages: chunk headers, hybrid BM25 and vector search, reranking and top-k.
- [RAG citations from web sources: cite the passage, URL and date](https://contextretrieval.net/guides/rag-citations-from-web-sources): How to make RAG answers cite web pages correctly: final URLs, quote-level spans, verification after generation, link rot, and citation formats readers trust.
- [RAG index freshness: re-fetch, re-embed, delete what is gone](https://contextretrieval.net/guides/rag-index-freshness): Keep a web-sourced RAG index true: refresh cadence per source type, content-hash change detection, incremental upserts, deleting removed pages, stale citations.

## Posts

- [Delete stale vectors without deleting pages that only failed once](https://contextretrieval.net/blog/delete-stale-vectors): Removed pages keep answering until their vectors go. Delete on a confirmed 404 or 410, never on a timeout, and watch sync modes that delete what is missing.
- [BM25 vs embeddings on documentation pages: which queries each wins](https://contextretrieval.net/blog/bm25-vs-embeddings): BM25 finds exact strings such as error codes and versions; embeddings find paraphrases. On docs pages you usually need both, fused into one ranking.
- [Chunk size for RAG: a starting point and one tuning pass](https://contextretrieval.net/blog/chunk-size-for-rag): There is no best chunk size for RAG. Start from headings with a cap, count in tokens, and change one setting at a time against real questions.

## Built on Live Web Data API

> Live web pages as clean Markdown, text or HTML for AI agents. Send up to 10 URLs per request to POST /v1/contents; failed pages cost nothing.

- API base URL: https://livewebdata.com/api
- Auth: `Authorization: Bearer <api_key>` (keys start with 1337_live_)
- OpenAPI: https://contextretrieval.net/openapi.json
- MCP server (Streamable HTTP, same bearer key): https://contextretrieval.net/mcp

### Docs

- [Overview](https://contextretrieval.net/docs.md): Live Web Data API turns any public URL into clean Markdown, text or HTML — fetched live through our fetch network.
- [API reference](https://contextretrieval.net/docs/api-reference.md): Two endpoints. HTTP 200 is returned even when individual pages failed — check statuses[] for the per-URL outcome.
- [Examples](https://contextretrieval.net/docs/examples.md): Copy-paste examples in cURL, Python and JavaScript. Replace YOUR_API_KEY with a key from the dashboard.
- [Automate requests](https://contextretrieval.net/docs/automation.md): Move from clicking Run in the playground to fetching URL lists from your own code — with rate limits, retries and scheduling handled.
- [SDKs](https://contextretrieval.net/docs/sdks.md): Thin, typed clients for JavaScript/TypeScript and Python — built-in retries, typed errors, one-line page fetches.
- [MCP for agents](https://contextretrieval.net/docs/mcp.md): Use the Live Web Data API as a tool in Claude, Cursor, VS Code and any other MCP client — with one URL and your existing API key.
- [No-code recipes](https://contextretrieval.net/docs/no-code.md): Call the API from n8n, Make or Zapier with each tool's generic HTTP step. Native integrations are on the roadmap.

### Optional

- [Full docs in one file](https://contextretrieval.net/llms-full.txt)
- [Postman collection](https://contextretrieval.net/live-web-data-api.postman_collection.json)
- [Example scripts (.zip)](https://contextretrieval.net/live-web-data-api-examples.zip)