RAG Chunk Previewer
Preview text chunking with per-chunk token estimates for your RAG setup.
Done calculating? Check these tools
Dify
Dify is an open-source LLM app development platform that enables you to create…
AnythingLLM
AnythingLLM is a full-stack application that lets you chat with any documents…
Flowise
Flowise is a drag-and-drop UI for building LLM flows using LangChain. Create…
Related categories
About RAG Chunk Previewer
The sneakiest failure in a RAG setup isn't the model — it's chunking: chunks too big for the embedder, too small to carry context, or cut mid-sentence. Paste your document, pick a strategy (character windows / sentence packing / heading sections), and drag the size and overlap sliders — every chunk's content, char count and token estimate renders instantly, with fragments and overflows flagged yellow and red, plus a chunk map for the overall shape. Tune here, then copy the parameters into Dify / AnythingLLM / LangChain instead of rebuilding your index by trial and error. All computation runs locally in your browser.
FAQ
- How do I choose chunk size and overlap?
- A sane starting point: 500–1000 character chunks with 10–15% overlap. Short FAQ content prefers small chunks with sentence packing; long documents prefer heading sections to preserve topical units. The test: can a chunk be understood on its own? A half-sentence chunk is a fragment, and both embedding and retrieval quality drop. Overlap rescues information crossing boundaries but adds storage and redundancy — bigger isn't better.
- What does the overflow warning mean?
- Embedding models cap their input: classic bge-family models at ~512 tokens, modern text-embedding models at ~8192. Overflowing chunks get truncated (the tail never makes it into the vector) or fail silently, making them unfindable forever. This tool flags chunks red beyond the 512-token threshold — if you run a modern long-window embedder you can stretch to 8192, but oversized chunks dilute the vector either way; smaller is still safer.
- Do the token estimates match what platforms report?
- Estimates use universal char/token ratios (labelled as such) — plenty for magnitude calls and overflow risk. Embedding models tokenize differently from LLMs (bge uses WordPiece, not BPE), so exact counts belong to your platform. Treat this as a parameter-tuning previewer: the relative effect of strategy/size/overlap is reliable, absolute numbers come from the platform.