If you’ve built a RAG pipeline in the last two years, you know the routine. Chunk the document. Embed the chunks. Stuff them into a vector database. Run a similarity search at query time. Hope the top-k chunks actually contain the answer.

It works — until it doesn’t. Anyone who has debugged a RAG system on a 200-page financial report or a dense legal contract has hit the same wall: semantically similar text isn’t always the relevant text, chunk boundaries slice apart ideas that belong together, and there’s no clean way to explain why the model retrieved what it retrieved. You just get a ranked list of cosine similarities and a shrug.

That’s the gap a newer approach called Vectorless RAG is trying to close.

What Vectorless RAG Actually Means

The name is a little more provocative than the idea deserves — it’s not that vectors are bad, it’s that similarity search is the wrong retrieval mechanism for certain kinds of documents. Vectorless RAG replaces embeddings and nearest-neighbor search with structure and reasoning.

The most visible implementation of this idea right now is PageIndex, an open-source framework that’s been gaining traction on GitHub. The core idea: instead of chunking a document into arbitrary 500-token windows and embedding each one, PageIndex has an LLM read the document and build a hierarchical table-of-contents-style tree — sections, subsections, headings, the natural structure the document already has.

At query time, there’s no similarity search at all. The LLM reasons over the tree itself, top-down, deciding which branch is likely to contain the answer, the way a person would flip to the right chapter of a textbook instead of scanning every page for keyword overlap.

How It Actually Works

The process breaks into two phases:

1. Structural indexing (done once per document)
The LLM scans the raw document and detects natural boundaries — headings, section titles, numbered clauses, scene breaks, whatever structure the source material actually has. This gets turned into a tree, typically stored as human-readable JSON, where each node represents a section and its children represent subsections.

2. Reasoning-based retrieval (done per query)
When a question comes in, the LLM doesn’t embed it and search a vector index. It walks the tree, at each level asking something like “given this question, which of these sections is most likely to contain the answer?” It narrows down node by node until it lands on the relevant leaf, then pulls that section’s actual text as context for generation.

Because every hop down the tree is a discrete decision the LLM made, the whole retrieval path is traceable. You can literally show the reasoning: “the model chose Section 4 → Section 4.2 → Section 4.2.1 because the question referenced quarterly liabilities.” Try getting that kind of explanation out of a cosine similarity score.

Where This Beats Vector-Based RAG

Long, structured documents. Financial filings, legal contracts, policy documents, academic papers — anything with a real hierarchy benefits from being read the way it was written, not shredded into token windows.
No chunking artifacts. Arbitrary chunk boundaries frequently cut a table in half or separate a clause from its exception. Structural indexing respects the document’s own boundaries instead.
Explainability. Regulatory and legal use cases often need to justify why a system retrieved a given passage. A reasoning trail through a tree is a much easier compliance story than “these embeddings were 0.83 cosine-similar.”
No embedding infrastructure. No vector DB, no re-embedding when you change models, no tuning of chunk size / overlap / index type. Fewer moving parts to operate.

Where Vector-Based RAG Still Wins

Vectorless isn’t a strict upgrade — it’s a different tool for a different shape of problem.

Large, loosely related corpora. If you’re searching across thousands of unrelated documents (support tickets, scattered wiki pages, mixed-format knowledge bases), there’s often no clean hierarchy to reason over. Semantic similarity search is still the more scalable default.
Latency and cost at query time. Vector search is a fast ANN lookup. Tree-based reasoning means multiple LLM calls per query to traverse the hierarchy — more tokens, more latency, more cost, especially on wide or deep trees.
Fuzzy, paraphrased queries. Embeddings are good at “this means roughly the same thing” matching even when a document doesn’t spell out an obvious structural path to the answer. Reasoning-based navigation depends on the document actually having exploitable structure.

In practice, the two aren’t mutually exclusive. Some implementations combine them — hybrid tree search that uses structure to narrow the search space and semantic similarity to refine within a section — which arguably gets the best of both.

A Practical Take

If you’re the kind of person who’s already built a FAISS + sentence-transformers pipeline and knows exactly where it falls apart on long documents, Vectorless RAG is worth prototyping rather than fully replacing your stack with. A reasonable way to evaluate it:

Take a batch of long, structured documents where your current chunk-and-embed pipeline underperforms (contracts, filings, technical specs).
Run them through a tree-based indexer like PageIndex and compare retrieval precision against your existing top-k vector search on the same question set.
Look specifically at cases where the right chunk exists in your vector index but doesn’t rank in the top-k — those are exactly the failures structural reasoning is designed to fix.
Weigh the latency/cost tradeoff of multi-hop LLM reasoning against your current single ANN lookup, since that’s the real operational cost of going vectorless.

The bigger shift underneath all of this is less about vectors specifically and more about a broader trend in RAG: retrieval that leans on LLM reasoning at query time rather than pre-computed similarity. As models get cheaper and faster, doing more “thinking” per query becomes economically viable in ways it wasn’t a couple of years ago. Vectorless RAG is one of the first concrete, working examples of that shift — treating retrieval less like a database lookup and more like a small agentic task.

Further reading: PageIndex on GitHub for the open-source implementation and cookbook examples of agentic vectorless retrieval.