Under 10 Million Tokens, Your RAG System Probably Doesn't Need a Vector Database

Almost every proposal for an AI system over your company documents includes a vector database. Pinecone, Weaviate, Chroma, pgvector, take your pick. It is treated as the obvious foundation.

There is a reasonable case for that at scale. What is missing from most proposals is where the scale threshold sits, and whether the client is anywhere near it.

A 2026 study puts a number on it.

The study and the crossover

Which RAG Paradigm Wins at Scale? tested retrieval approaches across 28 strictly nested corpus tiers spanning roughly 450-fold in size. Nested means each larger tier contains the smaller ones, so differences come from scale rather than from swapping content.

Neither approach wins outright. They cross over at a specific corpus size.

The authors' conclusion: "lexical retrieval is the strongest scalable default, while agentic reasoning works best after ranked discovery rather than in place of it."

BM25 is a keyword ranking algorithm from the 1990s. It is what a plain search index does. In this study it beat the sophisticated approaches at every tier above the crossover, without any embedding step at all.

Where a small business sits on that axis

Ten million tokens is roughly seven to eight million words. For comparison, that is somewhere in the region of eighty full-length books.

Now think about what a small business actually wants a RAG system to read. Support articles. Product documentation. Policy PDFs. A few years of email templates. Perhaps a manual and a price list.

Every corpus a client has brought us has come in somewhere in the hundreds of thousands of words. That is one to two orders of magnitude below the point where the scaling argument starts to bite. Count yours before you take our word for it; the number is usually smaller than people expect.

That is the gap worth naming. The vector database is being sold on properties that matter at a scale most buyers will never reach.

The honest counter-argument

The study does not show vector search is useless. Dense retrieval was found computationally efficient, just less accurate than BM25 in these tests. And semantic search does something keyword search cannot: it matches meaning. A customer asking "my order never showed up" and a document titled "delayed shipments" share no keywords at all.

So the strongest small setups we build are hybrid. Keyword search does the heavy lifting and semantic search covers the vocabulary mismatch. That is a search index plus embeddings. It is a different thing from running a dedicated vector database as your primary infrastructure.

The distinction is between using vectors and buying, hosting, syncing and monitoring a separate vector database system. The first is a feature. The second is infrastructure with a permanent operational cost.

What the extra infrastructure actually costs you

The bill is the smallest part. Adding a vector database means you now own an embedding pipeline, an indexing strategy, chunking decisions, sync jobs when documents change, freshness monitoring, and relevance evaluation. Each is a thing that can silently break.

The failure we see most often is staleness. A document gets updated, the re-embedding job fails quietly, and the system confidently answers from the old version. Nobody notices for weeks because the answer looks fine. Keyword search over the live documents cannot fail that way, because there is no second copy to fall out of date.

For a five-person company without an engineer on staff, that operational surface is the real cost, and it does not appear on any pricing page.

How to work out where you actually sit

• Count your corpus. Total the words in everything you want the system to read, then multiply by about 1.3 for tokens. Most people are surprised how small the number is.
• If you are well under 10 million tokens, ask any supplier proposing a vector database to justify it on something other than scale.
• Ask what happens when a document changes. If there is no clear answer, you have found the staleness problem before it finds you.
• Ask for a keyword-search baseline to compare against. If nobody has measured the simple option, nobody knows the complex one is better.
• Judge on your real questions, not on a demo. A curated demo makes every architecture look fine.

The second and last points do most of the work. A supplier who has measured a baseline and can show you where it fell short is doing engineering.

What we do

We start with the simplest retrieval that could work, measure it against a fixed set of the client's real questions, and add complexity only where the measurement demands it. Frequently it does not. Some of our RAG deployments are a search index and a well-scoped prompt, and they answer better than the vector setup they replaced because the retrieval was never the bottleneck.

When the corpus is large or the vocabulary mismatch is severe, we add embeddings. We have not yet had a small business client whose document set justified graph RAG.

Related reading: the chunking settings in every RAG tutorial covers a second place where standard advice costs money without buying accuracy, and AI chatbot vs RAG for small business covers whether you need retrieval at all.

If someone has quoted you a RAG build and you want to know whether the architecture fits your corpus size, send us the proposal. Our AI and automation work often starts by removing something from a plan rather than adding to it.