Blog · 2026-07-29 · Vynaris Team
What an enterprise RAG assistant costs per answered query: the 20% no-answer tax
An enterprise RAG assistant costs $1.60 to $61.94 per 1,000 answered queries. The 20% no-answer tax and embedding-refresh line set the bill, not the model.
An enterprise internal-knowledge RAG assistant costs $1.60 to $61.94 per 1,000 answered queries across six models, a 39x spread, with prices verified 2026-07-29. Three things set that bill, and none of them is the model you pick. A 20% no-answer rate taxes every answered query by 22%. Caching, the trick that decides a text-to-SQL bill, saves almost nothing here. And an embedding-refresh line can outrun your entire generation bill below about 13,000 queries a month. This is a reproducible model you can re-run with your own corpus size and no-answer rate.
The finding, before the model
An internal-knowledge assistant answers "where's our refund policy for EU orders" by retrieving chunks from your own documents and generating an answer over them. It is self-serve: no human agent sits behind it, so unlike a support desk there is no escalation floor to dominate the bill. Three levers decide the number, and all three cut against the advice you would carry over from other workloads.
First, retrieval is dynamic. Every query pulls a different set of chunks, so the 8,000-token context changes call to call. That breaks prompt caching, which needs an identical prefix to hit. The only cacheable part is the small static system prompt, so caching saves 4% to 5% here, against the 67% to 86% it saves on a static-schema text-to-SQL agent.
Second, roughly one in five queries returns no answer. The vector database finds nothing relevant, or the model reads the chunks and correctly declines. You still paid to retrieve and to run the generation attempt. That dead spend lands entirely on the answered queries as a tax.
Third, the corpus does not embed itself once and stay free. Documents change, so you re-embed on a cadence, and that fixed monthly cost amortizes over however many queries you serve.
We built this from public pricing only. No product data. Every token count below is an assumption you can edit, and every price is from a provider's live page, captured 2026-07-29.
The verdict, per 1,000 answered queries
Model Cost / 1k answered When to pick it
------------------------------------------------------------------------- ------------------ -----------------------------------------------------------
[deepseek-v4-flash](https://vynaris.com/models#deepseek-v4-flash) $1.60 Default floor, if it clears your grounding and accuracy bar
[gemini-3.5-flash-lite](https://vynaris.com/models#gemini-3-5-flash-lite) $4.03 A cheap non-DeepSeek option inside Google's stack
[gpt-5.4-mini](https://vynaris.com/models#gpt-5-4-mini) $9.53 OpenAI-native shops wanting a mid workhorse
[claude-haiku-4.5](https://vynaris.com/models#claude-haiku-4-5) $12.39 Strong grounded-answer quality per dollar
[gemini-3.6-flash](https://vynaris.com/models#gemini-3-6-flash) $18.58 When Flash-Lite misses on synthesis across many chunks
[claude-opus-5](https://vynaris.com/models#claude-opus-5) $61.94 Hard multi-hop synthesis only, never the defaultThe spread is 39x, from $1.60 to $61.94 per 1,000 answered queries. Both ends include the 20% no-answer tax and exclude the embedding-refresh line, which we cost separately below because it scales with corpus and cadence, not with the model.
The workload (edit these)
One answered query is the outcome unit: a user asks a question, retrieval returns chunks, the model writes a grounded answer. Queries that return no answer are counted, because you paid for them.
Assumption Value Note
-------------------- ---------------------------------------------------------------- -----------------------------------------------
Static system prompt 500 input tokens the only cacheable part
Retrieved chunks 8,000 input tokens dynamic per query, not cacheable across queries
Question text 150 input tokens the user's phrasing
Answered output 300 [output tokens](https://vynaris.com/glossary/cost-per-token) a grounded answer with citations
No-answer output 60 output tokens a short "no relevant result" reply
No-answer rate 20% share of queries that return nothing usable
Corpus 2,000,000 chunks x 512 tokens 1.024B tokens per full re-embed passThe no-answer rate is the number to argue with. Twenty percent is typical for a broad internal corpus where users ask about things that were never documented. Per answered query the input is 8,650 tokens against 300 out. Because retrieval is 92% of that input and it changes every call, the usual "just cache your prefix" advice barely moves this bill.
The no-answer tax
The tax is arithmetic. Serve 1,000 queries, answer 800, and the other 200 still consumed retrieval plus a generation attempt. Spread that spend over the 800 real answers and each one costs more.
On claude-haiku-4.5, an answered query is $0.01015 and a no-answer query is $0.00895 (same input, shorter output). Across 1,000 queries you spend $9.91, and dividing by 800 answers gives $12.39 per 1,000 answered, or $0.01239 each. Against the naive figure that pretends every query answers, that is a 1.22x multiplier, a flat 22% tax that rides on every model in the table. Push the no-answer rate to 30% and the tax climbs to 1.35x; get it down to 10% and it falls to 1.10x. The lever is retrieval quality, not model choice: a no-answer rate you halve is a tax you nearly halve.
Caching barely helps here
Carry over the text-to-SQL lesson and you will be disappointed. There, a static 6,000-token schema cached to a tenth of its price and cut the bill up to 86%. Here the retrieval context is different on every query, so only the 500-token system prompt caches.
Model Uncached / 1k answered Prefix-cached / 1k Caching saves
----------------- ---------------------- ------------------ -------------
deepseek-v4-flash $1.30 $1.23 5.3%
claude-haiku-4.5 $10.15 $9.70 4.4%
gemini-3.6-flash $15.23 $14.55 4.4%
claude-opus-5 $50.75 $48.50 4.4%(These are answered-query costs before the no-answer tax, to isolate the caching effect.) Four to five percent is real money at volume, so cache the system prompt. But if someone tells you caching will transform your RAG bill, they are describing a different workload. This bill falls by sending fewer wasted tokens, not by caching tokens that never repeat.
The embedding-refresh line
Every playbook so far has been pure generation cost. A RAG assistant carries a second bill: keeping the index current. A corpus of 2,000,000 chunks at 512 tokens is 1.024 billion tokens per full pass. Re-embed it and the cost depends on the model.
Embedding model Full monthly re-embed 10% monthly delta
---------------------- --------------------- -----------------
text-embedding-3-small $20.48 / mo $2.05 / mo
text-embedding-3-large $133.12 / mo $13.31 / mo
gemini-embedding-001 $153.60 / mo $15.36 / moThat looks small until you amortize it over low query volume. Take the cheapest option, text-embedding-3-small on a full monthly refresh at $20.48, and spread it across your answered queries.
Answered queries / month Embedding / 1k answered Share of Haiku generation Share of DeepSeek generation
------------------------ ----------------------- ------------------------- ----------------------------
2,000 $10.24 82.7% 639%
10,000 $2.05 16.5% 128%
50,000 $0.41 3.3% 25.6%
100,000 $0.20 1.7% 12.8%Read the DeepSeek column. Because the cheapest model makes generation nearly free, the refresh line dominates the total until volume is high. The break-even, where a full monthly refresh equals your whole generation bill, is 12,784 answered queries a month on deepseek-v4-flash, 1,653 on claude-haiku-4.5, and just 331 on claude-opus-5. Below those volumes you spend more re-embedding the corpus than answering questions from it: on a low-traffic assistant over a large corpus, your bill is an embedding bill, not a generation bill.
The refresh cadence is the control. If only 10% of your documents change per month and you re-embed just those, the line drops tenfold to $2.05 and stops mattering above a few thousand queries. Track document change rate, embed the delta, and this cost nearly vanishes.
Where routing changes the unit economics
A text-to-SQL agent gets a free correctness signal: the query runs or it errors. A RAG assistant does not. Nothing tells you for free whether an answer was right. But you do get one cheap signal before generation, and it targets the most wasteful spend directly: the retrieval score.
If the top-k similarity is below a threshold, the corpus almost certainly cannot answer the question. So gate on it. Return the no-answer response without calling the model at all, and you kill the dead spend before it happens. This is not model routing; it is deciding whether to generate.
Model a gate that catches 75% of the no-answer queries pre-generation. On claude-haiku-4.5, 150 of every 1,000 queries now skip the LLM entirely (the query embedding is a rounding error), while 50 slip through and still generate.
Strategy (Haiku 4.5) Cost / 1k answered
-------------------------------------- ------------------
No gate, every query generates $12.39
Retrieval-score gate before generation $10.71That is a 13.5% cut with zero change to the model. Compare it to switching from Haiku to a cheaper model for quality-sensitive answers, which you may not want to do, and the gate is the better first move. It also cuts latency on the queries it rejects, since a threshold check beats a full generation. You can pre-fill your own token sizes in the calculator to see the shape on your corpus.
The honest catch: set the threshold too aggressively and you convert answerable queries into no-answers, which lowers your bill while lowering quality. Tune the gate against a labeled question set, not against the invoice.
When a wider window beats a second round-trip
The other real decision is retrieval width. Retrieve more chunks per query and every query costs more input. Retrieve fewer, and some queries miss and need a second retrieval round-trip. There is a break-even.
On claude-haiku-4.5, one wide call with a 12,000-token context window of retrieved chunks costs $0.01415 per query. One narrow 8,000-token call costs $0.01015. A narrow call plus an occasional second hop costs $0.01015 x (1 + p), where p is the share of queries that need the second retrieval. Setting the two equal gives a break-even second-hop rate of 39%.
So if more than about 39% of your queries would otherwise need a second retrieval pass, retrieve wider in one call and pay the flat input premium. If your first retrieval usually suffices, keep the window narrow and eat the occasional re-hop, which is cheaper on tokens though it adds a round-trip of latency.
Build notes
A token model does not capture the constraints that decide a production RAG assistant.
- Cache the system prompt, but do not architect around it. The 500-token static prefix is the only thing that repeats. Take the 4% to 5%; expect nothing more from a workload whose context never repeats.
- Gate on retrieval score before you touch the model. The cheapest token is the one you never generate. A similarity threshold that rejects likely no-answer queries removes the single most wasteful line in the bill.
- Watch [chunking](https://vynaris.com/glossary/chunking) and top-k, because they set your input. Eight chunks at 1,000 tokens is 8,000 input tokens on every call, answered or not. Smaller chunks with better semantic search ranking hold quality at a lower top-k, and top-k is a direct multiplier on every query's cost.
- Re-embed the delta, not the corpus, and treat the no-answer rate as a retrieval metric. Embedding only changed documents keeps the refresh line small; better ranking lowers the no-answer rate, which cuts the tax on every answered query. Both fixes are upstream of the model and pay back faster than a model upgrade.
When this workload does not need a router
The honest tradeoff: most internal assistants do not need model routing at all. Cached deepseek-v4-flash is $1.60 per 1,000 answered queries, and cached claude-haiku-4.5 is $12.39 with strong grounded-answer quality. If one competent cheap model answers your questions correctly, a multi-model router is complexity with no payoff. The two moves that always pay off are upstream of the model: a retrieval-score gate to kill dead spend, and a delta-only embedding cadence. Reserve a frontier model for the hard multi-hop questions a cheap model provably cannot synthesize, and route to it on a measured quality gap, never as the default that answers the easy majority at 39x the floor. A support desk with human agents is a different calculation, where deflection dominates; see the RAG support agent playbook for that shape.
FAQ
What does an enterprise RAG assistant cost per answered query? In model fees, $0.0016 to $0.0619 per answered query across deepseek-v4-flash through claude-opus-5, on a workload of 8,000 retrieval tokens plus 650 fixed input, 300 output, and a 20% no-answer rate. Per 1,000 answered queries that is $1.60 to $61.94, a 39x spread. Prices verified 2026-07-29. Embedding refresh is a separate line.
What is the no-answer tax? About 20% of queries return nothing usable but still cost retrieval plus a generation attempt. Spreading that dead spend over the queries that do answer raises the per-answered cost by 22% at a 20% no-answer rate. Lowering the rate through better retrieval cuts the tax almost one for one.
Why doesn't caching help a RAG assistant much? The retrieved chunks differ on every query, so the cacheable prefix is only the small static system prompt. Caching saves 4% to 5% here, against 67% to 86% on a static-schema text-to-SQL agent whose whole prefix repeats.
How much does embedding refresh add? A 1-billion-token corpus re-embedded fully each month is $20.48 on text-embedding-3-small. Amortized, that is $2.05 per 1,000 answered queries at 10,000 queries a month and $0.20 at 100,000. Below roughly 1,700 queries a month on Haiku (or 12,800 on DeepSeek), the refresh costs more than all your generation. Re-embed only changed documents to shrink it.
Should I route to a frontier model? Rarely as the default. A cheap model at $1.60 to $12.39 per 1,000 answered handles most internal questions. Route to a frontier model only for hard multi-hop synthesis a cheap model measurably cannot do, and gate on retrieval score first to remove the dead spend that no model tier fixes.
Sources
- Anthropic pricing (claude-haiku-4.5 $1/$5, claude-opus-5 $5/$25, cache read 0.1x, 5-minute cache write 1.25x), captured 2026-07-29: https://platform.claude.com/docs/en/about-claude/pricing
- OpenAI API pricing (gpt-5.4-mini $0.75/$4.50, cached input $0.075, text-embedding-3-small $0.02, text-embedding-3-large $0.13), captured 2026-07-29: https://developers.openai.com/api/docs/pricing
- Google Gemini API pricing (gemini-3.6-flash $1.50/$7.50, gemini-3.5-flash-lite $0.30/$2.50, gemini-embedding-001 $0.15, context caching), captured 2026-07-29: https://ai.google.dev/gemini-api/docs/pricing
- DeepSeek API pricing (deepseek-v4-flash $0.14 miss / $0.0028 hit / $0.28 out), captured 2026-07-29: https://api-docs.deepseek.com/quick_start/pricing
- Cost model script and per-answered-query arithmetic: from the assumptions table above.
Prices change. We re-verify every figure in this post monthly and stamp updates. Numbers here are current as of 2026-07-29.
The lesson from the model: a RAG assistant's bill is set upstream of the model. Twenty percent of queries answer nothing and tax the rest 22%, caching cannot save you because retrieval never repeats, and a re-embedding line can outweigh generation on a low-traffic deployment. Vynaris is an OpenAI-compatible gateway that gates wasted calls, right-sizes the model per query, and shows the per-answered-query cost so you watch the number that matters. One base URL swap. Get an API key at vynaris.com.