A retrieval system that cannot refuse is not finished
Ask a typical RAG stack something its corpus does not contain and it will answer anyway, fluently, with no signal that it is guessing.
Take any RAG system that is working well and ask it something the corpus simply does not cover. Watch what happens. Almost always it retrieves the nearest few chunks, because vector search will always return its top k whether or not anything matched, and the model writes a confident paragraph out of material that does not answer the question.
Nothing in that pipeline is broken. Retrieval returned results, because that is what it does. Generation produced fluent text grounded in the context it was handed, because that is what it does. The system has no step whose job is to say no.
Why "just prompt it to say I don't know" only half works
It helps, and then it stops helping, for a reason worth understanding. By the time the model sees the context, the evidence that retrieval failed has already been thrown away. The chunks arrive as a flat list with no scores, no distribution, no indication that the best match was barely better than the worst. They look exactly like chunks from a question the corpus does answer well.
The information needed to refuse exists one stage earlier, and it does not survive the handoff.
Where the decision belongs
Refusal is a retrieval-time decision, made on retrieval-time evidence:
- The absolute similarity of the best match, not just its rank. Top-1 of a bad set is still bad.
- The spread between the best match and the rest. One strong hit among weak ones usually means a real answer. A flat distribution usually means nothing matched.
- Whether the retrieved chunks agree with each other. Contradiction is a signal of its own, and it deserves a different response than absence does.
Each of those becomes a gate before generation runs. Below threshold, the system does not call the model at all. It says it does not have the answer, and where it can, says what it searched.
The part that makes it real
None of this is testable without a set of questions the corpus genuinely does not answer, kept alongside the ones it does. Most eval sets contain only answerable questions, so they measure how well the system answers and never once measure whether it knows when to stop.
Build that negative set first. It is the cheapest artefact in the project and the only one that will tell you whether refusal actually works.