Does retrieval make medical QA more reliable?

The question

Retrieval is usually sold as the fix for a model that makes things up: let it look things up and it will be right more often. Accuracy is only half of that. In medicine the dangerous answer is the confident wrong one, so I want to know whether retrieval makes an open 8B model more reliable: fewer confident errors, and confidence that drops when the evidence isn’t there.

This is an evaluation study, not a new system. It’s a three-person course project, and I built the pipeline.

Four conditions

Every condition uses the same questions, the same model weights, the same prompt (only the context block changes), greedy decoding and the same answer parser. Only what the model reads changes:

The gaps are the findings: C2 − C1 is what retrieval buys, C3 − C2 is what failed retrieval costs, and C2b against C2 shows whether the model knows when it doesn’t know. A null or negative result counts; published work already shows retrieval hurting 8B models, so “it didn’t help” is an answer I’d report, not hide.

Closing the leaks

PubMedQA is easy to fool yourself with. Each question is its paper’s title rephrased, and each abstract’s conclusion states the answer. Left alone, a search would just find the question’s own paper and read the answer off it, and the RAG numbers would measure how the benchmark was built rather than retrieval. So:

• Answer leak. The source abstract’s conclusion is stripped from the search library. Copies of a source filed under another ID are removed too; I found two.
• Title leak. Every document is indexed without its title.
• Tuning leak. Prompt, k and thresholds are fixed on the 500 dev questions. The 500 test questions, the official PubMedQA test set, are run once at the end.

What’s built

A search library of 273,441 abstracts, deduplicated and with the leaks above removed. Two retrievers: BM25, which I wrote by hand and checked returns the same top 10 as the bm25s library on all 500 dev questions, and MedCPT, a biomedical neural retriever. BM25 finds the source paper in its top five for 84.6% of dev questions. Confidence is read straight from the model’s probability for yes, no and maybe, and every metric is written by hand so I can defend each one line by line. A separate probe checks whether Llama has simply memorised the papers’ conclusions, which would make the closed-book score look better than it is.

Before seeing any Llama output I fixed the decision rules: both retrievers are reported, k is picked per retriever by dev macro-F1, and the confident-error threshold is 0.9, with 0.8 and 0.95 as checks.

How it’s measured

Accuracy, macro-F1 and per-class recall, against a floor of 55.2% for always answering “yes”. For reliability: the confident-error rate (wrong with confidence of at least 0.9), answers that flip between closed-book and retrieved, risk–coverage curves and calibration error. Paired McNemar tests and bootstrap confidence intervals throughout.

Limits

500 test questions can’t detect differences much below 4–6 points, so small gaps will be reported as intervals, not claims. One benchmark and one model family, so nothing here generalises beyond them. The model runs in FP16 on free Kaggle T4s rather than BF16, the same in every condition. “Hallucination” here means a confident wrong label, since PubMedQA answers are yes, no or maybe and there’s no free text to fabricate. And nothing here is a claim about clinical safety.

Next

Run Llama on the dev set under all four conditions, choose k by the fixed rule, then run the test set once.

Code

github.com/Ashvinj2004/retrieval-reliability-medqa