SourceFetch: Grounded Q&A over billing & coding policy

Launch Live Demo

Billers and coders spend their day inside dense policy PDFs. The corpus includes the NCCI Policy Manual and the Claims Processing Manual as well as modifier guidance. They search for the one paragraph that settles a question. SourceFetch turns that corpus into a question answering service. It retrieves the relevant passages from a vector store and has a model answer strictly from them. Each passage is cited inline. SourceFetch is the companion to ScrubCheck. ScrubCheck flags that a claim will deny. SourceFetch answers why from the manual.

The code is on GitHub. The design is summarized below.

System Design

SourceFetch uses retrieval augmented generation (RAG). The governing rule is that the model may speak only from what it retrieves. A question is embedded. The most relevant passages are pulled from a ChromaDB vector store. A model then answers strictly from those passages. Each claim is cited with a [n] that links to the exact source. I built the system deliberately as retrieval rather than as a model fine tuned on the policy. The rulebook changes every quarter. An answer that traces back to a passage is more trustworthy than one baked opaquely into a model’s weights. The tool runs offline with no API key and returns the raw retrieved passages. It upgrades to prose composed by Claude and to semantic Hugging Face embeddings when configured.

Enforcing Abstention

The hardest part was getting the model to refuse. Ask it about a modifier that is not in the loaded manuals and an early version would answer from its own background knowledge or stretch a nearby passage to fit. That is exactly the failure a billing tool cannot have. I tightened the grounding so the model abstains and states what it does cover. This turned the tool’s most serious vulnerability into one of its strongest features.

Evaluation

Most RAG demos ship with no measurement. This one includes an evaluation harness over a small gold set. The harness reports retrieval quality (hit@k and MRR) alongside a grounding check that uses an LLM as a judge. The question “is it any good?” now has a numeric answer rather than an impression. Owning evaluation was the workstream I led on my graduate capstone. It is what separates a working RAG demo from a clear account of how good it is and where it fails.

Deployment and Hosting

The retrieval layer is cheap and safe to expose. SourceFetch is containerized and runs as a public demo on Cloud Run in retrieval only mode. The answers composed by Claude make paid API calls so that layer stays gated behind a key. As with the rest of my work: the source is on GitHub and the demo above shows it running.