Customers and service teams
People seeking factual explanations of policy wording, with page citations they can inspect.
02 · RAG · LLM evaluation
QuoteGuard
I built a policy-question assistant that finds supporting clauses, cites its sources and shows why it answers or refuses.
Problem
Find the clause behind the answer.
A simple insurance question can depend on definitions, exclusions and conditions several pages apart. I built QuoteGuard to retrieve those passages, answer with page references and refuse requests for advice or pricing. I tested retrieval and refusals separately so I could check each part of the system.
Information, not advice
The assistant explains the supplied document but does not recommend cover, personalise advice or generate a premium.
Require supporting text
If retrieval finds no support, the assistant should say it cannot answer from the document.
Data
One real policy, three purpose-built evaluation sets.
The source is an Allianz Business Insurance product disclosure statement. A structure-preserving parser recovered 619 sections and their page locations. Evaluation material is kept separate from the document chunks used by the retriever.
- Gold questions
- 66 total: 61 answerable and 5 deliberately unanswerable
- Evidence validation
- Each answerable question is anchored to a supporting span and source page
- Adversarial set
- 40 prompts across seven attack and boundary categories
- Benign controls
- 30 legitimate questions used to measure over-refusal
Approach
Separate retrieval quality, answer grounding and guardrail behaviour.
I combined two text-search methods, then reranked the candidate passages before producing a cited answer. Each response includes a trace showing which checks passed or caused a refusal: pricing, advice, prompt injection, document scope or supporting evidence.
Implementation scope: the professional implementation listed in my résumé used Azure OpenAI and Azure AI Search, with RAGAS evaluation work and MLflow experiment tracking. The public repository is intentionally portable: it uses TF-IDF/BM25 hybrid retrieval, a cross-encoder reranker, local or API language-model clients and an extractive fallback. The metrics on this page come from the versioned public evaluation artefacts unless stated otherwise.
Public source
The linked repository contains the portable pipeline, evaluation code and saved results. The professional Azure implementation is separate and is not included.
System architecture
Check the request, find evidence, then check the answer.
- Input guardDetect advice, pricing, injection and obvious personal data.
- Hybrid retrievalCombine TF-IDF and BM25 candidates with rank fusion.
- RerankUse a cross-encoder to reorder the most relevant passages.
- AnswerGenerate or extract only from supplied policy evidence.
- Output checksValidate citations, grounding and advice leakage.
The browser demo uses BM25 search, rule-based checks and quoted passages. It omits the language model, cross-encoder and learned injection detector; all processing stays in your browser.
Evaluation method
Test whether the clause was found before grading the answer built from it.
Retrieval evaluation: the 61 answerable questions are run through each retrieval configuration. A hit is defined by overlap with the gold supporting page. Recall@5 asks whether the answer page appears anywhere in the top five; hit@1, MRR@5 and nDCG@5 capture rank quality. Bootstrap confidence intervals are retained in the project artefacts.
Guardrail evaluation: 40 adversarial prompts and 30 benign controls are scored deterministically. Safe-rate measures correct handling of risky prompts; benign false-positive rate catches a system that becomes “safe” by refusing useful questions.
Answer evaluation: outputs record whether a cited page was actually retrieved and preserve the evidence shown to the answerer. In the Azure work, RAGAS experiments and MLflow runs were used to compare broader response-quality dimensions; those scores are not presented here because they are not in the public repository artefacts.
Results
Retrieval improved, and the revised checks passed the tuned prompt suite.
0.803
Recall@5 · hybrid retrieval plus reranking · n=61
0.632
nDCG@5 · rewards placing the supporting passage higher
72.5 → 100%
Safe-rate on the 40-prompt tuned adversarial suite
| Retriever | Hit@1 | Recall@5 | MRR@5 | nDCG@5 |
|---|---|---|---|---|
| TF-IDF | 0.410 | 0.689 | 0.509 | 0.540 |
| Dense (bge-small) | 0.377 | 0.607 | 0.466 | 0.498 |
| TF-IDF + rerank | 0.475 | 0.738 | 0.566 | 0.590 |
| Hybrid + rerank | 0.492 | 0.803 | 0.596 | 0.632 |
Failure cases & trade-offs
A larger question set changed which retriever I would choose.
The first benchmark was too small
TF-IDF looked strongest on an early 20-question set. On 61 answerable questions, BM25 clearly beat it. The earlier result had been overfit to a narrow sample.
Dense retrieval underperformed
Dense embeddings were weakest on this term-heavy policy. That does not imply dense retrieval is generally worse; it means the corpus and questions rewarded exact terminology.
The safety result is tuned, not universal
The 100% safe-rate is on 40 prompts used during one tighten-and-rerun cycle. It is evidence that the identified gaps were closed, not proof against unseen attacks.
Retrieval is not answer correctness
Recall@5 shows that supporting pages were surfaced. It does not by itself prove that every generated answer was complete, faithful or appropriately qualified.
One document limits generalisation
Layout, terminology and policy structure vary by insurer and product. Parser and retrieval choices need retesting across a broader document set.
What I would improve next
Add independent evaluation and production observability.
- Freeze an unseen adversarial holdout and commission independent prompt attacks.
- Evaluate multiple policies, scanned PDFs, tables and cross-document questions.
- Expand end-to-end answer grading for completeness, citation entailment and unsupported claims.
- Measure latency and cost by stage in both the portable and Azure deployments.
- Monitor retrieval drift, refusal rates and citation failures in MLflow or an equivalent audit trail.
Live demo & source
Ask a policy question — or try to cross a boundary.
This lightweight demo runs entirely in your browser. It loads the public policy chunks only when opened, uses BM25 retrieval and shows its guardrail trace and cited evidence. No question is sent to an API.
Load QuoteGuard in this page
Faithful but lighter: this browser version omits LLM synthesis, the cross-encoder reranker and the learned injection model. It demonstrates retrieval, heuristic guardrails, extractive grounding and citation checks.
Open this panel to load the in-browser engine.