Customers and service teams
People seeking factual explanations of policy wording, with page citations they can inspect.
02 · RAG · LLM evaluation
QuoteGuard
A guardrailed retrieval system for insurance policy questions: find the right clause, answer from quoted evidence, refuse unsafe requests, and record why each decision was made.
Problem
A policy answer is useful only if its evidence and boundaries are clear.
Insurance product disclosure statements are long, structured and easy to misread. A customer may ask a simple question whose answer depends on a definition, exclusion and section-specific condition several pages apart. A fluent answer without the right clause is worse than a clear refusal.
Information, not advice
The assistant explains the supplied document but does not recommend cover, personalise advice or generate a premium.
Evidence before eloquence
If retrieval does not surface support, the correct behaviour is “I don’t have that information,” not improvisation.
Data
One real policy, three purpose-built evaluation sets.
The source is an Allianz Business Insurance product disclosure statement. A structure-preserving parser recovered 619 sections and their page locations. Evaluation material is kept separate from the document chunks used by the retriever.
- Gold questions
- 66 total: 61 answerable and 5 deliberately unanswerable
- Evidence validation
- Each answerable question is anchored to a supporting span and source page
- Adversarial set
- 40 prompts across seven attack and boundary categories
- Benign controls
- 30 legitimate questions used to measure over-refusal
Approach
Separate retrieval quality, answer grounding and guardrail behaviour.
The public companion parses the policy, retrieves with lexical signals, fuses rankings, reranks candidates and produces a cited answer or explicit refusal. Each safety decision is recorded in a trace so a reviewer can see whether pricing, advice, injection, scope or grounding caused the response.
Implementation scope: the professional implementation listed in my résumé used Azure OpenAI and Azure AI Search, with RAGAS evaluation work and MLflow experiment tracking. The public repository is intentionally portable: it uses TF-IDF/BM25 hybrid retrieval, a cross-encoder reranker, local or API language-model clients and an extractive fallback. The metrics on this page come from the versioned public evaluation artefacts unless stated otherwise.
Why the distinction is explicit
Azure, RAGAS and MLflow are relevant experience, but the public repository does not pretend to contain proprietary work code. The open companion demonstrates the same evaluation logic with inspectable components.
System architecture
Guard before retrieval, ground after generation.
- Input guardDetect advice, pricing, injection and obvious personal data.
- Hybrid retrievalCombine TF-IDF and BM25 candidates with rank fusion.
- RerankUse a cross-encoder to reorder the most relevant passages.
- AnswerGenerate or extract only from supplied policy evidence.
- Output checksValidate citations, grounding and advice leakage.
The browser demo keeps the interaction entirely client-side. It uses BM25 plus heuristic guards and extractive answers; it omits the language-model synthesiser, cross-encoder and learned injection detector. That keeps the demo free and inspectable while labelling what is and is not faithful to the full pipeline.
Evaluation method
Test whether the clause was found before grading the answer built from it.
Retrieval evaluation: the 61 answerable questions are run through each retrieval configuration. A hit is defined by overlap with the gold supporting page. Recall@5 asks whether the answer page appears anywhere in the top five; hit@1, MRR@5 and nDCG@5 capture rank quality. Bootstrap confidence intervals are retained in the project artefacts.
Guardrail evaluation: 40 adversarial prompts and 30 benign controls are scored deterministically. Safe-rate measures correct handling of risky prompts; benign false-positive rate catches a system that becomes “safe” by refusing useful questions.
Answer evaluation: outputs record whether a cited page was actually retrieved and preserve the evidence shown to the answerer. In the Azure work, RAGAS experiments and MLflow runs were used to compare broader response-quality dimensions; those scores are not presented here because they are not in the public repository artefacts.
Results
Hybrid retrieval found the support; one guardrail iteration closed the tested gaps.
0.803
Recall@5 · hybrid retrieval plus reranking · n=61
0.632
nDCG@5 · rewards placing the supporting passage higher
72.5 → 100%
Safe-rate on the 40-prompt tuned adversarial suite
| Retriever | Hit@1 | Recall@5 | MRR@5 | nDCG@5 |
|---|---|---|---|---|
| TF-IDF | 0.410 | 0.689 | 0.509 | 0.540 |
| Dense (bge-small) | 0.377 | 0.607 | 0.466 | 0.498 |
| TF-IDF + rerank | 0.475 | 0.738 | 0.566 | 0.590 |
| Hybrid + rerank | 0.492 | 0.803 | 0.596 | 0.632 |
Failure cases & trade-offs
The strongest lesson came from an evaluation set that changed the winner.
The first benchmark was too small
TF-IDF looked strongest on an early 20-question set. On 61 answerable questions, BM25 clearly beat it. The earlier result had been overfit to a narrow sample.
Dense retrieval underperformed
Dense embeddings were weakest on this term-heavy policy. That does not imply dense retrieval is generally worse; it means the corpus and questions rewarded exact terminology.
The safety result is tuned, not universal
The 100% safe-rate is on 40 prompts used during one tighten-and-rerun cycle. It is evidence that the identified gaps were closed, not proof against unseen attacks.
Retrieval is not answer correctness
Recall@5 shows that supporting pages were surfaced. It does not by itself prove that every generated answer was complete, faithful or appropriately qualified.
One document limits generalisation
Layout, terminology and policy structure vary by insurer and product. Parser and retrieval choices need retesting across a broader document set.
What I would improve next
Add independent evaluation and production observability.
- Freeze an unseen adversarial holdout and commission independent prompt attacks.
- Evaluate multiple policies, scanned PDFs, tables and cross-document questions.
- Expand end-to-end answer grading for completeness, citation entailment and unsupported claims.
- Measure latency and cost by stage in both the portable and Azure deployments.
- Monitor retrieval drift, refusal rates and citation failures in MLflow or an equivalent audit trail.
Live demo & source
Ask a policy question — or try to cross a boundary.
This lightweight demo runs entirely in your browser. It loads the public policy chunks only when opened, uses BM25 retrieval and shows its guardrail trace and cited evidence. No question is sent to an API.
Load QuoteGuard in this page
Faithful but lighter: this browser version omits LLM synthesis, the cross-encoder reranker and the learned injection model. It demonstrates retrieval, heuristic guardrails, extractive grounding and citation checks.
Open this panel to load the in-browser engine.