Legal operations and knowledge teams
People preparing confidential matter text for review, analysis or an external language model.
01 · NLP · Privacy engineering
Legal Text Anonymiser
A RoBERTa-based system for detecting sensitive facts in legal text, choosing the right privacy treatment, and keeping reversible identity mappings inside the user’s trust boundary.
Problem
Removing a name does not necessarily remove an identity.
Legal teams can use language models to summarise files or build chronologies, but those files contain names, case references and contextual details that may identify a person when combined. The system therefore has two jobs: detect direct identifiers reliably, and make the remaining re-identification risk visible.
Misses matter more than clutter
Recall is important because a missed direct identifier can defeat the privacy treatment; false positives still damage readability and utility.
Identity stays local
In reversible mode, the mapping between real entities and stable labels never needs to leave the organisation.
Data
A legal benchmark with both entity labels and privacy roles.
The project uses the Text Anonymization Benchmark: 1,268 annotated European Court of Human Rights judgments. Its annotations distinguish entity types such as people, organisations and case codes, and whether an item is a direct identifier, quasi-identifier or does not require masking.
The held-out evaluation contains 555 annotator-document rows covering 127 unique judgments. That distinction matters: presenting those 555 rows as 555 independent judgments would overstate the breadth of the test data.
- Corpus
- 1,268 annotated legal judgments
- Held-out set
- 555 annotation rows across 127 unique judgments
- Label dimensions
- Entity type plus DIRECT, QUASI and NO_MASK privacy roles
- Evaluation unit
- Entity spans, with both partial and exact matching reported
Approach
Detect first, then choose a treatment that matches the threat model.
I put each detector behind the same span interface, measured a general-purpose baseline, and fine-tuned RoBERTa on the legal labels. Detection feeds three distinct product modes:
Remove direct identifiers
Replace names, organisations, dates, IDs and case codes when direct disclosure is the immediate risk.
Generalise indirect details
Broaden ages, locations and dates until the combination is less distinctive within the reference corpus.
Preserve relationships
Map entities to stable tokens so a model can follow who did what, then restore names locally if authorised.
Practical distinction
Redaction removes visible identifiers. Anonymisation tries to reduce the chance that remaining information can single someone out. Pseudonymisation is reversible and therefore still personal data; it protects the external processing step, not every downstream use.
System architecture
The sensitive mapping remains on the trusted side of the boundary.
- Ingest locallyRead legal text inside the organisation’s environment.
- Detect entitiesRoBERTa returns typed spans and confidence values.
- Assess roleDirect and quasi-identifiers take different paths.
- TransformRedact, generalise or replace with stable pseudonyms.
- Restore locallyOnly authorised reversible flows use the local vault.
Evaluation method
Model quality and privacy risk are measured as separate questions.
Detection: entity spans are scored with partial-match and exact-match precision, recall and F1 on the held-out annotations. Bootstrap confidence intervals quantify uncertainty. Partial matching gives credit when the right entity is found with an imperfect boundary; exact matching exposes those boundary errors.
Mosaic risk: after masking direct identifiers, the analysis selects increasingly many quasi-identifiers and asks when a record becomes unique within this corpus. This is a controlled corpus-uniqueness study, not a real-world re-identification probability.
Results
Better detection helped; it did not make contextual risk disappear.
0.8510
Partial-match F1 · 95% bootstrap CI 0.8367–0.8649
0.7842
Exact-match F1 · boundary errors included
86.4%
Overall recall at the selected 384/64 inference configuration
| Measure | Result | Interpretation |
|---|---|---|
| RoBERTa partial-match F1 | 0.8510 | Selected trained inference configuration |
| RoBERTa exact-match F1 | 0.7842 | More stringent span-boundary score |
| Unique after one selected quasi fact | 81.2% | Within this corpus only |
| Unique after three selected quasi facts | 96.5% | Within this corpus only |
| Never unique in the tested sequence | 0.9% | 12 of 1,268 judgments |
Failure cases & trade-offs
Several plausible improvements did not survive evaluation.
Ensembling made performance worse
A union-style three-model ensemble scored 0.553 F1 because weak organisation predictions from one member polluted every vote. Diversity is not useful when one member is systematically poor on a label.
Legal pretraining was not a clear win
LegalBERT reached 0.849 F1, effectively tied with RoBERTa at 0.851. Legal-domain pretraining did not justify extra complexity for this task.
Coreference rules barely moved recall
A post-processor intended to recover shorthand mentions changed macro recall by only 0.0001 because the detector already captured most cases where the rule could help.
Privacy and utility pull in opposite directions
Generalising more detail reduces uniqueness but can remove facts a summariser needs. The right threshold depends on the use case, adversary and review process.
The risk study has a narrow claim
Corpus uniqueness is evidence of mosaic risk, not proof of real-world identification. External data availability and attacker knowledge were not modelled.
What I would improve next
Validate the workflow, not only the detector.
- Run external validation on a different legal jurisdiction and document style.
- Measure how each privacy mode changes downstream summarisation and information-extraction utility.
- Add reviewer queues, span-level overrides and audit logs for human accountability.
- Threat-model storage, key rotation and access control around the pseudonym vault.
- Expand risk measurement beyond within-corpus uniqueness using explicitly documented auxiliary data.
Demo & source
Inspect the transformation, then inspect the implementation.
The hosted Hugging Face demo lets you compare transformation modes on supplied or synthetic text. Do not paste confidential information into a public demo.
Load the hosted demo on this page
The external application is loaded only after this panel is opened.