Legal Text Anonymiser

The project uses the Text Anonymization Benchmark: 1,268 annotated European Court of Human Rights judgments. Its annotations distinguish entity types such as people, organisations and case codes, and whether an item is a direct identifier, quasi-identifier or does not require masking.

The held-out evaluation contains 555 annotator-document rows covering 127 unique judgments. That distinction matters: presenting those 555 rows as 555 independent judgments would overstate the breadth of the test data.

I normalised every detector to the same contract: text in, typed character spans out. This made spaCy, Presidio, three fine-tunes and their voting rules comparable through one evaluator. The baseline exposed a missing category: spaCy had 0% recall on case-file codes. A custom recogniser helped that label, but noisy organisation predictions kept Presidio behind the fine-tunes. Detection feeds three product modes:

Detection: entity spans are scored with partial-match and exact-match precision, recall and F1 on the held-out annotations. Cluster-bootstrap confidence intervals resample the 127 unique judgments, not the 555 annotation rows. Repeated annotations of the same text are not independent observations. Partial matching gives credit when the right entity is found with an imperfect boundary; exact matching exposes those boundary errors.

Mosaic risk: after masking direct identifiers, the analysis selects increasingly many quasi-identifiers and asks when a record becomes unique within this corpus. This is a controlled corpus-uniqueness study, not a real-world re-identification probability.

Detection results below use 555 annotation rows covering 127 judgments. The ensemble votes over saved predictions from RoBERTa, LegalBERT and Longformer; the hosted Studio uses RoBERTa alone at 512/64, while the ensemble rows describe separate offline experiments.

Majority voting improves precision while recall is 0.0015 below RoBERTa. Union catches more mentions and has lower F1: the preferred rule depends on the cost of a missed identifier. Partial overlap can earn detection credit while leaving part of a name visible, so F1 alone cannot validate release quality.

Evidence: scores and cluster confidence intervals · ensemble evaluation.

Finding names, preparing useful text and avoiding missed identifiers are different jobs. These experiments show how the design could change when the priority changes.

Test scope: 555 annotation rows across 127 unique legal judgments. The small model was run on those judgments; the other policies replay saved predictions. These are exploratory comparisons on the existing test set, not fresh external validation or deployed features.

If the task is to extract people and organisations for indexing or review, keep the entity detector and omit the privacy-risk scorer, generalisation loop and identity vault. This is named-entity recognition (NER), with no claim that its output is anonymous.

Test run: the small, non-transformer en_core_web_sm model versus the existing fine-tuned RoBERTa, both restricted to PERSON and ORG. Here the target is every annotated name, including public organisations the privacy benchmark permits leaving visible.

The small model finds more of this broader target, but produces more false positives. RoBERTa was trained for the masking policy, which excludes many public institutions; it needs retraining for an “extract every name” brief. On just the 6,084 names marked for masking, the ranking is different: RoBERTa reaches 0.822 F1 versus 0.386 for the small model.

Possible adaptation: use the small model for a first-pass extraction assistant, with correction by a reader. Distil or retrain a compact model on the actual extraction target before treating it as a replacement. Latency and memory savings have not been benchmarked here.

If a missed name costs more than an unnecessary redaction, change the policy. Start by covering every span proposed by any of the three fine-tunes. Then try a broader rule: remove each sentence-like passage containing a detection.

A high F1 score—even 0.90—does not mean every document is clean. This simulation instead checks whether every character of a target name is removed, and how much text is lost in doing so.

Headline scope: the 6,084 PERSON/ORG mentions marked DIRECT or QUASI, across 555 annotation rows. Full removal is stricter than partial-match detection recall.

Anonymiser Studio lets you compare Source, Redact, Anonymise and Pseudonymise, inspect each detected phrase, and restore names using the demo identity map. It runs the RoBERTa fine-tune on CPU with a 512/64 inference window; the three-model ensemble results above are a separate comparison. Use the supplied synthetic example or your own synthetic text. Input and identity mappings are processed on Hugging Face infrastructure, not on your computer. Do not paste confidential information into this public demo.