← Insights

How Microsoft Presidio Detects PII: Questions Developers Ask

Answers to the questions developers ask about Presidio's recognizers, NER models and redaction, from real threads, with the sources.

8 min read
How Microsoft Presidio Detects PII: Questions Developers Ask

Microsoft Presidio is an open-source Python library. Its analyzer finds personal data in text, and its anonymizer redacts, masks or replaces what the analyzer found. It was created at Microsoft, but it is no longer a Microsoft product: the project now lives under the Data Privacy Stack organization as a community-governed project, and its FAQ says it is “not an official product of any company.” This article answers the questions developers ask most often about it, taken from the threads where they asked them. It ends with where phi-boundary-gate, a Tigerless Labs tool that we make, fits.

How do I make Presidio detect and mask names it misses?

Check the model before you change the rules. Presidio's PERSON recognizer runs on a spaCy model. In the Stack Overflow thread that asked this question, a banking team found the recognizer caught Western names but missed Indian ones such as “Priya Sharma” and “Rajesh Gupta.” A reply could not reproduce the miss on Presidio's demo site, where most NER models found the same names. It suggested checking the spaCy model and its version first, and trying transformers before training a custom model.

Run that check on your own data before anything else:

  1. Install the two packages with `pip install presidio_analyzer` and `pip install presidio_anonymizer`.
  2. Download the English spaCy model the analyzer loads by default with `python -m spacy download en_core_web_lg`.
  3. Analyze a sentence from your own data with `entities=["PERSON"]` and read every result it returns.
  4. Repeat the run with the model and version you will ship, and compare the two before you change any recognizer.

Patterns are the wrong fix for names. The same asker tried a list of Indian names in a pattern recognizer and found it too rigid for variations and for first and last names written together. Training a custom spaCy model was possible, but the asker judged it heavy for production and a standing maintenance cost.

How do I add a new entity recognizer to Presidio?

Write a recognizer, then add it to the analyzer's registry. For lists and simple patterns, PatternRecognizer is enough. The project's guide on adding recognizers starts with a deny-list:

from presidio_analyzer import PatternRecognizer

titles_recognizer = PatternRecognizer(supported_entity='TITLE',
                                      deny_list=['Mr.', 'Mrs.', 'Miss'])

Then register it with analyzer.registry.add_recognizer(titles_recognizer) and pass TITLE in the entities list of your analyze call.

Language is the trap. In a second Stack Overflow thread, a developer added an entity to a pipeline that handles English and Spanish, and the answer pointed to the constructor: PatternRecognizer takes supported_language, which defaults to 'en'. A recognizer built without it is English-only, so it is stale when you analyze Spanish text. Set the language on every recognizer you add for another language, and test each language on its own sample.

Can I load custom recognizers into the REST API from a YAML file?

Yes, but the path differs between the Python package and the container. A GitHub issue describes a user who loaded a YAML file of recognizers into the package, then could not see them in the REST service's recognizers list. The maintainers answered that YAML loading works in package use, that the container needed its own code change, and that ad-hoc recognizers were the way to send them with a request. The issue was closed in October 2024 with the note that YAML config is now supported.

In the package, the loading code from the project's docs looks like this:

from presidio_analyzer import AnalyzerEngine, RecognizerRegistry

registry = RecognizerRegistry()
registry.load_predefined_recognizers()
registry.add_recognizers_from_yaml('recognizers.yaml')
analyzer = AnalyzerEngine(registry=registry)

For one request, skip the file. The analyzer's REST API accepts ad_hoc_recognizers in the request body, and they apply only to that request:

{
  "text": "Mr. John Smith's drivers license is AC432223",
  "language": "en",
  "ad_hoc_recognizers": [
    {
      "name": "Mr. Recognizer",
      "supported_language": "en",
      "deny_list": ["Mr", "Mr.", "Mister"],
      "supported_entity": "MR_TITLE"
    }
  ]
}

Use ad-hoc recognizers for a pattern that changes per customer or per request. Use a YAML file for the same list on every run.

How do I add custom anonymization for a table?

Use presidio-structured for tables, and give it an analyzer you have configured yourself. The answer in issue 1353 shows the pattern: build the analyzer with your recognizers, pass it to PandasAnalysisBuilder, and give StructuredEngine an operator for each entity. The thread's example replaces names with a fixed value and passes emails through a custom function. Trimmed to its core, it looks like this:

import pandas as pd
from faker import Faker
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer.entities import OperatorConfig
from presidio_structured import StructuredEngine, PandasAnalysisBuilder

fake = Faker()
sample_df = pd.DataFrame({'name': ['Arthur', 'David'],
                          'email': ['a@example.com', 'd@example.com']})
analyzer = AnalyzerEngine()
analysis_builder = PandasAnalysisBuilder(analyzer=analyzer)
tabular_analysis = analysis_builder.generate_analysis(sample_df)

operators = {
    'DEFAULT': OperatorConfig('replace', {'new_value': ' '}),
    'PERSON': OperatorConfig('replace', {'new_value': 'REDACTED'}),
    'EMAIL_ADDRESS': OperatorConfig('custom', {'lambda': lambda x: fake.safe_email()}),
}
pandas_engine = StructuredEngine()
anonymized_df = pandas_engine.anonymize(sample_df, tabular_analysis, operators=operators)
print(anonymized_df)

The FAQ adds that presidio-structured scans datasets for PII and supports redaction of text, cells or columns, so one analyzer configuration can serve both documents and tables.

Redaction kills context, what do you use instead?

The poster in the discussion moved to deterministic, format-preserving replacement, where the same input always maps to the same token. They wrote that aggressive redaction had flattened the signal in their AI support-ticket pipeline: once every customer became a blank and every phone number a placeholder, the model and the people reading it could not tell tickets apart, correlate a repeat caller, or spot patterns across one person's history. A stable token keeps those links without exposing the raw value, and it stays structurally valid, so downstream parsers still accept it.

The catch is the one the poster named. The mapping keys on the exact string, so “Dan johnson” and “Dan Johnson”, or two spellings of one email address, tokenize differently unless you normalize first. Consistency across a whole corpus and reversal limited to authorized roles were not there out of the box either, and the poster said that layer is something you build and maintain yourself. Presidio's extension points for this work are its custom operators; its anonymizer's operators include hash and encrypt alongside replace, redact and mask.

When is Presidio the wrong tool?

Presidio's FAQ answers this itself. It describes Presidio as a library meant to be customized, and says hosted PII services such as Azure AI Language and Amazon Comprehend “often have better entity coverage or accuracy than Presidio.” Its own suggestion is to run Presidio alongside such a service when your data has entity types the service does not support. Choose a hosted service when you want a managed model and do not plan to tune recognizers. Choose Presidio when you need to add entities, control the rules, or run inside your own environment.

The FAQ also says detection is automated, and that “there is no guarantee that Presidio will find all sensitive information,” so additional systems and protections should be employed. Its REST endpoints have no built-in authentication by design, so put an authentication layer in front of them before they leave your own network. A detector is one control, not the whole control: test every miss you find on your own samples before you rely on its counts.

Where does phi-boundary-gate fit?

We make phi-boundary-gate, a Tigerless Labs project under the MIT license. It follows PHI candidates across the places they travel in an AI system: user messages, retrieved context, tool output, model input, memory, debug logs and provider requests. A YAML policy decides, per category and per layer, whether a value is allowed, redacted or reported as a violation. Its built-in rules cover common identifiers such as phone numbers, member and claim IDs, medical record numbers and addresses, and it has an optional local Presidio NER detector for the names that rules miss.

If your agent keeps memory between sessions, our guide to giving Claude Code memory across sessions is the place to start, because that memory is one of the places the same questions come back.

Install it from PyPI with pip install phi-boundary-gate on Python 3.11 or newer. The repository holds synthetic samples only. Read its limits before you adopt it: it makes no HIPAA-compliance claim, every hit is a PHI candidate that needs human review, and it cannot verify a business associate agreement. A value the detector misses cannot be redacted either, so the detector's coverage sets the gate's coverage. For the wider set of Labs tools, see the Labs group.

Conclusion

Presidio is a capable library with a clear extension model. Most of its hard questions reduce to three: which NLP model you run, how you write recognizers for the entities you care about, and what you do with the output once redaction is done. The answers come from your own data, measured, not from the defaults.

The decision rule: if you need to add entity types, control the rules, and run the detector inside your own environment, Presidio is a sound base. If you need a managed model and no tuning, a hosted PII service is the simpler choice, and you can still test Presidio beside it for the gaps. If you are choosing the coding agent that will write those recognizers, our Codex vs Claude Code comparison covers the trade-offs.

Your next step takes an afternoon: install Presidio, run ten sentences of your own text through the analyzer, and list every miss. If the text is medical, read the phi-boundary-gate limits before you add the gate to a pipeline. For how we check AI-written drafts before they go live, see how we review AI-written blog drafts.

FAQ

Is Microsoft Presidio a Microsoft product?

No. Microsoft created it, and the project has since moved to Data Privacy Stack as a community-governed, vendor-neutral project. Its FAQ says it is not an official product of any company and comes with no warranty or SLA. Its usage terms are set by the repository's license.

Will Presidio find every piece of PII in my data?

No, and the project says so. Its detection is automated, so there is no guarantee that it finds all sensitive information. The FAQ recommends additional systems and protections alongside it.

Can Presidio detect PII in tables?

Yes, through presidio-structured. It scans structured and semi-structured datasets for PII using the analyzer, and it supports redaction of text, cells or columns in a tabular dataset.

How do I install phi-boundary-gate?

Install it from PyPI with pip install phi-boundary-gate. It needs Python 3.11 or newer and is released under the MIT license. Its repository holds synthetic samples only.

Does phi-boundary-gate make a product HIPAA compliant?

No. We make no HIPAA-compliance claim. Every hit is a PHI candidate that needs human review, and the compliance guard enforces only the facts you give it. It cannot verify a business associate agreement itself.

Share this article

Keep reading

11 min read

Should Agent Memory Have an Expiration Date?

A fixed expiry can drop facts that are still true. Here is how TTL, decay and event-based expiry differ, and when a fact should stay valid until replaced.

Read story