Engineering Blog · Data Engineering
Entity Resolution Is the Hidden Problem Behind Enterprise AI
Before an AI system can reason about business data, it must know when different values refer to the same thing.
Five values or one entity?
SS304
304 SS
Stainless Steel 304 → Stainless Steel 304
Grade 304 Stainless Steel
AISI 304Those strings can represent one material. If a system treats them as five values, search fragments, analytics double-count, and retrieval misses relevant records.
Extraction and resolution solve different problems
Entity extraction finds a value. Entity resolution determines the real-world concept it represents: extracted SS304, canonical entity Stainless Steel 304. The distinction extends the structured-data foundation from finding mentions to establishing identity.
The problem appears everywhere
Product catalogs, customers, suppliers, healthcare entities, financial institutions, addresses, companies, materials, standards, part numbers, job titles, and locations all accumulate aliases, errors, abbreviations, and legacy identifiers.
Narrow the possible matches
Observed value
↓
Possible identities
↓
Contextual validation
↓
Canonical entityComparing every value with every known entity does not scale. Common resolution systems narrow the search space, then use lexical, semantic, contextual, or governed reference information to evaluate plausible matches.
Normalize deterministically first
Lowercasing, punctuation removal, whitespace and unit normalization, known-abbreviation expansion, and legal-suffix normalization reduce avoidable variation. These operations are cheap, testable, and should usually precede embedding or LLM inference.
Embeddings help—but similarity is not identity
Embeddings help discover aliases, generate candidates, and interpret unseen terminology. Yet Stainless Steel 304 and Stainless Steel 316 are semantically close and materially different. Exact grades, identifiers, and context must constrain semantic matching.
Context determines identity
Manufacturer, category, country, unit, source, neighbouring attributes, and domain change the answer. “Mercury” can mean an element, planet, brand, company, or product. Context helps distinguish semantically related but different identities.
Value + context
↓
Possible identity
↓
Validation
├── Resolved
└── Review neededGoverned canonical knowledge compounds value
A canonical record separates a stable identity from its changing display names and aliases. Provenance and review history help teams trust verified mappings. Once an unknown value is resolved and governed, future occurrences may require less interpretation.
Unknown value → Verified resolution → Governed knowledge
↓
Easier future matchingFailure modes and production impact
Over-merging destroys distinctions; under-merging fragments the truth. Stale aliases, missing context, multilingual values, and conflicting source systems require versioning and stewardship. Poor resolution degrades RAG retrieval, analytics, search, deduplication, recommendations, integration, reporting, and ML training.
Conclusion
Before an AI system can reason about business data reliably, it must know when two pieces of data refer to the same thing. That requires governed data engineering, not a one-off fuzzy match.