Entity resolution on messy public records

If you need to deduplicate company names in a large dataset, a single fuzzy-matching function will not get you there. At a few thousand rows it appears to work. At a few million it is too slow to run on every pair, too loose on short names, too strict on long ones, and it merges companies that merely share a common word.

What works is a pipeline with six stages: normalize the fields, block to generate a manageable set of candidate pairs, compare each pair on several fields, score the comparisons into a match probability, cluster the matched pairs into entities, and review the uncertain cases by hand. Each stage is simple. The quality comes from doing all of them and measuring the result.

This article uses customs shipment records as the worked example, because they are public in several countries and they are about as messy as company data gets.

The problem, concretely

Bill-of-lading style records carry a shipper and a consignee, each as a free-text name and a free-text address, typed by whoever filed the document. There is no company identifier. The same exporter appears as:

SHENZHEN BRIGHTSTAR ELECTRONICS CO.,LTD
Shenzhen Bright Star Electronic Co Ltd
BRIGHTSTAR ELECTRONICS (SHENZHEN) COMPANY LIMITED
SHENZHEN BRIGHTSTAR ELEC CO LTD 3F BLDG 2 BAOAN DIST

The last one has part of the address in the name field, which is common. Meanwhile a different company, "Shenzhen Brightway Electronics Co., Ltd.", is one edit away from the first line and must not be merged.

With one million distinct name-address strings there are roughly 500 billion possible pairs. Whatever you do, you cannot compare them all.

Decide what an entity is before writing code

This is the step most projects skip and then pay for. Is the entity a legal entity, a physical site, or a corporate family? "Acme Logistics GmbH" in Hamburg and "Acme Logistics Inc" in New Jersey are one brand, two legal entities, and possibly one ultimate parent.

Our recommendation: resolve at the lowest level the data supports, usually legal entity at a location, and model hierarchy as a separate relationship on top. You can always roll up. You cannot cleanly split entities that were merged too early.

Stage 1: Normalization

The goal is to remove differences that carry no information while keeping the ones that do.

Unicode, case, punctuation. Apply Unicode normalization, strip diacritics, casefold, and turn punctuation into spaces. Remove periods without adding a space so that "S.A. de C.V." collapses to "sa de cv" and "L.L.C." to "llc".

Legal suffixes. Ltd, Limited, LLC, Inc, GmbH, S.A. de C.V., Co., Ltd., Pte Ltd, Pvt Ltd, Sdn Bhd, and dozens more. Strip them from the matching key, longest first, but keep them in a separate field. A suffix mismatch is weak evidence of a different legal entity: "Acme GmbH" and "Acme LLC" are probably related and probably not the same company.

Abbreviations and noise words. "Intl" and "International", "Mfg" and "Manufacturing", "Elec" and "Electronics". Build the list from your own data by looking at the most frequent tokens; a generic list will miss domain terms.

Transliteration. Names that arrive in non-Latin scripts need a consistent romanization before string comparison means anything. Be aware that romanized names already in the data vary too ("Bright Star" versus "Brightstar", or different romanization schemes for the same characters), which is one reason to compare with spaces removed as well as token by token.

Addresses. Parse free-text addresses into components rather than comparing raw strings. libpostal is the standard open-source choice: a statistical parser trained on OpenStreetMap data, with Python bindings that expose parse_address and expand_address. Once you have a city, postcode, and country, those become strong comparison fields and good blocking keys. Geocoding adds distance as a feature where coverage is good.

A minimal name normalizer using only the standard library:

import re
import unicodedata

LEGAL_SUFFIXES = [
    "sa de cv", "s de rl de cv", "gmbh and co kg", "co ltd", "pvt ltd", "pte ltd",
    "pty ltd", "sdn bhd", "company limited", "gmbh", "limited", "ltd", "llc", "llp",
    "inc", "incorporated", "corp", "corporation", "company", "co", "plc", "sa",
    "srl", "spa", "bv", "nv", "ag", "kg", "ab", "oy", "as",
]
_SUFFIX_RE = re.compile(
    r"\s+(?:" + "|".join(sorted(map(re.escape, LEGAL_SUFFIXES), key=len, reverse=True)) + r")$"
)

def normalize_name(raw: str) -> tuple[str, str]:
    """Returns (matching_key, stripped_legal_suffixes)."""
    s = unicodedata.normalize("NFKD", raw)
    s = "".join(ch for ch in s if not unicodedata.combining(ch))
    s = s.casefold().replace("&", " and ").replace(".", "")
    s = re.sub(r"[^\w\s]", " ", s)
    s = re.sub(r"\s+", " ", s).strip()

    suffixes: list[str] = []
    while m := _SUFFIX_RE.search(s):
        suffixes.insert(0, m.group(0).strip())
        s = s[: m.start()]
    return s, " ".join(suffixes)

normalize_name("SHENZHEN BRIGHTSTAR ELECTRONICS CO.,LTD") returns ("shenzhen brightstar electronics", "co ltd").

Stage 2: Blocking

Blocking generates candidate pairs cheaply so that the expensive comparison runs on millions of pairs, not hundreds of billions. A record gets one or more blocking keys; only records that share a key are compared. Use several keys in parallel, because any single key will miss some true matches.

Technique	Catches	Misses
Prefix of name plus country	Trailing variation, suffix noise	Differences at the start of the name
Rare-token keys	Word-order changes, inserted words	Names made only of common words
Phonetic codes (Soundex, Metaphone)	Spelling variants of pronounceable names	Designed for English; weak elsewhere
Sorted neighbourhood	Near-neighbours in sort order	Anything that sorts far apart
MinHash with LSH on character n-grams	General string similarity at scale	Short names; needs threshold tuning
Embedding nearest-neighbour search	Abbreviations, translations, reordering	Pulls in semantically similar non-matches
Postcode plus first token	Same site, badly typed name	Records with no usable address

def blocking_keys(name_key: str, country: str | None, postcode: str | None,
                  rare_tokens: set[str]) -> set[str]:
    tokens = name_key.split()
    keys: set[str] = set()
    if tokens:
        keys.add(f"pfx:{name_key.replace(' ', '')[:5]}:{country or '??'}")
    keys.update(f"tok:{t}" for t in tokens if t in rare_tokens)
    if postcode and tokens:
        keys.add(f"pc:{postcode}:{tokens[0][:3]}")
    return keys

rare_tokens comes from your data: compute how many records each token appears in and keep those below a cutoff. "brightstar" is a useful key. "shenzhen", "trading", and "international" are not; they produce blocks with hundreds of thousands of members.

Measure blocking on its own. Two numbers: pair completeness (what fraction of known true matches survive blocking) and reduction ratio (how much of the pair space you eliminated). A true match lost at this stage can never be recovered later. Also cap block size; one oversized block can dominate the run time.

Stage 3: Pairwise comparison

For each candidate pair, compute a vector of comparisons rather than one score:

Jaro-Winkler on the name key, which rewards matching prefixes and suits short strings.
Token-set similarity, which ignores word order and duplicated tokens.
TF-IDF cosine on character n-grams, which down-weights common fragments and tolerates typos.
Name with spaces removed, exact or edit distance, for "Bright Star" versus "Brightstar".
Address agreement by component: country, city, postcode, street, plus geographic distance where geocoded.
Legal suffix agreement, disagreement, or missing.
Context, such as overlapping trading partners or product codes. Two similarly named consignees that receive the same goods from the same shipper are very likely the same company.

Libraries such as RapidFuzz provide fast implementations of the string measures. Bucket each comparison into levels (exact, close, somewhat close, different, missing) rather than using raw floats; it makes the next stage more robust.

Stage 4: Scoring

Probabilistic linkage. The Fellegi-Sunter model is the long-standing approach. For each comparison level it estimates how likely that level is among true matches (the m probability) and among non-matches (the u probability). The ratio gives a match weight, the weights sum across fields, and the total converts to a match probability. Parameters can be estimated from unlabelled data with expectation-maximization. Splink, an open-source library from the UK Ministry of Justice, implements this at scale on backends such as DuckDB and Spark, and includes term-frequency adjustments so that agreeing on a rare name counts for more than agreeing on a common one.

Supervised classification. If you have a few thousand labelled pairs, a gradient-boosted classifier on the comparison vector usually does at least as well, and it captures interactions between fields. The cost is the labelling, and the labels must come from the uncertain region, not from random pairs.

Either way, the output is a probability per pair, and you set two thresholds: above the upper one, accept; below the lower one, reject; in between, adjudicate.

Where LLMs and embeddings help, and where they hurt

Help: adjudicating the middle band. The ambiguous pairs are a small fraction of candidates, and they are where a reader with judgment adds value. Give the model both records in full, with addresses and context, ask for a structured verdict (same, different, or unsure) with a short reason, and store the verdict keyed on the pair so that a re-run never asks again or gets a different answer.

Help: embeddings for blocking recall. They catch candidates that string methods miss, such as abbreviations and translated names.

Hurt: running a model over every candidate pair. Even after blocking, that is millions of calls. It is slow and costly, and results are not repeatable unless you persist every verdict.

Hurt: embeddings as the final score. "Acme Trading Shanghai" and "Acme Trading Shenzhen" sit very close in embedding space. Semantic similarity is not identity.

Hurt: world knowledge. A model may "know" that two companies are related and call them the same, merging a parent with its subsidiary, or may confidently assert facts about an obscure firm it has never seen. Instruct it to judge only from the records supplied, and audit a sample of its verdicts like any other labeller.

Stage 5: Clustering

Matched pairs form a graph; entities are groups of connected records. The simplest method, connected components, takes the transitive closure: if A matches B and B matches C, all three merge. That is where over-merging comes from. One bad link between two large clusters fuses them, and chains of individually plausible links can join records that look nothing alike at either end.

Defences, in rough order of effort:

Use a higher threshold for links that join clusters than for links within them.
After forming components, check cohesion: if the weakest pairwise score inside a cluster is very low, split it at the weakest edges.
Use average-linkage or complete-linkage agglomerative clustering instead of connected components, so a record joins a cluster only if it resembles most of its members.
Flag every cluster above a size limit for review. Genuinely huge entities exist, but so do garbage clusters, and they look the same from outside.

Traps specific to trade records

Freight forwarders and NVOCCs. On many bills the named shipper or consignee is a logistics intermediary, not the buyer or seller. They appear on enormous numbers of records and share addresses with their customers' paperwork. Identify them early and treat them as their own entity type.

"To order" consignees. Entries such as "TO ORDER" or "TO THE ORDER OF" a bank are not companies. Filter them before blocking.

Shared addresses. Free-trade-zone agents, registered-office providers, and multi-tenant industrial parks put unrelated companies at one address. Address agreement alone must never be sufficient.

Same name, different entity. Common names recur across cities and countries. Require corroboration from location or context before merging.

Parents and subsidiaries. A strong name match with a different country or legal suffix is more likely a sister company than a duplicate.

Stable IDs and re-runs

New data arrives, the pipeline re-runs, and cluster membership shifts. Downstream users need identifiers that do not.

Assign persistent entity IDs, and on each run map new clusters to previous IDs by greatest overlap.
Record merges and splits in a history table; never reuse a retired ID.
Match incoming records against existing entities first, and re-cluster globally on a slower schedule.
Store human decisions as must-link and cannot-link constraints that every future run honours. Nothing erodes trust faster than a reviewer's correction silently reverting.

Evaluation

Build a labelled pair set, sampled by score band rather than at random. Random pairs are almost all obvious non-matches and tell you nothing.

Pairwise precision and recall at your chosen thresholds. Precision errors (false merges) are usually more damaging than recall errors, because a merged entity corrupts every aggregate built on it.
Cluster-level metrics such as B-cubed precision and recall, which score each record by how pure and how complete its cluster is. Pairwise numbers can look fine while one giant bad cluster ruins the dataset.
Blocking recall, measured separately, as above.
A fixed regression set of known hard cases, run on every change.

Treat the human review queue as part of the system. Rank it by impact (uncertain links touching large clusters first), show both records side by side with the evidence, and feed every decision back in as labels and constraints. This is the same measurement discipline we apply to model-based systems generally, described in the three measurements we insist on.

A checklist

If you have a dataset like this and want a second opinion on the pipeline, get in touch.