Fuzzy Matching Duplicate FNOL Submissions

This page is a focused implementation guide under Duplicate Claim Detection at Ingestion Scale: it works through the concrete rapidfuzz scoring step that turns a candidate pair of First Notice of Loss records into a numeric, explainable, threshold-driven duplicate verdict — the part where a nickname, a transposed VIN digit, and a one-day loss-date drift all have to be reconciled into a single defensible score.

Once blocking has narrowed the field to a handful of candidate pairs, the hard problem is comparing two dirty records without either over-matching or missing a real duplicate. The same loss submitted twice rarely arrives identical. A mobile-app FNOL says Robert Q. Mendez; the broker portal says Bob Mendez. One channel records the VIN as 1HGCM82633A004352, the other transposes two digits into 1HGCM82633A003452. The loss date differs by a day because one intake stamped it in the claimant’s local timezone and the other in UTC. The loss narratives — rear-ended at the Elm St light versus hit from behind, Elm Street intersection — share meaning but almost no exact tokens.

A single fused string comparison across the concatenated record collapses under this. Bob Mendez rear-ended at the Elm St light against Robert Q. Mendez hit from behind, Elm Street intersection scores low as one blob, even though every field is a plausible match. The defect classes are concrete: a scorer mismatch (using an order-sensitive ratio on a name where word order legitimately varies), a normalization gap (comparing 1HGCM82633A004352 against 1hgcm 82633a004352 as raw strings), and a boundary flip (a weighted score landing exactly on the cutoff and resolving differently across runs). This guide fixes all three: the right rapidfuzz scorer per field, deterministic normalization before comparison, and an inclusive, rounded threshold decision.

This pattern targets Python 3.10+ and uses rapidfuzz for the string comparisons and structlog for the audit line. It assumes the blocking stage described in the parent duplicate claim detection guide has already produced candidate pairs, so this code scores a pair rather than searching the book. Pin the scorer library so its ratio semantics cannot drift:

Copy
python -m venv .venv && source .venv/bin/activate
pip install "rapidfuzz==3.*" "structlog==24.*"

Fuzzy scoring is only as good as the strings fed to it. Normalize deterministically first: upper-case and strip separators from the VIN, canonicalize names and addresses to lower-case ASCII with collapsed whitespace, and parse the loss date to a date object so a comparison is a day-delta, not a string diff. Never score raw intake text.

Copy
from __future__ import annotations

import re
import unicodedata
from datetime import date

_WS = re.compile(r"\s+")
_NON_ALNUM = re.compile(r"[^a-z0-9 ]")
_ADDR_ABBR = {"street": "st", "avenue": "ave", "road": "rd", "boulevard": "blvd",
              "drive": "dr", "lane": "ln", "court": "ct", "apartment": "apt"}


def canon_text(value: str) -> str:
    """Lower-case, strip accents/punctuation, expand nothing, collapse spaces."""
    folded = unicodedata.normalize("NFKD", value).encode("ascii", "ignore").decode()
    return _WS.sub(" ", _NON_ALNUM.sub(" ", folded.lower())).strip()


def canon_address(value: str) -> str:
    """Canonical text plus street-suffix folding so 'Street' == 'St'."""
    tokens = canon_text(value).split()
    return " ".join(_ADDR_ABBR.get(t, t) for t in tokens)


def canon_vin(value: str) -> str:
    """VINs are 17 alphanumerics; strip everything else and upper-case."""
    return re.sub(r"[^A-Z0-9]", "", value.upper())


def parse_loss_date(value: str) -> date:
    """ISO 8601 only; raise on anything else rather than guessing a format."""
    return date.fromisoformat(value.strip())

Each field has a shape, and the scorer must match it. token_set_ratio ignores word order and duplicate tokens, which is exactly right for names (Bob Mendez versus Robert Q. Mendez) and free-text narratives where extra words appear. token_sort_ratio sorts tokens before comparing, suiting addresses where component order is stable but spacing varies. A VIN is a near-exact identifier, so a plain ratio (edit distance) catches a one- or two-character transposition without treating unrelated VINs as similar. The loss date is numeric proximity, not text.

Copy
from rapidfuzz import fuzz


def score_name(a: str, b: str) -> float:
    return float(fuzz.token_set_ratio(a, b))


def score_address(a: str, b: str) -> float:
    return float(fuzz.token_sort_ratio(a, b))


def score_vin(a: str, b: str) -> float:
    """Exact VINs score 100; a transposed digit degrades gracefully via edit distance."""
    if not a or not b:
        return 0.0
    if a == b:
        return 100.0
    return float(fuzz.ratio(a, b))


def score_loss_text(a: str, b: str) -> float:
    return float(fuzz.token_set_ratio(a, b))


def score_date(a: date, b: date, *, window_days: int = 3) -> float:
    """100 for same day, linearly decaying to 0 at window_days apart."""
    delta = abs((a - b).days)
    if delta >= window_days:
        return 0.0
    return round(100.0 * (1 - delta / window_days), 2)

Multiply each sub-score by a fixed weight, sum, round to a fixed precision, and compare against two thresholds with inclusive boundaries. Rounding before the comparison is what removes the boundary flip: two runs on the same inputs produce byte-identical scores. Retain the per-field breakdown in the result so the verdict is explainable.

Copy
from dataclasses import dataclass
from enum import Enum

import structlog

log = structlog.get_logger("fraud.dedup.fuzzy")

WEIGHTS = {"name": 0.30, "address": 0.20, "vin": 0.25, "loss_text": 0.15, "date": 0.10}


class Verdict(str, Enum):
    DUPLICATE = "DUPLICATE"
    AMBIGUOUS = "AMBIGUOUS"
    UNIQUE = "UNIQUE"


@dataclass(frozen=True, slots=True)
class FuzzyMatch:
    field_scores: dict[str, float]
    weighted_score: float
    verdict: Verdict


def classify(scores: dict[str, float], *, high: float, low: float) -> FuzzyMatch:
    if not (0 <= low < high <= 100):
        raise ValueError(f"require 0 <= low ({low}) < high ({high}) <= 100")
    weighted = round(sum(scores[k] * w for k, w in WEIGHTS.items()), 2)
    if weighted >= high:
        verdict = Verdict.DUPLICATE
    elif weighted >= low:
        verdict = Verdict.AMBIGUOUS
    else:
        verdict = Verdict.UNIQUE
    log.info("dedup.fuzzy.classified", weighted=weighted, verdict=verdict.value,
             **{f"s_{k}": v for k, v in scores.items()})
    return FuzzyMatch(dict(scores), weighted, verdict)

The three defect classes each get an explicit test: a name that varies only by order and nickname must still score as a duplicate, a normalized VIN comparison must survive a transposition, and the threshold boundary must resolve deterministically.

Copy
from datetime import date


def test_name_order_and_nickname_match() -> None:
    assert score_name(canon_text("Robert Q. Mendez"), canon_text("Bob Mendez")) < 100
    # token_set tolerates the reorder + extra token; full name vs full name is high
    assert score_name(canon_text("Robert Mendez"), canon_text("Mendez Robert")) == 100.0


def test_vin_transposition_scores_high_but_not_exact() -> None:
    a = canon_vin("1HGCM82633A004352")
    b = canon_vin("1hgcm82633a-003452")   # two digits transposed + dirty formatting
    s = score_vin(a, b)
    assert 80.0 < s < 100.0


def test_threshold_boundary_is_deterministic() -> None:
    scores = {"name": 92.0, "address": 88.0, "vin": 100.0,
              "loss_text": 80.0, "date": 100.0}
    m = classify(scores, high=92.0, low=78.0)
    assert m.verdict is Verdict.DUPLICATE          # inclusive: exactly-high clears
    # same inputs, second call → identical score, no float drift
    assert classify(scores, high=92.0, low=78.0).weighted_score == m.weighted_score


def test_date_decay_is_zero_outside_window() -> None:
    assert score_date(date(2026, 7, 1), date(2026, 7, 1)) == 100.0
    assert score_date(date(2026, 7, 1), date(2026, 7, 10)) == 0.0

Run the suite with python -m pytest -q. The decisive property is that every score is reproducible: the weighted total is rounded to a fixed precision, so the boundary comparison has no tolerance band and the verdict replays exactly from the recorded sub-scores.

Because the verdict is a pure function of the normalized fields, the field weights, and the two thresholds, it is fully reconstructable from the audit record. Persist the per-field sub-scores, the weighted total, the threshold set, and the rule version alongside both claim ids before any merge is published — written to the same append-only trail the audit log schema design guide specifies. When a policyholder disputes a suppressed claim, the sub-score breakdown (name 94, vin 100, date 100) shows exactly why the two were treated as one, which is what keeps an automated merge defensible under NAIC unfair-claims-settlement scrutiny. Never store only the final verdict; a bare DUPLICATE cannot be re-examined.

• Order-sensitive scorer under-matches names. Symptom: Bob Mendez versus Mendez, Bob scores low and a real duplicate slips through. Cause: ratio or partial_ratio used where token order varies. Fix: use token_set_ratio for names and narratives so word order and extra tokens do not penalize the match.
• Raw VIN comparison misses a formatting difference. Symptom: identical vehicles score below the cutoff. Cause: one VIN carries hyphens or spaces the other does not. Fix: run canon_vin on both before score_vin, and only then apply edit-distance so a real transposition still degrades gracefully.
• Boundary verdict flips between runs. Symptom: a pair alternates DUPLICATE and AMBIGUOUS on identical inputs. Cause: an unrounded floating-point sum straddling the cutoff. Fix: round the weighted score to a fixed precision before comparison and use inclusive >= boundaries consistently.
• Timezone drift zeroes the date sub-score. Symptom: the same loss reported through two channels loses its date match. Cause: one intake stamped local time, the other UTC, pushing the delta past the window. Fix: normalize loss dates to a single timezone at ingestion and keep window_days at 3 so a one-day drift still scores.
• Free-text narrative dominates or vanishes. Symptom: matches hinge entirely on the loss description, or ignore it. Cause: a miscalibrated loss_text weight. Fix: tune the weight against a labelled sample; narrative is corroborating signal, not the primary key — keep it a minority of the total.

• Duplicate Claim Detection at Ingestion Scale — the parent guide whose blocking stage feeds this scorer
• Entity Network Analysis for Organized Claims Fraud — where near-duplicate edges surface as split-claim rings
• Audit Log Schema Design — the append-only trail every merge verdict is written into
• SIU Referral Orchestration — where a confirmed duplicate pattern is escalated for investigation
• Fraud Pattern Detection — the domain this scoring step sits within