Duplicate Claim Detection at Ingestion Scale

Duplicate claim detection is the deduplication gate that inspects every incoming First Notice of Loss (FNOL) before it is allowed to open a reserve, and decides whether it is a genuinely new loss or a copy of one the book already holds. It is one detector in the broader Fraud Pattern Detection domain, sitting at the ingestion boundary so that a re-filed loss is caught before it ever reaches severity scoring or payment. For InsurTech developers and claims analysts, this is where the same water-damage event submitted through three channels — a mobile app, a broker portal, and a call-center transcript — has to collapse into one claim, or where a single loss deliberately split into two under-threshold claims has to surface as related.

At pilot volume you can eyeball duplicates. At production volume — tens of thousands of daily FNOL events across multiple intake channels, arriving out of order, some replayed by an at-least-once broker — three failure classes appear that a naive claim_number equality check cannot survive.

The first is the exact-key illusion. Insurers assign a fresh claim number to every intake, so two submissions of the same loss carry different identifiers by construction. Deduplication on the surrogate key finds nothing. The real duplicate signal lives in the loss itself — claimant identity, loss location, loss date, vehicle or property identifier, and the free-text loss description — none of which is a clean primary key, and all of which arrive dirty: transposed digits in a VIN, an abbreviated street suffix, a nickname instead of a legal first name, a loss date off by a day because of timezone handling at intake.

The second is the quadratic wall. Comparing every new FNOL against every open claim is O(n²) in the size of the book, and a fuzzy-string comparison across five fields is not cheap. A book of two million open claims cannot afford a full pairwise scan per intake, so a disciplined pipeline must first narrow the field to a small set of plausible candidates — the blocking step — before any expensive similarity scoring runs. Get blocking wrong and you either miss duplicates (blocks too tight) or reintroduce the quadratic cost (blocks too loose).

The third is the non-deterministic verdict. When the same pair of claims scores as a duplicate on one run and unique on the next — because a threshold was read from mutable state, or a floating-point similarity score straddled a hard cutoff — the decision is indefensible the moment it is disputed. A duplicate that was auto-merged, then later contested, must be reconstructable from the audit record alone: which fields matched, what each field’s similarity score was, and which threshold produced the verdict.

A disciplined deduplication layer treats duplicate detection as a two-stage, idempotent, fail-closed function: cheap deterministic blocking to generate candidates, then explainable similarity scoring to render a match, near-match, or unique verdict — every one of them written to an immutable trail.

This component targets Python 3.10+ for modern union syntax and structural pattern matching. The similarity engine is rapidfuzz (a C+±backed Levenshtein/token-ratio library that is an order of magnitude faster than pure-Python fuzzywuzzy), with structlog for the audit lines and pydantic for the intake schema. Pin them so a minor bump cannot silently shift scoring behaviour:

Copy
python -m venv .venv && source .venv/bin/activate
pip install "rapidfuzz==3.*" "pydantic==2.*" "structlog==24.*"

Infrastructure dependencies:

• A message broker delivering canonical FNOL events (Kafka with idempotent producers, or SQS FIFO), so deduplication consumes the same ordered stream the rest of intake uses. Because the broker is at-least-once, the pipeline must be idempotent: re-delivery of an already-processed FNOL must reproduce the identical verdict, not open a second claim.
• A candidate index — an inverted index from blocking key to open-claim ids, typically a Redis set per key or an indexed column in the claims store — so candidate generation is a key lookup, not a table scan.
• An append-only audit store — the same governance backbone described in the Audit Log Schema Design guide — into which every duplicate verdict is written before any merge or reserve action.

Establish the environment-driven thresholds before writing rule logic: the auto-merge cutoff (DUP_HIGH_THRESHOLD, default 92), the review floor (DUP_LOW_THRESHOLD, default 78), and the active rule version (DUP_RULE_VERSION, default v1.4). These are deployed as configuration, never hard-coded, so a threshold change is a config rollout, not a code release.

The pipeline is two ordered stages, and keeping them separate is what makes it both fast and explainable. Blocking is a cheap, deterministic function that maps a normalized FNOL to one or more string keys; two records can only be duplicates if they share a key. Scoring is the expensive, explainable function that runs only within a block, comparing a new claim against the handful of open claims that share its key.

The art is in the blocking key. Too coarse — say, just the loss month — and every block is enormous, so scoring degrades back toward quadratic. Too fine — say, the full concatenation of every field verbatim — and a single transposed VIN digit lands two copies of the same loss in different blocks, and the duplicate is never even compared. The production pattern is multiple overlapping blocking keys: a record emits several keys (soundex of surname plus loss month; VIN prefix plus loss month; normalized postcode plus loss date), and a pair is a candidate if it collides on any key. This tolerates a single dirty field without inflating the candidate set, because it is unlikely that a genuine duplicate is corrupted on every blocking field at once.

Scoring itself is deliberately not a single fused string comparison. Each field is compared with the rapidfuzz scorer appropriate to its shape — token_set_ratio for names and free-text loss descriptions where word order and extra tokens vary, token_sort_ratio for addresses, an exact-or-near check for VINs, and a day-delta for loss dates — and the field scores are combined into one weighted total. Per-field scores are retained, not collapsed, because the audit record and the downstream review UI both need to show why a pair matched, not merely that it did. This same explainability discipline is what lets a duplicate signal feed cleanly into the statistical anomaly scoring layer and the organized-fraud entity network analysis graph without re-deriving the comparison.

The implementation below normalizes an FNOL, emits its blocking keys, scores a candidate pair field by field, and renders a three-way verdict. Everything is typed, side-effect-free apart from the audit line, and raises explicit exceptions rather than defaulting on bad input.

Copy
from __future__ import annotations

import re
import unicodedata
from dataclasses import dataclass, field
from datetime import date
from enum import Enum

import structlog
from rapidfuzz import fuzz

log = structlog.get_logger("fraud.dedup")

_WS = re.compile(r"\s+")
_NON_ALNUM = re.compile(r"[^a-z0-9 ]")


class DedupError(Exception):
    """Raised when an FNOL cannot be normalized into a comparable record."""


def _canon(value: str) -> str:
    """Lower-case, strip accents and punctuation, collapse whitespace."""
    folded = unicodedata.normalize("NFKD", value).encode("ascii", "ignore").decode()
    return _WS.sub(" ", _NON_ALNUM.sub(" ", folded.lower())).strip()


@dataclass(frozen=True, slots=True)
class NormalizedFNOL:
    claim_id: str
    surname: str            # canonical
    full_name: str          # canonical
    address: str            # canonical
    vin: str                # upper, no separators; "" if none
    loss_date: date
    loss_text: str          # canonical free-text description


def normalize(raw: dict[str, object]) -> NormalizedFNOL:
    """Coerce a raw FNOL dict into a comparison-ready record.

    Raises DedupError rather than defaulting, so a malformed intake never
    silently produces a record that matches nothing.
    """
    try:
        surname = _canon(str(raw["claimant_surname"]))
        loss_date = date.fromisoformat(str(raw["loss_date"]))
    except (KeyError, ValueError) as exc:
        raise DedupError(f"unnormalizable FNOL: {exc}") from exc
    if not surname:
        raise DedupError("empty claimant surname after normalization")
    vin = re.sub(r"[^A-Z0-9]", "", str(raw.get("vin", "")).upper())
    return NormalizedFNOL(
        claim_id=str(raw["claim_id"]),
        surname=surname,
        full_name=_canon(str(raw.get("claimant_name", surname))),
        address=_canon(str(raw.get("loss_address", ""))),
        vin=vin,
        loss_date=loss_date,
        loss_text=_canon(str(raw.get("loss_description", ""))),
    )

The blocking function emits the overlapping key set. A soundex-style phonetic reduction of the surname collapses spelling variants (Smith/Smyth), while the loss month and VIN prefix add discriminating power without demanding an exact match on any single dirty field:

Copy
def _soundex(name: str) -> str:
    """Compact phonetic code so spelling variants share a blocking key."""
    if not name:
        return "0000"
    codes = {**dict.fromkeys("bfpv", "1"), **dict.fromkeys("cgjkqsxz", "2"),
             **dict.fromkeys("dt", "3"), "l": "4",
             **dict.fromkeys("mn", "5"), "r": "6"}
    head, tail = name[0].upper(), ""
    prev = codes.get(name[0], "")
    for ch in name[1:]:
        code = codes.get(ch, "")
        if code and code != prev:
            tail += code
        if ch not in "hw":
            prev = code
    return (head + tail + "000")[:4]


def blocking_keys(rec: NormalizedFNOL) -> frozenset[str]:
    """Overlapping keys: a pair is a candidate if it collides on ANY key."""
    ym = rec.loss_date.strftime("%Y%m")
    keys = {f"snd:{_soundex(rec.surname)}:{ym}"}
    if rec.vin:
        keys.add(f"vin:{rec.vin[:4]}:{ym}")
    if rec.address:
        keys.add(f"adr:{rec.address[:8]}:{rec.loss_date.isoformat()}")
    return frozenset(keys)

Scoring compares a candidate pair field by field and combines the results with explicit weights. The per-field scores are returned in the result object so the decision is fully explainable:

Copy
class Verdict(str, Enum):
    DUPLICATE = "DUPLICATE"
    AMBIGUOUS = "AMBIGUOUS"
    UNIQUE = "UNIQUE"


@dataclass(frozen=True, slots=True)
class MatchResult:
    new_claim_id: str
    candidate_claim_id: str
    field_scores: dict[str, float]
    weighted_score: float
    verdict: Verdict
    rule_version: str


_WEIGHTS = {"name": 0.30, "address": 0.20, "vin": 0.25, "loss_text": 0.15, "date": 0.10}


def _date_score(a: date, b: date) -> float:
    """100 for same day, decaying to 0 beyond a 3-day window."""
    delta = abs((a - b).days)
    return max(0.0, 100.0 - (delta / 3.0) * 100.0)


def score_pair(
    new: NormalizedFNOL,
    other: NormalizedFNOL,
    *,
    high: float,
    low: float,
    rule_version: str,
) -> MatchResult:
    if high <= low:
        raise DedupError(f"high threshold {high} must exceed low {low}")
    vin_score = 100.0 if new.vin and new.vin == other.vin else (
        float(fuzz.ratio(new.vin, other.vin)) if new.vin and other.vin else 0.0
    )
    scores = {
        "name": float(fuzz.token_set_ratio(new.full_name, other.full_name)),
        "address": float(fuzz.token_sort_ratio(new.address, other.address)),
        "vin": vin_score,
        "loss_text": float(fuzz.token_set_ratio(new.loss_text, other.loss_text)),
        "date": _date_score(new.loss_date, other.loss_date),
    }
    weighted = round(sum(scores[k] * w for k, w in _WEIGHTS.items()), 2)
    if weighted >= high:
        verdict = Verdict.DUPLICATE
    elif weighted >= low:
        verdict = Verdict.AMBIGUOUS
    else:
        verdict = Verdict.UNIQUE
    log.info("dedup.scored", new=new.claim_id, other=other.claim_id,
             weighted=weighted, verdict=verdict.value)
    return MatchResult(new.claim_id, other.claim_id, scores, weighted,
                       verdict, rule_version)

The orchestration that ties these together fetches candidates from the block index, scores each, and reduces to a single decision — the highest-scoring match wins, and a UNIQUE result opens a new claim. Because the whole path is a pure function of the normalized records and the threshold set, re-processing a replayed FNOL yields the same verdict, which is what makes the merge idempotent: a duplicate that was already linked to its canonical claim links to the same canonical claim again rather than spawning a third record.

The thresholds and field weights are configuration, not code. The gap between DUP_HIGH_THRESHOLD and DUP_LOW_THRESHOLD defines the ambiguous band routed to human review; widening it trades automation for recall, and the correct width is a business decision calibrated against a labelled sample, not a constant baked into the source.

Carrier-specific overrides layer on top of the base weight set — a commercial-auto program leans harder on the VIN weight, a homeowners program on address and loss text — resolved at load time into a single immutable configuration object passed into score_pair. Resolving overrides inside the hot path reintroduces the non-determinism the gate exists to eliminate.

Every duplicate verdict is an audit event, and an auto-merge is a consequential one: it can suppress a claim a policyholder legitimately filed. The audit record must let an examiner reconstruct the exact decision — the two claim ids, the per-field similarity scores, the weighted total, the threshold set, and the rule version — written to append-only storage before the merge is published. This is the same discipline the core architecture and compliance mapping domain enforces platform-wide, and it is what keeps an automated merge defensible under a market-conduct exam: a regulator asking why two claims were treated as one gets the field-level rationale, not a bare merged=true.

Fail-closed applies here too. When normalization raises DedupError, or a candidate block exceeds DUP_MAX_CANDIDATES, the safe action is to route the FNOL to human review rather than silently open a new claim (which risks a duplicate reserve) or silently merge (which risks suppressing a valid claim). A high-scoring auto-merge that a claimant later disputes is unwound by reading the immutable record and re-splitting, never by editing history in place.

The mechanics of the scoring step reward a dedicated treatment. The guide on fuzzy matching duplicate FNOL submissions works through the concrete rapidfuzz scorer choices field by field — why token_set_ratio suits claimant names and loss narratives while token_sort_ratio suits addresses, how to normalize a VIN and a loss date before comparison, and the exact assertion patterns that prove the threshold logic behaves at the boundary. Read it when you are implementing the score_pair step against real, dirty intake data and need the pytest suite that pins its behaviour.

• Fraud Pattern Detection — the parent domain this deduplication gate belongs to
• Fuzzy Matching Duplicate FNOL Submissions — the concrete rapidfuzz scoring recipe for the pair step
• Entity Network Analysis for Organized Claims Fraud — where related-claim edges surface as collusion rings
• Statistical Anomaly Scoring — consumes duplicate signals as a fraud-score feature
• SIU Referral Orchestration — where a confirmed duplicate pattern is escalated to investigators
• Claims Triage & Routing Engines — the control plane a de-duplicated claim flows into