Backtesting Threshold Changes Against Claim History

This page is a focused implementation guide under Dynamic Threshold Tuning: it shows how to replay a proposed routing or severity threshold over an immutable log of historical claims before it ships, so you can measure the routing shift, SLA impact, and misroute rate the change will cause instead of discovering them in production.

Routing thresholds are the tuning knobs of a triage engine: the severity score above which a claim goes to a senior adjuster, the dollar figure that trips catastrophe handling, the confidence floor below which a claim is diverted to manual review. Every one of these is a number someone will eventually want to move — because the queue for senior adjusters is too long, or a fraud pattern slipped through, or a state DOI changed a timeliness rule. The temptation is to nudge the number in a config file and watch production. That is how a “small” change from severity 0.70 to 0.68 quietly reassigns eleven percent of the daily volume to a team that is already at capacity, blows the acknowledgement SLA on a whole segment, and is only caught a week later in a metrics review.

The defect is that the change was never measured against the population it would actually affect. A threshold does not act on the average claim; it acts on the distribution of claims sitting near the boundary, and that distribution is knowable — it is your own claim history. This guide builds a deterministic backtest harness that replays a candidate threshold over an immutable event log of historical claims and reports three concrete metrics: how many claims change route (the routing shift), how the projected acknowledgement time moves under the new routing (the SLA impact), and the rate at which the new threshold would have misrouted claims whose correct disposition is known from the historical outcome (the misroute rate). The harness is deterministic and pure over the log, so the same candidate always yields the same verdict, and a guardrail gate blocks any change whose measured impact exceeds an approved envelope. It sits alongside implementing priority queues for catastrophic claims, which consumes the thresholds this harness validates, and feeds the approved change into the regulatory sync pipelines that version rule updates.

This harness targets Python 3.10+ with pandas for the replay and vectorized metrics. Pin the versions so grouping and rounding semantics stay reproducible across the team:

Copy
python -m venv .venv && source .venv/bin/activate
pip install "pandas==2.2.*" "structlog==24.*"

The one hard input requirement is an immutable claim-history log: an append-only table where each row is a historical claim with the severity score it received, the route it was actually assigned, the acknowledgement time it achieved, and — critically — the known-correct disposition derived from its final outcome (was it truly a senior-adjuster claim, was it truly catastrophic). If your history is mutable or the outcome labels are backfilled inconsistently, the misroute rate is meaningless. Freeze a snapshot of the log by content hash before the backtest so the same run is reproducible for audit.

Separate the policy (how a threshold maps a claim to a route) from the replay. The routing function is a pure map from a claim’s features and a threshold to a route label, so the same function serves both production and the backtest — you are never testing a re-implementation.

Copy
from __future__ import annotations

from dataclasses import dataclass
from decimal import Decimal
from enum import Enum

import structlog

log = structlog.get_logger("threshold.backtest")


class Route(str, Enum):
    SENIOR = "senior_adjuster"
    STANDARD = "standard_queue"
    MANUAL = "manual_review"


@dataclass(frozen=True, slots=True)
class ClaimRecord:
    claim_id: str
    severity: Decimal            # 0..1 score the claim received historically
    loss_amount: Decimal
    ack_hours: Decimal           # acknowledgement time actually achieved
    true_route: Route            # correct disposition from final outcome


@dataclass(frozen=True, slots=True)
class ThresholdConfig:
    senior_severity: Decimal     # >= this routes to a senior adjuster
    manual_floor: Decimal        # < this diverts to manual review


def route_claim(claim: ClaimRecord, cfg: ThresholdConfig) -> Route:
    if claim.severity < cfg.manual_floor:
        return Route.MANUAL
    if claim.severity >= cfg.senior_severity:
        return Route.SENIOR
    return Route.STANDARD

Load the frozen log into a DataFrame and apply the routing function twice — once under the live thresholds, once under the candidate. Because route_claim is pure and the log is immutable, the replay is deterministic: identical input yields an identical frame every time.

Copy
import pandas as pd


def replay(
    claims: list[ClaimRecord],
    baseline: ThresholdConfig,
    candidate: ThresholdConfig,
) -> pd.DataFrame:
    rows = [
        {
            "claim_id": c.claim_id,
            "severity": c.severity,
            "ack_hours": c.ack_hours,
            "true_route": c.true_route.value,
            "baseline_route": route_claim(c, baseline).value,
            "candidate_route": route_claim(c, candidate).value,
        }
        for c in claims
    ]
    df = pd.DataFrame(rows)
    if df.empty:
        raise EmptyBacktestWindow("no claims in the frozen history snapshot")
    log.info("backtest.replayed", claims=len(df))
    return df


class EmptyBacktestWindow(Exception):
    """The frozen history snapshot contained no claims to replay."""

Reduce the replay frame to the three decision metrics. Routing shift is the fraction of claims whose route changed. SLA impact is the change in mean acknowledgement time for the segment newly routed to the senior queue, using the historical ack_hours of the claims now landing there. Misroute rate is the fraction of claims whose candidate route disagrees with the known-correct disposition.

Copy
@dataclass(frozen=True, slots=True)
class BacktestReport:
    n_claims: int
    routing_shift: Decimal        # fraction of claims that changed route
    misroute_rate: Decimal        # fraction disagreeing with true_route
    senior_ack_delta_hours: Decimal   # mean ack-time change for senior queue


def _fraction(numerator: int, denominator: int) -> Decimal:
    if denominator == 0:
        return Decimal("0")
    return (Decimal(numerator) / Decimal(denominator)).quantize(Decimal("0.0001"))


def compute_metrics(df: pd.DataFrame) -> BacktestReport:
    n = len(df)
    changed = int((df["baseline_route"] != df["candidate_route"]).sum())
    misrouted = int((df["candidate_route"] != df["true_route"]).sum())

    base_senior = df.loc[df["baseline_route"] == Route.SENIOR.value, "ack_hours"]
    cand_senior = df.loc[df["candidate_route"] == Route.SENIOR.value, "ack_hours"]
    base_mean = Decimal(str(base_senior.mean())) if len(base_senior) else Decimal("0")
    cand_mean = Decimal(str(cand_senior.mean())) if len(cand_senior) else Decimal("0")
    ack_delta = (cand_mean - base_mean).quantize(Decimal("0.01"))

    report = BacktestReport(
        n_claims=n,
        routing_shift=_fraction(changed, n),
        misroute_rate=_fraction(misrouted, n),
        senior_ack_delta_hours=ack_delta,
    )
    log.info("backtest.metrics", routing_shift=str(report.routing_shift),
             misroute_rate=str(report.misroute_rate),
             senior_ack_delta=str(report.senior_ack_delta_hours))
    return report

A backtest is only a guardrail if it can block. Compare the report against an explicit envelope and raise on any breach, so an out-of-bounds change cannot reach the rule store without a human overriding the envelope on the record.

Copy
@dataclass(frozen=True, slots=True)
class ImpactEnvelope:
    max_routing_shift: Decimal
    max_misroute_rate: Decimal
    max_senior_ack_delta_hours: Decimal


def enforce_envelope(report: BacktestReport, envelope: ImpactEnvelope) -> None:
    breaches: list[str] = []
    if report.routing_shift > envelope.max_routing_shift:
        breaches.append(f"routing_shift {report.routing_shift} > {envelope.max_routing_shift}")
    if report.misroute_rate > envelope.max_misroute_rate:
        breaches.append(f"misroute_rate {report.misroute_rate} > {envelope.max_misroute_rate}")
    if report.senior_ack_delta_hours > envelope.max_senior_ack_delta_hours:
        breaches.append(
            f"senior_ack_delta {report.senior_ack_delta_hours} > "
            f"{envelope.max_senior_ack_delta_hours}"
        )
    if breaches:
        raise EnvelopeBreached("; ".join(breaches))
    log.info("backtest.envelope_ok", n_claims=report.n_claims)


class EnvelopeBreached(Exception):
    """The candidate threshold's measured impact exceeded the approved envelope."""

The harness itself must be tested so its verdicts are trustworthy. Prove that an identical candidate produces zero shift, that a lowered senior threshold shifts the expected claims, and that the envelope actually blocks.

Copy
from decimal import Decimal


def _claim(cid: str, sev: str, ack: str, true_route: Route) -> ClaimRecord:
    return ClaimRecord(cid, Decimal(sev), Decimal("10000"), Decimal(ack), true_route)


BASE = ThresholdConfig(senior_severity=Decimal("0.70"), manual_floor=Decimal("0.10"))
LOWER = ThresholdConfig(senior_severity=Decimal("0.68"), manual_floor=Decimal("0.10"))
CLAIMS = [
    _claim("c1", "0.69", "4", Route.SENIOR),    # standard under BASE, senior under LOWER
    _claim("c2", "0.95", "3", Route.SENIOR),    # senior under both
    _claim("c3", "0.40", "8", Route.STANDARD),  # standard under both
]


def test_identical_candidate_has_zero_shift() -> None:
    df = replay(CLAIMS, BASE, BASE)
    report = compute_metrics(df)
    assert report.routing_shift == Decimal("0.0000")


def test_lowered_threshold_shifts_boundary_claim() -> None:
    df = replay(CLAIMS, BASE, LOWER)
    report = compute_metrics(df)
    assert report.routing_shift == Decimal("0.3333")   # 1 of 3 changed
    assert report.misroute_rate == Decimal("0.0000")   # LOWER matches true_route


def test_envelope_blocks_excessive_shift() -> None:
    df = replay(CLAIMS, BASE, LOWER)
    report = compute_metrics(df)
    tight = ImpactEnvelope(Decimal("0.10"), Decimal("0.05"), Decimal("2.00"))
    import pytest
    with pytest.raises(EnvelopeBreached):
        enforce_envelope(report, tight)

Run under python -m pytest -q. The determinism is the point: the same frozen log and the same candidate always produce the same BacktestReport, which is what lets the number in the report stand as evidence in a change-approval record.

Threshold changes to a claims-routing engine are rule changes, and regulators treat rule changes as governed events. Because the backtest is deterministic over a content-hashed snapshot of the claim history, the BacktestReport is a reproducible artifact: attach it, the frozen-log hash, the baseline and candidate ThresholdConfig, and the ImpactEnvelope to the change record before the new threshold is written. When a state DOI or an internal auditor asks why a routing rule changed and what its measured effect was, the report reconstructs the projected routing shift and misroute rate at the moment the decision was made, and the versioned rule flows through the regulatory sync pipelines with that evidence attached. A change that skipped the backtest, or one whose envelope was overridden, is visible as such in the record — which is exactly the accountability an unfair-claims-settlement review looks for.

• Backtest looks clean but production regresses. Symptom: the report showed a small shift, yet queues blew out after ship. Cause: the frozen history was not representative — it predated a seasonal catastrophe surge. Fix: run the backtest over multiple windows including your worst historical month, and gate on the worst-case report, not the average.
• Misroute rate is suspiciously zero. Symptom: every candidate scores a perfect misroute rate. Cause: true_route was derived from the same threshold you are testing, so the label is circular. Fix: derive the known-correct disposition from the final claim outcome, never from a routing rule.
• Non-deterministic reports. Symptom: two runs of the same candidate disagree. Cause: the replay read a live table that changed between runs, or a float mean introduced platform rounding. Fix: replay only the content-hashed snapshot and quantize every metric with an explicit Decimal scale as shown.
• SLA delta hides a queue-capacity breach. Symptom: mean acknowledgement time barely moves but the senior team is overwhelmed. Cause: the mean masks a volume spike. Fix: report the count of claims newly routed to each queue alongside the ack-time delta, and add a max-volume term to the envelope.
• EmptyBacktestWindow on a valid change. Symptom: the harness raises with a real candidate. Cause: the snapshot filter excluded every claim (wrong date range). Fix: treat it as a snapshot-selection bug and fail closed — never let a change ship because the backtest found nothing to measure.

• Dynamic Threshold Tuning — the parent guide this backtest harness validates changes for
• Implementing Priority Queues for Catastrophic Claims — the surge-routing consumer of the thresholds this harness gates
• Regulatory Sync Pipelines — where an approved threshold change is versioned and propagated
• Building Append-Only Hash-Chained Audit Logs — the immutable-log discipline the frozen history snapshot depends on
• Claims Triage & Routing Engines — the control plane these thresholds tune