Calibrating Severity Thresholds with Historical Loss Data

This guide sits under Automated Severity Scoring Models: it shows how to derive severity band cutoffs from the loss distribution of historical closed claims — using quantile and cost-based methods — then validate them on a holdout, version them, and avoid the leakage that quietly inflates every backtest.

The tier boundaries that split a claim into low, medium, high, or critical severity are the single most consequential numbers in the routing engine, yet they are routinely set by intuition — “over fifty thousand is high” — and never revisited. Hand-picked cutoffs are wrong in two directions at once. Set them too low and the senior-adjuster queue floods with routine claims, inflating loss adjustment expense and burying the genuinely large losses. Set them too high and a claim that should have had immediate senior attention sits in the straight-through path until it develops, and the reserve is set late.

Worse, the few teams that do derive cutoffs from data usually derive them wrong. They compute quantiles over the entire history including still-open claims, whose incurred amounts have not developed to ultimate — so the distribution is understated and the bands drift low. They fit the cutoffs and evaluate them on the same rows, so the reported tier balance is optimistic and collapses in production. And they treat the resulting numbers as ephemeral constants in a config file, with no version, no provenance, and no way to answer “which cutoffs scored this claim last March?”. This page derives band cutoffs from closed-claim ultimate losses using two defensible methods, validates them against a temporally-separated holdout, and packages them in a versioned, auditable dataclass. The output is exactly the versioned threshold artifact consumed by the batch runs in Orchestrating Batch Severity Scoring with Celery.

This targets Python 3.10+ with pandas and NumPy for the distribution work, plus structlog for audit lines. Pin the majors so quantile interpolation semantics stay fixed:

Copy
python -m venv .venv && source .venv/bin/activate
pip install "pandas==2.2.*" "numpy==1.26.*" "structlog==24.*"

The input is a table of closed claims with a developed ultimate loss and a loss date. Two data conditions must hold before calibration: every row is genuinely closed (open claims carry undeveloped incurred amounts that bias the distribution low), and monetary amounts are exact — carried as Decimal at the boundary and only converted to float inside NumPy for the quantile computation, never stored back as float. The resulting cutoffs feed the same scoring node described across the parent Automated Severity Scoring Models guide and are the artifact re-scored against in Dynamic Threshold Tuning.

Randomly shuffling rows into train and test leaks the future into the past: a catastrophe that inflated late-period losses would appear in both sets, and the holdout would flatter cutoffs that will not generalize. Split on a cutoff date instead, so the holdout is strictly later than the fit window — the same temporal boundary the model will face in production.

Copy
from __future__ import annotations

from dataclasses import dataclass
from datetime import date

import pandas as pd


@dataclass(frozen=True, slots=True)
class CalibrationSplit:
    fit: pd.DataFrame
    holdout: pd.DataFrame
    cutoff_date: date


def temporal_split(claims: pd.DataFrame, cutoff_date: date) -> CalibrationSplit:
    if not claims["is_closed"].all():
        raise ValueError("calibrate on closed claims only; open incurred is undeveloped")
    loss_dates = pd.to_datetime(claims["loss_date"]).dt.date
    fit = claims[loss_dates < cutoff_date].copy()
    holdout = claims[loss_dates >= cutoff_date].copy()
    if len(fit) < 1000 or len(holdout) < 250:
        raise ValueError(f"insufficient sample: fit={len(fit)} holdout={len(holdout)}")
    return CalibrationSplit(fit, holdout, cutoff_date)

Two defensible methods. The quantile method places cutoffs at fixed percentiles of the fit-set loss distribution, so each band holds a controlled share of claims — simple, stable, and the right default when your objective is queue balance. The cost method searches for the cutoff that minimizes total expected cost: the adjustment expense of over-triaging small claims to senior review, plus the leakage cost of under-triaging large claims that develop adversely. Use cost when the two error directions have very different dollar consequences.

Copy
from decimal import Decimal
import numpy as np


def quantile_cutoffs(fit: pd.DataFrame, quantiles: tuple[float, ...]) -> list[Decimal]:
    losses = fit["ultimate_loss"].astype(float).to_numpy()
    raw = np.quantile(losses, quantiles, method="linear")
    # round each boundary up to whole dollars, keep exact as Decimal
    return [Decimal(int(np.ceil(x))) for x in raw]


def cost_based_high_cutoff(
    fit: pd.DataFrame,
    *,
    senior_review_cost: Decimal,       # LAE added by routing to senior
    leakage_rate: Decimal,             # fraction of a missed large loss lost
    grid: tuple[Decimal, ...],
) -> Decimal:
    losses = fit["ultimate_loss"]
    best_cutoff, best_cost = grid[0], None
    n = Decimal(len(fit))
    for cutoff in grid:
        over_triaged = Decimal(int((losses < cutoff).sum()))   # small claims sent up
        missed = losses[losses >= cutoff]
        under_cost = (missed.map(Decimal).sum() * leakage_rate)
        total = over_triaged * senior_review_cost + under_cost
        if best_cost is None or total < best_cost:
            best_cutoff, best_cost = cutoff, total
    return best_cutoff

Apply the candidate boundaries to the untouched holdout and check two things: the realized share of claims per band is close to the target, and the rate of large losses that fell into a band below critical is acceptable. Only boundaries that pass are frozen into an immutable, content-hashed record so the exact cutoffs that scored any claim are always reconstructable.

Copy
import hashlib
import json
import structlog

log = structlog.get_logger("severity.calibration")


@dataclass(frozen=True, slots=True)
class CalibratedThresholds:
    boundaries: tuple[Decimal, ...]     # ascending band edges, e.g. (low|med, med|high, high|crit)
    method: str                         # "quantile" | "cost"
    fit_window_end: date
    sample_size: int
    version: str

    @property
    def content_hash(self) -> str:
        payload = json.dumps({
            "boundaries": [str(b) for b in self.boundaries],
            "method": self.method,
            "fit_window_end": self.fit_window_end.isoformat(),
            "sample_size": self.sample_size,
        }, sort_keys=True)
        return hashlib.sha256(payload.encode()).hexdigest()[:16]


def band_of(loss: Decimal, boundaries: tuple[Decimal, ...]) -> int:
    band = 0
    for edge in boundaries:
        if loss >= edge:
            band += 1
    return band                          # 0=low ... len=critical


def validate(holdout: pd.DataFrame, boundaries: tuple[Decimal, ...],
             *, target_shares: tuple[float, ...], tolerance: float = 0.03) -> bool:
    bands = holdout["ultimate_loss"].map(lambda x: band_of(Decimal(str(x)), boundaries))
    realized = bands.value_counts(normalize=True).sort_index()
    for i, target in enumerate(target_shares):
        got = float(realized.get(i, 0.0))
        if abs(got - target) > tolerance:
            log.warning("calibration.share_off", band=i, target=target, got=got)
            return False
    return True

The tests pin the two properties that leakage and float error most often break: the fit/holdout split is strictly temporal, and quantile cutoffs are monotonically increasing and exact. A synthetic loss distribution with a known shape lets you assert the percentile placement directly.

Copy
import numpy as np
import pandas as pd
import pytest
from datetime import date
from decimal import Decimal


def _synthetic(n: int, seed: int = 7) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    losses = rng.lognormal(mean=8.5, sigma=1.1, size=n)   # right-skewed like real losses
    days = rng.integers(0, 720, size=n)
    return pd.DataFrame({
        "ultimate_loss": np.round(losses, 2),
        "loss_date": [date(2024, 1, 1) + pd.Timedelta(days=int(d)) for d in days],
        "is_closed": True,
    })


def test_temporal_split_is_disjoint_and_ordered():
    df = _synthetic(5000)
    split = temporal_split(df, cutoff_date=date(2025, 1, 1))
    assert pd.to_datetime(split.fit["loss_date"]).dt.date.max() < date(2025, 1, 1)
    assert pd.to_datetime(split.holdout["loss_date"]).dt.date.min() >= date(2025, 1, 1)


def test_quantile_cutoffs_monotonic_and_exact():
    df = _synthetic(20000)
    cuts = quantile_cutoffs(df, (0.5, 0.85, 0.97))
    assert cuts == sorted(cuts)                       # ascending band edges
    assert all(isinstance(c, Decimal) for c in cuts)  # no float leaked into storage


def test_open_claims_rejected():
    df = _synthetic(2000)
    df.loc[0, "is_closed"] = False
    with pytest.raises(ValueError):
        temporal_split(df, cutoff_date=date(2025, 1, 1))

Run with python -m pytest -q. The decisive checks are that the holdout minimum date is not less than the fit maximum date — proving no temporal overlap — and that every stored boundary is a Decimal, so no float ever becomes the number that decides a claim’s tier.

Severity band boundaries are a rating-adjacent decision: they determine which claims receive senior scrutiny and how reserves are initially set, so a regulator can ask you to justify them. Because CalibratedThresholds is immutable and content-hashed, the exact cutoffs, the method, the fit window, and the sample size that produced any claim’s tier are reconstructable from the version alone. Persist each calibrated artifact to the same append-only ledger the scoring engine reads, and stamp its version onto every scoring record. That closes the loop from “why was this claim tiered high?” back to a specific, dated, reproducible set of boundaries derived from disclosed historical losses — the transparency a market-conduct examiner expects and the provenance that lets you defend the cutoffs as data-driven rather than arbitrary.

• Cutoffs drift lower every recalibration. Symptom: bands creep down and senior queues grow. Cause: open or partially-developed claims leaked into the fit set, understating the tail. Fix: enforce is_closed and, if development is slow, restrict to claims closed long enough to be at ultimate.
• Holdout share matches perfectly but production does not. Symptom: validation passes yet live band balance is wrong. Cause: a random shuffle split leaked future rows into the fit set. Fix: always split on loss date so the holdout is strictly later, mirroring production time order.
• Off-by-a-penny tier flips near a boundary. Symptom: two identical claims land in different bands across runs. Cause: a float loss amount compared against the cutoff. Fix: carry losses as Decimal end-to-end and compare with Decimal in band_of; only convert to float inside NumPy for the quantile call.
• Cost-based cutoff swings wildly between runs. Symptom: the high boundary jumps thousands of dollars run to run. Cause: the leakage rate or senior-review cost is guessed and the grid is coarse. Fix: source both costs from actuals, widen the sample, and refine the grid near the current optimum rather than searching the whole range each time.
• Nobody can say which cutoffs scored a claim. Symptom: a disputed tier cannot be reproduced. Cause: cutoffs live as mutable config constants with no version. Fix: freeze every calibration into a content-hashed CalibratedThresholds, persist it, and stamp its version on each scoring record.

• Automated Severity Scoring Models — the parent guide whose tier boundaries this calibration produces
• Orchestrating Batch Severity Scoring with Celery — the run that applies these versioned thresholds across the whole book
• Dynamic Threshold Tuning — how calibrated cutoffs are adjusted and re-evaluated over time
• Implementing Priority Queues for Catastrophic Claims — where the critical-band cutoff drives surge routing
• Claims Triage & Routing Engines — the control plane these thresholds govern