ACORD and NAIC Field Validation Pipelines

ACORD and NAIC field validation is the gate that decides whether an extracted policy or claim payload is well-formed enough to enter the canonical store, and it is one component of the broader Core Architecture & Compliance Mapping domain. This page shows how to build a layered validator that checks ingested payloads against ACORD data standards and NAIC required data elements before anything downstream is allowed to trust them.

The payloads reaching this stage are never clean. They arrive from carrier PDF extraction, third-party administrator feeds, and partner APIs, each carrying a slightly different interpretation of the same ACORD form. One source sends a commercial application as a nested ACORD 125 object with a two-letter state code; another flattens the same data and encodes the state as a full name; a third omits the NAIC-mandated producer identifiers entirely because its legacy core never captured them. If any of these reach the canonical policy store unchecked, the defect surfaces days later as a mis-routed claim, a reserve calculated against a phantom coverage line, or a statutory data-quality finding during a market-conduct exam. Validation is the single place where all of that variability is forced to resolve, deterministically, into either an accepted record or a structured rejection routed to manual review.

At pilot volume a single Pydantic model with a handful of validators feels sufficient. The failure classes only appear once the pipeline ingests hundreds of thousands of heterogeneous ACORD payloads a day and the compliance team starts asking why specific records were accepted or rejected.

The first failure is conflated error semantics. When structural malformation (a missing required object), a type error (a premium sent as a string), a code-list violation (a line-of-business code that is not in the ACORD enumeration), and a cross-field business-rule breach (an expiration date before the effective date) all raise the same generic ValidationError, the pipeline cannot decide what to do with each. A structural error means the extraction upstream is broken and the payload should be re-queued; a code-list violation usually means a mapping gap that a human can resolve. Collapsing them into one exception type destroys that routing signal.

The second is non-deterministic rejection reasons. If two structurally identical payloads with the same defect can produce different error messages depending on validator evaluation order, the audit trail becomes indefensible. A regulator asking why one commercial application was rejected and a near-identical one accepted must receive a reproducible, ordered list of reason codes — not a stack trace whose contents depend on dictionary iteration.

The third is silent standard drift. ACORD publishes revised forms and NAIC updates its required data elements on their own cadences. When the validation rules live as scattered literals inside the model, an update to the NAIC required-element list becomes a risky code edit with no record of what changed or when it took effect. The rules must live in a versioned manifest so that a payload validated last quarter can be re-checked against the exact standard that was in force at the time.

A layered validator collapses all three risks. It evaluates payloads through four ordered layers — structural, type, code-list/enumeration, then cross-field business rules — short-circuits at the first layer that fails hard, accumulates every recoverable violation into a single ValidationResult, and stamps each outcome with the manifest version that produced it.

This component targets Python 3.10+ for modern type-hint syntax and structural pattern matching, and uses Pydantic v2 for the type layer and structlog for structured audit lines. Pin the libraries so a minor version bump cannot silently change coercion or serialization semantics:

Copy
python -m venv .venv && source .venv/bin/activate
pip install "pydantic==2.*" "structlog==24.*" "orjson==3.*"

Two infrastructure dependencies underpin the validator in production. The first is a versioned validation manifest store — a registry that holds each revision of the ACORD code lists and NAIC required-element sets keyed by a manifest version string, so a payload validated last quarter can be re-checked against the exact standard in force at the time. The second is the shared append-only audit store, the governance backbone common to the whole domain and modeled in Audit Log Schema Design, into which every validation outcome is written with its manifest version before the record proceeds or is diverted.

The validator sits immediately downstream of extraction. Raw carrier documents are turned into structured payloads by Policy PDF Parsing & Extraction Workflows, and carrier-specific identifiers are normalized onto canonical ACORD field names by Field Mapping Strategies — that mapping stage produces exactly the payloads this validator consumes. Only after validation passes does the record become the typed contract defined in Policy Schema Design.

Every payload passes through four layers in a fixed order, and the order is deliberate because each layer presupposes the one before it.

Layer 1 — Structural. Confirm the required ACORD objects and NAIC-mandated elements are present at all. For an ACORD 125 commercial application that means the applicant, producer, and at least one policy line object exist; for ACORD 140 property it means the premises and building schedule are present. A missing required object is a hard failure that short-circuits the remaining layers, because there is nothing coherent left to type-check. This failure class signals that extraction or mapping upstream is broken and the payload should be re-queued rather than sent to a human.

Layer 2 — Type and format. Coerce each present field to its declared type: premiums and limits to Decimal, dates to ISO 8601 date, identifiers to their pattern (a two-letter state code, a NAIC company number). A field that cannot coerce records a violation but does not short-circuit — later layers still run so the reviewer sees the full defect set at once.

Layer 3 — Code list and enumeration. Check that coded values are members of the reference set the standard defines: the line-of-business code against the ACORD enumeration, the state against the jurisdiction list, the producer identifiers against the NAIC required-element expectations. This is the layer most exposed to standard drift, which is why the reference sets come from the versioned manifest rather than inline literals.

Layer 4 — Cross-field business rules. Apply invariants that span multiple fields: the expiration date must strictly follow the effective date, a stated coverage line must have a corresponding limit, a commercial application flagged as a renewal must carry a prior policy number. These are the checks a JSON Schema or a single-field validator structurally cannot express, and they run last because they assume every field is already present, well-typed, and drawn from a valid code list.

Every layer contributes to one accumulating ValidationResult. Recoverable violations are collected rather than thrown, so a single pass yields a complete, ordered list of reason codes. Only a hard structural failure short-circuits. The result is deterministic: the same payload against the same manifest always yields the same ordered codes, which is what makes a rejection defensible in an exam.

The implementation expresses each layer as a composable validator over a Pydantic model, accumulating violations into an explicit ValidationResult. Reason codes are stable strings drawn from an enum so downstream routing and audit reporting never key off free-text messages.

Copy
from __future__ import annotations

from dataclasses import dataclass, field
from decimal import Decimal, InvalidOperation
from datetime import date
from enum import Enum
from typing import Any, Callable, Mapping

import structlog
from pydantic import BaseModel, ConfigDict, ValidationError

log = structlog.get_logger("acord.validator")


class Severity(str, Enum):
    HARD = "hard"      # short-circuits the pipeline; upstream extraction is broken
    SOFT = "soft"      # accumulate and continue; routable to manual review


class ReasonCode(str, Enum):
    MISSING_REQUIRED_OBJECT = "ACORD.STRUCT.MISSING_OBJECT"
    UNCOERCIBLE_TYPE = "ACORD.TYPE.UNCOERCIBLE"
    CODE_NOT_IN_ENUM = "ACORD.CODE.NOT_IN_ENUM"
    NAIC_ELEMENT_MISSING = "NAIC.ELEMENT.REQUIRED_MISSING"
    DATE_WINDOW_INVALID = "ACORD.RULE.DATE_WINDOW"
    COVERAGE_WITHOUT_LIMIT = "ACORD.RULE.COVERAGE_NO_LIMIT"


@dataclass(frozen=True, slots=True)
class Violation:
    code: ReasonCode
    severity: Severity
    field_path: str
    detail: str


@dataclass(slots=True)
class ValidationResult:
    manifest_version: str
    violations: list[Violation] = field(default_factory=list)

    def record(self, v: Violation) -> None:
        self.violations.append(v)

    @property
    def is_valid(self) -> bool:
        return not self.violations

    @property
    def short_circuited(self) -> bool:
        return any(v.severity is Severity.HARD for v in self.violations)

    def reason_codes(self) -> list[str]:
        # Deterministic ordering: layer order is preserved by insertion,
        # ties broken by field path so identical payloads always match.
        return [v.code.value for v in self.violations]

A Manifest carries the versioned reference sets and the required-object list, so the same validator code can validate against any historical standard by loading a different manifest:

Copy
@dataclass(frozen=True, slots=True)
class Manifest:
    version: str
    required_objects: frozenset[str]          # e.g. {"applicant", "producer", "policy_line"}
    naic_required_elements: frozenset[str]     # e.g. {"producer.naic_company_code"}
    line_of_business_codes: frozenset[str]     # ACORD LOB enumeration for this revision
    state_codes: frozenset[str]


Layer = Callable[[Mapping[str, Any], ValidationResult, Manifest], bool]
# A layer returns True to continue to the next layer, False to short-circuit.


def structural_layer(payload: Mapping[str, Any], result: ValidationResult, m: Manifest) -> bool:
    ok = True
    for obj in m.required_objects:
        if obj not in payload or payload[obj] in (None, {}, []):
            result.record(Violation(
                ReasonCode.MISSING_REQUIRED_OBJECT, Severity.HARD,
                obj, f"required ACORD object '{obj}' absent from payload",
            ))
            ok = False
    for element in m.naic_required_elements:
        if _dig(payload, element) is None:
            result.record(Violation(
                ReasonCode.NAIC_ELEMENT_MISSING, Severity.HARD,
                element, f"NAIC required data element '{element}' absent",
            ))
            ok = False
    return ok  # short-circuit if any required object/element is missing


def code_list_layer(payload: Mapping[str, Any], result: ValidationResult, m: Manifest) -> bool:
    lob = _dig(payload, "policy_line.lob_code")
    if lob is not None and lob not in m.line_of_business_codes:
        result.record(Violation(
            ReasonCode.CODE_NOT_IN_ENUM, Severity.SOFT,
            "policy_line.lob_code", f"LOB '{lob}' not in manifest {m.version}",
        ))
    state = _dig(payload, "applicant.state")
    if state is not None and state not in m.state_codes:
        result.record(Violation(
            ReasonCode.CODE_NOT_IN_ENUM, Severity.SOFT,
            "applicant.state", f"state '{state}' not a valid jurisdiction",
        ))
    return True  # code-list violations are recoverable; keep accumulating


def cross_field_layer(payload: Mapping[str, Any], result: ValidationResult, m: Manifest) -> bool:
    eff, exp = _dig(payload, "effective_date"), _dig(payload, "expiration_date")
    if isinstance(eff, date) and isinstance(exp, date) and eff >= exp:
        result.record(Violation(
            ReasonCode.DATE_WINDOW_INVALID, Severity.SOFT,
            "expiration_date", "expiration must strictly follow effective date",
        ))
    line = payload.get("policy_line") or {}
    if line.get("coverage") and _to_decimal(line.get("limit")) is None:
        result.record(Violation(
            ReasonCode.COVERAGE_WITHOUT_LIMIT, Severity.SOFT,
            "policy_line.limit", "coverage declared without a parseable limit",
        ))
    return True

The type layer is delegated to a Pydantic model so coercion and format rules are declarative, and its failures are translated into the same Violation vocabulary rather than a raw ValidationError:

Copy
def type_layer_factory(model: type[BaseModel]) -> Layer:
    def type_layer(payload: Mapping[str, Any], result: ValidationResult, m: Manifest) -> bool:
        try:
            model.model_validate(payload)
        except ValidationError as exc:
            for err in exc.errors():
                result.record(Violation(
                    ReasonCode.UNCOERCIBLE_TYPE, Severity.SOFT,
                    ".".join(str(p) for p in err["loc"]), err["msg"],
                ))
        return True
    return type_layer


def run_pipeline(
    payload: Mapping[str, Any],
    layers: tuple[Layer, ...],
    manifest: Manifest,
) -> ValidationResult:
    result = ValidationResult(manifest_version=manifest.version)
    for layer in layers:
        if not layer(payload, result, manifest):
            log.warning("acord.validation.short_circuit",
                        manifest=manifest.version, codes=result.reason_codes())
            break
    log.info("acord.validation.complete", manifest=manifest.version,
             valid=result.is_valid, codes=result.reason_codes())
    return result


def _dig(payload: Mapping[str, Any], dotted: str) -> Any:
    cur: Any = payload
    for part in dotted.split("."):
        if not isinstance(cur, Mapping):
            return None
        cur = cur.get(part)
    return cur


def _to_decimal(value: Any) -> Decimal | None:
    if value is None:
        return None
    try:
        return Decimal(str(value))
    except (InvalidOperation, ValueError):
        return None

Two design points make the gate auditable. First, run_pipeline never raises out of the function for a data defect — every recoverable violation is degraded into the ValidationResult with an explicit reason code and field path, and only a genuine programming error would propagate. Second, every result carries the manifest_version that produced it, so a rejection recorded today can be replayed against the exact code lists and required-element sets in force when it happened.

The validator holds no standard-specific literals of its own; every code list, required-element set, and severity assignment is loaded from the manifest at construction time. This is what turns an ACORD revision or a NAIC element change into a configuration deploy rather than a code edit.

Copy
import os

ACTIVE_MANIFEST_VERSION = os.environ.get("ACORD_MANIFEST_VERSION", "2026.06.0")

COMMERCIAL_125_MANIFEST = Manifest(
    version=ACTIVE_MANIFEST_VERSION,
    required_objects=frozenset({"applicant", "producer", "policy_line"}),
    naic_required_elements=frozenset({
        "producer.naic_company_code",
        "producer.producer_id",
    }),
    line_of_business_codes=frozenset({"CGL", "PROP", "AUTOB", "UMBR"}),
    state_codes=frozenset({"AL", "AK", "AZ", "CA", "FL", "NY", "TX"}),  # abridged
)

PIPELINE: tuple[Layer, ...] = (
    structural_layer,
    type_layer_factory(_Acord125TypeModel),  # Pydantic model, defined per form
    code_list_layer,
    cross_field_layer,
)

Carrier- and jurisdiction-specific overrides are resolved when the manifest is assembled, never at validation time. The state code list and any per-jurisdiction required elements derive from State Regulation Mapping, keeping statutory values out of application code. Because the manifest version is stamped onto every outcome, tuning a code list means publishing a new manifest with a new version string — never editing an existing one in place — so the relationship between a validated record and the standard that produced it is always reconstructable.

Validation is where two regulatory standards become executable. ACORD defines the structure and code lists that make a payload interoperable across carriers; NAIC defines the data elements a company must capture and the data-quality expectations examined during market-conduct reviews. The layered validator maps each standard onto a distinct layer, so a violation carries an unambiguous provenance: a NAIC.ELEMENT.REQUIRED_MISSING code points at a statutory data-completeness obligation, while an ACORD.CODE.NOT_IN_ENUM points at an interoperability defect.

Every outcome — accepted or rejected — is written to the shared append-only store described in Audit Log Schema Design, stamped with the manifest version, the ordered reason codes, and a hash of the canonical payload. That record is what lets a compliance team answer, months later and without replaying the live pipeline, exactly why a given commercial application was diverted to manual review. Because the reason codes are stable enum values rather than free text, they aggregate cleanly into the data-quality metrics a data-governance report presents to a regulator.

Validation failures cluster into a small set of named scenarios, each with a deterministic diagnostic path.

This validator is the entry gate; several focused guides expand the pieces it depends on. The concrete mechanics of expressing an ACORD form as a machine-checkable contract — a subset of ACORD 125 fields as JSON Schema, validated with the jsonschema library and layered with the cross-field checks a schema cannot express — are worked end to end in validating ACORD 125 fields against JSON Schema. The typed contract a validated payload becomes is defined in Policy Schema Design, the payloads this stage consumes are produced by Field Mapping Strategies, the outcomes are persisted through Audit Log Schema Design, and the per-jurisdiction reference sets come from State Regulation Mapping.

• Core Architecture & Compliance Mapping — the parent domain defining the contracts this validator enforces
• Validating ACORD 125 Fields Against JSON Schema — the concrete JSON Schema and cross-field implementation for a commercial application
• Field Mapping Strategies — normalizes carrier payloads into the canonical ACORD fields this stage validates
• Policy Schema Design — the typed contract a payload becomes once it clears this gate
• Audit Log Schema Design — the append-only store every validation outcome is written to
• State Regulation Mapping — supplies the per-jurisdiction code lists and required elements the manifest carries