Camelot Lattice vs Stream Mode for Declarations Tables

Camelot’s two extraction flavors — lattice, which reads ruling lines, and stream, which infers columns from whitespace — are not interchangeable, and this comparison under Table Parsing with Camelot settles which to run on ruled versus borderless declarations and rating tables. Force lattice on a borderless premium schedule and you get zero tables; force stream on a dense ruled grid and it merges adjacent columns until a deductible lands in the limit field. The right answer is a measured auto-selection heuristic, not a hardcoded guess.

A declarations page carries its most valuable data — coverage limits, per-peril deductibles, premium splits — inside table geometry, and how that geometry is drawn decides which flavor can recover it. Modern carrier declarations and ruled loss runs render explicit cell borders as vector strokes, which is exactly what lattice needs: it uses OpenCV to find horizontal and vertical lines, intersects them into a cell grid, and assigns each text fragment to the cell that contains it. Legacy mainframe-generated schedules render the same data as whitespace-aligned text with no strokes at all, which is what stream needs: it clusters text by its x-coordinates and guesses column boundaries from the gaps.

The failure mode that costs money is silent column misalignment. Run stream over a ruled grid and its whitespace heuristic splits on the internal spacing of a two-word coverage name, fabricating a phantom column that shifts every subsequent value one cell right — nothing throws, and a deductible silently persists as a limit. Run lattice over a borderless schedule and it finds no lines, returns an empty table list, and a naive pipeline logs “no tables found” and moves on. Two further hazards break both flavors in different ways: a vertically merged cell such as a spanned “Total Insured Value” header, and a multi-line column header that wraps across two text rows. This page shows the real Camelot call for each flavor, reads the accuracy and whitespace metrics out of parsing_report, and builds the heuristic that picks the flavor from measured geometry rather than assumption.

Camelot’s behaviour is dominated by native dependencies, so pin the whole stack — an unpinned Ghostscript or OpenCV changes lattice line detection and makes extraction non-reproducible across deploys. Lattice additionally requires the system Ghostscript binary in the runtime image; stream does not.

Copy
python -m venv .venv && source .venv/bin/activate
pip install "camelot-py[cv]==0.11.0" "opencv-python-headless==4.10.0.84" \
            "ghostscript==0.7" "pandas==2.2.2"
gs --version   # lattice shells out to system Ghostscript; must resolve

Confirm the page has a vector layer before either flavor runs. An image-only scan carries no strokes for lattice and no recoverable text for stream, so it must be diverted to the OCR Integration & Sync pathway rather than parsed here.

Lattice takes a line_scale sensitivity that governs how thin a rule it will treat as a table border. The decisive output is not the DataFrame but table.parsing_report: a dict carrying accuracy (Camelot’s confidence that text fell cleanly into cells) and whitespace (the percentage of blank cells, a proxy for over- or under-segmentation).

Copy
from __future__ import annotations

from dataclasses import dataclass

import camelot
import pandas as pd
import structlog

log = structlog.get_logger("camelot.flavor")


@dataclass(frozen=True, slots=True)
class FlavorRead:
    flavor: str
    df: pd.DataFrame
    accuracy: float
    whitespace: float


def run_lattice(pdf_path: str, pages: str, line_scale: int = 40) -> list[FlavorRead]:
    tables = camelot.read_pdf(
        pdf_path,
        pages=pages,
        flavor="lattice",
        line_scale=line_scale,
        split_text=True,
        strip_text="\n",
    )
    reads = [
        FlavorRead(
            flavor="lattice",
            df=t.df.copy(),
            accuracy=float(t.parsing_report["accuracy"]),
            whitespace=float(t.parsing_report["whitespace"]),
        )
        for t in tables
    ]
    log.info("lattice.done", tables=len(reads),
             best=max((r.accuracy for r in reads), default=0.0))
    return reads

Stream infers columns from whitespace, which is where a ruled grid trips it. For a genuinely borderless carrier form you constrain it with table_areas (the x1,y1,x2,y2 region of the table) and columns (the x-coordinates of the separators), both sourced per carrier from the versioned manifest. Without those hints, stream over-segments a two-word coverage label into two columns.

Copy
def run_stream(
    pdf_path: str,
    pages: str,
    *,
    table_areas: list[str] | None = None,
    columns: list[str] | None = None,
) -> list[FlavorRead]:
    tables = camelot.read_pdf(
        pdf_path,
        pages=pages,
        flavor="stream",
        table_areas=table_areas,   # e.g. ["50,700,560,120"]
        columns=columns,           # e.g. ["120,300,420"]
        split_text=True,
        strip_text="\n",
    )
    reads = [
        FlavorRead(
            flavor="stream",
            df=t.df.copy(),
            accuracy=float(t.parsing_report["accuracy"]),
            whitespace=float(t.parsing_report["whitespace"]),
        )
        for t in tables
    ]
    log.info("stream.done", tables=len(reads),
             best=max((r.accuracy for r in reads), default=0.0))
    return reads

The heuristic runs lattice first — it is precise when strokes exist and cheap to reject when they do not — reads the best accuracy, and only falls through to stream when lattice is below the floor. It picks the higher-accuracy flavor, breaking ties toward the lower whitespace percentage, and raises when neither clears the floor so the document leaves the fast path instead of poisoning the store.

Copy
class NoConfidentTable(RuntimeError):
    """Neither flavor reconstructed the grid above the accuracy floor."""


def select_flavor(
    pdf_path: str,
    pages: str,
    *,
    accuracy_floor: float = 90.0,
    carrier_columns: list[str] | None = None,
    carrier_areas: list[str] | None = None,
) -> FlavorRead:
    lattice = run_lattice(pdf_path, pages)
    best_lattice = max(lattice, key=lambda r: r.accuracy, default=None)
    if best_lattice and best_lattice.accuracy >= accuracy_floor:
        return best_lattice

    stream = run_stream(
        pdf_path, pages, table_areas=carrier_areas, columns=carrier_columns
    )
    candidates = [r for r in (*lattice, *stream) if not r.df.empty]
    if not candidates:
        raise NoConfidentTable(f"{pdf_path} pages {pages}: no tables at all")

    # Highest accuracy wins; lower whitespace breaks ties.
    best = max(candidates, key=lambda r: (r.accuracy, -r.whitespace))
    if best.accuracy < accuracy_floor:
        raise NoConfidentTable(
            f"{pdf_path} pages {pages}: best accuracy {best.accuracy:.1f}"
        )
    log.info("flavor.selected", flavor=best.flavor, accuracy=best.accuracy)
    return best

Prove the selector on two fixtures — a ruled declarations page and a borderless legacy schedule — and assert it picks the flavor each is drawn for. Then treat the accuracy score as the benchmark: the correct flavor should clear the floor comfortably while the wrong one lands well below it, which is precisely the signal the heuristic routes on.

Copy
def test_ruled_page_selects_lattice(ruled_pdf: str) -> None:
    read = select_flavor(ruled_pdf, "1")
    assert read.flavor == "lattice"
    assert read.accuracy >= 90.0


def test_borderless_page_selects_stream(legacy_pdf: str) -> None:
    read = select_flavor(
        legacy_pdf, "1",
        carrier_areas=["50,700,560,120"],
        carrier_columns=["120,300,420"],
    )
    assert read.flavor == "stream"


def test_wrong_flavor_scores_below_floor(ruled_pdf: str) -> None:
    # Forcing stream on a ruled grid must score poorly, not silently "succeed".
    forced = run_stream(ruled_pdf, "1")
    assert max((r.accuracy for r in forced), default=0.0) < 90.0

The comparison below summarizes what each flavor does well, where it breaks, and which declarations geometry it belongs to:

Choose lattice as the default for modern carrier declarations and any loss run rendered with visible cell borders. Its stroke-based cell grid is deterministic — a recovered value traces to the exact box a rule pair bounds — and its signature failure (an empty list on a borderless page) is loud and cheap to detect, which makes it safe to try first. When lattice under-detects thin or anti-aliased hairline rules, raise line_scale for that carrier in the manifest before abandoning it; do not jump straight to stream.

Choose stream only for genuinely borderless forms — the legacy mainframe schedules and typewriter-era rating tables that draw columns with spaces alone — and never run it unconstrained on those. Always pass carrier-calibrated columns and table_areas from the manifest, because stream’s whitespace inference is exactly what fabricates the phantom column that shifts a deductible into the limit field. The safest production posture is the auto-selector: lattice first, stream with hints as the fallback, and a hard NoConfidentTable when both stay below the floor so the document is quarantined rather than trusted. That accuracy-scored routing, extended with memory-guarded page chunking and merged-cell consolidation for large binders, is developed in depth in Optimizing Camelot for Complex Insurance Tables. Whichever flavor wins, hand the recovered cells to Field Mapping Strategies for canonical ACORD normalization, and when a page turns out to have no grid at all, fall back to the pdfplumber-versus-PyMuPDF native-text path rather than forcing either flavor into a degraded state.

The flavor decision is part of a recovered value’s provenance, so persist it. Alongside each validated row, write the source PDF’s SHA-256 hash, the selected flavor, the line_scale or columns hints used, the parsing_report accuracy and whitespace scores, and the pinned Camelot and Ghostscript versions — all append-only. That lineage lets the Coverage Validation Rules engine cross-check a recovered limit against the bound policy, and it satisfies NAIC data-governance and state department-of-insurance examination requests by letting an examiner reconstruct precisely how a premium figure was derived from a specific page under a specific flavor and configuration. Because financial matrices frequently intersect policyholder identifiers, the recovered cells cross a tier boundary only under the logged transitions defined by Data Boundary Enforcement.

• Lattice returns an empty table list on a clearly tabular page. Symptom: zero tables on a page you can see has a grid. Cause: hairline or anti-aliased rules Ghostscript does not surface as strokes. Fix: raise line_scale for that carrier in the manifest; only fall through to stream when no rules truly exist.
• Stream shifts a deductible into the limit column. Symptom: values are present but offset one cell right. Cause: whitespace inference split a multi-word label into a phantom column. Fix: pass explicit columns separators and a table_areas box from the manifest, and validate the recovered column count against the expected schema so a mismatch quarantines rather than persists.
• A merged cell fragments into duplicate rows. Symptom: one logical row appears twice, one copy with a blank leading column. Cause: a vertically spanned “Total Insured Value” header emitted a continuation fragment. Fix: coalesce any row whose first column is empty into its predecessor before the schema gate, as covered in the optimizing guide.
• A multi-line header wraps into a phantom header row. Symptom: the first data row is actually the second line of a wrapped header. Cause: split_text was off or the header spans two text rows. Fix: enable split_text=True and set the true header row count in the manifest so normalization aligns columns correctly.
• Same PDF yields different tables across deploys. Symptom: a previously clean parse regresses with no code change. Cause: an unpinned Ghostscript or OpenCV bump altered line detection. Fix: pin the full native stack, record the versions in the audit log, and replay quarantined documents against the frozen versions to isolate the regression.

• Table Parsing with Camelot — the parent stage this flavor decision sits inside
• Optimizing Camelot for Complex Insurance Tables — chunking, memory guards, and merged-cell consolidation at scale
• pdfplumber vs PyMuPDF for Carrier PDFs — the native-text fallback when a page has no grid
• OCR Integration & Sync — optical fallback for rasterized or scanned tables
• Field Mapping Strategies — normalizing recovered cells into canonical ACORD identifiers