Verisk's Data Team Trades Mainframe Black Box for Governed, AI-Ready Pipeline

Data engineers on Verisk Analytics' underwriting data team used a dbt Summit 2026 breakout session to walk through a four-stage modernization of a pipeline that feeds actuaries whose pricing decisions get filed with state regulators. Verisk aggregates insurance experience from hundreds of insurers, each translating its systems into a fixed-width format defined by a thousand-page PDF manual.

The data lives in tables with a few billion rows and a couple hundred columns each, and for decades it ran on a mainframe. "The mainframe wasn't broken," Casey Schwegel said, noting it held decades of underwriting rules and did its job well. But tracing where a number came from meant staring into a black box.

Moving to Snowflake with Jupyter notebooks brought version control and visibility, though notebooks weren't built for daily production. Native Snowflake objects like streams, tasks, and dynamic tables enabled daily refreshes but left hundreds of isolated objects to coordinate by hand.

dbt unified the stack with version-controlled models, self-documenting lineage, and tests that halt a pipeline before bad data reaches an actuary. The team organized its project by business line with staging, intermediate, and mart layers, plus a naming convention and a custom macro that resolves where each model lands.

Before dbt, the team juggled three sources of truth that never quite agreed: production logic patched directly in Snowflake, lagging Git code, and Confluence documentation that stopped reflecting reality. dbt doc blocks now keep a single description referenced everywhere.

That foundation is now supporting AI: a custom dbt Wizard skill propagates column descriptions through the DAG, project context improves AI-generated SQL, standards are enforced on pull requests, and job logs are cross-referenced with query history for debugging. Vijay Singh and Mateusz Geca noted the work stays human in the loop. "AI is going to thrive on patterns and context," Schwegel said.

Audience questions turned to onboarding staff without Git or SQL backgrounds, and the team's answer was that some people have to lead the shift. The unresolved challenge is scale: actuaries want ten-plus years of fully updated historical data, and incremental models are still catching up.

Source: dbt Labs Blog