Data Culture's Playbook for Trusting AI Analytics Agents Before They Ship

At dbt Summit 2026, Data Culture analytics engineering director Opeyemi Fabiyi presented a three-step framework for measuring whether an AI analytics agent can be trusted before it reaches a stakeholder. Borrowing the term evaluation-driven development from Chip Huyen's book AI Engineering, Fabiyi summed up the team's rule: don't ship what you can't measure.

Data Culture, a boutique data and AI consultancy, spent its first five years on modern data stack work such as data modeling, BI migrations, and platform migration. Over the past year, client demand shifted toward operationalizing workflows with AI and letting stakeholders chat with their data. That shift led the team to build its own methodology, evaluation framework, and playbook, since used on two production clients: a healthcare provider building clinical operations analytics and an e-commerce retailer building product and sales analytics.

The trigger came during the first client deployment, when Data Culture could not answer a basic question: what is the accuracy of this agent? The team had connected an agent to a Snowflake MCP server for test users but had no number to offer. Months later, dbt Labs' 2026 State of Analytics Engineering report showed the same problem industry-wide, with 71 percent of respondents worried about hallucinated or incorrect data reaching decision-makers and 83 percent naming trust a top organizational priority.

When the team built a first lightweight evaluation pipeline and reviewed results by eye, it found that several so-called failures were cosmetic mismatches a human analyst would have called correct. In one case the agent returned the right monthly gross revenue numbers under a differently named column; in another, the correct three product categories plus a fourth row of legitimate uncategorized products; in a third, the right numbers plus one extra supporting column. A naive exact-match comparison marked all three wrong.

The framework that followed has three steps: define what correct means, define how to compare results, and build layered evaluation logic. The first step means writing golden questions specific enough that two analysts would write the same SQL, then validating expected answers with a domain expert. The second compares query results rather than the query itself, using a four-tier cascade: exact match, value-only comparison ignoring column names, overlapping-column comparison tolerating extra context columns, and approximate-numeric matching tolerating rounding. The third splits evaluation into a deterministic layer handling roughly 80 percent of golden questions with fast, free, reproducible comparisons, and a large language model judge scoring the remaining 20 percent on groundedness, relevance, and comprehensiveness.

With the framework in place, Data Culture ran controlled experiments using the same golden questions and the same model throughout. On the e-commerce deployment, accuracy rose from 0 percent with no semantic context to 50 percent with modeled dbt tables instead of raw source data, 52.6 percent after adding column descriptions, 78.9 percent after adding business rules and default filters, and 94.7 percent after adding verified queries and named metrics. The healthcare deployment followed a similar curve, starting at 0 percent on raw data and clearing Data Culture's internal 90 percent production threshold once the same layers were added.

Fabiyi also cited Anthropic's published account of taking an agent from 21 percent to 95 percent accuracy after building a curated data foundation and an offline validation step. He argued that a dbt project's modeled tables and semantic context are the durable asset in any agentic analytics deployment, while the agent or LLM layer can be swapped out later. Data Culture open sourced its evaluation pipeline, golden question templates, comparison logic, and a technical walkthrough at github.com/datacult/dbt-agent-trust.

Source: dbt Labs Blog