Semantic Layers Boost AI Analyst Accuracy to 96%, dbt Labs Finds

A benchmark from dbt Labs' developer experience team found that an AI model answering questions through a semantic layer scored 96 to 100% accuracy, compared with 80 to 91% when the same model wrote raw SQL directly against a database schema.

Alex Noonan, senior developer experience advocate at dbt Labs, presented the findings at dbt Summit 2026, tracing self-service analytics failures from a 1971 natural-language moon-rock query system through 1990s OLAP cubes and later natural-language BI features. Noonan argued this attempt differs because large language models have solved teaching machines users' words; what remains is grounding - teaching the machine what those words mean inside a business.

Noonan said every point of the accuracy gap represents a confidently wrong number landing in someone's Slack channel. Teams are converging on three grounding approaches: a semantic layer with pre-defined metrics, an ontology-style entity and relationship model in the style of Palantir, and an agent context layer assembling relevant context at query time, which he described as the direction Fivetran and dbt are building toward together.

Much of the talk covered unglamorous fundamentals separating a demo from a durable system: data quality and observability practices such as tests, freshness checks and anomaly detection; documentation written to be read by machines as much as people; and consistent data modeling. Noonan split context engineering into "just in time" context - stable metadata like a semantic layer or agent skills - and "just in case" context, the search infrastructure needed for high-volume unstructured sources like call transcripts and support tickets.

On trust, Noonan recommended keeping the chat interface in a public channel like Slack so mistakes and corrections happen visibly rather than in a private thread, and cautioned against over-relying on "LLM as judge" evaluation since a judge model shares the same blind spots as the model it grades. Noonan cited a figure that roughly half of generative AI pilots get abandoned after proof of concept, and pointed to this year's State of Analytics Engineering report, in which about half of dbt users said they're actively building toward conversational analytics. Noonan returned to two lines: "the worst thing that you can do is give somebody a confidently wrong answer," and "silent failures are the killer in any IT system."

Noonan closed with a live demo of a personal AI trading assistant built on dbt's MCP server, pulling SEC filings and technical indicators, noting a trade it had recommended closed profitable the day before the talk.

Source: dbt Labs Blog