dbt v2 Demo Shows Linting Up to 250x Faster, Says dbt Labs Engineer

A dbt Labs engineering leader walked a packed room through dbt v2's new developer experience in a live-demo session, describing a rebuilt SQL engine that he said makes linting 40 to 250 times faster than standard SQLFluff on the same project. Alexander Bogdanowicz, who manages the organization behind dbt v2, said he has worked on the underlying problem for about four years - two at dbt Labs and two at SDF, the company dbt Labs acquired that built much of the language comprehension technology behind v2.

According to Bogdanowicz, dbt has always understood project configuration and how to orchestrate a DAG - a dependency graph of models - but never had an actual understanding of SQL grammar itself until v2. The release addresses that by rewriting the Jinja templating engine in Rust, giving dbt semantic comprehension of the SQL and Jinja it runs. That rewrite, he said, unlocks incremental project state management, baseline static analysis that always checks SQL is grammatically valid, and strict static analysis that catches errors locally before a project runs remotely.

In the demo, enabling the LSP linter - a language-server tool that surfaces errors in the editor - in VS Code showed lint problems inline as Bogdanowicz edited a model, and running dbt lint from the terminal produced the same diagnostics. The linter ships with five new dbt-specific SQLFluff rules, including one that flags hard-coded table references instead of refs or sources and one that flags OR conditions inside joins because they force unnecessary table scans. dbt format and dbt lint --fix cleaned up dozens of violations automatically, leaving only those needing a human judgment call. Bogdanowicz attributed the speed to dbt v2 never running the introspection queries that normally slow linting, since its semantic understanding fills in those gaps without hitting the warehouse.

dbt checks bring project-level assertions natively into SQL, replacing patterns that previously lived in Jinja-based tools like the dbt Project Evaluator package. Checks query an internal, project-generated information schema - a catalog of models, docs, lineage, groups, macros and metrics - through DuckDB, with configurable severities and filters so a check can be scoped to a single model. They also run automatically as part of dbt build.

Docs v2 is built on that same information schema and runs through DuckDB compiled to WebAssembly in the browser, so dbt docs generate and dbt docs serve produce a fast, statically hostable catalog. Full column-level lineage is available as long as the project compiles with strict static analysis on. The VS Code extension's Compare Changes feature runs a live data diff between a modified model and its production version; in the demo, Bogdanowicz said it caught an accidental 90,000-row drop caused by a single bad filter change.

Unit tests can now compile and run entirely on local DuckDB with zero network calls for Snowflake and BigQuery projects, currently an experimental feature. dbt v2 can also batch multiple not-null or uniqueness tests on one model into a single scan, which the company says cuts compute and run time on large projects. An audience poll found roughly 40 percent of the room had already tried dbt v2. Bogdanowicz closed by pointing attendees toward private betas for dbt State and Lake Compute, calling dbt State "a free lunch" that pays for itself in saved compute.

Source: dbt Labs Blog