dbt State helps teams cut warehouse compute and speed up runs

dbt Labs has introduced dbt State, a feature designed to reduce warehouse compute costs and accelerate data pipeline runs. The company reports that roughly 159,000 dbt jobs run daily, most on an hourly basis, even when underlying data hasn't changed. This leads to thousands of models being rebuilt unnecessarily, driving up cloud expenses as data usage grows.

dbt State addresses this by making the dbt engine stateful: it builds only what has changed and skips everything that hasn't. The approach, first previewed as state-aware orchestration (SAO) at a previous dbt Summit, showed customers saving 30% or more on warehouse bills. Feedback led to expanding the feature beyond production environments and the Fusion engine, making it available in development, staging, and CI, as well as in dbt Core v2.0 and as a plugin for v1.7 through v1.12.

The system works by maintaining a control plane that tracks hashes of data and code states, deciding for each model whether a rebuild is needed or if an existing version can be cloned. Pricing is based on daily active target tables (DATT), counting unique reuses per day rather than total rebuilds.

One early adopter, Fanatics Betting and Gaming, saw double-digit compute savings on its first pilot project. The team, led by senior analytics engineer Alvin Chai, runs about 11 projects with a couple thousand models on Snowflake and dbt. Initially, model reuse was near zero (0.2%), but after configuring source freshness and model-level SLAs, reuse rates climbed to roughly 15% in the pilot, with some overscheduled projects reaching 25% and an average around 8%.

Chai noted the shift in focus from managing job schedules to asking how fresh a model actually needs to be, reducing operational complexity. dbt State is available for a 30-day trial, and the company emphasizes that it works anywhere dbt runs, including with orchestrators like Dagster or Airflow.

Source: dbt Labs Blog