Figma's dbt Consistency Checkers Catch 100+ Bugs a Month

Figma data scientist Madison Kohls has explained how her team turned dbt singular tests into what they call consistency checkers - checks wired into alerting, task tracking and investigation tools so data and product bugs get caught before they do damage.

The work began with Figma's pre-IPO billing remodel, which moved customers from a "true-up" model, where users could be added to unpaid seats and settled later, to a request-approval model where every seat is approved up front. With more than 500,000 paid plans and 5 million paid users affected, ordinary unit tests and bug bashes weren't enough to surface billing mistakes.

Dagster runs the checks on a schedule against Snowflake, and failures are stored individually and in aggregate. PagerDuty and Slack handle on-call alerts, Asana and Linear track tasks, and Hex supports investigation, including a write-back path for marking a failure as expected.

During the remodel, teams built more than 100 consistency checkers and caught five or more billing discrepancies serious enough to have blocked the remodel or the IPO. Today Figma runs more than 450 checkers across 15 product and business functions, catching over 100 real code and data issues in a typical month. The pattern now also covers Figma's AI credit system, spanning more than 45 AI features used by over 700,000 customers.

Early results were noisy: Kohls said 67 percent of initial failures were false positives, split roughly between stale data syncs, wrong SQL logic and permission gaps. Checking failures against a later sync, using AI agents to help write and iterate on SQL, and getting legal sign-off for fuller datasets cut that rate to 33 percent.

In the Q&A, Kohls drew a line between routine dbt tests, which stay in the DAG to catch model-breaking issues, and consistency checkers, reserved for product or business-facing problems. Ownership sits with software engineering as with unit tests, with data science partnering on more complex SQL.

Source: dbt Labs Blog