Twilio Consolidates Three Data Systems into One ClickHouse Pipeline Governed by dbt

Twilio's Voice & Video Insights team outlined how it replaced three separate distributed systems with a single ClickHouse-backed pipeline managed through dbt, in a breakout session at dbt Summit 2026 titled "ClickHouse + dbt: Building a GitOps workflow at Twilio." Team members Marek Radonsky and Pavan Kotikalapudi presented alongside Alex Francoeur, director of product management at ClickHouse.

Twilio's Voice Insights product delivers near real-time analytics on calls and conferences, and the company processes more than 2.5 trillion interactions annually for customers including Best Buy, Netflix, and Uber. For a decade, that pipeline relied on Kafka for ingesting raw call metrics, Spark Streaming applications for processing, and Elasticsearch for serving results to Twilio's console and REST APIs.

Maintaining three enterprise-scale systems for one workflow carried a heavy operational and infrastructure burden. As schema changes spread across multiple developers and teams, making manual changes directly to the cluster grew increasingly risky. Twilio consolidated ingestion, transformation, and serving into ClickHouse alone, using materialized views for streaming transformations that previously required custom Spark Scala applications.

The workflow evolved in stages: from tracking DDL changes in shared documents, to version control that still permitted direct cluster changes, to a migration tool with a GitOps approach storing schema as code in GitHub, and finally to dbt with the open-source dbt-clickhouse adapter for transformations, testing, and lineage. As pipelines multiplied, Twilio split one monolithic dbt project into separate projects per product area with a shared layer for common logic, orchestrated through Argo Workflows.

Because ClickHouse is open source, engineers could run real ClickHouse test containers for local and CI integration tests instead of relying on mocks. Developers can each deploy a preview environment against a slice of production data, an approach the team calls a database per branch. Automated lineage from dbt artifacts replaced hand-maintained architecture diagrams, and Twilio contributed a materialized-view model update capability back to the open-source dbt-clickhouse project.

The result: three distributed systems became one, custom Spark applications became SQL models and materialized views in dbt, and manual production changes became version-controlled and audited, letting Twilio redeploy the same pipeline consistently across regions. Francoeur closed with product news: the long-established dbt v1 ClickHouse adapter remains fully supported, a dbt v2 adapter had just entered public beta and was nearing feature parity with v1, and ClickHouse support in the dbt platform was newly available in private beta.

Source: dbt Labs Blog