Verily Health Tracks 30+ Pathogens With a Five-Person Team and One dbt Package

A five-person data team at Verily Health runs near real-time wastewater surveillance for more than 30 pathogens using dbt, according to David Drummond, a data engineer and technical lead at the company. He spoke at a dbt Summit 2026 breakout session on wastewater monitoring at scale.

Verily's public Sightline dashboard lets anyone follow regional and seasonal trends for pathogens including COVID-19, influenza A, norovirus and rotavirus. The data comes from wastewater samples that government partners collect and Verily's labs process. It reaches academic researchers, government agencies and organizations like the CDC, usually arriving with a lag of only a day or two.

The team builds on FHIR, the open healthcare data standard maintained by the international HL7 organization. FHIR's more than 150 resource types, such as Patient, Observation, Specimen and Location, arrive as deeply nested JSON or XML, which raises recurring questions about how far to flatten the data, what to use as a primary key, and whether to track full history or only the latest snapshot.

Instead of solving those questions separately for each of the two or three dozen datasets it supports, the team built a shared dbt package with parameterized macros for each FHIR resource. The same observation macro that tracks influenza A levels in a wastewater sample also tracks a person's A1C reading from a continuous glucose monitor, with sensitivity and grain adjusted through macro parameters rather than duplicated code.

Two heuristics keep the models usable: composite keys are capped at around three columns, and most tables stay at roughly 30 columns or fewer, even when that means choosing a grain other than the finest one technically possible. The pipeline follows a medallion structure, from a restricted, permissioned bronze layer through source-aligned staging models that mirror FHIR terminology, optional intermediate models for testing, and domain-centric mart models written in language researchers and scientists use.

dbt freshness tests monitor not only the team's own pipeline but also the wide range of formats, from fixed-width files to CSVs, that arrive from academic partners. Because the team is small relative to its stakeholder base, Drummond said the group leans on dbt's semantic layer and documentation, including dbt docs, to support self-serve analysis rather than acting as a bottleneck.

The approach was tested at scale this past summer, when Verily partnered with health agencies and academic institutions to extend Sightline monitoring to World Cup host cities, including the stadium in Kansas City. Drummond reported no major outbreaks during the tournament.

Source: dbt Labs Blog