dbt Labs experiment: engineered summaries cut AI skill costs nearly in half

An internal experiment at Fivetran + dbt Labs found that serving a warehouse-built summary of customer call transcripts through dbt's MCP server cut token costs by 48.5% and run time by 43.2% compared with pulling full transcripts directly from a vendor's connector.

The test used one skill, four ways of gathering the same customer context, 80 runs and three measurements: cost, speed and output quality, all on real customer data. The skill, sales-pdf-builder, normally fetches Gong call transcripts through Gong's MCP and generates a branded recap PDF.

The problem, according to the company, was that skills across the organization each pulled raw context from vendor tools, driving up token spend and straining shared API rate limits. A single Gong call transcript can run past 70,000 characters.

Four variants were compared: the Gong MCP control, dbt MCP on the full transcript, dbt MCP on Gong's own brief, and dbt MCP on a summary engineered with the company's own extraction rules via Snowflake's AI_COMPLETE().

The Gong brief was cheapest and fastest, cutting cost and run time by more than half, but in all 20 runs it missed every date, number, job title, decision and next step, and named only one of five tools discussed. The engineered summary retained most of that detail, matched the full-transcript variants on dates and numbers, and was best at naming next steps, though it fell short on titles and missed decisions entirely.

The company said the shortfall can be fixed by editing its own dbt model rather than requesting a vendor feature, and that once context lives in the warehouse, every skill draws from the same modeled, versioned source.

Source: dbt Labs Blog