Data-Eng-Bench Establishes Open-Source DBA and Data Engineering Evaluation Standard for AI Agents
dbt Labs, Snowflake, and researchers have released Data-Eng-Bench, a comprehensive open benchmark assessing AI agents on live SQL, dbt, and DBA tasks.

By Aarav Mehta, Cloud & Enterprise Tech
, updated 9 min read
The open-source data community, led by dbt Labs, Snowflake research teams, and academic contributors, has released Data-Eng-Bench, an open evaluation suite designed to test artificial intelligence agents across end-to-end database administration and analytics engineering tasks. By shifting evaluation focus from toy text-to-SQL queries toward full data warehouse lifecycle operations, the benchmark establishes a rigorous standard for assessing autonomous data agents.
For years, foundation model developers evaluated database capabilities using legacy datasets like Spider and WikiSQL, which measured whether a model could write a single SELECT statement based on an English prompt. While frontier models now score above 85% on these static translation exercises, real-world data engineering involves far more than writing isolated SQL queries.
Data engineers must refactor complex data build tool (dbt) transformation pipelines, debug circular dependencies, write idempotent schema migration scripts, manage partition keys, and tune warehouse query costs without breaking production reporting dashboards.
Evaluating real-world data pipelines across four operational tiers
Data-Eng-Bench evaluates models by spinning up ephemeral, isolated container environments connected to real database backends, including PostgreSQL, Snowflake, BigQuery, and DuckDB. Rather than grading text strings, the evaluation framework executes the agent's code directly against live databases and verifies the final state of the schema and data tables.
The benchmark categorises operational challenges across four distinct tiers:
Tier 1: Analytics engineering and dbt pipeline refactoring
Agents are tasked with resolving compilation errors in multi-thousand-line dbt projects. Tasks include refactoring jinja macros, resolving circular model references, updating materialisation strategies from views to incremental tables, and repairing broken schema.yml validation tests.
Tier 2: Database administration and schema evolution
This track tests whether an agent can safely execute non-destructive database migrations. Challenges include adding columns with default values to tables with millions of rows without locking the table, refactoring foreign key relationships, and migrating data types across active transactional databases.
Tier 3: Query performance tuning and indexing
Agents receive slow-running analytical queries alongside database query execution plans (EXPLAIN Analyse output). The agent must diagnose full table scans, create appropriate composite indexes, rewrite inefficient correlated subqueries into common table expressions (CTEs), and restructure join orders to minimise intermediate memory spills.
Tier 4: Warehouse cost and partition optimisation
In cloud data warehouse targets like Snowflake and BigQuery, poorly designed queries can run up thousands of dollars in compute costs. Agents are evaluated on their ability to configure table clustering keys, partition time-series datasets, and configure auto-suspend warehouse policies to minimise credit consumption.
Comparing leading models across data engineering benchmarks
Comprehensive testing across proprietary and open-weight models revealed stark disparities between general coding proficiency and specialized data systems competence:
| Model | dbt Refactoring Success (%) | Schema Migration (%) | Index & Query Tuning (%) | Cost Optimisation (%) | Overall Score (%) | | :--- | :--- | :--- | :--- | :--- | :--- | | Claude Mythos 5.2 | 58.4 | 52.1 | 64.3 | 49.0 | 56.0 | | GPT-5 Turbo | 54.2 | 47.8 | 59.1 | 44.5 | 51.4 | | Claude 3.5 Sonnet | 48.7 | 42.0 | 53.8 | 38.2 | 45.7 | | DeepSeek-Coder-V3 | 43.1 | 38.6 | 47.9 | 34.0 | 40.9 | | Llama 3.3 70B Instruct | 36.5 | 31.2 | 39.4 | 28.1 | 33.8 |
Across all models tested, agents struggled most severely on Tier 2 schema migrations, achieving an average success rate of just 41.5%. Models frequently generated migration scripts that inadvertently dropped existing table data, failed to account for nullable constraints, or introduced deadlocks during concurrent write operations.
On dbt refactoring tasks, models frequently hallucinated jinja macro syntax or created circular dependency graphs in the dbt project directed acyclic graph (DAG), demonstrating an incomplete understanding of project-wide state.
Best practices for integrating AI agents into production data teams
The benchmark findings offer practical guidance for engineering organisations seeking to integrate coding assistants and autonomous agents into production data stacks:
Maintain isolated staging environments with realistic data samples: Never permit an AI agent to execute database migrations or DDL statements against production databases without testing the commands inside an ephemeral sandbox.
Implement automated CI/CD pipeline validation: Integrate dbt compile, dbt test, and sqlfluff linting checks directly into pull request workflows. Reject any agent-generated pull request that fails automated dependency checks.
Require explicit human-in-the-loop review for schema changes: Because models struggle with table-locking behaviour and schema evolution safety, senior analytics engineers must audit all migration scripts modifying primary or foreign keys.
Provide agents with full database schema context: Agents perform significantly better when supplied with complete table DDL, row count statistics, and query execution plans through Model Context Protocol (MCP) server connectors rather than plain text prompts.
Data-Eng-Bench source code, evaluation datasets, and container harnesses are available under the Apache 2.0 open-source licence on GitHub, providing data teams with a reproducible framework to benchmark autonomous data assistants.
Frequently Asked Questions
How does Data-Eng-Bench differ from text-to-SQL benchmarks like Spider?
Spider tests whether a model can write a single SQL query from text. Data-Eng-Bench evaluates whether an agent can manage real data systems: refactoring dbt projects, executing safe schema migrations, tuning slow query plans, and managing cloud warehouse costs.
Why do AI agents struggle with database migrations?
Schema migrations require understanding state over time and the physical realities of database storage engines. Models often write queries that work logically but lock entire tables, delete historical data, or crash under high transactional concurrency.
How can data teams use this benchmark internally?
Data teams can run Data-Eng-Bench inside local Docker containers to objectively measure how different AI models and coding agents perform on their specific database engines before granting them access to internal repositories.
Sources
- 1dbt Labs Developer Communitygetdbt.com
- 2BIRD-Bench Research Consortiumbird-bench.github.io