要点
- Databricks and Turing built a shareable enterprise benchmark around Calderwick Energy, a fictional UK electricity and gas supplier, with hundreds of tables, the largest about 120 million rows of half-hourly meter data, plus notebooks, dashboards, saved queries, and outdated copies.
- Each agent was run 5 times per task in the same live Databricks workspace. Answers were graded against a pre-set rubric.
- Genie recorded a 91.0% pass rate, 100.3 seconds average wall time, and 431,829 average tokens per run. Unnamed Agent A scored 45.0% in 434.6 seconds with 889,019 tokens. Agent B scored 58.0% in 306.5 seconds with 1,051,070 tokens.
- Genie ran natively with Genie Ontology. The other agents used a Databricks toolbelt that explicitly excluded the Genie MCP, so they did not get the context layer being tested.
- In a worked example, a table column showed 0.31 p/kWh for FY24/25, but a notebook recorded the historical Commercial planning rate as 0.52. Genie returned 0.52 in all five runs. The other agents did not.
Databricks published a benchmark blog during its Week of Genie, written with Turing, that compares Genie plus Genie Ontology with two leading frontier agents on business questions in a live Databricks workspace. The headline numbers are stark. Genie passed 91.0% of tasks, against 45.0% and 58.0% for the unnamed rivals, in about a quarter to a third of the time and with less than half the tokens. Those figures should not be read as a clean model-versus-model result. Genie ran at home, with a context layer built for that workspace. The other agents were not given that layer. The more durable point sits underneath the ranking. On messy enterprise questions, the table that looks like the answer is sometimes the wrong object, and the right number lives in a notebook that a search has to know how to find.
[1]What was tested
Databricks argues that existing data-agent benchmarks mostly give a set of tables, and sometimes the pipeline code that built them, and that scores on those tests look similar across agents. Real business questions, it says, depend on knowledge stored in notebooks, dashboards, saved SQL, documents, and other analytical assets, including copied notebooks, abandoned analyses, frozen reporting tables, and dashboards nobody has touched in months.
The company and Turing built a fictional firm, Calderwick Energy, a UK electricity and gas supplier, whose mess is meant to resemble a typical data team's workspace. There are hundreds of tables. The largest has about 120 million rows of half-hourly meter data. On top sit notebooks, scheduled jobs, dashboards, saved queries, code, metrics, documents, and Genie Agents. Databricks says a previous Genie-plus-ontology evaluation used a proprietary dataset, and that this one is designed so others can also run it.
Each agent received the same questions and was run five times per task against the same live workspace. Graders used a calibrated rubric, with acceptable numeric tolerances set in advance. A response passed only if it met the required checks. The post also recorded how agents found and used business context. The published overall scores are: Genie, 91.0% pass rate, 100.3 seconds, 431,829 tokens; Agent A, 45.0%, 434.6 seconds, 889,019 tokens; Agent B, 58.0%, 306.5 seconds, 1,051,070 tokens. The blog's title frames this as 1.6 times the accuracy with less than half the tokens, which matches 91 divided by Agent B's 58, not the larger gap against Agent A.
[1]The 0.31 versus 0.52 example
The most useful part of the post is not the leaderboard. It is a single task about a historical Commercial planning rate. A bill cost stack table has a column, cfd_rate_p, and the rows for FY24/25 contain 0.31 pence per kilowatt-hour. An agent that finds the table, filters to the period, and reads the column will return 0.31. Checking more rows would seem to confirm it, because the same rate appears throughout the table.
The notebook that builds the mart says something else. Commercial sets planning rates annually. The mart uses FY25/26 assumptions, including when those assumptions are applied to earlier billing periods. A section titled "Planning rates by year" keeps the history: FY24/25 is 0.52 p/kWh, FY25/26 is 0.31, and a FY26/27 draft is 0.45. The question asked which rate the Commercial team set for FY24/25. The number sitting in the table is the later assumption written backward onto earlier rows.
Databricks says Genie Ontology surfaced a snippet with the historical rates and linked it to the source notebook, 45_mart_bill_cost_stack. Across five Genie runs, the first answer was 0.52 every time. The other agents did not get the historical rate. One returned 0.31 throughout. The other reported 0.31, or questioned whether a year-specific rate was documented, or both.
That example is the thesis in miniature. The wrong answer is not a hallucination in the usual sense. It is a correct reading of the wrong object. A system that can find the notebook note has a path to the right number. A system that is good at reading tables, and has to wander through a workspace to discover that the table is stale, is easy to beat on questions of this kind.
[1]How the comparison was stacked
Genie ran natively. The other agents connected through a Databricks toolbelt. The toolbelt specifically excluded the Genie MCP, a connector that would have given those agents Genie itself, because, the post says, that would have given them access to the Genie capabilities under test. That is a clear design choice, and it is also the central caveat. The benchmark is evidence that an agent with a maintained ontology over workspace knowledge can beat frontier agents that must discover that knowledge through a more general toolbelt. It is not evidence that the same frontier models would still lose if they were given the same ontology.
The rivals are unnamed. The workspace is Databricks. Access to the benchmark, according to the post, is arranged through a Databricks account team. The authors themselves note that in new deployments with little accumulated workspace knowledge, or when questions can be answered from tables and schemas alone, the difference between agents may be smaller, and that this is why other data-agent benchmarks show closer scores.
Those limits do not make the example fake. They do mean the 91% figure is a result on home turf, under rules that withheld the context layer from the competition.
[1]The fair objection, and the point that survives it
The steelman is simple. Databricks built a test that rewards the product it is selling, then advertised a 1.6 times accuracy gap against anonymous systems that were denied the feature under test. A vendor is entitled to show its own software in the environment it was built for. A reader is entitled not to treat that as a general ranking of "enterprise agents."
What survives the objection is still worth writing down. Institutional knowledge often lives beside the table, in the note that says the column has been overwritten by a later planning assumption. Frontier models are already good at following written instructions and writing queries. If the next increment in enterprise accuracy comes from retrieving that note cheaply and reliably, then the scarce asset is a maintained map of the business, not another jump in raw model scores. The stacked benchmark is a poor way to crown a winner. It is a reasonable way to exhibit that failure mode.
[1]What to watch in the next six months
The useful next step is a rerun in which every agent, including the unnamed frontier systems, receives the same ontology or the same exported context pack, on a workspace that can be reached without going through a sales team. If the 91% to 58% gap shrinks once the context layer is shared, the lesson is about ontology, not about Genie as a unique reasoner. If the gap holds, Databricks will have a stronger result than the one it published this week. Until then, treat 0.52 versus 0.31 as the finding, and treat 91% as a home-turf score.
[1]