When agents write pipelines, declarative dry-runs beat procedural retries

Databricks published a blunt engineering claim this week: if autonomous agents are going to author production data pipelines, the programming model matters as much as the model weights. In a controlled bake-off of 528 agent-built pipelines—22 tasks, 12 messy datasets each, procedural PySpark versus Spark Declarative Pipelines (SDP), all on Opus 4.8—declarative pipelines burned far less cluster time, shipped roughly half as much code, and failed structurally before a run rather than midway through one.

The company also says more than 70% of SDP pipelines on its platform are now created by agents. That figure is self-reported, but the direction is familiar to anyone watching coding agents move from demos into tickets. Once agents write the majority of pipeline code, every retry is a bill—and every unreadable diff is a review tax.

In Databricks’ framing, a procedural pipeline tells Spark how to build each dataset step by step. The only way to know it works is to run it. A declarative pipeline describes the datasets you want and lets Spark decide order, materialization, and much of the boilerplate. Structural bugs—missing columns, missing producers, illegal graph edges—can be rejected in a dry run that does not fully execute on a cluster.

That sounds like an old compiler-versus-interpreter argument. The new twist is who writes the code. Agents iterate by generating, failing, and regenerating. If failure requires a partial cluster run, the cost curve is hostile. If failure is a cheap validation, the same agent can afford more attempts without torching executor-seconds.

The measured gaps are large enough to take seriously even after the usual vendor-blog discounts. Procedural builds used 596 executor-seconds across the measured set versus 17 for declarative—about 34×. Median completed declarative pipelines were 64 lines versus 131 for procedural (~51% less code). Completion rates were similar (255/264 procedural vs 258/264 declarative), so this is not a story about one approach “winning tasks.”

Two caveats, which Databricks mostly owns: dry runs catch structural errors, not silent logic errors. Silent wrong-answer rates were within roughly 15% of each other (about 28% procedural vs 32% declarative)—too close to crown a winner. And declarative spent more LLM tokens up front (~2.3× median token cost: about $0.50 vs $0.26 at Opus 4.8 list rates). The bet is that a fixed token premium beats recurring cluster waste as data grows.

Importantly, the team published a public repository with setup, prompts, datasets, grader, and provenance. That does not make the result independent science, but it raises the evidentiary bar above a slide with a single “34×” callout.

What the post underplays is organizational, not statistical. Declarative wins assume the platform’s checker is trusted and that agents are constrained to emit declarative forms. In mixed estates—legacy notebooks, ad-hoc Spark jobs, partial migrations—agents will still emit procedural shortcuts. Databricks itself notes 19 procedural builds cheated by pulling data into memory and processing row-by-row; none of the declarative builds did. That is an architectural forcing function, not merely a style guide.

Another gap: the benchmark uses one agent (Opus 4.8) and Databricks’ own SDP surface. A weaker agent, a different declarative DSL, or a warehouse without an equivalent dry-run may compress the gap. The thesis is portable—“separate what you want from how it runs so the first part can be checked”—but the magnitude is local until others replicate.

For data platform teams, the practical implication is sharper than “use SDP.” Treat the agent’s authoring contract as a production control. If agents can emit arbitrary PySpark, you have bought a probabilistic cluster bill. If agents can only propose declarative graphs that fail closed in validation, you have bought a reviewable artifact and a cheaper failure mode.

That also reframes code review. Half as many lines is not automatically half the risk—silent bugs still need tests—but it is a different review surface: schemas, dependencies, and declared outputs instead of imperative glue. In an agent-majority world, that may be the only review humans can sustain.

Over the next six months, watch whether other lakehouse and orchestration vendors publish comparable agent-vs-agent bake-offs with open harnesses—not marketing win rates, but executor-time and silent-error tables. If declarative (or SQL-first, or IaC-like) authoring consistently makes agent mistakes cheap, the industry argument shifts: model choice still matters, but the scarce resource becomes the cost of being wrong on a cluster.