Add basic sql benchmark runner for running sql benchmarks by Omega359 · Pull Request #23052 · apache/datafusion

Omega359 · 2026-06-19T20:27:35Z

Which issue does this PR close?

Part of CLI (non criterion) interface for sql runner #21937.

Rationale for this change

Running sql benchmarks using environment variables for configuration is awkward and error prone and strictly using criterion, while statistically much better, is quite slow compared to using simple iterations.

This PR is the first version of a benchmark runner for sql benchmarks that will eventually use arguments for all benchmark configuration options.

What changes are included in this PR?

A simple benchmark runner that can list out the sql benchmarks and run a benchmark using iterations or criterion allowing for specifying a single query if desired.

Future enhancements will use arguments for benchmark configuration vs just using environment variables as well as providing help and tying this into bench.sh

Are these changes tested?

Yes. I have a script that tests all current sql benchmarks both with and without criterion. Here is an portion of it for the single clickbench benchmark:

# clickbench single basic long flags
env DATA_DIR=data CLICKBENCH_TYPE=single cargo run -p datafusion-benchmarks --bin benchmark_runner -- clickbench --query 0 --iterations 5 --output results/benchmark_runner/clickbench_single_long.json

# clickbench single basic short flags
env DATA_DIR=data CLICKBENCH_TYPE=single cargo run -p datafusion-benchmarks --bin benchmark_runner -- clickbench --query 0 -i 5 -o results/benchmark_runner/clickbench_single_short.json

# clickbench single basic env iterations
env DATA_DIR=data CLICKBENCH_TYPE=single ITERATIONS=5 cargo run -p datafusion-benchmarks --bin benchmark_runner -- clickbench --query 0 --output results/benchmark_runner/clickbench_single_env_iterations.json

# clickbench single criterion with baseline
env DATA_DIR=data CLICKBENCH_TYPE=single cargo run -p datafusion-benchmarks --bin benchmark_runner -- clickbench --query 0 --criterion --save-baseline benchmark_runner_acceptance

# clickbench single criterion without baseline
env DATA_DIR=data CLICKBENCH_TYPE=single cargo run -p datafusion-benchmarks --bin benchmark_runner -- clickbench --query 0 --criterion

The existing cargo bench approach still works the same (criterion only):

env DATA_DIR=data CLICKBENCH_TYPE=single BENCH_NAME=clickbench BENCH_QUERY=0 cargo bench -p datafusion-benchmarks --bench sql`

Are there any user-facing changes?

No.

… with either simple iterations or using criterion.

Omega359 · 2026-06-22T03:15:20Z

The test failure seems unrelated and all tests pass locally.

alamb · 2026-06-23T16:34:18Z

will try and look at this one shortly

alamb · 2026-06-23T18:30:11Z

CI failure will be fixed via

chore: cargo update -p quinn to resolve security audit issue #23122

alamb

The code looks good to e -- thank you @Omega359 . I also tried it out and it works great

Let me know if you want me to merge this PR now and iterate in follow on PRs or if you want to make changes to this one

Some minor usability feedback (for a follow on PR):

Explicit list command

Right now I can run

nice cargo run --profile=ci --bin benchmark_runner

And it will print a nice list of available commands ❤️

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ DATA_DIR=benchmarks/data  nice cargo run --profile=ci --bin benchmark_runner --
    Finished `ci` profile [unoptimized] target(s) in 0.21s
     Running `target/ci/benchmark_runner`
SQL benchmarks:
  clickbench               43 queries
...
  tpch                     22 queries
  wide_schema              4 queries

However, it was not at all clear to me given the help text:

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ DATA_DIR=benchmarks/data  nice cargo run --profile=ci --bin benchmark_runner -- --help
    Finished `ci` profile [unoptimized] target(s) in 0.21s
     Running `target/ci/benchmark_runner --help`
Run DataFusion SQL benchmarks

Usage: benchmark_runner [OPTIONS] [BENCHMARK]

Arguments:
  [BENCHMARK]  SQL benchmark group to run

Options:
  -q, --query <QUERY>
          [env: BENCH_QUERY=]

....
  -o, --output <OUTPUT>
          Write simple runner results as JSON to this path
      --save-baseline <BASELINE>
          Save Criterion measurements to the named baseline
  -h, --help
          Print help

Suggestion: add an alias --list or something:

cargo run --profile=ci --bin benchmark_runner -- --list

Simpler command

It would be great if I could use a simpler command line with good defaults.

For example,

nice cargo run --profile=ci --bin benchmark_runner -- clickbench --query 0

However, that doesn't work:

...
     Running `target/ci/benchmark_runner clickbench --query 0`
Error: Error during planning: No files found at file:///Users/andrewlamb/Software/datafusion/data/hits.parquet. Cannot infer schema from an empty location; either add data files or declare an explicit schema for the table.

I had to explicitly specify the DATA_DIR

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ DATA_DIR=benchmarks/data  nice cargo run --profile=ci --bin benchmark_runner -- clickbench --query 0
    Finished `ci` profile [unoptimized] target(s) in 0.20s
     Running `target/ci/benchmark_runner clickbench --query 0`
clickbench/Q00 iteration 0: 4.2 ms, 1 rows
clickbench/Q00 iteration 1: 4.4 ms, 1 rows
clickbench/Q00 iteration 2: 4.4 ms, 1 rows

alamb · 2026-06-23T18:30:33Z

+// specific language governing permissions and limitations
+// under the License.
+
+//! Shared SQL benchmark runner used by `benchmark_runner` and the Criterion


alamb · 2026-06-23T18:33:30Z

+#[tokio::main]
+async fn main() {
+    env_logger::init();
+    if let Err(error) = sql_benchmark_runner::run_cli().await {


Minor nit -- it might make sense to move run_cli and relevant code such as argument processing, main, etc in the actual binary rather than a function in the a crate.

I don't think the argument processing will be used by anything other than this binary

alamb · 2026-06-23T18:44:15Z

+    },
+}
+
+#[derive(Debug, Parser)]


see comment above -- I recommend moving the criterion / arg parsing out of this module (so this is only the shared implementation with the criterion runner)

alamb · 2026-06-23T18:44:57Z

+}
+
+/// Runs the selected SQL benchmarks through a caller-provided Criterion instance.
+pub fn run_criterion_benchmarks_impl(


I recommend moving this into benchmarks/benches/sql.rs so that the criterion specific stuff is separated from the core benchmark running logic

Omega359 · 2026-06-23T20:06:31Z

As usual I'll work on applying the improvements to this PR @alamb rather than followup PR's :)

alamb · 2026-06-23T21:23:24Z

As usual I'll work on applying the improvements to this PR @alamb rather than followup PR's :)

I always want to ask though :)

Omega359 added 2 commits June 21, 2026 12:13

Add basic sql benchmark runner that allows for running sql benchmarks…

4894574

… with either simple iterations or using criterion.

Fixed clippy errors.

1edafef

Omega359 force-pushed the benchmark_runner_v2 branch from 1cfedab to 1edafef Compare June 21, 2026 16:13

Omega359 marked this pull request as ready for review June 22, 2026 03:15

Omega359 and others added 3 commits June 21, 2026 23:15

Merge branch 'main' into benchmark_runner_v2

d5c4ec4

Merge branch 'main' into benchmark_runner_v2

47d08bc

Merge branch 'main' into benchmark_runner_v2

f34ba7a

Removed test, it's not all that useful.

279cf99

Merge branch 'main' into benchmark_runner_v2

4e61b86

alamb approved these changes Jun 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add basic sql benchmark runner for running sql benchmarks#23052

Add basic sql benchmark runner for running sql benchmarks#23052
Omega359 wants to merge 7 commits into
apache:mainfrom
Omega359:benchmark_runner_v2

Omega359 commented Jun 19, 2026 •

edited

Loading

Uh oh!

Omega359 commented Jun 22, 2026

Uh oh!

alamb commented Jun 23, 2026

Uh oh!

alamb commented Jun 23, 2026

Uh oh!

alamb left a comment

Uh oh!

alamb Jun 23, 2026

Uh oh!

alamb Jun 23, 2026

Uh oh!

alamb Jun 23, 2026

Uh oh!

alamb Jun 23, 2026

Uh oh!

Omega359 commented Jun 23, 2026

Uh oh!

alamb commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Omega359 commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Omega359 commented Jun 22, 2026

Uh oh!

alamb commented Jun 23, 2026

Uh oh!

alamb commented Jun 23, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Explicit list command

Simpler command

Uh oh!

alamb Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Omega359 commented Jun 23, 2026

Uh oh!

alamb commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Omega359 commented Jun 19, 2026 •

edited

Loading