Language models generated GPU code that ran some analytical database queries faster, but the results varied by model and query. In a benchmark of the 22 TPC-H queries at scale factor 10 on an H100, GPT-5.5 in the full CUDA setup recorded a 100.0% pass rate and a 2.11× overall speedup over the compiled TorchPlan baseline. The reported Qwen3.5-397B-A17B configuration using Triton with full optimization also recorded a 100.0% pass rate and a 1.26× speedup. GPT-OSS-120B reached the same 1.26× speedup, but its pass rate was 86.4%.
The test began with validated plans
The comparison was built around validated query plans. The benchmark used a two-stage pipeline: it constructed and validated a baseline TorchPlan for each query, then asked language models to inject optimized Triton or CUDA kernels while preserving the query’s semantics. Baseline validation compared the result of the original SQL run in DuckDB with the result returned by run_query on the same input tables.
At its main scale, the evaluation covered 880 model-query-framework-scope trajectories across 22 queries, 10 models, two GPU frameworks and two optimization levels. It used TPC-H at scale factor 10, with 22 analytical queries over eight relational tables, on an H100 GPU. Compiled TorchPlan with torch.compile served as the primary generic baseline, and the reported speedups were measured against it.
Each query-model-framework-scope combination went through execution-guided repair. Candidates had to pass correctness checks and reach at least a 1.05× speedup, with a default maximum of 10 repair rounds. Across the 880 trajectories, 77.5% eventually produced a correct kernel meeting that threshold. Cumulative success reached 40% after round one, 57% after round two and 65% after round three.
Performance varied with the query
The comparison with Sirius varied with the number of tables a query referenced. GPT-5.5 was faster on eight of 13 queries referencing at most three tables, but on only two of nine queries referencing more than three tables. Overall runtime favored GPT-5.5.
The analyzed CUDA-full implementations showed recurring implementation patterns. Among them, 21 of 22 retained cuDF for string or DataFrame operations while using CUDA for the numeric hot path. Fourteen of 22 replaced torch.unique/scatter_add group-by pipelines with fused CUDA aggregation. On Q14, a lookup-and-probe rewrite reached an 11.2× speedup, compared with near 1.07× for alternatives that retained the cuDF merge.
The comparisons changed with scope and prompts
The study also compared full and core optimization. The full-versus-core comparison showed mean speedup gains of 0.31× for CUDA and 0.29× for Triton among the top-five models. Among the bottom-five models, the corresponding gains were 0.02× and 0.05×. The authors interpret this pattern as evidence that stronger models benefit more from full-query specialization.
Prompt context was tested in an ablation. Without the GPU prompt, the CUDA speedup changed by −0.27×; without the data prompt, it changed by −0.40×. For Triton, the corresponding changes were −0.06× and −0.22×. The reported change was larger when workload data was omitted than when GPU details were omitted in both backends, especially for CUDA.
A larger test remained preliminary
A separate partitioned test moved to TPC-H scale factor 100 on four H100 GPUs with 80 GB of memory each, using 10 GB chunks. It produced 22 correct implementations and a 36.44-second workload runtime, or a 2.54× speedup over the corresponding partitioned Dask-cuDF TorchPlan baseline. The authors describe this result as preliminary proof-of-concept evidence.
The primary evidence remains limited to the TPC-H scale factor 10 evaluation on one H100. The scale factor 100 result came from a separate partitioned proof of concept. The benchmark began with validated TorchPlans and asked models to inject CUDA or Triton kernels, so it did not test the complete path from arbitrary SQL to a final optimized kernel.
All 22 evaluated implementations passed both source-audit verdicts and both held-out validations. The authors state that these checks do not constitute a formal proof of semantic equivalence.
The work is an arXiv preprint, version 1, posted on 25 August 2026. The supplied metadata does not report peer-review or journal-publication status.
Paper data and sources
Original title: DataKernelBench: Can LLMs Optimize Database Queries on GPUs?
Authors: Gokul Karthik Kumar, Yotam Perlitz, Corey Lammie et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text