Skip to content

[experiment] Layout Reader#8518

Draft
gatesn wants to merge 51 commits into
developfrom
ngates/layout27
Draft

[experiment] Layout Reader#8518
gatesn wants to merge 51 commits into
developfrom
ngates/layout27

Conversation

@gatesn

@gatesn gatesn commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

What feels like the 27th time I've explored this space, I think I might finally be getting somewhere.

This design pulls out essentially a scan engine. Layouts are actually just one way take serialized arrays and construct a ScanPlan, but in theory we could build a ScanPlan by hand or by any other means.

A ScanPlan node can accept push-down of various operations:

  • try_push_expr - apply an expression over the plan node. This is the closest to our current layout readers that can take an expression, intercept some of it, pass some down to a child, and so on.

This plan can then be used to answer different types of questions:

  1. prepare_read - evaluate a row range of the plan to produce a resulting array.
  2. prepare_evidence - construct zero or more evidence providers that contribute evidence to a predicate. For example, zone maps are an evidence provider that emits definitely_false, but not definitely_true results, and with a much lower cost than a filter evaluation.
  3. prepare_aggregate - return the requested aggregate partial state. This allows zone maps or other layouts to answer aggregate push-down queries.
  4. prepare_stats - return a set of approximate aggregate partials for the given aggregates.

[more description to come]

gatesn and others added 17 commits June 17, 2026 14:38
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Checkpoint of in-progress V2 ScanNode work (segment scheduling driver,
scheduled segment source, scan scheduler) so agent fixes can be integrated
on a clean base. Reviewed/benchmarked state.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
The scan2 StructScanNode single-field fast paths (single get_item and
single-referenced-field expressions) routed straight to the child scan
node, bypassing the parent struct's validity mask. Projecting one field
out of a nullable struct therefore returned the child's own values and
validity with no parent null mask applied, producing wrong nulls (and a
non-nullable result where a nullable one was expected).

Mirror the v1 struct reader's `array.mask(validity)` behaviour: add a
small MaskScanNode that reads an input value and the struct's
non-nullable boolean validity child and produces `mask(input, validity)`.
Wrap the single-field fast-path results in MaskScanNode when the struct
is nullable. The full push_struct path already threads validity through
StructValueScanNode, so it is unchanged.

Add a V1-vs-V2 differential test harness in vortex-file that scans the
same ScanRequest through both paths and asserts equality across flat
(nullable + non-nullable), chunked, dict-encoded, zoned, and nested
nullable-struct fixtures, plus ports of the v1 struct-null regression
tests (test_struct_layout_nulls / test_struct_layout_nested) to the V2
path. Before the fix the five nested-nullable-struct cases failed with
"expected i32?, actual i32"; after the fix all 18 cases pass.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…filter-first

Port of the V1 multi-conjunct filter behavior to the V2 PartitionWorkScheduler
driver: (1) sort filter conjuncts cheapest-first in PreparedScanNodeFile::try_new
so expensive residuals (e.g. FSST LIKE) run after cheap selective ones; (2) when
the demanded-row density falls below EXPR_EVAL_THRESHOLD (0.2), read the residual
predicate with selection=need so the leaf returns the compacted array and the
expression evaluates over only the demanded rows, scattering the verdict back via
Mask::intersect_by_rank. Adds V1-vs-V2 differential cases (low- and high-density
multi-conjunct) and a predicate_cost unit test.

Improves ClickBench multi-conjunct filters (q22 701->547ms, q23 now < V1). A
separate single-LIKE FSST amplification (q21) remains and is tracked separately.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
V2 parallelizes the join probe, aggregate, and Arrow decode ACROSS
DataFusion partitions (V1 instead fans one partition into many split
tasks). When a query projected a heavily-encoded column (e.g. a single
RunEnd chunk for lineitem.l_orderkey), the opener fed split_aligned_row_range
coarse chunk boundaries, which collapsed every byte-range file_group onto
one partition and serialized the probe ~2-wide (TPC-H q4 ran 2.6x slower
than V1).

Feed split_aligned_row_range the scan's own morsel ranges instead: the
read-column chunk hints, or the 100k-row fallback when a read column is a
single chunk (mirroring PreparedScanNodeFile::splits). Each morsel lands
wholly in one partition, so the scan spreads across all of DataFusion's
byte-range file_groups with no collapse and no chunk straddling a partition
boundary. The assignment is contiguous per partition, so it is correct even
when the scan output must preserve order.

Also run the Vortex->Arrow conversion on the runtime CPU pool
(handle.spawn_cpu + buffered/buffer_unordered) so decode fans out within a
partition rather than running serially on the consumer poll thread.

TPC-H SF1 (datafusion-bench, VORTEX_SCAN_IMPL=v2): q4 goes from 2.6x slower
than V1 to faster than V1; overall ~parity.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…H_FULL_PLAN

With --show-metrics and VORTEX_BENCH_FULL_PLAN=1, print the DataFusion
EXPLAIN ANALYZE-style annotated plan (elapsed_compute / output_rows per
operator) to stderr, to localize where wall time goes across scan,
HashJoin build/probe, and aggregate.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Rename the runtime scan node API to ScanPlan and move the plan and segment primitives into vortex-scan. Layout v2 now expands directly through layout.new_scan_plan with a plan ScanRequest, and the docs describe the v2 path as the layout scan model.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the action/benchmark-sql Trigger SQL benchmarks to run on this PR label Jun 20, 2026
@github-actions github-actions Bot removed the action/benchmark-sql Trigger SQL benchmarks to run on this PR label Jun 20, 2026
@codspeed-hq

This comment was marked as off-topic.

gatesn added 6 commits June 19, 2026 23:37
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn added the action/benchmark-sql Trigger SQL benchmarks to run on this PR label Jun 21, 2026
@github-actions github-actions Bot removed the action/benchmark-sql Trigger SQL benchmarks to run on this PR label Jun 21, 2026
@gatesn gatesn added the action/benchmark Trigger full benchmarks to run on this PR label Jun 21, 2026
@github-actions github-actions Bot removed the action/benchmark Trigger full benchmarks to run on this PR label Jun 21, 2026
gatesn added 2 commits June 24, 2026 12:04
Signed-off-by: "Nicholas Gates" <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
@vortex-data vortex-data deleted a comment from github-actions Bot Jun 24, 2026
gatesn added 4 commits June 24, 2026 12:26
Signed-off-by: "Nicholas Gates" <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant