[ESQL] null-fill deferred Parquet extract #159717

Open
costin opened 12:22pm on September 18, 2026 wants to merge 257 Δ into elastic/elasticsearch main from
esql/deferred-parquet-extract-null

Pull Request Overview

  • Opened on September 18, 2026
  • Status Open
  • Commit count 4 with first commit September 18, 2026

Total Delta

257 Total Diff Delta

Open Days

Open 0 weekdays

Test Delta

95 Diff Delta in Test Files
Breakdown by Phase

How long has this pull request spent in each phase of its lifecycle?

Fraction of total time Business days Phase
 
0.0 days Authoring 2 commits before pull request opened for review
 
0.2 days Awaiting first review
 
0.5 days Revising work with 2 commits in response to 1 review that left 8 comments

Total time for pull request still awaiting merge: 0.7 business days

Author avatar

[ESQL] null-fill deferred Parquet extract

SORT|LIMIT late materialization threw when a deferred column was unsupported or missing from this file. Eager scan already constant-null-fills those slots; the throw made TopN fail after the same query without SORT succeeded.

Comments Threads Pending Resolution

Resolved Comment Threads

bpintea reviewed and approved this work on September 18, 2026
x-pack/plugin/esql-datasource-parquet/src/main/java/org/elasticsearch/xpack/esql/datasource/parquet/ParquetFormatReader.java
Outdated
329
     * applying the same primitive-type mapping that the iterator uses (see
330
     * {@link #convertParquetTypeToEsql}). Returns {@code null} when the column is absent or maps
331 331
     * to {@link DataType#UNSUPPORTED} / {@link DataType#NULL}. 
A
 
{@code
 
null}
 
result is not a
332
     * caller error: {@link ParquetColumnExtractor} emits a constant-null block for that column
333
     * rather than throwing. The returned descriptor carries {@code maxRepetitionLevel} so callers
x-pack/plugin/esql-datasource-parquet/src/test/java/org/elasticsearch/xpack/esql/datasource/parquet/ParquetColumnExtractorTests.java
Outdated
450
        byte[] data = writeIntAndUnsupportedListOfStructFile(10);
451
        StorageObject so = createStorageObject(data);
452
        try (ColumnExtractor extractor = newFullFileExtractor(so)) {
453
            long[] positions = { 0, 3, 9 };
454
            Block[] blocks = extractor.extract(
new String[] { "v", "outputs" }, null, positions, blockFactory);