π₯ Breaking changes
- Reject literal input in
sort_by_exprs()
(#17606)
π Performance improvements
- Skip parquet page when unneeded (#18192)
- Improve binview extend/ifthenelse (#18164)
- Start on better Parquet delta decoding (#18049)
- Tune jemalloc to not create muzzy pages (#18148)
- Reduce default async thread count (#18142)
- Use single threaded algorithms if only 1 core given (#18101)
- Use
Arc<Vec<_>>
instead of Arc<[_]>
for paths and hive partitions (#18066)
- SIMD View from
FixedSizeBinary
(#18059)
- Use bitmask to filter Parquet predicate-pushdown items (#17993)
- Zerocopy buffers for
FixedSizeBinary
to BinaryView
cast (#18043)
- Integer fast path Parquet dict encoding (#18030)
- Speedup writing of Parquet primitive values (#18020)
- Remove temporary allocations in Parquet (#18013)
- Delay selection expansion (#18011)
- Optimize strings slices (#17996)
- Make
.dt.weekday
20x faster (#17992)
- Shrink
MemSliceInner
enum (#17991)
- Push down slice with non-zero offset to Parquet (#17972)
- Reduce copy in
MemSlice
(#17983)
- Ensure metadata flags are maintained on vertical parallelization (#17804)
- Ensure only nodes that are not changed are cached in collapse optimizer (#17791)
- Use bitflags for OptState (#17788)
- Remove async directory auto-detection (#17779)
- Fix accidental quadratic horizontal concat (#17783)
- Batch parquet integer decoding (#17734)
- Use mmap-ed memory if possible in Parquet reader (#17725)
- Use bitflags for function options (#17723)
- Introduce
MemReader
to file buffer in Parquet reader (#17712)
- Better GC and push_view for binviews (#17627)
- Fix pathological perf issue in window-order-by (#17650)
- Cache path resolving of
scan
functions (#17616)
- Add
ArrayChunks
to optimize codegen of BatchDecoder (#17632)
- Rechunk before we go into grouped gathers (#17623)
- Cache schema resolve back to DSL (#17610)
- Add fastpath for when rounding by single constant durations (#17580)
- Improve parallelism in writing hive parquet (#17512)
- Support datetime in predicate during hive partition pruning (#17545)
- Batch nested embed parquet decoding (#17549)
- Batch nested Parquet decoding (#17542)
- Collect Parquet dictionary binary as view (#17475)
- Keep more parallelism when CSE plan cache hits (#17463)
- Batch parquet primitive decoding (#17462)
- Respect allow_threading in some more operators (#17450)
- Parallelize parquet metadata deserialization (#17399)
β¨ Enhancements
- Create literals for datetime/date expressions (#18184)
- Create literals in 'datetime' expression (#18182)
- Add missing impl for
Series
(#18166)
- Raise on invalid 'is_between' and improve error message quality (#18147)
- Add boolean Parquet HybridRle encoding (#18022)
- Add nested SQL join support (#18006)
- Push down slice with non-zero offset to Parquet (#17972)
- Add support for binary
size
method to Expr and Series "bin" namespace (#17924)
- Add
SQL
interface support for PostgreSQL dollar-quoted string literals (#17940)
- Allow for parsing parquet file where the time zone is stored as lowercase "utc" (#17925)
- Expose binary_elementwise_into_string_amortized for plugin authors, recommend
apply_into_string_amortized
instead of apply_to_buffer
(#17903)
- Decompress in CSV / NDJSON scan (#17841)
- Ensure unique names in HConcat (#17884)
- Support authentication with HuggingFace login (#17881)
- Support "BY NAME" qualifier for
SQL
"INTERSECT" and "EXCEPT" set ops (#17835)
- Raise informative error instead of panicking when passing invalid directives to
to_string
for Date dtype (#17670)
- Implement forward/backward fill for all types (#17861)
- Implement
is_in
operation on decimal type (#17832)
- Support
hf://
in read_(csv|ipc|ndjson)
functions (#17785)
- Allow literals in sort (#17780)
- Cloud support for NDJSON (#17717)
- Support API token for scanning
hf://
(#17682)
- Raise error instead of panic in unsupported serde (#17679)
- Include file path option for NDJSON (#17681)
- Hugging Face path expansion (#17665)
- Add DSL validation for cloud eligible check (#17287)
- Raise informative error message if non-IntoExpr is passed by name in *Frame.group_by (#17654)
- Change API for writing partitioned Parquet to reduce code duplication (#17586)
- Cache schema resolve back to DSL (#17610)
- Expose
returns_scalar
to map_elements (#17613)
- Add option to include file path for Parquet, IPC, CSV scans (#17563)
- Support
describe
on decimal (#15092)
- Support datetime in predicate during hive partition pruning (#17545)
- Raise more informative error message for directories containing files with mixed extensions (#17480)
- Exclude empty files from directory/glob expansion (#17478)
- Add "future" versioning (#17421)
- Apply slice pushdown immediately to in-memory frames (#17459)
- Support writing hive partitioned parquet (#17324)
- Add right join support (#17441)
- Support hive partitioning in
scan_ipc
(#17434)
π Bug fixes
- Fix struct shift and list builder (#18189)
- Don't load Parquet nested metadata (#18183)
- Throw bigidx error for Parquet row-count (#18154)
- Fix unpivot on empty df (#18179)
- Don't vertically parallelize cse contexts (#18177)
- Properly handle empty Parquet row groups with no dictionary (#18161)
- Struct outer nullabillity (#18156)
- Fix pyarrow predicate pushdown regression (#18145)
- Prevent unwanted supertype cast in 'search_sorted' (#18143)
- Parquet with
filter=None
(#18139)
- Don't raise when converting from pandas if index contains duplicate names when
include_index=False
(the default) (#18133)
- Don't remove leading whitespace in
read_csv
(#18131)
- Py-polars compilation with no features (#18129)
- String transform
to_titlecase
was too narrowly defined (#18122)
- Reading Parquet with Null dictionary page (#18112)
- Incorrect lazy CSV
select(len())
for compressed files (#18067)
- Fix
sink_ipc_cloud
panicking with runtime error (#18091)
- Properly write Parquet for sliced lists (#18073)
- Panic reading multiple CSV files from cloud (#18056)
- Fix
CloudWriter
to use buffer before making requests (#18027)
- Fix typos and remove trailing whitespace (#18024)
- Handle
cfg(feature)
for shrink_dtype
(#18038)
- Subtraction with overflow on negative slice offset in Parquet (#18036)
- Add nested SQL join support (#18006)
- Allow
read_csv
schema
to take unparsable types (#17765)
- Multi-output column expressions in frame
sort
method (#17947)
- Fix Asof join by schema (#17988)
- Fix glob resolution for Hugging Face (#17958)
- Several parquet reader/writer regressions (#17941)
- Incorrect filter on categorical columns from parquet files (#17950)
- SQL
COUNT(DISTINCT x)
should not include NULL values (#17930)
- Scanning '%' from cloud (#17890)
- Respect
glob=False
for cloud reads (#17860)
- Properly write nest-nulled values in Parquet (#17845)
- Allow full-null Object series to be built (#17870)
- Fix
from_arrow
for struct type (#17839)
- Infer decimal scales on mixed scale input (#17840)
- Raise on unsupported fill strategy dtype (#17837)
- Properly write nested
NullArray
in Parquet (#17807)
- Check input type on list.to_struct (#17834)
- Fix right join schema (#17833)
- Non-compliant Parquet list element name (#17803)
- Correctly set should_broadcast flag in HStack CSE rewrite (#17784)
- Fix projection pusdhown of literals without names (#17778)
- Don't expand HTTP paths (#17774)
- Check funtion input len at expansion (#17763)
- Don't panic in invalid agg_groups (#17762)
- Raise empty struct (#17736)
- Fix GC logic in
write_ipc
(#17752)
- Panic in pl.concat_list and list.concat on empty inputs (#17742)
- Fix out nullability for structs coming from arrow (#17738)
- Percent encode for Hugging Face paths (#17718)
- Use bytemuck in slice reinterpret for Parquet ArrayChunks (#17700)
- Propagate struct outer nullability eagerly (#17697)
- Use ETag for HTTP file cache invalidation (#17684)
- Fix type inference failure caused by double transpose (#17663)
- Interpret %y consistently with Chrono in to_date/to_datetime/strptime (#17661)
- Fix explode invalid check (#17651)
- Tighten up error checking on join keys (#17517)
- Expand brackets in async glob expansion (#17630)
- Fix row index disappearing after projection pushdown in NDJSON (#17631)
- Fix struct -> enum is_in (#17622)
- Don't needlessly unwrap in
pivot_schema
(#17611)
- Reject literal input in
sort_by_exprs()
(#17606)
- Bitmap collect into safety (#17588)
- Method dt.truncate was sometimes returning incorrect results for pre-1970 datetimes (#17582)
- Defer path expansion until
collect
in file scan methods (#17532)
- Correct logic for descending sort of BooleanChunked (#17558)
- Don't unwrap send attempt to oneshot channel (#17566)
- Fix scanning from HTTP cloud paths (#17571)
- Properly implement struct (#17522)
- Add missing commas in python IR interchange (#17518)
- Fix predicate pushdown for
.list.(get|gather)
(#17511)
- Turn panic into error when serializing Object types (#17353)
- Fix struct expansion and raise on exclude (#17489)
- Fix decimal dyn float supertype (#17464)
- Don't rechunk on phys_repr (#17461)
- Harden alchemy session for old sqlalchemy versions (#17366)
- Fix swapping rename schema (#17458)
- Raise on oob decimal precision (#17445)
- Don't allow json inference method to be chunked/streaming (#17396)
- avoid panic when projecting solitary count into empty frame (#17393)
- Set literal nesting to 0 (#17392)
- Fix scanning cloud paths with spaces (#17379)
- Fix
slice
length no longer allowing None
(#17372)
- Cull row index in scan if projection pushdown removes it (#17363)
- Fix typo in
SchemaError
exception message (#17350)
π Documentation
- Mention 'Array' in data types overview (#18060)
- Correct concat rechunk in user guide (#18080)
- Fix typo in title of Hugging Face docs page (#18097)
- Update pivot docstring for clarity (#18000)
- Fix aggregation guide discrepancies (#18003)
- Updating the join example input for rust for consistency with python example (#17898)
- Rename
str.lengths
to str.len_bytes
in description text (#11577) (#17626)
- Create example for
polars.Expr.bin.decode
(#17508)
- Add right join in the user guide (#17608)
- Update SQL examples in README (#17568)
- Fix typo in Getting Started section of user guide (#17465)
π¦ Build system
- Update Rust toolchain to
nightly-2024-07-26
(#17891)
π οΈ Other improvements
- Consistently use PlRandomState, PlHashMap/Set (#18190)
- Remove unneeded growable (#18165)
- Allow non-scoped tasks to be spawned (#18163)
- Update Cargo.lock to fix build error on Linux (#18153)
- Remove Nth,Wildcard from ExprIR and make conversion falllible (#18115)
- Allow zip node to extend with nulls or broadcast inputs (#18077)
- Add @coastalwhite to parquet codeowners (#18032)
- Minor bump to
comfy-table
version (#18028)
- Add unit tests for
str.contains_any
and str.replace_many
(#17961)
- Remove apply_generic, use unary_elementwise (#17902)
- Add general filters in Parquet (#17910)
- Fix broken
typos
command in make pre-commit
for py-polars folder (#17897)
- Remove HybridRLE iter / batch nested parquet decoding (#17889)
- Deal with re-entrant expressions locally (#17885)
- Unify Parquet nested decoders (#17816)
- Add zip node to streaming engine (#17866)
- Fix simple projection in streaming engine (#17871)
- Make better use of
typos
configuration features (#17800)
- Remove complicated Parquet traits (#17767)
- Rename Unit to Plain in Parquet reader (#17751)
- Add missing feature flag for prefetching on arm (#17740)
- Add the capability to run new streaming engine on test suite (#17706)
- Update CODEOWNERS (#17707)
- Allow compute nodes in streaming engine to decide whether they want serial or parallel input/output (#17699)
- Name tests so they actually run (#17690)
- Add reduce
ComputeNode
in new streaming engine (#17389)
- Add DSL validation for cloud eligible check (#17287)
- Rename Pipe to Connector (#17655)
- Document and simplify MutableBinView::push_ignore_validity (#17645)
- Remove has_validity, use has_nulls (#17519)
- Add
ArrayChunks
to optimize codegen of BatchDecoder (#17632)
- Move path logic to from
utils
to path_utils
in polars-io (#17635)
- Fix struct gather (#17621)
- Back to StructChunked name (#17609)
- Re-enable struct related tests (#17597)
- Completely redo structure of Parquet decoder (#17589)
- Fix struct outer validity;fmt;is_in;cast;cmp (#17590)
- Make no-default-features compile (#17516)
- No longer blanket allow unused items in polars-stream (#17551)
- Smaller filter / valid in Parquet (#17569)
- Move path expansion utils to
polars-io
crate (#17521)
- Add streaming slice node to new streaming engine (#17451)
- Add ordered union to new streaming engine (#17359)
- Add automated check for PR title formatting (#17412)
- Remove transmute for object store path (#17395)
- Fix Python version resolver in release drafter (#17390)
- Remove dead code (#17378)
Thank you to all our contributors for making this release possible!
@5j9, @ByteNybbler, @EricTulowetzke, @JamesCE2001, @Julian-J-S, @KDruzhkin, @MarcoGorelli, @Object905, @SandroCasagrande, @Vincenthays, @alexander-beedie, @anergictcell, @arnabanimesh, @atigbadr, @brandon-b-miller, @brunobbaraujo, @cmdlineluser, @coastalwhite, @davanstrien, @deanm0000, @deepyaman, @delsner, @dependabot, @dependabot[bot], @diegoglozano, @eitsupi, @ember91, @flisky, @henryharbeck, @implicit-apparatus, @itamarst, @jonaylor89, @jparag, @knl, @kylebarron, @lukapeschke, @mcrumiller, @moritzwilksch, @nameexhaustion, @orlp, @phi-friday, @philss, @r-brink, @ragyabraham, @rcorty, @ritchie46, @rosstitmarsh, @ruihe774, @sherlockbeard, @stinodego, @szepeviktor, @tylerriccio33, @wangxiaoying and @wence-