Polars: rs-0.42.0 Release

Release date:
August 14, 2024
Previous version:
rs-0.41.3 (released July 2, 2024)
Magnitude:
25,597 Diff Delta
Contributors:
52 total committers
Data confidence:
Commits:

315 Commits in this Release

Ordered by the degree to which they evolved the repo in this version.

Authored July 30, 2024
Authored August 14, 2024
Authored August 1, 2024
Authored August 13, 2024

Top Contributors in rs-0.42.0

coastalwhite
ritchie46
nameexhaustion
orlp
stinodego
alexander-beedie
ruihe774
MarcoGorelli
kylebarron
dependabot-bot

Directory Browser for rs-0.42.0

All files are compared to previous version, rs-0.41.3. Click here to browse diffs between other versions.

Loading File Browser...

Release Notes Published

πŸ’₯ Breaking changes

  • Reject literal input in sort_by_exprs() (#17606)

πŸš€ Performance improvements

  • Skip parquet page when unneeded (#18192)
  • Improve binview extend/ifthenelse (#18164)
  • Start on better Parquet delta decoding (#18049)
  • Tune jemalloc to not create muzzy pages (#18148)
  • Reduce default async thread count (#18142)
  • Use single threaded algorithms if only 1 core given (#18101)
  • Use Arc<Vec<_>> instead of Arc<[_]> for paths and hive partitions (#18066)
  • SIMD View from FixedSizeBinary (#18059)
  • Use bitmask to filter Parquet predicate-pushdown items (#17993)
  • Zerocopy buffers for FixedSizeBinary to BinaryView cast (#18043)
  • Integer fast path Parquet dict encoding (#18030)
  • Speedup writing of Parquet primitive values (#18020)
  • Remove temporary allocations in Parquet (#18013)
  • Delay selection expansion (#18011)
  • Optimize strings slices (#17996)
  • Make .dt.weekday 20x faster (#17992)
  • Shrink MemSliceInner enum (#17991)
  • Push down slice with non-zero offset to Parquet (#17972)
  • Reduce copy in MemSlice (#17983)
  • Ensure metadata flags are maintained on vertical parallelization (#17804)
  • Ensure only nodes that are not changed are cached in collapse optimizer (#17791)
  • Use bitflags for OptState (#17788)
  • Remove async directory auto-detection (#17779)
  • Fix accidental quadratic horizontal concat (#17783)
  • Batch parquet integer decoding (#17734)
  • Use mmap-ed memory if possible in Parquet reader (#17725)
  • Use bitflags for function options (#17723)
  • Introduce MemReader to file buffer in Parquet reader (#17712)
  • Better GC and push_view for binviews (#17627)
  • Fix pathological perf issue in window-order-by (#17650)
  • Cache path resolving of scan functions (#17616)
  • Add ArrayChunks to optimize codegen of BatchDecoder (#17632)
  • Rechunk before we go into grouped gathers (#17623)
  • Cache schema resolve back to DSL (#17610)
  • Add fastpath for when rounding by single constant durations (#17580)
  • Improve parallelism in writing hive parquet (#17512)
  • Support datetime in predicate during hive partition pruning (#17545)
  • Batch nested embed parquet decoding (#17549)
  • Batch nested Parquet decoding (#17542)
  • Collect Parquet dictionary binary as view (#17475)
  • Keep more parallelism when CSE plan cache hits (#17463)
  • Batch parquet primitive decoding (#17462)
  • Respect allow_threading in some more operators (#17450)
  • Parallelize parquet metadata deserialization (#17399)

✨ Enhancements

  • Create literals for datetime/date expressions (#18184)
  • Create literals in 'datetime' expression (#18182)
  • Add missing impl for Series (#18166)
  • Raise on invalid 'is_between' and improve error message quality (#18147)
  • Add boolean Parquet HybridRle encoding (#18022)
  • Add nested SQL join support (#18006)
  • Push down slice with non-zero offset to Parquet (#17972)
  • Add support for binary size method to Expr and Series "bin" namespace (#17924)
  • Add SQL interface support for PostgreSQL dollar-quoted string literals (#17940)
  • Allow for parsing parquet file where the time zone is stored as lowercase "utc" (#17925)
  • Expose binary_elementwise_into_string_amortized for plugin authors, recommend apply_into_string_amortized instead of apply_to_buffer (#17903)
  • Decompress in CSV / NDJSON scan (#17841)
  • Ensure unique names in HConcat (#17884)
  • Support authentication with HuggingFace login (#17881)
  • Support "BY NAME" qualifier for SQL "INTERSECT" and "EXCEPT" set ops (#17835)
  • Raise informative error instead of panicking when passing invalid directives to to_string for Date dtype (#17670)
  • Implement forward/backward fill for all types (#17861)
  • Implement is_in operation on decimal type (#17832)
  • Support hf:// in read_(csv|ipc|ndjson) functions (#17785)
  • Allow literals in sort (#17780)
  • Cloud support for NDJSON (#17717)
  • Support API token for scanning hf:// (#17682)
  • Raise error instead of panic in unsupported serde (#17679)
  • Include file path option for NDJSON (#17681)
  • Hugging Face path expansion (#17665)
  • Add DSL validation for cloud eligible check (#17287)
  • Raise informative error message if non-IntoExpr is passed by name in *Frame.group_by (#17654)
  • Change API for writing partitioned Parquet to reduce code duplication (#17586)
  • Cache schema resolve back to DSL (#17610)
  • Expose returns_scalar to map_elements (#17613)
  • Add option to include file path for Parquet, IPC, CSV scans (#17563)
  • Support describe on decimal (#15092)
  • Support datetime in predicate during hive partition pruning (#17545)
  • Raise more informative error message for directories containing files with mixed extensions (#17480)
  • Exclude empty files from directory/glob expansion (#17478)
  • Add "future" versioning (#17421)
  • Apply slice pushdown immediately to in-memory frames (#17459)
  • Support writing hive partitioned parquet (#17324)
  • Add right join support (#17441)
  • Support hive partitioning in scan_ipc (#17434)

🐞 Bug fixes

  • Fix struct shift and list builder (#18189)
  • Don't load Parquet nested metadata (#18183)
  • Throw bigidx error for Parquet row-count (#18154)
  • Fix unpivot on empty df (#18179)
  • Don't vertically parallelize cse contexts (#18177)
  • Properly handle empty Parquet row groups with no dictionary (#18161)
  • Struct outer nullabillity (#18156)
  • Fix pyarrow predicate pushdown regression (#18145)
  • Prevent unwanted supertype cast in 'search_sorted' (#18143)
  • Parquet with filter=None (#18139)
  • Don't raise when converting from pandas if index contains duplicate names when include_index=False (the default) (#18133)
  • Don't remove leading whitespace in read_csv (#18131)
  • Py-polars compilation with no features (#18129)
  • String transform to_titlecase was too narrowly defined (#18122)
  • Reading Parquet with Null dictionary page (#18112)
  • Incorrect lazy CSV select(len()) for compressed files (#18067)
  • Fix sink_ipc_cloud panicking with runtime error (#18091)
  • Properly write Parquet for sliced lists (#18073)
  • Panic reading multiple CSV files from cloud (#18056)
  • Fix CloudWriter to use buffer before making requests (#18027)
  • Fix typos and remove trailing whitespace (#18024)
  • Handle cfg(feature) for shrink_dtype (#18038)
  • Subtraction with overflow on negative slice offset in Parquet (#18036)
  • Add nested SQL join support (#18006)
  • Allow read_csv schema to take unparsable types (#17765)
  • Multi-output column expressions in frame sort method (#17947)
  • Fix Asof join by schema (#17988)
  • Fix glob resolution for Hugging Face (#17958)
  • Several parquet reader/writer regressions (#17941)
  • Incorrect filter on categorical columns from parquet files (#17950)
  • SQL COUNT(DISTINCT x) should not include NULL values (#17930)
  • Scanning '%' from cloud (#17890)
  • Respect glob=False for cloud reads (#17860)
  • Properly write nest-nulled values in Parquet (#17845)
  • Allow full-null Object series to be built (#17870)
  • Fix from_arrow for struct type (#17839)
  • Infer decimal scales on mixed scale input (#17840)
  • Raise on unsupported fill strategy dtype (#17837)
  • Properly write nested NullArray in Parquet (#17807)
  • Check input type on list.to_struct (#17834)
  • Fix right join schema (#17833)
  • Non-compliant Parquet list element name (#17803)
  • Correctly set should_broadcast flag in HStack CSE rewrite (#17784)
  • Fix projection pusdhown of literals without names (#17778)
  • Don't expand HTTP paths (#17774)
  • Check funtion input len at expansion (#17763)
  • Don't panic in invalid agg_groups (#17762)
  • Raise empty struct (#17736)
  • Fix GC logic in write_ipc (#17752)
  • Panic in pl.concat_list and list.concat on empty inputs (#17742)
  • Fix out nullability for structs coming from arrow (#17738)
  • Percent encode for Hugging Face paths (#17718)
  • Use bytemuck in slice reinterpret for Parquet ArrayChunks (#17700)
  • Propagate struct outer nullability eagerly (#17697)
  • Use ETag for HTTP file cache invalidation (#17684)
  • Fix type inference failure caused by double transpose (#17663)
  • Interpret %y consistently with Chrono in to_date/to_datetime/strptime (#17661)
  • Fix explode invalid check (#17651)
  • Tighten up error checking on join keys (#17517)
  • Expand brackets in async glob expansion (#17630)
  • Fix row index disappearing after projection pushdown in NDJSON (#17631)
  • Fix struct -> enum is_in (#17622)
  • Don't needlessly unwrap in pivot_schema (#17611)
  • Reject literal input in sort_by_exprs() (#17606)
  • Bitmap collect into safety (#17588)
  • Method dt.truncate was sometimes returning incorrect results for pre-1970 datetimes (#17582)
  • Defer path expansion until collect in file scan methods (#17532)
  • Correct logic for descending sort of BooleanChunked (#17558)
  • Don't unwrap send attempt to oneshot channel (#17566)
  • Fix scanning from HTTP cloud paths (#17571)
  • Properly implement struct (#17522)
  • Add missing commas in python IR interchange (#17518)
  • Fix predicate pushdown for .list.(get|gather) (#17511)
  • Turn panic into error when serializing Object types (#17353)
  • Fix struct expansion and raise on exclude (#17489)
  • Fix decimal dyn float supertype (#17464)
  • Don't rechunk on phys_repr (#17461)
  • Harden alchemy session for old sqlalchemy versions (#17366)
  • Fix swapping rename schema (#17458)
  • Raise on oob decimal precision (#17445)
  • Don't allow json inference method to be chunked/streaming (#17396)
  • avoid panic when projecting solitary count into empty frame (#17393)
  • Set literal nesting to 0 (#17392)
  • Fix scanning cloud paths with spaces (#17379)
  • Fix slice length no longer allowing None (#17372)
  • Cull row index in scan if projection pushdown removes it (#17363)
  • Fix typo in SchemaError exception message (#17350)

πŸ“– Documentation

  • Mention 'Array' in data types overview (#18060)
  • Correct concat rechunk in user guide (#18080)
  • Fix typo in title of Hugging Face docs page (#18097)
  • Update pivot docstring for clarity (#18000)
  • Fix aggregation guide discrepancies (#18003)
  • Updating the join example input for rust for consistency with python example (#17898)
  • Rename str.lengths to str.len_bytes in description text (#11577) (#17626)
  • Create example for polars.Expr.bin.decode (#17508)
  • Add right join in the user guide (#17608)
  • Update SQL examples in README (#17568)
  • Fix typo in Getting Started section of user guide (#17465)

πŸ“¦ Build system

  • Update Rust toolchain to nightly-2024-07-26 (#17891)

πŸ› οΈ Other improvements

  • Consistently use PlRandomState, PlHashMap/Set (#18190)
  • Remove unneeded growable (#18165)
  • Allow non-scoped tasks to be spawned (#18163)
  • Update Cargo.lock to fix build error on Linux (#18153)
  • Remove Nth,Wildcard from ExprIR and make conversion falllible (#18115)
  • Allow zip node to extend with nulls or broadcast inputs (#18077)
  • Add @coastalwhite to parquet codeowners (#18032)
  • Minor bump to comfy-table version (#18028)
  • Add unit tests for str.contains_any and str.replace_many (#17961)
  • Remove apply_generic, use unary_elementwise (#17902)
  • Add general filters in Parquet (#17910)
  • Fix broken typos command in make pre-commit for py-polars folder (#17897)
  • Remove HybridRLE iter / batch nested parquet decoding (#17889)
  • Deal with re-entrant expressions locally (#17885)
  • Unify Parquet nested decoders (#17816)
  • Add zip node to streaming engine (#17866)
  • Fix simple projection in streaming engine (#17871)
  • Make better use of typos configuration features (#17800)
  • Remove complicated Parquet traits (#17767)
  • Rename Unit to Plain in Parquet reader (#17751)
  • Add missing feature flag for prefetching on arm (#17740)
  • Add the capability to run new streaming engine on test suite (#17706)
  • Update CODEOWNERS (#17707)
  • Allow compute nodes in streaming engine to decide whether they want serial or parallel input/output (#17699)
  • Name tests so they actually run (#17690)
  • Add reduce ComputeNode in new streaming engine (#17389)
  • Add DSL validation for cloud eligible check (#17287)
  • Rename Pipe to Connector (#17655)
  • Document and simplify MutableBinView::push_ignore_validity (#17645)
  • Remove has_validity, use has_nulls (#17519)
  • Add ArrayChunks to optimize codegen of BatchDecoder (#17632)
  • Move path logic to from utils to path_utils in polars-io (#17635)
  • Fix struct gather (#17621)
  • Back to StructChunked name (#17609)
  • Re-enable struct related tests (#17597)
  • Completely redo structure of Parquet decoder (#17589)
  • Fix struct outer validity;fmt;is_in;cast;cmp (#17590)
  • Make no-default-features compile (#17516)
  • No longer blanket allow unused items in polars-stream (#17551)
  • Smaller filter / valid in Parquet (#17569)
  • Move path expansion utils to polars-io crate (#17521)
  • Add streaming slice node to new streaming engine (#17451)
  • Add ordered union to new streaming engine (#17359)
  • Add automated check for PR title formatting (#17412)
  • Remove transmute for object store path (#17395)
  • Fix Python version resolver in release drafter (#17390)
  • Remove dead code (#17378)

Thank you to all our contributors for making this release possible! @5j9, @ByteNybbler, @EricTulowetzke, @JamesCE2001, @Julian-J-S, @KDruzhkin, @MarcoGorelli, @Object905, @SandroCasagrande, @Vincenthays, @alexander-beedie, @anergictcell, @arnabanimesh, @atigbadr, @brandon-b-miller, @brunobbaraujo, @cmdlineluser, @coastalwhite, @davanstrien, @deanm0000, @deepyaman, @delsner, @dependabot, @dependabot[bot], @diegoglozano, @eitsupi, @ember91, @flisky, @henryharbeck, @implicit-apparatus, @itamarst, @jonaylor89, @jparag, @knl, @kylebarron, @lukapeschke, @mcrumiller, @moritzwilksch, @nameexhaustion, @orlp, @phi-friday, @philss, @r-brink, @ragyabraham, @rcorty, @ritchie46, @rosstitmarsh, @ruihe774, @sherlockbeard, @stinodego, @szepeviktor, @tylerriccio33, @wangxiaoying and @wence-