Polars: rs-0.40.0 Release

Release date:
May 21, 2024
Previous version:
rs-0.39.2 (released April 17, 2024)
Magnitude:
27,948 Diff Delta
Contributors:
46 total committers
Data confidence:
Commits:

282 Commits in this Release

Ordered by the degree to which they evolved the repo in this version.

Top Contributors in rs-0.40.0

ritchie46
stinodego
alexander-beedie
MarcoGorelli
itamarst
nameexhaustion
wence-
thalassemia
reswqa
CanglongCl

Directory Browser for rs-0.40.0

All files are compared to previous version, rs-0.39.2. Click here to browse diffs between other versions.

Loading File Browser...

Release Notes Published

πŸ’₯ Breaking changes

  • Remove incremental read based batched CSV reader (#16259)
  • separate rolling_*_by from rolling_*(..., by=...) in Rust (#16102)
  • Move CSV read options from CsvReader to CsvReadOptions (#16126)
  • Rename all 'Chunk's to RecordBatch (#16063)
  • prepare for join coalescing argument (#15418)
  • Rename to CsvParserOptions to CsvReaderOptions, use in CsvReader (#15919)
  • Add context trace to LazyFrame conversion errors (#15761)
  • Move schema resolving of file scan to IR phase (#15739)
  • Move schema resolving to IR phase. (#15714)
  • Rename LogicalPlan and builders to reflect their uses better (#15712)

πŸš€ Performance improvements

  • Use branchless uleb128 decoding for parquet (#16352)
  • Reduce error bubbling in parquet hybrid_rle (#16348)
  • use is_sorted in ewm_mean_by, deprecate check_sorted (#16335)
  • Optimize is_sorted for numeric data (#16333)
  • do not use pyo3-built (#16309)
  • Faster bitpacking for Parquet writer (#16278)
  • Avoid importing ctypes.util in CPU check script if possible (#16307)
  • Don't rechunk when converting DataFrame to numpy/ndarray (#16288)
  • use zeroed vec in ewm_mean_by for sorted fastpath (#16265)
  • use zeroable_vec in ewm_mean_by (#16166)
  • Improve cost of chunk_idx compute (#16154)
  • Don't rechunk by default in concat (#16128)
  • Ensure rechunk is parallel (#16127)
  • Don't traverse deep datasets that we repr as union in CSE (#16096)
  • Ensure better chunk sizes (#16071)
  • Don't rechunk in parallel collection (#15907)
  • Improve non-trivial list aggregations (#15888)
  • Ensure we hit specialized gather for binary/strings (#15886)
  • Limit the cache size for to_datetime (#15826)
  • skip initial null items and don't recompute slope in interpolate (#15819)
  • Fix quadratic in binview growable same source (#15734)

✨ Enhancements

  • Raise when joining on the same keys twice (#16329)
  • Don't require data to be sorted by by column in rolling_*_by operations (#16249)
  • Add struct.field expansion (regex, wildcard, columns) (#16320)
  • Faster bitpacking for Parquet writer (#16278)
  • Add struct.with_fields (#16305)
  • Handle implicit SQL string β†’ temporal conversion in the BETWEEN clause (#16279)
  • Add new index/range based selector cs.by_index, allow multiple indices for nth (#16217)
  • Show warning if expressions are very deep (#16233)
  • Native CSV file list reading (#16180)
  • Register memory mapped files and raise when written to (#16208)
  • Raise when encountering invalid supertype in functions during conversion (#16182)
  • Add SQL support for GROUP BY ALL syntax and fix several issues with aliased group keys (#16179)
  • Allow implicit string β†’ temporal conversion in SQL comparisons (#15958)
  • separate rolling_*_by from rolling_*(..., by=...) in Rust (#16102)
  • Add run-length encoding to Parquet writer (#16125)
  • add date pattern dd.mm.YYYY (#16045)
  • Add RLE to RLE_DICTIONARY encoder (#15959)
  • Support non-coalescing joins in default engine (#16036)
  • Move diagonal & horizontal concat schema resolving to IR phase (#16034)
  • raise more informative error messages in rolling_* aggregations instead of panicking (#15979)
  • Convert concat during IR conversion (#16016)
  • Improve dynamic supertypes (#16009)
  • Additional uint datatype support for the SQL interface (#15993)
  • Support Decimal read from IPC (#15965)
  • Add typed collection from par iterators (#15961)
  • Add by argument for Expr.top_k and Expr.bottom_k (#15468)
  • Add option to disable globbing in csv (#15930)
  • Add option to disable globbing in parquet (#15928)
  • Rename to CsvParserOptions to CsvReaderOptions, use in CsvReader (#15919)
  • Expressify dt.round (#15861)
  • Improve error messages in context stack (#15881)
  • Add dynamic literals to ensure schema correctness (#15832)
  • dt.truncate supports broadcasting lhs (#15768)
  • Expressify str.json_path_match (#15764)
  • Support decimal float parsing in CSV (#15774)
  • Add context trace to LazyFrame conversion errors (#15761)

🐞 Bug fixes

  • correct AExpr.to_field for bitwise and logical and/or (#16360)
  • cargo clippy for uleb128 safety comment (#16368)
  • Infer CSV schema as supertype of all files (#16349)
  • Address overflow combining u64 hashes in Debug builds (#16323)
  • Don't exclude explicitly named columns in group-by context' expr expansion (#16318)
  • Harden Series.reshape against invalid parameters (#16281)
  • Fix list.sum dtype for boolean (#16290)
  • Don't stackoverflow on all/any horizontal (#16287)
  • compilation error when both lazy and ipc features are enabled (#16284)
  • `rolling_*_by was throwing incorrect error when dataframe was sorted by contained multiple chunks (#16247)
  • Clippy Error for CPUID (#16241)
  • Reading CSV with low_memory gave no data (#16231)
  • Empty unique (#16214)
  • Fix empty drop nulls (#16213)
  • Fix get expression group-by state (#16189)
  • Fix rolling empty group OOB (#16186)
  • offset=-0i was being treated differently to offset=0i in rolling (#16184)
  • Fix panic on empty frame joins (#16181)
  • Fix streaming glob slice (#16174)
  • Fix CSV skip_rows_after_header for streaming (#16176)
  • Flush parquet at end of batches tick (#16073)
  • Check CSE name aliases for collisions. (#16149)
  • Don't override CSV reader encoding with lossy UTF-8 (#16151)
  • Add missing allow macros for windows (#16130)
  • Ensure hex and bitstring literals work inside SQL IN clauses (#16101)
  • Revert "Add RLE to RLE_DICTIONARY encoder" (#16113)
  • Respect user passed 'reader_schema' in 'scan_csv' (#16080)
  • Lazy csv + projection; respect null values arg (#16077)
  • Materialize dtypes when converting to arrow (#16074)
  • Fix casting decimal to decimal for high precision (#16049)
  • Fix printing max scale decimals (#16048)
  • Decimal supertype for dyn int (#16046)
  • Do not set sorted flag on lexical sorting (#16032)
  • properly handle nulls in DictionaryArray::iter_typed (#16013)
  • Fix CSE case where upper plan has no projection (#16011)
  • Crash/incorrect group_by/n_unique on categoricals created by (q)cut (#16006)
  • Ternary supertype dynamics (#15995)
  • Treat splitting by empty string as iterating over chars (#15922)
  • Fix PartialEq for DataType::Unknown (#15992)
  • Do not reverse null indices in descending arg_sort (#15974)
  • Finish adding typed_lit to help schema determination in SQL "extract" func (#15955)
  • do not panic when comparing against categorical with incompatible dtype (#15857)
  • Join validation for multiple keys (#15947)
  • Set default limit for String column display to 30 and fix edge cases (#15934)
  • typo in add_half_life takes ln(negative) (#15932)
  • Remove ffspec from parquet reader (#15927)
  • avoid WRITE+EXEC for CPUID check (#15912)
  • fix inconsistent decimal formatting (#15457)
  • Preserve NULLs for is_not_nan (#15889)
  • double projection check should only take the upstream projections into account (#15901)
  • Ensure we don't create invalid frames when combining unit lit + … (#15903)
  • Clear cached rename schema (#15902)
  • Fix OOB in struct lit/agg aggregation (#15891)
  • create (q)cut labels in fixed order (#15843)
  • Tag shrink_dtype as non-streaming (#15828)
  • drop-nulls edge case; remove drop-nulls special case (#15815)
  • ewm_mean_by was skipping initial nulls when it was already sorted by "by" column (#15812)
  • Consult cgroups to determine free memory (#15798)
  • raise if index count like 2i is used when performing rolling, group_by_dynamic, upsample, or other temporal operatios (#15751)
  • Don't deduplicate sort that has slice pushdown (#15784)
  • Fix incorrect is_between pushdown to scan_pyarrow_dataset (#15769)
  • Handle null index correctly for list take (#15737)
  • Preserve lexical ordering on concat (#15753)
  • Remove incorrect unsafe pointer cast for int -> enum (#15740)
  • pass series name to apply for cut/qcut (#15715)
  • count of null column shouldn't panic in agg context (#15710)

πŸ“– Documentation

  • Clarify arrow usage (#16152)
  • Solve inconsistency between code and comment (#16135)
  • add filter docstring examples to date and datetime (#15996)
  • update the link to R API docs (#15973)
  • Fix a typo in categorical section of the user guide (#15777)
  • Fix incorrect column name in LazyFrame.sort doc example (#15658)

πŸ“¦ Build system

  • Update Rust nightly toolchain version (#16222)
  • Don't import jemalloc (#15942)
  • Use default allocator for lts-cpu (#15941)
  • replace all macos-latest referrals with macos-13 (#15926)
  • pin mimalloc and macos-13 (#15925)
  • use jemalloc in lts-cpu (#15913)

πŸ› οΈ Other improvements

  • simplify interpolate code, add test for rolling_*_by with nulls (#16334)
  • Move expression expansion to conversion module (#16331)
  • Add polars-expr README (#16316)
  • Move physical expressions to new crate (#16306)
  • Use cls (not self) in classmethods (#16303)
  • conditionally print the CSEs (#16292)
  • Rename ChunkedArray.chunk_id to chunk_lengths (#16273)
  • Use Scalar instead of Series some aggregations (#16277)
  • Use CsvReadOptions in LazyCsvReader (#16283)
  • Do not hardcode bash path in Makefile (#16263)
  • Add IR::Reduce (not yet implemented) (#16216)
  • Remove incremental read based batched CSV reader (#16259)
  • move all describe, describe_tree and dot-viz code to IR instead of DslPlan (#16237)
  • move describe to IR instead of DSL (#16191)
  • Use Duration.is_zero instead of comparing Duration.duration_ns to 0 (#16195)
  • Remove unused code (#16175)
  • Don't override CSV reader encoding with lossy UTF-8 (#16151)
  • Move CSV read options from CsvReader to CsvReadOptions (#16126)
  • Bump sccache action (#16088)
  • Fix failures in test coverage workflow (#16083)
  • Rename all 'Chunk's to RecordBatch (#16063)
  • Use UnionArgs for DSL side (#16017)
  • Add some comments (#16008)
  • prepare for join coalescing argument (#15418)
  • Pin coverage job to MacOS 13 for now (#15918)
  • Reorganize from_iter and dispatch to collect_ca when possible (#15904)
  • More polars-io cleanup (#15885)
  • Improve type-coercion (#15879)
  • Move type coercion to IR conversion phase (#15868)
  • Reorganize polars_io::parquet module (#15860)
  • Reorganize polars_io::csv module (#15831)
  • Always expand horizontal_any/all (#15816)
  • Rename decimal_float to decimal_comma (#15817)
  • Move IO-related options structs to polars-io (#15806)
  • Split coverage calculation (#15780)
  • Update readme (#15787)
  • Move schema resolving of file scan to IR phase (#15739)
  • Factor out ensure_is_constant_duration (#15733)
  • Move schema resolving to IR phase. (#15714)
  • Rename LogicalPlan and builders to reflect their uses better (#15712)

Thank you to all our contributors for making this release possible! @CanglongCl, @JulianCologne, @KDruzhkin, @MarcoGorelli, @NedJWestern, @NexVeridian, @NickCondron, @Robinsane, @ShivMunagala, @TobiasDummschat, @YichiZhang0613, @alexander-beedie, @avimallu, @bertiewooster, @brandon-b-miller, @c-peters, @coastalwhite, @dangotbanned, @datenzauberai, @deanm0000, @dependabot, @dependabot[bot], @eitsupi, @gasmith, @haocheng6, @ion-elgreco, @itamarst, @janpipek, @jr200, @jrycw, @jsarbach, @luke396, @marenwestermann, @max-muoto, @mbuhidar, @nameexhaustion, @orlp, @pydanny, @r-brink, @reswqa, @ritchie46, @stinodego, @thalassemia, @tharunsuresh-code, @twoertwein, @wence- and @wsyxbcl