[SYSTEMDS-2411] - Performance codegen kmeans mnist80m w/ compression [SYSTEMDS-3088] - Prefetch instruction [SYSTEMDS-3098] - Broadcast instruction [SYSTEMDS-3256] - Create apply functions for cleaning primitives [SYSTEMDS-3286] - LogicalEnumerator change with transitions concept and cleanups [SYSTEMDS-3347] - Flatten the nested loop for parallel pipelines execution [SYSTEMDS-3376] - Adding apply_pipeline() builtin for cleaning pipelines API [SYSTEMDS-3401] - Release docker images with GitHub actions [SYSTEMDS-3422] - Add monitoring tool testing workflows [SYSTEMDS-3443] - Asynchronous Execution and Persist Spark Transformations [SYSTEMDS-3466] - Future-based asynchronous execution of Spark actions [SYSTEMDS-3469] - New operator linearization order to maximize inter-operator parallelism [SYSTEMDS-3470] - Lineage-based reuse of Spark actions [SYSTEMDS-3473] - Push down rmvar instructions for asynchronous instructions [SYSTEMDS-3474] - Lineage-based reuse of asynchronous operators [SYSTEMDS-3479] - Persist and reuse Spark RDDs [SYSTEMDS-3497] - Refactor to add LOP rewrite step in compilation

Bug

[SYSTEMDS-1026] - Fix memory configuration in sparkDML.sh [SYSTEMDS-1281] - OOM Error On Binary Write [SYSTEMDS-1283] - Out of memory error [SYSTEMDS-2948] - CLA Improved Run estimation [SYSTEMDS-3045] - AttributeError: Function definition not found [SYSTEMDS-3272] - applySchema built-in to set the schema of frame from DML [SYSTEMDS-3339] - CSR TSMM left with filled rows bug [SYSTEMDS-3353] - Sparse TSMM dense row blocks CSR [SYSTEMDS-3354] - py4j.Py4JException: Method exceptionString([class org.apache.spark.SparkConf]) does not exist [SYSTEMDS-3355] - MatrixBlock size using CSR when allowed [SYSTEMDS-3379] - Federated Nan Values [SYSTEMDS-3390] - countDistinctApprox() operation in AggregateUnaryCPInstruction is inefficient for row/col aggregations [SYSTEMDS-3391] - Correct the release artifact generation date [SYSTEMDS-3394] - Log4j incompatible dependencies [SYSTEMDS-3396] - ConcurrentModificationException in federated execution [SYSTEMDS-3398] - Jackson Core missing for json writing and reading in reduced binary [SYSTEMDS-3400] - Fix Java doc warnings [SYSTEMDS-3408] - Enque output not UTF-8 python [SYSTEMDS-3409] - Read CSV directly without mtd python [SYSTEMDS-3411] - Python configuration not loading defaults [SYSTEMDS-3412] - Matrix Multiplication crash in Spark [SYSTEMDS-3414] - Pipelines failing in Hybrid execution [SYSTEMDS-3415] - Built-in tests failure in Git actions [SYSTEMDS-3416] - Cleaning Pipelines failed with No space left on device [SYSTEMDS-3417] - IndexOutOfBounds due to int overflow on replace [SYSTEMDS-3418] - Cleaning Pipelines: Replace function failure in hybrid execution [SYSTEMDS-3419] - Cleaning Pipelines: Block Sizes mismatch [SYSTEMDS-3420] - Cleaning Pipelines in hybrid mode: Invalid block dimensions error [SYSTEMDS-3424] - Federated Statistics print in non federated scenario [SYSTEMDS-3425] - Spark Aggregate Binary operations parse to Fed instruction [SYSTEMDS-3432] - FederationUtils.bindResponses causes out of memory because of sparse matrices. [SYSTEMDS-3433] - Python IDE test Docs fail [SYSTEMDS-3435] - MSVM robustness for non-existing classes [SYSTEMDS-3436] - CLA ArrayOutOfBounds in sample [SYSTEMDS-3437] - CLA Invalid Unique estimate DDC [SYSTEMDS-3439] - Federated read cache cannot be disabled [SYSTEMDS-3442] - Monitoring Heavy hitters not always correct list [SYSTEMDS-3451] - Slow Federated Mlogreg on Criteo (dummy-coded) [SYSTEMDS-3452] - Incorrect warning when reading scalars [SYSTEMDS-3476] - Spark with default settings [SYSTEMDS-3477] - Cleaning Pipelines: Task Parallel Experiments failing in spark mode [SYSTEMDS-3498] - Unique() crashes with iterator EOF on vectors with >1K distinct items [SYSTEMDS-3500] - Perftest: Mlogreg on 1M_1k_dense w/ unnecessary spark jobs [SYSTEMDS-3501] - Perftest: lmDS on 1M_1k_dense with unnecessary spark tsmm [SYSTEMDS-3503] - Java doc warnings

Epic

[SYSTEMDS-450] - Extended spark interfaces [SYSTEMDS-3445] - Combining compression schemes together [SYSTEMDS-3459] - Reorganization and cleanup of the internal representation of FrameBlocks

New Feature

[SYSTEMDS-2551] - Federated Compression Instruction [SYSTEMDS-2699] - CLA IO Compressed Matrices [SYSTEMDS-2754] - Compressed Max/Min Index support. [SYSTEMDS-2830] - Functional Compression [SYSTEMDS-3280] - Homomorphic Encryption for Federated Parameter Servers [SYSTEMDS-3303] - NN Builtin: Attention Layer [SYSTEMDS-3325] - Multi-threaded tokenization [SYSTEMDS-3337] - CLA TSMM direct multiplication [SYSTEMDS-3360] - Federated async compression [SYSTEMDS-3361] - Federated Workload-aware Compression [SYSTEMDS-3369] - Timout setting for all federated tests [SYSTEMDS-3374] - Federated primitive for transferring a local data object to a federated representation [SYSTEMDS-3404] - Synchronous with backup workers mode for Parameter Servers [SYSTEMDS-3405] - Federated Write at site [SYSTEMDS-3438] - CLA RowSlice compressed return [SYSTEMDS-3478] - Bitset array for Frames [SYSTEMDS-3481] - Frame from MatrixBlock improvement [SYSTEMDS-3493] - Python windows Install [SYSTEMDS-3494] - Python 3.9 support [SYSTEMDS-3495] - Parallel Compressed Encode

Story

[SYSTEMDS-2783] - Lineage-based reuse in federated execution [SYSTEMDS-3087] - Memory management and lazy evaluation in dynamic environments [SYSTEMDS-3463] - Add unique() built-in function

Improvement

[SYSTEMDS-295] - Unexpected order when print !boolean [SYSTEMDS-1169] - Clean Up and Automate Python Tests [SYSTEMDS-1406] - Fix whitespace issues in main algorithms [SYSTEMDS-1532] - Introduce Python scripts to launch SystemML from the Command Line [SYSTEMDS-2513] - Improve the Development and user experience on Windows [SYSTEMDS-2897] - CLA decompressing write [SYSTEMDS-3185] - Federated Multi Tenant Backend [SYSTEMDS-3187] - Add documentation for the release scripts [SYSTEMDS-3192] - Test large dense block compression [SYSTEMDS-3254] - CountDistinct Col and Row & Unique [SYSTEMDS-3282] - Upper bound for number of decoders [SYSTEMDS-3283] - Multi-threaded ctable instruction [SYSTEMDS-3319] - CLA Generalize Bin Packing [SYSTEMDS-3323] - CLA move combine of empty to estim [SYSTEMDS-3328] - Federated transform for equi-height [SYSTEMDS-3359] - Sample-based Recode Map Size Estimation [SYSTEMDS-3386] - Refactor runtime replacement of CP or SP instructions with FED instructions [SYSTEMDS-3393] - Use Java (JDK17) SIMD Implementation [SYSTEMDS-3413] - Add row/col aggregation support to countDistinct() builtin function [SYSTEMDS-3429] - Use Local Level of Parallelism when Transformencoding in Federated Mode [SYSTEMDS-3440] - Federated Requests Coordinator Hostname [SYSTEMDS-3444] - Spark Write CLA [SYSTEMDS-3446] - DDC Append [SYSTEMDS-3447] - SDC Append [SYSTEMDS-3448] - Uncompressed Append [SYSTEMDS-3449] - Const/Empty append [SYSTEMDS-3450] - DDCFOR Append [SYSTEMDS-3453] - Offsets Append [SYSTEMDS-3454] - CLA Sheme primitive [SYSTEMDS-3455] - Improved multi-threaded unary operations [SYSTEMDS-3456] - MatrixBlock equals [SYSTEMDS-3457] - MatrixBlock equals Sparse Specialization [SYSTEMDS-3458] - Add support for Spark backend to countDistinct() builtin function [SYSTEMDS-3460] - Move FrameBlock out of MatrixBlock path. [SYSTEMDS-3461] - FrameBlock Arrays separation [SYSTEMDS-3462] - FrameBlock Iterators Factory Pattern [SYSTEMDS-3464] - Python Combine Write [SYSTEMDS-3465] - Typed return on CacheBlock Interface Slice [SYSTEMDS-3467] - Add support for MULTI_BLOCK Spark backend support for countDistinct() [SYSTEMDS-3471] - Enable multi-threaded transformencode/apply [SYSTEMDS-3472] - Spark Append Frame Bug [SYSTEMDS-3475] - Spark update version 3.3.1 [SYSTEMDS-3480] - Verify release scripts with github workflows [SYSTEMDS-3482] - Parallel Hadoop IO startup [SYSTEMDS-3484] - FrameAppend optimization [SYSTEMDS-3485] - Precompile detect type patterns [SYSTEMDS-3486] - Character Array Type [SYSTEMDS-3487] - Array primitives with null [SYSTEMDS-3488] - Compressed Frame Write [SYSTEMDS-3489] - CLA Compress NaN [SYSTEMDS-3490] - Compressed Transform Encode [SYSTEMDS-3491] - CLA Specialized Column Indexes

Test

[SYSTEMDS-3395] - ColGroup Equivalence Tests [SYSTEMDS-3397] - Python NN testExample

Wish

[SYSTEMDS-3171] - GIO - Mapping from binary data

Task

[SYSTEMDS-209] - Algorithm wrappers (ml pipelines, ml context) [SYSTEMDS-563] - MR operations over frames [SYSTEMDS-3148] - Federated Performance Tests [SYSTEMDS-3228] - Builtin for k nearest neighbor graph construction [SYSTEMDS-3229] - WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform [SYSTEMDS-3241] - Federated quantile [SYSTEMDS-3291] - Apply builtin for mice [SYSTEMDS-3348] - Federated Monitoring Tool [SYSTEMDS-3496] - New builtin function auc (area under ROC curve)

Dependency upgrade

[SYSTEMDS-3375] - CUDA11 / CUDNN8 support

Documentation

[SYSTEMDS-3407] - GMM is missing docs for seed and verbose [SYSTEMDS-3434] - Python API does not include params for all kvargs

New Contributors

@fathollahzadeh made their first contribution in https://github.com/apache/systemds/pull/1469
@morf1us made their first contribution in https://github.com/apache/systemds/pull/1520

Full Changelog: https://github.com/apache/systemds/compare/2.2.0-rc1...3.0.0-rc2