Home » Oreps » Systemds » All Pull Requests » #2600
apache/systemds
An open source ML system for the end-to-end data science lifecycle
Watch
1
Commit Activity
Pull Requests
Directory Browser
Velocity History
Releases

[SYSTEMDS-3835] Improve efficiency of Representations in Scuro #2600

Compare code on GitHub
Merged
christinadionysio opened 9:03am on August 31, 2026 wanted to merge 5586 Δ into apache/systemds main from
text_rep_efficiency

Pull request #2600 quality checks

Code files and functions touched by this pull request are analyzed below.

Quality targets

How does this pull request compare to chosen code quality targets? Read more about code quality targets.

Target name Target value This PR
Maximum Diff Delta per PR 500 5586
Maximum comment count per PR 10 0
Pull request maximum business days between activity 3 14
Pull request maximum business days open 5 15

Coupled files

Are there any files from this commit that are always committed alongside another file? This often suggests that there exists code coupling to potentially remove. Read more about code coupling

Committed together % Code file
Coupled code file 		Commits in PR with both
100 src/main/python/systemds/scuro/representations/x3d.py
src/main/python/systemds/scuro/representations/vgg.py 		1
100 src/main/python/systemds/scuro/representations/image_bind.py
src/main/python/systemds/scuro/representations/resnet.py 		1
100 src/main/python/systemds/scuro/representations/image_bind.py
src/main/python/systemds/scuro/dataloader/base_loader.py 		2
100 src/main/python/systemds/scuro/representations/image_bind.py
src/main/python/systemds/scuro/modality/unimodal_modality.py 		3
100 src/main/python/systemds/scuro/representations/image_bind.py
src/main/python/systemds/scuro/representations/swin_video_transformer.py 		1
100 src/main/python/systemds/scuro/representations/image_bind.py
src/main/python/systemds/scuro/representations/x3d.py 		2
100 src/main/python/systemds/scuro/representations/image_bind.py
src/main/python/systemds/scuro/representations/clip.py 		1
100 src/main/python/systemds/scuro/representations/image_bind.py
src/main/python/systemds/scuro/representations/vgg.py 		2
100 src/main/python/systemds/scuro/drsearch/representation_dag.py
src/main/python/systemds/scuro/drsearch/unimodal_optimizer.py 		1
100 src/main/python/systemds/scuro/drsearch/modality_shared_memory.py
src/main/python/systemds/scuro/modality/unimodal_modality.py 		1
100 src/main/python/systemds/scuro/drsearch/modality_shared_memory.py
src/main/python/systemds/scuro/drsearch/node_executor.py 		1
100 src/main/python/systemds/scuro/drsearch/modality_shared_memory.py
src/main/python/systemds/scuro/drsearch/node_scheduler.py 		1
100 src/main/python/systemds/scuro/drsearch/modality_shared_memory.py
src/main/python/systemds/scuro/representations/vgg.py 		1
100 src/main/python/systemds/scuro/drsearch/multimodal_optimizer.py
src/main/python/systemds/scuro/modality/unimodal_modality.py 		1
100 src/main/python/systemds/scuro/drsearch/multimodal_optimizer.py
src/main/python/systemds/scuro/drsearch/representation_dag.py 		1
100 src/main/python/systemds/scuro/drsearch/multimodal_optimizer.py
src/main/python/systemds/scuro/drsearch/unimodal_optimizer.py 		1
100 src/main/python/systemds/scuro/drsearch/multimodal_optimizer.py
src/main/python/systemds/scuro/representations/timeseries_representations.py 		1
100 src/main/python/systemds/scuro/representations/clip.py
src/main/python/systemds/scuro/representations/bert.py 		2
100 src/main/python/systemds/scuro/representations/utils.py
src/main/python/systemds/scuro/representations/bert.py 		1
100 src/main/python/systemds/scuro/representations/bow.py
src/main/python/systemds/scuro/representations/tfidf.py 		1
100 src/main/python/systemds/scuro/modality/joined.py
src/main/python/tests/scuro/data_generator.py 		1
100 src/main/python/systemds/scuro/representations/optical_flow.py
src/main/python/systemds/scuro/representations/resnet.py 		1
100 src/main/python/systemds/scuro/representations/optical_flow.py
src/main/python/systemds/scuro/modality/unimodal_modality.py 		1
100 src/main/python/systemds/scuro/representations/optical_flow.py
src/main/python/systemds/scuro/representations/swin_video_transformer.py 		1
100 src/main/python/systemds/scuro/representations/optical_flow.py
src/main/python/systemds/scuro/representations/x3d.py 		1
100 src/main/python/systemds/scuro/representations/swin_video_transformer.py
src/main/python/systemds/scuro/representations/bert.py 		1
100 src/main/python/systemds/scuro/representations/swin_video_transformer.py
src/main/python/systemds/scuro/representations/resnet.py 		2
100 src/main/python/systemds/scuro/representations/swin_video_transformer.py
src/main/python/systemds/scuro/representations/wav2vec.py 		1
100 src/main/python/systemds/scuro/representations/swin_video_transformer.py
src/main/python/systemds/scuro/representations/x3d.py 		1
100 src/main/python/systemds/scuro/representations/swin_video_transformer.py
src/main/python/systemds/scuro/representations/clip.py 		2
100 src/main/python/systemds/scuro/representations/swin_video_transformer.py
src/main/python/systemds/scuro/representations/vgg.py 		2
100 src/main/python/systemds/scuro/representations/wav2vec.py
src/main/python/systemds/scuro/representations/bert.py 		1
100 src/main/python/systemds/scuro/representations/wav2vec.py
src/main/python/systemds/scuro/representations/resnet.py 		1
100 src/main/python/systemds/scuro/representations/wav2vec.py
src/main/python/systemds/scuro/representations/swin_video_transformer.py 		1
100 src/main/python/systemds/scuro/representations/x3d.py
src/main/python/systemds/scuro/representations/resnet.py 		1
100 src/main/python/systemds/scuro/representations/x3d.py
src/main/python/systemds/scuro/representations/swin_video_transformer.py 		1
100 src/main/python/systemds/scuro/drsearch/worker_pool.py
src/main/python/systemds/scuro/utils/memory_utility.py 		1
100 src/main/python/systemds/scuro/drsearch/dag_group_scheduler.py
src/main/python/systemds/scuro/modality/unimodal_modality.py 		2
100 src/main/python/systemds/scuro/drsearch/worker_pool.py
src/main/python/systemds/scuro/drsearch/node_scheduler.py 		1
100 src/main/python/systemds/scuro/representations/color_histogram.py
src/main/python/systemds/scuro/representations/vgg.py 		1
100 src/main/python/systemds/scuro/representations/color_histogram.py
src/main/python/systemds/scuro/representations/clip.py 		1
100 src/main/python/systemds/scuro/representations/color_histogram.py
src/main/python/systemds/scuro/representations/x3d.py 		1
100 src/main/python/systemds/scuro/representations/color_histogram.py
src/main/python/systemds/scuro/representations/swin_video_transformer.py 		1
100 src/main/python/systemds/scuro/representations/color_histogram.py
src/main/python/systemds/scuro/representations/resnet.py 		1
100 src/main/python/systemds/scuro/drsearch/worker_pool.py
src/main/python/systemds/scuro/drsearch/node_executor.py 		2
100 src/main/python/systemds/scuro/representations/vgg.py
src/main/python/systemds/scuro/representations/color_histogram.py 		1
100 src/main/python/systemds/scuro/drsearch/dag_group_scheduler.py
src/main/python/systemds/scuro/representations/clip.py 		1
100 src/main/python/systemds/scuro/representations/vgg.py
src/main/python/systemds/scuro/representations/clip.py 		2
100 src/main/python/systemds/scuro/drsearch/worker_pool.py
src/main/python/systemds/scuro/drsearch/modality_shared_memory.py 		1
100 src/main/python/systemds/scuro/drsearch/worker_pool.py
src/main/python/systemds/scuro/drsearch/unimodal_optimizer.py 		1
100 src/main/python/systemds/scuro/drsearch/dag_group_scheduler.py
src/main/python/systemds/scuro/dataloader/image_loader.py 		1
100 src/main/python/systemds/scuro/drsearch/worker_pool.py
src/main/python/systemds/scuro/representations/image_bind.py 		2
100 src/main/python/systemds/scuro/representations/vgg.py
src/main/python/systemds/scuro/representations/x3d.py 		1
100 src/main/python/systemds/scuro/drsearch/worker_pool.py
src/main/python/systemds/scuro/modality/unimodal_modality.py 		2
100 src/main/python/systemds/scuro/representations/vgg.py
src/main/python/systemds/scuro/representations/swin_video_transformer.py 		2
100 src/main/python/systemds/scuro/representations/vgg.py
src/main/python/systemds/scuro/representations/resnet.py 		2
100 src/main/python/systemds/scuro/drsearch/dag_group_executor.py
src/main/python/systemds/scuro/drsearch/representation_dag.py 		1
100 src/main/python/systemds/scuro/representations/vgg.py
src/main/python/systemds/scuro/representations/bert.py 		1
100 src/main/python/systemds/scuro/representations/openface.py
src/main/python/systemds/scuro/representations/image_bind.py 		1
100 src/main/python/systemds/scuro/representations/openface.py
src/main/python/systemds/scuro/representations/vgg.py 		1
100 src/main/python/systemds/scuro/drsearch/dag_group_executor.py
src/main/python/systemds/scuro/drsearch/node_executor.py 		1
96 src/main/python/systemds/scuro/representations/word2vec.py
src/main/python/systemds/scuro/representations/tfidf.py 		1
92 src/main/python/systemds/scuro/representations/tfidf.py
src/main/python/systemds/scuro/representations/bow.py 		1
90 src/main/python/systemds/scuro/representations/mfcc.py
src/main/python/systemds/scuro/representations/mel_spectrogram.py 		1
89 src/main/python/systemds/scuro/representations/tfidf.py
src/main/python/systemds/scuro/representations/word2vec.py 		1
88 src/main/python/systemds/scuro/representations/bow.py
src/main/python/systemds/scuro/representations/word2vec.py 		1
88 src/main/python/systemds/scuro/dataloader/base_loader.py
src/main/python/systemds/scuro/modality/unimodal_modality.py 		2
88 src/main/python/systemds/scuro/representations/x3d.py
src/main/python/systemds/scuro/representations/clip.py 		1
88 src/main/python/systemds/scuro/representations/x3d.py
src/main/python/systemds/scuro/representations/image_bind.py 		2
88 src/main/python/systemds/scuro/representations/wav2vec.py
src/main/python/systemds/scuro/representations/clip.py 		1
87 src/main/python/systemds/scuro/representations/vgg.py
src/main/python/systemds/scuro/representations/wav2vec.py 		1
87 src/main/python/systemds/scuro/representations/vgg.py
src/main/python/systemds/scuro/representations/image_bind.py 		2
87 src/main/python/systemds/scuro/representations/swin_video_transformer.py
src/main/python/systemds/scuro/representations/color_histogram.py 		1
87 src/main/python/systemds/scuro/representations/vgg.py
src/main/python/systemds/scuro/drsearch/unimodal_optimizer.py 		1
85 src/main/python/tests/scuro/test_operator_registry.py
src/main/python/tests/scuro/test_unimodal_optimizer.py 		1
84 src/main/python/systemds/scuro/representations/word2vec.py
src/main/python/systemds/scuro/representations/bow.py 		1
84 src/main/python/systemds/scuro/drsearch/node_scheduler.py
src/main/python/systemds/scuro/drsearch/node_executor.py 		1
83 src/main/python/tests/scuro/test_window_operations.py
src/main/python/tests/scuro/test_unimodal_representations.py 		1
82 src/main/python/systemds/scuro/dataloader/base_loader.py
src/main/python/systemds/scuro/representations/resnet.py 		1
81 src/main/python/systemds/scuro/representations/mfcc.py
src/main/python/systemds/scuro/representations/window_aggregation.py 		1
81 src/main/python/systemds/scuro/modality/unimodal_modality.py
src/main/python/systemds/scuro/modality/modality.py 		1
80 src/main/python/systemds/scuro/modality/modality.py
src/main/python/systemds/scuro/modality/unimodal_modality.py 		1
78 src/main/python/systemds/scuro/representations/clip.py
src/main/python/systemds/scuro/representations/vgg.py 		2
78 src/main/python/systemds/scuro/representations/clip.py
src/main/python/systemds/scuro/representations/resnet.py 		2
77 src/main/python/systemds/scuro/representations/x3d.py
src/main/python/systemds/scuro/representations/color_histogram.py 		1
77 src/main/python/systemds/scuro/representations/wav2vec.py
src/main/python/systemds/scuro/representations/vgg.py 		1
76 src/main/python/systemds/scuro/modality/unimodal_modality.py
src/main/python/systemds/scuro/representations/resnet.py 		1
76 src/main/python/tests/scuro/test_unimodal_representations.py
src/main/python/tests/scuro/test_window_operations.py 		1
75 src/main/python/systemds/scuro/representations/swin_video_transformer.py
src/main/python/systemds/scuro/representations/image_bind.py 		1
75 src/main/python/systemds/scuro/representations/resnet.py
src/main/python/systemds/scuro/representations/bert.py 		1
75 src/main/python/systemds/scuro/representations/swin_video_transformer.py
src/main/python/systemds/scuro/drsearch/unimodal_optimizer.py 		1
75 src/main/python/systemds/scuro/drsearch/modality_shared_memory.py
src/main/python/systemds/scuro/drsearch/worker_pool.py 		1
75 src/main/python/systemds/scuro/drsearch/modality_shared_memory.py
src/main/python/systemds/scuro/utils/memory_utility.py 		1
75 src/main/python/systemds/scuro/drsearch/modality_shared_memory.py
src/main/python/systemds/scuro/drsearch/task.py 		1
75 src/main/python/systemds/scuro/drsearch/dag_group_scheduler.py
src/main/python/systemds/scuro/representations/color_histogram.py 		1
75 src/main/python/systemds/scuro/drsearch/dag_group_scheduler.py
src/main/python/systemds/scuro/dataloader/base_loader.py 		1
75 src/main/python/systemds/scuro/drsearch/dag_group_scheduler.py
src/main/python/systemds/scuro/representations/vgg.py 		1
75 src/main/python/systemds/scuro/drsearch/dag_group_scheduler.py
src/main/python/systemds/scuro/representations/resnet.py 		1
75 src/main/python/systemds/scuro/drsearch/dag_group_scheduler.py
src/main/python/systemds/scuro/drsearch/node_executor.py 		1
72 src/main/python/systemds/scuro/dataloader/video_loader.py
src/main/python/systemds/scuro/dataloader/base_loader.py 		1
72 src/main/python/systemds/scuro/dataloader/video_loader.py
src/main/python/systemds/scuro/representations/resnet.py 		1
71 src/main/python/systemds/scuro/representations/clip.py
src/main/python/systemds/scuro/drsearch/unimodal_optimizer.py 		1
71 src/main/python/systemds/scuro/representations/bert.py
src/main/python/systemds/scuro/representations/resnet.py 		1
71 src/main/python/systemds/scuro/drsearch/task.py
src/main/python/systemds/scuro/drsearch/node_executor.py 		1
71 src/main/python/systemds/scuro/drsearch/task.py
src/main/python/systemds/scuro/drsearch/unimodal_optimizer.py 		1
71 src/main/python/systemds/scuro/drsearch/task.py
src/main/python/systemds/scuro/modality/unimodal_modality.py 		1
71 src/main/python/systemds/scuro/drsearch/unimodal_optimizer.py
src/main/python/systemds/scuro/drsearch/representation_dag.py 		1
71 src/main/python/systemds/scuro/dataloader/image_loader.py
src/main/python/systemds/scuro/dataloader/video_loader.py 		1
71 src/main/python/systemds/scuro/dataloader/image_loader.py
src/main/python/systemds/scuro/representations/clip.py 		1
71 src/main/python/systemds/scuro/utils/memory_utility.py
src/main/python/systemds/scuro/drsearch/node_scheduler.py 		1
71 src/main/python/tests/scuro/test_operator_registry.py
src/main/python/tests/scuro/test_window_operations.py 		1
71 src/main/python/systemds/scuro/modality/unimodal_modality.py
src/main/python/systemds/scuro/dataloader/base_loader.py 		2
71 src/main/python/tests/scuro/test_operator_registry.py
src/main/python/tests/scuro/data_generator.py 		1
70 src/main/python/systemds/scuro/dataloader/base_loader.py
src/main/python/systemds/scuro/dataloader/video_loader.py 		1

Biggest files changes in pull request

Estimated line counts are shown. Values may change once change summary has finished building.

Path View whole file Lines removed Lines added Final line count
src/main/python/systemds/scuro/representations/utils.py BeforeAfter 0 287 ---
src/main/python/systemds/scuro/representations/image_bind.py BeforeAfter 46 285 ---
src/main/python/systemds/scuro/dataloader/video_loader.py BeforeAfter 38 119 ---
src/main/python/tests/scuro/test_unimodal_representations.py BeforeAfter 15 495 ---
src/main/python/systemds/scuro/representations/clip.py BeforeAfter 95 147 ---
src/main/python/tests/scuro/test_window_operations.py BeforeAfter 0 398 ---
src/main/python/tests/scuro/test_chunked_leaf_execution.py BeforeAfter 0 352 ---
src/main/python/tests/scuro/test_transformer_text_aggregation.py BeforeAfter 0 353 ---
src/main/python/systemds/scuro/representations/x3d.py BeforeAfter 68 178 ---
src/main/python/systemds/scuro/drsearch/multimodal_ga_optimizer.py BeforeAfter 0 1103 ---
src/main/python/systemds/scuro/representations/window_aggregation.py BeforeAfter 12 68 ---
src/main/python/systemds/scuro/representations/bert.py BeforeAfter 59 105 ---
src/main/python/tests/scuro/test_lazy_visual_loading.py BeforeAfter 0 177 ---
src/main/python/tests/scuro/test_window_representation_batching.py BeforeAfter 0 141 ---
src/main/python/systemds/scuro/representations/resnet.py BeforeAfter 60 68 ---
src/main/python/systemds/scuro/representations/vgg.py BeforeAfter 74 74 ---
src/main/python/systemds/scuro/drsearch/representation_dag.py BeforeAfter 36 8 ---
src/main/python/tests/scuro/test_unimodal_optimizer.py BeforeAfter 0 222 ---
src/main/python/tests/scuro/test_neural_encoder_batching.py BeforeAfter 0 156 ---
src/main/python/systemds/scuro/representations/swin_video_transformer.py BeforeAfter 37 55 ---
src/main/python/systemds/scuro/modality/joined.py BeforeAfter 0 52 ---
src/main/python/systemds/scuro/drsearch/unimodal_optimizer.py BeforeAfter 3 75 ---
src/main/python/systemds/scuro/representations/wav2vec.py BeforeAfter 14 79 ---
src/main/python/systemds/scuro/modality/unimodal_modality.py BeforeAfter 16 76 ---
src/main/python/systemds/scuro/dataloader/image_loader.py BeforeAfter 11 34 ---
src/main/python/tests/scuro/test_modality_pad.py BeforeAfter 0 105 ---
src/main/python/tests/scuro/data_generator.py BeforeAfter 0 37 ---
src/main/python/tests/scuro/test_multimodal_ga_optimizer.py BeforeAfter 0 1220 ---
src/main/python/systemds/scuro/dataloader/base_loader.py BeforeAfter 10 46 ---
src/main/python/systemds/scuro/modality/modality.py BeforeAfter 0 24 ---
src/main/python/systemds/scuro/__init__.py BeforeAfter 0 63 ---
src/main/python/systemds/scuro/drsearch/multimodal_optimizer.py BeforeAfter 2 0 ---
src/main/python/systemds/scuro/representations/mfcc.py BeforeAfter 1 4 ---
src/main/python/systemds/scuro/representations/optical_flow.py BeforeAfter 0 4 ---
src/main/python/systemds/scuro/representations/word2vec.py BeforeAfter 1 1 ---
src/main/python/systemds/scuro/representations/mel_spectrogram.py BeforeAfter 1 4 ---
src/main/python/systemds/scuro/representations/color_histogram.py BeforeAfter 0 5 ---
src/main/python/tests/scuro/test_operator_registry.py BeforeAfter 0 7 ---
.github/workflows/python.yml BeforeAfter 0 2 ---
src/main/python/systemds/scuro/drsearch/node_executor.py BeforeAfter 16 103 ---
src/main/python/systemds/scuro/representations/bow.py BeforeAfter 0 1 ---
src/main/python/systemds/scuro/representations/tfidf.py BeforeAfter 0 1 ---
src/main/python/systemds/scuro/drsearch/dag_group_executor.py BeforeAfter 0 0 ---
src/main/python/systemds/scuro/drsearch/worker_pool.py BeforeAfter 7 123 ---
src/main/python/systemds/scuro/drsearch/dag_group_scheduler.py BeforeAfter 0 0 ---
src/main/python/systemds/scuro/representations/timeseries_representations.py BeforeAfter 1 4 ---
src/main/python/systemds/scuro/drsearch/operator_registry.py BeforeAfter 0 0 ---
src/main/python/systemds/scuro/drsearch/task.py BeforeAfter 1 135 ---
src/main/python/systemds/scuro/drsearch/modality_shared_memory.py BeforeAfter 0 6 ---
src/main/python/systemds/scuro/representations/openface.py BeforeAfter 0 577 ---
src/main/python/systemds/scuro/utils/memory_utility.py BeforeAfter 0 20 ---
src/main/python/systemds/scuro/utils/checkpointing.py BeforeAfter 0 0 ---
src/main/python/systemds/scuro/drsearch/ranking.py BeforeAfter 0 126 ---
src/main/python/systemds/scuro/drsearch/node_scheduler.py BeforeAfter 15 138 ---
src/main/python/systemds/scuro/models/model.py BeforeAfter 0 2 ---
src/main/python/systemds/scuro/drsearch/test_set_evaluation.py BeforeAfter 0 121 ---
src/main/python/systemds/scuro/dataloader/timeseries_loader.py BeforeAfter 0 2 ---

️ Biggest functions grown or shrunk by pull request

Which functions had their line count change as part of this pull request?

Estimated line counts are shown. Values may change once change summary has finished building.

Qualified function name View whole function Lines removed Lines added Final line count
UnimodalModality::apply_representations BeforeAfter 10 10 111
VideoLoader::_decode_file BeforeAfter 2 2 37
Representation::UnimodalRepresentation::VGG19::estimate_peak_memory_bytes BeforeAfter 3 3 35
UnimodalModality::subset BeforeAfter 4 4 18
Representation::UnimodalRepresentation::VGG19::estimate_output_memory_bytes BeforeAfter 1 1 5
VideoLoader::extract BeforeAfter 4 4 4
GitClear uses cookies to ensure you get the best experience on our website. Learn more about our cookie policy