Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REFACTOR-#5005: Use finalize method instead of list comprehension + drain_call_queue #5006

Merged
merged 2 commits into from
Sep 22, 2022

Conversation

anmyachev
Copy link
Collaborator

@anmyachev anmyachev commented Sep 20, 2022

Signed-off-by: Myachev anatoly.myachev@intel.com

What do these changes do?

  • commit message follows format outlined here
  • passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
  • passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
  • signed commit with git commit -s
  • Resolves REFACTOR: use finalize method instead of list comprehension + drain_call_queue #5005
  • tests added and passing
  • module layout described at docs/development/architecture.rst is up-to-date
  • added (Issue Number: PR title (PR Number)) and github username to release notes for next major release

…l_queue

Signed-off-by: Myachev <anatoly.myachev@intel.com>
@anmyachev anmyachev marked this pull request as ready for review September 20, 2022 16:30
@anmyachev anmyachev requested a review from a team as a code owner September 20, 2022 16:30
@codecov
Copy link

codecov bot commented Sep 20, 2022

Codecov Report

Merging #5006 (b24cf22) into master (45879a6) will increase coverage by 4.70%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #5006      +/-   ##
==========================================
+ Coverage   84.89%   89.59%   +4.70%     
==========================================
  Files         267      268       +1     
  Lines       19749    20031     +282     
==========================================
+ Hits        16766    17947    +1181     
+ Misses       2983     2084     -899     
Impacted Files Coverage Δ
...dataframe/pandas/partitioning/partition_manager.py 89.22% <100.00%> (+3.84%) ⬆️
modin/logging/config.py 94.59% <0.00%> (-1.30%) ⬇️
modin/experimental/batch/test/test_pipeline.py 90.21% <0.00%> (ø)
modin/pandas/series.py 94.27% <0.00%> (+0.23%) ⬆️
modin/pandas/groupby.py 93.77% <0.00%> (+0.23%) ⬆️
modin/pandas/series_utils.py 98.89% <0.00%> (+0.55%) ⬆️
...odin/core/storage_formats/pandas/query_compiler.py 96.40% <0.00%> (+0.56%) ⬆️
modin/core/io/text/excel_dispatcher.py 94.01% <0.00%> (+0.85%) ⬆️
modin/core/io/column_stores/parquet_dispatcher.py 96.25% <0.00%> (+2.08%) ⬆️
...tations/pandas_on_python/partitioning/partition.py 93.75% <0.00%> (+2.08%) ⬆️
... and 38 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Signed-off-by: Myachev <anatoly.myachev@intel.com>
@@ -1316,7 +1316,6 @@ def get_right_block(right_partitions, row_idx, col_idx):
)

@classmethod
@wait_computations_if_benchmark_mode
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no need to make this operation synchronous, we can make the operation that calls add_to_apply_calls function synchronous (thereby creating a queue for execution), for example: lazy_map_partitions.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this wrapping is incorrect because wait_computations_if_benchmark_mode assumes the function returns partitions. See also #4273.

Copy link
Collaborator

@pyrito pyrito Sep 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mvashishtha will this fix #4273 as well? Should we add a test here and verify that it does resolve that issue?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it fixes. We can add a test, but it is not necessary, as there are others that will break if we change the implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

REFACTOR: use finalize method instead of list comprehension + drain_call_queue
3 participants