-
Notifications
You must be signed in to change notification settings - Fork 651
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PERF-#5268: Call get
on all partitions at once in to_pandas
#4776
Conversation
Codecov Report
@@ Coverage Diff @@
## master #4776 +/- ##
===========================================
- Coverage 84.82% 65.92% -18.91%
===========================================
Files 268 269 +1
Lines 19701 19986 +285
===========================================
- Hits 16712 13176 -3536
- Misses 2989 6810 +3821
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
23511c1
to
cca6978
Compare
get
on all partitions at once in to_pandas
@vnlitvinov could you take a look? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got benchmarks for this change?
modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition_manager.py
Outdated
Show resolved
Hide resolved
modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition_manager.py
Outdated
Show resolved
Hide resolved
@vnlitvinov results by using asv: the PR
Master branch
|
…pandas' Signed-off-by: Myachev <anatoly.myachev@intel.com>
Signed-off-by: Myachev <anatoly.myachev@intel.com>
Co-authored-by: Vasily Litvinov <fam1ly.n4me@yandex.ru>
Signed-off-by: Myachev <anatoly.myachev@intel.com>
Signed-off-by: Myachev <anatoly.myachev@intel.com>
@vnlitvinov ready for review |
Co-authored-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
modin/core/execution/dask/implementations/pandas_on_dask/partitioning/partition_manager.py
Show resolved
Hide resolved
Co-authored-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
Signed-off-by: Myachev <anatoly.myachev@intel.com>
Signed-off-by: Myachev <anatoly.myachev@intel.com>
Co-authored-by: Vasily Litvinov <fam1ly.n4me@yandex.ru>
Signed-off-by: Myachev <anatoly.myachev@intel.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, minor questions left only
Signed-off-by: Myachev <anatoly.myachev@intel.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
@anmyachev one note - this PR is solving only half of problem stated in #2814 (getting things serially), but not the other (using N*N |
get
on all partitions at once in to_pandas
get
on all partitions at once in to_pandas
I created a separate issue: #5268 |
Signed-off-by: Myachev anatoly.myachev@intel.com
What do these changes do?
flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
git commit -s
docs/development/architecture.rst
is up-to-date