Modin project suggestions

[devin-petersohn]

Modin Potential Projects

The purpose of this issue is to have a single place for all issues good for new users/projects we are working on/etc. It is also intended to give prospective contributors an idea of what would be highest impact to work on.

Overview

If you are interested in contributing to Modin, welcome! If you are interested in contributing something that isn't here, let us know and we'll add it! A good place to start looking for issues to contribute to is in the Good First Issue 🔰 or Help Wanted 🌐 tagged issues.

Table of Contents

1.) APIs
2.) Experimental Features
3.) Query Processing/Planning
4.) System Level Enhancements

1.) APIs

New APIs

The following APIs are highest priority in Modin (in no particular order). This is based on the number of people requesting it and the number of people using it from our study of usage. If you would like to contribute a particular API, please comment on the related issue:

• pivot_table: pivot_table function implementation #1200
• groupby with list of columns: NotImplementedError: Groupby with lists of columns not yet supported. #75 Added in Add support for grouping by multiple columns when doing a reduction #987
• parallel read_excel: Implementation for parallel read_excel #467 Added in Add parallel read_excel feature #1648
• parallel read_pickle: Request to implement read_pickle function #529
• parallel read_json: Request to implement read_json function #554 Added in Preliminary read_json implementation #715
• Compressed read_csv: Feature Request: read_csv on compressed files #630 Added in reading gzipped csv files #682
• drop_duplicates: New Feature Request: Add support for drop_duplicates() #634 Added in support for duplicated() and drop_duplicates() #892
• pd.value_counts/Series.value_counts: New method request: Series.value_counts and pd.value_counts #653 Added in Add value_counts implementation for Series and as free function #1535
• Series.str: New method Request: Series.str #654 Added in Add support for text manipulation operations #713
• read_fwf: Request to implement read_fwf  #752 Added in Add implementation of read_fwf #1448

Improve performance of existing methods

The following APIs need some improvement. These are typically issues requiring deeper knowledge of Modin internals.

• groupby performance: Enhancement: groupby performance #655 Added in Improve performance of Groupby #659
• sort_values performance: Improve sort_values #158
• merge/join performance: FEAT: Support right and outer types of join for merge/join #656
• insert performance: Enhancement: Rewriting insert to use apply_func_to_select_indices instead of apply_func_to_select_indices_along_full_axis #670 Added in Enhancement: insert and __setitem__ performance  #696

2.) Experimental Features

The following features will either augment the pandas API by providing hints to the system or add some new feature that doesn't exist in pandas.

• read_sql parallelism parameter: read_sql() parallelism parameter #455
• Progress bar: Modin progress bar #629
• Read multiple files from S3 into the same DataFrame: How to read multi parquets files from S3.  #700

3.) Query Processing/Planning

More here later...

4.) System Level Enhancements

• Add Windows support: Windows support #657 Added in Add Dask futures implementation #732
• GPU DataFrame support: GPU DataFrame support #150
• Fixed size Cluster support: How to achieve clustering in Modin and distributed machine learning #583
• Ray autoscaler support: Support for Ray autoscaler #660 Added in Initial ray autoscaler support #661
• Add Geopandas API: Support Geopandas #671

[devin-petersohn]

This is a bit outdated, and we have a lot of these implemented.

[devin-petersohn]

Closing to start a new thread, thanks to all who contributed!