-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improve the speed of from_dataframe with a MultiIndex (by 40x!) (#4184)
* Add MultiIndexSeries.time_to_xarray() benchmark * Improve the speed of from_dataframe with a MultiIndex Fixes GH-2459 Before: pandas.MultiIndexSeries.time_to_xarray ======= ========= ========== -- subset ------- -------------------- dtype True False ======= ========= ========== int 505±0ms 37.1±0ms float 485±0ms 38.3±0ms ======= ========= ========== After: pandas.MultiIndexSeries.time_to_xarray ======= ========= ========== -- subset ------- -------------------- dtype True False ======= ========= ========== int 11.5±0ms 39.2±0ms float 12.5±0ms 26.6±0ms ======= ========= ========== There are still some cases where we have to fall back to the existing slow implementation, but hopefully they should now be relatively rare. * remove unused import * Simplify converting MultiIndex dataframes * remove comments * remove types with NA * more multiindex dataframe tests * add whats new note * Preserve order of MultiIndex levels in from_dataframe * Add todo note * Rewrite from_dataframe to avoid passing around a dataframe * Require that MultiIndexes are unique even with sparse=True * clarify comment
- Loading branch information
Showing
5 changed files
with
127 additions
and
30 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
import numpy as np | ||
import pandas as pd | ||
|
||
from . import parameterized | ||
|
||
|
||
class MultiIndexSeries: | ||
def setup(self, dtype, subset): | ||
data = np.random.rand(100000).astype(dtype) | ||
index = pd.MultiIndex.from_product( | ||
[ | ||
list("abcdefhijk"), | ||
list("abcdefhijk"), | ||
pd.date_range(start="2000-01-01", periods=1000, freq="B"), | ||
] | ||
) | ||
series = pd.Series(data, index) | ||
if subset: | ||
series = series[::3] | ||
self.series = series | ||
|
||
@parameterized(["dtype", "subset"], ([int, float], [True, False])) | ||
def time_to_xarray(self, dtype, subset): | ||
self.series.to_xarray() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters