String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) #59376
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…(remove pyarrow_numpy)
| @@ -1305,7 +1304,6 @@ def string_storage(request): | |||
|
|
|||
| * 'python' | |||
| * 'pyarrow' | |||
| * 'pyarrow_numpy' | |||
There was a problem hiding this comment.
So the string_storage fixture will no longer cover the "pyarrow + NaN" variant of the string dtype directly with this change, but:
- Most tests using this fixture are inside the
pandas/tests/iosubmodule, which had an override of this fixture to not include "pyarrow_numpy" anyway (and I removed this override) - I checked/updated the other tests and none of them really need this.
Comment on lines
+463
to
+468
| if value == "pyarrow_numpy": | ||
| # TODO: we can remove extra message after 3.0 | ||
| msg += ( | ||
| ". 'pyarrow_numpy' was specified, but this option should be " | ||
| "enabled using pandas.options.future.infer_string instead" | ||
| ) |
There was a problem hiding this comment.
Added this custom message to the error in case someone actually did end up using pd.options.mode.string_storage = "pyarrow_numpy" to point them to the right option to enable.
Comment on lines
+903
to
+904
| def test_astype_to_string_dtype_not_modifying_input(any_string_dtype, val): | ||
| # GH#51073 - variant of the above test with explicit dtype instances |
There was a problem hiding this comment.
Added this variant because the test above with the string_storage option and astype("string") will now only test the NA-variants of the string dtype.
mroeschke
reviewed
Jul 31, 2024
| with cf.config_prefix("mode"): | ||
| cf.register_option( | ||
| "string_storage", | ||
| "python", | ||
| string_storage_doc, | ||
| validator=is_one_of_factory(["python", "pyarrow", "pyarrow_numpy"]), | ||
| # validator=is_one_of_factory(["python", "pyarrow"]), |
There was a problem hiding this comment.
Suggested change
| # validator=is_one_of_factory(["python", "pyarrow"]), |
There was a problem hiding this comment.
I would prefer to leave this line in so it's easier to go back to that, given that this extra message is only meant to stay for 2.3, and for 3.0 we should already remove it again (because the infer_string option will be enabled by default in 3.0, so then there is no point to let users set that manually)
mroeschke
reviewed
Jul 31, 2024
WillAyd
approved these changes
Aug 1, 2024
mroeschke
approved these changes
Aug 1, 2024
|
Thanks @jorisvandenbossche |
37 tasks
WillAyd
pushed a commit
that referenced
this pull request
Aug 13, 2024
…(remove pyarrow_numpy) (#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
WillAyd
pushed a commit
to WillAyd/pandas
that referenced
this pull request
Aug 14, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
WillAyd
pushed a commit
to WillAyd/pandas
that referenced
this pull request
Aug 15, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
WillAyd
pushed a commit
to WillAyd/pandas
that referenced
this pull request
Aug 15, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
WillAyd
pushed a commit
to WillAyd/pandas
that referenced
this pull request
Aug 15, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
WillAyd
pushed a commit
to WillAyd/pandas
that referenced
this pull request
Aug 21, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
WillAyd
pushed a commit
to WillAyd/pandas
that referenced
this pull request
Aug 22, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
WillAyd
pushed a commit
to WillAyd/pandas
that referenced
this pull request
Aug 22, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
WillAyd
pushed a commit
to WillAyd/pandas
that referenced
this pull request
Aug 27, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
WillAyd
pushed a commit
to WillAyd/pandas
that referenced
this pull request
Sep 20, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
jorisvandenbossche
added a commit
to WillAyd/pandas
that referenced
this pull request
Oct 2, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
jorisvandenbossche
added a commit
to WillAyd/pandas
that referenced
this pull request
Oct 2, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
jorisvandenbossche
added a commit
to WillAyd/pandas
that referenced
this pull request
Oct 3, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
jorisvandenbossche
added a commit
to WillAyd/pandas
that referenced
this pull request
Oct 7, 2024
…(remove pyarrow_numpy) (pandas-dev#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
jorisvandenbossche
added a commit
that referenced
this pull request
Oct 9, 2024
…(remove pyarrow_numpy) (#59376) * String dtype: restrict options.mode.string_storage to python|pyarrow (remove pyarrow_numpy) * add type annotation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We allowed to set
"pyarrow_numpy"as a value to themode.string_storageoption.I assume this was mostly done for testing purposes, because we explicitly documented that this option would be ignored when enabling the
future.infer_stringoption (i.e. when "pyarrow_numpy" is being used by default).In context of the new string dtype (PDEP-14), I think we should remove that option again (since we will generally deprecate/remove "pyarrow_numpy"). Given the above (not documented), I opted for directly removing it from the option instead of first deprecating it (for the StringDtype constructor / string alias, we will first deprecate it). But I did add a clarification to the option error message in case someone actually did use it.
xref #54792