BUG: Inserting array of same size with Series.loc raises ValueError #38266

ma3da · 2020-12-03T17:33:05Z

closes #37748
closes #37486
closes #38271

tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

(GH37748) added test

jbrockmendel · 2020-12-03T18:29:24Z

arw2019 · 2020-12-03T18:29:18Z

pandas/tests/indexing/test_iloc.py

+        # GH37748
+        ser = Series(0, index=range(5), dtype="object")
+
+        expected = np.zeros(size)


would you get the same bug with a length size python list?

arw2019 · 2020-12-03T18:33:49Z

doc/source/whatsnew/v1.2.0.rst

@@ -677,6 +677,7 @@ Indexing
 - Bug in :meth:`DataFrame.loc` and :meth:`DataFrame.__getitem__`  raising ``KeyError`` when columns were :class:`MultiIndex` with only one level (:issue:`29749`)
 - Bug in :meth:`Series.__getitem__` and :meth:`DataFrame.__getitem__` raising blank ``KeyError`` without missing keys for :class:`IntervalIndex` (:issue:`27365`)
 - Bug in setting a new label on a :class:`DataFrame` or :class:`Series` with a :class:`CategoricalIndex` incorrectly raising ``TypeError`` when the new label is not among the index's categories (:issue:`38098`)
+- Bug in :meth:`Series.loc` and :meth:`Series.iloc` raising ``ValueError`` when inserting an array in a ``object`` Series of equal length (:issue:`37748`)


array seems vague. Do you mean NumPy array, list, ExtensionArray (or selection of those)

all of those i guess :)

Indeed, np.array, list, tuple do cause the raise. From your example, ExtensionArray don't but also behave unexpectedly.

arw2019 · 2020-12-03T18:41:47Z

Also does this solve #38271

ma3da · 2020-12-03T18:59:04Z

@arw2019

Also does this solve #38271

It seems so:

In [1]: import numpy as np 
   ...:     ...: import pandas as pd 
   ...:     ...:  
   ...:     ...: ser = pd.Series(1, index=list("abcde"), dtype="object") 
   ...:     ...:  
   ...:     ...: expected = pd.array([0,0,3,0,0]) 
   ...:     ...: ser.loc["a"] = expected 
   ...:     ...: result = ser[0]                                                                                                                                                                                                               

In [2]: ser                                                                                                                                                                                                                                    
Out[2]: 
a    [0, 0, 3, 0, 0]
b                  1
c                  1
d                  1
e                  1
dtype: object

arw2019

Okay great can you add tests for the list, ExtensionArray cases and update whatsnew

arw2019 · 2020-12-03T19:04:05Z

doc/source/whatsnew/v1.2.0.rst

@@ -677,6 +677,7 @@ Indexing
 - Bug in :meth:`DataFrame.loc` and :meth:`DataFrame.__getitem__`  raising ``KeyError`` when columns were :class:`MultiIndex` with only one level (:issue:`29749`)
 - Bug in :meth:`Series.__getitem__` and :meth:`DataFrame.__getitem__` raising blank ``KeyError`` without missing keys for :class:`IntervalIndex` (:issue:`27365`)
 - Bug in setting a new label on a :class:`DataFrame` or :class:`Series` with a :class:`CategoricalIndex` incorrectly raising ``TypeError`` when the new label is not among the index's categories (:issue:`38098`)
+- Bug in :meth:`Series.loc` and :meth:`Series.iloc` raising ``ValueError`` when inserting an array in a ``object`` Series of equal length (:issue:`37748`)


all of those i guess :)

ma3da · 2020-12-03T22:22:33Z

pandas/_testing.py

+def assert_python_equal(left, right):
+    """
+    Check left and right are equal w.r.t the ``==`` operator.
+
+    Parameters
+    ----------
+    left : object
+    right : object
+    """
+    assert left == right


ok, I'm guessing this is too much, but I failed to find helper funcs to assert list/tuple equality.
I'll welcome advice on how to write the tests :)

yeah don't do this, just do assert left == right pytest already handles this

phofl · 2020-12-03T22:46:43Z

pandas/tests/indexing/test_loc.py

+
+        ser.loc["a"] = expected
+        result = ser[0]
+        assert_fn(result, expected)


Is there a reason for not checking the whole Series? This would also avoid your helper function

good point, thanks

jreback · 2020-12-03T23:18:40Z

pandas/_testing.py

+def assert_python_equal(left, right):
+    """
+    Check left and right are equal w.r.t the ``==`` operator.
+
+    Parameters
+    ----------
+    left : object
+    right : object
+    """
+    assert left == right


yeah don't do this, just do assert left == right pytest already handles this

…r_same_size

pandas/tests/indexing/test_loc.py

jreback · 2020-12-04T23:34:24Z

doc/source/whatsnew/v1.2.0.rst

@@ -677,6 +677,8 @@ Indexing
 - Bug in :meth:`DataFrame.loc` and :meth:`DataFrame.__getitem__`  raising ``KeyError`` when columns were :class:`MultiIndex` with only one level (:issue:`29749`)
 - Bug in :meth:`Series.__getitem__` and :meth:`DataFrame.__getitem__` raising blank ``KeyError`` without missing keys for :class:`IntervalIndex` (:issue:`27365`)
 - Bug in setting a new label on a :class:`DataFrame` or :class:`Series` with a :class:`CategoricalIndex` incorrectly raising ``TypeError`` when the new label is not among the index's categories (:issue:`38098`)
+- Bug in :meth:`Series.loc` and :meth:`Series.iloc` raising ``ValueError`` when inserting a listlike ``np.array``, ``list`` or ``tuple`` in an ``object`` Series of equal length (:issue:`37748`)


you list 3 issues in the top of PR, what's missing?

Ah, I added a whastnew note for #38271 and one for #37748.
In the PR, I listed #37486 because it is a particular case of #37748 (iloc on a Series with a list, both of size 1).

Pls add the issue number to the note for #37748 then

jreback · 2020-12-04T23:34:44Z

cc @phofl and @jbrockmendel if any comments.

phofl · 2020-12-05T00:11:04Z

lgtm otherwise

…r_same_size

jreback · 2020-12-05T17:14:10Z

thanks @ma3da

ma3da added 4 commits December 3, 2020 18:24

BUG: Error for Series.loc on array of same size

e2e9ad8

(GH37748) added test

added tests cases for is_scalar_indexer, ndim=1

482f349

allow non-tuple scalar for ndim=1 in is_scalar_indexer

92d1960

whatsnew entry

f376ec5

ma3da changed the title ~~Gh 37748 loc error same size~~ BUG: Error for Series.loc on array of same size Dec 3, 2020

ma3da changed the title ~~BUG: Error for Series.loc on array of same size~~ BUG: Inserting array of same size with Series.loc raises ValueError Dec 3, 2020

ma3da marked this pull request as ready for review December 3, 2020 17:45

arw2019 reviewed Dec 3, 2020

View reviewed changes

arw2019 suggested changes Dec 3, 2020

View reviewed changes

ma3da added 2 commits December 3, 2020 23:15

more testing

0165b30

reworked whatsnew entry

1aa4bed

ma3da commented Dec 3, 2020

View reviewed changes

phofl reviewed Dec 3, 2020

View reviewed changes

jreback requested changes Dec 3, 2020

View reviewed changes

jreback added the Indexing Related to indexing on series/frames, not to indexes themselves label Dec 3, 2020

ma3da added 2 commits December 4, 2020 06:59

test with assert_series_equal + fix np.zeros -> np.array

ed16223

Merge remote-tracking branch 'upstream/master' into GH_37748_loc_erro…

f054a0e

…r_same_size

jreback added the Bug label Dec 4, 2020

jreback added this to the 1.2 milestone Dec 4, 2020

jreback reviewed Dec 4, 2020

View reviewed changes

pandas/tests/indexing/test_loc.py Show resolved Hide resolved

jreback reviewed Dec 4, 2020

View reviewed changes

ma3da added 2 commits December 5, 2020 01:42

cr: add issue to whatsnew

039a6f9

Merge remote-tracking branch 'upstream/master' into GH_37748_loc_erro…

418254b

…r_same_size

jreback approved these changes Dec 5, 2020

View reviewed changes

jreback merged commit 5399c6d into pandas-dev:master Dec 5, 2020

ma3da deleted the GH_37748_loc_error_same_size branch December 6, 2020 00:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Inserting array of same size with Series.loc raises ValueError #38266

BUG: Inserting array of same size with Series.loc raises ValueError #38266

ma3da commented Dec 3, 2020 •

edited

Loading

jbrockmendel commented Dec 3, 2020

arw2019 Dec 3, 2020

ma3da Dec 3, 2020

arw2019 Dec 3, 2020

arw2019 Dec 3, 2020

ma3da Dec 3, 2020

arw2019 commented Dec 3, 2020

ma3da commented Dec 3, 2020

arw2019 left a comment

arw2019 Dec 3, 2020

ma3da Dec 3, 2020

jreback Dec 3, 2020

ma3da Dec 4, 2020

phofl Dec 3, 2020

ma3da Dec 4, 2020

jreback Dec 3, 2020

jreback Dec 4, 2020

ma3da Dec 5, 2020

phofl Dec 5, 2020

ma3da Dec 5, 2020

jreback commented Dec 4, 2020

phofl commented Dec 5, 2020

jreback commented Dec 5, 2020

BUG: Inserting array of same size with Series.loc raises ValueError #38266

BUG: Inserting array of same size with Series.loc raises ValueError #38266

Conversation

ma3da commented Dec 3, 2020 • edited Loading

jbrockmendel commented Dec 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arw2019 commented Dec 3, 2020

ma3da commented Dec 3, 2020

arw2019 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Dec 4, 2020

phofl commented Dec 5, 2020

jreback commented Dec 5, 2020

ma3da commented Dec 3, 2020 •

edited

Loading