BUG: replace coerces incorrect dtype #12780

sinhrks · 2016-04-03T08:20:04Z

closes BUG/API: Fix .replace dtype conversion rules #12747
tests added / passed
passes git diff upstream/master | flake8 --diff
whatsnew entry

jreback · 2016-04-03T17:31:26Z

pandas/core/internals.py

@@ -1355,6 +1365,12 @@ class NumericBlock(Block):
    is_numeric = True
    _can_hold_na = True

+    def _can_replace_element(self, element):
+        # coercion is handled by putmask
+


no

In [1]: issubclass(np.bool_, np.number) Out[1]: False In [2]: issubclass(np.float_, np.number) Out[2]: True

sinhrks · 2016-04-06T21:31:36Z

Found this affects more functions than I thought. Let me do it in 4 separate PRs.

1 Add numeric related tests to clarify the current behaviour (and future change)
2. Fix numeric coercion.
3. Add datetime-like tests.
4. Fix datetime-like coercion.

related to #12747 and #12780 Author: sinhrks <sinhrks@gmail.com> Closes #12841 from sinhrks/dtype_tests and squashes the following commits: e11cc2a [sinhrks] TST: Add numeric coercion tests

codecov-io · 2016-05-13T18:03:59Z

Current coverage is 85.97% (diff: 89.65%)

Merging #12780 into master will decrease coverage by <.01%

@@             master     #12780   diff @@
==========================================
  Files           140        140          
  Lines         51254      51275    +21   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          44068      44085    +17   
- Misses         7186       7190     +4   
  Partials          0          0

Powered by Codecov. Last update a62fdf8...f9154e8

jreback · 2016-05-13T18:44:57Z

after #13147 this should be slightly more standard

jreback · 2016-11-23T11:27:56Z

pandas/tests/indexing/test_coercion.py


    def _assert_replace_conversion(self, from_key, to_key, how):
        index = pd.Index([3, 4], name='xxx')
        obj = pd.Series(self.rep[from_key], index=index, name='yyy')
        self.assertEqual(obj.dtype, from_key)

+        if (from_key.startswith('datetime') and to_key.startswith('datetime')):
+            # different tz, currently mask_missing raises SystemError


is this for another issue?

Yeah, issued #15183. Let me fix it in a separate pr.

jreback · 2016-11-23T11:28:16Z

@sinhrks looks pretty good, just a couple of small comments.

jreback · 2017-01-19T02:04:48Z

pandas/core/internals.py

@@ -3233,6 +3236,16 @@ def comp(s):
            return _possibly_compare(values, getattr(s, 'asm8', s),
                                     operator.eq)

+        def _cast(block, scalar):


this might be a more general routine (IOW maybe defin as a utility function in internals rather than a nested function here), or maybe as a block method?

cast_scalar?

jreback · 2017-01-19T14:12:20Z

@sinhrks status? (lgtm so far)

sinhrks · 2017-01-21T05:21:42Z

Updated to fix the above points, and now ready for review.

jreback · 2017-01-21T15:48:57Z

looks fine. still some internal things ideally should clean up. but can do later.

though couple of comments if you can address.

jreback · 2017-01-23T16:30:25Z

pandas/types/cast.py

@@ -361,6 +377,11 @@ def _infer_dtype_from_scalar(val):
    elif is_complex(val):
        dtype = np.complex_

+    elif pandas_dtype:


why is this parameter necessary, why wouldn't we always want to infer a pandas dtype if its there (e.g. Period)?

This is not needed for the PR itself. The same func is required for #14145.

jreback · 2017-01-23T16:30:48Z

pandas/types/cast.py

+        if val is tslib.NaT or val.tz is None:
+            dtype = np.dtype('M8[ns]')
+        else:
+            if pandas_dtype:


e.g. here, is this a back-compat issue?

jreback · 2017-03-20T14:03:50Z

@sinhrks I am rebasing this and will merge after testing.

closes pandas-dev#12747 Author: sinhrks <sinhrks@gmail.com> This patch had conflicts when merged, resolved by Committer: Jeff Reback <jeff@reback.net> Closes pandas-dev#12780 from sinhrks/replace_type and squashes the following commits: f9154e8 [sinhrks] remove unnecessary comments 279fdf6 [sinhrks] remove import failure de44877 [sinhrks] BUG: replace coerces incorrect dtype

xref #15736 xref #12780 Author: Jeff Reback <jeff@reback.net> Closes #15765 from jreback/common_types and squashes the following commits: d472646 [Jeff Reback] try removing restriction on windows 8d07cae [Jeff Reback] CLN: replace _interleave_dtype with _find_common_type

closes pandas-dev#12747 Author: sinhrks <sinhrks@gmail.com> This patch had conflicts when merged, resolved by Committer: Jeff Reback <jeff@reback.net> Closes pandas-dev#12780 from sinhrks/replace_type and squashes the following commits: f9154e8 [sinhrks] remove unnecessary comments 279fdf6 [sinhrks] remove import failure de44877 [sinhrks] BUG: replace coerces incorrect dtype

xref pandas-dev#15736 xref pandas-dev#12780 Author: Jeff Reback <jeff@reback.net> Closes pandas-dev#15765 from jreback/common_types and squashes the following commits: d472646 [Jeff Reback] try removing restriction on windows 8d07cae [Jeff Reback] CLN: replace _interleave_dtype with _find_common_type

closes pandas-dev#12747 Author: sinhrks <sinhrks@gmail.com> This patch had conflicts when merged, resolved by Committer: Jeff Reback <jeff@reback.net> Closes pandas-dev#12780 from sinhrks/replace_type and squashes the following commits: f9154e8 [sinhrks] remove unnecessary comments 279fdf6 [sinhrks] remove import failure de44877 [sinhrks] BUG: replace coerces incorrect dtype

xref pandas-dev#15736 xref pandas-dev#12780 Author: Jeff Reback <jeff@reback.net> Closes pandas-dev#15765 from jreback/common_types and squashes the following commits: d472646 [Jeff Reback] try removing restriction on windows 8d07cae [Jeff Reback] CLN: replace _interleave_dtype with _find_common_type

sinhrks added Bug Dtype Conversions Unexpected or buggy dtype conversions API Design labels Apr 3, 2016

sinhrks added this to the 0.18.1 milestone Apr 3, 2016

sinhrks force-pushed the replace_type branch 2 times, most recently from 8ca4b9a to df2eae1 Compare April 3, 2016 15:55

sinhrks mentioned this pull request Apr 3, 2016

BUG/API: Fix .replace dtype conversion rules #12747

Closed

sinhrks force-pushed the replace_type branch from df2eae1 to 1d322fd Compare April 3, 2016 17:21

jreback reviewed Apr 3, 2016
View reviewed changes

sinhrks force-pushed the replace_type branch 3 times, most recently from 72d13eb to 0c7f4de Compare April 4, 2016 16:36

sinhrks mentioned this pull request Apr 9, 2016

TST: Add numeric coercion tests #12841

Closed

3 tasks

sinhrks modified the milestones: 0.19.0, 0.18.1 Apr 10, 2016

sinhrks force-pushed the replace_type branch from 0c7f4de to c7a8d3a Compare April 10, 2016 20:07

sinhrks force-pushed the replace_type branch from c7a8d3a to 5d6d0be Compare May 13, 2016 18:03

sinhrks force-pushed the replace_type branch from 5d6d0be to 1760706 Compare May 21, 2016 01:41

jorisvandenbossche modified the milestones: 0.19.0, 0.20.0 Jul 8, 2016

jreback reviewed Nov 23, 2016

View reviewed changes

sinhrks mentioned this pull request Nov 29, 2016

PERF: Improve replace perf #12745

Closed

3 tasks

sinhrks force-pushed the replace_type branch from 081d2e9 to 239c0ff Compare January 8, 2017 06:00

jreback reviewed Jan 19, 2017

View reviewed changes

BUG: replace coerces incorrect dtype

de44877

sinhrks force-pushed the replace_type branch from 206d301 to 806ebee Compare January 21, 2017 02:53

sinhrks mentioned this pull request Jan 21, 2017

BUG: Timestamp array comparison may raise RecursionError #15183

Closed

remove import failure

279fdf6

sinhrks force-pushed the replace_type branch from 806ebee to 279fdf6 Compare January 21, 2017 05:20

sinhrks changed the title ~~(WIP)BUG: replace coerces incorrect dtype~~ BUG: replace coerces incorrect dtype Jan 21, 2017

jreback added this to the 0.20.0 milestone Jan 21, 2017

remove unnecessary comments

f9154e8

sinhrks force-pushed the replace_type branch from 818ab4d to f9154e8 Compare January 23, 2017 11:16

jreback reviewed Jan 23, 2017

View reviewed changes

jreback mentioned this pull request Mar 20, 2017

BUG: replace coerces incorrect dtype #15742

Merged

jreback closed this Mar 20, 2017

This was referenced Mar 20, 2017

BUG: replace of numeric by string / dtype coversion #15743

Closed

CLN: replace _interleave_dtype with _find_common_type #15765

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: replace coerces incorrect dtype #12780

BUG: replace coerces incorrect dtype #12780

sinhrks commented Apr 3, 2016 •

edited

Loading

jreback Apr 3, 2016

sinhrks commented Apr 6, 2016

codecov-io commented May 13, 2016 •

edited

Loading

jreback commented May 13, 2016

jreback Nov 23, 2016

sinhrks Jan 21, 2017 •

edited

Loading

jreback commented Nov 23, 2016

jreback Jan 19, 2017

jreback commented Jan 19, 2017

sinhrks commented Jan 21, 2017

jreback commented Jan 21, 2017 •

edited

Loading

jreback Jan 23, 2017

sinhrks Feb 3, 2017

jreback Jan 23, 2017

jreback commented Mar 20, 2017

BUG: replace coerces incorrect dtype #12780

BUG: replace coerces incorrect dtype #12780

Conversation

sinhrks commented Apr 3, 2016 • edited Loading

jreback Apr 3, 2016

Choose a reason for hiding this comment

sinhrks commented Apr 6, 2016

codecov-io commented May 13, 2016 • edited Loading

Current coverage is 85.97% (diff: 89.65%)

jreback commented May 13, 2016

jreback Nov 23, 2016

Choose a reason for hiding this comment

sinhrks Jan 21, 2017 • edited Loading

Choose a reason for hiding this comment

jreback commented Nov 23, 2016

jreback Jan 19, 2017

Choose a reason for hiding this comment

jreback commented Jan 19, 2017

sinhrks commented Jan 21, 2017

jreback commented Jan 21, 2017 • edited Loading

jreback Jan 23, 2017

Choose a reason for hiding this comment

sinhrks Feb 3, 2017

Choose a reason for hiding this comment

jreback Jan 23, 2017

Choose a reason for hiding this comment

jreback commented Mar 20, 2017

sinhrks commented Apr 3, 2016 •

edited

Loading

codecov-io commented May 13, 2016 •

edited

Loading

sinhrks Jan 21, 2017 •

edited

Loading

jreback commented Jan 21, 2017 •

edited

Loading