Safer is dtype

[TomAugspurger]

Closes #22960

[pep8speaks]

Hello @TomAugspurger! Thanks for updating the PR.

• There are no PEP8 issues in the file pandas/core/dtypes/base.py !

• There are no PEP8 issues in the file pandas/core/frame.py !

• There are no PEP8 issues in the file pandas/core/series.py !

• There are no PEP8 issues in the file pandas/tests/dtypes/test_dtypes.py !

• There are no PEP8 issues in the file pandas/tests/frame/test_operators.py !

Comment last updated on October 03, 2018 at 20:20 Hours UTC

[TomAugspurger]

@jreback @pambot thoughts on broadening the caught exceptions in https://github.com/pandas-dev/pandas/pull/22975/files#diff-8c2f3d0cbd1523cafd79c45e87124103R100?. According to the docs, only TypeError is "allowed" to be raised, but python has no way to enforce that. I'm not sure what's best here.

[TomAugspurger]

Failure is from #22019 (more evidence that moving the numpydev warnings build to the main section is a good idea).

I'm going to include a fix for that here.

[codecov]

Codecov Report

Merging #22975 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #22975      +/-   ##
==========================================
+ Coverage   92.19%   92.19%   +<.01%     
==========================================
  Files         169      169              
  Lines       50821    50825       +4     
==========================================
+ Hits        46852    46857       +5     
+ Misses       3969     3968       -1

Flag	Coverage Δ
#multiple	90.61% <100%> (ø)	⬆️
#single	42.36% <60%> (-0.01%)	⬇️

Impacted Files	Coverage Δ
pandas/core/dtypes/base.py	100% <100%> (ø)	⬆️
pandas/core/series.py	93.75% <100%> (ø)	⬆️
pandas/core/frame.py	97.2% <100%> (ø)	⬆️
pandas/util/_depr_module.py	65.11% <0%> (-2.33%)	⬇️
pandas/core/resample.py	96.98% <0%> (+0.01%)	⬆️
pandas/core/groupby/grouper.py	98.16% <0%> (+0.73%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fea27f0...073df47. Read the comment docs.

[TomAugspurger]

cc @jbrockmendel for the ops changes to catch that warning from numpy. Does https://github.com/pandas-dev/pandas/pull/22975/files#diff-1e79abbbdd150d4771b91ea60a4e1cc7R4911 look like the right place to do it? Looking at the call stack, we can do it in either _combine_match_index or _combine_series_frame Given that combine_match_index dispatches to series (which I think may go to EA ops), I don't think we should be catching NumPy warnings there. I'd rather leave that up to the EA authors.

[TomAugspurger]

Darn, we're back to the failure in test_ndarray_compat, which I cannot reproduce locally. https://travis-ci.org/pandas-dev/pandas/jobs/436672887#L2765

______________________ TestSeriesMisc.test_ndarray_compat ______________________
[gw1] linux -- Python 3.7.0 /home/travis/miniconda3/envs/pandas/bin/python
self = <pandas.tests.series.test_api.TestSeriesMisc object at 0x7f94a64fdbe0>
    def test_ndarray_compat(self):
    
        # test numpy compat with Series as sub-class of NDFrame
        tsdf = DataFrame(np.random.randn(1000, 3), columns=['A', 'B', 'C'],
                         index=date_range('1/1/2000', periods=1000))
    
        def f(x):
            return x[x.idxmax()]
    
>       result = tsdf.apply(f)
pandas/tests/series/test_api.py:405: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/core/frame.py:6240: in apply
    return op.get_result()
pandas/core/apply.py:151: in get_result
    return self.apply_standard()
pandas/core/apply.py:245: in apply_standard
    dummy = Series(empty_arr, index=index, dtype=values.dtype)
pandas/core/series.py:277: in __init__
    raise_cast_failure=True)
pandas/core/series.py:4270: in _sanitize_array
    subarr = _try_cast(data, True)
pandas/core/series.py:4228: in _try_cast
    subarr = maybe_cast_to_integer_array(arr, dtype)
pandas/core/dtypes/cast.py:1350: in maybe_cast_to_integer_array
    if np.array_equal(arr, casted):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
a1 = array([ 6.93066801e-310,  6.93066801e-310,  4.63759004e-310,
        4.63759004e-310,  6.93058871e-310, -3.80607598e+0...93066821e-310, -9.21064254e+018,
        6.93058666e-310, -3.04661570e+290,  0.00000000e+000,
        0.00000000e+000])
a2 = array([ 6.93066801e-310,  6.93066801e-310,  4.63759004e-310,
        4.63759004e-310,  6.93058871e-310, -3.80607598e+0...93066821e-310, -9.21064254e+018,
        6.93058666e-310, -3.04661570e+290,  0.00000000e+000,
        0.00000000e+000])
    def array_equal(a1, a2):
        """
        True if two arrays have the same shape and elements, False otherwise.
    
        Parameters
        ----------
        a1, a2 : array_like
            Input arrays.
    
        Returns
        -------
        b : bool
            Returns True if the arrays are equal.
    
        See Also
        --------
        allclose: Returns True if two arrays are element-wise equal within a
                  tolerance.
        array_equiv: Returns True if input arrays are shape consistent and all
                     elements equal.
    
        Examples
        --------
        >>> np.array_equal([1, 2], [1, 2])
        True
        >>> np.array_equal(np.array([1, 2]), np.array([1, 2]))
        True
        >>> np.array_equal([1, 2], [1, 2, 3])
        False
        >>> np.array_equal([1, 2], [1, 4])
        False
    
        """
        try:
            a1, a2 = asarray(a1), asarray(a2)
        except Exception:
            return False
        if a1.shape != a2.shape:
            return False
>       return bool(asarray(a1 == a2).all())
E       DeprecationWarning: elementwise comparison failed; this will raise an error 

[jbrockmendel]

[...] look like the right place to do it?

I think it should be higher up the stack where func is defined.

[TomAugspurger]

You're thinking something like this?

def func2(...):
    with np.errstate(all='ignore'):
        func(...)

and func2 is whats actually passed down through the chain.

[jbrockmendel]

You're thinking something like this?

Close. _combine_match_index only gets called from one place in _combine_series_frame. _combine_series_frame gets called from 3 places. I'd want to track the problem back and see which of these paths is actually relevant. i.e. do this particular wrapping as early as possible so as to wrap as little as possible.

Does the problem show up exclusively in comparison(DataFrame, Series) ops? That would be the simplest explanation with the cleanest fix.

[TomAugspurger]

Will take a look at who hits this.

jumping back to the other failure, this seems strange. The error is coming in FrameApply.apply_standard when we do dummy = Series(empty_arr, index=index, dtype=values.dtype).

Eventually, we get to _sanitize_array, which calls maybe_cast_to_integer_dtype. But why the if is_float_dtype(dtype)?

In [1]: import pandas as pd

In [2]: pd.Series([1., 2], dtype=float)
> /Users/taugspurger/sandbox/pandas-travis-37-numpydev/pandas/core/series.py(4229)_try_cast()
-> subarr = maybe_cast_to_integer_array(arr, dtype)
(Pdb) l
4224            try:
4225                # gh-15832: Check if we are requesting a numeric dype and
4226                # that we can convert the data to the requested dtype.
4227                if is_float_dtype(dtype) or is_integer_dtype(dtype):
4228                    breakpoint()
4229 ->                 subarr = maybe_cast_to_integer_array(arr, dtype)
4230
4231                subarr = maybe_cast_to_datetime(arr, dtype)
4232                # Take care in creating object arrays (but iterators are not
4233                # supported):
4234                if is_object_dtype(dtype) and (is_list_like(subarr) and
(Pdb) dtype
dtype('float64')

cc @gfyoung that's from #21456, if you recall.

[TomAugspurger]

@jbrockmendel updated with what should be a more focused filter, though I'm not sure. https://github.com/pandas-dev/pandas/pull/22975/files#diff-f38e1e99cb32831f13ced48de4220eacR141

AFAICT, the only warning leaking through was truediv when the RHS had 0s. So I replaced operator.truediv with a wrapper that sets the errstate.

Compared to my earlier commit, I think the difference is that

1. The errstate is applied just to truediv and rtruediv, not every function that reached that else block in _combine_match_index.
2. The errstate is applied to truediv, regardless of which path it takes. In particular, ops.dispatch_to_series is also called with the silent_truediv function. That ends up not mattering here, since another errstate is applied down in expressions._evaluate_standard.

[TomAugspurger]

I'm not sure what was wrong with
d88929f, but there were warnings coming through everywhere. Gone with the original approach of all was calling func with the errstate set. That matches what we eventually do for series anyway.

[jbrockmendel]

Gone with the original approach of all was calling func with the errstate set.

OK. It's not my favorite thing, but I can live with it.

[jreback]

is there a reason you are removing this job?

[TomAugspurger]

I'm removing it from the allow_failures section.

[jreback]

oh ok, good

[TomAugspurger]

Any other thoughts @jreback, or are we good to merge?

[jreback]

thanks!