Merge remote-tracking branch 'upstream/master' into io_csv_docstring_…

…fixed

* upstream/master:
  DOC: more consistent flake8-commands in contributing.rst (pandas-dev#23724)
  DOC: Fixed the doctsring for _set_axis_name (GH 22895) (pandas-dev#22969)
  DOC: Improve GL03 message re: blank lines at end of docstrings. (pandas-dev#23649)
  TST: add tests for keeping dtype in Series.update (pandas-dev#23604)
  TST: For GH4861, Period and datetime in multiindex (pandas-dev#23776)
  TST: move .str-test to strings.py & parametrize it; precursor to pandas-dev#23582 (pandas-dev#23777)
  STY: isort tests/scalar, tests/tslibs, import libwindow instead of _window (pandas-dev#23787)
  BUG: fixed .str.contains(..., na=False) for categorical series (pandas-dev#22170)
  BUG: Maintain column order with groupby.nth (pandas-dev#22811)
  API/DEPR: replace kwarg "pat" with "sep" in str.[r]partition (pandas-dev#23767)
  CLN: Finish isort core (pandas-dev#23765)
  TST: Mark test_pct_max_many_rows with pytest.mark.single (pandas-dev#23799)

Makefile

@@ -13,7 +13,7 @@ build: clean_pyc
 	python setup.py build_ext --inplace
 
 lint-diff:
-	git diff master --name-only -- "*.py" | grep -E "pandas|scripts" | xargs flake8
+	git diff upstream/master --name-only -- "*.py" | xargs flake8
 
 develop: build
 	-python setup.py develop

doc/source/contributing.rst

@@ -575,7 +575,7 @@ the `flake8 <https://pypi.org/project/flake8>`_ tool
 and report any stylistic errors in your code. Therefore, it is helpful before
 submitting code to run the check yourself on the diff::
 
-   git diff master -u -- "*.py" | flake8 --diff
+   git diff upstream/master -u -- "*.py" | flake8 --diff
 
 This command will catch any stylistic errors in your changes specifically, but
 be beware it may not catch all of them. For example, if you delete the only
@@ -584,21 +584,21 @@ unused function. However, style-checking the diff will not catch this because
 the actual import is not part of the diff. Thus, for completeness, you should
 run this command, though it will take longer::
 
-   git diff master --name-only -- "*.py" | grep "pandas/" | xargs -r flake8
+   git diff upstream/master --name-only -- "*.py" | xargs -r flake8
 
 Note that on OSX, the ``-r`` flag is not available, so you have to omit it and
 run this slightly modified command::
 
-   git diff master --name-only -- "*.py" | grep "pandas/" | xargs flake8
+   git diff upstream/master --name-only -- "*.py" | xargs flake8
 
-Windows does not support the ``grep`` and ``xargs`` commands (unless installed
-for example via the `MinGW <http://www.mingw.org/>`__ toolchain), but one can
-imitate the behaviour as follows::
+Windows does not support the ``xargs`` command (unless installed for example
+via the `MinGW <http://www.mingw.org/>`__ toolchain), but one can imitate the
+behaviour as follows::
 
-    for /f %i in ('git diff upstream/master --name-only ^| findstr pandas/') do flake8 %i
+    for /f %i in ('git diff upstream/master --name-only -- "*.py"') do flake8 %i
 
-This will also get all the files being changed by the PR (and within the
-``pandas/`` folder), and run ``flake8`` on them one after the other.
+This will get all the files being changed by the PR (and ending with ``.py``),
+and run ``flake8`` on them, one after the other.
 
 .. _contributing.import-formatting:
 

doc/source/whatsnew/v0.24.0.rst

@@ -288,6 +288,7 @@ Other Enhancements
 - Added :meth:`Interval.overlaps`, :meth:`IntervalArray.overlaps`, and :meth:`IntervalIndex.overlaps` for determining overlaps between interval-like objects (:issue:`21998`)
 - :func:`~DataFrame.to_parquet` now supports writing a ``DataFrame`` as a directory of parquet files partitioned by a subset of the columns when ``engine = 'pyarrow'`` (:issue:`23283`)
 - :meth:`Timestamp.tz_localize`, :meth:`DatetimeIndex.tz_localize`, and :meth:`Series.tz_localize` have gained the ``nonexistent`` argument for alternative handling of nonexistent times. See :ref:`timeseries.timezone_nonexistent` (:issue:`8917`)
+- :meth:`Index.difference` now has an optional ``sort`` parameter to specify whether the results should be sorted if possible (:issue:`17839`)
 - :meth:`read_excel()` now accepts ``usecols`` as a list of column names or callable (:issue:`18273`)
 - :meth:`MultiIndex.to_flat_index` has been added to flatten multiple levels into a single-level :class:`Index` object.
 - :meth:`DataFrame.to_stata` and :class:` pandas.io.stata.StataWriter117` can write mixed sting columns to Stata strl format (:issue:`23633`)
@@ -1035,6 +1036,7 @@ Deprecations
 - :meth:`Timestamp.tz_localize`, :meth:`DatetimeIndex.tz_localize`, and :meth:`Series.tz_localize` have deprecated the ``errors`` argument in favor of the ``nonexistent`` argument (:issue:`8917`)
 - The class ``FrozenNDArray`` has been deprecated. When unpickling, ``FrozenNDArray`` will be unpickled to ``np.ndarray`` once this class is removed (:issue:`9031`)
 - The methods :meth:`DataFrame.update` and :meth:`Panel.update` have deprecated the ``raise_conflict=False|True`` keyword in favor of ``errors='ignore'|'raise'`` (:issue:`23585`)
+- The methods :meth:`Series.str.partition` and :meth:`Series.str.rpartition` have deprecated the ``pat`` keyword in favor of ``sep`` (:issue:`22676`)
 - Deprecated the `nthreads` keyword of :func:`pandas.read_feather` in favor of
   `use_threads` to reflect the changes in pyarrow 0.11.0. (:issue:`23053`)
 - :func:`pandas.read_excel` has deprecated accepting ``usecols`` as an integer. Please pass in a list of ints from 0 to ``usecols`` inclusive instead (:issue:`23527`)
@@ -1278,7 +1280,7 @@ Strings
 
 - Bug in :meth:`Index.str.partition` was not nan-safe (:issue:`23558`).
 - Bug in :meth:`Index.str.split` was not nan-safe (:issue:`23677`).
--
+- Bug :func:`Series.str.contains` not respecting the ``na`` argument for a ``Categorical`` dtype ``Series`` (:issue:`22158`)
 
 Interval
 ^^^^^^^^
@@ -1416,6 +1418,7 @@ Groupby/Resample/Rolling
 - Bug in :meth:`DataFrame.resample` and :meth:`Series.resample` when resampling by a weekly offset (``'W'``) across a DST transition (:issue:`9119`, :issue:`21459`)
 - Bug in :meth:`DataFrame.expanding` in which the ``axis`` argument was not being respected during aggregations (:issue:`23372`)
 - Bug in :meth:`pandas.core.groupby.DataFrameGroupBy.transform` which caused missing values when the input function can accept a :class:`DataFrame` but renames it (:issue:`23455`).
+- Bug in :func:`pandas.core.groupby.GroupBy.nth` where column order was not always preserved (:issue:`20760`)
 
 Reshaping
 ^^^^^^^^^

pandas/core/arrays/base.py

@@ -5,16 +5,18 @@
    This is an experimental API and subject to breaking changes
    without warning.
 """
-import numpy as np
-
 import operator
 
-from pandas.core.dtypes.generic import ABCSeries, ABCIndexClass
-from pandas.errors import AbstractMethodError
+import numpy as np
+
+from pandas.compat import PY3, set_function_name
 from pandas.compat.numpy import function as nv
-from pandas.compat import set_function_name, PY3
-from pandas.core import ops
+from pandas.errors import AbstractMethodError
+
 from pandas.core.dtypes.common import is_list_like
+from pandas.core.dtypes.generic import ABCIndexClass, ABCSeries
+
+from pandas.core import ops
 
 _not_implemented_message = "{} does not implement {}."
 

pandas/core/arrays/categorical.py

@@ -1,61 +1,46 @@
 # pylint: disable=E1101,W0232
 
-import numpy as np
-from warnings import warn
 import textwrap
+from warnings import warn
 
-from pandas import compat
-from pandas.compat import u, lzip
-from pandas._libs import lib, algos as libalgos
+import numpy as np
+
+from pandas._libs import algos as libalgos, lib
+import pandas.compat as compat
+from pandas.compat import lzip, u
+from pandas.compat.numpy import function as nv
+from pandas.util._decorators import (
+    Appender, Substitution, cache_readonly, deprecate_kwarg)
+from pandas.util._validators import validate_bool_kwarg, validate_fillna_kwargs
 
-from pandas.core.dtypes.generic import (
-    ABCSeries, ABCIndexClass, ABCCategoricalIndex)
-from pandas.core.dtypes.missing import isna, notna
-from pandas.core.dtypes.inference import is_hashable
 from pandas.core.dtypes.cast import (
-    maybe_infer_to_datetimelike,
-    coerce_indexer_dtype)
-from pandas.core.dtypes.dtypes import CategoricalDtype
+    coerce_indexer_dtype, maybe_infer_to_datetimelike)
 from pandas.core.dtypes.common import (
-    ensure_int64,
-    ensure_object,
-    ensure_platform_int,
-    is_extension_array_dtype,
-    is_dtype_equal,
-    is_datetimelike,
-    is_datetime64_dtype,
-    is_timedelta64_dtype,
-    is_categorical,
-    is_categorical_dtype,
-    is_float_dtype,
-    is_integer_dtype,
-    is_object_dtype,
-    is_list_like, is_sequence,
-    is_scalar, is_iterator,
-    is_dict_like)
-
-from pandas.core.algorithms import factorize, take_1d, unique1d, take
+    ensure_int64, ensure_object, ensure_platform_int, is_categorical,
+    is_categorical_dtype, is_datetime64_dtype, is_datetimelike, is_dict_like,
+    is_dtype_equal, is_extension_array_dtype, is_float_dtype, is_integer_dtype,
+    is_iterator, is_list_like, is_object_dtype, is_scalar, is_sequence,
+    is_timedelta64_dtype)
+from pandas.core.dtypes.dtypes import CategoricalDtype
+from pandas.core.dtypes.generic import (
+    ABCCategoricalIndex, ABCIndexClass, ABCSeries)
+from pandas.core.dtypes.inference import is_hashable
+from pandas.core.dtypes.missing import isna, notna
+
 from pandas.core.accessor import PandasDelegate, delegate_names
-from pandas.core.base import (PandasObject,
-                              NoNewAttributesMixin, _shared_docs)
+import pandas.core.algorithms as algorithms
+from pandas.core.algorithms import factorize, take, take_1d, unique1d
+from pandas.core.base import NoNewAttributesMixin, PandasObject, _shared_docs
 import pandas.core.common as com
+from pandas.core.config import get_option
 from pandas.core.missing import interpolate_2d
-from pandas.compat.numpy import function as nv
-from pandas.util._decorators import (
-    Appender, cache_readonly, deprecate_kwarg, Substitution)
-
-import pandas.core.algorithms as algorithms
-
 from pandas.core.sorting import nargsort
 
 from pandas.io.formats import console
 from pandas.io.formats.terminal import get_terminal_size
-from pandas.util._validators import validate_bool_kwarg, validate_fillna_kwargs
-from pandas.core.config import get_option
 
 from .base import ExtensionArray
 
-
 _take_msg = textwrap.dedent("""\
     Interpreting negative values in 'indexer' as missing values.
     In the future, this will change to meaning positional indices

pandas/core/arrays/datetimelike.py

@@ -5,44 +5,33 @@
 
 import numpy as np
 
-from pandas._libs import lib, iNaT, NaT
+from pandas._libs import NaT, iNaT, lib
 from pandas._libs.tslibs import timezones
-from pandas._libs.tslibs.timedeltas import delta_to_nanoseconds, Timedelta
-from pandas._libs.tslibs.timestamps import maybe_integer_op_deprecated
 from pandas._libs.tslibs.period import (
-    Period, DIFFERENT_FREQ_INDEX, IncompatibleFrequency)
-
+    DIFFERENT_FREQ_INDEX, IncompatibleFrequency, Period)
+from pandas._libs.tslibs.timedeltas import Timedelta, delta_to_nanoseconds
+from pandas._libs.tslibs.timestamps import maybe_integer_op_deprecated
+import pandas.compat as compat
 from pandas.errors import (
     AbstractMethodError, NullFrequencyError, PerformanceWarning)
-from pandas import compat
-
-from pandas.tseries import frequencies
-from pandas.tseries.offsets import Tick, DateOffset
+from pandas.util._decorators import deprecate_kwarg
 
 from pandas.core.dtypes.common import (
-    pandas_dtype,
-    needs_i8_conversion,
-    is_list_like,
-    is_offsetlike,
-    is_extension_array_dtype,
-    is_datetime64_dtype,
-    is_datetime64_any_dtype,
-    is_datetime64tz_dtype,
-    is_float_dtype,
-    is_integer_dtype,
-    is_bool_dtype,
-    is_period_dtype,
-    is_timedelta64_dtype,
-    is_object_dtype)
-from pandas.core.dtypes.generic import ABCSeries, ABCDataFrame, ABCIndexClass
+    is_bool_dtype, is_datetime64_any_dtype, is_datetime64_dtype,
+    is_datetime64tz_dtype, is_extension_array_dtype, is_float_dtype,
+    is_integer_dtype, is_list_like, is_object_dtype, is_offsetlike,
+    is_period_dtype, is_timedelta64_dtype, needs_i8_conversion, pandas_dtype)
 from pandas.core.dtypes.dtypes import DatetimeTZDtype
+from pandas.core.dtypes.generic import ABCDataFrame, ABCIndexClass, ABCSeries
 from pandas.core.dtypes.missing import isna
 
-import pandas.core.common as com
 from pandas.core.algorithms import checked_add_with_arr, take, unique1d
+import pandas.core.common as com
+
+from pandas.tseries import frequencies
+from pandas.tseries.offsets import DateOffset, Tick
 
 from .base import ExtensionOpsMixin
-from pandas.util._decorators import deprecate_kwarg
 
 
 def _make_comparison_op(cls, op):

pandas/core/arrays/datetimes.py

@@ -6,35 +6,28 @@
 from pytz import utc
 
 from pandas._libs import lib, tslib
-from pandas._libs.tslib import Timestamp, NaT, iNaT
+from pandas._libs.tslib import NaT, Timestamp, iNaT
 from pandas._libs.tslibs import (
-    ccalendar, normalize_date,
-    conversion, fields, timezones,
-    resolution as libresolution)
-
-from pandas.util._decorators import cache_readonly, Appender
+    ccalendar, conversion, fields, normalize_date, resolution as libresolution,
+    timezones)
+import pandas.compat as compat
 from pandas.errors import PerformanceWarning
-from pandas import compat
+from pandas.util._decorators import Appender, cache_readonly
 
 from pandas.core.dtypes.common import (
-    _NS_DTYPE,
-    is_object_dtype,
-    is_int64_dtype,
-    is_datetime64tz_dtype,
-    is_datetime64_dtype)
+    _NS_DTYPE, is_datetime64_dtype, is_datetime64tz_dtype, is_int64_dtype,
+    is_object_dtype)
 from pandas.core.dtypes.dtypes import DatetimeTZDtype
-from pandas.core.dtypes.missing import isna
 from pandas.core.dtypes.generic import ABCIndexClass, ABCSeries
+from pandas.core.dtypes.missing import isna
 
-import pandas.core.common as com
-from pandas.core.algorithms import checked_add_with_arr
 from pandas.core import ops
-
-from pandas.tseries.frequencies import to_offset, get_period_alias
-from pandas.tseries.offsets import Tick, generate_range
-
+from pandas.core.algorithms import checked_add_with_arr
 from pandas.core.arrays import datetimelike as dtl
+import pandas.core.common as com
 
+from pandas.tseries.frequencies import get_period_alias, to_offset
+from pandas.tseries.offsets import Tick, generate_range
 
 _midnight = time(0, 0)
 

pandas/core/arrays/integer.py

@@ -1,31 +1,27 @@
+import copy
 import sys
 import warnings
-import copy
-import numpy as np
 
+import numpy as np
 
 from pandas._libs import lib
+from pandas.compat import range, set_function_name, string_types, u
 from pandas.util._decorators import cache_readonly
-from pandas.compat import u, range, string_types
-from pandas.compat import set_function_name
 
-from pandas.core import nanops
+from pandas.core.dtypes.base import ExtensionDtype
 from pandas.core.dtypes.cast import astype_nansafe
-from pandas.core.dtypes.generic import ABCSeries, ABCIndexClass
 from pandas.core.dtypes.common import (
-    is_integer, is_scalar, is_float,
-    is_bool_dtype,
-    is_float_dtype,
-    is_integer_dtype,
-    is_object_dtype,
-    is_list_like)
-from pandas.core.arrays import ExtensionArray, ExtensionOpsMixin
-from pandas.core.dtypes.base import ExtensionDtype
+    is_bool_dtype, is_float, is_float_dtype, is_integer, is_integer_dtype,
+    is_list_like, is_object_dtype, is_scalar)
 from pandas.core.dtypes.dtypes import register_extension_dtype
+from pandas.core.dtypes.generic import ABCIndexClass, ABCSeries
 from pandas.core.dtypes.missing import isna, notna
 
+from pandas.core import nanops
+from pandas.core.arrays import ExtensionArray, ExtensionOpsMixin
+
 from pandas.io.formats.printing import (
-    format_object_summary, format_object_attrs, default_pprint)
+    default_pprint, format_object_attrs, format_object_summary)
 
 
 class _IntegerDtype(ExtensionDtype):

pandas/core/arrays/interval.py

@@ -1,31 +1,30 @@
+from operator import le, lt
 import textwrap
-import numpy as np
 
-from operator import le, lt
+import numpy as np
 
-from pandas._libs.interval import (Interval, IntervalMixin,
-                                   intervals_to_interval_bounds)
+from pandas._libs.interval import (
+    Interval, IntervalMixin, intervals_to_interval_bounds)
 from pandas.compat import add_metaclass
 from pandas.compat.numpy import function as nv
-import pandas.core.common as com
-from pandas.core.config import get_option
+from pandas.util._decorators import Appender
+from pandas.util._doctools import _WritableDoc
+
 from pandas.core.dtypes.cast import maybe_convert_platform
-from pandas.core.dtypes.common import (is_categorical_dtype, is_float_dtype,
-                                       is_integer_dtype, is_interval_dtype,
-                                       is_scalar, is_string_dtype,
-                                       is_datetime64_any_dtype,
-                                       is_timedelta64_dtype, is_interval,
-                                       pandas_dtype)
+from pandas.core.dtypes.common import (
+    is_categorical_dtype, is_datetime64_any_dtype, is_float_dtype,
+    is_integer_dtype, is_interval, is_interval_dtype, is_scalar,
+    is_string_dtype, is_timedelta64_dtype, pandas_dtype)
 from pandas.core.dtypes.dtypes import IntervalDtype
-from pandas.core.dtypes.generic import (ABCDatetimeIndex, ABCPeriodIndex,
-                                        ABCSeries, ABCIntervalIndex,
-                                        ABCInterval)
+from pandas.core.dtypes.generic import (
+    ABCDatetimeIndex, ABCInterval, ABCIntervalIndex, ABCPeriodIndex, ABCSeries)
 from pandas.core.dtypes.missing import isna, notna
+
+import pandas.core.common as com
+from pandas.core.config import get_option
 from pandas.core.indexes.base import Index, ensure_index
-from pandas.util._decorators import Appender
-from pandas.util._doctools import _WritableDoc
 
-from . import ExtensionArray, Categorical
+from . import Categorical, ExtensionArray
 
 _VALID_CLOSED = {'left', 'right', 'both', 'neither'}
 _interval_shared_docs = {}