DOC: Use official numpydoc extension

[FHaase]

• Replace custom numpydoc in doc/sphinxext/numpydoc with official
  numpydoc release
• Remove numpydoc_use_blockquotes parameter

[pep8speaks]

Hello @FHaase! Thanks for updating the PR.

• There are no PEP8 issues in the file doc/source/conf.py !

• There are no PEP8 issues in the file pandas/core/ops.py !

Comment last updated on December 14, 2018 at 18:25 Hours UTC

[FHaase]

@datapythonista is this what you meant in #23763 with:

avoid that numpydoc parameters are rendered inside a <blockquote> tag, breaking the formatting

Before:

After:

[TomAugspurger]

Some xref issues for context #11257, #6003, numpy/numpydoc#106 (comment).

@FHaase can you verify that those docstrings (e.g. Index.str) are rendering correctly?

[codecov]

Codecov Report

Merging #24098 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #24098   +/-   ##
=======================================
  Coverage   42.52%   42.52%           
=======================================
  Files         161      161           
  Lines       51697    51697           
=======================================
  Hits        21982    21982           
  Misses      29715    29715

Flag	Coverage Δ
#single	42.52% <ø> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d3ed91...b39f68e. Read the comment docs.

[codecov]

Codecov Report

Merging #24098 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #24098   +/-   ##
=======================================
  Coverage   92.22%   92.22%           
=======================================
  Files         162      162           
  Lines       51828    51828           
=======================================
  Hits        47798    47798           
  Misses       4030     4030

Flag	Coverage Δ
#multiple	90.62% <ø> (ø)	⬆️
#single	43% <ø> (ø)	⬆️

Impacted Files	Coverage Δ
pandas/core/ops.py	94.26% <ø> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 040f06f...afeb025. Read the comment docs.

[FHaase]

Index.str renders identical with release version of 0.23.4

We had an option to use a member_list over param_list to display the Attribute section which made
the attribute section look like the Methods section rather than the Parameter section.

The option is removed in favor of having this behavior always. So pandas.Index renders unchanged.

[FHaase]

Previously a condition existed:

            'attributes':
                self._str_param_list('Attributes', fake_autosummary=True)
                if self.attributes_as_param_list
                else self._str_member_list('Attributes'),

I assumed that self.attributes_as_param_list was always set as False

[datapythonista]

I don't understand this PR. Removing our copy of numpydoc and importing it as a dependency sounds good (not sure why was copied, I understand from the linked issues that it was because py3 compatibility?)

But what you're doing is have it as a dependency, and create an intermediate module to monkeypatch it? What is the advantage?

[datapythonista]

Regarding the blockquotes, the html wraps lists with a <blockquote>, because a list is indented, and indented code is used for a quote too. With the current style is not visible (I think we changed the css or something), but with the new theme I'm implementing the lists are rendered with a different font (because of the blockquote). See here:

[FHaase]

But what you're doing is have it as a dependency, and create an intermediate module to monkeypatch it? What is the advantage?

Moved the patching to conf.py. Didn't know that replacing a function has effect on a global level.

[datapythonista]

This looks much better. But I don't think the comment is clear enough to understand what/why we are doing this. I'd probably move it to the docstring of alternative_str and expand it (and I'd probably rename alternative_str to sphinxdocstring_str).

[FHaase]

Regarding the blockquotes, the html wraps lists with a <blockquote>, because a list is indented, and indented code is used for a quote too. With the current style is not visible (I think we changed the css or something), but with the new theme I'm implementing the lists are rendered with a different font (because of the blockquote).

@datapythonista I've compared the text in io.rst starting at line 1880 with the docutils example. When I indent by 2 spaces instead of 4 the <blockquote> doesn't get generated anymore. Can you confirm this?

If the JSON serializer cannot handle the container contents directly it will
fall back in the following manner:

* if the dtype is unsupported (e.g. ``np.complex``) then the ``default_handler``, if provided, will be called
  for each value, otherwise an exception is raised.

* if an object is unsupported it will attempt the following:


  * check if the object has defined a ``toDict`` method and call it.
    A ``toDict`` method should return a ``dict`` which will then be JSON serialized.

  * invoke the ``default_handler`` if one was provided.

  * convert the object to a ``dict`` by traversing its contents. However this will often fail
    with an ``OverflowError`` or give unexpected results.

[datapythonista]

I think the reStructuredText standard requires no indentation for first level lists, and 4 for sublists. We had some problems with the rendering in the past, you can take a look at #21520 and referenced issues, I don't remember the details.

[FHaase]

<ul>
<li><p class="first">if the dtype is unsupported (e.g. <code class="docutils literal notranslate"><span class="pre">np.complex</span></code>) then the <code class="docutils literal notranslate"><span class="pre">default_handler</span></code>, if provided, will be called
for each value, otherwise an exception is raised.</p>
</li>
<li><p class="first">if an object is unsupported it will attempt the following:</p>
<blockquote>
<div><ul class="simple">
<li>check if the object has defined a <code class="docutils literal notranslate"><span class="pre">toDict</span></code> method and call it.
A <code class="docutils literal notranslate"><span class="pre">toDict</span></code> method should return a <code class="docutils literal notranslate"><span class="pre">dict</span></code> which will then be JSON serialized.</li>
<li>invoke the <code class="docutils literal notranslate"><span class="pre">default_handler</span></code> if one was provided.</li>
<li>convert the object to a <code class="docutils literal notranslate"><span class="pre">dict</span></code> by traversing its contents. However this will often fail
with an <code class="docutils literal notranslate"><span class="pre">OverflowError</span></code> or give unexpected results.</li>
</ul>
</div></blockquote>
</li>
</ul>

vs

<ul class="simple">
<li>if the dtype is unsupported (e.g. <code class="docutils literal notranslate"><span class="pre">np.complex</span></code>) then the <code class="docutils literal notranslate"><span class="pre">default_handler</span></code>, if provided, will be called
for each value, otherwise an exception is raised.</li>
<li>if an object is unsupported it will attempt the following:<ul>
<li>check if the object has defined a <code class="docutils literal notranslate"><span class="pre">toDict</span></code> method and call it.
A <code class="docutils literal notranslate"><span class="pre">toDict</span></code> method should return a <code class="docutils literal notranslate"><span class="pre">dict</span></code> which will then be JSON serialized.</li>
<li>invoke the <code class="docutils literal notranslate"><span class="pre">default_handler</span></code> if one was provided.</li>
<li>convert the object to a <code class="docutils literal notranslate"><span class="pre">dict</span></code> by traversing its contents. However this will often fail
with an <code class="docutils literal notranslate"><span class="pre">OverflowError</span></code> or give unexpected results.</li>
</ul>
</li>
</ul>

[FHaase]

From A ReStructuredText Primer at bulleted list

* a bullet point using "*"

  - a sub-list using "-"

    + yet another sub-list

  - another item

From Sphinx Documentation

* this is
* a list

  * with a nested list
  * and some subitems

* and here the parent list continues

Both use 2 spaces as indent.

[datapythonista]

See #21518 (the problem I fixed at that time was that some top level lists were indented). Not sure where I checked the standard, but for what I saw, while sphinx is flexible, the standard is 4 spaces, and other software may fail to render the lists with a different number.

[FHaase]

You mentioned pandoc in that issue, looking at the documentation pandoc uses 2 spaces as well.

* fruits
  + apples
    - macintosh
    - red delicious
  + pears
  + peaches
* vegetables
  + broccoli
  + chard

I don't know from where you took 4 spaces as indentation for sublists, but docutils, sphinx, pandoc, ...
all seem to use 2.

pandoc uses 4 spaces for verbatim text:

A block of text indented four spaces (or one tab) is treated as verbatim text: that is, special characters do not trigger special formatting, and all spaces and line breaks are preserved. For example, ...

[datapythonista]

lgtm, thanks @FHaase

Regarding the lists indentation, I guess I was wrong, not sure where I got the information from. But let's better move the discussion to the appropriate issue, so we have it there when someone works in it.

[TomAugspurger]

cc @jorisvandenbossche.

I haven't had a chance to look at this yet. Will try to next week.

[jreback]

yeah @jorisvandenbossche needs to have a look at this. IIRC there were some rationale for using the vendored version.

[jreback]

do we have a min version required?

[FHaase]

I don't know exactly. Currently it uses the latest which is 0.8.0.

[jorisvandenbossche]

I am certainly fine with monkeypatching upstream numpydoc, instead of vendoring the full package as we do now. That makes it a bit more explicit which hacks we added / where the changes are compared to upstream.

I think in the past we had more hacks than just the attributes listing, but it might be that the others already have been solved in the meantime.
I quickly looked at the build log, and we have a whole bunch of warnings related to numpydoc / autodoc, but we also have them on master, so this PR does not seem to add new warnings.

[jorisvandenbossche]

Reading through #20383, it seems I have updated our vendored version to the upstream one earlier this year, with the attribute listing as the only 'hack' afterwards. And in that way removing the patches we had previously (#11257)

There is a note there (#20383 (comment)) about the numpydoc_use_blockquotes and **kwargs in the docstring. @FHaase this is still working now? (as I see you removed that setting; I don't really remember directly why it was needed at the time)

We had an option to use a member_list over param_list to display the Attribute section which made
the attribute section look like the Methods section rather than the Parameter section.

The option is removed in favor of having this behavior always.

@FHaase yes, fine to remove that. I originally did it like that in the hope to push that change upstream (for which there is a PR: numpy/numpydoc#161)

[jreback]

@jorisvandenbossche over to you; merge when satisfied

[FHaase]

There are two pictures added above. With a before and after comparison.
They feature a *args and **kwargs in parameters section. Which does look fine to me.

The vendored version was 0.8.0.dev0 (something like that) now it's 0.8.0. I guess they fixed all those issues for the release version.

[FHaase]

Patched numpy/numpydoc#150 into SphinxDocString.
Running python make.py html --single api doesn't generate any WARNING: Inline strong start-string without end-string.

Happened those issues were cut off after all the Warnings regarding the toctree were shown.

[jorisvandenbossche]

@FHaase you have one linting error: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=5106

[jreback]

lgtm. can you merge master and ping on green. any issues closed by this?

[FHaase]

Ping @jreback

[jreback]

https://travis-ci.org/pandas-dev/pandas/jobs/468121566

this failed building docs, and didn't fail the jobs: @datapythonista

[datapythonista]

I saw that couple of days ago working in some fixes to the doc build. We're using os.system() to call sphinx-build, and it returns the status code instead of raising an exception when the called command fails.

That will be fixed in the changes I'm doing, hopefully I'll be able to open a PR very soon, just need to fix some problems I'm getting with sphinx-autogen.

[jreback]

ok, but it appears our docs are completely broken now. so let's fix that before merging anything else.

[datapythonista]

I thought the failure was in this PR. Let me take a look then.

[jreback]

http://pandas-docs.github.io/pandas-docs-travis/

[datapythonista]

locally it fails because of the enchant (spellcheck) thing. Not sure if that's the only problem, as the error in the CI is different, but I'll get rid of that first, as it's something I wanted to do anyway (it can't be used, as the dependencies can't be installed)

[datapythonista]

I've got a different error locally, but I think something is wrong in my system (I'm getting pytest segfaults in an unrelated branch).

I opened #24287 which with some luck fixed the problem. And in any case should let us know if the doc build fails.

[jreback]

thanks @FHaase