Make DataFrame.to_string output full content by default

[lshepard]

I modeled this off of #24841. Some alternatives I considered:

• Instead of setting the option_context here, we could wind the param into the depths of the formatter. I tried this, actually, and started finding a number of edge cases and bugs. I realized that the issue only occurs in a pretty narrow case - if the user is explicitly calling to_string - because most of the time, when representing a DataFrame, the user will want long strings truncated for readability. So I think the safest way is to do it at the top level without interfering with lower-level formatters.
• Series.to_string() could arguably benefit from the same treatment, although that wasn't mentioned in the original issue (and I have never found the need to use it personally) so I didn't bring that in.

Here's an example on a real dataset showing long columns preserved in a text file produced by to_string():

Additional manual testing:

• Main use case- by default, no limits and ignores the display options, but can still override:

>>> print(df.to_string())
        A       B
0     NaN     NaN
1 -1.0000     foo
2 -2.1234   foooo
3  3.0000  fooooo
4  4.0000     bar
>>> with option_context('display.max_colwidth', 5):
...     print(df.to_string())
... 
        A       B
0     NaN     NaN
1 -1.0000     foo
2 -2.1234   foooo
3  3.0000  fooooo
4  4.0000     bar
>>> print(df.to_string(max_colwidth=5))
      A     B
0   NaN   NaN
1 -1...   foo
2 -2...  f...
3  3...  f...
4  4...   bar

• The string representation of DataFrame does still use the display options (so it's only the explicit to_string that doesn't:

>>> with option_context('display.max_colwidth', 5):
...     print(str(df))
... 
      A     B
0   NaN   NaN
1 -1...   foo
2 -2...  f...
3  3...  f...
4  4...   bar

• The new parameter validates for None and positive ints, but rejects anything else:

>>> print(df.to_string(max_colwidth=-5))
    ...
    raise ValueError(msg)
ValueError: Value must be a nonnegative integer or None

• closes DataFrame.to_string truncates long strings #9784
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[pep8speaks]

Hello @lshepard! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-09-14 14:09:02 UTC

[lshepard]

Wanted to note - one of the commenters on the issue asks: "With Pandas 0.25.0, setting display.max_colwidth to a large number stops the truncation but when trying to left justify columns with df.to_string(justify='left'), that same display setting somehow pads columns on the left so they are not left aligned. Is there any present way to prevent truncation and get left justified string columns when output to a terminal? I know a pull request is in process but I would like to do this now. Thanks."

I can accomplish this in my testing by setting max_colwidth=0, which switches the padding to left. It is weird, though, that passing justify="left" does not justify it correctly. Seems like maybe a separate bug or one that I could look further into.

[TomAugspurger]

I'm not sure I understand this cause, but I don't think we'll want a workaround like this.

Can you post the output with max_colwidth=0, and how it compares to this?

[lshepard]

This shows how the max_colwidth=0 strangely forces a left justification:

>>> print(df.to_string(max_colwidth=0))
        A       B
0 NaN      NaN   
1 -1.0000  foo   
2 -2.1234  foooo 
3  3.0000  fooooo
4  4.0000  bar   

Whereas the behavior in this PR preserves the right justification that is the current default:

>>> print(df.to_string(max_colwidth=99999))
        A       B
0     NaN     NaN
1 -1.0000     foo
2 -2.1234   foooo
3  3.0000  fooooo
4  4.0000     bar

And interestingly, passing justify='left' doesn't have an effect:

>>> print(df.to_string(justify='left'))
   A      B      
0     NaN     NaN
1 -1.0000     foo
2 -2.1234   foooo
3  3.0000  fooooo
4  4.0000     bar

This is because of these lines in _make_fixed_width:

def _make_fixed_width(
    ...
    max_len = max(adj.len(x) for x in strings)
    ...
    def just(x):
        if conf_max is not None:
            if (conf_max > 3) & (adj.len(x) > max_len):
                x = x[: max_len - 3] + "..."
        return x

    strings = [just(x) for x in strings]
    result = adj.justify(strings, max_len, mode=justify)
    return result

It checks if conf_max > 3 to apply the dot truncation ... so if it's <= 3 then that isn't called. So it's not just 0 but any of 0, 1, 2, or 3 that causes the justification to line up.

I can spend some more time to better understand why this is happening. I agree that we should not rely on some incidentals of the underlying implementation to determine whether to justify the text.

[lshepard]

Ok, I figured out the issue. There are two lines where format_array is called and the justify parameter is not passed all the way through -- so in some places, the justification is being overridden.

Note that the bug where justification doesn't happen if conf_max < 3 already appears - so I think it can probably be pulled out as a separate PR.

[lshepard]

Scratch all that. I re-read the docs and I see that I misinterpreted the justify param, as it only refers to the column headers, not the content. In that regard it is behaving correctly. So I think I'll leave the justification question out of this PR.

[TomAugspurger]

Thanks.

I think we'll need to deprecate the current behavior, rather than just changing the default. Users will need to be explicit about the max_colwidth for now I think.

[TomAugspurger]

I'm not sure I understand this cause, but I don't think we'll want a workaround like this.

Can you post the output with max_colwidth=0, and how it compares to this?

[lshepard]

I think we'll need to deprecate the current behavior, rather than just changing the default.
Users will need to be explicit about the max_colwidth for now I think.

To make sure I understand, what should happen when the user calls df.to_string() without parameters?

• Limits columns to the display.max_colwidth config option (the current default)
• No limit to column width (this PR)
• Make max_colwidth a required parameter (so vanilla to_string() no longer works)
• One of the above, but with a deprecation warning

Sounds like you're suggesting that we continue the current behavior, but with a deprecation warning? Should we only sound the deprecation warning if the data frame contains columns that would have otherwise been truncated? Seems like in most cases, the truncation won't be a difference in behavior and I would hate to make vanilla df.to_string() not work...

[lshepard]

To update: I chose to punt the justify question as I think that's a separate issue that pre-existed and I may not understand the use case that well anyway.

I changed the behavior of max_colwidth so that it will use None to mean unlimited instead of the sentinel of 999999. I also changed the other two places from where I had copied the style (in the clipboard formatter & the html formatter).

I have not yet added a deprecation warning - happy to do so if you think desired behavior is that the max_colwidth param should be required in the future. That doesn't make as much sense to me.

[TomAugspurger]

To make sure I understand, what should happen when the user calls df.to_string() without parameters?

That needs to be decided. It seems that in #28052 we decided that to_html not outputting the entire dataframe was a bug. Presumably we would say the same about this, but I'm not sure.

cc @simonjayhawkins

[simonjayhawkins]

To make sure I understand, what should happen when the user calls df.to_string() without parameters?

That needs to be decided. It seems that in #28052 we decided that to_html not outputting the entire dataframe was a bug. Presumably we would say the same about this, but I'm not sure.

cc @simonjayhawkins

we should probably maintain consistency. although long html would probably wrap within a cell and that does not apply to to_string.

[lshepard]

I agree- the original issue came from someone calling vanilla to_string and being surprised:

I am calling to_string() without any
parameters and it beautifully fixed-
formatted my dataframe apart from my
very wide filename column, that is being
truncated with "...". How can I avoid that?

I think no truncation by default is the most intuitive approach, and matches to_html behavior. Someone who is surprised can find the parameter to truncate pretty easily in the docs, and it’s unlikely to be a surprise the same way that truncation might be.

[TomAugspurger]

OK, let's call using the options.display value a bug then.

This will need a release note in doc/source/whatsnew/v1.0.0.rst under bug fixes.

[lshepard]

Thanks for the feedback! I added the whatsnew notice, swapped to the correct validator, and did some more manual testing to ensure it worked as expected.

[TomAugspurger]

Looking good overall. A few small requests.

[lshepard]

Ready for review, thanks!

[TomAugspurger]

@lshepard merge conflict in the release notes. Could you merge master & repush?

[lshepard]

Huh, failed with “worker 'gw0' crashed while running 'pandas/tests/test_sorting.py::TestSafeSort::test_labels_out_of_bound[-1]'”. I don’t think that’s related but I’ll take a look.

[TomAugspurger]

Could be a random failure. I'd start by repushing an empty commit.

[WillAyd]

lgtm

[lshepard]

Thanks for all the detailed comments on the review!

[WillAyd]

@lshepard can you fix the merge conflict on the whatsnew? @TomAugspurger mind taking a look at this one?

[TomAugspurger]

LGTM too. Just need to fix the merge conflict.

[WillAyd]

Nice PR - thanks a lot @lshepard

[rswgnu]

Excited to try this out and see it resolved. I hope proper left justification can be resolved soon too. Thanks.