DOC: Dataframe.apply does not always return a Series when reduce=True

[NelsonAndrew]

Code Sample, a copy-pastable example if possible

pd.DataFrame({'col_1':['val_1','val_2','val_3']}).apply(lambda row: [1], axis=1, reduce=True)

Problem description

Per the documentation, calling Dataframe.apply w/ reduce=True should always produce a Series. However, if the applied function returns list values Dataframe.apply returns a Dataframe, not a Series.

Expected Output

When called with reduce=True, I expect Dataframe.apply to produce a Series where each element is a list, not a Dataframe.

Output of pd.show_versions()

0.18.1

[jreback]

please show a copy pastable example

[NelsonAndrew]

updated to copy pastable example

[jreback]

The issue is .apply has to try to figure out what you are returning and how that maps to the starting data.

returning lists from a function are subject to interpretation and thus a single element list is ambiguous. You can return a tuple instead which works unambiguously.

Note that returning non-scalars is generally not recommended and is not efficiently supported.

In [5]: pd.DataFrame({'col_1':['val_1','val_2','val_3']}).apply(lambda row: [1, 2], axis=1)
Out[5]: 
0    [1, 2]
1    [1, 2]
2    [1, 2]
dtype: object

In [6]: pd.DataFrame({'col_1':['val_1','val_2','val_3']}).apply(lambda row: [1], axis=1)
Out[6]: 
   col_1
0      1
1      1
2      1

In [8]: pd.DataFrame({'col_1':['val_1','val_2','val_3']}).apply(lambda row: (1,), axis=1)
Out[8]: 
0    (1,)
1    (1,)
2    (1,)
dtype: object

closing as not a bug / won't fix.

[jreback]

a bit more commentary in #14370

[NelsonAndrew]

Thanks for the feedback.

[jorisvandenbossche]

@jreback I agree that apply does a lot of guessing in the default case, and it is sometimes difficult to predict what the output will be. But isn't that the point of the keyword? To be able to specify this and be sure about the resulting shape?
In any case, if this is a won't fix, then the documentation is wrong.

[jreback]

reduce : boolean or None, default None
    Try to apply reduction procedures. If the DataFrame is empty,
    apply will use reduce to determine whether the result should be a
    Series or a DataFrame. If reduce is None (the default), apply's
    return value will be guessed by calling func an empty Series (note:
    while guessing, exceptions raised by func will be ignored). If
    reduce is True a Series will always be returned, and if False a
    DataFrame will always be returned.

I guess the doc-string is misleading. reduce really only applies to empty frames and has no effect otherwise, which it says in the end.

I suppose some nice examples of what not to do in .apply (like returning a 1-element list and expecting it work, rather returning tuples) might be in order.

@NelsonAndrew I'll re-open as a doc issue.

.apply is already pretty magical, I think it could use a comprehensive fix but that is likely to break back-compat.