Ordering matters for @overload

[mgeisler]

I am trying to model a generic container (a Pandas Series though it shouldn't matter too much for this). The container has a generic "value type" and a generic "index type". When iterating over the container, you get the value type back. So I tried with:

class Series(Generic[VT, IT], Sequence[VT]):

    @overload
    def __getitem__(self, key: slice) -> List[VT]: ...

    @overload
    def __getitem__(self, key: int) -> VT: ...

This gives me an error:

stubs/pandas/__init__.pyi: note: In class "Series":
stubs/pandas/__init__.pyi:186: error: Signature of "__getitem__" incompatible with supertype "Sequence"

That baffled me for a while and on a whim, I tried reordering the two signatures:

class Series(Generic[VT, IT], Sequence[VT]):

    @overload
    def __getitem__(self, key: int) -> VT: ...

    @overload
    def __getitem__(self, key: slice) -> List[VT]: ...

The error is now gone! :-)

When I first got the error, I tried going into typing.py to see exactly what signature Sequence has. That didn't help me since Sequence is defined as

class Sequence(Sized, Iterable[T_co], Container[T_co],
               extra=collections_abc.Sequence):
    pass

I think it could have been helpful if mypy could have written the exact types it was comparing.

This was tested with 0ec0cb4.

[JukkaL]

Yeah, mypy should show the signature of the superclass method.

Also, mypy could perhaps allow you to switch the order of overload variants in case the signatures aren't overlapping, though this is a little questionable as the order could plausibly affect which overload gets picked in a call.

[ddfisher]

@JukkaL - what do you mean by "switch the order of overload variants"? Is there some obvious natural ordering that I'm missing?

[JukkaL]

By order of overload variants I mean the textual order of the @overload decorated functions. For example, this is not valid right now:

class A:
    @overload
    def f(self, x: X) -> int: ...
    @overload
    def f(self, x: Y) -> str: ...

class B(A):
    @overload   # note that Y -> str is now the first variant
    def f(self, x: Y) -> str: ...
    @overload
    def f(self, x: X) -> int: ...

Mypy generally picks the first overload variant that matches the argument types when type checking a call. If you change the ordering of variants in a subclass the inferred return type could arbitrarily change for a call depending on whether the receiver type is the base class or the subclass, which is undesirable. However, this is a pretty marginal thing in practice.

[mgeisler]

Interesting... it wasn't even clear to me that there was a defined order. I had somehow imagined that all overloaded methods would have different (non-overlapping) signatures and therefore order wouldn't matter. Since Python itself doesn't have overloaded functions, the semantics of this was not clear from the examples in the tutorial.

[JukkaL]

The reason that there is an order is partially an accident of history, as earlier versions of mypy actually let you use @overload to define functions with multiple bodies, and the decorator would implement runtime dispatch based on argument types. The first matching overload variant was always executed. The runtime dispatch is no longer supported, so mypy could ignore the order of overload variants. However, in practice functions can implement a particular order if they performance isinstance checks, for example. Consider this function:

def f(x):
    if isinstance(x, A):
        return 'x'
    elif isinstance(x, B):
        return 1
    assert False

We could have this overloaded signature in a stub file:

@overload
def f(x: A) -> str: ...
@overload
def f(x: B) -> int: ...

Now if we call f with an instance of a class that is a subclass of both A and B, the first variant would be picked at runtime and the return value would be a string. Thus it makes sense to expect the A -> str variant to be the first one, and give it preference over the B -> int variant.

[gvanrossum]

(Perhaps also relevant to this discussion is the idea that two unrelated user-defined classes A and B are considered potentially overlapping because someone might use multiple inheritance to derive a class from both of them simultaneously. But this is more of a theoretical issue -- in practice that's not going to happen, and we should change mypy's rules not to reject overloads on this basis alone.)

[JukkaL]

Actually mypy normally doesn't reject overloads based on potential overlapping that could be introduced via multiple inheritance. I think that the operator method checks might still be picky about this, though.

[mentalisttraceur]

Not sure if this is considered part of the same issue or not, but I just had a situation where order of overloads made the difference between type checks passing correctly and false Too many arguments for "compose" [call-arg] errors:

Broken:

from typing import Callable, TypeVar, ParamSpec, overload

P = ParamSpec('P')
R1 = TypeVar('R1')
@overload
def compose(f1: Callable[P, R1]) -> Callable[P, R1]:
    ...
R2 = TypeVar('R2')
@overload
def compose(f2: Callable[[R1], R2], f1: Callable[P, R1]) -> Callable[P, R2]:
    ...
R3 = TypeVar('R3')
@overload
def compose(f3: Callable[[R2], R3], f2: Callable[[R1], R2], f1: Callable[P, R1]) -> Callable[P, R3]:
    ...
R4 = TypeVar('R4')
@overload
def compose(f4: Callable[[R3], R4], f3: Callable[[R2], R3], f2: Callable[[R1], R2], f1: Callable[P, R1]) -> Callable[P, R4]:
    ...

Fixed (by changing order)

from typing import Callable, TypeVar, ParamSpec, overload

P = ParamSpec('P')
R1 = TypeVar('R1')
R2 = TypeVar('R2')
R3 = TypeVar('R3')
R4 = TypeVar('R4')

@overload
def compose(f4: Callable[[R3], R4], f3: Callable[[R2], R3], f2: Callable[[R1], R2], f1: Callable[P, R1]) -> Callable[P, R4]:
    ...
@overload
def compose(f3: Callable[[R2], R3], f2: Callable[[R1], R2], f1: Callable[P, R1]) -> Callable[P, R3]:
    ...
@overload
def compose(f2: Callable[[R1], R2], f1: Callable[P, R1]) -> Callable[P, R2]:
    ...
@overload
def compose(f1: Callable[P, R1]) -> Callable[P, R1]:
    ...

[JelleZijlstra]

We have made some changes to overload resolution since and it is expected that overload order matters. Any related problems should be discussed in a new issue, not in this ancient closed issue.

[mentalisttraceur]

Turns out the issue in my last comment wasn't even an ordering issue, it was an interleaving issue - eiher order works fine so long as all TypeVar declarations are before all the compose overloads.