spurious "Need type annotation" error with overloads

[chadrik]

I'm getting "Need type annotation for variable" errors which I think are the result of a bug with overloads.

Stub file overload_test.pyi:

from typing import overload, TypeVar, Any

_T = TypeVar('_T')

@overload
def attr(default: _T = ..., blah: int = ...) -> _T: ...
@overload
def attr(default: Any = ...) -> Any: ...

Test program:

from overload_test import attr

a = attr()
b = attr()

reveal_type(a)
reveal_type(b)

mypy output:

test.py:3: error: Need type annotation for variable
test.py:4: error: Need type annotation for variable
test.py:6: error: Revealed type is 'Any'
test.py:6: error: Cannot determine type of 'a'
test.py:7: error: Revealed type is 'Any'
test.py:7: error: Cannot determine type of 'b'

It may be interesting to note that if I swap the order of the overloads, the "Need type annotation for variable" and "Cannot determine type" errors go away.

[ethanhs]

This happens because mypy is getting confused between the overload and the TypeVar. I believe it is trying to deduce the TypeVar and cannot, but is not considering the alternative signature. It seems we need to consider other valid overloads if typevar deduction fails on one.

Note that removing the other overload entirely has the same output, thus the second overload is not used at all.

EDIT to say confirmed on 0.540-master

[gvanrossum]

It would be best to define three different overloads, with 0, 1 and 2 arguments.

[chadrik]

Note that removing the other overload entirely has the same output, thus the second overload is not used at all.

For my own edification, if we ditch the second overload...

from typing import overload, TypeVar, Any

_T = TypeVar('_T')

def attr(default: _T = ..., blah: int = ...) -> _T: ...

...is the "Need type annotation for variable" error valid? Isn't the unfilled TypeVar equivalent to Any?

[gvanrossum]

Isn't the unfilled TypeVar equivalent to Any?

No, it's not as simple as that. It becomes similar to

def attr() -> _T: ...

and the type of attr() then becomes "the uninhabited type" which is printed as <nothing>, but which essentially means "there is no value of this type".

[chadrik]

and the type of attr() then becomes "the uninhabited type"

and I assume that anytime you assign "the uninhabited type" to a variable it generates the "Need type annotation for variable" error?

It would be best to define three different overloads, with 0, 1 and 2
arguments.

The problem is that in reality I have 9 arguments. Creating an overload for every combination is a bit inconvenient.

[chadrik]

Let me fill you in on a bit more detail. I'm writing the pyi stubs for the attrs project, and we want all of these forms to work:

a = attr.ib()  # Any
b = attr.ib(type=int)  # int
c = attr.ib(default=0)  # int
d: int = attr.ib()   # int

I can get all the cases to work except the first.

The following pair of overloads generate the errors from the OP (same result as if there was only the first overload, as @ethanhs points out):

@overload
def attr(default: _T = ..., validator: Optional[Union[_ValidatorType[_T], List[_ValidatorType[_T]], Tuple[_ValidatorType[_T], ...]]] = ..., repr: bool = ..., cmp: bool = ..., hash: Optional[bool] = ..., init: bool = ..., convert: Optional[_ConverterType[_T]] = ..., metadata: Mapping = ..., type: Type[_T] = ...) -> _T: ...
@overload
def attr(default: _T = ..., validator: Optional[Union[_ValidatorType[_T], List[_ValidatorType[_T]], Tuple[_ValidatorType[_T], ...]]] = ..., repr: bool = ..., cmp: bool = ..., hash: Optional[bool] = ..., init: bool = ..., convert: Optional[_ConverterType[_T]] = ..., metadata: Mapping = ...) -> Any: ...

The following pair of overloads generates the "Overloaded function signatures overlap with incompatible return types" error:

@overload
def attr(default: _T = ..., validator: Optional[Union[_ValidatorType[_T], List[_ValidatorType[_T]], Tuple[_ValidatorType[_T], ...]]] = ..., repr: bool = ..., cmp: bool = ..., hash: Optional[bool] = ..., init: bool = ..., convert: Optional[_ConverterType[_T]] = ..., metadata: Mapping = ..., type: Type[_T] = ...) -> _T: ...
@overload
def attr(default: Any = ..., validator: Optional[Union[_ValidatorType[Any], List[_ValidatorType[Any]], Tuple[_ValidatorType[Any], ...]]] = ..., repr: bool = ..., cmp: bool = ..., hash: Optional[bool] = ..., init: bool = ..., convert: Optional[_ConverterType[Any]] = ..., metadata: Mapping = ...) -> Any: ...

I'm pretty stuck at the moment, and I can't tell if there is a bug that needs to be solved before I can move forward (and if so which one?), or if there is a workaround. Any tips would be greatly appreciated!

[gvanrossum]

Sadly I cannot think of a better solution than 10 overloads -- for each keyword arg, a version where that kw is mandatory and the rest are optional, and then an empty one. You can use this syntax to disallow positional args: `def attr(*, kw1: T, kw2: T = ...) -> T`.

[chadrik]

Sadly I cannot think of a better solution than 10 overloads -- for each
keyword arg, a version where that kw is mandatory and the rest are
optional, and then an empty one. You can use this syntax to disallow
positional args: def attr(*, kw1: T, kw2: T = ...) -> T.

The attrs stubs have to maintain compatibility with the actual attr.attr() function which cannot use the bare * syntax (attrs supports python 2, and besides, altering the signature would be a breaking change). So, I don't think that the 10 overload plan works.

I experimented a bit more and I think I would have to create a stub for every permutation.

For example, I tried these stubs for a 3 argument function:

from typing import overload, TypeVar, Any

T = TypeVar('T')

# keywords
@overload
def attr(*, one: T,       two: bool = ..., three: bool = ...) -> T: ...
@overload
def attr(*, one: T = ..., two: bool,       three: bool = ...) -> T: ...
@overload
def attr(*, one: T = ..., two: bool = ..., three: bool) -> T: ...
# positional
@overload
def attr(one: T, two: bool, three: bool) -> T: ...
@overload
def attr(one: T, two: bool) -> T: ...
@overload
def attr(one: T) -> T: ...
# empty
@overload
def attr() -> Any: ... 

which still does not cover all scenarios:

from overload_test import attr

a = attr()
b = attr(0, three=True, two=True)
c = attr(0, True, True)
d = attr(three=True, two=True)  # error: Need type annotation for variable

Seems like I'm stuck again unless I'm willing to write an ungodly number of overloads, and it's unclear which bug/feature gets me unstuck. I know that there's been a lot of discussion about overloads (#4159, #4159, python/typing#253), but I'm not sure where that work is coalescing. I'd love to subscribe so I can keep an eye on its progress.

@ethanhs suggested "we need to consider other valid overloads if typevar deduction fails on one." Is this an actionable idea? Can that be executed independently of the other work on overloads, or should it be passed along to @ilevkivskyi for consideration as part of the larger refactor?

[gvanrossum]

Stubs are always using Py3 syntax, so maybe that can help you get over the `*` idea (if it even matters). The only other idea I have (other than waiting to see if the planned changes to `@overload` help) is to write a mypy plugin.

[chadrik]

I've been struggling to find a set of overloads that even marginally works, even if it's not perfect.

Here's my contrived example:

from typing import overload, TypeVar, Any

T = TypeVar('T')

@overload
def attr(*, one: T,         two: bool = ..., three: bool = ...) -> T: ...
@overload
def attr(*, one: Any = ..., two: bool,       three: bool = ...) -> Any: ...
@overload
def attr(*, one: Any = ..., two: bool = ..., three: bool) -> Any: ...

And the program:

a = attr(one=0, two=True, three=True)
reveal_type(a)

I expect the revealed type to be int, but it is Any. Why would mypy not choose the first overload here? Doesn't order matter?

Also, I could use some insight into why these two overloads are considered overlapping:

@overload
def attr(*, one: T,         two: bool = ..., three: bool = ...) -> T: ...
@overload
def attr(*, two: bool,       three: bool = ...) -> Any: ...

The first has a required keyword argument one that the second doesn't possess at all, and the second has a required argument two that is optional in the first.

Thanks for the help.

[gvanrossum]

mypy currently doesn't pick the first overload -- it considers all of them, and returns either the common type, or in some cases `Any` (IIRC it chooses `Any` when that's one of the possible returns for a matching overload). We're planning to change this, your input is useful, but if you want a fix right now writing a mypy plugin seems the only option. I'm not sure why your second example is considered overlapping -- maybe that's a bug!

[chadrik]

We're planning to change this, your input is useful, but if you want a fix
right now writing a mypy plugin seems the only option.

It's not urgent. I was looking to gather more knowledge and hopefully give you all some more information to chew on when redesigning this.

FWIW, I think that order should be an important factor when mypy chooses an overload. Overloads are essentially a declarative form of the imperative instance checks found within a function body, and order certainly matters there.

For example, this seems intuitive to me, but it results in an overlap error:

from typing import overload, Any

class Foo:
    pass

class Bar:
    pass

@overload
def func(a: Foo, b: Bar) -> Bar: ...
@overload
def func(a: Foo, b: Any) -> Foo: ...
@overload
def func(a: Any, b: Any) -> None: ...

def func(a, b):
    if isinstance(a, Foo) and isinstance(b, Bar):
        return b
    elif isinstance(a, Foo):
        return a
    else:
        return None

func(Foo(), 1)

Certainly there are other complex considerations which are over my head, but I think that above all the new design should aim to be intuitive. Something that can be conveyed in a (short) paragraph to your average mypy user.

Thanks again for your help and patience.

[gvanrossum]

I have to stop arguing now. Please find the relevant issue in the typing tracker and add your feedback there.

[JukkaL]

We'll probably do a somewhat major overhaul of overloads at some point. Currently they are in a pretty messy state.

[ilevkivskyi]

@chadrik

Can that be executed independently of the other work on overloads, or should it be passed along to @ilevkivskyi for consideration as part of the larger refactor?

I agree with @JukkaL here. I will prepare a summary about how overloads should work and post it in python/typing#253

[euresti]

From a "gradual typing" perspective I find it kind of useful to have this raise error: Need type annotation for variable

@attr.s
class Foo:
    a = attr.ib()

It's akin to saying x = {} or x = None And if you're adding types to a project you will probably need to state what this is. Having worked on projects like this before, I consider Any to be a last resort thing and something that should be avoided at all costs.

However it would be nice if the other versions wouldn't have that problem:

@attr.s
class Foo:
    a = attr.ib(17)
    a = attr.ib(default=22)

But I don't know if that's the case right now.

[chadrik]

The other cases are working as expected.

[Michael0x2a]

@chadrik -- I'm not sure where exactly the status of attrs is at right now, but we did recently land some fixes to overloads -- in short, we do basically fix the first matching overloads in order, though we do mandate that the overloads are arranged roughly from least to most specific (and some other caveats).

Maybe this will help address some of the issues you had? But we've also landed an attrs plugin before the overload overhauls happened so maybe we've already found workarounds...

Not sure, so I'll leave this open (but I figured I ought to also leave some sort of update).

[ilevkivskyi]

@Michael0x2a hm, I think there is still a bug, I get this on master (note the bound on type variable):

from typing import overload, TypeVar, Any

_T = TypeVar('_T', bound=int)

@overload
def attr(default: _T = ..., blah: int = ...) -> _T: ...
@overload
def attr(default: Any = ...) -> Any: ...  # Overloaded function signature 2 will never be matched: signature 1's parameter type(s) are the same or broader

But if I call attr("hi") it looks like the second overload is actually matched.