What's the proper way to spell the type of a callable?

[gvanrossum]

Python supports many types of parameters -- positional, with defaults, keyword only, var-args, var-keyword-args. The intended signature can not always be unambiguously derived from the declaration (the 'def' syntax). Built-in functions provide additional issues (some don't have names for positional parameters).

The first point to make is that we needn't provide ways to spell every signature type. The most common use case for signatures is callback functions (of various kinds), and in practice these almost always have a fixed number of positional argument. (The exception is the callback for built-in functions like filter() and map(); the callback signatures for these correlate in a weird way to the other arguments passed, and many languages deal with these by overloading a number of fixed-argument versions.)

Second, Python tries hard to avoid the word "function" in favor of "callable", to emphasize that not just functions defined with 'def' qualify, but also built-ins, bound methods, class objects, instances of classes that define a call method, and so on. In mypy, function types are defined using Function, which I actually like better than Callable, but I feel that my hands are bound by the general desire to reuse the names of ABCs (the abc and collections.abc modules) -- collections.abc defines Callable.

Given a function with positional arguments of type (int, str) and returning float, how would we write its signature? In mypy this is written as Function[[int, str], float]. In general, mypy's Function is used with two values in square brackets, the first value being a list of argument types, the second being the return type. This is somewhat awkward because even the simplest function signature requires nested square brackets, and this negatively impacts the readability of signature types.

But what else to use? Using round parentheses runs into all sorts of problems; since Callable is a type it looks like creating an instance. We could drop the inner brackets and write Callable[int, str, float](still reusing the same example). My worry here is that in many cases the return value is irrelevant and/or unused, and it's easy to think that the type of a procedure taking two bool arguments is written Callable[bool, bool] -- but actually that's a function of one bool argument that also returns a bool. The mypy approach makes it easier to catch such mistakes early (or even to allow Callable[[bool, bool]] as a shorthand for Callable[[bool, bool], None]).

I think I've seen other languages where the return type comes first; arguably it's harder to forget the first argument than the last. But everywhere else in Python the return type always comes after the argument types.

In the end I think that mypy's notation, with Callable substituted for Function, is the best we can do. (And if you find Callable too awkward, you can define an alias "Function = Callable" or even "Fun = Callable".)

[mvitousek]

For the basic case with only positional arguments, I don't know of a better syntax than mypy's. It would be nice if we could use arrows for function types: int -> str rather than Callable[[int],str], but as far as I know this is impossible in the Python parser (would be cool if I'm wrong).

In our case studies using Reticulated, we actually found that it was practically useful to have our callable types contain more information than just the types of positional arguments. Including (optional) parameter names helped us detect a class of potential bugs where a subclass's version of a method has a different name than its superclass's*:

  class C:
    def f(self, x:int):
        return 10
  class D(C):
    def f(self, y:int):
        return 10
  def foo(c:C):
    c.f(x=20)
  foo(D())

Here, the call to c.f() in foo() fails at runtime because D's implementation of f has a different parameter name than C's. Reticulated would reject this program because D is not a subtype of C, since C().f has a type like Callable[[('x':int)], Any] while D().f has type Callable[[('y':int)], Any].** These types are incompatible because of their differing parameter names.

We didn't need to write down these types anywhere in the program to get this result, since the analyzer can pick them up from their definitions, so maybe this is immaterial. However, I'm worried that preventing users from writing down useful types like the ones above will prevent some reasonable programs from ever being "fully" statically typed. Further, any analyzer which supports named-parameter function types will need to have a way to spell them in error messages anyways, so why not allow them to be written down as annotations too?

* a real life version of this is in python 3.4's xml/dom/minidom.py, between Node and Entity classes
** technically, currently in Reticulated the type is Function([('x':int)], Dyn) but we'll transition to the PEP's syntax once it's finalized.

[gvanrossum]

Yeah, what I was trying to say is that the information a type checker
retains about a function signature should contain everything, so it can
reject your example. But the syntax we give users to specify callback types
in practice can be more limited, like mypy's.

I'm not worried about preventing some reasonable programs from being fully
typed. Given the abundance and popularity of Python's dynamic features only
the most trivial programs can be fully typed anyway. My goal is to be able
to express the types for those parts of programs that could reasonably
be statically typed (i.e. where the programmer already has a static type in
their mind, we can now let them express it). Function annotations are the
80% win here. The rest can be covered in later versions based on experience
with the first 80%. Python wasn't built in a day. :-)

On Thu, Nov 6, 2014 at 11:07 PM, Michael Vitousek notifications@github.com
wrote:

For the basic case with only positional arguments, I don't know of a
better syntax than mypy's. It would be nice if we could use arrows for
function types: int -> str rather than Callable[[int],str], but as far as
I know this is impossible in the Python parser (would be cool if I'm wrong).

In our case studies using Reticulated, we actually found that it was
practically useful to have our callable types contain more information than
just the types of positional arguments. Including (optional) parameter
names helped us detect a class of potential bugs where a subclass's version
of a method has a different name than its superclass's*:

class C:
def f(self, x:int):
return 10
class D(C):
def f(self, y:int):
return 10
def foo(c:C):
c.f(x=20)
foo(D())

Here, the call to c.f() in foo() fails at runtime because D's
implementation of f has a different parameter name than C's. Reticulated
would reject this program because D is not a subtype of C, since C().f
has a type like Callable[[('x':int)], Any] while D().f has type Callable[[('y':int)],
Any].** These types are incompatible because of their differing parameter
names.

We didn't need to write down these types anywhere in the program to get
this result, since the analyzer can pick them up from their definitions, so
maybe this is immaterial. However, I'm worried that preventing users from
writing down useful types like the ones above will prevent some reasonable
programs from ever being "fully" statically typed. Further, any analyzer
which supports named-parameter function types will need to have a way to
spell them in error messages anyways, so why not allow them to be written
down as annotations too?

• a real life version of this is in python 3.4's xml/dom/minidom.py,
  between Node and Entity classes
  ** technically, currently in Reticulated the type is Function([('x':int)],
  Dyn) but we'll transition to the PEP's syntax once it's finalized.

—
Reply to this email directly or view it on GitHub
#5 (comment).

--Guido van Rossum (python.org/~guido)

[gvanrossum]

Update: mypy renamed Function to Callable. I'm pretty set on having Callable[[arg1, arg2, ...], ret] and no way to spell the exact type of functions with keyword parameters, varargs etc. One exception: it's useful to be able to spell "don't care about the args, but it's callable, and it returns X" -- see python/mypy#540 for a proposal (AnyArgs).

[ambv]

I support Callable[[arg1, arg2, ...], ret], although I have to admit the example in the PEP confused me a little at first. Since TypeVar is a function-based factory, could Callable also be one? Example:
Callable(int, bytes, named_arg=bool).returns(int)

[gvanrossum]

I guess I have to explain my reasoning for when to use X(args) vs. X[args].

• When using X(args), you are instantiating X, and the relationship between X and the returned value is isinstance(X(args), X). This is the case for TypeVar.
• When using X[args], something else is happening, and the relationship is most likely issubclass(X[args], X). This is the case for a "classic" generic type such as List or Iterable, but also (in my view) for Union and Callable. In my prototype implementation this is the case, except for Union[] of a single type, which returns just that type. But I do claim that Union[int, str] is a subclass of Union.

[ambv]

Okay, that explanation is a good one, I like it. Looks like we have a winner:

Callable[[arg1, arg2], ret]

However, Python 3 sports keyword-only arguments and I think we should support that. A possible way out is:

Callable[[type1, type2], {'arg3': int, 'arg4': Optional[int]}, ret]

If there's only keyword arguments, we could skip defining the list. As ret will never be a dictionary instance, this is unambiguous for the type checker.

[gvanrossum]

Well, I think we should not support keyword-only arguments. We can add this extension later, but the use cases (where you'd actually want to define a callback with a keyword-only argument) are exceedingly rare.

[ambv]

We should resist the temptation to guess if the writer meant Any or None with Callable[[T1, T2]] and as such I'm against introducing that short-hand.

Otherwise fixed in e2e6fc4