Support interoperability with other annotations and type systems

[scoder]

Support interoperability with other annotations and type systems by requiring separate namespaces.

[gvanrossum]

I don't like the tuple notation. I see several options that you might pursue:

• decorators; the type checker can understand these (in most cases)
• use custom annotations and don't run the type checker
• use custom annotations and disable the type checker per module or per function using a magic comment (see Do we need a standard way of silencing checkers? #16)

Also, it would be more useful to open an issue for discussion before proposing a textual change (unless the change in non-controversial, which you know this isn't). And, "typing.integer" isn't a thing.

[scoder]

I thought it was ok to propose a text change as a basis for discussion. Pull requests are quite cheap and they're essentially just issues with changes.

I actually think that tuples are ok as long as their content is unambiguous, which is what I'm trying to assure with the namespace requirement. Lists (i.e. square brackets) might look a little nicer syntactically, but wouldn't catch the idea of different kinds of annotations as much as tuples do. The alternative would be dicts, but they are very verbose and that verbosity is not needed if each item in the tuple is semantically well defined by its namespaced type. I think tuples are as simple and straight forward as it can get.

Are you suggesting that interoperability isn't even worth the overhead of a tuple? What's your opinion on an extensible type system? Don't you think it'll be necessary?

[gvanrossum]

BTW, Steven D'Aprano wrote something pretty compelling on python-ideas. Maybe we can use some of his language in the PEP:

"""
In the past I have strongly defended the idea that type-hinting
annotations need to co-exist with other uses for annotations. By this I
mean that there should be a simple way for the author of a module who
uses annotations for something else to flag that module, or parts of the
module, so that the type-checker skips it.

There is no good way to have multiple uses of annotations be used in the
same function. You have suggested using a dict, but what of the module
that wants to give a completely different meaning to dicts as
annotations? Your interpetation of annotations is not compatible with
that module. Like multiple inheritence, multiple use of annotations is
only possible if all parties cooperate and agree on semantics.

Besides, the more information you try to squeeze into the function
parameter list, the more unwieldy, unreadable and ugly it gets.

Type-hinting was Guido's original motivation for introducing
annotations, and there is no other use of annotations which has become
popular or widespread enough to justify calling it a "standard use". So
in the absense of any other standard use of annotations, I am
comfortable with giving type-hints special status:

• whether type-checking is enabled by default or not, we understand that
  annotations are primarily for type-hinting, and any annotations are to
  be understood as type-hints by default;
• other uses for annotations are permitted, but it is up to the user to
  flag the module/class/function so the type-checker skips it;
• no provision is made for mixing type-hints and arbitrary other uses in
  the same annotation (in other words, if you want to use annotations
  for type-hinting and Foo at the same time, you're on your own);
• but of course, nothing stops you from creating your own custom
  annotation scheme that includes Foo + type-hints.

If some other use of annotations becomes wildly successful, then it
may be worth rethinking the special status of type-hinting in the
future.

If you want to interoperate with type-hinting, decorators may be a
second-class solution, but they are a solution.
"""

[gvanrossum]

@scoder:

I thought it was ok to propose a text change as a basis for discussion.
Pull requests are quite cheap and they're essentially just issues with changes.

The problem with a PR is that it obscures the essence of the proposal with a lot of editing details. That may make sense for code (where trying to explain it in English is often more verbose than showing the code) but not for text (where the diff is significantly more verbose than just the proposal, clearly worded). Also, a PR doesn't show the text in the same "stream" window as the rest of the discussion.

I actually think that tuples are ok as long as their content is unambiguous, [...]

Well, I really don't like them. And I want type hints to become the primary use of annotations.

Are you suggesting that interoperability isn't even worth the overhead of a tuple?
What's your opinion on an extensible type system? Don't you think it'll be necessary?

I'm not sure what you mean by an extensible type system. Clearly you're not referring to the ability to define new types. But then what? New constructs like Intersection (#18) or Protocols (#11)? Those can be discussed and added in a future version; it's hard to see how you could add those without changes to the type checker, and an "escape" notation like you propose isn't going to help. And sticking documentation in an annotation (like your doc() example) isn't extending the type system, it's an attempt to add something completely different to annotations. That just sounds like a bad idea to me -- you can only cram so much in the annotation for an argument before it just becomes a jumble.

[The-Compiler]

I'm one of the users of custom annotations (to export functions to the user as "commands" they can invoke in my application), and I've tried to follow the discussions so far.

but of course, nothing stops you from creating your own custom
annotation scheme that includes Foo + type-hints.

How would that work? I'd imagine something like a decorator @typing.custom_annotation(callable), where callable is a function which gets the annotation of a parameter, and returns the type for the checker to use. That way, having a dict is easy (@typing.custom_annotation(operator.itemgetter('pytype')) for example, if I'm not mistaken), and more complex things are possible.

I don't like the idea of being unable to use the type checker for functions with custom annotations, i.e. only being able to skip things via a decorator. I'd prefer to make them work using a decorator.

[gvanrossum]

The problem with your proposal is that it's not enough to specify how you would get the type annotation for a parameter -- you must specify it in a way that the static type checker can use. The static type checker reads your code but doesn't ever import it. IMO it's unreasonable to expect the type checker to understand operator.itemgetter('pytype') and expect then the annotation to be a dict from which is picks the ''pytype" key as the type. IOW you're still thinking of the type as something to be conjured up somehow at runtime -- while actually it is intentional that the type checker looks at your source code without any help from the runtime.

I'm not sure what Steven D'Aprano meant with "Foo + type-hints" but I don't see a reasonable interpretation except that it refers to you writing your own type checker, which you can make behave however you want to.

[The-Compiler]

That is a good point. Then I'd still be in favour of a dict-syntax being supported optionally. That'd work for me (I currently use dicts), but indeed, I'm not sure how useful that'd be universally.

An example of how I use the annotations:

    @cmdutils.register(...)
    def scroll_perc(self, perc: {'type': float}=None,
                    horizontal: {'flag': 'x'}=False,
                    count: {'special': 'count'}=None):

This is indeed rather heavy to read, but very convenient to me. This would expose a command with a perc-argument which is verified to be a float, a horizontal-flag with x as shorthand, and a count argument which is automatically filled with the count the user entered (think vim commands).

[gvanrossum]

But are you really interested in running mypy over your code and over the application code that is calling your commands? How does an application even call those commands? How doe the non-type-hint annotations come into play? If an application does a dynamic lookup of commands in your library and then calls them, the type checker isn't useful anyway -- the type checker only works for (shall we say) "static" calls where the name of the called function (or method) is in the caller's code. Example:

from somewhere.something import whatever
def main():
    whatever('Hello', 'world')

can easily be type-checked (assuming there are annotations on whatever()). However, in the following example, the static checker can do nothing, even though the exact same function is called (and the same annotations appear at runtime):

import importlib
def main():
    something = importlib.import_module('somewhere.something')
    whatever = getattr(something, 'whatever')
    whatever('Hello', 'world')

Now, if your use case looks more like the former, type annotations on whatever are useful. You can do two things: either add type annotations directly in the source code for somewhere/something.py, or you can create a stub file that is seen by the static checker but not at run time. If you use a stub file, you can still use annotations for your own nefarious (:-) purposes, but you must disable type checking own code (i.e. something.py) using a per-method or per-class decorator or a "# type: OFF" comment. Only the application will be type-checked (and only if the app developer runs mypy). If you want your own code to be checked by mypy too (or primarily), you will have to give up on using annotations for your own flags -- you're going to have to invent some decorator. (Because I still don't believe that the ugliness of supporting your dict proposal is worth it.)

[The-Compiler]

Indeed, I just realized the same while catching up with the thread on python-ideas. Those functions are indeed called dynamically, so it'd probably suffice to have a "don't check this" decorator.

Am I right when I assume the decorator has to be an actual decorator above the function so it can be inferred statically - i.e. I couldn't apply the decarator in my @cmdutils.register decorator?

[gvanrossum]

Yeah, see #51.

[gvanrossum]

Sorry, should have closed this back in January.