Add typing_extensions module for new / optional typing features to PyPI

[vlasovskikh]

Let's add typing_extensions module to PyPI in addition to Mypy-specific mypy_extensions.

Currently new stuff in typing appears in mypy_extensions even though most of the new typing features are not Mypy-specific.

In order to make new typing features available to 1) previous Python versions, 2) all type checkers I would suggest to move these features to typing_extensions.

The good candidates for including in typing_extensions could be:

• NoReturn (apparently already in PEP 484)
• Protocol (draft PEP 544)
• Argspecs?
• TypedDict?

[gvanrossum]

Agreed. I do think there should be a requirement for things to appear in a PEP (either 484 or a new one) before they go into typing_extensions. This means we should probably write up TypedDict and maybe wait for argspecs (for those there's still a lot of possible variation in the eventual design).

[Michael0x2a]

Type and maybe NewType would be other good candidates. They're present in the later versions of Python 3.5, but not the earlier ones, iirc.

[gvanrossum]

Oooh, might as well!

[Michael0x2a]

Other ones perhaps worth adding in:

• typing.Collection -- added in 3.6
• typing.Deque -- added in 3.6.1
• typing.ContextManager -- added in 3.6
• typing.Counter -- added in 3.6.1
• typing.ChainMap -- added in 3.6.1
• typing.AsyncGenerator -- added in 3.5.4
• typing.ClassVar -- added in 3.5.3

(I got this list by looking for the "New in" phrase in the typing docs, though it might not be comprehensive. For example, neither Type nor NewType have the "New in ..." phrases next to them in the index at the end (which might in part be my fault, tbh))

And for the sake of completeness, we could also add in the TYPE_CHECKING constant (idk when exactly it was added, but it's listed as being new in the Python 3.6 release notes). It's a bit unnecessary since we can just do if False or several other alternatives instead, but hey, as long as we're here...

[JelleZijlstra]

Sounds like it might be worthwhile to just shadow/duplicate all of regular typing, so that people aren't tied to the stdlib's version of typing. The disadvantage is that type checkers would have to add special-casing for things imported from typing_extensions.

I've been thinking of a related idea to create a library of commonly useful type aliases. Some things that could be included are:

• An alias for Union[bytes, Text] (which is needed very often in typeshed)
• A generic alias for Union[AnyStr, os.PathLike[AnyStr]] that falls back to AnyStr before 3.6 (another common typeshed need)
• The types from https://github.com/python/typeshed/blob/master/stdlib/3/wsgiref/types.pyi, which don't actually exist at runtime.

[Michael0x2a]

I was curious and decided to just install Python 3.5.0 and 3.6.1 and do a diff, and found a few more (non-private) names that I missed: AsyncIterable, AsyncIterator, Awaitable, MethodDescriptorType, MethodWrapperType, NamedTupleMeta, SlotWrapperType, Text. Full list here.

(Though, I'm not sure how many of these are meant to be publicly used or not? The last four items on the list certainly don't appear to be documented anywhere.)

In any case, I agree with @JelleZijlstra that just shadowing typing might be the simplest option at this point given the number of differences (16 to 20, depending on how you count the last four).

[JukkaL]

I'm not sure if I like shadowing everything, as it would potentially lead into a subset of users just importing everything from typing_extensions. This would lead into inconsistency -- you might start seeing odd-looking code like this (and this could confuse some users and they could start start cargo culting the idiom):

# Just to make sure everything works in 3.5.0
from typing_extensions import Optional, Tuple

Also, if implementations of things like List, Tuple and such in typing_extensions would track the latest typing version, they might not always support mixing with types inside the system typing (for example, if we change the representation of List[int] as has been discussed). This could mean that something like this could fail at runtime:

# a.py
from typing_extensions import List
A = List

# b.py
from a import A
from typing import List
List[A[int]]  # might fail

The memory usage and startup time cost of typing_extensions would also be bigger if it would reimplement everything in typing.

If we'd just import the basic primitives like List from the system typing and export them through typing_extensions, we wouldn't have this problem, but people might be surprised by this as well, if they are expecting that all typing changes get backported to typing_extensions. Also, this opens to doors to the cargo culting example I gave earlier.

My preference would be just to include new things added to the public API of typing (or significantly changed things) since 3.5.0 to typing_extensions.

[ilevkivskyi]

@JukkaL

This could mean that something like this could fail at runtime:

I would say it will most certainly fail. I remember having big troubles because of accidentally having two copies of typing in different places.

My preference would be just to include new things added to the public API of typing (or significantly changed things) since 3.5.0 to typing_extensions.

I agree we should not shadow everything. Also I could propose adding stuff from typing_inspect to typing_extensions (if anyone still cares about runtime introspection).

[Michael0x2a]

Is anybody working on this at the moment? If not, I'd be happy to volunteer :)

I don't think I should be the one to publish the package on pypi or anything, but I could definitely do the preliminary work of pulling together the module for somebody to review, and work on making whatever changes are necessary to mypy itself.

[ilevkivskyi]

@Michael0x2a
It looks like nobody is working on this now, so that you can go ahead. I would propose to just combine three things in typing_extensions:

• New types added after 3.5.0
• Things from mypy_extensions (note there is a PR Support TypedDicts with missing keys (total=False) mypy#3558 that will change TypedDict)
• Introspection functions from typing_inspect

[JukkaL]

I think that we should focus on things in mypy_extensions that we think are likely to be promoted to typing without API changes. Some of the features are likely to change or we don't have much experience with them yet, and we may want to only have them in mypy_extensions for now. Examples include TypedDict, as mentioned by @ilevkivskyi, and also the flexible callable syntax (at least naming might still change).

Similarly, maybe typing_inspect is still too experimental for typing_extensions. Also, it's already available on PyPI, so the benefit of including it in typing_extensions seems limited.

[ilevkivskyi]

@JukkaL

... without API changes ...

This is a reasonable approach. Then my list above would be even simpler: NoReturn is already in typing, and TypedDict and flexible Callable still might change, therefore we can start from the first bullet -- new types added after 3.5.0.

[Michael0x2a]

I've started work on this, and ran into some complications I'd like to get some feedback on.

---

First, typing.overload originally threw an exception if you tried to use it at runtime, outside of stub files-- this was changed in Python 3.5.2.

I know we discussed up above that we wanted to minimize duplication, but I think it makes sense here to include the modified version of overload in typing_extensions? I can't really see this causing subtle runtime issues, and defining overloads at runtime is handy/would make it easier to write code compatible across multiple Python versions.

---

Second, the typing module seems to have been refactored between Python 3.5.2 and 3.5.3. In the older variant, you create new types by doing something like this:

class UnionMeta(TypingMeta):
    # Complicated code here

class Union(Final, metaclass=UnionMeta, _root=True):
    # Simple code here

In the newer variant, you instead do this:

class _Union(_FinalTypingBase, _root=True):
    # Complex code here

Union = _Union(_root=True)

This causes some complications with the NoReturn and ClassVar types, since both are currently defined using the newer variant.

One solution is to copy over _FinalTypingBase and all associated helper functions/parent classes to typing_extensions. This means we don't need to change NoReturn and ClassVar at all -- my main concern is that it could lead to subtle compatibility issues when mixing types (though to be fair, you can't really do that with NoReturn and ClassVar...).

The other solution would be to re-implement NoReturn and ClassVar to use the older variant and export that when the user is using Python 3.5.0 - 3.5.2. This likely minimize the chances of weirdness, but would likely make it harder to backport more complex types (such as TypedDict?) in the future.

I'm tentatively leaning towards the second solution -- thoughts?

---

I don't think this last thing is that big of a deal, but the exact definition of some of the collections types were changed over time. For example, typing.Sequence was changed to subclass typing.Reversible in 3.5.2, and was changed again to subclass typing.Collection in 3.5.3.

The question is whether we should include these modified types in typing_extensions -- no matter what you do, you're always going to run into inconsistencies related to issubclass checks at runtime (e.g. doing things like issubclass(Sequence[...], Reversible)),.

However, I don't think this is a feature many people rely on (maybe except for people writing runtime type checkers?), so I'm planning on not including these modified types to minimize duplication.

Does that sound correct, or should we backport the modified collection types anyways for the sake of consistency? A third option might be to rename any modified types in typing_extensions (e.g. provide a ReversibleSequence type), but that also seems confusing.

[gvanrossum]

I hope Ivan can provide more guidance here, but I also hope that typing_extensions.py doesn't end up containing a whole bunch of reimplementations of stuff that's also in typing.py. I do accept that in order to support the full range of released versions of typing.py, you may need to implement some things in different ways -- hopefully checking sys.version_info should suffice there (since typing.py release Python versions anyway; I don't really want to support configurations where someone installed e.g. typing.py 3.5.3 on Python 3.5.1).

[Michael0x2a]

I agree -- I'm also hoping to keep things as simple as possible. So far, it looks like NoReturn and ClassVar are the only two things that's going to require re-implementation in some way.

Relatedly, maybe there's a third alternative to the NoReturn and ClassVar thing -- just eliminate as much of the runtime checks as possible and keep typing_extensions.py super minimal, just like mypy_extensions.py. We'd still have to re-implement NoReturn, ClassVar, and any future types we add, but would be free to keep that re-implementation really simple instead of trying to build it on top of older versions of typing.py.

After all, I don't think anybody who's aware of typing_extensions to begin with is actually going to start abusing types...

[gvanrossum]

Oh, of course. If a dummy works for mypy_extensions.py it should work for typing_extensions.py. It means they won't have great runtime introspection but that's probably okay. So this should work at least for older typing.py versions.

[ilevkivskyi]

@Michael0x2a
Here are some thoughts on your comments:

• I think @overload can be just copied from the new version of typing.

• While adding Collection, sys.version_info should be used, since Collection depends on changes in collections.abc. Also I would rather focus on types that are completely absent, like Deque and AsyncGenerator, etc. and later we can polish Reversible etc.

• NoReturn and ClassVar could be implemented differently than in typing. Ideally all tests from from test_typing.py should pass. But the minimal requirement is the "user-facing API" of these type forms.

If you have specific questions about the internal API, I will be glad to help.

[Michael0x2a]

I have a first draft for this project here, if anybody is interested in reviewing it: https://github.com/michael0x2a/typing_extensions. (If I should make a pull request somewhere instead, let me know).

While implementing this, I also made the following decisions:

• I decided to re-implement the current runtime behavior of NoReturn and ClassVar for older versions of Python, mainly to see what it would look like. However, it seems the consensus above was that going with a simpler implementation would be better (?). If so, I'm happy to modify NoReturn and ClassVar accordingly.
• typing.Counter could not cleanly be backported to Python 3.5.0 and 3.5.1 due to this bug -- Counter[T] is subclassing Dict[T, int]. I chose to circumvent this by hacking Counter so Counter[T] is always aliased to Counter[T, int] for those two Python versions. The other option would be just not define Counter for Python 3.5.0 and 3.5.1 at all.
• typing.Collection also could not be cleanly backported because it depends on the presence of collection.abc.Collection, which was only recently added to Python. Since Collection seemed like such a useful type, I decided to backport a (private) version of collection.abc.Collection to versions of Python that didn't include it so we can also define typing.Collection. The other alternative would be to just not backport typing.Collection at all.
• I chose to implement both a Python 2 and Python 3 version of this module, mostly to make life slightly easier for people who are trying to write Python 2 and 3 compatible code.

Basically, I chose to favor emulating typing as closely as possible and backporting as much as possible possibly at the cost of simplicity. That may have been the wrong tradeoff to make -- in that case, I'm happy to go back and simply and remove some of the hacks I added in.

Running tests:

Because installing seven different versions of the Python interpreter just to test this module seemed a bit extreme, I wrote a test runner that only depends on the existence of Python 2.7.x and Python 3.6.1, and modifies PYTHONPATH that points to snapshots of relevant modules from standard library for each Python release. This should emulate running the tests against different versions of the Python standard library.

test_typing_extensions.py itself is mostly subset of test_typing.py from this repo, with some tweaks/extra tests.

More specifically, the tests assume that python is aliased to Python 2.7 and python3 is aliased to Python 3.6.1 on Linux systems (and py -2.7 and py -3.6 for Windows). If that isn't the case, you can modify run_tests.py so it uses the right aliases.

[gvanrossum]

I wonder if this could be done as a subdirectory of the existing typing repo? That way the link between the two is more obvious, and we may be more likely to update them in sync. (I think the idea is that stuff gets added to both, but typing_extensions gets released frequently, while typing.py only gets released in sync with CPython bugfix releases.

[ilevkivskyi]

I wonder if this could be done as a subdirectory of the existing typing repo?

I think we should do it this way.

[ilevkivskyi]

typing_extensions are now published on PyPI with backported new classes form typing and Protocol. The remaining two candidates mentioned in the original post are TypedDict and extended Callable arguments. What are the opinions about these two?

[JukkaL]

I think that we should wait some more to see if these features will be actively used by users. Looking at Dropbox internal repos, extended Callable arguments are only used a handful of times. TypedDict has a similar number of uses, but we haven't made TypedDict officially supported yet so only early adopters are using it right now. My view is that we should consider promoting features to typing when we have evidence that they will be used widely enough. Having a feature in typing is a big commitment and causes work for every developer of a PEP 484 compatible tool. Perhaps we can make exceptions for features that are very simple, that are needed for stdlib stubs, or that solve a niche but important use case.

[dmwyatt]

One of the best features of TypeScript is interfaces (mostly for me because of REST APIs), and TypedDict is a nice python version of that functionality.

Jetbrains (aka @vlasovskikh) has said, that they will support typing_extensions but not mypy_extensions, so, @ilevkivskyi , this is my vote for getting TypedDict into typing_extensions.

Of course, I don't know how "in-flux" TypedDict is so maybe it's not stable enough to be pushed out to a wider audience. I guess @JukkaL would have to comment on that.

[JukkaL]

We are planning to add TypedDict to typing_extensions, probably in the next few months. It's pretty stable now and seems widely useful.

Since we now have typing_extensions, can this issue be closed? We can add new issues for particular features that might be added to typing_extensions.