asyncio.as_completed() raises TypeError when the first supplied parameter is a generator that yields awaitables

[alexdelorenzo]

BPO	44176
Nosy	@asvetlov, @1st1, @alexdelorenzo
PRs	bpo-44176: Allow asyncio.as_completed()'s first parameter to be a generator yielding awaitables #26228
Files	as_completed_gen.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-05-19.05:04:33.179>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.7', 'library', 'expert-asyncio']
title = 'asyncio.as_completed() raises TypeError when the first supplied parameter is a generator that yields awaitables'
updated_at = <Date 2021-05-19.05:53:09.349>
user = 'https://github.com/alexdelorenzo'

bugs.python.org fields:

activity = <Date 2021-05-19.05:53:09.349>
actor = 'alexdelorenzo'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'asyncio']
creation = <Date 2021-05-19.05:04:33.179>
creator = 'alexdelorenzo'
dependencies = []
files = ['50052']
hgrepos = []
issue_num = 44176
keywords = []
message_count = 1.0
messages = ['393923']
nosy_count = 3.0
nosy_names = ['asvetlov', 'yselivanov', 'alexdelorenzo']
pr_nums = ['26228']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44176'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• GH-88342: clarify that asyncio.as_completed accepts generators yielding tasks #103626

[alexdelorenzo]

According to the documentation, asyncio.as_completed() takes a positional argument, aws, as an iterable of awaitables[1]:

    asyncio.as_completed(aws, *, loop=None, timeout=None)
      Run awaitable objects in the aws iterable concurrently.

As seen in the attached as_completed_gen.py file, built-in containers like lists, and iterators over them, are accepted by as_completed() as the first parameter without raising an error.

However, as_completed() raises TypeError if it is called with an iterable of awaitables that is also a generator. There are examples of this behavior in as_completed_gen.py, but here is a short example using a generator expression in the main() coroutine function:

    from asyncio import run, as_completed
    
    async def example(): pass
    
    async def main():
      coros = (example() for _ in range(10))
    
      for coro in as_completed(coros):  # raises TypeError
        await coro
    
    run(main())

Running that example will raise a TypeError with this message:

TypeError: expect an iterable of futures, not generator

If we look at the first line in the body of as_completed(), we can see why this error is thrown for generators that yield awaitables:

    def as_completed(fs, *, loop=None, timeout=None):
      if futures.isfuture(fs) or coroutines.iscoroutine(fs):
        raise TypeError(f"expect an iterable of futures, not {type(fs).__name__}")
      ...

Because generators are coroutines, and the first condition in as_completed() is True, and TypeError gets raised:

    from asyncio import coroutines

# generators and generator expressions are coroutines
assert coroutines.iscoroutine(example() for _ in range(10))

Perhaps as_completed() can use inspect.isawaitable() instead, like so:

    from inspect import isawaitable

    def as_completed(fs, *, loop=None, timeout=None):
      if futures.isfuture(fs) or isawaitable(fs):
        ...

I made a pull request with that change here[2].

[1] https://docs.python.org/3/library/asyncio-task.html#asyncio.as_completed

[2] #26228

[kumaraditya303]

3.12 allows generators so this is completed.

[kumaraditya303]

#103626 clarifies this in doc.