Typing Documentation: NoReturn intent is ambiguous

[ChadBailey]

After lots of internal discussion, it seems that myself with some colleagues at work have concluded that the intent of NoReturn (official docs, pep484) is ambiguous, at least as listed in the docs.

Important: I mention mypy a lot in this, however, my concern isn't with mypy, it's with the intent becoming clear within the Python official documentation. I've only added all this context in hopes of making the concerns abundantly clear.

There are 2 schools of thought. The first is that NoReturn is only used when the function called never passes back control to the parent in the stack. This would mean all code below calling a function with NoReturn would be unreachable.

The second school of thought is that NoReturn is used for functions which perform processing but do not explicitly return a value. This would mean that code below calling a function with NoReturn may be reachable, but you should not ever assign the value of a function with NoReturn to a variable (this would be true in both schools of thought).

The closer we thought we were to an explanation, we found information that brought doubt back in. For now we have settled on Assumption1, but the docs could easily clear this up.

Here is some demo code to explain the point

from typing import NoReturn, Any

# Assumption1: NoReturn denotes that the function in question will not pass 
# control back to the calling function, making all code called below it 
# unreachable

# Assumption2: NoReturn denotes that the function in question may or may not
# pass control back to the calling function, but more importantly it does not 
# explicitly return anything and does not contain the return keyword or contains
# the return keyword by itself.


# Functions 1-6 pass mypy checks, comment out functions 7-8 to validate

def function1() -> NoReturn:
    """Exact behavior of scenario in official docs"""
    raise RuntimeError('no way')


def function2() -> NoReturn:
    """This function should be functionally identical to function7() in that it
    implicitly returns None but does not contain the return keyword. This 
    matches the 
    [mypy test implementation](https://github.com/ilevkivskyi/mypy/blob/5af9597a0a8e6b16075637145f15178130a659a1/test-data/unit/check-flags.test#L154)
    and does not generate the error 
    `error: Implicit return in function which does not return` in mypy that 
    function7 does, though it is functionally identical (as far as type goes).
    """
    pass


def function3() -> None:
    """This function implicitly returns None, but the return keyword is used. 
    If NoReturn is used here, the error 
    `error: Return statement in function which does not return` is generated
    in mypy."""
    return


def function4() -> None:
    """This function explicitly returns None"""
    return None


def function5() -> NoReturn:
    """Arbitrary function created to identify if mypy had special treatment of 
    functions with just `pass` in them"""
    pass


def function6() -> NoReturn:
    """This function should be functionally identical to function7() in that it
    implicitly returns None but does not contain the return keyword. This 
    matches the 
    [mypy test implementation](https://github.com/ilevkivskyi/mypy/blob/5af9597a0a8e6b16075637145f15178130a659a1/test-data/unit/check-flags.test#L154)
    and does not generate the error 
    `error: Implicit return in function which does not return` in mypy that 
    function7 does, though it is functionally identical (as far as type goes).
    """
    function5()
    function5()
    function5()


# Function7 does not pass mypy checks and function8 calls function7, comment out to validate

def function7() -> NoReturn:
    """This function performs actions, but does not explicitly return and does 
    not contain the return keyword. This generates the error 
    `error: Implicit return in function which does not return` in mypy. 
    Implicitly, the returned value is None"""
    1+1


def function8() -> NoReturn:
    """This function should be functionally identical to function7() in that it
    implicitly returns None but does not contain the return keyword. This 
    matches the 
    [mypy test implementation](https://github.com/ilevkivskyi/mypy/blob/5af9597a0a8e6b16075637145f15178130a659a1/test-data/unit/check-flags.test#L154)
    and does not generate the error 
    `error: Implicit return in function which does not return` in mypy that 
    function2 does, though it is functionally identical (as far as type goes).
    """
    function7()


def testing() -> None:  # This should be None if assumption1 is correct, NoReturn if assumption2 is correct
    a: None = function1()  # This should cause an error in all cases
    b: None = function2()  # This code is unreachable if assumption1 is correct. If assumption2 is correct it will cause an error like `assigning result of a function call, where the function has no return`
    c: None = function3()  # This code is unreachable if assumption1 is correct, and not necessarily cause an error if assumption2 is correct

    x = function2    # Valid use-case assigning a function to a variable
    y = function2()  # Valid use-case of when the intent might have been to assign a function to a variable, but instead the value of that function was assigned

Edited: Updated example code in efforts to be more clear

[srittau]

The first interpretation is the correct one. A bare or no return is equivalent to returning None and should be annotated as such.

[ChadBailey]

Thanks so much for your clarity on that. Do you think this warrants a tweak to the documentation to make that more obvious?

[Michael0x2a]

Regarding the behavior of your specific examples:

1. The behavior of function2 and function5 is actually due to a mypy feature/misfeature where it skips type checking the body of a function when it happens to be completely empty -- see Handle empty function bodies safely mypy#2350 for more details. For example, mypy would still continue to report no errors if you had changed the function signature to def function2() -> int: pass.

2. So since function7 isn't empty, mypy will type check as usual and report an error, since it would be inconsistent for a function to return None when it was declared to never return a value.

3. But with function8, you never actually do reach the end of the function: the only time when a function will never return a value is if it throws an exception. So, since you're calling function7(), the exception bubbles up and you never end up returning anything in function8() either. This is, of course, assuming that you first fix function7 to conform to the expected signature.

   (And same thing with function6: if function5 throws, so will function6.)

If it helps, mypy internally represents NoReturn as the equivalent to Haskell's bottom type or TypeScript's Never type -- basically, the type corresponding to the empty set of values.

This is strictly speaking an implementation detail though: all PEP 484 says is that NoReturn is used to "annotation functions that never return normally [...] for example by unconditionally raising an exception". Though personally, I find this restriction to be mildly unfortunate/inconvenient. python/mypy#5818 has some related discussion.

Also, just to give some context, I think the official typing docs historically have usually been updated only as an afterthought to the core typing design/implementation work. So I'm sure there's lots of low-hanging fruit there.

[ChadBailey]

If it helps, mypy internally represents NoReturn as the equivalent to Haskell's bottom type or TypeScript's Never type -- basically, the type corresponding to the empty set of values.

This makes sense, and is essentially what we had concluded on before I wrote up this bug report. None the less, in some languages there doesn't appear to be a clear distinction between the concepts of none and never so this may cause confusion among people who typing is new to and they come from a background where no distinction is made. C's void type is what I have in mind - but perhaps I've misunderstood the void type all along.

[ChadBailey]

1. The behavior of function2 and function5 is actually due to a mypy feature/misfeature where it skips type checking the body of a function when it happens to be completely empty -- see python/mypy#2350 for more details. For example, mypy would still continue to report no errors if you had changed the function signature to def function2() -> int: pass.

This is incredibly helpful, thanks for the explanation! This was the primary driver behind the confusion as I expected it to fail.

I had a few notes that were pointing to the wrong functions due to changes in the numbers, so I have edited my post to fix some of those (more might exist). I should have bothered figuring out nice names for them rather than arbitrary numbers. To be clear, the only one that actually fails is function7, but function8 depends on function7 so they both have to be commented to pass tests