Setting up for developing pfun

[chuckwondo]

In order to be able to submit PRs, I've forked pfun, and I'm trying to prepare for development by following the steps in the Development section of the README.

I first installed Python 3.7.13 since 3.7 is given as the earliest supported version in pyproject.toml. I used pyenv to do so. I then ran poetry env use 3.7.13 to create a virtualenv.

After activating the new environment, I ran the first command under the Development section of the README, but it failed while attempting to install pfun:

$ poetry install -E http -E sql
Installing dependencies from lock file

...(dependencies installed)...

Installing the current project: pfun (0.13.1)
  EnvCommandError

  Command ['/Users/charlie/Library/Caches/pypoetry/virtualenvs/pfun-meDSMSL9-py3.7/bin/pip', 'install', '-e', '/Users/charlie/src/chuckwondo/pfun', '--no-deps'] errored with the following return code 1, and output: 
  Obtaining file:///Users/charlie/src/chuckwondo/pfun
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'done'
  Installing collected packages: pfun
    Running setup.py develop for pfun
      error: subprocess-exited-with-error
      
      × python setup.py develop did not run successfully.
      │ exit code: 1
      ╰─> [15 lines of output]
          running develop
          running egg_info
          writing src/pfun.egg-info/PKG-INFO
          writing dependency_links to src/pfun.egg-info/dependency_links.txt
          writing requirements to src/pfun.egg-info/requires.txt
          writing top-level names to src/pfun.egg-info/top_level.txt
          reading manifest file 'src/pfun.egg-info/SOURCES.txt'
          adding license file 'LICENSE'
          writing manifest file 'src/pfun.egg-info/SOURCES.txt'
          running build_ext
          building 'pfun.effect' extension
          clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/charlie/Library/Caches/pypoetry/virtualenvs/pfun-meDSMSL9-py3.7/include -I/Users/charlie/.pyenv/versions/3.7.13/include/python3.7m -c src/pfun/effect.c -o build/temp.macosx-12.4-x86_64-3.7/src/pfun/effect.o
          clang: error: no such file or directory: 'src/pfun/effect.c'
          clang: error: no input files
          error: command 'clang' failed with exit status 1
          [end of output]
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error
  
  × python setup.py develop did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      running develop
      running egg_info
      writing src/pfun.egg-info/PKG-INFO
      writing dependency_links to src/pfun.egg-info/dependency_links.txt
      writing requirements to src/pfun.egg-info/requires.txt
      writing top-level names to src/pfun.egg-info/top_level.txt
      reading manifest file 'src/pfun.egg-info/SOURCES.txt'
      adding license file 'LICENSE'
      writing manifest file 'src/pfun.egg-info/SOURCES.txt'
      running build_ext
      building 'pfun.effect' extension
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Users/charlie/Library/Caches/pypoetry/virtualenvs/pfun-meDSMSL9-py3.7/include -I/Users/charlie/.pyenv/versions/3.7.13/include/python3.7m -c src/pfun/effect.c -o build/temp.macosx-12.4-x86_64-3.7/src/pfun/effect.o
      clang: error: no such file or directory: 'src/pfun/effect.c'
      clang: error: no input files
      error: command 'clang' failed with exit status 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  

  at ~/.poetry/lib/poetry/utils/env.py:1183 in _run
      1179│                 output = subprocess.check_output(
      1180│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1181│                 )
      1182│         except CalledProcessError as e:
    → 1183│             raise EnvCommandError(e, input=input_)
      1184│ 
      1185│         return decode(output)
      1186│ 
      1187│     def execute(self, bin, *args, **kwargs):

Any idea how I can resolve this?

[suned]

Just realised the readme is a bit outdated, sorry about that. Try

poetry run task install -E http -E sql

This alias will first compile the cython sources to C and then run poetry install.

[chuckwondo]

That did the trick. Thank you!

[chuckwondo]

One last issue with development steps:

$ poetry run task lint
trailing-whitespace-fixer................................................Passed
end-of-file-fixer........................................................Passed
isort....................................................................Passed
yapf.....................................................................Passed
flake8...................................................................Passed
mypy.....................................................................Failed
- hook id: mypy
- exit code: 1

src/pfun/aio_trampoline.py:81: error: Incompatible types in "await"
(actual type "Union[GeneratorType[Any, Any, Any], Coroutine[Any, Any, Any]]", expected type "Awaitable[Any]")
Found 1 error in 1 file (checked 27 source files)

[suned]

Ah, I was able to reproduce the error locally with python 3.7 (I'm normally on 3.10, which we use in CI as well for the lint step). Thats a bit incovenient, since really it should just be ignored. However we also use error on unused ignores, so that would become an error in Python 3.10.

[chuckwondo]

Interesting. I'm using Python 3.7 since that's the earliest version pfun supports.

I generally do that to prevent myself from accidentally using something introduced in a later of version of Python since that would then mean the code wouldn't work with earlier versions.

Aren't you concerned about developing in Python 3.10 and "accidentally" using newer syntax or functions when pfun should support as early as Python 3.7?

Using tox to run tests is one possible way to catch such cases.

[suned]

We test with all compatible python versions in CI so not very concerned no. But apparently we need to start linting with all compatible versions as well (or at least type checking) 😅

[chuckwondo]

Sounds good. So for the moment, it sounds like I should just develop with 3.10 to avoid this issue. Thanks for support and prompt replies!

[suned]

Yes that's probably the path of least resistance for now.