Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement extras_require #181

Merged
merged 19 commits into from
Jul 23, 2018
Merged

Implement extras_require #181

merged 19 commits into from
Jul 23, 2018

Conversation

flying-sheep
Copy link
Contributor

@flying-sheep flying-sheep commented May 29, 2018

Fixes #144

@takluyver
Copy link
Member

Thanks, I think this makes sense. Some thoughts coming up...

flit/common.py Outdated
dev_requires = ()
extras_require = {}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Metadata 2.0 was dropped, and metadata 2.1 is much closer to the 1.x formats. This metadata class is meant to closely represent the metadata we actually write/upload, so I'd rather add the extra requirements to requires_dist rather than adding a dict attribute to it.

The _prep_metadata() function in flit.inifile is responsible for converting the metadata we read from the file to a dict representing the standardised Python metadata.


These are not (yet) encoded in the wheel, but are used when doing
``flit install``.
These are installed by ``flit install`` and encoded in the wheel as extra ``dev``.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would make this undocumented (or maybe leave it for now but mark it as deprecated). The canonical way will become extras-require.dev.

@takluyver
Copy link
Member

takluyver commented Jun 2, 2018

Other stuff that I think this should have:

  • Validation (this should come for free if we put them in metadata.requires_dist as I suggested inline)
  • The docs for the --deps option should also explain that 'all' and 'develop' include the three special extras: test, docs and dev.
  • I think flit install should probably grow a --extras option to control these. E.g. flit install --deps production --extras test,gtk
    • Specifying --deps none and any extras should be an error
  • As you suggested on Support for extras_require #144, complain if both dev-requires and extras-require.dev are set. I would go as far as erroring out - "in the face of ambiguity, resist the temptation to guess"
  • Tests :-)

Also, one thing I'm still thinking about - input welcome: I don't like the name extras-require much. I think the idea is that "these extras require those packages", but that's not how I think about it, and it's an awkward difference from requires with an s. I'm considering calling it requires-extras instead, to make the connection with the main requires field clearer.

@flying-sheep
Copy link
Contributor Author

I added some of your recommendations and will get to the others eventually

@flying-sheep
Copy link
Contributor Author

OK, all done, but I have more ideas for tests:

  • Test installing for some configurations
  • Test if the dep; extra==blah lines are written correctly

@flying-sheep
Copy link
Contributor Author

OK, done! now i’m happy with it.

@flying-sheep
Copy link
Contributor Author

flying-sheep commented Jul 3, 2018

OK @takluyver now i’m really happy! could you please check this?

@Mariatta
Copy link
Contributor

Mariatta commented Jul 15, 2018

This doesn't seem to work for me.

I pulled this PR and installed it locally, by doing flit install in where flit is checked out.
(I also had to first remove [tool.flit.metadata.extras-require] in this PR's pyproject.toml)

Then I did flit install --deps all on another package.

I have the following in the other package's pyproject.toml

...
[tool.flit.metadata]
....
requires = ["uritemplate>=3.0.0"]
...

[tool.flit.metadata.extras-require]
test = ["pytest>=3.0.0", "pytest-asyncio"]
doc = ["sphinx"]
aiohttp = ["aiohttp"]
treq = ["treq", "twisted[tls]"]
tornado = ["tornado"]
setup = ["pytest-runner>=2.11.0"]

...

When I ran flit install --deps all, only uritemplate gets installed.

Using Python 3.6.5 if it matters.

the two reserved extras ``test`` and ``doc`` as well as the extra ``dev``
are installed by ``flit install``. For example:

.. code-block:: toml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will emit sphinx warning message: WARNING: Pygments lexer name 'toml' is not known.
I'm using Sphinx 1.7.4.

Copy link
Contributor Author

@flying-sheep flying-sheep Jul 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, we could add pygments-github-lexers to the doc extra requirements

@flying-sheep
Copy link
Contributor Author

flying-sheep commented Jul 16, 2018

That’s weird! the tests do work after all. There must be something I missed 🤔

What does flit --debug install --deps all say?

@Mariatta
Copy link
Contributor

$ flit --debug install --deps all
Parsed arguments Namespace(debug=True, deps='all', extras=(), ini_file=PosixPath('pyproject.toml'), logo=False, pth_file=False, python='/Users/mariatta/github/flit/venv36/bin/python3.6', repository=None, subcmd='install', symlink=False, user=None)  D-flit
User site packages not available - env install                                                                                D-flit.install
User install? False                                                                                                           D-flit.install
Loading module gidgethub/__init__.py                                                                                           D-flit.common
Loading module gidgethub/__init__.py                                                                                           D-flit.common
Copying package file(s) from gidgethub                                                                                          I-flit.wheel
Adding gidgethub/__init__.py to zip file                                                                                        D-flit.wheel
Adding gidgethub/abc.py to zip file                                                                                             D-flit.wheel
Adding gidgethub/aiohttp.py to zip file                                                                                         D-flit.wheel
Adding gidgethub/routing.py to zip file                                                                                         D-flit.wheel
Adding gidgethub/sansio.py to zip file                                                                                          D-flit.wheel
Adding gidgethub/tornado.py to zip file                                                                                         D-flit.wheel
Adding gidgethub/treq.py to zip file                                                                                            D-flit.wheel
Adding gidgethub/test/__init__.py to zip file                                                                                   D-flit.wheel
Adding gidgethub/test/test_abc.py to zip file                                                                                   D-flit.wheel
Adding gidgethub/test/test_aiohttp.py to zip file                                                                               D-flit.wheel
Adding gidgethub/test/test_exceptions.py to zip file                                                                            D-flit.wheel
Adding gidgethub/test/test_routing.py to zip file                                                                               D-flit.wheel
Adding gidgethub/test/test_sansio.py to zip file                                                                                D-flit.wheel
Adding gidgethub/test/test_tornado.py to zip file                                                                               D-flit.wheel
Adding gidgethub/test/test_treq.py to zip file                                                                                  D-flit.wheel
Adding gidgethub/test/samples/convert_headers.py to zip file                                                                    D-flit.wheel
Adding gidgethub/test/samples/ping_urlencoded/200.json to zip file                                                              D-flit.wheel
Adding gidgethub/test/samples/ping_urlencoded/body to zip file                                                                  D-flit.wheel
Adding gidgethub/test/samples/pr_diff/200.json to zip file                                                                      D-flit.wheel
Adding gidgethub/test/samples/pr_diff/body to zip file                                                                          D-flit.wheel
Adding gidgethub/test/samples/pr_merged/204.json to zip file                                                                    D-flit.wheel
Adding gidgethub/test/samples/pr_merged/body to zip file                                                                        D-flit.wheel
Adding gidgethub/test/samples/pr_not_found/404.json to zip file                                                                 D-flit.wheel
Adding gidgethub/test/samples/pr_not_found/body to zip file                                                                     D-flit.wheel
Adding gidgethub/test/samples/pr_page_1/200.json to zip file                                                                    D-flit.wheel
Adding gidgethub/test/samples/pr_page_1/body to zip file                                                                        D-flit.wheel
Adding gidgethub/test/samples/pr_page_2/200.json to zip file                                                                    D-flit.wheel
Adding gidgethub/test/samples/pr_page_2/body to zip file                                                                        D-flit.wheel
Adding gidgethub/test/samples/pr_page_last/200.json to zip file                                                                 D-flit.wheel
Adding gidgethub/test/samples/pr_page_last/body to zip file                                                                     D-flit.wheel
Adding gidgethub/test/samples/pr_single/200.json to zip file                                                                    D-flit.wheel
Adding gidgethub/test/samples/pr_single/body to zip file                                                                        D-flit.wheel
Writing metadata files                                                                                                          I-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/entry_points.txt in zip file                                                     D-flit.wheel
Adding LICENSE to zip file                                                                                                      D-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/WHEEL in zip file                                                                D-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/METADATA in zip file                                                             D-flit.wheel
Writing the record of files                                                                                                     I-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/RECORD in zip file                                                               D-flit.wheel
Requirement already satisfied: gidgethub==3.0.0.dev0 from file:///var/folders/xg/b8lym1jx549c9q1qc8cf7b0c0000gn/T/tmpyehqf78k/gidgethub-3.0.0.dev0-py3-none-any.whl in /Users/mariatta/github/flit/venv36/lib/python3.6/site-packages (3.0.0.dev0)
Requirement already satisfied: uritemplate>=3.0.0 in /Users/mariatta/github/flit/venv36/lib/python3.6/site-packages (from gidgethub==3.0.0.dev0) (3.0.0)

@flying-sheep
Copy link
Contributor Author

flying-sheep commented Jul 16, 2018

Hmm, before I debug myself, could you please quickly try with the newest commit?

Please also try flit --debug install --deps dev so I can see if the problem is that extra_reqs in my code is somehow empty

@Mariatta
Copy link
Contributor

I tried both --deps develop and --deps all:

$ flit --debug install --deps all
Parsed arguments Namespace(debug=True, deps='all', extras=(), ini_file=PosixPath('pyproject.toml'), logo=False, pth_file=False, python='/Users/mariatta/github/flit/venv36/bin/python3.6', repository=None, subcmd='install', symlink=False, user=None)  D-flit
User site packages not available - env install                                                                                                                      D-flit.install
User install? False                                                                                                                                                 D-flit.install
Loading module gidgethub/__init__.py                                                                                                                                 D-flit.common
Loading module gidgethub/__init__.py                                                                                                                                 D-flit.common
Copying package file(s) from gidgethub                                                                                                                                I-flit.wheel
Adding gidgethub/__init__.py to zip file                                                                                                                              D-flit.wheel
Adding gidgethub/abc.py to zip file                                                                                                                                   D-flit.wheel
Adding gidgethub/aiohttp.py to zip file                                                                                                                               D-flit.wheel
Adding gidgethub/routing.py to zip file                                                                                                                               D-flit.wheel
Adding gidgethub/sansio.py to zip file                                                                                                                                D-flit.wheel
Adding gidgethub/tornado.py to zip file                                                                                                                               D-flit.wheel
Adding gidgethub/treq.py to zip file                                                                                                                                  D-flit.wheel
Adding gidgethub/test/__init__.py to zip file                                                                                                                         D-flit.wheel
Adding gidgethub/test/test_abc.py to zip file                                                                                                                         D-flit.wheel
Adding gidgethub/test/test_aiohttp.py to zip file                                                                                                                     D-flit.wheel
Adding gidgethub/test/test_exceptions.py to zip file                                                                                                                  D-flit.wheel
Adding gidgethub/test/test_routing.py to zip file                                                                                                                     D-flit.wheel
Adding gidgethub/test/test_sansio.py to zip file                                                                                                                      D-flit.wheel
Adding gidgethub/test/test_tornado.py to zip file                                                                                                                     D-flit.wheel
Adding gidgethub/test/test_treq.py to zip file                                                                                                                        D-flit.wheel
Adding gidgethub/test/samples/convert_headers.py to zip file                                                                                                          D-flit.wheel
Adding gidgethub/test/samples/ping_urlencoded/200.json to zip file                                                                                                    D-flit.wheel
Adding gidgethub/test/samples/ping_urlencoded/body to zip file                                                                                                        D-flit.wheel
Adding gidgethub/test/samples/pr_diff/200.json to zip file                                                                                                            D-flit.wheel
Adding gidgethub/test/samples/pr_diff/body to zip file                                                                                                                D-flit.wheel
Adding gidgethub/test/samples/pr_merged/204.json to zip file                                                                                                          D-flit.wheel
Adding gidgethub/test/samples/pr_merged/body to zip file                                                                                                              D-flit.wheel
Adding gidgethub/test/samples/pr_not_found/404.json to zip file                                                                                                       D-flit.wheel
Adding gidgethub/test/samples/pr_not_found/body to zip file                                                                                                           D-flit.wheel
Adding gidgethub/test/samples/pr_page_1/200.json to zip file                                                                                                          D-flit.wheel
Adding gidgethub/test/samples/pr_page_1/body to zip file                                                                                                              D-flit.wheel
Adding gidgethub/test/samples/pr_page_2/200.json to zip file                                                                                                          D-flit.wheel
Adding gidgethub/test/samples/pr_page_2/body to zip file                                                                                                              D-flit.wheel
Adding gidgethub/test/samples/pr_page_last/200.json to zip file                                                                                                       D-flit.wheel
Adding gidgethub/test/samples/pr_page_last/body to zip file                                                                                                           D-flit.wheel
Adding gidgethub/test/samples/pr_single/200.json to zip file                                                                                                          D-flit.wheel
Adding gidgethub/test/samples/pr_single/body to zip file                                                                                                              D-flit.wheel
Writing metadata files                                                                                                                                                I-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/entry_points.txt in zip file                                                                                           D-flit.wheel
Adding LICENSE to zip file                                                                                                                                            D-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/WHEEL in zip file                                                                                                      D-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/METADATA in zip file                                                                                                   D-flit.wheel
Writing the record of files                                                                                                                                           I-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/RECORD in zip file                                                                                                     D-flit.wheel
Processing /var/folders/xg/b8lym1jx549c9q1qc8cf7b0c0000gn/T/tmpryu0hiqu/gidgethub-3.0.0.dev0-py3-none-any.whl
Requirement already satisfied: uritemplate>=3.0.0 in /Users/mariatta/github/flit/venv36/lib/python3.6/site-packages (from gidgethub==3.0.0.dev0) (3.0.0)
Installing collected packages: gidgethub
Successfully installed gidgethub-3.0.0.dev0
$ pip uninstall gidgethub
Uninstalling gidgethub-3.0.0.dev0:
  Would remove:
    /Users/mariatta/github/flit/venv36/bin/gidgethub
    /Users/mariatta/github/flit/venv36/lib/python3.6/site-packages/gidgethub-3.0.0.dev0.dist-info/*
    /Users/mariatta/github/flit/venv36/lib/python3.6/site-packages/gidgethub/*
Proceed (y/n)? y
  Successfully uninstalled gidgethub-3.0.0.dev0
$ flit --debug install --deps develop
Parsed arguments Namespace(debug=True, deps='develop', extras=(), ini_file=PosixPath('pyproject.toml'), logo=False, pth_file=False, python='/Users/mariatta/github/flit/venv36/bin/python3.6', repository=None, subcmd='install', symlink=False, user=None)  D-flit
User site packages not available - env install                                                                                                                      D-flit.install
User install? False                                                                                                                                                 D-flit.install
Loading module gidgethub/__init__.py                                                                                                                                 D-flit.common
Loading module gidgethub/__init__.py                                                                                                                                 D-flit.common
Copying package file(s) from gidgethub                                                                                                                                I-flit.wheel
Adding gidgethub/__init__.py to zip file                                                                                                                              D-flit.wheel
Adding gidgethub/abc.py to zip file                                                                                                                                   D-flit.wheel
Adding gidgethub/aiohttp.py to zip file                                                                                                                               D-flit.wheel
Adding gidgethub/routing.py to zip file                                                                                                                               D-flit.wheel
Adding gidgethub/sansio.py to zip file                                                                                                                                D-flit.wheel
Adding gidgethub/tornado.py to zip file                                                                                                                               D-flit.wheel
Adding gidgethub/treq.py to zip file                                                                                                                                  D-flit.wheel
Adding gidgethub/test/__init__.py to zip file                                                                                                                         D-flit.wheel
Adding gidgethub/test/test_abc.py to zip file                                                                                                                         D-flit.wheel
Adding gidgethub/test/test_aiohttp.py to zip file                                                                                                                     D-flit.wheel
Adding gidgethub/test/test_exceptions.py to zip file                                                                                                                  D-flit.wheel
Adding gidgethub/test/test_routing.py to zip file                                                                                                                     D-flit.wheel
Adding gidgethub/test/test_sansio.py to zip file                                                                                                                      D-flit.wheel
Adding gidgethub/test/test_tornado.py to zip file                                                                                                                     D-flit.wheel
Adding gidgethub/test/test_treq.py to zip file                                                                                                                        D-flit.wheel
Adding gidgethub/test/samples/convert_headers.py to zip file                                                                                                          D-flit.wheel
Adding gidgethub/test/samples/ping_urlencoded/200.json to zip file                                                                                                    D-flit.wheel
Adding gidgethub/test/samples/ping_urlencoded/body to zip file                                                                                                        D-flit.wheel
Adding gidgethub/test/samples/pr_diff/200.json to zip file                                                                                                            D-flit.wheel
Adding gidgethub/test/samples/pr_diff/body to zip file                                                                                                                D-flit.wheel
Adding gidgethub/test/samples/pr_merged/204.json to zip file                                                                                                          D-flit.wheel
Adding gidgethub/test/samples/pr_merged/body to zip file                                                                                                              D-flit.wheel
Adding gidgethub/test/samples/pr_not_found/404.json to zip file                                                                                                       D-flit.wheel
Adding gidgethub/test/samples/pr_not_found/body to zip file                                                                                                           D-flit.wheel
Adding gidgethub/test/samples/pr_page_1/200.json to zip file                                                                                                          D-flit.wheel
Adding gidgethub/test/samples/pr_page_1/body to zip file                                                                                                              D-flit.wheel
Adding gidgethub/test/samples/pr_page_2/200.json to zip file                                                                                                          D-flit.wheel
Adding gidgethub/test/samples/pr_page_2/body to zip file                                                                                                              D-flit.wheel
Adding gidgethub/test/samples/pr_page_last/200.json to zip file                                                                                                       D-flit.wheel
Adding gidgethub/test/samples/pr_page_last/body to zip file                                                                                                           D-flit.wheel
Adding gidgethub/test/samples/pr_single/200.json to zip file                                                                                                          D-flit.wheel
Adding gidgethub/test/samples/pr_single/body to zip file                                                                                                              D-flit.wheel
Writing metadata files                                                                                                                                                I-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/entry_points.txt in zip file                                                                                           D-flit.wheel
Adding LICENSE to zip file                                                                                                                                            D-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/WHEEL in zip file                                                                                                      D-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/METADATA in zip file                                                                                                   D-flit.wheel
Writing the record of files                                                                                                                                           I-flit.wheel
Writing data to gidgethub-3.0.0.dev0.dist-info/RECORD in zip file                                                                                                     D-flit.wheel
Processing /var/folders/xg/b8lym1jx549c9q1qc8cf7b0c0000gn/T/tmp1nzkuqlx/gidgethub-3.0.0.dev0-py3-none-any.whl
Requirement already satisfied: uritemplate>=3.0.0 in /Users/mariatta/github/flit/venv36/lib/python3.6/site-packages (from gidgethub==3.0.0.dev0) (3.0.0)
Installing collected packages: gidgethub
Successfully installed gidgethub-3.0.0.dev0

@flying-sheep
Copy link
Contributor Author

OK, so what I missed is that install_requirements is only called when not installing via pip (i.e. only when pth or symlink is set.

I fixed it, and it works now!

@Mariatta
Copy link
Contributor

Thanks!
flit install --deps all now works, it installs all the extras.

flit install --deps develop doesn't seem to be working, at least not to my expectation.
I had expected it to install test and docs extras, but it didn't install any of those.

@flying-sheep
Copy link
Contributor Author

flying-sheep commented Jul 16, 2018

Fixed, that was an oversight! Thank you for helping, that was a bigger PR than anticipated

@Mariatta
Copy link
Contributor

Mariatta commented Jul 16, 2018

Awesome!
I confirmed that with the latest change, --deps develop now installs test and doc extras, and --deps all still works too.

Thank you!

(I didn't look at your code).

@flying-sheep
Copy link
Contributor Author

Great! Now I also added the doc requirement. That should be it.

In order to get the TOML highlighting in the docs as well, we need to install pygments-github-lexers on readthedocs, but that needs @takluyver’s intervention

Copy link
Member

@takluyver takluyver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your patience, I know I've been slow to get round to reviewing this. The actual implementation is looking good now, but there are still a few changes I'd like.

dev-requires
Packages that are required for development. This field is in the same format
as ``requires``.
extras-require
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like the user facing name for this field to be requires-extra or requires-extras (I don't have a strong preference between these two - thoughts?). I like the symmetry between requires, requires-python and requires-extra[s].

I think of this like "the package requires these packages, this version of Python, and optionally these further packages for extra features."

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requires-extras and requires-extra imply that the package requires the extras. However it’s the extra features that require certain dependencies. So only extras-require says the right thing, don’t you agree?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I don't. The naming extras_require has always felt unintuitive to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, requires-extra makes a little more sense, I’m going for that one then.

inf = read_pkg_ini(samples_dir / 'module1-pkg.toml')
assert inf['module'] == 'module1'
assert inf['metadata']['home_page'] == 'http://github.com/sirrobin/module1'

def test_misspelled_key():
def test_misspelled_key(samples_dir):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand what you've done here, and I'm a big fan of py.test's capabilities, but in this case I prefer the simplicity of a global variable that the test functions use over the wizardry of a py.test fixture. Sorry, but I'm going to ask you to revert all these samples_dir changes.

@@ -82,7 +82,8 @@ Install the package on your system.
.. option:: --deps <dependency option>

Which dependencies to install. One of ``all``, ``production``, ``develop``,
or ``none``. Default ``all``.
or ``none``. ``all`` and ``develop`` install the extras ``test``, ``docs``,
and ``dev``. Default ``all``.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't forget to document the new --extras option too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@takluyver
Copy link
Member

Thanks! Now I think the last bit that needs to be done is adding the --extras command line option to cmdline.rst.

@flying-sheep
Copy link
Contributor Author

done!

@takluyver
Copy link
Member

Thank-you :-)

@takluyver takluyver merged commit fcdb3ee into pypa:master Jul 23, 2018
@flying-sheep flying-sheep deleted the extras branch July 23, 2018 09:26
@flying-sheep
Copy link
Contributor Author

No problem 😀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants