Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extra gcs and s3 #339

Draft
wants to merge 13 commits into
base: master
Choose a base branch
from
Draft

extra gcs and s3 #339

wants to merge 13 commits into from

Conversation

yokomotod
Copy link
Collaborator

@yokomotod yokomotod commented Dec 14, 2023

Problem

  • now we always have to install both s3 and gcs dependencies
  • it causes
    • more lock problem, since python's dependency resolving(like Poetry) is fragile
    • unnecessary install time, disk usage, container image size

Goal

  • from gokart USER
    • can separate [s3] and [gcs] extras to reduce unnecessary dependencies
      • pip install gokart: not install gcs, s3 related deps
      • pip install gokart[s3]: install with s3 related deps
      • pip install gokart[gcs]: install with gcs related deps
      • pip install gokart[s3][gcs]: install both (results same as with previous behavior)

Sub Goal

  • from gokart DEVELOPER
    • can unit tests that
      • core logic works without any extra deps. no unintended access to extra libs
      • each extra logic works with its dependency
      • extra dependencies are installed correctly
    • usually, developers can run all unit tests without being conscious of extras
      • extras are installed by default
      • (update)
        • this goal makes confusion of pyproject.toml, so after discussion, I drop this goal
        • and developer will poetry install --extras=s3 --extras=gcs --with=dev-s3 explicitly
      • all test for extras will be executed

Discussion

  • how it's confusing to need gokart[s3] or gokart[gcs] to works
  • do we have to MAJOR update because this is breaking change?
    • (update) discussed and maybe YES!

ToDO

test publishing on pypy dev

gokart/object_storage.py Outdated Show resolved Hide resolved
pyproject.toml Outdated Show resolved Hide resolved
tox.ini Outdated Show resolved Hide resolved
@yokomotod
Copy link
Collaborator Author

yokomotod commented Dec 20, 2023

other option:

  • separate pypi package?
    • gokart-gcs gokart-s3
      • (update)
        • NO. it will cause combinatorial explosion if we have more extras
        • and pip extra is straight forward for this purpose

@yokomotod
Copy link
Collaborator Author

@hirosassa Hi! How do your team feel, as another well know users? 😃

especially # discussion points on description

@KAZYPinkSaurus
Copy link

unnecessary install time, disk usage, container image size

How much does it actually affect disk size and install time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants