Fix for "Why the --no-deps in build.py?" (Issue #2529)

[pdallair]

I've created a very simple Test App app to test my changes.

Test App

main.py

iimport pkg_resources
import platform

from kivy.app import App
from kivy.uix.anchorlayout import AnchorLayout
from kivy.uix.boxlayout import BoxLayout
from kivy.uix.gridlayout import GridLayout
from kivy.uix.label import Label
from kivy.logger import Logger

import pymake    # Doesn't have a recipe. Has requirements that have recipes.
import telenium  # Doesn't have a recipe. Doesn't have requirements that have recipes.


class TestApp(App):
    def build(self):
        root_layout = AnchorLayout(anchor_x="center", anchor_y="center")
        intermediate_layout = BoxLayout(orientation="vertical", size_hint=(0.75, 0.75))
        widget_layout = GridLayout(cols=4)

        root_layout.add_widget(intermediate_layout)

        intermediate_layout.add_widget(Label(text='Test App', font_size=32, size_hint=(1.0, None)))
        intermediate_layout.add_widget(widget_layout)

        widget_layout.add_widget(Label(text=f'Python', font_size=16))
        widget_layout.add_widget(Label(text=platform.python_version(), font_size=16))
        for installed_package in pkg_resources.working_set:
            pkg_name = installed_package.key
            pkg_version = installed_package.version
            Logger.info(f'{pkg_name:>33s}=={pkg_version:<10s}')
            widget_layout.add_widget(Label(text=pkg_name, font_size=16))
            widget_layout.add_widget(Label(text=pkg_version, font_size=16))

        return root_layout


if __name__ == "__main__":
    TestApp().run()

buildozer.spec

[app]
title = Test App
package.name = testapp
package.domain = org.kivy
source.dir = .
source.include_exts = py,png,jpg,kv,atlas
version = 0.1
requirements = python3==3.8.10,kivy,mfpymake,telenium
orientation = portrait
fullscreen = 0
android.permissions = INTERNET,ACCESS_NETWORK_STATE
android.arch = armeabi-v7a

[buildozer]
log_level = 2
warn_on_root = 1

Logs/Output

buildozer android clean debug deploy output

download link

logcat

download link

Although it's not apparent here the user can still specify which version of a requirement they want using the usual syntax., even it's not a direct requirement. For e.g. in this case the default version of numpy (1.18.1) supported by the recipe was installed but if the user wants to they could specify numpy==<another_version> in the requirement list. The problem becomes whether or not the recipe supports that version or not.

[misl6]

Nice work!
Will definitely test it as soon I have a chance.

Meanwhile, some minor improvements:

• Flake8 tests are failing, can you take it a look ?
• How about adding a test, so the bug doesn't get introduced back in future ?
• Is the psutil recipe strictly related to this PR or could be moved into a separate PR?

[pdallair]

Thank you! 😄 I appreciate your compliment and collaboration. 👍

flake8 issues ✔️

It was just a few trailing white-spaces so I removed them and amended the commit.

adding a test ✔️

Like a unit test to make sure the list is expanded as expected? I'll start working on that. Let me know if that's not what you meant. Done! Though the test might fail at some point if the requirements of our requirements have changed.

psutil recipe ✔️

Yeah... I it's not necessary for this feature. psutils doesn't even get installed with my current requirement list. I think it was just for one of the previous libraries that I used to test my changes. Still, judging from the github insights, it's apparently being referred to in over 4k+ packages (100k+ repositories) so maybe it wouldn't be bad to keep this recipe now that it's made, correct? I will move it to another PR like you suggest.
Just going to drop it. All method calls I tried ran into permission issues and I simply don't personally need this library.

[misl6]

Yes, a unit test that checks the dependencies list expansion.
Do not consider that to be mandatory, just would be a nice to have.

Thank you for taking care about the flake8failure and the psutil recipe 😄

[misl6]

Yep, never tried it, butpsutil will unlikely work (apart some minor features), unless the device is rooted.
A recipe for psutil was present around 2014, but never been migrated to the newer toolchain (the migration has been done in later 2015)

[pdallair]

Man, I don't understand why this action failed. When I run make test locally, I don't get that error.

(p4a-dev) pdallair@ubuntu-vbox:~/dev/PycharmProjects/PythonForAndroid$ make test
`which tox` -- tests/ --ignore tests/test_pythonpackage.py
GLOB sdist-make: /home/pdallair/dev/PycharmProjects/PythonForAndroid/setup.py
pep8 inst-nodeps: /home/pdallair/dev/PycharmProjects/PythonForAndroid/.tox/.tmp/package/1/python-for-android-2021.9.5.zip
pep8 installed: appdirs==1.4.4,click==8.0.3,colorama==0.4.4,flake8==4.0.1,importlib-metadata==4.10.0,Jinja2==3.0.3,MarkupSafe==2.0.1,mccabe==0.6.1,pep517==0.6.0,pip-tools==6.4.0,pycodestyle==2.8.0,pyflakes==2.4.0,python-for-android==2021.9.5,pytoml==0.1.21,sh==1.14.2,six==1.16.0,toml==0.10.2,zipp==3.6.0
pep8 run-test-pre: PYTHONHASHSEED='2143996071'
pep8 run-test: commands[0] | flake8 pythonforandroid/ tests/ ci/ setup.py
py3 create: /home/pdallair/dev/PycharmProjects/PythonForAndroid/.tox/py3
py3 installdeps: pytest, coveralls, backports.tempfile
py3 inst: /home/pdallair/dev/PycharmProjects/PythonForAndroid/.tox/.tmp/package/1/python-for-android-2021.9.5.zip
py3 installed: appdirs==1.4.4,attrs==21.2.0,backports.tempfile==1.0,backports.weakref==1.0.post1,certifi==2021.10.8,charset-normalizer==2.0.9,click==8.0.3,colorama==0.4.4,coverage==6.2,coveralls==3.3.1,docopt==0.6.2,idna==3.3,importlib-metadata==4.10.0,iniconfig==1.1.1,Jinja2==3.0.3,MarkupSafe==2.0.1,packaging==21.3,pep517==0.6.0,pip-tools==6.4.0,pluggy==1.0.0,py==1.11.0,pyparsing==3.0.6,pytest==6.2.5,python-for-android==2021.9.5,pytoml==0.1.21,requests==2.26.0,sh==1.14.2,six==1.16.0,toml==0.10.2,urllib3==1.26.7,zipp==3.6.0
py3 run-test-pre: PYTHONHASHSEED='2143996071'
py3 run-test: commands[0] | coverage run --branch --source=pythonforandroid -m pytest tests --ignore tests/test_pythonpackage.py
=================================================================================================================================================== test session starts ===================================================================================================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
cachedir: .tox/py3/.pytest_cache
rootdir: /home/pdallair/dev/PycharmProjects/PythonForAndroid
collected 214 items                                                                                                                                                                                                                                                                                                       

tests/test_androidmodule_ctypes_finder.py ...                                                                                                                                                                                                                                                                       [  1%]
tests/test_archs.py ......                                                                                                                                                                                                                                                                                          [  4%]
tests/test_bootstrap.py ..........................                                                                                                                                                                                                                                                                  [ 16%]
tests/test_build.py ...                                                                                                                                                                                                                                                                                             [ 17%]
tests/test_distribution.py ..........                                                                                                                                                                                                                                                                               [ 22%]
tests/test_graph.py ............................                                                                                                                                                                                                                                                                    [ 35%]
tests/test_logger.py .                                                                                                                                                                                                                                                                                              [ 35%]
tests/test_pythonpackage_basic.py .........                                                                                                                                                                                                                                                                         [ 40%]
tests/test_recipe.py ................                                                                                                                                                                                                                                                                               [ 47%]
tests/test_recommendations.py ...........                                                                                                                                                                                                                                                                           [ 52%]
tests/test_toolchain.py ......                                                                                                                                                                                                                                                                                      [ 55%]
tests/test_util.py ......                                                                                                                                                                                                                                                                                           [ 58%]
tests/recipes/test_freetype.py ..                                                                                                                                                                                                                                                                                   [ 59%]
tests/recipes/test_gevent.py .                                                                                                                                                                                                                                                                                      [ 59%]
tests/recipes/test_harfbuzz.py ..                                                                                                                                                                                                                                                                                   [ 60%]
tests/recipes/test_hostpython3.py .......                                                                                                                                                                                                                                                                           [ 64%]
tests/recipes/test_icu.py ......                                                                                                                                                                                                                                                                                    [ 66%]
tests/recipes/test_jpeg.py ..                                                                                                                                                                                                                                                                                       [ 67%]
tests/recipes/test_leveldb.py ..                                                                                                                                                                                                                                                                                    [ 68%]
tests/recipes/test_libbz2.py .....                                                                                                                                                                                                                                                                                  [ 71%]
tests/recipes/test_libcurl.py ..                                                                                                                                                                                                                                                                                    [ 71%]
tests/recipes/test_libexpat.py ..                                                                                                                                                                                                                                                                                   [ 72%]
tests/recipes/test_libffi.py ...                                                                                                                                                                                                                                                                                    [ 74%]
tests/recipes/test_libgeos.py ..                                                                                                                                                                                                                                                                                    [ 75%]
tests/recipes/test_libiconv.py ..                                                                                                                                                                                                                                                                                   [ 76%]
tests/recipes/test_liblzma.py .....                                                                                                                                                                                                                                                                                 [ 78%]
tests/recipes/test_libmysqlclient.py ..                                                                                                                                                                                                                                                                             [ 79%]
tests/recipes/test_libogg.py ..                                                                                                                                                                                                                                                                                     [ 80%]
tests/recipes/test_libpq.py ..                                                                                                                                                                                                                                                                                      [ 81%]
tests/recipes/test_libsecp256k1.py ..                                                                                                                                                                                                                                                                               [ 82%]
tests/recipes/test_libshine.py ..                                                                                                                                                                                                                                                                                   [ 83%]
tests/recipes/test_libvorbis.py ..                                                                                                                                                                                                                                                                                  [ 84%]
tests/recipes/test_libvpx.py ..                                                                                                                                                                                                                                                                                     [ 85%]
tests/recipes/test_libx264.py ..                                                                                                                                                                                                                                                                                    [ 85%]
tests/recipes/test_libxml2.py ..                                                                                                                                                                                                                                                                                    [ 86%]
tests/recipes/test_libxslt.py ..                                                                                                                                                                                                                                                                                    [ 87%]
tests/recipes/test_openal.py ...                                                                                                                                                                                                                                                                                    [ 89%]
tests/recipes/test_openssl.py ......                                                                                                                                                                                                                                                                                [ 92%]
tests/recipes/test_pandas.py .                                                                                                                                                                                                                                                                                      [ 92%]
tests/recipes/test_png.py ..                                                                                                                                                                                                                                                                                        [ 93%]
tests/recipes/test_pyicu.py .                                                                                                                                                                                                                                                                                       [ 93%]
tests/recipes/test_python3.py ..........                                                                                                                                                                                                                                                                            [ 98%]
tests/recipes/test_reportlab.py .                                                                                                                                                                                                                                                                                   [ 99%]
tests/recipes/test_snappy.py ..                                                                                                                                                                                                                                                                                     [100%]

============================================================================================================================================= 214 passed in 397.15s (0:06:37) =============================================================================================================================================
py3 run-test: commands[1] | coverage report -m
Name                                                                     Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------------------------------------------------------
pythonforandroid/__init__.py                                                 1      0      0      0   100%
pythonforandroid/archs.py                                                  121      3     36      1    95%   239-241
pythonforandroid/bdistapk.py                                               102    102     66      0     0%   1-161
pythonforandroid/bootstrap.py                                              282     20    134     12    91%   33, 37, 118-121, 137-138, 165->167, 234-235, 279->276, 295, 305, 309, 313, 374->377, 384-389, 398->400
pythonforandroid/bootstraps/__init__.py                                      0      0      0      0   100%
pythonforandroid/bootstraps/empty/__init__.py                                9      0      2      0   100%
pythonforandroid/bootstraps/sdl2/__init__.py                                33      1      8      3    90%   21, 46->50, 50->52
pythonforandroid/bootstraps/service_library/__init__.py                      4      0      2      0   100%
pythonforandroid/bootstraps/service_only/__init__.py                        35      1      8      3    91%   29, 45->49, 49->51
pythonforandroid/bootstraps/webview/__init__.py                             34      1      8      3    90%   26, 42->46, 46->48
pythonforandroid/build.py                                                  594    313    244     32    41%   30-40, 44-56, 60-70, 74-78, 135, 173, 189, 202, 215, 228, 252, 273-279, 287-288, 290-291, 307, 314->317, 318-320, 322-324, 326-328, 331-340, 342, 349-350, 352-353, 366->369, 387, 395->394, 406-408, 427, 430, 480, 486, 499, 511-531, 539, 546-603, 610, 617-724, 756->761, 763, 804, 811, 833, 834->840, 848-883, 893-938, 942-1071
pythonforandroid/distribution.py                                           114      8     54     10    89%   118, 121, 136, 138, 140, 141->134, 173, 206, 218->220, 282
pythonforandroid/entrypoints.py                                              9      6      0      0    33%   10-20
pythonforandroid/graph.py                                                  178     11    112      9    93%   47, 50, 74, 76, 91, 135, 263->259, 289-292, 300, 321
pythonforandroid/logger.py                                                 148     39     62     11    64%   29, 38->45, 105, 118-119, 126, 142-143, 147, 160, 169, 172-174, 179->167, 180->167, 187, 192-231
pythonforandroid/patching.py                                                59     35      4      0    38%   6-8, 12-14, 19, 28-30, 34-36, 40-42, 46-48, 52-54, 58-60, 65, 70-72, 76-77, 81-83, 88
pythonforandroid/pythonpackage.py                                          316    127    159     25    58%   93, 96, 141-144, 168, 172, 191-192, 199-200, 214, 247-249, 257-259, 263, 274->272, 277, 301-436, 449-530, 606-608, 609->648, 632->640, 637, 659-660, 663->667, 687->684, 694, 725-730, 735, 736->738, 742->687, 783
pythonforandroid/recipe.py                                                 683    323    297     33    49%   160->exit, 181, 189, 193, 198-205, 208, 231-250, 267-270, 273-279, 291-295, 305-308, 311, 348-350, 375-380, 382, 392-405, 416-427, 430-502, 508, 517->520, 532, 545-564, 576, 582-584, 593, 606, 608->exit, 617, 633-651, 656-657, 662, 674, 681, 727, 736, 749-753, 772, 775-778, 784-792, 801-807, 810, 813, 816-820, 882-892, 901-902, 908, 913-916, 927->948, 941, 943->948, 945, 951-956, 961-962, 968-985, 988-990, 993-995, 1010-1012, 1015-1025, 1029-1031, 1034-1038, 1058-1060, 1063-1092, 1095-1101, 1107-1121, 1126-1132, 1135-1156, 1168-1169, 1199-1203, 1209-1212
pythonforandroid/recipes/Pillow/__init__.py                                 51     39     10      0    23%   36-95
pythonforandroid/recipes/__init__.py                                         0      0      0      0   100%
pythonforandroid/recipes/aiohttp/__init__.py                                 8      0      2      0   100%
pythonforandroid/recipes/android/__init__.py                                42     29     10      0    29%   20-22, 25-83
pythonforandroid/recipes/android/src/android/__init__.py                     1      0      0      0   100%
pythonforandroid/recipes/android/src/android/_ctypes_library_finder.py      38      3     21      6    85%   17->20, 21, 51, 53->55, 57, 65->60
pythonforandroid/recipes/apsw/__init__.py                                   23     11      2      0    56%   14-23, 26-31
pythonforandroid/recipes/argon2-cffi/__init__.py                            12      3      2      0    79%   12-14
pythonforandroid/recipes/atom/__init__.py                                    7      0      2      0   100%
pythonforandroid/recipes/audiostream/__init__.py                            26     13      2      0    54%   18-30, 40-43
pythonforandroid/recipes/babel/__init__.py                                   9      0      2      0   100%
pythonforandroid/recipes/bcrypt/__init__.py                                 15      6      2      0    65%   12-19
pythonforandroid/recipes/boost/__init__.py                                  47     27      6      0    42%   52-54, 60, 63-67, 73-80, 85-106
pythonforandroid/recipes/brokenrecipe/__init__.py                            5      0      2      0   100%
pythonforandroid/recipes/cdecimal/__init__.py                               16      6      6      0    55%   16-22
pythonforandroid/recipes/cffi/__init__.py                                   31     19      2      0    42%   22-26, 29-47
pythonforandroid/recipes/coverage/__init__.py                                9      0      2      0   100%
pythonforandroid/recipes/cppy/__init__.py                                    9      0      2      0   100%
pythonforandroid/recipes/cryptography/__init__.py                           15      6      2      0    65%   12-19
pythonforandroid/recipes/cymunk/__init__.py                                  6      0      2      0   100%
pythonforandroid/recipes/cython/__init__.py                                  9      0      2      0   100%
pythonforandroid/recipes/dateutil/__init__.py                                9      0      2      0   100%
pythonforandroid/recipes/decorator/__init__.py                               9      0      2      0   100%
pythonforandroid/recipes/enaml/__init__.py                                   8      0      2      0   100%
pythonforandroid/recipes/ethash/__init__.py                                  5      0      2      0   100%
pythonforandroid/recipes/evdev/__init__.py                                  13      3      2      0    80%   20-22
pythonforandroid/recipes/feedparser/__init__.py                              8      0      2      0   100%
pythonforandroid/recipes/ffmpeg/__init__.py                                 69     55     12      0    20%   15-16, 19, 22-24, 27-152
pythonforandroid/recipes/ffpyplayer/__init__.py                             21     11      4      0    48%   13-30
pythonforandroid/recipes/ffpyplayer_codecs/__init__.py                       6      1      2      0    88%   8
pythonforandroid/recipes/flask/__init__.py                                   9      0      2      0   100%
pythonforandroid/recipes/fontconfig/__init__.py                             13      3      2      0    80%   13-16
pythonforandroid/recipes/freetype/__init__.py                               55     16     16      5    65%   34-42, 54->exit, 73, 91-102, 118-121, 127-129
pythonforandroid/recipes/genericndkbuild/__init__.py                        19      7      2      0    67%   14, 17-22, 25-28
pythonforandroid/recipes/gevent/__init__.py                                 21      0      2      0   100%
pythonforandroid/recipes/greenlet/__init__.py                                7      0      2      0   100%
pythonforandroid/recipes/groestlcoin_hash/__init__.py                        7      0      2      0   100%
pythonforandroid/recipes/harfbuzz/__init__.py                               30      7      6      2    75%   31-40, 70-72
pythonforandroid/recipes/hostpython3/__init__.py                            58      0     16      2    97%   120->126, 122->120
pythonforandroid/recipes/icu/__init__.py                                    61      1     14      4    93%   47->49, 61, 74->91, 92->exit
pythonforandroid/recipes/ifaddrs/__init__.py                                37     22     10      0    36%   21-24, 28-51
pythonforandroid/recipes/jedi/__init__.py                                    7      0      2      0   100%
pythonforandroid/recipes/jpeg/__init__.py                                   19      0      2      0   100%
pythonforandroid/recipes/kivy3/__init__.py                                  13      3      2      0    80%   16-18
pythonforandroid/recipes/kivy/__init__.py                                   32     17     10      0    40%   18-30, 36-39, 42-53
pythonforandroid/recipes/kiwisolver/__init__.py                             15      7      4      0    53%   17-27
pythonforandroid/recipes/lapack/__init__.py                                 33     17      4      0    49%   24-30, 33-55
pythonforandroid/recipes/leveldb/__init__.py                                21      0      2      0   100%
pythonforandroid/recipes/libbz2/__init__.py                                 23      0      2      0   100%
pythonforandroid/recipes/libcurl/__init__.py                                23      0      2      0   100%
pythonforandroid/recipes/libexpat/__init__.py                               20      0      2      0   100%
pythonforandroid/recipes/libffi/__init__.py                                 23      0      4      1    96%   27->29
pythonforandroid/recipes/libgeos/__init__.py                                23      0      2      0   100%
pythonforandroid/recipes/libglob/__init__.py                                38     22     10      0    38%   30, 34-37, 41-63
pythonforandroid/recipes/libiconv/__init__.py                               16      0      2      0   100%
pythonforandroid/recipes/liblzma/__init__.py                                30      0      4      1    97%   22->24
pythonforandroid/recipes/libmysqlclient/__init__.py                         21      1      2      0    96%   23
pythonforandroid/recipes/libogg/__init__.py                                 15      0      2      0   100%
pythonforandroid/recipes/libpcre/__init__.py                                18      5      2      0    75%   16-25, 28
pythonforandroid/recipes/libpq/__init__.py                                  21      1      2      0    96%   18
pythonforandroid/recipes/librt/__init__.py                                  22     12     10      0    38%   23, 27-50
pythonforandroid/recipes/libsecp256k1/__init__.py                           17      0      4      1    95%   18->20
pythonforandroid/recipes/libshine/__init__.py                               24      0      2      0   100%
pythonforandroid/recipes/libsodium/__init__.py                              22      8      2      0    67%   16-27, 30-32
pythonforandroid/recipes/libtorrent/__init__.py                             64     46     22      0    23%   15-21, 57-60, 64-67, 71-134, 139-145
pythonforandroid/recipes/libtribler/__init__.py                              9      0      2      0   100%
pythonforandroid/recipes/libvorbis/__init__.py                              23      0      2      0   100%
pythonforandroid/recipes/libvpx/__init__.py                                 26      1      4      1    93%   39
pythonforandroid/recipes/libwebp/__init__.py                                23     10      2      0    60%   23-47
pythonforandroid/recipes/libx264/__init__.py                                21      1      4      1    92%   20
pythonforandroid/recipes/libxml2/__init__.py                                27      0      4      1    97%   19->21
pythonforandroid/recipes/libxslt/__init__.py                                36      0      4      1    98%   31->33
pythonforandroid/recipes/libzbar/__init__.py                                26     11      2      0    61%   22-27, 30-50
pythonforandroid/recipes/libzmq/__init__.py                                 21      8      2      0    65%   16-39
pythonforandroid/recipes/lxml/__init__.py                                   44     32      6      0    28%   15-24, 27-63
pythonforandroid/recipes/m2crypto/__init__.py                               26     12      4      0    53%   16-31, 35-37
pythonforandroid/recipes/matplotlib/__init__.py                             66     50      8      0    24%   37-68, 80-93, 96-104, 107-160
pythonforandroid/recipes/msgpack-python/__init__.py                          7      0      2      0   100%
pythonforandroid/recipes/mysqldb/__init__.py                                31     18      2      0    45%   19-23, 26-29, 32-49
pythonforandroid/recipes/netifaces/__init__.py                               9      0      2      0   100%
pythonforandroid/recipes/numpy/__init__.py                                  40     20      4      0    50%   27-38, 42-46, 50-52, 55-57
pythonforandroid/recipes/omemo-backend-signal/__init__.py                    9      0      2      0   100%
pythonforandroid/recipes/omemo/__init__.py                                   9      0      2      0   100%
pythonforandroid/recipes/openal/__init__.py                                 21      3      2      0    87%   16-18
pythonforandroid/recipes/opencv/__init__.py                                 40     24      4      0    41%   38, 41-44, 47-141
pythonforandroid/recipes/opencv_extras/__init__.py                           6      0      2      0   100%
pythonforandroid/recipes/openssl/__init__.py                                55      2     14      2    94%   63, 114
pythonforandroid/recipes/pandas/__init__.py                                 16      0      2      0   100%
pythonforandroid/recipes/pil/__init__.py                                    10      3      2      0    75%   16-20
pythonforandroid/recipes/png/__init__.py                                    18      0      2      0   100%
pythonforandroid/recipes/preppy/__init__.py                                  8      0      2      0   100%
pythonforandroid/recipes/protobuf_cpp/__init__.py                           81     56     16      0    28%   28-69, 72-96, 100-103, 109-126, 132-139
pythonforandroid/recipes/psycopg2/__init__.py                               26     13      4      0    50%   20-24, 29-32, 37-45
pythonforandroid/recipes/pyaml/__init__.py                                   8      0      2      0   100%
pythonforandroid/recipes/pycparser/__init__.py                               9      0      2      0   100%
pythonforandroid/recipes/pycrypto/__init__.py                               29     16      2      0    48%   19-30, 33-41
pythonforandroid/recipes/pycryptodome/__init__.py                            6      0      2      0   100%
pythonforandroid/recipes/pydantic/__init__.py                                8      0      2      0   100%
pythonforandroid/recipes/pygame/__init__.py                                 32     18      2      0    47%   27-54, 57-61
pythonforandroid/recipes/pyicu/__init__.py                                  17      0      4      0   100%
pythonforandroid/recipes/pyjnius/__init__.py                                18      4      2      0    80%   19-22
pythonforandroid/recipes/pyleveldb/__init__.py                              17      8      2      0    58%   13-24
pythonforandroid/recipes/pymunk/__init__.py                                 13      4      2      0    73%   12-15
pythonforandroid/recipes/pynacl/__init__.py                                 16      6      2      0    67%   14-26
pythonforandroid/recipes/pyogg/__init__.py                                   9      0      2      0   100%
pythonforandroid/recipes/pyopenal/__init__.py                                9      0      2      0   100%
pythonforandroid/recipes/pyopenssl/__init__.py                               8      0      2      0   100%
pythonforandroid/recipes/pyproj/__init__.py                                  7      0      2      0   100%
pythonforandroid/recipes/pyrxp/__init__.py                                   7      0      2      0   100%
pythonforandroid/recipes/pysdl2/__init__.py                                  6      0      2      0   100%
pythonforandroid/recipes/pysha3/__init__.py                                 17      8      2      0    58%   13-22
pythonforandroid/recipes/python3/__init__.py                               161     27     34     12    79%   76->83, 171, 188-189, 217, 232-234, 238-244, 249-253, 258-260, 274-275, 280->279, 283, 318->328, 383-384, 393-396
pythonforandroid/recipes/pytz/__init__.py                                    9      0      2      0   100%
pythonforandroid/recipes/pyusb/__init__.py                                   9      0      2      0   100%
pythonforandroid/recipes/pyzbar/__init__.py                                 17      8      2      0    58%   16-23
pythonforandroid/recipes/pyzmq/__init__.py                                  28     12      2      0    60%   22-31, 34-55
pythonforandroid/recipes/regex/__init__.py                                   8      0      2      0   100%
pythonforandroid/recipes/reportlab/__init__.py                              43      4     14      4    79%   15->exit, 25-27, 46, 48->exit
pythonforandroid/recipes/ruamel.yaml/__init__.py                             9      0      2      0   100%
pythonforandroid/recipes/scipy/__init__.py                                  39     26      2      0    37%   15-17, 20-22, 25-48
pythonforandroid/recipes/scrypt/__init__.py                                 17      8      2      0    58%   16-23
pythonforandroid/recipes/sdl2/__init__.py                                   18      6      2      0    70%   16-19, 22-25
pythonforandroid/recipes/sdl2_image/__init__.py                              7      0      2      0   100%
pythonforandroid/recipes/sdl2_mixer/__init__.py                              7      0      2      0   100%
pythonforandroid/recipes/sdl2_ttf/__init__.py                                6      0      2      0   100%
pythonforandroid/recipes/secp256k1/__init__.py                              16      6      2      0    67%   27-32
pythonforandroid/recipes/setuptools/__init__.py                              7      0      2      0   100%
pythonforandroid/recipes/shapely/__init__.py                                16      6      2      0    67%   21-31
pythonforandroid/recipes/six/__init__.py                                     6      0      2      0   100%
pythonforandroid/recipes/snappy/__init__.py                                 16      0      2      0   100%
pythonforandroid/recipes/spine/__init__.py                                   9      0      2      0   100%
pythonforandroid/recipes/sqlalchemy/__init__.py                              9      0      2      0   100%
pythonforandroid/recipes/sqlite3/__init__.py                                22      9      2      0    62%   14, 17-20, 24-26, 30-32
pythonforandroid/recipes/storm/__init__.py                                  12      2      2      0    86%   13-17
pythonforandroid/recipes/sympy/__init__.py                                   8      0      2      0   100%
pythonforandroid/recipes/twisted/__init__.py                                22      9      6      0    54%   18-26, 29-35
pythonforandroid/recipes/ujson/__init__.py                                   6      0      2      0   100%
pythonforandroid/recipes/vispy/__init__.py                                   7      0      2      0   100%
pythonforandroid/recipes/vlc/__init__.py                                    49     33     14      0    29%   21-43, 53-72
pythonforandroid/recipes/websocket-client/__init__.py                        6      0      2      0   100%
pythonforandroid/recipes/wsaccel/__init__.py                                 7      0      2      0   100%
pythonforandroid/recipes/x3dh/__init__.py                                   10      0      2      0   100%
pythonforandroid/recipes/xeddsa/__init__.py                                 19      6      2      0    71%   20-30
pythonforandroid/recipes/zbar/__init__.py                                   18      8      2      0    60%   23-30
pythonforandroid/recipes/zbarlight/__init__.py                              17      8      2      0    58%   16-23
pythonforandroid/recipes/zeroconf/__init__.py                                8      0      2      0   100%
pythonforandroid/recipes/zope/__init__.py                                   14      4      2      0    75%   14-19, 22
pythonforandroid/recipes/zope_interface/__init__.py                         21      6      2      0    74%   17-23, 26-28
pythonforandroid/recommendations.py                                         85      0     24      1    99%   107->exit
pythonforandroid/toolchain.py                                              628    279    244     27    50%   41-50, 56-57, 59-61, 64-65, 108, 109->112, 119->124, 147->153, 148->150, 177->179, 208->211, 227, 255-256, 308, 411, 613, 615, 617, 619, 634, 647-651, 664-687, 780-784, 851-852, 865-876, 880, 883, 887-897, 901-904, 911-918, 931, 936-941, 962-966, 970-985, 990-992, 997-999, 1003-1004, 1019-1026, 1037-1041, 1051-1066, 1076-1086, 1106-1124, 1132-1145, 1154-1207, 1219-1260, 1264-1266, 1270-1272, 1283-1286, 1290, 1295-1303, 1307-1312, 1321-1331, 1338, 1343, 1348-1361, 1364, 1368-1392
pythonforandroid/util.py                                                    57      2     20      2    95%   87-88, 104->106
--------------------------------------------------------------------------------------------------------------------
TOTAL                                                                     6569   2214   2051    216    63%
_________________________________________________________________________________________________________________________________________________________ summary _________________________________________________________________________________________________________________________________________________________
  pep8: commands succeeded
  py3: commands succeeded
  congratulations :)

[misl6]

get_ndk_platform_dir has been removed here: #2467.

Looks like your branch started before these changes. (As I can see, you started your branch from master, instead of develop).

[pdallair]

Alright, I rebased my changes onto develop and made the necessary changes to my unit test for it to pass locally. 🤞

[misl6]

Looks like the CI is now failing due to a missing dependency (one included in your tests) on python 3.6.

But, python 3.6 reached EOL 5 days ago. Let me open a PR to discuss the removal.

[misl6]

See: #2534

[misl6]

@pdallair #2534 has been merged, can you rebase on top of it? 😄

[pdallair]

Done.

[misl6]

Thank you for rebasing it!
Can you please address the requested changes?

[pdallair]

Hey @misl6 I've made and pushed some more changes. I may still have 1 pending change request because I only modified lines around the one you highlighted. Could review this again please? Thank you.

[misl6]

Hey @misl6 I've made and pushed some more changes. I may still have 1 pending change request because I only modified lines around the one you highlighted. Could review this again please? Thank you.

I was on vacation 😄 , sorry for the delay.
Will continue reviewing it tomorrow, meanwhile I approved the CI workflow run :)

[misl6]

Here, into this regex, we shouldn't only account for digits.
requirement_name = re.sub(r'==.+', '', requirement) or requirement.split('==')[0]should be good.

As an example: kivy==master is a pretty common use case and should be covered.

[pdallair]

Ah ok. I was not aware of this. I use this regex pattern in 3 places apparently. Would replacing them with ==[^\s]+, to include anything but whitespaces, do the trick?

[misl6]

Uhm, why whitespaces shouldn't be considered? I'm missing some context?

[misl6]

The results are mixed, but I'm sure we can find the right path.

Let's take this simplified real world example:
requirements=hostpython3==3.9.9, python3==3.9.9, kivy, android, pydal, git+https://github.com/kivy-garden/zbarcam@master

pip-compile result is:

[INFO]:    Compiling dependencies for: ['pydal', 'git+https://github.com/kivy-garden/zbarcam@master']
[DEBUG]:   -> running bash -c echo -e 'pydal
git+https://github.com/kivy-garden/zbarcam@master' > requirements.in && /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -m piptools compile --dry-run --annotation-style=line && rm requirements.in
[DEBUG]:   	#
[DEBUG]:   	# This file is autogenerated by pip-compile with python 3.9
[DEBUG]:   	# To update, run:
[DEBUG]:   	#
[DEBUG]:   	#    pip-compile --annotation-style=line
[DEBUG]:   	#
[DEBUG]:   	certifi==2021.10.8        # via requests
[DEBUG]:   	charset-normalizer==2.0.10  # via requests
[DEBUG]:   	docutils==0.18.1          # via kivy
[DEBUG]:   	idna==3.3                 # via requests
[DEBUG]:   	kivy==2.0.0               # via kivy-garden.xcamera, kivy-garden.zbarcam
[DEBUG]:   	kivy-garden==0.1.4        # via kivy
[DEBUG]:   	kivy-garden.xcamera==2020.613  # via xcamera
[DEBUG]:   	kivy-garden.zbarcam @ git+https://github.com/kivy-garden/zbarcam@master  # via -r requirements.in
[DEBUG]:   	numpy==1.22.0             # via kivy-garden.zbarcam, opencv-python
[DEBUG]:   	opencv-python==4.5.5.62   # via kivy-garden.xcamera, kivy-garden.zbarcam
[DEBUG]:   	pillow==9.0.0             # via kivy-garden.zbarcam
[DEBUG]:   	pydal==20210626.3         # via -r requirements.in
[DEBUG]:   	pygments==2.11.1          # via kivy
[DEBUG]:   	pyzbar==0.1.8             # via kivy-garden.zbarcam
[DEBUG]:   	requests==2.27.0          # via kivy-garden
[DEBUG]:   	urllib3==1.26.7           # via requests
[DEBUG]:   	xcamera==2020.613         # via kivy-garden.zbarcam
[DEBUG]:   	Dry-run, so nothing updated.

And I had the following expanded requirements list at the end:
[INFO]: Expanded Requirements List: ['hostpython3==3.9.9', 'python3==3.9.9', 'kivy', 'android', 'pydal', 'git+https://github.com/kivy-garden/zbarcam@master']

My expectations were something like that (the strikeout dependencies are not expected to be included):

certifi==2021.10.8 # via requests (Cause kivy-garden has been removed)
charset-normalizer==2.0.10 # via requests (Cause kivy-garden has been removed)
docutils==0.18.1 # via kivy (Requested by a parent which have a recipe)
idna==3.3 # via requests (Cause kivy-garden has been removed)
kivy==2.0.0 # via kivy-garden.xcamera, kivy-garden.zbarcam (Have a recipe)
kivy-garden==0.1.4 # via kivy (Requested by a parent which have a recipe)
kivy-garden.xcamera==2020.613 # via xcamera (plain python, installable via pip)
kivy-garden.zbarcam @ git+https://github.com/kivy-garden/zbarcam@master # via -r requirements.in
numpy==1.22.0 # via kivy-garden.zbarcam, opencv-python (We have a recipe, and the recipe should be built)
opencv-python==4.5.5.62 # via kivy-garden.xcamera, kivy-garden.zbarcam (A recipe is not present, will expect it to fail during install)
pillow==9.0.0 # via kivy-garden.zbarcam (We have a recipe, and the recipe should be built)
pydal==20210626.3 # via -r requirements.in (plain python, installable via pip)
pygments==2.11.1 # via kivy (Requested by a parent which have a recipe)
pyzbar==0.1.8 # via kivy-garden.zbarcam (We have a recipe, and the recipe should be built)
requests==2.27.0 # via kivy-garden (Cause kivy-garden has been removed)
urllib3==1.26.7 # via requests (Cause requests has been removed)
xcamera==2020.613 # via kivy-garden.zbarcam (plain python, installable via pip)

[pdallair]

git+https://github.com/kivy-garden/zbarcam@master

Aaaah, I was not aware we could specify requirements like this. I've managed to modify my code to work with the exact same syntax you've given me (lines up pretty well with the expectations you've given me).

Expanded Requirements List: ['hostpython3==3.9.9', 'python3==3.9.9', 'kivy', 'android', 'pydal', 'git+https://github.com/kivy-garden/zbarcam@master', 'kivy-garden.xcamera==2020.613', 'numpy', 'opencv-python==4.5.5.62', 'pillow', 'pyzbar', 'xcamera==2020.613']

However, there is still one case (at least) involving this syntax that I'm wondering about now:

kivy-garden.zbarcam did not have a recipe but what if the user specified something that did, using this syntax? Would it be correct to not look include it in the dependency compilation and simply use the recipe (as we do with the usual syntax)? If this is correct, I would need to parse the potential recipe name from the provided git+url statement. Is it safe to assume that @<branch_name> is always present? is it safe to assume everything between .com/ and @<branch_name> more or less equates to the related recipe name (e.g., git+https/github.com/foo/bar@master --> recipe named foo.bar)?

Are there any other supported syntaxes I should be aware of?

[misl6]

git+https://github.com/kivy-garden/zbarcam@master

Aaaah, I was not aware we could specify requirements like this. I've managed to modify my code to work with the exact same syntax you've given me (lines up pretty well with the expectations you've given me).

Expanded Requirements List: ['hostpython3==3.9.9', 'python3==3.9.9', 'kivy', 'android', 'pydal', 'git+https://github.com/kivy-garden/zbarcam@master', 'kivy-garden.xcamera==2020.613', 'numpy', 'opencv-python==4.5.5.62', 'pillow', 'pyzbar', 'xcamera==2020.613']

However, there is still one case (at least) involving this syntax that I'm wondering about now:

kivy-garden.zbarcam did not have a recipe but what if the user specified something that did, using this syntax? Would it be correct to not look include it in the dependency compilation and simply use the recipe (as we do with the usual syntax)? If this is correct, I would need to parse the potential recipe name from the provided git+url statement. Is it safe to assume that @<branch_name> is always present? is it safe to assume everything between .com/ and @<branch_name> more or less equates to the related recipe name (e.g., git+https/github.com/foo/bar@master --> recipe named foo.bar)?

Are there any other supported syntaxes I should be aware of?

• Recipes:
  Should look like kivy or kivy==version, so there's no need to "unpack" requirements like git+https://github.com/kivy-garden/zbarcam@master.
  If a user wants to target a specific url could do the following:
  export URL_kivy=git+https://github.com/misl6/kivy
  And, with this URL set, if I want to target a specific branch, I can set something like kivy==fix/keyboard-suggestions-android (fix/keyboard-suggestions-android is a branch of my kivy fork) in requirements.
  You can find examples of valid url for recipe here here:

  python-for-android/pythonforandroid/recipe.py

  Line 184 in 06dd1e0

  def download_file(self, url, target, cwd=None):

• pip installable packages:
  If is valid for pip, should be considered valid for us. 😄

[pdallair]

[Recipes] Should look like kivy or kivy==version, so there's no need to "unpack" requirements like git+https://github.com/kivy-garden/zbarcam@master.

Ah ok. So it appears safe to assume that any other syntax doesn't have to do with a recipe. I suppose the answer to my question — Would it be correct to not look include it in the dependency compilation and simply use the recipe (as we do with the usual syntax)? — is a convenient and resounding "No". 👍

[For pip installation packages] if [the syntax] is valid for pip, it should be considered valid for us. 😄

Well... even before applying any of my changes I've had varying degrees of success when applying version constraints... most notably with version constraints that included a comma (e.g., somepackage>=1.0.0,<2.0.0). The problem is the way we split the requirements list. It's always a challenge to find ways to include raw usage of the separator character for a list of unknown length and I believe that would be another issue entirely.

I'm not convinced that file:////<project_path> syntax would work as well with or without my changes because at some point the all pip packages to be installed are forced into a lowercase.

There are so many possibilities... Anyways, my point is and I'm sorry to say it but things probably weren't as flexible as we would like to say when it comes to pip-package syntaxes. As much as I could try, in theory, investigate this further, in practice, I'd like to move on until I personally need one of these more complex syntaxes to work.

Anyways, I've submitted the changes where the git+<web_url> syntax should work. The link you gave me ends up failing down the line during the build process but I doubt this is due to my changes.

[misl6]

Well... even before applying any of my changes I've had varying degrees of success when applying version constraints... most notably with version constraints that included a comma (e.g., somepackage>=1.0.0,<2.0.0). The problem is the way we split the requirements list. It's always a challenge to find ways to include raw usage of the separator character for a list of unknown length and I believe that would be another issue entirely.

Yes, that could be certainly improvable in a near future 😄

I'm not convinced that file:////<project_path> syntax would work as well with or without my changes because at some point the all pip packages to be installed are forced into a lowercase.

Never tested a file:////<project_path> syntax, also here is improvable in future but I think that the use rate of this syntax is quite low, so shouldn't be an issue for the time being.

Anyways, I've submitted the changes where the git+<web_url> syntax should work. The link you gave me ends up failing down the line during the build process but I doubt this is due to my changes.

Will test it extensively ASAP, but code looks good to my eyes. Thank you.

[pdallair]

Will test it extensively ASAP

Hey @misl6, how's the testing going? No need to rush, just asking in case you've simply forgotten. If you need help all I would really need is use case parameters, something I can't muster up easilly with the little experience I have using p4a.

[misl6]

Hey @pdallair , don't worry, this feature that you nicely implemented is so beginner-friendly that I can hardly forget about it 😄

My "Kivy schedule" is currently full of tasks in order to release a new kivy/kivy release, so I do not have a personal ETA for it, but is something that I will finish testing as soon I have a free spot.

Btw, I think that we already covered a high percentage of use cases, so I expect that also passes the final ones.

[misl6]

Finally, Kivy 2.1.0 is out, and all the tools are been updated accordingly.

Will be back on this one during the next two weeks :)

[misl6]

I finally had some time to review this PR and a couple of issues are holding me back from merging it, but more brains working on it means more processing power, and maybe we can find a solution.

• We have some dependencies that are part of the "Kivy family" and pretty used by the community that need to be updated in order to avoid building/installing broken dependencies ( kivy-garden.zbarcam is one of them, as opencv-python should not be installed on Android devices )

• While trying to find an effective way to update our "Kivy family" packages in order to support this PR, I found out that pip-compile supports the environment-markers (Kivy itself uses environment markers to install specific dependencies based on the OS), so it processes the dependencies based on the platform where pip-compile is executed, which means that it will probably be going to generate a wrong requirements.txt file (Targeting a different sys_platform or python_version).

Possible partial fix (As it fixes issues related to python_version, but sys_platform is still an issue): Instead of installing pip-tools as a requirement, that could be installed into the hostpython3 (which follows the same Python version of the python3 recipe).

Any thoughts?