Self-hosted (ghpages) pip archive html file

[ptheywood]

If we are going to release on pypi, we will likely only be able to host pyflamegpu packages built with a single cuda architecture, which Ideally should be the oldest version with broad support at the time of release. I.e. CUDA 11.2 would be my preference, as this will work with CUDA 11 releases >= 11.2, so wide support. Alternatively CUDA 11.0 for the oldest support, but this does not have forwards compatibility.

In some cases,we might want to provide multiple version. Ie CUDA 11.0 (collab) and 11.2 (broader support), as well as in the future CUDA 11.x and CUDA 12.x. In which case we use the python local package version number to embed this, i.e. +cuda112 for cuda 11.2.

We could support users installing these via pip, withtout them being uploaded to pypi by leveraging the github release artifacts, github actions, and github pages.

Users could then install a potential 2.0.0 release built with CUDA 11.2 via:

python3 -m pip install -f https://download.flamegpu.com/whl/stable.html pyflamegpu==2.0.0+cuda112

Or alternatively pre-release versions using the --pre flag

python3 -m pip install  --pre -f https://download.flamegpu.com/whl/prerelease.html pyflamegpu==2.0.0-rc1+cuda112

--no-index may also be useful in the above commands?

This is how pytorch handle pip installation of alternate versions.

As with pip distribution in general, it has downsides re: alterante builds. In this case you cannot ask for the latest pyflamegpu built with CUDA XY, instead you must specify a specific minor version and the local version.
Poetry / other python packaging might also not be able to resolve this if they do not support the -f flag for pip subpackages.

One way to implement this would be:

• Github action, which on release publication:
  • Queries github for the assets of the release (they may be available in the gha context already)
  • Insert html links to these assets into a .html file, commited to the gh_pages branch of an alternate repo, which is hosted via ghpages on a subdomain of flamegpu.com.

Some of the following commands may be useful to implemetn this.

Using the gh command line tool which is installed into actions by default, we can:

get a list of the last 10 releases using:

gh release list --limit 10 

get a json object containing all of the .whl assets for a specific release tag using:

gh release view v2.0.0-alpha --json assets --jq ".assets.[] | select(.name|endswith(\".whl\"))"

For more information, see comments on #605.

[ptheywood]

gh doesn't provide an obvious way to get the lists of assets for a release.

However, the GET /repos/{owner}/{repo}/releases REST API endpoint does.

I.e.

curl   -H "Accept: application/vnd.github.v3+json"   https://github.com/gitapi/repos/FLAMEGPU/FLAMEGPU2/releases

Returns a very long json response, which is a list ([]) of dictionaries ({}), with one per release (probably some pagination eventatiually.

The release dictionary includes "tag_name" fields along with other releasae metadata such as if it is a pre-release or not, or a draft, publication date etc, but most importantly assets

assets is a list of objets, where each object importantly contains "url", "name" and "label" fields (and "size"` which may be of interest)

Using the url and name components (labels maybe in the future) this should provide everything that is needed to generate a html page which allows python pip installation.

This API endpoint returns 30 release by default, which can be configured using the per_page parameter. The page parameter can be configured to select which page to view (1 indexed`.

A 404 status response will be issued if not found.

Alternatively, the GET /repos/{owner}/{repo}/releases/tags/{tag} endpoint exists, to get the release info for a specific tag (i.e. no pagination).

Ie.

curl   -H "Accept: application/vnd.github.v3+json"   https://github.com/gitapi/repos/FLAMEGPU/FLAMEGPU2/releases/tags/v2.0.0-alpha.1

If this output was stored to tag.json it can then be filtered using jq to extract various components

# Get the URL of the release
cat tag.json | jq -c '.["url"]'
# Get the url, name and label of the 0th asset for the release 
cat tag.json | jq -c '.["assets"] | .[0]["url", "name", "label"]'

This could be used in conjunction with another endpoint to get the list of releases?

As a quick POC to verify this does work with gh release asset urls:

A file pyflamegpu.html containing:

<a href="https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu_vis-2.0.0a1+cuda112-cp39-cp39-linux_x86_64.whl">pyflamegpu_vis-2.0.0a1+cuda112-cp39-cp39-linux_x86_64.whl</a><br>
<a href="https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu-2.0.0a1+cuda112-cp39-cp39-linux_x86_64.whl">pyflamegpu-2.0.0a1+cuda112-cp39-cp39-linux_x86_64.whl</a><br>
<a href="https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu_vis-2.0.0a1+cuda110-cp39-cp39-linux_x86_64.whl">pyflamegpu_vis-2.0.0a1+cuda110-cp39-cp39-linux_x86_64.whl</a><br>
<a href="https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu-2.0.0a1+cuda110-cp39-cp39-linux_x86_64.whl">pyflamegpu-2.0.0a1+cuda110-cp39-cp39-linux_x86_64.whl</a><br>

Then from a system with python 3.9 installed:

python3 -m pip install -U pyflamegpu -f pyflamegpu.html 
Looking in links: pyflamegpu.html
Collecting pyflamegpu
  Downloading https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu-2.0.0a1%2Bcuda112-cp39-cp39-linux_x86_64.whl (145.2 MB)
     |████████████████████████████████| 145.2 MB 18.2 MB/s 
Installing collected packages: pyflamegpu
Successfully installed pyflamegpu-2.0.0a1+cuda112

or the vis wheel:

python3 -m pip install -U pyflamegpu_vis -f pyflamegpu.html 
Looking in links: pyflamegpu.html
Collecting pyflamegpu_vis
  Downloading https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.1/pyflamegpu_vis-2.0.0a1%2Bcuda112-cp39-cp39-linux_x86_64.whl (146.2 MB)
     |████████████████████████████████| 146.2 MB 16.8 MB/s 
Installing collected packages: pyflamegpu-vis
Successfully installed pyflamegpu-vis-2.0.0a1+cuda112

However, both wheels can be installed into the same venv at the same time as they have different names, even though they both provide the pyflamegpu module, so whichever was installed later will be imported.

This is one of the downsides of providing vis and non vis wheels which import the same package name.

It might be better to produce a separate .html file for pyflamegpu and pyflamegpu_vis to further distinguish these as separate (and ideally in the future stop providing pyflamegpu_vis`

python3 -m pip list 
Package        Version
-------------- ---------------
pip            20.3.4
pkg-resources  0.0.0
pyflamegpu     2.0.0a1+cuda112
pyflamegpu-vis 2.0.0a1+cuda112
setuptools     44.1.1

Or if the .html file only contained python 3.8 URIs, then the following error would occur (i.e. what will happen on non x86_64 machines, or python < 3.6, or python > 3.9:

python3 -m pip install -U pyflamegpu -f pyflamegpu.html 
Looking in links: pyflamegpu.html
ERROR: Could not find a version that satisfies the requirement pyflamegpu
ERROR: No matching distribution found for pyflamegpu

[ptheywood]

Attaching a .html file to a github release as an atrifact is not served in the web browser when viewed, it is downloaded, so can't be used for a (hosted) wheelhouse, so would need to be gh pages. Conda still seems like the better solution overall, given our wheels are not strictly manylinux compliant without dlopen soup.

[ptheywood]

I've made a start on this, working in a standalone separate repository for now (so no one accidentally finds it and starts using it before it is "official")

https://github.com/ptheywood/FLAMEGPU2-wheelhouse

So far I have a script which will fetch upto the last 30 releases and all their artifcacts from teh github api into a json file.

Another script is then going to take that json file and dump out the appropriate HTML to disk.

CI will then be set up to do that, and commit the result to the gh-pages branch.

Once that all works, I'll make it more API friendly so it doesn't hit any rate limits etc, by only finding assets for releases it didn't already know about.

We can then either set the workflow to be triggered manually, or be triggered by workflow_dispatch, so once we publush a release (i.e. stop it beign a draft) it will be triggered.

[ptheywood]

I now have a pair of working scrtipts.

However, changes are needed for vis / non vis build selection, (and cuda selection?)

I.e. if I produce samples/pyflamepgu.html currently, containing all wheels, just asking for pyflamegpu gets 2.0.0rc+cuda112.vis. Is this what we want as the default?

While how wheel houses work, its possible to provide a directory strucutre with multiple files.

This allows people to query teh full set for explicit versions / just get the default, or to alos have ways to state "give me the latest cuda 11.2" or "give me the latest vis build, regardless of cuda version".

Otherwise the python version comparison rules apply, so the latest cuda version, and then .vis is priority over no ".vis" component.

The alternative would be to tweak the build / python packaing and ship vis and non vis wheeels together, so one (or both) can be installed. This would need changes to import statements however (off the top of my head), or dlopen magic to make everything a vis build, but only link against the vis dependence if opted in to.

[ptheywood]

Now have CI working for single-file wheelhouse, currently hostest at https://ptheywood.uk/FLAMEGPU2-wheelhouse/ for demonstration purposes, but this will not be the final location.

(most likely wheelhouse.flamegpu.com or something to that effect

python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/ pyflamegpu
Looking in links: https://ptheywood.uk/FLAMEGPU2-wheelhouse/
Collecting pyflamegpu
  Downloading https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-rc/pyflamegpu-2.0.0rc0%2Bcuda112.vis-cp310-cp310-linux_x86_64.whl (170.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 170.5/170.5 MB 11.2 MB/s eta 0:00:00
Collecting astpretty
  Using cached astpretty-3.0.0-py2.py3-none-any.whl (4.9 kB)
Installing collected packages: astpretty, pyflamegpu
Successfully installed astpretty-3.0.0 pyflamegpu-2.0.0rc0+cuda112.vis

It is also possible to install specific versions, getting the most "recent" build (i.e. cuda112, vis was specified differntly in alpha builds, which might need to change again in the next rc?)

python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/ pyflamegpu==v2.0.0-a2
Looking in links: https://ptheywood.uk/FLAMEGPU2-wheelhouse/
Collecting pyflamegpu==v2.0.0-a2
  Downloading https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-alpha.2/pyflamegpu-2.0.0a2%2Bcuda112-cp310-cp310-linux_x86_64.whl (153.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 153.6/153.6 MB 11.3 MB/s eta 0:00:00
Installing collected packages: pyflamegpu
  Attempting uninstall: pyflamegpu
    Found existing installation: pyflamegpu 2.0.0rc0+cuda112.vis
    Uninstalling pyflamegpu-2.0.0rc0+cuda112.vis:
      Successfully uninstalled pyflamegpu-2.0.0rc0+cuda112.vis
Successfully installed pyflamegpu-2.0.0a2+cuda112

Or the fully explicit version to get a different cuda / non vis build

python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/ pyflamegpu==v2.0.0-rc0+cuda110
Looking in links: https://ptheywood.uk/FLAMEGPU2-wheelhouse/
Collecting pyflamegpu==v2.0.0-rc0+cuda110
  Downloading https://github.com/FLAMEGPU/FLAMEGPU2/releases/download/v2.0.0-rc/pyflamegpu-2.0.0rc0%2Bcuda110-cp310-cp310-linux_x86_64.whl (80.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.3/80.3 MB 13.6 MB/s eta 0:00:00
Requirement already satisfied: astpretty in ./.venv-remote/lib/python3.10/site-packages (from pyflamegpu==v2.0.0-rc0+cuda110) (3.0.0)
Installing collected packages: pyflamegpu
  Attempting uninstall: pyflamegpu
    Found existing installation: pyflamegpu 2.0.0a2+cuda112
    Uninstalling pyflamegpu-2.0.0a2+cuda112:
      Successfully uninstalled pyflamegpu-2.0.0a2+cuda112
Successfully installed pyflamegpu-2.0.0rc0+cuda110

Notably this does solve the python selection process, but to avoid vis wheels for the tutorial we will want to separate the vis into it's own html file. Likely the same for the cuda version, as I'm not sure of a way to save "Give me the most recent build which has +cu110 as the build identifier" (I've not looked really though).

I.e. people will still need to be pretty specific, unless they just want latest cuda with vis.

If we have a tree like follows:

.
├── cu110
│   └── index.html
├── cu112
│   └── index.html
├── cu120
│   └── index.html
└── vis
    ├── cu110
    │   └── index.html
    ├── cu112
    │   └── index.html
    └── cu120
        └── index.html

then users can could use one of the following, depending what they want

# latest cuda, currenlty with vis, latest pyflamegpu
python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/ pyflamegpu
# latest cuda, specific pyflamegpu, iplicitly latest cuda + vis currently (though I do want to fix that still 
python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/ pyflamegpu==v2.0.0-rc
# specific cuda, no vis
python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/cu110 pyflamegpu
# specific cuda, vis
python3 -m pip install -f https://ptheywood.uk/FLAMEGPU2-wheelhouse/vis/cu110 pyflamegpu

Alternatively / additionally, we could adjust teh name of the non-vis wheels to include an additional local version identifier component so it is "newer" than the equivalent +cudaXXX.vis A version with more segements will be "greater" than a version with less, hence .vis wins. We would need to make it lexicographically greater than .vis to appear newer, so no-vis would not be defualt, nor console. So we might need to identify this separately. It might be better to rename the entire python package in that case (so both can be installed, and can explicitly choose the correct one, but that is a separate but related issue)

The local versions we embed to our wheels are also not allowed on pypi / should not be used when publishing "upstream" projects, so we might need to find an alternate way to deal with this in that case.

[ptheywood]

Partially implemented the above split wheelhouses to demonstrate it works.

ptheywood/FLAMEGPU2-wheelhouse#1

Can install

• Any version via index.hmtml, defualts to latest cuda with vis
• console-only CUDA 11.0 via cuda110/index.html
• console-only CUDA 11.2 via cuda112/index.html
• vis CUDA 11.0 via cuda110/vis/index.html
• vis CUDA 11.2 via cuda112/vis/index.html

No way to get a vis or non-vis CUDA X build. With the current single pyflamegpu package that is vis or not vis that doesn't make sense IMO.
Plus given local versions are not allowed in package indexes such as pypi, we would only be able to distribute vis or non vis builds that way. Ideally we should have a separate package imo, and then either:

• dynamically link against the vis library if it is there, otherwise its a console only. Would require changes to the API. We could then either effectively always do vis builds (and only make vis available if vis is available), or we could produce 2 python packages, and make the vis package an optional dependency. I.e. pip install pyflamegpu[pyflamegpuvis] would be how to do a vis install.

• Vis builds should produce a different python package name, pyflamegpuvis or wahtever. users would then have to change their import statements, but it would be as simple as doing import pyflamegpuvis as pyflamegpu and that would be the onyl difference. A common snippet that would load console if available otherwise load vis (or the reverse) would then be:

  try:
      import pyflamegpu
  except ImportError:
      import pyflamegpuvis as pyflamegpu
  

Both are grim in their own ways and a decent bit of effort though.

[ptheywood]

Paul has setup a CNAME record for whl.flamegpu.com. Once DNS propagates, I'll get this factored in and move the repo across to the flame org.

re: triggering the workflow on publication of a new release, need to make a PR against the FLAMEGPU2 repo, with a new github action workflow, which on publication of a release uses the gh api to trigger the workflow_dispatch (or repository_dispatch event on the wheelhouse repo's CI workflow.

Could do this with CURL, or the gh cli (though this apparently doesn't return the id of the new run, so have to query again for new runs and wait for it to complete. Roughly the following (given the published event for a repo only fires once)

on:
  release:
    types: [published]  
env:
  GH_TOKEN: ${{ secrets.MY_TOKEN }}

...

steps:
  - run: |
      gh workflow run ci.yml -R flamegpu/FLAMEGPU2
      sleep 5
      gh run watch -R flamegpu/FLAMEGPU2 $(gh run list -R flamegpu/FLAMEGPU2 -w ci.yaml -L1 --json databaseId --jq .[0].databaseId)

There is effectivly a race in the above, so the resulting status might be a lie (if the ci was triggered for another reason after the first gh command.

[ptheywood]

Slight issue with using gh to trigger a workflow in another repo, the GH_TOKEN generated in the workflow doens't seem to have enough permissions, even when adding write all. Might need some kind of org/account level token instead.

If the workflow is in the same repo it works as expected.

[ptheywood]

Using a PAT with repo action write scope stored in a secret let the remote dispatch work.

However PATS have a default expiration of 30 days, but also get deactivated after a year, so if I create one with a very long duration it will expire if there isn't a new flamegpu2 release for a while.

Grand scheme this is OK though, as we will be able to manually trigger it in that case, and I can add a comment to the workflow that would trigger it stating it would need a new PAT creating if it fails prior to re-running the job (or manually triggering the workflow in the other repo)