Merge branch 'main' into kira/deposition-related-filters

* main:
  chore(main): release web 1.25.0 (#1074)
  fix: ID detection for annotation object id link  (#1075)
  chore(main): release cryoet-data-portal-python-client 3.1.0 (#1071)
  chore: add instructions and commands to manually release the python package. (#1073)
  feat: annotation GO ID -> annotation Object ID: add UniProtKB support to annotation Object IDs (#1068)
  feat: Generate Python client code using GraphQL introspection (#1008)
  chore(main): release web 1.24.0 (#1067)
  chore: automate release of python client (#972)
  feat: add single deposition info panel (#986)
  fix: wait for graphql to be healthy in client tests (#1044)
  ci: fixes prod e2e tests (#1066)
  chore(main): release web 1.23.1 (#1051)

.github/workflows/release.yml

@@ -0,0 +1,62 @@
+name: Create Release PRs
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  release-please:
+    concurrency:
+      group: release-prs-${{ github.ref }}
+      cancel-in-progress: true
+
+    runs-on: ubuntu-latest
+    steps:
+      - name: release please
+        uses: googleapis/release-please-action@v4
+        id: release
+        with:
+          # TODO Configuring using manifest file as workaround until we move the frontend to its own repo
+          manifest-file: "release-please.manifest.json"
+          config-file: "release-please.config.json"
+          target-branch: "main"
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+    outputs:
+      paths_released: ${{ steps.release.outputs.paths_released }}
+
+  publish-pypi-package:
+    name: Build and publish Python package to PyPI
+    runs-on: ubuntu-latest
+    needs: release-please
+    if: contains(needs.release-please.outputs.paths_released, 'client/python/cryoet_data_portal')
+    environment:
+      name: pypi
+      url: https://pypi.org/p/cryoet-data-portal
+    permissions:
+      id-token: write # IMPORTANT: this permission is mandatory for trusted publishing
+    steps:
+      - name: Checkout ref branch
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.ref }}
+          fetch-depth: 0
+
+      - uses: actions/setup-python@v4
+        with:
+          pyton-version: "3.10"
+
+      - name: build
+        run: |
+          make build -C client/python/cryoet_data_portal
+
+      - name: Publish distribution 📦 to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: client/python/cryoet_data_portal/dist

client/python/cryoet_data_portal/.coveragerc

@@ -0,0 +1,4 @@
+[run]
+omit =
+    # omit temporary modules created by tests
+    */test_write_models*/_tmp_models.py

client/python/cryoet_data_portal/CHANGELOG.md

@@ -0,0 +1,23 @@
+# Changelog
+
+## [3.1.0](https://github.com/chanzuckerberg/cryoet-data-portal/compare/cryoet-data-portal-python-client-v3.0.3...cryoet-data-portal-python-client-v3.1.0) (2024-08-22)
+
+
+### ✨ Features
+
+* add user agent to client requests ([#966](https://github.com/chanzuckerberg/cryoet-data-portal/issues/966)) ([8209cd4](https://github.com/chanzuckerberg/cryoet-data-portal/commit/8209cd46cb8ab21341c7ee94672db3bae78f9aa2))
+* Generate Python client code using GraphQL introspection ([#1008](https://github.com/chanzuckerberg/cryoet-data-portal/issues/1008)) ([35b7265](https://github.com/chanzuckerberg/cryoet-data-portal/commit/35b72656e77132c9d64cc077705da8940bb29e44))
+
+
+### 🐞 Bug Fixes
+
+* create recursive_from_prefix path if it does not exist ([#940](https://github.com/chanzuckerberg/cryoet-data-portal/issues/940)) ([0069f08](https://github.com/chanzuckerberg/cryoet-data-portal/commit/0069f080987ac05efef82d024cb17f4dc307a0f3))
+* Use match with substring for exception check in client tests ([#895](https://github.com/chanzuckerberg/cryoet-data-portal/issues/895)) ([07352ec](https://github.com/chanzuckerberg/cryoet-data-portal/commit/07352ecdb8c6f50ffe97ff7be9777c0cf6dd66cb))
+* wait for graphql to be healthy in client tests ([#1044](https://github.com/chanzuckerberg/cryoet-data-portal/issues/1044)) ([65f0a4b](https://github.com/chanzuckerberg/cryoet-data-portal/commit/65f0a4b76783ad32bbe439f62fc32f0cae3ae646)), closes [#942](https://github.com/chanzuckerberg/cryoet-data-portal/issues/942)
+
+
+### 🧹 Miscellaneous Chores
+
+* Add additional test case to TestGetDestinationPath ([#955](https://github.com/chanzuckerberg/cryoet-data-portal/issues/955)) ([a9412a8](https://github.com/chanzuckerberg/cryoet-data-portal/commit/a9412a80f3b24ff94b0803fdd59d3583b4521706))
+* add instructions and commands to manually release the python package. ([#1073](https://github.com/chanzuckerberg/cryoet-data-portal/issues/1073)) ([4833eb9](https://github.com/chanzuckerberg/cryoet-data-portal/commit/4833eb95d32ee06a5608e69d6aebf013b1c9fd73))
+* automate release of python client ([#972](https://github.com/chanzuckerberg/cryoet-data-portal/issues/972)) ([073bff7](https://github.com/chanzuckerberg/cryoet-data-portal/commit/073bff7180e2ac3b390cac6a5665b63a7f00e472))

client/python/cryoet_data_portal/Makefile

@@ -1,6 +1,9 @@
 .PHONY: test-infra
 test-infra:
-	cd tests && docker compose up -d
+	# Use force-recreate to handle re-initializing the database schema and data
+	# rather than maintaining incremental migrations.
+	# See the test_infra/hasura/README.md for more details.
+	cd tests && docker compose up -d --wait --wait-timeout 300 --force-recreate
 	cd ./tests/test_infra/; ./seed_moto.sh
 
 .PHONY: coverage
@@ -20,3 +23,24 @@ test:
 		export BOTO_ENDPOINT_URL=http://localhost:4000; \
 		export BOTO_SIGNATURE_VERSION=s3v4; \
 		pytest -vvv -s . $(TEST)
+
+.PHONY: codegen
+codegen:
+	python -m cryoet_data_portal._codegen
+	# Need to run pre-commit twice because black and ruff fight with each other.
+	# Ignore the return code because that is non-zero when pre-commit applies a fix.
+	-pre-commit run --files src/cryoet_data_portal/_models.py src/cryoet_data_portal/data/schema.graphql
+	-pre-commit run --files src/cryoet_data_portal/_models.py src/cryoet_data_portal/data/schema.graphql
+
+.PHONY: build
+build:
+	python -m pip install --upgrade pip
+	pip install build twine
+	python -m build
+
+release/testpypi: build
+	python -m twine upload --repository testpypi dist/*
+
+release/pypi: build
+	# Manually release in case CI/CD pipeline is not working in ./.github/workflows/release.yml
+	python -m twine upload dist/*

client/python/cryoet_data_portal/pyproject.toml

@@ -1,10 +1,10 @@
 [build-system]
-requires = ["setuptools>=45", "setuptools_scm[toml]>=6.2"]
+requires = ["setuptools>=64"]
 build-backend = "setuptools.build_meta"
 
 [project]
 name = "cryoet_data_portal"
-dynamic = ["version"]
+version = "3.1.0"
 description = "API Client to facilitate the use of the CryoET Portal. For more information about the API and the project visit https://github.com/chanzuckerberg/cryoet-data-portal/"
 authors = [
     { name = "Chan Zuckerberg Initiative", email = "cryoetdataportal@chanzuckerberg.com" }
@@ -38,6 +38,7 @@ dependencies= [
 [project.urls]
 homepage = "https://github.com/chanzuckerberg/cryoet-data-portal"
 repository = "https://github.com/chanzuckerberg/cryoet-data-portal"
+documentation = "https://chanzuckerberg.github.io/cryoet-data-portal/python-api.html"
 
 [tool.setuptools.packages.find]
 where = ["src"]

client/python/cryoet_data_portal/release_process.md

@@ -1,105 +1,19 @@
 # Python package release process
-
-The following approach is used to manage releases of the Python package:
-
-1. The package is automatically built (sdist and wheels) in a GitHub action, and build artifacts are uploaded to GitHub.
-2. Release candidate testing is done by installing built assets from Github.
-3. Build versions are managed via [`setuptools_scm`](https://github.com/pypa/setuptools_scm) and the version is automatically determined from git tags.
-4. Releases are created and managed via GitHub Releases, leaving a tag in place from which future branches (eg, emergency fixes) can be created.
-5. Built packages are published to PyPi _from_ the GitHub assets, i.e. are never hand-created, to reduce errors.
-
-## Prerequisites
-
-While not strictly required, this process assumes you have met the following prerequisites:
-
-- You have write access to the `chanzuckerberg/cryoet-data-portal` repo
-- You have an account on pypi.org and test.pypi.org, both with access to the cryoet-data-portal project
-- You have the Github CLI tool (`gh`) installed. See [documentation](https://cli.github.com/).
-- You have the `pipx` CLI tool installed. See [documentation](https://pypa.github.io/pipx/).
-
-## Step 1: Building the package assets
-
-A build will occur automatically upon each commit to main, upon each commit to a PR, or when the build workflow is manually invoked. The build workflow is defined by the GH `py-build.yml` workflow. Build artifacts are the Python setuptools-created `sdist` and `wheel`, and are retained for a limited period of time (currently the GH default of 90 days).
-
-Unless you are revising and testing the build process itself, there is no need to manually perform a build.
-
-## Step 2: Release candidate testing
-
-Any pre-built asset on Github can be installed and tested from the Github URL. For example:
-
-1. Identify the GH workflow run ID that contains the asset you wish to test. A simple way to do this is:
-   ```shell
-   $ gh run list
-   ```
-   Alternatively, you can use the "Actions" tag in the GitHub web UI.
-2. Download the build artifact.zip from GitHub, using the GH Action run ID associated with the `build` action for your commit OR utilizing the web UI:
-
-   ```shell
-   $ gh run download <ID>
-   ```
-
-   If you download using the browser, unzip into a temp directory, e.g.,
-
-   ```shell
-   $ unzip artifact.zip -d ./artifact/
-   Archive:  artifact.zip
-     inflating: ./artifact/cryoet-data-portal-0.0.1.dev0-py3-none-any.whl
-     inflating: ./artifact/cryoet-data-portal-0.0.1.dev0.tar.gz
-   ```
-
-3. Install and test the downloaded build, e.g.,
-   ```shell
-   $ pip uninstall cryoet-data-portal
-   $ pip install ./artifact/cryoet-data-portal-*-any.whl
-   ```
-
-To test a release candidate:
-
-1. Identify the build you wish to test. Download and test the artifact built for that commit as described above.
-2. Perform end-user testing, using the above installation method
-3. If acceptable, proceed to Step 3 - create a release.
-
-If testing exposes problems, fix and commit a solution as you would any other change.
-
-## Step 3: Create a release
-
-Prior to this process, determine the correct semver version number for the new release. Please consider if this is a major, minor or patch release, and if it should have a tag (e.g., an alpha build, with a `a#` suffix or a pre-release candidate, with a `-rc` suffix). If you are not sure, please review [PEP 440](https://peps.python.org/pep-0440/) for more information.
-
-This process also assumes you are releasing from `main`. If you create a release PR, it should be merged to main before releasing.
-
-To create a release, perform the following:
-
-1. Identify both the (tested & validated) commit and semver for the release.
-2. Tag the commit with the release version (_including_ a `v` prefix) and push the tag to origin. **Important**: use an annotated tag, e.g., `git tag -a v1.9.4 -m 'Release 1.9.4`. For example (please replace <SEMVER> with your version, _including_ a `v`, e.g. `v1.9.4`:
-   ```shell
-   $ git tag -a <SEMVER> -m 'Release <SEMVER>'
-   $ git push origin <SEMVER>
-   ```
-3. Trigger a build for this tag by manually triggering the `py-build.yml` workflow. For example:
-   ```shell
-   $ gh workflow run py-build.yml --ref <SEMVER>
-   ```
-4. When the workflow completes, make note of the run ID (e.g., using `gh run list`).
-5. Optional, _but recommended_: download the asset from the build workflow and validate it.
-6. Create and publish a GitHub Release, using the `<SEMVER>` tag above (e.g., `v1.9.4`). It is recommended that you include a release summary and change log in the GH release.
-
-## Step 4: Publish assets to PyPi
-
-To publish built release assets to PyPi (_note_: this will require your pypi/testpypi login):
-
-1. Download the assets built for your release commit, using the same method as step 2 above, e.g.,
-   ```shell
-   $ gh run download <ID>
-   ```
-2. Optional: upload to TestPyPi (this assumes the downloaded assets are in ./artifact/).
-
-   ```shell
-   pipx run twine upload --repository testpypi ./artifact/*
-   ```
-
-   Following the upload, confirm correct presentation on the project page and ability to download install from TestPyPi. To test with TestPyPi, use `pipx run twine upload --repository testpypi ./artifact/*`. You can find more information [here](https://packaging.python.org/en/latest/guides/using-testpypi/).
-
-3. Use twine to upload to PyPi (this assumes the downloaded assets are in ./artifact/), e.g.,
-   ```shell
-   pipx run twine upload ./artifact/*
-   ```
+To release a new version of python client, merge a pull request to main with change to **./client/python/cryoet-data-portal**. Release-please will automatically create a PR with the a version bump if needed. The name of the PR will be **chore(main): release cryoet-data-portal-python-client vX.X.X**. Here is an [example](https://github.com/chanzuckerberg/cryoet-data-portal/pull/981) of a release PR. Merging the release-please PR will create a new release in github and a new version will be uploaded to pypi. If no changes were made that affect the python client, the python client will not be release. Closing the PR will skip the release of these changes and they will be included in the next release.
+
+If the version number needs to be change see [release-plesae documentation](https://github.com/googleapis/release-please?tab=readme-ov-file#how-do-i-change-the-version-number) for how to change the version number. This same process can be used to manually release the package.
+
+## Build Locally
+The python client can be build locally by running the following:
+```bash
+make build
+```
+
+# Manual Release Process
+After merging in the release-please PR, if the python package did not release it can be manually release by doing the following:
+1. Setup your environment to upload to pypi by following the instructions [here](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#create-an-account)
+2. Pull the latest changes from main
+3. Run the following command to release the python client:
+```bash
+make release/pypi
+```

client/python/cryoet_data_portal/scripts/requirements-dev.txt

@@ -5,3 +5,4 @@ requests-mock
 twine
 coverage
 nbqa
+Jinja2

client/python/cryoet_data_portal/src/cryoet_data_portal/__init__.py

@@ -12,15 +12,16 @@
     Dataset,
     DatasetAuthor,
     DatasetFunding,
+    Deposition,
+    DepositionAuthor,
     Run,
     TiltSeries,
     Tomogram,
     TomogramAuthor,
     TomogramVoxelSpacing,
 )
-from ._version import version
 
-__version__ = version
+__version__ = "3.1.0"
 
 __all__ = [
     "Client",
@@ -30,6 +31,8 @@
     "Dataset",
     "DatasetAuthor",
     "DatasetFunding",
+    "Deposition",
+    "DepositionAuthor",
     "Run",
     "TiltSeries",
     "Tomogram",