Add table containing version string of latest release of spyglass

[rly]

The Spyglass database and code are intricately linked. The tables defined in Python within Spyglass are intended to have a perfect correspondence with the tables in the MySQL database. If not, unexpected errors can occur when inserting or accessing data, e.g., tables or columns may no longer exist or new columns may be required.

In Version 0.3.0, we introduced changes to some Python classes that define tables and made the corresponding changes to the database, but anyone using an older version of spyglass to insert data will encounter errors because the older code assumes the database tables are structured in the previous way.

We will avoid making breaking changes but they are bound to happen, especially as a pipeline is being developed. To help catch cases where older code is being used, I suggest adding a "spyglass_version" table with a single row that contains the version string of the latest release of Spyglass (e.g., 0.3.3), and adding code that checks whether you are running the matching version when you connect to the database. If not, raise an error.

[lfrank]

Great idea...

[CBroz1]

I think this would be best addressed by hidden attributes, if and when they become available in DataJoint

[CBroz1]

Now that we manage version with hatch, the 'ground truth' of versioning is the GitHub tag, which pushes to PyPI, so either. I worry that adding another source of ground truth in the form of a SQL table would introduce complications.

Here's something I came up with for running everything in python

import requests
from packaging.version import parse as vparse

from spyglass import __version__ as spyglass_version
from spyglass.utils import logger

# spyglass_version from hatch: '0.4.3a5.dev21+g2b1093d1.d20240208'
# spyglass_version from pypi : '0.5.1'

def check_latest_version():
    try:
        response = requests.get("https://pypi.org/pypi/spyglass/json")
        response.raise_for_status()
        latest_version = response.json()["info"]["version"]

        if vparse(spyglass_version) < vparse(latest_version):
            logger.error(
                f"There is a newer version available: {latest_version}"
            )

    except Exception as e:
        logger.error(f"Error: {e}")

I'm running into trouble however because hatch doesn't update my local _version.py under our current squash merge approach of PRs.

@edeno Do you know how to trigger a refresh of _version.py?

[edeno]

If you pip install -e ., it will update your local version. However, it doesn't update to 0.5.1 I think because I tagged a commit in the PR branch instead of the master branch. We can rectify this on our next release.

[rly]

@CBroz1 That approach is reasonable for the Frank Lab database, which should always be up-to-date with the latest version of Spyglass.

Are we imagining that there might be multiple databases, e.g., created by different labs or as sandboxes? If so, and there is a breaking change in Spyglass (tables or columns get renamed or deleted), and the other database is not altered correspondingly, then if check_latest_version runs on import, users of that other database cannot keep using their older version of Spyglass with that database, and if they update Spyglass, the database might be out of sync.

[CBroz1]

Thanks, @edeno - yes, rerunning pip install updates. Not sure how to revise our workflow to make this version check u smoothly

I see what you mean @rly - The database should keep track of the version used on declaration, and perhaps not permit someone to run a version beyond a breaking change without adapting tables. I think that might mean for us that table changes require a minor version bump. Is that ok with you @edeno ?

[edeno]

I think the problem is that even you know the version when the table is declared or altered, there could still be a difference if the make function is changed or an underlying package it depends on changes.

I'm actually not sure about the version check because I think it could become very complicated very quickly.