Update CI/CD to fix PyPi deployment failure and improve documentation deployment

[jordanpadams]

Tasks

• Fix Github Action failure when trying to deploy to PyPi.org.
• Upon release, deploy documentation to https://nasa-pds-incubator.github.io/pds-deep-archive/
• Tag development releases in Github in addition to deploying to test PyPi (see https://github.com/NASA-PDS-Incubator/pds-doi-service for an example of how to do this)
• Improved Regression Test Suite - after the several bugs we have caught over the past few weeks, I would really like to improve this test suite. I don't even mind if we add a few of those bundles to this repo, or the test suite can simply contain the expected outputs and the data can be downloaded from our test bundle staging area. See https://github.com/NASA-PDS/pds-deep-archive/issues?q=is%3Aissue+label%3Atest+ for example test data
• Update CI/CD to deploy in-dev documentation to some other online directory (maybe https://nasa-pds-incubator.github.io/pds-deep-archive/development or https://nasa-pds-incubator.github.io/pds-deep-archive/dev?) Factored out into Organize the automated documentation generation for dev releases #74 for a project-centric approach

[jordanpadams]

@nutjob4life note the added task above for Improved Regression Test Suite

[nutjob4life]

@jordanpadams I would like that too. When time pressure isn't an issue, the way I like to work is:

1. Appraise the latest bug report.
2. Branch and run the test suite. They should all work.
3. Add a test case that exposes the new bug.
4. Run the test suite; it should fail, if not, fix the new test case.
5. Fix the bug.
6. Run the suite again; it should succeed; if not, back to step 5.
7. Commit & PR.

This way, you get a monotonically increasing suite.

Downloading on demand is nice but can present intermittent test failures, esp. in automated nightly test environments. If the bundles aren't too large, I sure don't mind including them in the repository. We can omit them from source distributions with the caveat to end-users that "test suite will run in degraded mode" or similar.

What do you think @jordanpadams?

[jordanpadams]

@nutjob4life i will leave it to you. I am definitely OK with this approach, I just know it can be time consuming. so whatever you think is best. at minimum, I would like us to have at least one regression test that runs on a bundle, outputs the AIP and SIP, and do some simple comparisons to ensure the outputs meet some success critieria (e.g. number of lines in .TABs, LIDs, some other maybe useful info)

[nutjob4life]

Hey @jordanpadams I am completely out of steam since resolving #71 for the PDS.20 milestone.

However the PR for it #72 contains much of the work for the last checkbox on this issue above, Improved Regression Suite. That PR includes:

• Creating a SIP with all collections = true from the LADEE bundle
• Creating an AIP with all collections = true from the LADEE bundle
• Creating a SIP with all collections = true from the Insight Documents bundle
• Creating an AIP with all collections = true from the Insight Documents bundle
• Creating a SIP with all collections = false from the Insight Documents bundle
• Creating an AIP with all collections = false from the Insight Documents bundle

There's a easy-to-plug-in framework in there now to add additional bundles with various parameters.

For this issue, then, that just leaves tagging dev releases which @tloubrieu-jpl may already have implemented. I need to take a closer look at the various GitHub workflows he's added to this repository.

And then deploying dev docs.

Upshot is, I've missed this issue for this milestone. I'm sorry 🙇

[jordanpadams]

@nutjob4life no problem. thanks for getting to #71 ! that is totally OK for this build. this issue is not time sensitive. the other one was so we can get moving on operational release. @tloubrieu-jpl is working on deploying dev vs operational docs so he can probably chime in here and may even wrap up this issue for you.

[tloubrieu-jpl]

Update CI/CD to deploy in-dev is difficult to implement because there is not action to push into a specific directory in gh-pages.

It will requires more work and discussion on how to handle automated documentation for software. This will be done in ticket #74

[nutjob4life]

Note to self @nutjob4life: "changing your point of view" might be a good way to make headway on this effort, similar to how the "corral" does it. For a common approach that can span repositories, we may want to investigate making our own GitHub Action that can be plugged into multiple workflows without mushrooming each to 100s of lines.