API dump and deeper trulens_eval compatibility test #1309
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sfc-gh-pmardziel
changed the title
sfc-gh-dhuang
pushed a commit
that referenced
this pull request
Aug 24, 2024
* dummy testing * writing record dump from dummy app invocation * write golden files * add golden tests * docs * add docs * template adjustment * fix mkdocs warnings * fix old test * change goldens to yaml for human readability * backtrack to json due to yaml issues, add checking of feedback result as well * more skipped json parts * format * add some additional checks on bindings * nit * add todo near comments * grammar fix * more docs * more docs * cleanup * fix validate nondeterminism * update gold * pretty print golden json output * extension warning * icons for dummy docs * fix docs * format * adjust template wording * update custom_example notebook * update docs around custom_example * update golden tests for custom app * fix outdated recording usage * adding custom app options * adding more example usage * fixing up async uses * format * format again * add streaming to dummy app * update golden * add comparison for unordered sequences * add frozendict to test reqs * typo * reorganize dummy app * cleaning up docs * updating docs and adding references * remove str from enums * revert and add comment about str Enums * format * update golden files * adjust name argument default in some signatures * PR comments * undo accidental addition * Factoring out part of static tests. * nit * format * work * api test case * address PR comments * remove unused * format * format again * add api golden list * format * pre-commit format * Add pydantic attributes to api printout * fix forward ref rendering for api dump * fix test imports * fix test path * fix import * format * typo again * make api test optional * another typo * type typo * fixes for python before 3.11 * use subtests in json compare * minor * remove old example from package root, another copy was in examples outside of package * typo * move api reference docs * remove unused docs * remove old stuff * revert goldens * fixing merge * rename test * Rename. * fix makefile * Fixing api dumps and splitting by pyversion. * skip version in json compare * fix static test * missing todo * skip _bundle mod * remove duplicate targets * fix to some imports * another fix * write golden api * add jsondiff to test deps * add 3.11 api dump * add 3.9 and 3.10 api dumps * run on py311 only * test only on 3.11 * api changes * fixing session * fixing session args * make app_id a read only instead of computed field * forgot to regen api * read_only -> frozen
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Some utilities to collect module contents and a dump of the current names in the
trulens_evalpackage. This is to be used for backwards compatibility tests. Also added dump of the newtrulenspackage to make sure we are aware of API changes when they happen.Other details good to know for developers
Factored out module enumeration and related from static tests to import utilities. Making use of this for some backwards compatibility checks.
Added
api_testthat compares the qualified names and types of all public members of trulens modules and classes to a fixed golden set. In the future, will also add function signatures to this ground truth.Type of change
not work as expected)