Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kar2019 testing score assertion errors #1198

Open
samwinebrake opened this issue Aug 22, 2024 · 0 comments
Open

Kar2019 testing score assertion errors #1198

samwinebrake opened this issue Aug 22, 2024 · 0 comments
Assignees

Comments

@samwinebrake
Copy link
Collaborator

Score assertion errors are present in the kar2019 benchmark testing. See below for an example:

    def test_Kar2019ost_cornet_s():
        benchmark = load_benchmark('Kar2019-ost')
        filename = 'cornet_s-kar2019.nc'
        filepath = Path(__file__).parent / filename
        s3.download_file_if_not_exists(local_path=filepath,
                                       bucket='brainscore-unittests', remote_filepath=f'tests/test_benchmarks/{filename}')
        precomputed_features = NeuroidAssembly.from_files(
            filepath,
            stimulus_set_identifier=benchmark._assembly.stimulus_set.identifier,
            stimulus_set=benchmark._assembly.stimulus_set)
        precomputed_features = PrecomputedFeatures(precomputed_features, visual_degrees=8)
        # score
        score = benchmark(precomputed_features).raw
>       assert score == approx(.316, abs=.005)
E       assert <xarray.Score...ay(0.02970973) == 0.316 ± 5.0e-03
E         comparison failed
E         Obtained: <xarray.Score ()>\narray(0.2408883)\nAttributes:\n    raw:      <xarray.Score (split: 10)>\narray([0.14566184, 0.12413281, 0.23...\n    error:    <xarray.Score ()>\narray(0.02970973)
E         Expected: 0.316 ± 5.0e-03

test.py:35: AssertionError

This assertion error is with brainscore version 2.0.0, whereas when using python 3.11 and the updated brainscore version, the output score is 0.23875906 which still causes an assertion error. This is possibly due to the continued use of tensorflow within the ost metric.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants