Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update requirements and add float16 support #110

Merged
merged 5 commits into from
Jan 24, 2024

Conversation

SumanthRH
Copy link

@SumanthRH SumanthRH commented Jan 22, 2024

What does this PR do?

I've been playing around with the library recently and noticed that some of the files were a bit outdated. This is a simple PR to

  • update requirements
  • cleanup a bit of the device casting and make it more automatic (still needs some work, but probably in a later PR)
  • allow passing kwargs to the model during initialization. This will allow us to, for example, use bfloat16 dtype for memory efficiency. I made a small change in the LM.to() method for dtype casting.

Ex: lm = ecco.from_pretrained("distilgpt2", activations=True, torch_dtype=torch.bfloat16)

I've tested attributions and neuron factors notebooks with the new changes and things seem to work well

@SumanthRH
Copy link
Author

Okay so I ran some more tests on using bfloat16 and it works for primary attributions, neuron factors, output token scores, token rankings and for visualizing the evolution of a token. Hoping to see this merged @jalammar !

@jalammar jalammar changed the base branch from main to update-py January 24, 2024 11:42
@jalammar
Copy link
Owner

Thanks @SumanthRH! Merging this into a branch to update the CI/CD python versions.

Which version of python did you test this on?

@jalammar jalammar merged commit 296a396 into jalammar:update-py Jan 24, 2024
0 of 3 checks passed
@SumanthRH
Copy link
Author

I tested this on python 3.10!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants