Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eps as minimum scale #3159

Merged
merged 2 commits into from
Jul 10, 2024
Merged

Conversation

quic-kyunggeu
Copy link
Contributor

Problem

Currently v2 encoding analyzer asserts scale >= torch.finfo(...).tiny.
However, this tiny is the smallest positive normal number, which is numerically extremely unstable

Fix

Use torch.finfo.eps as minimum scale instead.
eps is a much more moderate value than tiny, and hence more numerically stable

float32 float16 bfloat16 float64
tiny 1.18e-38 6.10e-05 1.18e-38 2.23e-308
eps 1.19e-07 9.77e-04 7.81e-03 2.22e-16

(p.s. 1.0 / (2**16-1) is equal to 1.53e-05, which is still larger than float32 eps)

Signed-off-by: Kyunggeun Lee <quic_kyunggeu@quicinc.com>
Signed-off-by: Kyunggeun Lee <quic_kyunggeu@quicinc.com>
@quic-kyunggeu quic-kyunggeu merged commit 51780c8 into quic:develop Jul 10, 2024
3 checks passed
@quic-kyunggeu quic-kyunggeu deleted the eps_as_minimum_scale branch July 10, 2024 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants