-
Notifications
You must be signed in to change notification settings - Fork 508
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
automatic batch size for dp test
#1165
Conversation
Resolves deepmodeling#1149. We start nbatch * natoms from 1024 (or we can set a different number), and iteratively multiply it by 2 until catching the OOM error. A small issue is that it's a bit slow to catch the TF OOM error. It's a problem of TF and I don't know how to resolve it. Luckily we only need to catch once.
Codecov Report
@@ Coverage Diff @@
## devel #1165 +/- ##
==========================================
+ Coverage 75.94% 76.08% +0.13%
==========================================
Files 90 91 +1
Lines 7172 7226 +54
==========================================
+ Hits 5447 5498 +51
- Misses 1725 1728 +3
Continue to review full report at Codecov.
|
deepmd/utils/batch_size.py
Outdated
self.maximum_working_batch_size = 0 | ||
self.minimal_not_working_batch_size = 2**31 | ||
|
||
def execuate(self, callable: Callable, start_index: int, natoms: int) -> Tuple[int, tuple]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo: execuate -> execute
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you please add a unittest for the AutoBatchSize
?
Resolves #1149.
We start nbatch * natoms from 1024 (or we can set a different number), and iteratively multiply it by 2 until catching the OOM error.
A small issue is that it's a bit slow to catch the TF OOM error. It's a problem of TF and I don't know how to resolve it. Luckily we only need to catch once.