Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Two changes to the gradient checker that improve performance pretty significantly without any cost to unit test coverage.
(1) 8901fe7 moves the analytic gradient computation outside the loop over bottom features and stores it throughout all finite differencing computation (extra storage should be pretty minimal since the unit tests use relatively tiny blobs).
(2) 061b0db adds a method
GradientCheckEltwise
that replacesGradientCheckExhaustive
in many layers. In any layer where we have element-wise computation where each output is a function only of the input at the corresponding index (e.g., neuron layers), we only need to do finite-differencing to compute the diagonal of the jacobian and we know all other entries are zero.In my tests of the entire unit test suite (
./build/test/test_all.testbin
on a k40) this gave a 73% speedup from 238,610 ms @ dev down to 66,322 ms.