Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gradient check speedups #363

Merged
merged 2 commits into from
Apr 25, 2014
Merged

Conversation

jeffdonahue
Copy link
Contributor

Two changes to the gradient checker that improve performance pretty significantly without any cost to unit test coverage.

(1) 8901fe7 moves the analytic gradient computation outside the loop over bottom features and stores it throughout all finite differencing computation (extra storage should be pretty minimal since the unit tests use relatively tiny blobs).

(2) 061b0db adds a method GradientCheckEltwise that replaces GradientCheckExhaustive in many layers. In any layer where we have element-wise computation where each output is a function only of the input at the corresponding index (e.g., neuron layers), we only need to do finite-differencing to compute the diagonal of the jacobian and we know all other entries are zero.

In my tests of the entire unit test suite (./build/test/test_all.testbin on a k40) this gave a 73% speedup from 238,610 ms @ dev down to 66,322 ms.

@shelhamer
Copy link
Member

Testing in a minute is sweet! Now all I have to do is bring up the build bot and our tests will be perpetual.

Thanks Jeff!

shelhamer added a commit that referenced this pull request Apr 25, 2014
@shelhamer shelhamer merged commit c6f1011 into BVLC:dev Apr 25, 2014
@jeffdonahue jeffdonahue deleted the speedup-gradient-check branch April 25, 2014 20:53
@shelhamer shelhamer mentioned this pull request May 20, 2014
mitmul pushed a commit to mitmul/caffe that referenced this pull request Sep 30, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants