use elementwise to optimize gelu backward implementation on GPU #38263
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
|
PR名字描述稍微详细一点，跟上个PR对应 |
Zjq9409
changed the title
Done. |
ZzSean
reviewed
Dec 21, 2021
paddle/fluid/operators/gelu_op.cu
Outdated
| tanh(kAlpha * x * (one + static_cast<MPType>(0.044715) * x * x)); | ||
| auto ans = | ||
| half * x * ((one - tanh_out * tanh_out) * | ||
| (kAlpha + static_cast<MPType>(0.1070322243) * x * x)) + |
paddle/fluid/operators/gelu_op.cu
Outdated
| auto tanh_out = | ||
| tanh(kAlpha * x * (one + static_cast<MPType>(0.044715) * x * x)); | ||
| auto ans = | ||
| half * x * ((one - tanh_out * tanh_out) * |
JamesLim-sy
reviewed
Dec 21, 2021
paddle/fluid/operators/gelu_op.cu
Outdated
| @@ -12,9 +12,76 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |||
| See the License for the specific language governing permissions and | |||
| limitations under the License. */ | |||
|
|
|||
| #include "paddle/fluid/operators/amp/fp16_type_traits.h" | |||
| #include "paddle/fluid/operators/elementwise/elementwise_op_broadcast.cu.h" | |||
| #include "paddle/fluid/operators/gelu_op.h" | |||
| #include "paddle/fluid/platform/float16.h" | |||
There was a problem hiding this comment.
这个头文件引用已经在paddle/fluid/operators/amp/fp16_type_traits.h 引用过了，可以删除
Zjq9409
commented
Dec 22, 2021
paddle/fluid/operators/gelu_op.cu
Outdated
| (one + tanh(static_cast<MPType>(0.79788456) * x * | ||
| (one + static_cast<MPType>(0.044715) * x * x))); | ||
| MPType half = static_cast<MPType>(0.5); | ||
| MPType decimal = static_cast<MPType>(0.044715); |
There was a problem hiding this comment.
// gelu(x) = 0.5 * x * (1 + tanh(sqrt(2 / \pi) * (x + 0.044715 * x^{3})))
gelu的公式，其中“0.044715”是一个固定的常量值
There was a problem hiding this comment.
0.044715
可以用宏定义的方式表达这个常量值, 比如：
#define GELU_CONSTANT 0.044715
同时将这个宏定义放在通用文件夹中，比如gelu_op.h 中，同步修改使用了0.044715 的代码
JamesLim-sy
reviewed
Dec 22, 2021
paddle/fluid/operators/gelu_op.cu
Outdated
| MPType dout = static_cast<MPType>(arg_dout); | ||
| MPType one = static_cast<MPType>(1); | ||
| MPType half = static_cast<MPType>(0.5); | ||
| MPType decimal = static_cast<MPType>(0.044715); |
paddle/fluid/operators/gelu_op.cu
Outdated
| (one + tanh(static_cast<MPType>(0.79788456) * x * | ||
| (one + static_cast<MPType>(0.044715) * x * x))); | ||
| MPType half = static_cast<MPType>(0.5); | ||
| MPType decimal = static_cast<MPType>(0.044715); |
There was a problem hiding this comment.
0.044715
可以用宏定义的方式表达这个常量值, 比如：
#define GELU_CONSTANT 0.044715
同时将这个宏定义放在通用文件夹中，比如gelu_op.h 中，同步修改使用了0.044715 的代码
paddle/fluid/operators/gelu_op.cu
Outdated
| MPType kBeta = kAlpha * decimal * static_cast<MPType>(3); | ||
| auto tanh_out = tanh(kAlpha * x * (one + decimal * x * x)); | ||
| auto temp = (one - tanh_out * tanh_out) * (kAlpha + kBeta * x * x); | ||
| auto ans = half * x * temp + half * (one + tanh_out); |
There was a problem hiding this comment.
这部分的计算中应该存在多次重复计算的参数，比如:x^3 ，可以把这类参数挑出来，减少计算量
zmxdream
pushed a commit
to zmxdream/Paddle
that referenced
this pull request
Dec 25, 2021
…lePaddle#38263) * optimize gelu backward * optimize gelu backward * optimize code * Number to expression * Replacement number
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Performance optimization
PR changes
OPs
Describe
使用elementwise优化gelu算子GPU反向计算，前向计算+反向计算优化后性能数据如下：
