Incremental compilation optimization of phi #59811
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
你的PR提交成功，感谢你对开源项目的贡献! |
risemeup1
changed the title
risemeup1
changed the title
risemeup1
changed the title
risemeup1
changed the title
risemeup1
changed the title
risemeup1
changed the title
… mv_.cc_from_phi_to_fluid
risemeup1
changed the title
chenwhql
reviewed
Dec 11, 2023
| @@ -15,8 +15,8 @@ limitations under the License. */ | |||
| #include <iostream> | |||
|
|
|||
| #include "gtest/gtest.h" | |||
| #include "paddle/fluid/operators/ops_signature/signatures.h" | |||
There was a problem hiding this comment.
这两个单测要不要也移到fluid
收到
zyfncg
approved these changes
Dec 11, 2023
YuanRisheng
approved these changes
Dec 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
others
PR changes
others
Description
背景：
phi增量编译存在的问题：
phi作为独立编译模块，理论上phi目录下的所有符号都应该编译到libphi.so中，但是phi/kernels/ops/compat/*.cc文件并没有编译在phi中，而是在paddle/fluid/operators/generator/中编译，故将这些文件放在这fluid目录下更加合理，也是导致phi重新编译的原因之一
phi/kernels/fusion/下面有多个通过一些.py文件生成大量.cu文件的操作(约150个)，cmake写法都是通过execute_process生成，每次cmake构建阶段都会重新生成，时间戳发生变化，导致phi下这些kernel文件都会重新编译(约20min)，也是增量编译耗时过长的原因，应设置tmp文件配合copy_if_different或者用add_custom_command，add_custom_target命令封装在make阶段编译进行优化。
不仅仅是phi/kernels目录，往往改了很简单的文件，phi基本上都会从0开始编译，主要原因是phi/CMakeLists.txt/下调用了phi.cmake中的generate_unify_header函数，此函数中使用了一些string(replace xxx)操作导致cmake阶段检测到phi下边的文件几乎都有变化，故需要使用更加规范的cmake代码进行重构
还有一些其它地方也是类似于生成代码的逻辑都会在增量编译存在这个问题，需要添加合适的跳过机制
收益：
本地测试(a100机器，cpu核心数159)，增量编译可以受益约15min，p40收益约30min，不同机器可能有差异
CI上也普遍存在这个问题，优化后CI配合ccache也有较大收益
优化之后，只有改phi或者common(phi依赖基础组件common)中的文件，phi才会重新编译(且只编译phi目录下改动的某个.cc文件)，若改了phi依赖的库文件，只会重新Linking CXX shared library libphi.so，不会重新编译phi，符合预期
To do
fluid等其它目录也存在这样的问题，此PR之后优化fluid目录
添加cmake代码相关监控
Pcard-67010