-
Notifications
You must be signed in to change notification settings - Fork 504
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Add paddle backend in devel #4157
Draft
HydrogenSulfate
wants to merge
46
commits into
deepmodeling:devel
Choose a base branch
from
HydrogenSulfate:add_paddle_backend
base: devel
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 26 commits
Commits
Show all changes
46 commits
Select commit
Hold shift + click to select a range
59b9af5
Add paddle backend code(WIP)
HydrogenSulfate 7f32618
update runnable code with water/se_e2_a
HydrogenSulfate 217bf36
update correct water/se_e2_a code
HydrogenSulfate 0ad720d
fix extra state
HydrogenSulfate 1d7b0d1
Fix typo and bugs
HydrogenSulfate f6eeef6
fix concat
HydrogenSulfate ec021f7
update fix code
HydrogenSulfate 55f71f6
add normalize composite impl
HydrogenSulfate ebdbed2
use prim function instead of vanilla function, supporting
HydrogenSulfate e0aeb73
Merge branch 'devel' into add_paddle_backend
HydrogenSulfate 0896c90
Merge branch 'devel' into add_paddle_backend
HydrogenSulfate 2e79d68
update inference code(WIP)
HydrogenSulfate 482d588
update CMAKE code(WIP)
HydrogenSulfate a3c4663
update CMAKE
HydrogenSulfate 2b4832a
add build script
HydrogenSulfate f1100e4
Update water/se_e2_a + LAMMPS code
HydrogenSulfate 3068869
Merge branch 'devel' into add_paddle_backend
HydrogenSulfate 68a2d62
fix get_pd_version
HydrogenSulfate ff7e0ef
fix read_env.py
HydrogenSulfate 023ba53
fix suffix
HydrogenSulfate 8a1834f
fix pd/cxx_op.py
HydrogenSulfate 396bd54
fix main.py
HydrogenSulfate 66734bc
fix get_item_paddle
HydrogenSulfate ba02ae8
restore in.lammps
HydrogenSulfate 40157dd
restore pyproject.toml
HydrogenSulfate 8d53aec
simplify CMAKE
HydrogenSulfate e39d466
restore c_api.cc
HydrogenSulfate b97571e
fix bugs
HydrogenSulfate 50092c6
change pt -> pd
HydrogenSulfate e02dd11
refactor DeepPotPD.cc
HydrogenSulfate 8343077
update commonPD.h
HydrogenSulfate 09c54f3
remove boost
HydrogenSulfate bc854b2
refine code
HydrogenSulfate 892fd80
update refined infer code
HydrogenSulfate 04b9064
Merge branch 'devel' into add_paddle_backend
HydrogenSulfate b3a6408
remove redundant code
HydrogenSulfate 15a7e75
refine docstring of get_buffer
HydrogenSulfate c792563
Merge branch 'devel' into add_paddle_backend
HydrogenSulfate 72241ea
update pd version code
HydrogenSulfate 8694476
restore non related files
HydrogenSulfate e246a34
add paddle to related docs
HydrogenSulfate 5ee8bcf
optimize cmake paddle macro name
HydrogenSulfate e3c1ceb
update parallel training with paddle backend
HydrogenSulfate 49ba5a5
use 0-D Tensor as buffer shape
HydrogenSulfate f1cae59
support DCU(rocm)
HydrogenSulfate a83fb63
simplify compile code
HydrogenSulfate File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,145 @@ | ||
# SPDX-License-Identifier: LGPL-3.0-or-later | ||
import importlib | ||
import os | ||
import site | ||
from functools import ( | ||
lru_cache, | ||
) | ||
from importlib.machinery import ( | ||
FileFinder, | ||
) | ||
from importlib.util import ( | ||
find_spec, | ||
) | ||
from pathlib import ( | ||
Path, | ||
) | ||
from sysconfig import ( | ||
get_path, | ||
) | ||
from typing import ( | ||
List, | ||
Optional, | ||
Tuple, | ||
Union, | ||
) | ||
|
||
from packaging.version import ( | ||
Version, | ||
) | ||
|
||
|
||
@lru_cache | ||
def find_paddle() -> Tuple[Optional[str], List[str]]: | ||
"""Find PaddlePadle library. | ||
|
||
Tries to find PaddlePadle in the order of: | ||
|
||
1. Environment variable `PADDLE_ROOT` if set | ||
2. The current Python environment. | ||
3. user site packages directory if enabled | ||
4. system site packages directory (purelib) | ||
|
||
Considering the default PaddlePadle package still uses old CXX11 ABI, we | ||
cannot install it automatically. | ||
|
||
Returns | ||
------- | ||
str, optional | ||
PaddlePadle library path if found. | ||
list of str | ||
TensorFlow requirement if not found. Empty if found. | ||
""" | ||
if os.environ.get("DP_ENABLE_PADDLE", "0") == "0": | ||
return None, [] | ||
requires = [] | ||
pd_spec = None | ||
|
||
if (pd_spec is None or not pd_spec) and os.environ.get("PADDLE_ROOT") is not None: | ||
site_packages = Path(os.environ.get("PADDLE_ROOT")).parent.absolute() | ||
pd_spec = FileFinder(str(site_packages)).find_spec("paddle") | ||
|
||
# get paddle spec | ||
# note: isolated build will not work for backend | ||
if pd_spec is None or not pd_spec: | ||
pd_spec = find_spec("paddle") | ||
|
||
if not pd_spec and site.ENABLE_USER_SITE: | ||
# first search TF from user site-packages before global site-packages | ||
site_packages = site.getusersitepackages() | ||
if site_packages: | ||
pd_spec = FileFinder(site_packages).find_spec("paddle") | ||
|
||
if not pd_spec: | ||
# purelib gets site-packages path | ||
site_packages = get_path("purelib") | ||
if site_packages: | ||
pd_spec = FileFinder(site_packages).find_spec("paddle") | ||
|
||
# get install dir from spec | ||
try: | ||
pd_install_dir = pd_spec.submodule_search_locations[0] # type: ignore | ||
# AttributeError if ft_spec is None | ||
# TypeError if submodule_search_locations are None | ||
# IndexError if submodule_search_locations is an empty list | ||
except (AttributeError, TypeError, IndexError): | ||
pd_install_dir = None | ||
requires.extend(get_pd_requirement()["paddle"]) | ||
return pd_install_dir, requires | ||
|
||
|
||
@lru_cache | ||
def get_pd_requirement(pd_version: str = "") -> dict: | ||
"""Get PaddlePadle requirement when Paddle is not installed. | ||
|
||
If pd_version is not given and the environment variable `PADDLE_VERSION` is set, use it as the requirement. | ||
|
||
Parameters | ||
---------- | ||
pd_version : str, optional | ||
Paddle version | ||
|
||
Returns | ||
------- | ||
dict | ||
PaddlePadle requirement. | ||
""" | ||
if pd_version is None: | ||
return {"paddle": []} | ||
if pd_version == "": | ||
pd_version = os.environ.get("PADDLE_VERSION", "") | ||
|
||
return { | ||
"paddle": [ | ||
# uv has different local version behaviors, i.e. `==2.3.1` cannot match `==2.3.1+cpu` | ||
# https://github.com/astral-sh/uv/blob/main/PIP_COMPATIBILITY.md#local-version-identifiers | ||
# luckily, .* (prefix matching) defined in PEP 440 can match any local version | ||
# https://peps.python.org/pep-0440/#version-matching | ||
f"paddle=={Version(pd_version).base_version}.*" | ||
if pd_version != "" | ||
else "paddle>=3.0.0", | ||
], | ||
} | ||
|
||
|
||
@lru_cache | ||
def get_pd_version(pd_path: Optional[Union[str, Path]]) -> str: | ||
"""Get Paddle version from a Paddle Python library path. | ||
|
||
Parameters | ||
---------- | ||
pd_path : str or Path | ||
pd Python library path | ||
|
||
Returns | ||
------- | ||
str | ||
version | ||
""" | ||
if pd_path is None or pd_path == "": | ||
return "" | ||
version_file = Path(pd_path) / "version.py" | ||
spec = importlib.util.spec_from_file_location("paddle.version", version_file) | ||
module = importlib.util.module_from_spec(spec) | ||
spec.loader.exec_module(module) | ||
return module.full_version |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix variable assignment for Paddle version.
The implementation for Paddle support looks good overall, but there's still an issue to address:
On line 128,
pt_version
is assigned instead ofpd_version
. This appears to be a copy-paste error that was not fixed in the previous review.Please apply the following change:
This change will correctly assign the Paddle version.
Committable suggestion