Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

【Hackathon 5th No.23】Add ConcatDataset API to Paddle #57720

Merged
merged 7 commits into from
Nov 17, 2023
Merged

【Hackathon 5th No.23】Add ConcatDataset API to Paddle #57720

merged 7 commits into from
Nov 17, 2023

Conversation

Patrick-Star125
Copy link
Contributor

@Patrick-Star125 Patrick-Star125 commented Sep 25, 2023 •
edited
Loading

PR types

New features

PR changes

APIs

Description

Add ConcatDataset API to Paddle

Link

Rfc PR: PaddlePaddle/community#646
docs PR: PaddlePaddle/docs#6290

待完成

  • RFC文档修改
  • 中文文档

@paddle-bot
Copy link

paddle-bot bot commented Sep 25, 2023

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@paddle-bot paddle-bot bot added the contributor External developers label Sep 25, 2023
@Ligoml Ligoml mentioned this pull request Sep 25, 2023
Open
@Patrick-Star125 Patrick-Star125 changed the title 【Hackathon 5 No.23】Add ConcatDataset API to Paddle 【Hackathon 5th No.23】Add ConcatDataset API to Paddle Sep 28, 2023
@paddle-ci-bot
Copy link

paddle-ci-bot bot commented Oct 3, 2023

Sorry to inform you that 2ae9ff2's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

@paddle-ci-bot
Copy link

paddle-ci-bot bot commented Oct 11, 2023

Sorry to inform you that 1c49d66's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

@paddle-ci-bot
Copy link

paddle-ci-bot bot commented Oct 19, 2023

Sorry to inform you that fac7400's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

@paddle-ci-bot
Copy link

paddle-ci-bot bot commented Oct 27, 2023

Sorry to inform you that 8a051f8's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

lxd-cumt
lxd-cumt reviewed Nov 6, 2023
View reviewed changes
python/paddle/io/dataloader/dataset.py Outdated
s += l
return r

def __init__(self, datasets) -> None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

添加datasets的类型判断

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

具体是对什么类型做判断呢，判断是否为sequence（list、tuple）吗？

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以加上typehint吗，Iterable[Dataset]表示Dataset是一个可迭代对象

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@Patrick-Star125 Patrick-Star125 mentioned this pull request Nov 6, 2023
Merged
lxd-cumt
lxd-cumt reviewed Nov 13, 2023
View reviewed changes
Copy link
Contributor

@lxd-cumt lxd-cumt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code stype check的流水线，可以在push前执行pre-commit，进行代码规范检查

@Patrick-Star125
Copy link
Contributor Author

Patrick-Star125 commented Nov 13, 2023 •
edited
Loading

Codestyle-Check已通过，Coverage不知道为什么没有覆盖，请问有可能是测试没执行的原因吗？

luotao1
luotao1 reviewed Nov 13, 2023
View reviewed changes
Copy link
Contributor

@luotao1 luotao1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image 你看下有没有跑进去，如果跑进去的话，就贴一个本地验证的截图上来

@Patrick-Star125
Copy link
Contributor Author

在线下打点测试是跑进去了的，手动打印堆栈

__init__方法

image

__getitem__方法

image

测试通过

image

lxd-cumt
lxd-cumt approved these changes Nov 15, 2023
View reviewed changes
Copy link
Contributor

@lxd-cumt lxd-cumt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

jeff41404
jeff41404 approved these changes Nov 15, 2023
View reviewed changes
Copy link
Contributor

@jeff41404 jeff41404 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

sunzhongkai588
sunzhongkai588 approved these changes Nov 16, 2023
View reviewed changes
Copy link
Contributor

@sunzhongkai588 sunzhongkai588 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for docs

@luotao1 luotao1 merged commit 1916775 into PaddlePaddle:develop Nov 17, 2023
28 checks passed
SecretXV pushed a commit to SecretXV/Paddle that referenced this pull request Nov 28, 2023
@Patrick-Star125 @SecretXV
【Hackathon 5th No.23】Add ConcatDataset API to Paddle (PaddlePaddle#57720
e7809c3 
)

* add ConcatDataset API

* remove redundant code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@luotao1 luotao1 luotao1 left review comments

@jeff41404 jeff41404 jeff41404 approved these changes

@lxd-cumt lxd-cumt lxd-cumt approved these changes

@sunzhongkai588 sunzhongkai588 sunzhongkai588 approved these changes

Assignees

@luotao1 luotao1

@jeff41404 jeff41404

@lxd-cumt lxd-cumt

@sunzhongkai588 sunzhongkai588

Labels
contributor External developers PaddlePaddle Hackathon
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@Patrick-Star125 @luotao1 @jeff41404 @lxd-cumt @sunzhongkai588