Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CTGAN optimization (Memory Usage) #24

Closed
MooooCat opened this issue Sep 26, 2023 · 0 comments · Fixed by #35
Closed

CTGAN optimization (Memory Usage) #24

MooooCat opened this issue Sep 26, 2023 · 0 comments · Fixed by #35
Assignees
Labels
difficulty-hard good first issue Good for newcomers help wanted Extra attention is needed

Comments

@MooooCat
Copy link
Contributor

MooooCat commented Sep 26, 2023

🚅Search before asking

I have searched for issues similar to this one.

🚅Description

In the current implementation of CTGAN, all training data will be loaded into memory at once, which will greatly increase memory consumption.

Thanks.

🏕Solution(optional)

When using csv files to store training data, the current implementation uses pd.dataframe to load the data once. I plan to load the batch data in CTGAN in real time, and load other batch data after the use is completed.

🍰Detail(optional)

There will be no parameter changes to the current implementation.

🍰Example(optional)

This optimized implementation will not change the existing example code.

@MooooCat MooooCat added good first issue Good for newcomers help wanted Extra attention is needed difficulty-hard labels Sep 26, 2023
@MooooCat MooooCat self-assigned this Sep 26, 2023
@MooooCat MooooCat changed the title CTGAN optimization () CTGAN optimization (Memory Usage) Sep 26, 2023
@MooooCat MooooCat pinned this issue Sep 26, 2023
@MooooCat MooooCat assigned Guo-Yunzhe and unassigned MooooCat Sep 26, 2023
@MooooCat MooooCat linked a pull request Oct 30, 2023 that will close this issue
7 tasks
@Wh1isper Wh1isper unpinned this issue Dec 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
difficulty-hard good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants