Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(python): account for both multi-process and distributed torch workers #2761

Merged
merged 9 commits into from
Aug 27, 2024

Conversation

tonyf
Copy link
Contributor

@tonyf tonyf commented Aug 20, 2024

Resolves #2713

Copy link

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

@tonyf tonyf changed the title [torch][dataset] Account for both multi-process and distributed torch workers fix: Account for both multi-process and distributed torch workers in torch LanceDataset Aug 20, 2024
@github-actions github-actions bot added the bug Something isn't working label Aug 20, 2024
@eddyxu eddyxu changed the title fix: Account for both multi-process and distributed torch workers in torch LanceDataset feat(python): account for both multi-process and distributed torch workers Aug 20, 2024
@github-actions github-actions bot added the enhancement New feature or request label Aug 20, 2024
@AyushExel AyushExel self-requested a review August 26, 2024 07:25
Copy link
Contributor

@AyushExel AyushExel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks. Can you fix the linting error

@wjones127 wjones127 removed the bug Something isn't working label Aug 26, 2024
@AyushExel AyushExel merged commit 37df1d5 into lancedb:main Aug 27, 2024
11 checks passed
gagan-bhullar-tech pushed a commit to gagan-bhullar-tech/lance that referenced this pull request Sep 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request python
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sharded samplers don't account for dataloader's multiprocess workers
3 participants