Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement DynamoDBRead class. #458

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

nasudadada
Copy link

Summary

  • Add functionality to read data from DynamoDB tables and output as CSV or JSONL

Key features

  • Specify table name, output directory, file name, and file format
  • Support for CSV and JSONL output formats

This commit enables users to efficiently export data from DynamoDB tables.

Brief

This PR adds a new feature to read data from DynamoDB tables and export it as CSV or JSONL format. Key features include:

  1. User-specifiable parameters:

    • Table name
    • Output directory
    • File name
    • File format (CSV or JSONL)
  2. Supported output formats:

    • CSV
    • JSONL

The current implementation uses a scan operation to retrieve all data stored in the table. This feature addition enables users to export data from DynamoDB tables.

Points to Check

  • Following other implementations, exception handling and logging have been kept minimal. Is this approach acceptable?
  • Are there any implementation aspects missing from this OSS implementation? Please check if I'm overlooked any considerations, particularly regarding data consistency or performance.
  • Given that we're using a scan operation to retrieve all data, please confirm if this method is appropriate in terms of impact and performance for large tables.
  • Does this implementation align with the style of this OSS project?

Test

Confirmed

  • Added test code to verify proper functionality.
  • Confirmed file output works correctly using the feature in actual scenarios.

Review Limit

None

Additional Information

  • In the current implementation, we are only using scan operations, but I am considering applying query operations with partition keys and sort keys when the opportunity arises.:
    • Adding partition key and sort key parameters to the interface to allow data filtering
    • Adding support for query operations

- Add functionality to read data from DynamoDB tables and output as CSV or JSONL

Key features:
- Specify table name, output directory, file name, and file format
- Flexible AWS authentication setup (region, access key, secret key, profile)
- Support for CSV and JSONL output formats

This commit enables users to efficiently export data from DynamoDB tables.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant