Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DynamoDB: Import table from S3 #21825

Closed
1 of 2 tasks
LeeroyHannigan opened this issue Aug 30, 2022 · 10 comments · Fixed by #28610
Closed
1 of 2 tasks

DynamoDB: Import table from S3 #21825

LeeroyHannigan opened this issue Aug 30, 2022 · 10 comments · Fixed by #28610
Assignees
@LeeroyHannigan
Labels
@aws-cdk/aws-dynamodb Related to Amazon DynamoDB effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. in-progress This issue is being actively worked on. p1

Comments

@LeeroyHannigan
Copy link
Contributor

Describe the feature

To import data into DynamoDB, your data must be in an Amazon S3 bucket in CSV, DynamoDB JSON, or Amazon Ion format. Data can be compressed in ZSTD or GZIP format, or can be directly imported in uncompressed form. Source data can either be a single Amazon S3 object or multiple Amazon S3 objects that use the same prefix.

CFN ImportSpecification

DynamoDB S3DataImport

Use Case

Feature parity with CFN for newly created feature.

Proposed Solution

No response

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

2.38.1

Environment details (OS name and version, etc.)

Mac OS 12.5
@LeeroyHannigan LeeroyHannigan added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Aug 30, 2022
@github-actions github-actions bot added the @aws-cdk/aws-dynamodb Related to Amazon DynamoDB label Aug 30, 2022
@LeeroyHannigan
Copy link
Contributor Author

This issue can be assigned to me, will work on adding the feature.

@peterwoodworth peterwoodworth added p1 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Aug 30, 2022
@peterwoodworth peterwoodworth added the in-progress This issue is being actively worked on. label Aug 30, 2022
@peterwoodworth
Copy link
Contributor

Thanks a bunch @LeeroyHannigan, please continue any discussions in this issue about design if you have any so the team can comment 🙂

@jarnohenneman
Copy link

Any process been made on this?

@LeeroyHannigan
Copy link
Contributor Author

Currently running tests. Should be pushed by end of the month.

@jarnohenneman
Copy link

Morning @LeeroyHannigan. Any update for us?

@jarnohenneman
Copy link

@LeeroyHannigan apologies for the second ping on this. Any update on the release for this?

@upstartjohnvandivier
Copy link

@peterwoodworth @LeeroyHannigan any update?

@jeremyforan
Copy link

.................any updates? I am asking for @jarnohenneman......and myself.

@sakurai-ryo
Copy link
Contributor

sakurai-ryo commented Jan 7, 2024 •
edited
Loading

I'll take this future request.

@sakurai-ryo sakurai-ryo mentioned this issue Jan 8, 2024
Merged
@mergify mergify bot closed this as completed in #28610 Jan 19, 2024
mergify bot pushed a commit that referenced this issue Jan 19, 2024
@sakurai-ryo
feat(dynamodb): import data from the bucket (#28610)
45b8398 
This PR adds support for importing data from a bucket.
This feature is not supported by `AWS::DynamoDB::GlobalTable`, so it is implemented only in the `Table` construct.

### Details of this feature
Users can create DynamoDB table using data from the bucket.
The data to be imported can be a single object or multiple objects.
This can be selected by specifying the prefix of the key.

Data format must be CSV, DynamoDB JSON, or Amazon Ion.
ZSTD or GZIP compressed data can also be imported.

For CSV files, delimiter and header can be specified as options.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3forDynamoDB.html

In CloudFormation, use this feat as follows.
```yaml
AWSTemplateFormatVersion: "2010-09-09"
Resources:
  MyTable:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        -
          AttributeName: "id"
          AttributeType: "S"
      KeySchema:
        -
          AttributeName: "id"
          KeyType: "HASH"
      TableName: "myTableName"
      ImportSourceSpecification:
        InputCompressionType: NONE # GZIP | ZSTD 
        InputFormat: DYNAMODB_JSON # ION | CSV
        S3BucketSource:
          S3Bucket: cloudfs-s3
          S3KeyPrefix: db
```
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-dynamodb-table-importsourcespecification.html

Closes #21825

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions GitHub Actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LeeroyHannigan LeeroyHannigan

Labels
@aws-cdk/aws-dynamodb Related to Amazon DynamoDB effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. in-progress This issue is being actively worked on. p1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(dynamodb): import data from the bucket sakurai-ryo/aws-cdk
7 participants
@jeremyforan @jarnohenneman @LeeroyHannigan @peterwoodworth @sakurai-ryo @upstartjohnvandivier @Naumel